Reverse proxy: Traefik on demand — implementation plan #281

New issue

Open

opened 2026-11-01 08:04:49 +00:00 by kayg · 192 comments

kayg commented 2026-11-01 08:04:49 +00:00

Owner

Copy link

Reverse proxy on demand: engine-neutral implementation plan

For implementers after owner approval: use the approved execution workflow, read the linked design decisions and execute the gated slices below task by task. This document authorizes planning only.

Status: P5 proposal for kayg to critique before code. The engine choice is OPEN following the owner's Caddy question. #261 comment 72438 selected Traefik earlier; this plan does not silently extend that selection to the reopened comparison. Keep the requested issue title for continuity.

Goal: expose an authorized service through a managed HTTPS name, downloading and running a proxy only on nodes with enabled exposures.

Architecture: the infraternal server owns desired state, authorization and audit; the local infraternal agent acquires and supervises one external proxy process per serving node. A narrow engine adapter renders our exposure model, applies revisions and reports observations. Only the selected adapter ships initially; this is a boundary for choice and future migration, not a commitment to maintain two engines.

Stack: existing Rust action/plugin framework, mesh coordinator and update trust; selected external Go executable; existing TypeScript product shell and shared UI. No new foundational library is selected here. All new packages/crates remain AGPL-3.0-only; review the engine and module licenses separately.

Inputs: D316 and repository decisions, R2 design, earlier owner selection, J20 measurements, edge updates #211, mesh DNS #266. Snapshot: origin/dev at f4b61cd6c3f52078cad7573c5367d7a6e75f84f0, branch research/p5-traefik-plan. No implementation changes.

Owner grill, recommendation first

1. Engine: recommend waiting for the Caddy extension of J20, then selecting one engine and one provider-capable artifact set. Traefik has measured zero-loss reloads and stock DNS integrations; Caddy offers a transactional local configuration API but DNS-01 adds a custom-build supply chain. Neither advantage settles speed or memory. Caddy has not been measured in the supplied J20 results. Does kayg prefer Traefik or Caddy after reviewing matching workload, module-enabled binary size/RSS, reload and certificate results? Do not start engine-specific implementation before this answer.
2. Trust: recommend an infraternal-signed component manifest and immutable release artifact mirror for either engine. Verify available upstream signatures in release CI; never represent our signature as an upstream signature. Traefik's inspected release lacks detached asset signatures. If Caddy wins, recommend CI-owned, pinned xcaddy builds with an approved DNS module set rather than dynamically building on hosts. Is this acquisition trust and build responsibility acceptable?
3. Scope/access: recommend mesh-only HTTP/HTTPS exposures first, bound to the mesh address, with exact host matching and existing mesh rules enforced per original service. A DNS answer alone cannot authorize traffic. Public/LAN listeners need an explicit audience and authentication decision. Should those ship in the first release or follow separately?
4. Names/TLS: recommend .internal remain the default mesh suffix; publicly trusted TLS requires an optional owned domain and DNS-01. For .internal, offer explicit private-CA TLS only after its issuance/trust-distribution contract is approved, or explicitly chosen HTTP. Never imply a public CA can issue .internal. Should private-CA TLS ship in slice 1, and which existing PKI authority is permitted to issue it?
5. Ports: recommend rootless high ports (8080/8443 proposed), with an explicit privileged mapping option for conventional URLs. High ports appear in the URL. Prefer narrow host-managed port mapping to granting the proxy process capabilities; Linux capability support is an opt-in alternative. Approve this default and the privileged mapping UX for Linux/macOS?
6. Custody: recommend node-sealed DNS credentials, only on the issuing node, with least-privilege challenge-zone tokens; later an OpenBao-backed reference uses the same custody interface. This follows the owner's agreement in principle, not a claim an unattended sealed store is implemented. Approve using the platform/node sealing contract for unattended renewal, and requiring explicit re-enrollment after node-key loss?
7. Lifecycle: recommend stopping the process when enabled exposure count reaches zero, retaining its private certificate state, and checking renewal immediately when re-enabled. This saves idle memory but cannot renew while stopped. Re-enabling after a long stop may wait for issuance; never fall back silently to HTTP. Accept that behavior, a proposed 10-second drain deadline, and the #211 adoption dependency for seamless infraternal updates?
8. CLI: recommend expose <machine> <service> --name <dns-name>, plus expose list, expose status <id>, expose remove <id>, and --json everywhere via D16/D314. Is expose the right noun/verb family? Details remain an action-schema grill, not a second command implementation.
9. Budgets: recommend the provisional thresholds below as regression review triggers, then ratchet actual measured selected-engine results. Approve them after Caddy measurements rather than inheriting an unmeasured cap or loosening D41's infraternal-agent RSS budget.

Evidence and engine-specific differences

J20 tested Traefik 3.7.17, executable 182,120,608 bytes, idle RSS 93.30 MiB, warm 95.77 MiB, 1,000 held connections 150.27 MiB, HTTP high-water 265.84 MiB. Idle CPU sampled 0.00%. Three reload trials lost 0 / 0 / 0 requests; route switch 27 / 561 / 1041 ms. These are standalone Linux x86 results, not macOS numbers or a universal zero-loss guarantee. HTTP/1 c64 was 32,956 req/s, p99 7.296 ms under J20's allocation and generator ceiling. Caddy cells stay unmeasured, including provider-enabled builds.

Concern	Traefik adapter	Caddy adapter
DNS-01	Supported provider integrations compiled into stock binary; configure named ACME resolvers. Pin and test our supported provider subset.	Stock release does not include the needed provider modules. Pin core + each DNS module commit/version + dependency closure + Go/xcaddy toolchain; build approved variants or one bounded module bundle in release CI.
Acquisition	Official versioned archive from github.com/traefik/traefik/releases, reviewed hashes, then our signed manifest/mirror. R2 inspected 3.7.17 without detached checksum signatures. Recheck the exact chosen release assets.	Official stock releases have documented checksum-signature verification. A custom download/build-service artifact is a different artifact and does not inherit the stock signature. Prefer our reproducible CI build, SBOM, provenance and release signature; build-service use needs independently pinned resulting bytes and verified provenance. No live module selection/download service at install time.
Configuration/reload	Static startup config for listeners/providers/resolvers; full dynamic snapshot in a watched directory. File changes are asynchronous and provider events throttled (documented default 2 s); no successful write acknowledgment proves application. Static changes require controlled restart.	Native JSON; whole config POST /load through a private permissioned Unix socket. API blocks until success/failure and documents rollback on failure without downtime. A JSON file is a durable source, not an automatic file watcher; caddy reload also uses the admin API. With admin disabled, use a controlled restart and report that interruption.
Admin surface	Disable API/dashboard/insecure mode, unused discovery providers, debug template dumping, remote plugin loading and telemetry/version checks. No api@internal route. Private ping only, separate from data listeners.	Replace default localhost:2019 with a permissioned private Unix socket, or disable admin if accepting restart-only changes. No public or mesh binding, remote admin, arbitrary module/config API or proxying the socket. Disable unintended automatic port-80 listeners and unapproved on-demand TLS.
TLS storage	Dedicated acme.json with mode 0600 and exclusive writer. Never share it among nodes.	Dedicated private data/storage directory holding accounts and certificate keys, not acme.json; treat all contents as secret and prohibit arbitrary storage modules.
Memory	Above J20 measurements; process only while active.	Pending parallel matched measurement with the exact approved DNS modules. Do not extrapolate stock-binary memory.

Sources: Traefik file provider, Traefik ACME, Traefik API/dashboard; Caddy JSON API, Caddy DNS challenge, Caddy build instructions, Caddy provider modules, Caddy asset signatures. Documentation describes capabilities; J20 supplies our measurements. Freeze versioned docs/source alongside the selected release recipe.

1. Acquisition, installation and release coupling

One signed component manifest per infraternal release binds release identity/channel, component ID, engine version, OS/architecture, module-set digest, source URL, archive/executable SHA-256, exact sizes, configuration/storage compatibility versions and license/provenance receipts. Release CI verifies upstream evidence where present, scans advisories/dependencies/licenses and signs the approved resulting bytes using the existing release trust policy. Keep identity-bearing manifest verification separate from a bare checksum list. No latest, host-side xcaddy, automatic engine self-update or arbitrary URL input.

The existing release_verify.rs::verify_release_file and release_manifest.rs::verify_release_manifest are useful trust precedents, not drop-in component installers: the former hard-caps payloads at 128 MiB, below measured Traefik size, and the latter authenticates infraternal release identity. Propose a separate bounded component payload policy, initially maximum 256 MiB archive / 512 MiB extracted executable, exact signed lengths stricter than those caps. Do not increase the existing application-update limit globally. Extract verifier primitives into a shared small module only where both server and infraternal agent genuinely need them; add no server/SQLx dependency to host-only builds.

On first enabled exposure: acquire an installer lock, stream to a private staging file, enforce time/byte/disk bounds, verify signed manifest and archive digest before extraction, extract exactly the allowlisted executable, verify its digest/target, then fsync and rename within the destination filesystem. Reject absolute/parent paths, duplicate entries, symlinks/hardlinks, special files, bombs and wrong-architecture artifacts. Verify the same immutable file ultimately executed; the install parent must not be writable by other users. No shell expansion or untrusted executable search path.

Use the existing resolved infraternal data root: rootful /var/lib/infraternal, rootless ~/.local/share/infraternal where applicable, and the established macOS resolver rather than assuming Linux paths. Proposed layout: components/proxy/<artifact-digest>/bin, proxy/config/, proxy/sealed/, proxy/tls/, and private runtime socket directory. Directories 0700; immutable executable 0500; manifests/config/certificates 0600. Keep secrets separate from engine configuration history.

D251/#211: stage the component selected by the candidate infraternal release before activating it on an already exposing node; keep the current working process/artifact until candidate readiness. Inactive nodes retain the manifest and need no download. Rollback selects the previous release's pinned artifact and compatible rendered config, never an arbitrary old engine. Signed downgrade authorization follows existing release rules. Retain previous artifact and metadata until rollback window ends; never garbage-collect the active binary. Cert storage compatibility is a gate: use a quiesced protected snapshot or approved forward/backward-compatible migration; never overwrite newer ACME account/cert state with a stale backup blindly.

Air-gapped install imports a release-signed component bundle from a file through exactly the same checks, no bypass flag. A verified mirror/cache is optional and content-addressed. Full isolation cannot contact public ACME or provider APIs: support explicit pre-provisioned certificates or approved private PKI, show renewal responsibility and expiry, and never promise public DNS-01 will work offline. Packaging must include engine/module license notices and verification evidence.

2. Supervision, limits and health

Desired enabled exposure count, not browser attachment or traffic, determines process lifetime. State machine: inactive → acquiring → verifying → starting → ready, with applying, degraded, retrying, draining, and typed failed observations. Download failure leaves desired revision pending, with a retry action. Disabled rows and cached artifacts alone do not start a process.

One owned process per node, not per service/session. Start with an absolute verified path, allowlisted environment, fixed working directory and restrictive umask. Acquire a node-scoped ownership lock before bind. Readiness requires process survival, private control/ping readiness and a synthetic expected-route/certificate probe; distinguish engine-ready, upstream-unhealthy and certificate-pending. Stop: retire routes/deny new traffic, signal graceful drain, wait the proposed 10 s, terminate the owned process group if needed, reap and remove private runtime material. When re-enabled, inspect retained certificates immediately before reporting ready.

Proposed crash backoff: 200 ms doubling to 30 s with jitter, reset after 60 s healthy; five failures in 60 s enter a cooled-down failure state with bounded automatic retries and explicit Retry. Retry timers exist only while enabled exposures require the engine. Never download again on every crash. Failed reload keeps the last permitted working revision; a security revocation is different and must fail closed even if that requires withdrawing a listener.

Linux: use existing process/descriptor ownership primitives, child rlimits and, when delegated, a cgroup for hard memory/task bounds. Proposed active-process MemoryMax review budget 512 MiB, file descriptor cap 8192 and capped connection/inflight settings; test overload behavior before approval. macOS: use supported rlimits and process-group/launchd lifecycle integration; do not claim Linux-style hard resident-memory enforcement exists there. A Go memory target is advisory. Observe RSS and pressure; report limit support truthfully. Do not impose process-wide rlimits on the infraternal agent merely to constrain its child.

Abrupt infraternal-agent exit and update ownership need real tests on both platforms. Linux parent-death/cgroup cleanup and macOS service-manager behavior differ. Track process identity and owned control descriptors; never kill/re-adopt a numeric PID alone. #211 generation handoff must transfer exclusive child ownership, log readers and the immutable component/config revision, or explicitly drain/restart. No candidate callbacks may alter exposure state before adoption commits. Seamless proxy survival across host updates is a gate dependency, not implemented in current dev. If unmanaged macOS SIGKILL leaves an orphan, require a scoped cleanup owner or surface that limitation before shipping; do not hand-wave portability.

Rootless listeners bind above 1024. DNS-01 requires no inbound port 80. For conventional 80/443, propose an explicit narrowly scoped OS port mapping on Linux/macOS, audited install/removal without touching unrelated rules. Linux CAP_NET_BIND_SERVICE can be an owner-approved alternative on the verified executable or constrained launcher; it grants low-port bind broadly, needs reapplication after replacement, and must not spread to the infraternal binary. macOS has no Linux file capabilities; use an approved privileged mapping/helper or high ports. Router PCP/NAT-PMP/UPnP mapping is a separate exposure consent, not a substitute for local privileged bind. No silent root promotion or global unprivileged-port sysctl change.

Engine stdout/stderr feed the existing infraternal log path with node/exposure/revision IDs, bounded line size/queue/rate and dropped-count summaries. Never forward arbitrary raw errors/configs to the browser. Keep only safe structured fields; redact Authorization/Cookie/query credentials, never log provider tokens/private keys. Access logs opt-in/minimized; health and certificate observations are pushed on change, not UI polling at idle. A proposed 30 s private active-engine health probe is measured separately from zero inactive wakeups.

3. Product model and adapter contract

Proposed ExposureSpec: stable exposure ID, owning node, stable service target (protocol/address/port), normalized exact DNS name, optional literal path prefix, listener audience/port, TLS mode (disabled, managed_dns, provided, future approved private PKI), certificate/provider references, access-rule reference, enabled flag and monotonically increasing desired revision. Status separately records applied revision, engine artifact ID, process/upstream/certificate states, expiry, safe error code and next action. No credentials in either public model. Product copy uses rules, following D252; the requested “policy” concept maps to those existing rules.

Server authorizes the actor, node, service and audience; normalizes/validates the whole transaction, stores it and audits intent. Node validates the authenticated revision and target bindings again. No raw engine JSON/YAML, arbitrary expressions, templates, file paths, exec hooks, middleware snippets or module names in the public API. Exact names are canonicalized using the existing mesh name rules; initially reject wildcards, overlapping host/path claims and credentials embedded in URLs. TCP/UDP and arbitrary middleware are separate future scope.

Proposed adapter boundary in the proxy plugin: render(snapshot, capabilities) -> RenderedRevision; apply(rendered) -> Applied | Pending | Rejected; observe() -> ProxyObservation; plus lifecycle startup requirements. RenderedRevision identifies nonsecret config bytes/hash, secret references and whether restart is required. Capabilities explicitly report compiled DNS providers and supported limits. Only one writer applies snapshots serially; stale generations cannot supersede a newer one.

Traefik: serialize a complete dynamic snapshot to a temporary sibling outside the watched directory, fsync, rename to the one canonical watched file and fsync the directory. This avoids mixed multi-file revisions. Watch the parent directory rather than a replaced-file mount. Keep a durable desired revision and last-known-good rendered snapshot. Because watch events are asynchronous/coalesced, observe expected route/cert behavior before acknowledging applied; process health alone is insufficient. API/dashboard stay disabled. Test a private synthetic revision route restricted to local probing if needed; do not expose a management route to clients. Startup-only resolver/listener changes are marked restart-required in status.

Caddy: serialize native JSON atomically for durable recovery, validate with the exact installed build, then submit the whole snapshot via private Unix-socket /load. One successful response is the engine acknowledgment, followed by route/TLS probes. Use our durable committed desired/applied revisions as authority; either disable autosave or protect and reconcile it, never blindly --resume a stale config after revocation. Disable the admin endpoint only if the owner accepts restart-only application. File-only mode is not claimed to hot-reload automatically. Long-lived WebSocket and HTTP/2 behavior must be measured for the chosen stream-drain settings.

For either engine, invalid nonsecurity edits retain the last working authorized state and report desired/applied mismatch. Revocation first removes authorization at the dataplane enforcement boundary and closes forbidden live flows; restoration of a last-known-good config must never restore revoked access. Partial writes, crash between persistence/apply and rapid rename/edit/remove races must converge deterministically after restart.

4. Certificates and custody

Managed custom-domain TLS uses engine DNS-01, explicitly disabling unwanted HTTP-01/TLS-ALPN fallback and implicit public listeners. The optional owned suffix is editable, while .internal can remain available as the default alias per D316. DNS-01 needs public authoritative TXT control even if A/AAAA records and services exist only on mesh. Normalize suffix boundaries, authorize domain use, and explain propagation/CAA/rate-limit/account/provider failures with a safe retry action. Stage new names/certificates before removing an old working custom-domain route where rules permit; suffix changes must also retire old DNS and access state coherently.

Provider tokens are sealed at rest and delivered only to the designated issuing node over the existing authenticated channel. Store only opaque references/status centrally. Scope tokens to the challenge zone; offer delegated _acme-challenge zones where supported. Unseal just for the owned process, via provider-supported credential files or private config; environment-only providers require an explicitly documented same-user/root inspection risk and scrubbing, never argv. Use file-backed provider options where supported, not a claim every provider supports them. Rotation writes new private material, validates issuance/renewal, switches, then retires old material. Node migration requires explicit custody reassignment, no mesh-wide replication of tokens.

Unattended renewal cannot depend on the user's passkey being present. Node sealing requires a reviewed persistent machine key/platform protection and recovery semantics; do not repurpose the passkey-wrapped SSH vault as if it were already suitable. OpenBao later implements the same credential reference → node-local material boundary; no new secret engine is selected by this plan.

Traefik owns mode-0600 acme.json in a 0700 directory, one writer, protected backups; Caddy owns an equivalently private storage tree. Both contain private keys and ACME accounts. Protect path ownership, no symlinks, no export in diagnostics, no unrestricted backup attachments. Backups are encrypted and explicitly included in disaster recovery scope. Certificate metadata extraction must not return key bytes. Never share a mutable engine certificate store across nodes. Stopping the last exposure stops renewal; retained certificates are checked on next activation, with safe issuance before ready and visible expiry failures.

.internal is not publicly registrable. Do not make public ACME attempts for it and do not silently install an engine-generated root CA into OS/browser trust. Private certificate issuance, separate server-auth EKUs and trust distribution require owner-approved integration with existing PKI; mesh/device identity keys must not be reused as website keys. Provided certificates are an explicit alternative with expiry/status tracking.

5. Mesh names and enforcement

Reuse crates/infraternal-mesh/src/names.rs::NameTable, coordinator canonical claims and #266's negotiated, revisioned name snapshots. #266 is active work, not a guarantee those listeners/suffix APIs are in this base. D316 requires <hostname>.internal and an optional custom suffix; a service also needs a unique routing key. Propose one root service per host name initially, explicit path prefix or explicit service alias for extras, with owner confirmation in the contract slice. DNS returns the actual serving node's mesh address, not an unreachable upstream or invented global gateway.

Use the same accepted mesh rule revision for name visibility, exposure authorization and dataplane admission. Denied peers must learn no protected aliases from a proxy status response. DNS denial is insufficient: direct-IP Host/SNI requests, stale DNS caches and existing connections must still be denied. Shared HTTPS 443/8443 also cannot collapse distinct service-port rules into “can reach the proxy”. Render per-route permitted peer sources from the original service rule, enforce mesh identity at ingress, and retain backend egress restrictions. Mesh source addresses are trusted only on the authenticated mesh interface. Never trust client-supplied X-Forwarded-For/Forwarded headers as identity; configure engine source matching accordingly.

Tests must prove a peer permitted for service A cannot reach B on the same listener, even with forged Host/SNI/forwarded headers or via an allowed backend. Address reassignment must not inherit an old peer's privileges. For requests requiring person-level authentication or public access, mesh IP is not enough: integrate an approved identity flow separately or deny. Close existing forbidden flows on revocation under the mesh cutoff contract, and fail closed when a newly accepted rule revision cannot be installed. Preserve last accepted rules during coordinator outage only as already allowed by the mesh contract; do not invent expiry semantics here.

6. UI and CLI proposal

Only a behavior brief here; D171 leaves design/mockups to the orchestrator. Read TASTE.md; reuse packages/ui components (entity-chip, pane-stepper, status-line, spinner, toast, chip-input, focus/keyboard helpers), not new primitives. Owner-approved mockups precede UI implementation.

Entry point beside a service in the host's existing view: Expose service. Show the exact service target, choose name/suffix, TLS and who can reach it, then preview the full URL including nonstandard port and audience before enabling. Domain/provider setup appears only when required. First activation shows downloading → verifying → starting → getting certificate → ready in the service's own surface. A pending certificate must not appear ready merely because the child is alive.

Exposure details show copy/open URL, desired/applied state, upstream health, certificate issuer/expiry/renewal state and a plain-language next action for errors. Disable/remove and domain/access edits stay beside the exposure. Provider secret controls show configured/rotate/remove, never reveal. Errors use appropriate icons, retry near the failed step, pending approvals in the existing toast/Inbox, and technical paths/log details behind Details. No engine branding/schema/admin dashboard in the normal flow. On mobile use one column, 44 pt targets, no hover-only controls, no colliding labels or long URL overflow; ink/paper and empty/loading/offline/retry/expired/revoked states get equal care.

CLI is the same registered action schema as UI/API/MCP, not a parallel parser or management channel. Proposed commands are in grill Q8. --json uses the existing envelope, stable typed errors and desired/applied revisions; an accepted mutation returns pending status rather than waiting indefinitely for DNS. Credentials use secure input/file-reference mechanisms, never command arguments. A read-only list/status command must not trigger acquisition. Add usage/examples and action availability/permissions through ActionDefinition/ActionRegistry and existing CLI output helpers.

7. Security review and budgets

Independent review focus: supply-chain authenticity vs hashes; malicious archives and replacement after verification; host/path/config injection; SSRF/unauthorized upstreams; shared-listener authorization bypass; revocation on live WebSockets/h2; secret custody and raw error leakage; ACME abuse; child/adoption ownership and privileged mapping; storage rollback/key loss. No arbitrary dynamic engine modules, discovery socket mounts, public admin endpoints or unbounded API/config bodies.

Required malicious cases: backticks/quotes/newlines in DNS/path/header fields, escaped separators/IDNA and suffix confusion, duplicate names, ../absolute/symlink archives, oversized compressed/uncompressed inputs, wrong target/digest/key/replayed release, writable-parent executable swap, Unix-socket ownership attacks, forged proxy headers, unknown SNI/default virtual host, loopback/admin/metadata upstream SSRF, malicious provider error containing a canary secret, cancellation after commit, concurrent enable/remove/update, resource exhaustion and disk-full/fsync failure. Parser/config limits are fuzzed; no invalid input creates a partially authorized route.

Budget distinction: external engine adds +0 embedded engine bytes to the infraternal binary; acquisition/verification/supervisor/UI glue has an unknown nonzero potential delta and must be measured per slice for x86_64/aarch64. Never claim whole feature binary delta is zero. No dormant child, probe timer or background download; idle infraternal-agent RSS stays under D41's 15 MB budget measured separately. Keep UI lazy-loaded with no idle polling.

Provisional selected-Traefik regression triggers: idle RSS ≤110 MiB per active node; 1,000-connection RSS ≤180 MiB; HTTP peak ≤320 MiB under the matched J20 workload; no idle CPU materially above its 0.00% sampled baseline without investigation. These are proposed, not owner-approved caps. At 50 active nodes J20 idle is 4,665 MiB (~4.56 GiB) across the fleet; 1,000 held connections each is 7,513.5 MiB (~7.34 GiB). Fifty terminals on one node share one proxy. Zero active exposures means zero engine RSS. Retaining two measured Traefik executable versions costs 364,241,216 bytes, excluding archives/state. Obtain Caddy figures for the actual provider module variant before setting its thresholds.

Reload gate: zero dropped/failed requests in three repeated hot-update trials with h1 keep-alive, h2 streams, in-flight responses and WebSockets; record switch latency, p50/p99, CPU and RSS, no universal zero-downtime claim from one benchmark. Proposed control-only route-application p99 ≤3 s under the matched workload (excludes certificate issuance); compare throughput/p99 within 5% of selected-engine baseline or investigate with paired reruns. Engine restart/binary upgrade is separately measured and labeled, never folded into a hot-reload claim. No public ACME time bound is promised.

8. Implementation slices and gates

Execution begins only after the relevant grill answers. Each slice starts with named failing regression tests, runs them to RED, implements the smallest change, reruns to GREEN, then creates an atomic issue-referenced commit. Run scripts/verify iterate during iteration and scripts/verify fast on exact final head on a build VM through build-slot, quoting its output verbatim. Build on compute01/dev-02, benchmark only perf-01 through perf-run; never build/test on the control host. No pushing, merging or deploying is authorized by this planning task.

S0: contracts and selected-engine acceptance

• Attach Caddy's module-enabled matched J20 comparison; record the owner's engine/artifact/provider answers, supported platforms and HTTP scope.
• Create contracts/proxy.md defining ExposureSpec, status/revisions, access and TLS semantics, secure secret references, ownership/adoption and bounded limits. Propose a narrow ProxyAdapter interface as above.
• Add contract fixtures for unknown fields, stale revisions, canonical names, unsupported provider and service A/B access separation. Reuse the existing action/schema conventions.
• Gate: owner answers + independent security contract review. No engine/library choice delegated to an implementer. Per-platform engine availability and module licenses proven.

S1: verified acquisition and offline import

Proposed files: crates/plugins/proxy/{Cargo.toml,src/lib.rs,src/component.rs,src/acquire.rs,tests/acquisition.rs}; shared verification extraction beside existing release verifier only if needed; release component recipe/manifests under release/. Wire workspace/features without linking server-only dependencies into the host.

• RED: correct signed tuple imports; wrong release/platform/module digest/signature/hash, >128 MiB valid component handling, traversal/link/bomb and interrupted/disk-full installs fail safely; active/previous binary cannot be GC'd.
• Implement acquire_component(release_binding, source) -> VerifiedComponent with immutable descriptor/path identity and atomic publication; online/offline use the same verifier.
• GREEN/gates: installer tests on Linux x86_64/aarch64 and macOS arm64, cargo-deny/license/SBOM review, compile-out, static D243 deltas, full fast gate. No network until first enable.

S2: owned lifecycle, status and logs

Proposed files: crates/plugins/proxy/src/{supervisor.rs,logs.rs,health.rs}, tests/lifecycle.rs; integrate via the existing plugin host registration in crates/infraternal-agent/src/ and installed-service ownership paths, rather than treating remote_supervisor.rs (server-link reconnect logic) as a child-process manager.

• RED: zero exposures means no child/timers; one/many exposures mean one child; last disable drains/stops; crash storm is bounded; startup cannot report ready early; canary secrets never enter logs; parent abrupt-stop/restart cannot duplicate or adopt an unrelated process.
• Implement the lifecycle state machine, verified spawn, exclusive lock/control identity, bounded logs/limits and private health observations.
• GREEN/gates: real engine process tests, log-flood/OOM/fd/port-conflict cases, Linux rootless + macOS lifecycle evidence, process/socket/orphan sweep. Measure inactive cost and active health-probe cost. #211 adoption gate required before claiming interruption-free host updates.

S3: exposure state and selected adapter

Proposed files: crates/plugins/proxy/src/{model.rs,descriptor.rs,server.rs,store.rs,reconcile.rs,adapter.rs}, one selected factual integration module traefik.rs or caddy.rs, tests/{model,render,reload}.rs; additive migration under crates/infraternal-server/migrations/ and action registration following crates/plugins/mesh/src/descriptor.rs.

• RED: authorization/audit, concurrent/stale revisions, injection/unknown host/path, complete atomic replacement, failed apply, security revocation refusal and crash-recovery convergence assertions.
• Implement persistent desired/observed state and one-writer reconciliation. Render structured serializers, never raw concatenated expressions supplied by callers. Assert every control/admin surface is disabled or private.
• GREEN/gates: real selected-engine valid/invalid reloads on Linux/macOS, three zero-loss reload trials on perf-01, control switch latency and request p99/throughput/RSS against J20. Test static restart separately. Full fast + parser fuzz smoke.

S4: TLS and sealed credential lifecycle

Proposed files: crates/plugins/proxy/src/{certificates.rs,custody.rs}, tests/certificates.rs; reuse reviewed node-secret facilities once their actual integration is established. Do not create new cryptography in the plugin.

• RED: public .internal requests refused; missing/wrong provider module reported; wrong/expired cert, DNS timeout/CAA/provider failure and token rotation retain safe state; stop/re-enable after expiry, permissions, encrypted backup and no-secret diagnostics asserted.
• Implement DNS-01 resolver/automation configuration, certificate observations, provided-cert import and sealed-reference retrieval on the issuing node only. Staging ACME by default in tests; production issuer is explicit.
• GREEN/gates: local ACME/DNS fixture issuance + renewal + restart/storage recovery, controlled provider test with externally delivered credentials, Linux/macOS custody/permission review, exact selected module artifact receipt. No screenshots/logs containing credentials. Air-gap test with no network and provided cert.

S5: mesh DNS/rules, privilege mapping and release integration

Proposed files: proxy mesh adapter plus narrow extensions in crates/plugins/mesh/src/{configuration.rs,access_updates.rs} and #266's accepted DNS interfaces; integration tests in both plugins. Coordinate with #266/#211 branches before editing their seams.

• RED: direct-IP and stale-DNS bypass denied; service A/B split on one listener, source spoof/address reuse, denied-name privacy, custom suffix rename/removal and live-flow revocation; mapping touches no unrelated host rules; updater rollback cannot resurrect revoked routes.
• Implement revision-bound rules/name projection, ingress source enforcement, owned privileged mapping (only if approved), and #211 child-generation adoption/prestage/rollback adapter.
• GREEN/gates: disposable Linux namespaces/real mesh peers, macOS listener/mapping checks, coordinator outage/recovery, cutoffs under existing mesh budget, host update/rollback while h1/h2/WebSocket traffic runs. If cross-platform identity-safe adoption cannot be proven, do not claim seamless upgrades. Full fast + integration + paired perf receipts.

S6: shared actions/CLI and approved UI

Proposed files: proxy action descriptor plus existing crates/infraternal/src/cli/ adapters/tests; lazy apps/tracer/src/canvas/proxy/ view/transport modules and production browser cases; extend shared packages/ui components only where an existing primitive lacks a needed capability.

• RED: schema permissions and --json envelope/errors/pending revisions agree across CLI/UI/MCP; read-only status never downloads; secure credential entry cannot leak into command history/argv.
• Orchestrator produces mockups for owner approval, then implements the behavior brief with existing shared components. No Codex visual self-approval.
• GREEN/gates: CLI envelope/action parity, production-browser tests at 1440×900, 393×852 and 375×667 in ink/paper including all unhappy states; attach raw screenshots for owner/independent review, real-iPhone focus/keyboard/scroll checks, contrast/touch targets and lazy-bundle/first-paint/idle measurements. Full fast plus browser reproducibility gates per PROCESS.

Planning verification and handoff

Self-review: all eight requested areas map to sections and slices above. Key unresolved dependencies are engine measurements/selection, unattended node sealing contract, private .internal PKI, shared-listener rule enforcement, #266 names, #211 generation adoption and macOS privileged mapping/cleanup behavior. These are owner questions or concrete acceptance gates, not permission to invent a foundation.

This issue is the reviewable deliverable. No code, dependency, release pin or DECISIONS entry has been changed; only kayg may settle the open choices. Gate receipt and exact planning-worktree status follow in a separate comment, with no claim the proposed implementation has passed tests.

# Reverse proxy on demand: engine-neutral implementation plan > For implementers after owner approval: use the approved execution workflow, read the linked design decisions and execute the gated slices below task by task. This document authorizes planning only. **Status:** P5 proposal for kayg to critique before code. The engine choice is OPEN following the owner's Caddy question. #261 comment 72438 selected Traefik earlier; this plan does not silently extend that selection to the reopened comparison. Keep the requested issue title for continuity. **Goal:** expose an authorized service through a managed HTTPS name, downloading and running a proxy only on nodes with enabled exposures. **Architecture:** the infraternal server owns desired state, authorization and audit; the local infraternal agent acquires and supervises one external proxy process per serving node. A narrow engine adapter renders our exposure model, applies revisions and reports observations. Only the selected adapter ships initially; this is a boundary for choice and future migration, not a commitment to maintain two engines. **Stack:** existing Rust action/plugin framework, mesh coordinator and update trust; selected external Go executable; existing TypeScript product shell and shared UI. No new foundational library is selected here. All new packages/crates remain AGPL-3.0-only; review the engine and module licenses separately. **Inputs:** [D316 and repository decisions](https://git.kayg.org/kayg/infraternal/src/branch/dev/DECISIONS.md), [R2 design](https://git.kayg.org/kayg/infraternal/issues/261#issuecomment-72433), [earlier owner selection](https://git.kayg.org/kayg/infraternal/issues/261#issuecomment-72438), [J20 measurements](https://git.kayg.org/kayg/infraternal/issues/261#issuecomment-71101), [edge updates #211](https://git.kayg.org/kayg/infraternal/issues/211), [mesh DNS #266](https://git.kayg.org/kayg/infraternal/issues/266). Snapshot: `origin/dev` at `f4b61cd6c3f52078cad7573c5367d7a6e75f84f0`, branch `research/p5-traefik-plan`. No implementation changes. ## Owner grill, recommendation first 1. **Engine: recommend waiting for the Caddy extension of J20, then selecting one engine and one provider-capable artifact set.** Traefik has measured zero-loss reloads and stock DNS integrations; Caddy offers a transactional local configuration API but DNS-01 adds a custom-build supply chain. Neither advantage settles speed or memory. Caddy has not been measured in the supplied J20 results. Does kayg prefer Traefik or Caddy after reviewing matching workload, module-enabled binary size/RSS, reload and certificate results? Do not start engine-specific implementation before this answer. 2. **Trust: recommend an infraternal-signed component manifest and immutable release artifact mirror for either engine.** Verify available upstream signatures in release CI; never represent our signature as an upstream signature. Traefik's inspected release lacks detached asset signatures. If Caddy wins, recommend CI-owned, pinned `xcaddy` builds with an approved DNS module set rather than dynamically building on hosts. Is this acquisition trust and build responsibility acceptable? 3. **Scope/access: recommend mesh-only HTTP/HTTPS exposures first, bound to the mesh address, with exact host matching and existing mesh rules enforced per original service.** A DNS answer alone cannot authorize traffic. Public/LAN listeners need an explicit audience and authentication decision. Should those ship in the first release or follow separately? 4. **Names/TLS: recommend `.internal` remain the default mesh suffix; publicly trusted TLS requires an optional owned domain and DNS-01.** For `.internal`, offer explicit private-CA TLS only after its issuance/trust-distribution contract is approved, or explicitly chosen HTTP. Never imply a public CA can issue `.internal`. Should private-CA TLS ship in slice 1, and which existing PKI authority is permitted to issue it? 5. **Ports: recommend rootless high ports (8080/8443 proposed), with an explicit privileged mapping option for conventional URLs.** High ports appear in the URL. Prefer narrow host-managed port mapping to granting the proxy process capabilities; Linux capability support is an opt-in alternative. Approve this default and the privileged mapping UX for Linux/macOS? 6. **Custody: recommend node-sealed DNS credentials, only on the issuing node, with least-privilege challenge-zone tokens; later an OpenBao-backed reference uses the same custody interface.** This follows the owner's agreement in principle, not a claim an unattended sealed store is implemented. Approve using the platform/node sealing contract for unattended renewal, and requiring explicit re-enrollment after node-key loss? 7. **Lifecycle: recommend stopping the process when enabled exposure count reaches zero, retaining its private certificate state, and checking renewal immediately when re-enabled.** This saves idle memory but cannot renew while stopped. Re-enabling after a long stop may wait for issuance; never fall back silently to HTTP. Accept that behavior, a proposed 10-second drain deadline, and the #211 adoption dependency for seamless infraternal updates? 8. **CLI: recommend `expose <machine> <service> --name <dns-name>`, plus `expose list`, `expose status <id>`, `expose remove <id>`, and `--json` everywhere via D16/D314.** Is `expose` the right noun/verb family? Details remain an action-schema grill, not a second command implementation. 9. **Budgets: recommend the provisional thresholds below as regression review triggers, then ratchet actual measured selected-engine results.** Approve them after Caddy measurements rather than inheriting an unmeasured cap or loosening D41's infraternal-agent RSS budget. ## Evidence and engine-specific differences J20 tested Traefik **3.7.17**, executable **182,120,608 bytes**, idle RSS **93.30 MiB**, warm **95.77 MiB**, 1,000 held connections **150.27 MiB**, HTTP high-water **265.84 MiB**. Idle CPU sampled **0.00%**. Three reload trials lost **0 / 0 / 0** requests; route switch **27 / 561 / 1041 ms**. These are standalone Linux x86 results, not macOS numbers or a universal zero-loss guarantee. HTTP/1 c64 was 32,956 req/s, p99 7.296 ms under J20's allocation and generator ceiling. Caddy cells stay **unmeasured**, including provider-enabled builds. | Concern | Traefik adapter | Caddy adapter | |---|---|---| | DNS-01 | Supported provider integrations compiled into stock binary; configure named ACME resolvers. Pin and test our supported provider subset. | Stock release does not include the needed provider modules. Pin core + each DNS module commit/version + dependency closure + Go/xcaddy toolchain; build approved variants or one bounded module bundle in release CI. | | Acquisition | Official versioned archive from `github.com/traefik/traefik/releases`, reviewed hashes, then our signed manifest/mirror. R2 inspected 3.7.17 without detached checksum signatures. Recheck the exact chosen release assets. | Official stock releases have documented checksum-signature verification. A custom download/build-service artifact is a different artifact and does not inherit the stock signature. Prefer our reproducible CI build, SBOM, provenance and release signature; build-service use needs independently pinned resulting bytes and verified provenance. No live module selection/download service at install time. | | Configuration/reload | Static startup config for listeners/providers/resolvers; full dynamic snapshot in a watched directory. File changes are asynchronous and provider events throttled (documented default 2 s); no successful write acknowledgment proves application. Static changes require controlled restart. | Native JSON; whole config `POST /load` through a private permissioned Unix socket. API blocks until success/failure and documents rollback on failure without downtime. A JSON file is a durable source, not an automatic file watcher; `caddy reload` also uses the admin API. With admin disabled, use a controlled restart and report that interruption. | | Admin surface | Disable API/dashboard/insecure mode, unused discovery providers, debug template dumping, remote plugin loading and telemetry/version checks. No `api@internal` route. Private ping only, separate from data listeners. | Replace default `localhost:2019` with a permissioned private Unix socket, or disable admin if accepting restart-only changes. No public or mesh binding, remote admin, arbitrary module/config API or proxying the socket. Disable unintended automatic port-80 listeners and unapproved on-demand TLS. | | TLS storage | Dedicated `acme.json` with mode 0600 and exclusive writer. Never share it among nodes. | Dedicated private data/storage directory holding accounts and certificate keys, not `acme.json`; treat all contents as secret and prohibit arbitrary storage modules. | | Memory | Above J20 measurements; process only while active. | Pending parallel matched measurement with the exact approved DNS modules. Do not extrapolate stock-binary memory. | Sources: [Traefik file provider](https://doc.traefik.io/traefik/reference/install-configuration/providers/others/file/), [Traefik ACME](https://doc.traefik.io/traefik/reference/install-configuration/tls/certificate-resolvers/acme/), [Traefik API/dashboard](https://doc.traefik.io/traefik/reference/install-configuration/api-dashboard/); [Caddy JSON API](https://caddyserver.com/docs/api), [Caddy DNS challenge](https://caddyserver.com/docs/automatic-https#dns-challenge), [Caddy build instructions](https://caddyserver.com/docs/build), [Caddy provider modules](https://caddyserver.com/docs/modules/), [Caddy asset signatures](https://caddyserver.com/docs/signature-verification). Documentation describes capabilities; J20 supplies our measurements. Freeze versioned docs/source alongside the selected release recipe. ## 1. Acquisition, installation and release coupling One signed component manifest per infraternal release binds release identity/channel, component ID, engine version, OS/architecture, module-set digest, source URL, archive/executable SHA-256, exact sizes, configuration/storage compatibility versions and license/provenance receipts. Release CI verifies upstream evidence where present, scans advisories/dependencies/licenses and signs the approved resulting bytes using the existing release trust policy. Keep identity-bearing manifest verification separate from a bare checksum list. No `latest`, host-side `xcaddy`, automatic engine self-update or arbitrary URL input. The existing `release_verify.rs::verify_release_file` and `release_manifest.rs::verify_release_manifest` are useful trust precedents, not drop-in component installers: the former hard-caps payloads at **128 MiB**, below measured Traefik size, and the latter authenticates infraternal release identity. Propose a separate bounded component payload policy, initially maximum **256 MiB archive / 512 MiB extracted executable**, exact signed lengths stricter than those caps. Do not increase the existing application-update limit globally. Extract verifier primitives into a shared small module only where both server and infraternal agent genuinely need them; add no server/SQLx dependency to host-only builds. On first enabled exposure: acquire an installer lock, stream to a private staging file, enforce time/byte/disk bounds, verify signed manifest and archive digest before extraction, extract exactly the allowlisted executable, verify its digest/target, then fsync and rename within the destination filesystem. Reject absolute/parent paths, duplicate entries, symlinks/hardlinks, special files, bombs and wrong-architecture artifacts. Verify the same immutable file ultimately executed; the install parent must not be writable by other users. No shell expansion or untrusted executable search path. Use the existing resolved infraternal data root: rootful `/var/lib/infraternal`, rootless `~/.local/share/infraternal` where applicable, and the established macOS resolver rather than assuming Linux paths. Proposed layout: `components/proxy/<artifact-digest>/bin`, `proxy/config/`, `proxy/sealed/`, `proxy/tls/`, and private runtime socket directory. Directories 0700; immutable executable 0500; manifests/config/certificates 0600. Keep secrets separate from engine configuration history. D251/#211: stage the component selected by the candidate infraternal release before activating it on an already exposing node; keep the current working process/artifact until candidate readiness. Inactive nodes retain the manifest and need no download. Rollback selects the previous release's pinned artifact and compatible rendered config, never an arbitrary old engine. Signed downgrade authorization follows existing release rules. Retain previous artifact and metadata until rollback window ends; never garbage-collect the active binary. Cert storage compatibility is a gate: use a quiesced protected snapshot or approved forward/backward-compatible migration; never overwrite newer ACME account/cert state with a stale backup blindly. Air-gapped install imports a release-signed component bundle from a file through exactly the same checks, no bypass flag. A verified mirror/cache is optional and content-addressed. Full isolation cannot contact public ACME or provider APIs: support explicit pre-provisioned certificates or approved private PKI, show renewal responsibility and expiry, and never promise public DNS-01 will work offline. Packaging must include engine/module license notices and verification evidence. ## 2. Supervision, limits and health Desired enabled exposure count, not browser attachment or traffic, determines process lifetime. State machine: `inactive → acquiring → verifying → starting → ready`, with `applying`, `degraded`, `retrying`, `draining`, and typed `failed` observations. Download failure leaves desired revision pending, with a retry action. Disabled rows and cached artifacts alone do not start a process. One owned process per node, not per service/session. Start with an absolute verified path, allowlisted environment, fixed working directory and restrictive umask. Acquire a node-scoped ownership lock before bind. Readiness requires process survival, private control/ping readiness and a synthetic expected-route/certificate probe; distinguish engine-ready, upstream-unhealthy and certificate-pending. Stop: retire routes/deny new traffic, signal graceful drain, wait the proposed 10 s, terminate the owned process group if needed, reap and remove private runtime material. When re-enabled, inspect retained certificates immediately before reporting ready. Proposed crash backoff: 200 ms doubling to 30 s with jitter, reset after 60 s healthy; five failures in 60 s enter a cooled-down failure state with bounded automatic retries and explicit Retry. Retry timers exist only while enabled exposures require the engine. Never download again on every crash. Failed reload keeps the last permitted working revision; a security revocation is different and must fail closed even if that requires withdrawing a listener. Linux: use existing process/descriptor ownership primitives, child rlimits and, when delegated, a cgroup for hard memory/task bounds. Proposed active-process `MemoryMax` review budget 512 MiB, file descriptor cap 8192 and capped connection/inflight settings; test overload behavior before approval. macOS: use supported rlimits and process-group/launchd lifecycle integration; do not claim Linux-style hard resident-memory enforcement exists there. A Go memory target is advisory. Observe RSS and pressure; report limit support truthfully. Do not impose process-wide rlimits on the infraternal agent merely to constrain its child. Abrupt infraternal-agent exit and update ownership need real tests on both platforms. Linux parent-death/cgroup cleanup and macOS service-manager behavior differ. Track process identity and owned control descriptors; never kill/re-adopt a numeric PID alone. #211 generation handoff must transfer exclusive child ownership, log readers and the immutable component/config revision, or explicitly drain/restart. No candidate callbacks may alter exposure state before adoption commits. Seamless proxy survival across host updates is a gate dependency, not implemented in current dev. If unmanaged macOS SIGKILL leaves an orphan, require a scoped cleanup owner or surface that limitation before shipping; do not hand-wave portability. Rootless listeners bind above 1024. DNS-01 requires no inbound port 80. For conventional 80/443, propose an explicit narrowly scoped OS port mapping on Linux/macOS, audited install/removal without touching unrelated rules. Linux `CAP_NET_BIND_SERVICE` can be an owner-approved alternative on the verified executable or constrained launcher; it grants low-port bind broadly, needs reapplication after replacement, and must not spread to the infraternal binary. macOS has no Linux file capabilities; use an approved privileged mapping/helper or high ports. Router PCP/NAT-PMP/UPnP mapping is a separate exposure consent, not a substitute for local privileged bind. No silent root promotion or global unprivileged-port sysctl change. Engine stdout/stderr feed the existing infraternal log path with node/exposure/revision IDs, bounded line size/queue/rate and dropped-count summaries. Never forward arbitrary raw errors/configs to the browser. Keep only safe structured fields; redact Authorization/Cookie/query credentials, never log provider tokens/private keys. Access logs opt-in/minimized; health and certificate observations are pushed on change, not UI polling at idle. A proposed 30 s private active-engine health probe is measured separately from zero inactive wakeups. ## 3. Product model and adapter contract Proposed `ExposureSpec`: stable exposure ID, owning node, stable service target (protocol/address/port), normalized exact DNS name, optional literal path prefix, listener audience/port, TLS mode (`disabled`, `managed_dns`, `provided`, future approved private PKI), certificate/provider references, access-rule reference, enabled flag and monotonically increasing desired revision. Status separately records applied revision, engine artifact ID, process/upstream/certificate states, expiry, safe error code and next action. No credentials in either public model. Product copy uses **rules**, following D252; the requested “policy” concept maps to those existing rules. Server authorizes the actor, node, service and audience; normalizes/validates the whole transaction, stores it and audits intent. Node validates the authenticated revision and target bindings again. No raw engine JSON/YAML, arbitrary expressions, templates, file paths, exec hooks, middleware snippets or module names in the public API. Exact names are canonicalized using the existing mesh name rules; initially reject wildcards, overlapping host/path claims and credentials embedded in URLs. TCP/UDP and arbitrary middleware are separate future scope. Proposed adapter boundary in the proxy plugin: `render(snapshot, capabilities) -> RenderedRevision`; `apply(rendered) -> Applied | Pending | Rejected`; `observe() -> ProxyObservation`; plus lifecycle startup requirements. `RenderedRevision` identifies nonsecret config bytes/hash, secret references and whether restart is required. Capabilities explicitly report compiled DNS providers and supported limits. Only one writer applies snapshots serially; stale generations cannot supersede a newer one. Traefik: serialize a complete dynamic snapshot to a temporary sibling **outside the watched directory**, fsync, rename to the one canonical watched file and fsync the directory. This avoids mixed multi-file revisions. Watch the parent directory rather than a replaced-file mount. Keep a durable desired revision and last-known-good rendered snapshot. Because watch events are asynchronous/coalesced, observe expected route/cert behavior before acknowledging applied; process health alone is insufficient. API/dashboard stay disabled. Test a private synthetic revision route restricted to local probing if needed; do not expose a management route to clients. Startup-only resolver/listener changes are marked restart-required in status. Caddy: serialize native JSON atomically for durable recovery, validate with the exact installed build, then submit the whole snapshot via private Unix-socket `/load`. One successful response is the engine acknowledgment, followed by route/TLS probes. Use our durable committed desired/applied revisions as authority; either disable autosave or protect and reconcile it, never blindly `--resume` a stale config after revocation. Disable the admin endpoint only if the owner accepts restart-only application. File-only mode is not claimed to hot-reload automatically. Long-lived WebSocket and HTTP/2 behavior must be measured for the chosen stream-drain settings. For either engine, invalid nonsecurity edits retain the last working authorized state and report desired/applied mismatch. Revocation first removes authorization at the dataplane enforcement boundary and closes forbidden live flows; restoration of a last-known-good config must never restore revoked access. Partial writes, crash between persistence/apply and rapid rename/edit/remove races must converge deterministically after restart. ## 4. Certificates and custody Managed custom-domain TLS uses engine DNS-01, explicitly disabling unwanted HTTP-01/TLS-ALPN fallback and implicit public listeners. The optional owned suffix is editable, while `.internal` can remain available as the default alias per D316. DNS-01 needs public authoritative TXT control even if A/AAAA records and services exist only on mesh. Normalize suffix boundaries, authorize domain use, and explain propagation/CAA/rate-limit/account/provider failures with a safe retry action. Stage new names/certificates before removing an old working custom-domain route where rules permit; suffix changes must also retire old DNS and access state coherently. Provider tokens are sealed at rest and delivered only to the designated issuing node over the existing authenticated channel. Store only opaque references/status centrally. Scope tokens to the challenge zone; offer delegated `_acme-challenge` zones where supported. Unseal just for the owned process, via provider-supported credential files or private config; environment-only providers require an explicitly documented same-user/root inspection risk and scrubbing, never argv. Use file-backed provider options where supported, not a claim every provider supports them. Rotation writes new private material, validates issuance/renewal, switches, then retires old material. Node migration requires explicit custody reassignment, no mesh-wide replication of tokens. Unattended renewal cannot depend on the user's passkey being present. Node sealing requires a reviewed persistent machine key/platform protection and recovery semantics; do not repurpose the passkey-wrapped SSH vault as if it were already suitable. OpenBao later implements the same `credential reference → node-local material` boundary; no new secret engine is selected by this plan. Traefik owns mode-0600 `acme.json` in a 0700 directory, one writer, protected backups; Caddy owns an equivalently private storage tree. Both contain private keys and ACME accounts. Protect path ownership, no symlinks, no export in diagnostics, no unrestricted backup attachments. Backups are encrypted and explicitly included in disaster recovery scope. Certificate metadata extraction must not return key bytes. Never share a mutable engine certificate store across nodes. Stopping the last exposure stops renewal; retained certificates are checked on next activation, with safe issuance before ready and visible expiry failures. `.internal` is not publicly registrable. Do not make public ACME attempts for it and do not silently install an engine-generated root CA into OS/browser trust. Private certificate issuance, separate server-auth EKUs and trust distribution require owner-approved integration with existing PKI; mesh/device identity keys must not be reused as website keys. Provided certificates are an explicit alternative with expiry/status tracking. ## 5. Mesh names and enforcement Reuse `crates/infraternal-mesh/src/names.rs::NameTable`, coordinator canonical claims and #266's negotiated, revisioned name snapshots. #266 is active work, not a guarantee those listeners/suffix APIs are in this base. D316 requires `<hostname>.internal` and an optional custom suffix; a service also needs a unique routing key. Propose one root service per host name initially, explicit path prefix or explicit service alias for extras, with owner confirmation in the contract slice. DNS returns the actual serving node's mesh address, not an unreachable upstream or invented global gateway. Use the same accepted mesh rule revision for name visibility, exposure authorization and dataplane admission. Denied peers must learn no protected aliases from a proxy status response. DNS denial is insufficient: direct-IP Host/SNI requests, stale DNS caches and existing connections must still be denied. Shared HTTPS 443/8443 also cannot collapse distinct service-port rules into “can reach the proxy”. Render per-route permitted peer sources from the original service rule, enforce mesh identity at ingress, and retain backend egress restrictions. Mesh source addresses are trusted only on the authenticated mesh interface. Never trust client-supplied `X-Forwarded-For`/Forwarded headers as identity; configure engine source matching accordingly. Tests must prove a peer permitted for service A cannot reach B on the same listener, even with forged Host/SNI/forwarded headers or via an allowed backend. Address reassignment must not inherit an old peer's privileges. For requests requiring person-level authentication or public access, mesh IP is not enough: integrate an approved identity flow separately or deny. Close existing forbidden flows on revocation under the mesh cutoff contract, and fail closed when a newly accepted rule revision cannot be installed. Preserve last accepted rules during coordinator outage only as already allowed by the mesh contract; do not invent expiry semantics here. ## 6. UI and CLI proposal Only a behavior brief here; D171 leaves design/mockups to the orchestrator. Read `TASTE.md`; reuse `packages/ui` components (`entity-chip`, `pane-stepper`, `status-line`, `spinner`, `toast`, `chip-input`, focus/keyboard helpers), not new primitives. Owner-approved mockups precede UI implementation. Entry point beside a service in the host's existing view: **Expose service**. Show the exact service target, choose name/suffix, TLS and who can reach it, then preview the full URL including nonstandard port and audience before enabling. Domain/provider setup appears only when required. First activation shows downloading → verifying → starting → getting certificate → ready in the service's own surface. A pending certificate must not appear ready merely because the child is alive. Exposure details show copy/open URL, desired/applied state, upstream health, certificate issuer/expiry/renewal state and a plain-language next action for errors. Disable/remove and domain/access edits stay beside the exposure. Provider secret controls show configured/rotate/remove, never reveal. Errors use appropriate icons, retry near the failed step, pending approvals in the existing toast/Inbox, and technical paths/log details behind Details. No engine branding/schema/admin dashboard in the normal flow. On mobile use one column, 44 pt targets, no hover-only controls, no colliding labels or long URL overflow; ink/paper and empty/loading/offline/retry/expired/revoked states get equal care. CLI is the same registered action schema as UI/API/MCP, not a parallel parser or management channel. Proposed commands are in grill Q8. `--json` uses the existing envelope, stable typed errors and desired/applied revisions; an accepted mutation returns pending status rather than waiting indefinitely for DNS. Credentials use secure input/file-reference mechanisms, never command arguments. A read-only list/status command must not trigger acquisition. Add usage/examples and action availability/permissions through `ActionDefinition`/`ActionRegistry` and existing CLI output helpers. ## 7. Security review and budgets Independent review focus: supply-chain authenticity vs hashes; malicious archives and replacement after verification; host/path/config injection; SSRF/unauthorized upstreams; shared-listener authorization bypass; revocation on live WebSockets/h2; secret custody and raw error leakage; ACME abuse; child/adoption ownership and privileged mapping; storage rollback/key loss. No arbitrary dynamic engine modules, discovery socket mounts, public admin endpoints or unbounded API/config bodies. Required malicious cases: backticks/quotes/newlines in DNS/path/header fields, escaped separators/IDNA and suffix confusion, duplicate names, `..`/absolute/symlink archives, oversized compressed/uncompressed inputs, wrong target/digest/key/replayed release, writable-parent executable swap, Unix-socket ownership attacks, forged proxy headers, unknown SNI/default virtual host, loopback/admin/metadata upstream SSRF, malicious provider error containing a canary secret, cancellation after commit, concurrent enable/remove/update, resource exhaustion and disk-full/fsync failure. Parser/config limits are fuzzed; no invalid input creates a partially authorized route. Budget distinction: external engine adds **+0 embedded engine bytes** to the infraternal binary; acquisition/verification/supervisor/UI glue has an **unknown nonzero potential delta** and must be measured per slice for x86_64/aarch64. Never claim whole feature binary delta is zero. No dormant child, probe timer or background download; idle infraternal-agent RSS stays under D41's 15 MB budget measured separately. Keep UI lazy-loaded with no idle polling. Provisional selected-Traefik regression triggers: idle RSS ≤110 MiB per active node; 1,000-connection RSS ≤180 MiB; HTTP peak ≤320 MiB under the matched J20 workload; no idle CPU materially above its 0.00% sampled baseline without investigation. These are proposed, not owner-approved caps. At 50 active nodes J20 idle is **4,665 MiB (~4.56 GiB)** across the fleet; 1,000 held connections each is **7,513.5 MiB (~7.34 GiB)**. Fifty terminals on one node share one proxy. Zero active exposures means zero engine RSS. Retaining two measured Traefik executable versions costs **364,241,216 bytes**, excluding archives/state. Obtain Caddy figures for the actual provider module variant before setting its thresholds. Reload gate: **zero dropped/failed requests in three repeated hot-update trials** with h1 keep-alive, h2 streams, in-flight responses and WebSockets; record switch latency, p50/p99, CPU and RSS, no universal zero-downtime claim from one benchmark. Proposed control-only route-application p99 ≤3 s under the matched workload (excludes certificate issuance); compare throughput/p99 within 5% of selected-engine baseline or investigate with paired reruns. Engine restart/binary upgrade is separately measured and labeled, never folded into a hot-reload claim. No public ACME time bound is promised. ## 8. Implementation slices and gates Execution begins only after the relevant grill answers. Each slice starts with named failing regression tests, runs them to RED, implements the smallest change, reruns to GREEN, then creates an atomic issue-referenced commit. Run `scripts/verify iterate` during iteration and **`scripts/verify fast` on exact final head** on a build VM through `build-slot`, quoting its output verbatim. Build on compute01/dev-02, benchmark only perf-01 through `perf-run`; never build/test on the control host. No pushing, merging or deploying is authorized by this planning task. ### S0: contracts and selected-engine acceptance - [ ] Attach Caddy's module-enabled matched J20 comparison; record the owner's engine/artifact/provider answers, supported platforms and HTTP scope. - [ ] Create `contracts/proxy.md` defining ExposureSpec, status/revisions, access and TLS semantics, secure secret references, ownership/adoption and bounded limits. Propose a narrow `ProxyAdapter` interface as above. - [ ] Add contract fixtures for unknown fields, stale revisions, canonical names, unsupported provider and service A/B access separation. Reuse the existing action/schema conventions. - [ ] Gate: owner answers + independent security contract review. No engine/library choice delegated to an implementer. Per-platform engine availability and module licenses proven. ### S1: verified acquisition and offline import Proposed files: `crates/plugins/proxy/{Cargo.toml,src/lib.rs,src/component.rs,src/acquire.rs,tests/acquisition.rs}`; shared verification extraction beside existing release verifier only if needed; release component recipe/manifests under `release/`. Wire workspace/features without linking server-only dependencies into the host. - [ ] RED: correct signed tuple imports; wrong release/platform/module digest/signature/hash, >128 MiB valid component handling, traversal/link/bomb and interrupted/disk-full installs fail safely; active/previous binary cannot be GC'd. - [ ] Implement `acquire_component(release_binding, source) -> VerifiedComponent` with immutable descriptor/path identity and atomic publication; online/offline use the same verifier. - [ ] GREEN/gates: installer tests on Linux x86_64/aarch64 and macOS arm64, cargo-deny/license/SBOM review, compile-out, static D243 deltas, full fast gate. No network until first enable. ### S2: owned lifecycle, status and logs Proposed files: `crates/plugins/proxy/src/{supervisor.rs,logs.rs,health.rs}`, `tests/lifecycle.rs`; integrate via the existing plugin host registration in `crates/infraternal-agent/src/` and installed-service ownership paths, rather than treating `remote_supervisor.rs` (server-link reconnect logic) as a child-process manager. - [ ] RED: zero exposures means no child/timers; one/many exposures mean one child; last disable drains/stops; crash storm is bounded; startup cannot report ready early; canary secrets never enter logs; parent abrupt-stop/restart cannot duplicate or adopt an unrelated process. - [ ] Implement the lifecycle state machine, verified spawn, exclusive lock/control identity, bounded logs/limits and private health observations. - [ ] GREEN/gates: real engine process tests, log-flood/OOM/fd/port-conflict cases, Linux rootless + macOS lifecycle evidence, process/socket/orphan sweep. Measure inactive cost and active health-probe cost. #211 adoption gate required before claiming interruption-free host updates. ### S3: exposure state and selected adapter Proposed files: `crates/plugins/proxy/src/{model.rs,descriptor.rs,server.rs,store.rs,reconcile.rs,adapter.rs}`, one selected factual integration module `traefik.rs` or `caddy.rs`, `tests/{model,render,reload}.rs`; additive migration under `crates/infraternal-server/migrations/` and action registration following `crates/plugins/mesh/src/descriptor.rs`. - [ ] RED: authorization/audit, concurrent/stale revisions, injection/unknown host/path, complete atomic replacement, failed apply, security revocation refusal and crash-recovery convergence assertions. - [ ] Implement persistent desired/observed state and one-writer reconciliation. Render structured serializers, never raw concatenated expressions supplied by callers. Assert every control/admin surface is disabled or private. - [ ] GREEN/gates: real selected-engine valid/invalid reloads on Linux/macOS, three zero-loss reload trials on perf-01, control switch latency and request p99/throughput/RSS against J20. Test static restart separately. Full fast + parser fuzz smoke. ### S4: TLS and sealed credential lifecycle Proposed files: `crates/plugins/proxy/src/{certificates.rs,custody.rs}`, `tests/certificates.rs`; reuse reviewed node-secret facilities once their actual integration is established. Do not create new cryptography in the plugin. - [ ] RED: public `.internal` requests refused; missing/wrong provider module reported; wrong/expired cert, DNS timeout/CAA/provider failure and token rotation retain safe state; stop/re-enable after expiry, permissions, encrypted backup and no-secret diagnostics asserted. - [ ] Implement DNS-01 resolver/automation configuration, certificate observations, provided-cert import and sealed-reference retrieval on the issuing node only. Staging ACME by default in tests; production issuer is explicit. - [ ] GREEN/gates: local ACME/DNS fixture issuance + renewal + restart/storage recovery, controlled provider test with externally delivered credentials, Linux/macOS custody/permission review, exact selected module artifact receipt. No screenshots/logs containing credentials. Air-gap test with no network and provided cert. ### S5: mesh DNS/rules, privilege mapping and release integration Proposed files: proxy mesh adapter plus narrow extensions in `crates/plugins/mesh/src/{configuration.rs,access_updates.rs}` and #266's accepted DNS interfaces; integration tests in both plugins. Coordinate with #266/#211 branches before editing their seams. - [ ] RED: direct-IP and stale-DNS bypass denied; service A/B split on one listener, source spoof/address reuse, denied-name privacy, custom suffix rename/removal and live-flow revocation; mapping touches no unrelated host rules; updater rollback cannot resurrect revoked routes. - [ ] Implement revision-bound rules/name projection, ingress source enforcement, owned privileged mapping (only if approved), and #211 child-generation adoption/prestage/rollback adapter. - [ ] GREEN/gates: disposable Linux namespaces/real mesh peers, macOS listener/mapping checks, coordinator outage/recovery, cutoffs under existing mesh budget, host update/rollback while h1/h2/WebSocket traffic runs. If cross-platform identity-safe adoption cannot be proven, do not claim seamless upgrades. Full fast + integration + paired perf receipts. ### S6: shared actions/CLI and approved UI Proposed files: proxy action descriptor plus existing `crates/infraternal/src/cli/` adapters/tests; lazy `apps/tracer/src/canvas/proxy/` view/transport modules and production browser cases; extend shared `packages/ui` components only where an existing primitive lacks a needed capability. - [ ] RED: schema permissions and `--json` envelope/errors/pending revisions agree across CLI/UI/MCP; read-only status never downloads; secure credential entry cannot leak into command history/argv. - [ ] Orchestrator produces mockups for owner approval, then implements the behavior brief with existing shared components. No Codex visual self-approval. - [ ] GREEN/gates: CLI envelope/action parity, production-browser tests at 1440×900, 393×852 and 375×667 in ink/paper including all unhappy states; attach raw screenshots for owner/independent review, real-iPhone focus/keyboard/scroll checks, contrast/touch targets and lazy-bundle/first-paint/idle measurements. Full fast plus browser reproducibility gates per PROCESS. ## Planning verification and handoff Self-review: all eight requested areas map to sections and slices above. Key unresolved dependencies are engine measurements/selection, unattended node sealing contract, private `.internal` PKI, shared-listener rule enforcement, #266 names, #211 generation adoption and macOS privileged mapping/cleanup behavior. These are owner questions or concrete acceptance gates, not permission to invent a foundation. This issue is the reviewable deliverable. No code, dependency, release pin or DECISIONS entry has been changed; only kayg may settle the open choices. Gate receipt and exact planning-worktree status follow in a separate comment, with no claim the proposed implementation has passed tests.

kayg commented 2026-11-01 08:05:03 +00:00

Author

Owner

Copy link

P5 started: branch research/p5-traefik-plan, base/head f4b61cd6c from origin/dev. Planning only in isolated worktree. Owner reopened the engine question; acquisition, supervision, exposure model, custody and UI will support either engine. No Caddy measurements assumed; parallel J20 extension is a dependency. No code changes or builds started.

P5 started: branch `research/p5-traefik-plan`, base/head `f4b61cd6c` from `origin/dev`. Planning only in isolated worktree. Owner reopened the engine question; acquisition, supervision, exposure model, custody and UI will support either engine. No Caddy measurements assumed; parallel J20 extension is a dependency. No code changes or builds started.

kayg commented 2026-11-01 08:11:39 +00:00

Author

Owner

Copy link

P5 findings before plan publication:

• Source at f4b61cd6c: crates/infraternal-server/src/release_verify.rs::verify_digest_expected refuses files above 128 MiB. J20 measured a 182,120,608-byte Traefik executable. Component payload limits must be separate; do not globally weaken app-update bounds.
• R2 #261 comment 72433 inspected Traefik 3.7.17 release assets without detached checksum signatures. Our signed component manifest must distinguish release authentication from upstream asset authentication. Caddy documents stock release signatures, but its DNS-module custom builds require their own pinned build/verification pipeline.
• Caddy /load documents synchronous success/failure and rollback; Traefik file-provider reload is asynchronous and throttled. Desired/applied state needs an engine-specific acknowledgment strategy. Sources: https://caddyserver.com/docs/api and https://doc.traefik.io/traefik/reference/install-configuration/providers/others/file/
• Shared proxy listeners must preserve each original service rule. DNS visibility and permission to reach port 8443 cannot authorize every route. #266 and #211 are active integration dependencies, not implemented seams to assume.
• Owner reopened engine choice. Caddy is unmeasured in the supplied J20 results; no ranking or final engine pin is asserted.

P5 findings before plan publication: - Source at `f4b61cd6c`: `crates/infraternal-server/src/release_verify.rs::verify_digest_expected` refuses files above 128 MiB. J20 measured a 182,120,608-byte Traefik executable. Component payload limits must be separate; do not globally weaken app-update bounds. - R2 #261 comment 72433 inspected Traefik 3.7.17 release assets without detached checksum signatures. Our signed component manifest must distinguish release authentication from upstream asset authentication. Caddy documents stock release signatures, but its DNS-module custom builds require their own pinned build/verification pipeline. - Caddy `/load` documents synchronous success/failure and rollback; Traefik file-provider reload is asynchronous and throttled. Desired/applied state needs an engine-specific acknowledgment strategy. Sources: https://caddyserver.com/docs/api and https://doc.traefik.io/traefik/reference/install-configuration/providers/others/file/ - Shared proxy listeners must preserve each original service rule. DNS visibility and permission to reach port 8443 cannot authorize every route. #266 and #211 are active integration dependencies, not implemented seams to assume. - Owner reopened engine choice. Caddy is unmeasured in the supplied J20 results; no ranking or final engine pin is asserted.

kayg commented 2026-11-01 08:12:09 +00:00

Author

Owner

Copy link

P5 planning deliverable posted and read back byte-for-byte. Engine choice is the first owner question; Caddy remains unmeasured pending the parallel J20 extension. Acquisition/custody/supervision/model/UI remain engine-neutral; only the selected adapter is proposed for initial implementation. No code, dependency, decision or release pin changed. No push/deploy/commit performed.

Branch: research/p5-traefik-plan. Base/head: f4b61cd6c3f52078cad7573c5367d7a6e75f84f0. Planning worktree is clean. Plan/evidence copy: /home/kayg/.cache/codex-scratch/p5/evidence/implementation-plan.md.

Gate attempt on compute01, unmodified exact-base worktree, through the required build-slot wrapper, with a 60-second queue limit:

build-slot: [09:06:20] all 4 slots busy; queue position 9, waiting...
Terminated

Wrapper exit: 124 (timeout). scripts/verify fast did not start; no green gate or test claim. This is a capacity limitation, not a test failure. The raw log timestamp is the host's output, quoted verbatim. VM log: /home/claude/.cache/codex-scratch/p5/evidence/verify-fast.log. No compiler/test/build was left running by this job; temporary VM source checkouts are being removed. No target directory was created.

Owner critique is requested on the recommendation-first grill in the issue body; no implementation begins from this research task. Issues remain open.

P5 planning deliverable posted and read back byte-for-byte. Engine choice is the first owner question; Caddy remains unmeasured pending the parallel J20 extension. Acquisition/custody/supervision/model/UI remain engine-neutral; only the selected adapter is proposed for initial implementation. No code, dependency, decision or release pin changed. No push/deploy/commit performed. Branch: `research/p5-traefik-plan`. Base/head: `f4b61cd6c3f52078cad7573c5367d7a6e75f84f0`. Planning worktree is clean. Plan/evidence copy: `/home/kayg/.cache/codex-scratch/p5/evidence/implementation-plan.md`. Gate attempt on compute01, unmodified exact-base worktree, through the required build-slot wrapper, with a 60-second queue limit: ```text build-slot: [09:06:20] all 4 slots busy; queue position 9, waiting... Terminated ``` Wrapper exit: `124` (timeout). `scripts/verify fast` did not start; no green gate or test claim. This is a capacity limitation, not a test failure. The raw log timestamp is the host's output, quoted verbatim. VM log: `/home/claude/.cache/codex-scratch/p5/evidence/verify-fast.log`. No compiler/test/build was left running by this job; temporary VM source checkouts are being removed. No target directory was created. Owner critique is requested on the recommendation-first grill in the issue body; no implementation begins from this research task. Issues remain open.

kayg commented 2026-11-01 08:13:05 +00:00

Author

Owner

Copy link

Cleanup confirmed: both temporary VM source checkouts removed; no P5 VM processes remain. The local planning worktree and small evidence files are retained for owner critique. No builds or tests ran, and no target directory was created.

Cleanup confirmed: both temporary VM source checkouts removed; no P5 VM processes remain. The local planning worktree and small evidence files are retained for owner critique. No builds or tests ran, and no target directory was created.

kayg commented 2026-11-01 09:58:11 +00:00

Author

Owner

Copy link

Owner decision (kayg, 2026-11-01), final: reverse-proxy engine = Traefik, downloaded on demand (R3 comparison 72700 on #261). Caddy idle RSS measured 48 MiB (not 17), half of Traefik's 94 MiB, but it drops existing connections on every config reload (Traefik kept all, 6/6), is slower under load (c512 13k vs 21–23k req/s; p99 70–74 vs 42–53 ms) and needs a custom build for Cloudflare DNS-01. Connection-preserving reloads are required because the UI changes proxy config while terminals/canvas run over long-lived WebSockets.

**Owner decision (kayg, 2026-11-01), final: reverse-proxy engine = Traefik, downloaded on demand** (R3 comparison 72700 on #261). Caddy idle RSS measured 48 MiB (not 17), half of Traefik's 94 MiB, but it drops existing connections on every config reload (Traefik kept all, 6/6), is slower under load (c512 13k vs 21–23k req/s; p99 70–74 vs 42–53 ms) and needs a custom build for Cloudflare DNS-01. Connection-preserving reloads are required because the UI changes proxy config while terminals/canvas run over long-lived WebSockets.

kayg commented 2026-11-01 10:16:24 +00:00

Author

Owner

Copy link

Owner answers (kayg, 2026-11-01) on the #281 grill:

• P2 trust: release CI verifies Traefik's published traefik_vX_checksums.txt (SHA-256) for the pinned release AND signs our own manifest of the verified bytes; nodes fetch from our immutable mirror and verify our manifest. Both checks.
• P3 scope: mesh-only exposure first; public exposure as a later opt-in slice; auto-detect when something else holds 80/443; rootless mode uses 10080/10443 (owner's choice over 8080/8443).
• P4 names/TLS: agreed, changeable; owner wants one wildcard certificate instead of one per service — naming shape pending (flat <service>.ssh.kayg.org under *.ssh.kayg.org vs per-machine wildcards).
• P5 ports: as above; INVESTIGATE: when running as root and another web server already holds 80/443, is there a safe way to take the mesh IP's 80/443 only (e.g. nftables redirect on the mesh interface to 10080/10443; behaviour if the other server binds the mesh IP explicitly).
• P6 custody: agreed; Cloudflare is only an example — support every DNS provider Traefik's ACME DNS-01 supports, chosen in the UI.
• P7 lifecycle, P9 budgets: agreed.
• P8 CLI: owner asked for a more human-friendly verb than expose; orchestrator proposed serve (pending).

**Owner answers (kayg, 2026-11-01) on the #281 grill:** - P2 trust: release CI verifies Traefik's published `traefik_vX_checksums.txt` (SHA-256) for the pinned release AND signs our own manifest of the verified bytes; nodes fetch from our immutable mirror and verify our manifest. Both checks. - P3 scope: mesh-only exposure first; public exposure as a later opt-in slice; auto-detect when something else holds 80/443; **rootless mode uses 10080/10443** (owner's choice over 8080/8443). - P4 names/TLS: agreed, changeable; owner wants one wildcard certificate instead of one per service — naming shape pending (flat `<service>.ssh.kayg.org` under `*.ssh.kayg.org` vs per-machine wildcards). - P5 ports: as above; INVESTIGATE: when running as root and another web server already holds 80/443, is there a safe way to take the mesh IP's 80/443 only (e.g. nftables redirect on the mesh interface to 10080/10443; behaviour if the other server binds the mesh IP explicitly). - P6 custody: agreed; Cloudflare is only an example — support every DNS provider Traefik's ACME DNS-01 supports, chosen in the UI. - P7 lifecycle, P9 budgets: agreed. - P8 CLI: owner asked for a more human-friendly verb than `expose`; orchestrator proposed `serve` (pending).

kayg referenced this issue 2026-11-01 10:16:25 +00:00

Secrets: an intuitive interface over OpenBao #287

kayg commented 2026-11-01 10:57:11 +00:00

Author

Owner

Copy link

J62 started on branch proxy-traefik-281, based on 7bf03044f0003b4b5280be93570ec0e1fe5b356d (origin/dev). I’m checking the owner’s #281 engine, trust, access, TLS and ports answers plus the #261/#266/#211 dependencies before writing failing tests. Next are the requested acquisition, supervision and CLI slices, then the compute01 build-slot gates and size report. No code changes yet.

J62 started on branch `proxy-traefik-281`, based on `7bf03044f0003b4b5280be93570ec0e1fe5b356d` (`origin/dev`). I’m checking the owner’s #281 engine, trust, access, TLS and ports answers plus the #261/#266/#211 dependencies before writing failing tests. Next are the requested acquisition, supervision and CLI slices, then the compute01 build-slot gates and size report. No code changes yet.

kayg referenced this issue 2026-11-01 11:11:36 +00:00

Secrets: node-sealed custody for DNS-01 tokens (replaces J62's owner-only file) #289

kayg commented 2026-11-01 11:12:51 +00:00

Author

Owner

Copy link

J62 audit finding (base 7bf03044f): still enforces the 128 MiB app payload cap; Traefik 3.7.17 is 182,120,608 bytes per J20/R3, so component payloads need their own strict cap. Mesh coordination exposes a , but the current server/node config does not yet carry #266's public-hostname-derived suffix; I will keep naming behind one function that consumes that suffix and avoid duplicating #266's DNS publication path.\n\nCustody finding: there is no generic node-sealed credential store in this base. The existing CA wrapper is user-scoped server-side key custody, not a node secret API. Per kayg's answer in this session, J62 will implement the interim interface with owner-only file storage under agent state (0700 directory; 0600 exclusive temp file + atomic rename; verify owner/mode), Traefik receives only provider paths, and provider removal deletes its files. No token values enter config, argv, environment, or logs. The module will state that node sealing/OpenBao replaces this implementation behind the interface (#287 follow-up).

J62 audit finding (base 7bf03044f): still enforces the 128 MiB app payload cap; Traefik 3.7.17 is 182,120,608 bytes per J20/R3, so component payloads need their own strict cap. Mesh coordination exposes a , but the current server/node config does not yet carry #266's public-hostname-derived suffix; I will keep naming behind one function that consumes that suffix and avoid duplicating #266's DNS publication path.\n\nCustody finding: there is no generic node-sealed credential store in this base. The existing CA wrapper is user-scoped server-side key custody, not a node secret API. Per kayg's answer in this session, J62 will implement the interim interface with owner-only file storage under agent state (0700 directory; 0600 exclusive temp file + atomic rename; verify owner/mode), Traefik receives only provider paths, and provider removal deletes its files. No token values enter config, argv, environment, or logs. The module will state that node sealing/OpenBao replaces this implementation behind the interface (#287 follow-up).

kayg commented 2026-11-01 11:13:08 +00:00

Author

Owner

Copy link

J62 audit finding (base 7bf03044f): `release_verify::verify_digest_expected` still enforces the 128 MiB app payload cap; Traefik 3.7.17 is 182,120,608 bytes per J20/R3, so component payloads need their own strict cap. Mesh coordination exposes a `dns_suffix`, but the current server/node config does not yet carry #266's public-hostname-derived suffix; I will keep naming behind one function that consumes that suffix and avoid duplicating #266's DNS publication path.

Custody finding: there is no generic node-sealed credential store in this base. The existing CA wrapper is user-scoped server-side key custody, not a node secret API. Per kayg's answer in this session, J62 will implement the interim `DnsTokenCustody` interface with owner-only file storage under agent state (0700 directory; 0600 exclusive temp file + atomic rename; verify owner/mode), Traefik receives only provider `*_FILE` paths, and provider removal deletes its files. No token values enter config, argv, environment, or logs. The module will state that node sealing/OpenBao replaces this implementation behind the interface (#287 follow-up).

J62 audit finding (base 7bf03044f): \`release_verify::verify_digest_expected\` still enforces the 128 MiB app payload cap; Traefik 3.7.17 is 182,120,608 bytes per J20/R3, so component payloads need their own strict cap. Mesh coordination exposes a \`dns_suffix\`, but the current server/node config does not yet carry #266's public-hostname-derived suffix; I will keep naming behind one function that consumes that suffix and avoid duplicating #266's DNS publication path. Custody finding: there is no generic node-sealed credential store in this base. The existing CA wrapper is user-scoped server-side key custody, not a node secret API. Per kayg's answer in this session, J62 will implement the interim \`DnsTokenCustody\` interface with owner-only file storage under agent state (0700 directory; 0600 exclusive temp file + atomic rename; verify owner/mode), Traefik receives only provider \`*_FILE\` paths, and provider removal deletes its files. No token values enter config, argv, environment, or logs. The module will state that node sealing/OpenBao replaces this implementation behind the interface (#287 follow-up).

kayg commented 2026-11-01 11:19:06 +00:00

Author

Owner

Copy link

Correction to the previous J62 audit notes; their shell formatting damaged the evidence text.

At base 7bf03044f, crates/infraternal-server/src/release_verify.rs::verify_digest_expected keeps the 128 MiB app payload limit. J20/R3 measured the Traefik executable at 182,120,608 bytes, so component downloads need their own bounded policy.

Mesh coordination currently has a dns_suffix, but the public-hostname-derived suffix from #266 is not carried by this base's server/node configuration. J62 will keep name construction behind one function that accepts the configured suffix.

Per kayg's custody decision in this session, J62 will define DnsTokenCustody with a temporary owner-only file implementation under the agent state directory: create and verify a 0700 directory, use 0600 exclusive temporary files and atomic rename, refuse owner/mode mismatches, pass only provider *_FILE paths to Traefik, and delete provider files on removal. The module will document that node sealing or OpenBao replaces the backend without changing callers (#287 follow-up).

Correction to the previous J62 audit notes; their shell formatting damaged the evidence text. At base 7bf03044f, crates/infraternal-server/src/release_verify.rs::verify_digest_expected keeps the 128 MiB app payload limit. J20/R3 measured the Traefik executable at 182,120,608 bytes, so component downloads need their own bounded policy. Mesh coordination currently has a dns_suffix, but the public-hostname-derived suffix from #266 is not carried by this base's server/node configuration. J62 will keep name construction behind one function that accepts the configured suffix. Per kayg's custody decision in this session, J62 will define DnsTokenCustody with a temporary owner-only file implementation under the agent state directory: create and verify a 0700 directory, use 0600 exclusive temporary files and atomic rename, refuse owner/mode mismatches, pass only provider *_FILE paths to Traefik, and delete provider files on removal. The module will document that node sealing or OpenBao replaces the backend without changing callers (#287 follow-up).

kayg commented 2026-11-01 11:47:19 +00:00

Author

Owner

Copy link

J62 port-mapping investigation (#281): do not install a generic nftables redirect for occupied 80/443.

Linux destination NAT in a prerouting chain runs before route selection and local socket lookup. A rule scoped to the mesh interface and mesh destination can therefore rewrite packets before the kernel selects a listener. If another service explicitly binds the mesh address on port 80/443, a redirect to 10080/10443 would take that service's mesh traffic too; the explicit bind does not protect it. Interface scoping preserves non-mesh ingress, but it does not preserve the existing mesh listener. A safe rule would need an additional owner-controlled port-binding contract and lifecycle ownership of the nftables table/rules, which this slice does not have.

J62 will bind root-mode 80/443 only when those ports are available on the selected mesh address. If occupied, it will use 10080/10443 and report the explicit ports in the serve result. No host firewall rules are changed. This conclusion follows nftables' documented prerouting DNAT order; it is not a live host firewall experiment.

J62 port-mapping investigation (#281): do not install a generic nftables redirect for occupied 80/443. Linux destination NAT in a prerouting chain runs before route selection and local socket lookup. A rule scoped to the mesh interface and mesh destination can therefore rewrite packets before the kernel selects a listener. If another service explicitly binds the mesh address on port 80/443, a redirect to 10080/10443 would take that service's mesh traffic too; the explicit bind does not protect it. Interface scoping preserves non-mesh ingress, but it does not preserve the existing mesh listener. A safe rule would need an additional owner-controlled port-binding contract and lifecycle ownership of the nftables table/rules, which this slice does not have. J62 will bind root-mode 80/443 only when those ports are available on the selected mesh address. If occupied, it will use 10080/10443 and report the explicit ports in the serve result. No host firewall rules are changed. This conclusion follows nftables' documented prerouting DNAT order; it is not a live host firewall experiment.

kayg commented 2026-11-01 12:05:22 +00:00

Author

Owner

Copy link

J62 base alignment: proxy-traefik-281 now starts at current origin/dev 5418f93574c6be84766f9a7dcc196b34b27ff3bc, which includes the D243 report-only size gate. The existing uncommitted crates/plugins/proxy custody scaffold and workspace entry were preserved. A targeted custody RED run remains queued on compute01; no test result yet. Next: observe the expected custody assertion failure, then implement the owner-only file store behind DnsTokenCustody.

J62 base alignment: `proxy-traefik-281` now starts at current `origin/dev` `5418f93574c6be84766f9a7dcc196b34b27ff3bc`, which includes the D243 report-only size gate. The existing uncommitted `crates/plugins/proxy` custody scaffold and workspace entry were preserved. A targeted custody RED run remains queued on compute01; no test result yet. Next: observe the expected custody assertion failure, then implement the owner-only file store behind `DnsTokenCustody`.

kayg commented 2026-11-01 12:13:50 +00:00

Author

Owner

Copy link

Crash-safety checkpoint pushed: proxy-traefik-281 at 1bf578a8ed2c6e271e1787b17cab2e95a6be6420, based on origin/dev 5418f9357. This is an explicitly unqualified WIP containing the proxy crate scaffold and custody RED tests; implementation remains a refusing stub. The assigned compute01 test has waited in build-slot for about 90 minutes and is currently reported at queue position 39. Log: /home/claude/.cache/codex-scratch/j62/custody-red.log. No test result is claimed; I’m waiting for its assertion output before implementing.

Crash-safety checkpoint pushed: `proxy-traefik-281` at `1bf578a8ed2c6e271e1787b17cab2e95a6be6420`, based on `origin/dev` `5418f9357`. This is an explicitly unqualified WIP containing the proxy crate scaffold and custody RED tests; implementation remains a refusing stub. The assigned compute01 test has waited in `build-slot` for about 90 minutes and is currently reported at queue position 39. Log: `/home/claude/.cache/codex-scratch/j62/custody-red.log`. No test result is claimed; I’m waiting for its assertion output before implementing.

kayg commented 2026-11-01 12:15:23 +00:00

Author

Owner

Copy link

J62 build-admission finding: compute01 reports /dev/sda1 at 457 GB used of 492 GB, 16 GB free (97%). build-slot is refusing admission until at least 25 GB is free. I canceled only the stale J62 test wrapper because its snapshot lacked the new proxy package in Cargo.lock; it never reached Cargo or ran an assertion. No other jobs' targets were removed. J62 has no test result yet. Please arrange cleanup of an appropriate finished-job target on compute01; I will resubmit the corrected RED test after admission is available.

J62 build-admission finding: compute01 reports `/dev/sda1` at 457 GB used of 492 GB, 16 GB free (97%). `build-slot` is refusing admission until at least 25 GB is free. I canceled only the stale J62 test wrapper because its snapshot lacked the new proxy package in `Cargo.lock`; it never reached Cargo or ran an assertion. No other jobs' targets were removed. J62 has no test result yet. Please arrange cleanup of an appropriate finished-job target on compute01; I will resubmit the corrected RED test after admission is available.

kayg referenced this issue 2026-11-01 13:39:03 +00:00

Mesh: SSH over the mesh (D197 M1, D235) #221

kayg commented 2026-11-01 13:51:58 +00:00

Author

Owner

Copy link

J62 progress on branch proxy-traefik-281 (base 5418f93574c6be84766f9a7dcc196b34b27ff3bc, head 1e3eaba89).

Owner direction is recorded in code: DnsTokenCustody with put/get/delete/rotate, backed for now by owner-only files. The module documents that node-key sealing or OpenBao under #289 can replace the implementation without changing callers. The store requires a 0700 state directory, uses 0700 provider directories and 0600 token files, stages with exclusive creation and atomically renames, and returns only the path used by a provider *_FILE variable. Debug output is redacted.

Compute01 red/green evidence via build-slot: expected red was 3 passed / 8 failed against the stub; green was 11 passed / 0 failed for cargo test --locked -p infraternal-plugin-proxy --lib -- --nocapture. The per-job target is /home/claude/.cache/codex-scratch/j62/target.

Next: acquisition manifest and installer, Traefik supervision/configuration and mesh grant wiring, CLI, lifecycle/namespace coverage, then the fast gate and size report.

J62 progress on branch `proxy-traefik-281` (base `5418f93574c6be84766f9a7dcc196b34b27ff3bc`, head `1e3eaba89`). Owner direction is recorded in code: `DnsTokenCustody` with `put/get/delete/rotate`, backed for now by owner-only files. The module documents that node-key sealing or OpenBao under #289 can replace the implementation without changing callers. The store requires a 0700 state directory, uses 0700 provider directories and 0600 token files, stages with exclusive creation and atomically renames, and returns only the path used by a provider `*_FILE` variable. Debug output is redacted. Compute01 red/green evidence via build-slot: expected red was 3 passed / 8 failed against the stub; green was 11 passed / 0 failed for `cargo test --locked -p infraternal-plugin-proxy --lib -- --nocapture`. The per-job target is `/home/claude/.cache/codex-scratch/j62/target`. Next: acquisition manifest and installer, Traefik supervision/configuration and mesh grant wiring, CLI, lifecycle/namespace coverage, then the fast gate and size report.

kayg commented 2026-11-01 14:25:27 +00:00

Author

Owner

Copy link

J62 resumed from pushed WIP and rebased proxy-traefik-281 onto current origin/dev 37bbda45554f6e603a6856fc23319e17d8a1b14f0. Current pushed head: 9a9dafbea6aad3d1eb0ce8d79bce5990533d6063.

Release acquisition checkpoint: release builds now download pinned Traefik 3.7.17 platform archives, verify the upstream checksum list, inspect the allowlisted executable and MIT licence, and place archives under the on-demand components/<release>/proxy/... path. The existing release signer includes the small per-target JSON manifests in its signed checksum list; the 128 MiB application payload limit stays unchanged. scripts/release/build, publish, verify and the fast gate now include the component files/tests.

Compute01 evidence: python3 -m unittest discover -s scripts/release -p traefik_component_test.py → Ran 5 tests in 0.009s; OK; python3 scripts/release/test.py → Ran 8 tests in 11.443s; OK. The detached cargo test --locked -p infraternal-plugin-proxy is still waiting in build-slot at queue position 3; no Rust test result yet. Log: /home/claude/.cache/codex-scratch/j62/proxy-test-resubmitted.log.

Next: observe the Rust test result, finish installer verification and the mesh-only Traefik render/supervisor tests, then run the final VM gate and size report. Interim DNS-token custody remains the DnsTokenCustody owner-only file implementation per #289; sealing/OpenBao stays behind that interface.

J62 resumed from pushed WIP and rebased `proxy-traefik-281` onto current `origin/dev` `37bbda45554f6e603a6856fc23319e17d8a1b14f0`. Current pushed head: `9a9dafbea6aad3d1eb0ce8d79bce5990533d6063`. Release acquisition checkpoint: release builds now download pinned Traefik 3.7.17 platform archives, verify the upstream checksum list, inspect the allowlisted executable and MIT licence, and place archives under the on-demand `components/<release>/proxy/...` path. The existing release signer includes the small per-target JSON manifests in its signed checksum list; the 128 MiB application payload limit stays unchanged. `scripts/release/build`, publish, verify and the fast gate now include the component files/tests. Compute01 evidence: `python3 -m unittest discover -s scripts/release -p traefik_component_test.py` → `Ran 5 tests in 0.009s; OK`; `python3 scripts/release/test.py` → `Ran 8 tests in 11.443s; OK`. The detached `cargo test --locked -p infraternal-plugin-proxy` is still waiting in `build-slot` at queue position 3; no Rust test result yet. Log: `/home/claude/.cache/codex-scratch/j62/proxy-test-resubmitted.log`. Next: observe the Rust test result, finish installer verification and the mesh-only Traefik render/supervisor tests, then run the final VM gate and size report. Interim DNS-token custody remains the `DnsTokenCustody` owner-only file implementation per #289; sealing/OpenBao stays behind that interface.

kayg commented 2026-11-01 14:37:36 +00:00

Author

Owner

Copy link

J62 acquisition RED finding, compute01 through build-slot on 2026-11-01. The test fixture originally failed while constructing ../outside, before the installer boundary; I changed it to write the malicious tar header directly and reran the focused suite.

The corrected RED result against the installer stub is:

running 3 tests
installer_tests::installer_rejects_bad_archive_checksum_and_wrong_architecture ... ok
installer_tests::installer_extracts_only_the_allowlisted_executable_with_private_mode ... FAILED
installer_tests::installer_rejects_symlink_and_traversal_entries ... ok

failures:
    installer_tests::installer_extracts_only_the_allowlisted_executable_with_private_mode

test result: FAILED. 2 passed; 1 failed; 0 ignored; 0 measured; 16 filtered out

The failing path returns Kind(Unsupported) from the current stub. The traversal/symlink test now reaches the installer boundary. Log: /home/claude/.cache/codex-scratch/j62/installer-red-fixed.log; target: /home/claude/.cache/codex-scratch/j62/target. Next: implement locked bounded staging, checksum-before-extract, allowlisted extraction, executable re-verification and atomic digest-path publication.

J62 acquisition RED finding, compute01 through `build-slot` on 2026-11-01. The test fixture originally failed while constructing `../outside`, before the installer boundary; I changed it to write the malicious tar header directly and reran the focused suite. The corrected RED result against the installer stub is: ```text running 3 tests installer_tests::installer_rejects_bad_archive_checksum_and_wrong_architecture ... ok installer_tests::installer_extracts_only_the_allowlisted_executable_with_private_mode ... FAILED installer_tests::installer_rejects_symlink_and_traversal_entries ... ok failures: installer_tests::installer_extracts_only_the_allowlisted_executable_with_private_mode test result: FAILED. 2 passed; 1 failed; 0 ignored; 0 measured; 16 filtered out ``` The failing path returns `Kind(Unsupported)` from the current stub. The traversal/symlink test now reaches the installer boundary. Log: `/home/claude/.cache/codex-scratch/j62/installer-red-fixed.log`; target: `/home/claude/.cache/codex-scratch/j62/target`. Next: implement locked bounded staging, checksum-before-extract, allowlisted extraction, executable re-verification and atomic digest-path publication.

kayg referenced this issue 2026-11-01 14:42:58 +00:00

infraternal up + one-command install #263

kayg commented 2026-11-01 14:42:58 +00:00

Author

Owner

Copy link

Owner answers, continued (kayg, 2026-11-01):

• Q20 (name shape): both shapes should be possible, with one wildcard certificate per pattern. The owner asked whether a stolen certificate is a concern; the orchestrator is answering (key custody and CT-log exposure).
• Q21 (verb): the owner prefers publish; serve reads like it changes how the infraternal server serves. Open: how to tell internal publishing from external. Orchestrator proposal: publish defaults to mesh-only, and --public exposes to the internet.
• #289: (a). Move DNS tokens to OpenBao once #287 exists.

Owner answers, continued (kayg, 2026-11-01): - **Q20 (name shape):** both shapes should be possible, with one wildcard certificate per pattern. The owner asked whether a stolen certificate is a concern; the orchestrator is answering (key custody and CT-log exposure). - **Q21 (verb):** the owner prefers **`publish`**; `serve` reads like it changes how the infraternal server serves. Open: how to tell internal publishing from external. Orchestrator proposal: `publish` defaults to mesh-only, and `--public` exposes to the internet. - **#289:** (a). Move DNS tokens to OpenBao once #287 exists.

kayg commented 2026-11-01 14:46:52 +00:00

Author

Owner

Copy link

Owner answers (kayg, 2026-11-01):

• Q20: both name shapes are supported. Per-machine wildcard (*.<machine>.<suffix>) is the default, and its private key never leaves that machine. Flat <service>.<suffix> names get exact per-service certificates issued on the serving node. No wildcard key is ever copied between nodes. Accepted trade-off: exact names appear in public Certificate Transparency logs.
• Q21: the verb is publish.
  • infraternal publish <port> is mesh-only: reachable at the mesh name by members with access.
  • --public also exposes it to the internet, with confirmation the first time.
  • infraternal publish with no argument lists what is published and where.

Owner answers (kayg, 2026-11-01): - **Q20:** both name shapes are supported. **Per-machine wildcard (`*.<machine>.<suffix>`) is the default**, and its private key never leaves that machine. Flat `<service>.<suffix>` names get **exact per-service certificates issued on the serving node**. No wildcard key is ever copied between nodes. Accepted trade-off: exact names appear in public Certificate Transparency logs. - **Q21:** the verb is **`publish`**. - `infraternal publish <port>` is mesh-only: reachable at the mesh name by members with access. - `--public` also exposes it to the internet, with confirmation the first time. - `infraternal publish` with no argument lists what is published and where.

kayg commented 2026-11-01 14:55:58 +00:00

Author

Owner

Copy link

J62 resumed from pushed head 9a9dafbea6aad3d1eb0ce8d79bce5990533d6063 on proxy-traefik-281, based on origin/dev 37bbda45554f6e603a6856fc23319e17d8a1b14f0.

The latest owner decisions are recorded here: publish is the verb, bare publish lists entries, a port publishes mesh-only by default, and --public needs first-use confirmation. Both certificate shapes are supported; per-machine wildcard is the default and stays on its node, while flat names use exact per-service certificates whose names enter CT logs. I added a focused plan and test-first coverage for both naming forms and the one-command table.

Compute01 currently reports 8 GB free (build-slot floor: 25 GB). The focused red run was submitted detached as PID 1033773 and is waiting for disk admission; it has not started compiling. Log: /home/claude/.cache/codex-scratch/j62/proxy-owner-decisions-red.log. No result is claimed. Once space is available I will observe the expected red result, then implement and verify the naming/CLI behavior.

J62 resumed from pushed head `9a9dafbea6aad3d1eb0ce8d79bce5990533d6063` on `proxy-traefik-281`, based on `origin/dev` `37bbda45554f6e603a6856fc23319e17d8a1b14f0`. The latest owner decisions are recorded here: `publish` is the verb, bare `publish` lists entries, a port publishes mesh-only by default, and `--public` needs first-use confirmation. Both certificate shapes are supported; per-machine wildcard is the default and stays on its node, while flat names use exact per-service certificates whose names enter CT logs. I added a focused plan and test-first coverage for both naming forms and the one-command table. Compute01 currently reports 8 GB free (build-slot floor: 25 GB). The focused red run was submitted detached as PID 1033773 and is waiting for disk admission; it has not started compiling. Log: `/home/claude/.cache/codex-scratch/j62/proxy-owner-decisions-red.log`. No result is claimed. Once space is available I will observe the expected red result, then implement and verify the naming/CLI behavior.

kayg commented 2026-11-01 15:10:11 +00:00

Author

Owner

Copy link

J62 owner-decision RED evidence on pushed head 521a4b347:

failures:
    cli_tests::publish_is_the_single_root_command_with_optional_port_and_public_scope
    names_tests::flat_service_names_receive_an_exact_service_certificate
    names_tests::per_machine_wildcard_is_the_default_and_covers_the_machine_service_name

test result: FAILED. 22 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

This is the expected red result: the two owner-approved naming forms and the single publish action are still stubs. Log: /home/claude/.cache/codex-scratch/j62/proxy-owner-decisions-red-5.log. No run is pending. I am implementing the tested contracts now.

J62 owner-decision RED evidence on pushed head `521a4b347`: ```text failures: cli_tests::publish_is_the_single_root_command_with_optional_port_and_public_scope names_tests::flat_service_names_receive_an_exact_service_certificate names_tests::per_machine_wildcard_is_the_default_and_covers_the_machine_service_name test result: FAILED. 22 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s ``` This is the expected red result: the two owner-approved naming forms and the single `publish` action are still stubs. Log: `/home/claude/.cache/codex-scratch/j62/proxy-owner-decisions-red-5.log`. No run is pending. I am implementing the tested contracts now.

kayg commented 2026-11-01 15:17:13 +00:00

Author

Owner

Copy link

Crash-safe checkpoint pushed: proxy-traefik-281 at c701f9423.

The naming and CLI RED evidence remains the expected 22 passed / 3 failed recorded above. The implementation now maps both owner-approved certificate shapes, defaults to node-local per-machine wildcards, exposes one optional-port publish action with --public, --confirm-public, and --flat-name, and documents CT-log visibility for exact names.

A focused crate GREEN rerun is queued on compute01 as PID 1209626, behind four active build-slot jobs. Log: /home/claude/.cache/codex-scratch/j62/proxy-naming-green-2.log. It uses the VM copy that contains the naming implementation and excludes the newer config/download stubs; no result is claimed yet. Next I will observe that run, then drive the acquisition and config tests RED/GREEN before implementing their behavior.

Crash-safe checkpoint pushed: `proxy-traefik-281` at `c701f9423`. The naming and CLI RED evidence remains the expected 22 passed / 3 failed recorded above. The implementation now maps both owner-approved certificate shapes, defaults to node-local per-machine wildcards, exposes one optional-port `publish` action with `--public`, `--confirm-public`, and `--flat-name`, and documents CT-log visibility for exact names. A focused crate GREEN rerun is queued on compute01 as PID 1209626, behind four active build-slot jobs. Log: `/home/claude/.cache/codex-scratch/j62/proxy-naming-green-2.log`. It uses the VM copy that contains the naming implementation and excludes the newer config/download stubs; no result is claimed yet. Next I will observe that run, then drive the acquisition and config tests RED/GREEN before implementing their behavior.

kayg commented 2026-11-01 15:24:24 +00:00

Author

Owner

Copy link

J62 resumed from pushed proxy-traefik-281 head c701f9423, based on current origin/dev 37bbda45554f6e603a6856fc23319e17d8a1b14f. The compute01 cargo test -p infraternal-plugin-proxy --lib run remains queued (PID 1209626); four slots are busy and free disk is 208 GB, so there is no result yet.

Source audit findings: download.rs::ComponentDownloader::install, installer.rs::install_stream, and all three config.rs render/write entry points still return Unsupported; the proxy crate has no supervisor module or process integration yet. The signed manifest, offline bundle import, custody store, installer extraction tests, naming function, and publish action schema are present. These findings are from the current branch source, not test claims.

Plan: use the queued crate result as the first RED receipt; finish downloader/config behavior and add lifecycle/security coverage; wire one process owner to the existing agent component lifecycle and action registry; run the crate tests, required fast gate and size report on VMs; investigate/record occupied-port behavior (the existing owner note rejects a generic nftables redirect because it would capture an explicitly bound mesh listener); then clean the J62 target. Estimate: about 2 hours of implementation/verification time, with slot wait separate. No gate is claimed yet.

J62 resumed from pushed `proxy-traefik-281` head `c701f9423`, based on current `origin/dev` `37bbda45554f6e603a6856fc23319e17d8a1b14f`. The compute01 `cargo test -p infraternal-plugin-proxy --lib` run remains queued (PID 1209626); four slots are busy and free disk is 208 GB, so there is no result yet. Source audit findings: `download.rs::ComponentDownloader::install`, `installer.rs::install_stream`, and all three `config.rs` render/write entry points still return `Unsupported`; the proxy crate has no supervisor module or process integration yet. The signed manifest, offline bundle import, custody store, installer extraction tests, naming function, and publish action schema are present. These findings are from the current branch source, not test claims. Plan: use the queued crate result as the first RED receipt; finish downloader/config behavior and add lifecycle/security coverage; wire one process owner to the existing agent component lifecycle and action registry; run the crate tests, required fast gate and size report on VMs; investigate/record occupied-port behavior (the existing owner note rejects a generic nftables redirect because it would capture an explicitly bound mesh listener); then clean the J62 target. Estimate: about 2 hours of implementation/verification time, with slot wait separate. No gate is claimed yet.

kayg commented 2026-11-01 15:41:47 +00:00

Author

Owner

Copy link

J62 resumed on proxy-traefik-281 at pushed head c701f9423, based on origin/dev 37bbda455.

The resubmitted VM test completed with the expected installer RED on the pushed head: test result: FAILED. 17 passed; 2 failed. It identified the unsupported installer path and a traversal archive fixture rejected by the tar builder before reaching production parsing. The test log is /home/claude/.cache/codex-scratch/j62/proxy-test-resubmitted.log; the current WIP fixes the installer path and fixture.

The working tree now also contains implementations for bounded mirror/upstream streaming and atomic static/dynamic config rendering, following the already-committed RED tests. Next: run those tests GREEN on a VM, add the owned Traefik process lifecycle and CLI/agent integration, then exercise log redaction and staging ACME. Occupied 80/443 falls back to 10080/10443: an nftables redirect on the mesh interface would still divert traffic addressed to a server explicitly bound to that mesh IP, with no safe way to identify ownership, so no redirect is planned.

J62 resumed on `proxy-traefik-281` at pushed head `c701f9423`, based on `origin/dev` `37bbda455`. The resubmitted VM test completed with the expected installer RED on the pushed head: `test result: FAILED. 17 passed; 2 failed`. It identified the unsupported installer path and a traversal archive fixture rejected by the tar builder before reaching production parsing. The test log is `/home/claude/.cache/codex-scratch/j62/proxy-test-resubmitted.log`; the current WIP fixes the installer path and fixture. The working tree now also contains implementations for bounded mirror/upstream streaming and atomic static/dynamic config rendering, following the already-committed RED tests. Next: run those tests GREEN on a VM, add the owned Traefik process lifecycle and CLI/agent integration, then exercise log redaction and staging ACME. Occupied 80/443 falls back to 10080/10443: an nftables redirect on the mesh interface would still divert traffic addressed to a server explicitly bound to that mesh IP, with no safe way to identify ownership, so no redirect is planned.

kayg commented 2026-11-01 16:00:00 +00:00

Author

Owner

Copy link

J62 resumed from pushed head c701f9423 on proxy-traefik-281, based on origin/dev 37bbda455. No J62 build/test process is active in either VM queue.

Plan and estimate: about 2 hours of implementation plus required gates; build-slot wait excluded. (1) Run the current supervisor tests RED on dev-02 through build-slot. (2) Complete the owned Traefik process lifecycle and wire it through the existing infraternal-agent plugin/action path, including private ping, drain/backoff and redacted logs. (3) Close security/config gaps for provider *_FILE custody, listener selection, exact grant-bound routing, offline bundle parity and staging ACME coverage. (4) Run focused proxy tests, scripts/verify fast on dev-02, the x86 static-size report on compute01, then commit/push and remove only the J62 target.

Current branch worktree has uncommitted acquisition, config, signature-verification and supervisor scaffolding. Logs/evidence remain under /home/claude/.cache/codex-scratch/j62/; the VM target will be kept to one path and removed at completion.

J62 resumed from pushed head `c701f9423` on `proxy-traefik-281`, based on `origin/dev` `37bbda455`. No J62 build/test process is active in either VM queue. Plan and estimate: about 2 hours of implementation plus required gates; build-slot wait excluded. (1) Run the current supervisor tests RED on dev-02 through build-slot. (2) Complete the owned Traefik process lifecycle and wire it through the existing infraternal-agent plugin/action path, including private ping, drain/backoff and redacted logs. (3) Close security/config gaps for provider `*_FILE` custody, listener selection, exact grant-bound routing, offline bundle parity and staging ACME coverage. (4) Run focused proxy tests, `scripts/verify fast` on dev-02, the x86 static-size report on compute01, then commit/push and remove only the J62 target. Current branch worktree has uncommitted acquisition, config, signature-verification and supervisor scaffolding. Logs/evidence remain under `/home/claude/.cache/codex-scratch/j62/`; the VM target will be kept to one path and removed at completion.

kayg commented 2026-11-01 16:17:01 +00:00

Author

Owner

Copy link

J62 resumed from pushed head c701f9423 on proxy-traefik-281, based on origin/dev 37bbda45554f6e603a6856fc23319e17d8a1b14f (D243 report-only is in the base history).

The focused supervisor_tests run remains queued on dev-02 as PID 3078693; build-slot --status reports 19 GB free versus the 25 GB admission floor, so it has not started. I am leaving it queued.

Plan (about 2 hours of implementation and verification; queue wait separate): finish the current config/downloader WIP, connect the proxy supervisor and publish action to the infraternal agent, add lifecycle and secret-redaction coverage, run focused and fast gates on VMs, record static-size deltas, remove the J62 target, then push and post the exact gate output. I will preserve #266 grant/suffix checks and the #211 update-adoption boundary; no interruption-free update claim without adoption evidence. The occupied-port investigation rejects an nftables redirect because it would also divert traffic explicitly bound to the mesh address.

DNS tokens continue to use the owner-approved interim DnsTokenCustody file backend from #289: provider *_FILE path only, owner-only mode/ownership checks, deletion on provider removal, with sealing/OpenBao left behind the same interface.

J62 resumed from pushed head `c701f9423` on `proxy-traefik-281`, based on `origin/dev` `37bbda45554f6e603a6856fc23319e17d8a1b14f` (D243 report-only is in the base history). The focused `supervisor_tests` run remains queued on dev-02 as PID 3078693; `build-slot --status` reports 19 GB free versus the 25 GB admission floor, so it has not started. I am leaving it queued. Plan (about 2 hours of implementation and verification; queue wait separate): finish the current config/downloader WIP, connect the proxy supervisor and `publish` action to the infraternal agent, add lifecycle and secret-redaction coverage, run focused and fast gates on VMs, record static-size deltas, remove the J62 target, then push and post the exact gate output. I will preserve #266 grant/suffix checks and the #211 update-adoption boundary; no interruption-free update claim without adoption evidence. The occupied-port investigation rejects an nftables redirect because it would also divert traffic explicitly bound to the mesh address. DNS tokens continue to use the owner-approved interim `DnsTokenCustody` file backend from #289: provider `*_FILE` path only, owner-only mode/ownership checks, deletion on provider removal, with sealing/OpenBao left behind the same interface.

kayg commented 2026-11-01 16:29:23 +00:00

Author

Owner

Copy link

J62 progress: the previous focused-run PID 3078693 was no longer present on either VM, and its referenced log was absent. I pushed the crash-safe WIP checkpoint bfb2902d6 and resubmitted cargo test --offline -j4 -p infraternal-plugin-proxy --lib supervisor_tests -- --nocapture on compute01 through build-slot as PID 1946713; log: /home/claude/.cache/codex-scratch/j62/supervisor-resubmit.log. The pre-existing compute01 J62 source snapshot lacked the new supervisor module, so the run uses a clean detached checkout of the pushed head and reuses the existing J62 target. I am tracing the agent registry and mesh permission path while it compiles.

J62 progress: the previous focused-run PID 3078693 was no longer present on either VM, and its referenced log was absent. I pushed the crash-safe WIP checkpoint `bfb2902d6` and resubmitted `cargo test --offline -j4 -p infraternal-plugin-proxy --lib supervisor_tests -- --nocapture` on compute01 through build-slot as PID 1946713; log: `/home/claude/.cache/codex-scratch/j62/supervisor-resubmit.log`. The pre-existing compute01 J62 source snapshot lacked the new supervisor module, so the run uses a clean detached checkout of the pushed head and reuses the existing J62 target. I am tracing the agent registry and mesh permission path while it compiles.

kayg commented 2026-11-01 16:35:52 +00:00

Author

Owner

Copy link

J62 supervisor RED/GREEN checkpoint: before the lifecycle implementation, supervisor_tests reached the assertions and failed 4/4 as expected (startup/drain/backoff were no-ops; provider token remained in logs). After implementing the state transitions, bounded exponential backoff, and longest-first log redaction, the fresh compute01 run passed: test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 36 filtered out; finished in 0.00s. The renderer compile blockers are fixed; its TOML formatter is included in the same focused crate compile.

J62 supervisor RED/GREEN checkpoint: before the lifecycle implementation, `supervisor_tests` reached the assertions and failed 4/4 as expected (startup/drain/backoff were no-ops; provider token remained in logs). After implementing the state transitions, bounded exponential backoff, and longest-first log redaction, the fresh compute01 run passed: `test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 36 filtered out; finished in 0.00s`. The renderer compile blockers are fixed; its TOML formatter is included in the same focused crate compile.

kayg commented 2026-11-01 16:51:02 +00:00

Author

Owner

Copy link

J62 resumed on proxy-traefik-281, pushed head 54010eabc5e3828dec25edf8923f3751364821c2, based on origin/dev 37bbda45554f6e603a6856fc23319e17d8a1b14f0.

I re-read the J62 brief, #281 plan/owner decisions, D316/D314/D303, and the current branch state. The focused config log on compute01 confirms the last renderer run was 4 passed / 2 failed: the private ping renderer emits manualRouting = true where the test and health contract require automatic routing, and the HTTPS router is named docs-https while the route contract expects one stable service key. The supervisor transition tests were reported green in the prior checkpoint. No J62 build is queued or running now; compute01 has 234 GB free, while dev-02 has 41 GB free and a queued unrelated build.

Next: fix the two renderer findings test-first; connect the proxy lifecycle and publish action to the real infraternal-agent path; complete the remaining provider-file/config and exposure lifecycle wiring; add namespace/Pebble coverage; run focused suites and scripts/verify fast on dev-02, then x86 size reporting on compute01. The nftables investigation remains closed: a mesh-interface redirect could still capture a service explicitly bound to the mesh IP, so no redirect will be installed.

Estimate: about 3 hours of implementation and verification, excluding slot wait. I will keep one J62 target and remove it at completion. Current log: /home/claude/.cache/codex-scratch/j62/config-ping-red.log.

J62 resumed on `proxy-traefik-281`, pushed head `54010eabc5e3828dec25edf8923f3751364821c2`, based on `origin/dev` `37bbda45554f6e603a6856fc23319e17d8a1b14f0`. I re-read the J62 brief, #281 plan/owner decisions, D316/D314/D303, and the current branch state. The focused config log on compute01 confirms the last renderer run was 4 passed / 2 failed: the private ping renderer emits `manualRouting = true` where the test and health contract require automatic routing, and the HTTPS router is named `docs-https` while the route contract expects one stable service key. The supervisor transition tests were reported green in the prior checkpoint. No J62 build is queued or running now; compute01 has 234 GB free, while dev-02 has 41 GB free and a queued unrelated build. Next: fix the two renderer findings test-first; connect the proxy lifecycle and publish action to the real infraternal-agent path; complete the remaining provider-file/config and exposure lifecycle wiring; add namespace/Pebble coverage; run focused suites and `scripts/verify fast` on dev-02, then x86 size reporting on compute01. The nftables investigation remains closed: a mesh-interface redirect could still capture a service explicitly bound to the mesh IP, so no redirect will be installed. Estimate: about 3 hours of implementation and verification, excluding slot wait. I will keep one J62 target and remove it at completion. Current log: `/home/claude/.cache/codex-scratch/j62/config-ping-red.log`.

kayg commented 2026-11-01 17:15:51 +00:00

Author

Owner

Copy link

J62 crash-safe checkpoint: pushed proxy-traefik-281 at 607c97459. Added failing-first tests for proxy registry presence and fail-closed mesh context, disjoint router keys, and credential redaction. The compute01 focused router test is still queued as PID 2057611 at position 3; no result is claimed. Log: /home/claude/.cache/codex-scratch/j62/config-router-red.log. Next: observe RED, fix the renderer, then finish agent and process wiring. #266 suffix/grant context remains fail-closed until supplied by the trusted coordinator.

J62 crash-safe checkpoint: pushed proxy-traefik-281 at 607c97459. Added failing-first tests for proxy registry presence and fail-closed mesh context, disjoint router keys, and credential redaction. The compute01 focused router test is still queued as PID 2057611 at position 3; no result is claimed. Log: /home/claude/.cache/codex-scratch/j62/config-router-red.log. Next: observe RED, fix the renderer, then finish agent and process wiring. #266 suffix/grant context remains fail-closed until supplied by the trusted coordinator.

kayg commented 2026-11-01 17:30:54 +00:00

Author

Owner

Copy link

J62 WIP checkpoint pushed at ff57a8b44. Added a real child-process test fixture asserting provider credentials are passed only via *_FILE paths, no token bytes reach argv, child logs are scrubbed, and the owned child is stopped/reaped. The targeted renderer GREEN rerun remains queued on compute01 as PID 2451894, currently behind two jobs; no result yet. Log: /home/claude/.cache/codex-scratch/j62/config-router-green.log. Next: observe renderer GREEN, then run the proxy and agent RED tests before implementing child ownership and registry wiring.

J62 WIP checkpoint pushed at ff57a8b44. Added a real child-process test fixture asserting provider credentials are passed only via *_FILE paths, no token bytes reach argv, child logs are scrubbed, and the owned child is stopped/reaped. The targeted renderer GREEN rerun remains queued on compute01 as PID 2451894, currently behind two jobs; no result yet. Log: /home/claude/.cache/codex-scratch/j62/config-router-green.log. Next: observe renderer GREEN, then run the proxy and agent RED tests before implementing child ownership and registry wiring.

kayg commented 2026-11-01 17:36:55 +00:00

Author

Owner

Copy link

Correction on the queued config GREEN run (PID 2451894): it was admitted, then exited before Cargo because its shell started in /home/claude rather than /home/claude/worktrees/J62-build. The log ends with: build-slot: [18:34:37] got slot 1/4; path "/home/claude" does not contain a 'flake.nix', searching up; error: could not find a flake.nix file. No test ran and no result is claimed. I am pushing the two-file renderer WIP checkpoint, then will resubmit with an explicit cd to the repo. Existing target remains /home/claude/worktrees/J62/target; free disk was 132G at the last status.

Correction on the queued config GREEN run (PID 2451894): it was admitted, then exited before Cargo because its shell started in /home/claude rather than /home/claude/worktrees/J62-build. The log ends with: build-slot: [18:34:37] got slot 1/4; path "/home/claude" does not contain a 'flake.nix', searching up; error: could not find a flake.nix file. No test ran and no result is claimed. I am pushing the two-file renderer WIP checkpoint, then will resubmit with an explicit cd to the repo. Existing target remains /home/claude/worktrees/J62/target; free disk was 132G at the last status.

kayg commented 2026-11-01 17:39:24 +00:00

Author

Owner

Copy link

Crash-safe progress checkpoint: pushed renderer WIP on . The queued GREEN resubmission is PID 2644471, with an explicit and the existing single target ; log: . At 18:37 UTC, build-slot showed four active jobs, J62 at queue position 4, and 130G free. No test result is claimed. Next: observe the crate GREEN, then run the committed registry, redaction, and child-boundary RED checks before implementing those paths.

Crash-safe progress checkpoint: pushed renderer WIP on . The queued GREEN resubmission is PID 2644471, with an explicit and the existing single target ; log: . At 18:37 UTC, build-slot showed four active jobs, J62 at queue position 4, and 130G free. No test result is claimed. Next: observe the crate GREEN, then run the committed registry, redaction, and child-boundary RED checks before implementing those paths.

kayg commented 2026-11-01 17:39:37 +00:00

Author

Owner

Copy link

Correction to my previous progress comment: its inline Markdown backticks were interpreted by the shell, so that comment's command output may be incomplete. Current state: renderer WIP commit 2054659a1 is pushed to proxy-traefik-281. The corrected GREEN run is PID 2644471, launched after cd to /home/claude/worktrees/J62-build and using the existing single target /home/claude/worktrees/J62/target. Log: /home/claude/.cache/codex-scratch/j62/config-router-green.log. At 18:37 UTC, build-slot showed four active jobs, J62 at queue position 4, and 130G free. No test result is claimed. Next: observe the crate GREEN, then run the committed registry, redaction, and child-boundary RED checks before implementing those paths.

Correction to my previous progress comment: its inline Markdown backticks were interpreted by the shell, so that comment's command output may be incomplete. Current state: renderer WIP commit 2054659a1 is pushed to proxy-traefik-281. The corrected GREEN run is PID 2644471, launched after cd to /home/claude/worktrees/J62-build and using the existing single target /home/claude/worktrees/J62/target. Log: /home/claude/.cache/codex-scratch/j62/config-router-green.log. At 18:37 UTC, build-slot showed four active jobs, J62 at queue position 4, and 130G free. No test result is claimed. Next: observe the crate GREEN, then run the committed registry, redaction, and child-boundary RED checks before implementing those paths.

kayg commented 2026-11-01 17:50:07 +00:00

Author

Owner

Copy link

J62 finding from the corrected compute01 run: the proxy renderer tests passed, but the crate suite returned 38 passed / 4 failed. Three downloader failures share one cause: ComponentDownloader::install receives release version 0.1.0, while fetch_metadata joins proxy-traefik-linux-amd64.json directly to the mirror root. The release fixture and signed bundle place metadata at <mirror>/<release_version>/...; the transport therefore returns NotFound before signature verification, and the untrusted-manifest test never reaches its expected PermissionDenied. Evidence is in /home/claude/.cache/codex-scratch/j62/config-router-green.log; the tests are download_tests::{mirror_archive_is_streamed_after_the_product_signature_is_verified,upstream_archive_fallback_uses_the_signed_manifest_and_verified_digest,an_untrusted_manifest_never_triggers_an_archive_request}. I will keep archive URLs rooted at the signed mirror path and scope only metadata fetches under the validated release version. The fourth failure is the expected generic credential-redaction RED; next is to implement the sanitizer and then observe all downloader, redaction, renderer, and new private-ping tests GREEN.

J62 finding from the corrected compute01 run: the proxy renderer tests passed, but the crate suite returned 38 passed / 4 failed. Three downloader failures share one cause: `ComponentDownloader::install` receives release version `0.1.0`, while `fetch_metadata` joins `proxy-traefik-linux-amd64.json` directly to the mirror root. The release fixture and signed bundle place metadata at `<mirror>/<release_version>/...`; the transport therefore returns `NotFound` before signature verification, and the untrusted-manifest test never reaches its expected `PermissionDenied`. Evidence is in `/home/claude/.cache/codex-scratch/j62/config-router-green.log`; the tests are `download_tests::{mirror_archive_is_streamed_after_the_product_signature_is_verified,upstream_archive_fallback_uses_the_signed_manifest_and_verified_digest,an_untrusted_manifest_never_triggers_an_archive_request}`. I will keep archive URLs rooted at the signed mirror path and scope only metadata fetches under the validated release version. The fourth failure is the expected generic credential-redaction RED; next is to implement the sanitizer and then observe all downloader, redaction, renderer, and new private-ping tests GREEN.

kayg commented 2026-11-01 17:51:22 +00:00

Author

Owner

Copy link

Progress checkpoint: pushed test-first commit cd6173c63 with loopback-only private ping coverage. Compute01 is running none of these tests yet; detached RED job PID 2886438 waits in build-slot at queue position 4. It will run the proxy agent registry tests and child process boundary test against /home/claude/worktrees/J62-build, using the single target /home/claude/worktrees/J62/target. Master log: /home/claude/.cache/codex-scratch/j62/proxy-red-tests.log; per-test logs: agent-registry-red.log and proxy-process-red.log. At 18:50 UTC there were four active jobs and 116G free. Next: observe these RED results, implement the child boundary, health probe and log sanitizer, then run the proxy suite GREEN.

Progress checkpoint: pushed test-first commit cd6173c63 with loopback-only private ping coverage. Compute01 is running none of these tests yet; detached RED job PID 2886438 waits in build-slot at queue position 4. It will run the proxy agent registry tests and child process boundary test against /home/claude/worktrees/J62-build, using the single target /home/claude/worktrees/J62/target. Master log: /home/claude/.cache/codex-scratch/j62/proxy-red-tests.log; per-test logs: agent-registry-red.log and proxy-process-red.log. At 18:50 UTC there were four active jobs and 116G free. Next: observe these RED results, implement the child boundary, health probe and log sanitizer, then run the proxy suite GREEN.

kayg commented 2026-11-01 18:05:34 +00:00

Author

Owner

Copy link

J62 progress on proxy-traefik-281, base 37bbda4555. The prior crate run confirmed the mirror metadata URL must include the validated release-version directory; the exact failures and request paths are in /home/claude/.cache/codex-scratch/j62/config-router-green.log. I added release-version/path-traversal regression tests locally and am continuing the downloader fix. The current registry/child RED run, PID 2886438, remains queued at position 4; compute01 reported 66 GB free. Its checkout and target remain untouched until it completes. No result is claimed for this queued run. Next: observe its RED output, finish the downloader and private child boundary, then run the required focused suites and gates.

J62 progress on proxy-traefik-281, base 37bbda45554f6e603a6856fc23319e17d8a1b14f0. The prior crate run confirmed the mirror metadata URL must include the validated release-version directory; the exact failures and request paths are in /home/claude/.cache/codex-scratch/j62/config-router-green.log. I added release-version/path-traversal regression tests locally and am continuing the downloader fix. The current registry/child RED run, PID 2886438, remains queued at position 4; compute01 reported 66 GB free. Its checkout and target remain untouched until it completes. No result is claimed for this queued run. Next: observe its RED output, finish the downloader and private child boundary, then run the required focused suites and gates.

kayg commented 2026-11-01 18:20:34 +00:00

Author

Owner

Copy link

J62 crash-safe checkpoint: pushed WIP commit a7ce898bc on proxy-traefik-281. It scopes signed metadata requests under the release-version directory, validates stable and edge versions against the release publisher grammar before URL construction, and adds path-format tests. The existing mirror/fallback tests had already shown the metadata URL RED in the previous compute01 run; this new commit has not been built yet. The queued registry/child RED run PID 2886438 remains at position 2 against the unchanged VM checkout at cd6173c63. I will let it finish before updating that checkout or submitting another run. Next: observe registry/child RED, validate this commit with focused VM tests, then implement the process boundary and registry wiring.

J62 crash-safe checkpoint: pushed WIP commit a7ce898bc on proxy-traefik-281. It scopes signed metadata requests under the release-version directory, validates stable and edge versions against the release publisher grammar before URL construction, and adds path-format tests. The existing mirror/fallback tests had already shown the metadata URL RED in the previous compute01 run; this new commit has not been built yet. The queued registry/child RED run PID 2886438 remains at position 2 against the unchanged VM checkout at cd6173c63. I will let it finish before updating that checkout or submitting another run. Next: observe registry/child RED, validate this commit with focused VM tests, then implement the process boundary and registry wiring.

kayg commented 2026-11-01 18:22:53 +00:00

Author

Owner

Copy link

J62 checkpoint: pushed WIP e3209c881 on proxy-traefik-281. It adds credential-shaped log redaction for Authorization, Cookie, and sensitive query fields; the existing focused run had reproduced a leak when no known token values were supplied. This sanitizer commit still needs its VM GREEN run. The registry/child RED run PID 2886438 remains queued at position 2 against checkout cd6173c63, so that checkout remains unchanged. The one target remains /home/claude/worktrees/J62/target. Next: observe the queued RED, move the VM checkout to the pushed head, run the focused download and sanitizer suites, and implement the child/registry boundary from its failing output.

J62 checkpoint: pushed WIP e3209c881 on proxy-traefik-281. It adds credential-shaped log redaction for Authorization, Cookie, and sensitive query fields; the existing focused run had reproduced a leak when no known token values were supplied. This sanitizer commit still needs its VM GREEN run. The registry/child RED run PID 2886438 remains queued at position 2 against checkout cd6173c63, so that checkout remains unchanged. The one target remains /home/claude/worktrees/J62/target. Next: observe the queued RED, move the VM checkout to the pushed head, run the focused download and sanitizer suites, and implement the child/registry boundary from its failing output.

kayg commented 2026-11-01 18:31:17 +00:00

Author

Owner

Copy link

J62 finding — failing-first run on compute01

The queued RED script finished against cd6173c6367aad437107892b15c18426c26d786e (logs: ~/.cache/codex-scratch/j62/agent-registry-red.log and proxy-process-red.log).

Evidence:

• proxy_registry_tests: 1 passed, 2 failed. proxy.publish must be present in the compiled node registry failed; publication without accepted mesh context also failed its expected Rejected result.
• Child boundary integration did not compile: unresolved import infraternal_plugin_proxy::supervisor::TraefikChild.

I’m implementing the registry entry, private loopback ping, and owned child boundary on proxy-traefik-281 (current pushed head e3209c881). Next: run the focused green tests on a VM, then finish namespace/Pebble coverage.

J62 finding — failing-first run on compute01 The queued RED script finished against `cd6173c6367aad437107892b15c18426c26d786e` (logs: `~/.cache/codex-scratch/j62/agent-registry-red.log` and `proxy-process-red.log`). Evidence: - `proxy_registry_tests`: 1 passed, 2 failed. `proxy.publish must be present in the compiled node registry` failed; publication without accepted mesh context also failed its expected `Rejected` result. - Child boundary integration did not compile: `unresolved import infraternal_plugin_proxy::supervisor::TraefikChild`. I’m implementing the registry entry, private loopback ping, and owned child boundary on `proxy-traefik-281` (current pushed head `e3209c881`). Next: run the focused green tests on a VM, then finish namespace/Pebble coverage.

kayg commented 2026-11-01 18:50:37 +00:00

Author

Owner

Copy link

J62 resumed from the pushed WIP branch proxy-traefik-281. I rebased its 13 J62 commits onto current origin/dev 65d9861914e8cd4a734d533d64ed1f4ea5c34119; local head is 4dc7d2f81b06dce95ffdcd0a1d05132e7510274e with the in-progress supervisor and registry changes still uncommitted. The rebased head has not been pushed yet.

I resubmitted the proxy crate suite detached through compute01 build-slot. It is queued at position 3; log: /home/claude/.cache/codex-scratch/j62/proxy-resume-test.log. Existing single target: /home/claude/worktrees/J62/target.

Finding: the accepted Mesh snapshot has directional peer policies and addresses, but the current origin/dev agent/config boundary does not provide the public-host-derived DNS suffix to proxy.publish. The action remains fail-closed while I trace a trusted integration path; no caller-supplied grant or suffix will authorize a route.

Plan (about 3 hours of implementation and verification, excluding slot wait): verify and finish the child/lifecycle WIP, finish publication/config integration that can be based on the accepted Mesh state, add namespace + Pebble staging coverage, run focused and fast gates on VMs, report the x86 static-size delta, then push and clean the sole J62 target.

J62 resumed from the pushed WIP branch `proxy-traefik-281`. I rebased its 13 J62 commits onto current `origin/dev` `65d9861914e8cd4a734d533d64ed1f4ea5c34119`; local head is `4dc7d2f81b06dce95ffdcd0a1d05132e7510274e` with the in-progress supervisor and registry changes still uncommitted. The rebased head has not been pushed yet. I resubmitted the proxy crate suite detached through compute01 build-slot. It is queued at position 3; log: `/home/claude/.cache/codex-scratch/j62/proxy-resume-test.log`. Existing single target: `/home/claude/worktrees/J62/target`. Finding: the accepted Mesh snapshot has directional peer policies and addresses, but the current `origin/dev` agent/config boundary does not provide the public-host-derived DNS suffix to `proxy.publish`. The action remains fail-closed while I trace a trusted integration path; no caller-supplied grant or suffix will authorize a route. Plan (about 3 hours of implementation and verification, excluding slot wait): verify and finish the child/lifecycle WIP, finish publication/config integration that can be based on the accepted Mesh state, add namespace + Pebble staging coverage, run focused and fast gates on VMs, report the x86 static-size delta, then push and clean the sole J62 target.

kayg commented 2026-11-01 19:36:45 +00:00

Author

Owner

Copy link

J62 crash-safe WIP checkpoint pushed on proxy-traefik-281: 5a096f41595f2213306fa4d48729eace0e8ab48b, based on origin/dev 65d9861914e8cd4a734d533d64ed1f4ea5c34119.

This checkpoint captures the single-child Traefik supervisor, atomic route snapshots, private health ping, provider *_FILE path handling, log redaction, node registry wiring, and the namespace/Pebble test harness. The prior compute01 log is a compile failure from an older source snapshot (raw libc calls rejected by forbid(unsafe_code)); the current source replaces those calls with safe nix locking and process-group setup. No current J62 test is running and no green result is claimed.

Next: refresh the VM checkout from this pushed head, run the focused proxy tests, finish trusted mesh name/suffix/grant propagation and actual publish behavior, then complete the acquisition/lifecycle/staging tests, VM gates and size report. I will remove the sole J62 target after final verification.

J62 crash-safe WIP checkpoint pushed on `proxy-traefik-281`: `5a096f41595f2213306fa4d48729eace0e8ab48b`, based on `origin/dev` `65d9861914e8cd4a734d533d64ed1f4ea5c34119`. This checkpoint captures the single-child Traefik supervisor, atomic route snapshots, private health ping, provider `*_FILE` path handling, log redaction, node registry wiring, and the namespace/Pebble test harness. The prior compute01 log is a compile failure from an older source snapshot (raw libc calls rejected by `forbid(unsafe_code)`); the current source replaces those calls with safe nix locking and process-group setup. No current J62 test is running and no green result is claimed. Next: refresh the VM checkout from this pushed head, run the focused proxy tests, finish trusted mesh name/suffix/grant propagation and actual `publish` behavior, then complete the acquisition/lifecycle/staging tests, VM gates and size report. I will remove the sole J62 target after final verification.

kayg commented 2026-11-01 20:07:32 +00:00

Author

Owner

Copy link

J62 resumed on proxy-traefik-281 at 5a096f41595f2213306fa4d48729eace0e8ab48b, based on origin/dev 65d9861914e8cd4a734d533d64ed1f4ea5c34119.

The compute01 build-slot item PID 4175181 remains queued and untouched; it runs compiled_mesh_snapshot_carries_stable_proxy_name_and_dns_suffix against the existing VM source snapshot. Its log is /home/claude/.cache/codex-scratch/j62/mesh-red.log.

Source audit finding: ProxyAgentPlugin::call currently returns Rejected for every non-list publication, so the already-built supervisor/config paths are not yet reachable through publish <port>. The accepted Mesh config carries directional peer policies but this branch has no accepted machine-name/DNS-suffix context. The queued integration test is expected to expose the missing snapshot fields. Publication must derive source addresses from the accepted inbound TCP grant for that port; request JSON cannot supply grants or suffixes.

Plan: observe that VM RED, then add the narrow accepted-Mesh context and connect publish to the existing supervisor; finish the namespace/Pebble and installer/lifecycle coverage; run focused tests and scripts/verify fast on the VMs; report the x86 static-size delta; remove the one J62 target; push and post the exact gate output. Occupied 80/443 falls back to 10080/10443. No nftables redirect is safe when another process may bind the mesh address explicitly. Estimate remains about 3 hours of code and verification, excluding slot wait.

J62 resumed on `proxy-traefik-281` at `5a096f41595f2213306fa4d48729eace0e8ab48b`, based on `origin/dev` `65d9861914e8cd4a734d533d64ed1f4ea5c34119`. The compute01 build-slot item PID 4175181 remains queued and untouched; it runs `compiled_mesh_snapshot_carries_stable_proxy_name_and_dns_suffix` against the existing VM source snapshot. Its log is `/home/claude/.cache/codex-scratch/j62/mesh-red.log`. Source audit finding: `ProxyAgentPlugin::call` currently returns `Rejected` for every non-list publication, so the already-built supervisor/config paths are not yet reachable through `publish <port>`. The accepted Mesh config carries directional peer policies but this branch has no accepted machine-name/DNS-suffix context. The queued integration test is expected to expose the missing snapshot fields. Publication must derive source addresses from the accepted inbound TCP grant for that port; request JSON cannot supply grants or suffixes. Plan: observe that VM RED, then add the narrow accepted-Mesh context and connect `publish` to the existing supervisor; finish the namespace/Pebble and installer/lifecycle coverage; run focused tests and `scripts/verify fast` on the VMs; report the x86 static-size delta; remove the one J62 target; push and post the exact gate output. Occupied 80/443 falls back to 10080/10443. No nftables redirect is safe when another process may bind the mesh address explicitly. Estimate remains about 3 hours of code and verification, excluding slot wait.

kayg commented 2026-11-01 20:19:44 +00:00

Author

Owner

Copy link

Finding: the first compute01 queue run completed with 0 passed; 5 filtered out, so it did not exercise the intended regression. The VM source copy was stale: crates/plugins/mesh/tests/configuration.rs was SHA-256 47384abc99841ea8452de8d4dd29ab402e3122bbe768fdc1c5d7da872a6e3c32 versus local 31e10c45ca65d74c55982eb093b5c294e42b17b74a9f3ebb6d624f915d5acd37. I synchronized the branch sources (kept the VM target) and resubmitted the focused test; PID 157710, log /home/claude/.cache/codex-scratch/j62/mesh-red.log.

Finding: the first compute01 queue run completed with `0 passed; 5 filtered out`, so it did not exercise the intended regression. The VM source copy was stale: `crates/plugins/mesh/tests/configuration.rs` was SHA-256 `47384abc99841ea8452de8d4dd29ab402e3122bbe768fdc1c5d7da872a6e3c32` versus local `31e10c45ca65d74c55982eb093b5c294e42b17b74a9f3ebb6d624f915d5acd37`. I synchronized the branch sources (kept the VM target) and resubmitted the focused test; PID 157710, log `/home/claude/.cache/codex-scratch/j62/mesh-red.log`.

kayg commented 2026-11-01 20:25:36 +00:00

Author

Owner

Copy link

Orchestrator choice for slice 1, pending owner rename if desired: use a separate infraternal unpublish <port|name> command as the inverse of publish. It removes the route, prints what stopped being reachable, is idempotent, and a missing target returns D303 missing (exit 4) with a one-line message. --json uses the shared CLI envelope. I will record this in the proxy module doc and test both selector forms and the missing result.

Orchestrator choice for slice 1, pending owner rename if desired: use a separate `infraternal unpublish <port|name>` command as the inverse of `publish`. It removes the route, prints what stopped being reachable, is idempotent, and a missing target returns D303 `missing` (exit 4) with a one-line message. `--json` uses the shared CLI envelope. I will record this in the proxy module doc and test both selector forms and the missing result.

kayg commented 2026-11-01 20:27:40 +00:00

Author

Owner

Copy link

Follow-up finding: the synchronized source SHA matched, but the second run also reused the old Cargo test binary because tar preserved the source timestamp; it again reported 0 passed; 5 filtered out. I touched crates/plugins/mesh/tests/configuration.rs on compute01 to invalidate the stale harness and resubmitted. Current queued PID 228639, same log /home/claude/.cache/codex-scratch/j62/mesh-red.log.

Follow-up finding: the synchronized source SHA matched, but the second run also reused the old Cargo test binary because tar preserved the source timestamp; it again reported `0 passed; 5 filtered out`. I touched `crates/plugins/mesh/tests/configuration.rs` on compute01 to invalidate the stale harness and resubmitted. Current queued PID 228639, same log `/home/claude/.cache/codex-scratch/j62/mesh-red.log`.

kayg commented 2026-11-01 20:36:11 +00:00

Author

Owner

Copy link

Orchestrator CLI choice for #281: use a separate infraternal unpublish <port|name> inverse to publish. It reports what stopped being reachable; a missing target is idempotent D303 missing (exit 4) with a one-line message, and --json uses the shared envelope. The module doc and CLI tests record this choice pending any owner rename.

Orchestrator CLI choice for #281: use a separate `infraternal unpublish <port|name>` inverse to `publish`. It reports what stopped being reachable; a missing target is idempotent D303 `missing` (exit 4) with a one-line message, and `--json` uses the shared envelope. The module doc and CLI tests record this choice pending any owner rename.

kayg commented 2026-11-01 20:46:45 +00:00

Author

Owner

Copy link

J62 continuation: compute01 test PID 228639 remains queued at position 1; /home/claude/.cache/codex-scratch/j62/mesh-red.log still contains only the queue notice. I preserved that VM source snapshot and added one more local regression requiring name-only Mesh config changes to reach the proxy. No production code changed in this continuation. Next: observe the intended RED, then implement accepted identity propagation and finish the publish/unpublish action wiring.

J62 continuation: compute01 test PID 228639 remains queued at position 1; `/home/claude/.cache/codex-scratch/j62/mesh-red.log` still contains only the queue notice. I preserved that VM source snapshot and added one more local regression requiring name-only Mesh config changes to reach the proxy. No production code changed in this continuation. Next: observe the intended RED, then implement accepted identity propagation and finish the publish/unpublish action wiring.

kayg commented 2026-11-01 20:59:56 +00:00

Author

Owner

Copy link

J62 resumed on proxy-traefik-281. I rebased the pushed WIP onto current origin/dev (0067e1255407f0f3ed0f1d6c5e5caf51667e00a0) and pushed the rebased checkpoint at 42a919e06de62611d3a84da3cb3f22a346fb49ff.

The accepted-Mesh RED completed on compute01 as expected: compiled_mesh_snapshot_carries_stable_proxy_name_and_dns_suffix failed because the serialized snapshot has null for machine_name instead of host-2. I am adding identity to the existing chunked revision path and wiring the proxy only from an accepted complete config. This keeps the same inbound TCP grants as Mesh and prevents caller JSON from asserting grants or suffixes.

Next: propagate the server-origin DNS suffix, connect the accepted context to publish, finish unpublish <port|name> and its D303 missing behavior, then run focused proxy/Mesh/CLI and namespace Pebble verification on the VMs. The current estimate is about three hours of implementation and VM verification, excluding build-slot wait. I will keep one J62 target and report the report-only static size delta. The occupied-port investigation still rejects an nftables redirect because it would capture another service explicitly bound to the mesh address.

J62 resumed on `proxy-traefik-281`. I rebased the pushed WIP onto current `origin/dev` (`0067e1255407f0f3ed0f1d6c5e5caf51667e00a0`) and pushed the rebased checkpoint at `42a919e06de62611d3a84da3cb3f22a346fb49ff`. The accepted-Mesh RED completed on compute01 as expected: `compiled_mesh_snapshot_carries_stable_proxy_name_and_dns_suffix` failed because the serialized snapshot has `null` for `machine_name` instead of `host-2`. I am adding identity to the existing chunked revision path and wiring the proxy only from an accepted complete config. This keeps the same inbound TCP grants as Mesh and prevents caller JSON from asserting grants or suffixes. Next: propagate the server-origin DNS suffix, connect the accepted context to `publish`, finish `unpublish <port|name>` and its D303 `missing` behavior, then run focused proxy/Mesh/CLI and namespace Pebble verification on the VMs. The current estimate is about three hours of implementation and VM verification, excluding build-slot wait. I will keep one J62 target and report the report-only static size delta. The occupied-port investigation still rejects an nftables redirect because it would capture another service explicitly bound to the mesh address.

kayg commented 2026-11-01 21:40:36 +00:00

Author

Owner

Copy link

J62 continuation on proxy-traefik-281. The latest focused compute01 run confirmed the new unpublish action and command-table entry, with 50 tests passing and one expected-message assertion still failing; its log is /home/claude/.cache/codex-scratch/j62/proxy-unpublish-green.log. The remaining mismatch was the missing-port line omitting the word port.

I have corrected the local D303 response and added action-level cases for unpublish by port/name and accepted-grant revocation. Current work wires Mesh's retained accepted config into the proxy controller, then connects on-demand verified acquisition and supervision to publish/unpublish. No UI or real ACME certificates are in this slice.

Next: sync and run the focused proxy/agent/Mesh tests on a VM, run the namespace Pebble staging case, fix any failures, run scripts/verify fast, report the D243 size delta, push the branch, and remove the sole target. Estimate: about 90 minutes of implementation and verification, plus slot wait.

J62 continuation on `proxy-traefik-281`. The latest focused compute01 run confirmed the new unpublish action and command-table entry, with 50 tests passing and one expected-message assertion still failing; its log is `/home/claude/.cache/codex-scratch/j62/proxy-unpublish-green.log`. The remaining mismatch was the missing-port line omitting the word `port`. I have corrected the local D303 response and added action-level cases for unpublish by port/name and accepted-grant revocation. Current work wires Mesh's retained accepted config into the proxy controller, then connects on-demand verified acquisition and supervision to publish/unpublish. No UI or real ACME certificates are in this slice. Next: sync and run the focused proxy/agent/Mesh tests on a VM, run the namespace Pebble staging case, fix any failures, run `scripts/verify fast`, report the D243 size delta, push the branch, and remove the sole target. Estimate: about 90 minutes of implementation and verification, plus slot wait.

kayg commented 2026-11-01 21:41:41 +00:00

Author

Owner

Copy link

J62 port-conflict investigation for #281: do not add an nftables redirect in slice 1. The current rootful probe tries to bind the node's exact Mesh address on 80 and 443 and uses 10080/10443 when either conventional listener is occupied. A redirect limited to the Mesh interface and Mesh destination address still runs before local socket delivery. It would therefore capture Mesh-IP:80/443 packets even when another server explicitly owns that address and port, diverting that server's Mesh traffic into Traefik. The redirect cannot establish socket ownership or distinguish the other server's routes. Without explicit coordination or an exclusive reserved listener, the behavior is not safe. Keep direct address binding and high-port fallback; no host firewall rules are changed.

J62 port-conflict investigation for #281: do not add an nftables redirect in slice 1. The current rootful probe tries to bind the node's exact Mesh address on 80 and 443 and uses 10080/10443 when either conventional listener is occupied. A redirect limited to the Mesh interface and Mesh destination address still runs before local socket delivery. It would therefore capture Mesh-IP:80/443 packets even when another server explicitly owns that address and port, diverting that server's Mesh traffic into Traefik. The redirect cannot establish socket ownership or distinguish the other server's routes. Without explicit coordination or an exclusive reserved listener, the behavior is not safe. Keep direct address binding and high-port fallback; no host firewall rules are changed.

kayg commented 2026-11-01 22:00:05 +00:00

Author

Owner

Copy link

Orchestrator choice for the CLI: use a separate infraternal unpublish <port|name> command as the inverse of publish. It accepts the local port or exact published name, reports what stopped being reachable, and removing the last publication starts the 10-second drain. Missing targets are idempotent and return D303 missing (exit 4) with a one-line message; --json follows the publish envelope. The module doc and command table record this as the orchestrator’s choice, open to kayg renaming it.

Orchestrator choice for the CLI: use a separate `infraternal unpublish <port|name>` command as the inverse of `publish`. It accepts the local port or exact published name, reports what stopped being reachable, and removing the last publication starts the 10-second drain. Missing targets are idempotent and return D303 `missing` (exit 4) with a one-line message; `--json` follows the `publish` envelope. The module doc and command table record this as the orchestrator’s choice, open to kayg renaming it.

kayg commented 2026-11-01 22:02:47 +00:00

Author

Owner

Copy link

J62 checkpoint pushed: cf0bef881 on proxy-traefik-281, based on current origin/dev 6093cb8a. The last focused proxy run on compute01 completed with 50 passing tests and one expected message assertion failure; the port-label fix is now in the commit. I am rerunning the crate suite from an isolated VM worktree, then will run the namespace Pebble case and scripts/verify fast. The occupied-port analysis is recorded in comment 73730; no nftables redirect is added.

J62 checkpoint pushed: `cf0bef881` on `proxy-traefik-281`, based on current `origin/dev` `6093cb8a`. The last focused proxy run on compute01 completed with 50 passing tests and one expected message assertion failure; the port-label fix is now in the commit. I am rerunning the crate suite from an isolated VM worktree, then will run the namespace Pebble case and `scripts/verify fast`. The occupied-port analysis is recorded in comment 73730; no nftables redirect is added.

kayg commented 2026-11-01 22:04:35 +00:00

Author

Owner

Copy link

The focused run on cf0bef881 completed with 54 passing tests and 2 failures. Both expose one formatting gap: the expected rootless :10443 listener port is absent from publish URLs and the unpublish reachability message. The unpublish port-label assertion now passes. I am tracing the selected listener ports into the controller and will rerun the failing tests after the fix.

The focused run on `cf0bef881` completed with 54 passing tests and 2 failures. Both expose one formatting gap: the expected rootless `:10443` listener port is absent from `publish` URLs and the `unpublish` reachability message. The `unpublish` port-label assertion now passes. I am tracing the selected listener ports into the controller and will rerun the failing tests after the fix.

kayg commented 2026-11-01 22:14:36 +00:00

Author

Owner

Copy link

The listener-port propagation change compiled far enough to identify one type mismatch: the scope-restart helper still returned Result<()> on its ineligible branch after changing to Result<ListenerPorts>. No tests ran on that attempt. I am correcting the branch and removing the associated unused imports/state assignments before rerunning.

The listener-port propagation change compiled far enough to identify one type mismatch: the scope-restart helper still returned `Result<()>` on its ineligible branch after changing to `Result<ListenerPorts>`. No tests ran on that attempt. I am correcting the branch and removing the associated unused imports/state assignments before rerunning.

kayg commented 2026-11-01 22:23:11 +00:00

Author

Owner

Copy link

J62 resumed on branch proxy-traefik-281 at cf0bef881 (merge base origin/dev 6093cb8a). Existing WIP adds signed Traefik acquisition, owner-only DNS token custody, mesh-grant route enforcement, lifecycle supervision, and CLI publish / unpublish actions. I confirmed the current module doc and command table record the orchestrator's separate unpublish <port|name> choice, pending owner rename.

Plan: (1) complete positional CLI and D303 missing/JSON regression coverage; (2) resolve the pending proxy crate test on compute01 and run the required fast gate on the emptier VM; (3) report-only size deltas, inspect the nftables finding, clean targets, commit/push atomic changes, and post the final receipts. Estimated remaining work: 60 minutes, subject to VM gate duration. The previous J62 test process is being checked before resubmission; I will keep its log path in the next progress note.

J62 resumed on branch `proxy-traefik-281` at `cf0bef881` (merge base `origin/dev` `6093cb8a`). Existing WIP adds signed Traefik acquisition, owner-only DNS token custody, mesh-grant route enforcement, lifecycle supervision, and CLI `publish` / `unpublish` actions. I confirmed the current module doc and command table record the orchestrator's separate `unpublish <port|name>` choice, pending owner rename. Plan: (1) complete positional CLI and D303 missing/JSON regression coverage; (2) resolve the pending proxy crate test on compute01 and run the required fast gate on the emptier VM; (3) report-only size deltas, inspect the nftables finding, clean targets, commit/push atomic changes, and post the final receipts. Estimated remaining work: 60 minutes, subject to VM gate duration. The previous J62 test process is being checked before resubmission; I will keep its log path in the next progress note.

kayg commented 2026-11-01 22:27:30 +00:00

Author

Owner

Copy link

J62 update: the detached compute01 proxy crate suite finished on the four-file listener-port snapshot with TEST_EXIT=0; receipt: test result: ok. 57 passed; 0 failed; 0 ignored; finished in 0.09s (doc tests also green; namespace/Pebble remains a separate ignored test). I committed and pushed e3fd798f6afd898aa4997cfc6a043ec6a2d89a13, which carries the selected HTTP/HTTPS listener ports into publication and drain URLs. Next: update the compute01 checkout to this exact head, run the full fast gate and the staging Pebble namespace test, capture size rows, then clean the J62 target and post final receipts.

J62 update: the detached compute01 proxy crate suite finished on the four-file listener-port snapshot with `TEST_EXIT=0`; receipt: `test result: ok. 57 passed; 0 failed; 0 ignored; finished in 0.09s` (doc tests also green; namespace/Pebble remains a separate ignored test). I committed and pushed `e3fd798f6afd898aa4997cfc6a043ec6a2d89a13`, which carries the selected HTTP/HTTPS listener ports into publication and drain URLs. Next: update the compute01 checkout to this exact head, run the full fast gate and the staging Pebble namespace test, capture size rows, then clean the J62 target and post final receipts.

kayg commented 2026-11-01 22:30:32 +00:00

Author

Owner

Copy link

J62 custody audit finding: FileDnsTokenCustody::delete is implemented and unit tested, but no production caller invokes it. DnsProviderProfile is loaded from the agent environment and the supervisor only calls get; removing INFRATERAL_DNS_PROVIDER therefore leaves its owner-only token file behind. I will add a failing regression and cleanup of inactive provider directories when the agent loads its current provider profile. I also found the TraefikChild::spawn doc says credential bytes are never read there, while the code reads them into zeroizing memory only to redact logs; the child still receives only _FILE paths. I’ll correct that comment for accuracy.

J62 custody audit finding: `FileDnsTokenCustody::delete` is implemented and unit tested, but no production caller invokes it. `DnsProviderProfile` is loaded from the agent environment and the supervisor only calls `get`; removing `INFRATERAL_DNS_PROVIDER` therefore leaves its owner-only token file behind. I will add a failing regression and cleanup of inactive provider directories when the agent loads its current provider profile. I also found the `TraefikChild::spawn` doc says credential bytes are never read there, while the code reads them into zeroizing memory only to redact logs; the child still receives only `_FILE` paths. I’ll correct that comment for accuracy.

kayg commented 2026-11-01 22:34:33 +00:00

Author

Owner

Copy link

J62 gate finding: the exact-head scripts/verify fast log reports error[E0433]: failed to resolve: use of unresolved module or unlinked crate tracing at crates/infraternal-agent/src/plugins.rs:91, where proxy child output is connected to the existing tracing pipeline. The agent crate did not declare the workspace tracing dependency. I added the existing workspace dependency locally; I will validate it on the VM after the current exact-head gate exits, then rerun the final gate on the fixed head.

J62 gate finding: the exact-head `scripts/verify fast` log reports `error[E0433]: failed to resolve: use of unresolved module or unlinked crate tracing` at `crates/infraternal-agent/src/plugins.rs:91`, where proxy child output is connected to the existing tracing pipeline. The agent crate did not declare the workspace tracing dependency. I added the existing workspace dependency locally; I will validate it on the VM after the current exact-head gate exits, then rerun the final gate on the fixed head.

kayg commented 2026-11-01 22:38:54 +00:00

Author

Owner

Copy link

J62 fast-gate finding: the current Clippy stage reports 25 errors in the new proxy crate and 3 in the Mesh snapshot integration. The proxy findings are mostly naming/ownership lints plus 64 KiB stack buffers and two overlong functions (render_dynamic_config, extract_executable, and run_supervisor); the Mesh findings are two clone assignments and one complex callback type. The log is /home/claude/.cache/codex-scratch/j62/verify-fast.log. I’m tracing each to the branch diff and will fix the code rather than add lint allowances.

J62 fast-gate finding: the current Clippy stage reports 25 errors in the new proxy crate and 3 in the Mesh snapshot integration. The proxy findings are mostly naming/ownership lints plus 64 KiB stack buffers and two overlong functions (`render_dynamic_config`, `extract_executable`, and `run_supervisor`); the Mesh findings are two clone assignments and one complex callback type. The log is `/home/claude/.cache/codex-scratch/j62/verify-fast.log`. I’m tracing each to the branch diff and will fix the code rather than add lint allowances.

kayg commented 2026-11-01 22:46:26 +00:00

Author

Owner

Copy link

J62 resumed on proxy-traefik-281 at e3fd798f6a, based on current origin/dev 6093cb8a83. I reread BRIEF-COMMON.md, the #281 plan and selected owner answers, D316/D314/D303, #261 R2/J20/R3, #266 DNS naming, and #211 edge-update context. The current local changes add the tracing dependency used by proxy child logs and a failing DNS-token cleanup regression. Compute01 still has the older exact-head scripts/verify fast run active in its J62 worktree; I have not submitted a competing build. Next: finish provider-removal cleanup and resolve the remaining installer test failures, then run the proxy suite, namespace/Pebble case, and exact-head fast gate through build-slot. The orchestrator's separate unpublish command is already in the module doc and recorded in #281 comments 73591 and 73753.

J62 resumed on proxy-traefik-281 at e3fd798f6afd898aa4997cfc6a043ec6a2d89a13, based on current origin/dev 6093cb8a83b8. I reread BRIEF-COMMON.md, the #281 plan and selected owner answers, D316/D314/D303, #261 R2/J20/R3, #266 DNS naming, and #211 edge-update context. The current local changes add the tracing dependency used by proxy child logs and a failing DNS-token cleanup regression. Compute01 still has the older exact-head scripts/verify fast run active in its J62 worktree; I have not submitted a competing build. Next: finish provider-removal cleanup and resolve the remaining installer test failures, then run the proxy suite, namespace/Pebble case, and exact-head fast gate through build-slot. The orchestrator's separate unpublish command is already in the module doc and recorded in #281 comments 73591 and 73753.

kayg commented 2026-11-01 22:55:09 +00:00

Author

Owner

Copy link

J62 resumed on proxy-traefik-281 at e3fd798f6afd898aa4997cfc6a043ec6a2d89a13, based on current origin/dev 6093cb8a83b810a119269d6045e8720a3e3922b6.

The prior J62 compute01 test is no longer present in the build-slot queue. Current status shows all slots occupied on both compute01 and dev-02, with unrelated J72 and J40 jobs queued. I have not started a competing build. The local worktree has follow-up changes for the tracing dependency, provider-token cleanup, and compile/lint fixes.

Plan: (1) finish and split the current WIP into atomic test-first commits; (2) once a slot opens, run the proxy crate suite and namespace/Pebble test on a VM; (3) run scripts/verify fast on the emptier VM and report the D243 static-size delta; (4) push the branch, post exact receipts, and remove the single J62 target. Estimate: about 2 hours of active work, excluding build-slot wait. The unpublish <port|name> choice is already recorded as the orchestrator's choice pending owner rename; the nftables finding remains that redirection is unsafe when another server explicitly binds the mesh address.

J62 resumed on `proxy-traefik-281` at `e3fd798f6afd898aa4997cfc6a043ec6a2d89a13`, based on current `origin/dev` `6093cb8a83b810a119269d6045e8720a3e3922b6`. The prior J62 compute01 test is no longer present in the build-slot queue. Current status shows all slots occupied on both compute01 and dev-02, with unrelated J72 and J40 jobs queued. I have not started a competing build. The local worktree has follow-up changes for the tracing dependency, provider-token cleanup, and compile/lint fixes. Plan: (1) finish and split the current WIP into atomic test-first commits; (2) once a slot opens, run the proxy crate suite and namespace/Pebble test on a VM; (3) run `scripts/verify fast` on the emptier VM and report the D243 static-size delta; (4) push the branch, post exact receipts, and remove the single J62 target. Estimate: about 2 hours of active work, excluding build-slot wait. The `unpublish <port|name>` choice is already recorded as the orchestrator's choice pending owner rename; the nftables finding remains that redirection is unsafe when another server explicitly binds the mesh address.

kayg commented 2026-11-01 23:29:10 +00:00

Author

Owner

Copy link

J62 resumed on proxy-traefik-281 from pushed head e97be5b5a, whose merge base is origin/dev 6093cb8a83b810a119269d6045e8720a3e3922b6. Current origin/dev has advanced to d1ec515e4 with the #266 mesh-DNS merge; I will integrate that base before final verification.

The stale-token custody regression is now RED on the pre-fix commit: agent_start_removes_custody_files_for_a_removed_dns_provider failed because the removed provider's token file survived startup (0 passed; 1 failed, expected assertion). Log: /home/claude/.cache/codex-scratch/j62/custody-cleanup-red-final.log. Earlier retries did not reach Cargo: one used a missing nix path and the other ran outside the flake worktree.

Next: wire provider cleanup behind DnsTokenCustody, verify configured-provider retention, then run the focused suite, namespace/Pebble case and scripts/verify fast on the least-loaded VM. I will report D243 size deltas and remove the J62 target before handoff.

J62 resumed on `proxy-traefik-281` from pushed head `e97be5b5a`, whose merge base is `origin/dev` `6093cb8a83b810a119269d6045e8720a3e3922b6`. Current `origin/dev` has advanced to `d1ec515e4` with the #266 mesh-DNS merge; I will integrate that base before final verification. The stale-token custody regression is now RED on the pre-fix commit: `agent_start_removes_custody_files_for_a_removed_dns_provider` failed because the removed provider's token file survived startup (`0 passed; 1 failed`, expected assertion). Log: `/home/claude/.cache/codex-scratch/j62/custody-cleanup-red-final.log`. Earlier retries did not reach Cargo: one used a missing nix path and the other ran outside the flake worktree. Next: wire provider cleanup behind `DnsTokenCustody`, verify configured-provider retention, then run the focused suite, namespace/Pebble case and `scripts/verify fast` on the least-loaded VM. I will report D243 size deltas and remove the J62 target before handoff.

kayg commented 2026-11-01 23:31:28 +00:00

Author

Owner

Copy link

Focused custody GREEN follow-up found a lockfile mismatch before tests started: the local WIP adds the existing tracing workspace dependency to the agent's proxy feature, but the current Cargo.lock lacks that agent dependency edge. cargo test --locked stopped with cannot update the lock file; no test ran. I am resolving the lock metadata on compute01 with the same source snapshot, then will rerun with --locked and sync the resulting Cargo.lock back.

Focused custody GREEN follow-up found a lockfile mismatch before tests started: the local WIP adds the existing `tracing` workspace dependency to the agent's `proxy` feature, but the current Cargo.lock lacks that agent dependency edge. `cargo test --locked` stopped with `cannot update the lock file`; no test ran. I am resolving the lock metadata on compute01 with the same source snapshot, then will rerun with `--locked` and sync the resulting Cargo.lock back.

kayg commented 2026-11-01 23:39:03 +00:00

Author

Owner

Copy link

The full proxy crate run on the current WIP reported 58 passed; 1 failed: the configured DNS provider's credential file did not survive ProxyController::for_agent startup. The removed-provider cleanup RED/GREEN passes, so I have not treated the cleanup as complete. I added safe in-child checks for the configured provider profile and token-file path before and after controller startup to isolate the mismatch; the focused diagnostic run is queued through compute01's build-slot.

The full proxy crate run on the current WIP reported `58 passed; 1 failed`: the configured DNS provider's credential file did not survive `ProxyController::for_agent` startup. The removed-provider cleanup RED/GREEN passes, so I have not treated the cleanup as complete. I added safe in-child checks for the configured provider profile and token-file path before and after controller startup to isolate the mismatch; the focused diagnostic run is queued through compute01's build-slot.

kayg commented 2026-11-01 23:47:46 +00:00

Author

Owner

Copy link

J62 resume checkpoint on proxy-traefik-281: pushed head e97be5b5abbccd9d0411228f7acd9f827530fa73; base origin/dev has advanced from merge base 6093cb8a83b810a119269d6045e8720a3e3922b6 to d1ec515e44dabccb640801c4a1ef159b32e004fe.

The custody diagnostic completed on compute01 and failed after ProxyController::for_agent startup: the selected provider token path was missing. Log: /home/claude/.cache/codex-scratch/j62/custody-retain-diagnostic.log. I added local assertions that distinguish the configured provider identity and direct cleanup from controller startup; those assertions still need a VM run. The prior queued J62 test is gone; current queue entries are unrelated.

Next: finish the custody regression, rebase onto current origin/dev for the merged #266 DNS suffix/grant path, run the proxy and Mesh focused tests plus the namespace Pebble staging case through build-slot, then run scripts/verify fast, report both static-size deltas, push and remove the J62 target. Estimated active implementation and verification: about 2 hours, excluding queue waits. The separate unpublish <port|name> choice and its D303 missing/exit-4 response remain recorded in the module doc, command tests, and prior #281 comments.

J62 resume checkpoint on `proxy-traefik-281`: pushed head `e97be5b5abbccd9d0411228f7acd9f827530fa73`; base `origin/dev` has advanced from merge base `6093cb8a83b810a119269d6045e8720a3e3922b6` to `d1ec515e44dabccb640801c4a1ef159b32e004fe`. The custody diagnostic completed on compute01 and failed after `ProxyController::for_agent` startup: the selected provider token path was missing. Log: `/home/claude/.cache/codex-scratch/j62/custody-retain-diagnostic.log`. I added local assertions that distinguish the configured provider identity and direct cleanup from controller startup; those assertions still need a VM run. The prior queued J62 test is gone; current queue entries are unrelated. Next: finish the custody regression, rebase onto current `origin/dev` for the merged #266 DNS suffix/grant path, run the proxy and Mesh focused tests plus the namespace Pebble staging case through `build-slot`, then run `scripts/verify fast`, report both static-size deltas, push and remove the J62 target. Estimated active implementation and verification: about 2 hours, excluding queue waits. The separate `unpublish <port|name>` choice and its D303 missing/exit-4 response remain recorded in the module doc, command tests, and prior #281 comments.

kayg commented 2026-11-01 23:56:51 +00:00

Author

Owner

Copy link

J62 resumed from pushed head fe43a8314 on proxy-traefik-281, rebased onto origin/dev d1ec515e44dabccb640801c4a1ef159b32e004fe. The worktree is clean. I am continuing with the custody restart regression and tracing the server-derived Mesh DNS suffix into the accepted publication context without enabling the default-off Mesh DNS feature. Plan remains about 2 hours of active work, excluding build-slot waits: finish those regressions, run focused proxy/Mesh and namespace/Pebble checks, run scripts/verify fast, report static-size deltas, push and post exact receipts, then remove J62 targets. Current queue check: compute01 is full with 122 GB free; dev-02 has free slots but only 16 GB free, below the build floor, so compute01 is the only usable Linux VM right now. No J62 build is pending.

J62 resumed from pushed head `fe43a8314` on `proxy-traefik-281`, rebased onto `origin/dev` `d1ec515e44dabccb640801c4a1ef159b32e004fe`. The worktree is clean. I am continuing with the custody restart regression and tracing the server-derived Mesh DNS suffix into the accepted publication context without enabling the default-off Mesh DNS feature. Plan remains about 2 hours of active work, excluding build-slot waits: finish those regressions, run focused proxy/Mesh and namespace/Pebble checks, run `scripts/verify fast`, report static-size deltas, push and post exact receipts, then remove J62 targets. Current queue check: compute01 is full with 122 GB free; dev-02 has free slots but only 16 GB free, below the build floor, so compute01 is the only usable Linux VM right now. No J62 build is pending.

kayg commented 2026-11-02 00:03:16 +00:00

Author

Owner

Copy link

Finding on #281: the feature-off Mesh store discards public_origin, then Snapshot::config calls configuration::compile, which hardcodes internal for ExposureIdentity. The proxy therefore receives the wrong naming suffix in normal builds. RED evidence on compute01 at fe43a8314: cargo test --locked -p infraternal-plugin-mesh --test store publication_identity_tracks_server_public_host_across_restart -- --exact --nocapture returned test result: FAILED. 0 passed; 1 failed with left: "internal", right: "ssh.example.test". I will pass the public-host-derived suffix through the accepted exposure identity and advance the configuration revision when that origin suffix changes, while keeping resolver fields and mesh-dns-spike default-off.

Finding on #281: the feature-off Mesh store discards `public_origin`, then `Snapshot::config` calls `configuration::compile`, which hardcodes `internal` for `ExposureIdentity`. The proxy therefore receives the wrong naming suffix in normal builds. RED evidence on compute01 at `fe43a8314`: `cargo test --locked -p infraternal-plugin-mesh --test store publication_identity_tracks_server_public_host_across_restart -- --exact --nocapture` returned `test result: FAILED. 0 passed; 1 failed` with `left: "internal"`, `right: "ssh.example.test"`. I will pass the public-host-derived suffix through the accepted exposure identity and advance the configuration revision when that origin suffix changes, while keeping resolver fields and `mesh-dns-spike` default-off.

kayg commented 2026-11-02 00:11:13 +00:00

Author

Owner

Copy link

J62 diagnostics: the custody restart regression was configured incorrectly. It set INFRATERAL_DNS_* (missing the N), while DnsProviderProfile::from_environment reads INFRATERNAL_DNS_*; the profile was therefore None, and startup correctly deleted the provider as unconfigured. I am correcting the test fixture before judging the implementation. The first suffix GREEN attempt also stopped at compile time because DEFAULT_DNS_SUFFIX is currently gated behind mesh-dns-spike; I will make the shared compatibility constant available to the server snapshot compiler without enabling resolver fields or the default-off feature.

J62 diagnostics: the custody restart regression was configured incorrectly. It set `INFRATERAL_DNS_*` (missing the `N`), while `DnsProviderProfile::from_environment` reads `INFRATERNAL_DNS_*`; the profile was therefore `None`, and startup correctly deleted the provider as unconfigured. I am correcting the test fixture before judging the implementation. The first suffix GREEN attempt also stopped at compile time because `DEFAULT_DNS_SUFFIX` is currently gated behind `mesh-dns-spike`; I will make the shared compatibility constant available to the server snapshot compiler without enabling resolver fields or the default-off feature.

kayg commented 2026-11-02 00:18:38 +00:00

Author

Owner

Copy link

J62 resumed from pushed head fe43a8314 on proxy-traefik-281; the branch merge base is d1ec515e4, and current origin/dev is ffebd0561.

The resumed compute01 checks in /home/claude/.cache/codex-scratch/j62/proxy-recheck.log now pass: publication_identity_tracks_server_public_host_across_restart (MESH_SUFFIX_EXIT=0) and agent_restart_retains_the_configured_dns_provider_files_for_renewal (PROXY_CUSTODY_EXIT=0). The earlier custody failure came from a test fixture spelling INFRATERAL_*; the corrected fixture now verifies the configured provider survives agent restart. No J62 process remains queued or running.

The local WIP includes the default-off Mesh DNS suffix propagation and the private-ping negative health regression. The module documentation and command table record the orchestrator's CLI choice: infraternal unpublish <port|name> removes one publication, reports what stopped being reachable, returns D303 missing / exit 4 for an absent target, and uses the shared --json envelope. Removing the last route starts the existing 10-second drain. This choice remains open for an owner rename. Next: rebase onto current origin/dev, run the focused proxy/Mesh/CLI and namespace/Pebble checks, then the required fast gate and report-only size comparison.

J62 resumed from pushed head `fe43a8314` on `proxy-traefik-281`; the branch merge base is `d1ec515e4`, and current `origin/dev` is `ffebd0561`. The resumed compute01 checks in `/home/claude/.cache/codex-scratch/j62/proxy-recheck.log` now pass: `publication_identity_tracks_server_public_host_across_restart` (`MESH_SUFFIX_EXIT=0`) and `agent_restart_retains_the_configured_dns_provider_files_for_renewal` (`PROXY_CUSTODY_EXIT=0`). The earlier custody failure came from a test fixture spelling `INFRATERAL_*`; the corrected fixture now verifies the configured provider survives agent restart. No J62 process remains queued or running. The local WIP includes the default-off Mesh DNS suffix propagation and the private-ping negative health regression. The module documentation and command table record the orchestrator's CLI choice: `infraternal unpublish <port|name>` removes one publication, reports what stopped being reachable, returns D303 `missing` / exit 4 for an absent target, and uses the shared `--json` envelope. Removing the last route starts the existing 10-second drain. This choice remains open for an owner rename. Next: rebase onto current `origin/dev`, run the focused proxy/Mesh/CLI and namespace/Pebble checks, then the required fast gate and report-only size comparison.

kayg commented 2026-11-02 00:22:18 +00:00

Author

Owner

Copy link

J62 branch update: proxy-traefik-281 is now rebased onto current origin/dev ffebd0561a010690583481abc4d583ecaca4542e; pushed checkpoint head 0d8e4b6a56ffc5109e116c1f51f859de4a6a305f. The seven-file local WIP remains uncommitted: it carries the public-host-derived Mesh suffix in normal builds and tightens the provider-restart/private-ping regressions. The two targeted GREEN results remain in proxy-recheck.log; the stale queued run is complete.

I am checking both VM slots and will run the full proxy suite, Mesh regression, namespace/Pebble staging case, and final fast gate on a VM with a free slot and enough disk. I will consolidate the crash-safety WIP commits before the final push.

J62 branch update: `proxy-traefik-281` is now rebased onto current `origin/dev` `ffebd0561a010690583481abc4d583ecaca4542e`; pushed checkpoint head `0d8e4b6a56ffc5109e116c1f51f859de4a6a305f`. The seven-file local WIP remains uncommitted: it carries the public-host-derived Mesh suffix in normal builds and tightens the provider-restart/private-ping regressions. The two targeted GREEN results remain in `proxy-recheck.log`; the stale queued run is complete. I am checking both VM slots and will run the full proxy suite, Mesh regression, namespace/Pebble staging case, and final fast gate on a VM with a free slot and enough disk. I will consolidate the crash-safety WIP commits before the final push.

kayg commented 2026-11-02 00:24:48 +00:00

Author

Owner

Copy link

The first full-format preflight on the rebased snapshot stopped before compiling: cargo fmt --all --check found existing formatting drift in mesh/src/node_actions.rs and proxy controller/supervisor files (the new health regression was also unformatted). I ran rustfmt only for the touched Mesh and proxy packages; no behavior changed and no test ran in that attempt. I am resyncing the formatted source and rerunning the focused checks through build-slot.

The first full-format preflight on the rebased snapshot stopped before compiling: `cargo fmt --all --check` found existing formatting drift in `mesh/src/node_actions.rs` and proxy controller/supervisor files (the new health regression was also unformatted). I ran rustfmt only for the touched Mesh and proxy packages; no behavior changed and no test ran in that attempt. I am resyncing the formatted source and rerunning the focused checks through `build-slot`.

kayg commented 2026-11-02 00:28:40 +00:00

Author

Owner

Copy link

Focused dev-02 suite finding on the current source: 58 passed; 2 failed. live_process_without_a_healthy_private_ping_is_reported_unhealthy failed because IsRunning only checked Child::try_wait and never probed the configured private /ping; the source path returned true for a live child even after HTTP 503. I added the loopback probe to that health observation, with a 250 ms bound.

The existing first-route lifecycle test also timed out once in the full suite, but a fresh isolated rerun passed (1 passed; 0 failed; finished in 1.15s) and its fixture answered the first healthy request. The test helper now reports how many requests it served; I will keep this finding open until a full rerun determines whether the timeout repeats.

Focused dev-02 suite finding on the current source: `58 passed; 2 failed`. `live_process_without_a_healthy_private_ping_is_reported_unhealthy` failed because `IsRunning` only checked `Child::try_wait` and never probed the configured private `/ping`; the source path returned `true` for a live child even after HTTP 503. I added the loopback probe to that health observation, with a 250 ms bound. The existing first-route lifecycle test also timed out once in the full suite, but a fresh isolated rerun passed (`1 passed; 0 failed; finished in 1.15s`) and its fixture answered the first healthy request. The test helper now reports how many requests it served; I will keep this finding open until a full rerun determines whether the timeout repeats.

kayg commented 2026-11-02 00:34:57 +00:00

Author

Owner

Copy link

J62 checkpoint on proxy-traefik-281, pushed head d267360998ba71b8e80038927776d8dd35accb44, based on current origin/dev ffebd0561a010690583481abc4d583ecaca4542e. The checkpoint preserves the normal-build public-host-derived proxy suffix, private-ping health observation/regression, and corrected custody-restart fixture. The same snapshot is queued on dev-02 in ~/.cache/codex-scratch/j62-final/focused-tests.sh (fmt, proxy tests/clippy, Mesh store test/clippy); its log is ~/.cache/codex-scratch/j62-final/focused-tests.log. After it completes I’ll run the namespace/Pebble check and final scripts/verify fast, then split this WIP checkpoint into atomic commits. The separate unpublish <port|name> command remains recorded in the module doc and #281 as the orchestrator’s choice, open to owner rename.

J62 checkpoint on `proxy-traefik-281`, pushed head `d267360998ba71b8e80038927776d8dd35accb44`, based on current `origin/dev` `ffebd0561a010690583481abc4d583ecaca4542e`. The checkpoint preserves the normal-build public-host-derived proxy suffix, private-ping health observation/regression, and corrected custody-restart fixture. The same snapshot is queued on dev-02 in `~/.cache/codex-scratch/j62-final/focused-tests.sh` (fmt, proxy tests/clippy, Mesh store test/clippy); its log is `~/.cache/codex-scratch/j62-final/focused-tests.log`. After it completes I’ll run the namespace/Pebble check and final `scripts/verify fast`, then split this WIP checkpoint into atomic commits. The separate `unpublish <port|name>` command remains recorded in the module doc and #281 as the orchestrator’s choice, open to owner rename.

kayg commented 2026-11-02 00:36:40 +00:00

Author

Owner

Copy link

J62 focused dev-02 run completed from the queued snapshot. Formatting passed; the proxy crate reported test result: ok. 60 passed; 0 failed; 0 ignored and its ordinary integration target reported 1 passed. Proxy Clippy then exited 101 with seven Rust 1.99 pedantic findings: similar listener variable names, missing backticks around OpenBao, one extraction function over 100 lines, two 64 KiB stack buffers, a derivable runtime default, and a collapsible lifecycle match. The script stopped there, so the Mesh store test/clippy did not run. Full receipt: /home/claude/.cache/codex-scratch/j62-final/focused-tests.log; no J62 process remains. Next I’m splitting the extraction helper and fixing the lint findings, then I’ll rerun the focused checks through build-slot.

J62 focused dev-02 run completed from the queued snapshot. Formatting passed; the proxy crate reported `test result: ok. 60 passed; 0 failed; 0 ignored` and its ordinary integration target reported `1 passed`. Proxy Clippy then exited 101 with seven Rust 1.99 pedantic findings: similar listener variable names, missing backticks around OpenBao, one extraction function over 100 lines, two 64 KiB stack buffers, a derivable runtime default, and a collapsible lifecycle match. The script stopped there, so the Mesh store test/clippy did not run. Full receipt: `/home/claude/.cache/codex-scratch/j62-final/focused-tests.log`; no J62 process remains. Next I’m splitting the extraction helper and fixing the lint findings, then I’ll rerun the focused checks through build-slot.

kayg commented 2026-11-02 00:41:32 +00:00

Author

Owner

Copy link

After the first focused run, I split the long archive walk into per-entry/executable/license helpers and moved its 64 KiB copy buffers to the heap; I also fixed the remaining strict Clippy findings without changing behavior. WIP checkpoint 9b6c5ff84239470e866e352051a3658d059c5049 is pushed on proxy-traefik-281. A fresh focused run is queued on dev-02 through build-slot at ~/.cache/codex-scratch/j62-final/focused-tests-2.sh; the receipt will be ~/.cache/codex-scratch/j62-final/focused-tests-2.log. Current queue position was 2 with all slots busy and 29 GB free; compute01 also had all slots occupied.

After the first focused run, I split the long archive walk into per-entry/executable/license helpers and moved its 64 KiB copy buffers to the heap; I also fixed the remaining strict Clippy findings without changing behavior. WIP checkpoint `9b6c5ff84239470e866e352051a3658d059c5049` is pushed on `proxy-traefik-281`. A fresh focused run is queued on dev-02 through `build-slot` at `~/.cache/codex-scratch/j62-final/focused-tests-2.sh`; the receipt will be `~/.cache/codex-scratch/j62-final/focused-tests-2.log`. Current queue position was 2 with all slots busy and 29 GB free; compute01 also had all slots occupied.

kayg commented 2026-11-02 00:59:22 +00:00

Author

Owner

Copy link

J62 focused Clippy finding on branch head 9b6c5ff (the lint-only edits are still uncommitted): the proxy unit suite passed 60/60 and the child process test passed 1/1. cargo clippy --locked --jobs 3 -p infraternal-plugin-proxy --all-targets -- -D warnings then failed only in integration-test crates: 13 expect_used diagnostics in tests/process.rs, and 30 expect_used diagnostics plus too_many_lines in tests/namespace_pebble.rs. The earlier five lints in the proxy library/unit tests no longer appeared. Full receipt: /home/claude/.cache/codex-scratch/j62-final/focused-post-lint.log on dev-02 (FOCUSED_EXIT=101).

I added test-crate-scoped failure-context allowances and one function-scoped line-count allowance; production linting remains unchanged. Those changes have not yet been rerun. Compute01 has no free slot and dev-02 is below its 25 GB admission floor, so I’m syncing the current source to the existing compute01 checkout and will submit the next focused run detached through build-slot.

J62 focused Clippy finding on branch head 9b6c5ff (the lint-only edits are still uncommitted): the proxy unit suite passed 60/60 and the child process test passed 1/1. `cargo clippy --locked --jobs 3 -p infraternal-plugin-proxy --all-targets -- -D warnings` then failed only in integration-test crates: 13 `expect_used` diagnostics in `tests/process.rs`, and 30 `expect_used` diagnostics plus `too_many_lines` in `tests/namespace_pebble.rs`. The earlier five lints in the proxy library/unit tests no longer appeared. Full receipt: `/home/claude/.cache/codex-scratch/j62-final/focused-post-lint.log` on dev-02 (FOCUSED_EXIT=101). I added test-crate-scoped failure-context allowances and one function-scoped line-count allowance; production linting remains unchanged. Those changes have not yet been rerun. Compute01 has no free slot and dev-02 is below its 25 GB admission floor, so I’m syncing the current source to the existing compute01 checkout and will submit the next focused run detached through build-slot.

kayg commented 2026-11-02 01:05:12 +00:00

Author

Owner

Copy link

J62 continuation: the current six-file Clippy cleanup is synced by SHA-256 to the detached dev-02 checkout at 9b6c5ff84239470e866e352051a3658d059c5049. The focused run covers formatting, proxy tests and Clippy, the #266 suffix persistence regression, Mesh Clippy, CLI and agent action tests, plus release component manifest/publication tests. It is queued through build-slot at position 2; no tests have started yet. Log: /home/claude/.cache/codex-scratch/j62-final/focused-final.log. Next: finish the focused run, run the x86 Pebble namespace test and exact-head fast gate, then post the final head and verbatim gate output.

J62 continuation: the current six-file Clippy cleanup is synced by SHA-256 to the detached dev-02 checkout at `9b6c5ff84239470e866e352051a3658d059c5049`. The focused run covers formatting, proxy tests and Clippy, the #266 suffix persistence regression, Mesh Clippy, CLI and agent action tests, plus release component manifest/publication tests. It is queued through `build-slot` at position 2; no tests have started yet. Log: `/home/claude/.cache/codex-scratch/j62-final/focused-final.log`. Next: finish the focused run, run the x86 Pebble namespace test and exact-head fast gate, then post the final head and verbatim gate output.

kayg commented 2026-11-02 01:07:59 +00:00

Author

Owner

Copy link

Focused-run launcher finding: the first detached invocation exited before tests with path "/home/claude" does not contain a flake.nix``. Root cause was invoking nix develop before changing into the J62 checkout; no compilation or test began. I added the missing checkout cd to the runner, confirmed dev-02 has free build slots and 30 GB free, and am resubmitting through build-slot.

Focused-run launcher finding: the first detached invocation exited before tests with `path "/home/claude" does not contain a `flake.nix``. Root cause was invoking `nix develop` before changing into the J62 checkout; no compilation or test began. I added the missing checkout `cd` to the runner, confirmed dev-02 has free build slots and 30 GB free, and am resubmitting through `build-slot`.

kayg commented 2026-11-02 01:10:12 +00:00

Author

Owner

Copy link

Focused verification finding: Mesh Clippy reports compile_inner at 101/100 lines after the public-host-derived suffix was added. I added a regression test for server-default, legacy custom, and explicit suffix selection; it failed as expected at compile time because the extracted helper was not yet defined (RED_EXIT=101, unresolved selected_dns_suffix). I extracted the existing selection rule into that helper without changing its cases. Next: run the regression and Mesh Clippy GREEN before restarting the full focused suite.

Focused verification finding: Mesh Clippy reports `compile_inner` at 101/100 lines after the public-host-derived suffix was added. I added a regression test for server-default, legacy custom, and explicit suffix selection; it failed as expected at compile time because the extracted helper was not yet defined (`RED_EXIT=101`, unresolved `selected_dns_suffix`). I extracted the existing selection rule into that helper without changing its cases. Next: run the regression and Mesh Clippy GREEN before restarting the full focused suite.

kayg commented 2026-11-02 01:11:06 +00:00

Author

Owner

Copy link

Mesh suffix lint fix is verified RED→GREEN: the new server_dns_suffix_is_default_and_explicit_or_legacy_choices_win test first failed on the absent helper, then passed 1/1 after extracting the existing selection rule. cargo clippy --locked --jobs 3 -p infraternal-plugin-mesh --all-targets -- -D warnings also passed. The full focused proxy/Mesh/CLI/release-test script has been resubmitted through build-slot on dev-02; log: /home/claude/.cache/codex-scratch/j62-final/focused-final-3.log.

Mesh suffix lint fix is verified RED→GREEN: the new `server_dns_suffix_is_default_and_explicit_or_legacy_choices_win` test first failed on the absent helper, then passed `1/1` after extracting the existing selection rule. `cargo clippy --locked --jobs 3 -p infraternal-plugin-mesh --all-targets -- -D warnings` also passed. The full focused proxy/Mesh/CLI/release-test script has been resubmitted through `build-slot` on dev-02; log: `/home/claude/.cache/codex-scratch/j62-final/focused-final-3.log`.

kayg commented 2026-11-02 01:12:17 +00:00

Author

Owner

Copy link

The full focused script reached the CLI check after these passed: formatting, all proxy tests, proxy Clippy, the #266 public-host suffix persistence test, and Mesh Clippy. The CLI check exited immediately with error: no library targets found in package infraternal``; source inspection confirms this package is a single binary and the publication tests are nested under component_cli::tests. I’m correcting the test invocation to --bin infraternal component_cli::tests, then I’ll rerun the complete focused command.

The full focused script reached the CLI check after these passed: formatting, all proxy tests, proxy Clippy, the #266 public-host suffix persistence test, and Mesh Clippy. The CLI check exited immediately with `error: no library targets found in package `infraternal``; source inspection confirms this package is a single binary and the publication tests are nested under `component_cli::tests`. I’m correcting the test invocation to `--bin infraternal component_cli::tests`, then I’ll rerun the complete focused command.

kayg commented 2026-11-02 01:16:28 +00:00

Author

Owner

Copy link

CLI test finding: compiling component_cli::tests on the J62 branch produced three errors: the new test imported missing add_cli_values, and run_action returned execute's Result<Exit> through block_on_io, whose existing contract is Result<()>. The missing mapper also meant registered positional and named values were never put into the request. I added schema-typed argument mapping (including positional port/name values and flags) and wrapped the Exit result while retaining the existing reactor helper contract. The new CLI tests have not yet passed; next is the focused RED→GREEN run.

CLI test finding: compiling `component_cli::tests` on the J62 branch produced three errors: the new test imported missing `add_cli_values`, and `run_action` returned `execute`'s `Result<Exit>` through `block_on_io`, whose existing contract is `Result<()>`. The missing mapper also meant registered positional and named values were never put into the request. I added schema-typed argument mapping (including positional port/name values and flags) and wrapped the `Exit` result while retaining the existing reactor helper contract. The new CLI tests have not yet passed; next is the focused RED→GREEN run.

kayg commented 2026-11-02 01:18:36 +00:00

Author

Owner

Copy link

The focused CLI regression is GREEN: cargo test --locked --jobs 3 -p infraternal --bin infraternal component_cli::tests -- --nocapture passed 5/5, including positional publish/unpublish mapping and D303 exit 4. run_action now captures the typed exit while keeping block_on_io's existing unit-return contract. The compiler also exposed an unused_mut warning in the J62 Mesh feature wiring; I restored the cfg-scoped mutation pattern and am checking Mesh with both agent/server features plus the default-off DNS feature before the full focused rerun.

The focused CLI regression is GREEN: `cargo test --locked --jobs 3 -p infraternal --bin infraternal component_cli::tests -- --nocapture` passed `5/5`, including positional publish/unpublish mapping and D303 exit 4. `run_action` now captures the typed exit while keeping `block_on_io`'s existing unit-return contract. The compiler also exposed an `unused_mut` warning in the J62 Mesh feature wiring; I restored the cfg-scoped mutation pattern and am checking Mesh with both agent/server features plus the default-off DNS feature before the full focused rerun.

kayg commented 2026-11-02 01:19:24 +00:00

Author

Owner

Copy link

Mesh feature verification is GREEN after the cfg-scoped mutation adjustment: cargo clippy --locked --jobs 3 -p infraternal-plugin-mesh --all-targets --features agent,server -- -D warnings passed, and cargo check --locked --jobs 3 -p infraternal-plugin-mesh --features mesh-dns-spike passed. The first retry exposed a missing semicolon in the cfg-only binding; that syntax error is corrected. The Mesh suffix selector regression and the five CLI tests are green. Next: rerun the full focused suite on the exact current source, then the x86 Pebble integration and final gate.

Mesh feature verification is GREEN after the cfg-scoped mutation adjustment: `cargo clippy --locked --jobs 3 -p infraternal-plugin-mesh --all-targets --features agent,server -- -D warnings` passed, and `cargo check --locked --jobs 3 -p infraternal-plugin-mesh --features mesh-dns-spike` passed. The first retry exposed a missing semicolon in the cfg-only binding; that syntax error is corrected. The Mesh suffix selector regression and the five CLI tests are green. Next: rerun the full focused suite on the exact current source, then the x86 Pebble integration and final gate.

kayg commented 2026-11-02 01:20:27 +00:00

Author

Owner

Copy link

The full focused runner stopped at cargo fmt --all --check before compilation, with two formatting diffs in the new component_cli.rs argument mapper. No tests ran in that invocation. I’m applying the repository formatter to the adapter and will rerun the full focused command.

The full focused runner stopped at `cargo fmt --all --check` before compilation, with two formatting diffs in the new `component_cli.rs` argument mapper. No tests ran in that invocation. I’m applying the repository formatter to the adapter and will rerun the full focused command.

kayg commented 2026-11-02 01:25:37 +00:00

Author

Owner

Copy link

J62 focused rerun is active on dev-02 (PID 3362040); log: /home/claude/.cache/codex-scratch/j62-final/focused-final-7.log. The preceding run reached the agent Mesh grant test and stopped at compile time: assert_eq! could not infer the type for expected .parse() values compared with Vec<IpAddr>. The expectation now uses explicit parse::<std::net::IpAddr>(); this is a test-fixture type inference fix, not a runtime behavior change.\n\nCommand-table choice from the orchestrator: use separate infraternal unpublish <port|name>. It prints what stopped being reachable, is idempotent with D303 missing / exit 4 and a one-line message, and supports --json. Removing the last publication enters the existing 10-second drain. This is an orchestrator choice pending any owner rename and is documented in the proxy module doc and command table.

J62 focused rerun is active on dev-02 (PID 3362040); log: /home/claude/.cache/codex-scratch/j62-final/focused-final-7.log. The preceding run reached the agent Mesh grant test and stopped at compile time: `assert_eq!` could not infer the type for expected `.parse()` values compared with `Vec<IpAddr>`. The expectation now uses explicit `parse::<std::net::IpAddr>()`; this is a test-fixture type inference fix, not a runtime behavior change.\n\nCommand-table choice from the orchestrator: use separate `infraternal unpublish <port|name>`. It prints what stopped being reachable, is idempotent with D303 `missing` / exit 4 and a one-line message, and supports `--json`. Removing the last publication enters the existing 10-second drain. This is an orchestrator choice pending any owner rename and is documented in the proxy module doc and command table.

kayg commented 2026-11-02 01:39:27 +00:00

Author

Owner

Copy link

J62 is aligned with current origin/dev 9f36dd8e0a5e9e799cd3264cb6e1e7605071c5e2. The merge had one Cargo.lock conflict; I kept the latest dev dependency pins and added only the proxy component graph (101 added lock lines, no unrelated dependency upgrades). cargo check --locked --jobs 3 -p infraternal-plugin-proxy passed on compute01. The full focused proxy/Mesh/CLI/release suite is now running on that merged source through build-slot; log: /home/claude/.cache/codex-scratch/j62/focused-merged.log. Next: x86 namespace/Pebble staging ACME, scripts/verify fast, static-size report, cleanup, final push.

J62 is aligned with current origin/dev `9f36dd8e0a5e9e799cd3264cb6e1e7605071c5e2`. The merge had one Cargo.lock conflict; I kept the latest dev dependency pins and added only the proxy component graph (101 added lock lines, no unrelated dependency upgrades). `cargo check --locked --jobs 3 -p infraternal-plugin-proxy` passed on compute01. The full focused proxy/Mesh/CLI/release suite is now running on that merged source through build-slot; log: `/home/claude/.cache/codex-scratch/j62/focused-merged.log`. Next: x86 namespace/Pebble staging ACME, `scripts/verify fast`, static-size report, cleanup, final push.

kayg commented 2026-11-02 01:49:58 +00:00

Author

Owner

Copy link

Namespace/Pebble setup RED finding on compute01: both pinned archives passed their SHA checks, but the runner expected binaries at archive root. tar -tzf showed pebble-linux-amd64/linux/amd64/pebble and pebble-challtestsrv-linux-amd64/linux/amd64/pebble-challtestsrv. I updated scripts/test-proxy-pebble.sh to extract those exact allowlisted paths to private test files, then chmod and validate them. The corrected staging ACME test is running through build-slot; log: /home/claude/.cache/codex-scratch/j62/pebble-e2e-2.log.

Namespace/Pebble setup RED finding on compute01: both pinned archives passed their SHA checks, but the runner expected binaries at archive root. `tar -tzf` showed `pebble-linux-amd64/linux/amd64/pebble` and `pebble-challtestsrv-linux-amd64/linux/amd64/pebble-challtestsrv`. I updated `scripts/test-proxy-pebble.sh` to extract those exact allowlisted paths to private test files, then chmod and validate them. The corrected staging ACME test is running through build-slot; log: `/home/claude/.cache/codex-scratch/j62/pebble-e2e-2.log`.

kayg commented 2026-11-02 01:50:27 +00:00

Author

Owner

Copy link

Pebble runner follow-up: after the archive paths were corrected, the wrapper exited before the test with cargo: command not found because a nested login shell reset the PATH supplied by nix develop. No ACME scenario ran. I am rerunning the same script directly from a non-login shell inside the dev shell, retaining the existing job target.

Pebble runner follow-up: after the archive paths were corrected, the wrapper exited before the test with `cargo: command not found` because a nested login shell reset the PATH supplied by `nix develop`. No ACME scenario ran. I am rerunning the same script directly from a non-login shell inside the dev shell, retaining the existing job target.

kayg commented 2026-11-02 01:56:35 +00:00

Author

Owner

Copy link

Pebble runner finding on proxy-traefik-281: the corrected allowlisted archive paths still failed on rerun because shell redirection attempted to overwrite an existing 0500 binary. Evidence: /home/claude/.cache/codex-scratch/j62/pebble-e2e-3.log ended with Permission denied at the binary path and PEBBLE_E2E_3_EXIT=1; no ACME scenario ran. I changed the runner to extract both binaries into a unique private staging directory, validate and chmod them, then atomically rename them into place.

Next: run the Pebble namespace staging-ACME test on compute01 through build-slot, run scripts/verify fast on the VM with the best free-slot/disk capacity, report the D243 static-size delta, then push and post the final head plus verbatim gate output. Active estimate: about 45 minutes, excluding build-slot waits. The separate unpublish <port|name> command choice and D303 missing/exit-4 behavior are already documented in the module and prior #281 comment as the orchestrator's recommendation pending owner rename.

Pebble runner finding on `proxy-traefik-281`: the corrected allowlisted archive paths still failed on rerun because shell redirection attempted to overwrite an existing `0500` binary. Evidence: `/home/claude/.cache/codex-scratch/j62/pebble-e2e-3.log` ended with `Permission denied` at the binary path and `PEBBLE_E2E_3_EXIT=1`; no ACME scenario ran. I changed the runner to extract both binaries into a unique private staging directory, validate and chmod them, then atomically rename them into place. Next: run the Pebble namespace staging-ACME test on compute01 through `build-slot`, run `scripts/verify fast` on the VM with the best free-slot/disk capacity, report the D243 static-size delta, then push and post the final head plus verbatim gate output. Active estimate: about 45 minutes, excluding build-slot waits. The separate `unpublish <port|name>` command choice and D303 missing/exit-4 behavior are already documented in the module and prior #281 comment as the orchestrator's recommendation pending owner rename.

kayg commented 2026-11-02 01:59:31 +00:00

Author

Owner

Copy link

Second Pebble rerun finding: staging and replacing the two pinned Pebble executables now succeeds, then extraction failed on the cached Traefik executable at scripts/test-proxy-pebble.sh:95 with Permission denied (PEBBLE_E2E_4_EXIT=1). The root cause was the same: shell redirection tried to truncate the existing 0500 executable before checking the signed-manifest executable digest. The runner now stages Traefik under a unique same-filesystem name, verifies its digest, chmods it, then atomically renames it into place; one exit trap cleans all remaining staging paths. I am resyncing and rerunning the namespace staging-ACME test, then the final fast gate. Estimate remains about 45 minutes active, excluding build-slot waits.

Second Pebble rerun finding: staging and replacing the two pinned Pebble executables now succeeds, then extraction failed on the cached Traefik executable at `scripts/test-proxy-pebble.sh:95` with `Permission denied` (`PEBBLE_E2E_4_EXIT=1`). The root cause was the same: shell redirection tried to truncate the existing `0500` executable before checking the signed-manifest executable digest. The runner now stages Traefik under a unique same-filesystem name, verifies its digest, chmods it, then atomically renames it into place; one exit trap cleans all remaining staging paths. I am resyncing and rerunning the namespace staging-ACME test, then the final fast gate. Estimate remains about 45 minutes active, excluding build-slot waits.

kayg commented 2026-11-02 02:03:09 +00:00

Author

Owner

Copy link

Pebble ACME finding: the runner now replaces the cached Pebble and Traefik executables and starts the namespace scenario, but the test times out after 45 seconds waiting for issuance and the mesh route (PEBBLE_E2E_5_EXIT=101). Its private acme.json remained the initial 2-byte {}, so no certificate was issued. The fixture currently drops all Traefik child logs, leaving the provider/ACME failure unlocalized. I will capture only a bounded tail of the supervisor's already-redacted log lines on test failure, then rerun before changing production behavior. No real CA endpoint is used.

Pebble ACME finding: the runner now replaces the cached Pebble and Traefik executables and starts the namespace scenario, but the test times out after 45 seconds waiting for issuance and the mesh route (`PEBBLE_E2E_5_EXIT=101`). Its private `acme.json` remained the initial 2-byte `{}`, so no certificate was issued. The fixture currently drops all Traefik child logs, leaving the provider/ACME failure unlocalized. I will capture only a bounded tail of the supervisor's already-redacted log lines on test failure, then rerun before changing production behavior. No real CA endpoint is used.

kayg commented 2026-11-02 02:10:31 +00:00

Author

Owner

Copy link

Pebble diagnostic result: on the 45-second failure, the HTTPREQ fixture recorded zero provider calls and no adapter errors; verified curl received Traefik's default certificate, with no SAN for docs.node.example.org. The persisted dynamic.json contains the exact Host router and certResolver: dns, and the static config contains the dns ACME resolver. I am adding a failure-only --insecure request to report the HTTP status while preserving certificate verification as the success condition; that will distinguish a missing file-provider route from an ACME activation failure before any production change.

Pebble diagnostic result: on the 45-second failure, the HTTPREQ fixture recorded zero provider calls and no adapter errors; verified curl received Traefik's default certificate, with no SAN for `docs.node.example.org`. The persisted `dynamic.json` contains the exact Host router and `certResolver: dns`, and the static config contains the `dns` ACME resolver. I am adding a failure-only `--insecure` request to report the HTTP status while preserving certificate verification as the success condition; that will distinguish a missing file-provider route from an ACME activation failure before any production change.

kayg commented 2026-11-02 02:14:35 +00:00

Author

Owner

Copy link

Confirmed file-provider bug: Traefik v3.7.17's isFileSupported accepts only .toml, .yaml, and .yml, so it silently ignores .json snapshots (source: https://github.com/traefik/traefik/blob/v3.7.17/pkg/provider/file/file.go#L516-L523). The namespace test's dynamic.json therefore never loaded: the failure-only request returned HTTP 404 and HTTPREQ saw zero calls. Production also wrote JSON bytes to services.toml, which is not TOML. I am aligning both paths on one services.yaml filename and keeping the JSON-serialized snapshot, which is valid YAML, then rerunning the Pebble ACME case.

Confirmed file-provider bug: Traefik v3.7.17's `isFileSupported` accepts only `.toml`, `.yaml`, and `.yml`, so it silently ignores `.json` snapshots (source: https://github.com/traefik/traefik/blob/v3.7.17/pkg/provider/file/file.go#L516-L523). The namespace test's `dynamic.json` therefore never loaded: the failure-only request returned HTTP 404 and HTTPREQ saw zero calls. Production also wrote JSON bytes to `services.toml`, which is not TOML. I am aligning both paths on one `services.yaml` filename and keeping the JSON-serialized snapshot, which is valid YAML, then rerunning the Pebble ACME case.

kayg commented 2026-11-02 02:18:45 +00:00

Author

Owner

Copy link

The services.yaml filename resolves the routing failure: the latest Pebble run returns HTTP 200 from the granted backend when certificate verification is bypassed. The remaining ACME failure is localized after the DNS provider hook: HTTPREQ received one /present call with no adapter error, the node-local Pebble account was registered, but its certificate list stayed empty; the verified request still saw the default certificate. Evidence: /home/claude/.cache/codex-scratch/j62/pebble-e2e-9.log. I am checking the fixture's TXT publication and Pebble validation path before changing production code.

The `services.yaml` filename resolves the routing failure: the latest Pebble run returns HTTP 200 from the granted backend when certificate verification is bypassed. The remaining ACME failure is localized after the DNS provider hook: HTTPREQ received one `/present` call with no adapter error, the node-local Pebble account was registered, but its certificate list stayed empty; the verified request still saw the default certificate. Evidence: `/home/claude/.cache/codex-scratch/j62/pebble-e2e-9.log`. I am checking the fixture's TXT publication and Pebble validation path before changing production code.

kayg commented 2026-11-02 02:31:16 +00:00

Author

Owner

Copy link

J62 reviewer-finding triage on the merged worktree. Evidence: release producer inspect_archive() accepts regular README.md, while the runtime installer allowlist is exactly traefik and LICENSE.md; the new producer regression ran on compute01 and failed as expected (ValueError not raised). Mixed public/mesh routes currently share wildcard-bound entrypoints, while mesh routes retain their exact-host and grant middleware. Log reading retains only the first 8 KiB before full-secret redaction, although provider values can be 64 KiB. I also checked the backoff claim: recv_timeout returns as soon as a replacement command is queued, so the review’s claimed full-delay wait is not reproduced by this std mpsc path; I am adding a regression that proves queued removal is handled before its deadline. Next: make archive and mixed-listener regressions pass, discard oversized log lines, complete Pebble/focused tests, then run the final VM gate.

J62 reviewer-finding triage on the merged worktree. Evidence: release producer `inspect_archive()` accepts regular `README.md`, while the runtime installer allowlist is exactly `traefik` and `LICENSE.md`; the new producer regression ran on compute01 and failed as expected (`ValueError not raised`). Mixed public/mesh routes currently share wildcard-bound entrypoints, while mesh routes retain their exact-host and grant middleware. Log reading retains only the first 8 KiB before full-secret redaction, although provider values can be 64 KiB. I also checked the backoff claim: `recv_timeout` returns as soon as a replacement command is queued, so the review’s claimed full-delay wait is not reproduced by this std mpsc path; I am adding a regression that proves queued removal is handled before its deadline. Next: make archive and mixed-listener regressions pass, discard oversized log lines, complete Pebble/focused tests, then run the final VM gate.

kayg commented 2026-11-02 02:35:31 +00:00

Author

Owner

Copy link

Additional #281 acquisition evidence: compute01 downloaded the pinned Traefik 3.7.17 linux/amd64 archive, verified its SHA-256 against traefik_v3.7.17_checksums.txt (UPSTREAM_CHECKSUM_MATCH=True), and listed its tar members: CHANGELOG.md, LICENSE.md, traefik. The producer and runtime importer need matching explicit allowlists including the signed changelog, while only the executable is extracted; arbitrary regular files remain rejected.

Additional #281 acquisition evidence: compute01 downloaded the pinned Traefik 3.7.17 linux/amd64 archive, verified its SHA-256 against `traefik_v3.7.17_checksums.txt` (`UPSTREAM_CHECKSUM_MATCH=True`), and listed its tar members: `CHANGELOG.md`, `LICENSE.md`, `traefik`. The producer and runtime importer need matching explicit allowlists including the signed changelog, while only the executable is extracted; arbitrary regular files remain rejected.

kayg commented 2026-11-02 02:50:44 +00:00

Author

Owner

Copy link

J62 resumed on proxy-traefik-281 at edbaffba10c98f8090471d2bae1e054038f41e59. The separate infraternal unpublish <port|name> action is in the root command table; it reports the stopped URL and 10-second drain, returns one-line D303 missing / exit 4 when absent, and shares --json. This remains the orchestrator’s choice pending owner rename, as noted in the proxy module doc.

Pending local changes cover upstream archive allowlisting, mixed public/mesh listener separation, oversized log-record discard, and Pebble fixture diagnostics. The latest staging-ACME run on compute01 reaches the granted backend over HTTP but still serves Traefik’s default certificate (HTTPREQ /present=1, no adapter errors). I’m fixing the fixture resolver path and syncing the merged source before rerunning focused tests and scripts/verify fast.

J62 resumed on `proxy-traefik-281` at `edbaffba10c98f8090471d2bae1e054038f41e59`. The separate `infraternal unpublish <port|name>` action is in the root command table; it reports the stopped URL and 10-second drain, returns one-line D303 `missing` / exit 4 when absent, and shares `--json`. This remains the orchestrator’s choice pending owner rename, as noted in the proxy module doc. Pending local changes cover upstream archive allowlisting, mixed public/mesh listener separation, oversized log-record discard, and Pebble fixture diagnostics. The latest staging-ACME run on compute01 reaches the granted backend over HTTP but still serves Traefik’s default certificate (HTTPREQ `/present`=1, no adapter errors). I’m fixing the fixture resolver path and syncing the merged source before rerunning focused tests and `scripts/verify fast`.

kayg commented 2026-11-02 03:21:27 +00:00

Author

Owner

Copy link

API-disabled test discrepancy on proxy-traefik-281 at edbaffba10c98f8090471d2bae1e054038f41e59: the earlier 1/1 pass came from a reused component target with stale test artifacts. The copied source hashes matched the current worktree, but the result was not reproducible after clearing the package artifacts. Through compute01's build-slot, cargo clean -p infraternal-plugin-proxy removed 2,728 files (638.5 MiB); rebuilding and rerunning the same named test then failed as expected at config_tests.rs:69: assertion failed: !rendered.contains("[api]") (exit 101). The renderer still emits the [api] table. I am removing that table so Traefik's API/dashboard stay at their disabled defaults, then will rerun this test green.

API-disabled test discrepancy on `proxy-traefik-281` at `edbaffba10c98f8090471d2bae1e054038f41e59`: the earlier 1/1 pass came from a reused component target with stale test artifacts. The copied source hashes matched the current worktree, but the result was not reproducible after clearing the package artifacts. Through compute01's `build-slot`, `cargo clean -p infraternal-plugin-proxy` removed 2,728 files (638.5 MiB); rebuilding and rerunning the same named test then failed as expected at `config_tests.rs:69`: `assertion failed: !rendered.contains("[api]")` (exit 101). The renderer still emits the `[api]` table. I am removing that table so Traefik's API/dashboard stay at their disabled defaults, then will rerun this test green.

kayg commented 2026-11-02 03:30:39 +00:00

Author

Owner

Copy link

J62 command table choice (orchestrator decision, pending an owner rename): use the separate infraternal unpublish <port|name> command as the inverse of publish. It reports what stopped being reachable and allows the supervisor's 10-second drain. A missing target returns D303 missing (exit 4) with a one-line message; --json uses the standard command envelope. The root action table and module docs now record this choice, with CLI and controller coverage.

J62 command table choice (orchestrator decision, pending an owner rename): use the separate `infraternal unpublish <port|name>` command as the inverse of `publish`. It reports what stopped being reachable and allows the supervisor's 10-second drain. A missing target returns D303 `missing` (exit 4) with a one-line message; `--json` uses the standard command envelope. The root action table and module docs now record this choice, with CLI and controller coverage.

kayg commented 2026-11-02 03:38:38 +00:00

Author

Owner

Copy link

Pebble ACME root-cause finding: fresh namespace runs consistently serve the granted backend over HTTPS with an unverified curl, but return Traefik's default certificate after 45.5 seconds; the HTTPREQ fixture records one successful present call and no adapter errors. Traefik v3.7.17's ACME source configures both authoritative-name-server and recursive DNS propagation checks by default, while this test publishes its TXT record only through the local challenge server for a synthetic example.org name. I’m adding a test-only setting to skip the authoritative check while retaining recursive validation against the explicit local resolver; production keeps the default checks.

Pebble ACME root-cause finding: fresh namespace runs consistently serve the granted backend over HTTPS with an unverified curl, but return Traefik's default certificate after 45.5 seconds; the HTTPREQ fixture records one successful `present` call and no adapter errors. Traefik v3.7.17's ACME source configures both authoritative-name-server and recursive DNS propagation checks by default, while this test publishes its TXT record only through the local challenge server for a synthetic `example.org` name. I’m adding a test-only setting to skip the authoritative check while retaining recursive validation against the explicit local resolver; production keeps the default checks.

kayg commented 2026-11-02 03:41:23 +00:00

Author

Owner

Copy link

ACME test diagnosis correction: disabling Traefik's authoritative-name-server check exposed the next boundary. Debug logs show the challenge provider completes (HTTPREQ calls=2), then Pebble rejects validation because its DNS lookup goes to the host stub 127.0.0.53:53 and is refused. The Pebble v2.9.0 README documents -dnsserver <host:port>; the fixture used an unsupported PEBBLE_VA_DNS_SERVER environment variable. I’ll point Pebble at the isolated challenge DNS server with that flag. The ACME CA's DNS validation remains active.

ACME test diagnosis correction: disabling Traefik's authoritative-name-server check exposed the next boundary. Debug logs show the challenge provider completes (`HTTPREQ calls=2`), then Pebble rejects validation because its DNS lookup goes to the host stub `127.0.0.53:53` and is refused. The Pebble v2.9.0 README documents `-dnsserver <host:port>`; the fixture used an unsupported `PEBBLE_VA_DNS_SERVER` environment variable. I’ll point Pebble at the isolated challenge DNS server with that flag. The ACME CA's DNS validation remains active.

kayg commented 2026-11-02 03:46:21 +00:00

Author

Owner

Copy link

Pebble resolver flag fix confirmed ACME issuance: the served certificate now matches docs.node.example.org, and HTTPREQ records both present and cleanup. The remaining test failure is trust-chain verification: curl reports issuer CN=Pebble Intermediate CA 7c23a4. Pebble v2.9.0 documents that its ACME roots and intermediates are regenerated on every launch and exposes the current root at /roots/0; pebble.minica.pem only trusts Pebble's HTTPS service certificate. I’ll fetch the ephemeral issuance root over Pebble's verified management HTTPS endpoint and use it only for the staging curl checks.

Pebble resolver flag fix confirmed ACME issuance: the served certificate now matches `docs.node.example.org`, and HTTPREQ records both `present` and `cleanup`. The remaining test failure is trust-chain verification: curl reports issuer `CN=Pebble Intermediate CA 7c23a4`. Pebble v2.9.0 documents that its ACME roots and intermediates are regenerated on every launch and exposes the current root at `/roots/0`; `pebble.minica.pem` only trusts Pebble's HTTPS service certificate. I’ll fetch the ephemeral issuance root over Pebble's verified management HTTPS endpoint and use it only for the staging curl checks.

kayg commented 2026-11-02 03:47:36 +00:00

Author

Owner

Copy link

Mesh-IP redirect investigation: a rule limited to the mesh interface and mesh destination IP would still be unsafe when another server owns TCP 80/443 on that address. The nftables NAT reference states that redirect is DNAT to the local host and only applies in prerouting/output NAT chains; its prerouting example rewrites inbound port 22 before local delivery. Thus mesh-IP:80/443 packets would be rewritten to 10080/10443 before the existing socket is selected, even when the other server explicitly binds the mesh IP. This steals that server's mesh traffic. No redirect is implemented; the proxy keeps the high-port fallback whenever 80/443 are occupied. Reference: https://wiki.nftables.org/wiki-nftables/index.php/Performing_Network_Address_Translation_(NAT)

Mesh-IP redirect investigation: a rule limited to the mesh interface and mesh destination IP would still be unsafe when another server owns TCP 80/443 on that address. The nftables NAT reference states that `redirect` is DNAT to the local host and only applies in prerouting/output NAT chains; its prerouting example rewrites inbound port 22 before local delivery. Thus mesh-IP:80/443 packets would be rewritten to 10080/10443 before the existing socket is selected, even when the other server explicitly binds the mesh IP. This steals that server's mesh traffic. No redirect is implemented; the proxy keeps the high-port fallback whenever 80/443 are occupied. Reference: https://wiki.nftables.org/wiki-nftables/index.php/Performing_Network_Address_Translation_(NAT)

kayg commented 2026-11-02 03:56:17 +00:00

Author

Owner

Copy link

J62 progress: WIP commit 538ba1506 is pushed to proxy-traefik-281. Fresh cargo test -p infraternal-plugin-proxy passed on compute01 (68 unit tests plus the process test); scripts/release/traefik_component_test.py passed 6/6. The Pebble test now uses its documented -dnsserver flag and fetches its per-run issuance root from the verified local management endpoint. That run is queued behind all four compute01 slots (queue PID 8579); its log is /home/claude/.cache/codex-scratch/j62/proxy-pebble-root.log. Next: collect the e2e result, run the final gate, report static sizes, and clean the target.

J62 progress: WIP commit `538ba1506` is pushed to `proxy-traefik-281`. Fresh `cargo test -p infraternal-plugin-proxy` passed on compute01 (68 unit tests plus the process test); `scripts/release/traefik_component_test.py` passed 6/6. The Pebble test now uses its documented `-dnsserver` flag and fetches its per-run issuance root from the verified local management endpoint. That run is queued behind all four compute01 slots (queue PID 8579); its log is `/home/claude/.cache/codex-scratch/j62/proxy-pebble-root.log`. Next: collect the e2e result, run the final gate, report static sizes, and clean the target.

kayg commented 2026-11-02 04:08:34 +00:00

Author

Owner

Copy link

J62 resumed from pushed WIP 538ba1506, rebased onto current origin/dev 9f36dd8e0, and pushed the rewritten branch at 1572c89d8. The one lockfile conflict was resolved by retaining current-dev entries and the J62 component entries. The unpublish <port|name> decision, D303 missing result, and 10-second drain are already recorded in the module and earlier #281 comments. The last pushed-head Pebble root run passed 1/1; focused tests and scripts/verify fast still need fresh receipts on the rebased head. Next: run them through compute01 build-slot, report both static sizes, then remove the J62 target.

J62 resumed from pushed WIP `538ba1506`, rebased onto current `origin/dev` `9f36dd8e0`, and pushed the rewritten branch at `1572c89d8`. The one lockfile conflict was resolved by retaining current-dev entries and the J62 component entries. The `unpublish <port|name>` decision, D303 missing result, and 10-second drain are already recorded in the module and earlier #281 comments. The last pushed-head Pebble root run passed 1/1; focused tests and `scripts/verify fast` still need fresh receipts on the rebased head. Next: run them through compute01 `build-slot`, report both static sizes, then remove the J62 target.

kayg commented 2026-11-02 04:22:25 +00:00

Author

Owner

Copy link

J62 final-review findings on proxy-traefik-281 at 1572c89d8 (independent review against base 9f36dd8e0): seven important end-to-end gaps remain. I verified the first root cause in source: the release producer omits executable, which Rust ComponentManifest requires; the other findings trace to the publisher/downloader path mismatch, no HTTPS port allowance in restricted Mesh grants, imported artifacts not being reused offline, child lifetime not tied to the infraternal agent, stale listener address after accepted Mesh identity changes, and redirect middleware defaulting to port 443. The existing component unit tests and the namespace fixture do not cross these boundaries.

Plan and estimate: add one failing regression for each boundary, fix producer and published mirror keys, wire restricted Mesh grants to the exact proxy listener ports, reuse signature-verified imported/current artifacts without network, make proxy child lifetime crash-safe, reconcile listener addresses on accepted Mesh updates, and render the audience-specific HTTPS redirect port. Then rerun proxy/CLI/Mesh/agent/release tests and the namespace Pebble ACME test, run scripts/verify fast, report static sizes, commit the fixes atomically, push, and post exact receipts. Estimated active work: about 2 hours, excluding build-slot waits. The exact-head proxy run passed 69 unit tests plus 1 process test; CLI tests passed 5/5. The fresh namespace run is queued/running on compute01; no final-gate claim yet.

J62 final-review findings on `proxy-traefik-281` at `1572c89d8` (independent review against base `9f36dd8e0`): seven important end-to-end gaps remain. I verified the first root cause in source: the release producer omits `executable`, which Rust `ComponentManifest` requires; the other findings trace to the publisher/downloader path mismatch, no HTTPS port allowance in restricted Mesh grants, imported artifacts not being reused offline, child lifetime not tied to the infraternal agent, stale listener address after accepted Mesh identity changes, and redirect middleware defaulting to port 443. The existing component unit tests and the namespace fixture do not cross these boundaries. Plan and estimate: add one failing regression for each boundary, fix producer and published mirror keys, wire restricted Mesh grants to the exact proxy listener ports, reuse signature-verified imported/current artifacts without network, make proxy child lifetime crash-safe, reconcile listener addresses on accepted Mesh updates, and render the audience-specific HTTPS redirect port. Then rerun proxy/CLI/Mesh/agent/release tests and the namespace Pebble ACME test, run `scripts/verify fast`, report static sizes, commit the fixes atomically, push, and post exact receipts. Estimated active work: about 2 hours, excluding build-slot waits. The exact-head proxy run passed 69 unit tests plus 1 process test; CLI tests passed 5/5. The fresh namespace run is queued/running on compute01; no final-gate claim yet.

kayg commented 2026-11-02 04:46:07 +00:00

Author

Owner

Copy link

J62 progress on #281. Working worktree: proxy-traefik-281-final, based on origin/dev 9f36dd8e0; final push remains authorized only to proxy-traefik-281.

Test-first review reproductions on compute01:

• cargo test --locked --jobs 4 -p infraternal-plugin-proxy --lib: 68 passed, 6 failed for the expected redirect-port, mesh-address listener restart, offline imported bundle reuse, existing archive reuse, and abrupt parent-exit child ownership gaps. Log: /home/claude/.cache/codex-scratch/j62/proxy-review-red.log.
• cargo test --locked --jobs 4 -p infraternal-plugin-mesh --features agent proxy_listener_ports_are_limited_to_peers_with_tcp_ingress_grants --lib: 0 passed, 2 failed in nft and iptables as expected. Log: /home/claude/.cache/codex-scratch/j62/proxy-ingress-agent-red.log.

The first firewall invocation omitted --features agent and ran zero matching tests; it is not treated as evidence. Fixes are in progress for the five review boundaries. The installer now persists the original signed release metadata beside the digest-addressed executable so offline startup can reverify it without adding a signature format. Current VM scratch target: /home/claude/.cache/codex-scratch/j62/target. No final gate has run yet.

J62 progress on #281. Working worktree: `proxy-traefik-281-final`, based on `origin/dev` `9f36dd8e0`; final push remains authorized only to `proxy-traefik-281`. Test-first review reproductions on compute01: - `cargo test --locked --jobs 4 -p infraternal-plugin-proxy --lib`: 68 passed, 6 failed for the expected redirect-port, mesh-address listener restart, offline imported bundle reuse, existing archive reuse, and abrupt parent-exit child ownership gaps. Log: `/home/claude/.cache/codex-scratch/j62/proxy-review-red.log`. - `cargo test --locked --jobs 4 -p infraternal-plugin-mesh --features agent proxy_listener_ports_are_limited_to_peers_with_tcp_ingress_grants --lib`: 0 passed, 2 failed in nft and iptables as expected. Log: `/home/claude/.cache/codex-scratch/j62/proxy-ingress-agent-red.log`. The first firewall invocation omitted `--features agent` and ran zero matching tests; it is not treated as evidence. Fixes are in progress for the five review boundaries. The installer now persists the original signed release metadata beside the digest-addressed executable so offline startup can reverify it without adding a signature format. Current VM scratch target: `/home/claude/.cache/codex-scratch/j62/target`. No final gate has run yet.

kayg commented 2026-11-02 05:15:31 +00:00

Author

Owner

Copy link

J62 resumed on proxy-traefik-281 from pushed head 1572c89d8 in the current-dev worktree. The focused compute01 run completed: proxy unit tests reported 73 passed, 1 failed. The failure is download_tests::an_untrusted_manifest_never_triggers_an_archive_request: the offline installed-component lookup created components/ before there was any trusted signed metadata. I changed the lookup to validate an existing tree without creating it; next I am syncing that one-file fix and rerunning the regression and focused suite through build-slot.

Command-table choice for this slice: use separate infraternal unpublish <port|name> as the inverse of publish. It reports what stopped being reachable, triggers the supervisor's 10-second drain, supports --json, and a missing target returns one-line D303 missing / exit 4. This is the orchestrator's choice pending owner rename; the proxy module doc records it.

J62 resumed on `proxy-traefik-281` from pushed head `1572c89d8` in the current-dev worktree. The focused compute01 run completed: proxy unit tests reported 73 passed, 1 failed. The failure is `download_tests::an_untrusted_manifest_never_triggers_an_archive_request`: the offline installed-component lookup created `components/` before there was any trusted signed metadata. I changed the lookup to validate an existing tree without creating it; next I am syncing that one-file fix and rerunning the regression and focused suite through `build-slot`. Command-table choice for this slice: use separate `infraternal unpublish <port|name>` as the inverse of `publish`. It reports what stopped being reachable, triggers the supervisor's 10-second drain, supports `--json`, and a missing target returns one-line D303 `missing` / exit 4. This is the orchestrator's choice pending owner rename; the proxy module doc records it.

kayg commented 2026-11-02 05:18:32 +00:00

Author

Owner

Copy link

Focused regression after the change: download_tests::an_untrusted_manifest_never_triggers_an_archive_request now passes 1/1 on compute01 through build-slot. The lookup returns without creating components/ when the local store is absent; the untrusted manifest is rejected before archive fetch. I am rerunning the complete focused proxy/Mesh/agent-context/release set against this tree next.

Focused regression after the change: `download_tests::an_untrusted_manifest_never_triggers_an_archive_request` now passes 1/1 on compute01 through `build-slot`. The lookup returns without creating `components/` when the local store is absent; the untrusted manifest is rejected before archive fetch. I am rerunning the complete focused proxy/Mesh/agent-context/release set against this tree next.

kayg commented 2026-11-02 05:36:33 +00:00

Author

Owner

Copy link

J62 progress on proxy-traefik-281-final (base origin/dev 9f36dd8e0, latest pushed checkpoint 1572c89d8).

Two test-first findings are fixed in the local WIP:

• Mesh context transition Some(old) → None → Some(new) reused the stale listener. RED: losing_mesh_identity_and_rejoining_on_a_new_address_restarts_the_proxy_listener expected 10.254.0.9 to start, observed only 10.254.0.2; GREEN: 1 passed. On context loss the controller now drops the old supervisor and clears stale routes while retaining publication intent.
• Production ACME initialization created a zero-byte file, while the Pebble fixture uses valid JSON {}. Both new initialization and existing-empty-file RED tests observed [] instead of [123,125]; GREEN: the controller test subset reports 10 passed. The initializer now creates or repairs an empty owner-only file as {}.

Receipts on compute01:

• /home/claude/.cache/codex-scratch/j62/mesh-transition-red-2.log
• /home/claude/.cache/codex-scratch/j62/mesh-transition-green.log
• /home/claude/.cache/codex-scratch/j62/acme-init-red.log
• /home/claude/.cache/codex-scratch/j62/acme-existing-red.log
• /home/claude/.cache/codex-scratch/j62/controller-green.log

Next: run focused proxy/Mesh/agent/CLI/release checks, the namespace Pebble ACME test, then scripts/verify fast; report D243 sizes, split/push the WIP fixes atomically, post final receipts, and remove the J62 target. The separate unpublish <port|name> choice, D303 missing/exit-4 behavior, JSON support, and 10-second drain are already documented in the module and prior #281 comment.

J62 progress on `proxy-traefik-281-final` (base `origin/dev` `9f36dd8e0`, latest pushed checkpoint `1572c89d8`). Two test-first findings are fixed in the local WIP: - Mesh context transition `Some(old) → None → Some(new)` reused the stale listener. RED: `losing_mesh_identity_and_rejoining_on_a_new_address_restarts_the_proxy_listener` expected `10.254.0.9` to start, observed only `10.254.0.2`; GREEN: 1 passed. On context loss the controller now drops the old supervisor and clears stale routes while retaining publication intent. - Production ACME initialization created a zero-byte file, while the Pebble fixture uses valid JSON `{}`. Both new initialization and existing-empty-file RED tests observed `[]` instead of `[123,125]`; GREEN: the controller test subset reports 10 passed. The initializer now creates or repairs an empty owner-only file as `{}`. Receipts on compute01: - `/home/claude/.cache/codex-scratch/j62/mesh-transition-red-2.log` - `/home/claude/.cache/codex-scratch/j62/mesh-transition-green.log` - `/home/claude/.cache/codex-scratch/j62/acme-init-red.log` - `/home/claude/.cache/codex-scratch/j62/acme-existing-red.log` - `/home/claude/.cache/codex-scratch/j62/controller-green.log` Next: run focused proxy/Mesh/agent/CLI/release checks, the namespace Pebble ACME test, then `scripts/verify fast`; report D243 sizes, split/push the WIP fixes atomically, post final receipts, and remove the J62 target. The separate `unpublish <port|name>` choice, D303 missing/exit-4 behavior, JSON support, and 10-second drain are already documented in the module and prior #281 comment.

kayg commented 2026-11-02 06:03:22 +00:00

Author

Owner

Copy link

J62 progress on proxy-traefik-281-final, continuing from pushed proxy-traefik-281 checkpoint 1572c89d8 (base 9f36dd8e0). The local WIP now includes the review fixes and updates the Pebble namespace fixture to use the listener ports selected by the supervisor; local/VM hashes match for the fixture, runner, and controller.

The earlier queued package-test PID is gone. Current build-slot check: compute01 has 3 free slots and 75 GB free; dev-02 has all 3 slots busy. I am resubmitting the proxy package test detached through compute01 build-slot, reusing /home/claude/.cache/codex-scratch/j62/target (14 GB). No test or final-gate result is claimed yet. Next: collect focused package/Pebble receipts, run scripts/verify fast, report D243 size rows, push the finished changes, and post the exact gate output.

J62 progress on `proxy-traefik-281-final`, continuing from pushed `proxy-traefik-281` checkpoint `1572c89d8` (base `9f36dd8e0`). The local WIP now includes the review fixes and updates the Pebble namespace fixture to use the listener ports selected by the supervisor; local/VM hashes match for the fixture, runner, and controller. The earlier queued package-test PID is gone. Current build-slot check: compute01 has 3 free slots and 75 GB free; dev-02 has all 3 slots busy. I am resubmitting the proxy package test detached through compute01 build-slot, reusing `/home/claude/.cache/codex-scratch/j62/target` (14 GB). No test or final-gate result is claimed yet. Next: collect focused package/Pebble receipts, run `scripts/verify fast`, report D243 size rows, push the finished changes, and post the exact gate output.

kayg commented 2026-11-02 06:21:07 +00:00

Author

Owner

Copy link

J62 progress on proxy-traefik-281 (active worktree: proxy-traefik-281-final), based on pushed checkpoint 1572c89d8 / origin/dev 9f36dd8e0.

The resubmitted VM command cargo test -p infraternal-plugin-proxy completed on compute01: 77 library tests passed, the process redaction test passed, and the namespace Pebble test was ignored by its explicit fixture gate. Log: /home/claude/worktrees/J62-proxy-test-20261102.log.

A new failing regression then reproduced the rootless redirect-port issue: supervisor_tests::owned_supervisor_starts_on_first_exposure_drains_routes_and_keeps_acme_state exits 101 because dynamic config contains redirect port 443 while the selected listener is 10443. The supervisor detects ports after the controller has built the route and currently renders the stale fallback port. Log: /home/claude/worktrees/J62-proxy-redirect-red.log. I will update every runtime route to the selected audience-specific HTTPS port before rendering and preserve those values in the supervisor snapshot.

J62 progress on `proxy-traefik-281` (active worktree: `proxy-traefik-281-final`), based on pushed checkpoint `1572c89d8` / `origin/dev` `9f36dd8e0`. The resubmitted VM command `cargo test -p infraternal-plugin-proxy` completed on compute01: 77 library tests passed, the process redaction test passed, and the namespace Pebble test was ignored by its explicit fixture gate. Log: `/home/claude/worktrees/J62-proxy-test-20261102.log`. A new failing regression then reproduced the rootless redirect-port issue: `supervisor_tests::owned_supervisor_starts_on_first_exposure_drains_routes_and_keeps_acme_state` exits 101 because dynamic config contains redirect port `443` while the selected listener is `10443`. The supervisor detects ports after the controller has built the route and currently renders the stale fallback port. Log: `/home/claude/worktrees/J62-proxy-redirect-red.log`. I will update every runtime route to the selected audience-specific HTTPS port before rendering and preserve those values in the supervisor snapshot.

kayg commented 2026-11-02 07:18:20 +00:00

Author

Owner

Copy link

J62 final-gate finding on proxy-traefik-281 at e9a9b8a84: scripts/verify fast on dev-02 found that the workspace test and clippy builds both omitted the newly optional mesh exposure field in remote_supervisor_review_tests.rs. Exact compiler diagnostic: error[E0063]: missing field exposure in initializer of infraternal_plugin_mesh::dataplane::Config at line 286.

The static-build row passed and is report-only under D243: x86_64 22,539,952 bytes (+1,096,128); aarch64 20,551,144 bytes (+921,904). Full log: /home/claude/.cache/codex-scratch/j62-dev02/verify-fast-final.log.

Added exposure: None to the test fixture because this case does not publish a service. The correctly feature-gated regression passed 1/1 with cargo test --locked -p infraternal-agent --features remote,mesh --lib acknowledged_network_access_survives_superseded_worker_completion. Fix committed and pushed as 3372c85aa037d15e334ff7b1772f76d1f017ce17; the full gate is being rerun on that head.

J62 final-gate finding on `proxy-traefik-281` at `e9a9b8a84`: `scripts/verify fast` on dev-02 found that the workspace test and clippy builds both omitted the newly optional mesh `exposure` field in `remote_supervisor_review_tests.rs`. Exact compiler diagnostic: `error[E0063]: missing field exposure in initializer of infraternal_plugin_mesh::dataplane::Config` at line 286. The static-build row passed and is report-only under D243: x86_64 22,539,952 bytes (+1,096,128); aarch64 20,551,144 bytes (+921,904). Full log: `/home/claude/.cache/codex-scratch/j62-dev02/verify-fast-final.log`. Added `exposure: None` to the test fixture because this case does not publish a service. The correctly feature-gated regression passed 1/1 with `cargo test --locked -p infraternal-agent --features remote,mesh --lib acknowledged_network_access_survives_superseded_worker_completion`. Fix committed and pushed as `3372c85aa037d15e334ff7b1772f76d1f017ce17`; the full gate is being rerun on that head.

kayg commented 2026-11-02 07:34:54 +00:00

Author

Owner

Copy link

J62 progress on proxy-traefik-281. The second full gate at 3372c85aa confirmed two independent findings: Clippy rejected duplicate D303 mappings for missing and not_found, and default_build_registers_no_unbuilt_command rejected publish even though plugin-proxy is enabled in the shipping default feature set. The workspace tests also confirmed the optional mesh exposure fixture fix.

Both fixes are now pushed as separate commits: b326417a9 merges the D303 aliases and tests exit 4 for both; 1fa66b199 expects publish and unpublish when proxy is compiled, while preserving the compiled-out case. Focused evidence on compute01: default_build_registers_no_unbuilt_command passed 1/1; missing_component_action_results_use_the_d303_not_found_exit passed 1/1; package Clippy and fmt check passed. Log: /home/claude/.cache/codex-scratch/j62/final-touched-checks.log.

D243 static rows passed as report-only: x86_64 22,539,952 bytes (+1,096,128); aarch64 20,551,144 bytes (+921,904). Second gate log: /home/claude/.cache/codex-scratch/j62-dev02/verify-fast-final-fixed.log.

Next: rerun scripts/verify fast on head 1fa66b1995e8bb69f447f9a130a2c8c6e0635157 through an available build slot.

J62 progress on `proxy-traefik-281`. The second full gate at `3372c85aa` confirmed two independent findings: Clippy rejected duplicate D303 mappings for `missing` and `not_found`, and `default_build_registers_no_unbuilt_command` rejected `publish` even though `plugin-proxy` is enabled in the shipping default feature set. The workspace tests also confirmed the optional mesh exposure fixture fix. Both fixes are now pushed as separate commits: `b326417a9` merges the D303 aliases and tests exit 4 for both; `1fa66b199` expects `publish` and `unpublish` when proxy is compiled, while preserving the compiled-out case. Focused evidence on compute01: `default_build_registers_no_unbuilt_command` passed 1/1; `missing_component_action_results_use_the_d303_not_found_exit` passed 1/1; package Clippy and fmt check passed. Log: `/home/claude/.cache/codex-scratch/j62/final-touched-checks.log`. D243 static rows passed as report-only: x86_64 22,539,952 bytes (+1,096,128); aarch64 20,551,144 bytes (+921,904). Second gate log: `/home/claude/.cache/codex-scratch/j62-dev02/verify-fast-final-fixed.log`. Next: rerun `scripts/verify fast` on head `1fa66b1995e8bb69f447f9a130a2c8c6e0635157` through an available build slot.

kayg commented 2026-11-02 08:21:03 +00:00

Author

Owner

Copy link

J62 resume check: the exact unmodified task base is 9f36dd8e0a. I created a detached VM worktree at that SHA and submitted the full agent_daemon integration target through compute01 build-slot (PID 2343516), reusing the existing J62 target. Log: /home/claude/.cache/codex-scratch/j62/agent-daemon-origin-base.log. Both VM queues were checked before submission; compute01 had one free slot and dev-02 was full. Next: collect the baseline result, then rerun the final head gate if appropriate.

J62 resume check: the exact unmodified task base is 9f36dd8e0a5e9e799cd3264cb6e1e7605071c5e2. I created a detached VM worktree at that SHA and submitted the full agent_daemon integration target through compute01 build-slot (PID 2343516), reusing the existing J62 target. Log: /home/claude/.cache/codex-scratch/j62/agent-daemon-origin-base.log. Both VM queues were checked before submission; compute01 had one free slot and dev-02 was full. Next: collect the baseline result, then rerun the final head gate if appropriate.

kayg commented 2026-11-02 08:26:42 +00:00

Author

Owner

Copy link

Baseline finding on unmodified J62 base 9f36dd8e0a: the full agent_daemon integration target passed all 11 tests, including real_binary_rootless_cleanup_survives_sender_disconnect_before_readiness (34.66 s; log /home/claude/.cache/codex-scratch/j62/agent-daemon-origin-base.log). This narrows the prior full-gate failure to an intermittent case or a J62 interaction. I am running the same full target on head 1fa66b1995e8bb69f447f9a130a2c8c6e0635157; log /home/claude/.cache/codex-scratch/j62/agent-daemon-j62-full-target.log.

Baseline finding on unmodified J62 base 9f36dd8e0a5e9e799cd3264cb6e1e7605071c5e2: the full `agent_daemon` integration target passed all 11 tests, including `real_binary_rootless_cleanup_survives_sender_disconnect_before_readiness` (34.66 s; log `/home/claude/.cache/codex-scratch/j62/agent-daemon-origin-base.log`). This narrows the prior full-gate failure to an intermittent case or a J62 interaction. I am running the same full target on head `1fa66b1995e8bb69f447f9a130a2c8c6e0635157`; log `/home/claude/.cache/codex-scratch/j62/agent-daemon-j62-full-target.log`.

kayg commented 2026-11-02 08:29:18 +00:00

Author

Owner

Copy link

J62 comparison result: the full agent_daemon integration target passes on both exact task base 9f36dd8e0a5e9e799cd3264cb6e1e7605071c5e2 (11/11, 34.66 s) and J62 head 1fa66b1995e8bb69f447f9a130a2c8c6e0635157 (11/11, 34.67 s). The one resource_unavailable from the earlier full gate did not reproduce in either full target run. Logs: /home/claude/.cache/codex-scratch/j62/agent-daemon-origin-base.log and /home/claude/.cache/codex-scratch/j62/agent-daemon-j62-full-target.log. I will run a fresh fast gate on the J62 head and record its exact result.

J62 comparison result: the full `agent_daemon` integration target passes on both exact task base `9f36dd8e0a5e9e799cd3264cb6e1e7605071c5e2` (11/11, 34.66 s) and J62 head `1fa66b1995e8bb69f447f9a130a2c8c6e0635157` (11/11, 34.67 s). The one `resource_unavailable` from the earlier full gate did not reproduce in either full target run. Logs: `/home/claude/.cache/codex-scratch/j62/agent-daemon-origin-base.log` and `/home/claude/.cache/codex-scratch/j62/agent-daemon-j62-full-target.log`. I will run a fresh fast gate on the J62 head and record its exact result.

kayg commented 2026-11-02 09:16:14 +00:00

Author

Owner

Copy link

J62 fresh gate finding on head 1fa66b1995e8bb69f447f9a130a2c8c6e0635157: the scripts/verify fast run on dev-02 completed its summary. Verbatim Rust gate table:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | FAIL (101)   |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+

The workspace test row reported remote_service::tests::link_failure_does_not_wait_forever_for_blocking_work and remote_service::tests::polling_panic_reaches_bounded_shutdown; both hit their 5-second receive timeout. crates/infraternal/src/remote_service.rs is unchanged from the task base. I will compare those cases against the exact base and then run the required x86_64 Pebble namespace receipt when compute01 has disk headroom.

D243 size rows passed report-only: x86_64 22,539,952 bytes (+1,096,128); aarch64 20,551,144 bytes (+921,904). Full log: /home/claude/.cache/codex-scratch/j62-dev02/verify-fast-resumed.log.

J62 fresh gate finding on head `1fa66b1995e8bb69f447f9a130a2c8c6e0635157`: the `scripts/verify fast` run on dev-02 completed its summary. Verbatim Rust gate table: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | FAIL (101) | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | static builds | PASS | +---------------------------+--------------+ ``` The workspace test row reported `remote_service::tests::link_failure_does_not_wait_forever_for_blocking_work` and `remote_service::tests::polling_panic_reaches_bounded_shutdown`; both hit their 5-second receive timeout. `crates/infraternal/src/remote_service.rs` is unchanged from the task base. I will compare those cases against the exact base and then run the required x86_64 Pebble namespace receipt when compute01 has disk headroom. D243 size rows passed report-only: x86_64 22,539,952 bytes (+1,096,128); aarch64 20,551,144 bytes (+921,904). Full log: `/home/claude/.cache/codex-scratch/j62-dev02/verify-fast-resumed.log`.

kayg commented 2026-11-02 09:40:09 +00:00

Author

Owner

Copy link

J62 follow-up evidence on head 1fa66b1995e8bb69f447f9a130a2c8c6e0635157:

• On exact base 9f36dd8e0a5e9e799cd3264cb6e1e7605071c5e2, cargo test --locked --jobs 4 -p infraternal --bin infraternal passed: 121 passed, 0 failed, 1 ignored; BASE_BINARY_TEST_EXIT=0. Log: /home/claude/.cache/codex-scratch/j62/remote-service-origin-base.log.
• The same command on J62 passed: 126 passed, 0 failed, 1 ignored; J62_BINARY_TEST_EXIT=0. Log: /home/claude/.cache/codex-scratch/j62/remote-service-j62-x86.log.
• The latest-head isolated mesh DNS-01 namespace test passed through scripts/test-proxy-pebble.sh: 1 passed, 0 failed; PEBBLE_E2E_EXIT=0. Log: /home/claude/.cache/codex-scratch/j62/proxy-pebble-final.log.

The two 5-second remote_service timeouts from the full dev-02 gate did not reproduce in either x86 binary suite; crates/infraternal/src/remote_service.rs is unchanged on this branch. The dev-02 gate remains recorded as a failed test row, while static builds passed report-only. I will rerun the full gate on x86 when compute01 has enough free disk; it currently has 42 GB free, and J62 has no active VM process.

J62 follow-up evidence on head `1fa66b1995e8bb69f447f9a130a2c8c6e0635157`: - On exact base `9f36dd8e0a5e9e799cd3264cb6e1e7605071c5e2`, `cargo test --locked --jobs 4 -p infraternal --bin infraternal` passed: 121 passed, 0 failed, 1 ignored; `BASE_BINARY_TEST_EXIT=0`. Log: `/home/claude/.cache/codex-scratch/j62/remote-service-origin-base.log`. - The same command on J62 passed: 126 passed, 0 failed, 1 ignored; `J62_BINARY_TEST_EXIT=0`. Log: `/home/claude/.cache/codex-scratch/j62/remote-service-j62-x86.log`. - The latest-head isolated mesh DNS-01 namespace test passed through `scripts/test-proxy-pebble.sh`: 1 passed, 0 failed; `PEBBLE_E2E_EXIT=0`. Log: `/home/claude/.cache/codex-scratch/j62/proxy-pebble-final.log`. The two 5-second `remote_service` timeouts from the full dev-02 gate did not reproduce in either x86 binary suite; `crates/infraternal/src/remote_service.rs` is unchanged on this branch. The dev-02 gate remains recorded as a failed test row, while static builds passed report-only. I will rerun the full gate on x86 when compute01 has enough free disk; it currently has 42 GB free, and J62 has no active VM process.

kayg commented 2026-11-02 10:51:49 +00:00

Author

Owner

Copy link

J62 slice 1 is complete on pushed branch proxy-traefik-281 at 1fa66b1995e8bb69f447f9a130a2c8c6e0635157.

The proxy module records the orchestrator's CLI choice, pending any owner rename: infraternal unpublish <port|name> is the separate inverse of publish. It accepts a local port or exact published name, reports what stopped being reachable, triggers the 10-second drain when the last publication is removed, supports --json, and returns one-line D303 missing / exit 4 for an absent target. Tests cover both selector forms and missing results.

The module also records interim DnsTokenCustody file custody under owner-only permissions, replaceable by node-key sealing or OpenBao without changing callers. Default per-machine wildcard keys remain on their node; flat exact-service certificate names appear in CT logs. The unsafe nftables redirect case is documented; no redirect is installed.

Verification on compute01 at this head:

• cargo test --locked --jobs 4 -p infraternal-plugin-proxy: 77 passed, 0 failed; process test 1 passed; the namespace test is separately run against Pebble.
• scripts/test-proxy-pebble.sh: namespace serving test 1 passed, 0 failed, staging ACME only.
• The first default-thread full gate run had two timing-sensitive remote_client tests fail under load. Both passed when run individually with the remote feature; the full gate passed with RUST_TEST_THREADS=1.
• Static sizes are report-only under D243: x86_64 22,548,432 bytes (+1,104,608); aarch64 20,557,624 bytes (+928,384).
• Final scripts/verify fast result: VERIFY_FAST_EXIT=0. Log: /home/claude/.cache/codex-scratch/j62/verify-fast-threads1-at-1fa66b1.log.

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+

J62 slice 1 is complete on pushed branch `proxy-traefik-281` at `1fa66b1995e8bb69f447f9a130a2c8c6e0635157`. The proxy module records the orchestrator's CLI choice, pending any owner rename: `infraternal unpublish <port|name>` is the separate inverse of `publish`. It accepts a local port or exact published name, reports what stopped being reachable, triggers the 10-second drain when the last publication is removed, supports `--json`, and returns one-line D303 `missing` / exit 4 for an absent target. Tests cover both selector forms and missing results. The module also records interim `DnsTokenCustody` file custody under owner-only permissions, replaceable by node-key sealing or OpenBao without changing callers. Default per-machine wildcard keys remain on their node; flat exact-service certificate names appear in CT logs. The unsafe nftables redirect case is documented; no redirect is installed. Verification on compute01 at this head: - `cargo test --locked --jobs 4 -p infraternal-plugin-proxy`: 77 passed, 0 failed; process test 1 passed; the namespace test is separately run against Pebble. - `scripts/test-proxy-pebble.sh`: namespace serving test 1 passed, 0 failed, staging ACME only. - The first default-thread full gate run had two timing-sensitive `remote_client` tests fail under load. Both passed when run individually with the `remote` feature; the full gate passed with `RUST_TEST_THREADS=1`. - Static sizes are report-only under D243: x86_64 22,548,432 bytes (+1,104,608); aarch64 20,557,624 bytes (+928,384). - Final `scripts/verify fast` result: `VERIFY_FAST_EXIT=0`. Log: `/home/claude/.cache/codex-scratch/j62/verify-fast-threads1-at-1fa66b1.log`. ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | static builds | PASS | +---------------------------+--------------+ ```

kayg commented 2026-11-02 11:03:09 +00:00

Author

Owner

Copy link

J99 independent review started. Target: proxy-traefik-281 at 1fa66b1995e8bb69f447f9a130a2c8c6e0635157; comparison ref: current origin/dev at 16bf736c7cc6b474474d6b148077b516b0893cf1.

Plan: inspect the acquisition, supervisor/config, Mesh exposure, DNS-token custody, certificate, and CLI paths against D316 and the owner answers in this issue/#289; run a focused test only if needed to confirm a concrete finding; measure and attribute static size against current origin/dev by crate; post the numbered verdict with file/line evidence. Estimate: about 60 minutes active review and measurement, excluding VM build-slot waits.

J99 independent review started. Target: `proxy-traefik-281` at `1fa66b1995e8bb69f447f9a130a2c8c6e0635157`; comparison ref: current `origin/dev` at `16bf736c7cc6b474474d6b148077b516b0893cf1`. Plan: inspect the acquisition, supervisor/config, Mesh exposure, DNS-token custody, certificate, and CLI paths against D316 and the owner answers in this issue/#289; run a focused test only if needed to confirm a concrete finding; measure and attribute static size against current `origin/dev` by crate; post the numbered verdict with file/line evidence. Estimate: about 60 minutes active review and measurement, excluding VM build-slot waits.

kayg commented 2026-11-02 11:19:34 +00:00

Author

Owner

Copy link

J99 review finding 1 — P1 — Mesh grant revocations are acknowledged before Traefik applies them.

Evidence: crates/plugins/proxy/src/controller.rs:675-690 rebuilds routes and source allowlists from the newly accepted Mesh context. crates/plugins/proxy/src/supervisor.rs:784-802 returns success after write_dynamic_snapshot; :995-1000 only renders and atomically replaces the watched file. crates/plugins/proxy/src/config.rs:206-208 enables the asynchronous file provider, and there is no applied-config acknowledgment or fail-closed listener transition on this path.

Impact: after a peer's service grant is revoked, its prior router and IP allowlist can remain active until Traefik reloads the file (or remain active if that reload is rejected). The caller has already received success, so the old grant can still reach the service after revocation. This misses the fail-closed behavior described in the #281 plan for authorization changes.

J99 review finding 1 — P1 — Mesh grant revocations are acknowledged before Traefik applies them. Evidence: `crates/plugins/proxy/src/controller.rs:675-690` rebuilds routes and source allowlists from the newly accepted Mesh context. `crates/plugins/proxy/src/supervisor.rs:784-802` returns success after `write_dynamic_snapshot`; `:995-1000` only renders and atomically replaces the watched file. `crates/plugins/proxy/src/config.rs:206-208` enables the asynchronous file provider, and there is no applied-config acknowledgment or fail-closed listener transition on this path. Impact: after a peer's service grant is revoked, its prior router and IP allowlist can remain active until Traefik reloads the file (or remain active if that reload is rejected). The caller has already received success, so the old grant can still reach the service after revocation. This misses the fail-closed behavior described in the #281 plan for authorization changes.

kayg commented 2026-11-02 11:19:35 +00:00

Author

Owner

Copy link

J99 review finding 2 — P2 — Rootless mixed Mesh routes use listener ports the Mesh firewall does not allow.

Evidence: crates/plugins/proxy/src/config.rs:131-141 selects 10080/10443 for rootless listeners. With public and Mesh routes together, detect_mixed_listener_ports skips that pair and selects 10081/10444 (:148-170); the supervisor installs those selected ports and writes them into routes (crates/plugins/proxy/src/supervisor.rs:906-924). Both Mesh firewall backends add only [80, 443, 10080, 10443] as proxy ingress ports (crates/plugins/mesh/src/firewall.rs:16,359-369; crates/plugins/mesh/src/iptables.rs:21,345-365).

Impact: in the rootless mixed-public/Mesh case, published Mesh routes are advertised on 10081/10444 but Mesh ingress grants open only the fixed ports. Mesh peers therefore cannot reach those routes. The ingress rule needs to cover the selected listener pair.

J99 review finding 2 — P2 — Rootless mixed Mesh routes use listener ports the Mesh firewall does not allow. Evidence: `crates/plugins/proxy/src/config.rs:131-141` selects 10080/10443 for rootless listeners. With public and Mesh routes together, `detect_mixed_listener_ports` skips that pair and selects 10081/10444 (`:148-170`); the supervisor installs those selected ports and writes them into routes (`crates/plugins/proxy/src/supervisor.rs:906-924`). Both Mesh firewall backends add only `[80, 443, 10080, 10443]` as proxy ingress ports (`crates/plugins/mesh/src/firewall.rs:16,359-369`; `crates/plugins/mesh/src/iptables.rs:21,345-365`). Impact: in the rootless mixed-public/Mesh case, published Mesh routes are advertised on 10081/10444 but Mesh ingress grants open only the fixed ports. Mesh peers therefore cannot reach those routes. The ingress rule needs to cover the selected listener pair.

kayg commented 2026-11-02 11:23:29 +00:00

Author

Owner

Copy link

J99 review finding 3 — P2 — Switching between Mesh-only, public-only, and mixed exposure scopes kills active proxy connections.

Evidence: crates/plugins/proxy/src/supervisor.rs:784-801 restarts the child whenever listener_scope changes. restart_for_listener_scope stops the old child before starting the new one (:962-981), and TraefikChild::stop kills and reaps it (:460-465). A scope transition occurs when adding or removing the last publication for either audience (:701-716).

Impact: publishing or unpublishing a public route while Mesh routes remain, or vice versa, terminates every connection through the proxy, including long-lived WebSockets. That conflicts with the #281 engine rationale that configuration changes must preserve existing connections. The dynamic route-update path preserves them; listener-scope transitions do not.

J99 review finding 3 — P2 — Switching between Mesh-only, public-only, and mixed exposure scopes kills active proxy connections. Evidence: `crates/plugins/proxy/src/supervisor.rs:784-801` restarts the child whenever `listener_scope` changes. `restart_for_listener_scope` stops the old child before starting the new one (`:962-981`), and `TraefikChild::stop` kills and reaps it (`:460-465`). A scope transition occurs when adding or removing the last publication for either audience (`:701-716`). Impact: publishing or unpublishing a public route while Mesh routes remain, or vice versa, terminates every connection through the proxy, including long-lived WebSockets. That conflicts with the #281 engine rationale that configuration changes must preserve existing connections. The dynamic route-update path preserves them; listener-scope transitions do not.

kayg referenced this issue 2026-11-02 12:59:05 +00:00

Head-to-head: infraternal mesh vs NetBird/Tailscale, infraternal sessions vs tmux/Zellij #260

kayg commented 2026-11-02 13:01:44 +00:00

Author

Owner

Copy link

Round 1 verdict: REQUEST CHANGES

Target: proxy-traefik-281 at 1fa66b1995e8bb69f447f9a130a2c8c6e0635157. The review comparison snapshot was origin/dev at 16bf736c7cc6b474474d6b148077b516b0893cf1.

1. P1 — Mesh grant revocations are acknowledged before Traefik applies them. controller.rs:675-690 rebuilds the routes and source allowlists from the accepted Mesh context. supervisor.rs:784-802,995-1000 returns success after rendering and atomically replacing the watched file; config.rs:206-208 enables Traefik's asynchronous file-provider reload. There is no applied-config acknowledgement or fail-closed transition. A revoked peer can therefore continue using the prior router and allowlist until reload, or longer if reload is rejected.

2. P2 — Rootless mixed Mesh routes select ports that the Mesh firewall does not allow. config.rs:131-170 selects 10081/10444 when public and Mesh routes coexist; supervisor.rs:906-924 installs those ports in the routes. Both firewall backends permit only 80, 443, 10080, and 10443 (firewall.rs:16,359-369; iptables.rs:21,345-365). Mesh peers cannot reach the mixed-scope Mesh routes.

3. P2 — Exposure-scope changes kill active proxy connections. supervisor.rs:701-716,784-801 restarts Traefik when the listener scope changes; supervisor.rs:962-981,460-465 stops and kills the old child before starting the replacement. Publishing or unpublishing the last route for one audience while another remains can terminate active requests and WebSockets.

Static size, x86_64 musl, default infraternal binary: canonical scripts/static-cargo build --locked --release --package infraternal --target x86_64-unknown-linux-musl measured 22,413,328 bytes at the review baseline and 22,548,432 bytes at the target, a +135,104-byte delta.

cargo bloat --release --target x86_64-unknown-linux-musl --package infraternal --bin infraternal --crates -n 0 gave approximate LTO .text attribution (its output warns these values are estimates). Largest increases: infraternal_plugin_proxy +168.5 KiB; std +102.4 KiB; serde_json +33.1 KiB; miniz_oxide +21.8 KiB; tar +13.7 KiB; flate2 +7.7 KiB; infraternal_plugin_host_facts +6.5 KiB; serde_core +5.7 KiB; infraternal_plugin_mesh +5.2 KiB; crc32fast +3.6 KiB; infraternal_plugin_canvas +3.2 KiB; infraternal_plugin_api +1.3 KiB. Largest reductions: infraternal_agent -115.5 KiB; infraternal_server -102.4 KiB; axum -20.5 KiB; tungstenite -18.6 KiB; tokio -12.5 KiB; infraternal -12.3 KiB; infraternal_plugin_ssh -6.7 KiB. The reported total .text section rounds from 16.3 to 16.4 MiB; crate attribution is approximate and does not sum exactly under LTO.

No additional actionable findings in the reviewed acquisition/extraction, proxy hardening, DNS-token custody, certificate scope, or D303/D314 CLI paths. No tests were run; the findings are supported by the code paths above, and the review brief limits test runs to confirming findings.

Round 1 verdict: **REQUEST CHANGES** Target: `proxy-traefik-281` at `1fa66b1995e8bb69f447f9a130a2c8c6e0635157`. The review comparison snapshot was `origin/dev` at `16bf736c7cc6b474474d6b148077b516b0893cf1`. 1. **P1 — Mesh grant revocations are acknowledged before Traefik applies them.** `controller.rs:675-690` rebuilds the routes and source allowlists from the accepted Mesh context. `supervisor.rs:784-802,995-1000` returns success after rendering and atomically replacing the watched file; `config.rs:206-208` enables Traefik's asynchronous file-provider reload. There is no applied-config acknowledgement or fail-closed transition. A revoked peer can therefore continue using the prior router and allowlist until reload, or longer if reload is rejected. 2. **P2 — Rootless mixed Mesh routes select ports that the Mesh firewall does not allow.** `config.rs:131-170` selects 10081/10444 when public and Mesh routes coexist; `supervisor.rs:906-924` installs those ports in the routes. Both firewall backends permit only 80, 443, 10080, and 10443 (`firewall.rs:16,359-369`; `iptables.rs:21,345-365`). Mesh peers cannot reach the mixed-scope Mesh routes. 3. **P2 — Exposure-scope changes kill active proxy connections.** `supervisor.rs:701-716,784-801` restarts Traefik when the listener scope changes; `supervisor.rs:962-981,460-465` stops and kills the old child before starting the replacement. Publishing or unpublishing the last route for one audience while another remains can terminate active requests and WebSockets. Static size, x86_64 musl, default `infraternal` binary: canonical `scripts/static-cargo build --locked --release --package infraternal --target x86_64-unknown-linux-musl` measured **22,413,328 bytes** at the review baseline and **22,548,432 bytes** at the target, a **+135,104-byte** delta. `cargo bloat --release --target x86_64-unknown-linux-musl --package infraternal --bin infraternal --crates -n 0` gave approximate LTO `.text` attribution (its output warns these values are estimates). Largest increases: `infraternal_plugin_proxy` +168.5 KiB; `std` +102.4 KiB; `serde_json` +33.1 KiB; `miniz_oxide` +21.8 KiB; `tar` +13.7 KiB; `flate2` +7.7 KiB; `infraternal_plugin_host_facts` +6.5 KiB; `serde_core` +5.7 KiB; `infraternal_plugin_mesh` +5.2 KiB; `crc32fast` +3.6 KiB; `infraternal_plugin_canvas` +3.2 KiB; `infraternal_plugin_api` +1.3 KiB. Largest reductions: `infraternal_agent` -115.5 KiB; `infraternal_server` -102.4 KiB; `axum` -20.5 KiB; `tungstenite` -18.6 KiB; `tokio` -12.5 KiB; `infraternal` -12.3 KiB; `infraternal_plugin_ssh` -6.7 KiB. The reported total `.text` section rounds from 16.3 to 16.4 MiB; crate attribution is approximate and does not sum exactly under LTO. No additional actionable findings in the reviewed acquisition/extraction, proxy hardening, DNS-token custody, certificate scope, or D303/D314 CLI paths. No tests were run; the findings are supported by the code paths above, and the review brief limits test runs to confirming findings.

kayg commented 2026-11-02 13:09:28 +00:00

Author

Owner

Copy link

J62b started on branch proxy-traefik-281-j62b, based on origin/proxy-traefik-281 at 1fa66b1995e8bb69f447f9a130a2c8c6e0635157.

Plan (estimated 120 minutes active work, excluding VM build-slot waits):

1. Add a regression test for mesh revocation racing asynchronous config reload; block the revoked peer before applying, confirm the exact generated config is active before success, and deny the route on timeout. Commit separately.
2. Define listener ports once and consume that set from route config and both firewall backends. Add a produced-port coverage test. Commit separately.
3. Keep Traefik entrypoints stable while route scope changes; add a real long-lived WebSocket test across publish/unpublish of another route. Commit separately.

I will run the proxy integration tests and scripts/verify fast on whichever VM has the available build slot, push only this branch, and report the exact gate output here.

J62b started on branch `proxy-traefik-281-j62b`, based on `origin/proxy-traefik-281` at `1fa66b1995e8bb69f447f9a130a2c8c6e0635157`. Plan (estimated 120 minutes active work, excluding VM build-slot waits): 1. Add a regression test for mesh revocation racing asynchronous config reload; block the revoked peer before applying, confirm the exact generated config is active before success, and deny the route on timeout. Commit separately. 2. Define listener ports once and consume that set from route config and both firewall backends. Add a produced-port coverage test. Commit separately. 3. Keep Traefik entrypoints stable while route scope changes; add a real long-lived WebSocket test across publish/unpublish of another route. Commit separately. I will run the proxy integration tests and `scripts/verify fast` on whichever VM has the available build slot, push only this branch, and report the exact gate output here.

kayg commented 2026-11-02 13:33:05 +00:00

Author

Owner

Copy link

J62b finding 1 evidence: the new test mesh_grant_revocation_removes_the_route_before_narrowing_its_allowlist fails on base 1fa66b1995e8bb69f447f9a130a2c8c6e0635157 before production changes. The fake loopback provider confirms the supervisor never requests an applied-config generation after replacing a narrowed Mesh grant:

thread 'supervisor_tests::mesh_grant_revocation_removes_the_route_before_narrowing_its_allowlist' panicked at crates/plugins/proxy/src/supervisor_tests.rs:467:14:
supervisor must await the route-removal generation: Timeout
...
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 77 filtered out
P1_RED_EXIT=101

J62b finding 1 evidence: the new test `mesh_grant_revocation_removes_the_route_before_narrowing_its_allowlist` fails on base `1fa66b1995e8bb69f447f9a130a2c8c6e0635157` before production changes. The fake loopback provider confirms the supervisor never requests an applied-config generation after replacing a narrowed Mesh grant: ```text thread 'supervisor_tests::mesh_grant_revocation_removes_the_route_before_narrowing_its_allowlist' panicked at crates/plugins/proxy/src/supervisor_tests.rs:467:14: supervisor must await the route-removal generation: Timeout ... test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 77 filtered out P1_RED_EXIT=101 ```

kayg commented 2026-11-02 14:45:33 +00:00

Author

Owner

Copy link

J62b start, continuing the review fixes on branch proxy-traefik-281-j62b, based on 1fa66b1995e8bb69f447f9a130a2c8c6e0635157.

Plan (estimate: 3 hours active work, excluding VM build-slot waits):

1. Finish P2 by using one shared proxy port definition in config and both Mesh firewall backends; run the existing failing regression test and package tests.
2. Add the real-Traefik long-lived WebSocket regression for route A across publish/unpublish of route B, observe it fail, then keep entrypoints stable and update only dynamic routes.
3. Run proxy integration tests and scripts/verify fast on the final head, then push this branch and post the final evidence.

J62b start, continuing the review fixes on branch `proxy-traefik-281-j62b`, based on `1fa66b1995e8bb69f447f9a130a2c8c6e0635157`. Plan (estimate: 3 hours active work, excluding VM build-slot waits): 1. Finish P2 by using one shared proxy port definition in config and both Mesh firewall backends; run the existing failing regression test and package tests. 2. Add the real-Traefik long-lived WebSocket regression for route A across publish/unpublish of route B, observe it fail, then keep entrypoints stable and update only dynamic routes. 3. Run proxy integration tests and `scripts/verify fast` on the final head, then push this branch and post the final evidence.

kayg commented 2026-11-02 14:59:08 +00:00

Author

Owner

Copy link

J62b finding 2 RED/GREEN evidence (commit 0068a441c): config_tests::every_mixed_listener_port_is_allowed_by_the_mesh_firewall_backends failed before the fix on missing ingress port 10081. After the fix, cargo test --jobs 3 -p infraternal-mesh -p infraternal-plugin-mesh -p infraternal-plugin-proxy on dev-02 passed; the proxy package reported 81 passed, 0 failed, and the new coverage test passed. Shared listener-pair/range definitions now drive config selection, nftables and iptables.

J62b finding 2 RED/GREEN evidence (commit `0068a441c`): `config_tests::every_mixed_listener_port_is_allowed_by_the_mesh_firewall_backends` failed before the fix on missing ingress port `10081`. After the fix, `cargo test --jobs 3 -p infraternal-mesh -p infraternal-plugin-mesh -p infraternal-plugin-proxy` on dev-02 passed; the proxy package reported 81 passed, 0 failed, and the new coverage test passed. Shared listener-pair/range definitions now drive config selection, nftables and iptables.

kayg commented 2026-11-02 15:22:47 +00:00

Author

Owner

Copy link

J62b progress: P2 is committed as 0068a441c; the Mesh/proxy package suite passed on dev-02. P3’s test-only long-lived WebSocket scenario is in the local worktree and synced to compute01. The RED run uses scripts/test-proxy-pebble.sh, is waiting in compute01’s build-slot queue, and writes /home/claude/.cache/codex-scratch/j62b/pebble-red.log. The compute01 worktree target is /home/claude/worktrees/j62b/target. Next: confirm the WebSocket failure, implement stable entrypoints/dynamic-only updates, then rerun the integration green.

J62b progress: P2 is committed as `0068a441c`; the Mesh/proxy package suite passed on dev-02. P3’s test-only long-lived WebSocket scenario is in the local worktree and synced to compute01. The RED run uses `scripts/test-proxy-pebble.sh`, is waiting in compute01’s build-slot queue, and writes `/home/claude/.cache/codex-scratch/j62b/pebble-red.log`. The compute01 worktree target is `/home/claude/worktrees/j62b/target`. Next: confirm the WebSocket failure, implement stable entrypoints/dynamic-only updates, then rerun the integration green.

kayg commented 2026-11-02 15:33:14 +00:00

Author

Owner

Copy link

Correction to my previous progress note: scripts/test-proxy-pebble.sh defaults CARGO_TARGET_DIR to the parent of its fixture root. For subsequent J62b runs I’ll set the fixture root to /home/claude/.cache/codex-scratch/j62b/pebble and CARGO_TARGET_DIR=/home/claude/.cache/codex-scratch/j62b/target. The completed first diagnostic run used the script default; it is no longer running.

Correction to my previous progress note: `scripts/test-proxy-pebble.sh` defaults `CARGO_TARGET_DIR` to the parent of its fixture root. For subsequent J62b runs I’ll set the fixture root to `/home/claude/.cache/codex-scratch/j62b/pebble` and `CARGO_TARGET_DIR=/home/claude/.cache/codex-scratch/j62b/target`. The completed first diagnostic run used the script default; it is no longer running.

kayg commented 2026-11-02 15:38:59 +00:00

Author

Owner

Copy link

J62b P1 integration finding: the first real-Traefik run timed out before any route was published. Its captured log shows the file provider rejecting the generation router rule: unsupported function: Headers (run log: /home/claude/.cache/codex-scratch/j62b/pebble-diag.log, lines 107-108). I’m adding a regression assertion for Traefik’s exact Header matcher, then I’ll fix the renderer and fold it into the P1 commit before resuming P3.

J62b P1 integration finding: the first real-Traefik run timed out before any route was published. Its captured log shows the file provider rejecting the generation router rule: `unsupported function: Headers` (run log: `/home/claude/.cache/codex-scratch/j62b/pebble-diag.log`, lines 107-108). I’m adding a regression assertion for Traefik’s exact `Header` matcher, then I’ll fix the renderer and fold it into the P1 commit before resuming P3.

kayg commented 2026-11-02 15:48:05 +00:00

Author

Owner

Copy link

J62b P1 correction is folded into commit 65de8feca. RED: cargo test --locked --jobs 4 -p infraternal-plugin-proxy config_generation_probe_is_gated_on_the_loopback_health_entrypoint failed because the generated rule contained unsupported Headers(...). GREEN: the same test passed with the exact Header(...) matcher (1 passed, 0 failed). The real-Traefik run then applied the initial generation and reached P3, where the existing restart closed route A’s WebSocket during route B publish.

J62b P1 correction is folded into commit `65de8feca`. RED: `cargo test --locked --jobs 4 -p infraternal-plugin-proxy config_generation_probe_is_gated_on_the_loopback_health_entrypoint` failed because the generated rule contained unsupported `Headers(...)`. GREEN: the same test passed with the exact `Header(...)` matcher (1 passed, 0 failed). The real-Traefik run then applied the initial generation and reached P3, where the existing restart closed route A’s WebSocket during route B publish.

kayg commented 2026-11-02 16:18:39 +00:00

Author

Owner

Copy link

J62b finding 3 RED evidence: the real Pebble integration held a verified TLS WebSocket on route A while publishing route B. With the pre-fix listener-scope restart, route A closed during that publish:

route A WebSocket survives publishing route B: Custom { kind: UnexpectedEof, error: "WebSocket connection closed" }
test namespace_serves_a_granted_mesh_backend_with_pebble_dns01 ... FAILED
PEBBLE_P3_RED_EXIT=101

Log: /home/claude/.cache/codex-scratch/j62b/pebble-p3-red.log. The same scenario now exercises both publish and unpublish of route B while requiring route A frames to continue.

J62b finding 3 RED evidence: the real Pebble integration held a verified TLS WebSocket on route A while publishing route B. With the pre-fix listener-scope restart, route A closed during that publish: ```text route A WebSocket survives publishing route B: Custom { kind: UnexpectedEof, error: "WebSocket connection closed" } test namespace_serves_a_granted_mesh_backend_with_pebble_dns01 ... FAILED PEBBLE_P3_RED_EXIT=101 ``` Log: `/home/claude/.cache/codex-scratch/j62b/pebble-p3-red.log`. The same scenario now exercises both publish and unpublish of route B while requiring route A frames to continue.

kayg commented 2026-11-02 16:21:20 +00:00

Author

Owner

Copy link

J62b finding 3 GREEN on pushed head b36bec2b9.

• cargo test --locked --jobs 4 -p infraternal-mesh -p infraternal-plugin-mesh -p infraternal-plugin-proxy: proxy package, 81 passed, 0 failed.
• scripts/test-proxy-pebble.sh: namespace_serves_a_granted_mesh_backend_with_pebble_dns01, 1 passed, 0 failed.

The Pebble test keeps a real verified TLS WebSocket on route A and confirms it continues receiving frames across both publishing and unpublishing route B. The final scripts/verify fast run is next.

J62b finding 3 GREEN on pushed head `b36bec2b9`. - `cargo test --locked --jobs 4 -p infraternal-mesh -p infraternal-plugin-mesh -p infraternal-plugin-proxy`: proxy package, 81 passed, 0 failed. - `scripts/test-proxy-pebble.sh`: `namespace_serves_a_granted_mesh_backend_with_pebble_dns01`, 1 passed, 0 failed. The Pebble test keeps a real verified TLS WebSocket on route A and confirms it continues receiving frames across both publishing and unpublishing route B. The final `scripts/verify fast` run is next.

kayg commented 2026-11-02 17:25:33 +00:00

Author

Owner

Copy link

J62b gate correction on the first final-head run: static builds passed, while fmt and clippy identified formatting plus one unnested_or_patterns lint in the stable listener config. The workspace test row exposed the P2 rule-count assertion still expecting 40 after the IPv4 and IPv6 proxy-ingress accept rules were added; the renderer produced 42.

The P2 assertion is now 42 and is squashed into the P2 commit (d76686d88); its focused test passes (1 passed, 0 failed). P3 formatting and the Clippy pattern are corrected in 98892f633. The three-finding history remains one commit per finding; I am rerunning proxy integration and the fast gate on 98892f633.

J62b gate correction on the first final-head run: static builds passed, while fmt and clippy identified formatting plus one `unnested_or_patterns` lint in the stable listener config. The workspace test row exposed the P2 rule-count assertion still expecting 40 after the IPv4 and IPv6 proxy-ingress accept rules were added; the renderer produced 42. The P2 assertion is now 42 and is squashed into the P2 commit (`d76686d88`); its focused test passes (1 passed, 0 failed). P3 formatting and the Clippy pattern are corrected in `98892f633`. The three-finding history remains one commit per finding; I am rerunning proxy integration and the fast gate on `98892f633`.

kayg commented 2026-11-02 20:06:17 +00:00

Author

Owner

Copy link

J62b finished on branch proxy-traefik-281-j62b at 7e944b1caf3b9dd7bc4009178ea112182f982c4b.

The three J99 findings are fixed in separate commits:

• P1: wait for the generation-matched rawdata router before returning a revocation success; remove the route before narrowing its source list, and stop Traefik and return an error if application is not confirmed before timeout.
• P2: config and both Mesh firewall backends consume the shared listener port definitions; the regression checks every port config can produce.
• P2: keep listener entrypoints stable and update dynamic routes only; the real Pebble test holds route A's WebSocket across publishing and unpublishing route B.

Proxy verification on compute01: cargo test --locked --jobs 4 -p infraternal-mesh -p infraternal-plugin-mesh -p infraternal-plugin-proxy passed (proxy: 81 passed, 0 failed); scripts/test-proxy-pebble.sh passed (1 passed, 0 failed).

Final scripts/verify fast output:

VERIFY_FAST_EXIT=0
+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+

Static D243 result for this slice: x86_64 22,564,784 bytes (+16,352 vs task head 1fa66b199); aarch64 20,574,712 bytes (+17,088 vs task head 1fa66b199). Full gate log: /home/claude/.cache/codex-scratch/j62b/verify-fast-final-head.log.

J62b finished on branch `proxy-traefik-281-j62b` at `7e944b1caf3b9dd7bc4009178ea112182f982c4b`. The three J99 findings are fixed in separate commits: - P1: wait for the generation-matched rawdata router before returning a revocation success; remove the route before narrowing its source list, and stop Traefik and return an error if application is not confirmed before timeout. - P2: config and both Mesh firewall backends consume the shared listener port definitions; the regression checks every port config can produce. - P2: keep listener entrypoints stable and update dynamic routes only; the real Pebble test holds route A's WebSocket across publishing and unpublishing route B. Proxy verification on compute01: `cargo test --locked --jobs 4 -p infraternal-mesh -p infraternal-plugin-mesh -p infraternal-plugin-proxy` passed (proxy: 81 passed, 0 failed); `scripts/test-proxy-pebble.sh` passed (1 passed, 0 failed). Final `scripts/verify fast` output: ```text VERIFY_FAST_EXIT=0 +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | static builds | PASS | +---------------------------+--------------+ ``` Static D243 result for this slice: x86_64 22,564,784 bytes (+16,352 vs task head `1fa66b199`); aarch64 20,574,712 bytes (+17,088 vs task head `1fa66b199`). Full gate log: `/home/claude/.cache/codex-scratch/j62b/verify-fast-final-head.log`.

kayg commented 2026-11-02 20:11:39 +00:00

Author

Owner

Copy link

Starting read-only review of proxy-traefik-281 at 7e944b1caf against merge base 9f36dd8e0a. Reviewing #281 decisions and J99/J62b findings plus the full slice; no code changes.

Starting read-only review of proxy-traefik-281 at 7e944b1caf3b9dd7bc4009178ea112182f982c4b against merge base 9f36dd8e0a5e9e799cd3264cb6e1e7605071c5e2. Reviewing #281 decisions and J99/J62b findings plus the full slice; no code changes.

kayg commented 2026-11-02 20:33:21 +00:00

Author

Owner

Copy link

REQUEST CHANGES — read-only review of proxy-traefik-281 at 7e944b1caf3b9dd7bc4009178ea112182f982c4b against merge base 9f36dd8e0a5e9e799cd3264cb6e1e7605071c5e2.

The three J99 findings are closed:

• J99 P1 (revocation acknowledgment): supervisor.rs:819-858, 1020-1050 first applies a route-removal snapshot for narrowed Mesh source lists, then applies the narrower snapshot; each write gets a new UUID and waits for its own header-gated /api/rawdata router on loopback (config.rs:335-345, supervisor.rs:344-415). An old config cannot match the new UUID. Timeout stops the child and returns an error (supervisor.rs:831-841, 871-876); the Mesh observer propagates that error as a rejected action (node_actions.rs:475-480). The supervisor backs off, restarts, and renders/reapplies the retained desired services with a fresh generation (supervisor.rs:894-923, 950-1007). Full route removal also waits on the empty snapshot before success.
• J99 P2 (mixed listener ports): infraternal-mesh/src/proxy_ports.rs:29-65 is consumed by the proxy selector and both nftables/iptables renderers; config_tests.rs checks produced listener ports against the shared ingress set.
• J99 P2 (route-change restarts): entrypoints stay static while route snapshots change dynamically (config.rs:140-143, supervisor.rs:843-858). J62b’s real Pebble test holds route A’s TLS WebSocket across publishing and unpublishing route B; its passing result is recorded below.

New findings:

1. P2 — The firewall exception opens every candidate proxy port to any peer with any TCP grant. proxy_ports.rs:44-65 defines broad ranges for all 256 candidate listener pairs, while the selector chooses only one available pair per process (config.rs:147-174). The nftables renderer includes a peer in incoming*_proxy_sources if it has any inbound TCP port (firewall.rs:392-415), then accepts that source to every port in the full ranges (firewall.rs:438-439); iptables does the same (iptables.rs:425-443). Scenario: a peer has a narrow grant for TCP/3000 and another local service listens on the Mesh address at TCP/10082. If Traefik selected a different free pair, the peer can still reach 10082 through the proxy exception, despite having no grant for 10082. This is an access-policy bypass introduced by allowing the whole candidate range.

2. P2 — Traefik inherits the rootful infraternal agent’s full privileges. Rootful mode is an approved host mode (DECISIONS.md:73-78)); with effective root and free conventional ports, the proxy selects 80/443 (supervisor.rs:955-960, config.rs:127-138). The child launch at supervisor.rs:481-500` clears the environment and sets lifecycle controls, but does not drop uid/gid or capabilities or apply a child sandbox. Scenario: a rootful node publishes a Mesh service, and a remotely reachable request exploits Traefik; the process has host-root privileges. Binding 80/443 does not require retaining all root privileges while processing network traffic.

3. P2 — Mesh-only publication still binds public entrypoints on wildcard addresses. config.rs:240-260 renders the public HTTP/TLS listeners on unspecified addresses on every start; a Mesh-only service is attached only to mesh-http/mesh-https (config.rs:391-395). Scenario: the first publication is Mesh-only, but a host with ordinary inbound access still accepts unauthenticated TCP/TLS on wildcard 80/443 in rootful mode (or 10080/10443 rootless); those requests reach Traefik’s parser and return no-route responses without --public. This exposes the proxy’s network parser outside the Mesh even when no public route exists.

Artifact acquisition and release verification match the owner’s two-check answer: CI verifies the pinned upstream checksum list and signs our manifest; runtime verifies our release signature, signed tuple, archive and executable hashes, bounds, and allowlisted extraction. Token custody is owner-only, the child receives only *_FILE paths after env_clear, and token values are not put in argv. The generation API route is on the loopback-only health entrypoint; dashboard and insecure API are disabled. Per-machine wildcard keys and ACME state stay node-local; flat names request exact certificates. I did not rerun tests or builds on the read-only review host.

J62b’s exact-head verification from #281, quoted verbatim:

VERIFY_FAST_EXIT=0
+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+

No source changes made.

REQUEST CHANGES — read-only review of `proxy-traefik-281` at `7e944b1caf3b9dd7bc4009178ea112182f982c4b` against merge base `9f36dd8e0a5e9e799cd3264cb6e1e7605071c5e2`. The three J99 findings are closed: - **J99 P1 (revocation acknowledgment):** `supervisor.rs:819-858, 1020-1050` first applies a route-removal snapshot for narrowed Mesh source lists, then applies the narrower snapshot; each write gets a new UUID and waits for its own header-gated `/api/rawdata` router on loopback (`config.rs:335-345`, `supervisor.rs:344-415`). An old config cannot match the new UUID. Timeout stops the child and returns an error (`supervisor.rs:831-841, 871-876`); the Mesh observer propagates that error as a rejected action (`node_actions.rs:475-480`). The supervisor backs off, restarts, and renders/reapplies the retained desired services with a fresh generation (`supervisor.rs:894-923, 950-1007`). Full route removal also waits on the empty snapshot before success. - **J99 P2 (mixed listener ports):** `infraternal-mesh/src/proxy_ports.rs:29-65` is consumed by the proxy selector and both nftables/iptables renderers; `config_tests.rs` checks produced listener ports against the shared ingress set. - **J99 P2 (route-change restarts):** entrypoints stay static while route snapshots change dynamically (`config.rs:140-143`, `supervisor.rs:843-858`). J62b’s real Pebble test holds route A’s TLS WebSocket across publishing and unpublishing route B; its passing result is recorded below. New findings: 1. **P2 — The firewall exception opens every candidate proxy port to any peer with any TCP grant.** `proxy_ports.rs:44-65` defines broad ranges for all 256 candidate listener pairs, while the selector chooses only one available pair per process (`config.rs:147-174`). The nftables renderer includes a peer in `incoming*_proxy_sources` if it has any inbound TCP port (`firewall.rs:392-415`), then accepts that source to every port in the full ranges (`firewall.rs:438-439`); iptables does the same (`iptables.rs:425-443`). Scenario: a peer has a narrow grant for TCP/3000 and another local service listens on the Mesh address at TCP/10082. If Traefik selected a different free pair, the peer can still reach 10082 through the proxy exception, despite having no grant for 10082. This is an access-policy bypass introduced by allowing the whole candidate range. 2. **P2 — Traefik inherits the rootful infraternal agent’s full privileges.** Rootful mode is an approved host mode (`DECISIONS.md:73-78)); with effective root and free conventional ports, the proxy selects 80/443 (`supervisor.rs:955-960`, `config.rs:127-138`). The child launch at `supervisor.rs:481-500` clears the environment and sets lifecycle controls, but does not drop uid/gid or capabilities or apply a child sandbox. Scenario: a rootful node publishes a Mesh service, and a remotely reachable request exploits Traefik; the process has host-root privileges. Binding 80/443 does not require retaining all root privileges while processing network traffic. 3. **P2 — Mesh-only publication still binds public entrypoints on wildcard addresses.** `config.rs:240-260` renders the public HTTP/TLS listeners on unspecified addresses on every start; a Mesh-only service is attached only to `mesh-http`/`mesh-https` (`config.rs:391-395`). Scenario: the first publication is Mesh-only, but a host with ordinary inbound access still accepts unauthenticated TCP/TLS on wildcard 80/443 in rootful mode (or 10080/10443 rootless); those requests reach Traefik’s parser and return no-route responses without `--public`. This exposes the proxy’s network parser outside the Mesh even when no public route exists. Artifact acquisition and release verification match the owner’s two-check answer: CI verifies the pinned upstream checksum list and signs our manifest; runtime verifies our release signature, signed tuple, archive and executable hashes, bounds, and allowlisted extraction. Token custody is owner-only, the child receives only `*_FILE` paths after `env_clear`, and token values are not put in argv. The generation API route is on the loopback-only health entrypoint; dashboard and insecure API are disabled. Per-machine wildcard keys and ACME state stay node-local; flat names request exact certificates. I did not rerun tests or builds on the read-only review host. J62b’s exact-head verification from #281, quoted verbatim: ```text VERIFY_FAST_EXIT=0 +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | static builds | PASS | +---------------------------+--------------+ ``` No source changes made.

kayg commented 2026-11-02 20:34:19 +00:00

Author

Owner

Copy link

Correction and one additional low-priority finding for my review above:

• The rootful-mode decision reference should render as DECISIONS.md:73-78.
• P3 — A newline-terminated DNS token can evade exact-value log redaction. Custody writes token bytes unchanged (custody.rs:140-159), and child startup retains those bytes in the redaction value (supervisor.rs:454-475). The log reader removes trailing CR/LF before exact-value replacement (supervisor.rs:577-580, 177-187). Scenario: a token file ends in a newline and a provider error/debug line emits the normalized raw token; after line-ending removal, the exact secret string no longer matches and the value can reach the log sink. This is conditional on a provider emitting the raw value, but the sanitizer does not uphold the stated invariant for that accepted file shape.

The verdict remains REQUEST CHANGES; the three P2 findings and evidence are in the review comment above.

Correction and one additional low-priority finding for my review above: - The rootful-mode decision reference should render as `DECISIONS.md:73-78`. - **P3 — A newline-terminated DNS token can evade exact-value log redaction.** Custody writes token bytes unchanged (`custody.rs:140-159`), and child startup retains those bytes in the redaction value (`supervisor.rs:454-475`). The log reader removes trailing CR/LF before exact-value replacement (`supervisor.rs:577-580`, `177-187`). Scenario: a token file ends in a newline and a provider error/debug line emits the normalized raw token; after line-ending removal, the exact secret string no longer matches and the value can reach the log sink. This is conditional on a provider emitting the raw value, but the sanitizer does not uphold the stated invariant for that accepted file shape. The verdict remains **REQUEST CHANGES**; the three P2 findings and evidence are in the review comment above.

kayg commented 2026-11-02 20:37:38 +00:00

Author

Owner

Copy link

J62b round 3 started on branch proxy-traefik-281-j62b, currently at 7e944b1caf3b9dd7bc4009178ea112182f982c4b; rebasing onto origin/dev base b5099e3be1721d01f12f8e7782723e9f23ab2350.

Plan (about 4 hours active work, excluding VM build-slot waits):

1. Rebase and verify the current J62b fixes against the latest base; trace J116's four paths and locate existing child-sandbox and firewall-grant patterns.
2. Add failing-first access tests, then constrain each firewall backend to the selected proxy pair and route-authorized peers.
3. Add a failing-first real-child privilege test; run Traefik as a dedicated unprivileged uid/gid with the existing child sandbox, minimal capabilities, and custody file ownership.
4. Add a failing-first public-listener exposure test; gate public access without disrupting active Mesh routes, including the rootless guarantee boundary.
5. Add failing-first raw/trimmed DNS-token redaction coverage, then run focused proxy checks and the final fast gate, push, and record the exact output here.

J62b round 3 started on branch `proxy-traefik-281-j62b`, currently at `7e944b1caf3b9dd7bc4009178ea112182f982c4b`; rebasing onto `origin/dev` base `b5099e3be1721d01f12f8e7782723e9f23ab2350`. Plan (about 4 hours active work, excluding VM build-slot waits): 1. Rebase and verify the current J62b fixes against the latest base; trace J116's four paths and locate existing child-sandbox and firewall-grant patterns. 2. Add failing-first access tests, then constrain each firewall backend to the selected proxy pair and route-authorized peers. 3. Add a failing-first real-child privilege test; run Traefik as a dedicated unprivileged uid/gid with the existing child sandbox, minimal capabilities, and custody file ownership. 4. Add a failing-first public-listener exposure test; gate public access without disrupting active Mesh routes, including the rootless guarantee boundary. 5. Add failing-first raw/trimmed DNS-token redaction coverage, then run focused proxy checks and the final fast gate, push, and record the exact output here.

kayg commented 2026-11-02 21:00:31 +00:00

Author

Owner

Copy link

J62b round 3 finding 1 evidence: both firewall backends currently admit the whole candidate interval to any peer with any inbound TCP grant. The corrected RED run on compute01 failed both new assertions: nft still rendered 10080-10335 and iptables still rendered --ctorigdstport 10080:10335 for a peer whose only grant is TCP/3000 and with no published proxy route. Log: /home/claude/.cache/codex-scratch/j62b/p2-red.log.

I’m changing the exception to use the selected Mesh listener pair and only backend ports of currently published Mesh routes; each backend will derive sources from the accepted grant for those route ports. The regression also checks the granted route remains reachable through that pair.

J62b round 3 finding 1 evidence: both firewall backends currently admit the whole candidate interval to any peer with any inbound TCP grant. The corrected RED run on compute01 failed both new assertions: nft still rendered `10080-10335` and iptables still rendered `--ctorigdstport 10080:10335` for a peer whose only grant is TCP/3000 and with no published proxy route. Log: `/home/claude/.cache/codex-scratch/j62b/p2-red.log`. I’m changing the exception to use the selected Mesh listener pair and only backend ports of currently published Mesh routes; each backend will derive sources from the accepted grant for those route ports. The regression also checks the granted route remains reachable through that pair.

kayg commented 2026-11-02 21:23:26 +00:00

Author

Owner

Copy link

Finding 1 is committed as 34cd2e69c and pushed. Mesh ingress now uses the selected listener pair and only peers whose accepted TCP grant matches an applied non-public backend route; the controller updates it after the Traefik generation is confirmed. Evidence: infraternal-plugin-mesh --lib: 45 passed, 0 failed, 13 ignored; infraternal-plugin-proxy --all-targets: 82 passed, 0 failed, 1 process integration passed, with namespace_pebble ignored because its pinned fixture is run separately.

Finding 1 is committed as 34cd2e69c and pushed. Mesh ingress now uses the selected listener pair and only peers whose accepted TCP grant matches an applied non-public backend route; the controller updates it after the Traefik generation is confirmed. Evidence: infraternal-plugin-mesh --lib: 45 passed, 0 failed, 13 ignored; infraternal-plugin-proxy --all-targets: 82 passed, 0 failed, 1 process integration passed, with namespace_pebble ignored because its pinned fixture is run separately.

kayg commented 2026-11-02 21:41:10 +00:00

Author

Owner

Copy link

J62b round 3 resumed on proxy-traefik-281-j62b. I fetched origin and found origin/dev at 816f9ab3e8dc708d88a5f774d605cced22d82d22; the current branch is based on b5099e3be1721d01f12f8e7782723e9f23ab2350, so I am rebasing before continuing.

Plan (about 4 hours active work, excluding build-slot waits):

1. Rebase and trace the four review paths plus existing child-sandbox patterns.
2. Finish the rootful child identity/capability test and implementation; commit it atomically.
3. Gate public listeners without disrupting other route connections; add a regression and commit it atomically.
4. Cover raw and trimmed DNS token redaction; fix and commit it atomically.
5. Run the full gate and proxy integration tests on an admitted VM, push, and record the exact receipts.

J62b round 3 resumed on `proxy-traefik-281-j62b`. I fetched origin and found `origin/dev` at `816f9ab3e8dc708d88a5f774d605cced22d82d22`; the current branch is based on `b5099e3be1721d01f12f8e7782723e9f23ab2350`, so I am rebasing before continuing. Plan (about 4 hours active work, excluding build-slot waits): 1. Rebase and trace the four review paths plus existing child-sandbox patterns. 2. Finish the rootful child identity/capability test and implementation; commit it atomically. 3. Gate public listeners without disrupting other route connections; add a regression and commit it atomically. 4. Cover raw and trimmed DNS token redaction; fix and commit it atomically. 5. Run the full gate and proxy integration tests on an admitted VM, push, and record the exact receipts.

kayg commented 2026-11-02 22:57:48 +00:00

Author

Owner

Copy link

J62b round 3 finding 2 (P2) is committed and pushed as 5ef6390db.

RED on the pre-fix source, from /home/claude/.cache/codex-scratch/j62b/p2-rootless-red.log:

assertion failed: left: Some("0"), right: Some("1")
test traefik_child_obeys_the_rootful_identity_and_capability_boundary ... FAILED
J62B_F2_RED_EXIT=101

GREEN on compute01, cargo test --locked --jobs 4 -p infraternal-plugin-proxy --test process traefik_child_obeys_the_rootful_identity_and_capability_boundary -- --exact --nocapture:

running 1 test
test traefik_child_obeys_the_rootful_identity_and_capability_boundary ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 1 filtered out
J62B_F2_GREEN_EXIT=0

Rootful launch now drops to UID/GID 65532, keeps only CAP_NET_BIND_SERVICE in the effective, permitted, inheritable, ambient and bounding sets, and enables no_new_privs. It reuses the shim's existing signal/fd3 isolation and parent-bound child hooks. The executable and config remain inaccessible through private state ancestors; DNS token custody files are owned by the child at mode 0600. Safe prior root-owned executable/config/custody files are migrated during verified installation, config replacement, or custody open.

The integration test reads /proc/self/status from the running child and asserts the rootful UID/GID and capability sets when the runner is root. This compute01 run was unprivileged, so it exercised the rootless process assertions; sudo -n is unavailable there, and its user namespace lacks newuidmap.

J62b round 3 finding 2 (P2) is committed and pushed as `5ef6390db`. RED on the pre-fix source, from `/home/claude/.cache/codex-scratch/j62b/p2-rootless-red.log`: ```text assertion failed: left: Some("0"), right: Some("1") test traefik_child_obeys_the_rootful_identity_and_capability_boundary ... FAILED J62B_F2_RED_EXIT=101 ``` GREEN on compute01, `cargo test --locked --jobs 4 -p infraternal-plugin-proxy --test process traefik_child_obeys_the_rootful_identity_and_capability_boundary -- --exact --nocapture`: ```text running 1 test test traefik_child_obeys_the_rootful_identity_and_capability_boundary ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 1 filtered out J62B_F2_GREEN_EXIT=0 ``` Rootful launch now drops to UID/GID 65532, keeps only CAP_NET_BIND_SERVICE in the effective, permitted, inheritable, ambient and bounding sets, and enables no_new_privs. It reuses the shim's existing signal/fd3 isolation and parent-bound child hooks. The executable and config remain inaccessible through private state ancestors; DNS token custody files are owned by the child at mode 0600. Safe prior root-owned executable/config/custody files are migrated during verified installation, config replacement, or custody open. The integration test reads `/proc/self/status` from the running child and asserts the rootful UID/GID and capability sets when the runner is root. This compute01 run was unprivileged, so it exercised the rootless process assertions; `sudo -n` is unavailable there, and its user namespace lacks `newuidmap`.

kayg commented 2026-11-02 23:57:00 +00:00

Author

Owner

Copy link

J62b round 3 finding 3 RED on 5ef6390db, dev-02:

config_tests::rootless_public_entrypoints_bind_to_loopback ... FAILED
controller::tests::public_listener_gate_is_installed_before_the_first_route_snapshot ... FAILED
test result: FAILED. 82 passed; 2 failed

The renderer produced wildcard public listeners for rootless selected ports. The controller's first firewall observer call happened after the route snapshot was already recorded. I will keep rootful public listeners stable and gate their selected pair in the owned Mesh firewall: close before starting the first listener, open after a public route is applied, and close before removing the last public route. Rootless mode will bind those entrypoints to loopback and reject --public, since it cannot manage the host firewall or provide remote public reachability while keeping listener config stable.

J62b round 3 finding 3 RED on `5ef6390db`, dev-02: ```text config_tests::rootless_public_entrypoints_bind_to_loopback ... FAILED controller::tests::public_listener_gate_is_installed_before_the_first_route_snapshot ... FAILED test result: FAILED. 82 passed; 2 failed ``` The renderer produced wildcard public listeners for rootless selected ports. The controller's first firewall observer call happened after the route snapshot was already recorded. I will keep rootful public listeners stable and gate their selected pair in the owned Mesh firewall: close before starting the first listener, open after a public route is applied, and close before removing the last public route. Rootless mode will bind those entrypoints to loopback and reject `--public`, since it cannot manage the host firewall or provide remote public reachability while keeping listener config stable.

kayg commented 2026-11-03 00:23:49 +00:00

Author

Owner

Copy link

J62b round 3 finding 3 is committed as ab7b79b3c.

Rootful public entrypoints keep one stable selected port pair. The owned nftables and iptables backends block non-loopback traffic to that pair before Traefik starts and before the last public route is removed; they open the gate only after the public route's dynamic snapshot is applied. Mesh routes continue using dynamic config, so this does not restart Traefik. Rootless mode binds those entrypoints to 127.0.0.1 and rejects --public: it can keep the parser off remote interfaces, but cannot control the host firewall or offer remote public reachability.

GREEN on dev-02, cargo test --locked --jobs 3 -p infraternal-mesh -p infraternal-plugin-mesh -p infraternal-plugin-proxy --all-targets: exit 0. The proxy package ran 85 tests with 0 failures; namespace_pebble remains ignored there and is run separately through scripts/test-proxy-pebble.sh.

J62b round 3 finding 3 is committed as `ab7b79b3c`. Rootful public entrypoints keep one stable selected port pair. The owned nftables and iptables backends block non-loopback traffic to that pair before Traefik starts and before the last public route is removed; they open the gate only after the public route's dynamic snapshot is applied. Mesh routes continue using dynamic config, so this does not restart Traefik. Rootless mode binds those entrypoints to `127.0.0.1` and rejects `--public`: it can keep the parser off remote interfaces, but cannot control the host firewall or offer remote public reachability. GREEN on dev-02, `cargo test --locked --jobs 3 -p infraternal-mesh -p infraternal-plugin-mesh -p infraternal-plugin-proxy --all-targets`: exit 0. The proxy package ran 85 tests with 0 failures; `namespace_pebble` remains ignored there and is run separately through `scripts/test-proxy-pebble.sh`.

kayg commented 2026-11-03 00:27:14 +00:00

Author

Owner

Copy link

J62b round 3 continued. The branch is rebased onto current origin/dev bce4164636; I preserved the newer Linux remote-install cfg and bytes dependency while carrying the proxy registry wiring. The J116 fixes and three of four J62b round-three findings are already recorded above. Remaining: add newline and CRLF DNS-token redaction regression cases, observe RED, implement raw-plus-trimmed replacement, then run Pebble integration and scripts/verify fast, push and record the exact gate receipt. Estimate: 45 minutes of active work, excluding build-slot waits.

J62b round 3 continued. The branch is rebased onto current origin/dev bce4164636dee480558760b571431296f4c65aac; I preserved the newer Linux remote-install cfg and bytes dependency while carrying the proxy registry wiring. The J116 fixes and three of four J62b round-three findings are already recorded above. Remaining: add newline and CRLF DNS-token redaction regression cases, observe RED, implement raw-plus-trimmed replacement, then run Pebble integration and scripts/verify fast, push and record the exact gate receipt. Estimate: 45 minutes of active work, excluding build-slot waits.

kayg commented 2026-11-03 00:44:42 +00:00

Author

Owner

Copy link

J62b round 3 finding 4 RED on the pre-fix source, dev-02, through build-slot: cargo test --locked --jobs 3 -p infraternal-plugin-proxy --lib newline_terminated_provider_values_are_redacted_after_log_normalization -- --nocapture failed as expected. Output: running 1 test; supervisor_tests::newline_terminated_provider_values_are_redacted_after_log_normalization ... FAILED; assertion !redacted.contains("secret-value-123") failed; test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 85 filtered out; exit 101. The test passes a line-normalized log entry and a raw newline-terminated custody value. I am changing the replacement set to include both the raw value and its CR/LF-trimmed form, then will rerun this test.

J62b round 3 finding 4 RED on the pre-fix source, dev-02, through build-slot: `cargo test --locked --jobs 3 -p infraternal-plugin-proxy --lib newline_terminated_provider_values_are_redacted_after_log_normalization -- --nocapture` failed as expected. Output: `running 1 test`; `supervisor_tests::newline_terminated_provider_values_are_redacted_after_log_normalization ... FAILED`; assertion `!redacted.contains("secret-value-123")` failed; `test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 85 filtered out`; exit 101. The test passes a line-normalized log entry and a raw newline-terminated custody value. I am changing the replacement set to include both the raw value and its CR/LF-trimmed form, then will rerun this test.

kayg commented 2026-11-03 00:53:39 +00:00

Author

Owner

Copy link

J62b round 3 finding 4 is fixed in 1d5a8d2b3. RED was recorded above. GREEN on compute01 against the committed source, cargo test --locked --jobs 4 -p infraternal-plugin-proxy --lib newline_terminated_provider_values_are_redacted_after_log_normalization -- --nocapture:

running 1 test
test supervisor_tests::newline_terminated_provider_values_are_redacted_after_log_normalization ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 85 filtered out

The sanitizer now matches both stored bytes and the value after trailing CR/LF removal. Next: run the mesh/proxy package integration tests, the Pebble WebSocket fixture, and scripts/verify fast on the final head.

J62b round 3 finding 4 is fixed in `1d5a8d2b3`. RED was recorded above. GREEN on compute01 against the committed source, `cargo test --locked --jobs 4 -p infraternal-plugin-proxy --lib newline_terminated_provider_values_are_redacted_after_log_normalization -- --nocapture`: ```text running 1 test test supervisor_tests::newline_terminated_provider_values_are_redacted_after_log_normalization ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 85 filtered out ``` The sanitizer now matches both stored bytes and the value after trailing CR/LF removal. Next: run the mesh/proxy package integration tests, the Pebble WebSocket fixture, and `scripts/verify fast` on the final head.

kayg commented 2026-11-03 01:29:14 +00:00

Author

Owner

Copy link

Additional finding 2 regression found by the required Pebble integration. Inside its unshare --user --map-root-user fixture, /proc/self/uid_map and gid_map each contain only 0 1500 1; UID/GID 65532 are unavailable. The current identity::rootful() checks only effective UID 0, so Traefik takes the host-root path and rejects the private fixture config directory before startup. RED on compute01: identity::tests::user_namespace_root_without_traefik_id_mappings_is_rootless failed at assert!(!super::rootful()) (0 passed, 1 failed, exit 101). I will require both fixed child IDs to be mapped before Linux selects rootful mode, and fold this correction into the existing rootful finding 2 commit.

Additional finding 2 regression found by the required Pebble integration. Inside its `unshare --user --map-root-user` fixture, `/proc/self/uid_map` and `gid_map` each contain only `0 1500 1`; UID/GID 65532 are unavailable. The current `identity::rootful()` checks only effective UID 0, so Traefik takes the host-root path and rejects the private fixture config directory before startup. RED on compute01: `identity::tests::user_namespace_root_without_traefik_id_mappings_is_rootless` failed at `assert!(!super::rootful())` (0 passed, 1 failed, exit 101). I will require both fixed child IDs to be mapped before Linux selects rootful mode, and fold this correction into the existing rootful finding 2 commit.

kayg commented 2026-11-03 01:32:51 +00:00

Author

Owner

Copy link

Finding 2 user-namespace boundary is corrected in the working tree. GREEN on compute01: cargo test --locked --jobs 4 -p infraternal-plugin-proxy --lib user_namespace_root_without_traefik_id_mappings_is_rootless -- --nocapture:

running 1 test
running 1 test
test identity::tests::user_namespace_root_without_traefik_id_mappings_is_rootless ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 86 filtered out

test identity::tests::user_namespace_root_without_traefik_id_mappings_is_rootless ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 86 filtered out

Linux rootful mode now requires euid 0 plus mappings for the fixed Traefik UID and GID. A one-ID user namespace therefore stays in rootless mode and keeps its private 0700 config directory. I am rerunning the Pebble integration now; this remains part of finding 2 and will be folded into its commit.

Finding 2 user-namespace boundary is corrected in the working tree. GREEN on compute01: `cargo test --locked --jobs 4 -p infraternal-plugin-proxy --lib user_namespace_root_without_traefik_id_mappings_is_rootless -- --nocapture`: ```text running 1 test running 1 test test identity::tests::user_namespace_root_without_traefik_id_mappings_is_rootless ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 86 filtered out test identity::tests::user_namespace_root_without_traefik_id_mappings_is_rootless ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 86 filtered out ``` Linux rootful mode now requires euid 0 plus mappings for the fixed Traefik UID and GID. A one-ID user namespace therefore stays in rootless mode and keeps its private 0700 config directory. I am rerunning the Pebble integration now; this remains part of finding 2 and will be folded into its commit.

kayg commented 2026-11-03 01:35:47 +00:00

Author

Owner

Copy link

The Pebble regression that exposed finding 2 now passes on compute01 after the user-namespace root check: CARGO_TARGET_DIR=/home/claude/worktrees/j62b-r3-codex/target scripts/test-proxy-pebble.sh:

running 1 test
running 1 test
test namespace_serves_a_granted_mesh_backend_with_pebble_dns01 ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

test namespace_serves_a_granted_mesh_backend_with_pebble_dns01 ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

The test now runs in the intended rootless user namespace instead of taking the host-root file ownership path. The identity boundary and its RED/GREEN test will be folded into the existing finding 2 commit before final verification.

The Pebble regression that exposed finding 2 now passes on compute01 after the user-namespace root check: `CARGO_TARGET_DIR=/home/claude/worktrees/j62b-r3-codex/target scripts/test-proxy-pebble.sh`: ```text running 1 test running 1 test test namespace_serves_a_granted_mesh_backend_with_pebble_dns01 ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out test namespace_serves_a_granted_mesh_backend_with_pebble_dns01 ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out ``` The test now runs in the intended rootless user namespace instead of taking the host-root file ownership path. The identity boundary and its RED/GREEN test will be folded into the existing finding 2 commit before final verification.

kayg commented 2026-11-03 02:10:45 +00:00

Author

Owner

Copy link

J62b round 3 rebased onto latest at ; candidate head is pushed. The four round-three fixes remain separate commits with RED/GREEN evidence above. Final proxy integration and are next on compute01; dev-02 currently has only 9 GB free, below the 25 GB build floor.

J62b round 3 rebased onto latest at ; candidate head is pushed. The four round-three fixes remain separate commits with RED/GREEN evidence above. Final proxy integration and are next on compute01; dev-02 currently has only 9 GB free, below the 25 GB build floor.

kayg commented 2026-11-03 02:11:11 +00:00

Author

Owner

Copy link

J62b round 3 has been rebased onto latest origin/dev at 299b7ad30b. Candidate head 9d4040b005 is pushed on proxy-traefik-281-j62b. The four finding fixes remain separate commits with RED/GREEN evidence above. Final proxy integration and scripts/verify fast are next on compute01; dev-02 currently has 9 GB free, below the 25 GB build floor.

J62b round 3 has been rebased onto latest origin/dev at 299b7ad30bdbf3c45b3436f7ba63d05b3599aa9f. Candidate head 9d4040b005d2c8822c65818a9bf6295b3fe6d420 is pushed on proxy-traefik-281-j62b. The four finding fixes remain separate commits with RED/GREEN evidence above. Final proxy integration and scripts/verify fast are next on compute01; dev-02 currently has 9 GB free, below the 25 GB build floor.

kayg commented 2026-11-03 02:43:14 +00:00

Author

Owner

Copy link

Queue update for J62b round 3: my compute01 build-slot job remained queued for 27 minutes and never entered a slot; I terminated it cleanly. Dev-02 now has 77 GB free and a shorter queue, so its worktree is prepared at 9d4040b005 for package tests and scripts/verify fast. The pinned Pebble integration is x86_64-only and will run on compute01 separately.

Queue update for J62b round 3: my compute01 build-slot job remained queued for 27 minutes and never entered a slot; I terminated it cleanly. Dev-02 now has 77 GB free and a shorter queue, so its worktree is prepared at 9d4040b005d2c8822c65818a9bf6295b3fe6d420 for package tests and scripts/verify fast. The pinned Pebble integration is x86_64-only and will run on compute01 separately.

kayg commented 2026-11-03 03:31:53 +00:00

Author

Owner

Copy link

The first post-rebase fast gate on 9d4040b005 found fmt and Clippy failures in the rootful child changes plus an inverted conditional lint in the public listener gate. I corrected those and folded the formatting/lint changes into the existing rootful and listener-gate commits; the history remains one commit per finding. The static-build row passed. The package suite passed (proxy: 87 passed; process: 2 passed), but the workspace vt_credit integration failed once with credit probe failed: t1-gate-approximate-1 after 25.45s while other build jobs were active. I am rerunning that test and the full gate on the updated pushed head before reporting completion. Gate log: /home/claude/.cache/codex-scratch/j62b/verify-fast-final-head.log.

The first post-rebase fast gate on 9d4040b005d2c8822c65818a9bf6295b3fe6d420 found fmt and Clippy failures in the rootful child changes plus an inverted conditional lint in the public listener gate. I corrected those and folded the formatting/lint changes into the existing rootful and listener-gate commits; the history remains one commit per finding. The static-build row passed. The package suite passed (proxy: 87 passed; process: 2 passed), but the workspace vt_credit integration failed once with `credit probe failed: t1-gate-approximate-1` after 25.45s while other build jobs were active. I am rerunning that test and the full gate on the updated pushed head before reporting completion. Gate log: /home/claude/.cache/codex-scratch/j62b/verify-fast-final-head.log.

kayg commented 2026-11-03 06:08:32 +00:00

Author

Owner

Copy link

J62b round 3 continuation: I fetched and rebased onto latest origin/dev, 7add126405 (parent a6812c3cf5), with no conflicts. The latest fast gate completed with VERIFY_EXIT=1: fmt, VT native fmt, compile-out, package tests, static builds and the remaining rows passed; Clippy failed because registry_at is 103 lines after the new Mesh observer wiring in finding 1. I’ll extract that observer into a focused helper and fold the lint fix into finding 1’s commit, then rerun the focused Mesh/proxy tests, Pebble integration, and full gate. Current VM queues are full; estimate 45 minutes of active work, excluding queue time.

J62b round 3 continuation: I fetched and rebased onto latest origin/dev, 7add126405a9205bb817a154af11fad2c438cd45 (parent a6812c3cf528fed73d8dbac004a89da8f8f77906), with no conflicts. The latest fast gate completed with VERIFY_EXIT=1: fmt, VT native fmt, compile-out, package tests, static builds and the remaining rows passed; Clippy failed because registry_at is 103 lines after the new Mesh observer wiring in finding 1. I’ll extract that observer into a focused helper and fold the lint fix into finding 1’s commit, then rerun the focused Mesh/proxy tests, Pebble integration, and full gate. Current VM queues are full; estimate 45 minutes of active work, excluding queue time.

kayg commented 2026-11-03 10:07:12 +00:00

Author

Owner

Copy link

J62b-c takeover of J62b round 3 (the previous job stopped on a provider rate limit at about 12:30 IST).

State found:

• Branch proxy-traefik-281-j62b, pushed head 5b76657c4, based on 7add12640. All four J116 round 3 fixes are committed separately, with RED/GREEN evidence above: da4aa7320 (scoped proxy ingress), 5bca1bed9 (rootful privilege drop), 857cb3856 (public listener gate), 5b76657c4 (newline token redaction).
• No uncommitted work in ~/Developer/Worktrees/infraternal-J62b, compute01:~/worktrees/j62b-r3-codex or dev-02:~/worktrees/j62b-r3-codex.
• No J62b processes are running on either VM.
• Last gate (dev-02, ~/.cache/codex-scratch/j62b/verify-fast-r3-final.log): clippy PASS, static builds PASS, test FAIL (101) on one test outside the slice: infraternal-server auth::tests::socket_tests::transient_authority_stall::terminal_stream_survives_authority_stalls_without_loss_or_reorder (748 passed, 1 failed). The earlier 06:00 run failed clippy too_many_lines on registry_at (crates/infraternal-agent/src/plugins.rs, 103 lines); the J62b note about extracting it is still open.

Remaining plan (about 40 min active work, plus queue time):

1. Rebased onto origin/dev 7c082bddf (clean, no conflicts).
2. Extract the Mesh-observer wiring out of registry_at and fold it into the finding 1 commit (da4aa7320).
3. Check whether the transient_authority_stall failure also happens on unmodified origin/dev, and rerun it on the branch.
4. Run the Pebble integration on compute01 and scripts/verify fast on the final head; push; record the receipts here.

J62b-c takeover of J62b round 3 (the previous job stopped on a provider rate limit at about 12:30 IST). State found: - Branch `proxy-traefik-281-j62b`, pushed head `5b76657c4`, based on `7add12640`. All four J116 round 3 fixes are committed separately, with RED/GREEN evidence above: `da4aa7320` (scoped proxy ingress), `5bca1bed9` (rootful privilege drop), `857cb3856` (public listener gate), `5b76657c4` (newline token redaction). - No uncommitted work in `~/Developer/Worktrees/infraternal-J62b`, `compute01:~/worktrees/j62b-r3-codex` or `dev-02:~/worktrees/j62b-r3-codex`. - No J62b processes are running on either VM. - Last gate (dev-02, `~/.cache/codex-scratch/j62b/verify-fast-r3-final.log`): clippy PASS, static builds PASS, test FAIL (101) on one test outside the slice: `infraternal-server` `auth::tests::socket_tests::transient_authority_stall::terminal_stream_survives_authority_stalls_without_loss_or_reorder` (748 passed, 1 failed). The earlier 06:00 run failed clippy `too_many_lines` on `registry_at` (`crates/infraternal-agent/src/plugins.rs`, 103 lines); the J62b note about extracting it is still open. Remaining plan (about 40 min active work, plus queue time): 1. Rebased onto origin/dev `7c082bddf` (clean, no conflicts). 2. Extract the Mesh-observer wiring out of `registry_at` and fold it into the finding 1 commit (`da4aa7320`). 3. Check whether the `transient_authority_stall` failure also happens on unmodified origin/dev, and rerun it on the branch. 4. Run the Pebble integration on compute01 and `scripts/verify fast` on the final head; push; record the receipts here.

kayg commented 2026-11-03 10:13:39 +00:00

Author

Owner

Copy link

J62b-c progress: the branch is rebased onto origin/dev 7c082bddf and pushed as b67e7a879; the force-push was leased on 5b76657c4.

The registry_at clippy too_many_lines fix is folded into the finding 1 commit (now a7bfa336c, "scope proxy ingress to applied routes"). Proxy plugin construction moves to proxy_plugin_at, and the two-way Mesh/proxy observer wiring moves to connect_proxy_and_mesh in crates/infraternal-agent/src/plugins.rs. There is no behaviour change: both hooks are plain setters. The tree outside that file is identical to the pre-rebase head.

Running now:

• scripts/verify fast on dev-02 at b67e7a879; log ~/.cache/codex-scratch/j62b/verify-fast-c.log.
• scripts/test-proxy-pebble.sh on compute01 at b67e7a879; log ~/.cache/codex-scratch/j62b/pebble-c.log.

Target dirs: ~/worktrees/j62b-r3-codex/target on both VMs.

J62b-c progress: the branch is rebased onto origin/dev `7c082bddf` and pushed as `b67e7a879`; the force-push was leased on `5b76657c4`. The `registry_at` clippy `too_many_lines` fix is folded into the finding 1 commit (now `a7bfa336c`, "scope proxy ingress to applied routes"). Proxy plugin construction moves to `proxy_plugin_at`, and the two-way Mesh/proxy observer wiring moves to `connect_proxy_and_mesh` in `crates/infraternal-agent/src/plugins.rs`. There is no behaviour change: both hooks are plain setters. The tree outside that file is identical to the pre-rebase head. Running now: - `scripts/verify fast` on dev-02 at `b67e7a879`; log `~/.cache/codex-scratch/j62b/verify-fast-c.log`. - `scripts/test-proxy-pebble.sh` on compute01 at `b67e7a879`; log `~/.cache/codex-scratch/j62b/pebble-c.log`. Target dirs: `~/worktrees/j62b-r3-codex/target` on both VMs.

kayg commented 2026-11-03 11:04:20 +00:00

Author

Owner

Copy link

J62b-c finish on branch proxy-traefik-281-j62b, head b67e7a87916af4a2aed0bad0d4842d6b3f165598, based on origin/dev 7c082bddfa36a7ea464e406522dbe67feda1630f.

scripts/verify fast on dev-02 at b67e7a879 (nix develop -c, through build-slot), verbatim (log dev-02:~/.cache/codex-scratch/j62b/verify-fast-c.log):

(static builds: 285s)

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | FAIL (101)   |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
VERIFY_EXIT=1

The only failing row is test, and it has one failure outside the slice: -p infraternal --test session_exit_events (idle_daemon_discovers_unattached_and_adopted_shim_exit_without_requests). The first client.open() on the real agent socket hit the fixture's 1 s receive timeout (TimeoutError: timed out, agent_fixture.py:129). The empty-agent idle window just before it passed (seconds=2.210). On the previous head, the previous run failed a different timing test in infraternal-server (transient_authority_stall) and this one passed.

Evidence that it is load flakiness, not this branch:

• Isolated reruns on dev-02 at b67e7a879, load about 7: 3 passed, 2 failed.
• Sequential on dev-02, same worktree: origin/dev 5/5 passed at load about 5; the branch 4/5 passed at load 9–14.
• Interleaved A/B on compute01. Both debug binaries were built from the same worktree and run alternately with the same scripts/verify-session-exit-events.py --samples 8 (unchanged by this branch): 12/12 passed, 6 per binary. Idle windows were identical and exit-census latency was the same. The second column is the 1-minute load average.

7c082bddf run=1 rc=0 load=9.41 idle: seconds=2.200 seconds=2.200 seconds=2.201 p50_us=1947 max_us=2665  :: 
b67e7a879 run=1 rc=0 load=8.58 idle: seconds=2.200 seconds=2.200 seconds=2.201 p50_us=2159 max_us=2650  :: 
7c082bddf run=2 rc=0 load=8.04 idle: seconds=2.201 seconds=2.200 seconds=2.200 p50_us=2086 max_us=3592  :: 
b67e7a879 run=2 rc=0 load=7.81 idle: seconds=2.201 seconds=2.201 seconds=2.201 p50_us=2038 max_us=3677  :: 
7c082bddf run=3 rc=0 load=7.76 idle: seconds=2.201 seconds=2.201 seconds=2.201 p50_us=2075 max_us=5866  :: 
b67e7a879 run=3 rc=0 load=8.27 idle: seconds=2.201 seconds=2.205 seconds=2.201 p50_us=4418 max_us=9833  :: 
7c082bddf run=4 rc=0 load=8.22 idle: seconds=2.201 seconds=2.206 seconds=2.201 p50_us=2591 max_us=4245  :: 
b67e7a879 run=4 rc=0 load=10.08 idle: seconds=2.201 seconds=2.201 seconds=2.202 p50_us=2468 max_us=10384  :: 
7c082bddf run=5 rc=0 load=9.90 idle: seconds=2.203 seconds=2.201 seconds=2.201 p50_us=2589 max_us=8606  :: 
b67e7a879 run=5 rc=0 load=9.69 idle: seconds=2.203 seconds=2.200 seconds=2.211 p50_us=2718 max_us=5159  :: 
7c082bddf run=6 rc=0 load=9.28 idle: seconds=2.201 seconds=2.203 seconds=2.201 p50_us=2341 max_us=86381  :: 
b67e7a879 run=6 rc=0 load=9.62 idle: seconds=2.201 seconds=2.201 seconds=2.201 p50_us=2928 max_us=6781  :: 

Static sizes (report only, D243 amended): x86_64 23028624 bytes, delta +610368 bytes against the gate's baseline; aarch64 20969840 bytes, delta +516560 bytes.

Proxy integration on compute01 at b67e7a879: CARGO_TARGET_DIR=/home/claude/worktrees/j62b-r3-codex/target scripts/test-proxy-pebble.sh:

test namespace_serves_a_granted_mesh_backend_with_pebble_dns01 ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 9.69s
PEBBLE_EXIT=0

J116 round 3 findings, one commit each, with RED/GREEN evidence in the comments above:

1. P2, firewall opens all proxy ports to any TCP-granted peer: a7bfa336c fix(mesh): scope proxy ingress to applied routes. The nft and iptables renderer tests failed on 10080-10335 / --ctorigdstport 10080:10335 for a peer granted only TCP/3000. The registry_at clippy extraction (connect_proxy_and_mesh, proxy_plugin_at) is folded into this commit.
2. P2, rootful Traefik runs as root: 03892bcad fix(proxy): drop rootful Traefik privileges. Tests: process::traefik_child_obeys_the_rootful_identity_and_capability_boundary and identity::tests::user_namespace_root_without_traefik_id_mappings_is_rootless. The rootful identity assertions run only when the test runner is root; the compute01 run was unprivileged.
3. P2, mesh-only binds the public wildcard: b57814b02 fix(proxy): gate wildcard listeners until public routes apply. Tests: config_tests::rootless_public_entrypoints_bind_to_loopback and controller::tests::public_listener_gate_is_installed_before_the_first_route_snapshot.
4. P3, token redaction misses newlines: b67e7a879 fix(proxy): redact newline-terminated DNS tokens. Test: supervisor_tests::newline_terminated_provider_values_are_redacted_after_log_normalization.

Not done: a fully green gate receipt. Both rerun options are on the orchestrator: rerun verify fast at b67e7a879 on a quiet slot, or accept the A/B above. The behaviour changes in rounds 2–3 that need the independent review: rootless mode now binds the public entrypoints to 127.0.0.1 and rejects --public; rootful mode drops Traefik to UID/GID 65532 with only CAP_NET_BIND_SERVICE. No merge, no deploy.

J62b-c finish on branch `proxy-traefik-281-j62b`, head `b67e7a87916af4a2aed0bad0d4842d6b3f165598`, based on origin/dev `7c082bddfa36a7ea464e406522dbe67feda1630f`. `scripts/verify fast` on dev-02 at `b67e7a879` (`nix develop -c`, through build-slot), verbatim (log `dev-02:~/.cache/codex-scratch/j62b/verify-fast-c.log`): ```text (static builds: 285s) +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | FAIL (101) | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | static builds | PASS | +---------------------------+--------------+ VERIFY_EXIT=1 ``` The only failing row is `test`, and it has one failure outside the slice: `-p infraternal --test session_exit_events` (`idle_daemon_discovers_unattached_and_adopted_shim_exit_without_requests`). The first `client.open()` on the real agent socket hit the fixture's 1 s receive timeout (`TimeoutError: timed out`, `agent_fixture.py:129`). The empty-agent idle window just before it passed (`seconds=2.210`). On the previous head, the previous run failed a different timing test in `infraternal-server` (`transient_authority_stall`) and this one passed. Evidence that it is load flakiness, not this branch: - Isolated reruns on dev-02 at `b67e7a879`, load about 7: 3 passed, 2 failed. - Sequential on dev-02, same worktree: origin/dev 5/5 passed at load about 5; the branch 4/5 passed at load 9–14. - Interleaved A/B on compute01. Both debug binaries were built from the same worktree and run alternately with the same `scripts/verify-session-exit-events.py --samples 8` (unchanged by this branch): **12/12 passed, 6 per binary**. Idle windows were identical and exit-census latency was the same. The second column is the 1-minute load average. ```text 7c082bddf run=1 rc=0 load=9.41 idle: seconds=2.200 seconds=2.200 seconds=2.201 p50_us=1947 max_us=2665 :: b67e7a879 run=1 rc=0 load=8.58 idle: seconds=2.200 seconds=2.200 seconds=2.201 p50_us=2159 max_us=2650 :: 7c082bddf run=2 rc=0 load=8.04 idle: seconds=2.201 seconds=2.200 seconds=2.200 p50_us=2086 max_us=3592 :: b67e7a879 run=2 rc=0 load=7.81 idle: seconds=2.201 seconds=2.201 seconds=2.201 p50_us=2038 max_us=3677 :: 7c082bddf run=3 rc=0 load=7.76 idle: seconds=2.201 seconds=2.201 seconds=2.201 p50_us=2075 max_us=5866 :: b67e7a879 run=3 rc=0 load=8.27 idle: seconds=2.201 seconds=2.205 seconds=2.201 p50_us=4418 max_us=9833 :: 7c082bddf run=4 rc=0 load=8.22 idle: seconds=2.201 seconds=2.206 seconds=2.201 p50_us=2591 max_us=4245 :: b67e7a879 run=4 rc=0 load=10.08 idle: seconds=2.201 seconds=2.201 seconds=2.202 p50_us=2468 max_us=10384 :: 7c082bddf run=5 rc=0 load=9.90 idle: seconds=2.203 seconds=2.201 seconds=2.201 p50_us=2589 max_us=8606 :: b67e7a879 run=5 rc=0 load=9.69 idle: seconds=2.203 seconds=2.200 seconds=2.211 p50_us=2718 max_us=5159 :: 7c082bddf run=6 rc=0 load=9.28 idle: seconds=2.201 seconds=2.203 seconds=2.201 p50_us=2341 max_us=86381 :: b67e7a879 run=6 rc=0 load=9.62 idle: seconds=2.201 seconds=2.201 seconds=2.201 p50_us=2928 max_us=6781 :: ``` Static sizes (report only, D243 amended): x86_64 23028624 bytes, delta +610368 bytes against the gate's baseline; aarch64 20969840 bytes, delta +516560 bytes. Proxy integration on compute01 at `b67e7a879`: `CARGO_TARGET_DIR=/home/claude/worktrees/j62b-r3-codex/target scripts/test-proxy-pebble.sh`: ```text test namespace_serves_a_granted_mesh_backend_with_pebble_dns01 ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 9.69s PEBBLE_EXIT=0 ``` J116 round 3 findings, one commit each, with RED/GREEN evidence in the comments above: 1. P2, firewall opens all proxy ports to any TCP-granted peer: `a7bfa336c` fix(mesh): scope proxy ingress to applied routes. The nft and iptables renderer tests failed on `10080-10335` / `--ctorigdstport 10080:10335` for a peer granted only TCP/3000. The `registry_at` clippy extraction (`connect_proxy_and_mesh`, `proxy_plugin_at`) is folded into this commit. 2. P2, rootful Traefik runs as root: `03892bcad` fix(proxy): drop rootful Traefik privileges. Tests: `process::traefik_child_obeys_the_rootful_identity_and_capability_boundary` and `identity::tests::user_namespace_root_without_traefik_id_mappings_is_rootless`. The rootful identity assertions run only when the test runner is root; the compute01 run was unprivileged. 3. P2, mesh-only binds the public wildcard: `b57814b02` fix(proxy): gate wildcard listeners until public routes apply. Tests: `config_tests::rootless_public_entrypoints_bind_to_loopback` and `controller::tests::public_listener_gate_is_installed_before_the_first_route_snapshot`. 4. P3, token redaction misses newlines: `b67e7a879` fix(proxy): redact newline-terminated DNS tokens. Test: `supervisor_tests::newline_terminated_provider_values_are_redacted_after_log_normalization`. Not done: a fully green gate receipt. Both rerun options are on the orchestrator: rerun `verify fast` at `b67e7a879` on a quiet slot, or accept the A/B above. The behaviour changes in rounds 2–3 that need the independent review: rootless mode now binds the public entrypoints to `127.0.0.1` and rejects `--public`; rootful mode drops Traefik to UID/GID 65532 with only `CAP_NET_BIND_SERVICE`. No merge, no deploy.

kayg commented 2026-11-03 11:08:10 +00:00

Author

Owner

Copy link

J133 independent security review started (read-only). Target: proxy-traefik-281-j62b at b67e7a87916af4a2aed0bad0d4842d6b3f165598, diffed against origin/dev. Plan (~2 h): verify J116's four round-3 fixes against their repros, review the whole slice (firewall, privilege drop, binds, revocation timing, secrets, config injection, traversal, supervision), and run the root-gated process tests as root in a disposable worktree under /root on compute01 (removed afterwards). No code changes.

J133 independent security review started (read-only). Target: `proxy-traefik-281-j62b` at `b67e7a87916af4a2aed0bad0d4842d6b3f165598`, diffed against `origin/dev`. Plan (~2 h): verify J116's four round-3 fixes against their repros, review the whole slice (firewall, privilege drop, binds, revocation timing, secrets, config injection, traversal, supervision), and run the root-gated process tests as root in a disposable worktree under `/root` on compute01 (removed afterwards). No code changes.

kayg commented 2026-11-03 12:10:31 +00:00

Author

Owner

Copy link

J133 independent security review: REQUEST CHANGES

Target: proxy-traefik-281-j62b at b67e7a87916af4a2aed0bad0d4842d6b3f165598, diffed against origin/dev. This was a read-only review; no code was pushed. The root runs used a disposable clone in /root/j133 on compute01. That clone, its target dir, the netns namespaces and the temp files are all removed. Logs are kept on code-minipc01 at ~/Developer/Worktrees/j133-evidence/{run,run2}.log.

J116 round-3 fixes, checked against their repros

J116 finding	Status	Evidence
1. Proxy ports open to any peer with a TCP grant	Fixed as stated (but see P1-A)	netns, real nft and iptables-nft rulesets rendered by the branch: peer .2, granted TCP/3000 only, is BLOCKED to the selected pair 10081/10444 and to an unrelated Mesh service on 10082. No candidate ranges are rendered any more.
2. Rootful Traefik runs as root	Fixed at the child boundary. The production path is broken (P1-B) and loses its parent-death signal (P2-A)	As root, traefik_child_obeys_the_rootful_identity_and_capability_boundary ... ok: UID/GID 65532, NoNewPrivs=1, only CAP_NET_BIND_SERVICE.
3. Mesh-only mode binds the public wildcard	Fixed for rootful	netns, nft and iptables: underlay → :80/:443 BLOCKED while the gate is closed and CONNECTED once it is open; a Mesh peer without a 443 grant stays BLOCKED. Rootless binds loopback (code).
4. Token redaction misses newlines	Fixed for CR/LF, with a residual (P3-A)	supervisor.rs:193

Root test run (compute01, cargo test -p infraternal-plugin-proxy, euid 0)

--test process:  2 passed; 1 failed
  traefik_child_obeys_the_rootful_identity_and_capability_boundary ... ok
  traefik_child_receives_only_credential_file_paths_and_redacts_its_logs ... FAILED
    process.rs:168:51 read child invocation: Os { code: 2, kind: NotFound }
--lib:           83 passed; 5 failed
  abrupt_infraternal_parent_exit_stops_the_traefik_child        (fixture did not start the child)
  live_process_without_a_healthy_private_ping_is_reported_unhealthy
  mesh_grant_revocation_removes_the_route_before_narrowing_its_allowlist
  owned_supervisor_starts_on_first_exposure_drains_routes_and_keeps_acme_state
  unconfirmed_route_removal_stops_the_proxy_and_returns_a_timeout
    -> "configuration directory has unsafe owner or mode"

The round-3 GREEN receipts were all unprivileged runs. As root, 6 of the slice's tests fail.

Findings

P1-A: Mesh routes are unreachable for peers with port grants. The firewall drops the handshake after the first SYN.
crates/plugins/mesh/src/firewall.rs:475,484,569-603; crates/plugins/mesh/src/iptables.rs:431,574-610.
The proxy exception (@incoming*_proxy_sources . @proxy_ingress_ports) is only evaluated for ct state new. Established and reply packets jump to incoming_state and outgoing_state in nft, or hit the ESTABLISHED checks in iptables. Those only accept @incoming*_all or the exact granted (peer, port) tuples, and then drop. Only peers with TrafficPolicy::All can use a proxy route.
Repro: netns, branch-rendered rules, peer .3 granted TCP/8080, route backend 8080, selected pair 10081/10444:

control: peer .3 direct grant tcp/8080            got=CONNECTED expected=CONNECTED
proxy: peer .3 -> 10.254.0.1:10444                got=BLOCKED   expected=CONNECTED   (nft and iptables)
with the proxy exception inserted into *_state:   got=CONNECTED; peer .2 still BLOCKED

Neither the unit tests nor the Pebble test (which has no Mesh firewall) cover this path.
A second effect: connection_state::retire (connection_state.rs:34,104) judges every flow by permits(peer.inbound, dport). A listener-pair port is never a grant, so after the state chains are fixed, any grant narrowing anywhere would delete every proxied flow from every port-granted peer. That would break the WebSockets that D-engine choice 72714 must keep.
Fix:

• Accept established and reply packets for (proxy source, selected pair) in both backends.
• Make retire treat a flow to the selected pair as permitted while its source is still in the proxy source set.
• Add a netns test that runs the real Mesh firewall plus Traefik for a peer with a port grant.

P1-B: Rootful mode cannot start Traefik.
crates/plugins/proxy/src/controller.rs:723-776 and crates/plugins/proxy/src/supervisor.rs:1350-1357.
ensure_proxy_directory creates proxy/ with mode 0750 owned root:root, then rejects it because the group is not 65532. Even with a correct group, acquire_supervisor_lock requires its parent (that same proxy/ dir) to be 0700. Root probe from a disposable test:

J133 rootful=true
J133 ensure_proxy_directory=Err(PermissionDenied "proxy state directory has unsafe group or mode") mode/uid/gid=Ok((0o750, 0, 0))
J133 TraefikSupervisor::start err=Some(PermissionDenied "proxy state directory has unsafe ownership or mode")

Every rootful proxy publish fails, and the privilege drop has never run end to end.
Fix:

• Create the dir at 0700, then fchown it to root:65532 and chmod it to 0750 through the fd.
• Keep the supervisor lock in the agent-only 0700 state dir.
• Run the supervisor tests as root, for example in a user namespace with 65532 mapped, as part of the gate.

P2-A: The rootful Traefik loses PR_SET_PDEATHSIG, so an agent crash leaves an orphan Traefik.
crates/infraternal-shim/src/pty.rs:932 sets the parent-death signal in an earlier pre_exec hook. Then drop_proxy_root calls setuid (pty.rs:1041), and the kernel's commit_creds clears pdeath_signal when euid changes. Root probe, run as a fake Traefik child:

J133 child line: J133_PDEATHSIG=0

Scenario: the agent is OOM-killed or SIGKILLed. Traefik (65532) keeps the wildcard 80/443 and the Mesh listener, along with its last routes and allowlists. The restarted agent's Mesh firewall starts with default proxy ingress, so the public gate is gone. The new Traefik cannot bind its old pair, and nothing owns the orphan. abrupt_infraternal_parent_exit_stops_the_traefik_child (supervisor_tests.rs:130) does not exercise the rootful path.
Fix:

• Set PDEATHSIG as the last step after the credential drop, and recheck getppid() after it.
• On start, reap any stale proxy child, for example through a lock fd the child inherits, or a cgroup.

P2-B: The fixed shared identity 65532 has no collision check.
crates/plugins/proxy/src/identity.rs:4-6. 65532 is the conventional nonroot UID in common container images. Any other process on the host running as 65532, such as a rootful container without userns, can:

• ptrace Traefik, which is dumpable after execve because no privilege is gained;
• read /proc/<pid>/fd/* and environ;
• signal it;
• reach the child-owned 0600 DNS token and ACME key files.

A compromised Traefik also gains every host file owned by 65532.
Fix: allocate a dedicated system user and group at install, or a recorded unused ID, and refuse to start if other processes or files already use that ID.

P2-C: A per-route revocation does not end live upgraded connections.
controller.rs:440 says "existing connections may drain for up to 10 seconds". In fact, Traefik keeps hijacked WebSockets across dynamic reloads. Once P1-A is fixed, here is the case. Peer X keeps a grant for route B but loses route A. Its open WebSocket or terminal on A then stays open with no time limit, because the firewall still admits X to the shared pair and retire sees no reach change for the pair.
Fix: when a source is removed from a Mesh route, delete conntrack flows from that source to the selected pair. This also drops X's other proxied flows, which is acceptable for a revocation. Also correct the message.

P3-A: Redaction misses other whitespace. supervisor.rs:193 only strips CR/LF. The DNS client library Traefik uses trims all leading and trailing whitespace from *_FILE values, so a file like "tok \n" or "\ttok" still leaks tok. Fix: also add secret.trim() (Unicode whitespace) to the replacement set.

P3-B: The public gate outlives the listener and has a broad scope. controller.rs:525 keeps public_listener_pair for as long as the controller holds a supervisor. That includes after the drain stops Traefik, so any later service on 80/443 is silently dropped on every non-lo interface. The nft rule (firewall.rs input chain, tcp dport {80,443} drop) also covers IPv6 services that bind specific addresses. Fix: clear the gate when the child stops, and scope it to the addresses Traefik actually binds.

P3-C: The health port is chosen by a TOCTOU bind-and-release. controller.rs:690. A local user who squats the port before Traefik binds it can answer /ping and /api/rawdata, which forges apply acks. Traefik then exits on the bind failure, so the impact is DoS rather than stale routes. Fix: verify that the listening socket belongs to the child (match the socket inode against its fds) before trusting acks.

P3-D: A test fails as root. tests/process.rs:168 (traefik_child_receives_only_credential_file_paths_and_redacts_its_logs) fails as root (log above). The capture path is not writable or reachable for 65532.

Clean areas

• Config injection: host and id labels are restricted to [a-z0-9-], dynamic config is serde JSON, and static config uses JSON-escaped TOML strings.
• ipAllowList: uses RemoteAddr; no forwarded-header trust is configured.
• Generation API route: loopback only and gated on a per-generation UUID.
• Tokens: never placed in argv or env values; only /proc/self/fd paths are passed.
• Custody: fd-based fchown, O_NOFOLLOW and nlink checks. A compromised child that holds the provider dir fd can only cause DoS, not escalate.
• Extraction: allowlisted, as J116 found.
• Revocation ack: J99 fix holds. The route is removed and its generation acked before the Mesh action is acknowledged.

For D330 (rootless --public allowed, wildcard 10080/10443 or the next free pair)

Things that would make the next round unsafe, for that job to design against:

1. No gate exists in rootless mode. controller.rs:525 only creates the public gate when state.rootful. If rootless binds the wildcard pair for the whole process lifetime, J116#3 comes back: Traefik's parser is exposed to the network while only Mesh routes exist. The wildcard must accept only while a public route is applied, and turning it on or off must not restart the Mesh listeners. One way is a separate public-only Traefik process, started after its first snapshot is acked on loopback and stopped after the last public route.
2. Mesh routes must never attach to http/https. Today render_published_service puts non-public routes only on mesh-http/mesh-https, including the redirect router. Keep a test that asserts this, because the rootless public entrypoint would now be reachable from the network.
3. Port selection must probe. config.rs:133 returns 10080/10443 for rootless without probing. "Next free pair" has to probe the wildcard on both v4 and v6 (Go listens dual-stack on 0.0.0.0) and must exclude the Mesh pair.
4. Ordering: apply and ack the public snapshot before the wildcard accepts traffic. On removal, the listener must close before the route is reported as removed. Without a firewall, the listener lifecycle is the only control.
5. P2-A (orphan) gets worse. A rootless orphan would hold a wildcard listener with no gate at all.
6. Warning text: say that only the host firewall or router limits reachability, and that the Mesh firewall does not filter the public pair.

## J133 independent security review: **REQUEST CHANGES** Target: `proxy-traefik-281-j62b` at `b67e7a87916af4a2aed0bad0d4842d6b3f165598`, diffed against `origin/dev`. This was a read-only review; no code was pushed. The root runs used a disposable clone in `/root/j133` on compute01. That clone, its target dir, the netns namespaces and the temp files are all removed. Logs are kept on code-minipc01 at `~/Developer/Worktrees/j133-evidence/{run,run2}.log`. ### J116 round-3 fixes, checked against their repros | J116 finding | Status | Evidence | |---|---|---| | 1. Proxy ports open to any peer with a TCP grant | **Fixed** as stated (but see P1-A) | netns, real nft and iptables-nft rulesets rendered by the branch: peer `.2`, granted TCP/3000 only, is BLOCKED to the selected pair 10081/10444 and to an unrelated Mesh service on 10082. No candidate ranges are rendered any more. | | 2. Rootful Traefik runs as root | **Fixed at the child boundary**. The production path is broken (P1-B) and loses its parent-death signal (P2-A) | As root, `traefik_child_obeys_the_rootful_identity_and_capability_boundary ... ok`: UID/GID 65532, NoNewPrivs=1, only CAP_NET_BIND_SERVICE. | | 3. Mesh-only mode binds the public wildcard | **Fixed for rootful** | netns, nft and iptables: underlay → `:80`/`:443` BLOCKED while the gate is closed and CONNECTED once it is open; a Mesh peer without a 443 grant stays BLOCKED. Rootless binds loopback (code). | | 4. Token redaction misses newlines | **Fixed for CR/LF**, with a residual (P3-A) | `supervisor.rs:193` | ### Root test run (compute01, `cargo test -p infraternal-plugin-proxy`, euid 0) ```text --test process: 2 passed; 1 failed traefik_child_obeys_the_rootful_identity_and_capability_boundary ... ok traefik_child_receives_only_credential_file_paths_and_redacts_its_logs ... FAILED process.rs:168:51 read child invocation: Os { code: 2, kind: NotFound } --lib: 83 passed; 5 failed abrupt_infraternal_parent_exit_stops_the_traefik_child (fixture did not start the child) live_process_without_a_healthy_private_ping_is_reported_unhealthy mesh_grant_revocation_removes_the_route_before_narrowing_its_allowlist owned_supervisor_starts_on_first_exposure_drains_routes_and_keeps_acme_state unconfirmed_route_removal_stops_the_proxy_and_returns_a_timeout -> "configuration directory has unsafe owner or mode" ``` The round-3 GREEN receipts were all unprivileged runs. As root, 6 of the slice's tests fail. ### Findings **P1-A: Mesh routes are unreachable for peers with port grants. The firewall drops the handshake after the first SYN.** `crates/plugins/mesh/src/firewall.rs:475,484,569-603`; `crates/plugins/mesh/src/iptables.rs:431,574-610`. The proxy exception (`@incoming*_proxy_sources . @proxy_ingress_ports`) is only evaluated for `ct state new`. Established and reply packets jump to `incoming_state` and `outgoing_state` in nft, or hit the ESTABLISHED checks in iptables. Those only accept `@incoming*_all` or the exact granted `(peer, port)` tuples, and then drop. Only peers with `TrafficPolicy::All` can use a proxy route. Repro: netns, branch-rendered rules, peer `.3` granted TCP/8080, route backend 8080, selected pair 10081/10444: ```text control: peer .3 direct grant tcp/8080 got=CONNECTED expected=CONNECTED proxy: peer .3 -> 10.254.0.1:10444 got=BLOCKED expected=CONNECTED (nft and iptables) with the proxy exception inserted into *_state: got=CONNECTED; peer .2 still BLOCKED ``` Neither the unit tests nor the Pebble test (which has no Mesh firewall) cover this path. A second effect: `connection_state::retire` (`connection_state.rs:34,104`) judges every flow by `permits(peer.inbound, dport)`. A listener-pair port is never a grant, so after the state chains are fixed, any grant narrowing anywhere would delete every proxied flow from every port-granted peer. That would break the WebSockets that D-engine choice 72714 must keep. Fix: - Accept established and reply packets for `(proxy source, selected pair)` in both backends. - Make `retire` treat a flow to the selected pair as permitted while its source is still in the proxy source set. - Add a netns test that runs the real Mesh firewall plus Traefik for a peer with a port grant. **P1-B: Rootful mode cannot start Traefik.** `crates/plugins/proxy/src/controller.rs:723-776` and `crates/plugins/proxy/src/supervisor.rs:1350-1357`. `ensure_proxy_directory` creates `proxy/` with mode 0750 owned `root:root`, then rejects it because the group is not 65532. Even with a correct group, `acquire_supervisor_lock` requires its parent (that same `proxy/` dir) to be 0700. Root probe from a disposable test: ```text J133 rootful=true J133 ensure_proxy_directory=Err(PermissionDenied "proxy state directory has unsafe group or mode") mode/uid/gid=Ok((0o750, 0, 0)) J133 TraefikSupervisor::start err=Some(PermissionDenied "proxy state directory has unsafe ownership or mode") ``` Every rootful `proxy publish` fails, and the privilege drop has never run end to end. Fix: - Create the dir at 0700, then `fchown` it to `root:65532` and chmod it to 0750 through the fd. - Keep the supervisor lock in the agent-only 0700 state dir. - Run the supervisor tests as root, for example in a user namespace with 65532 mapped, as part of the gate. **P2-A: The rootful Traefik loses PR_SET_PDEATHSIG, so an agent crash leaves an orphan Traefik.** `crates/infraternal-shim/src/pty.rs:932` sets the parent-death signal in an earlier `pre_exec` hook. Then `drop_proxy_root` calls `setuid` (`pty.rs:1041`), and the kernel's `commit_creds` clears `pdeath_signal` when euid changes. Root probe, run as a fake Traefik child: ```text J133 child line: J133_PDEATHSIG=0 ``` Scenario: the agent is OOM-killed or SIGKILLed. Traefik (65532) keeps the wildcard 80/443 and the Mesh listener, along with its last routes and allowlists. The restarted agent's Mesh firewall starts with default proxy ingress, so the public gate is gone. The new Traefik cannot bind its old pair, and nothing owns the orphan. `abrupt_infraternal_parent_exit_stops_the_traefik_child` (`supervisor_tests.rs:130`) does not exercise the rootful path. Fix: - Set PDEATHSIG as the last step after the credential drop, and recheck `getppid()` after it. - On start, reap any stale proxy child, for example through a lock fd the child inherits, or a cgroup. **P2-B: The fixed shared identity 65532 has no collision check.** `crates/plugins/proxy/src/identity.rs:4-6`. 65532 is the conventional `nonroot` UID in common container images. Any other process on the host running as 65532, such as a rootful container without userns, can: - ptrace Traefik, which is dumpable after `execve` because no privilege is gained; - read `/proc/<pid>/fd/*` and environ; - signal it; - reach the child-owned 0600 DNS token and ACME key files. A compromised Traefik also gains every host file owned by 65532. Fix: allocate a dedicated system user and group at install, or a recorded unused ID, and refuse to start if other processes or files already use that ID. **P2-C: A per-route revocation does not end live upgraded connections.** `controller.rs:440` says "existing connections may drain for up to 10 seconds". In fact, Traefik keeps hijacked WebSockets across dynamic reloads. Once P1-A is fixed, here is the case. Peer X keeps a grant for route B but loses route A. Its open WebSocket or terminal on A then stays open with no time limit, because the firewall still admits X to the shared pair and `retire` sees no reach change for the pair. Fix: when a source is removed from a Mesh route, delete conntrack flows from that source to the selected pair. This also drops X's other proxied flows, which is acceptable for a revocation. Also correct the message. **P3-A: Redaction misses other whitespace.** `supervisor.rs:193` only strips CR/LF. The DNS client library Traefik uses trims all leading and trailing whitespace from `*_FILE` values, so a file like `"tok \n"` or `"\ttok"` still leaks `tok`. Fix: also add `secret.trim()` (Unicode whitespace) to the replacement set. **P3-B: The public gate outlives the listener and has a broad scope.** `controller.rs:525` keeps `public_listener_pair` for as long as the controller holds a supervisor. That includes after the drain stops Traefik, so any later service on 80/443 is silently dropped on every non-`lo` interface. The nft rule (`firewall.rs` input chain, `tcp dport {80,443} drop`) also covers IPv6 services that bind specific addresses. Fix: clear the gate when the child stops, and scope it to the addresses Traefik actually binds. **P3-C: The health port is chosen by a TOCTOU bind-and-release.** `controller.rs:690`. A local user who squats the port before Traefik binds it can answer `/ping` and `/api/rawdata`, which forges apply acks. Traefik then exits on the bind failure, so the impact is DoS rather than stale routes. Fix: verify that the listening socket belongs to the child (match the socket inode against its fds) before trusting acks. **P3-D: A test fails as root.** `tests/process.rs:168` (`traefik_child_receives_only_credential_file_paths_and_redacts_its_logs`) fails as root (log above). The capture path is not writable or reachable for 65532. ### Clean areas - **Config injection:** host and id labels are restricted to `[a-z0-9-]`, dynamic config is serde JSON, and static config uses JSON-escaped TOML strings. - **ipAllowList:** uses RemoteAddr; no forwarded-header trust is configured. - **Generation API route:** loopback only and gated on a per-generation UUID. - **Tokens:** never placed in argv or env values; only `/proc/self/fd` paths are passed. - **Custody:** fd-based `fchown`, `O_NOFOLLOW` and `nlink` checks. A compromised child that holds the provider dir fd can only cause DoS, not escalate. - **Extraction:** allowlisted, as J116 found. - **Revocation ack:** J99 fix holds. The route is removed and its generation acked before the Mesh action is acknowledged. ### For D330 (rootless `--public` allowed, wildcard 10080/10443 or the next free pair) Things that would make the next round unsafe, for that job to design against: 1. **No gate exists in rootless mode.** `controller.rs:525` only creates the public gate when `state.rootful`. If rootless binds the wildcard pair for the whole process lifetime, J116#3 comes back: Traefik's parser is exposed to the network while only Mesh routes exist. The wildcard must accept only while a public route is applied, and turning it on or off must not restart the Mesh listeners. One way is a separate public-only Traefik process, started after its first snapshot is acked on loopback and stopped after the last public route. 2. **Mesh routes must never attach to `http`/`https`.** Today `render_published_service` puts non-public routes only on `mesh-http`/`mesh-https`, including the redirect router. Keep a test that asserts this, because the rootless public entrypoint would now be reachable from the network. 3. **Port selection must probe.** `config.rs:133` returns 10080/10443 for rootless without probing. "Next free pair" has to probe the wildcard on both v4 and v6 (Go listens dual-stack on `0.0.0.0`) and must exclude the Mesh pair. 4. **Ordering:** apply and ack the public snapshot before the wildcard accepts traffic. On removal, the listener must close before the route is reported as removed. Without a firewall, the listener lifecycle is the only control. 5. **P2-A (orphan) gets worse.** A rootless orphan would hold a wildcard listener with no gate at all. 6. **Warning text:** say that only the host firewall or router limits reachability, and that the Mesh firewall does not filter the public pair.

kayg commented 2026-11-03 12:13:41 +00:00

Author

Owner

Copy link

J62c (round 4 of slice 1) started on branch proxy-traefik-281-j62b (worktree branch j62c on code, pushed back to proxy-traefik-281-j62b).

• Previous head: b67e7a879. Rebased cleanly onto origin/dev 00ec71000 (37 branch commits replayed, no conflicts).
• Scope: J133's findings P1-A, P1-B, P2-A, P2-B, P2-C, P3-A to P3-D (comment 76880), each its own commit with a test that fails first, then D330 (rootless --public).
• Root-only tests run in a throwaway clone under /root on compute01, cleaned up after.

A plan with time estimates follows once I've read the controller and supervisor.

J62c (round 4 of slice 1) started on branch `proxy-traefik-281-j62b` (worktree branch `j62c` on code, pushed back to `proxy-traefik-281-j62b`). - Previous head: `b67e7a879`. Rebased cleanly onto origin/dev `00ec71000` (37 branch commits replayed, no conflicts). - Scope: J133's findings P1-A, P1-B, P2-A, P2-B, P2-C, P3-A to P3-D (comment 76880), each its own commit with a test that fails first, then D330 (rootless `--public`). - Root-only tests run in a throwaway clone under /root on compute01, cleaned up after. A plan with time estimates follows once I've read the controller and supervisor.

kayg commented 2026-11-03 12:16:11 +00:00

Author

Owner

Copy link

J62c plan (one commit per item, each with a test that fails first; RED and GREEN posted here):

1. P1-A (~1.5 h): in both firewall backends, accept established and reply packets for (proxy source, selected pair) in the *_state chains. connection_state::retire gets the old and new MeshProxyIngress: a flow to the selected pair stays permitted while its source is still a proxy source. Tests: rendered-rule unit tests for nft and iptables, a retire decision unit test, and an ignored netns test (real Mesh firewall, port-granted peer, data after the SYN) run as root.
2. P1-B (~1 h): the proxy dir is created at 0700, then fchown root:child-group and chmod 0750 through the fd. The supervisor lock moves into the agent-only 0700 state dir (a lock_directory field in SupervisorSettings). The 5 supervisor tests pass as root.
3. P2-A (~1.5 h): after the credential drop, the shim sets the parent-death signal (SIGKILL for the proxy) and rechecks getppid(). Every child inherits a child-lock fd. On start, while the agent holds the supervisor lock, a held child lock means an orphan from a dead agent: its holders are found by /proc/*/fd inode match, killed and reaped before Traefik starts. Root test: the child reports a nonzero PDEATHSIG after the drop, and a planted orphan holding the child lock is reaped.
4. P2-B (~1.5 h): dedicated system user/group infraternal-proxy, resolved at runtime and created with useradd --system (busybox adduser -S fallback) on first rootful start. Start is refused if the UID is 0, if another passwd/group entry shares the ID, or if any process outside our own child runs with it. rootful() requires the namespace to map the whole 0–65535 range. A full file-system scan for files owned by the ID is not done (cost); I note this in the commit.
5. P2-C (~1 h): when a source loses any proxied route, its conntrack flows to the selected pair are deleted. The unpublish message drops "may drain for up to 10 seconds".
6. P3-A…D (~1.5 h, one commit each): redact secret.trim(); clear the public gate when Traefik stops, and scope the nft/iptables drop to the IPv4 and IPv6 addresses Traefik binds; health-port ack is trusted only when the listening socket's inode is among the child's fds; tests/process.rs capture works as root.
7. D330 (~3 h): rootless --public runs a separate public-only Traefik process (own static/dynamic config, ACME file, health port, lock). It starts with its first public snapshot already written and acked, and is stopped (listener closed, child reaped) before the last public route's removal is reported. The rootless Mesh process gets no public entrypoints, so it never restarts for public changes. The public pair is probed on 0.0.0.0 and [::] from 10080/10443 and skips the Mesh pair. The publish result warns with the port, and says the Mesh firewall does not filter it: only this machine's firewall or the router limits who can reach it. Tests cover the rendered configs (Mesh routes never on http/https), the controller ordering, and the probe.
8. Gate (~1.5 h): root runs in a throwaway /root clone on compute01, unprivileged runs, scripts/verify fast.

Total ~12 h. A known residual for D330: Traefik binds its entrypoints a few milliseconds before its file provider loads the first snapshot, so a connection in that window gets a 404 from the default handler. No route other than the intended public one is ever served. I'll document it rather than build a socket handoff.

J62c plan (one commit per item, each with a test that fails first; RED and GREEN posted here): 1. **P1-A** (~1.5 h): in both firewall backends, accept established and reply packets for `(proxy source, selected pair)` in the `*_state` chains. `connection_state::retire` gets the old and new `MeshProxyIngress`: a flow to the selected pair stays permitted while its source is still a proxy source. Tests: rendered-rule unit tests for nft and iptables, a `retire` decision unit test, and an ignored netns test (real Mesh firewall, port-granted peer, data after the SYN) run as root. 2. **P1-B** (~1 h): the proxy dir is created at 0700, then `fchown` root:child-group and chmod 0750 through the fd. The supervisor lock moves into the agent-only 0700 state dir (a `lock_directory` field in `SupervisorSettings`). The 5 supervisor tests pass as root. 3. **P2-A** (~1.5 h): after the credential drop, the shim sets the parent-death signal (SIGKILL for the proxy) and rechecks `getppid()`. Every child inherits a child-lock fd. On start, while the agent holds the supervisor lock, a held child lock means an orphan from a dead agent: its holders are found by `/proc/*/fd` inode match, killed and reaped before Traefik starts. Root test: the child reports a nonzero PDEATHSIG after the drop, and a planted orphan holding the child lock is reaped. 4. **P2-B** (~1.5 h): dedicated system user/group `infraternal-proxy`, resolved at runtime and created with `useradd --system` (busybox `adduser -S` fallback) on first rootful start. Start is refused if the UID is 0, if another passwd/group entry shares the ID, or if any process outside our own child runs with it. `rootful()` requires the namespace to map the whole 0–65535 range. A full file-system scan for files owned by the ID is not done (cost); I note this in the commit. 5. **P2-C** (~1 h): when a source loses any proxied route, its conntrack flows to the selected pair are deleted. The unpublish message drops "may drain for up to 10 seconds". 6. **P3-A…D** (~1.5 h, one commit each): redact `secret.trim()`; clear the public gate when Traefik stops, and scope the nft/iptables drop to the IPv4 and IPv6 addresses Traefik binds; health-port ack is trusted only when the listening socket's inode is among the child's fds; `tests/process.rs` capture works as root. 7. **D330** (~3 h): rootless `--public` runs a separate public-only Traefik process (own static/dynamic config, ACME file, health port, lock). It starts with its first public snapshot already written and acked, and is stopped (listener closed, child reaped) before the last public route's removal is reported. The rootless Mesh process gets no public entrypoints, so it never restarts for public changes. The public pair is probed on 0.0.0.0 and [::] from 10080/10443 and skips the Mesh pair. The publish result warns with the port, and says the Mesh firewall does not filter it: only this machine's firewall or the router limits who can reach it. Tests cover the rendered configs (Mesh routes never on `http`/`https`), the controller ordering, and the probe. 8. Gate (~1.5 h): root runs in a throwaway /root clone on compute01, unprivileged runs, `scripts/verify fast`. Total ~12 h. A known residual for D330: Traefik binds its entrypoints a few milliseconds before its file provider loads the first snapshot, so a connection in that window gets a 404 from the default handler. No route other than the intended public one is ever served. I'll document it rather than build a socket handoff.

kayg commented 2026-11-03 13:05:16 +00:00

Author

Owner

Copy link

J62c progress. All fixes and D330 are committed and pushed on proxy-traefik-281-j62b, head 9073b90ca, rebased on origin/dev 00ec71000. Each commit is fmt-clean on its own.

Item	Commit
New, P1: nft Mesh table failed to install while no Mesh route is published (details below)	84040190a
P1-A: whole proxied flows for port-granted peers; retire keeps them	4f792ba4d
P1-B: rootful proxy dir created 0700, then repaired to 0750 through its fd; owner lock in an agent-only dir	a34aa40d6
P2-A: parent-death signal (SIGKILL) re-armed after the credential drop, parent rechecked; child-inherited lock reaps orphans on start	2e39ea816
P2-B: dedicated infraternal-proxy system account; collision checks on passwd, group and running processes	3433cf9da
P2-C: a source that loses any route loses its flows to the pair; message corrected	ac9dcf497
P3-A: redact secret.trim()	a2ff0e583
P3-B: the supervisor owns the public gate, closed before each spawn and cleared after each reap	e7346e299
P3-C: health acks are trusted only when the child's fds hold the listening socket	8bbdfcbcd
D330: rootless --public through a separate public-only process	f97d2d9f0
P3-D: the process test captures through the inherited custody fd	9073b90ca

New finding (P1), found while running the P1-A RED against the unfixed branch. On the nft backend, Firewall::apply fails with PermissionDenied on every node with no Mesh proxy route, so the Mesh table never installs. The renderer emitted set proxy_ingress_ports { type inet_service; elements = { }; }, which nft rejects:

$ printf 'table inet t {\n set s { type inet_service; elements = {  }; }\n}\n' | unshare -Urn nft -c -f -
/dev/stdin:2:43-43: Error: syntax error, unexpected '}'
$ printf 'table inet t {\n set s { type inet_service; }\n}\n' | unshare -Urn nft -c -f -   # exit 0

A unit test pinned the broken string. Fixed in 84040190a, which is now the first commit of this round.

P3-D root cause. Under nix develop, root's TMPDIR is private, and the dropped child can't traverse it to write its capture file. The other boundary tests reach their files only through inherited descriptors, so they passed.

P1-A unit RED is recorded (compute01, unfixed source plus the new tests):

test iptables::tests::proxied_flows_keep_passing_the_state_checks_after_their_syn ... FAILED
test firewall::tests::proxied_flows_keep_passing_the_state_chains_after_their_syn ... FAILED
test result: FAILED. 13 passed; 2 failed; 4 ignored

Queued now:

• compute01: namespace RED/GREEN for the empty set, P1-A and P2-C, and the root run (REDs for P1-B, P3-D, P2-A, P2-B, then GREEN of proxy, shim and Mesh as root, then cleanup).
• dev-02: unprivileged REDs for P2-A, P3-A and P3-C, then GREEN tests and clippy.

Logs: ~/j62c-logs/ on both VMs, /root/j62c-root.log on compute01. One target dir per VM: ~/.cache/codex-scratch/j62c/target.

J62c progress. All fixes and D330 are committed and pushed on `proxy-traefik-281-j62b`, head `9073b90ca`, rebased on origin/dev `00ec71000`. Each commit is fmt-clean on its own. | Item | Commit | |---|---| | **New, P1:** nft Mesh table failed to install while no Mesh route is published (details below) | `84040190a` | | P1-A: whole proxied flows for port-granted peers; `retire` keeps them | `4f792ba4d` | | P1-B: rootful proxy dir created 0700, then repaired to 0750 through its fd; owner lock in an agent-only dir | `a34aa40d6` | | P2-A: parent-death signal (SIGKILL) re-armed after the credential drop, parent rechecked; child-inherited lock reaps orphans on start | `2e39ea816` | | P2-B: dedicated `infraternal-proxy` system account; collision checks on passwd, group and running processes | `3433cf9da` | | P2-C: a source that loses any route loses its flows to the pair; message corrected | `ac9dcf497` | | P3-A: redact `secret.trim()` | `a2ff0e583` | | P3-B: the supervisor owns the public gate, closed before each spawn and cleared after each reap | `e7346e299` | | P3-C: health acks are trusted only when the child's fds hold the listening socket | `8bbdfcbcd` | | D330: rootless `--public` through a separate public-only process | `f97d2d9f0` | | P3-D: the process test captures through the inherited custody fd | `9073b90ca` | **New finding (P1), found while running the P1-A RED against the unfixed branch.** On the nft backend, `Firewall::apply` fails with PermissionDenied on every node with no Mesh proxy route, so the Mesh table never installs. The renderer emitted `set proxy_ingress_ports { type inet_service; elements = { }; }`, which nft rejects: ```text $ printf 'table inet t {\n set s { type inet_service; elements = { }; }\n}\n' | unshare -Urn nft -c -f - /dev/stdin:2:43-43: Error: syntax error, unexpected '}' $ printf 'table inet t {\n set s { type inet_service; }\n}\n' | unshare -Urn nft -c -f - # exit 0 ``` A unit test pinned the broken string. Fixed in `84040190a`, which is now the first commit of this round. **P3-D root cause.** Under `nix develop`, root's `TMPDIR` is private, and the dropped child can't traverse it to write its capture file. The other boundary tests reach their files only through inherited descriptors, so they passed. P1-A unit RED is recorded (compute01, unfixed source plus the new tests): ```text test iptables::tests::proxied_flows_keep_passing_the_state_checks_after_their_syn ... FAILED test firewall::tests::proxied_flows_keep_passing_the_state_chains_after_their_syn ... FAILED test result: FAILED. 13 passed; 2 failed; 4 ignored ``` Queued now: - compute01: namespace RED/GREEN for the empty set, P1-A and P2-C, and the root run (REDs for P1-B, P3-D, P2-A, P2-B, then GREEN of proxy, shim and Mesh as root, then cleanup). - dev-02: unprivileged REDs for P2-A, P3-A and P3-C, then GREEN tests and clippy. Logs: `~/j62c-logs/` on both VMs, `/root/j62c-root.log` on compute01. One target dir per VM: `~/.cache/codex-scratch/j62c/target`.

kayg commented 2026-11-03 16:01:07 +00:00

Author

Owner

Copy link

J62c: red-to-green evidence per finding

Branch proxy-traefik-281-j62b, head 36384e212. Each RED ran the fix's own new tests on the commit before that fix (as an overlay). Each GREEN ran at head. Some REDs ran before a later rebase and name the SHAs current then. Since then the commits changed only by formatting, clippy and the test fixups noted below. Hosts: compute01 (x86; namespace runs under unshare -Urn, root runs as real root in a throwaway clone under /root), and dev-02 (arm64, unprivileged).

New P1: nft Mesh table could not install with no Mesh route (84040190a)

RED (compute01, P1-A namespace test on b2965d89e):

nft_admits_a_port_granted_peer_to_the_proxy_listener_for_the_whole_flow panicked at proxy_ingress_namespace.rs:208:29:
called `Result::unwrap()` on an `Err` value: Kind(PermissionDenied)      <- firewall.apply() itself

Root cause: nft rejects elements = { } (repro in the comment above). GREEN: with the fix alone (84040190a), the same test gets past apply and fails only on P1-A's own assertion (next section).

P1-A: whole proxied flows for port-granted peers (fa894071f)

RED (compute01, tests on 84040190a):

test iptables::tests::proxied_flows_keep_passing_the_state_checks_after_their_syn ... FAILED
test firewall::tests::proxied_flows_keep_passing_the_state_chains_after_their_syn ... FAILED
nft_admits_a_port_granted_peer_to_the_proxy_listener_for_the_whole_flow:
  assertion failed: a port-granted peer reaches the pair   left: "BLOCKED"  right: "connected"
nf_tables_admits_a_port_granted_peer_to_the_proxy_listener_for_the_whole_flow:
  assertion failed: a port-granted peer reaches the pair   left: "BLOCKED"  right: "connected"

GREEN (compute01 unshare -Urn, and again as real root under unshare -n). The test covers several round trips on one flow from a peer granted only TCP/8080, a peer with no route grant refused, and the flow surviving an unrelated narrowing:

test nft_admits_a_port_granted_peer_to_the_proxy_listener_for_the_whole_flow ... ok
test nf_tables_admits_a_port_granted_peer_to_the_proxy_listener_for_the_whole_flow ... ok
test connection_state::tests::a_port_granted_peer_keeps_its_proxied_flows_while_it_is_a_proxy_source ... ok

P1-B: rootful proxy state (e13b6267a)

RED (root, compute01, proxy code before the fix):

test controller::tests::a_new_proxy_state_directory_is_prepared_for_the_child ... FAILED
test supervisor_tests::live_process_without_a_healthy_private_ping_is_reported_unhealthy ... FAILED
test supervisor_tests::abrupt_infraternal_parent_exit_stops_the_traefik_child ... FAILED
test supervisor_tests::owned_supervisor_starts_on_first_exposure_drains_routes_and_keeps_acme_state ... FAILED
test supervisor_tests::mesh_grant_revocation_removes_the_route_before_narrowing_its_allowlist ... FAILED
test supervisor_tests::unconfirmed_route_removal_stops_the_proxy_and_returns_a_timeout ... FAILED

GREEN (root, head 36384e212): all six ... ok, and the proxy lib test result: ok. 102 passed; 0 failed.

P2-A: parent-death signal after the drop, orphan reaping (da2f7ed3d)

RED. The probe reads the signal on the exec'd main thread of the child:

unprivileged (dev-02):  the_parent_death_signal_survives_the_credential_drop  left: "15"  right: "9"   (SIGTERM; a graceful drain kept the listeners)
root (compute01):       the_parent_death_signal_survives_the_credential_drop  left: "0"   right: "9"   (cleared by setuid, as J133 found)
both:                   a_new_supervisor_stops_an_orphan_that_holds_the_child_lock ... FAILED

GREEN (root and unprivileged, head):

test supervisor_tests::the_parent_death_signal_survives_the_credential_drop ... ok
test supervisor_tests::a_new_supervisor_stops_an_orphan_that_holds_the_child_lock ... ok
test supervisor_tests::abrupt_infraternal_parent_exit_stops_the_traefik_child ... ok

P2-B: dedicated account (ce9c401b2)

RED (root, on da2f7ed3d): the_rootful_child_identity_is_not_the_shared_conventional_id left: 65532 right: 65532.
GREEN (root, head): the_rootful_child_runs_as_the_dedicated_account ... ok. The run created infraternal-proxy:x:999:987::/nonexistent:/usr/sbin/nologin through useradd --system. The cleanup removed it (account after: none). Two issues were found and fixed during the root runs:

• First use raced when several processes created the account at once; the loser's useradd failed. Creation is now serialized, and the account is looked up again after a failed tool run.
• The collision check counted the agent's own children. They are now exempt.

P2-C: revoked route ends its source's flows (73cc570b5)

RED (compute01, namespace test on the commit before):

nft_revoking_a_route_ends_the_sources_open_proxied_flows:
  assertion failed: a revoked route ends that source's open flows and only that source's
  left: ["echo", "echo"]   right: ["lost", "echo"]
nf_tables_revoking_a_route_ends_the_sources_open_proxied_flows:  (same)

GREEN (compute01 userns, and as real root): both ... ok. The test covers several cases:

• The narrowed peer's open flow ends; an unaffected peer's flow survives.
• The narrowed peer can reconnect to the route it kept.
• Unpublishing a route ends every flow that could use it.

P3-A: trimmed token (c9ebab8ea)

RED (dev-02): provider_values_are_redacted_as_the_dns_client_trims_them ... FAILED. GREEN: ... ok.

P3-B: public gate tied to the child (a89756dce)

No RED can run on the old code: the gate moved from the controller to the supervisor, so the test drives an API that didn't exist. Before, the gate was set when the controller created a supervisor and stayed until the controller dropped it, the case J133 traced. GREEN (root and unprivileged): the_public_gate_is_closed_before_a_child_starts_and_cleared_after_it_stops ... ok, the_public_gate_is_removed_once_the_proxy_child_is_gone ... ok.

P3-C: health acks only from the child's socket (ea00e7ec9)

RED (dev-02, on the commit before): a_squatter_on_the_health_port_cannot_acknowledge_routes ... FAILED. A separate process answering 200 confirmed the route. GREEN (root and unprivileged): ... ok.

P3-D: process.rs:168 as root (36384e212)

RED (root, before): traefik_child_receives_only_credential_file_paths_and_redacts_its_logs ... FAILED read child invocation: Os { code: 2, kind: NotFound }.
GREEN (root, head): test result: ok. 2 passed; 0 failed for tests/process.rs, including the rootful identity and capability boundary.

D330 (842908ace)

GREEN (unprivileged and root):

test config_tests::rootless_processes_split_the_public_and_mesh_listeners ... ok
test config_tests::mesh_routes_never_attach_to_the_public_entrypoints ... ok
test config_tests::the_public_pair_is_probed_on_both_families_and_skips_the_mesh_pair ... ok
test controller::tests::rootless_public_publication_warns_that_only_the_host_firewall_limits_reach ... ok
test controller::tests::rootless_public_routes_never_restart_the_mesh_listeners ... ok

These cover:

• The rootless Mesh process has no http/https entrypoints or wildcard bind; the public process has no Mesh entrypoints.
• A Mesh route renders only on mesh-*.
• A port held on [::] is skipped, and so is the Mesh pair.
• The publish reply has a warning naming the port: "Public routes listen on ports 10080 and 10443 on every address of this machine. The Mesh firewall does not filter these ports; only this machine's firewall or your router limits who can reach them, and either may still block them."
• The public route is applied by the public process before the reply.
• The Mesh process starts once and never receives a public route; the public process is stopped before the removal reply.

Residual (documented in the commit): Traefik binds its entrypoints a few milliseconds before its file provider loads the first snapshot. A connection in that window gets Traefik's default 404; no other route is served.

## J62c: red-to-green evidence per finding Branch `proxy-traefik-281-j62b`, head `36384e212`. Each RED ran the fix's own new tests on the commit *before* that fix (as an overlay). Each GREEN ran at head. Some REDs ran before a later rebase and name the SHAs current then. Since then the commits changed only by formatting, clippy and the test fixups noted below. Hosts: compute01 (x86; namespace runs under `unshare -Urn`, root runs as real root in a throwaway clone under `/root`), and dev-02 (arm64, unprivileged). ### New P1: nft Mesh table could not install with no Mesh route (`84040190a`) RED (compute01, P1-A namespace test on `b2965d89e`): ```text nft_admits_a_port_granted_peer_to_the_proxy_listener_for_the_whole_flow panicked at proxy_ingress_namespace.rs:208:29: called `Result::unwrap()` on an `Err` value: Kind(PermissionDenied) <- firewall.apply() itself ``` Root cause: `nft` rejects `elements = { }` (repro in the comment above). GREEN: with the fix alone (`84040190a`), the same test gets past `apply` and fails only on P1-A's own assertion (next section). ### P1-A: whole proxied flows for port-granted peers (`fa894071f`) RED (compute01, tests on `84040190a`): ```text test iptables::tests::proxied_flows_keep_passing_the_state_checks_after_their_syn ... FAILED test firewall::tests::proxied_flows_keep_passing_the_state_chains_after_their_syn ... FAILED nft_admits_a_port_granted_peer_to_the_proxy_listener_for_the_whole_flow: assertion failed: a port-granted peer reaches the pair left: "BLOCKED" right: "connected" nf_tables_admits_a_port_granted_peer_to_the_proxy_listener_for_the_whole_flow: assertion failed: a port-granted peer reaches the pair left: "BLOCKED" right: "connected" ``` GREEN (compute01 `unshare -Urn`, and again as real root under `unshare -n`). The test covers several round trips on one flow from a peer granted only TCP/8080, a peer with no route grant refused, and the flow surviving an unrelated narrowing: ```text test nft_admits_a_port_granted_peer_to_the_proxy_listener_for_the_whole_flow ... ok test nf_tables_admits_a_port_granted_peer_to_the_proxy_listener_for_the_whole_flow ... ok test connection_state::tests::a_port_granted_peer_keeps_its_proxied_flows_while_it_is_a_proxy_source ... ok ``` ### P1-B: rootful proxy state (`e13b6267a`) RED (root, compute01, proxy code before the fix): ```text test controller::tests::a_new_proxy_state_directory_is_prepared_for_the_child ... FAILED test supervisor_tests::live_process_without_a_healthy_private_ping_is_reported_unhealthy ... FAILED test supervisor_tests::abrupt_infraternal_parent_exit_stops_the_traefik_child ... FAILED test supervisor_tests::owned_supervisor_starts_on_first_exposure_drains_routes_and_keeps_acme_state ... FAILED test supervisor_tests::mesh_grant_revocation_removes_the_route_before_narrowing_its_allowlist ... FAILED test supervisor_tests::unconfirmed_route_removal_stops_the_proxy_and_returns_a_timeout ... FAILED ``` GREEN (root, head `36384e212`): all six `... ok`, and the proxy lib `test result: ok. 102 passed; 0 failed`. ### P2-A: parent-death signal after the drop, orphan reaping (`da2f7ed3d`) RED. The probe reads the signal on the exec'd main thread of the child: ```text unprivileged (dev-02): the_parent_death_signal_survives_the_credential_drop left: "15" right: "9" (SIGTERM; a graceful drain kept the listeners) root (compute01): the_parent_death_signal_survives_the_credential_drop left: "0" right: "9" (cleared by setuid, as J133 found) both: a_new_supervisor_stops_an_orphan_that_holds_the_child_lock ... FAILED ``` GREEN (root and unprivileged, head): ```text test supervisor_tests::the_parent_death_signal_survives_the_credential_drop ... ok test supervisor_tests::a_new_supervisor_stops_an_orphan_that_holds_the_child_lock ... ok test supervisor_tests::abrupt_infraternal_parent_exit_stops_the_traefik_child ... ok ``` ### P2-B: dedicated account (`ce9c401b2`) RED (root, on `da2f7ed3d`): `the_rootful_child_identity_is_not_the_shared_conventional_id left: 65532 right: 65532`. GREEN (root, head): `the_rootful_child_runs_as_the_dedicated_account ... ok`. The run created `infraternal-proxy:x:999:987::/nonexistent:/usr/sbin/nologin` through `useradd --system`. The cleanup removed it (`account after: none`). Two issues were found and fixed during the root runs: - First use raced when several processes created the account at once; the loser's `useradd` failed. Creation is now serialized, and the account is looked up again after a failed tool run. - The collision check counted the agent's own children. They are now exempt. ### P2-C: revoked route ends its source's flows (`73cc570b5`) RED (compute01, namespace test on the commit before): ```text nft_revoking_a_route_ends_the_sources_open_proxied_flows: assertion failed: a revoked route ends that source's open flows and only that source's left: ["echo", "echo"] right: ["lost", "echo"] nf_tables_revoking_a_route_ends_the_sources_open_proxied_flows: (same) ``` GREEN (compute01 userns, and as real root): both `... ok`. The test covers several cases: - The narrowed peer's open flow ends; an unaffected peer's flow survives. - The narrowed peer can reconnect to the route it kept. - Unpublishing a route ends every flow that could use it. ### P3-A: trimmed token (`c9ebab8ea`) RED (dev-02): `provider_values_are_redacted_as_the_dns_client_trims_them ... FAILED`. GREEN: `... ok`. ### P3-B: public gate tied to the child (`a89756dce`) No RED can run on the old code: the gate moved from the controller to the supervisor, so the test drives an API that didn't exist. Before, the gate was set when the controller created a supervisor and stayed until the controller dropped it, the case J133 traced. GREEN (root and unprivileged): `the_public_gate_is_closed_before_a_child_starts_and_cleared_after_it_stops ... ok`, `the_public_gate_is_removed_once_the_proxy_child_is_gone ... ok`. ### P3-C: health acks only from the child's socket (`ea00e7ec9`) RED (dev-02, on the commit before): `a_squatter_on_the_health_port_cannot_acknowledge_routes ... FAILED`. A separate process answering 200 confirmed the route. GREEN (root and unprivileged): `... ok`. ### P3-D: `process.rs:168` as root (`36384e212`) RED (root, before): `traefik_child_receives_only_credential_file_paths_and_redacts_its_logs ... FAILED read child invocation: Os { code: 2, kind: NotFound }`. GREEN (root, head): `test result: ok. 2 passed; 0 failed` for `tests/process.rs`, including the rootful identity and capability boundary. ### D330 (`842908ace`) GREEN (unprivileged and root): ```text test config_tests::rootless_processes_split_the_public_and_mesh_listeners ... ok test config_tests::mesh_routes_never_attach_to_the_public_entrypoints ... ok test config_tests::the_public_pair_is_probed_on_both_families_and_skips_the_mesh_pair ... ok test controller::tests::rootless_public_publication_warns_that_only_the_host_firewall_limits_reach ... ok test controller::tests::rootless_public_routes_never_restart_the_mesh_listeners ... ok ``` These cover: - The rootless Mesh process has no `http`/`https` entrypoints or wildcard bind; the public process has no Mesh entrypoints. - A Mesh route renders only on `mesh-*`. - A port held on `[::]` is skipped, and so is the Mesh pair. - The publish reply has a `warning` naming the port: "Public routes listen on ports 10080 and 10443 on every address of this machine. The Mesh firewall does not filter these ports; only this machine's firewall or your router limits who can reach them, and either may still block them." - The public route is applied by the public process before the reply. - The Mesh process starts once and never receives a public route; the public process is stopped before the removal reply. Residual (documented in the commit): Traefik binds its entrypoints a few milliseconds before its file provider loads the first snapshot. A connection in that window gets Traefik's default 404; no other route is served.

kayg commented 2026-11-03 16:59:42 +00:00

Author

Owner

Copy link

J62c progress. Head is b5dcdd3a9; the per-finding evidence is in my comment above. Since then I've made clippy and test-fixture fixups only, squashed into their commits, plus two P2-B fixes found by the root runs:

• First-use account creation raced across processes. It is now serialized, and the account is looked up again after a failed tool run.
• The agent's own children are exempt from the collision check.

Root GREEN at 36384e212 (compute01, real root): proxy lib 102 passed; 0 failed, tests/process.rs 2 passed, shim lib 102 passed. The account was created as infraternal-proxy:x:999:987, then removed with the clone. The only failure is infraternal-shim --test mouse, which fails only outside the gate because zig isn't on PATH (FileNotFoundError: 'zig').

scripts/verify fast at head: every row passes except test. That row fails in infraternal-agent --lib:

• remote_client::tests::unavailable_udp_does_not_add_seconds_before_tcp_admission;
• winning_quic_drops_the_unfinished_tls_dial_before_control_dispatch;
• sometimes server_path_capability_is_advertised_on_native_and_fallback_transports.

It happens on both VMs (dev-02 four times out of six; compute01 once out of once).

I'm treating this as caused by the branch until shown otherwise:

• Interleaved on dev-02 at load 4: the module alone, head failed 2 of 3 runs and origin/dev 00ec71000 passed 3 of 3. A later 12-pair run of the module alone at load 6 passed 12 of 12 on both.
• With the whole agent lib (as the gate runs it), pair 1 at load 6: head failed 2 remote_client tests, and dev passed 219 passed; 0 failed.
• Run alone, the test passes at head in 0.12 s. Failures always take exactly ~3.0 s, so it's a timeout under parallel contention, not a crash.

A localization run is queued on dev-02: full agent lib, interleaved over dev, the commit before P2-C, P2-C and head. Log: ~/j62c-logs/loc.log.

J62c progress. Head is `b5dcdd3a9`; the per-finding evidence is in my comment above. Since then I've made clippy and test-fixture fixups only, squashed into their commits, plus two P2-B fixes found by the root runs: - First-use account creation raced across processes. It is now serialized, and the account is looked up again after a failed tool run. - The agent's own children are exempt from the collision check. Root GREEN at `36384e212` (compute01, real root): proxy lib `102 passed; 0 failed`, `tests/process.rs` `2 passed`, shim lib `102 passed`. The account was created as `infraternal-proxy:x:999:987`, then removed with the clone. The only failure is `infraternal-shim --test mouse`, which fails only outside the gate because `zig` isn't on PATH (`FileNotFoundError: 'zig'`). `scripts/verify fast` at head: every row passes except `test`. That row fails in `infraternal-agent --lib`: - `remote_client::tests::unavailable_udp_does_not_add_seconds_before_tcp_admission`; - `winning_quic_drops_the_unfinished_tls_dial_before_control_dispatch`; - sometimes `server_path_capability_is_advertised_on_native_and_fallback_transports`. It happens on both VMs (dev-02 four times out of six; compute01 once out of once). I'm treating this as caused by the branch until shown otherwise: - Interleaved on dev-02 at load 4: the module alone, head failed 2 of 3 runs and origin/dev `00ec71000` passed 3 of 3. A later 12-pair run of the module alone at load 6 passed 12 of 12 on both. - With the whole agent lib (as the gate runs it), pair 1 at load 6: head failed 2 `remote_client` tests, and dev passed `219 passed; 0 failed`. - Run alone, the test passes at head in 0.12 s. Failures always take exactly ~3.0 s, so it's a timeout under parallel contention, not a crash. A localization run is queued on dev-02: full agent lib, interleaved over dev, the commit before P2-C, P2-C and head. Log: `~/j62c-logs/loc.log`.

kayg commented 2026-11-03 19:12:14 +00:00

Author

Owner

Copy link

J62c finished: round 4 of slice 1

Branch proxy-traefik-281-j62b, head ad2148fb91d3296d809db48d443de42bd5cfdce0, on origin/dev 00ec71000. Pushed; not merged, not deployed. Per-finding RED/GREEN evidence is in my comment above. The commits from this round, in order:

Commit	Change
84040190a	new P1: nft Mesh table failed to install with no Mesh route (elements = { })
fa894071f	P1-A
e13b6267a	P1-B
b4964427b	P2-A
2fda07647	P2-B
9e804def5	P2-C
f5a2a565e	P3-A
840c0e87b	P3-B
3db1e1b61	P3-C
e075d1e1c	D330
b5dcdd3a9	P3-D
ad2148fb9	test: the Pebble namespace precondition accepts the kernel's DOWN fallback tunnels (compute01 now has the ipip/gre modules loaded)

Every commit is fmt-clean on its own.

Gate

scripts/verify fast at ad2148fb9 (dev-02, load 9–12), verbatim:

== verify fast at ad2148fb91d3296d809db48d443de42bd5cfdce0
+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | FAIL (101)   |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
VERIFY_EXIT=1

The test failures in that run:

test restart::tests::cold_exec_with_closed_stdin_keeps_the_image_pinned_and_maps_null ... FAILED
test restart::tests::path_review::late_recovery_from_path_invocation_keeps_install_target ... FAILED
test restart::tests::restart_request_logs_kernel_peer_target_and_cold_outcome ... FAILED      ("exceeded 60s; status=signal: 9")
test real_binary_keeps_background_backend_probe_off_the_control_path ... FAILED             (Python fixture oracle)

git diff origin/dev...HEAD touches none of these files (restart, tests/agent_daemon.rs, remote_client*). Across eight gate runs today, the failing test cases changed from run to run:

• status_restart (passes alone);
• replay_size (passes alone);
• cli_sessions;
• the restart tests above;
• agent_daemon;
• remote_client.

The cargo test runner stops at the first failing binary, so I also ran the whole workspace with --no-fail-fast. 210 test binaries passed, and every failure was one of these timing tests.

remote_client needs a look by someone with a quieter host. It failed in most gate runs today, on both VMs. Interleaved full infraternal-agent --lib runs on dev-02, at load 9–12:

Commit	Runs with a remote_client failure
origin/dev 00ec71000	1 of 2
before P2-C 2fda07647	0 of 2
P2-C 9e804def5	1 of 2
head	2 of 2

An earlier pair at load 6: head failed, dev passed. Run on its own module, 12 interleaved pairs at load 6 passed 24 of 24, and the single test passes in 0.12 s at head. So the flake exists on unmodified origin/dev. With this sample I can't rule out that the branch makes it more likely under load; I found no startup work in this round's diff that would cause it.

Other runs

• Real Traefik + Pebble DNS-01 (scripts/test-proxy-pebble.sh, compute01, b9a0f9de9; the only change since moves a test const): test namespace_serves_a_granted_mesh_backend_with_pebble_dns01 ... ok, 1 passed; 0 failed. This covers P3-C's socket-ownership check against Traefik's own listener, the child lock and the supervisor changes.

• Root (compute01, real root, throwaway clone /root/j62c, head 36384e212; later changes are lint and test-only):

  proxy lib: test result: ok. 102 passed; 0 failed
  proxy tests/process.rs: test result: ok. 2 passed; 0 failed
  shim lib: test result: ok. 102 passed; 0 failed
  

  The Mesh namespace tests (4) pass as real root under unshare -n. The run created infraternal-proxy:x:999:987, and the cleanup removed it: account after: none; clone: removed.

• Unprivileged: the Mesh lib and namespace tests pass (compute01 unshare -Urn), and the proxy lib passes (dev-02).

• infraternal-shim --test mouse fails only outside the gate, because zig isn't on PATH (FileNotFoundError: 'zig'). Inside verify fast, VT fmt and the shim run normally.

Size (report only, D243)

Target	Size	vs D243 baseline	vs previous round
x86_64	23,118,640 B	+700,384 B	+90,016 B
aarch64	21,042,448 B	+589,168 B	+72,608 B

For the security reviewer

• D330 residual: Traefik binds its entrypoints a few milliseconds before its file provider loads the first snapshot. A connection in that window gets the default 404; no other route is served.
• P2-B does no file-system scan for files already owned by the account's IDs (cost). Running processes are checked before every start; the agent's own children are exempt.
• P2-C deletes all of a source's flows to the listener pair when it loses any route, as J133 accepted. The comparison is against the config from the last confirmed proxy snapshot, so the reload window is covered.
• The unit-test build trusts the test process as a health-socket owner (#[cfg(test)] only).

Ops notes

• Root's ~/infraternal-ops/slots is missing, so build-slot run as root assumes 2 slots. At the head of the queue it then waits while slots 3 and 4 are free, and blocks everyone behind it. I hit this and re-queued with claude's slot config. /root/j79d-root/run.sh (not mine) was queued the same way.
• Root's ssh goes through a su wrapper, so scp to root exits 1. Use ssh root@… 'cat > file' < file.

Cleanup: worktrees, target dirs, refs and bundles are removed on both VMs, and the root clone and account on compute01. Small logs stay in ~/j62c-logs/ on both VMs.

## J62c finished: round 4 of slice 1 Branch `proxy-traefik-281-j62b`, head **`ad2148fb91d3296d809db48d443de42bd5cfdce0`**, on origin/dev `00ec71000`. Pushed; not merged, not deployed. Per-finding RED/GREEN evidence is in my comment above. The commits from this round, in order: | Commit | Change | |---|---| | `84040190a` | **new P1:** nft Mesh table failed to install with no Mesh route (`elements = { }`) | | `fa894071f` | P1-A | | `e13b6267a` | P1-B | | `b4964427b` | P2-A | | `2fda07647` | P2-B | | `9e804def5` | P2-C | | `f5a2a565e` | P3-A | | `840c0e87b` | P3-B | | `3db1e1b61` | P3-C | | `e075d1e1c` | D330 | | `b5dcdd3a9` | P3-D | | `ad2148fb9` | test: the Pebble namespace precondition accepts the kernel's DOWN fallback tunnels (compute01 now has the ipip/gre modules loaded) | Every commit is fmt-clean on its own. ### Gate `scripts/verify fast` at `ad2148fb9` (dev-02, load 9–12), verbatim: ```text == verify fast at ad2148fb91d3296d809db48d443de42bd5cfdce0 +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | FAIL (101) | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | static builds | PASS | +---------------------------+--------------+ VERIFY_EXIT=1 ``` The `test` failures in that run: ```text test restart::tests::cold_exec_with_closed_stdin_keeps_the_image_pinned_and_maps_null ... FAILED test restart::tests::path_review::late_recovery_from_path_invocation_keeps_install_target ... FAILED test restart::tests::restart_request_logs_kernel_peer_target_and_cold_outcome ... FAILED ("exceeded 60s; status=signal: 9") test real_binary_keeps_background_backend_probe_off_the_control_path ... FAILED (Python fixture oracle) ``` `git diff origin/dev...HEAD` touches none of these files (restart, `tests/agent_daemon.rs`, `remote_client*`). Across eight gate runs today, the failing `test` cases changed from run to run: - `status_restart` (passes alone); - `replay_size` (passes alone); - `cli_sessions`; - the restart tests above; - `agent_daemon`; - `remote_client`. The cargo test runner stops at the first failing binary, so I also ran the whole workspace with `--no-fail-fast`. 210 test binaries passed, and every failure was one of these timing tests. **`remote_client` needs a look by someone with a quieter host.** It failed in most gate runs today, on both VMs. Interleaved full `infraternal-agent --lib` runs on dev-02, at load 9–12: | Commit | Runs with a `remote_client` failure | |---|---| | origin/dev `00ec71000` | 1 of 2 | | before P2-C `2fda07647` | 0 of 2 | | P2-C `9e804def5` | 1 of 2 | | head | 2 of 2 | An earlier pair at load 6: head failed, dev passed. Run on its own module, 12 interleaved pairs at load 6 passed 24 of 24, and the single test passes in 0.12 s at head. So the flake exists on unmodified origin/dev. With this sample I can't rule out that the branch makes it more likely under load; I found no startup work in this round's diff that would cause it. ### Other runs - **Real Traefik + Pebble DNS-01** (`scripts/test-proxy-pebble.sh`, compute01, `b9a0f9de9`; the only change since moves a test const): `test namespace_serves_a_granted_mesh_backend_with_pebble_dns01 ... ok`, `1 passed; 0 failed`. This covers P3-C's socket-ownership check against Traefik's own listener, the child lock and the supervisor changes. - **Root** (compute01, real root, throwaway clone `/root/j62c`, head `36384e212`; later changes are lint and test-only): ```text proxy lib: test result: ok. 102 passed; 0 failed proxy tests/process.rs: test result: ok. 2 passed; 0 failed shim lib: test result: ok. 102 passed; 0 failed ``` The Mesh namespace tests (4) pass as real root under `unshare -n`. The run created `infraternal-proxy:x:999:987`, and the cleanup removed it: `account after: none; clone: removed`. - **Unprivileged:** the Mesh lib and namespace tests pass (compute01 `unshare -Urn`), and the proxy lib passes (dev-02). - `infraternal-shim --test mouse` fails only outside the gate, because `zig` isn't on PATH (`FileNotFoundError: 'zig'`). Inside `verify fast`, VT fmt and the shim run normally. ### Size (report only, D243) | Target | Size | vs D243 baseline | vs previous round | |---|---|---|---| | x86_64 | 23,118,640 B | +700,384 B | **+90,016 B** | | aarch64 | 21,042,448 B | +589,168 B | **+72,608 B** | ### For the security reviewer - D330 residual: Traefik binds its entrypoints a few milliseconds before its file provider loads the first snapshot. A connection in that window gets the default 404; no other route is served. - P2-B does no file-system scan for files already owned by the account's IDs (cost). Running processes are checked before every start; the agent's own children are exempt. - P2-C deletes all of a source's flows to the listener pair when it loses any route, as J133 accepted. The comparison is against the config from the last confirmed proxy snapshot, so the reload window is covered. - The unit-test build trusts the test process as a health-socket owner (`#[cfg(test)]` only). ### Ops notes - Root's `~/infraternal-ops/slots` is missing, so `build-slot` run as root assumes 2 slots. At the head of the queue it then waits while slots 3 and 4 are free, and blocks everyone behind it. I hit this and re-queued with claude's slot config. `/root/j79d-root/run.sh` (not mine) was queued the same way. - Root's ssh goes through a `su` wrapper, so scp to root exits 1. Use `ssh root@… 'cat > file' < file`. Cleanup: worktrees, target dirs, refs and bundles are removed on both VMs, and the root clone and account on compute01. Small logs stay in `~/j62c-logs/` on both VMs.

kayg commented 2026-11-03 19:14:27 +00:00

Author

Owner

Copy link

J152 independent security review (round 4) started. Read-only: no code is pushed.

Target: proxy-traefik-281-j62b at ad2148fb91d3296d809db48d443de42bd5cfdce0, diffed against origin/dev e81eb5bb4 (merge base bcca3054f).

Plan (~3 h): read the round-4 diff against J133's findings (76880) and D330; run the proxy and Mesh suites as real root in a throwaway clone under /root/j152 on compute01, through build-slot; run remote_client (infraternal-agent) 5× interleaved at head and at origin/dev; post one verdict comment here.

J152 independent security review (round 4) started. Read-only: no code is pushed. Target: `proxy-traefik-281-j62b` at `ad2148fb91d3296d809db48d443de42bd5cfdce0`, diffed against `origin/dev` `e81eb5bb4` (merge base `bcca3054f`). Plan (~3 h): read the round-4 diff against J133's findings (76880) and D330; run the proxy and Mesh suites as real root in a throwaway clone under `/root/j152` on compute01, through build-slot; run `remote_client` (`infraternal-agent`) 5× interleaved at head and at origin/dev; post one verdict comment here.

kayg commented 2026-11-03 20:56:55 +00:00

Author

Owner

Copy link

J152 independent security review, round 4: REQUEST CHANGES

Target: proxy-traefik-281-j62b at ad2148fb91d3296d809db48d443de42bd5cfdce0, diffed against origin/dev e81eb5bb4 (merge base bcca3054f). This was a read-only review; no code was pushed. Root runs used a throwaway clone in /root/j152 on compute01, through build-slot. The clone, its target dir, and the infraternal-proxy account the tests created are all removed (userdel ok; account after: none; group after: none). Logs are on code-minipc01 in ~/Developer/Worktrees/j152-evidence/.

One open P2 (the dedicated account's ID policy) keeps this at REQUEST CHANGES. Everything J133 raised is fixed or narrowed. No P1 remains.

Round-4 fixes, checked

Item	Status	Evidence
new: nft empty set	Fixed	nft namespace tests install and pass as root (below).
P1-A: whole flows	Fixed, nothing broader	Continuation accepts in both backends match only TCP, original source in the proxy-source set, original destination equal to this node's Mesh address, and original destination port in the selected pair (firewall.rs:604-612, iptables.rs:562-585). New flows still need SYN, and conntrack retirement keeps the same test (connection_state.rs:66-72). Inside the flow, routes are still enforced by Traefik's per-route ipAllowList on RemoteAddr, as before. Real-root netns runs: nft and iptables-nft pass.
P1-B: rootful proxy dir	Fixed	Root: proxy lib 102 passed; 0 failed.
P2-A: PDEATHSIG and orphans	Fixed for the crash case	Root: the_parent_death_signal_survives_the_credential_drop ... ok, a_new_supervisor_stops_an_orphan_that_holds_the_child_lock ... ok. The reaper itself has a new problem (P3-1). Containment of a compromised child is still open (P3-4).
P2-B: dedicated account	Partly fixed	The account and its collision check work, but the ID policy is a new problem (P2-1).
P2-C: flows end on revocation	Fixed for the stated scenario	Root netns: nft_/nf_tables_revoking_a_route_ends_the_sources_open_proxied_flows ... ok. One bookkeeping residual remains (P3-3).
P3-A: trimmed token	Fixed	redact_log_line adds trim(). Oversized lines are dropped, not truncated.
P3-B: gate lifetime	Fixed, with a regression	The gate now follows the child's life. A crash restart leaves public routes gated (P3-2).
P3-C: health-ack ownership	Fixed	verify_health_listener runs before the first ack. An empty listener set and listeners on both the exact and wildcard address are both rejected. Root: a_squatter_on_the_health_port_cannot_acknowledge_routes ... ok. Later acks can only come from a process holding the bound port, and only the child holds it.
P3-D: test as root	Fixed	Root: tests/process.rs 2 passed; 0 failed.

Conntrack races. Rules are installed before retirement in every path. A deleted flow can't come back: a non-SYN ct state new is dropped in both directions. The second sweep after Traefik's ack (against proxy_config) closes the reload window. The one gap is that proxy_config also advances on updates Traefik never confirmed (P3-3).

D330 (rootless --public)

No findings.

• Bind ordering: the public process starts only from replace(), with its snapshot already written, and the reply waits for that generation's ack.
• The 404 window: benign. Before the file provider loads, Traefik answers only its default 404, with its default certificate. The dashboard and insecure API are off, and ping and rawdata are only on the loopback health entrypoint.
• Mesh-only routes stay off the public entrypoints: they run in separate processes. The MeshOnly static config has no http/https entrypoints, and the PublicOnly process is only ever given service.public routes (controller.rs:482-489, 553-600).
• Probing: each port is probed on 0.0.0.0 and [::]. Std sets SO_REUSEADDR, so a port that another process listens on, on any address, counts as taken. EAFNOSUPPORT and EADDRNOTAVAIL count as free when there is no IPv6. The Mesh pair is skipped.
• Stop before report: dropping the owner sends Shutdown, and the worker kills and wait()s the child before join returns, all before the unpublish reply. A failed public apply also drops the owner.
• Warning: it names both ports and says that only this machine's firewall or the router limits reach. That matches D330.

Secret redaction: no new findings. Token values never reach argv or the environment, and log lines are redacted as raw, CR/LF-trimmed and fully trimmed values, plus named credential keys.

Root runs (compute01, euid 0, head ad2148fb9)

build-proxy exit=0
build-mesh exit=0
ns nft_admits_a_port_granted_peer_to_the_proxy_listener_for_the_whole_flow exit=0 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 1.91s
ns nf_tables_admits_a_port_granted_peer_to_the_proxy_listener_for_the_whole_flow exit=0 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 2.21s
ns nft_revoking_a_route_ends_the_sources_open_proxied_flows exit=0 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 4.99s
ns nf_tables_revoking_a_route_ends_the_sources_open_proxied_flows exit=0 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 5.46s
mesh-fw-ignored exit=0 test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 89 filtered out; finished in 1.22s
mesh-lib exit=0 test result: ok. 79 passed; 0 failed; 14 ignored; 0 measured; 0 filtered out; finished in 0.85s
proxy lib (--skip j152_probe): test result: ok. 102 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out; finished in 5.19s
proxy tests/process.rs: test result: ok. 2 passed; 0 failed

The namespace tests ran as real root under unshare -n. The first proxy-lib pass had 5 failures. That run was contaminated by my own identity probe: its uid-999 workload correctly made other tests' starts refuse (another process runs as the proxy account). The clean rerun above skips the two probes.

Two throwaway probe tests were added to the root clone only (sources in the evidence dir):

J152 account uid=999 gid=987; foreign workload with that uid -> verify_unshared=Err(Custom { kind: PermissionDenied, error: "another process runs as the proxy account; refusing to share its identity" })
J152 start_ok=true orphan=Some(ExitStatus(unix_wait_status(9))) bystander(uid 65534, no flock)=Some(ExitStatus(unix_wait_status(9)))

Findings

P2-1: The dedicated account gets UID 999, a UID that common container images also use. Rootful publishing either refuses permanently or shares that UID without noticing.
crates/plugins/proxy/src/identity.rs:182-195 (useradd --system), :13-15 (rationale), :73-105 (verify_unshared), :226 (check_unique).
useradd --system takes the highest free ID at or below SYS_UID_MAX. On a stock host that is 999. It was 999 here, and in J62c's run. The official postgres and redis images (and many others) also run as UID 999. Without a user namespace, a container's processes and files have host UID 999.

Scenario A (deterministic): the host already runs such a container. Every rootful start then fails with "another process runs as the proxy account". The probe above reproduces this. proxy publish can't work on that host until the operator renumbers something.

Scenario B: the container starts after Traefik. Nothing checks again until the next child start, so the UID is silently shared. A compromised Traefik can then read and write anything that container wrote to bind mounts on the host, such as a ./pgdata owned by 999 under a world-traversable /srv or /home/<user>. The module doc says "IDs allocated fresh by the account tools have no prior files". That is false: the account tools consult only /etc/passwd, not the file system.

Also, check_unique ignores the member list of the infraternal-proxy group, and other users whose primary GID is the proxy GID. Such users can traverse the 0750 proxy dir and read the 0640 config.

Fix:

• Allocate the IDs explicitly (useradd -r -u <uid> -g <gid>) from a high band that images and /etc/subuid//etc/subgid don't use, and record the choice in agent state. maps_system_range will need adjusting.
• Reject a group that has members, or that another user has as their primary group.
• Rewrite the module doc so it doesn't claim fresh IDs own no files.

P3-1: The orphan reaper SIGKILLs any process that holds the lock's inode, without checking who it is.
crates/plugins/proxy/src/supervisor.rs:1652-1686.
The comment says a match "identifies an orphaned child, never an unrelated process". The probe disproves this: a bystander running as UID 65534 that held only a read descriptor (no flock) was killed alongside the real orphan.

How a descriptor can reach another process:

• SCM_RIGHTS: a compromised rootful Traefik passes its descriptor to another process, for example a message parked in the system bus broker.
• Any root process that has the state file open at that moment (backup, inspection).
• PID reuse between the /proc scan and kill.

After the next agent crash, the root agent kills that process. This needs an agent crash to trigger, so it's P3.

Fix: pidfd_open the candidate, then re-verify that it still holds the inode, that its UIDs are the child identity, and that /proc/<pid>/exe is the installed Traefik inode. Then signal it with pidfd_send_signal.

P3-2 (regression from P3-B): after a crash restart, rootful public routes stay behind the gate.
crates/plugins/proxy/src/ingress.rs:76-80, supervisor.rs:1266-1296, 1353, 1456-1468, controller.rs:605-633.
Reaping the crashed child calls gate(None), which also sets public_route_active = false. The restart calls gate(Some(pair)), which closes the gate but never re-opens it. Only the controller's update_mesh_ingress sets the flag again, and that runs only on the next publish, unpublish or Mesh context change.

Scenario: Traefik is OOM-killed. The supervisor restarts it with the same public routes, but every non-loopback packet to 80/443 is dropped until some unrelated event happens. This fails closed, so it is availability only.

Fix: have the supervisor restore the previous public_route_active once the restarted child acks a snapshot that contains a public route. Add a test that crashes the fake child and asserts the gate opens again.

P3-3: P2-C's "last confirmed" reference also advances on unconfirmed updates.
crates/plugins/mesh/src/firewall.rs:170-189.
apply_proxy_ingress sets proxy_config to the node's current config on every call. That includes gate-only updates from the supervisor thread (ingress.rs:76) and the controller's update(false, …) (controller.rs:505-507, and update_mesh_ingress(…, false)).

Scenario:

1. The Mesh config narrows X: X loses route A and keeps route B. complete() deletes X's flows.
2. Before the controller's replace, the supervisor restarts a crashed Traefik from its retained, pre-narrowing services. Its gate update advances proxy_config to the new config.
3. X's client reconnects to A, which the old allowlist still admits.
4. The confirmation of the new config then compares it with itself, so X's new flow survives.

This needs a restart inside the window, so it's P3.

Fix: pass confirmed through to the node, and advance proxy_config only on confirmed snapshots. Better still, tag each snapshot with the Mesh config version it was rendered from, and advance proxy_config to that version.

P3-4 (design): a compromised child can escape orphan reaping.
The child holds a duplicate of the agent's open file description, so flock(LOCK_UN) from the child releases the agent's lock. A child process forked from Traefik doesn't inherit PDEATHSIG and can keep the 80/443 listening sockets. After an agent restart, the Mesh firewall comes up with no gate, and verify_unshared refuses to start a new Traefik rather than reclaiming the ports. The escaped process keeps serving the public ports with the node's ACME keys. Only a cgroup (cgroup.kill) or a PID namespace contains this. This belongs in the threat model below, not in another patch.

Narrowing or new classes?

Judged by area, not by round count:

• Mesh firewall and conntrack: narrowing, converged. In round 3 this area had a P1 (whole flows) and a P2 (revocation). Both fixes hold under real root on both backends, and what's left is one P3 bookkeeping race (P3-3).

• D330 public process: a new area, but clean. It follows J133's notes point by point.

• Containment and identity of the Traefik child: new classes in new code, not narrowing. Each round added a mechanism here:

  • round 2: the child ran as root;
  • round 3: a fixed 65532 identity plus privilege drop;
  • round 4: a created account, a /proc identity scan, PDEATHSIG re-armed, and a /proc fd-ownership killer.

  Each mechanism brought its own class of defect: directory modes and lost PDEATHSIG (round 3), and now an ID-allocation policy (P2-1), a kill without identity check (P3-1), and a compromised child escaping (P3-4). The cause is that nothing written says what containment has to guarantee.

Recommendation: before more patching in this area, write a short threat model and design note. It should answer three questions:

1. Is a compromised Traefik in scope?
2. What may it reach (host UIDs, files, ACME keys, DNS tokens)?
3. May it outlive the agent?

It should then pick one mechanism with measured costs. Options:

• (a) a dedicated account with an explicit ID policy plus a cgroup per child (needs a delegated subtree: Delegate=yes);
• (b) a user and PID namespace for the child with pre-bound sockets passed in (no host UID, so the collision question goes away);
• (c) declare a compromised Traefik out of scope and remove the mechanisms that don't serve the crash case.

The owner decides. Fixing P2-1 alone in the current design is fine as a stopgap.

remote_client (infraternal-agent --lib), head vs origin/dev, interleaved, compute01

Features matched cargo test --workspace unification (head native-streams,mesh,proxy,plugin-host-facts; dev the same without proxy), with INFRATERNAL_TEST_STATE_ROOT set. A watchdog killed any test binary running past 240 s and recorded the hung test.

run 1 head exit=101 load=7.92  rc_tests=65 other_failed=24 rc_failed=[server_path_capability_is_advertised_on_native_and_fallback_transports unavailable_udp_does_not_add_seconds_before_tcp_admission]
run 1 dev  exit=101 load=16.39 rc_tests=65 208 passed; 14 failed other_failed=15 rc_failed=[]
run 2 head exit=101 load=24.17 rc_tests=65 other_failed=24 rc_failed=[server_path_capability_… unavailable_udp_… winning_quic_drops_the_unfinished_tls_dial_before_control_dispatch]
run 2 dev  exit=101 load=13.57 rc_tests=65 208 passed; 14 failed other_failed=15 rc_failed=[]
run 3 head exit=101 load=13.90 rc_tests=65 other_failed=24 rc_failed=[]
run 3 dev  exit=101 load=10.27 rc_tests=65 208 passed; 14 failed other_failed=15 rc_failed=[]
run 4 head exit=101 load=11.93 rc_tests=65 other_failed=24 rc_failed=[]
run 4 dev  exit=101 load=7.97  rc_tests=65 208 passed; 14 failed other_failed=15 rc_failed=[]
run 5 head exit=101 load=10.99 rc_tests=65 other_failed=24 rc_failed=[unavailable_udp_… winning_quic_drops_…]
run 5 dev  exit=101 load=13.41 rc_tests=65 208 passed; 14 failed other_failed=15 rc_failed=[]
watchdog: head runs 1-5 all hung in daemon::remote_tests::cancelling_remote_dispatch_retires_the_borrowed_authenticated_channel; dev never did.

Head fails remote_client in 3 of 5 runs, dev in 0 of 5. The cause is not branch code.

• All of head's remote_client failures happen in runs where cancelling_remote_dispatch_retires_the_borrowed_authenticated_channel hangs.
• With only that test skipped, head passes all 65 remote_client tests: 213 passed; 10 failed, and the 10 are the lifecycle/sessions tests that fail on dev too in this environment.
• That test hangs on unmodified dev as well when run alone (timeout 60: head 3/3 hung, dev 3/3 hung) and when daemon::remote_tests runs as a module (head 3/3, dev 3/3).

So it is a pre-existing, order-dependent hang. Dev's full-lib order happens to avoid it, and head's slightly different test set doesn't. The branch doesn't touch daemon* or remote* code except one line in remote_supervisor_review_tests.rs. Even so, the gate will be flakier after this merges, so the hang needs its own issue: a test-isolation fix and a timeout on that test's awaits.

No code was changed, so the gate is not applicable. The size delta is unchanged from J62c's report.

## J152 independent security review, round 4: **REQUEST CHANGES** Target: `proxy-traefik-281-j62b` at `ad2148fb91d3296d809db48d443de42bd5cfdce0`, diffed against `origin/dev` `e81eb5bb4` (merge base `bcca3054f`). This was a read-only review; no code was pushed. Root runs used a throwaway clone in `/root/j152` on compute01, through build-slot. The clone, its target dir, and the `infraternal-proxy` account the tests created are all removed (`userdel ok; account after: none; group after: none`). Logs are on code-minipc01 in `~/Developer/Worktrees/j152-evidence/`. One open P2 (the dedicated account's ID policy) keeps this at REQUEST CHANGES. Everything J133 raised is fixed or narrowed. No P1 remains. ### Round-4 fixes, checked | Item | Status | Evidence | |---|---|---| | new: nft empty set | **Fixed** | nft namespace tests install and pass as root (below). | | P1-A: whole flows | **Fixed, nothing broader** | Continuation accepts in both backends match only TCP, original source in the proxy-source set, original destination equal to this node's Mesh address, and original destination port in the *selected* pair (`firewall.rs:604-612`, `iptables.rs:562-585`). New flows still need SYN, and conntrack retirement keeps the same test (`connection_state.rs:66-72`). Inside the flow, routes are still enforced by Traefik's per-route `ipAllowList` on RemoteAddr, as before. Real-root netns runs: nft and iptables-nft pass. | | P1-B: rootful proxy dir | **Fixed** | Root: proxy lib `102 passed; 0 failed`. | | P2-A: PDEATHSIG and orphans | **Fixed for the crash case** | Root: `the_parent_death_signal_survives_the_credential_drop ... ok`, `a_new_supervisor_stops_an_orphan_that_holds_the_child_lock ... ok`. The reaper itself has a new problem (P3-1). Containment of a *compromised* child is still open (P3-4). | | P2-B: dedicated account | **Partly fixed** | The account and its collision check work, but the ID policy is a new problem (P2-1). | | P2-C: flows end on revocation | **Fixed for the stated scenario** | Root netns: `nft_/nf_tables_revoking_a_route_ends_the_sources_open_proxied_flows ... ok`. One bookkeeping residual remains (P3-3). | | P3-A: trimmed token | **Fixed** | `redact_log_line` adds `trim()`. Oversized lines are dropped, not truncated. | | P3-B: gate lifetime | **Fixed, with a regression** | The gate now follows the child's life. A crash restart leaves public routes gated (P3-2). | | P3-C: health-ack ownership | **Fixed** | `verify_health_listener` runs before the first ack. An empty listener set and listeners on both the exact and wildcard address are both rejected. Root: `a_squatter_on_the_health_port_cannot_acknowledge_routes ... ok`. Later acks can only come from a process holding the bound port, and only the child holds it. | | P3-D: test as root | **Fixed** | Root: `tests/process.rs` `2 passed; 0 failed`. | **Conntrack races.** Rules are installed before retirement in every path. A deleted flow can't come back: a non-SYN `ct state new` is dropped in both directions. The second sweep after Traefik's ack (against `proxy_config`) closes the reload window. The one gap is that `proxy_config` also advances on updates Traefik never confirmed (P3-3). ### D330 (rootless `--public`) No findings. - **Bind ordering:** the public process starts only from `replace()`, with its snapshot already written, and the reply waits for that generation's ack. - **The 404 window:** benign. Before the file provider loads, Traefik answers only its default 404, with its default certificate. The dashboard and insecure API are off, and ping and rawdata are only on the loopback health entrypoint. - **Mesh-only routes stay off the public entrypoints:** they run in separate processes. The `MeshOnly` static config has no `http`/`https` entrypoints, and the `PublicOnly` process is only ever given `service.public` routes (`controller.rs:482-489, 553-600`). - **Probing:** each port is probed on `0.0.0.0` and `[::]`. Std sets SO_REUSEADDR, so a port that another process listens on, on any address, counts as taken. EAFNOSUPPORT and EADDRNOTAVAIL count as free when there is no IPv6. The Mesh pair is skipped. - **Stop before report:** dropping the owner sends Shutdown, and the worker kills and `wait()`s the child before `join` returns, all before the unpublish reply. A failed public apply also drops the owner. - **Warning:** it names both ports and says that only this machine's firewall or the router limits reach. That matches D330. **Secret redaction:** no new findings. Token values never reach argv or the environment, and log lines are redacted as raw, CR/LF-trimmed and fully trimmed values, plus named credential keys. ### Root runs (compute01, euid 0, head `ad2148fb9`) ```text build-proxy exit=0 build-mesh exit=0 ns nft_admits_a_port_granted_peer_to_the_proxy_listener_for_the_whole_flow exit=0 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 1.91s ns nf_tables_admits_a_port_granted_peer_to_the_proxy_listener_for_the_whole_flow exit=0 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 2.21s ns nft_revoking_a_route_ends_the_sources_open_proxied_flows exit=0 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 4.99s ns nf_tables_revoking_a_route_ends_the_sources_open_proxied_flows exit=0 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 5.46s mesh-fw-ignored exit=0 test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 89 filtered out; finished in 1.22s mesh-lib exit=0 test result: ok. 79 passed; 0 failed; 14 ignored; 0 measured; 0 filtered out; finished in 0.85s proxy lib (--skip j152_probe): test result: ok. 102 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out; finished in 5.19s proxy tests/process.rs: test result: ok. 2 passed; 0 failed ``` The namespace tests ran as real root under `unshare -n`. The first proxy-lib pass had 5 failures. That run was contaminated by my own identity probe: its uid-999 workload correctly made other tests' starts refuse (`another process runs as the proxy account`). The clean rerun above skips the two probes. Two throwaway probe tests were added to the root clone only (sources in the evidence dir): ```text J152 account uid=999 gid=987; foreign workload with that uid -> verify_unshared=Err(Custom { kind: PermissionDenied, error: "another process runs as the proxy account; refusing to share its identity" }) J152 start_ok=true orphan=Some(ExitStatus(unix_wait_status(9))) bystander(uid 65534, no flock)=Some(ExitStatus(unix_wait_status(9))) ``` ### Findings **P2-1: The dedicated account gets UID 999, a UID that common container images also use. Rootful publishing either refuses permanently or shares that UID without noticing.** `crates/plugins/proxy/src/identity.rs:182-195` (`useradd --system`), `:13-15` (rationale), `:73-105` (`verify_unshared`), `:226` (`check_unique`). `useradd --system` takes the highest free ID at or below SYS_UID_MAX. On a stock host that is 999. It was 999 here, and in J62c's run. The official postgres and redis images (and many others) also run as UID 999. Without a user namespace, a container's processes and files have host UID 999. Scenario A (deterministic): the host already runs such a container. Every rootful start then fails with "another process runs as the proxy account". The probe above reproduces this. `proxy publish` can't work on that host until the operator renumbers something. Scenario B: the container starts after Traefik. Nothing checks again until the next child start, so the UID is silently shared. A compromised Traefik can then read and write anything that container wrote to bind mounts on the host, such as a `./pgdata` owned by 999 under a world-traversable `/srv` or `/home/<user>`. The module doc says "IDs allocated fresh by the account tools have no prior files". That is false: the account tools consult only `/etc/passwd`, not the file system. Also, `check_unique` ignores the member list of the `infraternal-proxy` group, and other users whose primary GID is the proxy GID. Such users can traverse the 0750 proxy dir and read the 0640 config. Fix: - Allocate the IDs explicitly (`useradd -r -u <uid> -g <gid>`) from a high band that images and `/etc/subuid`/`/etc/subgid` don't use, and record the choice in agent state. `maps_system_range` will need adjusting. - Reject a group that has members, or that another user has as their primary group. - Rewrite the module doc so it doesn't claim fresh IDs own no files. **P3-1: The orphan reaper SIGKILLs any process that holds the lock's inode, without checking who it is.** `crates/plugins/proxy/src/supervisor.rs:1652-1686`. The comment says a match "identifies an orphaned child, never an unrelated process". The probe disproves this: a bystander running as UID 65534 that held only a read descriptor (no flock) was killed alongside the real orphan. How a descriptor can reach another process: - SCM_RIGHTS: a compromised rootful Traefik passes its descriptor to another process, for example a message parked in the system bus broker. - Any root process that has the state file open at that moment (backup, inspection). - PID reuse between the `/proc` scan and `kill`. After the next agent crash, the root agent kills that process. This needs an agent crash to trigger, so it's P3. Fix: `pidfd_open` the candidate, then re-verify that it still holds the inode, that its UIDs are the child identity, and that `/proc/<pid>/exe` is the installed Traefik inode. Then signal it with `pidfd_send_signal`. **P3-2 (regression from P3-B): after a crash restart, rootful public routes stay behind the gate.** `crates/plugins/proxy/src/ingress.rs:76-80`, `supervisor.rs:1266-1296, 1353, 1456-1468`, `controller.rs:605-633`. Reaping the crashed child calls `gate(None)`, which also sets `public_route_active = false`. The restart calls `gate(Some(pair))`, which closes the gate but never re-opens it. Only the controller's `update_mesh_ingress` sets the flag again, and that runs only on the next publish, unpublish or Mesh context change. Scenario: Traefik is OOM-killed. The supervisor restarts it with the same public routes, but every non-loopback packet to 80/443 is dropped until some unrelated event happens. This fails closed, so it is availability only. Fix: have the supervisor restore the previous `public_route_active` once the restarted child acks a snapshot that contains a public route. Add a test that crashes the fake child and asserts the gate opens again. **P3-3: P2-C's "last confirmed" reference also advances on unconfirmed updates.** `crates/plugins/mesh/src/firewall.rs:170-189`. `apply_proxy_ingress` sets `proxy_config` to the node's *current* config on every call. That includes gate-only updates from the supervisor thread (`ingress.rs:76`) and the controller's `update(false, …)` (`controller.rs:505-507`, and `update_mesh_ingress(…, false)`). Scenario: 1. The Mesh config narrows X: X loses route A and keeps route B. `complete()` deletes X's flows. 2. Before the controller's `replace`, the supervisor restarts a crashed Traefik from its retained, pre-narrowing services. Its gate update advances `proxy_config` to the new config. 3. X's client reconnects to A, which the old allowlist still admits. 4. The confirmation of the new config then compares it with itself, so X's new flow survives. This needs a restart inside the window, so it's P3. Fix: pass `confirmed` through to the node, and advance `proxy_config` only on confirmed snapshots. Better still, tag each snapshot with the Mesh config version it was rendered from, and advance `proxy_config` to that version. **P3-4 (design): a compromised child can escape orphan reaping.** The child holds a duplicate of the agent's open file description, so `flock(LOCK_UN)` from the child releases the agent's lock. A child process forked from Traefik doesn't inherit PDEATHSIG and can keep the 80/443 listening sockets. After an agent restart, the Mesh firewall comes up with no gate, and `verify_unshared` *refuses* to start a new Traefik rather than reclaiming the ports. The escaped process keeps serving the public ports with the node's ACME keys. Only a cgroup (`cgroup.kill`) or a PID namespace contains this. This belongs in the threat model below, not in another patch. ### Narrowing or new classes? Judged by area, not by round count: - **Mesh firewall and conntrack: narrowing, converged.** In round 3 this area had a P1 (whole flows) and a P2 (revocation). Both fixes hold under real root on both backends, and what's left is one P3 bookkeeping race (P3-3). - **D330 public process: a new area, but clean.** It follows J133's notes point by point. - **Containment and identity of the Traefik child: new classes in new code, not narrowing.** Each round added a mechanism here: - round 2: the child ran as root; - round 3: a fixed 65532 identity plus privilege drop; - round 4: a created account, a `/proc` identity scan, PDEATHSIG re-armed, and a `/proc` fd-ownership killer. Each mechanism brought its own class of defect: directory modes and lost PDEATHSIG (round 3), and now an ID-allocation policy (P2-1), a kill without identity check (P3-1), and a compromised child escaping (P3-4). The cause is that nothing written says what containment has to guarantee. **Recommendation:** before more patching in this area, write a short threat model and design note. It should answer three questions: 1. Is a compromised Traefik in scope? 2. What may it reach (host UIDs, files, ACME keys, DNS tokens)? 3. May it outlive the agent? It should then pick one mechanism with measured costs. Options: - (a) a dedicated account with an explicit ID policy plus a cgroup per child (needs a delegated subtree: `Delegate=yes`); - (b) a user and PID namespace for the child with pre-bound sockets passed in (no host UID, so the collision question goes away); - (c) declare a compromised Traefik out of scope and remove the mechanisms that don't serve the crash case. The owner decides. Fixing P2-1 alone in the current design is fine as a stopgap. ### `remote_client` (`infraternal-agent --lib`), head vs origin/dev, interleaved, compute01 Features matched `cargo test --workspace` unification (head `native-streams,mesh,proxy,plugin-host-facts`; dev the same without `proxy`), with `INFRATERNAL_TEST_STATE_ROOT` set. A watchdog killed any test binary running past 240 s and recorded the hung test. ```text run 1 head exit=101 load=7.92 rc_tests=65 other_failed=24 rc_failed=[server_path_capability_is_advertised_on_native_and_fallback_transports unavailable_udp_does_not_add_seconds_before_tcp_admission] run 1 dev exit=101 load=16.39 rc_tests=65 208 passed; 14 failed other_failed=15 rc_failed=[] run 2 head exit=101 load=24.17 rc_tests=65 other_failed=24 rc_failed=[server_path_capability_… unavailable_udp_… winning_quic_drops_the_unfinished_tls_dial_before_control_dispatch] run 2 dev exit=101 load=13.57 rc_tests=65 208 passed; 14 failed other_failed=15 rc_failed=[] run 3 head exit=101 load=13.90 rc_tests=65 other_failed=24 rc_failed=[] run 3 dev exit=101 load=10.27 rc_tests=65 208 passed; 14 failed other_failed=15 rc_failed=[] run 4 head exit=101 load=11.93 rc_tests=65 other_failed=24 rc_failed=[] run 4 dev exit=101 load=7.97 rc_tests=65 208 passed; 14 failed other_failed=15 rc_failed=[] run 5 head exit=101 load=10.99 rc_tests=65 other_failed=24 rc_failed=[unavailable_udp_… winning_quic_drops_…] run 5 dev exit=101 load=13.41 rc_tests=65 208 passed; 14 failed other_failed=15 rc_failed=[] watchdog: head runs 1-5 all hung in daemon::remote_tests::cancelling_remote_dispatch_retires_the_borrowed_authenticated_channel; dev never did. ``` **Head fails `remote_client` in 3 of 5 runs, dev in 0 of 5. The cause is not branch code.** - All of head's `remote_client` failures happen in runs where `cancelling_remote_dispatch_retires_the_borrowed_authenticated_channel` hangs. - With only that test skipped, head passes all 65 `remote_client` tests: `213 passed; 10 failed`, and the 10 are the lifecycle/sessions tests that fail on dev too in this environment. - That test hangs on **unmodified dev as well** when run alone (`timeout 60`: head 3/3 hung, dev 3/3 hung) and when `daemon::remote_tests` runs as a module (head 3/3, dev 3/3). So it is a pre-existing, order-dependent hang. Dev's full-lib order happens to avoid it, and head's slightly different test set doesn't. The branch doesn't touch `daemon*` or `remote*` code except one line in `remote_supervisor_review_tests.rs`. Even so, the gate will be flakier after this merges, so the hang needs its own issue: a test-isolation fix and a timeout on that test's awaits. No code was changed, so the gate is not applicable. The size delta is unchanged from J62c's report.

kayg commented 2026-11-03 20:59:09 +00:00

Author

Owner

Copy link

J62d start (round 5)

Branch proxy-traefik-281-j62d, rebased onto origin/dev e8c7acd44 (clean, 49 commits), head 01f6d8b6f (= j62b ad2148fb9 rebased).

Plan (~3–4 h):

1. P2-1: explicit uid/gid from 60000–60999 (passwd, group, running processes, group members, primary-gid users checked; refuse on collision; never useradd --system allocation). Failing-first test. (~60 min)
2. P3-1: orphan reaper verifies uid + exe inode through a pidfd before pidfd_send_signal. (~40 min)
3. P3-2: supervisor restores the public gate after a crash restart acks a snapshot with public routes. (~30 min)
4. P3-3: Mesh proxy_config advances only on confirmed snapshots. (~30 min)
5. Part B: docs/design/proxy-containment.md (threat model + 3 options + recommendation), no implementation. (~30 min)
6. Separate issue for the pre-existing cancelling_remote_dispatch_retires_the_borrowed_authenticated_channel hang.
7. Root runs in a throwaway clone on compute01, then scripts/verify fast.
   Each item is one commit, pushed as it lands.

## J62d start (round 5) Branch `proxy-traefik-281-j62d`, rebased onto origin/dev `e8c7acd44` (clean, 49 commits), head `01f6d8b6f` (= j62b `ad2148fb9` rebased). Plan (~3–4 h): 1. P2-1: explicit uid/gid from 60000–60999 (passwd, group, running processes, group members, primary-gid users checked; refuse on collision; never `useradd --system` allocation). Failing-first test. (~60 min) 2. P3-1: orphan reaper verifies uid + exe inode through a pidfd before `pidfd_send_signal`. (~40 min) 3. P3-2: supervisor restores the public gate after a crash restart acks a snapshot with public routes. (~30 min) 4. P3-3: Mesh `proxy_config` advances only on confirmed snapshots. (~30 min) 5. Part B: `docs/design/proxy-containment.md` (threat model + 3 options + recommendation), no implementation. (~30 min) 6. Separate issue for the pre-existing `cancelling_remote_dispatch_retires_the_borrowed_authenticated_channel` hang. 7. Root runs in a throwaway clone on compute01, then `scripts/verify fast`. Each item is one commit, pushed as it lands.

kayg referenced this issue 2026-11-03 20:59:25 +00:00

J155: cancelling_remote_dispatch_retires_the_borrowed_authenticated_channel hangs on dev #332

kayg referenced this issue 2026-11-03 21:11:31 +00:00

Test hang: cancelling_remote_dispatch_retires_the_borrowed_authenticated_channel hangs on dev (order-dependent) #333

kayg commented 2026-11-03 21:43:49 +00:00

Author

Owner

Copy link

J62d progress

Branch proxy-traefik-281-j62d, head a0c0f3d47 (rebased onto dev e8c7acd44). Pushed commits:

• 7523941ec P2-1: explicit proxy UID/GID from 60514–60999, never useradd --system's choice
• e071902fc P3-1: orphan reaper checks identity through a pidfd (pidfd_send_signal)
• 6695cc9a5 P3-2: the public gate reopens when a restarted child confirms its routes
• ee579e9c8 P3-3: the Mesh flow reference moves only on a confirmed snapshot
• a0c0f3d47 Part B: docs/design/proxy-containment.md (for the owner's decision)

Done so far (dev-02, ordinary user): proxy lib 106 passed, mesh lib 79 passed, and the namespace test binary builds. Each commit passes cargo check on its own. Workspace clippy and fmt PASS in verify quick. Its unit stage fails only in infraternal-agent sessions/lifecycle/remote_client tests (NotFound), the same environment failures J152 saw on dev. verify fast on dev will show it.

Running now: root failing-first and fixed runs on compute01 (unit j62d-root2, log /root/j62d-root.log), and verify fast on compute01 (~claude/.cache/codex-scratch/j62d/fast-full.log). One target dir per VM: ~/.cache/codex-scratch/j62d/target, plus /root/j62d/target.

The pre-existing hang is filed as #333.

## J62d progress Branch `proxy-traefik-281-j62d`, head `a0c0f3d47` (rebased onto dev `e8c7acd44`). Pushed commits: - `7523941ec` P2-1: explicit proxy UID/GID from 60514–60999, never `useradd --system`'s choice - `e071902fc` P3-1: orphan reaper checks identity through a pidfd (`pidfd_send_signal`) - `6695cc9a5` P3-2: the public gate reopens when a restarted child confirms its routes - `ee579e9c8` P3-3: the Mesh flow reference moves only on a confirmed snapshot - `a0c0f3d47` Part B: `docs/design/proxy-containment.md` (for the owner's decision) Done so far (dev-02, ordinary user): proxy lib 106 passed, mesh lib 79 passed, and the namespace test binary builds. Each commit passes `cargo check` on its own. Workspace clippy and fmt PASS in `verify quick`. Its unit stage fails only in `infraternal-agent` sessions/lifecycle/remote_client tests (`NotFound`), the same environment failures J152 saw on dev. `verify fast` on dev will show it. Running now: root failing-first and fixed runs on compute01 (unit `j62d-root2`, log `/root/j62d-root.log`), and `verify fast` on compute01 (`~claude/.cache/codex-scratch/j62d/fast-full.log`). One target dir per VM: `~/.cache/codex-scratch/j62d/target`, plus `/root/j62d/target`. The pre-existing hang is filed as #333.

kayg commented 2026-11-03 22:51:55 +00:00

Author

Owner

Copy link

J62d finish: Part A and Part B done; ready for review (not merged)

Branch proxy-traefik-281-j62d, head a0c0f3d47, on origin/dev e8c7acd44. Commits:

• 7523941ec P2-1: the proxy account gets an explicit ID, the lowest free one in 60514–60999 with UID == GID. A candidate is free only when no passwd UID or primary GID uses it, no group uses it, no subuid/subgid range covers it and no running process uses it. The ID is passed to groupadd/useradd (or BusyBox addgroup/adduser); the tools' own allocation is never used. An existing account outside the band is refused with a clear message. A group with members, or one that is another user's primary group, is refused. The passwd/group entries are the record.
• e071902fc P3-1: the orphan reaper pins each candidate with pidfd_open. It then re-checks that the process holds the lock inode, that all its UIDs are the child identity's, and that /proc/<pid>/exe is the installed Traefik inode. The signal goes through pidfd_send_signal (new shim wrapper).
• 6695cc9a5 P3-2: the gate now carries Closed/Open/Cleared. A restarted child that confirms a snapshot with a public route reopens the gate.
• ee579e9c8 P3-3: confirmed is passed from the publisher to the node to the firewall, and proxy_config advances only on confirmed snapshots.
• a0c0f3d47 Part B: docs/design/proxy-containment.md. It proposes a threat model, compares three options, and recommends option 1 (a cgroup per child) now, with option 2 (namespaces) later for rootless. The owner's decision; nothing is built.

Separate issue for the pre-existing hang: #333. (It passed in this gate run: cancelling_remote_dispatch_retires_the_borrowed_authenticated_channel ... ok.)

Root, compute01, throwaway clone /root/j62d (run as a systemd unit through build-slot; removed afterwards, account removed)

Each fix was first reverted to show its test failing, then restored.

a0c0f3d47 docs(proxy): threat model and containment options for the Traefik child (#281)
=== passwd/group state: 999=[] [systemd-journal:x:999:] proxy=[]
account after cleanup: [] []
=== FF P2-1: old identity.rs (01f6d8b6f) + new range assertion
   Compiling uuid v1.27.0
test identity::tests::the_rootful_child_runs_as_the_dedicated_account ... FAILED
thread 'identity::tests::the_rootful_child_runs_as_the_dedicated_account' (290348) panicked at crates/plugins/proxy/src/identity.rs:352:9:
uid 999 outside the reserved range
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 102 filtered out; finished in 0.12s
account the old code created: [infraternal-proxy:x:999:987::/nonexistent:/usr/sbin/nologin]
=== FIX P2-1 refusal: an account left at the old allocation is not adopted
test identity::tests::the_rootful_child_runs_as_the_dedicated_account ... FAILED
thread 'identity::tests::the_rootful_child_runs_as_the_dedicated_account' (291127) panicked at crates/plugins/proxy/src/identity.rs:554:41:
resolve the proxy account: Custom { kind: PermissionDenied, error: "the infraternal-proxy account has UID 999 and GID 987, outside the reserved range 60514-60999; remove the account and its group so infraternal can recreate it" }
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 105 filtered out; finished in 0.00s
account after cleanup: [] []
=== FIX P2-1 allocation, with foreign workloads at 999 and at 60514 (the first candidate)
    PID   UID   GID COMMAND
 291166   999   999 sleep 900
 291167 60514 60514 sleep 900
test identity::tests::a_process_running_with_the_proxy_ids_is_a_collision ... ok
test identity::tests::the_agents_own_children_are_not_a_collision ... ok
test identity::tests::a_proxy_id_shared_with_another_account_or_group_is_refused ... ok
test identity::tests::a_proxy_group_that_other_users_belong_to_is_refused ... ok
test identity::tests::an_account_outside_the_proxy_range_is_not_adopted ... ok
test identity::tests::the_proxy_id_is_chosen_from_its_range_past_every_used_id ... ok
test identity::tests::user_namespace_root_without_a_system_id_range_is_rootless ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 105 filtered out; finished in 0.00s
test identity::tests::user_namespace_root_without_a_system_id_range_is_rootless ... ok
test identity::tests::the_rootful_child_runs_as_the_dedicated_account ... ok
test result: ok. 8 passed; 0 failed; 0 ignored; 0 measured; 98 filtered out; finished in 0.15s
chosen: passwd=[infraternal-proxy:x:60515:60515::/nonexistent:/usr/sbin/nologin] group=[infraternal-proxy:x:60515:]
=== FIX proxy lib as root (whole crate)
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 105 filtered out; finished in 0.00s
test result: ok. 106 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.16s
=== FIX proxy tests/process.rs as root
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s
=== FF P3-1: reaper without the pidfd identity checks
warning: field `uid` is never read
1623 |     uid: u32,
1736 | fn runs_only_as(pid: u32, uid: u32) -> bool {
test supervisor_tests::a_new_supervisor_stops_an_orphan_that_holds_the_child_lock ... FAILED
thread 'supervisor_tests::a_new_supervisor_stops_an_orphan_that_holds_the_child_lock' (294238) panicked at crates/plugins/proxy/src/supervisor_tests.rs:976:5:
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 105 filtered out; finished in 0.04s
=== FIX P3-1
test supervisor_tests::a_new_supervisor_stops_an_orphan_that_holds_the_child_lock ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 105 filtered out; finished in 0.04s
=== FF P3-2: restarted child never reopens the gate
test supervisor_tests::a_crash_restart_reopens_the_public_gate_for_applied_routes ... FAILED
thread 'supervisor_tests::a_crash_restart_reopens_the_public_gate_for_applied_routes' (295477) panicked at crates/plugins/proxy/src/supervisor_tests.rs:1137:5:
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 105 filtered out; finished in 20.08s
=== FIX P3-2
test supervisor_tests::the_public_gate_is_closed_before_a_child_starts_and_cleared_after_it_stops ... ok
test supervisor_tests::a_crash_restart_reopens_the_public_gate_for_applied_routes ... ok
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 104 filtered out; finished in 3.53s
=== FF P3-3: every update advances the reference
ns nft_an_unconfirmed_update_does_not_move_the_flow_reference
thread 'nft_an_unconfirmed_update_does_not_move_the_flow_reference' (314438) panicked at crates/plugins/mesh/tests/proxy_ingress_namespace.rs:394:5:
assertion `left == right` failed: a flow opened under routes Traefik had not yet narrowed must end at the confirmation
  left: "echo"
 right: "lost"
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 5 filtered out; finished in 0.62s
ns nf_tables_an_unconfirmed_update_does_not_move_the_flow_reference
thread 'nf_tables_an_unconfirmed_update_does_not_move_the_flow_reference' (314512) panicked at crates/plugins/mesh/tests/proxy_ingress_namespace.rs:394:5:
assertion `left == right` failed: a flow opened under routes Traefik had not yet narrowed must end at the confirmation
  left: "echo"
 right: "lost"
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 5 filtered out; finished in 1.42s
=== FIX P3-3 and every namespace test
ns nft_an_unconfirmed_update_does_not_move_the_flow_reference
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 5 filtered out; finished in 2.07s
ns nf_tables_an_unconfirmed_update_does_not_move_the_flow_reference
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 5 filtered out; finished in 3.07s
ns nft_revoking_a_route_ends_the_sources_open_proxied_flows
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 5 filtered out; finished in 5.18s
ns nf_tables_revoking_a_route_ends_the_sources_open_proxied_flows
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 5 filtered out; finished in 6.24s
ns nft_admits_a_port_granted_peer_to_the_proxy_listener_for_the_whole_flow
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 5 filtered out; finished in 2.18s
ns nf_tables_admits_a_port_granted_peer_to_the_proxy_listener_for_the_whole_flow
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 5 filtered out; finished in 3.12s
=== mesh lib as root
test result: ok. 79 passed; 0 failed; 14 ignored; 0 measured; 0 filtered out; finished in 0.98s
account after cleanup: [] []
leftover sleeps: 0
0
ROOT_DONE

Reading it:

• P2-1 failing-first: the old code allocated infraternal-proxy:x:999:987.
• P2-1 refusal: the new code refuses that leftover account by name.
• P2-1 allocation: with foreign workloads running at 999 and at 60514, the new account got 60515:60515.
• P3-1 failing-first: the bystander was killed. With the fix it survives and the orphan is reaped.
• P3-2 failing-first: the gate never reopened (20 s deadline). With the fix it reopens.
• P3-3 failing-first: left: "echo" right: "lost" on nft and iptables-nft. With the fix, all 6 namespace tests pass.

Ordinary user, dev-02: proxy lib 106 passed, mesh lib 79 passed. Each commit passes cargo check on its own.

scripts/verify fast, compute01, head a0c0f3d47

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
VERIFY_EXIT=0

Static size, D243 report only:

x86_64:  PT_INTERP: absent; size: 23148208 bytes; baseline 22418256; delta +729952 bytes (report only, D243 amended 2026-11-01)
aarch64: PT_INTERP: absent; size: 21057936 bytes; baseline 20453280; delta +604656 bytes (report only, D243 amended 2026-11-01)

These deltas are for the whole #281 branch against the gate's baseline. Round 5 adds only a few hundred lines of logic.

Full gate log on compute01: ~claude/.cache/codex-scratch/j62d/fast-full.log; root log: ~claude/.cache/codex-scratch/j62d/root.log.

Open items: one owner decision (Part B containment option). P3-4 (a compromised child that forks away) is deliberately left to that decision.

## J62d finish: Part A and Part B done; ready for review (not merged) Branch `proxy-traefik-281-j62d`, **head `a0c0f3d47`**, on origin/dev `e8c7acd44`. Commits: - `7523941ec` **P2-1**: the proxy account gets an explicit ID, the lowest free one in **60514–60999** with UID == GID. A candidate is free only when no passwd UID or primary GID uses it, no group uses it, no subuid/subgid range covers it and no running process uses it. The ID is passed to groupadd/useradd (or BusyBox addgroup/adduser); the tools' own allocation is never used. An existing account outside the band is refused with a clear message. A group with members, or one that is another user's primary group, is refused. The passwd/group entries are the record. - `e071902fc` **P3-1**: the orphan reaper pins each candidate with `pidfd_open`. It then re-checks that the process holds the lock inode, that all its UIDs are the child identity's, and that `/proc/<pid>/exe` is the installed Traefik inode. The signal goes through `pidfd_send_signal` (new shim wrapper). - `6695cc9a5` **P3-2**: the gate now carries `Closed`/`Open`/`Cleared`. A restarted child that confirms a snapshot with a public route reopens the gate. - `ee579e9c8` **P3-3**: `confirmed` is passed from the publisher to the node to the firewall, and `proxy_config` advances only on confirmed snapshots. - `a0c0f3d47` **Part B**: `docs/design/proxy-containment.md`. It proposes a threat model, compares three options, and recommends option 1 (a cgroup per child) now, with option 2 (namespaces) later for rootless. **The owner's decision**; nothing is built. Separate issue for the pre-existing hang: #333. (It passed in this gate run: `cancelling_remote_dispatch_retires_the_borrowed_authenticated_channel ... ok`.) ### Root, compute01, throwaway clone `/root/j62d` (run as a systemd unit through build-slot; removed afterwards, account removed) Each fix was first reverted to show its test failing, then restored. ```text a0c0f3d47 docs(proxy): threat model and containment options for the Traefik child (#281) === passwd/group state: 999=[] [systemd-journal:x:999:] proxy=[] account after cleanup: [] [] === FF P2-1: old identity.rs (01f6d8b6f) + new range assertion Compiling uuid v1.27.0 test identity::tests::the_rootful_child_runs_as_the_dedicated_account ... FAILED thread 'identity::tests::the_rootful_child_runs_as_the_dedicated_account' (290348) panicked at crates/plugins/proxy/src/identity.rs:352:9: uid 999 outside the reserved range test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 102 filtered out; finished in 0.12s account the old code created: [infraternal-proxy:x:999:987::/nonexistent:/usr/sbin/nologin] === FIX P2-1 refusal: an account left at the old allocation is not adopted test identity::tests::the_rootful_child_runs_as_the_dedicated_account ... FAILED thread 'identity::tests::the_rootful_child_runs_as_the_dedicated_account' (291127) panicked at crates/plugins/proxy/src/identity.rs:554:41: resolve the proxy account: Custom { kind: PermissionDenied, error: "the infraternal-proxy account has UID 999 and GID 987, outside the reserved range 60514-60999; remove the account and its group so infraternal can recreate it" } test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 105 filtered out; finished in 0.00s account after cleanup: [] [] === FIX P2-1 allocation, with foreign workloads at 999 and at 60514 (the first candidate) PID UID GID COMMAND 291166 999 999 sleep 900 291167 60514 60514 sleep 900 test identity::tests::a_process_running_with_the_proxy_ids_is_a_collision ... ok test identity::tests::the_agents_own_children_are_not_a_collision ... ok test identity::tests::a_proxy_id_shared_with_another_account_or_group_is_refused ... ok test identity::tests::a_proxy_group_that_other_users_belong_to_is_refused ... ok test identity::tests::an_account_outside_the_proxy_range_is_not_adopted ... ok test identity::tests::the_proxy_id_is_chosen_from_its_range_past_every_used_id ... ok test identity::tests::user_namespace_root_without_a_system_id_range_is_rootless ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 105 filtered out; finished in 0.00s test identity::tests::user_namespace_root_without_a_system_id_range_is_rootless ... ok test identity::tests::the_rootful_child_runs_as_the_dedicated_account ... ok test result: ok. 8 passed; 0 failed; 0 ignored; 0 measured; 98 filtered out; finished in 0.15s chosen: passwd=[infraternal-proxy:x:60515:60515::/nonexistent:/usr/sbin/nologin] group=[infraternal-proxy:x:60515:] === FIX proxy lib as root (whole crate) test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 105 filtered out; finished in 0.00s test result: ok. 106 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.16s === FIX proxy tests/process.rs as root test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s === FF P3-1: reaper without the pidfd identity checks warning: field `uid` is never read 1623 | uid: u32, 1736 | fn runs_only_as(pid: u32, uid: u32) -> bool { test supervisor_tests::a_new_supervisor_stops_an_orphan_that_holds_the_child_lock ... FAILED thread 'supervisor_tests::a_new_supervisor_stops_an_orphan_that_holds_the_child_lock' (294238) panicked at crates/plugins/proxy/src/supervisor_tests.rs:976:5: test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 105 filtered out; finished in 0.04s === FIX P3-1 test supervisor_tests::a_new_supervisor_stops_an_orphan_that_holds_the_child_lock ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 105 filtered out; finished in 0.04s === FF P3-2: restarted child never reopens the gate test supervisor_tests::a_crash_restart_reopens_the_public_gate_for_applied_routes ... FAILED thread 'supervisor_tests::a_crash_restart_reopens_the_public_gate_for_applied_routes' (295477) panicked at crates/plugins/proxy/src/supervisor_tests.rs:1137:5: test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 105 filtered out; finished in 20.08s === FIX P3-2 test supervisor_tests::the_public_gate_is_closed_before_a_child_starts_and_cleared_after_it_stops ... ok test supervisor_tests::a_crash_restart_reopens_the_public_gate_for_applied_routes ... ok test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 104 filtered out; finished in 3.53s === FF P3-3: every update advances the reference ns nft_an_unconfirmed_update_does_not_move_the_flow_reference thread 'nft_an_unconfirmed_update_does_not_move_the_flow_reference' (314438) panicked at crates/plugins/mesh/tests/proxy_ingress_namespace.rs:394:5: assertion `left == right` failed: a flow opened under routes Traefik had not yet narrowed must end at the confirmation left: "echo" right: "lost" test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 5 filtered out; finished in 0.62s ns nf_tables_an_unconfirmed_update_does_not_move_the_flow_reference thread 'nf_tables_an_unconfirmed_update_does_not_move_the_flow_reference' (314512) panicked at crates/plugins/mesh/tests/proxy_ingress_namespace.rs:394:5: assertion `left == right` failed: a flow opened under routes Traefik had not yet narrowed must end at the confirmation left: "echo" right: "lost" test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 5 filtered out; finished in 1.42s === FIX P3-3 and every namespace test ns nft_an_unconfirmed_update_does_not_move_the_flow_reference test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 5 filtered out; finished in 2.07s ns nf_tables_an_unconfirmed_update_does_not_move_the_flow_reference test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 5 filtered out; finished in 3.07s ns nft_revoking_a_route_ends_the_sources_open_proxied_flows test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 5 filtered out; finished in 5.18s ns nf_tables_revoking_a_route_ends_the_sources_open_proxied_flows test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 5 filtered out; finished in 6.24s ns nft_admits_a_port_granted_peer_to_the_proxy_listener_for_the_whole_flow test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 5 filtered out; finished in 2.18s ns nf_tables_admits_a_port_granted_peer_to_the_proxy_listener_for_the_whole_flow test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 5 filtered out; finished in 3.12s === mesh lib as root test result: ok. 79 passed; 0 failed; 14 ignored; 0 measured; 0 filtered out; finished in 0.98s account after cleanup: [] [] leftover sleeps: 0 0 ROOT_DONE ``` Reading it: - **P2-1 failing-first**: the old code allocated `infraternal-proxy:x:999:987`. - **P2-1 refusal**: the new code refuses that leftover account by name. - **P2-1 allocation**: with foreign workloads running at 999 and at 60514, the new account got **60515:60515**. - **P3-1 failing-first**: the bystander was killed. With the fix it survives and the orphan is reaped. - **P3-2 failing-first**: the gate never reopened (20 s deadline). With the fix it reopens. - **P3-3 failing-first**: `left: "echo" right: "lost"` on nft and iptables-nft. With the fix, all 6 namespace tests pass. Ordinary user, dev-02: proxy lib 106 passed, mesh lib 79 passed. Each commit passes `cargo check` on its own. ### `scripts/verify fast`, compute01, head `a0c0f3d47` ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | static builds | PASS | +---------------------------+--------------+ VERIFY_EXIT=0 ``` Static size, D243 report only: ```text x86_64: PT_INTERP: absent; size: 23148208 bytes; baseline 22418256; delta +729952 bytes (report only, D243 amended 2026-11-01) aarch64: PT_INTERP: absent; size: 21057936 bytes; baseline 20453280; delta +604656 bytes (report only, D243 amended 2026-11-01) ``` These deltas are for the whole #281 branch against the gate's baseline. Round 5 adds only a few hundred lines of logic. Full gate log on compute01: `~claude/.cache/codex-scratch/j62d/fast-full.log`; root log: `~claude/.cache/codex-scratch/j62d/root.log`. Open items: one owner decision (Part B containment option). P3-4 (a compromised child that forks away) is deliberately left to that decision.

kayg commented 2026-11-03 23:28:20 +00:00

Author

Owner

Copy link

J156 independent security review of the round-5 commits: APPROVE

Target: proxy-traefik-281-j62d at a0c0f3d47. Scope: only 7523941ec, e071902fc, 6695cc9a5 and ee579e9c8, against J152's round-4 review (77190). The design note a0c0f3d47 is the owner's decision and is not reviewed here. This review was read-only; nothing was pushed.

The four fixes do what they claim. None of them breaks an item J152 passed. Two findings are non-blocking P3s, both on edges where the code fails closed.

Root run (compute01, euid 0, /root/j156 git-archive of a0c0f3d47, through build-slot)

cargo test -p infraternal-plugin-proxy --lib
test identity::tests::an_account_outside_the_proxy_range_is_not_adopted ... ok
test identity::tests::a_proxy_group_that_other_users_belong_to_is_refused ... ok
test identity::tests::the_proxy_id_is_chosen_from_its_range_past_every_used_id ... ok
test identity::tests::the_rootful_child_runs_as_the_dedicated_account ... ok
test supervisor_tests::a_new_supervisor_stops_an_orphan_that_holds_the_child_lock ... ok
test supervisor_tests::a_crash_restart_reopens_the_public_gate_for_applied_routes ... ok
test supervisor_tests::the_public_gate_is_closed_before_a_child_starts_and_cleared_after_it_stops ... ok
test result: ok. 106 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.50s
account: infraternal-proxy:x:60514:60514::/nonexistent:/usr/sbin/nologin infraternal-proxy:x:60514:

The tests leave the account behind, so I removed it afterwards (after: [] []). No process was left running as 60514, and /root/j156 is gone. I did not rerun the mesh namespace tests: ee579e9c8 is a one-bit change, and J62d's fail-first evidence for it on both backends is convincing.

Checked

• P2-1 (ID allocation), 7523941ec: fixed. The ID is chosen from 60514–60999 with UID == GID.
  • Every candidate is checked against passwd UIDs and primary GIDs, group GIDs, /etc/subuid and /etc/subgid ranges (u64, saturating), and all Uid/Gid/Groups of running processes. The ID is then passed explicitly to groupadd/useradd or addgroup/adduser.
  • TOCTOU: someone else can take the ID between choose_id and useradd. useradd and adduser then fail on the duplicate ID without -o, the group is rolled back, and start fails. A concurrent agent that wins is caught by the second lookup(), and check_in_range and check_unique run on whatever account exists.
  • A process that starts as that ID later is caught by verify_unshared before every child start (unchanged from round 4).
  • An account left outside the band is refused, not adopted. Group members and other users with the GID as primary group are refused. The module doc no longer claims fresh IDs own no files.
  • maps_proxy_range now requires the mapping from 0 to reach 61000.
• P3-1 (reaper), e071902fc: fixed.
  • Lock held: the caller holds the supervisor lock before acquire_child_lock (supervisor.rs:1056-1066), unchanged.
  • Order: scan, pidfd_open, then re-check after the pin: the process holds the inode, all four UIDs are the child UID, and /proc/<pid>/exe matches the installed Traefik's dev and inode. The signal then goes through the pidfd.
  • If the signal lands, the pinned task was alive the whole time, so the /proc/<pid> reads described it. A PID reused before the pin fails the re-check. A PID reused after it gives ESRCH.
  • If the executable is missing, nothing matches (executable: None, returns early). A holder that doesn't match is never signalled, and start fails closed with the existing timeout.
• Unsafe signal_process_descriptor (infraternal-shim/src/pty.rs:1590-1614): sound.
  • fd lifetime: it takes a BorrowedFd<'_> that outlives the call, and the OwnedFd from process_exit_descriptor stays alive in the caller across the call.
  • Signal number: nix::sys::signal::Signal is a closed enum of valid signals, so no out-of-range number can be built.
  • The siginfo argument is null, flags are 0, and the kernel reads no user memory.
  • errno: last_os_error() is read right after the syscall, with nothing in between.
  • Variadic arguments: fd and sig are c_int and flags is u32; the kernel takes int/unsigned int and truncates, so the upper register bits don't matter.
  • Kernels without pidfd return ENOSYS from pidfd_open or pidfd_send_signal. The caller ignores the error and nothing is signalled, so this fails closed.
• P3-2 (three-state gate), 6695cc9a5: the gate can't open for an unconfirmed config.
  • Open is sent only from start_runtime_child after start_managed_child returns Ok (supervisor.rs:1266-1278).
  • That return happens only after verify_health_listener (child-owned port) and wait_for_config_generation acked the snapshot written for these exact services (:1368, :1415-1420). It is sent only when one of them is public.
  • A failed start sends Cleared, and a failed Open leaves the gate Closed.
  • Closed now also resets public_route_active, which is correct: a new child isn't live until it confirms.
  • J152's P3-B (the gate follows the child's life) holds: Cleared on reap or failed start, Closed before spawn.
• P3-3 (confirmed flag), ee579e9c8: fixed.
  • proxy_config advances only when confirmed is true (mesh/src/firewall.rs:193-195).
  • The supervisor's gate updates, including Open, go through update(false, …), so a crash restart from retained routes can't move the reference.
  • Rules are still installed, and retirement still runs against the unmoved reference, so unconfirmed calls fail safe.
  • P2-C from round 4 is unaffected.
• No regressions to J152's passing items. The nft and iptables continuation rules, the redaction, health-ack ownership, PDEATHSIG and the D330 split are not touched by these commits, apart from the gate-type and observer signature changes above.

Findings (non-blocking)

P3-a: ID collision checks see only local files, while the account lookup uses NSS.
crates/plugins/proxy/src/identity.rs:178-187 (choose_id) and :384-411 (check_unique) parse /etc/passwd and /etc/group. lookup() (:269-271) goes through NSS.

• Scenario: a host with SSSD/LDAP or systemd-userdb has a directory user with UID 60514 who is not logged in. No process runs as that UID, so the process check misses it. shadow's useradd checks only local files and creates infraternal-proxy with 60514. The proxy now shares an identity with a real user's files on NFS or home directories.
• Fix: in choose_id's taken closure and in check_unique, also reject an ID when User::from_uid(id) or Group::from_gid(id) returns an entry whose name isn't infraternal-proxy. This is cheap, at most a few hundred NSS lookups the first time only.

P3-b: an orphan from an older Traefik release is never reaped, and start then fails permanently.
crates/plugins/proxy/src/supervisor.rs:1060-1065 and :1700-1702. Releases are installed content-addressed (installer.rs:261), so an upgrade gives settings.executable a new inode.

• Scenario: PDEATHSIG fails to fire, the agent crashes, and a component update installs a new Traefik. On the next start the orphan runs the old inode, does not match, keeps holding the lock and the 80/443 ports, and every start fails with "orphaned proxy process" until an operator kills it by hand.
• This fails closed, so it is availability only.
• Fix: match the exe inode against every installed bundle executable, not only the current one. Or, at minimum, name the holding PID in the timeout error so the operator can act on it.

Gate not applicable: no code was changed by this job.

## J156 independent security review of the round-5 commits: **APPROVE** Target: `proxy-traefik-281-j62d` at `a0c0f3d47`. Scope: only `7523941ec`, `e071902fc`, `6695cc9a5` and `ee579e9c8`, against J152's round-4 review (77190). The design note `a0c0f3d47` is the owner's decision and is not reviewed here. This review was read-only; nothing was pushed. The four fixes do what they claim. None of them breaks an item J152 passed. Two findings are non-blocking P3s, both on edges where the code fails closed. ### Root run (compute01, euid 0, `/root/j156` git-archive of `a0c0f3d47`, through build-slot) ```text cargo test -p infraternal-plugin-proxy --lib test identity::tests::an_account_outside_the_proxy_range_is_not_adopted ... ok test identity::tests::a_proxy_group_that_other_users_belong_to_is_refused ... ok test identity::tests::the_proxy_id_is_chosen_from_its_range_past_every_used_id ... ok test identity::tests::the_rootful_child_runs_as_the_dedicated_account ... ok test supervisor_tests::a_new_supervisor_stops_an_orphan_that_holds_the_child_lock ... ok test supervisor_tests::a_crash_restart_reopens_the_public_gate_for_applied_routes ... ok test supervisor_tests::the_public_gate_is_closed_before_a_child_starts_and_cleared_after_it_stops ... ok test result: ok. 106 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.50s account: infraternal-proxy:x:60514:60514::/nonexistent:/usr/sbin/nologin infraternal-proxy:x:60514: ``` The tests leave the account behind, so I removed it afterwards (`after: [] []`). No process was left running as 60514, and `/root/j156` is gone. I did not rerun the mesh namespace tests: `ee579e9c8` is a one-bit change, and J62d's fail-first evidence for it on both backends is convincing. ### Checked - **P2-1 (ID allocation), `7523941ec`: fixed.** The ID is chosen from 60514–60999 with UID == GID. - Every candidate is checked against passwd UIDs and primary GIDs, group GIDs, `/etc/subuid` and `/etc/subgid` ranges (u64, saturating), and all Uid/Gid/Groups of running processes. The ID is then passed explicitly to groupadd/useradd or addgroup/adduser. - TOCTOU: someone else can take the ID between `choose_id` and `useradd`. useradd and adduser then fail on the duplicate ID without `-o`, the group is rolled back, and start fails. A concurrent agent that wins is caught by the second `lookup()`, and `check_in_range` and `check_unique` run on whatever account exists. - A process that starts as that ID later is caught by `verify_unshared` before every child start (unchanged from round 4). - An account left outside the band is refused, not adopted. Group members and other users with the GID as primary group are refused. The module doc no longer claims fresh IDs own no files. - `maps_proxy_range` now requires the mapping from 0 to reach 61000. - **P3-1 (reaper), `e071902fc`: fixed.** - Lock held: the caller holds the supervisor lock before `acquire_child_lock` (`supervisor.rs:1056-1066`), unchanged. - Order: scan, `pidfd_open`, then re-check *after* the pin: the process holds the inode, all four UIDs are the child UID, and `/proc/<pid>/exe` matches the installed Traefik's dev and inode. The signal then goes through the pidfd. - If the signal lands, the pinned task was alive the whole time, so the `/proc/<pid>` reads described it. A PID reused before the pin fails the re-check. A PID reused after it gives ESRCH. - If the executable is missing, nothing matches (`executable: None`, returns early). A holder that doesn't match is never signalled, and start fails closed with the existing timeout. - **Unsafe `signal_process_descriptor` (`infraternal-shim/src/pty.rs:1590-1614`): sound.** - fd lifetime: it takes a `BorrowedFd<'_>` that outlives the call, and the `OwnedFd` from `process_exit_descriptor` stays alive in the caller across the call. - Signal number: `nix::sys::signal::Signal` is a closed enum of valid signals, so no out-of-range number can be built. - The siginfo argument is null, flags are 0, and the kernel reads no user memory. - errno: `last_os_error()` is read right after the syscall, with nothing in between. - Variadic arguments: fd and sig are `c_int` and flags is `u32`; the kernel takes `int`/`unsigned int` and truncates, so the upper register bits don't matter. - Kernels without pidfd return ENOSYS from `pidfd_open` or `pidfd_send_signal`. The caller ignores the error and nothing is signalled, so this fails closed. - **P3-2 (three-state gate), `6695cc9a5`: the gate can't open for an unconfirmed config.** - `Open` is sent only from `start_runtime_child` after `start_managed_child` returns Ok (`supervisor.rs:1266-1278`). - That return happens only after `verify_health_listener` (child-owned port) and `wait_for_config_generation` acked the snapshot written for these exact services (`:1368`, `:1415-1420`). It is sent only when one of them is `public`. - A failed start sends `Cleared`, and a failed `Open` leaves the gate `Closed`. - `Closed` now also resets `public_route_active`, which is correct: a new child isn't live until it confirms. - J152's P3-B (the gate follows the child's life) holds: `Cleared` on reap or failed start, `Closed` before spawn. - **P3-3 (`confirmed` flag), `ee579e9c8`: fixed.** - `proxy_config` advances only when `confirmed` is true (`mesh/src/firewall.rs:193-195`). - The supervisor's gate updates, including `Open`, go through `update(false, …)`, so a crash restart from retained routes can't move the reference. - Rules are still installed, and retirement still runs against the unmoved reference, so unconfirmed calls fail safe. - P2-C from round 4 is unaffected. - **No regressions to J152's passing items.** The nft and iptables continuation rules, the redaction, health-ack ownership, PDEATHSIG and the D330 split are not touched by these commits, apart from the gate-type and observer signature changes above. ### Findings (non-blocking) **P3-a: ID collision checks see only local files, while the account lookup uses NSS.** `crates/plugins/proxy/src/identity.rs:178-187` (`choose_id`) and `:384-411` (`check_unique`) parse `/etc/passwd` and `/etc/group`. `lookup()` (`:269-271`) goes through NSS. - **Scenario:** a host with SSSD/LDAP or systemd-userdb has a directory user with UID 60514 who is not logged in. No process runs as that UID, so the process check misses it. shadow's `useradd` checks only local files and creates `infraternal-proxy` with 60514. The proxy now shares an identity with a real user's files on NFS or home directories. - **Fix:** in `choose_id`'s `taken` closure and in `check_unique`, also reject an ID when `User::from_uid(id)` or `Group::from_gid(id)` returns an entry whose name isn't `infraternal-proxy`. This is cheap, at most a few hundred NSS lookups the first time only. **P3-b: an orphan from an older Traefik release is never reaped, and start then fails permanently.** `crates/plugins/proxy/src/supervisor.rs:1060-1065` and `:1700-1702`. Releases are installed content-addressed (`installer.rs:261`), so an upgrade gives `settings.executable` a new inode. - **Scenario:** PDEATHSIG fails to fire, the agent crashes, and a component update installs a new Traefik. On the next start the orphan runs the old inode, does not match, keeps holding the lock and the 80/443 ports, and every start fails with "orphaned proxy process" until an operator kills it by hand. - This fails closed, so it is availability only. - **Fix:** match the exe inode against every installed bundle executable, not only the current one. Or, at minimum, name the holding PID in the timeout error so the operator can act on it. Gate not applicable: no code was changed by this job.

kayg referenced this issue 2026-11-04 02:56:10 +00:00

Head-to-head: infraternal mesh vs NetBird/Tailscale, infraternal sessions vs tmux/Zellij #260

kayg commented 2026-11-04 02:56:10 +00:00

Author

Owner

Copy link

Finding from J103e (#260): 13b6836a5 slows the remote supervisor's first dial

On a fresh agent state dir, the proxy plugin creates proxy-secrets with a durable parent-directory fsync during plugin start. The remote supervisor dials only after plugin start. On compute01, beside the login_binary tests, that fsync took 6.5 s (strace), and dev e0633114b fails remote_client::tests::unavailable_udp_does_not_add_seconds_before_tcp_admission / winning_quic_drops_the_unfinished_tls_dial_before_control_dispatch in 3 of 5 runs of cargo test --workspace --locked --lib -- remote_client:: login_binary::.

Full evidence: #260 (the J103e dev finding). The same class of bug was fixed for the mesh engine record on mesh-gotatun-linux (92f4c1dba). Not fixed here.

## Finding from J103e (#260): `13b6836a5` slows the remote supervisor's first dial On a fresh agent state dir, the proxy plugin creates `proxy-secrets` with a durable parent-directory fsync during plugin start. The remote supervisor dials only after plugin start. On compute01, beside the `login_binary` tests, that fsync took **6.5 s** (strace), and dev `e0633114b` fails `remote_client::tests::unavailable_udp_does_not_add_seconds_before_tcp_admission` / `winning_quic_drops_the_unfinished_tls_dial_before_control_dispatch` in 3 of 5 runs of `cargo test --workspace --locked --lib -- remote_client:: login_binary::`. Full evidence: #260 (the J103e dev finding). The same class of bug was fixed for the mesh engine record on `mesh-gotatun-linux` (`92f4c1dba`). Not fixed here.

kayg referenced this issue 2026-11-04 02:57:23 +00:00

macOS mesh node #265

kayg commented 2026-11-04 03:03:31 +00:00

Author

Owner

Copy link

J159 done for both P3s (J156, comment 77247). Branch proxy-p3-j159, head b985e2b1d5be329e26d3824adf8c7c8a89547503, base 91112e9da (dev). Not merged, not deployed.

Commits (atomic, pushed):

• 028a4bd3b proxy: look up candidate IDs through the system user directory. choose_id and check_unique now also call getpwuid_r/getgrgid_r (nix) through a small Directory trait. An ID the directory knows is skipped, our account must resolve to its own name only, and a failing lookup counts as taken or refused. Rejected: a getent subprocess. Test: ids_held_by_a_directory_service_are_never_chosen_or_shared, using a fake lookup.
• b985e2b1d proxy: stop an orphan left by an older Traefik release. The orphan check now matches any installed release's executable (<store>/<digest>/bin/traefik siblings of the current one, only when the path has that exact shape). The pidfd, lock-holder and UID checks are unchanged. A holder that still isn't ours is named in the error with its PID and executable. Tests: an_orphan_from_an_older_traefik_release_is_stopped_too and a_holder_that_is_not_an_installed_traefik_is_named_in_the_error. The existing orphan test is unchanged in behaviour.

Real-environment proof (root, compute01, throwaway clone /root/j159): cargo test -p infraternal-plugin-proxy -- --test-threads=1 gave test result: ok. 109 passed; 0 failed. That included the_rootful_child_runs_as_the_dedicated_account, all three orphan tests, the directory test and traefik_child_obeys_the_rootful_identity_and_capability_boundary. Afterwards I removed the clone, its target dir and the infraternal-proxy user and group (checked with getent). verify quick on compute01: fmt, clippy and unit tests (changed) all PASS, VERIFY_EXIT=0.

Gate status: scripts/verify fast is NOT green, and I am not claiming it is. Three full runs failed in test, each on a different test outside crates/plugins/proxy. Every other row passed, including clippy and static builds (x86_64 +102400 B accepted; aarch64 +611664 B vs baseline, report-only).

• compute01 run 1: remote_client::tests::winning_quic_drops_the_unfinished_tls_dial_before_control_dispatch (Elapsed at remote_client_tests.rs:205). Passes alone: cargo test --workspace -- winning_quic gave ok.
• compute01 run 2: status_restart::service_states_are_truthful_in_text_and_json (state "unknown", 1.85 s). Passes alone on my branch (ok, 4.97 s) and on unmodified origin/dev 91112e9da (ok, 4.93 s).
• compute01 run 3: remote_client::tests::unavailable_udp_does_not_add_seconds_before_tcp_admission (remote_client_tests.rs:167) and winning_quic_… again. Both are wall-clock assertions.
• dev-02 run: agent_queues::saturated_terminal_queues_preserve_every_byte_and_daemon_liveness (agent_queues.rs:14, 53 s).
  Both VMs were at load average 8–16 with other gates running. These look like load-sensitive timing flakes. I did not prove them on origin/dev under identical load, since that needs a full dev gate per failure. My diff touches only identity.rs, supervisor.rs, supervisor_tests.rs and the visibility of valid_component_digest in installer.rs. Logs: compute01 ~/j159-fast-run1.log, ~/j159-fast-run2.log, ~/j159-fast.log; dev-02 ~/j159-fast.log.

Needs the orchestrator's call: accept the flakes as known, or rerun fast on a quieter VM.

J159 done for both P3s (J156, comment 77247). Branch `proxy-p3-j159`, head `b985e2b1d5be329e26d3824adf8c7c8a89547503`, base `91112e9da` (dev). Not merged, not deployed. Commits (atomic, pushed): - `028a4bd3b` proxy: look up candidate IDs through the system user directory. `choose_id` and `check_unique` now also call getpwuid_r/getgrgid_r (nix) through a small `Directory` trait. An ID the directory knows is skipped, our account must resolve to its own name only, and a failing lookup counts as taken or refused. Rejected: a `getent` subprocess. Test: `ids_held_by_a_directory_service_are_never_chosen_or_shared`, using a fake lookup. - `b985e2b1d` proxy: stop an orphan left by an older Traefik release. The orphan check now matches any installed release's executable (`<store>/<digest>/bin/traefik` siblings of the current one, only when the path has that exact shape). The pidfd, lock-holder and UID checks are unchanged. A holder that still isn't ours is named in the error with its PID and executable. Tests: `an_orphan_from_an_older_traefik_release_is_stopped_too` and `a_holder_that_is_not_an_installed_traefik_is_named_in_the_error`. The existing orphan test is unchanged in behaviour. Real-environment proof (root, compute01, throwaway clone /root/j159): `cargo test -p infraternal-plugin-proxy -- --test-threads=1` gave `test result: ok. 109 passed; 0 failed`. That included `the_rootful_child_runs_as_the_dedicated_account`, all three orphan tests, the directory test and `traefik_child_obeys_the_rootful_identity_and_capability_boundary`. Afterwards I removed the clone, its target dir and the `infraternal-proxy` user and group (checked with getent). `verify quick` on compute01: fmt, clippy and unit tests (changed) all PASS, `VERIFY_EXIT=0`. **Gate status: `scripts/verify fast` is NOT green, and I am not claiming it is.** Three full runs failed in `test`, each on a different test outside `crates/plugins/proxy`. Every other row passed, including clippy and static builds (x86_64 +102400 B accepted; aarch64 +611664 B vs baseline, report-only). - compute01 run 1: `remote_client::tests::winning_quic_drops_the_unfinished_tls_dial_before_control_dispatch` (Elapsed at remote_client_tests.rs:205). Passes alone: `cargo test --workspace -- winning_quic` gave ok. - compute01 run 2: `status_restart::service_states_are_truthful_in_text_and_json` (state "unknown", 1.85 s). Passes alone on my branch (ok, 4.97 s) and on unmodified `origin/dev` `91112e9da` (ok, 4.93 s). - compute01 run 3: `remote_client::tests::unavailable_udp_does_not_add_seconds_before_tcp_admission` (remote_client_tests.rs:167) and `winning_quic_…` again. Both are wall-clock assertions. - dev-02 run: `agent_queues::saturated_terminal_queues_preserve_every_byte_and_daemon_liveness` (agent_queues.rs:14, 53 s). Both VMs were at load average 8–16 with other gates running. These look like load-sensitive timing flakes. I did not prove them on `origin/dev` under identical load, since that needs a full dev gate per failure. My diff touches only `identity.rs`, `supervisor.rs`, `supervisor_tests.rs` and the visibility of `valid_component_digest` in `installer.rs`. Logs: compute01 `~/j159-fast-run1.log`, `~/j159-fast-run2.log`, `~/j159-fast.log`; dev-02 `~/j159-fast.log`. Needs the orchestrator's call: accept the flakes as known, or rerun `fast` on a quieter VM.

kayg referenced this issue 2026-11-04 03:26:23 +00:00

infraternal up + one-command install #263

kayg commented 2026-11-04 03:37:52 +00:00

Author

Owner

Copy link

J164 start: a cgroup per Traefik child (D336, option 1 of docs/design/proxy-containment.md)

Branch proxy-cgroup-j164, based on origin/dev 109e14cb7. Not merged, not deployed.

Plan (estimate ~5 h):

1. Measure first (~1 h): on compute01, as root and as claude, time creating a leaf cgroup, moving a process into it (cgroup.procs and CLONE_INTO_CGROUP), cgroup.kill plus waiting for cgroup.events populated 0, and rmdir. Compare that with today's spawn and stop path, plus steady-state CPU and RSS. Numbers go here before I build anything further.
2. Build (~2.5 h): the child is spawned straight into <agent cgroup>/proxy-<generation>. Stop means cgroup.kill, wait for it to empty, then rmdir. Rootful uses Delegate=yes on the infraternal agent unit, with KillMode=process kept. Rootless uses the user manager's delegation. If neither is writable, it falls back to today's pidfd reaping, and mesh status / proxy status say so.
3. Adoption (~0.5 h): on start, any leftover proxy-* leaf is killed and removed. J159's executable-based check stays as the fallback path only.
4. Tests (~1 h): as root in /root/j164 on compute01, a fake Traefik that double-forks a grandchild must leave nothing behind after a stop and after an agent crash (SIGKILL). Rootless too, if claude has delegation.

## J164 start: a cgroup per Traefik child (D336, option 1 of `docs/design/proxy-containment.md`) Branch `proxy-cgroup-j164`, based on `origin/dev` `109e14cb7`. Not merged, not deployed. Plan (estimate ~5 h): 1. **Measure first** (~1 h): on compute01, as root and as `claude`, time creating a leaf cgroup, moving a process into it (`cgroup.procs` and `CLONE_INTO_CGROUP`), `cgroup.kill` plus waiting for `cgroup.events populated 0`, and `rmdir`. Compare that with today's spawn and stop path, plus steady-state CPU and RSS. Numbers go here before I build anything further. 2. **Build** (~2.5 h): the child is spawned straight into `<agent cgroup>/proxy-<generation>`. Stop means `cgroup.kill`, wait for it to empty, then `rmdir`. Rootful uses `Delegate=yes` on the infraternal agent unit, with `KillMode=process` kept. Rootless uses the user manager's delegation. If neither is writable, it falls back to today's pidfd reaping, and `mesh status` / `proxy status` say so. 3. **Adoption** (~0.5 h): on start, any leftover `proxy-*` leaf is killed and removed. J159's executable-based check stays as the fallback path only. 4. **Tests** (~1 h): as root in `/root/j164` on compute01, a fake Traefik that double-forks a grandchild must leave nothing behind after a stop and after an agent crash (SIGKILL). Rootless too, if `claude` has delegation.

kayg commented 2026-11-04 03:44:01 +00:00

Author

Owner

Copy link

J164 measurements: cost of a cgroup per Traefik child

The benchmark is cgbench.rs, a standalone file with no dependencies, built with the flake's rustc. Each iteration starts sh -c 'sleep 1000 & exec sleep 1000', so it has a direct child plus a forked grandchild.

• Today: the same fork path production already uses (std falls back from posix_spawn to fork because of the pre-exec hooks). Stop kills and reaps the direct child only. The grandchild survives, so the benchmark removes it afterwards, untimed.
• cgroup: mkdir a leaf under the process's own cgroup. The pre-exec hook writes 0 to an inherited cgroup.procs descriptor, so the child is inside the leaf before exec and before it can fork. Stop is cgroup.kill, reap the direct child, poll cgroup.events until populated 0, then rmdir.

Rootful: perf VM, perf-run --root (cpus 1–3), kernel 6.12.113, 500 iterations, run 2 of 2 (run 1 within ±5%)

today: spawn (fork path)           p50=187.8us p90=223.1us p99=359.0us
today: kill+reap child only        p50= 41.0us p90= 60.0us p99=120.8us   (grandchild left alive)
cgroup: mkdir leaf                 p50= 16.0us p90= 22.0us p99= 30.9us
cgroup: spawn joining leaf         p50=199.4us p90=241.7us p99=478.0us
cgroup: start total (mkdir+spawn)  p50=224.6us p90=267.7us p99=501.2us
cgroup: kill+reap+empty (2 procs)  p50= 63.1us p90=169.5us p99=250.8us
cgroup: rmdir                      p50= 10.8us p90= 14.1us p99= 24.8us
cgroup: stop total                 p50= 76.9us p90=182.2us p99=261.5us
steal during run: 2 of 3377 jiffies

• Start: +37 µs at p50 (mkdir 16 µs, plus about 12 µs for the cgroup.procs write in the child). Traefik's own start-up to a healthy ping takes hundreds of milliseconds, so this is below 0.1% of a child start.
• Stop: +36 µs at p50, +122 µs at p90. That buys killing both processes and confirming the leaf is empty. Today's stop kills one process and leaves the grandchild behind.
• A restart is a stop plus a start: about +75 µs at p50.

Steady state (same perf VM run)

• CPU: a fixed 400M-round integer loop takes plain 733.8–738.8 ms and in the leaf 733.0–742.4 ms, across 5 alternating pairs. That is within noise. No cgroup controller is enabled in the leaf, so the kernel charges nothing extra per tick, and the infraternal agent does no cgroup work between start and stop: no polling, no timers.
• Child RSS: 2224–2300 KiB in both cases.
• Kernel memory per leaf: 5.1–9.8 KiB (Slab+Percpu delta over 1000 empty leaves, 6 rounds; this VM is shared, so treat it as an upper bound). There are at most two leaves per node: the mixed process, plus the D330 rootless public process.

Rootless: compute01, systemd-run --user -p Delegate=yes, 300 iterations (load average 5–17, so only the relative numbers mean anything)

today: spawn p50=715us   today: kill+reap p50=173us
cgroup: start total p50=756us   cgroup: stop total p50=352us   mkdir 44us   rmdir 29us

Same shape as rootful: the deltas are mkdir plus rmdir plus one write. This is the same kernel path, so I use the perf VM numbers as the reference.

Delegation, observed on compute01 (systemd 257)

• System manager: a Delegate=yes unit's cgroup carries the trusted.delegate=1 and user.delegate=1 xattrs. A unit without it carries neither. Rootful detection uses this xattr.
• User manager: with or without Delegate=yes, no delegate xattr is set. The whole user@1500.service tree is owned by claude, though, while a login session-*.scope is owned by root. Rootless therefore uses the agent's own cgroup when that cgroup is owned by the agent's user and writable, and otherwise falls back.

What this means for the build

• The leaf is created under the process's own cgroup (proxy-<n>). No controllers are enabled in it (subtree_control stays empty), so the cgroup v2 "no internal processes" rule never applies, and the infraternal agent itself stays where systemd put it.
• I add Delegate=yes to both generated units (the server unit with its embedded infraternal agent, and the host bootstrap unit). KillMode=process stays. Session shims already run in their own infraternal-session-*.scope, so delegation doesn't touch them.
• Existing installs: their unit has no Delegate=yes, so they fall back to today's pidfd reaping, and mesh status and proxy status say so until the unit is rewritten.
• Rootless limit (for the security review): a rootless child runs as the infraternal agent's own user, and that user owns the cgroup tree. A compromised rootless Traefik could therefore write its PID into the parent's cgroup.procs and leave the leaf. Moving it requires write access to the common ancestor, which the user has. The crash case and ordinary forks are still contained. A rootful child (UID 60514, no capabilities apart from NET_BIND_SERVICE) can't write the root-owned files. This matches the design note's "rootless: accepted for now" line, and option 2 (a cgroup namespace inside a user namespace) is the fix.

Raw logs: perf VM ~/j164/rootful/. Benchmark source and binary: compute01 ~/.cache/codex-scratch/j164/.

## J164 measurements: cost of a cgroup per Traefik child The benchmark is `cgbench.rs`, a standalone file with no dependencies, built with the flake's rustc. Each iteration starts `sh -c 'sleep 1000 & exec sleep 1000'`, so it has a direct child plus a forked grandchild. - **Today:** the same fork path production already uses (std falls back from posix_spawn to fork because of the pre-exec hooks). Stop kills and reaps the direct child only. The grandchild survives, so the benchmark removes it afterwards, untimed. - **cgroup:** `mkdir` a leaf under the process's own cgroup. The pre-exec hook writes `0` to an inherited `cgroup.procs` descriptor, so the child is inside the leaf before `exec` and before it can fork. Stop is `cgroup.kill`, reap the direct child, poll `cgroup.events` until `populated 0`, then `rmdir`. ### Rootful: perf VM, `perf-run --root` (cpus 1–3), kernel 6.12.113, 500 iterations, run 2 of 2 (run 1 within ±5%) ``` today: spawn (fork path) p50=187.8us p90=223.1us p99=359.0us today: kill+reap child only p50= 41.0us p90= 60.0us p99=120.8us (grandchild left alive) cgroup: mkdir leaf p50= 16.0us p90= 22.0us p99= 30.9us cgroup: spawn joining leaf p50=199.4us p90=241.7us p99=478.0us cgroup: start total (mkdir+spawn) p50=224.6us p90=267.7us p99=501.2us cgroup: kill+reap+empty (2 procs) p50= 63.1us p90=169.5us p99=250.8us cgroup: rmdir p50= 10.8us p90= 14.1us p99= 24.8us cgroup: stop total p50= 76.9us p90=182.2us p99=261.5us steal during run: 2 of 3377 jiffies ``` - **Start:** +37 µs at p50 (mkdir 16 µs, plus about 12 µs for the `cgroup.procs` write in the child). Traefik's own start-up to a healthy ping takes hundreds of milliseconds, so this is below 0.1% of a child start. - **Stop:** +36 µs at p50, +122 µs at p90. That buys killing **both** processes and confirming the leaf is empty. Today's stop kills one process and leaves the grandchild behind. - **A restart** is a stop plus a start: about +75 µs at p50. ### Steady state (same perf VM run) - **CPU:** a fixed 400M-round integer loop takes plain 733.8–738.8 ms and in the leaf 733.0–742.4 ms, across 5 alternating pairs. That is within noise. No cgroup controller is enabled in the leaf, so the kernel charges nothing extra per tick, and the infraternal agent does no cgroup work between start and stop: no polling, no timers. - **Child RSS:** 2224–2300 KiB in both cases. - **Kernel memory per leaf:** 5.1–9.8 KiB (Slab+Percpu delta over 1000 empty leaves, 6 rounds; this VM is shared, so treat it as an upper bound). There are at most two leaves per node: the mixed process, plus the D330 rootless public process. ### Rootless: compute01, `systemd-run --user -p Delegate=yes`, 300 iterations (load average 5–17, so only the relative numbers mean anything) ``` today: spawn p50=715us today: kill+reap p50=173us cgroup: start total p50=756us cgroup: stop total p50=352us mkdir 44us rmdir 29us ``` Same shape as rootful: the deltas are mkdir plus rmdir plus one write. This is the same kernel path, so I use the perf VM numbers as the reference. ### Delegation, observed on compute01 (systemd 257) - **System manager:** a `Delegate=yes` unit's cgroup carries the `trusted.delegate=1` and `user.delegate=1` xattrs. A unit without it carries neither. Rootful detection uses this xattr. - **User manager:** with or without `Delegate=yes`, no delegate xattr is set. The whole `user@1500.service` tree is owned by `claude`, though, while a login `session-*.scope` is owned by root. Rootless therefore uses the agent's own cgroup when that cgroup is owned by the agent's user and writable, and otherwise falls back. ### What this means for the build - The leaf is created under the process's own cgroup (`proxy-<n>`). No controllers are enabled in it (`subtree_control` stays empty), so the cgroup v2 "no internal processes" rule never applies, and the infraternal agent itself stays where systemd put it. - I add `Delegate=yes` to both generated units (the server unit with its embedded infraternal agent, and the host bootstrap unit). `KillMode=process` stays. Session shims already run in their own `infraternal-session-*.scope`, so delegation doesn't touch them. - **Existing installs:** their unit has no `Delegate=yes`, so they fall back to today's pidfd reaping, and `mesh status` and `proxy status` say so until the unit is rewritten. - **Rootless limit (for the security review):** a rootless child runs as the infraternal agent's own user, and that user owns the cgroup tree. A *compromised* rootless Traefik could therefore write its PID into the parent's `cgroup.procs` and leave the leaf. Moving it requires write access to the common ancestor, which the user has. The crash case and ordinary forks are still contained. A rootful child (UID 60514, no capabilities apart from NET_BIND_SERVICE) can't write the root-owned files. This matches the design note's "rootless: accepted for now" line, and option 2 (a cgroup namespace inside a user namespace) is the fix. Raw logs: perf VM `~/j164/rootful/`. Benchmark source and binary: compute01 `~/.cache/codex-scratch/j164/`.

kayg commented 2026-11-04 04:15:10 +00:00

Author

Owner

Copy link

J164 progress. Branch proxy-cgroup-j164, head fd9a4c803, four atomic commits on base 109e14cb7:

• 8a635b94f proxy: each Traefik child runs in its own cgroup leaf. It joins in the shim's pre-exec hook. Stop is cgroup.kill, then wait for empty, then rmdir. On start the supervisor adopts its orphaned leaves; J159's reaper stays as the second line. Tests are in supervisor_cgroup_tests.rs.
• 2522d94e3 status: containment is reported in infraternal publish (the proxy listing) and in mesh status (a "Proxy" row).
• 2c80bd8f9 units: Delegate=yes on the generated server and host units. KillMode=process is unchanged.
• fd9a4c803 docs: the design note now records what was built, and corrects its rootless claim.

verify quick on dev-02 (1b1da8238, the same tree): fmt and clippy passed. Unit tests stopped in infraternal-agent on three remote_client timing tests (Elapsed, and fallback_local_admission_us=2003775) at load average 9–15. These are the same flakes J152 and J159 recorded on dev. The fast gate will tell.

Queued now: the root run on compute01 (/root/j164, red = undelegated login scope, green = Delegate=yes unit, plus the whole proxy crate), and the rootless run on dev-02 (session scope against a systemd-run --user -p Delegate=yes unit). Target dirs: dev-02 ~/worktrees/j164/target, compute01 /root/j164/target.

J164 progress. Branch `proxy-cgroup-j164`, head `fd9a4c803`, four atomic commits on base `109e14cb7`: - `8a635b94f` proxy: each Traefik child runs in its own cgroup leaf. It joins in the shim's pre-exec hook. Stop is `cgroup.kill`, then wait for empty, then rmdir. On start the supervisor adopts its orphaned leaves; J159's reaper stays as the second line. Tests are in `supervisor_cgroup_tests.rs`. - `2522d94e3` status: containment is reported in `infraternal publish` (the proxy listing) and in `mesh status` (a "Proxy" row). - `2c80bd8f9` units: `Delegate=yes` on the generated server and host units. `KillMode=process` is unchanged. - `fd9a4c803` docs: the design note now records what was built, and corrects its rootless claim. `verify quick` on dev-02 (1b1da8238, the same tree): fmt and clippy passed. Unit tests stopped in `infraternal-agent` on three `remote_client` timing tests (`Elapsed`, and `fallback_local_admission_us=2003775`) at load average 9–15. These are the same flakes J152 and J159 recorded on dev. The fast gate will tell. Queued now: the root run on compute01 (`/root/j164`, red = undelegated login scope, green = `Delegate=yes` unit, plus the whole proxy crate), and the rootless run on dev-02 (session scope against a `systemd-run --user -p Delegate=yes` unit). Target dirs: dev-02 `~/worktrees/j164/target`, compute01 `/root/j164/target`.

kayg referenced this issue 2026-11-04 04:30:24 +00:00

Machine trust: self-renewal and 7-day key rotation #285

kayg referenced this issue 2026-11-04 04:35:23 +00:00

Head-to-head: infraternal mesh vs NetBird/Tailscale, infraternal sessions vs tmux/Zellij #260

kayg commented 2026-11-04 05:04:10 +00:00

Author

Owner

Copy link

J164 real-environment proof: a double-forking Traefik leaves nothing behind

Head 31fabf153. The test is supervisor_cgroup_tests.rs. The fake Traefik setsid-forks a grandchild, /bin/sleep <unique>, which leaves the process group and session and has no parent-death signal. The test looks for that process by its unique argument, not through the leaf. Two scenarios:

• stop: the supervisor stops normally.
• crash: an infraternal agent (the re-executed test binary) is SIGKILLed, then a fresh supervisor starts on the same lock directory.

Red is today's mechanism: no delegation, so fallback, with INFRATERNAL_REQUIRE_PROXY_CGROUP=1. Green is a Delegate=yes unit.

Rootful: root on compute01, throwaway clone /root/j164, through build-slot (Debian 13, systemd 257, 6.12)

build-slot: [06:02:10] got slot 2/4
=== head 31fabf1537048b4fb6b6c25a5b0c578291a257ec, kernel 6.12.113+deb13-amd64, systemd 257 (257.13-1~deb13u1)
=== RED: today's mechanism (root in an undelegated login scope: 0::/user.slice/user-0.slice/session-c36249.scope)
red exit=101
test supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind ... stop: containment=fallback survivors=[350265] leaves=[]
thread 'supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind' (350260) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:142:9:
test supervisor_tests::cgroup_containment::an_agent_crash_leaves_no_forked_process_behind_once_the_next_agent_starts ... crash: containment=fallback survivors=[350574] leaves=[]
thread 'supervisor_tests::cgroup_containment::an_agent_crash_leaves_no_forked_process_behind_once_the_next_agent_starts' (350557) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:142:9:
test supervisor_tests::cgroup_containment::proxy_crash_helper ... ok
test result: FAILED. 1 passed; 2 failed; 0 ignored; 0 measured; 112 filtered out; finished in 9.63s
=== GREEN: root in a Delegate=yes unit
green exit=0
test supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind ... stop: containment=cgroup survivors=[] leaves=[]
test supervisor_tests::cgroup_containment::an_agent_crash_leaves_no_forked_process_behind_once_the_next_agent_starts ... crash: containment=cgroup survivors=[] leaves=[]
test supervisor_tests::cgroup_containment::proxy_crash_helper ... ok
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 112 filtered out; finished in 0.50s
=== whole proxy crate (lib + integration) as root in a Delegate=yes unit
crate exit=0
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 114 filtered out; finished in 0.00s
test result: ok. 115 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 17.90s
test result: ok. 0 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
=== leftovers
leaves: 0
processes as the proxy account: 0
account after cleanup: [] []
ROOT_RUN_DONE
--- red-orphan.txt (earlier run, test bug left it alive: rootful fallback escape)
    PID   UID ELAPSED COMMAND
 118451 60514     673 /bin/sleep 3742.668742
PPid:	1
Uid:	60514	60514	60514	60514
0::/user.slice/user-0.slice/session-c36105.scope
total 0
lr-x------ 1 infraternal-proxy infraternal-proxy 64 Nov  4 05:43 0 -> /dev/null
l-wx------ 1 infraternal-proxy infraternal-proxy 64 Nov  4 05:43 1 -> /dev/null
l-wx------ 1 infraternal-proxy infraternal-proxy 64 Nov  4 05:43 2 -> /dev/null
lrwx------ 1 infraternal-proxy infraternal-proxy 64 Nov  4 05:43 4 -> socket:[73618981]
lr-x------ 1 infraternal-proxy infraternal-proxy 64 Nov  4 05:43 64 -> /tmp/nix-shell.GuFoIS/proxy-cg-crash-88fc72f5-5a34-4660-99e0-da328ee92362/fake-traefik
lr-x------ 1 infraternal-proxy infraternal-proxy 64 Nov  4 05:43 65 -> /tmp/nix-shell.GuFoIS/proxy-cg-crash-88fc72f5-5a34-4660-99e0-da328ee92362/private/.traefik-child.lock
l-wx------ 1 infraternal-proxy infraternal-proxy 64 Nov  4 05:43 9 -> /tmp/infraternal-build-slots/slot.4

red-orphan.txt comes from an earlier run whose crash test had a bug (fixed before this head) and so never cleaned up. It shows what P3-4 looks like for real: the escaped process runs as 60514 under PID 1 and still holds the child lock (fd 65), while sitting in the login scope with no cgroup of its own. The new agent's J159 reaper refuses to kill it, because it isn't an installed Traefik. With containment it is killed on start.

Rootless: claude on dev-02 (Ubuntu 24.04, systemd 255, 6.8), systemd-run --user -p Delegate=yes against the login session scope

build-slot: [04:49:40] got slot 1/2
=== head 88c80cf40e38ff5a9ab2b9f08ec073ee157961a9, kernel 6.8.0-147-generic, systemd 255 (255.4-1ubuntu8.18)
=== RED: rootless in a login session scope (0::/user.slice/user-1500.slice/session-c25027.scope)
red exit=101
test supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind ... stop: containment=fallback survivors=[673478] leaves=[]
thread 'supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind' (673181) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:142:9:
test supervisor_tests::cgroup_containment::an_agent_crash_leaves_no_forked_process_behind_once_the_next_agent_starts ... crash: containment=fallback survivors=[674071] leaves=[]
thread 'supervisor_tests::cgroup_containment::an_agent_crash_leaves_no_forked_process_behind_once_the_next_agent_starts' (674059) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:142:9:
test supervisor_tests::cgroup_containment::proxy_crash_helper ... ok
test result: FAILED. 1 passed; 2 failed; 0 ignored; 0 measured; 112 filtered out; finished in 30.26s
=== GREEN: rootless in a user service with Delegate=yes
green exit=0
test supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind ... stop: containment=cgroup survivors=[] leaves=[]
test supervisor_tests::cgroup_containment::an_agent_crash_leaves_no_forked_process_behind_once_the_next_agent_starts ... crash: containment=cgroup survivors=[] leaves=[]
test supervisor_tests::cgroup_containment::proxy_crash_helper ... ok
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 112 filtered out; finished in 0.33s
=== whole proxy crate, rootless, in the delegated user service
crate exit=101
test cgroup::tests::only_a_pure_cgroup_v2_line_names_the_agent_cgroup ... FAILED
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 114 filtered out; finished in 0.00s
thread 'cgroup::tests::only_a_pure_cgroup_v2_line_names_the_agent_cgroup' (675418) panicked at crates/plugins/proxy/src/cgroup.rs:329:9:
test result: FAILED. 114 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 9.06s
=== mesh and shim unit tests (skipped by quick after the remote_client flakes)
mesh exit=0
test result: ok. 28 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.67s
shim exit=0
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 115 filtered out; finished in 0.00s
test result: ok. 102 passed; 0 failed; 14 ignored; 0 measured; 0 filtered out; finished in 10.03s
leaves left: 0
ROOTLESS_RUN_DONE

build-slot: [04:53:53] got slot 2/2
=== head 31fabf1537048b4fb6b6c25a5b0c578291a257ec: whole proxy crate, rootless, Delegate=yes user service
crate exit=101
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 114 filtered out; finished in 0.00s
stop: containment=cgroup survivors=[] leaves=[]
crash: containment=cgroup survivors=[] leaves=[]
test result: ok. 115 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 8.09s
test result: ok. 0 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.00s
thread 'traefik_child_receives_only_credential_file_paths_and_redacts_its_logs' (709774) panicked at crates/plugins/proxy/tests/process.rs:165:53:
test traefik_child_receives_only_credential_file_paths_and_redacts_its_logs ... FAILED
test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
leaves left: 0
CRATE_DONE

Notes:

• The first rootless crate run caught a real bug of mine: an infraternal agent in the root cgroup (0::/, an init without systemd) was refused and fell back. It is fixed in a15f9c43c, and the rerun above passes 115/115.
• tests/process.rs::traefik_child_receives_only_credential_file_paths_and_redacts_its_logs failed once on dev-02 under load. It's a pre-existing race: the shell creates the capture file with > before printf writes it, and the test reads it as soon as it exists. That path spawns without a cgroup. Rerunning the same binary 10× gave pass=10 fail=0, and as root it passed (2 passed).
• VM gotcha, for the brief: on both VMs, nix develop … | grep can hang after the command exits. nix's detached crashpad_handler keeps the pipe's write end open. Redirect to a file instead.

Cleanup: /root/j164 src, target and state removed (logs kept). No proxy-* leaves remain on either VM, no processes run as the proxy account, and infraternal-proxy was removed ([] []).

## J164 real-environment proof: a double-forking Traefik leaves nothing behind Head `31fabf153`. The test is `supervisor_cgroup_tests.rs`. The fake Traefik `setsid`-forks a grandchild, `/bin/sleep <unique>`, which leaves the process group and session and has no parent-death signal. The test looks for that process by its unique argument, not through the leaf. Two scenarios: - **stop:** the supervisor stops normally. - **crash:** an infraternal agent (the re-executed test binary) is SIGKILLed, then a fresh supervisor starts on the same lock directory. **Red** is today's mechanism: no delegation, so fallback, with `INFRATERNAL_REQUIRE_PROXY_CGROUP=1`. **Green** is a `Delegate=yes` unit. ### Rootful: root on compute01, throwaway clone `/root/j164`, through build-slot (Debian 13, systemd 257, 6.12) ``` build-slot: [06:02:10] got slot 2/4 === head 31fabf1537048b4fb6b6c25a5b0c578291a257ec, kernel 6.12.113+deb13-amd64, systemd 257 (257.13-1~deb13u1) === RED: today's mechanism (root in an undelegated login scope: 0::/user.slice/user-0.slice/session-c36249.scope) red exit=101 test supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind ... stop: containment=fallback survivors=[350265] leaves=[] thread 'supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind' (350260) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:142:9: test supervisor_tests::cgroup_containment::an_agent_crash_leaves_no_forked_process_behind_once_the_next_agent_starts ... crash: containment=fallback survivors=[350574] leaves=[] thread 'supervisor_tests::cgroup_containment::an_agent_crash_leaves_no_forked_process_behind_once_the_next_agent_starts' (350557) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:142:9: test supervisor_tests::cgroup_containment::proxy_crash_helper ... ok test result: FAILED. 1 passed; 2 failed; 0 ignored; 0 measured; 112 filtered out; finished in 9.63s === GREEN: root in a Delegate=yes unit green exit=0 test supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind ... stop: containment=cgroup survivors=[] leaves=[] test supervisor_tests::cgroup_containment::an_agent_crash_leaves_no_forked_process_behind_once_the_next_agent_starts ... crash: containment=cgroup survivors=[] leaves=[] test supervisor_tests::cgroup_containment::proxy_crash_helper ... ok test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 112 filtered out; finished in 0.50s === whole proxy crate (lib + integration) as root in a Delegate=yes unit crate exit=0 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 114 filtered out; finished in 0.00s test result: ok. 115 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 17.90s test result: ok. 0 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s === leftovers leaves: 0 processes as the proxy account: 0 account after cleanup: [] [] ROOT_RUN_DONE --- red-orphan.txt (earlier run, test bug left it alive: rootful fallback escape) PID UID ELAPSED COMMAND 118451 60514 673 /bin/sleep 3742.668742 PPid: 1 Uid: 60514 60514 60514 60514 0::/user.slice/user-0.slice/session-c36105.scope total 0 lr-x------ 1 infraternal-proxy infraternal-proxy 64 Nov 4 05:43 0 -> /dev/null l-wx------ 1 infraternal-proxy infraternal-proxy 64 Nov 4 05:43 1 -> /dev/null l-wx------ 1 infraternal-proxy infraternal-proxy 64 Nov 4 05:43 2 -> /dev/null lrwx------ 1 infraternal-proxy infraternal-proxy 64 Nov 4 05:43 4 -> socket:[73618981] lr-x------ 1 infraternal-proxy infraternal-proxy 64 Nov 4 05:43 64 -> /tmp/nix-shell.GuFoIS/proxy-cg-crash-88fc72f5-5a34-4660-99e0-da328ee92362/fake-traefik lr-x------ 1 infraternal-proxy infraternal-proxy 64 Nov 4 05:43 65 -> /tmp/nix-shell.GuFoIS/proxy-cg-crash-88fc72f5-5a34-4660-99e0-da328ee92362/private/.traefik-child.lock l-wx------ 1 infraternal-proxy infraternal-proxy 64 Nov 4 05:43 9 -> /tmp/infraternal-build-slots/slot.4 ``` `red-orphan.txt` comes from an earlier run whose crash test had a bug (fixed before this head) and so never cleaned up. It shows what P3-4 looks like for real: the escaped process runs as 60514 under PID 1 and still holds the child lock (fd 65), while sitting in the login scope with no cgroup of its own. The new agent's J159 reaper refuses to kill it, because it isn't an installed Traefik. With containment it is killed on start. ### Rootless: `claude` on dev-02 (Ubuntu 24.04, systemd 255, 6.8), `systemd-run --user -p Delegate=yes` against the login session scope ``` build-slot: [04:49:40] got slot 1/2 === head 88c80cf40e38ff5a9ab2b9f08ec073ee157961a9, kernel 6.8.0-147-generic, systemd 255 (255.4-1ubuntu8.18) === RED: rootless in a login session scope (0::/user.slice/user-1500.slice/session-c25027.scope) red exit=101 test supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind ... stop: containment=fallback survivors=[673478] leaves=[] thread 'supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind' (673181) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:142:9: test supervisor_tests::cgroup_containment::an_agent_crash_leaves_no_forked_process_behind_once_the_next_agent_starts ... crash: containment=fallback survivors=[674071] leaves=[] thread 'supervisor_tests::cgroup_containment::an_agent_crash_leaves_no_forked_process_behind_once_the_next_agent_starts' (674059) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:142:9: test supervisor_tests::cgroup_containment::proxy_crash_helper ... ok test result: FAILED. 1 passed; 2 failed; 0 ignored; 0 measured; 112 filtered out; finished in 30.26s === GREEN: rootless in a user service with Delegate=yes green exit=0 test supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind ... stop: containment=cgroup survivors=[] leaves=[] test supervisor_tests::cgroup_containment::an_agent_crash_leaves_no_forked_process_behind_once_the_next_agent_starts ... crash: containment=cgroup survivors=[] leaves=[] test supervisor_tests::cgroup_containment::proxy_crash_helper ... ok test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 112 filtered out; finished in 0.33s === whole proxy crate, rootless, in the delegated user service crate exit=101 test cgroup::tests::only_a_pure_cgroup_v2_line_names_the_agent_cgroup ... FAILED test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 114 filtered out; finished in 0.00s thread 'cgroup::tests::only_a_pure_cgroup_v2_line_names_the_agent_cgroup' (675418) panicked at crates/plugins/proxy/src/cgroup.rs:329:9: test result: FAILED. 114 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 9.06s === mesh and shim unit tests (skipped by quick after the remote_client flakes) mesh exit=0 test result: ok. 28 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.67s shim exit=0 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 115 filtered out; finished in 0.00s test result: ok. 102 passed; 0 failed; 14 ignored; 0 measured; 0 filtered out; finished in 10.03s leaves left: 0 ROOTLESS_RUN_DONE build-slot: [04:53:53] got slot 2/2 === head 31fabf1537048b4fb6b6c25a5b0c578291a257ec: whole proxy crate, rootless, Delegate=yes user service crate exit=101 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 114 filtered out; finished in 0.00s stop: containment=cgroup survivors=[] leaves=[] crash: containment=cgroup survivors=[] leaves=[] test result: ok. 115 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 8.09s test result: ok. 0 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.00s thread 'traefik_child_receives_only_credential_file_paths_and_redacts_its_logs' (709774) panicked at crates/plugins/proxy/tests/process.rs:165:53: test traefik_child_receives_only_credential_file_paths_and_redacts_its_logs ... FAILED test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s leaves left: 0 CRATE_DONE ``` Notes: - The first rootless crate run caught a real bug of mine: an infraternal agent in the root cgroup (`0::/`, an init without systemd) was refused and fell back. It is fixed in `a15f9c43c`, and the rerun above passes 115/115. - `tests/process.rs::traefik_child_receives_only_credential_file_paths_and_redacts_its_logs` failed once on dev-02 under load. It's a pre-existing race: the shell creates the capture file with `>` before `printf` writes it, and the test reads it as soon as it exists. That path spawns without a cgroup. Rerunning the same binary 10× gave `pass=10 fail=0`, and as root it passed (2 passed). - VM gotcha, for the brief: on both VMs, `nix develop … | grep` can hang after the command exits. nix's detached `crashpad_handler` keeps the pipe's write end open. Redirect to a file instead. Cleanup: `/root/j164` src, target and state removed (logs kept). No `proxy-*` leaves remain on either VM, no processes run as the proxy account, and `infraternal-proxy` was removed (`[] []`).

kayg commented 2026-11-04 05:55:17 +00:00

Author

Owner

Copy link

J164 gate: scripts/verify fast on compute01, head 31fabf1537048b4fb6b6c25a5b0c578291a257ec

Not green. Every row passes except test. That row stopped in infraternal-agent --lib on two remote_client wall-clock tests: unavailable_udp_does_not_add_seconds_before_tcp_admission and winning_quic_drops_the_unfinished_tls_dial_before_control_dispatch. They are the same tests J152 and J159 recorded as load flakes, and the load average was 25–40 during this run. This branch touches nothing in remote_client. Its only agent change is the probe wiring in plugins.rs. A base-against-head run of those tests on this machine is queued, and I'll post it.

The tests the test row didn't reach after that stop were run separately, and all passed:

• proxy crate as root in a Delegate=yes unit: lib 115/115 and integration 2/2;
• proxy crate rootless in a delegated user unit: lib 115/115;
• mesh lib: 28/28;
• shim lib: 102 passed, 14 ignored.

Log: compute01 ~/j164-fast.log. Excerpt, verbatim:

failures:

---- remote_client::tests::unavailable_udp_does_not_add_seconds_before_tcp_admission stdout ----
fallback_local_admission_us=2032405

thread 'remote_client::tests::unavailable_udp_does_not_add_seconds_before_tcp_admission' (519583) panicked at crates/infraternal-agent/src/remote_client_tests.rs:167:5:
UDP unavailability delayed TCP establishment by seconds
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- remote_client::tests::winning_quic_drops_the_unfinished_tls_dial_before_control_dispatch stdout ----

thread 'remote_client::tests::winning_quic_drops_the_unfinished_tls_dial_before_control_dispatch' (519608) panicked at crates/infraternal-agent/src/remote_client_tests.rs:205:10:
called `Result::unwrap()` on an `Err` value: Elapsed(())


failures:
    remote_client::tests::unavailable_udp_does_not_add_seconds_before_tcp_admission
    remote_client::tests::winning_quic_drops_the_unfinished_tls_dial_before_control_dispatch

test result: FAILED. 224 passed; 2 failed; 22 ignored; 0 measured; 0 filtered out; finished in 33.31s

error: test failed, to rerun pass `-p infraternal-agent --lib`
...
/home/claude/worktrees/j164/target/x86_64-unknown-linux-musl/release/infraternal: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, stripped
PT_INTERP: absent; size: 23174800 bytes; baseline 22418256; delta +756544 bytes (report only, D243 amended 2026-11-01)
/home/claude/worktrees/j164/target/aarch64-unknown-linux-musl/release/infraternal: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, stripped
PT_INTERP: absent; size: 21086464 bytes; baseline 20453280; delta +633184 bytes (report only, D243 amended 2026-11-01)
SKIP key memory: aarch64-unknown-linux-musl (host x86_64; native execution required)
x86_64-unknown-linux-musl: +102400 bytes accepted with delta; zero/negative deltas reported; missing/wrong/dynamic/PT_INTERP/inspection defects rejected PASS
aarch64-unknown-linux-musl: +102400 bytes accepted with delta; zero/negative deltas reported; missing/wrong/dynamic/PT_INTERP/inspection defects rejected PASS
unsupported target: rejected PASS
bun test v1.4.2 (744846f84)

scripts/release/static-metadata-review.test.ts:
(pass) x86_64-unknown-linux-musl compiler metadata is independent of the checkout location [37.21ms]
(pass) aarch64-unknown-linux-musl compiler metadata is independent of the checkout location [31.20ms]

 2 pass
 0 fail
 14 expect() calls
Ran 2 tests across 1 file. [2.28s]
server subcommand: exit 2, versioned JSON unavailable result
agent group: exit 0, one parsed versioned JSON listing of its commands
(static builds: 1629s)

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | FAIL (101)   |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
VERIFY_EXIT=1

Size: the x86 and aarch64 rows report against the gate's stale baseline (+756544 B and +633184 B), so they are not this slice's delta. The same queued job builds the base 109e14cb7 x86 static binary to give the real delta.

## J164 gate: `scripts/verify fast` on compute01, head `31fabf1537048b4fb6b6c25a5b0c578291a257ec` **Not green.** Every row passes except `test`. That row stopped in `infraternal-agent --lib` on two `remote_client` wall-clock tests: `unavailable_udp_does_not_add_seconds_before_tcp_admission` and `winning_quic_drops_the_unfinished_tls_dial_before_control_dispatch`. They are the same tests J152 and J159 recorded as load flakes, and the load average was 25–40 during this run. This branch touches nothing in `remote_client`. Its only agent change is the probe wiring in `plugins.rs`. A base-against-head run of those tests on this machine is queued, and I'll post it. The tests the `test` row didn't reach after that stop were run separately, and all passed: - proxy crate as root in a `Delegate=yes` unit: lib 115/115 and integration 2/2; - proxy crate rootless in a delegated user unit: lib 115/115; - mesh lib: 28/28; - shim lib: 102 passed, 14 ignored. Log: compute01 `~/j164-fast.log`. Excerpt, verbatim: ``` failures: ---- remote_client::tests::unavailable_udp_does_not_add_seconds_before_tcp_admission stdout ---- fallback_local_admission_us=2032405 thread 'remote_client::tests::unavailable_udp_does_not_add_seconds_before_tcp_admission' (519583) panicked at crates/infraternal-agent/src/remote_client_tests.rs:167:5: UDP unavailability delayed TCP establishment by seconds note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace ---- remote_client::tests::winning_quic_drops_the_unfinished_tls_dial_before_control_dispatch stdout ---- thread 'remote_client::tests::winning_quic_drops_the_unfinished_tls_dial_before_control_dispatch' (519608) panicked at crates/infraternal-agent/src/remote_client_tests.rs:205:10: called `Result::unwrap()` on an `Err` value: Elapsed(()) failures: remote_client::tests::unavailable_udp_does_not_add_seconds_before_tcp_admission remote_client::tests::winning_quic_drops_the_unfinished_tls_dial_before_control_dispatch test result: FAILED. 224 passed; 2 failed; 22 ignored; 0 measured; 0 filtered out; finished in 33.31s error: test failed, to rerun pass `-p infraternal-agent --lib` ... /home/claude/worktrees/j164/target/x86_64-unknown-linux-musl/release/infraternal: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, stripped PT_INTERP: absent; size: 23174800 bytes; baseline 22418256; delta +756544 bytes (report only, D243 amended 2026-11-01) /home/claude/worktrees/j164/target/aarch64-unknown-linux-musl/release/infraternal: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, stripped PT_INTERP: absent; size: 21086464 bytes; baseline 20453280; delta +633184 bytes (report only, D243 amended 2026-11-01) SKIP key memory: aarch64-unknown-linux-musl (host x86_64; native execution required) x86_64-unknown-linux-musl: +102400 bytes accepted with delta; zero/negative deltas reported; missing/wrong/dynamic/PT_INTERP/inspection defects rejected PASS aarch64-unknown-linux-musl: +102400 bytes accepted with delta; zero/negative deltas reported; missing/wrong/dynamic/PT_INTERP/inspection defects rejected PASS unsupported target: rejected PASS bun test v1.4.2 (744846f84) scripts/release/static-metadata-review.test.ts: (pass) x86_64-unknown-linux-musl compiler metadata is independent of the checkout location [37.21ms] (pass) aarch64-unknown-linux-musl compiler metadata is independent of the checkout location [31.20ms] 2 pass 0 fail 14 expect() calls Ran 2 tests across 1 file. [2.28s] server subcommand: exit 2, versioned JSON unavailable result agent group: exit 0, one parsed versioned JSON listing of its commands (static builds: 1629s) +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | FAIL (101) | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | static builds | PASS | +---------------------------+--------------+ VERIFY_EXIT=1 ``` **Size:** the x86 and aarch64 rows report against the gate's stale baseline (+756544 B and +633184 B), so they are not this slice's delta. The same queued job builds the base `109e14cb7` x86 static binary to give the real delta.

kayg commented 2026-11-04 07:01:42 +00:00

Author

Owner

Copy link

J164 finished: a cgroup per Traefik child (D336). Not merged, not deployed; ready for the security review.

Branch proxy-cgroup-j164, head 31fabf1537048b4fb6b6c25a5b0c578291a257ec, base 109e14cb7:

• a15f9c43c proxy: run each Traefik child in its own cgroup (shim pre-exec join, cgroup.kill stop, adoption on start, tests)
• 1820bb15d proxy, mesh: say how Traefik is contained in status
• da5685db4 units: Delegate=yes on the generated server and host units
• 31fabf153 docs(proxy): record option 1 as built, and the rootless limit

Measurements: comment 77343. Root and rootless proof: comment 77394. Gate, verbatim: comment 77408. Every row passes except test, which stopped on two remote_client wall-clock tests.

Are the remote_client failures ours? Base against head, compute01, the same cargo test --workspace binaries, filtered to remote_client::tests, interleaved

build-slot: [07:44:28] got slot 3/4
head x86 static size (from the gate): 23174800
head bins: /home/claude/worktrees/j164/target/debug/deps/infraternal_agent-827173b4de74dd4c
base bins: /home/claude/worktrees/j164/target/debug/deps/infraternal_agent-0923f786da3dfe27
--- round 1
head infraternal_agent-827173b4de74dd4c exit=0 load=15.35 test result: ok. 19 passed; 0 failed; 6 ignored; 0 measured; 223 filtered out; finished in 0.70s failed=[]
base infraternal_agent-0923f786da3dfe27 exit=0 load=15.35 test result: ok. 19 passed; 0 failed; 6 ignored; 0 measured; 223 filtered out; finished in 0.66s failed=[]
--- round 2
head infraternal_agent-827173b4de74dd4c exit=0 load=14.92 test result: ok. 19 passed; 0 failed; 6 ignored; 0 measured; 223 filtered out; finished in 0.66s failed=[]
base infraternal_agent-0923f786da3dfe27 exit=0 load=14.92 test result: ok. 19 passed; 0 failed; 6 ignored; 0 measured; 223 filtered out; finished in 0.66s failed=[]
--- round 3
head infraternal_agent-827173b4de74dd4c exit=0 load=14.92 test result: ok. 19 passed; 0 failed; 6 ignored; 0 measured; 223 filtered out; finished in 0.67s failed=[]
base infraternal_agent-0923f786da3dfe27 exit=0 load=14.92 test result: ok. 19 passed; 0 failed; 6 ignored; 0 measured; 223 filtered out; finished in 0.65s failed=[]
--- base x86 static build
base static exit=0
x86_64 static: base 23155568, head 23174800, delta 19232 bytes
BASE_AB_DONE

• Alone, both pass 3/3 on head and on base.
• I did not reproduce the full-lib failure on base. It needs the whole infraternal-agent lib under heavy load, which is J152's order- and load-dependent flake.
• This branch adds and changes no infraternal-agent tests. Its only agent change is the probe closure in plugins.rs, so the lib's test set and order are the same as base.
• The orchestrator can accept this as the known flake or rerun fast on a quieter VM.

Size

x86_64 static: base 23,155,568 B, head 23,174,800 B, +19,232 B for this slice. The gate's +756544 is measured against its stale baseline. xattr was already linked through tar.

For the security review

• Rootless limit: the cgroup tree belongs to the infraternal agent's user, so a compromised rootless child can write itself into the parent cgroup. Ordinary forks and crashes are contained. A rootful child can't do this (root-owned leaf, UID 60514, no CAP_SYS_ADMIN).
• Fallback triggers: an install whose unit lacks Delegate=yes, every existing install until its unit is rewritten, a rootless agent outside a user service, a hybrid cgroup v1 system, and a kernel older than 5.14. Status then says "limited: ; a process Traefik forks may outlive it".
• Pre-existing, not changed here: the Traefik child inherits any non-CLOEXEC descriptor the agent itself inherited. In the tests, build-slot's fd 9 reached the orphan and kept the slot busy. In production the agent's own descriptors are CLOEXEC, but a unit or launcher could pass others.

Cleanup: compute01 and dev-02 worktrees and targets removed. /root/j164 holds logs only. No proxy leaves or test processes are left, and the infraternal-proxy account was removed.

## J164 finished: a cgroup per Traefik child (D336). Not merged, not deployed; ready for the security review. Branch `proxy-cgroup-j164`, head `31fabf1537048b4fb6b6c25a5b0c578291a257ec`, base `109e14cb7`: - `a15f9c43c` proxy: run each Traefik child in its own cgroup (shim pre-exec join, `cgroup.kill` stop, adoption on start, tests) - `1820bb15d` proxy, mesh: say how Traefik is contained in status - `da5685db4` units: `Delegate=yes` on the generated server and host units - `31fabf153` docs(proxy): record option 1 as built, and the rootless limit **Measurements:** comment 77343. **Root and rootless proof:** comment 77394. **Gate, verbatim:** comment 77408. Every row passes except `test`, which stopped on two `remote_client` wall-clock tests. ### Are the `remote_client` failures ours? Base against head, compute01, the same `cargo test --workspace` binaries, filtered to `remote_client::tests`, interleaved ``` build-slot: [07:44:28] got slot 3/4 head x86 static size (from the gate): 23174800 head bins: /home/claude/worktrees/j164/target/debug/deps/infraternal_agent-827173b4de74dd4c base bins: /home/claude/worktrees/j164/target/debug/deps/infraternal_agent-0923f786da3dfe27 --- round 1 head infraternal_agent-827173b4de74dd4c exit=0 load=15.35 test result: ok. 19 passed; 0 failed; 6 ignored; 0 measured; 223 filtered out; finished in 0.70s failed=[] base infraternal_agent-0923f786da3dfe27 exit=0 load=15.35 test result: ok. 19 passed; 0 failed; 6 ignored; 0 measured; 223 filtered out; finished in 0.66s failed=[] --- round 2 head infraternal_agent-827173b4de74dd4c exit=0 load=14.92 test result: ok. 19 passed; 0 failed; 6 ignored; 0 measured; 223 filtered out; finished in 0.66s failed=[] base infraternal_agent-0923f786da3dfe27 exit=0 load=14.92 test result: ok. 19 passed; 0 failed; 6 ignored; 0 measured; 223 filtered out; finished in 0.66s failed=[] --- round 3 head infraternal_agent-827173b4de74dd4c exit=0 load=14.92 test result: ok. 19 passed; 0 failed; 6 ignored; 0 measured; 223 filtered out; finished in 0.67s failed=[] base infraternal_agent-0923f786da3dfe27 exit=0 load=14.92 test result: ok. 19 passed; 0 failed; 6 ignored; 0 measured; 223 filtered out; finished in 0.65s failed=[] --- base x86 static build base static exit=0 x86_64 static: base 23155568, head 23174800, delta 19232 bytes BASE_AB_DONE ``` - Alone, both pass 3/3 on head and on base. - I did **not** reproduce the full-lib failure on base. It needs the whole `infraternal-agent` lib under heavy load, which is J152's order- and load-dependent flake. - This branch adds and changes no `infraternal-agent` tests. Its only agent change is the probe closure in `plugins.rs`, so the lib's test set and order are the same as base. - The orchestrator can accept this as the known flake or rerun `fast` on a quieter VM. ### Size x86_64 static: base 23,155,568 B, head 23,174,800 B, **+19,232 B** for this slice. The gate's `+756544` is measured against its stale baseline. `xattr` was already linked through `tar`. ### For the security review - **Rootless limit:** the cgroup tree belongs to the infraternal agent's user, so a *compromised* rootless child can write itself into the parent cgroup. Ordinary forks and crashes are contained. A rootful child can't do this (root-owned leaf, UID 60514, no `CAP_SYS_ADMIN`). - **Fallback triggers:** an install whose unit lacks `Delegate=yes`, every existing install until its unit is rewritten, a rootless agent outside a user service, a hybrid cgroup v1 system, and a kernel older than 5.14. Status then says "limited: <reason>; a process Traefik forks may outlive it". - **Pre-existing, not changed here:** the Traefik child inherits any non-CLOEXEC descriptor the agent itself inherited. In the tests, build-slot's fd 9 reached the orphan and kept the slot busy. In production the agent's own descriptors are CLOEXEC, but a unit or launcher could pass others. Cleanup: compute01 and dev-02 worktrees and targets removed. `/root/j164` holds logs only. No proxy leaves or test processes are left, and the `infraternal-proxy` account was removed.

kayg referenced this issue 2026-11-04 07:02:16 +00:00

Add-host: a failed install leaves a bootstrap lock that blocks every re-add forever #305

kayg commented 2026-11-04 07:10:45 +00:00

Author

Owner

Copy link

J172 start: security review of proxy-cgroup-j164

Reviewing head 31fabf1537048b4fb6b6c25a5b0c578291a257ec against base 109e14cb7 (D336, J164 comments 77343/77394/77408). Read-only: the branch is not changed.

Plan (about 2 h, most of it build-slot queue):

1. Read cgroup.rs, the supervisor/shim diff, the units and the status wiring. Done.
2. Shell experiments as root on compute01 (Delegate=yes + KillMode=process lifecycle, xattrs, root cgroup files) and rootless on dev-02 (escape reach). Done, findings follow in the verdict.
3. Review-only probe tests appended to a throwaway copy of supervisor_cgroup_tests.rs (never pushed): a child that writes itself into the parent cgroup, a child that makes a nested cgroup, a dump of the child's descriptors and cgroup. Root: /root/j172 on compute01 (queued, position 9). Rootless: ~/worktrees/j172 on dev-02 (queued, position 3).
4. Verdict with P1/P2/P3 findings and evidence.

## J172 start: security review of `proxy-cgroup-j164` Reviewing head `31fabf1537048b4fb6b6c25a5b0c578291a257ec` against base `109e14cb7` (D336, J164 comments 77343/77394/77408). Read-only: the branch is not changed. Plan (about 2 h, most of it build-slot queue): 1. Read `cgroup.rs`, the supervisor/shim diff, the units and the status wiring. Done. 2. Shell experiments as root on compute01 (`Delegate=yes` + `KillMode=process` lifecycle, xattrs, root cgroup files) and rootless on dev-02 (escape reach). Done, findings follow in the verdict. 3. Review-only probe tests appended to a throwaway copy of `supervisor_cgroup_tests.rs` (never pushed): a child that writes itself into the parent cgroup, a child that makes a nested cgroup, a dump of the child's descriptors and cgroup. Root: `/root/j172` on compute01 (queued, position 9). Rootless: `~/worktrees/j172` on dev-02 (queued, position 3). 4. Verdict with P1/P2/P3 findings and evidence.

kayg commented 2026-11-04 07:52:54 +00:00

Author

Owner

Copy link

J172 verdict on proxy-cgroup-j164 @ 31fabf1537048b4fb6b6c25a5b0c578291a257ec: REQUEST CHANGES

No P0 or P1. Rootful containment holds. The reaping cannot reach outside the infraternal agent's own subtree, and the join happens before exec with no window. There is one P2: a rootless child that leaves its leaf makes stop hang forever. Five P3s follow.

Method. I read the diff. Then I appended review-only probe tests to a throwaway copy of supervisor_cgroup_tests.rs; they were never pushed. Each probe uses a fake Traefik that records its uid, cgroup and fds, records the cgroup of a forked grandchild, then (by mode) tries to move itself into the parent cgroup (escape) or mkdirs a nested cgroup inside its leaf (nested). It then setsid-forks and execs sleep. Root ran on compute01 in /root/j172 (built through build-slot, removed afterwards). Rootless ran as claude on dev-02 in a systemd-run --user -p Delegate=yes unit. Logs: compute01 /root/j172/root-run.log, dev-02 ~/j172/user-run.log. No leftovers on either VM (0 marker processes, 0 proxy-* leaves). I left the existing infraternal-proxy account on compute01 in place.

What holds (with evidence)

• Join before exec: no window. The child and the grandchild it forks are both in the leaf, on both hosts:
  uid=60514 cg=/system.slice/run-p2265822-i52592164.service/proxy-2d80a5-0 / grandchild cg=…/proxy-2d80a5-0. The write happens first in the pre-exec hook, so nothing runs as Traefik before it. A failed write fails the spawn, and the leaf is removed.
• Inherited descriptors. The child's fds are 0/1/2, the exec fd 64, the child lock 65, plus the shell's own 10/11. No cgroup.procs fd reaches the child (it is O_CLOEXEC), and the write runs before dup2, so a remap onto its number can't matter. Same on both hosts.
• Rootful escape is impossible. escape-exit=2 … cannot create …/cgroup.procs: Permission denied, and mkdir … nested: Permission denied. The stop then empties and removes the leaf (grandchildren=[] direct-child=[] leaves=[]), and the next start is fine.
• Reaping stays in its own subtree. cgroupfs has no symlinks, and cgroup v2 refuses rename. own_cgroup/valid_recorded_base reject .., . and empty segments, and anything outside /sys/fs/cgroup. Rootful, the record lives in the root-only lock directory. Rootless, the child can rewrite the record (same uid), but then the kernel limits cgroup.kill to that user's own tree.
• Rootless escape is limited to the user's own tree (dev-02, shell): moving to the parent: ok; to root-owned user.slice: refused; to user@1500.service (internal node): refused.
• Delegate=yes + KillMode=process (root, systemd 257, shell). The unit gets trusted.delegate=1 and user.delegate=1. subtree_control stays empty, so systemd enables no controllers. systemctl stop kills only the main process: the unit reads inactive while the leaf and its members live on, and the next start reuses the cgroup, where adopt_orphans kills them. This matches today's KillMode=process behaviour; the forked grandchild escaped before this branch too. Existing installs have no xattr, so they fall back, which the status reports.
• No systemd, ordinary cgroup: root in /j172-nosd with /run/systemd hidden gets kind=cgroup, and a stop leaves nothing.

P2-1. A rootless child that leaves its leaf hangs stop() forever, and status still says "contained"

Evidence (dev-02, mode=escape):

escape-exit=0 now=/user.slice/user-1500.slice/user@1500.service/app.slice/run-u5344.service
J172 stop returned=false after 15.002697527s
J172 after stop: grandchildren=[1374931] direct-child=[1374905] leaves=[".../run-u5344.service/proxy-e46f2-0"]

TraefikChild::stop skips child.kill() whenever the cgroup.kill write succeeds, then blocks in child.wait(). A direct child that is no longer in the leaf never gets SIGKILL. So the supervisor worker thread hangs, and so does TraefikSupervisor::drop (it joins the worker), which wedges proxy reconfiguration and infraternal agent shutdown. The escaped process keeps its listeners. This needs a compromised rootless Traefik (accepted in the design note), but the result should be "the escapee runs, and stop still finishes", not a wedged infraternal agent.
Fix: always kill() the direct child as well (it is pidfd-safe: std still holds the unreaped child), whether or not cgroup.kill succeeded. Optionally, before waiting, check that /proc/<pid>/cgroup still names the leaf, and if not, log that the child left its cgroup.

P3-1. A nested cgroup made by a rootless child blocks every later start (EBUSY)

Evidence (dev-02, mode=nested):

mkdir-nested-exit=0
J172 stop returned=true after 474.834µs        (Drop swallows the remove error)
J172 after stop: grandchildren=[] direct-child=[] leaves=[".../proxy-120b98-0"]
J172 next agent start: Err(Os { code: 16, kind: ResourceBusy, message: "Device or resource busy" })

rmdir of a leaf with an empty child cgroup fails with EBUSY, and adopt_orphans propagates that through ?, so the proxy never starts again until someone removes the cgroup by hand. Only a compromised rootless child can do this (rootful mkdir is denied, see above).
Fix: in kill_and_remove, after populated 0, remove the child cgroups depth-first (this is safe: cgroup.kill already covered the whole subtree, and populated reflects the whole subtree). Then remove the leaf.

P3-2. Rootless status overstates containment

Containment::Cgroup.describe() gives the same text rootful and rootless: "each Traefik process runs in its own cgroup; stopping it stops everything it started". Rootless, P2-1 shows that isn't true for a compromised child. The design note and comment 77343 are honest about this, but mesh status/publish are not.
Fix: add a rootless variant (e.g. kind: "cgroup-rootless"), with text like "…; a compromised Traefik running as your user could still leave it".

P3-3. Root in the root cgroup (no systemd) gets a false reason

Evidence (compute01, mount ns without /run/systemd, agent moved to /):
J172 own=0::/ kind=fallback detail=limited: the kernel is older than Linux 5.14 (no cgroup.kill) on kernel 6.12.
The root cgroup never has cgroup.kill (or cgroup.events), on any kernel. Falling back is safe, but the stated reason is wrong.
Fix: say "the infraternal agent runs in the root cgroup". Or support it: leaves under / are legal, and the root cgroup is exempt from the no-internal-processes rule. Detect cgroup.kill support by checking a created leaf, or /sys/kernel/cgroup/features, rather than the base.
Kernels before 5.14 (no cgroup.kill anywhere): I found that by reading the code, not by running it (no such kernel was available). They take the same check and fall back with a correct message.

P3-4. Leaves that outlive a unit losing Delegate=yes are never reaped

Evidence (root, shell): a Delegate=yes + KillMode=process unit left a populated proxy-* leaf on stop. After Delegate=yes was removed and the unit restarted, the xattr was absent 61, the leaf was still there, and all its members were alive. detect then falls back, and adopt_orphans never runs, so those processes live for as long as the unit's cgroup does. This happens on a downgrade, or when a hand-written or distro unit replaces a generated one.
Fix: in fallback, when the agent's own cgroup is writable (root, or the owner), still run adopt_orphans on its own cgroup and the recorded base. Killing your own tagged leaves needs no delegation.

P3-5. Leaf tag uses the inode without the device

tag = ino(lock_directory). Two infraternal agents in the same cgroup, with lock directories on different filesystems that share an inode number, would reap each other's live leaves at start. This is unlikely (one agent per unit), but cheap to rule out.
Fix: proxy-<dev:x>-<ino:x>-<n>.

Also noted, not a finding: systemd versions that set no trusted.delegate/user.delegate xattr would report "lacks Delegate=yes" even when the unit has it. That is safe, but the message is misleading. I didn't test it (both VMs run systemd 255/257).

Gate: no code changed in this review, so the gate is not applicable. The branch's own gate is in 77408.

## J172 verdict on `proxy-cgroup-j164` @ `31fabf1537048b4fb6b6c25a5b0c578291a257ec`: **REQUEST CHANGES** No P0 or P1. Rootful containment holds. The reaping cannot reach outside the infraternal agent's own subtree, and the join happens before exec with no window. There is one P2: a rootless child that leaves its leaf makes stop hang forever. Five P3s follow. **Method.** I read the diff. Then I appended review-only probe tests to a throwaway copy of `supervisor_cgroup_tests.rs`; they were never pushed. Each probe uses a fake Traefik that records its uid, cgroup and fds, records the cgroup of a forked grandchild, then (by mode) tries to move itself into the parent cgroup (`escape`) or `mkdir`s a nested cgroup inside its leaf (`nested`). It then setsid-forks and execs `sleep`. Root ran on compute01 in `/root/j172` (built through build-slot, removed afterwards). Rootless ran as `claude` on dev-02 in a `systemd-run --user -p Delegate=yes` unit. Logs: compute01 `/root/j172/root-run.log`, dev-02 `~/j172/user-run.log`. No leftovers on either VM (0 marker processes, 0 `proxy-*` leaves). I left the existing `infraternal-proxy` account on compute01 in place. ### What holds (with evidence) - **Join before exec: no window.** The child and the grandchild it forks are both in the leaf, on both hosts: `uid=60514 cg=/system.slice/run-p2265822-i52592164.service/proxy-2d80a5-0` / `grandchild cg=…/proxy-2d80a5-0`. The write happens first in the pre-exec hook, so nothing runs as Traefik before it. A failed write fails the spawn, and the leaf is removed. - **Inherited descriptors.** The child's fds are 0/1/2, the exec fd 64, the child lock 65, plus the shell's own 10/11. No `cgroup.procs` fd reaches the child (it is `O_CLOEXEC`), and the write runs before `dup2`, so a remap onto its number can't matter. Same on both hosts. - **Rootful escape is impossible.** `escape-exit=2 … cannot create …/cgroup.procs: Permission denied`, and `mkdir … nested: Permission denied`. The stop then empties and removes the leaf (`grandchildren=[] direct-child=[] leaves=[]`), and the next start is fine. - **Reaping stays in its own subtree.** cgroupfs has no symlinks, and cgroup v2 refuses rename. `own_cgroup`/`valid_recorded_base` reject `..`, `.` and empty segments, and anything outside `/sys/fs/cgroup`. Rootful, the record lives in the root-only lock directory. Rootless, the child can rewrite the record (same uid), but then the kernel limits `cgroup.kill` to that user's own tree. - **Rootless escape is limited to the user's own tree** (dev-02, shell): moving to the parent: ok; to root-owned `user.slice`: refused; to `user@1500.service` (internal node): refused. - **`Delegate=yes` + `KillMode=process`** (root, systemd 257, shell). The unit gets `trusted.delegate=1` and `user.delegate=1`. `subtree_control` stays empty, so systemd enables no controllers. `systemctl stop` kills only the main process: the unit reads `inactive` while the leaf and its members live on, and the next start reuses the cgroup, where `adopt_orphans` kills them. This matches today's KillMode=process behaviour; the forked grandchild escaped before this branch too. Existing installs have no xattr, so they fall back, which the status reports. - **No systemd, ordinary cgroup:** root in `/j172-nosd` with `/run/systemd` hidden gets `kind=cgroup`, and a stop leaves nothing. ### P2-1. A rootless child that leaves its leaf hangs `stop()` forever, and status still says "contained" Evidence (dev-02, `mode=escape`): ``` escape-exit=0 now=/user.slice/user-1500.slice/user@1500.service/app.slice/run-u5344.service J172 stop returned=false after 15.002697527s J172 after stop: grandchildren=[1374931] direct-child=[1374905] leaves=[".../run-u5344.service/proxy-e46f2-0"] ``` `TraefikChild::stop` skips `child.kill()` whenever the `cgroup.kill` write succeeds, then blocks in `child.wait()`. A direct child that is no longer in the leaf never gets SIGKILL. So the supervisor worker thread hangs, and so does `TraefikSupervisor::drop` (it joins the worker), which wedges proxy reconfiguration and infraternal agent shutdown. The escaped process keeps its listeners. This needs a compromised rootless Traefik (accepted in the design note), but the result should be "the escapee runs, and stop still finishes", not a wedged infraternal agent. **Fix:** always `kill()` the direct child as well (it is pidfd-safe: std still holds the unreaped child), whether or not `cgroup.kill` succeeded. Optionally, before waiting, check that `/proc/<pid>/cgroup` still names the leaf, and if not, log that the child left its cgroup. ### P3-1. A nested cgroup made by a rootless child blocks every later start (EBUSY) Evidence (dev-02, `mode=nested`): ``` mkdir-nested-exit=0 J172 stop returned=true after 474.834µs (Drop swallows the remove error) J172 after stop: grandchildren=[] direct-child=[] leaves=[".../proxy-120b98-0"] J172 next agent start: Err(Os { code: 16, kind: ResourceBusy, message: "Device or resource busy" }) ``` `rmdir` of a leaf with an empty child cgroup fails with EBUSY, and `adopt_orphans` propagates that through `?`, so the proxy never starts again until someone removes the cgroup by hand. Only a compromised rootless child can do this (rootful mkdir is denied, see above). **Fix:** in `kill_and_remove`, after `populated 0`, remove the child cgroups depth-first (this is safe: `cgroup.kill` already covered the whole subtree, and `populated` reflects the whole subtree). Then remove the leaf. ### P3-2. Rootless status overstates containment `Containment::Cgroup.describe()` gives the same text rootful and rootless: "each Traefik process runs in its own cgroup; stopping it stops everything it started". Rootless, P2-1 shows that isn't true for a compromised child. The design note and comment 77343 are honest about this, but `mesh status`/`publish` are not. **Fix:** add a rootless variant (e.g. `kind: "cgroup-rootless"`), with text like "…; a compromised Traefik running as your user could still leave it". ### P3-3. Root in the root cgroup (no systemd) gets a false reason Evidence (compute01, mount ns without `/run/systemd`, agent moved to `/`): `J172 own=0::/ kind=fallback detail=limited: the kernel is older than Linux 5.14 (no cgroup.kill)` on kernel 6.12. The root cgroup never has `cgroup.kill` (or `cgroup.events`), on any kernel. Falling back is safe, but the stated reason is wrong. **Fix:** say "the infraternal agent runs in the root cgroup". Or support it: leaves under `/` are legal, and the root cgroup is exempt from the no-internal-processes rule. Detect `cgroup.kill` support by checking a created leaf, or `/sys/kernel/cgroup/features`, rather than the base. Kernels before 5.14 (no `cgroup.kill` anywhere): I found that by reading the code, not by running it (no such kernel was available). They take the same check and fall back with a correct message. ### P3-4. Leaves that outlive a unit losing `Delegate=yes` are never reaped Evidence (root, shell): a `Delegate=yes` + `KillMode=process` unit left a populated `proxy-*` leaf on stop. After `Delegate=yes` was removed and the unit restarted, the xattr was `absent 61`, the leaf was still there, and all its members were alive. `detect` then falls back, and `adopt_orphans` never runs, so those processes live for as long as the unit's cgroup does. This happens on a downgrade, or when a hand-written or distro unit replaces a generated one. **Fix:** in fallback, when the agent's own cgroup is writable (root, or the owner), still run `adopt_orphans` on its own cgroup and the recorded base. Killing your own tagged leaves needs no delegation. ### P3-5. Leaf tag uses the inode without the device `tag = ino(lock_directory)`. Two infraternal agents in the same cgroup, with lock directories on different filesystems that share an inode number, would reap each other's live leaves at start. This is unlikely (one agent per unit), but cheap to rule out. **Fix:** `proxy-<dev:x>-<ino:x>-<n>`. Also noted, not a finding: systemd versions that set no `trusted.delegate`/`user.delegate` xattr would report "lacks Delegate=yes" even when the unit has it. That is safe, but the message is misleading. I didn't test it (both VMs run systemd 255/257). **Gate:** no code changed in this review, so the gate is not applicable. The branch's own gate is in 77408.

kayg referenced this issue 2026-11-04 08:05:02 +00:00

macOS mesh node #265

kayg referenced this issue 2026-11-04 08:18:58 +00:00

Machine trust: self-renewal and 7-day key rotation #285

kayg referenced this issue 2026-11-04 08:21:58 +00:00

infraternal up + one-command install #263

kayg commented 2026-11-04 08:23:26 +00:00

Author

Owner

Copy link

J164 round 2 (J172 review, comment 77452) started and pushed. Branch proxy-cgroup-j164, head b9467d190, on top of 31fabf153. One commit per finding, each with its test:

• 5ccab8cb5 P3-5: the leaf is named proxy-<dev>-<ino>-<n>. Test: the_leaf_name_carries_device_and_inode.
• a13d7f65d P2-1: stop always kills the direct child as well. Test: a_child_that_leaves_its_cgroup_is_still_stopped (the child writes itself into the parent cgroup; stop must finish within 10 s).
• f1bbf1399 P3-1: nested cgroups are removed deepest first, bounded at 4096, on both stop and adoption. Test: a_cgroup_made_inside_the_leaf_never_blocks_removal.
• 0f2414a7f P3-2: a rootless variant, kind cgroup-rootless, whose text says a compromised Traefik running as your user could still move out. Test: rootless_status_says_what_is_really_contained.
• 815f36353 P3-3: "the infraternal agent runs in the root cgroup". Test: the_root_cgroup_is_named_as_the_fallback_reason.
• c7a933ee6 P3-4: adoption also runs in fallback, via ProxyCgroups::for_reaping, which makes no new leaves. Test: leftover_cgroups_are_reaped_even_without_delegation.
• b9467d190 the design note.

Queued: red (the final tests over the old cgroup.rs/supervisor.rs) and then green.

• Rootless on dev-02, in a delegated user service and in the login scope, then verify quick.
• Root on compute01 (/root/j164), in an undelegated login scope and in a Delegate=yes unit.

Gate: waiting for J168 to merge, then rebase and run verify fast.

J164 round 2 (J172 review, comment 77452) started and pushed. Branch `proxy-cgroup-j164`, head `b9467d190`, on top of `31fabf153`. One commit per finding, each with its test: - `5ccab8cb5` **P3-5:** the leaf is named `proxy-<dev>-<ino>-<n>`. Test: `the_leaf_name_carries_device_and_inode`. - `a13d7f65d` **P2-1:** stop always kills the direct child as well. Test: `a_child_that_leaves_its_cgroup_is_still_stopped` (the child writes itself into the parent cgroup; stop must finish within 10 s). - `f1bbf1399` **P3-1:** nested cgroups are removed deepest first, bounded at 4096, on both stop and adoption. Test: `a_cgroup_made_inside_the_leaf_never_blocks_removal`. - `0f2414a7f` **P3-2:** a rootless variant, kind `cgroup-rootless`, whose text says a compromised Traefik running as your user could still move out. Test: `rootless_status_says_what_is_really_contained`. - `815f36353` **P3-3:** "the infraternal agent runs in the root cgroup". Test: `the_root_cgroup_is_named_as_the_fallback_reason`. - `c7a933ee6` **P3-4:** adoption also runs in fallback, via `ProxyCgroups::for_reaping`, which makes no new leaves. Test: `leftover_cgroups_are_reaped_even_without_delegation`. - `b9467d190` the design note. Queued: red (the final tests over the old `cgroup.rs`/`supervisor.rs`) and then green. - **Rootless** on dev-02, in a delegated user service and in the login scope, then `verify quick`. - **Root** on compute01 (`/root/j164`), in an undelegated login scope and in a `Delegate=yes` unit. Gate: waiting for J168 to merge, then rebase and run `verify fast`.

kayg commented 2026-11-04 09:39:25 +00:00

Author

Owner

Copy link

J164 round 2 finished: all six J172 findings fixed. Head bc6e5880ad3dc4f09ce4d3295bb35a8bf490d5a6. Not merged, not deployed.

Commits on 31fabf153, one per finding, each with its test:

• 85b605446 P3-5: tag is device and inode
• df283a507 P2-1: stop always kills the direct child
• e2ce145fe P3-1: nested cgroups removed deepest first, bounded
• a31f9973c P3-2: cgroup-rootless kind and status text
• a2ae82a7c P3-3: root cgroup named as the reason
• 0f83fb820 P3-4: adoption also in fallback (for_reaping)
• bc6e5880a docs

I replayed the series twice while testing (the heads in comment 77494 are superseded). Once to fix a bug in the tag test: it computed the expected prefix after removing its fixture. Once because verify quick found assigning_clones in the P3-1 loop.

How red was made: the final test file over the pre-fix cgroup.rs and supervisor.rs from 31fabf153. Green is the head. The tests find processes by a unique sleep argument, never through the code under test.

Finding	Red (fixes reverted)	Green (head)	Where
P2-1 escape	rootless: stop never returned; the test failed at its 10 s expect (:339)	escape: stop=Ok(()) direct child alive=false	dev-02, rootless delegated
P3-1 nested	rootless: stop=Err(…ResourceBusy), leaf left, adopt EBUSY. Root: adopt EBUSY	stop=Ok(()), adopt=Ok(1), nothing left	dev-02 and compute01
P3-2 status	rootless kind=cgroup, "stops everything it started"	kind=cgroup-rootless, "…a compromised Traefik running as your user could still move out of it". Root unchanged	dev-02 and compute01
P3-3 root cgroup	"the kernel is older than Linux 5.14"	"the infraternal agent runs in the root cgroup"	both
P3-4 fallback leftovers	root, undelegated login scope: containment=fallback survivors=[3306605] leaf left=true	containment=fallback survivors=[] leaf left=false	compute01
P3-5 tag	proxy-313601-0	proxy-22-313c77-0	both

Rootful escape and nested-on-stop pass even before the fix (made=false, because the move and the mkdir are refused), as J172 found.

Whole proxy crate at head: rootless delegated 121 lib / 2 integration, root in Delegate=yes 121 / 2. Cleanup: no proxy-* leaves, no marker sleeps, no proxy-account processes. infraternal-proxy removed, and /root/j164 holds logs only.

verify quick (dev-02, head): fmt PASS, clippy PASS. Unit tests failed only in infraternal-agent --lib, on four remote_client tests (server_path_capability…, unavailable_udp…, winning_quic…, durable_state::acknowledged_network_access…). The orchestrator identified these as J168's dev regression. I'll rebase onto dev and run verify fast once J168 merges.

dev-02 rootless log (red, green, scope, crate, quick)

build-slot: [09:26:39] got slot 2/2
=== head bc6e5880ad3dc4f09ce4d3295bb35a8bf490d5a6, kernel 6.8.0-147-generic, systemd 255 (255.4-1ubuntu8.18)
=== RED (fixes reverted, final tests), rootless delegated user service
red exit=101
test supervisor_tests::cgroup_containment::a_cgroup_made_inside_the_leaf_never_blocks_removal ... nested: made=true stop=Err(Os { code: 16, kind: ResourceBusy, message: "Device or resource busy" }) leaf left=true
nested: leftover made=true adopt=Err(Os { code: 16, kind: ResourceBusy, message: "Device or resource busy" }) left=true
thread 'supervisor_tests::cgroup_containment::a_cgroup_made_inside_the_leaf_never_blocks_removal' (2001769) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:440:5:
test supervisor_tests::cgroup_containment::a_child_that_leaves_its_cgroup_is_still_stopped ... 
thread 'supervisor_tests::cgroup_containment::a_child_that_leaves_its_cgroup_is_still_stopped' (2001776) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:339:6:
test supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind ... 
thread 'supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind' (2001777) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:196:9:
test supervisor_tests::cgroup_containment::an_agent_crash_leaves_no_forked_process_behind_once_the_next_agent_starts ... crash: containment=cgroup survivors=[] leaves=[]
test supervisor_tests::cgroup_containment::leftover_cgroups_are_reaped_even_without_delegation ... leftover: containment=cgroup survivors=[2001797] leaf left=true
thread 'supervisor_tests::cgroup_containment::leftover_cgroups_are_reaped_even_without_delegation' (2001796) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:548:5:
test supervisor_tests::cgroup_containment::proxy_crash_helper ... ok
test supervisor_tests::cgroup_containment::rootless_status_says_what_is_really_contained ... status: kind=cgroup text=each Traefik process runs in its own cgroup; stopping it stops everything it started
thread 'supervisor_tests::cgroup_containment::rootless_status_says_what_is_really_contained' (2001821) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:465:9:
test supervisor_tests::cgroup_containment::the_leaf_name_carries_device_and_inode ... tag: proxy-4457a-0
thread 'supervisor_tests::cgroup_containment::the_leaf_name_carries_device_and_inode' (2001822) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:574:5:
test supervisor_tests::cgroup_containment::the_root_cgroup_is_named_as_the_fallback_reason ... root cgroup: the kernel is older than Linux 5.14 (no cgroup.kill)
thread 'supervisor_tests::cgroup_containment::the_root_cgroup_is_named_as_the_fallback_reason' (2001823) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:480:5:
test result: FAILED. 2 passed; 7 failed; 0 ignored; 0 measured; 112 filtered out; finished in 2.51s
=== GREEN, rootless delegated user service
green exit=0
test supervisor_tests::cgroup_containment::a_cgroup_made_inside_the_leaf_never_blocks_removal ... nested: made=true stop=Ok(()) leaf left=false
nested: leftover made=true adopt=Ok(1) left=false
test supervisor_tests::cgroup_containment::a_child_that_leaves_its_cgroup_is_still_stopped ... escape: stop=Ok(()) direct child alive=false
test supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind ... stop: containment=cgroup-rootless survivors=[] leaves=[]
test supervisor_tests::cgroup_containment::an_agent_crash_leaves_no_forked_process_behind_once_the_next_agent_starts ... crash: containment=cgroup-rootless survivors=[] leaves=[]
test supervisor_tests::cgroup_containment::leftover_cgroups_are_reaped_even_without_delegation ... leftover: containment=cgroup-rootless survivors=[] leaf left=false
test supervisor_tests::cgroup_containment::proxy_crash_helper ... ok
test supervisor_tests::cgroup_containment::rootless_status_says_what_is_really_contained ... status: kind=cgroup-rootless text=each Traefik process runs in its own cgroup, so stopping it also stops what it forks; a compromised Tra
test supervisor_tests::cgroup_containment::the_leaf_name_carries_device_and_inode ... tag: proxy-fd03-442f1-0
test supervisor_tests::cgroup_containment::the_root_cgroup_is_named_as_the_fallback_reason ... root cgroup: the infraternal agent runs in the root cgroup
test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 112 filtered out; finished in 6.46s
=== GREEN, rootless login session scope (fallback)
scope exit=0
test supervisor_tests::cgroup_containment::a_cgroup_made_inside_the_leaf_never_blocks_removal ... nested: skipped, no containment here: the infraternal agent does not run in a user service it owns
test supervisor_tests::cgroup_containment::a_child_that_leaves_its_cgroup_is_still_stopped ... escape: skipped, no containment here: the infraternal agent does not run in a user service it owns
test supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind ... stop: containment=fallback survivors=[2002472] leaves=[]
test supervisor_tests::cgroup_containment::an_agent_crash_leaves_no_forked_process_behind_once_the_next_agent_starts ... crash: containment=fallback survivors=[2002540] leaves=[]
test supervisor_tests::cgroup_containment::leftover_cgroups_are_reaped_even_without_delegation ... leftover: skipped, /sys/fs/cgroup/user.slice/user-1500.slice/session-c25451.scope is not ours to write
test supervisor_tests::cgroup_containment::proxy_crash_helper ... ok
test supervisor_tests::cgroup_containment::rootless_status_says_what_is_really_contained ... ok
test supervisor_tests::cgroup_containment::the_leaf_name_carries_device_and_inode ... tag: skipped, no containment here: the infraternal agent does not run in a user service it owns
test supervisor_tests::cgroup_containment::the_root_cgroup_is_named_as_the_fallback_reason ... root cgroup: the infraternal agent runs in the root cgroup
test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 112 filtered out; finished in 9.82s
=== whole proxy crate, rootless delegated
crate exit=0
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 120 filtered out; finished in 0.00s
test result: ok. 121 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 8.43s
test result: ok. 0 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
leaves left: 0; marker sleeps: 0
test supervisor_tests::cgroup_containment::a_cgroup_made_inside_the_leaf_never_blocks_removal ... nested: made=true stop=Err(Os { code: 16, kind: ResourceBusy, message: "Device or resource busy" }) leaf left=true
nested: leftover made=true adopt=Err(Os { code: 16, kind: ResourceBusy, message: "Device or resource busy" }) left=true
called `Result::unwrap()` on an `Err` value: Elapsed(())


failures:
    remote_client::supervisor::review_tests::durable_state::acknowledged_network_access_survives_superseded_worker_completion
    remote_client::tests::server_path_capability_is_advertised_on_native_and_fallback_transports
    remote_client::tests::unavailable_udp_does_not_add_seconds_before_tcp_admission
    remote_client::tests::winning_quic_drops_the_unfinished_tls_dial_before_control_dispatch

test result: FAILED. 222 passed; 4 failed; 22 ignored; 0 measured; 0 filtered out; finished in 286.31s

error: test failed, to rerun pass `-p infraternal-agent --lib`
(unit tests (changed): 609s)

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| clippy (workspace)        | PASS         |
| unit tests (changed)      | FAIL (101)   |
+---------------------------+--------------+
REVIEW_ROOTLESS_DONE

compute01 root log

build-slot: [10:29:36] got slot 4/4
=== head bc6e5880ad3dc4f09ce4d3295bb35a8bf490d5a6, kernel 6.12.113+deb13-amd64, systemd 257 (257.13-1~deb13u1)
=== RED (fixes reverted), root in an undelegated login scope: 0::/user.slice/user-0.slice/session-c36998.scope
exit=101
test supervisor_tests::cgroup_containment::a_cgroup_made_inside_the_leaf_never_blocks_removal ... nested: skipped, no containment here: the infraternal agent's unit lacks Delegate=yes
test supervisor_tests::cgroup_containment::a_child_that_leaves_its_cgroup_is_still_stopped ... escape: skipped, no containment here: the infraternal agent's unit lacks Delegate=yes
test supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind ... stop: containment=fallback survivors=[3306377] leaves=[]
test supervisor_tests::cgroup_containment::an_agent_crash_leaves_no_forked_process_behind_once_the_next_agent_starts ... crash: containment=fallback survivors=[3306455] leaves=[]
test supervisor_tests::cgroup_containment::leftover_cgroups_are_reaped_even_without_delegation ... leftover: containment=fallback survivors=[3306605] leaf left=true
thread 'supervisor_tests::cgroup_containment::leftover_cgroups_are_reaped_even_without_delegation' (3306604) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:548:5:
test supervisor_tests::cgroup_containment::proxy_crash_helper ... ok
test supervisor_tests::cgroup_containment::rootless_status_says_what_is_really_contained ... ok
test supervisor_tests::cgroup_containment::the_leaf_name_carries_device_and_inode ... tag: skipped, no containment here: the infraternal agent's unit lacks Delegate=yes
test supervisor_tests::cgroup_containment::the_root_cgroup_is_named_as_the_fallback_reason ... root cgroup: the kernel is older than Linux 5.14 (no cgroup.kill)
thread 'supervisor_tests::cgroup_containment::the_root_cgroup_is_named_as_the_fallback_reason' (3306640) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:480:5:
test result: FAILED. 7 passed; 2 failed; 0 ignored; 0 measured; 112 filtered out; finished in 11.61s
=== RED (fixes reverted), root in a Delegate=yes unit
exit=101
test supervisor_tests::cgroup_containment::a_cgroup_made_inside_the_leaf_never_blocks_removal ... nested: made=false stop=Ok(()) leaf left=false
nested: leftover made=true adopt=Err(Os { code: 16, kind: ResourceBusy, message: "Device or resource busy" }) left=true
thread 'supervisor_tests::cgroup_containment::a_cgroup_made_inside_the_leaf_never_blocks_removal' (3306682) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:445:5:
test supervisor_tests::cgroup_containment::a_child_that_leaves_its_cgroup_is_still_stopped ... escape: stop=Ok(()) direct child alive=false
test supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind ... 
thread 'supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind' (3306696) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:196:9:
test supervisor_tests::cgroup_containment::an_agent_crash_leaves_no_forked_process_behind_once_the_next_agent_starts ... crash: containment=cgroup survivors=[] leaves=[]
test supervisor_tests::cgroup_containment::leftover_cgroups_are_reaped_even_without_delegation ... leftover: containment=cgroup survivors=[3306716] leaf left=true
thread 'supervisor_tests::cgroup_containment::leftover_cgroups_are_reaped_even_without_delegation' (3306715) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:548:5:
test supervisor_tests::cgroup_containment::proxy_crash_helper ... ok
test supervisor_tests::cgroup_containment::rootless_status_says_what_is_really_contained ... status: kind=cgroup text=each Traefik process runs in its own cgroup; stopping it stops everything it started
test supervisor_tests::cgroup_containment::the_leaf_name_carries_device_and_inode ... tag: proxy-313601-0
thread 'supervisor_tests::cgroup_containment::the_leaf_name_carries_device_and_inode' (3306796) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:574:5:
test supervisor_tests::cgroup_containment::the_root_cgroup_is_named_as_the_fallback_reason ... root cgroup: the kernel is older than Linux 5.14 (no cgroup.kill)
thread 'supervisor_tests::cgroup_containment::the_root_cgroup_is_named_as_the_fallback_reason' (3306797) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:480:5:
test result: FAILED. 4 passed; 5 failed; 0 ignored; 0 measured; 112 filtered out; finished in 2.66s
=== GREEN, root in an undelegated login scope
exit=0
test supervisor_tests::cgroup_containment::a_cgroup_made_inside_the_leaf_never_blocks_removal ... nested: skipped, no containment here: the infraternal agent's unit lacks Delegate=yes
test supervisor_tests::cgroup_containment::a_child_that_leaves_its_cgroup_is_still_stopped ... escape: skipped, no containment here: the infraternal agent's unit lacks Delegate=yes
test supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind ... stop: containment=fallback survivors=[3307323] leaves=[]
test supervisor_tests::cgroup_containment::an_agent_crash_leaves_no_forked_process_behind_once_the_next_agent_starts ... crash: containment=fallback survivors=[3307358] leaves=[]
test supervisor_tests::cgroup_containment::leftover_cgroups_are_reaped_even_without_delegation ... leftover: containment=fallback survivors=[] leaf left=false
test supervisor_tests::cgroup_containment::proxy_crash_helper ... ok
test supervisor_tests::cgroup_containment::rootless_status_says_what_is_really_contained ... ok
test supervisor_tests::cgroup_containment::the_leaf_name_carries_device_and_inode ... tag: skipped, no containment here: the infraternal agent's unit lacks Delegate=yes
test supervisor_tests::cgroup_containment::the_root_cgroup_is_named_as_the_fallback_reason ... root cgroup: the infraternal agent runs in the root cgroup
test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 112 filtered out; finished in 9.49s
=== GREEN, root in a Delegate=yes unit
exit=0
test supervisor_tests::cgroup_containment::a_cgroup_made_inside_the_leaf_never_blocks_removal ... nested: made=false stop=Ok(()) leaf left=false
nested: leftover made=true adopt=Ok(1) left=false
test supervisor_tests::cgroup_containment::a_child_that_leaves_its_cgroup_is_still_stopped ... escape: stop=Ok(()) direct child alive=false
test supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind ... stop: containment=cgroup survivors=[] leaves=[]
test supervisor_tests::cgroup_containment::an_agent_crash_leaves_no_forked_process_behind_once_the_next_agent_starts ... crash: containment=cgroup survivors=[] leaves=[]
test supervisor_tests::cgroup_containment::leftover_cgroups_are_reaped_even_without_delegation ... leftover: containment=cgroup survivors=[] leaf left=false
test supervisor_tests::cgroup_containment::proxy_crash_helper ... ok
test supervisor_tests::cgroup_containment::rootless_status_says_what_is_really_contained ... status: kind=cgroup text=each Traefik process runs in its own cgroup; stopping it stops everything it started
test supervisor_tests::cgroup_containment::the_leaf_name_carries_device_and_inode ... tag: proxy-22-313c77-0
test supervisor_tests::cgroup_containment::the_root_cgroup_is_named_as_the_fallback_reason ... root cgroup: the infraternal agent runs in the root cgroup
test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 112 filtered out; finished in 0.59s
=== whole proxy crate as root, Delegate=yes unit
crate exit=0
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 120 filtered out; finished in 0.00s
test result: ok. 121 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 17.91s
test result: ok. 0 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
leaves: 0; proxy-account processes: 0; marker sleeps: 0
account after cleanup: [] []
REVIEW_ROOT_DONE

## J164 round 2 finished: all six J172 findings fixed. Head `bc6e5880ad3dc4f09ce4d3295bb35a8bf490d5a6`. Not merged, not deployed. Commits on `31fabf153`, one per finding, each with its test: - `85b605446` P3-5: tag is device and inode - `df283a507` P2-1: stop always kills the direct child - `e2ce145fe` P3-1: nested cgroups removed deepest first, bounded - `a31f9973c` P3-2: `cgroup-rootless` kind and status text - `a2ae82a7c` P3-3: root cgroup named as the reason - `0f83fb820` P3-4: adoption also in fallback (`for_reaping`) - `bc6e5880a` docs I replayed the series twice while testing (the heads in comment 77494 are superseded). Once to fix a bug in the tag test: it computed the expected prefix after removing its fixture. Once because `verify quick` found `assigning_clones` in the P3-1 loop. **How red was made:** the final test file over the pre-fix `cgroup.rs` and `supervisor.rs` from `31fabf153`. Green is the head. The tests find processes by a unique `sleep` argument, never through the code under test. | Finding | Red (fixes reverted) | Green (head) | Where | |---|---|---|---| | P2-1 escape | rootless: stop never returned; the test failed at its 10 s expect (`:339`) | `escape: stop=Ok(()) direct child alive=false` | dev-02, rootless delegated | | P3-1 nested | rootless: `stop=Err(…ResourceBusy)`, leaf left, adopt EBUSY. Root: adopt EBUSY | `stop=Ok(())`, `adopt=Ok(1)`, nothing left | dev-02 and compute01 | | P3-2 status | rootless `kind=cgroup`, "stops everything it started" | `kind=cgroup-rootless`, "…a compromised Traefik running as your user could still move out of it". Root unchanged | dev-02 and compute01 | | P3-3 root cgroup | "the kernel is older than Linux 5.14" | "the infraternal agent runs in the root cgroup" | both | | P3-4 fallback leftovers | root, undelegated login scope: `containment=fallback survivors=[3306605] leaf left=true` | `containment=fallback survivors=[] leaf left=false` | compute01 | | P3-5 tag | `proxy-313601-0` | `proxy-22-313c77-0` | both | Rootful escape and nested-on-stop pass even before the fix (`made=false`, because the move and the mkdir are refused), as J172 found. Whole proxy crate at head: rootless delegated 121 lib / 2 integration, root in `Delegate=yes` 121 / 2. Cleanup: no `proxy-*` leaves, no marker sleeps, no proxy-account processes. `infraternal-proxy` removed, and `/root/j164` holds logs only. **`verify quick`** (dev-02, head): fmt PASS, clippy PASS. Unit tests failed only in `infraternal-agent --lib`, on four `remote_client` tests (`server_path_capability…`, `unavailable_udp…`, `winning_quic…`, `durable_state::acknowledged_network_access…`). The orchestrator identified these as J168's dev regression. I'll rebase onto dev and run `verify fast` once J168 merges. <details><summary>dev-02 rootless log (red, green, scope, crate, quick)</summary> ``` build-slot: [09:26:39] got slot 2/2 === head bc6e5880ad3dc4f09ce4d3295bb35a8bf490d5a6, kernel 6.8.0-147-generic, systemd 255 (255.4-1ubuntu8.18) === RED (fixes reverted, final tests), rootless delegated user service red exit=101 test supervisor_tests::cgroup_containment::a_cgroup_made_inside_the_leaf_never_blocks_removal ... nested: made=true stop=Err(Os { code: 16, kind: ResourceBusy, message: "Device or resource busy" }) leaf left=true nested: leftover made=true adopt=Err(Os { code: 16, kind: ResourceBusy, message: "Device or resource busy" }) left=true thread 'supervisor_tests::cgroup_containment::a_cgroup_made_inside_the_leaf_never_blocks_removal' (2001769) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:440:5: test supervisor_tests::cgroup_containment::a_child_that_leaves_its_cgroup_is_still_stopped ... thread 'supervisor_tests::cgroup_containment::a_child_that_leaves_its_cgroup_is_still_stopped' (2001776) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:339:6: test supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind ... thread 'supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind' (2001777) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:196:9: test supervisor_tests::cgroup_containment::an_agent_crash_leaves_no_forked_process_behind_once_the_next_agent_starts ... crash: containment=cgroup survivors=[] leaves=[] test supervisor_tests::cgroup_containment::leftover_cgroups_are_reaped_even_without_delegation ... leftover: containment=cgroup survivors=[2001797] leaf left=true thread 'supervisor_tests::cgroup_containment::leftover_cgroups_are_reaped_even_without_delegation' (2001796) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:548:5: test supervisor_tests::cgroup_containment::proxy_crash_helper ... ok test supervisor_tests::cgroup_containment::rootless_status_says_what_is_really_contained ... status: kind=cgroup text=each Traefik process runs in its own cgroup; stopping it stops everything it started thread 'supervisor_tests::cgroup_containment::rootless_status_says_what_is_really_contained' (2001821) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:465:9: test supervisor_tests::cgroup_containment::the_leaf_name_carries_device_and_inode ... tag: proxy-4457a-0 thread 'supervisor_tests::cgroup_containment::the_leaf_name_carries_device_and_inode' (2001822) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:574:5: test supervisor_tests::cgroup_containment::the_root_cgroup_is_named_as_the_fallback_reason ... root cgroup: the kernel is older than Linux 5.14 (no cgroup.kill) thread 'supervisor_tests::cgroup_containment::the_root_cgroup_is_named_as_the_fallback_reason' (2001823) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:480:5: test result: FAILED. 2 passed; 7 failed; 0 ignored; 0 measured; 112 filtered out; finished in 2.51s === GREEN, rootless delegated user service green exit=0 test supervisor_tests::cgroup_containment::a_cgroup_made_inside_the_leaf_never_blocks_removal ... nested: made=true stop=Ok(()) leaf left=false nested: leftover made=true adopt=Ok(1) left=false test supervisor_tests::cgroup_containment::a_child_that_leaves_its_cgroup_is_still_stopped ... escape: stop=Ok(()) direct child alive=false test supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind ... stop: containment=cgroup-rootless survivors=[] leaves=[] test supervisor_tests::cgroup_containment::an_agent_crash_leaves_no_forked_process_behind_once_the_next_agent_starts ... crash: containment=cgroup-rootless survivors=[] leaves=[] test supervisor_tests::cgroup_containment::leftover_cgroups_are_reaped_even_without_delegation ... leftover: containment=cgroup-rootless survivors=[] leaf left=false test supervisor_tests::cgroup_containment::proxy_crash_helper ... ok test supervisor_tests::cgroup_containment::rootless_status_says_what_is_really_contained ... status: kind=cgroup-rootless text=each Traefik process runs in its own cgroup, so stopping it also stops what it forks; a compromised Tra test supervisor_tests::cgroup_containment::the_leaf_name_carries_device_and_inode ... tag: proxy-fd03-442f1-0 test supervisor_tests::cgroup_containment::the_root_cgroup_is_named_as_the_fallback_reason ... root cgroup: the infraternal agent runs in the root cgroup test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 112 filtered out; finished in 6.46s === GREEN, rootless login session scope (fallback) scope exit=0 test supervisor_tests::cgroup_containment::a_cgroup_made_inside_the_leaf_never_blocks_removal ... nested: skipped, no containment here: the infraternal agent does not run in a user service it owns test supervisor_tests::cgroup_containment::a_child_that_leaves_its_cgroup_is_still_stopped ... escape: skipped, no containment here: the infraternal agent does not run in a user service it owns test supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind ... stop: containment=fallback survivors=[2002472] leaves=[] test supervisor_tests::cgroup_containment::an_agent_crash_leaves_no_forked_process_behind_once_the_next_agent_starts ... crash: containment=fallback survivors=[2002540] leaves=[] test supervisor_tests::cgroup_containment::leftover_cgroups_are_reaped_even_without_delegation ... leftover: skipped, /sys/fs/cgroup/user.slice/user-1500.slice/session-c25451.scope is not ours to write test supervisor_tests::cgroup_containment::proxy_crash_helper ... ok test supervisor_tests::cgroup_containment::rootless_status_says_what_is_really_contained ... ok test supervisor_tests::cgroup_containment::the_leaf_name_carries_device_and_inode ... tag: skipped, no containment here: the infraternal agent does not run in a user service it owns test supervisor_tests::cgroup_containment::the_root_cgroup_is_named_as_the_fallback_reason ... root cgroup: the infraternal agent runs in the root cgroup test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 112 filtered out; finished in 9.82s === whole proxy crate, rootless delegated crate exit=0 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 120 filtered out; finished in 0.00s test result: ok. 121 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 8.43s test result: ok. 0 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s leaves left: 0; marker sleeps: 0 test supervisor_tests::cgroup_containment::a_cgroup_made_inside_the_leaf_never_blocks_removal ... nested: made=true stop=Err(Os { code: 16, kind: ResourceBusy, message: "Device or resource busy" }) leaf left=true nested: leftover made=true adopt=Err(Os { code: 16, kind: ResourceBusy, message: "Device or resource busy" }) left=true called `Result::unwrap()` on an `Err` value: Elapsed(()) failures: remote_client::supervisor::review_tests::durable_state::acknowledged_network_access_survives_superseded_worker_completion remote_client::tests::server_path_capability_is_advertised_on_native_and_fallback_transports remote_client::tests::unavailable_udp_does_not_add_seconds_before_tcp_admission remote_client::tests::winning_quic_drops_the_unfinished_tls_dial_before_control_dispatch test result: FAILED. 222 passed; 4 failed; 22 ignored; 0 measured; 0 filtered out; finished in 286.31s error: test failed, to rerun pass `-p infraternal-agent --lib` (unit tests (changed): 609s) +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | clippy (workspace) | PASS | | unit tests (changed) | FAIL (101) | +---------------------------+--------------+ REVIEW_ROOTLESS_DONE ``` </details> <details><summary>compute01 root log</summary> ``` build-slot: [10:29:36] got slot 4/4 === head bc6e5880ad3dc4f09ce4d3295bb35a8bf490d5a6, kernel 6.12.113+deb13-amd64, systemd 257 (257.13-1~deb13u1) === RED (fixes reverted), root in an undelegated login scope: 0::/user.slice/user-0.slice/session-c36998.scope exit=101 test supervisor_tests::cgroup_containment::a_cgroup_made_inside_the_leaf_never_blocks_removal ... nested: skipped, no containment here: the infraternal agent's unit lacks Delegate=yes test supervisor_tests::cgroup_containment::a_child_that_leaves_its_cgroup_is_still_stopped ... escape: skipped, no containment here: the infraternal agent's unit lacks Delegate=yes test supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind ... stop: containment=fallback survivors=[3306377] leaves=[] test supervisor_tests::cgroup_containment::an_agent_crash_leaves_no_forked_process_behind_once_the_next_agent_starts ... crash: containment=fallback survivors=[3306455] leaves=[] test supervisor_tests::cgroup_containment::leftover_cgroups_are_reaped_even_without_delegation ... leftover: containment=fallback survivors=[3306605] leaf left=true thread 'supervisor_tests::cgroup_containment::leftover_cgroups_are_reaped_even_without_delegation' (3306604) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:548:5: test supervisor_tests::cgroup_containment::proxy_crash_helper ... ok test supervisor_tests::cgroup_containment::rootless_status_says_what_is_really_contained ... ok test supervisor_tests::cgroup_containment::the_leaf_name_carries_device_and_inode ... tag: skipped, no containment here: the infraternal agent's unit lacks Delegate=yes test supervisor_tests::cgroup_containment::the_root_cgroup_is_named_as_the_fallback_reason ... root cgroup: the kernel is older than Linux 5.14 (no cgroup.kill) thread 'supervisor_tests::cgroup_containment::the_root_cgroup_is_named_as_the_fallback_reason' (3306640) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:480:5: test result: FAILED. 7 passed; 2 failed; 0 ignored; 0 measured; 112 filtered out; finished in 11.61s === RED (fixes reverted), root in a Delegate=yes unit exit=101 test supervisor_tests::cgroup_containment::a_cgroup_made_inside_the_leaf_never_blocks_removal ... nested: made=false stop=Ok(()) leaf left=false nested: leftover made=true adopt=Err(Os { code: 16, kind: ResourceBusy, message: "Device or resource busy" }) left=true thread 'supervisor_tests::cgroup_containment::a_cgroup_made_inside_the_leaf_never_blocks_removal' (3306682) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:445:5: test supervisor_tests::cgroup_containment::a_child_that_leaves_its_cgroup_is_still_stopped ... escape: stop=Ok(()) direct child alive=false test supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind ... thread 'supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind' (3306696) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:196:9: test supervisor_tests::cgroup_containment::an_agent_crash_leaves_no_forked_process_behind_once_the_next_agent_starts ... crash: containment=cgroup survivors=[] leaves=[] test supervisor_tests::cgroup_containment::leftover_cgroups_are_reaped_even_without_delegation ... leftover: containment=cgroup survivors=[3306716] leaf left=true thread 'supervisor_tests::cgroup_containment::leftover_cgroups_are_reaped_even_without_delegation' (3306715) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:548:5: test supervisor_tests::cgroup_containment::proxy_crash_helper ... ok test supervisor_tests::cgroup_containment::rootless_status_says_what_is_really_contained ... status: kind=cgroup text=each Traefik process runs in its own cgroup; stopping it stops everything it started test supervisor_tests::cgroup_containment::the_leaf_name_carries_device_and_inode ... tag: proxy-313601-0 thread 'supervisor_tests::cgroup_containment::the_leaf_name_carries_device_and_inode' (3306796) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:574:5: test supervisor_tests::cgroup_containment::the_root_cgroup_is_named_as_the_fallback_reason ... root cgroup: the kernel is older than Linux 5.14 (no cgroup.kill) thread 'supervisor_tests::cgroup_containment::the_root_cgroup_is_named_as_the_fallback_reason' (3306797) panicked at crates/plugins/proxy/src/supervisor_cgroup_tests.rs:480:5: test result: FAILED. 4 passed; 5 failed; 0 ignored; 0 measured; 112 filtered out; finished in 2.66s === GREEN, root in an undelegated login scope exit=0 test supervisor_tests::cgroup_containment::a_cgroup_made_inside_the_leaf_never_blocks_removal ... nested: skipped, no containment here: the infraternal agent's unit lacks Delegate=yes test supervisor_tests::cgroup_containment::a_child_that_leaves_its_cgroup_is_still_stopped ... escape: skipped, no containment here: the infraternal agent's unit lacks Delegate=yes test supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind ... stop: containment=fallback survivors=[3307323] leaves=[] test supervisor_tests::cgroup_containment::an_agent_crash_leaves_no_forked_process_behind_once_the_next_agent_starts ... crash: containment=fallback survivors=[3307358] leaves=[] test supervisor_tests::cgroup_containment::leftover_cgroups_are_reaped_even_without_delegation ... leftover: containment=fallback survivors=[] leaf left=false test supervisor_tests::cgroup_containment::proxy_crash_helper ... ok test supervisor_tests::cgroup_containment::rootless_status_says_what_is_really_contained ... ok test supervisor_tests::cgroup_containment::the_leaf_name_carries_device_and_inode ... tag: skipped, no containment here: the infraternal agent's unit lacks Delegate=yes test supervisor_tests::cgroup_containment::the_root_cgroup_is_named_as_the_fallback_reason ... root cgroup: the infraternal agent runs in the root cgroup test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 112 filtered out; finished in 9.49s === GREEN, root in a Delegate=yes unit exit=0 test supervisor_tests::cgroup_containment::a_cgroup_made_inside_the_leaf_never_blocks_removal ... nested: made=false stop=Ok(()) leaf left=false nested: leftover made=true adopt=Ok(1) left=false test supervisor_tests::cgroup_containment::a_child_that_leaves_its_cgroup_is_still_stopped ... escape: stop=Ok(()) direct child alive=false test supervisor_tests::cgroup_containment::a_stopped_child_leaves_no_forked_process_behind ... stop: containment=cgroup survivors=[] leaves=[] test supervisor_tests::cgroup_containment::an_agent_crash_leaves_no_forked_process_behind_once_the_next_agent_starts ... crash: containment=cgroup survivors=[] leaves=[] test supervisor_tests::cgroup_containment::leftover_cgroups_are_reaped_even_without_delegation ... leftover: containment=cgroup survivors=[] leaf left=false test supervisor_tests::cgroup_containment::proxy_crash_helper ... ok test supervisor_tests::cgroup_containment::rootless_status_says_what_is_really_contained ... status: kind=cgroup text=each Traefik process runs in its own cgroup; stopping it stops everything it started test supervisor_tests::cgroup_containment::the_leaf_name_carries_device_and_inode ... tag: proxy-22-313c77-0 test supervisor_tests::cgroup_containment::the_root_cgroup_is_named_as_the_fallback_reason ... root cgroup: the infraternal agent runs in the root cgroup test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 112 filtered out; finished in 0.59s === whole proxy crate as root, Delegate=yes unit crate exit=0 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 120 filtered out; finished in 0.00s test result: ok. 121 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 17.91s test result: ok. 0 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s leaves: 0; proxy-account processes: 0; marker sleeps: 0 account after cleanup: [] [] REVIEW_ROOT_DONE ``` </details>

kayg commented 2026-11-04 09:44:52 +00:00

Author

Owner

Copy link

J168 finish: head cb05abcaf1d142e69f5ec64a71358b55a729d72f on branch proxy-start-io-j168 (base 226a569ac, pushed)

Commit	What
6774296f5	Test, red first: opening_custody_on_fresh_state_writes_nothing (open + cleanup + get + delete on a fresh state dir leave the state dir empty; timing-independent) and first_secret_creates_the_private_store_lazily.
752e57d0a	Fix. FileDnsTokenCustody::open only validates an existing proxy-secrets; put creates root and provider dir (same parent+self fsyncs) on first write. Cleanup/delete on a never-written store are no-ops, get is NotFound. Durability for stored secrets is unchanged. Rejected: a background-thread fsync (IO stays on a path with no reason to run, races the first put).
cb05abcaf	rustfmt of the new test. 752e57d0a alone passes fmt only with this commit on top.

Red/green (compute01): at 6774296f5 opening_custody_on_fresh_state_writes_nothing ... FAILED (7 passed, 1 failed). At 752e57d0a all 111 proxy lib tests pass.

Proof (the brief): cargo test --workspace --locked -j4 --lib -- remote_client:: login_binary:: at 752e57d0a (same code as the head apart from test formatting): 20 of 20 pass, load 13 to 44 during the runs (logs: ~claude/j168-evidence/loop-*.log on compute01).

scripts/verify fast is NOT green at the head, twice, with a different unrelated test each time:

• Run 1 (752e57d0a): fmt FAIL (mine, fixed by cb05abcaf), test FAIL: status_restart::service_states_are_truthful_in_text_and_json, "service manager query timed out".
• Run 2 (cb05abcaf): fmt PASS, clippy PASS, static builds PASS, test FAIL: agent_daemon::real_binary_rootless_cleanup_survives_sender_disconnect_before_readiness, terminal.login.revoke -> resource_unavailable.
• Neither touches proxy custody. Both pass 6 of 6 in isolation at cb05abcaf and 6 of 6 at base 226a569ac (cargo test -p infraternal --test agent_daemon --test status_restart, load 3 to 8). I did NOT reproduce either failure on unmodified base, so I cannot say they also fail there; they look like load flakes in the gate (the gate ran at high load). Logs: ~claude/j168-evidence/fast-run{1,2}.log.
• I did not rerun the gate a third time. Needs a decision: rerun once more, or accept the two isolated 6/6 results.

Static-size delta (report only): aarch64 +611536 bytes vs baseline in run 1 (this change adds no code beyond a few lines; the baseline predates other dev merges).

## J168 finish: head `cb05abcaf1d142e69f5ec64a71358b55a729d72f` on branch `proxy-start-io-j168` (base `226a569ac`, pushed) | Commit | What | |---|---| | `6774296f5` | Test, red first: `opening_custody_on_fresh_state_writes_nothing` (open + cleanup + get + delete on a fresh state dir leave the state dir empty; timing-independent) and `first_secret_creates_the_private_store_lazily`. | | `752e57d0a` | Fix. `FileDnsTokenCustody::open` only validates an existing `proxy-secrets`; `put` creates root and provider dir (same parent+self fsyncs) on first write. Cleanup/delete on a never-written store are no-ops, `get` is NotFound. Durability for stored secrets is unchanged. Rejected: a background-thread fsync (IO stays on a path with no reason to run, races the first `put`). | | `cb05abcaf` | rustfmt of the new test. `752e57d0a` alone passes fmt only with this commit on top. | **Red/green (compute01):** at `6774296f5` `opening_custody_on_fresh_state_writes_nothing ... FAILED` (7 passed, 1 failed). At `752e57d0a` all 111 proxy lib tests pass. **Proof (the brief):** `cargo test --workspace --locked -j4 --lib -- remote_client:: login_binary::` at `752e57d0a` (same code as the head apart from test formatting): **20 of 20 pass**, load 13 to 44 during the runs (logs: `~claude/j168-evidence/loop-*.log` on compute01). **`scripts/verify fast` is NOT green at the head, twice, with a different unrelated test each time:** - Run 1 (`752e57d0a`): fmt FAIL (mine, fixed by `cb05abcaf`), test FAIL: `status_restart::service_states_are_truthful_in_text_and_json`, "service manager query timed out". - Run 2 (`cb05abcaf`): fmt PASS, clippy PASS, static builds PASS, test FAIL: `agent_daemon::real_binary_rootless_cleanup_survives_sender_disconnect_before_readiness`, `terminal.login.revoke` -> `resource_unavailable`. - Neither touches proxy custody. Both pass **6 of 6** in isolation at `cb05abcaf` and **6 of 6** at base `226a569ac` (`cargo test -p infraternal --test agent_daemon --test status_restart`, load 3 to 8). I did NOT reproduce either failure on unmodified base, so I cannot say they also fail there; they look like load flakes in the gate (the gate ran at high load). Logs: `~claude/j168-evidence/fast-run{1,2}.log`. - I did not rerun the gate a third time. Needs a decision: rerun once more, or accept the two isolated 6/6 results. Static-size delta (report only): aarch64 +611536 bytes vs baseline in run 1 (this change adds no code beyond a few lines; the baseline predates other dev merges).

kayg referenced this issue 2026-11-04 09:55:52 +00:00

DECISIONS consistency audit 2026-11-04 #341

kayg referenced this issue 2026-11-04 10:03:41 +00:00

Engineering practices proposal #344

kayg referenced this issue 2026-11-04 10:22:03 +00:00

Engineering practices proposal #344

kayg commented 2026-11-04 11:22:01 +00:00

Author

Owner

Copy link

J177 verdict on proxy-cgroup-j164 @ bc6e5880ad3dc4f09ce4d3295bb35a8bf490d5a6: APPROVE

Scope: only the diff 31fabf153..bc6e5880a. Each of the six J172 findings (77452) is closed, and I found no new hole. One P3 note below, which does not block.

Code review, finding by finding

• P2-1 (escape hangs stop). stop() now always calls child.kill() after the best-effort cgroup.kill, and wait() stays the authoritative reap. The kill is kill(pid), not a pidfd, but the PID can't be reused before it: std has not reaped the child yet, so the PID stays a zombie and can't be handed out again. That only fails if other code reaps it first. I checked: the proxy crate and the infraternal agent have no waitpid(-1) and no SIGCHLD=SIG_IGN. The only waitpid calls (server embedded.rs, and the shim, which is a separate process) name a specific PID. A pidfd isn't needed. If std's pidfd spawn becomes stable, it would make this robust against future reapers.
• P3-1 (nested EBUSY). The breadth-first walk starts at the leaf and only follows read_dir entries that are directories. cgroupfs has no symlinks, DirEntry::file_type doesn't follow links anyway, and cgroup v2 refuses rename, so the walk can't leave the leaf's subtree. Reversing the breadth-first order removes every child before its parent. The cap is checked before each push, so the walk fails closed on the 4097th entry. The cost is bounded: at most 4096 paths and 4097 read_dir calls.
• P3-2. There is a separate CgroupRootless kind, chosen by the effective uid. Its text no longer says "stops everything", and it states the escape. The rootful text is unchanged.
• P3-3. check_base compares with /sys/fs/cgroup before it looks for cgroup.kill, so the reason is now right.
• P3-4. for_reaping uses only /proc/self/cgroup (through the existing own_cgroup validation) plus the recorded base (valid_recorded_base). It never creates leaves. It touches only directories whose names start with this supervisor's own proxy-<dev>-<ino>- tag, so the J172 subtree argument still holds.
• P3-5. The tag is proxy-{dev:x}-{ino:x}-. Hex has no -, and the tag ends with -, so no tag is a prefix of another (proxy-a-b- vs proxy-a-bc- vs proxy-ab-…). The name is never parsed, only compared with starts_with. Leaves with the old inode-only names aren't matched, but that format never shipped.

Non-blocking P3: a rootless child can block its own restarts with the 4096 cap

A compromised rootless child owns its leaf. It can make more than 4096 nested cgroups, or, after it moves out (P2-1), keep running mkdir inside the leaf while the stop runs. Either way kill_and_remove returns an error and adopt_orphans refuses every later start until someone cleans up by hand. This is fail-closed. It only affects that user's own proxy, and it needs the compromised rootless Traefik that the design note already accepts. A possible follow-up: write cgroup.max.descendants on the leaf when it is created, so the kernel enforces the limit.

Reproduction (the branch's own tests at head, cgroup_containment, INFRATERNAL_REQUIRE_PROXY_CGROUP=1)

Rootless, dev-02 (systemd-run --user -p Delegate=yes, kernel 6.8, systemd 255): 9 passed; 0 failed

nested: made=true stop=Ok(()) leaf left=false
nested: leftover made=true adopt=Ok(1) left=false
escape: stop=Ok(()) direct child alive=false
stop: containment=cgroup-rootless survivors=[] leaves=[]
crash: containment=cgroup-rootless survivors=[] leaves=[]
leftover: containment=cgroup-rootless survivors=[] leaf left=false
status: kind=cgroup-rootless text=each Traefik process runs in its own cgroup, so stopping it also stops what it forks; a compromised Traefik running as your user could still move out of it
tag: proxy-fd03-45035-0
root cgroup: the infraternal agent runs in the root cgroup

Root, compute01 (systemd-run -p Delegate=yes -p KillMode=process): 9 passed; 0 failed

nested: made=false stop=Ok(()) leaf left=false        (rootful mkdir refused, as J172 found)
nested: leftover made=true adopt=Ok(1) left=false
escape: stop=Ok(()) direct child alive=false
stop: containment=cgroup survivors=[] leaves=[]
crash: containment=cgroup survivors=[] leaves=[]
leftover: containment=cgroup survivors=[] leaf left=false
status: kind=cgroup text=each Traefik process runs in its own cgroup; stopping it stops everything it started
tag: proxy-22-3390df-0
root cgroup: the infraternal agent runs in the root cgroup

Root, undelegated unit (P3-4 fallback path): leftover: containment=fallback survivors=[] leaf left=false, 1 passed.

I didn't repeat red: J164's red runs (77584) show each test failing against the code before the fixes.

Cleanup: 0 proxy-* cgroups on either VM, and no test processes left. I removed the scratch targets, the test binaries and the ~/worktrees/j177 worktrees. Logs remain at dev-02 ~/j177/user-run.log and compute01 /root/j177/root-run.log. The branch is unchanged.

Gate: I changed no code in this review, so the gate doesn't apply. J164 still owes verify fast once it has rebased onto dev after J168 merges (see 77584).

## J177 verdict on `proxy-cgroup-j164` @ `bc6e5880ad3dc4f09ce4d3295bb35a8bf490d5a6`: **APPROVE** Scope: only the diff `31fabf153..bc6e5880a`. Each of the six J172 findings (77452) is closed, and I found no new hole. One P3 note below, which does not block. ### Code review, finding by finding - **P2-1 (escape hangs stop).** `stop()` now always calls `child.kill()` after the best-effort `cgroup.kill`, and `wait()` stays the authoritative reap. The kill is `kill(pid)`, not a pidfd, but the PID can't be reused before it: std has not reaped the child yet, so the PID stays a zombie and can't be handed out again. That only fails if other code reaps it first. I checked: the proxy crate and the infraternal agent have no `waitpid(-1)` and no `SIGCHLD=SIG_IGN`. The only `waitpid` calls (server `embedded.rs`, and the shim, which is a separate process) name a specific PID. A pidfd isn't needed. If std's pidfd spawn becomes stable, it would make this robust against future reapers. - **P3-1 (nested EBUSY).** The breadth-first walk starts at the leaf and only follows `read_dir` entries that are directories. cgroupfs has no symlinks, `DirEntry::file_type` doesn't follow links anyway, and cgroup v2 refuses rename, so the walk can't leave the leaf's subtree. Reversing the breadth-first order removes every child before its parent. The cap is checked before each push, so the walk fails closed on the 4097th entry. The cost is bounded: at most 4096 paths and 4097 `read_dir` calls. - **P3-2.** There is a separate `CgroupRootless` kind, chosen by the effective uid. Its text no longer says "stops everything", and it states the escape. The rootful text is unchanged. - **P3-3.** `check_base` compares with `/sys/fs/cgroup` before it looks for `cgroup.kill`, so the reason is now right. - **P3-4.** `for_reaping` uses only `/proc/self/cgroup` (through the existing `own_cgroup` validation) plus the recorded base (`valid_recorded_base`). It never creates leaves. It touches only directories whose names start with this supervisor's own `proxy-<dev>-<ino>-` tag, so the J172 subtree argument still holds. - **P3-5.** The tag is `proxy-{dev:x}-{ino:x}-`. Hex has no `-`, and the tag ends with `-`, so no tag is a prefix of another (`proxy-a-b-` vs `proxy-a-bc-` vs `proxy-ab-…`). The name is never parsed, only compared with `starts_with`. Leaves with the old inode-only names aren't matched, but that format never shipped. ### Non-blocking P3: a rootless child can block its own restarts with the 4096 cap A compromised rootless child owns its leaf. It can make more than 4096 nested cgroups, or, after it moves out (P2-1), keep running `mkdir` inside the leaf while the stop runs. Either way `kill_and_remove` returns an error and `adopt_orphans` refuses every later start until someone cleans up by hand. This is fail-closed. It only affects that user's own proxy, and it needs the compromised rootless Traefik that the design note already accepts. A possible follow-up: write `cgroup.max.descendants` on the leaf when it is created, so the kernel enforces the limit. ### Reproduction (the branch's own tests at head, `cgroup_containment`, `INFRATERNAL_REQUIRE_PROXY_CGROUP=1`) **Rootless, dev-02** (`systemd-run --user -p Delegate=yes`, kernel 6.8, systemd 255): `9 passed; 0 failed` ``` nested: made=true stop=Ok(()) leaf left=false nested: leftover made=true adopt=Ok(1) left=false escape: stop=Ok(()) direct child alive=false stop: containment=cgroup-rootless survivors=[] leaves=[] crash: containment=cgroup-rootless survivors=[] leaves=[] leftover: containment=cgroup-rootless survivors=[] leaf left=false status: kind=cgroup-rootless text=each Traefik process runs in its own cgroup, so stopping it also stops what it forks; a compromised Traefik running as your user could still move out of it tag: proxy-fd03-45035-0 root cgroup: the infraternal agent runs in the root cgroup ``` **Root, compute01** (`systemd-run -p Delegate=yes -p KillMode=process`): `9 passed; 0 failed` ``` nested: made=false stop=Ok(()) leaf left=false (rootful mkdir refused, as J172 found) nested: leftover made=true adopt=Ok(1) left=false escape: stop=Ok(()) direct child alive=false stop: containment=cgroup survivors=[] leaves=[] crash: containment=cgroup survivors=[] leaves=[] leftover: containment=cgroup survivors=[] leaf left=false status: kind=cgroup text=each Traefik process runs in its own cgroup; stopping it stops everything it started tag: proxy-22-3390df-0 root cgroup: the infraternal agent runs in the root cgroup ``` **Root, undelegated unit (P3-4 fallback path):** `leftover: containment=fallback survivors=[] leaf left=false`, `1 passed`. I didn't repeat red: J164's red runs (77584) show each test failing against the code before the fixes. **Cleanup:** 0 `proxy-*` cgroups on either VM, and no test processes left. I removed the scratch targets, the test binaries and the `~/worktrees/j177` worktrees. Logs remain at dev-02 `~/j177/user-run.log` and compute01 `/root/j177/root-run.log`. The branch is unchanged. **Gate:** I changed no code in this review, so the gate doesn't apply. J164 still owes `verify fast` once it has rebased onto dev after J168 merges (see 77584).

kayg commented 2026-11-04 11:49:51 +00:00

Author

Owner

Copy link

J164 gate: scripts/verify fast on compute01, green

Head e7030b6991d3352f1d9be93c85bbce41ffb7f8d5: proxy-cgroup-j164 rebased onto origin/dev, which includes J168 (4fd5adf0c). The 11 commits replayed cleanly, with no conflicts. Pushed with a lease on bc6e5880a. Not merged, not deployed. J177 is re-reviewing the six fixes.

Log: compute01 ~/j164-fast2.log. Excerpt, verbatim:

PT_INTERP: absent; size: 23398192 bytes; baseline 22418256; delta +979936 bytes (report only, D243 amended 2026-11-01)
PT_INTERP: absent; size: 21269152 bytes; baseline 20453280; delta +815872 bytes (report only, D243 amended 2026-11-01)
SKIP key memory: aarch64-unknown-linux-musl (host x86_64; native execution required)
x86_64-unknown-linux-musl: +102400 bytes accepted with delta; zero/negative deltas reported; missing/wrong/dynamic/PT_INTERP/inspection defects rejected PASS
aarch64-unknown-linux-musl: +102400 bytes accepted with delta; zero/negative deltas reported; missing/wrong/dynamic/PT_INTERP/inspection defects rejected PASS
unsupported target: rejected PASS
(static builds: 839s)

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
...
 48 pass
 0 fail
 77248 expect() calls
Ran 48 tests across 12 files. [11.69s]
VERIFY_EXIT=0

Every row passes, including test. The remote_client failures from the earlier runs are gone with J168.

Size: the gate's +979936 (x86) and +815872 (aarch64) are against its stale baseline, not this slice. My measurement of this slice against its base was +19,232 B on x86, at 31fabf153. The J172 fixes add a few hundred lines of logic and no dependency; I didn't build dev statically again to remeasure.

## J164 gate: `scripts/verify fast` on compute01, **green** Head `e7030b6991d3352f1d9be93c85bbce41ffb7f8d5`: `proxy-cgroup-j164` rebased onto `origin/dev`, which includes J168 (`4fd5adf0c`). The 11 commits replayed cleanly, with no conflicts. Pushed with a lease on `bc6e5880a`. Not merged, not deployed. J177 is re-reviewing the six fixes. Log: compute01 `~/j164-fast2.log`. Excerpt, verbatim: ``` PT_INTERP: absent; size: 23398192 bytes; baseline 22418256; delta +979936 bytes (report only, D243 amended 2026-11-01) PT_INTERP: absent; size: 21269152 bytes; baseline 20453280; delta +815872 bytes (report only, D243 amended 2026-11-01) SKIP key memory: aarch64-unknown-linux-musl (host x86_64; native execution required) x86_64-unknown-linux-musl: +102400 bytes accepted with delta; zero/negative deltas reported; missing/wrong/dynamic/PT_INTERP/inspection defects rejected PASS aarch64-unknown-linux-musl: +102400 bytes accepted with delta; zero/negative deltas reported; missing/wrong/dynamic/PT_INTERP/inspection defects rejected PASS unsupported target: rejected PASS (static builds: 839s) +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | static builds | PASS | +---------------------------+--------------+ ... 48 pass 0 fail 77248 expect() calls Ran 48 tests across 12 files. [11.69s] VERIFY_EXIT=0 ``` Every row passes, including `test`. The `remote_client` failures from the earlier runs are gone with J168. **Size:** the gate's `+979936` (x86) and `+815872` (aarch64) are against its stale baseline, not this slice. My measurement of this slice against its base was +19,232 B on x86, at `31fabf153`. The J172 fixes add a few hundred lines of logic and no dependency; I didn't build dev statically again to remeasure.

kayg referenced this issue from a commit 2026-11-04 11:51:09 +00:00

mesh: run Linux nodes on the GotaTun fork by default (#260, D163)

kayg referenced this issue 2026-11-04 12:52:56 +00:00

macOS mesh node #265

kayg referenced this issue 2026-11-04 13:11:03 +00:00

Machine trust: self-renewal and 7-day key rotation #285

kayg referenced this issue 2026-11-04 15:32:09 +00:00

Add-host: a failed install leaves a bootstrap lock that blocks every re-add forever #305

kayg referenced this issue 2026-11-05 00:01:51 +00:00

Engineering practices proposal #344

kayg referenced this issue 2026-11-07 10:43:32 +00:00

static binary +5.4 MB since 2026-11-02: attribution and trims #413

kayg referenced this issue 2026-11-08 18:53:25 +00:00

DECISIONS consistency audit 2026-11-04 #341

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

j470b-server-mesh-rejoin

mesh-compat-382

j444b-d381-key-wins-flood

j461b-macos-utun-reader-thread

j475b-remove-host-motion

fix-reconnect-new-link

j481-d380

dev

j479-mesh-soak

fix/j476-host-facts

j475-motion-pill-split

j470-mac-node-join

root-upgrade-d376

j474-review-merge

latency-targets-120hz

nightly-suite-shards

gotatun-pin-j469

macos-utun-reader-thread

fix-flood-echo-stall

wip/j416

motion-fast-ease-out

mesh-addhost-j259

takeover-close-terminals-d368

kbd-instant-followups

fix-trust-key-double-click

nightly-suite-partial-report

appwin-icons-j207

rusqlite-slice1

swipe-content-first-d374

mesh-mtu-381

addhost-instant-open

glass-contrast-d365-mock

glass-contrast-d365

signout-followups-442

fix-startup-toast-red

research-gotatun-matrix-j317

themes-sync-j304

fix-load-flakes

fix-nix-zig-patch

fix-compute01-browser-baseline

ratchet-size-baseline-419

predict-auto-d130

revoke-coverage-d367

nightly-suite-queue-timeout

nightly-browser-suite

fix-rustsec-2026-0459

fonts-ioskeley-obel

log-retention-d375

fix-kbd-no-anim-427

fix-dev-red-specs-2

takeover-proof-305-j205

wip/j410-server-pump

ui-consistency-d372

fix-osc52-clipboard

libghostty-vt-1.0

forget-retired-host-d369

j404-merge-426

fix-setup-lock-race

fix-touch-select-426

revoke-access-report-ui

fix-predictive-echo-specs

fix-selinux-agent-exec

wip/j395-checkpoint2

wip/j395

j391-review-scratch

ui-consistency-j195-deferred-rebased

fix-rustc-paths-421

decisions-wording-341

fix-install-suite-red

fix-dev-rtt-specs

fix-pending-removal-align

fix-privileged-skip

fix-install-suite-404

review-j368

mesh-dns-j254

revoke-access-report-d356b

revoke-privilege-round5

fix-removal-confirm

review-j360

gotatun-rotation-367

deps-refresh-340

review-j354

review-J351

review-j350

addhost-takeover-d359

revoke-report-items-347

revoke-end-sessions-d362

review-j348

size-mesh-ssh-346

size-trim-337

revoke-access-diff-d356

j259-pre-rebase-b06c5a6

j305-ended-at-start-spec

addhost-sudo-consent

fix-layout-real-specs-411

fix-shell-open-411

fix-console-hosts-373

fix-shell-exit

fuzz-host-name-353

fix-host-details-410

macos-filter-j197

research-gotatun-tricks

revoke-p3-j216l

stale-state-pill

revoke-followups-j216k

revoke-progress-j216e

backup/macos01-20261106/lume/j197

backup/macos01-20261106/lume/appwin-icons-j207-stash0

backup/macos01-20261106/claude/stash-0

backup/macos01-20261106/claude/stash-1

backup/macos01-20261106/claude/J80-macos-s1

backup/macos01-20261106/claude/J80

backup/macos01-20261106/claude/J80-slice1-20261102

backup/macos01-20261106/claude/J16b-appwin-on-dev

backup/macos01-20261106/claude/appwin-launch-idle-146

backup/macos01-20261106/claude/J80-s1-run

wip/j259

trust-285-p3c

runtime-lock-j216f

mesh-ssh-p3-j175

tun-proof-p3b

trust-285-p3

terminal-polish-j271

tun-proof-p3

ssh-install-flakes-j216i

fuzz-fixes-j264

trust-285-a

trust-285-b

restart-oversized-fixture-j198

backup/macos01-20261106/lume/appwin-icons-j207

restart-bound-j216h

up-preflight-p3-j198

backup/macos01-20261106/lume/j251

up-preflight-j198

wip/up-preflight-j198-j245-p2

wip/up-preflight-j198-j245-p1

agent-fsync-offloop-j196

wip/up-preflight-j198-j245

settings-offscreen-j231

wip/up-preflight-j198-j226-inspection

wip/j103f

themes-j206

wip/up-preflight-j198-j226

flakes-j216c

backup/macos01-20261106/claude/j197

flakes-j216b

backup/macos01-20261106/claude/j197-red

wip/up-preflight-j198-review

flakes-j216

backup/macos01-20261106/claude/appwin-icons-j207

engineering-j208b

appwin-mock-j209

toast-stack-j165

engineering-j208

decisions-d351

decisions-d350

decisions-d349

appearance-presets-mock-j129

decisions-d348

docs-audit-spec

doc-coverage-baseline-raise

decisions-d347

decisions-d346b

decisions-d346

j88h-profile

ui-consistency-j195-deferred

ui-consistency-j195

engineering-practices-j194

preferred-login-ui-j166

signin-pools-263-j174

macos-node-265-s1

decisions-d345

decisions-d344

proxy-cgroup-j164

verify-quick-fuzz-329

proxy-start-io-j168

timing-flakes-j163

up-263-s1

mcp-events-j147

verify-quick-fuzz-j158

proxy-p3-j159

agent-resource-j157-detect

wip/j103d-final

remote-cancel-hang-j155

proxy-traefik-281-j62d

j149-verify-quick

proxy-traefik-281-j62b

j150-drop-unnamed

wip/j79d-red-r4

wip/j103d-d332

rootless-toast-j142

wip/j103d-head

wip/j103d

j125c-mcp

macos-build-j136

backup/macos01-20261106/claude/j136

wip/j79d-red-cli

wip/j79d-red-server

j121-link-liveness

j118c-tracer-failures

wip/j79c-head

release-desktop-j131

j124c-signed-macos

wip/j79c-red-p3

wip/j79c-red-p23

wip/j79c-red-p22

merge-j128-check

perf-deferred-j128

sidebar-toggle-260

j127-shortcut-changed-dot

j125-mcp

j118-tracer-failures

j126-account-summary

settings-sections-mock-j92

j120-pinch-lock

sidebar-292

backup/macos01-20261106/claude/j120

taste-2026-11-03

row-actions-242-215

backup/macos01-20261106/claude/stash-0-untracked

backup/macos01-20261106/claude/J80-265-s1-run

backup/macos01-20261106/claude/J80-macos-s1-20261102

j112-canvas-commands

ui-no-overshoot-scroll

fix-recovery-restart

backup/macos01-20261106/claude/stash-1-untracked

taste-2026-11-02

size-baseline-ratchet

machine-cards-structure-j108

j109-boot-copy-mockups

fix-dev-red

wip/up-263-s1

j95b-settings-rows

proxy-traefik-281

j95-option-rows

docs-human-j91

docs-keys-palette

fix-flood-1013-270-j38d

mesh-ssh-s1

mesh-ssh-s0

addhost-d291-tunnel

test-stability-load

mesh-relay-220

repro-296

shim-thp-off-24

mesh-pins-258-v2

mesh-dns

signin-d311-merged

docs-pages

ui-size-283

fix-ui-batch-specs

wip/j12b-edge-update

gate-on-arm

fix-rename-sync-293

trust-285-j60b

fix-focus-ring-clip

integ-ui-1101

edge-update-v2

signin-d311

taste-2026-11-01

layout-per-tab-250

ui-approved-splitopt-224

ui-approved-splitopt-224-on-dev

icon-text-centre-282

ui-approved-189-212-216

research-reverse-proxy

transfer-budget-320

review-mesh-pins-258-r2

sidebar-pill-259

review-mesh-relay-220

wip/j12-edge-update-v2

nix-package

status-crash-loop-237

fix-dev-test-failures

fix-fixture-socket-path

wip/mesh-relay-220-pre-dev-advance-rebase

fix-canvas-side-clip

research-head-to-head

frosted-real-256

review-mesh-pins-258

wip/mesh-relay-220-pre-final-rebase

review-layout-per-tab-250-r2

broad-suite-triage

nix-devshell

review-status-crash-loop-237-r3

wip/nix-devshell-pre-p3

taste-frosted-real

wip/code-serve-demo-2026-10-30

wip/code-serve-glass-2026-10-30

wip/code-serve-clusters-2026-10-30

wip/code-merge-release94-dev-2026-10-30

wip/rproxy-delta-scratch-2026-10-30

wip/stale-note-257-staged-2026-10-30

wip/d241-size-staged-2026-10-30

review-broad-suite-207

wip/mesh-pins-258-pre-split

wip/mesh-relay-220-pre-rebase

mesh-measure-220

plan-211-update

review-edge-autoupdate-r3

perf24-reconnect

review-stale-note-r2

release-prod-live

d241-shell-marker

review-mesh-pins-server-258

review-revocation-r3

review-setup-token-r2

review-sync-journal-r3

wip/review-mesh-pins-258-prune-review-1995fee46

frosted-scroll-256

sync-journal-r3

revocation-per-account-wip

review-tokio-asyncfd-0386

review-sync-p3s

wip/stash-8by64-stash0-2026-10-30

mesh-drag-r2

setup-token-data-wip

release-9-4

auth-busy-1013

mesh-view-overlap

sync-p3s

review-auth-busy-wake

review-readd-r8-wip

mesh-drag-bounds

audit-auth-93

app-windows-stream-start

review-authority-cap-r3

app-windows-r7-measure

readd-minimal-r6

readd-minimal-r5

app-windows-r7-soak

readd-minimal-r4

app-windows-r7

app-windows-r6

readd-minimal-r3

readd-minimal-r2

readd-minimal

merge-release8-dev

review-busy-page-liveness-r2

review-busy-page-liveness

mesh-view-trim

review-authority-read-retry-r2

review-remove-uninstall-r2

release-retry-reattach

release-retry

review-remove-uninstall

review-authority-read-retry

app-windows-ship

mesh-autoswitch-p2

mesh-ssh

mesh-relay

mesh-macos-node

screen-and-control

session-labels

review-real-logins-r5

review-real-logins-r4

review-real-logins-r3

shim-memory-r2

review-real-logins-r2

shim-memory

shadow-addhost-instant

shadow-real-logins

shadow-same-session-flood

shadow-prod-perf-fix

hosts-from-server

app-windows-real

app-windows-transport

release-prod-6w

mesh-policy-verify

quiet-close-toast

dev-merge-r4

review-mesh-agent-transport

review-app-windows-transport

settings-roundtrip

review-release-pipeline-r4

mesh-agent-transport

ended-strip-fix

review-audit-ca-update-d244

settings-rows

phone-width

signout-clear

fix-restart-tests

dev-merge-rp

signer-hardening

review-audit-ca-update

terminal-fidelity

review-audit-ca-update-wip

mesh-integrate

review-mesh-agent-transport-515d3712

review-mesh-dataplane-feec1988

mesh-dataplane

signout

research-canvas-node

wip/review-renderer-cost

wip/review-terminal-fidelity

padlock-glyph

review-release-pipeline-2e95b689

phone-polish

sidebar-polish

inbox-generic

review-mesh-dataplane-378d4055

prod-batch

ui-inbox

ui-ca-unlock

ui-machine-sections

review-mesh-dataplane-a5f744fb

tls-session-prod

session-bounds-prod

fix-dead-session

session-bounds

tls-session

palette-polish

shortcuts-d222

research-appmeta

phone-gutter

ui-centre-button

wip/ui-centre-button

wip/tls-session

review-addhost-breakit

ui-platform

renderer-cost

remote-agents

mesh-api

ssh-bootstrap

prod-hotfix2

prod-fixes

review-remote-agents

review-mesh-dataplane

session-life

research-browser

review-mesh-api

research-capture

review-mesh-api-round3

review-mesh-api-round1

gate-stability

review-gate-stability

review-relay-direct

review-mesh-api-findings

proto-sessions

font-picker

proto-addhost

proto-windows

mesh-core

latency-gate

research-bsd

research-discovery

keytray-mods

mock-core

ui-tweaks

fix-motion-sel

fix-motion

integrate2

fix-robust

t-web

t-server

vt-ghostty

fix-snap-echo

t-web-auth

review-vt2

t-agent

lane-protocol

lane-shim

lane-auth

s1-base

s1-contracts

s1-workspace

No results found.

Labels

Clear labels   

launch-qa

Findings from launch QA.

No labels

launch-qa

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kayg/infraternal#281

No description provided.