Production add-host fails at Connect on hosts whose SSH is the mesh client's SSH server #195
Closed
opened 2026-10-12 18:11:21 +00:00 by kayg
·
57 comments
No Branch/Tag specified
main
j470b-server-mesh-rejoin
mesh-compat-382
j444b-d381-key-wins-flood
j461b-macos-utun-reader-thread
j475b-remove-host-motion
fix-reconnect-new-link
j481-d380
dev
j479-mesh-soak
fix/j476-host-facts
j475-motion-pill-split
j470-mac-node-join
root-upgrade-d376
j474-review-merge
latency-targets-120hz
nightly-suite-shards
gotatun-pin-j469
macos-utun-reader-thread
fix-flood-echo-stall
wip/j416
motion-fast-ease-out
mesh-addhost-j259
takeover-close-terminals-d368
kbd-instant-followups
fix-trust-key-double-click
nightly-suite-partial-report
appwin-icons-j207
rusqlite-slice1
swipe-content-first-d374
mesh-mtu-381
addhost-instant-open
glass-contrast-d365-mock
glass-contrast-d365
signout-followups-442
fix-startup-toast-red
research-gotatun-matrix-j317
themes-sync-j304
fix-load-flakes
fix-nix-zig-patch
fix-compute01-browser-baseline
ratchet-size-baseline-419
predict-auto-d130
revoke-coverage-d367
nightly-suite-queue-timeout
nightly-browser-suite
fix-rustsec-2026-0459
fonts-ioskeley-obel
log-retention-d375
fix-kbd-no-anim-427
fix-dev-red-specs-2
takeover-proof-305-j205
wip/j410-server-pump
ui-consistency-d372
fix-osc52-clipboard
libghostty-vt-1.0
forget-retired-host-d369
j404-merge-426
fix-setup-lock-race
fix-touch-select-426
revoke-access-report-ui
fix-predictive-echo-specs
fix-selinux-agent-exec
wip/j395-checkpoint2
wip/j395
j391-review-scratch
ui-consistency-j195-deferred-rebased
fix-rustc-paths-421
decisions-wording-341
fix-install-suite-red
fix-dev-rtt-specs
fix-pending-removal-align
fix-privileged-skip
fix-install-suite-404
review-j368
mesh-dns-j254
revoke-access-report-d356b
revoke-privilege-round5
fix-removal-confirm
review-j360
gotatun-rotation-367
deps-refresh-340
review-j354
review-J351
review-j350
addhost-takeover-d359
revoke-report-items-347
revoke-end-sessions-d362
review-j348
size-mesh-ssh-346
size-trim-337
revoke-access-diff-d356
j259-pre-rebase-b06c5a6
j305-ended-at-start-spec
addhost-sudo-consent
fix-layout-real-specs-411
fix-shell-open-411
fix-console-hosts-373
fix-shell-exit
fuzz-host-name-353
fix-host-details-410
macos-filter-j197
research-gotatun-tricks
revoke-p3-j216l
stale-state-pill
revoke-followups-j216k
revoke-progress-j216e
backup/macos01-20261106/lume/j197
backup/macos01-20261106/lume/appwin-icons-j207-stash0
backup/macos01-20261106/claude/stash-0
backup/macos01-20261106/claude/stash-1
backup/macos01-20261106/claude/J80-macos-s1
backup/macos01-20261106/claude/J80
backup/macos01-20261106/claude/J80-slice1-20261102
backup/macos01-20261106/claude/J16b-appwin-on-dev
backup/macos01-20261106/claude/appwin-launch-idle-146
backup/macos01-20261106/claude/J80-s1-run
wip/j259
trust-285-p3c
runtime-lock-j216f
mesh-ssh-p3-j175
tun-proof-p3b
trust-285-p3
terminal-polish-j271
tun-proof-p3
ssh-install-flakes-j216i
fuzz-fixes-j264
trust-285-a
trust-285-b
restart-oversized-fixture-j198
backup/macos01-20261106/lume/appwin-icons-j207
restart-bound-j216h
up-preflight-p3-j198
backup/macos01-20261106/lume/j251
up-preflight-j198
wip/up-preflight-j198-j245-p2
wip/up-preflight-j198-j245-p1
agent-fsync-offloop-j196
wip/up-preflight-j198-j245
settings-offscreen-j231
wip/up-preflight-j198-j226-inspection
wip/j103f
themes-j206
wip/up-preflight-j198-j226
flakes-j216c
backup/macos01-20261106/claude/j197
flakes-j216b
backup/macos01-20261106/claude/j197-red
wip/up-preflight-j198-review
flakes-j216
backup/macos01-20261106/claude/appwin-icons-j207
engineering-j208b
appwin-mock-j209
toast-stack-j165
engineering-j208
decisions-d351
decisions-d350
decisions-d349
appearance-presets-mock-j129
decisions-d348
docs-audit-spec
doc-coverage-baseline-raise
decisions-d347
decisions-d346b
decisions-d346
j88h-profile
ui-consistency-j195-deferred
ui-consistency-j195
engineering-practices-j194
preferred-login-ui-j166
signin-pools-263-j174
macos-node-265-s1
decisions-d345
decisions-d344
proxy-cgroup-j164
verify-quick-fuzz-329
proxy-start-io-j168
timing-flakes-j163
up-263-s1
mcp-events-j147
verify-quick-fuzz-j158
proxy-p3-j159
agent-resource-j157-detect
wip/j103d-final
remote-cancel-hang-j155
proxy-traefik-281-j62d
j149-verify-quick
proxy-traefik-281-j62b
j150-drop-unnamed
wip/j79d-red-r4
wip/j103d-d332
rootless-toast-j142
wip/j103d-head
wip/j103d
j125c-mcp
macos-build-j136
backup/macos01-20261106/claude/j136
wip/j79d-red-cli
wip/j79d-red-server
j121-link-liveness
j118c-tracer-failures
wip/j79c-head
release-desktop-j131
j124c-signed-macos
wip/j79c-red-p3
wip/j79c-red-p23
wip/j79c-red-p22
merge-j128-check
perf-deferred-j128
sidebar-toggle-260
j127-shortcut-changed-dot
j125-mcp
j118-tracer-failures
j126-account-summary
settings-sections-mock-j92
j120-pinch-lock
sidebar-292
backup/macos01-20261106/claude/j120
taste-2026-11-03
row-actions-242-215
backup/macos01-20261106/claude/stash-0-untracked
backup/macos01-20261106/claude/J80-265-s1-run
backup/macos01-20261106/claude/J80-macos-s1-20261102
j112-canvas-commands
ui-no-overshoot-scroll
fix-recovery-restart
backup/macos01-20261106/claude/stash-1-untracked
taste-2026-11-02
size-baseline-ratchet
machine-cards-structure-j108
j109-boot-copy-mockups
fix-dev-red
wip/up-263-s1
j95b-settings-rows
proxy-traefik-281
j95-option-rows
docs-human-j91
docs-keys-palette
fix-flood-1013-270-j38d
mesh-ssh-s1
mesh-ssh-s0
addhost-d291-tunnel
test-stability-load
mesh-relay-220
repro-296
shim-thp-off-24
mesh-pins-258-v2
mesh-dns
signin-d311-merged
docs-pages
ui-size-283
fix-ui-batch-specs
wip/j12b-edge-update
gate-on-arm
fix-rename-sync-293
trust-285-j60b
fix-focus-ring-clip
integ-ui-1101
edge-update-v2
signin-d311
taste-2026-11-01
layout-per-tab-250
ui-approved-splitopt-224
ui-approved-splitopt-224-on-dev
icon-text-centre-282
ui-approved-189-212-216
research-reverse-proxy
transfer-budget-320
review-mesh-pins-258-r2
sidebar-pill-259
review-mesh-relay-220
wip/j12-edge-update-v2
nix-package
status-crash-loop-237
fix-dev-test-failures
fix-fixture-socket-path
wip/mesh-relay-220-pre-dev-advance-rebase
fix-canvas-side-clip
research-head-to-head
frosted-real-256
review-mesh-pins-258
wip/mesh-relay-220-pre-final-rebase
review-layout-per-tab-250-r2
broad-suite-triage
nix-devshell
review-status-crash-loop-237-r3
wip/nix-devshell-pre-p3
taste-frosted-real
wip/code-serve-demo-2026-10-30
wip/code-serve-glass-2026-10-30
wip/code-serve-clusters-2026-10-30
wip/code-merge-release94-dev-2026-10-30
wip/rproxy-delta-scratch-2026-10-30
wip/stale-note-257-staged-2026-10-30
wip/d241-size-staged-2026-10-30
review-broad-suite-207
wip/mesh-pins-258-pre-split
wip/mesh-relay-220-pre-rebase
mesh-measure-220
plan-211-update
review-edge-autoupdate-r3
perf24-reconnect
review-stale-note-r2
release-prod-live
d241-shell-marker
review-mesh-pins-server-258
review-revocation-r3
review-setup-token-r2
review-sync-journal-r3
wip/review-mesh-pins-258-prune-review-1995fee46
frosted-scroll-256
sync-journal-r3
revocation-per-account-wip
review-tokio-asyncfd-0386
review-sync-p3s
wip/stash-8by64-stash0-2026-10-30
mesh-drag-r2
setup-token-data-wip
release-9-4
auth-busy-1013
mesh-view-overlap
sync-p3s
review-auth-busy-wake
review-readd-r8-wip
mesh-drag-bounds
audit-auth-93
app-windows-stream-start
review-authority-cap-r3
app-windows-r7-measure
readd-minimal-r6
readd-minimal-r5
app-windows-r7-soak
readd-minimal-r4
app-windows-r7
app-windows-r6
readd-minimal-r3
readd-minimal-r2
readd-minimal
merge-release8-dev
review-busy-page-liveness-r2
review-busy-page-liveness
mesh-view-trim
review-authority-read-retry-r2
review-remove-uninstall-r2
release-retry-reattach
release-retry
review-remove-uninstall
review-authority-read-retry
app-windows-ship
mesh-autoswitch-p2
mesh-ssh
mesh-relay
mesh-macos-node
screen-and-control
session-labels
review-real-logins-r5
review-real-logins-r4
review-real-logins-r3
shim-memory-r2
review-real-logins-r2
shim-memory
shadow-addhost-instant
shadow-real-logins
shadow-same-session-flood
shadow-prod-perf-fix
hosts-from-server
app-windows-real
app-windows-transport
release-prod-6w
mesh-policy-verify
quiet-close-toast
dev-merge-r4
review-mesh-agent-transport
review-app-windows-transport
settings-roundtrip
review-release-pipeline-r4
mesh-agent-transport
ended-strip-fix
review-audit-ca-update-d244
settings-rows
phone-width
signout-clear
fix-restart-tests
dev-merge-rp
signer-hardening
review-audit-ca-update
terminal-fidelity
review-audit-ca-update-wip
mesh-integrate
review-mesh-agent-transport-515d3712
review-mesh-dataplane-feec1988
mesh-dataplane
signout
research-canvas-node
wip/review-renderer-cost
wip/review-terminal-fidelity
padlock-glyph
review-release-pipeline-2e95b689
phone-polish
sidebar-polish
inbox-generic
review-mesh-dataplane-378d4055
prod-batch
ui-inbox
ui-ca-unlock
ui-machine-sections
review-mesh-dataplane-a5f744fb
tls-session-prod
session-bounds-prod
fix-dead-session
session-bounds
tls-session
palette-polish
shortcuts-d222
research-appmeta
phone-gutter
ui-centre-button
wip/ui-centre-button
wip/tls-session
review-addhost-breakit
ui-platform
renderer-cost
remote-agents
mesh-api
ssh-bootstrap
prod-hotfix2
prod-fixes
review-remote-agents
review-mesh-dataplane
session-life
research-browser
review-mesh-api
research-capture
review-mesh-api-round3
review-mesh-api-round1
gate-stability
review-gate-stability
review-relay-direct
review-mesh-api-findings
proto-sessions
font-picker
proto-addhost
proto-windows
mesh-core
latency-gate
research-bsd
research-discovery
keytray-mods
mock-core
ui-tweaks
fix-motion-sel
fix-motion
integrate2
fix-robust
t-web
t-server
vt-ghostty
fix-snap-echo
t-web-auth
review-vt2
t-agent
lane-protocol
lane-shim
lane-auth
s1-base
s1-contracts
s1-workspace
No results found.
No labels
launch-qa
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set
Reference
kayg/infraternal#195
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Owner 2026-10-12 23:35 IST: adding 10.69.69.52 (and o1, o2, minipc01, .217, .155) from production fails at "Connect to the host" ("Connecting to the host failed… / The connection to the host dropped…"), while
ssh kayg@10.69.69.52from monitor01 works.Evidence:
step="connect" exit_status=unavailable(rootless and root).ssh -vshows remoteNetBird-SSH-Server-0.79.0,Authenticated … using "none".5bf3f42e(add-host rounds 2–3 deployed 17:38 IST).Also UX: the failure card shows two contradictory sentences (connect failed + connection dropped) and the same text again in Details.
Job: addhost-netbird-connect.
Start #195: branch addhost-netbird-connect, base/head
5bf3f42ede(release-prod-5). Read issue body and all comments (0); no parent referenced. Reproduce isolated connect/check; retain strict host-key verification; no deployment or host writes.Orchestrator note (owner reminder 2026-10-12): network auth is DECIDED, not optional — D22 (network identity: NetBird/Tailscale, no secret stored), D116 (first cut: NetBird SSH, Tailscale SSH, passwords, keys) and the add-host decision (DECISIONS.md ~line 1288: add-host detects automatically whether network auth or local auth applies; local auth order = server key → passkey-encrypted key → one-time password; choice asked only as a last resort). The fix must restore automatic network-auth detection for NetBird SSH ("none" method) and Tailscale SSH, with a test for each, and show why it regressed (the test that should have caught it). The owner tested this flow on infraternal-dev earlier today and got past Connect, so compare release-prod-3/4 against release-prod-5.
Root cause #195: release-prod-5 Bootstrap::run passes previous_attempt::check_script(mode) (or CHECK_SCRIPT + INFRATERNAL_SETUP_CLEAR) as its first command, but auth_order::Pending::run admits only exact CHECK_SCRIPT. It returns "SSH authentication requires the read-only system check." locally, before opening TCP; .at(Connect) replaces this evidence with generic connecting text, exit_status remains None. NetBird none auth is supported: real pinned gliderlabs v0.3.8/x-crypto v0.55.0 loopback passes network connect/check, same-connection command and changed-key refusal. Trace confirms using "none". Production-4 passed CHECK_SCRIPT unchanged; production-5 introduced the composite ownership check. Proposed additive contract clarification: permit only the exact generated read-only ownership check (root/rootless and the cleanup-owned clear record) in the initial automatic-auth allowlist; repeat this fixed read-only check on authentication fallback, never arbitrary scripts or input. Keeps the intended one SSH command/round trip and D182 trust unchanged.
Completed dependency step:
4ad3562epins test-only gliderlabs/ssh v0.3.8 and x/crypto v0.55.0 from NetBird v0.79.0. BSD-style licenses compatible with AGPL; no Cargo/production dependency added. go mod verify: all modules verified; cargo fmt --all --check passed. Local functional loopback trace demonstrates none authentication; no local performance claim.Completed fixture and contract steps:
90de0e40runs a loopback-only real SSH engine with in-memory synthetic Ed25519 key and disposable HOME; go test ./... passes.bebca047records the exact read-only allowlist clarification proposed earlier. No production library, wire category or host-key confirmation changes. First bad commit identified by git log -S previous_attempt::check_script:436dfb97; its unchanged Pending adapter rejects the new first command before TCP.#195 progress: loopback real-engine regression failed on release-5's exact initial ownership command with "SSH authentication requires the read-only system check." and no exit status; it passes with the exact-script allowlist. First bad commit
436dfb97. SSH crate clippy and functional tests pass; UI copy regression failed before the edit and passes afterward. Full static gate is running. Host load is high (load average 55.94/47.55/33.16 on 8 cores); initial unprioritized dispatcher stopped at its unchanged latency self-test. A CPUWeight=10000 scope passed that row and is continuing. Local test durations are not performance evidence.Gate finding: the prioritized full dispatcher reaches workspace clippy but fails on the production base's crates/infraternal-agent/src/daemon.rs:1139-1140: readiness_exit_tests is included without #[cfg(test)], unlike the adjacent test modules; strict non-test compilation reports unused import super::* in readiness_exit_tests.rs:2. This module is unchanged from
5bf3f42e. Add the missing test-only guard as its own atomic gate repair; no runtime behavior changes. Full gate continues through the remaining rows. Fixc8c2fb5badmits only generated read-only checks; SSH crate clippy/tests passed and real-engine functional counts are check_variants=3, physical_connections=3, later_credentials_requested=0, changed_key_refused=1.Completed authentication fix
c8c2fb5band regression tests39726289. Exact read-only allowlist preserves a single initial check command; arbitrary appended writes remain refused. Real pinned loopback engine: check_variants=3, physical_connections=3, later_credentials_requested=0, changed_key_refused=1; production-4 sniff and both production-5 rootless/owned-cleanup commands pass. Controlled root/rootless fallback tests: 7 passed, 0 failed. Strict SSH crate clippy and fmt pass. No timing or throughput claim from the build host.Source-contract git bisect completed between
3aec316e(release-prod-4) and5bf3f42e(release-prod-5): first bad commit436dfb97(see evidence/addhost-netbird-connect/bisect.txt for full SHA). Predicate checks when Bootstrap first sends the composite ownership sniff while Pending still requires exact CHECK_SCRIPT; the real-engine before/after test independently reproduces that rejected command. New failure-card cases pass at 1440x900 and 393x852 on a production build; distinct Details cause and no claimed drop. Full browser run 30 passed, 2 credential-flow tests queued/timed out; focused rerun passes all 4 (the 2 failures plus both new cases). Screenshots: evidence/addhost-netbird-connect/connect-failure-{1440,393}.png.Static gate finding: x86_64 musl built successfully and is static (no PT_INTERP), but is 20,333,104 bytes against the unchanged 16,404,816-byte cap. The production base already records this failure in evidence/addhost-round3/final-fast.log and static-size.md (20,132,112-byte x86 build against the same cap); no budget or decision will be changed in this job. Building an isolated exact
5bf3f42eartifact for a direct size comparison. Final head gate will be reported verbatim, including this blocker if it remains. Earlier milestone also failed the unchanged readiness ownership-EOF test; targeted rerun and final workspace run are pending. New SSH/UI tests remain green.Resume #195 after disk-full interruption: branch addhost-netbird-connect, original base
5bf3f42ede, head3c1b8482. Read all 10 issue comments; no parent referenced. Existing commits and pending source files are intact. Resume diagnostics WARN, empty-Details regression and test-only clippy guard as separate steps. Reuse n195/target; rerun final fast gate and report existing budget failures without changing budgets. No deployment or remote host writes.Gate repair finding #195: readiness_exit_tests fails deterministically at src/readiness_exit_tests.rs:87 because it expects Output immediately after the initial InputAck, but daemon.rs emits terminal.replay_complete for Hello.head=0 before live output. The existing regression already proves the relay waits for credit; its queue assertion omits the ordered replay event. Consume and verify that event before asserting the output drain. Keep this test-only repair with the missing cfg(test) guard; no daemon runtime behavior changes. Evidence: resume-agent-tests.txt (before), resume-agent-after.txt (after).
Completed #195 resume steps:
a1a455c9adds NetBird, Tailscale and absent provider metadata to the real none-auth regression: 3 scripts x 3 provider cases = 9 physical connections, 0 later credential requests, 1 changed-pin refusal. Fmt and strict SSH clippy pass. This covers Tailscale metadata over the shared SSH authentication path; it does not claim a deployed Tailscale-server integration run.acff8b95adds WARN failure_kind; its sentinel regression fails without the field (warn-before.txt) and both failure-log tests pass with it (warn-after.txt), with no host output or private sentinel in the WARN. Strict server clippy passes.592f3cafrepairs test-only compilation and the readiness fixture's missing replay-complete assertion. The original output-drain assertion passes; no daemon runtime behavior changes. Strict agent clippy passes. Evidence: resume-agent-tests.txt, resume-agent-after.txt, resume-native-clippy-after.txt.Completed UI steps #195:
8d949ac4replaces the empty Details fallback with a factual no-additional-evidence sentence.c81a56b4retains the failing-before regression and scopes the Connect assertion to the failure page, since transitions temporarily retain outgoing progress copy. bun run check and all 8 copy tests pass; focused production browser cases 3 passed (desktop, phone, empty Details), screenshots at connect-failure-1440.png and connect-failure-393.png. Local harness durations are not performance evidence. Full functional wizard rerun and final fast gate pending.Full functional wizard rerun #195: all 33 production-build Playwright cases pass with unique TRACER_TEST_PORT=4395, under the job scope. Includes Tailscale network-access display, all local credential fallbacks, strict host-key confirmation, phone flow, leftovers recovery, both Connect cards and empty Details. No performance result claimed from build-host test durations. Screenshots refreshed at evidence/addhost-netbird-connect/connect-failure-{1440,393}.png; unrelated #169 screenshot generated by the suite restored from git. All test-server children retired by Playwright.
Direct baseline #195: exact unmodified release-prod-5
5bf3f42edebuilds x86_64 musl successfully after restoring two absent files in this job's checksum-verified VT source cache. Static stripped artifact = 20,329,648 bytes, already 3,924,832 bytes over the frozen 16,404,816-byte cap. No size budget changed. Evidence: baseline-build-after-cache-repair.txt. Current head's complete two-architecture gate is pending; the shared target directory is reused. One unchanged CA unlock audit unit test has reported a failure in the workspace run; awaiting its diagnostic and running it alone before drawing a conclusion.Workspace finding #195: unchanged CA unlock audit test fails only its log capture assertion (ca_unlock_audit_tests.rs:97, 0 events captured vs 4 expected), not the unlock/refusal assertions. Workspace server result: 325 passed, 1 failed, 39 ignored. The identical test executable passes when the audit test runs alone (audit-rerun.txt). No audit/auth runtime source changes in this lane. Final branch gate will run with RUST_TEST_THREADS=1 to remove overlapping tracing subscribers; no assertion, timeout or authentication code will be weakened. The parallel-run failure remains recorded in verify-fast-resume.txt.
Completed evidence snapshot
e29845a2c1: README/review index, before/after regressions, source-contract bisect, nine successful real SSH checks, strict changed-pin refusal, 33 passing production wizard cases and both screenshots. Both static builds succeed; x86=20,333,168 bytes vs exact production base=20,329,648 (+3,520), aarch64=18,112,240 bytes; frozen caps unchanged. Raw engine/gate logs retain their original whitespace. Final serialized scripts/verify fast is running on this exact head; TS gate also reruns because the dispatcher exits before TS on Rust failure.Finish #195: head
e29845a2c1, branch addhost-netbird-connect, pushed successfully to origin/addhost-netbird-connect. Final head gate ran with RUST_TEST_THREADS=1 in the job scope; no assertions or budgets changed.Workspace: 740 passed, 0 failed, 78 ignored as configured in the repository. The formerly failing audit capture test passes. Separate final TS/production build/transfer/protocol gate passes (protocol: 36 passed, 0 failed, 34,462 assertions). Real pinned SSH regression: 9 successful checks/9 connections, 0 later credential requests, changed pin refused. Production wizard: 33 passed; screenshots and complete author checklist in evidence/addhost-netbird-connect/.
scripts/verify fast exits 1 solely on the unchanged static-size caps. Exact production x86 base already exceeds its cap: 20,329,648 bytes; fixed x86 adds 3,520 bytes. Both shipping artifacts build static; no budget relaxed. Gate output verbatim:
Review: evidence/addhost-netbird-connect/README.md and review.md. No deployment, bootstrap upload/install or lab configuration writes; only the requested repository push and issue comments. Issue remains open.
Cleanup complete #195: head
e29845a2c1matches origin/addhost-netbird-connect; worktree clean. Removed this job's n195 scratch/build caches and detached baseline. No job-owned SSH fixture remained. Final gate summary and results are in the preceding finish comment; no deployment.Independent review start #195: local branch review-addhost-netbird-connect at
e29845a2c1; merge base with dev93b00f5ab9. Read all 20 comments and issue body; no parent referenced. Review exact merge-base diff and distinguish inherited changes from #195. No push, deployment, live fixture changes or perf VM use. Functional checks only; local gate timings are not performance evidence.Independent verification at
e29845a2: bun check passes; 45 focused unit tests pass (136 assertions); real none-auth fixture passes 9 checks over 9 physical connections, requests 0 later credentials, and refuses 1 changed pin without changing it. All 15 real-filesystem/listener ownership recovery tests pass. Production Connect cards pass at 1440x900 and 393x852; empty Details and 3 initial browser failures pass on focused rerun. Initial unprioritized suite interrupted after repeated harness deadline failures; prioritized full functional rerun pending with 120s test timeout, unchanged assertions. Local durations are not performance evidence. No confirmed finding yet; exact-head fast gate and bounded parser fuzz pending.Independent verification progress at
e29845a2: prioritized production wizard rerun passes all 33 cases (TRACER_TEST_PORT=4496, 120s test timeout, unchanged assertions). Separate TS gate passes compiler, production build, checksum/transfer checks and 36 protocol tests with 34,462 assertions. Native head gate has passed fmt and Clippy; serialized workspace tests continuing without failures. Initial coverage-enabled parser fuzz failed at linking (_sancov_gen symbols); retry uses harness-only CARGO_PROFILE_RELEASE_LTO=false and opt-level=1, without changing shipping settings. No timing or throughput claim from the build host. No confirmed finding; additional refused-input/concurrent-auth regressions prepared pending the exact-head gate.Independent finding R01 (P2), commit
a13769c685: previous_attempt.sh:42-48 matches /proc descriptor links against the raw base path. A permitted home alias misses the canonical descriptor path. The exact generated rootless check returns canonical_status=73, alias_status=0, alias_reports_stale=true while the same private setup marker inode remains open. Rust regression home_alias_keeps_open_setup_descriptor_live fails (0 passed / 1 failed); no implementation fix. This incorrectly offers Start fresh despite an open setup descriptor. P2 is fixed forward under the owner verdict rule; no P0/P1 found so far.Review test step
0df6dea709: 9 auth-order tests pass, including 20 changed/nonempty first-command refusals with 0 opens/credential requests and 64 overlapping valid checks across all 4 generated variants, with 1 selected connection per group and 0 later credential requests. The controlled transport yields with auth pending to exercise the actual selection lock without a timer. Strict Clippy for both review test targets and formatting pass. Production code/dependencies unchanged. Exacte29845a2gate: 740 passed, 0 failed, 78 ignored; both static builds succeed (x86 20,333,104 bytes, aarch64 18,112,240 bytes) but exceed existing caps. Fuzzer rerun passes 225,943 inputs using coverage+ASan, LTO off and opt-level=1; default LTO link failure retained. Final review gate will retain the intentional R01 failure; no implementation fix.Independent review finish #195: verdict merge — MERGE (no P0/P1). Findings P0=0, P1=0, P2=1, P3=0. R01 is fixed forward under the owner verdict rule.
Target
e29845a2c1; merge base with dev93b00f5ab9. Local review branch review-addhost-netbird-connect; head6fa2b32a6e. No implementation fix, push, deployment, perf VM or live-owner-fixture changes.R01 P2: previous_attempt.sh:42-48 misses an open setup descriptor through a permitted home alias. Exact production rootless check: canonical status=73; alias status=0 with a stale record. Ordinary Rust test home_alias_keeps_open_setup_descriptor_live fails and is committed in
a13769c6. Canonical control verifies descriptor visibility. No recovery mutation or data-loss claim. Commit0df6dea7adds passing security/concurrency coverage: 20 refused inputs produce 0 opens/credential requests; 64 overlapping checks select 1 connection per group, 0 later credentials. 9 auth-order tests and both test-target Clippy checks pass.Independent evidence: target native workspace 740 passed, 0 failed, 78 configured ignores; real none-auth SSH 9 successful checks/9 successful-check connections, 0 later credentials, changed pin refused; all 33 production wizard cases and 4 replay/inventory cases pass; 15 real recovery tests pass; TS gate passes with 36 protocol tests/34,462 assertions; coverage+ASan fuzz passes 225,943 inputs with harness-only LTO off/opt-level=1. Original harness failures retained. Durations are not performance evidence. Tailscale cases cover metadata over the shared fixture engine, not deployed Tailscale integration.
Both static artifacts build, but existing caps fail: x86=20,333,104 bytes; aarch64=18,112,240. Review commits change shipping size by 0 bytes; x86 SHA256 unchanged. Final review-source gate intentionally fails only R01 plus those caps; fmt, VT fmt, Clippy and cargo-deny pass. Cargo stops on R01 (295 passed, 1 failed, 21 ignored), so the complete target-head result is preserved separately. No assertions or budgets weakened.
Evidence/captures and checklist: evidence/review-addhost-netbird-connect/README.md and compressed raw logs. Findings also recorded in /home/kayg/.local/state/codex-jobs/infraternal/review-addhost-netbird-connect/findings.md.
Exact target
e29845a2scripts/verify fast exits 1; gate output verbatim:Issue remains open. Build/toolchain/corpus and short-fixture scratch cleanup follows; committed evidence remains local.
Review cleanup complete #195: local head
6fa2b32a6e, worktree clean. Removed job build/toolchain/corpus caches, read-only Go modules, generated Python caches and private short fixture root. Named test ports have no remaining listeners. Committed evidence and findings record remain; no push or deployment.DEPLOYED to production 2026-10-13 02:33 IST: release-prod-6
e29845a2(= release-prod-5 + Connect regression fix + quiet close toast? no: web from this tree). x86 sha256 2c634516 (20,333,104 B, -2.1 MB vs prod-5), arm 81c842e5; key-memory check ready under 8 MiB memlock; server active, MainPID 196841, web 200. Review: MERGE P0=0 P1=0 P2=1 (R01 forward fix queued). The restart re-locked the CA: owner unlocks in the morning, then retries add-host on 10.69.69.52 and infraternal-dev.Start #195: branch dev-merge-prod6, base
21b37bca70(origin/dev). Read all 28 comments on #195 and all 260 on parent #169. Merging origin/release-prod-live9ab4329c, retaining both histories and tests; then importing R01 and fixing forward while preserving SF-1 symlink-ancestor refusal and bound-setup-socket refusal. No deployment.Merge finding: two content conflicts in apps/tracer/src/canvas/addhost/model.ts and crates/infraternal-agent/src/readiness_exit_tests.rs. Resolved by retaining dev's arm64/x86_64 download-step guidance and replay session-id assertion, plus production's Connect failure wording; both sides' test cases/assertions remain.
git diff --checkpasses. Running targeted gates before recording the merge commit.Merge finding resolved: the first combined Connect advice violated the production regression tests (6 passed, 2 failed: invented “dropped” wording and repeated Details). Retained production's exact “Check SSH access and the host service, then try again.” advice while keeping dev's signed arm64/x86_64 download labels.
bun test test/addhost-copy.test.ts: 8 passed, 0 failed, 22 assertions. The merged readiness test also retains dev's session_id assertion in addition to production's ordered replay event checks.Completed merged SSH plugin suite: 116 passed, 0 failed, 9 ignored. The real fixture root is
/home/kayg/.cache/codex-scratch/dev-merge-prod6(no symlink),RUST_TEST_THREADS=1;fresh_runtime_symlink_never_removes_foreign_empty_directoriesandstale_previous_attempt_socket_can_be_retired_without_manual_cleanupboth pass. Initial fixture run found the credential-helper scratch threshold at 108 bytes; removing the extra/fpath component resolved it without source or assertion changes.Completed restart suite: 31 passed, 0 failed, 1 ignored. The first run's two agent_review fixtures failed pre-assertion because the
.upstreamsocket path was exactly 108 bytes. Replayed unchanged with the real compact per-job fixture directory/home/kayg/.cache/codex-scratch/dm6(96-byte longest path); both restart replay tests pass. No assertion, timeout or source changes; no #118 exemption.Completed release suites: server release_verify 4 pass/1 ignored; release_payload 9/2; release_latest 3/2; release plugin 10/3; release version API 1/0. Aggregate: 27 passed, 0 failed, 8 ignored.
scripts/release/test.py: 5/5;bun test scripts/release/review.test.ts scripts/release/worker.test.ts: 5/5, 21 assertions;scripts/release/channel_cli_test.py: 1/1. Ignored performance measurements were not run on the build host.Add-host wizard production browser suite passed:
TRACER_TEST_PORT=4537 bun x playwright test test/addhost-wizard.spec.ts— 37 passed (8.2m), 0 failed. Production screenshots:evidence/addhost-netbird-connect/connect-failure-1440.png,evidence/addhost-netbird-connect/connect-failure-393.png; wizard fixture also refreshedevidence/addhost-169/failure-details.png. Both parent test sets remain in the merged test files.Merged
origin/release-prod-liveintodev-merge-prod6inbdf68a6c. Kept both sides’ tests, including the zero-head replay event and devsession_idassertion. Gates before commit: rustfmt passed; clippy passed; add-host copy 8 passed/0 failed; SSH plugin 116 passed/0 failed/9 ignored; restart 31 passed/0 failed/1 ignored; release suites 27 passed/0 failed/8 ignored plus release scripts 11 passed; TypeScript check passed; add-host wizard 37 passed/0 failed (8.2m, production build). Screenshots are inevidence/addhost-netbird-connect/. Next: apply the reviewer’s R01 test and reproduce the alias descriptor failure before fixing it.Cherry-picked the review’s exact R01 test as
10c900e5(crates/plugins/ssh/tests/previous_attempt_liveness_review.rs, 85 lines). No recovery implementation change is included. Running the new integration test next to confirm the canonical-home control passes and the permitted-alias check incorrectly reports stale recovery.R01 red baseline is confirmed on commit
10c900e5: the production check returnedcanonical_status=73 alias_status=Some(0) alias_reports_stale=true;home_alias_keeps_open_setup_descriptor_livefailed because the alias result was 0 instead of 73. The same marker inode remained open across both calls. Existingno_redirectand socket-liveness logic are unchanged so far.R01 fixed in
52f64477.previous_attempt.shresolves the base physically only afterno_redirectsucceeds, then matches/proclinks under the lexical and physical base. The/proc/net/unixlive-socket refusal remains before descriptor scanning; symlink-ancestor refusal is unchanged. Evidence: R01 now printscanonical_status=73 alias_status=Some(73) alias_reports_stale=falseand passes;previous_attempt_shell.py16/16; SSH plugin suite 117 passed, 0 failed, 9 ignored; rustfmt and SSH clippy passed.Finished on
52f64477628a3b79413a3730c9dbd25bc6268f0a; pusheddev-merge-prod6toorigin/dev-merge-prod6.origin/devis an ancestor of the branch. R01’s red run and green fix are recorded above. The SSH plugin suite passed 117/0/9 ignored, the ownership shell suite 16/16, restart 31/0/1 ignored, release suites 27/0/8 ignored plus release scripts 11/11, add-host copy 8/8, and production add-host wizard 37/37. Rustfmt, workspace clippy, TypeScript check, cargo-deny, and the fast gate test row passed. The only fast-gate failure is the unchanged D243 static size cap; TypeScript check was run separately because the dispatcher exits after the Rust size failure.Final
scripts/verify fastoutput (verbatim):Capture note:
scripts/verify fastcompleted and wrote its full output; the post-run zsh wrapper then failed to assign the result to zsh’s read-onlystatusvariable. The final Rust table in the captured output reportsstatic builds | FAIL (1)and the two D243 overages above; the wrapper error happened after the gate and did not change its result. The TypeScript check had already passed separately.Continuation start #195: branch dev-merge-prod6 at
52f6447762; current origin/dev is5d098cb828. Read #195 and parent #169 history. Merging current dev into the existing production merge + R01 fix; checking migration order, auth/mesh/SSH/add-host/restart/release suites, then final fast gate. Preserve the D243 failure if unchanged; push only this branch. No deployment.Merged-tree finding #195: the server auth/socket suite passes on the
52f64477+ origin/dev5d098cb8tree: 33 passed, 0 failed, 2 ignored. This retains the liveness/review3 socket, mesh inventory, revocation, lifecycle and control-credit cases; no test was removed. Raw output:/home/kayg/.cache/codex-scratch/dev-merge-prod6/server-auth-sockets.log. Migration, SSH, mesh, restart/release and browser suites remain in progress.Migration check #195: the merged tree has one migration for each version 0001–0024. Incoming host revocation is version 0024; existing migrations 0001–0023 are retained. The migration compatibility suite passes 11/11, including fresh/dev/production histories, exact-checksum version-20 retagging, the version-23 retirement journal, unknown/dirty history refusal, and concurrent resume. Evidence:
/home/kayg/.cache/codex-scratch/dev-merge-prod6/server-migrations.log. Server Mesh revocation and the remaining requested suites are still running.Merged-tree native suites #195: auth sockets 33/0/2 ignored; review3 13/0/1 ignored; migration histories 11/0 and remote legacy-history 1/0; Mesh revocation unit + integration 1/0 each; Mesh plugin with agent+server features 43/0/47 ignored (the ignored cases require disposable kernel namespaces/CAP_NET_ADMIN or are measurements); SSH plugin 113/0/9 ignored; R01 1/1 and ownership shell 16/16; restart 28/0/1 ignored; release Rust 24/0/5 ignored. The first feature-gated Mesh integration invocation selected zero tests; rerun with
--features remoteran and passed. Logs are in/home/kayg/.cache/codex-scratch/dev-merge-prod6/. Add-host browser and release script suites remain.Merge and focused suites complete. Head
5e383c3d(merge of prior52f64477with current dev5d098cb8). Migration ledger is unique and contiguous 0001–0024; 0023 bootstrap retirement remains stable, mesh host revocation is 0024, and migration compatibility tests pass. No test files were deleted. Mesh author checklist reviewed; source/docs whitespace check clean (historical evidence logs retain their recorded formatting).Results on the merged tree: server auth sockets 33 passed/2 ignored; review3 13/1; migration compatibility 11 passed; remote mesh revocation and host removal 1 each passed; mesh plugin agent+server 43 passed/47 ignored (namespace/CAP_NET_ADMIN or measurement-only); SSH plugin 113 passed/9 ignored across 23 targets plus R01 1/1 and shell fixture 16/16; restart 28 passed/1 ignored; release Rust 24 passed/5 ignored; release Python 23 passed; release Bun 8 passed; add-host production browser 37/37 at TRACER_TEST_PORT=4786. Logs are in /home/kayg/.cache/codex-scratch/dev-merge-prod6 (server-auth-sockets.log, server-review3.log, server-migrations.log, server-mesh-revocation.log, mesh-plugin-agent-server.log, ssh-plugin.log, ssh-r01.log, ssh-ownership-shell.log, restart-cli.log, server-release.log, release-scripts-final.log, release-bun.log). Browser suite output ended
37 passed (7.6m); generated screenshot diffs were restored.Running branch-head scripts/verify fast and TypeScript gate now; final gate output and push status will follow.
Fast-gate finding: the first workspace test run had 95 passed, 1 failed, 14 ignored in infraternal-agent. The failure was test setup: daemon_gone_tests.rs:111 attempted a Unix socket under INFRATERAL_TEST_STATE_ROOT=/home/kayg/.cache/codex-scratch/dm6/state and the path exceeded SUN_LEN (
InvalidInput: path must be shorter than SUN_LEN). I’m rerunning with the same private scratch root shortened to /home/kayg/.cache/codex-scratch/dm6. No product source changed. Static builds continue from this run; the known D243 cap is being recorded separately.Correction to the prior gate-path note: this test derives its fixture from std::env::temp_dir() (TMPDIR), not INFRATERNAL_TEST_STATE_ROOT. The configured TMPDIR /home/kayg/.cache/codex-scratch/dev-merge-prod6/tmp has length 51; the fixture's rendezvous socket path is 124 bytes, over the Unix socket limit. I’ll rerun with private TMPDIR /home/kayg/.cache/codex-scratch/d6; the same candidate path is 107 bytes. This is test configuration only; no source change.
Static-size finding from branch-head fast gate: x86_64-unknown-linux-musl passed production-trust checks and has no PT_INTERP, but is 22,534,320 bytes against the D243 maximum 16,404,816 (baseline 16,142,672 + 262,144). Artifact: /home/kayg/.cache/codex-scratch/dev-merge-prod6/target/x86_64-unknown-linux-musl/release/infraternal. The gate is still building/checking AArch64; no size baseline or waiver was changed.
AArch64 result from the same fast run: production-trust passed and PT_INTERP is absent; binary is 19,950,152 bytes against the D243 maximum 13,817,816 (baseline 13,555,672 + 262,144). Artifact: /home/kayg/.cache/codex-scratch/dev-merge-prod6/target/aarch64-unknown-linux-musl/release/infraternal. Both static sizes exceed their unchanged ceilings; the Rust test row also needs a rerun with the shorter TMPDIR documented above.
Path setup confirmed: the previously failing agent test passed 1/1 with TMPDIR=/home/kayg/.cache/codex-scratch/d6 and INFRATERNAL_TEST_STATE_ROOT=/home/kayg/.cache/codex-scratch/d6/s. The exact test output is /home/kayg/.cache/codex-scratch/dev-merge-prod6/gone-short-tmpdir.log. I’m rerunning the full fast gate with these short private roots; no product code changed.
Corrected full fast run: the server test binary reported 414 passed, 2 failed, 46 ignored. The only test failures were auth socket tests
mesh_calls_and_subscriptions_deliver_scoped_live_inventoryandsession_gone::an_ended_or_unknown_session_gets_the_final_close_without_losing_the_socket; both ended withserver closed websocket unexpectedly: Noneat auth/tests.rs:1731. No other test failed. Full output: /home/kayg/.cache/codex-scratch/dev-merge-prod6/final-fast-rerun.log. I’m running these exact tests on a detached origin/dev baseline at5d098cb8in the current build-host load window as required; D243 size results are separately recorded.Baseline comparison update: on detached dev
5d098cb8,mesh_calls_and_subscriptions_deliver_scoped_live_inventorypassed 1/1 (baseline-auth-mesh.log). The branch failure therefore did not reproduce on the first baseline run; the second baseline test and branch retests are still pending. I’m investigating rather than classifying the full-suite failure as baseline noise.Baseline comparison complete: both exact auth socket tests passed 1/1 on detached dev
5d098cb8(baseline-auth-mesh.log,baseline-auth-gone.log). Their failures in the full merged workspace run are not reproduced on baseline, so I’m retesting them on the branch and will inspect any repeated failures before pushing.Source-isolated retest: both failing tests passed 1/1 on branch
5e383c3dfrom its separate clean target (branch-auth-mesh-clean-target.log,branch-auth-gone-clean-target.log). The same tests also passed 1/1 each on detached dev5d098cb8. The full workspace run had 2 unexpected websocket closes among 462 server tests; isolated reruns pass on both trees. No source changes were made. I’m starting a clean-target final fast gate so all rows use branch artifacts without cross-worktree reuse.Finished on branch dev-merge-prod6 at
5e383c3da8. Pushed the branch; it fast-forwards from origin/dev at5d098cb828. Worktree is clean.R01 passes: the permitted home alias recovers its setup descriptor. Symlink-ancestor refusal and refusal while a setup socket remains bound are retained.
Focused evidence: browser add-host 37/37; server auth sockets 33 passed, 2 ignored; review3 13 passed, 1 ignored; migration compatibility 11 passed; mesh plugin 43 passed, 47 ignored; SSH plugin 113 passed, 9 ignored across 23 targets, R01 1/1, SSH shell fixture 16/16; restart CLI 28 passed, 1 ignored; release Rust 24 passed, 5 ignored, Python 23 passed, Bun 8 passed. The clean-target full workspace test gate passed, including both auth socket tests that failed in the earlier loaded run.
Final
scripts/verify fast: fmt, VT native fmt, CLI plugin compile-out, clippy, workspace tests and cargo-deny pass. Static builds fail only the known D243 size budget. Exact architecture diagnostics from the gate:Gate final output (last 100 lines printed after
scripts/verify fast, verbatim):Closed by the orchestrator during the 2026-10-28 handover (#254), on the owner's request to close finished issues. Evidence: Fix merged into dev and release-prod-live (
52f644776,bdf68a6c) and deployed; merged-tree tests green. Reopen if anything is still missing.kayg referenced this issue2026-10-28 17:53:55 +00:00
kayg referenced this issue2026-10-28 17:53:57 +00:00