Research: complete host and fact discovery coverage (Linux families, macOS, Windows, BSD) #64

Open
opened 2026-10-07 08:30:45 +00:00 by kayg · 16 comments
Owner

Owner (2026-10-07): make host and fact discovery complete and well researched. Which Linux distro families, macOS, Windows and BSDs are covered; which facts; how hosts are found from the server host. Probes are cheap and parallelisable, so cover everything sensible. Research first (matrix + probe design + owner questions), then implement after the grill.

Owner (2026-10-07): make host and fact discovery complete and well researched. Which Linux distro families, macOS, Windows and BSDs are covered; which facts; how hosts are found from the server host. Probes are cheap and parallelisable, so cover everything sensible. Research first (matrix + probe design + owner questions), then implement after the grill.
Author
Owner

Started research on branch research-discovery from main a85ca2e265. Read the worktree contract, DECISIONS 17–22/126/138/141/144, docs/ARCHITECTURE.md, docs/PROCESS.md, and dev contracts/bootstrap.md. Issue #64 has no prior comments; checked tracking context (#5, #9, #24).

Started research on branch research-discovery from main a85ca2e265ce83572ae86b6ce322256af6ee3a10. Read the worktree contract, DECISIONS 17–22/126/138/141/144, docs/ARCHITECTURE.md, docs/PROCESS.md, and dev contracts/bootstrap.md. Issue #64 has no prior comments; checked tracking context (#5, #9, #24).
Author
Owner

Owner steer, D145 (posted while the research runs): fact collection is an infraternal agent feature (Rust, native /proc /sys / OS APIs, parallel external commands with a timeout each, hang detection and diagnosis), not a portable shell script. Before install, SSH does only a tiny OS/arch sniff, then the same binary runs once in a one-shot probe mode over SSH and returns JSON without installing. The research report's section 5 (probe design) should be read through this lens: the matrix of facts and commands stays fully valid, and the per-OS command list becomes the agent's collector table.

Owner steer, D145 (posted while the research runs): fact collection is an **infraternal agent feature** (Rust, native /proc /sys / OS APIs, parallel external commands with a timeout each, hang detection and diagnosis), not a portable shell script. Before install, SSH does only a tiny OS/arch sniff, then the same binary runs once in a one-shot probe mode over SSH and returns JSON without installing. The research report's section 5 (probe design) should be read through this lens: the matrix of facts and commands stays fully valid, and the per-OS command list becomes the agent's collector table.
Author
Owner

Finding (live-host access): local NetBird status contains 3 peers but no requested o1/o2/minipc01 names. Failed to connect to kayg@o1:22

Troubleshooting steps:

  1. Check peer connectivity: netbird status -d
  2. Verify SSH server is enabled on the peer
  3. Ensure correct hostname/IP is used failed before remote execution: (8.2 s wall). No host state changed; probe still pending for remaining targets.
Finding (live-host access): local NetBird status contains 3 peers but no requested o1/o2/minipc01 names. Failed to connect to kayg@o1:22 Troubleshooting steps: 1. Check peer connectivity: netbird status -d 2. Verify SSH server is enabled on the peer 3. Ensure correct hostname/IP is used failed before remote execution: (8.2 s wall). No host state changed; probe still pending for remaining targets.
Author
Owner

Correction: the previous comment was corrupted by shell expansion while posting. Actual evidence from the earlier attempt: netbird ssh --no-browser kayg@o1 failed before remote execution with dial tcp lookup o1 on 127.0.0.53:53 server misbehaving; wall time 8.2 s. NetBird status exposed no requested target name; no host commands ran.

Correction: the previous comment was corrupted by shell expansion while posting. Actual evidence from the earlier attempt: netbird ssh --no-browser kayg@o1 failed before remote execution with dial tcp lookup o1 on 127.0.0.53:53 server misbehaving; wall time 8.2 s. NetBird status exposed no requested target name; no host commands ran.
Author
Owner

D145 incorporated into probe design: recommend a tiny preinstall OS/architecture sniff, then the matching one-shot infraternal agent probe over the same SSH connection. The collector is native Rust with OS interfaces and bounded parallel commands, JSON output, and per-command timeout/hang diagnosis; the shell-family coverage remains documented for the initial sniff and fallback research.

D145 incorporated into probe design: recommend a tiny preinstall OS/architecture sniff, then the matching one-shot infraternal agent probe over the same SSH connection. The collector is native Rust with OS interfaces and bounded parallel commands, JSON output, and per-command timeout/hang diagnosis; the shell-family coverage remains documented for the initial sniff and fallback research.
Author
Owner

Finding (read-only host probe attempts, 2026-10-07): all three Linux peers failed before remote execution with DNS failure: o1 7.916 s (stdout 190 B/stderr 116 B), o2 7.902 s (190/116 B), minipc01 7.899 s (196/134 B); zero probe rows. macOS VM ssh failed host-key verification in 1.721 s (0 stdout/31 B stderr); no command ran. I preserved strict host-key checking and did not change known_hosts. Needs NetBird DNS/peer access and a trusted VM host key before measurements can complete.

Finding (read-only host probe attempts, 2026-10-07): all three Linux peers failed before remote execution with DNS failure: o1 7.916 s (stdout 190 B/stderr 116 B), o2 7.902 s (190/116 B), minipc01 7.899 s (196/134 B); zero probe rows. macOS VM ssh failed host-key verification in 1.721 s (0 stdout/31 B stderr); no command ran. I preserved strict host-key checking and did not change known_hosts. Needs NetBird DNS/peer access and a trusted VM host key before measurements can complete.
Author
Owner

Section 1 complete in research/discovery/REPORT.md: documented 12 discovery sources with yielded data, SSH-acceptance limits, privacy, cost and failure modes. Passive mesh/config/host-file reads are separated from the optional unauthenticated service check; cloud credential files and API calls are excluded by default.

Section 1 complete in research/discovery/REPORT.md: documented 12 discovery sources with yielded data, SSH-acceptance limits, privacy, cost and failure modes. Passive mesh/config/host-file reads are separated from the optional unauthenticated service check; cloud credential files and API calls are excluded by default.
Author
Owner

Section 2 complete in research/discovery/REPORT.md and matrix.json: 30 target families are mapped. Linux identity uses parsed os-release plus special markers; the shell-polyglot first sniff is marked a candidate because cmd and Windows PowerShell could not be validated on a host. OpenWrt 25.12+ uses apk; earlier releases use opkg.

Section 2 complete in research/discovery/REPORT.md and matrix.json: 30 target families are mapped. Linux identity uses parsed os-release plus special markers; the shell-polyglot first sniff is marked a candidate because cmd and Windows PowerShell could not be validated on a host. OpenWrt 25.12+ uses apk; earlier releases use opkg.
Author
Owner

Section 3 complete: matrix.json covers 34 facts × 30 families = 1,020 explicit cells. It preserves D126's 11 default facts, leaves its 5 optional facts as optional, and marks all 18 additional facts opt-in. Each cell includes command, fallback, root need, relative cost, parallelism, privacy, parsing and status.

Section 3 complete: matrix.json covers 34 facts × 30 families = 1,020 explicit cells. It preserves D126's 11 default facts, leaves its 5 optional facts as optional, and marks all 18 additional facts opt-in. Each cell includes command, fallback, root need, relative cost, parallelism, privacy, parsing and status.
Author
Owner

Section 4 complete: read-only access attempts produced 0 target fact rows. o1/o2/minipc01 failed before remote execution in 7.916/7.902/7.899 s; the macOS VM failed strict host-key verification in 1.721 s. Output sizes and causes are in research/discovery/REPORT.md; no host state changed.

Section 4 complete: read-only access attempts produced 0 target fact rows. o1/o2/minipc01 failed before remote execution in 7.916/7.902/7.899 s; the macOS VM failed strict host-key verification in 1.721 s. Output sizes and causes are in research/discovery/REPORT.md; no host state changed.
Author
Owner

Section 5 complete: D145's native-first Rust one-shot --probe-once design, bounded JSON/timeouts, static/live refresh split and unresolved transient binary-delivery question are recorded. No collector timing or output-size measurement is claimed because no remote command ran.

Section 5 complete: D145's native-first Rust one-shot --probe-once design, bounded JSON/timeouts, static/live refresh split and unresolved transient binary-delivery question are recorded. No collector timing or output-size measurement is claimed because no remote command ran.
Author
Owner

Section 6 complete: owner questions cover privacy defaults, fact defaults, Windows/BSD priority, API-only families, macOS/Intel floor, transient binary delivery and explicit exclusions.

Section 6 complete: owner questions cover privacy defaults, fact defaults, Windows/BSD priority, API-only families, macOS/Intel floor, transient binary delivery and explicit exclusions.
Author
Owner

Section 7 reviewer checklist complete in research/discovery/REPORT.md: confirms full cross-product, source/privacy limits, real-host evidence, unverified areas and unchanged decisions. No product code or host configuration changed.

Section 7 reviewer checklist complete in research/discovery/REPORT.md: confirms full cross-product, source/privacy limits, real-host evidence, unverified areas and unchanged decisions. No product code or host configuration changed.
Author
Owner

Finding (verification gate): attempted scripts/verify fast on branch research-discovery at a85ca2e265; this main-based worktree has no scripts/verify (exit 127: zsh:1: no such file or directory: scripts/verify). The dev ref has a different verifier tree, so I did not copy or run it against this branch. JSON parsing and the 1,020-cell structural audit passed; no product code changed.

Finding (verification gate): attempted scripts/verify fast on branch research-discovery at a85ca2e265ce83572ae86b6ce322256af6ee3a10; this main-based worktree has no scripts/verify (exit 127: zsh:1: no such file or directory: scripts/verify). The dev ref has a different verifier tree, so I did not copy or run it against this branch. JSON parsing and the 1,020-cell structural audit passed; no product code changed.
Author
Owner

Finished: committed research/discovery/REPORT.md and matrix.json at 2a986fc276. Matrix audit: 30 OS families × 34 facts = 1,020 cells, all recipes and required fields present; 0 target fact rows because access failed before remote execution. scripts/verify fast output verbatim: zsh:1: no such file or directory: scripts/verify (exit 127; verifier absent on this main-based branch). Worktree is clean.

Finished: committed research/discovery/REPORT.md and matrix.json at 2a986fc2767cf7720527aa92c7d0cca6a1de70b8. Matrix audit: 30 OS families × 34 facts = 1,020 cells, all recipes and required fields present; 0 target fact rows because access failed before remote execution. scripts/verify fast output verbatim: zsh:1: no such file or directory: scripts/verify (exit 127; verifier absent on this main-based branch). Worktree is clean.
Author
Owner

Real-host evidence (orchestrator, read-only, NetBird SSH), completing report §4, which could not reach the hosts:

Host Result Wall time (incl. NetBird SSH setup) Facts
o1 (10.69.69.51) Ubuntu 24.04.5, aarch64, kernel 7.0.0-1013-oracle, 4 CPU, 24 GB, QEMU KVM, systemd, docker.sock, docker+podman+kubectl, apt, netbird+tailscale, no passwordless sudo 1.82 s all
o2 (10.69.69.52) Ubuntu 24.04.5, aarch64, 6.17.0-1018-oracle, same shape 1.29 s all
monitor01 (10.69.69.50) Debian 13, x86_64, 4 CPU, 8 GB, QEMU/KVM, systemd, docker and containerd sockets, docker+podman, apt, netbird+tailscale, no passwordless sudo ok all
calternal-macos (10.69.69.21) macOS 27.0 (26A428), arm64, VirtualMac2,1, 4 CPU, 8 GB, launchd, netbird, no package manager found ok all
minipc01 (10.69.69.217) name resolves; TCP 22 times out: the live #58 "blocked by mesh policy" case fail 0

Findings for the collector and discovery design:

  1. Mesh names do not resolve on this server host (lookup o1.vpn.kayg.org on 127.0.0.53: no such host, while netbird status -d lists o1/o2 with IPs). Discovery must use the mesh-reported IPs (and keep the names as labels), not system DNS.
  2. netbird status -d on the server host lists 13+ peers (hass, m5p, npve-compute01, stash, calternal-macos, admin-rook, shipmelol, compute01, calternal-dev, monitor01, o1, o2, …), so quiet discovery (D144) finds them with no token.
  3. The POSIX probe covered Debian, Ubuntu (aarch64 Oracle kernels) and macOS without changes; the macOS fallbacks (sw_vers, sysctl) worked.
  4. Passwordless sudo is "no" on all four reachable hosts. Root agents therefore need the H13 consent plus a sudo password, or a user agent. Good real-world input for the add-host design.
  5. One SSH round trip including the NetBird SSH handshake is about 1.3–1.8 s from Germany, which supports the one-shot single-round-trip probe design (D145).
**Real-host evidence (orchestrator, read-only, NetBird SSH)**, completing report §4, which could not reach the hosts: | Host | Result | Wall time (incl. NetBird SSH setup) | Facts | |---|---|---|---| | o1 (10.69.69.51) | Ubuntu 24.04.5, aarch64, kernel 7.0.0-1013-oracle, 4 CPU, 24 GB, QEMU KVM, systemd, docker.sock, docker+podman+kubectl, apt, netbird+tailscale, no passwordless sudo | 1.82 s | all | | o2 (10.69.69.52) | Ubuntu 24.04.5, aarch64, 6.17.0-1018-oracle, same shape | 1.29 s | all | | monitor01 (10.69.69.50) | Debian 13, x86_64, 4 CPU, 8 GB, QEMU/KVM, systemd, docker and containerd sockets, docker+podman, apt, netbird+tailscale, no passwordless sudo | ok | all | | calternal-macos (10.69.69.21) | macOS 27.0 (26A428), arm64, VirtualMac2,1, 4 CPU, 8 GB, launchd, netbird, no package manager found | ok | all | | minipc01 (10.69.69.217) | name resolves; **TCP 22 times out**: the live #58 "blocked by mesh policy" case | fail | 0 | **Findings for the collector and discovery design:** 1. **Mesh names do not resolve on this server host** (`lookup o1.vpn.kayg.org on 127.0.0.53: no such host`, while `netbird status -d` lists o1/o2 with IPs). Discovery must use the **mesh-reported IPs** (and keep the names as labels), not system DNS. 2. `netbird status -d` on the server host lists **13+ peers** (hass, m5p, npve-compute01, stash, calternal-macos, admin-rook, shipmelol, compute01, calternal-dev, monitor01, o1, o2, …), so quiet discovery (D144) finds them with no token. 3. The POSIX probe covered Debian, Ubuntu (aarch64 Oracle kernels) and macOS without changes; the macOS fallbacks (`sw_vers`, `sysctl`) worked. 4. **Passwordless sudo is "no" on all four** reachable hosts. Root agents therefore need the H13 consent plus a sudo password, or a user agent. Good real-world input for the add-host design. 5. One SSH round trip including the NetBird SSH handshake is about 1.3–1.8 s from Germany, which supports the one-shot single-round-trip probe design (D145).
Sign in to join this conversation.
No labels
launch-qa
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/infraternal#64
No description provided.