Orchestrator handover 2026-10-28: moving to a new machine #254

Closed
opened 2026-10-28 17:50:46 +00:00 by kayg · 2 comments
Owner

Orchestrator handover, written 2026-10-28 ~23:45 IST by the outgoing orchestrator (Claude, session session_01GHLxFr6gB7AcU2GRKHB6DE). The owner is moving the orchestrator to a new machine. This issue is the single starting point for the next orchestrator: read it top to bottom before doing anything.

Read first, in this order: CLAUDE.md → DECISIONS.md (canonical; D1–D310 on dev) → docs/ARCHITECTURE.md → docs/PROCESS.md → docs/design/TASTE.md (before any UI) → this issue → #24 (status board, kept current).

Every time below is in IST. The owner wants all times in IST (CEST + 3:30; CET + 4:30 since 25 Oct).


0. Before infraternal-dev is shut down or decommissioned: copy these off it

The old orchestrator host is infraternal-dev (NetBird 10.69.69.125, LAN 10.70.3.157, user kayg). The git history is safe: everything is pushed to Forgejo, and unique branch tips are kept as archive/2026-10-28/* tags. The files below exist only on that machine. Copy them to the new host before it goes away, as the same user, preserving permissions. None of them may ever be posted to fj.

Must copy Path on infraternal-dev Why
Handover bundle ~/infraternal-orchestrator-handover-2026-10-28.tar.gz (mode 600) orchestrator state, job prompts, deploy/preview scripts (incl. the local preview-auth patch the serve-* worktrees carry), systemd units, memory (§4). SHA-256 816fe0bf2339047735be369b5691bef3bb5a99440774d7984bddf477b38008a2
1Password service-account token ~/.config/infraternal/op-service-account-token never display it; copy the file as is
Forgejo push key ~/.ssh/git.kayg.org, ~/.ssh/git.kayg.org.pub pushes to ssh://git@10.69.69.52:2222
GitHub mirror key ~/.ssh/github.com, ~/.ssh/github.com.pub, plus the Host github.com block in ~/.ssh/config mirror only; push only on the owner's say-so
fj (Forgejo CLI) token ~/.local/share/forgejo-cli/keys.json fj and issue-attach
Codex accounts ~/.config/openai1/, ~/.config/openai2/ or run a fresh codex login on the new host
Firecrawl CLI login ~/.config/firecrawl-cli/ web research
Claude Code state ~/.claude/ (1.7 GB) and ~/.claude.json settings, skills, agents/ (designer), and projects/-home-kayg-Developer-infraternal/ (the memory and this session's full transcript 15bbfdbc-0147-45a8-a97b-90d4ce6d4d03.jsonl)
git config ~/.config/git/ identity kayg / team@kayg.org
Optional Path Why
Full job folders ~/.local/state/codex-jobs/infraternal/ (14 GB) every past job's full log and result; the bundle holds only what the queue needs
Build cache ~/.cache/sccache-infraternal/ (43 GB) saves hours of cold builds; can be regenerated
Owner's add-host test fixture ~/.local/share/infraternal, ~/.local/share/infraternal.bootstrap-cache, ~/.local/share/infraternal.dev-2026-10-06 the owner's live acceptance case for add-host (§2). If the machine itself is decommissioned, not just the orchestrator moved, this case is lost unless it is copied or the VM is kept

Example, run on the new host as kayg. It copies everything in the "Must copy" table into the same paths, keeping permissions. sshd is not running on infraternal-dev, so use NetBird SSH:

netbird ssh --no-browser kayg@10.69.69.125 'cd ~ && tar czf - infraternal-orchestrator-handover-2026-10-28.tar.gz .config/infraternal .ssh/git.kayg.org .ssh/git.kayg.org.pub .ssh/github.com .ssh/github.com.pub .local/share/forgejo-cli .config/openai1 .config/openai2 .config/firecrawl-cli .claude .claude.json .config/git' | tar xzf - -C ~

Merge the Host github.com block from the old ~/.ssh/config by hand. Afterwards, check that ~/.config/infraternal/op-service-account-token is mode 400 and its folder 700.

Then, on the running system:

  1. Stop the services on infraternal-dev: systemctl --user stop serve-demo serve-recovery serve-clusters status-preview and the three infraternal-* timers.
  2. infraternal-agent.service on infraternal-dev is an infraternal host in production. If the machine goes away, remove it in production's host list, or expect it to show as offline.
  3. Tell the owner the new preview links (the old ones are http://10.69.69.125:47361, :47381, :47401, :47391).

1. Where things stand (2026-10-28)

Nothing is running. No Codex jobs, no subagents. The scheduler is paused (orchestrator/pause exists). Nothing has happened in the repo since 2026-10-22 ~08:45 IST, when Codex sign-in expired.

Production (monitor01, root@10.69.69.50, https://infraternal.kayg.org):

Binary release 9.4, b30bb81cf; MainPID 211431 since 2026-10-22 06:31 IST; exe SHA-256 71ced02f01ef832a… (= artifact)
Web db4add238 = 9.4 + mesh-view-overlap + mesh-drag r4 + a test typing fix. Swapped in live with no restart
release-prod-live db4add238 (always = what runs in prod)
healthz 200 (checked 2026-10-28 23:15 IST)
Public access since 2026-10-22 20:10 IST (#253): the Traefik allow-list was removed. The infraternal agent port 7443 stays NetBird-only (D64)
CA re-locked by the 9.4 restart on 2026-10-22. The owner was asked to unlock it; not confirmed.

Branches (all remote branches were cleaned today; see §6):

Branch Head Meaning
dev 81962ab09 all development. Does not yet contain release 9.4 or the web update (see §3, task 1)
release-prod-live db4add238 production
release-9-4 b30bb81cf 9.4 release line (binary in prod)
main 5a0ec9ee7 stable builds only; stale since 2026-10-10. Owner rule: main = stable builds; nothing promoted since
others the paused jobs' branches (§3) plus proto-status (a preview)

Codex: codex login status reports both ~/.config/openai1 and ~/.config/openai2 signed in. On 2026-10-22, openai1's refresh token had expired (401) and openai2 hit its usage limit (reset 2026-10-27 07:32). The owner may have run codex login since then. Before relaunching anything, run one small codex exec per account to confirm it works.


2. Owner rules and standing orders (verbatim where quoted)

Canonical decisions are in DECISIONS.md. Only the owner can change one.

The decisions that drive the queued work:

Decision What it says
D64 Managed hosts open no inbound ports; infraternal agents dial out
D206 Iteration loop
D217 Updates never lock the CA; deploys use an in-memory handoff. Not built yet (#154), which is why every restart still re-locks the CA
D241 Auto-started multiplexers give way to infraternal sessions
D243 Binary size goes down, not up
D250 How changes reach production
D251 Production follows edge and updates itself (#211)
D262 Evidence lives on the issue
D288 infraternal mux: sessions automatable from inside a session (#251)
D291 New hosts join through the add-host SSH connection
D295 Undoing a failed server update stays simple
D309/D310 The latest owner answers

These orders and constraints are not in DECISIONS.md, or only partly:

  • Latest work order (2026-10-21): "Work autonomously and parallelly through your roadmap. Use Firecrawl to research, web search any information that you might need. Use Codex subagents for backend tasks and your own subagent, Sonnet 5.5, for UI and UX tasks."
  • Priority (2026-10-21): "let's prioritise running mux and mesh from the client side instead of the add-host side." The owner picked "Finish r8, then freeze". Add-host (#231) is now frozen after round 8. The client-side work is laptop CLI mux/attach plus a host joining by itself (#251, D288).
  • Tonight's roadmap (2026-10-21): it must not need owner decisions; owner questions wait for the morning, with concrete examples.
  • Deploy without asking: reviewed and staged fixes go to prod without the owner's OK. Then tell the owner to unlock the CA, because every server restart re-locks it. A web-only deploy swaps live; verify it by MainPID, not by restart count.
  • Break-it reviews: every Codex backend job gets a review that tries to prove it wrong with failing tests. Use neutral wording ("defensive audit with test cases"), never "attack/exploit/attacker". The content filter kills such jobs (#231 r8 review died this way).
  • Merge bar (D206 iteration loop): merge when there is no P0/P1, with delta reviews; prod is the test target. Before any merge the user can see, check that the break-it review is clean, there are perf-VM numbers, the size does not grow, the diff has been read, and a real demo video exists.
  • D243 size bands: growth under 16 KB the orchestrator approves; 16–64 KB needs a written reason; over 64 KB needs the owner. Binary size caps in scripts/verify.d/rust.sh ratchet to measured bytes, with no allowance.
  • Codex fast mode is ON (owner 2026-10-15/19, "Codex jobs fast from now on"): flag file orchestrator/fast-mode; jobs run with -c service_tier="priority".
  • No invented limits: every limit must trace back to an owner message or a hard safety need.
  • History: atomic commits; push every commit. Commit trailers:
    Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> and Claude-Session: <session url>. No other products' names in code or comments.
  • Tracking: one fj issue per piece of work. Comment at start (branch + base), at each finding, when blocked, and at the end (head + gate output verbatim). Evidence (screenshots, logs) goes on the issue via issue-attach, never into the repo (D262). Jobs never close issues; the orchestrator does when the owner asks.
  • Foundational libraries (terminal state, crypto, DB, transport, renderer): never chosen inside a job. They go to the owner as a grill question with measured trade-offs.

Security and access constraints (verbatim from the owner or from hard safety needs):

  • 1Password service-account token: ~/.config/infraternal/op-service-account-token. Never cat or display it; pass it only as an env var. Vault "Apps / Infraternal" (Cloudflare token). Orchestrator only.
  • No secrets in the repo, logs, argv or URLs. Fewest secrets: use a credential once, seal it end to end, never store it.
  • Host keys: never accept a changed host key. ssh monitor01 fails host-key verification; use netbird ssh --no-browser root@10.69.69.50.
  • Test hosts: "minipc01, ash1, ash2 can be connected via ssh on monitor01, you're free to access them via there and then clean up known hosts on monitor01 afterwards! you may not modify any vms or containers on minipc01, ash1, ash2. you're free to test infraternal agent though." On minipc01 (Proxmox), create and delete only our own infraternal-test-* guests.
  • infraternal-dev (10.69.69.125) is the owner's acceptance host. From rules.md: "no contact of any kind from jobs: no ssh, no host-key probe, no ping, no inventory scan". When you list hosts from production data, filter it out first.
    • /home/kayg/.local/share/infraternal* on it is the owner's live add-host test fixture: never modify, move or delete it (owner: "please don't, rather build the better UX and let me try if it works!").
    • It is also the old orchestrator/build host (see §5). Once the orchestrator moves, treat it like any other owner host. It stays an acceptance test case.
  • MacBook kayg@10.69.69.101 (the Mac VM also runs on it; one lock for both):
    • Connect with netbird ssh --no-browser only, never --tty.
    • Take /tmp/infraternal-bench.lock; stop below 30% battery.
    • Never change TCC, trust or keychain settings; never click prompts.
    • Capture only inside the signed companion via open -n -W. On any prompt or denial, stop and never retry.
    • Restore the shipped helper (SHA-256 352d671b823b352f3c12ad1f71dcf6c55e82e5857f0b7c15956f2e404c1858cc).
    • Test keychain ~/Library/Keychains/infraternal-signing.keychain-db, unlocked in the same command right before each codesign. Sign by SHA-1 3E6AE7C6BA4CFE1E6280535EB93CF7A7959A7BA6.
    • Give cleanup proof with every PID.
  • Perf VM root@10.69.69.208 (infraternal-perf-test, ours alone, 4 vCPU): use netbird ssh and flock /var/lock/perf-bench.lock. All perf numbers come from here.
  • Production DB: read-only, via monitor01 (/var/lib/infraternal/infraternal.sqlite; sqlite3 is not installed there).
  • GitHub mirror kaygdotorg/infraternal: push only when the owner says so. Issues stay on fj. #218 tracks the history secrets audit before any public push.

3. Work queue, in priority order

3a. Owner actions (only the owner can do these)

  1. Unlock the CA in prod (pending since the 9.4 restart on 2026-10-22), if not done already.
  2. Codex sign-in on the new machine: CODEX_HOME=~/.config/openai1 codex login (and openai2 if needed).
  3. Owner questions parked for a grill (each has recommended answers and examples on its issue): #251 Q1–Q6 (client-side mux/mesh), #220 Q1–Q8 (NAT traversal/relay), #146 encoder question (app windows).
  4. MacBook: it needs to be charging (≥30%) before the parked Mac lane parked/53-app-windows-second-capture (#146) can run.

3b. First orchestrator tasks on the new machine

  1. Merge production back into dev (#24). git merge origin/release-prod-live into origin/dev conflicts in exactly two files:

    • scripts/verify.d/rust.sh: the static size caps. dev has 21,389,232 / 19,561,656; 9.4 has 21,389,424 / 19,562,296.
    • docs/PROCESS.md: the size-ceiling table and its text.

    Resolution: build the merged tree with the pinned static recipe (scripts/static-cargo / auth-toolchain Zig, as in the 9.3/9.4 release prompts), then set both caps to the measured bytes. Keep both PROCESS.md paragraphs and add one for the merge. Run scripts/verify fast with CARGO_TARGET_DIR set. The original job spec is in the bundle at jobs/merge-release94-dev/prompt.md (Luna max).

  2. Resume the stopped jobs (they stopped mid-task when Codex sign-in expired on 2026-10-22). Use orchestrator/resume-job.sh <job> <model> <effort> <wip-sha>. Its continuation text says "host resize"; edit it to say "Codex sign-in expired / host move". Each job's WIP commit must be reset --soft HEAD~1-ed, then split into atomic commits.

# Job Issue Model / effort (fast) Branch → head State when stopped
1 setup-token-data #252 P1 (launch QA S2) Luna max setup-token-data-wip 21cb39947 (base dev) Rust and TS done and tested: the setup screen prints sudo infraternal server setup-token --data /var/lib/infraternal for a system service. The final scripts/verify iterate/static size step was not finished
2 revocation-per-account-r2 #240 Sol medium revocation-per-account 57185fd9b (+6,304 B) Review HOLD (review-revocation-per-account f52fd309b): R1, a logout waiter cancelled after commit loses the notification; R2, bootstrap keeps its socket when the notification source is lost. Round 2 had just started
3 review-sync-p3s #250 Sol medium sync-p3s 3b94b1f28 (+0 B, gate PASS) Break-it review had just started
4 sync-journal-r3 #250 Sol medium sync-journal-r3 573dc412e Perf VM: 6/6 pass (max save 12.2 ms / flush 2.3 ms at 60 windows). The final full gate was queued behind the merge-base native build. Merge-base size measured: 21,389,296 B, delta 0
5 edge-autoupdate-resume #211 (D251) Sol medium edge-autoupdate-resume-wip d4a1bf6ea (local branch edge-autoupdate-r2; remote edge-autoupdate-r2 643a5af84 is older) Edge auto-update round 2 in progress. A partial review exists: review-edge-autoupdate-d295 b1555e44f (171 lines of recovery tests, WIP)
6 d241-shell-marker #251 / D241 Luna max d241-shell-marker-wip e6b0c5a16 (remote d241-shell-marker e3a06ba42 is older) Perf (perf VM, 2 runs: p99 −1,462 µs / −793 µs) and size (−192,864 B) are done. "Profile commit and final gates are next"
7 broad-real-suite-triage #207 Luna max broad-suite-triage d69ce64ed Triage of the broad real-browser suite; continue from its last #207 comment
8 merge-release94-dev #24 Luna max none yet See 3b.1. It looped on Codex errors, so it was stopped

review-mesh-drag-r3 is not needed (a Sonnet review covered it; the drag fix is in prod).
3. Follow-ups found in the last rounds (no job yet):

  • Mesh pins (P3): a pin for a host that was already gone at page load is never pruned and counts toward the 256-pin limit. In apps/tracer/src/canvas/mesh/api.ts, pruning only drops hosts that disappear while the page is open.
  • The WebKit 1,000-host pin-limit test fails, and already did before mesh-drag round 4. WebKit touch/pen drags can't be measured in headless mode.
  • scripts/release/build-metadata.test.ts:162 fails depending on the checkout path (it passes in the dev worktree). Backend fix.
  • #237: a crash loop while infraternal status says Running (found by the launch-QA dry run, #247).
  • #253: queue a defensive audit of everything reachable without signing in now that the site is public (rate limits on sign-in, setup and join, header handling behind Traefik).
  1. Frozen, do not resume without the owner: #231 add-host re-add. Round 8 review HOLD (P1: a state directory created between join and activation is adopted, then removed by rollback). WIP regressions on review-readd-r8-wip 287065c2f; round-9 plan on #231. Nothing from #231 is in prod.
  2. Parked lanes (specs in orchestrator/parked/): 53-app-windows-second-capture (#146, base app-windows-second-stream, needs the MacBook), 90-research-wasm-core (#177, base research-wasm-core).
  3. The orchestrator queue.md has the long-tail queue, and resume.md has the per-day notes. Both are in the bundle.

4. Tooling outside git (bundle on the old host)

Bundle (not attached here, because this repo and its issues are public): ~/infraternal-orchestrator-handover-2026-10-28.tar.gz on the old host infraternal-dev (mode 600; 476 KB; sha256 c37d785b76f08ea717a0…). Copy it to the new machine by hand and keep it private. It holds the orchestrator memory and job prompts, including the Mac's test-keychain password. It was scanned for tokens and private keys and holds none. It contains:

In bundle Restore to What
orchestrator/ ~/.local/state/codex-jobs/infraternal/orchestrator/ scheduler.sh (slot scheduler; ready/NN-name/{meta,prompt.md}, meta = MODEL= EFFORT= BASE= WEIGHT= ISSUE=), rules.md (preamble put at the top of every job prompt), run-job-retry.sh, resume-job.sh, build-slot (serialises heavy builds), disk-guard.sh (scheduler needs ≥ 80 GB free), branch-sweep.sh (nightly: deletes branches merged into dev), scratch-sweep.sh, issue-attach (uploads evidence to an issue; the token is read from fj's key store), git-ssh (no ControlMaster; key ~/.ssh/git.kayg.org), pause, fast-mode, priority-jobs, queue.md, resume.md, parked/, held/, launched/ (every past job's meta + prompt), deleted-branches.log (SHA of every deleted branch)
jobs/<job>/ ~/.local/state/codex-jobs/infraternal/<job>/ prompt, run scripts, result and log tail for the 8 stopped jobs + release-9-4 (the release recipe)
preview/ ~/.local/share/infraternal-preview/ deploy-prod.sh (the only deploy path: refuses a SHA that doesn't contain release-prod-live, backs up the DB, rolls back automatically if the server doesn't come back healthy), publish.sh, serve.ts, preview-forward.ts, perf-watch.py
systemd-user/ ~/.config/systemd/user/ timers infraternal-{disk-guard,scratch-sweep,branch-sweep}; owner previews serve-demo (47361), serve-recovery (47381), serve-clusters (47401), status-preview (47391); infraternal-agent.service (this host as an infraternal host)
claude/memory/ ~/.claude/projects/<repo-path>/memory/ the orchestrator's memory: 70+ files of owner preferences and rules. Start from MEMORY.md
repo-untracked/.cargo/rust-lld <repo>/.cargo/rust-lld untracked linker wrapper in the old main checkout (finds the pinned toolchain's LLD)
claude/agents/, claude/skill-codex-subagents/ ~/.claude/agents/, ~/.claude/skills/codex-subagents/ designer/designer-high subagent definitions; the Codex job launcher skill (names, systemd units, model table)

Secrets to bring by hand (not in the bundle; exact paths and a copy command are in §0):

  • the 1Password token file;
  • ~/.ssh/git.kayg.org (Forgejo push key);
  • fj's key store ~/.local/share/forgejo-cli/keys.json;
  • Codex homes ~/.config/openai{1,2} (or a fresh codex login);
  • NetBird enrolment for the new machine;
  • GitHub key (mirror only).

Rebuild checklist for the new machine:

  1. Install the toolchain via nix (bun, fj, firecrawl at ~/.npm-global/bin/firecrawl, Codex at ~/.npm-global/bin/codex; the skill calls the vendored musl ELF by absolute path).
  2. Enable linger with loginctl enable-linger.
  3. Set up sccache.
  4. Builds only on SSD; the old host had 12 vCPU / 64 GB, with ~8–9 jobs at once and ≤ 5 build-heavy.
  5. Clone ssh://git@10.69.69.52:2222/kayg/infraternal.git. Worktrees go in <repo>/.worktrees/<name>.
  6. Recreate the serve-* worktrees at their pinned SHAs: serve-demo d9564c89a, serve-recovery e52ef1388, serve-clusters 30faecc14 (each gets a one-line, never-committed apps/tracer/test/mock-server.ts tweak, applied by preview/preview-auth.patch.sh <worktree>; also proto-status 145bf0e93). Run bun install --frozen-lockfile in apps/tracer.
  7. The preview forwarders bind to 10.69.69.125 (the old host's NetBird IP). Change them to the new host's IP and tell the owner the new preview links.
  8. Copy the systemd units and the timers.
  9. Update orchestrator/rules.md for the new host. It still says "infraternal-dev is the machine you are running on", lets dev servers bind to 10.69.69.125, and says to benchmark locally on infraternal-dev. Change these to the new host's name and IP. Perf numbers still come from the perf VM (10.69.69.208), which the orchestrator books.
  10. Paths in the scripts are absolute under /home/kayg/… (deploy-prod.sh uses /home/kayg/Developer/infraternal and /tmp/claude-1001). Keep the same user and paths, or update them.
  11. Remove orchestrator/pause only after the Codex check in §1.

5. Infrastructure map

  • Old orchestrator/build host: infraternal-dev, NetBird 10.69.69.125, LAN 10.70.3.157, 12 vCPU / 62 GB, Debian 13, /dev/sda1 492 GB SSD (117 GB used after cleanup). It also runs an infraternal agent (infraternal-agent.service) and the owner previews.
  • Production: monitor01, root@10.69.69.50, public 37.157.255.66. Traefik in Docker (kayg-traefik); route /opt/infra/traefik/dynamic/infraternal.yml → http://172.18.0.1:7011. Server unit infraternal-server (--behind-proxy --trusted-proxy 172.18.0.0/16 --remote-listen 10.69.69.50:7443). Backups in /var/backups/infraternal/. healthz http://172.18.0.1:7011/healthz.
  • Forgejo: git.kayg.org (SSH 10.69.69.52:2222). Use fj --host https://git.kayg.org; issue create --repo kayg/infraternal --body-file f "<title>"; issue view kayg/infraternal#N [comments]; issue search --repo kayg/infraternal --state open. Known: large pushes print Forgejo: Internal Server Error from a hook, but the refs land; verify with git ls-remote.
  • Perf VM: root@10.69.69.208. MacBook: kayg@10.69.69.101. Test hosts: minipc01 / ash1 / ash2, reached via monitor01.
  • Cloudflare DNS: infraternal.kayg.org A → 37.157.255.66 (the owner changed it on 2026-10-22). Traefik uses DNS-01 via Cloudflare; the certificate is valid until 2027-01-02.

6. Cleanup done today (2026-10-28)

  • Unfinished code committed and pushed as WIP:
  • Branches archived, not lost. Every branch tip that wasn't already in dev, main, release-prod-live or a kept branch is now a lightweight tag archive/2026-10-28/<branch> (290 tags; -remote suffix where the remote tip differed from the local one). Each was verified on Forgejo against the local commit ID. Then:
    • 354 remote branches were deleted, leaving 21;
    • 438 local branches were deleted, leaving 21;
    • 328 worktrees were removed, leaving 23: main, the 18 job/lane worktrees, the 3 serve-* previews and proto-status;
    • about 67 GB of disk was freed.
  • Restoring a branch: git push origin archive/2026-10-28/<name>:refs/heads/<name>. Every deleted SHA is also listed in orchestrator/deleted-branches.log.
  • Kept as-is: .worktrees/infraternal/ (52 KB of old spike files, not a worktree) and .worktrees/release-pipeline/target/root-proof (root-owned leftover; needs sudo rm).
  • Issues closed today: 90 issues that were verified done; each has a closing comment with its evidence. Appendix A lists them, and Appendix B gives the state of every open issue.
  • Lost in the cleanup: untracked perf-VM browser evidence folders from 2026-10-12/13 in the old prod-perf-fix worktree (evidence/perf-vm-browser/*, .job-state/) were deleted along with that worktree. Whatever had been attached to its issue then is still there; the rest is gone.

7. Gotchas that cost time

  • git.kayg.org/kayg/infraternal is public (private: false). Anyone can read it, its issues and every evidence attachment, without signing in. Never put credentials, private keys, test passwords or personal data in an issue, a comment or an attachment. The owner may want to review this (see #218 for the mirror audit). A full-history secrets scan on 2026-10-29 (gitleaks over every ref and tag, plus a regex pass) found no live secret; only false positives, test keys and local test-server cookies. Details are on #218. Issue bodies, comments and attachments were not scanned.
  • The Bash tool runs zsh:
    • $S:refs/... is read as a zsh modifier, so write "${S}:refs/...".
    • for x in $VAR does not word-split; use an array or ${=VAR}.
    • An unmatched glob is an error.
  • scripts/verify fast needs CARGO_TARGET_DIR set; without it, the release-plugin tests fail. Run heavy gates through orchestrator/build-slot.
  • Web build: in apps/tracer, run bun install --frozen-lockfile && bun run check && bun test … && bun run build && bun run transfer-budget.ts. Critical-JS Brotli cap is 256,000 B (prod is at 255,839).
  • Deploy: deploy-prod.sh <sha> [--bin x86] [--bundle dir] [--web dist]. Never pass --arm together with --bundle. Verify with systemctl show -p MainPID infraternal-server + sha256sum /proc/$pid/exe + healthz, then force-push release-prod-live to the deployed SHA and comment on #24.
  • Codex's content filter stops VT/FFI/auth jobs that use attack words. The recipe is in memory codex-filter-vt-jobs.md.
  • WebKit headless has no backdrop blur: check glass effects in Chromium or on a real iPhone.
  • Owner previews ("serve-*") are stable copies: jobs never restart them.

Appendix A. Issues closed on 2026-10-28

Issues closed on 2026-10-28 (90; each has a closing comment with its evidence). Done items were checked by git merge-base --is-ancestor against dev/prod, or a recorded decision or superseding issue was cited. Borderline cases were left open: #79 (Q18 open), #169 (260 comments, findings may live elsewhere), #207 (its triage job is paused), #230 and #244 (owner retest pending).

#10 #11 #12 #13 #15 #16 #17 #19 #20 #21 #22 #23 #25 #26 #27 #28 #29 #30 #31 #32 #33 #34 #36 #39 #40 #51 #55 #57 #75 #81 #84 #85 #90 #91 #113 #114 #115 #116 #119 #122 #123 #124 #125 #126 #127 #128 #138 #139 #144 #149 #151 #152 #155 #161 #162 #163 #164 #165 #166 #167 #168 #171 #173 #174 #181 #182 #185 #188 #189 #190 #191 #193 #195 #196 #198 #199 #201 #210 #214 #219 #228 #229 #234 #236 #241 #245 #246 #248 #249 #253

Reason per issue
  • #10: Contracts (shim, plugin, bootstrap, wire) drafted at d731975 which is in dev and prod; wire.md updated for D63 and implemented by the merged lanes.
  • #11: Workspace/gates merged as 39630d6 (in dev and prod) after orchestrator split into atomic commits; gate PASS in the issue.
  • #12: Spike result delivered and owner decided WebSocket-only on leg A in D63 (custom codec stands); wire.md updated fd9a498.
  • #13: Renderer spike delivered; ghostty-web was chosen (D14) and pinned from source in D109/#36 (845c293 in dev and prod).
  • #15: Codec lane merged to dev as 7174d6b (verified ancestor of dev); P1 split-header finding fixed and fuzz clean.
  • #16: Shim lane merged to dev as 7ae0b83 (verified in dev); review findings answered, gate PASS.
  • #17: Auth lane merged to dev as 7668d0c (verified in dev) after rework and second review verdict MERGE; D86 approved crypto.
  • #19: Server core merged and deployed (t-server 7551ff6/f1b996c and integration acf83c7 are in dev and prod); audit findings fixed under #28/#29.
  • #20: Agent core merged (6c64ede, 48d7bab in dev and prod); review R3 fixed (e72589e), integrated via #34.
  • #21: Web client merged (58040df, faa3154, ff505cb in dev and prod); audit F1 fixed in #29 round.
  • #22: libghostty-vt switch (D94) merged via a5250c3 (#22,#26,#34) in dev and prod; remaining resize-refusal limitation tracked on #37.
  • #23: VNC research delivered (137a569, cdc7c59 in dev); owner answers recorded as D104-107, follow-up spike is #35.
  • #25: fix-vt snapshot fidelity finished at 234baf0 (in dev and prod); nightly 19,864 replays, 0 divergences; second review was #26.
  • #26: Second review verdict rework accepted as known limitation by orchestrator; real fix (transactional resize) is tracked on #37 and 565105f is in dev and prod.
  • #27: Idle canary fixed at 48d7bab (in dev and prod); zero idle wakeups measured, gate PASS.
  • #28: Six server findings fixed and merged (7551ff6, acf83c7 in dev and prod); the beyond-120s writer gap is a documented owner choice.
  • #29: Auth audit rounds ended with fixes merged (7551ff6, faa3154 in dev and prod); A3 answered as D108 and regressions added.
  • #30: D108 single passkey prompt implemented (faa3154 in dev and prod) and independently verified by review-auth3.
  • #31: Superseded by #36/D109: owner chose no upstream release request and a pinned source build, shipped in 845c293.
  • #32: Sign-in/onboarding port merged as 56bb6f51d (in dev) with D98 updated in d2ce17a; tests and screenshots posted.
  • #33: Pane cards, branched tree and tooltips merged in 0c47cd42e (#33, #36) which is in dev and prod; production screenshots and gate posted.
  • #34: Lanes merged into dev and deployed to monitor01 (acf83c7 in dev and prod; prod now runs release 9.4 line); later comments are layout-table notes.
  • #36: Ghostty pinned to a source build at 845c293 (in dev and prod), accepted by orchestrator, D109.
  • #39: Snapshot paint bug fixed and merged as 7d17212 (in dev), regression snapshot-resume.spec.ts passes.
  • #40: Fixes merged as 6016ff7a0 (in dev); owner answered the two design questions as D111 and D112.
  • #51: Setup-token socket retired; merged as b5e677956, deployed to monitor01 and verified socket absent after start with an owner present; b5e677956 is in dev and prod.
  • #55: Gate merged as ac03b1913 (verified in dev and prod): 300 ms RTT default profile and input-to-photon gate; remaining budget misses were split into #52 #61 #62 #63 #73 #74.
  • #57: Size-budget commit 2de8da2 taken and measured-ratchet follow-ups landed (4b2cd2ad4, e273fd309 in dev); mold measured as no gain and deliberately not merged, sccache set up.
  • #75: Lossless local terminal themes deployed (9e09924, then 7eca507 after the cursor regression was fixed by bd12725fe); all verified in dev and prod and checked on the production build.
  • #81: Engine measurement report research/mesh-engines/REPORT.md delivered and owner decisions recorded as D169 (a24ee0bd6 in dev).
  • #84: Cell metrics and on-grid glyphs deployed (7eca507) and the letter-spacing hotfix bd12725fe deployed; both verified in dev and prod with cursor position checks at DPR 1/2.
  • #85: Palette keyboard selection fix merged (e3e00eaea in dev) and deployed to prod in 5ed4f29 (ancestor of release-prod-live), owner-visible deploy comment, no open findings.
  • #90: Font picker, per-font weights and glyph placement merged into dev (ac10a5e22, 4c5c08a16, 459f10a19); fonts and Maple Mono weight present in dev tree.
  • #91: Owner answers recorded as D177; both toolbar rows, pill swipe, in-place toolbar editing and device layout merged in dev (c8708d9b2, f27bafdc8, 49bbe5bf2).
  • #113: Wheel/touch goes to the locked terminal: 664e45e88 in dev, deployed in 1c3ae5f (ancestor of release-prod-live).
  • #114: Sidebar overlays the canvas: fix b054f237d/1c3ae5f40 in dev, deployed in 1c3ae5f (in release-prod-live).
  • #115: Droplet split/close finished (head b0c9a4c, gate exit 0), droplet code present in dev (apps/tracer/src/canvas/create.ts), D190 recorded.
  • #116: Frame meter and freeze recorder finished and merged (perf-hud.on-demand.ts in dev; later commits d010d7ceb, 930675def under #116).
  • #119: Phone fixes deployed in 63b1f10e (in release-prod-live and dev), 27/27 real-server specs, D191.
  • #122: Quick-swipe fix (80c276796, 8a3d5b045) merged and deployed in 63b1f10e (in release-prod-live).
  • #123: Stray reach arrows removed (488cded1; equivalent 167b06c32 in dev, Mesh-only arrows, D194) with spec.
  • #124: Bisect found no regression across the commits; real zoom cost is tracked under #121 and the blur mitigation was reverted.
  • #125: Full screen finished (b9d05364, D193); full screen code and shortcuts present in dev.
  • #126: Resize-from-dragged-edge and lone-window behavior per D195 deployed in 63b1f10e (in release-prod-live).
  • #127: Window details popover merged into dev (7d6df4470, 90c3d9328 "popover is in the product canvas", head 42e50d42 is ancestor of origin/dev); finish comment has gate output, no open findings.
  • #128: Fix canvas-switch 9a770037 is in dev and was deployed to prod 2026-10-09 (web build of 01a6d848, also in dev and prod-live) with measured before/after table.
  • #138: Research delivered (research/terminal-images/REPORT.md at c8624b58) and owner decisions recorded as D203 (images pipeline, limits, HEIC/AVIF placeholder until trusted decoder, sandboxed helper).
  • #139: Research delivered (research/native-capture/MEASURE.md, 57346fc3) and owner decision recorded as D204 (macOS capture bindings and encoder).
  • #144: Host facts for popover merged into dev and prod-live (5cf106de ancestor of both); extended info explicitly out of this slice; final review and gate were clean.
  • #149: TLS session fix merged and deployed in prod-batch (6320f5b52 / 5d02b8d6b ancestor of release-prod-live and dev); delta review found no gap.
  • #151: Session cap/dead session fixes merged and deployed in prod-batch 707a4572 (6320f5b52 and b67d03455 in dev; release-prod-live contains it), listed in the deploy merge message.
  • #152: UX audit report delivered via orchestrator; owner decisions on its grill questions recorded as D219.
  • #155: Research delivered and re-grill result recorded as D222; shortcuts-d222 (e6969edc) is in dev and prod-live.
  • #161: Fixed (settings-roundtrip 051d0a40 in dev and prod) and deployed in release-prod-4 3aec316e with before/after evidence (0 stray PTY bytes across five cycles).
  • #162: Research note delivered (research/mesh-ssh) and owner grill outcome recorded as D235.
  • #163: Back-merge completed: dev-backmerge 8d0fc8ba merged into dev as 50a89572 (ancestor of origin/dev), checks recorded.
  • #164: All five items shipped: phone-polish c512cc07 is in dev and prod-live, lock-icons (withLockBadge) in dev; glyph question resolved by the owner's lock-badge rule.
  • #165: Generic Inbox card 89145f81 deployed web-only to production and is in dev and prod-live, gate and specs recorded.
  • #166: Sidebar polish 15d15c25 deployed web-only to production and is in dev and prod-live, captures verified.
  • #167: Research delivered (research/canvas-node, 2202eaf9) and owner decisions recorded as D248.
  • #168: Ended-sessions fix deployed in release-prod-4 3aec316e (in dev and prod-live) after review rounds 2-3; last comment is the deploy record.
  • #171: Sign-out clear (d8fb45c8, d472ee3a) is in dev and release-prod-live; deployed web-only in 4855e033.
  • #173: Merged into dev at c75c362b after a clean break-it review (P0-P3 none); c75c362b verified in dev and prod.
  • #174: Restart tests fixed and merged as cc3297508 (fix-restart-tests, #174) in dev; gate-flakes confirmed 50/50 loaded passes.
  • #181: Phone width fix and settings-rows (d6642c21) are in dev and release-prod-live; deployed in 4855e033 and 3aec316e.
  • #182: Cursor-blink (D249) deployed in 4855e033, which is in dev and release-prod-live.
  • #185: Ended-strip fix (361fc7ff) is in dev and release-prod-live; deployed as 4d1b1950.
  • #188: Deploy pipeline merged into dev (21b37bca7 is in dev and release-prod-live) and used for the production releases.
  • #189: Add-host speed work (addhost-instant 53ef70af6) is in dev and prod; deployed to production as 3048580c1 and release 2 (103815e70).
  • #190: Duplicate of #189; the addhost-fast review said MERGE and its follow-up fixes went in through addhost-instant (53ef70af6), which is in dev and prod.
  • #191: Superseded by #192 (the same feels-local research brief, with the superlogical note carried over).
  • #193: Observability research and log-volume numbers delivered; decisions recorded as D260/D261 (D261 in DECISIONS.md).
  • #195: Fix merged into dev and release-prod-live (52f644776, bdf68a6c) and deployed; merged-tree tests green.
  • #196: Deployed web-only as release-prod-5w ade40a36 and pushed to dev.
  • #198: Merged to dev as 9e84426c4 (in dev and prod); the size caps are green again.
  • #199: Switch-over merged (fa904fdaa) and deployed (036bc2bb0); both are in dev and release-prod-live.
  • #201: Mesh auto-switch deployed 2026-10-17 as 3b2183eaa, which is in dev and prod.
  • #210: Merged web-budget-headroom (804ac0e85) in dev: deferred brotli 44,807 to 42,272 of 45,056.
  • #214: Terminal links merged as f210a4f25 (ancestor of origin/dev and release-prod-live), deployed web-only and owner-approved; underline/CJK follow-ups tracked in #232.
  • #219: Linux mesh problem survey completed (report attached, snapshot 5abc64bb8); its findings drive the relay work in #220.
  • #228: Tooltip fix merged to dev and deployed web-only at 1bcb7a87b (ancestor of dev and release-prod-live); final gate passed.
  • #229: Mesh unreachable fix released in 3b2183eaa (ancestor of dev and prod), deployed 2026-10-17, review verdict MERGE.
  • #234: Ping/control-queue fix and link-doubt indicator released in 103815e70 (ancestor of dev and prod, release 2) with measurements.
  • #236: Honest Mesh view and host names deployed in release 5 (c00c6364e) and release 7 (173a9ba14), both ancestors of dev and prod.
  • #241: WASM page reuse fixed by renderer-zero-pages, merged as cef7224ee (in dev and prod) and deployed web-only 2026-10-17.
  • #245: Liveness retirement fix deployed web-only at 857823ce4 (ancestor of dev and prod); production journal showed zero stream closes after deploy.
  • #246: Cross-root reproducible static builds merged to dev (702e671a0, 709efb451 ancestor of dev) with passing final gate.
  • #248: Stale-screen note merged (c8aa36727 in dev) and deployed in release 8 (85b0285b7, ancestor of dev and prod).
  • #249: Quiet idle warm-up merged (d37691790) and deployed in release 9 (bec4eb7dd, ancestor of dev and prod).
  • #253: Public web access completed and verified on 2026-10-22 (Traefik allowlist removed, agent port stays NetBird-only).

Appendix B. Open issues on 2026-10-28

One-line state per open issue (2026-10-28 triage, read-only, from each issue's latest comments). Use this to pick work; the issue itself is the record.

All open issues
  • #1: Decision-record issue still lists unchecked items (tier prices, KCP benchmark, snapshot-vs-state-diff resume, S1 details).
  • #2: S0 website rejected by owner and parked on 2026-10-05; branch s0-website never merged, redesign/design grill not done.
  • #3: Pages/D1/secrets set up but the workflow site.yml, first deploy and custom domains/TLS are still pending (waits on S0 website).
  • #4: Perf VM and bench runner checklist all unchecked; no comments (only perf-run lock script landed).
  • #5: S1 grill round 1 answered (D48-53) but last comment leaves the TLS strategy for infraternal.kayg.org open and topic list unchecked.
  • #6: DNS fixed but the owner decision on old status-page containers/files on monitor01 is unanswered.
  • #7: Standing real-iPhone checklist template with every item unchecked and no comments.
  • #8: GitHub push mirror and README pointer both unchecked, no comments; push is only on owner say-so.
  • #9: S1 umbrella with many unchecked backend/web/quality items (SSH plugin, bootstrap, inventory, perf gate, etc.).
  • #14: Piece-by-piece visual grill still ongoing (latest piece ui-tweaks merged 56bb6f51d); product UI identity not finished.
  • #18: Stress test and crash analysis delivered (fixed via #36/D109) and search design recorded in D89-91, but cross-session search implementation not evidenced in dev.
  • #24: Status board, always open.
  • #35: Spike reports A-C merged (d57516aec) but owner questions from SPIKE-B/C (session count, scaling, JPEG vs H.264, shared encoder) remain unanswered.
  • #37: vt-export parked after content-filter stops; WIP branch only, transactional resize added to scope, nothing merged.
  • #38: Owner question on server-side Snapshot coalescing (815 to 202 MiB per session) unanswered; no D### found; spike a82523c not in dev.
  • #41: Codex review was stopped by content filter; only fuzz targets ran clean, HTTP/WebSocket robustness checks still undone.
  • #42: fix-perf (967d278) is not in dev; merge plan pending a D130 check of the coalescing commits, so findings are not shown shipped.
  • #43: Fix merged (a20ae2989, in dev and prod) but owner retest with the real 1Password extension is still pending and the welcome-UI provider lane was handed off.
  • #44: Scroll fix 760490ce4 is in dev and prod; the last comment asks the owner to confirm on a real iPhone.
  • #45: Recovery backend deployed (5ed4f295c) but the recovery sign-in UI is held behind a preview opt-in (1c12295c6); production UI not shipped.
  • #46: D118 decision recorded (nested scroll + canvas gestures + accent border) but no implementation commits reference #46 and the issue was never confirmed done.
  • #47: Research report research/data-path/REPORT.md delivered, but its comment says no decision made; the owner questions are unanswered on the issue.
  • #48: Canvas/pill/popover design: prototype work on dev but the last comments are mid-work findings with no closing deliverable or owner sign-off.
  • #49: Idea for later (auto-install mesh clients), zero comments, not started.
  • #50: Predictive echo deployed only behind ?predict=1 (e19c4eb); default-on and felt-latency measurement on the India-Germany path not confirmed.
  • #52: Umbrella: returning sign-in still measured p50 ~3.96 s, p99 ~5.0 s vs 5000 ms budget (FAIL in latest runs); phase B (#67) not done.
  • #53: Merged in dev/prod (444b839e4) but the issue says real iPhone check remains owner-side and no confirmation is recorded.
  • #54: Compose mode (D130) has no comments and no commits; needs design grill and prototype.
  • #56: infraternal relay subcommand: no comments, no commits referencing it.
  • #58: Mesh-policy SSH block diagnosis and 'add a policy' feature: no comments, no commits.
  • #59: infraternal migrate command: no comments, design and grill first, not started.
  • #60: Svelte 5 migration (D142/D143): only a step-0 prototype instruction comment, no migration commits.
  • #61: Reconnect-to-resumed still measured p99 ~1009 ms vs 900 ms budget (FAIL) in the latest run.
  • #62: Cold access-screen render p99 ~1.9-2.0 s vs 1 s budget (FAIL) in the latest run.
  • #63: Echo-confirmed p99 ~364 ms vs 334 ms budget (FAIL) in the latest run; predictive echo is only behind a flag.
  • #64: Discovery coverage report delivered (research/discovery/REPORT.md on dev) but the issue says implement after the grill; implementation and owner questions remain.
  • #65: Reviewer-clock fix merged (5ed4f295c) but motion-audit is still observed failing in later runs (#74), so flakiness is not resolved.
  • #66: Phase A merged (fca6e093d) but the 5 RTT + 150 ms target is not shown met; #52 latest runs still p50 ~3.96 s at 300 ms RTT.
  • #67: Phase B (combined auth/attach protocol, 2 RTT + 150 ms) has no commits referencing it; waits on phase A.
  • #68: Design round G (users, groups, OIDC) settled in D150-D154 but implementation not done; no commits.
  • #69: Releases and auto-update (D155) implementation not done by this issue; no commits reference #69.
  • #70: Windows support is a later umbrella (priority raised by D161), not started.
  • #71: Ranking report delivered (research/bsd/REPORT.md) but deferred by owner under D161; to be picked up later.
  • #72: Design of the 'Gather extended information' switch: owner grill questions unanswered, no mockup.
  • #73: Same-session PTY flood still FAILs (p99 382 ms, growth 53 ms vs 16.67 ms budget) in the latest run.
  • #74: Long main-thread tasks in the 4x CPU motion audit not resolved; latest run failed earlier on a different motion assertion.
  • #76: Research and demo delivered (research/reflective-card on dev) but owner questions on permissions, export angle and CSS vs spike are open; product feature not built.
  • #77: Waiting on the owner to provide a Proxmox VM and NetBird key; test hosts later arrived (minipc01) but the described lab is not confirmed.
  • #78: S1 remote agents umbrella: 395 comments, last comments show add-host round 2 with issues left open for the orchestrator VM rerun; ongoing.
  • #80: Slice 1 crate merged (9c62448a3) but the issue states it stays open; slice 2 wiring and real packet filtering still to come.
  • #82: Add-host mockup: last comments are in-progress phone fixes with merge conflicts; no owner approval recorded.
  • #83: Recovery mockup: last comments are in-progress steps (token block missing, shared primitives); no owner approval recorded.
  • #86: Grill for deep links has no comments yet; DECISIONS only references it (#86 IDs), not decided or built.
  • #87: Add-host round 2 finished on release-addhost but comments say issues left open for orchestrator VM rerun; not confirmed merged/closed out.
  • #88: Research delivered and owner chose quinn+rustls+ring (D176), but promised follow-ups remain: matched-loss rerun and BBR at 300 ms RTT.
  • #89: Umbrella: canvas product merged but grill/containers-VMs-Kubernetes fit and later rounds still open.
  • #92: Resume p99 / same-session flood echo over budget; no comments, needs backend investigation.
  • #93: Owner answers recorded as D180, but exports (JSON, Markdown, image, Excalidraw) not confirmed built.
  • #94: Share studio: only the Machine-view mockup rounds exist (r2 with review fixes); real-device numbers owed and share studio not built.
  • #95: Idea for standalone infrafetch project; no comments, not started.
  • #96: Live sharing is an idea with open questions (core vs ee, link encryption); no comments.
  • #97: Break-it review reproduced open P2 (RC-6 grouped parked-terminal memory 6.4 MiB vs 2 MiB budget); no fix recorded.
  • #98: Canvas doc backend: review delta still in progress, fast gate failed only on binary size (D205); merge not confirmed.
  • #99: Independent review verdict "merge after fixes" with 3 open P2 (MS-R4/R5/R6); fix and merge not recorded.
  • #100: Session lifecycle: review has open findings (SL-R9) and last comment is a gate run; merge not confirmed.
  • #101: Later idea (history/log search); grill first, no comments.
  • #102: Self-update spec only has an owner note; implementation not confirmed.
  • #103: Relay/direct path still in run5 incremental slices; str0m path not in dev.
  • #104: Sessions mockup ported to canvas (6dd7ea2 in dev and prod) but the deferred-chunk limit raise to 58 KiB awaits owner approval.
  • #105: Backups spec with no comments; no evidence of finished work or numbers.
  • #106: Later: native PBS destination, needs library measurements and owner decision; no comments.
  • #107: Canvas streams review has open P2 CS-R6 (input writer removed under send pressure) and a native test failure noted.
  • #108: Mesh API deployed (db8c5270) but comments still carry open items: binary size over frozen budget and D198 lifecycle work in progress.
  • #109: Umbrella plan for canvas lanes; no comments, lanes (e.g. #98-#108) not all done.
  • #110: terminal_writers leak fix: no comments, no commit referencing it found in dev.
  • #111: Zero-downtime restarts: no comments or fix recorded.
  • #112: Plain names decided and deployed (1c3ae5f) but the audit of every remaining message is not confirmed complete.
  • #117: Single observation of a dropped window open under load; no comments or investigation.
  • #118: Test-only fix merged to dev (459070fe) but independent verification review is still running.
  • #120: System-limits test plan with no comments; not run.
  • #121: Target of 120 fps zoom/spam; round 5 deployed but p95 34.9 ms vs target and work continues.
  • #129: Mesh data plane umbrella: last handoff (dev-merge-d215 7d70353f) states acceptance blocked by static size, native same-session budget and a failing browser fixture; no green merge claimed.
  • #130: Mesh "why can't A reach B": idea, no comments, to be grilled before building.
  • #131: Mesh preview/undo of changes: idea, no comments, to be grilled before building.
  • #132: Mesh temporary access: idea, no comments, to be grilled before building.
  • #133: Mesh service discovery: idea, no comments, to be grilled before building.
  • #134: Mesh live paths on arrows: idea, no comments, not started.
  • #135: Mesh doctor: idea, no comments, not started.
  • #136: Mesh per-arrow traffic: idea waiting on the later metrics component, not started.
  • #137: Backup grill (growth, HA, auto rollback): no comments, measurement and grill not done.
  • #140: Browser-as-canvas-window research parked by owner decision D207; W1-W3 explicitly left open.
  • #141: No comments on the issue; dev has several #141 commits (focus framing, keyboard split, close X) but nothing on the issue confirms the three reported items (split stuck, focus after split, close bounce) are verified.
  • #142: macOS app plan: no comments, to be grilled once server is stable.
  • #143: CLI layer merged into dev (47f2b31b / 30487a0c) but the issue says a binary-size cut under D205 is a pending follow-up and D202 requires every component (relay, ssh, backup, ...) to use it.
  • #145: Account preferences store: no comments and no server prefs store found in origin/dev.
  • #146: App windows from a Mac: latest comment says blocked on the Mac (power/NetBird), no native root cause or fix claimed, issue remains open.
  • #147: Access rule list and explain(a,b): no comments and no implementation found in origin/dev.
  • #148: Feedback frames plugin: review ended with failing static size gate and unverified perf; not in dev, and D281 later redesigned in-app feedback via GitHub Discussions.
  • #150: Missing-glyph boxes on logout: no comments and no fix found.
  • #153: Research report pushed on branch research-prf (61482a0c, not in dev), recommendation made but no decision recorded; D206 still lists device keys as pending PRF research.
  • #154: D217/D244 CA handoff and key memory deployed (release-prod-5 5bf3f42e) but delta review HOLD with UP-01..03 (P1) deliberately deferred to release pipeline round 4.
  • #156: PXE grill: D253 recorded but the issue lists open owner questions and D-text says Boot Machines questions wait until mesh and dogfooding are stable.
  • #157: Release pipeline umbrella with 270 comments; latest branch dev-merge-r4 (261e4499) fails the static size gate and round 4 / UP-01..03 follow-ups are pending; not confirmed closed out.
  • #158: Fix deployed (1c08c4b7) but the delta review left a P2 (logs-body download has no retry) promised as a later fix-forward.
  • #159: Benchmark harness delivered, but an orchestrator-found harness bug (root scratch dir 0700, chown to measured user needed) is still noted as needing a fix.
  • #160: Fix deployed in release-prod-4 3aec316e, but a later delta review (b429e566, review-query-replies-r2) ended with verdict HOLD and a P1 with no resolution recorded.
  • #170: Multiplexer-skip work (archive tag multiplexer-skip 00299ee9f) is not in dev; D297 fit-review owner questions and D243 size overage were still pending.
  • #172: real-logins branch (tag b3e2b5b7f) is not in dev; review findings A-F (agent uninstall panic, UI fallback note, window identity, stale cells) and RLR-13 P1 remain.
  • #175: Size-cut push is partly delivered via #198/D264 (9e84426c4), but the S2 one-crypto-library decision (D257) and further cuts are not shown as done.
  • #176: prod-perf-fix (tag 28bbc0b6f) is not in dev; reconnect budgets D265 and the resume-token continuation were still pending.
  • #177: WASM client-core spike parked by the owner 2026-10-12 until mesh and app windows are stable; no result delivered.
  • #178: D236 session labels not found in dev; the last comments are a review with a failing job-head gate and the issue stays open.
  • #179: Queued issue with no comments; no sign of D230 account names work in dev.
  • #180: Queued issue with no comments; no WireGuard stack measurement found.
  • #183: Mesh mockup round 1 is on branch mockup-mesh-r1 and not merged; the owner answers were recorded but follow-up rounds were on #186 and not confirmed complete.
  • #184: perf-remeasure (e8b98754) is not in dev and scripts/perf-vm is absent; the VM browser timing harness is not delivered.
  • #186: Access-rules spec and mockup rounds are not in dev (no docs/design/access-rules.md); owner answers continue and the model is not approved.
  • #187: idle-rss branch (tag 2c5f0849b) is not in dev; the RSS attribution and reduction are unfinished.
  • #192: Research delivered and D259 recorded, but the owner asked for measurements on Q1-Q3 before deciding and the research-feels-instant follow-up is still unfinished.
  • #194: Host-inventory merge 85368c7fd is in dev and prod, but the "+" host picker (ui-new-window-host 79006368) is not in dev.
  • #197: Flood latency is still being measured; D268 records only that the flood server part ships, and review verdict was HOLD with unresolved gate failures.
  • #200: Evidence tooling and D262 exist, but the job was blocked on the final gate and the repo is still large (.git is 607 MB); the history rewrite is not done.
  • #202: Planning-only issue with no comments; D261 says logs nodes are planned next, and no plan was delivered.
  • #203: Cipher-order work has not landed in dev (no merge found) and the independent review C203-R1 (P2) was still open.
  • #204: Cheaper gate iterations work is on a branch with the gate still red on size; no merge to dev found.
  • #205: User-screen pool merged into dev (7ee79fb6d) but stays off by default until the D270 soak gates and the owner's go.
  • #206: Research on tmux and zellij issues has no delivered report visible; the latest comments are unrelated deploy notes.
  • #208: Only a plan comment; no measurements or report delivered.
  • #209: Terminal OSC work is not in dev; review findings OSC-R06 and OSC-R07 (P2) were open.
  • #211: D251 edge auto-update backend still in progress (head 643a5af84, UI/proofs/final gates pending, last comments are build-slot waits).
  • #212: UX gap audit with ranked gaps and continuing gate/measurement comments; no closing result or per-gap resolution recorded.
  • #213: Mac userspace mesh measurement failed (direct UDP to perf VM timed out); real mesh throughput not measured, nothing merged.
  • #215: Mockup issue (sidebar rename/remove, privileged consent) with only screenshots posted; no owner approval or decision recorded.
  • #216: Version pill/licensing mockup rounds posted with screenshots; no owner approval or merge recorded.
  • #217: Native spike (D238) final fast gate failed (size cap, server test); orchestrator reconciliation and owner review pending.
  • #218: History audit done but rewrite/cut-over not authorized or performed; waits on owner.
  • #220: Owner questions Q1-Q8 pending; no relay/NAT code built.
  • #221: Mesh SSH still under research/review; dependencies on process-per-user and login access not built; issue remains open.
  • #223: VNC research delivered with 7 open owner questions and failing gate rows; waits on owner.
  • #224: Person's card mockup first round posted for owner judgment; no approval recorded.
  • #225: Fix 492e1505a deployed to prod, but the stuck old agent field case moved to frozen #231 and no verified reconnect recorded.
  • #227: Close-reason logging shipped (5b2a1b9ee in prod) but the flicker fix on pill-flicker (9e01a0331) is not in dev/prod and server-side close trigger is unisolated.
  • #231: Frozen after round 8 with an unfixed P1 (late-created state directory); round 9 not started.
  • #232: Link underline merged and deployed (f627a2e09) but two queued follow-ups (CJK URL detection, clipped last column) remain.
  • #233: Join-over-SSH (D291) branch join-over-ssh 47619c27d is not in dev or prod; add-host work frozen.
  • #235: Re-add failure diagnostics still in progress: missing-account-after-channel case has a failing regression, not fixed or merged.
  • #237: Status truth shipped in release 4 (a906ede87) but a new launch-qa finding (status says Running in a crash loop) is unresolved.
  • #238: Audit found the D126 default fields hidden behind the extended-info switch; UI fix still needed and no merge recorded.
  • #239: CLI overhaul first slices deployed in release 4 but the proposal lists further build slices not yet done.
  • #240: Busy-store authority fix live in release 9.4 (b30bb81cf, in prod but not yet in dev) after review rounds; merge-back to dev and closing evidence pending.
  • #242: Own context menu and swipe actions: only mockup screenshots posted; no owner approval or merge.
  • #243: Server-held layout shipped (release 6, 9.2) but p3/settings follow-up head f63884f58 is not in dev or prod.
  • #247: Launch QA run stopped at S2 by the #252 P1; final verify fast failed on JS transfer budget; remaining personas not run.
  • #250: Protocol-close fix partially deployed (9.2) but follow-up fix-sync-journal head 573dc412e has open P2 and is not in dev or prod.
  • #251: Research delivered but owner design questions Q1-Q6 are pending.
  • #252: P1 setup-token fix job stopped mid-gate (TypeScript check/verify still running or paused); not merged.
  • #79 (left open, borderline): Grill delivered: owner answered Q1-Q17 and the result is recorded in D163-D167 (commit 159f43797 in dev); build work tracked in #80/#81.
  • #169 (left open, borderline): Add-host rounds 1-3 and start-fresh deployed (5bf3f42e in dev and release-prod-live); NetBird-SSH-only case continued and fixed in #195; speed work moved to #189/#190.
  • #207 (left open, borderline): Test fixes fast-forwarded into dev (21c31f906, verified in the log) and the diff was reviewed by the orchestrator.
  • #230 (left open, borderline): Add-host Retry fix released in 173a9ba14 (ancestor of dev and prod, release 7) with staging proof; only the owner's retest remains.
  • #244 (left open, borderline): Phone resume recovery shipped in release 6 (8f8c84e37, ancestor of dev and prod) with owner test steps listed.
Orchestrator handover, written 2026-10-28 ~23:45 IST by the outgoing orchestrator (Claude, session `session_01GHLxFr6gB7AcU2GRKHB6DE`). The owner is moving the orchestrator to a new machine. This issue is the single starting point for the next orchestrator: read it top to bottom before doing anything. **Read first, in this order:** `CLAUDE.md` → `DECISIONS.md` (canonical; D1–D310 on dev) → `docs/ARCHITECTURE.md` → `docs/PROCESS.md` → `docs/design/TASTE.md` (before any UI) → this issue → #24 (status board, kept current). Every time below is in IST. The owner wants all times in IST (CEST + 3:30; CET + 4:30 since 25 Oct). --- ## 0. Before infraternal-dev is shut down or decommissioned: copy these off it The old orchestrator host is `infraternal-dev` (NetBird 10.69.69.125, LAN 10.70.3.157, user `kayg`). The git history is safe: everything is pushed to Forgejo, and unique branch tips are kept as `archive/2026-10-28/*` tags. **The files below exist only on that machine.** Copy them to the new host before it goes away, as the same user, preserving permissions. None of them may ever be posted to fj. | Must copy | Path on infraternal-dev | Why | |---|---|---| | **Handover bundle** | `~/infraternal-orchestrator-handover-2026-10-28.tar.gz` (mode 600) | orchestrator state, job prompts, deploy/preview scripts (incl. the local preview-auth patch the serve-* worktrees carry), systemd units, memory (§4). SHA-256 `816fe0bf2339047735be369b5691bef3bb5a99440774d7984bddf477b38008a2` | | 1Password service-account token | `~/.config/infraternal/op-service-account-token` | never display it; copy the file as is | | Forgejo push key | `~/.ssh/git.kayg.org`, `~/.ssh/git.kayg.org.pub` | pushes to `ssh://git@10.69.69.52:2222` | | GitHub mirror key | `~/.ssh/github.com`, `~/.ssh/github.com.pub`, plus the `Host github.com` block in `~/.ssh/config` | mirror only; push only on the owner's say-so | | fj (Forgejo CLI) token | `~/.local/share/forgejo-cli/keys.json` | `fj` and `issue-attach` | | Codex accounts | `~/.config/openai1/`, `~/.config/openai2/` | or run a fresh `codex login` on the new host | | Firecrawl CLI login | `~/.config/firecrawl-cli/` | web research | | Claude Code state | `~/.claude/` (1.7 GB) and `~/.claude.json` | settings, skills, `agents/` (designer), and `projects/-home-kayg-Developer-infraternal/` (the memory and this session's full transcript `15bbfdbc-0147-45a8-a97b-90d4ce6d4d03.jsonl`) | | git config | `~/.config/git/` | identity `kayg` / team@kayg.org | | Optional | Path | Why | |---|---|---| | Full job folders | `~/.local/state/codex-jobs/infraternal/` (14 GB) | every past job's full log and result; the bundle holds only what the queue needs | | Build cache | `~/.cache/sccache-infraternal/` (43 GB) | saves hours of cold builds; can be regenerated | | Owner's add-host test fixture | `~/.local/share/infraternal`, `~/.local/share/infraternal.bootstrap-cache`, `~/.local/share/infraternal.dev-2026-10-06` | the owner's live acceptance case for add-host (§2). **If the machine itself is decommissioned, not just the orchestrator moved, this case is lost unless it is copied or the VM is kept** | Example, run on the new host as `kayg`. It copies everything in the "Must copy" table into the same paths, keeping permissions. sshd is not running on infraternal-dev, so use NetBird SSH: ``` netbird ssh --no-browser kayg@10.69.69.125 'cd ~ && tar czf - infraternal-orchestrator-handover-2026-10-28.tar.gz .config/infraternal .ssh/git.kayg.org .ssh/git.kayg.org.pub .ssh/github.com .ssh/github.com.pub .local/share/forgejo-cli .config/openai1 .config/openai2 .config/firecrawl-cli .claude .claude.json .config/git' | tar xzf - -C ~ ``` Merge the `Host github.com` block from the old `~/.ssh/config` by hand. Afterwards, check that `~/.config/infraternal/op-service-account-token` is mode 400 and its folder 700. **Then, on the running system:** 1. Stop the services on infraternal-dev: `systemctl --user stop serve-demo serve-recovery serve-clusters status-preview` and the three `infraternal-*` timers. 2. `infraternal-agent.service` on infraternal-dev is an infraternal host in production. If the machine goes away, remove it in production's host list, or expect it to show as offline. 3. Tell the owner the new preview links (the old ones are `http://10.69.69.125:47361`, `:47381`, `:47401`, `:47391`). --- ## 1. Where things stand (2026-10-28) **Nothing is running.** No Codex jobs, no subagents. The scheduler is paused (`orchestrator/pause` exists). Nothing has happened in the repo since 2026-10-22 ~08:45 IST, when Codex sign-in expired. **Production** (monitor01, `root@10.69.69.50`, https://infraternal.kayg.org): | | | |---|---| | Binary | release 9.4, `b30bb81cf`; MainPID 211431 since 2026-10-22 06:31 IST; exe SHA-256 `71ced02f01ef832a…` (= artifact) | | Web | `db4add238` = 9.4 + mesh-view-overlap + mesh-drag r4 + a test typing fix. Swapped in live with no restart | | `release-prod-live` | `db4add238` (always = what runs in prod) | | healthz | 200 (checked 2026-10-28 23:15 IST) | | Public access | since 2026-10-22 20:10 IST (#253): the Traefik allow-list was removed. The infraternal agent port 7443 stays NetBird-only (D64) | | CA | re-locked by the 9.4 restart on 2026-10-22. **The owner was asked to unlock it; not confirmed.** | **Branches** (all remote branches were cleaned today; see §6): | Branch | Head | Meaning | |---|---|---| | `dev` | `81962ab09` | all development. **Does not yet contain release 9.4 or the web update** (see §3, task 1) | | `release-prod-live` | `db4add238` | production | | `release-9-4` | `b30bb81cf` | 9.4 release line (binary in prod) | | `main` | `5a0ec9ee7` | stable builds only; stale since 2026-10-10. Owner rule: main = stable builds; nothing promoted since | | others | | the paused jobs' branches (§3) plus `proto-status` (a preview) | **Codex:** `codex login status` reports both `~/.config/openai1` and `~/.config/openai2` signed in. On 2026-10-22, openai1's refresh token had expired (401) and openai2 hit its usage limit (reset 2026-10-27 07:32). The owner may have run `codex login` since then. Before relaunching anything, run one small `codex exec` per account to confirm it works. --- ## 2. Owner rules and standing orders (verbatim where quoted) Canonical decisions are in `DECISIONS.md`. Only the owner can change one. **The decisions that drive the queued work:** | Decision | What it says | |---|---| | D64 | Managed hosts open no inbound ports; infraternal agents dial out | | D206 | Iteration loop | | D217 | Updates never lock the CA; deploys use an in-memory handoff. Not built yet (#154), which is why every restart still re-locks the CA | | D241 | Auto-started multiplexers give way to infraternal sessions | | D243 | Binary size goes down, not up | | D250 | How changes reach production | | D251 | Production follows edge and updates itself (#211) | | D262 | Evidence lives on the issue | | D288 | `infraternal mux`: sessions automatable from inside a session (#251) | | D291 | New hosts join through the add-host SSH connection | | D295 | Undoing a failed server update stays simple | | D309/D310 | The latest owner answers | These orders and constraints are **not** in DECISIONS.md, or only partly: - **Latest work order (2026-10-21):** "Work autonomously and parallelly through your roadmap. Use Firecrawl to research, web search any information that you might need. Use Codex subagents for backend tasks and your own subagent, Sonnet 5.5, for UI and UX tasks." - **Priority (2026-10-21):** "let's prioritise running mux and mesh from the client side instead of the add-host side." The owner picked "Finish r8, then freeze". Add-host (#231) is now **frozen** after round 8. The client-side work is laptop CLI mux/attach plus a host joining by itself (#251, D288). - **Tonight's roadmap (2026-10-21):** it must not need owner decisions; owner questions wait for the morning, with concrete examples. - **Deploy without asking:** reviewed and staged fixes go to prod without the owner's OK. Then tell the owner to unlock the CA, because every server restart re-locks it. A web-only deploy swaps live; verify it by MainPID, not by restart count. - **Break-it reviews:** every Codex backend job gets a review that tries to prove it wrong with failing tests. Use neutral wording ("defensive audit with test cases"), never "attack/exploit/attacker". The content filter kills such jobs (#231 r8 review died this way). - **Merge bar (D206 iteration loop):** merge when there is no P0/P1, with delta reviews; prod is the test target. Before any merge the user can see, check that the break-it review is clean, there are perf-VM numbers, the size does not grow, the diff has been read, and a real demo video exists. - **D243 size bands:** growth under 16 KB the orchestrator approves; 16–64 KB needs a written reason; over 64 KB needs the owner. Binary size caps in `scripts/verify.d/rust.sh` ratchet to measured bytes, with no allowance. - **Codex fast mode is ON** (owner 2026-10-15/19, "Codex jobs fast from now on"): flag file `orchestrator/fast-mode`; jobs run with `-c service_tier="priority"`. - **No invented limits:** every limit must trace back to an owner message or a hard safety need. - **History:** atomic commits; push every commit. Commit trailers: `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>` and `Claude-Session: <session url>`. No other products' names in code or comments. - **Tracking:** one fj issue per piece of work. Comment at start (branch + base), at each finding, when blocked, and at the end (head + gate output verbatim). Evidence (screenshots, logs) goes on the issue via `issue-attach`, never into the repo (D262). Jobs never close issues; the orchestrator does when the owner asks. - **Foundational libraries** (terminal state, crypto, DB, transport, renderer): never chosen inside a job. They go to the owner as a grill question with measured trade-offs. **Security and access constraints (verbatim from the owner or from hard safety needs):** - **1Password service-account token:** `~/.config/infraternal/op-service-account-token`. Never cat or display it; pass it only as an env var. Vault "Apps / Infraternal" (Cloudflare token). Orchestrator only. - **No secrets** in the repo, logs, argv or URLs. Fewest secrets: use a credential once, seal it end to end, never store it. - **Host keys:** never accept a changed host key. `ssh monitor01` fails host-key verification; use `netbird ssh --no-browser root@10.69.69.50`. - **Test hosts:** "minipc01, ash1, ash2 can be connected via ssh on monitor01, you're free to access them via there and then clean up known hosts on monitor01 afterwards! you may not modify any vms or containers on minipc01, ash1, ash2. you're free to test infraternal agent though." On minipc01 (Proxmox), create and delete only our own `infraternal-test-*` guests. - **infraternal-dev (10.69.69.125)** is the owner's acceptance host. From `rules.md`: "no contact of any kind from jobs: no ssh, no host-key probe, no ping, no inventory scan". When you list hosts from production data, filter it out first. - `/home/kayg/.local/share/infraternal*` on it is the owner's live add-host test fixture: never modify, move or delete it (owner: "please don't, rather build the better UX and let me try if it works!"). - **It is also the old orchestrator/build host** (see §5). Once the orchestrator moves, treat it like any other owner host. It stays an acceptance test case. - **MacBook `kayg@10.69.69.101`** (the Mac VM also runs on it; one lock for both): - Connect with `netbird ssh --no-browser` only, never `--tty`. - Take `/tmp/infraternal-bench.lock`; stop below 30% battery. - Never change TCC, trust or keychain settings; never click prompts. - Capture only inside the signed companion via `open -n -W`. On any prompt or denial, stop and never retry. - Restore the shipped helper (SHA-256 `352d671b823b352f3c12ad1f71dcf6c55e82e5857f0b7c15956f2e404c1858cc`). - Test keychain `~/Library/Keychains/infraternal-signing.keychain-db`, unlocked in the same command right before each codesign. Sign by SHA-1 `3E6AE7C6BA4CFE1E6280535EB93CF7A7959A7BA6`. - Give cleanup proof with every PID. - **Perf VM `root@10.69.69.208`** (infraternal-perf-test, ours alone, 4 vCPU): use netbird ssh and `flock /var/lock/perf-bench.lock`. All perf numbers come from here. - **Production DB:** read-only, via monitor01 (`/var/lib/infraternal/infraternal.sqlite`; sqlite3 is not installed there). - **GitHub mirror `kaygdotorg/infraternal`:** push only when the owner says so. Issues stay on fj. #218 tracks the history secrets audit before any public push. --- ## 3. Work queue, in priority order ### 3a. Owner actions (only the owner can do these) 1. **Unlock the CA** in prod (pending since the 9.4 restart on 2026-10-22), if not done already. 2. **Codex sign-in on the new machine:** `CODEX_HOME=~/.config/openai1 codex login` (and openai2 if needed). 3. **Owner questions parked for a grill** (each has recommended answers and examples on its issue): #251 Q1–Q6 (client-side mux/mesh), #220 Q1–Q8 (NAT traversal/relay), #146 encoder question (app windows). 4. **MacBook:** it needs to be charging (≥30%) before the parked Mac lane `parked/53-app-windows-second-capture` (#146) can run. ### 3b. First orchestrator tasks on the new machine 1. **Merge production back into dev** (#24). `git merge origin/release-prod-live` into `origin/dev` conflicts in exactly two files: - `scripts/verify.d/rust.sh`: the static size caps. dev has 21,389,232 / 19,561,656; 9.4 has 21,389,424 / 19,562,296. - `docs/PROCESS.md`: the size-ceiling table and its text. Resolution: build the merged tree with the pinned static recipe (`scripts/static-cargo` / auth-toolchain Zig, as in the 9.3/9.4 release prompts), then set both caps to the **measured** bytes. Keep both PROCESS.md paragraphs and add one for the merge. Run `scripts/verify fast` with `CARGO_TARGET_DIR` set. The original job spec is in the bundle at `jobs/merge-release94-dev/prompt.md` (Luna max). 2. **Resume the stopped jobs** (they stopped mid-task when Codex sign-in expired on 2026-10-22). Use `orchestrator/resume-job.sh <job> <model> <effort> <wip-sha>`. Its continuation text says "host resize"; edit it to say "Codex sign-in expired / host move". Each job's WIP commit must be `reset --soft HEAD~1`-ed, then split into atomic commits. | # | Job | Issue | Model / effort (fast) | Branch → head | State when stopped | |---|---|---|---|---|---| | 1 | setup-token-data | **#252 P1** (launch QA S2) | Luna max | `setup-token-data-wip` 21cb39947 (base dev) | Rust and TS done and tested: the setup screen prints `sudo infraternal server setup-token --data /var/lib/infraternal` for a system service. The final `scripts/verify iterate`/static size step was not finished | | 2 | revocation-per-account-r2 | #240 | Sol medium | `revocation-per-account` 57185fd9b (+6,304 B) | Review HOLD (`review-revocation-per-account` f52fd309b): R1, a logout waiter cancelled after commit loses the notification; R2, bootstrap keeps its socket when the notification source is lost. Round 2 had just started | | 3 | review-sync-p3s | #250 | Sol medium | `sync-p3s` 3b94b1f28 (+0 B, gate PASS) | Break-it review had just started | | 4 | sync-journal-r3 | #250 | Sol medium | `sync-journal-r3` 573dc412e | Perf VM: 6/6 pass (max save 12.2 ms / flush 2.3 ms at 60 windows). The final full gate was queued behind the merge-base native build. Merge-base size measured: 21,389,296 B, delta 0 | | 5 | edge-autoupdate-resume | #211 (D251) | Sol medium | `edge-autoupdate-resume-wip` d4a1bf6ea (local branch `edge-autoupdate-r2`; remote `edge-autoupdate-r2` 643a5af84 is older) | Edge auto-update round 2 in progress. A partial review exists: `review-edge-autoupdate-d295` b1555e44f (171 lines of recovery tests, WIP) | | 6 | d241-shell-marker | #251 / D241 | Luna max | `d241-shell-marker-wip` e6b0c5a16 (remote `d241-shell-marker` e3a06ba42 is older) | Perf (perf VM, 2 runs: p99 −1,462 µs / −793 µs) and size (−192,864 B) are done. "Profile commit and final gates are next" | | 7 | broad-real-suite-triage | #207 | Luna max | `broad-suite-triage` d69ce64ed | Triage of the broad real-browser suite; continue from its last #207 comment | | 8 | merge-release94-dev | #24 | Luna max | none yet | See 3b.1. It looped on Codex errors, so it was stopped | review-mesh-drag-r3 is **not** needed (a Sonnet review covered it; the drag fix is in prod). 3. **Follow-ups found in the last rounds** (no job yet): - Mesh pins (P3): a pin for a host that was already gone at page load is never pruned and counts toward the 256-pin limit. In `apps/tracer/src/canvas/mesh/api.ts`, pruning only drops hosts that disappear while the page is open. - The WebKit 1,000-host pin-limit test fails, and already did before mesh-drag round 4. WebKit touch/pen drags can't be measured in headless mode. - `scripts/release/build-metadata.test.ts:162` fails depending on the checkout path (it passes in the dev worktree). Backend fix. - #237: a crash loop while `infraternal status` says Running (found by the launch-QA dry run, #247). - #253: queue a defensive audit of everything reachable without signing in now that the site is public (rate limits on sign-in, setup and join, header handling behind Traefik). 4. **Frozen, do not resume without the owner:** #231 add-host re-add. Round 8 review HOLD (P1: a state directory created between join and activation is adopted, then removed by rollback). WIP regressions on `review-readd-r8-wip` 287065c2f; round-9 plan on #231. Nothing from #231 is in prod. 5. **Parked lanes** (specs in `orchestrator/parked/`): `53-app-windows-second-capture` (#146, base `app-windows-second-stream`, needs the MacBook), `90-research-wasm-core` (#177, base `research-wasm-core`). 6. The orchestrator `queue.md` has the long-tail queue, and `resume.md` has the per-day notes. Both are in the bundle. --- ## 4. Tooling outside git (bundle on the old host) **Bundle (not attached here, because this repo and its issues are public):** `~/infraternal-orchestrator-handover-2026-10-28.tar.gz` on the old host infraternal-dev (mode 600; 476 KB; sha256 `c37d785b76f08ea717a0…`). **Copy it to the new machine by hand and keep it private.** It holds the orchestrator memory and job prompts, including the Mac's test-keychain password. It was scanned for tokens and private keys and holds none. It contains: | In bundle | Restore to | What | |---|---|---| | `orchestrator/` | `~/.local/state/codex-jobs/infraternal/orchestrator/` | `scheduler.sh` (slot scheduler; `ready/NN-name/{meta,prompt.md}`, `meta` = `MODEL= EFFORT= BASE= WEIGHT= ISSUE=`), `rules.md` (preamble put at the top of every job prompt), `run-job-retry.sh`, `resume-job.sh`, `build-slot` (serialises heavy builds), `disk-guard.sh` (scheduler needs ≥ 80 GB free), `branch-sweep.sh` (nightly: deletes branches merged into dev), `scratch-sweep.sh`, `issue-attach` (uploads evidence to an issue; the token is read from fj's key store), `git-ssh` (no ControlMaster; key `~/.ssh/git.kayg.org`), `pause`, `fast-mode`, `priority-jobs`, `queue.md`, `resume.md`, `parked/`, `held/`, `launched/` (every past job's meta + prompt), `deleted-branches.log` (SHA of every deleted branch) | | `jobs/<job>/` | `~/.local/state/codex-jobs/infraternal/<job>/` | prompt, run scripts, result and log tail for the 8 stopped jobs + release-9-4 (the release recipe) | | `preview/` | `~/.local/share/infraternal-preview/` | **`deploy-prod.sh`** (the only deploy path: refuses a SHA that doesn't contain `release-prod-live`, backs up the DB, rolls back automatically if the server doesn't come back healthy), `publish.sh`, `serve.ts`, `preview-forward.ts`, `perf-watch.py` | | `systemd-user/` | `~/.config/systemd/user/` | timers `infraternal-{disk-guard,scratch-sweep,branch-sweep}`; owner previews `serve-demo` (47361), `serve-recovery` (47381), `serve-clusters` (47401), `status-preview` (47391); `infraternal-agent.service` (this host as an infraternal host) | | `claude/memory/` | `~/.claude/projects/<repo-path>/memory/` | the orchestrator's memory: 70+ files of owner preferences and rules. Start from `MEMORY.md` | | `repo-untracked/.cargo/rust-lld` | `<repo>/.cargo/rust-lld` | untracked linker wrapper in the old main checkout (finds the pinned toolchain's LLD) | | `claude/agents/`, `claude/skill-codex-subagents/` | `~/.claude/agents/`, `~/.claude/skills/codex-subagents/` | `designer`/`designer-high` subagent definitions; the Codex job launcher skill (names, systemd units, model table) | **Secrets to bring by hand (not in the bundle; exact paths and a copy command are in §0):** - the 1Password token file; - `~/.ssh/git.kayg.org` (Forgejo push key); - fj's key store `~/.local/share/forgejo-cli/keys.json`; - Codex homes `~/.config/openai{1,2}` (or a fresh `codex login`); - NetBird enrolment for the new machine; - GitHub key (mirror only). **Rebuild checklist for the new machine:** 1. Install the toolchain via nix (`bun`, `fj`, `firecrawl` at `~/.npm-global/bin/firecrawl`, Codex at `~/.npm-global/bin/codex`; the skill calls the vendored musl ELF by absolute path). 2. Enable linger with `loginctl enable-linger`. 3. Set up sccache. 4. Builds only on SSD; the old host had 12 vCPU / 64 GB, with ~8–9 jobs at once and ≤ 5 build-heavy. 5. Clone `ssh://git@10.69.69.52:2222/kayg/infraternal.git`. Worktrees go in `<repo>/.worktrees/<name>`. 6. Recreate the `serve-*` worktrees at their pinned SHAs: `serve-demo` d9564c89a, `serve-recovery` e52ef1388, `serve-clusters` 30faecc14 (each gets a one-line, never-committed `apps/tracer/test/mock-server.ts` tweak, applied by `preview/preview-auth.patch.sh <worktree>`; also `proto-status` 145bf0e93). Run `bun install --frozen-lockfile` in `apps/tracer`. 7. The preview forwarders bind to `10.69.69.125` (the old host's NetBird IP). Change them to the new host's IP and tell the owner the new preview links. 8. Copy the systemd units and the timers. 9. **Update `orchestrator/rules.md` for the new host.** It still says "infraternal-dev is the machine you are running on", lets dev servers bind to `10.69.69.125`, and says to benchmark locally on infraternal-dev. Change these to the new host's name and IP. Perf numbers still come from the perf VM (10.69.69.208), which the orchestrator books. 10. Paths in the scripts are absolute under `/home/kayg/…` (deploy-prod.sh uses `/home/kayg/Developer/infraternal` and `/tmp/claude-1001`). Keep the same user and paths, or update them. 11. Remove `orchestrator/pause` only after the Codex check in §1. --- ## 5. Infrastructure map - **Old orchestrator/build host:** `infraternal-dev`, NetBird 10.69.69.125, LAN 10.70.3.157, 12 vCPU / 62 GB, Debian 13, `/dev/sda1` 492 GB SSD (117 GB used after cleanup). It also runs an infraternal agent (`infraternal-agent.service`) and the owner previews. - **Production:** monitor01, `root@10.69.69.50`, public `37.157.255.66`. Traefik in Docker (`kayg-traefik`); route `/opt/infra/traefik/dynamic/infraternal.yml` → `http://172.18.0.1:7011`. Server unit `infraternal-server` (`--behind-proxy --trusted-proxy 172.18.0.0/16 --remote-listen 10.69.69.50:7443`). Backups in `/var/backups/infraternal/`. healthz `http://172.18.0.1:7011/healthz`. - **Forgejo:** git.kayg.org (SSH `10.69.69.52:2222`). Use `fj --host https://git.kayg.org`; `issue create --repo kayg/infraternal --body-file f "<title>"`; `issue view kayg/infraternal#N [comments]`; `issue search --repo kayg/infraternal --state open`. **Known:** large pushes print `Forgejo: Internal Server Error` from a hook, but the refs land; verify with `git ls-remote`. - **Perf VM:** `root@10.69.69.208`. **MacBook:** `kayg@10.69.69.101`. **Test hosts:** minipc01 / ash1 / ash2, reached via monitor01. - **Cloudflare DNS:** `infraternal.kayg.org` A → 37.157.255.66 (the owner changed it on 2026-10-22). Traefik uses DNS-01 via Cloudflare; the certificate is valid until 2027-01-02. --- ## 6. Cleanup done today (2026-10-28) - **Unfinished code committed and pushed as WIP:** - Uncommitted changes in 11 worktrees, each as a WIP commit: - 7 with code changes: client-resilience 4660395e3, mesh-relay 1e490da43, process-per-user d98143b1a, readd-minimal-r5 179b84b0e, review-edge-autoupdate-d295 b1555e44f, review-mesh-api cd5e068db, shim-relocs b9251c278; - 4 with untracked files: research-webrtc 704f8c493, review-mesh-drag-r3b 8ab0177ab, review-remote-agents dedd9b91f, ui-gap-fixes 9de070726. - Locally modified capture PNGs (regenerated test screenshots) were discarded. - **Branches archived, not lost.** Every branch tip that wasn't already in `dev`, `main`, `release-prod-live` or a kept branch is now a lightweight tag `archive/2026-10-28/<branch>` (290 tags; `-remote` suffix where the remote tip differed from the local one). Each was verified on Forgejo against the local commit ID. Then: - 354 remote branches were deleted, leaving 21; - 438 local branches were deleted, leaving 21; - 328 worktrees were removed, leaving 23: main, the 18 job/lane worktrees, the 3 `serve-*` previews and `proto-status`; - about 67 GB of disk was freed. - **Restoring a branch:** `git push origin archive/2026-10-28/<name>:refs/heads/<name>`. Every deleted SHA is also listed in `orchestrator/deleted-branches.log`. - Kept as-is: `.worktrees/infraternal/` (52 KB of old spike files, not a worktree) and `.worktrees/release-pipeline/target/root-proof` (root-owned leftover; needs `sudo rm`). - **Issues closed today:** 90 issues that were verified done; each has a closing comment with its evidence. Appendix A lists them, and Appendix B gives the state of every open issue. - **Lost in the cleanup:** untracked perf-VM browser evidence folders from 2026-10-12/13 in the old `prod-perf-fix` worktree (`evidence/perf-vm-browser/*`, `.job-state/`) were deleted along with that worktree. Whatever had been attached to its issue then is still there; the rest is gone. --- ## 7. Gotchas that cost time - **git.kayg.org/kayg/infraternal is public** (`private: false`). Anyone can read it, its issues and every evidence attachment, without signing in. Never put credentials, private keys, test passwords or personal data in an issue, a comment or an attachment. The owner may want to review this (see #218 for the mirror audit). A full-history secrets scan on 2026-10-29 (gitleaks over every ref and tag, plus a regex pass) found no live secret; only false positives, test keys and local test-server cookies. Details are on #218. Issue bodies, comments and attachments were not scanned. - The Bash tool runs **zsh**: - `$S:refs/...` is read as a zsh modifier, so write `"${S}:refs/..."`. - `for x in $VAR` does not word-split; use an array or `${=VAR}`. - An unmatched glob is an error. - `scripts/verify fast` needs `CARGO_TARGET_DIR` set; without it, the release-plugin tests fail. Run heavy gates through `orchestrator/build-slot`. - Web build: in `apps/tracer`, run `bun install --frozen-lockfile && bun run check && bun test … && bun run build && bun run transfer-budget.ts`. Critical-JS Brotli cap is 256,000 B (prod is at 255,839). - Deploy: `deploy-prod.sh <sha> [--bin x86] [--bundle dir] [--web dist]`. Never pass `--arm` together with `--bundle`. Verify with `systemctl show -p MainPID infraternal-server` + `sha256sum /proc/$pid/exe` + healthz, then force-push `release-prod-live` to the deployed SHA and comment on #24. - Codex's content filter stops VT/FFI/auth jobs that use attack words. The recipe is in memory `codex-filter-vt-jobs.md`. - WebKit headless has no backdrop blur: check glass effects in Chromium or on a real iPhone. - Owner previews ("serve-*") are stable copies: jobs never restart them. --- ## Appendix A. Issues closed on 2026-10-28 Issues closed on 2026-10-28 (90; each has a closing comment with its evidence). Done items were checked by `git merge-base --is-ancestor` against dev/prod, or a recorded decision or superseding issue was cited. Borderline cases were left open: #79 (Q18 open), #169 (260 comments, findings may live elsewhere), #207 (its triage job is paused), #230 and #244 (owner retest pending). #10 #11 #12 #13 #15 #16 #17 #19 #20 #21 #22 #23 #25 #26 #27 #28 #29 #30 #31 #32 #33 #34 #36 #39 #40 #51 #55 #57 #75 #81 #84 #85 #90 #91 #113 #114 #115 #116 #119 #122 #123 #124 #125 #126 #127 #128 #138 #139 #144 #149 #151 #152 #155 #161 #162 #163 #164 #165 #166 #167 #168 #171 #173 #174 #181 #182 #185 #188 #189 #190 #191 #193 #195 #196 #198 #199 #201 #210 #214 #219 #228 #229 #234 #236 #241 #245 #246 #248 #249 #253 <details><summary>Reason per issue</summary> - #10: Contracts (shim, plugin, bootstrap, wire) drafted at d731975 which is in dev and prod; wire.md updated for D63 and implemented by the merged lanes. - #11: Workspace/gates merged as 39630d6 (in dev and prod) after orchestrator split into atomic commits; gate PASS in the issue. - #12: Spike result delivered and owner decided WebSocket-only on leg A in D63 (custom codec stands); wire.md updated fd9a498. - #13: Renderer spike delivered; ghostty-web was chosen (D14) and pinned from source in D109/#36 (845c293 in dev and prod). - #15: Codec lane merged to dev as 7174d6b (verified ancestor of dev); P1 split-header finding fixed and fuzz clean. - #16: Shim lane merged to dev as 7ae0b83 (verified in dev); review findings answered, gate PASS. - #17: Auth lane merged to dev as 7668d0c (verified in dev) after rework and second review verdict MERGE; D86 approved crypto. - #19: Server core merged and deployed (t-server 7551ff6/f1b996c and integration acf83c7 are in dev and prod); audit findings fixed under #28/#29. - #20: Agent core merged (6c64ede, 48d7bab in dev and prod); review R3 fixed (e72589e), integrated via #34. - #21: Web client merged (58040df, faa3154, ff505cb in dev and prod); audit F1 fixed in #29 round. - #22: libghostty-vt switch (D94) merged via a5250c3 (#22,#26,#34) in dev and prod; remaining resize-refusal limitation tracked on #37. - #23: VNC research delivered (137a569, cdc7c59 in dev); owner answers recorded as D104-107, follow-up spike is #35. - #25: fix-vt snapshot fidelity finished at 234baf0 (in dev and prod); nightly 19,864 replays, 0 divergences; second review was #26. - #26: Second review verdict rework accepted as known limitation by orchestrator; real fix (transactional resize) is tracked on #37 and 565105f is in dev and prod. - #27: Idle canary fixed at 48d7bab (in dev and prod); zero idle wakeups measured, gate PASS. - #28: Six server findings fixed and merged (7551ff6, acf83c7 in dev and prod); the beyond-120s writer gap is a documented owner choice. - #29: Auth audit rounds ended with fixes merged (7551ff6, faa3154 in dev and prod); A3 answered as D108 and regressions added. - #30: D108 single passkey prompt implemented (faa3154 in dev and prod) and independently verified by review-auth3. - #31: Superseded by #36/D109: owner chose no upstream release request and a pinned source build, shipped in 845c293. - #32: Sign-in/onboarding port merged as 56bb6f51d (in dev) with D98 updated in d2ce17a; tests and screenshots posted. - #33: Pane cards, branched tree and tooltips merged in 0c47cd42e (#33, #36) which is in dev and prod; production screenshots and gate posted. - #34: Lanes merged into dev and deployed to monitor01 (acf83c7 in dev and prod; prod now runs release 9.4 line); later comments are layout-table notes. - #36: Ghostty pinned to a source build at 845c293 (in dev and prod), accepted by orchestrator, D109. - #39: Snapshot paint bug fixed and merged as 7d17212 (in dev), regression snapshot-resume.spec.ts passes. - #40: Fixes merged as 6016ff7a0 (in dev); owner answered the two design questions as D111 and D112. - #51: Setup-token socket retired; merged as b5e677956, deployed to monitor01 and verified socket absent after start with an owner present; b5e677956 is in dev and prod. - #55: Gate merged as ac03b1913 (verified in dev and prod): 300 ms RTT default profile and input-to-photon gate; remaining budget misses were split into #52 #61 #62 #63 #73 #74. - #57: Size-budget commit 2de8da2 taken and measured-ratchet follow-ups landed (4b2cd2ad4, e273fd309 in dev); mold measured as no gain and deliberately not merged, sccache set up. - #75: Lossless local terminal themes deployed (9e09924, then 7eca507 after the cursor regression was fixed by bd12725fe); all verified in dev and prod and checked on the production build. - #81: Engine measurement report research/mesh-engines/REPORT.md delivered and owner decisions recorded as D169 (a24ee0bd6 in dev). - #84: Cell metrics and on-grid glyphs deployed (7eca507) and the letter-spacing hotfix bd12725fe deployed; both verified in dev and prod with cursor position checks at DPR 1/2. - #85: Palette keyboard selection fix merged (e3e00eaea in dev) and deployed to prod in 5ed4f29 (ancestor of release-prod-live), owner-visible deploy comment, no open findings. - #90: Font picker, per-font weights and glyph placement merged into dev (ac10a5e22, 4c5c08a16, 459f10a19); fonts and Maple Mono weight present in dev tree. - #91: Owner answers recorded as D177; both toolbar rows, pill swipe, in-place toolbar editing and device layout merged in dev (c8708d9b2, f27bafdc8, 49bbe5bf2). - #113: Wheel/touch goes to the locked terminal: 664e45e88 in dev, deployed in 1c3ae5f (ancestor of release-prod-live). - #114: Sidebar overlays the canvas: fix b054f237d/1c3ae5f40 in dev, deployed in 1c3ae5f (in release-prod-live). - #115: Droplet split/close finished (head b0c9a4c, gate exit 0), droplet code present in dev (apps/tracer/src/canvas/create.ts), D190 recorded. - #116: Frame meter and freeze recorder finished and merged (perf-hud.on-demand.ts in dev; later commits d010d7ceb, 930675def under #116). - #119: Phone fixes deployed in 63b1f10e (in release-prod-live and dev), 27/27 real-server specs, D191. - #122: Quick-swipe fix (80c276796, 8a3d5b045) merged and deployed in 63b1f10e (in release-prod-live). - #123: Stray reach arrows removed (488cded1; equivalent 167b06c32 in dev, Mesh-only arrows, D194) with spec. - #124: Bisect found no regression across the commits; real zoom cost is tracked under #121 and the blur mitigation was reverted. - #125: Full screen finished (b9d05364, D193); full screen code and shortcuts present in dev. - #126: Resize-from-dragged-edge and lone-window behavior per D195 deployed in 63b1f10e (in release-prod-live). - #127: Window details popover merged into dev (7d6df4470, 90c3d9328 "popover is in the product canvas", head 42e50d42 is ancestor of origin/dev); finish comment has gate output, no open findings. - #128: Fix canvas-switch 9a770037 is in dev and was deployed to prod 2026-10-09 (web build of 01a6d848, also in dev and prod-live) with measured before/after table. - #138: Research delivered (research/terminal-images/REPORT.md at c8624b58) and owner decisions recorded as D203 (images pipeline, limits, HEIC/AVIF placeholder until trusted decoder, sandboxed helper). - #139: Research delivered (research/native-capture/MEASURE.md, 57346fc3) and owner decision recorded as D204 (macOS capture bindings and encoder). - #144: Host facts for popover merged into dev and prod-live (5cf106de ancestor of both); extended info explicitly out of this slice; final review and gate were clean. - #149: TLS session fix merged and deployed in prod-batch (6320f5b52 / 5d02b8d6b ancestor of release-prod-live and dev); delta review found no gap. - #151: Session cap/dead session fixes merged and deployed in prod-batch 707a4572 (6320f5b52 and b67d03455 in dev; release-prod-live contains it), listed in the deploy merge message. - #152: UX audit report delivered via orchestrator; owner decisions on its grill questions recorded as D219. - #155: Research delivered and re-grill result recorded as D222; shortcuts-d222 (e6969edc) is in dev and prod-live. - #161: Fixed (settings-roundtrip 051d0a40 in dev and prod) and deployed in release-prod-4 3aec316e with before/after evidence (0 stray PTY bytes across five cycles). - #162: Research note delivered (research/mesh-ssh) and owner grill outcome recorded as D235. - #163: Back-merge completed: dev-backmerge 8d0fc8ba merged into dev as 50a89572 (ancestor of origin/dev), checks recorded. - #164: All five items shipped: phone-polish c512cc07 is in dev and prod-live, lock-icons (withLockBadge) in dev; glyph question resolved by the owner's lock-badge rule. - #165: Generic Inbox card 89145f81 deployed web-only to production and is in dev and prod-live, gate and specs recorded. - #166: Sidebar polish 15d15c25 deployed web-only to production and is in dev and prod-live, captures verified. - #167: Research delivered (research/canvas-node, 2202eaf9) and owner decisions recorded as D248. - #168: Ended-sessions fix deployed in release-prod-4 3aec316e (in dev and prod-live) after review rounds 2-3; last comment is the deploy record. - #171: Sign-out clear (d8fb45c8, d472ee3a) is in dev and release-prod-live; deployed web-only in 4855e033. - #173: Merged into dev at c75c362b after a clean break-it review (P0-P3 none); c75c362b verified in dev and prod. - #174: Restart tests fixed and merged as cc3297508 (fix-restart-tests, #174) in dev; gate-flakes confirmed 50/50 loaded passes. - #181: Phone width fix and settings-rows (d6642c21) are in dev and release-prod-live; deployed in 4855e033 and 3aec316e. - #182: Cursor-blink (D249) deployed in 4855e033, which is in dev and release-prod-live. - #185: Ended-strip fix (361fc7ff) is in dev and release-prod-live; deployed as 4d1b1950. - #188: Deploy pipeline merged into dev (21b37bca7 is in dev and release-prod-live) and used for the production releases. - #189: Add-host speed work (addhost-instant 53ef70af6) is in dev and prod; deployed to production as 3048580c1 and release 2 (103815e70). - #190: Duplicate of #189; the addhost-fast review said MERGE and its follow-up fixes went in through addhost-instant (53ef70af6), which is in dev and prod. - #191: Superseded by #192 (the same feels-local research brief, with the superlogical note carried over). - #193: Observability research and log-volume numbers delivered; decisions recorded as D260/D261 (D261 in DECISIONS.md). - #195: Fix merged into dev and release-prod-live (52f644776, bdf68a6c) and deployed; merged-tree tests green. - #196: Deployed web-only as release-prod-5w ade40a36 and pushed to dev. - #198: Merged to dev as 9e84426c4 (in dev and prod); the size caps are green again. - #199: Switch-over merged (fa904fdaa) and deployed (036bc2bb0); both are in dev and release-prod-live. - #201: Mesh auto-switch deployed 2026-10-17 as 3b2183eaa, which is in dev and prod. - #210: Merged web-budget-headroom (804ac0e85) in dev: deferred brotli 44,807 to 42,272 of 45,056. - #214: Terminal links merged as f210a4f25 (ancestor of origin/dev and release-prod-live), deployed web-only and owner-approved; underline/CJK follow-ups tracked in #232. - #219: Linux mesh problem survey completed (report attached, snapshot 5abc64bb8); its findings drive the relay work in #220. - #228: Tooltip fix merged to dev and deployed web-only at 1bcb7a87b (ancestor of dev and release-prod-live); final gate passed. - #229: Mesh unreachable fix released in 3b2183eaa (ancestor of dev and prod), deployed 2026-10-17, review verdict MERGE. - #234: Ping/control-queue fix and link-doubt indicator released in 103815e70 (ancestor of dev and prod, release 2) with measurements. - #236: Honest Mesh view and host names deployed in release 5 (c00c6364e) and release 7 (173a9ba14), both ancestors of dev and prod. - #241: WASM page reuse fixed by renderer-zero-pages, merged as cef7224ee (in dev and prod) and deployed web-only 2026-10-17. - #245: Liveness retirement fix deployed web-only at 857823ce4 (ancestor of dev and prod); production journal showed zero stream closes after deploy. - #246: Cross-root reproducible static builds merged to dev (702e671a0, 709efb451 ancestor of dev) with passing final gate. - #248: Stale-screen note merged (c8aa36727 in dev) and deployed in release 8 (85b0285b7, ancestor of dev and prod). - #249: Quiet idle warm-up merged (d37691790) and deployed in release 9 (bec4eb7dd, ancestor of dev and prod). - #253: Public web access completed and verified on 2026-10-22 (Traefik allowlist removed, agent port stays NetBird-only). </details> ## Appendix B. Open issues on 2026-10-28 One-line state per open issue (2026-10-28 triage, read-only, from each issue's latest comments). Use this to pick work; the issue itself is the record. <details><summary>All open issues</summary> - #1: Decision-record issue still lists unchecked items (tier prices, KCP benchmark, snapshot-vs-state-diff resume, S1 details). - #2: S0 website rejected by owner and parked on 2026-10-05; branch s0-website never merged, redesign/design grill not done. - #3: Pages/D1/secrets set up but the workflow site.yml, first deploy and custom domains/TLS are still pending (waits on S0 website). - #4: Perf VM and bench runner checklist all unchecked; no comments (only perf-run lock script landed). - #5: S1 grill round 1 answered (D48-53) but last comment leaves the TLS strategy for infraternal.kayg.org open and topic list unchecked. - #6: DNS fixed but the owner decision on old status-page containers/files on monitor01 is unanswered. - #7: Standing real-iPhone checklist template with every item unchecked and no comments. - #8: GitHub push mirror and README pointer both unchecked, no comments; push is only on owner say-so. - #9: S1 umbrella with many unchecked backend/web/quality items (SSH plugin, bootstrap, inventory, perf gate, etc.). - #14: Piece-by-piece visual grill still ongoing (latest piece ui-tweaks merged 56bb6f51d); product UI identity not finished. - #18: Stress test and crash analysis delivered (fixed via #36/D109) and search design recorded in D89-91, but cross-session search implementation not evidenced in dev. - #24: Status board, always open. - #35: Spike reports A-C merged (d57516aec) but owner questions from SPIKE-B/C (session count, scaling, JPEG vs H.264, shared encoder) remain unanswered. - #37: vt-export parked after content-filter stops; WIP branch only, transactional resize added to scope, nothing merged. - #38: Owner question on server-side Snapshot coalescing (815 to 202 MiB per session) unanswered; no D### found; spike a82523c not in dev. - #41: Codex review was stopped by content filter; only fuzz targets ran clean, HTTP/WebSocket robustness checks still undone. - #42: fix-perf (967d278) is not in dev; merge plan pending a D130 check of the coalescing commits, so findings are not shown shipped. - #43: Fix merged (a20ae2989, in dev and prod) but owner retest with the real 1Password extension is still pending and the welcome-UI provider lane was handed off. - #44: Scroll fix 760490ce4 is in dev and prod; the last comment asks the owner to confirm on a real iPhone. - #45: Recovery backend deployed (5ed4f295c) but the recovery sign-in UI is held behind a preview opt-in (1c12295c6); production UI not shipped. - #46: D118 decision recorded (nested scroll + canvas gestures + accent border) but no implementation commits reference #46 and the issue was never confirmed done. - #47: Research report research/data-path/REPORT.md delivered, but its comment says no decision made; the owner questions are unanswered on the issue. - #48: Canvas/pill/popover design: prototype work on dev but the last comments are mid-work findings with no closing deliverable or owner sign-off. - #49: Idea for later (auto-install mesh clients), zero comments, not started. - #50: Predictive echo deployed only behind ?predict=1 (e19c4eb); default-on and felt-latency measurement on the India-Germany path not confirmed. - #52: Umbrella: returning sign-in still measured p50 ~3.96 s, p99 ~5.0 s vs 5000 ms budget (FAIL in latest runs); phase B (#67) not done. - #53: Merged in dev/prod (444b839e4) but the issue says real iPhone check remains owner-side and no confirmation is recorded. - #54: Compose mode (D130) has no comments and no commits; needs design grill and prototype. - #56: infraternal relay subcommand: no comments, no commits referencing it. - #58: Mesh-policy SSH block diagnosis and 'add a policy' feature: no comments, no commits. - #59: infraternal migrate command: no comments, design and grill first, not started. - #60: Svelte 5 migration (D142/D143): only a step-0 prototype instruction comment, no migration commits. - #61: Reconnect-to-resumed still measured p99 ~1009 ms vs 900 ms budget (FAIL) in the latest run. - #62: Cold access-screen render p99 ~1.9-2.0 s vs 1 s budget (FAIL) in the latest run. - #63: Echo-confirmed p99 ~364 ms vs 334 ms budget (FAIL) in the latest run; predictive echo is only behind a flag. - #64: Discovery coverage report delivered (research/discovery/REPORT.md on dev) but the issue says implement after the grill; implementation and owner questions remain. - #65: Reviewer-clock fix merged (5ed4f295c) but motion-audit is still observed failing in later runs (#74), so flakiness is not resolved. - #66: Phase A merged (fca6e093d) but the 5 RTT + 150 ms target is not shown met; #52 latest runs still p50 ~3.96 s at 300 ms RTT. - #67: Phase B (combined auth/attach protocol, 2 RTT + 150 ms) has no commits referencing it; waits on phase A. - #68: Design round G (users, groups, OIDC) settled in D150-D154 but implementation not done; no commits. - #69: Releases and auto-update (D155) implementation not done by this issue; no commits reference #69. - #70: Windows support is a later umbrella (priority raised by D161), not started. - #71: Ranking report delivered (research/bsd/REPORT.md) but deferred by owner under D161; to be picked up later. - #72: Design of the 'Gather extended information' switch: owner grill questions unanswered, no mockup. - #73: Same-session PTY flood still FAILs (p99 382 ms, growth 53 ms vs 16.67 ms budget) in the latest run. - #74: Long main-thread tasks in the 4x CPU motion audit not resolved; latest run failed earlier on a different motion assertion. - #76: Research and demo delivered (research/reflective-card on dev) but owner questions on permissions, export angle and CSS vs spike are open; product feature not built. - #77: Waiting on the owner to provide a Proxmox VM and NetBird key; test hosts later arrived (minipc01) but the described lab is not confirmed. - #78: S1 remote agents umbrella: 395 comments, last comments show add-host round 2 with issues left open for the orchestrator VM rerun; ongoing. - #80: Slice 1 crate merged (9c62448a3) but the issue states it stays open; slice 2 wiring and real packet filtering still to come. - #82: Add-host mockup: last comments are in-progress phone fixes with merge conflicts; no owner approval recorded. - #83: Recovery mockup: last comments are in-progress steps (token block missing, shared primitives); no owner approval recorded. - #86: Grill for deep links has no comments yet; DECISIONS only references it (#86 IDs), not decided or built. - #87: Add-host round 2 finished on release-addhost but comments say issues left open for orchestrator VM rerun; not confirmed merged/closed out. - #88: Research delivered and owner chose quinn+rustls+ring (D176), but promised follow-ups remain: matched-loss rerun and BBR at 300 ms RTT. - #89: Umbrella: canvas product merged but grill/containers-VMs-Kubernetes fit and later rounds still open. - #92: Resume p99 / same-session flood echo over budget; no comments, needs backend investigation. - #93: Owner answers recorded as D180, but exports (JSON, Markdown, image, Excalidraw) not confirmed built. - #94: Share studio: only the Machine-view mockup rounds exist (r2 with review fixes); real-device numbers owed and share studio not built. - #95: Idea for standalone infrafetch project; no comments, not started. - #96: Live sharing is an idea with open questions (core vs ee, link encryption); no comments. - #97: Break-it review reproduced open P2 (RC-6 grouped parked-terminal memory 6.4 MiB vs 2 MiB budget); no fix recorded. - #98: Canvas doc backend: review delta still in progress, fast gate failed only on binary size (D205); merge not confirmed. - #99: Independent review verdict "merge after fixes" with 3 open P2 (MS-R4/R5/R6); fix and merge not recorded. - #100: Session lifecycle: review has open findings (SL-R9) and last comment is a gate run; merge not confirmed. - #101: Later idea (history/log search); grill first, no comments. - #102: Self-update spec only has an owner note; implementation not confirmed. - #103: Relay/direct path still in run5 incremental slices; str0m path not in dev. - #104: Sessions mockup ported to canvas (6dd7ea2 in dev and prod) but the deferred-chunk limit raise to 58 KiB awaits owner approval. - #105: Backups spec with no comments; no evidence of finished work or numbers. - #106: Later: native PBS destination, needs library measurements and owner decision; no comments. - #107: Canvas streams review has open P2 CS-R6 (input writer removed under send pressure) and a native test failure noted. - #108: Mesh API deployed (db8c5270) but comments still carry open items: binary size over frozen budget and D198 lifecycle work in progress. - #109: Umbrella plan for canvas lanes; no comments, lanes (e.g. #98-#108) not all done. - #110: terminal_writers leak fix: no comments, no commit referencing it found in dev. - #111: Zero-downtime restarts: no comments or fix recorded. - #112: Plain names decided and deployed (1c3ae5f) but the audit of every remaining message is not confirmed complete. - #117: Single observation of a dropped window open under load; no comments or investigation. - #118: Test-only fix merged to dev (459070fe) but independent verification review is still running. - #120: System-limits test plan with no comments; not run. - #121: Target of 120 fps zoom/spam; round 5 deployed but p95 34.9 ms vs target and work continues. - #129: Mesh data plane umbrella: last handoff (dev-merge-d215 7d70353f) states acceptance blocked by static size, native same-session budget and a failing browser fixture; no green merge claimed. - #130: Mesh "why can't A reach B": idea, no comments, to be grilled before building. - #131: Mesh preview/undo of changes: idea, no comments, to be grilled before building. - #132: Mesh temporary access: idea, no comments, to be grilled before building. - #133: Mesh service discovery: idea, no comments, to be grilled before building. - #134: Mesh live paths on arrows: idea, no comments, not started. - #135: Mesh doctor: idea, no comments, not started. - #136: Mesh per-arrow traffic: idea waiting on the later metrics component, not started. - #137: Backup grill (growth, HA, auto rollback): no comments, measurement and grill not done. - #140: Browser-as-canvas-window research parked by owner decision D207; W1-W3 explicitly left open. - #141: No comments on the issue; dev has several #141 commits (focus framing, keyboard split, close X) but nothing on the issue confirms the three reported items (split stuck, focus after split, close bounce) are verified. - #142: macOS app plan: no comments, to be grilled once server is stable. - #143: CLI layer merged into dev (47f2b31b / 30487a0c) but the issue says a binary-size cut under D205 is a pending follow-up and D202 requires every component (relay, ssh, backup, ...) to use it. - #145: Account preferences store: no comments and no server prefs store found in origin/dev. - #146: App windows from a Mac: latest comment says blocked on the Mac (power/NetBird), no native root cause or fix claimed, issue remains open. - #147: Access rule list and explain(a,b): no comments and no implementation found in origin/dev. - #148: Feedback frames plugin: review ended with failing static size gate and unverified perf; not in dev, and D281 later redesigned in-app feedback via GitHub Discussions. - #150: Missing-glyph boxes on logout: no comments and no fix found. - #153: Research report pushed on branch research-prf (61482a0c, not in dev), recommendation made but no decision recorded; D206 still lists device keys as pending PRF research. - #154: D217/D244 CA handoff and key memory deployed (release-prod-5 5bf3f42e) but delta review HOLD with UP-01..03 (P1) deliberately deferred to release pipeline round 4. - #156: PXE grill: D253 recorded but the issue lists open owner questions and D-text says Boot Machines questions wait until mesh and dogfooding are stable. - #157: Release pipeline umbrella with 270 comments; latest branch dev-merge-r4 (261e4499) fails the static size gate and round 4 / UP-01..03 follow-ups are pending; not confirmed closed out. - #158: Fix deployed (1c08c4b7) but the delta review left a P2 (logs-body download has no retry) promised as a later fix-forward. - #159: Benchmark harness delivered, but an orchestrator-found harness bug (root scratch dir 0700, chown to measured user needed) is still noted as needing a fix. - #160: Fix deployed in release-prod-4 3aec316e, but a later delta review (b429e566, review-query-replies-r2) ended with verdict HOLD and a P1 with no resolution recorded. - #170: Multiplexer-skip work (archive tag multiplexer-skip 00299ee9f) is not in dev; D297 fit-review owner questions and D243 size overage were still pending. - #172: real-logins branch (tag b3e2b5b7f) is not in dev; review findings A-F (agent uninstall panic, UI fallback note, window identity, stale cells) and RLR-13 P1 remain. - #175: Size-cut push is partly delivered via #198/D264 (9e84426c4), but the S2 one-crypto-library decision (D257) and further cuts are not shown as done. - #176: prod-perf-fix (tag 28bbc0b6f) is not in dev; reconnect budgets D265 and the resume-token continuation were still pending. - #177: WASM client-core spike parked by the owner 2026-10-12 until mesh and app windows are stable; no result delivered. - #178: D236 session labels not found in dev; the last comments are a review with a failing job-head gate and the issue stays open. - #179: Queued issue with no comments; no sign of D230 account names work in dev. - #180: Queued issue with no comments; no WireGuard stack measurement found. - #183: Mesh mockup round 1 is on branch mockup-mesh-r1 and not merged; the owner answers were recorded but follow-up rounds were on #186 and not confirmed complete. - #184: perf-remeasure (e8b98754) is not in dev and scripts/perf-vm is absent; the VM browser timing harness is not delivered. - #186: Access-rules spec and mockup rounds are not in dev (no docs/design/access-rules.md); owner answers continue and the model is not approved. - #187: idle-rss branch (tag 2c5f0849b) is not in dev; the RSS attribution and reduction are unfinished. - #192: Research delivered and D259 recorded, but the owner asked for measurements on Q1-Q3 before deciding and the research-feels-instant follow-up is still unfinished. - #194: Host-inventory merge 85368c7fd is in dev and prod, but the "+" host picker (ui-new-window-host 79006368) is not in dev. - #197: Flood latency is still being measured; D268 records only that the flood server part ships, and review verdict was HOLD with unresolved gate failures. - #200: Evidence tooling and D262 exist, but the job was blocked on the final gate and the repo is still large (.git is 607 MB); the history rewrite is not done. - #202: Planning-only issue with no comments; D261 says logs nodes are planned next, and no plan was delivered. - #203: Cipher-order work has not landed in dev (no merge found) and the independent review C203-R1 (P2) was still open. - #204: Cheaper gate iterations work is on a branch with the gate still red on size; no merge to dev found. - #205: User-screen pool merged into dev (7ee79fb6d) but stays off by default until the D270 soak gates and the owner's go. - #206: Research on tmux and zellij issues has no delivered report visible; the latest comments are unrelated deploy notes. - #208: Only a plan comment; no measurements or report delivered. - #209: Terminal OSC work is not in dev; review findings OSC-R06 and OSC-R07 (P2) were open. - #211: D251 edge auto-update backend still in progress (head 643a5af84, UI/proofs/final gates pending, last comments are build-slot waits). - #212: UX gap audit with ranked gaps and continuing gate/measurement comments; no closing result or per-gap resolution recorded. - #213: Mac userspace mesh measurement failed (direct UDP to perf VM timed out); real mesh throughput not measured, nothing merged. - #215: Mockup issue (sidebar rename/remove, privileged consent) with only screenshots posted; no owner approval or decision recorded. - #216: Version pill/licensing mockup rounds posted with screenshots; no owner approval or merge recorded. - #217: Native spike (D238) final fast gate failed (size cap, server test); orchestrator reconciliation and owner review pending. - #218: History audit done but rewrite/cut-over not authorized or performed; waits on owner. - #220: Owner questions Q1-Q8 pending; no relay/NAT code built. - #221: Mesh SSH still under research/review; dependencies on process-per-user and login access not built; issue remains open. - #223: VNC research delivered with 7 open owner questions and failing gate rows; waits on owner. - #224: Person's card mockup first round posted for owner judgment; no approval recorded. - #225: Fix 492e1505a deployed to prod, but the stuck old agent field case moved to frozen #231 and no verified reconnect recorded. - #227: Close-reason logging shipped (5b2a1b9ee in prod) but the flicker fix on pill-flicker (9e01a0331) is not in dev/prod and server-side close trigger is unisolated. - #231: Frozen after round 8 with an unfixed P1 (late-created state directory); round 9 not started. - #232: Link underline merged and deployed (f627a2e09) but two queued follow-ups (CJK URL detection, clipped last column) remain. - #233: Join-over-SSH (D291) branch join-over-ssh 47619c27d is not in dev or prod; add-host work frozen. - #235: Re-add failure diagnostics still in progress: missing-account-after-channel case has a failing regression, not fixed or merged. - #237: Status truth shipped in release 4 (a906ede87) but a new launch-qa finding (status says Running in a crash loop) is unresolved. - #238: Audit found the D126 default fields hidden behind the extended-info switch; UI fix still needed and no merge recorded. - #239: CLI overhaul first slices deployed in release 4 but the proposal lists further build slices not yet done. - #240: Busy-store authority fix live in release 9.4 (b30bb81cf, in prod but not yet in dev) after review rounds; merge-back to dev and closing evidence pending. - #242: Own context menu and swipe actions: only mockup screenshots posted; no owner approval or merge. - #243: Server-held layout shipped (release 6, 9.2) but p3/settings follow-up head f63884f58 is not in dev or prod. - #247: Launch QA run stopped at S2 by the #252 P1; final verify fast failed on JS transfer budget; remaining personas not run. - #250: Protocol-close fix partially deployed (9.2) but follow-up fix-sync-journal head 573dc412e has open P2 and is not in dev or prod. - #251: Research delivered but owner design questions Q1-Q6 are pending. - #252: P1 setup-token fix job stopped mid-gate (TypeScript check/verify still running or paused); not merged. - #79 (left open, borderline): Grill delivered: owner answered Q1-Q17 and the result is recorded in D163-D167 (commit 159f43797 in dev); build work tracked in #80/#81. - #169 (left open, borderline): Add-host rounds 1-3 and start-fresh deployed (5bf3f42e in dev and release-prod-live); NetBird-SSH-only case continued and fixed in #195; speed work moved to #189/#190. - #207 (left open, borderline): Test fixes fast-forwarded into dev (21c31f906, verified in the log) and the diff was reviewed by the orchestrator. - #230 (left open, borderline): Add-host Retry fix released in 173a9ba14 (ancestor of dev and prod, release 7) with staging proof; only the owner's retest remains. - #244 (left open, borderline): Phone resume recovery shipped in release 6 (8f8c84e37, ancestor of dev and prod) with owner test steps listed. </details>
Author
Owner

Superseded by the handover of 2026-10-30: #262 (all work paused and pushed, mesh-first roadmap, Claude-only rule, open decisions).

Superseded by the handover of 2026-10-30: #262 (all work paused and pushed, mesh-first roadmap, Claude-only rule, open decisions).
Author
Owner

J256 housekeeping, authorized by kayg. Closing as superseded: Its final comment explicitly supersedes this operational handover with #262; no separate implementation work remains in this historical handover record.

Canonical handover: #262 (left open). Evidence: the final handover comment dated 2026-10-30 explicitly redirects all work and open decisions there. Merge commit: not applicable; this is an operational handover record, not a code implementation. Re-fetched the issue and all comments; no changed activity or activity within 12 hours. Audit: #369.

J256 housekeeping, authorized by kayg. Closing as superseded: Its final comment explicitly supersedes this operational handover with #262; no separate implementation work remains in this historical handover record. Canonical handover: https://git.kayg.org/kayg/infraternal/issues/262 (left open). Evidence: the final handover comment dated 2026-10-30 explicitly redirects all work and open decisions there. Merge commit: not applicable; this is an operational handover record, not a code implementation. Re-fetched the issue and all comments; no changed activity or activity within 12 hours. Audit: #369.
kayg closed this issue 2026-11-05 18:07:42 +00:00
Sign in to join this conversation.
No labels
launch-qa
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/infraternal#254
No description provided.