kayg/infraternal

Watch

Fork 0

Orchestrator handover 2026-10-30: all work paused; mesh-first roadmap; Claude-only #262

New issue

Open

opened 2026-10-30 17:36:12 +00:00 by kayg · 1 comment

kayg commented

2026-10-30 17:36:12 +00:00

Owner

Handover from the infraternal orchestrator on code (10.69.69.92), 2026-10-30 23:06 IST, updated 23:13 IST. All work is paused, committed and pushed. Nothing of ours is running on any VM (only idle Claude sessions of the paused jobs remain), and every build-pool lease is released. At 23:10 IST four resume jobs were started and cancelled within two minutes on kayg's order; they made no changes (sweep re-run: 69 worktrees clean). Calternal is also paused tonight (its handover is kayg/calternal#774). Earlier handover: #254. The live status board stays #24.

This repo is public. No credentials appear here. Secret files are referenced by path only and are never to be printed.

1. Rules in force

Claude subagents only, no Codex (kayg, 2026-10-30 21:10 IST, for at least 24 h, until kayg lifts it). The shared Codex account on every build VM hit its usage limit until 2026-11-05 07:26. Use delegate_task with providerInstanceId: claudeAgent: Opus 5.5 (high) for security, privacy and protocol work; Sonnet 5.5 for UI and mechanical work. Break-it reviews also run on Claude.
Builds and tests never run on code; they go to the build VMs. Every job reads ~/infraternal-ops/rules.md (the master copy is ~/.local/state/codex-jobs/infraternal/orchestrator/rules.md on code; copy it to each VM after any change). New section 2026-10-30, "Finish what you start": a job never ends its turn while its own background work still runs. Claude jobs repeatedly did this and needed a "continue" message.
Three review rounds max: if round 3 still finds new P0/P1, stop patching and write a plan.
D243 sizes: under 16 KiB is approved by the orchestrator; 16–64 KiB needs a written reason; over 64 KiB needs kayg.
Reviewed and staged fixes deploy without asking; kayg unlocks the CA after each restart.
Coordinate builds and perf runs with calternal (thread import:claudeAgent:b6baf59e…). The perf VMs perf-test-01/02/03 are reached only through /usr/local/bin/perf-run.sh, or as root through perf-lock.sh + systemd-run --scope --slice=perf.slice, leaving an empty before/after machine diff.
Times go to kayg in IST.

2. Production

Release 9.5 (0c8639496) is live on monitor01 since 2026-10-30 17:02 IST, via deploy-prod.sh from code. Backup: /var/backups/infraternal/20261030-113220-0c8639496. release-prod-live = 0c8639496. kayg unlocked the CA.
Shim memory 9.5 vs 9.4 at 50 sessions (#24): PSS/USS flat. RSS +2.7% comes from shared pages. Open: in 1 of 4 fixtures, about 17 of 50 shims grew by 2 MiB USS while idle; needs ≥10 fixtures to find the cause.
59 leftover sessions from 17 Oct were closed on kayg's OK; kayg's 3 sessions remain.
dev is 7 commits ahead of production: decisions and docs only (D313–D316, the PROCESS.md release rule), no product code.

3. Decisions recorded today (on dev)

D312 2fb412d01: layout edits live per tab; the server is the only saved copy (#250).
D313 db8c1bdf6: mesh direct paths and relays, #220 grill Q1–Q7 (seam C, home-relay rendezvous, any relay, hard-NAT gated on counters, switch rule max(5 ms, 15%), TLS+WebSocket 443, QUIC DATAGRAM).
D314 c9671cb79 + f9d0a9a8f: one CLI command set (sessions …, canvas …, open), plus a tmux-compatible subset for AI tools. kayg: "support tmux style commands yes because lots of training data".
D315 c9671cb79: never interfere with auto-started multiplexers; offer the guard line with a confirmed "add" button (replaces D241's transparent skip).
D316 a4f15f8f7 + 42ab33035: str0m's ICE core for mesh path finding (+208 KB accepted), our own port mapping; .internal names by default (D165) plus custom subdomains. The reverse proxy runs on every client, without root above port 1024, with DNS-01 certificates; we never write our own proxy engine. The engine is chosen by kayg after the #261 measurement (Traefik can't be embedded in the Rust binary).
docs/PROCESS.md 63d8a03dc: release candidates pass the full browser suite twice in a row, retries 0 (#207 proposal, approved).

4. Roadmap (kayg 2026-10-30: "accelerate the mesh to replace NetBird first; the multiplexer in parallel")

Mesh lane, in order:

#220 relay + seam C (slice 1 done, branch mesh-relay-220);
infraternal sign-in <server> (link, code, QR, passkey; D311);
mesh DNS (.internal + custom suffix, records, upstream; D165/D197);
infraternal up laptop join + one-command install;
the macOS node (launchd + utun, on macos-02-std);
SSH over the mesh (#221).

Later in the same lane: rendezvous + simultaneous open, upgrade and flip-back on the #201 selector, own port mapping, str0m path checks, the reverse proxy per D316.

Multiplexer lane, in parallel: the D314 sessions/canvas commands + tmux subset + MCP (spec slices on #251 first); D315 guard line (rework branch d241-shell-marker, which must shrink from D241's +40 KB).

Measure first, to answer kayg's question: #260 head-to-head (our mesh vs NetBird/Tailscale, our sessions vs tmux/Zellij) and #261 (reverse-proxy engines).

5. Work in flight (all paused; heads on origin)

Issue	Branch @ head	State	Next
#258 Mesh pins: no fixed limit, visible-ID rule, 30-day drop (kayg-approved design)	`mesh-pins-258-v2` @ `6b4211583` (WIP)	Opus continuation was midway: renamed-tag fix, reads without the writer lock, clippy/TS fixed. 18/20 pin tests were passing; 2 float-vs-int expectations were corrected and not yet re-run	Split the WIP into atomic commits, rebase, both sizes, `verify fast`, then a break-it review. The canvas operation-log rule is D183/D248 (not D235)
#250 per-tab layout (D312)	impl `layout-per-tab-250` @ `7e2490f07`; review `review-layout-per-tab-250-r2` @ `80c2a3578` (WIP)	Round-2 review paused mid phase A (gate + Chromium + WebKit); findings so far are on #250	Finish the round-2 review: phase A, then phase B (dev `2fb412d01` static + WebKit). Perf kit (`361fac63…`) fails on perf-test-03 as real root: terminal stuck "connecting". The next fix round must reproduce it as real root on a build VM
#207 broad suite triage	`broad-suite-triage` @ `fc65de7b0`; review `review-broad-suite-207` @ `d9a16ed2c` (WIP)	Provisional merge after fixes: P0 0, P1 0, P2 7, P3 1. R207-02: old session's pixels stay after "New shell". R207-03: the attach ≤60 ms bound is lost and `REAL_SWITCH_BUDGETS` is never set. 4 extra broad runs 112/15/0	`verify fast` on `fc65de7b0` (worktree `~/.cache/codex-scratch/review-broad-suite-207/head-fc65` on 8by64), sizes, final verdict, then a fix round
#237 status during crash loop	`status-crash-loop-237` @ `f9859712c`; r3 review `review-status-crash-loop-237-r3` @ `9bf549962` (WIP)	Fix round 2 done (status p50/p99 14.8/18.8 ms); the final round-3 review was mid-run	Finish round 3. A new P0/P1 means stop and write a plan (three-round rule)
#211 edge auto-update (plan `plan-211-update` @ `c9c1877e9` accepted)	`edge-update-v2` @ `fd1892201` (WIP)	5 gated commits + 1 WIP; e2e under real systemd 6/7 (the 7th is a test-script bug: `open()` vs `os.open`)	Automatic updater, manual rollback, hosts follow the server's version, full-disk cases, the perf-VM kit, sizes, gate. Its ~56 GB target dir on 6by32 is kept
#220 mesh relay slice 1	`mesh-relay-220` @ `4fe5cd912` (WIP)	QUIC DATAGRAM relay, seam C on the WG port, failover. Namespace proof 6/6; fixed the 1452→1472 MTU drop	Proper commits, release numbers, `contracts/mesh-relay.md`, gates, size (expected 0 B shipped)
#220 measurement	`mesh-measure-220` @ `ced872e68`	Done. Results are on #220 (seam B ≈ kernel; seam A 2× CPU; 270 B per relayed pair)	—
#260 head-to-head kit	`research-head-to-head` @ `78ccce47e` (WIP)	`h2h` tool built; smoke passes for 5 mesh cases + tmux/Zellij. Possible product bug: server closes the stream with 1013 during a 100 MB `cat` (keepalive ≈500 ms)	Triage the 1013 case (is it real for browsers?), WAN/loss/scale cases, x86 kit, run on a perf VM
#261 reverse-proxy engines	`research-reverse-proxy` @ `ad7daa89f` (WIP); scratch `wip/rproxy-delta-scratch-2026-10-30`	All four pass the functional matrix. x86 adds: Pingora +5.3 MB (panics on tokio 1.54.1), Sozu +6.4 MB (AGPL/LGPL, not in deny allow-list), rpxy +4.7 MB (git-only, aws-lc). instant-acme DNS-01 works	Full-duration timings, x86 kit, write-up; then kayg decides
#259 sidebar pill drift + double-tap rename	`sidebar-pill-259` @ `e8c969d00`	Implementation pushed (9 commits over dev)	Production screenshots + bundle size + x86 `verify fast`, then my visual review
#146 app windows	helper `appwin-helper-macos02` @ `a2b8dbf9e`	Helper built + signed on macos-02-std, no prompt. `--check-permissions` → both false	Waiting on kayg's TCC choice (see §6). Then capture/latency work
#256 frosted sideways scroll	`frosted-scroll-256` @ `88ae0a23d`	Fix by default; `.cl-viewport` clip-path limits Chromium blur	kayg device test (Aside/Chromium on M2 Max); the preview at :47411 was unreachable from kayg's device
#241/D315	`d241-shell-marker` @ `e18cd7a0e`	Superseded by D315	Rework to D315 (queued)

Safety snapshots pushed today: wip/d241-size-staged-2026-10-30, wip/stale-note-257-staged-2026-10-30 (half-finished merges found staged in those worktrees; the worktrees are left as found), wip/stash-8by64-stash0-2026-10-30 (stash "setup-token-data-step-status"), perf24-reconnect (2 commits pushed). On code: wip/code-merge-release94-dev-2026-10-30 (a stale unfinished 9.4 merge with 2 conflicts; the real merge f069bb2a9 is on dev) and wip/code-serve-{clusters,demo,glass}-2026-10-30 (the local preview tweak to mock-server.ts). All snapshots were taken with a temporary index, so the worktrees are unchanged. Final sweep at 23:20 IST: every worktree on all VMs and on code is either clean and pushed, or saved to one of these branches; the preview worktrees sit on archive tags that are on origin. Evidence from the old std VM is archived on code at ~/.local/share/infraternal-std-archive-2026-10-30/.

6. Decisions waiting for kayg

#146 Screen Recording + Accessibility + PostEvent for com.infraternal.desktop.helper on macos-02-std: kayg said at 23:10 IST that they will grant these themselves, for /Users/lume/Applications/InfraternalCaptureHelper.app. Verify afterwards with the helper's --check-permissions (see #146), which must return both true. Accessibility already has an "off" row that macOS created itself. csreq: fade0c00000000580000000100000006000000020000001e636f6d2e696e6672617465726e616c2e6465736b746f702e68656c706572000000000004000000000000001462cbdf2863333c4631609ee3972d32c0a3791ba7.
macOS deployment target for the helper: the build is for macOS 26.0, while the crate claims 14.2. Build for 14.2?
10by32-02-std reboot (maintainer, to enable free-page reporting): not done yet (still up since 14:33 UTC at 23:13 IST). All our jobs there are paused, so it can reboot now. The T3 maintenance thread (6bfdd7b3…) no longer exists, so kayg must give it the go. The 10by32s will be phased out onto the new dev VMs (infraternal-dev-02-std).
DNS-01 provider token (e.g. Cloudflare): stored sealed and used only by the node requesting certificates, then moved to OpenBao? Recommended yes.
Reverse-proxy engine (after #261 completes): every embeddable option costs +4.7–6.4 MB, about 300× the 16 KiB band.
Mockups to approve (a rebuild of all of them at one URL is approved; the job isn't started yet): row menu + swipes #242; glassy machine cards #94; account card #224; version pill #216; host rename/consent #215; boot machines #156; sessions UI #104 (partly superseded by D311/D314); connection status icons. The add-host pill #189 is approved and only needs building.
#256: keep or remove the .cl-viewport clip-path (real blur vs cost), and which URL/device couldn't open the preview.
VM-to-VM SSH: kayg allowed it, but it only partly works (std↔std and 8by64→6by32/std time out; OrbStack DNS broke short names on the old std VMs).
OpenBao deployment (kayg), then move secret use onto it. After infraternal-dev is deleted, remove it from production's host list.
#146 encoder grill: which video encoder path app windows use (hardware VideoToolbox H.264 is proven on the VM; the codec and bitrate policy aren't decided). Ask it as a grill with measured options once the helper has its permissions.

7. Infrastructure

Machine split, agreed with calternal at 23:08 IST and reported to kayg:

Infraternal: 8by64-compute02, 6by32-compute02, 10by32-02-std, macos-02-std, and infraternal-dev-02-std when it exists.
Calternal: 8by64-compute01 (CI/staging), 6by32-compute01, 10by32-01-std, macos-01-std, and calternal-dev-02-std.
Shared, measurement only, by lock: perf-test-01/02/03.
Borrowing: ask in the calternal thread first, then pool lease --host … --placed.
Tonight: calternal lent us 6by32-compute01 (agent, x86) while it's paused. Our setup there: ~/infraternal, ~/infraternal-env.sh, ~/infraternal-ops, T3 env 2f385a8d-4422-48d7-ab4e-4bd35ddcb7f8, project mcp:4e961d17-e2cd-4a7b-a3b2-1e94ef3745ec. Hand it back when calternal resumes.

Machine	Access	T3 environmentId / project	Notes
8by64-compute02 (x86)	`netbird ssh agent@8by64-compute02`	`82ceea33-a5d6-4ddf-9315-9fe0eee9774f` / `mcp:2f293fb9…`	≤3 jobs; about 67 GB free; review target dirs (~38 GB for #207) can be deleted when those lanes close
6by32-compute02 (x86)	`agent@6by32-compute02`	`f8fd0eac-f81a-4562-b382-73f0ba8cb25d` / `mcp:6b7d764e…`	≤2 jobs; #211 target ~56 GB kept
10by32-02-std (arm64, cua spaces)	`netbird ssh cua@10by32-02-std.vpn.kayg.org`	`21b9f017-1060-4df4-9e17-3b2ccf8a0a1c` / `mcp:866ac412-bc6d-4133-9b92-031b0acbfa6a`	Ours by agreement (calternal has -01). `~/infraternal`, `~/infraternal-env.sh` (5 jobs), `~/infraternal-ops`. Static-size gate rows run on x86
macos-02-std (macOS 26.5.2, cua spaces)	`netbird ssh lume@macos-02-std.vpn.kayg.org`	`72ff8e68-41b2-4a5d-ba03-14d22ba85b6c` / `65a265e6-12d5-4d39-82b6-99948f7e1630`	Ours (calternal has macos-01). NOPASSWD sudo for lume; source `/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh` for nix. No Forgejo key: git is relayed by `orchestrator/vm/macos-sync.sh to
perf-test-01/02/03 (x86)	via calternal agreement	—	perf-test-03 has only our user-level `~/infraternal-perf` (Chromium, nix out-link, kits). Delete it when the #250/#211 perf runs are done
monitor01 (prod)	`netbird ssh root@10.69.69.50` from code	—	`deploy-prod.sh` in `~/.local/share/infraternal-preview/`

Build pool ~/.local/state/build-pool/ (78 tests): --host/--placed to register jobs that are already placed, a per-host user (cua on std), [groups.std] shared_storage=false. Register every job as placed and run a heartbeat unit (pool-hb-<job>); release it when the job ends.
Previews on code: :47361 demo, :47381 recovery, :47391 status, :47401 clusters (row menu + infrafetch combo), :47411 #256 glass. All are old; the mockup rebuild replaces them.
Orchestrator state: ~/.local/state/codex-jobs/infraternal/orchestrator/ (queue.md log, rules.md, spawn-log.md, vm/ scripts incl. sweep-unpushed.sh). Memory index: ~/.claude/projects/-home-kayg-Developer-Infraternal/memory/MEMORY.md.

7a. Problems found in the infrastructure

On 8by64, about 300 orphaned __shim processes (8.5 h old) were left by the first #250 review. Paused or cancelled jobs leave test servers behind. Sweep by age after each job (ps -eo etimes).
pkill -f <pattern> over netbird ssh matches its own command line. Use pgrep + kill, or split the pattern.
bc isn't installed on code.

7b. Slot plan for the next session (planned 23:08 IST, not started)

Machine	Jobs (all Claude Opus unless noted)
8by64-compute02	#220 relay slice 1 finish (moved to x86 for the gate); #258 finish; #250 round-2 review finish
6by32-compute02	#237 round-3 final review finish, then the next slot; #211 continue
6by32-compute01 (borrowed)	mesh step 2: `infraternal sign-in <server>` (link, code, QR, passkey; D311)
10by32-02-std (after the reboot)	#260 head-to-head finish + triage the 1013 disconnect on heavy output; #261 proxy finish (Sonnet); multiplexer: the D314 `sessions` commands + tmux subset + MCP
macos-02-std	#146 first real capture and latency, once kayg's grant is in

Then, as slots free:

mesh DNS
#207 fix round
#259 screenshots + gate (Sonnet)
D315 guard line
mockup rebuild at one URL (Sonnet)
perf-VM runs of the #260 and #261 kits, booked with calternal

Each brief must say "Finish what you start", and start from the branch head in §5.

8. Resume checklist

Confirm the Claude-only rule is still in force, and the Codex reset date.
Ask the maintainer whether the 10by32-02 reboot is done, and whether infraternal-dev-02-std exists (move work there).
Relaunch, one per free slot, in this order: #220 slice-1 finish, #258 finish, #250 round-2 review, #237 round-3 review, #211, #207 review, #259 finish, #260, #261, then roadmap items 2–3 (sign-in, mesh DNS), D314, D315, the mockup rebuild. Each brief starts from the branch head above, and the previous job's findings file stays in ~/.cache/codex-scratch/<job>/ on that VM.
Re-check §6 with kayg.

Handover from the infraternal orchestrator on `code` (10.69.69.92), 2026-10-30 23:06 IST, **updated 23:13 IST**. **All work is paused, committed and pushed.** Nothing of ours is running on any VM (only idle Claude sessions of the paused jobs remain), and every build-pool lease is released. At 23:10 IST four resume jobs were started and cancelled within two minutes on kayg's order; they made no changes (sweep re-run: 69 worktrees clean). Calternal is also paused tonight (its handover is kayg/calternal#774). Earlier handover: #254. The live status board stays #24. This repo is public. No credentials appear here. Secret files are referenced by path only and are never to be printed. ## 1. Rules in force - **Claude subagents only, no Codex** (kayg, 2026-10-30 21:10 IST, for at least 24 h, until kayg lifts it). The shared Codex account on every build VM hit its usage limit until **2026-11-05 07:26**. Use `delegate_task` with `providerInstanceId: claudeAgent`: Opus 5.5 (high) for security, privacy and protocol work; Sonnet 5.5 for UI and mechanical work. Break-it reviews also run on Claude. - Builds and tests never run on `code`; they go to the build VMs. Every job reads `~/infraternal-ops/rules.md` (the master copy is `~/.local/state/codex-jobs/infraternal/orchestrator/rules.md` on code; copy it to each VM after any change). New section 2026-10-30, **"Finish what you start"**: a job never ends its turn while its own background work still runs. Claude jobs repeatedly did this and needed a "continue" message. - **Three review rounds max**: if round 3 still finds new P0/P1, stop patching and write a plan. - D243 sizes: under 16 KiB is approved by the orchestrator; 16–64 KiB needs a written reason; over 64 KiB needs kayg. - Reviewed and staged fixes deploy without asking; kayg unlocks the CA after each restart. - Coordinate builds and perf runs with calternal (thread `import:claudeAgent:b6baf59e…`). The perf VMs perf-test-01/02/03 are reached only through `/usr/local/bin/perf-run.sh`, or as root through `perf-lock.sh` + `systemd-run --scope --slice=perf.slice`, leaving an empty before/after machine diff. - Times go to kayg in IST. ## 2. Production - **Release 9.5 (`0c8639496`) is live on monitor01** since 2026-10-30 17:02 IST, via `deploy-prod.sh` from code. Backup: `/var/backups/infraternal/20261030-113220-0c8639496`. `release-prod-live` = `0c8639496`. kayg unlocked the CA. - Shim memory 9.5 vs 9.4 at 50 sessions (#24): PSS/USS flat. RSS +2.7% comes from shared pages. **Open:** in 1 of 4 fixtures, about 17 of 50 shims grew by 2 MiB USS while idle; needs ≥10 fixtures to find the cause. - 59 leftover sessions from 17 Oct were closed on kayg's OK; kayg's 3 sessions remain. - `dev` is 7 commits ahead of production: decisions and docs only (D313–D316, the PROCESS.md release rule), no product code. ## 3. Decisions recorded today (on dev) - **D312** `2fb412d01`: layout edits live per tab; the server is the only saved copy (#250). - **D313** `db8c1bdf6`: mesh direct paths and relays, #220 grill Q1–Q7 (seam C, home-relay rendezvous, any relay, hard-NAT gated on counters, switch rule max(5 ms, 15%), TLS+WebSocket 443, QUIC DATAGRAM). - **D314** `c9671cb79` + `f9d0a9a8f`: one CLI command set (`sessions …`, `canvas …`, `open`), plus a tmux-compatible subset for AI tools. kayg: "support tmux style commands yes because lots of training data". - **D315** `c9671cb79`: never interfere with auto-started multiplexers; offer the guard line with a confirmed "add" button (replaces D241's transparent skip). - **D316** `a4f15f8f7` + `42ab33035`: str0m's ICE core for mesh path finding (+208 KB accepted), our own port mapping; `.internal` names by default (D165) plus custom subdomains. **The reverse proxy runs on every client, without root above port 1024, with DNS-01 certificates; we never write our own proxy engine.** The engine is chosen by kayg after the #261 measurement (Traefik can't be embedded in the Rust binary). - `docs/PROCESS.md` `63d8a03dc`: release candidates pass the full browser suite twice in a row, retries 0 (#207 proposal, approved). ## 4. Roadmap (kayg 2026-10-30: "accelerate the mesh to replace NetBird first; the multiplexer in parallel") **Mesh lane, in order:** 1. #220 relay + seam C (slice 1 done, branch `mesh-relay-220`); 2. `infraternal sign-in <server>` (link, code, QR, passkey; D311); 3. mesh DNS (`.internal` + custom suffix, records, upstream; D165/D197); 4. `infraternal up` laptop join + one-command install; 5. the macOS node (launchd + utun, on macos-02-std); 6. SSH over the mesh (#221). Later in the same lane: rendezvous + simultaneous open, upgrade and flip-back on the #201 selector, own port mapping, str0m path checks, the reverse proxy per D316. **Multiplexer lane, in parallel:** the D314 `sessions`/`canvas` commands + tmux subset + MCP (spec slices on #251 first); D315 guard line (rework branch `d241-shell-marker`, which must shrink from D241's +40 KB). **Measure first, to answer kayg's question:** #260 head-to-head (our mesh vs NetBird/Tailscale, our sessions vs tmux/Zellij) and #261 (reverse-proxy engines). ## 5. Work in flight (all paused; heads on origin) | Issue | Branch @ head | State | Next | |---|---|---|---| | #258 Mesh pins: no fixed limit, visible-ID rule, 30-day drop (kayg-approved design) | `mesh-pins-258-v2` @ `6b4211583` (WIP) | Opus continuation was midway: renamed-tag fix, reads without the writer lock, clippy/TS fixed. 18/20 pin tests were passing; 2 float-vs-int expectations were corrected and not yet re-run | Split the WIP into atomic commits, rebase, both sizes, `verify fast`, then a break-it review. The canvas operation-log rule is D183/D248 (not D235) | | #250 per-tab layout (D312) | impl `layout-per-tab-250` @ `7e2490f07`; review `review-layout-per-tab-250-r2` @ `80c2a3578` (WIP) | Round-2 review paused mid phase A (gate + Chromium + WebKit); findings so far are on #250 | Finish the round-2 review: phase A, then phase B (dev `2fb412d01` static + WebKit). **Perf kit** (`361fac63…`) fails on perf-test-03 as real root: terminal stuck "connecting". The next fix round must reproduce it as real root on a build VM | | #207 broad suite triage | `broad-suite-triage` @ `fc65de7b0`; review `review-broad-suite-207` @ `d9a16ed2c` (WIP) | Provisional **merge after fixes**: P0 0, P1 0, P2 7, P3 1. R207-02: old session's pixels stay after "New shell". R207-03: the attach ≤60 ms bound is lost and `REAL_SWITCH_BUDGETS` is never set. 4 extra broad runs 112/15/0 | `verify fast` on `fc65de7b0` (worktree `~/.cache/codex-scratch/review-broad-suite-207/head-fc65` on 8by64), sizes, final verdict, then a fix round | | #237 status during crash loop | `status-crash-loop-237` @ `f9859712c`; r3 review `review-status-crash-loop-237-r3` @ `9bf549962` (WIP) | Fix round 2 done (status p50/p99 14.8/18.8 ms); the **final round-3 review** was mid-run | Finish round 3. A new P0/P1 means stop and write a plan (three-round rule) | | #211 edge auto-update (plan `plan-211-update` @ `c9c1877e9` accepted) | `edge-update-v2` @ `fd1892201` (WIP) | 5 gated commits + 1 WIP; e2e under real systemd 6/7 (the 7th is a test-script bug: `open()` vs `os.open`) | Automatic updater, manual rollback, hosts follow the server's version, full-disk cases, the perf-VM kit, sizes, gate. Its ~56 GB target dir on 6by32 is kept | | #220 mesh relay slice 1 | `mesh-relay-220` @ `4fe5cd912` (WIP) | QUIC DATAGRAM relay, seam C on the WG port, failover. Namespace proof 6/6; fixed the 1452→1472 MTU drop | Proper commits, release numbers, `contracts/mesh-relay.md`, gates, size (expected 0 B shipped) | | #220 measurement | `mesh-measure-220` @ `ced872e68` | Done. Results are on #220 (seam B ≈ kernel; seam A 2× CPU; 270 B per relayed pair) | — | | #260 head-to-head kit | `research-head-to-head` @ `78ccce47e` (WIP) | `h2h` tool built; smoke passes for 5 mesh cases + tmux/Zellij. **Possible product bug:** server closes the stream with 1013 during a 100 MB `cat` (keepalive ≈500 ms) | Triage the 1013 case (is it real for browsers?), WAN/loss/scale cases, x86 kit, run on a perf VM | | #261 reverse-proxy engines | `research-reverse-proxy` @ `ad7daa89f` (WIP); scratch `wip/rproxy-delta-scratch-2026-10-30` | All four pass the functional matrix. x86 adds: Pingora +5.3 MB (panics on tokio 1.54.1), Sozu +6.4 MB (AGPL/LGPL, not in deny allow-list), rpxy +4.7 MB (git-only, aws-lc). instant-acme DNS-01 works | Full-duration timings, x86 kit, write-up; then kayg decides | | #259 sidebar pill drift + double-tap rename | `sidebar-pill-259` @ `e8c969d00` | Implementation pushed (9 commits over dev) | Production screenshots + bundle size + x86 `verify fast`, then my visual review | | #146 app windows | helper `appwin-helper-macos02` @ `a2b8dbf9e` | Helper built + signed on macos-02-std, no prompt. `--check-permissions` → both false | **Waiting on kayg's TCC choice** (see §6). Then capture/latency work | | #256 frosted sideways scroll | `frosted-scroll-256` @ `88ae0a23d` | Fix by default; `.cl-viewport` clip-path limits Chromium blur | kayg device test (Aside/Chromium on M2 Max); the preview at :47411 was unreachable from kayg's device | | #241/D315 | `d241-shell-marker` @ `e18cd7a0e` | Superseded by D315 | Rework to D315 (queued) | **Safety snapshots pushed today:** `wip/d241-size-staged-2026-10-30`, `wip/stale-note-257-staged-2026-10-30` (half-finished merges found staged in those worktrees; the worktrees are left as found), `wip/stash-8by64-stash0-2026-10-30` (stash "setup-token-data-step-status"), `perf24-reconnect` (2 commits pushed). On `code`: `wip/code-merge-release94-dev-2026-10-30` (a stale unfinished 9.4 merge with 2 conflicts; the real merge f069bb2a9 is on dev) and `wip/code-serve-{clusters,demo,glass}-2026-10-30` (the local preview tweak to `mock-server.ts`). All snapshots were taken with a temporary index, so the worktrees are unchanged. Final sweep at 23:20 IST: every worktree on all VMs and on `code` is either clean and pushed, or saved to one of these branches; the preview worktrees sit on archive tags that are on origin. Evidence from the old std VM is archived on code at `~/.local/share/infraternal-std-archive-2026-10-30/`. ## 6. Decisions waiting for kayg 1. **#146 Screen Recording + Accessibility + PostEvent for `com.infraternal.desktop.helper`** on macos-02-std: **kayg said at 23:10 IST that they will grant these themselves**, for `/Users/lume/Applications/InfraternalCaptureHelper.app`. Verify afterwards with the helper's `--check-permissions` (see #146), which must return both true. Accessibility already has an "off" row that macOS created itself. csreq: `fade0c00000000580000000100000006000000020000001e636f6d2e696e6672617465726e616c2e6465736b746f702e68656c706572000000000004000000000000001462cbdf2863333c4631609ee3972d32c0a3791ba7`. 2. **macOS deployment target** for the helper: the build is for macOS 26.0, while the crate claims 14.2. Build for 14.2? 3. **10by32-02-std reboot** (maintainer, to enable free-page reporting): **not done yet** (still up since 14:33 UTC at 23:13 IST). All our jobs there are paused, so it can reboot now. The T3 maintenance thread (`6bfdd7b3…`) no longer exists, so kayg must give it the go. The 10by32s will be phased out onto the new dev VMs (`infraternal-dev-02-std`). 4. **DNS-01 provider token** (e.g. Cloudflare): stored sealed and used only by the node requesting certificates, then moved to OpenBao? Recommended yes. 5. **Reverse-proxy engine** (after #261 completes): every embeddable option costs +4.7–6.4 MB, about 300× the 16 KiB band. 6. **Mockups to approve** (a rebuild of all of them at one URL is approved; the job isn't started yet): row menu + swipes #242; glassy machine cards #94; account card #224; version pill #216; host rename/consent #215; boot machines #156; sessions UI #104 (partly superseded by D311/D314); connection status icons. The add-host pill #189 is approved and only needs building. 7. **#256:** keep or remove the `.cl-viewport` clip-path (real blur vs cost), and which URL/device couldn't open the preview. 8. **VM-to-VM SSH:** kayg allowed it, but it only partly works (std↔std and 8by64→6by32/std time out; OrbStack DNS broke short names on the old std VMs). 9. OpenBao deployment (kayg), then move secret use onto it. After infraternal-dev is deleted, remove it from production's host list. 10. **#146 encoder grill:** which video encoder path app windows use (hardware VideoToolbox H.264 is proven on the VM; the codec and bitrate policy aren't decided). Ask it as a grill with measured options once the helper has its permissions. ## 7. Infrastructure **Machine split, agreed with calternal at 23:08 IST and reported to kayg:** - **Infraternal:** 8by64-compute02, 6by32-compute02, 10by32-02-std, macos-02-std, and infraternal-dev-02-std when it exists. - **Calternal:** 8by64-compute01 (CI/staging), 6by32-compute01, 10by32-01-std, macos-01-std, and calternal-dev-02-std. - **Shared, measurement only, by lock:** perf-test-01/02/03. - **Borrowing:** ask in the calternal thread first, then `pool lease --host … --placed`. - **Tonight:** calternal lent us **6by32-compute01** (agent, x86) while it's paused. Our setup there: `~/infraternal`, `~/infraternal-env.sh`, `~/infraternal-ops`, T3 env `2f385a8d-4422-48d7-ab4e-4bd35ddcb7f8`, project `mcp:4e961d17-e2cd-4a7b-a3b2-1e94ef3745ec`. Hand it back when calternal resumes. | Machine | Access | T3 environmentId / project | Notes | |---|---|---|---| | 8by64-compute02 (x86) | `netbird ssh agent@8by64-compute02` | `82ceea33-a5d6-4ddf-9315-9fe0eee9774f` / `mcp:2f293fb9…` | ≤3 jobs; about 67 GB free; review target dirs (~38 GB for #207) can be deleted when those lanes close | | 6by32-compute02 (x86) | `agent@6by32-compute02` | `f8fd0eac-f81a-4562-b382-73f0ba8cb25d` / `mcp:6b7d764e…` | ≤2 jobs; #211 target ~56 GB kept | | 10by32-02-std (arm64, cua spaces) | `netbird ssh cua@10by32-02-std.vpn.kayg.org` | `21b9f017-1060-4df4-9e17-3b2ccf8a0a1c` / `mcp:866ac412-bc6d-4133-9b92-031b0acbfa6a` | Ours by agreement (calternal has -01). `~/infraternal`, `~/infraternal-env.sh` (5 jobs), `~/infraternal-ops`. Static-size gate rows run on x86 | | macos-02-std (macOS 26.5.2, cua spaces) | `netbird ssh lume@macos-02-std.vpn.kayg.org` | `72ff8e68-41b2-4a5d-ba03-14d22ba85b6c` / `65a265e6-12d5-4d39-82b6-99948f7e1630` | Ours (calternal has macos-01). NOPASSWD sudo for lume; source `/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh` for nix. No Forgejo key: git is relayed by `orchestrator/vm/macos-sync.sh to|from` over NetBird SSH. Signing keychain `~/Library/Keychains/infraternal-signing.keychain-db` (unlock with `~/.config/infraternal-signing/unlock.sh`; never read the password file); identity SHA-1 `62CBDF28…1BA7`. Never touch the login keychain (it's locked over SSH and raises a SecurityAgent window). GUI-session commands: `sudo launchctl asuser 501 sudo -u lume …` | | perf-test-01/02/03 (x86) | via calternal agreement | — | perf-test-03 has only our user-level `~/infraternal-perf` (Chromium, nix out-link, kits). Delete it when the #250/#211 perf runs are done | | monitor01 (prod) | `netbird ssh root@10.69.69.50` from code | — | `deploy-prod.sh` in `~/.local/share/infraternal-preview/` | - **Build pool** `~/.local/state/build-pool/` (78 tests): `--host`/`--placed` to register jobs that are already placed, a per-host `user` (cua on std), `[groups.std] shared_storage=false`. Register every job as placed and run a heartbeat unit (`pool-hb-<job>`); release it when the job ends. - **Previews on code:** :47361 demo, :47381 recovery, :47391 status, :47401 clusters (row menu + infrafetch combo), :47411 #256 glass. All are old; the mockup rebuild replaces them. - **Orchestrator state:** `~/.local/state/codex-jobs/infraternal/orchestrator/` (`queue.md` log, `rules.md`, `spawn-log.md`, `vm/` scripts incl. `sweep-unpushed.sh`). Memory index: `~/.claude/projects/-home-kayg-Developer-Infraternal/memory/MEMORY.md`. ## 7a. Problems found in the infrastructure - On 8by64, about 300 orphaned `__shim` processes (8.5 h old) were left by the first #250 review. Paused or cancelled jobs leave test servers behind. Sweep by age after each job (`ps -eo etimes`). - `pkill -f <pattern>` over `netbird ssh` matches its own command line. Use `pgrep` + `kill`, or split the pattern. - `bc` isn't installed on code. ## 7b. Slot plan for the next session (planned 23:08 IST, not started) | Machine | Jobs (all Claude Opus unless noted) | |---|---| | 8by64-compute02 | #220 relay slice 1 finish (moved to x86 for the gate); #258 finish; #250 round-2 review finish | | 6by32-compute02 | #237 round-3 final review finish, then the next slot; #211 continue | | 6by32-compute01 (borrowed) | mesh step 2: `infraternal sign-in <server>` (link, code, QR, passkey; D311) | | 10by32-02-std (after the reboot) | #260 head-to-head finish + triage the 1013 disconnect on heavy output; #261 proxy finish (Sonnet); multiplexer: the D314 `sessions` commands + tmux subset + MCP | | macos-02-std | #146 first real capture and latency, once kayg's grant is in | **Then, as slots free:** 1. mesh DNS 2. #207 fix round 3. #259 screenshots + gate (Sonnet) 4. D315 guard line 5. mockup rebuild at one URL (Sonnet) 6. perf-VM runs of the #260 and #261 kits, booked with calternal Each brief must say "Finish what you start", and start from the branch head in §5. ## 8. Resume checklist 1. Confirm the Claude-only rule is still in force, and the Codex reset date. 2. Ask the maintainer whether the 10by32-02 reboot is done, and whether `infraternal-dev-02-std` exists (move work there). 3. Relaunch, one per free slot, in this order: #220 slice-1 finish, #258 finish, #250 round-2 review, #237 round-3 review, #211, #207 review, #259 finish, #260, #261, then roadmap items 2–3 (sign-in, mesh DNS), D314, D315, the mockup rebuild. Each brief starts from the branch head above, and the previous job's findings file stays in `~/.cache/codex-scratch/<job>/` on that VM. 4. Re-check §6 with kayg.

kayg commented

2026-10-30 17:36:36 +00:00

Author

Owner

Addendum to §6 (decisions waiting for kayg):

#146 encoder grill (carried over from #254): which video encoder path app windows use (hardware VideoToolbox H.264 is proven on the VM; the codec and bitrate policy aren't decided). Ask it as a grill with measured options once the helper has its permissions.

Addendum to §6 (decisions waiting for kayg): 10. **#146 encoder grill (carried over from #254):** which video encoder path app windows use (hardware VideoToolbox H.264 is proven on the VM; the codec and bitrate policy aren't decided). Ask it as a grill with measured options once the helper has its permissions.

kayg referenced this issue

2026-10-30 17:36:37 +00:00

Orchestrator handover 2026-10-28: moving to a new machine #254

kayg referenced this issue

2026-10-31 16:12:46 +00:00

Mesh: NAT traversal and relay for mesh peers #220

kayg referenced this issue

2026-10-31 18:12:30 +00:00

Move production from monitor01 to ssh #274

kayg referenced this issue

2026-11-01 10:16:25 +00:00

Secrets: an intuitive interface over OpenBao #287