kayg/infraternal

Watch

Fork 0

Release: nix build fails on dev, Zig 0.17 Module.zig patch anchor gone (blocks deploy) #476

New issue

Open

opened 2026-11-09 14:04:35 +00:00 by kayg · 4 comments

kayg commented

2026-11-09 14:04:35 +00:00

Owner

nix build .#packages.x86_64-linux.default fails on origin/dev 198c4a736 (compute01, log ~/deploy-198c4a736.log):

error: Cannot build '/nix/store/...-infraternal-zig-0.17.0.drv'.
  > AssertionError: native build target boundary changed

The python step in nix/package.nix that patches the pinned Zig's lib/std/Build/Module.zig (native targets get an explicit -target/-mcpu/--dynamic-linker) no longer finds its anchor. Suspect: 2c7135211 (#419) moved Zig 0.15.2 → 0.17.0. Production last deployed at 54a9da2a8. This blocks every deploy.

Gate gap: verify fast builds statics through scripts/static-cargo, not the Nix package, and merge.sh only checks cargo, so a broken Nix release build merged.

Plan (J442, ~2 h):

Confirm the breaking commit by building the zig derivation at 2c7135211^ and 2c7135211 on compute01 (build-slot).
Port the Module.zig patch to Zig 0.17.0 with the same semantics, keep the exactly-once assertion and the pinned archive (D213/D328); check the system.zig /usr/bin/env substitution.
Full nix build on x86_64 (compute01) and aarch64 (dev-02 if the flake has it); verify SHA256SUMS.
Add a Nix package build row to verify fast or merge.sh; measure and report the added time.

Branch fix-nix-zig-patch from origin/dev.

`nix build .#packages.x86_64-linux.default` fails on origin/dev 198c4a736 (compute01, log `~/deploy-198c4a736.log`): ``` error: Cannot build '/nix/store/...-infraternal-zig-0.17.0.drv'. > AssertionError: native build target boundary changed ``` The python step in `nix/package.nix` that patches the pinned Zig's `lib/std/Build/Module.zig` (native targets get an explicit `-target`/`-mcpu`/`--dynamic-linker`) no longer finds its anchor. Suspect: 2c7135211 (#419) moved Zig 0.15.2 → 0.17.0. Production last deployed at 54a9da2a8. This blocks every deploy. Gate gap: `verify fast` builds statics through scripts/static-cargo, not the Nix package, and merge.sh only checks cargo, so a broken Nix release build merged. Plan (J442, ~2 h): 1. Confirm the breaking commit by building the zig derivation at 2c7135211^ and 2c7135211 on compute01 (build-slot). 2. Port the Module.zig patch to Zig 0.17.0 with the same semantics, keep the exactly-once assertion and the pinned archive (D213/D328); check the `system.zig` `/usr/bin/env` substitution. 3. Full `nix build` on x86_64 (compute01) and aarch64 (dev-02 if the flake has it); verify SHA256SUMS. 4. Add a Nix package build row to `verify fast` or merge.sh; measure and report the added time. Branch `fix-nix-zig-patch` from origin/dev.

kayg commented

2026-11-09 14:17:01 +00:00

Author

Owner

Start: branch fix-nix-zig-patch from origin/dev 198c4a736.

Bisect (compute01, building only the package's infraternal-zig-* derivation per commit; log ~/.cache/codex-scratch/j442-nix/bisect.log on code):

== 54a9da2a8  -> infraternal-zig-0.15.2  built /nix/store/09zlpcz6...-infraternal-zig-0.15.2
== 2c7135211^ -> infraternal-zig-0.15.2  built (same drv)
== 2c7135211  -> infraternal-zig-0.17.0  AssertionError: native build target boundary changed
== f4e947e7d  -> infraternal-zig-0.17.0  AssertionError (same drv)

2c7135211 ("build(shim): upgrade to signed libghostty-vt 1.0", #419) is the first bad commit. It moved Zig 0.15.2 → 0.17.0 (nix/zig.nix + scripts/zig.sha256). Nothing else in nix/ or the pins changed between 54a9da2a8 and dev.

Cause: Zig 0.17 no longer builds compiler argv in lib/std/Build/Module.zig (it has no isNative at all). The make step does it in lib/compiler/Maker/Step/Compile.zig:

if (m.resolved_target.get(conf)) |resolved_target| {
    if (resolved_target.unwrapQuery(conf)) |query| {   // null for host targets
        -target / -mcpu / --dynamic-linker ...

A host target's query is stored as .none, so the patch is still needed: stock 0.17 passes no -target and the unpatched compiler binary re-detects the host via /usr/bin/env, which the sandbox lacks. system.zig still has "/usr/bin/env" (lines 993, 1015) and --replace-fail still applies.

Fix (9749cc236): the patch now anchors on that unwrapQuery line (asserted exactly once). For .none it rebuilds the query from the configured result, using std.Target.Query.fromTarget's add/sub feature arithmetic. 0.17 stores result as "model + every enabled feature", so without subtracting, the model's defaults could switch on features the build host lacks. Cross queries are untouched. The patched Zig derivation builds: /nix/store/q756662d...-infraternal-zig-0.17.0.

Gate row (515bdf9a2): scripts/check-nix-release-inputs is now in fast/iterate. It builds binaries.inputDerivation + bunDeps (patched Zig, signed VT sources, Cargo/Bun vendor trees, Rust toolchain) without compiling.

on unmodified origin/dev: rc=1, Cannot build ...infraternal-zig-0.17.0.drv > AssertionError: native build target boundary changed (red, as wanted)
at the branch head: rc=0. Warm 0.25/0.25/0.33 s (eval cache); 3.6 s with --option eval-cache false.

Full nix build for x86_64 (compute01) and aarch64 (dev-02) is queued in build-slot (compute01 queue position 12). Next: SHA256SUMS check, the full-build time, and a merge.sh proposal.

Start: branch `fix-nix-zig-patch` from origin/dev 198c4a736. **Bisect (compute01, building only the package's `infraternal-zig-*` derivation per commit; log `~/.cache/codex-scratch/j442-nix/bisect.log` on code):** ``` == 54a9da2a8 -> infraternal-zig-0.15.2 built /nix/store/09zlpcz6...-infraternal-zig-0.15.2 == 2c7135211^ -> infraternal-zig-0.15.2 built (same drv) == 2c7135211 -> infraternal-zig-0.17.0 AssertionError: native build target boundary changed == f4e947e7d -> infraternal-zig-0.17.0 AssertionError (same drv) ``` 2c7135211 ("build(shim): upgrade to signed libghostty-vt 1.0", #419) is the first bad commit. It moved Zig 0.15.2 → 0.17.0 (nix/zig.nix + scripts/zig.sha256). Nothing else in nix/ or the pins changed between 54a9da2a8 and dev. **Cause:** Zig 0.17 no longer builds compiler argv in `lib/std/Build/Module.zig` (it has no `isNative` at all). The make step does it in `lib/compiler/Maker/Step/Compile.zig`: ```zig if (m.resolved_target.get(conf)) |resolved_target| { if (resolved_target.unwrapQuery(conf)) |query| { // null for host targets -target / -mcpu / --dynamic-linker ... ``` A host target's query is stored as `.none`, so the patch is still needed: stock 0.17 passes no `-target` and the unpatched compiler binary re-detects the host via `/usr/bin/env`, which the sandbox lacks. `system.zig` still has `"/usr/bin/env"` (lines 993, 1015) and `--replace-fail` still applies. **Fix (9749cc236):** the patch now anchors on that `unwrapQuery` line (asserted exactly once). For `.none` it rebuilds the query from the configured `result`, using `std.Target.Query.fromTarget`'s add/sub feature arithmetic. 0.17 stores `result` as "model + every enabled feature", so without subtracting, the model's defaults could switch on features the build host lacks. Cross queries are untouched. The patched Zig derivation builds: `/nix/store/q756662d...-infraternal-zig-0.17.0`. **Gate row (515bdf9a2):** `scripts/check-nix-release-inputs` is now in fast/iterate. It builds `binaries.inputDerivation` + `bunDeps` (patched Zig, signed VT sources, Cargo/Bun vendor trees, Rust toolchain) without compiling. - on unmodified origin/dev: rc=1, `Cannot build ...infraternal-zig-0.17.0.drv > AssertionError: native build target boundary changed` (red, as wanted) - at the branch head: rc=0. Warm 0.25/0.25/0.33 s (eval cache); 3.6 s with `--option eval-cache false`. Full `nix build` for x86_64 (compute01) and aarch64 (dev-02) is queued in build-slot (compute01 queue position 12). Next: SHA256SUMS check, the full-build time, and a merge.sh proposal.

kayg commented

2026-11-09 15:40:44 +00:00

Author

Owner

Progress (head 515bdf9a2):

aarch64 full package build: green (dev-02, through build-slot, at 9749cc236; log ~/.cache/codex-scratch/j442/full-arm.log + build-aarch64-linux.log on dev-02):

nix build rc=0 wall=802s out=/nix/store/1xss9m4n3rxm0wg1svsdl7yx7dj7x57q-infraternal-0.1.0
sha256sums rc=0      (552 entries: OK)
SOURCE = 9749cc236c19270ed223adfb7a7b25d603485c93
bin/infraternal --version -> infraternal 0.1.0 (stable)
bundle: infraternal-{aarch64,x86_64}-musl, *-session-musl, licenses, web, SHA256SUMS

802 s includes the cold static builds (the binaries derivation changed with the source).

verify quick on dev-02: QUICK pre-check passed, VERIFY_EXIT=0 (~/.cache/codex-scratch/j442/quick.log).

Waiting: x86_64 full nix build (compute01 build-slot, queue position 4 after 1 h 29 min) and verify fast (dev-02 queue).

Progress (head 515bdf9a2): **aarch64 full package build: green** (dev-02, through build-slot, at 9749cc236; log `~/.cache/codex-scratch/j442/full-arm.log` + `build-aarch64-linux.log` on dev-02): ``` nix build rc=0 wall=802s out=/nix/store/1xss9m4n3rxm0wg1svsdl7yx7dj7x57q-infraternal-0.1.0 sha256sums rc=0 (552 entries: OK) SOURCE = 9749cc236c19270ed223adfb7a7b25d603485c93 bin/infraternal --version -> infraternal 0.1.0 (stable) bundle: infraternal-{aarch64,x86_64}-musl, *-session-musl, licenses, web, SHA256SUMS ``` 802 s includes the cold static builds (the binaries derivation changed with the source). `verify quick` on dev-02: `QUICK pre-check passed`, VERIFY_EXIT=0 (`~/.cache/codex-scratch/j442/quick.log`). Waiting: x86_64 full `nix build` (compute01 build-slot, queue position 4 after 1 h 29 min) and `verify fast` (dev-02 queue).

kayg commented

2026-11-09 16:47:58 +00:00

Author

Owner

Finding: x86_64 full build failed at 9749cc236 — bug in my first patch, fixed in fc4c52e78.

compute01 nix build (rc=1, 535 s; log ~/.cache/codex-scratch/j442/build-x86_64-linux.log): every native generator's compile SEGVs, e.g.

error: process terminated with signal SEGV
failed command: .../infraternal-zig-0.17.0/zig build-exe -target x86_64-linux.6.12.115...6.12.115-gnu.2.44 -mcpu skylake --dynamic-linker "96\024\001" ... --name combine_archives

Cause: I sent the rebuilt host query through upstream's if (query.dynamic_linker) |*dynamic_linker| emission. It appends a slice of the DynamicLinker's inline buffer, and that buffer belongs to a stack-local copy, so the slice dangles. Reproduced outside Nix with the same zig and flags: a real loader path compiles; "96\024\001" segfaults. aarch64 happened to survive.

Fix (fc4c52e78, force-pushed over the WIP): the host branch emits its own flags and arena.dupes the loader path, in the same if (host) {...} else {upstream} shape as the 0.15 patch, so upstream's cross path is untouched. A tiny native zig build --verbose with the patched compiler (log argv.log):

-target x86_64-linux.6.12.115...6.12.115-gnu.2.44 -mcpu skylake
--dynamic-linker /nix/store/pqyiidz5...-glibc-2.44-57/lib/ld-linux-x86-64.so.2
binary ran

verify fast at 3575f6b9d (dev-02, branch merged with origin/dev 5eb968744): VERIFY_EXIT=1. Every Rust row passes, Nix release inputs included; static builds PASS; privileged integration SKIP. One TS failure: (fail) a closing notice cannot hide its replacement (apps/tracer startup-toast.test.ts). It fails 5/5 on unmodified origin/dev 5eb968744 too (bun test src/canvas/sessions/startup-toast.test.ts → 0 pass 1 fail ×5), so it predates this branch. The first fast run (before the dev merge) also hit a load flake, daemon::gone_tests::a_vanished_session_gets_a_final_close_and_the_channel_stays_up (got close code 1 "terminal shim disconnected" instead of 4404); it passed 8/8 in isolation and in the second run.

Disk: dev-02 j442 target deleted (35G) and nix store gc freed 8.2 GiB; dev-02 is at 88G free.

Queued: x86_64 and aarch64 full builds at fc4c52e78.

**Finding: x86_64 full build failed at 9749cc236 — bug in my first patch, fixed in fc4c52e78.** compute01 `nix build` (rc=1, 535 s; log `~/.cache/codex-scratch/j442/build-x86_64-linux.log`): every native generator's compile SEGVs, e.g. ``` error: process terminated with signal SEGV failed command: .../infraternal-zig-0.17.0/zig build-exe -target x86_64-linux.6.12.115...6.12.115-gnu.2.44 -mcpu skylake --dynamic-linker "96\024\001" ... --name combine_archives ``` Cause: I sent the rebuilt host query through upstream's `if (query.dynamic_linker) |*dynamic_linker|` emission. It appends a slice of the DynamicLinker's inline buffer, and that buffer belongs to a stack-local copy, so the slice dangles. Reproduced outside Nix with the same zig and flags: a real loader path compiles; `"96\024\001"` segfaults. aarch64 happened to survive. Fix (fc4c52e78, force-pushed over the WIP): the host branch emits its own flags and `arena.dupe`s the loader path, in the same `if (host) {...} else {upstream}` shape as the 0.15 patch, so upstream's cross path is untouched. A tiny native `zig build --verbose` with the patched compiler (log `argv.log`): ``` -target x86_64-linux.6.12.115...6.12.115-gnu.2.44 -mcpu skylake --dynamic-linker /nix/store/pqyiidz5...-glibc-2.44-57/lib/ld-linux-x86-64.so.2 binary ran ``` **verify fast at 3575f6b9d (dev-02, branch merged with origin/dev 5eb968744):** VERIFY_EXIT=1. Every Rust row passes, `Nix release inputs` included; static builds PASS; privileged integration SKIP. One TS failure: `(fail) a closing notice cannot hide its replacement` (apps/tracer startup-toast.test.ts). **It fails 5/5 on unmodified origin/dev 5eb968744 too** (`bun test src/canvas/sessions/startup-toast.test.ts` → `0 pass 1 fail` ×5), so it predates this branch. The first fast run (before the dev merge) also hit a load flake, `daemon::gone_tests::a_vanished_session_gets_a_final_close_and_the_channel_stays_up` (got close code 1 "terminal shim disconnected" instead of 4404); it passed 8/8 in isolation and in the second run. Disk: dev-02 j442 target deleted (35G) and `nix store gc` freed 8.2 GiB; dev-02 is at 88G free. Queued: x86_64 and aarch64 full builds at fc4c52e78.

kayg commented

2026-11-09 23:42:47 +00:00

Author

Owner

Finish: head fc4c52e785 (git ls-remote confirms it on fix-nix-zig-patch). Not merged, not deployed.

x86_64 full package build, green (compute01, build-slot; log ~/.cache/codex-scratch/j442/full-x86-2.log + build-x86_64-linux.log):

nix build rc=0 wall=4022s out=/nix/store/gfmr4s3yhgkzrdsxkzva0ji2dpcyr01j-infraternal-0.1.0
sha256sums rc=0   (552 entries OK)
SOURCE fc4c52e7853e0547edca26d5af88dcd6a562bd32
bin/infraternal --version -> infraternal 0.1.0 (stable)
SEGV in log: 0

4022 s is wall time inside the slot while compute01 ran at load 14–25 with five other gates. It is not an idle-machine number.

aarch64 full package build, green (dev-02):

nix build rc=0 wall=903s out=/nix/store/wr0dpxiqaprwbbka2k71rgqi6l7ynyi6-infraternal-0.1.0
sha256sums rc=0   (552 entries OK); --version -> infraternal 0.1.0 (stable)

Gate: verify fast ran at 3575f6b9d (dev-02, origin/dev 5eb968744 merged): VERIFY_EXIT=1. All Rust rows PASS (Nix release inputs PASS, static builds PASS, privileged integration SKIP). The only failure is the TS test a closing notice cannot hide its replacement, which also fails 5/5 on unmodified origin/dev (comment above). Since that run, head has changed only nix/package.nix (fc4c52e78). The new row passes at fc4c52e78 in 19.2 s on compute01; the zig input drv was rebuilt on that run.

Gate-gap cost and proposal:

New verify fast/iterate row Nix release inputs: 0.25 s with a warm eval cache, 3.6 s without, 19 s when the patched Zig must rebuild. Red on origin/dev with the exact deploy error, green here.
A full nix build per merge costs 903 s (aarch64) to ~67 min (x86 under load), because it recompiles both static images a second time. Too slow for every merge. Proposed for merge.sh (orchestrator-owned, not edited): run nix build --no-link .#packages.x86_64-linux.default (timeout 2400) only when the merged diff touches nix/ flake.nix flake.lock rust-toolchain.toml Cargo.lock apps/tracer/bun.lock scripts/{zig.sha256,libghostty-vt.sha256,auth-toolchain,build-vt,patch-libghostty-vt.py,static-cargo,package-release} scripts/release/ crates/infraternal-shim/build.rs. 2c7135211 touched three of these. Also run it nightly on dev.

Disk: dev-02 target deleted, gc'd; dev-02 at 88G free. compute01 has no j442 target (logs only).

**Finish: head fc4c52e7853e0547edca26d5af88dcd6a562bd32** (`git ls-remote` confirms it on `fix-nix-zig-patch`). Not merged, not deployed. **x86_64 full package build, green** (compute01, build-slot; log `~/.cache/codex-scratch/j442/full-x86-2.log` + `build-x86_64-linux.log`): ``` nix build rc=0 wall=4022s out=/nix/store/gfmr4s3yhgkzrdsxkzva0ji2dpcyr01j-infraternal-0.1.0 sha256sums rc=0 (552 entries OK) SOURCE fc4c52e7853e0547edca26d5af88dcd6a562bd32 bin/infraternal --version -> infraternal 0.1.0 (stable) SEGV in log: 0 ``` 4022 s is wall time inside the slot while compute01 ran at load 14–25 with five other gates. It is not an idle-machine number. **aarch64 full package build, green** (dev-02): ``` nix build rc=0 wall=903s out=/nix/store/wr0dpxiqaprwbbka2k71rgqi6l7ynyi6-infraternal-0.1.0 sha256sums rc=0 (552 entries OK); --version -> infraternal 0.1.0 (stable) ``` **Gate:** `verify fast` ran at 3575f6b9d (dev-02, origin/dev 5eb968744 merged): VERIFY_EXIT=1. All Rust rows PASS (Nix release inputs PASS, static builds PASS, privileged integration SKIP). The only failure is the TS test `a closing notice cannot hide its replacement`, which also fails 5/5 on unmodified origin/dev (comment above). Since that run, head has changed only nix/package.nix (fc4c52e78). The new row passes at fc4c52e78 in 19.2 s on compute01; the zig input drv was rebuilt on that run. **Gate-gap cost and proposal:** - New `verify fast`/`iterate` row `Nix release inputs`: 0.25 s with a warm eval cache, 3.6 s without, 19 s when the patched Zig must rebuild. Red on origin/dev with the exact deploy error, green here. - A full `nix build` per merge costs 903 s (aarch64) to ~67 min (x86 under load), because it recompiles both static images a second time. Too slow for every merge. Proposed for merge.sh (orchestrator-owned, not edited): run `nix build --no-link .#packages.x86_64-linux.default` (timeout 2400) only when the merged diff touches `nix/ flake.nix flake.lock rust-toolchain.toml Cargo.lock apps/tracer/bun.lock scripts/{zig.sha256,libghostty-vt.sha256,auth-toolchain,build-vt,patch-libghostty-vt.py,static-cargo,package-release} scripts/release/ crates/infraternal-shim/build.rs`. 2c7135211 touched three of these. Also run it nightly on dev. Disk: dev-02 target deleted, gc'd; dev-02 at 88G free. compute01 has no j442 target (logs only).

No labels

launch-qa

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kayg/infraternal#476

No description provided.

Rows
Columns