kayg/infraternal
Watch
1
0
Fork
You've already forked infraternal
 0
Code Issues 356 Pull requests Projects Releases Packages Wiki Activity Actions

J324: profile and reduce GotaTun shared packet pool memory #407

New issue
Open
opened 2026-11-06 18:31:14 +00:00 by kayg · 22 comments
kayg commented 2026-11-06 18:31:14 +00:00
Owner
Copy link

Implementation job for #260, D348. Start from j88-linux-offloads; profile 1/100/1000 peers idle and 8-flow load, attribute heap/pools and retained RSS, reduce endpoint-shared packet storage and lazy queue retention with correctness tests first. Coordinate J316/J225; no infraternal dependency pin. Plan: inspect and coordinate (30m), reproduce/profile baseline (60m), one focused tested memory cut (90m), paired D348 measurements and x86/aarch64 tests (90m). Estimate 4.5h, subject to shared build/perf slots. Fork branch j324-mem-pools; base ebeb8f0071c4103168b570407f9dc555562f4b79. Push atomic passing commits to both fork remotes.

Implementation job for #260, D348. Start from j88-linux-offloads; profile 1/100/1000 peers idle and 8-flow load, attribute heap/pools and retained RSS, reduce endpoint-shared packet storage and lazy queue retention with correctness tests first. Coordinate J316/J225; no infraternal dependency pin. Plan: inspect and coordinate (30m), reproduce/profile baseline (60m), one focused tested memory cut (90m), paired D348 measurements and x86/aarch64 tests (90m). Estimate 4.5h, subject to shared build/perf slots. Fork branch j324-mem-pools; base ebeb8f0071c4103168b570407f9dc555562f4b79. Push atomic passing commits to both fork remotes.
kayg commented 2026-11-06 18:31:46 +00:00
Author
Owner
Copy link

start: branch j324-mem-pools, base ebeb8f0071c4103168b570407f9dc555562f4b79. brief, memory gotchas, D348 and J318 report read. coordination on #260. plan and 4.5h estimate in issue body. no allocator replacement.

start: branch j324-mem-pools, base ebeb8f0071c4103168b570407f9dc555562f4b79. brief, memory gotchas, D348 and J318 report read. coordination on #260. plan and 4.5h estimate in issue body. no allocator replacement.
kayg commented 2026-11-06 18:39:42 +00:00
Author
Owner
Copy link

j324 baseline finding: current ebeb8f0 already uses endpoint-shared lazy pool and lazy session rings/peer queues. D348-pinned memory diagnostic at 1/100/1000 configured peers per endpoint (one active established peer; others inert/unhandshaken) measured combined two-endpoint idle RSS 9432/12020/16344 KiB; loaded RSS at 3 seconds of 8-flow traffic 30120/33240/36940 KiB. This is not comparable with old R4b unpinned numbers or a calibrated acceptance claim. Artifact: perf VM /home/claude/j324/evidence/base/base/mesh-gotatun, console /home/claude/j324/evidence/base-console.log. Run exit 0, raw cpuset audits and machine diffs retained. Next: same-way competitors and jemalloc bin attribution; correctness-first small-buffer and released peer-queue-storage experiments. compute01 baseline build remains queued (position 17 at enqueue).

j324 baseline finding: current ebeb8f0 already uses endpoint-shared lazy pool and lazy session rings/peer queues. D348-pinned memory diagnostic at 1/100/1000 configured peers per endpoint (one active established peer; others inert/unhandshaken) measured combined two-endpoint idle RSS 9432/12020/16344 KiB; loaded RSS at 3 seconds of 8-flow traffic 30120/33240/36940 KiB. This is not comparable with old R4b unpinned numbers or a calibrated acceptance claim. Artifact: perf VM /home/claude/j324/evidence/base/base/mesh-gotatun, console /home/claude/j324/evidence/base-console.log. Run exit 0, raw cpuset audits and machine diffs retained. Next: same-way competitors and jemalloc bin attribution; correctness-first small-buffer and released peer-queue-storage experiments. compute01 baseline build remains queued (position 17 at enqueue).
kayg commented 2026-11-06 18:47:06 +00:00
Author
Owner
Copy link

progress checkpoint: report-only commit 3bdbeb2 pushed to both fork remotes on j324-mem-pools. Baseline/competitor scale overlays and exact artifact paths are in j324-memory/REPORT.md. All three scale cases completed exit 0 with empty case machine diffs; no shipping code changed yet, gate not applicable to report slice. Correctness RED tests for small packet backing and queue storage are prepared but remain uncommitted until executed. compute01 build queued behind other jobs; one target will be /home/claude/worktrees/gotatun-j324/target. Three short, separate D348 control/control calibration windows are running now; no candidate acceptance before spread passes.

progress checkpoint: report-only commit 3bdbeb2 pushed to both fork remotes on j324-mem-pools. Baseline/competitor scale overlays and exact artifact paths are in j324-memory/REPORT.md. All three scale cases completed exit 0 with empty case machine diffs; no shipping code changed yet, gate not applicable to report slice. Correctness RED tests for small packet backing and queue storage are prepared but remain uncommitted until executed. compute01 build queued behind other jobs; one target will be /home/claude/worktrees/gotatun-j324/target. Three short, separate D348 control/control calibration windows are running now; no candidate acceptance before spread passes.
kayg commented 2026-11-06 18:49:01 +00:00
Author
Owner
Copy link

j324 D348 acceptance blocker: three unchanged ebeb8f0 control windows failed calibration. 1-flow loss-aware p99 spread 11.575% > 3.2%; 8-flow throughput spread 2.186% > 0.8%. Other gated spreads: 1-flow throughput 1.575% <=1.6%, 8-flow p99 4.688% <=5.9%. Artifacts: perf VM /home/claude/j324/evidence/calibration/spread.json and calibration-control-{a,b,c}. First control also had machine diff +claude netbird (all CPU audits still recorded); later two machine diffs empty. No candidate acceptance enabled. Recommendation: continue memory/correctness diagnostics, retain any candidate as unqualified on the fork branch, and do not integrate/pin until a clean calibrated hold is available. I will not weaken limits or describe diagnostic performance as proof. Question for orchestrator: investigate background process/spread or explicitly authorize diagnostic-only candidate timing; no change to D348 requested.

j324 D348 acceptance blocker: three unchanged ebeb8f0 control windows failed calibration. 1-flow loss-aware p99 spread 11.575% > 3.2%; 8-flow throughput spread 2.186% > 0.8%. Other gated spreads: 1-flow throughput 1.575% <=1.6%, 8-flow p99 4.688% <=5.9%. Artifacts: perf VM /home/claude/j324/evidence/calibration/spread.json and calibration-control-{a,b,c}. First control also had machine diff +claude netbird (all CPU audits still recorded); later two machine diffs empty. No candidate acceptance enabled. Recommendation: continue memory/correctness diagnostics, retain any candidate as unqualified on the fork branch, and do not integrate/pin until a clean calibrated hold is available. I will not weaken limits or describe diagnostic performance as proof. Question for orchestrator: investigate background process/spread or explicitly authorize diagnostic-only candidate timing; no change to D348 requested.
kayg commented 2026-11-06 19:13:00 +00:00
Author
Owner
Copy link

checkpoint: report commit 81411ad pushed to both fork remotes. Fresh paired baseline (rustc 1.95.0, binary SHA256 84e342a6...) measured idle/load RSS MiB at 1/100/1000: 9.125/30.043, 11.641/31.871, 15.914/35.156 (both endpoints combined). Artifact: perf VM /home/claude/j324/evidence/base-built/base/mesh-gotatun. Fresh three-control cases all exited 0 with clean audits/restoration, but spread failed: throughput 1.995%/0.327%, p99 17.625%/11.290%. No acceptance claim. Source regressions now legitimately RED: both queues retain capacity 256 after drain/reset, and MTU packets retain >2048 bytes; logs compute01 /home/claude/j324-evidence/valid-red-{handshake_queue,mtu_datagrams_use_small_pool}.log. Candidate patch attached for crash recovery; production commits wait for GREEN. x86 batch queued pid 1700654 (/home/claude/j324-evidence/queue-green.log), one arm batch queued pid 475303 (/home/claude/j324-evidence/arm-checks.log). Each will verify queue-only then full candidate. x86 also builds baseline/candidate jemalloc stats probes. One target per host: /home/claude/worktrees/gotatun-j324/target; arm CARGO_INCREMENTAL=0 and exit cleanup configured.

checkpoint: report commit 81411ad pushed to both fork remotes. Fresh paired baseline (rustc 1.95.0, binary SHA256 84e342a6...) measured idle/load RSS MiB at 1/100/1000: 9.125/30.043, 11.641/31.871, 15.914/35.156 (both endpoints combined). Artifact: perf VM /home/claude/j324/evidence/base-built/base/mesh-gotatun. Fresh three-control cases all exited 0 with clean audits/restoration, but spread failed: throughput 1.995%/0.327%, p99 17.625%/11.290%. No acceptance claim. Source regressions now legitimately RED: both queues retain capacity 256 after drain/reset, and MTU packets retain >2048 bytes; logs compute01 /home/claude/j324-evidence/valid-red-{handshake_queue,mtu_datagrams_use_small_pool}.log. Candidate patch attached for crash recovery; production commits wait for GREEN. x86 batch queued pid 1700654 (/home/claude/j324-evidence/queue-green.log), one arm batch queued pid 475303 (/home/claude/j324-evidence/arm-checks.log). Each will verify queue-only then full candidate. x86 also builds baseline/candidate jemalloc stats probes. One target per host: /home/claude/worktrees/gotatun-j324/target; arm CARGO_INCREMENTAL=0 and exit cleanup configured.
j324-candidate.patch
19 KiB
kayg commented 2026-11-06 19:24:57 +00:00
Author
Owner
Copy link

j324 checkpoint: the single aarch64 batch completed with CARGO_INCREMENTAL=0. Verbatim markers:

J324_ARM_QUEUE_CHECKS_EXIT=0
J324_ARM_CHECKS_EXIT=0
J324_ARM_EXIT=0
J324_ARM_TARGET_REMOVED

Ring library tests: 142 passed, 0 failed, 7 ignored; CLI tests and all-target ring clippy passed. Log: dev-02 /home/claude/j324-evidence/arm-checks.log, copied to control scratch j324/evidence/arm-checks.log. Own target removed by batch trap. x86 PID 1700654 remains queued (seven jobs ahead including active slots); no new arm batch. D348 calibration remains failed as previously reported; production changes remain unqualified for integration.

j324 checkpoint: the single aarch64 batch completed with CARGO_INCREMENTAL=0. Verbatim markers: ``` J324_ARM_QUEUE_CHECKS_EXIT=0 J324_ARM_CHECKS_EXIT=0 J324_ARM_EXIT=0 J324_ARM_TARGET_REMOVED ``` Ring library tests: 142 passed, 0 failed, 7 ignored; CLI tests and all-target ring clippy passed. Log: dev-02 `/home/claude/j324-evidence/arm-checks.log`, copied to control scratch `j324/evidence/arm-checks.log`. Own target removed by batch trap. x86 PID 1700654 remains queued (seven jobs ahead including active slots); no new arm batch. D348 calibration remains failed as previously reported; production changes remain unqualified for integration.
kayg commented 2026-11-06 19:30:43 +00:00
Author
Owner
Copy link

j324 atomic implementation checkpoint, pushed to both fork remotes: queue release 96239d4; smaller shared pool buffers 77fcd66aab2ce3697802d833d47f6912c9d436be. Both have baseline failing-test evidence and a passing single aarch64 batch as reported above. x86 remains queued; candidate RSS/heap/disk evidence is pending. D348 calibration remains failed. No infraternal pin change; no TUN writer/coalescing or crypto scheduler changes. Production patch is no longer only a scratch attachment: https://github.com/kaygdotorg/gotatun/compare/ebeb8f0...j324-mem-pools

j324 atomic implementation checkpoint, pushed to both fork remotes: queue release `96239d4`; smaller shared pool buffers `77fcd66aab2ce3697802d833d47f6912c9d436be`. Both have baseline failing-test evidence and a passing single aarch64 batch as reported above. x86 remains queued; candidate RSS/heap/disk evidence is pending. D348 calibration remains failed. No infraternal pin change; no TUN writer/coalescing or crypto scheduler changes. Production patch is no longer only a scratch attachment: https://github.com/kaygdotorg/gotatun/compare/ebeb8f0...j324-mem-pools
kayg commented 2026-11-06 19:35:12 +00:00
Author
Owner
Copy link

x86 launch failure, not a test result: initial queued batch invoked nix develop from /home/claude and exited before checks because there was no flake.nix. Preserved log /home/claude/j324-evidence/queue-green-launch-failed.log. Corrected launcher now changes to /home/claude/worktrees/gotatun-j324 before entering the build slot and nix shell. Requeued PID 2328810, same batch/log /home/claude/j324-evidence/queue-green.log. No extra aarch64 run. Candidate and aarch64 report checkpoint pushed to both fork remotes.

x86 launch failure, not a test result: initial queued batch invoked `nix develop` from /home/claude and exited before checks because there was no flake.nix. Preserved log `/home/claude/j324-evidence/queue-green-launch-failed.log`. Corrected launcher now changes to `/home/claude/worktrees/gotatun-j324` before entering the build slot and nix shell. Requeued PID 2328810, same batch/log `/home/claude/j324-evidence/queue-green.log`. No extra aarch64 run. Candidate and aarch64 report checkpoint pushed to both fork remotes.
kayg commented 2026-11-06 19:44:04 +00:00
Author
Owner
Copy link

j324 kernel memory finding: same-way 1/100/1000 scale run completed exit 0 with host placement restored. Dedicated wg_peer objects are 1552 bytes; host-wide reserved slab pages correspond to 0.5625 / 0.59375 / 3.28125 MiB at the three scales (unchanged idle/load/recovery). Counts include existing/cached capacity and do not cover sessions, allowed-IP nodes, sockets or packet buffers, so these are a component, not total kernel memory or a GotaTun kernel-memory win. Raw /home/claude/j324/evidence/kernel-slab/kernel/mesh-kernel-plain/slab-*.txt; console /home/claude/j324/evidence/kernel-slab-console.log. Original daemon-RSS kernel rows remain N/A. x86 candidate batch remains queued.

j324 kernel memory finding: same-way 1/100/1000 scale run completed exit 0 with host placement restored. Dedicated `wg_peer` objects are 1552 bytes; host-wide reserved slab pages correspond to 0.5625 / 0.59375 / 3.28125 MiB at the three scales (unchanged idle/load/recovery). Counts include existing/cached capacity and do not cover sessions, allowed-IP nodes, sockets or packet buffers, so these are a component, not total kernel memory or a GotaTun kernel-memory win. Raw `/home/claude/j324/evidence/kernel-slab/kernel/mesh-kernel-plain/slab-*.txt`; console `/home/claude/j324/evidence/kernel-slab-console.log`. Original daemon-RSS kernel rows remain N/A. x86 candidate batch remains queued.
kayg commented 2026-11-06 19:55:46 +00:00
Author
Owner
Copy link

j324 disk finding: matched baseline executable 2,769,040 bytes; its four resolved runtime files add 3,885,024 bytes (6,654,064 before launcher). Harness wireguard-go executable: 3,408,137 bytes. Kernel module plus recursive module dependency files: 149,452 compressed bytes, excluding OS/control tools. Units are intentionally explicit, not equivalent deployment closures; no disk win established. Forwarding /proc/PID/io write_bytes deltas remain zero in baseline userspace scale cases. Candidate executable delta awaits x86. Evidence: perf /home/claude/j324/evidence/disk-inventory.json, disk-inventory.log, disk-closure.log; inventory scripts run under perf-run, both exit 0.

j324 disk finding: matched baseline executable 2,769,040 bytes; its four resolved runtime files add 3,885,024 bytes (6,654,064 before launcher). Harness wireguard-go executable: 3,408,137 bytes. Kernel module plus recursive module dependency files: 149,452 compressed bytes, excluding OS/control tools. Units are intentionally explicit, not equivalent deployment closures; no disk win established. Forwarding /proc/PID/io write_bytes deltas remain zero in baseline userspace scale cases. Candidate executable delta awaits x86. Evidence: perf `/home/claude/j324/evidence/disk-inventory.json`, `disk-inventory.log`, `disk-closure.log`; inventory scripts run under perf-run, both exit 0.
kayg commented 2026-11-06 20:25:39 +00:00
Author
Owner
Copy link

j324 x86 evidence: queue-only tests passed (J324_QUEUE_CHECKS_EXIT=0). The complete candidate all-feature suite then reported 152 passed, 3 failed, 7 ignored. Failures are device::tests::{number_of_packets,one_handshake,one_keepalive}, on existing one-second eavesdrop timeout deadlines in device/tests.rs:686. No production attribution yet. Failed artifact compute01 /home/claude/j324-evidence/queue-green.log, preserved control copy /home/kayg/.cache/codex-scratch/j324/evidence/x86-first-checks.log. Follow-up PID 3345273 queued in the correct worktree: frozen-baseline parallel control, baseline serial, candidate serial full tests, ring suites, clippy and matched release/heap builds. All existing deadlines and assertions unchanged; no second arm batch or production code change. Follow-up log /home/claude/j324-evidence/x86-followup.log. Baseline jemalloc snapshots completed exit 0; /home/claude/j324/evidence/heap-base, parsed /home/claude/j324/evidence/heap-summary.json; D348 calibration remains failed.

j324 x86 evidence: queue-only tests passed (`J324_QUEUE_CHECKS_EXIT=0`). The complete candidate all-feature suite then reported 152 passed, 3 failed, 7 ignored. Failures are device::tests::{number_of_packets,one_handshake,one_keepalive}, on existing one-second `eavesdrop timeout` deadlines in device/tests.rs:686. No production attribution yet. Failed artifact compute01 `/home/claude/j324-evidence/queue-green.log`, preserved control copy `/home/kayg/.cache/codex-scratch/j324/evidence/x86-first-checks.log`. Follow-up PID 3345273 queued in the correct worktree: frozen-baseline parallel control, baseline serial, candidate serial full tests, ring suites, clippy and matched release/heap builds. All existing deadlines and assertions unchanged; no second arm batch or production code change. Follow-up log `/home/claude/j324-evidence/x86-followup.log`. Baseline jemalloc snapshots completed exit 0; `/home/claude/j324/evidence/heap-base`, parsed `/home/claude/j324/evidence/heap-summary.json`; D348 calibration remains failed.
kayg commented 2026-11-06 20:28:49 +00:00
Author
Owner
Copy link

j324 baseline heap attribution: stats enabled, 8 arenas, dirty_decay_ms=10000, muzzy_decay_ms=0, background_thread=false, retain=true. At 1,000-peer 8-flow load, 4 KiB bin allocations total 14,069,760 bytes across endpoints; after recovery 167,936. Fixed x86 ring sizes: Tunn 1552, Handshake 576, Timers 568, PeerState 1616, mutex-wrapped PeerState 1664, lazy 8-slot session ring 17408, Packet 72 bytes. One 28 MiB jemalloc extent per endpoint backs the 25 MiB GRO scratch reservation; allocator resident estimates overcount OS RSS because existing code directly discards scratch pages. Optional jemalloc OS RSS recovered to 39.809 MiB after first load versus shipping glibc 11.242 MiB, so no allocator switch is justified. Source/bin attribution is not a stack trace; no CPU/timing acceptance. Raw perf /home/claude/j324/evidence/heap-base/heap-base/mesh-gotatun, parsed heap-summary.json, heap-base-table.md, console exit 0. Candidate profile waits for x86 follow-up; J225 existing timeout evidence is linked in report, own direct-base reproduction pending.

j324 baseline heap attribution: stats enabled, 8 arenas, dirty_decay_ms=10000, muzzy_decay_ms=0, background_thread=false, retain=true. At 1,000-peer 8-flow load, 4 KiB bin allocations total 14,069,760 bytes across endpoints; after recovery 167,936. Fixed x86 ring sizes: Tunn 1552, Handshake 576, Timers 568, PeerState 1616, mutex-wrapped PeerState 1664, lazy 8-slot session ring 17408, Packet 72 bytes. One 28 MiB jemalloc extent per endpoint backs the 25 MiB GRO scratch reservation; allocator resident estimates overcount OS RSS because existing code directly discards scratch pages. Optional jemalloc OS RSS recovered to 39.809 MiB after first load versus shipping glibc 11.242 MiB, so no allocator switch is justified. Source/bin attribution is not a stack trace; no CPU/timing acceptance. Raw perf `/home/claude/j324/evidence/heap-base/heap-base/mesh-gotatun`, parsed `heap-summary.json`, `heap-base-table.md`, console exit 0. Candidate profile waits for x86 follow-up; J225 existing timeout evidence is linked in report, own direct-base reproduction pending.
kayg commented 2026-11-06 20:33:02 +00:00
Author
Owner
Copy link

j324 targeted timeout evidence: isolated cargo test --locked -p gotatun --all-features --lib device::tests::one_handshake -- --exact --test-threads=1 passes on the unchanged candidate with the original deadline: 1 passed; 0 failed; 161 filtered out; finished in 0.70s, J324_ISOLATED_EXIT=0. Artifact compute01 /home/claude/j324-evidence/x86-one-handshake-isolated.log. This is one focused test, not a replacement for the full suite or direct-base comparison. The queued follow-up retains all controls and failed results. Baseline heap report and source are now pushed on both fork remotes at a63f39c.

j324 targeted timeout evidence: isolated `cargo test --locked -p gotatun --all-features --lib device::tests::one_handshake -- --exact --test-threads=1` passes on the unchanged candidate with the original deadline: `1 passed; 0 failed; 161 filtered out; finished in 0.70s`, `J324_ISOLATED_EXIT=0`. Artifact compute01 `/home/claude/j324-evidence/x86-one-handshake-isolated.log`. This is one focused test, not a replacement for the full suite or direct-base comparison. The queued follow-up retains all controls and failed results. Baseline heap report and source are now pushed on both fork remotes at a63f39c.
kayg commented 2026-11-06 20:38:18 +00:00
Author
Owner
Copy link

maintenance checkpoint: orchestrator froze compute01 admission at slots=0 for the announced 02:15–02:45 IST CPU expansion/restart. J324 follow-up PID 3345273 is queued, not running, and will be interrupted by maintenance. This is not a test failure. No new compute01 work will start until #402 explicitly posts “compute01 back, 5 slots”; then I will resubmit the preserved /home/claude/j324-x86-followup.sh once in the foreground. Current branch a63f39c is pushed to both remotes, production source remains 77fcd66. Completed arm tests and baseline/kernel/disk/heap artifacts are preserved. The prior genuine all-feature timeout result remains separately recorded and is not being relabeled as maintenance.

maintenance checkpoint: orchestrator froze compute01 admission at slots=0 for the announced 02:15–02:45 IST CPU expansion/restart. J324 follow-up PID 3345273 is queued, not running, and will be interrupted by maintenance. This is not a test failure. No new compute01 work will start until #402 explicitly posts “compute01 back, 5 slots”; then I will resubmit the preserved `/home/claude/j324-x86-followup.sh` once in the foreground. Current branch a63f39c is pushed to both remotes, production source remains 77fcd66. Completed arm tests and baseline/kernel/disk/heap artifacts are preserved. The prior genuine all-feature timeout result remains separately recorded and is not being relabeled as maintenance.
kayg commented 2026-11-06 21:15:30 +00:00
Author
Owner
Copy link

maintenance wait checkpoint: no “compute01 back, 5 slots” comment seen on #402 yet; no new compute01 work submitted. Branch a63f39c remains pushed to both fork remotes. Latest follow-up script is preserved on control /home/kayg/.cache/codex-scratch/j324/x86-followup.sh; upload this version after the explicit release, because the frozen remote waiter has the earlier script. It adds paired baseline/candidate default-concurrency controls as well as serial/standalone checks, preserves every exit status, and builds diagnostics even when a known test deadline fails. Resubmit once in the foreground after restart; keep the old queued /home/claude/j324-evidence/x86-followup.log as maintenance evidence. Production patch remains /home/kayg/.cache/codex-scratch/j324/candidate.patch; own x86 target is /home/claude/worktrees/gotatun-j324/target. Arm target removed. No perf job remains active. Outstanding: direct-base timeout comparison, candidate release/heap binaries, candidate RSS, counterbalanced timing diagnostics, final report/table. D348 calibration remains failed.

maintenance wait checkpoint: no “compute01 back, 5 slots” comment seen on #402 yet; no new compute01 work submitted. Branch a63f39c remains pushed to both fork remotes. Latest follow-up script is preserved on control `/home/kayg/.cache/codex-scratch/j324/x86-followup.sh`; upload this version after the explicit release, because the frozen remote waiter has the earlier script. It adds paired baseline/candidate default-concurrency controls as well as serial/standalone checks, preserves every exit status, and builds diagnostics even when a known test deadline fails. Resubmit once in the foreground after restart; keep the old queued `/home/claude/j324-evidence/x86-followup.log` as maintenance evidence. Production patch remains `/home/kayg/.cache/codex-scratch/j324/candidate.patch`; own x86 target is `/home/claude/worktrees/gotatun-j324/target`. Arm target removed. No perf job remains active. Outstanding: direct-base timeout comparison, candidate release/heap binaries, candidate RSS, counterbalanced timing diagnostics, final report/table. D348 calibration remains failed.
kayg commented 2026-11-06 21:17:36 +00:00
Author
Owner
Copy link

restart receipt: #402 comment 83012 confirms compute01 back, 5 slots at 02:45 IST, now 16 vCPUs. Earlier queued J324 PID 3345273 was cut by the 02:43 IST restart; it was not a running test or test failure. Old PID is absent, reboot confirmed. Uploaded latest control follow-up script and resubmitted exactly once in the foreground after that notice. Durable log /home/claude/j324-evidence/x86-post-restart.log; old queue log remains x86-followup.log. Builds remain j4, same target and source patch. Both default-concurrency baseline/candidate controls now run on the same post-restart host; prior pre-restart failures remain preserved separately.

restart receipt: #402 comment 83012 confirms compute01 back, 5 slots at 02:45 IST, now 16 vCPUs. Earlier queued J324 PID 3345273 was cut by the 02:43 IST restart; it was not a running test or test failure. Old PID is absent, reboot confirmed. Uploaded latest control follow-up script and resubmitted exactly once in the foreground after that notice. Durable log `/home/claude/j324-evidence/x86-post-restart.log`; old queue log remains `x86-followup.log`. Builds remain j4, same target and source patch. Both default-concurrency baseline/candidate controls now run on the same post-restart host; prior pre-restart failures remain preserved separately.
kayg commented 2026-11-06 21:33:36 +00:00
Author
Owner
Copy link

important correction: post-restart logs labeled x86-baseline-{parallel,serial,focused} are INVALID baseline controls. Archive extraction restored old mtimes and Cargo reused candidate test binaries: the “baseline” parallel log contains J324-only test names. No direct-base attribution is claimed from them. The seven archive files touched by production/instrumentation match ebeb8f0 Git blobs; the problem is build invalidation, not archive contents. The attempted next-stage source-refresh hook missed the already-finished batch and did not execute; original instrumenter restored. A separate explicit source verification is now queued in the foreground, /home/claude/j324-evidence/x86-source-refresh.log, using touch after each extraction and source hashes before full baseline/candidate parallel+serial controls. This is a new evidence correction, not a duplicate maintenance resubmission. Candidate ring tests passed 142/0/7, CLI tests and all-feature clippy passed; candidate serial all-feature suite passed, parallel suite retains timeouts, batch exit101. Exact log /home/claude/j324-evidence/x86-post-restart.log. Shipping binary hash 0c8224ed8854566634ad4a0b5171aacedd45236c0c99d7b7b0c4191af590332c; stripped base/candidate 2307080/2313456 bytes (+6376). Correct candidate binaries are ready for unqualified memory diagnostics; no D348 acceptance.

important correction: post-restart logs labeled x86-baseline-{parallel,serial,focused} are INVALID baseline controls. Archive extraction restored old mtimes and Cargo reused candidate test binaries: the “baseline” parallel log contains J324-only test names. No direct-base attribution is claimed from them. The seven archive files touched by production/instrumentation match ebeb8f0 Git blobs; the problem is build invalidation, not archive contents. The attempted next-stage source-refresh hook missed the already-finished batch and did not execute; original instrumenter restored. A separate explicit source verification is now queued in the foreground, `/home/claude/j324-evidence/x86-source-refresh.log`, using `touch` after each extraction and source hashes before full baseline/candidate parallel+serial controls. This is a new evidence correction, not a duplicate maintenance resubmission. Candidate ring tests passed 142/0/7, CLI tests and all-feature clippy passed; candidate serial all-feature suite passed, parallel suite retains timeouts, batch exit101. Exact log `/home/claude/j324-evidence/x86-post-restart.log`. Shipping binary hash 0c8224ed8854566634ad4a0b5171aacedd45236c0c99d7b7b0c4191af590332c; stripped base/candidate 2307080/2313456 bytes (+6376). Correct candidate binaries are ready for unqualified memory diagnostics; no D348 acceptance.
kayg commented 2026-11-06 21:38:19 +00:00
Author
Owner
Copy link

j324 shipping-system-allocator memory result (UNQUALIFIED for D348 timing): both endpoints combined; peers on each endpoint; one active established peer carries eight flows, all remaining peers inert/unhandshaken. Scale grows sequentially, later idle includes earlier load. MTU1420; idle sampled5s; load RSS at3s into10s; recovered12s later.

engine peers per endpoint idle RSS MiB loaded RSS MiB peak RSS MiB recovered RSS MiB idle CPU % disk write bytes during load
base-built 1 9.125 30.043 33.086 11.242 0.000 0
base-built 100 11.641 31.871 32.105 11.805 0.000 0
base-built 1000 15.914 35.156 36.477 15.688 0.400 0
candidate 1 9.094 25.957 27.477 11.113 0.000 0
candidate 100 11.520 25.578 27.492 11.754 0.000 0
candidate 1000 15.621 31.754 32.230 15.770 0.000 0
wireguard-go 1 29.309 642.695 929.754 929.770 0.000 0
wireguard-go 100 933.910 938.848 1078.598 1078.656 0.000 0
wireguard-go 1000 1111.145 1115.727 1201.840 1203.863 0.200 0

Candidate case and post-scope restoration both exit0: perf /home/claude/j324/evidence/candidate/candidate/mesh-gotatun, candidate-console.log, candidate-restore-console.log, candidate/post-scope-restored.txt; summary memory-summary.md. At1,000 peers loaded RSS falls3.402MiB (9.7%); recovered RSS rises0.082MiB, so no recovered-memory or fixed-peer-state win is claimed. Zero CPU ticks over5s do not prove zero CPU/wakeups; no idle-CPU optimization claim. Kernel daemon RSS is N/A; the separately reported wg_peer slab is only one kernel component. Forwarding write_bytes deltas are0 for these userspace runs. Stripped base/candidate2307080/2313456bytes (+6376); unstripped2769040/2776264 (+7224), so no disk win. Heap candidate/timing diagnostics are still running in separate short perf windows. Exact-base test comparison awaits the timestamp-refreshed source verification; invalid baseline test logs remain excluded. No infraternal pin change.

j324 shipping-system-allocator memory result (UNQUALIFIED for D348 timing): both endpoints combined; peers on each endpoint; one active established peer carries eight flows, all remaining peers inert/unhandshaken. Scale grows sequentially, later idle includes earlier load. MTU1420; idle sampled5s; load RSS at3s into10s; recovered12s later. | engine | peers per endpoint | idle RSS MiB | loaded RSS MiB | peak RSS MiB | recovered RSS MiB | idle CPU % | disk write bytes during load | |---|---:|---:|---:|---:|---:|---:|---:| | base-built | 1 | 9.125 | 30.043 | 33.086 | 11.242 | 0.000 | 0 | | base-built | 100 | 11.641 | 31.871 | 32.105 | 11.805 | 0.000 | 0 | | base-built | 1000 | 15.914 | 35.156 | 36.477 | 15.688 | 0.400 | 0 | | candidate | 1 | 9.094 | 25.957 | 27.477 | 11.113 | 0.000 | 0 | | candidate | 100 | 11.520 | 25.578 | 27.492 | 11.754 | 0.000 | 0 | | candidate | 1000 | 15.621 | 31.754 | 32.230 | 15.770 | 0.000 | 0 | | wireguard-go | 1 | 29.309 | 642.695 | 929.754 | 929.770 | 0.000 | 0 | | wireguard-go | 100 | 933.910 | 938.848 | 1078.598 | 1078.656 | 0.000 | 0 | | wireguard-go | 1000 | 1111.145 | 1115.727 | 1201.840 | 1203.863 | 0.200 | 0 | Candidate case and post-scope restoration both exit0: perf `/home/claude/j324/evidence/candidate/candidate/mesh-gotatun`, `candidate-console.log`, `candidate-restore-console.log`, `candidate/post-scope-restored.txt`; summary `memory-summary.md`. At1,000 peers loaded RSS falls3.402MiB (9.7%); recovered RSS rises0.082MiB, so no recovered-memory or fixed-peer-state win is claimed. Zero CPU ticks over5s do not prove zero CPU/wakeups; no idle-CPU optimization claim. Kernel daemon RSS is N/A; the separately reported wg_peer slab is only one kernel component. Forwarding write_bytes deltas are0 for these userspace runs. Stripped base/candidate2307080/2313456bytes (+6376); unstripped2769040/2776264 (+7224), so no disk win. Heap candidate/timing diagnostics are still running in separate short perf windows. Exact-base test comparison awaits the timestamp-refreshed source verification; invalid baseline test logs remain excluded. No infraternal pin change.
kayg commented 2026-11-06 21:55:34 +00:00
Author
Owner
Copy link

j324 diagnostic checkpoint at 5ea421b, pushed to both forks. Completed candidate heap profile: at 1,000 peers per endpoint, both endpoints combined, jemalloc allocated load drops 77.405 -> 71.477 MiB and packet 2+4 KiB bins drop 13.461 -> 6.270 MiB. Shipping RSS load drops 35.156 -> 31.754 MiB. Recovery is essentially unchanged. Raw perf heap-candidate/heap-candidate/mesh-gotatun; heap-comparison-table.md; both case and restoration exit0.

Counterbalanced short LAN diagnostics are complete, all cases/restoration exit0, but remain UNQUALIFIED because three-control calibration failed. At8 flows throughput control-before/candidate/control-after =2.175/2.185/2.175 Gbit/s, p99=13.6/13.0/12.9 ms; whole-machine CPU s/Gbit=1.440/1.483/1.443. This does not establish CPU or p99 non-regression. Stripped candidate grows6376bytes. No disk/CPU/kernel-memory win is claimed. Artifacts perf /home/claude/j324/evidence/candidate-timing-summary.md, scale-cpu-summary.md, disk-final.log. Report j324-memory/REPORT.md preserves all results and limitations.

Explicit x86 source-refresh controls remain foreground-queued on compute01; latest log /home/claude/j324-evidence/x86-source-refresh.log, queue position19, all5 slots busy, approximately25min elapsed. No new compute or perf jobs submitted. The earlier maintenance cut and invalid baseline logs remain documented separately. J316/J225 coordination: pool changes only affect packet pool copying and handshake deque release; no crypto completion or send_many contract change. Production source remains77fcd66. No infraternal pin change. D348 acceptance remains blocked pending passing calibration; this is an experimental memory candidate.

j324 diagnostic checkpoint at 5ea421b, pushed to both forks. Completed candidate heap profile: at 1,000 peers per endpoint, both endpoints combined, jemalloc allocated load drops 77.405 -> 71.477 MiB and packet 2+4 KiB bins drop 13.461 -> 6.270 MiB. Shipping RSS load drops 35.156 -> 31.754 MiB. Recovery is essentially unchanged. Raw perf heap-candidate/heap-candidate/mesh-gotatun; heap-comparison-table.md; both case and restoration exit0. Counterbalanced short LAN diagnostics are complete, all cases/restoration exit0, but remain UNQUALIFIED because three-control calibration failed. At8 flows throughput control-before/candidate/control-after =2.175/2.185/2.175 Gbit/s, p99=13.6/13.0/12.9 ms; whole-machine CPU s/Gbit=1.440/1.483/1.443. This does not establish CPU or p99 non-regression. Stripped candidate grows6376bytes. No disk/CPU/kernel-memory win is claimed. Artifacts perf /home/claude/j324/evidence/candidate-timing-summary.md, scale-cpu-summary.md, disk-final.log. Report j324-memory/REPORT.md preserves all results and limitations. Explicit x86 source-refresh controls remain foreground-queued on compute01; latest log /home/claude/j324-evidence/x86-source-refresh.log, queue position19, all5 slots busy, approximately25min elapsed. No new compute or perf jobs submitted. The earlier maintenance cut and invalid baseline logs remain documented separately. J316/J225 coordination: pool changes only affect packet pool copying and handshake deque release; no crypto completion or send_many contract change. Production source remains77fcd66. No infraternal pin change. D348 acceptance remains blocked pending passing calibration; this is an experimental memory candidate.
kayg commented 2026-11-06 22:12:59 +00:00
Author
Owner
Copy link

j324 cost/checkpoint: report head dfa117ae7ac9b3cc052b4e1f868a22e2ae14ef84 is pushed to both fork remotes. Candidate diagnostic measures Packet=80 bytes versus baseline72; a full256-entry blocked deque therefore has20KiB descriptors versus18KiB, freed on drain/reset. PeerState/Tunn/timers are unchanged. This overhead is included in the interpretation of the lower loaded RSS and heap bins, not omitted. Evidence perf /home/claude/j324/evidence/heap-{base,candidate}/heap-{base,candidate}/mesh-gotatun/heap-idle-1-a.log, J324_TYPE lines; report j324-memory/REPORT.md.

Corrected x86 source comparison has waited about40min, currently two queued jobs ahead. The sole J324 waiter remains PID259165; artifact /home/claude/j324-evidence/x86-source-refresh.log. No duplicate check, additional arm batch, or perf job is active. D348 calibration remains failed; no integration/non-regression claim.

j324 cost/checkpoint: report head dfa117ae7ac9b3cc052b4e1f868a22e2ae14ef84 is pushed to both fork remotes. Candidate diagnostic measures Packet=80 bytes versus baseline72; a full256-entry blocked deque therefore has20KiB descriptors versus18KiB, freed on drain/reset. PeerState/Tunn/timers are unchanged. This overhead is included in the interpretation of the lower loaded RSS and heap bins, not omitted. Evidence perf /home/claude/j324/evidence/heap-{base,candidate}/heap-{base,candidate}/mesh-gotatun/heap-idle-1-a.log, J324_TYPE lines; report j324-memory/REPORT.md. Corrected x86 source comparison has waited about40min, currently two queued jobs ahead. The sole J324 waiter remains PID259165; artifact /home/claude/j324-evidence/x86-source-refresh.log. No duplicate check, additional arm batch, or perf job is active. D348 calibration remains failed; no integration/non-regression claim.
kayg commented 2026-11-06 22:37:47 +00:00
Author
Owner
Copy link

j324 foreground queue checkpoint: explicit source-refresh controls remain queued after about65min, one job ahead, all5 slots busy. Existing waiter PID259165, wrapper259160. No duplicate admission, second arm batch, or perf work. Script /home/claude/j324-correct-baseline.sh; log /home/claude/j324-evidence/x86-source-refresh.log; expected final marker J324_SOURCE_REFRESH_FINISHED=1. It will refresh source mtimes, hash the five production files, then run untouched baseline and candidate all-feature full suites with both default and serial concurrency. Existing invalid baseline logs stay excluded.

Branch dfa117ae7ac9b3cc052b4e1f868a22e2ae14ef84 is pushed to both remotes, clean control worktree /home/kayg/Developer/Worktrees/gotatun-j324. Production source77fcd66, matched candidate memory/heap/timing completed and documented. Qualified D348 non-regression remains blocked by failed calibration. After the queued comparison finishes, preserve fresh logs, update the report/table and remove own x86 /home/claude/worktrees/gotatun-j324/target. Arm target already removed. No infraternal pin change.

j324 foreground queue checkpoint: explicit source-refresh controls remain queued after about65min, one job ahead, all5 slots busy. Existing waiter PID259165, wrapper259160. No duplicate admission, second arm batch, or perf work. Script /home/claude/j324-correct-baseline.sh; log /home/claude/j324-evidence/x86-source-refresh.log; expected final marker J324_SOURCE_REFRESH_FINISHED=1. It will refresh source mtimes, hash the five production files, then run untouched baseline and candidate all-feature full suites with both default and serial concurrency. Existing invalid baseline logs stay excluded. Branch dfa117ae7ac9b3cc052b4e1f868a22e2ae14ef84 is pushed to both remotes, clean control worktree /home/kayg/Developer/Worktrees/gotatun-j324. Production source77fcd66, matched candidate memory/heap/timing completed and documented. Qualified D348 non-regression remains blocked by failed calibration. After the queued comparison finishes, preserve fresh logs, update the report/table and remove own x86 /home/claude/worktrees/gotatun-j324/target. Arm target already removed. No infraternal pin change.
kayg commented 2026-11-06 22:46:25 +00:00
Author
Owner
Copy link

J324 final experimental handoff: branch j324-mem-pools, head b36d575a20e1f8747ca9e883afdd9aab4c9cd0c5, pushed to BOTH fork remotes. No infraternal pin change. Production commits: 96239d4 frees blocked handshake deque capacity after drain/reset; 77fcd66 adds a 2KiB class to the existing shared pool under one combined retention bound, preserving full receive capacity and large/offload fallback. J316 writer/coalescer and J225 crypto ordering remain untouched. Report: https://git.kayg.org/kayg/gotatun-fork/src/branch/j324-mem-pools/j324-memory/REPORT.md

Matched shipping-system-allocator RSS, both endpoints combined, peers on each endpoint. One established peer carries8 TCP flows, remaining peers inert/unhandshaken. Scale grows sequentially, later idle includes earlier load. MTU1420, idle5s, loaded sample3s into10s load, recovery12s later. Memory diagnostics, NOT D348 timing acceptance.

engine peers/endpoint idle MiB loaded MiB recovered MiB process idle CPU %
base 1 9.125 30.043 11.242 0.0
candidate 1 9.094 25.957 11.113 0.0
base 100 11.641 31.871 11.805 0.0
candidate 100 11.520 25.578 11.754 0.0
base 1000 15.914 35.156 15.688 0.4
candidate 1000 15.621 31.754 15.770 0.0
wireguard-go 1 29.309 642.695 929.770 0.0
wireguard-go 100 933.910 938.848 1078.656 0.0
wireguard-go 1000 1111.145 1115.727 1203.863 0.2
kernel 1/100/1000 N/A N/A N/A N/A

Loaded RSS at1000 peers falls3.402MiB (9.7%). Recovery does not improve there. Kernel has no daemon RSS; dedicated wg_peer objects are1552bytes and reserved host-wide slab pages are0.5625/0.59375/3.28125MiB, only one kernel component. Idle CPU sampling does not establish a CPU/wakeup win. Forwarding write_bytes deltas are0. At1000 load diagnostic jemalloc allocated77.405->71.477MiB, 2+4KiB bins13.461->6.270MiB. Packet descriptor grows72->80bytes; full256-entry queue descriptors18->20KiB. The existing shared pool, lazy session ring, allocator and decay configuration are retained. Stripped executable grows6376bytes; no disk superiority.

Acceptance remains blocked. Three-control D348 calibration failed. Unqualified control/candidate/control8-flow throughput2.175/2.185/2.175Gbit/s, p99=13.6/13.0/12.9ms, whole-machine CPU s/Gbit=1.440/1.483/1.443. This does NOT establish throughput/p99/CPU non-regression, and there is no qualified kernel/Go CPU win. Keep this as an experimental fork candidate; do not integrate from these diagnostics.

Corrected x86 controls acquired a slot after70min and finished. All ten source hashes match Git blobs and test-name presence confirms distinct156-test base/162-test candidate binaries. Base default+serial each148 passed/1 failed/7 ignored; candidate default+serial each154 passed/1 failed/7 ignored. All four fail existing device::tests::test_indices timeout assertions; all six added tests pass. This reproduces that failure on unchanged base but does not identify its cause or attribute the earlier three-test failure. Full all-feature x86 suite remains FAILED. Earlier stale-timestamp baseline logs remain excluded. No test deadline/assertion was changed.

Verbatim corrected markers:

J324_FRESH_base_PARALLEL_EXIT=101
J324_FRESH_base_SERIAL_EXIT=101
J324_FRESH_candidate_PARALLEL_EXIT=101
J324_FRESH_candidate_SERIAL_EXIT=101
J324_SOURCE_REFRESH_FINISHED=1

The final marker means comparison finished, not tests passed.

Other preserved verification markers, verbatim:

J324_X86_RING_EXIT=0
J324_X86_CLI_EXIT=0
J324_X86_CLIPPY_EXIT=0
scripts/verify fast: unavailable (fork has no scripts/verify; infraternal is unchanged)
J324_X86_BATCH_EXIT=101
J324_ARM_QUEUE_CHECKS_EXIT=0
J324_ARM_CHECKS_EXIT=0
J324_ARM_EXIT=0

Shipping ring tests on both architectures reported142 passed/0 failed/7 ignored; UDP integration, CLI and doctests passed. Aarch64 ran as one CARGO_INCREMENTAL=0 batch. This does not erase full all-feature x86 failures. Maintenance-cut waiter and once-only post-restart resubmission are recorded above; maintenance was not a test failure.

Evidence: compute01 /home/claude/j324-evidence/x86-source-refresh.log, x86-fresh-{base,candidate}-{parallel,serial}.log, x86-post-restart.log, cleanup.log; control copies and source-hash-proof.txt under /home/kayg/.cache/codex-scratch/j324/evidence/; arm arm-checks.log there and dev-02 /home/claude/j324-evidence/arm-checks.log. Perf /home/claude/j324/evidence/: memory-summary.md, heap-comparison-table.md, candidate-timing-summary.md, scale-cpu-summary.md, disk-final.log, raw candidate/heap-candidate/kernel-slab/competitors directories, and case/restoration logs. Every completed memory/heap/timing scope and restoration check exits0. Both own build targets are removed; no J324 run remains active. Issues remain open. Remaining blockers: qualified D348 calibration/paired acceptance and unresolved existing all-feature timeout failure.

J324 final experimental handoff: branch `j324-mem-pools`, head `b36d575a20e1f8747ca9e883afdd9aab4c9cd0c5`, pushed to BOTH fork remotes. No infraternal pin change. Production commits: `96239d4` frees blocked handshake deque capacity after drain/reset; `77fcd66` adds a 2KiB class to the existing shared pool under one combined retention bound, preserving full receive capacity and large/offload fallback. J316 writer/coalescer and J225 crypto ordering remain untouched. Report: https://git.kayg.org/kayg/gotatun-fork/src/branch/j324-mem-pools/j324-memory/REPORT.md Matched shipping-system-allocator RSS, **both endpoints combined**, peers on each endpoint. One established peer carries8 TCP flows, remaining peers inert/unhandshaken. Scale grows sequentially, later idle includes earlier load. MTU1420, idle5s, loaded sample3s into10s load, recovery12s later. Memory diagnostics, NOT D348 timing acceptance. | engine | peers/endpoint | idle MiB | loaded MiB | recovered MiB | process idle CPU % | |---|---:|---:|---:|---:|---:| | base | 1 | 9.125 | 30.043 | 11.242 | 0.0 | | candidate | 1 | 9.094 | 25.957 | 11.113 | 0.0 | | base | 100 | 11.641 | 31.871 | 11.805 | 0.0 | | candidate | 100 | 11.520 | 25.578 | 11.754 | 0.0 | | base | 1000 | 15.914 | 35.156 | 15.688 | 0.4 | | candidate | 1000 | 15.621 | 31.754 | 15.770 | 0.0 | | wireguard-go | 1 | 29.309 | 642.695 | 929.770 | 0.0 | | wireguard-go | 100 | 933.910 | 938.848 | 1078.656 | 0.0 | | wireguard-go | 1000 | 1111.145 | 1115.727 | 1203.863 | 0.2 | | kernel | 1/100/1000 | N/A | N/A | N/A | N/A | Loaded RSS at1000 peers falls3.402MiB (9.7%). Recovery does not improve there. Kernel has no daemon RSS; dedicated `wg_peer` objects are1552bytes and reserved host-wide slab pages are0.5625/0.59375/3.28125MiB, only one kernel component. Idle CPU sampling does not establish a CPU/wakeup win. Forwarding write_bytes deltas are0. At1000 load diagnostic jemalloc allocated77.405->71.477MiB, 2+4KiB bins13.461->6.270MiB. Packet descriptor grows72->80bytes; full256-entry queue descriptors18->20KiB. The existing shared pool, lazy session ring, allocator and decay configuration are retained. Stripped executable grows6376bytes; no disk superiority. **Acceptance remains blocked.** Three-control D348 calibration failed. Unqualified control/candidate/control8-flow throughput2.175/2.185/2.175Gbit/s, p99=13.6/13.0/12.9ms, whole-machine CPU s/Gbit=1.440/1.483/1.443. This does NOT establish throughput/p99/CPU non-regression, and there is no qualified kernel/Go CPU win. Keep this as an experimental fork candidate; do not integrate from these diagnostics. Corrected x86 controls acquired a slot after70min and finished. All ten source hashes match Git blobs and test-name presence confirms distinct156-test base/162-test candidate binaries. Base default+serial each148 passed/1 failed/7 ignored; candidate default+serial each154 passed/1 failed/7 ignored. All four fail existing `device::tests::test_indices` timeout assertions; all six added tests pass. This reproduces that failure on unchanged base but does not identify its cause or attribute the earlier three-test failure. Full all-feature x86 suite remains **FAILED**. Earlier stale-timestamp baseline logs remain excluded. No test deadline/assertion was changed. Verbatim corrected markers: ```text J324_FRESH_base_PARALLEL_EXIT=101 J324_FRESH_base_SERIAL_EXIT=101 J324_FRESH_candidate_PARALLEL_EXIT=101 J324_FRESH_candidate_SERIAL_EXIT=101 J324_SOURCE_REFRESH_FINISHED=1 ``` The final marker means comparison finished, not tests passed. Other preserved verification markers, verbatim: ```text J324_X86_RING_EXIT=0 J324_X86_CLI_EXIT=0 J324_X86_CLIPPY_EXIT=0 scripts/verify fast: unavailable (fork has no scripts/verify; infraternal is unchanged) J324_X86_BATCH_EXIT=101 J324_ARM_QUEUE_CHECKS_EXIT=0 J324_ARM_CHECKS_EXIT=0 J324_ARM_EXIT=0 ``` Shipping ring tests on both architectures reported142 passed/0 failed/7 ignored; UDP integration, CLI and doctests passed. Aarch64 ran as one CARGO_INCREMENTAL=0 batch. This does not erase full all-feature x86 failures. Maintenance-cut waiter and once-only post-restart resubmission are recorded above; maintenance was not a test failure. Evidence: compute01 `/home/claude/j324-evidence/x86-source-refresh.log`, `x86-fresh-{base,candidate}-{parallel,serial}.log`, `x86-post-restart.log`, `cleanup.log`; control copies and `source-hash-proof.txt` under `/home/kayg/.cache/codex-scratch/j324/evidence/`; arm `arm-checks.log` there and dev-02 `/home/claude/j324-evidence/arm-checks.log`. Perf `/home/claude/j324/evidence/`: `memory-summary.md`, `heap-comparison-table.md`, `candidate-timing-summary.md`, `scale-cpu-summary.md`, `disk-final.log`, raw candidate/heap-candidate/kernel-slab/competitors directories, and case/restoration logs. Every completed memory/heap/timing scope and restoration check exits0. Both own build targets are removed; no J324 run remains active. Issues remain open. Remaining blockers: qualified D348 calibration/paired acceptance and unresolved existing all-feature timeout failure.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
j470b-server-mesh-rejoin
mesh-compat-382
j444b-d381-key-wins-flood
j461b-macos-utun-reader-thread
j475b-remove-host-motion
fix-reconnect-new-link
j481-d380
dev
j479-mesh-soak
fix/j476-host-facts
j475-motion-pill-split
j470-mac-node-join
root-upgrade-d376
j474-review-merge
latency-targets-120hz
nightly-suite-shards
gotatun-pin-j469
macos-utun-reader-thread
fix-flood-echo-stall
wip/j416
motion-fast-ease-out
mesh-addhost-j259
takeover-close-terminals-d368
kbd-instant-followups
fix-trust-key-double-click
nightly-suite-partial-report
appwin-icons-j207
rusqlite-slice1
swipe-content-first-d374
mesh-mtu-381
addhost-instant-open
glass-contrast-d365-mock
glass-contrast-d365
signout-followups-442
fix-startup-toast-red
research-gotatun-matrix-j317
themes-sync-j304
fix-load-flakes
fix-nix-zig-patch
fix-compute01-browser-baseline
ratchet-size-baseline-419
predict-auto-d130
revoke-coverage-d367
nightly-suite-queue-timeout
nightly-browser-suite
fix-rustsec-2026-0459
fonts-ioskeley-obel
log-retention-d375
fix-kbd-no-anim-427
fix-dev-red-specs-2
takeover-proof-305-j205
wip/j410-server-pump
ui-consistency-d372
fix-osc52-clipboard
libghostty-vt-1.0
forget-retired-host-d369
j404-merge-426
fix-setup-lock-race
fix-touch-select-426
revoke-access-report-ui
fix-predictive-echo-specs
fix-selinux-agent-exec
wip/j395-checkpoint2
wip/j395
j391-review-scratch
ui-consistency-j195-deferred-rebased
fix-rustc-paths-421
decisions-wording-341
fix-install-suite-red
fix-dev-rtt-specs
fix-pending-removal-align
fix-privileged-skip
fix-install-suite-404
review-j368
mesh-dns-j254
revoke-access-report-d356b
revoke-privilege-round5
fix-removal-confirm
review-j360
gotatun-rotation-367
deps-refresh-340
review-j354
review-J351
review-j350
addhost-takeover-d359
revoke-report-items-347
revoke-end-sessions-d362
review-j348
size-mesh-ssh-346
size-trim-337
revoke-access-diff-d356
j259-pre-rebase-b06c5a6
j305-ended-at-start-spec
addhost-sudo-consent
fix-layout-real-specs-411
fix-shell-open-411
fix-console-hosts-373
fix-shell-exit
fuzz-host-name-353
fix-host-details-410
macos-filter-j197
research-gotatun-tricks
revoke-p3-j216l
stale-state-pill
revoke-followups-j216k
revoke-progress-j216e
backup/macos01-20261106/lume/j197
backup/macos01-20261106/lume/appwin-icons-j207-stash0
backup/macos01-20261106/claude/stash-0
backup/macos01-20261106/claude/stash-1
backup/macos01-20261106/claude/J80-macos-s1
backup/macos01-20261106/claude/J80
backup/macos01-20261106/claude/J80-slice1-20261102
backup/macos01-20261106/claude/J16b-appwin-on-dev
backup/macos01-20261106/claude/appwin-launch-idle-146
backup/macos01-20261106/claude/J80-s1-run
wip/j259
trust-285-p3c
runtime-lock-j216f
mesh-ssh-p3-j175
tun-proof-p3b
trust-285-p3
terminal-polish-j271
tun-proof-p3
ssh-install-flakes-j216i
fuzz-fixes-j264
trust-285-a
trust-285-b
restart-oversized-fixture-j198
backup/macos01-20261106/lume/appwin-icons-j207
restart-bound-j216h
up-preflight-p3-j198
backup/macos01-20261106/lume/j251
up-preflight-j198
wip/up-preflight-j198-j245-p2
wip/up-preflight-j198-j245-p1
agent-fsync-offloop-j196
wip/up-preflight-j198-j245
settings-offscreen-j231
wip/up-preflight-j198-j226-inspection
wip/j103f
themes-j206
wip/up-preflight-j198-j226
flakes-j216c
backup/macos01-20261106/claude/j197
flakes-j216b
backup/macos01-20261106/claude/j197-red
wip/up-preflight-j198-review
flakes-j216
backup/macos01-20261106/claude/appwin-icons-j207
engineering-j208b
appwin-mock-j209
toast-stack-j165
engineering-j208
decisions-d351
decisions-d350
decisions-d349
appearance-presets-mock-j129
decisions-d348
docs-audit-spec
doc-coverage-baseline-raise
decisions-d347
decisions-d346b
decisions-d346
j88h-profile
ui-consistency-j195-deferred
ui-consistency-j195
engineering-practices-j194
preferred-login-ui-j166
signin-pools-263-j174
macos-node-265-s1
decisions-d345
decisions-d344
proxy-cgroup-j164
verify-quick-fuzz-329
proxy-start-io-j168
timing-flakes-j163
up-263-s1
mcp-events-j147
verify-quick-fuzz-j158
proxy-p3-j159
agent-resource-j157-detect
wip/j103d-final
remote-cancel-hang-j155
proxy-traefik-281-j62d
j149-verify-quick
proxy-traefik-281-j62b
j150-drop-unnamed
wip/j79d-red-r4
wip/j103d-d332
rootless-toast-j142
wip/j103d-head
wip/j103d
j125c-mcp
macos-build-j136
backup/macos01-20261106/claude/j136
wip/j79d-red-cli
wip/j79d-red-server
j121-link-liveness
j118c-tracer-failures
wip/j79c-head
release-desktop-j131
j124c-signed-macos
wip/j79c-red-p3
wip/j79c-red-p23
wip/j79c-red-p22
merge-j128-check
perf-deferred-j128
sidebar-toggle-260
j127-shortcut-changed-dot
j125-mcp
j118-tracer-failures
j126-account-summary
settings-sections-mock-j92
j120-pinch-lock
sidebar-292
backup/macos01-20261106/claude/j120
taste-2026-11-03
row-actions-242-215
backup/macos01-20261106/claude/stash-0-untracked
backup/macos01-20261106/claude/J80-265-s1-run
backup/macos01-20261106/claude/J80-macos-s1-20261102
j112-canvas-commands
ui-no-overshoot-scroll
fix-recovery-restart
backup/macos01-20261106/claude/stash-1-untracked
taste-2026-11-02
size-baseline-ratchet
machine-cards-structure-j108
j109-boot-copy-mockups
fix-dev-red
wip/up-263-s1
j95b-settings-rows
proxy-traefik-281
j95-option-rows
docs-human-j91
docs-keys-palette
fix-flood-1013-270-j38d
mesh-ssh-s1
mesh-ssh-s0
addhost-d291-tunnel
test-stability-load
mesh-relay-220
repro-296
shim-thp-off-24
mesh-pins-258-v2
mesh-dns
signin-d311-merged
docs-pages
ui-size-283
fix-ui-batch-specs
wip/j12b-edge-update
gate-on-arm
fix-rename-sync-293
trust-285-j60b
fix-focus-ring-clip
integ-ui-1101
edge-update-v2
signin-d311
taste-2026-11-01
layout-per-tab-250
ui-approved-splitopt-224
ui-approved-splitopt-224-on-dev
icon-text-centre-282
ui-approved-189-212-216
research-reverse-proxy
transfer-budget-320
review-mesh-pins-258-r2
sidebar-pill-259
review-mesh-relay-220
wip/j12-edge-update-v2
nix-package
status-crash-loop-237
fix-dev-test-failures
fix-fixture-socket-path
wip/mesh-relay-220-pre-dev-advance-rebase
fix-canvas-side-clip
research-head-to-head
frosted-real-256
review-mesh-pins-258
wip/mesh-relay-220-pre-final-rebase
review-layout-per-tab-250-r2
broad-suite-triage
nix-devshell
review-status-crash-loop-237-r3
wip/nix-devshell-pre-p3
taste-frosted-real
wip/code-serve-demo-2026-10-30
wip/code-serve-glass-2026-10-30
wip/code-serve-clusters-2026-10-30
wip/code-merge-release94-dev-2026-10-30
wip/rproxy-delta-scratch-2026-10-30
wip/stale-note-257-staged-2026-10-30
wip/d241-size-staged-2026-10-30
review-broad-suite-207
wip/mesh-pins-258-pre-split
wip/mesh-relay-220-pre-rebase
mesh-measure-220
plan-211-update
review-edge-autoupdate-r3
perf24-reconnect
review-stale-note-r2
release-prod-live
d241-shell-marker
review-mesh-pins-server-258
review-revocation-r3
review-setup-token-r2
review-sync-journal-r3
wip/review-mesh-pins-258-prune-review-1995fee46
frosted-scroll-256
sync-journal-r3
revocation-per-account-wip
review-tokio-asyncfd-0386
review-sync-p3s
wip/stash-8by64-stash0-2026-10-30
mesh-drag-r2
setup-token-data-wip
release-9-4
auth-busy-1013
mesh-view-overlap
sync-p3s
review-auth-busy-wake
review-readd-r8-wip
mesh-drag-bounds
audit-auth-93
app-windows-stream-start
review-authority-cap-r3
app-windows-r7-measure
readd-minimal-r6
readd-minimal-r5
app-windows-r7-soak
readd-minimal-r4
app-windows-r7
app-windows-r6
readd-minimal-r3
readd-minimal-r2
readd-minimal
merge-release8-dev
review-busy-page-liveness-r2
review-busy-page-liveness
mesh-view-trim
review-authority-read-retry-r2
review-remove-uninstall-r2
release-retry-reattach
release-retry
review-remove-uninstall
review-authority-read-retry
app-windows-ship
mesh-autoswitch-p2
mesh-ssh
mesh-relay
mesh-macos-node
screen-and-control
session-labels
review-real-logins-r5
review-real-logins-r4
review-real-logins-r3
shim-memory-r2
review-real-logins-r2
shim-memory
shadow-addhost-instant
shadow-real-logins
shadow-same-session-flood
shadow-prod-perf-fix
hosts-from-server
app-windows-real
app-windows-transport
release-prod-6w
mesh-policy-verify
quiet-close-toast
dev-merge-r4
review-mesh-agent-transport
review-app-windows-transport
settings-roundtrip
review-release-pipeline-r4
mesh-agent-transport
ended-strip-fix
review-audit-ca-update-d244
settings-rows
phone-width
signout-clear
fix-restart-tests
dev-merge-rp
signer-hardening
review-audit-ca-update
terminal-fidelity
review-audit-ca-update-wip
mesh-integrate
review-mesh-agent-transport-515d3712
review-mesh-dataplane-feec1988
mesh-dataplane
signout
research-canvas-node
wip/review-renderer-cost
wip/review-terminal-fidelity
padlock-glyph
review-release-pipeline-2e95b689
phone-polish
sidebar-polish
inbox-generic
review-mesh-dataplane-378d4055
prod-batch
ui-inbox
ui-ca-unlock
ui-machine-sections
review-mesh-dataplane-a5f744fb
tls-session-prod
session-bounds-prod
fix-dead-session
session-bounds
tls-session
palette-polish
shortcuts-d222
research-appmeta
phone-gutter
ui-centre-button
wip/ui-centre-button
wip/tls-session
review-addhost-breakit
ui-platform
renderer-cost
remote-agents
mesh-api
ssh-bootstrap
prod-hotfix2
prod-fixes
review-remote-agents
review-mesh-dataplane
session-life
research-browser
review-mesh-api
research-capture
review-mesh-api-round3
review-mesh-api-round1
gate-stability
review-gate-stability
review-relay-direct
review-mesh-api-findings
proto-sessions
font-picker
proto-addhost
proto-windows
mesh-core
latency-gate
research-bsd
research-discovery
keytray-mods
mock-core
ui-tweaks
fix-motion-sel
fix-motion
integrate2
fix-robust
t-web
t-server
vt-ghostty
fix-snap-echo
t-web-auth
review-vt2
t-agent
lane-protocol
lane-shim
lane-auth
s1-base
s1-contracts
s1-workspace
No results found.
Labels
Clear labels   
launch-qa
Findings from launch QA.

No labels
launch-qa
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/infraternal#407
No description provided.