kayg/infraternal
Watch
1
0
Fork
You've already forked infraternal
 0
Code Issues 356 Pull requests Projects Releases Packages Wiki Activity Actions

Typing does nothing in a window left open across a server restart (prod, 2026-10-13) #199

New issue
Closed
opened 2026-10-13 05:16:28 +00:00 by kayg · 221 comments
kayg commented 2026-10-13 05:16:28 +00:00
Owner
Copy link

Owner report 2026-10-13 ~10:35 IST: a terminal window left open overnight on production (https://infraternal.kayg.org, session kayg@monitor01 · o1/update, session id 7f3044cc1b8d40f89e91fb04c3db3ee4) shows its last screen and a green dot, but typing does nothing.

Evidence (read-only, monitor01, 2026-10-13 05:15 UTC):

  • Production server was restarted at 2026-10-12 21:03:23 UTC (02:33 IST) for the #195 deploy. Journal: Unit process 186207 (infraternal) remains running after unit stopped, then infraternal agent failed: unexpected end of file; new server 196841 and new local daemon infraternal agent --fd 3 pid 196850 started. No log line since.
  • The session's shim (pid 186513, __shim --session 7f3044cc…, started 2026-10-12 13:03 UTC) is alive. Its shell runs ssh 10.69.69.51 (pid 193009), and that TCP connection is ESTABLISHED and healthy (rtt 164 ms, nothing queued). So the session itself is fine.
  • The new daemon 196850 holds no socket to that shim (ss -xp shows no peer for pid 186513). Across all 72 shims, only 1 has any socket at all.
  • The CA is locked since the restart (every restart re-locks it; owner has not unlocked yet).
  • The browser still shows the session as live (green dot) and the console says 69 sessions.

So input is dropped silently: either sessions created before a restart are not re-adopted by the new local daemon, or attach/input is refused while the CA is locked, and the UI does not say so.

Wanted:

  1. Root cause with a failing regression test: open a session, keep a browser client attached, restart the server the way production does (systemd, KillMode=process), and type — with the CA locked and unlocked.
  2. Fix: a client attached across a restart regains typing on its own within the D259 budgets, with no reload. If something the owner must do (unlock) blocks input, the window says so plainly with the action, and never shows a live dot while input goes nowhere.
  3. The owner's stuck tab is the acceptance test: do not touch production sessions or processes.
Owner report 2026-10-13 ~10:35 IST: a terminal window left open overnight on production (https://infraternal.kayg.org, session `kayg@monitor01 · o1/update`, session id 7f3044cc1b8d40f89e91fb04c3db3ee4) shows its last screen and a green dot, but typing does nothing. Evidence (read-only, monitor01, 2026-10-13 05:15 UTC): - Production server was restarted at 2026-10-12 21:03:23 UTC (02:33 IST) for the #195 deploy. Journal: `Unit process 186207 (infraternal) remains running after unit stopped`, then `infraternal agent failed: unexpected end of file`; new server 196841 and new local daemon `infraternal agent --fd 3` pid 196850 started. No log line since. - The session's shim (pid 186513, `__shim --session 7f3044cc…`, started 2026-10-12 13:03 UTC) is alive. Its shell runs `ssh 10.69.69.51` (pid 193009), and that TCP connection is ESTABLISHED and healthy (rtt 164 ms, nothing queued). So the session itself is fine. - The new daemon 196850 holds no socket to that shim (`ss -xp` shows no peer for pid 186513). Across all 72 shims, only 1 has any socket at all. - The CA is locked since the restart (every restart re-locks it; owner has not unlocked yet). - The browser still shows the session as live (green dot) and the console says 69 sessions. So input is dropped silently: either sessions created before a restart are not re-adopted by the new local daemon, or attach/input is refused while the CA is locked, and the UI does not say so. Wanted: 1. Root cause with a failing regression test: open a session, keep a browser client attached, restart the server the way production does (systemd, KillMode=process), and type — with the CA locked and unlocked. 2. Fix: a client attached across a restart regains typing on its own within the D259 budgets, with no reload. If something the owner must do (unlock) blocks input, the window says so plainly with the action, and never shows a live dot while input goes nowhere. 3. The owner's stuck tab is the acceptance test: do not touch production sessions or processes.
real-restarts-final.log
11 KiB
real-three-captures.log
6.9 KiB
browser-units-final.log
1.5 KiB
restart-locked-ready.png
88 KiB
restart-unlock-ready.png
97 KiB
restart-unlock-ready.png
35 KiB
native-primed-green.log
1.1 KiB
merged-review3.log
1.1 KiB
merged-review3-green.log
605 B
baseline-review3.log
9.4 KiB
held-open-green.log
718 B
plugin-lifecycle-green.log
750 B
restart-locked-catching-up.png
70 KiB
restart-unlock-catching-up.png
29 KiB
verify-iterate-scope.log
255 KiB
resume-sink-own-target.log
12 KiB
clippy-sink-own-target.log
7.8 KiB
real-current-control.log
1 KiB
staging-proof.txt
5.9 KiB
mock-browser-repeat.log
2.4 KiB
verify-fast-final.log
265 KiB
ts-final-head.log
24 KiB
ts-final-green.log
29 KiB
ts-verify-final-head.log
29 KiB
artifact-sha256-final.txt
331 B
push.log
101 B
staging-diagnostic.txt
9.1 KiB
independent-replies-gates.log
62 KiB
corrected-client-comparison.log
6.3 KiB
shipping-sha256.txt
321 B
staging-proof-transport.txt
5.4 KiB
native-probe-red-remote.log
2.5 KiB
mock-browser-repeat.log
2.4 KiB
local-static-server-trace.log
4.2 KiB
local-static-exit-reasons.log
179 B
socket-suite-transport.log
4.1 KiB
staging-49e809d9.txt
5.5 KiB
browser-units.log
1,008 B
paired-budget-six-retry.log
3.8 KiB
vm-browser-systemd-nsenter.log
1.8 KiB
vm-browser-liveness.log
3.4 KiB
vm-browser-three-window.log
7.7 KiB
vm-browser-mock.log
745 B
deadline-browser-green.log
3.5 KiB
deadline-browser-locked.log
764 B
deadline-restart-unlocked-ready.png
35 KiB
lease-delay-red.log
1.9 KiB
lease-delay-green.log
704 B
lease-clippy.log
335 B
route-after-stop.png
45 KiB
route-after-stop.png
45 KiB
route-after-stop.png
45 KiB
verify-iterate-d64e12c8.log
295 KiB
ts-resume.log
29 KiB
staging-resume.txt
6 KiB
vm-browser-resume.log
13 KiB
vm-mock-resume-fixture.log
1.1 KiB
idle-proof-module-gates.log
2.1 KiB
idle-proof-remote-tests.log
1.6 KiB
restart-locked-catching-up.png
44 KiB
restart-locked-ready.png
53 KiB
restart-unlock-catching-up.png
29 KiB
restart-unlock-ready.png
35 KiB
verify-fast-17fae13f.log
268 KiB
final-artifacts-resume.txt
468 B
browser-probe-scheduling-red.log
761 B
native-probe-correlation-red.log
1.6 KiB
real-controls.log
6.4 KiB
real-locked-repeat.log
851 B
restart-locked-catching-up.png
46 KiB
restart-locked-ready.png
51 KiB
protocol-repeat.log
3.6 KiB
real-three-controls.log
6.7 KiB
candidate-0.json
1.2 KiB
candidate-before.txt
3.9 KiB
baseline.txt
3.6 KiB
perf-before.txt
8.5 KiB
deadline-restart-locked-ready.png
53 KiB
browser-probe-scheduling-expanded-red.log
1.4 KiB
native-probe-final-red.log
1.2 KiB
ts-fast-final.log
29 KiB
input-exit-controls.log
811 B
transport-controls-expanded.log
931 B
verify-fast-final.log
294 KiB
independent-comparison-1.log
6.2 KiB
independent-comparison-6.log
4.4 KiB
native-artifacts-sha256.txt
322 B
fixed2-six.txt
2 KiB
authority-red2.log
1.2 KiB
authority-green.log
621 B
final-idle.txt
4.6 KiB
browser-recovery-final.log
9.5 KiB
restart-locked-ready.png
58 KiB
restart-unlock-ready.png
40 KiB
idle-authority-red4.log
1.3 KiB
idle-authority-final-tests.log
1.6 KiB
broken-close-cleanup.log
149 B
broken-close-red.log
1.4 KiB
broken-close-green.log
105 B
incoming-proof-red.log
2.4 KiB
incoming-proof-final-tests.log
4 KiB
incoming-proof-clippy3.log
280 B
mesh-exchange-red.log
1.2 KiB
mesh-exchange-green.log
4.2 KiB
mesh-exchange-clippy.log
330 B
size-current.txt
793 B
typescript-head.log
29 KiB
static-head-checks.log
865 B
workspace-head-final-tests.log
136 KiB
rust-head-independent.log
250 KiB
separate-control-gates.log
7.5 KiB
independent-reverse-safe-1.log
6.6 KiB
peer-path-red.log
989 B
peer-path-green.log
192 B
size-validated.txt
754 B
native-fixture-cleanup-check.txt
77 B
deadline-stress.log
5.6 KiB
verify-fast-final.log
277 KiB
verify-iterate5.log
1.1 KiB
build-base-validated.log
8.5 KiB
build-17-invalidated.log
1.8 KiB
terminal-close-red.log
1.2 KiB
reconnecting-route-red.log
1.2 KiB
reconnecting-route-green.log
109 B
accounting-red.log
1.1 KiB
accounting-green.log
100 B
browser-duplex-final.log
12 KiB
browser-duplex-retry.log
1.8 KiB
baseline-zero-strict.txt
1.4 KiB
duplex-matrix.txt
13 KiB
auth-timeout-baseline.log
4.7 KiB
auth-timeout-head-invalidated.log
4.1 KiB
verify-fast-strict-fixture.log
271 KiB
shared-cadence-red.log
1.1 KiB
shared-cadence-green.log
101 B
bench-perf-stat-source.txt
2.8 KiB
baseline-six-perf-stat.txt
1.7 KiB
shared-cadence-red.log
1.1 KiB
shared-cadence-green.log
101 B
terminal-close-green.log
106 B
deadline-verify-fast.log
286 KiB
deadline-final-sizes.json
743 B
deadline-vm-cleanup.txt
623 B
deadline-host-cleanup.json
305 B
final-cpu-table.txt
1.9 KiB
shared-cadence-matrix.txt
11 KiB
counter-comparison.txt
8.1 KiB
staging-one-corrected-counters.txt
2 KiB
fixed-one-corrected-counters.txt
2 KiB
final-binary-sizes.txt
440 B
perf-machine.txt
241 B
verify-iterate-head13.log
293 KiB
final-native-cleanup.txt
36 B
final-local-cleanup.txt
36 B
candidate-profile-report.txt
6.2 KiB
candidate-profile-receipt.txt
2.4 KiB
candidate-final.perf.gz
13 KiB
final-runtime-cleanup.txt
42 B
final-runtime-cleanup.txt
42 B
build-cleanup.txt
213 B
perf-cleanup.txt
64 B
m1-red.log
1.4 KiB
m1-green.log
524 B
m2-driver-red.log
3.1 KiB
m2-driver-green.log
2.5 KiB
adaptive-red.log
2.9 KiB
adaptive-green.log
1.1 KiB
control-queue-scoped-retry.log
450 B
m2-workspace-regression.log
417 B
first-delayed-probe-red.log
2.1 KiB
first-delayed-probe-green.log
2.7 KiB
native-cache-reuse.log
354 B
deadline-vm-artifact-cleanup.json
362 B
deadline-driver-green.log
4.8 KiB
idle-cpu-acceptance-red.log
2.3 KiB
candidate-active-symbols.txt
1.6 KiB
candidate-map-build.log
436 B
separate-comparison-1.log
6.5 KiB
separate-browser-idle.log
1.5 KiB
separate-c538-flat-compact.txt
2.4 KiB
deadline-browser-green.log
3.5 KiB
fixture-deadline-red.log
1.7 KiB
browser-units.log
3.5 KiB
cadence-jitter-red.log
3.7 KiB
cadence-jitter-green.log
2.9 KiB
read32k-6-answer-paced.log
3.6 KiB
final-browser-idle-old.log
1.5 KiB
final-browser-idle-new.log
1.5 KiB
copoll-6-answer-paced.log
1.8 KiB
copoll-socket-configured.log
6.4 KiB
idle-browser-ready.png
42 KiB
final-browser-mock.log
987 B
final-browser-restart.spec.ts.log
1.8 KiB
final-browser-restart-liveness-review.spec.ts.log
3.9 KiB
late-timer-response-red.log
4.4 KiB
late-timer-response-green.log
3.1 KiB
late-timer-fixture-red.log
1.2 KiB
scalar-6-answer-paced.log
3.5 KiB
scalar-1-answer-paced.log
2.8 KiB
perf-vm-environment-final.txt
208 B
separate-c538.perf.gz
20 KiB
deadline-pair-1.log
6.6 KiB
deadline-pair-6.log
7.9 KiB
deadline-browser-tests-navigation-repeat.log
8.9 KiB
native-virtual-clock.log
2.1 KiB
fixture-units.log
109 B
scalar-active-symbolized-flat.txt
47 KiB
scalar-active-symbolized-callgraph.txt
214 KiB
scalar-six-higher-profile.log
3.5 KiB
scalar-one-tcp-diagnostic.log
3.3 KiB
grouped-gates.log
6.4 KiB
handshake-proof-init-gates.log
5.8 KiB
remove-group-helper-gates.log
2 KiB
deadline-browser-tests-initial-navigation.log
9.6 KiB
deadline-verify-iterate.log
300 KiB
deadline-baseline-artifact.txt
262 B
fixture-deadline-issued-red.log
1.7 KiB
separate-comparison-0.log
3.5 KiB
serial26-recent-gates.log
9.6 KiB
serial26-6-final.log
3.5 KiB
serial26-browser-idle.log
1.5 KiB
serial26-one-tcp.log
3.4 KiB
serial26-0-final.log
1.5 KiB
serial26-browser-restart-liveness-review.spec.ts.log
3.4 KiB
serial26-browser-restart.spec.ts.log
1.9 KiB
serial26-browser-three-session.log
10 KiB
serial26-one-profile.log
3.5 KiB
serial26-one-high-profile.log
1.5 KiB
buffered-controls-red.log
1.1 KiB
paired-budget-gates.log
10 KiB
serial26-one-high-retry.log
3 KiB
serial26-high-flat.txt
15 KiB
serial26-high-inclusive.txt
29 KiB
paired-budget-gates.log
10 KiB
serial26-high.perf.gz
16 KiB
paired-budget-server-unit.log
52 KiB
paired-one-comparison.log
5.8 KiB
paired-budget-comparison.log
1.4 KiB
paired-budget-0.log
1.5 KiB
paired-budget-6.log
3.4 KiB
native-batch-gates.log
1 KiB
native-batch-gates.log
54 KiB
deadline-native-red.log
2.6 KiB
deadline-browser-red.log
3.8 KiB
deadline-driver-green.log
4.7 KiB
separate-comparison-6.log
8.1 KiB
artifacts.json
1,013 B
release-build.log
26 KiB
toolchain.json
389 B
browser-restart.log
1.4 KiB
browser-liveness.log
2.9 KiB
browser-three-window.log
4.2 KiB
browser-mock.log
1.1 KiB
browser-results.json
194 B
browser-restart-initial.log
213 B
restart-unlock-ready.png
35 KiB
restart-locked-catching-up.png
44 KiB
restart-unlock-catching-up.png
29 KiB
browser-production-captures.log
1.3 KiB
browser-final-cleanup.txt
106 B
staging-proof.txt
6 KiB
staging-command.log
6.5 KiB
verify-fast-generated-preview.log
6.6 KiB
size-comparison.json
646 B
operational-adapters.txt
16 KiB
verify-fast.log
287 KiB
final-validation.json
1.1 KiB
local-final-cleanup.json
962 B
vm-final-cleanup.json
469 B
production-key-memory.txt
158 B
deadline-real-socket-red.log
1.1 KiB
continuation-bench-production-6-original.log
2.7 KiB
candidate-six-thread-profile.log
3.7 KiB
restart-locked-ready.png
53 KiB
final-gate-summary.txt
662 B
vm-production-captures-resume.log
1.6 KiB
verify-typescript.log
29 KiB
staging-script-sha256.txt
196 B
final-browser-restart-three-session.spec.ts.log
9.2 KiB
real-baseline-control.log
1.2 KiB
staging-final-head.txt
5.3 KiB
serial26-browser-review-rerun.log
3.9 KiB
late-timer-fixture-green.log
104 B
verify-fast-head13.log
267 KiB
staging-idle-sockets.txt
6 KiB
native-control-batch-red.log
1.6 KiB
separate-browser-recovery-suite.log
18 KiB
restart-locked-ready.png restart-unlock-ready.png restart-unlock-ready.png restart-locked-catching-up.png restart-unlock-catching-up.png deadline-restart-unlocked-ready.png route-after-stop.png route-after-stop.png route-after-stop.png restart-locked-catching-up.png restart-locked-ready.png restart-unlock-catching-up.png restart-unlock-ready.png restart-locked-catching-up.png restart-locked-ready.png deadline-restart-locked-ready.png restart-locked-ready.png restart-unlock-ready.png idle-browser-ready.png restart-unlock-ready.png restart-locked-catching-up.png restart-unlock-catching-up.png restart-locked-ready.png
kayg commented 2026-10-13 05:17:30 +00:00
Author
Owner
Copy link

Start #199: branch stuck-after-restart, base/head 9ab4329c. Read contracts and all issue comments (0); no parent referenced. Read related deploy issue #195 history. Reproduce with private local fixtures only; no production access, processes or sessions. Push atomic commits only to origin/stuck-after-restart.

Start #199: branch stuck-after-restart, base/head 9ab4329c. Read contracts and all issue comments (0); no parent referenced. Read related deploy issue #195 history. Reproduce with private local fixtures only; no production access, processes or sessions. Push atomic commits only to origin/stuck-after-restart.
kayg commented 2026-10-13 05:29:39 +00:00
Author
Owner
Copy link

Finding #199: apps/tracer/src/connection.ts:395 originally sets resume deadline at Welcome +120s, while websocket.rs:1887-1895 starts server recovery grace at restart/disconnect. Unit overnight-before.log fails [false,false] vs expected [false,true]; deadline fix passes. This loses the durable writer and clears queued input via terminal.ts:1044-1060; it also forces a full repaint/two-round-trip attach. Embedded terminal_agent (websocket.rs:1648) returns the local daemon without consulting CA state. A simple real-PTY/server-PID restart with an expired browser clock passes before the fix (browser-before2.log), so the clock mismatch alone does not yet reproduce the owner’s permanently stuck tab. Extending reproduction to queued input, actual systemd KillMode=process and CA locked/unlocked; no production access.

Finding #199: apps/tracer/src/connection.ts:395 originally sets resume deadline at Welcome +120s, while websocket.rs:1887-1895 starts server recovery grace at restart/disconnect. Unit overnight-before.log fails [false,false] vs expected [false,true]; deadline fix passes. This loses the durable writer and clears queued input via terminal.ts:1044-1060; it also forces a full repaint/two-round-trip attach. Embedded terminal_agent (websocket.rs:1648) returns the local daemon without consulting CA state. A simple real-PTY/server-PID restart with an expired browser clock passes before the fix (browser-before2.log), so the clock mismatch alone does not yet reproduce the owner’s permanently stuck tab. Extending reproduction to queued input, actual systemd KillMode=process and CA locked/unlocked; no production access.
kayg commented 2026-10-13 05:34:28 +00:00
Author
Owner
Copy link

Completed resume fix 1e9d8a6b, pushed origin/stuck-after-restart. Grace starts on disconnect; failed retries cannot renew it. Strict tracer TypeScript check and cargo fmt pass; 3 focused unit tests pass, 9 assertions, with the overnight test failing before the change. Evidence overnight-before.log / overnight-after.log. Simple real PTY restart control passes before the fix; no claim yet that the owner’s permanent freeze is reproduced. Systemd/CA fixture currently rejects its remote-server startup configuration; resolving the isolated fixture.

Completed resume fix 1e9d8a6b, pushed origin/stuck-after-restart. Grace starts on disconnect; failed retries cannot renew it. Strict tracer TypeScript check and cargo fmt pass; 3 focused unit tests pass, 9 assertions, with the overnight test failing before the change. Evidence overnight-before.log / overnight-after.log. Simple real PTY restart control passes before the fix; no claim yet that the owner’s permanent freeze is reproduced. Systemd/CA fixture currently rejects its remote-server startup configuration; resolving the isolated fixture.
kayg commented 2026-10-13 05:55:04 +00:00
Author
Owner
Copy link

Real-engine red regression #199: exact 9ab4329c static binary (20,333,104 bytes), production tracer dist, real owner passkey/CA, real PTY, systemd transient service with KillMode=process. Initial typing computes before-42; stop closes browser transport (bridge closedBy server 1006, page data-connected=false). Type echo waiting-$((7*7)) + Return while stopped, restart with CA locked: real shim is reattached and old screen returns, but waiting-49 never appears. systemd-before11.log records the failure. The browser discards its queue in restartCursors on fresh Welcome because its original 120s deadline expired during the simulated overnight interval. Also window-card initial status and session-ui.on-demand.ts:122 label surviving sessions live independently of transport, explaining the per-window false green dot. Adding a separate red assertion for that dot. No production access.

Real-engine red regression #199: exact 9ab4329c static binary (20,333,104 bytes), production tracer dist, real owner passkey/CA, real PTY, systemd transient service with KillMode=process. Initial typing computes before-42; stop closes browser transport (bridge closedBy server 1006, page data-connected=false). Type echo waiting-$((7*7)) + Return while stopped, restart with CA locked: real shim is reattached and old screen returns, but waiting-49 never appears. systemd-before11.log records the failure. The browser discards its queue in restartCursors on fresh Welcome because its original 120s deadline expired during the simulated overnight interval. Also window-card initial status and session-ui.on-demand.ts:122 label surviving sessions live independently of transport, explaining the per-window false green dot. Adding a separate red assertion for that dot. No production access.
kayg commented 2026-10-13 06:29:00 +00:00
Author
Owner
Copy link

New evidence, 2026-10-13 ~12:00 IST (06:28 UTC): the owner's stuck window started accepting input again without a reload and without any deploy or restart. Server MainPID still 196841, NRestarts=0, no journal lines in the last 3 h. The shim for session 7f3044cc… (pid 186513), which had no socket peer at 05:15 UTC, now has 2 unix-socket connections to the daemon.

So the session and the daemon were able to attach all along; the browser was holding a connection that looked live (green dot) but carried no input, until something on the client side (tab focus, laptop wake, network change, or a timer) made it reconnect. Prime suspect: the client's liveness detection — a dead or half-open WebSocket/route after the server restart is not detected, and nothing forces a resubscribe. The fix must detect it within the D259 budgets (one RTT + 50 ms after the server is back), not minutes or hours later. Please make the regression test cover a client that stays open across the restart and does not change focus.

New evidence, 2026-10-13 ~12:00 IST (06:28 UTC): the owner's stuck window started accepting input again **without a reload and without any deploy or restart**. Server MainPID still 196841, NRestarts=0, no journal lines in the last 3 h. The shim for session 7f3044cc… (pid 186513), which had no socket peer at 05:15 UTC, now has 2 unix-socket connections to the daemon. So the session and the daemon were able to attach all along; the browser was holding a connection that looked live (green dot) but carried no input, until something on the client side (tab focus, laptop wake, network change, or a timer) made it reconnect. Prime suspect: the client's liveness detection — a dead or half-open WebSocket/route after the server restart is not detected, and nothing forces a resubscribe. The fix must detect it within the D259 budgets (one RTT + 50 ms after the server is back), not minutes or hours later. Please make the regression test cover a client that stays open across the restart and does not change focus.
kayg commented 2026-10-13 06:34:13 +00:00
Author
Owner
Copy link

Progress #199: both real systemd/production-browser restart cases now pass (systemd-captures2.log, 2 passed), including retained queued input, CA locked and unlocked, window connecting during the stop and live after replay. Four production screenshots captured. Shared snapshot/locked-input regressions: 3 passed; existing CA UI cases: 4 passed (desktop and phone). Full native fast rerun: 740 passed, 0 failed, 78 configured ignores; fmt/VT fmt/clippy/cargo-deny pass. Both static sizes match 9ab4329c exactly (x86 20,333,104; arm 18,112,240), delta 0, inherited frozen caps still fail. Perf VM 10.69.69.208 at 72 real PTYs and RTT150: old backoff recovery 219.776ms failed 200ms; first shortened window 204.756ms failed, retained picture true/first frame5ms. Further shortened confirmed-terminal recovery window is queued behind the shared perf lock (flock -w3600). Existing sessions do not require CA unlock; the retained CA UI state now refreshes on a new Welcome. No production access or deployment.

Progress #199: both real systemd/production-browser restart cases now pass (systemd-captures2.log, 2 passed), including retained queued input, CA locked and unlocked, window connecting during the stop and live after replay. Four production screenshots captured. Shared snapshot/locked-input regressions: 3 passed; existing CA UI cases: 4 passed (desktop and phone). Full native fast rerun: 740 passed, 0 failed, 78 configured ignores; fmt/VT fmt/clippy/cargo-deny pass. Both static sizes match 9ab4329c exactly (x86 20,333,104; arm 18,112,240), delta 0, inherited frozen caps still fail. Perf VM 10.69.69.208 at 72 real PTYs and RTT150: old backoff recovery 219.776ms failed 200ms; first shortened window 204.756ms failed, retained picture true/first frame5ms. Further shortened confirmed-terminal recovery window is queued behind the shared perf lock (flock -w3600). Existing sessions do not require CA unlock; the retained CA UI state now refreshes on a new Welcome. No production access or deployment.
kayg commented 2026-10-13 06:53:56 +00:00
Author
Owner
Copy link

Completed implementation steps through bcfd484646, pushed only origin/stuck-after-restart. Separate commits cover adapter readiness, per-binding hooks, inventory repaint, the existing window card, painted replay confirmation, bounded fast reconnect, CA state refresh and transport regressions. Per-commit fmt/strict TS/8 focused unit gates pass (29 assertions). Real systemd cases pass at desktop1440x900 and touch393x852 (systemd-head2.log); screenshots attached under evidence/stuck-after-restart/. Perf VM 72 PTYs, RTT150: desktop169.312ms, phone163.537ms vs200ms; last bitmap retained in both cases. Infraternal daemon RSS5,496,832B; 72 shims total299,012,096B, mean4,152,945B in phone run. Original failed perf runs remain. Full native gate740/0/78 ignored; static deltas0. Finishing fixture/evidence commits and exact-head gate.

Completed implementation steps through bcfd4846462da7117928b4530be6699b6b35c3e2, pushed only origin/stuck-after-restart. Separate commits cover adapter readiness, per-binding hooks, inventory repaint, the existing window card, painted replay confirmation, bounded fast reconnect, CA state refresh and transport regressions. Per-commit fmt/strict TS/8 focused unit gates pass (29 assertions). Real systemd cases pass at desktop1440x900 and touch393x852 (systemd-head2.log); screenshots attached under evidence/stuck-after-restart/. Perf VM 72 PTYs, RTT150: desktop169.312ms, phone163.537ms vs200ms; last bitmap retained in both cases. Infraternal daemon RSS5,496,832B; 72 shims total299,012,096B, mean4,152,945B in phone run. Original failed perf runs remain. Full native gate740/0/78 ignored; static deltas0. Finishing fixture/evidence commits and exact-head gate.
kayg commented 2026-10-13 07:52:44 +00:00
Author
Owner
Copy link

Completed fixture/evidence steps through 2838826447. Separate commits add server-only systemd restart, fixture-owned RSS measurement, shared real browser fixture, functional specs, perf entrypoint and evidence. Every fixture has strict TS/Python/fmt gates; systemd-head2.log passes desktop and phone, perf-final2.log records 169.312/163.537ms at72 sessions and RTT150. Final author review preserved existing approximation/input-pressure status after confirmation; warning-snapshot.log passes. Exact implementation/evidence head605a1109 full fast gate:740 passed,0 failed,78 configured ignores; only inherited static caps fail, both delta0. Final source-head repetition pending. Production acceptance of the original already-loaded tab is unverified; browser-only fixes cannot retroactively replace that bundle. No deploy or production access.

Completed fixture/evidence steps through 283882644730ed23235d7d4d063141686a3db383. Separate commits add server-only systemd restart, fixture-owned RSS measurement, shared real browser fixture, functional specs, perf entrypoint and evidence. Every fixture has strict TS/Python/fmt gates; systemd-head2.log passes desktop and phone, perf-final2.log records 169.312/163.537ms at72 sessions and RTT150. Final author review preserved existing approximation/input-pressure status after confirmation; warning-snapshot.log passes. Exact implementation/evidence head605a1109 full fast gate:740 passed,0 failed,78 configured ignores; only inherited static caps fail, both delta0. Final source-head repetition pending. Production acceptance of the original already-loaded tab is unverified; browser-only fixes cannot retroactively replace that bundle. No deploy or production access.
kayg commented 2026-10-13 08:09:48 +00:00
Author
Owner
Copy link

Result #199: branch stuck-after-restart, head da5321ba72, pushed only origin/stuck-after-restart. No deployment or production access.

Confirmed browser faults: resume grace expired from Welcome rather than disconnect, clearing overnight queued input on a fresh writer; surviving-session inventory made windows look live without a writable attachment. Grace now matches server recovery, window readiness follows session-validated painted replay, confirmed terminals have bounded rapid recovery, and later Welcome refreshes the existing CA state/action. Existing local PTYs resume while the CA is locked.

Real systemd KillMode=process, production browser, real PTYs/SQLite/virtual passkeys: failing-before queue and green-dot regressions retained; both same-page/no-reload desktop and phone cases pass. 8 focused unit cases / 29 assertions, wrapped-ring/locked-input cases (3), existing CA UI cases (4), strict tracer/fixture TypeScript checks pass. Four production screenshots and author checklist: evidence/stuck-after-restart/README.md.

Pinned VM 10.69.69.208 (4 CPUs), 72 real PTYs, fixed RTT 150: final desktop 168.630 ms and phone 158.742 ms against 200 ms; same bitmap retained. Daemon RSS about 5.5 MB; mean shim 4.15 MB × 72, about 299 MB summed RSS. Measurements are scenario results, not p99; locked cold recovery is measured before the second case additionally unlocks and verifies typing.

Native binary deltas against 9ab4329c: x86=20,333,104B (+0; SHA256 identical), arm=18,112,240B (+0; native source unchanged). Existing caps still fail; no budget altered.

Acceptance limitation: the original production tab remains untouched/unverified. An already loaded old browser bundle cannot receive this browser-only repair from a server binary swap alone. Same-page recovery is proven for the fixed client; no claim that the original persistent production stall was inspected or every cause excluded. Owner/orchestrator acceptance and independent review remain. Issue stays open.

Final exact-head scripts/verify fast: 740 passed, 0 failed, 78 configured ignores. Exit 1 solely on inherited static caps. Static diagnostics and gate summary below are verbatim; full equivalent logs are committed in evidence/stuck-after-restart/.

PT_INTERP: absent; size: 20333104 bytes (baseline 16142672 + allowance 262144 = max 16404816)
binary exceeds the 16404816-byte size budget: 20333104 bytes
PT_INTERP: absent; size: 18112240 bytes (baseline 13555672 + allowance 262144 = max 13817816)
binary exceeds the 13817816-byte size budget: 18112240 bytes

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+
Result #199: branch stuck-after-restart, head da5321ba72b846527f72866eb9de922209cf4c9e, pushed only origin/stuck-after-restart. No deployment or production access. Confirmed browser faults: resume grace expired from Welcome rather than disconnect, clearing overnight queued input on a fresh writer; surviving-session inventory made windows look live without a writable attachment. Grace now matches server recovery, window readiness follows session-validated painted replay, confirmed terminals have bounded rapid recovery, and later Welcome refreshes the existing CA state/action. Existing local PTYs resume while the CA is locked. Real systemd KillMode=process, production browser, real PTYs/SQLite/virtual passkeys: failing-before queue and green-dot regressions retained; both same-page/no-reload desktop and phone cases pass. 8 focused unit cases / 29 assertions, wrapped-ring/locked-input cases (3), existing CA UI cases (4), strict tracer/fixture TypeScript checks pass. Four production screenshots and author checklist: evidence/stuck-after-restart/README.md. Pinned VM 10.69.69.208 (4 CPUs), 72 real PTYs, fixed RTT 150: final desktop 168.630 ms and phone 158.742 ms against 200 ms; same bitmap retained. Daemon RSS about 5.5 MB; mean shim 4.15 MB × 72, about 299 MB summed RSS. Measurements are scenario results, not p99; locked cold recovery is measured before the second case additionally unlocks and verifies typing. Native binary deltas against 9ab4329c: x86=20,333,104B (+0; SHA256 identical), arm=18,112,240B (+0; native source unchanged). Existing caps still fail; no budget altered. Acceptance limitation: the original production tab remains untouched/unverified. An already loaded old browser bundle cannot receive this browser-only repair from a server binary swap alone. Same-page recovery is proven for the fixed client; no claim that the original persistent production stall was inspected or every cause excluded. Owner/orchestrator acceptance and independent review remain. Issue stays open. Final exact-head scripts/verify fast: 740 passed, 0 failed, 78 configured ignores. Exit 1 solely on inherited static caps. Static diagnostics and gate summary below are verbatim; full equivalent logs are committed in evidence/stuck-after-restart/. ```text PT_INTERP: absent; size: 20333104 bytes (baseline 16142672 + allowance 262144 = max 16404816) binary exceeds the 16404816-byte size budget: 20333104 bytes PT_INTERP: absent; size: 18112240 bytes (baseline 13555672 + allowance 262144 = max 13817816) binary exceeds the 13817816-byte size budget: 18112240 bytes +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ```
kayg commented 2026-10-13 08:11:51 +00:00
Author
Owner
Copy link

Cleanup #199: head da5321ba72 matches origin/stuck-after-restart; worktree clean. Removed this job's s199/99 build/runtime scratch (17 GB) and its isolated perf-VM directory. No s199 test service remains. Final gate and acceptance limits are recorded in the preceding result; issue remains open. No deployment or production process/session changes.

Cleanup #199: head da5321ba72b846527f72866eb9de922209cf4c9e matches origin/stuck-after-restart; worktree clean. Removed this job's s199/99 build/runtime scratch (17 GB) and its isolated perf-VM directory. No s199 test service remains. Final gate and acceptance limits are recorded in the preceding result; issue remains open. No deployment or production process/session changes.
kayg commented 2026-10-13 08:15:40 +00:00
Author
Owner
Copy link

Independent review start #199: review-stuck-after-restart at da5321ba72; merge base with local dev 93b00f5ab9. Read all 10 issue comments and related #195 history; no parent is referenced. Author actual source base is 9ab4329c. Verify explicit-close recovery and the owner’s open-but-unresponsive connection case with private fixtures. No implementation changes, pushes, deployments or perf-VM use. Local functional output is not performance evidence.

Independent review start #199: review-stuck-after-restart at da5321ba72b846527f72866eb9de922209cf4c9e; merge base with local dev 93b00f5ab965949b1a76a808e5734f98fed82fd4. Read all 10 issue comments and related #195 history; no parent is referenced. Author actual source base is 9ab4329c. Verify explicit-close recovery and the owner’s open-but-unresponsive connection case with private fixtures. No implementation changes, pushes, deployments or perf-VM use. Local functional output is not performance evidence.
kayg commented 2026-10-13 08:21:37 +00:00
Author
Owner
Copy link

P1 R199-1, independent review at da5321ba: the owner’s open-but-unresponsive connection case remains unresolved. Failing production-bundle test committed d9ca487d on review-stuck-after-restart: apps/tracer/test/restart-liveness-review.spec.ts (typing recovers when the old browser route stays open after the upstream stops). The route forwards normally before stop, retains its browser socket after upstream close, and permits healthy new routes. After typing, backend echoed_text stays before-route-stop rather than after-route-stop; routes remain 2 (the pre-stop total), stopped_routes=1, window_status=live, connected=true. No reload/focus change. Functional oracle only, not a latency claim. connection.ts sampleRoundTrip retains unanswered pings without expiring them; automatic retry begins only at socket.onclose. This is a modeled stalled route, not proof of the production incident’s exact network cause. Evidence: production screenshot in apps/tracer/test-results/restart-liveness-review-ty-0364d-en-after-the-upstream-stops/route-after-stop.png and scratch liveness-browser.log. Explicit-close author tests cannot exercise this state. Real-engine corroboration pending.

P1 R199-1, independent review at da5321ba: the owner’s open-but-unresponsive connection case remains unresolved. Failing production-bundle test committed d9ca487d on review-stuck-after-restart: apps/tracer/test/restart-liveness-review.spec.ts (typing recovers when the old browser route stays open after the upstream stops). The route forwards normally before stop, retains its browser socket after upstream close, and permits healthy new routes. After typing, backend echoed_text stays before-route-stop rather than after-route-stop; routes remain 2 (the pre-stop total), stopped_routes=1, window_status=live, connected=true. No reload/focus change. Functional oracle only, not a latency claim. connection.ts sampleRoundTrip retains unanswered pings without expiring them; automatic retry begins only at socket.onclose. This is a modeled stalled route, not proof of the production incident’s exact network cause. Evidence: production screenshot in apps/tracer/test-results/restart-liveness-review-ty-0364d-en-after-the-upstream-stops/route-after-stop.png and scratch liveness-browser.log. Explicit-close author tests cannot exercise this state. Real-engine corroboration pending.
kayg commented 2026-10-13 08:29:43 +00:00
Author
Owner
Copy link

Independent review continuation #199: review-stuck-after-restart at d9ca487dd8, target da5321ba72, merge base 93b00f5ab9. Preserved earlier P1 R199-1 and repeated the production-bundle failing test: no replacement route, window remains live. Extending the existing real systemd/PTy restart fixture to corroborate the retained-browser-route case. Native gate first resumed invocation hit fixture AF_UNIX pathname limit (121 bytes); rerun will use a shorter job-owned runtime root. No implementation changes, push, deploy or perf VM; local checks are functional only.

Independent review continuation #199: review-stuck-after-restart at d9ca487dd825c0f77fe7c3a8fdc84f525f703595, target da5321ba72b846527f72866eb9de922209cf4c9e, merge base 93b00f5ab965949b1a76a808e5734f98fed82fd4. Preserved earlier P1 R199-1 and repeated the production-bundle failing test: no replacement route, window remains live. Extending the existing real systemd/PTy restart fixture to corroborate the retained-browser-route case. Native gate first resumed invocation hit fixture AF_UNIX pathname limit (121 bytes); rerun will use a shorter job-owned runtime root. No implementation changes, push, deploy or perf VM; local checks are functional only.
kayg commented 2026-10-13 08:31:45 +00:00
Author
Owner
Copy link

P1 R199-1 corroborated against the real server (same finding, no additional count): apps/tracer/real/restart-liveness-review.spec.ts reuses systemd KillMode=process, production browser bundle, real passkey/CA/SQLite/daemon/PTy. Initial before-42 executes. Server stops and restarts successfully; an intermediary retains the browser socket, healthy new routes remain available. Post-restart typing never produces recovered-64. Functional assertion fails; window remains live, connected=true, no replacement route. This models intermediary socket retention, not the exact production network cause. No focus change/reload, implementation fix, push, deployment or perf VM. Raw evidence real-liveness-short.log under job scratch; commit on review-stuck-after-restart recorded next.

P1 R199-1 corroborated against the real server (same finding, no additional count): apps/tracer/real/restart-liveness-review.spec.ts reuses systemd KillMode=process, production browser bundle, real passkey/CA/SQLite/daemon/PTy. Initial before-42 executes. Server stops and restarts successfully; an intermediary retains the browser socket, healthy new routes remain available. Post-restart typing never produces recovered-64. Functional assertion fails; window remains live, connected=true, no replacement route. This models intermediary socket retention, not the exact production network cause. No focus change/reload, implementation fix, push, deployment or perf VM. Raw evidence real-liveness-short.log under job scratch; commit on review-stuck-after-restart recorded next.
kayg commented 2026-10-13 08:40:29 +00:00
Author
Owner
Copy link

Independent review step #199: real retained-route regression committed 50321514a2; strict optional close-event adapter correction committed 19e958cf. Real regression reproduces after correction: routes=3, superseded_routes=3, stopped_routes=1, window_status=live, connected=true; recovered-64 never executes after successful systemd restart. Existing real explicit-close cases pass (2: CA locked/unlocked). Tracer and explicit real-fixture strict TypeScript checks pass; focused transport tests 3 pass / 13 assertions. One P1 total, R199-1. Native fast gate still completing; corrected short TMPDIR avoids fixture pathname limits. Evidence will be committed under evidence/review-stuck-after-restart. No push/deploy/perf VM.

Independent review step #199: real retained-route regression committed 50321514a26e1ea704a34543daca1e71950b16bd; strict optional close-event adapter correction committed 19e958cf. Real regression reproduces after correction: routes=3, superseded_routes=3, stopped_routes=1, window_status=live, connected=true; recovered-64 never executes after successful systemd restart. Existing real explicit-close cases pass (2: CA locked/unlocked). Tracer and explicit real-fixture strict TypeScript checks pass; focused transport tests 3 pass / 13 assertions. One P1 total, R199-1. Native fast gate still completing; corrected short TMPDIR avoids fixture pathname limits. Evidence will be committed under evidence/review-stuck-after-restart. No push/deploy/perf VM.
kayg commented 2026-10-13 08:47:22 +00:00
Author
Owner
Copy link

Independent verification verdict #199: HOLD — merge after fixes. Counts P0=0, P1=1, P2=0, P3=0. Review branch review-stuck-after-restart, head 70dcd58939; target stuck-after-restart at da5321ba72; merge base with local dev 93b00f5ab9.

P1 R199-1: an intermediary retaining the browser socket after upstream stop leaves typing unable to trigger recovery, with the window still live. Committed failing tests: apps/tracer/test/restart-liveness-review.spec.ts (d9ca487d) and apps/tracer/real/restart-liveness-review.spec.ts (50321514, adapter types corrected 19e958cf). Both production-bundle oracles reproduce; real case uses systemd KillMode=process, passkey/CA/SQLite/daemon/surviving PTY, checks successful server restart, and never executes recovered-64. Latest real repetition: routes=3, superseded_routes=3, stopped_routes=1, status=live, connected=true. The modeled intermediary does not identify the exact production network cause.

Passing controls: 2 real explicit-close restart cases (CA locked/unlocked), 3 focused transport tests / 13 assertions; tracer and explicit real-fixture strict TypeScript pass. Native tests 740 passed, 0 failed, 78 configured ignores. Final scripts/verify fast exit1 only for inherited static caps. The review branch changes no application/native/build source or dependencies; binary delta0. The Rust gate stops before TS on cap failure, so TS was checked separately. Short HOME-cache TMPDIR/runtime roots avoid initial fixture pathname mistakes; no assertions changed.

Evidence/report/raw logs and capture: evidence/review-stuck-after-restart/README.md at 70dcd58939. No implementation repair, push, deployment, production access, or perf-VM use. No independent performance claim. Original tab acceptance remains unverified. Issue stays open.

Verbatim final static diagnostics and gate summary:

PT_INTERP: absent; size: 20333104 bytes (baseline 16142672 + allowance 262144 = max 16404816)
binary exceeds the 16404816-byte size budget: 20333104 bytes
PT_INTERP: absent; size: 18112240 bytes (baseline 13555672 + allowance 262144 = max 13817816)
binary exceeds the 13817816-byte size budget: 18112240 bytes

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+
Independent verification verdict #199: HOLD — merge after fixes. Counts P0=0, P1=1, P2=0, P3=0. Review branch review-stuck-after-restart, head 70dcd58939468869e5ca1de555ab465d4eaf113f; target stuck-after-restart at da5321ba72b846527f72866eb9de922209cf4c9e; merge base with local dev 93b00f5ab965949b1a76a808e5734f98fed82fd4. P1 R199-1: an intermediary retaining the browser socket after upstream stop leaves typing unable to trigger recovery, with the window still live. Committed failing tests: apps/tracer/test/restart-liveness-review.spec.ts (d9ca487d) and apps/tracer/real/restart-liveness-review.spec.ts (50321514, adapter types corrected 19e958cf). Both production-bundle oracles reproduce; real case uses systemd KillMode=process, passkey/CA/SQLite/daemon/surviving PTY, checks successful server restart, and never executes recovered-64. Latest real repetition: routes=3, superseded_routes=3, stopped_routes=1, status=live, connected=true. The modeled intermediary does not identify the exact production network cause. Passing controls: 2 real explicit-close restart cases (CA locked/unlocked), 3 focused transport tests / 13 assertions; tracer and explicit real-fixture strict TypeScript pass. Native tests 740 passed, 0 failed, 78 configured ignores. Final scripts/verify fast exit1 only for inherited static caps. The review branch changes no application/native/build source or dependencies; binary delta0. The Rust gate stops before TS on cap failure, so TS was checked separately. Short HOME-cache TMPDIR/runtime roots avoid initial fixture pathname mistakes; no assertions changed. Evidence/report/raw logs and capture: evidence/review-stuck-after-restart/README.md at 70dcd58939468869e5ca1de555ab465d4eaf113f. No implementation repair, push, deployment, production access, or perf-VM use. No independent performance claim. Original tab acceptance remains unverified. Issue stays open. Verbatim final static diagnostics and gate summary: ```text PT_INTERP: absent; size: 20333104 bytes (baseline 16142672 + allowance 262144 = max 16404816) binary exceeds the 16404816-byte size budget: 20333104 bytes PT_INTERP: absent; size: 18112240 bytes (baseline 13555672 + allowance 262144 = max 13817816) binary exceeds the 13817816-byte size budget: 18112240 bytes +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ```
kayg commented 2026-10-13 08:49:47 +00:00
Author
Owner
Copy link

Continuation r2 start/blocker: branch stuck-after-restart, head da5321ba72, production source base 9ab4329c. Read CLAUDE.md, DECISIONS.md, docs/ARCHITECTURE.md, docs/PROCESS.md and every #199 comment; no parent referenced. R199-1 remains: unanswered client pings never expire and reconnect starts only on socket.onclose (apps/tracer/src/connection.ts). Fetching review-stuck-after-restart/dev to port the committed regressions and read D265 failed: git@10.69.69.52: Permission denied (publickey); fatal: Could not read from remote repository. Stopped under the explicit auth-error rule; no credential inspection, code changes, commits, pushes, deployment or production access. Owner/orchestrator must restore existing Git access before continuation. No tests or scripts/verify fast run in this blocked continuation.

Continuation r2 start/blocker: branch stuck-after-restart, head da5321ba72b846527f72866eb9de922209cf4c9e, production source base 9ab4329c. Read CLAUDE.md, DECISIONS.md, docs/ARCHITECTURE.md, docs/PROCESS.md and every #199 comment; no parent referenced. R199-1 remains: unanswered client pings never expire and reconnect starts only on socket.onclose (apps/tracer/src/connection.ts). Fetching review-stuck-after-restart/dev to port the committed regressions and read D265 failed: git@10.69.69.52: Permission denied (publickey); fatal: Could not read from remote repository. Stopped under the explicit auth-error rule; no credential inspection, code changes, commits, pushes, deployment or production access. Owner/orchestrator must restore existing Git access before continuation. No tests or scripts/verify fast run in this blocked continuation.
kayg commented 2026-10-13 08:52:17 +00:00
Author
Owner
Copy link

Continuation r3 start: stuck-after-restart at da5321ba, source base 9ab4329c. Read every #199 comment; no parent referenced. Git auth outage respected: no fetch/push. Ported local review regressions as b81af837, 475dc478, ffba73dc; no review evidence files imported (D262). Reproduce open browser routes and both transport directions, then implement D265 server-driven liveness and retain input. Private fixtures only; no production access. Push pending.

Continuation r3 start: stuck-after-restart at da5321ba, source base 9ab4329c. Read every #199 comment; no parent referenced. Git auth outage respected: no fetch/push. Ported local review regressions as b81af837, 475dc478, ffba73dc; no review evidence files imported (D262). Reproduce open browser routes and both transport directions, then implement D265 server-driven liveness and retain input. Private fixtures only; no production access. Push pending.
kayg commented 2026-10-13 08:55:23 +00:00
Author
Owner
Copy link

R199-1 reproduced on ffba73dc: imported production-browser mock test fails; backend echoed_text=before-route-stop, expected after-route-stop; retained socket leaves window live. Raw mock-before.log retained outside repo. Additive contract proposal for D265: reuse existing Ping/Pong on control and terminal streams, server-driven probes on both transport legs; terminal-stream replies must cross the shim's existing InputStateRequest fence. No wire tags, engine, dependency or resume-token transport change. Missing replies invalidate readiness without waiting for a browser close event; unacknowledged input keeps its durable writer. This intentionally supersedes D81 idle-ping prohibition as D265 requires liveness; measure cost at 72 sessions on the pinned VM. No production access.

R199-1 reproduced on ffba73dc: imported production-browser mock test fails; backend echoed_text=before-route-stop, expected after-route-stop; retained socket leaves window live. Raw mock-before.log retained outside repo. Additive contract proposal for D265: reuse existing Ping/Pong on control and terminal streams, server-driven probes on both transport legs; terminal-stream replies must cross the shim's existing InputStateRequest fence. No wire tags, engine, dependency or resume-token transport change. Missing replies invalidate readiness without waiting for a browser close event; unacknowledged input keeps its durable writer. This intentionally supersedes D81 idle-ping prohibition as D265 requires liveness; measure cost at 72 sessions on the pinned VM. No production access.
kayg commented 2026-10-13 09:08:15 +00:00
Author
Owner
Copy link

Completed browser module 54dd397d: route leases expire independently of WebSocket close events; resume/input cursors are retained, late superseded events ignored. Imported mock regression fails before (before-route-stop only, live=true) and passes after. Real production-browser/systemd KillMode=process/PTy review regression also fails before (recovered-64 never executes) and passes after with queued waiting-49 and later commands; no reload/focus change. Four deterministic transport tests/20 assertions, strict tracer TypeScript, fmt/diff pass. Raw logs remain in s199r3 HOME-cache scratch, no evidence committed (D262). Next: server-driven browser/daemon/session probes, both direction failures, scale measurements. Push pending; no production access.

Completed browser module 54dd397d: route leases expire independently of WebSocket close events; resume/input cursors are retained, late superseded events ignored. Imported mock regression fails before (before-route-stop only, live=true) and passes after. Real production-browser/systemd KillMode=process/PTy review regression also fails before (recovered-64 never executes) and passes after with queued waiting-49 and later commands; no reload/focus change. Four deterministic transport tests/20 assertions, strict tracer TypeScript, fmt/diff pass. Raw logs remain in s199r3 HOME-cache scratch, no evidence committed (D262). Next: server-driven browser/daemon/session probes, both direction failures, scale measurements. Push pending; no production access.
kayg commented 2026-10-13 09:22:26 +00:00
Author
Owner
Copy link

Completed infraternal daemon module 49fd1e26: terminal probes reply only after the existing selected-writer InputStateRequest fence, preserving native stream identity; one fence in flight, no wire tags/dependencies/shim changes. Gate: 44 agent unit tests +2 transport tests pass, 7 configured ignores; fmt/all-target clippy pass. Added a held-fence assertion proving no premature Pong. Shared upstream actor regression now has a valid red oracle: peer remains open, attach succeeds, inventory never becomes Unavailable without EOF (3-second functional timeout, not performance evidence). A first harness attempt omitted Hello and used an odd backend stream; corrected before counting evidence. Browser mock and real retained-route red/green already recorded. Server module and 72-session VM checks in progress. No production access. Push pending.

Completed infraternal daemon module 49fd1e26: terminal probes reply only after the existing selected-writer InputStateRequest fence, preserving native stream identity; one fence in flight, no wire tags/dependencies/shim changes. Gate: 44 agent unit tests +2 transport tests pass, 7 configured ignores; fmt/all-target clippy pass. Added a held-fence assertion proving no premature Pong. Shared upstream actor regression now has a valid red oracle: peer remains open, attach succeeds, inventory never becomes Unavailable without EOF (3-second functional timeout, not performance evidence). A first harness attempt omitted Hello and used an odd backend stream; corrected before counting evidence. Browser mock and real retained-route red/green already recorded. Server module and 72-session VM checks in progress. No production access. Push pending.
kayg commented 2026-10-13 09:49:02 +00:00
Author
Owner
Copy link

Completed server module 11e4b0e2 and browser counterpart (head recorded below): server Ping monitors browser, shared daemon and shim-fenced session routes; a stalled session reattaches on the existing browser socket with the same writer. Shared daemon replacement does not wait for graceful child reaping. No new wire tags, dependencies or engines. Real test head passes all 6 cases: retained route CA locked/unlocked, stalled daemon, stalled shim, ordinary closes locked/unlocked; production screenshots in private scratch. Three mock direction cases pass and queued text appears once. Native server suite: 326 passed, 2 test adapters initially assumed no server Ping, 39 ignored; independent responder adapters preserve the original renewal assertions and both now pass. Strict native clippy/fmt and TS pass. Functional host profile, no local performance claim. A short private HOME-cache root avoids inherited AF_UNIX test pathname limits. Next: VM at 72 attached PTYs, true TCP/TLS delay, static size and final fast gate. Push pending; no production access.
25e962aeac

Completed server module 11e4b0e2 and browser counterpart (head recorded below): server Ping monitors browser, shared daemon and shim-fenced session routes; a stalled session reattaches on the existing browser socket with the same writer. Shared daemon replacement does not wait for graceful child reaping. No new wire tags, dependencies or engines. Real test head passes all 6 cases: retained route CA locked/unlocked, stalled daemon, stalled shim, ordinary closes locked/unlocked; production screenshots in private scratch. Three mock direction cases pass and queued text appears once. Native server suite: 326 passed, 2 test adapters initially assumed no server Ping, 39 ignored; independent responder adapters preserve the original renewal assertions and both now pass. Strict native clippy/fmt and TS pass. Functional host profile, no local performance claim. A short private HOME-cache root avoids inherited AF_UNIX test pathname limits. Next: VM at 72 attached PTYs, true TCP/TLS delay, static size and final fast gate. Push pending; no production access. 25e962aeacbdfe3b3209cf3ddbeb1345c7ce0ed8
kayg commented 2026-10-13 09:58:33 +00:00
Author
Owner
Copy link

Author-review finding/fix: a parked shim data link retained its outstanding probe and suppressed future probes after relink. The extended shim-fence test fails before (replacement probe timeout) and passes after clearing probe state on each data-link drop. Completed daemon refinement; head appended below. Agent 44 unit +2 transport tests pass, 7 ignored; fmt/all-target clippy pass. CPU load on the shared build host also exposed a first-startup browser probe race: initial unpainted windows are not yet live; monitoring previously confirmed windows must remain continuous across replay. Refining startup scope and retaining the former 2-second latency-pill update cadence before final VM checks. No production access. Push pending.
cbbb580ffb

Author-review finding/fix: a parked shim data link retained its outstanding probe and suppressed future probes after relink. The extended shim-fence test fails before (replacement probe timeout) and passes after clearing probe state on each data-link drop. Completed daemon refinement; head appended below. Agent 44 unit +2 transport tests pass, 7 ignored; fmt/all-target clippy pass. CPU load on the shared build host also exposed a first-startup browser probe race: initial unpainted windows are not yet live; monitoring previously confirmed windows must remain continuous across replay. Refining startup scope and retaining the former 2-second latency-pill update cadence before final VM checks. No production access. Push pending. cbbb580ffbba4b6389943cc9f47135461d880528
kayg commented 2026-10-13 11:20:28 +00:00
Author
Owner
Copy link

Milestone head a6b346b5dd, local only/push pending. Completed atomic commits: 7619c5b2 probe phase/explicit retirement, b890d394 directional/mock probes and safe reporter, fa353cf3 private pidfd pause, eefd61f0 scale/per-window regressions, 7877b111 perf/RSS/CPU, a6b346b5 contract following proposal. Phase regression red/green; explicit close/error retires readiness without awaiting close. 4 transport unit cases/23 assertions and strict TS pass. Mock both/up/down + wrapped-ring + same-writer transient input: 5 pass, original assertions. Real retained/closed restart cases CA locked/unlocked and daemon/shim pauses pass, including three sessions; per-window readiness permits healthy siblings to stay live. Parked-fence red/green and 44+2 daemon tests pass. Head fast gate running. First VM GNU artifact attempt hit ENOENT (Nix loader absent); no measurement counted, waiting for static artifact for true TCP/TLS netem runs. No production access/deploy/push. Framework failure output exposed an ephemeral mock cookie; log removed, safe reporter now omits request dumps. No evidence committed. fj comment has no attachment option; text evidence will be inline and capture paths retained for handoff.

Milestone head a6b346b5dddb0ae89757b0ee1895f5e867b3d5e9, local only/push pending. Completed atomic commits: 7619c5b2 probe phase/explicit retirement, b890d394 directional/mock probes and safe reporter, fa353cf3 private pidfd pause, eefd61f0 scale/per-window regressions, 7877b111 perf/RSS/CPU, a6b346b5 contract following proposal. Phase regression red/green; explicit close/error retires readiness without awaiting close. 4 transport unit cases/23 assertions and strict TS pass. Mock both/up/down + wrapped-ring + same-writer transient input: 5 pass, original assertions. Real retained/closed restart cases CA locked/unlocked and daemon/shim pauses pass, including three sessions; per-window readiness permits healthy siblings to stay live. Parked-fence red/green and 44+2 daemon tests pass. Head fast gate running. First VM GNU artifact attempt hit ENOENT (Nix loader absent); no measurement counted, waiting for static artifact for true TCP/TLS netem runs. No production access/deploy/push. Framework failure output exposed an ephemeral mock cookie; log removed, safe reporter now omits request dumps. No evidence committed. fj comment has no attachment option; text evidence will be inline and capture paths retained for handoff.
kayg commented 2026-10-13 12:11:54 +00:00
Author
Owner
Copy link

Head d999e5cc: all six real production-browser regressions pass at three attached PTYs, including retained sockets with CA locked/unlocked and daemon/shim pauses. Corrected perf-VM TLS profile (port-scoped 150 ms RTT, 72 attached PTYs) retains the screen and delivers queued commands, but path-to-typable is 1181.873 ms against D265 fresh-connection 500 ms. Not done: tracing connection stages; no budget changed. VM 4 cores, memory 8262782976 bytes, load 0.27/0.23/0.31. Idle five-second sample: server 27.400% of one core, daemon 3.600%, shims 0.000%; server RSS 22130688, daemon 6103040, 72 shims total 297705472 bytes. Earlier all-loopback delay run is invalid because it also delayed the internal proxy leg. Production untouched; push pending (known auth outage).

Head d999e5cc: all six real production-browser regressions pass at three attached PTYs, including retained sockets with CA locked/unlocked and daemon/shim pauses. Corrected perf-VM TLS profile (port-scoped 150 ms RTT, 72 attached PTYs) retains the screen and delivers queued commands, but path-to-typable is 1181.873 ms against D265 fresh-connection 500 ms. Not done: tracing connection stages; no budget changed. VM 4 cores, memory 8262782976 bytes, load 0.27/0.23/0.31. Idle five-second sample: server 27.400% of one core, daemon 3.600%, shims 0.000%; server RSS 22130688, daemon 6103040, 72 shims total 297705472 bytes. Earlier all-loopback delay run is invalid because it also delayed the internal proxy leg. Production untouched; push pending (known auth outage).
kayg commented 2026-10-13 12:42:17 +00:00
Author
Owner
Copy link

Perf-VM diagnostics: direct real-TLS retained browser case passes at 72 attached PTYs (desktop 432.623 ms, phone 459.000 ms; RTT 150 ms, fresh budget 500 ms); paused shim 77.135 ms against 200 ms. A conservative daemon-process-spawn boundary produced 251.146 ms against 200 ms; another run automatically replayed before any window outage, so the fixture now records completed real replays instead of requiring a synthetic connecting transition. CPU burst/recovery remains under investigation: internal terminal Pongs currently run database authorization despite sending no data or gaining authority (websocket.rs output loop). Removing that internal-only authorization; revocation and terminal input/output remain gated. Earlier self-signed certificate retries and intercepted-socket TLS stages are not production recovery measurements; real TLS retention is now implemented in the private bridge. Auxiliary attachment clients explicitly model the RTT in their Pong replies while the measured browser traverses real TLS/netem; zero-RTT auxiliary clients could otherwise expire during batch setup. No production access, push pending.

Perf-VM diagnostics: direct real-TLS retained browser case passes at 72 attached PTYs (desktop 432.623 ms, phone 459.000 ms; RTT 150 ms, fresh budget 500 ms); paused shim 77.135 ms against 200 ms. A conservative daemon-process-spawn boundary produced 251.146 ms against 200 ms; another run automatically replayed before any window outage, so the fixture now records completed real replays instead of requiring a synthetic connecting transition. CPU burst/recovery remains under investigation: internal terminal Pongs currently run database authorization despite sending no data or gaining authority (websocket.rs output loop). Removing that internal-only authorization; revocation and terminal input/output remain gated. Earlier self-signed certificate retries and intercepted-socket TLS stages are not production recovery measurements; real TLS retention is now implemented in the private bridge. Auxiliary attachment clients explicitly model the RTT in their Pong replies while the measured browser traverses real TLS/netem; zero-RTT auxiliary clients could otherwise expire during batch setup. No production access, push pending.
kayg commented 2026-10-13 13:41:56 +00:00
Author
Owner
Copy link

Committed 1e56cdbf: an old Pong queued during renderer suspension could arrive before the overdue timer, replace the learned RTT with the suspension duration, and retain false readiness. Deterministic 60-second fake-clock regression fails before (expected disconnect count 3, got 2) and passes after; four transport cases, 25 assertions, check pass. Native Liveness had the equivalent late-reply renewal: added assertion fails before and passes after enforcing expiry before renewal (liveness.rs). No wall-time unit result is used as latency evidence. Final scaled perf runs before this extra guard: 72 attached PTYs, true TLS/netem; daemon 123.506/200 ms, retained phone 387.094/500 ms at RTT150, retained desktop 700.843/950 ms at RTT300, shim 76.991/200 ms; screen retained in all. Internal Pong fast path idle sample server 9.600%, daemon 3.800% of one core; server RSS 20127744, daemon 6107136, shims total 297693184 bytes. Repeating with the final guard/build. Six current real-TLS/systemd functional cases pass. Bulk socket tests failed in existing Mesh/expiry/session-gone cases under host load; all five pass when paired back-to-back with detached origin/dev 9894aff6. Do not claim a failed-same-way baseline exception: both paired runs passed. Gate head and artifact deltas still pending; no production access and no push.

Committed 1e56cdbf: an old Pong queued during renderer suspension could arrive before the overdue timer, replace the learned RTT with the suspension duration, and retain false readiness. Deterministic 60-second fake-clock regression fails before (expected disconnect count 3, got 2) and passes after; four transport cases, 25 assertions, check pass. Native Liveness had the equivalent late-reply renewal: added assertion fails before and passes after enforcing expiry before renewal (liveness.rs). No wall-time unit result is used as latency evidence. Final scaled perf runs before this extra guard: 72 attached PTYs, true TLS/netem; daemon 123.506/200 ms, retained phone 387.094/500 ms at RTT150, retained desktop 700.843/950 ms at RTT300, shim 76.991/200 ms; screen retained in all. Internal Pong fast path idle sample server 9.600%, daemon 3.800% of one core; server RSS 20127744, daemon 6107136, shims total 297693184 bytes. Repeating with the final guard/build. Six current real-TLS/systemd functional cases pass. Bulk socket tests failed in existing Mesh/expiry/session-gone cases under host load; all five pass when paired back-to-back with detached origin/dev 9894aff6. Do not claim a failed-same-way baseline exception: both paired runs passed. Gate head and artifact deltas still pending; no production access and no push.
kayg commented 2026-10-13 14:46:47 +00:00
Author
Owner
Copy link

58bbf34c: input handshaking overflow regression is red before (Input sent while unconfirmed) and green after (one buffered flush); real systemd/PTY cases six pass. Scaled real TLS shim recovery red 745.138/200 ms becomes green 77.709/200 ms. Repeated daemon replacement still shows a warm-path failure 235.554/200 ms: per-terminal expiry can retire a stream just before the supervisor automatically rebinds its surviving writer. Proposed additive contract event terminal.reconnecting, bound to the existing session_id: clear readiness and retain input/stream while the supervisor rebinds; the existing replay_complete fence confirms recovery. Prioritize detection of the shared local daemon link over its terminal route deadlines. No new library or wire tag. This implements D265 rather than widening a budget. Final 58bbf34c gate had a real-binary 24-restart oracle timeout at iteration22; previous b9b3cb9f full tests pass, unchanged Rust code. Rerunning on the final implementation; no failure exception claimed outside #118. Production untouched, push pending.

58bbf34c: input handshaking overflow regression is red before (Input sent while unconfirmed) and green after (one buffered flush); real systemd/PTY cases six pass. Scaled real TLS shim recovery red 745.138/200 ms becomes green 77.709/200 ms. Repeated daemon replacement still shows a warm-path failure 235.554/200 ms: per-terminal expiry can retire a stream just before the supervisor automatically rebinds its surviving writer. Proposed additive contract event terminal.reconnecting, bound to the existing session_id: clear readiness and retain input/stream while the supervisor rebinds; the existing replay_complete fence confirms recovery. Prioritize detection of the shared local daemon link over its terminal route deadlines. No new library or wire tag. This implements D265 rather than widening a budget. Final 58bbf34c gate had a real-binary 24-restart oracle timeout at iteration22; previous b9b3cb9f full tests pass, unchanged Rust code. Rerunning on the final implementation; no failure exception claimed outside #118. Production untouched, push pending.
kayg commented 2026-10-13 15:57:03 +00:00
Author
Owner
Copy link

dcb941d1 retains the browser stream/writer and rebinds a stalled terminal route on the server, avoiding a client round trip; be877aeb documents the session-bound terminal.reconnecting notice proposed above. Normal healthy-link deadlines restored: tighter control deadlines caused false replacement and were rejected. Six real systemd KillMode=process/PTY restart tests pass, CA locked/unlocked, including a retained TLS browser socket and paused daemon/shim. No reload/focus. Final perf VM (4 cores, 7.7 GiB, 72 attached sessions, TLS/netem): desktop RTT300 666.045/950 ms; phone RTT150 301.130/500 ms; daemon RTT150 repeated 131.078 and 93.634/200 ms; shim RTT150 77.793/200 ms. All retained their last screen. Five-second idle sample: server 9.2%, daemon 3.8%, all shims 0% at clock-tick resolution; shims 292962304 bytes aggregate, daemon 6189056 bytes, server 22056960 bytes, load 0.34/0.09/0.06. Auxiliary scale clients delay application Pongs by RTT; measured browser uses real TCP/TLS netem. be877aeb full gate: fmt/VT fmt/clippy/test/cargo-deny PASS; inherited static size caps FAIL. x86 20357232 vs production 20333104 (+24128); ARM 18131312 vs18112240 (+19072). Still investigating two legacy mock browser failures; directional mock cases and real cases pass. Production untouched; push pending.

dcb941d1 retains the browser stream/writer and rebinds a stalled terminal route on the server, avoiding a client round trip; be877aeb documents the session-bound terminal.reconnecting notice proposed above. Normal healthy-link deadlines restored: tighter control deadlines caused false replacement and were rejected. Six real systemd KillMode=process/PTY restart tests pass, CA locked/unlocked, including a retained TLS browser socket and paused daemon/shim. No reload/focus. Final perf VM (4 cores, 7.7 GiB, 72 attached sessions, TLS/netem): desktop RTT300 666.045/950 ms; phone RTT150 301.130/500 ms; daemon RTT150 repeated 131.078 and 93.634/200 ms; shim RTT150 77.793/200 ms. All retained their last screen. Five-second idle sample: server 9.2%, daemon 3.8%, all shims 0% at clock-tick resolution; shims 292962304 bytes aggregate, daemon 6189056 bytes, server 22056960 bytes, load 0.34/0.09/0.06. Auxiliary scale clients delay application Pongs by RTT; measured browser uses real TCP/TLS netem. be877aeb full gate: fmt/VT fmt/clippy/test/cargo-deny PASS; inherited static size caps FAIL. x86 20357232 vs production 20333104 (+24128); ARM 18131312 vs18112240 (+19072). Still investigating two legacy mock browser failures; directional mock cases and real cases pass. Production untouched; push pending.
kayg commented 2026-10-13 16:04:59 +00:00
Author
Owner
Copy link

Legacy mock failure identified: transient-action-vm.log records host.facts.read after Welcome followed by Close1002 unexpected test-double action; the next attach makes its exact attach_count=1/2 assertions fail. Product canvas legitimately asks for host facts; real server returns unknown-action failure, not a socket-wide close. Correcting only the mock to preserve this behavior. Snapshot/theme state equality also fails with the saved pre-continuation bundle under the same VM profile (browser-baseline-legacy-vm.log); investigating renderer-state sampling before changing that fixture. a0d18d3c removes inherited committed evidence per D262; files preserved outside the worktree. Its final gate passes all rows except the inherited static size cap.

Legacy mock failure identified: transient-action-vm.log records host.facts.read after Welcome followed by Close1002 unexpected test-double action; the next attach makes its exact attach_count=1/2 assertions fail. Product canvas legitimately asks for host facts; real server returns unknown-action failure, not a socket-wide close. Correcting only the mock to preserve this behavior. Snapshot/theme state equality also fails with the saved pre-continuation bundle under the same VM profile (browser-baseline-legacy-vm.log); investigating renderer-state sampling before changing that fixture. a0d18d3c removes inherited committed evidence per D262; files preserved outside the worktree. Its final gate passes all rows except the inherited static size cap.
kayg commented 2026-10-13 16:07:54 +00:00
Author
Owner
Copy link

a915293c corrects the unsupported-action mock refusal; e9c9a576 waits for settled scrolling before the Snapshot/theme comparison (same pattern as terminal-theme.spec.ts). Red diagnostic: viewport66.62 before,66.67 after; cursorX2, absolute cursor row257 and screen text identical; saved pre-continuation bundle also fails. Strict state equality preserved. All five original production-bundle mock browser cases now pass on the perf VM, browser-settled-final-vm.log; check passes, 11 input/transport/exit cases pass with48 assertions. No production changes in these two fixture commits. Removing the owned detached baseline exposed its compile-time manifest path in a cached protocol test binary (hello.bin missing); cleaned only that crate dev artifacts and rebuilding final gate on e9c9a576. No gate exception claimed. Author review: source inventory selection is per attached host; recovery and data remain authorization-gated; internal Pongs carry no data or authority; notice is bounded and session-bound; durable writer/cursors/remaining credit preserved; stale sockets cannot mutate replacements; input awaits replay paint; repeated failures retain backoff; no dependencies, schema, secrets or production changes. Push pending.

a915293c corrects the unsupported-action mock refusal; e9c9a576 waits for settled scrolling before the Snapshot/theme comparison (same pattern as terminal-theme.spec.ts). Red diagnostic: viewport66.62 before,66.67 after; cursorX2, absolute cursor row257 and screen text identical; saved pre-continuation bundle also fails. Strict state equality preserved. All five original production-bundle mock browser cases now pass on the perf VM, browser-settled-final-vm.log; check passes, 11 input/transport/exit cases pass with48 assertions. No production changes in these two fixture commits. Removing the owned detached baseline exposed its compile-time manifest path in a cached protocol test binary (hello.bin missing); cleaned only that crate dev artifacts and rebuilding final gate on e9c9a576. No gate exception claimed. Author review: source inventory selection is per attached host; recovery and data remain authorization-gated; internal Pongs carry no data or authority; notice is bounded and session-bound; durable writer/cursors/remaining credit preserved; stale sockets cannot mutate replacements; input awaits replay paint; repeated failures retain backoff; no dependencies, schema, secrets or production changes. Push pending.
kayg commented 2026-10-13 16:15:53 +00:00
Author
Owner
Copy link

Finish: head e9c9a576 (stuck-after-restart), push pending per r3; no fetch/push/deploy or production access.

Root cause: socket-open/last attached status was treated as readiness without continuous end-to-end proof. A healthy browser/server socket could conceal a stalled server/session route indefinitely. The reviewer retained-route mock and real PTY/systemd oracle reproduce it; saved old binary keeps live while shim input is unavailable (real-before-final-oracle-safe.log, restart-fixture.ts:203). DAEMON control activity alone is insufficient: terminal probes now require a shim InputState fence. Queued late Pongs after renderer suspension cannot renew an expired lease.

Fix: pipelined server/browser and per-terminal probes; expire stale routes; direct server rebind with the same durable writer/cursors/remaining credit; bounded session-bound terminal.reconnecting notice clears readiness; only replay paint confirms live; buffered input flushes once then awaits acknowledgment. Hello+Resume is the first WS message, never URL/header. Repeated failures retain bounded backoff and stale events cannot mutate replacement routes. No new dependencies/schema/decisions.

Evidence: six real systemd KillMode=process/PTY cases pass (CA locked/unlocked, TLS socket retained, daemon/shim paused, no reload/focus); existing-session typing works while CA locked, so unlock is not required for this recovery. Five production-bundle mock cases pass; 11 input/transport/exit unit cases,48 assertions and TypeScript check pass. Additional red/green cases cover shim input fence, late native/browser Pongs, stale batched socket events and input held until replay. Native embedded18 and actual renewal2 cases pass. Evidence files are outside the repo per D262.

Final perf VM 4 cores,8262770688 bytes RAM,72 attached sessions: desktop RTT300, CA locked,666.045/950 ms; phone393x852 RTT150, CA unlocked,301.130/500 ms; daemon RTT150,131.078 and93.634/200 ms; shim RTT150,77.793/200 ms. True TLS and port-scoped TCP netem for measured browser; auxiliary scale-client Pongs delayed by RTT. Screens retained in all. Sample load0.34/0.09/0.06; 5-second idle CPU server9.2%,daemon3.8%, all shims0% at tick resolution; RSS server22056960,daemon6189056,shims292962304 aggregate (mean4068920). Logs perf-direct-*.log under /home/kayg/.cache/codex-scratch/s199r3.

D243 delta vs9ab4329c: x86_64 musl20357232 vs20333104 =+24128 bytes; ARM musl18131312 vs18112240 =+19072 bytes. Both static, no PT_INTERP; inherited cap failure is explicit, no budget raised. Final gate table below is verbatim. Full stdout retained outside the repository.

Acceptance limit: tests keep the updated production bundle open across restarts. The owner’s live old tab and upgrade from its old JavaScript are not verified; pre-fix clients do not understand terminal.reconnecting. No production acceptance claimed. Screenshots retained under /home/kayg/.cache/codex-scratch/s199r3/captures; visual judgment remains owner/reviewer work. Perf VM job and detached baseline removed; large build targets removed after final gate.

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

Static build diagnostics, verbatim:

PT_INTERP: absent; size: 20357232 bytes (baseline 16142672 + allowance 262144 = max 16404816)
binary exceeds the 16404816-byte size budget: 20357232 bytes
PT_INTERP: absent; size: 18131312 bytes (baseline 13555672 + allowance 262144 = max 13817816)
binary exceeds the 13817816-byte size budget: 18131312 bytes

Full gate stdout: /home/kayg/.cache/codex-scratch/s199r3/verify-fast-e9c9a576.log (exit1 due static caps only). Measured/final x86 SHA256 both88f1e7a5d35002a8ff1f98f7b35eca3668707089034b30d5ebfe5480f646b95c.
Reproduce functional cases from apps/tracer with a built binary: TRACER_TEST_PORT=4299 INFRATERNAL_BIN= bun x playwright test -c playwright.real.config.ts restart.spec.ts restart-liveness-review.spec.ts --reporter=./test/safe-reporter.ts; mock cases use playwright.config.ts (restart-liveness-review.spec.ts, snapshot-resume.spec.ts, transient-session-input.spec.ts). Build tracer first. Use isolated fixture roots as in real/server.ts, never owner state. Timing cases run only on the locked perf VM via real/perf-restart.ts.

Finish: head e9c9a576 (stuck-after-restart), push pending per r3; no fetch/push/deploy or production access. Root cause: socket-open/last attached status was treated as readiness without continuous end-to-end proof. A healthy browser/server socket could conceal a stalled server/session route indefinitely. The reviewer retained-route mock and real PTY/systemd oracle reproduce it; saved old binary keeps live while shim input is unavailable (real-before-final-oracle-safe.log, restart-fixture.ts:203). DAEMON control activity alone is insufficient: terminal probes now require a shim InputState fence. Queued late Pongs after renderer suspension cannot renew an expired lease. Fix: pipelined server/browser and per-terminal probes; expire stale routes; direct server rebind with the same durable writer/cursors/remaining credit; bounded session-bound terminal.reconnecting notice clears readiness; only replay paint confirms live; buffered input flushes once then awaits acknowledgment. Hello+Resume is the first WS message, never URL/header. Repeated failures retain bounded backoff and stale events cannot mutate replacement routes. No new dependencies/schema/decisions. Evidence: six real systemd KillMode=process/PTY cases pass (CA locked/unlocked, TLS socket retained, daemon/shim paused, no reload/focus); existing-session typing works while CA locked, so unlock is not required for this recovery. Five production-bundle mock cases pass; 11 input/transport/exit unit cases,48 assertions and TypeScript check pass. Additional red/green cases cover shim input fence, late native/browser Pongs, stale batched socket events and input held until replay. Native embedded18 and actual renewal2 cases pass. Evidence files are outside the repo per D262. Final perf VM 4 cores,8262770688 bytes RAM,72 attached sessions: desktop RTT300, CA locked,666.045/950 ms; phone393x852 RTT150, CA unlocked,301.130/500 ms; daemon RTT150,131.078 and93.634/200 ms; shim RTT150,77.793/200 ms. True TLS and port-scoped TCP netem for measured browser; auxiliary scale-client Pongs delayed by RTT. Screens retained in all. Sample load0.34/0.09/0.06; 5-second idle CPU server9.2%,daemon3.8%, all shims0% at tick resolution; RSS server22056960,daemon6189056,shims292962304 aggregate (mean4068920). Logs perf-direct-*.log under /home/kayg/.cache/codex-scratch/s199r3. D243 delta vs9ab4329c: x86_64 musl20357232 vs20333104 =+24128 bytes; ARM musl18131312 vs18112240 =+19072 bytes. Both static, no PT_INTERP; inherited cap failure is explicit, no budget raised. Final gate table below is verbatim. Full stdout retained outside the repository. Acceptance limit: tests keep the updated production bundle open across restarts. The owner’s live old tab and upgrade from its old JavaScript are not verified; pre-fix clients do not understand terminal.reconnecting. No production acceptance claimed. Screenshots retained under /home/kayg/.cache/codex-scratch/s199r3/captures; visual judgment remains owner/reviewer work. Perf VM job and detached baseline removed; large build targets removed after final gate. ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ``` Static build diagnostics, verbatim: ```text PT_INTERP: absent; size: 20357232 bytes (baseline 16142672 + allowance 262144 = max 16404816) binary exceeds the 16404816-byte size budget: 20357232 bytes PT_INTERP: absent; size: 18131312 bytes (baseline 13555672 + allowance 262144 = max 13817816) binary exceeds the 13817816-byte size budget: 18131312 bytes ``` Full gate stdout: /home/kayg/.cache/codex-scratch/s199r3/verify-fast-e9c9a576.log (exit1 due static caps only). Measured/final x86 SHA256 both88f1e7a5d35002a8ff1f98f7b35eca3668707089034b30d5ebfe5480f646b95c. Reproduce functional cases from apps/tracer with a built binary: TRACER_TEST_PORT=4299 INFRATERNAL_BIN=<binary> bun x playwright test -c playwright.real.config.ts restart.spec.ts restart-liveness-review.spec.ts --reporter=./test/safe-reporter.ts; mock cases use playwright.config.ts (restart-liveness-review.spec.ts, snapshot-resume.spec.ts, transient-session-input.spec.ts). Build tracer first. Use isolated fixture roots as in real/server.ts, never owner state. Timing cases run only on the locked perf VM via real/perf-restart.ts.
kayg commented 2026-10-13 16:45:19 +00:00
Author
Owner
Copy link

Independent review continuation: target e9c9a576, merge base with dev 93b00f5ab9. Read all 32 comments on #199; no parent is referenced. Existing review branch 70dcd589 preserved and target merged locally as 2bb47684, using the target versions of the ported fixtures. Reverify R199-1 and inspect new liveness/input recovery. No implementation changes, push, deploy, production access or perf VM. Local checks are functional only.

Independent review continuation: target e9c9a576, merge base with dev 93b00f5ab965949b1a76a808e5734f98fed82fd4. Read all 32 comments on #199; no parent is referenced. Existing review branch 70dcd589 preserved and target merged locally as 2bb47684, using the target versions of the ported fixtures. Reverify R199-1 and inspect new liveness/input recovery. No implementation changes, push, deploy, production access or perf VM. Local checks are functional only.
kayg commented 2026-10-13 16:48:22 +00:00
Author
Owner
Copy link

Review scratch directory disappeared during native compilation; interrupted gate cannot be used as verification. Five production mock cases passed before deletion (both/up/down retained route, snapshot resume, transient input); logs must be regenerated. Restarting under e9-verification; no implementation changes or timing claim.

Review scratch directory disappeared during native compilation; interrupted gate cannot be used as verification. Five production mock cases passed before deletion (both/up/down retained route, snapshot resume, transient input); logs must be regenerated. Restarting under e9-verification; no implementation changes or timing claim.
kayg commented 2026-10-13 16:49:58 +00:00
Author
Owner
Copy link

Review step aba553e1: P2 R199-2 recorded as a committed failing test, apps/tracer/src/connection-correlation-review.test.ts. Reply nonce matches but sentUs differs; deterministic initial grace elapses with disconnects=0, expected1. connection.ts discards sentUs before acceptPong, contrary to wire section9 echo/correlation rule. No authority bypass demonstrated. No implementation changes. Five mock functional recovery controls pass; rebuilding native gate after scratch interruption.

Review step aba553e1: P2 R199-2 recorded as a committed failing test, apps/tracer/src/connection-correlation-review.test.ts. Reply nonce matches but sentUs differs; deterministic initial grace elapses with disconnects=0, expected1. connection.ts discards sentUs before acceptPong, contrary to wire section9 echo/correlation rule. No authority bypass demonstrated. No implementation changes. Five mock functional recovery controls pass; rebuilding native gate after scratch interruption.
kayg commented 2026-10-13 17:31:53 +00:00
Author
Owner
Copy link

Review step 6071eaf6: P2 R199-3 preserved as a failing deterministic test in apps/tracer/src/connection-startup-review.test.ts. All probes correctly echo nonce/timestamp on a simulated RTT1200ms link; initial estimate1000ms +50ms expires before the first valid response (expected disconnects0, actual1), so startup cannot learn that RTT. Wire section9 describes an unknown initial RTT, with expiry after sampling. No wall-clock performance claim or implementation change. R199-2 remains committed at aba553e1. Validation:11 existing unit cases/48 assertions pass; strict tracer check passes. Native verification has encountered private socket path limits (corrected TMPDIR) and unchanged CLI compiler/exec/startup timeout tests under host load; separate dev baseline and scoped final gate are running. Debug real-browser runs failed startup/scale controls and recovery assertions; no production-binary confirmation yet. No push/deploy/perf VM.

Review step 6071eaf6: P2 R199-3 preserved as a failing deterministic test in apps/tracer/src/connection-startup-review.test.ts. All probes correctly echo nonce/timestamp on a simulated RTT1200ms link; initial estimate1000ms +50ms expires before the first valid response (expected disconnects0, actual1), so startup cannot learn that RTT. Wire section9 describes an unknown initial RTT, with expiry after sampling. No wall-clock performance claim or implementation change. R199-2 remains committed at aba553e1. Validation:11 existing unit cases/48 assertions pass; strict tracer check passes. Native verification has encountered private socket path limits (corrected TMPDIR) and unchanged CLI compiler/exec/startup timeout tests under host load; separate dev baseline and scoped final gate are running. Debug real-browser runs failed startup/scale controls and recovery assertions; no production-binary confirmation yet. No push/deploy/perf VM.
kayg commented 2026-10-13 18:36:02 +00:00
Author
Owner
Copy link

Review step bb99c7bc: P2 R199-4 confirmed with a real WebSocket/SQLite ordering test (restart_open_liveness_review.rs). Existing attachment, sampled healthy route, every probe answered; hold only terminal.open backend reply, then release. The action succeeds, followed by socket closure rather than another server probe. websocket.rs awaits open_terminal inside receive handling, queues valid Pongs and rejects them as expired afterward. The regression fails at line220 and is committed, explicitly ignored in ordinary gates; run with --ignored. Initial invalid fixture lacked px_w/px_h; corrected valid request is the evidence. No performance measurement or implementation fix. This also provides an ordering explanation to investigate for auxiliary scale sockets closing during session creation; no claim that every loaded-host failure is this finding. P0/P1 new findings0, P2 confirmed3. Production retained-route CA-locked case passed at3 sessions; other production cases were refused extra fixture sessions during setup. Final gate daemon unit failure is another SUN_LEN fixture boundary (107-byte path with a shorter private TMPDIR is required), not an implementation regression. Repeating exact-source gate with that corrected environment. No push/deploy/perf VM.

Review step bb99c7bc: P2 R199-4 confirmed with a real WebSocket/SQLite ordering test (restart_open_liveness_review.rs). Existing attachment, sampled healthy route, every probe answered; hold only terminal.open backend reply, then release. The action succeeds, followed by socket closure rather than another server probe. websocket.rs awaits open_terminal inside receive handling, queues valid Pongs and rejects them as expired afterward. The regression fails at line220 and is committed, explicitly ignored in ordinary gates; run with --ignored. Initial invalid fixture lacked px_w/px_h; corrected valid request is the evidence. No performance measurement or implementation fix. This also provides an ordering explanation to investigate for auxiliary scale sockets closing during session creation; no claim that every loaded-host failure is this finding. P0/P1 new findings0, P2 confirmed3. Production retained-route CA-locked case passed at3 sessions; other production cases were refused extra fixture sessions during setup. Final gate daemon unit failure is another SUN_LEN fixture boundary (107-byte path with a shorter private TMPDIR is required), not an implementation regression. Repeating exact-source gate with that corrected environment. No push/deploy/perf VM.
kayg commented 2026-10-13 19:02:49 +00:00
Author
Owner
Copy link

Independent verification verdict: MERGE (no P0/P1), merge. Counts P0=0, P1=0, P2=3, P3=0. Target e9c9a576; review-stuck-after-restart head bb99c7bc53; start merge base93b00f5ab965949b1a76a808e5734f98fed82fd4.

P2 failing tests committed: R199-2 wrong Pong timestamp (aba553e1), R199-3 unsampled startup RTT ceiling (6071eaf6), R199-4 successful held terminal.open retires a responsive existing socket (bb99c7bc, explicit --ignored). Scenarios, paths and evidence are in preceding comments and findings.md; fix forward under the owner rule.

Historical P1 R199-1 resolved for the updated bundle: both retained-browser-route production-binary/systemd/PTY cases pass, CA locked/unlocked, no reload/focus assistance. All6 production cases pass at1 attached session. Three-session runs retained a passing locked retained-route case but had extra-session setup refusals/closures; all-six coverage at3 is not claimed. Original already-loaded owner tab and performance claims remain unverified; no production/perf-VM access.

Exact target source/test-set fast gate:743 passed,0 failed,78 configured ignores. Corrected short private TMPDIR, serialized tests, job-owned CPUWeight10000 scope; no assertions/deadlines changed or baseline exception claimed. Earlier path/timeout failures retained. Review tests restored afterward; workspace/all-target Clippy and strict tracer TypeScript pass. Focused controls11 pass/48 assertions; new TypeScript tests both fail, held-open Rust ordering oracle fails after successful operation. Source/dependencies unchanged by review; shipping sizes match author head, both static. Owned build/baseline/fixture scratch removed; logs/screenshots retained.

Full record: /home/kayg/.local/state/codex-jobs/infraternal/review-stuck-after-restart/result.md and findings.md. Evidence: /home/kayg/.cache/codex-scratch/stuck-after-restart-review/e9-verification; production-one-captures/. No implementation fixes, push or deploy. Issue stays open.

Verbatim scripts/verify fast summary:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

Verbatim static diagnostics:

PT_INTERP: absent; size: 20357232 bytes (baseline 16142672 + allowance 262144 = max 16404816)
binary exceeds the 16404816-byte size budget: 20357232 bytes
PT_INTERP: absent; size: 18131312 bytes (baseline 13555672 + allowance 262144 = max 13817816)
binary exceeds the 13817816-byte size budget: 18131312 bytes
Independent verification verdict: MERGE (no P0/P1), merge. Counts P0=0, P1=0, P2=3, P3=0. Target e9c9a576; review-stuck-after-restart head bb99c7bc53183c63ac98000a9b8891802ca94e55; start merge base93b00f5ab965949b1a76a808e5734f98fed82fd4. P2 failing tests committed: R199-2 wrong Pong timestamp (aba553e1), R199-3 unsampled startup RTT ceiling (6071eaf6), R199-4 successful held terminal.open retires a responsive existing socket (bb99c7bc, explicit --ignored). Scenarios, paths and evidence are in preceding comments and findings.md; fix forward under the owner rule. Historical P1 R199-1 resolved for the updated bundle: both retained-browser-route production-binary/systemd/PTY cases pass, CA locked/unlocked, no reload/focus assistance. All6 production cases pass at1 attached session. Three-session runs retained a passing locked retained-route case but had extra-session setup refusals/closures; all-six coverage at3 is not claimed. Original already-loaded owner tab and performance claims remain unverified; no production/perf-VM access. Exact target source/test-set fast gate:743 passed,0 failed,78 configured ignores. Corrected short private TMPDIR, serialized tests, job-owned CPUWeight10000 scope; no assertions/deadlines changed or baseline exception claimed. Earlier path/timeout failures retained. Review tests restored afterward; workspace/all-target Clippy and strict tracer TypeScript pass. Focused controls11 pass/48 assertions; new TypeScript tests both fail, held-open Rust ordering oracle fails after successful operation. Source/dependencies unchanged by review; shipping sizes match author head, both static. Owned build/baseline/fixture scratch removed; logs/screenshots retained. Full record: /home/kayg/.local/state/codex-jobs/infraternal/review-stuck-after-restart/result.md and findings.md. Evidence: /home/kayg/.cache/codex-scratch/stuck-after-restart-review/e9-verification; production-one-captures/. No implementation fixes, push or deploy. Issue stays open. Verbatim scripts/verify fast summary: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ``` Verbatim static diagnostics: ```text PT_INTERP: absent; size: 20357232 bytes (baseline 16142672 + allowance 262144 = max 16404816) binary exceeds the 16404816-byte size budget: 20357232 bytes PT_INTERP: absent; size: 18131312 bytes (baseline 13555672 + allowance 262144 = max 13817816) binary exceeds the 13817816-byte size budget: 18131312 bytes ```
kayg commented 2026-10-13 19:05:48 +00:00
Author
Owner
Copy link

Continuation r5 start: stuck-after-restart at e9c9a576, production base 9ab4329c. Read all #199 comments; no parent referenced. Ported local review tests a9d2fe58/f0946cfa/f5fc0afa. Fix only three P2 findings and add all-six three-session restart coverage; final perf-VM rerun and fast gate. Offline Git, push pending; no production access. This release branch has no verify iterate mode; use equivalent focused fmt/clippy/TypeScript/touched-test gates without changing verification tooling.

Continuation r5 start: stuck-after-restart at e9c9a576, production base 9ab4329c. Read all #199 comments; no parent referenced. Ported local review tests a9d2fe58/f0946cfa/f5fc0afa. Fix only three P2 findings and add all-six three-session restart coverage; final perf-VM rerun and fast gate. Offline Git, push pending; no production access. This release branch has no verify iterate mode; use equivalent focused fmt/clippy/TypeScript/touched-test gates without changing verification tooling.
kayg commented 2026-10-13 19:11:15 +00:00
Author
Owner
Copy link

Completed browser fixes 7b1ac159 and 94d2ea36: R199-2 exact timestamp correlation; R199-3 explicit unsampled state with bounded 2s grace (native equivalent), then learned RTT+50ms. Both ported tests fail before, all four focused cases pass after (22 assertions), strict TypeScript/fmt pass. Correlation oracle advances to 2100ms to exceed the new unsampled grace; suspension oracle now echoes the captured correct timestamp, preserving its late-valid-reply assertion. Red/green logs under s199r5 outside repo. Native held-open regression still compiling under shared host load. No production access; push pending.

Completed browser fixes 7b1ac159 and 94d2ea36: R199-2 exact timestamp correlation; R199-3 explicit unsampled state with bounded 2s grace (native equivalent), then learned RTT+50ms. Both ported tests fail before, all four focused cases pass after (22 assertions), strict TypeScript/fmt pass. Correlation oracle advances to 2100ms to exceed the new unsampled grace; suspension oracle now echoes the captured correct timestamp, preserving its late-valid-reply assertion. Red/green logs under s199r5 outside repo. Native held-open regression still compiling under shared host load. No production access; push pending.
kayg commented 2026-10-13 19:19:57 +00:00
Author
Owner
Copy link

R199-4 reproduced with the unchanged native binary in a CPUWeight=10000 scope: held terminal.open succeeds, then next observation is socket closure (restart_open_liveness_review.rs:220). Initial unscoped run instead lost the mock backend under load; not counted as the intended oracle. websocket.rs awaited open_terminal inside inbound handling, preventing consumption of correctly echoed Pongs. Minimal repair reuses the existing four-call FuturesUnordered queue, preserves authority cancellation and result/refusal formats; normal regression gate enabled. Red evidence server-red-scoped.log outside repo; this is functional ordering, not a host latency claim.

R199-4 reproduced with the unchanged native binary in a CPUWeight=10000 scope: held terminal.open succeeds, then next observation is socket closure (restart_open_liveness_review.rs:220). Initial unscoped run instead lost the mock backend under load; not counted as the intended oracle. websocket.rs awaited open_terminal inside inbound handling, preventing consumption of correctly echoed Pongs. Minimal repair reuses the existing four-call FuturesUnordered queue, preserves authority cancellation and result/refusal formats; normal regression gate enabled. Red evidence server-red-scoped.log outside repo; this is functional ordering, not a host latency claim.
kayg commented 2026-10-13 19:42:18 +00:00
Author
Owner
Copy link

Broader pre-commit server gate: clippy PASS; 326 unit tests passed, 3 failed, 39 ignored. One failure was setup-token Unix socket pathname length with the first TMPDIR (corrected to private scratch/r5). Other two were existing paused-clock auth timeout assertions: review3_busy_recovery:285 (30s virtual timeout), review4_open_reply_and_revocation:920 (10s virtual timeout). Both pass unchanged in isolated scopes. Detached origin/dev 804ac0e8 review3_busy_recovery also passes, so no failed-baseline exception is claimed. Repeat full server unit gate on shorter root running. New held-open regression passes both focused and broader runs; no deadlines/assertions weakened.

Broader pre-commit server gate: clippy PASS; 326 unit tests passed, 3 failed, 39 ignored. One failure was setup-token Unix socket pathname length with the first TMPDIR (corrected to private scratch/r5). Other two were existing paused-clock auth timeout assertions: review3_busy_recovery:285 (30s virtual timeout), review4_open_reply_and_revocation:920 (10s virtual timeout). Both pass unchanged in isolated scopes. Detached origin/dev 804ac0e8 review3_busy_recovery also passes, so no failed-baseline exception is claimed. Repeat full server unit gate on shorter root running. New held-open regression passes both focused and broader runs; no deadlines/assertions weakened.
kayg commented 2026-10-13 20:03:38 +00:00
Author
Owner
Copy link

R199-4 refinement: original 350ms held-open oracle fails before (successful open then socket closure) and passes focused after; it hit healthy-route scheduling expiry in a full host-loaded suite. Replace timed hold with a stronger receive-dispatch barrier: server must answer a client Ping before open release, then succeed and send another probe. Old e9c9a576 fails this barrier (server-barrier-before.log); fixed source passes. Reuse shared FrozenSocketTime (#118) during real socket/SQLite I/O; advance one final 25ms probe tick explicitly, without changing production deadlines. Three visible-window production/systemd suite now passes all 6 cases on desktop, both CA states, same-page recovery, queued-input identity and sibling isolation (three-current.log). Phone 1-window perf profile retained. Fixture fixes select canvas input, identify sessions by computed markers, and derive private CA port from TRACER_TEST_PORT; no UI/product changes. First VM run refused an auxiliary open during setup, so no valid measurement; subsequent final desktop at72 attached sessions passes 664.173/950ms, RTT300, bitmap retained. Remaining VM cases and final gates running.

R199-4 refinement: original 350ms held-open oracle fails before (successful open then socket closure) and passes focused after; it hit healthy-route scheduling expiry in a full host-loaded suite. Replace timed hold with a stronger receive-dispatch barrier: server must answer a client Ping before open release, then succeed and send another probe. Old e9c9a576 fails this barrier (server-barrier-before.log); fixed source passes. Reuse shared FrozenSocketTime (#118) during real socket/SQLite I/O; advance one final 25ms probe tick explicitly, without changing production deadlines. Three visible-window production/systemd suite now passes all 6 cases on desktop, both CA states, same-page recovery, queued-input identity and sibling isolation (three-current.log). Phone 1-window perf profile retained. Fixture fixes select canvas input, identify sessions by computed markers, and derive private CA port from TRACER_TEST_PORT; no UI/product changes. First VM run refused an auxiliary open during setup, so no valid measurement; subsequent final desktop at72 attached sessions passes 664.173/950ms, RTT300, bitmap retained. Remaining VM cases and final gates running.
kayg commented 2026-10-13 20:13:34 +00:00
Author
Owner
Copy link

Completed steps: browser timestamp 7b1ac159, unknown startup RTT 94d2ea36, server pending-open b16956a8, three visible-window tests 09c3237d (head). Server fmt/all-target Clippy and 329 unit +23 integration tests pass, 39 unit/1 integration ignores. All6 desktop three-window systemd/PTY production-bundle cases pass; strict tracer and fixture TS pass. Final VM72 attached PTYs: desktop RTT300 664.173/950ms; phone RTT150 326.748/500ms; daemon RTT150 134.732/200ms; shim RTT150 77.887/200ms. All bitmap retained. VM4 CPUs/8262770688B RAM; measured loads 2.29–3.47 (five-second idle samples). RSS about21.6–22.1MB server,6.2MB daemon,298.0–298.1MB shims aggregate (about4.14MB per shim ×72). Idle samples server9.0–14.8%,daemon3.8–5.8%,shims0–0.2% of one CPU; scenario measurements, not p99 or a statistical regression comparison. First scaled setup refusal remains recorded as invalid measurement, not counted as a pass. x86 shipping size20362128B vs9ab4329c20333104B =+29024B; round delta+4896B vs e9c9a576. Final head fast gate and ARM size pending. No production access/deploy; offline Git, push pending.

Completed steps: browser timestamp 7b1ac159, unknown startup RTT 94d2ea36, server pending-open b16956a8, three visible-window tests 09c3237d (head). Server fmt/all-target Clippy and 329 unit +23 integration tests pass, 39 unit/1 integration ignores. All6 desktop three-window systemd/PTY production-bundle cases pass; strict tracer and fixture TS pass. Final VM72 attached PTYs: desktop RTT300 664.173/950ms; phone RTT150 326.748/500ms; daemon RTT150 134.732/200ms; shim RTT150 77.887/200ms. All bitmap retained. VM4 CPUs/8262770688B RAM; measured loads 2.29–3.47 (five-second idle samples). RSS about21.6–22.1MB server,6.2MB daemon,298.0–298.1MB shims aggregate (about4.14MB per shim ×72). Idle samples server9.0–14.8%,daemon3.8–5.8%,shims0–0.2% of one CPU; scenario measurements, not p99 or a statistical regression comparison. First scaled setup refusal remains recorded as invalid measurement, not counted as a pass. x86 shipping size20362128B vs9ab4329c20333104B =+29024B; round delta+4896B vs e9c9a576. Final head fast gate and ARM size pending. No production access/deploy; offline Git, push pending.
kayg commented 2026-10-13 20:50:21 +00:00
Author
Owner
Copy link

Finish r5: stuck-after-restart, head 09c3237d36. Seven local atomic commits from e9c9a576; push pending per offline r3. No deployment or production access; original owner tab remains untouched/unverified. Issue stays open.

Fixed the three P2s: exact nonce/timestamp correlation (7b1ac159); explicitly unknown startup RTT with bounded native-equivalent 2s grace, then measured RTT+50ms (94d2ea36); terminal opens polled in the existing bounded call queue so probes/input/revocation continue (b16956a8). Deadlines, protocol, dependencies and engines unchanged. Original root fault was readiness without continuous end-to-end route proof; prior D265 recovery remains intact.

Evidence: both imported browser regressions fail before and pass after (4 focused cases/22 assertions). Original held-open regression fails before with successful operation then closure; strengthened control-before-release barrier also fails on e9c9a576 and passes fixed source. Shared FrozenSocketTime prevents host scheduling from consuming the ordering oracle’s lease; no production deadline widened. Server module gates: Clippy,329 unit +23 integration pass. Final full native gate:744 passed,0 failed,78 configured ignores. Earlier auth timeouts pass isolated and on repeated suite; dev804ac0e8 control passes, so no failed-baseline exception claimed. First full gate’s unchanged CLI queue test refused its initial PTY (resource_unavailable); it passes unchanged isolated and in final full gate.

09c3237d adds all6 real production-browser/systemd KillMode=process/PTY cases at three visible desktop windows: CA locked/unlocked, retained/closed socket, daemon/shim stalls. Computed markers establish session identity, queued-input delivery and sibling isolation; no reload/focus recovery. All6 pass; strict tracer/fixture TypeScript pass separately (fast stops before TS on inherited size caps). Existing phone profile remains 393x852; no three-window phone claim. Screenshot captures are outside repo at /home/kayg/.cache/codex-scratch/s199r5/captures (four PNGs); visual judgment remains owner/reviewer work.

Final pinned VM:4 CPUs,8262770688B RAM,72 attached PTYs,true TLS/port-scoped TCP netem. Desktop RTT300:664.173/950ms; phone RTT150:326.748/500ms; daemon RTT150:134.732/200ms; shim RTT150:77.887/200ms. All retain last bitmap. Auxiliary Pongs model RTT; measured browser traverses actual netem. Point scenarios, not p99. First setup refusal is retained as an invalid measurement. RSS about21.6–22.1MB server,6.2MB daemon,298.0–298.1MB shims (about4.14MB ×72). Five-second idle samples server9.0–14.8%,daemon3.8–5.8%,shims0–0.2% of one CPU; sample loads2.29–3.47. No new idle work added; no statistical CPU regression comparison claimed.

D243 shipping deltas vs recorded9ab4329c: x8620362128−20333104=+29024B; ARM18135776−18112240=+23536B. Round deltas vs e9c9a576:+4896/+4464B for the pending-open liveness repair; no dependencies/profile changes. Both static/no PT_INTERP. Final x86 SHA256 a4ec163b2fc0ecb00f97f57a28a484571aa522593801cb1366f3f3334ce0f695 matches measured artifact. Only inherited static caps fail; no threshold altered.

Self review: pair correlation before renewal; bounded unsampled grace; authorization at dispatch/during wait/before reply and SQL mutation; bounded queue/cancellation; same writer/cursors and per-window replay proof preserved; resume token remains first WS message, never header/URL; no new wire/schema/library/UI components. Evidence/logs and final shipping artifacts remain under /home/kayg/.cache/codex-scratch/s199r5. Temporary baseline/build targets/runtime roots and VM job removed. Worktree clean. scripts/verify iterate is absent on this release branch; focused per-commit gates used, final fast run on head.

Final scripts/verify fast exit1. Verbatim summary:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

Verbatim static diagnostics:

PT_INTERP: absent; size: 20362128 bytes (baseline 16142672 + allowance 262144 = max 16404816)
binary exceeds the 16404816-byte size budget: 20362128 bytes
PT_INTERP: absent; size: 18135776 bytes (baseline 13555672 + allowance 262144 = max 13817816)
binary exceeds the 13817816-byte size budget: 18135776 bytes

Full stdout: /home/kayg/.cache/codex-scratch/s199r5/verify-fast-final.log. Red/green: browser-red.log/browser-green.log, server-red-scoped.log/server-barrier-before.log/server-frozen-green.log. Functional: three-current.log; perf: perf-desktop-diagnostic.log,perf-phone.log,perf-daemon.log,perf-shim.log. Reproduce browser cases from apps/tracer with production dist and built binary: TRACER_TEST_PORT=4295 INFRATERNAL_BIN= bun x playwright test -c playwright.real.config.ts restart-three-session.spec.ts --reporter=./test/safe-reporter.ts. Timing harness real/perf-restart.ts requires isolated VM driver/namespaces and lock as documented in prior issue steps.

Finish r5: stuck-after-restart, head 09c3237d362104b7821f711365a49af170715ddd. Seven local atomic commits from e9c9a576; push pending per offline r3. No deployment or production access; original owner tab remains untouched/unverified. Issue stays open. Fixed the three P2s: exact nonce/timestamp correlation (7b1ac159); explicitly unknown startup RTT with bounded native-equivalent 2s grace, then measured RTT+50ms (94d2ea36); terminal opens polled in the existing bounded call queue so probes/input/revocation continue (b16956a8). Deadlines, protocol, dependencies and engines unchanged. Original root fault was readiness without continuous end-to-end route proof; prior D265 recovery remains intact. Evidence: both imported browser regressions fail before and pass after (4 focused cases/22 assertions). Original held-open regression fails before with successful operation then closure; strengthened control-before-release barrier also fails on e9c9a576 and passes fixed source. Shared FrozenSocketTime prevents host scheduling from consuming the ordering oracle’s lease; no production deadline widened. Server module gates: Clippy,329 unit +23 integration pass. Final full native gate:744 passed,0 failed,78 configured ignores. Earlier auth timeouts pass isolated and on repeated suite; dev804ac0e8 control passes, so no failed-baseline exception claimed. First full gate’s unchanged CLI queue test refused its initial PTY (resource_unavailable); it passes unchanged isolated and in final full gate. 09c3237d adds all6 real production-browser/systemd KillMode=process/PTY cases at three visible desktop windows: CA locked/unlocked, retained/closed socket, daemon/shim stalls. Computed markers establish session identity, queued-input delivery and sibling isolation; no reload/focus recovery. All6 pass; strict tracer/fixture TypeScript pass separately (fast stops before TS on inherited size caps). Existing phone profile remains 393x852; no three-window phone claim. Screenshot captures are outside repo at /home/kayg/.cache/codex-scratch/s199r5/captures (four PNGs); visual judgment remains owner/reviewer work. Final pinned VM:4 CPUs,8262770688B RAM,72 attached PTYs,true TLS/port-scoped TCP netem. Desktop RTT300:664.173/950ms; phone RTT150:326.748/500ms; daemon RTT150:134.732/200ms; shim RTT150:77.887/200ms. All retain last bitmap. Auxiliary Pongs model RTT; measured browser traverses actual netem. Point scenarios, not p99. First setup refusal is retained as an invalid measurement. RSS about21.6–22.1MB server,6.2MB daemon,298.0–298.1MB shims (about4.14MB ×72). Five-second idle samples server9.0–14.8%,daemon3.8–5.8%,shims0–0.2% of one CPU; sample loads2.29–3.47. No new idle work added; no statistical CPU regression comparison claimed. D243 shipping deltas vs recorded9ab4329c: x8620362128−20333104=+29024B; ARM18135776−18112240=+23536B. Round deltas vs e9c9a576:+4896/+4464B for the pending-open liveness repair; no dependencies/profile changes. Both static/no PT_INTERP. Final x86 SHA256 a4ec163b2fc0ecb00f97f57a28a484571aa522593801cb1366f3f3334ce0f695 matches measured artifact. Only inherited static caps fail; no threshold altered. Self review: pair correlation before renewal; bounded unsampled grace; authorization at dispatch/during wait/before reply and SQL mutation; bounded queue/cancellation; same writer/cursors and per-window replay proof preserved; resume token remains first WS message, never header/URL; no new wire/schema/library/UI components. Evidence/logs and final shipping artifacts remain under /home/kayg/.cache/codex-scratch/s199r5. Temporary baseline/build targets/runtime roots and VM job removed. Worktree clean. scripts/verify iterate is absent on this release branch; focused per-commit gates used, final fast run on head. Final scripts/verify fast exit1. Verbatim summary: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ``` Verbatim static diagnostics: ```text PT_INTERP: absent; size: 20362128 bytes (baseline 16142672 + allowance 262144 = max 16404816) binary exceeds the 16404816-byte size budget: 20362128 bytes PT_INTERP: absent; size: 18135776 bytes (baseline 13555672 + allowance 262144 = max 13817816) binary exceeds the 13817816-byte size budget: 18135776 bytes ``` Full stdout: /home/kayg/.cache/codex-scratch/s199r5/verify-fast-final.log. Red/green: browser-red.log/browser-green.log, server-red-scoped.log/server-barrier-before.log/server-frozen-green.log. Functional: three-current.log; perf: perf-desktop-diagnostic.log,perf-phone.log,perf-daemon.log,perf-shim.log. Reproduce browser cases from apps/tracer with production dist and built binary: TRACER_TEST_PORT=4295 INFRATERNAL_BIN=<binary> bun x playwright test -c playwright.real.config.ts restart-three-session.spec.ts --reporter=./test/safe-reporter.ts. Timing harness real/perf-restart.ts requires isolated VM driver/namespaces and lock as documented in prior issue steps.
kayg commented 2026-10-13 20:53:36 +00:00
Author
Owner
Copy link

Start dev-merge-199: branch dev-merge-199, base 804ac0e854 (origin/dev). Read repository contracts and all 45 comments on #199; no parent referenced. Merge reviewed origin/stuck-after-restart 09c3237d36, preserving dev mesh/inventory/handoff/size/test changes and #199 readiness/liveness. Run functional restart/browser/unit regressions, iterate and final fast. Push only dev-merge-199; no deploy or production access. Evidence outside checkout per D262.

Start dev-merge-199: branch dev-merge-199, base 804ac0e854ad0d489d4182270d5956244a3708f0 (origin/dev). Read repository contracts and all 45 comments on #199; no parent referenced. Merge reviewed origin/stuck-after-restart 09c3237d362104b7821f711365a49af170715ddd, preserving dev mesh/inventory/handoff/size/test changes and #199 readiness/liveness. Run functional restart/browser/unit regressions, iterate and final fast. Push only dev-merge-199; no deploy or production access. Evidence outside checkout per D262.
kayg commented 2026-10-13 21:04:34 +00:00
Author
Owner
Copy link

Merge resolution progress: all nine conflicts resolved. Both wire sections retained; fresh browser budget follows D267 (4 RTT + 50ms). Real fixture keeps dev JSON setup-token envelope; live bindings retain dev first-paint/inventory settling and #199 painted replay readiness. Shared actor retains native priming/writes and #199 liveness; reattach retains dev replay flag/credit. Auto-merge mock fallback initially shadowed dev remote.host.list/host.facts.read responders; reordered so known actions keep their answers. Tracer/real-fixture strict TypeScript pass; 7 focused transport/input tests pass, 40 assertions. Directional mock cases pass; wrapped-ring initial readiness failed twice, full original assertions passed in diagnostic run; unchanged full repetition running. Native compilation ongoing. No performance measurements or production access.

Merge resolution progress: all nine conflicts resolved. Both wire sections retained; fresh browser budget follows D267 (4 RTT + 50ms). Real fixture keeps dev JSON setup-token envelope; live bindings retain dev first-paint/inventory settling and #199 painted replay readiness. Shared actor retains native priming/writes and #199 liveness; reattach retains dev replay flag/credit. Auto-merge mock fallback initially shadowed dev remote.host.list/host.facts.read responders; reordered so known actions keep their answers. Tracer/real-fixture strict TypeScript pass; 7 focused transport/input tests pass, 40 assertions. Directional mock cases pass; wrapped-ring initial readiness failed twice, full original assertions passed in diagnostic run; unchanged full repetition running. Native compilation ongoing. No performance measurements or production access.
kayg commented 2026-10-13 21:13:52 +00:00
Author
Owner
Copy link

Merge finding: native input retention (embedded.rs retain_owned_input) rejected terminal Ping as Some(false), retiring a healthy native route. New regression native_input_retention_leaves_terminal_probes_to_the_route fails on the merged pre-repair source with Some(false) vs None (0 passed / 1 failed). Ping now bypasses retained-input admission and reaches the ordinary shim-fenced route; native input/Credit/Resize ownership stays intact. Primed fallback promotion starts a fresh lease, matching ordinary link replacement. Compile adaptation: embedded_path_tests removes await on #199 synchronous connection_lost; replay-suffix assertions unchanged. Clippy decoder exceeded 100 lines by 3; compact equivalent decoding removes the excess with no lint suppression. All five original mock browser cases pass together; 7 browser unit cases / 40 assertions and production transfer budget pass (critical Brotli246141/256000, deferred42454/45056). Native full gate and real restart tests pending.native-probe-red-remote.log
mock-browser-repeat.log
browser-units.log

Merge finding: native input retention (embedded.rs retain_owned_input) rejected terminal Ping as Some(false), retiring a healthy native route. New regression native_input_retention_leaves_terminal_probes_to_the_route fails on the merged pre-repair source with Some(false) vs None (0 passed / 1 failed). Ping now bypasses retained-input admission and reaches the ordinary shim-fenced route; native input/Credit/Resize ownership stays intact. Primed fallback promotion starts a fresh lease, matching ordinary link replacement. Compile adaptation: embedded_path_tests removes await on #199 synchronous connection_lost; replay-suffix assertions unchanged. Clippy decoder exceeded 100 lines by 3; compact equivalent decoding removes the excess with no lint suppression. All five original mock browser cases pass together; 7 browser unit cases / 40 assertions and production transfer budget pass (critical Brotli246141/256000, deferred42454/45056). Native full gate and real restart tests pending.[native-probe-red-remote.log](https://git.kayg.org/attachments/74bc0db2-f73d-4a38-9a94-a0b8110b5875) [mock-browser-repeat.log](https://git.kayg.org/attachments/4fe1200c-31de-400b-9054-841ba24a8a99) [browser-units.log](https://git.kayg.org/attachments/1ecf7e69-4c55-4938-ad45-fc5e66b75415)
kayg commented 2026-10-13 21:21:13 +00:00
Author
Owner
Copy link

Native regression green: native_input_retention_leaves_terminal_probes_to_the_route 1 passed after Ping passthrough. First iterate clippy failed at read_incoming 103/100 lines; equivalent compact decoding now passes workspace clippy. First iterate stopped on daemon_gone_tests.rs:111 SUN_LEN: TMPDIR38 bytes + g<24 hex> + sessions/<32 hex>.sock =111-byte socket pathname. Rerun uses private /home/kayg/.cache/codex-scratch/m9 (34 bytes; resulting107-byte path), serialized tests; no production timeout or assertion changed. Real-binary 24 abrupt daemon restart oracle passes on rerun. Full iterate, static artifacts and real browser systemd suite still running. No baseline exception claimed. Fresh-connection perf reporter metadata also follows owner D267 (4 RTT+50ms); no new timing measurement.

Native regression green: native_input_retention_leaves_terminal_probes_to_the_route 1 passed after Ping passthrough. First iterate clippy failed at read_incoming 103/100 lines; equivalent compact decoding now passes workspace clippy. First iterate stopped on daemon_gone_tests.rs:111 SUN_LEN: TMPDIR38 bytes + g<24 hex> + sessions/<32 hex>.sock =111-byte socket pathname. Rerun uses private /home/kayg/.cache/codex-scratch/m9 (34 bytes; resulting107-byte path), serialized tests; no production timeout or assertion changed. Real-binary 24 abrupt daemon restart oracle passes on rerun. Full iterate, static artifacts and real browser systemd suite still running. No baseline exception claimed. Fresh-connection perf reporter metadata also follows owner D267 (4 RTT+50ms); no new timing measurement.
kayg commented 2026-10-13 21:33:21 +00:00
Author
Owner
Copy link

Real production-browser/systemd suite first run:11 passed/1 failed. Three-window shim-stall fails restart-fixture.ts:241: helper selected first historical attached session, while assertions/queued input target first canvas window. Arrival order can select a sibling. Fixture now passes computed marker-derived sessionIds[0] into pauseUpstream; no product source, deadlines or assertions changed. Rerun has additional setup failures at141/165 during native/baseline compilation; retain these and rerun without own build overlap. Native iterate2 reaches review3_busy_recovery timeout at review3_tests.rs:285; isolated merged test repeats same timeout in CPUWeight10000 scope. Detached dev804ac0e8 baseline under job scratch compiling for same oracle; no exception claimed before its result. Evidence real-restarts.log, real-three-selected.log, iterate-2.log, merged-review3.log outside checkout. No production access or performance claim.

Real production-browser/systemd suite first run:11 passed/1 failed. Three-window shim-stall fails restart-fixture.ts:241: helper selected first historical attached session, while assertions/queued input target first canvas window. Arrival order can select a sibling. Fixture now passes computed marker-derived sessionIds[0] into pauseUpstream; no product source, deadlines or assertions changed. Rerun has additional setup failures at141/165 during native/baseline compilation; retain these and rerun without own build overlap. Native iterate2 reaches review3_busy_recovery timeout at review3_tests.rs:285; isolated merged test repeats same timeout in CPUWeight10000 scope. Detached dev804ac0e8 baseline under job scratch compiling for same oracle; no exception claimed before its result. Evidence real-restarts.log, real-three-selected.log, iterate-2.log, merged-review3.log outside checkout. No production access or performance claim.
kayg commented 2026-10-13 21:39:42 +00:00
Author
Owner
Copy link

No #118 exception: detached origin/dev804ac0e8 review3_busy_recovery passes1/0 in CPUWeight10000 scope, merged source fails0/1 at review3_tests.rs:285 (same scoped oracle). Investigating merged socket-loop integration; no deadline changes. Iterate2 also stalled in mesh_plugin_discovery_disable_and_restart_preserve_control_socket without output for more than ten minutes; stopped only its own verified test PID4178557, preserving partial transcript. ARM static artifact built; native diagnostic focused run in progress. Three-window fixture now selects paused shim from its executed session marker; selected-session rerun has setup and later sibling-echo failures retained, no complete green claim.

No #118 exception: detached origin/dev804ac0e8 review3_busy_recovery passes1/0 in CPUWeight10000 scope, merged source fails0/1 at review3_tests.rs:285 (same scoped oracle). Investigating merged socket-loop integration; no deadline changes. Iterate2 also stalled in mesh_plugin_discovery_disable_and_restart_preserve_control_socket without output for more than ten minutes; stopped only its own verified test PID4178557, preserving partial transcript. ARM static artifact built; native diagnostic focused run in progress. Three-window fixture now selects paused shim from its executed session marker; selected-session rerun has setup and later sibling-echo failures retained, no complete green claim.
kayg commented 2026-10-13 21:44:40 +00:00
Author
Owner
Copy link

Confirmed merge regression and repair: dev804ac0e8 review3_busy_recovery passes1/0, merged in-loop FuturesUnordered open times out0/1; temporary diagnostics see both inner and outer revocation checks start without completion. The fixture has one SQLite connection. A queued future can retain it across Pending; the outer socket loop then awaits its own authority read while no longer polling that queued future. Reuse Tokio JoinSet (already in server) for the same four-call bound, preserving queued-open watchdogs, dispatch/reply/SQL authority checks and refusal formats. Socket teardown aborts and joins tasks. Original review3_busy_recovery now passes1/0; no deadline/assertion changed, no baseline/load exception. Held-open and plugin-lifecycle controls running. Temporary diagnostics removed. Shipping artifacts will be rebuilt; earlier sizes are superseded. No library/dependency/wire-tag change.

Confirmed merge regression and repair: dev804ac0e8 review3_busy_recovery passes1/0, merged in-loop FuturesUnordered open times out0/1; temporary diagnostics see both inner and outer revocation checks start without completion. The fixture has one SQLite connection. A queued future can retain it across Pending; the outer socket loop then awaits its own authority read while no longer polling that queued future. Reuse Tokio JoinSet (already in server) for the same four-call bound, preserving queued-open watchdogs, dispatch/reply/SQL authority checks and refusal formats. Socket teardown aborts and joins tasks. Original review3_busy_recovery now passes1/0; no deadline/assertion changed, no baseline/load exception. Held-open and plugin-lifecycle controls running. Temporary diagnostics removed. Shipping artifacts will be rebuilt; earlier sizes are superseded. No library/dependency/wire-tag change.
kayg commented 2026-10-13 21:56:18 +00:00
Author
Owner
Copy link

Corrected production-browser/systemd/PTY suite: all12 pass on rebuilt x86 static binary (22275888B). Six one-window scenarios (desktop/phone where defined) plus all six three-visible-window scenarios: closed/retained browser routes, CA locked/unlocked, daemon/shim stalls; queued computed commands, later input, session identity and sibling isolation preserved. Pause selects marker-proven focused session. New independent queued tasks remove single-connection SQLite polling deadlock; original review3_busy_recovery, held-open barrier and plugin discovery/disable/restart all pass unchanged. No #118 exception claimed. Third iterate still completing; ARM rebuilt for final source; final fast/head/merge SHA pending. Tests are functional; no new timing/CPU/RSS claim. Screenshots/logs in external evidence, will attach final receipts. No production access/deploy.

Corrected production-browser/systemd/PTY suite: all12 pass on rebuilt x86 static binary (22275888B). Six one-window scenarios (desktop/phone where defined) plus all six three-visible-window scenarios: closed/retained browser routes, CA locked/unlocked, daemon/shim stalls; queued computed commands, later input, session identity and sibling isolation preserved. Pause selects marker-proven focused session. New independent queued tasks remove single-connection SQLite polling deadlock; original review3_busy_recovery, held-open barrier and plugin discovery/disable/restart all pass unchanged. No #118 exception claimed. Third iterate still completing; ARM rebuilt for final source; final fast/head/merge SHA pending. Tests are functional; no new timing/CPU/RSS claim. Screenshots/logs in external evidence, will attach final receipts. No production access/deploy.
kayg commented 2026-10-13 22:03:46 +00:00
Author
Owner
Copy link

Third iterate: fmt, VT fmt, CLI compile-out, workspace clippy and cargo-deny pass; workspace tests stop at server421 pass/2 fail/46 ignored. Both failures remote_standby_tests.rs:715 treated any post-promotion control frame as a second Hello; D265 now permits Ping there. Keep the existing25ms no-extra-greeting oracle, answer only Ping with correlated Pong, reject every other frame. All6 primed native fallback tests now pass, including primary loss, blocked primary writes/control and detached writer budgets. All other touched-crate cases passed in iterate3, including recovery and plugin lifecycle repaired by bounded independent calls. Final touched-crate clippy, merge commit and full head fast pending. All12 real browser cases, all5 mocks and13 unit cases/52 assertions pass. No changed assertions/deadlines in production, no baseline exception or budget ratchet.

Third iterate: fmt, VT fmt, CLI compile-out, workspace clippy and cargo-deny pass; workspace tests stop at server421 pass/2 fail/46 ignored. Both failures remote_standby_tests.rs:715 treated any post-promotion control frame as a second Hello; D265 now permits Ping there. Keep the existing25ms no-extra-greeting oracle, answer only Ping with correlated Pong, reject every other frame. All6 primed native fallback tests now pass, including primary loss, blocked primary writes/control and detached writer budgets. All other touched-crate cases passed in iterate3, including recovery and plugin lifecycle repaired by bounded independent calls. Final touched-crate clippy, merge commit and full head fast pending. All12 real browser cases, all5 mocks and13 unit cases/52 assertions pass. No changed assertions/deadlines in production, no baseline exception or budget ratchet.
kayg commented 2026-10-13 22:06:18 +00:00
Author
Owner
Copy link

Merge committed:55f0a7f5b72d8a1048041b82e66990f90b86019b; parents804ac0e8 and09c3237d. Atomic upstream history retained. Final full head fast running; push only dev-merge-199 afterward.

Nine conflicts resolved:

  • apps/tracer/real/server.ts: dev JSON setup-token envelope plus restart/retained-route/pidfd pause fixture; pause selects marker-proven session.
  • canvas/sessions/live.ts: dev first-paint/inventory settling plus per-window painted-replay connection hooks.
  • contracts/wire.md: keep native parked/primed fallback section and route-liveness section; no new wire tags, D267 four-RTT fresh budget.
  • agent/daemon.rs: keep remote Cursor and add shim-fenced Ping/Pong.
  • agent/session_commands_tests.rs: keep both exhaustive command variants.
  • server/auth/socket_tests.rs: retain input-credit and held-open liveness modules.
  • server/embedded.rs: retain native priming, owned writes, plugin invalidations, credit/replay flag; add probes/rebind/readiness notices, probe passthrough and fresh promoted lease.
  • server/embedded_regressions.rs: retain all dev tests plus half-open/probe regressions.
  • server/lib.rs: retain node/release modules and add liveness.

Auto-merge integration: preserve known mock inventory/facts responses before unknown-action refusal; adapt dev synchronous cleanup call; answer native Ping while preserving no-second-greeting checks. Reuse bounded independent Tokio calls to avoid single-connection SQLite deadlock; abort/join on socket teardown, existing authority checks unchanged.

Module fmt/clippy/strict TS pass. All12 real production-browser/systemd/PTY restart cases pass (six at three visible windows), all5 mocks,13 browser units/52 assertions, all6 primed native controls and unchanged recovery/held-open/plugin lifecycle controls pass. Earlier failed gates and baseline control retained on issue; no waiver or assertion/deadline widening. Full fast and exact final sizes pending. No production access/deploy; original loaded owner tab unverified.

Merge committed:55f0a7f5b72d8a1048041b82e66990f90b86019b; parents804ac0e8 and09c3237d. Atomic upstream history retained. Final full head fast running; push only dev-merge-199 afterward. Nine conflicts resolved: - apps/tracer/real/server.ts: dev JSON setup-token envelope plus restart/retained-route/pidfd pause fixture; pause selects marker-proven session. - canvas/sessions/live.ts: dev first-paint/inventory settling plus per-window painted-replay connection hooks. - contracts/wire.md: keep native parked/primed fallback section and route-liveness section; no new wire tags, D267 four-RTT fresh budget. - agent/daemon.rs: keep remote Cursor and add shim-fenced Ping/Pong. - agent/session_commands_tests.rs: keep both exhaustive command variants. - server/auth/socket_tests.rs: retain input-credit and held-open liveness modules. - server/embedded.rs: retain native priming, owned writes, plugin invalidations, credit/replay flag; add probes/rebind/readiness notices, probe passthrough and fresh promoted lease. - server/embedded_regressions.rs: retain all dev tests plus half-open/probe regressions. - server/lib.rs: retain node/release modules and add liveness. Auto-merge integration: preserve known mock inventory/facts responses before unknown-action refusal; adapt dev synchronous cleanup call; answer native Ping while preserving no-second-greeting checks. Reuse bounded independent Tokio calls to avoid single-connection SQLite deadlock; abort/join on socket teardown, existing authority checks unchanged. Module fmt/clippy/strict TS pass. All12 real production-browser/systemd/PTY restart cases pass (six at three visible windows), all5 mocks,13 browser units/52 assertions, all6 primed native controls and unchanged recovery/held-open/plugin lifecycle controls pass. Earlier failed gates and baseline control retained on issue; no waiver or assertion/deadline widening. Full fast and exact final sizes pending. No production access/deploy; original loaded owner tab unverified.
kayg commented 2026-10-13 22:15:40 +00:00
Author
Owner
Copy link

Head55f0a7f5: separate production three-window capture repetition all6 pass; prior complete real suite all12 pass (including phone one-window profile), five mock cases and13 transport/input/exit units with52 assertions pass. Native red/green receipts and selected actual production screenshots attached below; visual judgment belongs to owner/reviewer, no new timing claim. Final fast continuing; no production/deploy access.

real-restarts-final.log
real-three-captures.log
browser-units-final.log
mock-browser-repeat.log
native-primed-green.log
merged-review3.log
merged-review3-green.log
baseline-review3.log
held-open-green.log
plugin-lifecycle-green.log
restart-locked-catching-up.png
restart-locked-ready.png
restart-unlock-ready.png
restart-unlock-catching-up.png
restart-unlock-ready.png

Head55f0a7f5: separate production three-window capture repetition all6 pass; prior complete real suite all12 pass (including phone one-window profile), five mock cases and13 transport/input/exit units with52 assertions pass. Native red/green receipts and selected actual production screenshots attached below; visual judgment belongs to owner/reviewer, no new timing claim. Final fast continuing; no production/deploy access. [real-restarts-final.log](https://git.kayg.org/attachments/7b9c9522-a85e-432f-9327-3d892b35b661) [real-three-captures.log](https://git.kayg.org/attachments/6e7c76f5-fd59-44bb-9069-8ca7dd7efa08) [browser-units-final.log](https://git.kayg.org/attachments/705914bc-ae9b-4c79-877e-141ee30a0186) [mock-browser-repeat.log](https://git.kayg.org/attachments/2cd4ad7c-da4b-45b0-a5cb-3c956599f878) [native-primed-green.log](https://git.kayg.org/attachments/16d2e06a-94ff-40e5-a0a0-26e0888dd43d) [merged-review3.log](https://git.kayg.org/attachments/d08faca2-cbdd-42c4-b2c8-42194080947a) [merged-review3-green.log](https://git.kayg.org/attachments/96f82cb5-9c38-4266-abe7-117b4eec705a) [baseline-review3.log](https://git.kayg.org/attachments/c9a1ddba-53e6-4d00-a191-34c066bb3594) [held-open-green.log](https://git.kayg.org/attachments/5767cf43-bc66-47d3-811f-d8dc5bb93928) [plugin-lifecycle-green.log](https://git.kayg.org/attachments/dd56675c-73a7-4cd1-8933-e23e741b13e0) ![restart-locked-catching-up.png](https://git.kayg.org/attachments/6b32c614-35e1-4ab8-8e77-90e386c4f81f) ![restart-locked-ready.png](https://git.kayg.org/attachments/35965daf-ba5b-4460-8500-ec603bd0fbc3) ![restart-unlock-ready.png](https://git.kayg.org/attachments/a458c83a-fd70-4861-8e4d-afee4061ce7d) ![restart-unlock-catching-up.png](https://git.kayg.org/attachments/c56ea376-4017-4c23-b094-366c7ec88090) ![restart-unlock-ready.png](https://git.kayg.org/attachments/df6fd577-7ea7-41ef-abeb-c7bb91472415)
kayg commented 2026-10-13 22:22:28 +00:00
Author
Owner
Copy link

55f0a7f5 full fast: native workspace1013 passed/0 failed/138 configured ignores; fmt, VT fmt, CLI compile-out, clippy, cargo-deny pass. Static only size exceptions: x8622275888B (+71584 over22204304 cap), ARM19704712B (+65968 over19638744 cap), both no PT_INTERP; native x86 key-heap initialization1 passes, ARM runtime check skipped on x86 per gate.

Fast stops before TypeScript on size failure. Separate ts_verify_fast: compiler passes; canvas/unit batch191 pass/0 fail but7 unhandled async errors before build/protocol stages. Every error host-icons.ts:55: status.test.ts global document fixture omitted querySelectorAll, which dev59391585 lazy mark completion uses. Add empty SVG query result to that existing non-rendering fixture; no production source or assertion changed. Test-only atomic follow-up and exact-head full gate pending; no budget/exception beyond D263 claimed. Red receipt ts-final-head.log; green full TS rerun running.

55f0a7f5 full fast: native workspace1013 passed/0 failed/138 configured ignores; fmt, VT fmt, CLI compile-out, clippy, cargo-deny pass. Static only size exceptions: x8622275888B (+71584 over22204304 cap), ARM19704712B (+65968 over19638744 cap), both no PT_INTERP; native x86 key-heap initialization1 passes, ARM runtime check skipped on x86 per gate. Fast stops before TypeScript on size failure. Separate ts_verify_fast: compiler passes; canvas/unit batch191 pass/0 fail but7 unhandled async errors before build/protocol stages. Every error host-icons.ts:55: status.test.ts global document fixture omitted querySelectorAll, which dev59391585 lazy mark completion uses. Add empty SVG query result to that existing non-rendering fixture; no production source or assertion changed. Test-only atomic follow-up and exact-head full gate pending; no budget/exception beyond D263 claimed. Red receipt ts-final-head.log; green full TS rerun running.
kayg commented 2026-10-13 22:26:10 +00:00
Author
Owner
Copy link

Test-only follow-up committed57b0c3ddca8e190f674071e39dcc0aed1ca76cf6 on dev-merge-199 (merge55f0a7f5). Add the missing empty querySelectorAll result to the existing status DOM fixture; no runtime source/metadata/dependency change. Red TS run191 pass/0 fail plus7 unhandled lazy-mark completions; full ts_verify_fast green191 canvas/build tests and37 protocol tests, zero failures/errors. Compiler, production precompression/transfer and Rust differential oracle pass. Final production JS critical Brotli246140/256000, deferred42445/45056, largest on-demand14185/18432. Exact-head full fast rerun in progress because the test fixture changed; first merge-head fast had1013/0/138 ignores and only D263 static size failures. Both commits will push only to dev-merge-199 after final checks; no deploy or production access. Evidence ts-final-head.log (red), ts-final-green.log (green), verify-fast-head.log outside checkout.

Test-only follow-up committed57b0c3ddca8e190f674071e39dcc0aed1ca76cf6 on dev-merge-199 (merge55f0a7f5). Add the missing empty querySelectorAll result to the existing status DOM fixture; no runtime source/metadata/dependency change. Red TS run191 pass/0 fail plus7 unhandled lazy-mark completions; full ts_verify_fast green191 canvas/build tests and37 protocol tests, zero failures/errors. Compiler, production precompression/transfer and Rust differential oracle pass. Final production JS critical Brotli246140/256000, deferred42445/45056, largest on-demand14185/18432. Exact-head full fast rerun in progress because the test fixture changed; first merge-head fast had1013/0/138 ignores and only D263 static size failures. Both commits will push only to dev-merge-199 after final checks; no deploy or production access. Evidence ts-final-head.log (red), ts-final-green.log (green), verify-fast-head.log outside checkout.
kayg commented 2026-10-13 22:44:39 +00:00
Author
Owner
Copy link

Finish dev-merge-199: head57b0c3ddca8e190f674071e39dcc0aed1ca76cf6, merge55f0a7f5b72d8a1048041b82e66990f90b86019b (parents dev804ac0e8 + reviewed09c3237d). Both atomic commits pushed only to origin/dev-merge-199; remote SHA verified. No dev/main push, deploy, production access or issue closure. Conflict resolution is in the preceding merge-step comment.

Result: preserve dev mesh/primed fallback/cursors, hosts, handoff/release pipeline, size cuts and tests, with #199 bidirectional/shim-fenced liveness, server-side rebind, retained input and painted readiness. Integration repairs: native probes bypass retained-input admission; promoted links get a fresh lease; queued calls use existing Tokio tasks so a pending SQLite read cannot deadlock the main socket loop (same4-call bound, abort/join on teardown); pause the marker-proven shim; native fixtures answer Ping while rejecting a second greeting; complete the non-rendering status DOM fixture for dev's lazy marks. No new dependencies, foundational choice, wire tag, budget ratchet or production timeout change.

Final validation: native workspace1013 passed/0 failed/138 configured ignores, including real24-stop PTY/input oracle, auth recovery/revocation/late-result barriers, held-open liveness and native fallback/backpressure controls. No #118/load exception claimed. Production-browser/systemd KillMode=process/real PTYs:12/0 (six one-window profiles plus all six three-visible-window cases); separate three-window screenshot repetition6/0. Five mock cases pass;13 transport/input/exit units/52 assertions pass. Full ts_verify_fast on final head passes:191 canvas/build tests and37 protocol/differential tests, zero failures/unhandled errors; strict compiler, production precompression and transfer budgets pass. Final JS Brotli critical246140/256000, deferred42445/45056; largest on-demand14185/18432 bytes.

scripts/verify fast on final head exits1 only for D263's authorized one-time static-size exception: x8622275888B, +71584 over22204304; ARM19704712B, +65968 over19638744. Both stripped static binaries have no PT_INTERP. x86 secure key heap initialization1 passes under8MiB memlock; ARM runtime check is skipped on x86 per the gate. The size-boundary fixture rejects the over-cap shipping image for that same size failure. Fast stops before TS, so the full TS gate was run separately on the exact head.

Author review: nonce/timestamp correlation before renewal; expired/stale replies cannot retain readiness; one shim fence per link; same writer/offsets/unspent credit across rebind; per-window ready only after painted replay; bounded task/input queues and cancellation; authority at dispatch/during waits/before reply and SQL mutation; CA-locked existing sessions reconnect; Hello/Resume first message, never headers/URLs; plugin compile-out and dev handoff preserved; no receipts in checkout. Independent upstream review is of09c3237d; merge-only repairs are supported by the red/green regressions and full gate, not claimed as a separate independent review.

Limits: original already-loaded production tab untouched/unverified; headless Chromium desktop/phone profiles are functional checks, not physical iPhone or new latency/CPU/RSS measurements. Screenshots attached for owner/reviewer judgment. Earlier failures, red/green controls and baseline result remain in issue comments/receipts.

Cleanup: worktree clean; detached baseline removed; owned build targets and runtime scratch removed, no own target process remained. Selected evidence and final shipping binaries retained outside checkout at /home/kayg/.cache/codex-scratch/dev-merge-199/{evidence,artifacts}. Final artifacts SHA256-identical to the real-browser-tested merge binaries (test-only follow-up).

Verbatim final fast summary:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

Verbatim static diagnostics:

PT_INTERP: absent; size: 22275888 bytes; cap: 22204304 bytes; delta: +71584 bytes
binary exceeds the 22204304-byte size budget: 22275888 bytes
PT_INTERP: absent; size: 19704712 bytes; cap: 19638744 bytes; delta: +65968 bytes
binary exceeds the 19638744-byte size budget: 19704712 bytes

Full final gate/TypeScript/red receipt and shipping checksums:
verify-fast-final.log
ts-final-head.log
ts-final-green.log
ts-verify-final-head.log
artifact-sha256-final.txt
push.log
shipping-sha256.txt

Finish dev-merge-199: head57b0c3ddca8e190f674071e39dcc0aed1ca76cf6, merge55f0a7f5b72d8a1048041b82e66990f90b86019b (parents dev804ac0e8 + reviewed09c3237d). Both atomic commits pushed only to origin/dev-merge-199; remote SHA verified. No dev/main push, deploy, production access or issue closure. Conflict resolution is in the preceding merge-step comment. Result: preserve dev mesh/primed fallback/cursors, hosts, handoff/release pipeline, size cuts and tests, with #199 bidirectional/shim-fenced liveness, server-side rebind, retained input and painted readiness. Integration repairs: native probes bypass retained-input admission; promoted links get a fresh lease; queued calls use existing Tokio tasks so a pending SQLite read cannot deadlock the main socket loop (same4-call bound, abort/join on teardown); pause the marker-proven shim; native fixtures answer Ping while rejecting a second greeting; complete the non-rendering status DOM fixture for dev's lazy marks. No new dependencies, foundational choice, wire tag, budget ratchet or production timeout change. Final validation: native workspace1013 passed/0 failed/138 configured ignores, including real24-stop PTY/input oracle, auth recovery/revocation/late-result barriers, held-open liveness and native fallback/backpressure controls. No #118/load exception claimed. Production-browser/systemd KillMode=process/real PTYs:12/0 (six one-window profiles plus all six three-visible-window cases); separate three-window screenshot repetition6/0. Five mock cases pass;13 transport/input/exit units/52 assertions pass. Full ts_verify_fast on final head passes:191 canvas/build tests and37 protocol/differential tests, zero failures/unhandled errors; strict compiler, production precompression and transfer budgets pass. Final JS Brotli critical246140/256000, deferred42445/45056; largest on-demand14185/18432 bytes. scripts/verify fast on final head exits1 only for D263's authorized one-time static-size exception: x8622275888B, +71584 over22204304; ARM19704712B, +65968 over19638744. Both stripped static binaries have no PT_INTERP. x86 secure key heap initialization1 passes under8MiB memlock; ARM runtime check is skipped on x86 per the gate. The size-boundary fixture rejects the over-cap shipping image for that same size failure. Fast stops before TS, so the full TS gate was run separately on the exact head. Author review: nonce/timestamp correlation before renewal; expired/stale replies cannot retain readiness; one shim fence per link; same writer/offsets/unspent credit across rebind; per-window ready only after painted replay; bounded task/input queues and cancellation; authority at dispatch/during waits/before reply and SQL mutation; CA-locked existing sessions reconnect; Hello/Resume first message, never headers/URLs; plugin compile-out and dev handoff preserved; no receipts in checkout. Independent upstream review is of09c3237d; merge-only repairs are supported by the red/green regressions and full gate, not claimed as a separate independent review. Limits: original already-loaded production tab untouched/unverified; headless Chromium desktop/phone profiles are functional checks, not physical iPhone or new latency/CPU/RSS measurements. Screenshots attached for owner/reviewer judgment. Earlier failures, red/green controls and baseline result remain in issue comments/receipts. Cleanup: worktree clean; detached baseline removed; owned build targets and runtime scratch removed, no own target process remained. Selected evidence and final shipping binaries retained outside checkout at /home/kayg/.cache/codex-scratch/dev-merge-199/{evidence,artifacts}. Final artifacts SHA256-identical to the real-browser-tested merge binaries (test-only follow-up). Verbatim final fast summary: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ``` Verbatim static diagnostics: ```text PT_INTERP: absent; size: 22275888 bytes; cap: 22204304 bytes; delta: +71584 bytes binary exceeds the 22204304-byte size budget: 22275888 bytes PT_INTERP: absent; size: 19704712 bytes; cap: 19638744 bytes; delta: +65968 bytes binary exceeds the 19638744-byte size budget: 19704712 bytes ``` Full final gate/TypeScript/red receipt and shipping checksums: [verify-fast-final.log](https://git.kayg.org/attachments/a007399c-0dbb-4ef5-a281-d9d71f397933) [ts-final-head.log](https://git.kayg.org/attachments/d9e3bc3b-aa15-4400-9669-88e0e2186881) [ts-final-green.log](https://git.kayg.org/attachments/6dd30cf1-2382-4332-bd71-fdde6e833be4) [ts-verify-final-head.log](https://git.kayg.org/attachments/f4dbe258-e12d-43ff-98b9-1e47a14a90b3) [artifact-sha256-final.txt](https://git.kayg.org/attachments/eedde2fe-c69b-4229-854d-53e6b9a1a5e2) [push.log](https://git.kayg.org/attachments/7b09de9f-2eda-4c82-a8dd-dba49ea9135a) [shipping-sha256.txt](https://git.kayg.org/attachments/b0741777-820a-4851-b5b8-5ced748fc2a9)
kayg commented 2026-10-13 23:48:11 +00:00
Author
Owner
Copy link

Continuation start: dev-merge-199 at 57b0c3ddca; existing merge 55f0a7f5, dev base804ac0e8. Read #199 history (59 comments), no parent referenced; related #195 already recorded. Reproduce staging unexpected terminal message locally, report only message type/code (never payload), determine client responsiveness versus product handoff failure, then full signed staging proof and gates. No production deployment; push only dev-merge-199.

Continuation start: dev-merge-199 at 57b0c3ddca8e190f674071e39dcc0aed1ca76cf6; existing merge 55f0a7f5, dev base804ac0e8. Read #199 history (59 comments), no parent referenced; related #195 already recorded. Reproduce staging unexpected terminal message locally, report only message type/code (never payload), determine client responsiveness versus product handoff failure, then full signed staging proof and gates. No production deployment; push only dev-merge-199.
kayg commented 2026-10-14 00:02:28 +00:00
Author
Owner
Copy link

Local functional control against unchanged 57b0c3dd shipping binary: isolated systemd private /run, 128 real PTYs created, browser clients parked between commands, no handoff. Fails in wire_send (BrokenPipe); receipt local-before-isolated.log. Rootless/startup controls were fixture failures and are not counted. Existing wire_read answers Ping only when the test caller reads; attach_all parks prior clients while setting up siblings and the deploy driver parks all clients. Added test-only continuous event loop with correlated control Pong/transport Pong, bounded queues and cancellation on Drop. First adapted local repetition also fails under host load; no green claim or performance evidence. Full signed staging proof is building for the locked perf VM; product deadlines/frames remain unchanged pending its result. Safe diagnostics print Close type/code only.

Local functional control against unchanged 57b0c3dd shipping binary: isolated systemd private /run, 128 real PTYs created, browser clients parked between commands, no handoff. Fails in wire_send (BrokenPipe); receipt local-before-isolated.log. Rootless/startup controls were fixture failures and are not counted. Existing wire_read answers Ping only when the test caller reads; attach_all parks prior clients while setting up siblings and the deploy driver parks all clients. Added test-only continuous event loop with correlated control Pong/transport Pong, bounded queues and cancellation on Drop. First adapted local repetition also fails under host load; no green claim or performance evidence. Full signed staging proof is building for the locked perf VM; product deadlines/frames remain unchanged pending its result. Safe diagnostics print Close type/code only.
kayg commented 2026-10-14 00:39:33 +00:00
Author
Owner
Copy link

Confirmed local cause (optimized shipping image with temporary exit tags): Close(None) occurs before any handoff; the responsive client is retired by websocket.rs browser_expired at24 active attachments while setting up the next attachment. local-static-server-trace.log and local-static-exit-reasons.log capture it; the deliberately unattended negative control expires as expected at1 attachment. The test-only continuous client is necessary but insufficient: the full signed perf-VM proof still fails before handoff after128 PTYs on the adapted client (staging-command.log; VM load0.50/0.27/0.12). Main socket work shares probe/read scheduling with SQL, attach and leg-B sends, so local work can age a healthy browser lease. Repairing the product transport loop so control proofs continue independently of that work; preserve existing25ms authority checks, correlated/late-Pong rules, per-session shim fence/rebind and all bounded queues. Native unoptimized fixture attempts fail earlier in terminal creation and are not evidence for the optimized-path cause. No performance or successful staging claim yet.

Confirmed local cause (optimized shipping image with temporary exit tags): Close(None) occurs before any handoff; the responsive client is retired by websocket.rs browser_expired at24 active attachments while setting up the next attachment. local-static-server-trace.log and local-static-exit-reasons.log capture it; the deliberately unattended negative control expires as expected at1 attachment. The test-only continuous client is necessary but insufficient: the full signed perf-VM proof still fails before handoff after128 PTYs on the adapted client (staging-command.log; VM load0.50/0.27/0.12). Main socket work shares probe/read scheduling with SQL, attach and leg-B sends, so local work can age a healthy browser lease. Repairing the product transport loop so control proofs continue independently of that work; preserve existing25ms authority checks, correlated/late-Pong rules, per-session shim fence/rebind and all bounded queues. Native unoptimized fixture attempts fail earlier in terminal creation and are not evidence for the optimized-path cause. No performance or successful staging claim yet.
kayg commented 2026-10-14 01:28:30 +00:00
Author
Owner
Copy link

Staging acceptance PASS on the transport-repair candidate (commits/gates pending). Confirmed cause: optimized local shipping server retires a responsive socket via browser_expired at24 attachments, before any handoff. Product now polls browser control reads/proofs independently of SQL/attach/leg-B work; same authority checks, serialized writer, correlated/late-Pong rules and shim fences; bounded input bytes/slots and worker cancellation. The shipping test client also needs a continuous event loop rather than pull-only wire_read.

Perf VM (4 vCPU), load before0.41/0.35/0.52, after0.42/0.35/0.52:12 signed handoffs,128/128 shim PIDs survive and128/128 input echoes, one real passkey unlock, CA stays unlocked; broken-image refusal and verified warm rollback/backup integrity at10000 nodes and50 users pass. Listener gap conservative bracket p50=18.312ms,p99=102.780ms; max probe interval=101.395ms (coarse brackets are not exact outage durations). Steady5-second sample at128 PTYs/6 browser sockets: server10.198% of one core, RSS21696512B. VM lock released and directory absent. Full proof1/0.

Authenticated socket suite34 passed/0 failed/2 configured ignores, including10k-node headroom, revocation, held-open work and input-pressure isolation. First native server run420/4: fixed new malformed-frame EOF race; overload fixture now requires both Close/Pong without cross-stream ordering. Both direct remote queue/renewal failures pass unchanged in isolation; no load waiver claimed. Final iterate/fast, atomic commits and real restart repetitions remain.

staging-proof-transport.txt
local-static-server-trace.log
local-static-exit-reasons.log
socket-suite-transport.log

Staging acceptance PASS on the transport-repair candidate (commits/gates pending). Confirmed cause: optimized local shipping server retires a responsive socket via browser_expired at24 attachments, before any handoff. Product now polls browser control reads/proofs independently of SQL/attach/leg-B work; same authority checks, serialized writer, correlated/late-Pong rules and shim fences; bounded input bytes/slots and worker cancellation. The shipping test client also needs a continuous event loop rather than pull-only wire_read. Perf VM (4 vCPU), load before0.41/0.35/0.52, after0.42/0.35/0.52:12 signed handoffs,128/128 shim PIDs survive and128/128 input echoes, one real passkey unlock, CA stays unlocked; broken-image refusal and verified warm rollback/backup integrity at10000 nodes and50 users pass. Listener gap conservative bracket p50=18.312ms,p99=102.780ms; max probe interval=101.395ms (coarse brackets are not exact outage durations). Steady5-second sample at128 PTYs/6 browser sockets: server10.198% of one core, RSS21696512B. VM lock released and directory absent. Full proof1/0. Authenticated socket suite34 passed/0 failed/2 configured ignores, including10k-node headroom, revocation, held-open work and input-pressure isolation. First native server run420/4: fixed new malformed-frame EOF race; overload fixture now requires both Close/Pong without cross-stream ordering. Both direct remote queue/renewal failures pass unchanged in isolation; no load waiver claimed. Final iterate/fast, atomic commits and real restart repetitions remain. [staging-proof-transport.txt](https://git.kayg.org/attachments/b1810327-33c2-45bb-9ff0-e4c05f4488bd) [local-static-server-trace.log](https://git.kayg.org/attachments/f26d5ee9-dec2-450c-a4e8-6f48612e674d) [local-static-exit-reasons.log](https://git.kayg.org/attachments/267ca399-b0a5-47c8-b163-6b3561fbcd32) [socket-suite-transport.log](https://git.kayg.org/attachments/29eaf7e5-2011-42de-ba83-b80fc2aaa878)
kayg commented 2026-10-14 01:34:46 +00:00
Author
Owner
Copy link

Atomic steps pushed only to dev-merge-199:6a1fbbe2 (overload test: require both outcomes, keep exactly-once/stream isolation);37f63cfd (product bounded transport workers independent of application waits); client-helper commit follows. Fmt/all-target remote clippy pass; socket suite34/0/2 ignores, four targeted failures all pass, full signed VM proof1/0 at128/128 preserved sessions and echoes. No dev push or production deploy. Correcting the previous comment’s VM load_before transcription: the attached passing proof records0.10/0.29/0.51; load_after0.42/0.35/0.52. Other quoted metrics unchanged. Build-host128-client repetition fails under concurrent load and is not waived; adding the same positive/negative control to the locked VM proof. Final iterate/fast and restart repetition remain.

Atomic steps pushed only to dev-merge-199:6a1fbbe2 (overload test: require both outcomes, keep exactly-once/stream isolation);37f63cfd (product bounded transport workers independent of application waits); client-helper commit follows. Fmt/all-target remote clippy pass; socket suite34/0/2 ignores, four targeted failures all pass, full signed VM proof1/0 at128/128 preserved sessions and echoes. No dev push or production deploy. Correcting the previous comment’s VM load_before transcription: the attached passing proof records0.10/0.29/0.51; load_after0.42/0.35/0.52. Other quoted metrics unchanged. Build-host128-client repetition fails under concurrent load and is not waived; adding the same positive/negative control to the locked VM proof. Final iterate/fast and restart repetition remain.
kayg commented 2026-10-14 02:54:46 +00:00
Author
Owner
Copy link

Final workspace iterate found one product regression: server_setup_terminal_echo_and_resume_preserve_offsets_once panicked in futures-util sink::unfold (start_send without poll_ready), 423 passed / 1 failed / 47 ignored. The adapter consumed its context on a failed write, while existing socket teardown retries best-effort terminal Close frames. Replaced it with a serialized adapter retaining its writer after an error; targeted test and per-commit gates running. Extended signed staging proof at b9b76413 passed both ignored cases: 128/128 PTYs retained across 12 handoffs, one real passkey unlock, CA unlocked, six independently polling browsers responsive; unattended negative control receives Close(None). Transcript: staging-proof.txt. Final-head proof will be rerun after the sink correction.

Final workspace iterate found one product regression: server_setup_terminal_echo_and_resume_preserve_offsets_once panicked in futures-util sink::unfold (start_send without poll_ready), 423 passed / 1 failed / 47 ignored. The adapter consumed its context on a failed write, while existing socket teardown retries best-effort terminal Close frames. Replaced it with a serialized adapter retaining its writer after an error; targeted test and per-commit gates running. Extended signed staging proof at b9b76413 passed both ignored cases: 128/128 PTYs retained across 12 handoffs, one real passkey unlock, CA unlocked, six independently polling browsers responsive; unattended negative control receives Close(None). Transcript: [staging-proof.txt](https://git.kayg.org/attachments/378db549-f7be-4b65-8e1b-400594e1b822). Final-head proof will be rerun after the sink correction.
kayg commented 2026-10-14 03:00:32 +00:00
Author
Owner
Copy link

Gate finding receipts and browser status: verify-iterate-scope.log
real-current-control.log
real-baseline-control.log

The paired isolated retained-route/locked-CA browser regression passes with both the pre-fix shipping binary and repaired binary. Two broader restart-suite attempts failed setup/input assertions under shared host load; these are not claimed green or waived. Native iterate had 423 passing server tests and the one new adapter panic documented above; fix is being gated. The requested staging7 scratch disappeared during compilation (Cargo object/rmeta ENOENT, then missing vendored OpenSSL source files); the already-uploaded staging transcript is preserved on the issue. Further validation uses this job’s own scratch target to avoid that collision.

Gate finding receipts and browser status: [verify-iterate-scope.log](https://git.kayg.org/attachments/662bfd5f-2d32-4e80-868c-65eb0dcc1451) [real-current-control.log](https://git.kayg.org/attachments/8a83693f-f581-4251-b05d-115632e75403) [real-baseline-control.log](https://git.kayg.org/attachments/30ced017-36d5-4d00-9c83-72d775f14720) The paired isolated retained-route/locked-CA browser regression passes with both the pre-fix shipping binary and repaired binary. Two broader restart-suite attempts failed setup/input assertions under shared host load; these are not claimed green or waived. Native iterate had 423 passing server tests and the one new adapter panic documented above; fix is being gated. The requested staging7 scratch disappeared during compilation (Cargo object/rmeta ENOENT, then missing vendored OpenSSL source files); the already-uploaded staging transcript is preserved on the issue. Further validation uses this job’s own scratch target to avoid that collision.
kayg commented 2026-10-14 03:38:48 +00:00
Author
Owner
Copy link

Completed and pushed cleanup correction 49e809d9: the serialized sink now restores its writer context before returning a failed write, so terminal Close cleanup cannot panic. The original resume regression is green (1 passed / 0 failed); cargo fmt --all --check, git diff --check and cargo clippy -p infraternal-server --features remote --all-targets -- -D warnings pass. resume-sink-own-target.log
clippy-sink-own-target.log

Full signed staging proof is rerunning on this head. Only its scratch/evidence paths differ from the brief: ~/.cache/codex-scratch/dev-merge-199/staging and this job’s evidence directory avoid the staging7 deletion collision. No production deployment; only dev-merge-199 was pushed.

Completed and pushed cleanup correction 49e809d9: the serialized sink now restores its writer context before returning a failed write, so terminal Close cleanup cannot panic. The original resume regression is green (1 passed / 0 failed); cargo fmt --all --check, git diff --check and cargo clippy -p infraternal-server --features remote --all-targets -- -D warnings pass. [resume-sink-own-target.log](https://git.kayg.org/attachments/868e7bd6-082f-4ae7-a660-d1ad18d6e9c6) [clippy-sink-own-target.log](https://git.kayg.org/attachments/58634ca5-e3e2-4018-80be-6d96e69d5c86) Full signed staging proof is rerunning on this head. Only its scratch/evidence paths differ from the brief: ~/.cache/codex-scratch/dev-merge-199/staging and this job’s evidence directory avoid the staging7 deletion collision. No production deployment; only dev-merge-199 was pushed.
kayg commented 2026-10-14 05:11:29 +00:00
Author
Owner
Copy link

Final-head signed proof at 49e809d9 is RED: all 12 handoffs kept CA unlocked and 128 shim PIDs, signed bad-image refusal/automatic rollback and backup integrity (10,000 nodes / 50 users) passed, then final attach/echo received Close(None), failing ca_handoff_browser_tests.rs:111. No waiver or PASS claim. Perf VM load before 3.31 / 6.83 / 4.09. staging-49e809d9.txt

Investigating the additional socket retirement. Full iterate is running on the head. Browser harness startup failed because its private /run hid the system manager socket; fixture-only mounts corrected, real systemd cases rerunning.

Final-head signed proof at 49e809d9 is RED: all 12 handoffs kept CA unlocked and 128 shim PIDs, signed bad-image refusal/automatic rollback and backup integrity (10,000 nodes / 50 users) passed, then final attach/echo received Close(None), failing ca_handoff_browser_tests.rs:111. No waiver or PASS claim. Perf VM load before 3.31 / 6.83 / 4.09. [staging-49e809d9.txt](https://git.kayg.org/attachments/8820c30e-f1fe-4b3a-b9ec-bf97d861b3ee) Investigating the additional socket retirement. Full iterate is running on the head. Browser harness startup failed because its private /run hid the system manager socket; fixture-only mounts corrected, real systemd cases rerunning.
kayg commented 2026-10-14 05:46:57 +00:00
Author
Owner
Copy link

49e809d9 browser regressions on perf VM: 12/12 real systemd/PTY cases pass (2 restart, 4 retained-route/stalled-upstream, 6 three-window isolation), plus 3/3 mock liveness cases. Exact static image SHA256 97bfdc74661e66977854a424d7b4ca8ff18c29383b0fdde807f764c3524f1415; production assets built on build host. Private /run and passwd, real system-manager units via fixture-only wrappers; lock released between each spec file. No timing claims from earlier build-host diagnostics. vm-browser-systemd-nsenter.log
vm-browser-liveness.log
vm-browser-three-window.log
vm-browser-mock.log

Temporary static-string-only transport diagnostics were removed. Instrumented signed proof passed both cases (128/128 PIDs and input/output, CA unlocked, six responsive clients), but the earlier uninstrumented final-reattach Close(None) remains recorded; unmodified-head proof is rerunning, no size/auth/budget changes. staging-diagnostic.txt

49e809d9 browser regressions on perf VM: 12/12 real systemd/PTY cases pass (2 restart, 4 retained-route/stalled-upstream, 6 three-window isolation), plus 3/3 mock liveness cases. Exact static image SHA256 97bfdc74661e66977854a424d7b4ca8ff18c29383b0fdde807f764c3524f1415; production assets built on build host. Private /run and passwd, real system-manager units via fixture-only wrappers; lock released between each spec file. No timing claims from earlier build-host diagnostics. [vm-browser-systemd-nsenter.log](https://git.kayg.org/attachments/c4a3553b-9a26-4ed5-95ee-82f159364ad1) [vm-browser-liveness.log](https://git.kayg.org/attachments/9e4190b7-fe5a-4de4-b6f7-fc4afe5c8b42) [vm-browser-three-window.log](https://git.kayg.org/attachments/6e34a4a2-c0b5-4a9e-b34f-ce02bfb55cd4) [vm-browser-mock.log](https://git.kayg.org/attachments/6d99e044-0732-4533-adc5-941437e9399f) Temporary static-string-only transport diagnostics were removed. Instrumented signed proof passed both cases (128/128 PIDs and input/output, CA unlocked, six responsive clients), but the earlier uninstrumented final-reattach Close(None) remains recorded; unmodified-head proof is rerunning, no size/auth/budget changes. [staging-diagnostic.txt](https://git.kayg.org/attachments/fccc1de0-8148-48f4-9f6a-49c92328c21e)
kayg commented 2026-10-14 07:51:52 +00:00
Author
Owner
Copy link

Second clean 49e809d9 signed staging run still FAILED on terminal Close(None) after all 12 handoffs and bad-image rollback; 128 shim PIDs and CA unlock were preserved. This is not waived by the passing diagnostic run.

Found a deterministic product bug in liveness.rs: a fully answered route expires when server work postpones its next Ping, before the peer has an outstanding probe to answer. Paused-clock regression: 0 passed/1 failed before, 2 passed/0 failed after. d64e12c8 starts the response deadline when a fully answered route issues a probe, expires only with outstanding probes, and retains the same RTT budget and no-late-revival invariant (D265). fmt and remote all-target clippy pass. Commit pushed to dev-merge-199. Clean head staging and iterate started; preceding lease diagnostic is waiting for the exclusive perf-VM lock.

lease-delay-red.log
staging-final-head.txt
lease-delay-green.log
lease-clippy.log

Second clean 49e809d9 signed staging run still FAILED on terminal Close(None) after all 12 handoffs and bad-image rollback; 128 shim PIDs and CA unlock were preserved. This is not waived by the passing diagnostic run. Found a deterministic product bug in liveness.rs: a fully answered route expires when server work postpones its next Ping, before the peer has an outstanding probe to answer. Paused-clock regression: 0 passed/1 failed before, 2 passed/0 failed after. d64e12c8 starts the response deadline when a fully answered route issues a probe, expires only with outstanding probes, and retains the same RTT budget and no-late-revival invariant (D265). fmt and remote all-target clippy pass. Commit pushed to dev-merge-199. Clean head staging and iterate started; preceding lease diagnostic is waiting for the exclusive perf-VM lock. [lease-delay-red.log](https://git.kayg.org/attachments/6dbae7f0-a641-447d-a801-79550d69f99d) [staging-final-head.txt](https://git.kayg.org/attachments/e9a5d284-e7ba-4a38-b073-5e033c4f0446) [lease-delay-green.log](https://git.kayg.org/attachments/21e0c555-eff8-48ba-89d2-275b7190b4d4) [lease-clippy.log](https://git.kayg.org/attachments/ff649a7f-e11c-43ff-b49a-b24255907543)
kayg commented 2026-10-14 08:12:45 +00:00
Author
Owner
Copy link

d64e12c8: scripts/verify iterate PASS. Server suite 425 passed/0 failed/47 ignored, including the formerly failing unchanged native control-queue test; TypeScript protocol 37 passed/0 failed, 34,463 assertions, zero differential divergences. All compiler, bundle-transfer and artifact checks pass. Full verify fast started. Both signed staging proofs and the exact-head browser rerun remain queued under flock -w 3600 on the shared perf VM; no staging acceptance claimed yet.

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
+---------------------------+--------------+
ITERATE gate passed. This is not the final gate: run scripts/verify fast before reporting done.

verify-iterate-d64e12c8.log

Prior mock-browser functional receipts (production build; route closed indication, no visual-quality verdict):
route-after-stop.png
route-after-stop.png
route-after-stop.png

d64e12c8: scripts/verify iterate PASS. Server suite 425 passed/0 failed/47 ignored, including the formerly failing unchanged native control-queue test; TypeScript protocol 37 passed/0 failed, 34,463 assertions, zero differential divergences. All compiler, bundle-transfer and artifact checks pass. Full verify fast started. Both signed staging proofs and the exact-head browser rerun remain queued under flock -w 3600 on the shared perf VM; no staging acceptance claimed yet. ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | +---------------------------+--------------+ ITERATE gate passed. This is not the final gate: run scripts/verify fast before reporting done. ``` [verify-iterate-d64e12c8.log](https://git.kayg.org/attachments/564f77e8-b44c-40fd-bf97-8823cadb3080) Prior mock-browser functional receipts (production build; route closed indication, no visual-quality verdict): ![route-after-stop.png](https://git.kayg.org/attachments/04430f8f-a5f1-4733-b29f-a98a094372b0) ![route-after-stop.png](https://git.kayg.org/attachments/7f489cba-4f90-4342-bd77-115a1b97755e) ![route-after-stop.png](https://git.kayg.org/attachments/a2ab606a-5a8f-4c79-bf91-78046c0e4f12)
kayg commented 2026-10-14 08:27:28 +00:00
Author
Owner
Copy link

Resume after disk-full interruption: branch dev-merge-199, head d64e12c89f, merged dev base 804ac0e8. Read all 71 issue comments; no parent referenced. No uncommitted source changes; only generated scripts/pycache. Previous iterate PASS retained, final fast and clean signed staging were interrupted. Rebuild missing debug output and repeat final gates/proof on this head; only job branch push, no production deploy. Scratch remains job-owned after earlier staging7 deletion.

Resume after disk-full interruption: branch dev-merge-199, head d64e12c89fa652b1d01a78324cf61eb42c4d5172, merged dev base 804ac0e8. Read all 71 issue comments; no parent referenced. No uncommitted source changes; only generated scripts/__pycache__. Previous iterate PASS retained, final fast and clean signed staging were interrupted. Rebuild missing debug output and repeat final gates/proof on this head; only job branch push, no production deploy. Scratch remains job-owned after earlier staging7 deletion.
kayg commented 2026-10-14 08:38:53 +00:00
Author
Owner
Copy link

Resume checkpoint at d64e12c8: full ts_verify_fast PASS (191 canvas/build tests, 37 protocol tests / 34,463 assertions, zero failures). Compiler, vendor checksums, production precompression/transfer and Rust differential oracle pass. No source changes. Clean signed staging built and queued under flock -w 3600; final fast rebuilding deleted native output. Production image rebuilding for exact-head browser repetition. No new build-host performance claim.
ts-resume.log

Resume checkpoint at d64e12c8: full ts_verify_fast PASS (191 canvas/build tests, 37 protocol tests / 34,463 assertions, zero failures). Compiler, vendor checksums, production precompression/transfer and Rust differential oracle pass. No source changes. Clean signed staging built and queued under flock -w 3600; final fast rebuilding deleted native output. Production image rebuilding for exact-head browser repetition. No new build-host performance claim. [ts-resume.log](https://git.kayg.org/attachments/432654bb-f3af-4f25-91fb-aa39f929a714)
kayg commented 2026-10-14 08:55:07 +00:00
Author
Owner
Copy link

Clean d64e12c8 signed proof passes both ignored cases: 128/128 shims and input echoes, 12 sampled handoffs, retained unlocked CA, rollback/backup integrity, six local responsive clients and unattended negative control. VM load 1.96/2.80/2.01 before, 2.57/2.81/2.05 after; listener-gap bracket p50 46.566ms / p99 58.764ms, max probe interval 27.446ms; idle sample 41.987% of one core / 24,678,400B RSS at128 PTYs. However one Browser Close(None) is printed during the final idle sample before teardown. Not reporting healthy socket preservation complete: add a fresh control exchange on all six retained clients after the sample and rerun to distinguish a missed test assertion from fixture cleanup. Native fast and exact-head browser repetition continue.
staging-resume.txt

Clean d64e12c8 signed proof passes both ignored cases: 128/128 shims and input echoes, 12 sampled handoffs, retained unlocked CA, rollback/backup integrity, six local responsive clients and unattended negative control. VM load 1.96/2.80/2.01 before, 2.57/2.81/2.05 after; listener-gap bracket p50 46.566ms / p99 58.764ms, max probe interval 27.446ms; idle sample 41.987% of one core / 24,678,400B RSS at128 PTYs. However one Browser Close(None) is printed during the final idle sample before teardown. Not reporting healthy socket preservation complete: add a fresh control exchange on all six retained clients after the sample and rerun to distinguish a missed test assertion from fixture cleanup. Native fast and exact-head browser repetition continue. [staging-resume.txt](https://git.kayg.org/attachments/1b2f39c9-33bf-4249-8cef-d99444b7c1d6)
kayg commented 2026-10-14 09:09:45 +00:00
Author
Owner
Copy link

Stronger signed staging proof PASS on unchanged d64e12c8 product plus pending test coverage: final128/128 shim identities and echoes, CA remains unlocked, verified rollback/backup integrity, and all6 retained sockets answer a fresh control exchange after5s idle. Fixture ordinals8–13 recorded; no retained-socket Close in this run. Earlier unclassified Close is retained, not declared benign. VM load1.02/1.81/2.05 before,1.43/1.83/2.05 after; listener-gap brackets p50=47.300ms/p99=104.306ms, max probe interval102.649ms (coarse brackets, not exact outage); idle34.387% one core /22822912B server RSS at128 PTYs/6 browsers. Both ignored cases pass, including unattended Close and6/6 local polling clients. No new budget or performance comparison claim.

Exact product image aa83faee755bea941d9972ea081d344c22515da1fcf87bd2e85b38c12bdcb15c: all12 production-browser/systemd/PTy restart cases pass; all5 mock cases pass with their standard clusters-preview fixture build. First mock attempts used the wrong prebuilt profile and failed setup; no product source/assertions changed. Native workspace tests pass; static builds and test-coverage module gates pending. Only dev-merge-199 push, no production deploy.
staging-idle-sockets.txt
vm-browser-resume.log
vm-mock-resume-fixture.log

Stronger signed staging proof PASS on unchanged d64e12c8 product plus pending test coverage: final128/128 shim identities and echoes, CA remains unlocked, verified rollback/backup integrity, and all6 retained sockets answer a fresh control exchange after5s idle. Fixture ordinals8–13 recorded; no retained-socket Close in this run. Earlier unclassified Close is retained, not declared benign. VM load1.02/1.81/2.05 before,1.43/1.83/2.05 after; listener-gap brackets p50=47.300ms/p99=104.306ms, max probe interval102.649ms (coarse brackets, not exact outage); idle34.387% one core /22822912B server RSS at128 PTYs/6 browsers. Both ignored cases pass, including unattended Close and6/6 local polling clients. No new budget or performance comparison claim. Exact product image aa83faee755bea941d9972ea081d344c22515da1fcf87bd2e85b38c12bdcb15c: all12 production-browser/systemd/PTy restart cases pass; all5 mock cases pass with their standard clusters-preview fixture build. First mock attempts used the wrong prebuilt profile and failed setup; no product source/assertions changed. Native workspace tests pass; static builds and test-coverage module gates pending. Only dev-merge-199 push, no production deploy. [staging-idle-sockets.txt](https://git.kayg.org/attachments/8bd81364-3bdd-406e-a09c-ffd81d600449) [vm-browser-resume.log](https://git.kayg.org/attachments/a40e2ea8-cb2c-4ba3-becf-26bc91d1ee0d) [vm-mock-resume-fixture.log](https://git.kayg.org/attachments/e07264f1-2528-4183-9fd0-e105084d706f)
kayg commented 2026-10-14 09:18:37 +00:00
Author
Owner
Copy link

Completed and pushed test-only17fae13fc080bbcde47b704fa2799bfb75bcd449: fresh correlated control replies from all6 retained browsers after the idle sample, with safe socket ordinals. Product stays d64e12c8; signed proof source matches the committed test changes and passes both ignored cases (128/128 shims/echoes, unlocked CA,6 retained and6 local clients). Fmt/remote all-target clippy pass; focused real-socket event-loop test1/0 and liveness tests2/0 pass with --features remote. Initial default-feature client-loop filter selected0 and is not counted. Final scripts/verify fast started after this commit; prior full native run passed, ARM static build still completing. Remote branch SHA verified. No dev/main push or production deploy.
idle-proof-module-gates.log
idle-proof-remote-tests.log

Completed and pushed test-only17fae13fc080bbcde47b704fa2799bfb75bcd449: fresh correlated control replies from all6 retained browsers after the idle sample, with safe socket ordinals. Product stays d64e12c8; signed proof source matches the committed test changes and passes both ignored cases (128/128 shims/echoes, unlocked CA,6 retained and6 local clients). Fmt/remote all-target clippy pass; focused real-socket event-loop test1/0 and liveness tests2/0 pass with --features remote. Initial default-feature client-loop filter selected0 and is not counted. Final scripts/verify fast started after this commit; prior full native run passed, ARM static build still completing. Remote branch SHA verified. No dev/main push or production deploy. [idle-proof-module-gates.log](https://git.kayg.org/attachments/ee9649e7-d63f-4faf-8b47-5715644c5fa0) [idle-proof-remote-tests.log](https://git.kayg.org/attachments/785fed82-8607-47de-8344-83cbf82e0a6b)
kayg commented 2026-10-14 09:43:29 +00:00
Author
Owner
Copy link

Finish resume: dev-merge-199 head17fae13fc080bbcde47b704fa2799bfb75bcd449, pushed and remote SHA verified. Merge55f0a7f5 (dev804ac0e8 + reviewed09c3237d) remains intact; nine conflict resolutions are recorded in the merge-step comment above. No dev/main push, production deploy or issue closure. Worktree clean; owned VM directories, runtime roots and build targets removed after retaining artifacts/receipts.

Cause/fix: actual unexpected message was Close(None), before handoff: attachment/SQL/leg-B work prevented polling healthy browser Pongs. Bounded independent read/proof workers preserve the serialized writer and authority checks. Failed writes retain writer context for cleanup. A second deterministic fault expired fully answered routes before a delayed next probe; d64e12c8 begins the deadline when an unanswered probe exists, retaining the same RTT budget and late-reply refusal. The shipping client now polls continuously as the browser does. No wire tags, dependencies, foundational choices, timeout widening or budget ratchet. Dev native fallback, inventory, mesh and handoff behavior remain.

17fae13f adds test-only safe socket IDs and fresh correlated replies from every retained browser after the idle sample. Stronger signed proof passes both ignored cases:12 sampled handoffs,128/128 shim identities and input echoes, CA unlocked after one real passkey ceremony; signed refusal/automatic rollback and backup integrity at10000 nodes/50 users;6/6 retained clients after idle,6/6 local polling clients, unattended Close. Earlier unclassified Close remains recorded, not declared benign. VM load1.02/1.81/2.05 before and1.43/1.83/2.05 after; listener-gap bracket p50=47.300ms/p99=104.306ms, max probe interval102.649ms. Coarse brackets, not exact outage durations. Five-second idle sample34.387% of one core /22822912B server RSS at128 PTYs/6 browsers; scenario samples, no statistical typing/CPU comparison claim.

Validation: final-head workspace1015 passed/0 failed/139 configured ignores; no load waiver. Fmt, VT fmt, CLI compile-out, clippy and cargo-deny pass. Prior d64e12c8 iterate PASS; final head fast rechecks those native rows. Full TypeScript gate191 canvas/build plus37 protocol tests/34463 assertions passes, compiler/transfer/differential checks green (TS unchanged by17fa). Real production-browser/systemd/PTy12/12, standard mock fixture5/5; extra production capture controls2/2. Focused remote client-loop1/0 and liveness2/0; initial zero-test default-feature filter is not counted. Production screenshots attached for owner/reviewer judgment.

Final scripts/verify fast exits1 solely under D263's one-time size exception: x8622319696B (+115392 over22204304), ARM19748120B (+109376 over19638744), both stripped static/no PT_INTERP and staging authority absent. x86 key-heap initialization succeeds; ARM runtime check skipped on x86 per gate. The size-boundary fixture rejects the same over-cap image. Fast stops before TS, so its full gate was run separately. Browser image aa83faee… is marked release (INFRATERNAL_RELEASE_BUILD=1); native-gate image72c82e3d… uses the ordinary development marker (0), same x86 size; checksums recorded separately, not claimed identical.

Author checklist: bounded queues/cancellation, authority at dispatch/waits/replies, correlated/late proof handling, retained durable input/offsets/credit, per-session painted readiness and shim fence, CA-locked existing-session recovery, plugin compile-out, no secrets/receipts in repo. Original owner's already-loaded production tab remains untouched/unverified; no physical-iPhone or new latency-budget claim. Independent merge/repair review remains orchestrator work.

Verbatim final gate summary:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

Verbatim sizes:

PT_INTERP: absent; size: 22319696 bytes; cap: 22204304 bytes; delta: +115392 bytes
PT_INTERP: absent; size: 19748120 bytes; cap: 19638744 bytes; delta: +109376 bytes

verify-fast-17fae13f.log
final-artifacts-resume.txt
staging-idle-sockets.txt
vm-browser-resume.log
vm-mock-resume-fixture.log
ts-resume.log
vm-production-captures-resume.log
restart-locked-catching-up.png
restart-locked-ready.png
restart-unlock-catching-up.png
restart-unlock-ready.png

Finish resume: dev-merge-199 head17fae13fc080bbcde47b704fa2799bfb75bcd449, pushed and remote SHA verified. Merge55f0a7f5 (dev804ac0e8 + reviewed09c3237d) remains intact; nine conflict resolutions are recorded in the merge-step comment above. No dev/main push, production deploy or issue closure. Worktree clean; owned VM directories, runtime roots and build targets removed after retaining artifacts/receipts. Cause/fix: actual unexpected message was Close(None), before handoff: attachment/SQL/leg-B work prevented polling healthy browser Pongs. Bounded independent read/proof workers preserve the serialized writer and authority checks. Failed writes retain writer context for cleanup. A second deterministic fault expired fully answered routes before a delayed next probe; d64e12c8 begins the deadline when an unanswered probe exists, retaining the same RTT budget and late-reply refusal. The shipping client now polls continuously as the browser does. No wire tags, dependencies, foundational choices, timeout widening or budget ratchet. Dev native fallback, inventory, mesh and handoff behavior remain. 17fae13f adds test-only safe socket IDs and fresh correlated replies from every retained browser after the idle sample. Stronger signed proof passes both ignored cases:12 sampled handoffs,128/128 shim identities and input echoes, CA unlocked after one real passkey ceremony; signed refusal/automatic rollback and backup integrity at10000 nodes/50 users;6/6 retained clients after idle,6/6 local polling clients, unattended Close. Earlier unclassified Close remains recorded, not declared benign. VM load1.02/1.81/2.05 before and1.43/1.83/2.05 after; listener-gap bracket p50=47.300ms/p99=104.306ms, max probe interval102.649ms. Coarse brackets, not exact outage durations. Five-second idle sample34.387% of one core /22822912B server RSS at128 PTYs/6 browsers; scenario samples, no statistical typing/CPU comparison claim. Validation: final-head workspace1015 passed/0 failed/139 configured ignores; no load waiver. Fmt, VT fmt, CLI compile-out, clippy and cargo-deny pass. Prior d64e12c8 iterate PASS; final head fast rechecks those native rows. Full TypeScript gate191 canvas/build plus37 protocol tests/34463 assertions passes, compiler/transfer/differential checks green (TS unchanged by17fa). Real production-browser/systemd/PTy12/12, standard mock fixture5/5; extra production capture controls2/2. Focused remote client-loop1/0 and liveness2/0; initial zero-test default-feature filter is not counted. Production screenshots attached for owner/reviewer judgment. Final scripts/verify fast exits1 solely under D263's one-time size exception: x8622319696B (+115392 over22204304), ARM19748120B (+109376 over19638744), both stripped static/no PT_INTERP and staging authority absent. x86 key-heap initialization succeeds; ARM runtime check skipped on x86 per gate. The size-boundary fixture rejects the same over-cap image. Fast stops before TS, so its full gate was run separately. Browser image aa83faee… is marked release (INFRATERNAL_RELEASE_BUILD=1); native-gate image72c82e3d… uses the ordinary development marker (0), same x86 size; checksums recorded separately, not claimed identical. Author checklist: bounded queues/cancellation, authority at dispatch/waits/replies, correlated/late proof handling, retained durable input/offsets/credit, per-session painted readiness and shim fence, CA-locked existing-session recovery, plugin compile-out, no secrets/receipts in repo. Original owner's already-loaded production tab remains untouched/unverified; no physical-iPhone or new latency-budget claim. Independent merge/repair review remains orchestrator work. Verbatim final gate summary: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ``` Verbatim sizes: ```text PT_INTERP: absent; size: 22319696 bytes; cap: 22204304 bytes; delta: +115392 bytes PT_INTERP: absent; size: 19748120 bytes; cap: 19638744 bytes; delta: +109376 bytes ``` [verify-fast-17fae13f.log](https://git.kayg.org/attachments/68147d70-260f-4cbd-b0f7-1ed9a69de66c) [final-artifacts-resume.txt](https://git.kayg.org/attachments/a5476b4c-cd26-4648-a11f-fe88fdde556f) [staging-idle-sockets.txt](https://git.kayg.org/attachments/8bd81364-3bdd-406e-a09c-ffd81d600449) [vm-browser-resume.log](https://git.kayg.org/attachments/a40e2ea8-cb2c-4ba3-becf-26bc91d1ee0d) [vm-mock-resume-fixture.log](https://git.kayg.org/attachments/e07264f1-2528-4183-9fd0-e105084d706f) [ts-resume.log](https://git.kayg.org/attachments/432654bb-f3af-4f25-91fb-aa39f929a714) [vm-production-captures-resume.log](https://git.kayg.org/attachments/1316bf6e-f0d1-416f-b1e5-43e0d93e0cdb) ![restart-locked-catching-up.png](https://git.kayg.org/attachments/a1ba611c-9248-4f4b-9ae4-164f3f81aa9a) ![restart-locked-ready.png](https://git.kayg.org/attachments/f2e38eb0-59d9-4686-b8f4-9b1defe4c12d) ![restart-unlock-catching-up.png](https://git.kayg.org/attachments/dc23a4a6-b983-40ee-b035-4c2e1459a078) ![restart-unlock-ready.png](https://git.kayg.org/attachments/d516e08f-579c-411c-a7b8-d909dd0d6e68)
kayg commented 2026-10-14 09:50:55 +00:00
Author
Owner
Copy link

Start idle-cpu-199: branch idle-cpu-199, base/head 17fae13fc0. Read repository contract, decisions, architecture/process and all 77 #199 comments; no parent referenced. Compare static 9ab4329c and 17fae13f at 128 real PTYs, 0/1/6 browser sockets, 60s settle + 60s CPU/scheduler sample on locked perf VM 10.69.69.208. Preserve D265 deadlines and all #199 controls. Push only idle-cpu-199; no deploy or production access.

Start idle-cpu-199: branch idle-cpu-199, base/head 17fae13fc080bbcde47b704fa2799bfb75bcd449. Read repository contract, decisions, architecture/process and all 77 #199 comments; no parent referenced. Compare static 9ab4329c and 17fae13f at 128 real PTYs, 0/1/6 browser sockets, 60s settle + 60s CPU/scheduler sample on locked perf VM 10.69.69.208. Preserve D265 deadlines and all #199 controls. Push only idle-cpu-199; no deploy or production access.
kayg commented 2026-10-14 09:52:47 +00:00
Author
Owner
Copy link

Independent verification start: review-dev-merge-199 at target 17fae13fc0; merge base with local dev 8e7467c07c. Read all 78 comments on #199 and job report; no parent referenced. Full merge-base diff is 1509 files; concentrate on #199 restart/liveness and merge integration, preserving prior regression controls. No implementation fixes, push, deploy, production access or perf VM. Local checks are functional only; no independent performance claim.

Independent verification start: review-dev-merge-199 at target 17fae13fc080bbcde47b704fa2799bfb75bcd449; merge base with local dev 8e7467c07cd2ec1668a775a846076b278f68ef38. Read all 78 comments on #199 and job report; no parent referenced. Full merge-base diff is 1509 files; concentrate on #199 restart/liveness and merge integration, preserving prior regression controls. No implementation fixes, push, deploy, production access or perf VM. Local checks are functional only; no independent performance claim.
kayg commented 2026-10-14 09:55:14 +00:00
Author
Owner
Copy link

Independent review R199-M1 (P2), committed b49265f42c: apps/tracer/src/connection-probe-scheduling-review.test.ts fails expected disconnects=0, actual1. Every issued Ping received its exact Pong; postpone only the next browser timer to virtual100ms. connection.ts:427 retires the fully answered route before sending another probe. Equivalent server scheduling fault is repaired by d64e12c8; browser still measures from last Pong. Deterministic functional oracle, no timing measurement or implementation fix. Evidence browser-probe-scheduling-red.log under review scratch.

Independent review R199-M1 (P2), committed b49265f42cbc5575a7009a2f620d78742f0f7593: apps/tracer/src/connection-probe-scheduling-review.test.ts fails expected disconnects=0, actual1. Every issued Ping received its exact Pong; postpone only the next browser timer to virtual100ms. connection.ts:427 retires the fully answered route before sending another probe. Equivalent server scheduling fault is repaired by d64e12c8; browser still measures from last Pong. Deterministic functional oracle, no timing measurement or implementation fix. Evidence browser-probe-scheduling-red.log under review scratch.
kayg commented 2026-10-14 10:00:35 +00:00
Author
Owner
Copy link

Comparison fixture finding: both 9ab4329c and 17fae13f retain MAX_ATTACHMENTS=32 (crates/infraternal-server/src/websocket.rs). Reuse staging attach_all semantics: 25 streams/socket. At 128 existing real PTYs, 0/1/6 sockets therefore view 0/25/128 PTYs. Report attached count explicitly; same assignment on both revisions. Continuous polling answers probes while setup/settle/sample runs; fresh correlated replies after sampling verify socket survival. No product changes yet.

Comparison fixture finding: both 9ab4329c and 17fae13f retain MAX_ATTACHMENTS=32 (crates/infraternal-server/src/websocket.rs). Reuse staging attach_all semantics: 25 streams/socket. At 128 existing real PTYs, 0/1/6 sockets therefore view 0/25/128 PTYs. Report attached count explicitly; same assignment on both revisions. Continuous polling answers probes while setup/settle/sample runs; fresh correlated replies after sampling verify socket survival. No product changes yet.
kayg commented 2026-10-14 10:00:47 +00:00
Author
Owner
Copy link

Independent review R199-M2 (P2), committed 12f06e0c4d: native_probe_correlation_review.rs imports shipping Liveness, pinned Tokio1.53.2 and paused clock. Issued probes25000 and50000; unissued reply37500/37500 is accepted and extends readiness beyond unanswered deadline. Test unissued_native_probe_reply_cannot_extend_readiness fails0/1 (explicit --ignored); evidence native-probe-correlation-red.log. No authority/data bypass demonstrated. New Rust test temporarily parked until target-head fast finishes to preserve its exact native test set. Browser controls currently3 pass/2 failures under shared load; no full-suite green or baseline waiver claimed. No implementation fixes/push/deploy/perf VM.

Independent review R199-M2 (P2), committed 12f06e0c4d065c25c763f8f869a6e0f11fd21a7d: native_probe_correlation_review.rs imports shipping Liveness, pinned Tokio1.53.2 and paused clock. Issued probes25000 and50000; unissued reply37500/37500 is accepted and extends readiness beyond unanswered deadline. Test unissued_native_probe_reply_cannot_extend_readiness fails0/1 (explicit --ignored); evidence native-probe-correlation-red.log. No authority/data bypass demonstrated. New Rust test temporarily parked until target-head fast finishes to preserve its exact native test set. Browser controls currently3 pass/2 failures under shared load; no full-suite green or baseline waiver claimed. No implementation fixes/push/deploy/perf VM.
kayg commented 2026-10-14 10:09:58 +00:00
Author
Owner
Copy link

Review checkpoint at12f06e0c: six existing transport unit cases pass/30 assertions; two native Liveness controls pass through the pinned-Tokio driver. Fresh production browser assets with recorded release artifact aa83faee755bea941d9972ea081d344c22515da1fcf87bd2e85b38c12bdcb15c: retained-browser real systemd restart passes CA locked/unlocked; stalled daemon and stalled shim pass (4 controls). Explicit-close locked case fails restart-fixture.ts:271 while checking acknowledged replacement route; remaining case pending. Mock directional recovery all3 have passed across two runs; transient-input control fails at varying assertions, no full-suite green/waiver claimed. Evidence real-controls.log/browser-controls*.log; native fast still compiling, TypeScript compiler and transfer checks pass so far. No new P0/P1 finding or independent performance claim.

Review checkpoint at12f06e0c: six existing transport unit cases pass/30 assertions; two native Liveness controls pass through the pinned-Tokio driver. Fresh production browser assets with recorded release artifact aa83faee755bea941d9972ea081d344c22515da1fcf87bd2e85b38c12bdcb15c: retained-browser real systemd restart passes CA locked/unlocked; stalled daemon and stalled shim pass (4 controls). Explicit-close locked case fails restart-fixture.ts:271 while checking acknowledged replacement route; remaining case pending. Mock directional recovery all3 have passed across two runs; transient-input control fails at varying assertions, no full-suite green/waiver claimed. Evidence real-controls.log/browser-controls*.log; native fast still compiling, TypeScript compiler and transfer checks pass so far. No new P0/P1 finding or independent performance claim.
kayg commented 2026-10-14 10:18:26 +00:00
Author
Owner
Copy link

Review checkpoint at12f06e0c: TypeScript compiler,191 canvas/build cases and transfer checks pass; protocol37/0,34463 assertions, zero differential divergences pass with an independent oracle target. Earlier protocol timeouts were cargo-run waiting for the native gate target lock; same30000ms guard passes after target isolation, no assertions changed. All6 single-window real restart scenarios have passed across initial/repeat runs, including phone unlock profile; production captures attached. Three-visible-window suite fails6/6 at initial readiness restart-fixture.ts:141 before disruption; no sibling-recovery coverage claimed. Raw real-three-controls.log retained. Findings remain P0=0/P1=0/P2=2/P3=0; native exact-target fast still building tests. No performance claim or host-load waiver.

Review checkpoint at12f06e0c: TypeScript compiler,191 canvas/build cases and transfer checks pass; protocol37/0,34463 assertions, zero differential divergences pass with an independent oracle target. Earlier protocol timeouts were cargo-run waiting for the native gate target lock; same30000ms guard passes after target isolation, no assertions changed. All6 single-window real restart scenarios have passed across initial/repeat runs, including phone unlock profile; production captures attached. Three-visible-window suite fails6/6 at initial readiness restart-fixture.ts:141 before disruption; no sibling-recovery coverage claimed. Raw real-three-controls.log retained. Findings remain P0=0/P1=0/P2=2/P3=0; native exact-target fast still building tests. No performance claim or host-load waiver.
kayg commented 2026-10-14 10:22:46 +00:00
Author
Owner
Copy link

Native gate verification interrupted: GNU target/debug directory disappeared during CLI tests. verify-fast.log records ld.bfd cannot open output probe fixture path, followed by snapshot/startup ENOENT; CLI result73 pass/3 fail/1 ignored, workspace stopped before remaining crates. fmt/VT fmt/compile-out/clippy/cargo-deny pass; static builds continue. Cause of deletion unidentified; no product-regression or host-load waiver claimed. Repeat exact-target native checks under separately named /home/kayg/.cache/codex-scratch/review199-native/build. Confirmed findings/commits remain12f06e0c, P2=2, no P0/P1.

Native gate verification interrupted: GNU target/debug directory disappeared during CLI tests. verify-fast.log records ld.bfd cannot open output probe fixture path, followed by snapshot/startup ENOENT; CLI result73 pass/3 fail/1 ignored, workspace stopped before remaining crates. fmt/VT fmt/compile-out/clippy/cargo-deny pass; static builds continue. Cause of deletion unidentified; no product-regression or host-load waiver claimed. Repeat exact-target native checks under separately named /home/kayg/.cache/codex-scratch/review199-native/build. Confirmed findings/commits remain12f06e0c, P2=2, no P0/P1.
kayg commented 2026-10-14 10:30:42 +00:00
Author
Owner
Copy link

R199-M1 refined and committed6518efb1: both pre-attachment and attached fully answered control routes retire before issuing a delayed next probe (0 pass/2 fail, expected disconnects0/actual1). Before attachment the real server has no active-route probe timer, so this variant does not rely on peer-side expiry during renderer suspension. Same P2 finding; counts unchanged. Native P2 remains12f06e0c. Native repeat is building under review199-native/build after the original GNU output disappeared; static builds continue. No implementation repair or deadline change.

R199-M1 refined and committed6518efb1: both pre-attachment and attached fully answered control routes retire before issuing a delayed next probe (0 pass/2 fail, expected disconnects0/actual1). Before attachment the real server has no active-route probe timer, so this variant does not rely on peer-side expiry during renderer suspension. Same P2 finding; counts unchanged. Native P2 remains12f06e0c. Native repeat is building under review199-native/build after the original GNU output disappeared; static builds continue. No implementation repair or deadline change.
kayg commented 2026-10-14 11:06:18 +00:00
Author
Owner
Copy link

Real idle work reproduced on locked perf VM infraternal-perf-test (4 CPUs), retained exact-head native-gate artifact 72c82e3db220b592624793b6f4ee07f1ef654881ebf0f793bead9de5b291b0ad (17fae13f receipt). Fresh source rebuild still running; no product changes. 128 real PTYs; 60s settling then 60s sampling:

Browser sockets Viewed PTYs CPU % one core schedstat schedule-ins/s
0 0 0.000 0.000
1 25 5.059 889.237
6 128 20.497 4451.884

Six-socket sample: Tokio workers 10.894 CPU seconds/60s, SQLite workers 1.205 CPU seconds/60s; VM load 1.369/1.437/1.914 before, 0.762/1.261/1.823 after. Fresh correlated replies and all 128 shim identities preserved. No startup/handoff ambiguity remains.

Source path: websocket.rs tick sends each terminal proof through AttachmentHandle::send_frame -> per-attachment forward_inputs task -> actor; each Pong traverses per-attachment output task back to browser loop. Added a deterministic real-Unix-socket regression requiring content-free proofs not to queue/wake payload consumers at 128 routes; red execution queued behind the fresh build. Proposed internal repair: let the existing shared host actor own route probes/leases, consume correlated Pongs there, batch local wire writes, and send payload consumers only actual transitions. Same wire tags, 25ms cadence, shim fence, deadlines, writer/offsets, authority checks and per-stream recovery; no new dependency or engine. Profile capture and 9ab4329c 0/1/6 rows queued.

candidate-before.txt

Real idle work reproduced on locked perf VM infraternal-perf-test (4 CPUs), retained exact-head native-gate artifact 72c82e3db220b592624793b6f4ee07f1ef654881ebf0f793bead9de5b291b0ad (17fae13f receipt). Fresh source rebuild still running; no product changes. 128 real PTYs; 60s settling then 60s sampling: | Browser sockets | Viewed PTYs | CPU % one core | schedstat schedule-ins/s | |---|---:|---:|---:| |0|0|0.000|0.000| |1|25|5.059|889.237| |6|128|20.497|4451.884| Six-socket sample: Tokio workers 10.894 CPU seconds/60s, SQLite workers 1.205 CPU seconds/60s; VM load 1.369/1.437/1.914 before, 0.762/1.261/1.823 after. Fresh correlated replies and all 128 shim identities preserved. No startup/handoff ambiguity remains. Source path: websocket.rs tick sends each terminal proof through AttachmentHandle::send_frame -> per-attachment forward_inputs task -> actor; each Pong traverses per-attachment output task back to browser loop. Added a deterministic real-Unix-socket regression requiring content-free proofs not to queue/wake payload consumers at 128 routes; red execution queued behind the fresh build. Proposed internal repair: let the existing shared host actor own route probes/leases, consume correlated Pongs there, batch local wire writes, and send payload consumers only actual transitions. Same wire tags, 25ms cadence, shim fence, deadlines, writer/offsets, authority checks and per-stream recovery; no new dependency or engine. Profile capture and 9ab4329c 0/1/6 rows queued. [candidate-before.txt](https://git.kayg.org/attachments/e299d53d-6dd6-4989-9a33-b586201aa439)
kayg commented 2026-10-14 11:07:33 +00:00
Author
Owner
Copy link

Correction to the preceding per-thread transcription: SQLite workers total 1.404761 CPU seconds over the six-socket 60s window (seven workers), not 1.205. The total CPU 20.497%, Tokio subtotal 10.894s, schedule-ins rate 4451.884/s and attached raw receipt are unchanged.

Correction to the preceding per-thread transcription: SQLite workers total 1.404761 CPU seconds over the six-socket 60s window (seven workers), not 1.205. The total CPU 20.497%, Tokio subtotal 10.894s, schedule-ins rate 4451.884/s and attached raw receipt are unchanged.
kayg commented 2026-10-14 11:08:11 +00:00
Author
Owner
Copy link

Second native-output disappearance: original target/aarch64-unknown-linux-musl/release and x86 release directories vanished while ARM CLI was linking; rustc ENOENT at infraternal rcgu.o. First scripts/verify fast exits1 with test FAIL101 and static builds FAIL101; not a completed gate. x86 had completed and passed static/key-memory checks before deletion,22319696B (+115392 over cap). Protected GNU repeat passed425 server cases; direct-cargo mouse fixture first lacked zig on PATH (my invocation error), corrected to the verify wrapper PATH and full cached workspace rerun in progress. Full separate ts_verify_fast on6518efb1 PASS191+37/34463 assertions. Complete final fast will run under review199-native/build; no source fixes, deadline changes or waiver.

Second native-output disappearance: original target/aarch64-unknown-linux-musl/release and x86 release directories vanished while ARM CLI was linking; rustc ENOENT at infraternal rcgu.o. First scripts/verify fast exits1 with test FAIL101 and static builds FAIL101; not a completed gate. x86 had completed and passed static/key-memory checks before deletion,22319696B (+115392 over cap). Protected GNU repeat passed425 server cases; direct-cargo mouse fixture first lacked zig on PATH (my invocation error), corrected to the verify wrapper PATH and full cached workspace rerun in progress. Full separate ts_verify_fast on6518efb1 PASS191+37/34463 assertions. Complete final fast will run under review199-native/build; no source fixes, deadline changes or waiver.
kayg commented 2026-10-14 11:22:42 +00:00
Author
Owner
Copy link

Fresh 9ab4329c static release completed: 20,333,104B, SHA256 271a9e4a8c1ab6cebeb3bdad7e9c734901637c182219afe70831ab491373206c. All three 128-PTY, 60s-settle/60s-sample VM rows report 0.000 CPU% and 0.000 schedstat schedule-ins/s at 0/1/6 browser sockets (0/25/128 viewed PTYs); retained socket control replies and 128 shim identities pass.

Fresh 17fae13f release is 22,637,008B, SHA256 bf8843982d27be97b5d7772bc0f4640cec35b5036bf189487574f6926494ab4b, no PT_INTERP. It differs from the retained native-gate image 72c82e3d (22,319,696B); cause not established, no product source/profile/dependency edit. Do not treat the retained-image rows as the final fresh-build comparison. Fresh 0/1/6 rows queued. Use this freshly built merge-base image for the final same-recipe size delta.

Separate retained-image 10s perf capture: 192 samples, no lost samples; significant kernel futex wake_up_q/try_to_wake_up and Unix socket work, consistent with per-route queue churn. Per-thread 60s counts are primary evidence; no exact user-function attribution from the stripped image. GNU regression build still compiling; no fix yet.

baseline.txt
perf-before.txt

Fresh 9ab4329c static release completed: 20,333,104B, SHA256 271a9e4a8c1ab6cebeb3bdad7e9c734901637c182219afe70831ab491373206c. All three 128-PTY, 60s-settle/60s-sample VM rows report 0.000 CPU% and 0.000 schedstat schedule-ins/s at 0/1/6 browser sockets (0/25/128 viewed PTYs); retained socket control replies and 128 shim identities pass. Fresh 17fae13f release is 22,637,008B, SHA256 bf8843982d27be97b5d7772bc0f4640cec35b5036bf189487574f6926494ab4b, no PT_INTERP. It differs from the retained native-gate image 72c82e3d (22,319,696B); cause not established, no product source/profile/dependency edit. Do not treat the retained-image rows as the final fresh-build comparison. Fresh 0/1/6 rows queued. Use this freshly built merge-base image for the final same-recipe size delta. Separate retained-image 10s perf capture: 192 samples, no lost samples; significant kernel futex wake_up_q/try_to_wake_up and Unix socket work, consistent with per-route queue churn. Per-thread 60s counts are primary evidence; no exact user-function attribution from the stripped image. GNU regression build still compiling; no fix yet. [baseline.txt](https://git.kayg.org/attachments/a6681131-9901-47b8-a1e7-a7fdec36f39c) [perf-before.txt](https://git.kayg.org/attachments/f25c4209-9532-480f-9701-c75d0149ed6c)
kayg commented 2026-10-14 11:30:12 +00:00
Author
Owner
Copy link

Review checkpoint6518efb1: focused browser transport/input/exit controls13/0,52 assertions; full TypeScript gate191+37/0,34463 protocol assertions. Native review test passes pedantic Clippy via same pinned-Tokio driver. GNU repeat425 server cases pass; mouse probe initially lacked Zig PATH, then missing public headers from my partial native-cache copy. Restored exact checksum-pinned public headers; mouse1/0, no implementation/test assertions changed. Original fast failed after both GNU and ARM output directories disappeared; completed failed targets removed. Final full fast running under protected review199-native/build. Findings unchanged P2=2/no P0/P1; raw failed receipts retained.

Review checkpoint6518efb1: focused browser transport/input/exit controls13/0,52 assertions; full TypeScript gate191+37/0,34463 protocol assertions. Native review test passes pedantic Clippy via same pinned-Tokio driver. GNU repeat425 server cases pass; mouse probe initially lacked Zig PATH, then missing public headers from my partial native-cache copy. Restored exact checksum-pinned public headers; mouse1/0, no implementation/test assertions changed. Original fast failed after both GNU and ARM output directories disappeared; completed failed targets removed. Final full fast running under protected review199-native/build. Findings unchanged P2=2/no P0/P1; raw failed receipts retained.
kayg commented 2026-10-14 11:45:16 +00:00
Author
Owner
Copy link

Completed/pushed harness e0bb83cc: real PTY/socket functional checks, Python AST, cargo fmt --all --check and diff whitespace pass. Default observation is 60s settle +60s sample on locked VM; 25 streams/socket matches staging.

Broker regression red: proof-queues-red2.log reports 0 passed/1 failed, idle proofs woke per-session payload consumers at128 routes. Initial fixture compiler E0382 was corrected before counting this red. First actor-owned implementation green:1 passed/0 failed; cadence/deadline/fence unchanged, local probe writes batched. Added explicit monitored attachment entrypoint so plain native/parked writer lifecycles retain their scope. All payload/reconnect replies still go through browser authority gates. Full server tests, clippy and fresh static measurement remain; no completion claim.

Fresh unchanged candidate attempt after zero-socket sample closed during PTY creation; retained as failed setup, not a valid CPU row or waived control. Retry reached128 PTYs/one socket and is sampling. No production access/deploy.

Completed/pushed harness e0bb83cc: real PTY/socket functional checks, Python AST, cargo fmt --all --check and diff whitespace pass. Default observation is 60s settle +60s sample on locked VM; 25 streams/socket matches staging. Broker regression red: proof-queues-red2.log reports 0 passed/1 failed, idle proofs woke per-session payload consumers at128 routes. Initial fixture compiler E0382 was corrected before counting this red. First actor-owned implementation green:1 passed/0 failed; cadence/deadline/fence unchanged, local probe writes batched. Added explicit monitored attachment entrypoint so plain native/parked writer lifecycles retain their scope. All payload/reconnect replies still go through browser authority gates. Full server tests, clippy and fresh static measurement remain; no completion claim. Fresh unchanged candidate attempt after zero-socket sample closed during PTY creation; retained as failed setup, not a valid CPU row or waived control. Retry reached128 PTYs/one socket and is sampling. No production access/deploy.
kayg commented 2026-10-14 11:46:10 +00:00
Author
Owner
Copy link

Final protected fast checkpoint at6518efb1: exact target native test set1015 passed/0 failed/139 configured ignores; fmt, VT fmt, CLI compile-out, Clippy and cargo-deny pass. Mouse public probe now passes with complete pinned headers and Zig PATH; no test assertions/deadlines changed. Rust review regression restored from temporary parking after native checks completed. Completed GNU outputs removed only after test/cargo-deny completion to keep disk bounded; both static release builds still running. Review tests/production source status unchanged: two confirmed P2s, no P0/P1.

Final protected fast checkpoint at6518efb1: exact target native test set1015 passed/0 failed/139 configured ignores; fmt, VT fmt, CLI compile-out, Clippy and cargo-deny pass. Mouse public probe now passes with complete pinned headers and Zig PATH; no test assertions/deadlines changed. Rust review regression restored from temporary parking after native checks completed. Completed GNU outputs removed only after test/cargo-deny completion to keep disk bounded; both static release builds still running. Review tests/production source status unchanged: two confirmed P2s, no P0/P1.
kayg commented 2026-10-14 12:07:07 +00:00
Author
Owner
Copy link

Independent source review of the uncommitted actor batching found an authority gap: expiry could issue Close/Attach and replay retained input before the WebSocket consumer rechecked committed browser-session revocation. Also found dropped reconnect notices on full queues and a remote all-route write sweep that could delay incoming proofs. Correcting these before the next performance run; no proposed Rust change has been committed. Prior full server suite: 426 passed, 0 failed, 47 ignored. Added 128-route batch test: 1 passed, 0 failed. Fresh 17fa build six-socket reruns retire sockets during settle/sample; these rows are invalid, not idle proof. Retained native-gate artifact comparison remains valid.

Independent source review of the uncommitted actor batching found an authority gap: expiry could issue Close/Attach and replay retained input before the WebSocket consumer rechecked committed browser-session revocation. Also found dropped reconnect notices on full queues and a remote all-route write sweep that could delay incoming proofs. Correcting these before the next performance run; no proposed Rust change has been committed. Prior full server suite: 426 passed, 0 failed, 47 ignored. Added 128-route batch test: 1 passed, 0 failed. Fresh 17fa build six-socket reruns retire sockets during settle/sample; these rows are invalid, not idle proof. Retained native-gate artifact comparison remains valid.
kayg commented 2026-10-14 12:13:14 +00:00
Author
Owner
Copy link

Independent verification verdict #199: MERGE (no P0/P1) — merge. Counts P0=0, P1=0, P2=2, P3=0. Fix P2 tests forward under the owner rule.

Target dev-merge-199 at17fae13fc080bbcde47b704fa2799bfb75bcd449; merge base with local dev8e7467c07cd2ec1668a775a846076b278f68ef38. Review branch review-dev-merge-199, head6518efb1107b3f0f0262e60983ff2ab27b7e8ac5; three atomic local commits, two test files,94 lines. No implementation/dependency/contract changes, push, deploy, production access or perf VM.

P2 R199-M1: connection.ts:427 retires a fully answered browser control route when its next probe is delayed. Committed connection-probe-scheduling-review.test.ts (b49265f4, refined6518efb1): pre-attachment and attached variants both fail expected disconnects0/actual1. The pre-attachment case has no peer-side active-route timer. Deterministic virtual clock, not a timing measurement.
P2 R199-M2: liveness.rs:45 accepts an unissued nonce/timestamp pair between two issued probes, extending readiness past the unanswered deadline. Committed native_probe_correlation_review.rs (12f06e0c), unissued_native_probe_reply_cannot_extend_readiness, explicit --ignored. Production module imported directly by pinned-Tokio driver;0 pass/1 fail. No authority/data bypass shown. Format/pedantic Clippy pass. Findings and red receipts attached.

Final protected scripts/verify fast: workspace1015 passed/0 failed/139 configured ignores; fmt, VT fmt, CLI compile-out, Clippy, cargo-deny pass. Only D263 one-time static size exception fails: x8622319696B (+115392 over cap), ARM19748120B (+109376). Both static/no PT_INTERP, staging authority absent; x86 key-memory initialization1 succeeds, ARM runtime skipped on x86 as specified. Review tests add no shipping implementation/dependencies. Full separate TypeScript gate passes191 canvas/build plus37 protocol tests/34463 assertions, zero divergences;13 focused transport/input/exit cases/52 assertions pass. Exact native target test set ran with the new ignored Rust review file temporarily parked, then restored.

Real private production-browser/systemd KillMode=process/passkey/CA/SQLite/daemon/PTy controls: all6 single-window scenarios passed across initial/repeat runs (including phone unlock profile). One explicit-close locked case failed first then passed unchanged; failures retained. These use the recorded author release image aa83faee… and fresh browser assets, not the fresh development-marked gate image. Three-visible-window suite fails6/6 at initial readiness before disruption; sibling recovery not independently verified. Mock directional recovery all3 passed across runs; snapshot passes; transient-input fails at varying assertions, no full-suite pass/waiver claimed. These checks are functional only.

Limits: no independent performance/128-session signed handoff/physical-iPhone/original-owner-tab acceptance. The1509-file merge-base diff was reviewed primarily for #199 restart/liveness integration. Earlier native target/debug and both release output directories disappeared during execution; those interrupted gate failures are retained, not waived. Protected final gate completes. Retry invocation/cache errors (Zig PATH, omitted public headers) were corrected without source/assertion/deadline changes.

Evidence/report: /home/kayg/.cache/codex-scratch/dev-merge-199-review/evidence and /home/kayg/.local/state/codex-jobs/infraternal/review-dev-merge-199/{result.md,findings.md}. Selected red/control/protocol logs and production screenshots attached; final gate attachment follows. Issue stays open.

Verbatim final scripts/verify fast summary (exit1):

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

Verbatim size diagnostics:

PT_INTERP: absent; size: 22319696 bytes; cap: 22204304 bytes; delta: +115392 bytes
PT_INTERP: absent; size: 19748120 bytes; cap: 19638744 bytes; delta: +109376 bytes
PT_INTERP: absent; size: 22319696 bytes; cap: 22204304 bytes; delta: +115392 bytes
Independent verification verdict #199: MERGE (no P0/P1) — merge. Counts P0=0, P1=0, P2=2, P3=0. Fix P2 tests forward under the owner rule. Target dev-merge-199 at17fae13fc080bbcde47b704fa2799bfb75bcd449; merge base with local dev8e7467c07cd2ec1668a775a846076b278f68ef38. Review branch review-dev-merge-199, head6518efb1107b3f0f0262e60983ff2ab27b7e8ac5; three atomic local commits, two test files,94 lines. No implementation/dependency/contract changes, push, deploy, production access or perf VM. P2 R199-M1: connection.ts:427 retires a fully answered browser control route when its next probe is delayed. Committed connection-probe-scheduling-review.test.ts (b49265f4, refined6518efb1): pre-attachment and attached variants both fail expected disconnects0/actual1. The pre-attachment case has no peer-side active-route timer. Deterministic virtual clock, not a timing measurement. P2 R199-M2: liveness.rs:45 accepts an unissued nonce/timestamp pair between two issued probes, extending readiness past the unanswered deadline. Committed native_probe_correlation_review.rs (12f06e0c), unissued_native_probe_reply_cannot_extend_readiness, explicit --ignored. Production module imported directly by pinned-Tokio driver;0 pass/1 fail. No authority/data bypass shown. Format/pedantic Clippy pass. Findings and red receipts attached. Final protected scripts/verify fast: workspace1015 passed/0 failed/139 configured ignores; fmt, VT fmt, CLI compile-out, Clippy, cargo-deny pass. Only D263 one-time static size exception fails: x8622319696B (+115392 over cap), ARM19748120B (+109376). Both static/no PT_INTERP, staging authority absent; x86 key-memory initialization1 succeeds, ARM runtime skipped on x86 as specified. Review tests add no shipping implementation/dependencies. Full separate TypeScript gate passes191 canvas/build plus37 protocol tests/34463 assertions, zero divergences;13 focused transport/input/exit cases/52 assertions pass. Exact native target test set ran with the new ignored Rust review file temporarily parked, then restored. Real private production-browser/systemd KillMode=process/passkey/CA/SQLite/daemon/PTy controls: all6 single-window scenarios passed across initial/repeat runs (including phone unlock profile). One explicit-close locked case failed first then passed unchanged; failures retained. These use the recorded author release image aa83faee… and fresh browser assets, not the fresh development-marked gate image. Three-visible-window suite fails6/6 at initial readiness before disruption; sibling recovery not independently verified. Mock directional recovery all3 passed across runs; snapshot passes; transient-input fails at varying assertions, no full-suite pass/waiver claimed. These checks are functional only. Limits: no independent performance/128-session signed handoff/physical-iPhone/original-owner-tab acceptance. The1509-file merge-base diff was reviewed primarily for #199 restart/liveness integration. Earlier native target/debug and both release output directories disappeared during execution; those interrupted gate failures are retained, not waived. Protected final gate completes. Retry invocation/cache errors (Zig PATH, omitted public headers) were corrected without source/assertion/deadline changes. Evidence/report: /home/kayg/.cache/codex-scratch/dev-merge-199-review/evidence and /home/kayg/.local/state/codex-jobs/infraternal/review-dev-merge-199/{result.md,findings.md}. Selected red/control/protocol logs and production screenshots attached; final gate attachment follows. Issue stays open. Verbatim final scripts/verify fast summary (exit1): ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ``` Verbatim size diagnostics: ```text PT_INTERP: absent; size: 22319696 bytes; cap: 22204304 bytes; delta: +115392 bytes PT_INTERP: absent; size: 19748120 bytes; cap: 19638744 bytes; delta: +109376 bytes PT_INTERP: absent; size: 22319696 bytes; cap: 22204304 bytes; delta: +115392 bytes ```
kayg commented 2026-10-14 12:18:15 +00:00
Author
Owner
Copy link

Review complete: MERGE (no P0/P1), merge; P0=0, P1=0, P2=2, P3=0. Local review-dev-merge-199 head 6518efb110; no push/deploy. Two committed P2 regressions are fixed forward under the owner rule.

Final fast: 1,015 workspace tests pass, zero fail, 139 ignores; only D263 size caps fail. x86: 22,319,696 bytes (+115,392); ARM: 19,748,120 bytes (+109,376), both static/no PT_INTERP. Full separate TS: 191+37 tests pass, 34,463 protocol assertions; focused controls: 13 pass/52 assertions. Three-window setup and mock transient-input coverage gaps remain recorded; no independent performance/128-session handoff/original-tab claim.

Final gate receipt attached: https://git.kayg.org/attachments/afb6de9a-6cc5-489d-b2ec-c2b84182989a. Report/findings and raw evidence remain under the review's state/cache directories. Owned build/runtime scratch removed; issue stays open.

Verbatim gate:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+
Review complete: MERGE (no P0/P1), merge; P0=0, P1=0, P2=2, P3=0. Local review-dev-merge-199 head 6518efb1107b3f0f0262e60983ff2ab27b7e8ac5; no push/deploy. Two committed P2 regressions are fixed forward under the owner rule. Final fast: 1,015 workspace tests pass, zero fail, 139 ignores; only D263 size caps fail. x86: 22,319,696 bytes (+115,392); ARM: 19,748,120 bytes (+109,376), both static/no PT_INTERP. Full separate TS: 191+37 tests pass, 34,463 protocol assertions; focused controls: 13 pass/52 assertions. Three-window setup and mock transient-input coverage gaps remain recorded; no independent performance/128-session handoff/original-tab claim. Final gate receipt attached: https://git.kayg.org/attachments/afb6de9a-6cc5-489d-b2ec-c2b84182989a. Report/findings and raw evidence remain under the review's state/cache directories. Owned build/runtime scratch removed; issue stays open. Verbatim gate: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ```
kayg commented 2026-10-14 12:28:15 +00:00
Author
Owner
Copy link

Step committed/pushed: 90e17439 batches local browser shim proofs on the host actor tick and consumes correlated Pongs without per-terminal payload notifications; remote writes retain actor interleaving. Separate notice/rebind authority gates, full-queue retirement and stale-request rejection preserved. Red queue and rebind-authority regressions now green; fmt, touched-server clippy and 6 route regressions pass. Independent source review approved corrected implementation. Preliminary fixed artifact85347a14: healthy128PTY/6socket, settle60/sample60, CPU16.261251% one core,3293.262875 schedule-ins/s; still not idle-quiet. Static size22,637,392B vs same-recipe pristine17fa22,637,008B (+384B). Later current-source artifactae23a0c2 size22,637,360B (+352B) awaits final measurement. Browser recovery:11/12 passed; remaining failure is initial live-attachment setup before stalled-shim injection, not waived. Iterate failed7 CLI restart fixture path checks because my TMPDIR was a symlink: logs show alias vs canonical path mismatch and PermissionDenied. Corrected TMPDIR to its real path and rerunning. Remaining measured hotspot: server-generated browser probes read/renew SQLite authority every25ms; evaluating event/deadline lease without weakening actual data authority or dead-link detection.

Step committed/pushed: 90e17439 batches local browser shim proofs on the host actor tick and consumes correlated Pongs without per-terminal payload notifications; remote writes retain actor interleaving. Separate notice/rebind authority gates, full-queue retirement and stale-request rejection preserved. Red queue and rebind-authority regressions now green; fmt, touched-server clippy and 6 route regressions pass. Independent source review approved corrected implementation. Preliminary fixed artifact85347a14: healthy128PTY/6socket, settle60/sample60, CPU16.261251% one core,3293.262875 schedule-ins/s; still not idle-quiet. Static size22,637,392B vs same-recipe pristine17fa22,637,008B (+384B). Later current-source artifactae23a0c2 size22,637,360B (+352B) awaits final measurement. Browser recovery:11/12 passed; remaining failure is initial live-attachment setup before stalled-shim injection, not waived. Iterate failed7 CLI restart fixture path checks because my TMPDIR was a symlink: logs show alias vs canonical path mismatch and PermissionDenied. Corrected TMPDIR to its real path and rerunning. Remaining measured hotspot: server-generated browser probes read/renew SQLite authority every25ms; evaluating event/deadline lease without weakening actual data authority or dead-link detection.
kayg commented 2026-10-14 12:51:40 +00:00
Author
Owner
Copy link

Step c73aa190 committed/pushed harness diagnostics; no exception payloads or close reasons are logged. Corrected scratch fixture root to a real short directory: all28 CLI restart tests passed (1 ignored). Independent source review approves the new uncommitted probe-authority lease: server-generated content-free Ping uses committed revocation events plus exact absolute expiry, without renewing user activity or polling SQL each25ms; browser requests/payload/rebind retain fresh gates. The deterministic regression failed before the change: idle proof extended expiry1791982249 to1794573649. After change,10 idle/auth/route regressions passed, including cached128 probes while the real SQL pool connection is held and committed logout cancellation; touched-server clippy passes. Added expiry-refresh-after-authorized-renewal coverage, rerunning. Same-recipe actual merge-base57b0c3dd static x86 size22,593,200B SHA73fd93e17e888df8cff2f6b3ca253c3e9a2b8eee6e8fae8065f1547019f3bd5b; pristine17fa22,637,008B (inherited +43,808B), batching-only current-source artifact22,637,360B (+352B vs17fa, +44,160B vsmergebase). Final build/measurement/fast gate still pending. Preliminary batch measurement and red/green rebind receipts:
fixed2-six.txt
authority-red2.log
authority-green.log

Step c73aa190 committed/pushed harness diagnostics; no exception payloads or close reasons are logged. Corrected scratch fixture root to a real short directory: all28 CLI restart tests passed (1 ignored). Independent source review approves the new uncommitted probe-authority lease: server-generated content-free Ping uses committed revocation events plus exact absolute expiry, without renewing user activity or polling SQL each25ms; browser requests/payload/rebind retain fresh gates. The deterministic regression failed before the change: idle proof extended expiry1791982249 to1794573649. After change,10 idle/auth/route regressions passed, including cached128 probes while the real SQL pool connection is held and committed logout cancellation; touched-server clippy passes. Added expiry-refresh-after-authorized-renewal coverage, rerunning. Same-recipe actual merge-base57b0c3dd static x86 size22,593,200B SHA73fd93e17e888df8cff2f6b3ca253c3e9a2b8eee6e8fae8065f1547019f3bd5b; pristine17fa22,637,008B (inherited +43,808B), batching-only current-source artifact22,637,360B (+352B vs17fa, +44,160B vsmergebase). Final build/measurement/fast gate still pending. Preliminary batch measurement and red/green rebind receipts: [fixed2-six.txt](https://git.kayg.org/attachments/1e7f24bf-dbea-40e4-92cf-0ce4a47b6ee7) [authority-red2.log](https://git.kayg.org/attachments/301c0970-357b-429e-b262-3c896219f98e) [authority-green.log](https://git.kayg.org/attachments/da207428-3f7b-4f41-a9cf-4e2767b7e4ea)
kayg commented 2026-10-14 13:27:57 +00:00
Author
Owner
Copy link

Head039a5502 pushed. Final fixed artifactf67a1e00 size22,643,536B: +6,528B versus pristine17fa, +50,336B versus actual merge-base57b0c3dd (43,808B inherited before this job). Perf VM128PTY, settle60/sample60: clients0/1/6 => CPU0/2.603145/4.382569% one core, schedule-ins0/339.688761/1188.224324 per second. All retained sockets and128 shim identities survived, all cleanup PIDs gone; SQLite workers recorded0CPU and0 schedule-ins in each window. Final production browser recovery12/12 pass. Full server suite432passed/1failed/47ignored: existing bootstrap SQLite retirement retry assertion timed out after writer rollback; rerunning, not waived. Gate execution location correction: iterate calibration self-test measured host scheduling jitter and stopped before Rust. The brief requires actual timing budgets on the perf VM, so the unchanged latency_proxy.py self-test will run there under the lock via a scratch interpreter dispatcher; other verification commands remain local. No gate source or threshold changes. Full iterate/fast transcripts still pending.

Head039a5502 pushed. Final fixed artifactf67a1e00 size22,643,536B: +6,528B versus pristine17fa, +50,336B versus actual merge-base57b0c3dd (43,808B inherited before this job). Perf VM128PTY, settle60/sample60: clients0/1/6 => CPU0/2.603145/4.382569% one core, schedule-ins0/339.688761/1188.224324 per second. All retained sockets and128 shim identities survived, all cleanup PIDs gone; SQLite workers recorded0CPU and0 schedule-ins in each window. Final production browser recovery12/12 pass. Full server suite432passed/1failed/47ignored: existing bootstrap SQLite retirement retry assertion timed out after writer rollback; rerunning, not waived. Gate execution location correction: iterate calibration self-test measured host scheduling jitter and stopped before Rust. The brief requires actual timing budgets on the perf VM, so the unchanged latency_proxy.py self-test will run there under the lock via a scratch interpreter dispatcher; other verification commands remain local. No gate source or threshold changes. Full iterate/fast transcripts still pending.
kayg commented 2026-10-14 14:06:48 +00:00
Author
Owner
Copy link

Correction to the preceding CPU table: scripts/bench-idle-routes.py answered server probes but omitted the browser-originated 25 ms control Ping loop in apps/tracer/src/connection.ts. Those receipts measure a responses-only profile, not the full staging traffic. I am adding the client loop and repeating both revisions and the fix.

Source review found Context.reply still performs renewing SQL reads for these incoming content-free probes. Proposed additive clarification to contracts/keywrap.md: only content-free Ping/Pong (including bounded transport echoes) may reuse a non-renewing authority snapshot watched by committed revocation and absolute expiry; blocked writes are cancelled on revocation/expiry, and application frames, terminal payloads and backend rebinds keep their existing fresh reads. Independent source review has no security objection to this narrow exception. A real-WebSocket failing regression will precede the change.

Iteration gate Rust tests passed; cargo-deny then rejected rustls 0.23.45 for RUSTSEC-2026-0346/0347/0348/0349. The upstream 0.23.46 release fixes all four (https://github.com/rustls/rustls/releases/tag/v/0.23.46). I will update the existing dependency in a separate atomic commit, with no new features or engine choice.

Correction to the preceding CPU table: scripts/bench-idle-routes.py answered server probes but omitted the browser-originated 25 ms control Ping loop in apps/tracer/src/connection.ts. Those receipts measure a responses-only profile, not the full staging traffic. I am adding the client loop and repeating both revisions and the fix. Source review found Context.reply still performs renewing SQL reads for these incoming content-free probes. Proposed additive clarification to contracts/keywrap.md: only content-free Ping/Pong (including bounded transport echoes) may reuse a non-renewing authority snapshot watched by committed revocation and absolute expiry; blocked writes are cancelled on revocation/expiry, and application frames, terminal payloads and backend rebinds keep their existing fresh reads. Independent source review has no security objection to this narrow exception. A real-WebSocket failing regression will precede the change. Iteration gate Rust tests passed; cargo-deny then rejected rustls 0.23.45 for RUSTSEC-2026-0346/0347/0348/0349. The upstream 0.23.46 release fixes all four (https://github.com/rustls/rustls/releases/tag/v/0.23.46). I will update the existing dependency in a separate atomic commit, with no new features or engine choice.
kayg commented 2026-10-14 14:20:18 +00:00
Author
Owner
Copy link

Completed steps: 4f507ee7 adds full-duplex client probes to the fixture; 0c5401c3 scopes native attachments to remote builds/tests; d308b1cc caches both directions of content-free proofs; 04608d6f adds the proposed narrow authority-contract exception. All commits pushed to idle-cpu-199. No deploy.

Real-WebSocket regression failed before the fix: idle browser proof renewed expiry from 1791987707 to 1794579107. After the fix, all 10 focused idle/authority tests pass, including responses with the SQL connection held. cargo fmt and targeted default-feature clippy pass. Independent source review approved revocation/expiry/write cancellation and unchanged application/rebind gates.

incoming-proof-red.log
incoming-proof-final-tests.log
incoming-proof-clippy3.log

The full-duplex comparison is queued with flock -w 3600; no new timing evidence yet. rustls patch update/rebuild is running separately.

Completed steps: 4f507ee7 adds full-duplex client probes to the fixture; 0c5401c3 scopes native attachments to remote builds/tests; d308b1cc caches both directions of content-free proofs; 04608d6f adds the proposed narrow authority-contract exception. All commits pushed to idle-cpu-199. No deploy. Real-WebSocket regression failed before the fix: idle browser proof renewed expiry from 1791987707 to 1794579107. After the fix, all 10 focused idle/authority tests pass, including responses with the SQL connection held. cargo fmt and targeted default-feature clippy pass. Independent source review approved revocation/expiry/write cancellation and unchanged application/rebind gates. [incoming-proof-red.log](https://git.kayg.org/attachments/fe40e933-e17a-4fb9-beaf-916ef794e773) [incoming-proof-final-tests.log](https://git.kayg.org/attachments/1fe94ce6-00f4-40f3-a106-265693397ce7) [incoming-proof-clippy3.log](https://git.kayg.org/attachments/d4367b5f-3f00-4669-b46d-6166c9decf8b) The full-duplex comparison is queued with flock -w 3600; no new timing evidence yet. rustls patch update/rebuild is running separately.
kayg commented 2026-10-14 14:23:54 +00:00
Author
Owner
Copy link

Build-host disk reached zero free space during the static rebuild: rustc failed creating a temp directory with OS error 28 (build-duplex-fixed2.log). Removed only this job’s completed Zig source/cache scratch, retaining installed native archives; free space has since recovered to 5.2 GB. Retrying the same build. No source workaround or gate waiver.

Full-duplex performance comparisons remain queued behind the perf VM lock (flock -w 3600). Current source head 04608d6f; rustls 0.23.46 patch is uncommitted pending consumer tests. Security scan now reports: advisories ok, bans ok, licenses ok, sources ok. Workspace clippy passed.

Build-host disk reached zero free space during the static rebuild: rustc failed creating a temp directory with OS error 28 (build-duplex-fixed2.log). Removed only this job’s completed Zig source/cache scratch, retaining installed native archives; free space has since recovered to 5.2 GB. Retrying the same build. No source workaround or gate waiver. Full-duplex performance comparisons remain queued behind the perf VM lock (flock -w 3600). Current source head 04608d6f; rustls 0.23.46 patch is uncommitted pending consumer tests. Security scan now reports: advisories ok, bans ok, licenses ok, sources ok. Workspace clippy passed.
kayg commented 2026-10-14 14:40:38 +00:00
Author
Owner
Copy link

Completed dependency step 2854f7e1 (pushed): rustls 0.23.45 -> 0.23.46 only; feature graph identical. cargo-deny: advisories ok, bans ok, licenses ok, sources ok. Workspace clippy passed. Real QUIC/TLS consumer tests passed; server suite: 434 passed, 0 failed, 47 ignored. The broader workspace run stopped at the C mouse probe because disk recovery removed its scratch headers. Restored those headers; unchanged mouse test now passes (1 passed, 0 failed). Iteration gate will rerun the whole suite.

Static x86_64 artifact: 22330768 bytes, no PT_INTERP, SHA-256 0177c4d2a603097df601415e0fb95f88b67bd3150e5d50033f12c58d1d18fbdf (matching VM copy). Repeating the merge-base build with the same recipe before finalizing the size delta.

Performance acceptance and the iteration gate calibration are currently blocked on /var/lock/perf-bench.lock: holder PID 935095 has held it over 40 minutes, with multiple waiters. This job is waiting via flock -w 3600 and has not touched the holder. The current holder must finish/release the lock; orchestration may need to identify it if it remains stuck. No corrected full-duplex CPU rows exist yet.

Completed dependency step 2854f7e1 (pushed): rustls 0.23.45 -> 0.23.46 only; feature graph identical. cargo-deny: advisories ok, bans ok, licenses ok, sources ok. Workspace clippy passed. Real QUIC/TLS consumer tests passed; server suite: 434 passed, 0 failed, 47 ignored. The broader workspace run stopped at the C mouse probe because disk recovery removed its scratch headers. Restored those headers; unchanged mouse test now passes (1 passed, 0 failed). Iteration gate will rerun the whole suite. Static x86_64 artifact: 22330768 bytes, no PT_INTERP, SHA-256 0177c4d2a603097df601415e0fb95f88b67bd3150e5d50033f12c58d1d18fbdf (matching VM copy). Repeating the merge-base build with the same recipe before finalizing the size delta. Performance acceptance and the iteration gate calibration are currently blocked on /var/lock/perf-bench.lock: holder PID 935095 has held it over 40 minutes, with multiple waiters. This job is waiting via flock -w 3600 and has not touched the holder. The current holder must finish/release the lock; orchestration may need to identify it if it remains stuck. No corrected full-duplex CPU rows exist yet.
kayg commented 2026-10-14 14:53:56 +00:00
Author
Owner
Copy link

Perf-lock blocker identified by process metadata only: PID 935095 holds /var/lock/perf-bench.lock; its cwd is /root/infraternal-bench/ppu-soak/r3long, child PID 935320 is timeout. Holder elapsed about one hour. This is another job; I have not signalled it or read its argv/files. Our full-duplex tables, browser rerun and gate calibration remain queued with flock -w 3600. Performance acceptance requires that soak to release the lock, or orchestration to provide a measurement window.

Current HEAD 2854f7e1 is pushed. Both static release builds succeed. Independent TypeScript gate passes. Matched stripped x86 sizes: merge-base57b0c3dd 22275888 B; head22330768 B; delta+54880 B (under64KB). Existing caps remain exceeded: x86+126464 B; ARM19758632 B, cap19638744 B, +119888 B. D263 records the one-time cutover exception; caps were not edited. Full branch-head fast gate still required after iteration/calibration can proceed.

Perf-lock blocker identified by process metadata only: PID 935095 holds /var/lock/perf-bench.lock; its cwd is /root/infraternal-bench/ppu-soak/r3long, child PID 935320 is timeout. Holder elapsed about one hour. This is another job; I have not signalled it or read its argv/files. Our full-duplex tables, browser rerun and gate calibration remain queued with flock -w 3600. Performance acceptance requires that soak to release the lock, or orchestration to provide a measurement window. Current HEAD 2854f7e1 is pushed. Both static release builds succeed. Independent TypeScript gate passes. Matched stripped x86 sizes: merge-base57b0c3dd 22275888 B; head22330768 B; delta+54880 B (under64KB). Existing caps remain exceeded: x86+126464 B; ARM19758632 B, cap19638744 B, +119888 B. D263 records the one-time cutover exception; caps were not edited. Full branch-head fast gate still required after iteration/calibration can proceed.
kayg commented 2026-10-14 15:00:09 +00:00
Author
Owner
Copy link

Author auth/server checklist (docs/verification/auth.md B1/B2/B3/S1/S2/N1) run on 2854f7e1: application/backend/persistence authority and writer ordering traced unchanged; only content-free proofs reuse non-renewing snapshots, with watched committed revocation and actual expiry before/after waits. Control-slot/byte bounds and admission partitions retained. Passkey/recovery algorithms and browser envelopes unchanged; existing server tests passed. New helper has role/invariant docs; native proof and contract changes split into atomic commits. Both static binaries pass production-trust checks and have no PT_INTERP; existing size caps fail, recorded separately. Independent source reviewer approved both-direction proof cancellation and fresh application/rebind gates. No UI code changed; prior production-build desktop/phone receipts and browser liveness run are attached above; the patched-binary browser rerun is still queued.

Size growth justification pending full-duplex acceptance: +54880 B against the repeated merge base buys shared actor proof batching, removal of per-session payload wakes, non-renewing watched authority and the mandatory rustls security patch. The responses-only six-client receipt reduced server CPU from20.4972% to4.38257% and schedule-ins from4451.88/s to1188.22/s, with SQL workers at zero during the corrected server-probe window. That is expressly not a full-duplex staging-equivalent proof; the requested full-duplex table remains blocked by the ppu-soak lock. No size cap was raised.

Author auth/server checklist (docs/verification/auth.md B1/B2/B3/S1/S2/N1) run on 2854f7e1: application/backend/persistence authority and writer ordering traced unchanged; only content-free proofs reuse non-renewing snapshots, with watched committed revocation and actual expiry before/after waits. Control-slot/byte bounds and admission partitions retained. Passkey/recovery algorithms and browser envelopes unchanged; existing server tests passed. New helper has role/invariant docs; native proof and contract changes split into atomic commits. Both static binaries pass production-trust checks and have no PT_INTERP; existing size caps fail, recorded separately. Independent source reviewer approved both-direction proof cancellation and fresh application/rebind gates. No UI code changed; prior production-build desktop/phone receipts and browser liveness run are attached above; the patched-binary browser rerun is still queued. Size growth justification pending full-duplex acceptance: +54880 B against the repeated merge base buys shared actor proof batching, removal of per-session payload wakes, non-renewing watched authority and the mandatory rustls security patch. The responses-only six-client receipt reduced server CPU from20.4972% to4.38257% and schedule-ins from4451.88/s to1188.22/s, with SQL workers at zero during the corrected server-probe window. That is expressly not a full-duplex staging-equivalent proof; the requested full-duplex table remains blocked by the ppu-soak lock. No size cap was raised.
kayg commented 2026-10-14 15:15:34 +00:00
Author
Owner
Copy link

Fresh independent Rust section: fmt, VT fmt, CLI plugin compile-out, clippy and cargo-deny PASS; static builds FAIL only existing size caps. Test row: 433 server tests passed, 1 failed, 47 ignored. Failure is plugin_lifecycle.rs:36, unexpected control reply. Isolated lifecycle rerun passed.

The helper assumes the next binary frame is its Call Result while an active subscription shares control stream0 (contracts/wire.md has separate call_id/sub_id). A legitimate queued Event can therefore fail the test. Adding a deterministic real-WebSocket framed interleaving regression before changing this test helper; only unrelated Event snapshots will be skipped, with wrong Result IDs and other frames still failing. Production code is unchanged. No timeout waiver.

The first requested full-duplex perf row exhausted its one-hour lock wait with no measurement; ppu-soak/r3long still holds the lock. Corrected idle acceptance remains blocked.

Fresh independent Rust section: fmt, VT fmt, CLI plugin compile-out, clippy and cargo-deny PASS; static builds FAIL only existing size caps. Test row: 433 server tests passed, 1 failed, 47 ignored. Failure is plugin_lifecycle.rs:36, unexpected control reply. Isolated lifecycle rerun passed. The helper assumes the next binary frame is its Call Result while an active subscription shares control stream0 (contracts/wire.md has separate call_id/sub_id). A legitimate queued Event can therefore fail the test. Adding a deterministic real-WebSocket framed interleaving regression before changing this test helper; only unrelated Event snapshots will be skipped, with wrong Result IDs and other frames still failing. Production code is unchanged. No timeout waiver. The first requested full-duplex perf row exhausted its one-hour lock wait with no measurement; ppu-soak/r3long still holds the lock. Corrected idle acceptance remains blocked.
kayg commented 2026-10-14 15:40:34 +00:00
Author
Owner
Copy link

Completed test-only repair 90739285 (pushed): plugin exchange now correlates its Result across independent subscription Events. Deterministic framed-WebSocket regression failed first at the same unexpected-control-reply panic; focused lifecycle run: 6 passed, 0 failed. fmt and touched-crate clippy passed; independent source review approved retained lifecycle assertions.

Full workspace cargo test --workspace --locked now exits0 on 90739285; server suite: 435 passed, 0 failed, 47 ignored. Shipping x86 SHA remains0177c4d2a603097df601415e0fb95f88b67bd3150e5d50033f12c58d1d18fbdf (bit-identical to2854f7e1).

mesh-exchange-red.log
mesh-exchange-green.log
mesh-exchange-clippy.log
size-current.txt
typescript-head.log
static-head-checks.log

workspace-head-final-tests.log
rust-head-independent.log

The full scripts/verify iterate invocation exited1 after its one-hour perf-lock calibration wait, before entering Rust gates. scripts/verify fast remains blocked at the same calibration. No complete gate success claimed.

Archived-source reproducibility finding: replacing revisions at one path preserves commit timestamps, so Cargo can incorrectly regard changed source as fresh. The attempted candidate repeat returned without compiling after57b0c3dd ->17fae13f replacement. Source mtimes are now invalidated and comparison rebuilds are being audited; the preceding repeated size comparison is provisional until that audit completes. Earlier fresh17 build was in this worktree, and production9/base57 were in the private archive path; their raw build logs remain attached/retained.

Completed test-only repair 90739285 (pushed): plugin exchange now correlates its Result across independent subscription Events. Deterministic framed-WebSocket regression failed first at the same unexpected-control-reply panic; focused lifecycle run: 6 passed, 0 failed. fmt and touched-crate clippy passed; independent source review approved retained lifecycle assertions. Full workspace cargo test --workspace --locked now exits0 on 90739285; server suite: 435 passed, 0 failed, 47 ignored. Shipping x86 SHA remains0177c4d2a603097df601415e0fb95f88b67bd3150e5d50033f12c58d1d18fbdf (bit-identical to2854f7e1). [mesh-exchange-red.log](https://git.kayg.org/attachments/cd658cc5-a10e-46c1-b43a-9baae2ab0774) [mesh-exchange-green.log](https://git.kayg.org/attachments/424e7b81-11da-4032-8efd-dd4db9206acb) [mesh-exchange-clippy.log](https://git.kayg.org/attachments/6a4c8dca-c0f5-41b0-b38d-caffdff05540) [size-current.txt](https://git.kayg.org/attachments/ee594a05-ec77-4f27-8031-206324546882) [typescript-head.log](https://git.kayg.org/attachments/20db9f84-2577-445e-897e-3fe6e692587b) [static-head-checks.log](https://git.kayg.org/attachments/252df289-5e65-4f24-833e-c45df18e615a) [workspace-head-final-tests.log](https://git.kayg.org/attachments/32feaccf-3a04-41ca-8335-f50fe7d467e1) [rust-head-independent.log](https://git.kayg.org/attachments/ce3376d9-92a5-47bb-b9aa-0c73961e8e5c) The full scripts/verify iterate invocation exited1 after its one-hour perf-lock calibration wait, before entering Rust gates. scripts/verify fast remains blocked at the same calibration. No complete gate success claimed. Archived-source reproducibility finding: replacing revisions at one path preserves commit timestamps, so Cargo can incorrectly regard changed source as fresh. The attempted candidate repeat returned without compiling after57b0c3dd ->17fae13f replacement. Source mtimes are now invalidated and comparison rebuilds are being audited; the preceding repeated size comparison is provisional until that audit completes. Earlier fresh17 build was in this worktree, and production9/base57 were in the private archive path; their raw build logs remain attached/retained.
kayg commented 2026-10-14 16:39:22 +00:00
Author
Owner
Copy link

Head cc3129764c is pushed; worktree clean. Runtime fix batches host proofs and consumes Pong metadata directly, and both socket proof directions use non-renewing watched authority. Application/backend gates and dead-link cadence remain unchanged. Added real-engine red/green regressions, repaired independent Event/Result test correlation, and patched rustls 0.23.46 without feature changes.

Completed full scripts/verify fast on this head, exit 1. Output verbatim:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

Static row failures are the existing size caps only: x86 22330768 B vs cap 22204304 B (+126464 B); ARM 19758632 B vs cap 19638744 B (+119888 B). Production trust, static linkage and native x86 key-heap checks pass. D263 records the one-time cutover exception; caps unchanged. TypeScript compiler/tests/production bundle checks independently pass (fast stops before TS on the Rust failure).

Validated size comparison after source timestamp invalidation: merge base 57b0c3dd 22275888 B; candidate 17fae13f 22319696 B; fixed head 22330768 B. Delta +54880 B vs merge base, +11072 B vs candidate. Branch rebuilt without compiler cache; source/body confirmed. Final-gate rebuild SHA is 63497f01d7593ece12291b2a660da187b89447ec9001920ad5fec27c17899008; queued fixed rows now use that exact artifact. Earlier 0177c4d2 artifact remains historical; sizes equal.

peer-path-red.log
peer-path-green.log
size-validated.txt
native-fixture-cleanup-check.txt

verify-fast-final.log
verify-iterate5.log
build-base-validated.log
build-17-invalidated.log

Performance acceptance remains blocked, not done: corrected full-duplex 0/1/6 tables and patched-binary 12-browser-test rerun are incomplete. ppu-soak/r3long held the perf lock over two hours, briefly released it, then started another long phase (holder PID 1017225). Calibration ran in that gap, but first matrix attempts hit AF_UNIX path length; cc312976 fixes inspection/teardown through directory-fd paths (red/green attached). Failed-fixture native processes are confirmed gone. New complete matrix is queued per row with flock -w 3600 in duplex-matrix.txt. Orchestration needs to provide a measurement window; no idle-quiet or deployment approval claimed. No deploy; issue left open.

Head cc3129764ce10c8ce72360d974a56e841f991b38 is pushed; worktree clean. Runtime fix batches host proofs and consumes Pong metadata directly, and both socket proof directions use non-renewing watched authority. Application/backend gates and dead-link cadence remain unchanged. Added real-engine red/green regressions, repaired independent Event/Result test correlation, and patched rustls 0.23.46 without feature changes. Completed full scripts/verify fast on this head, exit 1. Output verbatim: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ``` Static row failures are the existing size caps only: x86 22330768 B vs cap 22204304 B (+126464 B); ARM 19758632 B vs cap 19638744 B (+119888 B). Production trust, static linkage and native x86 key-heap checks pass. D263 records the one-time cutover exception; caps unchanged. TypeScript compiler/tests/production bundle checks independently pass (fast stops before TS on the Rust failure). Validated size comparison after source timestamp invalidation: merge base 57b0c3dd 22275888 B; candidate 17fae13f 22319696 B; fixed head 22330768 B. Delta +54880 B vs merge base, +11072 B vs candidate. Branch rebuilt without compiler cache; source/body confirmed. Final-gate rebuild SHA is 63497f01d7593ece12291b2a660da187b89447ec9001920ad5fec27c17899008; queued fixed rows now use that exact artifact. Earlier 0177c4d2 artifact remains historical; sizes equal. [peer-path-red.log](https://git.kayg.org/attachments/d18b06a7-c650-4150-926c-0d818e07e28c) [peer-path-green.log](https://git.kayg.org/attachments/6dcaf974-5817-4f65-b671-5f22d586995c) [size-validated.txt](https://git.kayg.org/attachments/530e7812-506d-40a7-8059-0d1aa669c4ed) [native-fixture-cleanup-check.txt](https://git.kayg.org/attachments/aef0ae4a-bf3d-4143-87d6-c2ecf49b5d63) [verify-fast-final.log](https://git.kayg.org/attachments/065d4ba6-8630-42fb-9194-b0ca63979482) [verify-iterate5.log](https://git.kayg.org/attachments/897742e9-f0cc-4fa3-b92a-44748c185092) [build-base-validated.log](https://git.kayg.org/attachments/8d670df3-022e-49b2-abc6-0c7c2b010772) [build-17-invalidated.log](https://git.kayg.org/attachments/309ee7d1-de3e-46ca-ac19-1955bd078ed4) Performance acceptance remains blocked, not done: corrected full-duplex 0/1/6 tables and patched-binary 12-browser-test rerun are incomplete. ppu-soak/r3long held the perf lock over two hours, briefly released it, then started another long phase (holder PID 1017225). Calibration ran in that gap, but first matrix attempts hit AF_UNIX path length; cc312976 fixes inspection/teardown through directory-fd paths (red/green attached). Failed-fixture native processes are confirmed gone. New complete matrix is queued per row with flock -w 3600 in duplex-matrix.txt. Orchestration needs to provide a measurement window; no idle-quiet or deployment approval claimed. No deploy; issue left open.
kayg commented 2026-10-14 16:58:24 +00:00
Author
Owner
Copy link

Completed fixture review correction at 8ef4d49e: client probes now schedule from probe start (same 25 ms cadence as connection.ts), and thread churn invalidates a CPU sample instead of omitting exited workers. Deterministic exited-worker regression fails before the guard and passes after; both Python fixture tests pass. Runtime/binary source unchanged. Uploaded strict fixture atomically before remaining matrix rows execute; the first no-client baseline row used the earlier fixture and will be repeated. Final fast gate rerunning on this head.

First preliminary full-duplex baseline row (9ab4329c, 128 PTYs, 0 clients): 60 s settle, 60.0602 s sample, 0 CPU%, 0 scheduler activations/s, all threads zero; load before 2.533/1.890/1.085, after 0.928/1.542/1.015. Cleanup found no remaining owned shim. Attached-client measurements and browser rerun remain queued on the shared perf lock. This row alone does not establish staging acceptance.

Completed fixture review correction at 8ef4d49e: client probes now schedule from probe start (same 25 ms cadence as connection.ts), and thread churn invalidates a CPU sample instead of omitting exited workers. Deterministic exited-worker regression fails before the guard and passes after; both Python fixture tests pass. Runtime/binary source unchanged. Uploaded strict fixture atomically before remaining matrix rows execute; the first no-client baseline row used the earlier fixture and will be repeated. Final fast gate rerunning on this head. First preliminary full-duplex baseline row (9ab4329c, 128 PTYs, 0 clients): 60 s settle, 60.0602 s sample, 0 CPU%, 0 scheduler activations/s, all threads zero; load before 2.533/1.890/1.085, after 0.928/1.542/1.015. Cleanup found no remaining owned shim. Attached-client measurements and browser rerun remain queued on the shared perf lock. This row alone does not establish staging acceptance.
kayg commented 2026-10-14 17:02:50 +00:00
Author
Owner
Copy link

Fixture accounting correction receipts for 8ef4d49e:

Fixture accounting correction receipts for 8ef4d49e: - Red: [exited worker was silently omitted](https://git.kayg.org/attachments/c7223739-c28f-48d8-9628-28aaeb2ce926). - Green: [both fixture regressions pass](https://git.kayg.org/attachments/4b27bf19-9124-466b-bae2-895698f1e638). The push is currently waiting in git/SSH (no authentication error reported); earlier head cc312976 is already pushed. Strict matrix fixture is installed on the perf VM. No staging acceptance claim until all healthy 60 s rows complete.
kayg commented 2026-10-14 17:05:33 +00:00
Author
Owner
Copy link

8ef4d49e is now pushed to idle-cpu-199. Existing SSH multiplexing socket stalled the first push; bounded retry succeeded without changing keys or credentials. Both runtime fixes and stricter fixture are on the remote branch. Remaining acceptance matrix, browser rerun, and final gate timing calibration still wait on the shared perf lock (current holder PID 1085141; no interference).

8ef4d49e is now pushed to idle-cpu-199. Existing SSH multiplexing socket stalled the first push; bounded retry succeeded without changing keys or credentials. Both runtime fixes and stricter fixture are on the remote branch. Remaining acceptance matrix, browser rerun, and final gate timing calibration still wait on the shared perf lock (current holder PID 1085141; no interference).
kayg commented 2026-10-14 17:11:38 +00:00
Author
Owner
Copy link

Acceptance blocked on perf access: current lock holder PID 1085141 has held /var/lock/perf-bench.lock for over 30 minutes, with another job's timeout child 1112404. Matrix next row (production, 1 client), browser rerun, baseline-zero strict repeat, and final gate's timing calibration are queued with flock -w 3600. No owned benchmark is measuring or holding the lock now. Orchestrator coordination of a short measurement window is needed if this booking continues; queued commands remain active. No other job was signalled. Head 8ef4d49e is pushed; no deploy.

Acceptance blocked on perf access: current lock holder PID 1085141 has held /var/lock/perf-bench.lock for over 30 minutes, with another job's timeout child 1112404. Matrix next row (production, 1 client), browser rerun, baseline-zero strict repeat, and final gate's timing calibration are queued with flock -w 3600. No owned benchmark is measuring or holding the lock now. Orchestrator coordination of a short measurement window is needed if this booking continues; queued commands remain active. No other job was signalled. Head 8ef4d49e is pushed; no deploy.
kayg commented 2026-10-14 17:17:46 +00:00
Author
Owner
Copy link

Strict full-duplex fixture row completed on perf VM: production 9ab4329c, 128 real PTYs, 1 browser client/25 attached terminals, 60 s settle + 60.000804 s sample: CPU 1.792090% one core, scheduler activations 385.9115/s, 4,628 correlated client probe replies across settle/sample. Thread sets stable, retained socket/shim identities healthy, cleanup remaining_shims=[]. Load before 0.605/0.905/0.998, after 0.221/0.738/0.934. This includes client-originated 25 ms probes; earlier responses-only numbers are not the equivalent comparison. Remaining matrix rows queued; no staging acceptance yet.

Strict full-duplex fixture row completed on perf VM: production 9ab4329c, 128 real PTYs, 1 browser client/25 attached terminals, 60 s settle + 60.000804 s sample: CPU 1.792090% one core, scheduler activations 385.9115/s, 4,628 correlated client probe replies across settle/sample. Thread sets stable, retained socket/shim identities healthy, cleanup remaining_shims=[]. Load before 0.605/0.905/0.998, after 0.221/0.738/0.934. This includes client-originated 25 ms probes; earlier responses-only numbers are not the equivalent comparison. Remaining matrix rows queued; no staging acceptance yet.
kayg commented 2026-10-14 17:19:53 +00:00
Author
Owner
Copy link

Final binary browser rerun: 10/12 restart/recovery cases passed. Both failures were page.goto ERR_NETWORK_CHANGED before sign-in (restart-fixture.ts:107), before stalled-shim stimulus. Two failed cases queued for isolated rerun with unchanged assertions/deadlines. Post-run ownership scan found owned_native_pids=[] on perf VM; no fixture native processes remain. Raw receipt:
browser-duplex-final.log

Final binary browser rerun: 10/12 restart/recovery cases passed. Both failures were page.goto ERR_NETWORK_CHANGED before sign-in (restart-fixture.ts:107), before stalled-shim stimulus. Two failed cases queued for isolated rerun with unchanged assertions/deadlines. Post-run ownership scan found owned_native_pids=[] on perf VM; no fixture native processes remain. Raw receipt: [browser-duplex-final.log](https://git.kayg.org/attachments/67856466-897e-4893-9032-0440822fc47d)
kayg commented 2026-10-14 17:25:11 +00:00
Author
Owner
Copy link

Strict accounting caught a real invalid sample: production 9ab4329c with 6 clients/128 terminals changed its thread set during the 60 s window (bench-idle-routes.py:202). No CPU number accepted for this row; cleanup remaining_shims=[]. Same 60 s settle/sample queued for fresh retry; guard not weakened. Strict production 0-client repeat passed: 60.046855 s, CPU=0%, scheduler activations=0/s, stable threads/shims; load before 0.736/1.518/1.287, after 0.345/1.268/1.214. Final gate passed its perf calibration and is rebuilding Rust checks. Browser failed-case rerun has started.

Strict accounting caught a real invalid sample: production 9ab4329c with 6 clients/128 terminals changed its thread set during the 60 s window (bench-idle-routes.py:202). No CPU number accepted for this row; cleanup remaining_shims=[]. Same 60 s settle/sample queued for fresh retry; guard not weakened. Strict production 0-client repeat passed: 60.046855 s, CPU=0%, scheduler activations=0/s, stable threads/shims; load before 0.736/1.518/1.287, after 0.345/1.268/1.214. Final gate passed its perf calibration and is rebuilding Rust checks. Browser failed-case rerun has started.
kayg commented 2026-10-14 17:26:45 +00:00
Author
Owner
Copy link

Final binary browser coverage: main run 10/12 passed; targeted rerun passed both prior ERR_NETWORK_CHANGED cases with unchanged assertions/deadlines (2/2). All twelve requested restart/recovery cases now have passing final-binary executions. Strict production-zero CPU receipt and browser retry attached:
browser-duplex-retry.log
baseline-zero-strict.txt

Final binary browser coverage: main run 10/12 passed; targeted rerun passed both prior ERR_NETWORK_CHANGED cases with unchanged assertions/deadlines (2/2). All twelve requested restart/recovery cases now have passing final-binary executions. Strict production-zero CPU receipt and browser retry attached: [browser-duplex-retry.log](https://git.kayg.org/attachments/dfca9e5e-c837-4f45-9c0c-eb86eb3ada47) [baseline-zero-strict.txt](https://git.kayg.org/attachments/bfdf52ad-1951-4b40-9d26-04f1c0165f2a)
kayg commented 2026-10-14 17:31:19 +00:00
Author
Owner
Copy link

Strict staging row 17fae13f: 128 real PTYs, 0 clients, 60 s settle + 60.060369 s sample, CPU=0%, scheduler activations=0/s, all worker runtimes/schedule counts unchanged; thread sets and shim identities stable, cleanup remaining_shims=[]. No-client idle is quiet. Attached-client rows still required before acceptance. VM load before 9.663/14.065/7.009, after 3.547/11.502/6.567 (4 vCPU); recorded rather than treating the window as an unloaded-host measurement. Production 6-client strict retry now settling.

Strict staging row 17fae13f: 128 real PTYs, 0 clients, 60 s settle + 60.060369 s sample, CPU=0%, scheduler activations=0/s, all worker runtimes/schedule counts unchanged; thread sets and shim identities stable, cleanup remaining_shims=[]. No-client idle is quiet. Attached-client rows still required before acceptance. VM load before 9.663/14.065/7.009, after 3.547/11.502/6.567 (4 vCPU); recorded rather than treating the window as an unloaded-host measurement. Production 6-client strict retry now settling.
kayg commented 2026-10-14 17:37:35 +00:00
Author
Owner
Copy link

Strict full-duplex staging 17fae13f row passed: 128 real PTYs, 1 browser/25 attached terminals, 60 s settle + 60.000282 s sample, CPU 5.786500% one core, scheduler activations 1265.0607/s, 4,642 correlated client replies, stable threads/shims/socket, cleanup remaining_shims=[]. Production same fixture: 1.792090%, 385.9115/s (staging ~3.23x CPU, ~3.28x activations). Load before 1.037/5.186/5.166, after 0.700/4.343/4.876. Six-client staging row now settling. Runtime profile/source findings already identify per-route proof fanout/SQLite work; fixes pushed, full-duplex fixed rows pending.

Strict full-duplex staging 17fae13f row passed: 128 real PTYs, 1 browser/25 attached terminals, 60 s settle + 60.000282 s sample, CPU 5.786500% one core, scheduler activations 1265.0607/s, 4,642 correlated client replies, stable threads/shims/socket, cleanup remaining_shims=[]. Production same fixture: 1.792090%, 385.9115/s (staging ~3.23x CPU, ~3.28x activations). Load before 1.037/5.186/5.166, after 0.700/4.343/4.876. Six-client staging row now settling. Runtime profile/source findings already identify per-route proof fanout/SQLite work; fixes pushed, full-duplex fixed rows pending.
kayg commented 2026-10-14 17:39:22 +00:00
Author
Owner
Copy link

Strict full-duplex staging 17fae13f, 128 real PTYs, 6 browsers/all128 terminals: 60 s settle + 60.000372 s sample passed. CPU 24.027599% one core, scheduler activations 5920.4299/s, ~4,670 correlated client replies per socket, stable thread sets/socket/shims, cleanup remaining_shims=[]. Eight SQLite workers consumed 2.142723416 CPU-s total during the window. Load before 1.503/3.558/4.534, after 2.493/3.497/4.452. This reproduces substantial attached idle work without relaxing proof deadlines. Fixed binary rows next. Baseline 6-client diagnostic will use a separately labelled 180 s settle to inspect thread churn; original required 60 s rows remain rejected, not replaced silently.

Strict full-duplex staging 17fae13f, 128 real PTYs, 6 browsers/all128 terminals: 60 s settle + 60.000372 s sample passed. CPU 24.027599% one core, scheduler activations 5920.4299/s, ~4,670 correlated client replies per socket, stable thread sets/socket/shims, cleanup remaining_shims=[]. Eight SQLite workers consumed 2.142723416 CPU-s total during the window. Load before 1.503/3.558/4.534, after 2.493/3.497/4.452. This reproduces substantial attached idle work without relaxing proof deadlines. Fixed binary rows next. Baseline 6-client diagnostic will use a separately labelled 180 s settle to inspect thread churn; original required 60 s rows remain rejected, not replaced silently.
kayg commented 2026-10-14 17:51:23 +00:00
Author
Owner
Copy link

Partial strict full-duplex matrix (128 real PTYs, 60 s settle + 60 s sample; CPU% one core / scheduler activations per s):

Binary 0 clients 1 client/25 terminals 6 clients/128 terminals
9ab4329c 0 / 0 1.792090 / 385.9115 rejected: thread churn; retries also setup timeouts
17fae13f 0 / 0 5.786500 / 1265.0607 24.027599 / 5920.4299
fixed (runtime cc312976, fixture 8ef4d49e) 0 / 0 2.815845 / 373.6807 rejected: thread churn; identity diagnostic now settling

Fixed one-client SQL worker CPU is exactly 0 ns during the sample. All accepted rows retained their sockets/thread sets/shim identities and cleaned up. Six-client acceptance is unfinished; no deploy recommendation.

Full fast on 8ef4d49e failed one auth socket test (434 pass,1 fail,47 ignored) plus static caps. Detached origin/dev 1b23ef31 isolated test passed, so no #118 baseline waiver is claimed. Initial branch isolated run reused baseline artifact and is excluded; after touching branch Rust inputs and confirmed source rebuild, branch isolated test passes. Full fast rerunning, with source invalidation retained. Raw receipts:
duplex-matrix.txt
auth-timeout-baseline.log
auth-timeout-head-invalidated.log
verify-fast-strict-fixture.log

Partial strict full-duplex matrix (128 real PTYs, 60 s settle + 60 s sample; CPU% one core / scheduler activations per s): | Binary | 0 clients | 1 client/25 terminals | 6 clients/128 terminals | |---|---:|---:|---:| | 9ab4329c | 0 / 0 | 1.792090 / 385.9115 | rejected: thread churn; retries also setup timeouts | | 17fae13f | 0 / 0 | 5.786500 / 1265.0607 | 24.027599 / 5920.4299 | | fixed (runtime cc312976, fixture 8ef4d49e) | 0 / 0 | 2.815845 / 373.6807 | rejected: thread churn; identity diagnostic now settling | Fixed one-client SQL worker CPU is exactly 0 ns during the sample. All accepted rows retained their sockets/thread sets/shim identities and cleaned up. Six-client acceptance is unfinished; no deploy recommendation. Full fast on 8ef4d49e failed one auth socket test (434 pass,1 fail,47 ignored) plus static caps. Detached origin/dev 1b23ef31 isolated test passed, so no #118 baseline waiver is claimed. Initial branch isolated run reused baseline artifact and is excluded; after touching branch Rust inputs and confirmed source rebuild, branch isolated test passes. Full fast rerunning, with source invalidation retained. Raw receipts: [duplex-matrix.txt](https://git.kayg.org/attachments/2d11f5f5-ddb1-4d40-a887-5468c941df46) [auth-timeout-baseline.log](https://git.kayg.org/attachments/7c6e1d89-ca59-467b-acba-ebff8aedb929) [auth-timeout-head-invalidated.log](https://git.kayg.org/attachments/6e3c31f9-8076-4eee-a682-9bb09ab020d8) [verify-fast-strict-fixture.log](https://git.kayg.org/attachments/984e83fa-42c2-4c45-b8f6-a550e34db658)
kayg commented 2026-10-14 17:57:55 +00:00
Author
Owner
Copy link

13b70239 pushed: corrected browser fixture control(Ping) behavior. Production replies Pong then sampleRoundTrip(); previous fixture only replied, omitting fallback sampling when timers are delayed. Timer/receive now share one 25 ms cadence, bounded pending samples, and min(next probe, proof deadline) timer scheduling. Regression red before helper; all3 fixture tests green. Independent source review approves corrected workload, no new P0/P1/P2. Dead-link deadlines unchanged.

Fresh matrix queued, fixed6 first, then0/1/6 across both comparison revisions. Earlier rows stay explicitly labelled pre-correction, not silently presented as the corrected fixture comparison. Candidate6 will capture a separate post-sample frame-pointer profile. Current full gate rerun started before this Python-only commit; final head gate still required. Runtime/static artifact source unchanged.

13b70239 pushed: corrected browser fixture control(Ping) behavior. Production replies Pong then sampleRoundTrip(); previous fixture only replied, omitting fallback sampling when timers are delayed. Timer/receive now share one 25 ms cadence, bounded pending samples, and min(next probe, proof deadline) timer scheduling. Regression red before helper; all3 fixture tests green. Independent source review approves corrected workload, no new P0/P1/P2. Dead-link deadlines unchanged. Fresh matrix queued, fixed6 first, then0/1/6 across both comparison revisions. Earlier rows stay explicitly labelled pre-correction, not silently presented as the corrected fixture comparison. Candidate6 will capture a separate post-sample frame-pointer profile. Current full gate rerun started before this Python-only commit; final head gate still required. Runtime/static artifact source unchanged.
kayg commented 2026-10-14 18:00:55 +00:00
Author
Owner
Copy link

Corrected shared-cadence fixture13b70239, final fixed binary SHA63497f01..., 128 real PTYs, 6 clients/all128 terminals: PASSED 60 s settle +60.000710 s sample. CPU8.225909% one core, scheduler activations1387.4169/s, SQL worker CPU=0ns and activations=0 across all6 SQL workers. ~4670 correlated client replies per socket; stable TIDs/socket/shim identities, cleanup remaining_shims=[]. Load before1.290/2.353/2.869, after1.904/2.297/2.813. Corrected staging/baseline repeats remain necessary for final same-fixture table. Earlier pre-correction staging6 was24.027599%/5920.4299; do not present this mixed-version comparison as final acceptance.

Corrected shared-cadence fixture13b70239, final fixed binary SHA63497f01..., 128 real PTYs, 6 clients/all128 terminals: PASSED 60 s settle +60.000710 s sample. CPU8.225909% one core, scheduler activations1387.4169/s, SQL worker CPU=0ns and activations=0 across all6 SQL workers. ~4670 correlated client replies per socket; stable TIDs/socket/shim identities, cleanup remaining_shims=[]. Load before1.290/2.353/2.869, after1.904/2.297/2.813. Corrected staging/baseline repeats remain necessary for final same-fixture table. Earlier pre-correction staging6 was24.027599%/5920.4299; do not present this mixed-version comparison as final acceptance.
kayg commented 2026-10-14 18:20:16 +00:00
Author
Owner
Copy link

Production6 accounting resolved with inherited perf stat counters (same corrected13b70239 browser fixture): 128PTYS/6clients/all128attached,60s settle+60.003556s counter window PASSED. CPU4.158161% one core,1382.9014 context-switches/s,100% counter coverage. Worker TIDs grew11->13 during window (two new SQLite workers), explaining strict endpoint-set rejections; no exited-worker accounting is discarded by inherited counters. All sockets/shims healthy; cleanup remaining_shims=[]. Load before0.372/1.447/1.762,after0.650/1.296/1.687. Same-counter fixed6/staging6 and production1 rows queued. Original /proc guard remains unchanged. Temporary counter helper will be issue-attached for reproducibility; no runtime or deadline change.

Production6 accounting resolved with inherited perf stat counters (same corrected13b70239 browser fixture): 128PTYS/6clients/all128attached,60s settle+60.003556s counter window PASSED. CPU4.158161% one core,1382.9014 context-switches/s,100% counter coverage. Worker TIDs grew11->13 during window (two new SQLite workers), explaining strict endpoint-set rejections; no exited-worker accounting is discarded by inherited counters. All sockets/shims healthy; cleanup remaining_shims=[]. Load before0.372/1.447/1.762,after0.650/1.296/1.687. Same-counter fixed6/staging6 and production1 rows queued. Original /proc guard remains unchanged. Temporary counter helper will be issue-attached for reproducibility; no runtime or deadline change.
kayg commented 2026-10-14 18:22:26 +00:00
Author
Owner
Copy link

Full fast rerun completed with Rust test PASS (all workspace tests); fmt,VTfmt,compile-out,clippy,cargo-deny PASS. Static builds alone fail unchanged caps. Static x86 output remains byte-identical to measured fixed binary SHA63497f01d7593ece12291b2a660da187b89447ec9001920ad5fec27c17899008,22330768B. Final ordered iterate->fast now running on pushed13b70239; no further source changes planned.

Verbatim rerun Rust summary:
+---------------------------+--------------+
| Rust gate | Result |
+---------------------------+--------------+
| fmt | PASS |
| VT native fmt | PASS |
| CLI plugin compile-out | PASS |
| clippy | PASS |
| test | PASS |
| cargo-deny | PASS |
| static builds | FAIL (1) |
+---------------------------+--------------+

Accounting precision: first production6 perf receipt records two newly created worker TIDs; their SQLite classification was inferred from pool shape, not captured names. Counter helper now captures names at both endpoints. Inherited task-clock/context-switch counts retain all worker work regardless of that classification.

Full fast rerun completed with Rust test PASS (all workspace tests); fmt,VTfmt,compile-out,clippy,cargo-deny PASS. Static builds alone fail unchanged caps. Static x86 output remains byte-identical to measured fixed binary SHA63497f01d7593ece12291b2a660da187b89447ec9001920ad5fec27c17899008,22330768B. Final ordered iterate->fast now running on pushed13b70239; no further source changes planned. Verbatim rerun Rust summary: +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ Accounting precision: first production6 perf receipt records two newly created worker TIDs; their SQLite classification was inferred from pool shape, not captured names. Counter helper now captures names at both endpoints. Inherited task-clock/context-switch counts retain all worker work regardless of that classification.
kayg commented 2026-10-14 18:28:00 +00:00
Author
Owner
Copy link

Counter helper and production6 receipt attached for reproducibility. Fixed6 same-counter row passed:CPU7.381285%,context-switches1341.6783/s,100%counter coverage;stable worker/socket/shim identities;cleanup[]. Load before1.944/1.776/1.755,after0.935/1.500/1.659. CPU remains higher than production6(4.158161%);context switches are slightly lower than production6(1382.9014/s). Corrected staging6 counter row pending; no zero-work claim for attached clients.
bench-perf-stat-source.txt
baseline-six-perf-stat.txt
shared-cadence-red.log
shared-cadence-green.log

Counter helper and production6 receipt attached for reproducibility. Fixed6 same-counter row passed:CPU7.381285%,context-switches1341.6783/s,100%counter coverage;stable worker/socket/shim identities;cleanup[]. Load before1.944/1.776/1.755,after0.935/1.500/1.659. CPU remains higher than production6(4.158161%);context switches are slightly lower than production6(1382.9014/s). Corrected staging6 counter row pending; no zero-work claim for attached clients. [bench-perf-stat-source.txt](https://git.kayg.org/attachments/e72956f2-684f-4529-82e7-486bbe5232c2) [baseline-six-perf-stat.txt](https://git.kayg.org/attachments/f98a814f-6078-4738-9659-640fe651211d) [shared-cadence-red.log](https://git.kayg.org/attachments/1cf1c552-714e-47ef-b5c1-fe60d16840ae) [shared-cadence-green.log](https://git.kayg.org/attachments/3496bd86-6a86-425c-8763-7f761fde883b)
kayg commented 2026-10-14 19:21:03 +00:00
Author
Owner
Copy link

Finish/handoff head13b70239a0afe00ca13d879c6aa0be1da3dc6177, pushed to idle-cpu-199; no deploy or merge. Candidate17fae13f is not idle-quiet with attached clients. Fixes batch native probes, consume Pongs without per-terminal payload notifications, and replace repeated probe SQL authorization/renewal with a watched, non-renewing deadline lease. Payload/backend operations retain fresh authorization. Committed revocation, expiry, blocked writers and stale route generations are covered;25ms cadence and existing dead-link deadlines unchanged.

Final workload:128 real shipping PTYs,0/1/6 real WebSocket clients (1attaches25 terminals,6all128),60s settle+60s sample, locked perf VM only. Corrected fixture shares browser timer/incoming-Ping cadence. Counter-based samples preserve worker churn; endpoint/probe/client failures excluded and retained in raw receipts. The0-client rows are all exactly0 CPU/scheduler events. Primary6 samples:production4.158161%/1382.9014context switches/s,staging17.959773%/5288.1216/s,fixed7.381285%/1341.6783/s. CPU reduction59%,activation reduction75%;fixed CPU still above production. Valid fixed1 repeat2.504415% (/proc) vs4.652394% (inherited counters) and staging6 repeat22.939864% are disclosed, not averaged away. This is loopback server work with40Hz client proofs, not the historical browser profile or a new signed staging/TLS proof.

Profile:136samples,0lost;15.44% sampled time in kernel spin-unlock/wakeup paths (futex/socket), consistent with excessive event/SQL fanout. Stripped userspace symbols limit attribution; real128-route/held-SQL regressions and steady samples prove the repaired paths. Steady fixed1/6 /proc windows show0SQL CPU; not claimed for every repeat.

Checks: final iterate PASS; final fast exit1 solely unchanged static caps. Server435pass,0fail,47ignored; all workspace Rust tests pass. Browser final binary10/12main+2/2unchanged failed-case retry; initial2failures were ERR_NETWORK_CHANGED before sign-in. Three benchmark fixture regressions pass. Independent runtime/fixture source review: no newP0/P1/P2. Author checklist reviewed correctness/concurrency, strict authorization/cancellation, bounded queues, cold/off behavior, secret-free receipts, resource ownership, licensing/docs and measured scope; no foundational engine chosen, no parser change, no UI code change. Independent orchestrator review remains required.

Stripped static x86:22,330,768B vs validated merge-base57b0c3dd22,275,888B (+54,880B); vs staging22,319,696B (+11,072B). >16KB overall growth justified by measured fanout/SQL reduction; own fix delta below16KB. No cap edits. Existing cap excess:x86+126,464B,ARM+119,888B (D263 cutover exception/#198 follow-up remains owner/orchestrator work). Final measured/gated x86 SHA63497f01d7593ece12291b2a660da187b89447ec9001920ad5fec27c17899008 is byte-identical after source-invalidated gate rebuild. No deploy approval claimed.

Final fast Rust summary, verbatim:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

Receipts:
final-cpu-table.txt
shared-cadence-matrix.txt
counter-comparison.txt
staging-one-corrected-counters.txt
fixed-one-corrected-counters.txt
final-binary-sizes.txt
perf-machine.txt
verify-iterate-head13.log
verify-fast-head13.log
final-native-cleanup.txt
final-local-cleanup.txt
candidate-profile-report.txt
candidate-profile-receipt.txt
candidate-final.perf.gz

Finish/handoff head13b70239a0afe00ca13d879c6aa0be1da3dc6177, pushed to idle-cpu-199; no deploy or merge. Candidate17fae13f is not idle-quiet with attached clients. Fixes batch native probes, consume Pongs without per-terminal payload notifications, and replace repeated probe SQL authorization/renewal with a watched, non-renewing deadline lease. Payload/backend operations retain fresh authorization. Committed revocation, expiry, blocked writers and stale route generations are covered;25ms cadence and existing dead-link deadlines unchanged. Final workload:128 real shipping PTYs,0/1/6 real WebSocket clients (1attaches25 terminals,6all128),60s settle+60s sample, locked perf VM only. Corrected fixture shares browser timer/incoming-Ping cadence. Counter-based samples preserve worker churn; endpoint/probe/client failures excluded and retained in raw receipts. The0-client rows are all exactly0 CPU/scheduler events. Primary6 samples:production4.158161%/1382.9014context switches/s,staging17.959773%/5288.1216/s,fixed7.381285%/1341.6783/s. CPU reduction59%,activation reduction75%;fixed CPU still above production. Valid fixed1 repeat2.504415% (/proc) vs4.652394% (inherited counters) and staging6 repeat22.939864% are disclosed, not averaged away. This is loopback server work with40Hz client proofs, not the historical browser profile or a new signed staging/TLS proof. Profile:136samples,0lost;15.44% sampled time in kernel spin-unlock/wakeup paths (futex/socket), consistent with excessive event/SQL fanout. Stripped userspace symbols limit attribution; real128-route/held-SQL regressions and steady samples prove the repaired paths. Steady fixed1/6 /proc windows show0SQL CPU; not claimed for every repeat. Checks: final iterate PASS; final fast exit1 solely unchanged static caps. Server435pass,0fail,47ignored; all workspace Rust tests pass. Browser final binary10/12main+2/2unchanged failed-case retry; initial2failures were ERR_NETWORK_CHANGED before sign-in. Three benchmark fixture regressions pass. Independent runtime/fixture source review: no newP0/P1/P2. Author checklist reviewed correctness/concurrency, strict authorization/cancellation, bounded queues, cold/off behavior, secret-free receipts, resource ownership, licensing/docs and measured scope; no foundational engine chosen, no parser change, no UI code change. Independent orchestrator review remains required. Stripped static x86:22,330,768B vs validated merge-base57b0c3dd22,275,888B (+54,880B); vs staging22,319,696B (+11,072B). >16KB overall growth justified by measured fanout/SQL reduction; own fix delta below16KB. No cap edits. Existing cap excess:x86+126,464B,ARM+119,888B (D263 cutover exception/#198 follow-up remains owner/orchestrator work). Final measured/gated x86 SHA63497f01d7593ece12291b2a660da187b89447ec9001920ad5fec27c17899008 is byte-identical after source-invalidated gate rebuild. No deploy approval claimed. Final fast Rust summary, verbatim: ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ``` Receipts: [final-cpu-table.txt](https://git.kayg.org/attachments/39c92972-f8b0-48d9-ba0d-a79d8bcb7ba1) [shared-cadence-matrix.txt](https://git.kayg.org/attachments/9a6302b3-e915-4f07-a0f8-67e4ac6e620c) [counter-comparison.txt](https://git.kayg.org/attachments/65ac53a3-a51a-4cdc-a8d6-443fba916b48) [staging-one-corrected-counters.txt](https://git.kayg.org/attachments/09e10bdb-be83-44a9-aaeb-84715693553c) [fixed-one-corrected-counters.txt](https://git.kayg.org/attachments/6cc1f657-f561-47c1-b9b9-b17a5e463c95) [final-binary-sizes.txt](https://git.kayg.org/attachments/98f4275f-587e-4ce2-ad72-0ab4c854d070) [perf-machine.txt](https://git.kayg.org/attachments/24dae016-1ad6-49df-9156-68b2b5249145) [verify-iterate-head13.log](https://git.kayg.org/attachments/2a013217-4525-4b40-8d5e-2f21b967db58) [verify-fast-head13.log](https://git.kayg.org/attachments/0345e86c-77c0-471f-9b0c-9748c578ac1e) [final-native-cleanup.txt](https://git.kayg.org/attachments/e7eccb92-1589-413b-96bd-454370915e74) [final-local-cleanup.txt](https://git.kayg.org/attachments/4811278a-7520-4792-8464-3a3e40f352b5) [candidate-profile-report.txt](https://git.kayg.org/attachments/da4d79da-1e73-4964-bb47-f686227f8cdb) [candidate-profile-receipt.txt](https://git.kayg.org/attachments/00febc32-b0d1-4c12-b030-46efbc939392) [candidate-final.perf.gz](https://git.kayg.org/attachments/a3c22011-96ec-441d-abf5-50aab92e5a91)
kayg commented 2026-10-14 19:22:00 +00:00
Author
Owner
Copy link

128 real PTYs; corrected browser cadence; 60 s settle + 60 s sample. Nonzero-client rows use inherited perf stat; zero-client rows use /proc task schedstat. Counts are context switches/s or schedule-ins/s, not hardware interrupts.

Revision Clients Attached CPU% one core Scheduler activations/s Load before -> after (1/5/15m)
9ab4329c 0 0 0.000000 0.0000 0.296/1.068/1.580 -> 0.188/0.888/1.484
9ab4329c 1 25 1.699533 379.6350 2.540/2.289/1.777 -> 0.972/1.884/1.668
9ab4329c 6 128 4.158161 1382.9014 0.372/1.447/1.762 -> 0.650/1.296/1.687
17fae13f 0 0 0.000000 0.0000 0.423/1.028/1.406 -> 0.154/0.838/1.315
17fae13f 1 25 5.710502 1223.0991 0.355/1.526/1.557 -> 0.276/1.291/1.473
17fae13f 6 128 17.959773 5288.1216 0.791/1.279/1.529 -> 0.996/1.231/1.495
13b70239 0 0 0.000000 0.0000 0.326/1.061/2.086 -> 0.118/0.865/1.953
13b70239 1 25 4.652394 492.6610 0.000/0.710/1.304 -> 0.855/0.837/1.309
13b70239 6 128 7.381285 1341.6783 1.944/1.776/1.755 -> 0.935/1.500/1.659

Valid repeat variation: fixed1 /proc=2.504415% /364.9652 schedule-ins/s, inherited-counter repeat=4.652394% /492.6610 context-switches/s; the latter created one SQLite worker during the window. Staging6 counter repeat during profile fixture=22.939864% /5125.2412/s vs primary17.959773% /5288.1216/s. No averaging or rejection of these valid repeats.

Primary6 comparison: fixed vs staging CPU reduction59% and scheduler activation reduction75%; fixed CPU remains above production. Fixed steady /proc1 and6 samples show exactly0 SQL worker CPU; this is not claimed for every repeat.

All accepted rows retained socket/shim identity and passed correlated final replies. Rejected thread-churn/retired-client/setup-timeout rows are preserved in the raw receipts, excluded from this table.

128 real PTYs; corrected browser cadence; 60 s settle + 60 s sample. Nonzero-client rows use inherited perf stat; zero-client rows use /proc task schedstat. Counts are context switches/s or schedule-ins/s, not hardware interrupts. | Revision | Clients | Attached | CPU% one core | Scheduler activations/s | Load before -> after (1/5/15m) | |---|---:|---:|---:|---:|---| | 9ab4329c | 0 | 0 | 0.000000 | 0.0000 | 0.296/1.068/1.580 -> 0.188/0.888/1.484 | | 9ab4329c | 1 | 25 | 1.699533 | 379.6350 | 2.540/2.289/1.777 -> 0.972/1.884/1.668 | | 9ab4329c | 6 | 128 | 4.158161 | 1382.9014 | 0.372/1.447/1.762 -> 0.650/1.296/1.687 | | 17fae13f | 0 | 0 | 0.000000 | 0.0000 | 0.423/1.028/1.406 -> 0.154/0.838/1.315 | | 17fae13f | 1 | 25 | 5.710502 | 1223.0991 | 0.355/1.526/1.557 -> 0.276/1.291/1.473 | | 17fae13f | 6 | 128 | 17.959773 | 5288.1216 | 0.791/1.279/1.529 -> 0.996/1.231/1.495 | | 13b70239 | 0 | 0 | 0.000000 | 0.0000 | 0.326/1.061/2.086 -> 0.118/0.865/1.953 | | 13b70239 | 1 | 25 | 4.652394 | 492.6610 | 0.000/0.710/1.304 -> 0.855/0.837/1.309 | | 13b70239 | 6 | 128 | 7.381285 | 1341.6783 | 1.944/1.776/1.755 -> 0.935/1.500/1.659 | Valid repeat variation: fixed1 /proc=2.504415% /364.9652 schedule-ins/s, inherited-counter repeat=4.652394% /492.6610 context-switches/s; the latter created one SQLite worker during the window. Staging6 counter repeat during profile fixture=22.939864% /5125.2412/s vs primary17.959773% /5288.1216/s. No averaging or rejection of these valid repeats. Primary6 comparison: fixed vs staging CPU reduction59% and scheduler activation reduction75%; fixed CPU remains above production. Fixed steady /proc1 and6 samples show exactly0 SQL worker CPU; this is not claimed for every repeat. All accepted rows retained socket/shim identity and passed correlated final replies. Rejected thread-churn/retired-client/setup-timeout rows are preserved in the raw receipts, excluded from this table.
kayg commented 2026-10-14 19:26:07 +00:00
Author
Owner
Copy link

Cleanup complete at13b70239: worktree clean; no owned native/browser runtime processes on perf VM or build host. Private compiler cache stopped; own target/artifact/runtime/short-path scratch removed; detached baseline removed; owned perf-VM fixture directory removed after receipts attached. Evidence preserved outside checkout and on this issue. No deploy,merge,issue close or other-job process changes.
final-runtime-cleanup.txt
build-cleanup.txt
perf-cleanup.txt

Cleanup complete at13b70239: worktree clean; no owned native/browser runtime processes on perf VM or build host. Private compiler cache stopped; own target/artifact/runtime/short-path scratch removed; detached baseline removed; owned perf-VM fixture directory removed after receipts attached. Evidence preserved outside checkout and on this issue. No deploy,merge,issue close or other-job process changes. [final-runtime-cleanup.txt](https://git.kayg.org/attachments/481fbf21-2073-4a91-8f66-8ca2bd687659) [build-cleanup.txt](https://git.kayg.org/attachments/41b2681b-9b91-49b5-aa04-d93b753ef9fc) [perf-cleanup.txt](https://git.kayg.org/attachments/9359bdf0-4985-4478-bf84-c3b4328743d5)
kayg commented 2026-10-14 19:29:10 +00:00
Author
Owner
Copy link

Continuation start #199: idle-cpu-199 at 13b70239a0. Read repository/process rules and #199 history; no parent is referenced. Import R199-M1/M2 review regressions, reproduce before fixes, then adaptive healthy-idle browser probes while preserving D265/D267 recovery. Heavy checks use build-slot; measurements only on the locked perf VM, 128 PTYs, 0/1/6 clients, 60s settle/sample. Merge origin/dev if advanced, final iterate/fast and size comparison, push only idle-cpu-199. No deploy.

Continuation start #199: idle-cpu-199 at 13b70239a0afe00ca13d879c6aa0be1da3dc6177. Read repository/process rules and #199 history; no parent is referenced. Import R199-M1/M2 review regressions, reproduce before fixes, then adaptive healthy-idle browser probes while preserving D265/D267 recovery. Heavy checks use build-slot; measurements only on the locked perf VM, 128 PTYs, 0/1/6 clients, 60s settle/sample. Merge origin/dev if advanced, final iterate/fast and size comparison, push only idle-cpu-199. No deploy.
kayg commented 2026-10-14 19:38:34 +00:00
Author
Owner
Copy link

Review regressions imported as f56b19b4/545493be/3808308e. R199-M1 reproduces 0/2, fixed d939fdf5: 4 focused controls pass/25 assertions; preserve fresh deadline and only restart an answered lease after local scheduling consumed it. R199-M2 reproduces 0/1 in a pinned Tokio 1.53.2 driver importing shipping source; bounded exact issued-probe correlation passes 3/3, full server gates compiling through build-slot.

Adaptive browser regression first fails all5 cases (40 probes per virtual second). Tests332a71ed and fix85e8701c:13 connection cases/60 assertions pass, strict tracer TypeScript passes. Healthy idle uses40ms; pending input/startup/recent RTT doubt retains25ms. Existing RTT+50ms deadline/correlation unchanged. Timer cancellation on replies avoids obsolete-deadline wakeups. These are deterministic functional counts, not host timing evidence. Perf-VM table and full gates remain pending; no acceptance/deploy claim.

m1-red.log
m1-green.log
m2-driver-red.log
m2-driver-green.log
adaptive-red.log
adaptive-green.log

Review regressions imported as f56b19b4/545493be/3808308e. R199-M1 reproduces 0/2, fixed d939fdf5: 4 focused controls pass/25 assertions; preserve fresh deadline and only restart an answered lease after local scheduling consumed it. R199-M2 reproduces 0/1 in a pinned Tokio 1.53.2 driver importing shipping source; bounded exact issued-probe correlation passes 3/3, full server gates compiling through build-slot. Adaptive browser regression first fails all5 cases (40 probes per virtual second). Tests332a71ed and fix85e8701c:13 connection cases/60 assertions pass, strict tracer TypeScript passes. Healthy idle uses40ms; pending input/startup/recent RTT doubt retains25ms. Existing RTT+50ms deadline/correlation unchanged. Timer cancellation on replies avoids obsolete-deadline wakeups. These are deterministic functional counts, not host timing evidence. Perf-VM table and full gates remain pending; no acceptance/deploy claim. [m1-red.log](https://git.kayg.org/attachments/926e5247-45b1-444b-97ad-5200310b3815) [m1-green.log](https://git.kayg.org/attachments/df9a57ec-56a7-4e46-9e38-e9d26e36f94e) [m2-driver-red.log](https://git.kayg.org/attachments/341d385d-68cf-4b63-804a-fa7f9e929a53) [m2-driver-green.log](https://git.kayg.org/attachments/fe820570-3350-4b41-b2fd-53bf606f7a36) [adaptive-red.log](https://git.kayg.org/attachments/cee60508-b8c9-4f52-b213-996f98fb81c4) [adaptive-green.log](https://git.kayg.org/attachments/7f662f0f-b175-4337-a2d5-e85ec7db1f26)
kayg commented 2026-10-14 19:43:06 +00:00
Author
Owner
Copy link

Steps completed: native exact issued correlation eeede2ed (paused Tokio shipping-source driver3/3; remote all-target Clippy/fmt pass; full server tests compiling). Enable R199-M2 in ordinary gates. Browser input watermark51614850 preserves the highest cursor across replay; failing-before regression25vs39 probes, now26 connection tests/73 assertions pass with strict TypeScript. Fixturecb9dc934 adds explicitly labelled adaptive browser workload and per-window client activity;4 Python tests pass. Merged origin/dev1b23ef31 as6d0439b3: only owner decisions/taste changed; rustls0.23.46 already identical. No runtime conflict or new dependency. Perf VM tools staged; measurement and final gates pending.

Steps completed: native exact issued correlation eeede2ed (paused Tokio shipping-source driver3/3; remote all-target Clippy/fmt pass; full server tests compiling). Enable R199-M2 in ordinary gates. Browser input watermark51614850 preserves the highest cursor across replay; failing-before regression25vs39 probes, now26 connection tests/73 assertions pass with strict TypeScript. Fixturecb9dc934 adds explicitly labelled adaptive browser workload and per-window client activity;4 Python tests pass. Merged origin/dev1b23ef31 as6d0439b3: only owner decisions/taste changed; rustls0.23.46 already identical. No runtime conflict or new dependency. Perf VM tools staged; measurement and final gates pending.
kayg commented 2026-10-14 20:04:03 +00:00
Author
Owner
Copy link

Checkpoint head 96b947fe: browser startup grace fix c38f6f10 and matching fixture 96b947fe. A first timer delayed to virtual1999ms on a healthy1200ms RTT route reproduced a false disconnect; start the bounded unknown-RTT grace when its first probe is issued, matching native behavior. 27 connection tests/75 assertions and strict tracer TypeScript pass; 5 Python fixture tests pass. Existing sampled RTT+50ms deadline is unchanged.

Native remote all-target Clippy/fmt pass. First full server run:434 passed/1 failed/47 ignored; saturated_native_control_queue_does_not_hide_primary_terminal_output lost its connection during setup. Unchanged isolated weighted-scope retry passes; no baseline waiver or full-suite green claimed yet. The enabled R199-M2 workspace regression passes explicitly (1/0), in addition to its three-case paused-clock driver. Final weighted iterate/fast will repeat the complete suite.

Static production/base/fixed builds are serial in one target directory with source invalidation. Perf matrix waits for immutable artifacts, then locks each60s-settle/60s-sample row separately. Gate calibration uses the unchanged SHA119b87ef... latency_proxy.py on the perf VM via a scratch dispatcher; no gate source/threshold edited. Browser plugin unavailable; existing Playwright fixtures use production bundles/real private PTYs on the perf VM. No deploy, production access or acceptance claim.

control-queue-scoped-retry.log
m2-workspace-regression.log
first-delayed-probe-red.log
first-delayed-probe-green.log

Checkpoint head 96b947fe: browser startup grace fix c38f6f10 and matching fixture 96b947fe. A first timer delayed to virtual1999ms on a healthy1200ms RTT route reproduced a false disconnect; start the bounded unknown-RTT grace when its first probe is issued, matching native behavior. 27 connection tests/75 assertions and strict tracer TypeScript pass; 5 Python fixture tests pass. Existing sampled RTT+50ms deadline is unchanged. Native remote all-target Clippy/fmt pass. First full server run:434 passed/1 failed/47 ignored; saturated_native_control_queue_does_not_hide_primary_terminal_output lost its connection during setup. Unchanged isolated weighted-scope retry passes; no baseline waiver or full-suite green claimed yet. The enabled R199-M2 workspace regression passes explicitly (1/0), in addition to its three-case paused-clock driver. Final weighted iterate/fast will repeat the complete suite. Static production/base/fixed builds are serial in one target directory with source invalidation. Perf matrix waits for immutable artifacts, then locks each60s-settle/60s-sample row separately. Gate calibration uses the unchanged SHA119b87ef... latency_proxy.py on the perf VM via a scratch dispatcher; no gate source/threshold edited. Browser plugin unavailable; existing Playwright fixtures use production bundles/real private PTYs on the perf VM. No deploy, production access or acceptance claim. [control-queue-scoped-retry.log](https://git.kayg.org/attachments/89cf54e1-9868-42ba-beec-80c0f52092a2) [m2-workspace-regression.log](https://git.kayg.org/attachments/d5f93e01-b74f-4374-b6d2-192585b0a2bc) [first-delayed-probe-red.log](https://git.kayg.org/attachments/cd6bc3a8-8213-48f2-a400-06e7233d332c) [first-delayed-probe-green.log](https://git.kayg.org/attachments/93fed657-7e01-40dc-95f6-b7a02407632e)
kayg commented 2026-10-14 20:14:12 +00:00
Author
Owner
Copy link

Fresh production9ab4329c static build20333104B; local/VM SHA25672c0881c19fc590c8371a3f86ecc84287678c3db904b4c71bbab6da31933bdac match, stripped static/no PT_INTERP. Perf VM6-client/all128-attached row passes60s settle+60s inherited-counter sample: CPU5.302399% one core,1570.7275 context-switches/s (261.7879/client); equivalent original-cadence driver39.0474 probes/timer-expirations per second per client. Load0.247/1.121/1.210 ->0.089/0.914/1.131. Retained sockets/128 shim identities healthy, cleanup[]. This fresh valid production row differs from prior4.158161%/1382.9014; neither is discarded. Fixed rows remain pending; no acceptance claim.

Only setup fault so far was a missing latency_proxy.py import in the staged driver before any server launch; copied the unchanged dependency and restarted. Build pipeline retains one target directory. Base/head native adapter and build recipe are byte-identical; complete installed archive+headers reused for the same pinned source/x86-musl/ReleaseFast to avoid redundant C compilation. Inputs/archive SHA receipt attached; Rust source invalidated for every comparison build.

continuation-bench-production-6-original.log
native-cache-reuse.log

Fresh production9ab4329c static build20333104B; local/VM SHA25672c0881c19fc590c8371a3f86ecc84287678c3db904b4c71bbab6da31933bdac match, stripped static/no PT_INTERP. Perf VM6-client/all128-attached row passes60s settle+60s inherited-counter sample: CPU5.302399% one core,1570.7275 context-switches/s (261.7879/client); equivalent original-cadence driver39.0474 probes/timer-expirations per second per client. Load0.247/1.121/1.210 ->0.089/0.914/1.131. Retained sockets/128 shim identities healthy, cleanup[]. This fresh valid production row differs from prior4.158161%/1382.9014; neither is discarded. Fixed rows remain pending; no acceptance claim. Only setup fault so far was a missing latency_proxy.py import in the staged driver before any server launch; copied the unchanged dependency and restarted. Build pipeline retains one target directory. Base/head native adapter and build recipe are byte-identical; complete installed archive+headers reused for the same pinned source/x86-musl/ReleaseFast to avoid redundant C compilation. Inputs/archive SHA receipt attached; Rust source invalidated for every comparison build. [continuation-bench-production-6-original.log](https://git.kayg.org/attachments/1e123ab1-8dc9-40b2-b54e-9be9b913ad66) [native-cache-reuse.log](https://git.kayg.org/attachments/65fa43ac-d5e4-4357-946e-05233d0d0d6e)
kayg commented 2026-10-14 20:32:21 +00:00
Author
Owner
Copy link

Orchestrator, 02:02 IST — evidence for the liveness deadline (relevant to the switch-over and R199-M1): the app-window UI work on the perf VM (4 vCPU, under load from 1–10 decoding streams) saw the browser liveness check — deadline ≈ last RTT + 50 ms ≈ 51 ms at 1 ms RTT — repeatedly drop healthy sockets while windows opened. A deadline derived from one sample of a tiny RTT is too tight under CPU load (the UX audit #212 gap 8 saw 3–5 reconnects in 30 s idle on a loaded host). The deadline must use a smoothed RTT plus variance with a floor (and be measured from when a probe was actually sent, not when a timer fired), without weakening detection of dead links. Please make sure the switch-over round covers it.

Orchestrator, 02:02 IST — evidence for the liveness deadline (relevant to the switch-over and R199-M1): the app-window UI work on the perf VM (4 vCPU, under load from 1–10 decoding streams) saw the browser liveness check — deadline ≈ last RTT + 50 ms ≈ 51 ms at 1 ms RTT — repeatedly drop healthy sockets while windows opened. A deadline derived from one sample of a tiny RTT is too tight under CPU load (the UX audit #212 gap 8 saw 3–5 reconnects in 30 s idle on a loaded host). The deadline must use a smoothed RTT plus variance with a floor (and be measured from when a probe was actually sent, not when a timer fired), without weakening detection of dead links. Please make sure the switch-over round covers it.
kayg commented 2026-10-14 20:54:07 +00:00
Author
Owner
Copy link

CPU acceptance remains RED on96b947fe: fresh1 client1.746226%/340.1169ctx/s vs production1.569521%/384.2169;6 clients6.810104%/1281.4161 vs production5.302399%/1570.7275. Activations pass, CPU fails both (4-case comparison oracle:2 pass/2 fail). Fixed0 clients is exactly0 CPU/0schedule-ins. A six-client repeat is valid6.252996%; all SQL workers record0 CPU and0schedule-ins; four Tokio workers account for the CPU. Earlier retired-client row remains invalid/excluded.

Separate10s fp profile:55 samples,0lost; coarse attribution includes memset12.7% and kernel wake/scheduling paths. Link-map-only rebuild is SHA-identical to measured83a4bac1...; corrected executable file-offset→VMA bias is+0x1000. No stack payloads captured. Profile is too small for precise percentages; evidence points at transport scheduling/IO rather than SQL.

Next trial: co-poll the existing read and proof futures on one socket-owner Tokio task, retaining independent polling when either awaits, bounded queues, serialized writes, authority cancellation and identical probe/deadline/correlation behavior. This targets cross-worker wakeups and split-half contention without changing wire semantics or foundational engines. Benchmark remains the failing-before oracle; existing held-SQL/blocked-writer/recovery tests must remain green, then rerun the same VM rows.

idle-cpu-acceptance-red.log
candidate-six-thread-profile.log
candidate-active-symbols.txt
candidate-map-build.log

CPU acceptance remains RED on96b947fe: fresh1 client1.746226%/340.1169ctx/s vs production1.569521%/384.2169;6 clients6.810104%/1281.4161 vs production5.302399%/1570.7275. Activations pass, CPU fails both (4-case comparison oracle:2 pass/2 fail). Fixed0 clients is exactly0 CPU/0schedule-ins. A six-client repeat is valid6.252996%; all SQL workers record0 CPU and0schedule-ins; four Tokio workers account for the CPU. Earlier retired-client row remains invalid/excluded. Separate10s fp profile:55 samples,0lost; coarse attribution includes memset12.7% and kernel wake/scheduling paths. Link-map-only rebuild is SHA-identical to measured83a4bac1...; corrected executable file-offset→VMA bias is+0x1000. No stack payloads captured. Profile is too small for precise percentages; evidence points at transport scheduling/IO rather than SQL. Next trial: co-poll the existing read and proof futures on one socket-owner Tokio task, retaining independent polling when either awaits, bounded queues, serialized writes, authority cancellation and identical probe/deadline/correlation behavior. This targets cross-worker wakeups and split-half contention without changing wire semantics or foundational engines. Benchmark remains the failing-before oracle; existing held-SQL/blocked-writer/recovery tests must remain green, then rerun the same VM rows. [idle-cpu-acceptance-red.log](https://git.kayg.org/attachments/07aa8375-0e1b-4e37-ade1-e38bc11d213d) [candidate-six-thread-profile.log](https://git.kayg.org/attachments/8902d005-b046-491f-93e3-1df1e132bfb3) [candidate-active-symbols.txt](https://git.kayg.org/attachments/00b05f00-d0fa-4522-9168-4ad1c7a71302) [candidate-map-build.log](https://git.kayg.org/attachments/efd762ef-698e-4577-8e61-798c7f507041)
kayg commented 2026-10-14 21:00:53 +00:00
Author
Owner
Copy link

Source finding after symbolication: the pinned framing engine zero-initializes its receive window before each read attempt, including WouldBlock; default128KiB (tungstenite0.29.0 src/protocol/frame/mod.rs read_in:236, axum0.8.9 WebSocketUpgrade::read_buffer_size). The sampled memset IPs are real (link-map-only rebuild SHA-identical83a4bac1...), not a terminal renderer; executable segment file-offset→VMA bias+0x1000 resolves them. SQL workers remain exactly0CPU.

Trying the smaller, isolated configuration change first:32KiB WebSocket receive chunks, enough for the existing16KiB browser Input plus envelope; action JSON is bounded8KiB. Existing4MiB wire/message limits and incremental assembly are unchanged. No parser, protocol, probe interval, deadline, authority or worker scheduling change. This reduces per-poll clearing and baseline receive storage by96KiB/socket (576KiB at6;24MiB at256). Native gates and fresh6/1/0 VM rows run before committing. The four-case healthy-row comparison is already red on the pre-change binary. Co-polling remains a separate possible experiment if this is insufficient.

Source finding after symbolication: the pinned framing engine zero-initializes its receive window before each read attempt, including WouldBlock; default128KiB (tungstenite0.29.0 src/protocol/frame/mod.rs read_in:236, axum0.8.9 WebSocketUpgrade::read_buffer_size). The sampled memset IPs are real (link-map-only rebuild SHA-identical83a4bac1...), not a terminal renderer; executable segment file-offset→VMA bias+0x1000 resolves them. SQL workers remain exactly0CPU. Trying the smaller, isolated configuration change first:32KiB WebSocket receive chunks, enough for the existing16KiB browser Input plus envelope; action JSON is bounded8KiB. Existing4MiB wire/message limits and incremental assembly are unchanged. No parser, protocol, probe interval, deadline, authority or worker scheduling change. This reduces per-poll clearing and baseline receive storage by96KiB/socket (576KiB at6;24MiB at256). Native gates and fresh6/1/0 VM rows run before committing. The four-case healthy-row comparison is already red on the pre-change binary. Co-polling remains a separate possible experiment if this is insufficient.
kayg commented 2026-10-14 21:18:17 +00:00
Author
Owner
Copy link

Safe diagnostics identify the retired sample: RTT estimate1.995ms, last-Pong age53.636ms at expiry, pending probe age10.483ms; its valid reply arrives at19.013ms after send. Fixed40ms idle spacing spent most of the50ms allowance before probing. Deterministic new tests reproduce false retirement for a1ms→20ms RTT step at1/28 streams. Two high-RTT resource cases also fail the intended RTT-aware budget (17 pass/4 fail).

Additive wire§9 clarification before implementation, within the approved adaptive brief: startup, pending input and recent link doubt retain25ms pipelining. Healthy idle keeps at most one outstanding probe; next sample is due at max(last send+25ms,last correlated answer+10ms). The existing last-answer+measured-RTT+50ms expiry, bounded unknown grace, exact correlation and no-late-revival rule are unchanged. This leaves40ms of scheduling allowance after a full RTT at high latency; short local links retain the original25ms minimum spacing. All missing-direction phase tests remain strict. No wire tags/layouts, capabilities, engines or authority rules change. The arbitrary25Hz test bound is replaced with ceil(1000/max(25,RTT+10)), independent of stream count; CPU-vs-production acceptance limits are unchanged.

Read-window32KiB trial is independently still under native gates. Its first wire-fixture row is invalid due to the above client deadline and remains excluded; rerun both prior/native32KiB artifacts with the corrected client policy before claiming its CPU effect. Actual browser before/after measurements remain required.

Safe diagnostics identify the retired sample: RTT estimate1.995ms, last-Pong age53.636ms at expiry, pending probe age10.483ms; its valid reply arrives at19.013ms after send. Fixed40ms idle spacing spent most of the50ms allowance before probing. Deterministic new tests reproduce false retirement for a1ms→20ms RTT step at1/28 streams. Two high-RTT resource cases also fail the intended RTT-aware budget (17 pass/4 fail). Additive wire§9 clarification before implementation, within the approved adaptive brief: startup, pending input and recent link doubt retain25ms pipelining. Healthy idle keeps at most one outstanding probe; next sample is due at max(last send+25ms,last correlated answer+10ms). The existing last-answer+measured-RTT+50ms expiry, bounded unknown grace, exact correlation and no-late-revival rule are unchanged. This leaves40ms of scheduling allowance after a full RTT at high latency; short local links retain the original25ms minimum spacing. All missing-direction phase tests remain strict. No wire tags/layouts, capabilities, engines or authority rules change. The arbitrary25Hz test bound is replaced with ceil(1000/max(25,RTT+10)), independent of stream count; CPU-vs-production acceptance limits are unchanged. Read-window32KiB trial is independently still under native gates. Its first wire-fixture row is invalid due to the above client deadline and remains excluded; rerun both prior/native32KiB artifacts with the corrected client policy before claiming its CPU effect. Actual browser before/after measurements remain required.
kayg commented 2026-10-14 21:26:29 +00:00
Author
Owner
Copy link

Checkpoint: 339da0da fixes idle cadence after an RTT step without extending D265/D267 response deadlines (29 connection tests, 79 assertions; TypeScript check passes). 118fc7fd mirrors the schedule in the 128-PTY driver (5 fixture tests pass); 2060b158 records the proposed additive contract clarification. 37e4a482 bounds WebSocket read initialization to 32 KiB; frame/message limits unchanged, 96 KiB less read capacity per socket. fmt and touched-server clippy pass; native suite 434 pass, one scratch path length failure, 47 ignored. The unchanged failing socket test passes using a short scratch-directory path. Corrected-cadence VM matrix and real-browser restart tests are running; CPU acceptance remains unproven. No deploy.

Checkpoint: 339da0da fixes idle cadence after an RTT step without extending D265/D267 response deadlines (29 connection tests, 79 assertions; TypeScript check passes). 118fc7fd mirrors the schedule in the 128-PTY driver (5 fixture tests pass); 2060b158 records the proposed additive contract clarification. 37e4a482 bounds WebSocket read initialization to 32 KiB; frame/message limits unchanged, 96 KiB less read capacity per socket. fmt and touched-server clippy pass; native suite 434 pass, one scratch path length failure, 47 ignored. The unchanged failing socket test passes using a short scratch-directory path. Corrected-cadence VM matrix and real-browser restart tests are running; CPU acceptance remains unproven. No deploy.
kayg commented 2026-10-14 21:29:38 +00:00
Author
Owner
Copy link

32 KiB read-window trial: 128 PTYs / 6 clients, 60 s settling + 60.0026 s sample, CPU 5.48695% one core, 1319.993 schedule-ins/s, no SQL worker CPU/schedule-ins, cleanup retained no shims. This row still used the previous 40 ms fixture: the updated script was first copied beside the helper rather than into its imported scripts/ directory. The imported path is now corrected. It is a valid 40 ms diagnostic, not final cadence acceptance; CPU remains above the fresh production 5.30240% reference. Next measured trial co-polls the independent read/proof futures in one task to avoid cross-worker transfers while keeping bounded queues and deadlines. Production comparison budgets remain unchanged.

cadence-jitter-red.log
cadence-jitter-green.log
read32k-6-answer-paced.log

32 KiB read-window trial: 128 PTYs / 6 clients, 60 s settling + 60.0026 s sample, CPU 5.48695% one core, 1319.993 schedule-ins/s, no SQL worker CPU/schedule-ins, cleanup retained no shims. This row still used the previous 40 ms fixture: the updated script was first copied beside the helper rather than into its imported scripts/ directory. The imported path is now corrected. It is a valid 40 ms diagnostic, not final cadence acceptance; CPU remains above the fresh production 5.30240% reference. Next measured trial co-polls the independent read/proof futures in one task to avoid cross-worker transfers while keeping bounded queues and deadlines. Production comparison budgets remain unchanged. [cadence-jitter-red.log](https://git.kayg.org/attachments/6c0207eb-d8e0-44e2-bd93-e022a7203048) [cadence-jitter-green.log](https://git.kayg.org/attachments/e511872c-da81-4a9f-9f37-f1b323385b0a) [read32k-6-answer-paced.log](https://git.kayg.org/attachments/227a5c0d-bd81-4a67-87ec-b5c886b2c503)
kayg commented 2026-10-14 21:33:33 +00:00
Author
Owner
Copy link

37e4a482 checkpoint: the corrected production browser bundle passes all 12 real-PTY recovery cases on the perf VM, native read32k SHA8543164933dd2a4dd960dc2ac8825d6763f406543cb04b26388af55fcbc843be.
restart.spec.ts: 2 passed
restart-liveness-review.spec.ts: 4 passed
restart-three-session.spec.ts: 6 passed
Fixture mode uses isolated direct server-PID restart (not actual systemd service control); the orchestrator signed staging proof remains next. origin/dev remains1b23ef31 and is included. Browser CPU/wakeup and final native CPU acceptance still pending. Co-polling native trial not committed; its first test filter matched no tests and will be corrected before acceptance.

37e4a482 checkpoint: the corrected production browser bundle passes all 12 real-PTY recovery cases on the perf VM, native read32k SHA8543164933dd2a4dd960dc2ac8825d6763f406543cb04b26388af55fcbc843be. restart.spec.ts: 2 passed restart-liveness-review.spec.ts: 4 passed restart-three-session.spec.ts: 6 passed Fixture mode uses isolated direct server-PID restart (not actual systemd service control); the orchestrator signed staging proof remains next. origin/dev remains1b23ef31 and is included. Browser CPU/wakeup and final native CPU acceptance still pending. Co-polling native trial not committed; its first test filter matched no tests and will be corrected before acceptance.
kayg commented 2026-10-14 21:41:38 +00:00
Author
Owner
Copy link

Adaptive browser result on the perf VM,128 real PTYs,150 ms message RTT,one attached browser,60 s settle+60 s sample; native read32k identical for both samples. Old/new control probes36.0281→6.3264/s; liveness timer callbacks65.8462→8.10779/s; whole-owned-browser process CPU11.8528→7.38024% one core; process schedule-ins902.128→476.567/s. Both samples retained the socket,128 PTYs,typing afterward,title infraternal,zero page errors. No deadline increase. CPU counters cover100% of each sample. Native server acceptance remains pending.

Rejected uncommitted co-poll experiment: six-client fixture retired routes at51–53 ms with1 ms learned RTT; socket suite37 pass/2 fail/2 ignored (idle socket after31 s clock advance and fresh-delete). Separate transport tasks restored. No relaxed deadlines or test skips. Scalar Pong consumption is the next narrower measured trial.

final-browser-idle-old.log
final-browser-idle-new.log
copoll-6-answer-paced.log
copoll-socket-configured.log

Adaptive browser result on the perf VM,128 real PTYs,150 ms message RTT,one attached browser,60 s settle+60 s sample; native read32k identical for both samples. Old/new control probes36.0281→6.3264/s; liveness timer callbacks65.8462→8.10779/s; whole-owned-browser process CPU11.8528→7.38024% one core; process schedule-ins902.128→476.567/s. Both samples retained the socket,128 PTYs,typing afterward,title infraternal,zero page errors. No deadline increase. CPU counters cover100% of each sample. Native server acceptance remains pending. Rejected uncommitted co-poll experiment: six-client fixture retired routes at51–53 ms with1 ms learned RTT; socket suite37 pass/2 fail/2 ignored (idle socket after31 s clock advance and fresh-delete). Separate transport tasks restored. No relaxed deadlines or test skips. Scalar Pong consumption is the next narrower measured trial. [final-browser-idle-old.log](https://git.kayg.org/attachments/2eff1b8c-1994-4e5b-83f2-89d7763dbd62) [final-browser-idle-new.log](https://git.kayg.org/attachments/e18f4d61-d254-4195-90a1-512499eb9c44) [copoll-6-answer-paced.log](https://git.kayg.org/attachments/cdc4da61-15e2-4d62-a6b4-cd8e88998895) [copoll-socket-configured.log](https://git.kayg.org/attachments/6711e3fb-676f-4c51-af06-748e9b29c9de)
kayg commented 2026-10-14 21:45:40 +00:00
Author
Owner
Copy link

Browser receipts for37e4a482:12/12 real-engine recovery cases and5/5 targeted mock cases pass on the perf VM. Screenshot is the production bundle after the adaptive128-PTY sample; socket live,one visible window,typing verified,zero page errors. Visual-quality judgment left to owner. Native scalar-Pong trial remains uncommitted; full crate tests running.

idle-browser-ready.png
final-browser-mock.log
final-browser-restart.spec.ts.log
final-browser-restart-liveness-review.spec.ts.log
final-browser-restart-three-session.spec.ts.log

Browser receipts for37e4a482:12/12 real-engine recovery cases and5/5 targeted mock cases pass on the perf VM. Screenshot is the production bundle after the adaptive128-PTY sample; socket live,one visible window,typing verified,zero page errors. Visual-quality judgment left to owner. Native scalar-Pong trial remains uncommitted; full crate tests running. ![idle-browser-ready.png](https://git.kayg.org/attachments/1d451170-4345-4078-adf7-980d48ec249f) [final-browser-mock.log](https://git.kayg.org/attachments/126d4626-a7db-466b-8bdc-28219984ab69) [final-browser-restart.spec.ts.log](https://git.kayg.org/attachments/20bdb206-a55f-4325-9f27-cfd85c76c41b) [final-browser-restart-liveness-review.spec.ts.log](https://git.kayg.org/attachments/d704f915-559d-49a4-9fe2-a54bbd095330) [final-browser-restart-three-session.spec.ts.log](https://git.kayg.org/attachments/9a65c86f-20b2-4fd3-8412-5195df6ef922)
kayg commented 2026-10-14 21:49:59 +00:00
Author
Owner
Copy link

5e0184c9 consumes validated scalar route Pongs directly, preserving the existing shared-codec admission and exact issued-probe checks. No payload, transport-task, deadline or dependency changes. Fmt and touched-server clippy pass. Full native server unit suite:435 passed;0 failed;47 ignored. The idle-socket and fresh-delete cases which failed in the rejected co-poll experiment both pass with separate transport tasks restored. Static release build and corrected-cadence VM matrix next. CPU acceptance remains unproven; no deploy.

5e0184c9 consumes validated scalar route Pongs directly, preserving the existing shared-codec admission and exact issued-probe checks. No payload, transport-task, deadline or dependency changes. Fmt and touched-server clippy pass. Full native server unit suite:435 passed;0 failed;47 ignored. The idle-socket and fresh-delete cases which failed in the rejected co-poll experiment both pass with separate transport tasks restored. Static release build and corrected-cadence VM matrix next. CPU acceptance remains unproven; no deploy.
kayg commented 2026-10-14 22:02:34 +00:00
Author
Owner
Copy link

d28e63f7 completes M1 for a partially late local callback: with RTT1 ms, a fully answered route and11 ms late next callback, a20 ms reply was rejected even though no probe was outstanding during the callback delay. The two new regressions fail before the fix;31 connection tests/85 assertions and TypeScript checking pass afterward. Preserve only the scheduled reply allowance for pre-issuance local delay; cap at latest actual answer and never move a pending deadline. All existing directional-failure phase tests remain green. ee9dcb5a mirrors it in the minimal browser driver;6 fixture tests pass. Native binary remains5e0184c9, SHA8596d3579750738e5a6eb1f4fca11c1fd0ed9cece7f90dec841073de2a4a6f03. The earlier six-client scalar run retired routes and is invalid; final matrix repeats the corrected driver, without changing CPU/activation budgets.

late-timer-response-red.log
late-timer-response-green.log
late-timer-fixture-red.log
late-timer-fixture-green.log
scalar-6-answer-paced.log

d28e63f7 completes M1 for a partially late local callback: with RTT1 ms, a fully answered route and11 ms late next callback, a20 ms reply was rejected even though no probe was outstanding during the callback delay. The two new regressions fail before the fix;31 connection tests/85 assertions and TypeScript checking pass afterward. Preserve only the scheduled reply allowance for pre-issuance local delay; cap at latest actual answer and never move a pending deadline. All existing directional-failure phase tests remain green. ee9dcb5a mirrors it in the minimal browser driver;6 fixture tests pass. Native binary remains5e0184c9, SHA8596d3579750738e5a6eb1f4fca11c1fd0ed9cece7f90dec841073de2a4a6f03. The earlier six-client scalar run retired routes and is invalid; final matrix repeats the corrected driver, without changing CPU/activation budgets. [late-timer-response-red.log](https://git.kayg.org/attachments/01973932-d81e-4ba1-a317-289bb3f2c3ec) [late-timer-response-green.log](https://git.kayg.org/attachments/11bcc92f-74a6-4526-8047-c6fb87f86f8a) [late-timer-fixture-red.log](https://git.kayg.org/attachments/3a9df3a0-cd56-408f-ba16-3179719f9ccc) [late-timer-fixture-green.log](https://git.kayg.org/attachments/6b7719b0-4bea-4155-afa4-f517dae14338) [scalar-6-answer-paced.log](https://git.kayg.org/attachments/c6bb30f2-ea48-47e8-8f8c-c3d43f47f66c)
kayg commented 2026-10-14 22:07:44 +00:00
Author
Owner
Copy link

CPU acceptance still red at ee9dcb5a: valid128-PTY/one-client sample (25 attached sessions),60 s settle+60.002609 s sample. Native scalar CPU2.132334% one core vs production1.569521%; schedule-ins351.8014/s vs production384.2169/s. All SQLite workers0 CPU and0 schedule-ins; client38.5317 probes/timer expirations per second; socket retained, cleanup no shims. Perf counters100% coverage. Local final gate run interrupted before Rust checks because further native work is required; not a gate failure waiver. Higher-sample997 Hz20 s address-only profile queued separately after an uncontaminated CPU sample, with an exact-binary linker map building locally.

scalar-1-answer-paced.log
perf-vm-environment-final.txt

CPU acceptance still red at ee9dcb5a: valid128-PTY/one-client sample (25 attached sessions),60 s settle+60.002609 s sample. Native scalar CPU2.132334% one core vs production1.569521%; schedule-ins351.8014/s vs production384.2169/s. All SQLite workers0 CPU and0 schedule-ins; client38.5317 probes/timer expirations per second; socket retained, cleanup no shims. Perf counters100% coverage. Local final gate run interrupted before Rust checks because further native work is required; not a gate failure waiver. Higher-sample997 Hz20 s address-only profile queued separately after an uncontaminated CPU sample, with an exact-binary linker map building locally. [scalar-1-answer-paced.log](https://git.kayg.org/attachments/6b3250e2-e032-4e8b-a9fa-fa20e2745b1b) [perf-vm-environment-final.txt](https://git.kayg.org/attachments/6afc9afa-2be1-4eed-8e2b-8d9476d6fe7c)
kayg commented 2026-10-14 22:42:51 +00:00
Author
Owner
Copy link

Detailed scalar profile:1097 samples,0 lost,997 Hz for20 s after (not inside) the60 s CPU sample. Exact link-map build SHA8596d3579750738e5a6eb1f4fca11c1fd0ed9cece7f90dec841073de2a4a6f03 matches the measured ELF. Analysis-only labels are never executed. Kernel sendto/packet-processing22.79% inclusive; scheduler wake/switch19.14% leaf; memset1.28% leaf. Repeat six-client CPU5.473108% and1379.5137 schedule-ins/s; earlier6.652396% retained. TCP-FD-filtered post-sample diagnostic confirms78.489 TCP sends/s for one client; CPU for that independent repeat2.719408%, not accepted as passing.

Grouped-write helper committed4cda2f0b (11 relevant tests pass,3 diagnostics ignored). Uncommitted ready-proof grouping hit the existing idle-socket-after31s-clock-jump case (38 pass/1 fail/2 ignored), revealing that the proof lease and initial socket expiry were queried after Welcome. Candidate correctness fix prepares the non-renewing lease before Welcome and reuses that validated expiry; removes two redundant initial reads. Clippy passes and the formerly failing case now passes; full socket suite still running. No deadline weakening, timing cap changes or new dependencies.

scalar-active-symbolized-flat.txt
scalar-active-symbolized-callgraph.txt
scalar-six-higher-profile.log
scalar-one-tcp-diagnostic.log
grouped-gates.log

Detailed scalar profile:1097 samples,0 lost,997 Hz for20 s after (not inside) the60 s CPU sample. Exact link-map build SHA8596d3579750738e5a6eb1f4fca11c1fd0ed9cece7f90dec841073de2a4a6f03 matches the measured ELF. Analysis-only labels are never executed. Kernel sendto/packet-processing22.79% inclusive; scheduler wake/switch19.14% leaf; memset1.28% leaf. Repeat six-client CPU5.473108% and1379.5137 schedule-ins/s; earlier6.652396% retained. TCP-FD-filtered post-sample diagnostic confirms78.489 TCP sends/s for one client; CPU for that independent repeat2.719408%, not accepted as passing. Grouped-write helper committed4cda2f0b (11 relevant tests pass,3 diagnostics ignored). Uncommitted ready-proof grouping hit the existing idle-socket-after31s-clock-jump case (38 pass/1 fail/2 ignored), revealing that the proof lease and initial socket expiry were queried after Welcome. Candidate correctness fix prepares the non-renewing lease before Welcome and reuses that validated expiry; removes two redundant initial reads. Clippy passes and the formerly failing case now passes; full socket suite still running. No deadline weakening, timing cap changes or new dependencies. [scalar-active-symbolized-flat.txt](https://git.kayg.org/attachments/22cf096e-f541-4f0d-a1ea-704e986f0004) [scalar-active-symbolized-callgraph.txt](https://git.kayg.org/attachments/b3e1737f-6686-402c-bc65-1b8f55eb3cf1) [scalar-six-higher-profile.log](https://git.kayg.org/attachments/9bca1fbf-57e3-4719-84eb-9731c8d77f83) [scalar-one-tcp-diagnostic.log](https://git.kayg.org/attachments/d6627d63-5cc8-4608-9dcd-b7c24fd1a878) [grouped-gates.log](https://git.kayg.org/attachments/c8fb59ac-25a2-44d3-9891-71041913fb87)
kayg commented 2026-10-14 22:50:59 +00:00
Author
Owner
Copy link

29b47d57 fixes the post-Welcome authority initialization race: subscribe/read the non-renewing proof lease before Welcome, reuse its validated absolute expiry for the shared socket loop, and remove two redundant initial reads. Formatting/clippy pass;39 socket tests pass,0 fail,2 ignored, including the former31 s idle-clock failure.

395c5223 removes the unused group-flush experiment (9 relevant WebSocket tests pass,3 diagnostics ignored). Repeating co-polling in isolation is now justified by two independently fixed confounders: native proof initialization after Welcome and browser pre-issuance callback delay. Current uncommitted co-polling keeps original timers, messages, queues and correlation; read/proof futures share one task with cooperative read-budget yielding. Its39 active socket tests now pass; native correlation and static build are running. CPU acceptance remains red until measured. No deploy; branch not yet pushed.

handshake-proof-init-gates.log
remove-group-helper-gates.log

29b47d57 fixes the post-Welcome authority initialization race: subscribe/read the non-renewing proof lease before Welcome, reuse its validated absolute expiry for the shared socket loop, and remove two redundant initial reads. Formatting/clippy pass;39 socket tests pass,0 fail,2 ignored, including the former31 s idle-clock failure. 395c5223 removes the unused group-flush experiment (9 relevant WebSocket tests pass,3 diagnostics ignored). Repeating co-polling in isolation is now justified by two independently fixed confounders: native proof initialization after Welcome and browser pre-issuance callback delay. Current uncommitted co-polling keeps original timers, messages, queues and correlation; read/proof futures share one task with cooperative read-budget yielding. Its39 active socket tests now pass; native correlation and static build are running. CPU acceptance remains red until measured. No deploy; branch not yet pushed. [handshake-proof-init-gates.log](https://git.kayg.org/attachments/46cfbe13-0041-4bfc-b835-7f54577597f5) [remove-group-helper-gates.log](https://git.kayg.org/attachments/3d26924e-5ca7-445b-b912-154ffecf06fb)
kayg commented 2026-10-14 23:01:08 +00:00
Author
Owner
Copy link

Post-initialization co-poll trial now passes all39 active socket tests (2 ignored) and all3 native liveness/correlation tests; fmt/clippy pass. Static artifact SHA1ab1611b975070a8fe806be859e3af567c3f964941ef52e9f235a126c028f9e0,22330736 B,merge-base22277392 B,delta53344 B. Source is395c5223 plus uncommitted one-task read/proof futures with cooperative read-budget yielding; no timer/message/deadline changes. One-client VM measurement running. Fresh production9ab references:one client1.431775% CPU/386.7844 schedule-ins/s and38.5447 TCP sends/s; six clients4.495748%/1557.1787. Earlier references retained, including original prior-run6-client4.158%. CPU acceptance has not been claimed.

The former co-poll failures were confounded by the independently fixed post-Welcome initial authority read and browser late pre-issuance callback. New39-test green run confirms those failures are fixed; release-stage recovery tests and final repository gates remain after CPU acceptance.

Post-initialization co-poll trial now passes all39 active socket tests (2 ignored) and all3 native liveness/correlation tests; fmt/clippy pass. Static artifact SHA1ab1611b975070a8fe806be859e3af567c3f964941ef52e9f235a126c028f9e0,22330736 B,merge-base22277392 B,delta53344 B. Source is395c5223 plus uncommitted one-task read/proof futures with cooperative read-budget yielding; no timer/message/deadline changes. One-client VM measurement running. Fresh production9ab references:one client1.431775% CPU/386.7844 schedule-ins/s and38.5447 TCP sends/s; six clients4.495748%/1557.1787. Earlier references retained, including original prior-run6-client4.158%. CPU acceptance has not been claimed. The former co-poll failures were confounded by the independently fixed post-Welcome initial authority read and browser late pre-issuance callback. New39-test green run confirms those failures are fixed; release-stage recovery tests and final repository gates remain after CPU acceptance.
kayg commented 2026-10-14 23:05:02 +00:00
Author
Owner
Copy link

Co-polling now passes correctness but fails performance: one-client CPU4.395714%,370.5622 schedule-ins/s,77.5228 TCP sends/s, all clients retained and cleanup empty. Rejected and restored separate tasks.

Additive wire §9 implementation clarification proposed before code: after learning RTT, the server can keep one browser-control probe in flight, start its next fallback no later than25 ms after the correlated answer, and pull that probe forward into an already-ready reply once25 ms has elapsed since its previous send. Give that serial browser-control probe RTT+25 ms to answer, with an exact response-deadline timer: maximum healthy-idle detection gap is25+(RTT+25)=RTT+50 ms. Startup stays pipelined at25 ms until the first sample. Per-PTY and daemon probes keep their existing policy. No timed coalescing delay, no wire layout/capability change, no ordinary-traffic readiness renewal, and no pending deadline extension. Accepted Pongs wake scheduling; unsolicited/duplicate Pongs do not. Test the deadline algebra and RTT step, then all socket/restart cases and the same VM matrix. The current failing CPU and syscall receipts are the before-change resource oracle.

Co-polling now passes correctness but fails performance: one-client CPU4.395714%,370.5622 schedule-ins/s,77.5228 TCP sends/s, all clients retained and cleanup empty. Rejected and restored separate tasks. Additive wire §9 implementation clarification proposed before code: after learning RTT, the server can keep one browser-control probe in flight, start its next fallback no later than25 ms after the correlated answer, and pull that probe forward into an already-ready reply once25 ms has elapsed since its previous send. Give that serial browser-control probe RTT+25 ms to answer, with an exact response-deadline timer: maximum healthy-idle detection gap is25+(RTT+25)=RTT+50 ms. Startup stays pipelined at25 ms until the first sample. Per-PTY and daemon probes keep their existing policy. No timed coalescing delay, no wire layout/capability change, no ordinary-traffic readiness renewal, and no pending deadline extension. Accepted Pongs wake scheduling; unsolicited/duplicate Pongs do not. Test the deadline algebra and RTT step, then all socket/restart cases and the same VM matrix. The current failing CPU and syscall receipts are the before-change resource oracle.
kayg commented 2026-10-14 23:25:55 +00:00
Author
Owner
Copy link

The first serial-control prototype is valid but not accepted: one-client CPU1.996258%,409.2478 schedule-ins/s; TCP sends57.3932/s vs prior78.4890/s and production38.5447/s. All clients retained and cleanup empty. Correctness gates pass, including7 native correlation/scheduling tests and39 socket tests.

Timing refinement proposed before code: reserve26 ms (rather than25) after a correlated answer for the fallback, and24 ms (rather than25) plus measured RTT for the response. Sum remains exactly RTT+50 ms; ready replies can still pull the probe forward after25 ms since the last send. The1 ms headroom accommodates timer quantization observed in the~25.93 ms client cadence, without delaying replies or extending a pending deadline. Skip a writer/authority operation entirely on answer-only scheduler wakes when no reply/probe/expiry needs work. Native per-PTY/daemon budgets remain unchanged. Keep all failure-phase, RTT-step, correlation and restart tests. No timed batching wait or dependency additions.

The first serial-control prototype is valid but not accepted: one-client CPU1.996258%,409.2478 schedule-ins/s; TCP sends57.3932/s vs prior78.4890/s and production38.5447/s. All clients retained and cleanup empty. Correctness gates pass, including7 native correlation/scheduling tests and39 socket tests. Timing refinement proposed before code: reserve26 ms (rather than25) after a correlated answer for the fallback, and24 ms (rather than25) plus measured RTT for the response. Sum remains exactly RTT+50 ms; ready replies can still pull the probe forward after25 ms since the last send. The1 ms headroom accommodates timer quantization observed in the~25.93 ms client cadence, without delaying replies or extending a pending deadline. Skip a writer/authority operation entirely on answer-only scheduler wakes when no reply/probe/expiry needs work. Native per-PTY/daemon budgets remain unchanged. Keep all failure-phase, RTT-step, correlation and restart tests. No timed batching wait or dependency additions.
kayg commented 2026-10-14 23:37:48 +00:00
Author
Owner
Copy link

26/24 refinement hit the unchanged #199 held-open ordering test: it explicitly advances25 ms after an ordinary operation completes and expects another native probe. FrozenSocketTime reported its90 s hang bound (38 other socket tests passed); this is a cadence regression, not a host-load waiver. Preserve the existing25 ms cadence for250 ms after application writes and monitoring restart, matching the client startup/recent-doubt window. Content-free Ping/Pong traffic cannot extend this window. Idle uses26 ms fallback; response margin remains24 ms in both cases, so detection is no later than RTT+50 ms (RTT+49 during the fast window). No new expiry timer for the window itself; inspect it on existing events. Keep the review test unchanged. Proposed before implementation; rerun the complete socket suite and CPU matrix.

26/24 refinement hit the unchanged #199 held-open ordering test: it explicitly advances25 ms after an ordinary operation completes and expects another native probe. FrozenSocketTime reported its90 s hang bound (38 other socket tests passed); this is a cadence regression, not a host-load waiver. Preserve the existing25 ms cadence for250 ms after application writes and monitoring restart, matching the client startup/recent-doubt window. Content-free Ping/Pong traffic cannot extend this window. Idle uses26 ms fallback; response margin remains24 ms in both cases, so detection is no later than RTT+50 ms (RTT+49 during the fast window). No new expiry timer for the window itself; inspect it on existing events. Keep the review test unchanged. Proposed before implementation; rerun the complete socket suite and CPU matrix.
kayg commented 2026-10-14 23:50:01 +00:00
Author
Owner
Copy link

26/24 idle scheduler refinement: fmt and server remote/all-target clippy pass. Liveness unit tests7 pass, WebSocket tests11 pass/3 ignored, complete socket suite39 pass/2 ignored, native correlation suite8 pass. Held-terminal-open review regression is fixed without editing its test; only application writes/restart retain the25ms recent-activity cadence. Static release build in progress; CPU acceptance is still pending. Raw gate receipt: serial26-recent-gates.log (will attach with measurement).

26/24 idle scheduler refinement: fmt and server remote/all-target clippy pass. Liveness unit tests7 pass, WebSocket tests11 pass/3 ignored, complete socket suite39 pass/2 ignored, native correlation suite8 pass. Held-terminal-open review regression is fixed without editing its test; only application writes/restart retain the25ms recent-activity cadence. Static release build in progress; CPU acceptance is still pending. Raw gate receipt: serial26-recent-gates.log (will attach with measurement).
kayg commented 2026-10-14 23:56:05 +00:00
Author
Owner
Copy link

Current serial26 static artifact e7a0ebb9ad26efeb353504d6c2a49f5479323ab53717234e63384e430b41c586,22341424 bytes (+64032 versus dev1b23ef31). Focused gates green. One-client60s settle/60s sample queued on perf-bench.lock; lock queue currently includes a >30-minute python process whose cwd is /root/infraternal-bench/ppu-soak/r4long (not this job). No process touched. Waiting with flock-w3600 as required; CPU acceptance remains pending.

Current serial26 static artifact e7a0ebb9ad26efeb353504d6c2a49f5479323ab53717234e63384e430b41c586,22341424 bytes (+64032 versus dev1b23ef31). Focused gates green. One-client60s settle/60s sample queued on perf-bench.lock; lock queue currently includes a >30-minute python process whose cwd is /root/infraternal-bench/ppu-soak/r4long (not this job). No process touched. Waiting with flock-w3600 as required; CPU acceptance remains pending.
kayg commented 2026-10-14 23:56:38 +00:00
Author
Owner
Copy link

Checkpoint commits5162818e (bounded answer-paced liveness module) and0ca60d20 (coalesced transport scheduling). Focused gate receipt below; complete socket suite39 pass/2 ignored, correlation8 pass. Static artifact e7a0ebb9 matches this source (no source edits after build). CPU acceptance is still queued behind the shared VM lock; this checkpoint is not a deploy-ready result.

serial26-recent-gates.log

Checkpoint commits5162818e (bounded answer-paced liveness module) and0ca60d20 (coalesced transport scheduling). Focused gate receipt below; complete socket suite39 pass/2 ignored, correlation8 pass. Static artifact e7a0ebb9 matches this source (no source edits after build). CPU acceptance is still queued behind the shared VM lock; this checkpoint is not a deploy-ready result. [serial26-recent-gates.log](https://git.kayg.org/attachments/9ac6107b-3a16-48b1-ab8a-3c9d1ae791c9)
kayg commented 2026-10-14 23:59:38 +00:00
Author
Owner
Copy link

d8c2072a documents the same bounded server-control schedule in contracts/wire.md§9. git diff --check passes. Self-review: exact issued nonce correlation survives monitoring restart; pending expiry cannot move on sends/duplicates; ready reply grouping preserves WS boundaries and whole-group timeout; bounded queues and committed revocation/absolute expiry remain authoritative; no dependencies/features introduced. Fresh origin/dev remains1b23ef31 and is an ancestor. Final iterate timing calibration, one-client CPU row, and12 final-native browser cases are waiting on the shared VM lock. No deploy-ready claim.

d8c2072a documents the same bounded server-control schedule in contracts/wire.md§9. git diff --check passes. Self-review: exact issued nonce correlation survives monitoring restart; pending expiry cannot move on sends/duplicates; ready reply grouping preserves WS boundaries and whole-group timeout; bounded queues and committed revocation/absolute expiry remain authoritative; no dependencies/features introduced. Fresh origin/dev remains1b23ef31 and is an ancestor. Final iterate timing calibration, one-client CPU row, and12 final-native browser cases are waiting on the shared VM lock. No deploy-ready claim.
kayg commented 2026-10-15 00:08:45 +00:00
Author
Owner
Copy link

External validation blocker: perf-bench.lock PID1981650 still owns the shared VM lock; its python child1986198 has cwd /root/infraternal-bench/ppu-soak/r4long and has run43 minutes. More than a dozen measurements are queued behind it, including this job's one-client CPU row, final-native browser recovery and verify calibration. Orchestrator: coordinate/release a short measurement window for199; do not treat d8c2072a as idle-accepted until its0/1/6 rows exist. This job is waiting with flock-w3600 and has not touched the other job's processes.

External validation blocker: perf-bench.lock PID1981650 still owns the shared VM lock; its python child1986198 has cwd /root/infraternal-bench/ppu-soak/r4long and has run43 minutes. More than a dozen measurements are queued behind it, including this job's one-client CPU row, final-native browser recovery and verify calibration. Orchestrator: coordinate/release a short measurement window for199; do not treat d8c2072a as idle-accepted until its0/1/6 rows exist. This job is waiting with flock-w3600 and has not touched the other job's processes.
kayg commented 2026-10-15 00:43:16 +00:00
Author
Owner
Copy link

Shared lock turned over after the long ppu-soak run. This job's final-native real-browser idle case is now running (128 real PTYs,150ms configured RTT,60s settle+60s sample). CPU0/1/6 rows, restart fixtures and gate calibration remain queued, each in its own short lock acquisition. No intervention on the other job was needed.

Shared lock turned over after the long ppu-soak run. This job's final-native real-browser idle case is now running (128 real PTYs,150ms configured RTT,60s settle+60s sample). CPU0/1/6 rows, restart fixtures and gate calibration remain queued, each in its own short lock acquisition. No intervention on the other job was needed.
kayg commented 2026-10-15 00:45:25 +00:00
Author
Owner
Copy link

d8c2072a / static SHA e7a0ebb9: final6-client128-PTY row is below production:3.907449% one core,1185.351494 activations/s (latest production9ab4329c4.495748%/1557.178658; also below older4.158161%/1382.9014 reference).60s settle+60.002455s sample,100% inherited counter coverage, all SQL worker deltas0, load0.761/0.370/0.352→0.514/0.363/0.351, cleanup empty.

Final-native real Chromium,128 real PTYs/one visible window/150ms RTT:60s settle+60s sample;7.9925 timer callbacks/s,6.4939 control probes/s,2.17665% whole-browser CPU,176.158 browser context switches/s. One retained socket, typing still works,0 page errors. Original browser reference65.846 callbacks/s/11.8528% CPU/902.128 switches/s; previous adaptive client with fixed native ticks8.108/7.380%/476.567.

Native+browser changes justify the64032-byte stripped x86 delta versus dev1b23ef31 (22341424−22277392), under65536; no dependencies/features added. One-client andzero-client rows and final gates still pending; not yet deploy-ready.

serial26-6-final.log
serial26-browser-idle.log

d8c2072a / static SHA e7a0ebb9: final6-client128-PTY row is below production:3.907449% one core,1185.351494 activations/s (latest production9ab4329c4.495748%/1557.178658; also below older4.158161%/1382.9014 reference).60s settle+60.002455s sample,100% inherited counter coverage, all SQL worker deltas0, load0.761/0.370/0.352→0.514/0.363/0.351, cleanup empty. Final-native real Chromium,128 real PTYs/one visible window/150ms RTT:60s settle+60s sample;7.9925 timer callbacks/s,6.4939 control probes/s,2.17665% whole-browser CPU,176.158 browser context switches/s. One retained socket, typing still works,0 page errors. Original browser reference65.846 callbacks/s/11.8528% CPU/902.128 switches/s; previous adaptive client with fixed native ticks8.108/7.380%/476.567. Native+browser changes justify the64032-byte stripped x86 delta versus dev1b23ef31 (22341424−22277392), under65536; no dependencies/features added. One-client andzero-client rows and final gates still pending; not yet deploy-ready. [serial26-6-final.log](https://git.kayg.org/attachments/28b48154-69b1-4e8e-b98f-7b7305ef5e1d) [serial26-browser-idle.log](https://git.kayg.org/attachments/ae1ae62c-bd4f-490b-929f-d8eff4dd15d7)
kayg commented 2026-10-15 00:52:18 +00:00
Author
Owner
Copy link

d8c2072a matrix:0clients0CPU/0activations;1client1.810443%/370.400760 activations/s (production1.431775%/386.784376);6clients3.907449%/1185.351494 (production4.495748%/1557.178658). One-client CPU is still RED. TCP writes45.1447/s versus production38.5447/s. All60s settle+60s samples; fixtures cleaned up. Profiling that row before further optimization. Full iterate interrupted in clippy (SIGTERM to this job's unique scope) because CPU remains red; no final gate claim.

Browser restart two cases pass; liveness review3/4pass, stalled-daemon case hit Chromium goto ERR_NETWORK_CHANGED before its action. Unchanged spec rerun queued; six three-session cases separately queued.

serial26-one-tcp.log
serial26-0-final.log
serial26-browser-restart-liveness-review.spec.ts.log

d8c2072a matrix:0clients0CPU/0activations;1client1.810443%/370.400760 activations/s (production1.431775%/386.784376);6clients3.907449%/1185.351494 (production4.495748%/1557.178658). One-client CPU is still RED. TCP writes45.1447/s versus production38.5447/s. All60s settle+60s samples; fixtures cleaned up. Profiling that row before further optimization. Full iterate interrupted in clippy (SIGTERM to this job's unique scope) because CPU remains red; no final gate claim. Browser restart two cases pass; liveness review3/4pass, stalled-daemon case hit Chromium goto ERR_NETWORK_CHANGED before its action. Unchanged spec rerun queued; six three-session cases separately queued. [serial26-one-tcp.log](https://git.kayg.org/attachments/d22c8063-90a5-403c-b701-6d275d851045) [serial26-0-final.log](https://git.kayg.org/attachments/fbba2dee-0d0f-4fc4-a8cf-6cb70415b77a) [serial26-browser-restart-liveness-review.spec.ts.log](https://git.kayg.org/attachments/c863671f-9b54-4ccf-9d67-94cdbf807cb4)
kayg commented 2026-10-15 00:59:37 +00:00
Author
Owner
Copy link

Unchanged serial26 one-client repeat is3.930862% CPU/445.295525 activations/s versus first1.810443%/370.400760. This within-binary variation means the earlier single co-poll comparison alone did not establish causality. Low-frequency follow-up capture has only15 samples (too few for quantitative attribution), including four finish_task_switch samples. Exact native link map rebuilt byte-identically to e7a0ebb9;997Hz/20s capture queued separately after its primary60s row.

Next isolated trial: poll read and proof futures under one task with tokio::select; when either ends, cancel its peer immediately. Keep the26/24 schedule, nonce correlation, queues, authority and all deadlines unchanged; no explicit additional cooperative budget consumption. This tests worker handoff cost after ready-reply grouping (the earlier trial preceded serial scheduling). Existing failing one-client CPU row is the red benchmark. Run complete socket/correlation gates and compare exact artifacts on VM before retaining. No cadence/test threshold relaxation.

Unchanged serial26 one-client repeat is3.930862% CPU/445.295525 activations/s versus first1.810443%/370.400760. This within-binary variation means the earlier single co-poll comparison alone did not establish causality. Low-frequency follow-up capture has only15 samples (too few for quantitative attribution), including four finish_task_switch samples. Exact native link map rebuilt byte-identically to e7a0ebb9;997Hz/20s capture queued separately after its primary60s row. Next isolated trial: poll read and proof futures under one task with tokio::select; when either ends, cancel its peer immediately. Keep the26/24 schedule, nonce correlation, queues, authority and all deadlines unchanged; no explicit additional cooperative budget consumption. This tests worker handoff cost after ready-reply grouping (the earlier trial preceded serial scheduling). Existing failing one-client CPU row is the red benchmark. Run complete socket/correlation gates and compare exact artifacts on VM before retaining. No cadence/test threshold relaxation.
kayg commented 2026-10-15 01:04:08 +00:00
Author
Owner
Copy link

Serial26 final-native browser coverage: restart2/2, liveness review4/4 on unchanged rerun, three-session6/6—all12pass. Original liveness run3/4passed; stalled-daemon case failed at goto with ERR_NETWORK_CHANGED before its action (original failure receipt attached previously).

CPU one-client repeat3.930862%/445.295525activations/s remains RED. Low-rate follow-up profile only15samples,0lost; insufficient for quantitative attribution. Higher997Hz/20s retry was invalid before profiling: browser pending proof crossed the existing deadline (RTT1ms,last-answer age64.83ms,last-send age40.56ms), then a diagnostic thread-name read raced a departing thread. No idle acceptance row from that run. Retry adds child-PID timeline and race-tolerant diagnostic comm names; kernel counter and retained-socket assertions unchanged. Single-task select trial queued behind build-slot.

serial26-browser-restart.spec.ts.log
serial26-browser-review-rerun.log
serial26-browser-three-session.log
serial26-one-profile.log
serial26-one-high-profile.log

Serial26 final-native browser coverage: restart2/2, liveness review4/4 on unchanged rerun, three-session6/6—all12pass. Original liveness run3/4passed; stalled-daemon case failed at goto with ERR_NETWORK_CHANGED before its action (original failure receipt attached previously). CPU one-client repeat3.930862%/445.295525activations/s remains RED. Low-rate follow-up profile only15samples,0lost; insufficient for quantitative attribution. Higher997Hz/20s retry was invalid before profiling: browser pending proof crossed the existing deadline (RTT1ms,last-answer age64.83ms,last-send age40.56ms), then a diagnostic thread-name read raced a departing thread. No idle acceptance row from that run. Retry adds child-PID timeline and race-tolerant diagnostic comm names; kernel counter and retained-socket assertions unchanged. Single-task select trial queued behind build-slot. [serial26-browser-restart.spec.ts.log](https://git.kayg.org/attachments/ea0fb2ec-66d6-43ac-b65e-e93e95d7a691) [serial26-browser-review-rerun.log](https://git.kayg.org/attachments/989ce060-62fe-4275-baee-c7c124ccc6e2) [serial26-browser-three-session.log](https://git.kayg.org/attachments/68b065cf-6c4f-4c81-b15e-1752ebad1ac8) [serial26-one-profile.log](https://git.kayg.org/attachments/13f73fca-5f6d-470d-a228-10abfe28c543) [serial26-one-high-profile.log](https://git.kayg.org/attachments/730f3b7a-5df6-4613-a06b-8cd548171e6e)
kayg commented 2026-10-15 01:20:55 +00:00
Author
Owner
Copy link

New deterministic regression test is RED before fix: buffered_control_frames_cannot_monopolize_a_transport_turn feeds4096 already-buffered valid control frames, biases selection toward the receiver, and requires a separate proof future to get polled. Without a cooperative bound, receive consumes the entire burst first. This is a real fairness hazard for the single-task layout even though existing39 socket tests pass. Add consume_budget inside receive_wire's loop, including transport-control/incomplete-frame iterations, so buffered data cannot bypass I/O cooperation. No timer, cadence, readiness or deadline change. Re-run clippy, WebSocket tests, complete socket/correlation suites and rebuild as separate paired-budget artifact.

buffered-controls-red.log

New deterministic regression test is RED before fix: buffered_control_frames_cannot_monopolize_a_transport_turn feeds4096 already-buffered valid control frames, biases selection toward the receiver, and requires a separate proof future to get polled. Without a cooperative bound, receive consumes the entire burst first. This is a real fairness hazard for the single-task layout even though existing39 socket tests pass. Add consume_budget inside receive_wire's loop, including transport-control/incomplete-frame iterations, so buffered data cannot bypass I/O cooperation. No timer, cadence, readiness or deadline change. Re-run clippy, WebSocket tests, complete socket/correlation suites and rebuild as separate paired-budget artifact. [buffered-controls-red.log](https://git.kayg.org/attachments/aa01975c-fee2-43f8-86f0-317f50422910)
kayg commented 2026-10-15 01:22:28 +00:00
Author
Owner
Copy link

Fairness regression is GREEN with consume_budget in receive_wire (fmt/clippy pass;12 WebSocket tests pass/3 ignored). Complete socket/correlation gates and new paired-budget static build are in progress.

VM contention again: holder2062385 owns perf-bench.lock, cwd /root/infraternal-bench/ppu-soak/r4seg; its python child2065762 has held the run20m42s. This job's high-frequency retry (queued03:02 CEST) and short production/trial one-client comparison (queued03:16) have not started. Orchestrator: coordinate a short199 measurement window; previous table is still red at one client. No other job's process touched.

Fairness regression is GREEN with consume_budget in receive_wire (fmt/clippy pass;12 WebSocket tests pass/3 ignored). Complete socket/correlation gates and new paired-budget static build are in progress. VM contention again: holder2062385 owns perf-bench.lock, cwd /root/infraternal-bench/ppu-soak/r4seg; its python child2065762 has held the run20m42s. This job's high-frequency retry (queued03:02 CEST) and short production/trial one-client comparison (queued03:16) have not started. Orchestrator: coordinate a short199 measurement window; previous table is still red at one client. No other job's process touched.
kayg commented 2026-10-15 01:25:47 +00:00
Author
Owner
Copy link

Completed14b96ef4: buffered control decoding consumes the existing cooperative budget before another decode/transport iteration. Regression0pass/1fail before, green after; fmt/clippy green,12 WebSocket tests pass/3 ignored,39 socket tests pass/2 ignored,8 native correlation tests pass. No timer/deadline/cadence change. Single-task worker experiment remains uncommitted and performance-unaccepted. Corrected paired-budget static build is running; receipt below records completed focused gates, not a final full gate.

paired-budget-gates.log

Completed14b96ef4: buffered control decoding consumes the existing cooperative budget before another decode/transport iteration. Regression0pass/1fail before, green after; fmt/clippy green,12 WebSocket tests pass/3 ignored,39 socket tests pass/2 ignored,8 native correlation tests pass. No timer/deadline/cadence change. Single-task worker experiment remains uncommitted and performance-unaccepted. Corrected paired-budget static build is running; receipt below records completed focused gates, not a final full gate. [paired-budget-gates.log](https://git.kayg.org/attachments/88329ccc-cb37-425d-b2c5-c033cb42369c)
kayg commented 2026-10-15 01:38:05 +00:00
Author
Owner
Copy link

Valid serial26 one-client retry:1.695593% CPU/362.612281 activations/s,60s settle+60.003483s sample,100% counter coverage; child2094113 stable across1s PID observations, all SQL worker CPU/schedule deltas0,128 shim identities retained and cleanup empty. This rules out a backend-child restart during this primary window. Still above freshest production1.431775% CPU.

Separate post-primary997Hz/20s profile:339samples,0lost; exact shipping link map and analysis-only labeled ELF (.text unchanged, never executed). Two largest kernel leaves (_raw_spin_unlock_irqrestore and finish_task_switch)74/339=21.83%; syscall-inclusive57.82%; native actor4.13% self/7.37% inclusive, memset1.18% self. Inclusive categories overlap. Confirms syscall/scheduling cost remains the next target; not a quantitative claim from the earlier15-sample capture.

Corrected single-task+budget artifact SHA636bbe1d1d8e878a987bd6b8e4206eba153bbbe4093c5a50f355cec568f77eae,22341392 bytes (+64000 over dev1b23ef31). Focused gates pass (39socket/8correlation/12WebSocket); complete server unit suite and back-to-back production comparison plus0/6rows queued. Worker change still uncommitted pending CPU acceptance.

serial26-one-high-retry.log

Valid serial26 one-client retry:1.695593% CPU/362.612281 activations/s,60s settle+60.003483s sample,100% counter coverage; child2094113 stable across1s PID observations, all SQL worker CPU/schedule deltas0,128 shim identities retained and cleanup empty. This rules out a backend-child restart during this primary window. Still above freshest production1.431775% CPU. Separate post-primary997Hz/20s profile:339samples,0lost; exact shipping link map and analysis-only labeled ELF (.text unchanged, never executed). Two largest kernel leaves (_raw_spin_unlock_irqrestore and finish_task_switch)74/339=21.83%; syscall-inclusive57.82%; native actor4.13% self/7.37% inclusive, memset1.18% self. Inclusive categories overlap. Confirms syscall/scheduling cost remains the next target; not a quantitative claim from the earlier15-sample capture. Corrected single-task+budget artifact SHA636bbe1d1d8e878a987bd6b8e4206eba153bbbe4093c5a50f355cec568f77eae,22341392 bytes (+64000 over dev1b23ef31). Focused gates pass (39socket/8correlation/12WebSocket); complete server unit suite and back-to-back production comparison plus0/6rows queued. Worker change still uncommitted pending CPU acceptance. [serial26-one-high-retry.log](https://git.kayg.org/attachments/025df4d8-10e8-4e33-9f38-4d5c72141ce2)
kayg commented 2026-10-15 01:40:58 +00:00
Author
Owner
Copy link

Attachment correction: Forgejo rejected raw .perf with HTTP422, so the prior comment initially linked only the primary log. Text reports and the gzip-compressed raw trace are now attached below. The two largest kernel leaves are generic spin-unlock and task-switch leaves, not an exclusive wakeup category; inclusive categories overlap. No profile data was altered.

serial26-high-flat.txt
serial26-high-inclusive.txt
paired-budget-gates.log
serial26-high.perf.gz

Attachment correction: Forgejo rejected raw .perf with HTTP422, so the prior comment initially linked only the primary log. Text reports and the gzip-compressed raw trace are now attached below. The two largest kernel leaves are generic spin-unlock and task-switch leaves, not an exclusive wakeup category; inclusive categories overlap. No profile data was altered. [serial26-high-flat.txt](https://git.kayg.org/attachments/2a87f257-eae7-4297-a629-7b5a821ad3aa) [serial26-high-inclusive.txt](https://git.kayg.org/attachments/89b89674-76b3-4741-a1f8-a615828e7165) [paired-budget-gates.log](https://git.kayg.org/attachments/75d407ce-fdde-48fd-b8ee-b3b37de68de0) [serial26-high.perf.gz](https://git.kayg.org/attachments/ce3035d7-0bb9-4ef9-b5d0-97debe5583f4)
kayg commented 2026-10-15 01:46:33 +00:00
Author
Owner
Copy link

Complete server unit suite passed for 14b96ef4 plus the uncommitted single-task trial (artifact 636bbe1d). This is the crate gate, not the final workspace gate.

test result: ok. 443 passed; 0 failed; 47 ignored; 0 measured; 0 filtered out; finished in 381.07s

Production comparison and corrected 0/1/6-client rows are still queued on the shared perf VM. No new orchestration instruction appeared in the refreshed issue history.

paired-budget-server-unit.log

Complete server unit suite passed for 14b96ef4 plus the uncommitted single-task trial (artifact 636bbe1d). This is the crate gate, not the final workspace gate. ```text test result: ok. 443 passed; 0 failed; 47 ignored; 0 measured; 0 filtered out; finished in 381.07s ``` Production comparison and corrected 0/1/6-client rows are still queued on the shared perf VM. No new orchestration instruction appeared in the refreshed issue history. [paired-budget-server-unit.log](https://git.kayg.org/attachments/4452ca1c-7925-4fbb-b1d8-f86963d73123)
kayg commented 2026-10-15 02:14:31 +00:00
Author
Owner
Copy link

Source inspection while queued: native read_incoming already runs inside the same actor select as timers (embedded.rs:1004), so there is no separate native reader task to combine. The host tick separately writes its control Ping before embedded_route_proofs::tick builds and writes local route headers (embedded.rs:945-948; embedded_route_proofs.rs:54-95). That permits two separately awaited local write_all operations on a healthy tick with route proofs. No native change made; this is an unmeasured follow-up option if the browser worker trial still misses CPU.

VM queue moved again: the older unbudgeted paired trial's fresh production control is now settling; corrected artifact 636bbe1d and its 0/6 rows remain queued. The unbudgeted artifact is diagnostic only; the new fairness regression makes it unsuitable to ship.

Source inspection while queued: native read_incoming already runs inside the same actor select as timers (embedded.rs:1004), so there is no separate native reader task to combine. The host tick separately writes its control Ping before embedded_route_proofs::tick builds and writes local route headers (embedded.rs:945-948; embedded_route_proofs.rs:54-95). That permits two separately awaited local write_all operations on a healthy tick with route proofs. No native change made; this is an unmeasured follow-up option if the browser worker trial still misses CPU. VM queue moved again: the older unbudgeted paired trial's fresh production control is now settling; corrected artifact 636bbe1d and its 0/6 rows remain queued. The unbudgeted artifact is diagnostic only; the new fairness regression makes it unsuitable to ship.
kayg commented 2026-10-15 02:44:23 +00:00
Author
Owner
Copy link

Older diagnostic trial 9c8041b0: back-to-back production control 1.646455% CPU / 386.513853 activations/s; trial 1.951698% / 378.134519. CPU still misses. This artifact lacks the new fairness bound and is not suitable to ship.

Corrected 636bbe1d: zero clients passed at exactly 0 CPU / 0 schedule-ins, 60 s settle + 60.036027 s sample; cleanup empty. Its fresh production control became invalid when the legacy client timer fired late with no pending request (58 ms after the last answer); that stopped the paired script before the candidate. Six-client row also invalid: retained sockets closed and pending proofs exceeded their unchanged deadlines. No CPU row accepted from either failed run.

Retry uses the same current adaptive client for both static server versions. Healthy low-RTT cadence remains 25 ms; this applies the explicitly requested M1 behavior to the baseline client too, rather than comparing against its known delayed-timer bug. Pending proof assertions and server deadlines remain unchanged. Added GC observation only; GC is not disabled.

Correction: earlier measurement-exception attribution to a departing thread was too specific—the exception type was not logged. New diagnostics log only its safe type. Client retirement already made those rows invalid regardless of the secondary exception.

Native batching preparation: add optional host control to the existing tick API, preserving existing data-only fixture assertions with None; new Some-mode tests must fail before combining local control and route headers. No production behavior changed by the test scaffold yet.

paired-one-comparison.log
paired-budget-comparison.log
paired-budget-0.log
paired-budget-6.log

Older diagnostic trial 9c8041b0: back-to-back production control 1.646455% CPU / 386.513853 activations/s; trial 1.951698% / 378.134519. CPU still misses. This artifact lacks the new fairness bound and is not suitable to ship. Corrected 636bbe1d: zero clients passed at exactly 0 CPU / 0 schedule-ins, 60 s settle + 60.036027 s sample; cleanup empty. Its fresh production control became invalid when the legacy client timer fired late with no pending request (58 ms after the last answer); that stopped the paired script before the candidate. Six-client row also invalid: retained sockets closed and pending proofs exceeded their unchanged deadlines. No CPU row accepted from either failed run. Retry uses the same current adaptive client for both static server versions. Healthy low-RTT cadence remains 25 ms; this applies the explicitly requested M1 behavior to the baseline client too, rather than comparing against its known delayed-timer bug. Pending proof assertions and server deadlines remain unchanged. Added GC observation only; GC is not disabled. Correction: earlier measurement-exception attribution to a departing thread was too specific—the exception type was not logged. New diagnostics log only its safe type. Client retirement already made those rows invalid regardless of the secondary exception. Native batching preparation: add optional host control to the existing tick API, preserving existing data-only fixture assertions with None; new Some-mode tests must fail before combining local control and route headers. No production behavior changed by the test scaffold yet. [paired-one-comparison.log](https://git.kayg.org/attachments/c6612fd9-a954-4f65-92cd-74b30c186cd7) [paired-budget-comparison.log](https://git.kayg.org/attachments/be69938b-d825-44fb-85ff-6c742ab4ff1f) [paired-budget-0.log](https://git.kayg.org/attachments/2088136b-4600-4598-ac35-626922ba10f4) [paired-budget-6.log](https://git.kayg.org/attachments/80618b16-fd65-415f-bc09-270a7e1e79ca)
kayg commented 2026-10-15 03:00:16 +00:00
Author
Owner
Copy link

Native batching regressions: 0 passed / 2 failed before the change; both pass after. Fmt and server remote/all-target clippy pass. The host tick now passes its control lease to the existing route batch: local control is first in the same write as ready terminal headers, stamped after expiry work; remote peers retain their owned control writer. Existing data-only fixture assertions retain None mode unchanged. No wire layout, interval or deadline change.

Static native-batch build is in progress, followed by the complete server suite and native correlation tests. Worker and native batching changes remain uncommitted and performance-unaccepted. Current client GC diagnostics and the corrected same-client production comparison are still queued.

native-control-batch-red.log
native-batch-gates.log

Native batching regressions: 0 passed / 2 failed before the change; both pass after. Fmt and server remote/all-target clippy pass. The host tick now passes its control lease to the existing route batch: local control is first in the same write as ready terminal headers, stamped after expiry work; remote peers retain their owned control writer. Existing data-only fixture assertions retain None mode unchanged. No wire layout, interval or deadline change. Static native-batch build is in progress, followed by the complete server suite and native correlation tests. Worker and native batching changes remain uncommitted and performance-unaccepted. Current client GC diagnostics and the corrected same-client production comparison are still queued. [native-control-batch-red.log](https://git.kayg.org/attachments/039c4dc3-b973-46b1-bf7e-d78097cffda4) [native-batch-gates.log](https://git.kayg.org/attachments/c934b14d-7c50-4b0d-a9cb-b63adfde6113)
kayg commented 2026-10-15 03:41:09 +00:00
Author
Owner
Copy link

Checkpoints 81e3849d and 0fed4495: local host control and terminal route proofs share one IPC write; queued browser replies no longer add an unconditional yield. Independent transport workers retained: co-polling failed CPU acceptance (same adaptive client, production 1.5496% / 385.81 activations/s; co-poll 1.7408% / 377.73; six-client co-poll 5.6865% / 1302.83). GC observation remained enabled: maximum pauses below 0.6ms. These trials are rejected, not acceptance evidence.

Both changes have targeted red/green regression coverage from the preceding steps. Current fmt/clippy pass, 445 server unit tests pass (47 ignored), eight native probe-correlation tests pass. Static x86_64 artifact fcc1ccbb626710c4a354f095b5423abed257e5486da120555405777b032ba0c5: 22,328,464 bytes, +51,072 versus dev merge base. Final paired 0/1/6 CPU measurements queued; full iterate/fast started. origin/dev remains 1b23ef31 and is already integrated. No deploy.

Checkpoints 81e3849d and 0fed4495: local host control and terminal route proofs share one IPC write; queued browser replies no longer add an unconditional yield. Independent transport workers retained: co-polling failed CPU acceptance (same adaptive client, production 1.5496% / 385.81 activations/s; co-poll 1.7408% / 377.73; six-client co-poll 5.6865% / 1302.83). GC observation remained enabled: maximum pauses below 0.6ms. These trials are rejected, not acceptance evidence. Both changes have targeted red/green regression coverage from the preceding steps. Current fmt/clippy pass, 445 server unit tests pass (47 ignored), eight native probe-correlation tests pass. Static x86_64 artifact fcc1ccbb626710c4a354f095b5423abed257e5486da120555405777b032ba0c5: 22,328,464 bytes, +51,072 versus dev merge base. Final paired 0/1/6 CPU measurements queued; full iterate/fast started. origin/dev remains 1b23ef31 and is already integrated. No deploy.
kayg commented 2026-10-15 03:42:25 +00:00
Author
Owner
Copy link

Current receipts for 81e3849d / 0fed4495:

independent-replies-gates.log
corrected-client-comparison.log
paired-budget-six-retry.log

Perf access blocker at 05:41 CEST: the shared lock is held by PID 2169590, ppu-soak/r5lazy, a 7,200-second soak with 600-second warmup (elapsed 1:14:41). Our final comparisons and browser checks are waiting with flock -w 3600. No other job was interrupted. Local full gates are progressing. One obsolete own queued native-batch six-client trial was cancelled before it launched; the new candidate comparisons remain queued.

Current receipts for 81e3849d / 0fed4495: [independent-replies-gates.log](https://git.kayg.org/attachments/5b5f1315-5969-46f5-a919-a2a35fad1fce) [corrected-client-comparison.log](https://git.kayg.org/attachments/84451de6-dd07-4fe5-b74a-763c1be244c3) [paired-budget-six-retry.log](https://git.kayg.org/attachments/8be3ce15-17c1-4d5c-a6f7-885bcbed0c2c) Perf access blocker at 05:41 CEST: the shared lock is held by PID 2169590, ppu-soak/r5lazy, a 7,200-second soak with 600-second warmup (elapsed 1:14:41). Our final comparisons and browser checks are waiting with flock -w 3600. No other job was interrupted. Local full gates are progressing. One obsolete own queued native-batch six-client trial was cancelled before it launched; the new candidate comparisons remain queued.
kayg commented 2026-10-15 03:44:13 +00:00
Author
Owner
Copy link

Correction to gate status: the final 0fed4495 full-gate driver is waiting for the build-slot wrapper, not yet compiling. The prior iterate log was from the earlier interrupted run and has now been renamed prior-interrupted-final-verify-iterate.log to prevent confusion. Current-head targeted gates remain 445 + 8 passed; final iterate/fast remain pending. Both host build slots and the perf VM are occupied by other jobs.

Correction to gate status: the final 0fed4495 full-gate driver is waiting for the build-slot wrapper, not yet compiling. The prior iterate log was from the earlier interrupted run and has now been renamed prior-interrupted-final-verify-iterate.log to prevent confusion. Current-head targeted gates remain 445 + 8 passed; final iterate/fast remain pending. Both host build slots and the perf VM are occupied by other jobs.
kayg commented 2026-10-15 04:18:02 +00:00
Author
Owner
Copy link

FCC one-client comparison failed CPU acceptance: production9ab4329c 1.995869% / 375.247 activations/s; fcc1ccbb 3.109664% / 365.081. Both 128 real PTYs, 25 attached, 60s settle +60s sample. Production load3.660/3.057/1.620→1.404/2.515/1.521; candidate0.545/2.068/1.428→0.273/1.706/1.341.

The native IPC batching experiment also increased CPU in the preceding worker arrangement (paired-budget1.7408% before batching, native-batch3.5480% after). Reverting81e3849d; retaining independent workers, exact nonce checks, adaptive browser timing and bounded decoder fairness. The predeclared reverse-order FCC repeat remains queued and will be reported, not hidden. The FCC full-gate driver was stopped while waiting only on VM calibration; no Rust gate result is claimed. Obsolete queued FCC browser/zero checks will be cancelled before launch.

FCC one-client comparison failed CPU acceptance: production9ab4329c 1.995869% / 375.247 activations/s; fcc1ccbb 3.109664% / 365.081. Both 128 real PTYs, 25 attached, 60s settle +60s sample. Production load3.660/3.057/1.620→1.404/2.515/1.521; candidate0.545/2.068/1.428→0.273/1.706/1.341. The native IPC batching experiment also increased CPU in the preceding worker arrangement (paired-budget1.7408% before batching, native-batch3.5480% after). Reverting81e3849d; retaining independent workers, exact nonce checks, adaptive browser timing and bounded decoder fairness. The predeclared reverse-order FCC repeat remains queued and will be reported, not hidden. The FCC full-gate driver was stopped while waiting only on VM calibration; no Rust gate result is claimed. Obsolete queued FCC browser/zero checks will be cancelled before launch.
kayg commented 2026-10-15 04:27:57 +00:00
Author
Owner
Copy link

Fixture teardown defect fixed: 278f55ae adds the failing BrokenPipeError close test; d5023069 catches connection-close errors and collects every close before private unit/shim teardown, reporting unexpected close failures afterward. All seven fixture tests pass. Retired clients are also rejected immediately after settling, before sampling; proof deadlines and workload are unchanged.

Both six-client FCC comparison rows are invalid (client proof retirement). The candidate run then raised BrokenPipeError during finally, leaving its own server PID2321698, unit infraternal-idle-199-60050cd167e940f99e8a7094c208587e.service. Stopped that verified private unit at approximately06:24 CEST; remaining owned PIDs=[], shim peers=[]; removed its p1 directory. Any other measurement overlapping that leaked process window should be rechecked. The FCC zero row failed FileExistsError from this leftover directory and is invalid. The one-client3.1097% result completed before this leak and remains valid.

The predeclared reverse-order FCC repeat was cancelled while still queued, then rescheduled unchanged with teardown-safe fixture code. Native IPC batching revert is staged: fmt/clippy and52 native tests (3 ignored) plus8 correlation tests pass; release build ongoing. No full-gate success or deploy readiness claimed.

independent-comparison-1.log
independent-comparison-6.log
broken-close-cleanup.log
broken-close-red.log
broken-close-green.log

Fixture teardown defect fixed: 278f55ae adds the failing BrokenPipeError close test; d5023069 catches connection-close errors and collects every close before private unit/shim teardown, reporting unexpected close failures afterward. All seven fixture tests pass. Retired clients are also rejected immediately after settling, before sampling; proof deadlines and workload are unchanged. Both six-client FCC comparison rows are invalid (client proof retirement). The candidate run then raised BrokenPipeError during finally, leaving its own server PID2321698, unit infraternal-idle-199-60050cd167e940f99e8a7094c208587e.service. Stopped that verified private unit at approximately06:24 CEST; remaining owned PIDs=[], shim peers=[]; removed its p1 directory. Any other measurement overlapping that leaked process window should be rechecked. The FCC zero row failed FileExistsError from this leftover directory and is invalid. The one-client3.1097% result completed before this leak and remains valid. The predeclared reverse-order FCC repeat was cancelled while still queued, then rescheduled unchanged with teardown-safe fixture code. Native IPC batching revert is staged: fmt/clippy and52 native tests (3 ignored) plus8 correlation tests pass; release build ongoing. No full-gate success or deploy readiness claimed. [independent-comparison-1.log](https://git.kayg.org/attachments/5d1bebe8-af66-4845-8cca-e18016ddd845) [independent-comparison-6.log](https://git.kayg.org/attachments/479466ef-5359-4d80-8a41-0b6d4f571c21) [broken-close-cleanup.log](https://git.kayg.org/attachments/d9216ac2-bdf0-4aad-bf6c-3c2eafec4aad) [broken-close-red.log](https://git.kayg.org/attachments/f2008a10-1f5a-4430-b92e-3276530686d0) [broken-close-green.log](https://git.kayg.org/attachments/321aa628-e078-4a4c-8212-13b80752e1fe)
kayg commented 2026-10-15 04:36:46 +00:00
Author
Owner
Copy link

Head f924ffd9ae reverts the rejected local host-control IPC batching experiment. Independent boxed transport workers and ready-reply scheduling remain. fmt/clippy pass;52 native unit tests pass (3 ignored) and8 native correlation tests pass. Fixture teardown fix d5023069 passes7 unit tests after the new BrokenPipe regression failed first.

Static x86_64 c538bd0204f65ac51ecbccb720359b5837805454427d9b585882b4741e6d90d1 is22,328,400 bytes, +51,008 versus dev merge base1b23ef31 (same locked musl release recipe). Fresh origin/dev fetch remains1b23ef31, already integrated. New teardown-safe 0/1/6 paired comparisons are queued since06:31 CEST; the predeclared reverse-order rejected FCC repeat is also rescheduled with safe teardown. Full iterate→fast driver started for this head; no full-gate success claimed. Perf lock at06:35 was held by ppu-soak/r5lazy.

separate-control-gates.log

Head f924ffd9ae7e1025b771c8a40728665d6d2bfcec reverts the rejected local host-control IPC batching experiment. Independent boxed transport workers and ready-reply scheduling remain. fmt/clippy pass;52 native unit tests pass (3 ignored) and8 native correlation tests pass. Fixture teardown fix d5023069 passes7 unit tests after the new BrokenPipe regression failed first. Static x86_64 c538bd0204f65ac51ecbccb720359b5837805454427d9b585882b4741e6d90d1 is22,328,400 bytes, +51,008 versus dev merge base1b23ef31 (same locked musl release recipe). Fresh origin/dev fetch remains1b23ef31, already integrated. New teardown-safe 0/1/6 paired comparisons are queued since06:31 CEST; the predeclared reverse-order rejected FCC repeat is also rescheduled with safe teardown. Full iterate→fast driver started for this head; no full-gate success claimed. Perf lock at06:35 was held by ppu-soak/r5lazy. [separate-control-gates.log](https://git.kayg.org/attachments/860fe65e-2744-4f33-9bdd-f69aa99fa1e6)
kayg commented 2026-10-15 04:56:39 +00:00
Author
Owner
Copy link

Head8c4eb46f strengthens the benchmark oracle only; native artifact remains c538bd02 from f924ffd9 (Rust/TypeScript unchanged). Added failing regressions for terminal Close (7c90ad51→41837b6b) and terminal.reconnecting Event (5244533c→8c4eb46f), plus a healthy metadata preservation case. All ten fixture unit tests pass. Prior socket/shim-only rows are diagnostic, not sufficient proof that every terminal route remained ready. Final acceptance requires the stricter fixture.

Teardown-safe VM fixture SHA256220e01c8f06fa9482e82b9d8c7f59d42a6e079ef40f8a38b624910e137432c1a; future runs print this source hash. Updated it atomically while every queued comparison log was still empty. CPU sampling/probe timing unchanged. Final iterate is still waiting at VM calibration; fast will run on the branch head afterward.

terminal-close-red.log
terminal-close-green.log
reconnecting-route-red.log
reconnecting-route-green.log

Head8c4eb46f strengthens the benchmark oracle only; native artifact remains c538bd02 from f924ffd9 (Rust/TypeScript unchanged). Added failing regressions for terminal Close (7c90ad51→41837b6b) and terminal.reconnecting Event (5244533c→8c4eb46f), plus a healthy metadata preservation case. All ten fixture unit tests pass. Prior socket/shim-only rows are diagnostic, not sufficient proof that every terminal route remained ready. Final acceptance requires the stricter fixture. Teardown-safe VM fixture SHA256220e01c8f06fa9482e82b9d8c7f59d42a6e079ef40f8a38b624910e137432c1a; future runs print this source hash. Updated it atomically while every queued comparison log was still empty. CPU sampling/probe timing unchanged. Final iterate is still waiting at VM calibration; fast will run on the branch head afterward. [terminal-close-red.log](https://git.kayg.org/attachments/fc1bbe0d-c0cf-4332-b5ed-db7dca076560) [terminal-close-green.log](https://git.kayg.org/attachments/b1094749-6d61-44f3-ae13-b7477a2f5b80) [reconnecting-route-red.log](https://git.kayg.org/attachments/a24e3176-bf67-40e0-98f3-e59354ca24da) [reconnecting-route-green.log](https://git.kayg.org/attachments/e8ef75b2-801f-450b-bab5-d035a175b3c6)
kayg commented 2026-10-15 05:14:42 +00:00
Author
Owner
Copy link

Blocked on shared perf access,07:13 CEST: latest0/1/6 paired comparisons have been queued since06:31 and have not started. Full iterate is waiting at VM calibration (queued approximately06:38), so fast has not started. Current lock holderPID2366930 is mesh-autoswitch-r14-standing-150 with a1,500s command timeout; process elapsed46:46 includes lock waiting and is not a lock-hold duration. The orchestrator needs to make a short exclusive measurement window available; existing flock -w3600 waiters remain queued and will be retried on timeout. No other process is interrupted.

Current head8c4eb46f; native artifactc538bd02 unchanged sincef924ffd9. Ten fixture tests pass,52 targeted native tests plus8 correlation tests pass. Browser recovery and idle callback measurements also queued. Removed3.8GiB of this job's obsolete debug incremental cache under disk pressure; release artifacts, baseline and evidence retained. No full-gate success, CPU acceptance, push or deploy claimed.

Lock receipt:

2026-10-15T05:13:54.704965+00:00
2366930 WRITE  /run/lock/perf-bench.lock
    PID     ELAPSED COMMAND
2366930       46:46 flock -E 75 -w 3600 /var/lock/perf-bench.lock timeout --signal=TERM --kill-after=30 1500 systemd-run --quiet --scope --unit mesh-autoswitch-r14-standing-150 -p CPUWeight=10000 -p RuntimeMaxSec=1520 /usr/bin/python3 /root/infraternal-bench/mesh-autoswitch/r14/diag-standing-runner.py --fixture /root/infraternal-bench/mesh-autoswitch/r13/release/server-fixture --payload /root/infraternal-bench/mesh-autoswitch/r13/release/infraternal --sshd-source /root/infraternal-bench/mesh-autoswitch/r6/acceptance/sshd.py --output /root/infraternal-bench/mesh-autoswitch/r14/actual-ssh --label diag-standing82bb-31-a --profile release --transport-expectation preserved --acceptance-mode standing-switch --samples 31 --venues 0 150 --venue-locked --single-venue 150
Blocked on shared perf access,07:13 CEST: latest0/1/6 paired comparisons have been queued since06:31 and have not started. Full iterate is waiting at VM calibration (queued approximately06:38), so fast has not started. Current lock holderPID2366930 is mesh-autoswitch-r14-standing-150 with a1,500s command timeout; process elapsed46:46 includes lock waiting and is not a lock-hold duration. The orchestrator needs to make a short exclusive measurement window available; existing flock -w3600 waiters remain queued and will be retried on timeout. No other process is interrupted. Current head8c4eb46f; native artifactc538bd02 unchanged sincef924ffd9. Ten fixture tests pass,52 targeted native tests plus8 correlation tests pass. Browser recovery and idle callback measurements also queued. Removed3.8GiB of this job's obsolete debug incremental cache under disk pressure; release artifacts, baseline and evidence retained. No full-gate success, CPU acceptance, push or deploy claimed. Lock receipt: ``` 2026-10-15T05:13:54.704965+00:00 2366930 WRITE /run/lock/perf-bench.lock PID ELAPSED COMMAND 2366930 46:46 flock -E 75 -w 3600 /var/lock/perf-bench.lock timeout --signal=TERM --kill-after=30 1500 systemd-run --quiet --scope --unit mesh-autoswitch-r14-standing-150 -p CPUWeight=10000 -p RuntimeMaxSec=1520 /usr/bin/python3 /root/infraternal-bench/mesh-autoswitch/r14/diag-standing-runner.py --fixture /root/infraternal-bench/mesh-autoswitch/r13/release/server-fixture --payload /root/infraternal-bench/mesh-autoswitch/r13/release/infraternal --sshd-source /root/infraternal-bench/mesh-autoswitch/r6/acceptance/sshd.py --output /root/infraternal-bench/mesh-autoswitch/r14/actual-ssh --label diag-standing82bb-31-a --profile release --transport-expectation preserved --acceptance-mode standing-switch --samples 31 --venues 0 150 --venue-locked --single-venue 150 ```
kayg commented 2026-10-15 05:21:08 +00:00
Author
Owner
Copy link

Predeclared reverse-order repeat completed with stricter fixture220e01c8:128 real PTYs,1 WebSocket/25 attached terminals;60s settle +60s sample each. Rejected FCC artifact2.250172% CPU /347.645 activations/s, production9ab1.742542% /376.917. FCC CPU is29.1% higher despite fewer activations. Both retained sockets/terminal routes pass; cleanup reports no remaining shim peers. No child restart in either sampled window.

FCC load0.570/1.484/1.916→0.454/1.299/1.825; production0.261/1.102/1.723→0.175/0.927/1.623. This confirms rejection, not acceptance of the IPC batching trial. Replacementc538bd02 comparisons0/1/6 remain queued; native sourcef924ffd9, fixture head8c4eb46f. Browser recovery files are now one12-test invocation under one short locked run, avoiding three separate lock waits. Full iterate still awaiting VM calibration.

independent-reverse-safe-1.log

Predeclared reverse-order repeat completed with stricter fixture220e01c8:128 real PTYs,1 WebSocket/25 attached terminals;60s settle +60s sample each. Rejected FCC artifact2.250172% CPU /347.645 activations/s, production9ab1.742542% /376.917. FCC CPU is29.1% higher despite fewer activations. Both retained sockets/terminal routes pass; cleanup reports no remaining shim peers. No child restart in either sampled window. FCC load0.570/1.484/1.916→0.454/1.299/1.825; production0.261/1.102/1.723→0.175/0.927/1.623. This confirms rejection, not acceptance of the IPC batching trial. Replacementc538bd02 comparisons0/1/6 remain queued; native sourcef924ffd9, fixture head8c4eb46f. Browser recovery files are now one12-test invocation under one short locked run, avoiding three separate lock waits. Full iterate still awaiting VM calibration. [independent-reverse-safe-1.log](https://git.kayg.org/attachments/be4a9f90-6977-4f6d-9081-417f72db8406)
kayg commented 2026-10-15 05:28:43 +00:00
Author
Owner
Copy link

Six-client acceptance row passes in the paired run:128 real PTYs and128 retained attached routes across6 sockets;60s settle +60s sample each; strict fixture220e01c8.

Native build CPU% one core Activations/s Activations/s/client Mean wire-client timer expirations/s/client
9ab4329c-separate-control 5.686600 1479.325 246.554 38.982
f924ffd9-c538bd02 5.035589 1142.230 190.372 38.748

CPU reduction11.45%; activation reduction22.79%. Production load[0.65234375, 0.96484375, 1.5537109375]→[0.8828125, 0.94580078125, 1.5078125]; candidate load[2.88720703125, 1.44580078125, 1.634765625]→[1.296875, 1.24267578125, 1.55126953125]. Both kernel counters ran100%; retained shim identities match and no terminal Close/reconnect notice was accepted. Cleanup remaining shims=[]; child PID stable in both windows.

Machine configuration observed on the named perf VM:

{"host": "infraternal-perf-test", "kernel": "6.12.111+deb13-amd64", "cpu_count": 4, "affinity_count": 4, "physical_memory_bytes": 8262770688}

Native artifactc538bd02 remains the f924ffd9 code; branch head8c4eb46f adds fixture validation only. Zero-client pair is running; one-client pair, Chromium wakeups and recovery suite are queued. Full iterate still waiting at VM calibration; no overall green or deploy claim. Wire-client timer counts above are not Chromium process wakeups; actual Chromium measurement is separate.

separate-comparison-6.log

Six-client acceptance row passes in the paired run:128 real PTYs and128 retained attached routes across6 sockets;60s settle +60s sample each; strict fixture220e01c8. | Native build | CPU% one core | Activations/s | Activations/s/client | Mean wire-client timer expirations/s/client | |---|---:|---:|---:|---:| | 9ab4329c-separate-control | 5.686600 | 1479.325 | 246.554 | 38.982 | | f924ffd9-c538bd02 | 5.035589 | 1142.230 | 190.372 | 38.748 | CPU reduction11.45%; activation reduction22.79%. Production load[0.65234375, 0.96484375, 1.5537109375]→[0.8828125, 0.94580078125, 1.5078125]; candidate load[2.88720703125, 1.44580078125, 1.634765625]→[1.296875, 1.24267578125, 1.55126953125]. Both kernel counters ran100%; retained shim identities match and no terminal Close/reconnect notice was accepted. Cleanup remaining shims=[]; child PID stable in both windows. Machine configuration observed on the named perf VM: ```json {"host": "infraternal-perf-test", "kernel": "6.12.111+deb13-amd64", "cpu_count": 4, "affinity_count": 4, "physical_memory_bytes": 8262770688} ``` Native artifactc538bd02 remains the f924ffd9 code; branch head8c4eb46f adds fixture validation only. Zero-client pair is running; one-client pair, Chromium wakeups and recovery suite are queued. Full iterate still waiting at VM calibration; no overall green or deploy claim. Wire-client timer counts above are not Chromium process wakeups; actual Chromium measurement is separate. [separate-comparison-6.log](https://git.kayg.org/attachments/0aa73c82-b257-4e21-b96b-f63fa3608841)
kayg commented 2026-10-15 05:49:33 +00:00
Author
Owner
Copy link

C538 full paired table: FAIL one-client CPU.128 real PTYs each;0/25/128 attached routes for0/1/6 sockets;60s settle +60s sample; strict fixture220e01c8.

Clients Production CPU% C538 CPU% Production activations/s C538 activations/s Result
0 0.000000 0.000000 0.000 0.000 PASS
1 1.641269 2.420881 383.719 339.467 FAIL CPU
6 5.686600 5.035589 1479.325 1142.230 PASS

One-client production load0.022/0.482/1.126→0.064/0.407/1.059; C5380.079/0.352/0.994→0.068/0.301/0.936. Child PID stable; SQL worker CPU remains zero on the candidate. The separate post-window997Hz CPU profile has649 samples,0 lost;275/649 sampled leaves are generic kernel spin unlock/task-switch functions. Unix socket wake paths are visible; these are not exclusively wakeup samples and do not establish which source change caused the extra CPU. Reverting0fed4495 (boxed workers plus skipped ready-reply yield) for the next measured trial, preserving all deadlines and the decoder fairness fix.

C538 passes all12 real-browser recovery cases. Real Chromium at128 PTYs,1 visible window,150ms configured message RTT:7.574 timer callbacks/s,6.326 control probes/s,2.700392% whole-browser CPU and176.023 activations/s over60s;1 socket retained, page errors0, typing still live. Original browser65.846 callbacks/s and11.8528% CPU. All4 measured Chromium PIDs2499909,2499973,2499927,2499930 are gone; private p1 runtime absent.

Full iterate reached VM calibration PASS (p50310.09ms,p99311.19ms) and native compilation, then was stopped after the CPU miss; no completed iterate/fast claim. Head8c4eb46f plus staged revert0fed4495; restored-worker targeted gates/build waiting for a build slot. No push/deploy.

separate-comparison-0.log
separate-comparison-1.log
separate-browser-idle.log
separate-browser-recovery-suite.log
separate-c538.perf.gz
separate-c538-flat-compact.txt
separate-comparison-6.log

C538 full paired table: FAIL one-client CPU.128 real PTYs each;0/25/128 attached routes for0/1/6 sockets;60s settle +60s sample; strict fixture220e01c8. | Clients | Production CPU% | C538 CPU% | Production activations/s | C538 activations/s | Result | |---:|---:|---:|---:|---:|---| |0|0.000000|0.000000|0.000|0.000|PASS| |1|1.641269|2.420881|383.719|339.467|FAIL CPU| |6|5.686600|5.035589|1479.325|1142.230|PASS| One-client production load0.022/0.482/1.126→0.064/0.407/1.059; C5380.079/0.352/0.994→0.068/0.301/0.936. Child PID stable; SQL worker CPU remains zero on the candidate. The separate post-window997Hz CPU profile has649 samples,0 lost;275/649 sampled leaves are generic kernel spin unlock/task-switch functions. Unix socket wake paths are visible; these are not exclusively wakeup samples and do not establish which source change caused the extra CPU. Reverting0fed4495 (boxed workers plus skipped ready-reply yield) for the next measured trial, preserving all deadlines and the decoder fairness fix. C538 passes all12 real-browser recovery cases. Real Chromium at128 PTYs,1 visible window,150ms configured message RTT:7.574 timer callbacks/s,6.326 control probes/s,2.700392% whole-browser CPU and176.023 activations/s over60s;1 socket retained, page errors0, typing still live. Original browser65.846 callbacks/s and11.8528% CPU. All4 measured Chromium PIDs2499909,2499973,2499927,2499930 are gone; private p1 runtime absent. Full iterate reached VM calibration PASS (p50310.09ms,p99311.19ms) and native compilation, then was stopped after the CPU miss; no completed iterate/fast claim. Head8c4eb46f plus staged revert0fed4495; restored-worker targeted gates/build waiting for a build slot. No push/deploy. [separate-comparison-0.log](https://git.kayg.org/attachments/ea93450f-fa09-4904-9db0-fbdca4978ccd) [separate-comparison-1.log](https://git.kayg.org/attachments/e36ae60e-1b07-423a-a8ad-672b49f3ffbd) [separate-browser-idle.log](https://git.kayg.org/attachments/5425d10c-7665-43a6-a4c4-383323b207b0) [separate-browser-recovery-suite.log](https://git.kayg.org/attachments/4cde580d-a515-4ee2-ae77-335bdf0af02f) [separate-c538.perf.gz](https://git.kayg.org/attachments/77c8f1ed-4e75-4fc2-96f8-3b64bc4b6f91) [separate-c538-flat-compact.txt](https://git.kayg.org/attachments/ac615564-920f-454e-86ab-f24be018ef7c) [separate-comparison-6.log](https://git.kayg.org/attachments/0aa73c82-b257-4e21-b96b-f63fa3608841)
kayg commented 2026-10-15 05:55:13 +00:00
Author
Owner
Copy link

Next experiment: head8c4eb46f plus staged revert0fed4495 restores concrete read/proof futures and the one-turn ready-reply collection. The failing benchmark is the strict one-client C538 row2.420881% versus production1.641269%. fmt and remote/all-target clippy pass;12 WebSocket tests (3 ignored),39 socket-authority tests (2 ignored) and8 native correlation tests pass. Static release building; no performance success claimed.

Temporary diagnostic change: .cargo/rust-lld.sh appends -Map=$HOME/.cache/codex-scratch/idle-cpu-199/evidence/restored-workers-link.map for this link. Restore its original final line exec -a ld.lld "$rust_lld" "$@" as soon as the release finishes, before committing. This emits an external receipt only; no change to optimization or static-link settings. Native scheduler revert remains uncommitted until measured acceptance.

C538 current full table remains FAIL one-client CPU, despite0/6 passing and all12 recovery cases/Chromium backoff passing. Full iterate was stopped on that rejected artifact and archived; fast has not run.

Next experiment: head8c4eb46f plus staged revert0fed4495 restores concrete read/proof futures and the one-turn ready-reply collection. The failing benchmark is the strict one-client C538 row2.420881% versus production1.641269%. fmt and remote/all-target clippy pass;12 WebSocket tests (3 ignored),39 socket-authority tests (2 ignored) and8 native correlation tests pass. Static release building; no performance success claimed. Temporary diagnostic change: .cargo/rust-lld.sh appends -Map=$HOME/.cache/codex-scratch/idle-cpu-199/evidence/restored-workers-link.map for this link. Restore its original final line exec -a ld.lld "$rust_lld" "$@" as soon as the release finishes, before committing. This emits an external receipt only; no change to optimization or static-link settings. Native scheduler revert remains uncommitted until measured acceptance. C538 current full table remains FAIL one-client CPU, despite0/6 passing and all12 recovery cases/Chromium backoff passing. Full iterate was stopped on that rejected artifact and archived; fast has not run.
kayg commented 2026-10-15 06:03:14 +00:00
Author
Owner
Copy link

Unboxed restored-worker experiment65378e6120f1ad3e79953f5067d73ad6186ea3a73c66d5c291a145ede59cf926 builds and passes12 WebSocket,39 socket-authority and8 correlation tests, but is22,345,936 bytes: +68,544 versus base,3,008 above the65,536-byte growth limit. Not committed or accepted; no owner exception requested. Temporary linker map helper was restored after its build.

Next experiment retains the existing once-per-connection future type erasure and restores only unconditional ready-reply collection before sending. Current staged diff is8 lines in websocket_transport.rs; head8c4eb46f. This isolates scheduling and retains the size savings. Its failing benchmark remains C538 one-client2.420881% vs production1.641269%. Targeted gates/build running or waiting for slot; no performance success claimed.

For this next link only, .cargo/rust-lld.sh temporarily appends -Map=$HOME/.cache/codex-scratch/idle-cpu-199/evidence/reply-turn-link.map. Restore the original exec -a ld.lld "$rust_lld" "$@" immediately after the build, before any commit. No probe timing or deadline changes.

Unboxed restored-worker experiment65378e6120f1ad3e79953f5067d73ad6186ea3a73c66d5c291a145ede59cf926 builds and passes12 WebSocket,39 socket-authority and8 correlation tests, but is22,345,936 bytes: +68,544 versus base,3,008 above the65,536-byte growth limit. Not committed or accepted; no owner exception requested. Temporary linker map helper was restored after its build. Next experiment retains the existing once-per-connection future type erasure and restores only unconditional ready-reply collection before sending. Current staged diff is8 lines in websocket_transport.rs; head8c4eb46f. This isolates scheduling and retains the size savings. Its failing benchmark remains C538 one-client2.420881% vs production1.641269%. Targeted gates/build running or waiting for slot; no performance success claimed. For this next link only, .cargo/rust-lld.sh temporarily appends -Map=$HOME/.cache/codex-scratch/idle-cpu-199/evidence/reply-turn-link.map. Restore the original exec -a ld.lld "$rust_lld" "$@" immediately after the build, before any commit. No probe timing or deadline changes.
kayg commented 2026-10-15 06:54:01 +00:00
Author
Owner
Copy link

Final continuation start: idle-cpu-199 at f2cfb4da1 (accepted/pushed head). Read #199 history; no parent issue is referenced. Owner accepts the one-client idle gap: candidate 2.420881% vs production 1.641269% of one core (2.42% vs 1.64%). No further idle-CPU experiments; the saved unfinished reply-turn patch will not be reapplied.

Only remaining change: correlated per-route SRTT/RTTVAR (alpha 1/8, beta 1/4), response allowance max(SRTT + 4*RTTVAR + 50ms, measured floor), preserving immutable outstanding deadlines and existing probe cadence. Initial floor candidate 250ms, to cover the required 150ms stall. This proposes the corresponding additive wire §9 clarification before implementation (D265/D267); no wire tags or recovery/input rules change. Red tests first, real socket under bounded same-core CPU stress, paired 1/6-client VM checks, build-slot iterate/fast, atomic commits/push only idle-cpu-199. No deploy.

Final continuation start: idle-cpu-199 at f2cfb4da1 (accepted/pushed head). Read #199 history; no parent issue is referenced. Owner accepts the one-client idle gap: candidate 2.420881% vs production 1.641269% of one core (2.42% vs 1.64%). No further idle-CPU experiments; the saved unfinished reply-turn patch will not be reapplied. Only remaining change: correlated per-route SRTT/RTTVAR (alpha 1/8, beta 1/4), response allowance max(SRTT + 4*RTTVAR + 50ms, measured floor), preserving immutable outstanding deadlines and existing probe cadence. Initial floor candidate 250ms, to cover the required 150ms stall. This proposes the corresponding additive wire §9 clarification before implementation (D265/D267); no wire tags or recovery/input rules change. Red tests first, real socket under bounded same-core CPU stress, paired 1/6-client VM checks, build-slot iterate/fast, atomic commits/push only idle-cpu-199. No deploy.
kayg commented 2026-10-15 06:58:44 +00:00
Author
Owner
Copy link

Deadline red/green: new paused native cases fail 3/3 before (150ms stall, floor bound, jitter); real Unix-socket peer fails healthy reply after 150ms server stall. Browser stall tests (1 and 28 streams) and floor-bound case fail before. After estimator/floor: native driver 14/14, browser connection tests 35/35 (93 assertions), including R199-M1 and R199-M2 correlation controls. No performance claim from build-host test durations.

Both routes now learn SRTT/RTTVAR only from correlated unexpired replies; variation updates before SRTT. Response allowance max(SRTT+4*RTTVAR+50ms,250ms); startup grace remains2s, probe cadence unchanged. A new probe on a fully answered route starts its allowance at actual issuance; further requests do not move an outstanding deadline. 250ms provisionally covers the required150ms stall plus100ms dispatch headroom; real same-core-stress VM oracle is queued. Heavy per-module gates waiting for a build slot; no commit/gate-complete claim yet.

deadline-native-red.log
deadline-real-socket-red.log
deadline-browser-red.log
deadline-driver-green.log
deadline-browser-green.log

Deadline red/green: new paused native cases fail 3/3 before (150ms stall, floor bound, jitter); real Unix-socket peer fails healthy reply after 150ms server stall. Browser stall tests (1 and 28 streams) and floor-bound case fail before. After estimator/floor: native driver 14/14, browser connection tests 35/35 (93 assertions), including R199-M1 and R199-M2 correlation controls. No performance claim from build-host test durations. Both routes now learn SRTT/RTTVAR only from correlated unexpired replies; variation updates before SRTT. Response allowance max(SRTT+4*RTTVAR+50ms,250ms); startup grace remains2s, probe cadence unchanged. A new probe on a fully answered route starts its allowance at actual issuance; further requests do not move an outstanding deadline. 250ms provisionally covers the required150ms stall plus100ms dispatch headroom; real same-core-stress VM oracle is queued. Heavy per-module gates waiting for a build slot; no commit/gate-complete claim yet. [deadline-native-red.log](https://git.kayg.org/attachments/ea17b62c-b160-45da-9dcb-75957d41c1b3) [deadline-real-socket-red.log](https://git.kayg.org/attachments/e64d3282-4299-422a-bb41-d1018798d6b8) [deadline-browser-red.log](https://git.kayg.org/attachments/c9ceddbc-3886-4167-bd1b-ce1bdb115034) [deadline-driver-green.log](https://git.kayg.org/attachments/e27c5056-2224-47b6-b3f3-d02ac76997bc) [deadline-browser-green.log](https://git.kayg.org/attachments/9f77d7f4-5fe2-4c7d-86ef-7949fde8e438)
kayg commented 2026-10-15 07:21:41 +00:00
Author
Owner
Copy link

Floor measurement on locked perf VM:20/20 real Unix-socket trials passed with healthy peer, server scheduling turn held150ms and a bounded CPU busy loop pinned with the test to CPU0 in this job’s own CPUWeight=10000 scope. Quiet correlated replies 33–84us; held replies 150110–150182us. The250ms response floor leaves 99818us above the largest observed held reply. Load and cleanup proof are in the raw receipt; generator stopped before any further measurement.

D265 trade-off is explicit: settled low-RTT silence is detected within250ms plus the existing≤26ms probe gap; RTT-dominated routes follow SRTT+4*RTTVAR+50ms. This protects the observed150ms stall rather than treating it as a1ms peer failure. No extra probes/timers. Actual dead-peer and decay bounds are covered with paused time, not host timing. The owner’s final brief authorizes this scheduling floor; connection recovery/input/correlation invariants remain.

Earlier full server gate:448 passed/1 failed/47 ignored; held-open test hit the shared90s paused-clock watchdog. Identical isolated repeat passes1/0; no source timeout or assertion changed and no baseline waiver claimed. Final full iterate/fast will run serial tests. Focused per-commit checks are waiting for a build slot.

deadline-stress.log

Floor measurement on locked perf VM:20/20 real Unix-socket trials passed with healthy peer, server scheduling turn held150ms and a bounded CPU busy loop pinned with the test to CPU0 in this job’s own CPUWeight=10000 scope. Quiet correlated replies 33–84us; held replies 150110–150182us. The250ms response floor leaves 99818us above the largest observed held reply. Load and cleanup proof are in the raw receipt; generator stopped before any further measurement. D265 trade-off is explicit: settled low-RTT silence is detected within250ms plus the existing≤26ms probe gap; RTT-dominated routes follow SRTT+4*RTTVAR+50ms. This protects the observed150ms stall rather than treating it as a1ms peer failure. No extra probes/timers. Actual dead-peer and decay bounds are covered with paused time, not host timing. The owner’s final brief authorizes this scheduling floor; connection recovery/input/correlation invariants remain. Earlier full server gate:448 passed/1 failed/47 ignored; held-open test hit the shared90s paused-clock watchdog. Identical isolated repeat passes1/0; no source timeout or assertion changed and no baseline waiver claimed. Final full iterate/fast will run serial tests. Focused per-commit checks are waiting for a build slot. [deadline-stress.log](https://git.kayg.org/attachments/f7b53ba2-c3b3-40d9-bf6a-2d0c8ba8c0af)
kayg commented 2026-10-15 07:25:17 +00:00
Author
Owner
Copy link

Completed/pushed atomic steps:1819a23b7 native estimator/floor and real-socket regressions;33a2836e3 browser estimator/floor preserving the independent existing link-doubt/cadence signal;c87fc063 protocol clarification.

Per-commit checks:fmt and remote all-target server Clippy PASS;13 liveness unit cases pass in the full server attempt;native integration M2 and real-socket cases pass separately;unchanged held-open regression isolated repeat passes. Tracer strict TypeScript PASS;36 connection tests/95 assertions PASS, including M1 strengthened to1s timer delay and queued healthy150ms reply. VM stress20/20 PASS (150.110–150.182ms);quiet dead-peer paused bound≤276ms;RTT variation shrinks after a stable link. The earlier full-server watchdog failure remains recorded; no full-suite waiver.

Final serial iterate/fast on c87fc063 is queued through build-slot. Next:one paired1/6-client perf run against accepted f2cfb4da1 artifact, production-bundle real restart controls, size delta against dev-merge-199 (17fae13f), exact-head gate receipts. No more idle-CPU changes or reply-turn experiments. No deploy.

Completed/pushed atomic steps:1819a23b7 native estimator/floor and real-socket regressions;33a2836e3 browser estimator/floor preserving the independent existing link-doubt/cadence signal;c87fc063 protocol clarification. Per-commit checks:fmt and remote all-target server Clippy PASS;13 liveness unit cases pass in the full server attempt;native integration M2 and real-socket cases pass separately;unchanged held-open regression isolated repeat passes. Tracer strict TypeScript PASS;36 connection tests/95 assertions PASS, including M1 strengthened to1s timer delay and queued healthy150ms reply. VM stress20/20 PASS (150.110–150.182ms);quiet dead-peer paused bound≤276ms;RTT variation shrinks after a stable link. The earlier full-server watchdog failure remains recorded; no full-suite waiver. Final serial iterate/fast on c87fc063 is queued through build-slot. Next:one paired1/6-client perf run against accepted f2cfb4da1 artifact, production-bundle real restart controls, size delta against dev-merge-199 (17fae13f), exact-head gate receipts. No more idle-CPU changes or reply-turn experiments. No deploy.
kayg commented 2026-10-15 07:30:19 +00:00
Author
Owner
Copy link

Validation resource wait at07:30 UTC: final-head gate is still queued for a host build slot (the existing waiter began07:14 UTC; no second slot-wrapped job was started). All implementation commits through c87fc0631 are pushed;20/20 VM stress passed. A build slot must become available before the full iterate/fast and new static artifact can run; paired1/6-client measurements and restart controls depend on that artifact. Continuing to wait, no other job interrupted and no gate waived.

Validation resource wait at07:30 UTC: final-head gate is still queued for a host build slot (the existing waiter began07:14 UTC; no second slot-wrapped job was started). All implementation commits through c87fc0631 are pushed;20/20 VM stress passed. A build slot must become available before the full iterate/fast and new static artifact can run; paired1/6-client measurements and restart controls depend on that artifact. Continuing to wait, no other job interrupted and no gate waived.
kayg commented 2026-10-15 07:43:21 +00:00
Author
Owner
Copy link

Resource blocker at07:43 UTC: final gate waiterPID140982 has waited since07:14 UTC (about29min) for one of the four build slots. It is still only sleeping in build-slot; no heavy command from this job is running. All four slot locks remain occupied. The orchestrator needs to make a build slot available/prioritize this release blocker; this job cannot safely free another lane’s slot. Continuing to wait, no other process interrupted.

Head c87fc0631 is pushed. Code/focused checks and20/20 same-core VM stress are complete; full serial iterate/fast, the new static artifact, its size delta and the paired1/6-client CPU/restart validation remain dependent on a slot. No deploy or completed-release-gate claim.

Resource blocker at07:43 UTC: final gate waiterPID140982 has waited since07:14 UTC (about29min) for one of the four build slots. It is still only sleeping in build-slot; no heavy command from this job is running. All four slot locks remain occupied. The orchestrator needs to make a build slot available/prioritize this release blocker; this job cannot safely free another lane’s slot. Continuing to wait, no other process interrupted. Head c87fc0631 is pushed. Code/focused checks and20/20 same-core VM stress are complete; full serial iterate/fast, the new static artifact, its size delta and the paired1/6-client CPU/restart validation remain dependent on a slot. No deploy or completed-release-gate claim.
kayg commented 2026-10-15 08:07:44 +00:00
Author
Owner
Copy link

Final iterate attempt stopped at CLI69passed/7failed/1ignored. This is my invocation error: TMPDIR was /proc/463182/fd/197, an alias to the private i199 root. Restart security tests intentionally reject alias paths and compare their canonical admitted path; the failure explicitly compares /proc/463182/fd/197/restart-tests/.../installed with /home/kayg/.cache/codex-scratch/i199/restart-tests/.../installed. No deadline-source failure is claimed from this attempt.

Corrected both TMPDIR and INFRATERNAL_TEST_STATE_ROOT to a new real, mode0700 job-owned directory /home/kayg/.cache/codex-scratch/i9 (34bytes; the longest g<24hex>/sessions/<32hex>.sock fixture fits107bytes). No symlink/proc alias; no test assertion, timeout, source or gate threshold changed. Full serial iterate/fast on unchanged pushed c87fc0631 is queued again through build-slot/own CPUWeight10000 scope. Failed transcript retained as deadline-verify-iterate-proc-root.log; focused canonical-path verification in progress. No waiver.

Final iterate attempt stopped at CLI69passed/7failed/1ignored. This is my invocation error: TMPDIR was /proc/463182/fd/197, an alias to the private i199 root. Restart security tests intentionally reject alias paths and compare their canonical admitted path; the failure explicitly compares /proc/463182/fd/197/restart-tests/.../installed with /home/kayg/.cache/codex-scratch/i199/restart-tests/.../installed. No deadline-source failure is claimed from this attempt. Corrected both TMPDIR and INFRATERNAL_TEST_STATE_ROOT to a new real, mode0700 job-owned directory /home/kayg/.cache/codex-scratch/i9 (34bytes; the longest g<24hex>/sessions/<32hex>.sock fixture fits107bytes). No symlink/proc alias; no test assertion, timeout, source or gate threshold changed. Full serial iterate/fast on unchanged pushed c87fc0631 is queued again through build-slot/own CPUWeight10000 scope. Failed transcript retained as deadline-verify-iterate-proc-root.log; focused canonical-path verification in progress. No waiver.
kayg commented 2026-10-15 08:17:51 +00:00
Author
Owner
Copy link

Comparison provenance finding before measuring: retained separate-control artifact SHA256 c538bd0204f65ac51ecbccb720359b5837805454427d9b585882b4741e6d90d1 is native f924ffd9, not exact accepted f2cfb4da1. f2cf changes production Liveness::ping in crates/infraternal-server/src/liveness.rs as well as tests. It must not be labelled an exact accepted-head baseline. The owner-accepted historical CPU numbers remain2.420881% vs1.641269%; this correction concerns the new paired deadline validation.

The existing clean scratch baseline checkout now points to exact f2cfb4da16. The queued single build-slot scope will build that stripped x86 musl release, then run current c87fc0631 iterate/fast; same one target folder and recipe, Rust source mtimes invalidated between revisions, artifact SHA/size retained. Paired1/6-client checks will use fresh deadline-baseline vs deadline with matching old/new browser response estimators. No current-source or idle-CPU changes, no extra CPU trial. Full gate retry still uses the corrected canonical34-byte runtime root.

Comparison provenance finding before measuring: retained separate-control artifact SHA256 c538bd0204f65ac51ecbccb720359b5837805454427d9b585882b4741e6d90d1 is native f924ffd9, not exact accepted f2cfb4da1. f2cf changes production Liveness::ping in crates/infraternal-server/src/liveness.rs as well as tests. It must not be labelled an exact accepted-head baseline. The owner-accepted historical CPU numbers remain2.420881% vs1.641269%; this correction concerns the new paired deadline validation. The existing clean scratch baseline checkout now points to exact f2cfb4da16a193df87b42c976548d272c4c7db1a. The queued single build-slot scope will build that stripped x86 musl release, then run current c87fc0631 iterate/fast; same one target folder and recipe, Rust source mtimes invalidated between revisions, artifact SHA/size retained. Paired1/6-client checks will use fresh deadline-baseline vs deadline with matching old/new browser response estimators. No current-source or idle-CPU changes, no extra CPU trial. Full gate retry still uses the corrected canonical34-byte runtime root.
kayg commented 2026-10-15 08:54:00 +00:00
Author
Owner
Copy link

Completed exact-head iterate on pushed c87fc0631:PASS. Corrected canonical runtime root;76 CLI cases pass, full native workspace1054passed/0failed/139configured ignores. Earlier held-open watchdog case passes unchanged in this full serial run. Workspace Clippy and cargo-deny PASS. Full TypeScript gate PASS (191 canvas/build cases plus37 protocol cases/34463 assertions); existing36 connection cases/95 assertions also pass. No failure waiver or threshold change.

Fresh exact accepted f2cfb4da1 baseline:22328784B, SHA256 ef8bdc9291d4d711e279e9a507e254e56c8cb0af771ac690e3a9a4ec3baf8d91; perf-VM copy matches. Current-head full fast (including static builds) now runs in the same canonical build slot/own weighted scope. Paired1/6-client/restart runs await that artifact. No additional idle-CPU changes or deployment.

Verbatim iterate Rust summary (not the final gate):

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
+---------------------------+--------------+

deadline-verify-iterate.log
deadline-baseline-artifact.txt

Completed exact-head iterate on pushed c87fc0631:PASS. Corrected canonical runtime root;76 CLI cases pass, full native workspace1054passed/0failed/139configured ignores. Earlier held-open watchdog case passes unchanged in this full serial run. Workspace Clippy and cargo-deny PASS. Full TypeScript gate PASS (191 canvas/build cases plus37 protocol cases/34463 assertions); existing36 connection cases/95 assertions also pass. No failure waiver or threshold change. Fresh exact accepted f2cfb4da1 baseline:22328784B, SHA256 ef8bdc9291d4d711e279e9a507e254e56c8cb0af771ac690e3a9a4ec3baf8d91; perf-VM copy matches. Current-head full fast (including static builds) now runs in the same canonical build slot/own weighted scope. Paired1/6-client/restart runs await that artifact. No additional idle-CPU changes or deployment. Verbatim iterate Rust summary (not the final gate): ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | +---------------------------+--------------+ ``` [deadline-verify-iterate.log](https://git.kayg.org/attachments/5cb812f2-6675-47df-b544-954320ce267d) [deadline-baseline-artifact.txt](https://git.kayg.org/attachments/7e558854-0ddd-42d1-a64a-83ae1ec2cf19)
kayg commented 2026-10-15 09:16:39 +00:00
Author
Owner
Copy link

Current x86 static22329104B, SHA256 d1da1c07f9947a944ef0ff501f2bfd2e05ec51998f75228d0f84e85dd081d794 (VM copy matches). Delta+320B vs exact accepted f2cf22328784B;+9408B vs dev-merge-19917fa22319696B;+51712B vs actual merge-base1b23ef3122277392B (includes accepted earlier work). Static/no PT_INTERP, production trust and x86 key-memory initialization1 PASS; existing x86 cap22204304B fails by124800B. ARM/final fast summary still building. All1054 workspace tests pass0fail/139ignores in fast too.

First new production browser suite11/12PASS; remaining restart.spec CA-locked case fails initial page.goto at restart-fixture.ts:107 with ERR_NETWORK_CHANGED, before sign-in or any route probe. Retain this failure; whole12-case suite reruns unchanged. Both first paired commands fail before server creation because the stopped earlier run left private p1 directory (safe-fixture.py:274 root.mkdir guard). No CPU window started and no row is accepted. Driver now uses fresh, confirmed-absent d1/d6 private runtime directories; guards/assertions unchanged. Valid paired1/6 runs and browser rerun are queued under separate short perf locks/owner files. No source or idle-CPU changes; no failure waiver.

Current x86 static22329104B, SHA256 d1da1c07f9947a944ef0ff501f2bfd2e05ec51998f75228d0f84e85dd081d794 (VM copy matches). Delta+320B vs exact accepted f2cf22328784B;+9408B vs dev-merge-19917fa22319696B;+51712B vs actual merge-base1b23ef3122277392B (includes accepted earlier work). Static/no PT_INTERP, production trust and x86 key-memory initialization1 PASS; existing x86 cap22204304B fails by124800B. ARM/final fast summary still building. All1054 workspace tests pass0fail/139ignores in fast too. First new production browser suite11/12PASS; remaining restart.spec CA-locked case fails initial page.goto at restart-fixture.ts:107 with ERR_NETWORK_CHANGED, before sign-in or any route probe. Retain this failure; whole12-case suite reruns unchanged. Both first paired commands fail before server creation because the stopped earlier run left private p1 directory (safe-fixture.py:274 root.mkdir guard). No CPU window started and no row is accepted. Driver now uses fresh, confirmed-absent d1/d6 private runtime directories; guards/assertions unchanged. Valid paired1/6 runs and browser rerun are queued under separate short perf locks/owner files. No source or idle-CPU changes; no failure waiver.
kayg commented 2026-10-15 09:27:23 +00:00
Author
Owner
Copy link

Paired deadline validation on the perf VM: 128 real PTYs, 60 s settle + 60 s sample, inherited perf stat counters at 100% coverage. Exact accepted f2cfb4da1 versus c87fc0631 (artifact d1da1c07); no active CPU generator. All retained sockets/PTY identities survived, server child PID stayed stable, and every run reports remaining_shims=[].

Clients Viewed PTYs Revision CPU % one core Activations/s Wire-client timers/s/client Load before → after (1/5/15m)
1 25 f2cfb4da1-accepted 1.862517 339.580 38.698 1.177/1.453/0.822 → 0.504/1.203/0.775
1 25 deadline-srtt-floor250 2.006826 341.153 38.682 0.251/0.994/0.729 → 0.132/0.824/0.688
6 128 f2cfb4da1-accepted 5.085750 1147.484 38.765 0.294/0.760/0.677 → 5.831/2.367/1.236
6 128 deadline-srtt-floor250 5.127020 1156.722 38.781 2.843/2.157/1.236 → 1.373/1.841/1.183

Measured point deltas: one client +0.144309 percentage points CPU (+7.75%) and +1.573 activations/s; six clients +0.041270 points CPU (+0.81%) and +9.238 activations/s. These single paired samples do not establish zero CPU change or statistical significance. I will not claim the requested “no measurable idle CPU change” as proven. No further idle-CPU changes or experiments, per the owner’s stop instruction. The new one-client point (2.007%) is below the historically accepted 2.42% point, but that historical difference is not a same-window comparison.

The old saved p1-directory attempts started no server/sample and are excluded. New pairs use clean d1/d6 directories and unchanged validity guards. Browser controls: 11/12 then 10/12 passed, with every failure at initial page.goto ERR_NETWORK_CHANGED before sign-in/liveness; the remaining ordinary CA-locked case is queued alone in a fresh browser with all assertions unchanged. All other unique restart scenarios have valid passing executions. Full fast ARM build/final summary is still running; x86 linkage/trust/key-memory and all 1054 native tests pass.
deadline-pair-1.log
deadline-pair-6.log
deadline-browser-tests-initial-navigation.log
deadline-browser-tests-navigation-repeat.log

Paired deadline validation on the perf VM: 128 real PTYs, 60 s settle + 60 s sample, inherited perf stat counters at 100% coverage. Exact accepted f2cfb4da1 versus c87fc0631 (artifact d1da1c07); no active CPU generator. All retained sockets/PTY identities survived, server child PID stayed stable, and every run reports remaining_shims=[]. | Clients | Viewed PTYs | Revision | CPU % one core | Activations/s | Wire-client timers/s/client | Load before → after (1/5/15m) | |---:|---:|---|---:|---:|---:|---| | 1 | 25 | f2cfb4da1-accepted | 1.862517 | 339.580 | 38.698 | 1.177/1.453/0.822 → 0.504/1.203/0.775 | | 1 | 25 | deadline-srtt-floor250 | 2.006826 | 341.153 | 38.682 | 0.251/0.994/0.729 → 0.132/0.824/0.688 | | 6 | 128 | f2cfb4da1-accepted | 5.085750 | 1147.484 | 38.765 | 0.294/0.760/0.677 → 5.831/2.367/1.236 | | 6 | 128 | deadline-srtt-floor250 | 5.127020 | 1156.722 | 38.781 | 2.843/2.157/1.236 → 1.373/1.841/1.183 | Measured point deltas: one client +0.144309 percentage points CPU (+7.75%) and +1.573 activations/s; six clients +0.041270 points CPU (+0.81%) and +9.238 activations/s. These single paired samples do not establish zero CPU change or statistical significance. I will not claim the requested “no measurable idle CPU change” as proven. No further idle-CPU changes or experiments, per the owner’s stop instruction. The new one-client point (2.007%) is below the historically accepted 2.42% point, but that historical difference is not a same-window comparison. The old saved p1-directory attempts started no server/sample and are excluded. New pairs use clean d1/d6 directories and unchanged validity guards. Browser controls: 11/12 then 10/12 passed, with every failure at initial page.goto ERR_NETWORK_CHANGED before sign-in/liveness; the remaining ordinary CA-locked case is queued alone in a fresh browser with all assertions unchanged. All other unique restart scenarios have valid passing executions. Full fast ARM build/final summary is still running; x86 linkage/trust/key-memory and all 1054 native tests pass. [deadline-pair-1.log](https://git.kayg.org/attachments/7dfe8c7c-4cf4-4c5d-a101-396007d68984) [deadline-pair-6.log](https://git.kayg.org/attachments/c7039750-0a7c-4bdd-bb4f-07251026cdc1) [deadline-browser-tests-initial-navigation.log](https://git.kayg.org/attachments/5e6c94de-e83f-459e-be46-d01028bb8239) [deadline-browser-tests-navigation-repeat.log](https://git.kayg.org/attachments/26bb1e29-eaa3-4822-90bb-d207c88dbe90)
kayg commented 2026-10-15 09:29:50 +00:00
Author
Owner
Copy link

Remaining CA-locked ordinary restart case passed unchanged in a fresh browser (BROWSER PASSED). All 12 unique real production-browser/PTY restart scenarios now have passing executions on c87fc0631 artifact d1da1c07: retained/closed browser route, CA locked/unlocked, daemon/shim stalls, three-window queued-input identity and sibling isolation. No reload/focus recovery added. Earlier whole invocations were 11/12 and 10/12; every failure was initial page.goto ERR_NETWORK_CHANGED before sign-in or route proof, retained in prior attachments. This is passing coverage across executions, not a claim that either whole invocation was green. No fixture, assertion, timeout or retry setting changed.

Current production-build locked/unlocked ready screenshots attached for owner review; no author visual-quality judgment. All paired CPU samples and route/cleanup validity checks are recorded in the preceding table. ARM/final fast summary remains in progress.

deadline-browser-locked.log
deadline-restart-locked-ready.png
deadline-restart-unlocked-ready.png

Remaining CA-locked ordinary restart case passed unchanged in a fresh browser (BROWSER PASSED). All 12 unique real production-browser/PTY restart scenarios now have passing executions on c87fc0631 artifact d1da1c07: retained/closed browser route, CA locked/unlocked, daemon/shim stalls, three-window queued-input identity and sibling isolation. No reload/focus recovery added. Earlier whole invocations were 11/12 and 10/12; every failure was initial page.goto ERR_NETWORK_CHANGED before sign-in or route proof, retained in prior attachments. This is passing coverage across executions, not a claim that either whole invocation was green. No fixture, assertion, timeout or retry setting changed. Current production-build locked/unlocked ready screenshots attached for owner review; no author visual-quality judgment. All paired CPU samples and route/cleanup validity checks are recorded in the preceding table. ARM/final fast summary remains in progress. [deadline-browser-locked.log](https://git.kayg.org/attachments/c51d6cfa-cb8d-47b6-a4e6-0f93b3f88c3b) ![deadline-restart-locked-ready.png](https://git.kayg.org/attachments/ed6afc91-d542-48f7-80c1-8a9b43d8acab) ![deadline-restart-unlocked-ready.png](https://git.kayg.org/attachments/9790e08f-0b2d-4ae0-afb4-ecb188851f1a)
kayg commented 2026-10-15 09:41:37 +00:00
Author
Owner
Copy link

Final deadline continuation result — branch idle-cpu-199, pushed head c87fc06314. Atomic commits: 1819a23b7 native response estimator/tests; 33a2836e3 browser estimator/tests; c87fc0631 protocol contract. No deploy, dev/main push, issue closure, further idle-CPU changes or reply-turn experiment.

Owner acceptance is recorded: historical one-client CPU 2.420881% versus production 1.641269% of one core (2.42% versus 1.64%). The required correctness repair is implemented: correlated per-route SRTT/RTTVAR (alpha 1/8, beta 1/4), response allowance max(SRTT + 4*RTTVAR + 50 ms, 250 ms). The 250 ms floor covers 20/20 healthy real-socket same-core-stress trials: 150.110–150.182 ms held replies, at least 99.818 ms headroom. Extra sends cannot renew an outstanding lease; an answered route arms only at actual issuance. Startup grace, exact correlation, probe cadence and the existing browser link-doubt signal are preserved.

Validation:

  • Full iterate PASS. Full fast: 1,054 native tests pass, zero failures, 139 configured ignores; fmt, VT fmt, CLI compile-out, workspace Clippy and cargo-deny PASS.
  • Strict TypeScript and full TypeScript gate PASS (191 canvas/build cases; 37 protocol cases, 34,463 assertions). Focused connection cases: 36 PASS / 95 assertions, including strengthened R199-M1. Native correlation/M2 and stall/dead-peer/jitter-decay cases PASS.
  • All 12 unique production-browser/real-PTY restart scenarios have passing executions. Whole invocations were 11/12 and 10/12 because initial page.goto saw ERR_NETWORK_CHANGED before sign-in/probes. Remaining ordinary CA-locked case passes unchanged in a fresh browser. This is passing coverage across executions, not a green single full-browser invocation. No assertion/timeout/retry setting changed; failed receipts remain attached.

CPU limitation: the single valid paired measurements do not prove the requested “no measurable idle CPU change.” The measured points are slightly higher; do not treat this as a statistical zero-change result. No more CPU work/runs were done after these valid pairs, per the stop instruction. Same perf VM, 128 real PTYs, 60 s settling + 60 s samples; inherited kernel counters at 100% coverage, stable child/shim identity and cleanup checks pass. Wire-client timer counts, loads and complete receipts are in the preceding table.

Clients Viewed PTYs Accepted f2cf CPU % Fix CPU % Accepted activations/s Fix activations/s
1 25 1.862517 2.006826 339.580 341.153
6 128 5.085750 5.127020 1147.484 1156.722

Static release sizes (same recipe, source freshness invalidated between baseline/current builds):

  • x86_64: 22,329,104 B; SHA256 d1da1c07f9947a944ef0ff501f2bfd2e05ec51998f75228d0f84e85dd081d794. +320 B versus exact accepted f2cf (22,328,784 B); +9,408 B versus dev-merge-199 17fae13f (22,319,696 B); +51,712 B versus actual merge base 1b23ef31 (22,277,392 B), including previously accepted work. This deadline slice adds 320 B with no dependency/features change; the inherited aggregate buys the earlier measured proof batching/non-renewing authority work already accepted on #199.
  • ARM: 19,752,920 B; SHA256 ee00ad22e5181e850bc3fb734591abe44b26678b4e0b32da80d879ac583c50de.
  • Both stripped/static, no PT_INTERP; production trust PASS. x86 key-memory initialization=1 under the 8 MiB memlock gate; ARM runtime check skipped on x86 as designed.
  • Only existing absolute caps fail: x86 +124,800 B; ARM +114,176 B. The test-static-size.py assertion also fails solely because it correctly rejects that same x86 cap. No cap/threshold changed. Fast stops before its TypeScript dispatch on this Rust failure; exact-head iterate includes the full TypeScript gate.

Verbatim final scripts/verify fast summary (exit 1):

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

Verbatim size diagnostics:

PT_INTERP: absent; size: 22329104 bytes; cap: 22204304 bytes; delta: +124800 bytes
PT_INTERP: absent; size: 19752920 bytes; cap: 19638744 bytes; delta: +114176 bytes

Author server/auth checklist completed: correlation/expiry precede training; variance uses previous SRTT; bounded issued-probe state and restart IDs preserved; ordinary traffic/extra sends cannot renew silence; watched revocation and absolute authority expiry remain independent of route allowance; application/rebind/writer/input gates unchanged; no wire/schema/library/role changes. New code/test roles and timing invariants are documented. Screenshots are attached for owner review, without author visual-quality judgment.

Cleanup: all eight measured server/daemon PIDs are gone, d1/d6 are absent, every owned VM stress/measurement/browser scope is inactive. New VM artifacts/scripts were removed; prior job receipts were untouched. Removed this job’s cargo target, temporary comparison worktree, old-browser red-test copy and private i9 runtime root. Kept evidence, reproducible harness inputs and immutable comparison artifacts outside the checkout; the saved unfinished patch was never reapplied. Tracked worktree is clean; preexisting untracked scripts/pycache remains untouched. Final push reports Everything up-to-date. Issue stays open; orchestrator owns the next release/staging step.

deadline-verify-fast.log
deadline-final-sizes.json
deadline-vm-cleanup.txt
deadline-host-cleanup.json
deadline-vm-artifact-cleanup.json
deadline-browser-green.log
deadline-driver-green.log

Final deadline continuation result — branch idle-cpu-199, pushed head c87fc063141d33658715eea42687330352e8bd5c. Atomic commits: 1819a23b7 native response estimator/tests; 33a2836e3 browser estimator/tests; c87fc0631 protocol contract. No deploy, dev/main push, issue closure, further idle-CPU changes or reply-turn experiment. Owner acceptance is recorded: historical one-client CPU 2.420881% versus production 1.641269% of one core (2.42% versus 1.64%). The required correctness repair is implemented: correlated per-route SRTT/RTTVAR (alpha 1/8, beta 1/4), response allowance max(SRTT + 4*RTTVAR + 50 ms, 250 ms). The 250 ms floor covers 20/20 healthy real-socket same-core-stress trials: 150.110–150.182 ms held replies, at least 99.818 ms headroom. Extra sends cannot renew an outstanding lease; an answered route arms only at actual issuance. Startup grace, exact correlation, probe cadence and the existing browser link-doubt signal are preserved. Validation: - Full iterate PASS. Full fast: 1,054 native tests pass, zero failures, 139 configured ignores; fmt, VT fmt, CLI compile-out, workspace Clippy and cargo-deny PASS. - Strict TypeScript and full TypeScript gate PASS (191 canvas/build cases; 37 protocol cases, 34,463 assertions). Focused connection cases: 36 PASS / 95 assertions, including strengthened R199-M1. Native correlation/M2 and stall/dead-peer/jitter-decay cases PASS. - All 12 unique production-browser/real-PTY restart scenarios have passing executions. Whole invocations were 11/12 and 10/12 because initial page.goto saw ERR_NETWORK_CHANGED before sign-in/probes. Remaining ordinary CA-locked case passes unchanged in a fresh browser. This is passing coverage across executions, not a green single full-browser invocation. No assertion/timeout/retry setting changed; failed receipts remain attached. CPU limitation: the single valid paired measurements do not prove the requested “no measurable idle CPU change.” The measured points are slightly higher; do not treat this as a statistical zero-change result. No more CPU work/runs were done after these valid pairs, per the stop instruction. Same perf VM, 128 real PTYs, 60 s settling + 60 s samples; inherited kernel counters at 100% coverage, stable child/shim identity and cleanup checks pass. Wire-client timer counts, loads and complete receipts are in the preceding table. | Clients | Viewed PTYs | Accepted f2cf CPU % | Fix CPU % | Accepted activations/s | Fix activations/s | |---:|---:|---:|---:|---:|---:| | 1 | 25 | 1.862517 | 2.006826 | 339.580 | 341.153 | | 6 | 128 | 5.085750 | 5.127020 | 1147.484 | 1156.722 | Static release sizes (same recipe, source freshness invalidated between baseline/current builds): - x86_64: 22,329,104 B; SHA256 d1da1c07f9947a944ef0ff501f2bfd2e05ec51998f75228d0f84e85dd081d794. +320 B versus exact accepted f2cf (22,328,784 B); +9,408 B versus dev-merge-199 17fae13f (22,319,696 B); +51,712 B versus actual merge base 1b23ef31 (22,277,392 B), including previously accepted work. This deadline slice adds 320 B with no dependency/features change; the inherited aggregate buys the earlier measured proof batching/non-renewing authority work already accepted on #199. - ARM: 19,752,920 B; SHA256 ee00ad22e5181e850bc3fb734591abe44b26678b4e0b32da80d879ac583c50de. - Both stripped/static, no PT_INTERP; production trust PASS. x86 key-memory initialization=1 under the 8 MiB memlock gate; ARM runtime check skipped on x86 as designed. - Only existing absolute caps fail: x86 +124,800 B; ARM +114,176 B. The test-static-size.py assertion also fails solely because it correctly rejects that same x86 cap. No cap/threshold changed. Fast stops before its TypeScript dispatch on this Rust failure; exact-head iterate includes the full TypeScript gate. Verbatim final scripts/verify fast summary (exit 1): ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ``` Verbatim size diagnostics: ```text PT_INTERP: absent; size: 22329104 bytes; cap: 22204304 bytes; delta: +124800 bytes PT_INTERP: absent; size: 19752920 bytes; cap: 19638744 bytes; delta: +114176 bytes ``` Author server/auth checklist completed: correlation/expiry precede training; variance uses previous SRTT; bounded issued-probe state and restart IDs preserved; ordinary traffic/extra sends cannot renew silence; watched revocation and absolute authority expiry remain independent of route allowance; application/rebind/writer/input gates unchanged; no wire/schema/library/role changes. New code/test roles and timing invariants are documented. Screenshots are attached for owner review, without author visual-quality judgment. Cleanup: all eight measured server/daemon PIDs are gone, d1/d6 are absent, every owned VM stress/measurement/browser scope is inactive. New VM artifacts/scripts were removed; prior job receipts were untouched. Removed this job’s cargo target, temporary comparison worktree, old-browser red-test copy and private i9 runtime root. Kept evidence, reproducible harness inputs and immutable comparison artifacts outside the checkout; the saved unfinished patch was never reapplied. Tracked worktree is clean; preexisting untracked scripts/__pycache__ remains untouched. Final push reports Everything up-to-date. Issue stays open; orchestrator owns the next release/staging step. [deadline-verify-fast.log](https://git.kayg.org/attachments/c1aa2384-0d8f-43b6-9204-004a83331d47) [deadline-final-sizes.json](https://git.kayg.org/attachments/0e68e1b4-32f7-4fa7-af54-21a1edf66843) [deadline-vm-cleanup.txt](https://git.kayg.org/attachments/82c3de28-daae-4f0b-b5f5-0bee87883f9b) [deadline-host-cleanup.json](https://git.kayg.org/attachments/be1624ee-9f40-49d1-a35e-e92e269efeac) [deadline-vm-artifact-cleanup.json](https://git.kayg.org/attachments/aebe67e2-3ab0-49c5-a99b-c27e1186b23e) [deadline-browser-green.log](https://git.kayg.org/attachments/a9eb61a3-d9da-4cbe-b689-c2c842043de9) [deadline-driver-green.log](https://git.kayg.org/attachments/e1e487ea-eed9-4615-8eea-09397f9cad1d)
kayg commented 2026-10-15 09:45:26 +00:00
Author
Owner
Copy link

Start switchover-release: branch switchover-release, base/head 036bc2bb06 (clean). Reading all #199 and related #195 history and dev-merge-199 invocation receipts. Build exact pinned-Zig x86/ARM release artifacts, web, signed isolated staging proof and 12 real restart/mock cases, then scripts/verify fast. One target directory; build-slot and CPUWeight=10000. No deployment, production access, owner fixture changes, or source changes planned. Preserve ordinary release bytes separately before staging-trust rebuild.

Start switchover-release: branch switchover-release, base/head 036bc2bb067a231736f219f32897aaa7d8942b70 (clean). Reading all #199 and related #195 history and dev-merge-199 invocation receipts. Build exact pinned-Zig x86/ARM release artifacts, web, signed isolated staging proof and 12 real restart/mock cases, then scripts/verify fast. One target directory; build-slot and CPUWeight=10000. No deployment, production access, owner fixture changes, or source changes planned. Preserve ordinary release bytes separately before staging-trust rebuild.
kayg commented 2026-10-15 09:45:38 +00:00
Author
Owner
Copy link

Independent review start: review-idle-cpu-199 at c87fc06314; requested merge base with local dev 8e7467c07c (origin/dev merge base 1b23ef31). Reading all 196 comments and report; issue has no parent reference. Functional review only: no perf VM, push, deployment or implementation fixes. scripts/verify fast started through build-slot on untouched target head; raw evidence under ~/.cache/codex-scratch/idle-cpu-199-review/evidence/.

Independent review start: review-idle-cpu-199 at c87fc063141d33658715eea42687330352e8bd5c; requested merge base with local dev 8e7467c07cd2ec1668a775a846076b278f68ef38 (origin/dev merge base 1b23ef31). Reading all 196 comments and report; issue has no parent reference. Functional review only: no perf VM, push, deployment or implementation fixes. scripts/verify fast started through build-slot on untouched target head; raw evidence under ~/.cache/codex-scratch/idle-cpu-199-review/evidence/.
kayg commented 2026-10-15 09:49:12 +00:00
Author
Owner
Copy link

Review step committed 9f63cda30 on review-idle-cpu-199: P2 R199-I1, scripts/bench-idle-routes-review.test.ts reproduces drift in the committed benchmark receiver (0 pass/1 fail, fixture late client proof at scripts/bench-idle-routes.py:163). A synthetic 1ms initial RTT then a correlated reply 150ms after issuance is accepted by shipping's 250ms floor but refused by the fixture's old RTT+50ms deadline. Both fixture deadline paths still omit SRTT/RTTVAR/floor; no claim that externally staged final drivers are identical to this committed source. Shipping focused connection controls: 34 pass/0 fail/87 assertions; existing Python fixture controls: 10 pass/0 fail. No new P0/P1, implementation fixes, push or performance measurements. Gate remains queued through build-slot, with no output yet. Evidence: ~/.cache/codex-scratch/idle-cpu-199-review/evidence/{fixture-deadline-red,browser-units,fixture-units,verify-fast}.log.

Review step committed 9f63cda30 on review-idle-cpu-199: P2 R199-I1, scripts/bench-idle-routes-review.test.ts reproduces drift in the committed benchmark receiver (0 pass/1 fail, fixture late client proof at scripts/bench-idle-routes.py:163). A synthetic 1ms initial RTT then a correlated reply 150ms after issuance is accepted by shipping's 250ms floor but refused by the fixture's old RTT+50ms deadline. Both fixture deadline paths still omit SRTT/RTTVAR/floor; no claim that externally staged final drivers are identical to this committed source. Shipping focused connection controls: 34 pass/0 fail/87 assertions; existing Python fixture controls: 10 pass/0 fail. No new P0/P1, implementation fixes, push or performance measurements. Gate remains queued through build-slot, with no output yet. Evidence: ~/.cache/codex-scratch/idle-cpu-199-review/evidence/{fixture-deadline-red,browser-units,fixture-units,verify-fast}.log.
kayg commented 2026-10-15 09:50:42 +00:00
Author
Owner
Copy link

Head: 036bc2bb06
Branch: switchover-release
rustls: 0.23.46
No source changes. Production artifacts use INFRATERNAL_RELEASE_BUILD=1 and default features. Staging procedure adds its required ephemeral staging-trust authority, uses a synthetic ARM placeholder and fixture web page as designed; it does not authenticate or execute the production ARM/web artifacts. Ordinary x86/ARM artifacts are retained separately and checked for absence of staging trust. Runtime staging test source remains unchanged; external operational adapter confines its remote directory to switchover-release, records lock owner, and bounds each hold to 1800s. Browser tests use the ordinary retained x86 image and production dist; mocks use a separate standard clusters-preview build. Gate calibration uses unchanged latency_proxy.py (119b87ef39ee7022c76dbc641a1428dd5eae7254265e6ddeea6697b56a4a8339) on locked perf VM, not shared host. No production access, deploy or owner fixture writes.

Head: 036bc2bb067a231736f219f32897aaa7d8942b70 Branch: switchover-release rustls: 0.23.46 No source changes. Production artifacts use INFRATERNAL_RELEASE_BUILD=1 and default features. Staging procedure adds its required ephemeral staging-trust authority, uses a synthetic ARM placeholder and fixture web page as designed; it does not authenticate or execute the production ARM/web artifacts. Ordinary x86/ARM artifacts are retained separately and checked for absence of staging trust. Runtime staging test source remains unchanged; external operational adapter confines its remote directory to switchover-release, records lock owner, and bounds each hold to 1800s. Browser tests use the ordinary retained x86 image and production dist; mocks use a separate standard clusters-preview build. Gate calibration uses unchanged latency_proxy.py (119b87ef39ee7022c76dbc641a1428dd5eae7254265e6ddeea6697b56a4a8339) on locked perf VM, not shared host. No production access, deploy or owner fixture writes.
kayg commented 2026-10-15 09:56:44 +00:00
Author
Owner
Copy link

Resource wait 2026-10-15 09:56 UTC: switchover-release build-slot PID1458788 has waited since09:46 UTC (~10min); compilation has not started. All four slot records are occupied. This job holds no build slot and has not interrupted other jobs. Highest-priority CPUWeight=10000 scope is configured; one target directory and one queued heavy command. Release artifacts, signed staging and final gate depend on that slot; private harness preparation is complete. Continuing to wait.

Resource wait 2026-10-15 09:56 UTC: switchover-release build-slot PID1458788 has waited since09:46 UTC (~10min); compilation has not started. All four slot records are occupied. This job holds no build slot and has not interrupted other jobs. Highest-priority CPUWeight=10000 scope is configured; one target directory and one queued heavy command. Release artifacts, signed staging and final gate depend on that slot; private harness preparation is complete. Continuing to wait.
kayg commented 2026-10-15 09:59:04 +00:00
Author
Owner
Copy link

Independent review validation wait: review-idle-cpu-199 at 9f63cda301, target c87fc0631. scripts/verify fast is still waiting in the required host build-slot wrapper and has produced zero bytes of gate output; no native gate has run. A host build slot must become available. No other job was interrupted. Functional controls pass: 36 connection tests/95 assertions, 10 existing fixture tests, repository artifact gate. P2 R199-I1 is committed (one failing fixture receiver regression). No P0/P1 confirmed. Full gate remains required; continuing to wait. No independent performance/restart acceptance or deployment claim.

Independent review validation wait: review-idle-cpu-199 at 9f63cda3011fe59fa583cbbb4a6f0274b10f18e7, target c87fc0631. scripts/verify fast is still waiting in the required host build-slot wrapper and has produced zero bytes of gate output; no native gate has run. A host build slot must become available. No other job was interrupted. Functional controls pass: 36 connection tests/95 assertions, 10 existing fixture tests, repository artifact gate. P2 R199-I1 is committed (one failing fixture receiver regression). No P0/P1 confirmed. Full gate remains required; continuing to wait. No independent performance/restart acceptance or deployment claim.
kayg commented 2026-10-15 10:02:24 +00:00
Author
Owner
Copy link

Independent native liveness check: 14 virtual-clock tests pass/0 fail, directly importing committed crates/infraternal-server/tests/native_probe_correlation_review.rs and shipping src/liveness.rs using pinned Tokio 1.53.2. The real-time socket measurement is filtered out; no independent CPU/latency claim. This is an isolated module check, not the full server/workspace gate. Source review covers exact probe IDs/timestamps, no renewal by extra sends, startup/restart, response floors/variance, proof queue/byte bounds, and non-renewing authority with committed revocation/absolute expiry. No additional confirmed findings. Full fast remains queued through build-slot.

fixture-deadline-red.log
browser-units.log
native-virtual-clock.log
fixture-units.log

Independent native liveness check: 14 virtual-clock tests pass/0 fail, directly importing committed crates/infraternal-server/tests/native_probe_correlation_review.rs and shipping src/liveness.rs using pinned Tokio 1.53.2. The real-time socket measurement is filtered out; no independent CPU/latency claim. This is an isolated module check, not the full server/workspace gate. Source review covers exact probe IDs/timestamps, no renewal by extra sends, startup/restart, response floors/variance, proof queue/byte bounds, and non-renewing authority with committed revocation/absolute expiry. No additional confirmed findings. Full fast remains queued through build-slot. [fixture-deadline-red.log](https://git.kayg.org/attachments/a40a9845-4ecc-4e6f-8241-558c041d6246) [browser-units.log](https://git.kayg.org/attachments/1420eb18-c45f-476c-bd0c-8f47564f0cd6) [native-virtual-clock.log](https://git.kayg.org/attachments/61be98c1-ca2c-421d-8fda-7467b12b7257) [fixture-units.log](https://git.kayg.org/attachments/ce8e6b47-efe8-4c56-9040-be57e505d197)
kayg commented 2026-10-15 10:02:56 +00:00
Author
Owner
Copy link

Release resource blocker at 2026-10-15T10:02:55.917098+00:00: switchover-release waiter PID1458788 has waited16min; no compilation has begun. Orchestrator: the approved highest-priority release needs a build slot; all four are occupied. Continuing one existing waiter, no other process interrupted. Prepared exact-head release/staging/browser/gate procedures remain unchanged. Slot ownership records:
slot-1.who: started 2026-10-15T11:04:21+02:00, job ?, PID 849674
slot-2.who: started 2026-10-15T11:45:00+02:00, job ?, PID 1420344
slot-3.who: started 2026-10-15T11:39:23+02:00, job ?, PID 980509
slot-4.who: started 2026-10-15T11:40:12+02:00, job research-nat, PID 1083815

Release resource blocker at 2026-10-15T10:02:55.917098+00:00: switchover-release waiter PID1458788 has waited16min; no compilation has begun. Orchestrator: the approved highest-priority release needs a build slot; all four are occupied. Continuing one existing waiter, no other process interrupted. Prepared exact-head release/staging/browser/gate procedures remain unchanged. Slot ownership records: slot-1.who: started 2026-10-15T11:04:21+02:00, job ?, PID 849674 slot-2.who: started 2026-10-15T11:45:00+02:00, job ?, PID 1420344 slot-3.who: started 2026-10-15T11:39:23+02:00, job ?, PID 980509 slot-4.who: started 2026-10-15T11:40:12+02:00, job research-nat, PID 1083815
kayg commented 2026-10-15 10:06:46 +00:00
Author
Owner
Copy link

Review checkpoint committed cf701478 on review-idle-cpu-199: docs/verification/idle-cpu-199-review.md records P2 R199-I1, scope, 36 passing browser tests/95 assertions, 14 passing shipping-source virtual-clock tests, 10 passing fixture tests, artifact/whitespace checks and limits. Reproducer remains 9f63cda30. No shipping source/dependency changes or push. Full fast is still queued with no output; serial native tests and a short canonical runtime root are configured. This is a checkpoint, not a completed gate or final verdict.

Review checkpoint committed cf701478 on review-idle-cpu-199: docs/verification/idle-cpu-199-review.md records P2 R199-I1, scope, 36 passing browser tests/95 assertions, 14 passing shipping-source virtual-clock tests, 10 passing fixture tests, artifact/whitespace checks and limits. Reproducer remains 9f63cda30. No shipping source/dependency changes or push. Full fast is still queued with no output; serial native tests and a short canonical runtime root are configured. This is a checkpoint, not a completed gate or final verdict.
kayg commented 2026-10-15 10:08:06 +00:00
Author
Owner
Copy link

Resource finding at10:09 UTC (process names only, no argv): slot2 has been held since09:45 by PID1420344; child1445739 is serve-loop.sh, with python3/bun serving children and a sleep. Other slot trees are compiling (cargo/make, cargo, cargo-clippy). The release waiter1458788 is still queued after22min. Orchestrator: a persistent serving loop is occupying one of the four heavy-build slots; please coordinate its release/prioritize switchover-release. This job has not signalled or modified any holder. Continuing the same waiter.

Resource finding at10:09 UTC (process names only, no argv): slot2 has been held since09:45 by PID1420344; child1445739 is serve-loop.sh, with python3/bun serving children and a sleep. Other slot trees are compiling (cargo/make, cargo, cargo-clippy). The release waiter1458788 is still queued after22min. Orchestrator: a persistent serving loop is occupying one of the four heavy-build slots; please coordinate its release/prioritize switchover-release. This job has not signalled or modified any holder. Continuing the same waiter.
kayg commented 2026-10-15 10:10:09 +00:00
Author
Owner
Copy link

Review test refinement b24e2b1d on review-idle-cpu-199: R199-I1 now issues both requests through actual Browser.sample_probe rather than manually constructing its pending map. The same committed receiver assertion at line163 fails (0 pass/1 fail, fixture late client proof) for the correlated reply after a virtual150ms delay. Same P2 finding/count; no implementation changes. Full fast remains queued through build-slot, with zero output.

fixture-deadline-issued-red.log

Review test refinement b24e2b1d on review-idle-cpu-199: R199-I1 now issues both requests through actual Browser.sample_probe rather than manually constructing its pending map. The same committed receiver assertion at line163 fails (0 pass/1 fail, fixture late client proof) for the correlated reply after a virtual150ms delay. Same P2 finding/count; no implementation changes. Full fast remains queued through build-slot, with zero output. [fixture-deadline-issued-red.log](https://git.kayg.org/attachments/cf7a66b9-5af4-4058-ada4-26f43b5c0c10)
kayg commented 2026-10-15 10:22:03 +00:00
Author
Owner
Copy link

Review report provenance updated in 1efdb05d on review-idle-cpu-199: records strengthened reproducer b24e2b1d and keeps release-size comparison explicitly pending, rather than inferring a measured zero delta from test/documentation scope. Four local atomic review commits; worktree clean, no push or implementation changes. Full fast still has zero bytes of output while queued through build-slot. All four lock files have live descriptors; three slots include active native builds and one includes serve-loop.sh/python3. Process names/descriptors only were inspected; no other process signalled. Orchestrator coordination is needed if the slot wait persists. Confirmed counts unchanged: P0=0/P1=0/P2=1/P3=0.

Review report provenance updated in 1efdb05d on review-idle-cpu-199: records strengthened reproducer b24e2b1d and keeps release-size comparison explicitly pending, rather than inferring a measured zero delta from test/documentation scope. Four local atomic review commits; worktree clean, no push or implementation changes. Full fast still has zero bytes of output while queued through build-slot. All four lock files have live descriptors; three slots include active native builds and one includes serve-loop.sh/python3. Process names/descriptors only were inspected; no other process signalled. Orchestrator coordination is needed if the slot wait persists. Confirmed counts unchanged: P0=0/P1=0/P2=1/P3=0.
kayg commented 2026-10-15 10:22:59 +00:00
Author
Owner
Copy link

2026-10-15T10:22:58.657207+00:00: release build acquired a host slot after ~36min and x86_64 compilation started. Pinned Zig0.15.2 executable SHA2562858dc89dbbfdd08cceda1b841e7fd0a793a1a67b49f150bc3d0d1de44ed7f51, Rust1.99.0, Bun1.4.2. One target folder, release marker1, default features, exact locked recipe. No other job was interrupted by this job.

2026-10-15T10:22:58.657207+00:00: release build acquired a host slot after ~36min and x86_64 compilation started. Pinned Zig0.15.2 executable SHA2562858dc89dbbfdd08cceda1b841e7fd0a793a1a67b49f150bc3d0d1de44ed7f51, Rust1.99.0, Bun1.4.2. One target folder, release marker1, default features, exact locked recipe. No other job was interrupted by this job.
kayg commented 2026-10-15 10:36:09 +00:00
Author
Owner
Copy link

Independent review resource blocker: approximately50min after the start comment, scripts/verify fast has still not acquired a host build slot and its log remains0bytes. Current serial-gate waiter is active; no native workspace/static gate output exists to quote. All four slot locks have live descriptors (last metadata-only check: three native builds; one serve-loop.sh/python3). No other job was interrupted, no slot bypassed, no perf VM used. Orchestrator must provide/coordinate a slot if the wait persists. Review branch remains clean at1efdb05d; P0=0/P1=0/P2=1/P3=0. Confirmed finding and focused passing checks are committed, attached and recorded above; full verification remains incomplete. Continuing the existing waiter.

Independent review resource blocker: approximately50min after the start comment, scripts/verify fast has still not acquired a host build slot and its log remains0bytes. Current serial-gate waiter is active; no native workspace/static gate output exists to quote. All four slot locks have live descriptors (last metadata-only check: three native builds; one serve-loop.sh/python3). No other job was interrupted, no slot bypassed, no perf VM used. Orchestrator must provide/coordinate a slot if the wait persists. Review branch remains clean at1efdb05d; P0=0/P1=0/P2=1/P3=0. Confirmed finding and focused passing checks are committed, attached and recorded above; full verification remains incomplete. Continuing the existing waiter.
kayg commented 2026-10-15 10:37:46 +00:00
Author
Owner
Copy link

Independent review verdict #199: MERGE (no P0/P1) — merge under the owner severity rule. Counts P0=0, P1=0, P2=1, P3=0. Independent full-gate verification remains INCOMPLETE / BLOCKED on host build capacity.

Target idle-cpu-199 at c87fc06314. Local review-idle-cpu-199 head 2dc926cc19, five atomic commits; worktree clean. Requested local-dev merge base 8e7467c07c; origin/dev base 1b23ef311c. Review scope and checklist: docs/verification/idle-cpu-199-review.md.

P2 R199-I1: scripts/bench-idle-routes.py:109/162 still use the old RTT+50ms response allowance. Committed TypeScript regression scripts/bench-idle-routes-review.test.ts (9f63cda30, strengthened b24e2b1d) issues both probes through actual Browser.sample_probe and feeds bounded in-memory replies at synthetic times. Initial1ms correlation passes; the next correlated reply150ms after issuance fails at receiver line163, fixture late client proof. Shipping's250ms floor admits it. Red0/1; same finding, no implementation repair. This establishes committed workload drift, not the source identity of externally staged final measurement drivers. P2 fixes forward under the owner rule.

Passing independent controls:36 browser connection tests/95 assertions;14 shipping-source native virtual-clock tests (real-time socket case excluded);10 existing Python fixture tests; artifact and review whitespace checks. No new dependency or shipping source. No independent performance/size/restart/signed-handoff/physical-iPhone/original-tab acceptance claim. The author's zero CPU-change limitation remains unchanged. No push, deploy, perf VM or other-job process changes.

Gate blocker: scripts/verify fast was invoked through required build-slot, with serial tests, short canonical scratch and TRACER_TEST_PORT4399. It never acquired a slot during this review; verify-fast.log stayed0bytes and the target directory was never created. Stopped only this review's queued waiter after the prolonged wait (verified child was sleep). There is no independent gate output or exit status to quote; it is neither PASS nor a test failure. Orchestrator must make a host build slot available and rerun the required gate to complete verification. Evidence and the exact invocation are retained below/in the local report. No other process was signalled and no slot was bypassed.

Author-reported scripts/verify fast summary ON TARGET c87fc0631, quoted verbatim from the provided job result.md; this is an attributed claim, not an independent run:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

Raw review receipts attached above; strengthened red receipt: https://git.kayg.org/attachments/cf7a66b9-5af4-4058-ada4-26f43b5c0c10 . Local evidence: ~/.cache/codex-scratch/idle-cpu-199-review/evidence/. Findings: ~/.local/state/codex-jobs/infraternal/review-idle-cpu-199/findings.md. All review build output was removed; no review test server or native fixture remains. Issue stays open.

Independent review verdict #199: MERGE (no P0/P1) — merge under the owner severity rule. Counts P0=0, P1=0, P2=1, P3=0. Independent full-gate verification remains INCOMPLETE / BLOCKED on host build capacity. Target idle-cpu-199 at c87fc063141d33658715eea42687330352e8bd5c. Local review-idle-cpu-199 head 2dc926cc1988c2355b6f41c84b8dfbd609d66990, five atomic commits; worktree clean. Requested local-dev merge base 8e7467c07cd2ec1668a775a846076b278f68ef38; origin/dev base 1b23ef311ce8f850ff01a60d06a420db33d26888. Review scope and checklist: docs/verification/idle-cpu-199-review.md. P2 R199-I1: scripts/bench-idle-routes.py:109/162 still use the old RTT+50ms response allowance. Committed TypeScript regression scripts/bench-idle-routes-review.test.ts (9f63cda30, strengthened b24e2b1d) issues both probes through actual Browser.sample_probe and feeds bounded in-memory replies at synthetic times. Initial1ms correlation passes; the next correlated reply150ms after issuance fails at receiver line163, fixture late client proof. Shipping's250ms floor admits it. Red0/1; same finding, no implementation repair. This establishes committed workload drift, not the source identity of externally staged final measurement drivers. P2 fixes forward under the owner rule. Passing independent controls:36 browser connection tests/95 assertions;14 shipping-source native virtual-clock tests (real-time socket case excluded);10 existing Python fixture tests; artifact and review whitespace checks. No new dependency or shipping source. No independent performance/size/restart/signed-handoff/physical-iPhone/original-tab acceptance claim. The author's zero CPU-change limitation remains unchanged. No push, deploy, perf VM or other-job process changes. Gate blocker: scripts/verify fast was invoked through required build-slot, with serial tests, short canonical scratch and TRACER_TEST_PORT4399. It never acquired a slot during this review; verify-fast.log stayed0bytes and the target directory was never created. Stopped only this review's queued waiter after the prolonged wait (verified child was sleep). There is no independent gate output or exit status to quote; it is neither PASS nor a test failure. Orchestrator must make a host build slot available and rerun the required gate to complete verification. Evidence and the exact invocation are retained below/in the local report. No other process was signalled and no slot was bypassed. Author-reported scripts/verify fast summary ON TARGET c87fc0631, quoted verbatim from the provided job result.md; this is an attributed claim, not an independent run: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ``` Raw review receipts attached above; strengthened red receipt: https://git.kayg.org/attachments/cf7a66b9-5af4-4058-ada4-26f43b5c0c10 . Local evidence: ~/.cache/codex-scratch/idle-cpu-199-review/evidence/. Findings: ~/.local/state/codex-jobs/infraternal/review-idle-cpu-199/findings.md. All review build output was removed; no review test server or native fixture remains. Issue stays open.
kayg commented 2026-10-15 10:47:22 +00:00
Author
Owner
Copy link

Completed x86_64 release on unchanged036bc2bb0: /home/kayg/.cache/codex-scratch/switchover-release/artifacts/infraternal-x86_64-musl
SHA256 29d659256cf62d0e4b0c743797e44b19f4b85b3908553cd9141ef03a642acc24
Size 22329104B; stripped static PIE, no staging authority (check-production-trust PASS). Release marker1; size matches c87fc0631 recorded22329104B, delta0 versus input source. Hash differs from the prior development-marked gate image as expected; this is the retained production-marked image. ARM compilation started in the same target folder. Staging directory correction before execution: the unchanged test enforces /root/infraternal-bench/deploy; use a unique sr<8hex> child there (longest physical session socket102B), preserving its guard. Lock-owner/1800s operational controls remain. No source changes or deployment.

Completed x86_64 release on unchanged036bc2bb0: /home/kayg/.cache/codex-scratch/switchover-release/artifacts/infraternal-x86_64-musl SHA256 29d659256cf62d0e4b0c743797e44b19f4b85b3908553cd9141ef03a642acc24 Size 22329104B; stripped static PIE, no staging authority (check-production-trust PASS). Release marker1; size matches c87fc0631 recorded22329104B, delta0 versus input source. Hash differs from the prior development-marked gate image as expected; this is the retained production-marked image. ARM compilation started in the same target folder. Staging directory correction before execution: the unchanged test enforces /root/infraternal-bench/deploy; use a unique sr<8hex> child there (longest physical session socket102B), preserving its guard. Lock-owner/1800s operational controls remain. No source changes or deployment.
kayg commented 2026-10-15 11:04:33 +00:00
Author
Owner
Copy link

Completed release artifacts at unchanged036bc2bb067a231736f219f32897aaa7d8942b70 (no source commits/deploy):

  • /home/kayg/.cache/codex-scratch/switchover-release/artifacts/infraternal-x86_64-musl — SHA256 29d659256cf62d0e4b0c743797e44b19f4b85b3908553cd9141ef03a642acc24; 22329104B
  • /home/kayg/.cache/codex-scratch/switchover-release/artifacts/infraternal-aarch64-musl — SHA256 e120c1adedabdc7da69e9ee4cb78daabdca058eae19631deaca6bba1a5668f27; 19752920B
  • /home/kayg/.cache/codex-scratch/switchover-release/artifacts/web-dist.tar — SHA256 99414fea1dea409d500ca33936cf8deb510d4a0426a6b9eb4b6e7f9a7df8c7dc; 3543040B
    Web dist: /home/kayg/.cache/codex-scratch/switchover-release/artifacts/web (253files,3333781B); per-file SHA manifest digest 455ba0fd45b8d832ba70a4b2db4228370ce4e0ab8f8eff6d447bc8b687866e60. web-dist.tar is a canonical sorted,mtime0,owner0 archive of exactly that dist.
    Both native images are stripped static PIE/no PT_INTERP and production-trust PASS; release marker1. Sizes match input c87fc0631, delta0. Inherited caps would be exceeded x86+124800B/ARM+114176B; no thresholds changed.
    Production web transfer PASS: critical gzip296285/307200B, Brotli246354/256000B; deferred47848/51200B and42485/45056B; largest on-demand15949/20480B and14185/18432B. rustls0.23.46 verified.
    Staging/browser/gate remain pending. Preview fixture preparation is queued for the next host slot (single queued command); retained production bytes are frozen separately.
    artifacts.json
    release-build.log
    toolchain.json
Completed release artifacts at unchanged036bc2bb067a231736f219f32897aaa7d8942b70 (no source commits/deploy): - /home/kayg/.cache/codex-scratch/switchover-release/artifacts/infraternal-x86_64-musl — SHA256 29d659256cf62d0e4b0c743797e44b19f4b85b3908553cd9141ef03a642acc24; 22329104B - /home/kayg/.cache/codex-scratch/switchover-release/artifacts/infraternal-aarch64-musl — SHA256 e120c1adedabdc7da69e9ee4cb78daabdca058eae19631deaca6bba1a5668f27; 19752920B - /home/kayg/.cache/codex-scratch/switchover-release/artifacts/web-dist.tar — SHA256 99414fea1dea409d500ca33936cf8deb510d4a0426a6b9eb4b6e7f9a7df8c7dc; 3543040B Web dist: /home/kayg/.cache/codex-scratch/switchover-release/artifacts/web (253files,3333781B); per-file SHA manifest digest 455ba0fd45b8d832ba70a4b2db4228370ce4e0ab8f8eff6d447bc8b687866e60. web-dist.tar is a canonical sorted,mtime0,owner0 archive of exactly that dist. Both native images are stripped static PIE/no PT_INTERP and production-trust PASS; release marker1. Sizes match input c87fc0631, delta0. Inherited caps would be exceeded x86+124800B/ARM+114176B; no thresholds changed. Production web transfer PASS: critical gzip296285/307200B, Brotli246354/256000B; deferred47848/51200B and42485/45056B; largest on-demand15949/20480B and14185/18432B. rustls0.23.46 verified. Staging/browser/gate remain pending. Preview fixture preparation is queued for the next host slot (single queued command); retained production bytes are frozen separately. [artifacts.json](https://git.kayg.org/attachments/bdd76825-ad23-4bca-8bb7-c461a9785183) [release-build.log](https://git.kayg.org/attachments/d643c842-6036-4237-a083-5b28e3d2c5b4) [toolchain.json](https://git.kayg.org/attachments/a3e17679-a282-44ab-96fd-3da762282f0f)
kayg commented 2026-10-15 11:06:49 +00:00
Author
Owner
Copy link

Retained production x86 artifact29d65925 passes prlimit --memlock=8388608 __key-memory-check: initialization=1, locked pages and dump exclusion enabled. ARM file confirms aarch64, statically linked, stripped; no native ARM runtime claim on this x86 host. Correction to preceding generic wording: x86 is static PIE, ARM is a statically linked ELF (not claimed PIE). Both have no PT_INTERP and no staging authority.
Remaining sequential validation is queued through one build-slot command; it will keep staging/native fast/TS in that acquired slot and run browser groups under separate short VM locks. No source changes or deployment.

Retained production x86 artifact29d65925 passes prlimit --memlock=8388608 __key-memory-check: initialization=1, locked pages and dump exclusion enabled. ARM file confirms aarch64, statically linked, stripped; no native ARM runtime claim on this x86 host. Correction to preceding generic wording: x86 is static PIE, ARM is a statically linked ELF (not claimed PIE). Both have no PT_INTERP and no staging authority. Remaining sequential validation is queued through one build-slot command; it will keep staging/native fast/TS in that acquired slot and run browser groups under separate short VM locks. No source changes or deployment.
kayg commented 2026-10-15 11:10:43 +00:00
Author
Owner
Copy link

Validation waiter2143590 ended by SIGTERM/exit143 before acquiring a build slot; prepare-browser-interrupted.log contains only the scope banner, no preview/staging/browser/gate code ran and validation-exits.json does not exist. Cause is not known; no new orchestration instruction appears on refreshed #199. Verified old process gone; restarted exactly one weighted build-slot waiter with the prepared sequential validation pipeline. Release artifacts remain retained and hash-checked. This is an interrupted wait, not a test result or waiver.

Validation waiter2143590 ended by SIGTERM/exit143 before acquiring a build slot; prepare-browser-interrupted.log contains only the scope banner, no preview/staging/browser/gate code ran and validation-exits.json does not exist. Cause is not known; no new orchestration instruction appears on refreshed #199. Verified old process gone; restarted exactly one weighted build-slot waiter with the prepared sequential validation pipeline. Release artifacts remain retained and hash-checked. This is an interrupted wait, not a test result or waiver.
kayg commented 2026-10-15 11:16:47 +00:00
Author
Owner
Copy link

Completed browser acceptance on production-marked x86 artifact29d659256cf62d0e4b0c743797e44b19f4b85b3908553cd9141ef03a642acc24 and the frozen production dist:12/12 real systemd KillMode=process/PTY restart cases PASS (2 ordinary,4 retained-route/stalled-upstream,6 three-window identity/queued-input/sibling isolation);5/5 mock cases PASS (both/up/down route,ring snapshot,same-writer retry). All groups ran on locked perf VM with owner receipts, TRACER_TEST_PORT4499, no retries/assertion/deadline edits. Node22.23.3 and bundled Playwright1.61.1/Chromium1228.
Initial attempts failed before test execution because bun x tried network package resolution (DNS failure); retained as browser-*-initial.log. Invoke the exact bundled CLI directly to avoid network resolution; clean rerun groups all exit0. No shipping/test-source changes. Separate ordinary-restart capture repetition is running solely for production desktop/phone screenshots.
Signed derived staging image compiled: SHA2562b73c917cdf55e7d4f2b724b701e401bd6752b697e177b85da69fbd0c56bc0f3,22333200B. It includes required per-run staging authority and is not the production artifact. Native staging harness compilation and final fast/TS gates remain pending. No deploy.
browser-restart.log
browser-liveness.log
browser-three-window.log
browser-mock.log
browser-results.json
browser-restart-initial.log

Completed browser acceptance on production-marked x86 artifact29d659256cf62d0e4b0c743797e44b19f4b85b3908553cd9141ef03a642acc24 and the frozen production dist:12/12 real systemd KillMode=process/PTY restart cases PASS (2 ordinary,4 retained-route/stalled-upstream,6 three-window identity/queued-input/sibling isolation);5/5 mock cases PASS (both/up/down route,ring snapshot,same-writer retry). All groups ran on locked perf VM with owner receipts, TRACER_TEST_PORT4499, no retries/assertion/deadline edits. Node22.23.3 and bundled Playwright1.61.1/Chromium1228. Initial attempts failed before test execution because bun x tried network package resolution (DNS failure); retained as browser-*-initial.log. Invoke the exact bundled CLI directly to avoid network resolution; clean rerun groups all exit0. No shipping/test-source changes. Separate ordinary-restart capture repetition is running solely for production desktop/phone screenshots. Signed derived staging image compiled: SHA2562b73c917cdf55e7d4f2b724b701e401bd6752b697e177b85da69fbd0c56bc0f3,22333200B. It includes required per-run staging authority and is not the production artifact. Native staging harness compilation and final fast/TS gates remain pending. No deploy. [browser-restart.log](https://git.kayg.org/attachments/3b217cb4-2e93-41e9-82db-205bfe6fc7af) [browser-liveness.log](https://git.kayg.org/attachments/b9fe0638-ccda-4e56-afad-0634dbe5701a) [browser-three-window.log](https://git.kayg.org/attachments/1ec35763-a04a-48f3-9970-909d44988781) [browser-mock.log](https://git.kayg.org/attachments/a6408fcc-23d0-4c75-8654-5ac2418e95dd) [browser-results.json](https://git.kayg.org/attachments/531f51ed-d3a7-493f-874a-f43ee326dd41) [browser-restart-initial.log](https://git.kayg.org/attachments/e364b7a0-bf76-4b2a-99f1-8ba154d7ae48)
kayg commented 2026-10-15 11:20:29 +00:00
Author
Owner
Copy link

Production capture repetition also PASS2/2, unchanged ordinary restart cases with existing RESTART_CAPTURE_DIR option. Desktop1440x900 and phone393x852 catching-up/ready screenshots attached for owner review; no visual-quality judgment. All12 main real cases and5 mocks already passed in clean grouped runs.
Browser VM cleanup complete:24 recorded transient unit launches inactive/MainPID0; no remaining processes by private cwd or executable inode; one residual rendezvous socket collected through the repository fixture cleanup helper; private browser directory absent. Calibration/staging work is separate and remains owned. Three initial real groups had DNS resolution failures before tests; the initial mock runner also failed before any test result (cause not independently diagnosed). Direct bundled CLI clean rerun passes all groups.
restart-locked-ready.png
restart-unlock-ready.png
restart-locked-catching-up.png
restart-unlock-catching-up.png
browser-production-captures.log
browser-final-cleanup.txt

Production capture repetition also PASS2/2, unchanged ordinary restart cases with existing RESTART_CAPTURE_DIR option. Desktop1440x900 and phone393x852 catching-up/ready screenshots attached for owner review; no visual-quality judgment. All12 main real cases and5 mocks already passed in clean grouped runs. Browser VM cleanup complete:24 recorded transient unit launches inactive/MainPID0; no remaining processes by private cwd or executable inode; one residual rendezvous socket collected through the repository fixture cleanup helper; private browser directory absent. Calibration/staging work is separate and remains owned. Three initial real groups had DNS resolution failures before tests; the initial mock runner also failed before any test result (cause not independently diagnosed). Direct bundled CLI clean rerun passes all groups. ![restart-locked-ready.png](https://git.kayg.org/attachments/9df67011-87fe-40bc-90ff-4f2fe179c928) ![restart-unlock-ready.png](https://git.kayg.org/attachments/488a7982-131e-4fcd-b39e-e539b3ab9ca9) ![restart-locked-catching-up.png](https://git.kayg.org/attachments/2b84ac2d-ba64-4129-a41b-59587d192797) ![restart-unlock-catching-up.png](https://git.kayg.org/attachments/37cd968d-542c-4d73-a58d-31c279a9adde) [browser-production-captures.log](https://git.kayg.org/attachments/c4d501fc-1745-4dfe-8f3f-7cf5fd3fd074) [browser-final-cleanup.txt](https://git.kayg.org/attachments/9d7a27cc-4d96-4b25-91c5-f1fd3bdaa6eb)
kayg commented 2026-10-15 11:58:00 +00:00
Author
Owner
Copy link

Signed staging acceptance PASS at unchanged036bc2bb0, both ignored shipping tests1/0 each.12 sampled signed handoffs preserve128/128 real shim PIDs and128/128 input echoes; one real passkey unlock, CA remains unlocked; all6 retained sockets answer fresh control after5s idle; local polling control6/6 responsive, unattended negative control closes. Signed unusable-image refusal, automatic warm rollback and backup integrity with10000 nodes/50 users PASS. VM directory absent and measurement lock released.
Derived staging image SHA2b73c917cdf55e7d4f2b724b701e401bd6752b697e177b85da69fbd0c56bc0f3,22333200B. Standard script uses per-run trust plus fixture web/ARM placeholder; production artifact hashes remain separate. Original staging-proof/staging_remote.py source hashes attached.
VM receipts (listener gaps are conservative brackets, not exact outage durations;5s CPU/RSS is a point sample, no new regression or zero-change claim):

STAGING load_before=0.65 0.65 0.60
METRIC name=signed_deploy_idle server_cpu_percent_one_core=12.774 server_rss_bytes=21610496 real_pty_sessions=128 browser_connections=6 sample_seconds=5
STAGING retained_browser_sockets_responsive=6
METRIC name=signed_deploy_handoff n=12 listener_gap_bracket_p50_ms=49.682 listener_gap_bracket_p99_ms=105.810 probe_max_interval_ms=38.838 users=50 nodes=10000 real_pty_sessions=128 surviving_shim_pids=128 resumed_input_output=128 unlock_ceremonies=1 ca_unlocked=true
STAGING local_clients_responsive=6
STAGING load_after=5.87 1.87 1.00
STAGING measurement_lock_released=true
STAGING teardown_directory_absent=true

Full TypeScript gate PASS separately (37 protocol cases/34463 assertions plus full canvas/build suite). First fast attempt stopped before Rust on my generated apps/tracer/mock-dist precompressed files; retained transcript. Moved that temporary build outside checkout. Final full fast rerun has acquired a slot and passed the repository artifact row; no source, assertion or cap changes. No deploy.
staging-proof.txt
staging-command.log
staging-script-sha256.txt
verify-typescript.log
verify-fast-generated-preview.log

Signed staging acceptance PASS at unchanged036bc2bb0, both ignored shipping tests1/0 each.12 sampled signed handoffs preserve128/128 real shim PIDs and128/128 input echoes; one real passkey unlock, CA remains unlocked; all6 retained sockets answer fresh control after5s idle; local polling control6/6 responsive, unattended negative control closes. Signed unusable-image refusal, automatic warm rollback and backup integrity with10000 nodes/50 users PASS. VM directory absent and measurement lock released. Derived staging image SHA2b73c917cdf55e7d4f2b724b701e401bd6752b697e177b85da69fbd0c56bc0f3,22333200B. Standard script uses per-run trust plus fixture web/ARM placeholder; production artifact hashes remain separate. Original staging-proof/staging_remote.py source hashes attached. VM receipts (listener gaps are conservative brackets, not exact outage durations;5s CPU/RSS is a point sample, no new regression or zero-change claim): ```text STAGING load_before=0.65 0.65 0.60 METRIC name=signed_deploy_idle server_cpu_percent_one_core=12.774 server_rss_bytes=21610496 real_pty_sessions=128 browser_connections=6 sample_seconds=5 STAGING retained_browser_sockets_responsive=6 METRIC name=signed_deploy_handoff n=12 listener_gap_bracket_p50_ms=49.682 listener_gap_bracket_p99_ms=105.810 probe_max_interval_ms=38.838 users=50 nodes=10000 real_pty_sessions=128 surviving_shim_pids=128 resumed_input_output=128 unlock_ceremonies=1 ca_unlocked=true STAGING local_clients_responsive=6 STAGING load_after=5.87 1.87 1.00 STAGING measurement_lock_released=true STAGING teardown_directory_absent=true ``` Full TypeScript gate PASS separately (37 protocol cases/34463 assertions plus full canvas/build suite). First fast attempt stopped before Rust on my generated apps/tracer/mock-dist precompressed files; retained transcript. Moved that temporary build outside checkout. Final full fast rerun has acquired a slot and passed the repository artifact row; no source, assertion or cap changes. No deploy. [staging-proof.txt](https://git.kayg.org/attachments/4c5558bd-b7e4-4338-be81-b91339adb9eb) [staging-command.log](https://git.kayg.org/attachments/35994179-10bf-40c2-afc6-cfbd356d177d) [staging-script-sha256.txt](https://git.kayg.org/attachments/9eb0421d-36f1-47f1-935c-5dde440225b1) [verify-typescript.log](https://git.kayg.org/attachments/a9c44486-62dd-4154-bf76-09917367b176) [verify-fast-generated-preview.log](https://git.kayg.org/attachments/6f4e98e7-46ef-40ca-bf01-5ec0d70fcb73)
kayg commented 2026-10-15 12:34:08 +00:00
Author
Owner
Copy link

Exact-head036bc2bb0 fast milestone: fmt,VT fmt,25 CLI compile-out tests,workspace Clippy,1054 workspace Rust tests (0failed/139configured ignores),and cargo-deny PASS (advisories/bans/licenses/sources all ok). Static builds/linkage/key-memory/size rows are still running; this is not the final gate result.
Size comparison remeasured read-only from the retained prior same-recipe merge-base artifact:22277392B,SHA380c9ab1c62f1a6bb94d8b33f7900eab773d22c475966b3f7c84c1c4b2f462d0. Current merge base67010de9199e has identical native build inputs to recorded1b23ef31 (Cargo manifest/lock,crates,.cargo,toolchain,auth-toolchain,build-vt). Production x8622329104B gives inherited+51712B; this job adds0native source change. The aggregate is the previously accepted #199 liveness/proof work,below64KiB;existing absolute caps remain unchanged. No new baseline compilation is claimed.
Operational adapter source is attached as text documentation (script extensions were rejected by the attachment service); per-file hashes included. Test assertions and repository staging sources unchanged.
size-comparison.json
operational-adapters.txt

Exact-head036bc2bb0 fast milestone: fmt,VT fmt,25 CLI compile-out tests,workspace Clippy,1054 workspace Rust tests (0failed/139configured ignores),and cargo-deny PASS (advisories/bans/licenses/sources all ok). Static builds/linkage/key-memory/size rows are still running; this is not the final gate result. Size comparison remeasured read-only from the retained prior same-recipe merge-base artifact:22277392B,SHA380c9ab1c62f1a6bb94d8b33f7900eab773d22c475966b3f7c84c1c4b2f462d0. Current merge base67010de9199e has identical native build inputs to recorded1b23ef31 (Cargo manifest/lock,crates,.cargo,toolchain,auth-toolchain,build-vt). Production x8622329104B gives inherited+51712B; this job adds0native source change. The aggregate is the previously accepted #199 liveness/proof work,below64KiB;existing absolute caps remain unchanged. No new baseline compilation is claimed. Operational adapter source is attached as text documentation (script extensions were rejected by the attachment service); per-file hashes included. Test assertions and repository staging sources unchanged. [size-comparison.json](https://git.kayg.org/attachments/96a35bf4-3e61-419a-84ae-4ab5b382f698) [operational-adapters.txt](https://git.kayg.org/attachments/f4d3026d-a530-408b-97ff-3816e646f8c4)
kayg commented 2026-10-15 12:47:07 +00:00
Author
Owner
Copy link

Finish switchover-release: branch switchover-release, head 036bc2bb06; clean worktree. No source changes/commits, push, production access or deployment. Artifacts remain available after build-target cleanup.

  • /home/kayg/.cache/codex-scratch/switchover-release/artifacts/infraternal-x86_64-musl
    SHA256 29d659256cf62d0e4b0c743797e44b19f4b85b3908553cd9141ef03a642acc24; 22329104 bytes
  • /home/kayg/.cache/codex-scratch/switchover-release/artifacts/infraternal-aarch64-musl
    SHA256 e120c1adedabdc7da69e9ee4cb78daabdca058eae19631deaca6bba1a5668f27; 19752920 bytes
  • /home/kayg/.cache/codex-scratch/switchover-release/artifacts/web-dist.tar
    SHA256 99414fea1dea409d500ca33936cf8deb510d4a0426a6b9eb4b6e7f9a7df8c7dc; 3543040 bytes
    Web dist: /home/kayg/.cache/codex-scratch/switchover-release/artifacts/web;253 files,3333781B. Per-file manifest /home/kayg/.cache/codex-scratch/switchover-release/artifacts/web-SHA256SUMS; manifest SHA256 455ba0fd45b8d832ba70a4b2db4228370ce4e0ab8f8eff6d447bc8b687866e60. The web-dist.tar checksum above covers a sorted,mtime0,owner0 archive of this exact dist.

Acceptance PASS: signed staging both tests pass;12 sampled handoffs preserve128/128 shim PIDs and128/128 real input echoes;one real passkey unlock;CA stays unlocked;6/6 retained sockets responsive after idle;6/6 polling controls responsive;unattended negative control closes;bad-image refusal,warm rollback,and backup integrity at10000 nodes/50 users pass. Staging uses the standard per-run temporary-trust x86 fixture, fixture web and unused ARM placeholder; production ARM receives build/link/trust checks, production web is verified separately by browser tests.
Browser PASS:12/12 real systemd/PTY restart cases and5/5 mocks in clean grouped runs;production screenshot repetition2/2 also passes. Pinned bundled Playwright1.61.1/Chromium1228 and Node22.23.3;unique ports and short VM locks. Initial package-resolution attempts and the generated-preview artifact-check failure are retained on this issue; operational fixes only,no assertion/deadline/source changes.

Final scripts/verify fast on this head exits1 ONLY for inherited absolute size caps and their same size-boundary assertion. Workspace Rust1054passed/0failed/139configured ignores;CLI compile-out25/0;fmt,VTfmt,Clippy,cargo-deny PASS. Both static builds/no PT_INTERP/production trust pass;native x86 key-memory initialization1 under8MiB memlock. ARM runtime check skipped on x86 as designed. Separate full TypeScript gate PASS,37 protocol cases/34463 assertions plus full canvas/build suite;fast stops before TS on Rust size failure. rustls0.23.46,checksum-pinned Zig0.15.2,Rust1.99.0 verified.

Verbatim final gate summary and size diagnostics:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

PT_INTERP: absent; size: 22329104 bytes; cap: 22204304 bytes; delta: +124800 bytes
PT_INTERP: absent; size: 19752920 bytes; cap: 19638744 bytes; delta: +114176 bytes

Footprint:production x86 size matches c87fc0631 (+0 for this release job). Remeasured retained same-recipe merge-base image22277392B versus22329104B gives inherited+51712B;actual merge base67010de9199e has native build inputs identical to recorded1b23ef31. The accepted #199 liveness/proof work owns that aggregate growth;no cap edits or fresh baseline compilation claimed.
Release checklist complete:exact source/toolchain/locked inputs;production trust isolated from staging;production hashes rechecked;no dependency/profile/schema/library/cap changes;real-engine and browser receipts attached;no author visual-quality verdict. Cleanup proves recorded VM units inactive,owned processes absent,staging/browser/calibration directories absent,local target/private runtime/tests removed. Generated native-probe receipts moved outside checkout. Retained artifacts,evidence,and small operational adapters only. Issue stays open;orchestrator owns deployment.

verify-fast.log
final-gate-summary.txt
final-validation.json
local-final-cleanup.json
vm-final-cleanup.json
production-key-memory.txt
artifacts.json
release-build.log
toolchain.json
size-comparison.json
operational-adapters.txt

Finish switchover-release: branch switchover-release, head 036bc2bb067a231736f219f32897aaa7d8942b70; clean worktree. No source changes/commits, push, production access or deployment. Artifacts remain available after build-target cleanup. - /home/kayg/.cache/codex-scratch/switchover-release/artifacts/infraternal-x86_64-musl SHA256 29d659256cf62d0e4b0c743797e44b19f4b85b3908553cd9141ef03a642acc24; 22329104 bytes - /home/kayg/.cache/codex-scratch/switchover-release/artifacts/infraternal-aarch64-musl SHA256 e120c1adedabdc7da69e9ee4cb78daabdca058eae19631deaca6bba1a5668f27; 19752920 bytes - /home/kayg/.cache/codex-scratch/switchover-release/artifacts/web-dist.tar SHA256 99414fea1dea409d500ca33936cf8deb510d4a0426a6b9eb4b6e7f9a7df8c7dc; 3543040 bytes Web dist: /home/kayg/.cache/codex-scratch/switchover-release/artifacts/web;253 files,3333781B. Per-file manifest /home/kayg/.cache/codex-scratch/switchover-release/artifacts/web-SHA256SUMS; manifest SHA256 455ba0fd45b8d832ba70a4b2db4228370ce4e0ab8f8eff6d447bc8b687866e60. The web-dist.tar checksum above covers a sorted,mtime0,owner0 archive of this exact dist. Acceptance PASS: signed staging both tests pass;12 sampled handoffs preserve128/128 shim PIDs and128/128 real input echoes;one real passkey unlock;CA stays unlocked;6/6 retained sockets responsive after idle;6/6 polling controls responsive;unattended negative control closes;bad-image refusal,warm rollback,and backup integrity at10000 nodes/50 users pass. Staging uses the standard per-run temporary-trust x86 fixture, fixture web and unused ARM placeholder; production ARM receives build/link/trust checks, production web is verified separately by browser tests. Browser PASS:12/12 real systemd/PTY restart cases and5/5 mocks in clean grouped runs;production screenshot repetition2/2 also passes. Pinned bundled Playwright1.61.1/Chromium1228 and Node22.23.3;unique ports and short VM locks. Initial package-resolution attempts and the generated-preview artifact-check failure are retained on this issue; operational fixes only,no assertion/deadline/source changes. Final scripts/verify fast on this head exits1 ONLY for inherited absolute size caps and their same size-boundary assertion. Workspace Rust1054passed/0failed/139configured ignores;CLI compile-out25/0;fmt,VTfmt,Clippy,cargo-deny PASS. Both static builds/no PT_INTERP/production trust pass;native x86 key-memory initialization1 under8MiB memlock. ARM runtime check skipped on x86 as designed. Separate full TypeScript gate PASS,37 protocol cases/34463 assertions plus full canvas/build suite;fast stops before TS on Rust size failure. rustls0.23.46,checksum-pinned Zig0.15.2,Rust1.99.0 verified. Verbatim final gate summary and size diagnostics: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ PT_INTERP: absent; size: 22329104 bytes; cap: 22204304 bytes; delta: +124800 bytes PT_INTERP: absent; size: 19752920 bytes; cap: 19638744 bytes; delta: +114176 bytes ``` Footprint:production x86 size matches c87fc0631 (+0 for this release job). Remeasured retained same-recipe merge-base image22277392B versus22329104B gives inherited+51712B;actual merge base67010de9199e has native build inputs identical to recorded1b23ef31. The accepted #199 liveness/proof work owns that aggregate growth;no cap edits or fresh baseline compilation claimed. Release checklist complete:exact source/toolchain/locked inputs;production trust isolated from staging;production hashes rechecked;no dependency/profile/schema/library/cap changes;real-engine and browser receipts attached;no author visual-quality verdict. Cleanup proves recorded VM units inactive,owned processes absent,staging/browser/calibration directories absent,local target/private runtime/tests removed. Generated native-probe receipts moved outside checkout. Retained artifacts,evidence,and small operational adapters only. Issue stays open;orchestrator owns deployment. [verify-fast.log](https://git.kayg.org/attachments/6ef6c151-f23c-4f6e-8718-6da88566b3f6) [final-gate-summary.txt](https://git.kayg.org/attachments/5a960f62-eea8-4fcc-9ebc-66cd80016aea) [final-validation.json](https://git.kayg.org/attachments/4fd111ec-cf48-4245-8807-9ede38e6a356) [local-final-cleanup.json](https://git.kayg.org/attachments/e027d7c0-bff6-4e75-8141-0c009c726908) [vm-final-cleanup.json](https://git.kayg.org/attachments/d18bfae2-3013-4111-b2bb-8ddbfbc9e572) [production-key-memory.txt](https://git.kayg.org/attachments/f5c36de9-e0c5-4d5a-b69a-7d146a8c05b0) [artifacts.json](https://git.kayg.org/attachments/bdd76825-ad23-4bca-8bb7-c461a9785183) [release-build.log](https://git.kayg.org/attachments/d643c842-6036-4237-a083-5b28e3d2c5b4) [toolchain.json](https://git.kayg.org/attachments/a3e17679-a282-44ab-96fd-3da762282f0f) [size-comparison.json](https://git.kayg.org/attachments/96a35bf4-3e61-419a-84ae-4ab5b382f698) [operational-adapters.txt](https://git.kayg.org/attachments/f4d3026d-a530-408b-97ff-3816e646f8c4)
kayg commented 2026-10-28 17:53:15 +00:00
Author
Owner
Copy link

Closed by the orchestrator during the 2026-10-28 handover (#254), on the owner's request to close finished issues. Evidence: Switch-over merged (fa904fdaa) and deployed (036bc2bb0); both are in dev and release-prod-live. Reopen if anything is still missing.

Closed by the orchestrator during the 2026-10-28 handover (#254), on the owner's request to close finished issues. Evidence: Switch-over merged (fa904fdaa) and deployed (036bc2bb0); both are in dev and release-prod-live. Reopen if anything is still missing.
kayg closed this issue 2026-10-28 17:53:15 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
j470b-server-mesh-rejoin
mesh-compat-382
j444b-d381-key-wins-flood
j461b-macos-utun-reader-thread
j475b-remove-host-motion
fix-reconnect-new-link
j481-d380
dev
j479-mesh-soak
fix/j476-host-facts
j475-motion-pill-split
j470-mac-node-join
root-upgrade-d376
j474-review-merge
latency-targets-120hz
nightly-suite-shards
gotatun-pin-j469
macos-utun-reader-thread
fix-flood-echo-stall
wip/j416
motion-fast-ease-out
mesh-addhost-j259
takeover-close-terminals-d368
kbd-instant-followups
fix-trust-key-double-click
nightly-suite-partial-report
appwin-icons-j207
rusqlite-slice1
swipe-content-first-d374
mesh-mtu-381
addhost-instant-open
glass-contrast-d365-mock
glass-contrast-d365
signout-followups-442
fix-startup-toast-red
research-gotatun-matrix-j317
themes-sync-j304
fix-load-flakes
fix-nix-zig-patch
fix-compute01-browser-baseline
ratchet-size-baseline-419
predict-auto-d130
revoke-coverage-d367
nightly-suite-queue-timeout
nightly-browser-suite
fix-rustsec-2026-0459
fonts-ioskeley-obel
log-retention-d375
fix-kbd-no-anim-427
fix-dev-red-specs-2
takeover-proof-305-j205
wip/j410-server-pump
ui-consistency-d372
fix-osc52-clipboard
libghostty-vt-1.0
forget-retired-host-d369
j404-merge-426
fix-setup-lock-race
fix-touch-select-426
revoke-access-report-ui
fix-predictive-echo-specs
fix-selinux-agent-exec
wip/j395-checkpoint2
wip/j395
j391-review-scratch
ui-consistency-j195-deferred-rebased
fix-rustc-paths-421
decisions-wording-341
fix-install-suite-red
fix-dev-rtt-specs
fix-pending-removal-align
fix-privileged-skip
fix-install-suite-404
review-j368
mesh-dns-j254
revoke-access-report-d356b
revoke-privilege-round5
fix-removal-confirm
review-j360
gotatun-rotation-367
deps-refresh-340
review-j354
review-J351
review-j350
addhost-takeover-d359
revoke-report-items-347
revoke-end-sessions-d362
review-j348
size-mesh-ssh-346
size-trim-337
revoke-access-diff-d356
j259-pre-rebase-b06c5a6
j305-ended-at-start-spec
addhost-sudo-consent
fix-layout-real-specs-411
fix-shell-open-411
fix-console-hosts-373
fix-shell-exit
fuzz-host-name-353
fix-host-details-410
macos-filter-j197
research-gotatun-tricks
revoke-p3-j216l
stale-state-pill
revoke-followups-j216k
revoke-progress-j216e
backup/macos01-20261106/lume/j197
backup/macos01-20261106/lume/appwin-icons-j207-stash0
backup/macos01-20261106/claude/stash-0
backup/macos01-20261106/claude/stash-1
backup/macos01-20261106/claude/J80-macos-s1
backup/macos01-20261106/claude/J80
backup/macos01-20261106/claude/J80-slice1-20261102
backup/macos01-20261106/claude/J16b-appwin-on-dev
backup/macos01-20261106/claude/appwin-launch-idle-146
backup/macos01-20261106/claude/J80-s1-run
wip/j259
trust-285-p3c
runtime-lock-j216f
mesh-ssh-p3-j175
tun-proof-p3b
trust-285-p3
terminal-polish-j271
tun-proof-p3
ssh-install-flakes-j216i
fuzz-fixes-j264
trust-285-a
trust-285-b
restart-oversized-fixture-j198
backup/macos01-20261106/lume/appwin-icons-j207
restart-bound-j216h
up-preflight-p3-j198
backup/macos01-20261106/lume/j251
up-preflight-j198
wip/up-preflight-j198-j245-p2
wip/up-preflight-j198-j245-p1
agent-fsync-offloop-j196
wip/up-preflight-j198-j245
settings-offscreen-j231
wip/up-preflight-j198-j226-inspection
wip/j103f
themes-j206
wip/up-preflight-j198-j226
flakes-j216c
backup/macos01-20261106/claude/j197
flakes-j216b
backup/macos01-20261106/claude/j197-red
wip/up-preflight-j198-review
flakes-j216
backup/macos01-20261106/claude/appwin-icons-j207
engineering-j208b
appwin-mock-j209
toast-stack-j165
engineering-j208
decisions-d351
decisions-d350
decisions-d349
appearance-presets-mock-j129
decisions-d348
docs-audit-spec
doc-coverage-baseline-raise
decisions-d347
decisions-d346b
decisions-d346
j88h-profile
ui-consistency-j195-deferred
ui-consistency-j195
engineering-practices-j194
preferred-login-ui-j166
signin-pools-263-j174
macos-node-265-s1
decisions-d345
decisions-d344
proxy-cgroup-j164
verify-quick-fuzz-329
proxy-start-io-j168
timing-flakes-j163
up-263-s1
mcp-events-j147
verify-quick-fuzz-j158
proxy-p3-j159
agent-resource-j157-detect
wip/j103d-final
remote-cancel-hang-j155
proxy-traefik-281-j62d
j149-verify-quick
proxy-traefik-281-j62b
j150-drop-unnamed
wip/j79d-red-r4
wip/j103d-d332
rootless-toast-j142
wip/j103d-head
wip/j103d
j125c-mcp
macos-build-j136
backup/macos01-20261106/claude/j136
wip/j79d-red-cli
wip/j79d-red-server
j121-link-liveness
j118c-tracer-failures
wip/j79c-head
release-desktop-j131
j124c-signed-macos
wip/j79c-red-p3
wip/j79c-red-p23
wip/j79c-red-p22
merge-j128-check
perf-deferred-j128
sidebar-toggle-260
j127-shortcut-changed-dot
j125-mcp
j118-tracer-failures
j126-account-summary
settings-sections-mock-j92
j120-pinch-lock
sidebar-292
backup/macos01-20261106/claude/j120
taste-2026-11-03
row-actions-242-215
backup/macos01-20261106/claude/stash-0-untracked
backup/macos01-20261106/claude/J80-265-s1-run
backup/macos01-20261106/claude/J80-macos-s1-20261102
j112-canvas-commands
ui-no-overshoot-scroll
fix-recovery-restart
backup/macos01-20261106/claude/stash-1-untracked
taste-2026-11-02
size-baseline-ratchet
machine-cards-structure-j108
j109-boot-copy-mockups
fix-dev-red
wip/up-263-s1
j95b-settings-rows
proxy-traefik-281
j95-option-rows
docs-human-j91
docs-keys-palette
fix-flood-1013-270-j38d
mesh-ssh-s1
mesh-ssh-s0
addhost-d291-tunnel
test-stability-load
mesh-relay-220
repro-296
shim-thp-off-24
mesh-pins-258-v2
mesh-dns
signin-d311-merged
docs-pages
ui-size-283
fix-ui-batch-specs
wip/j12b-edge-update
gate-on-arm
fix-rename-sync-293
trust-285-j60b
fix-focus-ring-clip
integ-ui-1101
edge-update-v2
signin-d311
taste-2026-11-01
layout-per-tab-250
ui-approved-splitopt-224
ui-approved-splitopt-224-on-dev
icon-text-centre-282
ui-approved-189-212-216
research-reverse-proxy
transfer-budget-320
review-mesh-pins-258-r2
sidebar-pill-259
review-mesh-relay-220
wip/j12-edge-update-v2
nix-package
status-crash-loop-237
fix-dev-test-failures
fix-fixture-socket-path
wip/mesh-relay-220-pre-dev-advance-rebase
fix-canvas-side-clip
research-head-to-head
frosted-real-256
review-mesh-pins-258
wip/mesh-relay-220-pre-final-rebase
review-layout-per-tab-250-r2
broad-suite-triage
nix-devshell
review-status-crash-loop-237-r3
wip/nix-devshell-pre-p3
taste-frosted-real
wip/code-serve-demo-2026-10-30
wip/code-serve-glass-2026-10-30
wip/code-serve-clusters-2026-10-30
wip/code-merge-release94-dev-2026-10-30
wip/rproxy-delta-scratch-2026-10-30
wip/stale-note-257-staged-2026-10-30
wip/d241-size-staged-2026-10-30
review-broad-suite-207
wip/mesh-pins-258-pre-split
wip/mesh-relay-220-pre-rebase
mesh-measure-220
plan-211-update
review-edge-autoupdate-r3
perf24-reconnect
review-stale-note-r2
release-prod-live
d241-shell-marker
review-mesh-pins-server-258
review-revocation-r3
review-setup-token-r2
review-sync-journal-r3
wip/review-mesh-pins-258-prune-review-1995fee46
frosted-scroll-256
sync-journal-r3
revocation-per-account-wip
review-tokio-asyncfd-0386
review-sync-p3s
wip/stash-8by64-stash0-2026-10-30
mesh-drag-r2
setup-token-data-wip
release-9-4
auth-busy-1013
mesh-view-overlap
sync-p3s
review-auth-busy-wake
review-readd-r8-wip
mesh-drag-bounds
audit-auth-93
app-windows-stream-start
review-authority-cap-r3
app-windows-r7-measure
readd-minimal-r6
readd-minimal-r5
app-windows-r7-soak
readd-minimal-r4
app-windows-r7
app-windows-r6
readd-minimal-r3
readd-minimal-r2
readd-minimal
merge-release8-dev
review-busy-page-liveness-r2
review-busy-page-liveness
mesh-view-trim
review-authority-read-retry-r2
review-remove-uninstall-r2
release-retry-reattach
release-retry
review-remove-uninstall
review-authority-read-retry
app-windows-ship
mesh-autoswitch-p2
mesh-ssh
mesh-relay
mesh-macos-node
screen-and-control
session-labels
review-real-logins-r5
review-real-logins-r4
review-real-logins-r3
shim-memory-r2
review-real-logins-r2
shim-memory
shadow-addhost-instant
shadow-real-logins
shadow-same-session-flood
shadow-prod-perf-fix
hosts-from-server
app-windows-real
app-windows-transport
release-prod-6w
mesh-policy-verify
quiet-close-toast
dev-merge-r4
review-mesh-agent-transport
review-app-windows-transport
settings-roundtrip
review-release-pipeline-r4
mesh-agent-transport
ended-strip-fix
review-audit-ca-update-d244
settings-rows
phone-width
signout-clear
fix-restart-tests
dev-merge-rp
signer-hardening
review-audit-ca-update
terminal-fidelity
review-audit-ca-update-wip
mesh-integrate
review-mesh-agent-transport-515d3712
review-mesh-dataplane-feec1988
mesh-dataplane
signout
research-canvas-node
wip/review-renderer-cost
wip/review-terminal-fidelity
padlock-glyph
review-release-pipeline-2e95b689
phone-polish
sidebar-polish
inbox-generic
review-mesh-dataplane-378d4055
prod-batch
ui-inbox
ui-ca-unlock
ui-machine-sections
review-mesh-dataplane-a5f744fb
tls-session-prod
session-bounds-prod
fix-dead-session
session-bounds
tls-session
palette-polish
shortcuts-d222
research-appmeta
phone-gutter
ui-centre-button
wip/ui-centre-button
wip/tls-session
review-addhost-breakit
ui-platform
renderer-cost
remote-agents
mesh-api
ssh-bootstrap
prod-hotfix2
prod-fixes
review-remote-agents
review-mesh-dataplane
session-life
research-browser
review-mesh-api
research-capture
review-mesh-api-round3
review-mesh-api-round1
gate-stability
review-gate-stability
review-relay-direct
review-mesh-api-findings
proto-sessions
font-picker
proto-addhost
proto-windows
mesh-core
latency-gate
research-bsd
research-discovery
keytray-mods
mock-core
ui-tweaks
fix-motion-sel
fix-motion
integrate2
fix-robust
t-web
t-server
vt-ghostty
fix-snap-echo
t-web-auth
review-vt2
t-agent
lane-protocol
lane-shim
lane-auth
s1-base
s1-contracts
s1-workspace
No results found.
Labels
Clear labels   
launch-qa
Findings from launch QA.

No labels
launch-qa
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/infraternal#199
No description provided.