Gate: credit, snapshot and socket tests fail under load #118
Open
opened 2026-10-08 21:35:48 +00:00 by kayg
·
41 comments
No Branch/Tag specified
main
j470b-server-mesh-rejoin
mesh-compat-382
j444b-d381-key-wins-flood
j461b-macos-utun-reader-thread
j475b-remove-host-motion
fix-reconnect-new-link
j481-d380
dev
j479-mesh-soak
fix/j476-host-facts
j475-motion-pill-split
j470-mac-node-join
root-upgrade-d376
j474-review-merge
latency-targets-120hz
nightly-suite-shards
gotatun-pin-j469
macos-utun-reader-thread
fix-flood-echo-stall
wip/j416
motion-fast-ease-out
mesh-addhost-j259
takeover-close-terminals-d368
kbd-instant-followups
fix-trust-key-double-click
nightly-suite-partial-report
appwin-icons-j207
rusqlite-slice1
swipe-content-first-d374
mesh-mtu-381
addhost-instant-open
glass-contrast-d365-mock
glass-contrast-d365
signout-followups-442
fix-startup-toast-red
research-gotatun-matrix-j317
themes-sync-j304
fix-load-flakes
fix-nix-zig-patch
fix-compute01-browser-baseline
ratchet-size-baseline-419
predict-auto-d130
revoke-coverage-d367
nightly-suite-queue-timeout
nightly-browser-suite
fix-rustsec-2026-0459
fonts-ioskeley-obel
log-retention-d375
fix-kbd-no-anim-427
fix-dev-red-specs-2
takeover-proof-305-j205
wip/j410-server-pump
ui-consistency-d372
fix-osc52-clipboard
libghostty-vt-1.0
forget-retired-host-d369
j404-merge-426
fix-setup-lock-race
fix-touch-select-426
revoke-access-report-ui
fix-predictive-echo-specs
fix-selinux-agent-exec
wip/j395-checkpoint2
wip/j395
j391-review-scratch
ui-consistency-j195-deferred-rebased
fix-rustc-paths-421
decisions-wording-341
fix-install-suite-red
fix-dev-rtt-specs
fix-pending-removal-align
fix-privileged-skip
fix-install-suite-404
review-j368
mesh-dns-j254
revoke-access-report-d356b
revoke-privilege-round5
fix-removal-confirm
review-j360
gotatun-rotation-367
deps-refresh-340
review-j354
review-J351
review-j350
addhost-takeover-d359
revoke-report-items-347
revoke-end-sessions-d362
review-j348
size-mesh-ssh-346
size-trim-337
revoke-access-diff-d356
j259-pre-rebase-b06c5a6
j305-ended-at-start-spec
addhost-sudo-consent
fix-layout-real-specs-411
fix-shell-open-411
fix-console-hosts-373
fix-shell-exit
fuzz-host-name-353
fix-host-details-410
macos-filter-j197
research-gotatun-tricks
revoke-p3-j216l
stale-state-pill
revoke-followups-j216k
revoke-progress-j216e
backup/macos01-20261106/lume/j197
backup/macos01-20261106/lume/appwin-icons-j207-stash0
backup/macos01-20261106/claude/stash-0
backup/macos01-20261106/claude/stash-1
backup/macos01-20261106/claude/J80-macos-s1
backup/macos01-20261106/claude/J80
backup/macos01-20261106/claude/J80-slice1-20261102
backup/macos01-20261106/claude/J16b-appwin-on-dev
backup/macos01-20261106/claude/appwin-launch-idle-146
backup/macos01-20261106/claude/J80-s1-run
wip/j259
trust-285-p3c
runtime-lock-j216f
mesh-ssh-p3-j175
tun-proof-p3b
trust-285-p3
terminal-polish-j271
tun-proof-p3
ssh-install-flakes-j216i
fuzz-fixes-j264
trust-285-a
trust-285-b
restart-oversized-fixture-j198
backup/macos01-20261106/lume/appwin-icons-j207
restart-bound-j216h
up-preflight-p3-j198
backup/macos01-20261106/lume/j251
up-preflight-j198
wip/up-preflight-j198-j245-p2
wip/up-preflight-j198-j245-p1
agent-fsync-offloop-j196
wip/up-preflight-j198-j245
settings-offscreen-j231
wip/up-preflight-j198-j226-inspection
wip/j103f
themes-j206
wip/up-preflight-j198-j226
flakes-j216c
backup/macos01-20261106/claude/j197
flakes-j216b
backup/macos01-20261106/claude/j197-red
wip/up-preflight-j198-review
flakes-j216
backup/macos01-20261106/claude/appwin-icons-j207
engineering-j208b
appwin-mock-j209
toast-stack-j165
engineering-j208
decisions-d351
decisions-d350
decisions-d349
appearance-presets-mock-j129
decisions-d348
docs-audit-spec
doc-coverage-baseline-raise
decisions-d347
decisions-d346b
decisions-d346
j88h-profile
ui-consistency-j195-deferred
ui-consistency-j195
engineering-practices-j194
preferred-login-ui-j166
signin-pools-263-j174
macos-node-265-s1
decisions-d345
decisions-d344
proxy-cgroup-j164
verify-quick-fuzz-329
proxy-start-io-j168
timing-flakes-j163
up-263-s1
mcp-events-j147
verify-quick-fuzz-j158
proxy-p3-j159
agent-resource-j157-detect
wip/j103d-final
remote-cancel-hang-j155
proxy-traefik-281-j62d
j149-verify-quick
proxy-traefik-281-j62b
j150-drop-unnamed
wip/j79d-red-r4
wip/j103d-d332
rootless-toast-j142
wip/j103d-head
wip/j103d
j125c-mcp
macos-build-j136
backup/macos01-20261106/claude/j136
wip/j79d-red-cli
wip/j79d-red-server
j121-link-liveness
j118c-tracer-failures
wip/j79c-head
release-desktop-j131
j124c-signed-macos
wip/j79c-red-p3
wip/j79c-red-p23
wip/j79c-red-p22
merge-j128-check
perf-deferred-j128
sidebar-toggle-260
j127-shortcut-changed-dot
j125-mcp
j118-tracer-failures
j126-account-summary
settings-sections-mock-j92
j120-pinch-lock
sidebar-292
backup/macos01-20261106/claude/j120
taste-2026-11-03
row-actions-242-215
backup/macos01-20261106/claude/stash-0-untracked
backup/macos01-20261106/claude/J80-265-s1-run
backup/macos01-20261106/claude/J80-macos-s1-20261102
j112-canvas-commands
ui-no-overshoot-scroll
fix-recovery-restart
backup/macos01-20261106/claude/stash-1-untracked
taste-2026-11-02
size-baseline-ratchet
machine-cards-structure-j108
j109-boot-copy-mockups
fix-dev-red
wip/up-263-s1
j95b-settings-rows
proxy-traefik-281
j95-option-rows
docs-human-j91
docs-keys-palette
fix-flood-1013-270-j38d
mesh-ssh-s1
mesh-ssh-s0
addhost-d291-tunnel
test-stability-load
mesh-relay-220
repro-296
shim-thp-off-24
mesh-pins-258-v2
mesh-dns
signin-d311-merged
docs-pages
ui-size-283
fix-ui-batch-specs
wip/j12b-edge-update
gate-on-arm
fix-rename-sync-293
trust-285-j60b
fix-focus-ring-clip
integ-ui-1101
edge-update-v2
signin-d311
taste-2026-11-01
layout-per-tab-250
ui-approved-splitopt-224
ui-approved-splitopt-224-on-dev
icon-text-centre-282
ui-approved-189-212-216
research-reverse-proxy
transfer-budget-320
review-mesh-pins-258-r2
sidebar-pill-259
review-mesh-relay-220
wip/j12-edge-update-v2
nix-package
status-crash-loop-237
fix-dev-test-failures
fix-fixture-socket-path
wip/mesh-relay-220-pre-dev-advance-rebase
fix-canvas-side-clip
research-head-to-head
frosted-real-256
review-mesh-pins-258
wip/mesh-relay-220-pre-final-rebase
review-layout-per-tab-250-r2
broad-suite-triage
nix-devshell
review-status-crash-loop-237-r3
wip/nix-devshell-pre-p3
taste-frosted-real
wip/code-serve-demo-2026-10-30
wip/code-serve-glass-2026-10-30
wip/code-serve-clusters-2026-10-30
wip/code-merge-release94-dev-2026-10-30
wip/rproxy-delta-scratch-2026-10-30
wip/stale-note-257-staged-2026-10-30
wip/d241-size-staged-2026-10-30
review-broad-suite-207
wip/mesh-pins-258-pre-split
wip/mesh-relay-220-pre-rebase
mesh-measure-220
plan-211-update
review-edge-autoupdate-r3
perf24-reconnect
review-stale-note-r2
release-prod-live
d241-shell-marker
review-mesh-pins-server-258
review-revocation-r3
review-setup-token-r2
review-sync-journal-r3
wip/review-mesh-pins-258-prune-review-1995fee46
frosted-scroll-256
sync-journal-r3
revocation-per-account-wip
review-tokio-asyncfd-0386
review-sync-p3s
wip/stash-8by64-stash0-2026-10-30
mesh-drag-r2
setup-token-data-wip
release-9-4
auth-busy-1013
mesh-view-overlap
sync-p3s
review-auth-busy-wake
review-readd-r8-wip
mesh-drag-bounds
audit-auth-93
app-windows-stream-start
review-authority-cap-r3
app-windows-r7-measure
readd-minimal-r6
readd-minimal-r5
app-windows-r7-soak
readd-minimal-r4
app-windows-r7
app-windows-r6
readd-minimal-r3
readd-minimal-r2
readd-minimal
merge-release8-dev
review-busy-page-liveness-r2
review-busy-page-liveness
mesh-view-trim
review-authority-read-retry-r2
review-remove-uninstall-r2
release-retry-reattach
release-retry
review-remove-uninstall
review-authority-read-retry
app-windows-ship
mesh-autoswitch-p2
mesh-ssh
mesh-relay
mesh-macos-node
screen-and-control
session-labels
review-real-logins-r5
review-real-logins-r4
review-real-logins-r3
shim-memory-r2
review-real-logins-r2
shim-memory
shadow-addhost-instant
shadow-real-logins
shadow-same-session-flood
shadow-prod-perf-fix
hosts-from-server
app-windows-real
app-windows-transport
release-prod-6w
mesh-policy-verify
quiet-close-toast
dev-merge-r4
review-mesh-agent-transport
review-app-windows-transport
settings-roundtrip
review-release-pipeline-r4
mesh-agent-transport
ended-strip-fix
review-audit-ca-update-d244
settings-rows
phone-width
signout-clear
fix-restart-tests
dev-merge-rp
signer-hardening
review-audit-ca-update
terminal-fidelity
review-audit-ca-update-wip
mesh-integrate
review-mesh-agent-transport-515d3712
review-mesh-dataplane-feec1988
mesh-dataplane
signout
research-canvas-node
wip/review-renderer-cost
wip/review-terminal-fidelity
padlock-glyph
review-release-pipeline-2e95b689
phone-polish
sidebar-polish
inbox-generic
review-mesh-dataplane-378d4055
prod-batch
ui-inbox
ui-ca-unlock
ui-machine-sections
review-mesh-dataplane-a5f744fb
tls-session-prod
session-bounds-prod
fix-dead-session
session-bounds
tls-session
palette-polish
shortcuts-d222
research-appmeta
phone-gutter
ui-centre-button
wip/ui-centre-button
wip/tls-session
review-addhost-breakit
ui-platform
renderer-cost
remote-agents
mesh-api
ssh-bootstrap
prod-hotfix2
prod-fixes
review-remote-agents
review-mesh-dataplane
session-life
research-browser
review-mesh-api
research-capture
review-mesh-api-round3
review-mesh-api-round1
gate-stability
review-gate-stability
review-relay-direct
review-mesh-api-findings
proto-sessions
font-picker
proto-addhost
proto-windows
mesh-core
latency-gate
research-bsd
research-discovery
keytray-mods
mock-core
ui-tweaks
fix-motion-sel
fix-motion
integrate2
fix-robust
t-web
t-server
vt-ghostty
fix-snap-echo
t-web-auth
review-vt2
t-agent
lane-protocol
lane-shim
lane-auth
s1-base
s1-contracts
s1-workspace
No results found.
No labels
launch-qa
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set
Reference
kayg/infraternal#118
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Every Codex job tonight hit intermittent failures in the native gate under host load (avg 20-50 on 8 cores): crates/infraternal/tests/vt_credit.rs:22 (t1-gate-approximate-1, t1-gate-faithful-1 'credit probe failed'), scripts/verify-agent-snapshot.py:48 repeated_tiny_credit_pauses_preserve_shell_progress (snapshot count / output cursor), and auth socket_tests timeouts (recovery_subscription_updates_every_signed_in_socket, half_open_sockets_obey_user_limit, review3_busy_*). They pass alone. Find out whether each is a test that depends on wall-clock timing (fix the test to be deterministic) or a real ordering issue in credit/snapshot/socket handling that load exposes (fix the code, failing test first). Never loosen a threshold to pass.
Launched gate-stability (Codex GPT 6.1 Sol Medium: may be a real credit/snapshot ordering issue in the data path, not only test timing). Branch gate-stability from dev
1c3ae5f.Started gate-stability at base
1c3ae5f40bon infraternal-dev (8 cores). Read #118 and its only comment; no parent is linked in the issue body. Will reproduce credit, repeated Snapshot pauses, and auth socket/review3 tests under a fixed local CPU load, classify from evidence, commit each gated logical fix, and run the fast gate twice under that load. No push or deploy.Credit baseline at
1c3ae5fon infraternal-dev, 8 cores, fixed 16 CPU workers (default 300 ms RTT + jitter). Ten runs/case: approximate/1 = 1/10 pass, approximate/16384 = 1/10, faithful/1 = 0/10, faithful/16384 = 0/10. Failures: 33 initial-producer 5 s marker waits (scripts/verify-vt-credit.py:30), 5 second-producer 1.5 s waits (:43); 2 pass. No threshold changed. Investigating event-based detachment/readiness; the 249/250 ms boundary is already checked exactly in crates/infraternal-agent/tests/unit/relay.rs.before-credit-approximate-1.json: one-minute load 30.79–46.79
before-credit-approximate-16384.json: one-minute load 30.79–46.79
before-credit-faithful-1.json: one-minute load 30.79–45.99
before-credit-faithful-16384.json: one-minute load 30.79–46.79
Auth evidence: the ordinary 16-worker, all-8-core profile passed 10/10 review3 module runs (13 tests/run); socket module still measuring. Additional diagnostic: 16 extra workers and the auth test confined to CPU 0, scope CPUWeight=10000. review3_backend_reply_has_a_deadline failed at review3_tests.rs:662: measured 1.998570571 s against its 2 s assertion, although the server timer is armed before the test observes the backend Call and starts its stopwatch. Classification (a): test stopwatch starts after the timer, so scheduling delay changes the measured interval. Will retain the exact 2 s configured deadline and test it with paused time and an observed Call; no production deadline change. Evidence: scratch gate-stability/before-contended-review3-01.log.
Socket baseline: ordinary fixed 16-worker profile passed 10/10 complete socket modules (8 correctness tests/run). A separate CPU-throttled diagnostic (CPUWeight=10000, CPUQuota=10%, same host and generator) reproduced unexpected Close in recovery_subscription_updates_every_signed_in_socket and half_open_sockets_obey_user_limit; first runs both failed in 8.14/9.00 s. Evidence: before-throttled-{capacity,half-open}-01.log. Classification (a): these correctness/admission fixtures inherit production 25 ms authority reads and 5 s Hello deadlines, so scheduling delay changes the state being asserted. Frozen test time retains those exact production values and lets real SQLite/network readiness complete before explicit expiry. No production code or budget changed. Ordinary and throttled before/after counts will be reported separately.
Credit classification (a): the probe coupled readiness to 5 s and 2 MiB producer throughput to 1.5 s, rather than observing detachment. Event-based probe now records the actual attempted repaint cost (> partial grant), shim EOF, exact drained output window, unchanged cursor/no premature event, producer marker, and one funded Snapshot. All four first checks passed under load with second-producer completion 2245.5–4599.6 ms, zero notices/repaints before funding. Production relay is unchanged; its exact 249/250 ms unit boundary remains. Evidence: credit-green-{approximate,faithful}-{1,16384}.log. The 50-run matched-load measurements remain pending.
Snapshot baseline completed: 3/10 pass on infraternal-dev with the fixed 16-worker load (additional CPU-0 auth diagnostic workers overlapped later runs; load recorded per run). Seven failures when reusing stream 1: stale old Output was read after out reset to zero; example cursor (38614774,0), scripts/agent_fixture.py:164 via verify-agent-snapshot.py:65. A separate RTT=1200 ms diagnostic also reproduced the :48 cursor/count assertion. Classification (a): 500 ms of socket silence is not proof that funded output has drained, and 800 ms after Credit is not a readiness signal. Fix preserves all four original yes/head 12 MiB floods, three 1-byte grants and 1 MiB recoveries, and the >8 MiB ring-wrap check. Exact funded-window drain, shim EOF and control Pong now fence the observations and final stream reuse. No native production path or threshold changed. Evidence: before-snapshot-*.log/json and diagnostic-snapshot-delay.log.
Correction to the preceding Snapshot comment: failures were five cursor/count assertions at verify-agent-snapshot.py:48, one stale Output on reattachment (38614774,0), and one completed-2 marker hang at 90 s. Total remains 3/10 pass. They were not all stream-reuse failures. Exact credit drainage addresses both the queued-frame assertions and the marker hang caused when the fixture stops receiving before consuming its granted transport window. The first corrected probe is still running; no final success claim.
half_open_sockets_obey_user_limit matched throttled profile: before 7/10 pass; after 50/50 pass. infraternal-dev, 8 CPUs; fixed 16 all-CPU workers plus 16 CPU-0 diagnostic workers; test CPUWeight=10000, CPUQuota=10%. After one-minute load range 67.60–84.63. Production 5 s Hello and 25 ms authority deadlines unchanged; test time is frozen until explicitly advanced. No code committed yet: first complete fast gate is compiling. Evidence: before-throttled-half-open.json and after-throttled-half-open.json, per-run logs under scratch gate-stability.
Benchmark finding: managed native shims run in independent
infraternal-session-<UUID>.scopeunits with default CPUWeight=100, so caller CPUWeight=10000 does not propagate. Under the fixed 32-worker load, the revised snapshot probe completed 15/22 attempts before stopping; several producer markers exceeded their unchanged 90-second hang bound. The fixture now has an opt-in INFRATERNAL_TEST_CPU_WEIGHT setting which authenticates its own shim UUID/PID through Hello before weighting only that session scope. Matched reruns will weight caller and owned shim equally, retaining all four 12 MiB floods, original credit sizes, and per-step hang bounds. The loop harness overall bound is 600 seconds to cover four separate 90-second producer bounds; it is not a performance assertion. No production scope policy changes. Evidence: scripts/agent_fixture.py:benchmark_scope; prepriority-after-matched-snapshot.json and per-run logs in the job scratch directory.The post-launch weighting experiment exposed terminal.open refusals before viewer attachment (unchanged production two-second shim readiness bound) at load 90+. Cause: the managed shim still starts at default CPUWeight=100 until Hello returns. The opt-in benchmark fixture now wraps only its child PATH systemd-run to apply the requested CPUWeight from launch; it still uses real manager scopes and validates the returned owned UUID/PID. No retry, readiness-budget increase, or production-policy change. Restarting snapshot cohorts from zero with this correction applied to original and revised probes alike. Earlier post-launch-weight cohorts remain recorded, including failures.
Credit classification (a), test timing: original matched fixed 32-worker profiles passed approximate/1=0/10, approximate/16384=0/10, faithful/1=1/10, faithful/16384=0/10. Revised probes passed 50/50 in each of all four combinations under that same load. Producer completion exceeded the old 1.5-second assertion in successful runs without a stalled PTY. They now observe actual shim EOF, exact consumed credit, attempted repaint cost exceeding the partial grant, ordered control Pong, and an explicit FIFO producer-release marker. Original 12 MiB + 2 MiB payloads and credit sizes remain; exact 249/250 ms production starvation boundaries remain covered by relay unit tests. No production code or budget change. Evidence: before-matched-credit-.json and prepriority-after-matched-credit-.json (individual logs retained), scripts/verify-vt-credit.py and scripts/credit_fixture.py. All runs on local infraternal-dev under systemd-run --user --scope -p CPUWeight=10000, 16 all-core CPU workers plus 16 CPU0 diagnostic workers.
Snapshot classification (a), test timing and fixture stream ordering: initial default-weight loaded baseline passed 3/10 (five cursor/count failures, one stale queued Output after resetting reattach cursor, one producer hang). With caller and owned shim weighted equally from launch, matched default RTT original passed 10/10 and revised 50/50. A matched RTT=1200 ms condition exposes the old 0.5-second socket-silence fence: original 0/10; revised cohort still running. Revised probe observes the exact consumed window, actual shim EOF and unaffordable attempted Snapshot, then drains an ordered Pong; reattach waits for old viewer output to drain before resetting the cursor. It preserves every original 12 MiB flood (48 MiB/run), three one-byte grants, three one-MiB refunds, and >8 MiB ring-wrap assertion. No production credit/snapshot code changes.
before-launch-weight-snapshot: 10/10; median 20.974s; observed load1 79.87–80.60
after-launch-weight-snapshot: 50/50; median 22.547s; observed load1 76.16–83.95
before-launch-weight-snapshot-rtt1200: 0/10; median 11.869s; observed load1 79.03–80.52
after-launch-weight-snapshot-rtt1200: 26/26; median 48.115s; observed load1 76.57–82.52
Evidence: scripts/verify-agent-snapshot.py, scripts/credit_fixture.py; individual before/after-launch-weight-snapshot*.json/log files under /home/kayg/.cache/codex-scratch/gate-stability/. All runs local under CPUWeight=10000 with the same fixed 32 CPU workers; RTT/jitter are fixture network conditions, not relaxed thresholds.
Auth capacity classification (a), wall-clock test dependence: under unchanged CPUWeight=10000 / CPUQuota=10% with the fixed 32-worker load, recovery_subscription_updates_every_signed_in_socket passed 0/10 originally and 50/50 after; half_open_sockets_obey_user_limit passed 7/10 originally and 50/50 after. Real loopback sockets and SQLite remain. Tests start with the paused runtime epoch and a blocking-task guard prevents automatic virtual-time jumps while real I/O is pending. Readiness is observed through Hello/Welcome/Call/Pong; time only advances explicitly. Sixteen reserved sockets remain full at 4999 ms, then release at the exact unchanged 5000 ms boundary; fragment completion uses the same boundary. Recovery still requires 16 start + 16 cancellation events per run and a different-account state snapshot plus Pong fence, replacing a 250 ms silence assertion. Admission alone exercises 800 held sockets across 50 runs; the recovery test checks 1600 notification deliveries. Production timeouts, authority code and socket limits are unchanged.
Evidence: before-throttled-capacity.json, after-throttled-capacity.json, before-throttled-half-open.json, after-throttled-half-open.json; crates/infraternal-server/src/auth/socket_clock.rs and socket_tests.rs. Complete module 50-run cohorts are still running.
Snapshot high-RTT matched cohort complete: original 0/10, revised 50/50 under the same fixed 32-worker load, RTT=1200 ms and configured jitter, with caller and fixture-owned shim scopes CPUWeight=10000 from launch. All four original 12 MiB floods, three partial/funded snapshot cycles and final >8 MiB ring-wrap check passed in every revised run (2,400 MiB total shell output over 50 runs). Default RTT revised also passed 50/50. Evidence: before-launch-weight-snapshot-rtt1200.json, after-launch-weight-snapshot-rtt1200.json and their individual logs. Per-step marker deadlines remained 90 seconds; no native production changes. The six atomic change steps are now being gated against their own trees.
Complete socket_tests cohort finished: 50/50 module runs, eight correctness cases per run = 400 passing cases, including recovery subscription, admission, exact handshake/frame deadlines, authenticated idle, revoked idle and clean Close. Original same CPU0-affinity / fixed 32-worker profile was 10/10 module runs; the separate quota profile reproduced recovery/admission failures as recorded above. Correctness runtimes use paused time; the pre-existing ignored release measurement stays on ordinary time and is unchanged (no new skips/ignores). Median module runtime 20.131s; load1 range 62.49–92.81. Evidence: before-contended-auth.json, after-contended-auth.json; all per-run logs. Revised test executable infraternal_server-02fb6cfd6f1f8c56, source snapshots preserved in scratch steps/socket/. Review3 50-run cohort continues.
Complete review3_tests cohort finished: 50/50 module runs, 13 correctness cases/run = 650 passing cases, including busy logout, recovery and expiry, pending routes/idle expiry, revoked handshake/resume, backend reply deadline and late reply revocation. Original same CPU0-affinity / fixed 32-worker profile passed 9/10; the failing backend probe measured 1.998570571 s from observed Call against a timer armed earlier with a two-second deadline. Classification (a): test stopwatch origin and wall-clock authority/read deadlines. Revised runtime epoch is paused from construction, actual Hello/Welcome/Call events establish readiness, and configured deadlines advance exactly (no extra millisecond allowance). The near-term expiry fixture now obtains Welcome before explicitly backdating the real stored authority and crossing its armed timer. Production socket/authority code and budgets unchanged. The pre-existing release-only ignored measurement remains ordinary-time and unchanged. Median 31.555s; load1 range 50.28–93.31. Evidence: before-contended-review3.json, after-contended-review3.json and per-run logs; preserved steps/review3/ sources. All requested corrected cohorts have now reached 50/50. Owned diagnostic CPU workers have been stopped for intermediate commit gates; the same load will be restored for both final gates.
Fixture scheduling follow-up: the test launcher now copies the caller cgroup CPU weight automatically on unified hierarchies, with the explicit INFRATERNAL_TEST_CPU_WEIGHT override retained. Thus the requested ordinary
systemd-run --user --scope -p CPUWeight=10000 scripts/verify fastinvocation also weights independently managed test shims from launch. Legacy hierarchies retain ordinary setup; detached shims inherit directly, and an absent manager executable remains supported. This is fixture-only: no production manager policy change, global configuration, deadline change or retry. The completed matched snapshot cohorts explicitly used that same effective 10000 weight. The first isolated gate was interrupted before commit to restart against this final helper form; its tests had passed. Both final gates will exercise automatic inheritance without the override.Completed fixture step: Observe exact native credit windows and shim detachment. HEAD
31bb5b065d. scripts/verify fast passed on this exact commit tree. Gate output retained at /home/kayg/.cache/codex-scratch/gate-stability/commit-fixture-fast.log; all budgets unchanged. Baseline/load pass-rate evidence is recorded in preceding comments.Baseline evidence limits, by requested review3 case:
review3_busy_logout: 10/10 original baseline attempts passed; revised 50/50.
review3_busy_recovery: 10/10 original baseline attempts passed; revised 50/50.
review3_pending_routes_and_idle_expiry_recovery: 10/10 original baseline attempts passed; revised 50/50.
These three reported failures were not individually reproduced in the ten baseline module loops on this host. The module failure actually reproduced was the backend stopwatch (1.998570571 s vs 2 s). Their remaining real-I/O/wall-time dependence is now controlled by the same paused-clock/readiness fixture, with their authority assertions retained. No production ordering issue was demonstrated in these experiments. Case-level after evidence covers all 13 cases in every one of the 50 complete revised module runs.
Fixture infrastructure committed at
31bb5b065dafter scripts/verify fast passed on that exact tree. Automatic caller-weight smoke read 10000 inside its requested scope; gate used automatic inheritance (no override). x86_64 static binary 15,965,376 bytes vs unchanged max 16,404,816; aarch64 13,426,040 vs unchanged max 13,817,816. Both static, no PT_INTERP. TypeScript: 54/54 tracer tests and 7/7 protocol tests; compiler, production build and transfer budget passed. Source: scripts/agent_fixture.py and scripts/credit_fixture.py. Optional observation callbacks reuse the existing shim proxy; exact credit/Pong draining and inherited fixture-only scheduling are the shared prerequisites for the subsequent probe commits. Gate output: commit-fixture-fast.log in the job scratch directory.Completed credit step: Wait for real partial-credit detachment and producer progress. HEAD
3e291f2ec2. scripts/verify fast passed on this exact commit tree. Gate output retained at /home/kayg/.cache/codex-scratch/gate-stability/commit-credit-fast.log; all budgets unchanged. Baseline/load pass-rate evidence is recorded in preceding comments.Partial-credit probe committed at
3e291f2eafter its complete scripts/verify fast gate passed on that tree (54 tracer tests, 7 protocol tests; both static size budgets unchanged and passing). Loaded probe evidence: approximate/1 0/10 -> 50/50; approximate/16384 0/10 -> 50/50; faithful/1 1/10 -> 50/50; faithful/16384 0/10 -> 50/50, same fixed CPU-generator profile. scripts/verify-vt-credit.py replaces readiness/producer wall-clock assertions with observed EOF, exact credit, attempted unaffordable repaint and explicit producer completion; preserves original 12 MiB + 2 MiB payloads. Gate log: commit-credit-fast.log; all individual loaded results retained in the before-matched-credit and prepriority-after-matched-credit JSON/log files.Completed snapshot step: Fence funded snapshot windows before checking cursors. HEAD
274f794216. scripts/verify fast passed on this exact commit tree. Gate output retained at /home/kayg/.cache/codex-scratch/gate-stability/commit-snapshot-fast.log; all budgets unchanged. Baseline/load pass-rate evidence is recorded in preceding comments.Snapshot probe committed at
274f794216after its complete scripts/verify fast gate passed on that exact tree. Real loaded comparisons: revised default RTT 50/50; matched RTT=1200 ms original 0/10 -> revised 50/50. Each run retains four 12 MiB floods, three one-byte pauses, three one-MiB refunds, exactly one funded snapshot per cycle and >8 MiB ring-wrap. CPU weighting is automatically inherited from the caller for managed test shims; this gate used no override. Both static size budgets, compiler/checks, 54 tracer tests and 7 protocol tests passed. Gate output: commit-snapshot-fast.log. All three native steps are now separately committed; auth dependency, socket-clock/socket tests, and review3 test steps follow independently.Completed tokio step: Enable virtual time only for auth tests. HEAD
b60f19f88d. scripts/verify fast passed on this exact commit tree. Gate output retained at /home/kayg/.cache/codex-scratch/gate-stability/commit-tokio-fast.log; all budgets unchanged. Baseline/load pass-rate evidence is recorded in preceding comments.Dev dependency step committed at
b60f19f88dafter scripts/verify fast passed on that exact tree: 54 tracer tests, 7 protocol tests, both static binaries within unchanged budgets. One existing Tokio dev-dependency feature (test-util) added; zero new libraries, zero lockfile changes and zero production feature changes. This is the prerequisite for the explicitly controlled correctness clocks in the next two commits. Gate log: commit-tokio-fast.log. Loaded socket/review3 evidence remains 50/50 complete modules, as recorded above.Completed socket step: Control socket deadlines across real database and network I/O. HEAD
64ed588f98. scripts/verify fast passed on this exact commit tree. Gate output retained at /home/kayg/.cache/codex-scratch/gate-stability/commit-socket-fast.log; all budgets unchanged. Baseline/load pass-rate evidence is recorded in preceding comments.Socket clock/tests step committed at
64ed588f98after scripts/verify fast passed on that tree. Complete loaded module: 50/50 x 8 correctness cases = 400 passes. Quota diagnostics: recovery 0/10 -> 50/50; half-open capacity 7/10 -> 50/50. Exact 4999/5000 ms handshake and partial-frame boundaries are asserted; all 16 sockets/recipients and the other-account isolation remain. The shared blocking-task clock guard holds virtual time across real SQLite/TCP without CPU spinning; wall timeout only bounds a hang. Existing ignored release measurement is unchanged and remains real-time. Gate: commit-socket-fast.log; both static budgets, compiler/build/transfer checks, 54 tracer tests and 7 protocol tests passed. Review3 is now the final atomic step being gated.Completed review3 step: Advance auth deadlines after observed socket readiness. HEAD
8b7cccde7d. scripts/verify fast passed on this exact commit tree. Gate output retained at /home/kayg/.cache/codex-scratch/gate-stability/commit-review3-fast.log; all budgets unchanged. Baseline/load pass-rate evidence is recorded in preceding comments.Review3 step committed at
8b7cccde7dafter scripts/verify fast passed on that exact tree. Loaded complete module: original 9/10 -> revised 50/50 x 13 correctness cases = 650 passes. Backend timeout advances exactly two seconds after observed Call, retaining late-result rejection and zero terminal-row mutations; busy logout/recovery/expiry and pending-route checks remain real TCP/SQLite. Original busy logout, recovery and pending expiry were 10/10 individually in this reproduction (limitations recorded above). No new skips/ignores, no production policy/timeout changes. Gate: commit-review3-fast.log, including 54 tracer and 7 protocol passes and both static budgets. Six local atomic commits now exist; no push/deploy. Starting the required final two complete scripts/verify fast runs under the same fixed 32-worker profile, automatic caller CPUWeight inheritance, unique TRACER_TEST_PORT=4318.Final loaded workspace gate found an additional test-clock dependence: review3_revoked_resume_must_not_welcome observed Close only after the frozen-clock watchdog released at about 89 s, then failed the guard assertion. No Welcome was issued. The isolated 50-run module cohort did not expose this shutdown path. Its negative phase has already committed logout and queued Resume; only Close is valid, but shutdown may need the existing bounded writer fallback. The test now releases its frozen clock after that readiness point, allowing the unchanged real close/read budgets to run. Its original 30-second read deadline remains a hang bound, and every binary Welcome remains rejected. No production code, deadline or threshold change. Failing test first evidence: failed final-fast-1.log lines 308–319, head
8b7cccde, 32 workers, load1 finish 59.91. This fix will be separately gated/committed and the required two final gates restarted.Corrected revoked Resume negative phase passed 50/50 under the restored fixed 32-worker load (same all-core/CPU0 generator scopes CPUWeight=10000). Median 0.675s; load1 30.72–35.77. Positive readiness/authority mutation remains frozen; after committed logout + queued Resume the clock is released so shutdown writer fallback and the unchanged 30-second hang bound can progress. Every run still rejected Welcome and observed closure. Evidence: after-fixed-revoked-resume.json + per-run logs. Full workspace gate for this additional atomic fix is still running under that same load.
Additional fix validation: the loaded workspace test suite passed, including revoked Resume and all other auth cases. The same full gate failed clippy only: explicitly dropping an underscore-prefixed _clock binding violates the existing used_underscore_binding lint. Renamed that local binding clock (no behavior/deadline change), then restarting the full gate. Corrected revoked Resume 50/50 evidence remains valid. No commit made before a complete green gate. Evidence: lint-fix-close-phase-fast.log.
Additional negative-close clock fix committed at
3b51a63c99after complete scripts/verify fast passed on this exact tree under the fixed 32-worker load. Corrected revoked Resume case: 50/50; workspace gate exit 0, elapsed 351.175 s, start load ['24.55', '31.75', '29.14'], finish load ['54.14', '47.82', '37.36']. Both static budgets and 54 tracer/7 protocol tests passed. Only this test releases its clock after committed revocation + queued Resume, retaining its original no-Welcome assertion and 30-second hang bound. Failing gate and lint correction remain recorded; no production changes. Seven local atomic commits now. Restarting both required final gates on this head under the same load. Gate output: fix-close-phase-fast.log.Final scripts/verify fast 1/2: PASS, head
3b51a63c99. Local infraternal-dev, 8 cores, same fixed 32 CPU workers; all callers/shims CPUWeight=10000 by automatic inheritance, no override. Elapsed 304.695 s; load start ['37.39', '44.21', '36.59']; finish ['49.64', '50.38', '41.67']. TRACER_TEST_PORT=4318. Complete output below, verbatim. Second gate is running under the same load.Verbatim final gate 1 output
Final scripts/verify fast 2/2: PASS, head
3b51a63c99. Local infraternal-dev, 8 cores, same fixed 32 CPU workers; callers/shims CPUWeight=10000 by automatic inheritance, no override. Elapsed 307.863 s; load start ['49.64', '50.38', '41.67']; finish ['41.52', '44.61', '41.53']. TRACER_TEST_PORT=4318. Both final gates passed. Complete output below, verbatim.Verbatim final gate 2 output
Finished #118 at head
3b51a63c99: seven atomic commits, each with a passing fast gate. Classification (a): timing dependence in tests and fixture stream draining; no demonstrated production ordering defect. Real readiness, exact credit windows, ordered Pong fences and controlled auth time replace silence/stopwatch assertions. No production deadlines, payloads, grants, admission limits, wire behavior or budgets were relaxed; no new skips/ignores.Matched before/after cohorts on local infraternal-dev (8 cores), fixed 32 CPU workers, CPUWeight=10000 benchmark scopes; pinned/10%-quota variants are identified below.
The complete socket/review3 cohorts cover 400/650 passing test executions. Corrected revoked-resume negative-close phase separately passed 50/50 under the same load. Original busy_logout, busy_recovery and pending_routes/idle-expiry each passed 10/10 individually here: their reported intermittent failures were not independently reproduced; the shared deterministic fixture covers them. Baseline failures, corrected/invalidated cohorts and the failed first final attempt remain recorded above.
Final fast gates on this head both passed under unchanged 32-worker load: 304.695 s and 307.863 s. One-minute load at boundaries: 37.39 -> 49.64 -> 41.52. Complete outputs are quoted verbatim in the preceding two gate comments. Static binary sizes remain x86_64 15,965,376 bytes / aarch64 13,426,040 bytes, within unchanged budgets; no PT_INTERP.
Author reviewer checklist completed: real SQLite/TCP authority checks, cross-account recovery isolation, all sixteen recipients, exact virtual boundaries, late backend replies, shutdown clock release, retained ring-wrap/payload checks, module invariant documentation, dev-only Tokio feature, fixture-owned scope handling and unchanged production security/performance policy reviewed. Build targets and temporary target symlink removed; owned load scopes stopped. No push/deploy; issue remains open.
Evidence: /home/kayg/.cache/codex-scratch/gate-stability/pass-rate-summary.json, final-environment.json, final-fast-1.log, final-fast-2.log, author-review.md and per-cohort JSON/logs.
Merged into dev as
459070fe(test-only: fixtures, paused-time socket tests, tokio test-util dev-dependency; dev was its exact base, so the job's two green gates cover the merged tree). Running jobs pick it up when they merge dev. Verification review queued.Independent verification review started on review-gate-stability, target
3b51a63c99, original base1c3ae5f40b. Read all 39 comments on #118; no parent linked. dev advanced to the merge during initial reads, so review retains original base. Verifying assertions and loaded fixture behaviour; no implementation fixes, push or deploy.Independent source review step complete at
3b51a63c99(review-gate-stability unchanged). Nine files: test/fixture-only; no production source, lockfile, contract, skips/ignores or wire changes. Checked D13/D41/D75/D81/D170 and relevant shim/wire contracts. Original 4 x 12 MiB snapshot floods, 12 MiB + 2 MiB credit payloads, all sixteen recovery recipients and other-account checks retained. Exact 249/250 ms relay coverage remains. No confirmed finding. Fresh fast gate: fmt/native fmt/clippy passed; test build continues. Independent loaded attempts will be 10 per case, not a repeat of the author's 50-run cohort.kayg referenced this issue2026-10-28 17:53:56 +00:00