Test: motion-audit spec is flaky under load #65
Closed
opened 2026-10-07 08:40:47 +00:00 by kayg
·
18 comments
No Branch/Tag specified
main
j470b-server-mesh-rejoin
mesh-compat-382
j444b-d381-key-wins-flood
j461b-macos-utun-reader-thread
j475b-remove-host-motion
fix-reconnect-new-link
j481-d380
dev
j479-mesh-soak
fix/j476-host-facts
j475-motion-pill-split
j470-mac-node-join
root-upgrade-d376
j474-review-merge
latency-targets-120hz
nightly-suite-shards
gotatun-pin-j469
macos-utun-reader-thread
fix-flood-echo-stall
wip/j416
motion-fast-ease-out
mesh-addhost-j259
takeover-close-terminals-d368
kbd-instant-followups
fix-trust-key-double-click
nightly-suite-partial-report
appwin-icons-j207
rusqlite-slice1
swipe-content-first-d374
mesh-mtu-381
addhost-instant-open
glass-contrast-d365-mock
glass-contrast-d365
signout-followups-442
fix-startup-toast-red
research-gotatun-matrix-j317
themes-sync-j304
fix-load-flakes
fix-nix-zig-patch
fix-compute01-browser-baseline
ratchet-size-baseline-419
predict-auto-d130
revoke-coverage-d367
nightly-suite-queue-timeout
nightly-browser-suite
fix-rustsec-2026-0459
fonts-ioskeley-obel
log-retention-d375
fix-kbd-no-anim-427
fix-dev-red-specs-2
takeover-proof-305-j205
wip/j410-server-pump
ui-consistency-d372
fix-osc52-clipboard
libghostty-vt-1.0
forget-retired-host-d369
j404-merge-426
fix-setup-lock-race
fix-touch-select-426
revoke-access-report-ui
fix-predictive-echo-specs
fix-selinux-agent-exec
wip/j395-checkpoint2
wip/j395
j391-review-scratch
ui-consistency-j195-deferred-rebased
fix-rustc-paths-421
decisions-wording-341
fix-install-suite-red
fix-dev-rtt-specs
fix-pending-removal-align
fix-privileged-skip
fix-install-suite-404
review-j368
mesh-dns-j254
revoke-access-report-d356b
revoke-privilege-round5
fix-removal-confirm
review-j360
gotatun-rotation-367
deps-refresh-340
review-j354
review-J351
review-j350
addhost-takeover-d359
revoke-report-items-347
revoke-end-sessions-d362
review-j348
size-mesh-ssh-346
size-trim-337
revoke-access-diff-d356
j259-pre-rebase-b06c5a6
j305-ended-at-start-spec
addhost-sudo-consent
fix-layout-real-specs-411
fix-shell-open-411
fix-console-hosts-373
fix-shell-exit
fuzz-host-name-353
fix-host-details-410
macos-filter-j197
research-gotatun-tricks
revoke-p3-j216l
stale-state-pill
revoke-followups-j216k
revoke-progress-j216e
backup/macos01-20261106/lume/j197
backup/macos01-20261106/lume/appwin-icons-j207-stash0
backup/macos01-20261106/claude/stash-0
backup/macos01-20261106/claude/stash-1
backup/macos01-20261106/claude/J80-macos-s1
backup/macos01-20261106/claude/J80
backup/macos01-20261106/claude/J80-slice1-20261102
backup/macos01-20261106/claude/J16b-appwin-on-dev
backup/macos01-20261106/claude/appwin-launch-idle-146
backup/macos01-20261106/claude/J80-s1-run
wip/j259
trust-285-p3c
runtime-lock-j216f
mesh-ssh-p3-j175
tun-proof-p3b
trust-285-p3
terminal-polish-j271
tun-proof-p3
ssh-install-flakes-j216i
fuzz-fixes-j264
trust-285-a
trust-285-b
restart-oversized-fixture-j198
backup/macos01-20261106/lume/appwin-icons-j207
restart-bound-j216h
up-preflight-p3-j198
backup/macos01-20261106/lume/j251
up-preflight-j198
wip/up-preflight-j198-j245-p2
wip/up-preflight-j198-j245-p1
agent-fsync-offloop-j196
wip/up-preflight-j198-j245
settings-offscreen-j231
wip/up-preflight-j198-j226-inspection
wip/j103f
themes-j206
wip/up-preflight-j198-j226
flakes-j216c
backup/macos01-20261106/claude/j197
flakes-j216b
backup/macos01-20261106/claude/j197-red
wip/up-preflight-j198-review
flakes-j216
backup/macos01-20261106/claude/appwin-icons-j207
engineering-j208b
appwin-mock-j209
toast-stack-j165
engineering-j208
decisions-d351
decisions-d350
decisions-d349
appearance-presets-mock-j129
decisions-d348
docs-audit-spec
doc-coverage-baseline-raise
decisions-d347
decisions-d346b
decisions-d346
j88h-profile
ui-consistency-j195-deferred
ui-consistency-j195
engineering-practices-j194
preferred-login-ui-j166
signin-pools-263-j174
macos-node-265-s1
decisions-d345
decisions-d344
proxy-cgroup-j164
verify-quick-fuzz-329
proxy-start-io-j168
timing-flakes-j163
up-263-s1
mcp-events-j147
verify-quick-fuzz-j158
proxy-p3-j159
agent-resource-j157-detect
wip/j103d-final
remote-cancel-hang-j155
proxy-traefik-281-j62d
j149-verify-quick
proxy-traefik-281-j62b
j150-drop-unnamed
wip/j79d-red-r4
wip/j103d-d332
rootless-toast-j142
wip/j103d-head
wip/j103d
j125c-mcp
macos-build-j136
backup/macos01-20261106/claude/j136
wip/j79d-red-cli
wip/j79d-red-server
j121-link-liveness
j118c-tracer-failures
wip/j79c-head
release-desktop-j131
j124c-signed-macos
wip/j79c-red-p3
wip/j79c-red-p23
wip/j79c-red-p22
merge-j128-check
perf-deferred-j128
sidebar-toggle-260
j127-shortcut-changed-dot
j125-mcp
j118-tracer-failures
j126-account-summary
settings-sections-mock-j92
j120-pinch-lock
sidebar-292
backup/macos01-20261106/claude/j120
taste-2026-11-03
row-actions-242-215
backup/macos01-20261106/claude/stash-0-untracked
backup/macos01-20261106/claude/J80-265-s1-run
backup/macos01-20261106/claude/J80-macos-s1-20261102
j112-canvas-commands
ui-no-overshoot-scroll
fix-recovery-restart
backup/macos01-20261106/claude/stash-1-untracked
taste-2026-11-02
size-baseline-ratchet
machine-cards-structure-j108
j109-boot-copy-mockups
fix-dev-red
wip/up-263-s1
j95b-settings-rows
proxy-traefik-281
j95-option-rows
docs-human-j91
docs-keys-palette
fix-flood-1013-270-j38d
mesh-ssh-s1
mesh-ssh-s0
addhost-d291-tunnel
test-stability-load
mesh-relay-220
repro-296
shim-thp-off-24
mesh-pins-258-v2
mesh-dns
signin-d311-merged
docs-pages
ui-size-283
fix-ui-batch-specs
wip/j12b-edge-update
gate-on-arm
fix-rename-sync-293
trust-285-j60b
fix-focus-ring-clip
integ-ui-1101
edge-update-v2
signin-d311
taste-2026-11-01
layout-per-tab-250
ui-approved-splitopt-224
ui-approved-splitopt-224-on-dev
icon-text-centre-282
ui-approved-189-212-216
research-reverse-proxy
transfer-budget-320
review-mesh-pins-258-r2
sidebar-pill-259
review-mesh-relay-220
wip/j12-edge-update-v2
nix-package
status-crash-loop-237
fix-dev-test-failures
fix-fixture-socket-path
wip/mesh-relay-220-pre-dev-advance-rebase
fix-canvas-side-clip
research-head-to-head
frosted-real-256
review-mesh-pins-258
wip/mesh-relay-220-pre-final-rebase
review-layout-per-tab-250-r2
broad-suite-triage
nix-devshell
review-status-crash-loop-237-r3
wip/nix-devshell-pre-p3
taste-frosted-real
wip/code-serve-demo-2026-10-30
wip/code-serve-glass-2026-10-30
wip/code-serve-clusters-2026-10-30
wip/code-merge-release94-dev-2026-10-30
wip/rproxy-delta-scratch-2026-10-30
wip/stale-note-257-staged-2026-10-30
wip/d241-size-staged-2026-10-30
review-broad-suite-207
wip/mesh-pins-258-pre-split
wip/mesh-relay-220-pre-rebase
mesh-measure-220
plan-211-update
review-edge-autoupdate-r3
perf24-reconnect
review-stale-note-r2
release-prod-live
d241-shell-marker
review-mesh-pins-server-258
review-revocation-r3
review-setup-token-r2
review-sync-journal-r3
wip/review-mesh-pins-258-prune-review-1995fee46
frosted-scroll-256
sync-journal-r3
revocation-per-account-wip
review-tokio-asyncfd-0386
review-sync-p3s
wip/stash-8by64-stash0-2026-10-30
mesh-drag-r2
setup-token-data-wip
release-9-4
auth-busy-1013
mesh-view-overlap
sync-p3s
review-auth-busy-wake
review-readd-r8-wip
mesh-drag-bounds
audit-auth-93
app-windows-stream-start
review-authority-cap-r3
app-windows-r7-measure
readd-minimal-r6
readd-minimal-r5
app-windows-r7-soak
readd-minimal-r4
app-windows-r7
app-windows-r6
readd-minimal-r3
readd-minimal-r2
readd-minimal
merge-release8-dev
review-busy-page-liveness-r2
review-busy-page-liveness
mesh-view-trim
review-authority-read-retry-r2
review-remove-uninstall-r2
release-retry-reattach
release-retry
review-remove-uninstall
review-authority-read-retry
app-windows-ship
mesh-autoswitch-p2
mesh-ssh
mesh-relay
mesh-macos-node
screen-and-control
session-labels
review-real-logins-r5
review-real-logins-r4
review-real-logins-r3
shim-memory-r2
review-real-logins-r2
shim-memory
shadow-addhost-instant
shadow-real-logins
shadow-same-session-flood
shadow-prod-perf-fix
hosts-from-server
app-windows-real
app-windows-transport
release-prod-6w
mesh-policy-verify
quiet-close-toast
dev-merge-r4
review-mesh-agent-transport
review-app-windows-transport
settings-roundtrip
review-release-pipeline-r4
mesh-agent-transport
ended-strip-fix
review-audit-ca-update-d244
settings-rows
phone-width
signout-clear
fix-restart-tests
dev-merge-rp
signer-hardening
review-audit-ca-update
terminal-fidelity
review-audit-ca-update-wip
mesh-integrate
review-mesh-agent-transport-515d3712
review-mesh-dataplane-feec1988
mesh-dataplane
signout
research-canvas-node
wip/review-renderer-cost
wip/review-terminal-fidelity
padlock-glyph
review-release-pipeline-2e95b689
phone-polish
sidebar-polish
inbox-generic
review-mesh-dataplane-378d4055
prod-batch
ui-inbox
ui-ca-unlock
ui-machine-sections
review-mesh-dataplane-a5f744fb
tls-session-prod
session-bounds-prod
fix-dead-session
session-bounds
tls-session
palette-polish
shortcuts-d222
research-appmeta
phone-gutter
ui-centre-button
wip/ui-centre-button
wip/tls-session
review-addhost-breakit
ui-platform
renderer-cost
remote-agents
mesh-api
ssh-bootstrap
prod-hotfix2
prod-fixes
review-remote-agents
review-mesh-dataplane
session-life
research-browser
review-mesh-api
research-capture
review-mesh-api-round3
review-mesh-api-round1
gate-stability
review-gate-stability
review-relay-direct
review-mesh-api-findings
proto-sessions
font-picker
proto-addhost
proto-windows
mesh-core
latency-gate
research-bsd
research-discovery
keytray-mods
mock-core
ui-tweaks
fix-motion-sel
fix-motion
integrate2
fix-robust
t-web
t-server
vt-ghostty
fix-snap-echo
t-web-auth
review-vt2
t-agent
lane-protocol
lane-shim
lane-auth
s1-base
s1-contracts
s1-workspace
No results found.
No labels
launch-qa
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set
Reference
kayg/infraternal#65
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The motion-audit spec (apps/tracer/test/motion-audit.spec.ts) is flaky under CPU load: it timed out waiting for #copy-recovery data-done, and earlier runs failed other auth timing assertions. Make it deterministic: wait on state, not wall-clock assumptions; run it with the CPU-priority scope in the gate; and keep its real budgets.
Start: branch fix-socket from dev
ac03b1913b. I am addressing the four deadline/synchronization flakes requested by the orchestrator before the separate #51 socket fix; measuring baseline repetitions under host load and a CPUWeight scope.Finding: on infraternal-dev (x86_64, 8 cores; starting load 9.84/14.92/19.07), baseline review3_backend_reply_has_a_deadline failed on repetition 8 after 32,960 ms; repetitions 1–7 passed at about 60.5 s each. The test waits 61 s against the production 60 s terminal-open deadline. The remaining baseline repetitions are running.
Baseline finding on infraternal-dev (x86_64, 8 cores; load started 9.84/14.92/19.07 and ended 22.53/25.11/20.67): review3_backend_reply_has_a_deadline failed 10/20, review3_busy_expiry failed 13/20, and attached_backpressure_and_detached_progress failed 20/20 before reaching its assertion because the shim child reported ENOENT and its socket did not appear within 10 s. The initial review4 filter matched zero tests; those rows are discarded and I am rerunning with the exact module path.
Corrected review4 baseline (exact filter auth::tests::review3_tests::review4_ten_second_open_preserves_authority): 5/20 passed and 15/20 failed on infraternal-dev; every failure ended with 'server closed websocket unexpectedly'. Test elapsed 2.47–43.88 s (median 15.11 s), including two real 10 s waits in passing runs; load ended 38.28/32.71/25.38. The earlier zero-test rows are excluded.
Correction: the first direct shim-binary repetition used the worktree root as CWD, but crates/infraternal/tests/shim.rs resolves its fixture path from the package root. Its 20/20 ENOENT rows are invalid and must be ignored; I stopped that run and will repeat from crates/infraternal. The server-test baseline rows are unaffected.
Valid base measurements complete (infraternal-dev x86_64, 8 cores; load varied about 10–38): under load, backend deadline passed 10/20 (median 46.70 s; 10 failures), busy expiry 7/20 (13 failures), review4 ten-second open 5/20 (15 failures; median 15.11 s), and detached producer 7/20 (13 failures; failures hit the fixed 2 s output assertion). Under systemd-run --user --scope -p CPUWeight=10000: backend 19/20, busy expiry 20/20, review4 19/20 (median 22.00 s), detached producer 20/20 (median 2.53 s). All shim runs now used crates/infraternal as CWD; invalid wrong-CWD rows are discarded.
Finding: the PTY login shell changes cwd to /home/kayg, so the relative completion marker was never created; the output still reached sequence 1,490,028 (observed from the live shim socket). The test marker will use an absolute path under its session state.
Additional finding from the fast gate: review3_revoked_resume_must_not_welcome also has a 100ms close assertion. Before changing it, it passed 19/20 under ambient load and 5/20 under systemd-run --user --scope -p CPUWeight=10000 (8-core infraternal-dev; close event still arrives later). I am adding it to this determinism repair.
Determinism repair committed as
c5af3d1. On the 8-core x86_64 host, all five cases now pass 20/20 under ambient load and 20/20 under systemd-run CPUWeight=10000: backend deadline, busy expiry, review4 open/revocation, detached shim, and revoked Resume. Post medians under load: 3.016s, 0.994s, 2.217s, 2.295s, 0.525s; CPUWeight: 2.864s, 0.766s, 1.434s, 0.790s, 0.555s. Before counts and failure evidence are in the preceding comments and commit body.The successful combined-worktree gate ran as
systemd-run --user --scope --quiet --property=CPUWeight=10000 -- scripts/verify fast. Verbatim summary output:Final lane head is
fbe257906a; #65 changes are isolated inc5af3d1, followed by the requested #51 fix. All five flake cases pass 20/20 under ambient load and 20/20 under CPUWeight=10000; before/after counts and medians are recorded in the prior comment andc5af3d1.Final combined-tree gate command:
systemd-run --user --scope --quiet --property=CPUWeight=10000 -- scripts/verify fast(exit 0). Verbatim summary:Self-review complete: every changed deadline override is behind cfg(test); release code still uses the original handshake, authority-read, terminal-open and embedded-agent values. The shim synchronization is confined to the real-process test fixture.
Start: branch fix-reviewer-clock from dev
907e89e2cb. I read the #65 and #45 histories. I will make both expired-finish tests control expiry deterministically around writer contention, preserve production expiry behavior, and verify both tests under ambient and CPUWeight=10000 load plus a commit-time-check removal mutation.Finding: reviewer_tests.rs:184-197 samples SystemTime, then sets recovery expiry to now+1s before recovery-ready. HTTP uses the live global clock (http.rs:206-213), so scheduling/storage delay can correctly yield 401 before the test reaches its writer-wait assertion. #45 records this exact setup failure (recovery-ready expected 200, got 401); this host's load varied 22.53 to 15.70 on the 1-minute average. A baseline capability run passed 1/1 in 4.51s, confirming the failure window is intermittent. I will use a cfg(test) HTTP clock override while retaining the production clock and commit-time SQL predicates.
Mutation proof: temporarily removed
expires>MAX(?,unixepoch())from recovery::complete in crates/infraternal-server/src/auth/recovery.rs, leaving the rest unchanged. reviewer_http_expired_capability_cannot_complete_after_writer_wait then returned HTTP 200 (expected 401) and failed at reviewer_tests.rs:257 (0/1 passed, 3.30s; expiry-mutation.log). The pre-mutation recovery.rs is saved in job scratch and is being restored now.Step complete (uncommitted diff on head
907e89e2cb): added a cfg(test)-only shared HTTP clock and advanced expiry only after the existing writer wait. Both tests passed 20/20 under ambient load and 20/20 undersystemd-run --user --scope -p CPUWeight=10000(80/80 total). Ambient host infraternal-dev, 8 cores, load average start/end 2.30/8.35/12.61 and 4.04/7.72/11.72; CPUWeight scope start/end 3.27/7.24/11.46 and 1.49/5.20/10.12. Removing the statement-time recovery-expiry predicate makes the capability test return 200 and fail its expected-401 assertion (0/1). Running scripts/verify fast next.Fast-gate environment finding: the first scoped run exited 1 because the agent attach-credit fixture could not create relative
target/ac1206527(FileNotFoundError at scripts/verify-agent-attach-credit.py:54), and the musl wrappers could not findtarget/auth-tools/zig/zigwhile CARGO_TARGET_DIR pointed to scratch (verify-fast.log). This is the known target-path assumption recorded on #45; no source test failed in the recovery code. I will make the ignored worktreetargetan alias to the job scratch target and rerun.Finished: head
9e99820ff5(test: control recovery reviewer HTTP expiry clock (#65 #45)). The cfg(test) clock is opt-in and shared by cloned HTTP routers. The test keeps time before expiry while the writer is held, then advances it; production time, expiry checks and constants are unchanged. Auth self-review checklist B1/S1/S2/N1 is complete.Repetitions on infraternal-dev (8 cores): each test passed 20/20 under ambient load and 20/20 in
systemd-run --user --scope -p CPUWeight=10000(80/80 total). Ambient load average start/end: 2.30/8.35/12.61 and 4.04/7.72/11.72. CPUWeight scope start/end: 3.27/7.24/11.46 and 1.49/5.20/10.12.Mutation proof: removing
expires>MAX(?,unixepoch())fromrecovery::completemadereviewer_http_expired_capability_cannot_complete_after_writer_waitreturn HTTP 200 instead of 401; 0 passed / 1 failed. Mutation reverted.scripts/verify fastcommand:systemd-run --user --scope -p CPUWeight=10000 env CARGO_TARGET_DIR=$HOME/.cache/codex-scratch/fix-reviewer-clock/verify-target RUST_TEST_THREADS=1 TRACER_TEST_PORT=4865 scripts/verify fast— exit 0.Verbatim gate summary:
kayg referenced this issue2026-10-28 17:53:56 +00:00
Closed by the orchestrator during the 2026-11-09 night housekeeping (owner order: "close stale prs/issues"). The work has landed on dev, and the latest note here is a finish or merge note (J387 inventory on #436). Reopen if anything is still missing.