Remove unnecessary authorization fallback polling while preserving deadlines #86

Open

opened 2026-10-04 18:56:23 +00:00 by kayg · 0 comments

kayg commented

2026-10-04 18:56:23 +00:00

Owner

Problem and evidence

A 60-second wakeup trace counted 7,136 server wakeups (~119/s) and 921 standalone infraternal agent wakeups (~15.4/s). Idle CPU over 62.31 s was 1.04% server and 0.24% standalone infraternal agent; no file-write-byte increase. Load1 ranged 6.43–9.61. Current authorization scheduling has commit notifications plus a fixed 100 ms fallback poll (crates/server/src/stream/revalidation.rs:130-133, 183-235).

The wakeup trace is idle-wakeups.trace; CPU/load summary is idle-summary.json. Changing server workers from eight to two saved about 1.7–2 MiB RSS but did not reduce CPU or wakeups; the standalone runtime already uses two workers.

Recommendation

Replace fixed fallback polling only where commit notifications and bounded expiry timers provide the required freshness. Consolidate freshness-bounded authorization reads while retaining close deadlines and fail-closed behavior. Measure each timer removal separately.

Payoff, effort and risk

Expected gain: target 2–4× fewer idle wakeups and server idle mean CPU at or below 0.5%.
Quick win vs deeper change: identify the wake sources and map each timer to its freshness/deadline contract in ≤½ day; remove unnecessary polls and retest liveness in 1–2 days.
Risk: medium; missed notifications or expiry reads could delay authorization loss or leave stale state.

Priority: Next milestone.

2026-10-04 owner decisions: Matches event-driven server caches and the single shared WebSocket event route. Keep explicit freshness and revocation bounds. Do not make worker-count changes as a CPU optimization. Related: #67 for the shared authorization model; keep that scope coordinated rather than duplicating it.

Report: perf-audit report
Source: perf-audit 2026-10-04

## Problem and evidence A 60-second wakeup trace counted 7,136 server wakeups (~119/s) and 921 standalone infraternal agent wakeups (~15.4/s). Idle CPU over 62.31 s was 1.04% server and 0.24% standalone infraternal agent; no file-write-byte increase. Load1 ranged 6.43–9.61. Current authorization scheduling has commit notifications plus a fixed 100 ms fallback poll (crates/server/src/stream/revalidation.rs:130-133, 183-235). The wakeup trace is [idle-wakeups.trace](/mnt/hdd/infraternal/jobs/perf-audit/idle-wakeups.trace); CPU/load summary is [idle-summary.json](/mnt/hdd/infraternal/jobs/perf-audit/idle-summary.json). Changing server workers from eight to two saved about 1.7–2 MiB RSS but did not reduce CPU or wakeups; the standalone runtime already uses two workers. ## Recommendation Replace fixed fallback polling only where commit notifications and bounded expiry timers provide the required freshness. Consolidate freshness-bounded authorization reads while retaining close deadlines and fail-closed behavior. Measure each timer removal separately. ## Payoff, effort and risk **Expected gain:** target 2–4× fewer idle wakeups and server idle mean CPU at or below 0.5%. **Quick win vs deeper change:** identify the wake sources and map each timer to its freshness/deadline contract in ≤½ day; remove unnecessary polls and retest liveness in 1–2 days. **Risk:** medium; missed notifications or expiry reads could delay authorization loss or leave stale state. **Priority:** Next milestone. **2026-10-04 owner decisions:** Matches event-driven server caches and the single shared WebSocket event route. Keep explicit freshness and revocation bounds. Do not make worker-count changes as a CPU optimization. Related: #67 for the shared authorization model; keep that scope coordinated rather than duplicating it. **Report:** [perf-audit report](/mnt/hdd/infraternal/jobs/perf-audit/report.md) Source: perf-audit 2026-10-04

kayg referenced this issue

2026-10-04 19:04:03 +00:00

Performance audit: release end-to-end hot spots at d1eefa9 #51

Commenting is not possible because the repository is archived.

No labels

No milestone

No project

No assignees

1 participant

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kayg/infraternal-v0#86

No description provided.

Rows
Columns