OPS: resize the build VM to 12 vCPU / 96 GB RAM in the maintenance window (pre-stop, handoff, resume jobs) #595

Open
opened 2026-10-01 08:01:30 +00:00 by kayg · 0 comments
Owner

Owner decision (2026-09-30 and 2026-10-01)

  • 2026-09-30: the build host is an 8-core VM that runs many Codex jobs, so builds and gates time out. The owner: "sadly this would require a poweroff of the build vm! remember last time? but i can, at max, give you 12 cores and 96 GB RAM as otherwise it will crash the proxmox host and other vms that run on the node."
  • 2026-10-01: do it "whenever we have the least amount of jobs running (maybe after the next deploy?)", in the same maintenance window as the git history purge.

Procedure

  1. Before power-off (Claude): pause new jobs; run the pre-stop script (record running jobs, stop them cleanly, push every branch and a snapshot of uncommitted work); write a handoff note to memory.
  2. Owner: power off the build VM, set 12 vCPU and 96 GB RAM (not more, to protect the Proxmox host), and boot.
  3. After boot (Claude): resume the jobs from the handoff note. Raise the job cap from 14 to about 20–22, and re-check load, memory and disk alerts.

Acceptance

  • nproc reports 12 and free -g reports about 96 GB on the build host.
  • Every job that ran before the stop resumes with no lost commits.
  • One merge round runs its full gates without a timeout.

Related: the git history purge issue (same window), the job cap rule in the codex-jobs scripts.

## Owner decision (2026-09-30 and 2026-10-01) - 2026-09-30: the build host is an 8-core VM that runs many Codex jobs, so builds and gates time out. The owner: "sadly this would require a poweroff of the build vm! remember last time? but i can, at max, give you 12 cores and 96 GB RAM as otherwise it will crash the proxmox host and other vms that run on the node." - 2026-10-01: do it "whenever we have the least amount of jobs running (maybe after the next deploy?)", in the same maintenance window as the git history purge. ## Procedure 1. **Before power-off (Claude):** pause new jobs; run the pre-stop script (record running jobs, stop them cleanly, push every branch and a snapshot of uncommitted work); write a handoff note to memory. 2. **Owner:** power off the build VM, set **12 vCPU and 96 GB RAM** (not more, to protect the Proxmox host), and boot. 3. **After boot (Claude):** resume the jobs from the handoff note. Raise the job cap from 14 to about 20–22, and re-check load, memory and disk alerts. ## Acceptance - `nproc` reports 12 and `free -g` reports about 96 GB on the build host. - Every job that ran before the stop resumes with no lost commits. - One merge round runs its full gates without a timeout. Related: the git history purge issue (same window), the job cap rule in the codex-jobs scripts.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal#595
No description provided.