Keep vector queries available during per-User migration #456

Open
opened 2026-09-29 13:44:26 +00:00 by kayg · 0 comments
Owner

Follow-up from #435. The per-User vector copy is idempotent and verifies counts before shared rows are deleted. Its current lazy opener waits for a User's copy and cleanup before returning a read handle. A large first-open migration can stall that User's Search, and the shared cache lock can stall other Users. DESIGN §48 requires Search to continue reading the old store until each new file is verified.

Acceptance: migrate each User with a per-User single-flight task; serve reads from the old store while the new file is unverified; atomically switch that User to the verified file; only then clean old rows. Bound open files, migration memory, and per-User work. On restart at each copy, marker and cleanup step, queries return complete old or new results, never a partial set. Test concurrent A/B queries during a large migration and a request storm with one User stalled. Keep the negative owner-filter control.

Follow-up from #435. The per-User vector copy is idempotent and verifies counts before shared rows are deleted. Its current lazy opener waits for a User's copy and cleanup before returning a read handle. A large first-open migration can stall that User's Search, and the shared cache lock can stall other Users. DESIGN §48 requires Search to continue reading the old store until each new file is verified. Acceptance: migrate each User with a per-User single-flight task; serve reads from the old store while the new file is unverified; atomically switch that User to the verified file; only then clean old rows. Bound open files, migration memory, and per-User work. On restart at each copy, marker and cleanup step, queries return complete old or new results, never a partial set. Test concurrent A/B queries during a large migration and a request storm with one User stalled. Keep the negative owner-filter control.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal#456
No description provided.