kayg/calternal

Watch

Fork 0

PERF: bound the collaboration filesystem event queue (#663) #761

New issue

Open

opened 2026-10-02 13:09:51 +00:00 by kayg · 4 comments

kayg commented

2026-10-02 13:09:51 +00:00

Owner

Parent audit: #663. Baseline: c4a61e8cf0; unchanged in round 7a.

M1: Collaboration watcher has no queue limit

Status: source-confirmed performance finding. No crash or security exploit was reproduced.

Evidence: crates/calternal-collab/src/session.rs:699 uses
mpsc::unbounded_channel and watches all Homes recursively. Filtering starts
in the consumer, after allocation and enqueue. Reads, non-Note files and
errors can enter the queue. The consumer awaits external_change for each
Note event. The same queue remains in job/merge-round-7a.

Impact: allocations grow with the difference between event arrival and
processing rates. Estimate: 100,000 pending events at 200–500 bytes per event
need 19–48 MiB before allocator overhead. Multiple paths increase the cost.
There is no configured upper limit.

Fix: filter change events before enqueue; coalesce Note identities in a
bounded queue; set one reconciliation flag on overflow. Do not drop the only
external change to an active room. Reuse the Search overflow pattern.

Test: a deterministic local producer and paused consumer must keep slots
and bytes below the cap. After overflow, the active room must converge to
the final file contents. Include unrelated file and read events.

Duplicate search: all-state collaboration, unbounded, memory and idle.
#325 concerns Search watcher loss, not this collaboration queue.

Parent audit: #663. Baseline: c4a61e8cf090170f35b1bed3350d9de20c83ecd5; unchanged in round 7a. M1: Collaboration watcher has no queue limit Status: source-confirmed performance finding. No crash or security exploit was reproduced. Evidence: crates/calternal-collab/src/session.rs:699 uses mpsc::unbounded_channel and watches all Homes recursively. Filtering starts in the consumer, after allocation and enqueue. Reads, non-Note files and errors can enter the queue. The consumer awaits external_change for each Note event. The same queue remains in job/merge-round-7a. Impact: allocations grow with the difference between event arrival and processing rates. Estimate: 100,000 pending events at 200–500 bytes per event need 19–48 MiB before allocator overhead. Multiple paths increase the cost. There is no configured upper limit. Fix: filter change events before enqueue; coalesce Note identities in a bounded queue; set one reconciliation flag on overflow. Do not drop the only external change to an active room. Reuse the Search overflow pattern. Test: a deterministic local producer and paused consumer must keep slots and bytes below the cap. After overflow, the active room must converge to the final file contents. Include unrelated file and read events. Duplicate search: all-state collaboration, unbounded, memory and idle. #325 concerns Search watcher loss, not this collaboration queue.

kayg referenced this issue

2026-10-02 13:11:32 +00:00

PERF: use one resettable flush task per collaboration room (#663) #778

kayg commented

2026-10-02 13:11:35 +00:00

Author

Owner

Confirmed #761 in the sync architecture audit at round-7a 2f4482ded. Companion findings: #769 targets external_change to the changed room before reading source files (currently one read per loaded room); #778 bounds per-frame persistence timers to one task per room. Keep #761 as the sole watcher-ingress owner and combine its dirty-set overflow recovery with #769 identity routing. No hostile load or crash was reproduced here.

Confirmed #761 in the sync architecture audit at round-7a 2f4482ded. Companion findings: #769 targets external_change to the changed room before reading source files (currently one read per loaded room); #778 bounds per-frame persistence timers to one task per room. Keep #761 as the sole watcher-ingress owner and combine its dirty-set overflow recovery with #769 identity routing. No hostile load or crash was reproduced here.

kayg referenced this issue

2026-10-02 13:13:24 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg referenced this issue

2026-10-02 13:16:50 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg referenced this issue

2026-10-02 13:19:27 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg referenced this issue

2026-10-02 13:22:10 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg commented

2026-10-02 13:25:36 +00:00

Author

Owner

Starting work on job/notesperf, based on job/merge-round-7a at 2f4482ded066d9c5d9c59130377907f7fd2916c9. I will address the issue with focused changes and regression coverage, then report the final head SHA and verbatim gate output here.

Starting work on `job/notesperf`, based on `job/merge-round-7a` at `2f4482ded066d9c5d9c59130377907f7fd2916c9`. I will address the issue with focused changes and regression coverage, then report the final head SHA and verbatim gate output here.

kayg commented

2026-10-02 18:26:44 +00:00

Author

Owner

Finding from review: the watcher coalesced a duplicate path while its source read was already in flight. If another write landed after that read began, the room could stay stale until a later event. The bounded inbox now turns that specific in-flight duplicate into one room reconciliation sweep; the new regression test covers the race. I am running the Collab crate tests before the commit.

kayg commented

2026-10-02 19:10:46 +00:00

Author

Owner

The external-change ingress is now bounded to 256 unique Note hints and 64 KiB of path bytes, filters non-Notes events before enqueue, coalesces duplicate paths, and sweeps active rooms on overflow. A duplicate arriving during an in-flight source read also schedules one follow-up sweep. Tests passed: cargo test -p calternal-collab --lib external_change_queue_tests (4 passed). Commit: 750b36129. Final gates and one performance profile are pending.

The external-change ingress is now bounded to 256 unique Note hints and 64 KiB of path bytes, filters non-Notes events before enqueue, coalesces duplicate paths, and sweeps active rooms on overflow. A duplicate arriving during an in-flight source read also schedules one follow-up sweep. Tests passed: `cargo test -p calternal-collab --lib external_change_queue_tests` (4 passed). Commit: 750b36129. Final gates and one performance profile are pending.

kayg referenced this issue

2026-10-02 21:09:16 +00:00

PERF: mount Notes before optional syntax grammars load (#663) #747

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kayg/calternal#761

No description provided.

Rows
Columns