Cache normalized container search text per inventory generation #79
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem and evidence
The audit counted 36,054,590 allocation calls in the final heap profile. Container-summary filtering accounted for 20,229,060 (56.11%); constructing normalized search text accounted for 6,622,967. The named full-list CPU profile attributed 26.09% to filtering and 9.94% to search (malloc 11.18% inclusive), with 614 rows at load1 26.83–27.60. The separate 500-container real measurement recorded list p99 of 22.576, 23.581 and 21.070 ms against the 20 ms plan limit; load1 fell 20.59→17.95.
Current code builds a fresh lowercase String for each candidate row and collects cloned matching summaries (crates/server/src/resources.rs:2490-2549). The relevant profiles are named-list200.svg and heap-final.svg; the 500-resource runs are in audit-results.json.
Recommendation
Cache normalized searchable text alongside each immutable inventory generation. Keep filtering borrowed rows through predicate evaluation and avoid cloning all matches before response serialization where the response path permits it. Preserve exact case-folding, label order, filter semantics and bounded cache lifetime.
Payoff, effort and risk
Expected gain: remove the observed 6.62M search allocations for this workload and target roughly 15–25% lower full-list CPU; improve the 500-resource p99 while building toward 500 hosts / 20,000 containers feeling instant.
Quick win vs deeper change: generation-scoped search-text caching is estimated at ≤½ day; borrowed serialization is a deeper 1–2 day change.
Risk: medium; stale cache keys or changed normalization could alter query results. Cache memory must stay within inventory bounds.
Priority: Now (≤1 day for the cache quick win).
2026-10-04 owner decisions: Matches the 500-host / 20,000-container scale target, event-driven server cache and code/tests-as-spec. Tie future list optimizations to that workload, not only a 500-container fixture. Related: #57 for generic resource list routing.
Report: perf-audit report
Source: perf-audit 2026-10-04
Starting web-perf-now on branch
web-perf-now, base head069059d31fcd91366e76072e6be4871c01e5cde6. I am reading the three issue scopes and existing perf evidence, then will implement the precompressed asset serving, cold-route JS reductions, and normalized container search cache in this worktree.#79 progress (branch web-perf-now, base
069059d)Container search text is now normalized once per immutable inventory generation and reused for nonempty text queries; the existing row order and predicates are preserved. Empty text queries do not build the cache, and conditional 304 responses still bypass filtering. The cached and uncached test outputs match, and the fast gate passed including the workspace tests. The benchmark covers cold index construction and cached/uncached filtering at 500 and 5,000 rows; I am not claiming an A/B performance result because the perf-test-vm measurement was not run.
#79 finished
Branch: web-perf-now
Base:
069059d31fHead:
52fe3ae1daBefore, the audit profile attributed 6,622,967 allocation calls to normalized search-text construction (20,229,060 total container-summary filtering allocations). After, normalized search text is built once per row when an immutable generation is first queried with nonempty text and reused for subsequent queries; no post-change perf-test-VM A/B was run, so there is no measured after number. Cache lifetime follows the generation. Empty text does not build it; conditional 304 responses bypass filtering. The unit test compares cached and uncached results and checks stable cached storage. Criterion cases cover index construction plus cached/uncached filtering at 500 and 5,000 rows.
scripts/verify fastexited 0. Verbatim output:The separately run browser gate recorded passing cold-route budgets but failed later on /api/v1/containers 503/401 console errors under high VM load. No performance A/B claim is made.
Correction: the browser retry against committed head
52fe3ae1daagain passed cold-route budgets, then failed at the light/desktop page-clean assertion with /api/v1/containers 503/401 console errors:FAIL browser: Playwright against release server (318.534s). The cause is not established, so I am not attributing it solely to load. The cache change itself is covered by the passing workspace tests; no after A/B measurement is claimed.