kayg/hermternal-apple
Watch
1
0
Fork
You've already forked hermternal-apple
 0
Code Issues 44 Pull requests Projects Releases 4 Packages Wiki Activity Actions

Add deterministic performance regression harness and hitch profiling lane #14

New issue
Open
opened 2026-08-21 09:01:12 +00:00 by kayg · 1 comment
kayg commented 2026-08-21 09:01:12 +00:00
Owner
Copy link

Goal

Add deterministic performance regression coverage for Hermternal's local hot paths, plus a scheduled native UI-hitch lane and repeatable Instruments review.

Performance is priority one, but network/auth/server timing and arbitrary UI wall-clock thresholds are too noisy for per-commit gates.

Architecture first

Extract a Foundation-focused HermternalCore library target and add HermternalCoreTests; keep the executable target and Scripts/build-app.sh unchanged.

Inject or extract seams for:

  • History cache directory/filesystem/codec.
  • TranscriptSource and bounded prefetch coordinator.
  • Cache-first session opener/reducer with generation control.
  • Markdown segment parser.
  • Pure gateway NDJSON frame parser and streaming accumulator.

Per-commit deterministic contracts

Fixed local fixtures, no SSO/network/live Hermes:

  1. Cold and warm cache load for 500 messages; warm hit performs no disk decode.
  2. Reconcile 30 session IDs with valid/stale/corrupt/old-version entries; exact count/bytes and pruning.
  3. Prefetch 30 sessions; maximum four in flight, once each, cancellation/generation prevents stale counter or state updates.
  4. Cached-open race: delayed session A resume cannot overwrite later selected session B.
  5. Large Markdown corpus including fenced/unterminated code; correct output without per-delta parsing.
  6. 10k NDJSON deltas plus malformed/final frames; exact ordering, no loss/duplication, authoritative completion text.

Use correctness assertions as hard gates. Where native macOS 26 SDK support compiles, collect XCTest clock/CPU/memory/storage metrics as controlled-run trends, not host-independent fixed limits. Run measurements in Release with coverage/sanitizers disabled and fixture setup outside the measured region.

Nightly / controlled Mac lane

Add deterministic app fixture/launch mode and semantic accessibility identifiers. Use XCUITest for functional launch/sidebar/chat flows and Xcode 26 XCTHitchMetric(application:) around fixed sidebar/transcript scrolling. Baseline only on controlled hardware; reset gestures outside the measured interval. Do not use sleeps or generic XCUITest elapsed time as performance gates.

Release profiling

Add stable OSSignposter intervals for cache reconcile/hit/open, prefetch, gateway ingest, Markdown completion parse, and transcript first paint. Capture Xcode 26 Instruments SwiftUI + Time Profiler/Hangs & Hitches/Allocations/File Activity/Points of Interest scenarios:

  • 500-message cold vs warm open and rapid 30-chat switching.
  • Long Markdown streaming/autoscroll.
  • Sidebar scrolling during four-lane prefetch.

Attach xctrace artifacts for diagnosis; do not use Instruments sampling output as a brittle per-commit pass/fail gate. MetricKit on macOS 26 is daily/field observation, not CI.

Official references

Acceptance

The deterministic suite fails on plausible cache/race/parser regressions, the scheduled lane records repeatable hitch trends on one controlled Mac, and release traces isolate each named hot path without requiring live credentials or a live agent.

## Goal Add deterministic performance regression coverage for Hermternal's local hot paths, plus a scheduled native UI-hitch lane and repeatable Instruments review. Performance is priority one, but network/auth/server timing and arbitrary UI wall-clock thresholds are too noisy for per-commit gates. ## Architecture first Extract a Foundation-focused `HermternalCore` library target and add `HermternalCoreTests`; keep the executable target and `Scripts/build-app.sh` unchanged. Inject or extract seams for: - History cache directory/filesystem/codec. - `TranscriptSource` and bounded prefetch coordinator. - Cache-first session opener/reducer with generation control. - Markdown segment parser. - Pure gateway NDJSON frame parser and streaming accumulator. ## Per-commit deterministic contracts Fixed local fixtures, no SSO/network/live Hermes: 1. Cold and warm cache load for 500 messages; warm hit performs no disk decode. 2. Reconcile 30 session IDs with valid/stale/corrupt/old-version entries; exact count/bytes and pruning. 3. Prefetch 30 sessions; maximum four in flight, once each, cancellation/generation prevents stale counter or state updates. 4. Cached-open race: delayed session A resume cannot overwrite later selected session B. 5. Large Markdown corpus including fenced/unterminated code; correct output without per-delta parsing. 6. 10k NDJSON deltas plus malformed/final frames; exact ordering, no loss/duplication, authoritative completion text. Use correctness assertions as hard gates. Where native macOS 26 SDK support compiles, collect XCTest clock/CPU/memory/storage metrics as controlled-run trends, not host-independent fixed limits. Run measurements in Release with coverage/sanitizers disabled and fixture setup outside the measured region. ## Nightly / controlled Mac lane Add deterministic app fixture/launch mode and semantic accessibility identifiers. Use XCUITest for functional launch/sidebar/chat flows and Xcode 26 `XCTHitchMetric(application:)` around fixed sidebar/transcript scrolling. Baseline only on controlled hardware; reset gestures outside the measured interval. Do not use sleeps or generic XCUITest elapsed time as performance gates. ## Release profiling Add stable `OSSignposter` intervals for cache reconcile/hit/open, prefetch, gateway ingest, Markdown completion parse, and transcript first paint. Capture Xcode 26 Instruments SwiftUI + Time Profiler/Hangs & Hitches/Allocations/File Activity/Points of Interest scenarios: - 500-message cold vs warm open and rapid 30-chat switching. - Long Markdown streaming/autoscroll. - Sidebar scrolling during four-lane prefetch. Attach `xctrace` artifacts for diagnosis; do not use Instruments sampling output as a brittle per-commit pass/fail gate. MetricKit on macOS 26 is daily/field observation, not CI. ## Official references - Writing/running performance tests: https://developer.apple.com/documentation/xcode/writing-and-running-performance-tests - XCTHitchMetric: https://developer.apple.com/documentation/xctest/xcthitchmetric - WWDC25 What's new in Xcode 26 (hitch metric example): https://developer.apple.com/videos/play/wwdc2025/247/ - Optimize SwiftUI performance with Instruments: https://developer.apple.com/videos/play/wwdc2025/306/ - Understanding hitches: https://developer.apple.com/documentation/xcode/understanding-hitches-in-your-app - Improving app performance: https://developer.apple.com/documentation/xcode/improving-your-app-s-performance - XCTOSSignpostMetric: https://developer.apple.com/documentation/xctest/xctossignpostmetric ## Acceptance The deterministic suite fails on plausible cache/race/parser regressions, the scheduled lane records repeatable hitch trends on one controlled Mac, and release traces isolate each named hot path without requiring live credentials or a live agent.
kayg commented 2026-08-22 04:23:52 +00:00
Author
Owner
Copy link

Status update for architecture planning: a HermternalTests test target was added in commit b944320, targeting the executable target directly. This may conflict with the plan here to extract a HermternalCore library with HermternalCoreTests; reconcile that target structure when #14 is picked up.

The extraction is now intended to happen BEFORE the search-index work, because the index needs the same seams listed here. The six identity tests currently in HermternalTests should move to HermternalCoreTests as part of that extraction.

Status update for architecture planning: a `HermternalTests` test target was added in commit `b944320`, targeting the executable target directly. This may conflict with the plan here to extract a `HermternalCore` library with `HermternalCoreTests`; reconcile that target structure when #14 is picked up. The extraction is now intended to happen BEFORE the search-index work, because the index needs the same seams listed here. The six identity tests currently in `HermternalTests` should move to `HermternalCoreTests` as part of that extraction.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
dev
feat/sidebar-selection-archive-purge
v0.0.4
v0.0.3
v0.0.2
v0.0.1
Labels
Clear labels   
delivered
Shipped and recorded in project history

  
outstanding
Not yet delivered or blocked

No labels
delivered
outstanding
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/hermternal-apple#14
No description provided.