kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 191 Pull requests 61 Projects Releases Packages Wiki Activity Actions 20

Plan DAV capture/export/replay v3 #818

Open
kayg wants to merge 1 commit from docs/dav-recorder-v2-plan into dev
pull from: docs/dav-recorder-v2-plan
merge into: kayg:dev
Conversation 5 Commits 1 Files changed 1 +101
kayg commented 2026-10-31 21:56:28 +00:00
Owner
Copy link

DAV capture/export/replay plan v3, revised

Design only. This description supersedes docs/plans/0064-dav-recorder-v2.md and the earlier v3 description. Architecture accepted; revised against plan review 72220. #820 remains paused at 58e503944542a8a84b7e9a6a4389433a877fa099 under the three-round rule. No implementation, capture or build is authorized by this design task. Future PRs target dev; never merge here.

Goal and architecture

Capture Apple Contacts ↔ CardDAV traffic with stock Nix mitmdump, export a narrow credential-free typed fixture offline, then reuse #820's Rust replay/comparison. No custom proxy or inline validator. Protocol-valid unknown responses must reach the client unchanged and remain in raw flows; schema errors can only refuse export, retaining raw evidence. Disk, network, TLS and proxy defects remain operational failures: “no schema-induced abort” is not a claim that recording cannot fail.

Evidence: #820 reviews round 1, round 2, round 3. Round 3 caught 23/25 replay differences, yet ordinary responses became 502s and saved fixtures retained Apple auth, Digest nonces and Link URL credentials. Repatching that recorder or building another HTTP proxy repeats the wrong boundary. The small Python exporter uses upstream FlowReader/FlowWriter, not a replacement flow decoder. Reuse existing typed Rust comparison rather than redesigning it.

First PR scope and sizes

Only contacts-crud and contacts-initial-sync, Basic app-password authentication, sequential synthetic workloads. Prove on Linux first. Actual Apple profile coverage, Mac GUI access and real-iPhone requirements remain separate PENDING gates.

Component Files/interfaces Review budget
Capture/finalization README and scripts/capture-finalize.py: workload wrapper, final scripted request, graceful stop and private receipt 70–100 Python lines plus 50–70 documentation/Nix lines; zero proxy addons
Bounded raw reader export/reader.py: file/record preflight and containment; upstream FlowReader -> bounded flows 80–110 Python lines; no new flow format/decoder
Closed conversion export/policy.py, export/bodies.py, export/export.py: Basic/URI/header policy, small card/XML adapters, CLI/atomic writer 280–340 Python lines combined; split files above about 200 lines
Typed Rust adaptation src/schema.rs, wire.rs, fixture.rs, main.rs, retain replay.rs 100–180 changed Rust lines: remove raw fallback, v3 strict loading, narrow discovery/error variants/auth state; reuse comparator
Gates export/tests/test_export.py, tests/export_contract.rs, tests/review_v3_mutations.rs, focused existing suites 300–400 new/adapted test lines, table-driven historical cases; reuse existing probe/harness code

Total new Python production budget 430–550 lines, plus 100–180 changed Rust lines, 300–400 test lines, 50–70 documentation/Nix lines. These are estimates, not counts or acceptance ceilings. Keep correctness gates if implementation exceeds a budget; revisit decomposition rather than add fallbacks. Nix/dependency closure sizes are not part of these source-line estimates.

export --capture PATH --receipt PATH --profile contacts-v1 --scenario ENUM --macos X.Y.Z --out PATH produces format 3 JSON for the Rust loader. Fixed metadata enums only; CLI free text cannot enter the fixture. Exit 0 only after complete staged output passes the strict Rust loader, then no-overwrite atomic publication. Export refusal exits 2 with fixed error code, exchange index and schema position, never source values/names/paths/tracebacks. Existing inventory check exit 3 remains PENDING until real required captures exist. Reject format 2 and disable/remove old unsafe record; never silently migrate raw fields.

Capture and observable completion

Raw flows and their receipt stay outside all checkouts on the originating VM, directory 0700/files 0600, umask 077. Raw includes secrets: never commit, upload to issues/CI, print, or copy to a repository staging directory. No mapping sidecar or source-valued diagnostics. Use a disposable synthetic account and target-owned credentials read from files, never argv. Stock mitmdump only, configured to retain bodies and avoid host/redirect rewriting. No old schema/body cap on capture. Configure temporary TLS trust only in the disposable guest if required; no production changes.

A clean EOF or sufficient scenario milestones does not establish completion. The private finalizer owns the scripted workload/proxy processes and follows this protocol:

  1. Run the requested workload successfully and wait for all its requests to complete. Stop background clients; require a quiescent, sequential scenario. For a future manual Apple capture, explicitly stop further app traffic before finalization; unverifiable quiescence refuses publication.
  2. Send exactly one final sentinel request through the same proxy to the disposable origin: GET /__dav_capture_complete__/<fresh-random-nonce>, no auth/query/fragment/body. Expect a completed 404 from the unmodified own-server router. This reserved test path needs no product endpoint. Unexpected status, retry, missing response or further scenario traffic prevents finalization. The nonce distinguishes this run; it is not a fixture identifier.
  3. Gracefully stop only this mitmdump process, wait for its successful exit and closed writer. Fail on unsuccessful exit or timeout. Flush the closed raw file to stable storage; do not create a receipt while the proxy may still append.
  4. Atomically create a small private receipt with fixed version/scenario, expected nonce/status, exact raw byte length and SHA-256, workload success and proxy exit status. Receipt is completion evidence from the trusted local workload wrapper, not an unforgeable proof of user actions. Never produce one after a failed workload/stop. The hash and nonce are local control data and never enter fixtures/logs/issues.
  5. Export requires both the receipt and a byte-matching raw file, exactly one complete matching sentinel as the final HTTP flow, no error/incomplete flows, unambiguous ordering and the scenario's required milestones. Remove the sentinel from exported exchanges. Reading EOF alone is never enough. Reject missing/malformed/stale receipt, missing/duplicate/mismatched marker, traffic after marker or changed raw bytes. Retain raw evidence on refusal.

This detects a readable prefix cut between complete records, including after apparent scenario coverage but before finalization. It does not turn an arbitrary receipt into proof of actual Contacts behavior. Once a real fixture is accepted and replay evidence recorded, remove local raw/receipt, disposable account and proxy CA material; accurately report deletion without promising storage-level erasure.

Resource boundary before FlowReader

The review identified that mitmproxy 12.2.3 FlowReader calls tnetstring.load, which allocates the record before semantic checks, and can auto-load HAR. Do not claim the 4 MiB body check bounds that allocation.

Concrete importer contract, implemented around the upstream decoder:

  • Open only a regular raw file (no symlink/device/FIFO), hold that descriptor, fstat it and reject length above 64 MiB before FlowReader or any whole-file load. Receipt length must match; hash the same descriptor in bounded chunks. No unbounded read() or mmap. Limit receipt to 4 KiB before parsing.
  • Pre-scan top-level native tnetstring framing in fixed-size reads: bounded decimal length prefix (at most nine digits), colon, payload within 12 MiB per record, final expected native dictionary tag. Verify every length against remaining file bytes; reject partial prefix/payload/tag, trailing garbage, more than 10,000 records, and every HAR/JSON input. Seek/skip bounded payloads, never allocate advertised lengths during this pass. Do not decode nested values or duplicate upstream serialization logic.
  • Feed upstream FlowReader one preflight-bounded record view at a time; cap every underlying read at that record's remaining bytes and require exact consumption. Its HAR autodetection cannot consume beyond a record and HAR was already refused. No generator expression that loads all flows. Re-fstat and recheck raw digest before publication so concurrent mutation invalidates the receipt; even changed bytes cannot bypass read caps.
  • Record size caps bound FlowReader's raw allocation, not its Python object amplification or recursion. Run importer/decoder in a child with 1 GiB address-space cap, 60 seconds wall time, sanitized error handling and no core dump. Linux VM gate must demonstrate these limits actually apply (e.g. inherited RLIMIT_AS and a parent wall deadline). Process exit/signal/decoder exception yields a fixed refusal, no traceback or partially published fixture. Receipt/hash/preflight work also runs within these limits. A smaller cap can be chosen only with measured supported-baseline headroom.
  • After decoding one bounded flow: at most 64 KiB aggregate raw request+response header bytes, 4 MiB decoded body each, nesting 32, 100,000 slots, 16 MiB aggregate slot-key bytes for the session. Reject metadata-heavy and excessive-header flows even if their framed record is under 12 MiB.
  • Never access mitmproxy's auto-decoding .content/.text before checking content encoding. Support identity/gzip only; incrementally decompress raw content in bounded chunks with a 4 MiB output ceiling and reject overrun, incomplete stream, unsupported coding or bad data. Do not materialize expansion before checking it. Semantic headers/trailers cannot be silently hidden as framing.

Costs: preflight+hash are O(raw bytes), conversion O(B log I) with ordered slot maps, memory O(one bounded decoded flow + bounded slot keys + upstream decoder object overhead). The process cap contains that overhead; exceeding it is an explicit refusal, not a claim of exact Python allocation accounting. Staged files contain only new typed objects; never serialize Flow, Flow state, exception or arbitrary dictionaries from input.

Closed allowlist, contacts-v1

Default for every unlisted request/response header, URI feature, XML QName/attribute, card property/parameter and body variant: refuse offline export. Never copy an unknown name or infer that an unlisted request header is irrelevant. No Raw/Other strings, name substrings, hashes of captured secrets, whole-header opaque fallback or source-to-slot map in output. Every retained scalar is a closed grammar value or a typed slot assigned by first-seen identity. Preserve repeated identity within its semantic domain; distinct UID identities cannot collapse.

Field First-PR treatment and reason
Session/method/status Fixed profile/scenario/app/version fields; supported request methods OPTIONS, GET, HEAD, PROPFIND, REPORT, PUT, DELETE. Numeric HTTP status only, never reason phrase. Advertised Allow may also include MKCOL, MKCALENDAR, PROPPATCH without adding request replay support.
Cards VCARD 3.0 with VERSION, UID, FN, N, EMAIL, REV only, no parameters initially. VERSION is a fixed literal, UID its own bijective slot, structured N preserves components, FN/EMAIL/N content independently seeded, REV validated timestamp slot. The existing seeded replay tests exercise EMAIL and REV; CRUD/initial-sync gate workloads must exercise all six fields including N, otherwise remove unexercised entries. FN/N/UID/VERSION are baseline fields. Unknown fields/parameters, PHOTO/binary and vCard 4.0 refuse. NOTE is deferred; keep its sentinel refusal test. Adding a real Contacts-required field requires evidence, its grammar/seed assertions and a focused test, not the old broad enum surface.
Authorization / WWW-Authenticate Basic only. Authorization payload replaced by a typed auth-presence slot; do not preserve base64. Basic challenge realm becomes a typed slot, optional charset only fixed UTF-8. Unknown Basic parameters refuse. Preserve initial unauthenticated request -> 401 -> authenticated request. Inject target credentials only at authenticated exchanges; disable implicit retries that erase 401 or redirect steps. Never learn expected card content from an initial response.
Deferred auth/state Bearer, Digest, Authentication-Info, Cookie, Set-Cookie, Proxy-Authorization, X-Apple-ID-Account-Web-Auth, X-Apple-Password all refuse immediately with no grammar conversion or fixture artifact. Capture remains intact. Do not strip required auth and claim replay equivalence.
Accept / Content-Type Closed media ranges/types XML, text/vcard, text/plain and text/html; supported charset UTF-8/US-ASCII and numeric q where needed. Preserve semantic charset differences; unknown parameters refuse. Plain/HTML body rules below restrict status.
Depth / Prefer / Preference-Applied Depth 0/1/infinity; only return=minimal/representation preferences. Fixed grammar, unknown preference refuses.
DAV / Allow Closed capability/method lists including 1/2/3, addressbook, calendar-access, extended-mkcol, sync-collection and the advertised methods above. Preserve identity and list structure. Unknown capability refuses only export.
ETag / If-Match / If-None-Match / Sync-Token Quoted syntax, strength/wildcard/list shape, typed identity, empty/nonempty token and advancement. Target-owned request binding, uniqueness and stale-write behavior retained.
Location / Content-Location / DAV href Only root, relative and HTTP/HTTPS URI forms with approved well-known/principal/collection/resource route structure; principal/resource/authority are typed slots. Preserve /dav, contacts vs addressbook, slash shape and origin. Queries, fragments and userinfo refuse, not scrub. Unknown routes refuse. Target origin translation is explicit; external redirects need an approved disposable target origin binding. Follow/check target-advertised hrefs, never repair them to a working canonical URL.
Link Refuse first PR, including credential-bearing URLs. Neither current minimal scenario needs conversion. If real evidence later requires it, a separate reviewed URI/relation policy is needed.
Date / Last-Modified / XML dates Valid timestamp slots. Accept IMF-fixdate, RFC850 and asctime with real date/time/weekday and standard two-digit-year rules. Impossible timestamps fail; equivalent valid legacy formats normalize.
Content-Length Validate source representation length using method/status semantics, including HEAD; regenerate after target rendering. XML getcontentlength uses independent target representation length/seed, never arbitrary first-response learning.
X-Token-Required Explicit strict boolean regression entry only; gate must exercise true and false. No generic token/session substring redaction.
Explicit transport drops Host, Connection, Keep-Alive, Transfer-Encoding, TE, Trailer, Proxy-Connection where legal; Accept-Encoding/Content-Encoding only after checked framing and bounded identity/gzip decode. A Connection-nominated retained/unknown semantic field or semantic trailer refuses, never disappears. Recompute target framing and request identity encoding.
Profile-specific drops User-Agent and Server omitted for fixed synthetic/Apple Contacts profile. If source behavior depends on client identity, refuse this profile and add evidence-backed target identity policy.

Header names are fixed enums matched case-insensitively; preserve presence, multiplicity and order among same-name values. Ignore different-header ordering. Any unlisted value/name refuses without echoing it.

Body variants: Empty, DavXml, VCard, OmittedErrorBody. Plain/HTML at 4xx/5xx or supported discovery redirects becomes only presence plus fixed media type and an explicit error-prose-omitted comparison flag; no contents, lengths or digests. Successful HTML 200 refuses rather than masquerading as DAV. Unknown successful bodies refuse. Embedded address-data parses as vCard even through XML escaping/CDATA.

XML allowlist starts from the DAV/CardDAV entries only in #820 parse_xml_element at 58e50394, restricted further to the two scripted scenarios and historical discovery/error controls. Record the final exact enum table with those tests, not a wildcard namespace. Fixed supported attributes use checked enums or typed slots; unknown names/attributes refuse. Preserve namespace/node shape/order, typed href, ETag, sync-token, media type/charset, representation length, display-text slots and validated creationdate/getlastmodified. No DTD/entities. Explicit additions: DAV principal-collection-set, report capabilities with their supported-report-set/supported-report/report children, root href, and CardDAV {urn:ietf:params:xml:ns:carddav}no-uid-conflict under DAV error at 403. Never equate wrong namespaces.

The round-3 review did not name the Radicale extension QName. Reproduce it privately on a Linux VM before first-PR gate completion; record a source-backed decision and a minimal synthetic regression. Add a closed typed entry if relevant, or an exact named drop with evidence of irrelevance. Unknown extension refuses until that decision; successful export of the supported discovery reproduction is required. No generic Radicale exemption. Radicale is research-only, not a product or routine regression dependency.

Explicit ancillary drop boundary: Flow connection/client/server addresses, TLS certificates, arbitrary metadata, absolute timestamps/durations, capture IDs and proxy options never enter fixture or diagnostics. Request times may be inspected privately for sequential-order checks only. Do not serialize them. Private receipt hash/nonce and file paths are likewise excluded. Add sentinel tests in each ancillary field.

Reuse and own-DAV evidence

Keep #820 field-by-field comparison, independent content seeds, TargetBindings bijection, structural route checks, SyncProgress, ETag/If-Match and stale-write 412, ordered multistatus namespace/shape, timestamp/charset/length checks, required-scenario inventory and PENDING logic. Retain existing replay limits (4 MiB bodies, nesting 32, 30-second exchange deadline). Reuse focused tests in replay.rs, fixtures.rs, synthetic.rs, review_round1_mutations.rs, review_etag_xml.rs, review_r2_mutations.rs and review_regressions.rs. Strengthen tests that fail on missing seeds before the intended mutation. Port independent dev-02 review820-r3 probes as synthetic tests; do not copy raw logs/captures.

Do not reuse schema.rs raw-header fallback or whole-value redaction, record.rs proxy/inline validation or unconditional cleanup messages. Construct new closed output objects. Delete assertions that ordinary valid source responses should abort capture.

Own-DAV round-3 reproduction: source-derived wire 403 containing CardDAV no-uid-conflict, from carddav-writes-r2 (crates/carddav/src/write.rs:90–95 via crates/dav/src/serve.rs:915–923), caused recorder 502 because it knew only DAV no-uid-conflict at 409. The available carddav-r2 live binary was not DAV-mounted; its well-known HTML 200 proves no live own-DAV compatibility. New gate must drive a disposable DAV-mounted own server through stock mitmdump on Linux: discovery, initial 401, Basic authenticated depth-0 PROPFIND, two independently seeded contacts, duplicate-UID write yielding exact 403, then successful read/recovery and completion sentinel. Capture must retain that response; export/replay must check status/QName/href. Attach sanitized commands, mounted route and source head only. An unmounted HTML 200 fails; a wire stub alone is insufficient.

Failure table

Failure Effect Detection Recovery Test
Ordinary/unknown valid HTTP Client retains original status/body, raw capture continues Through-proxy response/flow assertion Supported export or offline refusal, never schema-induced 502 Every historical ordinary-response P1
Clean EOF at record boundary before finalization No fixture publication Missing receipt/final sentinel despite apparent milestones Finish/repeat workload and finalize; retain raw Stop after last apparent scenario flow but before marker/receipt
Failed workload/stop, missing or duplicate marker, stale receipt, post-marker traffic No complete fixture Marker+receipt+digest+milestone checks Fix capture/finalize; never fabricate success receipt Each failure, including valid marker but failed proxy exit
Oversized/truncated/HAR input Import refuses before large allocation stat, native frame pre-scan, bounded upstream reads and process cap Retain source, smaller supported scenario 64 MiB file boundary; oversized record/header; decimal/payload/tag truncation; HAR
Decoder object expansion or compressed body expansion Child terminates/refuses with fixed error 1 GiB/60s containment, bounded decompression Narrow input or reviewed limit change Metadata-heavy flow, nested input, gzip overrun/truncation; no traceback
Unknown field or deferred grammar Offline refusal, no output Closed policy Reviewed small follow-up, rerun export Unknown names and Digest/cookie/Link/userinfo/query variants
Source credential or arbitrary metadata No literal copy path Closed constructors and artifact/diagnostic sentinels Slot supported scalars or refuse unsupported field Apple/Digest/Link, body/card/ancillary matrix
Missing seed, collapsed UID or wrong FN Replay fails before dependent send or at intended comparison Independent seed map and bijection Fix seeds/server, not expected-value learning Passing baseline then FN/UID swap/collapse
Broken target href Replay fails, no normalization repair Structural route comparison and actual advertised-route follow Fix route or explicit mapping /dav removal and collection swap
Stale token/tag, wrong 412/type/length/date Intended replay mismatch Retained typed comparisons Fix server Historical seeded mutations
Overlapping dependent requests / incomplete scenario No completeness claim Sequential private ordering/milestones Recapture sequentially; no guessed reordering Overlap/reorder/zero/partial coverage
Publish/cleanup/output conflict Existing output/input retained, no success claim Typed staging, no-overwrite atomic rename, actual cleanup status Remove only owned staging; report failure accurately Write/rename/cleanup error, existing directory

First-PR Linux gates and historical inventory

All automated first-PR gates run on Linux VMs, not the control host or Mac. They do not depend on GUI permission. Use stock mitmdump with a disposable DAV-mounted Calternal server for product integration, scripted HTTP clients for both scenarios, and upstream HTTPFlow/FlowWriter for focused exporter cases. Minimal synthetic responders may mutate or serve historical response shapes through the stock proxy; no custom production recorder. Real Apple coverage stays PENDING.

Every response mutation starts with an independently seeded passing HTTP replay baseline. Change one semantic value, keep Date/Content-Length controls and actual PUT/If-Match flows, and assert intended rejection code/exchange. Missing-seed failures or tests that assert historical misses do not count as mutation detection.

Source Required Linux test
R1 P1.1 Two independently seeded contacts: unchanged pass, FN change/UID swap/UID collapse fail for intended reason.
R1 P1.2 Serve and follow advertised href; missing /dav or contacts→addressbook fails without repair. Root/relative/HTTP/HTTPS controls pass.
R1 P1.3 and R2 P1 OPTIONS DAV/Allow extended-mkcol and MKCALENDAR reaches client intact, exports and replays. Changed supported capability fails comparison; unknown capability survives capture and refuses export.
R3 P1 credential persistence Basic supported path uses typed slots. Apple headers, Bearer/Digest auth/challenges, Authentication-Info nextnonce/rspauth, Cookie/Set-Cookie and Link userinfo survive raw capture but refuse export with zero fixture output. Test exact historical sentinel strings and encoded variants in files/stdout/stderr.
R3 P1 ordinary responses Well-known 301 Location /, HTTP/HTTPS Location/Content-Location, 401 text/plain, depth-0 207 principal/report/root discovery including exact reproduced Radicale extension, and own-DAV CardDAV no-uid-conflict 403 reach client unchanged, export and replay for supported shapes. Own-server mounted gate is mandatory; unknown XML refuses offline.
Earlier privacy P1s from #789 Unknown request targets, Location/XML href values, principal/resource segments, query/fragment/userinfo and unknown card/XML names never leak; supported shapes slot, deferred shapes refuse.
R1 P2 Media family, independently derived getcontentlength, weak/unquoted ETag mutations fail.
R2 P2 Impossible REV/creationdate/getlastmodified and utf-8→us-ascii header/XML type mutations fail. Valid leap-day/date and all three HTTP-date grammars pass.
R3 P2 Cookie secret-only and Secure-removed variants both refuse export until later cookie support; no misleading successful compare. X-Token-Required boolean flip fails. Basic→Bearer refuses. Initial unauthenticated 401 then authenticated success is preserved.
R3 continuity Nonadvancing sync-token, reused ETag after changed PUT and stale PUT 204 instead of 412 fail. Listed-header addition/removal/duplicates/reversed same-name values fail where meaningful. Header case and order between distinct names remain equivalent; unknown additions refuse.
Carried P3 Cleanup error does not claim removal; source capture and existing fixture survive export failure.

Additional mandatory gates:

  • Completion: missing receipt/marker, complete-record prefix after all apparent milestones, duplicate/wrong nonce, stale hash/length, traffic after marker, marker success with workload/proxy failure, malformed receipt; no publication. Successfully finalized two-scenario controls export. Exclude marker from replay.
  • Pre-reader resource tests: sparse oversize raw file rejected before decoder invocation; a short file advertising an enormous record rejected without allocating that length; truncated record/header, HAR, >64 KiB headers, excessive metadata, gzip expansion/partial stream, child timeout/memory termination. Assert bounded reads/decoder invocation order and sanitized stderr; raw bytes unchanged and no published fixture.
  • Strict schema/privacy: producer -> strict Rust v3 loader round trip; deny unknown nested fields and slot references; v2 rejected; deterministic repeat exports; repeated typed identity/UID distinctness. Whole-file scans across all staged/published files and stdout/stderr, including refusal cases, not just JSON in memory.
  • Distinct generated sentinels in header names/values, Basic payload/realm, deferred Digest/Apple/cookie/Link, URI segments/userinfo/query/fragment, XML QName/namespace/attributes/text/CDATA/embedded card, card fields/parameters including deferred NOTE/PHOTO, reason phrase/error prose, CLI metadata and Flow ancillary addresses/TLS certificates/metadata/timing/IDs. Exercise literal, percent/double-percent, XML escapes, base64/URL-safe base64, uppercase and non-UTF-8. Supported inputs slot/omit through named rules; unsupported ones refuse without echoing source strings. No raw Flow state or map sidecar.
  • Publication/I/O/ordering: fail write/rename/cleanup, existing destination, overlapping dependent writes and zero/incomplete scenarios; only typed staging, existing output untouched, accurate error and no complete-session marker on refusal.

Nix and commands for implementation

The reviewer evaluated nixpkgs pin 1e7da2489c0ba3a06bceec1b444fbc02c5223ab1: Python 3.14.7/mitmproxy 12.2.3 available on x86_64-linux and aarch64-linux. That is evaluation evidence only, not build/import proof. Provide pkgs.python3.withPackages (ps: [ ps.mitmproxy ]) and mitmdump from that same package set. PATH executable alone is insufficient. No pip or alternate decoder. Temporary Nix files follow BRIEF-COMMON until #779 merges; never commit the temporary overlay.

On both dev-01 and dev-02, through netbird ssh and Nix, require import of FlowReader, FlowWriter, a synthetic writer→reader round trip and stock mitmdump smoke test. On dev-02 run focused functional suites and mounted own-server scenario through ~/bin/slot, with production builds at -j4. Planned commands (not results):

  • nix develop -c bash -c '~/bin/slot python3 -m unittest discover -s crates/dav-replay/export/tests' (includes package import, writer/reader, mitmdump integration, finalization/resource/privacy gates).
  • nix develop -c bash -c 'cargo fmt --all --check'.
  • nix develop -c bash -c '~/bin/slot cargo clippy -j4 -p calternal-dav-replay --locked -- -D warnings'.
  • nix develop -c bash -c '~/bin/slot cargo test -j4 -p calternal-dav-replay --locked' (includes v3 contract, typed historical mutation matrix).
  • nix develop -c bash -c '~/bin/slot cargo run -j4 -p calternal-dav-replay --locked -- check crates/dav-replay/fixtures' must retain exit 3/PENDING until real captures exist.

Use the existing VM job slots; stop only owned processes. Capture/finalization integration failures fail the first-PR gate, not merely a pending native-capture result. A real Mac capture later requires the Mac lock/Nix/GUI permissions; Linux scripted traffic must not be labeled Apple capture. Calendar/Reminders and real-iPhone gates remain visible. ADR 0067 traffic storage stays on the issue, not in the repository; this contacts-only plan does not authorize calendar fixtures.

Follow-ups and unresolved evidence

Separate reviewed follow-ups, only when source evidence requires them: Digest/Bearer and target challenge/state handling; cookies with typed names/values/flags and target-owned jar (then Secure-removal comparison); Link relations; URI userinfo/query/fragment policy; additional card fields/parameters/NOTE/photo/vCard versions; Calendar/Reminders profile; concurrent replay. None is a hidden first-PR dependency: unsupported sessions refuse without publication.

First-PR owner must establish the exact Radicale QName and mounted own-server head/start/seed commands, preserving the historical tests. Default: export refusal / product gate fails, never silently ignore. Real Mac routing/TLS trust/GUI access and whether actual Contacts needs extra auth/fields remain pending source evidence. The narrow profile does not claim full Apple coverage. No code changes, builds, tests or captures were run for this plan revision.

# DAV capture/export/replay plan v3, revised Design only. This description supersedes `docs/plans/0064-dav-recorder-v2.md` and the earlier v3 description. Architecture accepted; revised against [plan review 72220](https://git.kayg.org/kayg/calternal/pulls/818#issuecomment-72220). #820 remains paused at `58e503944542a8a84b7e9a6a4389433a877fa099` under the three-round rule. No implementation, capture or build is authorized by this design task. Future PRs target `dev`; never merge here. ## Goal and architecture Capture Apple Contacts ↔ CardDAV traffic with stock Nix mitmdump, export a narrow credential-free typed fixture offline, then reuse #820's Rust replay/comparison. No custom proxy or inline validator. Protocol-valid unknown responses must reach the client unchanged and remain in raw flows; schema errors can only refuse export, retaining raw evidence. Disk, network, TLS and proxy defects remain operational failures: “no schema-induced abort” is not a claim that recording cannot fail. Evidence: #820 reviews [round 1](https://git.kayg.org/kayg/calternal/pulls/820#issuecomment-72055), [round 2](https://git.kayg.org/kayg/calternal/pulls/820#issuecomment-72126), [round 3](https://git.kayg.org/kayg/calternal/pulls/820#issuecomment-72191). Round 3 caught 23/25 replay differences, yet ordinary responses became 502s and saved fixtures retained Apple auth, Digest nonces and Link URL credentials. Repatching that recorder or building another HTTP proxy repeats the wrong boundary. The small Python exporter uses upstream FlowReader/FlowWriter, not a replacement flow decoder. Reuse existing typed Rust comparison rather than redesigning it. ## First PR scope and sizes Only `contacts-crud` and `contacts-initial-sync`, Basic app-password authentication, sequential synthetic workloads. Prove on Linux first. Actual Apple profile coverage, Mac GUI access and real-iPhone requirements remain separate PENDING gates. | Component | Files/interfaces | Review budget | |---|---|---| | Capture/finalization | README and `scripts/capture-finalize.py`: workload wrapper, final scripted request, graceful stop and private receipt | 70–100 Python lines plus 50–70 documentation/Nix lines; zero proxy addons | | Bounded raw reader | `export/reader.py`: file/record preflight and containment; upstream FlowReader -> bounded flows | 80–110 Python lines; no new flow format/decoder | | Closed conversion | `export/policy.py`, `export/bodies.py`, `export/export.py`: Basic/URI/header policy, small card/XML adapters, CLI/atomic writer | 280–340 Python lines combined; split files above about 200 lines | | Typed Rust adaptation | `src/schema.rs`, `wire.rs`, `fixture.rs`, `main.rs`, retain `replay.rs` | 100–180 changed Rust lines: remove raw fallback, v3 strict loading, narrow discovery/error variants/auth state; reuse comparator | | Gates | `export/tests/test_export.py`, `tests/export_contract.rs`, `tests/review_v3_mutations.rs`, focused existing suites | 300–400 new/adapted test lines, table-driven historical cases; reuse existing probe/harness code | Total new Python production budget **430–550 lines**, plus **100–180 changed Rust lines**, **300–400 test lines**, **50–70 documentation/Nix lines**. These are estimates, not counts or acceptance ceilings. Keep correctness gates if implementation exceeds a budget; revisit decomposition rather than add fallbacks. Nix/dependency closure sizes are not part of these source-line estimates. `export --capture PATH --receipt PATH --profile contacts-v1 --scenario ENUM --macos X.Y.Z --out PATH` produces format 3 JSON for the Rust loader. Fixed metadata enums only; CLI free text cannot enter the fixture. Exit 0 only after complete staged output passes the strict Rust loader, then no-overwrite atomic publication. Export refusal exits 2 with fixed error code, exchange index and schema position, never source values/names/paths/tracebacks. Existing inventory `check` exit 3 remains PENDING until real required captures exist. Reject format 2 and disable/remove old unsafe `record`; never silently migrate raw fields. ## Capture and observable completion Raw flows and their receipt stay outside all checkouts on the originating VM, directory 0700/files 0600, umask 077. Raw includes secrets: never commit, upload to issues/CI, print, or copy to a repository staging directory. No mapping sidecar or source-valued diagnostics. Use a disposable synthetic account and target-owned credentials read from files, never argv. Stock mitmdump only, configured to retain bodies and avoid host/redirect rewriting. No old schema/body cap on capture. Configure temporary TLS trust only in the disposable guest if required; no production changes. A clean EOF or sufficient scenario milestones **does not establish completion**. The private finalizer owns the scripted workload/proxy processes and follows this protocol: 1. Run the requested workload successfully and wait for all its requests to complete. Stop background clients; require a quiescent, sequential scenario. For a future manual Apple capture, explicitly stop further app traffic before finalization; unverifiable quiescence refuses publication. 2. Send exactly one final sentinel request through the same proxy to the disposable origin: GET `/__dav_capture_complete__/<fresh-random-nonce>`, no auth/query/fragment/body. Expect a completed 404 from the unmodified own-server router. This reserved test path needs no product endpoint. Unexpected status, retry, missing response or further scenario traffic prevents finalization. The nonce distinguishes this run; it is not a fixture identifier. 3. Gracefully stop only this mitmdump process, wait for its successful exit and closed writer. Fail on unsuccessful exit or timeout. Flush the closed raw file to stable storage; do not create a receipt while the proxy may still append. 4. Atomically create a small private receipt with fixed version/scenario, expected nonce/status, exact raw byte length and SHA-256, workload success and proxy exit status. Receipt is completion evidence from the trusted local workload wrapper, not an unforgeable proof of user actions. Never produce one after a failed workload/stop. The hash and nonce are local control data and never enter fixtures/logs/issues. 5. Export requires both the receipt and a byte-matching raw file, exactly one complete matching sentinel **as the final HTTP flow**, no error/incomplete flows, unambiguous ordering and the scenario's required milestones. Remove the sentinel from exported exchanges. Reading EOF alone is never enough. Reject missing/malformed/stale receipt, missing/duplicate/mismatched marker, traffic after marker or changed raw bytes. Retain raw evidence on refusal. This detects a readable prefix cut between complete records, including after apparent scenario coverage but before finalization. It does not turn an arbitrary receipt into proof of actual Contacts behavior. Once a real fixture is accepted and replay evidence recorded, remove local raw/receipt, disposable account and proxy CA material; accurately report deletion without promising storage-level erasure. ## Resource boundary before FlowReader The review identified that mitmproxy 12.2.3 FlowReader calls tnetstring.load, which allocates the record before semantic checks, and can auto-load HAR. Do not claim the 4 MiB body check bounds that allocation. Concrete importer contract, implemented around the upstream decoder: - Open only a regular raw file (no symlink/device/FIFO), hold that descriptor, fstat it and reject length above **64 MiB** before FlowReader or any whole-file load. Receipt length must match; hash the same descriptor in bounded chunks. No unbounded `read()` or mmap. Limit receipt to **4 KiB** before parsing. - Pre-scan top-level native tnetstring framing in fixed-size reads: bounded decimal length prefix (at most nine digits), colon, payload within **12 MiB per record**, final expected native dictionary tag. Verify every length against remaining file bytes; reject partial prefix/payload/tag, trailing garbage, more than **10,000 records**, and every HAR/JSON input. Seek/skip bounded payloads, never allocate advertised lengths during this pass. Do not decode nested values or duplicate upstream serialization logic. - Feed upstream FlowReader one preflight-bounded record view at a time; cap every underlying read at that record's remaining bytes and require exact consumption. Its HAR autodetection cannot consume beyond a record and HAR was already refused. No generator expression that loads all flows. Re-fstat and recheck raw digest before publication so concurrent mutation invalidates the receipt; even changed bytes cannot bypass read caps. - Record size caps bound FlowReader's raw allocation, not its Python object amplification or recursion. Run importer/decoder in a child with **1 GiB address-space cap**, **60 seconds wall time**, sanitized error handling and no core dump. Linux VM gate must demonstrate these limits actually apply (e.g. inherited RLIMIT_AS and a parent wall deadline). Process exit/signal/decoder exception yields a fixed refusal, no traceback or partially published fixture. Receipt/hash/preflight work also runs within these limits. A smaller cap can be chosen only with measured supported-baseline headroom. - After decoding one bounded flow: at most **64 KiB aggregate raw request+response header bytes**, **4 MiB decoded body each**, nesting **32**, **100,000 slots**, **16 MiB aggregate slot-key bytes** for the session. Reject metadata-heavy and excessive-header flows even if their framed record is under 12 MiB. - Never access mitmproxy's auto-decoding `.content`/`.text` before checking content encoding. Support identity/gzip only; incrementally decompress raw content in bounded chunks with a 4 MiB output ceiling and reject overrun, incomplete stream, unsupported coding or bad data. Do not materialize expansion before checking it. Semantic headers/trailers cannot be silently hidden as framing. Costs: preflight+hash are O(raw bytes), conversion O(B log I) with ordered slot maps, memory O(one bounded decoded flow + bounded slot keys + upstream decoder object overhead). The process cap contains that overhead; exceeding it is an explicit refusal, not a claim of exact Python allocation accounting. Staged files contain only new typed objects; never serialize Flow, Flow state, exception or arbitrary dictionaries from input. ## Closed allowlist, contacts-v1 Default for every unlisted request/response header, URI feature, XML QName/attribute, card property/parameter and body variant: **refuse offline export**. Never copy an unknown name or infer that an unlisted request header is irrelevant. No Raw/Other strings, name substrings, hashes of captured secrets, whole-header opaque fallback or source-to-slot map in output. Every retained scalar is a closed grammar value or a typed slot assigned by first-seen identity. Preserve repeated identity within its semantic domain; distinct UID identities cannot collapse. | Field | First-PR treatment and reason | |---|---| | Session/method/status | Fixed profile/scenario/app/version fields; supported request methods OPTIONS, GET, HEAD, PROPFIND, REPORT, PUT, DELETE. Numeric HTTP status only, never reason phrase. Advertised Allow may also include MKCOL, MKCALENDAR, PROPPATCH without adding request replay support. | | Cards | VCARD 3.0 with **VERSION, UID, FN, N, EMAIL, REV only**, no parameters initially. VERSION is a fixed literal, UID its own bijective slot, structured N preserves components, FN/EMAIL/N content independently seeded, REV validated timestamp slot. The existing seeded replay tests exercise EMAIL and REV; CRUD/initial-sync gate workloads must exercise all six fields including N, otherwise remove unexercised entries. FN/N/UID/VERSION are baseline fields. Unknown fields/parameters, PHOTO/binary and vCard 4.0 refuse. NOTE is deferred; keep its sentinel refusal test. Adding a real Contacts-required field requires evidence, its grammar/seed assertions and a focused test, not the old broad enum surface. | | Authorization / WWW-Authenticate | **Basic only**. Authorization payload replaced by a typed auth-presence slot; do not preserve base64. Basic challenge realm becomes a typed slot, optional charset only fixed UTF-8. Unknown Basic parameters refuse. Preserve initial unauthenticated request -> 401 -> authenticated request. Inject target credentials only at authenticated exchanges; disable implicit retries that erase 401 or redirect steps. Never learn expected card content from an initial response. | | Deferred auth/state | Bearer, Digest, Authentication-Info, Cookie, Set-Cookie, Proxy-Authorization, X-Apple-ID-Account-Web-Auth, X-Apple-Password all refuse immediately with no grammar conversion or fixture artifact. Capture remains intact. Do not strip required auth and claim replay equivalence. | | Accept / Content-Type | Closed media ranges/types XML, text/vcard, text/plain and text/html; supported charset UTF-8/US-ASCII and numeric q where needed. Preserve semantic charset differences; unknown parameters refuse. Plain/HTML body rules below restrict status. | | Depth / Prefer / Preference-Applied | Depth 0/1/infinity; only return=minimal/representation preferences. Fixed grammar, unknown preference refuses. | | DAV / Allow | Closed capability/method lists including 1/2/3, addressbook, calendar-access, extended-mkcol, sync-collection and the advertised methods above. Preserve identity and list structure. Unknown capability refuses only export. | | ETag / If-Match / If-None-Match / Sync-Token | Quoted syntax, strength/wildcard/list shape, typed identity, empty/nonempty token and advancement. Target-owned request binding, uniqueness and stale-write behavior retained. | | Location / Content-Location / DAV href | Only root, relative and HTTP/HTTPS URI forms with approved well-known/principal/collection/resource route structure; principal/resource/authority are typed slots. Preserve `/dav`, contacts vs addressbook, slash shape and origin. **Queries, fragments and userinfo refuse**, not scrub. Unknown routes refuse. Target origin translation is explicit; external redirects need an approved disposable target origin binding. Follow/check target-advertised hrefs, never repair them to a working canonical URL. | | Link | Refuse first PR, including credential-bearing URLs. Neither current minimal scenario needs conversion. If real evidence later requires it, a separate reviewed URI/relation policy is needed. | | Date / Last-Modified / XML dates | Valid timestamp slots. Accept IMF-fixdate, RFC850 and asctime with real date/time/weekday and standard two-digit-year rules. Impossible timestamps fail; equivalent valid legacy formats normalize. | | Content-Length | Validate source representation length using method/status semantics, including HEAD; regenerate after target rendering. XML getcontentlength uses independent target representation length/seed, never arbitrary first-response learning. | | X-Token-Required | Explicit strict boolean regression entry only; gate must exercise true and false. No generic token/session substring redaction. | | Explicit transport drops | Host, Connection, Keep-Alive, Transfer-Encoding, TE, Trailer, Proxy-Connection where legal; Accept-Encoding/Content-Encoding only after checked framing and bounded identity/gzip decode. A Connection-nominated retained/unknown semantic field or semantic trailer refuses, never disappears. Recompute target framing and request identity encoding. | | Profile-specific drops | User-Agent and Server omitted for fixed synthetic/Apple Contacts profile. If source behavior depends on client identity, refuse this profile and add evidence-backed target identity policy. | Header names are fixed enums matched case-insensitively; preserve presence, multiplicity and order among same-name values. Ignore different-header ordering. Any unlisted value/name refuses without echoing it. Body variants: Empty, DavXml, VCard, OmittedErrorBody. Plain/HTML at 4xx/5xx or supported discovery redirects becomes only presence plus fixed media type and an explicit error-prose-omitted comparison flag; no contents, lengths or digests. Successful HTML 200 refuses rather than masquerading as DAV. Unknown successful bodies refuse. Embedded address-data parses as vCard even through XML escaping/CDATA. XML allowlist starts from the **DAV/CardDAV entries only** in #820 `parse_xml_element` at `58e50394`, restricted further to the two scripted scenarios and historical discovery/error controls. Record the final exact enum table with those tests, not a wildcard namespace. Fixed supported attributes use checked enums or typed slots; unknown names/attributes refuse. Preserve namespace/node shape/order, typed href, ETag, sync-token, media type/charset, representation length, display-text slots and validated creationdate/getlastmodified. No DTD/entities. Explicit additions: DAV principal-collection-set, report capabilities with their supported-report-set/supported-report/report children, root href, and **CardDAV** `{urn:ietf:params:xml:ns:carddav}no-uid-conflict` under DAV error at **403**. Never equate wrong namespaces. The round-3 review did not name the Radicale extension QName. Reproduce it privately on a Linux VM before first-PR gate completion; record a source-backed decision and a minimal synthetic regression. Add a closed typed entry if relevant, or an exact named drop with evidence of irrelevance. Unknown extension refuses until that decision; successful export of the supported discovery reproduction is required. No generic Radicale exemption. Radicale is research-only, not a product or routine regression dependency. Explicit ancillary drop boundary: Flow connection/client/server addresses, TLS certificates, arbitrary metadata, absolute timestamps/durations, capture IDs and proxy options never enter fixture **or diagnostics**. Request times may be inspected privately for sequential-order checks only. Do not serialize them. Private receipt hash/nonce and file paths are likewise excluded. Add sentinel tests in each ancillary field. ## Reuse and own-DAV evidence Keep #820 field-by-field comparison, independent content seeds, TargetBindings bijection, structural route checks, SyncProgress, ETag/If-Match and stale-write 412, ordered multistatus namespace/shape, timestamp/charset/length checks, required-scenario inventory and PENDING logic. Retain existing replay limits (4 MiB bodies, nesting 32, 30-second exchange deadline). Reuse focused tests in replay.rs, fixtures.rs, synthetic.rs, review_round1_mutations.rs, review_etag_xml.rs, review_r2_mutations.rs and review_regressions.rs. Strengthen tests that fail on missing seeds before the intended mutation. Port independent dev-02 review820-r3 probes as synthetic tests; do not copy raw logs/captures. Do not reuse schema.rs raw-header fallback or whole-value redaction, record.rs proxy/inline validation or unconditional cleanup messages. Construct new closed output objects. Delete assertions that ordinary valid source responses should abort capture. Own-DAV round-3 reproduction: source-derived wire 403 containing CardDAV no-uid-conflict, from `carddav-writes-r2` (`crates/carddav/src/write.rs:90–95` via `crates/dav/src/serve.rs:915–923`), caused recorder 502 because it knew only DAV no-uid-conflict at 409. The available carddav-r2 live binary was **not DAV-mounted**; its well-known HTML 200 proves no live own-DAV compatibility. New gate must drive a disposable **DAV-mounted own server** through stock mitmdump on Linux: discovery, initial 401, Basic authenticated depth-0 PROPFIND, two independently seeded contacts, duplicate-UID write yielding exact 403, then successful read/recovery and completion sentinel. Capture must retain that response; export/replay must check status/QName/href. Attach sanitized commands, mounted route and source head only. An unmounted HTML 200 fails; a wire stub alone is insufficient. ## Failure table | Failure | Effect | Detection | Recovery | Test | |---|---|---|---|---| | Ordinary/unknown valid HTTP | Client retains original status/body, raw capture continues | Through-proxy response/flow assertion | Supported export or offline refusal, never schema-induced 502 | Every historical ordinary-response P1 | | Clean EOF at record boundary before finalization | No fixture publication | Missing receipt/final sentinel despite apparent milestones | Finish/repeat workload and finalize; retain raw | Stop after last apparent scenario flow but before marker/receipt | | Failed workload/stop, missing or duplicate marker, stale receipt, post-marker traffic | No complete fixture | Marker+receipt+digest+milestone checks | Fix capture/finalize; never fabricate success receipt | Each failure, including valid marker but failed proxy exit | | Oversized/truncated/HAR input | Import refuses before large allocation | stat, native frame pre-scan, bounded upstream reads and process cap | Retain source, smaller supported scenario | 64 MiB file boundary; oversized record/header; decimal/payload/tag truncation; HAR | | Decoder object expansion or compressed body expansion | Child terminates/refuses with fixed error | 1 GiB/60s containment, bounded decompression | Narrow input or reviewed limit change | Metadata-heavy flow, nested input, gzip overrun/truncation; no traceback | | Unknown field or deferred grammar | Offline refusal, no output | Closed policy | Reviewed small follow-up, rerun export | Unknown names and Digest/cookie/Link/userinfo/query variants | | Source credential or arbitrary metadata | No literal copy path | Closed constructors and artifact/diagnostic sentinels | Slot supported scalars or refuse unsupported field | Apple/Digest/Link, body/card/ancillary matrix | | Missing seed, collapsed UID or wrong FN | Replay fails before dependent send or at intended comparison | Independent seed map and bijection | Fix seeds/server, not expected-value learning | Passing baseline then FN/UID swap/collapse | | Broken target href | Replay fails, no normalization repair | Structural route comparison and actual advertised-route follow | Fix route or explicit mapping | /dav removal and collection swap | | Stale token/tag, wrong 412/type/length/date | Intended replay mismatch | Retained typed comparisons | Fix server | Historical seeded mutations | | Overlapping dependent requests / incomplete scenario | No completeness claim | Sequential private ordering/milestones | Recapture sequentially; no guessed reordering | Overlap/reorder/zero/partial coverage | | Publish/cleanup/output conflict | Existing output/input retained, no success claim | Typed staging, no-overwrite atomic rename, actual cleanup status | Remove only owned staging; report failure accurately | Write/rename/cleanup error, existing directory | ## First-PR Linux gates and historical inventory All automated first-PR gates run on **Linux VMs**, not the control host or Mac. They do not depend on GUI permission. Use stock mitmdump with a disposable DAV-mounted Calternal server for product integration, scripted HTTP clients for both scenarios, and upstream HTTPFlow/FlowWriter for focused exporter cases. Minimal synthetic responders may mutate or serve historical response shapes through the stock proxy; no custom production recorder. Real Apple coverage stays PENDING. Every response mutation starts with an independently seeded **passing HTTP replay baseline**. Change one semantic value, keep Date/Content-Length controls and actual PUT/If-Match flows, and assert intended rejection code/exchange. Missing-seed failures or tests that assert historical misses do not count as mutation detection. | Source | Required Linux test | |---|---| | R1 P1.1 | Two independently seeded contacts: unchanged pass, FN change/UID swap/UID collapse fail for intended reason. | | R1 P1.2 | Serve and follow advertised href; missing /dav or contacts→addressbook fails without repair. Root/relative/HTTP/HTTPS controls pass. | | R1 P1.3 and R2 P1 | OPTIONS DAV/Allow extended-mkcol and MKCALENDAR reaches client intact, exports and replays. Changed supported capability fails comparison; unknown capability survives capture and refuses export. | | R3 P1 credential persistence | Basic supported path uses typed slots. Apple headers, Bearer/Digest auth/challenges, Authentication-Info nextnonce/rspauth, Cookie/Set-Cookie and Link userinfo survive raw capture but refuse export with zero fixture output. Test exact historical sentinel strings and encoded variants in files/stdout/stderr. | | R3 P1 ordinary responses | Well-known 301 Location `/`, HTTP/HTTPS Location/Content-Location, 401 text/plain, depth-0 207 principal/report/root discovery including exact reproduced Radicale extension, and own-DAV CardDAV no-uid-conflict 403 reach client unchanged, export and replay for supported shapes. Own-server mounted gate is mandatory; unknown XML refuses offline. | | Earlier privacy P1s from #789 | Unknown request targets, Location/XML href values, principal/resource segments, query/fragment/userinfo and unknown card/XML names never leak; supported shapes slot, deferred shapes refuse. | | R1 P2 | Media family, independently derived getcontentlength, weak/unquoted ETag mutations fail. | | R2 P2 | Impossible REV/creationdate/getlastmodified and utf-8→us-ascii header/XML type mutations fail. Valid leap-day/date and all three HTTP-date grammars pass. | | R3 P2 | Cookie secret-only and Secure-removed variants both **refuse export** until later cookie support; no misleading successful compare. X-Token-Required boolean flip fails. Basic→Bearer refuses. Initial unauthenticated 401 then authenticated success is preserved. | | R3 continuity | Nonadvancing sync-token, reused ETag after changed PUT and stale PUT 204 instead of 412 fail. Listed-header addition/removal/duplicates/reversed same-name values fail where meaningful. Header case and order between distinct names remain equivalent; unknown additions refuse. | | Carried P3 | Cleanup error does not claim removal; source capture and existing fixture survive export failure. | Additional mandatory gates: - Completion: missing receipt/marker, complete-record prefix after all apparent milestones, duplicate/wrong nonce, stale hash/length, traffic after marker, marker success with workload/proxy failure, malformed receipt; no publication. Successfully finalized two-scenario controls export. Exclude marker from replay. - Pre-reader resource tests: sparse oversize raw file rejected before decoder invocation; a short file advertising an enormous record rejected without allocating that length; truncated record/header, HAR, >64 KiB headers, excessive metadata, gzip expansion/partial stream, child timeout/memory termination. Assert bounded reads/decoder invocation order and sanitized stderr; raw bytes unchanged and no published fixture. - Strict schema/privacy: producer -> strict Rust v3 loader round trip; deny unknown nested fields and slot references; v2 rejected; deterministic repeat exports; repeated typed identity/UID distinctness. Whole-file scans across all staged/published files and stdout/stderr, including refusal cases, not just JSON in memory. - Distinct generated sentinels in header names/values, Basic payload/realm, deferred Digest/Apple/cookie/Link, URI segments/userinfo/query/fragment, XML QName/namespace/attributes/text/CDATA/embedded card, card fields/parameters including deferred NOTE/PHOTO, reason phrase/error prose, CLI metadata and Flow ancillary addresses/TLS certificates/metadata/timing/IDs. Exercise literal, percent/double-percent, XML escapes, base64/URL-safe base64, uppercase and non-UTF-8. Supported inputs slot/omit through named rules; unsupported ones refuse without echoing source strings. No raw Flow state or map sidecar. - Publication/I/O/ordering: fail write/rename/cleanup, existing destination, overlapping dependent writes and zero/incomplete scenarios; only typed staging, existing output untouched, accurate error and no complete-session marker on refusal. ## Nix and commands for implementation The reviewer evaluated nixpkgs pin `1e7da2489c0ba3a06bceec1b444fbc02c5223ab1`: Python 3.14.7/mitmproxy 12.2.3 available on x86_64-linux and aarch64-linux. That is evaluation evidence only, not build/import proof. Provide `pkgs.python3.withPackages (ps: [ ps.mitmproxy ])` and mitmdump from that same package set. PATH executable alone is insufficient. No pip or alternate decoder. Temporary Nix files follow BRIEF-COMMON until #779 merges; never commit the temporary overlay. On **both dev-01 and dev-02**, through `netbird ssh` and Nix, require import of `FlowReader, FlowWriter`, a synthetic writer→reader round trip and stock mitmdump smoke test. On dev-02 run focused functional suites and mounted own-server scenario through `~/bin/slot`, with production builds at `-j4`. Planned commands (not results): - `nix develop -c bash -c '~/bin/slot python3 -m unittest discover -s crates/dav-replay/export/tests'` (includes package import, writer/reader, mitmdump integration, finalization/resource/privacy gates). - `nix develop -c bash -c 'cargo fmt --all --check'`. - `nix develop -c bash -c '~/bin/slot cargo clippy -j4 -p calternal-dav-replay --locked -- -D warnings'`. - `nix develop -c bash -c '~/bin/slot cargo test -j4 -p calternal-dav-replay --locked'` (includes v3 contract, typed historical mutation matrix). - `nix develop -c bash -c '~/bin/slot cargo run -j4 -p calternal-dav-replay --locked -- check crates/dav-replay/fixtures'` must retain exit 3/PENDING until real captures exist. Use the existing VM job slots; stop only owned processes. Capture/finalization integration failures fail the first-PR gate, not merely a pending native-capture result. A real Mac capture later requires the Mac lock/Nix/GUI permissions; Linux scripted traffic must not be labeled Apple capture. Calendar/Reminders and real-iPhone gates remain visible. ADR 0067 traffic storage stays on the issue, not in the repository; this contacts-only plan does not authorize calendar fixtures. ## Follow-ups and unresolved evidence Separate reviewed follow-ups, only when source evidence requires them: Digest/Bearer and target challenge/state handling; cookies with typed names/values/flags and target-owned jar (then Secure-removal comparison); Link relations; URI userinfo/query/fragment policy; additional card fields/parameters/NOTE/photo/vCard versions; Calendar/Reminders profile; concurrent replay. None is a hidden first-PR dependency: unsupported sessions refuse without publication. First-PR owner must establish the exact Radicale QName and mounted own-server head/start/seed commands, preserving the historical tests. Default: export refusal / product gate fails, never silently ignore. Real Mac routing/TLS trust/GUI access and whether actual Contacts needs extra auth/fields remain pending source evidence. The narrow profile does not claim full Apple coverage. No code changes, builds, tests or captures were run for this plan revision.
docs: plan typed DAV recorder v2
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
39fc6dc4ea
kayg commented 2026-10-31 22:02:26 +00:00
Author
Owner
Copy link

Plan review

Blocking

  • None for the core design. Header names and XML qualified names are closed schema choices, unknown names abort, and variable XML text is placeholdered. The typed allowlist is a reasonable size after the three #789 review rounds.

Should change

  • State the response boundary explicitly: persist only the numeric status code, never the reason phrase/status line; accept only named structured body variants for every status, placeholder all variable text, and abort/clean up on free-form or unknown error bodies. Add capture tests with unique sentinels in a reason phrase and free-form 4xx/5xx body that prove no fixture bytes remain. The current status and typed body wording implies this but does not make the invariant explicit.
  • Require replay assertions for sync-token progression, current and stale ETag/If-Match flows (including the expected 412), and multistatus namespace/order/shape. Placeholder equality and exchange order can preserve these while binding values to target responses; the current generic correlation tests do not require these gate behaviors.
  • In the no-verbatim property test, use a distinct sentinel per value slot and exercise escaped/percent-encoded URI and XML forms. An exact search for the original string can miss an encoded representation.

Nits

  • None.

Verdict: approve with changes.

## Plan review ### Blocking - None for the core design. Header names and XML qualified names are closed schema choices, unknown names abort, and variable XML text is placeholdered. The typed allowlist is a reasonable size after the three #789 review rounds. ### Should change - State the response boundary explicitly: persist only the numeric status code, never the reason phrase/status line; accept only named structured body variants for every status, placeholder all variable text, and abort/clean up on free-form or unknown error bodies. Add capture tests with unique sentinels in a reason phrase and free-form 4xx/5xx body that prove no fixture bytes remain. The current `status` and `typed body` wording implies this but does not make the invariant explicit. - Require replay assertions for sync-token progression, current and stale ETag/If-Match flows (including the expected 412), and multistatus namespace/order/shape. Placeholder equality and exchange order can preserve these while binding values to target responses; the current generic correlation tests do not require these gate behaviors. - In the no-verbatim property test, use a distinct sentinel per value slot and exercise escaped/percent-encoded URI and XML forms. An exact search for the original string can miss an encoded representation. ### Nits - None. **Verdict: approve with changes.**
kayg changed title from Plan typed DAV recorder v2 to Plan DAV capture/export/replay v3 2026-11-01 01:25:50 +00:00
kayg commented 2026-11-01 01:30:29 +00:00
Author
Owner
Copy link

Independent plan review (v3)

Reviewed #818 at dfcd2fdfa577d58a9153760577dcd30c2c45fb1f, its description, and all three #820 reviews: 72055, 72126, 72191. Verdict: revise, preserving the proposed three-part architecture.

Structural assessment

The architecture removes both repeated mechanisms. Stock mitmdump has no application schema validation that can turn an ordinary DAV response into a 502. Unsupported semantics now stop offline export while preserving capture. Proxy setup, disk failures and upstream proxy defects remain separate operational failures.

For fixtures, closed names plus grammars/typed slots remove the raw-header fallback. The plan explicitly covers free text in allowed headers, XML text/attributes/CDATA, vCard NOTE and parameters, URL userinfo/query/path, Digest material, and error prose. PHOTO/binary content refuses export. No remaining arbitrary-text copy path is authorized by the design. This conclusion depends on constructing new output objects rather than serializing any Flow state or reusing #820's raw variants. Sentinel tests support that boundary; they do not establish it alone.

Findings P0–P3

P0: none found.

P1: capture completeness has no observable completion contract. The plan promises that interrupted saves/timeouts cannot publish a complete scenario, but a .mitm file is a sequence of independent records. FlowReader accepts EOF between records. A proxy stopped after a complete flow, with a later request pending or not yet saved, can leave a readable prefix containing all currently required scenario milestones. No error flow or parse failure need survive. Required-scenario checks alone cannot distinguish that prefix from a finished session. Specify a small external capture-finalization protocol: finish the workload, drain/stop the proxy successfully, then create a private completion receipt tied to the exact raw file; export requires that evidence plus scenario checks. No addon or custom flow format is needed. Missing/failed finalization must retain raw evidence and refuse publication. Gate: stop capture at a complete-record boundary after apparent scenario coverage but before successful finalization; export must refuse. Do not promise detection from FlowReader alone.

P2: the stated resource bound starts after the dependency has allocated the flow. In pinned mitmproxy 12.2.3, FlowReader calls tnetstring.load; tnetstring.load reads the advertised complete record into memory before constructing the Flow. FlowReader also auto-detects HAR and loads that whole document. A 4 MiB decoded-body check and flow/slot caps do not bound that allocation, headers, or imported metadata. Revise the memory claim and put a concrete raw-input/file limit and process memory/time containment before FlowReader, plus bounded decompression before materializing decoded bodies. Keep the upstream decoder. Gate oversized single records/headers, HAR input, and compressed expansion; failures must leave no published fixture or source-valued traceback.

P2: the first slice is larger than these two scenarios require. The 800–1,000 Python / 450–650 test line budgets are estimates, not evidence that this is the simplest implementation. Make the cuts below explicit before implementation; retain the closed contract and regression coverage.

P3: explicitly exclude Flow timing and ancillary state. Say that connection addresses, TLS certificates, Flow metadata, timestamps/durations and capture IDs never enter v3 or diagnostics. Request times may be used privately for ordering checks. Ordering must not serialize absolute times or durations. The existing closed-output rule implies this; an explicit drop rule and sentinel checks make the boundary reviewable.

Concrete first-PR cuts

  • Basic authentication only. Refuse Bearer/Digest exchanges before interpreting their parameter grammars. Keep Basic challenge/authenticated-state behavior and tests. Keep Digest/Apple-header sentinel refusal tests; defer Digest algorithm/qop/domain/nonce-count conversion and Bearer replay support.
  • Refuse Cookie/Set-Cookie sessions. Defer the cookie jar, target bindings, expiry/domain/path grammar and cookie mutation comparator. Keep raw-capture survival and zero-output refusal tests for secret-only variation and removed Secure. Restore successful cookie replay and its flag mutations only in a later supported slice.
  • No URI queries/fragments or URL userinfo support initially. Capture them unchanged; refuse export rather than implementing query registries or userinfo-presence fields. Keep root/relative/HTTP/HTTPS discovery and actual target-href checks. Implement Link conversion only if the two minimal supported scenarios require it; otherwise refuse Link and retain the credential regression.
  • Start with cards actually used by independently seeded CRUD/initial-sync cases. Support required FN/N/UID/VERSION and exercised contact fields/parameters. Refuse unexercised properties, including PHOTO; add individual entries when source evidence requires them. NOTE can be a typed slot when included, with a sentinel regression. Do not build the full listed 29-property surface speculatively.

These cuts trade breadth for explicit unsupported-session refusal. They do not relax capture survival or fixture privacy. Keep discovered Contacts-required fields, the exact Radicale extension decision, legacy dates, CardDAV 403, ordering, seed/bijection checks, limits and publication tests. Do not cut those to meet a line budget.

Test feasibility

Every historical P1 can be automated on Linux without a Mac capture. Use mitmproxy HTTPFlow/FlowWriter for focused exporter cases and stock mitmdump against minimal synthetic HTTP servers for capture integration. Drive a disposable DAV-mounted Calternal server through mitmdump for the product integration gate. A real Apple session remains a separate PENDING gate.

Historical P1 Automated gate
R1 initial content/identity Independently seed two contacts; passing baseline, then FN change, UID swap and UID collapse fail for the intended reason.
R1 href normalization Serve advertised target hrefs; remove /dav or change collection route; replay must reject/fail the actual advertised route without repair.
R1/R2 capabilities OPTIONS with extended-mkcol/MKCALENDAR reaches the client intact and exports; unknown capability survives capture and refuses offline.
R3 credentials Unique sentinels in Apple headers, auth/challenges, Link userinfo, XML/card values and encodings; inspect persisted files and stdout/stderr. Supported forms sanitize; deferred forms refuse without fixture artifacts.
R3 ordinary responses Root/HTTPS redirects, 401 text, depth-0 discovery and correct CardDAV 403 retain upstream status/body in raw capture; supported forms export and replay. Reproduce the exact Radicale QName locally before claiming its export gate passed.

Keep passing seeded baselines before every response mutation, and assert its rejection code/exchange. Synthetic Linux coverage can prove these regressions independently of Apple GUI access; it cannot prove that the chosen profile covers actual Apple traffic.

Nix packaging evidence and limits

No packaging blocker found. At the prescribed nixpkgs pin 1e7da2489c0ba3a06bceec1b444fbc02c5223ab1, mitmproxy is a buildPythonPackage, version 12.2.3; the top-level executable package is its toPythonApplication wrapper. Use pkgs.python3.withPackages (ps: [ ps.mitmproxy ]) for the exporter, with the executable from the same package set. Putting pkgs.mitmproxy on PATH alone is insufficient.

Read-only nix eval --json --impure --expr … via netbird ssh on dev-01 x86_64-linux and dev-02 aarch64-linux successfully evaluated that Python environment's drvPath; both report Python 3.14.7, mitmproxy 12.2.3, and lib.meta.availableOn true for mitmproxy and mitmproxy-rs. This is evaluation evidence, not build/import proof. First implementation gates should run from mitmproxy.io import FlowReader, FlowWriter plus a FlowWriter→FlowReader round trip and mitmdump smoke test on both architectures. No pip or alternate decoder is needed.

No implementation, builds, test runs, commits or pushes were performed in this review. Only package evaluation ran on the VMs. Revise the completion contract, pre-reader resource boundary and first-slice scope; then the architecture is suitable to implement.

## Independent plan review (v3) Reviewed #818 at `dfcd2fdfa577d58a9153760577dcd30c2c45fb1f`, its description, and all three #820 reviews: [72055](https://git.kayg.org/kayg/calternal/pulls/820#issuecomment-72055), [72126](https://git.kayg.org/kayg/calternal/pulls/820#issuecomment-72126), [72191](https://git.kayg.org/kayg/calternal/pulls/820#issuecomment-72191). **Verdict: revise**, preserving the proposed three-part architecture. ### Structural assessment **The architecture removes both repeated mechanisms.** Stock mitmdump has no application schema validation that can turn an ordinary DAV response into a 502. Unsupported semantics now stop offline export while preserving capture. Proxy setup, disk failures and upstream proxy defects remain separate operational failures. For fixtures, closed names plus grammars/typed slots remove the raw-header fallback. The plan explicitly covers free text in allowed headers, XML text/attributes/CDATA, vCard NOTE and parameters, URL userinfo/query/path, Digest material, and error prose. PHOTO/binary content refuses export. No remaining arbitrary-text copy path is authorized by the design. This conclusion depends on constructing new output objects rather than serializing any Flow state or reusing #820's raw variants. Sentinel tests support that boundary; they do not establish it alone. ### Findings P0–P3 **P0:** none found. **P1: capture completeness has no observable completion contract.** The plan promises that interrupted saves/timeouts cannot publish a complete scenario, but a `.mitm` file is a sequence of independent records. FlowReader accepts EOF between records. A proxy stopped after a complete flow, with a later request pending or not yet saved, can leave a readable prefix containing all currently required scenario milestones. No error flow or parse failure need survive. Required-scenario checks alone cannot distinguish that prefix from a finished session. Specify a small external capture-finalization protocol: finish the workload, drain/stop the proxy successfully, then create a private completion receipt tied to the exact raw file; export requires that evidence plus scenario checks. No addon or custom flow format is needed. Missing/failed finalization must retain raw evidence and refuse publication. Gate: stop capture at a complete-record boundary after apparent scenario coverage but before successful finalization; export must refuse. Do not promise detection from FlowReader alone. **P2: the stated resource bound starts after the dependency has allocated the flow.** In pinned mitmproxy 12.2.3, [FlowReader](https://github.com/mitmproxy/mitmproxy/blob/v12.2.3/mitmproxy/io/io.py) calls tnetstring.load; [tnetstring.load](https://github.com/mitmproxy/mitmproxy/blob/v12.2.3/mitmproxy/io/tnetstring.py) reads the advertised complete record into memory before constructing the Flow. FlowReader also auto-detects HAR and loads that whole document. A 4 MiB decoded-body check and flow/slot caps do not bound that allocation, headers, or imported metadata. Revise the memory claim and put a concrete raw-input/file limit and process memory/time containment before FlowReader, plus bounded decompression before materializing decoded bodies. Keep the upstream decoder. Gate oversized single records/headers, HAR input, and compressed expansion; failures must leave no published fixture or source-valued traceback. **P2: the first slice is larger than these two scenarios require.** The 800–1,000 Python / 450–650 test line budgets are estimates, not evidence that this is the simplest implementation. Make the cuts below explicit before implementation; retain the closed contract and regression coverage. **P3: explicitly exclude Flow timing and ancillary state.** Say that connection addresses, TLS certificates, Flow metadata, timestamps/durations and capture IDs never enter v3 or diagnostics. Request times may be used privately for ordering checks. Ordering must not serialize absolute times or durations. The existing closed-output rule implies this; an explicit drop rule and sentinel checks make the boundary reviewable. ### Concrete first-PR cuts - **Basic authentication only.** Refuse Bearer/Digest exchanges before interpreting their parameter grammars. Keep Basic challenge/authenticated-state behavior and tests. Keep Digest/Apple-header sentinel refusal tests; defer Digest algorithm/qop/domain/nonce-count conversion and Bearer replay support. - **Refuse Cookie/Set-Cookie sessions.** Defer the cookie jar, target bindings, expiry/domain/path grammar and cookie mutation comparator. Keep raw-capture survival and zero-output refusal tests for secret-only variation and removed Secure. Restore successful cookie replay and its flag mutations only in a later supported slice. - **No URI queries/fragments or URL userinfo support initially.** Capture them unchanged; refuse export rather than implementing query registries or userinfo-presence fields. Keep root/relative/HTTP/HTTPS discovery and actual target-href checks. Implement Link conversion only if the two minimal supported scenarios require it; otherwise refuse Link and retain the credential regression. - **Start with cards actually used by independently seeded CRUD/initial-sync cases.** Support required FN/N/UID/VERSION and exercised contact fields/parameters. Refuse unexercised properties, including PHOTO; add individual entries when source evidence requires them. NOTE can be a typed slot when included, with a sentinel regression. Do not build the full listed 29-property surface speculatively. These cuts trade breadth for explicit unsupported-session refusal. They do not relax capture survival or fixture privacy. Keep discovered Contacts-required fields, the exact Radicale extension decision, legacy dates, CardDAV 403, ordering, seed/bijection checks, limits and publication tests. Do not cut those to meet a line budget. ### Test feasibility **Every historical P1 can be automated on Linux without a Mac capture.** Use mitmproxy HTTPFlow/FlowWriter for focused exporter cases and stock mitmdump against minimal synthetic HTTP servers for capture integration. Drive a disposable DAV-mounted Calternal server through mitmdump for the product integration gate. A real Apple session remains a separate PENDING gate. | Historical P1 | Automated gate | |---|---| | R1 initial content/identity | Independently seed two contacts; passing baseline, then FN change, UID swap and UID collapse fail for the intended reason. | | R1 href normalization | Serve advertised target hrefs; remove `/dav` or change collection route; replay must reject/fail the actual advertised route without repair. | | R1/R2 capabilities | OPTIONS with extended-mkcol/MKCALENDAR reaches the client intact and exports; unknown capability survives capture and refuses offline. | | R3 credentials | Unique sentinels in Apple headers, auth/challenges, Link userinfo, XML/card values and encodings; inspect persisted files and stdout/stderr. Supported forms sanitize; deferred forms refuse without fixture artifacts. | | R3 ordinary responses | Root/HTTPS redirects, 401 text, depth-0 discovery and correct CardDAV 403 retain upstream status/body in raw capture; supported forms export and replay. Reproduce the exact Radicale QName locally before claiming its export gate passed. | Keep passing seeded baselines before every response mutation, and assert its rejection code/exchange. Synthetic Linux coverage can prove these regressions independently of Apple GUI access; it cannot prove that the chosen profile covers actual Apple traffic. ### Nix packaging evidence and limits **No packaging blocker found.** At the prescribed nixpkgs pin `1e7da2489c0ba3a06bceec1b444fbc02c5223ab1`, [mitmproxy is a buildPythonPackage](https://github.com/NixOS/nixpkgs/blob/1e7da2489c0ba3a06bceec1b444fbc02c5223ab1/pkgs/development/python-modules/mitmproxy/default.nix), version 12.2.3; the top-level executable package is its `toPythonApplication` wrapper. Use `pkgs.python3.withPackages (ps: [ ps.mitmproxy ])` for the exporter, with the executable from the same package set. Putting `pkgs.mitmproxy` on PATH alone is insufficient. Read-only `nix eval --json --impure --expr …` via `netbird ssh` on **dev-01 x86_64-linux and dev-02 aarch64-linux** successfully evaluated that Python environment's `drvPath`; both report Python 3.14.7, mitmproxy 12.2.3, and `lib.meta.availableOn` true for mitmproxy and mitmproxy-rs. This is evaluation evidence, not build/import proof. First implementation gates should run `from mitmproxy.io import FlowReader, FlowWriter` plus a FlowWriter→FlowReader round trip and mitmdump smoke test on both architectures. No pip or alternate decoder is needed. No implementation, builds, test runs, commits or pushes were performed in this review. Only package evaluation ran on the VMs. Revise the completion contract, pre-reader resource boundary and first-slice scope; then the architecture is suitable to implement.
kayg commented 2026-11-01 01:43:37 +00:00
Author
Owner
Copy link

Revised plan v3 in the description against review 72220:

  • Completion now requires a final through-proxy sentinel plus a private receipt tied to the exact closed raw file, successful workload/proxy exit and scenario milestones. Clean EOF and readable prefixes cannot publish.
  • Added pre-FlowReader regular-file/64 MiB checks, 12 MiB native-record pre-scan and bounded reads, HAR rejection, 1 GiB/60-second process containment and bounded decompression.
  • First PR is Basic-only with six exercised contact fields; cookies, Digest/Bearer, Link, URI userinfo/query/fragment and unused card grammars are explicit follow-ups. Their credential regressions remain capture-survival/zero-output-refusal tests.
  • Added explicit ancillary Flow-state exclusion, same-package Nix Python packaging and both-architecture import/round-trip/smoke gates.
  • All first-PR integration gates run on Linux, driving a disposable DAV-mounted own server through stock mitmdump, with every historical P1 covered. Native Apple evidence remains separately PENDING.

New estimated budget: 430–550 Python production lines, 100–180 changed Rust lines, 300–400 new/adapted test lines, plus 50–70 documentation/Nix lines. Estimates do not waive correctness gates. No code, builds, tests or captures changed/run; #820 remains paused.

Revised plan v3 in the description against [review 72220](https://git.kayg.org/kayg/calternal/pulls/818#issuecomment-72220): - Completion now requires a final through-proxy sentinel plus a private receipt tied to the exact closed raw file, successful workload/proxy exit and scenario milestones. Clean EOF and readable prefixes cannot publish. - Added pre-FlowReader regular-file/64 MiB checks, 12 MiB native-record pre-scan and bounded reads, HAR rejection, 1 GiB/60-second process containment and bounded decompression. - First PR is Basic-only with six exercised contact fields; cookies, Digest/Bearer, Link, URI userinfo/query/fragment and unused card grammars are explicit follow-ups. Their credential regressions remain capture-survival/zero-output-refusal tests. - Added explicit ancillary Flow-state exclusion, same-package Nix Python packaging and both-architecture import/round-trip/smoke gates. - All first-PR integration gates run on Linux, driving a disposable DAV-mounted own server through stock mitmdump, with every historical P1 covered. Native Apple evidence remains separately PENDING. New estimated budget: 430–550 Python production lines, 100–180 changed Rust lines, 300–400 new/adapted test lines, plus 50–70 documentation/Nix lines. Estimates do not waive correctness gates. No code, builds, tests or captures changed/run; #820 remains paused.
kayg commented 2026-11-01 06:28:11 +00:00
Author
Owner
Copy link

GUI prerequisites on macos-01-std are done (2026-11-01, IST morning):

  1. Automation (Apple Events) allowed for the ssh login process (it shows up as "NetBird"): System Events, Contacts (control and data access), Mail, Notes. Checked over ssh with no prompt: System Events and Notes, Contacts and Mail scripts all return values.
  2. Test CA trust: no fixed CA file exists. The CA is created fresh per run (mkcert.sh), and slice 0 used a public wildcard cert. Trusting needs no GUI: ~/bin/trust-test-ca.sh <ca.pem> on the Mac (sudo security add-trusted-cert -d -r trustRoot -p ssl -k /Library/Keychains/System.keychain). Proved with a throwaway CA: verify-cert NOT_TRUSTED before, "successful" after; removed again afterwards. Remove a run's CA at the end with sudo security remove-trusted-cert -d ca.pem.
  3. The old slice-0 account ("Example", user imap766/t1) is deleted: Internet Accounts says "No accounts". A leftover certificate warning dialog in Notes was cancelled.

Screen Sharing is enabled on the VM. Reconnect: GUI clicks are easiest through the guest's cua-spacesd (MCP at 127.0.0.1:3211, token in ~/.cua/spacesd/token). The VNC framebuffer showed black, so VNC is not useful here.

GUI prerequisites on `macos-01-std` are done (2026-11-01, IST morning): 1. Automation (Apple Events) allowed for the ssh login process (it shows up as "NetBird"): System Events, Contacts (control and data access), Mail, Notes. Checked over ssh with no prompt: System Events and Notes, Contacts and Mail scripts all return values. 2. Test CA trust: no fixed CA file exists. The CA is created fresh per run (`mkcert.sh`), and slice 0 used a public wildcard cert. Trusting needs no GUI: `~/bin/trust-test-ca.sh <ca.pem>` on the Mac (`sudo security add-trusted-cert -d -r trustRoot -p ssl -k /Library/Keychains/System.keychain`). Proved with a throwaway CA: verify-cert NOT_TRUSTED before, "successful" after; removed again afterwards. Remove a run's CA at the end with `sudo security remove-trusted-cert -d ca.pem`. 3. The old slice-0 account ("Example", user imap766/t1) is deleted: Internet Accounts says "No accounts". A leftover certificate warning dialog in Notes was cancelled. Screen Sharing is enabled on the VM. Reconnect: GUI clicks are easiest through the guest's cua-spacesd (MCP at 127.0.0.1:3211, token in ~/.cua/spacesd/token). The VNC framebuffer showed black, so VNC is not useful here.
kayg commented 2026-11-07 19:33:44 +00:00
Author
Owner
Copy link

housekeeping #1009: rebased+green, left open for review.

rebased DAV recorder plan; proposal retained.

head: 5347df21aab3fe9311fd1b37f2e8c5c3c69f941a. rebased onto dev cd7f84432febfc63b788fbb9b9f9950452982d74 and pushed with the original exact force-with-lease.

validation on the Mac VM, through Nix and the shared Mac lock/slot: root bun install --frozen-lockfile; in apps/web, bun run check, bun run lint, bun run build, bun run test; PERF_DIST=<worktree>/apps/web/dist bash perf/web.sh --bundle-only --gate. all passed. this documentation-only branch has no targeted browser specs or Rust implementation changes. exact commands, logs, and dev bundle control.

original-head missing-commit/patch inventory, file pr-818.md.

housekeeping #1009: rebased+green, left open for review. rebased DAV recorder plan; proposal retained. head: `5347df21aab3fe9311fd1b37f2e8c5c3c69f941a`. rebased onto dev `cd7f84432febfc63b788fbb9b9f9950452982d74` and pushed with the original exact force-with-lease. validation on the Mac VM, through Nix and the shared Mac lock/slot: root `bun install --frozen-lockfile`; in `apps/web`, `bun run check`, `bun run lint`, `bun run build`, `bun run test`; `PERF_DIST=<worktree>/apps/web/dist bash perf/web.sh --bundle-only --gate`. all passed. this documentation-only branch has no targeted browser specs or Rust implementation changes. [exact commands, logs, and dev bundle control](https://git.kayg.org/attachments/64244177-fd81-4992-860e-4d5d22ca01cd). [original-head missing-commit/patch inventory](https://git.kayg.org/attachments/fe3c09dc-dea3-47a0-92e2-b44a777083e5), file `pr-818.md`.
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
Merge conflict checking is in progress. Try again in few moments.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin docs/dav-recorder-v2-plan:docs/dav-recorder-v2-plan
git switch docs/dav-recorder-v2-plan

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch dev
git merge --no-ff docs/dav-recorder-v2-plan
git switch docs/dav-recorder-v2-plan
git rebase dev
git switch dev
git merge --ff-only docs/dav-recorder-v2-plan
git switch docs/dav-recorder-v2-plan
git rebase dev
git switch dev
git merge --no-ff docs/dav-recorder-v2-plan
git switch dev
git merge --squash docs/dav-recorder-v2-plan
git switch dev
git merge --ff-only docs/dav-recorder-v2-plan
git switch dev
git merge docs/dav-recorder-v2-plan
git push origin dev
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!818
No description provided.