PERF: benchmark and optimise per-User WebDAV write serialisation (#457) with fsync (#429) #476

New issue

Open

opened 2026-09-30 04:46:10 +00:00 by kayg · 31 comments

kayg commented 2026-09-30 04:46:10 +00:00

Owner

Copy link

Request (owner, 2026-09-30): "please benchmark this, we want to maximize performance"

#457 serialises each User's WebDAV writes (uploads included) so lock checks cannot race, and keeps locks in memory. Measure what that costs, then optimise.

• Protocol: on the perf VM under flock /root/perf.lock, a release build, interleaved runs (serialised vs a build with the per-User write mutex removed) at N = 1, 4, 8 and 16 parallel PUTs of 1 KiB, 1 MiB and 64 MiB files, one User and then two Users. Report p50/p95 latency, throughput and CPU. Include Finder-like behaviour (LOCK, PUT, UNLOCK per file; .DS_Store and AppleDouble files) using the #457 litmus and the Mac capture.
• Decision rule up front: if serialisation costs more than 10% throughput at N ≥ 4, replace the per-User mutex with a finer lock (per path or per lock token, checked atomically with the write) that keeps the #457 guarantees (a write without the token gets 423; no race). Prove it with a concurrency test and a rerun of the litmus locks suite.
• Tie in #429 (per-file fsync cost): measure both together and report which dominates.
  Gates: per crate (calternal-dav, calternal-fs, calternal-server), the adversarial DAV round, and the numbers verbatim.

## Request (owner, 2026-09-30): "please benchmark this, we want to maximize performance" #457 serialises each User's WebDAV writes (uploads included) so lock checks cannot race, and keeps locks in memory. Measure what that costs, then optimise. - Protocol: on the perf VM under `flock /root/perf.lock`, a release build, interleaved runs (serialised vs a build with the per-User write mutex removed) at N = 1, 4, 8 and 16 parallel PUTs of 1 KiB, 1 MiB and 64 MiB files, one User and then two Users. Report p50/p95 latency, throughput and CPU. Include Finder-like behaviour (LOCK, PUT, UNLOCK per file; `.DS_Store` and AppleDouble files) using the #457 litmus and the Mac capture. - Decision rule up front: if serialisation costs more than 10% throughput at N ≥ 4, replace the per-User mutex with a finer lock (per path or per lock token, checked atomically with the write) that keeps the #457 guarantees (a write without the token gets 423; no race). Prove it with a concurrency test and a rerun of the litmus locks suite. - Tie in #429 (per-file fsync cost): measure both together and report which dominates. Gates: per crate (calternal-dav, calternal-fs, calternal-server), the adversarial DAV round, and the numbers verbatim.

staging-calendar-first-evidence.png

320 KiB

staging-reminders-list.png

585 KiB

staging-finder-final.png

286 KiB

staging-calendar-final.png

267 KiB

calendar-before.png

272 KiB

safari-trust-required.png

168 KiB

profile-pending.png

331 KiB

kayg referenced this issue 2026-09-30 06:05:59 +00:00

calternaldav: break it every way (conformance suites, real Apple clients, cursed data, concurrency, isolation, DoS) #457

kayg referenced this issue 2026-09-30 06:48:57 +00:00

calternaldav: break it every way (conformance suites, real Apple clients, cursed data, concurrency, isolation, DoS) #457

kayg referenced this issue 2026-09-30 07:41:18 +00:00

calternaldav: break it every way (conformance suites, real Apple clients, cursed data, concurrency, isolation, DoS) #457

kayg referenced this issue 2026-09-30 08:43:21 +00:00

calternaldav: break it every way (conformance suites, real Apple clients, cursed data, concurrency, isolation, DoS) #457

kayg referenced this issue 2026-10-01 05:20:15 +00:00

calternaldav: break it every way (conformance suites, real Apple clients, cursed data, concurrency, isolation, DoS) #457

kayg commented 2026-10-01 13:34:02 +00:00

Author

Owner

Copy link

Starting #476 on branch job/webdav-lock-476. Initial HEAD/base SHA: cc25c441b7. I am checking current DAV lock semantics and building a repeatable concurrent PUT profile before narrowing the mutation gate.

Starting #476 on branch job/webdav-lock-476. Initial HEAD/base SHA: cc25c441b7a974185622a1dee853cf38686d2b67. I am checking current DAV lock semantics and building a repeatable concurrent PUT profile before narrowing the mutation gate.

kayg referenced this issue from a commit 2026-10-01 13:54:08 +00:00

bench(webdav): add lock contention profile

kayg commented 2026-10-01 14:03:08 +00:00

Author

Owner

Copy link

Baseline evidence from the isolated perf VM: the 8-way, 1 MB, same-folder, unlocked PUT burst completed at 8.57 MiB/s with p95 PUT latency 889.09 ms. The VM load average inside was 0.28 / 0.14 / 0.09. The first Finder-style lock burst did not produce a timing row: all 8 LOCK requests to new file URLs returned HTTP 403. No 5xx or data-loss result appeared. I am checking lock-null creation against existing-resource LOCK before treating this as a route defect.

Baseline evidence from the isolated perf VM: the 8-way, 1 MB, same-folder, unlocked PUT burst completed at 8.57 MiB/s with p95 PUT latency 889.09 ms. The VM load average inside was 0.28 / 0.14 / 0.09. The first Finder-style lock burst did not produce a timing row: all 8 LOCK requests to new file URLs returned HTTP 403. No 5xx or data-loss result appeared. I am checking lock-null creation against existing-resource LOCK before treating this as a route defect.

kayg commented 2026-10-01 14:03:20 +00:00

Author

Owner

Copy link

Baseline evidence from the isolated perf VM: the 8-way, 1 MB, same-folder, unlocked PUT burst completed at 8.57 MiB/s with p95 PUT latency 889.09 ms. The VM load average inside the perf lock was 0.28 / 0.14 / 0.09. The first Finder-style lock burst did not produce a timing row: all 8 LOCK requests to new file URLs returned HTTP 403. No 5xx or data-loss result appeared. I am checking lock-null creation against existing-resource LOCK before treating this as a route defect.

Baseline evidence from the isolated perf VM: the 8-way, 1 MB, same-folder, unlocked PUT burst completed at 8.57 MiB/s with p95 PUT latency 889.09 ms. The VM load average inside the perf lock was 0.28 / 0.14 / 0.09. The first Finder-style lock burst did not produce a timing row: all 8 LOCK requests to new file URLs returned HTTP 403. No 5xx or data-loss result appeared. I am checking lock-null creation against existing-resource LOCK before treating this as a route defect.

kayg commented 2026-10-01 14:06:59 +00:00

Author

Owner

Copy link

Baseline correction: the first perf service binary is not current enough for #476. Under the VM lock, OPTIONS returned HTTP 200 with DAV class 1 only, and Allow omitted LOCK and UNLOCK. The earlier 8-way unlocked PUT number is not a valid #476 baseline and will be discarded. I stopped that isolated server and will use a release build of the merged branch before recording the baseline matrix.

Baseline correction: the first perf service binary is not current enough for #476. Under the VM lock, OPTIONS returned HTTP 200 with DAV class 1 only, and Allow omitted LOCK and UNLOCK. The earlier 8-way unlocked PUT number is not a valid #476 baseline and will be discarded. I stopped that isolated server and will use a release build of the merged branch before recording the baseline matrix.

kayg commented 2026-10-01 14:26:55 +00:00

Author

Owner

Copy link

Current-source release build first stopped because the ignored production embed directory apps/web/build/ is absent (RustEmbed in crates/calternal-server/src/main.rs:73). The app build then confirmed that workspace JS dependencies are not installed (vite: command not found). I’m installing from the frozen root bun.lock and building the frontend so I can complete the valid lock-enabled perf baseline.

Current-source release build first stopped because the ignored production embed directory `apps/web/build/` is absent (`RustEmbed` in `crates/calternal-server/src/main.rs:73`). The app build then confirmed that workspace JS dependencies are not installed (`vite: command not found`). I’m installing from the frozen root `bun.lock` and building the frontend so I can complete the valid lock-enabled perf baseline.

kayg commented 2026-10-01 14:41:20 +00:00

Author

Owner

Copy link

The current-source class 2 server is staged in the isolated perf instance. At the first measurement attempt, lslocks showed /root/perf.lock held by another Python workload (elapsed 03:46 at inspection), so the baseline matrix is queued on flock; it will record uptime after acquiring the lock. The local source changes do not affect the staged pre-change server.

The current-source class 2 server is staged in the isolated perf instance. At the first measurement attempt, `lslocks` showed `/root/perf.lock` held by another Python workload (elapsed 03:46 at inspection), so the baseline matrix is queued on `flock`; it will record `uptime` after acquiring the lock. The local source changes do not affect the staged pre-change server.

kayg commented 2026-10-01 14:57:56 +00:00

Author

Owner

Copy link

RFC/code review found that the existing custom mutation helper flattened every positive token in If, regardless of its tagged resource URI. dav-server evaluates each list against its tagged path, so using the flattened list for parent-collection checks could let a token for a different resource satisfy that check. I’ve changed the DAV helper to associate tokens with the request or tagged resource path and added a focused regression test; the adversarial round will exercise the behavior through LOCK/PUT/MOVE/DELETE.

RFC/code review found that the existing custom mutation helper flattened every positive token in `If`, regardless of its tagged resource URI. `dav-server` evaluates each list against its tagged path, so using the flattened list for parent-collection checks could let a token for a different resource satisfy that check. I’ve changed the DAV helper to associate tokens with the request or tagged resource path and added a focused regression test; the adversarial round will exercise the behavior through LOCK/PUT/MOVE/DELETE.

kayg commented 2026-10-01 15:19:30 +00:00

Author

Owner

Copy link

Benchmark correction: the pre-change matrix has not started. lslocks and ps on the perf VM showed /root/perf.lock held by this job’s own /run/webdav-lock-476-log.py, which inherited the lock file descriptor during setup; a queued baseline was waiting behind it. I am restarting the isolated log collector without the inherited descriptor. No benchmark results from this attempt are valid.

Benchmark correction: the pre-change matrix has not started. `lslocks` and `ps` on the perf VM showed `/root/perf.lock` held by this job’s own `/run/webdav-lock-476-log.py`, which inherited the lock file descriptor during setup; a queued baseline was waiting behind it. I am restarting the isolated log collector without the inherited descriptor. No benchmark results from this attempt are valid.

kayg commented 2026-10-01 15:22:19 +00:00

Author

Owner

Copy link

Correction: I found both the isolated server launcher (runuser) and log collector inherited the /root/perf.lock descriptor. I restarted both outside the lock. One queued attempt acquired the lock while the server was restarting, failed its OPTIONS connection, and exited before writing a benchmark result; that attempt is discarded. The valid pre-change matrix has now started under the lock. The staged server SHA-256 matches the local pre-change release: 487bc19e183bd17d7f21a61b7bf9fe983843d85e372dae27bb7aefadd13a4344.

Correction: I found both the isolated server launcher (`runuser`) and log collector inherited the `/root/perf.lock` descriptor. I restarted both outside the lock. One queued attempt acquired the lock while the server was restarting, failed its OPTIONS connection, and exited before writing a benchmark result; that attempt is discarded. The valid pre-change matrix has now started under the lock. The staged server SHA-256 matches the local pre-change release: `487bc19e183bd17d7f21a61b7bf9fe983843d85e372dae27bb7aefadd13a4344`.

kayg commented 2026-10-01 15:28:44 +00:00

Author

Owner

Copy link

Pre-change ext4 baseline finding (perf VM, load sampled while /root/perf.lock was held): 32 concurrent unlocked 100 MB PUTs into different folders completed in 87.48 s, with 84.91 s p95 per PUT and 34.89 MiB/s aggregate throughput. The same-folder case was comparable. This is the global write contention #476 targets. The matrix is finishing its remaining locked/different-folder row; full before/after data will follow.

Pre-change ext4 baseline finding (perf VM, load sampled while `/root/perf.lock` was held): 32 concurrent unlocked 100 MB PUTs into different folders completed in 87.48 s, with 84.91 s p95 per PUT and 34.89 MiB/s aggregate throughput. The same-folder case was comparable. This is the global write contention #476 targets. The matrix is finishing its remaining locked/different-folder row; full before/after data will follow.

kayg commented 2026-10-01 15:56:01 +00:00

Author

Owner

Copy link

HDD emulation baseline (all 8 combinations at 1 MB, on /dev/mapper/calternal-hdd-emu, 8 ms read/write delay) is complete under /root/perf.lock. The 32-upload same-folder case took 92.65 s unlocked and 180.37 s with LOCK; locked throughput was 0.17 MiB/s and p95 PUT latency was 111.97 s. A linear projection for one 32 × 100 MB case is over five hours, so the 100 MB HDD rows are not measured and will remain a reported timebox gap; I am not treating the projection as a result. The complete 100 MB matrix continues on ext4.

HDD emulation baseline (all 8 combinations at 1 MB, on `/dev/mapper/calternal-hdd-emu`, 8 ms read/write delay) is complete under `/root/perf.lock`. The 32-upload same-folder case took 92.65 s unlocked and 180.37 s with LOCK; locked throughput was 0.17 MiB/s and p95 PUT latency was 111.97 s. A linear projection for one 32 × 100 MB case is over five hours, so the 100 MB HDD rows are not measured and will remain a reported timebox gap; I am not treating the projection as a result. The complete 100 MB matrix continues on ext4.

kayg commented 2026-10-01 15:58:06 +00:00

Author

Owner

Copy link

Current-source ext4 benchmark finding: the first two 8-worker, 1 MB same-folder cases passed (unlocked throughput 8.33 MiB/s; locked 5.68 MiB/s). The 32-worker same-folder unlocked case returned HTTP 500 for 9 of 32 PUTs. The benchmark aborted, so its partial after results are invalid and no after matrix has been recorded. I am tracing and fixing this 5xx before restarting performance or adversarial probes.

Current-source ext4 benchmark finding: the first two 8-worker, 1 MB same-folder cases passed (unlocked throughput 8.33 MiB/s; locked 5.68 MiB/s). The 32-worker same-folder unlocked case returned HTTP 500 for 9 of 32 PUTs. The benchmark aborted, so its partial after results are invalid and no after matrix has been recorded. I am tracing and fixing this 5xx before restarting performance or adversarial probes.

kayg commented 2026-10-01 16:05:59 +00:00

Author

Owner

Copy link

Code finding while tracing the 500s: FileDavError::ServiceUnavailable and FileDavError::Internal both become FsError::GeneralFailure, so the DAV response loses its original status and has no body. This may explain the observed 500, but I have not confirmed the source failure yet. I added temporary-safe diagnostics in the Files DAV adapter that log only the static status, error code and message, without a User, path, payload or token. I will keep only the diagnostics that help the finished route remain operable.

Code finding while tracing the 500s: `FileDavError::ServiceUnavailable` and `FileDavError::Internal` both become `FsError::GeneralFailure`, so the DAV response loses its original status and has no body. This may explain the observed 500, but I have not confirmed the source failure yet. I added temporary-safe diagnostics in the Files DAV adapter that log only the static status, error code and message, without a User, path, payload or token. I will keep only the diagnostics that help the finished route remain operable.

kayg commented 2026-10-01 16:32:34 +00:00

Author

Owner

Copy link

Source inspection found that Files permits only 16 active uploads per User (MAX_ACTIVE_UPLOADS), while the requested Finder profile opens 32 simultaneous PUTs. A rejected slot uses HTTP 429, which the DAV adapter maps to FileDavError::ServiceUnavailable; the DAV callback layer currently turns that into a generic 500. The 32-PUT probe saw 13/32 responses as 500, consistent with the slot limit, although the perf VM is now unreachable so its new diagnostic log is not available to confirm the exact code. I am raising the slot limit above the requested burst, preserving 503 for genuine service-unavailable errors, and adding a 32-upload regression test. SSH to 10.69.69.63 currently returns No route to host; I will retry before final measurements.

Source inspection found that Files permits only 16 active uploads per User (`MAX_ACTIVE_UPLOADS`), while the requested Finder profile opens 32 simultaneous PUTs. A rejected slot uses HTTP 429, which the DAV adapter maps to `FileDavError::ServiceUnavailable`; the DAV callback layer currently turns that into a generic 500. The 32-PUT probe saw 13/32 responses as 500, consistent with the slot limit, although the perf VM is now unreachable so its new diagnostic log is not available to confirm the exact code. I am raising the slot limit above the requested burst, preserving 503 for genuine service-unavailable errors, and adding a 32-upload regression test. SSH to 10.69.69.63 currently returns `No route to host`; I will retry before final measurements.

kayg commented 2026-10-01 17:08:47 +00:00

Author

Owner

Copy link

The local WebDAV probe exposed a lock-scope bug: a token-authorized PUT below a Depth-infinity Home lock returned 423 before the DAV lock system checked the child URL. The extra parent-membership check treated the recursive lock as a direct Depth-0 collection lock. I changed the check to cover only a direct Depth-0 parent lock; DAV still checks Depth-infinity locks against the member URL. The 32-worker status storm also reached the four-permit Argon2 verifier's bounded wait and returned its intended 429 before lock checks. I narrowed the conflict probe to eight concurrent operations so it tests lock statuses; the PUT benchmark still exercises 32 concurrent transfers.

The local WebDAV probe exposed a lock-scope bug: a token-authorized PUT below a Depth-infinity Home lock returned 423 before the DAV lock system checked the child URL. The extra parent-membership check treated the recursive lock as a direct Depth-0 collection lock. I changed the check to cover only a direct Depth-0 parent lock; DAV still checks Depth-infinity locks against the member URL. The 32-worker status storm also reached the four-permit Argon2 verifier's bounded wait and returned its intended 429 before lock checks. I narrowed the conflict probe to eight concurrent operations so it tests lock statuses; the PUT benchmark still exercises 32 concurrent transfers.

kayg commented 2026-10-01 17:22:48 +00:00

Author

Owner

Copy link

The focused real-server WebDAV adversarial rerun passed after the path-scope and Depth-infinity fixes: WebDAV scripted probes passed. Litmus completed 41 checks with 36 passing and the same five recognized failures: owner_modify (three cases), complex_cond_put, and fail_complex_cond_put (all documented as known by the probe). The LOCK/PUT/MOVE/DELETE contention storm returned the expected 423 responses, and the stale strong ETag race returned 412.

The perf VM remains unreachable (ssh root@10.69.69.63: No route to host). I am running the requested before/after matrix locally against the same ext4 host and real DAV fixture, with per-case host load recorded. The HDD emulation exists only on that VM, so it cannot be measured in this run.

The focused real-server WebDAV adversarial rerun passed after the path-scope and Depth-infinity fixes: `WebDAV scripted probes passed`. Litmus completed 41 checks with 36 passing and the same five recognized failures: `owner_modify` (three cases), `complex_cond_put`, and `fail_complex_cond_put` (all documented as known by the probe). The LOCK/PUT/MOVE/DELETE contention storm returned the expected 423 responses, and the stale strong ETag race returned 412. The perf VM remains unreachable (`ssh root@10.69.69.63`: `No route to host`). I am running the requested before/after matrix locally against the same ext4 host and real DAV fixture, with per-case host load recorded. The HDD emulation exists only on that VM, so it cannot be measured in this run.

kayg commented 2026-10-01 17:51:27 +00:00

Author

Owner

Copy link

Implementation decisions for #476 where DESIGN did not prescribe the internal gate layout:

• Keep the Files API upload cap at 16. Give DAV a separate cap of 64 so Finder can stage 32 parallel PUTs with headroom without changing Files API behavior.
• Key mutation gates by User and canonical DAV resource bytes. Same-resource writes and tree removals take exclusive gates; distinct collection members share the parent membership gate. Acquire gates in key order so MOVE and COPY cannot deadlock.
• Apply a direct collection lock to member URL changes for Depth 0. Let Depth infinity cover member resources through the normal DAV lock check, as RFC 4918 §7.4 requires.
• Bound in-memory live lock state to 1024 leases per User. Each lease has a finite maximum timeout.

Implementation decisions for #476 where DESIGN did not prescribe the internal gate layout: - Keep the Files API upload cap at 16. Give DAV a separate cap of 64 so Finder can stage 32 parallel PUTs with headroom without changing Files API behavior. - Key mutation gates by User and canonical DAV resource bytes. Same-resource writes and tree removals take exclusive gates; distinct collection members share the parent membership gate. Acquire gates in key order so MOVE and COPY cannot deadlock. - Apply a direct collection lock to member URL changes for Depth 0. Let Depth infinity cover member resources through the normal DAV lock check, as RFC 4918 §7.4 requires. - Bound in-memory live lock state to 1024 leases per User. Each lease has a finite maximum timeout.

kayg commented 2026-10-01 18:10:27 +00:00

Author

Owner

Copy link

The local pre-change benchmark exposed the 32-session cap: in the 100M / same folder / n=32 / unlocked case, 6 of 32 PUTs returned HTTP 429. The old DAV provider called the shared Files API upload creator, which enforced MAX_ACTIVE_UPLOADS = 16. Ten earlier cases completed. This is a before-side failure, not a current-source result. I am updating the benchmark to record failed upload counts and continue so it can finish all cases and compare the DAV-specific cap of 64.

The local pre-change benchmark exposed the 32-session cap: in the `100M / same folder / n=32 / unlocked` case, 6 of 32 PUTs returned HTTP 429. The old DAV provider called the shared Files API upload creator, which enforced `MAX_ACTIVE_UPLOADS = 16`. Ten earlier cases completed. This is a before-side failure, not a current-source result. I am updating the benchmark to record failed upload counts and continue so it can finish all cases and compare the DAV-specific cap of 64.

kayg commented 2026-10-01 18:18:07 +00:00

Author

Owner

Copy link

Correction to my previous benchmark comment: a later run of the same pre-change binary and same 100 MB / same-folder / n=32 / unlocked case completed 32 of 32 PUTs. It took 226.24 seconds with p95 PUT latency 219,937.83 ms; load average was 19.79 at start and 14.23 at end. The old DAV path does share the 16-session Files API cap, but these runs do not establish that it caused the earlier six 429s. I am recording overload statuses by case and will report the initial 429 as transient rather than attribute it to that cap.

Correction to my previous benchmark comment: a later run of the same pre-change binary and same 100 MB / same-folder / n=32 / unlocked case completed 32 of 32 PUTs. It took 226.24 seconds with p95 PUT latency 219,937.83 ms; load average was 19.79 at start and 14.23 at end. The old DAV path does share the 16-session Files API cap, but these runs do not establish that it caused the earlier six 429s. I am recording overload statuses by case and will report the initial 429 as transient rather than attribute it to that cap.

kayg commented 2026-10-01 19:10:35 +00:00

Author

Owner

Copy link

Local WebDAV lock benchmark finding for #476:

The perf VM was unreachable (ssh returned No route to host), so this is a local ext4 measurement with concurrent host work. The 100 MB same-folder, 32-worker unlocked case accepted 32/32 PUTs in 271.77 s at 11.23 MiB/s; PUT p95 was 260.72 s (host load 21.72 to 21.60). The matching locked case accepted 31/32 PUTs, then recorded 31 transport timeouts and one connection reset across its LOCK/PUT/UNLOCK transactions; it took 1,231.34 s and host load rose from 21.60 to 33.31. The local comparison is load-confounded. I am continuing the remaining matrix pairs once; no rerun is planned.

Local WebDAV lock benchmark finding for #476: The perf VM was unreachable (`ssh` returned `No route to host`), so this is a local ext4 measurement with concurrent host work. The 100 MB same-folder, 32-worker unlocked case accepted 32/32 PUTs in 271.77 s at 11.23 MiB/s; PUT p95 was 260.72 s (host load 21.72 to 21.60). The matching locked case accepted 31/32 PUTs, then recorded 31 transport timeouts and one connection reset across its LOCK/PUT/UNLOCK transactions; it took 1,231.34 s and host load rose from 21.60 to 33.31. The local comparison is load-confounded. I am continuing the remaining matrix pairs once; no rerun is planned.

kayg commented 2026-10-01 19:12:24 +00:00

Author

Owner

Copy link

Additional local diagnostic during the 100 MB, 32-worker locked case:

Unauthenticated DAV OPTIONS returned 401 in 35 ms, and an authenticated PROPFIND returned 207 in 333 ms. A single authenticated LOCK to a different absent path timed out after 15 s through the proxy and again through the backend; LOCK on an existing benchmark file and an empty PUT to a separate path also timed out after 15 s. These probes were not part of the measured matrix. They show that reads/authentication stayed responsive while lock and write requests stalled under the concurrent run. Host load was above 20. I am letting the one local matrix finish to capture whether later cases recover; cause is not established yet.

Additional local diagnostic during the 100 MB, 32-worker locked case: Unauthenticated DAV OPTIONS returned 401 in 35 ms, and an authenticated PROPFIND returned 207 in 333 ms. A single authenticated LOCK to a different absent path timed out after 15 s through the proxy and again through the backend; LOCK on an existing benchmark file and an empty PUT to a separate path also timed out after 15 s. These probes were not part of the measured matrix. They show that reads/authentication stayed responsive while lock and write requests stalled under the concurrent run. Host load was above 20. I am letting the one local matrix finish to capture whether later cases recover; cause is not established yet.

kayg commented 2026-10-01 19:16:03 +00:00

Author

Owner

Copy link

The local matrix is now at 12/16 cases. The next 100 MB, different-folder, 8-worker case has not reached measurement: its path-preparation MKCOL is still waiting after more than 7 minutes, before the benchmark creates upload workers. The host one-minute load average is about 31. An unauthenticated DAV OPTIONS and authenticated PROPFIND still returned promptly. I am preserving this run's results and will not repeat it to seek a quieter host.

The local matrix is now at 12/16 cases. The next 100 MB, different-folder, 8-worker case has not reached measurement: its path-preparation MKCOL is still waiting after more than 7 minutes, before the benchmark creates upload workers. The host one-minute load average is about 31. An unauthenticated DAV OPTIONS and authenticated PROPFIND still returned promptly. I am preserving this run's results and will not repeat it to seek a quieter host.

kayg commented 2026-10-01 19:25:01 +00:00

Author

Owner

Copy link

#476 final report

Head: 3eb63adb2b54633c96d5064ef2c2ca17be03fe85

Built

• DAV writes now take ordered gates for the exact resource, its collection membership, and affected subtrees. Sibling resource writes can share parent checks; overlapping moves, deletes, locks, and writes still wait.
• Added RFC 4918 Depth 0 parent-lock checks and kept Depth infinity member checks in the DAV lock system. The adversarial probe now races LOCK, PUT, MOVE, and DELETE on overlapping paths, and checks stale 412 and complete-file outcomes.
• Kept the Files API cap at 16 active uploads. DAV has a separate cap of 64 so Finder can stage 32 parallel PUTs.
• Recorded the available before/after measurements in docs/perf/baseline.json.

Files: crates/calternal-dav/src/files.rs, crates/plugins/files/src/dav.rs, crates/plugins/files/src/uploads.rs, tests/adversarial/webdav.py, bench/webdav-lock-476.py, docs/perf/baseline.json.

Performance

The perf VM returned No route to host, so both binaries ran on the same local ext4 host. The baseline has 16 before cases and 12 after cases. For 1 MB, different-folder, 32-worker unlocked PUTs, throughput changed from 1.77 to 4.50 MiB/s and PUT p95 from 17,006.76 to 6,633.63 ms; both runs accepted 32/32 uploads.

The large-file results are load-confounded. Same-folder, 100 MB, 32-worker unlocked PUTs changed from 13.49 MiB/s and 219,937.83 ms p95 to 11.23 MiB/s and 260,718.69 ms p95; both accepted 32/32. In the matching locked after case, 31/32 PUTs succeeded, and the client transactions recorded 31 timeouts and one connection reset over 1,231.34 s as host load rose from 21.60 to 33.31.

The next after case timed out during folder setup: its MKCOL waited for the benchmark client's 900 second timeout before measurement. The last three 100 MB different-folder cases are marked unmeasured in the baseline. I did not repeat the local matrix. The perf VM and HDD emulation remain unmeasured.

Gates and probes

cargo fmt --all --check
(no output; exit 0)

cargo clippy -p calternal-dav --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 16.24s

cargo test -p calternal-dav
test result: ok. 45 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.14s
test result: ok. 36 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04s

cargo clippy -p calternal-plugin-files --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 12.36s

cargo test -p calternal-plugin-files
147 passed; 0 failed; 1 ignored (reported result; the full harness line and duration were not retained)

The real-server WebDAV scripted probe passed. Litmus reported 36/41; its five known 4xx exceptions remain: three dead-property PROPPATCH cases return 403, and two long conditional PUT cases return 400 because Litmus 0.13 truncates the ETag-bearing If header. The configured probe treats those as known exceptions.

Decisions and gaps

• Keep locks in memory, scoped by User and canonical DAV resource path, with a 1,024 live-lock cap per User and a finite timeout of up to 600 seconds.
• Use shared parent membership gates for sibling members, exclusive resource gates for direct writes, and exclusive subtree gates for tree removal or Depth infinity lock scope. Acquire gate keys in order.
• Keep the Files API upload cap at 16 and use 64 only for DAV.
• Use local ext4 measurements because the designated perf VM was unreachable. Do not treat the overloaded 100 MB comparison as a stable speedup measurement.
• Known gap: four after cases have no measurements because the case setup MKCOL timed out under high host load. The stalled local write/lock behavior is recorded on this issue and in the baseline for owner review.

#476 final report Head: `3eb63adb2b54633c96d5064ef2c2ca17be03fe85` ## Built - DAV writes now take ordered gates for the exact resource, its collection membership, and affected subtrees. Sibling resource writes can share parent checks; overlapping moves, deletes, locks, and writes still wait. - Added RFC 4918 Depth 0 parent-lock checks and kept Depth infinity member checks in the DAV lock system. The adversarial probe now races LOCK, PUT, MOVE, and DELETE on overlapping paths, and checks stale 412 and complete-file outcomes. - Kept the Files API cap at 16 active uploads. DAV has a separate cap of 64 so Finder can stage 32 parallel PUTs. - Recorded the available before/after measurements in `docs/perf/baseline.json`. Files: `crates/calternal-dav/src/files.rs`, `crates/plugins/files/src/dav.rs`, `crates/plugins/files/src/uploads.rs`, `tests/adversarial/webdav.py`, `bench/webdav-lock-476.py`, `docs/perf/baseline.json`. ## Performance The perf VM returned `No route to host`, so both binaries ran on the same local ext4 host. The baseline has 16 before cases and 12 after cases. For 1 MB, different-folder, 32-worker unlocked PUTs, throughput changed from 1.77 to 4.50 MiB/s and PUT p95 from 17,006.76 to 6,633.63 ms; both runs accepted 32/32 uploads. The large-file results are load-confounded. Same-folder, 100 MB, 32-worker unlocked PUTs changed from 13.49 MiB/s and 219,937.83 ms p95 to 11.23 MiB/s and 260,718.69 ms p95; both accepted 32/32. In the matching locked after case, 31/32 PUTs succeeded, and the client transactions recorded 31 timeouts and one connection reset over 1,231.34 s as host load rose from 21.60 to 33.31. The next after case timed out during folder setup: its MKCOL waited for the benchmark client's 900 second timeout before measurement. The last three 100 MB different-folder cases are marked unmeasured in the baseline. I did not repeat the local matrix. The perf VM and HDD emulation remain unmeasured. ## Gates and probes ```text cargo fmt --all --check (no output; exit 0) cargo clippy -p calternal-dav --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 16.24s cargo test -p calternal-dav test result: ok. 45 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.14s test result: ok. 36 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04s cargo clippy -p calternal-plugin-files --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 12.36s cargo test -p calternal-plugin-files 147 passed; 0 failed; 1 ignored (reported result; the full harness line and duration were not retained) ``` The real-server WebDAV scripted probe passed. Litmus reported 36/41; its five known 4xx exceptions remain: three dead-property PROPPATCH cases return 403, and two long conditional PUT cases return 400 because Litmus 0.13 truncates the ETag-bearing If header. The configured probe treats those as known exceptions. ## Decisions and gaps - Keep locks in memory, scoped by User and canonical DAV resource path, with a 1,024 live-lock cap per User and a finite timeout of up to 600 seconds. - Use shared parent membership gates for sibling members, exclusive resource gates for direct writes, and exclusive subtree gates for tree removal or Depth infinity lock scope. Acquire gate keys in order. - Keep the Files API upload cap at 16 and use 64 only for DAV. - Use local ext4 measurements because the designated perf VM was unreachable. Do not treat the overloaded 100 MB comparison as a stable speedup measurement. - Known gap: four after cases have no measurements because the case setup MKCOL timed out under high host load. The stalled local write/lock behavior is recorded on this issue and in the baseline for owner review.

kayg referenced this issue 2026-10-02 06:35:28 +00:00

PERF: tab switch to fully painted on a hard disk with heavy data (Calendar much slower than Photos); stress, measure, fix #549

kayg commented 2026-10-02 10:28:30 +00:00

Author

Owner

Copy link

Results

Profile	dev c4a61e8	job/webdav-lock-476 3eb63ad
WebDAV PUT matrix: 1 MB/100 MB, 32 workers, same/different folder, locked/unlocked	Not measured before the reporting cutoff	Not measured before the reporting cutoff

No latency, CPU or RSS samples exist; no regression is inferred. The six interleaved A/B runs needed for this matrix did not fit in the perf-VM time window.

Command

The prepared runner would invoke /root/perf-rerun/run_aux_profile.py webdav --server <dev-or-feature-release-binary> --output /root/perf-rerun/output/webdav-<label>.json --commit <matching-SHA>. It seeds a real User and runs the eight requested matrix cases with --sizes 1M,100M --concurrency 32 --layouts same,different --locks off,on --continue-on-failure, under /root/hdd-emu.sh run-limited, TMPDIR=/srv/hdd-emu/tmp and flock -w 14400 /root/perf.lock.

## Results | Profile | dev `c4a61e8` | `job/webdav-lock-476` `3eb63ad` | | --- | --- | --- | | WebDAV PUT matrix: 1 MB/100 MB, 32 workers, same/different folder, locked/unlocked | Not measured before the reporting cutoff | Not measured before the reporting cutoff | No latency, CPU or RSS samples exist; no regression is inferred. The six interleaved A/B runs needed for this matrix did not fit in the perf-VM time window. ## Command The prepared runner would invoke `/root/perf-rerun/run_aux_profile.py webdav --server <dev-or-feature-release-binary> --output /root/perf-rerun/output/webdav-<label>.json --commit <matching-SHA>`. It seeds a real User and runs the eight requested matrix cases with `--sizes 1M,100M --concurrency 32 --layouts same,different --locks off,on --continue-on-failure`, under `/root/hdd-emu.sh run-limited`, `TMPDIR=/srv/hdd-emu/tmp` and `flock -w 14400 /root/perf.lock`.

kayg referenced this issue 2026-10-02 10:30:31 +00:00

PERF: tab switch to fully painted on a hard disk with heavy data (Calendar much slower than Photos); stress, measure, fix #549

kayg referenced this issue 2026-10-02 13:08:54 +00:00

PERF: Files startup reconciliation holds the shared mutation lock across a whole Home (#663) #750

kayg referenced this issue 2026-10-02 13:11:39 +00:00

PERF: isolate WebDAV file reads from the global mutation queue (#663) #800

kayg referenced this issue 2026-10-02 13:18:27 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg commented 2026-10-02 14:45:05 +00:00

Author

Owner

Copy link

Starting the macinterop job requested by the owner. Scope: real Calendar, Reminders, Notes and Finder tests, with defects filed separately; no product optimisation in this job. Tested build is detached origin/job/merge-round-7a at daea49a7f0. Original job branch: job/macinterop. Mac lock acquired. Existing 2026-10-01/02 lab results are reference only; this run will retest the requested build.

Starting the macinterop job requested by the owner. Scope: real Calendar, Reminders, Notes and Finder tests, with defects filed separately; no product optimisation in this job. Tested build is detached origin/job/merge-round-7a at daea49a7f0a16fbc382614761e5f9f05ec077c7b. Original job branch: job/macinterop. Mac lock acquired. Existing 2026-10-01/02 lab results are reference only; this run will retest the requested build.

kayg commented 2026-10-02 15:00:57 +00:00

Author

Owner

Copy link

Lab finding: the old local data directory and TLS key files recorded in macdav-lab/start-server.sh do not exist after the reboot. The Mac still has the calternal profile, its cached calendars and the old trusted CA. A fresh User needs a new lab profile and certificate. The real Calendar window currently shows a connection warning; this is lab state, not evidence of a product defect. Production web build passed. The server build ended with signal 15 before a binary was produced; retrying once, with four build jobs and without the stalled shared compiler cache.

Lab finding: the old local data directory and TLS key files recorded in macdav-lab/start-server.sh do not exist after the reboot. The Mac still has the calternal profile, its cached calendars and the old trusted CA. A fresh User needs a new lab profile and certificate. The real Calendar window currently shows a connection warning; this is lab state, not evidence of a product defect. Production web build passed. The server build ended with signal 15 before a binary was produced; retrying once, with four build jobs and without the stalled shared compiler cache.

kayg commented 2026-10-02 15:27:39 +00:00

Author

Owner

Copy link

Status: BLOCKED. The real-client interop matrix is not complete.

The Mac is reachable and runs macOS 27.0 (26A428), with cua-driver 0.30.2. I held the Mac lock for all Mac sessions. The existing calternal profile and cached calendars remain on the Mac. The local data and TLS key files named by the old lab scripts do not exist after the reboot. A replacement certificate reaches the Mac, but the Mac does not trust its new CA.

Build and files

• Checkout/head: daea49a7f0a16fbc382614761e5f9f05ec077c7b, detached from origin/job/merge-round-7a.
• The worktree server build did not complete. The initial build ended with signal 15; the retry was stopped when the trust blocker was established.
• For lab preparation I used the merge job's compiled server, read only. /api/v1/system/info reported 2f4482ded066d9c5d9c59130377907f7fd2916c9. git diff 2f4482ded..daea49a7f contains only 11 added lines in tests/adversarial/authz_matrix.py and tests/adversarial/xuser_matrix.py; product code is identical. This is not a claim that the requested checkout's own build passed.
• The production web build passed. The real passkey APIs created a disposable User, App Password and Mac configuration profile. The profile was copied privately to Mac Downloads and opened, but was not installed.
• No tracked files changed and no commits were made: this job was testing only. No product fix, push, deploy or merge was performed.
• Evidence and local preparation scripts: artifacts/macinterop/ (ignored). Screenshots are attached below.

App × flow matrix

App	Flow	Result / issue
Lab setup	Mac access, existing profile and cached calendars	PASS, read only; does not prove authentication
Lab setup	HTTPS with normal certificate verification	BLOCKED: replacement CA is not trusted; lab prerequisite, no product issue
Calendar	Fresh account discovery	UNTESTED
Calendar	Create, edit, move and delete timed events	UNTESTED
Calendar	All-day, recurrence with an exception, alarm and time zone	UNTESTED
Calendar	Web ↔ Mac edits and refresh	UNTESTED
Calendar	Two calendars and calendar deletion	UNTESTED
Reminders	Create, complete, due dates and lists	UNTESTED
Reminders	Tasks ↔ Mac round trip	UNTESTED
Notes	Create, edit and delete on both sides	UNTESTED
Notes	Attachments, folders and duplicate checks across sync cycles	UNTESTED
Finder	Cmd+K, mixed folder, 500 MB file, NFD/emoji/spaces	UNTESTED
Finder	TextEdit open/save, rename, move, delete and Trash	UNTESTED
Finder	Hidden Apple metadata and disconnect/reconnect	UNTESTED

No new product defect is confirmed. Existing #643–#648 results in the previous lab notes were not retested and are not counted as passes or failures for this run.

Evidence and logs

• Real Calendar window: cached lab calendars and connection warning.
• Real Safari window: lab secure connection failure. This image was taken during tunnel diagnosis; it is not evidence of a server API defect.
• Device Management: replacement profile downloaded, not installed.
• After the tunnel was corrected, real Mac curl reported SSL certificate problem: self signed certificate in certificate chain and curl: (60) SSL certificate problem: self signed certificate in certificate chain. The transcript is in artifacts/macinterop/mac-tls-verification.txt.
• Zero DAV requests reached the server. The proxy recorded one 502 from a local health check before the upstream server was started; this is a lab startup error, not a product finding. There is no evidence for any DAV retry or request-latency result.
• Server startup logged slow SQLite acquisition/queries on this busy host. These are SLOW-only observations, not an interop or benchmark result.

Owner recovery steps

1. Prefer restoring the old lab CA signing key and leaf certificate, if a backup exists. This preserves the Mac's existing lab certificate trust.
2. Otherwise use the replacement public root.pem in ~/.local/state/codex-jobs/calternal/macdav-lab/macinterop-2026-10-02-tls/. Import only that lab CA into Keychain Access → System, open its Trust settings, set it to Always Trust, and complete the macOS authorization prompt. The SHA-256 fingerprint is 95:33:54:D8:43:45:EE:89:EE:46:4A:81:4C:69:E3:37:92:27:06:AE:FC:F1:84:92:17:11:75:39:A8:AD:AA:DD. This certificate expires on 2026-10-05 at 14:54:55 UTC; replace it if recovery occurs later. I did not change certificate trust or other Mac settings.
3. A subsequent job must start a fresh disposable server, create a fresh User and App Password, and replace only the old calternal lab profile in System Settings → General → Device Management. The profile from this run is invalid after cleanup and must not be installed. macOS may show its downloaded copy until the pending profile expires.
4. Run the full matrix above with normal TLS verification. The job's Mac rule requires these extra trust/authorization steps to be listed for the owner, not performed through an unavailable password prompt.

Gate output, verbatim

✓ built in 1m 16s

> Using @sveltejs/adapter-static
  Wrote site to "build"
  ✔ done

cargo fmt --check exit: 0
     Removed 3797 files, 1.6GiB total
cargo clean exit: 0

No crate or web source changed, so per-crate clippy/test and web check/test were not run. No final integration merge was needed for a source change; the detached build under test was kept unchanged. No feature or route was added, so no bench profile was added. No adversarial product round ran: the required real-client prerequisite did not pass.

Cleanup, verified

MAC_TEMPORARY_FILES_REMOVED
PORT CLOSED 18088
PORT CLOSED 8443
PORT CLOSED 31993
PORT CLOSED 31465
LAB_DATA_AND_CREDENTIALS_REMOVED
Request count: 1 DAV request count: 0

The lab server, TLS proxy and three tunnels are stopped. The disposable Home, index, App Password/profile/session files and web output are removed. The retained server log has its setup token redacted. The Mac lock is released. Only private lab TLS recovery material is kept outside the repo; it contains no User data.

Decisions: treat connection/trust failures as lab prerequisites, not product defects; use the existing binary only after proving product-code equality and report its embedded revision; retain lab TLS material privately so the next run can reuse the same CA. No product design decision was made.

UX gaps closed: none; no product UI was changed. UX gaps left / known gaps: all requested authenticated client flows, all prior defect retests, the 500 MB transfer, sync-cycle duplicate checks and latency/retry observations remain untested. This report does not satisfy the real-client deploy gate.

**Status: BLOCKED. The real-client interop matrix is not complete.** The Mac is reachable and runs macOS 27.0 (26A428), with cua-driver 0.30.2. I held the Mac lock for all Mac sessions. The existing calternal profile and cached calendars remain on the Mac. The local data and TLS key files named by the old lab scripts do not exist after the reboot. A replacement certificate reaches the Mac, but the Mac does not trust its new CA. **Build and files** - Checkout/head: `daea49a7f0a16fbc382614761e5f9f05ec077c7b`, detached from `origin/job/merge-round-7a`. - The worktree server build did not complete. The initial build ended with signal 15; the retry was stopped when the trust blocker was established. - For lab preparation I used the merge job's compiled server, read only. `/api/v1/system/info` reported `2f4482ded066d9c5d9c59130377907f7fd2916c9`. `git diff 2f4482ded..daea49a7f` contains only 11 added lines in `tests/adversarial/authz_matrix.py` and `tests/adversarial/xuser_matrix.py`; product code is identical. This is not a claim that the requested checkout's own build passed. - The production web build passed. The real passkey APIs created a disposable User, App Password and Mac configuration profile. The profile was copied privately to Mac Downloads and opened, but was not installed. - No tracked files changed and no commits were made: this job was testing only. No product fix, push, deploy or merge was performed. - Evidence and local preparation scripts: `artifacts/macinterop/` (ignored). Screenshots are attached below. **App × flow matrix** | App | Flow | Result / issue | |---|---|---| | Lab setup | Mac access, existing profile and cached calendars | PASS, read only; does not prove authentication | | Lab setup | HTTPS with normal certificate verification | BLOCKED: replacement CA is not trusted; lab prerequisite, no product issue | | Calendar | Fresh account discovery | UNTESTED | | Calendar | Create, edit, move and delete timed events | UNTESTED | | Calendar | All-day, recurrence with an exception, alarm and time zone | UNTESTED | | Calendar | Web ↔ Mac edits and refresh | UNTESTED | | Calendar | Two calendars and calendar deletion | UNTESTED | | Reminders | Create, complete, due dates and lists | UNTESTED | | Reminders | Tasks ↔ Mac round trip | UNTESTED | | Notes | Create, edit and delete on both sides | UNTESTED | | Notes | Attachments, folders and duplicate checks across sync cycles | UNTESTED | | Finder | Cmd+K, mixed folder, 500 MB file, NFD/emoji/spaces | UNTESTED | | Finder | TextEdit open/save, rename, move, delete and Trash | UNTESTED | | Finder | Hidden Apple metadata and disconnect/reconnect | UNTESTED | No new product defect is confirmed. Existing #643–#648 results in the previous lab notes were not retested and are not counted as passes or failures for this run. **Evidence and logs** - [Real Calendar window: cached lab calendars and connection warning](https://git.kayg.org/attachments/0d0ddc93-4668-4828-8d77-1c5f1e0f066c). - [Real Safari window: lab secure connection failure](https://git.kayg.org/attachments/2a40c0a4-4dd6-4b0c-af04-fe816224ecb3). This image was taken during tunnel diagnosis; it is not evidence of a server API defect. - [Device Management: replacement profile downloaded, not installed](https://git.kayg.org/attachments/dc713427-296c-4918-b0e0-b11c2aa49b36). - After the tunnel was corrected, real Mac curl reported `SSL certificate problem: self signed certificate in certificate chain` and `curl: (60) SSL certificate problem: self signed certificate in certificate chain`. The transcript is in `artifacts/macinterop/mac-tls-verification.txt`. - Zero DAV requests reached the server. The proxy recorded one 502 from a local health check before the upstream server was started; this is a lab startup error, not a product finding. There is no evidence for any DAV retry or request-latency result. - Server startup logged slow SQLite acquisition/queries on this busy host. These are SLOW-only observations, not an interop or benchmark result. **Owner recovery steps** 1. Prefer restoring the old lab CA signing key and leaf certificate, if a backup exists. This preserves the Mac's existing lab certificate trust. 2. Otherwise use the replacement public `root.pem` in `~/.local/state/codex-jobs/calternal/macdav-lab/macinterop-2026-10-02-tls/`. Import only that lab CA into Keychain Access → System, open its Trust settings, set it to Always Trust, and complete the macOS authorization prompt. The SHA-256 fingerprint is `95:33:54:D8:43:45:EE:89:EE:46:4A:81:4C:69:E3:37:92:27:06:AE:FC:F1:84:92:17:11:75:39:A8:AD:AA:DD`. This certificate expires on 2026-10-05 at 14:54:55 UTC; replace it if recovery occurs later. I did not change certificate trust or other Mac settings. 3. A subsequent job must start a fresh disposable server, create a fresh User and App Password, and replace only the old calternal lab profile in System Settings → General → Device Management. The profile from this run is invalid after cleanup and must not be installed. macOS may show its downloaded copy until the pending profile expires. 4. Run the full matrix above with normal TLS verification. The job's Mac rule requires these extra trust/authorization steps to be listed for the owner, not performed through an unavailable password prompt. **Gate output, verbatim** ```text ✓ built in 1m 16s > Using @sveltejs/adapter-static Wrote site to "build" ✔ done ``` ```text cargo fmt --check exit: 0 Removed 3797 files, 1.6GiB total cargo clean exit: 0 ``` No crate or web source changed, so per-crate clippy/test and web check/test were not run. No final integration merge was needed for a source change; the detached build under test was kept unchanged. No feature or route was added, so no bench profile was added. No adversarial product round ran: the required real-client prerequisite did not pass. **Cleanup, verified** ```text MAC_TEMPORARY_FILES_REMOVED PORT CLOSED 18088 PORT CLOSED 8443 PORT CLOSED 31993 PORT CLOSED 31465 LAB_DATA_AND_CREDENTIALS_REMOVED Request count: 1 DAV request count: 0 ``` The lab server, TLS proxy and three tunnels are stopped. The disposable Home, index, App Password/profile/session files and web output are removed. The retained server log has its setup token redacted. The Mac lock is released. Only private lab TLS recovery material is kept outside the repo; it contains no User data. **Decisions:** treat connection/trust failures as lab prerequisites, not product defects; use the existing binary only after proving product-code equality and report its embedded revision; retain lab TLS material privately so the next run can reuse the same CA. No product design decision was made. **UX gaps closed:** none; no product UI was changed. **UX gaps left / known gaps:** all requested authenticated client flows, all prior defect retests, the 500 MB transfer, sync-cycle duplicate checks and latency/retry observations remain untested. This report does not satisfy the real-client deploy gate.

kayg commented 2026-10-02 15:51:14 +00:00

Author

Owner

Copy link

Starting Round 2 on job/macinterop-staging-r2, base daea49a7f0a16fbc382614761e5f9f05ec077c7b. Scope: the owner-requested staging Apple-client matrix. HTTPS edge at 168.119.145.115 returns 200 with TLS verification 0. Public DNS and macOS dscacheutil both return 10.69.69.52. Server-side preparation proceeds; no Mac account changes until DNS resolves to the specified edge. Credentials remain in the private staging directory. No production changes.

Starting Round 2 on `job/macinterop-staging-r2`, base `daea49a7f0a16fbc382614761e5f9f05ec077c7b`. Scope: the owner-requested staging Apple-client matrix. HTTPS edge at 168.119.145.115 returns 200 with TLS verification 0. Public DNS and macOS dscacheutil both return 10.69.69.52. Server-side preparation proceeds; no Mac account changes until DNS resolves to the specified edge. Credentials remain in the private staging directory. No production changes.

kayg commented 2026-10-02 16:19:26 +00:00

Author

Owner

Copy link

Preparation commit: 9a0a3fc39. node --check tests/manual/apple-staging-setup.mjs exited 0. A fresh node tests/manual/apple-staging-setup.mjs run printed:

interop-admin saved passkey sign-in passed
macinterop saved passkey sign-in passed
Test User, App Password and profile ready; secrets saved privately.

The admin and member User use separate virtual passkeys. Setup and invitation registration used the real browser forms. The setup admin needed recovery after the first automation checkbox timeout; its new key is private. The Mac DNS check now returns 168.119.145.115. Staging ss -ltn shows HTTP on 10.70.3.158:8080, no IMAP listener. The generated profile advertises dev.calternal.com:993; Notes is not exposed. No product code changed.

Preparation commit: `9a0a3fc39`. `node --check tests/manual/apple-staging-setup.mjs` exited 0. A fresh `node tests/manual/apple-staging-setup.mjs` run printed: ```text interop-admin saved passkey sign-in passed macinterop saved passkey sign-in passed Test User, App Password and profile ready; secrets saved privately. ``` The admin and member User use separate virtual passkeys. Setup and invitation registration used the real browser forms. The setup admin needed recovery after the first automation checkbox timeout; its new key is private. The Mac DNS check now returns `168.119.145.115`. Staging `ss -ltn` shows HTTP on `10.70.3.158:8080`, no IMAP listener. The generated profile advertises `dev.calternal.com:993`; Notes is not exposed. No product code changed.

kayg referenced this issue 2026-10-02 16:21:22 +00:00

Staging: expose Notes IMAP with a valid certificate for the generated Mac profile #858

kayg commented 2026-10-02 16:59:33 +00:00

Author

Owner

Copy link

Real Mac evidence expands the impact of #858: installing the combined server-generated Calendar + Notes profile failed and rolled back Calendar as well. On macOS 27, mdmclient reported MailPayloadPlugin account-save failure, com.apple.accounts Code=1, nested error Code=60, Operation timed out connecting to dev.calternal.com on the default ports. The log then reported Install Failed and removed the calternal Calendar account payload. profiles list showed no calternal staging profile; Internet Accounts showed No accounts after Calendar/Reminders relaunch. No owner-password prompt appeared. HTTPS from the Mac returned 200 with TLS verification 0, and authenticated DAV discovery returned 207. The job will use a new server-generated profile with CalDAV access and no Notes scope to continue independent Calendar/Reminders testing. This does not verify Notes and does not change the edge.

Real Mac evidence expands the impact of #858: installing the combined server-generated Calendar + Notes profile failed and rolled back Calendar as well. On macOS 27, `mdmclient` reported `MailPayloadPlugin` account-save failure, `com.apple.accounts Code=1`, nested error `Code=60`, `Operation timed out` connecting to `dev.calternal.com` on the default ports. The log then reported `Install Failed` and removed the `calternal Calendar account` payload. `profiles list` showed no calternal staging profile; Internet Accounts showed `No accounts` after Calendar/Reminders relaunch. No owner-password prompt appeared. HTTPS from the Mac returned 200 with TLS verification 0, and authenticated DAV discovery returned 207. The job will use a new server-generated profile with CalDAV access and no Notes scope to continue independent Calendar/Reminders testing. This does not verify Notes and does not change the edge.

kayg referenced this issue 2026-10-02 17:41:00 +00:00

Reminders over CalDAV: calternal edits of priority/due on an Apple-created Reminder never reach the Mac and are reverted by the next Apple write #643

kayg referenced this issue 2026-10-02 17:41:01 +00:00

Reminders over CalDAV: a Task with only a scheduled time shows undated in Apple Reminders (DTSTART only, floating) #647

kayg referenced this issue 2026-10-02 17:43:05 +00:00

Task retitle exposes the old canonical checkbox as an extra inline Task #940

kayg commented 2026-10-02 17:49:10 +00:00

Author

Owner

Copy link

Final Round 2 report. Branch job/macinterop-staging-r2, head c75a5b99a1514b3322778021e5447146eb957455. Merged origin/dev once at 6009232d7; no push or deploy.

Built tests/manual/apple-staging-setup.mjs: real browser setup/invitation/recovery and saved-passkey sign-in, private App Password/profile output, and an option to omit unavailable Notes. Recorded the matrix in docs/research/apple-staging-2026-10-02.md. No product changes. Module comments were read again before this report.

Apple staging check, 2026-10-02

This report records Round 2 of #476. It tests the staging image
c4a61e8cf090170f35b1bed3350d9de20c83ecd5. It does not certify a newer build.
See DESIGN §§7, 30, 31 and 48 for the account and adapter contracts.

Setup

The public and Mac DNS records reached 168.119.145.115. Native Mac HTTPS
validation passed. No hosts file, DNS setting or certificate trust setting
changed. The first administrator and a separate test User were created with
real browser forms and virtual passkeys. Credentials are in the private
staging directory, with mode 0600.

The old calternal lab profile was removed. The combined staging profile failed:
the IMAP connection timed out and macOS removed its Calendar payload. The
server-generated profile without Notes installed. Native Calendar and
Reminders discovered the staging calendars. Native NetFS mounted Files over
WebDAV. The first Connect to Server attempt stalled; the mount passed after
the test's authentication helpers were stopped and native NetFS retried.
The Mac lock was held for the entire check.

Results

“API” below means an authenticated request for this test User. A native check
uses the actual Mac application or its filesystem client. API-to-Mac checks
verify the adapter, not the browser's editor controls.

Surface and check	Result	Evidence or limit
Calendar: discovery and create in both directions	PASS	Untagged, Work and Fitness appeared; native and API Log entries arrived.
Calendar: title and time in both directions	PASS	Native 10:30–12:00 edit and API 13:00–14:00 edit matched.
Calendar: move to the next day	PASS	Same block ID moved from September 29 to September 30.
Calendar: delete in both directions	PASS	Native cross-midnight delete and API event delete reached the other side.
Calendar: Unicode, cross-midnight and display alarm	PASS	Unicode title and 23:00–01:00 time survived; the native 15-minute alarm remained in DAV. Notification delivery was not tested.
Calendar: two areas, edit one copy	PASS	One block ID and both tags remained after a native Fitness edit.
Calendar: delete one area copy	PASS	Fitness tag and calendar disappeared; Work copy and block ID remained.
Calendar: repeated sync	PASS	No duplicate R2 events in native readback.
Calendar: stale revision	PASS	Stale PUT returned 412; the stored event did not change.
Calendar: all-day, recurrence, description, location, URL	LIMITED BY CONTRACT	Functional DAV checks returned 422 for all-day and 403 for the other unsupported fields. No new native rejection-dialog check.
Calendar: create an area calendar from DAV	LIMITED BY CONTRACT	MKCALENDAR returned 403.
Calendar: native move between area calendars	NOT VERIFIED	No successful native MOVE check in this round.
Reminders: plain, timed and date-only create	PASS	Native records reached Tasks with correct date, time, priority and multiline body.
Reminders: native rename, body, priority and date edit	PASS	Date-only edit reached Tasks as October 9, low priority and the changed title/body.
Reminders: completion and API uncompletion	PASS	Status changed in Tasks; native readback showed uncompleted.
Reminders: delete in both directions	PASS	Native plain Task delete and API-created Task delete reached the other side.
Reminders: API-created title, due and priority edits	PASS	Native readback showed the edited title, October 10 and priority 9.
Reminders: API edit of an Apple-created Task	FAIL #643	Due October 12 and low priority did not reach the Mac. Completion reverted both to October 8 and urgent.
Reminders: scheduled-only Task	FAIL #647	Native due date was missing. Later native activity cleared the stored scheduled date and time.
Reminders: 21-item Unicode batch	PASS	21 file Tasks and 21 distinct IDs; native list showed the Unicode names.
Reminders: standalone Task retitle	FAIL #940	An extra inline Task appeared with the old canonical checkbox title.
Reminders: Flag, subtask and extra list controls	NOT RETESTED	Round 1 Apple limitations were not checked again.
Files: native mount, Unicode folder and filename	PASS	Native NetFS mount; NFD filename became NFC on the server.
Files: copy, read, append, rename, move and delete	PASS	Native byte checks passed; deleted entries were absent in DAV listing.
Files: 50 MB copy	PASS	Source, mounted read and independent DAV GET had identical SHA-256.
Files: 30 files in three levels	PASS	Recursive byte comparison, rename and delete passed.
Files: API folder create and rename to Mac	PASS	DAV MKCOL/MOVE returned 201; renamed folder was visible on the mount.
Files: copy extended attributes with cp	FAIL #648	Both copies returned exit 1. All 17 content bytes survived; attributes did not. The earlier zero-byte result was not reproduced.
Files: Finder GUI metadata copy	CONTENT PASS, METADATA FAIL #648	After copy completion, all 17 bytes matched. The extended attribute was absent.
Files: volume name	KNOWN GAP #648	Finder uses the User ID as the volume name.
Notes: discovery, create, edit, move, delete and rich content	NOT EXPOSED #858	No public IMAP route or staging IMAP listener with a valid certificate.

Verification and limits

The setup helper passed saved-passkey sign-in for both test accounts and
obtained a real App Password and profile. Setup and recovery browser flows
also passed during preparation. Final syntax, format and whitespace checks
passed. No Rust crate, route, background job or product UI changed, so crate
clippy/tests, web gates, responsive screenshots and a hot-path benchmark do
not apply to this change. The 50 MB check measures integrity, not performance.

A single functional DAV rejection check covered stale revision and documented
unsupported Calendar writes. It did not run a hostile-input campaign against
staging. Server journal capture showed no ERROR or panic lines; it is not a
complete HTTP status trace. No staging deployment or production change occurred.

UX gaps

No product UX gap was fixed by this test job. Findings are on #643, #647,
#648, #858 and #940. The two observed losses of Task properties must block
release. The Mac has notifications disabled for Reminders; the owner must
enable them before testing notification delivery. No password prompt was
needed for the final account installation.

Decisions

Use the server-generated profile without Notes after macOS rolled back the
combined profile. Leave Notes untested until its edge is available. Keep the
staging build unchanged so results refer to the owner's selected image.
These are test choices. No product design choice was added.

Mac screenshots and the full gate output are attached or quoted on #476.
Raw non-secret test evidence remains in artifacts/macinterop/staging-r2/.
Screenshots are not committed.

Gate output (verbatim)

cargo fmt --check: exit 0; no output
node --check tests/manual/apple-staging-setup.mjs: exit 0; no output
git diff --check: exit 0; no output
interop-admin saved passkey sign-in passed
macinterop saved passkey sign-in passed
Test User, App Password and profile ready; secrets saved privately.

Mac screenshots

Calendar before deletes

Calendar after deleting one area

Reminders Unicode batch and date-only Task

Finder 50 MB file and Unicode filename

The screenshots are from the real Mac. No product UI changed, so there is no responsive web screenshot set. Notifications remain disabled in macOS; enable them for a delivery test. The installed staging profile remains available; its private copy is preserved separately from later helper-generated profiles. The Mac lock and staging journal watch are released. cargo clean returned Removed 1 file, 356B total. The worktree is clean.

Final Round 2 report. Branch `job/macinterop-staging-r2`, head `c75a5b99a1514b3322778021e5447146eb957455`. Merged `origin/dev` once at `6009232d7`; no push or deploy. Built `tests/manual/apple-staging-setup.mjs`: real browser setup/invitation/recovery and saved-passkey sign-in, private App Password/profile output, and an option to omit unavailable Notes. Recorded the matrix in `docs/research/apple-staging-2026-10-02.md`. No product changes. Module comments were read again before this report. # Apple staging check, 2026-10-02 This report records Round 2 of #476. It tests the staging image `c4a61e8cf090170f35b1bed3350d9de20c83ecd5`. It does not certify a newer build. See DESIGN §§7, 30, 31 and 48 for the account and adapter contracts. ## Setup The public and Mac DNS records reached `168.119.145.115`. Native Mac HTTPS validation passed. No hosts file, DNS setting or certificate trust setting changed. The first administrator and a separate test User were created with real browser forms and virtual passkeys. Credentials are in the private staging directory, with mode 0600. The old calternal lab profile was removed. The combined staging profile failed: the IMAP connection timed out and macOS removed its Calendar payload. The server-generated profile without Notes installed. Native Calendar and Reminders discovered the staging calendars. Native NetFS mounted Files over WebDAV. The first Connect to Server attempt stalled; the mount passed after the test's authentication helpers were stopped and native NetFS retried. The Mac lock was held for the entire check. ## Results “API” below means an authenticated request for this test User. A native check uses the actual Mac application or its filesystem client. API-to-Mac checks verify the adapter, not the browser's editor controls. | Surface and check | Result | Evidence or limit | | --- | --- | --- | | Calendar: discovery and create in both directions | PASS | Untagged, Work and Fitness appeared; native and API Log entries arrived. | | Calendar: title and time in both directions | PASS | Native 10:30–12:00 edit and API 13:00–14:00 edit matched. | | Calendar: move to the next day | PASS | Same block ID moved from September 29 to September 30. | | Calendar: delete in both directions | PASS | Native cross-midnight delete and API event delete reached the other side. | | Calendar: Unicode, cross-midnight and display alarm | PASS | Unicode title and 23:00–01:00 time survived; the native 15-minute alarm remained in DAV. Notification delivery was not tested. | | Calendar: two areas, edit one copy | PASS | One block ID and both tags remained after a native Fitness edit. | | Calendar: delete one area copy | PASS | Fitness tag and calendar disappeared; Work copy and block ID remained. | | Calendar: repeated sync | PASS | No duplicate R2 events in native readback. | | Calendar: stale revision | PASS | Stale PUT returned 412; the stored event did not change. | | Calendar: all-day, recurrence, description, location, URL | LIMITED BY CONTRACT | Functional DAV checks returned 422 for all-day and 403 for the other unsupported fields. No new native rejection-dialog check. | | Calendar: create an area calendar from DAV | LIMITED BY CONTRACT | MKCALENDAR returned 403. | | Calendar: native move between area calendars | NOT VERIFIED | No successful native MOVE check in this round. | | Reminders: plain, timed and date-only create | PASS | Native records reached Tasks with correct date, time, priority and multiline body. | | Reminders: native rename, body, priority and date edit | PASS | Date-only edit reached Tasks as October 9, low priority and the changed title/body. | | Reminders: completion and API uncompletion | PASS | Status changed in Tasks; native readback showed uncompleted. | | Reminders: delete in both directions | PASS | Native plain Task delete and API-created Task delete reached the other side. | | Reminders: API-created title, due and priority edits | PASS | Native readback showed the edited title, October 10 and priority 9. | | Reminders: API edit of an Apple-created Task | FAIL #643 | Due October 12 and low priority did not reach the Mac. Completion reverted both to October 8 and urgent. | | Reminders: scheduled-only Task | FAIL #647 | Native due date was missing. Later native activity cleared the stored scheduled date and time. | | Reminders: 21-item Unicode batch | PASS | 21 file Tasks and 21 distinct IDs; native list showed the Unicode names. | | Reminders: standalone Task retitle | FAIL #940 | An extra inline Task appeared with the old canonical checkbox title. | | Reminders: Flag, subtask and extra list controls | NOT RETESTED | Round 1 Apple limitations were not checked again. | | Files: native mount, Unicode folder and filename | PASS | Native NetFS mount; NFD filename became NFC on the server. | | Files: copy, read, append, rename, move and delete | PASS | Native byte checks passed; deleted entries were absent in DAV listing. | | Files: 50 MB copy | PASS | Source, mounted read and independent DAV GET had identical SHA-256. | | Files: 30 files in three levels | PASS | Recursive byte comparison, rename and delete passed. | | Files: API folder create and rename to Mac | PASS | DAV MKCOL/MOVE returned 201; renamed folder was visible on the mount. | | Files: copy extended attributes with cp | FAIL #648 | Both copies returned exit 1. All 17 content bytes survived; attributes did not. The earlier zero-byte result was not reproduced. | | Files: Finder GUI metadata copy | CONTENT PASS, METADATA FAIL #648 | After copy completion, all 17 bytes matched. The extended attribute was absent. | | Files: volume name | KNOWN GAP #648 | Finder uses the User ID as the volume name. | | Notes: discovery, create, edit, move, delete and rich content | NOT EXPOSED #858 | No public IMAP route or staging IMAP listener with a valid certificate. | ## Verification and limits The setup helper passed saved-passkey sign-in for both test accounts and obtained a real App Password and profile. Setup and recovery browser flows also passed during preparation. Final syntax, format and whitespace checks passed. No Rust crate, route, background job or product UI changed, so crate clippy/tests, web gates, responsive screenshots and a hot-path benchmark do not apply to this change. The 50 MB check measures integrity, not performance. A single functional DAV rejection check covered stale revision and documented unsupported Calendar writes. It did not run a hostile-input campaign against staging. Server journal capture showed no ERROR or panic lines; it is not a complete HTTP status trace. No staging deployment or production change occurred. ## UX gaps No product UX gap was fixed by this test job. Findings are on #643, #647, #648, #858 and #940. The two observed losses of Task properties must block release. The Mac has notifications disabled for Reminders; the owner must enable them before testing notification delivery. No password prompt was needed for the final account installation. ## Decisions Use the server-generated profile without Notes after macOS rolled back the combined profile. Leave Notes untested until its edge is available. Keep the staging build unchanged so results refer to the owner's selected image. These are test choices. No product design choice was added. Mac screenshots and the full gate output are attached or quoted on #476. Raw non-secret test evidence remains in `artifacts/macinterop/staging-r2/`. Screenshots are not committed. ## Gate output (verbatim) ```text cargo fmt --check: exit 0; no output node --check tests/manual/apple-staging-setup.mjs: exit 0; no output git diff --check: exit 0; no output interop-admin saved passkey sign-in passed macinterop saved passkey sign-in passed Test User, App Password and profile ready; secrets saved privately. ``` ## Mac screenshots [Calendar before deletes](https://git.kayg.org/attachments/b5ae839d-5595-41ff-9f9d-64add05ac6f0) [Calendar after deleting one area](https://git.kayg.org/attachments/24e111ab-84ca-45cd-86af-797c25d36c4c) [Reminders Unicode batch and date-only Task](https://git.kayg.org/attachments/83d72b6d-290c-4837-ba6a-a4e3f07f8210) [Finder 50 MB file and Unicode filename](https://git.kayg.org/attachments/f3da7286-a298-4dd5-997f-adb5390f891a) The screenshots are from the real Mac. No product UI changed, so there is no responsive web screenshot set. Notifications remain disabled in macOS; enable them for a delivery test. The installed staging profile remains available; its private copy is preserved separately from later helper-generated profiles. The Mac lock and staging journal watch are released. `cargo clean` returned `Removed 1 file, 356B total`. The worktree is clean.

kayg referenced this issue 2026-10-02 23:30:19 +00:00

Task retitle exposes the old canonical checkbox as an extra inline Task #940

kayg referenced this issue 2026-10-03 12:51:31 +00:00

Issue hygiene sweep 2026-10-03 #1008

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

dev

wip/rev2-money-ident

wip/restyle-notes

wip/previewcard-1098

wip/palette2-1123

wip/palette-1093

wip/onboard2-1141

wip/onboard-1141.aborted-early

wip/onboard-1141

wip/nlpchip-1127

wip/morph-1104

wip/merge-round-7c5

wip/merge-round-7c4

job/notifloop-1194

wip/merge-round-7c3

wip/merge-round-7c2

wip/merge-round-7c

wip/mchrome-1084

wip/mailghost2-1094

wip/mailghost-1094

wip/kbpreview2-1118

wip/kbpreview-1118

wip/kanban-1092

wip/importhang-1121

wip/hiderev-1153

wip/hide4-1153

wip/hide3-1153

wip/hide2-1153

wip/hide-1153

wip/editreg-1132

wip/editorrail3-1113

wip/editorrail2-1113

wip/editorrail-1113

wip/e2e-b2-1071

wip/e2e-b-1071

wip/draw4-1101

wip/draw3-1101

wip/draw2-1101

wip/draw-1101

wip/directory-1199-r

wip/directory-1199

wip/delete-1119

wip/collabrev-1197

wip/collabloss-1197

wip/cards2-1083

wip/cards-1083

wip/canvas-visual

wip/canvasvis2-976

wip/calhdr-1112

wip/calcards-1115

wip/browserfix

wip/blocks-1125

wip/allday-1107

wip/agenda-decks

wip/agenda-1086

wip/adv7c-1105

wip/txentry-1198

wip/trayicons2-1095

wip/trayicons-1095

job/onboard-1141

wip/sidebar3-1094

wip/rev2-webperf

job/collabloss-1197

job/hide-1153

job/perf-1124

job/perf2-1124

job/tocrail-1191

job/restyle-settings

wip/restyle-settings

job/segmented-1200

wip/notifloop-1194

job/tagperf-1186

wip/tagperf-1186

wip/segmented-1200

job/restyle-files

job/tagdnd-1187

job/merge30

job/cards-1179

wip/cards2-1179

wip/cards-1179

wip/tocrail-1191

wip/tagdnd-1187

wip/restyle-files

wip/perf-1124

wip/merge30j

job/restyle-notes

job/wizchoices-1140

job/adv-1202

wip/wizchoices-1140

wip/restyle-1190

job/moneyfmt-1180

job/txentry-1198

wip/moneyfmt2-1180

wip/moneyfmt-1180-r

wip/moneyfmt-1180

job/pillglass-1189

job/flags-1181

wip/flags-1181

job/restyle-1190

job/restyle-mailmoney

job/restyle-search

job/settingsreg-1195

job/wizard-1140

site/website

wip/wizardrev2-1140

wip/wizardrev-1140

wip/wizard5-1140

wip/wizard4-1140

wip/wizard3-1140

wip/wizard2-1140

wip/wizard-1140

wip/pillglass-1189

wip/settingsreg-1195

job/merge29

job/fu-1171

wip/merge29j

wip/fu-1171

job/fu-1166

job/directory-1199

job/proflog-1204

job/txresearch-1188

wip/fu-1166

job/merge28

job/search-1066

wip/search-1066

wip/merge28j

job/gateslot-1182

job/bulkimport-1157

job/mailnet-1160

wip/mailnetrev-1160

wip/mailnet-1160

wip/bulkrev-1157

wip/bulkimport-1157

job/startup-1161

wip/startup-1161

job/merge27

job/linkcards-1151

wip/linkcards3-1151

wip/linkcards2-1151

wip/linkcards-1151

job/traydate-1144

wip/traydate3-1144

wip/traydate2-1144

wip/traydate-1144

job/draw-1101

wip/merge27j

job/blockpill-1152

wip/blockpill3-1152

wip/blockpill2-1152

wip/blockpill-1152

job/minihover-1149

wip/minihover2-1149

wip/minihover-1149

job/merge25

wip/merge25-r

wip/merge25b

wip/merge25

job/inspector-1129

job/tags-1110

wip/inspector3-1129

wip/inspector2-1129

wip/inspector-1129

wip/tagsrev-1110

wip/tags2-1110

wip/tags-1110

job/dates-1148

wip/datesrev-1148

wip/dates2-1148

wip/dates-1148

job/licence-1145

wip/licence2-1145

wip/licence-1145

job/selfhost-1156

job/merge23

wip/merge23

job/tagfilter-1109

wip/tagfilter2-1109

wip/tagfilter-1109

job/kbd-1134

wip/kbd2-1134

wip/kbd-1134

job/palfoot-1137

wip/selfhost-1156

wip/palfoot2-1137

wip/palfoot-1137

job/toggle-1158

wip/toggle-1158

job/kbpreview-1118

job/docratchet-1155

job/perflint-1133

job/devtests-1159

wip/docratchet-1155

wip/devtests-1159

job/segv-1136

wip/toast-1142

wip/segv-1136

job/toast-1142

job/blockreload-1147

wip/blockreload-1147

job/font-1150

wip/font-1150

job/importui-1120

job/minimonth-1149

wip/importui-1120

wip/minimonth-1149

job/depcheck-1146

wip/perflint-1133

wip/depcheck-1146

job/calcards-1115

job/blocks-1125

job/plus-1128

job/shift-1138

wip/plus2-1128

wip/plus-1128

wip/shift-1138

job/moneyfid-1130

job/editorrail-1113

wip/moneyrev-1130

wip/moneyfid-1130

job/noext-851

wip/noext-851

wip/noext3-851

wip/noext2-851

job/week-1135

wip/week-1135

job/editreg-1132

job/smoke-1122

wip/smoke-1122

job/docs-1143

job/palette2-1123

job/calhdr-1112

job/nlpchip-1127

job/mailghost-1094

job/reconnect-1131

wip/reconnect-1131

job/trayicons-1095

job/delete-1119

job/importhang-1121

job/cards-1083

job/palette-1093

job/mchrome-1084

job/e2e-a-1071

job/canvas-visual

job/previewcard-1098

job/allday-1107

wip/e2e-a2-1071

wip/e2e-a-1071

job/e2e-b-1071

job/adv7c-1105

job/kanban-1092

job/agenda-1086

job/merge-round-7c

job/morph-1104

wip/surfaces-p2

job/merge-round-9

wip/merge-round-9

job/7cfix-small

wip/7cfix-small

job/mailui-1078

job/merge-round-8

wip/merge-round-8

wip/mailui-1078

job/mailround-1038

job/applemail-accept

wip/settitle-1068

wip/mailround2-1038

wip/mailround-1038

wip/e2e-7b

job/crash-1069

wip/crash-1069

job/searchlost-1066

wip/searchlost-1066

job/7b-reconcile

job/flake-1065

wip/flake-1065

wip/merge-round-7b7

wip/merge-round-7b6

wip/merge-round-7b5

wip/merge-round-7b4

wip/7b-reconcile

job/appupdate-1059

job/nfd-1044

wip/appupdate-1059

job/e2e-7b

job/loop-1062

wip/loop-1062

job/pdfprev-1045

job/invtoggle-1053

wip/pdfprev-1045

wip/nfd-1044

wip/invtoggle-1053

job/7bfix-e2e

job/mailstress-b

wip/7bfix-e2e

wip/mailstress-b

job/7bfix-adv

wip/7bfix-adv

job/mailstress-a

job/stack-1054

wip/stack-1054

wip/mailstress-a

job/mailstress-1038

wip/mailstress-1038

job/upload500-1051

wip/upload500-1051

job/share-1034

wip/share-1034

job/syncerr-1037

job/7bfix-photos

wip/7bfix-photos

job/paste-1036

job/setside-1039

wip/setside-1039

wip/paste-1036

job/lease-1042

wip/syncerr-1037

wip/lease-1042

job/7bfix-data

job/passkeybind-1043

wip/apprevoke-1041

job/invite-1035

wip/invite-1035

job/merge-round-7b2

wip/merge-round-7b2

job/mailproxy-486

job/apprevoke-1041

job/rebuild-1033

job/pillborder-1029

wip/pillborder-1029

wip/mailproxy-486

wip/applemail-486

job/headless-998

wip/headless-998

job/groups-1028

wip/groups-1028

job/rebuildwarn-1016

wip/rebuildwarn-1016

job/startup-1011

wip/startup-1011

job/monthpill-1009

job/bgthumb-1025

job/sharetitle-1012

wip/monthpill-1009

wip/bgthumb-1025

wip/sharetitle-1012

job/canvas-cards-977

wip/canvas-cards-977

job/canvas-pencil-978

job/canvas-sketch-990

wip/canvas-sketch-990

wip/canvas-pencil-978

job/canvas-files-989

wip/canvas-files-989

job/canvas-collab-991

wip/canvas-collab-991

job/weekscroll-1018

wip/weekscroll-1018

wip/canvas-core-976

job/canvas-core-976

job/round-drag

wip/round-drag

job/round-settings

job/browserfix

wip/oapi-974

job/oapi-974

job/hist2-integrate

job/mailhtml-726

wip/mailhtml-726

wip/hist2-integrate

job/moneyfu-984

job/drag-1015

wip/drag-1015

job/rename-1017

wip/rename-1017

job/hist2-api

wip/hist2-api

job/oneacct-1014

wip/oneacct-1014

wip/moneyfu-984

job/hist2-bench

job/hist2-restore

wip/hist2-bench

job/hist2-write

job/hotfix-724

wip/hotfix-724

wip/hist2-write

wip/hist2-restore

job/hist2-store

job/hist2-ui

wip/hist2-ui

wip/hist2-store

job/searchstarve-965

job/shutdown-963

wip/shutdown-963

wip/pubedit-981

job/pubedit-981

job/analytics-973

wip/searchstarve-965

job/authflash-850

job/weeklane-969

job/pvtitle-1004

job/hist-975

wip/authflash-850

job/voicepill-617

wip/pvtitle-1004

job/headring-1003

wip/weeklane-969

wip/voicepill-617

wip/headring-1003

wip/analytics-973

job/agentscope-980

wip/thumbsandbox-988

job/thumbsandbox-988

wip/hist-975

job/links-856

wip/links-856

job/davetag-966

wip/davetag-966

job/filesstorm-1000

job/hoverpad-725

wip/filesstorm-1000

job/ffmpegblas-993

job/merge-round-7a

wip/hoverpad-725

wip/ffmpegblas-993

job/nowdot-1002

wip/verify-7a

job/noteid-857

wip/nowdot-1002

wip/noteid-857

wip/merge-round-7a

wip/agentscope-980

job/imapedge

job/a11yfix2

wip/imapedge-941

wip/imapedge

wip/a11yfix2

job/notetask-986

job/logheading

wip/logheading-998

job/textthumb-652

job/photolive-987

wip/photolive-987

job/davactive-983

job/savefix-985

job/tabicons-607

wip/davactive-983

wip/tabicons-607

wip/notetask-986

wip/savefix-985

job/dirid-627

job/buildspeed-1007

wip/dirid-627

job/agenda-decks

job/perfguards-impl

job/undo-a11y

wip/undo-a11y

job/mailperf

job/wal-824

wip/settings-50

job/settings-50

job/notesfilter-606

wip/notesfilter-606

job/surfaces-p2

wip/wal-824

job/maillayouts

wip/mailperf

wip/maillayouts

job/taskmeta-659

job/money-ident

wip/money-ident

wip/taskmeta-659

job/errstates

wip/perfguards-impl

job/headings-881

wip/headings-881

wip/errstates

job/voice-619

job/gaps-827

job/notesperf

wip/notesperf

wip/voice-619

job/hddsql-549

job/perf-stream-668

wip/perf-stream-668

wip/deeplinks-fix

job/deeplinks-fix

job/authfix

job/docsfix-rust

wip/docsfix-rust

job/webperf

job/docsfix-web

job/datafix2

job/webdav-lock-476

job/copyfix

wip/copyfix

wip/webperf

job/focus-658

wip/protofix

job/mediafix

job/protofix

wip/mediafix

job/agentfix

job/hhmm-724

wip/agentfix

job/undo-722

job/reuse

wip/webdav-lock-476

wip/reuse

job/scopefix

job/datafix

wip/hhmm-724

wip/undo-722

job/surfaces-p1

wip/hddsql-549

job/voicememos-618

wip/datafix2

wip/surfaces-p1

job/fix-940

wip/fix-940

job/blaze-surfaces

wip/datafix

wip/blaze-surfaces

job/taskday-655

job/linknav-639

wip/linknav-639

wip/gaps-827

job/isolation-707

job/audiophotos-720

wip/audiophotos-720

job/advfind-664

wip/voicememos-618

wip/taskday-655

wip/isolation-707

wip/advfind-664

wip/scopefix

wip/focus-658

job/testgaps

wip/testgaps

job/overscroll-718

wip/authfix

job/deps

wip/overscroll-718

job/rev2-agentfix

job/rev2-money-ident

job/rev2-mailperf

wip/deps

job/hardening-728

wip/hardening-728

job/searchgen-832

wip/searchgen-832

job/photopw-849

job/mailsql-825

wip/photopw-849

job/sharefix

wip/sharefix

job/rev2-mailhtml-726

job/rev2-perfguards

job/copyval-723

job/lightglass-r2

wip/lightglass-r2

wip/docsfix-web

job/copy-audit

job/macinterop-staging-r2

job/design-sync

job/rev2-taskmeta-659

job/rev2-webperf

job/docs-audit

job/rev2-advfind-664

job/rev2-mailproxy-486

job/states-audit

job/rev2-datafix

job/design-drift

job/test-gaps

job/rev2-voicememos-618

job/rev2-mediafix

job/rev2-deps

job/rev2-datafix2

job/licence-audit

job/issue-hygiene

job/rev2-protofix

job/rev2-voice-619

job/rev2-isolation-707

job/rev2-surfaces-p1

job/deeplink-audit2

job/rev2-audiophotos-720

wip/test-gaps

job/rev2-overscroll-718

job/rev2-undo-722

wip/states-audit

job/rev2-dropmd-719

job/rev2-linknav-639

job/merge-7b-plan

wip/merge-7b-plan

job/rev2-taskday-655

wip/mailsql-825

job/rev2-webdav-lock-476

job/rev2-browserfix

wip/design-drift

job/rev2-hddsql-549

wip/deeplink-audit2

job/rev2-scopefix

job/rev2-authfix

job/rev2-hardening-728

job/rev2-wal-824

job/rev2-sharefix

job/calsidebar-638

job/chrome-audit

job/ioperf

wip/ioperf

wip/chrome-audit

wip/calsidebar-638

job/dropmd-719

wip/dropmd-719

job/ocr-build

wip/ocr-build

job/blaze-settings

wip/copyval-723

job/toastring-721

wip/toastring-721

job/deployfix-732

wip/deployfix-732

wip/blaze-settings

job/money-import-recheck

job/rev-a11y

job/perf-arch-db

job/rev-7b-data

wip/textthumb-652

wip/perf-arch-db

job/sec-protocols

job/sidehdr-660

job/rev-7b-security

job/research-surfaces

job/rev-design-gaps

job/rev-mcp-api

wip/sidehdr-660

job/perf-arch-memory

wip/sec-protocols

job/perf-arch-bundle

job/snapedge-714

wip/rev-mcp-api

job/sec-supplychain

wip/research-surfaces

job/perf-arch-sync

job/rev-consistency

job/perf-arch-server

wip/perf-arch-server

wip/perf-arch-memory

job/perf-arch-io

job/perf-arch-client

job/sec-fs

job/sec-mcp-scopes

job/sec-sharing

job/perf-guards

job/sec-browser

job/sec-admin-deploy

job/sec-auth

wip/snapedge-714

job/bgpicker-717

wip/perf-arch-bundle

wip/money-import-recheck

job/advsetup-654

wip/bgpicker-717

wip/advsetup-654

job/burst-709

job/kbdcaps-710

job/app-pw-chooser

wip/burst-709

wip/app-pw-chooser

job/imaptest-625

wip/kbdcaps-710

job/fix-499

wip/fix-499

job/perf-mut-667

job/calimg-589

job/perf-snap-666

wip/calimg-589

wip/perf-snap-666

wip/perf-mut-667

job/perf-cache-665

wip/perf-cache-665

job/voicefiles-620

wip/voicefiles-620

job/admin-burst-705

wip/admin-burst-705

job/voicememos-review

wip/voicememos-review

wip/ryw-653

job/ryw-653

job/writeonopen-661

job/instant-663

wip/writeonopen-661

job/money-import-review

wip/money-import-review

wip/importjs-610

review/integrations-407-round6

wip/integrations-review

job/dragghost-612

wip/dragghost-612

job/integrations

wip/integrations

job/decider-656

job/merge-round-6

job/perf-rerun

wip/merge-round-6

job/integrations-review-round5

job/selalign-576

wip/selalign-576

job/mcp-events-491

job/files-631

job/cal-e2e-569

wip/cal-e2e-569

job/reload-423

wip/reload-423

wip/mcp-events-491

wip/files-631

job/notesbridge-644

wip/notesbridge-644

job/editor-series

job/calcard-series

wip/calcard-series

job/mcp-events-review-491

wip/mcp-events-review

wip/editor-series

job/quirks-546

job/integrations-recheck

job/tocrail-636

wip/tocrail-636

wip/quirks-546

wip/reminders-643

job/reminders-643

wip/davscale-573

job/davscale-573

job/integrations-review

wip/ocr-eval-584

job/ocr-eval-584

job/esc-537

wip/esc-537

job/toastname-586

wip/toastname-586

job/submenu-579

wip/submenu-579

job/tasks-mode

wip/tasks-mode

job/agentdocs-630

job/dupwrite-634

wip/agentdocs-630

wip/dupwrite-634

job/lightglass-588

wip/lightglass-588

job/tabswitch-549

job/ghosttask-623

wip/ghosttask-623

job/toaststack-616

job/weekstate-609

job/mailsync-613

wip/mailsync-613

wip/weekstate-609

job/maildup-626

wip/tabswitch-549

wip/maildup-626

wip/toaststack-616

job/motion-611

wip/motion-611

job/tlstest-601

wip/tlstest-601

job/perf-495

job/floating-sheet

wip/floating-sheet

job/remdup-585

wip/remdup-585

job/fix-502

wip/fix-502

job/attachplay-622

job/perf-batch

wip/perf-batch-563

wip/perf-495

hotfix/mail-sync-diag

job/mail-m3

wip/mail-m3

job/attach-poof-603

job/calhover-608

job/editorbar-604

job/mentions-605

job/merge-round-4

job/allday-514

wip/merge-round-4

wip/allday-514

job/merge-round-4a

wip/merge-round-4a

job/sharestack-580

job/fix-501

wip/sharestack-580

wip/fix-501

job/perf-batch-563

job/apw-cache-review

wip/apw-cache-review

job/probe-520

wip/probe-520

job/mac-393

wip/mac-393

job/header-571

job/flake-513

wip/flake-513

job/docs-thumb-547

wip/header-571

job/webcal-572

wip/webcal-572

wip/shortcuts-542

job/shortcuts-542

wip/docs-thumb-547

job/caldav-stress

wip/caldav-stress

wip/sweep-478

job/apw-cache-512

wip/apw-cache-512

job/money-empty-540

wip/restart-505

wip/money-empty-540

wip/fix-510

job/restart-505

job/fix-503

job/perf-496

wip/perf-496

job/fix-498

wip/fix-498

job/info-inspector-465

wip/info-inspector-465

job/fix-510

job/fix-507

wip/fix-507

wip/fix-503

job/fix-493

job/money-kinds

wip/money-kinds

job/hygiene-548

job/merge-round-3

wip/fix-493

job/drag-snap-536

wip/merge-round-3

wip/merge-round-0930

wip/drag-snap-536

job/align-538

wip/align-538

job/bg-flash

wip/bg-flash

job/money-import

job/search-count-544

wip/search-count-544

wip/money-import

job/settings-key-541

wip/settings-key-541

job/toast-539

job/preview-421

wip/preview-421

wip/toast-539

job/tasks-500-531

job/title-plain-526

wip/title-plain-526

wip/tasks-500-531

job/notes-bridge

wip/parity-484

job/parity-484

job/files-slow

job/crash-525

wip/notes-bridge

wip/files-slow

wip/crash-525

job/kbd-motion-527

wip/bg-422

job/analytics-504

wip/analytics-504

wip/kbd-motion-527

job/upload-pill-523

wip/upload-pill-523

wip/tray-order

job/tray-order

wip/overflow-mid

wip/merge-round-2

job/perf-494

wip/perf-494

wip/mcp-fast-492

wip/motion-477

wip/asr-ab-489

wip/theme-variants-506

wip/overflow-511

wip/week-header-508

wip/attach-427

job/dav-delete-471

job/iso-435

wip/iso-435

wip/files-sel-keys

wip/dav-delete-471

job/align-253

job/siwc-490

wip/siwc-490

job/money-kinds-review

wip/align-253

wip/money-kinds-review

job/small-bugs-3

wip/overlay-title-487

wip/multiget-500

wip/hidden-420

wip/webcal-ui

wip/webcal-431

job/perf-367

job/location

wip/small-bugs-3

wip/location

wip/perf-367

wip/admin-deny-483

job/tag-unicode-473

wip/tag-unicode-473

job/blur-436

wip/photos-470

wip/blur-436

wip/small-bugs-4

wip/hunt-20260930

wip/settings-hdr-482

wip/chips-416

job/dedup-375

wip/dedup-375

job/doc-stack

wip/doc-stack

job/tokens-literals

wip/tokens-literals

job/jobs-leftovers

wip/send-fast

wip/paste-467

wip/money-numbers

job/money-plugin

wip/money-plugin

job/break-dav

wip/merge-batch

wip/crossday-469

wip/mac-verify

wip/mail-m2

wip/break-dav

wip/money-review2

job/money-md

job/modes-424

wip/money-md

wip/jobs-leftovers

job/agenda-413

wip/agenda-413

wip/modes-424

job/recog-417

wip/recog-417

wip/bounce-425

wip/ab-384-luna

job/webdav-perf

wip/webdav-perf

job/toast-ring

wip/toast-ring

job/money-review

wip/money-review

wip/micro-motion

wip/settings-card

wip/minical

job/notes-imap-428

job/least-priv

wip/ui-small-2

wip/flaky-426

wip/drag-end-418

job/jank

wip/jank

wip/least-priv

wip/docs-site

job/agenda

job/sec-batch

wip/sec-batch

wip/per-user-index

job/area-calendars

wip/area-calendars

job/parity

wip/parity

job/documents-research

wip/documents-research

job/test-infra

job/reminders-sync

wip/small-bugs-2

wip/reminders-sync

wip/gestures

job/google-oauth

wip/tags-merge

wip/tags

job/e2e-theme

wip/e2e-theme

job/icon-align

wip/test-infra

wip/select-align

wip/editor-385

job/voice

wip/webdav

job/webdav

job/app-pw-ui

job/editor-integrity

wip/editor-integrity

wip/voice

wip/quota

wip/cal-followups

wip/icon-align

job/composer-scale

wip/composer-scale

job/jobs-page

wip/jobs-page

job/hig-type

wip/hig-type

wip/app-pw-ui

job/motion-spring

job/mcp

wip/motion-spring

wip/mcp

job/small-bugs

wip/push-hosts

job/profile-sign

wip/touch-369

wip/profile-sign

job/mobile-focus

wip/mobile-focus

wip/ui-polish-354

wip/small-bugs

wip/dup-task

job/toast-polish

job/app-pw-scopes

wip/toast-polish

wip/app-pw-scopes

wip/cli-agent

wip/selection-pills

job/preview-attach

wip/preview-attach

job/dav-proppatch

wip/dav-proppatch

wip/cal-switcher

job/atomic-race

wip/atomic-race

job/photos-shared

wip/photos-shared

wip/cal-grid

wip/note-rewrite

wip/search-rebuild

job/mail-m1

job/paperless-import

wip/paperless-import

wip/mail-m1

wip/hidden-activity

wip/search-d

wip/pricing-research

wip/cursors

wip/auto-scheme

job/single-pills

wip/single-pills

wip/xuser-matrix

wip/money-format

wip/app-pw-setup

wip/purge-dos

wip/vault-health

wip/caldav-apple

wip/xuser-audit

wip/e2e-green

wip/tabbar

wip/adv-harness

wip/maple-mono

job/search-fix

wip/search-fix

wip/search-perf-c

job/adv-harness

wip/sidebar-headers

job/glass

wip/temp-index

job/polish

wip/polish

wip/file-protocols

wip/money-research

wip/glass

wip/voice-models

wip/collab-redo

job/voice-research

wip/hunt-20260928

wip/notes-actions-research

wip/search-pad

wip/search-perf

wip/search-sticky

wip/editor-undo

wip/chrome-rules

wip/motion

wip/appearance-research

wip/appearance

wip/audit-bugs

wip/cal-glass

wip/block-actions

wip/authz-order

wip/event-stripes

wip/chrome-sidebar

wip/auth-flaky

wip/robust-2

wip/gate-fix

wip/menu-blur

wip/import-calternaljs

wip/tray-fix

job/import-calternaljs

wip/index-order

wip/audit-fixes

wip/search-chevrons

research/mail

wip/phone-chrome

wip/dedup-break

wip/csp

wip/ui-audit

wip/select-toast

wip/perf

wip/flat-layout

wip/fonts

wip/event-tint

wip/sync-converge

wip/data-split

wip/glass-audit

wip/robustness

wip/sync-chaos

wip/search-thumbs

wip/fuzz

wip/menu-icons

wip/search-pill

wip/sync-changing

wip/heading-links

wip/date-formats

wip/a11y

wip/break-editor

wip/e2e-fix

wip/settings-sections

wip/sync-root-guard

wip/search-palette

wip/share-edit

job/toasts

wip/toasts

wip/cont-analytics

wip/authz-review

wip/popovers

wip/overlay-glass

wip/change-feed

wip/editor-modes

wip/composer-align

wip/cont-agenda

wip/agenda-merge

job/agent-conventions

wip/agent-conventions

wip/backend-misc

job/route-audit

wip/route-audit

wip/ui-batch

wip/heif-hardening

wip/grid-resize

wip/ask-page

wip/webmcp

job/deeplink-audit

wip/deeplinks

wip/shortcuts

wip/cont-tz-days

main

No results found.

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kayg/calternal#476

No description provided.