Large Tag acceptance stalls another User Note write beyond 30 seconds

kayg commented

2026-10-09 05:55:53 +00:00

Owner

The first tagperf4 5,000-Note candidate sample shows a cross-User write stall. Tags page/sidebar names appear at 195.40 ms, but POST /api/v1/tags/rewrites accepts only after 37,101.40 ms. One separate-User conditional Note-body PUT exceeds its 30-second transport cap (30,027.68 ms, status -1). The writer then stops because its ETag outcome is unknown. Five writes were attempted; four returned 200 and one timed out. Do not call the timeout a successful write p95.

The forward receipt completes at 128,188.45 ms, every source Note has its expected final Tag, and Tag reads return no errors (p95 189.66 ms). Server peak descriptors are 306; this is not the earlier 1,024-descriptor failure. load5 start/end is 4.16 / 5.23.

Candidate Rust source a77e71920, binary SHA-256 05cb9d89046ddc5ad5a410c9623c1b986a173add05239118235534a12139bdfd. The run uses perf-run on perf-test-69-63 and a fresh disposable fixture with a second real User/Note. Frozen SPA e1672b919. Caller start_rewrite holds the namespace/User locks through acceptance (crates/calternal-tags/src/lib.rs:3052). project_accepted joins json_each witnesses with tags before staging memberships (rewrite.rs); query-plan cost is a lead, not a measured cause. The observation does not isolate lock wait from database work or startup work.

The profile now allows 120 seconds for an independent write so it can record a slow success and continue the ETag chain. It separately reports successful-write p95, attempt maximum, counts and errors. It never resubmits an unknown timed-out write. The original sample is retained as a diagnostic; a corrected full pair is needed for successful other-User p95. No production algorithm changes are made by this measurement update.

The first tagperf4 5,000-Note candidate sample shows a cross-User write stall. Tags page/sidebar names appear at 195.40 ms, but POST /api/v1/tags/rewrites accepts only after 37,101.40 ms. One separate-User conditional Note-body PUT exceeds its 30-second transport cap (30,027.68 ms, status -1). The writer then stops because its ETag outcome is unknown. Five writes were attempted; four returned 200 and one timed out. Do not call the timeout a successful write p95. The forward receipt completes at 128,188.45 ms, every source Note has its expected final Tag, and Tag reads return no errors (p95 189.66 ms). Server peak descriptors are 306; this is not the earlier 1,024-descriptor failure. load5 start/end is 4.16 / 5.23. Candidate Rust source a77e71920, binary SHA-256 05cb9d89046ddc5ad5a410c9623c1b986a173add05239118235534a12139bdfd. The run uses perf-run on perf-test-69-63 and a fresh disposable fixture with a second real User/Note. Frozen SPA e1672b919. Caller start_rewrite holds the namespace/User locks through acceptance (crates/calternal-tags/src/lib.rs:3052). project_accepted joins json_each witnesses with tags before staging memberships (rewrite.rs); query-plan cost is a lead, not a measured cause. The observation does not isolate lock wait from database work or startup work. The profile now allows 120 seconds for an independent write so it can record a slow success and continue the ETag chain. It separately reports successful-write p95, attempt maximum, counts and errors. It never resubmits an unknown timed-out write. The original sample is retained as a diagnostic; a corrected full pair is needed for successful other-User p95. No production algorithm changes are made by this measurement update.

kayg commented

2026-10-09 06:33:54 +00:00

Author

Owner

Second paired 5,000-Note candidate sample, under perf-run with a 120-second write cap: both UI views 100.40 ms; acceptance 42,000.3 ms; background completion 137,393.19 ms. Concurrent Tag-read p95 211.48 ms, zero read errors. The independent conditional writer completed four requests (successful p95 125.98 ms) then received HTTP 503 after a 35,536.03 ms stall and stopped. This confirms that the earlier 30-second timeout was not merely an overly short probe cap. The successful percentile describes only four requests; it is not evidence that independent writes stayed responsive through the rewrite. load5 3.60 → 4.38. No UI code changed. Public raw evidence is being committed in docs/perf/runs/tag-rewrite-1186-tagperf4.json.

Second paired 5,000-Note candidate sample, under `perf-run` with a 120-second write cap: both UI views 100.40 ms; acceptance 42,000.3 ms; background completion 137,393.19 ms. Concurrent Tag-read p95 211.48 ms, zero read errors. The independent conditional writer completed four requests (successful p95 125.98 ms) then received HTTP 503 after a 35,536.03 ms stall and stopped. This confirms that the earlier 30-second timeout was not merely an overly short probe cap. The successful percentile describes only four requests; it is not evidence that independent writes stayed responsive through the rewrite. load5 3.60 → 4.38. No UI code changed. Public raw evidence is being committed in `docs/perf/runs/tag-rewrite-1186-tagperf4.json`.

kayg referenced this issue

2026-10-09 06:38:04 +00:00

Tags: rename/merge is slow — no full reconcile, batched parallel rewrite, optimistic UI #1186

kayg commented

2026-10-09 07:26:16 +00:00

Author

Owner

Third interleaved 5,000-Note candidate run confirms the initial write stall even without an error response: acceptance 41,002.7 ms; both UI views 435.20 ms; rewrite completion 154,355.03 ms. All 286 other-User conditional writes succeeded. Successful/attempt p95 is 285.37 ms, but maximum write latency is 40,675.46 ms. Concurrent Tag-read p95 229.12 ms, zero read errors. load5 6.78 → 6.80. This is a separate fresh fixture under its own perf-run reservation. The second sample had HTTP 503 after 35,536.03 ms, while the first reached the old 30-second client timeout. Failure and maximum latency must accompany p95; p95 alone hides the initial stall. All source bytes passed the completed-receipt check.

kayg referenced this issue

2026-10-09 07:28:49 +00:00

Tags: rename/merge is slow — no full reconcile, batched parallel rewrite, optimistic UI #1186

kayg referenced this issue

2026-10-09 07:31:59 +00:00

Tags: rename/merge is slow — no full reconcile, batched parallel rewrite, optimistic UI #1186

kayg referenced this issue

2026-10-09 07:40:34 +00:00

Tags: rename/merge is slow — no full reconcile, batched parallel rewrite, optimistic UI #1186

kayg commented

2026-10-09 07:40:35 +00:00

Author

Owner

Measured #1501 cause in the Rust test build: the former acceptance query holds the FULL authority SQLite write transaction while a different writer obtains its ordinary pool connection in 0.098 ms, then waits 5006.733 ms in BEGIN IMMEDIATE and returns SQLite BUSY. The diagnostic requests no User/namespace mutex, so SQLite write-lock contention alone is sufficient. Notes projection uses this same BEGIN IMMEDIATE path and bounded BUSY retries (crates/plugins/notes/src/store.rs), which explains repeated five-second waits and eventual 503. The namespace lock is global but a plain conditional Note-body write does not acquire it; its Notes mutex is per User. The current fix removes the acceptance namespace lock, saves intent only, stages 64 witnesses per background transaction, and forces witness-first owner+path indexed probes. The 5,000-Note admission regression passes while the namespace lock is held. Full safety gates and real-server/paired evidence are in progress.

kayg referenced this issue

2026-10-09 07:56:45 +00:00

Tags: rename/merge is slow — no full reconcile, batched parallel rewrite, optimistic UI #1186

kayg commented

2026-10-09 08:56:04 +00:00

Author

Owner

The final release binary passed the real-server two-User regression under a reserved perf-run session on perf-test-69-63 (load5 4.01). User A renamed a Tag on 2,000 Notes while User B made 20 conditional Note-body PUTs. All 20 succeeded. Acceptance: 149.601 ms; other-User write p95: 311.465 ms; maximum: 330.594 ms. The test enforces an independent 1-second cap for every write and verifies completed receipt and final bytes. No timeout assertion was weakened.

Two 5,000-Note candidate samples now completed: acceptance 246.8 / 393.8 ms; other-User write maxima 511.36 / 445.66 ms; durable completion 109.17 / 120.65 s. The third paired sample is running. Tag-read p95 is mixed (196.00 / 234.42 ms against origin/dev 215.81 / 203.31 ms), so #1499 is not marked resolved; a matched 1,000-Note pair is queued.

The headful real-macOS production Tags view showed the new Tag on page and sidebar 340.6 ms after trusted CDP Enter. HTTP/1.1 queue was 0.5 ms. This uses a remote browser with a local-server tunnel; server response timing includes the shared build host. Mac job directory and reservation lock are verified removed.

The final release binary passed the real-server two-User regression under a reserved perf-run session on perf-test-69-63 (load5 4.01). User A renamed a Tag on 2,000 Notes while User B made 20 conditional Note-body PUTs. All 20 succeeded. Acceptance: 149.601 ms; other-User write p95: 311.465 ms; maximum: 330.594 ms. The test enforces an independent 1-second cap for every write and verifies completed receipt and final bytes. No timeout assertion was weakened. Two 5,000-Note candidate samples now completed: acceptance 246.8 / 393.8 ms; other-User write maxima 511.36 / 445.66 ms; durable completion 109.17 / 120.65 s. The third paired sample is running. Tag-read p95 is mixed (196.00 / 234.42 ms against origin/dev 215.81 / 203.31 ms), so #1499 is not marked resolved; a matched 1,000-Note pair is queued. The headful real-macOS production Tags view showed the new Tag on page and sidebar 340.6 ms after trusted CDP Enter. HTTP/1.1 queue was 0.5 ms. This uses a remote browser with a local-server tunnel; server response timing includes the shared build host. Mac job directory and reservation lock are verified removed.

kayg referenced this issue

2026-10-09 10:12:51 +00:00

Tags: rename/merge is slow — no full reconcile, batched parallel rewrite, optimistic UI #1186

kayg commented

2026-10-09 10:12:51 +00:00

Author

Owner

tagfix5 / #1186 / #1501 — final report

Head: 4c4734791c on job/tagperf-1186. Start: e1f7e49252. Merged origin/dev once at 9dc65437e8 before gates. No push, deploy or merge into dev/main.

Built

Acceptance records the immutable intent and queue receipt in a FULL transaction; forward projection is deferred to the background worker.
Projection uses indexed witness seeks, at most 64 source paths per transaction, and yields after each commit. Per-User admission locks remain keyed by User; acceptance does not acquire the root namespace lock.
Saved exact membership names make child-to-parent and nested namespace replay idempotent. Legacy receipts remain readable. Distinct item counts do not count each membership twice.
Added a real-server two-User test with >=2,000 Notes and 20 conditional writes, each capped at 1 second; no unknown outcome is retried.

Files changed by this job

crates/calternal-tags/src/{lib.rs,rewrite.rs}; docs/DESIGN.md
tests/adversarial/tag-rewrite-cross-user-1501.mjs
bench/tag-rewrite-1186-paired.sh (isolated run root)
docs/perf/2026-10-09-tagfix5-1501.md, docs/perf/runs/tag-rewrite-1186-tagfix5.json, docs/perf/baseline.json
Merge resolutions: calternal-fs/src/write.rs; calternal-server/src/upgrade_tests.rs; exact live contracts/perf pins. Both directory-proof and batch-proof invariants are retained. Server upgrade frontiers include Tags 7 and the new dev migrations. The merged frontier-count assertion is 34 (dev 33 plus this branch’s existing Tags migration); content assertions are retained. No new migration was added by tagfix5.

Cause
A test-build diagnostic measured writer-pool checkout 0.098 ms, then BEGIN IMMEDIATE wait 5,006.733 ms with SQLITE_BUSY, while the former authority projection held its write transaction for 37,783.513 ms. The competing writer requested neither namespace nor User lock. The normal Note-body route uses a User-keyed mutex and retries SQLite busy writes. The blocking resource was SQLite's writer, not pool checkout or a cross-User mutex.

Performance (three 5,000-Note pairs on perf-test-69-63)
Frozen baseline: origin/dev 9dc65437e; frozen forward-path candidate source: 9f24ac50a; production SPA: e1672b919. Every measured run held perf-run's single shared lock. The later 6b000c795 retry-lock fix does not change the measured new-intent forward path. Raw records preserve hashes, CPU/RSS, actual run order and load5 in docs/perf/runs/tag-rewrite-1186-tagfix5.json.

Pair	Build	Acceptance ms	UI page/sidebar ms	Completion s	Other-User write p95/max ms	Tag read p95 ms	load5 before/after
1	origin/dev	490.20	>900,376	>900.05	649.23 / 1814.89	215.81	3.05 / 7.10
1	candidate	246.80	90.80	109.17	197.57 / 511.36	196.00	2.17 / 3.47
2	origin/dev	146.10	>900,003.80	>900.05	854.48 / 2670.44	203.31	1.08 / 6.72
2	candidate	393.80	138.30	120.65	233.29 / 445.66	234.42	2.69 / 4.01
3	origin/dev	260.10	>900,626.40	>900.01	635.40 / 2443.00	208.33	3.68 / 6.78
3	candidate	365.50	194.00	111.95	223.17 / 418.92	197.99	5.75 / 6.27

All candidate samples completed and verified every source. Their 309/327/303 independent writes had zero errors. No capped baseline is represented as completed latency. Expected cost: accepted witness rows times <=2 exact membership-name seeks per projection table, bounded transactions of <=64 source paths, candidate source bytes plus one Home quota walk per worker attempt. Candidate CPU: 232.99/259.90/238.29 s; average RSS: 373946317/387788114/371558625 bytes; peak: 411766784/422440960/405184512 bytes. Baseline CPU: 1400.15/1416.29/1386.69 s across its longer windows; raw record includes all RSS and descriptor values.

#1499 / UX gap left: the matched 1,000-Note pair still shows a Tag-read tail regression: p95 47.36 -> 60.81 ms (+28.4%), p50 22.12 -> 20.47 ms. Candidate completed in 24.249 s; baseline was incomplete at 847/1000 after 900 s. Terminal candidate ordinary-read p95 was 29.04 ms. The reader-pool recursive prefix grouping still does work per request; shorter startup-overlapped windows weight initial bursts more heavily. CPU/query and window weighting are supported explanations, not isolated attribution. #1499 remains open, with its numbers posted. This is a performance gap, not a claimed resolution.

Safety and UX gaps closed
The 71-test Tags suite covers subprocess SIGKILL windows, retry, failed-prefix repair, concurrent source edits, byte-exact Undo and quota. The new overlapping namespace replay test passed. Final review also fixed an admission/repair self-deadlock on saved receipt retries: release the User mutex before terminal repair. Its regression verifies stable receipt identity, failure, source-backed projection, Undo and unchanged source bytes. The new 5,000-Note timing fixture uses a separate User to avoid unrelated parallel fixture mutex waits; its one-second bound is unchanged. The real-server quota probe passed at a 64/128 prefix: refused Undo returned 507 without changing files; raising quota allowed exact Undo. The real-server failure probe passed with 64 old/64 new Notes, exact foreign bytes (10 Notes), cross-User receipt/Undo 404 and exact Undo of 128 Notes. Malformed receipt and rename probes returned 4xx. The separate two-User probe passed under perf-run: acceptance 149.601 ms, 20/20 writes succeeded, p95 311.465 ms, max 330.594 ms, load5 4.01. Final receipt and other-User bytes were checked.

No visible control or CSS was added. Existing TagRow and ItemActionMenu remain the UI controls. Headful macOS Chromium 155.0.8059.39 showed page/sidebar Tag projection at 340.6 ms after trusted CDP Enter; HTTP/1.1 queue 0.5 ms. Observed response 790.1 ms includes the tunnel and shared build host. This confirms rendering, not the reserved-server acceptance bound. Mac browser, download directory and /tmp/mac-bench.lock were removed and verified.

Gates (verbatim summaries)
cargo fmt --check: exit 0, no output.
cargo clippy -p <crate> --all-targets -- -D warnings and cargo test -p <crate> for calternal-fs, calternal-tags and calternal-server.

calternal-fs-clippy-final
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 10.17s

calternal-tags-clippy-last
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 58s

calternal-server-clippy-last
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 51s

calternal-fs-test-final
    Finished `test` profile [unoptimized + debuginfo] target(s) in 11.15s
test result: ok. 105 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 52.97s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.22s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.13s
test result: ok. 54 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 317.31s
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.14s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-tags-test-last
    Finished `test` profile [unoptimized + debuginfo] target(s) in 34.85s
test result: ok. 71 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 170.78s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

calternal-server-test-last
    Finished `test` profile [unoptimized + debuginfo] target(s) in 1m 27s
test result: ok. 274 passed; 0 failed; 12 ignored; 0 measured; 0 filtered out; finished in 251.60s

server-integration-exact
    Finished `test` profile [unoptimized + debuginfo] target(s) in 1m 00s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 32.44s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.34s

web-check-exact
perf-lint: PASS; 0 violations; 22260 scoped exceptions
svelte-check found 0 errors and 3 warnings in 1 file

web-test-last
 Test Files  305 passed (305)
      Tests  2218 passed (2218)
   Duration  204.89s (environment 28%, transform 25%, import 24%, tests 17%, setup 5%)

Final server unit command: cargo test -p calternal-server -- --test-threads=4 (274 passed). Final integration command after exact pin refresh: cargo test -p calternal-server --test perf_guards --test private_index_permissions (both passed). The first retry-check server run had a startup/backfill timeout; its failing group passed alone, then all server units passed. That run also exposed a stale performance exception fingerprint after the retry-body change. Only the existing live start_rewrite:mutation hash was refreshed in exceptions/adoption; no rule, limit or ratchet changed. The final integration checks and bun run check both passed with 22,260 scoped exceptions. The web test command passed all 305 files / 2,218 tests. These corrected failures are retained in artifacts.

All Rust commands used CARGO_PROFILE_DEV_DEBUG=line-tables-only, CARGO_INCREMENTAL=0 and CARGO_BUILD_JOBS=4. Web gates were cd apps/web && bun run check and bun run test --maxWorkers=2. The ignored diagnostic was run separately and passed. Only the documented projection staging boundary changed two existing test setups; their assertions were retained. Existing test expectations were not weakened.

Decisions

Keep the existing HTTP 200 pending receipt contract rather than change all clients to 202. Response status does not change acceptance work.
Reuse the existing 64-source batch bound for projection. Keep all memberships for one path in one batch.
Store each original Tag spelling in the durable witness and add serde defaults for old receipts. This prevents overlapping namespace replay from transforming a projected Tag twice.
Admission keeps this User's writer mutex for Index planning and the short intent commit. It no longer holds a root namespace lock or runs projection in that transaction.

Known gaps / UX gaps left

The comparative profile uses a 65,536 descriptor limit; #1497 remains a production limit issue.
Two local release probes exceeded the first write's 1-second timing cap on the shared build host. They remain failed timing diagnostics, not successful samples. The reserved strict real-server regression passed without changing its bound.
Initial perf attempts failed before measurement because of missing browser helper/broken supplied runtime symlink. They do not count as paired results.
The new forward-path measurements use the frozen 9f24ac50a release binary; the subsequent saved-receipt mutex-release fix is covered by focused/full Rust checks and leaves that measured forward path unchanged.
GPU renderer metadata was not retained from the real Mac run. No new visual styling was changed in this job.

For the merge round

bash tests/adversarial/run.sh — full combined authz/robustness/XUser coverage.
CALTERNAL_SERVER_BIN=<merge binary> CALTERNAL_E2E_ASSET_OVERRIDE=1 bun tests/adversarial/tag-rewrite-cross-user-1501.mjs — repeat the strict User-isolation contract on the combined build.
cd apps/web && bun run test:e2e (use the configured shared-server harness) — full combined UI behaviour; the focused probes and full web unit suite passed here.
The orchestrator owns full workspace gates and deployment checks. No push, deploy or merge into dev/main occurred.

Cleanup
cargo clean first removed 28,022 files / 17.3 GiB. After the late retry checks it removed 18,096 files / 12.3 GiB. apps/web/build was deleted again after its temporary restoration for RustEmbed. Worktree changes are committed. The own perf-VM run directory, binaries and disposable data were also removed.

Diagnostic disclosure
A disposable test session cookie appeared in a local transport-error trace. That disposable Instance was destroyed, so the session is invalid. The local trace was redacted and the new regression now hides transport headers. No live owner credential was read or printed.

tagfix5 / #1186 / #1501 — final report Head: 4c4734791ceeb13e2ba66b9fd58a8c9b45457551 on job/tagperf-1186. Start: e1f7e492524e50323e7d4d105b9c819ad21d5130. Merged origin/dev once at 9dc65437e86c74efe92d723f9a99774246e60650 before gates. No push, deploy or merge into dev/main. Built - Acceptance records the immutable intent and queue receipt in a FULL transaction; forward projection is deferred to the background worker. - Projection uses indexed witness seeks, at most 64 source paths per transaction, and yields after each commit. Per-User admission locks remain keyed by User; acceptance does not acquire the root namespace lock. - Saved exact membership names make child-to-parent and nested namespace replay idempotent. Legacy receipts remain readable. Distinct item counts do not count each membership twice. - Added a real-server two-User test with >=2,000 Notes and 20 conditional writes, each capped at 1 second; no unknown outcome is retried. Files changed by this job - crates/calternal-tags/src/{lib.rs,rewrite.rs}; docs/DESIGN.md - tests/adversarial/tag-rewrite-cross-user-1501.mjs - bench/tag-rewrite-1186-paired.sh (isolated run root) - docs/perf/2026-10-09-tagfix5-1501.md, docs/perf/runs/tag-rewrite-1186-tagfix5.json, docs/perf/baseline.json - Merge resolutions: calternal-fs/src/write.rs; calternal-server/src/upgrade_tests.rs; exact live contracts/perf pins. Both directory-proof and batch-proof invariants are retained. Server upgrade frontiers include Tags 7 and the new dev migrations. The merged frontier-count assertion is 34 (dev 33 plus this branch’s existing Tags migration); content assertions are retained. No new migration was added by tagfix5. Cause A test-build diagnostic measured writer-pool checkout 0.098 ms, then BEGIN IMMEDIATE wait 5,006.733 ms with SQLITE_BUSY, while the former authority projection held its write transaction for 37,783.513 ms. The competing writer requested neither namespace nor User lock. The normal Note-body route uses a User-keyed mutex and retries SQLite busy writes. The blocking resource was SQLite's writer, not pool checkout or a cross-User mutex. Performance (three 5,000-Note pairs on perf-test-69-63) Frozen baseline: origin/dev 9dc65437e; frozen forward-path candidate source: 9f24ac50a; production SPA: e1672b919. Every measured run held perf-run's single shared lock. The later 6b000c795 retry-lock fix does not change the measured new-intent forward path. Raw records preserve hashes, CPU/RSS, actual run order and load5 in docs/perf/runs/tag-rewrite-1186-tagfix5.json. | Pair | Build | Acceptance ms | UI page/sidebar ms | Completion s | Other-User write p95/max ms | Tag read p95 ms | load5 before/after | | --- | --- | ---: | ---: | ---: | ---: | ---: | --- | | 1 | origin/dev | 490.20 | >900,376 | >900.05 | 649.23 / 1814.89 | 215.81 | 3.05 / 7.10 | | 1 | candidate | 246.80 | 90.80 | 109.17 | 197.57 / 511.36 | 196.00 | 2.17 / 3.47 | | 2 | origin/dev | 146.10 | >900,003.80 | >900.05 | 854.48 / 2670.44 | 203.31 | 1.08 / 6.72 | | 2 | candidate | 393.80 | 138.30 | 120.65 | 233.29 / 445.66 | 234.42 | 2.69 / 4.01 | | 3 | origin/dev | 260.10 | >900,626.40 | >900.01 | 635.40 / 2443.00 | 208.33 | 3.68 / 6.78 | | 3 | candidate | 365.50 | 194.00 | 111.95 | 223.17 / 418.92 | 197.99 | 5.75 / 6.27 | All candidate samples completed and verified every source. Their 309/327/303 independent writes had zero errors. No capped baseline is represented as completed latency. Expected cost: accepted witness rows times <=2 exact membership-name seeks per projection table, bounded transactions of <=64 source paths, candidate source bytes plus one Home quota walk per worker attempt. Candidate CPU: 232.99/259.90/238.29 s; average RSS: 373946317/387788114/371558625 bytes; peak: 411766784/422440960/405184512 bytes. Baseline CPU: 1400.15/1416.29/1386.69 s across its longer windows; raw record includes all RSS and descriptor values. #1499 / UX gap left: the matched 1,000-Note pair still shows a Tag-read tail regression: p95 47.36 -> 60.81 ms (+28.4%), p50 22.12 -> 20.47 ms. Candidate completed in 24.249 s; baseline was incomplete at 847/1000 after 900 s. Terminal candidate ordinary-read p95 was 29.04 ms. The reader-pool recursive prefix grouping still does work per request; shorter startup-overlapped windows weight initial bursts more heavily. CPU/query and window weighting are supported explanations, not isolated attribution. #1499 remains open, with its numbers posted. This is a performance gap, not a claimed resolution. Safety and UX gaps closed The 71-test Tags suite covers subprocess SIGKILL windows, retry, failed-prefix repair, concurrent source edits, byte-exact Undo and quota. The new overlapping namespace replay test passed. Final review also fixed an admission/repair self-deadlock on saved receipt retries: release the User mutex before terminal repair. Its regression verifies stable receipt identity, failure, source-backed projection, Undo and unchanged source bytes. The new 5,000-Note timing fixture uses a separate User to avoid unrelated parallel fixture mutex waits; its one-second bound is unchanged. The real-server quota probe passed at a 64/128 prefix: refused Undo returned 507 without changing files; raising quota allowed exact Undo. The real-server failure probe passed with 64 old/64 new Notes, exact foreign bytes (10 Notes), cross-User receipt/Undo 404 and exact Undo of 128 Notes. Malformed receipt and rename probes returned 4xx. The separate two-User probe passed under perf-run: acceptance 149.601 ms, 20/20 writes succeeded, p95 311.465 ms, max 330.594 ms, load5 4.01. Final receipt and other-User bytes were checked. No visible control or CSS was added. Existing TagRow and ItemActionMenu remain the UI controls. Headful macOS Chromium 155.0.8059.39 showed page/sidebar Tag projection at 340.6 ms after trusted CDP Enter; HTTP/1.1 queue 0.5 ms. Observed response 790.1 ms includes the tunnel and shared build host. This confirms rendering, not the reserved-server acceptance bound. Mac browser, download directory and /tmp/mac-bench.lock were removed and verified. Gates (verbatim summaries) `cargo fmt --check`: exit 0, no output. `cargo clippy -p <crate> --all-targets -- -D warnings` and `cargo test -p <crate>` for calternal-fs, calternal-tags and calternal-server. ```text calternal-fs-clippy-final Finished `dev` profile [unoptimized + debuginfo] target(s) in 10.17s calternal-tags-clippy-last Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 58s calternal-server-clippy-last Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 51s calternal-fs-test-final Finished `test` profile [unoptimized + debuginfo] target(s) in 11.15s test result: ok. 105 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 52.97s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.22s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.13s test result: ok. 54 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 317.31s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.14s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s calternal-tags-test-last Finished `test` profile [unoptimized + debuginfo] target(s) in 34.85s test result: ok. 71 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 170.78s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s calternal-server-test-last Finished `test` profile [unoptimized + debuginfo] target(s) in 1m 27s test result: ok. 274 passed; 0 failed; 12 ignored; 0 measured; 0 filtered out; finished in 251.60s server-integration-exact Finished `test` profile [unoptimized + debuginfo] target(s) in 1m 00s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 32.44s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.34s web-check-exact perf-lint: PASS; 0 violations; 22260 scoped exceptions svelte-check found 0 errors and 3 warnings in 1 file web-test-last Test Files 305 passed (305) Tests 2218 passed (2218) Duration 204.89s (environment 28%, transform 25%, import 24%, tests 17%, setup 5%) ``` Final server unit command: `cargo test -p calternal-server -- --test-threads=4` (274 passed). Final integration command after exact pin refresh: `cargo test -p calternal-server --test perf_guards --test private_index_permissions` (both passed). The first retry-check server run had a startup/backfill timeout; its failing group passed alone, then all server units passed. That run also exposed a stale performance exception fingerprint after the retry-body change. Only the existing live `start_rewrite:mutation` hash was refreshed in exceptions/adoption; no rule, limit or ratchet changed. The final integration checks and `bun run check` both passed with 22,260 scoped exceptions. The web test command passed all 305 files / 2,218 tests. These corrected failures are retained in artifacts. All Rust commands used CARGO_PROFILE_DEV_DEBUG=line-tables-only, CARGO_INCREMENTAL=0 and CARGO_BUILD_JOBS=4. Web gates were `cd apps/web && bun run check` and `bun run test --maxWorkers=2`. The ignored diagnostic was run separately and passed. Only the documented projection staging boundary changed two existing test setups; their assertions were retained. Existing test expectations were not weakened. Decisions - Keep the existing HTTP 200 pending receipt contract rather than change all clients to 202. Response status does not change acceptance work. - Reuse the existing 64-source batch bound for projection. Keep all memberships for one path in one batch. - Store each original Tag spelling in the durable witness and add serde defaults for old receipts. This prevents overlapping namespace replay from transforming a projected Tag twice. - Admission keeps this User's writer mutex for Index planning and the short intent commit. It no longer holds a root namespace lock or runs projection in that transaction. Known gaps / UX gaps left - The comparative profile uses a 65,536 descriptor limit; #1497 remains a production limit issue. - Two local release probes exceeded the first write's 1-second timing cap on the shared build host. They remain failed timing diagnostics, not successful samples. The reserved strict real-server regression passed without changing its bound. - Initial perf attempts failed before measurement because of missing browser helper/broken supplied runtime symlink. They do not count as paired results. - The new forward-path measurements use the frozen 9f24ac50a release binary; the subsequent saved-receipt mutex-release fix is covered by focused/full Rust checks and leaves that measured forward path unchanged. - GPU renderer metadata was not retained from the real Mac run. No new visual styling was changed in this job. For the merge round - `bash tests/adversarial/run.sh` — full combined authz/robustness/XUser coverage. - `CALTERNAL_SERVER_BIN=<merge binary> CALTERNAL_E2E_ASSET_OVERRIDE=1 bun tests/adversarial/tag-rewrite-cross-user-1501.mjs` — repeat the strict User-isolation contract on the combined build. - `cd apps/web && bun run test:e2e` (use the configured shared-server harness) — full combined UI behaviour; the focused probes and full web unit suite passed here. - The orchestrator owns full workspace gates and deployment checks. No push, deploy or merge into dev/main occurred. Cleanup `cargo clean` first removed 28,022 files / 17.3 GiB. After the late retry checks it removed 18,096 files / 12.3 GiB. apps/web/build was deleted again after its temporary restoration for RustEmbed. Worktree changes are committed. The own perf-VM run directory, binaries and disposable data were also removed. Diagnostic disclosure A disposable test session cookie appeared in a local transport-error trace. That disposable Instance was destroyed, so the session is invalid. The local trace was redacted and the new regression now hides transport headers. No live owner credential was read or printed.

kayg referenced this issue

2026-10-09 11:13:24 +00:00

Merge round 36 #1526

kayg referenced this issue

2026-10-09 12:35:36 +00:00

Merge round 36 #1526

kayg commented

2026-10-10 09:14:51 +00:00

Author

Owner

Deployed to calternal.cloud in merge round 36 (build dbd621434, live 2026-10-10 ~14:40 IST). Closing.

Deployed to calternal.cloud in merge round 36 (build dbd621434, live 2026-10-10 ~14:40 IST). Closing.