kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 190 Pull requests 61 Projects Releases Packages Wiki Activity Actions 26

Train BM9 #869

Closed
kayg wants to merge 120 commits from train/2026-11-02-bm9 into dev
pull from: train/2026-11-02-bm9
merge into: kayg:dev
Conversation 16 Commits 120 Files changed 192 +8609 -70036
kayg commented 2026-11-02 16:27:47 +00:00
Owner
Copy link

Replaces #862. Assembly and first-run gate evidence are recorded in PR comments. Never merge as part of this test task.

Replaces #862. Assembly and first-run gate evidence are recorded in PR comments. Never merge as part of this test task.
kayg added 186 commits 2026-11-02 16:27:47 +00:00
test: write authority fixtures atomically
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
b130c95f60
test(notes): cover cache reuse and first page during rebuild
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
e0e78626ed
docs(perf): record attachment index flush baseline
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-native-contract (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-editor-contract (pull_request) Has been cancelled
Details
8f57b68941
fix(tasks): match Upcoming date selection and group order to mockup
Some checks failed
CI / web (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-native-contract (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-editor-contract (pull_request) Has been cancelled
Details
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
3f62ac8fd6
Replace svelte-sonner with our own small toast stack
Some checks failed
CI / web (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-native-contract (pull_request) Has been cancelled
Details
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-editor-contract (pull_request) Has been cancelled
Details
c98b6ee9e6 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
test(notes): run crash rounds until confirmed append coverage
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-editor-contract (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-native-contract (pull_request) Has been cancelled
Details
50cce9dcce 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
e2e: drop the recovery receipt wait (the receipt tag does not fire on rejoin)
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-native-contract (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-editor-contract (pull_request) Has been cancelled
Details
85845e3d11 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
test(notes): reproduce capture racing a daily placeholder reader
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-native-contract (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-editor-contract (pull_request) Has been cancelled
Details
68cecd2ba8
test(web): hold Undo toast during composer server assertions
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-native-contract (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-editor-contract (pull_request) Has been cancelled
Details
24fdae745d
test(auth): observe idle upload completion before checking workers
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-native-contract (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-editor-contract (pull_request) Has been cancelled
Details
144e717145
docs: link historical CI allocation observations
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-native-contract (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-editor-contract (pull_request) Has been cancelled
Details
9c05d871a1
perf: gate named app chunks with explicit brotli budgets
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-native-contract (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-editor-contract (pull_request) Has been cancelled
Details
da1071ed09 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	Cargo.lock
# Conflicts:
#	crates/engine/src/execute.rs
chore: keep inherited attachment measurements in PR comments
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-native-contract (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-editor-contract (pull_request) Has been cancelled
Details
383eb8f88b
kayg commented 2026-11-02 16:28:33 +00:00
Author
Owner
Copy link

Assembly head 383eb8f88b45b4cdd1604274dea26d642fd04ba6, base 55f35fcde3bc91ca3781055577b93e93819f6cd4. Current source heads captured immediately before each merge.

PR Source head Decision
#779 7b97e4d14231be07f6a80cb9ca172d02442c487e included, --no-ff
#858 626388da9052eaecb159d735f0cadc5e59e88312 included, --no-ff
#866 ba1f47dc94797d4a8a937298464b5e4828e301c2 included, --no-ff
#824 f1c1aa961e169ef99e132063a97ccbd9a89a5def included, --no-ff
#827 0543321b34f9a739275f70e31c79c5575d680259 included, --no-ff
#829 d455842942731aabc05474c7929597d57090a7cc authorization/admission conflicts with #827 perf cache require implementation choices
#835 19071be8455d718822ba98f6b9a402c5fd345242 depends on excluded #829
#828 5d7b9b36cd008de78851809724c90a85d92b40a7 included, --no-ff
#833 ccaece3dfecbe2048634fc792d7d4a2c6e31f324 included, --no-ff
#836 5f7d1a59776411e7f3353d0023817b43affae51e included, --no-ff
#831 316b43e9578c3e4d2944df8df43caebc611356f7 included, --no-ff
#844 50cce9dcceb87f23a9f346300b3336e315ff6c0c included, --no-ff
#847 40448327a15730271d56226b898e48bcc87b2a4d included, --no-ff
#855 c38c3083d51f4adcf5cd76a70f1f6960b427413e included, --no-ff
#857 144e717145e35f70356f593b7c04a527e00f7b17 included, --no-ff
#856 85845e3d11bea351608bde7481779ba2303372e5 included, --no-ff

#840 excluded as requested. #829 conflicts with #827 0543321b in authorize_write/admit: resource authorization and cleanup overlap the new never-shared cache and writer locking. This requires implementation choices, so #829 and dependent #835 are excluded. Mechanical resolutions only: #824 keeps both Cargo.lock package entries in name order; #828 retains credential checks then starts runtime timing. #831 measurement files moved to a PR comment and removed in cleanup commit.

First runs only, CI=1, retries=0. Nix with strace; heavy commands through ~/bin/slot, -j4, browser workers=1. Browser halves retain all native projects, admin-setup with dependents; only cross-half scheduling dependencies removed. All evidence stays in comments. Never merge.

Assembly head `383eb8f88b45b4cdd1604274dea26d642fd04ba6`, base `55f35fcde3bc91ca3781055577b93e93819f6cd4`. Current source heads captured immediately before each merge. | PR | Source head | Decision | |---|---|---| | #779 | `7b97e4d14231be07f6a80cb9ca172d02442c487e` | included, --no-ff | | #858 | `626388da9052eaecb159d735f0cadc5e59e88312` | included, --no-ff | | #866 | `ba1f47dc94797d4a8a937298464b5e4828e301c2` | included, --no-ff | | #824 | `f1c1aa961e169ef99e132063a97ccbd9a89a5def` | included, --no-ff | | #827 | `0543321b34f9a739275f70e31c79c5575d680259` | included, --no-ff | | #829 | `d455842942731aabc05474c7929597d57090a7cc` | authorization/admission conflicts with #827 perf cache require implementation choices | | #835 | `19071be8455d718822ba98f6b9a402c5fd345242` | depends on excluded #829 | | #828 | `5d7b9b36cd008de78851809724c90a85d92b40a7` | included, --no-ff | | #833 | `ccaece3dfecbe2048634fc792d7d4a2c6e31f324` | included, --no-ff | | #836 | `5f7d1a59776411e7f3353d0023817b43affae51e` | included, --no-ff | | #831 | `316b43e9578c3e4d2944df8df43caebc611356f7` | included, --no-ff | | #844 | `50cce9dcceb87f23a9f346300b3336e315ff6c0c` | included, --no-ff | | #847 | `40448327a15730271d56226b898e48bcc87b2a4d` | included, --no-ff | | #855 | `c38c3083d51f4adcf5cd76a70f1f6960b427413e` | included, --no-ff | | #857 | `144e717145e35f70356f593b7c04a527e00f7b17` | included, --no-ff | | #856 | `85845e3d11bea351608bde7481779ba2303372e5` | included, --no-ff | #840 excluded as requested. #829 conflicts with #827 0543321b in authorize_write/admit: resource authorization and cleanup overlap the new never-shared cache and writer locking. This requires implementation choices, so #829 and dependent #835 are excluded. Mechanical resolutions only: #824 keeps both Cargo.lock package entries in name order; #828 retains credential checks then starts runtime timing. #831 measurement files moved to a PR comment and removed in cleanup commit. First runs only, CI=1, retries=0. Nix with strace; heavy commands through ~/bin/slot, -j4, browser workers=1. Browser halves retain all native projects, admin-setup with dependents; only cross-half scheduling dependencies removed. All evidence stays in comments. Never merge.
kayg commented 2026-11-02 16:28:33 +00:00
Author
Owner
Copy link

Artifacts inherited from #831, moved here under ADR 0003.

docs/perf/attachment-index-flush.md

# Attachment index flushes

## Plan and cost

Keep one SQLite connection per recent user in a bounded cache of 64 entries.
The duplicate lookup and optional row insert use that shared connection. Use
WAL with `synchronous=FULL`, so a changed row is durable in the WAL before the
action is acknowledged; SQLite can checkpoint it later instead of flushing the
database and directory on every upload. Leave the attachment file fsync and
its parent-directory fsync unchanged. A new file gets one index write and one
WAL sync; a duplicate or an action with no attachment change gets no index
write or sync.

The lookup remains O(log F) for F indexed files. Cache lookup is O(1), with an
O(64) least-recently-used update, and at most 64 cached connections are open.
If all entries are in use when the cache is full, the action uses a temporary
connection; SQLite still commits the row durably, with an extra close-time
checkpoint possible in that case. The stronger FULL setting preserves the
index row's durability while deferring database checkpoint work.

| Failure | Effect | Detection | Recovery | Test |
|---|---|---|---|---|
| Index open or duplicate lookup fails | The add action fails before placement | Returned action error | Retry with the same idempotency key | Existing attachment action tests |
| Index row insert fails after placement | The file remains durable; the index remains a hint | Existing warning | A later upload can retry the index write | Existing upload and retry tests |
| No attachment changed | No index row is written or flushed | `strace` count between save markers | No recovery needed | Linux flush-count regression test |
| One attachment changed | One row is committed with one WAL sync | `strace` count between save markers | SQLite's FULL-synced WAL is replayed after restart | Linux flush-count regression test |
| Cache is full and all entries are active | One action uses a temporary connection and may checkpoint at close | Cache-full debug event and sync trace | Reopen the per-user index on the next action | Bounded-cache test and Linux flush-count regression test |

Gate tests: `cargo test --locked -p calternal-attachments -j4`; the Linux
flush-count regression test; the unchanged `1k / SSD / 4 clients / 256 KiB`
matrix cell on dev-01 before and after; and one matched baseline/head pair on
perf-01 under `~/bin/perf-lock`. The perf budgets stay unchanged.

## Baseline on current dev

Commit: `55f35fcde3bc91ca3781055577b93e93819f6cd4`. The server was built on
dev-01 as `~/bin/calternal-dev-55f35fc` (SHA256
`9e3269898ce6bd45e3ef9326149f41995d176c26394841ef9eef376d2b1f5dc4`). The
matrix driver was `~/bin/calternal-perf-0f38de31` (SHA256
`e83e22618a1ce06bc4bc42ee16ed966eff0fe4ff922c42da1cc42ae4f8700a8b`, built
from the #788 matrix source). It used seed `75931004`, one 1k-note Home, four
upload clients, 256 KiB unique attachments, 60 seconds, and 60 overlapping
note saves.

The plain run used:

```sh
calternal-perf --bin ~/bin/calternal-dev-55f35fc --work <run>/work \
  --sizes 1k --runs 1 --cold-runs 0 --idle-secs 0 --soak-secs 0 \
  --seed 75931004 --upload-sizes 256k --upload-clients 4 \
  --upload-secs 60 --upload-only --json <run>/result.json --gate

The trace run used the same arguments under:

nix shell nixpkgs#strace -c strace -f -tt -T -yy \
  -e trace=fsync,fdatasync,openat,close,renameat2,rename,fcntl \
  -o <run>/trace <calternal-perf command above>

Untraced warm run: 696 uploads; 60 save acknowledgements; p50 17.88 ms and p95
56.26 ms. This is a single dev-01 diagnostic run, not perf-VM qualification.

The matching strace -f -tt -T -yy run completed 465 uploads. It found 932
attachment WAL syncs, 469 attachment database syncs, and 467 attachment
directory syncs: 1,868 total, or 4.017 attachment-index sync calls per upload.
Two WAL syncs, one database sync, and one directory sync occurred per new
attachment. The trace also showed a separate pending-marker sync per upload.
Traced latency is excluded from the before/after comparison.

This reproduces the inherited amplification reported in
#788 comment 73181
on current dev. The original profile at 133cf2c2 is not used as the current
baseline.

Fix and current-dev result

The implementation keeps up to 64 per-user index connections in an LRU cache.
The connection uses WAL with synchronous=FULL, so each changed row is durable
in the WAL before acknowledgement. SQLite can checkpoint after its normal
page threshold, cache eviction, or shutdown. Attachment bytes still use the
existing file and parent-directory durability barriers.

The Linux regression test measures sync syscalls between real upload action
markers under strace. It passed with 0 attachment-index sync calls for a
duplicate/no-change save and 1 WAL sync call for one new attachment. The
bounded-cache test also passed with 65 busy users: 64 cached connections and
one scoped connection.

On dev-01, the same untraced matrix cell after the change completed 1,520
uploads and 59 overlapping note saves. Save-ack p50 was 13.29 ms and p95 was
40.62 ms. Against the single baseline run, p50 improved 25.7% and p95 improved
27.8%. These shared-host samples are diagnostic only, not qualification.

The matching trace run completed 807 uploads and 60 save samples. It found
814 attachment WAL syncs, 5 database syncs, and 2 attachment-directory syncs:
821 total, or 1.017 attachment-index sync calls per upload. The seven calls
above one per upload came from SQLite checkpoints during the run. Traced
save-ack p95 was 58.37 ms and is excluded from the latency comparison.

Targeted gates on dev-01 passed:

cargo fmt --all --check
cargo test --locked -p calternal-attachments -j4 -- --nocapture
cargo clippy --locked -p calternal-attachments --all-targets -j4 -- -D warnings
cargo build --release --locked -p calternal -j4

The test result was 21 passed, 0 failed; targeted Clippy passed with warnings
denied. The current-dev binary was built from implementation commit
eef6473203e20df1be29346b8ef03ff47a269522 and has SHA256
196ff9b8ea473f0cf22ea0546c76056d6efb2d413317ec0a8c6063a759271bbd.

Perf-01 matched comparison

On perf-01, I ran five alternating matched pairs with the same 1k-note Home,
four upload clients, 256 KiB attachments, 60-second upload window, and seed
75931004. Homes and app state were on the VM's ext4 non-rotational SSD. The
baseline was current origin/dev at 55f35fcde3bc91ca3781055577b93e93819f6cd4;
the fixed binary was built from eef6473203e20df1be29346b8ef03ff47a269522.
Both were measured with the same matrix driver and unchanged budgets file
(SHA256 8feee55baeb3940c09015f37db8da85bd0cf2c5f33d5673568bd976fcf02ace3).
Each run exited 0 and collected 59 or 60 save acknowledgements.

Pair Order Base save p50 / p95 Fixed save p50 / p95 Save p95 ratio Response p95 ratio Throughput ratio
1 base / fixed 8.05 / 11.28 ms 9.95 / 16.15 ms 143.2% 98.3% 114.4%
2 fixed / base 11.69 / 20.98 ms 12.20 / 20.45 ms 97.5% 47.8% 159.9%
3 base / fixed 12.36 / 17.88 ms 12.03 / 17.67 ms 98.8% 70.4% 147.3%
4 fixed / base 11.90 / 18.94 ms 12.56 / 18.33 ms 96.8% 66.6% 148.9%
5 base / fixed 11.70 / 20.47 ms 13.23 / 22.93 ms 112.0% 68.0% 150.4%
Median 11.70 / 18.94 ms 12.20 / 18.33 ms 98.8% paired 68.0% paired 148.9% paired

The five-pair median meets the existing #788 small all-SSD cell limits: upload
throughput at least 95% of current dev, upload response p95 at most 105%, and
overlapping save-ack p95 at most 105%. Save-ack p50 was 4.2% higher by the
median of run p50s; median run p95 fell 3.2%, and the median paired p95 ratio
was 98.8%. The first and fifth save-ack pairs were noisy and slower on the
fixed run, which is why the full five-pair table is retained. The full 18-cell
matrix still needs requalification after rebasing #788.

The unchanged perf/budgets.json was passed to --gate; it has no entry for
uploads.256KiB.save_ack, so the upload comparison uses the existing relative
matrix limits from #788 comment 72886.
No budget was edited. Individual JSON and Markdown outputs are retained on
perf-01. The compact per-run data is in
attachment-index-flush-perf01.json.


### docs/perf/attachment-index-flush-perf01.json

```json
{
  "machine": "perf-01: Linux 6.12.113+deb13-amd64, 4 CPUs, 7.7 GiB RAM, ext4 non-rotational SSD for Homes and state",
  "baseline": {
    "origin_dev": "55f35fcde3bc91ca3781055577b93e93819f6cd4",
    "binary_sha256": "9e3269898ce6bd45e3ef9326149f41995d176c26394841ef9eef376d2b1f5dc4"
  },
  "fixed": {
    "implementation_commit": "eef6473203e20df1be29346b8ef03ff47a269522",
    "binary_sha256": "196ff9b8ea473f0cf22ea0546c76056d6efb2d413317ec0a8c6063a759271bbd"
  },
  "matrix_driver_sha256": "e83e22618a1ce06bc4bc42ee16ed966eff0fe4ff922c42da1cc42ae4f8700a8b",
  "config": {
    "notes": 1000,
    "upload_bytes": 262144,
    "upload_clients": 4,
    "upload_seconds": 60,
    "seed": 75931004,
    "matched_pairs": 5,
    "order": [
      "base/fixed",
      "fixed/base",
      "base/fixed",
      "fixed/base",
      "base/fixed"
    ]
  },
  "command": "calternal-perf --bin <base-or-fixed-server> --work <run>/work --sizes 1k --runs 1 --cold-runs 0 --idle-secs 0 --soak-secs 0 --seed 75931004 --upload-sizes 256k --upload-clients 4 --upload-secs 60 --upload-only --json <run>/result.json --md <run>/result.md --budgets <unchanged perf/budgets.json> --gate",
  "budget_file_sha256": "8feee55baeb3940c09015f37db8da85bd0cf2c5f33d5673568bd976fcf02ace3",
  "matrix_thresholds": {
    "throughput_ratio_pct_min": 95,
    "upload_response_p95_ratio_pct_max": 105,
    "save_ack_p95_ratio_pct_max": 105
  },
  "pairs": [
    {
      "pair": 1,
      "order": "base/fixed",
      "base": {
        "save_ack_ms": {
          "p50": 8.051699000000001,
          "p95": 11.276453,
          "samples": 60
        },
        "upload_response_ms": {
          "p50": 122.414671,
          "p95": 143.195845,
          "samples": 1924
        },
        "throughput_mib_s": 7.8760662691913295,
        "exit_code": 0
      },
      "fixed": {
        "save_ack_ms": {
          "p50": 9.95182,
          "p95": 16.15159,
          "samples": 59
        },
        "upload_response_ms": {
          "p50": 116.10380400000001,
          "p95": 140.805797,
          "samples": 2201
        },
        "throughput_mib_s": 9.010913115334635,
        "exit_code": 0
      },
      "paired_ratio_pct": {
        "save_ack_p95": 143.23289424431601,
        "upload_response_p95": 98.33092363818238,
        "throughput": 114.40880266056756
      }
    },
    {
      "pair": 2,
      "order": "fixed/base",
      "base": {
        "save_ack_ms": {
          "p50": 11.694977,
          "p95": 20.984858,
          "samples": 60
        },
        "upload_response_ms": {
          "p50": 181.23751900000002,
          "p95": 305.759208,
          "samples": 1220
        },
        "throughput_mib_s": 4.994277119581212,
        "exit_code": 0
      },
      "fixed": {
        "save_ack_ms": {
          "p50": 12.195746,
          "p95": 20.453059999999997,
          "samples": 59
        },
        "upload_response_ms": {
          "p50": 122.14196,
          "p95": 146.29314,
          "samples": 1951
        },
        "throughput_mib_s": 7.987411446142995,
        "exit_code": 0
      },
      "paired_ratio_pct": {
        "save_ack_p95": 97.46580129348503,
        "upload_response_p95": 47.84586569180281,
        "throughput": 159.93128244378977
      }
    },
    {
      "pair": 3,
      "order": "base/fixed",
      "base": {
        "save_ack_ms": {
          "p50": 12.356477,
          "p95": 17.879756999999998,
          "samples": 60
        },
        "upload_response_ms": {
          "p50": 179.02521199999998,
          "p95": 211.695008,
          "samples": 1317
        },
        "throughput_mib_s": 5.391684547783172,
        "exit_code": 0
      },
      "fixed": {
        "save_ack_ms": {
          "p50": 12.034103,
          "p95": 17.673111,
          "samples": 60
        },
        "upload_response_ms": {
          "p50": 123.075991,
          "p95": 149.01044299999998,
          "samples": 1940
        },
        "throughput_mib_s": 7.942181138908136,
        "exit_code": 0
      },
      "paired_ratio_pct": {
        "save_ack_p95": 98.84424603757199,
        "upload_response_p95": 70.38920965013969,
        "throughput": 147.30426211922241
      }
    },
    {
      "pair": 4,
      "order": "fixed/base",
      "base": {
        "save_ack_ms": {
          "p50": 11.903978,
          "p95": 18.941137,
          "samples": 60
        },
        "upload_response_ms": {
          "p50": 179.25838,
          "p95": 220.128913,
          "samples": 1307
        },
        "throughput_mib_s": 5.350753287242168,
        "exit_code": 0
      },
      "fixed": {
        "save_ack_ms": {
          "p50": 12.555529,
          "p95": 18.334640999999998,
          "samples": 60
        },
        "upload_response_ms": {
          "p50": 122.91566,
          "p95": 146.669366,
          "samples": 1946
        },
        "throughput_mib_s": 7.967140604998429,
        "exit_code": 0
      },
      "paired_ratio_pct": {
        "save_ack_p95": 96.79799581197263,
        "upload_response_p95": 66.6288512495403,
        "throughput": 148.8975510045385
      }
    },
    {
      "pair": 5,
      "order": "base/fixed",
      "base": {
        "save_ack_ms": {
          "p50": 11.702833,
          "p95": 20.466426,
          "samples": 60
        },
        "upload_response_ms": {
          "p50": 179.29509800000002,
          "p95": 215.944592,
          "samples": 1311
        },
        "throughput_mib_s": 5.367088973256576,
        "exit_code": 0
      },
      "fixed": {
        "save_ack_ms": {
          "p50": 13.234993000000001,
          "p95": 22.926043,
          "samples": 59
        },
        "upload_response_ms": {
          "p50": 123.30149399999999,
          "p95": 146.776267,
          "samples": 1940
        },
        "throughput_mib_s": 8.074598087107052,
        "exit_code": 0
      },
      "paired_ratio_pct": {
        "save_ack_p95": 112.01781395540189,
        "upload_response_p95": 67.96941087554534,
        "throughput": 150.446510712634
      }
    }
  ],
  "summary": {
    "base": {
      "save_ack_p50_ms": 11.702833,
      "save_ack_p95_ms": 18.941137,
      "upload_response_p95_ms": 215.944592,
      "throughput_mib_s": 5.367088973256576
    },
    "fixed": {
      "save_ack_p50_ms": 12.195746,
      "save_ack_p95_ms": 18.334640999999998,
      "upload_response_p95_ms": 146.669366,
      "throughput_mib_s": 7.987411446142995
    },
    "median_paired_ratio_pct": {
      "save_ack_p95": 98.84424603757199,
      "upload_response_p95": 67.96941087554534,
      "throughput": 148.8975510045385
    }
  }
}
Artifacts inherited from #831, moved here under ADR 0003. ### docs/perf/attachment-index-flush.md ```markdown # Attachment index flushes ## Plan and cost Keep one SQLite connection per recent user in a bounded cache of 64 entries. The duplicate lookup and optional row insert use that shared connection. Use WAL with `synchronous=FULL`, so a changed row is durable in the WAL before the action is acknowledged; SQLite can checkpoint it later instead of flushing the database and directory on every upload. Leave the attachment file fsync and its parent-directory fsync unchanged. A new file gets one index write and one WAL sync; a duplicate or an action with no attachment change gets no index write or sync. The lookup remains O(log F) for F indexed files. Cache lookup is O(1), with an O(64) least-recently-used update, and at most 64 cached connections are open. If all entries are in use when the cache is full, the action uses a temporary connection; SQLite still commits the row durably, with an extra close-time checkpoint possible in that case. The stronger FULL setting preserves the index row's durability while deferring database checkpoint work. | Failure | Effect | Detection | Recovery | Test | |---|---|---|---|---| | Index open or duplicate lookup fails | The add action fails before placement | Returned action error | Retry with the same idempotency key | Existing attachment action tests | | Index row insert fails after placement | The file remains durable; the index remains a hint | Existing warning | A later upload can retry the index write | Existing upload and retry tests | | No attachment changed | No index row is written or flushed | `strace` count between save markers | No recovery needed | Linux flush-count regression test | | One attachment changed | One row is committed with one WAL sync | `strace` count between save markers | SQLite's FULL-synced WAL is replayed after restart | Linux flush-count regression test | | Cache is full and all entries are active | One action uses a temporary connection and may checkpoint at close | Cache-full debug event and sync trace | Reopen the per-user index on the next action | Bounded-cache test and Linux flush-count regression test | Gate tests: `cargo test --locked -p calternal-attachments -j4`; the Linux flush-count regression test; the unchanged `1k / SSD / 4 clients / 256 KiB` matrix cell on dev-01 before and after; and one matched baseline/head pair on perf-01 under `~/bin/perf-lock`. The perf budgets stay unchanged. ## Baseline on current dev Commit: `55f35fcde3bc91ca3781055577b93e93819f6cd4`. The server was built on dev-01 as `~/bin/calternal-dev-55f35fc` (SHA256 `9e3269898ce6bd45e3ef9326149f41995d176c26394841ef9eef376d2b1f5dc4`). The matrix driver was `~/bin/calternal-perf-0f38de31` (SHA256 `e83e22618a1ce06bc4bc42ee16ed966eff0fe4ff922c42da1cc42ae4f8700a8b`, built from the #788 matrix source). It used seed `75931004`, one 1k-note Home, four upload clients, 256 KiB unique attachments, 60 seconds, and 60 overlapping note saves. The plain run used: ```sh calternal-perf --bin ~/bin/calternal-dev-55f35fc --work <run>/work \ --sizes 1k --runs 1 --cold-runs 0 --idle-secs 0 --soak-secs 0 \ --seed 75931004 --upload-sizes 256k --upload-clients 4 \ --upload-secs 60 --upload-only --json <run>/result.json --gate ``` The trace run used the same arguments under: ```sh nix shell nixpkgs#strace -c strace -f -tt -T -yy \ -e trace=fsync,fdatasync,openat,close,renameat2,rename,fcntl \ -o <run>/trace <calternal-perf command above> ``` Untraced warm run: 696 uploads; 60 save acknowledgements; p50 17.88 ms and p95 56.26 ms. This is a single dev-01 diagnostic run, not perf-VM qualification. The matching `strace -f -tt -T -yy` run completed 465 uploads. It found 932 attachment WAL syncs, 469 attachment database syncs, and 467 attachment directory syncs: 1,868 total, or 4.017 attachment-index sync calls per upload. Two WAL syncs, one database sync, and one directory sync occurred per new attachment. The trace also showed a separate pending-marker sync per upload. Traced latency is excluded from the before/after comparison. This reproduces the inherited amplification reported in [#788 comment 73181](https://git.kayg.org/kayg/calternal/pulls/788#issuecomment-73181) on current dev. The original profile at `133cf2c2` is not used as the current baseline. ## Fix and current-dev result The implementation keeps up to 64 per-user index connections in an LRU cache. The connection uses WAL with `synchronous=FULL`, so each changed row is durable in the WAL before acknowledgement. SQLite can checkpoint after its normal page threshold, cache eviction, or shutdown. Attachment bytes still use the existing file and parent-directory durability barriers. The Linux regression test measures sync syscalls between real upload action markers under `strace`. It passed with **0 attachment-index sync calls** for a duplicate/no-change save and **1 WAL sync call** for one new attachment. The bounded-cache test also passed with 65 busy users: 64 cached connections and one scoped connection. On dev-01, the same untraced matrix cell after the change completed 1,520 uploads and 59 overlapping note saves. Save-ack p50 was 13.29 ms and p95 was 40.62 ms. Against the single baseline run, p50 improved 25.7% and p95 improved 27.8%. These shared-host samples are diagnostic only, not qualification. The matching trace run completed 807 uploads and 60 save samples. It found 814 attachment WAL syncs, 5 database syncs, and 2 attachment-directory syncs: 821 total, or 1.017 attachment-index sync calls per upload. The seven calls above one per upload came from SQLite checkpoints during the run. Traced save-ack p95 was 58.37 ms and is excluded from the latency comparison. Targeted gates on dev-01 passed: ```sh cargo fmt --all --check cargo test --locked -p calternal-attachments -j4 -- --nocapture cargo clippy --locked -p calternal-attachments --all-targets -j4 -- -D warnings cargo build --release --locked -p calternal -j4 ``` The test result was 21 passed, 0 failed; targeted Clippy passed with warnings denied. The current-dev binary was built from implementation commit `eef6473203e20df1be29346b8ef03ff47a269522` and has SHA256 `196ff9b8ea473f0cf22ea0546c76056d6efb2d413317ec0a8c6063a759271bbd`. ## Perf-01 matched comparison On perf-01, I ran five alternating matched pairs with the same 1k-note Home, four upload clients, 256 KiB attachments, 60-second upload window, and seed `75931004`. Homes and app state were on the VM's ext4 non-rotational SSD. The baseline was current `origin/dev` at `55f35fcde3bc91ca3781055577b93e93819f6cd4`; the fixed binary was built from `eef6473203e20df1be29346b8ef03ff47a269522`. Both were measured with the same matrix driver and unchanged budgets file (SHA256 `8feee55baeb3940c09015f37db8da85bd0cf2c5f33d5673568bd976fcf02ace3`). Each run exited 0 and collected 59 or 60 save acknowledgements. | Pair | Order | Base save p50 / p95 | Fixed save p50 / p95 | Save p95 ratio | Response p95 ratio | Throughput ratio | |---:|---|---:|---:|---:|---:|---:| | 1 | base / fixed | 8.05 / 11.28 ms | 9.95 / 16.15 ms | 143.2% | 98.3% | 114.4% | | 2 | fixed / base | 11.69 / 20.98 ms | 12.20 / 20.45 ms | 97.5% | 47.8% | 159.9% | | 3 | base / fixed | 12.36 / 17.88 ms | 12.03 / 17.67 ms | 98.8% | 70.4% | 147.3% | | 4 | fixed / base | 11.90 / 18.94 ms | 12.56 / 18.33 ms | 96.8% | 66.6% | 148.9% | | 5 | base / fixed | 11.70 / 20.47 ms | 13.23 / 22.93 ms | 112.0% | 68.0% | 150.4% | | Median | | 11.70 / 18.94 ms | 12.20 / 18.33 ms | 98.8% paired | 68.0% paired | 148.9% paired | The five-pair median meets the existing #788 small all-SSD cell limits: upload throughput at least 95% of current dev, upload response p95 at most 105%, and overlapping save-ack p95 at most 105%. Save-ack p50 was 4.2% higher by the median of run p50s; median run p95 fell 3.2%, and the median paired p95 ratio was 98.8%. The first and fifth save-ack pairs were noisy and slower on the fixed run, which is why the full five-pair table is retained. The full 18-cell matrix still needs requalification after rebasing #788. The unchanged `perf/budgets.json` was passed to `--gate`; it has no entry for `uploads.256KiB.save_ack`, so the upload comparison uses the existing relative matrix limits from [#788 comment 72886](https://git.kayg.org/kayg/calternal/pulls/788#issuecomment-72886). No budget was edited. Individual JSON and Markdown outputs are retained on perf-01. The compact per-run data is in [`attachment-index-flush-perf01.json`](attachment-index-flush-perf01.json). ``` ### docs/perf/attachment-index-flush-perf01.json ```json { "machine": "perf-01: Linux 6.12.113+deb13-amd64, 4 CPUs, 7.7 GiB RAM, ext4 non-rotational SSD for Homes and state", "baseline": { "origin_dev": "55f35fcde3bc91ca3781055577b93e93819f6cd4", "binary_sha256": "9e3269898ce6bd45e3ef9326149f41995d176c26394841ef9eef376d2b1f5dc4" }, "fixed": { "implementation_commit": "eef6473203e20df1be29346b8ef03ff47a269522", "binary_sha256": "196ff9b8ea473f0cf22ea0546c76056d6efb2d413317ec0a8c6063a759271bbd" }, "matrix_driver_sha256": "e83e22618a1ce06bc4bc42ee16ed966eff0fe4ff922c42da1cc42ae4f8700a8b", "config": { "notes": 1000, "upload_bytes": 262144, "upload_clients": 4, "upload_seconds": 60, "seed": 75931004, "matched_pairs": 5, "order": [ "base/fixed", "fixed/base", "base/fixed", "fixed/base", "base/fixed" ] }, "command": "calternal-perf --bin <base-or-fixed-server> --work <run>/work --sizes 1k --runs 1 --cold-runs 0 --idle-secs 0 --soak-secs 0 --seed 75931004 --upload-sizes 256k --upload-clients 4 --upload-secs 60 --upload-only --json <run>/result.json --md <run>/result.md --budgets <unchanged perf/budgets.json> --gate", "budget_file_sha256": "8feee55baeb3940c09015f37db8da85bd0cf2c5f33d5673568bd976fcf02ace3", "matrix_thresholds": { "throughput_ratio_pct_min": 95, "upload_response_p95_ratio_pct_max": 105, "save_ack_p95_ratio_pct_max": 105 }, "pairs": [ { "pair": 1, "order": "base/fixed", "base": { "save_ack_ms": { "p50": 8.051699000000001, "p95": 11.276453, "samples": 60 }, "upload_response_ms": { "p50": 122.414671, "p95": 143.195845, "samples": 1924 }, "throughput_mib_s": 7.8760662691913295, "exit_code": 0 }, "fixed": { "save_ack_ms": { "p50": 9.95182, "p95": 16.15159, "samples": 59 }, "upload_response_ms": { "p50": 116.10380400000001, "p95": 140.805797, "samples": 2201 }, "throughput_mib_s": 9.010913115334635, "exit_code": 0 }, "paired_ratio_pct": { "save_ack_p95": 143.23289424431601, "upload_response_p95": 98.33092363818238, "throughput": 114.40880266056756 } }, { "pair": 2, "order": "fixed/base", "base": { "save_ack_ms": { "p50": 11.694977, "p95": 20.984858, "samples": 60 }, "upload_response_ms": { "p50": 181.23751900000002, "p95": 305.759208, "samples": 1220 }, "throughput_mib_s": 4.994277119581212, "exit_code": 0 }, "fixed": { "save_ack_ms": { "p50": 12.195746, "p95": 20.453059999999997, "samples": 59 }, "upload_response_ms": { "p50": 122.14196, "p95": 146.29314, "samples": 1951 }, "throughput_mib_s": 7.987411446142995, "exit_code": 0 }, "paired_ratio_pct": { "save_ack_p95": 97.46580129348503, "upload_response_p95": 47.84586569180281, "throughput": 159.93128244378977 } }, { "pair": 3, "order": "base/fixed", "base": { "save_ack_ms": { "p50": 12.356477, "p95": 17.879756999999998, "samples": 60 }, "upload_response_ms": { "p50": 179.02521199999998, "p95": 211.695008, "samples": 1317 }, "throughput_mib_s": 5.391684547783172, "exit_code": 0 }, "fixed": { "save_ack_ms": { "p50": 12.034103, "p95": 17.673111, "samples": 60 }, "upload_response_ms": { "p50": 123.075991, "p95": 149.01044299999998, "samples": 1940 }, "throughput_mib_s": 7.942181138908136, "exit_code": 0 }, "paired_ratio_pct": { "save_ack_p95": 98.84424603757199, "upload_response_p95": 70.38920965013969, "throughput": 147.30426211922241 } }, { "pair": 4, "order": "fixed/base", "base": { "save_ack_ms": { "p50": 11.903978, "p95": 18.941137, "samples": 60 }, "upload_response_ms": { "p50": 179.25838, "p95": 220.128913, "samples": 1307 }, "throughput_mib_s": 5.350753287242168, "exit_code": 0 }, "fixed": { "save_ack_ms": { "p50": 12.555529, "p95": 18.334640999999998, "samples": 60 }, "upload_response_ms": { "p50": 122.91566, "p95": 146.669366, "samples": 1946 }, "throughput_mib_s": 7.967140604998429, "exit_code": 0 }, "paired_ratio_pct": { "save_ack_p95": 96.79799581197263, "upload_response_p95": 66.6288512495403, "throughput": 148.8975510045385 } }, { "pair": 5, "order": "base/fixed", "base": { "save_ack_ms": { "p50": 11.702833, "p95": 20.466426, "samples": 60 }, "upload_response_ms": { "p50": 179.29509800000002, "p95": 215.944592, "samples": 1311 }, "throughput_mib_s": 5.367088973256576, "exit_code": 0 }, "fixed": { "save_ack_ms": { "p50": 13.234993000000001, "p95": 22.926043, "samples": 59 }, "upload_response_ms": { "p50": 123.30149399999999, "p95": 146.776267, "samples": 1940 }, "throughput_mib_s": 8.074598087107052, "exit_code": 0 }, "paired_ratio_pct": { "save_ack_p95": 112.01781395540189, "upload_response_p95": 67.96941087554534, "throughput": 150.446510712634 } } ], "summary": { "base": { "save_ack_p50_ms": 11.702833, "save_ack_p95_ms": 18.941137, "upload_response_p95_ms": 215.944592, "throughput_mib_s": 5.367088973256576 }, "fixed": { "save_ack_p50_ms": 12.195746, "save_ack_p95_ms": 18.334640999999998, "upload_response_p95_ms": 146.669366, "throughput_mib_s": 7.987411446142995 }, "median_paired_ratio_pct": { "save_ack_p95": 98.84424603757199, "upload_response_p95": 67.96941087554534, "throughput": 148.8975510045385 } } } ```
kayg commented 2026-11-02 16:32:05 +00:00
Author
Owner
Copy link

First-run clippy FAIL (101): CI=1 cargo clippy --workspace --all-targets --locked -j4 -- -D warnings. No retry or source fix. First error:

   Compiling asn1-rs-derive v0.5.1
   Compiling pulldown-cmark v0.13.4 (/home/claude/Developer/Worktrees/bm9/vendor/pulldown-cmark)
    Checking idna_adapter v1.2.2
    Checking num-integer v0.1.47
   Compiling openssl v0.10.81
    Checking base64 v0.21.7
    Checking zeroize v1.9.1
    Checking foreign-types-shared v0.1.1
   Compiling pastey v0.1.1
error: this `if` statement can be collapsed
  --> crates/notes/src/room/edit.rs:43:9
   |
43 | /         if let Some(caller) = &e.caller {
44 | |             if !caller.credential.has_scope("notes.write") {
45 | |                 return Err(Error::new(
46 | |                     ErrorCode::Forbidden,
...  |
50 | |         }
   | |_________^
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/rust-1.99.0/index.html#collapsible_if
   = note: `-D clippy::collapsible-if` implied by `-D warnings`
   = help: to override `-D warnings` add `#[allow(clippy::collapsible_if)]`
help: collapse nested if block
   |
43 ~         if let Some(caller) = &e.caller
44 ~             && !caller.credential.has_scope("notes.write") {
45 |                 return Err(Error::new(
...
48 |                 ));
49 ~             }
   |

error: could not compile `calternal-notes` (lib) due to 1 previous error
warning: build failed, waiting for other jobs to finish...

The same baseline command will run once on origin/dev 55f35fcde3bc91ca3781055577b93e93819f6cd4, with only the #779 Nix tooling overlay.

First-run clippy FAIL (101): `CI=1 cargo clippy --workspace --all-targets --locked -j4 -- -D warnings`. No retry or source fix. First error: ```text Compiling asn1-rs-derive v0.5.1 Compiling pulldown-cmark v0.13.4 (/home/claude/Developer/Worktrees/bm9/vendor/pulldown-cmark) Checking idna_adapter v1.2.2 Checking num-integer v0.1.47 Compiling openssl v0.10.81 Checking base64 v0.21.7 Checking zeroize v1.9.1 Checking foreign-types-shared v0.1.1 Compiling pastey v0.1.1 error: this `if` statement can be collapsed --> crates/notes/src/room/edit.rs:43:9 | 43 | / if let Some(caller) = &e.caller { 44 | | if !caller.credential.has_scope("notes.write") { 45 | | return Err(Error::new( 46 | | ErrorCode::Forbidden, ... | 50 | | } | |_________^ | = help: for further information visit https://rust-lang.github.io/rust-clippy/rust-1.99.0/index.html#collapsible_if = note: `-D clippy::collapsible-if` implied by `-D warnings` = help: to override `-D warnings` add `#[allow(clippy::collapsible_if)]` help: collapse nested if block | 43 ~ if let Some(caller) = &e.caller 44 ~ && !caller.credential.has_scope("notes.write") { 45 | return Err(Error::new( ... 48 | )); 49 ~ } | error: could not compile `calternal-notes` (lib) due to 1 previous error warning: build failed, waiting for other jobs to finish... ``` The same baseline command will run once on origin/dev `55f35fcde3bc91ca3781055577b93e93819f6cd4`, with only the #779 Nix tooling overlay.
kayg commented 2026-11-02 16:34:36 +00:00
Author
Owner
Copy link

Clippy attribution: the identical CI=1 cargo clippy --workspace --all-targets --locked -j4 -- -D warnings command passed (0) on origin/dev 55f35fcde3bc91ca3781055577b93e93819f6cd4, once, using the #779 tooling overlay. The train collapsible_if failure at crates/notes/src/room/edit.rs:43 is new, so the verdict cannot be green. Remaining gates and full browser halves continue; no retries.

Clippy attribution: the identical `CI=1 cargo clippy --workspace --all-targets --locked -j4 -- -D warnings` command passed (0) on origin/dev `55f35fcde3bc91ca3781055577b93e93819f6cd4`, once, using the #779 tooling overlay. The train collapsible_if failure at `crates/notes/src/room/edit.rs:43` is new, so the verdict cannot be green. Remaining gates and full browser halves continue; no retries.
kayg commented 2026-11-02 16:39:10 +00:00
Author
Owner
Copy link

First-run gates complete, CI=1, Nix with strace, all heavy commands through ~/bin/slot. No retries.

Gate Exact command Exit
fmt cargo fmt --all --check 0
clippy cargo clippy --workspace --all-targets --locked -j4 -- -D warnings 101
test cargo test --workspace -j4 0
build cargo build -j4 0
web bun run build 0
bundle env PERF_DIST=/home/claude/Developer/Worktrees/bm9/apps/web/dist PERF_OUT=/home/claude/bm9-train/bundle PERF_WORK=/home/claude/bm9-train/bundle-work bash perf/web.sh --bundle-only --gate 0
case-paths node ci/check-case-paths.mjs 0
no-evidence node ci/check-no-evidence.mjs 0

Head 383eb8f88b45b4cdd1604274dea26d642fd04ba6. Full workspace tests and per-app bundle budgets PASS. Only clippy fails, confirmed new by one identical baseline run (see preceding comments). Both full browser halves are now released together: non-WebKit native projects on dev-01, WebKit plus admin-setup on dev-02. Workers=1, --retries=0, CI=1, disk >50 GB on each VM.

First-run gates complete, CI=1, Nix with strace, all heavy commands through ~/bin/slot. No retries. | Gate | Exact command | Exit | |---|---|---:| | fmt | `cargo fmt --all --check` | 0 | | clippy | `cargo clippy --workspace --all-targets --locked -j4 -- -D warnings` | 101 | | test | `cargo test --workspace -j4` | 0 | | build | `cargo build -j4` | 0 | | web | `bun run build` | 0 | | bundle | `env PERF_DIST=/home/claude/Developer/Worktrees/bm9/apps/web/dist PERF_OUT=/home/claude/bm9-train/bundle PERF_WORK=/home/claude/bm9-train/bundle-work bash perf/web.sh --bundle-only --gate` | 0 | | case-paths | `node ci/check-case-paths.mjs` | 0 | | no-evidence | `node ci/check-no-evidence.mjs` | 0 | Head `383eb8f88b45b4cdd1604274dea26d642fd04ba6`. Full workspace tests and per-app bundle budgets PASS. Only clippy fails, confirmed new by one identical baseline run (see preceding comments). Both full browser halves are now released together: non-WebKit native projects on dev-01, WebKit plus admin-setup on dev-02. Workers=1, --retries=0, CI=1, disk >50 GB on each VM.
kayg commented 2026-11-02 17:18:00 +00:00
Author
Owner
Copy link

Early single-case baseline attribution: Chromium-1440 search-corpus.spec.ts:40, “every prefix, the full word, one-letter and two-letter typos find their note”, PASS on origin/dev 55f35fcde3bc91ca3781055577b93e93819f6cd4: 1 test passed, 1492 queries, 0 missed. The train failed with miss ratio 0.7412868632707775 against 0.02. Classification: new/unresolved, not reproduced on baseline. The exact failed train case ran on baseline once, CI=1, --workers=1, --retries=0; it will not run again. Full first-error catalog and per-project tables follow at suite completion.

Baseline selection: bunx playwright test e2e/search-corpus.spec.ts --config=playwright.bm9-baseline.config.ts --project=chromium-1440 --grep "every prefix, the full word, one-letter and two-letter typos find their note$" --workers=1 --retries=0 --reporter=line,json. Baseline config imports native config and retains only admin-setup dependencies, removing scheduling-only dependencies for exact-case isolation.

Early single-case baseline attribution: Chromium-1440 `search-corpus.spec.ts:40`, “every prefix, the full word, one-letter and two-letter typos find their note”, PASS on origin/dev `55f35fcde3bc91ca3781055577b93e93819f6cd4`: 1 test passed, 1492 queries, 0 missed. The train failed with miss ratio `0.7412868632707775` against `0.02`. Classification: new/unresolved, not reproduced on baseline. The exact failed train case ran on baseline once, CI=1, --workers=1, --retries=0; it will not run again. Full first-error catalog and per-project tables follow at suite completion. Baseline selection: `bunx playwright test e2e/search-corpus.spec.ts --config=playwright.bm9-baseline.config.ts --project=chromium-1440 --grep "every prefix, the full word, one-letter and two-letter typos find their note$" --workers=1 --retries=0 --reporter=line,json`. Baseline config imports native config and retains only admin-setup dependencies, removing scheduling-only dependencies for exact-case isolation.
kayg commented 2026-11-02 17:55:02 +00:00
Author
Owner
Copy link

Full chromium half first-run results. CI=1, --workers=1, --retries=0.

Project Passed Failed Skipped Not run
admin-chromium-1440 12 0 9 0
admin-chromium-390 12 0 9 0
admin-setup 1 0 0 0
chromium-1440 416 1 597 0
chromium-390 859 0 155 0
contrast 9 0 0 0
flow 1 0 0 0
look-chromium-1440 5 0 0 0
look-chromium-390 5 0 0 0
passkeys 1 0 0 0
prefs-chromium 7 0 0 0
sidebar-band-chromium 3 0 0 0
throttled-1440 0 0 2 2
throttled-390 0 0 2 2

Native project selection preserved, admin-setup ran with dependents. Cross-half scheduling dependencies only were removed. Each failed case receives exactly one origin/dev run.

Runner stats: {"startTime": "2026-11-02T16:39:12.206Z", "duration": 4444747.241, "expected": 1331, "skipped": 774, "unexpected": 1, "flaky": 0}

Full chromium half first-run results. CI=1, --workers=1, --retries=0. | Project | Passed | Failed | Skipped | Not run | |---|---:|---:|---:|---:| | admin-chromium-1440 | 12 | 0 | 9 | 0 | | admin-chromium-390 | 12 | 0 | 9 | 0 | | admin-setup | 1 | 0 | 0 | 0 | | chromium-1440 | 416 | 1 | 597 | 0 | | chromium-390 | 859 | 0 | 155 | 0 | | contrast | 9 | 0 | 0 | 0 | | flow | 1 | 0 | 0 | 0 | | look-chromium-1440 | 5 | 0 | 0 | 0 | | look-chromium-390 | 5 | 0 | 0 | 0 | | passkeys | 1 | 0 | 0 | 0 | | prefs-chromium | 7 | 0 | 0 | 0 | | sidebar-band-chromium | 3 | 0 | 0 | 0 | | throttled-1440 | 0 | 0 | 2 | 2 | | throttled-390 | 0 | 0 | 2 | 2 | Native project selection preserved, admin-setup ran with dependents. Cross-half scheduling dependencies only were removed. Each failed case receives exactly one origin/dev run. Runner stats: `{"startTime": "2026-11-02T16:39:12.206Z", "duration": 4444747.241, "expected": 1331, "skipped": 774, "unexpected": 1, "flaky": 0}`
kayg commented 2026-11-02 17:55:03 +00:00
Author
Owner
Copy link

Every first-attempt failure, chromium. No retries.

1. chromium-1440 · search-corpus.spec.ts:40

every prefix, the full word, one-letter and two-letter typos find their note

First attempt errors:

Error: missed: prefix "ma" -> marmalade (rank 5); 1 typo "mrmalade" -> marmalade (rank -1); 1 typo "amrmalade" -> marmalade (rank -1); 1 typo "mermalade" -> marmalade (rank -1); 1 typo "mxarmalade" -> marmalade (rank -1); 1 typo "mamalade" -> marmalade (rank -1); 1 typo "mramalade" -> marmalade (rank -1); 1 typo "maemalade" -> marmalade (rank -1); 1 typo "maxrmalade" -> marmalade (rank -1); 1 typo "maralade" -> marmalade (rank -1); 1 typo "mamralade" -> marmalade (rank -1); 1 typo "marealade" -> marmalade (rank -1)

expect(received).toBeLessThanOrEqual(expected)

Expected: <= 0.02
Received:    0.7412868632707775

  103 |   console.log(`search corpus: ${asked} queries, ${misses.length} missed, p50 ${p(0.5)} ms, p95 ${p(0.95)} ms (round trip through the action route, tunnel included)`)
  104 |   if (misses.length) console.log(misses.slice(0, 40).join('\n'))
> 105 |   expect(misses.length / asked, `missed: ${misses.slice(0, 12).join('; ')}`).toBeLessThanOrEqual(0.02)
      |                                                                              ^
  106 |   } finally { await clearCorpus() }
  107 |   // Latency is not asserted here: this goes through whatever sits between the test and the server (the Studio's
  108 |   // tunnel adds ~150 ms). The server's own numbers come from the perf gate (perf/budgets.json, search.query).
    at /home/claude/Developer/Worktrees/bm9/apps/web/e2e/search-corpus.spec.ts:105:78
Every first-attempt failure, chromium. No retries. ### 1. chromium-1440 · search-corpus.spec.ts:40 every prefix, the full word, one-letter and two-letter typos find their note First attempt errors: ```text Error: missed: prefix "ma" -> marmalade (rank 5); 1 typo "mrmalade" -> marmalade (rank -1); 1 typo "amrmalade" -> marmalade (rank -1); 1 typo "mermalade" -> marmalade (rank -1); 1 typo "mxarmalade" -> marmalade (rank -1); 1 typo "mamalade" -> marmalade (rank -1); 1 typo "mramalade" -> marmalade (rank -1); 1 typo "maemalade" -> marmalade (rank -1); 1 typo "maxrmalade" -> marmalade (rank -1); 1 typo "maralade" -> marmalade (rank -1); 1 typo "mamralade" -> marmalade (rank -1); 1 typo "marealade" -> marmalade (rank -1) expect(received).toBeLessThanOrEqual(expected) Expected: <= 0.02 Received: 0.7412868632707775 103 | console.log(`search corpus: ${asked} queries, ${misses.length} missed, p50 ${p(0.5)} ms, p95 ${p(0.95)} ms (round trip through the action route, tunnel included)`) 104 | if (misses.length) console.log(misses.slice(0, 40).join('\n')) > 105 | expect(misses.length / asked, `missed: ${misses.slice(0, 12).join('; ')}`).toBeLessThanOrEqual(0.02) | ^ 106 | } finally { await clearCorpus() } 107 | // Latency is not asserted here: this goes through whatever sits between the test and the server (the Studio's 108 | // tunnel adds ~150 ms). The server's own numbers come from the perf gate (perf/budgets.json, search.query). at /home/claude/Developer/Worktrees/bm9/apps/web/e2e/search-corpus.spec.ts:105:78 ```
kayg commented 2026-11-02 17:55:36 +00:00
Author
Owner
Copy link

The complete Chromium invocation finished with 1331 passed, 1 failed, 770 explicit skips, and 4 cases with zero attempts. The four are the two tests in each throttled project, blocked by native scheduling dependencies after the main project failed. To satisfy the FULL suite, those four are now receiving their FIRST attempts in one separate CI=1 invocation, --workers=1, --retries=0, Nix + slot, same train head. Only their already-completed scheduling dependencies are removed. No attempted test is rerun. The initial per-project table counts these four under both skipped and not-run; the final consolidated table will separate those categories and include their first attempts. Search-corpus baseline PASS is reused, not rerun.

The complete Chromium invocation finished with 1331 passed, 1 failed, 770 explicit skips, and 4 cases with zero attempts. The four are the two tests in each throttled project, blocked by native scheduling dependencies after the main project failed. To satisfy the FULL suite, those four are now receiving their FIRST attempts in one separate CI=1 invocation, --workers=1, --retries=0, Nix + slot, same train head. Only their already-completed scheduling dependencies are removed. No attempted test is rerun. The initial per-project table counts these four under both skipped and not-run; the final consolidated table will separate those categories and include their first attempts. Search-corpus baseline PASS is reused, not rerun.
kayg commented 2026-11-02 17:55:42 +00:00
Author
Owner
Copy link

Single-run origin/dev attribution (55f35fcde3bc91ca3781055577b93e93819f6cd4, #779 tooling overlay only). CI=1, Nix, slot, --workers=1, --retries=0. Each exact failed case selected once; scheduling dependencies removed, admin-setup retained when needed. The early search-corpus comparison is reused, not rerun.

Project Failed train case Baseline result Attribution
chromium-1440 search-corpus.spec.ts:40 every prefix, the full word, one-letter and two-letter typos find their note PASS (exit 0) new/unresolved; not reproduced

Exact commands, from apps/web, with CALTERNAL_BIN/E2E_DIST pointing to the baseline build and TMPDIR outside the repo:

bunx playwright test e2e/search-corpus.spec.ts --config=playwright.bm9-baseline.config.ts --project=chromium-1440 --grep 'every\ prefix,\ the\ full\ word,\ one\-letter\ and\ two\-letter\ typos\ find\ their\ note$' --workers=1 --retries=0 --reporter=line,json --output="$out/case-4c2f286b5ab8-results"
Single-run origin/dev attribution (`55f35fcde3bc91ca3781055577b93e93819f6cd4`, #779 tooling overlay only). CI=1, Nix, slot, --workers=1, --retries=0. Each exact failed case selected once; scheduling dependencies removed, admin-setup retained when needed. The early search-corpus comparison is reused, not rerun. | Project | Failed train case | Baseline result | Attribution | |---|---|---|---| | chromium-1440 | `search-corpus.spec.ts:40` every prefix, the full word, one-letter and two-letter typos find their note | PASS (exit 0) | new/unresolved; not reproduced | Exact commands, from apps/web, with CALTERNAL_BIN/E2E_DIST pointing to the baseline build and TMPDIR outside the repo: ```sh bunx playwright test e2e/search-corpus.spec.ts --config=playwright.bm9-baseline.config.ts --project=chromium-1440 --grep 'every\ prefix,\ the\ full\ word,\ one\-letter\ and\ two\-letter\ typos\ find\ their\ note$' --workers=1 --retries=0 --reporter=line,json --output="$out/case-4c2f286b5ab8-results" ```
kayg commented 2026-11-02 17:57:23 +00:00
Author
Owner
Copy link

The four dependency-blocked cases completed their FIRST attempts: all 4 PASS, 0 retries. No attempted case was rerun.

Project Passed Failed Skipped
throttled-1440 2 0 0
throttled-390 2 0 0

Consolidated Chromium half: 1335 passed, 1 failed, 770 skipped, 0 unattempted. The first invocation’s unattempted records are replaced by these actual first attempts in the final table.

Command in apps/web: CI=1 bunx playwright test --config=playwright.bm9-unrun.config.ts --output="$HOME/bm9-train/unrun-results" --workers=1 --retries=0 --reporter=line,json, CALTERNAL_BIN/E2E_DIST at the train build. The config imports native config, selects both throttled projects, and removes their already-completed scheduling dependencies. Nix + slot; >50 GB free.

The four dependency-blocked cases completed their FIRST attempts: all 4 PASS, 0 retries. No attempted case was rerun. | Project | Passed | Failed | Skipped | |---|---:|---:|---:| | throttled-1440 | 2 | 0 | 0 | | throttled-390 | 2 | 0 | 0 | Consolidated Chromium half: **1335 passed, 1 failed, 770 skipped, 0 unattempted**. The first invocation’s unattempted records are replaced by these actual first attempts in the final table. Command in apps/web: `CI=1 bunx playwright test --config=playwright.bm9-unrun.config.ts --output="$HOME/bm9-train/unrun-results" --workers=1 --retries=0 --reporter=line,json`, CALTERNAL_BIN/E2E_DIST at the train build. The config imports native config, selects both throttled projects, and removes their already-completed scheduling dependencies. Nix + slot; >50 GB free.
kayg commented 2026-11-02 18:17:10 +00:00
Author
Owner
Copy link

Full webkit half first-run results. CI=1, --workers=1, --retries=0.

Project Passed Failed Skipped Not run
admin-setup 1 0 0 0
admin-webkit-1440 9 0 9 0
admin-webkit-390 9 0 9 0
look-webkit-1440 4 0 1 0
look-webkit-390 4 0 1 0
prefs-webkit 3 0 4 0
sidebar-band-webkit 3 0 0 0
webkit-1440 412 0 602 0
webkit-390 858 0 156 0

Native project selection preserved, admin-setup ran with dependents. Cross-half scheduling dependencies only were removed. Each failed case receives exactly one origin/dev run.

Runner stats: {"startTime": "2026-11-02T16:39:08.540Z", "duration": 5799901.539, "expected": 1303, "skipped": 782, "unexpected": 0, "flaky": 0}

Full webkit half first-run results. CI=1, --workers=1, --retries=0. | Project | Passed | Failed | Skipped | Not run | |---|---:|---:|---:|---:| | admin-setup | 1 | 0 | 0 | 0 | | admin-webkit-1440 | 9 | 0 | 9 | 0 | | admin-webkit-390 | 9 | 0 | 9 | 0 | | look-webkit-1440 | 4 | 0 | 1 | 0 | | look-webkit-390 | 4 | 0 | 1 | 0 | | prefs-webkit | 3 | 0 | 4 | 0 | | sidebar-band-webkit | 3 | 0 | 0 | 0 | | webkit-1440 | 412 | 0 | 602 | 0 | | webkit-390 | 858 | 0 | 156 | 0 | Native project selection preserved, admin-setup ran with dependents. Cross-half scheduling dependencies only were removed. Each failed case receives exactly one origin/dev run. Runner stats: `{"startTime": "2026-11-02T16:39:08.540Z", "duration": 5799901.539, "expected": 1303, "skipped": 782, "unexpected": 0, "flaky": 0}`
kayg commented 2026-11-02 18:17:39 +00:00
Author
Owner
Copy link

Final Train BM9 verdict: RED. Do not merge. Head 383eb8f88b45b4cdd1604274dea26d642fd04ba6, origin/dev base 55f35fcde3bc91ca3781055577b93e93819f6cd4. BM8 #862 closed with a link to this replacement.

PR Captured current source head Assembly decision
#779 7b97e4d14231be07f6a80cb9ca172d02442c487e included, --no-ff
#858 626388da9052eaecb159d735f0cadc5e59e88312 included, --no-ff
#866 ba1f47dc94797d4a8a937298464b5e4828e301c2 included, --no-ff
#824 f1c1aa961e169ef99e132063a97ccbd9a89a5def included, --no-ff
#827 0543321b34f9a739275f70e31c79c5575d680259 included, --no-ff
#829 d455842942731aabc05474c7929597d57090a7cc authorization/admission conflicts with #827 perf cache require implementation choices
#835 19071be8455d718822ba98f6b9a402c5fd345242 depends on excluded #829
#828 5d7b9b36cd008de78851809724c90a85d92b40a7 included, --no-ff
#833 ccaece3dfecbe2048634fc792d7d4a2c6e31f324 included, --no-ff
#836 5f7d1a59776411e7f3353d0023817b43affae51e included, --no-ff
#831 316b43e9578c3e4d2944df8df43caebc611356f7 included, --no-ff
#844 50cce9dcceb87f23a9f346300b3336e315ff6c0c included, --no-ff
#847 40448327a15730271d56226b898e48bcc87b2a4d included, --no-ff
#855 c38c3083d51f4adcf5cd76a70f1f6960b427413e included, --no-ff
#857 144e717145e35f70356f593b7c04a527e00f7b17 included, --no-ff
#856 85845e3d11bea351608bde7481779ba2303372e5 included, --no-ff

#840 intentionally omitted. #829 conflicted with #827’s 0543321b cache path in sharing authorization and admission; resolving resource authorization, cleanup, locking, and cache interactions needs implementation choices. Its dependent #835 was also excluded. Mechanical resolutions: retain both Cargo.lock packages for #824; retain credential checks then start runtime timing for #828. #831’s inherited measurements were posted in the artifact comment and removed from the train.

First-run gates, CI=1, Nix with strace, heavy work through ~/bin/slot:

Exact command Result
cargo fmt --all --check PASS
cargo clippy --workspace --all-targets --locked -j4 -- -D warnings FAIL (101), new
cargo test --workspace -j4 PASS
cargo build -j4 PASS on both VMs
bun run build (apps/web) PASS on both VMs
PERF_DIST="$PWD/apps/web/dist" PERF_OUT="$HOME/bm9-train/bundle" PERF_WORK="$HOME/bm9-train/bundle-work" bash perf/web.sh --bundle-only --gate PASS, #866 per-app budgets
node ci/check-case-paths.mjs PASS
node ci/check-no-evidence.mjs PASS

Clippy’s first error: clippy::collapsible_if at crates/notes/src/room/edit.rs:43. The identical baseline command passed once. Full first error, baseline attribution.

Consolidated full Playwright project matrix, each executed attempt counted once:

VM Project Passed Failed Skipped Unattempted
dev-01 admin-chromium-1440 12 0 9 0
dev-01 admin-chromium-390 12 0 9 0
dev-01 admin-setup 1 0 0 0
dev-01 chromium-1440 416 1 597 0
dev-01 chromium-390 859 0 155 0
dev-01 contrast 9 0 0 0
dev-01 flow 1 0 0 0
dev-01 look-chromium-1440 5 0 0 0
dev-01 look-chromium-390 5 0 0 0
dev-01 passkeys 1 0 0 0
dev-01 prefs-chromium 7 0 0 0
dev-01 sidebar-band-chromium 3 0 0 0
dev-01 throttled-1440 2 0 0 0
dev-01 throttled-390 2 0 0 0
dev-02 admin-setup 1 0 0 0
dev-02 admin-webkit-1440 9 0 9 0
dev-02 admin-webkit-390 9 0 9 0
dev-02 look-webkit-1440 4 0 1 0
dev-02 look-webkit-390 4 0 1 0
dev-02 prefs-webkit 3 0 4 0
dev-02 sidebar-band-webkit 3 0 0 0
dev-02 webkit-1440 412 0 602 0
dev-02 webkit-390 858 0 156 0

Totals: 2638 passed, 1 failed, 1552 skipped, 0 unattempted. Admin-setup appears on each VM because each half provisions its own admin session in the same invocation as its dependents.

The first Chromium invocation dependency-blocked four throttled cases without attempting them. Those four then passed their first attempts in a dedicated invocation with completed scheduling dependencies removed. No attempted case was rerun. Details, four first-attempt passes.

Browser command in apps/web, BM9_HALF=chromium on dev-01 and webkit on dev-02, released together:

export CI=1 CALTERNAL_BIN="$PWD/../../target/debug/calternal" E2E_DIST="$PWD/dist"
export TMPDIR="$HOME/bm9-train/browser-tmp"
export PLAYWRIGHT_JSON_OUTPUT_NAME="$HOME/bm9-train/playwright.json"
bunx playwright test --config=playwright.bm9.config.ts --output="$HOME/bm9-train/test-results" --workers=1 --retries=0 --reporter=line,json

The split config imports the native config, selects every non-WebKit project on dev-01 and every WebKit project plus admin-setup on dev-02, retaining same-half dependencies. Only cross-half scheduling dependencies are removed. Preparation: bun install --frozen-lockfile at root, Rust debug symbols/incremental disabled (CARGO_PROFILE_DEV_DEBUG=0 CARGO_PROFILE_TEST_DEBUG=0 CARGO_INCREMENTAL=0).

Failure attribution: the exact Chromium-1440 search-corpus case passed once on baseline (1492 queries, 0 misses), while the train missed 1106/1492 queries (ratio 0.7412868632707775, limit 0.02). New/unresolved, not established as pre-existing. Every first Chromium error, exact baseline result and command.

WebKit had no failures requiring baseline attribution. Complete WebKit project table.

All own build/test/baseline jobs finished before handback. Both VMs passed df -h / checks with >50 GB free before heavy work. Own throwaway targets, built dist copies, test-result traces and temporary test Homes were removed on each VM after its jobs finished; only small logs/JSON remain outside the repository. No build/test ran on the control host. No source PR branch or repository settings changed. The train was never merged.

The verdict is RED because every failure must be pre-existing for green, and both the clippy and search-corpus failures passed their exact baseline comparisons.

Final Train BM9 verdict: **RED. Do not merge.** Head `383eb8f88b45b4cdd1604274dea26d642fd04ba6`, origin/dev base `55f35fcde3bc91ca3781055577b93e93819f6cd4`. BM8 #862 closed with a link to this replacement. | PR | Captured current source head | Assembly decision | |---|---|---| | #779 | `7b97e4d14231be07f6a80cb9ca172d02442c487e` | included, --no-ff | | #858 | `626388da9052eaecb159d735f0cadc5e59e88312` | included, --no-ff | | #866 | `ba1f47dc94797d4a8a937298464b5e4828e301c2` | included, --no-ff | | #824 | `f1c1aa961e169ef99e132063a97ccbd9a89a5def` | included, --no-ff | | #827 | `0543321b34f9a739275f70e31c79c5575d680259` | included, --no-ff | | #829 | `d455842942731aabc05474c7929597d57090a7cc` | authorization/admission conflicts with #827 perf cache require implementation choices | | #835 | `19071be8455d718822ba98f6b9a402c5fd345242` | depends on excluded #829 | | #828 | `5d7b9b36cd008de78851809724c90a85d92b40a7` | included, --no-ff | | #833 | `ccaece3dfecbe2048634fc792d7d4a2c6e31f324` | included, --no-ff | | #836 | `5f7d1a59776411e7f3353d0023817b43affae51e` | included, --no-ff | | #831 | `316b43e9578c3e4d2944df8df43caebc611356f7` | included, --no-ff | | #844 | `50cce9dcceb87f23a9f346300b3336e315ff6c0c` | included, --no-ff | | #847 | `40448327a15730271d56226b898e48bcc87b2a4d` | included, --no-ff | | #855 | `c38c3083d51f4adcf5cd76a70f1f6960b427413e` | included, --no-ff | | #857 | `144e717145e35f70356f593b7c04a527e00f7b17` | included, --no-ff | | #856 | `85845e3d11bea351608bde7481779ba2303372e5` | included, --no-ff | #840 intentionally omitted. #829 conflicted with #827’s 0543321b cache path in sharing authorization and admission; resolving resource authorization, cleanup, locking, and cache interactions needs implementation choices. Its dependent #835 was also excluded. Mechanical resolutions: retain both Cargo.lock packages for #824; retain credential checks then start runtime timing for #828. #831’s inherited measurements were posted in [the artifact comment](https://git.kayg.org/kayg/calternal/pulls/869#issuecomment-75319) and removed from the train. First-run gates, CI=1, Nix with strace, heavy work through ~/bin/slot: | Exact command | Result | |---|---| | `cargo fmt --all --check` | PASS | | `cargo clippy --workspace --all-targets --locked -j4 -- -D warnings` | FAIL (101), new | | `cargo test --workspace -j4` | PASS | | `cargo build -j4` | PASS on both VMs | | `bun run build (apps/web)` | PASS on both VMs | | `PERF_DIST="$PWD/apps/web/dist" PERF_OUT="$HOME/bm9-train/bundle" PERF_WORK="$HOME/bm9-train/bundle-work" bash perf/web.sh --bundle-only --gate` | PASS, #866 per-app budgets | | `node ci/check-case-paths.mjs` | PASS | | `node ci/check-no-evidence.mjs` | PASS | Clippy’s first error: `clippy::collapsible_if` at `crates/notes/src/room/edit.rs:43`. The identical baseline command passed once. [Full first error](https://git.kayg.org/kayg/calternal/pulls/869#issuecomment-75333), [baseline attribution](https://git.kayg.org/kayg/calternal/pulls/869#issuecomment-75337). Consolidated full Playwright project matrix, each executed attempt counted once: | VM | Project | Passed | Failed | Skipped | Unattempted | |---|---|---:|---:|---:|---:| | dev-01 | admin-chromium-1440 | 12 | 0 | 9 | 0 | | dev-01 | admin-chromium-390 | 12 | 0 | 9 | 0 | | dev-01 | admin-setup | 1 | 0 | 0 | 0 | | dev-01 | chromium-1440 | 416 | 1 | 597 | 0 | | dev-01 | chromium-390 | 859 | 0 | 155 | 0 | | dev-01 | contrast | 9 | 0 | 0 | 0 | | dev-01 | flow | 1 | 0 | 0 | 0 | | dev-01 | look-chromium-1440 | 5 | 0 | 0 | 0 | | dev-01 | look-chromium-390 | 5 | 0 | 0 | 0 | | dev-01 | passkeys | 1 | 0 | 0 | 0 | | dev-01 | prefs-chromium | 7 | 0 | 0 | 0 | | dev-01 | sidebar-band-chromium | 3 | 0 | 0 | 0 | | dev-01 | throttled-1440 | 2 | 0 | 0 | 0 | | dev-01 | throttled-390 | 2 | 0 | 0 | 0 | | dev-02 | admin-setup | 1 | 0 | 0 | 0 | | dev-02 | admin-webkit-1440 | 9 | 0 | 9 | 0 | | dev-02 | admin-webkit-390 | 9 | 0 | 9 | 0 | | dev-02 | look-webkit-1440 | 4 | 0 | 1 | 0 | | dev-02 | look-webkit-390 | 4 | 0 | 1 | 0 | | dev-02 | prefs-webkit | 3 | 0 | 4 | 0 | | dev-02 | sidebar-band-webkit | 3 | 0 | 0 | 0 | | dev-02 | webkit-1440 | 412 | 0 | 602 | 0 | | dev-02 | webkit-390 | 858 | 0 | 156 | 0 | Totals: **2638 passed, 1 failed, 1552 skipped**, 0 unattempted. Admin-setup appears on each VM because each half provisions its own admin session in the same invocation as its dependents. The first Chromium invocation dependency-blocked four throttled cases without attempting them. Those four then passed their first attempts in a dedicated invocation with completed scheduling dependencies removed. No attempted case was rerun. [Details](https://git.kayg.org/kayg/calternal/pulls/869#issuecomment-75468), [four first-attempt passes](https://git.kayg.org/kayg/calternal/pulls/869#issuecomment-75471). Browser command in apps/web, BM9_HALF=chromium on dev-01 and webkit on dev-02, released together: ```sh export CI=1 CALTERNAL_BIN="$PWD/../../target/debug/calternal" E2E_DIST="$PWD/dist" export TMPDIR="$HOME/bm9-train/browser-tmp" export PLAYWRIGHT_JSON_OUTPUT_NAME="$HOME/bm9-train/playwright.json" bunx playwright test --config=playwright.bm9.config.ts --output="$HOME/bm9-train/test-results" --workers=1 --retries=0 --reporter=line,json ``` The split config imports the native config, selects every non-WebKit project on dev-01 and every WebKit project plus admin-setup on dev-02, retaining same-half dependencies. Only cross-half scheduling dependencies are removed. Preparation: `bun install --frozen-lockfile` at root, Rust debug symbols/incremental disabled (`CARGO_PROFILE_DEV_DEBUG=0 CARGO_PROFILE_TEST_DEBUG=0 CARGO_INCREMENTAL=0`). Failure attribution: the exact Chromium-1440 search-corpus case passed once on baseline (1492 queries, 0 misses), while the train missed 1106/1492 queries (ratio 0.7412868632707775, limit 0.02). **New/unresolved, not established as pre-existing.** [Every first Chromium error](https://git.kayg.org/kayg/calternal/pulls/869#issuecomment-75467), [exact baseline result and command](https://git.kayg.org/kayg/calternal/pulls/869#issuecomment-75469). WebKit had no failures requiring baseline attribution. [Complete WebKit project table](https://git.kayg.org/kayg/calternal/pulls/869#issuecomment-75506). All own build/test/baseline jobs finished before handback. Both VMs passed `df -h /` checks with >50 GB free before heavy work. Own throwaway targets, built dist copies, test-result traces and temporary test Homes were removed on each VM after its jobs finished; only small logs/JSON remain outside the repository. No build/test ran on the control host. No source PR branch or repository settings changed. The train was never merged. The verdict is RED because every failure must be pre-existing for green, and both the clippy and search-corpus failures passed their exact baseline comparisons.
kayg commented 2026-11-02 18:37:47 +00:00
Author
Owner
Copy link

BM9 source-fix investigation update. #827 clippy repair is pushed at 76f11dc4ba06468a2a0ed4d4592cea6038206696; the requested workspace clippy command passes on dev-01: #827 (comment).

Executed every cumulative BM9 merge in order, rebuilding each server and web app on dev-01. No first bad merge appeared in isolated search-corpus runs. Each row below passed both tests and reported 1492 queries / 0 misses:

Added PR Cumulative merge
#779 b71504bb
#858 2ef9605d
#866 ac68890b
#824 19cf5805
#827 92429527
#828 e621bbfc
#833 70982e1a
#836 af5c14d1
#831 538bb9d5
#844 888d2479
#847 1b42557e
#855 88916c09
#857 4488b5d6
#856 e4af9a94

Exact gates per merge, CI=1, CARGO_PROFILE_DEV_DEBUG=0, CARGO_PROFILE_TEST_DEBUG=0, CARGO_INCREMENTAL=0, all heavy commands through Nix and ~/bin/slot:

  • cargo build --locked -j4
  • bun install --frozen-lockfile
  • from apps/web: bun run build
  • from apps/web, CALTERNAL_BIN and E2E_DIST set to that merge's outputs: bunx playwright test e2e/search-corpus.spec.ts --project=chromium-1440 --no-deps --retries=0 --workers=1 --reporter=line

One runner-only error between #779 and #858 removed an exported TMPDIR; fixed the runner and resumed #858 before its test ran. No failed browser attempt was rerun in this sequence.

This does not invalidate the full-train failure or establish a flake. Investigation continues into shared search state: crates/search/src/index.rs:229-275 permits a stale typo vocabulary during its five-second background-refresh interval; the corpus setup only checks exact endpoint matches. A temporary existing-corpus probe is running to test that hypothesis. No search source fix or culprit claim yet. No train branch changed.

BM9 source-fix investigation update. #827 clippy repair is pushed at `76f11dc4ba06468a2a0ed4d4592cea6038206696`; the requested workspace clippy command passes on dev-01: https://git.kayg.org/kayg/calternal/pulls/827#issuecomment-75535. Executed every cumulative BM9 merge in order, rebuilding each server and web app on dev-01. **No first bad merge appeared in isolated search-corpus runs.** Each row below passed both tests and reported 1492 queries / 0 misses: | Added PR | Cumulative merge | |---|---| | #779 | b71504bb | | #858 | 2ef9605d | | #866 | ac68890b | | #824 | 19cf5805 | | #827 | 92429527 | | #828 | e621bbfc | | #833 | 70982e1a | | #836 | af5c14d1 | | #831 | 538bb9d5 | | #844 | 888d2479 | | #847 | 1b42557e | | #855 | 88916c09 | | #857 | 4488b5d6 | | #856 | e4af9a94 | Exact gates per merge, CI=1, CARGO_PROFILE_DEV_DEBUG=0, CARGO_PROFILE_TEST_DEBUG=0, CARGO_INCREMENTAL=0, all heavy commands through Nix and `~/bin/slot`: - `cargo build --locked -j4` - `bun install --frozen-lockfile` - from apps/web: `bun run build` - from apps/web, CALTERNAL_BIN and E2E_DIST set to that merge's outputs: `bunx playwright test e2e/search-corpus.spec.ts --project=chromium-1440 --no-deps --retries=0 --workers=1 --reporter=line` One runner-only error between #779 and #858 removed an exported TMPDIR; fixed the runner and resumed #858 before its test ran. No failed browser attempt was rerun in this sequence. This does not invalidate the full-train failure or establish a flake. Investigation continues into shared search state: `crates/search/src/index.rs:229-275` permits a stale typo vocabulary during its five-second background-refresh interval; the corpus setup only checks exact endpoint matches. A temporary existing-corpus probe is running to test that hypothesis. No search source fix or culprit claim yet. No train branch changed.
kayg commented 2026-11-02 18:50:15 +00:00
Author
Owner
Copy link

Source repairs pushed; no train branch changed:

  • #827 76f11dc4ba06468a2a0ed4d4592cea6038206696: let-chain clippy repair. Requested workspace clippy passes on dev-01, including again on the full BM9 scratch.
  • #856 b89e39bb313bb06b92da274124c6bbde30e2250a: corpus setup waits for all exact notes and the separately refreshed typo vocabulary; preserves all 1492 assertions and executes both requested desktop projects.

Full BM9 scratch 73abdbce9821f30e903668f8fe7781028410a1ec: clippy/build/web build pass; three executed invocations each of chromium-1440 and webkit-1440 pass, 12 tests / 8952 corpus queries / 0 misses. One extra WebKit launch failed before tests on an already-used ephemeral port; retained and replaced by one independent launch, with no test retries.

Important attribution limit: all 14 cumulative isolated merges passed, so no historical first-bad product merge was demonstrated. A warmed-vocabulary probe reproduced the incomplete readiness contract (48 misses with indexing=false), then passed with the fix (0 misses). This establishes the setup fault repaired in #856, not the exact historical 1106-miss cache state. Do not infer full-train green without rerunning the train.

Complete root-cause evidence, red/green probe, full source/scratch heads, exact gates, decisions and risks. Cold cumulative bisection. All own jobs are finished and large VM outputs cleaned. No merge or deployment.

Source repairs pushed; no train branch changed: - #827 `76f11dc4ba06468a2a0ed4d4592cea6038206696`: let-chain clippy repair. Requested workspace clippy passes on dev-01, including again on the full BM9 scratch. - #856 `b89e39bb313bb06b92da274124c6bbde30e2250a`: corpus setup waits for all exact notes and the separately refreshed typo vocabulary; preserves all 1492 assertions and executes both requested desktop projects. Full BM9 scratch `73abdbce9821f30e903668f8fe7781028410a1ec`: clippy/build/web build pass; three executed invocations each of chromium-1440 and webkit-1440 pass, 12 tests / 8952 corpus queries / 0 misses. One extra WebKit launch failed before tests on an already-used ephemeral port; retained and replaced by one independent launch, with no test retries. Important attribution limit: all 14 cumulative isolated merges passed, so no historical first-bad product merge was demonstrated. A warmed-vocabulary probe reproduced the incomplete readiness contract (48 misses with indexing=false), then passed with the fix (0 misses). This establishes the setup fault repaired in #856, not the exact historical 1106-miss cache state. Do not infer full-train green without rerunning the train. [Complete root-cause evidence, red/green probe, full source/scratch heads, exact gates, decisions and risks](https://git.kayg.org/kayg/calternal/pulls/856#issuecomment-75576). [Cold cumulative bisection](https://git.kayg.org/kayg/calternal/pulls/869#issuecomment-75542). All own jobs are finished and large VM outputs cleaned. No merge or deployment.
kayg commented 2026-11-02 18:54:02 +00:00
Author
Owner
Copy link

Replaced by Train BM10 #880: #880. Fresh assembly uses the current source heads and new first-run gates.

Replaced by Train BM10 #880: https://git.kayg.org/kayg/calternal/pulls/880. Fresh assembly uses the current source heads and new first-run gates.
kayg closed this pull request 2026-11-02 18:54:02 +00:00
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-native-contract (pull_request) Has been cancelled
Details
Markdown extraction contract / markdown-editor-contract (pull_request) Has been cancelled
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!869
No description provided.