Mesh rotation: every completed rotation logs ERROR 'Mesh node identity does not match rotation state' (store.rs config_for_identity race) #385

New issue

Open

opened 2026-11-06 02:35:10 +00:00 by kayg · 0 comments

kayg commented

2026-11-06 02:35:10 +00:00

Owner

Found by J267 (#375), dev 0b72f638a, kernel-engine server node, rotation interval 120 s. Severity P3 (an error at every successful rotation; there may be a short config gap).

Each completed key rotation of the server's own node logs an ERROR on the server at the moment the rotation ends:

02:05:56.327700Z ERROR infraternal_plugin_mesh::diagnostics: Mesh storage unavailable event="mesh_storage_error" operation="mesh.storage" reason="Mesh node identity does not match rotation state." file="crates/plugins/mesh/src/store.rs" line=199 count=1 previous_count=0
02:05:56.328879Z ERROR … (same)
02:09:56.409329Z ERROR … (same)

02:05:56 and 02:09:56 are exactly the two rotations' last_rotation times (1793930756 and 1793930996), when the key went a502df2c… → fad66746… → bf2a7791…. config_for_identity (store.rs:180-201) compares the identity the node presents with the store's loaded snapshot. When the phase leaves overlap, the node already presents the candidate while the snapshot still holds the old local_key, so the else if branch errors. The node stayed live and the next rotation worked, so this looks like a race between the node's switch and the snapshot reload. It still surfaces as "Mesh storage unavailable" at ERROR level on every rotation (default 7 days, per node), and that config request fails.

Repro: a server on the kernel engine (INFRATERNAL_MESH_ENGINE=kernel), mesh.rotation.interval.set {rotation_period_secs:120}, wait for two rotations, grep "does not match rotation state" server.log. Log: root@compute01:/root/j267/server2.log (deleted at J267 cleanup; the lines are quoted above).

Expected: no error for the node's own expected key switch (accept the new key once the phase is committed, or reload before comparing).

Links: #285, #375, D166.

Found by J267 (#375), dev `0b72f638a`, kernel-engine server node, rotation interval 120 s. Severity **P3** (an error at every successful rotation; there may be a short config gap). Each completed key rotation of the server's own node logs an ERROR on the server at the moment the rotation ends: ``` 02:05:56.327700Z ERROR infraternal_plugin_mesh::diagnostics: Mesh storage unavailable event="mesh_storage_error" operation="mesh.storage" reason="Mesh node identity does not match rotation state." file="crates/plugins/mesh/src/store.rs" line=199 count=1 previous_count=0 02:05:56.328879Z ERROR … (same) 02:09:56.409329Z ERROR … (same) ``` 02:05:56 and 02:09:56 are exactly the two rotations' `last_rotation` times (1793930756 and 1793930996), when the key went `a502df2c…` → `fad66746…` → `bf2a7791…`. `config_for_identity` (`store.rs:180-201`) compares the identity the node presents with the store's loaded snapshot. When the phase leaves `overlap`, the node already presents the candidate while the snapshot still holds the old `local_key`, so the `else if` branch errors. The node stayed live and the next rotation worked, so this looks like a race between the node's switch and the snapshot reload. It still surfaces as "Mesh storage unavailable" at ERROR level on every rotation (default 7 days, per node), and that config request fails. **Repro:** a server on the kernel engine (`INFRATERNAL_MESH_ENGINE=kernel`), `mesh.rotation.interval.set {rotation_period_secs:120}`, wait for two rotations, `grep "does not match rotation state" server.log`. Log: `root@compute01:/root/j267/server2.log` (deleted at J267 cleanup; the lines are quoted above). Expected: no error for the node's own expected key switch (accept the new key once the phase is committed, or reload before comparing). Links: #285, #375, D166.

kayg referenced this issue

2026-11-06 02:38:15 +00:00

Mac mesh break-it on dev 9b0c731e8 (J267) #375

kayg referenced this issue

2026-11-06 16:54:37 +00:00

Housekeeping 2026-11-06 #402