Investigate XUser removal of A Notification subscription #1445

Closed
opened 2026-10-08 04:56:06 +00:00 by kayg · 7 comments
Owner

Observed during Forgejo #1402's XUser matrix against the shared release server at d395b6e25.

The A-owned Notification subscription fixture was created through POST /api/v1/notifications/subscriptions and returned HTTP 204. After cross-User replay, the matrix's owner-state verification queried A's subscription row by its fixture installation ID and found no row. The runner reported: A VERIFY owner state: A's Notification subscription changed after cross-User replay.

The probe did not identify which request or background action changed the row. No unauthorized action or product-code change is claimed. Please investigate the state change; the evidence is the successful fixture response followed by the failed owner-state check in the same run.

Observed during Forgejo #1402's XUser matrix against the shared release server at `d395b6e25`. The A-owned Notification subscription fixture was created through `POST /api/v1/notifications/subscriptions` and returned HTTP 204. After cross-User replay, the matrix's owner-state verification queried A's subscription row by its fixture installation ID and found no row. The runner reported: `A VERIFY owner state: A's Notification subscription changed after cross-User replay`. The probe did not identify which request or background action changed the row. No unauthorized action or product-code change is claimed. Please investigate the state change; the evidence is the successful fixture response followed by the failed owner-state check in the same run.
Author
Owner

Starting #1445 on job/xsub-1445, base d395b6e25da20d8c67b8b9fc1a593b813939c8a7. I read #1402’s final report and am isolating each matrix replay from session cleanup and background delivery. Findings will be recorded in artifacts/xsub-findings.md. No product defect is claimed yet.

Starting #1445 on `job/xsub-1445`, base `d395b6e25da20d8c67b8b9fc1a593b813939c8a7`. I read #1402’s final report and am isolating each matrix replay from session cleanup and background delivery. Findings will be recorded in `artifacts/xsub-findings.md`. No product defect is claimed yet.
Author
Owner

Reused #1402’s fixture commits on this branch (ce31d0617, 15e78025e) and committed the focused probe as d78fd33bb. The Notification fixture uses a synthetic fcm.googleapis.com path; #1402’s local TLS callback belongs to MCP Events. The store already uses (user_id, installation_id). Permanent push failures and exact-session cleanup can remove bindings.

The first live run stopped in fixture setup: the Money preview did not report ready within 45 seconds; the retained Index showed its Job completed. The focused rerun uses a bounded 180-second fixture wait and retains deletion context on failure. Offline classification/probe gates passed 24 tests. The Notifications crate gate is compiling. No product isolation defect is claimed yet.

Reused #1402’s fixture commits on this branch (`ce31d0617`, `15e78025e`) and committed the focused probe as `d78fd33bb`. The Notification fixture uses a synthetic `fcm.googleapis.com` path; #1402’s local TLS callback belongs to MCP Events. The store already uses `(user_id, installation_id)`. Permanent push failures and exact-session cleanup can remove bindings. The first live run stopped in fixture setup: the Money preview did not report ready within 45 seconds; the retained Index showed its Job completed. The focused rerun uses a bounded 180-second fixture wait and retains deletion context on failure. Offline classification/probe gates passed 24 tests. The Notifications crate gate is compiling. No product isolation defect is claimed yet.
Author
Owner

Reproduced the missing subscription on the d395b6e25 release server before any B/C/D/anonymous replay. A notifications.dispatch Job removed it while delivering a money_import_finished Notification. The deletion trace records revoked_at=NULL, idle expiry 1791542039, absolute expiry 1799231579, and deletion time 1791455665; the session was live.

The fixture registers a synthetic FCM path, which returns HTTP 404. The server logged no endpoint-validation rejection. The push sender expires a binding after provider 404/410, scoped by User, Installation, session and generation. The local TLS callback in #1402 serves MCP Events, not this push fixture. This is a background fixture failure; no foreign request is needed to reproduce it.

The matrix now uses a controlled local TLS push endpoint that acknowledges encrypted deliveries. The per-request B/C/D/anonymous replay and separate duplicate-destination/session/background cases are running.

Reproduced the missing subscription on the `d395b6e25` release server **before any B/C/D/anonymous replay**. A `notifications.dispatch` Job removed it while delivering a `money_import_finished` Notification. The deletion trace records `revoked_at=NULL`, idle expiry `1791542039`, absolute expiry `1799231579`, and deletion time `1791455665`; the session was live. The fixture registers a synthetic FCM path, which returns HTTP 404. The server logged no endpoint-validation rejection. The push sender expires a binding after provider 404/410, scoped by User, Installation, session and generation. The local TLS callback in #1402 serves MCP Events, not this push fixture. This is a background fixture failure; no foreign request is needed to reproduce it. The matrix now uses a controlled local TLS push endpoint that acknowledges encrypted deliveries. The per-request B/C/D/anonymous replay and separate duplicate-destination/session/background cases are running.
Author
Owner

The controlled release-server round at d395b6e25 passed. It replayed 1,680 individual requests across 224 operations as B, C, D and anonymous, reading A’s complete subscription row after each request. A’s binding stayed unchanged; zero transport/5xx failures. A’s deletion trace stayed empty.

B/C/D could register, replace and unsubscribe the same endpoint and Installation ID without changing A. An epoch-age binding with a live session stayed registered and received real encrypted pushes through local TLS. Provider 404/410 and permanent DNS rejection removed only the failing extra binding. Idle/absolute session expiry and sign-out removed only the ended session binding. B sign-out with A’s endpoint/Installation preserved A.

No cross-User isolation defect was found. The original removal is the background fixture failure reported above. Product behavior stays unchanged; code comments and a Rust regression document the boundaries. Rust gates are still compiling. Evidence is in artifacts/xsub-controlled.log, artifacts/xsub-findings.md, and the retained per-request trace.

The controlled release-server round at `d395b6e25` passed. It replayed **1,680 individual requests across 224 operations** as B, C, D and anonymous, reading A’s complete subscription row after each request. A’s binding stayed unchanged; zero transport/5xx failures. A’s deletion trace stayed empty. B/C/D could register, replace and unsubscribe the same endpoint and Installation ID without changing A. An epoch-age binding with a live session stayed registered and received real encrypted pushes through local TLS. Provider 404/410 and permanent DNS rejection removed only the failing extra binding. Idle/absolute session expiry and sign-out removed only the ended session binding. B sign-out with A’s endpoint/Installation preserved A. No cross-User isolation defect was found. The original removal is the background fixture failure reported above. Product behavior stays unchanged; code comments and a Rust regression document the boundaries. Rust gates are still compiling. Evidence is in `artifacts/xsub-controlled.log`, `artifacts/xsub-findings.md`, and the retained per-request trace.
Author
Owner

The endpoint-only collision follow-up passed and is committed as b20c0633f: identical endpoints with distinct Installation IDs across B/C/D preserve A. The complete requested live campaign remains green.

Rust formatting and crate clippy passed. Clippy output:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 32m 20s

cargo test -p calternal-plugin-notifications -- --test-threads=4 is now compiling. The remaining Rust change contains documentation and the regression test; production behavior is unchanged.

The endpoint-only collision follow-up passed and is committed as `b20c0633f`: identical endpoints with distinct Installation IDs across B/C/D preserve A. The complete requested live campaign remains green. Rust formatting and crate clippy passed. Clippy output: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 32m 20s ``` `cargo test -p calternal-plugin-notifications -- --test-threads=4` is now compiling. The remaining Rust change contains documentation and the regression test; production behavior is unchanged.
Author
Owner

#1445 complete: background push failure removed the fixture

The release server at d395b6e25 removed A's original binding before any foreign replay. The Notification fixture used a fake FCM provider path; the local TLS callback in #1402 served MCP Events, not push delivery. A Money import queued money_import_finished. Background push dispatch attempted delivery and pruned the permanently failed binding. The session was still live. A POST to the same fake provider path format returned HTTP 404. Controlled provider 404/410 and permanent DNS failure reproduced the scoped removal. No cross-User isolation defect was found.

Evidence from the original real-server run:

XSUB original fixture after idle dispatch: binding exists=False; deletion context=[(1791455665, None, 1791542039, 1799231579, 'money_import_finished')]; no foreign replay ran

The controlled real-server replay passed:

XSUB replay: 224 operations; 1680 individual requests; 0 transport/5xx failures; A binding unchanged
XSUB age/pruning: epoch creation retained; local TLS accepted 2 encrypted push(es)
XSUB provider 404: only the failing binding removed
XSUB provider 410: only the failing binding removed
XSUB permanent DNS rejection: only the failing binding removed
XSUB idle/absolute session expiry and sign-out: only the ended session binding removed
XSUB B sign-out with identical endpoint/Installation: A binding unchanged
PASS: #1445 per-request isolation and background-cause round

The follow-up checked identical endpoints with both identical and distinct Installation IDs:

XSUB duplicate endpoint/Installation: B/C/D register, replace, unsubscribe preserve A (same and distinct Installation IDs)
PASS: #1445 per-request isolation and background-cause round

A's deletion trace was empty in the controlled round. All B/C/D/anonymous requests were replayed individually, with the exact A row checked at every boundary.

Built and files

  • tests/adversarial/xuser_matrix.py: reuse #1402 fixtures; local TLS push fixture; exact A binding assertions after every foreign request; bounded optional busy-host fixture timeout.
  • tests/adversarial/xsub_1445.py: per-request real-server replay and background-cause investigation, including deletion context and endpoint/Installation collisions.
  • tests/adversarial/run.sh and dns_rebind_preload.c: extend the existing TLS/DNS test harness for local push acknowledgements and delivery-time DNS rejection.
  • tests/adversarial/test_xsub_1445.py and test_xuser_classification.py: focused harness/classification regressions, preserving existing expectations.
  • crates/plugins/notifications/src/push.rs and store.rs: document intentional permanent-failure pruning and User scoping; add identical_push_destinations_remain_user_scoped Rust regression.
  • docs/audits/xsub-1445.md: reproducible findings and commands.

Production behavior is unchanged. No dependencies, migrations or UI changes. Reused the existing store factories, matrix plan, MCP TLS callback and DNS shim. Comments were reviewed before reporting.

Head: 87cf73df9ba6bcd27c289e36a6f3984871b056c7 on job/xsub-1445. Commits: ce31d0617, 15e78025e, d78fd33bb, b074ac80e, b20c0633f, 87cf73df9. Fetched and merged origin/dev once before final gates: Already up to date. No push or deploy.

Gates (verbatim output excerpts)

cargo fmt --check had no formatter output; captured result:

cargo fmt --check exit=0

cargo clippy -p calternal-plugin-notifications --all-targets -- -D warnings exited 0:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 32m 20s

cargo test -p calternal-plugin-notifications -- --test-threads=4 exited 0:

    Finished `test` profile [unoptimized + debuginfo] target(s) in 65m 56s
test store::tests::identical_push_destinations_remain_user_scoped ... ok
test result: ok. 45 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.55s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

XUSER_CLASSIFY_ONLY=1 python3 -m unittest discover -s tests/adversarial -p 'test_x*py':

........................
----------------------------------------------------------------------
Ran 24 tests in 0.749s

OK

Shell syntax, Python compilation, C shim compilation with -Wall -Wextra -Werror, and git diff --check passed. Full logs and running findings remain in ignored artifacts/. Cleanup completed:

     Removed 11610 files, 7.1GiB total
Web build output removed

Working tree is clean.

Decisions

  • Reuse the existing local TLS/DNS callback for successful push delivery, so unrelated Notification dispatch cannot invalidate matrix fixtures.
  • Preserve the existing User/Installation schema and intentional permanent-failure/session cleanup behavior. Existing deletes also guard session and generation.
  • Keep the normal 45-second Money fixture wait; allow an explicit bounded 180-second wait on the busy shared host.

Expected cost: no new production operation. The ordinary matrix adds one indexed A-binding lookup per foreign request. The focused investigation reads before and after each request; the measured real-server round covered 1,680 requests and had zero transport/5xx failures. No performance benchmark is required for this fixture/documentation change.

Known gaps

None for #1445. The first 45-second fixture setup timed out; the completed 180-second rounds supersede it. No test expectation was relaxed. All requested subscription scenarios ran to completion. Broader combined-branch suites belong to the merge round.

For the merge round

Run the combined branch's full XUser response-profile and owner-state matrix:

source /mnt/hdd/targets/shared-server/CURRENT
XUSER_MATRIX_ONLY=1 XUSER_FIXTURE_TIMEOUT=180 \
  ADVERSARIAL_SERVER_BIN="$CALTERNAL_SERVER_RELEASE_BIN" \
  bash tests/adversarial/run.sh

The focused #1445 replay and background scenarios are already complete; this broader command checks the combined changes.

# #1445 complete: background push failure removed the fixture The release server at d395b6e25 removed A's original binding before any foreign replay. The Notification fixture used a fake FCM provider path; the local TLS callback in #1402 served MCP Events, not push delivery. A Money import queued `money_import_finished`. Background push dispatch attempted delivery and pruned the permanently failed binding. The session was still live. A POST to the same fake provider path format returned HTTP 404. Controlled provider 404/410 and permanent DNS failure reproduced the scoped removal. No cross-User isolation defect was found. Evidence from the original real-server run: ```text XSUB original fixture after idle dispatch: binding exists=False; deletion context=[(1791455665, None, 1791542039, 1799231579, 'money_import_finished')]; no foreign replay ran ``` The controlled real-server replay passed: ```text XSUB replay: 224 operations; 1680 individual requests; 0 transport/5xx failures; A binding unchanged XSUB age/pruning: epoch creation retained; local TLS accepted 2 encrypted push(es) XSUB provider 404: only the failing binding removed XSUB provider 410: only the failing binding removed XSUB permanent DNS rejection: only the failing binding removed XSUB idle/absolute session expiry and sign-out: only the ended session binding removed XSUB B sign-out with identical endpoint/Installation: A binding unchanged PASS: #1445 per-request isolation and background-cause round ``` The follow-up checked identical endpoints with both identical and distinct Installation IDs: ```text XSUB duplicate endpoint/Installation: B/C/D register, replace, unsubscribe preserve A (same and distinct Installation IDs) PASS: #1445 per-request isolation and background-cause round ``` A's deletion trace was empty in the controlled round. All B/C/D/anonymous requests were replayed individually, with the exact A row checked at every boundary. ## Built and files - `tests/adversarial/xuser_matrix.py`: reuse #1402 fixtures; local TLS push fixture; exact A binding assertions after every foreign request; bounded optional busy-host fixture timeout. - `tests/adversarial/xsub_1445.py`: per-request real-server replay and background-cause investigation, including deletion context and endpoint/Installation collisions. - `tests/adversarial/run.sh` and `dns_rebind_preload.c`: extend the existing TLS/DNS test harness for local push acknowledgements and delivery-time DNS rejection. - `tests/adversarial/test_xsub_1445.py` and `test_xuser_classification.py`: focused harness/classification regressions, preserving existing expectations. - `crates/plugins/notifications/src/push.rs` and `store.rs`: document intentional permanent-failure pruning and User scoping; add `identical_push_destinations_remain_user_scoped` Rust regression. - `docs/audits/xsub-1445.md`: reproducible findings and commands. Production behavior is unchanged. No dependencies, migrations or UI changes. Reused the existing store factories, matrix plan, MCP TLS callback and DNS shim. Comments were reviewed before reporting. Head: `87cf73df9ba6bcd27c289e36a6f3984871b056c7` on `job/xsub-1445`. Commits: ce31d0617, 15e78025e, d78fd33bb, b074ac80e, b20c0633f, 87cf73df9. Fetched and merged origin/dev once before final gates: Already up to date. No push or deploy. ## Gates (verbatim output excerpts) `cargo fmt --check` had no formatter output; captured result: ```text cargo fmt --check exit=0 ``` `cargo clippy -p calternal-plugin-notifications --all-targets -- -D warnings` exited 0: ```text Finished `dev` profile [unoptimized + debuginfo] target(s) in 32m 20s ``` `cargo test -p calternal-plugin-notifications -- --test-threads=4` exited 0: ```text Finished `test` profile [unoptimized + debuginfo] target(s) in 65m 56s test store::tests::identical_push_destinations_remain_user_scoped ... ok test result: ok. 45 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.55s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` `XUSER_CLASSIFY_ONLY=1 python3 -m unittest discover -s tests/adversarial -p 'test_x*py'`: ```text ........................ ---------------------------------------------------------------------- Ran 24 tests in 0.749s OK ``` Shell syntax, Python compilation, C shim compilation with `-Wall -Wextra -Werror`, and `git diff --check` passed. Full logs and running findings remain in ignored `artifacts/`. Cleanup completed: ```text Removed 11610 files, 7.1GiB total Web build output removed ``` Working tree is clean. ## Decisions - Reuse the existing local TLS/DNS callback for successful push delivery, so unrelated Notification dispatch cannot invalidate matrix fixtures. - Preserve the existing User/Installation schema and intentional permanent-failure/session cleanup behavior. Existing deletes also guard session and generation. - Keep the normal 45-second Money fixture wait; allow an explicit bounded 180-second wait on the busy shared host. Expected cost: no new production operation. The ordinary matrix adds one indexed A-binding lookup per foreign request. The focused investigation reads before and after each request; the measured real-server round covered 1,680 requests and had zero transport/5xx failures. No performance benchmark is required for this fixture/documentation change. ## Known gaps None for #1445. The first 45-second fixture setup timed out; the completed 180-second rounds supersede it. No test expectation was relaxed. All requested subscription scenarios ran to completion. Broader combined-branch suites belong to the merge round. ## For the merge round Run the combined branch's full XUser response-profile and owner-state matrix: ```sh source /mnt/hdd/targets/shared-server/CURRENT XUSER_MATRIX_ONLY=1 XUSER_FIXTURE_TIMEOUT=180 \ ADVERSARIAL_SERVER_BIN="$CALTERNAL_SERVER_RELEASE_BIN" \ bash tests/adversarial/run.sh ``` The focused #1445 replay and background scenarios are already complete; this broader command checks the combined changes.
kayg referenced this issue from a commit 2026-10-08 19:43:32 +00:00
Author
Owner

Deployed to calternal.cloud in merge round 35b (build 9dc65437e, live 2026-10-09 ~11:07 IST). Closing.

Deployed to calternal.cloud in merge round 35b (build 9dc65437e, live 2026-10-09 ~11:07 IST). Closing.
kayg closed this issue 2026-10-09 05:36:33 +00:00
Commenting is not possible because the repository is archived.
No labels
No milestone
No project
No assignees
1 participant
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal-v0#1445
No description provided.