kayg/calternal

Watch

Fork 0

Feature flags: on by default, kill switch on Admin → Features, removed by day 14 #1181

New issue

Open

opened 2026-10-06 05:04:32 +00:00 by kayg · 7 comments

kayg commented

2026-10-06 05:04:32 +00:00

Owner

Owner decisions (2026-10-05/06): feature flags

New features land ON by default; each has a flag that works as a kill switch on Admin → Features (name, issue link, owner, created date, age, on/off), toggling takes effect without a restart and is audit-logged.
Flags are temporary: the daily digest lists flags older than 10 days; a job removes the flag and its dead branch by day 14 (the feature stays). A design-rule test fails the build for any flag older than 14 days.
One small shared flag primitive (server + web), read from the existing settings store; no third-party flag service.

## Owner decisions (2026-10-05/06): feature flags - New features land ON by default; each has a flag that works as a kill switch on Admin → Features (name, issue link, owner, created date, age, on/off), toggling takes effect without a restart and is audit-logged. - Flags are temporary: the daily digest lists flags older than 10 days; a job removes the flag and its dead branch by day 14 (the feature stays). A design-rule test fails the build for any flag older than 14 days. - One small shared flag primitive (server + web), read from the existing settings store; no third-party flag service.

features-1440-dark-macos.png

387 KiB

features-1440-light-macos.png

574 KiB

features-390-dark-macos.png

129 KiB

features-390-light-macos.png

164 KiB

features-820-dark-macos.png

280 KiB

features-820-light-macos.png

406 KiB

kayg commented

2026-10-06 05:13:51 +00:00

Author

Owner

Starting #1181 on branch job/flags-1181, based on d0061ec3df127d81c86729d07d899b0bf2b6de91 (origin/dev at job start). I am mapping the existing settings and admin flows before implementing the shared feature flag primitive.

Starting #1181 on branch `job/flags-1181`, based on `d0061ec3df127d81c86729d07d899b0bf2b6de91` (origin/dev at job start). I am mapping the existing settings and admin flows before implementing the shared feature flag primitive.

kayg commented

2026-10-06 05:20:24 +00:00

Author

Owner

Finding: Admin → Features currently lists Plugin enablement only (apps/web/src/routes/settings/plugins/PluginsSection.svelte); there is no temporary feature flag catalog or lifecycle digest. The server already stores instance controls in the Index and writes their audit record in the same FULL authority transaction (crates/calternal-plugin/src/state.rs, crates/calternal-server/src/wire.rs). I will reuse that authority store and existing Admin Features route, with a separate flag override table and a bounded lifecycle checker.

Finding: Admin → Features currently lists Plugin enablement only (`apps/web/src/routes/settings/plugins/PluginsSection.svelte`); there is no temporary feature flag catalog or lifecycle digest. The server already stores instance controls in the Index and writes their audit record in the same FULL authority transaction (`crates/calternal-plugin/src/state.rs`, `crates/calternal-server/src/wire.rs`). I will reuse that authority store and existing Admin Features route, with a separate flag override table and a bounded lifecycle checker.

kayg commented

2026-10-06 05:57:32 +00:00

Author

Owner

The repository had no daily scheduler for this digest; the existing scheduled workflows ran weekly. Added a Forgejo daily workflow on dev at 04:00 UTC. It prints the day-10 digest and fails after day 14. Evidence: python3 -m unittest discover -s scripts -p 'test_feature_flag_lifecycle.py' passed (3 tests). Commit: 80561c668.

The repository had no daily scheduler for this digest; the existing scheduled workflows ran weekly. Added a Forgejo daily workflow on `dev` at 04:00 UTC. It prints the day-10 digest and fails after day 14. Evidence: `python3 -m unittest discover -s scripts -p 'test_feature_flag_lifecycle.py'` passed (3 tests). Commit: `80561c668`.

kayg commented

2026-10-06 07:53:03 +00:00

Author

Owner

The first cargo test -p calternal-server run built successfully and reported 259 passed, 10 ignored, and 2 failed. wire::upgrade_tests::production_round_9_schema_upgrades_once_with_original_receipts expected the pre-#1181 pending migration set; I updated it to include calternal-plugin migration 3. serve::tests::direct_untrusted_peer_keeps_the_256_connection_cap hit OS error 24 (Too many open files) under default test parallelism. I am rerunning the server suite with four test threads to check whether the resource failure clears.

The first `cargo test -p calternal-server` run built successfully and reported 259 passed, 10 ignored, and 2 failed. `wire::upgrade_tests::production_round_9_schema_upgrades_once_with_original_receipts` expected the pre-#1181 pending migration set; I updated it to include `calternal-plugin` migration 3. `serve::tests::direct_untrusted_peer_keeps_the_256_connection_cap` hit OS error 24 (`Too many open files`) under default test parallelism. I am rerunning the server suite with four test threads to check whether the resource failure clears.

kayg commented

2026-10-06 07:58:38 +00:00

Author

Owner

The four-thread server suite cleared serve::tests::direct_untrusted_peer_keeps_the_256_connection_cap. Its remaining upgrade fixture assertion counted one fewer migration; I updated the post-upgrade inventory for plugin migration 3. The suite also reported the ignored-test supervisor wire::tests::live_apps_run_in_separate_processes failed because its child startup_serves_http_while_upgrade_backfills_wait exceeded the existing 15-second HTTP-startup deadline under shared-host load. This is a SLOW-only test failure; the #1181 tests passed. I will verify the migration fixture directly and report the startup timing result.

The four-thread server suite cleared `serve::tests::direct_untrusted_peer_keeps_the_256_connection_cap`. Its remaining upgrade fixture assertion counted one fewer migration; I updated the post-upgrade inventory for plugin migration 3. The suite also reported the ignored-test supervisor `wire::tests::live_apps_run_in_separate_processes` failed because its child `startup_serves_http_while_upgrade_backfills_wait` exceeded the existing 15-second HTTP-startup deadline under shared-host load. This is a SLOW-only test failure; the #1181 tests passed. I will verify the migration fixture directly and report the startup timing result.

kayg commented

2026-10-06 09:50:33 +00:00

Author

Owner

After merging the current origin/dev, the source parser found stale exact performance pins in the root layout, PluginsSection, AccountList, and the /api/v1/plugins operation. I refreshed 61 hashes while preserving each limit, moved 12 existing AccountList scopes to its new keyed list identity, and added exact tested pure-call bindings for the Admin catalog response. This removed three obsolete unresolved-call exceptions and reduced the ledger and ratchet from 22,359 to 22,356; the current parser reports no unpinned findings.

After merging the current `origin/dev`, the source parser found stale exact performance pins in the root layout, PluginsSection, AccountList, and the `/api/v1/plugins` operation. I refreshed 61 hashes while preserving each limit, moved 12 existing AccountList scopes to its new keyed list identity, and added exact tested pure-call bindings for the Admin catalog response. This removed three obsolete unresolved-call exceptions and reduced the ledger and ratchet from 22,359 to 22,356; the current parser reports no unpinned findings.

kayg commented

2026-10-06 11:19:34 +00:00

Author

Owner

Built

Added a default-on feature flag catalog, Instance-level kill switches in the existing settings store, audited Admin writes, and live updates without a restart.
Added the Admin Settings catalog with stable links, copy link, menu actions, and real empty, loading, and error states.
Added a daily lifecycle digest for flags older than 10 days and a design-rule test that rejects definitions older than 14 days.
Added the API/OpenAPI/client contract, authorization coverage, regression tests, the focused browser profile, and six production screenshots.
Fixed an effect dependency loop found during production E2E: Resource.load() reads its reactive snapshot, so the auth refresh callback now calls it under untrack().

Files

Rust/API: crates/calternal-plugin/{migrations/0003_feature_flag_state.sql,src/feature_flag_admin.rs,src/feature_flags.rs,src/state.rs,src/lib.rs}, crates/calternal-api/src/lib.rs, crates/calternal-server/src/{wire.rs,main.rs,upgrade_tests.rs}.

Web: apps/web/src/lib/featureFlags.ts, its tests, apps/web/src/routes/settings/{plugins/PluginsSection.svelte,sections.ts}, the settings tests, AccountList.svelte, +layout.svelte, and search/providers.ts.

Contracts and generated API: contracts/{feature-flags.json,actions.json,action-policy.json,openapi.json}, packages/api-client/src/generated.ts.

Lifecycle, CI, verification and perf: .forgejo/workflows/{ci.yml,feature-flag-lifecycle.yml}, scripts/feature_flag_lifecycle.py, its tests, tests/adversarial/authz_matrix.py, apps/web/e2e/feature-flags-1181.mjs, bench/feature-flags-1181.mjs, and exact perf registry pins under contracts/perf/.

Decisions

Used a checked-in canonical JSON catalog for source-owned metadata. The existing settings store holds only Instance overrides; a missing override means enabled.
Kept the shared server/web flag primitive and did not add a third-party flag service.

UX gaps closed

Production E2E exposed and verified the fix for repeated catalog requests that kept the Admin empty state loading. The real empty state now renders after the two catalog reads.
Captured macOS production screenshots at 390, 820, and 1440 px in light and dark modes.

UX gaps left

No unresolved interaction gap surfaced in the focused tests or production E2E. Visual review of the attached screenshots remains with the orchestrator.

Performance

The single local profile recorded 25 average rounds (50 requests): p50 24.56 ms, p95 72.17 ms, mean server CPU 16.22%, and mean RSS 231,359,521 bytes. The 100-round burst (200 requests) recorded p50 30.88 ms, p95 147.73 ms, mean CPU 40.36%, and mean RSS 246,124,803 bytes. Local one-minute load average before the run was 30.97. The perf VM lock was busy or unreachable. docs/perf/baseline.json has no feature-flag endpoint metric, so there is no prior number for comparison.

Gates (output excerpts verbatim)

cargo fmt --all -- --check
(no output; exit 0)

Finished `dev` profile [unoptimized + debuginfo] target(s) in 6.39s
Finished `dev` profile [unoptimized + debuginfo] target(s) in 18.35s
Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 24s

test result: ok. 18 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.27s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 57 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 20.21s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
test result: ok. 267 passed; 0 failed; 10 ignored; 0 measured; 0 filtered out; finished in 309.94s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 59.75s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.40s

svelte-check found 0 errors and 2 warnings in 2 files
Test Files  275 passed (275)
Tests  1912 passed (1912)
Test Files  2 passed (2)
Tests  23 passed (23)
Compressed 905 static variants; saved 21097443 bytes.
PASS #1181: six macOS production screenshots at 390, 820 and 1440px in light and dark (/home/kayg/Developer/calternal-wt/flags-1181/apps/web/artifacts/feature-flags-1181)
perf-lint: PASS; 0 violations; 22356 scoped exceptions

bun run test passed after the origin/dev merge. After the later untrack() fix, bun run check, the focused Vitest files (23 tests), production build, and production E2E passed again.

Screenshots

Head: 6257a37db5e4917f6e163bc4e30c4ea5505a08fb.

## Built - Added a default-on feature flag catalog, Instance-level kill switches in the existing settings store, audited Admin writes, and live updates without a restart. - Added the Admin Settings catalog with stable links, copy link, menu actions, and real empty, loading, and error states. - Added a daily lifecycle digest for flags older than 10 days and a design-rule test that rejects definitions older than 14 days. - Added the API/OpenAPI/client contract, authorization coverage, regression tests, the focused browser profile, and six production screenshots. - Fixed an effect dependency loop found during production E2E: `Resource.load()` reads its reactive snapshot, so the auth refresh callback now calls it under `untrack()`. ## Files Rust/API: `crates/calternal-plugin/{migrations/0003_feature_flag_state.sql,src/feature_flag_admin.rs,src/feature_flags.rs,src/state.rs,src/lib.rs}`, `crates/calternal-api/src/lib.rs`, `crates/calternal-server/src/{wire.rs,main.rs,upgrade_tests.rs}`. Web: `apps/web/src/lib/featureFlags.ts`, its tests, `apps/web/src/routes/settings/{plugins/PluginsSection.svelte,sections.ts}`, the settings tests, `AccountList.svelte`, `+layout.svelte`, and `search/providers.ts`. Contracts and generated API: `contracts/{feature-flags.json,actions.json,action-policy.json,openapi.json}`, `packages/api-client/src/generated.ts`. Lifecycle, CI, verification and perf: `.forgejo/workflows/{ci.yml,feature-flag-lifecycle.yml}`, `scripts/feature_flag_lifecycle.py`, its tests, `tests/adversarial/authz_matrix.py`, `apps/web/e2e/feature-flags-1181.mjs`, `bench/feature-flags-1181.mjs`, and exact perf registry pins under `contracts/perf/`. ## Decisions - Used a checked-in canonical JSON catalog for source-owned metadata. The existing settings store holds only Instance overrides; a missing override means enabled. - Kept the shared server/web flag primitive and did not add a third-party flag service. ## UX gaps closed - Production E2E exposed and verified the fix for repeated catalog requests that kept the Admin empty state loading. The real empty state now renders after the two catalog reads. - Captured macOS production screenshots at 390, 820, and 1440 px in light and dark modes. ## UX gaps left - No unresolved interaction gap surfaced in the focused tests or production E2E. Visual review of the attached screenshots remains with the orchestrator. ## Performance The single local profile recorded 25 average rounds (50 requests): p50 24.56 ms, p95 72.17 ms, mean server CPU 16.22%, and mean RSS 231,359,521 bytes. The 100-round burst (200 requests) recorded p50 30.88 ms, p95 147.73 ms, mean CPU 40.36%, and mean RSS 246,124,803 bytes. Local one-minute load average before the run was 30.97. The perf VM lock was busy or unreachable. `docs/perf/baseline.json` has no feature-flag endpoint metric, so there is no prior number for comparison. ## Gates (output excerpts verbatim) ```text cargo fmt --all -- --check (no output; exit 0) Finished `dev` profile [unoptimized + debuginfo] target(s) in 6.39s Finished `dev` profile [unoptimized + debuginfo] target(s) in 18.35s Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 24s test result: ok. 18 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.27s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 57 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 20.21s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s test result: ok. 267 passed; 0 failed; 10 ignored; 0 measured; 0 filtered out; finished in 309.94s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 59.75s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.40s svelte-check found 0 errors and 2 warnings in 2 files Test Files 275 passed (275) Tests 1912 passed (1912) Test Files 2 passed (2) Tests 23 passed (23) Compressed 905 static variants; saved 21097443 bytes. PASS #1181: six macOS production screenshots at 390, 820 and 1440px in light and dark (/home/kayg/Developer/calternal-wt/flags-1181/apps/web/artifacts/feature-flags-1181) perf-lint: PASS; 0 violations; 22356 scoped exceptions ``` `bun run test` passed after the `origin/dev` merge. After the later `untrack()` fix, `bun run check`, the focused Vitest files (23 tests), production build, and production E2E passed again. ## Screenshots ![390 px light, macOS](https://git.kayg.org/attachments/3e65f4cf-b3aa-46ac-9290-37c0483e791d) ![390 px dark, macOS](https://git.kayg.org/attachments/c0e8e20a-8712-410f-b9d3-54a2d408a085) ![820 px light, macOS](https://git.kayg.org/attachments/827a3995-e8ea-4b2a-ab09-263abd889fec) ![820 px dark, macOS](https://git.kayg.org/attachments/0992507f-0f53-4b10-9f8b-2f4c45b5ff4c) ![1440 px light, macOS](https://git.kayg.org/attachments/84579655-f217-4715-8a48-1488dd1d6425) ![1440 px dark, macOS](https://git.kayg.org/attachments/a4e19a1b-948a-4669-9bfe-75fbae97c723) Head: `6257a37db5e4917f6e163bc4e30c4ea5505a08fb`.

kayg referenced this issue

2026-10-06 12:09:05 +00:00

Reliability round: probe the routes added 2026-10-05/06 on a real server #1202

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kayg/calternal#1181

No description provided.

Rows
Columns