pjandro/OmniRoute
1
0
Fork 0
mirror of https://github.com/diegosouzapw/OmniRoute.git synced 2026-08-03 13:52:09 +03:00
Code Issues Packages Projects Releases Wiki Activity

ci(quality): wire Stryker mutation testing as advisory nightly (Fase 7 Task 11) (#3898)

Browse Source 
Integrated into release/v3.8.26
This commit is contained in:
Diego Rodrigues de Sa e Souza
2026-06-15 14:11:06 -03:00
committed by GitHub
parent e9d789822f
commit 9f5e651b42
6 changed files with 1567 additions and 84 deletions
Download Patch File Download Diff File Expand all files Collapse all files

37
.github/workflows/nightly-mutation.yml vendored Normal file
View File

@@ -0,0 +1,37 @@
name: Nightly Mutation
on:
schedule:
- cron: "17 3 * * *"
workflow_dispatch:
permissions:
contents: read
jobs:
stryker:
name: Stryker mutation testing (8 critical modules — advisory)
runs-on: ubuntu-latest
# Mutation testing is expensive (~200-500 mutants, 30-90 min). It runs only
# on the nightly schedule / manual dispatch, never on PRs. The score is NOT
# yet enforced as a ratchet (wired in a later INT phase) — for now the job
# just produces the HTML/JSON report and uploads it as an artifact.
timeout-minutes: 120
steps:
- uses: actions/checkout@v6
- uses: actions/setup-node@v6
with:
node-version: "24"
cache: npm
- run: npm ci
- name: Run Stryker (advisory)
id: stryker
continue-on-error: true
run: npx stryker run
- name: Upload mutation report
if: always()
uses: actions/upload-artifact@v4
with:
name: mutation-report
path: reports/mutation/
if-no-files-found: warn
retention-days: 14

5
.gitignore vendored
View File

@@ -23,6 +23,11 @@ coverage/
.build/**
.out/**
# Stryker mutation testing — ephemeral sandbox + generated reports (never commit)
.stryker-tmp/
reports/mutation/
stryker-output-*.json
# Memory Bank and Cursor rules (local-only AI agent context)
memory-bank/

2
config/quality/dependency-allowlist.json
View File

@@ -13,6 +13,8 @@
"@ngrok/ngrok",
"@opencode-ai/plugin",
"@playwright/test",
"@stryker-mutator/core",
"@stryker-mutator/tap-runner",
"@swc/helpers",
"@tailwindcss/postcss",
"@testing-library/jest-dom",

1390
package-lock.json generated
View File

File diff suppressed because it is too large Load Diff

3
package.json
View File

@@ -163,6 +163,7 @@
"test:protocols:e2e": "node scripts/dev/run-protocol-clients-tests.mjs",
"test:vitest": "vitest run --config vitest.mcp.config.ts",
"test:vitest:ui": "vitest run --config vitest.config.ts tests/unit/ui",
"test:mutation": "stryker run",
"test:ecosystem": "node scripts/dev/run-ecosystem-tests.mjs",
"test:system": "cross-env DISABLE_SQLITE_AUTO_BACKUP=true node --import tsx --import ./open-sse/utils/setupPolyfill.ts --test --test-force-exit --test-concurrency=1 tests/e2e/system-failover.test.ts",
"test:coverage": "cross-env DISABLE_SQLITE_AUTO_BACKUP=true c8 --output-dir coverage --exclude=tests/** --exclude=**/*.test.* --reporter=text-summary --reporter=html --reporter=json-summary --reporter=lcov --check-coverage --statements 60 --lines 60 --functions 60 --branches 60 node --max-old-space-size=8192 --import tsx --import ./open-sse/utils/setupPolyfill.ts --test --test-force-exit --test-concurrency=8 tests/unit/*.test.ts \"tests/unit/{api,auth,authz,build,cli,cli-helper,compression,correctness,cors,dashboard,db,db-adapters,docs,gamification,guardrails,lib,mcp,runtime,security,services,settings,shared,ui}/**/*.test.ts\"",
@@ -261,6 +262,8 @@
"devDependencies": {
"@cyclonedx/cyclonedx-npm": "4.2.1",
"@playwright/test": "^1.60.0",
"@stryker-mutator/core": "^9.6.1",
"@stryker-mutator/tap-runner": "^9.6.1",
"@tailwindcss/postcss": "^4.3.0",
"@testing-library/jest-dom": "^6.9.1",
"@testing-library/react": "^16.3.2",

214
stryker.conf.json
View File

@@ -5,23 +5,181 @@
"NIGHTLY ONLY — DO NOT run on every PR. Mutation testing is expensive:",
" - Each mutant requires a full test suite execution.",
" - The 8 modules produce ~200–500 mutants; est. 30–90 min per run.",
" - Wired to the 'nightly' CI job (not 'lint' or 'quality-gate').",
" - Wired to the nightly CI workflow (.github/workflows/nightly-mutation.yml),",
" NOT to the 'lint' / 'quality-gate' PR jobs.",
"",
"TEST RUNNER — @stryker-mutator/tap-runner (NOT vitest):",
" The 8 critical modules are covered by node:test files in tests/unit/",
" (run via `node --import tsx --test`), NOT by vitest. The vitest config",
" only includes a small set of .test.tsx + open-sse/**/__tests__ files, so",
" the vitest-runner would find ZERO covering tests for these modules. The",
" tap-runner spawns each node:test file individually and parses its TAP",
" output, which matches how this repo actually exercises the modules.",
" Each test file is loaded with tsx (TypeScript/ESM) + the project polyfill,",
" the same way `npm run test:unit` runs.",
"",
"Install before running (not bundled to avoid E2E / CI bloat):",
" npm install --save-dev @stryker-mutator/core @stryker-mutator/vitest-runner",
" (or @stryker-mutator/tap-runner if using node:test natively)",
" npm install --save-dev @stryker-mutator/core @stryker-mutator/tap-runner",
"",
"Run manually:",
" npx stryker run",
" npx stryker run --filePattern 'open-sse/handlers/chatCore.ts'",
" npm run test:mutation # full run (slow — nightly budget)",
" npx stryker run --dryRunOnly # validate the baseline only (no mutants)",
" (single-module probe: temporarily narrow `mutate` + `tap.testFiles` in this file)",
"",
"VALIDATED 2026-06-15: `npx stryker run --dryRunOnly` exits 0 — all 129 covering",
"test files run green in the Stryker sandbox and the perTest coverage map builds for",
"all 8 instrumented modules (15k+ mutants). The baseline dry-run takes ~20 min with",
"concurrency=1; the full mutation phase runs on top (advisory, capped by the workflow",
"timeout). So the nightly produces REAL mutation scores for the 8 modules.",
"",
"Mutation score per module → quality-baseline.json key 'mutationScore.<module>'",
"Direction: up (score can only improve; ratchet blocks drops — wired in INT phase)."
"Direction: up (score can only improve; ratchet blocks drops — wired in a later INT phase)."
],
"packageManager": "npm",
"testRunner": "vitest",
"vitest": {
"configFile": "vitest.config.ts"
"testRunner": "tap",
"plugins": ["@stryker-mutator/tap-runner"],
"tap": {
"testFiles": [
"tests/unit/account-fallback-anthropic-quota.test.ts",
"tests/unit/account-fallback-route-restriction-403.test.ts",
"tests/unit/account-fallback-service.test.ts",
"tests/unit/api-key-rotator-health.test.ts",
"tests/unit/appearance-widget-settings-schema.test.ts",
"tests/unit/auth-clear-account-error.test.ts",
"tests/unit/auth-disable-cooling-2997.test.ts",
"tests/unit/auth-extract-api-key.test.ts",
"tests/unit/auth-noauth-fallback-loop-3061.test.ts",
"tests/unit/auth-ollama-cloud-per-model-403-3027.test.ts",
"tests/unit/auth-opencode-zen-noauth-fallback.test.ts",
"tests/unit/auth-terminal-status.test.ts",
"tests/unit/authz/routeGuard.test.ts",
"tests/unit/auto-combo-context-advertising.test.ts",
"tests/unit/auto-combo-engine.test.ts",
"tests/unit/auto-combo-scoring-clamp.test.ts",
"tests/unit/build/check-circular-deps.test.ts",
"tests/unit/cache-sweeps.test.ts",
"tests/unit/cc-compatible-provider.test.ts",
"tests/unit/chat-context-relay.test.ts",
"tests/unit/chat-cooldown-aware-retry.test.ts",
"tests/unit/chat-helpers.test.ts",
"tests/unit/chat-route-coverage.test.ts",
"tests/unit/chat-route-edge-cases.test.ts",
"tests/unit/chatcore-compression-integration.test.ts",
"tests/unit/chatcore-extracted-modules-3821.test.ts",
"tests/unit/chatcore-imports-cleanly.test.ts",
"tests/unit/chatcore-sanitization.test.ts",
"tests/unit/chatcore-strip-stale-headers.test.ts",
"tests/unit/chatcore-translation-paths.test.ts",
"tests/unit/check-error-helper.test.ts",
"tests/unit/check-route-guard-membership.test.ts",
"tests/unit/check-test-discovery.test.ts",
"tests/unit/circuit-breaker-failure-kind.test.ts",
"tests/unit/claude-code-parity.test.ts",
"tests/unit/claude-effort-suffix-strip.test.ts",
"tests/unit/claude-oauth-provider.test.ts",
"tests/unit/claude-passthrough-stream-boolean.test.ts",
"tests/unit/claude-passthrough-thinking-2454.test.ts",
"tests/unit/cli-simulate.test.ts",
"tests/unit/codex-failover.test.ts",
"tests/unit/codex-stream-false.test.ts",
"tests/unit/collect-metrics-module-coverage.test.ts",
"tests/unit/combo-499-abort.test.ts",
"tests/unit/combo-auto-candidate-expansion.test.ts",
"tests/unit/combo-cache-invalidation.test.ts",
"tests/unit/combo-config.test.ts",
"tests/unit/combo-context-relay.test.ts",
"tests/unit/combo-health-autopilot.test.ts",
"tests/unit/combo-health-dashboard.test.ts",
"tests/unit/combo-health-route.test.ts",
"tests/unit/combo-hedging.test.ts",
"tests/unit/combo-max-depth-config.test.ts",
"tests/unit/combo-omnimodel-tag-stripping.test.ts",
"tests/unit/combo-prescreen.test.ts",
"tests/unit/combo-provider-cooldown.test.ts",
"tests/unit/combo-provider-diversity-wiring.test.ts",
"tests/unit/combo-quality-validator-reasoning.test.ts",
"tests/unit/combo-quota-soft-penalty.test.ts",
"tests/unit/combo-round-robin-streaming-lock-3811.test.ts",
"tests/unit/combo-routing-engine.test.ts",
"tests/unit/combo-scoring-inspector.test.ts",
"tests/unit/combo-sessionless-pin-3825.test.ts",
"tests/unit/combo-strategies.test.ts",
"tests/unit/combo-strategy-fallbacks.test.ts",
"tests/unit/combo-streaming-empty-content-failover.test.ts",
"tests/unit/combo-target-defensive-modelstr.test.ts",
"tests/unit/complexity-aware-scoring-wiring.test.ts",
"tests/unit/context-pinning-tool-calls.test.ts",
"tests/unit/correctness/combo.property.test.ts",
"tests/unit/correctness/sanitizers.property.test.ts",
"tests/unit/custom-model-target-format.test.ts",
"tests/unit/db-reset-module-state.test.ts",
"tests/unit/domain-persistence.test.ts",
"tests/unit/embeddings-auth.test.ts",
"tests/unit/error-classification.test.ts",
"tests/unit/error-message-sanitization.test.ts",
"tests/unit/executor-antigravity.test.ts",
"tests/unit/executor-web-cookie-sweep.test.ts",
"tests/unit/gemini-web-missing-browser-3516.test.ts",
"tests/unit/guardrails-api-3496.test.ts",
"tests/unit/memory-embedding-remote.test.ts",
"tests/unit/memory-embedding-transformers.test.ts",
"tests/unit/model-cooldowns-route-auth.test.ts",
"tests/unit/model-cooldowns-route.test.ts",
"tests/unit/model-lockout-decay.test.ts",
"tests/unit/oauth-providers-config.test.ts",
"tests/unit/oauth-redirect-uri-mismatch.test.ts",
"tests/unit/observability-fase04.test.ts",
"tests/unit/observability-payloads.test.ts",
"tests/unit/plan3-p0.test.ts",
"tests/unit/plugin-sandbox-permissions.test.ts",
"tests/unit/plugins-route-error-sanitization.test.ts",
"tests/unit/provider-error-rules.test.ts",
"tests/unit/provider-health-autopilot.test.ts",
"tests/unit/provider-health-matrix.test.ts",
"tests/unit/provider-request-failure-pipeline.test.ts",
"tests/unit/public-client-ids-3493.test.ts",
"tests/unit/publicCreds.test.ts",
"tests/unit/qoder-oauth-config.test.ts",
"tests/unit/quota-groups-route.test.ts",
"tests/unit/quota-key-models-route.test.ts",
"tests/unit/quota-policy-generalization.test.ts",
"tests/unit/quota-pool-log-route.test.ts",
"tests/unit/quota-streaming-consumption-usd.test.ts",
"tests/unit/rate-limit-enhanced.test.ts",
"tests/unit/rate-limit-manager.test.ts",
"tests/unit/responses-handler.test.ts",
"tests/unit/route-explainability.test.ts",
"tests/unit/route-guard-plugins-local-only.test.ts",
"tests/unit/route-guard-private-lan.test.ts",
"tests/unit/route-guard-provider-login-local-only.test.ts",
"tests/unit/router-strategies.test.ts",
"tests/unit/service-combo-metrics.test.ts",
"tests/unit/services-branch-hardening.test.ts",
"tests/unit/services/combo-metrics-memory.test.ts",
"tests/unit/settings/authz-bypass.test.ts",
"tests/unit/skip-provider-breaker-consumer-2743.test.ts",
"tests/unit/sse-auth.test.ts",
"tests/unit/strict-random-deck.test.ts",
"tests/unit/system-role-extraction.test.ts",
"tests/unit/t23-t24-fallback-resilience.test.ts",
"tests/unit/tag-routing.test.ts",
"tests/unit/thundering-herd.test.ts",
"tests/unit/token-refresh-race-comprehensive.test.ts",
"tests/unit/token-refresh-service.test.ts",
"tests/unit/tools-filter-anthropic-format.test.ts",
"tests/unit/usage-service-hardening.test.ts",
"tests/unit/validate-response-quality.test.ts"
],
"nodeArgs": [
"--import",
"tsx",
"--import",
"./open-sse/utils/setupPolyfill.ts",
"--test-reporter=tap",
"-r",
"{{hookFile}}",
"{{testFile}}"
]
},
"mutate": [
"open-sse/handlers/chatCore.ts",
@@ -33,15 +191,25 @@
"open-sse/utils/publicCreds.ts",
"src/shared/utils/circuitBreaker.ts"
],
"_ignorePatterns_comment": [
"ignorePatterns = files NOT copied into the Stryker sandbox. It does NOT scope",
"what gets mutated (that is the `mutate` array above). The test files MUST be",
"copied so the tap-runner can find covering tests, so DO NOT ignore tests/ here.",
"We only exclude heavy, mutation-irrelevant trees to keep sandbox creation fast:",
"build output, coverage, the huge docs/i18n translation tree, and other worktrees."
],
"ignorePatterns": [
"node_modules",
".next",
"dist",
"dist-electron",
".build",
"coverage",
"tests",
"**/*.test.ts",
"**/*.spec.ts",
"**/__tests__/**"
"playwright-report",
"test-results",
"reports",
"docs/i18n",
".worktrees",
".stryker-tmp"
],
"reporters": ["progress", "html", "json"],
"htmlReporter": {
@@ -53,7 +221,7 @@
"coverageAnalysis": "perTest",
"timeoutMS": 60000,
"timeoutFactor": 2.5,
"concurrency": 2,
"concurrency": 1,
"disableTypeChecks": true,
"checkers": [],
"thresholds": {
@@ -62,5 +230,19 @@
"break": null
},
"tempDirName": ".stryker-tmp",
"cleanTempDir": true
"cleanTempDir": true,
"_tapTestFiles_comment": [
"tap.testFiles is the explicit set of node:test files that cover the 8 mutated",
"modules (union of files importing any of them), MINUS a few timing/heap/streaming-",
"sensitive integration tests that can flake under Stryker concurrent runners and",
"would break the required all-green baseline dry-run (e.g. body-timeout-integration,",
"heap-pressure, sse-heartbeat-integration, *-stream-readiness, chatcore-memory-pressure).",
"It is enumerated (not a broad glob) so the Stryker dry-run stays tractable for the",
"nightly budget — a glob over the full ~1300-file unit suite would make the per-test",
"dry-run take hours. coverageAnalysis:perTest then narrows which files run per mutant.",
"Regenerate the base union after adding/renaming covering tests, then re-prune flaky ones:",
" grep -rlE \"circuitBreaker|publicCreds|accountFallback|routeGuard|services/auth|chatCore|services/combo|utils/error|public-client|account-fallback|route-guard|circuit-breaker\" tests/unit --include=\"*.test.ts\" | sort -u"
],
"dryRunTimeoutMinutes": 30,
"_concurrency_comment": "concurrency=1: the covering node:test files share SQLite/module state; running multiple files concurrently in the Stryker sandbox causes cross-file state races that fail the all-green baseline dry-run. Single-runner trades speed for a deterministic baseline (acceptable for a nightly)."
}