pjandro/OmniRoute
1
0
Fork 0
mirror of https://github.com/diegosouzapw/OmniRoute.git synced 2026-08-05 06:42:12 +03:00
Code Issues Packages Projects Releases Wiki Activity
Files
a7ff81bb95bba52eebe1ed09e215cc2e2e64a677
OmniRoute/stryker.conf.json
Diego Rodrigues de Sa e Souza 240f466f54 test(infra): isolate DATA_DIR per test process; raise Stryker concurrency 1→4 (#4078) 
* test(infra): isolate DATA_DIR per test process; raise Stryker concurrency 1→4

Every test process resolved DATA_DIR to the same default (~/.omniroute) when the env
var was unset (src/lib/dataPaths.ts::resolveDataDir), so concurrent test files opened
the SAME on-disk storage.sqlite. node:test spawns a process per file and Stryker spawns
one per sandbox, so this shared file caused cross-file state races:
- SQLite lock contention that hung `npm run test:unit` under high --test-concurrency
  (the ~95-min local hang), and
- the non-deterministic baseline that forced stryker.conf.json to concurrency: 1, which
  in turn could not finish the ~15k-mutant run inside the nightly timeout (the cancelled
  2026-06-16/17 nightly-mutation runs) — blocking Quality Gate v2 / Fase 9 Onda 2.

open-sse/utils/setupPolyfill.ts could NOT host the fix: it is imported by production
(bin/omniroute.mjs, proxyFetch.ts, proxyDispatcher.ts), where redirecting DATA_DIR would
point the live SQLite DB at a throwaway temp dir. So this adds a TEST-ONLY
tests/_setup/isolateDataDir.ts that gives each process its own temp DATA_DIR when none is
set (tests that set DATA_DIR explicitly still win), wired via --import into the test,
mutation and CI invocations.

Verified:
- Stryker dry-run A/B at concurrency=4: FAILS without the isolation import
  (account-fallback-service tap exit 9, a cross-file race) and PASSES with it.
- Full `npm run test:unit` green with isolation (0 fail; a one-off
  chatcore-translation-paths timeout flake did not reproduce and passes 3/3 isolated)
  and noticeably faster — the DB lock contention is gone.
- New tests/unit/isolate-datadir.test.ts guards the contract (unique temp DATA_DIR when
  unset; explicit DATA_DIR respected).

Wired the --import into: package.json (13 test scripts), stryker.conf.json (tap.nodeArgs
+ concurrency 1→4), .github/workflows/quality.yml (TIA step), ci.yml (the 5
unit/coverage/integration commands), and bumped nightly-mutation.yml timeout 120→180 for
the first cold run before the incremental cache is seeded.

* ci(quality): run the TIA gate at CI concurrency (4) to stop oversubscription flakes

The TIA "Impacted unit tests" step (made blocking in #4069) ran its fail-safe via
`npm run test:unit` — concurrency=20, tuned for multi-core dev machines. On a 4-vCPU CI
runner that is 5x oversubscribed, so timing-sensitive tests flake under the load (e.g.
`db-backup-extended` "The database connection is not open", `chatcore-translation-paths`
upstream-timeout). That intermittently fails a blocking gate on legitimate PRs — exactly
what surfaced on the DATA_DIR-isolation PR, whose package.json/workflow changes trip the
__RUN_ALL__ fail-safe.

Run both the impacted set and the fail-safe at --test-concurrency=4, matching the stable
ci.yml unit job. Adds a `test:unit:ci` script (test:unit at concurrency=4). The DATA_DIR
isolation in this PR keeps the parallel run race-free, so the only change here is matching
the runner's core count. Verified locally: db-backup-extended passes 8/8 in isolation
(5 with isolation, 3 without).
2026-06-17 13:46:28 -03:00

253 lines
12 KiB
JSON
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
{
"$schema": "https://stryker-mutator.io/schemas/stryker-schema.json",
"_comment": [
"Mutation testing for the ~8 critical modules (Task 11 — Fase 7).",
"NIGHTLY ONLY — DO NOT run on every PR. Mutation testing is expensive:",
" - Each mutant requires a full test suite execution.",
" - The 8 modules produce ~200–500 mutants; est. 30–90 min per run.",
" - Wired to the nightly CI workflow (.github/workflows/nightly-mutation.yml),",
" NOT to the 'lint' / 'quality-gate' PR jobs.",
"",
"TEST RUNNER — @stryker-mutator/tap-runner (NOT vitest):",
" The 8 critical modules are covered by node:test files in tests/unit/",
" (run via `node --import tsx --test`), NOT by vitest. The vitest config",
" only includes a small set of .test.tsx + open-sse/**/__tests__ files, so",
" the vitest-runner would find ZERO covering tests for these modules. The",
" tap-runner spawns each node:test file individually and parses its TAP",
" output, which matches how this repo actually exercises the modules.",
" Each test file is loaded with tsx (TypeScript/ESM) + the project polyfill,",
" the same way `npm run test:unit` runs.",
"",
"Install before running (not bundled to avoid E2E / CI bloat):",
" npm install --save-dev @stryker-mutator/core @stryker-mutator/tap-runner",
"",
"Run manually:",
" npm run test:mutation # full run (slow — nightly budget)",
" npx stryker run --dryRunOnly # validate the baseline only (no mutants)",
" (single-module probe: temporarily narrow `mutate` + `tap.testFiles` in this file)",
"",
"VALIDATED 2026-06-15: `npx stryker run --dryRunOnly` exits 0 — all 129 covering",
"test files run green in the Stryker sandbox and the perTest coverage map builds for",
"all 8 instrumented modules (15k+ mutants). The baseline dry-run takes ~20 min with",
"concurrency=1; the full mutation phase runs on top (advisory, capped by the workflow",
"timeout). So the nightly produces REAL mutation scores for the 8 modules.",
"",
"Mutation score per module → quality-baseline.json key 'mutationScore.<module>'",
"Direction: up (score can only improve; ratchet blocks drops — wired in a later INT phase)."
],
"packageManager": "npm",
"incremental": true,
"incrementalFile": "reports/mutation/stryker-incremental.json",
"testRunner": "tap",
"plugins": ["@stryker-mutator/tap-runner"],
"tap": {
"testFiles": [
"tests/unit/account-fallback-anthropic-quota.test.ts",
"tests/unit/account-fallback-route-restriction-403.test.ts",
"tests/unit/account-fallback-service.test.ts",
"tests/unit/api-key-rotator-health.test.ts",
"tests/unit/appearance-widget-settings-schema.test.ts",
"tests/unit/auth-clear-account-error.test.ts",
"tests/unit/auth-disable-cooling-2997.test.ts",
"tests/unit/auth-extract-api-key.test.ts",
"tests/unit/auth-noauth-fallback-loop-3061.test.ts",
"tests/unit/auth-ollama-cloud-per-model-403-3027.test.ts",
"tests/unit/auth-opencode-zen-noauth-fallback.test.ts",
"tests/unit/auth-terminal-status.test.ts",
"tests/unit/authz/routeGuard.test.ts",
"tests/unit/auto-combo-context-advertising.test.ts",
"tests/unit/auto-combo-engine.test.ts",
"tests/unit/auto-combo-scoring-clamp.test.ts",
"tests/unit/build/check-circular-deps.test.ts",
"tests/unit/cache-sweeps.test.ts",
"tests/unit/cc-compatible-provider.test.ts",
"tests/unit/chat-context-relay.test.ts",
"tests/unit/chat-cooldown-aware-retry.test.ts",
"tests/unit/chat-helpers.test.ts",
"tests/unit/chat-route-coverage.test.ts",
"tests/unit/chat-route-edge-cases.test.ts",
"tests/unit/chatcore-compression-integration.test.ts",
"tests/unit/chatcore-extracted-modules-3821.test.ts",
"tests/unit/chatcore-imports-cleanly.test.ts",
"tests/unit/chatcore-sanitization.test.ts",
"tests/unit/chatcore-strip-stale-headers.test.ts",
"tests/unit/chatcore-translation-paths.test.ts",
"tests/unit/check-error-helper.test.ts",
"tests/unit/check-route-guard-membership.test.ts",
"tests/unit/check-test-discovery.test.ts",
"tests/unit/circuit-breaker-failure-kind.test.ts",
"tests/unit/claude-code-parity.test.ts",
"tests/unit/claude-effort-suffix-strip.test.ts",
"tests/unit/claude-oauth-provider.test.ts",
"tests/unit/claude-passthrough-stream-boolean.test.ts",
"tests/unit/claude-passthrough-thinking-2454.test.ts",
"tests/unit/cli-simulate.test.ts",
"tests/unit/codex-failover.test.ts",
"tests/unit/codex-stream-false.test.ts",
"tests/unit/collect-metrics-module-coverage.test.ts",
"tests/unit/combo-499-abort.test.ts",
"tests/unit/combo-auto-candidate-expansion.test.ts",
"tests/unit/combo-cache-invalidation.test.ts",
"tests/unit/combo-config.test.ts",
"tests/unit/combo-context-relay.test.ts",
"tests/unit/combo-health-autopilot.test.ts",
"tests/unit/combo-health-dashboard.test.ts",
"tests/unit/combo-health-route.test.ts",
"tests/unit/combo-hedging.test.ts",
"tests/unit/combo-max-depth-config.test.ts",
"tests/unit/combo-omnimodel-tag-stripping.test.ts",
"tests/unit/combo-prescreen.test.ts",
"tests/unit/combo-provider-cooldown.test.ts",
"tests/unit/combo-provider-diversity-wiring.test.ts",
"tests/unit/combo-quality-validator-reasoning.test.ts",
"tests/unit/combo-quota-soft-penalty.test.ts",
"tests/unit/combo-round-robin-streaming-lock-3811.test.ts",
"tests/unit/combo-routing-engine.test.ts",
"tests/unit/combo-scoring-inspector.test.ts",
"tests/unit/combo-sessionless-pin-3825.test.ts",
"tests/unit/combo-strategies.test.ts",
"tests/unit/combo-strategy-fallbacks.test.ts",
"tests/unit/combo-streaming-empty-content-failover.test.ts",
"tests/unit/combo-target-defensive-modelstr.test.ts",
"tests/unit/complexity-aware-scoring-wiring.test.ts",
"tests/unit/context-pinning-tool-calls.test.ts",
"tests/unit/correctness/combo.property.test.ts",
"tests/unit/correctness/sanitizers.property.test.ts",
"tests/unit/custom-model-target-format.test.ts",
"tests/unit/db-reset-module-state.test.ts",
"tests/unit/domain-persistence.test.ts",
"tests/unit/embeddings-auth.test.ts",
"tests/unit/error-classification.test.ts",
"tests/unit/error-message-sanitization.test.ts",
"tests/unit/executor-antigravity.test.ts",
"tests/unit/executor-web-cookie-sweep.test.ts",
"tests/unit/gemini-web-missing-browser-3516.test.ts",
"tests/unit/guardrails-api-3496.test.ts",
"tests/unit/memory-embedding-remote.test.ts",
"tests/unit/memory-embedding-transformers.test.ts",
"tests/unit/model-cooldowns-route-auth.test.ts",
"tests/unit/model-cooldowns-route.test.ts",
"tests/unit/model-lockout-decay.test.ts",
"tests/unit/oauth-providers-config.test.ts",
"tests/unit/oauth-redirect-uri-mismatch.test.ts",
"tests/unit/observability-fase04.test.ts",
"tests/unit/observability-payloads.test.ts",
"tests/unit/plan3-p0.test.ts",
"tests/unit/plugin-sandbox-permissions.test.ts",
"tests/unit/plugins-route-error-sanitization.test.ts",
"tests/unit/provider-error-rules.test.ts",
"tests/unit/provider-health-autopilot.test.ts",
"tests/unit/provider-health-matrix.test.ts",
"tests/unit/provider-request-failure-pipeline.test.ts",
"tests/unit/public-client-ids-3493.test.ts",
"tests/unit/publicCreds.test.ts",
"tests/unit/qoder-oauth-config.test.ts",
"tests/unit/quota-groups-route.test.ts",
"tests/unit/quota-key-models-route.test.ts",
"tests/unit/quota-policy-generalization.test.ts",
"tests/unit/quota-pool-log-route.test.ts",
"tests/unit/quota-streaming-consumption-usd.test.ts",
"tests/unit/rate-limit-enhanced.test.ts",
"tests/unit/rate-limit-manager.test.ts",
"tests/unit/responses-handler.test.ts",
"tests/unit/route-explainability.test.ts",
"tests/unit/route-guard-plugins-local-only.test.ts",
"tests/unit/route-guard-private-lan.test.ts",
"tests/unit/route-guard-provider-login-local-only.test.ts",
"tests/unit/router-strategies.test.ts",
"tests/unit/service-combo-metrics.test.ts",
"tests/unit/services-branch-hardening.test.ts",
"tests/unit/services/combo-metrics-memory.test.ts",
"tests/unit/settings/authz-bypass.test.ts",
"tests/unit/skip-provider-breaker-consumer-2743.test.ts",
"tests/unit/sse-auth.test.ts",
"tests/unit/strict-random-deck.test.ts",
"tests/unit/system-role-extraction.test.ts",
"tests/unit/t23-t24-fallback-resilience.test.ts",
"tests/unit/tag-routing.test.ts",
"tests/unit/thundering-herd.test.ts",
"tests/unit/token-refresh-race-comprehensive.test.ts",
"tests/unit/token-refresh-service.test.ts",
"tests/unit/tools-filter-anthropic-format.test.ts",
"tests/unit/usage-service-hardening.test.ts",
"tests/unit/validate-response-quality.test.ts"
],
"nodeArgs": [
"--import",
"tsx",
"--import",
"./open-sse/utils/setupPolyfill.ts",
"--import",
"./tests/_setup/isolateDataDir.ts",
"--test-reporter=tap",
"-r",
"{{hookFile}}",
"{{testFile}}"
]
},
"mutate": [
"open-sse/handlers/chatCore.ts",
"open-sse/services/combo.ts",
"open-sse/services/accountFallback.ts",
"src/sse/services/auth.ts",
"src/server/authz/routeGuard.ts",
"open-sse/utils/error.ts",
"open-sse/utils/publicCreds.ts",
"src/shared/utils/circuitBreaker.ts"
],
"_ignorePatterns_comment": [
"ignorePatterns = files NOT copied into the Stryker sandbox. It does NOT scope",
"what gets mutated (that is the `mutate` array above). The test files MUST be",
"copied so the tap-runner can find covering tests, so DO NOT ignore tests/ here.",
"We only exclude heavy, mutation-irrelevant trees to keep sandbox creation fast:",
"build output, coverage, the huge docs/i18n translation tree, and other worktrees."
],
"ignorePatterns": [
".next",
"dist",
"dist-electron",
".build",
"coverage",
"playwright-report",
"test-results",
"reports",
"docs/i18n",
".worktrees",
".stryker-tmp"
],
"reporters": ["progress", "html", "json"],
"htmlReporter": {
"fileName": "reports/mutation/mutation.html"
},
"jsonReporter": {
"fileName": "reports/mutation/mutation.json"
},
"coverageAnalysis": "perTest",
"timeoutMS": 60000,
"timeoutFactor": 2.5,
"concurrency": 4,
"disableTypeChecks": true,
"checkers": [],
"thresholds": {
"high": 70,
"low": 50,
"break": null
},
"tempDirName": ".stryker-tmp",
"cleanTempDir": true,
"_tapTestFiles_comment": [
"tap.testFiles is the explicit set of node:test files that cover the 8 mutated",
"modules (union of files importing any of them), MINUS a few timing/heap/streaming-",
"sensitive integration tests that can flake under Stryker concurrent runners and",
"would break the required all-green baseline dry-run (e.g. body-timeout-integration,",
"heap-pressure, sse-heartbeat-integration, *-stream-readiness, chatcore-memory-pressure).",
"It is enumerated (not a broad glob) so the Stryker dry-run stays tractable for the",
"nightly budget — a glob over the full ~1300-file unit suite would make the per-test",
"dry-run take hours. coverageAnalysis:perTest then narrows which files run per mutant.",
"Regenerate the base union after adding/renaming covering tests, then re-prune flaky ones:",
" grep -rlE \"circuitBreaker|publicCreds|accountFallback|routeGuard|services/auth|chatCore|services/combo|utils/error|public-client|account-fallback|route-guard|circuit-breaker\" tests/unit --include=\"*.test.ts\" | sort -u"
],
"dryRunTimeoutMinutes": 30,
"_concurrency_comment": "concurrency=4 (was 1): the covering node:test files used to share SQLite/module state via the default DATA_DIR (~/.omniroute), so running them concurrently in the Stryker sandbox caused cross-file races that failed the all-green baseline. tap.nodeArgs now imports ./tests/_setup/isolateDataDir.ts, which gives each spawned test process its own temp DATA_DIR — eliminating the shared on-disk DB, so concurrency>1 is deterministic. A/B verified 2026-06-17: dry-run at concurrency=4 fails WITHOUT the isolation import (account-fallback-service tap exit 9) and passes WITH it. Raise further only if the runner has spare cores."
}
Reference in New Issue View Git Blame Copy Permalink