fix(ci): derive oasdiff base-ref from package version + flag mutation-toolchain regression (#4134)

Quality-Gate v2 Fase 9 maintenance (two findings from the Onda 0-1 follow-up):

- check-openapi-breaking.mjs: DEFAULT_BASE_REF was hard-coded "origin/release/
  v3.8.27" and drifted into the v3.8.29 cycle — LOCAL runs of the oasdiff
  breaking-change gate diffed the PR spec against the wrong release (CI passes
  BASE_REF=${{ github.base_ref }} and was unaffected). Derive it from
  package.json via releaseBranchForVersion() so it tracks every release bump
  automatically; literal fallback bumped to v3.8.29. +3 TDD cases.

- docs/ops/MUTATION_GATE_STRYKER_REGRESSION.md: NEW toolchain regression —
  `npx stryker run` now fails at config validation under Stryker 9.6.1 + ajv
  8.18.0 with a false "concurrency must match pattern" (reproduced with the
  UNMODIFIED stryker.conf.json, and with concurrency removed). The project
  config is valid — a standalone ajv 8.18.0 (Stryker's own dep) accepts it — so
  it's an OptionsValidator regression, not our config. This is NOT a re-opening
  of the Task 12 spike (perTest/killedBy was settled GO in v3.8.27; the nightly
  ran as recently as v3.8.28, timing out on budget). The nightly now can't even
  start; clearing this (a dependency pin needing a non-shared reinstall) is the
  prerequisite to producing scores again. Doc records evidence + fix direction.
This commit is contained in:
Diego Rodrigues de Sa e Souza
2026-06-17 23:46:45 -03:00
committed by GitHub
parent 9f14c1294a
commit ee1d0fb27e
3 changed files with 141 additions and 4 deletions

View File

@@ -0,0 +1,84 @@
---
title: "Mutation gate — Stryker 9.6.1 OptionsValidator regression (npx stryker run fails at config validation)"
---
# Mutation gate is currently non-functional (Stryker 9.6.1 + ajv 8.18.0)
> **Discovered:** 2026-06-17 (v3.8.29 cycle), re-running a scoped Stryker probe.
> `npx stryker run` now fails at **config validation**, before any sandbox/mutant work.
>
> **This is a NEW toolchain regression, not a re-opening of the spike.** The Onda-1
> Task 12 per-test (`killedBy`) attribution question was already settled **GO** in
> v3.8.27, and the nightly mutation run executed as recently as the **v3.8.28** cycle
> (it reached the run phase and timed out on **budget** — the standing Onda-2 blocker,
> tracked separately and addressed by the god-file splits in Onda 3). Something in the
> shared `node_modules` changed since that last nightly so that the validator now
> rejects the (valid) config — stopping the nightly from even starting.
## Symptom
`npx stryker run` (and therefore `npm run test:mutation` and the
`.github/workflows/nightly-mutation.yml` job) fails immediately with:
```
ERROR OptionsValidator Config option "concurrency" must match pattern "^(100|[1-9]?[0-9])%$"
ERROR Stryker Please correct this configuration error and try again.
at OptionsValidator.schemaValidate (.../@stryker-mutator/core/dist/src/config/options-validator.js:161)
```
Reproduced with the **unmodified** `stryker.conf.json` (`npx stryker run -c stryker.conf.json --dryRunOnly`)
and with a copy that **removes** `concurrency` entirely — the error persists either way.
## Why it is a false positive (the project config is valid)
- `stryker.conf.json` sets `"concurrency": 4`. The Stryker JSON schema
(`node_modules/@stryker-mutator/api/dist/schema/stryker-core.json`) defines
`concurrency` as `oneOf: [ {type:number, minimum:1}, {type:string, pattern:"^(100|[1-9]?[0-9])%$"} ]`
with **`examples: [4, "50%", "100%"]`** — so a number `4` is explicitly valid.
- Validating the config (and a bare `{concurrency: 4}`) against that schema with a
**standalone ajv 8.18.0** — the exact version Stryker depends on (`~8.18.0`, no nested
copy, no `overrides`) — and even with Stryker's **exact** ajv options
(`useDefaults, allErrors, jsPropertySyntax, verbose, logger:false, strict:false`)
returns **valid**, no concurrency error.
- A percentage string (`"30%"`, which _does_ match the pattern) is **also** rejected,
and the error appears even when `concurrency` is omitted. So the message is
**mis-attributed**: Stryker's compiled `validateFn` fails on the merged full-options
object and reports it against `concurrency`.
## Likely root cause
The interaction is inside Stryker 9.6.1's `OptionsValidator`, not the project config.
`jsPropertySyntax: true` (line 18 of `options-validator.js`) is an **ajv 6** option that
was **removed in ajv 7+**; under ajv 8.18.0 it is a no-op, and the error-path translation
in `describeErrors` can mislabel the genuinely-failing field as `concurrency`. The true
offending value is somewhere in the default-filled options object — not surfaced because
`describeErrors` collapses the ajv `oneOf` errors to a single line.
## Impact
- The mutation gate is **nightly + advisory** (not PR-blocking), so this **reds no PR**.
- But the nightly now **fails before it starts** — strictly worse than the prior state,
where it reached the run phase and only timed out on budget. It produces **zero scores**:
1. **P0 #1** — promoting `mutationScore` to a ratchet — has no fresh nightly values.
2. **Onda 2** (radiography + mutation-proved pruning) needs ≥1 complete run; this
config-validation regression must be cleared first, then the **budget** blocker
(god-files dominating ~⅔ of the 15k mutants) still remains, per the Onda-3 splits.
## Recommended fix (focused follow-up — NOT in this PR)
Out of scope for the oasdiff base-ref drift fix this PR carries, and it needs a
dependency change + reinstall that must happen on a **non-shared** `node_modules`
(the dev worktrees here symlink a shared `node_modules`; never `npm install` into it).
1. Reproduce on a clean install: `npm ci` in an isolated checkout, then
`npx stryker run -c stryker.conf.json --dryRunOnly`.
2. Surface the real failing field: temporarily log `validateFn.errors` (verbose) in
`options-validator.js`, or compile the schema with a plain ajv 8.18.0 and validate the
**full default options** object (not just the partial config) to see which field+value fails.
3. Most probable remedies, in order of preference:
- Pin `@stryker-mutator/core` + `@stryker-mutator/tap-runner` to the last version whose
`OptionsValidator` validates cleanly against ajv 8.18.0 (bisect 9.6.1 ← down).
- Or add a tested `overrides` for the ajv version Stryker's validator actually expects.
4. Verify by getting `npx stryker run --dryRunOnly` to exit 0 again (the conf comment claims
this passed on 2026-06-15/17 — confirm what changed in `node_modules` since), **then** run
the Task 12 spike to record the real `killedBy` verdict.

View File

@@ -29,13 +29,14 @@
// neste repo: report → ratchet → block).
//
// Base ref:
// • CI passa BASE_REF=${{ github.base_ref }} (ex.: "release/v3.8.27").
// • Local: default origin/release/v3.8.27.
// • CI passa BASE_REF=${{ github.base_ref }} (ex.: "release/vX.Y.Z").
// • Local: default derivado da versão do package.json (releaseBranchForVersion),
// ex.: package 3.8.29 → "origin/release/v3.8.29" — nunca fica stale entre ciclos.
// A spec base é extraída com `git show <BASE_REF>:docs/reference/openapi.yaml`.
//
// Uso:
// node scripts/check/check-openapi-breaking.mjs
// BASE_REF=origin/release/v3.8.27 node scripts/check/check-openapi-breaking.mjs
// BASE_REF=origin/release/vX.Y.Z node scripts/check/check-openapi-breaking.mjs
// node scripts/check/check-openapi-breaking.mjs --json # imprime JSON bruto do oasdiff
// node scripts/check/check-openapi-breaking.mjs --quiet # suprime logs de diagnóstico
// node scripts/check/check-openapi-breaking.mjs --ratchet # falha (exit 1) numa regressão
@@ -53,7 +54,35 @@ const RATCHET = process.argv.includes("--ratchet");
const SPEC_REL = "docs/reference/openapi.yaml";
const SPEC_PATH = path.join(ROOT, "docs", "reference", "openapi.yaml");
const DEFAULT_BASE_REF = "origin/release/v3.8.27";
/**
* Deriva o branch base de release a partir de uma versão semver
* (ex.: "3.8.29" → "origin/release/v3.8.29"). Mantém o default sincronizado com
* o ciclo de release SEM hard-code: o version-bump atualiza package.json a cada
* ciclo, então o default nunca fica stale (era "origin/release/v3.8.27" fixo).
* Ignora sufixos de prerelease/build (ex.: "3.8.29-dev.2" → v3.8.29).
*
* @param {string|null|undefined} version
* @returns {string|null} branch base (sem `origin/` ausente) ou null se não-semver
*/
export function releaseBranchForVersion(version) {
const m = String(version ?? "")
.trim()
.match(/^(\d+)\.(\d+)\.(\d+)/);
return m ? `origin/release/v${m[1]}.${m[2]}.${m[3]}` : null;
}
function readPackageVersion() {
try {
return JSON.parse(fs.readFileSync(path.join(ROOT, "package.json"), "utf8")).version;
} catch {
return null;
}
}
// CI sempre passa BASE_REF=${{ github.base_ref }} e vence; este default só vale
// para runs locais. Derivado da versão para não re-driftar a cada release.
const DEFAULT_BASE_REF = releaseBranchForVersion(readPackageVersion()) || "origin/release/v3.8.29";
const BASELINE_PATH = path.join(ROOT, "config/quality/quality-baseline.json");
// ---------------------------------------------------------------------------

View File

@@ -17,6 +17,7 @@ import { spawnSync } from "node:child_process";
import {
evaluateOpenapiRatchet,
readBaselineOpenapiValue,
releaseBranchForVersion,
// @ts-expect-error — .mjs helper has no type declarations; runtime shape is known.
} from "../../scripts/check/check-openapi-breaking.mjs";
@@ -26,11 +27,34 @@ const evaluate = evaluateOpenapiRatchet as (args: {
baseline: number | null;
}) => RatchetVerdict;
const readBaseline = readBaselineOpenapiValue as (p?: string) => number | null;
const releaseBranch = releaseBranchForVersion as (v: string | null | undefined) => string | null;
const SCRIPT_PATH = fileURLToPath(
new URL("../../scripts/check/check-openapi-breaking.mjs", import.meta.url)
);
// ---------------------------------------------------------------------------
// releaseBranchForVersion — the default base ref derives from the package
// version so it never goes stale across release cycles (was a hard-coded
// "origin/release/v3.8.27" that drifted into the v3.8.29 cycle).
// ---------------------------------------------------------------------------
test("releaseBranchForVersion: a clean semver derives the matching release branch", () => {
assert.equal(releaseBranch("3.8.29"), "origin/release/v3.8.29");
});
test("releaseBranchForVersion: a prerelease/build suffix is ignored", () => {
assert.equal(releaseBranch("3.8.29-dev.2"), "origin/release/v3.8.29");
assert.equal(releaseBranch("10.0.0+build.7"), "origin/release/v10.0.0");
});
test("releaseBranchForVersion: a non-semver value yields null (caller falls back)", () => {
assert.equal(releaseBranch(""), null);
assert.equal(releaseBranch(null), null);
assert.equal(releaseBranch(undefined), null);
assert.equal(releaseBranch("not-a-version"), null);
});
// ---------------------------------------------------------------------------
// evaluateOpenapiRatchet — the three contract cases from the plan
// ---------------------------------------------------------------------------