mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-08-02 05:12:11 +03:00
fix(ci): derive oasdiff base-ref from package version + flag mutation-toolchain regression (#4134)
Quality-Gate v2 Fase 9 maintenance (two findings from the Onda 0-1 follow-up):
- check-openapi-breaking.mjs: DEFAULT_BASE_REF was hard-coded "origin/release/
v3.8.27" and drifted into the v3.8.29 cycle — LOCAL runs of the oasdiff
breaking-change gate diffed the PR spec against the wrong release (CI passes
BASE_REF=${{ github.base_ref }} and was unaffected). Derive it from
package.json via releaseBranchForVersion() so it tracks every release bump
automatically; literal fallback bumped to v3.8.29. +3 TDD cases.
- docs/ops/MUTATION_GATE_STRYKER_REGRESSION.md: NEW toolchain regression —
`npx stryker run` now fails at config validation under Stryker 9.6.1 + ajv
8.18.0 with a false "concurrency must match pattern" (reproduced with the
UNMODIFIED stryker.conf.json, and with concurrency removed). The project
config is valid — a standalone ajv 8.18.0 (Stryker's own dep) accepts it — so
it's an OptionsValidator regression, not our config. This is NOT a re-opening
of the Task 12 spike (perTest/killedBy was settled GO in v3.8.27; the nightly
ran as recently as v3.8.28, timing out on budget). The nightly now can't even
start; clearing this (a dependency pin needing a non-shared reinstall) is the
prerequisite to producing scores again. Doc records evidence + fix direction.
This commit is contained in:
committed by
GitHub
parent
9f14c1294a
commit
ee1d0fb27e
84
docs/ops/MUTATION_GATE_STRYKER_REGRESSION.md
Normal file
84
docs/ops/MUTATION_GATE_STRYKER_REGRESSION.md
Normal file
@@ -0,0 +1,84 @@
|
||||
---
|
||||
title: "Mutation gate — Stryker 9.6.1 OptionsValidator regression (npx stryker run fails at config validation)"
|
||||
---
|
||||
|
||||
# Mutation gate is currently non-functional (Stryker 9.6.1 + ajv 8.18.0)
|
||||
|
||||
> **Discovered:** 2026-06-17 (v3.8.29 cycle), re-running a scoped Stryker probe.
|
||||
> `npx stryker run` now fails at **config validation**, before any sandbox/mutant work.
|
||||
>
|
||||
> **This is a NEW toolchain regression, not a re-opening of the spike.** The Onda-1
|
||||
> Task 12 per-test (`killedBy`) attribution question was already settled **GO** in
|
||||
> v3.8.27, and the nightly mutation run executed as recently as the **v3.8.28** cycle
|
||||
> (it reached the run phase and timed out on **budget** — the standing Onda-2 blocker,
|
||||
> tracked separately and addressed by the god-file splits in Onda 3). Something in the
|
||||
> shared `node_modules` changed since that last nightly so that the validator now
|
||||
> rejects the (valid) config — stopping the nightly from even starting.
|
||||
|
||||
## Symptom
|
||||
|
||||
`npx stryker run` (and therefore `npm run test:mutation` and the
|
||||
`.github/workflows/nightly-mutation.yml` job) fails immediately with:
|
||||
|
||||
```
|
||||
ERROR OptionsValidator Config option "concurrency" must match pattern "^(100|[1-9]?[0-9])%$"
|
||||
ERROR Stryker Please correct this configuration error and try again.
|
||||
at OptionsValidator.schemaValidate (.../@stryker-mutator/core/dist/src/config/options-validator.js:161)
|
||||
```
|
||||
|
||||
Reproduced with the **unmodified** `stryker.conf.json` (`npx stryker run -c stryker.conf.json --dryRunOnly`)
|
||||
and with a copy that **removes** `concurrency` entirely — the error persists either way.
|
||||
|
||||
## Why it is a false positive (the project config is valid)
|
||||
|
||||
- `stryker.conf.json` sets `"concurrency": 4`. The Stryker JSON schema
|
||||
(`node_modules/@stryker-mutator/api/dist/schema/stryker-core.json`) defines
|
||||
`concurrency` as `oneOf: [ {type:number, minimum:1}, {type:string, pattern:"^(100|[1-9]?[0-9])%$"} ]`
|
||||
with **`examples: [4, "50%", "100%"]`** — so a number `4` is explicitly valid.
|
||||
- Validating the config (and a bare `{concurrency: 4}`) against that schema with a
|
||||
**standalone ajv 8.18.0** — the exact version Stryker depends on (`~8.18.0`, no nested
|
||||
copy, no `overrides`) — and even with Stryker's **exact** ajv options
|
||||
(`useDefaults, allErrors, jsPropertySyntax, verbose, logger:false, strict:false`)
|
||||
returns **valid**, no concurrency error.
|
||||
- A percentage string (`"30%"`, which _does_ match the pattern) is **also** rejected,
|
||||
and the error appears even when `concurrency` is omitted. So the message is
|
||||
**mis-attributed**: Stryker's compiled `validateFn` fails on the merged full-options
|
||||
object and reports it against `concurrency`.
|
||||
|
||||
## Likely root cause
|
||||
|
||||
The interaction is inside Stryker 9.6.1's `OptionsValidator`, not the project config.
|
||||
`jsPropertySyntax: true` (line 18 of `options-validator.js`) is an **ajv 6** option that
|
||||
was **removed in ajv 7+**; under ajv 8.18.0 it is a no-op, and the error-path translation
|
||||
in `describeErrors` can mislabel the genuinely-failing field as `concurrency`. The true
|
||||
offending value is somewhere in the default-filled options object — not surfaced because
|
||||
`describeErrors` collapses the ajv `oneOf` errors to a single line.
|
||||
|
||||
## Impact
|
||||
|
||||
- The mutation gate is **nightly + advisory** (not PR-blocking), so this **reds no PR**.
|
||||
- But the nightly now **fails before it starts** — strictly worse than the prior state,
|
||||
where it reached the run phase and only timed out on budget. It produces **zero scores**:
|
||||
1. **P0 #1** — promoting `mutationScore` to a ratchet — has no fresh nightly values.
|
||||
2. **Onda 2** (radiography + mutation-proved pruning) needs ≥1 complete run; this
|
||||
config-validation regression must be cleared first, then the **budget** blocker
|
||||
(god-files dominating ~⅔ of the 15k mutants) still remains, per the Onda-3 splits.
|
||||
|
||||
## Recommended fix (focused follow-up — NOT in this PR)
|
||||
|
||||
Out of scope for the oasdiff base-ref drift fix this PR carries, and it needs a
|
||||
dependency change + reinstall that must happen on a **non-shared** `node_modules`
|
||||
(the dev worktrees here symlink a shared `node_modules`; never `npm install` into it).
|
||||
|
||||
1. Reproduce on a clean install: `npm ci` in an isolated checkout, then
|
||||
`npx stryker run -c stryker.conf.json --dryRunOnly`.
|
||||
2. Surface the real failing field: temporarily log `validateFn.errors` (verbose) in
|
||||
`options-validator.js`, or compile the schema with a plain ajv 8.18.0 and validate the
|
||||
**full default options** object (not just the partial config) to see which field+value fails.
|
||||
3. Most probable remedies, in order of preference:
|
||||
- Pin `@stryker-mutator/core` + `@stryker-mutator/tap-runner` to the last version whose
|
||||
`OptionsValidator` validates cleanly against ajv 8.18.0 (bisect 9.6.1 ← down).
|
||||
- Or add a tested `overrides` for the ajv version Stryker's validator actually expects.
|
||||
4. Verify by getting `npx stryker run --dryRunOnly` to exit 0 again (the conf comment claims
|
||||
this passed on 2026-06-15/17 — confirm what changed in `node_modules` since), **then** run
|
||||
the Task 12 spike to record the real `killedBy` verdict.
|
||||
@@ -29,13 +29,14 @@
|
||||
// neste repo: report → ratchet → block).
|
||||
//
|
||||
// Base ref:
|
||||
// • CI passa BASE_REF=${{ github.base_ref }} (ex.: "release/v3.8.27").
|
||||
// • Local: default origin/release/v3.8.27.
|
||||
// • CI passa BASE_REF=${{ github.base_ref }} (ex.: "release/vX.Y.Z").
|
||||
// • Local: default derivado da versão do package.json (releaseBranchForVersion),
|
||||
// ex.: package 3.8.29 → "origin/release/v3.8.29" — nunca fica stale entre ciclos.
|
||||
// A spec base é extraída com `git show <BASE_REF>:docs/reference/openapi.yaml`.
|
||||
//
|
||||
// Uso:
|
||||
// node scripts/check/check-openapi-breaking.mjs
|
||||
// BASE_REF=origin/release/v3.8.27 node scripts/check/check-openapi-breaking.mjs
|
||||
// BASE_REF=origin/release/vX.Y.Z node scripts/check/check-openapi-breaking.mjs
|
||||
// node scripts/check/check-openapi-breaking.mjs --json # imprime JSON bruto do oasdiff
|
||||
// node scripts/check/check-openapi-breaking.mjs --quiet # suprime logs de diagnóstico
|
||||
// node scripts/check/check-openapi-breaking.mjs --ratchet # falha (exit 1) numa regressão
|
||||
@@ -53,7 +54,35 @@ const RATCHET = process.argv.includes("--ratchet");
|
||||
|
||||
const SPEC_REL = "docs/reference/openapi.yaml";
|
||||
const SPEC_PATH = path.join(ROOT, "docs", "reference", "openapi.yaml");
|
||||
const DEFAULT_BASE_REF = "origin/release/v3.8.27";
|
||||
|
||||
/**
|
||||
* Deriva o branch base de release a partir de uma versão semver
|
||||
* (ex.: "3.8.29" → "origin/release/v3.8.29"). Mantém o default sincronizado com
|
||||
* o ciclo de release SEM hard-code: o version-bump atualiza package.json a cada
|
||||
* ciclo, então o default nunca fica stale (era "origin/release/v3.8.27" fixo).
|
||||
* Ignora sufixos de prerelease/build (ex.: "3.8.29-dev.2" → v3.8.29).
|
||||
*
|
||||
* @param {string|null|undefined} version
|
||||
* @returns {string|null} branch base (sem `origin/` ausente) ou null se não-semver
|
||||
*/
|
||||
export function releaseBranchForVersion(version) {
|
||||
const m = String(version ?? "")
|
||||
.trim()
|
||||
.match(/^(\d+)\.(\d+)\.(\d+)/);
|
||||
return m ? `origin/release/v${m[1]}.${m[2]}.${m[3]}` : null;
|
||||
}
|
||||
|
||||
function readPackageVersion() {
|
||||
try {
|
||||
return JSON.parse(fs.readFileSync(path.join(ROOT, "package.json"), "utf8")).version;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// CI sempre passa BASE_REF=${{ github.base_ref }} e vence; este default só vale
|
||||
// para runs locais. Derivado da versão para não re-driftar a cada release.
|
||||
const DEFAULT_BASE_REF = releaseBranchForVersion(readPackageVersion()) || "origin/release/v3.8.29";
|
||||
const BASELINE_PATH = path.join(ROOT, "config/quality/quality-baseline.json");
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
@@ -17,6 +17,7 @@ import { spawnSync } from "node:child_process";
|
||||
import {
|
||||
evaluateOpenapiRatchet,
|
||||
readBaselineOpenapiValue,
|
||||
releaseBranchForVersion,
|
||||
// @ts-expect-error — .mjs helper has no type declarations; runtime shape is known.
|
||||
} from "../../scripts/check/check-openapi-breaking.mjs";
|
||||
|
||||
@@ -26,11 +27,34 @@ const evaluate = evaluateOpenapiRatchet as (args: {
|
||||
baseline: number | null;
|
||||
}) => RatchetVerdict;
|
||||
const readBaseline = readBaselineOpenapiValue as (p?: string) => number | null;
|
||||
const releaseBranch = releaseBranchForVersion as (v: string | null | undefined) => string | null;
|
||||
|
||||
const SCRIPT_PATH = fileURLToPath(
|
||||
new URL("../../scripts/check/check-openapi-breaking.mjs", import.meta.url)
|
||||
);
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// releaseBranchForVersion — the default base ref derives from the package
|
||||
// version so it never goes stale across release cycles (was a hard-coded
|
||||
// "origin/release/v3.8.27" that drifted into the v3.8.29 cycle).
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
test("releaseBranchForVersion: a clean semver derives the matching release branch", () => {
|
||||
assert.equal(releaseBranch("3.8.29"), "origin/release/v3.8.29");
|
||||
});
|
||||
|
||||
test("releaseBranchForVersion: a prerelease/build suffix is ignored", () => {
|
||||
assert.equal(releaseBranch("3.8.29-dev.2"), "origin/release/v3.8.29");
|
||||
assert.equal(releaseBranch("10.0.0+build.7"), "origin/release/v10.0.0");
|
||||
});
|
||||
|
||||
test("releaseBranchForVersion: a non-semver value yields null (caller falls back)", () => {
|
||||
assert.equal(releaseBranch(""), null);
|
||||
assert.equal(releaseBranch(null), null);
|
||||
assert.equal(releaseBranch(undefined), null);
|
||||
assert.equal(releaseBranch("not-a-version"), null);
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// evaluateOpenapiRatchet — the three contract cases from the plan
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
Reference in New Issue
Block a user