feat(quality): seed per-module mutationScore floors + blocking aggregation ratchet (T3) (#4305)

First full mutation measurement landed (run 27823984918, the split nightly from #4272):
31 modules now have a COVERED mutation score. T3 turns that into an enforced gate.

Seed: 31 `mutationScore.<path>` floors in quality-baseline.json at ~2pt below the measured
score (absorbs run-to-run variance), direction:up, dedicatedGate:true. dedicatedGate means
the generic check-quality-ratchet SKIPS them (check-quality-ratchet.mjs:62) — they are
enforced only by check-mutation-ratchet.mjs. Range: memorySkillsInjection 13.49 (weakest)
to headers 94.29 (strongest); the security/critical floors: auth 52.57, accountFallback
68.38, routeGuard 76.08, circuitBreaker 56.94, error 43.83, publicCreds 59.76.

Gate: a new `mutation-ratchet` job in nightly-mutation.yml runs AFTER all batches
(needs: stryker, if: always()), downloads every mutation report, and ratchets the MERGED
per-module scores with `check-mutation-ratchet --ratchet` (blocking). It must aggregate
because the split batches each emit a PARTIAL view of a file (auth.ts in a1+a2,
accountFallback in b1+b2) — a per-batch ratchet would compare half a file against the
whole-file floor. check-mutation-ratchet unions same-file mutants across reports (#4272).
A module dropping below its floor fails the run; missing reports (upload flake) are skipped.

Verified: ratchet exits 0 on the seeded measurements, exits 1 on a synthetic regression
(auth 33.33 < 52.57), exits 0 advisory without --ratchet. Baseline change is additive
(31 floors + one comment; existing keys untouched). check-mutation-ratchet tests 8/8.
This commit is contained in:
Diego Rodrigues de Sa e Souza
2026-06-19 18:28:06 -03:00
committed by GitHub
parent 23455fdb0a
commit bbc9d1e1c5
2 changed files with 186 additions and 1 deletions

View File

@@ -132,3 +132,32 @@ jobs:
path: reports/mutation/
if-no-files-found: warn
retention-days: 14
# Aggregation gate (T3): each split batch emits a PARTIAL view of a mutated file
# (auth.ts lives in a1+a2, accountFallback in b1+b2), so a PER-BATCH ratchet would
# only ever see half a file vs the whole-file baseline. This job runs AFTER every
# batch, downloads all reports, and ratchets the MERGED per-module scores
# (check-mutation-ratchet UNIONS same-file mutants across reports) against the
# dedicatedGate `mutationScore.*` floors in quality-baseline.json (seeded ~2pt below
# the first full measurement). Blocking: a module dropping below its floor fails the
# run. Missing reports (e.g. an artifact-upload flake) are skipped, never failed.
mutation-ratchet:
name: Mutation score ratchet (blocking)
needs: stryker
if: always()
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false
- uses: actions/setup-node@v6
with:
node-version: "24"
- name: Download all mutation reports
uses: actions/download-artifact@v8
with:
pattern: mutation-report-*
path: reports/all
- name: Ratchet merged per-module mutation scores
run: node scripts/check/check-mutation-ratchet.mjs reports/all/*/mutation.json --ratchet

View File

@@ -141,6 +141,161 @@
"direction": "down",
"dedicatedGate": true,
"_note": "semgrep owasp/secrets findings. ADVISORY until first CI value is frozen, then flip blocking (Fase 9)."
},
"mutationScore.src/sse/services/auth.ts": {
"value": 52.57,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.open-sse/services/accountFallback.ts": {
"value": 68.38,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.src/server/authz/routeGuard.ts": {
"value": 76.08,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.src/shared/utils/circuitBreaker.ts": {
"value": 56.94,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.open-sse/utils/error.ts": {
"value": 43.83,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.open-sse/utils/publicCreds.ts": {
"value": 59.76,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.open-sse/services/combo/autoStrategy.ts": {
"value": 41.33,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.open-sse/services/combo/comboStructure.ts": {
"value": 57.82,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.open-sse/services/combo/validateQuality.ts": {
"value": 61.33,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.open-sse/services/combo/comboPredicates.ts": {
"value": 56.62,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.open-sse/services/combo/rrState.ts": {
"value": 70.88,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.open-sse/services/combo/shadowRouting.ts": {
"value": 48,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.open-sse/services/combo/targetSorters.ts": {
"value": 68.3,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.open-sse/services/combo/comboData.ts": {
"value": 76.94,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.open-sse/services/combo/quotaScoring.ts": {
"value": 39.73,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.open-sse/services/combo/quotaStrategies.ts": {
"value": 50.3,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.open-sse/handlers/chatCore/passthroughHelpers.ts": {
"value": 80.89,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.open-sse/handlers/chatCore/sanitization.ts": {
"value": 70.15,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.open-sse/handlers/chatCore/upstreamTimeouts.ts": {
"value": 33,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.open-sse/handlers/chatCore/comboContextCache.ts": {
"value": 13.62,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.open-sse/handlers/chatCore/idempotency.ts": {
"value": 42.82,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.open-sse/handlers/chatCore/responseHeaders.ts": {
"value": 62.7,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.open-sse/handlers/chatCore/executorHelpers.ts": {
"value": 70.39,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.open-sse/handlers/chatCore/memoryExtraction.ts": {
"value": 62.06,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.open-sse/handlers/chatCore/nonStreamingSse.ts": {
"value": 72.82,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.open-sse/handlers/chatCore/passthroughToolNames.ts": {
"value": 66.42,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.open-sse/handlers/chatCore/headers.ts": {
"value": 94.29,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.open-sse/handlers/chatCore/logTruncation.ts": {
"value": 77.64,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.open-sse/handlers/chatCore/memorySkillsInjection.ts": {
"value": 13.49,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.open-sse/handlers/chatCore/semanticCache.ts": {
"value": 60.16,
"direction": "up",
"dedicatedGate": true
},
"mutationScore.open-sse/handlers/chatCore/telemetryHelpers.ts": {
"value": 83.18,
"direction": "up",
"dedicatedGate": true
}
},
"_coverage_note": "Pisos anti-flake ~2pt abaixo do real do CI mergeado MEDIDO COM os 135 testes religados (run 27247237268: statements 78.4 / lines 78.4 / functions 83.84 / branches 75.73). O religamento da 6A.1 HONESTIFICOU a regua: os ~82.5 anteriores eram inflados porque modulos nunca importados ficavam fora do denominador do c8. Apertar via --require-tighten na Fase 6A (2026-06-16).",
@@ -166,5 +321,6 @@
"_osv_flip_blocking_2026_06_16_v3827": "Fim do ciclo v3.8.27: vulnCount (value 10, direction down, dedicatedGate) PROMOVIDO de ADVISORY para RATCHET BLOQUEANTE. check-vuln-ratchet.mjs ganhou um modo --ratchet (espelhando check-secrets/check-bundle-size/check-workflows da Etapa 2): le metrics.vulnCount.value daqui, compara a contagem MEDIDA pelo osv-scanner e sai 1 SOMENTE numa regressao real (medida > baseline). Sem --ratchet permanece advisory (exit 0). QUALQUER SKIP gracioso (osv-scanner ausente do PATH, osv.dev/rede inacessivel, timeout, JSON invalido) sai 0 MESMO com --ratchet — uma falha de MEDICAO nunca bloqueia, so uma regressao MEDIDA bloqueia. No ci.yml (job quality-extended) o step 'Vulnerability ratchet' passou a rodar 'npm run check:vuln-ratchet -- --ratchet' sem continue-on-error. Verificado no tip de release/v3.8.27: osv mede vulnCount=10 == baseline 10 (exit 0); forcar baseline=9 da exit 1 (10>9); PATH vazio da SKIP binary-absent exit 0. NB de VARIANCIA DE CVE: um CVE recem-divulgado numa dep ja presente pode redar o gate sem mudanca de dependencia — comportamento esperado, remedio = bumpar a dep ou re-baseline vulnCount com justificativa+issue. Ver docs/security/SUPPLY_CHAIN.md -> 'Variancia de CVE'.",
"_trivy_flip_blocking_2026_06_16_v3827": "Fim do ciclo v3.8.27: Trivy (scan de CVE da imagem Docker em docker-publish.yml) promovido para BLOQUEANTE em CRITICAL. Abordagem de DOIS PASSOS: o passo SARIF existente (severity HIGH,CRITICAL / exit-code 0 / upload SARIF) fica INTACTO para visibilidade na aba Security; um novo passo 'Trivy CRITICAL gate (blocking)' (severity CRITICAL / ignore-unfixed:true / exit-code 1) falha o release num CVE CRITICO FIXAVEL. ignore-unfixed evita travar por CVE de base-image sem patch upstream (reduz falso-bloqueio). Mesma variancia-de-CVE do osv: um novo CRITICAL fixavel divulgado pode redar; remedio = rebuild sobre base patcheada, bumpar dep, ou .trivyignore com justificativa+issue. Ver docs/security/SUPPLY_CHAIN.md. vulnCount permanece 10 (intocado neste flip — so a postura advisory->bloqueante mudou).",
"_rebaseline_2026_06_18_v3828_cycle_close": "Fim do ciclo v3.8.28 (RELEASED; ciclo v3.8.29 aberto): 3 metricas re-baselineadas para o valor REAL medido no push->main pos-release (run 27725117464, step 'Ratchet check') — eslintWarnings 3769->3779, openapiCoverage.pct 38.3->37.6, i18nUiCoverage.pct 80.1->79.1. Reproduzido localmente em release/v3.8.29 (9f14c1294): identico ao CI. Drift de fim-de-ciclo de features legitimas, NAO hand-cleanable: (a) eslint +10 = 'any' PERMITIDO (warn) em testes do ciclo + 4 react-hooks/exhaustive-deps em RequestLoggerV2.tsx (componente com bugs sutis de refresh #4103/#3972, arriscado mexer em deps de hook sem teste de UI); (b) openapi -0.7 = drop por rotas NOVAS INTERNAS (/api/tools/agent-bridge/* LOCAL_ONLY, spawnam MITM/DNS) — documenta-las no spec PUBLICO seria gaming; (c) i18n -1.0 = 37/41 locales em 79.1% (1741 chaves faltando cada, ~3000 traducoes via 'npm run i18n:run' que exige creds OMNIROUTE_TRANSLATION_API_KEY indisponiveis localmente). Mesmo precedente do _eslint_rebaseline_2026_06_16_v3826_forward_merge. Apertar no fim do ciclo: eslint/openapi via --require-tighten; i18n via i18n:run com creds. Autorizado pelo operador (decisao explicita).",
"_rebaseline_2026_06_19_v3829_cycle_close": "Release do ciclo v3.8.29: eslintWarnings re-baselineado 3779->3816 para o valor REAL medido em release/v3.8.29 (tip da3...; `npm run lint` local = 3816, identico ao Quality Ratchet do CI no PR #4126). O +37 e drift de fim-de-ciclo de 115 commits de features legitimas — `any` PERMITIDO (warn) em open-sse/ e tests/ do ciclo; os arquivos de reconciliacao deste release nao adicionam warnings (scripts/check/*.mjs sao eslint-ignored, o teste novo de check-fabricated-docs nao usa any). Mesmo precedente de _rebaseline_2026_06_18_v3828_cycle_close. Apertar via --require-tighten no fim do ciclo seguinte. ALÉM disso, o step Require-tighten (blocking) exigiu apertar 2 métricas que MELHORARAM no ciclo (medidas no CI do PR #4126): coverage.auth.lines 69->90 (CI mediu 92.6; piso ~2pt-abaixo-do-real anti-flake, dentro do tightenSlack 10) e openapiCoverage.pct 37.6->38.4 (rotas novas documentadas). Melhorias legitimas travadas no baseline, nao gaming. Autorizado pelo operador (release end-to-end, validado na VPS)."
"_rebaseline_2026_06_19_v3829_cycle_close": "Release do ciclo v3.8.29: eslintWarnings re-baselineado 3779->3816 para o valor REAL medido em release/v3.8.29 (tip da3...; `npm run lint` local = 3816, identico ao Quality Ratchet do CI no PR #4126). O +37 e drift de fim-de-ciclo de 115 commits de features legitimas — `any` PERMITIDO (warn) em open-sse/ e tests/ do ciclo; os arquivos de reconciliacao deste release nao adicionam warnings (scripts/check/*.mjs sao eslint-ignored, o teste novo de check-fabricated-docs nao usa any). Mesmo precedente de _rebaseline_2026_06_18_v3828_cycle_close. Apertar via --require-tighten no fim do ciclo seguinte. ALÉM disso, o step Require-tighten (blocking) exigiu apertar 2 métricas que MELHORARAM no ciclo (medidas no CI do PR #4126): coverage.auth.lines 69->90 (CI mediu 92.6; piso ~2pt-abaixo-do-real anti-flake, dentro do tightenSlack 10) e openapiCoverage.pct 37.6->38.4 (rotas novas documentadas). Melhorias legitimas travadas no baseline, nao gaming. Autorizado pelo operador (release end-to-end, validado na VPS).",
"_comment_mutationScore": "Per-module COVERED mutation score floors (detected/(detected+survived)), seeded ~2pt below the first full measurement (run 27823984918: split batches a1/a2/b1/b2/c1/c2/d + e/f/g/h/i). direction:up + dedicatedGate:true -> enforced ONLY by check-mutation-ratchet.mjs (the generic check-quality-ratchet skips dedicatedGate metrics), in the nightly-mutation aggregation job."
}