Files
OmniRoute/scripts/quality/collect-metrics.mjs
Diego Rodrigues de Sa e Souza cadc3f10b7 Release v3.8.35 (#4743)
* chore(release): open v3.8.35 development cycle

* fix db vacuum scheduler settings (#4726)

Scheduled VACUUM now follows Storage page settings (scheduledVacuum/vacuumHour) as single source of truth; env-flag control path removed. 11/11 vacuum-scheduler tests pass against release/v3.8.35 tip; no orphaned env refs. Integrated into release/v3.8.35.

* fix(tier): noAuth providers count as free; free filter returns empty … (#4753)

noAuth providers now classified free (union of legacy list + NOAUTH_PROVIDERS chat-tier derivation), -free arena_elo alias, and auto/<cat>:free returns an empty pool when no free candidate matches (opt-in legacy fallback via OMNIROUTE_AUTO_FREE_FALLBACK_TO_FULL_POOL). New env var documented in .env.example + ENVIRONMENT.md; CHANGELOG bullet added (maintainer co-author). 46/46 node + 56/56 vitest tests pass on release tip; env-doc-sync, docs-sync, typecheck:core, lint, file-size all green. Integrated into release/v3.8.35.

* refactor(chatCore): extrai 11 helpers de nível superior para 6 leaves puros (#3501) (#4571)

chatCore god-file decomposition (#3501): extract 6 pure leaves (cacheUsageMeta, executorClientHeaders, nonStreamingResponseBody, skillsFormat, streamErrorResult, streamFinalize) from chatCore.ts. Rebased onto release/v3.8.35 tip (resolved single chatCore.ts conflict — removed now-extracted inline buildExecutorClientHeaders). 265/265 chatcore tests, 26/26 new leaf tests, typecheck:core, cycles, file-size all green. Integrated into release/v3.8.35.

* refactor(chatCore): extrai resolveExecutorWithProxy + getExecutionCredentials para leaves (#3501) (#4646)

chatCore #3501: extract resolveExecutorWithProxy + getExecutionCredentials to leaves (executorProxy.ts, executionCredentials.ts). Clean cherry-pick onto release tip post-#4571. 12/12 new leaf tests, typecheck:core, cycles, file-size green. Integrated into release/v3.8.35.

* refactor(chatCore): extrai transforms de mensagens Claude p/ leaf (#3501) (#4708)

chatCore #3501: extract Claude upstream-message transforms to leaf (claudeUpstreamMessages.ts + claudeMessageTypes.ts). Clean cherry-pick post-#4646. 8/8 new leaf tests, typecheck/cycles/file-size green. Integrated into release/v3.8.35.

* refactor(chatCore): extrai persistAttemptLogs para leaf (#3501) (#4717)

chatCore #3501: extract persistAttemptLogs to leaf (attemptLogging.ts). Rebased onto release tip post-#4708 (resolved imports conflict: kept tip's resolveCompressionHeader from compression Phase 3, dropped now-unused logTruncation import moved into the leaf). 288/288 chatcore tests, typecheck/cycles/file-size green. Integrated into release/v3.8.35.

* refactor(chatCore): extrai stageTrace + compressionUsageReceipt para leaves (#3501) (#4721)

chatCore #3501: extract stageTrace + compressionUsageReceipt to leaves. Clean cherry-pick post-#4717. 6/6 new leaf tests, typecheck/cycles/file-size green. Integrated into release/v3.8.35.

* refactor(chatCore): extrai prepareUpstreamBody (1ª sub-fatia do executeProviderRequest, #3501) (#4730)

chatCore #3501: extract prepareUpstreamBody (first sub-slice of executeProviderRequest) to leaf (upstreamBody.ts). Clean cherry-pick post-#4721. 7/7 new leaf tests, full 301/301 chatcore suite, typecheck/cycles/file-size green. Completes the 6-PR chatCore decomposition stack into release/v3.8.35.

* fix(db): make db-backup import size cap configurable (#4719) (#4757)

Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>

* chore(quality): expand check:release-green to the FULL release-PR gate set (#4758)

The release-green pre-flight (Solution C) previously covered only a subset of the
gates that run exclusively on the release PR (PR→main), so reds still accrued
silently on release/** and surfaced in ~40-min layers at release time (v3.8.34:
3 CI rounds — CodeQL sanitization, then the fail-fast Quality Ratchet revealing
openapi then cyclomatic-complexity one push at a time, plus zizmor/integration).

Now check:release-green reproduces the COMPLETE release-PR gate set and reports
EVERY red in one pass (collected, not fail-fast):

- New DRIFT ratchets (report-only, rebaselined at release, never block):
  cyclomatic complexity, dead-code, type-coverage, compression-budget,
  openapi-coverage, workflow-lint (zizmor), codeql-ratchet.
- New HARD gates (real defects): docs-all (fabricated-docs strict + i18n mirror
  sync) and the integration test suite (gated behind !--quick).

The only release-PR gates it still cannot reproduce locally are GitHub-side CodeQL
semantic analysis and SonarQube/SonarCloud (external services).

The nightly-release-green workflow and /green-prs inherit the expanded coverage
automatically (they invoke this script), so cycle drift is now surfaced
continuously and the release PR is green on its first CI run.

Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>

* fix(dashboard): add missing onboarding.tiers step title (#4698) (#4755)

Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>

* feat(compression): Output Styles registry + D0 telemetry (Phase 4A) (#4694)

Phase 4A: Output Styles registry + D0 telemetry. Integrated into release/v3.8.35.

* feat(compression): SLM tier for ultra (Phase 4B) [stacked on #4694] (#4707)

Phase 4B: SLM tier for ultra. Integrated into release/v3.8.35.

* feat(compression): context-budget adaptive compression (Phase 4C) [stacked on #4707] (#4716)

Phase 4C: adaptive context-budget compression. Integrated into release/v3.8.35.

* feat(compression): offline evaluation harness (Phase 4 D1) [stacked on #4716] (#4720)

Phase 4 D1: offline evaluation harness. Integrated into release/v3.8.35.

* fix(sse): deepseek-web folds role:tool results into prompt transcript (#4712) (#4756)

Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>

* fix(dashboard): remove dead unconditional useLiveRequests call in HomePageClient (#4759, #4745, #4596) (#4761)

Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>

* fix(dashboard): dedupe provider nodes by id on compatible-provider add (#4746) (#4768)

Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>

* chore(db): re-export compressionRunTelemetry from localDb to satisfy db-rules (#4775)

Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>

* docs(security): add canonical STRIDE-based threat model (#4783)

Canonical STRIDE threat model. Integrated into release/v3.8.35.

* test(dashboard): add smoke test for home client dashboard (#4793)

Smoke test guarding the dashboard home client render (regression #4745/#4759). Code fix already landed via #4761; this PR's jsdom smoke test is the net-new regression guard. Integrated into release/v3.8.35.

* fix(combos): auto-promote zeroLatencyOptimizationsEnabled so legacy configs (pre-3.8.33 fallbackCompressionMode="lite") round-trip on the first GUI edit (#4774)

Auto-promote zeroLatencyOptimizationsEnabled + strip v3.8.31-era removed keys so legacy combo configs round-trip through PUT /api/combos/{id} on first GUI edit (closes #4382 followup). Pre-merge: rewrote the now-stale reject test to assert auto-promotion + added passthrough/round-trip regression guards; reconciled combos/page.tsx file-size baseline. Integrated into release/v3.8.35.

* refactor(chatCore): extrai parse + usage-stats não-streaming do executeProviderRequest (#3501) (#4762)

chatCore #3501: extract parseNonStreamingResponseBody + recordNonStreamingUsageStats. Integrated into release/v3.8.35.

* refactor(chatCore): extrai recordContextEditingTelemetryHook (#3501) (#4779)

chatCore #3501: extract recordContextEditingTelemetryHook. Integrated into release/v3.8.35.

* refactor(chatCore): extrai recordCompressionCacheStats (#3501) (#4792)

chatCore #3501: extract recordCompressionCacheStats. Integrated into release/v3.8.35.

* refactor(chatCore): extrai writeCavemanOutputAnalytics (#3501) (#4794)

chatCore #3501: extract writeCavemanOutputAnalytics. Integrated into release/v3.8.35.

* refactor(chatCore): extrai scheduleQuotaShareConsumption (POST-hook não-streaming, #3501) (#4780)

chatCore #3501: extract scheduleQuotaShareConsumption (non-streaming POST-hook). Integrated into release/v3.8.35.

* refactor(chatCore): extrai emitRequestGamificationEvent (helper compartilhado DRY, #3501) (#4776)

chatCore #3501: extract emitRequestGamificationEvent (DRY streaming/non-streaming). Integrated into release/v3.8.35.

* refactor(chatCore): extrai runPluginOnResponseHook (#3501) (#4782)

chatCore #3501: extract runPluginOnResponseHook. Integrated into release/v3.8.35.

* refactor(chatCore): extrai scheduleStreamingQuotaShareConsumption (POST-hook streaming, #3501) (#4784)

chatCore #3501: extract scheduleStreamingQuotaShareConsumption (streaming POST-hook). Integrated into release/v3.8.35.

* refactor(chatCore): extrai recordStreamingUsageStats (analytics de usage streaming, #3501) (#4791)

chatCore #3501: extract recordStreamingUsageStats. Integrated into release/v3.8.35.

* refactor(chatCore): extrai recordStreamingCost (custo por-request streaming, #3501) (#4790)

chatCore #3501: extract recordStreamingCost (per-request streaming cost). Integrated into release/v3.8.35.

* docs(readme): credit ponytail + OmniCompress; restore env-doc-sync release-green (#4799)

README compression credits (ponytail/OmniCompress) + env-doc-sync ignore for eval-only OMNIROUTE_EVAL_CREDENTIALS (restores release-green after #4720). Integrated into release/v3.8.35.

* chore(quality): trim combo-config.test.ts comments under file-size cap (#4774 follow-up) (#4800)

Restore file-size release-green. Integrated into release/v3.8.35.

* feat(api-docs): Redoc-rendered /api/docs + consolidate OpenAPI spec to docs/openapi.yaml (#4781)

Redoc /api/docs + OpenAPI spec consolidated to docs/openapi.yaml (canonical 201-path complete spec; old path → legacy fallback). All refs/gates/tests/CI updated. Integrated into release/v3.8.35.

* docs(compression): declare Phase 4 layers — Output Styles, adaptive dial, per-request control (#4801)

The README compression section listed the 9 input engines but not the Phase 4
layers now in production:
- Output Styles (output-axis steering: terse-prose / less-code / terse-cjk, lite/full/ultra)
- adaptive context-budget dial (reserve-output|percentage|absolute · floor|replace-autotrigger|off)
- per-request x-omniroute-compression precedence + the offline eval harness
Also bumped the highlights range to v3.8.35, expanded the compression feature bullet,
and marked the GUIDE's Phase 4 row Shipped (was 'Planned' — it's merged on v3.8.35).

Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore(release): finalize v3.8.35 CHANGELOG + docs reconciliation

- CHANGELOG: complete 3.8.35 section (all 35 commits since v3.8.34,
  contributor attribution: @rdself @megamen32 @KooshaPari @JxnLexn)
- docs(security): align THREAT_MODEL.md refs with real code
  (routeGuard.ts, tokenLimits.ts, /api/monitoring/health) — fabricated-docs gate
- check:fabricated-docs: skip docs/superpowers/specs (dated research reports)
- i18n: sync 3.8.35 section into 41 CHANGELOG mirrors (docs-sync size gate)
- ratchet rebaseline: cyclomatic 1916->1920, eslintWarnings 3907->3912
  (inherited cycle drift; release-finalize diff is docs-only)

* fix(release): resolve inherited base-reds surfaced by v3.8.35 release CI

Cycle base-reds that only run on PR→main (not the PR→release fast-path):

- test(autoCombo): suffixComposition-4517 used node:test in a vitest-only dir
  (#4753) → vitest found no suite. Switch to the vitest API. (Vitest job)
- test(agentSkills): openapiParser fixture wrote docs/reference/openapi.yaml;
  parser reads docs/openapi.yaml since #4781 → point fixture at the new path.
  (Unit/Coverage/Node24/Node26 shard 4)
- test(integration): proxy-pipeline source-scan expected inline streaming-cost
  code that #4790/#3501 extracted to the recordStreamingCost leaf → assert the
  delegation instead. (Integration 1/2)
- fix(chatCore): derive the log trace id from crypto, not Math.random
  (CodeQL js/insecure-randomness — log-correlation id, not a secret).
- test(resilience): circuit-breaker invalid-cooldown fallback asserted t>29000,
  flaking on slow CI where ~1.6s elapsed gave t=28401 → tolerate wall-clock
  drift (t>25000). (Unit 6/8)

* fix(usage): derive pending-request id from crypto, not Math.random

CodeQL js/insecure-randomness (#669): the pending-request id generated in
trackPendingRequest (usageHistory.ts) flows into attempt logging and was flagged
as insecure randomness in a security context. It's a log-correlation id, not a
secret — switch to crypto RNG to clear the alert. Pairs with the chatCore traceId
fix in 37c49781a (same sink).

---------

Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>
Co-authored-by: Randi <55005611+rdself@users.noreply.github.com>
Co-authored-by: Demiurge The Single <megamen932@gmail.com>
Co-authored-by: KooshaPari <42529354+KooshaPari@users.noreply.github.com>
Co-authored-by: Jan Leon <Jan.gaschler@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 17:06:18 -03:00

259 lines
9.6 KiB
JavaScript

#!/usr/bin/env node
// scripts/quality/collect-metrics.mjs — emite quality-metrics.json
// Coletores incrementais: Fase 1 traz ESLint warnings + cobertura.
// Fases 3/4 estendem com duplicação (jscpd), tamanho de arquivo e cobertura por módulo.
// Fase 6A.11: openapiCoverage.pct + i18nUiCoverage.pct (mínimo entre locales).
// Task 7.9: coverage.<modulo>.lines para ~8 módulos críticos, lidos do
// coverage/coverage-summary.json se existir (sem erro se ausente).
import fs from "node:fs";
import { promises as fsAsync } from "node:fs";
import path from "node:path";
import { pathToFileURL } from "node:url";
import { execFileSync } from "node:child_process";
import * as yaml from "js-yaml";
const cwd = process.cwd();
const out = {};
// 1) ESLint: contagem de warnings (errors devem ser 0; o lint já gata isso)
function eslintCounts() {
let stdout;
try {
stdout = execFileSync("npx", ["eslint", ".", "--format", "json"], {
encoding: "utf8",
maxBuffer: 256 * 1024 * 1024,
});
} catch (e) {
// eslint sai com código != 0 quando há errors; o JSON ainda vem no stdout
stdout = e.stdout?.toString() || "[]";
}
const results = JSON.parse(stdout);
out.eslintWarnings = results.reduce((n, r) => n + (r.warningCount || 0), 0);
out.eslintErrors = results.reduce((n, r) => n + (r.errorCount || 0), 0);
}
// 2) Cobertura: lê coverage/coverage-summary.json se existir (gerado por c8)
function coverage() {
const p = path.join(cwd, "coverage", "coverage-summary.json");
if (!fs.existsSync(p)) return;
const t = JSON.parse(fs.readFileSync(p, "utf8")).total;
out["coverage.statements"] = t.statements.pct;
out["coverage.lines"] = t.lines.pct;
out["coverage.functions"] = t.functions.pct;
out["coverage.branches"] = t.branches.pct;
}
// 3) Coverage per critical module (Task 7.9)
// Reads coverage/coverage-summary.json (produced by `npm run test:coverage` via c8).
// If the file is absent → silently skips (no error). This allows the gate to
// function normally in environments where coverage was not run (e.g. lint-only CI).
//
// The summary JSON produced by c8 looks like:
// { "total": {...}, "/abs/path/to/file.ts": { lines: { pct: 78 }, ... }, ... }
//
// modulePaths is a record of { metricSuffix: relPathFromRoot[] } — the first
// matching key in the summary wins.
/**
* Pure function — extracts per-module line-coverage percentages from a
* coverage-summary.json object.
*
* @param {Record<string, { lines?: { pct: number } }>} summaryJson
* The parsed coverage-summary.json (keys are absolute file paths or "total").
* @param {Record<string, string[]>} modulePaths
* Map of { metricKey: [relPath, ...fallbacks] } where relPath is relative to
* the repo root (forward slashes). Returns the lines.pct of the first match.
* @param {string} repoRoot Absolute path to the repo root (used to build keys).
* @returns {Record<string, number>} Map of metricKey → lines.pct (0-100).
*/
export function extractModuleCoverage(summaryJson, modulePaths, repoRoot) {
const result = {};
// Build a normalised lookup: absolute path (forward slashes) → pct
const lookup = new Map();
for (const [rawKey, data] of Object.entries(summaryJson)) {
if (rawKey === "total") continue;
const norm = rawKey.replace(/\\/g, "/");
const pct = data?.lines?.pct;
if (typeof pct === "number") lookup.set(norm, pct);
}
const normRoot = repoRoot.replace(/\\/g, "/").replace(/\/$/, "");
for (const [metricKey, candidates] of Object.entries(modulePaths)) {
for (const rel of candidates) {
const abs = `${normRoot}/${rel.replace(/\\/g, "/").replace(/^\//, "")}`;
if (lookup.has(abs)) {
result[metricKey] = lookup.get(abs);
break;
}
}
}
return result;
}
/** The 8 critical modules tracked by Task 7.9 (relative paths from repo root). */
export const CRITICAL_MODULE_PATHS = {
"coverage.chatCore.lines": ["open-sse/handlers/chatCore.ts"],
"coverage.combo.lines": ["open-sse/services/combo.ts"],
"coverage.accountFallback.lines": ["open-sse/services/accountFallback.ts"],
"coverage.auth.lines": ["src/sse/services/auth.ts"],
"coverage.routeGuard.lines": ["src/server/authz/routeGuard.ts"],
"coverage.error.lines": ["open-sse/utils/error.ts"],
"coverage.publicCreds.lines": ["open-sse/utils/publicCreds.ts"],
"coverage.circuitBreaker.lines": ["src/shared/utils/circuitBreaker.ts"],
};
function coverageByModule() {
const p = path.join(cwd, "coverage", "coverage-summary.json");
if (!fs.existsSync(p)) return; // absent → skip silently (Task 7.9 spec)
let summaryJson;
try {
summaryJson = JSON.parse(fs.readFileSync(p, "utf8"));
} catch {
return; // malformed file → skip
}
const moduleMetrics = extractModuleCoverage(summaryJson, CRITICAL_MODULE_PATHS, cwd);
Object.assign(out, moduleMetrics);
}
// 4) OpenAPI coverage: percentage of implemented routes documented in openapi.yaml
function openapiCoverage() {
const API_ROOT = path.join(cwd, "src", "app", "api");
const OPENAPI_PATH = path.join(cwd, "docs", "openapi.yaml");
if (!fs.existsSync(API_ROOT) || !fs.existsSync(OPENAPI_PATH)) return;
function collectRoutePaths(dir) {
const entries = fs.readdirSync(dir, { withFileTypes: true });
const paths = [];
for (const entry of entries) {
const fullPath = path.join(dir, entry.name);
if (entry.isDirectory()) {
paths.push(...collectRoutePaths(fullPath));
continue;
}
if (entry.isFile() && entry.name === "route.ts") {
const apiPath = path
.dirname(fullPath)
.replace(API_ROOT, "")
.replace(/\[([^\]]+)\]/g, "{$1}");
paths.push(`/api${apiPath}`);
}
}
return paths;
}
function normalizePath(p) {
return p.replace(/\/\[\.\.\.([^\]]+)\]/g, "/{$1}").replace(/\[([^\]]+)\]/g, "{$1}");
}
const implementedPaths = collectRoutePaths(API_ROOT).map(normalizePath);
const raw = yaml.load(fs.readFileSync(OPENAPI_PATH, "utf-8"));
const documentedPaths = new Set(Object.keys(raw.paths || {}));
const covered = implementedPaths.filter((p) => documentedPaths.has(p)).length;
const total = implementedPaths.length;
if (total > 0) out["openapiCoverage.pct"] = parseFloat(((covered / total) * 100).toFixed(1));
}
// 4) i18n UI coverage: minimum real coverage across all non-en locales
async function i18nUiCoverage() {
const MESSAGES_DIR = path.join(cwd, "src", "i18n", "messages");
const CONFIG_PATH = path.join(cwd, "config", "i18n.json");
const SOURCE_LOCALE = "en";
const PLACEHOLDER_PREFIX = "__MISSING__:";
if (!fs.existsSync(MESSAGES_DIR)) return;
const sourcePath = path.join(MESSAGES_DIR, `${SOURCE_LOCALE}.json`);
if (!fs.existsSync(sourcePath)) return;
function isPlainObject(value) {
return value !== null && typeof value === "object" && !Array.isArray(value);
}
function collectLeafPaths(obj, prefix = []) {
const paths = [];
for (const [key, value] of Object.entries(obj)) {
const next = [...prefix, key];
if (isPlainObject(value)) {
paths.push(...collectLeafPaths(value, next));
} else {
paths.push(next);
}
}
return paths;
}
const FORBIDDEN_KEY_SEGMENTS = new Set(["__proto__", "prototype", "constructor"]);
function lookupPath(obj, parts) {
let cur = obj;
for (const part of parts) {
if (!isPlainObject(cur)) return undefined;
if (FORBIDDEN_KEY_SEGMENTS.has(part)) return undefined;
if (!Object.prototype.hasOwnProperty.call(cur, part)) return undefined;
const entry = Object.entries(cur).find(([k]) => k === part);
cur = entry ? entry[1] : undefined;
}
return cur;
}
const source = JSON.parse(fs.readFileSync(sourcePath, "utf8"));
const enPaths = collectLeafPaths(source);
const totalEn = enPaths.length;
if (totalEn === 0) return;
let configCodes = null;
if (fs.existsSync(CONFIG_PATH)) {
try {
const cfg = JSON.parse(fs.readFileSync(CONFIG_PATH, "utf8"));
if (Array.isArray(cfg.locales)) configCodes = new Set(cfg.locales.map((l) => l.code));
} catch {
/* ignore */
}
}
const onDisk = (await fsAsync.readdir(MESSAGES_DIR))
.filter((f) => f.endsWith(".json") && f !== `${SOURCE_LOCALE}.json`)
.map((f) => f.slice(0, -5))
.filter((code) => (configCodes ? configCodes.has(code) : true));
let minCoverage = 100;
for (const locale of onDisk) {
const localePath = path.join(MESSAGES_DIR, `${locale}.json`);
let target;
try {
target = JSON.parse(fs.readFileSync(localePath, "utf8"));
} catch {
minCoverage = 0;
continue;
}
let present = 0;
let placeholder = 0;
for (const pathParts of enPaths) {
const value = lookupPath(target, pathParts);
if (value === undefined || isPlainObject(value)) continue;
present++;
if (typeof value === "string" && value.startsWith(PLACEHOLDER_PREFIX)) placeholder++;
}
const coverage = ((present - placeholder) / totalEn) * 100;
if (coverage < minCoverage) minCoverage = coverage;
}
if (onDisk.length > 0) out["i18nUiCoverage.pct"] = parseFloat(minCoverage.toFixed(1));
}
// Only run the collection pipeline when this file is executed directly.
// When imported (e.g. in tests), only the exported pure functions are available
// without triggering the expensive ESLint + i18n filesystem walks.
if (import.meta.url === pathToFileURL(process.argv[1] || "").href) {
eslintCounts();
coverage();
coverageByModule();
openapiCoverage();
await i18nUiCoverage();
fs.writeFileSync(
path.join(cwd, "config/quality/quality-metrics.json"),
JSON.stringify(out, null, 2) + "\n"
);
console.log("[collect-metrics]", JSON.stringify(out));
}