pjandro/OmniRoute
1
0
Fork 0
mirror of https://github.com/diegosouzapw/OmniRoute.git synced 2026-08-12 02:02:13 +03:00
Code Issues Packages Projects Releases Wiki Activity
Files
342cd87e12335143f2de416ec8af76bceeb104de
OmniRoute/docs/architecture/QUALITY_GATES.md
Diego Rodrigues de Sa e Souza cadc3f10b7 Release v3.8.35 (#4743) 
* chore(release): open v3.8.35 development cycle

* fix db vacuum scheduler settings (#4726)

Scheduled VACUUM now follows Storage page settings (scheduledVacuum/vacuumHour) as single source of truth; env-flag control path removed. 11/11 vacuum-scheduler tests pass against release/v3.8.35 tip; no orphaned env refs. Integrated into release/v3.8.35.

* fix(tier): noAuth providers count as free; free filter returns empty … (#4753)

noAuth providers now classified free (union of legacy list + NOAUTH_PROVIDERS chat-tier derivation), -free arena_elo alias, and auto/<cat>:free returns an empty pool when no free candidate matches (opt-in legacy fallback via OMNIROUTE_AUTO_FREE_FALLBACK_TO_FULL_POOL). New env var documented in .env.example + ENVIRONMENT.md; CHANGELOG bullet added (maintainer co-author). 46/46 node + 56/56 vitest tests pass on release tip; env-doc-sync, docs-sync, typecheck:core, lint, file-size all green. Integrated into release/v3.8.35.

* refactor(chatCore): extrai 11 helpers de nível superior para 6 leaves puros (#3501) (#4571)

chatCore god-file decomposition (#3501): extract 6 pure leaves (cacheUsageMeta, executorClientHeaders, nonStreamingResponseBody, skillsFormat, streamErrorResult, streamFinalize) from chatCore.ts. Rebased onto release/v3.8.35 tip (resolved single chatCore.ts conflict — removed now-extracted inline buildExecutorClientHeaders). 265/265 chatcore tests, 26/26 new leaf tests, typecheck:core, cycles, file-size all green. Integrated into release/v3.8.35.

* refactor(chatCore): extrai resolveExecutorWithProxy + getExecutionCredentials para leaves (#3501) (#4646)

chatCore #3501: extract resolveExecutorWithProxy + getExecutionCredentials to leaves (executorProxy.ts, executionCredentials.ts). Clean cherry-pick onto release tip post-#4571. 12/12 new leaf tests, typecheck:core, cycles, file-size green. Integrated into release/v3.8.35.

* refactor(chatCore): extrai transforms de mensagens Claude p/ leaf (#3501) (#4708)

chatCore #3501: extract Claude upstream-message transforms to leaf (claudeUpstreamMessages.ts + claudeMessageTypes.ts). Clean cherry-pick post-#4646. 8/8 new leaf tests, typecheck/cycles/file-size green. Integrated into release/v3.8.35.

* refactor(chatCore): extrai persistAttemptLogs para leaf (#3501) (#4717)

chatCore #3501: extract persistAttemptLogs to leaf (attemptLogging.ts). Rebased onto release tip post-#4708 (resolved imports conflict: kept tip's resolveCompressionHeader from compression Phase 3, dropped now-unused logTruncation import moved into the leaf). 288/288 chatcore tests, typecheck/cycles/file-size green. Integrated into release/v3.8.35.

* refactor(chatCore): extrai stageTrace + compressionUsageReceipt para leaves (#3501) (#4721)

chatCore #3501: extract stageTrace + compressionUsageReceipt to leaves. Clean cherry-pick post-#4717. 6/6 new leaf tests, typecheck/cycles/file-size green. Integrated into release/v3.8.35.

* refactor(chatCore): extrai prepareUpstreamBody (1ª sub-fatia do executeProviderRequest, #3501) (#4730)

chatCore #3501: extract prepareUpstreamBody (first sub-slice of executeProviderRequest) to leaf (upstreamBody.ts). Clean cherry-pick post-#4721. 7/7 new leaf tests, full 301/301 chatcore suite, typecheck/cycles/file-size green. Completes the 6-PR chatCore decomposition stack into release/v3.8.35.

* fix(db): make db-backup import size cap configurable (#4719) (#4757)

Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>

* chore(quality): expand check:release-green to the FULL release-PR gate set (#4758)

The release-green pre-flight (Solution C) previously covered only a subset of the
gates that run exclusively on the release PR (PR→main), so reds still accrued
silently on release/** and surfaced in ~40-min layers at release time (v3.8.34:
3 CI rounds — CodeQL sanitization, then the fail-fast Quality Ratchet revealing
openapi then cyclomatic-complexity one push at a time, plus zizmor/integration).

Now check:release-green reproduces the COMPLETE release-PR gate set and reports
EVERY red in one pass (collected, not fail-fast):

- New DRIFT ratchets (report-only, rebaselined at release, never block):
  cyclomatic complexity, dead-code, type-coverage, compression-budget,
  openapi-coverage, workflow-lint (zizmor), codeql-ratchet.
- New HARD gates (real defects): docs-all (fabricated-docs strict + i18n mirror
  sync) and the integration test suite (gated behind !--quick).

The only release-PR gates it still cannot reproduce locally are GitHub-side CodeQL
semantic analysis and SonarQube/SonarCloud (external services).

The nightly-release-green workflow and /green-prs inherit the expanded coverage
automatically (they invoke this script), so cycle drift is now surfaced
continuously and the release PR is green on its first CI run.

Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>

* fix(dashboard): add missing onboarding.tiers step title (#4698) (#4755)

Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>

* feat(compression): Output Styles registry + D0 telemetry (Phase 4A) (#4694)

Phase 4A: Output Styles registry + D0 telemetry. Integrated into release/v3.8.35.

* feat(compression): SLM tier for ultra (Phase 4B) [stacked on #4694] (#4707)

Phase 4B: SLM tier for ultra. Integrated into release/v3.8.35.

* feat(compression): context-budget adaptive compression (Phase 4C) [stacked on #4707] (#4716)

Phase 4C: adaptive context-budget compression. Integrated into release/v3.8.35.

* feat(compression): offline evaluation harness (Phase 4 D1) [stacked on #4716] (#4720)

Phase 4 D1: offline evaluation harness. Integrated into release/v3.8.35.

* fix(sse): deepseek-web folds role:tool results into prompt transcript (#4712) (#4756)

Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>

* fix(dashboard): remove dead unconditional useLiveRequests call in HomePageClient (#4759, #4745, #4596) (#4761)

Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>

* fix(dashboard): dedupe provider nodes by id on compatible-provider add (#4746) (#4768)

Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>

* chore(db): re-export compressionRunTelemetry from localDb to satisfy db-rules (#4775)

Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>

* docs(security): add canonical STRIDE-based threat model (#4783)

Canonical STRIDE threat model. Integrated into release/v3.8.35.

* test(dashboard): add smoke test for home client dashboard (#4793)

Smoke test guarding the dashboard home client render (regression #4745/#4759). Code fix already landed via #4761; this PR's jsdom smoke test is the net-new regression guard. Integrated into release/v3.8.35.

* fix(combos): auto-promote zeroLatencyOptimizationsEnabled so legacy configs (pre-3.8.33 fallbackCompressionMode="lite") round-trip on the first GUI edit (#4774)

Auto-promote zeroLatencyOptimizationsEnabled + strip v3.8.31-era removed keys so legacy combo configs round-trip through PUT /api/combos/{id} on first GUI edit (closes #4382 followup). Pre-merge: rewrote the now-stale reject test to assert auto-promotion + added passthrough/round-trip regression guards; reconciled combos/page.tsx file-size baseline. Integrated into release/v3.8.35.

* refactor(chatCore): extrai parse + usage-stats não-streaming do executeProviderRequest (#3501) (#4762)

chatCore #3501: extract parseNonStreamingResponseBody + recordNonStreamingUsageStats. Integrated into release/v3.8.35.

* refactor(chatCore): extrai recordContextEditingTelemetryHook (#3501) (#4779)

chatCore #3501: extract recordContextEditingTelemetryHook. Integrated into release/v3.8.35.

* refactor(chatCore): extrai recordCompressionCacheStats (#3501) (#4792)

chatCore #3501: extract recordCompressionCacheStats. Integrated into release/v3.8.35.

* refactor(chatCore): extrai writeCavemanOutputAnalytics (#3501) (#4794)

chatCore #3501: extract writeCavemanOutputAnalytics. Integrated into release/v3.8.35.

* refactor(chatCore): extrai scheduleQuotaShareConsumption (POST-hook não-streaming, #3501) (#4780)

chatCore #3501: extract scheduleQuotaShareConsumption (non-streaming POST-hook). Integrated into release/v3.8.35.

* refactor(chatCore): extrai emitRequestGamificationEvent (helper compartilhado DRY, #3501) (#4776)

chatCore #3501: extract emitRequestGamificationEvent (DRY streaming/non-streaming). Integrated into release/v3.8.35.

* refactor(chatCore): extrai runPluginOnResponseHook (#3501) (#4782)

chatCore #3501: extract runPluginOnResponseHook. Integrated into release/v3.8.35.

* refactor(chatCore): extrai scheduleStreamingQuotaShareConsumption (POST-hook streaming, #3501) (#4784)

chatCore #3501: extract scheduleStreamingQuotaShareConsumption (streaming POST-hook). Integrated into release/v3.8.35.

* refactor(chatCore): extrai recordStreamingUsageStats (analytics de usage streaming, #3501) (#4791)

chatCore #3501: extract recordStreamingUsageStats. Integrated into release/v3.8.35.

* refactor(chatCore): extrai recordStreamingCost (custo por-request streaming, #3501) (#4790)

chatCore #3501: extract recordStreamingCost (per-request streaming cost). Integrated into release/v3.8.35.

* docs(readme): credit ponytail + OmniCompress; restore env-doc-sync release-green (#4799)

README compression credits (ponytail/OmniCompress) + env-doc-sync ignore for eval-only OMNIROUTE_EVAL_CREDENTIALS (restores release-green after #4720). Integrated into release/v3.8.35.

* chore(quality): trim combo-config.test.ts comments under file-size cap (#4774 follow-up) (#4800)

Restore file-size release-green. Integrated into release/v3.8.35.

* feat(api-docs): Redoc-rendered /api/docs + consolidate OpenAPI spec to docs/openapi.yaml (#4781)

Redoc /api/docs + OpenAPI spec consolidated to docs/openapi.yaml (canonical 201-path complete spec; old path → legacy fallback). All refs/gates/tests/CI updated. Integrated into release/v3.8.35.

* docs(compression): declare Phase 4 layers — Output Styles, adaptive dial, per-request control (#4801)

The README compression section listed the 9 input engines but not the Phase 4
layers now in production:
- Output Styles (output-axis steering: terse-prose / less-code / terse-cjk, lite/full/ultra)
- adaptive context-budget dial (reserve-output|percentage|absolute · floor|replace-autotrigger|off)
- per-request x-omniroute-compression precedence + the offline eval harness
Also bumped the highlights range to v3.8.35, expanded the compression feature bullet,
and marked the GUIDE's Phase 4 row Shipped (was 'Planned' — it's merged on v3.8.35).

Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore(release): finalize v3.8.35 CHANGELOG + docs reconciliation

- CHANGELOG: complete 3.8.35 section (all 35 commits since v3.8.34,
  contributor attribution: @rdself @megamen32 @KooshaPari @JxnLexn)
- docs(security): align THREAT_MODEL.md refs with real code
  (routeGuard.ts, tokenLimits.ts, /api/monitoring/health) — fabricated-docs gate
- check:fabricated-docs: skip docs/superpowers/specs (dated research reports)
- i18n: sync 3.8.35 section into 41 CHANGELOG mirrors (docs-sync size gate)
- ratchet rebaseline: cyclomatic 1916->1920, eslintWarnings 3907->3912
  (inherited cycle drift; release-finalize diff is docs-only)

* fix(release): resolve inherited base-reds surfaced by v3.8.35 release CI

Cycle base-reds that only run on PR→main (not the PR→release fast-path):

- test(autoCombo): suffixComposition-4517 used node:test in a vitest-only dir
  (#4753) → vitest found no suite. Switch to the vitest API. (Vitest job)
- test(agentSkills): openapiParser fixture wrote docs/reference/openapi.yaml;
  parser reads docs/openapi.yaml since #4781 → point fixture at the new path.
  (Unit/Coverage/Node24/Node26 shard 4)
- test(integration): proxy-pipeline source-scan expected inline streaming-cost
  code that #4790/#3501 extracted to the recordStreamingCost leaf → assert the
  delegation instead. (Integration 1/2)
- fix(chatCore): derive the log trace id from crypto, not Math.random
  (CodeQL js/insecure-randomness — log-correlation id, not a secret).
- test(resilience): circuit-breaker invalid-cooldown fallback asserted t>29000,
  flaking on slow CI where ~1.6s elapsed gave t=28401 → tolerate wall-clock
  drift (t>25000). (Unit 6/8)

* fix(usage): derive pending-request id from crypto, not Math.random

CodeQL js/insecure-randomness (#669): the pending-request id generated in
trackPendingRequest (usageHistory.ts) flows into attempt logging and was flagged
as insecure randomness in a security context. It's a log-correlation id, not a
secret — switch to crypto RNG to clear the alert. Pairs with the chatCore traceId
fix in 37c49781a (same sink).

---------

Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>
Co-authored-by: Randi <55005611+rdself@users.noreply.github.com>
Co-authored-by: Demiurge The Single <megamen932@gmail.com>
Co-authored-by: KooshaPari <42529354+KooshaPari@users.noreply.github.com>
Co-authored-by: Jan Leon <Jan.gaschler@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 17:06:18 -03:00

280 lines
24 KiB
Markdown
Raw Blame History

---
title: Quality Gates Reference
---
# Quality Gates Reference
This document is the authoritative reference for all CI quality gates in OmniRoute.
It describes each gate, what it validates, which CI job it runs in, whether it uses
a ratchet baseline or a pass/fail policy, and whether it blocks the build or is advisory.
For a short summary and the allowlist policy, see the "Quality Gates & Ratchets" section
in `CLAUDE.md`.
---
## Gate Inventory (~48 scripts)
Scripts live under `scripts/check/` (policy gates) and `scripts/quality/` (ratchet engine).
The CI source of truth is `.github/workflows/ci.yml`.
### Job: `lint`
Runs on every PR to `main`. Blocks merge on failure.
| Script (`npm run ...`) | Validates | Blocking |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------- |
| `check:node-runtime` | Node.js version is within the supported range | Yes |
| `check:cycles` | Circular imports — all `src/` + `open-sse/` modules | Yes |
| `check:route-validation:t06` | Zod schemas present on all routes (Tier 6 policy) | Yes |
| `check:any-budget:t11` | `@ts-expect-error // any` count does not exceed budget (Tier 11 catraca) | Yes |
| `check:provider-consistency` | Every provider in `providers.ts` has a matching entry in `providerRegistry.ts` (and vice-versa, within the allowlist) | Yes |
| `check:fetch-targets` | Every `fetch("/api/...")` in client-side `src/` resolves to a real `route.ts` | Yes |
| `check:deps` | All `npm install`-able deps across every `package.json` in the repo are in `dependency-allowlist.json`; new unpinned or slopsquatted packages flagged | Yes |
| `audit:deps` | `npm audit` (root + electron) — no high/critical advisories (overlaps osv `check:vuln-ratchet`; see Rationalization Backlog) | Yes |
| `check:lockfile` | `package-lock.json` integrity — https registry, integrity hashes, no host overrides | Yes |
| `check:licenses` | SPDX license allowlist for production dependencies | Yes |
| `check:tracked-artifacts` | No build artifacts / committed `node_modules` symlinks (also runs in husky pre-push) | Yes |
| `check:file-size` | No source file exceeds the per-extension cap (ratchet: frozen large files in `frozen` list) | Yes |
| `check:error-helper` | Error responses in executors/handlers use `buildErrorBody()` / `sanitizeErrorMessage()` (Hard Rule #12) | Yes |
| `check:migration-numbering` | Migration SQL files are sequentially numbered, no gaps or duplicates | Yes |
| `check:public-creds` | No literal OAuth `client_id`/`client_secret` or Firebase Web keys outside `publicCreds.ts` (Hard Rule #11) | Yes |
| `check:db-rules` | No raw SQL outside `src/lib/db/` modules; no barrel-imports from `localDb.ts` (Hard Rules #2/#5) | Yes |
| `check:known-symbols` | Provider executors, routing strategies, and translators registered in their dispatch tables match the files on disk — no orphaned or undeclared symbols | Yes |
| `check:route-guard-membership` | Every route that spawns a child process is classified by `isLocalOnlyPath()` (Hard Rules #15/#17) | Yes |
| `check:test-discovery` | Every `*.test.ts` / `*.spec.ts` file in the repo is collected by at least one test runner (ratchet: orphan list in `test-discovery-baseline.json` can only shrink) | Yes |
| `check:docs-sync` | CHANGELOG version, OpenAPI version, and `llm.txt` are in sync | Yes |
| `typecheck:core` | TypeScript compilation without errors (advisory warnings only) | Yes |
| `typecheck:noimplicit:core` | Strict `noImplicitAny` — forward-looking; many pre-existing call sites still need annotations | **Advisory** (`continue-on-error: true`) |
### Job: `quality-gate`
Runs after `test-coverage`. Blocks merge on failure.
| Script | Validates | Blocking |
| ------------------- | ---------------------------------------------------------------------------------------------------------- | ------------------------- |
| `quality:collect` | Emits `quality-metrics.json` (ESLint warning count, coverage from merged shard report) | Yes (upstream of ratchet) |
| `quality:ratchet` | Each metric in `quality-baseline.json` has not regressed (ESLint warnings ≤ baseline; coverage ≥ baseline) | Yes |
| `check:duplication` | Code duplication (jscpd@4) does not exceed baseline in `quality-baseline.json` | Yes |
| `check:complexity` | File-level cyclomatic complexity does not exceed the cap (core ESLint `complexity` + `max-lines-per-function`) | Yes |
| `check:cognitive-complexity` | Cognitive complexity ratchet (`eslint-plugin-sonarjs`) — separate ESLint pass; mergeable with `check:complexity` (see Backlog) | Yes |
| `check:dead-code` | Unused exports / files ratchet (knip) does not regress vs baseline | Yes |
| `check:type-coverage` | Percent-typed ratchet (`type-coverage`) does not regress; largely subsumes `typecheck:noimplicit:core` | Yes |
| `check:codeql-ratchet` | Open CodeQL alert count does not regress (reads via `gh api`; graceful-skip without token) | Yes |
### Job: `quality-extended`
Entire job is advisory (`continue-on-error: true`). The npm-based ratchets run for
real; the external scanners install via `gh release download` and self-skip (exit 0)
when a binary is still absent.
| Script | Validates | Blocking |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------ | ------------ |
| `check:circular-deps` | No circular dependencies (dpdm) | **Advisory** |
| `check:bundle-size` | Bundle size does not exceed the cap | **Advisory** |
| `check:secrets` | Secret scanning (gitleaks) — skips if binary absent | **Advisory** |
| `check:vuln-ratchet` | Dependency vulnerabilities (osv-scanner) do not regress — skips if binary absent | **Advisory** |
| `check:workflows` | Workflow lint (actionlint + zizmor) — skips if binaries absent | **Advisory** |
| `check:openapi-breaking` | Breaking changes to the public API contract (`openapi.yaml`) vs the base branch (oasdiff) — emits `openapiBreaking=N`; skips if oasdiff absent or base spec unresolvable | **Advisory** |
### Job: `docs-sync-strict`
Runs on every PR to `main`. Blocks merge on failure.
| Script | Validates | Blocking |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------- |
| `check:docs-all` | Meta-gate that runs the 6 sub-gates below sequentially | Yes |
| ↳ `check:docs-sync` | CHANGELOG / OpenAPI / llm.txt version consistency | Yes |
| ↳ `check:docs-counts` | Counts in prose (provider count, migration count, etc.) are within the ratchet window of the real counts | Yes |
| ↳ `check:env-doc-sync` | Every env var in `.env.example` is documented in a docs table, and vice versa | Yes |
| ↳ `check:deprecated-versions` | No deprecated version strings in docs | Yes |
| ↳ `check:doc-links` | Internal markdown links in docs resolve to real files (`[text]`/`(path)` form) | Yes |
| ↳ `check:fabricated-docs` | Routes, env vars, CLI commands, hook names, and file paths cited in docs exist in the codebase. Hard gate via `--strict`; soft-fail without flag. | Yes (via `--strict` in CI) |
| `check:cli-i18n` | CLI command strings are present in all i18n locale files | Yes |
| `check:openapi-coverage` | OpenAPI spec covers at least a ratcheted floor of real routes | Yes |
| `check:openapi-security-tiers` | Security tier annotations in `openapi.yaml` are consistent with `routeGuard.ts` classifications | **Advisory** |
| `check:openapi-routes` | Every path in `openapi.yaml` resolves to a real `route.ts` (anti-hallucination) | Yes |
| `check:docs-symbols` | Every `/api/...` reference in `docs/**/*.md` resolves to a real `route.ts` (anti-hallucination) | Yes |
| `i18n translation drift` | Untranslated keys in i18n locale files — warn only | **Advisory** |
### Job: `i18n-ui-coverage`
| Script | Validates | Blocking |
| --------------------------------- | ----------------------------- | -------- |
| `check-ui-keys-coverage` (inline) | UI i18n key coverage is ≥ 65% | Yes |
### Job: `i18n`
Full i18n validation matrix (one job per locale). Entire job is advisory.
| Script | Validates | Blocking |
| ------------------------------- | ----------------------------------- | ----------------------------------------------------- |
| `validate_translation.py quick` | Translation completeness per locale | **Advisory** (`continue-on-error: true` on whole job) |
### Job: `pr-test-policy`
Runs on pull requests only.
| Script | Validates | Blocking |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------- | -------- |
| `check:pr-test-policy` | PRs that change production code in `src/`, `open-sse/`, `electron/`, or `bin/` must include or update tests (Hard Rule #8) | Yes |
| `check:test-masking` | Changed test files do not reduce net assert count or add `assert.ok(true)` tautologies | Yes |
| `check:pr-evidence` | PR body cites test/VPS evidence for the change (mechanizes Hard Rule #18 by grepping PR prose — fragile, see Backlog) | Yes |
### Job: `test-vitest`
Runs after `build`. Blocks merge on failure.
| Suite | Validates | Blocking |
| ---------------- | ------------------------------------------------------- | -------------------------------------------------------------------------- |
| `test:vitest` | MCP server (87 tools), autoCombo, cache — vitest runner | Yes |
| `test:vitest:ui` | UI component tests — vitest runner | **Advisory** (`continue-on-error: true`) — failing until Fase 6A UI triage |
### Nightly workflows (scheduled, advisory)
These run on a cron schedule (and `workflow_dispatch`), never on PRs. All are advisory.
| Workflow | Validates | Blocking |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------ |
| `nightly-property` | fast-check property tests with a random seed + high run count | **Advisory** |
| `nightly-resilience` | heap-growth gate, chaos fault-injection, k6 load/soak | **Advisory** |
| `nightly-llm-security` | promptfoo injection guard (block mode) + garak probes (skipped without a provider secret) | **Advisory** |
| `nightly-schemathesis` | OpenAPI contract fuzzing (schemathesis) against a live OmniRoute using `docs/openapi.yaml` — surfaces spec violations / unhandled 500s (Fase 8 B.4) | **Advisory** |
---
## Ratchet Baseline (`quality-baseline.json`)
The ratchet engine (`scripts/quality/check-quality-ratchet.mjs`) reads `quality-baseline.json`
and compares it against the freshly collected `quality-metrics.json`. Any metric that regresses
beyond its epsilon fails the build.
Current tracked metrics:
| Metric | Direction | Meaning |
| --------------------- | --------- | ---------------------------------- |
| `eslintWarnings` | `down` | ESLint warning count must not grow |
| `coverage.statements` | `up` | Statement coverage must not fall |
| `coverage.lines` | `up` | Line coverage must not fall |
| `coverage.functions` | `up` | Function coverage must not fall |
| `coverage.branches` | `up` | Branch coverage must not fall |
To update the baseline after a genuine improvement:
```bash
npm run quality:ratchet -- --update
git add quality-baseline.json
```
The `--update` flag writes the current measured values into `quality-baseline.json`.
Commit this file alongside the change that improved the metric. A PR that improves a
metric without updating the baseline will be caught by `--require-tighten` (Fase 6A.5,
pending implementation).
---
## Allowlist Policy
Every gate that cannot fail on pre-existing violations uses a frozen allowlist
(e.g., `KNOWN_STALE_DOC_REFS`, `KNOWN_MISSING`, `KNOWN_RAW_SQL`). The policy is:
**Fix the root cause; use the allowlist only when the violation is pre-existing and
cannot be fixed in the same PR.**
When adding an entry to an allowlist:
1. Include a comment with the justification.
2. Reference the tracking issue (e.g., `// #3498 — Phase 2 feature, not yet implemented`).
3. Remove the entry in the same PR that fixes the violation — a stale entry that no longer
suppresses an active violation is itself a defect (6A.3 stale-enforcement will
fail the gate on an orphaned allowlist entry once implemented).
Do **not** add allowlist entries to make tests pass faster. A green gate with a growing
allowlist is a false sense of quality.
### When a gate fails on your PR
1. **Read the gate output carefully** — it tells you exactly which file or symbol violated
the rule.
2. **Fix the violation** — most gates are deterministic filesystem checks that pass as soon
as the code is correct.
3. **If the violation is pre-existing** (i.e., you did not introduce it but the gate now
covers it): add an allowlist entry with a justification comment and a tracking issue.
4. **If the gate is a ratchet** (coverage, ESLint warnings, duplication, complexity):
your change made the metric worse. Fix the underlying issue, or (rarely) run
`npm run quality:ratchet -- --update` if the change is intentional and the metric
degradation is acceptable — but document why in the PR description.
5. **Advisory gates** (`continue-on-error: true`) are informational — they do not block
merge but appear in the CI summary. Fix them anyway.
---
## Adding a New Gate
1. Create `scripts/check/check-<name>.mjs` (or `.ts`). Policy gates exit 0/1.
Ratchet-style gates emit a metric to `quality-metrics.json` via `collect-metrics.mjs`.
2. Add `"check:<name>": "node scripts/check/check-<name>.mjs"` to `package.json`.
3. Wire it in `.github/workflows/ci.yml` under the appropriate job
(policy → `lint` or `docs-sync-strict`; ratchet → `quality-gate`).
4. If it has an allowlist, apply `reportStaleEntries()` from
`scripts/check/lib/allowlist.mjs` so stale entries are detected automatically.
5. Write a test in `tests/unit/build/` covering the gate's detection logic.
6. Update this document (add a row to the relevant job table).
---
## Agent tooling: LSP-in-the-loop (opt-in)
Beyond the CI gates, OmniRoute ships an **opt-in** `agent-lsp` scaffold
([`.mcp.json.example`](../../.mcp.json.example), Fase 7 Task 15). Copy it to `.mcp.json`
to expose a TypeScript language server to coding agents, so they resolve symbols /
diagnostics **before** writing code — a compile-before-claim companion to
`typecheck:core` that cuts "invented symbol" errors at the source. It is intentionally
not auto-loaded (you pick and verify the MCP↔LSP bridge); a broken entry only logs a
connection error and never breaks sessions.
---
## Rationalization Backlog (ROI review — Fase 9 Onda 3)
This inventory was reconciled against `ci.yml` on 2026-06-17 (the prior version omitted
`audit:deps`, `check:tracked-artifacts`, `check:lockfile`, `check:licenses`,
`check:dead-code`, `check:cognitive-complexity`, `check:type-coverage`,
`check:codeql-ratchet`, `check:pr-evidence`). An ROI review of the reconciled set
identified the following rationalization candidates. **The merges are mechanical CI
changes; the flips/drops are policy decisions reserved for the operator.** Nothing below
is applied yet.
**Also undocumented above** (advisory, low signal): the `docs-lint` job
(markdownlint + Vale, whole job `continue-on-error`) and the standalone scanner workflows
`semgrep.yml` / `codeql.yml` / `scorecard.yml`. `semgrepFindings: 0` is in
`quality-baseline.json` but is not wired to a blocking ratchet in `ci.yml` — the metric is
currently orphaned.
### Merge / dedup (mechanical, lower risk)
Each candidate was validated against the live gate state on 2026-06-17 (trust-but-verify);
several "obvious" merges turned out to hide debt and are **not** clean drop-ins.
- **`check:docs-sync` runs twice** — standalone in the `lint` job and again inside `check:docs-all` (`docs-sync-strict`) and the husky pre-commit hook. ✅ **DONE** — standalone `lint` invocation removed.
- **CVE scanning** — ❌ **NOT a clean merge.** `audit:deps` hard-fails on any high/critical CVE; `check:vuln-ratchet` (osv) only fails on a *regression* vs baseline (currently 1 MODERATE). Different semantics — dropping `audit:deps` would lose the absolute high/critical gate. Keep both.
- **Cycle detection** — ❌ **NOT a clean merge.** `check:circular-deps` (dpdm) reports **91 cycles** (that is why it is advisory); it cannot be promoted to blocking without first resolving them, and it has a broader scope than the green, curated `check:cycles`. Keep `check:cycles` blocking; resolving the 91 dpdm cycles is its own backlog.
- **Complexity** — ⏳ valid but real surgery. `check:complexity` (core ESLint) + `check:cognitive-complexity` (sonarjs) are two ESLint passes over `src` + `open-sse`; merging into one config emitting both metrics needs careful ratchet re-wiring. Deferred.
- **`/api` anti-hallucination** — ⏳ valid but script surgery. `check:openapi-routes` (spec→route) + `check:docs-symbols` (prose→route) share resolution logic; collapsing them is a non-trivial script change. Deferred.
- **`check:node-runtime` runs in 11 jobs** — ⚠️ **low ROI.** Each is a separate runner and the check is <1s; total savings ~10s, against losing a cheap per-job guard. Not worth the churn.
### Flip / decide (operator policy)
- `check:openapi-security-tiers` (advisory) — ❌ **NOT cleanly flippable.** It exits 0 but warns that several `traffic-inspector` routes under `LOCAL_ONLY_API_PREFIXES` lack the `x-loopback-only: true` annotation. Enforcing it requires adding those annotations to `openapi.yaml` first.
- `typecheck:noimplicit:core` (advisory) — largely subsumed by the blocking `check:type-coverage` ratchet. Flip to a ratchet or drop the redundant second `tsc` pass.
- `test:vitest:ui` (advisory, 14 parked fails) — fix-and-block or delete; don't leave rotting.
- `check:secrets` (gitleaks, blocking ratchet frozen at 3 documented false-positives) — allowlist the 3 to reach 0, or demote to advisory. Overlaps GitHub native secret-scanning + `check:public-creds`.
- `check:pr-evidence` (blocking, greps PR-body prose) — high false-positive risk; weakens Hard Rule #18 enforcement if dropped, so this is a genuine policy call.
- `semgrep` (advisory standalone) — overlaps CodeQL for the OWASP families; wire its baseline to a ratchet or drop.
---
## Related Documentation
- Supply-chain (provenance, SBOM, Trivy, Scorecard): [`docs/security/SUPPLY_CHAIN.md`](../security/SUPPLY_CHAIN.md)
Reference in New Issue View Git Blame Copy Permalink