OmniRoute/semcheck.yaml

# semcheck.yaml — docs↔code semantic consistency (Task 14 — Fase 7)
#
# PURPOSE
#   Fuzzy (LLM-assisted) layer that catches documentation describing behaviour
#   that the code no longer implements. Complements the deterministic
#   check-docs-symbols.mjs (Fase 6) which catches renamed/deleted exports.
#   semcheck catches semantic drift: a doc that says "returns 200 on success"
#   when the code now returns 204, or that describes a feature that was removed.
#
# OPERATING MODE  — ADVISORY / NON-BLOCKING
#   - This gate is LLM-backed → runs in the NIGHTLY job only (not per-PR).
#   - It NEVER blocks a PR merge automatically.
#   - Findings appear as annotations in the CI summary and as a comment on the
#     PR if triggered by a label (see ci.yml `semcheck` job).
#   - To trigger on a specific PR: add the label `semcheck` to the PR.
#   - To promote to blocking: set `fail-on-issues: true` and wire the job
#     under `required-checks` in the branch protection rule.
#
# FAIL MODE
#   fail-on-issues: false   ← advisory (current)
#   fail-on-issues: true    ← blocking (future, opt-in)
#
# TOOL
#   semcheck OSS (https://github.com/semcheck/semcheck — MIT).
#   Install: npm install --save-dev semcheck  (not bundled; nightly-only)
#   Run:     npx semcheck --config semcheck.yaml
#
# ADDING NEW RULES
#   1. Pick a doc file and the matching source file(s).
#   2. Write a `claim` that the doc makes about the code behaviour.
#   3. Optionally add `context-file` paths for the LLM to read.
#   4. Set `severity: warning` for informational, `error` for blocking candidates.
#   Rules are evaluated independently; a failure in one does not block others.

version: "1"

fail-on-issues: false   # ADVISORY — change to true to make blocking

model: "gpt-4o-mini"    # cheap model for advisory pass; upgrade to gpt-4o for precision

rules:

  # ── Routing & Combo ──────────────────────────────────────────────────────

  - id: combo-strategies-count
    doc: docs/routing/AUTO-COMBO.md
    claim: >
      The document states that OmniRoute supports 14 combo routing strategies.
      Verify that open-sse/services/combo.ts exports or references exactly 14
      strategy identifiers (or that any discrepancy has a comment explaining why).
    context-files:
      - open-sse/services/combo.ts
    severity: warning

  - id: auto-combo-9-factors
    doc: docs/routing/AUTO-COMBO.md
    claim: >
      The document describes exactly 9 scoring factors for the Auto-Combo
      strategy. Verify that the implementation references 9 distinct factors
      (e.g. in the autoCombo scoring function or its constants/comments).
    context-files:
      - open-sse/services/autoCombo/
    severity: warning

  - id: resilience-3-layers
    doc: docs/architecture/RESILIENCE_GUIDE.md
    claim: >
      The document describes 3 and only 3 resilience mechanisms: Provider
      Circuit Breaker, Connection Cooldown, and Model Lockout. Verify that no
      4th mechanism is implemented without being documented.
    context-files:
      - src/shared/utils/circuitBreaker.ts
      - open-sse/services/accountFallback.ts
    severity: warning

  # ── Security ─────────────────────────────────────────────────────────────

  - id: error-sanitization-flow
    doc: docs/security/ERROR_SANITIZATION.md
    claim: >
      The document states that all error responses MUST route through
      buildErrorBody() or sanitizeErrorMessage() from open-sse/utils/error.ts.
      Verify that the described API (function names, module path) matches what
      is actually exported in that file.
    context-files:
      - open-sse/utils/error.ts
    severity: error

  - id: public-creds-pattern
    doc: docs/security/PUBLIC_CREDS.md
    claim: >
      The document describes a mandatory resolvePublicCred() function in
      open-sse/utils/publicCreds.ts. Verify that function exists with roughly
      the API signature the doc describes (accepts a key, returns a string).
    context-files:
      - open-sse/utils/publicCreds.ts
    severity: error

  # ── MCP Server ───────────────────────────────────────────────────────────

  - id: mcp-tool-count
    doc: docs/frameworks/MCP-SERVER.md
    claim: >
      The document states the MCP server exposes 43 tools in total (30 base +
      3 memory + 4 skills + 6 notion). Verify this count is consistent with
      the number of tool definitions found in open-sse/mcp-server/tools/.
    context-files:
      - open-sse/mcp-server/tools/
    severity: warning

  # ── A2A Server ───────────────────────────────────────────────────────────

  - id: a2a-skills-count
    doc: docs/frameworks/A2A-SERVER.md
    claim: >
      The document lists exactly 5 built-in A2A skills: smart-routing,
      quota-management, provider-discovery, cost-analysis, health-report.
      Verify that src/lib/a2a/skills/ contains entries matching these names
      and that no additional undocumented skills exist.
    context-files:
      - src/lib/a2a/skills/
      - src/lib/a2a/taskExecution.ts
    severity: warning

  # ── API Reference ─────────────────────────────────────────────────────────

  - id: api-route-v1-prefix
    doc: docs/reference/API_REFERENCE.md
    claim: >
      The document states all API routes are under /v1/ (e.g. /v1/chat/completions).
      Verify that src/app/api/v1/ exists and that the key routes mentioned in
      the doc (chat/completions, embeddings, models) are present.
    context-files:
      - src/app/api/v1/
    severity: warning

  # ── Embedded Services ─────────────────────────────────────────────────────

  - id: embedded-services-api-pattern
    doc: docs/frameworks/EMBEDDED-SERVICES.md
    claim: >
      The document says each embedded service exposes 7 API endpoints:
      install, start, stop, restart, update, status, auto-start (plus a shared
      logs endpoint). Verify that at least one service under src/app/api/services/
      follows this pattern.
    context-files:
      - src/app/api/services/
    severity: warning

  # ── Route Guard ───────────────────────────────────────────────────────────

  - id: route-guard-local-only
    doc: docs/security/ROUTE_GUARD_TIERS.md
    claim: >
      The document describes LOCAL_ONLY_API_PREFIXES enforced by
      src/server/authz/routeGuard.ts via isLocalOnlyPath(). Verify these
      identifiers exist in the source file with roughly the described semantics.
    context-files:
      - src/server/authz/routeGuard.ts
    severity: error