# semcheck.yaml — docs↔code semantic consistency (Task 14 — Fase 7) # # PURPOSE # Fuzzy (LLM-assisted) layer that catches documentation describing behaviour # that the code no longer implements. Complements the deterministic # check-docs-symbols.mjs (Fase 6) which catches renamed/deleted exports. # semcheck catches semantic drift: a doc that says "returns 200 on success" # when the code now returns 204, or that describes a feature that was removed. # # OPERATING MODE — ADVISORY / NON-BLOCKING # - This gate is LLM-backed → runs in the NIGHTLY job only (not per-PR). # - It NEVER blocks a PR merge automatically. # - Findings appear as annotations in the CI summary and as a comment on the # PR if triggered by a label (see ci.yml `semcheck` job). # - To trigger on a specific PR: add the label `semcheck` to the PR. # - To promote to blocking: set `fail-on-issues: true` and wire the job # under `required-checks` in the branch protection rule. # # FAIL MODE # fail-on-issues: false ← advisory (current) # fail-on-issues: true ← blocking (future, opt-in) # # TOOL # semcheck OSS (https://github.com/semcheck/semcheck — MIT). # Install: npm install --save-dev semcheck (not bundled; nightly-only) # Run: npx semcheck --config semcheck.yaml # # ADDING NEW RULES # 1. Pick a doc file and the matching source file(s). # 2. Write a `claim` that the doc makes about the code behaviour. # 3. Optionally add `context-file` paths for the LLM to read. # 4. Set `severity: warning` for informational, `error` for blocking candidates. # Rules are evaluated independently; a failure in one does not block others. version: "1" fail-on-issues: false # ADVISORY — change to true to make blocking model: "gpt-4o-mini" # cheap model for advisory pass; upgrade to gpt-4o for precision rules: # ── Routing & Combo ────────────────────────────────────────────────────── - id: combo-strategies-count doc: docs/routing/AUTO-COMBO.md claim: > The document states that OmniRoute supports 14 combo routing strategies. Verify that open-sse/services/combo.ts exports or references exactly 14 strategy identifiers (or that any discrepancy has a comment explaining why). context-files: - open-sse/services/combo.ts severity: warning - id: auto-combo-9-factors doc: docs/routing/AUTO-COMBO.md claim: > The document describes exactly 9 scoring factors for the Auto-Combo strategy. Verify that the implementation references 9 distinct factors (e.g. in the autoCombo scoring function or its constants/comments). context-files: - open-sse/services/autoCombo/ severity: warning - id: resilience-3-layers doc: docs/architecture/RESILIENCE_GUIDE.md claim: > The document describes 3 and only 3 resilience mechanisms: Provider Circuit Breaker, Connection Cooldown, and Model Lockout. Verify that no 4th mechanism is implemented without being documented. context-files: - src/shared/utils/circuitBreaker.ts - open-sse/services/accountFallback.ts severity: warning # ── Security ───────────────────────────────────────────────────────────── - id: error-sanitization-flow doc: docs/security/ERROR_SANITIZATION.md claim: > The document states that all error responses MUST route through buildErrorBody() or sanitizeErrorMessage() from open-sse/utils/error.ts. Verify that the described API (function names, module path) matches what is actually exported in that file. context-files: - open-sse/utils/error.ts severity: error - id: public-creds-pattern doc: docs/security/PUBLIC_CREDS.md claim: > The document describes a mandatory resolvePublicCred() function in open-sse/utils/publicCreds.ts. Verify that function exists with roughly the API signature the doc describes (accepts a key, returns a string). context-files: - open-sse/utils/publicCreds.ts severity: error # ── MCP Server ─────────────────────────────────────────────────────────── - id: mcp-tool-count doc: docs/frameworks/MCP-SERVER.md claim: > The document states the MCP server exposes 43 tools in total (30 base + 3 memory + 4 skills + 6 notion). Verify this count is consistent with the number of tool definitions found in open-sse/mcp-server/tools/. context-files: - open-sse/mcp-server/tools/ severity: warning # ── A2A Server ─────────────────────────────────────────────────────────── - id: a2a-skills-count doc: docs/frameworks/A2A-SERVER.md claim: > The document lists exactly 5 built-in A2A skills: smart-routing, quota-management, provider-discovery, cost-analysis, health-report. Verify that src/lib/a2a/skills/ contains entries matching these names and that no additional undocumented skills exist. context-files: - src/lib/a2a/skills/ - src/lib/a2a/taskExecution.ts severity: warning # ── API Reference ───────────────────────────────────────────────────────── - id: api-route-v1-prefix doc: docs/reference/API_REFERENCE.md claim: > The document states all API routes are under /v1/ (e.g. /v1/chat/completions). Verify that src/app/api/v1/ exists and that the key routes mentioned in the doc (chat/completions, embeddings, models) are present. context-files: - src/app/api/v1/ severity: warning # ── Embedded Services ───────────────────────────────────────────────────── - id: embedded-services-api-pattern doc: docs/frameworks/EMBEDDED-SERVICES.md claim: > The document says each embedded service exposes 7 API endpoints: install, start, stop, restart, update, status, auto-start (plus a shared logs endpoint). Verify that at least one service under src/app/api/services/ follows this pattern. context-files: - src/app/api/services/ severity: warning # ── Route Guard ─────────────────────────────────────────────────────────── - id: route-guard-local-only doc: docs/security/ROUTE_GUARD_TIERS.md claim: > The document describes LOCAL_ONLY_API_PREFIXES enforced by src/server/authz/routeGuard.ts via isLocalOnlyPath(). Verify these identifiers exist in the source file with roughly the described semantics. context-files: - src/server/authz/routeGuard.ts severity: error