pjandro/OmniRoute
1
0
Fork 0
mirror of https://github.com/diegosouzapw/OmniRoute.git synced 2026-08-15 11:43:10 +03:00
Code Issues Packages Projects Releases Wiki Activity

docs: refresh every stale count to measured values + harden docs-counts gate (#10196)

Browse Source 
* docs(reference): regenerate PROVIDER_REFERENCE from live provider modules

The catalog was hand-stale at 291 since 2026-08-05 while the live provider
modules define 338 unique IDs. The generator also omitted the NOAUTH_PROVIDERS
module entirely (10 providers) and hardcoded the executor count in its footer;
both are now sourced from the live modules.

Refs #9985

* docs: refresh stale counts across README/AGENTS/llm.txt and architecture docs

Every count updated to values measured from the live code on 2026-08-12:
providers 291/271/248/236/226/212->338, migrations 110/117/130->144, MCP tools
94/99/104->105 (base 42->43), scopes 13/32->31, strategies 17/18->19,
Auto-Combo factors 12/13->14 (sessionAvailability row added to the table),
executors 67/78/84/89->101, quality gates ~48->~80, locales 29/30/39/40+->43
(41 non-source), A2A skills 5->6 (list-capabilities), free tier 43 pools/516
models/~1.53B/~2.15B->42/495/~1.51B/~2.13B, contributors 500+->320+ (324
unique emails), llm.txt version 3.8.47->3.8.50. llm.txt i18n mirrors resynced
(headers preserved, body mirrored).

Refs #9985

* docs(diagrams): sync SVG hero/pillars/comparison/cli/tier numbers

Text nodes and aria-labels only; layout, coordinates and animation values
untouched. providers 278/290->338 (cli list footer 264->334 more), MCP tools
104->105, strategies 18->19, free tier 43 pools/460+/516 models->42/495,
headline ~1.53B/~2.15B->~1.51B/~2.13B. All six SVGs re-validated as XML.

Refs #9985

* feat(check): harden docs-counts gate - live provider source, llm.txt, migrations, SVGs

The gate trusted PROVIDER_REFERENCE.md as the provider total, so a hand-stale
doc (291 vs 338 live) kept it falsely green. New STRICT checks: doc total vs
the live provider modules (same collections the generator unions), provider
count in llm.txt and package.json description, migration count vs
README/AGENTS/llm.txt, and a canonical-number sweep (providers/MCP
tools/strategies/pools) over the six README SVG diagrams with
coordinate/attribute-safe patterns. TDD: 9 new unit tests (red first on the
missing exports, green after) in tests/unit/check-docs-counts-sync.test.ts.

Refs #9985

* docs(readme): refresh What's New range and add v3.8.50 cycle highlights

---------

Co-authored-by: backryun <bakryun0718@proton.me>
This commit is contained in:
Diego Rodrigues de Sa e Souza
2026-08-12 14:24:56 -03:00
committed by GitHub
parent be62ca04f7
commit 8bd17be8aa
69 changed files with 1699 additions and 1361 deletions
Download Patch File Download Diff File Expand all files Collapse all files

149
scripts/check/check-docs-counts-sync.mjs
View File

@@ -18,9 +18,13 @@
// Exits 0 on success, 1 on STRICT drift (or any drift with --strict).
// Run: node scripts/check/check-docs-counts-sync.mjs
//
// NOTE: the provider check trusts PROVIDER_REFERENCE.md as the canonical total. If a
// provider is added to the code but the reference is not regenerated, this guard will
// not catch it — regenerate with `npm run gen:provider-reference` before relying on it.
// NOTE: PROVIDER_REFERENCE.md is no longer blindly trusted — a STRICT check compares
// the doc's `Total providers` against the live provider modules (the same collections
// the generator reads), so a hand-stale doc is a red, not a silently propagated total.
// Fix by running `npm run gen:provider-reference`. Additional STRICT coverage added in
// the 2026-08-12 hardening: llm.txt + package.json description (providers), migration
// count (README/AGENTS/llm.txt), and canonical numbers inside the README SVG diagrams
// (providers / MCP tools / routing strategies / free-tier pools).
import fs from "node:fs";
import { spawnSync } from "node:child_process";
@@ -76,6 +80,13 @@ export function readProviderTotal() {
return parseProviderTotal(fs.readFileSync(abs, "utf8"));
}
// STRICT: number of SQL migration files shipped with the app.
export function countMigrations() {
const abs = path.join(ROOT, "src", "lib", "db", "migrations");
if (!fs.existsSync(abs)) return 0;
return fs.readdirSync(abs).filter((f) => f.endsWith(".sql")).length;
}
// STRICT: canonical i18n locale count, read from the shared config.
export function countLocales() {
const abs = path.join(ROOT, "config", "i18n.json");
@@ -148,6 +159,15 @@ function readCodeFacts() {
'import {notionTools} from "./open-sse/mcp-server/tools/notionTools.ts";',
'import {obsidianTools} from "./open-sse/mcp-server/tools/obsidianTools.ts";',
'import {compressionTools} from "./open-sse/mcp-server/tools/compressionTools.ts";',
// Live provider total — the SAME collections gen-provider-reference.ts unions, so the
// doc-vs-live check below cannot drift from the generator's definition of "provider".
'import * as PROV from "./src/shared/constants/providers.ts";',
"const provCols=[PROV.FREE_PROVIDERS,PROV.NOAUTH_PROVIDERS,PROV.OAUTH_PROVIDERS,",
"PROV.WEB_COOKIE_PROVIDERS,PROV.APIKEY_PROVIDERS,PROV.LOCAL_PROVIDERS,PROV.SEARCH_PROVIDERS,",
"PROV.AUDIO_ONLY_PROVIDERS,PROV.UPSTREAM_PROXY_PROVIDERS,PROV.CLOUD_AGENT_PROVIDERS,",
"PROV.SYSTEM_PROVIDERS];",
"const pids=new Set();",
"for(const c of provCols)for(const p of Object.values(c||{}))if(p&&p.id)pids.add(p.id);",
"const cols={MCP_TOOLS,memoryTools,skillTools,agentSkillTools,githubSkillTools,poolTools,",
"gamificationTools,pluginTools,notionTools,obsidianTools,compressionTools};",
"const sc=new Set();",
@@ -158,7 +178,7 @@ function readCodeFacts() {
'console.log("@@"+JSON.stringify({freeSteady:t.steadyRecurringTokens,',
"freeFirst:t.firstMonthRealisticTokens,freePools:t.poolCount,engines:ENGINE_IDS.length,",
"cliTotal:cli.length,cliCode:by('code'),cliAgent:by('agent'),",
"mcpTools:countUniqueMcpTools(cols),mcpScopes:sc.size}));",
"mcpTools:countUniqueMcpTools(cols),mcpScopes:sc.size,providers:pids.size}));",
].join("");
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "docs-counts-"));
try {
@@ -252,6 +272,82 @@ export function makeNumberClaimValidator(expected, opts) {
};
}
// --- v3.8.50 hardening validators --------------------------------------------
// PURE: doc total must equal the live provider-module total (closes the falso-verde
// found in the 2026-08-12 audit: the doc sat hand-stale at 291 while the modules
// defined 338, and every downstream check inherited the stale total).
export function makeProviderReferenceValidator(expected) {
return (content) => {
const total = parseProviderTotal(content);
if (!total) return { ok: false, detail: "no `Total providers: **N**` marker found" };
if (total === expected)
return { ok: true, detail: `doc total ${total} matches the live provider modules` };
return {
ok: false,
detail:
`doc total ${total} is stale — the live provider modules define ${expected} ` +
`(run npm run gen:provider-reference)`,
};
};
}
// PURE: the npm package description must carry the live provider count.
export function makePackageDescriptionValidator(expected) {
return (content) => {
let desc = "";
try {
desc = String(JSON.parse(content).description || "");
} catch {
return { ok: false, detail: "package.json could not be parsed" };
}
if (desc.includes(String(expected)))
return { ok: true, detail: `description mentions the live provider count ${expected}` };
return {
ok: false,
detail: `description does not mention the live provider count ${expected}: "${desc}"`,
};
};
}
// PURE: sweep an SVG's text/aria content for the canonical numbers. Patterns are
// deliberately narrow — they anchor on the surrounding words so path coordinates,
// width/font-size attributes and small unrelated counts ("15 providers ToS-flagged",
// "100+ providers") can never register as claims. Providers require 3+ digits for the
// same reason.
const SVG_CANONICAL_PATTERNS = [
{ key: "providers", what: "providers", pattern: /(\d{3,4}) (?:AI )?providers\b/g },
{ key: "mcpTools", what: "MCP tools", pattern: /MCP (?:server with |with |\()(\d+)/g },
{ key: "strategies", what: "routing strategies", pattern: /(\d+) routing strategies\b/g },
{ key: "pools", what: "free-tier pools", pattern: /(\d+) provider pools\b/g },
];
export function checkSvgCanonicalNumbers(content, expected) {
const stale = [];
let claims = 0;
for (const { key, what, pattern } of SVG_CANONICAL_PATTERNS) {
if (expected[key] == null) continue;
for (const m of content.matchAll(pattern)) {
claims++;
const value = Number(m[1]);
if (value !== expected[key]) stale.push(`"${m[0]}" (${what} — code has ${expected[key]})`);
}
}
if (!claims) return { ok: true, detail: "no canonical-number claims in this SVG" };
if (!stale.length) return { ok: true, detail: `${claims} canonical claim(s) match the code` };
return { ok: false, detail: `stale: ${[...new Set(stale)].join(", ")}` };
}
// The README-embedded diagrams that historically rotted because no gate read them
// (the alt-text in README.md is checked, the SVG text nodes never were).
const SVG_DIAGRAM_FILES = [
"docs/diagrams/readme-hero.svg",
"docs/diagrams/free-tier-budget.svg",
"docs/diagrams/promise-pillars.svg",
"docs/diagrams/comparison-table.svg",
"docs/diagrams/cli-terminal.svg",
"docs/diagrams/tier-cascade.svg",
];
export function buildChecks() {
return [
{
@@ -259,7 +355,26 @@ export function buildChecks() {
actual: readProviderTotal(),
docKey: "providers",
strict: true,
files: ["README.md", "AGENTS.md"],
files: ["README.md", "AGENTS.md", "llm.txt"],
},
{
label: "Provider count (package.json description)",
actual: readProviderTotal(),
docKey: "providers",
strict: true,
files: ["package.json"],
validate: makePackageDescriptionValidator(readProviderTotal()),
},
{
label: "DB migrations count",
actual: countMigrations(),
docKey: "migrations",
strict: true,
files: ["README.md", "AGENTS.md", "llm.txt"],
validate: makeNumberClaimValidator(countMigrations(), {
what: "migrations",
pattern: /(\d+)\+? migrations?\b/gi,
}),
},
{
label: "i18n locales count",
@@ -289,6 +404,30 @@ export function buildChecks() {
validate: makeNumberClaimValidator(expected, { what, ...opts }),
});
return [
{
label: "Provider reference total (doc vs live modules)",
actual: f.providers,
docKey: "providers (live)",
strict: true,
files: ["docs/reference/PROVIDER_REFERENCE.md"],
validate: makeProviderReferenceValidator(f.providers),
},
{
label: "SVG canonical numbers (live code)",
actual:
`${f.providers} providers / ${f.mcpTools} MCP tools / ` +
`${countRoutingStrategies()} strategies / ${f.freePools} pools`,
docKey: "SVG canonical numbers",
strict: true,
files: SVG_DIAGRAM_FILES,
validate: (content) =>
checkSvgCanonicalNumbers(content, {
providers: f.providers,
mcpTools: f.mcpTools,
strategies: countRoutingStrategies(),
pools: f.freePools,
}),
},
{
label: "Free-tier headline (live catalog)",
actual: `~${(f.freeSteady / 1e9).toFixed(2)}B steady / ${f.freePools} pools`,

24
scripts/docs/gen-provider-reference.ts
View File

@@ -7,6 +7,7 @@ import path from "node:path";
import { fileURLToPath } from "node:url";
import {
FREE_PROVIDERS,
NOAUTH_PROVIDERS,
OAUTH_PROVIDERS,
WEB_COOKIE_PROVIDERS,
APIKEY_PROVIDERS,
@@ -136,6 +137,7 @@ function buildHeader(total: number): string {
"## Categories",
"",
"- **Free** — free tier with API key (configured via dashboard)",
"- **No-auth** — public endpoints that require no key or sign-in at all",
"- **OAuth** — sign-in flow handled by OmniRoute, no API key needed",
"- **Web cookie** — wraps the provider's web app via cookie auth",
"- **API key** — paid provider configured via API key (free credits may apply)",
@@ -159,8 +161,19 @@ function buildHeader(total: number): string {
].join("\n");
}
function countExecutorImpls(): number {
const dir = path.join(ROOT, "open-sse", "executors");
const nonImpl = new Set(["index.ts", "index.mts", "types.ts", "base.ts", "constants.ts"]);
return fs
.readdirSync(dir)
.filter(
(f) => f.endsWith(".ts") && !f.endsWith(".test.ts") && !f.startsWith("__") && !nonImpl.has(f)
).length;
}
function main() {
const free = asRecords(FREE_PROVIDERS);
const noauth = asRecords(NOAUTH_PROVIDERS as Record<string, ProviderRecord>);
const oauth = asRecords(OAUTH_PROVIDERS);
const webCookie = asRecords(WEB_COOKIE_PROVIDERS);
const apiKey = asRecords(APIKEY_PROVIDERS);
@@ -173,6 +186,7 @@ function main() {
const allIds = new Set<string>([
...free.map((p) => p.id),
...noauth.map((p) => p.id),
...oauth.map((p) => p.id),
...webCookie.map((p) => p.id),
...apiKey.map((p) => p.id),
@@ -186,6 +200,7 @@ function main() {
const sections = [
buildSection("Free Tier (OAuth-first or no-key)", free, "Free"),
buildSection("No-auth Providers (no key required)", noauth, "No-auth"),
buildSection("OAuth Providers", oauth, "OAuth"),
buildSection("Web Cookie Providers", webCookie, "Web cookie"),
buildSection("API Key Providers (paid / paid-with-free-credits)", apiKey, "API key"),
@@ -202,7 +217,7 @@ function main() {
"",
"- Catalog: [`src/shared/constants/providers.ts`](../../src/shared/constants/providers.ts)",
"- Registry (per-model details): [`open-sse/config/providerRegistry.ts`](../../open-sse/config/providerRegistry.ts)",
"- Executors: [`open-sse/executors/`](../../open-sse/executors/) (31 files)",
`- Executors: [\`open-sse/executors/\`](../../open-sse/executors/) (${countExecutorImpls()} implementations)`,
"- Translators: [`open-sse/translator/`](../../open-sse/translator/)",
"",
"## See Also",
@@ -218,9 +233,10 @@ function main() {
console.log(`✓ Wrote ${OUT_FILE}`);
console.log(` Providers: ${allIds.size} unique IDs`);
console.log(
` Sections: free=${free.length}, oauth=${oauth.length}, web=${webCookie.length}, ` +
`apikey=${apiKey.length}, local=${local.length}, search=${search.length}, ` +
`audio=${audio.length}, proxy=${upstreamProxy.length}, cloud=${cloudAgent.length}, system=${system.length}`
` Sections: free=${free.length}, noauth=${noauth.length}, oauth=${oauth.length}, ` +
`web=${webCookie.length}, apikey=${apiKey.length}, local=${local.length}, ` +
`search=${search.length}, audio=${audio.length}, proxy=${upstreamProxy.length}, ` +
`cloud=${cloudAgent.length}, system=${system.length}`
);
}