mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-08-05 06:42:12 +03:00
* chore(release): open v3.8.23 development cycle
* fix(anthropic): strip top_p when temperature is set to avoid 400 (#3691)
Integrated into release/v3.8.23
* fix(vertex): support Vertex AI Express-mode API keys (#3690)
Integrated into release/v3.8.23
* fix(stream): error on empty Claude SSE instead of synthetic success (#3689)
Integrated into release/v3.8.23
* fix(oauth): stop token-refresh invalidation loop + harden proxy resolution (#3692)
Integrated into release/v3.8.23
* docs: add FUNDING.yml and Support section to README (#3698)
Integrated into release/v3.8.23
* feat: gemini - handle known ratelimits (#3686)
Integrated into release/v3.8.23
* fix: stream combo fails over on empty content-filtered response (#3685) (#3702)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(antigravity): preserve gemini-3.1-pro high/low budget tiers (#3696) (#3703)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(auto-combo): add auto-updating model intelligence scoring (#3660)
Integrated into release/v3.8.23
* fix(gemini): context-mode fallback for signatureless tool calls (#3688) (#3704)
* chore(quality-gate): reconcile file-size baseline (27 files + providerLimits.ts) (#3705)
* feat(vertex): dynamic model discovery via Generative Language models API (#3712)
Integrated into release/v3.8.23. Vertex dynamic model discovery — surfaces image models (imagen-*, gemini-*-image), embeddings and audio from the live Generative Language catalog, with cached→static fallback and the shared parseGeminiModelsList helper. Validated: parser test 5/5, typecheck:core clean.
* fix(combo): gate reasoning token buffer (#3700)
Integrated into release/v3.8.23. Makes the #3588 reasoning token buffer safe and configurable: only inflates max_tokens when the model has a known, non-default output cap and the buffered value fits inside it; otherwise preserves/clamps the client limit. Adds the reasoningTokenBufferEnabled kill switch (default ON). Validated: combo-routing-engine 81/81, combo-config 25/25, combo-quality-validator-reasoning 12/12, phase1f 10/10, typecheck:core clean.
* refactor(#3501): god-component Phase 1g-1j — client 4062→3408 LOC (-654) (#3717)
Phase 1g-1j of #3501: client 4062→3408 LOC. Pure extraction (ProviderPlaygroundPanel, useCommandCodeAuth, useExternalLinkFlow+ExternalLinkModal, useAuthFileHandlers) + loadConnProxies ReferenceError fix + phase1f test path fix.
Co-authored-by: oyi77 <14921983+oyi77@users.noreply.github.com>
* refactor(#3501): god-component Phase 1k-1m — client 3408→2553 LOC (-855) (#3721)
Phase 1k-1m of #3501: client 3408→2553 LOC. Pure extraction (useModelImportHandlers+ImportProgressModal, useModelVisibilityHandlers, ProviderModelsSection).
Co-authored-by: oyi77 <14921983+oyi77@users.noreply.github.com>
* docs(changelog): restore #3590 bullet lost on the v3.8.20 release branch
The fix itself reached main pre-tag via cherry-pick #3591, but its changelog
bullet (commit e33fdd4ab) only ever existed on release/v3.8.20 after the
squash-merge. Restored under [3.8.20] per the 2026-06-12 release-branch
leftover audit (_tasks/release-audit/release-leftovers-audit-2026-06-12.md).
* fix(kiro): resolve quota for IAM Identity Center accounts missing a profileArn (#3722)
Integrated into release/v3.8.23
* refactor(#3501): god-component Phase 1n-1s — client 2553→1376 LOC (-1177) (#3725)
Phase 1n-1s of #3501: client 2553→1376 LOC. Pure extraction (ConnectionsListPanel, ConnectionsHeaderToolbar, ZedImportCard, BatchTestResultsModal, AdaptaTutorialModal, useApiKeySave + helpers).
Co-authored-by: oyi77 <14921983+oyi77@users.noreply.github.com>
* feat(model-lockout): settings UI, backend integration, error classification, and success-decay recovery (#3629)
Integrated into release/v3.8.23
* refactor(#3501): god-component Phase 1t — client 1376→781 LOC (≤800 TARGET REACHED ✅) (#3727)
Phase 1t of #3501: client 1376→781 LOC (≤800 reached). Original god-component 12,882→781 (−94%).
Co-authored-by: oyi77 <14921983+oyi77@users.noreply.github.com>
* fix: bundle @omniroute/opencode-plugin inside omniroute + add 'setup opencode' CLI command (#3726)
Integrated into release/v3.8.23
* feat(vertex): self-tracked USD spend since account added (#3724)
Integrated into release/v3.8.23
* fix(qwen-web): migrate to v2 chat API with full cookie-jar replay (#3288) (#3723)
Integrated into release/v3.8.23
* fix(sse): make safeLogEvents async — 'await' in a sync function broke every chatHelpers import
#3692 added a lazy 'await import(proxyEgress)' for egress-IP visibility inside
safeLogEvents, which is a sync function — an ES syntax error. It went unnoticed
because typecheck:core does not cover src/sse and no test in the merge gates
loaded chatHelpers via tsx; any consumer that did (chat-context-relay and
chat-route-coverage suites, integration harnesses) failed at module load with
'await can only be used inside an async function'.
safeLogEvents is fire-and-forget logging with an outer try/catch, so making it
async (and 'void'-ing the single chat.ts call site) preserves behavior exactly.
Validation: tests/unit/chat-context-relay.test.ts + chat-route-coverage.test.ts
went from failing-at-load to green (+14 tests destravados).
* fix(sse): remove cross-provider credential leak in emergency fallback + combo/proxy audit fixes (#3699)
Integrated into release/v3.8.23
* fix(executors): inject MiMoCode anti-abuse marker so free endpoint stops 403ing (#3728)
Integrated into release/v3.8.23
* fix(dashboard): repair "Test all models" — toast crash, status icons, auto-hide (#3729)
Integrated into release/v3.8.23
* chore(deps): bump actions/upload-artifact from 4 to 7 (#3735)
Integrated into release/v3.8.23 — aligns upload-artifact to v7 (already used across ci.yml).
* chore(deps): bump actions/cache from 4 to 5 (#3734)
Integrated into release/v3.8.23 — actions/cache v4→v5.
* chore(deps): bump actions/download-artifact from 4 to 8 (#3733)
Integrated into release/v3.8.23 — download-artifact v4→v8.
* feat(fallback): add OMNIROUTE_EMERGENCY_FALLBACK env switch (#3741)
Adds an OMNIROUTE_EMERGENCY_FALLBACK env switch to disable the emergency budget-exhaustion fallback (reroute to free nvidia/gpt-oss-120b). Default unchanged (enabled). Closes #3739, related #2879.
Integrated into release/v3.8.23.
* i18n: comprehensive zh-CN translation improvements (#3736)
Aligns zh-CN to en (hundreds of entries), translates batch-action labels + settings sidebar menu, adds categoryConfig/endpointTokenSaver keys, resolves __MISSING__ stubs. Sidebar/SidebarTab hardcoded strings replaced with t(). en.json purely additive (8 new sidebar.* keys, 0 removed); cli-i18n gate green.
Integrated into release/v3.8.23.
* chore(release): v3.8.23 — 2026-06-12
- CHANGELOG: complete v3.8.23 section (28 bullets, 27 commits)
- fix(webdav): resolve promise on writeStream finish, not req end — eliminates
intermittent 500 on PUT update (writeStream may not have flushed at rename time)
- test(autoCombo): stub DB calls from PR #3660 in tieredRotation.test.ts to prevent
5s timeout in vitest (getModelIntelligenceBySource DB init path)
- chore(env-sync): add XDG_DATA_HOME + OMNIROUTE_OPENCODE_PLUGIN_DIR to IGNORE_FROM_CODE
allowlist (introduced by PR #3726 setup-open-code.mjs, not OmniRoute config vars)
- chore(cli): regenerated bin/cli/api-commands/*.mjs (7 new, 27 updated)
* fix(model-family): fallback lookup also tries bare model name with dots
getNextFamilyFallback normalized dots-to-hyphens ("gemini-3.1-pro-high" →
"gemini-3-1-pro-high") but MODEL_FAMILIES keys use the literal dot form. The
lookup always missed, returning null for any model whose dots are part of the
name rather than a version separator.
Fallback: try MODEL_FAMILIES[lookupKey] ?? MODEL_FAMILIES[bareModel] so both
naming conventions are covered. Fixes T30 test (pre-existing since v3.8.22).
* feat: expose API key cost drilldown + quota % used (#3742)
Adds all-time USD cost per API key in the API Key Manager, a per-key deep-link into the Cost Explorer (filtered + grouped by model), URL-param hydration of range/groupBy/apiKeyIds, and a '% used' quota display. Review adjustments: extracted URL-param parsers to a tested module (Rule #18), i18n'd the new strings (en + zh-CN), dropped the redundant webdav-handler entry already on release.
Integrated into release/v3.8.23.
* feat: add provider display modes — All / Configured / Compact (#3743)
Replaces the Providers page configured-only toggle with All/Configured/Compact display modes (Compact = flat deduped grid, no-auth last). Persists the preference and migrates the legacy localStorage key. Rebased onto release/v3.8.23.
Integrated into release/v3.8.23.
* fix(cache): scope semantic-cache signature to API key (#3740)
Adds the api_key_id dimension to generateSignature's SHA-256 hash so two callers with different API keys never receive each other's cached responses. Threads apiKeyId through checkSemanticCache + both write sites; migration 098 clears pre-existing key-less entries; unauthenticated requests stay isolated from keyed ones. 3 TDD tests.
Integrated into release/v3.8.23.
* fix(responses): apply OpenAI Responses API stream=false spec default (#3708)
resolveStreamFlag now applies the stream=false-when-omitted default for sourceFormat=openai-responses (same as the existing claude path), so spec-compliant /v1/responses upstreams that return JSON no longer fall through to the wildcard-Accept heuristic and trigger STREAM_EARLY_EOF / 502. Codex CLI (stream:true) and explicit text/event-stream clients unaffected.
Integrated into release/v3.8.23.
* chore(release): reconcile CI gates for v3.8.23
- file-size baseline: re-freeze 8 files grown by PRs #3742/#3743/#3740
(cost drilldown, provider display modes, cache key isolation)
- ARCHITECTURE.md: update executor count 55→60 (check:docs-counts drift)
- .env.example: add OMNIROUTE_EMERGENCY_FALLBACK (#3741, env-doc-sync)
- CHANGELOG: add formatted bullets for #3742, #3743, #3708, #3740,
model-family-fallback fix; remove duplicate raw ### Fixed section
* test: restore assert count to satisfy check:test-masking gate
Three test files had net assertion removals after behavior-changing PRs:
- chatcore-translation-paths: emergency fallback moved to routing layer
(#3699) — add body error assertion + model-name guard
- executor-vertex-extended: non-JSON is now Express API key (#3690) —
add projects/-path guard to the express-key URL test
- stream-utils: empty streams now emit error (#3685) — add code/message/
status/completePayload guards to both passthrough and translate variants
All new assertions are meaningful (code enum value, 5xx range, non-empty
message, onComplete must-not-fire contract).
* fix(ci): move rtl-logical-classes test to ui/ so vitest:ui runner collects it
---------
Co-authored-by: Felipe Almeman <4226997+zhiru@users.noreply.github.com>
Co-authored-by: NOXX - Commiter <artur1992123@mail.ru>
Co-authored-by: Nick Sullivan <142708+TechNickAI@users.noreply.github.com>
Co-authored-by: Markus Hartung <mail@hartmark.se>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: PizzaV <103120356+pizzav-xyz@users.noreply.github.com>
Co-authored-by: Randi <55005611+rdself@users.noreply.github.com>
Co-authored-by: oyi77 <14921983+oyi77@users.noreply.github.com>
Co-authored-by: Chewji <126886556+Chewji9875@users.noreply.github.com>
Co-authored-by: Hernan Javier Ardila Sanchez <hjasgr@gmail.com>
Co-authored-by: Felipe Sartori <felipesartori.ti@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Zois Pagoulatos <zpagoulatos@hotmail.com>
Co-authored-by: sdfsdfw2 <167810361+sdfsdfw2@users.noreply.github.com>
Co-authored-by: Witroch4 <witalo_rocha@hotmail.com>
828 lines
27 KiB
TypeScript
828 lines
27 KiB
TypeScript
/**
|
|
* Unit tests for src/lib/arenaEloSync.ts
|
|
*
|
|
* Uses Node.js native test runner. All external fetch calls are mocked.
|
|
* DB functions use a real in-memory SQLite instance via node:sqlite (DatabaseSync),
|
|
* injected through the core module's globalThis.__omnirouteDb singleton.
|
|
* backupDbFile is called during first sync but safely no-ops (no file on disk).
|
|
*/
|
|
|
|
import { describe, it, beforeEach, afterEach } from "node:test";
|
|
import assert from "node:assert/strict";
|
|
import fs from "node:fs";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
|
|
const TEST_DATA_DIR = fs.mkdtempSync(
|
|
path.join(os.tmpdir(), "omniroute-arena-elo-test-"),
|
|
);
|
|
process.env.DATA_DIR = TEST_DATA_DIR;
|
|
|
|
const MIGRATION_SQL = fs.readFileSync(
|
|
path.resolve(
|
|
import.meta.dirname ?? __dirname,
|
|
"../../src/lib/db/migrations/097_model_intelligence.sql",
|
|
),
|
|
"utf8",
|
|
);
|
|
|
|
import { tryOpenSync } from "../../src/lib/db/adapters/driverFactory";
|
|
import type { SqliteAdapter } from "../../src/lib/db/adapters/types";
|
|
|
|
const core = await import("../../src/lib/db/core.ts");
|
|
|
|
const {
|
|
normalizeModelName,
|
|
transformToModelIntelligence,
|
|
fetchArenaLeaderboards,
|
|
syncArenaElo,
|
|
getArenaEloSyncStatus,
|
|
stopArenaEloSync,
|
|
} = await import("../../src/lib/arenaEloSync.ts");
|
|
|
|
import type {
|
|
ArenaLeaderboardData,
|
|
ArenaLeaderboardMap,
|
|
ArenaModelEntry,
|
|
} from "../../src/lib/arenaEloSync.ts";
|
|
|
|
const originalFetch = globalThis.fetch;
|
|
|
|
function mockFetch(
|
|
impl: (url: string, opts?: RequestInit) => Promise<Response>,
|
|
): void {
|
|
globalThis.fetch = impl as typeof fetch;
|
|
}
|
|
|
|
function restoreFetch(): void {
|
|
globalThis.fetch = originalFetch;
|
|
}
|
|
|
|
function jsonResponse(data: unknown, status = 200): Response {
|
|
return new Response(JSON.stringify(data), {
|
|
status,
|
|
headers: { "Content-Type": "application/json" },
|
|
});
|
|
}
|
|
|
|
function makeModelEntry(overrides: Partial<ArenaModelEntry> = {}): ArenaModelEntry {
|
|
return {
|
|
rank: 1,
|
|
model: "anthropic/claude-sonnet",
|
|
vendor: "Anthropic",
|
|
score: 1350,
|
|
ci: 10,
|
|
votes: 5000,
|
|
license: "proprietary",
|
|
...overrides,
|
|
};
|
|
}
|
|
|
|
function makeLeaderboardData(
|
|
models: ArenaModelEntry[] = [],
|
|
category = "text",
|
|
): ArenaLeaderboardData {
|
|
return {
|
|
meta: { leaderboard: category, model_count: models.length },
|
|
models,
|
|
};
|
|
}
|
|
|
|
function makeLeaderboardMap(
|
|
categories: Partial<Record<string, ArenaModelEntry[]>>,
|
|
): ArenaLeaderboardMap {
|
|
const map: ArenaLeaderboardMap = {};
|
|
for (const [cat, models] of Object.entries(categories)) {
|
|
map[cat] = makeLeaderboardData(models ?? [], cat);
|
|
}
|
|
return map;
|
|
}
|
|
|
|
let testAdapter: SqliteAdapter;
|
|
|
|
function createTestAdapter(): SqliteAdapter {
|
|
const patchedSql = MIGRATION_SQL.replace(
|
|
/\n\s*synced_at TEXT NOT NULL DEFAULT \(datetime\('now'\)\)/,
|
|
"\n synced_at TEXT NOT NULL",
|
|
);
|
|
const adapter = tryOpenSync(":memory:")!;
|
|
adapter.exec(patchedSql);
|
|
return adapter;
|
|
}
|
|
|
|
function countArenaEloEntries(): number {
|
|
const row = testAdapter
|
|
.prepare("SELECT COUNT(*) as cnt FROM model_intelligence WHERE source = 'arena_elo'")
|
|
.get() as Record<string, unknown> | undefined;
|
|
return Number(row?.cnt ?? 0);
|
|
}
|
|
|
|
function getAllEntries(): Array<Record<string, unknown>> {
|
|
return testAdapter
|
|
.prepare("SELECT * FROM model_intelligence WHERE source = 'arena_elo' ORDER BY model, category")
|
|
.all() as Array<Record<string, unknown>>;
|
|
}
|
|
|
|
beforeEach(() => {
|
|
core.resetDbInstance();
|
|
testAdapter = createTestAdapter();
|
|
globalThis.__omnirouteDb = testAdapter as never;
|
|
stopArenaEloSync();
|
|
});
|
|
|
|
afterEach(() => {
|
|
restoreFetch();
|
|
stopArenaEloSync();
|
|
delete globalThis.__omnirouteDb;
|
|
});
|
|
|
|
// ═══════════════════════════════════════════════════════════
|
|
// 1. normalizeModelName()
|
|
// ═══════════════════════════════════════════════════════════
|
|
|
|
describe("normalizeModelName()", () => {
|
|
it("strips 'anthropic/' vendor prefix", () => {
|
|
assert.strictEqual(
|
|
normalizeModelName("anthropic/claude-opus-4-6-thinking"),
|
|
"claude-opus-4-6-thinking",
|
|
);
|
|
});
|
|
|
|
it("strips 'openai/' vendor prefix", () => {
|
|
assert.strictEqual(normalizeModelName("openai/gpt-5.5"), "gpt-5.5");
|
|
});
|
|
|
|
it("strips 'google/' vendor prefix", () => {
|
|
assert.strictEqual(normalizeModelName("google/gemini-3-flash"), "gemini-3-flash");
|
|
});
|
|
|
|
it("strips 'meta/' vendor prefix", () => {
|
|
assert.strictEqual(normalizeModelName("meta/llama-4"), "llama-4");
|
|
});
|
|
|
|
it("strips 'deepseek/' vendor prefix", () => {
|
|
assert.strictEqual(normalizeModelName("deepseek/deepseek-r1"), "deepseek-r1");
|
|
});
|
|
|
|
it("strips 'xai/' vendor prefix", () => {
|
|
assert.strictEqual(normalizeModelName("xai/grok-4"), "grok-4");
|
|
});
|
|
|
|
it("lowercases the model name", () => {
|
|
assert.strictEqual(normalizeModelName("Claude-Sonnet-4"), "claude-sonnet-4");
|
|
});
|
|
|
|
it("lowercases vendor prefix before matching", () => {
|
|
assert.strictEqual(normalizeModelName("OpenAI/GPT-5.5"), "gpt-5.5");
|
|
});
|
|
|
|
it("returns name unchanged when no vendor prefix matches", () => {
|
|
assert.strictEqual(normalizeModelName("my-custom-model"), "my-custom-model");
|
|
});
|
|
});
|
|
|
|
// ═══════════════════════════════════════════════════════════
|
|
// 2. transformToModelIntelligence()
|
|
// ═══════════════════════════════════════════════════════════
|
|
|
|
describe("transformToModelIntelligence()", () => {
|
|
it("ELO normalization: 1500 ELO (max) with range 1000-1500 → score ≈ 0.98", () => {
|
|
const data = makeLeaderboardMap({
|
|
text: [
|
|
makeModelEntry({ model: "top-model", score: 1500, votes: 5000, rank: 1 }),
|
|
makeModelEntry({ model: "low-model", score: 1000, votes: 5000, rank: 2 }),
|
|
],
|
|
});
|
|
|
|
const entries = transformToModelIntelligence(data);
|
|
const topEntry = entries.find(
|
|
(e) => e.model === "top-model" && e.category === "default",
|
|
);
|
|
|
|
assert.ok(topEntry);
|
|
// taskFit = 0.4 + 0.58 * ((1500-1000) / 500) = 0.98
|
|
assert.ok(Math.abs(topEntry.score - 0.98) < 0.001, `got ${topEntry.score}`);
|
|
});
|
|
|
|
it("ELO normalization: 1000 ELO (min) with range 1000-1500 → score ≈ 0.40", () => {
|
|
const data = makeLeaderboardMap({
|
|
text: [
|
|
makeModelEntry({ model: "top-model", score: 1500, votes: 5000, rank: 1 }),
|
|
makeModelEntry({ model: "low-model", score: 1000, votes: 5000, rank: 2 }),
|
|
],
|
|
});
|
|
|
|
const entries = transformToModelIntelligence(data);
|
|
const lowEntry = entries.find(
|
|
(e) => e.model === "low-model" && e.category === "default",
|
|
);
|
|
|
|
assert.ok(lowEntry);
|
|
// taskFit = 0.4 + 0.58 * ((1000-1000) / 500) = 0.4
|
|
assert.ok(Math.abs(lowEntry.score - 0.4) < 0.001, `got ${lowEntry.score}`);
|
|
});
|
|
|
|
it("votes < 100 → confidence='low'", () => {
|
|
const data = makeLeaderboardMap({
|
|
text: [
|
|
makeModelEntry({ model: "sparse-model", score: 1200, votes: 50, rank: 5 }),
|
|
makeModelEntry({ model: "baseline", score: 1100, votes: 5000, rank: 10 }),
|
|
],
|
|
});
|
|
|
|
const entries = transformToModelIntelligence(data);
|
|
const entry = entries.find(
|
|
(e) => e.model === "sparse-model" && e.category === "default",
|
|
);
|
|
|
|
assert.ok(entry);
|
|
assert.strictEqual(entry.confidence, "low");
|
|
});
|
|
|
|
it("votes >= 1000 → confidence='medium'", () => {
|
|
const data = makeLeaderboardMap({
|
|
text: [
|
|
makeModelEntry({ model: "mid-model", score: 1200, votes: 1500, rank: 3 }),
|
|
makeModelEntry({ model: "baseline", score: 1100, votes: 5000, rank: 10 }),
|
|
],
|
|
});
|
|
|
|
const entries = transformToModelIntelligence(data);
|
|
const entry = entries.find(
|
|
(e) => e.model === "mid-model" && e.category === "default",
|
|
);
|
|
|
|
assert.ok(entry);
|
|
assert.strictEqual(entry.confidence, "medium");
|
|
});
|
|
|
|
it("votes >= 5000 → confidence='high'", () => {
|
|
const data = makeLeaderboardMap({
|
|
text: [
|
|
makeModelEntry({ model: "popular-model", score: 1300, votes: 8000, rank: 1 }),
|
|
makeModelEntry({ model: "baseline", score: 1100, votes: 3000, rank: 5 }),
|
|
],
|
|
});
|
|
|
|
const entries = transformToModelIntelligence(data);
|
|
const entry = entries.find(
|
|
(e) => e.model === "popular-model" && e.category === "default",
|
|
);
|
|
|
|
assert.ok(entry);
|
|
assert.strictEqual(entry.confidence, "high");
|
|
});
|
|
|
|
it("category mapping: 'text' → [default, review, documentation, debugging]", () => {
|
|
const data = makeLeaderboardMap({
|
|
text: [
|
|
makeModelEntry({ model: "text-model", score: 1200, votes: 5000, rank: 1 }),
|
|
],
|
|
});
|
|
|
|
const entries = transformToModelIntelligence(data);
|
|
const categories = entries
|
|
.filter((e) => e.model === "text-model")
|
|
.map((e) => e.category)
|
|
.sort();
|
|
|
|
assert.deepStrictEqual(categories, [
|
|
"debugging",
|
|
"default",
|
|
"documentation",
|
|
"review",
|
|
]);
|
|
});
|
|
|
|
it("category mapping: 'code' → [coding]", () => {
|
|
const data = makeLeaderboardMap({
|
|
code: [
|
|
makeModelEntry({ model: "code-model", score: 1300, votes: 5000, rank: 1 }),
|
|
],
|
|
});
|
|
|
|
const entries = transformToModelIntelligence(data);
|
|
const categories = entries
|
|
.filter((e) => e.model === "code-model")
|
|
.map((e) => e.category);
|
|
|
|
assert.deepStrictEqual(categories, ["coding"]);
|
|
});
|
|
|
|
it("expires_at is set to ~7 days in the future", () => {
|
|
const before = Date.now();
|
|
const data = makeLeaderboardMap({
|
|
text: [
|
|
makeModelEntry({ model: "test-model", score: 1200, votes: 5000, rank: 1 }),
|
|
],
|
|
});
|
|
|
|
const entries = transformToModelIntelligence(data);
|
|
const after = Date.now();
|
|
|
|
const entry = entries.find(
|
|
(e) => e.model === "test-model" && e.category === "default",
|
|
);
|
|
assert.ok(entry);
|
|
assert.ok(entry.expiresAt);
|
|
|
|
const expiresMs = new Date(entry.expiresAt).getTime();
|
|
const sevenDaysMs = 7 * 24 * 60 * 60 * 1000;
|
|
|
|
assert.ok(
|
|
expiresMs >= before + sevenDaysMs - 2000,
|
|
`expiresAt too early: ${entry.expiresAt}`,
|
|
);
|
|
assert.ok(
|
|
expiresMs <= after + sevenDaysMs + 2000,
|
|
`expiresAt too late: ${entry.expiresAt}`,
|
|
);
|
|
});
|
|
|
|
it("source is 'arena_elo' for all entries", () => {
|
|
const data = makeLeaderboardMap({
|
|
text: [
|
|
makeModelEntry({ model: "test-model", score: 1200, votes: 5000, rank: 1 }),
|
|
],
|
|
});
|
|
|
|
const entries = transformToModelIntelligence(data);
|
|
for (const entry of entries) {
|
|
assert.strictEqual(entry.source, "arena_elo");
|
|
}
|
|
});
|
|
|
|
it("expands model aliases for known models", () => {
|
|
const data = makeLeaderboardMap({
|
|
text: [
|
|
makeModelEntry({
|
|
model: "anthropic/claude-opus-4-6-thinking",
|
|
score: 1400,
|
|
votes: 5000,
|
|
rank: 1,
|
|
}),
|
|
],
|
|
});
|
|
|
|
const entries = transformToModelIntelligence(data);
|
|
const models = entries.map((e) => e.model);
|
|
|
|
assert.ok(models.includes("claude-opus-4-6-thinking"));
|
|
assert.ok(models.includes("claude-opus-4"));
|
|
assert.ok(models.includes("anthropic/claude-opus-4"));
|
|
});
|
|
|
|
it("empty leaderboard → no entries", () => {
|
|
const data = makeLeaderboardMap({ text: [] });
|
|
const entries = transformToModelIntelligence(data);
|
|
assert.strictEqual(entries.length, 0);
|
|
});
|
|
|
|
it("skips unknown leaderboard categories (e.g. vision)", () => {
|
|
const data = makeLeaderboardMap({
|
|
vision: [
|
|
makeModelEntry({ model: "vision-model", score: 1300, votes: 5000, rank: 1 }),
|
|
],
|
|
});
|
|
|
|
const entries = transformToModelIntelligence(data);
|
|
assert.strictEqual(entries.length, 0);
|
|
});
|
|
|
|
it("preserves eloRaw from the leaderboard", () => {
|
|
const data = makeLeaderboardMap({
|
|
text: [
|
|
makeModelEntry({ model: "test-model", score: 1337, votes: 5000, rank: 1 }),
|
|
makeModelEntry({ model: "other-model", score: 1100, votes: 3000, rank: 2 }),
|
|
],
|
|
});
|
|
|
|
const entries = transformToModelIntelligence(data);
|
|
const entry = entries.find(
|
|
(e) => e.model === "test-model" && e.category === "default",
|
|
);
|
|
assert.ok(entry);
|
|
assert.strictEqual(entry.eloRaw, 1337);
|
|
});
|
|
|
|
it("handles single-model leaderboard (eloRange = 1, avoids division by zero)", () => {
|
|
const data = makeLeaderboardMap({
|
|
text: [
|
|
makeModelEntry({ model: "only-model", score: 1200, votes: 5000, rank: 1 }),
|
|
],
|
|
});
|
|
|
|
const entries = transformToModelIntelligence(data);
|
|
assert.ok(entries.length > 0);
|
|
|
|
const entry = entries.find(
|
|
(e) => e.model === "only-model" && e.category === "default",
|
|
);
|
|
assert.ok(entry);
|
|
assert.ok(Math.abs(entry.score - 0.4) < 0.001);
|
|
});
|
|
|
|
it("rounds score to 4 decimal places", () => {
|
|
const data = makeLeaderboardMap({
|
|
text: [
|
|
makeModelEntry({ model: "model-a", score: 1300, votes: 200, rank: 1 }),
|
|
makeModelEntry({ model: "model-b", score: 1000, votes: 200, rank: 2 }),
|
|
],
|
|
});
|
|
|
|
const entries = transformToModelIntelligence(data);
|
|
for (const entry of entries) {
|
|
const str = entry.score.toString();
|
|
const dot = str.indexOf(".");
|
|
if (dot !== -1) {
|
|
assert.ok(str.length - dot - 1 <= 4, `score ${entry.score} > 4 decimals`);
|
|
}
|
|
}
|
|
});
|
|
});
|
|
|
|
// ═══════════════════════════════════════════════════════════
|
|
// 3. fetchArenaLeaderboards()
|
|
// ═══════════════════════════════════════════════════════════
|
|
|
|
describe("fetchArenaLeaderboards()", () => {
|
|
it("successful fetch with valid JSON returns both leaderboards", async () => {
|
|
const textData = makeLeaderboardData(
|
|
[makeModelEntry({ model: "text-model", score: 1200, votes: 5000, rank: 1 })],
|
|
"text",
|
|
);
|
|
const codeData = makeLeaderboardData(
|
|
[makeModelEntry({ model: "code-model", score: 1300, votes: 5000, rank: 1 })],
|
|
"code",
|
|
);
|
|
|
|
mockFetch(async (url: string) => {
|
|
if (url.includes("name=text")) return jsonResponse(textData);
|
|
if (url.includes("name=code")) return jsonResponse(codeData);
|
|
return new Response("Not found", { status: 404 });
|
|
});
|
|
|
|
const result = await fetchArenaLeaderboards();
|
|
|
|
assert.ok(result.text);
|
|
assert.ok(result.code);
|
|
assert.strictEqual(result.text.models.length, 1);
|
|
assert.strictEqual(result.code.models.length, 1);
|
|
});
|
|
|
|
it("failed fetch (non-200 status) throws with descriptive message", async () => {
|
|
mockFetch(async () => {
|
|
return new Response("Internal Server Error", { status: 500 });
|
|
});
|
|
|
|
await assert.rejects(
|
|
() => fetchArenaLeaderboards(),
|
|
(err: unknown) => {
|
|
assert.ok(err instanceof Error);
|
|
assert.ok(err.message.includes("All Arena leaderboard fetches failed"));
|
|
return true;
|
|
},
|
|
);
|
|
});
|
|
|
|
it("all fetches fail (network error) → throws", async () => {
|
|
mockFetch(async () => {
|
|
throw new Error("Network error: ECONNREFUSED");
|
|
});
|
|
|
|
await assert.rejects(
|
|
() => fetchArenaLeaderboards(),
|
|
(err: unknown) => {
|
|
assert.ok(err instanceof Error);
|
|
assert.ok(err.message.includes("All Arena leaderboard fetches failed"));
|
|
return true;
|
|
},
|
|
);
|
|
});
|
|
|
|
it("succeeds when one category fails but another succeeds", async () => {
|
|
const textData = makeLeaderboardData(
|
|
[makeModelEntry({ model: "text-model", score: 1200, votes: 5000, rank: 1 })],
|
|
"text",
|
|
);
|
|
|
|
mockFetch(async (url: string) => {
|
|
if (url.includes("name=text")) return jsonResponse(textData);
|
|
if (url.includes("name=code"))
|
|
return new Response("Error", { status: 500 });
|
|
return new Response("Not found", { status: 404 });
|
|
});
|
|
|
|
const result = await fetchArenaLeaderboards();
|
|
assert.ok(result.text);
|
|
assert.strictEqual(result.code, undefined);
|
|
});
|
|
|
|
it("invalid JSON in response → throws when all responses are invalid", async () => {
|
|
mockFetch(async () => {
|
|
return new Response("not-json", {
|
|
status: 200,
|
|
headers: { "Content-Type": "text/plain" },
|
|
});
|
|
});
|
|
|
|
await assert.rejects(
|
|
() => fetchArenaLeaderboards(),
|
|
(err: unknown) => {
|
|
assert.ok(err instanceof Error);
|
|
assert.ok(err.message.includes("All Arena leaderboard fetches failed"));
|
|
return true;
|
|
},
|
|
);
|
|
});
|
|
});
|
|
|
|
// ═══════════════════════════════════════════════════════════
|
|
// 4. syncArenaElo()
|
|
// ═══════════════════════════════════════════════════════════
|
|
|
|
describe("syncArenaElo()", () => {
|
|
it("happy path: returns success=true with correct modelCount", async () => {
|
|
const textData = makeLeaderboardData(
|
|
[makeModelEntry({ model: "gpt-5.5", score: 1200, votes: 5000, rank: 1 })],
|
|
"text",
|
|
);
|
|
const codeData = makeLeaderboardData(
|
|
[makeModelEntry({ model: "deepseek-r1", score: 1300, votes: 5000, rank: 1 })],
|
|
"code",
|
|
);
|
|
|
|
mockFetch(async (url: string) => {
|
|
if (url.includes("name=text")) return jsonResponse(textData);
|
|
if (url.includes("name=code")) return jsonResponse(codeData);
|
|
return new Response("Not found", { status: 404 });
|
|
});
|
|
|
|
const result = await syncArenaElo();
|
|
|
|
assert.strictEqual(result.success, true);
|
|
assert.strictEqual(result.source, "arena_elo");
|
|
assert.ok(result.modelCount > 0);
|
|
|
|
const dbCount = countArenaEloEntries();
|
|
assert.ok(dbCount > 0);
|
|
assert.strictEqual(dbCount, result.modelCount);
|
|
});
|
|
|
|
it("happy path: entries in DB have correct source and categories", async () => {
|
|
const textData = makeLeaderboardData(
|
|
[makeModelEntry({ model: "unique-test-model", score: 1200, votes: 5000, rank: 1 })],
|
|
"text",
|
|
);
|
|
|
|
mockFetch(async (url: string) => {
|
|
if (url.includes("name=text")) return jsonResponse(textData);
|
|
if (url.includes("name=code"))
|
|
return jsonResponse(makeLeaderboardData([], "code"));
|
|
return new Response("Not found", { status: 404 });
|
|
});
|
|
|
|
await syncArenaElo();
|
|
|
|
const entries = getAllEntries().filter((e) => String(e.model) === "unique-test-model");
|
|
const categories = entries.map((e) => String(e.category)).sort();
|
|
assert.deepStrictEqual(categories, [
|
|
"debugging",
|
|
"default",
|
|
"documentation",
|
|
"review",
|
|
]);
|
|
|
|
for (const entry of entries) {
|
|
assert.strictEqual(entry.source, "arena_elo");
|
|
}
|
|
});
|
|
|
|
it("dryRun=true → does not call bulkUpsertModelIntelligence", async () => {
|
|
const textData = makeLeaderboardData(
|
|
[makeModelEntry({ model: "test-model", score: 1200, votes: 5000, rank: 1 })],
|
|
"text",
|
|
);
|
|
|
|
mockFetch(async (url: string) => {
|
|
if (url.includes("name=text")) return jsonResponse(textData);
|
|
if (url.includes("name=code"))
|
|
return jsonResponse(makeLeaderboardData([], "code"));
|
|
return new Response("Not found", { status: 404 });
|
|
});
|
|
|
|
const result = await syncArenaElo(true);
|
|
|
|
assert.strictEqual(result.success, true);
|
|
assert.ok(result.modelCount > 0);
|
|
|
|
const dbCount = countArenaEloEntries();
|
|
assert.strictEqual(dbCount, 0, "dryRun should not write to DB");
|
|
});
|
|
|
|
it("dryRun=true does not update lastSyncTime", async () => {
|
|
// Reset module-level state by observing that before this test's dryRun,
|
|
// lastSync should remain unchanged from whatever it was.
|
|
// Since we can't reset module-level vars, we verify that a dryRun
|
|
// after a non-dryRun doesn't overwrite the model count.
|
|
const statusBefore = getArenaEloSyncStatus();
|
|
|
|
const textData = makeLeaderboardData(
|
|
[makeModelEntry({ model: "test-model", score: 1200, votes: 5000, rank: 1 })],
|
|
"text",
|
|
);
|
|
|
|
mockFetch(async (url: string) => {
|
|
if (url.includes("name=text")) return jsonResponse(textData);
|
|
if (url.includes("name=code"))
|
|
return jsonResponse(makeLeaderboardData([], "code"));
|
|
return new Response("Not found", { status: 404 });
|
|
});
|
|
|
|
await syncArenaElo(true);
|
|
|
|
const statusAfter = getArenaEloSyncStatus();
|
|
// dryRun should not change the modelCount
|
|
assert.strictEqual(statusAfter.lastSyncModelCount, statusBefore.lastSyncModelCount);
|
|
});
|
|
|
|
it("API failure → returns success=false with error", async () => {
|
|
mockFetch(async () => {
|
|
throw new Error("Network down");
|
|
});
|
|
|
|
const result = await syncArenaElo();
|
|
|
|
assert.strictEqual(result.success, false);
|
|
assert.strictEqual(result.source, "arena_elo");
|
|
assert.strictEqual(result.modelCount, 0);
|
|
assert.ok(result.error);
|
|
assert.ok(
|
|
result.error!.includes("All Arena leaderboard fetches failed"),
|
|
`unexpected: ${result.error}`,
|
|
);
|
|
});
|
|
|
|
it("calls deleteExpiredIntelligence before writing new entries", async () => {
|
|
testAdapter.prepare(
|
|
"INSERT INTO model_intelligence (model, source, category, score, elo_raw, confidence, synced_at, expires_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
|
|
).run("old-model", "arena_elo", "default", 0.5, 1000, "low", "2025-01-01T00:00:00Z", "2020-01-01T00:00:00Z");
|
|
|
|
assert.strictEqual(countArenaEloEntries(), 1);
|
|
|
|
const textData = makeLeaderboardData(
|
|
[makeModelEntry({ model: "new-model", score: 1200, votes: 5000, rank: 1 })],
|
|
"text",
|
|
);
|
|
|
|
mockFetch(async (url: string) => {
|
|
if (url.includes("name=text")) return jsonResponse(textData);
|
|
if (url.includes("name=code"))
|
|
return jsonResponse(makeLeaderboardData([], "code"));
|
|
return new Response("Not found", { status: 404 });
|
|
});
|
|
|
|
const syncResult = await syncArenaElo();
|
|
assert.strictEqual(syncResult.success, true);
|
|
|
|
const expiredEntry = testAdapter
|
|
.prepare("SELECT * FROM model_intelligence WHERE model = 'old-model'")
|
|
.get();
|
|
assert.strictEqual(expiredEntry, undefined);
|
|
});
|
|
|
|
it("handles empty leaderboard gracefully (0 entries, no DB write)", async () => {
|
|
mockFetch(async (url: string) => {
|
|
if (url.includes("name=text"))
|
|
return jsonResponse(makeLeaderboardData([], "text"));
|
|
if (url.includes("name=code"))
|
|
return jsonResponse(makeLeaderboardData([], "code"));
|
|
return new Response("Not found", { status: 404 });
|
|
});
|
|
|
|
const result = await syncArenaElo();
|
|
|
|
assert.strictEqual(result.success, true);
|
|
assert.strictEqual(result.modelCount, 0);
|
|
assert.strictEqual(countArenaEloEntries(), 0);
|
|
});
|
|
|
|
it("updates lastSyncTime after successful sync", async () => {
|
|
const textData = makeLeaderboardData(
|
|
[makeModelEntry({ model: "test-model", score: 1200, votes: 5000, rank: 1 })],
|
|
"text",
|
|
);
|
|
|
|
mockFetch(async (url: string) => {
|
|
if (url.includes("name=text")) return jsonResponse(textData);
|
|
if (url.includes("name=code"))
|
|
return jsonResponse(makeLeaderboardData([], "code"));
|
|
return new Response("Not found", { status: 404 });
|
|
});
|
|
|
|
await syncArenaElo();
|
|
|
|
const status = getArenaEloSyncStatus();
|
|
assert.ok(status.lastSync);
|
|
assert.ok(status.lastSyncModelCount > 0);
|
|
});
|
|
|
|
it("model aliases are stored in DB alongside canonical names", async () => {
|
|
const textData = makeLeaderboardData(
|
|
[makeModelEntry({
|
|
model: "anthropic/claude-opus-4-6-thinking",
|
|
score: 1400,
|
|
votes: 5000,
|
|
rank: 1,
|
|
})],
|
|
"text",
|
|
);
|
|
|
|
mockFetch(async (url: string) => {
|
|
if (url.includes("name=text")) return jsonResponse(textData);
|
|
if (url.includes("name=code"))
|
|
return jsonResponse(makeLeaderboardData([], "code"));
|
|
return new Response("Not found", { status: 404 });
|
|
});
|
|
|
|
await syncArenaElo();
|
|
|
|
const entries = getAllEntries();
|
|
const models = entries.map((e) => String(e.model));
|
|
|
|
assert.ok(models.includes("claude-opus-4-6-thinking"));
|
|
assert.ok(models.includes("claude-opus-4"));
|
|
});
|
|
});
|
|
|
|
// ═══════════════════════════════════════════════════════════
|
|
// 5. getArenaEloSyncStatus()
|
|
// ═══════════════════════════════════════════════════════════
|
|
|
|
describe("getArenaEloSyncStatus()", () => {
|
|
it("returns correct structure with all expected keys", () => {
|
|
const status = getArenaEloSyncStatus();
|
|
|
|
assert.ok("enabled" in status);
|
|
assert.ok("lastSync" in status);
|
|
assert.ok("lastSyncModelCount" in status);
|
|
assert.ok("nextSync" in status);
|
|
assert.ok("intervalMs" in status);
|
|
assert.ok("sources" in status);
|
|
});
|
|
|
|
it("returns sources containing 'arena_elo'", () => {
|
|
const status = getArenaEloSyncStatus();
|
|
assert.deepStrictEqual(status.sources, ["arena_elo"]);
|
|
});
|
|
|
|
it("returns intervalMs as a positive number", () => {
|
|
const status = getArenaEloSyncStatus();
|
|
assert.ok(typeof status.intervalMs === "number");
|
|
assert.ok(status.intervalMs > 0);
|
|
});
|
|
|
|
it("returns lastSyncModelCount as 0 before any non-dryRun sync completes in this suite context", () => {
|
|
// Module-level lastSyncTime may leak from earlier tests in the suite,
|
|
// but the structural fields are always present.
|
|
const status = getArenaEloSyncStatus();
|
|
assert.ok(typeof status.lastSync === "string" || status.lastSync === null);
|
|
assert.ok(typeof status.lastSyncModelCount === "number");
|
|
assert.ok(typeof status.nextSync === "string" || status.nextSync === null);
|
|
});
|
|
|
|
it("reflects ARENA_ELO_SYNC_ENABLED env var", () => {
|
|
const original = process.env.ARENA_ELO_SYNC_ENABLED;
|
|
|
|
process.env.ARENA_ELO_SYNC_ENABLED = "true";
|
|
const enabledStatus = getArenaEloSyncStatus();
|
|
assert.strictEqual(enabledStatus.enabled, true);
|
|
|
|
process.env.ARENA_ELO_SYNC_ENABLED = "false";
|
|
const disabledStatus = getArenaEloSyncStatus();
|
|
assert.strictEqual(disabledStatus.enabled, false);
|
|
|
|
if (original !== undefined) {
|
|
process.env.ARENA_ELO_SYNC_ENABLED = original;
|
|
} else {
|
|
delete process.env.ARENA_ELO_SYNC_ENABLED;
|
|
}
|
|
});
|
|
});
|
|
|
|
// ═══════════════════════════════════════════════════════════
|
|
// 6. stopArenaEloSync()
|
|
// ═══════════════════════════════════════════════════════════
|
|
|
|
describe("stopArenaEloSync()", () => {
|
|
it("does not throw when no timer is running", () => {
|
|
assert.doesNotThrow(() => stopArenaEloSync());
|
|
});
|
|
|
|
it("can be called multiple times without error", () => {
|
|
stopArenaEloSync();
|
|
assert.doesNotThrow(() => stopArenaEloSync());
|
|
assert.doesNotThrow(() => stopArenaEloSync());
|
|
});
|
|
});
|