Files
OmniRoute/tests/unit/arena-elo-sync.test.ts
Diego Rodrigues de Sa e Souza de60b4b171 Release v3.8.23
* chore(release): open v3.8.23 development cycle

* fix(anthropic): strip top_p when temperature is set to avoid 400 (#3691)

Integrated into release/v3.8.23

* fix(vertex): support Vertex AI Express-mode API keys (#3690)

Integrated into release/v3.8.23

* fix(stream): error on empty Claude SSE instead of synthetic success (#3689)

Integrated into release/v3.8.23

* fix(oauth): stop token-refresh invalidation loop + harden proxy resolution (#3692)

Integrated into release/v3.8.23

* docs: add FUNDING.yml and Support section to README (#3698)

Integrated into release/v3.8.23

* feat: gemini - handle known ratelimits (#3686)

Integrated into release/v3.8.23

* fix: stream combo fails over on empty content-filtered response (#3685) (#3702)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(antigravity): preserve gemini-3.1-pro high/low budget tiers (#3696) (#3703)

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(auto-combo): add auto-updating model intelligence scoring (#3660)

Integrated into release/v3.8.23

* fix(gemini): context-mode fallback for signatureless tool calls (#3688) (#3704)

* chore(quality-gate): reconcile file-size baseline (27 files + providerLimits.ts) (#3705)

* feat(vertex): dynamic model discovery via Generative Language models API (#3712)

Integrated into release/v3.8.23. Vertex dynamic model discovery — surfaces image models (imagen-*, gemini-*-image), embeddings and audio from the live Generative Language catalog, with cached→static fallback and the shared parseGeminiModelsList helper. Validated: parser test 5/5, typecheck:core clean.

* fix(combo): gate reasoning token buffer (#3700)

Integrated into release/v3.8.23. Makes the #3588 reasoning token buffer safe and configurable: only inflates max_tokens when the model has a known, non-default output cap and the buffered value fits inside it; otherwise preserves/clamps the client limit. Adds the reasoningTokenBufferEnabled kill switch (default ON). Validated: combo-routing-engine 81/81, combo-config 25/25, combo-quality-validator-reasoning 12/12, phase1f 10/10, typecheck:core clean.

* refactor(#3501): god-component Phase 1g-1j — client 4062→3408 LOC (-654) (#3717)

Phase 1g-1j of #3501: client 4062→3408 LOC. Pure extraction (ProviderPlaygroundPanel, useCommandCodeAuth, useExternalLinkFlow+ExternalLinkModal, useAuthFileHandlers) + loadConnProxies ReferenceError fix + phase1f test path fix.

Co-authored-by: oyi77 <14921983+oyi77@users.noreply.github.com>

* refactor(#3501): god-component Phase 1k-1m — client 3408→2553 LOC (-855) (#3721)

Phase 1k-1m of #3501: client 3408→2553 LOC. Pure extraction (useModelImportHandlers+ImportProgressModal, useModelVisibilityHandlers, ProviderModelsSection).

Co-authored-by: oyi77 <14921983+oyi77@users.noreply.github.com>

* docs(changelog): restore #3590 bullet lost on the v3.8.20 release branch

The fix itself reached main pre-tag via cherry-pick #3591, but its changelog
bullet (commit e33fdd4ab) only ever existed on release/v3.8.20 after the
squash-merge. Restored under [3.8.20] per the 2026-06-12 release-branch
leftover audit (_tasks/release-audit/release-leftovers-audit-2026-06-12.md).

* fix(kiro): resolve quota for IAM Identity Center accounts missing a profileArn (#3722)

Integrated into release/v3.8.23

* refactor(#3501): god-component Phase 1n-1s — client 2553→1376 LOC (-1177) (#3725)

Phase 1n-1s of #3501: client 2553→1376 LOC. Pure extraction (ConnectionsListPanel, ConnectionsHeaderToolbar, ZedImportCard, BatchTestResultsModal, AdaptaTutorialModal, useApiKeySave + helpers).

Co-authored-by: oyi77 <14921983+oyi77@users.noreply.github.com>

* feat(model-lockout): settings UI, backend integration, error classification, and success-decay recovery (#3629)

Integrated into release/v3.8.23

* refactor(#3501): god-component Phase 1t — client 1376→781 LOC (≤800 TARGET REACHED ) (#3727)

Phase 1t of #3501: client 1376→781 LOC (≤800 reached). Original god-component 12,882→781 (−94%).

Co-authored-by: oyi77 <14921983+oyi77@users.noreply.github.com>

* fix: bundle @omniroute/opencode-plugin inside omniroute + add 'setup opencode' CLI command (#3726)

Integrated into release/v3.8.23

* feat(vertex): self-tracked USD spend since account added (#3724)

Integrated into release/v3.8.23

* fix(qwen-web): migrate to v2 chat API with full cookie-jar replay (#3288) (#3723)

Integrated into release/v3.8.23

* fix(sse): make safeLogEvents async — 'await' in a sync function broke every chatHelpers import

#3692 added a lazy 'await import(proxyEgress)' for egress-IP visibility inside
safeLogEvents, which is a sync function — an ES syntax error. It went unnoticed
because typecheck:core does not cover src/sse and no test in the merge gates
loaded chatHelpers via tsx; any consumer that did (chat-context-relay and
chat-route-coverage suites, integration harnesses) failed at module load with
'await can only be used inside an async function'.

safeLogEvents is fire-and-forget logging with an outer try/catch, so making it
async (and 'void'-ing the single chat.ts call site) preserves behavior exactly.

Validation: tests/unit/chat-context-relay.test.ts + chat-route-coverage.test.ts
went from failing-at-load to green (+14 tests destravados).

* fix(sse): remove cross-provider credential leak in emergency fallback + combo/proxy audit fixes (#3699)

Integrated into release/v3.8.23

* fix(executors): inject MiMoCode anti-abuse marker so free endpoint stops 403ing (#3728)

Integrated into release/v3.8.23

* fix(dashboard): repair "Test all models" — toast crash, status icons, auto-hide (#3729)

Integrated into release/v3.8.23

* chore(deps): bump actions/upload-artifact from 4 to 7 (#3735)

Integrated into release/v3.8.23 — aligns upload-artifact to v7 (already used across ci.yml).

* chore(deps): bump actions/cache from 4 to 5 (#3734)

Integrated into release/v3.8.23 — actions/cache v4→v5.

* chore(deps): bump actions/download-artifact from 4 to 8 (#3733)

Integrated into release/v3.8.23 — download-artifact v4→v8.

* feat(fallback): add OMNIROUTE_EMERGENCY_FALLBACK env switch (#3741)

Adds an OMNIROUTE_EMERGENCY_FALLBACK env switch to disable the emergency budget-exhaustion fallback (reroute to free nvidia/gpt-oss-120b). Default unchanged (enabled). Closes #3739, related #2879.

Integrated into release/v3.8.23.

* i18n: comprehensive zh-CN translation improvements (#3736)

Aligns zh-CN to en (hundreds of entries), translates batch-action labels + settings sidebar menu, adds categoryConfig/endpointTokenSaver keys, resolves __MISSING__ stubs. Sidebar/SidebarTab hardcoded strings replaced with t(). en.json purely additive (8 new sidebar.* keys, 0 removed); cli-i18n gate green.

Integrated into release/v3.8.23.

* chore(release): v3.8.23 — 2026-06-12

- CHANGELOG: complete v3.8.23 section (28 bullets, 27 commits)
- fix(webdav): resolve promise on writeStream finish, not req end — eliminates
  intermittent 500 on PUT update (writeStream may not have flushed at rename time)
- test(autoCombo): stub DB calls from PR #3660 in tieredRotation.test.ts to prevent
  5s timeout in vitest (getModelIntelligenceBySource DB init path)
- chore(env-sync): add XDG_DATA_HOME + OMNIROUTE_OPENCODE_PLUGIN_DIR to IGNORE_FROM_CODE
  allowlist (introduced by PR #3726 setup-open-code.mjs, not OmniRoute config vars)
- chore(cli): regenerated bin/cli/api-commands/*.mjs (7 new, 27 updated)

* fix(model-family): fallback lookup also tries bare model name with dots

getNextFamilyFallback normalized dots-to-hyphens ("gemini-3.1-pro-high" →
"gemini-3-1-pro-high") but MODEL_FAMILIES keys use the literal dot form. The
lookup always missed, returning null for any model whose dots are part of the
name rather than a version separator.

Fallback: try MODEL_FAMILIES[lookupKey] ?? MODEL_FAMILIES[bareModel] so both
naming conventions are covered. Fixes T30 test (pre-existing since v3.8.22).

* feat: expose API key cost drilldown + quota % used (#3742)

Adds all-time USD cost per API key in the API Key Manager, a per-key deep-link into the Cost Explorer (filtered + grouped by model), URL-param hydration of range/groupBy/apiKeyIds, and a '% used' quota display. Review adjustments: extracted URL-param parsers to a tested module (Rule #18), i18n'd the new strings (en + zh-CN), dropped the redundant webdav-handler entry already on release.

Integrated into release/v3.8.23.

* feat: add provider display modes — All / Configured / Compact (#3743)

Replaces the Providers page configured-only toggle with All/Configured/Compact display modes (Compact = flat deduped grid, no-auth last). Persists the preference and migrates the legacy localStorage key. Rebased onto release/v3.8.23.

Integrated into release/v3.8.23.

* fix(cache): scope semantic-cache signature to API key (#3740)

Adds the api_key_id dimension to generateSignature's SHA-256 hash so two callers with different API keys never receive each other's cached responses. Threads apiKeyId through checkSemanticCache + both write sites; migration 098 clears pre-existing key-less entries; unauthenticated requests stay isolated from keyed ones. 3 TDD tests.

Integrated into release/v3.8.23.

* fix(responses): apply OpenAI Responses API stream=false spec default (#3708)

resolveStreamFlag now applies the stream=false-when-omitted default for sourceFormat=openai-responses (same as the existing claude path), so spec-compliant /v1/responses upstreams that return JSON no longer fall through to the wildcard-Accept heuristic and trigger STREAM_EARLY_EOF / 502. Codex CLI (stream:true) and explicit text/event-stream clients unaffected.

Integrated into release/v3.8.23.

* chore(release): reconcile CI gates for v3.8.23

- file-size baseline: re-freeze 8 files grown by PRs #3742/#3743/#3740
  (cost drilldown, provider display modes, cache key isolation)
- ARCHITECTURE.md: update executor count 55→60 (check:docs-counts drift)
- .env.example: add OMNIROUTE_EMERGENCY_FALLBACK (#3741, env-doc-sync)
- CHANGELOG: add formatted bullets for #3742, #3743, #3708, #3740,
  model-family-fallback fix; remove duplicate raw ### Fixed section

* test: restore assert count to satisfy check:test-masking gate

Three test files had net assertion removals after behavior-changing PRs:
- chatcore-translation-paths: emergency fallback moved to routing layer
  (#3699) — add body error assertion + model-name guard
- executor-vertex-extended: non-JSON is now Express API key (#3690) —
  add projects/-path guard to the express-key URL test
- stream-utils: empty streams now emit error (#3685) — add code/message/
  status/completePayload guards to both passthrough and translate variants

All new assertions are meaningful (code enum value, 5xx range, non-empty
message, onComplete must-not-fire contract).

* fix(ci): move rtl-logical-classes test to ui/ so vitest:ui runner collects it

---------

Co-authored-by: Felipe Almeman <4226997+zhiru@users.noreply.github.com>
Co-authored-by: NOXX - Commiter <artur1992123@mail.ru>
Co-authored-by: Nick Sullivan <142708+TechNickAI@users.noreply.github.com>
Co-authored-by: Markus Hartung <mail@hartmark.se>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: PizzaV <103120356+pizzav-xyz@users.noreply.github.com>
Co-authored-by: Randi <55005611+rdself@users.noreply.github.com>
Co-authored-by: oyi77 <14921983+oyi77@users.noreply.github.com>
Co-authored-by: Chewji <126886556+Chewji9875@users.noreply.github.com>
Co-authored-by: Hernan Javier Ardila Sanchez <hjasgr@gmail.com>
Co-authored-by: Felipe Sartori <felipesartori.ti@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Zois Pagoulatos <zpagoulatos@hotmail.com>
Co-authored-by: sdfsdfw2 <167810361+sdfsdfw2@users.noreply.github.com>
Co-authored-by: Witroch4 <witalo_rocha@hotmail.com>
2026-06-12 23:49:22 -03:00

828 lines
27 KiB
TypeScript

/**
* Unit tests for src/lib/arenaEloSync.ts
*
* Uses Node.js native test runner. All external fetch calls are mocked.
* DB functions use a real in-memory SQLite instance via node:sqlite (DatabaseSync),
* injected through the core module's globalThis.__omnirouteDb singleton.
* backupDbFile is called during first sync but safely no-ops (no file on disk).
*/
import { describe, it, beforeEach, afterEach } from "node:test";
import assert from "node:assert/strict";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
const TEST_DATA_DIR = fs.mkdtempSync(
path.join(os.tmpdir(), "omniroute-arena-elo-test-"),
);
process.env.DATA_DIR = TEST_DATA_DIR;
const MIGRATION_SQL = fs.readFileSync(
path.resolve(
import.meta.dirname ?? __dirname,
"../../src/lib/db/migrations/097_model_intelligence.sql",
),
"utf8",
);
import { tryOpenSync } from "../../src/lib/db/adapters/driverFactory";
import type { SqliteAdapter } from "../../src/lib/db/adapters/types";
const core = await import("../../src/lib/db/core.ts");
const {
normalizeModelName,
transformToModelIntelligence,
fetchArenaLeaderboards,
syncArenaElo,
getArenaEloSyncStatus,
stopArenaEloSync,
} = await import("../../src/lib/arenaEloSync.ts");
import type {
ArenaLeaderboardData,
ArenaLeaderboardMap,
ArenaModelEntry,
} from "../../src/lib/arenaEloSync.ts";
const originalFetch = globalThis.fetch;
function mockFetch(
impl: (url: string, opts?: RequestInit) => Promise<Response>,
): void {
globalThis.fetch = impl as typeof fetch;
}
function restoreFetch(): void {
globalThis.fetch = originalFetch;
}
function jsonResponse(data: unknown, status = 200): Response {
return new Response(JSON.stringify(data), {
status,
headers: { "Content-Type": "application/json" },
});
}
function makeModelEntry(overrides: Partial<ArenaModelEntry> = {}): ArenaModelEntry {
return {
rank: 1,
model: "anthropic/claude-sonnet",
vendor: "Anthropic",
score: 1350,
ci: 10,
votes: 5000,
license: "proprietary",
...overrides,
};
}
function makeLeaderboardData(
models: ArenaModelEntry[] = [],
category = "text",
): ArenaLeaderboardData {
return {
meta: { leaderboard: category, model_count: models.length },
models,
};
}
function makeLeaderboardMap(
categories: Partial<Record<string, ArenaModelEntry[]>>,
): ArenaLeaderboardMap {
const map: ArenaLeaderboardMap = {};
for (const [cat, models] of Object.entries(categories)) {
map[cat] = makeLeaderboardData(models ?? [], cat);
}
return map;
}
let testAdapter: SqliteAdapter;
function createTestAdapter(): SqliteAdapter {
const patchedSql = MIGRATION_SQL.replace(
/\n\s*synced_at TEXT NOT NULL DEFAULT \(datetime\('now'\)\)/,
"\n synced_at TEXT NOT NULL",
);
const adapter = tryOpenSync(":memory:")!;
adapter.exec(patchedSql);
return adapter;
}
function countArenaEloEntries(): number {
const row = testAdapter
.prepare("SELECT COUNT(*) as cnt FROM model_intelligence WHERE source = 'arena_elo'")
.get() as Record<string, unknown> | undefined;
return Number(row?.cnt ?? 0);
}
function getAllEntries(): Array<Record<string, unknown>> {
return testAdapter
.prepare("SELECT * FROM model_intelligence WHERE source = 'arena_elo' ORDER BY model, category")
.all() as Array<Record<string, unknown>>;
}
beforeEach(() => {
core.resetDbInstance();
testAdapter = createTestAdapter();
globalThis.__omnirouteDb = testAdapter as never;
stopArenaEloSync();
});
afterEach(() => {
restoreFetch();
stopArenaEloSync();
delete globalThis.__omnirouteDb;
});
// ═══════════════════════════════════════════════════════════
// 1. normalizeModelName()
// ═══════════════════════════════════════════════════════════
describe("normalizeModelName()", () => {
it("strips 'anthropic/' vendor prefix", () => {
assert.strictEqual(
normalizeModelName("anthropic/claude-opus-4-6-thinking"),
"claude-opus-4-6-thinking",
);
});
it("strips 'openai/' vendor prefix", () => {
assert.strictEqual(normalizeModelName("openai/gpt-5.5"), "gpt-5.5");
});
it("strips 'google/' vendor prefix", () => {
assert.strictEqual(normalizeModelName("google/gemini-3-flash"), "gemini-3-flash");
});
it("strips 'meta/' vendor prefix", () => {
assert.strictEqual(normalizeModelName("meta/llama-4"), "llama-4");
});
it("strips 'deepseek/' vendor prefix", () => {
assert.strictEqual(normalizeModelName("deepseek/deepseek-r1"), "deepseek-r1");
});
it("strips 'xai/' vendor prefix", () => {
assert.strictEqual(normalizeModelName("xai/grok-4"), "grok-4");
});
it("lowercases the model name", () => {
assert.strictEqual(normalizeModelName("Claude-Sonnet-4"), "claude-sonnet-4");
});
it("lowercases vendor prefix before matching", () => {
assert.strictEqual(normalizeModelName("OpenAI/GPT-5.5"), "gpt-5.5");
});
it("returns name unchanged when no vendor prefix matches", () => {
assert.strictEqual(normalizeModelName("my-custom-model"), "my-custom-model");
});
});
// ═══════════════════════════════════════════════════════════
// 2. transformToModelIntelligence()
// ═══════════════════════════════════════════════════════════
describe("transformToModelIntelligence()", () => {
it("ELO normalization: 1500 ELO (max) with range 1000-1500 → score ≈ 0.98", () => {
const data = makeLeaderboardMap({
text: [
makeModelEntry({ model: "top-model", score: 1500, votes: 5000, rank: 1 }),
makeModelEntry({ model: "low-model", score: 1000, votes: 5000, rank: 2 }),
],
});
const entries = transformToModelIntelligence(data);
const topEntry = entries.find(
(e) => e.model === "top-model" && e.category === "default",
);
assert.ok(topEntry);
// taskFit = 0.4 + 0.58 * ((1500-1000) / 500) = 0.98
assert.ok(Math.abs(topEntry.score - 0.98) < 0.001, `got ${topEntry.score}`);
});
it("ELO normalization: 1000 ELO (min) with range 1000-1500 → score ≈ 0.40", () => {
const data = makeLeaderboardMap({
text: [
makeModelEntry({ model: "top-model", score: 1500, votes: 5000, rank: 1 }),
makeModelEntry({ model: "low-model", score: 1000, votes: 5000, rank: 2 }),
],
});
const entries = transformToModelIntelligence(data);
const lowEntry = entries.find(
(e) => e.model === "low-model" && e.category === "default",
);
assert.ok(lowEntry);
// taskFit = 0.4 + 0.58 * ((1000-1000) / 500) = 0.4
assert.ok(Math.abs(lowEntry.score - 0.4) < 0.001, `got ${lowEntry.score}`);
});
it("votes < 100 → confidence='low'", () => {
const data = makeLeaderboardMap({
text: [
makeModelEntry({ model: "sparse-model", score: 1200, votes: 50, rank: 5 }),
makeModelEntry({ model: "baseline", score: 1100, votes: 5000, rank: 10 }),
],
});
const entries = transformToModelIntelligence(data);
const entry = entries.find(
(e) => e.model === "sparse-model" && e.category === "default",
);
assert.ok(entry);
assert.strictEqual(entry.confidence, "low");
});
it("votes >= 1000 → confidence='medium'", () => {
const data = makeLeaderboardMap({
text: [
makeModelEntry({ model: "mid-model", score: 1200, votes: 1500, rank: 3 }),
makeModelEntry({ model: "baseline", score: 1100, votes: 5000, rank: 10 }),
],
});
const entries = transformToModelIntelligence(data);
const entry = entries.find(
(e) => e.model === "mid-model" && e.category === "default",
);
assert.ok(entry);
assert.strictEqual(entry.confidence, "medium");
});
it("votes >= 5000 → confidence='high'", () => {
const data = makeLeaderboardMap({
text: [
makeModelEntry({ model: "popular-model", score: 1300, votes: 8000, rank: 1 }),
makeModelEntry({ model: "baseline", score: 1100, votes: 3000, rank: 5 }),
],
});
const entries = transformToModelIntelligence(data);
const entry = entries.find(
(e) => e.model === "popular-model" && e.category === "default",
);
assert.ok(entry);
assert.strictEqual(entry.confidence, "high");
});
it("category mapping: 'text' → [default, review, documentation, debugging]", () => {
const data = makeLeaderboardMap({
text: [
makeModelEntry({ model: "text-model", score: 1200, votes: 5000, rank: 1 }),
],
});
const entries = transformToModelIntelligence(data);
const categories = entries
.filter((e) => e.model === "text-model")
.map((e) => e.category)
.sort();
assert.deepStrictEqual(categories, [
"debugging",
"default",
"documentation",
"review",
]);
});
it("category mapping: 'code' → [coding]", () => {
const data = makeLeaderboardMap({
code: [
makeModelEntry({ model: "code-model", score: 1300, votes: 5000, rank: 1 }),
],
});
const entries = transformToModelIntelligence(data);
const categories = entries
.filter((e) => e.model === "code-model")
.map((e) => e.category);
assert.deepStrictEqual(categories, ["coding"]);
});
it("expires_at is set to ~7 days in the future", () => {
const before = Date.now();
const data = makeLeaderboardMap({
text: [
makeModelEntry({ model: "test-model", score: 1200, votes: 5000, rank: 1 }),
],
});
const entries = transformToModelIntelligence(data);
const after = Date.now();
const entry = entries.find(
(e) => e.model === "test-model" && e.category === "default",
);
assert.ok(entry);
assert.ok(entry.expiresAt);
const expiresMs = new Date(entry.expiresAt).getTime();
const sevenDaysMs = 7 * 24 * 60 * 60 * 1000;
assert.ok(
expiresMs >= before + sevenDaysMs - 2000,
`expiresAt too early: ${entry.expiresAt}`,
);
assert.ok(
expiresMs <= after + sevenDaysMs + 2000,
`expiresAt too late: ${entry.expiresAt}`,
);
});
it("source is 'arena_elo' for all entries", () => {
const data = makeLeaderboardMap({
text: [
makeModelEntry({ model: "test-model", score: 1200, votes: 5000, rank: 1 }),
],
});
const entries = transformToModelIntelligence(data);
for (const entry of entries) {
assert.strictEqual(entry.source, "arena_elo");
}
});
it("expands model aliases for known models", () => {
const data = makeLeaderboardMap({
text: [
makeModelEntry({
model: "anthropic/claude-opus-4-6-thinking",
score: 1400,
votes: 5000,
rank: 1,
}),
],
});
const entries = transformToModelIntelligence(data);
const models = entries.map((e) => e.model);
assert.ok(models.includes("claude-opus-4-6-thinking"));
assert.ok(models.includes("claude-opus-4"));
assert.ok(models.includes("anthropic/claude-opus-4"));
});
it("empty leaderboard → no entries", () => {
const data = makeLeaderboardMap({ text: [] });
const entries = transformToModelIntelligence(data);
assert.strictEqual(entries.length, 0);
});
it("skips unknown leaderboard categories (e.g. vision)", () => {
const data = makeLeaderboardMap({
vision: [
makeModelEntry({ model: "vision-model", score: 1300, votes: 5000, rank: 1 }),
],
});
const entries = transformToModelIntelligence(data);
assert.strictEqual(entries.length, 0);
});
it("preserves eloRaw from the leaderboard", () => {
const data = makeLeaderboardMap({
text: [
makeModelEntry({ model: "test-model", score: 1337, votes: 5000, rank: 1 }),
makeModelEntry({ model: "other-model", score: 1100, votes: 3000, rank: 2 }),
],
});
const entries = transformToModelIntelligence(data);
const entry = entries.find(
(e) => e.model === "test-model" && e.category === "default",
);
assert.ok(entry);
assert.strictEqual(entry.eloRaw, 1337);
});
it("handles single-model leaderboard (eloRange = 1, avoids division by zero)", () => {
const data = makeLeaderboardMap({
text: [
makeModelEntry({ model: "only-model", score: 1200, votes: 5000, rank: 1 }),
],
});
const entries = transformToModelIntelligence(data);
assert.ok(entries.length > 0);
const entry = entries.find(
(e) => e.model === "only-model" && e.category === "default",
);
assert.ok(entry);
assert.ok(Math.abs(entry.score - 0.4) < 0.001);
});
it("rounds score to 4 decimal places", () => {
const data = makeLeaderboardMap({
text: [
makeModelEntry({ model: "model-a", score: 1300, votes: 200, rank: 1 }),
makeModelEntry({ model: "model-b", score: 1000, votes: 200, rank: 2 }),
],
});
const entries = transformToModelIntelligence(data);
for (const entry of entries) {
const str = entry.score.toString();
const dot = str.indexOf(".");
if (dot !== -1) {
assert.ok(str.length - dot - 1 <= 4, `score ${entry.score} > 4 decimals`);
}
}
});
});
// ═══════════════════════════════════════════════════════════
// 3. fetchArenaLeaderboards()
// ═══════════════════════════════════════════════════════════
describe("fetchArenaLeaderboards()", () => {
it("successful fetch with valid JSON returns both leaderboards", async () => {
const textData = makeLeaderboardData(
[makeModelEntry({ model: "text-model", score: 1200, votes: 5000, rank: 1 })],
"text",
);
const codeData = makeLeaderboardData(
[makeModelEntry({ model: "code-model", score: 1300, votes: 5000, rank: 1 })],
"code",
);
mockFetch(async (url: string) => {
if (url.includes("name=text")) return jsonResponse(textData);
if (url.includes("name=code")) return jsonResponse(codeData);
return new Response("Not found", { status: 404 });
});
const result = await fetchArenaLeaderboards();
assert.ok(result.text);
assert.ok(result.code);
assert.strictEqual(result.text.models.length, 1);
assert.strictEqual(result.code.models.length, 1);
});
it("failed fetch (non-200 status) throws with descriptive message", async () => {
mockFetch(async () => {
return new Response("Internal Server Error", { status: 500 });
});
await assert.rejects(
() => fetchArenaLeaderboards(),
(err: unknown) => {
assert.ok(err instanceof Error);
assert.ok(err.message.includes("All Arena leaderboard fetches failed"));
return true;
},
);
});
it("all fetches fail (network error) → throws", async () => {
mockFetch(async () => {
throw new Error("Network error: ECONNREFUSED");
});
await assert.rejects(
() => fetchArenaLeaderboards(),
(err: unknown) => {
assert.ok(err instanceof Error);
assert.ok(err.message.includes("All Arena leaderboard fetches failed"));
return true;
},
);
});
it("succeeds when one category fails but another succeeds", async () => {
const textData = makeLeaderboardData(
[makeModelEntry({ model: "text-model", score: 1200, votes: 5000, rank: 1 })],
"text",
);
mockFetch(async (url: string) => {
if (url.includes("name=text")) return jsonResponse(textData);
if (url.includes("name=code"))
return new Response("Error", { status: 500 });
return new Response("Not found", { status: 404 });
});
const result = await fetchArenaLeaderboards();
assert.ok(result.text);
assert.strictEqual(result.code, undefined);
});
it("invalid JSON in response → throws when all responses are invalid", async () => {
mockFetch(async () => {
return new Response("not-json", {
status: 200,
headers: { "Content-Type": "text/plain" },
});
});
await assert.rejects(
() => fetchArenaLeaderboards(),
(err: unknown) => {
assert.ok(err instanceof Error);
assert.ok(err.message.includes("All Arena leaderboard fetches failed"));
return true;
},
);
});
});
// ═══════════════════════════════════════════════════════════
// 4. syncArenaElo()
// ═══════════════════════════════════════════════════════════
describe("syncArenaElo()", () => {
it("happy path: returns success=true with correct modelCount", async () => {
const textData = makeLeaderboardData(
[makeModelEntry({ model: "gpt-5.5", score: 1200, votes: 5000, rank: 1 })],
"text",
);
const codeData = makeLeaderboardData(
[makeModelEntry({ model: "deepseek-r1", score: 1300, votes: 5000, rank: 1 })],
"code",
);
mockFetch(async (url: string) => {
if (url.includes("name=text")) return jsonResponse(textData);
if (url.includes("name=code")) return jsonResponse(codeData);
return new Response("Not found", { status: 404 });
});
const result = await syncArenaElo();
assert.strictEqual(result.success, true);
assert.strictEqual(result.source, "arena_elo");
assert.ok(result.modelCount > 0);
const dbCount = countArenaEloEntries();
assert.ok(dbCount > 0);
assert.strictEqual(dbCount, result.modelCount);
});
it("happy path: entries in DB have correct source and categories", async () => {
const textData = makeLeaderboardData(
[makeModelEntry({ model: "unique-test-model", score: 1200, votes: 5000, rank: 1 })],
"text",
);
mockFetch(async (url: string) => {
if (url.includes("name=text")) return jsonResponse(textData);
if (url.includes("name=code"))
return jsonResponse(makeLeaderboardData([], "code"));
return new Response("Not found", { status: 404 });
});
await syncArenaElo();
const entries = getAllEntries().filter((e) => String(e.model) === "unique-test-model");
const categories = entries.map((e) => String(e.category)).sort();
assert.deepStrictEqual(categories, [
"debugging",
"default",
"documentation",
"review",
]);
for (const entry of entries) {
assert.strictEqual(entry.source, "arena_elo");
}
});
it("dryRun=true → does not call bulkUpsertModelIntelligence", async () => {
const textData = makeLeaderboardData(
[makeModelEntry({ model: "test-model", score: 1200, votes: 5000, rank: 1 })],
"text",
);
mockFetch(async (url: string) => {
if (url.includes("name=text")) return jsonResponse(textData);
if (url.includes("name=code"))
return jsonResponse(makeLeaderboardData([], "code"));
return new Response("Not found", { status: 404 });
});
const result = await syncArenaElo(true);
assert.strictEqual(result.success, true);
assert.ok(result.modelCount > 0);
const dbCount = countArenaEloEntries();
assert.strictEqual(dbCount, 0, "dryRun should not write to DB");
});
it("dryRun=true does not update lastSyncTime", async () => {
// Reset module-level state by observing that before this test's dryRun,
// lastSync should remain unchanged from whatever it was.
// Since we can't reset module-level vars, we verify that a dryRun
// after a non-dryRun doesn't overwrite the model count.
const statusBefore = getArenaEloSyncStatus();
const textData = makeLeaderboardData(
[makeModelEntry({ model: "test-model", score: 1200, votes: 5000, rank: 1 })],
"text",
);
mockFetch(async (url: string) => {
if (url.includes("name=text")) return jsonResponse(textData);
if (url.includes("name=code"))
return jsonResponse(makeLeaderboardData([], "code"));
return new Response("Not found", { status: 404 });
});
await syncArenaElo(true);
const statusAfter = getArenaEloSyncStatus();
// dryRun should not change the modelCount
assert.strictEqual(statusAfter.lastSyncModelCount, statusBefore.lastSyncModelCount);
});
it("API failure → returns success=false with error", async () => {
mockFetch(async () => {
throw new Error("Network down");
});
const result = await syncArenaElo();
assert.strictEqual(result.success, false);
assert.strictEqual(result.source, "arena_elo");
assert.strictEqual(result.modelCount, 0);
assert.ok(result.error);
assert.ok(
result.error!.includes("All Arena leaderboard fetches failed"),
`unexpected: ${result.error}`,
);
});
it("calls deleteExpiredIntelligence before writing new entries", async () => {
testAdapter.prepare(
"INSERT INTO model_intelligence (model, source, category, score, elo_raw, confidence, synced_at, expires_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
).run("old-model", "arena_elo", "default", 0.5, 1000, "low", "2025-01-01T00:00:00Z", "2020-01-01T00:00:00Z");
assert.strictEqual(countArenaEloEntries(), 1);
const textData = makeLeaderboardData(
[makeModelEntry({ model: "new-model", score: 1200, votes: 5000, rank: 1 })],
"text",
);
mockFetch(async (url: string) => {
if (url.includes("name=text")) return jsonResponse(textData);
if (url.includes("name=code"))
return jsonResponse(makeLeaderboardData([], "code"));
return new Response("Not found", { status: 404 });
});
const syncResult = await syncArenaElo();
assert.strictEqual(syncResult.success, true);
const expiredEntry = testAdapter
.prepare("SELECT * FROM model_intelligence WHERE model = 'old-model'")
.get();
assert.strictEqual(expiredEntry, undefined);
});
it("handles empty leaderboard gracefully (0 entries, no DB write)", async () => {
mockFetch(async (url: string) => {
if (url.includes("name=text"))
return jsonResponse(makeLeaderboardData([], "text"));
if (url.includes("name=code"))
return jsonResponse(makeLeaderboardData([], "code"));
return new Response("Not found", { status: 404 });
});
const result = await syncArenaElo();
assert.strictEqual(result.success, true);
assert.strictEqual(result.modelCount, 0);
assert.strictEqual(countArenaEloEntries(), 0);
});
it("updates lastSyncTime after successful sync", async () => {
const textData = makeLeaderboardData(
[makeModelEntry({ model: "test-model", score: 1200, votes: 5000, rank: 1 })],
"text",
);
mockFetch(async (url: string) => {
if (url.includes("name=text")) return jsonResponse(textData);
if (url.includes("name=code"))
return jsonResponse(makeLeaderboardData([], "code"));
return new Response("Not found", { status: 404 });
});
await syncArenaElo();
const status = getArenaEloSyncStatus();
assert.ok(status.lastSync);
assert.ok(status.lastSyncModelCount > 0);
});
it("model aliases are stored in DB alongside canonical names", async () => {
const textData = makeLeaderboardData(
[makeModelEntry({
model: "anthropic/claude-opus-4-6-thinking",
score: 1400,
votes: 5000,
rank: 1,
})],
"text",
);
mockFetch(async (url: string) => {
if (url.includes("name=text")) return jsonResponse(textData);
if (url.includes("name=code"))
return jsonResponse(makeLeaderboardData([], "code"));
return new Response("Not found", { status: 404 });
});
await syncArenaElo();
const entries = getAllEntries();
const models = entries.map((e) => String(e.model));
assert.ok(models.includes("claude-opus-4-6-thinking"));
assert.ok(models.includes("claude-opus-4"));
});
});
// ═══════════════════════════════════════════════════════════
// 5. getArenaEloSyncStatus()
// ═══════════════════════════════════════════════════════════
describe("getArenaEloSyncStatus()", () => {
it("returns correct structure with all expected keys", () => {
const status = getArenaEloSyncStatus();
assert.ok("enabled" in status);
assert.ok("lastSync" in status);
assert.ok("lastSyncModelCount" in status);
assert.ok("nextSync" in status);
assert.ok("intervalMs" in status);
assert.ok("sources" in status);
});
it("returns sources containing 'arena_elo'", () => {
const status = getArenaEloSyncStatus();
assert.deepStrictEqual(status.sources, ["arena_elo"]);
});
it("returns intervalMs as a positive number", () => {
const status = getArenaEloSyncStatus();
assert.ok(typeof status.intervalMs === "number");
assert.ok(status.intervalMs > 0);
});
it("returns lastSyncModelCount as 0 before any non-dryRun sync completes in this suite context", () => {
// Module-level lastSyncTime may leak from earlier tests in the suite,
// but the structural fields are always present.
const status = getArenaEloSyncStatus();
assert.ok(typeof status.lastSync === "string" || status.lastSync === null);
assert.ok(typeof status.lastSyncModelCount === "number");
assert.ok(typeof status.nextSync === "string" || status.nextSync === null);
});
it("reflects ARENA_ELO_SYNC_ENABLED env var", () => {
const original = process.env.ARENA_ELO_SYNC_ENABLED;
process.env.ARENA_ELO_SYNC_ENABLED = "true";
const enabledStatus = getArenaEloSyncStatus();
assert.strictEqual(enabledStatus.enabled, true);
process.env.ARENA_ELO_SYNC_ENABLED = "false";
const disabledStatus = getArenaEloSyncStatus();
assert.strictEqual(disabledStatus.enabled, false);
if (original !== undefined) {
process.env.ARENA_ELO_SYNC_ENABLED = original;
} else {
delete process.env.ARENA_ELO_SYNC_ENABLED;
}
});
});
// ═══════════════════════════════════════════════════════════
// 6. stopArenaEloSync()
// ═══════════════════════════════════════════════════════════
describe("stopArenaEloSync()", () => {
it("does not throw when no timer is running", () => {
assert.doesNotThrow(() => stopArenaEloSync());
});
it("can be called multiple times without error", () => {
stopArenaEloSync();
assert.doesNotThrow(() => stopArenaEloSync());
assert.doesNotThrow(() => stopArenaEloSync());
});
});