pjandro/OmniRoute
1
0
Fork 0
mirror of https://github.com/diegosouzapw/OmniRoute.git synced 2026-07-26 09:52:11 +03:00
Code Issues Packages Projects Releases Wiki Activity

feat: v3.6.4 — Combo Builder v2, Composite Tiers, P2C Credentials, Observability Layer

Browse Source 
## New Features
- Combo Builder v2 wizard UI (multi-stage: Basics → Steps → Strategy → Review)
- Combo Step Architecture Schema v2 (ComboModelStep, ComboRefStep, pinned accounts)
- Composite Tiers system for tiered model routing with fallback chains
- Model Capabilities Registry (unified resolver merging specs + registry + synced data)
- Observability module (buildHealthPayload, buildTelemetryPayload, buildSessionsSummary)
- Session & Quota Monitor panels on Health dashboard
- Combo Health per-target analytics via resolveNestedComboTargets()
- Combo Builder Options API (GET /api/combos/builder/options)

## Performance
- Middleware lazy loading (apiAuth, db/settings, modelSyncScheduler)
- E2E auth bypass mode (NEXT_PUBLIC_OMNIROUTE_E2E_MODE)

## Bug Fixes
- P2C credential selection with quota headroom awareness
- Fixed-account combo steps bypass model cooldowns/circuit breakers
- Combo metrics per-target tracking (byTarget with executionKey)
- Call logs schema expansion (7 new columns + composite index)
- Quota monitor lifecycle enrichment (status, snapshots, summary)
- Codex quota fetcher hardening

## Maintenance
- DB migration 021 (combo_call_log_targets)
- Combo CRUD normalization on read
- Playwright config + build script improvements
- OpenAPI spec version sync to 3.6.4

## Tests
- 16 new test suites + 12 existing test updates
- 86 files changed, +8318 -1378 lines
This commit is contained in:
diegosouzapw
2026-04-12 10:34:10 -03:00
parent f15576fd98
commit ea61d00cf7
87 changed files with 8431 additions and 1436 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
.env.example
View File

@@ -25,9 +25,8 @@ API_KEY_SECRET=
# Initial admin login password — CHANGE THIS before first use!
# Used by: bootstrap only — sets the initial dashboard password on first boot.
# After first login you can change it from Dashboard → Settings → Security.
# Default: 123456 (insecure, for local dev only)
INITIAL_PASSWORD=123456
# Default: CHANGEME (insecure, for local dev only)
INITIAL_PASSWORD=CHANGEME
# ═══════════════════════════════════════════════════════════════════════════════
# 2. STORAGE & DATABASE

55
CHANGELOG.md
View File

@@ -4,6 +4,61 @@
---
## [3.6.4] — 2026-04-12
### ✨ New Features
- **Combo Builder v2 (Wizard UI):** Completely redesigned the combo creation/editing interface as a multi-stage wizard with stages: Basics → Steps → Strategy → Review. The builder fetches provider, model, and connection metadata via a new `GET /api/combos/builder/options` endpoint, enabling precise provider/model/account selection with duplicate detection and automatic next-connection suggestion. Heavy UI components (`ModelSelectModal`, `ProxyConfigModal`, `ModelRoutingSection`) are now lazily loaded via `next/dynamic` for faster initial page render
- **Combo Step Architecture (Schema v2):** Introduced a structured step model (`ComboModelStep`, `ComboRefStep`) replacing the legacy flat string/object combo entries. Steps carry explicit `id`, `kind`, `providerId`, `connectionId`, `weight`, and `label` fields, enabling pinned-account routing, cross-combo references, and per-step metrics. All combo CRUD operations normalize entries through the new `src/lib/combos/steps.ts` module. Zod schemas updated with `comboModelStepInputSchema` and `comboRefStepInputSchema` unions
- **Composite Tiers System:** Added tiered model routing via `config.compositeTiers` — each tier maps a named stage to a specific combo step with optional fallback chains. Includes comprehensive validation (`src/lib/combos/compositeTiers.ts`) ensuring step existence, preventing circular fallback, and validating default tier references. Zod schema enforcement blocks composite tiers on global defaults (concrete combos only)
- **Model Capabilities Registry:** Created `src/lib/modelCapabilities.ts` providing `getResolvedModelCapabilities()` — a unified resolver that merges static specs, provider registry data, and live-synced capabilities into a single `ResolvedModelCapabilities` object covering tool calling, reasoning, vision, context window, thinking budget, modalities, and model lifecycle metadata
- **Observability Module:** Extracted health and telemetry payload construction into `src/lib/monitoring/observability.ts` with `buildHealthPayload()`, `buildTelemetryPayload()`, and `buildSessionsSummary()` builders. The health endpoint now returns session activity, quota monitor status, and per-provider breakdowns alongside existing system metrics
- **Session & Quota Monitor Dashboard:** Added live Session Activity and Quota Monitors panels to the Health dashboard, showing active session counts, sticky-bound sessions, per-API-key breakdowns, and top session details alongside quota monitor alerting/exhausted/error status with per-provider drill-down
- **Combo Health Per-Target Analytics:** The combo-health API now resolves per-target metrics using the new `resolveNestedComboTargets()` function, providing step-level success rates, latency, and historical usage breakdowns per execution key — enabling per-account, per-connection health visibility
### ⚡ Performance
- **Middleware Lazy Loading:** Refactored `src/proxy.ts` to lazy-import `apiAuth`, `db/settings`, and `modelSyncScheduler` modules, reducing middleware cold-start overhead. Added inline `isPublicApiRoute()` to avoid loading the full auth module for public routes
- **E2E Auth Bypass:** Added `NEXT_PUBLIC_OMNIROUTE_E2E_MODE` environment flag to bypass authentication gates for dashboard and management API routes during Playwright E2E test runs
### 🐛 Bug Fixes
- **P2C Credential Selection:** Implemented Power-of-Two-Choices (P2C) connection scoring in `src/sse/services/auth.ts` with quota headroom awareness, error/recency penalties, and forced/excluded connection support. The new `getProviderCredentialsWithQuotaPreflight()` function integrates quota preflight checks directly into credential selection, eliminating the separate Codex-only preflight path
- **Fixed-Account Combo Steps:** Combo steps with explicit `connectionId` now correctly bypass provider-level model cooldowns and circuit breakers, preventing a single account failure from blocking pinned-connection routing for the same model
- **Combo Metrics Per-Target Tracking:** Extended `comboMetrics.ts` to track `byTarget` metrics keyed by execution path, recording per-step `provider`, `providerId`, `connectionId`, and `label` alongside existing per-model aggregates
- **Call Logs Schema Expansion:** Added `requested_model`, `request_type`, `tokens_cache_read`, `tokens_cache_creation`, `tokens_reasoning`, `combo_step_id`, and `combo_execution_key` columns to `call_logs` with auto-migration. Added composite index `idx_cl_combo_target` for efficient per-target historical queries
- **Quota Monitor Enrichment:** Expanded `quotaMonitor.ts` with full lifecycle state tracking (`status`, `startedAt`, `lastPolledAt`, `consecutiveFailures`, `totalPolls`, `totalAlerts`), ISO-formatted snapshots via `getQuotaMonitorSnapshots()`, and sorted summary via `getQuotaMonitorSummary()`
- **Codex Quota Fetcher Hardening:** Improved `codexQuotaFetcher.ts` with safer connection registration and quota fetch error handling
### 🔧 Maintenance & Infrastructure
- **DB Migration 021:** Added `combo_call_log_targets` migration for `combo_step_id` and `combo_execution_key` columns in call_logs
- **Combo CRUD Normalization:** `db/combos.ts` now normalizes all stored combo entries through the step normalization pipeline on read, ensuring consistent step IDs and kind annotations regardless of when the combo was created
- **Playwright Config:** Updated Playwright configuration and `run-next-playwright.mjs` script for improved E2E test orchestration
- **Build Script:** Updated `build-next-isolated.mjs` with additional reliability improvements
### 🧪 Tests
- **16 New Test Suites:** Added comprehensive test coverage including:
- `combo-builder-draft.test.mjs` (186 lines) — Builder draft step construction and validation
- `combo-builder-options-route.test.mjs` (228 lines) — Builder options API endpoint
- `combo-health-route.test.mjs` (266 lines) — Combo health analytics with per-target metrics
- `combo-routes-composite-tiers.test.mjs` (157 lines) — Composite tiers API integration
- `composite-tiers-validation.test.mjs` (131 lines) — Composite tier validation rules
- `db-combos-crud.test.mjs` — Combo CRUD with step normalization
- `db-core-init.test.mjs` (129 lines) — DB initialization and column migrations
- `model-capabilities-registry.test.mjs` (105 lines) — Model capabilities resolution
- `observability-payloads.test.mjs` (165 lines) — Health/telemetry payload construction
- `openapi-spec-route.test.mjs` — OpenAPI spec generation
- `proxy-e2e-mode.test.mjs` (74 lines) — E2E mode auth bypass
- `quota-monitor.test.mjs` — Quota monitor lifecycle state
- `run-next-playwright.test.mjs` (119 lines) — Playwright runner script
- `sse-auth.test.mjs` (154 lines) — P2C credential selection and quota preflight
- `telemetry-summary-route.test.mjs` (35 lines) — Telemetry summary endpoint
- Plus updates to 12 existing test files for compatibility with new step architecture
---
## [3.6.3] — 2026-04-11
### ✨ New Features

2
docs/openapi.yaml
View File

@@ -1,7 +1,7 @@
openapi: 3.1.0
info:
title: OmniRoute API
version: 3.6.3
version: 3.6.4
description: |
OmniRoute is a local-first AI API proxy router. It provides an OpenAI-compatible
endpoint that routes requests to multiple AI providers with load balancing,

2
open-sse/config/providerRegistry.ts
View File

@@ -14,6 +14,8 @@ export interface RegistryModel {
id: string;
name: string;
toolCalling?: boolean;
supportsReasoning?: boolean;
supportsVision?: boolean;
targetFormat?: string;
unsupportedParams?: readonly string[];
/** Maximum context window in tokens */

4
open-sse/handlers/chatCore.ts
View File

@@ -479,6 +479,8 @@ export async function handleChatCore({
comboName,
comboStrategy = null,
isCombo = false,
comboStepId = null,
comboExecutionKey = null,
disableEmergencyFallback = false,
}) {
let { provider, model, extendedContext } = modelInfo;
@@ -746,6 +748,8 @@ export async function handleChatCore({
sourceFormat,
targetFormat,
comboName,
comboStepId,
comboExecutionKey,
apiKeyId: apiKeyInfo?.id || null,
apiKeyName: apiKeyInfo?.name || null,
noLog: noLogEnabled,

19
open-sse/mcp-server/server.ts
View File

@@ -12,6 +12,11 @@
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import {
getComboModelProvider,
getComboModelString,
getComboStepTarget,
} from "../../src/lib/combos/steps.ts";
import {
MCP_TOOLS,
@@ -105,11 +110,17 @@ function normalizeComboModels(
rawModels: unknown
): Array<{ provider: string; model: string; priority: number }> {
return toArray(rawModels).map((rawModel, index) => {
const model = toRecord(rawModel);
const modelRecord = toRecord(rawModel);
const modelString = getComboModelString(rawModel);
const target = getComboStepTarget(rawModel);
const provider =
getComboModelProvider(rawModel) ||
(modelString ? "unknown" : target ? "combo" : toString(modelRecord.provider, "unknown"));
return {
provider: toString(model.provider, "unknown"),
model: toString(model.model, "unknown"),
priority: toNumber(model.priority, index + 1),
provider,
model: modelString || target || toString(modelRecord.model, "unknown"),
priority: toNumber(modelRecord.priority, index + 1),
};
});
}

9
open-sse/mcp-server/tools/advancedTools.ts
View File

@@ -18,6 +18,11 @@
import { logToolCall } from "../audit.ts";
import { normalizeQuotaResponse } from "../../../src/shared/contracts/quota.ts";
import { resolveOmniRouteBaseUrl } from "../../../src/shared/utils/resolveOmniRouteBaseUrl.ts";
import {
getComboModelProvider,
getComboModelString,
getComboStepTarget,
} from "../../../src/lib/combos/steps.ts";
const OMNIROUTE_BASE_URL = resolveOmniRouteBaseUrl();
const OMNIROUTE_API_KEY = process.env.OMNIROUTE_API_KEY || "";
@@ -80,8 +85,8 @@ function getComboModels(combo: JsonRecord): ComboModel[] {
const nestedModels = toArrayOfRecords(toRecord(combo.data).models);
const sourceModels = directModels.length > 0 ? directModels : nestedModels;
return sourceModels.map((model) => ({
provider: toString(model.provider, "unknown"),
model: toString(model.model, ""),
provider: getComboModelProvider(model) || (getComboModelString(model) ? "unknown" : "combo"),
model: getComboModelString(model) || getComboStepTarget(model) || "",
inputCostPer1M: toNumber(model.inputCostPer1M, 3.0),
}));
}

55
open-sse/services/codexQuotaFetcher.ts
View File

@@ -76,6 +76,50 @@ export function registerCodexConnection(connectionId: string, meta: CodexConnect
connectionRegistry.set(connectionId, meta);
}
function getCodexConnectionMeta(
connectionId: string,
connection?: Record<string, unknown>
): CodexConnectionMeta | null {
if (connection && typeof connection === "object") {
const providerSpecificData =
connection.providerSpecificData &&
typeof connection.providerSpecificData === "object" &&
!Array.isArray(connection.providerSpecificData)
? (connection.providerSpecificData as Record<string, unknown>)
: {};
const accessToken =
typeof connection.accessToken === "string" && connection.accessToken.trim().length > 0
? connection.accessToken
: null;
const workspaceId =
typeof providerSpecificData.workspaceId === "string" &&
providerSpecificData.workspaceId.trim().length > 0
? providerSpecificData.workspaceId
: undefined;
if (accessToken) {
const meta = { accessToken, ...(workspaceId ? { workspaceId } : {}) };
connectionRegistry.set(connectionId, meta);
return meta;
}
}
return connectionRegistry.get(connectionId) || null;
}
function getDominantResetAt(quota: {
window5h: { percentUsed: number; resetAt: string | null };
window7d: { percentUsed: number; resetAt: string | null };
}): string | null {
if (quota.window7d.percentUsed > quota.window5h.percentUsed) {
return quota.window7d.resetAt || quota.window5h.resetAt;
}
if (quota.window5h.percentUsed > quota.window7d.percentUsed) {
return quota.window5h.resetAt || quota.window7d.resetAt;
}
return quota.window7d.resetAt || quota.window5h.resetAt;
}
// ─── Core Fetcher ────────────────────────────────────────────────────────────
/**
@@ -85,7 +129,10 @@ export function registerCodexConnection(connectionId: string, meta: CodexConnect
* @param connectionId - Connection ID from the DB (used to look up credentials)
* @returns QuotaInfo or null if fetch fails / no credentials
*/
export async function fetchCodexQuota(connectionId: string): Promise<QuotaInfo | null> {
export async function fetchCodexQuota(
connectionId: string,
connection?: Record<string, unknown>
): Promise<QuotaInfo | null> {
// Check cache first
const cached = quotaCache.get(connectionId);
if (cached && Date.now() - cached.fetchedAt < CACHE_TTL_MS) {
@@ -93,7 +140,7 @@ export async function fetchCodexQuota(connectionId: string): Promise<QuotaInfo |
}
// Look up credentials
const meta = connectionRegistry.get(connectionId);
const meta = getCodexConnectionMeta(connectionId, connection);
if (!meta?.accessToken) {
// No credentials registered — skip preflight gracefully
return null;
@@ -206,6 +253,10 @@ function parseCodexUsageResponse(data: unknown): CodexDualWindowQuota | null {
used: worstPercentUsed,
total: 100,
percentUsed: percentUsedNormalized,
resetAt: getDominantResetAt({
window5h: { percentUsed: usedPercent5h / 100, resetAt: resetAt5h },
window7d: { percentUsed: usedPercent7d / 100, resetAt: resetAt7d },
}),
window5h: { percentUsed: usedPercent5h / 100, resetAt: resetAt5h },
window7d: { percentUsed: usedPercent7d / 100, resetAt: resetAt7d },
limitReached,

590
open-sse/services/combo.ts
View File

@@ -18,10 +18,17 @@ import { applyComboAgentMiddleware, injectModelTag } from "./comboAgentMiddlewar
import { classifyWithConfig, DEFAULT_INTENT_CONFIG } from "./intentClassifier.ts";
import { selectProvider as selectAutoProvider } from "./autoCombo/engine.ts";
import { selectWithStrategy } from "./autoCombo/routerStrategy.ts";
import { DEFAULT_WEIGHTS, scorePool } from "./autoCombo/scoring.ts";
import { getTaskFitness } from "./autoCombo/taskFitness.ts";
import { calculateFactors, calculateScore, DEFAULT_WEIGHTS } from "./autoCombo/scoring.ts";
import { supportsToolCalling } from "./modelCapabilities.ts";
import { getSessionConnection } from "./sessionManager.ts";
import { getModelContextLimit } from "../../src/lib/modelsDevSync";
import { getModelContextLimit } from "../../src/lib/modelCapabilities";
import {
getComboModelString,
getComboStepTarget,
getComboStepWeight,
normalizeComboStep,
} from "../../src/lib/combos/steps.ts";
// Status codes that should mark semaphore + record circuit breaker failures
const TRANSIENT_FOR_BREAKER = [429, 502, 503, 504];
@@ -51,6 +58,29 @@ const DEFAULT_MODEL_P95_MS = {
};
const MIN_HISTORY_SAMPLES = 10;
type ResolvedComboTarget = {
kind: "model";
stepId: string;
executionKey: string;
modelStr: string;
provider: string;
providerId: string | null;
connectionId: string | null;
weight: number;
label: string | null;
};
type ComboRuntimeStep =
| ResolvedComboTarget
| {
kind: "combo-ref";
stepId: string;
executionKey: string;
comboName: string;
weight: number;
label: string | null;
};
/**
* Validate that a successful (HTTP 200) non-streaming response actually contains
* meaningful content. Returns { valid: true } or { valid: false, reason }.
@@ -140,8 +170,128 @@ const rrCounters = new Map();
* Supports both legacy string format and new object format
*/
function normalizeModelEntry(entry) {
if (typeof entry === "string") return { model: entry, weight: 0 };
return { model: entry.model, weight: entry.weight || 0 };
return {
model: getComboStepTarget(entry) || "",
weight: getComboStepWeight(entry),
};
}
function getTargetProvider(modelStr: string, providerId?: string | null): string {
const parsed = parseModel(modelStr);
return providerId || parsed.provider || parsed.providerAlias || "unknown";
}
function getComboBreakerKey(comboName: string, executionKey: string): string {
return `combo:${comboName}:${executionKey}`;
}
function toRecordedTarget(target: ResolvedComboTarget) {
return {
executionKey: target.executionKey,
stepId: target.stepId,
provider: target.provider,
providerId: target.providerId,
connectionId: target.connectionId,
label: target.label,
};
}
function buildExecutionKey(path: string[], stepId: string): string {
return [...path, stepId].join(">");
}
function normalizeRuntimeStep(entry, comboName, index, allCombos, path = []) {
const step = normalizeComboStep(entry, {
comboName,
index,
allCombos,
});
if (!step) return null;
const executionKey = buildExecutionKey(path, step.id);
const label = typeof step.label === "string" ? step.label : null;
const weight = step.weight || 0;
if (step.kind === "combo-ref") {
return {
kind: "combo-ref",
stepId: step.id,
executionKey,
comboName: step.comboName,
weight,
label,
};
}
const modelStr = getComboModelString(step);
if (!modelStr) return null;
return {
kind: "model",
stepId: step.id,
executionKey,
modelStr,
provider: getTargetProvider(modelStr, step.providerId),
providerId: step.providerId || null,
connectionId: step.connectionId || null,
weight,
label,
} satisfies ResolvedComboTarget;
}
function getDirectComboTargets(combo) {
return (combo.models || [])
.map((entry, index) => normalizeRuntimeStep(entry, combo.name, index, null))
.filter((entry): entry is ResolvedComboTarget => entry?.kind === "model");
}
function getTopLevelRuntimeSteps(combo, allCombos, path = []) {
return (combo.models || [])
.map((entry, index) => normalizeRuntimeStep(entry, combo.name, index, allCombos, path))
.filter((entry): entry is ComboRuntimeStep => entry !== null);
}
function expandRuntimeStep(step, allCombos, visited = new Set(), depth = 0, path = []) {
if (step.kind === "model") return [step];
if (depth > MAX_COMBO_DEPTH) return [];
const combos = Array.isArray(allCombos) ? allCombos : allCombos?.combos || [];
const nestedCombo = combos.find((combo) => combo.name === step.comboName);
if (!nestedCombo || visited.has(step.comboName)) return [];
return resolveNestedComboTargets(nestedCombo, combos, new Set(visited), depth + 1, [
...path,
step.stepId,
]);
}
export function resolveNestedComboTargets(
combo,
allCombos,
visited = new Set(),
depth = 0,
path = []
) {
const directTargets = (combo.models || [])
.map((entry, index) => normalizeRuntimeStep(entry, combo.name, index, null, path))
.filter((entry): entry is ResolvedComboTarget => entry?.kind === "model");
if (depth > MAX_COMBO_DEPTH) return directTargets;
if (visited.has(combo.name)) return [];
visited.add(combo.name);
const runtimeSteps = getTopLevelRuntimeSteps(combo, allCombos, path);
const resolved: ResolvedComboTarget[] = [];
for (const step of runtimeSteps) {
if (step.kind === "combo-ref") {
resolved.push(...expandRuntimeStep(step, allCombos, new Set(visited), depth, path));
continue;
}
resolved.push(step);
}
return resolved;
}
/**
@@ -232,37 +382,32 @@ export function resolveNestedComboModels(combo, allCombos, visited = new Set(),
return resolved;
}
/**
* Select a model using weighted random distribution
* @param {Array} models - Array of { model, weight } entries
* @returns {string} Selected model string
*/
function selectWeightedModel(models) {
const entries = models.map((m) => normalizeModelEntry(m));
const totalWeight = entries.reduce((sum, m) => sum + m.weight, 0);
function selectWeightedTarget(targets: Array<{ weight: number }>) {
if (targets.length === 0) return null;
const totalWeight = targets.reduce((sum, target) => sum + (target.weight || 0), 0);
if (totalWeight <= 0) {
// All weights are 0 → uniform random
return entries[Math.floor(Math.random() * entries.length)].model;
return targets[Math.floor(Math.random() * targets.length)];
}
let random = Math.random() * totalWeight;
for (const entry of entries) {
random -= entry.weight;
if (random <= 0) return entry.model;
for (const target of targets) {
random -= target.weight || 0;
if (random <= 0) return target;
}
return entries[entries.length - 1].model; // safety fallback
return targets[targets.length - 1];
}
/**
* Order models for weighted fallback (selected first, then by descending weight)
*/
function orderModelsForWeightedFallback(models, selectedModel) {
const entries = models.map((m) => normalizeModelEntry(m));
const selected = entries.find((e) => e.model === selectedModel);
const rest = entries.filter((e) => e.model !== selectedModel).sort((a, b) => b.weight - a.weight); // highest weight first for fallback
return [selected, ...rest].filter(Boolean).map((e) => e.model);
function orderTargetsForWeightedFallback(
targets: Array<{ executionKey: string; weight: number }>,
selectedExecutionKey: string
) {
const selected = targets.find((target) => target.executionKey === selectedExecutionKey);
const rest = targets
.filter((target) => target.executionKey !== selectedExecutionKey)
.sort((a, b) => b.weight - a.weight);
return [selected, ...rest].filter(Boolean);
}
// shuffleArray and getNextModelFromDeck moved to src/shared/utils/shuffleDeck.ts
@@ -297,6 +442,22 @@ async function sortModelsByCost(models) {
}
}
async function sortTargetsByCost(targets: ResolvedComboTarget[]) {
const orderedModels = await sortModelsByCost(targets.map((target) => target.modelStr));
const byModel = new Map<string, ResolvedComboTarget[]>();
for (const target of targets) {
const queue = byModel.get(target.modelStr) || [];
queue.push(target);
byModel.set(target.modelStr, queue);
}
return orderedModels
.map((modelStr) => {
const queue = byModel.get(modelStr);
return queue?.shift() || null;
})
.filter((target): target is ResolvedComboTarget => target !== null);
}
/**
* Sort models by usage count (least-used first) for least-used strategy
* @param {Array<string>} models - Model strings
@@ -315,6 +476,25 @@ function sortModelsByUsage(models, comboName) {
return withUsage.map((e) => e.modelStr);
}
function sortTargetsByUsage(targets: ResolvedComboTarget[], comboName: string) {
const orderedModels = sortModelsByUsage(
targets.map((target) => target.modelStr),
comboName
);
const byModel = new Map<string, ResolvedComboTarget[]>();
for (const target of targets) {
const queue = byModel.get(target.modelStr) || [];
queue.push(target);
byModel.set(target.modelStr, queue);
}
return orderedModels
.map((modelStr) => {
const queue = byModel.get(modelStr);
return queue?.shift() || null;
})
.filter((target): target is ResolvedComboTarget => target !== null);
}
/**
* Sort models by context window size (largest first) for context-optimized strategy.
* Uses models.dev synced capabilities to get context limits.
@@ -333,6 +513,22 @@ function sortModelsByContextSize(models) {
return withContext.map((e) => e.modelStr);
}
function sortTargetsByContextSize(targets: ResolvedComboTarget[]) {
const orderedModels = sortModelsByContextSize(targets.map((target) => target.modelStr));
const byModel = new Map<string, ResolvedComboTarget[]>();
for (const target of targets) {
const queue = byModel.get(target.modelStr) || [];
queue.push(target);
byModel.set(target.modelStr, queue);
}
return orderedModels
.map((modelStr) => {
const queue = byModel.get(modelStr);
return queue?.shift() || null;
})
.filter((target): target is ResolvedComboTarget => target !== null);
}
function toTextContent(content) {
if (typeof content === "string") return content;
if (!Array.isArray(content)) return "";
@@ -437,7 +633,7 @@ function getBootstrapLatencyMs(modelId) {
return DEFAULT_MODEL_P95_MS[normalized] ?? 1500;
}
async function buildAutoCandidates(modelStrings, comboName) {
async function buildAutoCandidates(targets, comboName) {
const metrics = getComboMetrics(comboName);
const { getPricingForModel } = await import("../../src/lib/localDb");
let historicalLatencyStats = {};
@@ -453,9 +649,10 @@ async function buildAutoCandidates(modelStrings, comboName) {
}
const candidates = await Promise.all(
modelStrings.map(async (modelStr) => {
targets.map(async (target) => {
const modelStr = target.modelStr;
const parsed = parseModel(modelStr);
const provider = parsed.provider || parsed.providerAlias || "unknown";
const provider = target.provider || parsed.provider || parsed.providerAlias || "unknown";
const model = parsed.model || modelStr;
const historicalKey = `${provider}/${model}`;
const historicalModelMetric = historicalLatencyStats[historicalKey] || null;
@@ -503,11 +700,16 @@ async function buildAutoCandidates(modelStrings, comboName) {
? Math.max(10, historicalStdDev)
: Math.max(10, p95LatencyMs * 0.1);
const breakerStateRaw = getCircuitBreaker(`combo:${modelStr}`)?.getStatus?.()?.state;
const breakerStateRaw = getCircuitBreaker(
getComboBreakerKey(comboName, target.executionKey)
)?.getStatus?.()?.state;
const circuitBreakerState =
breakerStateRaw === "OPEN" || breakerStateRaw === "HALF_OPEN" ? breakerStateRaw : "CLOSED";
return {
stepId: target.stepId,
executionKey: target.executionKey,
modelStr,
provider,
model,
quotaRemaining: 100,
@@ -526,6 +728,64 @@ async function buildAutoCandidates(modelStrings, comboName) {
return candidates;
}
function dedupeTargetsByExecutionKey(targets: ResolvedComboTarget[]) {
const seen = new Set<string>();
return targets.filter((target) => {
if (seen.has(target.executionKey)) return false;
seen.add(target.executionKey);
return true;
});
}
export function resolveComboTargets(combo, allCombos) {
return allCombos ? resolveNestedComboTargets(combo, allCombos) : getDirectComboTargets(combo);
}
function resolveWeightedTargets(combo, allCombos) {
const topLevelSteps = allCombos
? getTopLevelRuntimeSteps(combo, allCombos)
: getDirectComboTargets(combo);
if (topLevelSteps.length === 0) {
return { orderedTargets: [], selectedStep: null };
}
const selectedStep = selectWeightedTarget(topLevelSteps);
if (!selectedStep) {
return { orderedTargets: [], selectedStep: null };
}
const orderedSteps = orderTargetsForWeightedFallback(topLevelSteps, selectedStep.executionKey);
const expandedTargets = orderedSteps.flatMap((step) => {
if (!allCombos) {
return step.kind === "model" ? [step] : [];
}
return expandRuntimeStep(step, allCombos, new Set([combo.name]));
});
return {
orderedTargets: dedupeTargetsByExecutionKey(expandedTargets),
selectedStep,
};
}
function scoreAutoTargets(targets, candidates, taskType, weights) {
const candidateByExecutionKey = new Map(
candidates.map((candidate) => [candidate.executionKey, candidate])
);
return targets
.map((target) => {
const candidate = candidateByExecutionKey.get(target.executionKey);
if (!candidate) return null;
const factors = calculateFactors(candidate, candidates, taskType, getTaskFitness);
return {
target,
score: calculateScore(factors, weights),
};
})
.filter(Boolean)
.sort((a, b) => b.score - a.score);
}
/**
* Handle combo chat with fallback.
* @param {Object} options
@@ -548,7 +808,6 @@ export async function handleComboChat({
relayOptions,
}) {
const strategy = combo.strategy || "priority";
const models = combo.models || [];
const relayConfig =
strategy === "context-relay" ? resolveContextRelayConfig(relayOptions?.config || null) : null;
@@ -566,8 +825,8 @@ export async function handleComboChat({
}
// Wrap handleSingleModel to inject context caching tag on response (#401)
const handleSingleModelWrapped = combo.context_cache_protection
? async (b, modelStr) => {
const res = await handleSingleModel(b, modelStr);
? async (b, modelStr, target) => {
const res = await handleSingleModel(b, modelStr, target);
if (!res.ok) return res;
// Non-streaming: inject tag into JSON response
@@ -749,47 +1008,28 @@ export async function handleComboChat({
const maxRetries = config.maxRetries ?? 1;
const retryDelayMs = config.retryDelayMs ?? 2000;
let orderedModels;
let orderedTargets =
strategy === "weighted"
? resolveWeightedTargets(combo, allCombos)?.orderedTargets || []
: resolveComboTargets(combo, allCombos);
// Resolve nested combos if allCombos provided
if (allCombos) {
const flatModels = resolveNestedComboModels(combo, allCombos);
if (strategy === "weighted") {
// For weighted + nested, select from original models then fallback sequentially
const selected = selectWeightedModel(models);
orderedModels = orderModelsForWeightedFallback(models, selected);
// If entries were nested, they are already resolved to flat
orderedModels = orderedModels.flatMap((m) => {
const combos = Array.isArray(allCombos) ? allCombos : allCombos?.combos || [];
const nested = combos.find((c) => c.name === m);
if (nested) return resolveNestedComboModels(nested, allCombos);
return [m];
});
log.info(
"COMBO",
`Weighted selection with nested resolution: ${orderedModels.length} total models`
);
} else {
orderedModels = flatModels;
log.info("COMBO", `${strategy} with nested resolution: ${orderedModels.length} total models`);
}
} else if (strategy === "weighted") {
const selected = selectWeightedModel(models);
orderedModels = orderModelsForWeightedFallback(models, selected);
log.info("COMBO", `Weighted selection: ${selected} (from ${models.length} models)`);
} else {
orderedModels = models.map((m) => normalizeModelEntry(m).model);
if (strategy === "weighted") {
log.info(
"COMBO",
`Weighted selection${allCombos ? " with nested resolution" : ""}: ${orderedTargets.length} total targets`
);
} else if (allCombos) {
log.info("COMBO", `${strategy} with nested resolution: ${orderedTargets.length} total targets`);
}
// Apply strategy-specific ordering
if (strategy === "auto") {
const requestHasTools = Array.isArray(body?.tools) && body.tools.length > 0;
let eligibleModels = [...orderedModels];
let eligibleTargets = [...orderedTargets];
if (requestHasTools) {
const filtered = eligibleModels.filter((m) => supportsToolCalling(m));
const filtered = eligibleTargets.filter((target) => supportsToolCalling(target.modelStr));
if (filtered.length > 0) {
eligibleModels = filtered;
eligibleTargets = filtered;
} else {
log.warn(
"COMBO",
@@ -816,14 +1056,7 @@ export async function handleComboChat({
const candidatePool = Array.isArray(autoConfigSource.candidatePool)
? autoConfigSource.candidatePool
: [
...new Set(
eligibleModels.map((m) => {
const parsed = parseModel(m);
return parsed.provider || parsed.providerAlias || "unknown";
})
),
];
: [...new Set(eligibleTargets.map((target) => target.provider))];
const weights =
autoConfigSource.weights && typeof autoConfigSource.weights === "object"
@@ -838,7 +1071,6 @@ export async function handleComboChat({
const modePack =
typeof autoConfigSource.modePack === "string" ? autoConfigSource.modePack : undefined;
// Retrieve last known good provider (LKGP) for this combo/model (#919)
let lastKnownGoodProvider: string | undefined;
try {
const { getLKGP } = await import("../../src/lib/localDb");
@@ -848,7 +1080,7 @@ export async function handleComboChat({
log.warn("COMBO", "Failed to retrieve Last Known Good Provider. This is non-fatal.", { err });
}
const candidates = await buildAutoCandidates(eligibleModels, combo.name);
const candidates = await buildAutoCandidates(eligibleTargets, combo.name);
if (candidates.length > 0) {
let selectedProvider = null;
let selectedModel = null;
@@ -892,51 +1124,57 @@ export async function handleComboChat({
selectionReason = `score=${selection.score.toFixed(3)}${selection.isExploration ? " (exploration)" : ""}`;
}
const modelLookup = new Map();
for (const modelStr of eligibleModels) {
const parsed = parseModel(modelStr);
const provider = parsed.provider || parsed.providerAlias || "unknown";
const modelId = parsed.model || modelStr;
modelLookup.set(`${provider}/${modelId}`, modelStr);
}
const scoredTargets = scoreAutoTargets(eligibleTargets, candidates, taskType, weights);
const rankedTargets = scoredTargets.map((entry) => entry.target);
const selectedTarget =
scoredTargets.find((entry) => {
const parsed = parseModel(entry.target.modelStr);
const modelId = parsed.model || entry.target.modelStr;
return entry.target.provider === selectedProvider && modelId === selectedModel;
})?.target ||
rankedTargets[0] ||
eligibleTargets[0];
const ranked = scorePool(candidates, taskType, weights)
.map((r) => modelLookup.get(`${r.provider}/${r.model}`) || `${r.provider}/${r.model}`)
.filter(Boolean);
const selectedModelStr =
modelLookup.get(`${selectedProvider}/${selectedModel}`) ||
`${selectedProvider}/${selectedModel}`;
orderedModels = [...new Set([selectedModelStr, ...ranked, ...eligibleModels])];
orderedTargets = dedupeTargetsByExecutionKey(
[selectedTarget, ...rankedTargets, ...eligibleTargets].filter(Boolean)
);
log.info(
"COMBO",
`Auto selection: ${selectedModelStr} | intent=${intent} task=${taskType} | strategy=${routingStrategy} | ${selectionReason}`
`Auto selection: ${selectedTarget?.modelStr || `${selectedProvider}/${selectedModel}`} | intent=${intent} task=${taskType} | strategy=${routingStrategy} | ${selectionReason}`
);
} else {
log.warn("COMBO", "Auto strategy has no candidates, keeping default ordering");
}
} else if (strategy === "strict-random") {
const selectedId = await getNextFromDeck(`combo:${combo.name}`, orderedModels);
// Put selected model first so the fallback loop tries it first
const rest = orderedModels.filter((m) => m !== selectedId);
orderedModels = [selectedId, ...rest];
const selectedExecutionKey = await getNextFromDeck(
`combo:${combo.name}`,
orderedTargets.map((target) => target.executionKey)
);
const selectedTarget =
orderedTargets.find((target) => target.executionKey === selectedExecutionKey) || null;
const rest = orderedTargets.filter((target) => target.executionKey !== selectedExecutionKey);
orderedTargets = [selectedTarget, ...rest].filter(Boolean);
log.info(
"COMBO",
`Strict-random deck: ${selectedId} selected (${orderedModels.length} models)`
`Strict-random deck: ${selectedExecutionKey} selected (${orderedTargets.length} targets)`
);
} else if (strategy === "random") {
orderedModels = fisherYatesShuffle([...orderedModels]);
log.info("COMBO", `Random shuffle: ${orderedModels.length} models`);
orderedTargets = fisherYatesShuffle([...orderedTargets]);
log.info("COMBO", `Random shuffle: ${orderedTargets.length} targets`);
} else if (strategy === "least-used") {
orderedModels = sortModelsByUsage(orderedModels, combo.name);
log.info("COMBO", `Least-used ordering: ${orderedModels[0]} has fewest requests`);
orderedTargets = sortTargetsByUsage(orderedTargets, combo.name);
log.info("COMBO", `Least-used ordering: ${orderedTargets[0]?.modelStr} has fewest requests`);
} else if (strategy === "cost-optimized") {
orderedModels = await sortModelsByCost(orderedModels);
log.info("COMBO", `Cost-optimized ordering: cheapest first (${orderedModels[0]})`);
orderedTargets = await sortTargetsByCost(orderedTargets);
log.info("COMBO", `Cost-optimized ordering: cheapest first (${orderedTargets[0]?.modelStr})`);
} else if (strategy === "context-optimized") {
orderedModels = sortModelsByContextSize(orderedModels);
log.info("COMBO", `Context-optimized ordering: largest first (${orderedModels[0]})`);
orderedTargets = sortTargetsByContextSize(orderedTargets);
log.info("COMBO", `Context-optimized ordering: largest first (${orderedTargets[0]?.modelStr})`);
}
if (orderedTargets.length === 0) {
return unavailableResponse(503, "Combo has no executable targets");
}
let lastError = null;
@@ -944,13 +1182,14 @@ export async function handleComboChat({
let lastStatus = null;
const startTime = Date.now();
let fallbackCount = 0;
let recordedAttempts = 0;
for (let i = 0; i < orderedModels.length; i++) {
const modelStr = orderedModels[i];
const parsed = parseModel(modelStr);
const provider = parsed.provider || parsed.providerAlias || "unknown";
for (let i = 0; i < orderedTargets.length; i++) {
const target = orderedTargets[i];
const modelStr = target.modelStr;
const provider = target.provider;
const profile = getProviderProfile(provider);
const breakerKey = `combo:${modelStr}`;
const breakerKey = getComboBreakerKey(combo.name, target.executionKey);
const breaker = getCircuitBreaker(breakerKey, {
failureThreshold: profile.circuitBreakerThreshold,
resetTimeout: profile.circuitBreakerReset,
@@ -965,7 +1204,7 @@ export async function handleComboChat({
// Pre-check: skip models where all accounts are in cooldown
if (isModelAvailable) {
const available = await isModelAvailable(modelStr);
const available = await isModelAvailable(modelStr, target);
if (!available) {
log.info("COMBO", `Skipping ${modelStr} (all accounts in cooldown)`);
if (i > 0) fallbackCount++;
@@ -985,10 +1224,10 @@ export async function handleComboChat({
log.info(
"COMBO",
`Trying model ${i + 1}/${orderedModels.length}: ${modelStr}${retry > 0 ? ` (retry ${retry})` : ""}`
`Trying model ${i + 1}/${orderedTargets.length}: ${modelStr}${retry > 0 ? ` (retry ${retry})` : ""}`
);
const result = await handleSingleModelWrapped(body, modelStr);
const result = await handleSingleModelWrapped(body, modelStr, target);
// Success — validate response quality before returning
if (result.ok) {
@@ -1004,7 +1243,9 @@ export async function handleComboChat({
latencyMs: Date.now() - startTime,
fallbackCount,
strategy,
target: toRecordedTarget(target),
});
recordedAttempts++;
if (i > 0) fallbackCount++;
break; // move to next model
}
@@ -1019,7 +1260,9 @@ export async function handleComboChat({
latencyMs,
fallbackCount,
strategy,
target: toRecordedTarget(target),
});
recordedAttempts++;
// Context-relay intentionally splits responsibilities:
// combo.ts decides whether a successful turn should generate a handoff,
@@ -1063,7 +1306,12 @@ export async function handleComboChat({
// Record last known good provider (LKGP) for this combo/model (#919)
if (provider) {
import("../../src/lib/localDb")
.then(({ setLKGP }) => setLKGP(combo.name, combo.id || combo.name, provider))
.then(({ setLKGP }) =>
Promise.all([
setLKGP(combo.name, target.executionKey, provider),
setLKGP(combo.name, combo.id || combo.name, provider),
])
)
.catch((err) =>
log.warn("COMBO", "Failed to record Last Known Good Provider. This is non-fatal.", {
err,
@@ -1129,6 +1377,14 @@ export async function handleComboChat({
if (!shouldFallback && !comboBadRequestFallback) {
log.warn("COMBO", `Model ${modelStr} failed (no fallback)`, { status: result.status });
recordComboRequest(combo.name, modelStr, {
success: false,
latencyMs: Date.now() - startTime,
fallbackCount,
strategy,
target: toRecordedTarget(target),
});
recordedAttempts++;
return result;
}
@@ -1146,6 +1402,14 @@ export async function handleComboChat({
}
// Done retrying this model
recordComboRequest(combo.name, modelStr, {
success: false,
latencyMs: Date.now() - startTime,
fallbackCount,
strategy,
target: toRecordedTarget(target),
});
recordedAttempts++;
lastError = errorText || String(result.status);
if (!lastStatus) lastStatus = result.status;
if (i > 0) fallbackCount++;
@@ -1161,13 +1425,15 @@ export async function handleComboChat({
}
// Early exit: check if all models have breaker OPEN
const allBreakersOpen = orderedModels.every((m) => {
return !getCircuitBreaker(`combo:${m}`).canExecute();
const allBreakersOpen = orderedTargets.every((target) => {
return !getCircuitBreaker(getComboBreakerKey(combo.name, target.executionKey)).canExecute();
});
// All models failed
const latencyMs = Date.now() - startTime;
recordComboRequest(combo.name, null, { success: false, latencyMs, fallbackCount, strategy });
if (recordedAttempts === 0) {
recordComboRequest(combo.name, null, { success: false, latencyMs, fallbackCount, strategy });
}
if (allBreakersOpen) {
log.warn("COMBO", "All models have circuit breaker OPEN — aborting");
@@ -1226,7 +1492,6 @@ async function handleRoundRobinCombo({
settings,
allCombos,
}) {
const models = combo.models || [];
const config = settings
? resolveComboConfig(combo, settings)
: { ...getDefaultComboConfig(), ...(combo.config || {}) };
@@ -1235,17 +1500,10 @@ async function handleRoundRobinCombo({
const maxRetries = config.maxRetries ?? 1;
const retryDelayMs = config.retryDelayMs ?? 2000;
// Resolve models (support nested combos)
let orderedModels;
if (allCombos) {
orderedModels = resolveNestedComboModels(combo, allCombos);
} else {
orderedModels = models.map((m) => normalizeModelEntry(m).model);
}
const modelCount = orderedModels.length;
const orderedTargets = resolveComboTargets(combo, allCombos);
const modelCount = orderedTargets.length;
if (modelCount === 0) {
return unavailableResponse(503, "Round-robin combo has no models");
return unavailableResponse(503, "Round-robin combo has no executable targets");
}
// Get and increment atomic counter
@@ -1258,15 +1516,17 @@ async function handleRoundRobinCombo({
let lastStatus = null;
let earliestRetryAfter = null;
let fallbackCount = 0;
let recordedAttempts = 0;
// Try each model starting from the round-robin target
for (let offset = 0; offset < modelCount; offset++) {
const modelIndex = (startIndex + offset) % modelCount;
const modelStr = orderedModels[modelIndex];
const parsed = parseModel(modelStr);
const provider = parsed.provider || parsed.providerAlias || "unknown";
const target = orderedTargets[modelIndex];
const modelStr = target.modelStr;
const provider = target.provider;
const profile = getProviderProfile(provider);
const breakerKey = `combo:${modelStr}`;
const breakerKey = getComboBreakerKey(combo.name, target.executionKey);
const semaphoreKey = `combo:${combo.name}:${target.executionKey}`;
const breaker = getCircuitBreaker(breakerKey, {
failureThreshold: profile.circuitBreakerThreshold,
resetTimeout: profile.circuitBreakerReset,
@@ -1281,7 +1541,7 @@ async function handleRoundRobinCombo({
// Pre-check availability
if (isModelAvailable) {
const available = await isModelAvailable(modelStr);
const available = await isModelAvailable(modelStr, target);
if (!available) {
log.info("COMBO-RR", `Skipping ${modelStr} (all accounts in cooldown)`);
if (offset > 0) fallbackCount++;
@@ -1292,7 +1552,7 @@ async function handleRoundRobinCombo({
// Acquire semaphore slot (may wait in queue)
let release;
try {
release = await semaphore.acquire(modelStr, {
release = await semaphore.acquire(semaphoreKey, {
maxConcurrency: concurrency,
timeoutMs: queueTimeout,
});
@@ -1321,7 +1581,7 @@ async function handleRoundRobinCombo({
`[RR #${counter}] → ${modelStr}${offset > 0 ? ` (fallback +${offset})` : ""}${retry > 0 ? ` (retry ${retry})` : ""}`
);
const result = await handleSingleModel(body, modelStr);
const result = await handleSingleModel(body, modelStr, target);
// Success — validate response quality before returning
if (result.ok) {
@@ -1337,7 +1597,9 @@ async function handleRoundRobinCombo({
latencyMs: Date.now() - startTime,
fallbackCount,
strategy: "round-robin",
target: toRecordedTarget(target),
});
recordedAttempts++;
if (offset > 0) fallbackCount++;
break; // move to next model
}
@@ -1352,7 +1614,27 @@ async function handleRoundRobinCombo({
latencyMs,
fallbackCount,
strategy: "round-robin",
target: toRecordedTarget(target),
});
recordedAttempts++;
if (provider) {
import("../../src/lib/localDb")
.then(({ setLKGP }) =>
Promise.all([
setLKGP(combo.name, target.executionKey, provider),
setLKGP(combo.name, combo.id || combo.name, provider),
])
)
.catch((err) =>
log.warn(
"COMBO-RR",
"Failed to record Last Known Good Provider. This is non-fatal.",
{
err,
}
)
);
}
return result;
}
@@ -1404,7 +1686,7 @@ async function handleRoundRobinCombo({
// Transient errors → mark in semaphore AND record circuit breaker failure
if (TRANSIENT_FOR_BREAKER.includes(result.status) && cooldownMs > 0) {
semaphore.markRateLimited(modelStr, cooldownMs);
semaphore.markRateLimited(semaphoreKey, cooldownMs);
breaker._onFailure();
log.warn(
"COMBO-RR",
@@ -1414,6 +1696,14 @@ async function handleRoundRobinCombo({
if (!shouldFallback && !comboBadRequestFallback) {
log.warn("COMBO-RR", `${modelStr} failed (no fallback)`, { status: result.status });
recordComboRequest(combo.name, modelStr, {
success: false,
latencyMs: Date.now() - startTime,
fallbackCount,
strategy: "round-robin",
target: toRecordedTarget(target),
});
recordedAttempts++;
return result;
}
@@ -1431,6 +1721,14 @@ async function handleRoundRobinCombo({
}
// Done with this model
recordComboRequest(combo.name, modelStr, {
success: false,
latencyMs: Date.now() - startTime,
fallbackCount,
strategy: "round-robin",
target: toRecordedTarget(target),
});
recordedAttempts++;
lastError = errorText || String(result.status);
if (!lastStatus) lastStatus = result.status;
if (offset > 0) fallbackCount++;
@@ -1451,16 +1749,18 @@ async function handleRoundRobinCombo({
// All models exhausted
const latencyMs = Date.now() - startTime;
recordComboRequest(combo.name, null, {
success: false,
latencyMs,
fallbackCount,
strategy: "round-robin",
});
if (recordedAttempts === 0) {
recordComboRequest(combo.name, null, {
success: false,
latencyMs,
fallbackCount,
strategy: "round-robin",
});
}
// Early exit: check if all models have breaker OPEN
const allBreakersOpen = orderedModels.every((m) => {
return !getCircuitBreaker(`combo:${m}`).canExecute();
const allBreakersOpen = orderedTargets.every((target) => {
return !getCircuitBreaker(getComboBreakerKey(combo.name, target.executionKey)).canExecute();
});
if (allBreakersOpen) {

244
open-sse/services/comboMetrics.ts
View File

@@ -1,7 +1,7 @@
/**
* In-memory combo metrics tracker
* Tracks per-combo and per-model request counts, latency, success/failure rates
* Provides API for reading metrics from the dashboard
* Tracks per-combo, per-model, and per-target request counts, latency, success/failure rates.
* Provides API for reading metrics from the dashboard.
*/
interface ModelMetrics {
@@ -13,6 +13,16 @@ interface ModelMetrics {
lastUsedAt: string | null;
}
interface ComboTargetMetrics extends ModelMetrics {
executionKey: string;
stepId: string | null;
model: string;
provider: string | null;
providerId: string | null;
connectionId: string | null;
label: string | null;
}
interface ComboMetricsEntry {
totalRequests: number;
totalSuccesses: number;
@@ -23,33 +33,140 @@ interface ComboMetricsEntry {
lastUsedAt: string | null;
intentCounts: Record<string, number>;
byModel: Record<string, ModelMetrics>;
byTarget: Record<string, ComboTargetMetrics>;
}
interface ModelMetricsView extends ModelMetrics {
avgLatencyMs: number;
successRate: number;
}
interface ComboTargetMetricsView extends ComboTargetMetrics {
avgLatencyMs: number;
successRate: number;
}
interface ComboMetricsView extends ComboMetricsEntry {
avgLatencyMs: number;
successRate: number;
fallbackRate: number;
byModel: Record<
string,
ModelMetrics & {
avgLatencyMs: number;
successRate: number;
}
>;
byModel: Record<string, ModelMetricsView>;
byTarget: Record<string, ComboTargetMetricsView>;
}
export interface ComboRequestTargetMeta {
executionKey?: string | null;
stepId?: string | null;
provider?: string | null;
providerId?: string | null;
connectionId?: string | null;
label?: string | null;
}
function toNonEmptyString(value: unknown): string | null {
return typeof value === "string" && value.trim().length > 0 ? value.trim() : null;
}
function inferProvider(modelStr: string | null): string | null {
const model = toNonEmptyString(modelStr);
if (!model) return null;
const [provider] = model.split("/");
return toNonEmptyString(provider);
}
function createModelMetrics(): ModelMetrics {
return {
requests: 0,
successes: 0,
failures: 0,
totalLatencyMs: 0,
lastStatus: null,
lastUsedAt: null,
};
}
function createComboEntry(strategy: string): ComboMetricsEntry {
return {
totalRequests: 0,
totalSuccesses: 0,
totalFailures: 0,
totalFallbacks: 0,
totalLatencyMs: 0,
strategy,
lastUsedAt: null,
intentCounts: {},
byModel: {},
byTarget: {},
};
}
function applyMetricOutcome(
metric: ModelMetrics,
success: boolean,
latencyMs: number,
usedAt: string
): void {
metric.requests++;
metric.totalLatencyMs += latencyMs;
metric.lastUsedAt = usedAt;
if (success) {
metric.successes++;
metric.lastStatus = "ok";
return;
}
metric.failures++;
metric.lastStatus = "error";
}
function buildTargetMetric(
modelStr: string,
target: ComboRequestTargetMeta
): ComboTargetMetrics | null {
const executionKey = toNonEmptyString(target.executionKey) || toNonEmptyString(modelStr);
const model = toNonEmptyString(modelStr);
if (!executionKey || !model) return null;
return {
executionKey,
stepId: toNonEmptyString(target.stepId),
model,
provider: toNonEmptyString(target.provider) || inferProvider(model),
providerId: toNonEmptyString(target.providerId),
connectionId:
target.connectionId === null ? null : (toNonEmptyString(target.connectionId) ?? null),
label: target.label === null ? null : (toNonEmptyString(target.label) ?? null),
...createModelMetrics(),
};
}
function toMetricView<T extends ModelMetrics>(
metric: T
): T & {
avgLatencyMs: number;
successRate: number;
} {
return {
...metric,
avgLatencyMs: metric.requests > 0 ? Math.round(metric.totalLatencyMs / metric.requests) : 0,
successRate: metric.requests > 0 ? Math.round((metric.successes / metric.requests) * 100) : 0,
};
}
// In-memory store
const metrics = new Map<string, ComboMetricsEntry>();
/**
* Record a combo request result
* Record a combo request result.
* @param {string} comboName
* @param {string} modelStr - The model that handled the request (or null if all failed)
* @param {Object} options
* @param {boolean} options.success
* @param {number} options.latencyMs
* @param {number} options.fallbackCount - How many fallbacks occurred
* @param {string} [options.strategy] - "priority" or "weighted"
* @param {string} [options.strategy] - Routing strategy name
* @param {Object} [options.target] - Step/execution metadata for structured combos
*/
export function recordComboRequest(
comboName: string,
@@ -59,28 +176,27 @@ export function recordComboRequest(
latencyMs,
fallbackCount = 0,
strategy = "priority",
}: { success: boolean; latencyMs: number; fallbackCount?: number; strategy?: string }
target,
}: {
success: boolean;
latencyMs: number;
fallbackCount?: number;
strategy?: string;
target?: ComboRequestTargetMeta | null;
}
): void {
if (!metrics.has(comboName)) {
metrics.set(comboName, {
totalRequests: 0,
totalSuccesses: 0,
totalFailures: 0,
totalFallbacks: 0,
totalLatencyMs: 0,
strategy,
lastUsedAt: null,
intentCounts: {},
byModel: {},
});
metrics.set(comboName, createComboEntry(strategy));
}
const combo = metrics.get(comboName);
if (!combo) return;
const usedAt = new Date().toISOString();
combo.totalRequests++;
combo.totalLatencyMs += latencyMs;
combo.totalFallbacks += fallbackCount;
combo.lastUsedAt = new Date().toISOString();
combo.lastUsedAt = usedAt;
combo.strategy = strategy;
if (success) {
@@ -89,35 +205,36 @@ export function recordComboRequest(
combo.totalFailures++;
}
// Per-model tracking
if (modelStr) {
if (!combo.byModel[modelStr]) {
combo.byModel[modelStr] = {
requests: 0,
successes: 0,
failures: 0,
totalLatencyMs: 0,
lastStatus: null,
lastUsedAt: null,
};
}
const modelMetric = combo.byModel[modelStr];
modelMetric.requests++;
modelMetric.totalLatencyMs += latencyMs;
modelMetric.lastUsedAt = new Date().toISOString();
if (!modelStr) return;
if (success) {
modelMetric.successes++;
modelMetric.lastStatus = "ok";
} else {
modelMetric.failures++;
modelMetric.lastStatus = "error";
}
if (!combo.byModel[modelStr]) {
combo.byModel[modelStr] = createModelMetrics();
}
applyMetricOutcome(combo.byModel[modelStr], success, latencyMs, usedAt);
const targetMetric = buildTargetMetric(modelStr, target || {});
if (!targetMetric) return;
if (!combo.byTarget[targetMetric.executionKey]) {
combo.byTarget[targetMetric.executionKey] = targetMetric;
}
const existingTargetMetric = combo.byTarget[targetMetric.executionKey];
existingTargetMetric.stepId = targetMetric.stepId || existingTargetMetric.stepId;
existingTargetMetric.provider = targetMetric.provider || existingTargetMetric.provider;
existingTargetMetric.providerId = targetMetric.providerId || existingTargetMetric.providerId;
existingTargetMetric.connectionId =
target?.connectionId === null
? null
: (targetMetric.connectionId ?? existingTargetMetric.connectionId);
existingTargetMetric.label =
target?.label === null ? null : (targetMetric.label ?? existingTargetMetric.label);
applyMetricOutcome(existingTargetMetric, success, latencyMs, usedAt);
}
/**
* Get metrics for a specific combo
* Get metrics for a specific combo.
* @param {string} comboName
* @returns {Object|null}
*/
@@ -135,20 +252,19 @@ export function getComboMetrics(comboName: string): ComboMetricsView | null {
combo.totalRequests > 0 ? Math.round((combo.totalFallbacks / combo.totalRequests) * 100) : 0,
intentCounts: { ...combo.intentCounts },
byModel: Object.fromEntries(
Object.entries(combo.byModel).map(([model, m]) => [
model,
{
...m,
avgLatencyMs: m.requests > 0 ? Math.round(m.totalLatencyMs / m.requests) : 0,
successRate: m.requests > 0 ? Math.round((m.successes / m.requests) * 100) : 0,
},
Object.entries(combo.byModel).map(([model, metric]) => [model, toMetricView(metric)])
),
byTarget: Object.fromEntries(
Object.entries(combo.byTarget).map(([executionKey, metric]) => [
executionKey,
toMetricView(metric),
])
),
};
}
/**
* Get metrics for all combos
* Get metrics for all combos.
* @returns {Object} Map of comboName → metrics
*/
export function getAllComboMetrics(): Record<string, ComboMetricsView | null> {
@@ -164,17 +280,7 @@ export function getAllComboMetrics(): Record<string, ComboMetricsView | null> {
*/
export function recordComboIntent(comboName: string, intent: string): void {
if (!metrics.has(comboName)) {
metrics.set(comboName, {
totalRequests: 0,
totalSuccesses: 0,
totalFailures: 0,
totalFallbacks: 0,
totalLatencyMs: 0,
strategy: "priority",
lastUsedAt: null,
intentCounts: {},
byModel: {},
});
metrics.set(comboName, createComboEntry("priority"));
}
const combo = metrics.get(comboName);
@@ -184,14 +290,14 @@ export function recordComboIntent(comboName: string, intent: string): void {
}
/**
* Reset metrics for a specific combo
* Reset metrics for a specific combo.
*/
export function resetComboMetrics(comboName: string): void {
metrics.delete(comboName);
}
/**
* Reset all combo metrics
* Reset all combo metrics.
*/
export function resetAllComboMetrics(): void {
metrics.clear();

2
open-sse/services/contextManager.ts
View File

@@ -6,7 +6,7 @@
*/
import { REGISTRY } from "../config/providerRegistry.ts";
import { getModelContextLimit } from "../../src/lib/modelsDevSync";
import { getModelContextLimit } from "../../src/lib/modelCapabilities";
// Default token limits per provider (fallbacks when not in registry)
const DEFAULT_LIMITS: Record<string, number> = {

19
open-sse/services/model.ts
View File

@@ -71,6 +71,25 @@ function resolveProviderModelAlias(providerOrAlias, modelId) {
return aliases?.[modelId] || modelId;
}
/**
* Resolve a provider/model pair into canonical provider ID + provider-scoped model ID.
* Keeps provider-specific legacy aliases out of downstream capability and budget lookups.
*/
export function resolveCanonicalProviderModel(providerOrAlias, modelId) {
if (!modelId || typeof modelId !== "string") {
return {
provider: resolveProviderAlias(providerOrAlias),
model: modelId || null,
};
}
const provider = resolveProviderAlias(providerOrAlias);
return {
provider,
model: resolveProviderModelAlias(provider, modelId),
};
}
/**
* Parse model string: "alias/model" or "provider/model" or just alias
* Supports [1m] suffix for extended 1M context window (e.g. "claude-sonnet-4-6[1m]")

97
open-sse/services/modelCapabilities.ts
View File

@@ -1,92 +1,5 @@
import { PROVIDER_ID_TO_ALIAS, PROVIDER_MODELS } from "../config/providerModels.ts";
import { parseModel } from "./model.ts";
// Conservative denylist fallback used when registry metadata is absent.
// Keep small and explicit to avoid false negatives.
const TOOL_CALLING_UNSUPPORTED_PATTERNS = ["gpt-oss-120b", "deepseek-reasoner"];
function getRegistryToolCallingFlag(providerIdOrAlias: string, modelId: string): boolean | null {
const providerAlias = PROVIDER_ID_TO_ALIAS[providerIdOrAlias] || providerIdOrAlias;
const models = PROVIDER_MODELS[providerAlias];
if (!Array.isArray(models)) return null;
const found = models.find((m) => m?.id === modelId);
if (!found) return null;
return typeof found.toolCalling === "boolean" ? found.toolCalling : null;
}
/**
* Returns whether a model should be considered safe for structured function/tool calling.
*
* Decision order:
* 1) Provider registry metadata (toolCalling flag) when available.
* 2) Conservative denylist fallback for known problematic model families.
* 3) Default true.
*/
export function supportsToolCalling(modelStr: string): boolean {
const parsed = parseModel(modelStr);
const provider = parsed.provider || parsed.providerAlias || "";
const model = parsed.model || modelStr;
if (provider) {
const fromRegistry = getRegistryToolCallingFlag(provider, model);
if (fromRegistry !== null) return fromRegistry;
}
const normalized = String(modelStr || "").toLowerCase();
if (!normalized) return false;
const blocked = TOOL_CALLING_UNSUPPORTED_PATTERNS.some((pattern) => {
if (normalized === pattern) return true;
if (normalized.endsWith(`/${pattern}`)) return true;
return normalized.includes(pattern);
});
return !blocked;
}
// Models that do NOT support reasoning/thinking parameters.
// AG (Antigravity) claude-sonnet-4-6 routes through a Google internal API
// that returns 400 if thinking params are included.
const REASONING_UNSUPPORTED_PATTERNS = [
"antigravity/claude-sonnet-4-6",
"antigravity/claude-sonnet-4-5",
"antigravity/claude-sonnet-4",
];
function getRegistryReasoningFlag(providerIdOrAlias: string, modelId: string): boolean | null {
const providerAlias = PROVIDER_ID_TO_ALIAS[providerIdOrAlias] || providerIdOrAlias;
const models = PROVIDER_MODELS[providerAlias];
if (!Array.isArray(models)) return null;
const found = models.find((m) => m?.id === modelId);
if (!found) return null;
return typeof found.supportsReasoning === "boolean" ? found.supportsReasoning : null;
}
/**
* Returns whether a model supports reasoning/thinking parameters.
*
* Decision order:
* 1) Provider registry metadata (supportsReasoning flag) when available.
* 2) Explicit denylist for known unsupported models (e.g. AG Claude Sonnet).
* 3) Default true (pass through — safe, provider will ignore if unsupported).
*/
export function supportsReasoning(modelStr: string): boolean {
const parsed = parseModel(modelStr);
const provider = parsed.provider || parsed.providerAlias || "";
const model = parsed.model || modelStr;
if (provider) {
const fromRegistry = getRegistryReasoningFlag(provider, model);
if (fromRegistry !== null) return fromRegistry;
}
const normalized = String(modelStr || "").toLowerCase();
if (!normalized) return true;
const blocked = REASONING_UNSUPPORTED_PATTERNS.some(
(pattern) =>
normalized === pattern || normalized.endsWith(`/${pattern}`) || normalized.includes(pattern)
);
return !blocked;
}
export {
getResolvedModelCapabilities,
supportsReasoning,
supportsToolCalling,
} from "../../src/lib/modelCapabilities.ts";

2
open-sse/services/modelFamilyFallback.ts
View File

@@ -11,7 +11,7 @@
* (commit 6cea566, Mar 8 2026).
*/
import { getModelContextLimit } from "../../src/lib/modelsDevSync";
import { getModelContextLimit } from "../../src/lib/modelCapabilities";
import { parseModel } from "./model.ts";
import { CONTEXT_OVERFLOW_REGEX } from "./errorClassifier.ts";

259
open-sse/services/quotaMonitor.ts
View File

@@ -9,6 +9,7 @@
*/
import { registerQuotaFetcher, type QuotaFetcher } from "./quotaPreflight.ts";
import { getSessionInfo } from "./sessionManager.ts";
export { registerQuotaFetcher };
export type { QuotaFetcher };
@@ -24,6 +25,55 @@ interface MonitorState {
stopped: boolean;
provider: string;
accountId: string;
connectionSnapshot: Record<string, unknown> | null;
sessionBound: boolean;
status: "starting" | "idle" | "healthy" | "warning" | "exhausted" | "error";
startedAt: number;
lastPolledAt: number | null;
lastSuccessAt: number | null;
lastErrorAt: number | null;
lastError: string | null;
lastQuotaPercent: number | null;
lastQuotaUsed: number | null;
lastQuotaTotal: number | null;
lastResetAt: string | null;
lastAlertAt: number | null;
nextPollDelayMs: number | null;
nextPollAt: number | null;
totalPolls: number;
totalAlerts: number;
consecutiveFailures: number;
}
export interface QuotaMonitorSnapshot {
sessionId: string;
provider: string;
accountId: string;
status: "starting" | "idle" | "healthy" | "warning" | "exhausted" | "error";
startedAt: string;
lastPolledAt: string | null;
lastSuccessAt: string | null;
lastErrorAt: string | null;
lastError: string | null;
lastQuotaPercent: number | null;
lastQuotaUsed: number | null;
lastQuotaTotal: number | null;
lastResetAt: string | null;
lastAlertAt: string | null;
nextPollDelayMs: number | null;
nextPollAt: string | null;
totalPolls: number;
totalAlerts: number;
consecutiveFailures: number;
}
export interface QuotaMonitorSummary {
active: number;
alerting: number;
exhausted: number;
errors: number;
statusCounts: Record<QuotaMonitorSnapshot["status"], number>;
byProvider: Record<string, number>;
}
const activeMonitors = new Map<string, MonitorState>();
@@ -45,47 +95,148 @@ function suppressedAlert(
provider: string,
accountId: string,
percentUsed: number
): void {
): boolean {
const key = `${sessionId}:${provider}:${accountId}`;
const last = alertSuppression.get(key) ?? 0;
if (Date.now() - last < ALERT_SUPPRESS_WINDOW_MS) return;
if (Date.now() - last < ALERT_SUPPRESS_WINDOW_MS) return false;
alertSuppression.set(key, Date.now());
console.warn(
`[QuotaMonitor] session=${sessionId} ${provider}/${accountId}: ${(percentUsed * 100).toFixed(1)}% quota used`
);
return true;
}
function toIsoTimestamp(value: number | null): string | null {
return typeof value === "number" && Number.isFinite(value) ? new Date(value).toISOString() : null;
}
function getMonitorStatus(percentUsed: number | null): MonitorState["status"] {
if (!Number.isFinite(percentUsed)) return "idle";
if ((percentUsed as number) >= EXHAUSTION_THRESHOLD) return "exhausted";
if ((percentUsed as number) >= WARN_THRESHOLD) return "warning";
return "healthy";
}
function toPublicSnapshot(sessionId: string, state: MonitorState): QuotaMonitorSnapshot {
return {
sessionId,
provider: state.provider,
accountId: state.accountId,
status: state.status,
startedAt: new Date(state.startedAt).toISOString(),
lastPolledAt: toIsoTimestamp(state.lastPolledAt),
lastSuccessAt: toIsoTimestamp(state.lastSuccessAt),
lastErrorAt: toIsoTimestamp(state.lastErrorAt),
lastError: state.lastError,
lastQuotaPercent: state.lastQuotaPercent,
lastQuotaUsed: state.lastQuotaUsed,
lastQuotaTotal: state.lastQuotaTotal,
lastResetAt: state.lastResetAt,
lastAlertAt: toIsoTimestamp(state.lastAlertAt),
nextPollDelayMs: state.nextPollDelayMs,
nextPollAt: toIsoTimestamp(state.nextPollAt),
totalPolls: state.totalPolls,
totalAlerts: state.totalAlerts,
consecutiveFailures: state.consecutiveFailures,
};
}
function sortSnapshots(snapshots: QuotaMonitorSnapshot[]): QuotaMonitorSnapshot[] {
const severityRank: Record<QuotaMonitorSnapshot["status"], number> = {
exhausted: 5,
warning: 4,
error: 3,
starting: 2,
idle: 1,
healthy: 0,
};
return [...snapshots].sort((left, right) => {
const severityDelta = severityRank[right.status] - severityRank[left.status];
if (severityDelta !== 0) return severityDelta;
const quotaDelta = (right.lastQuotaPercent ?? -1) - (left.lastQuotaPercent ?? -1);
if (quotaDelta !== 0) return quotaDelta;
return (
(right.lastPolledAt ? Date.parse(right.lastPolledAt) : 0) -
(left.lastPolledAt ? Date.parse(left.lastPolledAt) : 0)
);
});
}
function scheduleNextPoll(sessionId: string, intervalMs: number): void {
const state = activeMonitors.get(sessionId);
if (!state || state.stopped) return;
state.nextPollDelayMs = intervalMs;
state.nextPollAt = Date.now() + intervalMs;
const { provider, accountId } = state;
const timer = setTimeout(async () => {
const current = activeMonitors.get(sessionId);
if (!current || current.stopped) return;
if (current.sessionBound && !getSessionInfo(sessionId)) {
stopQuotaMonitor(sessionId);
return;
}
try {
const fetcher = quotaFetcherRegistry.get(provider);
if (!fetcher) {
current.status = current.lastQuotaPercent === null ? "idle" : current.status;
scheduleNextPoll(sessionId, NORMAL_INTERVAL_MS);
return;
}
const quota = await fetcher(accountId);
const percentUsed = quota?.percentUsed ?? 0;
current.lastPolledAt = Date.now();
current.totalPolls += 1;
const previousStatus = current.status;
const quota = await fetcher(accountId, current.connectionSnapshot || undefined);
const percentUsed =
quota && typeof quota.percentUsed === "number" && Number.isFinite(quota.percentUsed)
? quota.percentUsed
: null;
current.lastSuccessAt = Date.now();
current.lastError = null;
current.lastErrorAt = null;
current.consecutiveFailures = 0;
current.lastQuotaPercent = percentUsed;
current.lastQuotaUsed =
quota && typeof quota.used === "number" && Number.isFinite(quota.used) ? quota.used : null;
current.lastQuotaTotal =
quota && typeof quota.total === "number" && Number.isFinite(quota.total)
? quota.total
: null;
current.lastResetAt =
quota && typeof quota.resetAt === "string" && quota.resetAt.trim().length > 0
? quota.resetAt
: null;
current.status = getMonitorStatus(percentUsed);
if (percentUsed >= EXHAUSTION_THRESHOLD) {
suppressedAlert(sessionId, provider, accountId, percentUsed);
console.info(
`[QuotaMonitor] session=${sessionId}: marking ${accountId} for next-session cooldown`
);
if (percentUsed !== null && percentUsed >= EXHAUSTION_THRESHOLD) {
const emittedAlert = suppressedAlert(sessionId, provider, accountId, percentUsed);
if (emittedAlert) {
current.lastAlertAt = Date.now();
current.totalAlerts += 1;
}
if (emittedAlert || previousStatus !== "exhausted") {
console.info(
`[QuotaMonitor] session=${sessionId}: marking ${accountId} for next-session cooldown`
);
}
scheduleNextPoll(sessionId, CRITICAL_INTERVAL_MS);
} else if (percentUsed >= WARN_THRESHOLD) {
suppressedAlert(sessionId, provider, accountId, percentUsed);
} else if (percentUsed !== null && percentUsed >= WARN_THRESHOLD) {
const emittedAlert = suppressedAlert(sessionId, provider, accountId, percentUsed);
if (emittedAlert) {
current.lastAlertAt = Date.now();
current.totalAlerts += 1;
}
scheduleNextPoll(sessionId, CRITICAL_INTERVAL_MS);
} else {
scheduleNextPoll(sessionId, NORMAL_INTERVAL_MS);
}
} catch {
} catch (error) {
current.lastErrorAt = Date.now();
current.lastError = error instanceof Error ? error.message : String(error);
current.consecutiveFailures += 1;
current.status = "error";
scheduleNextPoll(sessionId, NORMAL_INTERVAL_MS);
}
}, intervalMs);
@@ -101,9 +252,40 @@ export function startQuotaMonitor(
connection: Record<string, unknown>
): void {
if (!isQuotaMonitorEnabled(connection)) return;
if (activeMonitors.has(sessionId)) return;
const current = activeMonitors.get(sessionId);
if (current && !current.stopped) {
if (current.provider === provider && current.accountId === accountId) {
current.connectionSnapshot = connection;
current.sessionBound = current.sessionBound || getSessionInfo(sessionId) !== null;
return;
}
stopQuotaMonitor(sessionId);
}
activeMonitors.set(sessionId, { timer: null, stopped: false, provider, accountId });
activeMonitors.set(sessionId, {
timer: null,
stopped: false,
provider,
accountId,
connectionSnapshot: connection,
sessionBound: getSessionInfo(sessionId) !== null,
status: "starting",
startedAt: Date.now(),
lastPolledAt: null,
lastSuccessAt: null,
lastErrorAt: null,
lastError: null,
lastQuotaPercent: null,
lastQuotaUsed: null,
lastQuotaTotal: null,
lastResetAt: null,
lastAlertAt: null,
nextPollDelayMs: null,
nextPollAt: null,
totalPolls: 0,
totalAlerts: 0,
consecutiveFailures: 0,
});
scheduleNextPoll(sessionId, NORMAL_INTERVAL_MS);
}
@@ -124,3 +306,52 @@ export function stopQuotaMonitor(sessionId: string): void {
export function getActiveMonitorCount(): number {
return activeMonitors.size;
}
export function getQuotaMonitorSnapshot(sessionId: string): QuotaMonitorSnapshot | null {
const state = activeMonitors.get(sessionId);
if (!state || state.stopped) return null;
return toPublicSnapshot(sessionId, state);
}
export function getQuotaMonitorSnapshots(): QuotaMonitorSnapshot[] {
const snapshots: QuotaMonitorSnapshot[] = [];
for (const [sessionId, state] of activeMonitors) {
if (state.stopped) continue;
snapshots.push(toPublicSnapshot(sessionId, state));
}
return sortSnapshots(snapshots);
}
export function getQuotaMonitorSummary(): QuotaMonitorSummary {
const snapshots = getQuotaMonitorSnapshots();
const statusCounts: Record<QuotaMonitorSnapshot["status"], number> = {
starting: 0,
idle: 0,
healthy: 0,
warning: 0,
exhausted: 0,
error: 0,
};
const byProvider: Record<string, number> = {};
for (const snapshot of snapshots) {
statusCounts[snapshot.status] += 1;
byProvider[snapshot.provider] = (byProvider[snapshot.provider] || 0) + 1;
}
return {
active: snapshots.length,
alerting: statusCounts.warning + statusCounts.exhausted,
exhausted: statusCounts.exhausted,
errors: statusCounts.error,
statusCounts,
byProvider,
};
}
export function clearQuotaMonitors(): void {
for (const sessionId of [...activeMonitors.keys()]) {
stopQuotaMonitor(sessionId);
}
alertSuppression.clear();
}

16
open-sse/services/quotaPreflight.ts
View File

@@ -11,15 +11,20 @@ export interface PreflightQuotaResult {
proceed: boolean;
reason?: string;
quotaPercent?: number;
resetAt?: string | null;
}
export interface QuotaInfo {
used: number;
total: number;
percentUsed: number;
resetAt?: string | null;
}
export type QuotaFetcher = (connectionId: string) => Promise<QuotaInfo | null>;
export type QuotaFetcher = (
connectionId: string,
connection?: Record<string, unknown>
) => Promise<QuotaInfo | null>;
const EXHAUSTION_THRESHOLD = 0.95;
const WARN_THRESHOLD = 0.8;
@@ -51,7 +56,7 @@ export async function preflightQuota(
let quota: QuotaInfo | null = null;
try {
quota = await fetcher(connectionId);
quota = await fetcher(connectionId, connection);
} catch {
return { proceed: true };
}
@@ -66,7 +71,12 @@ export async function preflightQuota(
console.info(
`[QuotaPreflight] ${provider}/${connectionId}: ${(percentUsed * 100).toFixed(1)}% used — switching`
);
return { proceed: false, reason: "quota_exhausted", quotaPercent: percentUsed };
return {
proceed: false,
reason: "quota_exhausted",
quotaPercent: percentUsed,
resetAt: quota.resetAt ?? null,
};
}
if (percentUsed >= WARN_THRESHOLD) {

11
open-sse/services/thinkingBudget.ts
View File

@@ -13,8 +13,12 @@ export const ThinkingMode = {
ADAPTIVE: "adaptive", // Scale based on request complexity
};
import { capThinkingBudget, getDefaultThinkingBudget } from "@/shared/constants/modelSpecs";
import { supportsReasoning } from "./modelCapabilities.ts";
import {
capThinkingBudget,
getDefaultThinkingBudget,
getResolvedModelCapabilities,
supportsReasoning,
} from "@/lib/modelCapabilities";
// Effort → budget token mapping
export const EFFORT_BUDGETS = {
@@ -289,6 +293,9 @@ function applyAdaptiveBudget(body, cfg) {
*/
export function hasThinkingCapableModel(body) {
const model = body.model || "";
const resolved = getResolvedModelCapabilities(model);
if (resolved.supportsThinking === true) return true;
if (resolved.supportsThinking === false) return false;
return (
model.includes("claude") ||
model.includes("o1") ||

2
open-sse/translator/request/openai-to-gemini.ts
View File

@@ -8,7 +8,7 @@ import {
capMaxOutputTokens,
capThinkingBudget,
getDefaultThinkingBudget,
} from "../../../src/shared/constants/modelSpecs.ts";
} from "../../../src/lib/modelCapabilities.ts";
import * as crypto from "crypto";

4
package-lock.json generated
View File

@@ -1,12 +1,12 @@
{
"name": "omniroute",
"version": "3.6.3",
"version": "3.6.4",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "omniroute",
"version": "3.6.3",
"version": "3.6.4",
"hasInstallScript": true,
"license": "MIT",
"workspaces": [

2
package.json
View File

@@ -1,6 +1,6 @@
{
"name": "omniroute",
"version": "3.6.3",
"version": "3.6.4",
"description": "Smart AI Router with auto fallback — route to FREE & cheap models, zero downtime. Works with Cursor, Cline, Claude Desktop, Codex, and any OpenAI-compatible tool.",
"type": "module",
"bin": {

6
playwright.config.ts
View File

@@ -4,6 +4,10 @@ const dashboardPort = process.env.DASHBOARD_PORT || process.env.PORT || "20128";
const dashboardBaseUrl = `http://localhost:${dashboardPort}`;
const webServerReadyUrl = `${dashboardBaseUrl}/api/monitoring/health`;
const playwrightServerMode = process.env.OMNIROUTE_PLAYWRIGHT_SERVER_MODE || "start";
const playwrightWebServerTimeout = Number.parseInt(
process.env.OMNIROUTE_PLAYWRIGHT_WEB_SERVER_TIMEOUT || "900000",
10
);
export default defineConfig({
testDir: "./tests/e2e",
@@ -33,6 +37,6 @@ export default defineConfig({
command: `node scripts/run-next-playwright.mjs ${playwrightServerMode}`,
url: webServerReadyUrl,
reuseExistingServer: !process.env.CI,
timeout: 300_000,
timeout: Number.isFinite(playwrightWebServerTimeout) ? playwrightWebServerTimeout : 900_000,
},
});

9
scripts/build-next-isolated.mjs
View File

@@ -52,7 +52,7 @@ function runNextBuild() {
const child = spawn(process.execPath, [nextBin, "build"], {
cwd: projectRoot,
stdio: "inherit",
env: process.env,
env: resolveNextBuildEnv(process.env),
});
const forward = (signal) => {
@@ -74,6 +74,13 @@ function runNextBuild() {
});
}
export function resolveNextBuildEnv(baseEnv = process.env) {
return {
...baseEnv,
NEXT_PRIVATE_BUILD_WORKER: baseEnv.NEXT_PRIVATE_BUILD_WORKER || "0",
};
}
export async function main() {
let moved = false;

171
scripts/run-next-playwright.mjs
View File

@@ -1,8 +1,9 @@
#!/usr/bin/env node
import { spawn } from "node:child_process";
import { existsSync, renameSync } from "node:fs";
import { join } from "node:path";
import { cpSync, existsSync, mkdirSync, readdirSync, renameSync } from "node:fs";
import { dirname, join } from "node:path";
import { pathToFileURL } from "node:url";
import {
resolveRuntimePorts,
sanitizeColorEnv,
@@ -16,10 +17,19 @@ const cwd = process.cwd();
const appDir = join(cwd, "app");
const srcAppDir = join(cwd, "src", "app");
const appPage = join(appDir, "page.tsx");
const backupDir = join(cwd, "app.__qa_backup");
const defaultBackupDir = join(cwd, "app.__qa_backup");
const backupDir = resolvePlaywrightAppBackupDir({
cwd,
baseBackupExists: existsSync(defaultBackupDir),
appDirExists: existsSync(appDir),
});
const usingAlternativeBackupDir = backupDir !== defaultBackupDir;
const buildScript = join(cwd, "scripts", "build-next-isolated.mjs");
const standaloneServer = join(cwd, testDistDir(), "standalone", "server.js");
const buildIdFile = join(cwd, testDistDir(), "BUILD_ID");
const rootStaticDir = join(cwd, testDistDir(), "static");
const rootPublicDir = join(cwd, "public");
const standaloneStaticDir = join(cwd, testDistDir(), "standalone", ".next", "static");
const standalonePublicDir = join(cwd, testDistDir(), "standalone", "public");
let appDirMoved = false;
@@ -27,19 +37,90 @@ function testDistDir() {
return process.env.NEXT_DIST_DIR || ".next";
}
export function resolvePlaywrightAppBackupDir({
cwd,
baseBackupExists,
appDirExists,
pid = process.pid,
now = Date.now(),
}) {
const baseBackupDir = join(cwd, "app.__qa_backup");
if (!baseBackupExists || !appDirExists) {
return baseBackupDir;
}
return join(cwd, `app.__qa_backup.${pid}.${now}`);
}
function shouldMoveAppDir() {
return existsSync(appDir) && !existsSync(appPage) && existsSync(srcAppDir);
}
export function directoryHasEntries(dirPath) {
try {
return readdirSync(dirPath).length > 0;
} catch {
return false;
}
}
export function standaloneAssetsNeedSync({
standaloneServerPath,
rootStaticDirPath,
standaloneStaticDirPath,
}) {
return (
existsSync(standaloneServerPath) &&
existsSync(rootStaticDirPath) &&
!directoryHasEntries(standaloneStaticDirPath)
);
}
export function syncStandaloneRuntimeAssets({
standaloneServerPath,
rootStaticDirPath,
standaloneStaticDirPath,
rootPublicDirPath,
standalonePublicDirPath,
log = console,
}) {
if (!existsSync(standaloneServerPath)) return false;
let changed = false;
if (existsSync(rootPublicDirPath) && !directoryHasEntries(standalonePublicDirPath)) {
cpSync(rootPublicDirPath, standalonePublicDirPath, {
recursive: true,
force: true,
});
changed = true;
}
if (existsSync(rootStaticDirPath) && !directoryHasEntries(standaloneStaticDirPath)) {
mkdirSync(dirname(standaloneStaticDirPath), {
recursive: true,
});
cpSync(rootStaticDirPath, standaloneStaticDirPath, {
recursive: true,
force: true,
});
changed = true;
}
if (changed) {
log.log("[Playwright WebServer] Rehydrated standalone static/public assets");
}
return changed;
}
function prepareAppDir() {
if (!shouldMoveAppDir()) return;
if (existsSync(backupDir)) {
if (usingAlternativeBackupDir) {
console.warn(
"[Playwright WebServer] app.__qa_backup already exists; leaving app/ in place. " +
"If tests hit 404 on every route, clear app/ artifacts before running e2e."
"[Playwright WebServer] Existing app.__qa_backup detected; using a per-run backup dir instead."
);
return;
}
renameSync(appDir, backupDir);
@@ -55,14 +136,6 @@ function restoreAppDir() {
console.log("[Playwright WebServer] Restored app/ directory");
}
process.on("exit", restoreAppDir);
process.on("uncaughtException", (error) => {
restoreAppDir();
throw error;
});
prepareAppDir();
const bootstrapEnvVars = bootstrapEnv({ quiet: true });
const runtimePorts = resolveRuntimePorts(bootstrapEnvVars);
const testServerEnv = {
@@ -73,8 +146,17 @@ const testServerEnv = {
OMNIROUTE_DISABLE_TOKEN_HEALTHCHECK: process.env.OMNIROUTE_DISABLE_TOKEN_HEALTHCHECK || "1",
OMNIROUTE_DISABLE_LOCAL_HEALTHCHECK: process.env.OMNIROUTE_DISABLE_LOCAL_HEALTHCHECK || "1",
OMNIROUTE_HIDE_HEALTHCHECK_LOGS: process.env.OMNIROUTE_HIDE_HEALTHCHECK_LOGS || "1",
...(process.env.OMNIROUTE_USE_TURBOPACK
? {
OMNIROUTE_USE_TURBOPACK: process.env.OMNIROUTE_USE_TURBOPACK,
}
: {}),
};
export function shouldUseWebpackForPlaywrightDev({ mode, env }) {
return mode === "dev" && env.OMNIROUTE_USE_TURBOPACK !== "1";
}
function runChild(command, args, env) {
return new Promise((resolve) => {
const child = spawn(command, args, {
@@ -100,7 +182,7 @@ function runChild(command, args, env) {
async function runBuildForStart() {
if (mode !== "start") return;
if (process.env.OMNIROUTE_PLAYWRIGHT_SKIP_BUILD === "1") return;
if (existsSync(buildIdFile)) return;
console.log("[Playwright WebServer] Building fresh standalone app for this run...");
const buildEnv = withRuntimePortEnv(testServerEnv, runtimePorts);
const result = await runChild(process.execPath, [buildScript], buildEnv);
@@ -115,18 +197,37 @@ async function runBuildForStart() {
}
}
await runBuildForStart();
if (mode === "start") {
if (existsSync(standaloneServer)) {
spawnWithForwardedSignals(process.execPath, [standaloneServer], {
stdio: "inherit",
env: {
...withRuntimePortEnv(testServerEnv, runtimePorts),
PORT: String(runtimePorts.dashboardPort),
HOSTNAME: process.env.HOSTNAME || "127.0.0.1",
},
});
} else {
export async function main() {
process.on("exit", restoreAppDir);
process.on("uncaughtException", (error) => {
restoreAppDir();
throw error;
});
prepareAppDir();
await runBuildForStart();
if (mode === "start") {
if (existsSync(standaloneServer)) {
syncStandaloneRuntimeAssets({
standaloneServerPath: standaloneServer,
rootStaticDirPath: rootStaticDir,
standaloneStaticDirPath: standaloneStaticDir,
rootPublicDirPath: rootPublicDir,
standalonePublicDirPath: standalonePublicDir,
});
spawnWithForwardedSignals(process.execPath, [standaloneServer], {
stdio: "inherit",
env: {
...withRuntimePortEnv(testServerEnv, runtimePorts),
PORT: String(runtimePorts.dashboardPort),
HOSTNAME: process.env.HOSTNAME || "127.0.0.1",
},
});
return;
}
const args = [
"./node_modules/next/dist/bin/next",
"start",
@@ -138,18 +239,26 @@ if (mode === "start") {
stdio: "inherit",
env: withRuntimePortEnv(testServerEnv, runtimePorts),
});
return;
}
} else {
const args = [
"./node_modules/next/dist/bin/next",
mode,
"--webpack",
"--port",
String(runtimePorts.dashboardPort),
];
if (shouldUseWebpackForPlaywrightDev({ mode, env: testServerEnv })) {
args.splice(2, 0, "--webpack");
}
spawnWithForwardedSignals(process.execPath, args, {
stdio: "inherit",
env: withRuntimePortEnv(testServerEnv, runtimePorts),
});
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
await main();
}

72
src/app/(dashboard)/dashboard/analytics/ComboHealthTab.tsx
View File

@@ -20,6 +20,10 @@ function formatShare(value: number) {
return formatPercent(value * 100, 1);
}
function formatPercentOrDash(value: number | null, digits = 1) {
return typeof value === "number" ? formatPercent(value, digits) : "n/a";
}
function formatLatency(value: number) {
return `${Math.round(value).toLocaleString()}ms`;
}
@@ -95,6 +99,7 @@ function ComboHealthCard({ combo }: { combo: ComboHealthMetrics }) {
),
[combo.usageSkew.modelDistribution]
);
const targetHealth = combo.targetHealth || [];
return (
<Card className="overflow-hidden p-0">
@@ -251,6 +256,73 @@ function ComboHealthCard({ combo }: { combo: ComboHealthMetrics }) {
</div>
</section>
</div>
{targetHealth.length > 0 ? (
<div className="border-t border-black/5 px-6 py-5 dark:border-white/5">
<div>
<div className="text-sm font-semibold text-text-main">Execution targets</div>
<div className="mt-1 text-xs text-text-muted">
Step-level runtime metrics and quota visibility for structured combo targets.
</div>
</div>
<div className="mt-4 grid gap-3 lg:grid-cols-2">
{targetHealth.map((target) => (
<div
key={target.executionKey}
className="rounded-lg border border-black/5 bg-black/[0.02] p-4 dark:border-white/5 dark:bg-white/[0.02]"
>
<div className="flex flex-wrap items-start justify-between gap-3">
<div className="min-w-0">
<div className="truncate text-sm font-medium text-text-main">
{target.label || target.model}
</div>
<div className="mt-1 text-xs text-text-muted">
{target.provider}
{target.connectionId ? ` · ${target.connectionId.slice(0, 8)}` : ""}
</div>
<div className="mt-1 text-[11px] text-text-muted">{target.stepId}</div>
</div>
<div className="flex items-center gap-2">
{target.lastStatus ? (
<Badge size="sm" variant={target.lastStatus === "ok" ? "success" : "error"}>
{target.lastStatus}
</Badge>
) : null}
<Badge size="sm" variant="default">
{target.requests} req
</Badge>
</div>
</div>
<div className="mt-3 grid gap-2 sm:grid-cols-3">
<DistributionBar
label="Success"
value={Math.max(target.successRate, 0) / 100}
meta={formatPercent(target.successRate, 0)}
/>
<DistributionBar
label="Latency"
value={target.avgLatencyMs > 0 ? 1 : 0}
meta={formatLatency(target.avgLatencyMs)}
/>
<DistributionBar
label="Quota"
value={Math.max(target.quotaRemainingPct || 0, 0) / 100}
meta={formatPercentOrDash(target.quotaRemainingPct)}
/>
</div>
<div className="mt-3 flex flex-wrap items-center gap-2 text-[11px] text-text-muted">
<span>Quota scope: {target.quotaScope}</span>
{target.quotaTrend ? <span>Trend: {target.quotaTrend}</span> : null}
{target.quotaIsExhausted ? <span>Exhausted</span> : null}
</div>
</div>
))}
</div>
</div>
) : null}
</Card>
);
}

2242
src/app/(dashboard)/dashboard/combos/page.tsx
View File

File diff suppressed because it is too large Load Diff

142
src/app/(dashboard)/dashboard/health/page.tsx
View File

@@ -137,7 +137,15 @@ export default function HealthPage() {
);
}
const { system, providerHealth, providerSummary, rateLimitStatus, lockouts } = data;
const {
system,
providerHealth,
providerSummary,
rateLimitStatus,
lockouts,
sessions,
quotaMonitor,
} = data;
const cbEntries = Object.entries(providerHealth || {});
const lockoutEntries = Object.entries(lockouts || {});
@@ -273,6 +281,138 @@ export default function HealthPage() {
</Card>
</div>
{/* Session & Quota Observability */}
<div className="grid grid-cols-1 xl:grid-cols-2 gap-4">
<Card className="p-5">
<div className="flex items-center justify-between mb-4">
<h2 className="text-lg font-semibold text-text-main flex items-center gap-2">
<span className="material-symbols-outlined text-[20px] text-primary">groups</span>
Session Activity
</h2>
<span className="text-xs text-text-muted">{sessions?.activeCount ?? 0} active</span>
</div>
<div className="grid grid-cols-2 gap-3 mb-4">
<div className="rounded-xl border border-border/40 bg-surface/30 p-3">
<div className="text-xs text-text-muted">Sticky-bound sessions</div>
<div className="text-2xl font-semibold text-text-main mt-1">
{sessions?.stickyBoundCount ?? 0}
</div>
</div>
<div className="rounded-xl border border-border/40 bg-surface/30 p-3">
<div className="text-xs text-text-muted">Sessions by API key</div>
<div className="text-2xl font-semibold text-text-main mt-1">
{Object.keys(sessions?.byApiKey || {}).length}
</div>
</div>
</div>
{sessions?.top?.length > 0 ? (
<div className="space-y-2">
{sessions.top.slice(0, 5).map((session: any) => (
<div
key={session.sessionId}
className="rounded-lg border border-border/30 bg-surface/20 p-3 flex items-center justify-between gap-3"
>
<div className="min-w-0">
<div className="font-mono text-xs text-text-main truncate">
{session.sessionId}
</div>
<div className="text-xs text-text-muted mt-1">
{session.requestCount} requests
{session.connectionId ? ` • ${session.connectionId.slice(0, 8)}…` : ""}
</div>
</div>
<div className="text-right text-xs text-text-muted shrink-0">
<div>{Math.round((session.idleMs || 0) / 1000)}s idle</div>
<div>{Math.round((session.ageMs || 0) / 1000)}s age</div>
</div>
</div>
))}
</div>
) : (
<p className="text-sm text-text-muted">No active sessions tracked yet.</p>
)}
</Card>
<Card className="p-5">
<div className="flex items-center justify-between mb-4">
<h2 className="text-lg font-semibold text-text-main flex items-center gap-2">
<span className="material-symbols-outlined text-[20px] text-primary">radar</span>
Quota Monitors
</h2>
<span className="text-xs text-text-muted">{quotaMonitor?.active ?? 0} active</span>
</div>
<div className="grid grid-cols-2 md:grid-cols-4 gap-3 mb-4">
<div className="rounded-xl border border-border/40 bg-surface/30 p-3">
<div className="text-xs text-text-muted">Alerting</div>
<div className="text-2xl font-semibold text-amber-400 mt-1">
{quotaMonitor?.alerting ?? 0}
</div>
</div>
<div className="rounded-xl border border-border/40 bg-surface/30 p-3">
<div className="text-xs text-text-muted">Exhausted</div>
<div className="text-2xl font-semibold text-red-400 mt-1">
{quotaMonitor?.exhausted ?? 0}
</div>
</div>
<div className="rounded-xl border border-border/40 bg-surface/30 p-3">
<div className="text-xs text-text-muted">Errors</div>
<div className="text-2xl font-semibold text-orange-400 mt-1">
{quotaMonitor?.errors ?? 0}
</div>
</div>
<div className="rounded-xl border border-border/40 bg-surface/30 p-3">
<div className="text-xs text-text-muted">Providers</div>
<div className="text-2xl font-semibold text-text-main mt-1">
{Object.keys(quotaMonitor?.byProvider || {}).length}
</div>
</div>
</div>
{quotaMonitor?.monitors?.length > 0 ? (
<div className="space-y-2">
{quotaMonitor.monitors.slice(0, 5).map((monitor: any) => (
<div
key={`${monitor.sessionId}:${monitor.accountId}`}
className="rounded-lg border border-border/30 bg-surface/20 p-3 flex items-center justify-between gap-3"
>
<div className="min-w-0">
<div className="text-sm font-medium text-text-main truncate">
{monitor.provider} • {monitor.accountId.slice(0, 8)}…
</div>
<div className="text-xs text-text-muted mt-1 truncate">
{monitor.sessionId} • {monitor.status}
</div>
</div>
<div className="text-right text-xs shrink-0">
<div
className={
monitor.status === "exhausted"
? "text-red-400"
: monitor.status === "warning"
? "text-amber-400"
: monitor.status === "error"
? "text-orange-400"
: "text-text-main"
}
>
{typeof monitor.lastQuotaPercent === "number"
? `${Math.round(monitor.lastQuotaPercent * 100)}%`
: "—"}
</div>
<div className="text-text-muted">
{monitor.nextPollDelayMs
? `${Math.round(monitor.nextPollDelayMs / 1000)}s`
: "—"}
</div>
</div>
</div>
))}
</div>
) : (
<p className="text-sm text-text-muted">No session quota monitors active.</p>
)}
</Card>
</div>
{/* Graceful Degradation Status */}
{degradation && degradation.features && degradation.features.length > 0 && (
<Card className="p-5" role="region" aria-label="Graceful Degradation Status">

8
src/app/(dashboard)/dashboard/usage/components/BudgetTelemetryCards.tsx
View File

@@ -51,6 +51,14 @@ export default function BudgetTelemetryCards() {
<span className="text-text-muted">{t("totalRequests")}</span>
<span className="font-mono">{telemetry.totalRequests ?? 0}</span>
</div>
<div className="flex justify-between">
<span className="text-text-muted">Active sessions</span>
<span className="font-mono">{telemetry.sessions?.activeCount ?? 0}</span>
</div>
<div className="flex justify-between">
<span className="text-text-muted">Quota alerts</span>
<span className="font-mono">{telemetry.quotaMonitor?.alerting ?? 0}</span>
</div>
</div>
) : (
<p className="text-sm text-text-muted">{t("noDataYet")}</p>

14
src/app/api/cli-tools/openclaw/auto-order/route.ts
View File

@@ -5,6 +5,7 @@
*/
import { NextResponse } from "next/server";
import { getComboModelProvider } from "@/lib/combos/steps";
import { resolveOmniRouteBaseUrl } from "@/shared/utils/resolveOmniRouteBaseUrl";
const OMNIROUTE_BASE_URL = resolveOmniRouteBaseUrl();
@@ -18,7 +19,12 @@ export async function GET() {
]);
const health = healthRes.status === "fulfilled" ? await healthRes.value.json() : {};
const combos = combosRes.status === "fulfilled" ? await combosRes.value.json() : [];
const combosPayload = combosRes.status === "fulfilled" ? await combosRes.value.json() : [];
const combos = Array.isArray(combosPayload)
? combosPayload
: Array.isArray(combosPayload?.combos)
? combosPayload.combos
: [];
// Build provider scores from circuit breaker state
const breakers: any[] = health?.circuitBreakers || [];
@@ -29,8 +35,10 @@ export async function GET() {
if (Array.isArray(combos)) {
for (const combo of combos) {
for (const model of combo.models || combo.data?.models || []) {
allProviders.add(model.provider);
providerScores.set(model.provider, (providerScores.get(model.provider) || 0) + 1);
const provider = getComboModelProvider(model);
if (!provider) continue;
allProviders.add(provider);
providerScores.set(provider, (providerScores.get(provider) || 0) + 1);
}
}
}

31
src/app/api/combos/[id]/route.ts
View File

@@ -9,6 +9,8 @@ import {
} from "@/lib/localDb";
import { getConsistentMachineId } from "@/shared/utils/machineId";
import { syncToCloud } from "@/lib/cloudSync";
import { validateCompositeTiersConfig } from "@/lib/combos/compositeTiers";
import { normalizeComboModels } from "@/lib/combos/steps";
import { validateComboDAG } from "@omniroute/open-sse/services/combo.ts";
import { updateComboSchema } from "@/shared/validation/schemas";
import { isValidationFailure, validateBody } from "@/shared/validation/helpers";
@@ -53,7 +55,29 @@ export async function PUT(request, { params }) {
if (isValidationFailure(validation)) {
return NextResponse.json({ error: validation.error }, { status: 400 });
}
const body = validation.data;
const currentCombo = await getComboById(id);
if (!currentCombo) {
return NextResponse.json({ error: "Combo not found" }, { status: 404 });
}
const comboName = validation.data.name || currentCombo.name;
const body = validation.data.models
? {
...validation.data,
models: normalizeComboModels(validation.data.models, {
comboName,
}),
}
: validation.data;
const nextComboState = {
...currentCombo,
...body,
name: comboName,
};
const compositeValidation = validateCompositeTiersConfig(nextComboState);
if (!compositeValidation.success) {
return NextResponse.json({ error: compositeValidation.error }, { status: 400 });
}
// Check if name already exists (exclude current combo)
if (body.name) {
@@ -68,7 +92,6 @@ export async function PUT(request, { params }) {
const allCombos = await getCombos();
// Update the combo in the list temporarily for validation
const updatedCombos = allCombos.map((c) => (c.id === id ? { ...c, ...body } : c));
const comboName = body.name || (await getComboById(id))?.name;
if (comboName) {
try {
validateComboDAG(comboName, updatedCombos);
@@ -80,10 +103,6 @@ export async function PUT(request, { params }) {
const combo = await updateCombo(id, body);
if (!combo) {
return NextResponse.json({ error: "Combo not found" }, { status: 404 });
}
// Auto sync to Cloud if enabled
await syncToCloudIfEnabled();

12
src/app/api/combos/builder/options/route.ts Normal file
View File

@@ -0,0 +1,12 @@
import { NextResponse } from "next/server";
import { getComboBuilderOptions } from "@/lib/combos/builderOptions";
export async function GET() {
try {
const options = await getComboBuilderOptions();
return NextResponse.json(options);
} catch (error) {
console.log("Error fetching combo builder options:", error);
return NextResponse.json({ error: "Failed to fetch combo builder options" }, { status: 500 });
}
}

24
src/app/api/combos/route.ts
View File

@@ -2,6 +2,8 @@ import { NextResponse } from "next/server";
import { getCombos, createCombo, getComboByName, isCloudEnabled } from "@/lib/localDb";
import { getConsistentMachineId } from "@/shared/utils/machineId";
import { syncToCloud } from "@/lib/cloudSync";
import { validateCompositeTiersConfig } from "@/lib/combos/compositeTiers";
import { normalizeComboModels } from "@/lib/combos/steps";
import { validateComboDAG } from "@omniroute/open-sse/services/combo.ts";
import { createComboSchema } from "@/shared/validation/schemas";
import { isValidationFailure, validateBody } from "@/shared/validation/helpers";
@@ -27,7 +29,18 @@ export async function POST(request) {
if (isValidationFailure(validation)) {
return NextResponse.json({ error: validation.error }, { status: 400 });
}
const { name, models, strategy, config } = validation.data;
const normalizedModels = normalizeComboModels(validation.data.models, {
comboName: validation.data.name,
});
const comboInput = {
...validation.data,
models: normalizedModels,
};
const { name, strategy, config } = comboInput;
const compositeValidation = validateCompositeTiersConfig(comboInput);
if (!compositeValidation.success) {
return NextResponse.json({ error: compositeValidation.error }, { status: 400 });
}
// Check if name already exists
const existing = await getComboByName(name);
@@ -38,14 +51,19 @@ export async function POST(request) {
// Validate nested combo DAG (no circular references, max depth)
const allCombos = await getCombos();
// Temporarily add the new combo to validate its graph
const tempCombo = { name, models: models || [], strategy, config };
const tempCombo = {
...comboInput,
name,
strategy,
config,
};
try {
validateComboDAG(name, [...allCombos, tempCombo]);
} catch (dagError) {
return NextResponse.json({ error: dagError.message }, { status: 400 });
}
const combo = await createCombo({ name, models: models || [], strategy, config });
const combo = await createCombo(comboInput);
// Auto sync to Cloud if enabled
await syncToCloudIfEnabled();

57
src/app/api/combos/test/route.ts
View File

@@ -1,16 +1,29 @@
import { randomUUID } from "node:crypto";
import { NextResponse } from "next/server";
import { buildComboTestRequestBody, extractComboTestResponseText } from "@/lib/combos/testHealth";
import { getComboByName } from "@/lib/localDb";
import { getComboByName, getCombos } from "@/lib/localDb";
import { resolveNestedComboTargets } from "@omniroute/open-sse/services/combo.ts";
import { testComboSchema } from "@/shared/validation/schemas";
import { isValidationFailure, validateBody } from "@/shared/validation/helpers";
async function testComboModel(modelStr, internalUrl) {
function buildComboTestResult(target, partial = {}) {
return {
model: target.modelStr,
provider: target.provider,
stepId: target.stepId,
executionKey: target.executionKey,
connectionId: target.connectionId,
label: target.label,
...partial,
};
}
async function testComboTarget(target, internalUrl) {
const startTime = Date.now();
try {
// Send a minimal but real chat request through the same internal
// endpoint an external OpenAI-compatible client would use.
const testBody = buildComboTestRequestBody(modelStr);
const testBody = buildComboTestRequestBody(target.modelStr);
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 20000);
@@ -48,16 +61,15 @@ async function testComboModel(modelStr, internalUrl) {
const responseText = extractComboTestResponseText(responseBody);
if (!responseText) {
return {
model: modelStr,
return buildComboTestResult(target, {
status: "error",
statusCode: res.status,
error: "Provider returned HTTP 200 but no text content.",
latencyMs,
};
});
}
return { model: modelStr, status: "ok", latencyMs, responseText };
return buildComboTestResult(target, { status: "ok", latencyMs, responseText });
}
let errorMsg = "";
@@ -68,21 +80,19 @@ async function testComboModel(modelStr, internalUrl) {
errorMsg = res.statusText;
}
return {
model: modelStr,
return buildComboTestResult(target, {
status: "error",
statusCode: res.status,
error: errorMsg,
latencyMs,
};
});
} catch (error) {
const latencyMs = Date.now() - startTime;
return {
model: modelStr,
return buildComboTestResult(target, {
status: "error",
error: error.name === "AbortError" ? "Timeout (20s)" : error.message,
latencyMs,
};
});
}
}
@@ -119,22 +129,35 @@ export async function POST(request) {
return NextResponse.json({ error: "Combo not found" }, { status: 404 });
}
const models = (combo.models || []).map((m) => (typeof m === "string" ? m : m.model));
const allCombos = await getCombos();
const targets = resolveNestedComboTargets(combo, allCombos);
if (models.length === 0) {
if (targets.length === 0) {
return NextResponse.json({ error: "Combo has no models" }, { status: 400 });
}
const internalUrl = `${getBaseUrl(request)}/v1/chat/completions`;
const results = await Promise.all(
models.map((modelStr) => testComboModel(modelStr, internalUrl))
targets.map((target) => testComboTarget(target, internalUrl))
);
const resolvedBy = results.find((result) => result.status === "ok")?.model || null;
const resolvedResult = results.find((result) => result.status === "ok") || null;
const resolvedBy = resolvedResult?.model || null;
return NextResponse.json({
comboName,
strategy: combo.strategy || "priority",
resolvedBy,
resolvedByExecutionKey: resolvedResult?.executionKey || null,
resolvedByTarget: resolvedResult
? {
model: resolvedResult.model,
provider: resolvedResult.provider,
stepId: resolvedResult.stepId,
executionKey: resolvedResult.executionKey,
connectionId: resolvedResult.connectionId,
label: resolvedResult.label,
}
: null,
results,
testedAt: new Date().toISOString(),
});

74
src/app/api/monitoring/health/route.ts
View File

@@ -1,5 +1,6 @@
import { NextResponse } from "next/server";
import { getProviderConnections, getSettings } from "@/lib/localDb";
import { buildHealthPayload } from "@/lib/monitoring/observability";
import { APP_CONFIG } from "@/shared/constants/config";
import { AI_PROVIDERS } from "@/shared/constants/providers";
@@ -15,67 +16,38 @@ export async function GET() {
const { getAllRateLimitStatus } = await import("@omniroute/open-sse/services/rateLimitManager");
const { getAllModelLockouts } = await import("@omniroute/open-sse/services/accountFallback");
const { getInflightCount } = await import("@omniroute/open-sse/services/requestDedup.ts");
const { getQuotaMonitorSummary, getQuotaMonitorSnapshots } =
await import("@omniroute/open-sse/services/quotaMonitor.ts");
const { getActiveSessions, getAllActiveSessionCountsByKey } =
await import("@omniroute/open-sse/services/sessionManager.ts");
const settings = await getSettings();
const connections = await getProviderConnections();
const circuitBreakers = getAllCircuitBreakerStatuses();
const rateLimitStatus = getAllRateLimitStatus();
const lockouts = getAllModelLockouts();
const quotaMonitorSummary = getQuotaMonitorSummary();
const quotaMonitorMonitors = getQuotaMonitorSnapshots();
const activeSessions = getActiveSessions();
const activeSessionsByKey = getAllActiveSessionCountsByKey();
const { getAllHealthStatuses } = await import("@/lib/localHealthCheck");
// System info
const system = {
version: APP_CONFIG.version,
nodeVersion: process.version,
uptime: process.uptime(),
memoryUsage: process.memoryUsage(),
pid: process.pid,
platform: process.platform,
};
// Provider health summary (circuitBreakers is an Array of { name, state, ... })
const providerHealth = {};
for (const cb of circuitBreakers) {
// Skip test circuit breakers (leftover from unit tests)
if (cb.name.startsWith("test-") || cb.name.startsWith("test_")) continue;
providerHealth[cb.name] = {
state: cb.state,
failures: cb.failureCount || 0,
lastFailure: cb.lastFailureTime,
};
}
const configuredProviders = new Set(connections.map((c: any) => c.provider));
const activeProviders = new Set(
connections.filter((c: any) => c.isActive !== false).map((c: any) => c.provider)
);
return NextResponse.json({
status: "healthy",
timestamp: new Date().toISOString(),
system,
providerHealth,
providerSummary: {
catalogCount: Object.keys(AI_PROVIDERS).length,
configuredCount: configuredProviders.size,
activeCount: activeProviders.size,
monitoredCount: Object.keys(providerHealth).length,
},
localProviders: getAllHealthStatuses(),
const payload = buildHealthPayload({
appVersion: APP_CONFIG.version,
catalogCount: Object.keys(AI_PROVIDERS).length,
settings,
connections,
circuitBreakers,
rateLimitStatus,
lockouts,
dedup: {
inflightRequests: getInflightCount(),
},
cryptography: {
status:
process.env.STORAGE_ENCRYPTION_KEY && process.env.STORAGE_ENCRYPTION_KEY.length >= 32
? "healthy"
: "missing_or_invalid",
provider: "aes-256-gcm",
},
setupComplete: settings?.setupComplete || false,
localProviders: getAllHealthStatuses(),
inflightRequests: getInflightCount(),
quotaMonitorSummary,
quotaMonitorMonitors,
activeSessions,
activeSessionsByKey,
});
return NextResponse.json(payload);
} catch (error) {
console.error("[API] GET /api/monitoring/health error:", error);
return NextResponse.json({ status: "error", error: "Health check failed" }, { status: 500 });

15
src/app/api/openapi/spec/route.ts
View File

@@ -6,20 +6,21 @@
import { NextResponse } from "next/server";
import fs from "fs";
import path from "path";
import { fileURLToPath } from "url";
import yaml from "js-yaml";
let cachedSpec: { data: any; mtime: number } | null = null;
const ROUTE_DIR = path.dirname(fileURLToPath(import.meta.url));
const PROJECT_ROOT = path.resolve(ROUTE_DIR, "../../../../../");
const OPENAPI_SPEC_CANDIDATES = [
path.join(PROJECT_ROOT, "docs", "openapi.yaml"),
path.join(PROJECT_ROOT, "app", "docs", "openapi.yaml"),
];
export async function GET() {
try {
// Try multiple locations for the spec file
const candidates = [
path.join(process.cwd(), "docs", "openapi.yaml"),
path.join(process.cwd(), "app", "docs", "openapi.yaml"),
];
let specPath = "";
for (const p of candidates) {
for (const p of OPENAPI_SPEC_CANDIDATES) {
if (fs.existsSync(p)) {
specPath = p;
break;

10
src/app/api/telemetry/summary/route.ts
View File

@@ -1,4 +1,5 @@
import { NextResponse } from "next/server";
import { buildTelemetryPayload } from "@/lib/monitoring/observability";
import { getTelemetrySummary } from "@/shared/utils/requestTelemetry";
export async function GET(request) {
@@ -6,7 +7,14 @@ export async function GET(request) {
const { searchParams } = new URL(request.url);
const windowMs = parseInt(searchParams.get("windowMs") || "300000", 10);
const summary = getTelemetrySummary(windowMs);
return NextResponse.json(summary);
const { getQuotaMonitorSummary } = await import("@omniroute/open-sse/services/quotaMonitor.ts");
const { getActiveSessions } = await import("@omniroute/open-sse/services/sessionManager.ts");
const payload = buildTelemetryPayload({
summary,
quotaMonitorSummary: getQuotaMonitorSummary(),
activeSessions: getActiveSessions(),
});
return NextResponse.json(payload);
} catch (error) {
return NextResponse.json({ error: error.message }, { status: 500 });
}

273
src/app/api/usage/combo-health/route.ts
View File

@@ -3,6 +3,8 @@ import { z } from "zod";
import { getDbInstance } from "@/lib/db/core";
import { getComboById, getCombos } from "@/lib/db/combos";
import { getQuotaSnapshots } from "@/lib/db/quotaSnapshots";
import { getComboMetrics } from "@omniroute/open-sse/services/comboMetrics.ts";
import { resolveNestedComboTargets } from "@omniroute/open-sse/services/combo.ts";
import type {
ComboHealthMetrics,
ComboHealthResponse,
@@ -10,13 +12,11 @@ import type {
UtilizationTimeRange,
} from "@/shared/types/utilization";
type ComboModelNode = string | { model?: string | null };
type ComboRecord = {
id?: string;
name?: string;
strategy?: string;
models?: ComboModelNode[];
models?: unknown[];
};
type ModelUsageRow = {
@@ -45,6 +45,40 @@ type ProviderHealth = {
trend: "improving" | "stable" | "declining";
};
type ResolvedComboTargetView = {
stepId: string;
executionKey: string;
modelStr: string;
provider: string;
connectionId: string | null;
label: string | null;
};
type RuntimeTargetMetricView = {
requests?: number;
successRate?: number;
avgLatencyMs?: number;
lastStatus?: "ok" | "error" | null;
lastUsedAt?: string | null;
};
type HistoricalTargetUsageRow = {
combo_execution_key: string | null;
combo_step_id: string | null;
status: number | null;
duration: number | null;
timestamp: string | null;
};
type HistoricalTargetMetricView = {
stepId: string | null;
requests: number;
successRate: number;
avgLatencyMs: number;
lastStatus: "ok" | "error" | null;
lastUsedAt: string | null;
};
const querySchema = z.object({
range: z.enum(["1h", "24h", "7d", "30d"]),
comboId: z
@@ -84,25 +118,15 @@ function toSafeNumber(value: number | null | undefined): number {
return typeof value === "number" && Number.isFinite(value) ? value : 0;
}
function normalizeComboModels(models: ComboModelNode[] | undefined): string[] {
if (!Array.isArray(models)) return [];
return models
.map((entry) => {
if (typeof entry === "string") return entry;
if (entry && typeof entry === "object" && typeof entry.model === "string") {
return entry.model;
}
return "";
})
.filter((entry): entry is string => entry.trim().length > 0);
}
function extractProvider(model: string): string {
const [provider] = model.split("/");
return provider?.trim() || "unknown";
}
function toNonEmptyString(value: unknown): string | null {
return typeof value === "string" && value.trim().length > 0 ? value.trim() : null;
}
function calculateGini(values: number[]): number {
if (values.length === 0) return 0;
const sorted = [...values].sort((a, b) => a - b);
@@ -198,6 +222,50 @@ function buildProviderHealth(provider: string, snapshots: QuotaSnapshotRow[]): P
};
}
function buildConnectionHealth(
provider: string,
connectionId: string,
snapshots: QuotaSnapshotRow[]
): ProviderHealth | null {
if (snapshots.length === 0) return null;
const ordered = [...snapshots].sort((left, right) => {
const leftView = left as unknown as QuotaSnapshotView;
const rightView = right as unknown as QuotaSnapshotView;
return (leftView.createdAt || "").localeCompare(rightView.createdAt || "");
});
const firstSnapshot = ordered.find((entry) => {
const snapshotView = entry as unknown as QuotaSnapshotView;
return (
snapshotView.remainingPercentage !== null && snapshotView.remainingPercentage !== undefined
);
});
const lastSnapshot = [...ordered].reverse().find((entry) => {
const snapshotView = entry as unknown as QuotaSnapshotView;
return (
snapshotView.remainingPercentage !== null && snapshotView.remainingPercentage !== undefined
);
});
const firstRemaining =
(firstSnapshot as unknown as QuotaSnapshotView | undefined)?.remainingPercentage ?? 0;
const lastRemaining =
(lastSnapshot as unknown as QuotaSnapshotView | undefined)?.remainingPercentage ?? 0;
const delta = lastRemaining - firstRemaining;
let trend: ProviderHealth["trend"] = "stable";
if (delta >= 5) trend = "improving";
if (delta <= -5) trend = "declining";
return {
provider: `${provider}:${connectionId}`,
remainingPct: roundNumber(lastRemaining),
isExhausted:
(ordered[ordered.length - 1] as unknown as QuotaSnapshotView | undefined)?.isExhausted === 1,
trend,
};
}
function buildUsageSkew(
comboName: string,
comboModels: string[],
@@ -298,13 +366,168 @@ function buildQuotaHealth(providers: string[], since: string): ComboHealthMetric
};
}
function buildComboHealth(combo: ComboRecord, since: string): ComboHealthMetrics | null {
function getHistoricalTargetMetrics(
comboName: string,
since: string
): Map<string, HistoricalTargetMetricView> {
const db = getDbInstance();
const rows = db
.prepare(
`SELECT
combo_execution_key,
combo_step_id,
status,
duration,
timestamp
FROM call_logs
WHERE combo_name = ?
AND timestamp >= ?
AND COALESCE(NULLIF(combo_execution_key, ''), NULLIF(combo_step_id, '')) IS NOT NULL
ORDER BY combo_execution_key ASC, combo_step_id ASC, timestamp DESC`
)
.all(comboName, since) as HistoricalTargetUsageRow[];
const metrics = new Map<
string,
{
stepId: string | null;
requests: number;
successCount: number;
totalLatencyMs: number;
lastStatus: "ok" | "error" | null;
lastUsedAt: string | null;
}
>();
for (const row of rows) {
const executionKey =
toNonEmptyString(row.combo_execution_key) || toNonEmptyString(row.combo_step_id);
if (!executionKey) continue;
let metric = metrics.get(executionKey);
if (!metric) {
const statusCode = toSafeNumber(row.status);
metric = {
stepId: toNonEmptyString(row.combo_step_id),
requests: 0,
successCount: 0,
totalLatencyMs: 0,
lastStatus: statusCode > 0 ? (statusCode < 400 ? "ok" : "error") : null,
lastUsedAt: toNonEmptyString(row.timestamp),
};
metrics.set(executionKey, metric);
}
metric.requests += 1;
metric.totalLatencyMs += toSafeNumber(row.duration);
if (toSafeNumber(row.status) >= 200 && toSafeNumber(row.status) < 400) {
metric.successCount += 1;
}
if (!metric.stepId) {
metric.stepId = toNonEmptyString(row.combo_step_id);
}
}
return new Map(
Array.from(metrics.entries()).map(([executionKey, metric]) => [
executionKey,
{
stepId: metric.stepId,
requests: metric.requests,
successRate:
metric.requests > 0 ? Math.round((metric.successCount / metric.requests) * 100) : 0,
avgLatencyMs: metric.requests > 0 ? Math.round(metric.totalLatencyMs / metric.requests) : 0,
lastStatus: metric.lastStatus,
lastUsedAt: metric.lastUsedAt,
},
])
);
}
function buildTargetHealth(
comboName: string,
targets: ResolvedComboTargetView[],
since: string
): NonNullable<ComboHealthMetrics["targetHealth"]> {
const comboMetrics = getComboMetrics(comboName);
const historicalMetrics = getHistoricalTargetMetrics(comboName, since);
return targets.map((target) => {
const historicalMetric =
historicalMetrics.get(target.executionKey) || historicalMetrics.get(target.stepId) || null;
const runtimeMetric =
historicalMetric === null
? ((comboMetrics?.byTarget?.[target.executionKey] ||
comboMetrics?.byTarget?.[target.stepId] ||
null) as RuntimeTargetMetricView | null)
: null;
let quotaRemainingPct: number | null = null;
let quotaIsExhausted: boolean | null = null;
let quotaTrend: "improving" | "stable" | "declining" | null = null;
let quotaScope: "connection" | "provider" | "none" = "none";
if (target.connectionId) {
const connectionHealth = buildConnectionHealth(
target.provider,
target.connectionId,
getQuotaSnapshots({
provider: target.provider,
connectionId: target.connectionId,
since,
})
);
if (connectionHealth) {
quotaRemainingPct = connectionHealth.remainingPct;
quotaIsExhausted = connectionHealth.isExhausted;
quotaTrend = connectionHealth.trend;
quotaScope = "connection";
}
}
if (quotaScope === "none") {
const providerSnapshots = getQuotaSnapshots({ provider: target.provider, since });
const providerHealth = buildProviderHealth(target.provider, providerSnapshots);
if (providerSnapshots.length > 0) {
quotaRemainingPct = providerHealth.remainingPct;
quotaIsExhausted = providerHealth.isExhausted;
quotaTrend = providerHealth.trend;
quotaScope = "provider";
}
}
return {
executionKey: target.executionKey,
stepId: target.stepId,
model: target.modelStr,
provider: target.provider,
connectionId: target.connectionId,
label: target.label,
requests: toSafeNumber(historicalMetric?.requests ?? runtimeMetric?.requests),
successRate: toSafeNumber(historicalMetric?.successRate ?? runtimeMetric?.successRate),
avgLatencyMs: toSafeNumber(historicalMetric?.avgLatencyMs ?? runtimeMetric?.avgLatencyMs),
lastStatus: historicalMetric?.lastStatus ?? runtimeMetric?.lastStatus ?? null,
lastUsedAt: historicalMetric?.lastUsedAt ?? runtimeMetric?.lastUsedAt ?? null,
quotaRemainingPct,
quotaIsExhausted,
quotaTrend,
quotaScope,
};
});
}
function buildComboHealth(
combo: ComboRecord,
since: string,
allCombos: ComboRecord[]
): ComboHealthMetrics | null {
const comboId = typeof combo.id === "string" ? combo.id : "";
const comboName = typeof combo.name === "string" ? combo.name : "";
if (!comboId || !comboName) return null;
const models = normalizeComboModels(combo.models);
const providers = Array.from(new Set(models.map(extractProvider)));
const targets = resolveNestedComboTargets(combo, allCombos) as ResolvedComboTargetView[];
const models = targets.map((target) => target.modelStr);
const providers = Array.from(new Set(targets.map((target) => target.provider)));
return {
comboId,
@@ -314,6 +537,7 @@ function buildComboHealth(combo: ComboRecord, since: string): ComboHealthMetrics
? combo.strategy
: "priority",
models,
targetHealth: buildTargetHealth(comboName, targets, since),
quotaHealth: buildQuotaHealth(providers, since),
usageSkew: buildUsageSkew(comboName, models, since),
performance: buildPerformance(comboName, since),
@@ -340,21 +564,24 @@ export async function GET(request: Request) {
const { range, comboId } = parsedQuery.data;
const since = getRangeStartIso(range);
const allCombos = (await getCombos()) as ComboRecord[];
let combos: ComboRecord[] = [];
if (comboId) {
const combo = (await getComboById(comboId)) as ComboRecord | null;
const combo =
allCombos.find((entry) => entry.id === comboId) ||
((await getComboById(comboId)) as ComboRecord | null);
if (!combo) {
return NextResponse.json({ error: "Combo not found" }, { status: 404 });
}
combos = [combo];
} else {
combos = (await getCombos()) as ComboRecord[];
combos = allCombos;
}
const response: ComboHealthResponse = {
timeRange: range,
combos: combos
.map((combo) => buildComboHealth(combo, since))
.map((combo) => buildComboHealth(combo, since, allCombos))
.filter((combo): combo is ComboHealthMetrics => combo !== null),
};

27
src/app/api/v1beta/models/route.ts
View File

@@ -1,6 +1,8 @@
import { CORS_ORIGIN } from "@/shared/utils/cors";
import { PROVIDER_MODELS } from "@/shared/constants/models";
import { getAllCustomModels, getSyncedAvailableModels } from "@/lib/db/models";
import { getResolvedModelCapabilities } from "@/lib/modelCapabilities";
import { getSyncedCapabilities } from "@/lib/modelsDevSync";
/**
* Handle CORS preflight
@@ -21,18 +23,21 @@ export async function OPTIONS() {
*/
export async function GET() {
try {
getSyncedCapabilities();
const models = [];
// Built-in models (hardcoded defaults)
for (const [provider, providerModels] of Object.entries(PROVIDER_MODELS)) {
for (const model of providerModels) {
const resolved = getResolvedModelCapabilities({ provider, model: model.id });
models.push({
name: `models/${provider}/${model.id}`,
displayName: model.name || model.id,
description: `${provider} model: ${model.name || model.id}`,
supportedGenerationMethods: ["generateContent"],
inputTokenLimit: 128000,
outputTokenLimit: 8192,
inputTokenLimit: resolved.maxInputTokens || resolved.contextWindow || 128000,
outputTokenLimit: resolved.maxOutputTokens || 8192,
...(resolved.supportsThinking === true ? { thinking: true } : {}),
});
}
}
@@ -76,14 +81,26 @@ export async function GET() {
continue;
const m = model as Record<string, unknown>;
if (m.isHidden === true) continue;
const resolved = getResolvedModelCapabilities({
provider: providerId,
model: String(m.id),
});
models.push({
name: `models/${providerId}/${m.id}`,
displayName: m.name || m.id,
...(typeof m.description === "string" ? { description: m.description } : {}),
supportedGenerationMethods: ["generateContent"],
inputTokenLimit: typeof m.inputTokenLimit === "number" ? m.inputTokenLimit : 128000,
outputTokenLimit: typeof m.outputTokenLimit === "number" ? m.outputTokenLimit : 8192,
...(m.supportsThinking === true ? { thinking: true } : {}),
inputTokenLimit:
typeof m.inputTokenLimit === "number"
? m.inputTokenLimit
: resolved.maxInputTokens || resolved.contextWindow || 128000,
outputTokenLimit:
typeof m.outputTokenLimit === "number"
? m.outputTokenLimit
: resolved.maxOutputTokens || 8192,
...(m.supportsThinking === true || resolved.supportsThinking === true
? { thinking: true }
: {}),
});
}
}

1
src/app/globals.css
View File

@@ -4,7 +4,6 @@
(e.g. Next.js route groups like "(dashboard)"). Explicit @source
directives ensure all utility classes in route groups are included. */
@source "../app/(dashboard)";
@source "../../open-sse";
@source not "../../*.sqlite*";
@source not "../../.claude*";
@source not "../../.claude-memory";

12
src/domain/comboResolver.ts
View File

@@ -11,6 +11,7 @@
/**
* @typedef {import('./types.js').Combo} Combo
*/
import { getComboStepTarget, getComboStepWeight } from "@/lib/combos/steps";
/** @type {Map<string, number>} Persistent round-robin counters per combo */
const roundRobinCounters = new Map();
@@ -30,7 +31,12 @@ export function resolveComboModel(combo: any, context: any = {}) {
}
// Normalize models to { model, weight } format
const normalized = models.map((m) => (typeof m === "string" ? { model: m, weight: 1 } : m));
const normalized = models
.map((entry) => ({
model: getComboStepTarget(entry) || "",
weight: getComboStepWeight(entry) || 1,
}))
.filter((entry) => entry.model);
const strategy = combo.strategy || "priority";
@@ -93,6 +99,8 @@ export function resolveComboModel(combo: any, context: any = {}) {
* @returns {string[]} Remaining models in order
*/
export function getComboFallbacks(combo, primaryIndex) {
const models = (combo.models || []).map((m) => (typeof m === "string" ? m : m.model));
const models = (combo.models || [])
.map((entry) => getComboStepTarget(entry))
.filter((entry): entry is string => !!entry);
return [...models.slice(primaryIndex + 1), ...models.slice(0, primaryIndex)];
}

153
src/lib/combos/builderDraft.ts Normal file
View File

@@ -0,0 +1,153 @@
import type { ComboModelStep } from "@/lib/combos/steps";
type JsonRecord = Record<string, unknown>;
export const COMBO_BUILDER_AUTO_CONNECTION = "__auto__";
export const COMBO_BUILDER_STAGES = ["basics", "steps", "strategy", "review"] as const;
export type ComboBuilderStage = (typeof COMBO_BUILDER_STAGES)[number];
function isRecord(value: unknown): value is JsonRecord {
return !!value && typeof value === "object" && !Array.isArray(value);
}
function toTrimmedString(value: unknown): string | null {
return typeof value === "string" && value.trim().length > 0 ? value.trim() : null;
}
export function parseQualifiedModel(
value: unknown
): { providerId: string; modelId: string } | null {
const qualifiedModel = toTrimmedString(value);
if (!qualifiedModel) return null;
const firstSlashIndex = qualifiedModel.indexOf("/");
if (firstSlashIndex <= 0 || firstSlashIndex >= qualifiedModel.length - 1) return null;
return {
providerId: qualifiedModel.slice(0, firstSlashIndex),
modelId: qualifiedModel.slice(firstSlashIndex + 1),
};
}
export function getComboDraftTarget(entry: unknown): string | null {
if (typeof entry === "string") return toTrimmedString(entry);
if (!isRecord(entry)) return null;
if (entry.kind === "combo-ref") return toTrimmedString(entry.comboName);
return toTrimmedString(entry.model);
}
export function buildPrecisionComboModelStep({
providerId,
modelId,
connectionId = null,
connectionLabel,
weight = 0,
}: {
providerId: string;
modelId: string;
connectionId?: string | null;
connectionLabel?: string | null;
weight?: number;
}): ComboModelStep {
const normalizedProviderId = toTrimmedString(providerId) || "provider";
const normalizedModelId = toTrimmedString(modelId) || "model";
const normalizedConnectionId = toTrimmedString(connectionId);
const normalizedConnectionLabel = toTrimmedString(connectionLabel);
return {
kind: "model",
providerId: normalizedProviderId,
model: `${normalizedProviderId}/${normalizedModelId}`,
...(normalizedConnectionId ? { connectionId: normalizedConnectionId } : {}),
...(normalizedConnectionLabel ? { label: normalizedConnectionLabel } : {}),
weight: Number.isFinite(weight) ? Math.max(0, Math.min(100, Number(weight))) : 0,
};
}
export function getExactModelStepSignature(entry: unknown): string | null {
if (!isRecord(entry) || entry.kind === "combo-ref") return null;
const modelValue = toTrimmedString(entry.model);
const parsed = parseQualifiedModel(modelValue);
if (!parsed) return null;
const normalizedProviderId = toTrimmedString(entry.providerId) || parsed.providerId;
const normalizedConnectionId =
toTrimmedString(entry.connectionId) || COMBO_BUILDER_AUTO_CONNECTION;
return `model:${normalizedProviderId}:${parsed.modelId}:${normalizedConnectionId}`;
}
export function hasExactModelStepDuplicate(entries: unknown[], candidate: unknown): boolean {
const candidateSignature = getExactModelStepSignature(candidate);
if (!candidateSignature) return false;
return entries.some((entry) => getExactModelStepSignature(entry) === candidateSignature);
}
export function findNextSuggestedConnectionId(
entries: unknown[],
providerId: string,
modelId: string,
connections: Array<{ id?: string | null }> = []
): string {
for (const connection of connections) {
const connectionId = toTrimmedString(connection?.id);
if (!connectionId) continue;
const step = buildPrecisionComboModelStep({
providerId,
modelId,
connectionId,
});
if (!hasExactModelStepDuplicate(entries, step)) {
return connectionId;
}
}
return COMBO_BUILDER_AUTO_CONNECTION;
}
export function getComboBuilderStageChecks({
name,
nameError,
modelsCount,
hasInvalidWeightedTotal,
hasCostOptimizedWithoutPricing,
}: {
name: string;
nameError?: string | null;
modelsCount: number;
hasInvalidWeightedTotal?: boolean;
hasCostOptimizedWithoutPricing?: boolean;
}) {
return {
basics: Boolean(toTrimmedString(name)) && !toTrimmedString(nameError),
steps: modelsCount > 0,
strategy: !Boolean(hasInvalidWeightedTotal) && !Boolean(hasCostOptimizedWithoutPricing),
review: false,
};
}
export function canAccessComboBuilderStage(
stage: ComboBuilderStage,
checks: ReturnType<typeof getComboBuilderStageChecks>
): boolean {
if (stage === "basics") return true;
if (stage === "steps") return checks.basics;
if (stage === "strategy") return checks.basics && checks.steps;
if (stage === "review") return checks.basics && checks.steps;
return false;
}
export function getNextComboBuilderStage(stage: ComboBuilderStage): ComboBuilderStage {
const stageIndex = COMBO_BUILDER_STAGES.indexOf(stage);
if (stageIndex === -1 || stageIndex >= COMBO_BUILDER_STAGES.length - 1) {
return "review";
}
return COMBO_BUILDER_STAGES[stageIndex + 1];
}
export function getPreviousComboBuilderStage(stage: ComboBuilderStage): ComboBuilderStage {
const stageIndex = COMBO_BUILDER_STAGES.indexOf(stage);
if (stageIndex <= 0) return "basics";
return COMBO_BUILDER_STAGES[stageIndex - 1];
}

544
src/lib/combos/builderOptions.ts Normal file
View File

@@ -0,0 +1,544 @@
import {
getAllCustomModels,
getAllSyncedAvailableModels,
getCombos,
getModelIsHidden,
getProviderConnections,
getProviderNodes,
} from "@/lib/localDb";
import { getAccountDisplayName, getProviderDisplayName } from "@/lib/display/names";
import { getCompatibleFallbackModels } from "@/lib/providers/managedAvailableModels";
import { getResolvedModelCapabilities } from "@/lib/modelCapabilities";
import { getSyncedCapabilities } from "@/lib/modelsDevSync";
import { getModelsByProviderId } from "@/shared/constants/models";
import {
AI_PROVIDERS,
isAnthropicCompatibleProvider,
isClaudeCodeCompatibleProvider,
isOpenAICompatibleProvider,
} from "@/shared/constants/providers";
import type { RegistryModel } from "@omniroute/open-sse/config/providerRegistry.ts";
type JsonRecord = Record<string, unknown>;
type BuilderModelSource = "api-sync" | "system" | "custom" | "fallback";
type BuilderConnectionStatus = "active" | "inactive" | "rate-limited" | "error";
type ProviderVisual = { icon: string; color: string; source: "system" | "provider-node" };
type CustomModelLike = {
id?: string;
name?: string;
source?: string;
apiFormat?: string;
supportedEndpoints?: string[];
inputTokenLimit?: number;
outputTokenLimit?: number;
supportsThinking?: boolean;
isHidden?: boolean;
};
type SyncedModelLike = {
id?: string;
name?: string;
source?: string;
supportedEndpoints?: string[];
inputTokenLimit?: number;
outputTokenLimit?: number;
description?: string;
supportsThinking?: boolean;
};
type ProviderConnectionLike = {
id?: string;
provider?: string;
authType?: string;
name?: string;
displayName?: string;
email?: string;
priority?: number;
isActive?: boolean;
defaultModel?: string;
rateLimitedUntil?: number | null;
lastError?: string | null;
lastTested?: string | null;
updatedAt?: string | null;
testStatus?: string | null;
};
type ProviderNodeLike = {
id?: string;
type?: string;
name?: string;
prefix?: string;
};
export interface ComboBuilderModelOption {
id: string;
qualifiedModel: string;
name: string;
source: BuilderModelSource;
sources: BuilderModelSource[];
supportedEndpoints?: string[];
apiFormat?: string;
contextLength?: number;
outputTokenLimit?: number;
supportsThinking?: boolean;
}
export interface ComboBuilderConnectionOption {
id: string;
label: string;
type: string;
status: BuilderConnectionStatus;
priority: number;
isActive: boolean;
defaultModel?: string | null;
rateLimitedUntil?: number | null;
lastError?: string | null;
lastTested?: string | null;
}
export interface ComboBuilderProviderOption {
providerId: string;
providerType: string;
displayName: string;
alias: string;
prefix?: string | null;
icon: string;
color: string;
source: "system" | "provider-node";
acceptsArbitraryModel: boolean;
connectionCount: number;
activeConnectionCount: number;
modelCount: number;
connections: ComboBuilderConnectionOption[];
models: ComboBuilderModelOption[];
}
export interface ComboBuilderComboRefOption {
id: string;
name: string;
strategy: string;
stepCount: number;
version: number;
sortOrder?: number;
}
export interface ComboBuilderOptionsPayload {
schemaVersion: number;
generatedAt: string;
providers: ComboBuilderProviderOption[];
comboRefs: ComboBuilderComboRefOption[];
}
function toStringOrNull(value: unknown): string | null {
return typeof value === "string" && value.trim().length > 0 ? value.trim() : null;
}
function toNumberOrNull(value: unknown): number | null {
if (typeof value === "number" && Number.isFinite(value)) return value;
if (typeof value === "string" && value.trim().length > 0) {
const parsed = Number(value);
return Number.isFinite(parsed) ? parsed : null;
}
return null;
}
function toStringArray(value: unknown): string[] | undefined {
if (!Array.isArray(value)) return undefined;
const normalized = value
.map((item) => toStringOrNull(item))
.filter((item): item is string => Boolean(item));
return normalized.length > 0 ? normalized : undefined;
}
function isChatCapable(supportedEndpoints: string[] | undefined): boolean {
if (!supportedEndpoints || supportedEndpoints.length === 0) return true;
return supportedEndpoints.includes("chat");
}
function getSourcePriority(source: BuilderModelSource): number {
switch (source) {
case "api-sync":
return 0;
case "system":
return 1;
case "custom":
return 2;
case "fallback":
return 3;
default:
return 99;
}
}
function getCompatibleProviderVisual(providerNodeType: string | null): ProviderVisual {
if (providerNodeType === "openai-compatible") {
return { icon: "api", color: "#10A37F", source: "provider-node" };
}
if (providerNodeType === "anthropic-compatible") {
return { icon: "api", color: "#D97757", source: "provider-node" };
}
if (providerNodeType === "anthropic-compatible-cc") {
return { icon: "smart_toy", color: "#D97757", source: "provider-node" };
}
return { icon: "api", color: "#6B7280", source: "provider-node" };
}
function getProviderVisual(
providerId: string,
providerNode: ProviderNodeLike | null
): ProviderVisual & { alias: string; providerType: string } {
const providerEntry = AI_PROVIDERS[providerId];
if (providerEntry) {
return {
alias: providerEntry.alias || providerEntry.id,
providerType: providerEntry.id,
icon: providerEntry.icon || "hub",
color: providerEntry.color || "#6B7280",
source: "system",
};
}
const providerNodeType = toStringOrNull(providerNode?.type);
const compatibleVisual = getCompatibleProviderVisual(providerNodeType);
return {
alias: toStringOrNull(providerNode?.prefix) || providerId,
providerType: providerNodeType || providerId,
...compatibleVisual,
};
}
function deriveConnectionStatus(connection: ProviderConnectionLike): BuilderConnectionStatus {
if (connection.isActive === false) return "inactive";
const rateLimitedUntil = toNumberOrNull(connection.rateLimitedUntil);
if (typeof rateLimitedUntil === "number" && rateLimitedUntil > Date.now()) {
return "rate-limited";
}
if (typeof connection.testStatus === "string" && /error|fail/i.test(connection.testStatus)) {
return "error";
}
return "active";
}
function buildConnectionOption(
connection: ProviderConnectionLike
): ComboBuilderConnectionOption | null {
const id = toStringOrNull(connection.id);
if (!id) return null;
return {
id,
label: getAccountDisplayName(connection),
type: toStringOrNull(connection.authType) || "unknown",
status: deriveConnectionStatus(connection),
priority: typeof connection.priority === "number" ? connection.priority : 0,
isActive: connection.isActive !== false,
defaultModel: toStringOrNull(connection.defaultModel),
rateLimitedUntil: toNumberOrNull(connection.rateLimitedUntil),
lastError: toStringOrNull(connection.lastError),
lastTested: toStringOrNull(connection.lastTested),
};
}
function addModelOption(
modelMap: Map<string, ComboBuilderModelOption>,
providerId: string,
input: {
id: string | null;
name?: string | null;
source: BuilderModelSource;
supportedEndpoints?: string[];
apiFormat?: string | null;
contextLength?: number | null;
outputTokenLimit?: number | null;
supportsThinking?: boolean;
}
) {
const modelId = toStringOrNull(input.id);
if (!modelId) return;
if (getModelIsHidden(providerId, modelId)) return;
if (!isChatCapable(input.supportedEndpoints)) return;
const nextSourcePriority = getSourcePriority(input.source);
const existing = modelMap.get(modelId);
if (!existing) {
modelMap.set(modelId, {
id: modelId,
qualifiedModel: `${providerId}/${modelId}`,
name: toStringOrNull(input.name) || modelId,
source: input.source,
sources: [input.source],
...(input.supportedEndpoints && input.supportedEndpoints.length > 0
? { supportedEndpoints: input.supportedEndpoints }
: {}),
...(toStringOrNull(input.apiFormat) ? { apiFormat: input.apiFormat || undefined } : {}),
...(typeof input.contextLength === "number" ? { contextLength: input.contextLength } : {}),
...(typeof input.outputTokenLimit === "number"
? { outputTokenLimit: input.outputTokenLimit }
: {}),
...(typeof input.supportsThinking === "boolean"
? { supportsThinking: input.supportsThinking }
: {}),
});
return;
}
const existingPriority = getSourcePriority(existing.source);
const mergedSources = new Set<BuilderModelSource>([...existing.sources, input.source]);
if (nextSourcePriority < existingPriority) {
existing.source = input.source;
}
if (!existing.name || existing.name === existing.id) {
existing.name = toStringOrNull(input.name) || existing.name;
}
if (!existing.supportedEndpoints && input.supportedEndpoints?.length) {
existing.supportedEndpoints = input.supportedEndpoints;
}
if (!existing.apiFormat && toStringOrNull(input.apiFormat)) {
existing.apiFormat = input.apiFormat || undefined;
}
if (existing.contextLength == null && typeof input.contextLength === "number") {
existing.contextLength = input.contextLength;
}
if (existing.outputTokenLimit == null && typeof input.outputTokenLimit === "number") {
existing.outputTokenLimit = input.outputTokenLimit;
}
if (existing.supportsThinking == null && typeof input.supportsThinking === "boolean") {
existing.supportsThinking = input.supportsThinking;
}
existing.sources = Array.from(mergedSources).sort(
(left, right) => getSourcePriority(left) - getSourcePriority(right)
);
}
function compareConnections(
left: ComboBuilderConnectionOption,
right: ComboBuilderConnectionOption
): number {
if (left.isActive !== right.isActive) return left.isActive ? -1 : 1;
if (left.priority !== right.priority) return left.priority - right.priority;
return left.label.localeCompare(right.label, undefined, { sensitivity: "base" });
}
function compareModels(left: ComboBuilderModelOption, right: ComboBuilderModelOption): number {
const sourceDelta = getSourcePriority(left.source) - getSourcePriority(right.source);
if (sourceDelta !== 0) return sourceDelta;
const nameDelta = left.name.localeCompare(right.name, undefined, { sensitivity: "base" });
if (nameDelta !== 0) return nameDelta;
return left.id.localeCompare(right.id, undefined, { sensitivity: "base" });
}
function compareProviders(
left: ComboBuilderProviderOption,
right: ComboBuilderProviderOption
): number {
if (left.activeConnectionCount !== right.activeConnectionCount) {
return right.activeConnectionCount - left.activeConnectionCount;
}
return left.displayName.localeCompare(right.displayName, undefined, { sensitivity: "base" });
}
function normalizeCustomModels(raw: unknown): CustomModelLike[] {
return Array.isArray(raw)
? raw.filter(
(model): model is CustomModelLike =>
Boolean(model) && typeof model === "object" && !Array.isArray(model)
)
: [];
}
function normalizeSyncedModels(raw: unknown): SyncedModelLike[] {
return Array.isArray(raw)
? raw.filter(
(model): model is SyncedModelLike =>
Boolean(model) && typeof model === "object" && !Array.isArray(model)
)
: [];
}
export async function getComboBuilderOptions(): Promise<ComboBuilderOptionsPayload> {
getSyncedCapabilities();
const [connections, providerNodes, customModelsMap, syncedModelsMap, combos] = await Promise.all([
getProviderConnections(),
getProviderNodes(),
getAllCustomModels(),
getAllSyncedAvailableModels(),
getCombos(),
]);
const providerNodeMap = new Map<string, ProviderNodeLike>();
for (const providerNode of providerNodes as ProviderNodeLike[]) {
const providerId = toStringOrNull(providerNode.id);
if (!providerId) continue;
providerNodeMap.set(providerId, providerNode);
}
const connectionsByProvider = new Map<string, ProviderConnectionLike[]>();
for (const connection of connections as ProviderConnectionLike[]) {
const providerId = toStringOrNull(connection.provider);
if (!providerId) continue;
const list = connectionsByProvider.get(providerId) || [];
list.push(connection);
connectionsByProvider.set(providerId, list);
}
const providers: ComboBuilderProviderOption[] = [];
for (const [providerId, providerConnections] of connectionsByProvider) {
const providerNode = providerNodeMap.get(providerId) || null;
const providerVisual = getProviderVisual(providerId, providerNode);
const modelMap = new Map<string, ComboBuilderModelOption>();
const builtInModels = getModelsByProviderId(providerId);
const syncedModels = normalizeSyncedModels(
(syncedModelsMap as Record<string, unknown>)[providerId]
);
const customModels = normalizeCustomModels(
(customModelsMap as Record<string, unknown>)[providerId]
);
const fallbackModels = getCompatibleFallbackModels(providerId, builtInModels);
const acceptsArbitraryModel =
Boolean((AI_PROVIDERS[providerId] as JsonRecord | undefined)?.passthroughModels) ||
isOpenAICompatibleProvider(providerId) ||
isAnthropicCompatibleProvider(providerId) ||
isClaudeCodeCompatibleProvider(providerId);
for (const model of syncedModels) {
const resolved = getResolvedModelCapabilities({
provider: providerId,
model: toStringOrNull(model.id),
});
addModelOption(modelMap, providerId, {
id: toStringOrNull(model.id),
name: toStringOrNull(model.name),
source: "api-sync",
supportedEndpoints: toStringArray(model.supportedEndpoints),
contextLength: toNumberOrNull(model.inputTokenLimit) ?? resolved.contextWindow,
outputTokenLimit: toNumberOrNull(model.outputTokenLimit) ?? resolved.maxOutputTokens,
supportsThinking:
typeof model.supportsThinking === "boolean"
? model.supportsThinking
: (resolved.supportsThinking ?? undefined),
});
}
for (const model of builtInModels as RegistryModel[]) {
const resolved = getResolvedModelCapabilities({
provider: providerId,
model: toStringOrNull(model.id),
});
addModelOption(modelMap, providerId, {
id: toStringOrNull(model.id),
name: toStringOrNull(model.name),
source: "system",
contextLength: toNumberOrNull(model.contextLength) ?? resolved.contextWindow,
outputTokenLimit: resolved.maxOutputTokens,
supportsThinking: resolved.supportsThinking ?? undefined,
});
}
for (const model of customModels) {
if (model.isHidden === true) continue;
const source =
toStringOrNull(model.source) === "api-sync" ? "api-sync" : ("custom" as BuilderModelSource);
const resolved = getResolvedModelCapabilities({
provider: providerId,
model: toStringOrNull(model.id),
});
addModelOption(modelMap, providerId, {
id: toStringOrNull(model.id),
name: toStringOrNull(model.name),
source,
supportedEndpoints: toStringArray(model.supportedEndpoints),
apiFormat: toStringOrNull(model.apiFormat),
contextLength: toNumberOrNull(model.inputTokenLimit) ?? resolved.contextWindow,
outputTokenLimit: toNumberOrNull(model.outputTokenLimit) ?? resolved.maxOutputTokens,
supportsThinking:
typeof model.supportsThinking === "boolean"
? model.supportsThinking
: (resolved.supportsThinking ?? undefined),
});
}
if (Array.isArray(fallbackModels)) {
for (const model of fallbackModels) {
const resolved = getResolvedModelCapabilities({
provider: providerId,
model: toStringOrNull(model.id),
});
addModelOption(modelMap, providerId, {
id: toStringOrNull(model.id),
name: toStringOrNull(model.name),
source: "fallback",
contextLength:
typeof (model as { contextLength?: number }).contextLength === "number"
? (model as { contextLength?: number }).contextLength || null
: resolved.contextWindow,
outputTokenLimit: resolved.maxOutputTokens,
supportsThinking: resolved.supportsThinking ?? undefined,
});
}
}
const normalizedConnections = providerConnections
.map((connection) => buildConnectionOption(connection))
.filter((connection): connection is ComboBuilderConnectionOption => Boolean(connection))
.sort(compareConnections);
const activeConnectionCount = normalizedConnections.filter(
(connection) => connection.isActive
).length;
const displayName = (providerEntryName(providerId) ||
getProviderDisplayName(providerId, providerNode) ||
providerId) as string;
providers.push({
providerId,
providerType: providerVisual.providerType,
displayName,
alias: providerVisual.alias,
prefix: toStringOrNull(providerNode?.prefix),
icon: providerVisual.icon,
color: providerVisual.color,
source: providerVisual.source,
acceptsArbitraryModel,
connectionCount: normalizedConnections.length,
activeConnectionCount,
modelCount: modelMap.size,
connections: normalizedConnections,
models: Array.from(modelMap.values()).sort(compareModels),
});
}
const comboRefs = (combos as JsonRecord[])
.filter((combo) => combo.isHidden !== true && combo.isActive !== false)
.map((combo) => ({
id: toStringOrNull(combo.id) || toStringOrNull(combo.name) || "combo",
name: toStringOrNull(combo.name) || "combo",
strategy: toStringOrNull(combo.strategy) || "priority",
stepCount: Array.isArray(combo.models) ? combo.models.length : 0,
version: typeof combo.version === "number" ? combo.version : 2,
...(typeof combo.sortOrder === "number" ? { sortOrder: combo.sortOrder } : {}),
}))
.sort((left, right) => {
const leftSort =
typeof left.sortOrder === "number" ? left.sortOrder : Number.MAX_SAFE_INTEGER;
const rightSort =
typeof right.sortOrder === "number" ? right.sortOrder : Number.MAX_SAFE_INTEGER;
if (leftSort !== rightSort) return leftSort - rightSort;
return left.name.localeCompare(right.name, undefined, { sensitivity: "base" });
});
return {
schemaVersion: 2,
generatedAt: new Date().toISOString(),
providers: providers.sort(compareProviders),
comboRefs,
};
}
function providerEntryName(providerId: string): string | null {
const providerEntry = AI_PROVIDERS[providerId] as { name?: string } | undefined;
return toStringOrNull(providerEntry?.name);
}

200
src/lib/combos/compositeTiers.ts Normal file
View File

@@ -0,0 +1,200 @@
import { normalizeComboModels } from "@/lib/combos/steps";
type JsonRecord = Record<string, unknown>;
type ValidationErrorDetail = {
field: string;
message: string;
};
type CompositeTierValidationFailure = {
success: false;
error: {
message: string;
details: ValidationErrorDetail[];
};
};
type CompositeTierValidationSuccess = {
success: true;
};
export type CompositeTierValidationResult =
| CompositeTierValidationFailure
| CompositeTierValidationSuccess;
function isRecord(value: unknown): value is JsonRecord {
return !!value && typeof value === "object" && !Array.isArray(value);
}
function toTrimmedString(value: unknown): string | null {
return typeof value === "string" && value.trim().length > 0 ? value.trim() : null;
}
function createFailure(details: ValidationErrorDetail[]): CompositeTierValidationFailure {
return {
success: false,
error: {
message: "Invalid composite tiers",
details,
},
};
}
export function validateCompositeTiersConfig(combo: {
name?: unknown;
models?: unknown;
config?: unknown;
}): CompositeTierValidationResult {
if (!isRecord(combo.config)) {
return { success: true };
}
const compositeTiers = combo.config.compositeTiers;
if (compositeTiers === undefined || compositeTiers === null) {
return { success: true };
}
if (!isRecord(compositeTiers)) {
return createFailure([
{
field: "config.compositeTiers",
message: "compositeTiers must be an object",
},
]);
}
const defaultTier = toTrimmedString(compositeTiers.defaultTier);
const tiers = isRecord(compositeTiers.tiers) ? compositeTiers.tiers : null;
const details: ValidationErrorDetail[] = [];
if (!defaultTier) {
details.push({
field: "config.compositeTiers.defaultTier",
message: "defaultTier is required",
});
}
if (!tiers || Object.keys(tiers).length === 0) {
details.push({
field: "config.compositeTiers.tiers",
message: "tiers must define at least one tier",
});
return createFailure(details);
}
const normalizedSteps = normalizeComboModels(combo.models, {
comboName: toTrimmedString(combo.name),
});
const stepIds = new Set(
normalizedSteps
.map((step) => toTrimmedString(step.id))
.filter((value): value is string => !!value)
);
const tierEntries = new Map<string, { stepId: string; fallbackTier: string | null }>();
const stepIdOwners = new Map<string, string>();
for (const [rawTierName, rawTierValue] of Object.entries(tiers)) {
const tierName = toTrimmedString(rawTierName);
const fieldBase = `config.compositeTiers.tiers.${rawTierName}`;
if (!tierName) {
details.push({
field: fieldBase,
message: "tier name must be a non-empty string",
});
continue;
}
if (!isRecord(rawTierValue)) {
details.push({
field: fieldBase,
message: "tier config must be an object",
});
continue;
}
const stepId = toTrimmedString(rawTierValue.stepId);
const fallbackTier = toTrimmedString(rawTierValue.fallbackTier);
if (!stepId) {
details.push({
field: `${fieldBase}.stepId`,
message: "stepId is required",
});
continue;
}
if (!stepIds.has(stepId)) {
details.push({
field: `${fieldBase}.stepId`,
message: `stepId "${stepId}" does not exist in combo.models`,
});
}
const previousTierForStep = stepIdOwners.get(stepId);
if (previousTierForStep && previousTierForStep !== tierName) {
details.push({
field: `${fieldBase}.stepId`,
message: `stepId "${stepId}" is already assigned to tier "${previousTierForStep}"`,
});
} else {
stepIdOwners.set(stepId, tierName);
}
if (fallbackTier && fallbackTier === tierName) {
details.push({
field: `${fieldBase}.fallbackTier`,
message: "fallbackTier cannot reference the same tier",
});
}
tierEntries.set(tierName, {
stepId,
fallbackTier,
});
}
if (defaultTier && !tierEntries.has(defaultTier)) {
details.push({
field: "config.compositeTiers.defaultTier",
message: `defaultTier "${defaultTier}" does not exist in tiers`,
});
}
for (const [tierName, entry] of tierEntries.entries()) {
if (entry.fallbackTier && !tierEntries.has(entry.fallbackTier)) {
details.push({
field: `config.compositeTiers.tiers.${tierName}.fallbackTier`,
message: `fallbackTier "${entry.fallbackTier}" does not exist in tiers`,
});
}
}
const visitState = new Map<string, "visiting" | "visited">();
function visit(tierName: string, path: string[]) {
const state = visitState.get(tierName);
if (state === "visited") return;
if (state === "visiting") {
details.push({
field: `config.compositeTiers.tiers.${tierName}.fallbackTier`,
message: `fallbackTier cycle detected: ${[...path, tierName].join(" -> ")}`,
});
return;
}
visitState.set(tierName, "visiting");
const fallbackTier = tierEntries.get(tierName)?.fallbackTier;
if (fallbackTier && tierEntries.has(fallbackTier)) {
visit(fallbackTier, [...path, tierName]);
}
visitState.set(tierName, "visited");
}
for (const tierName of tierEntries.keys()) {
visit(tierName, []);
}
return details.length > 0 ? createFailure(details) : { success: true };
}

318
src/lib/combos/steps.ts Normal file
View File

@@ -0,0 +1,318 @@
type JsonRecord = Record<string, unknown>;
export const COMBO_SCHEMA_VERSION = 2;
export interface ComboModelStep {
id: string;
kind: "model";
model: string;
providerId?: string | null;
connectionId?: string | null;
weight: number;
label?: string;
tags?: string[];
}
export interface ComboRefStep {
id: string;
kind: "combo-ref";
comboName: string;
weight: number;
label?: string;
}
export type ComboStep = ComboModelStep | ComboRefStep;
type ComboCollectionLike =
| Array<{ name?: unknown } | string>
| { combos?: Array<{ name?: unknown }> }
| Iterable<string>
| null
| undefined;
interface NormalizeComboStepOptions {
comboName?: string | null;
index?: number;
allCombos?: ComboCollectionLike;
}
interface NormalizeComboRecordOptions {
allCombos?: ComboCollectionLike;
}
function isRecord(value: unknown): value is JsonRecord {
return !!value && typeof value === "object" && !Array.isArray(value);
}
function toTrimmedString(value: unknown): string | null {
return typeof value === "string" && value.trim().length > 0 ? value.trim() : null;
}
function toWeight(value: unknown): number {
const parsed =
typeof value === "number"
? value
: typeof value === "string" && value.trim().length > 0
? Number(value)
: 0;
if (!Number.isFinite(parsed)) return 0;
return Math.max(0, Math.min(100, parsed));
}
function collectComboNames(allCombos: ComboCollectionLike): Set<string> {
const names = new Set<string>();
if (!allCombos) return names;
if (
!Array.isArray(allCombos) &&
typeof (allCombos as { [Symbol.iterator]?: unknown })[Symbol.iterator] === "function" &&
!(isRecord(allCombos) && Array.isArray(allCombos.combos))
) {
for (const value of allCombos as Iterable<string>) {
const name = toTrimmedString(value);
if (name) names.add(name);
}
return names;
}
const combosFromRecord =
isRecord(allCombos) && Array.isArray(allCombos.combos) ? allCombos.combos : [];
const combos = Array.isArray(allCombos) ? allCombos : combosFromRecord;
for (const combo of combos) {
const name = typeof combo === "string" ? toTrimmedString(combo) : toTrimmedString(combo?.name);
if (name) names.add(name);
}
return names;
}
function parseProviderId(model: string): string | null {
const trimmed = model.trim();
const slashIndex = trimmed.indexOf("/");
if (slashIndex <= 0) return null;
const providerId = trimmed.slice(0, slashIndex).trim();
return providerId.length > 0 ? providerId : null;
}
function toFullModelString(model: string, providerId?: string | null): string {
const trimmedModel = model.trim();
if (trimmedModel.includes("/")) return trimmedModel;
const normalizedProviderId = toTrimmedString(providerId);
return normalizedProviderId ? `${normalizedProviderId}/${trimmedModel}` : trimmedModel;
}
function slugify(value: string): string {
const slug = value
.toLowerCase()
.replace(/[^a-z0-9]+/g, "-")
.replace(/^-+|-+$/g, "");
return slug.length > 0 ? slug : "step";
}
function buildStepId(
kind: ComboStep["kind"],
comboName: string | null,
index: number,
seed: string
) {
const parts = [
slugify(comboName || "combo"),
kind === "combo-ref" ? "ref" : "model",
String(index + 1),
slugify(seed),
];
return parts.join("-").slice(0, 200);
}
function shouldTreatAsComboRef(
target: string,
providerId: string | null,
options: NormalizeComboStepOptions
): boolean {
if (providerId) return false;
const comboNames = collectComboNames(options.allCombos);
return comboNames.has(target) || target === toTrimmedString(options.comboName);
}
export function isComboRefStep(value: unknown): value is ComboRefStep {
return isRecord(value) && value.kind === "combo-ref" && !!toTrimmedString(value.comboName);
}
export function isComboModelStep(value: unknown): value is ComboModelStep {
return isRecord(value) && value.kind === "model" && !!toTrimmedString(value.model);
}
export function getComboStepWeight(value: unknown): number {
if (typeof value === "string") return 0;
if (!isRecord(value)) return 0;
return toWeight(value.weight);
}
export function getComboModelString(value: unknown): string | null {
if (typeof value === "string") return toTrimmedString(value);
if (!isRecord(value) || value.kind === "combo-ref") return null;
const rawModel = toTrimmedString(value.model);
if (!rawModel) return null;
const providerId =
toTrimmedString(value.providerId) ||
toTrimmedString(value.provider) ||
parseProviderId(rawModel);
return toFullModelString(rawModel, providerId);
}
export function getComboModelProvider(value: unknown): string | null {
if (typeof value === "string") return parseProviderId(value);
if (!isRecord(value) || value.kind === "combo-ref") return null;
return (
toTrimmedString(value.providerId) ||
toTrimmedString(value.provider) ||
parseProviderId(toTrimmedString(value.model) || "")
);
}
export function getComboStepTarget(
value: unknown,
options: NormalizeComboStepOptions = {}
): string | null {
if (typeof value === "string") {
const target = toTrimmedString(value);
if (!target) return null;
return shouldTreatAsComboRef(target, null, options) ? target : target;
}
if (!isRecord(value)) return null;
if (value.kind === "combo-ref") return toTrimmedString(value.comboName);
const rawModel = toTrimmedString(value.model);
if (!rawModel) return null;
const isExplicitModel = value.kind === "model";
const providerId =
toTrimmedString(value.providerId) ||
toTrimmedString(value.provider) ||
parseProviderId(rawModel);
if (!isExplicitModel && shouldTreatAsComboRef(rawModel, providerId, options)) {
return rawModel;
}
return toFullModelString(rawModel, providerId);
}
export function normalizeComboStep(
value: unknown,
options: NormalizeComboStepOptions = {}
): ComboStep | null {
const comboName = toTrimmedString(options.comboName);
const index = typeof options.index === "number" ? options.index : 0;
if (typeof value === "string") {
const target = toTrimmedString(value);
if (!target) return null;
if (shouldTreatAsComboRef(target, null, options)) {
return {
id: buildStepId("combo-ref", comboName, index, target),
kind: "combo-ref",
comboName: target,
weight: 0,
};
}
const providerId = parseProviderId(target);
return {
id: buildStepId("model", comboName, index, target),
kind: "model",
model: target,
...(providerId ? { providerId } : {}),
weight: 0,
};
}
if (!isRecord(value)) return null;
const explicitId = toTrimmedString(value.id);
const weight = toWeight(value.weight);
const label = toTrimmedString(value.label);
if (value.kind === "combo-ref") {
const comboRefName = toTrimmedString(value.comboName);
if (!comboRefName) return null;
return {
id: explicitId || buildStepId("combo-ref", comboName, index, comboRefName),
kind: "combo-ref",
comboName: comboRefName,
weight,
...(label ? { label } : {}),
};
}
const rawModel = toTrimmedString(value.model);
if (!rawModel) return null;
const isExplicitModel = value.kind === "model";
const providerId =
toTrimmedString(value.providerId) ||
toTrimmedString(value.provider) ||
parseProviderId(rawModel);
if (!isExplicitModel && shouldTreatAsComboRef(rawModel, providerId, options)) {
return {
id: explicitId || buildStepId("combo-ref", comboName, index, rawModel),
kind: "combo-ref",
comboName: rawModel,
weight,
...(label ? { label } : {}),
};
}
const model = toFullModelString(rawModel, providerId);
const connectionId =
value.connectionId === null ? null : toTrimmedString(value.connectionId) || undefined;
const tags = Array.isArray(value.tags)
? value.tags.map((tag) => toTrimmedString(tag)).filter((tag): tag is string => !!tag)
: undefined;
return {
id:
explicitId ||
buildStepId("model", comboName, index, connectionId ? `${model}:${connectionId}` : model),
kind: "model",
model,
...(providerId ? { providerId } : {}),
...(connectionId !== undefined ? { connectionId } : {}),
weight,
...(label ? { label } : {}),
...(tags && tags.length > 0 ? { tags } : {}),
};
}
export function normalizeComboModels(
models: unknown,
options: Omit<NormalizeComboStepOptions, "index"> = {}
): ComboStep[] {
const list = Array.isArray(models) ? models : [];
return list
.map((value, index) => normalizeComboStep(value, { ...options, index }))
.filter((value): value is ComboStep => value !== null);
}
export function normalizeComboRecord<T extends JsonRecord>(
combo: T,
options: NormalizeComboRecordOptions = {}
): T & { version: 2; models: ComboStep[] } {
const comboName = toTrimmedString(combo.name);
return {
...combo,
version: COMBO_SCHEMA_VERSION,
models: normalizeComboModels(combo.models, {
comboName,
allCombos: options.allCombos,
}),
};
}

123
src/lib/db/combos.ts
View File

@@ -5,6 +5,7 @@
import { v4 as uuidv4 } from "uuid";
import { getDbInstance } from "./core";
import { backupDbFile } from "./backup";
import { normalizeComboRecord } from "@/lib/combos/steps";
type JsonRecord = Record<string, unknown>;
@@ -30,6 +31,45 @@ function withSortOrder(payload: string, sortOrder: number | null): JsonRecord {
return parsed;
}
function getComboNameSet(
db: ReturnType<typeof getDbInstance>,
extraNames: string[] = []
): Set<string> {
const rows = db.prepare("SELECT name FROM combos").all();
const names = new Set<string>();
for (const row of rows) {
const record = asRecord(row);
if (typeof record.name === "string" && record.name.trim().length > 0) {
names.add(record.name.trim());
}
}
for (const name of extraNames) {
if (typeof name === "string" && name.trim().length > 0) {
names.add(name.trim());
}
}
return names;
}
function normalizeStoredCombo(
combo: JsonRecord,
db: ReturnType<typeof getDbInstance>,
extraNames: string[] = []
): JsonRecord {
return normalizeComboRecord(combo, {
allCombos: getComboNameSet(db, extraNames),
}) as JsonRecord;
}
function parseComboRow(row: unknown): JsonRecord | null {
const payload = getSerializedData(row);
if (!payload) return null;
return withSortOrder(payload, getSortOrder(row));
}
function getNextSortOrder() {
const db = getDbInstance();
const row = db.prepare("SELECT COALESCE(MAX(sort_order), 0) AS sort_order FROM combos").get();
@@ -39,47 +79,60 @@ function getNextSortOrder() {
export async function getCombos() {
const db = getDbInstance();
return db
const rawCombos = db
.prepare("SELECT data, sort_order FROM combos ORDER BY sort_order ASC, name COLLATE NOCASE ASC")
.all()
.map((row) => {
const payload = getSerializedData(row);
if (!payload) return null;
return withSortOrder(payload, getSortOrder(row));
})
.map((row) => parseComboRow(row))
.filter((row): row is JsonRecord => row !== null);
const comboNames = rawCombos
.map((combo) => (typeof combo.name === "string" ? combo.name.trim() : ""))
.filter((name): name is string => name.length > 0);
return rawCombos.map((combo) =>
normalizeComboRecord(combo, {
allCombos: comboNames,
})
);
}
export async function getComboById(id: string) {
const db = getDbInstance();
const row = db.prepare("SELECT data, sort_order FROM combos WHERE id = ?").get(id);
const payload = getSerializedData(row);
return payload ? withSortOrder(payload, getSortOrder(row)) : null;
const combo = parseComboRow(row);
if (!combo) return null;
return normalizeStoredCombo(combo, db, typeof combo.name === "string" ? [combo.name] : []);
}
export async function getComboByName(name: string) {
const db = getDbInstance();
const row = db.prepare("SELECT data, sort_order FROM combos WHERE name = ?").get(name);
const payload = getSerializedData(row);
return payload ? withSortOrder(payload, getSortOrder(row)) : null;
const combo = parseComboRow(row);
if (!combo) return null;
return normalizeStoredCombo(combo, db, [name]);
}
export async function createCombo(data: JsonRecord) {
const db = getDbInstance();
const now = new Date().toISOString();
const sortOrder = typeof data.sortOrder === "number" ? data.sortOrder : getNextSortOrder();
const combo = {
id: uuidv4(),
name: data.name,
models: data.models || [],
strategy: data.strategy || "priority",
config: data.config || {},
isHidden: Boolean(data.isHidden),
sortOrder,
createdAt: now,
updatedAt: now,
};
const comboId = typeof data.id === "string" && data.id.trim().length > 0 ? data.id : uuidv4();
const combo = normalizeStoredCombo(
{
...data,
id: comboId,
name: data.name,
models: data.models || [],
strategy: data.strategy || "priority",
config: data.config || {},
isHidden: Boolean(data.isHidden),
sortOrder,
createdAt: now,
updatedAt: now,
},
db,
typeof data.name === "string" ? [data.name] : []
);
db.prepare(
"INSERT INTO combos (id, name, data, sort_order, created_at, updated_at) VALUES (?, ?, ?, ?, ?, ?)"
@@ -94,9 +147,8 @@ export async function updateCombo(id: string, data: JsonRecord) {
const existing = db.prepare("SELECT data, sort_order FROM combos WHERE id = ?").get(id);
if (!existing) return null;
const serializedCurrent = getSerializedData(existing);
if (!serializedCurrent) return null;
const current = withSortOrder(serializedCurrent, getSortOrder(existing));
const current = parseComboRow(existing);
if (!current) return null;
const sortOrder =
typeof data.sortOrder === "number"
? data.sortOrder
@@ -114,13 +166,14 @@ export async function updateCombo(id: string, data: JsonRecord) {
typeof merged["name"] === "string" && merged["name"].trim().length > 0
? merged["name"]
: currentName;
const normalizedMerged = normalizeStoredCombo({ ...merged, name: nextName }, db, [nextName]);
db.prepare(
"UPDATE combos SET name = ?, data = ?, sort_order = ?, updated_at = ? WHERE id = ?"
).run(nextName, JSON.stringify({ ...merged, name: nextName }), sortOrder, merged.updatedAt, id);
).run(nextName, JSON.stringify(normalizedMerged), sortOrder, normalizedMerged.updatedAt, id);
backupDbFile("pre-write");
return { ...merged, name: nextName };
return normalizedMerged;
}
export async function reorderCombos(comboIds: string[]) {
@@ -168,15 +221,23 @@ export async function reorderCombos(comboIds: string[]) {
return [String(record.id), row];
})
);
const comboNames = rows
.map((row) => {
const combo = parseComboRow(row);
return combo && typeof combo.name === "string" ? combo.name.trim() : "";
})
.filter((name): name is string => name.length > 0);
const reorderTransaction = db.transaction(() => {
orderedIds.forEach((id, index) => {
const row = rowById.get(id);
const payload = row ? getSerializedData(row) : null;
if (!payload) return;
const combo = withSortOrder(payload, getSortOrder(row));
const combo = row ? parseComboRow(row) : null;
if (!combo) return;
const sortOrder = index + 1;
const updatedCombo = { ...combo, sortOrder, updatedAt: now };
const updatedCombo = normalizeComboRecord(
{ ...combo, sortOrder, updatedAt: now },
{ allCombos: comboNames }
);
update.run(JSON.stringify(updatedCombo), sortOrder, now, id);
});
});

75
src/lib/db/core.ts
View File

@@ -162,17 +162,24 @@ const SCHEMA_SQL = `
path TEXT,
status INTEGER,
model TEXT,
requested_model TEXT,
provider TEXT,
account TEXT,
connection_id TEXT,
duration INTEGER DEFAULT 0,
tokens_in INTEGER DEFAULT 0,
tokens_out INTEGER DEFAULT 0,
tokens_cache_read INTEGER DEFAULT NULL,
tokens_cache_creation INTEGER DEFAULT NULL,
tokens_reasoning INTEGER DEFAULT NULL,
request_type TEXT,
source_format TEXT,
target_format TEXT,
api_key_id TEXT,
api_key_name TEXT,
combo_name TEXT,
combo_step_id TEXT,
combo_execution_key TEXT,
request_body TEXT,
response_body TEXT,
error TEXT,
@@ -401,6 +408,42 @@ function ensureCallLogsColumns(db: SqliteDatabase) {
db.exec("ALTER TABLE call_logs ADD COLUMN has_pipeline_details INTEGER DEFAULT 0");
console.log("[DB] Added call_logs.has_pipeline_details column");
}
if (!columnNames.has("requested_model")) {
db.exec("ALTER TABLE call_logs ADD COLUMN requested_model TEXT DEFAULT NULL");
console.log("[DB] Added call_logs.requested_model column");
}
if (!columnNames.has("request_type")) {
db.exec("ALTER TABLE call_logs ADD COLUMN request_type TEXT DEFAULT NULL");
console.log("[DB] Added call_logs.request_type column");
}
if (!columnNames.has("tokens_cache_read")) {
db.exec("ALTER TABLE call_logs ADD COLUMN tokens_cache_read INTEGER DEFAULT NULL");
console.log("[DB] Added call_logs.tokens_cache_read column");
}
if (!columnNames.has("tokens_cache_creation")) {
db.exec("ALTER TABLE call_logs ADD COLUMN tokens_cache_creation INTEGER DEFAULT NULL");
console.log("[DB] Added call_logs.tokens_cache_creation column");
}
if (!columnNames.has("tokens_reasoning")) {
db.exec("ALTER TABLE call_logs ADD COLUMN tokens_reasoning INTEGER DEFAULT NULL");
console.log("[DB] Added call_logs.tokens_reasoning column");
}
if (!columnNames.has("combo_step_id")) {
db.exec("ALTER TABLE call_logs ADD COLUMN combo_step_id TEXT DEFAULT NULL");
console.log("[DB] Added call_logs.combo_step_id column");
}
if (!columnNames.has("combo_execution_key")) {
db.exec("ALTER TABLE call_logs ADD COLUMN combo_execution_key TEXT DEFAULT NULL");
console.log("[DB] Added call_logs.combo_execution_key column");
}
db.exec(
"CREATE INDEX IF NOT EXISTS idx_call_logs_requested_model ON call_logs(requested_model)"
);
db.exec("CREATE INDEX IF NOT EXISTS idx_call_logs_request_type ON call_logs(request_type)");
db.exec(
"CREATE INDEX IF NOT EXISTS idx_cl_combo_target ON call_logs(combo_name, combo_execution_key, timestamp)"
);
} catch (error: unknown) {
const message = error instanceof Error ? error.message : String(error);
console.warn("[DB] Failed to verify call_logs schema:", message);
@@ -424,6 +467,7 @@ export function getDbInstance(): SqliteDatabase {
memoryDb.pragma("journal_mode = WAL");
memoryDb.exec(SCHEMA_SQL);
ensureUsageHistoryColumns(memoryDb);
ensureCallLogsColumns(memoryDb);
setDb(memoryDb);
return memoryDb;
}
@@ -524,6 +568,37 @@ export function getDbInstance(): SqliteDatabase {
"combo_sort_order"
);
}
if (hasColumn(db, "call_logs", "request_type")) {
db.prepare("INSERT OR IGNORE INTO _omniroute_migrations (version, name) VALUES (?, ?)").run(
"007",
"search_request_type"
);
}
if (hasColumn(db, "call_logs", "requested_model")) {
db.prepare("INSERT OR IGNORE INTO _omniroute_migrations (version, name) VALUES (?, ?)").run(
"009",
"requested_model"
);
}
if (
hasColumn(db, "call_logs", "tokens_cache_read") &&
hasColumn(db, "call_logs", "tokens_cache_creation") &&
hasColumn(db, "call_logs", "tokens_reasoning")
) {
db.prepare("INSERT OR IGNORE INTO _omniroute_migrations (version, name) VALUES (?, ?)").run(
"018",
"call_logs_detailed_tokens"
);
}
if (
hasColumn(db, "call_logs", "combo_step_id") &&
hasColumn(db, "call_logs", "combo_execution_key")
) {
db.prepare("INSERT OR IGNORE INTO _omniroute_migrations (version, name) VALUES (?, ?)").run(
"021",
"combo_call_log_targets"
);
}
runMigrations(db);
// Auto-migrate from db.json if exists

11
src/lib/db/migrations/001_initial_schema.sql
View File

@@ -115,17 +115,24 @@ CREATE TABLE IF NOT EXISTS call_logs (
path TEXT,
status INTEGER,
model TEXT,
requested_model TEXT,
provider TEXT,
account TEXT,
connection_id TEXT,
duration INTEGER DEFAULT 0,
tokens_in INTEGER DEFAULT 0,
tokens_out INTEGER DEFAULT 0,
tokens_cache_read INTEGER DEFAULT NULL,
tokens_cache_creation INTEGER DEFAULT NULL,
tokens_reasoning INTEGER DEFAULT NULL,
request_type TEXT,
source_format TEXT,
target_format TEXT,
api_key_id TEXT,
api_key_name TEXT,
combo_name TEXT,
combo_step_id TEXT,
combo_execution_key TEXT,
request_body TEXT,
response_body TEXT,
error TEXT,
@@ -134,6 +141,10 @@ CREATE TABLE IF NOT EXISTS call_logs (
);
CREATE INDEX IF NOT EXISTS idx_cl_timestamp ON call_logs(timestamp);
CREATE INDEX IF NOT EXISTS idx_cl_status ON call_logs(status);
CREATE INDEX IF NOT EXISTS idx_call_logs_requested_model ON call_logs(requested_model);
CREATE INDEX IF NOT EXISTS idx_call_logs_request_type ON call_logs(request_type);
CREATE INDEX IF NOT EXISTS idx_cl_combo_target
ON call_logs(combo_name, combo_execution_key, timestamp);
CREATE TABLE IF NOT EXISTS proxy_logs (
id TEXT PRIMARY KEY,

5
src/lib/db/migrations/021_combo_call_log_targets.sql Normal file
View File

@@ -0,0 +1,5 @@
ALTER TABLE call_logs ADD COLUMN combo_step_id TEXT DEFAULT NULL;
ALTER TABLE call_logs ADD COLUMN combo_execution_key TEXT DEFAULT NULL;
CREATE INDEX IF NOT EXISTS idx_cl_combo_target
ON call_logs(combo_name, combo_execution_key, timestamp);

20
src/lib/db/settings.ts
View File

@@ -7,6 +7,7 @@ import { backupDbFile } from "./backup";
import { PROVIDER_ID_TO_ALIAS } from "@omniroute/open-sse/config/providerModels.ts";
import { invalidateDbCache } from "./readCache";
import { resolveProxyForConnectionFromRegistry } from "./proxies";
import { getComboModelProvider as getComboEntryProvider } from "@/lib/combos/steps";
type JsonRecord = Record<string, unknown>;
type PricingModels = Record<string, JsonRecord>;
@@ -313,23 +314,8 @@ function resolveProviderAliasOrId(providerOrAlias: string): string {
}
function getComboModelProvider(modelEntry: unknown): string | null {
const record = toRecord(modelEntry);
if (typeof record.provider === "string") {
return resolveProviderAliasOrId(record.provider);
}
const modelValue =
typeof modelEntry === "string"
? modelEntry
: typeof record.model === "string"
? record.model
: null;
if (!modelValue) return null;
const [providerOrAlias] = modelValue.split("/", 1);
if (!providerOrAlias) return null;
return resolveProviderAliasOrId(providerOrAlias);
const providerOrAlias = getComboEntryProvider(modelEntry);
return providerOrAlias ? resolveProviderAliasOrId(providerOrAlias) : null;
}
function migrateProxyEntry(value: unknown): JsonRecord | null {

280
src/lib/modelCapabilities.ts Normal file
View File

@@ -0,0 +1,280 @@
import {
PROVIDER_ID_TO_ALIAS,
PROVIDER_MODELS,
} from "@omniroute/open-sse/config/providerModels.ts";
import { parseModel, resolveCanonicalProviderModel } from "@omniroute/open-sse/services/model.ts";
import { MODEL_SPECS, getModelSpec, type ModelSpec } from "@/shared/constants/modelSpecs";
import { getSyncedCapability } from "@/lib/modelsDevSync";
const TOOL_CALLING_UNSUPPORTED_PATTERNS = ["gpt-oss-120b", "deepseek-reasoner"];
const REASONING_UNSUPPORTED_PATTERNS = [
"antigravity/claude-sonnet-4-6",
"antigravity/claude-sonnet-4-5",
"antigravity/claude-sonnet-4",
];
type CapabilityInput =
| string
| {
provider?: string | null;
model?: string | null;
};
type SyncedCapabilities = ReturnType<typeof getSyncedCapability>;
export interface ResolvedModelCapabilities {
provider: string | null;
model: string | null;
rawModel: string | null;
toolCalling: boolean;
reasoning: boolean;
supportsThinking: boolean | null;
supportsTools: boolean | null;
supportsVision: boolean | null;
attachment: boolean | null;
structuredOutput: boolean | null;
temperature: boolean | null;
contextWindow: number | null;
maxInputTokens: number | null;
maxOutputTokens: number;
defaultThinkingBudget: number;
thinkingBudgetCap: number | null;
thinkingOverhead: number | null;
adaptiveMaxTokens: number | null;
family: string | null;
status: string | null;
openWeights: boolean | null;
knowledgeCutoff: string | null;
releaseDate: string | null;
lastUpdated: string | null;
modalitiesInput: string[];
modalitiesOutput: string[];
interleavedField: string | null;
}
function toNonEmptyString(value: unknown): string | null {
return typeof value === "string" && value.trim().length > 0 ? value.trim() : null;
}
function parseModalities(value: string | null | undefined): string[] {
if (typeof value !== "string" || value.trim().length === 0) return [];
try {
const parsed = JSON.parse(value);
return Array.isArray(parsed)
? parsed.filter((entry): entry is string => typeof entry === "string" && entry.length > 0)
: [];
} catch {
return [];
}
}
function getRegistryModel(providerIdOrAlias: string | null, modelId: string | null) {
if (!providerIdOrAlias || !modelId) return null;
const providerAlias = PROVIDER_ID_TO_ALIAS[providerIdOrAlias] || providerIdOrAlias;
const models = PROVIDER_MODELS[providerAlias];
if (!Array.isArray(models)) return null;
return models.find((model) => model?.id === modelId) || null;
}
function resolveCapabilityInput(input: CapabilityInput) {
if (typeof input === "string") {
const parsed = parseModel(input);
const rawModel = toNonEmptyString(parsed.model);
if (parsed.provider) {
const canonical = resolveCanonicalProviderModel(parsed.provider, rawModel);
return {
provider: canonical.provider,
model: toNonEmptyString(canonical.model),
rawModel,
lookupKey: input,
};
}
return {
provider: null,
model: rawModel,
rawModel,
lookupKey: input,
};
}
const rawProvider = toNonEmptyString(input.provider);
const rawModel = toNonEmptyString(input.model);
if (rawProvider) {
const canonical = resolveCanonicalProviderModel(rawProvider, rawModel);
return {
provider: canonical.provider,
model: toNonEmptyString(canonical.model),
rawModel,
lookupKey: rawModel ? `${canonical.provider}/${rawModel}` : canonical.provider,
};
}
return {
provider: null,
model: rawModel,
rawModel,
lookupKey: rawModel || "",
};
}
function heuristicToolCalling(modelStr: string): boolean {
const normalized = String(modelStr || "").toLowerCase();
if (!normalized) return false;
const blocked = TOOL_CALLING_UNSUPPORTED_PATTERNS.some((pattern) => {
if (normalized === pattern) return true;
if (normalized.endsWith(`/${pattern}`)) return true;
return normalized.includes(pattern);
});
return !blocked;
}
function heuristicReasoning(modelStr: string): boolean {
const normalized = String(modelStr || "").toLowerCase();
if (!normalized) return true;
const blocked = REASONING_UNSUPPORTED_PATTERNS.some(
(pattern) =>
normalized === pattern || normalized.endsWith(`/${pattern}`) || normalized.includes(pattern)
);
return !blocked;
}
function getStaticSpec(modelId: string | null, rawModel: string | null): ModelSpec | undefined {
if (modelId) {
const byCanonical = getModelSpec(modelId);
if (byCanonical) return byCanonical;
}
if (rawModel && rawModel !== modelId) {
return getModelSpec(rawModel);
}
return undefined;
}
function resolveVisionCapability(
spec: ModelSpec | undefined,
registryModel: { supportsVision?: boolean } | null,
synced: SyncedCapabilities,
modalitiesInput: string[],
modalitiesOutput: string[]
): boolean | null {
if (typeof spec?.supportsVision === "boolean") return spec.supportsVision;
if (typeof registryModel?.supportsVision === "boolean") return registryModel.supportsVision;
const allModalities = [...modalitiesInput, ...modalitiesOutput].map((entry) =>
String(entry).toLowerCase()
);
if (allModalities.some((entry) => entry.includes("image"))) {
return true;
}
return null;
}
export function getResolvedModelCapabilities(input: CapabilityInput): ResolvedModelCapabilities {
const resolved = resolveCapabilityInput(input);
const spec = getStaticSpec(resolved.model, resolved.rawModel);
const registryModel = getRegistryModel(resolved.provider, resolved.model);
const synced =
resolved.provider && resolved.model
? getSyncedCapability(resolved.provider, resolved.model)
: null;
const modalitiesInput = parseModalities(synced?.modalities_input);
const modalitiesOutput = parseModalities(synced?.modalities_output);
const lookupKey =
toNonEmptyString(
resolved.provider && resolved.model
? `${resolved.provider}/${resolved.model}`
: resolved.model || resolved.rawModel || resolved.lookupKey
) || "";
const supportsTools =
typeof spec?.supportsTools === "boolean"
? spec.supportsTools
: typeof registryModel?.toolCalling === "boolean"
? registryModel.toolCalling
: (synced?.tool_call ?? null);
const supportsThinking =
typeof spec?.supportsThinking === "boolean"
? spec.supportsThinking
: typeof registryModel?.supportsReasoning === "boolean"
? registryModel.supportsReasoning
: (synced?.reasoning ?? null);
return {
provider: resolved.provider,
model: resolved.model,
rawModel: resolved.rawModel,
toolCalling: supportsTools ?? heuristicToolCalling(lookupKey),
reasoning: supportsThinking ?? heuristicReasoning(lookupKey),
supportsThinking,
supportsTools,
supportsVision: resolveVisionCapability(
spec,
registryModel,
synced,
modalitiesInput,
modalitiesOutput
),
attachment: synced?.attachment ?? null,
structuredOutput: synced?.structured_output ?? null,
temperature: synced?.temperature ?? null,
contextWindow:
spec?.contextWindow ??
(typeof registryModel?.contextLength === "number" ? registryModel.contextLength : null) ??
synced?.limit_context ??
null,
maxInputTokens: synced?.limit_input ?? spec?.contextWindow ?? null,
maxOutputTokens:
spec?.maxOutputTokens ?? synced?.limit_output ?? MODEL_SPECS.__default__.maxOutputTokens,
defaultThinkingBudget: spec?.defaultThinkingBudget ?? 0,
thinkingBudgetCap: spec?.thinkingBudgetCap ?? null,
thinkingOverhead: spec?.thinkingOverhead ?? null,
adaptiveMaxTokens: spec?.adaptiveMaxTokens ?? null,
family: synced?.family ?? null,
status: synced?.status ?? null,
openWeights: synced?.open_weights ?? null,
knowledgeCutoff: synced?.knowledge_cutoff ?? null,
releaseDate: synced?.release_date ?? null,
lastUpdated: synced?.last_updated ?? null,
modalitiesInput,
modalitiesOutput,
interleavedField: synced?.interleaved_field ?? null,
};
}
export function supportsToolCalling(input: CapabilityInput): boolean {
if (typeof input === "string" && !String(input || "").trim()) return false;
return getResolvedModelCapabilities(input).toolCalling;
}
export function supportsReasoning(input: CapabilityInput): boolean {
if (typeof input === "string" && !String(input || "").trim()) return true;
return getResolvedModelCapabilities(input).reasoning;
}
export function capMaxOutputTokens(input: CapabilityInput, requested?: number): number {
const cap = getResolvedModelCapabilities(input).maxOutputTokens;
return requested ? Math.min(requested, cap) : cap;
}
export function getDefaultThinkingBudget(input: CapabilityInput): number {
return getResolvedModelCapabilities(input).defaultThinkingBudget;
}
export function capThinkingBudget(input: CapabilityInput, budget: number): number {
const cap = getResolvedModelCapabilities(input).thinkingBudgetCap ?? budget;
return Math.min(budget, cap);
}
export function getModelContextLimit(
providerOrInput: CapabilityInput,
modelId?: string
): number | null {
const resolved =
typeof providerOrInput === "string" && modelId !== undefined
? getResolvedModelCapabilities({ provider: providerOrInput, model: modelId })
: getResolvedModelCapabilities(providerOrInput);
return resolved.contextWindow;
}

106
src/lib/modelsDevSync.ts
View File

@@ -34,7 +34,7 @@ type PricingEntry = {
type PricingModels = Record<string, PricingEntry>;
type PricingByProvider = Record<string, PricingModels>;
interface CapabilityEntry {
export interface ModelCapabilityEntry {
tool_call: boolean | null;
reasoning: boolean | null;
attachment: boolean | null;
@@ -54,7 +54,7 @@ interface CapabilityEntry {
interleaved_field: string | null;
}
type CapabilitiesByProvider = Record<string, Record<string, CapabilityEntry>>;
export type CapabilitiesByProvider = Record<string, Record<string, ModelCapabilityEntry>>;
interface SyncStatus {
enabled: boolean;
@@ -209,6 +209,8 @@ let activeSyncIntervalMs = SYNC_INTERVAL_MS;
let cachedData: ModelsDevData | null = null;
let cacheTime = 0;
const CACHE_TTL_MS = 24 * 60 * 60 * 1000; // 24 hours
let cachedCapabilities: CapabilitiesByProvider | null = null;
let cachedCapabilitiesLoadedAll = false;
// ─── Core: Fetch ─────────────────────────────────────────
@@ -297,7 +299,7 @@ export function transformModelsDevToCapabilities(raw: ModelsDevData): Capabiliti
const omniRouteProviders = mapProviderId(providerId);
for (const [modelId, model] of Object.entries(providerData.models || {})) {
const cap: CapabilityEntry = {
const cap: ModelCapabilityEntry = {
tool_call: model.tool_call ?? null,
reasoning: model.reasoning ?? null,
attachment: model.attachment ?? null,
@@ -338,6 +340,31 @@ function toRecord(value: unknown): Record<string, unknown> {
return value && typeof value === "object" ? (value as Record<string, unknown>) : {};
}
function mapCapabilityRecord(record: Record<string, unknown>): ModelCapabilityEntry {
return {
tool_call: record.tool_call === 1 ? true : record.tool_call === 0 ? false : null,
reasoning: record.reasoning === 1 ? true : record.reasoning === 0 ? false : null,
attachment: record.attachment === 1 ? true : record.attachment === 0 ? false : null,
structured_output:
record.structured_output === 1 ? true : record.structured_output === 0 ? false : null,
temperature: record.temperature === 1 ? true : record.temperature === 0 ? false : null,
modalities_input: typeof record.modalities_input === "string" ? record.modalities_input : "[]",
modalities_output:
typeof record.modalities_output === "string" ? record.modalities_output : "[]",
knowledge_cutoff: typeof record.knowledge_cutoff === "string" ? record.knowledge_cutoff : null,
release_date: typeof record.release_date === "string" ? record.release_date : null,
last_updated: typeof record.last_updated === "string" ? record.last_updated : null,
status: typeof record.status === "string" ? record.status : null,
family: typeof record.family === "string" ? record.family : null,
open_weights: record.open_weights === 1 ? true : record.open_weights === 0 ? false : null,
limit_context: typeof record.limit_context === "number" ? record.limit_context : null,
limit_input: typeof record.limit_input === "number" ? record.limit_input : null,
limit_output: typeof record.limit_output === "number" ? record.limit_output : null,
interleaved_field:
typeof record.interleaved_field === "string" ? record.interleaved_field : null,
};
}
/**
* Read synced pricing from `models_dev_pricing` namespace.
*/
@@ -429,10 +456,20 @@ export function ensureCapabilitiesTable(): void {
/**
* Read synced capabilities from `model_capabilities` table.
*/
export function getSyncedCapabilities(
provider?: string,
modelId?: string
): Record<string, Record<string, CapabilityEntry>> {
export function getSyncedCapabilities(provider?: string, modelId?: string): CapabilitiesByProvider {
if (cachedCapabilitiesLoadedAll) {
if (!provider) {
return cachedCapabilities || {};
}
if (!modelId) {
return cachedCapabilities?.[provider] ? { [provider]: cachedCapabilities[provider] } : {};
}
const providerCaps = cachedCapabilities?.[provider];
return providerCaps?.[modelId] ? { [provider]: { [modelId]: providerCaps[modelId] } } : {};
}
const db = getDbInstance();
ensureCapabilitiesTable();
@@ -449,7 +486,7 @@ export function getSyncedCapabilities(
}
const rows = db.prepare(query).all(...params);
const result: Record<string, Record<string, CapabilityEntry>> = {};
const result: CapabilitiesByProvider = {};
for (const row of rows) {
const record = toRecord(row);
@@ -458,35 +495,36 @@ export function getSyncedCapabilities(
if (!prov || !mid) continue;
if (!result[prov]) result[prov] = {};
result[prov][mid] = {
tool_call: record.tool_call === 1 ? true : record.tool_call === 0 ? false : null,
reasoning: record.reasoning === 1 ? true : record.reasoning === 0 ? false : null,
attachment: record.attachment === 1 ? true : record.attachment === 0 ? false : null,
structured_output:
record.structured_output === 1 ? true : record.structured_output === 0 ? false : null,
temperature: record.temperature === 1 ? true : record.temperature === 0 ? false : null,
modalities_input:
typeof record.modalities_input === "string" ? record.modalities_input : "[]",
modalities_output:
typeof record.modalities_output === "string" ? record.modalities_output : "[]",
knowledge_cutoff:
typeof record.knowledge_cutoff === "string" ? record.knowledge_cutoff : null,
release_date: typeof record.release_date === "string" ? record.release_date : null,
last_updated: typeof record.last_updated === "string" ? record.last_updated : null,
status: typeof record.status === "string" ? record.status : null,
family: typeof record.family === "string" ? record.family : null,
open_weights: record.open_weights === 1 ? true : record.open_weights === 0 ? false : null,
limit_context: typeof record.limit_context === "number" ? record.limit_context : null,
limit_input: typeof record.limit_input === "number" ? record.limit_input : null,
limit_output: typeof record.limit_output === "number" ? record.limit_output : null,
interleaved_field:
typeof record.interleaved_field === "string" ? record.interleaved_field : null,
};
result[prov][mid] = mapCapabilityRecord(record);
}
if (!provider && !modelId) {
cachedCapabilities = result;
cachedCapabilitiesLoadedAll = true;
}
return result;
}
export function getSyncedCapability(
provider: string,
modelId: string
): ModelCapabilityEntry | null {
if (!provider || !modelId) return null;
if (cachedCapabilitiesLoadedAll) {
return cachedCapabilities?.[provider]?.[modelId] ?? null;
}
const db = getDbInstance();
ensureCapabilitiesTable();
const row = db
.prepare("SELECT * FROM model_capabilities WHERE provider = ? AND model_id = ? LIMIT 1")
.get(provider, modelId);
if (!row) return null;
return mapCapabilityRecord(toRecord(row));
}
/**
* Save synced capabilities to `model_capabilities` table (full replace).
*/
@@ -536,6 +574,8 @@ export function saveModelsDevCapabilities(data: CapabilitiesByProvider): void {
});
tx();
backupDbFile("pre-write");
cachedCapabilities = data;
cachedCapabilitiesLoadedAll = true;
}
/**
@@ -546,6 +586,8 @@ export function clearModelsDevCapabilities(): void {
ensureCapabilitiesTable();
db.prepare("DELETE FROM model_capabilities").run();
backupDbFile("pre-write");
cachedCapabilities = {};
cachedCapabilitiesLoadedAll = true;
}
// ─── Main sync function ──────────────────────────────────

227
src/lib/monitoring/observability.ts Normal file
View File

@@ -0,0 +1,227 @@
type JsonRecord = Record<string, unknown>;
interface CircuitBreakerStatus {
name: string;
state: string;
failureCount?: number;
lastFailureTime?: string | null;
}
interface SessionSnapshot {
sessionId: string;
createdAt: number;
lastActive: number;
requestCount: number;
connectionId: string | null;
ageMs: number;
}
interface QuotaMonitorSnapshot {
sessionId: string;
provider: string;
accountId: string;
status: "starting" | "idle" | "healthy" | "warning" | "exhausted" | "error";
startedAt: string;
lastPolledAt: string | null;
lastSuccessAt: string | null;
lastErrorAt: string | null;
lastError: string | null;
lastQuotaPercent: number | null;
lastQuotaUsed: number | null;
lastQuotaTotal: number | null;
lastResetAt: string | null;
lastAlertAt: string | null;
nextPollDelayMs: number | null;
nextPollAt: string | null;
totalPolls: number;
totalAlerts: number;
consecutiveFailures: number;
}
interface QuotaMonitorSummary {
active: number;
alerting: number;
exhausted: number;
errors: number;
statusCounts: Record<QuotaMonitorSnapshot["status"], number>;
byProvider: Record<string, number>;
}
interface BuildSessionsSummaryOptions {
activeSessions: SessionSnapshot[];
activeSessionsByKey?: Record<string, number>;
}
interface BuildTelemetryPayloadOptions {
summary: {
count: number;
p50: number;
p95: number;
p99: number;
phaseBreakdown: JsonRecord;
};
quotaMonitorSummary: QuotaMonitorSummary;
activeSessions: SessionSnapshot[];
}
interface BuildHealthPayloadOptions {
appVersion: string;
catalogCount?: number;
settings: { setupComplete?: boolean } | null | undefined;
connections: Array<{ provider?: string; isActive?: boolean | null }>;
circuitBreakers: CircuitBreakerStatus[];
rateLimitStatus: JsonRecord;
lockouts: JsonRecord;
localProviders: JsonRecord;
inflightRequests: number;
quotaMonitorSummary: QuotaMonitorSummary;
quotaMonitorMonitors: QuotaMonitorSnapshot[];
activeSessions: SessionSnapshot[];
activeSessionsByKey?: Record<string, number>;
}
function limitMonitors(monitors: QuotaMonitorSnapshot[], maxItems = 8): QuotaMonitorSnapshot[] {
return monitors.slice(0, maxItems);
}
export function buildSessionsSummary({
activeSessions,
activeSessionsByKey = {},
}: BuildSessionsSummaryOptions) {
const ordered = [...activeSessions].sort((left, right) => right.lastActive - left.lastActive);
const stickyBoundCount = ordered.filter((entry) => entry.connectionId).length;
return {
activeCount: ordered.length,
stickyBoundCount,
byApiKey: activeSessionsByKey,
top: ordered.slice(0, 8).map((entry) => ({
sessionId: entry.sessionId,
requestCount: entry.requestCount,
connectionId: entry.connectionId,
ageMs: entry.ageMs,
idleMs: Math.max(0, Date.now() - entry.lastActive),
createdAt: new Date(entry.createdAt).toISOString(),
lastActiveAt: new Date(entry.lastActive).toISOString(),
})),
};
}
export function buildTelemetryPayload({
summary,
quotaMonitorSummary,
activeSessions,
}: BuildTelemetryPayloadOptions) {
const sessions = buildSessionsSummary({ activeSessions });
return {
...summary,
totalRequests: summary.count,
sessions: {
activeCount: sessions.activeCount,
stickyBoundCount: sessions.stickyBoundCount,
},
quotaMonitor: {
active: quotaMonitorSummary.active,
alerting: quotaMonitorSummary.alerting,
exhausted: quotaMonitorSummary.exhausted,
errors: quotaMonitorSummary.errors,
statusCounts: quotaMonitorSummary.statusCounts,
},
};
}
export function buildHealthPayload({
appVersion,
catalogCount = 0,
settings,
connections,
circuitBreakers,
rateLimitStatus,
lockouts,
localProviders,
inflightRequests,
quotaMonitorSummary,
quotaMonitorMonitors,
activeSessions,
activeSessionsByKey = {},
}: BuildHealthPayloadOptions) {
const timestamp = new Date().toISOString();
const system = {
version: appVersion,
nodeVersion: process.version,
uptime: process.uptime(),
memoryUsage: process.memoryUsage(),
pid: process.pid,
platform: process.platform,
};
const providerHealth: Record<string, JsonRecord> = {};
for (const cb of circuitBreakers) {
if (cb.name.startsWith("test-") || cb.name.startsWith("test_")) continue;
providerHealth[cb.name] = {
state: cb.state,
failures: cb.failureCount || 0,
lastFailure: cb.lastFailureTime || null,
};
}
const configuredProviders = new Set(
connections.map((connection) => connection.provider).filter(Boolean)
);
const activeProviders = new Set(
connections
.filter((connection) => connection.isActive !== false)
.map((connection) => connection.provider)
.filter(Boolean)
);
const breakerCounts = circuitBreakers.reduce(
(acc, cb) => {
if (cb.name.startsWith("test-") || cb.name.startsWith("test_")) return acc;
if (cb.state === "OPEN") acc.open += 1;
else if (cb.state === "HALF_OPEN") acc.halfOpen += 1;
else acc.closed += 1;
return acc;
},
{ open: 0, halfOpen: 0, closed: 0 }
);
return {
status: "healthy",
timestamp,
system,
version: system.version,
uptime: system.uptime,
memoryUsage: system.memoryUsage,
activeConnections: connections.length,
circuitBreakers: {
...breakerCounts,
total: breakerCounts.open + breakerCounts.halfOpen + breakerCounts.closed,
},
providerHealth,
providerSummary: {
catalogCount,
configuredCount: configuredProviders.size,
activeCount: activeProviders.size,
monitoredCount: Object.keys(providerHealth).length,
},
localProviders,
rateLimitStatus,
lockouts,
quotaMonitor: {
...quotaMonitorSummary,
monitors: limitMonitors(quotaMonitorMonitors),
},
sessions: buildSessionsSummary({ activeSessions, activeSessionsByKey }),
dedup: {
inflightRequests,
},
cryptography: {
status:
process.env.STORAGE_ENCRYPTION_KEY && process.env.STORAGE_ENCRYPTION_KEY.length >= 32
? "healthy"
: "missing_or_invalid",
provider: "aes-256-gcm",
},
setupComplete: settings?.setupComplete || false,
};
}

23
src/lib/usage/callLogs.ts
View File

@@ -59,6 +59,8 @@ type CallLogArtifact = {
apiKeyId: string | null;
apiKeyName: string | null;
comboName: string | null;
comboStepId: string | null;
comboExecutionKey: string | null;
};
requestBody: unknown;
responseBody: unknown;
@@ -202,6 +204,8 @@ function buildArtifact(
apiKeyId: string | null;
apiKeyName: string | null;
comboName: string | null;
comboStepId: string | null;
comboExecutionKey: string | null;
},
requestBody: unknown,
responseBody: unknown,
@@ -235,6 +239,8 @@ function buildArtifact(
apiKeyId: logEntry.apiKeyId,
apiKeyName: logEntry.apiKeyName,
comboName: logEntry.comboName,
comboStepId: logEntry.comboStepId,
comboExecutionKey: logEntry.comboExecutionKey,
},
requestBody: requestBody ?? null,
responseBody: responseBody ?? null,
@@ -418,6 +424,9 @@ export async function saveCallLog(entry: any) {
apiKeyId,
apiKeyName: entry.apiKeyName || null,
comboName: entry.comboName || null,
comboStepId: toStringOrNull(entry.comboStepId),
comboExecutionKey:
toStringOrNull(entry.comboExecutionKey) || toStringOrNull(entry.comboStepId),
requestBody: serializePayloadForStorage(protectedRequestBody, 8192),
responseBody: serializePayloadForStorage(protectedResponseBody, 8192),
error: toStoredErrorString(protectedError),
@@ -431,16 +440,17 @@ export async function saveCallLog(entry: any) {
account, connection_id, duration, tokens_in, tokens_out,
tokens_cache_read, tokens_cache_creation, tokens_reasoning,
request_type, source_format,
target_format, api_key_id, api_key_name, combo_name, request_body, response_body, error,
artifact_relpath, has_pipeline_details
target_format, api_key_id, api_key_name, combo_name, combo_step_id,
combo_execution_key, request_body, response_body, error, artifact_relpath,
has_pipeline_details
)
VALUES (
@id, @timestamp, @method, @path, @status, @model, @requestedModel, @provider,
@account, @connectionId, @duration, @tokensIn, @tokensOut,
@tokensCacheRead, @tokensCacheCreation, @tokensReasoning,
@requestType, @sourceFormat,
@targetFormat, @apiKeyId, @apiKeyName, @comboName, @requestBody, @responseBody, @error,
NULL, 0
@targetFormat, @apiKeyId, @apiKeyName, @comboName, @comboStepId,
@comboExecutionKey, @requestBody, @responseBody, @error, NULL, 0
)
`
).run(logEntry);
@@ -544,6 +554,7 @@ export async function getCallLogs(filter: any = {}) {
requested_model LIKE @searchQ OR provider LIKE @searchQ OR
api_key_name LIKE @searchQ OR api_key_id LIKE @searchQ OR
combo_name LIKE @searchQ OR CAST(status AS TEXT) LIKE @searchQ
OR combo_step_id LIKE @searchQ OR combo_execution_key LIKE @searchQ
)`);
params.searchQ = `%${filter.search}%`;
}
@@ -581,6 +592,8 @@ export async function getCallLogs(filter: any = {}) {
targetFormat: toStringOrNull(l.target_format),
error: toStringOrNull(l.error),
comboName: toStringOrNull(l.combo_name),
comboStepId: toStringOrNull(l.combo_step_id),
comboExecutionKey: toStringOrNull(l.combo_execution_key),
apiKeyId: toStringOrNull(l.api_key_id),
apiKeyName: toStringOrNull(l.api_key_name),
hasRequestBody: typeof l.request_body === "string" && l.request_body.length > 0,
@@ -634,6 +647,8 @@ export async function getCallLogById(id: string) {
apiKeyId: toStringOrNull(entryRow.api_key_id),
apiKeyName: toStringOrNull(entryRow.api_key_name),
comboName: toStringOrNull(entryRow.combo_name),
comboStepId: toStringOrNull(entryRow.combo_step_id),
comboExecutionKey: toStringOrNull(entryRow.combo_execution_key),
requestBody: parseStoredPayload(entryRow.request_body),
responseBody: parseStoredPayload(entryRow.response_body),
error: toStringOrNull(entryRow.error),

59
src/proxy.ts
View File

@@ -1,13 +1,52 @@
import { NextResponse } from "next/server";
import { jwtVerify, SignJWT } from "jose";
import { generateRequestId } from "./shared/utils/requestId";
import { getSettings } from "./lib/localDb";
import { isPublicRoute, verifyAuth, isAuthRequired } from "./shared/utils/apiAuth";
import { checkBodySize, getBodySizeLimit } from "./shared/middleware/bodySizeGuard";
import { isDraining } from "./lib/gracefulShutdown";
import { isModelSyncInternalRequest } from "./shared/services/modelSyncScheduler";
const SECRET = new TextEncoder().encode(process.env.JWT_SECRET || "");
const E2E_MODE = process.env.NEXT_PUBLIC_OMNIROUTE_E2E_MODE === "1";
const PUBLIC_API_ROUTES = [
"/api/auth/login",
"/api/auth/logout",
"/api/auth/status",
"/api/settings/require-login",
"/api/init",
"/api/monitoring/health",
"/api/v1/",
"/api/cloud/",
"/api/oauth/",
];
let apiAuthModulePromise: Promise<typeof import("./shared/utils/apiAuth")> | null = null;
let settingsModulePromise: Promise<typeof import("./lib/db/settings")> | null = null;
let modelSyncModulePromise: Promise<typeof import("./shared/services/modelSyncScheduler")> | null =
null;
function isPublicApiRoute(pathname: string): boolean {
return PUBLIC_API_ROUTES.some((route) => pathname.startsWith(route));
}
async function getApiAuthModule() {
if (!apiAuthModulePromise) {
apiAuthModulePromise = import("./shared/utils/apiAuth");
}
return apiAuthModulePromise;
}
async function getSettingsModule() {
if (!settingsModulePromise) {
settingsModulePromise = import("./lib/db/settings");
}
return settingsModulePromise;
}
async function getModelSyncModule() {
if (!modelSyncModulePromise) {
modelSyncModulePromise = import("./shared/services/modelSyncScheduler");
}
return modelSyncModulePromise;
}
export async function proxy(request: any) {
const { pathname } = request.nextUrl;
@@ -37,14 +76,24 @@ export async function proxy(request: any) {
if (bodySizeRejection) return bodySizeRejection;
}
if (E2E_MODE) {
if (pathname.startsWith("/dashboard")) {
return response;
}
if (pathname.startsWith("/api/") && !pathname.startsWith("/api/v1/")) {
return response;
}
}
// ──────────────── Protect Management API Routes ────────────────
if (pathname.startsWith("/api/") && !pathname.startsWith("/api/v1/")) {
// Allow public routes (login, logout, health, etc.)
if (isPublicRoute(pathname)) {
if (isPublicApiRoute(pathname)) {
return response;
}
// Allow the model auto-sync scheduler to reach only its internal provider routes.
const { isModelSyncInternalRequest } = await getModelSyncModule();
if (
isModelSyncInternalRequest(request) &&
/^\/api\/providers\/[^/]+\/(sync-models|models)$/.test(pathname)
@@ -53,6 +102,7 @@ export async function proxy(request: any) {
}
// Check if auth is required at all (respects requireLogin setting)
const { isAuthRequired, verifyAuth } = await getApiAuthModule();
const authRequired = await isAuthRequired();
if (!authRequired) {
return response;
@@ -83,6 +133,7 @@ export async function proxy(request: any) {
try {
// Direct import — no HTTP self-fetch overhead
const { getSettings } = await getSettingsModule();
const settings = await getSettings();
// Skip auth if login is not required
if (settings.requireLogin === false) {

17
src/shared/types/utilization.ts
View File

@@ -31,6 +31,23 @@ export interface ComboHealthMetrics {
comboName: string;
strategy: string;
models: string[];
targetHealth?: Array<{
executionKey: string;
stepId: string;
model: string;
provider: string;
connectionId: string | null;
label: string | null;
requests: number;
successRate: number;
avgLatencyMs: number;
lastStatus: "ok" | "error" | null;
lastUsedAt: string | null;
quotaRemainingPct: number | null;
quotaIsExhausted: boolean | null;
quotaTrend: "improving" | "stable" | "declining" | null;
quotaScope: "connection" | "provider" | "none";
}>;
quotaHealth: {
providers: Array<{
provider: string;

13
src/shared/utils/machineId.ts
View File

@@ -1,5 +1,5 @@
import { execSync, execFileSync } from "child_process";
import { existsSync, readFileSync } from "fs";
import { existsSync } from "fs";
/**
* Get raw machine ID using OS-specific methods.
@@ -58,9 +58,16 @@ function getMachineIdRaw(): string {
// Strategy 3: Linux — read machine-id files directly (no `head` or pipe)
try {
for (const filePath of ["/etc/machine-id", "/var/lib/dbus/machine-id"]) {
if (existsSync(filePath)) {
const content = readFileSync(filePath, "utf8").trim().toLowerCase();
try {
const content = execFileSync("cat", [filePath], {
encoding: "utf8",
timeout: 5000,
})
.trim()
.toLowerCase();
if (content.length > 8) return content;
} catch {
// Try the next candidate file
}
}
} catch {

67
src/shared/validation/schemas.ts
View File

@@ -63,13 +63,33 @@ export const createKeySchema = z.object({
// ──── Combo Schemas ────
// A model entry can be a plain string (legacy) or an object with weight
const comboStepMetaSchema = {
id: z.string().trim().min(1).max(200).optional(),
weight: z.number().min(0).max(100).optional().default(0),
label: z.string().trim().min(1).max(200).optional(),
};
const comboModelStepInputSchema = z.object({
kind: z.literal("model").optional(),
provider: z.string().trim().min(1).max(120).optional(),
providerId: z.string().trim().min(1).max(120).optional(),
model: z.string().trim().min(1).max(300),
connectionId: z.string().trim().min(1).max(200).nullable().optional(),
tags: z.array(z.string().trim().min(1).max(100)).max(20).optional(),
...comboStepMetaSchema,
});
const comboRefStepInputSchema = z.object({
kind: z.literal("combo-ref"),
comboName: z.string().trim().min(1).max(100),
...comboStepMetaSchema,
});
// A combo entry can be a plain string (legacy), a legacy object, or a structured ComboStep.
const comboModelEntry = z.union([
z.string(),
z.object({
model: z.string().min(1),
weight: z.number().min(0).max(100).default(0),
}),
z.string().trim().min(1).max(300),
comboModelStepInputSchema,
comboRefStepInputSchema,
]);
const comboStrategySchema = z.enum([
@@ -102,6 +122,22 @@ const scoringWeightsSchema = z
})
.optional();
const compositeTierEntrySchema = z
.object({
stepId: z.string().trim().min(1).max(200),
fallbackTier: z.string().trim().min(1).max(100).optional(),
label: z.string().trim().min(1).max(200).optional(),
description: z.string().trim().min(1).max(500).optional(),
})
.strict();
const compositeTiersSchema = z
.object({
defaultTier: z.string().trim().min(1).max(100),
tiers: z.record(z.string().trim().min(1).max(100), compositeTierEntrySchema),
})
.strict();
const comboRuntimeConfigSchema = z
.object({
strategy: comboStrategySchema.optional(),
@@ -125,6 +161,7 @@ const comboRuntimeConfigSchema = z
budgetCap: z.number().positive().optional(),
explorationRate: z.number().min(0).max(1).optional(),
routerStrategy: z.string().optional(),
compositeTiers: compositeTiersSchema.optional(),
})
.strict();
@@ -537,6 +574,24 @@ export const updateComboDefaultsSchema = z
path: [],
});
}
if (value.comboDefaults?.compositeTiers) {
ctx.addIssue({
code: z.ZodIssueCode.custom,
message: "compositeTiers is only supported on concrete combos",
path: ["comboDefaults", "compositeTiers"],
});
}
for (const [providerId, config] of Object.entries(value.providerOverrides || {})) {
if (config?.compositeTiers) {
ctx.addIssue({
code: z.ZodIssueCode.custom,
message: "compositeTiers is only supported on concrete combos",
path: ["providerOverrides", providerId, "compositeTiers"],
});
}
}
});
export const updateRequireLoginSchema = z

126
src/sse/handlers/chat.ts
View File

@@ -1,6 +1,6 @@
import { randomUUID } from "crypto";
import {
getProviderCredentials,
getProviderCredentialsWithQuotaPreflight,
markAccountUnavailable,
extractApiKey,
isValidApiKey,
@@ -56,15 +56,12 @@ import {
registerKeySession,
isSessionRegisteredForKey,
} from "@omniroute/open-sse/services/sessionManager.ts";
import { startQuotaMonitor } from "@omniroute/open-sse/services/quotaMonitor.ts";
import {
isFallbackDecision,
shouldUseFallback,
} from "@omniroute/open-sse/services/emergencyFallback.ts";
import {
registerCodexQuotaFetcher,
registerCodexConnection,
fetchCodexQuota,
} from "@omniroute/open-sse/services/codexQuotaFetcher.ts";
import { registerCodexQuotaFetcher } from "@omniroute/open-sse/services/codexQuotaFetcher.ts";
// Register Codex quota fetcher at module load (once per server start).
// This hooks into the quotaPreflight + quotaMonitor systems so that combos
@@ -249,7 +246,10 @@ export async function handleChat(request: any, clientRawRequest: any = null) {
// Pre-check function used by combo routing. For explicit combo live tests,
// avoid pre-skipping so each model gets a real execution attempt.
const checkModelAvailable = async (modelString: string) => {
const checkModelAvailable = async (
modelString: string,
target?: { connectionId?: string | null }
) => {
if (isComboLiveTest) return true;
// Use getModelInfo to properly resolve custom prefixes
@@ -257,48 +257,29 @@ export async function handleChat(request: any, clientRawRequest: any = null) {
const provider = modelInfo.provider;
if (!provider) return true; // can't determine provider, let it try
// Check domain-level availability (cooldown)
if (!isModelAvailable(provider, modelInfo.model || modelString)) {
const resolvedModel = modelInfo.model || modelString;
const hasForcedConnection =
typeof target?.connectionId === "string" && target.connectionId.trim().length > 0;
// Fixed-account combo steps must bypass the provider/model cooldown gate here.
// A previous account failure can quarantine the model globally, but the next
// step may intentionally pin a different connection for the same model.
if (!hasForcedConnection && !isModelAvailable(provider, resolvedModel)) {
log.debug("AVAILABILITY", `${provider}/${modelInfo.model} in cooldown, skipping`);
return false;
}
const creds = await getProviderCredentials(
const creds = await getProviderCredentialsWithQuotaPreflight(
provider,
null,
apiKeyInfo?.allowedConnections ?? null,
modelInfo.model || modelString
resolvedModel,
{
...(target?.connectionId ? { forcedConnectionId: target.connectionId } : {}),
}
);
if (!creds || creds.allRateLimited) return false;
// ── Codex Quota Preflight (Item 1-2) ──────────────────────────────────
// Proactively skip Codex accounts that have consumed >= 95% of either
// their 5h or 7d quota window. This prevents requests from failing with
// a 429 and then retrying — we switch accounts early instead.
if (provider === "codex" && creds.connectionId) {
// Register connection metadata so the fetcher can call the usage API
if (creds.accessToken) {
registerCodexConnection(creds.connectionId, {
accessToken: creds.accessToken,
workspaceId:
typeof creds.providerSpecificData?.workspaceId === "string"
? creds.providerSpecificData.workspaceId
: undefined,
});
}
const quotaInfo = await fetchCodexQuota(creds.connectionId);
if (quotaInfo && quotaInfo.percentUsed >= 0.95) {
const pct = (quotaInfo.percentUsed * 100).toFixed(1);
log.info(
"QUOTA_PREFLIGHT",
`Skipping Codex account ${creds.connectionId.slice(0, 8)}...: quota at ${pct}% (preflight)`
);
return false;
}
}
// ──────────────────────────────────────────────────────────────────────
return true;
};
@@ -316,7 +297,15 @@ export async function handleChat(request: any, clientRawRequest: any = null) {
const response = await (handleComboChat as any)({
body,
combo,
handleSingleModel: (b: any, m: string) =>
handleSingleModel: (
b: any,
m: string,
target?: {
connectionId?: string | null;
executionKey?: string | null;
stepId?: string | null;
}
) =>
handleSingleModelChat(
b,
m,
@@ -328,6 +317,9 @@ export async function handleChat(request: any, clientRawRequest: any = null) {
{
sessionId,
forceLiveComboTest: isComboLiveTest,
forcedConnectionId: target?.connectionId ?? null,
comboStepId: target?.stepId || null,
comboExecutionKey: target?.executionKey || target?.stepId || null,
},
combo.strategy,
true
@@ -437,6 +429,9 @@ async function handleSingleModelChat(
emergencyFallbackTried?: boolean;
forceLiveComboTest?: boolean;
sessionId?: string | null;
forcedConnectionId?: string | null;
comboStepId?: string | null;
comboExecutionKey?: string | null;
} = {},
comboStrategy: string | null = null,
isCombo: boolean = false
@@ -447,11 +442,20 @@ async function handleSingleModelChat(
const { provider, model, sourceFormat, targetFormat, extendedContext } = resolved;
const forceLiveComboTest = runtimeOptions.forceLiveComboTest === true;
const hasForcedConnection =
typeof runtimeOptions.forcedConnectionId === "string" &&
runtimeOptions.forcedConnectionId.trim().length > 0;
const bypassReason = forceLiveComboTest
? "combo live test"
: hasForcedConnection
? "fixed combo step connection"
: undefined;
// 2. Pipeline gates (availability + circuit breaker)
const gate = checkPipelineGates(provider, model, {
ignoreCircuitBreaker: forceLiveComboTest,
ignoreModelCooldown: forceLiveComboTest,
ignoreCircuitBreaker: forceLiveComboTest || hasForcedConnection,
ignoreModelCooldown: forceLiveComboTest || hasForcedConnection,
...(bypassReason ? { bypassReason } : {}),
});
if (gate) return gate;
@@ -471,17 +475,22 @@ async function handleSingleModelChat(
let lastCooldownMs = 0;
while (true) {
const credentials = await getProviderCredentials(
const credentials = await getProviderCredentialsWithQuotaPreflight(
provider,
excludeConnectionId,
apiKeyInfo?.allowedConnections ?? null,
model,
forceLiveComboTest
? {
allowSuppressedConnections: true,
bypassQuotaPolicy: true,
}
: undefined
{
...(forceLiveComboTest
? {
allowSuppressedConnections: true,
bypassQuotaPolicy: true,
}
: {}),
...(runtimeOptions.forcedConnectionId
? { forcedConnectionId: runtimeOptions.forcedConnectionId }
: {}),
}
);
if (!credentials || credentials.allRateLimited) {
@@ -531,11 +540,16 @@ async function handleSingleModelChat(
);
}
}
const refreshedCredentials = await checkAndRefreshToken(provider, credentials);
if (runtimeOptions.sessionId && body?._omnirouteInternalRequest !== "context-handoff") {
touchSession(runtimeOptions.sessionId, credentials.connectionId);
startQuotaMonitor(
runtimeOptions.sessionId,
provider,
credentials.connectionId,
refreshedCredentials
);
}
const refreshedCredentials = await checkAndRefreshToken(provider, credentials);
const proxyInfo = await safeResolveProxy(credentials.connectionId);
const proxyStartTime = Date.now();
@@ -557,6 +571,8 @@ async function handleSingleModelChat(
comboName,
comboStrategy,
isCombo,
comboStepId: runtimeOptions.comboStepId ?? null,
comboExecutionKey: runtimeOptions.comboExecutionKey ?? runtimeOptions.comboStepId ?? null,
extendedContext,
});
if (telemetry) telemetry.endPhase();
@@ -634,7 +650,13 @@ async function handleSingleModelChat(
comboName,
apiKeyInfo,
telemetry,
{ ...runtimeOptions, emergencyFallbackTried: true },
{
...runtimeOptions,
emergencyFallbackTried: true,
forcedConnectionId: null,
comboStepId: null,
comboExecutionKey: null,
},
null, // no strategy for emergency fallback
Boolean(comboName) // isCombo if comboName exists
);

15
src/sse/handlers/chatHelpers.ts
View File

@@ -64,11 +64,16 @@ export async function resolveModelOrError(modelStr: string, body: any, endpointP
export function checkPipelineGates(
provider: string,
model: string,
options: { ignoreCircuitBreaker?: boolean; ignoreModelCooldown?: boolean } = {}
options: {
ignoreCircuitBreaker?: boolean;
ignoreModelCooldown?: boolean;
bypassReason?: string;
} = {}
) {
const bypassReason = options.bypassReason || "pipeline override";
const modelAvailable = isModelAvailable(provider, model);
if (!modelAvailable && options.ignoreModelCooldown) {
log.info("AVAILABILITY", `${provider}/${model} cooldown bypassed for combo live test`);
log.info("AVAILABILITY", `${provider}/${model} cooldown bypassed (${bypassReason})`);
} else if (!modelAvailable) {
log.warn("AVAILABILITY", `${provider}/${model} is in cooldown, rejecting request`);
return unavailableResponse(
@@ -85,7 +90,7 @@ export function checkPipelineGates(
log.info("CIRCUIT", `${name}: ${from} → ${to}`),
});
if (options.ignoreCircuitBreaker && !breaker.canExecute()) {
log.info("CIRCUIT", `Bypassing OPEN circuit breaker for combo live test: ${provider}`);
log.info("CIRCUIT", `Bypassing OPEN circuit breaker for ${provider} (${bypassReason})`);
} else if (!breaker.canExecute()) {
log.warn("CIRCUIT", `Circuit breaker OPEN for ${provider}, rejecting request`);
return unavailableResponse(
@@ -114,6 +119,8 @@ export async function executeChatWithBreaker({
comboName,
comboStrategy,
isCombo,
comboStepId,
comboExecutionKey,
extendedContext,
}: any): Promise<{ result: any; tlsFingerprintUsed: boolean }> {
let tlsFingerprintUsed = false;
@@ -133,6 +140,8 @@ export async function executeChatWithBreaker({
comboName,
comboStrategy,
isCombo,
comboStepId,
comboExecutionKey,
onCredentialsRefreshed: async (newCreds: any) => {
await updateProviderCredentials(credentials.connectionId, {
accessToken: newCreds.accessToken,

324
src/sse/services/auth.ts
View File

@@ -6,7 +6,7 @@ import {
getSettings,
getCachedSettings,
} from "@/lib/localDb";
import { getQuotaWindowStatus, isAccountQuotaExhausted } from "@/domain/quotaCache";
import { getQuotaCache, getQuotaWindowStatus, isAccountQuotaExhausted } from "@/domain/quotaCache";
import {
isAccountUnavailable,
getUnavailableUntil,
@@ -22,6 +22,7 @@ import {
getPassthroughProviders,
} from "@omniroute/open-sse/config/providerRegistry.ts";
import { COOLDOWN_MS } from "@omniroute/open-sse/config/constants.ts";
import { preflightQuota } from "@omniroute/open-sse/services/quotaPreflight.ts";
import { getCodexModelScope } from "@omniroute/open-sse/executors/codex.ts";
import { getProviderAlias, resolveProviderId } from "@/shared/constants/providers";
import * as log from "../utils/logger";
@@ -64,6 +65,8 @@ interface RecoverableConnectionState {
interface CredentialSelectionOptions {
allowSuppressedConnections?: boolean;
bypassQuotaPolicy?: boolean;
forcedConnectionId?: string | null;
excludeConnectionIds?: string[] | null;
}
const CODEX_QUOTA_THRESHOLD_PERCENT = 90;
@@ -134,6 +137,16 @@ interface QuotaLimitPolicy {
windows: string[];
}
interface QuotaCacheView {
quotas?: Record<
string,
{
remainingPercentage?: number;
resetAt?: string | null;
}
>;
}
function normalizeQuotaThreshold(value: unknown, fallback = CODEX_QUOTA_THRESHOLD_PERCENT): number {
const parsed = toNumber(value, fallback);
return Math.min(MAX_QUOTA_THRESHOLD_PERCENT, Math.max(MIN_QUOTA_THRESHOLD_PERCENT, parsed));
@@ -300,6 +313,192 @@ function getEarliestFutureDate(candidates: Array<string | null>): string | null
);
}
function getConnectionQuotaHeadroomPercent(
provider: string,
connection: ProviderConnectionView
): number | null {
const policy = resolveQuotaLimitPolicy(provider, connection.providerSpecificData);
const percentages: number[] = [];
const seenWindows = new Set<string>();
const collectWindow = (windowName: string) => {
const normalizedWindow = normalizeWindowName(windowName);
if (!normalizedWindow || seenWindows.has(normalizedWindow)) return;
seenWindows.add(normalizedWindow);
const status = getQuotaWindowStatus(connection.id, normalizedWindow, policy.thresholdPercent);
if (!status) return;
percentages.push(Math.max(0, Math.min(100, status.remainingPercentage)));
};
for (const windowName of policy.windows) {
collectWindow(windowName);
}
if (percentages.length > 0) {
return Math.min(...percentages);
}
const quotaEntry = getQuotaCache(connection.id) as QuotaCacheView | null;
const rawQuotas = quotaEntry?.quotas || {};
for (const quota of Object.values(rawQuotas)) {
if (!quota) continue;
const resetAt = toStringOrNull(quota.resetAt);
if (resetAt) {
const resetMs = new Date(resetAt).getTime();
if (Number.isFinite(resetMs) && resetMs <= Date.now()) {
continue;
}
}
const remaining = toNumber(quota.remainingPercentage, Number.NaN);
if (Number.isFinite(remaining)) {
percentages.push(Math.max(0, Math.min(100, remaining)));
}
}
return percentages.length > 0 ? Math.min(...percentages) : null;
}
function getConnectionErrorPenalty(connection: ProviderConnectionView): number {
const errorType = normalizeStatus(connection.lastErrorType);
const errorSource = normalizeStatus(connection.lastErrorSource);
const numericErrorCode = toNumber(connection.errorCode, 0);
let penalty = 0;
if (connection.lastError) penalty += 6;
if (
errorType === "rate_limited" ||
errorType === "quota_exhausted" ||
errorType === "quota" ||
numericErrorCode === 429
) {
penalty += 24;
} else if (numericErrorCode === 401 || numericErrorCode === 403 || errorSource === "oauth") {
penalty += 18;
} else if (numericErrorCode >= 500) {
penalty += 10;
}
return penalty;
}
function getConnectionRecencyPenalty(connection: ProviderConnectionView): number {
if (!connection.lastUsedAt) return 0;
const ageMs = Date.now() - new Date(connection.lastUsedAt).getTime();
if (!Number.isFinite(ageMs)) return 0;
if (ageMs < 15_000) return 3;
if (ageMs < 60_000) return 2;
if (ageMs < 5 * 60_000) return 1;
return 0;
}
function getP2CConnectionScore(
provider: string,
connection: ProviderConnectionView
): { score: number; quotaHeadroomPercent: number | null } {
const quotaBlocked = evaluateQuotaLimitPolicy(provider, connection).blocked;
const quotaExhausted = isAccountQuotaExhausted(connection.id);
const quotaHeadroomPercent = getConnectionQuotaHeadroomPercent(provider, connection);
let quotaPenalty = 0;
if (quotaHeadroomPercent !== null) {
quotaPenalty += Math.round((100 - quotaHeadroomPercent) / 8);
if (quotaHeadroomPercent <= 10) quotaPenalty += 10;
else if (quotaHeadroomPercent <= 25) quotaPenalty += 4;
} else if (!quotaBlocked && !quotaExhausted) {
quotaPenalty += 4;
}
const score =
(quotaExhausted ? 200 : 0) +
(quotaBlocked ? 80 : 0) +
getConnectionErrorPenalty(connection) +
Math.min(40, (connection.backoffLevel || 0) * 8) +
quotaPenalty +
Math.min(12, (connection.consecutiveUseCount || 0) * 2) +
getConnectionRecencyPenalty(connection) +
Math.min(6, Math.max(0, connection.priority || 0) - 1);
return { score, quotaHeadroomPercent };
}
function compareP2CConnections(
provider: string,
a: ProviderConnectionView,
b: ProviderConnectionView
): number {
const aScore = getP2CConnectionScore(provider, a);
const bScore = getP2CConnectionScore(provider, b);
if (aScore.score !== bScore.score) {
return aScore.score - bScore.score;
}
const aHeadroom = aScore.quotaHeadroomPercent ?? -1;
const bHeadroom = bScore.quotaHeadroomPercent ?? -1;
if (aHeadroom !== bHeadroom) {
return bHeadroom - aHeadroom;
}
if ((a.priority || 999) !== (b.priority || 999)) {
return (a.priority || 999) - (b.priority || 999);
}
return a.id.localeCompare(b.id);
}
function normalizeExcludedConnectionIds(
excludeConnectionId: string | null,
extraExcludedConnectionIds: string[] | null | undefined
): Set<string> {
const normalized = new Set<string>();
if (typeof excludeConnectionId === "string" && excludeConnectionId.trim().length > 0) {
normalized.add(excludeConnectionId.trim());
}
if (Array.isArray(extraExcludedConnectionIds)) {
for (const connectionId of extraExcludedConnectionIds) {
if (typeof connectionId === "string" && connectionId.trim().length > 0) {
normalized.add(connectionId.trim());
}
}
}
return normalized;
}
function buildQuotaPreflightRateLimitedResult(
provider: string,
blockedByPreflight: Array<{
id: string;
quotaPercent?: number;
resetAt?: string | null;
}>
) {
const retryAfter =
getEarliestFutureDate(blockedByPreflight.map((entry) => entry.resetAt ?? null)) ||
new Date(Date.now() + 5 * 60 * 1000).toISOString();
const blockedSummary = blockedByPreflight
.map((entry) => {
const percent = Number.isFinite(entry.quotaPercent)
? `${Math.round((entry.quotaPercent as number) * 100)}%`
: "quota exhausted";
return `${entry.id.slice(0, 8)}(${percent})`;
})
.join("; ");
log.info("AUTH", `${provider} | quota preflight filtered account(s): ${blockedSummary}`);
return {
allRateLimited: true,
retryAfter,
retryAfterHuman: formatRetryAfter(retryAfter),
lastError: `All ${provider} accounts blocked by quota preflight`,
lastErrorCode: 429,
};
}
// Mutex to prevent race conditions during account selection
let selectionMutex = Promise.resolve();
@@ -355,6 +554,14 @@ export async function getProviderCredentials(
const allowSuppressedConnections = options.allowSuppressedConnections === true;
const bypassQuotaPolicy = options.bypassQuotaPolicy === true;
const forcedConnectionId =
typeof options.forcedConnectionId === "string" && options.forcedConnectionId.trim().length > 0
? options.forcedConnectionId.trim()
: null;
const excludedConnectionIds = normalizeExcludedConnectionIds(
excludeConnectionId,
options.excludeConnectionIds
);
// Fix #922: Check for aliases (nvidia/nvidia_nim) to ensure credentials are found
const providersToSearch = getProviderSearchPool(provider);
@@ -370,9 +577,14 @@ export async function getProviderCredentials(
if (allowedConnections && allowedConnections.length > 0) {
connections = connections.filter((conn) => allowedConnections.includes(conn.id));
}
if (forcedConnectionId) {
connections = connections.filter((conn) => conn.id === forcedConnectionId);
}
log.debug(
"AUTH",
`${provider} | total connections: ${connections.length}, excludeId: ${excludeConnectionId || "none"}`
`${provider} | total connections: ${connections.length}, excludeIds: ${
excludedConnectionIds.size > 0 ? Array.from(excludedConnectionIds).join(",") : "none"
}, forcedId: ${forcedConnectionId || "none"}`
);
if (connections.length === 0) {
@@ -381,10 +593,15 @@ export async function getProviderCredentials(
const allConnectionsResults = await Promise.all(
providersToSearch.map((p) => getProviderConnections({ provider: p }))
);
const allConnectionsRaw = allConnectionsResults.filter(Array.isArray).flat();
const allConnections = (Array.isArray(allConnectionsRaw) ? allConnectionsRaw : [])
let allConnections = (allConnectionsResults.filter(Array.isArray).flat() as unknown[])
.map(toProviderConnection)
.filter((conn) => conn.id.length > 0);
if (allowedConnections && allowedConnections.length > 0) {
allConnections = allConnections.filter((conn) => allowedConnections.includes(conn.id));
}
if (forcedConnectionId) {
allConnections = allConnections.filter((conn) => conn.id === forcedConnectionId);
}
log.debug("AUTH", `${provider} | all connections (incl inactive): ${allConnections.length}`);
if (allConnections.length > 0) {
const earliest = getEarliestRateLimitedUntil(allConnections);
@@ -436,7 +653,7 @@ export async function getProviderCredentials(
// Filter out unavailable accounts and excluded connection
const availableConnections = connections.filter((c) => {
if (excludeConnectionId && c.id === excludeConnectionId) return false;
if (excludedConnectionIds.has(c.id)) return false;
if (!allowSuppressedConnections) {
if (isAccountUnavailable(c.rateLimitedUntil)) return false;
if (isTerminalConnectionStatus(c)) return false;
@@ -452,7 +669,7 @@ export async function getProviderCredentials(
`${provider} | available: ${availableConnections.length}/${connections.length}`
);
connections.forEach((c) => {
const excluded = excludeConnectionId && c.id === excludeConnectionId;
const excluded = excludedConnectionIds.has(c.id);
const rateLimited = isAccountUnavailable(c.rateLimitedUntil);
const terminalStatus = isTerminalConnectionStatus(c);
const codexScopeLimited = provider === "codex" && isCodexScopeUnavailable(c, requestedModel);
@@ -662,22 +879,20 @@ export async function getProviderCredentials(
});
}
} else if (strategy === "p2c") {
// Power of Two Choices: pick 2 random, choose the one with fewer failures
if (orderedConnections.length <= 2) {
connection = orderedConnections[0];
const candidatePool = withQuota.length > 0 ? withQuota : orderedConnections;
// Power of Two Choices: sample from the quota-eligible pool and compare
// health instead of defaulting to random-first selection.
if (candidatePool.length <= 2) {
connection = [...candidatePool].sort((a, b) => compareP2CConnections(provider, a, b))[0];
} else {
const i =
parseInt(randomUUID().replace(/-/g, "").substring(0, 8), 16) % orderedConnections.length;
parseInt(randomUUID().replace(/-/g, "").substring(0, 8), 16) % candidatePool.length;
let j =
parseInt(randomUUID().replace(/-/g, "").substring(0, 8), 16) %
(orderedConnections.length - 1);
parseInt(randomUUID().replace(/-/g, "").substring(0, 8), 16) % (candidatePool.length - 1);
if (j >= i) j++;
const a = orderedConnections[i];
const b = orderedConnections[j];
// Prefer the one with fewer consecutive uses / better health
const scoreA = (a.consecutiveUseCount || 0) + (a.lastError ? 10 : 0);
const scoreB = (b.consecutiveUseCount || 0) + (b.lastError ? 10 : 0);
connection = scoreA <= scoreB ? a : b;
const a = candidatePool[i];
const b = candidatePool[j];
connection = compareP2CConnections(provider, a, b) <= 0 ? a : b;
}
} else if (strategy === "random") {
// Random: Fisher-Yates-inspired random pick
@@ -735,6 +950,79 @@ export async function getProviderCredentials(
}
}
export async function getProviderCredentialsWithQuotaPreflight(
provider: string,
excludeConnectionId: string | null = null,
allowedConnections: string[] | null = null,
requestedModel: string | null = null,
options: CredentialSelectionOptions = {}
) {
if (options.bypassQuotaPolicy === true) {
return getProviderCredentials(
provider,
excludeConnectionId,
allowedConnections,
requestedModel,
options
);
}
const blockedByPreflight: Array<{
id: string;
quotaPercent?: number;
resetAt?: string | null;
}> = [];
const excludedConnectionIds = normalizeExcludedConnectionIds(
excludeConnectionId,
options.excludeConnectionIds
);
while (true) {
const credentials = await getProviderCredentials(
provider,
null,
allowedConnections,
requestedModel,
{
...options,
excludeConnectionIds: Array.from(excludedConnectionIds),
}
);
if (!credentials) {
if (blockedByPreflight.length > 0) {
return buildQuotaPreflightRateLimitedResult(provider, blockedByPreflight);
}
return null;
}
if (credentials.allRateLimited) {
return credentials;
}
const preflight = await preflightQuota(provider, credentials.connectionId, credentials);
if (preflight.proceed) {
return credentials;
}
blockedByPreflight.push({
id: credentials.connectionId,
quotaPercent: preflight.quotaPercent,
resetAt: preflight.resetAt ?? null,
});
excludedConnectionIds.add(credentials.connectionId);
log.info(
"AUTH",
`${provider} | preflight blocked ${credentials.connectionId.slice(0, 8)}${
Number.isFinite(preflight.quotaPercent)
? ` at ${Math.round((preflight.quotaPercent as number) * 100)}%`
: ""
}`
);
}
}
/**
* Mark account as unavailable — reads backoffLevel from DB, calculates cooldown with exponential backoff, saves new level
* @param {string} connectionId

5
src/sse/services/model.ts
View File

@@ -1,6 +1,7 @@
// Re-export from open-sse with localDb integration
import { getModelAliases, getComboByName, getProviderNodes, getCustomModels } from "@/lib/localDb";
import { getSettings } from "@/lib/localDb";
import { getComboStepTarget } from "@/lib/combos/steps";
import {
parseModel,
resolveModelAliasFromMap,
@@ -147,5 +148,7 @@ export async function getComboForModel(modelStr) {
export async function getComboModels(modelStr) {
const combo = await getCombo(modelStr);
if (!combo) return null;
return combo.models.map((m) => (typeof m === "string" ? m : m.model));
return (combo.models || [])
.map((entry) => getComboStepTarget(entry))
.filter((entry): entry is string => typeof entry === "string" && entry.length > 0);
}

101
tests/e2e/combos-flow.spec.ts
View File

@@ -17,6 +17,20 @@ type ComboCreatePayload = {
config?: Record<string, unknown>;
};
async function dispatchHtml5DragAndDrop(
page: import("@playwright/test").Page,
source: import("@playwright/test").Locator,
target: import("@playwright/test").Locator
) {
const dataTransfer = await page.evaluateHandle(() => new DataTransfer());
await source.dispatchEvent("dragstart", { dataTransfer });
await target.dispatchEvent("dragenter", { dataTransfer });
await target.dispatchEvent("dragover", { dataTransfer });
await target.dispatchEvent("drop", { dataTransfer });
await source.dispatchEvent("dragend", { dataTransfer });
await dataTransfer.dispose();
}
test.describe("Combos flow", () => {
test("applies template, creates combo, and runs quick test CTA", async ({ page }) => {
const state: {
@@ -110,6 +124,33 @@ test.describe("Combos flow", () => {
});
});
await page.route("**/api/combos/builder/options", async (route) => {
await route.fulfill({
status: 200,
contentType: "application/json",
body: JSON.stringify({
providers: [
{
providerId: "openai",
displayName: "OpenAI",
connectionCount: 1,
models: [{ id: "qa-test-model", name: "QA Test Model" }],
connections: [
{
id: "conn-openai",
label: "OpenAI Primary",
status: "active",
priority: 1,
defaultModel: "qa-test-model",
},
],
},
],
comboRefs: [],
}),
});
});
await page.route("**/api/combos", async (route) => {
const method = route.request().method();
if (method === "GET") {
@@ -151,8 +192,10 @@ test.describe("Combos flow", () => {
});
});
await page.goto("/dashboard/combos");
await page.waitForLoadState("networkidle");
await page.goto("/dashboard/combos", { waitUntil: "domcontentloaded" });
await expect(
page.getByRole("button", { name: /create combo|criar combo/i }).first()
).toBeVisible();
const redirectedToLogin = page.url().includes("/login");
test.skip(redirectedToLogin, "Authentication enabled without a login fixture.");
@@ -164,15 +207,22 @@ test.describe("Combos flow", () => {
const comboDialog = page.getByRole("dialog").first();
await expect(comboDialog).toBeVisible();
const comboCreateButton = comboDialog
.getByRole("button", { name: /create combo|criar combo/i })
.last();
const readinessPanel = comboDialog.locator('[data-testid="combo-readiness-panel"]');
const saveBlockers = comboDialog.locator('[data-testid="combo-save-blockers"]');
const comboNextButton = comboDialog.locator('[data-testid="combo-builder-next"]');
await expect(comboNextButton).toBeDisabled();
await comboDialog.locator('[data-testid="combo-template-high-availability"]').click();
await expect(comboNextButton).toBeEnabled();
await comboNextButton.click();
await expect(comboDialog.locator('[data-testid="combo-builder-stage-steps"]')).toBeVisible();
await expect(comboNextButton).toBeDisabled();
await comboDialog.locator('[data-testid="combo-builder-provider"]').selectOption("openai");
await comboDialog.locator('[data-testid="combo-builder-model"]').selectOption("qa-test-model");
await comboDialog.locator('[data-testid="combo-builder-account"]').selectOption("conn-openai");
await comboDialog.locator('[data-testid="combo-builder-add-step"]').click();
await expect(comboNextButton).toBeEnabled();
await comboNextButton.click();
await expect(readinessPanel).toBeVisible();
await expect(saveBlockers).toBeVisible();
await expect(comboCreateButton).toBeDisabled();
const applyRecommendationsButton = comboDialog
.getByRole("button", { name: /apply recommendations|aplicar recomendações/i })
.first();
@@ -184,14 +234,15 @@ test.describe("Combos flow", () => {
await expect(comboDialog.locator('[data-testid="strategy-change-nudge"]')).toBeVisible();
await applyRecommendationsButton.click();
await comboDialog
.getByRole("button", { name: /high availability|alta disponibilidade/i })
.click();
await comboDialog.getByRole("button", { name: /add model|adicionar modelo/i }).click();
await comboNextButton.click();
const modelDialog = page.getByRole("dialog").last();
await expect(modelDialog.getByRole("button", { name: /qa test model/i })).toBeVisible();
await modelDialog.getByRole("button", { name: /qa test model/i }).click();
const comboCreateButton = comboDialog
.getByRole("button", { name: /create combo|criar combo/i })
.last();
const readinessPanel = comboDialog.locator('[data-testid="combo-readiness-panel"]');
const saveBlockers = comboDialog.locator('[data-testid="combo-save-blockers"]');
await expect(readinessPanel).toBeVisible();
await expect(saveBlockers).toHaveCount(0);
await expect(comboCreateButton).toBeEnabled();
@@ -308,8 +359,8 @@ test.describe("Combos flow", () => {
});
});
await page.goto("/dashboard/combos");
await page.waitForLoadState("networkidle");
await page.goto("/dashboard/combos", { waitUntil: "domcontentloaded" });
await expect(page.getByTestId("combo-card-combo-1")).toBeVisible();
const redirectedToLogin = page.url().includes("/login");
test.skip(redirectedToLogin, "Authentication enabled without a login fixture.");
@@ -321,9 +372,11 @@ test.describe("Combos flow", () => {
)
.toEqual(["combo-card-combo-1", "combo-card-combo-2", "combo-card-combo-3"]);
await page
.getByTestId("combo-drag-handle-combo-3")
.dragTo(page.getByTestId("combo-card-combo-1"));
await dispatchHtml5DragAndDrop(
page,
page.getByTestId("combo-drag-handle-combo-3"),
page.getByTestId("combo-card-combo-1")
);
await expect.poll(() => state.reorderRequests).toBe(1);
await expect
@@ -332,8 +385,8 @@ test.describe("Combos flow", () => {
)
.toEqual(["combo-card-combo-3", "combo-card-combo-1", "combo-card-combo-2"]);
await page.reload();
await page.waitForLoadState("networkidle");
await page.reload({ waitUntil: "domcontentloaded" });
await expect(page.getByTestId("combo-card-combo-3")).toBeVisible();
await expect
.poll(async () =>

98
tests/integration/combo-routing-e2e.test.mjs
View File

@@ -4,6 +4,7 @@ import assert from "node:assert/strict";
import { createChatPipelineHarness } from "./_chatPipelineHarness.mjs";
const harness = await createChatPipelineHarness("combo-routing");
const callLogs = await import("../../src/lib/usage/callLogs.ts");
const {
BaseExecutor,
buildClaudeResponse,
@@ -16,6 +17,7 @@ const {
resetStorage,
seedConnection,
toPlainHeaders,
waitFor,
} = harness;
test.beforeEach(async () => {
@@ -172,6 +174,102 @@ test("priority combo falls back to the secondary model when the first one fails"
assert.equal(json.choices[0].message.content, "Fallback answered");
});
test("priority combo can repeat the same provider/model with different fixed accounts", async () => {
const firstConn = await seedConnection("openai", {
name: "openai-fixed-1",
apiKey: "sk-openai-fixed-1",
});
const secondConn = await seedConnection("openai", {
name: "openai-fixed-2",
apiKey: "sk-openai-fixed-2",
});
assert.notEqual(firstConn.id, secondConn.id);
await combosDb.createCombo({
name: "router-fixed-accounts",
strategy: "priority",
config: { maxRetries: 0, retryDelayMs: 0 },
models: [
{
id: "step-openai-primary",
kind: "model",
providerId: "openai",
model: "gpt-4o-mini",
connectionId: firstConn.id,
},
{
id: "step-openai-secondary",
kind: "model",
providerId: "openai",
model: "gpt-4o-mini",
connectionId: secondConn.id,
},
],
});
const authHeaders = [];
let firstAttemptHeader = null;
globalThis.fetch = async (_url, init = {}) => {
const headers = toPlainHeaders(init.headers);
authHeaders.push(headers.Authorization);
if (!firstAttemptHeader) {
firstAttemptHeader = headers.Authorization;
return new Response(JSON.stringify({ error: { message: "first account down" } }), {
status: 503,
headers: { "Content-Type": "application/json" },
});
}
return buildOpenAIResponse("Second fixed account answered");
};
const response = await handleChat(
buildRequest({
body: buildOpenAIChatBody("router-fixed-accounts"),
})
);
const json = await response.json();
assert.equal(response.status, 200);
assert.equal(authHeaders.length, 2);
assert.notEqual(authHeaders[0], authHeaders[1]);
assert.deepEqual(
new Set(authHeaders),
new Set(["Bearer sk-openai-fixed-1", "Bearer sk-openai-fixed-2"])
);
assert.equal(json.choices[0].message.content, "Second fixed account answered");
const comboLogs = await waitFor(async () => {
const logs = await callLogs.getCallLogs({ combo: true, limit: 10 });
return logs.length >= 2 ? logs : null;
});
assert.ok(comboLogs, "expected combo call logs to be persisted");
const targetLogs = comboLogs
.filter((entry) => entry.comboName === "router-fixed-accounts")
.sort((left, right) => (left.timestamp || "").localeCompare(right.timestamp || ""));
assert.equal(targetLogs.length, 2);
assert.deepEqual(
targetLogs.map((entry) => ({
comboStepId: entry.comboStepId,
comboExecutionKey: entry.comboExecutionKey,
status: entry.status,
})),
[
{
comboStepId: "step-openai-primary",
comboExecutionKey: "step-openai-primary",
status: 503,
},
{
comboStepId: "step-openai-secondary",
comboExecutionKey: "step-openai-secondary",
status: 200,
},
]
);
});
test("model combo mappings route explicit model ids through the configured combo", async () => {
await seedConnection("openai", { apiKey: "sk-openai-mapped" });
const combo = await combosDb.createCombo({

15
tests/unit/build-next-isolated.test.mjs
View File

@@ -5,7 +5,7 @@ import fsSync from "node:fs";
import os from "node:os";
import path from "node:path";
const { movePath } = await import("../../scripts/build-next-isolated.mjs");
const { movePath, resolveNextBuildEnv } = await import("../../scripts/build-next-isolated.mjs");
async function withTempDir(fn) {
const tempDir = await fs.mkdtemp(path.join(os.tmpdir(), "omniroute-build-next-isolated-"));
@@ -88,3 +88,16 @@ test("movePath rethrows non-EXDEV rename failures", async () => {
);
});
});
test("resolveNextBuildEnv forces stable build worker mode unless already provided", () => {
const defaultEnv = resolveNextBuildEnv({ NODE_ENV: "test" });
assert.equal(defaultEnv.NEXT_PRIVATE_BUILD_WORKER, "0");
assert.equal(defaultEnv.NODE_ENV, "test");
const preservedEnv = resolveNextBuildEnv({
NODE_ENV: "production",
NEXT_PRIVATE_BUILD_WORKER: "1",
});
assert.equal(preservedEnv.NEXT_PRIVATE_BUILD_WORKER, "1");
assert.equal(preservedEnv.NODE_ENV, "production");
});

37
tests/unit/call-log-cap.test.mjs
View File

@@ -41,6 +41,9 @@ test("call logs store a single per-request artifact with pipeline details", asyn
requestedModel: "openai/gpt-5",
provider: "openai",
duration: 42,
comboName: "combo-a",
comboStepId: "step-openai-a",
comboExecutionKey: "combo-a:0:step-openai-a",
requestBody: { messages: [{ role: "user", content: "hello" }] },
responseBody: { id: "resp_1", choices: [{ message: { content: "world" } }] },
pipelinePayloads: {
@@ -57,6 +60,9 @@ test("call logs store a single per-request artifact with pipeline details", asyn
const detail = await callLogs.getCallLogById(logId);
assert.equal(detail?.requestedModel, "openai/gpt-5");
assert.equal(detail?.comboName, "combo-a");
assert.equal(detail?.comboStepId, "step-openai-a");
assert.equal(detail?.comboExecutionKey, "combo-a:0:step-openai-a");
assert.equal(detail?.pipelinePayloads?.clientRawRequest?.body?.raw, true);
assert.equal(detail?.pipelinePayloads?.providerRequest?.body?.translated, true);
assert.equal(detail?.pipelinePayloads?.providerResponse?.body?.upstream, true);
@@ -70,6 +76,9 @@ test("call logs store a single per-request artifact with pipeline details", asyn
const artifact = JSON.parse(fs.readFileSync(artifactPath, "utf8"));
assert.equal(artifact.summary.id, logId);
assert.equal(artifact.summary.requestedModel, "openai/gpt-5");
assert.equal(artifact.summary.comboName, "combo-a");
assert.equal(artifact.summary.comboStepId, "step-openai-a");
assert.equal(artifact.summary.comboExecutionKey, "combo-a:0:step-openai-a");
assert.equal(artifact.pipeline.clientRawRequest.body.raw, true);
assert.equal("sourceRequest" in artifact.pipeline, false);
});
@@ -469,3 +478,31 @@ test("saveCallLog logs and returns when sqlite persistence throws unexpectedly",
assert.match(consoleCalls[0], /Failed to save call log/);
assert.match(consoleCalls[0], /simulated sqlite prepare failure/);
});
test("getCallLogs and getCallLogById expose combo target identifiers", async () => {
await callLogs.saveCallLog({
id: "combo-target-log",
timestamp: "2026-03-31T08:15:00.000Z",
method: "POST",
path: "/v1/chat/completions",
status: 503,
model: "openai/gpt-4o-mini",
requestedModel: "router-fixed-accounts",
provider: "openai",
connectionId: "conn-fixed-2",
comboName: "router-fixed-accounts",
comboStepId: "step-openai-secondary",
comboExecutionKey: "router-fixed-accounts:1:step-openai-secondary",
error: "upstream unavailable",
});
const logs = await callLogs.getCallLogs({ search: "step-openai-secondary" });
assert.equal(logs.length, 1);
assert.equal(logs[0].comboStepId, "step-openai-secondary");
assert.equal(logs[0].comboExecutionKey, "router-fixed-accounts:1:step-openai-secondary");
const detail = await callLogs.getCallLogById("combo-target-log");
assert.equal(detail?.comboName, "router-fixed-accounts");
assert.equal(detail?.comboStepId, "step-openai-secondary");
assert.equal(detail?.comboExecutionKey, "router-fixed-accounts:1:step-openai-secondary");
});

47
tests/unit/codex-quota-fetcher.test.mjs
View File

@@ -10,15 +10,19 @@ import {
} from "../../open-sse/services/codexQuotaFetcher.ts";
import { preflightQuota } from "../../open-sse/services/quotaPreflight.ts";
import {
clearQuotaMonitors,
getActiveMonitorCount,
startQuotaMonitor,
stopQuotaMonitor,
} from "../../open-sse/services/quotaMonitor.ts";
import { clearSessions, touchSession } from "../../open-sse/services/sessionManager.ts";
const originalFetch = globalThis.fetch;
test.afterEach(() => {
globalThis.fetch = originalFetch;
clearQuotaMonitors();
clearSessions();
});
test("fetchCodexQuota returns null when no registered credentials exist", async () => {
@@ -26,6 +30,48 @@ test("fetchCodexQuota returns null when no registered credentials exist", async
assert.equal(quota, null);
});
test("fetchCodexQuota can read credentials directly from the provided connection snapshot", async () => {
const connectionId = `codex-inline-${Date.now()}`;
const calls = [];
globalThis.fetch = async (url, init) => {
calls.push({ url, init });
return new Response(
JSON.stringify({
rate_limit: {
primary_window: {
used_percent: 70,
reset_after_seconds: 45,
},
secondary_window: {
used_percent: 20,
reset_after_seconds: 300,
},
},
}),
{
status: 200,
headers: { "content-type": "application/json" },
}
);
};
const quota = await fetchCodexQuota(connectionId, {
accessToken: "inline-token",
providerSpecificData: {
workspaceId: "workspace-inline",
},
});
assert.equal(calls.length, 1);
assert.equal(calls[0].init.headers.Authorization, "Bearer inline-token");
assert.equal(calls[0].init.headers["chatgpt-account-id"], "workspace-inline");
assert.equal(quota.percentUsed, 0.7);
assert.ok(typeof quota.resetAt === "string");
invalidateCodexQuotaCache(connectionId);
});
test("fetchCodexQuota parses dual-window usage, forwards workspace headers, and caches results", async () => {
const connectionId = `codex-cache-${Date.now()}`;
const calls = [];
@@ -141,6 +187,7 @@ test("registerCodexQuotaFetcher exposes Codex quota to preflight and monitor flo
providerSpecificData: { quotaPreflightEnabled: true },
});
touchSession("session-codex", connectionId);
startQuotaMonitor("session-codex", "codex", connectionId, {
providerSpecificData: { quotaMonitorEnabled: true },
});

186
tests/unit/combo-builder-draft.test.mjs Normal file
View File

@@ -0,0 +1,186 @@
import test from "node:test";
import assert from "node:assert/strict";
const builderDraft = await import("../../src/lib/combos/builderDraft.ts");
test("parseQualifiedModel keeps provider prefix and the full tail model id", () => {
assert.deepEqual(builderDraft.parseQualifiedModel("openrouter/openai/gpt-5.4"), {
providerId: "openrouter",
modelId: "openai/gpt-5.4",
});
assert.deepEqual(builderDraft.parseQualifiedModel("codex/gpt-5.3-codex"), {
providerId: "codex",
modelId: "gpt-5.3-codex",
});
assert.equal(builderDraft.parseQualifiedModel("combo-only"), null);
});
test("buildPrecisionComboModelStep preserves provider/model/account triple", () => {
assert.deepEqual(
builderDraft.buildPrecisionComboModelStep({
providerId: "codex",
modelId: "gpt-5.3-codex",
connectionId: "conn-codex-a",
connectionLabel: "Codex A",
weight: 35,
}),
{
kind: "model",
providerId: "codex",
model: "codex/gpt-5.3-codex",
connectionId: "conn-codex-a",
label: "Codex A",
weight: 35,
}
);
});
test("hasExactModelStepDuplicate blocks only exact provider/model/connection repeats", () => {
const existing = [
builderDraft.buildPrecisionComboModelStep({
providerId: "codex",
modelId: "gpt-5.3-codex",
connectionId: "conn-a",
}),
builderDraft.buildPrecisionComboModelStep({
providerId: "codex",
modelId: "gpt-5.3-codex",
connectionId: "conn-b",
}),
builderDraft.buildPrecisionComboModelStep({
providerId: "codex",
modelId: "gpt-5.3-codex",
}),
{ kind: "combo-ref", comboName: "fallback", weight: 0 },
];
assert.equal(
builderDraft.hasExactModelStepDuplicate(
existing,
builderDraft.buildPrecisionComboModelStep({
providerId: "codex",
modelId: "gpt-5.3-codex",
connectionId: "conn-a",
})
),
true
);
assert.equal(
builderDraft.hasExactModelStepDuplicate(
existing,
builderDraft.buildPrecisionComboModelStep({
providerId: "codex",
modelId: "gpt-5.3-codex",
connectionId: "conn-c",
})
),
false
);
assert.equal(
builderDraft.hasExactModelStepDuplicate(
existing,
builderDraft.buildPrecisionComboModelStep({
providerId: "codex",
modelId: "gpt-5.3-codex",
})
),
true
);
});
test("findNextSuggestedConnectionId advances to the next unused connection for the same model", () => {
const existing = [
builderDraft.buildPrecisionComboModelStep({
providerId: "codex",
modelId: "gpt-5.3-codex",
connectionId: "conn-a",
}),
builderDraft.buildPrecisionComboModelStep({
providerId: "codex",
modelId: "gpt-5.3-codex",
connectionId: "conn-b",
}),
];
assert.equal(
builderDraft.findNextSuggestedConnectionId(existing, "codex", "gpt-5.3-codex", [
{ id: "conn-a" },
{ id: "conn-b" },
{ id: "conn-c" },
]),
"conn-c"
);
assert.equal(
builderDraft.findNextSuggestedConnectionId(existing, "codex", "gpt-5.3-codex", [
{ id: "conn-a" },
{ id: "conn-b" },
]),
builderDraft.COMBO_BUILDER_AUTO_CONNECTION
);
});
test("combo builder stage helpers expose completion state and linear navigation", () => {
assert.deepEqual(
builderDraft.getComboBuilderStageChecks({
name: "codex-stack",
nameError: "",
modelsCount: 2,
hasInvalidWeightedTotal: false,
hasCostOptimizedWithoutPricing: false,
}),
{
basics: true,
steps: true,
strategy: true,
review: false,
}
);
assert.deepEqual(
builderDraft.getComboBuilderStageChecks({
name: "",
nameError: "Required",
modelsCount: 0,
hasInvalidWeightedTotal: true,
hasCostOptimizedWithoutPricing: false,
}),
{
basics: false,
steps: false,
strategy: false,
review: false,
}
);
assert.equal(builderDraft.getNextComboBuilderStage("basics"), "steps");
assert.equal(builderDraft.getNextComboBuilderStage("steps"), "strategy");
assert.equal(builderDraft.getNextComboBuilderStage("strategy"), "review");
assert.equal(builderDraft.getNextComboBuilderStage("review"), "review");
assert.equal(builderDraft.getPreviousComboBuilderStage("review"), "strategy");
assert.equal(builderDraft.getPreviousComboBuilderStage("basics"), "basics");
const checks = builderDraft.getComboBuilderStageChecks({
name: "codex-stack",
nameError: "",
modelsCount: 1,
hasInvalidWeightedTotal: true,
hasCostOptimizedWithoutPricing: false,
});
assert.equal(builderDraft.canAccessComboBuilderStage("basics", checks), true);
assert.equal(builderDraft.canAccessComboBuilderStage("steps", checks), true);
assert.equal(builderDraft.canAccessComboBuilderStage("strategy", checks), true);
assert.equal(builderDraft.canAccessComboBuilderStage("review", checks), true);
const lockedChecks = builderDraft.getComboBuilderStageChecks({
name: "",
nameError: "Required",
modelsCount: 0,
hasInvalidWeightedTotal: false,
hasCostOptimizedWithoutPricing: false,
});
assert.equal(builderDraft.canAccessComboBuilderStage("steps", lockedChecks), false);
assert.equal(builderDraft.canAccessComboBuilderStage("strategy", lockedChecks), false);
assert.equal(builderDraft.canAccessComboBuilderStage("review", lockedChecks), false);
});

228
tests/unit/combo-builder-options-route.test.mjs Normal file
View File

@@ -0,0 +1,228 @@
import test from "node:test";
import assert from "node:assert/strict";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-combo-builder-options-"));
process.env.DATA_DIR = TEST_DATA_DIR;
const core = await import("../../src/lib/db/core.ts");
const providersDb = await import("../../src/lib/db/providers.ts");
const modelsDb = await import("../../src/lib/db/models.ts");
const combosDb = await import("../../src/lib/db/combos.ts");
const modelsDevSync = await import("../../src/lib/modelsDevSync.ts");
const route = await import("../../src/app/api/combos/builder/options/route.ts");
async function resetStorage() {
core.resetDbInstance();
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
fs.mkdirSync(TEST_DATA_DIR, { recursive: true });
}
async function seedConnection(provider, overrides = {}) {
const payload = {
provider,
authType: overrides.authType || "apikey",
displayName: overrides.displayName,
email: overrides.email,
apiKey: overrides.apiKey === undefined ? "sk-test" : overrides.apiKey,
accessToken: overrides.accessToken,
isActive: overrides.isActive ?? true,
priority: overrides.priority || 1,
testStatus: overrides.testStatus || "active",
rateLimitedUntil: overrides.rateLimitedUntil,
defaultModel: overrides.defaultModel,
providerSpecificData: overrides.providerSpecificData || {},
};
if (overrides.name !== undefined) {
payload.name = overrides.name;
} else if ((overrides.authType || "apikey") !== "oauth") {
payload.name = `${provider}-${Math.random().toString(16).slice(2, 8)}`;
}
return providersDb.createProviderConnection(payload);
}
test.beforeEach(async () => {
await resetStorage();
});
test.after(() => {
core.resetDbInstance();
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
});
test("combo builder options route aggregates providers, connections, models and combo refs", async () => {
const nowPlusMinute = Date.now() + 60_000;
modelsDevSync.saveModelsDevCapabilities({
openai: {
"gpt-4o": {
tool_call: true,
reasoning: false,
attachment: true,
structured_output: true,
temperature: true,
modalities_input: JSON.stringify(["text", "image"]),
modalities_output: JSON.stringify(["text"]),
knowledge_cutoff: "2024-10",
release_date: "2024-05-13",
last_updated: "2024-10-01",
status: "stable",
family: "gpt-4",
open_weights: false,
limit_context: 128000,
limit_input: 128000,
limit_output: 16384,
interleaved_field: null,
},
},
});
await seedConnection("openai", {
name: "OpenAI Primary",
priority: 2,
defaultModel: "gpt-4o",
});
await seedConnection("openai", {
authType: "oauth",
email: "disabled@example.com",
accessToken: "oauth-token",
isActive: false,
priority: 1,
testStatus: "failed",
});
await seedConnection("codex", {
authType: "oauth",
email: "codex@example.com",
accessToken: "codex-token",
rateLimitedUntil: nowPlusMinute,
defaultModel: "gpt-5.3-codex",
});
await modelsDb.addCustomModel("openai", "custom-ops", "Custom Ops");
await modelsDb.addCustomModel(
"openai",
"text-embedding-hidden",
"Hidden Embedding",
"manual",
"chat-completions",
["embeddings"]
);
modelsDb.mergeModelCompatOverride("openai", "gpt-4o-mini", { isHidden: true });
const visibleCombo = await combosDb.createCombo({
name: "team-router",
strategy: "priority",
models: ["openai/gpt-4o"],
});
await combosDb.createCombo({
name: "hidden-router",
strategy: "priority",
models: ["openai/gpt-4o"],
isHidden: true,
});
const response = await route.GET();
const body = await response.json();
assert.equal(response.status, 200);
assert.equal(body.schemaVersion, 2);
assert.ok(Array.isArray(body.providers));
assert.ok(Array.isArray(body.comboRefs));
const openai = body.providers.find((provider) => provider.providerId === "openai");
const codex = body.providers.find((provider) => provider.providerId === "codex");
assert.ok(openai);
assert.equal(openai.displayName, "OpenAI");
assert.equal(openai.connectionCount, 2);
assert.equal(openai.activeConnectionCount, 1);
assert.ok(openai.models.some((model) => model.id === "gpt-4o"));
assert.equal(openai.models.find((model) => model.id === "gpt-4o").outputTokenLimit, 16384);
assert.equal(openai.models.find((model) => model.id === "gpt-4o").supportsThinking, false);
assert.equal(
openai.models.some((model) => model.id === "gpt-4o-mini"),
false
);
assert.ok(openai.models.some((model) => model.id === "custom-ops"));
assert.equal(
openai.models.some((model) => model.id === "text-embedding-hidden"),
false
);
assert.deepEqual(
openai.connections.map((connection) => ({
label: connection.label,
status: connection.status,
isActive: connection.isActive,
})),
[
{
label: "OpenAI Primary",
status: "active",
isActive: true,
},
{
label: "disabled@example.com",
status: "inactive",
isActive: false,
},
]
);
assert.equal(
openai.connections.some((connection) =>
Object.prototype.hasOwnProperty.call(connection, "apiKey")
),
false
);
assert.equal(
openai.connections.some((connection) =>
Object.prototype.hasOwnProperty.call(connection, "accessToken")
),
false
);
assert.ok(codex);
assert.equal(codex.connections[0].status, "rate-limited");
assert.equal(codex.connections[0].defaultModel, "gpt-5.3-codex");
assert.deepEqual(
body.comboRefs.map((combo) => combo.name),
[visibleCombo.name]
);
});
test("combo builder options route exposes compatible provider nodes with node metadata", async () => {
await providersDb.createProviderNode({
id: "openai-compatible-demo",
type: "openai-compatible",
name: "Gateway Demo",
prefix: "gd",
baseUrl: "https://proxy.example.com",
chatPath: "/v1/chat/completions",
modelsPath: "/v1/models",
});
await seedConnection("openai-compatible-demo", {
name: "Gateway Account",
providerSpecificData: {
baseUrl: "https://proxy.example.com",
},
});
await modelsDb.addCustomModel("openai-compatible-demo", "gpt-custom", "GPT Custom");
const response = await route.GET();
const body = await response.json();
const provider = body.providers.find((entry) => entry.providerId === "openai-compatible-demo");
assert.equal(response.status, 200);
assert.ok(provider);
assert.equal(provider.displayName, "Gateway Demo");
assert.equal(provider.providerType, "openai-compatible");
assert.equal(provider.alias, "gd");
assert.equal(provider.prefix, "gd");
assert.equal(provider.source, "provider-node");
assert.equal(provider.acceptsArbitraryModel, true);
assert.ok(provider.models.some((model) => model.id === "gpt-custom"));
assert.equal(provider.models[0].qualifiedModel, "openai-compatible-demo/gpt-custom");
});

121
tests/unit/combo-config.test.mjs
View File

@@ -3,7 +3,8 @@ import assert from "node:assert/strict";
const { resolveComboConfig, getDefaultComboConfig } =
await import("../../open-sse/services/comboConfig.ts");
const { createComboSchema } = await import("../../src/shared/validation/schemas.ts");
const { createComboSchema, updateComboDefaultsSchema } =
await import("../../src/shared/validation/schemas.ts");
test("getDefaultComboConfig returns a fresh copy of the defaults", () => {
const first = getDefaultComboConfig();
@@ -133,3 +134,121 @@ test("createComboSchema accepts context-relay strategy with handoff config", ()
assert.equal(parsed.config.handoffThreshold, 0.85);
assert.equal(parsed.config.maxMessagesForSummary, 24);
});
test("createComboSchema accepts structured combo steps with pinned connection and combo refs", () => {
const parsed = createComboSchema.parse({
name: "codex-pinned",
strategy: "priority",
models: [
{
kind: "model",
id: "step-codex-a",
providerId: "codex",
model: "gpt-5.4",
connectionId: "conn-codex-a",
weight: 10,
},
{
kind: "combo-ref",
id: "step-fallback",
comboName: "backup-codex",
weight: 5,
},
],
});
assert.equal(parsed.models[0].kind, "model");
assert.equal(parsed.models[0].providerId, "codex");
assert.equal(parsed.models[0].connectionId, "conn-codex-a");
assert.equal(parsed.models[1].kind, "combo-ref");
assert.equal(parsed.models[1].comboName, "backup-codex");
});
test("createComboSchema accepts composite tiers that reference normalized combo steps", () => {
const parsed = createComboSchema.parse({
name: "tiered-codex",
strategy: "priority",
models: [
{
kind: "model",
id: "step-primary",
providerId: "codex",
model: "gpt-5.4",
connectionId: "conn-codex-a",
},
{
kind: "model",
id: "step-backup",
providerId: "codex",
model: "gpt-5.4",
connectionId: "conn-codex-b",
},
],
config: {
compositeTiers: {
defaultTier: "primary",
tiers: {
primary: {
stepId: "step-primary",
fallbackTier: "backup",
label: "Codex A",
},
backup: {
stepId: "step-backup",
description: "Fallback account",
},
},
},
},
});
assert.equal(parsed.config.compositeTiers.defaultTier, "primary");
assert.equal(parsed.config.compositeTiers.tiers.primary.stepId, "step-primary");
assert.equal(parsed.config.compositeTiers.tiers.primary.fallbackTier, "backup");
assert.equal(parsed.config.compositeTiers.tiers.backup.stepId, "step-backup");
});
test("updateComboDefaultsSchema rejects composite tiers in global defaults and provider overrides", () => {
const result = updateComboDefaultsSchema.safeParse({
comboDefaults: {
compositeTiers: {
defaultTier: "primary",
tiers: {
primary: {
stepId: "step-primary",
},
},
},
},
providerOverrides: {
codex: {
compositeTiers: {
defaultTier: "backup",
tiers: {
backup: {
stepId: "step-backup",
},
},
},
},
},
});
assert.equal(result.success, false);
assert.deepEqual(
result.error.issues.map((issue) => ({
path: issue.path.join("."),
message: issue.message,
})),
[
{
path: "comboDefaults.compositeTiers",
message: "compositeTiers is only supported on concrete combos",
},
{
path: "providerOverrides.codex.compositeTiers",
message: "compositeTiers is only supported on concrete combos",
},
]
);
});

266
tests/unit/combo-health-route.test.mjs Normal file
View File

@@ -0,0 +1,266 @@
import test from "node:test";
import assert from "node:assert/strict";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-combo-health-route-"));
process.env.DATA_DIR = TEST_DATA_DIR;
const core = await import("../../src/lib/db/core.ts");
const combosDb = await import("../../src/lib/db/combos.ts");
const callLogs = await import("../../src/lib/usage/callLogs.ts");
const quotaSnapshotsDb = await import("../../src/lib/db/quotaSnapshots.ts");
const comboMetrics = await import("../../open-sse/services/comboMetrics.ts");
const route = await import("../../src/app/api/usage/combo-health/route.ts");
const { normalizeComboStep } = await import("../../src/lib/combos/steps.ts");
async function resetStorage() {
comboMetrics.resetAllComboMetrics();
core.resetDbInstance();
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
fs.mkdirSync(TEST_DATA_DIR, { recursive: true });
}
test.beforeEach(async () => {
await resetStorage();
});
test.after(() => {
comboMetrics.resetAllComboMetrics();
core.resetDbInstance();
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
});
test("combo health route exposes step-level target health for structured combos", async () => {
const comboInput = {
name: "combo-health-structured",
strategy: "priority",
models: [
{
kind: "model",
providerId: "openai",
model: "openai/gpt-4o-mini",
connectionId: "conn-openai-a",
label: "Account A",
},
{
kind: "model",
providerId: "openai",
model: "openai/gpt-4o-mini",
connectionId: "conn-openai-b",
label: "Account B",
},
],
};
const combo = await combosDb.createCombo(comboInput);
const firstStep = normalizeComboStep(comboInput.models[0], {
comboName: comboInput.name,
index: 0,
});
const secondStep = normalizeComboStep(comboInput.models[1], {
comboName: comboInput.name,
index: 1,
});
quotaSnapshotsDb.saveQuotaSnapshot({
provider: "openai",
connection_id: "conn-openai-a",
window_key: "daily",
remaining_percentage: 60,
is_exhausted: 0,
next_reset_at: null,
window_duration_ms: 86_400_000,
raw_data: null,
});
quotaSnapshotsDb.saveQuotaSnapshot({
provider: "openai",
connection_id: "conn-openai-b",
window_key: "daily",
remaining_percentage: 85,
is_exhausted: 0,
next_reset_at: null,
window_duration_ms: 86_400_000,
raw_data: null,
});
comboMetrics.recordComboRequest(comboInput.name, "openai/gpt-4o-mini", {
success: false,
latencyMs: 240,
strategy: "priority",
target: {
executionKey: firstStep.id,
stepId: firstStep.id,
provider: "openai",
connectionId: "conn-openai-a",
label: "Account A",
},
});
comboMetrics.recordComboRequest(comboInput.name, "openai/gpt-4o-mini", {
success: true,
latencyMs: 120,
strategy: "priority",
target: {
executionKey: secondStep.id,
stepId: secondStep.id,
provider: "openai",
connectionId: "conn-openai-b",
label: "Account B",
},
});
const response = await route.GET(
new Request(`http://localhost/api/usage/combo-health?range=24h&comboId=${combo.id}`)
);
const body = await response.json();
assert.equal(response.status, 200);
assert.equal(body.combos.length, 1);
assert.deepEqual(body.combos[0].models, ["openai/gpt-4o-mini", "openai/gpt-4o-mini"]);
assert.equal(body.combos[0].targetHealth.length, 2);
assert.deepEqual(
body.combos[0].targetHealth.map((entry) => ({
executionKey: entry.executionKey,
connectionId: entry.connectionId,
label: entry.label,
requests: entry.requests,
successRate: entry.successRate,
quotaRemainingPct: entry.quotaRemainingPct,
quotaScope: entry.quotaScope,
})),
[
{
executionKey: firstStep.id,
connectionId: "conn-openai-a",
label: "Account A",
requests: 1,
successRate: 0,
quotaRemainingPct: 60,
quotaScope: "connection",
},
{
executionKey: secondStep.id,
connectionId: "conn-openai-b",
label: "Account B",
requests: 1,
successRate: 100,
quotaRemainingPct: 85,
quotaScope: "connection",
},
]
);
});
test("combo health route prefers historical call log target metrics over volatile runtime memory", async () => {
const comboInput = {
name: "combo-health-history",
strategy: "priority",
models: [
{
kind: "model",
providerId: "openai",
model: "openai/gpt-4o-mini",
connectionId: "conn-openai-a",
label: "Account A",
},
{
kind: "model",
providerId: "openai",
model: "openai/gpt-4o-mini",
connectionId: "conn-openai-b",
label: "Account B",
},
],
};
const combo = await combosDb.createCombo(comboInput);
const firstStep = normalizeComboStep(comboInput.models[0], {
comboName: comboInput.name,
index: 0,
});
const secondStep = normalizeComboStep(comboInput.models[1], {
comboName: comboInput.name,
index: 1,
});
const firstTimestamp = new Date(Date.now() - 5 * 60 * 1000).toISOString();
const secondTimestamp = new Date(Date.now() - 4 * 60 * 1000).toISOString();
await callLogs.saveCallLog({
id: "combo-history-1",
timestamp: firstTimestamp,
method: "POST",
path: "/v1/chat/completions",
status: 503,
model: "openai/gpt-4o-mini",
requestedModel: comboInput.name,
provider: "openai",
connectionId: "conn-openai-a",
duration: 240,
comboName: comboInput.name,
comboStepId: firstStep.id,
comboExecutionKey: firstStep.id,
error: "first account unavailable",
});
await callLogs.saveCallLog({
id: "combo-history-2",
timestamp: secondTimestamp,
method: "POST",
path: "/v1/chat/completions",
status: 200,
model: "openai/gpt-4o-mini",
requestedModel: comboInput.name,
provider: "openai",
connectionId: "conn-openai-b",
duration: 120,
comboName: comboInput.name,
comboStepId: secondStep.id,
comboExecutionKey: secondStep.id,
});
comboMetrics.recordComboRequest(comboInput.name, "openai/gpt-4o-mini", {
success: true,
latencyMs: 10,
strategy: "priority",
target: {
executionKey: firstStep.id,
stepId: firstStep.id,
provider: "openai",
connectionId: "conn-openai-a",
label: "Account A",
},
});
const response = await route.GET(
new Request(`http://localhost/api/usage/combo-health?range=24h&comboId=${combo.id}`)
);
const body = await response.json();
assert.equal(response.status, 200);
assert.equal(body.combos.length, 1);
assert.deepEqual(
body.combos[0].targetHealth.map((entry) => ({
executionKey: entry.executionKey,
requests: entry.requests,
successRate: entry.successRate,
avgLatencyMs: entry.avgLatencyMs,
lastStatus: entry.lastStatus,
})),
[
{
executionKey: firstStep.id,
requests: 1,
successRate: 0,
avgLatencyMs: 240,
lastStatus: "error",
},
{
executionKey: secondStep.id,
requests: 1,
successRate: 100,
avgLatencyMs: 120,
lastStatus: "ok",
},
]
);
});

157
tests/unit/combo-routes-composite-tiers.test.mjs Normal file
View File

@@ -0,0 +1,157 @@
import test from "node:test";
import assert from "node:assert/strict";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-combo-composite-tiers-"));
process.env.DATA_DIR = TEST_DATA_DIR;
const core = await import("../../src/lib/db/core.ts");
const combosDb = await import("../../src/lib/db/combos.ts");
const createRoute = await import("../../src/app/api/combos/route.ts");
const comboRoute = await import("../../src/app/api/combos/[id]/route.ts");
async function resetStorage() {
core.resetDbInstance();
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
fs.mkdirSync(TEST_DATA_DIR, { recursive: true });
}
function makeCreateRequest(body) {
return new Request("http://localhost/api/combos", {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify(body),
});
}
function makeUpdateRequest(body) {
return new Request("http://localhost/api/combos/combo-1", {
method: "PUT",
headers: { "content-type": "application/json" },
body: JSON.stringify(body),
});
}
function createTieredComboInput() {
return {
name: "tiered-codex",
strategy: "priority",
models: [
{
kind: "model",
id: "step-primary",
providerId: "codex",
model: "gpt-5.4",
connectionId: "conn-codex-a",
},
{
kind: "model",
id: "step-backup",
providerId: "codex",
model: "gpt-5.4",
connectionId: "conn-codex-b",
},
],
config: {
compositeTiers: {
defaultTier: "primary",
tiers: {
primary: {
stepId: "step-primary",
fallbackTier: "backup",
},
backup: {
stepId: "step-backup",
},
},
},
},
};
}
test.beforeEach(async () => {
await resetStorage();
});
test.after(() => {
core.resetDbInstance();
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
});
test("POST /api/combos rejects composite tiers that point to unknown steps", async () => {
const response = await createRoute.POST(
makeCreateRequest({
...createTieredComboInput(),
config: {
compositeTiers: {
defaultTier: "primary",
tiers: {
primary: {
stepId: "step-missing",
},
},
},
},
})
);
const body = await response.json();
assert.equal(response.status, 400);
assert.deepEqual(body, {
error: {
message: "Invalid composite tiers",
details: [
{
field: "config.compositeTiers.tiers.primary.stepId",
message: 'stepId "step-missing" does not exist in combo.models',
},
],
},
});
});
test("POST /api/combos persists valid composite tiers", async () => {
const response = await createRoute.POST(makeCreateRequest(createTieredComboInput()));
const body = await response.json();
const stored = await combosDb.getComboByName("tiered-codex");
assert.equal(response.status, 201);
assert.equal(body.config.compositeTiers.defaultTier, "primary");
assert.equal(stored.config.compositeTiers.tiers.primary.stepId, "step-primary");
assert.equal(stored.models[0].id, "step-primary");
});
test("PUT /api/combos rejects updates that orphan an existing composite tier step reference", async () => {
const combo = await combosDb.createCombo(createTieredComboInput());
const response = await comboRoute.PUT(
makeUpdateRequest({
models: [
{
kind: "model",
id: "step-backup",
providerId: "codex",
model: "gpt-5.4",
connectionId: "conn-codex-b",
},
],
}),
{ params: Promise.resolve({ id: combo.id }) }
);
const body = await response.json();
assert.equal(response.status, 400);
assert.deepEqual(body, {
error: {
message: "Invalid composite tiers",
details: [
{
field: "config.compositeTiers.tiers.primary.stepId",
message: 'stepId "step-primary" does not exist in combo.models',
},
],
},
});
});

132
tests/unit/combo-routing-engine.test.mjs
View File

@@ -15,6 +15,7 @@ const {
handleComboChat,
shouldFallbackComboBadRequest,
} = await import("../../open-sse/services/combo.ts");
const { normalizeComboStep } = await import("../../src/lib/combos/steps.ts");
const { registerStrategy } = await import("../../open-sse/services/autoCombo/routerStrategy.ts");
const core = await import("../../src/lib/db/core.ts");
const settingsDb = await import("../../src/lib/db/settings.ts");
@@ -81,6 +82,12 @@ function capabilityEntry(limitContext) {
};
}
function getComboTargetBreakerKey(comboName, index, stepInput) {
const step = normalizeComboStep(stepInput, { comboName, index });
if (!step) throw new Error(`Failed to normalize combo step for ${comboName}#${index}`);
return `combo:${comboName}:${step.id}`;
}
async function resetStorage() {
core.resetDbInstance();
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
@@ -144,6 +151,30 @@ test("validateComboDAG rejects circular references and resolveNestedComboModels
);
});
test("resolveNestedComboModels expands explicit combo-ref steps", () => {
const combos = [
{
name: "root",
models: [
{ id: "root-ref-child", kind: "combo-ref", comboName: "child", weight: 0 },
{ id: "root-model", kind: "model", providerId: "openai", model: "openai/gpt-4o-mini" },
],
},
{
name: "child",
models: [
{ id: "child-model", kind: "model", providerId: "anthropic", model: "claude/sonnet" },
],
},
];
validateComboDAG("root", combos);
assert.deepEqual(resolveNestedComboModels(combos[0], combos), [
"claude/sonnet",
"openai/gpt-4o-mini",
]);
});
test("validateComboDAG enforces maximum nesting depth", () => {
const combos = [
{ name: "c1", models: ["c2"] },
@@ -177,12 +208,18 @@ test("handleComboChat priority strategy defaults to first model and records succ
});
const metrics = getComboMetrics("priority-default");
const firstStep = normalizeComboStep(combo.models[0], {
comboName: combo.name,
index: 0,
});
assert.equal(result.ok, true);
assert.deepEqual(calls, ["openai/gpt-4o-mini"]);
assert.equal(metrics.totalRequests, 1);
assert.equal(metrics.totalSuccesses, 1);
assert.equal(metrics.byModel["openai/gpt-4o-mini"].requests, 1);
assert.equal(metrics.byTarget[firstStep.id].requests, 1);
assert.equal(metrics.byTarget[firstStep.id].model, "openai/gpt-4o-mini");
assert.equal(metrics.strategy, "priority");
});
@@ -384,6 +421,65 @@ test("handleComboChat falls through empty successful responses and records failu
assert.equal(metrics.byModel["model-b"].lastStatus, "ok");
});
test("handleComboChat records per-target metrics separately when the same model repeats with different accounts", async () => {
const calls = [];
const combo = {
name: "per-target-repeat",
strategy: "priority",
models: [
{
kind: "model",
providerId: "openai",
model: "openai/gpt-4o-mini",
connectionId: "conn-openai-a",
label: "Account A",
},
{
kind: "model",
providerId: "openai",
model: "openai/gpt-4o-mini",
connectionId: "conn-openai-b",
label: "Account B",
},
],
config: { maxRetries: 0 },
};
const result = await handleComboChat({
body: {},
combo,
handleSingleModel: async (_body, modelStr, target) => {
calls.push(`${modelStr}:${target?.connectionId || "none"}`);
if (target?.connectionId === "conn-openai-a") {
return errorResponse(503, "account-a down");
}
return okResponse();
},
isModelAvailable: async () => true,
log: createLog(),
settings: null,
allCombos: null,
});
const firstStep = normalizeComboStep(combo.models[0], {
comboName: combo.name,
index: 0,
});
const secondStep = normalizeComboStep(combo.models[1], {
comboName: combo.name,
index: 1,
});
const metrics = getComboMetrics(combo.name);
assert.equal(result.ok, true);
assert.deepEqual(calls, ["openai/gpt-4o-mini:conn-openai-a", "openai/gpt-4o-mini:conn-openai-b"]);
assert.equal(metrics.byModel["openai/gpt-4o-mini"].requests, 2);
assert.equal(metrics.byTarget[firstStep.id].failures, 1);
assert.equal(metrics.byTarget[firstStep.id].connectionId, "conn-openai-a");
assert.equal(metrics.byTarget[secondStep.id].successes, 1);
assert.equal(metrics.byTarget[secondStep.id].connectionId, "conn-openai-b");
});
test("handleComboChat preserves the first failure status but surfaces the last error message", async () => {
const result = await handleComboChat({
body: {},
@@ -674,11 +770,17 @@ test("handleComboChat round-robin returns 503 when no models are configured", as
});
assert.equal(result.status, 503);
assert.match((await result.json()).error.message, /Round-robin combo has no models/);
assert.match((await result.json()).error.message, /Round-robin combo has no executable targets/);
});
test("handleComboChat round-robin falls through semaphore timeouts and malformed success payloads", async () => {
const release = await acquireSemaphore("model-a", { maxConcurrency: 1, timeoutMs: 100 });
const release = await acquireSemaphore(
getComboTargetBreakerKey("rr-timeout-fallback", 0, "model-a"),
{
maxConcurrency: 1,
timeoutMs: 100,
}
);
const calls = [];
try {
@@ -995,11 +1097,14 @@ test("handleComboChat returns a 503 when every model is unavailable before execu
});
test("handleComboChat returns the circuit-breaker unavailable response when all breakers are open", async () => {
for (const modelStr of ["openai/model-a", "openai/model-b"]) {
const breaker = getCircuitBreaker(`combo:${modelStr}`, {
failureThreshold: 1,
resetTimeout: 60000,
});
for (const [index, modelStr] of ["openai/model-a", "openai/model-b"].entries()) {
const breaker = getCircuitBreaker(
getComboTargetBreakerKey("all-breakers-open", index, modelStr),
{
failureThreshold: 1,
resetTimeout: 60000,
}
);
breaker._onFailure();
}
@@ -1315,11 +1420,14 @@ test("handleComboChat round-robin resolves nested combos and returns inactive wh
});
test("handleComboChat round-robin returns circuit-breaker unavailable when every model is open", async () => {
for (const modelStr of ["openai/model-a", "openai/model-b"]) {
const breaker = getCircuitBreaker(`combo:${modelStr}`, {
failureThreshold: 1,
resetTimeout: 60000,
});
for (const [index, modelStr] of ["openai/model-a", "openai/model-b"].entries()) {
const breaker = getCircuitBreaker(
getComboTargetBreakerKey("rr-breakers-open", index, modelStr),
{
failureThreshold: 1,
resetTimeout: 60000,
}
);
breaker._onFailure();
}

58
tests/unit/combo-test-route.test.mjs
View File

@@ -302,6 +302,64 @@ test("combo test route launches model probes concurrently while preserving combo
);
});
test("combo test route preserves structured step metadata for repeated model/account targets", async () => {
await createTestCombo([
{
kind: "model",
providerId: "openai",
model: "openai/gpt-4o-mini",
connectionId: "conn-openai-a",
label: "Account A",
},
{
kind: "model",
providerId: "openai",
model: "openai/gpt-4o-mini",
connectionId: "conn-openai-b",
label: "Account B",
},
]);
const fetchCalls = [];
globalThis.fetch = async (url, init = {}) => {
fetchCalls.push({ url: String(url), init });
const body = JSON.parse(init.body);
return new Response(
JSON.stringify({
choices: [
{
message: {
role: "assistant",
content: `OK:${body.model}`,
},
},
],
}),
{
status: 200,
headers: { "content-type": "application/json" },
}
);
};
const response = await route.POST(makeRequest());
const body = await response.json();
assert.equal(response.status, 200);
assert.equal(fetchCalls.length, 2);
assert.deepEqual(
fetchCalls.map(({ init }) => JSON.parse(init.body).model),
["openai/gpt-4o-mini", "openai/gpt-4o-mini"]
);
assert.equal(body.results[0].connectionId, "conn-openai-a");
assert.equal(body.results[0].label, "Account A");
assert.equal(body.results[1].connectionId, "conn-openai-b");
assert.equal(body.results[1].label, "Account B");
assert.notEqual(body.results[0].executionKey, body.results[1].executionKey);
assert.equal(body.resolvedByExecutionKey, body.results[0].executionKey);
assert.equal(body.resolvedByTarget.connectionId, "conn-openai-a");
});
test("combo test route rejects empty combos and respects forwarded base URLs", async () => {
await createTestCombo([]);

131
tests/unit/composite-tiers-validation.test.mjs Normal file
View File

@@ -0,0 +1,131 @@
import test from "node:test";
import assert from "node:assert/strict";
const { validateCompositeTiersConfig } = await import("../../src/lib/combos/compositeTiers.ts");
function createComboInput(overrides = {}) {
return {
name: "tiered-codex",
strategy: "priority",
models: [
{
kind: "model",
id: "step-primary",
providerId: "codex",
model: "gpt-5.4",
connectionId: "conn-codex-a",
},
{
kind: "model",
id: "step-backup",
providerId: "codex",
model: "gpt-5.4",
connectionId: "conn-codex-b",
},
],
config: {
compositeTiers: {
defaultTier: "primary",
tiers: {
primary: {
stepId: "step-primary",
fallbackTier: "backup",
},
backup: {
stepId: "step-backup",
},
},
},
},
...overrides,
};
}
test("validateCompositeTiersConfig accepts a valid tier graph that references combo steps", () => {
const result = validateCompositeTiersConfig(createComboInput());
assert.deepEqual(result, { success: true });
});
test("validateCompositeTiersConfig rejects tiers that point to missing step ids", () => {
const result = validateCompositeTiersConfig(
createComboInput({
config: {
compositeTiers: {
defaultTier: "primary",
tiers: {
primary: {
stepId: "step-missing",
},
},
},
},
})
);
assert.equal(result.success, false);
assert.deepEqual(result.error.details, [
{
field: "config.compositeTiers.tiers.primary.stepId",
message: 'stepId "step-missing" does not exist in combo.models',
},
]);
});
test("validateCompositeTiersConfig rejects duplicate step ownership across tiers", () => {
const result = validateCompositeTiersConfig(
createComboInput({
config: {
compositeTiers: {
defaultTier: "primary",
tiers: {
primary: {
stepId: "step-primary",
},
backup: {
stepId: "step-primary",
},
},
},
},
})
);
assert.equal(result.success, false);
assert.deepEqual(result.error.details, [
{
field: "config.compositeTiers.tiers.backup.stepId",
message: 'stepId "step-primary" is already assigned to tier "primary"',
},
]);
});
test("validateCompositeTiersConfig rejects fallback cycles", () => {
const result = validateCompositeTiersConfig(
createComboInput({
config: {
compositeTiers: {
defaultTier: "primary",
tiers: {
primary: {
stepId: "step-primary",
fallbackTier: "backup",
},
backup: {
stepId: "step-backup",
fallbackTier: "primary",
},
},
},
},
})
);
assert.equal(result.success, false);
assert.deepEqual(result.error.details, [
{
field: "config.compositeTiers.tiers.primary.fallbackTier",
message: "fallbackTier cycle detected: primary -> backup -> primary",
},
]);
});

72
tests/unit/db-combos-crud.test.mjs
View File

@@ -46,8 +46,18 @@ test("createCombo stores default strategy and supports lookup by id and name", a
models: [{ provider: "openai", model: "gpt-4.1" }],
});
assert.equal(combo.version, 2);
assert.equal(combo.strategy, "priority");
assert.equal(combo.sortOrder, 1);
assert.deepEqual(combo.models, [
{
id: "priority-combo-model-1-openai-gpt-4-1",
kind: "model",
providerId: "openai",
model: "openai/gpt-4.1",
weight: 0,
},
]);
assert.deepEqual(await combosDb.getComboById(combo.id), combo);
assert.deepEqual(await combosDb.getComboByName("Priority Combo"), combo);
});
@@ -135,3 +145,65 @@ test("deleteCombo reports missing ids and removes existing rows", async () => {
assert.equal(await combosDb.deleteCombo(combo.id), true);
assert.equal(await combosDb.getComboById(combo.id), null);
});
test("getCombos upgrades legacy persisted entries to version 2 and resolves combo refs", async () => {
const db = core.getDbInstance();
const now = new Date().toISOString();
db.prepare(
"INSERT INTO combos (id, name, data, sort_order, created_at, updated_at) VALUES (?, ?, ?, ?, ?, ?)"
).run(
"combo-child",
"child",
JSON.stringify({
id: "combo-child",
name: "child",
models: ["openai/gpt-4o-mini"],
strategy: "priority",
createdAt: now,
updatedAt: now,
}),
1,
now,
now
);
db.prepare(
"INSERT INTO combos (id, name, data, sort_order, created_at, updated_at) VALUES (?, ?, ?, ?, ?, ?)"
).run(
"combo-parent",
"parent",
JSON.stringify({
id: "combo-parent",
name: "parent",
models: ["child", { model: "anthropic/claude-sonnet-4", weight: 2 }],
strategy: "priority",
createdAt: now,
updatedAt: now,
}),
2,
now,
now
);
const combos = await combosDb.getCombos();
const parent = combos.find((combo) => combo.name === "parent");
assert.ok(parent);
assert.equal(parent.version, 2);
assert.deepEqual(parent.models, [
{
id: "parent-ref-1-child",
kind: "combo-ref",
comboName: "child",
weight: 0,
},
{
id: "parent-model-2-anthropic-claude-sonnet-4",
kind: "model",
providerId: "anthropic",
model: "anthropic/claude-sonnet-4",
weight: 2,
},
]);
});

129
tests/unit/db-core-init.test.mjs
View File

@@ -131,6 +131,78 @@ function createLegacySchemaDb(sqliteFile, { withData = false } = {}) {
seedDb.close();
}
function createLegacyCallLogsDb(sqliteFile) {
const seedDb = new Database(sqliteFile);
seedDb.exec(`
CREATE TABLE provider_connections (
id TEXT PRIMARY KEY,
provider TEXT NOT NULL,
auth_type TEXT,
name TEXT,
email TEXT,
priority INTEGER DEFAULT 0,
is_active INTEGER DEFAULT 1,
access_token TEXT,
refresh_token TEXT,
expires_at TEXT,
token_expires_at TEXT,
scope TEXT,
project_id TEXT,
test_status TEXT,
error_code TEXT,
last_error TEXT,
last_error_at TEXT,
last_error_type TEXT,
last_error_source TEXT,
backoff_level INTEGER DEFAULT 0,
rate_limited_until TEXT,
health_check_interval INTEGER,
last_health_check_at TEXT,
last_tested TEXT,
api_key TEXT,
id_token TEXT,
provider_specific_data TEXT,
expires_in INTEGER,
display_name TEXT,
global_priority INTEGER,
default_model TEXT,
token_type TEXT,
consecutive_use_count INTEGER DEFAULT 0,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
);
CREATE INDEX idx_pc_provider ON provider_connections(provider);
CREATE INDEX idx_pc_active ON provider_connections(is_active);
CREATE INDEX idx_pc_priority ON provider_connections(provider, priority);
CREATE TABLE call_logs (
id TEXT PRIMARY KEY,
timestamp TEXT NOT NULL,
method TEXT,
path TEXT,
status INTEGER,
model TEXT,
provider TEXT,
account TEXT,
connection_id TEXT,
duration INTEGER DEFAULT 0,
tokens_in INTEGER DEFAULT 0,
tokens_out INTEGER DEFAULT 0,
source_format TEXT,
target_format TEXT,
api_key_id TEXT,
api_key_name TEXT,
combo_name TEXT,
request_body TEXT,
response_body TEXT,
error TEXT
);
CREATE INDEX idx_cl_timestamp ON call_logs(timestamp);
CREATE INDEX idx_cl_status ON call_logs(status);
`);
seedDb.close();
}
test.beforeEach(() => {
restoreEnv();
cleanupGlobalDb();
@@ -399,3 +471,60 @@ test(
}
}
);
test(
"legacy call_logs schemas are upgraded before combo target indexes are created",
serial,
async () => {
const dataDir = makeTempDir("omniroute-db-legacy-call-logs-");
const sqliteFile = path.join(dataDir, "storage.sqlite");
createLegacyCallLogsDb(sqliteFile);
try {
await withEnv({ DATA_DIR: dataDir }, async () => {
const core = await importFresh("src/lib/db/core.ts");
const db = core.getDbInstance();
assert.ok(
db
.prepare("SELECT name FROM pragma_table_info('call_logs') WHERE name = ?")
.get("requested_model")
);
assert.ok(
db
.prepare("SELECT name FROM pragma_table_info('call_logs') WHERE name = ?")
.get("request_type")
);
assert.ok(
db
.prepare("SELECT name FROM pragma_table_info('call_logs') WHERE name = ?")
.get("combo_step_id")
);
assert.ok(
db
.prepare("SELECT name FROM pragma_table_info('call_logs') WHERE name = ?")
.get("combo_execution_key")
);
assert.ok(
db
.prepare("SELECT name FROM sqlite_master WHERE type = 'index' AND name = ?")
.get("idx_call_logs_requested_model")
);
assert.ok(
db
.prepare("SELECT name FROM sqlite_master WHERE type = 'index' AND name = ?")
.get("idx_call_logs_request_type")
);
assert.ok(
db
.prepare("SELECT name FROM sqlite_master WHERE type = 'index' AND name = ?")
.get("idx_cl_combo_target")
);
core.resetDbInstance();
});
} finally {
removePath(dataDir);
}
}
);

11
tests/unit/machine-id.test.mjs
View File

@@ -16,7 +16,6 @@ const originalWindir = process.env.windir;
const originalExecSync = childProcess.execSync;
const originalExecFileSync = childProcess.execFileSync;
const originalExistsSync = fs.existsSync;
const originalReadFileSync = fs.readFileSync;
function setPlatform(value) {
Object.defineProperty(process, "platform", {
@@ -33,7 +32,6 @@ test.afterEach(() => {
childProcess.execSync = originalExecSync;
childProcess.execFileSync = originalExecFileSync;
fs.existsSync = originalExistsSync;
fs.readFileSync = originalReadFileSync;
if (originalPlatformDescriptor) {
Object.defineProperty(process, "platform", originalPlatformDescriptor);
@@ -83,10 +81,11 @@ test("machineId: reads the Windows MachineGuid via REG.exe when available", asyn
test("machineId: falls back to Linux machine-id files before hostname", async () => {
setPlatform("linux");
fs.existsSync = (filePath) => filePath === "/etc/machine-id";
fs.readFileSync = (filePath, encoding) => {
assert.equal(filePath, "/etc/machine-id");
assert.equal(encoding, "utf8");
fs.existsSync = () => false;
childProcess.execFileSync = (command, args, options) => {
assert.equal(command, "cat");
assert.deepEqual(args, ["/etc/machine-id"]);
assert.equal(options.encoding, "utf8");
return "LINUX-MACHINE-ID\n";
};
childProcess.execSync = () => {

105
tests/unit/model-capabilities-registry.test.mjs Normal file
View File

@@ -0,0 +1,105 @@
import test from "node:test";
import assert from "node:assert/strict";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-model-capabilities-"));
process.env.DATA_DIR = TEST_DATA_DIR;
const core = await import("../../src/lib/db/core.ts");
const modelsDevSync = await import("../../src/lib/modelsDevSync.ts");
const modelCapabilities = await import("../../src/lib/modelCapabilities.ts");
function buildCapability(overrides = {}) {
return {
tool_call: null,
reasoning: null,
attachment: null,
structured_output: null,
temperature: null,
modalities_input: "[]",
modalities_output: "[]",
knowledge_cutoff: null,
release_date: null,
last_updated: null,
status: null,
family: null,
open_weights: null,
limit_context: null,
limit_input: null,
limit_output: null,
interleaved_field: null,
...overrides,
};
}
function resetStorage() {
core.resetDbInstance();
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
fs.mkdirSync(TEST_DATA_DIR, { recursive: true });
}
test.beforeEach(() => {
resetStorage();
});
test.after(() => {
core.resetDbInstance();
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
});
test("canonical model capability resolver merges models.dev data and keeps static overrides authoritative", () => {
modelsDevSync.saveModelsDevCapabilities({
openai: {
"gpt-4o": buildCapability({
tool_call: false,
reasoning: false,
attachment: true,
structured_output: true,
temperature: true,
modalities_input: JSON.stringify(["text", "image"]),
modalities_output: JSON.stringify(["text"]),
family: "gpt-4",
status: "stable",
limit_context: 256000,
limit_input: 256000,
limit_output: 12345,
}),
},
antigravity: {
"gemini-3.1-pro-high": buildCapability({
tool_call: false,
reasoning: false,
modalities_input: JSON.stringify(["text"]),
modalities_output: JSON.stringify(["text"]),
limit_context: 1024,
limit_output: 9999,
}),
},
});
const gpt4o = modelCapabilities.getResolvedModelCapabilities("openai/gpt-4o");
assert.equal(gpt4o.toolCalling, false);
assert.equal(gpt4o.reasoning, false);
assert.equal(gpt4o.supportsVision, true);
assert.equal(gpt4o.contextWindow, 256000);
assert.equal(gpt4o.maxInputTokens, 256000);
assert.equal(gpt4o.maxOutputTokens, 12345);
assert.equal(modelCapabilities.getModelContextLimit("openai", "gpt-4o"), 256000);
assert.equal(modelCapabilities.capMaxOutputTokens("openai/gpt-4o", 999999), 12345);
const geminiHigh = modelCapabilities.getResolvedModelCapabilities(
"antigravity/gemini-3.1-pro-high"
);
assert.equal(geminiHigh.toolCalling, true);
assert.equal(geminiHigh.reasoning, true);
assert.equal(geminiHigh.supportsThinking, true);
assert.equal(geminiHigh.contextWindow, 1048576);
assert.equal(geminiHigh.maxOutputTokens, 65535);
assert.equal(geminiHigh.defaultThinkingBudget, 24576);
assert.equal(
modelCapabilities.capThinkingBudget("antigravity/gemini-3.1-pro-high", 40000),
32768
);
});

10
tests/unit/model-combo-mappings-db.test.mjs
View File

@@ -146,7 +146,15 @@ test("resolveComboForModel returns the highest-priority enabled combo", async ()
assert.ok(resolved);
assert.equal(resolved.name, "priority");
assert.deepEqual(resolved.models, [{ provider: "openai", model: "gpt-4o" }]);
assert.deepEqual(resolved.models, [
{
id: "priority-model-1-openai-gpt-4o",
kind: "model",
providerId: "openai",
model: "openai/gpt-4o",
weight: 0,
},
]);
});
test("resolveComboForModel skips corrupted combo payloads and keeps scanning", async () => {

165
tests/unit/observability-payloads.test.mjs Normal file
View File

@@ -0,0 +1,165 @@
import test from "node:test";
import assert from "node:assert/strict";
import {
buildHealthPayload,
buildSessionsSummary,
buildTelemetryPayload,
} from "../../src/lib/monitoring/observability.ts";
test("buildSessionsSummary returns sticky counts and ordered top sessions", () => {
const summary = buildSessionsSummary({
activeSessions: [
{
sessionId: "sess-b",
createdAt: 1_000,
lastActive: 5_000,
requestCount: 3,
connectionId: "conn-b",
ageMs: 4_000,
},
{
sessionId: "sess-a",
createdAt: 1_000,
lastActive: 8_000,
requestCount: 5,
connectionId: null,
ageMs: 7_000,
},
],
activeSessionsByKey: { key1: 2 },
});
assert.equal(summary.activeCount, 2);
assert.equal(summary.stickyBoundCount, 1);
assert.equal(summary.byApiKey.key1, 2);
assert.equal(summary.top[0].sessionId, "sess-a");
assert.equal(summary.top[1].sessionId, "sess-b");
});
test("buildTelemetryPayload exposes totalRequests alias plus quota/session signals", () => {
const payload = buildTelemetryPayload({
summary: {
count: 7,
p50: 120,
p95: 320,
p99: 450,
phaseBreakdown: {
connect: { p50: 80, p95: 200, avg: 110, count: 7 },
},
},
quotaMonitorSummary: {
active: 3,
alerting: 2,
exhausted: 1,
errors: 1,
statusCounts: {
starting: 0,
idle: 0,
healthy: 1,
warning: 1,
exhausted: 1,
error: 0,
},
byProvider: { codex: 3 },
},
activeSessions: [
{
sessionId: "sess-a",
createdAt: 1_000,
lastActive: 8_000,
requestCount: 5,
connectionId: "conn-a",
ageMs: 7_000,
},
],
});
assert.equal(payload.totalRequests, 7);
assert.equal(payload.sessions.activeCount, 1);
assert.equal(payload.sessions.stickyBoundCount, 1);
assert.equal(payload.quotaMonitor.active, 3);
assert.equal(payload.quotaMonitor.exhausted, 1);
});
test("buildHealthPayload keeps legacy aliases and adds session/quota observability blocks", () => {
const payload = buildHealthPayload({
appVersion: "1.2.3",
catalogCount: 99,
settings: { setupComplete: true },
connections: [
{ provider: "codex", isActive: true },
{ provider: "openai", isActive: false },
],
circuitBreakers: [
{ name: "codex", state: "OPEN", failureCount: 2, lastFailureTime: "2026-04-12T12:00:00Z" },
{ name: "test-ignore", state: "OPEN", failureCount: 9, lastFailureTime: null },
],
rateLimitStatus: { codex: { blocked: 1 } },
lockouts: { codex: { "conn-1": { until: "2026-04-12T13:00:00Z" } } },
localProviders: { ollama: { ok: true } },
inflightRequests: 4,
quotaMonitorSummary: {
active: 1,
alerting: 1,
exhausted: 0,
errors: 0,
statusCounts: {
starting: 0,
idle: 0,
healthy: 0,
warning: 1,
exhausted: 0,
error: 0,
},
byProvider: { codex: 1 },
},
quotaMonitorMonitors: [
{
sessionId: "sess-a",
provider: "codex",
accountId: "conn-1",
status: "warning",
startedAt: "2026-04-12T12:00:00.000Z",
lastPolledAt: "2026-04-12T12:01:00.000Z",
lastSuccessAt: "2026-04-12T12:01:00.000Z",
lastErrorAt: null,
lastError: null,
lastQuotaPercent: 0.91,
lastQuotaUsed: 91,
lastQuotaTotal: 100,
lastResetAt: "2026-04-12T17:00:00.000Z",
lastAlertAt: "2026-04-12T12:01:00.000Z",
nextPollDelayMs: 15000,
nextPollAt: "2026-04-12T12:01:15.000Z",
totalPolls: 1,
totalAlerts: 1,
consecutiveFailures: 0,
},
],
activeSessions: [
{
sessionId: "sess-a",
createdAt: 1_000,
lastActive: 8_000,
requestCount: 5,
connectionId: "conn-1",
ageMs: 7_000,
},
],
activeSessionsByKey: { key1: 1 },
});
assert.equal(payload.version, "1.2.3");
assert.equal(payload.providerSummary.catalogCount, 99);
assert.equal(payload.providerSummary.configuredCount, 2);
assert.equal(payload.providerSummary.activeCount, 1);
assert.equal(payload.providerSummary.monitoredCount, 1);
assert.equal(payload.activeConnections, 2);
assert.equal(payload.circuitBreakers.open, 1);
assert.equal(payload.sessions.activeCount, 1);
assert.equal(payload.sessions.stickyBoundCount, 1);
assert.equal(payload.quotaMonitor.active, 1);
assert.equal(payload.quotaMonitor.monitors[0].provider, "codex");
assert.equal(payload.setupComplete, true);
});

15
tests/unit/openapi-spec-route.test.mjs Normal file
View File

@@ -0,0 +1,15 @@
import test from "node:test";
import assert from "node:assert/strict";
const { GET } = await import("../../src/app/api/openapi/spec/route.ts");
test("openapi spec route resolves the repository spec file and returns a parsed catalog", async () => {
const response = await GET();
assert.equal(response.status, 200);
const payload = await response.json();
assert.equal(typeof payload.info, "object");
assert.ok(Array.isArray(payload.endpoints));
assert.ok(Array.isArray(payload.schemas));
assert.ok(payload.endpoints.length > 0);
});

74
tests/unit/proxy-e2e-mode.test.mjs Normal file
View File

@@ -0,0 +1,74 @@
import test from "node:test";
import assert from "node:assert/strict";
import path from "node:path";
import { pathToFileURL } from "node:url";
const originalEnv = {
NEXT_PUBLIC_OMNIROUTE_E2E_MODE: process.env.NEXT_PUBLIC_OMNIROUTE_E2E_MODE,
};
function restoreEnv() {
for (const [key, value] of Object.entries(originalEnv)) {
if (value === undefined) {
delete process.env[key];
} else {
process.env[key] = value;
}
}
}
async function importFresh(modulePath) {
const url = pathToFileURL(path.resolve(modulePath)).href;
return import(`${url}?test=${Date.now()}-${Math.random().toString(16).slice(2)}`);
}
function makeRequest(pathname) {
return {
nextUrl: {
pathname,
protocol: "http:",
},
method: "GET",
cookies: {
get() {
return undefined;
},
},
headers: new Headers(),
url: `http://localhost:20128${pathname}`,
};
}
test.beforeEach(() => {
restoreEnv();
});
test.afterEach(() => {
restoreEnv();
});
test.after(() => {
restoreEnv();
});
test("proxy bypasses dashboard auth in Playwright e2e mode", async () => {
process.env.NEXT_PUBLIC_OMNIROUTE_E2E_MODE = "1";
const { proxy } = await importFresh("src/proxy.ts");
const response = await proxy(makeRequest("/dashboard/combos"));
assert.equal(response.status, 200);
assert.equal(response.headers.get("location"), null);
assert.ok(response.headers.get("x-request-id"));
});
test("proxy bypasses management API auth in Playwright e2e mode", async () => {
process.env.NEXT_PUBLIC_OMNIROUTE_E2E_MODE = "1";
const { proxy } = await importFresh("src/proxy.ts");
const response = await proxy(makeRequest("/api/combos"));
assert.equal(response.status, 200);
assert.equal(response.headers.get("location"), null);
assert.ok(response.headers.get("x-request-id"));
});

65
tests/unit/quota-monitor.test.mjs
View File

@@ -10,9 +10,13 @@ const {
startQuotaMonitor,
stopQuotaMonitor,
getActiveMonitorCount,
getQuotaMonitorSnapshot,
getQuotaMonitorSummary,
clearQuotaMonitors,
} = quotaMonitor;
const { preflightQuota } = quotaPreflight;
const { touchSession, clearSessions } = await import("../../open-sse/services/sessionManager.ts");
function createConnection(providerSpecificData = {}) {
return { providerSpecificData };
@@ -100,6 +104,11 @@ test("isQuotaMonitorEnabled reads the provider flag strictly", () => {
assert.equal(isQuotaMonitorEnabled(createConnection()), false);
});
test.afterEach(() => {
clearQuotaMonitors();
clearSessions();
});
test("startQuotaMonitor is a no-op when the monitor is disabled", async () => {
await withFakeTimers(async ({ scheduled }) => {
startQuotaMonitor("quota-disabled-session", "provider-disabled", "conn-1", createConnection());
@@ -112,6 +121,7 @@ test("startQuotaMonitor schedules the initial normal poll once per session", asy
const sessionId = `quota-normal-${Date.now()}`;
await withFakeTimers(async ({ scheduled }) => {
touchSession(sessionId, "conn-2");
startQuotaMonitor(
sessionId,
"provider-normal",
@@ -160,6 +170,7 @@ test("quota monitor keeps normal polling when no fetcher is registered", async (
const sessionId = `quota-missing-fetcher-${Date.now()}`;
await withFakeTimers(async ({ scheduled, runNextTimer }) => {
touchSession(sessionId, "conn-3");
startQuotaMonitor(
sessionId,
"provider-missing-fetcher",
@@ -191,6 +202,7 @@ test("quota monitor switches to critical polling and suppresses duplicate warnin
await withFakeTimers(async ({ scheduled, runNextTimer }) => {
await withPatchedConsole({ warn: (message) => warnings.push(message) }, async () => {
await withMockedNow(1_700_000_000_000, async () => {
touchSession(sessionId, "conn-4");
startQuotaMonitor(
sessionId,
provider,
@@ -232,6 +244,7 @@ test("quota monitor logs exhaustion and clears suppression state on stop", async
},
async () => {
await withMockedNow(1_700_000_000_000, async () => {
touchSession(sessionId, "conn-5");
startQuotaMonitor(
sessionId,
provider,
@@ -260,6 +273,7 @@ test("quota monitor logs exhaustion and clears suppression state on stop", async
},
async () => {
await withMockedNow(1_700_000_000_000, async () => {
touchSession(sessionId, "conn-5");
startQuotaMonitor(
sessionId,
provider,
@@ -285,6 +299,7 @@ test("quota monitor falls back to normal polling when the fetcher throws", async
});
await withFakeTimers(async ({ scheduled, runNextTimer }) => {
touchSession(sessionId, "conn-6");
startQuotaMonitor(
sessionId,
provider,
@@ -299,3 +314,53 @@ test("quota monitor falls back to normal polling when the fetcher throws", async
stopQuotaMonitor(sessionId);
});
});
test("quota monitor exposes runtime snapshot and restarts when the session account changes", async () => {
const provider = `quota-snapshot-${Date.now()}`;
const sessionId = `quota-snapshot-session-${Date.now()}`;
let calls = 0;
registerMonitorFetcher(provider, async () => {
calls += 1;
return {
used: 91,
total: 100,
percentUsed: 0.91,
resetAt: "2026-04-12T15:00:00.000Z",
};
});
await withFakeTimers(async ({ runNextTimer }) => {
touchSession(sessionId, "conn-7");
startQuotaMonitor(
sessionId,
provider,
"conn-7",
createConnection({ quotaMonitorEnabled: true })
);
await runNextTimer();
const firstSnapshot = getQuotaMonitorSnapshot(sessionId);
assert.equal(firstSnapshot?.provider, provider);
assert.equal(firstSnapshot?.accountId, "conn-7");
assert.equal(firstSnapshot?.status, "warning");
assert.equal(firstSnapshot?.lastQuotaPercent, 0.91);
assert.equal(firstSnapshot?.lastResetAt, "2026-04-12T15:00:00.000Z");
startQuotaMonitor(
sessionId,
provider,
"conn-8",
createConnection({ quotaMonitorEnabled: true })
);
const restartedSnapshot = getQuotaMonitorSnapshot(sessionId);
assert.equal(restartedSnapshot?.accountId, "conn-8");
assert.equal(restartedSnapshot?.status, "starting");
const summary = getQuotaMonitorSummary();
assert.equal(summary.active, 1);
assert.equal(summary.statusCounts.starting, 1);
assert.equal(calls, 1);
});
});

1
tests/unit/quota-preflight.test.mjs
View File

@@ -102,6 +102,7 @@ test("preflightQuota blocks when usage reaches the exhaustion threshold", async
proceed: false,
reason: "quota_exhausted",
quotaPercent: 0.95,
resetAt: null,
});
assert.equal(infos.length, 1);
assert.match(infos[0], /switching/i);

119
tests/unit/run-next-playwright.test.mjs Normal file
View File

@@ -0,0 +1,119 @@
import test from "node:test";
import assert from "node:assert/strict";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
const playwrightRunner = await import("../../scripts/run-next-playwright.mjs");
test("resolvePlaywrightAppBackupDir uses a per-run backup when a stale backup already exists", () => {
const cwd = "/tmp/omniroute-playwright-runner";
assert.equal(
playwrightRunner.resolvePlaywrightAppBackupDir({
cwd,
baseBackupExists: false,
appDirExists: true,
pid: 123,
now: 456,
}),
path.join(cwd, "app.__qa_backup")
);
assert.equal(
playwrightRunner.resolvePlaywrightAppBackupDir({
cwd,
baseBackupExists: true,
appDirExists: true,
pid: 123,
now: 456,
}),
path.join(cwd, "app.__qa_backup.123.456")
);
});
test("shouldUseWebpackForPlaywrightDev only opts into webpack when turbopack is disabled", () => {
assert.equal(
playwrightRunner.shouldUseWebpackForPlaywrightDev({
mode: "dev",
env: { OMNIROUTE_USE_TURBOPACK: "1" },
}),
false
);
assert.equal(
playwrightRunner.shouldUseWebpackForPlaywrightDev({
mode: "dev",
env: { OMNIROUTE_USE_TURBOPACK: "0" },
}),
true
);
assert.equal(
playwrightRunner.shouldUseWebpackForPlaywrightDev({
mode: "start",
env: { OMNIROUTE_USE_TURBOPACK: "1" },
}),
false
);
});
test("standalone asset helpers detect and rehydrate missing standalone static assets", () => {
const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-playwright-assets-"));
const standaloneServerPath = path.join(tempRoot, ".next", "standalone", "server.js");
const rootStaticDirPath = path.join(tempRoot, ".next", "static");
const standaloneStaticDirPath = path.join(tempRoot, ".next", "standalone", ".next", "static");
const rootPublicDirPath = path.join(tempRoot, "public");
const standalonePublicDirPath = path.join(tempRoot, ".next", "standalone", "public");
fs.mkdirSync(path.dirname(standaloneServerPath), { recursive: true });
fs.writeFileSync(standaloneServerPath, "server");
fs.mkdirSync(rootStaticDirPath, { recursive: true });
fs.mkdirSync(rootPublicDirPath, { recursive: true });
fs.writeFileSync(path.join(rootStaticDirPath, "chunk.js"), "console.log('chunk');");
fs.writeFileSync(path.join(rootPublicDirPath, "favicon.svg"), "<svg />");
assert.equal(
playwrightRunner.standaloneAssetsNeedSync({
standaloneServerPath,
rootStaticDirPath,
standaloneStaticDirPath,
}),
true
);
const logs = [];
const changed = playwrightRunner.syncStandaloneRuntimeAssets({
standaloneServerPath,
rootStaticDirPath,
standaloneStaticDirPath,
rootPublicDirPath,
standalonePublicDirPath,
log: {
log(message) {
logs.push(message);
},
},
});
assert.equal(changed, true);
assert.equal(
fs.readFileSync(path.join(standaloneStaticDirPath, "chunk.js"), "utf8"),
"console.log('chunk');"
);
assert.equal(
fs.readFileSync(path.join(standalonePublicDirPath, "favicon.svg"), "utf8"),
"<svg />"
);
assert.equal(
playwrightRunner.standaloneAssetsNeedSync({
standaloneServerPath,
rootStaticDirPath,
standaloneStaticDirPath,
}),
false
);
assert.match(logs[0] || "", /Rehydrated standalone static\/public assets/);
fs.rmSync(tempRoot, { recursive: true, force: true });
});

25
tests/unit/services-branch-hardening.test.mjs
View File

@@ -81,12 +81,15 @@ test("model capability helpers cover denylist, empty input and default-safe path
assert.equal(modelCapabilities.supportsToolCalling(""), false);
assert.equal(modelCapabilities.supportsToolCalling("openai/gpt-oss-120b"), false);
assert.equal(modelCapabilities.supportsToolCalling("deepseek-reasoner"), false);
assert.equal(modelCapabilities.supportsToolCalling("openai/gpt-4o"), true);
assert.equal(
modelCapabilities.supportsToolCalling("openai/nonexistent-default-safe-model"),
true
);
assert.equal(modelCapabilities.supportsReasoning(""), true);
assert.equal(modelCapabilities.supportsReasoning("antigravity/claude-sonnet-4-6"), false);
assert.equal(modelCapabilities.supportsReasoning("antigravity/claude-sonnet-4"), false);
assert.equal(modelCapabilities.supportsReasoning("openai/gpt-4o"), true);
assert.equal(modelCapabilities.supportsReasoning("openai/nonexistent-default-safe-model"), true);
});
test("combo agent middleware covers system override, tool filtering, tag stripping and pin propagation", () => {
@@ -214,12 +217,26 @@ test("combo metrics cover empty reads, intent tracking, per-model stats and rese
latencyMs: 120,
fallbackCount: 1,
strategy: "priority",
target: {
executionKey: "writer>step-openai-a",
stepId: "step-openai-a",
provider: "openai",
connectionId: "conn-openai-a",
label: "Primary OpenAI",
},
});
comboMetrics.recordComboRequest("writer", "openai/gpt-4o", {
success: false,
latencyMs: 80,
fallbackCount: 0,
strategy: "least-used",
target: {
executionKey: "writer>step-openai-a",
stepId: "step-openai-a",
provider: "openai",
connectionId: "conn-openai-a",
label: "Primary OpenAI",
},
});
comboMetrics.recordComboRequest("writer", null, {
success: true,
@@ -241,6 +258,10 @@ test("combo metrics cover empty reads, intent tracking, per-model stats and rese
assert.equal(writer.byModel["openai/gpt-4o"].requests, 2);
assert.equal(writer.byModel["openai/gpt-4o"].successRate, 50);
assert.equal(writer.byModel["openai/gpt-4o"].avgLatencyMs, 100);
assert.equal(writer.byTarget["writer>step-openai-a"].requests, 2);
assert.equal(writer.byTarget["writer>step-openai-a"].successRate, 50);
assert.equal(writer.byTarget["writer>step-openai-a"].connectionId, "conn-openai-a");
assert.equal(writer.byTarget["writer>step-openai-a"].label, "Primary OpenAI");
const allMetrics = comboMetrics.getAllComboMetrics();
assert.equal(Object.keys(allMetrics).length, 2);

154
tests/unit/sse-auth.test.mjs
View File

@@ -40,6 +40,8 @@ async function seedConnection(provider, overrides = {}) {
priority: overrides.priority,
rateLimitedUntil: overrides.rateLimitedUntil,
lastError: overrides.lastError,
lastErrorType: overrides.lastErrorType,
lastErrorSource: overrides.lastErrorSource,
errorCode: overrides.errorCode,
backoffLevel: overrides.backoffLevel,
providerSpecificData: overrides.providerSpecificData || {},
@@ -147,6 +149,60 @@ test("getProviderCredentials enforces generic quota policy unless explicitly byp
assert.equal(bypassed.connectionId, connection.id);
});
test("getProviderCredentialsWithQuotaPreflight skips exhausted preflight accounts and selects the next healthy one", async () => {
const blocked = await seedConnection("openai", {
name: "quota-preflight-blocked",
apiKey: "sk-preflight-blocked",
providerSpecificData: {
quotaPreflightEnabled: true,
},
});
const healthy = await seedConnection("openai", {
name: "quota-preflight-healthy",
apiKey: "sk-preflight-healthy",
providerSpecificData: {
quotaPreflightEnabled: true,
},
});
const quotaPreflight = await import("../../open-sse/services/quotaPreflight.ts");
quotaPreflight.registerQuotaFetcher("openai", async (connectionId) => ({
used: connectionId === blocked.id ? 96 : 40,
total: 100,
percentUsed: connectionId === blocked.id ? 0.96 : 0.4,
}));
const selected = await auth.getProviderCredentialsWithQuotaPreflight("openai");
assert.equal(selected.connectionId, healthy.id);
});
test("getProviderCredentialsWithQuotaPreflight returns allRateLimited when a forced connection is blocked by preflight", async () => {
const blocked = await seedConnection("openai", {
name: "quota-preflight-forced",
apiKey: "sk-preflight-forced",
providerSpecificData: {
quotaPreflightEnabled: true,
},
});
const quotaPreflight = await import("../../open-sse/services/quotaPreflight.ts");
quotaPreflight.registerQuotaFetcher("openai", async (connectionId) => ({
used: connectionId === blocked.id ? 99 : 20,
total: 100,
percentUsed: connectionId === blocked.id ? 0.99 : 0.2,
resetAt: futureIso(120_000),
}));
const selected = await auth.getProviderCredentialsWithQuotaPreflight("openai", null, null, null, {
forcedConnectionId: blocked.id,
});
assert.equal(selected.allRateLimited, true);
assert.equal(selected.lastErrorCode, 429);
assert.match(selected.lastError, /quota preflight/i);
});
test("resolveQuotaLimitPolicy normalizes Codex windows, thresholds, and defaults", () => {
const normalized = auth.resolveQuotaLimitPolicy("codex", {
limitPolicy: {
@@ -269,6 +325,43 @@ test("getProviderCredentials honors allowedConnections filters", async () => {
assert.notEqual(selected.connectionId, skipped.id);
});
test("getProviderCredentials honors forcedConnectionId even when another account is preferred", async () => {
await seedConnection("openai", {
name: "forced-default",
priority: 1,
apiKey: "sk-default",
});
const forcedConn = await seedConnection("openai", {
name: "forced-target",
priority: 99,
apiKey: "sk-forced",
});
const selected = await auth.getProviderCredentials("openai", null, null, null, {
forcedConnectionId: forcedConn.id,
});
assert.equal(selected.connectionId, forcedConn.id);
assert.equal(selected.apiKey, "sk-forced");
});
test("getProviderCredentials intersects forcedConnectionId with allowedConnections", async () => {
const allowedConn = await seedConnection("openai", {
name: "forced-allowed",
apiKey: "sk-allowed",
});
const blockedConn = await seedConnection("openai", {
name: "forced-blocked",
apiKey: "sk-blocked",
});
const selected = await auth.getProviderCredentials("openai", null, [allowedConn.id], null, {
forcedConnectionId: blockedConn.id,
});
assert.equal(selected, null);
});
test("getProviderCredentials retains rate-limited accounts when allowSuppressedConnections is enabled", async () => {
const connection = await seedConnection("openai", {
name: "suppressed-rate-limit",
@@ -532,6 +625,67 @@ test("getProviderCredentials cost-optimized selects the lowest priority account"
assert.equal(selected.connectionId, cheapest.id);
});
test("getProviderCredentials p2c prefers the account with more quota headroom over raw priority", async () => {
await settingsDb.updateSettings({ fallbackStrategy: "p2c" });
const nearLimit = await seedConnection("openai", {
name: "p2c-near-limit",
priority: 1,
apiKey: "sk-p2c-near-limit",
providerSpecificData: {
limitPolicy: {
enabled: true,
thresholdPercent: 80,
windows: ["daily"],
},
},
});
const healthy = await seedConnection("openai", {
name: "p2c-healthy-headroom",
priority: 9,
apiKey: "sk-p2c-healthy",
providerSpecificData: {
limitPolicy: {
enabled: true,
thresholdPercent: 80,
windows: ["daily"],
},
},
});
quotaCache.setQuotaCache(nearLimit.id, "openai", {
daily: { remainingPercentage: 12, resetAt: futureIso(180_000) },
});
quotaCache.setQuotaCache(healthy.id, "openai", {
daily: { remainingPercentage: 78, resetAt: futureIso(180_000) },
});
const selected = await auth.getProviderCredentials("openai");
assert.equal(selected.connectionId, healthy.id);
});
test("getProviderCredentials p2c deprioritizes accounts with recent rate-limit/backoff signals", async () => {
await settingsDb.updateSettings({ fallbackStrategy: "p2c" });
await seedConnection("openai", {
name: "p2c-rate-limited-history",
priority: 1,
apiKey: "sk-p2c-rate-limited",
lastError: "rate limit",
lastErrorType: "rate_limited",
errorCode: 429,
backoffLevel: 3,
});
const healthy = await seedConnection("openai", {
name: "p2c-clean-account",
priority: 8,
apiKey: "sk-p2c-clean",
});
const selected = await auth.getProviderCredentials("openai");
assert.equal(selected.connectionId, healthy.id);
});
test("getProviderCredentials resolves the nvidia special alias pool", async () => {
const connection = await seedConnection("nvidia_nim", {
name: "nvidia-special-alias",

35
tests/unit/telemetry-summary-route.test.mjs Normal file
View File

@@ -0,0 +1,35 @@
import test from "node:test";
import assert from "node:assert/strict";
import { GET } from "../../src/app/api/telemetry/summary/route.ts";
import { RequestTelemetry, recordTelemetry } from "../../src/shared/utils/requestTelemetry.ts";
import { clearQuotaMonitors, startQuotaMonitor } from "../../open-sse/services/quotaMonitor.ts";
import { clearSessions, touchSession } from "../../open-sse/services/sessionManager.ts";
test.afterEach(() => {
clearQuotaMonitors();
clearSessions();
});
test("telemetry summary route includes totalRequests alias plus session/quota monitor signals", async () => {
const telemetry = new RequestTelemetry("telemetry-route");
telemetry.startPhase("parse");
telemetry.endPhase();
recordTelemetry(telemetry);
touchSession("sess-route", "conn-route");
startQuotaMonitor("sess-route", "codex", "conn-route", {
providerSpecificData: { quotaMonitorEnabled: true },
});
const response = await GET(
new Request("http://localhost:20128/api/telemetry/summary?windowMs=600000")
);
const payload = await response.json();
assert.equal(response.status, 200);
assert.ok(payload.totalRequests >= 1);
assert.equal(payload.sessions.activeCount, 1);
assert.equal(payload.sessions.stickyBoundCount, 1);
assert.equal(payload.quotaMonitor.active, 1);
});