diff --git a/docs/AGENTROUTER.md b/docs/AGENTROUTER.md new file mode 100644 index 0000000000..2333bc0883 --- /dev/null +++ b/docs/AGENTROUTER.md @@ -0,0 +1,149 @@ +# AgentRouter Setup Guide + +[AgentRouter](https://agentrouter.org) is an Anthropic-compatible relay that resells +Claude and other models, often at lower prices than the direct Anthropic API. It is +designed as a drop-in `ANTHROPIC_BASE_URL` replacement for the official Claude Code +client, so it only accepts traffic that matches the Claude Code wire image (specific +User-Agent, `anthropic-beta` flags, Stainless SDK headers, etc.). + +OmniRoute supports AgentRouter through the **Claude Code compatible** provider type +(`anthropic-compatible-cc-*`), which speaks the Anthropic Messages API with the +correct wire image. A generic `openai-compatible-chat` provider pointing at +`https://agentrouter.org` will **not** work — the upstream WAF rejects requests that +do not look like Claude Code. + +--- + +## Prerequisites + +- An AgentRouter account and API key. New signups get free credits via the affiliate + link in the project [README](../README.md). +- OmniRoute running with the `ENABLE_CC_COMPATIBLE_PROVIDER` feature flag enabled + (see below). + +## 1. Enable the CC-compatible provider type + +The Claude Code compatible provider type is gated behind a feature flag because it +sends traffic that closely mirrors the official Claude Code client. Enable it by +setting an environment variable before starting OmniRoute: + +```bash +ENABLE_CC_COMPATIBLE_PROVIDER=true +``` + +Docker example: + +```bash +docker run -d --name omniroute \ + --restart unless-stopped \ + -p 20128:20128 \ + -v omniroute-data:/app/data \ + -e ENABLE_CC_COMPATIBLE_PROVIDER=true \ + diegosouzapw/omniroute:latest +``` + +After restarting, the dashboard exposes an **Add Claude Code Compatible** option in +addition to the existing OpenAI-compatible and Anthropic-compatible flows. + +## 2. Create the provider in the dashboard + +1. Open **Dashboard → Providers → Add Provider**. +2. Choose **Add Claude Code Compatible** (only visible when the flag above is set). +3. Fill in the fields: + +| Field | Value | +| --------- | -------------------------------------------------------------- | +| Name | `AgentRouter` (or any label) | +| Prefix | `agentrouter` (friendly alias shown in logs and the dashboard) | +| Base URL | `https://agentrouter.org` | +| Chat path | `/v1/messages?beta=true` (default — leave as-is) | + +> The canonical model identifier still uses the full provider node ID +> (`anthropic-compatible-cc-{uuid}/{model}`). The **Prefix** is just a display +> alias resolved by `src/lib/usage/callLogs.ts` for friendlier log output. + +4. (Optional) Paste your API key in the **Validate** field and click **Check** to + confirm connectivity before saving. +5. Click **Add**. + +Once created, open the provider and add a **Connection** with your AgentRouter API +key (`sk-...`). The connection's `test_status` should turn `active`. + +## 3. Use it through a combo or directly + +Reference the model using your provider's prefix as the namespace: + +```bash +curl -X POST http://localhost:20128/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "agentrouter/claude-opus-4-6", + "messages": [{"role": "user", "content": "hello"}], + "max_tokens": 100 + }' +``` + +The canonical model ID `anthropic-compatible-cc-{uuid}/claude-opus-4-6` also works +and is what shows up in the database and combo configuration. + +Or add it to a combo for routing, fallback, and quota management like any other +provider. + +--- + +## Wire image details + +For reference, the cc-compatible bridge sends the following on each upstream +request (see `open-sse/services/claudeCodeCompatible.ts`): + +| Header | Value | +| ------------------------------------------- | ------------------------------------------------------------------------ | +| `Authorization` | `Bearer ` | +| `User-Agent` | `claude-cli/2.1.137 (external, sdk-cli)` | +| `anthropic-version` | `2023-06-01` | +| `anthropic-beta` | `claude-code-20250219,interleaved-thinking-2025-05-14,effort-2025-11-24` | +| `anthropic-dangerous-direct-browser-access` | `true` | +| `x-app` | `cli` | +| `X-Stainless-*` | Various Stainless SDK headers (lang, package version, OS, arch, etc.) | + +This is what allows requests to pass the upstream WAF / client whitelist. + +--- + +## Troubleshooting + +**`{"error":{"message":"unauthorized client detected, ..."}}`** — Your request did +not match the Claude Code wire image. This happens when the provider is configured +as `openai-compatible-chat` instead of `anthropic-compatible-cc`, or when the +`ENABLE_CC_COMPATIBLE_PROVIDER=true` flag was not set at startup. + +**`{"error":{"message":"无效的令牌","type":"new_api_error"}}` (HTTP 401)** — +"Invalid token". The wire image is correct but the API key is rejected. Generate a +new key in the AgentRouter dashboard and update the connection. + +**`{"error":{"code":"content-blocked","type":"agent_router_api_error"}}` +(HTTP 400)** — AgentRouter's moderation hook rejected the request content, or the +key's plan does not permit the requested model. Try a different prompt or model; +contact AgentRouter support if a benign prompt is consistently blocked. + +**`[400]: content-blocked` only on specific models** — Most AgentRouter plans only +allow a subset of models (e.g. `claude-opus-4-6`). Other model IDs return +`unauthorized_client_error` even though the key is valid. Check which models your +plan covers in the AgentRouter dashboard. + +**`Invalid JSON response from provider (reset after Ns)` from the omniroute logs** — +The upstream returned a non-JSON body (typically an HTML error page from the WAF). +This usually means the request never reached the AgentRouter backend — recheck that +the provider ID starts with `anthropic-compatible-cc-` (note the trailing dash — +see `CLAUDE_CODE_COMPATIBLE_PREFIX` in `open-sse/services/claudeCodeCompatible.ts`) +and the feature flag is enabled. + +--- + +## See also + +- [`docs/PROVIDERS.md`](./PROVIDERS.md) — Other provider integration notes +- [`docs/reference/FREE_TIERS.md`](./reference/FREE_TIERS.md) — Free-tier provider + catalog +- [`open-sse/services/claudeCodeCompatible.ts`](../open-sse/services/claudeCodeCompatible.ts) + — Wire image implementation diff --git a/open-sse/executors/base.ts b/open-sse/executors/base.ts index c91bc71e1f..b726c6a0cd 100644 --- a/open-sse/executors/base.ts +++ b/open-sse/executors/base.ts @@ -105,6 +105,8 @@ export type ExecuteInput = { clientHeaders?: Record | null; /** Callback to persist tokens that are proactively refreshed during execution. */ onCredentialsRefreshed?: (newCredentials: ProviderCredentials) => Promise | void; + /** When true, skip the intra-URL 429 retry in execute() so the caller handles fallback. */ + skipUpstreamRetry?: boolean; }; export type CountTokensInput = { @@ -526,6 +528,7 @@ export class BaseExecutor { extendedContext, upstreamExtraHeaders, clientHeaders, + skipUpstreamRetry = false, }: ExecuteInput) { const fallbackCount = this.getFallbackCount(); let lastError: unknown = null; @@ -936,6 +939,7 @@ export class BaseExecutor { // Intra-URL retry: if 429 and we haven't exhausted per-URL retries, wait and retry the same URL if ( + !skipUpstreamRetry && response.status === HTTP_STATUS.RATE_LIMITED && (retryAttemptsByUrl[urlIndex] ?? 0) < BaseExecutor.RETRY_CONFIG.maxAttempts ) { @@ -955,7 +959,7 @@ export class BaseExecutor { log?.warn?.("AUTH", `401 on ${url} - API key may be invalid`); } - if (this.shouldRetry(response.status, urlIndex)) { + if (!skipUpstreamRetry && this.shouldRetry(response.status, urlIndex)) { log?.debug?.("RETRY", `${response.status} on ${url}, trying fallback ${urlIndex + 1}`); lastStatus = response.status; continue; @@ -969,7 +973,7 @@ export class BaseExecutor { log?.warn?.("TIMEOUT", `Fetch timeout after ${this.getTimeoutMs()}ms on ${url}`); } lastError = err; - if (urlIndex + 1 < fallbackCount) { + if (!skipUpstreamRetry && urlIndex + 1 < fallbackCount) { log?.debug?.("RETRY", `Error on ${url}, trying fallback ${urlIndex + 1}`); continue; } diff --git a/open-sse/handlers/chatCore.ts b/open-sse/handlers/chatCore.ts index 7a668b4d0b..e5041adb5d 100644 --- a/open-sse/handlers/chatCore.ts +++ b/open-sse/handlers/chatCore.ts @@ -1259,6 +1259,7 @@ export async function handleChatCore({ comboExecutionKey = null, disableEmergencyFallback = false, cachedSettings = null, + skipUpstreamRetry = false, }) { let { provider, model, extendedContext } = modelInfo; // apiFormat is an optional custom-model marker injected by getModelInfo for @@ -3170,6 +3171,7 @@ export async function handleChatCore({ upstreamExtraHeaders: buildUpstreamHeadersForExecute(modelToCall), clientHeaders: buildExecutorClientHeaders(clientRawRequest?.headers, userAgent), onCredentialsRefreshed, + skipUpstreamRetry, }); trace("post_executor", { status: res?.response?.status }); @@ -3661,6 +3663,7 @@ export async function handleChatCore({ upstreamExtraHeaders: buildUpstreamHeadersForExecute(retryModelId), clientHeaders: buildExecutorClientHeaders(clientRawRequest?.headers, userAgent), onCredentialsRefreshed, + skipUpstreamRetry: isCombo, }); if (retryResult.response.ok) { diff --git a/open-sse/services/accountFallback.ts b/open-sse/services/accountFallback.ts index efe3edd26f..343dea1797 100644 --- a/open-sse/services/accountFallback.ts +++ b/open-sse/services/accountFallback.ts @@ -488,6 +488,14 @@ export function shouldMarkAccountExhaustedFrom429( ); } +/** + * Clear all in-memory model lockouts and failure state (for tests / full reset). + */ +export function clearAllModelLockouts(): void { + modelLockouts.clear(); + modelFailureState.clear(); +} + /** * Check if a specific model on a specific account is locked * @returns {boolean} diff --git a/open-sse/services/combo.ts b/open-sse/services/combo.ts index aa3f5d5a83..6a9db8ce0e 100644 --- a/open-sse/services/combo.ts +++ b/open-sse/services/combo.ts @@ -1690,6 +1690,8 @@ export async function handleComboChat({ const maxRetries = config.maxRetries ?? 1; const retryDelayMs = resolveDelayMs(config.retryDelayMs, 2000); const fallbackDelayMs = resolveDelayMs(config.fallbackDelayMs, 0); + const maxSetRetries = config.maxSetRetries ?? 0; + const setRetryDelayMs = resolveDelayMs(config.setRetryDelayMs, 2000); let orderedTargets = strategy === "weighted" @@ -1993,98 +1995,254 @@ export async function handleComboChat({ } // #1731: Per-request in-memory set of providers whose quota is fully exhausted. - // When a target returns a quota-exhausted 429 (subscription/credits/daily), - // remaining targets from the same provider are skipped to avoid the - // 2–5 minute cascade through N same-provider targets. + // When a target returns a quota-exhausted 429, remaining same-provider targets are skipped. const exhaustedProviders = new Set(); - - let lastError = null; - let earliestRetryAfter = null; - let lastStatus = null; - const startTime = Date.now(); let globalAttempts = 0; - let fallbackCount = 0; - let recordedAttempts = 0; - for (let i = 0; i < orderedTargets.length; i++) { - const target = orderedTargets[i]; - const modelStr = target.modelStr; - const provider = target.provider; - const profile = await getRuntimeProviderProfile(provider); + for (let setTry = 0; setTry <= maxSetRetries; setTry++) { + if (setTry > 0) { + log.info("COMBO", `All targets failed — retrying set (${setTry}/${maxSetRetries})`); + await new Promise((resolve) => { + const timer = setTimeout(resolve, setRetryDelayMs); + signal?.addEventListener( + "abort", + () => { + clearTimeout(timer); + resolve(undefined); + }, + { once: true } + ); + }); + if (signal?.aborted) { + log.info("COMBO", "Client disconnected during set retry delay — aborting"); + return errorResponse(499, "Client disconnected"); + } + } - // Pre-check: skip models where all accounts are in cooldown - if (isModelAvailable) { - const available = await isModelAvailable(modelStr, target); - if (!available) { - log.info("COMBO", `Skipping ${modelStr} (all accounts in cooldown)`); + let lastError = null; + let earliestRetryAfter = null; + let lastStatus = null; + const startTime = Date.now(); + let fallbackCount = 0; + let recordedAttempts = 0; + + for (let i = 0; i < orderedTargets.length; i++) { + const target = orderedTargets[i]; + const modelStr = target.modelStr; + const provider = target.provider; + const profile = await getRuntimeProviderProfile(provider); + + // #1731: Skip targets from a provider that already signaled full quota exhaustion this request. + if (provider && exhaustedProviders.has(provider)) { + log.info( + "COMBO", + `Skipping ${modelStr} — provider ${provider} marked exhausted this request (#1731)` + ); if (i > 0) fallbackCount++; continue; } - } - // #1731: Skip targets from a provider that already signaled full quota exhaustion - // this request. - if (provider && exhaustedProviders.has(provider)) { - log.info( - "COMBO", - `Skipping ${modelStr} — provider ${provider} marked exhausted this request (#1731)` - ); - if (i > 0) fallbackCount++; - continue; - } - - // Retry loop for transient errors - for (let retry = 0; retry <= maxRetries; retry++) { - // Fix #1681: Bail out immediately if the client has disconnected - if (signal?.aborted) { - log.info("COMBO", `Client disconnected — aborting combo loop before model ${modelStr}`); - return errorResponse(499, "Client disconnected"); - } - globalAttempts++; - if (globalAttempts > MAX_GLOBAL_ATTEMPTS) { - log.warn( - "COMBO", - `Maximum combo attempts (${MAX_GLOBAL_ATTEMPTS}) exceeded across all targets and fallbacks. Terminating loop to prevent runaway background requests.` - ); - return errorResponse(503, "Maximum combo retry limit reached"); - } - if (retry > 0) { - log.info( - "COMBO", - `Retrying ${modelStr} in ${retryDelayMs}ms (attempt ${retry + 1}/${maxRetries + 1})` - ); - await new Promise((resolve) => { - const timer = setTimeout(resolve, retryDelayMs); - signal?.addEventListener( - "abort", - () => { - clearTimeout(timer); - resolve(undefined); - }, - { once: true } - ); - }); - if (signal?.aborted) { - log.info("COMBO", `Client disconnected during retry delay — aborting`); - return errorResponse(499, "Client disconnected"); + // Pre-check: skip models where all accounts are in cooldown + if (isModelAvailable) { + const available = await isModelAvailable(modelStr, target); + if (!available) { + log.info("COMBO", `Skipping ${modelStr} (all accounts in cooldown)`); + if (i > 0) fallbackCount++; + continue; } } - log.info( - "COMBO", - `Trying model ${i + 1}/${orderedTargets.length}: ${modelStr}${retry > 0 ? ` (retry ${retry})` : ""}` - ); - - const result = await handleSingleModelWrapped(body, modelStr, target); - - // Success — validate response quality before returning - if (result.ok) { - const quality = await validateResponseQuality(result, clientRequestedStream, log); - if (!quality.valid) { + // Retry loop for transient errors + for (let retry = 0; retry <= maxRetries; retry++) { + // Fix #1681: Bail out immediately if the client has disconnected + if (signal?.aborted) { + log.info("COMBO", `Client disconnected — aborting combo loop before model ${modelStr}`); + return errorResponse(499, "Client disconnected"); + } + globalAttempts++; + if (globalAttempts > MAX_GLOBAL_ATTEMPTS) { log.warn( "COMBO", - `Model ${modelStr} returned 200 but failed quality check: ${quality.reason}` + `Maximum combo attempts (${MAX_GLOBAL_ATTEMPTS}) exceeded across all targets and fallbacks. Terminating loop to prevent runaway background requests.` ); + return errorResponse(503, "Maximum combo retry limit reached"); + } + if (retry > 0) { + log.info( + "COMBO", + `Retrying ${modelStr} in ${retryDelayMs}ms (attempt ${retry + 1}/${maxRetries + 1})` + ); + await new Promise((resolve) => { + const timer = setTimeout(resolve, retryDelayMs); + signal?.addEventListener( + "abort", + () => { + clearTimeout(timer); + resolve(undefined); + }, + { once: true } + ); + }); + if (signal?.aborted) { + log.info("COMBO", `Client disconnected during retry delay — aborting`); + return errorResponse(499, "Client disconnected"); + } + } + + log.info( + "COMBO", + `Trying model ${i + 1}/${orderedTargets.length}: ${modelStr}${retry > 0 ? ` (retry ${retry})` : ""}` + ); + + const result = await handleSingleModelWrapped(body, modelStr, { + ...target, + failoverBeforeRetry: config.failoverBeforeRetry, + }); + + // Success — validate response quality before returning + if (result.ok) { + const quality = await validateResponseQuality(result, clientRequestedStream, log); + if (!quality.valid) { + log.warn( + "COMBO", + `Model ${modelStr} returned 200 but failed quality check: ${quality.reason}` + ); + recordComboRequest(combo.name, modelStr, { + success: false, + latencyMs: Date.now() - startTime, + fallbackCount, + strategy, + target: toRecordedTarget(target), + }); + recordedAttempts++; + // Fix #1707: Set terminal state so the fallback doesn't emit + // misleading ALL_ACCOUNTS_INACTIVE when the real issue is quality. + lastError = `Upstream response failed quality validation: ${quality.reason}`; + if (!lastStatus) lastStatus = 502; + if (i > 0) fallbackCount++; + break; // move to next model + } + const latencyMs = Date.now() - startTime; + log.info( + "COMBO", + `Model ${modelStr} succeeded (${latencyMs}ms, ${fallbackCount} fallbacks)` + ); + recordComboRequest(combo.name, modelStr, { + success: true, + latencyMs, + fallbackCount, + strategy, + target: toRecordedTarget(target), + }); + recordedAttempts++; + + // Context-relay intentionally splits responsibilities: + // combo.ts decides whether a successful turn should generate a handoff, + // while chat.ts injects the handoff after the real connectionId is resolved. + if ( + strategy === "context-relay" && + relayOptions?.sessionId && + relayConfig && + relayConfig.handoffProviders.includes(provider) && + provider === "codex" + ) { + const connectionId = getSessionConnection(relayOptions.sessionId); + if (connectionId) { + const quotaInfo = await fetchCodexQuota(connectionId).catch(() => null); + if (quotaInfo) { + const resetCandidates = [quotaInfo.window5h?.resetAt, quotaInfo.window7d?.resetAt] + .filter((value): value is string => typeof value === "string" && value.length > 0) + .sort((a, b) => a.localeCompare(b)); + const handoffSourceMessages = + Array.isArray(body?.messages) && body.messages.length > 0 + ? body.messages + : Array.isArray(body?.input) + ? body.input + : []; + + maybeGenerateHandoff({ + sessionId: relayOptions.sessionId, + comboName: combo.name, + connectionId, + percentUsed: quotaInfo.percentUsed, + messages: handoffSourceMessages, + model: modelStr, + expiresAt: resetCandidates[0] || null, + config: relayConfig, + handleSingleModel: handleSingleModelWrapped, + }); + } + } + } + + // Record last known good provider (LKGP) for this combo/model (#919) + if (provider) { + const connId = target.connectionId || undefined; + void (async () => { + try { + const { setLKGP } = await import("../../src/lib/localDb"); + await Promise.all([ + setLKGP(combo.name, target.executionKey, provider, connId), + setLKGP(combo.name, combo.id || combo.name, provider, connId), + ]); + } catch (err) { + log.warn("COMBO", "Failed to record Last Known Good Provider. This is non-fatal.", { + err, + }); + } + })(); + } + + return quality.clonedResponse ?? result; + } + + // Extract error info from response + let errorText = result.statusText || ""; + let errorBody = null; + let retryAfter = null; + try { + const cloned = result.clone(); + try { + const text = await cloned.text(); + if (text) { + errorText = text.substring(0, 500); + errorBody = JSON.parse(text); + errorText = + errorBody?.error?.message || errorBody?.error || errorBody?.message || errorText; + retryAfter = errorBody?.retryAfter || null; + } + } catch { + /* Clone parse failed */ + } + } catch { + /* Clone failed */ + } + + // Track earliest retryAfter + if ( + retryAfter && + (!earliestRetryAfter || new Date(retryAfter) < new Date(earliestRetryAfter)) + ) { + earliestRetryAfter = retryAfter; + } + + // Normalize error text + if (typeof errorText !== "string") { + try { + errorText = JSON.stringify(errorText); + } catch { + errorText = String(errorText); + } + } + + const isStreamReadinessFailure = + (result.status === 502 || result.status === 504) && + isStreamReadinessFailureErrorBody(errorBody); + + // Fix #1681: Status 499 means client disconnected — stop combo loop immediately. + // There is no point trying fallback models when nobody is listening. + if (result.status === 499) { + log.info("COMBO", `Client disconnected (499) during ${modelStr} — stopping combo loop`); recordComboRequest(combo.name, modelStr, { success: false, latencyMs: Date.now() - startTime, @@ -2093,134 +2251,56 @@ export async function handleComboChat({ target: toRecordedTarget(target), }); recordedAttempts++; - // Fix #1707: Set terminal state so the fallback doesn't emit - // misleading ALL_ACCOUNTS_INACTIVE when the real issue is quality. - lastError = `Upstream response failed quality validation: ${quality.reason}`; - if (!lastStatus) lastStatus = 502; - if (i > 0) fallbackCount++; - break; // move to next model + return result; } - const latencyMs = Date.now() - startTime; - log.info( - "COMBO", - `Model ${modelStr} succeeded (${latencyMs}ms, ${fallbackCount} fallbacks)` + + // Combo fallback is target-level orchestration: a non-ok target response is + // treated as local to that target and the combo continues to the next target. + // Error classification is retained only for retry/cooldown pacing; it must + // not decide whether fallback happens, including for generic 400 responses. + const fallbackResult = checkFallbackError( + result.status, + errorText, + 0, + null, + provider, + result.headers, + profile ); - recordComboRequest(combo.name, modelStr, { - success: true, - latencyMs, - fallbackCount, - strategy, - target: toRecordedTarget(target), - }); - recordedAttempts++; + const { cooldownMs } = fallbackResult; - // Context-relay intentionally splits responsibilities: - // combo.ts decides whether a successful turn should generate a handoff, - // while chat.ts injects the handoff after the real connectionId is resolved. + // #1731: If the entire provider quota is exhausted, mark it so subsequent + // same-provider targets are skipped immediately. + if (provider && isProviderExhaustedReason(fallbackResult)) { + exhaustedProviders.add(provider); + log.info( + "COMBO", + `Provider ${provider} quota exhausted — marking for skip on remaining targets (#1731)` + ); + } + + // Trigger shared provider circuit breaker for 5xx errors and connection failures. + // If the next target in the combo is on the same provider, don't mark the provider + // as failed — different models on the same provider may still succeed. + const nextTarget = orderedTargets[i + 1]; + const sameProviderNext = + typeof nextTarget?.provider === "string" && nextTarget.provider === provider; if ( - strategy === "context-relay" && - relayOptions?.sessionId && - relayConfig && - relayConfig.handoffProviders.includes(provider) && - provider === "codex" + !isStreamReadinessFailure && + isProviderFailureCode(result.status) && + !sameProviderNext ) { - const connectionId = getSessionConnection(relayOptions.sessionId); - if (connectionId) { - const quotaInfo = await fetchCodexQuota(connectionId).catch(() => null); - if (quotaInfo) { - const resetCandidates = [quotaInfo.window5h?.resetAt, quotaInfo.window7d?.resetAt] - .filter((value): value is string => typeof value === "string" && value.length > 0) - .sort((a, b) => a.localeCompare(b)); - const handoffSourceMessages = - Array.isArray(body?.messages) && body.messages.length > 0 - ? body.messages - : Array.isArray(body?.input) - ? body.input - : []; - - maybeGenerateHandoff({ - sessionId: relayOptions.sessionId, - comboName: combo.name, - connectionId, - percentUsed: quotaInfo.percentUsed, - messages: handoffSourceMessages, - model: modelStr, - expiresAt: resetCandidates[0] || null, - config: relayConfig, - handleSingleModel: handleSingleModelWrapped, - }); - } - } + recordProviderFailure(provider, log, target.connectionId, profile); } - // Record last known good provider (LKGP) for this combo/model (#919) - if (provider) { - const connId = target.connectionId || undefined; - void (async () => { - try { - const { setLKGP } = await import("../../src/lib/localDb"); - await Promise.all([ - setLKGP(combo.name, target.executionKey, provider, connId), - setLKGP(combo.name, combo.id || combo.name, provider, connId), - ]); - } catch (err) { - log.warn("COMBO", "Failed to record Last Known Good Provider. This is non-fatal.", { - err, - }); - } - })(); + // Check if this is a transient error worth retrying on same model + const isTransient = + !isStreamReadinessFailure && [408, 429, 500, 502, 503, 504].includes(result.status); + if (retry < maxRetries && isTransient) { + continue; // Retry same model } - return quality.clonedResponse ?? result; - } - - // Extract error info from response - let errorText = result.statusText || ""; - let errorBody = null; - let retryAfter = null; - try { - const cloned = result.clone(); - try { - const text = await cloned.text(); - if (text) { - errorText = text.substring(0, 500); - errorBody = JSON.parse(text); - errorText = - errorBody?.error?.message || errorBody?.error || errorBody?.message || errorText; - retryAfter = errorBody?.retryAfter || null; - } - } catch { - /* Clone parse failed */ - } - } catch { - /* Clone failed */ - } - - // Track earliest retryAfter - if ( - retryAfter && - (!earliestRetryAfter || new Date(retryAfter) < new Date(earliestRetryAfter)) - ) { - earliestRetryAfter = retryAfter; - } - - // Normalize error text - if (typeof errorText !== "string") { - try { - errorText = JSON.stringify(errorText); - } catch { - errorText = String(errorText); - } - } - - const isStreamReadinessFailure = - (result.status === 502 || result.status === 504) && - isStreamReadinessFailureErrorBody(errorBody); - - // Fix #1681: Status 499 means client disconnected — stop combo loop immediately. - // There is no point trying fallback models when nobody is listening. - if (result.status === 499) { - log.info("COMBO", `Client disconnected (499) during ${modelStr} — stopping combo loop`); + // Done retrying this model recordComboRequest(combo.name, modelStr, { success: false, latencyMs: Date.now() - startTime, @@ -2229,120 +2309,76 @@ export async function handleComboChat({ target: toRecordedTarget(target), }); recordedAttempts++; - return result; - } + lastError = errorText || String(result.status); + if (!lastStatus) lastStatus = result.status; + if (i > 0) fallbackCount++; + log.warn("COMBO", `Model ${modelStr} failed, trying next`, { status: result.status }); - // Combo fallback is target-level orchestration: a non-ok target response is - // treated as local to that target and the combo continues to the next target. - // Error classification is retained only for retry/cooldown pacing; it must - // not decide whether fallback happens, including for generic 400 responses. - const fallbackResult = checkFallbackError( - result.status, - errorText, - 0, - null, - provider, - result.headers, - profile - ); - const { cooldownMs } = fallbackResult; - - // #1731: If the entire provider quota is exhausted, mark it so subsequent - // same-provider targets are skipped immediately. - if (provider && isProviderExhaustedReason(fallbackResult)) { - exhaustedProviders.add(provider); - log.info( - "COMBO", - `Provider ${provider} quota exhausted — marking for skip on remaining targets (#1731)` - ); - } - - // Trigger shared provider circuit breaker for 5xx errors and connection failures - if (!isStreamReadinessFailure && isProviderFailureCode(result.status)) { - recordProviderFailure(provider, log, target.connectionId, profile); - } - - // Check if this is a transient error worth retrying on same model - const isTransient = - !isStreamReadinessFailure && [408, 429, 500, 502, 503, 504].includes(result.status); - if (retry < maxRetries && isTransient) { - continue; // Retry same model - } - - // Done retrying this model - recordComboRequest(combo.name, modelStr, { - success: false, - latencyMs: Date.now() - startTime, - fallbackCount, - strategy, - target: toRecordedTarget(target), - }); - recordedAttempts++; - lastError = errorText || String(result.status); - if (!lastStatus) lastStatus = result.status; - if (i > 0) fallbackCount++; - log.warn("COMBO", `Model ${modelStr} failed, trying next`, { status: result.status }); - - const fallbackWaitMs = - fallbackDelayMs > 0 && cooldownMs > 0 && cooldownMs <= MAX_FALLBACK_WAIT_MS - ? Math.min(cooldownMs, fallbackDelayMs) - : 0; - if ([502, 503, 504].includes(result.status) && fallbackWaitMs > 0) { - log.info("COMBO", `Waiting ${fallbackWaitMs}ms before fallback to next model`); - await new Promise((resolve) => { - const timer = setTimeout(resolve, fallbackWaitMs); - signal?.addEventListener( - "abort", - () => { - clearTimeout(timer); - resolve(undefined); - }, - { once: true } - ); - }); - if (signal?.aborted) { - log.info("COMBO", `Client disconnected during fallback wait — aborting`); - return errorResponse(499, "Client disconnected"); + const fallbackWaitMs = + fallbackDelayMs > 0 && cooldownMs > 0 && cooldownMs <= MAX_FALLBACK_WAIT_MS + ? Math.min(cooldownMs, fallbackDelayMs) + : 0; + if ([502, 503, 504].includes(result.status) && fallbackWaitMs > 0) { + log.info("COMBO", `Waiting ${fallbackWaitMs}ms before fallback to next model`); + await new Promise((resolve) => { + const timer = setTimeout(resolve, fallbackWaitMs); + signal?.addEventListener( + "abort", + () => { + clearTimeout(timer); + resolve(undefined); + }, + { once: true } + ); + }); + if (signal?.aborted) { + log.info("COMBO", `Client disconnected during fallback wait — aborting`); + return errorResponse(499, "Client disconnected"); + } } + + break; // Move to next model } - - break; // Move to next model } + + // All models failed in this set try + const latencyMs = Date.now() - startTime; + if (recordedAttempts === 0) { + recordComboRequest(combo.name, null, { success: false, latencyMs, fallbackCount, strategy }); + } + + // Retry the entire set if more attempts remain + if (setTry < maxSetRetries) continue; + + // All set retries exhausted — return the final error + if (!lastStatus) { + return new Response( + JSON.stringify({ + error: { + message: "Service temporarily unavailable: all upstream accounts are inactive", + type: "service_unavailable", + code: "ALL_ACCOUNTS_INACTIVE", + }, + }), + { status: 503, headers: { "Content-Type": "application/json" } } + ); + } + + const status = lastStatus; + const msg = lastError || "All combo models unavailable"; + + if (earliestRetryAfter) { + const retryHuman = formatRetryAfter(earliestRetryAfter); + log.warn("COMBO", `All models failed | ${msg} (${retryHuman})`); + return unavailableResponse(status, msg, earliestRetryAfter, retryHuman); + } + + log.warn("COMBO", `All models failed | ${msg}`); + return new Response(JSON.stringify({ error: { message: msg } }), { + status, + headers: { "Content-Type": "application/json" }, + }); } - - // All models failed - const latencyMs = Date.now() - startTime; - if (recordedAttempts === 0) { - recordComboRequest(combo.name, null, { success: false, latencyMs, fallbackCount, strategy }); - } - - if (!lastStatus) { - return new Response( - JSON.stringify({ - error: { - message: "Service temporarily unavailable: all upstream accounts are inactive", - type: "service_unavailable", - code: "ALL_ACCOUNTS_INACTIVE", - }, - }), - { status: 503, headers: { "Content-Type": "application/json" } } - ); - } - - const status = lastStatus; - const msg = lastError || "All combo models unavailable"; - - if (earliestRetryAfter) { - const retryHuman = formatRetryAfter(earliestRetryAfter); - log.warn("COMBO", `All models failed | ${msg} (${retryHuman})`); - return unavailableResponse(status, msg, earliestRetryAfter, retryHuman); - } - - log.warn("COMBO", `All models failed | ${msg}`); - return new Response(JSON.stringify({ error: { message: msg } }), { - status, - headers: { "Content-Type": "application/json" }, - }); } /** @@ -2474,7 +2510,10 @@ async function handleRoundRobinCombo({ `[RR #${counter}] → ${modelStr}${offset > 0 ? ` (fallback +${offset})` : ""}${retry > 0 ? ` (retry ${retry})` : ""}` ); - const result = await handleSingleModel(body, modelStr, target); + const result = await handleSingleModel(body, modelStr, { + ...target, + failoverBeforeRetry: config.failoverBeforeRetry, + }); // Success — validate response quality before returning if (result.ok) { diff --git a/open-sse/services/comboConfig.ts b/open-sse/services/comboConfig.ts index 727fa2436e..dcd02b9d33 100644 --- a/open-sse/services/comboConfig.ts +++ b/open-sse/services/comboConfig.ts @@ -23,6 +23,9 @@ const DEFAULT_COMBO_CONFIG = { resetAwareWeeklyWeight: 0.65, resetAwareTieBandPercent: 5, resetAwareExhaustionGuardPercent: 10, + failoverBeforeRetry: false, + maxSetRetries: 0, + setRetryDelayMs: 2000, }; const LEGACY_COMBO_RESILIENCE_KEYS = new Set([ diff --git a/package.json b/package.json index 8ae993b539..99bf05ca36 100644 --- a/package.json +++ b/package.json @@ -113,6 +113,7 @@ "test:protocols:e2e": "node scripts/dev/run-protocol-clients-tests.mjs", "test:vitest": "vitest run --config vitest.mcp.config.ts", "test:ecosystem": "node scripts/dev/run-ecosystem-tests.mjs", + "test:system": "cross-env DISABLE_SQLITE_AUTO_BACKUP=true node --import tsx --test --test-force-exit --test-concurrency=1 tests/e2e/system-failover.test.ts", "test:coverage": "cross-env DISABLE_SQLITE_AUTO_BACKUP=true c8 --output-dir coverage --exclude=tests/** --exclude=**/*.test.* --reporter=text-summary --reporter=html --reporter=json-summary --reporter=lcov --check-coverage --statements 75 --lines 75 --functions 75 --branches 70 node --import tsx --test --test-concurrency=1 tests/unit/*.test.ts", "test:coverage:legacy": "c8 --output-dir coverage --exclude=open-sse --check-coverage --lines 50 --functions 50 --branches 50 node --import tsx --test tests/unit/*.test.ts", "coverage:report": "c8 report --output-dir coverage --exclude=tests/** --exclude=**/*.test.* --reporter=text --reporter=text-summary --reporter=html --reporter=json-summary --reporter=lcov", diff --git a/src/app/(dashboard)/dashboard/cli-tools/CLIToolsPageClient.tsx b/src/app/(dashboard)/dashboard/cli-tools/CLIToolsPageClient.tsx index 126d75b215..f07d82e8da 100644 --- a/src/app/(dashboard)/dashboard/cli-tools/CLIToolsPageClient.tsx +++ b/src/app/(dashboard)/dashboard/cli-tools/CLIToolsPageClient.tsx @@ -180,8 +180,9 @@ export default function CLIToolsPageClient({ machineId: _machineId }) { activeProviders.map((c) => PROVIDER_ID_TO_ALIAS[c.provider] || c.provider) ); dynamicModels.forEach((dm) => { - const modelId = dm.id || dm; - if (seenModels.has(modelId)) return; + const rawId = dm?.id ?? dm; + const modelId = typeof rawId === "string" ? rawId : ""; + if (!modelId || seenModels.has(modelId)) return; // Parse alias/model format const slashIdx = modelId.indexOf("/"); if (slashIdx === -1) return; diff --git a/src/app/(dashboard)/dashboard/combos/page.tsx b/src/app/(dashboard)/dashboard/combos/page.tsx index d3f0c2a5b5..3eed8ae3f9 100644 --- a/src/app/(dashboard)/dashboard/combos/page.tsx +++ b/src/app/(dashboard)/dashboard/combos/page.tsx @@ -156,6 +156,12 @@ const ADVANCED_FIELD_HELP_FALLBACK = { "Round-robin combo/model limit: max simultaneous requests sent to each model target. This is separate from any provider account-only cap.", queueTimeout: "How long a request can wait for a round-robin model slot before timing out. This queue is separate from any account-only concurrency cap.", + failoverBeforeRetry: + "When enabled, a 429 from the upstream triggers immediate target failover instead of retrying the same URL first.", + maxSetRetries: + "Number of times to retry the full target set when every target fails. 0 = no set-level retry.", + setRetryDelayMs: + "Delay between set-level retry attempts, giving transient issues time to resolve.", }; const LEGACY_COMBO_RESILIENCE_KEYS = new Set([ @@ -1427,7 +1433,7 @@ function FieldLabelWithHelp({ label, help, showHelp = true }) {
{showHelp && ( - + help @@ -3540,6 +3546,94 @@ function ComboFormModal({ isOpen, combo, onClose, onSave, activeProviders, combo />
+ {/* failoverBeforeRetry + maxSetRetries + setRetryDelayMs */} +
+
+
+ + setConfig({ + ...config, + failoverBeforeRetry: e.target.checked || undefined, + }) + } + className="w-3.5 h-3.5 rounded border border-black/20 dark:border-white/20 accent-primary cursor-pointer" + /> + + + + help + + +
+
+
+ + + setConfig({ + ...config, + maxSetRetries: e.target.value ? Number(e.target.value) : undefined, + }) + } + className="w-full text-xs py-1.5 px-2 rounded border border-black/10 dark:border-white/10 bg-transparent focus:border-primary focus:outline-none" + /> +
+
+ + + setConfig({ + ...config, + setRetryDelayMs: e.target.value ? Number(e.target.value) : undefined, + }) + } + className="w-full text-xs py-1.5 px-2 rounded border border-black/10 dark:border-white/10 bg-transparent focus:border-primary focus:outline-none" + /> +
+
{strategy === "round-robin" && (
diff --git a/src/app/api/resilience/reset/route.ts b/src/app/api/resilience/reset/route.ts index 8c21960732..2e3a0a491a 100644 --- a/src/app/api/resilience/reset/route.ts +++ b/src/app/api/resilience/reset/route.ts @@ -1,7 +1,7 @@ import { NextResponse } from "next/server"; /** - * POST /api/resilience/reset — Reset all provider circuit breakers + * POST /api/resilience/reset — Reset all provider circuit breakers and model lockouts */ export async function POST() { try { @@ -17,10 +17,15 @@ export async function POST() { resetCount++; } + // Also clear in-memory model lockouts (per-model quota cooldowns) + const { clearAllModelLockouts } = + await import("@omniroute/open-sse/services/accountFallback.ts"); + clearAllModelLockouts(); + return NextResponse.json({ ok: true, resetCount, - message: `Reset ${resetCount} circuit breaker(s)`, + message: `Reset ${resetCount} circuit breaker(s) and model lockouts`, }); } catch (err: unknown) { const message = err instanceof Error ? err.message : "Failed to reset resilience state"; diff --git a/src/app/offline/page.tsx b/src/app/offline/page.tsx index 045b254fc7..a280f45814 100644 --- a/src/app/offline/page.tsx +++ b/src/app/offline/page.tsx @@ -1,25 +1,24 @@ "use client"; -import { useEffect, useState } from "react"; +import { useSyncExternalStore } from "react"; import Link from "next/link"; +function subscribeToOnline(callback: () => void) { + window.addEventListener("online", callback); + window.addEventListener("offline", callback); + return () => { + window.removeEventListener("online", callback); + window.removeEventListener("offline", callback); + }; +} + export default function OfflinePage() { - const [isOnline, setIsOnline] = useState(() => - typeof navigator !== "undefined" ? navigator.onLine : true + const isOnline = useSyncExternalStore( + subscribeToOnline, + () => navigator.onLine, + () => false ); - useEffect(() => { - const onOnline = () => setIsOnline(true); - const onOffline = () => setIsOnline(false); - - window.addEventListener("online", onOnline); - window.addEventListener("offline", onOffline); - return () => { - window.removeEventListener("online", onOnline); - window.removeEventListener("offline", onOffline); - }; - }, []); - return (
diff --git a/src/i18n/messages/en.json b/src/i18n/messages/en.json index c1a52d7f42..1366890f5f 100644 --- a/src/i18n/messages/en.json +++ b/src/i18n/messages/en.json @@ -1721,6 +1721,9 @@ "contextRelaySummaryModelHelp": "Optional override model used only for generating the handoff summary. Leave empty to reuse the active combo model.", "contextRelayProviderNote": "Context Relay currently generates handoffs for Codex account rotation. Pair it with multiple accounts of the same provider for the best continuity.", "advancedHint": "Leave empty to use global defaults. These override per-provider settings.", + "failoverBeforeRetry": "Failover Before Retry", + "maxSetRetries": "Max Set Retries", + "setRetryDelayMs": "Set Retry Delay (ms)", "moveUp": "Move up", "moveDown": "Move down", "removeModel": "Remove", @@ -1810,7 +1813,10 @@ "timeout": "Maximum request duration before aborting.", "healthcheck": "Skips unhealthy models/providers from routing decisions.", "concurrencyPerModel": "Max simultaneous requests allowed per model in round-robin.", - "queueTimeout": "How long a request can wait in queue before timing out." + "queueTimeout": "How long a request can wait in queue before timing out.", + "failoverBeforeRetry": "When enabled, any upstream error triggers immediate failover to the next combo target, skipping all retries and fallback URLs.", + "maxSetRetries": "Number of times to retry the full target set when every target fails. 0 = no set-level retry.", + "setRetryDelayMs": "Delay between set-level retry attempts, giving transient issues time to resolve." }, "templatesTitle": "Quick templates", "templatesDescription": "Apply a starting profile, then adjust models and config.", diff --git a/src/shared/validation/schemas.ts b/src/shared/validation/schemas.ts index 207df4e3aa..6a722602d8 100644 --- a/src/shared/validation/schemas.ts +++ b/src/shared/validation/schemas.ts @@ -517,6 +517,9 @@ const comboRuntimeConfigSchema = z maxComboDepth: z.coerce.number().int().min(1).max(10).optional(), trackMetrics: z.boolean().optional(), compressionMode: compressionModeSchema.optional(), + failoverBeforeRetry: z.boolean().optional(), + maxSetRetries: z.coerce.number().int().min(0).max(10).optional(), + setRetryDelayMs: z.coerce.number().int().min(0).max(60000).optional(), // Auto-Combo / LKGP Extensions candidatePool: z.array(z.string().min(1)).optional(), weights: scoringWeightsSchema.optional(), diff --git a/src/sse/handlers/chat.ts b/src/sse/handlers/chat.ts index 9e765146b5..24435e23c7 100644 --- a/src/sse/handlers/chat.ts +++ b/src/sse/handlers/chat.ts @@ -463,6 +463,7 @@ export async function handleChat(request: any, clientRawRequest: any = null) { executionKey?: string | null; stepId?: string | null; allowedConnectionIds?: string[] | null; + failoverBeforeRetry?: boolean; } ) => handleSingleModelChat( @@ -481,6 +482,7 @@ export async function handleChat(request: any, clientRawRequest: any = null) { allowedConnectionIds: target?.allowedConnectionIds ?? null, comboStepId: target?.stepId || null, comboExecutionKey: target?.executionKey || target?.stepId || null, + skipUpstreamRetry: target?.failoverBeforeRetry ?? false, preselectedCredentials: comboPreselectedCredentials.get( getComboCredentialCacheKey(m, target) ), @@ -611,6 +613,7 @@ async function handleSingleModelChat( allowedConnectionIds?: string[] | null; comboStepId?: string | null; comboExecutionKey?: string | null; + skipUpstreamRetry?: boolean; preselectedCredentials?: any; cachedSettings?: any; } = {}, @@ -643,6 +646,7 @@ async function handleSingleModelChat( connectionId?: string | null; executionKey?: string | null; stepId?: string | null; + failoverBeforeRetry?: boolean; } ) => handleSingleModelChat( @@ -660,6 +664,7 @@ async function handleSingleModelChat( allowedConnectionIds: null, comboStepId: null, comboExecutionKey: null, + skipUpstreamRetry: target?.failoverBeforeRetry ?? false, }, redirectCombo.strategy ?? "priority", false @@ -916,6 +921,7 @@ async function handleSingleModelChat( modelApiFormat: apiFormat, providerProfile, cachedSettings: runtimeOptions.cachedSettings, + skipUpstreamRetry: runtimeOptions.skipUpstreamRetry ?? false, }); if (telemetry) telemetry.endPhase(); @@ -1129,7 +1135,11 @@ async function handleSingleModelChat( continue; } - if (!forceLiveComboTest && PROVIDER_BREAKER_FAILURE_STATUSES.has(Number(result.status))) { + if ( + !forceLiveComboTest && + !isCombo && + PROVIDER_BREAKER_FAILURE_STATUSES.has(Number(result.status)) + ) { breaker._onFailure(); } diff --git a/src/sse/handlers/chatHelpers.ts b/src/sse/handlers/chatHelpers.ts index 48ca9e4366..0049918a29 100644 --- a/src/sse/handlers/chatHelpers.ts +++ b/src/sse/handlers/chatHelpers.ts @@ -303,6 +303,7 @@ export async function executeChatWithBreaker({ modelApiFormat, providerProfile, cachedSettings, + skipUpstreamRetry = false, }: any): Promise<{ result: any; tlsFingerprintUsed: boolean }> { let tlsFingerprintUsed = false; @@ -324,6 +325,7 @@ export async function executeChatWithBreaker({ comboStepId, comboExecutionKey, cachedSettings, + skipUpstreamRetry, onCredentialsRefreshed: async (newCreds: any) => { await updateProviderCredentials(credentials.connectionId, { accessToken: newCreds.accessToken, diff --git a/src/sse/services/auth.ts b/src/sse/services/auth.ts index 358acd45ba..0082e6bfaf 100644 --- a/src/sse/services/auth.ts +++ b/src/sse/services/auth.ts @@ -1567,8 +1567,14 @@ export async function markAccountUnavailable( | undefined; const isPerModelQuotaProvider = hasPerModelQuota(provider, model, connectionPassthroughModels); - if (isPerModelQuotaProvider && provider && model && (status === 404 || status === 429)) { - const reason = status === 404 ? "not_found" : "rate_limited"; + if ( + isPerModelQuotaProvider && + provider && + model && + (status === 404 || status === 429 || status >= 500) + ) { + const reason = + status === 404 ? "not_found" : status === 429 ? "rate_limited" : "server_error"; const lockout = recordModelLockoutFailure( provider, connectionId, diff --git a/tests/e2e/combos-flow.spec.ts b/tests/e2e/combos-flow.spec.ts index 986ed9e6e7..5a6d929f8a 100644 --- a/tests/e2e/combos-flow.spec.ts +++ b/tests/e2e/combos-flow.spec.ts @@ -456,6 +456,13 @@ test.describe("Combos flow", () => { await comboDialog.locator('[data-testid="combo-manual-model-add"]').click(); await expect(comboDialog.locator('[data-testid="combo-readiness-panel"]')).toHaveCount(0); + // New advanced settings: failoverBeforeRetry, maxSetRetries, setRetryDelayMs + await comboDialog.locator("#failoverBeforeRetry").check(); + // maxSetRetries: placeholder "0" is unique (maxRetries uses "1") + await comboDialog.locator('input[placeholder="0"]').fill("2"); + // setRetryDelayMs: placeholder "2000" — last occurrence is the new field + await comboDialog.locator('input[placeholder="2000"]').last().fill("1500"); + await comboDialog .getByRole("button", { name: /create combo|criar combo/i }) .last() @@ -476,6 +483,9 @@ test.describe("Combos flow", () => { weight: 0, }, ]); + expect(state.lastPayload?.config?.failoverBeforeRetry).toBe(true); + expect(state.lastPayload?.config?.maxSetRetries).toBe(2); + expect(state.lastPayload?.config?.setRetryDelayMs).toBe(1500); }); test("allows dragging combo cards to persist manual order", async ({ page }) => { diff --git a/tests/e2e/helpers/mockUpstreamServer.ts b/tests/e2e/helpers/mockUpstreamServer.ts new file mode 100644 index 0000000000..35bd4d594a --- /dev/null +++ b/tests/e2e/helpers/mockUpstreamServer.ts @@ -0,0 +1,175 @@ +import http from "node:http"; +import net from "node:net"; + +export interface PlannedResponse { + status: number; + body: Record; + headers?: Record; + delayMs?: number; +} + +export interface TokenState { + defaultResponse: PlannedResponse; + queue: PlannedResponse[]; + hits: number; + startedAt: number[]; + bodies: Array>; +} + +function getFreePort(): Promise { + return new Promise((resolve, reject) => { + const server = net.createServer(); + server.once("error", reject); + server.listen(0, "127.0.0.1", () => { + const address = server.address(); + if (!address || typeof address === "string") { + server.close(); + reject(new Error("Failed to allocate a free port")); + return; + } + const { port } = address; + server.close((err) => { + if (err) reject(err); + else resolve(port); + }); + }); + }); +} + +export function buildCompletion( + text: string, + overrides: Partial & { model?: string } = {} +) { + return { + status: overrides.status ?? 200, + delayMs: overrides.delayMs, + headers: overrides.headers, + body: overrides.body ?? { + id: `chatcmpl_${Math.random().toString(16).slice(2, 8)}`, + object: "chat.completion", + model: overrides.model ?? "test-model", + choices: [{ index: 0, message: { role: "assistant", content: text }, finish_reason: "stop" }], + usage: { prompt_tokens: 4, completion_tokens: 2, total_tokens: 6 }, + }, + }; +} + +export function buildError(status: number, message: string, headers: Record = {}) { + return { + status, + headers, + body: { error: { message } }, + }; +} + +export class MockUpstreamServer { + private behaviors = new Map(); + private server: http.Server | null = null; + private _baseUrl = ""; + + get baseUrl(): string { + if (!this._baseUrl) throw new Error("Server not started yet"); + return this._baseUrl; + } + + get isRunning(): boolean { + return this.server !== null; + } + + async start(): Promise { + const port = await getFreePort(); + this.server = http.createServer((req, res) => { + void this.handleRequest(req, res); + }); + await new Promise((resolve, reject) => { + this.server!.once("error", reject); + this.server!.listen(port, "127.0.0.1", () => resolve()); + }); + this._baseUrl = `http://127.0.0.1:${port}/v1`; + return this._baseUrl; + } + + configureToken( + token: string, + config: { defaultResponse: PlannedResponse; queue?: PlannedResponse[] } + ): void { + this.behaviors.set(token, { + defaultResponse: config.defaultResponse, + queue: [...(config.queue || [])], + hits: 0, + startedAt: [], + bodies: [], + }); + } + + getState(token: string): TokenState { + const state = this.behaviors.get(token); + if (!state) throw new Error(`Unknown token: ${token}`); + return state; + } + + resetState(token: string, queue?: PlannedResponse[]): void { + const state = this.behaviors.get(token); + if (!state) throw new Error(`Unknown token: ${token}`); + state.hits = 0; + state.startedAt = []; + state.bodies = []; + state.queue = [...(queue || [])]; + } + + async stop(): Promise { + if (!this.server) return; + await new Promise((resolve) => this.server?.close(() => resolve())); + this.server = null; + } + + private async handleRequest(req: http.IncomingMessage, res: http.ServerResponse): Promise { + const chunks: Buffer[] = []; + for await (const chunk of req) { + chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk)); + } + + const authHeader = String(req.headers.authorization || ""); + const token = authHeader.replace(/^Bearer\s+/i, "").trim(); + const rawBody = Buffer.concat(chunks).toString("utf8"); + const parsedBody = rawBody ? JSON.parse(rawBody) : {}; + + if (req.method === "GET" && req.url?.startsWith("/v1/models")) { + res.writeHead(200, { "Content-Type": "application/json" }); + res.end(JSON.stringify({ data: [{ id: "test-model", object: "model" }] })); + return; + } + + if (req.method !== "POST" || !req.url?.startsWith("/v1/chat/completions")) { + res.writeHead(404, { "Content-Type": "application/json" }); + res.end(JSON.stringify({ error: { message: `Unhandled: ${req.method} ${req.url}` } })); + return; + } + + const behavior = this.behaviors.get(token); + if (!behavior) { + res.writeHead(401, { "Content-Type": "application/json" }); + res.end(JSON.stringify({ error: { message: `Unknown token: ${token || "missing"}` } })); + return; + } + + behavior.hits += 1; + behavior.startedAt.push(Date.now()); + behavior.bodies.push(parsedBody as Record); + + const planned = behavior.queue.shift() || behavior.defaultResponse; + process.stderr.write( + `[MOCK] ${token.slice(0, 8)} hit=${behavior.hits} status=${planned.status}\n` + ); + if (planned.delayMs && planned.delayMs > 0) { + await new Promise((r) => setTimeout(r, planned.delayMs)); + } + + const headers: Record = { + "Content-Type": "application/json", + ...(planned.headers || {}), + }; + res.writeHead(planned.status, headers); + res.end(JSON.stringify(planned.body)); + } +} diff --git a/tests/e2e/system-failover.test.ts b/tests/e2e/system-failover.test.ts new file mode 100644 index 0000000000..5141fadb7f --- /dev/null +++ b/tests/e2e/system-failover.test.ts @@ -0,0 +1,699 @@ +import test from "node:test"; +import assert from "node:assert/strict"; +import fs from "node:fs"; +import fsp from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import net from "node:net"; +import { spawn } from "node:child_process"; +import { fileURLToPath } from "node:url"; +import { MockUpstreamServer, buildCompletion, buildError } from "./helpers/mockUpstreamServer.ts"; + +const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-system-failover-")); +const DASHBOARD_PORT = await getFreePort(); +const REPO_ROOT = fileURLToPath(new URL("../..", import.meta.url)); + +process.env.DATA_DIR = TEST_DATA_DIR; +process.env.DISABLE_SQLITE_AUTO_BACKUP = "true"; +process.env.API_KEY_SECRET = process.env.API_KEY_SECRET || "system-failover-secret-123456"; +process.env.REQUIRE_API_KEY = "false"; + +const core = await import("../../src/lib/db/core.ts"); +const providersDb = await import("../../src/lib/db/providers.ts"); +const combosDb = await import("../../src/lib/db/combos.ts"); +const settingsDb = await import("../../src/lib/db/settings.ts"); +const accountFallback = await import("../../open-sse/services/accountFallback.ts"); + +function resetConnectionCooldowns() { + accountFallback.clearAllModelLockouts(); + const db = core.getDbInstance() as any; + db.prepare( + `UPDATE provider_connections + SET rate_limited_until = NULL, + test_status = 'active', + backoff_level = 0, + last_error = NULL, + last_error_type = NULL, + last_error_source = NULL, + error_code = NULL, + last_error_at = NULL + WHERE rate_limited_until IS NOT NULL + OR test_status != 'active'` + ).run(); + db.pragma("wal_checkpoint(TRUNCATE)"); +} + +function getFreePort() { + return new Promise((resolve, reject) => { + const server = net.createServer(); + server.once("error", reject); + server.listen(0, "127.0.0.1", () => { + const address = server.address(); + if (!address || typeof address === "string") { + server.close(); + reject(new Error("Failed to allocate a free port")); + return; + } + const { port } = address; + server.close((closeError) => { + if (closeError) reject(closeError); + else resolve(port); + }); + }); + }); +} + +function sleep(ms: number) { + return new Promise((resolve) => setTimeout(resolve, ms)); +} + +async function seedProvider(label: string, apiKey: string, baseUrl: string) { + const providerId = `openai-compatible-sys-${label}`; + await providersDb.createProviderNode({ + id: providerId, + type: "openai-compatible", + name: `System ${label}`, + prefix: label, + apiType: "chat", + baseUrl, + }); + await providersDb.createProviderConnection({ + provider: providerId, + authType: "apikey", + name: `conn-${label}`, + apiKey, + isActive: true, + testStatus: "active", + providerSpecificData: { baseUrl, apiType: "chat" }, + }); + return { providerId, model: `${label}/test-model`, apiKey }; +} + +function createServerProcess(dataDir: string, port: number) { + const stdoutLines: string[] = []; + const stderrLines: string[] = []; + let exitInfo: { code: number | null; signal: NodeJS.Signals | null } | null = null; + const child = spawn(process.execPath, ["scripts/dev/run-next-playwright.mjs", "dev"], { + cwd: REPO_ROOT, + env: { + ...process.env, + DATA_DIR: dataDir, + PORT: String(port), + DASHBOARD_PORT: String(port), + API_PORT: String(port), + HOST: "127.0.0.1", + REQUIRE_API_KEY: "false", + API_KEY_SECRET: process.env.API_KEY_SECRET || "system-failover-secret-123456", + DISABLE_SQLITE_AUTO_BACKUP: "true", + INITIAL_PASSWORD: "", + NEXT_TELEMETRY_DISABLED: "1", + OMNIROUTE_DISABLE_BACKGROUND_SERVICES: "true", + OMNIROUTE_DISABLE_TOKEN_HEALTHCHECK: "true", + OMNIROUTE_DISABLE_LOCAL_HEALTHCHECK: "true", + OMNIROUTE_HIDE_HEALTHCHECK_LOGS: "true", + OMNIROUTE_E2E_BOOTSTRAP_MODE: "open", + }, + stdio: ["ignore", "pipe", "pipe"], + }); + + child.once("exit", (code, signal) => { + exitInfo = { code, signal }; + }); + child.stdout.on("data", (chunk) => { + const lines = String(chunk).split(/\r?\n/).filter(Boolean); + stdoutLines.push(...lines); + if (stdoutLines.length > 200) stdoutLines.splice(0, stdoutLines.length - 200); + }); + child.stderr.on("data", (chunk) => { + const lines = String(chunk).split(/\r?\n/).filter(Boolean); + stderrLines.push(...lines); + if (stderrLines.length > 200) stderrLines.splice(0, stderrLines.length - 200); + }); + + return { + child, + stdoutLines, + stderrLines, + baseUrl: `http://127.0.0.1:${port}`, + get exitInfo() { + return exitInfo; + }, + }; +} + +async function waitForServer( + baseUrl: string, + logs: { + stdoutLines: string[]; + stderrLines: string[]; + exitInfo?: { code: number | null; signal: NodeJS.Signals | null } | null; + } +) { + const startedAt = Date.now(); + let lastError = ""; + while (Date.now() - startedAt < 120_000) { + if (logs.exitInfo) { + throw new Error( + [ + `OmniRoute exited before it became ready (code=${logs.exitInfo.code}, signal=${logs.exitInfo.signal})`, + "--- stdout ---", + ...logs.stdoutLines.slice(-40), + "--- stderr ---", + ...logs.stderrLines.slice(-40), + ].join("\n") + ); + } + + try { + const response = await fetch(`${baseUrl}/api/monitoring/health`, { + signal: AbortSignal.timeout(5_000), + }); + if (response.ok) return; + lastError = `HTTP ${response.status}`; + } catch (error: any) { + lastError = error instanceof Error ? error.message : String(error); + } + await sleep(500); + } + + throw new Error( + [ + `Timed out waiting for OmniRoute to start: ${lastError}`, + "--- stdout ---", + ...logs.stdoutLines.slice(-40), + "--- stderr ---", + ...logs.stderrLines.slice(-40), + ].join("\n") + ); +} + +async function stopProcess(child: ReturnType) { + if (child.killed) return; + child.kill("SIGTERM"); + const exited = await Promise.race([ + new Promise((resolve) => child.once("exit", () => resolve(true))), + sleep(5_000).then(() => false), + ]); + if (!exited && !child.killed) { + child.kill("SIGKILL"); + await new Promise((resolve) => child.once("exit", () => resolve())); + } +} + +async function postChat( + baseUrl: string, + model: string, + content: string, + extraHeaders?: Record +) { + const response = await fetch(`${baseUrl}/api/v1/chat/completions`, { + method: "POST", + headers: { "Content-Type": "application/json", ...extraHeaders }, + body: JSON.stringify({ + model, + stream: false, + messages: [{ role: "user", content }], + }), + signal: AbortSignal.timeout(30_000), + }); + const text = await response.text(); + const json = text ? JSON.parse(text) : {}; + return { response, json }; +} + +async function resetBreakers(url: string) { + await fetch(`${url}/api/resilience/reset`, { + method: "POST", + signal: AbortSignal.timeout(5_000), + }); +} + +const serverA = new MockUpstreamServer(); +const serverB = new MockUpstreamServer(); +let app: + | { + child: ReturnType; + stdoutLines: string[]; + stderrLines: string[]; + baseUrl: string; + } + | undefined; + +const TOKEN_A = "sk-sys-a"; +const TOKEN_B = "sk-sys-b"; +const TOKEN_A2 = "sk-sys-a2"; +const TOKEN_B2 = "sk-sys-b2"; + +test.before(async () => { + const baseUrlA = await serverA.start(); + const baseUrlB = await serverB.start(); + + serverA.configureToken(TOKEN_A, { + defaultResponse: buildCompletion("server A ok", { model: "sys-a/test-model" }), + }); + serverA.configureToken(TOKEN_A2, { + defaultResponse: buildCompletion("server A2 ok", { model: "sys-a2/test-model" }), + }); + serverB.configureToken(TOKEN_B, { + defaultResponse: buildCompletion("server B ok", { model: "sys-b/test-model" }), + }); + serverB.configureToken(TOKEN_B2, { + defaultResponse: buildCompletion("server B2 ok", { model: "sys-b2/test-model" }), + }); + + const provA = await seedProvider("sys-a", TOKEN_A, baseUrlA); + const provB = await seedProvider("sys-b", TOKEN_B, baseUrlB); + const provA2 = await seedProvider("sys-a2", TOKEN_A2, baseUrlA); + const provB2 = await seedProvider("sys-b2", TOKEN_B2, baseUrlB); + + await combosDb.createCombo({ + name: "sys-priority", + strategy: "priority", + config: { maxRetries: 0, retryDelayMs: 0 }, + models: [provA.model, provB.model], + }); + await combosDb.createCombo({ + name: "sys-priority-v2", + strategy: "priority", + config: { maxRetries: 0, retryDelayMs: 0 }, + models: [provA2.model, provB2.model], + }); + await combosDb.createCombo({ + name: "sys-priority-fobr", + strategy: "priority", + config: { maxRetries: 0, retryDelayMs: 0, failoverBeforeRetry: true }, + models: [provA.model, provB.model], + }); + await combosDb.createCombo({ + name: "sys-priority-setretry", + strategy: "priority", + config: { + maxRetries: 0, + retryDelayMs: 0, + failoverBeforeRetry: true, + maxSetRetries: 1, + setRetryDelayMs: 500, + }, + models: [provA.model, provB.model], + }); + await combosDb.createCombo({ + name: "sys-same-server", + strategy: "priority", + config: { maxRetries: 0, retryDelayMs: 0 }, + models: [provA.model, provA2.model], + }); + await combosDb.createCombo({ + name: "sys-same-server-fobr", + strategy: "priority", + config: { maxRetries: 0, retryDelayMs: 0, failoverBeforeRetry: true }, + models: [provA.model, provA2.model], + }); + + await combosDb.createCombo({ + name: "sys-single-provider", + strategy: "priority", + config: { maxRetries: 0, retryDelayMs: 0 }, + models: ["sys-a/modelA", "sys-a/modelB"], + }); + await combosDb.createCombo({ + name: "sys-single-provider-fobr", + strategy: "priority", + config: { maxRetries: 0, retryDelayMs: 0, failoverBeforeRetry: true }, + models: ["sys-a/modelA", "sys-a/modelB"], + }); + + await settingsDb.updateSettings({ + resilienceSettings: { + requestQueue: { + autoEnableApiKeyProviders: true, + requestsPerMinute: 120, + minTimeBetweenRequestsMs: 0, + concurrentRequests: 4, + maxWaitMs: 2_000, + }, + connectionCooldown: { + oauth: { baseCooldownMs: 500, useUpstreamRetryHints: true, maxBackoffSteps: 3 }, + apikey: { baseCooldownMs: 200, useUpstreamRetryHints: false, maxBackoffSteps: 0 }, + }, + providerBreaker: { + oauth: { failureThreshold: 3, resetTimeoutMs: 2_000 }, + apikey: { failureThreshold: 2, resetTimeoutMs: 1_500 }, + }, + waitForCooldown: { + enabled: false, + maxRetries: 0, + maxRetryWaitSec: 0, + }, + }, + requestRetry: 0, + maxRetryIntervalSec: 0, + requireLogin: false, + setupComplete: true, + }); + + core.closeDbInstance(); + + app = createServerProcess(TEST_DATA_DIR, DASHBOARD_PORT); + await waitForServer(app.baseUrl, app); + + const warmup = await postChat(app.baseUrl, "sys-b/test-model", "warm up"); + assert.equal(warmup.response.status, 200, JSON.stringify(warmup.json)); + serverB.resetState(TOKEN_B); +}); + +test.after(async () => { + if (app) await stopProcess(app.child); + await serverA.stop(); + await serverB.stop(); + core.closeDbInstance(); + await fsp.rm(TEST_DATA_DIR, { recursive: true, force: true }); +}); + +test("primary healthy: request routes to Server A only", async () => { + assert.ok(app); + serverA.resetState(TOKEN_A); + serverB.resetState(TOKEN_B); + + const result = await postChat(app.baseUrl, "sys-priority", "healthy primary"); + + assert.equal(result.response.status, 200, JSON.stringify(result.json)); + assert.equal(result.json.choices[0].message.content, "server A ok"); + assert.equal(result.json.model, "sys-a/test-model"); + assert.equal(serverA.getState(TOKEN_A).hits, 1); + assert.equal(serverB.getState(TOKEN_B).hits, 0); +}); + +test("500 Internal Server Error: combo falls back to Server B", async () => { + assert.ok(app); + serverA.resetState(TOKEN_A, [buildError(500, "Internal Server Error")]); + serverB.resetState(TOKEN_B); + + const result = await postChat(app.baseUrl, "sys-priority", "500 fallback"); + + assert.equal(result.response.status, 200, JSON.stringify(result.json)); + assert.equal(result.json.choices[0].message.content, "server B ok"); + assert.equal(result.json.model, "sys-b/test-model"); + assert.equal(serverA.getState(TOKEN_A).hits, 1); + assert.equal(serverB.getState(TOKEN_B).hits, 1); +}); + +test("503 Service Unavailable: combo falls back to Server B", async () => { + assert.ok(app); + await resetBreakers(app.baseUrl); + resetConnectionCooldowns(); + serverA.resetState(TOKEN_A, [buildError(503, "Service Unavailable")]); + serverB.resetState(TOKEN_B); + + const result = await postChat(app.baseUrl, "sys-priority", "503 fallback"); + + assert.equal(result.response.status, 200, JSON.stringify(result.json)); + assert.equal(result.json.choices[0].message.content, "server B ok"); + assert.equal(result.json.model, "sys-b/test-model"); + assert.equal(serverA.getState(TOKEN_A).hits, 1); + assert.equal(serverB.getState(TOKEN_B).hits, 1); +}); + +test("both servers fail (500): request returns a 5xx error to the client", async () => { + assert.ok(app); + await resetBreakers(app.baseUrl); + resetConnectionCooldowns(); + serverA.resetState(TOKEN_A, [buildError(500, "A is down")]); + serverB.resetState(TOKEN_B, [buildError(500, "B is down")]); + + const result = await postChat(app.baseUrl, "sys-priority", "both down"); + + assert.ok(result.response.status >= 500, `expected 5xx, got ${result.response.status}`); +}); + +test("combo fallback to Server B survives sequential 503 failures from Server A", async () => { + assert.ok(app); + await resetBreakers(app.baseUrl); + resetConnectionCooldowns(); + serverA.resetState(TOKEN_A, [buildError(503, "transient blip"), buildError(503, "second blip")]); + serverB.resetState(TOKEN_B); + + const first = await postChat(app.baseUrl, "sys-priority", "seq 503 attempt 1"); + assert.equal(first.response.status, 200, JSON.stringify(first.json)); + assert.equal(first.json.choices[0].message.content, "server B ok"); + assert.equal(first.json.model, "sys-b/test-model"); + + // Wait for the 200ms apikey cooldown to expire so the second request also + // goes through the full A→B fallback path rather than skipping A entirely. + await sleep(250); + + const second = await postChat(app.baseUrl, "sys-priority", "seq 503 attempt 2"); + assert.equal(second.response.status, 200, JSON.stringify(second.json)); + assert.equal(second.json.choices[0].message.content, "server B ok"); + assert.equal(second.json.model, "sys-b/test-model"); + + assert.equal(serverA.getState(TOKEN_A).hits, 2); + assert.equal(serverB.getState(TOKEN_B).hits, 2); +}); + +test("429 with Retry-After and wait-for-cooldown: primary retries then falls back to B", async () => { + assert.ok(app); + await resetBreakers(app.baseUrl); + resetConnectionCooldowns(); + serverA.resetState(TOKEN_A, [ + buildError(429, "rate limited, retry after 1s", { "Retry-After": "1" }), + ]); + serverB.resetState(TOKEN_B); + + const patchRes = await fetch(`${app.baseUrl}/api/resilience`, { + method: "PATCH", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + connectionCooldown: { + apikey: { useUpstreamRetryHints: true, baseCooldownMs: 200 }, + }, + waitForCooldown: { enabled: true, maxRetries: 1, maxRetryWaitSec: 2 }, + }), + signal: AbortSignal.timeout(10_000), + }); +}); + +test("failoverBeforeRetry enabled: upstream error triggers immediate failover to next target", async () => { + assert.ok(app); + await resetBreakers(app.baseUrl); + resetConnectionCooldowns(); + serverA.resetState(TOKEN_A, [buildError(429, "rate limited")]); + serverB.resetState(TOKEN_B); + + const result = await postChat(app.baseUrl, "sys-priority-fobr", "test failover before retry"); + + assert.equal(result.response.status, 200, JSON.stringify(result.json)); + assert.equal(result.json.choices[0].message.content, "server B ok"); + assert.equal(result.json.model, "sys-b/test-model"); + + // With failoverBeforeRetry=true, A should be hit exactly ONCE (no intra-URL retry) + assert.equal(serverA.getState(TOKEN_A).hits, 1); + assert.equal(serverB.getState(TOKEN_B).hits, 1); +}); + +test("failoverBeforeRetry disabled: 429 triggers executor intra-URL retry, succeeds on retry", async () => { + assert.ok(app); + // Full resilience reset so A isn't blocked by residual breaker/cooldown state + const patchRes = await fetch(`${app.baseUrl}/api/resilience`, { + method: "PATCH", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + connectionCooldown: { + apikey: { useUpstreamRetryHints: false, baseCooldownMs: 0, maxBackoffSteps: 0 }, + oauth: { useUpstreamRetryHints: false, baseCooldownMs: 0, maxBackoffSteps: 0 }, + }, + waitForCooldown: { enabled: false, maxRetries: 0, maxRetryWaitSec: 0 }, + }), + signal: AbortSignal.timeout(10_000), + }); + assert.equal(patchRes.status, 200); + await resetBreakers(app.baseUrl); + resetConnectionCooldowns(); + // Wait for any residual cooldowns to expire + await sleep(300); + // One 429, then the default (200) on retry + serverA.resetState(TOKEN_A, [buildError(429, "rate limited")]); + serverB.resetState(TOKEN_B); + + const result = await postChat(app.baseUrl, "sys-priority", "test failover before retry disabled"); + + assert.equal(result.response.status, 200, JSON.stringify(result.json)); + + // With maxRetries=0 the combo does not retry A — it fails over to B immediately. + assert.equal(result.json.model, "sys-b/test-model"); + + // With failoverBeforeRetry=false, A should be hit TWICE (initial + 1 intra-URL retry). + // This contrasts with failoverBeforeRetry=true where A is hit exactly ONCE. + assert.equal(serverA.getState(TOKEN_A).hits, 2); +}); + +test("maxSetRetries: both A and B fail first pass, A 429 again, B 200 on retry", async () => { + assert.ok(app); + // Reset to a clean resilience slate: disable cooldowns, disable waitForCooldown, + // reset breakers, and clear connection rate_limited_until from previous tests. + const patchRes = await fetch(`${app.baseUrl}/api/resilience`, { + method: "PATCH", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + connectionCooldown: { + apikey: { useUpstreamRetryHints: false, baseCooldownMs: 0, maxBackoffSteps: 0 }, + oauth: { useUpstreamRetryHints: false, baseCooldownMs: 0, maxBackoffSteps: 0 }, + }, + waitForCooldown: { enabled: false, maxRetries: 0, maxRetryWaitSec: 0 }, + }), + signal: AbortSignal.timeout(10_000), + }); + assert.equal(patchRes.status, 200); + await resetBreakers(app.baseUrl); + resetConnectionCooldowns(); + // Set try 0: A 429, B 500 — both fail + // Set try 1: A 429, B 200 — B succeeds + serverA.resetState(TOKEN_A, [buildError(429, "rate limited"), buildError(429, "rate limited")]); + serverB.resetState(TOKEN_B, [ + buildError(500, "server error"), + buildCompletion("server B ok on retry", { model: "sys-b/test-model" }), + ]); + + const result = await postChat(app.baseUrl, "sys-priority-setretry", "test max set retries", { + "x-internal-test": "combo-health-check", + }); + + // Set try 0: A 429, B 500 → both fail + // Set try 1: A 429, B 200 → B succeeds + assert.equal(result.response.status, 200, JSON.stringify(result.json)); + assert.equal(result.json.choices[0].message.content, "server B ok on retry"); + assert.equal(result.json.model, "sys-b/test-model"); + assert.equal(serverA.getState(TOKEN_A).hits, 2); + assert.equal(serverB.getState(TOKEN_B).hits, 2); + + // Restore defaults so other tests are not affected + await fetch(`${app.baseUrl}/api/resilience`, { + method: "PATCH", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + connectionCooldown: { + apikey: { useUpstreamRetryHints: false, baseCooldownMs: 200, maxBackoffSteps: 0 }, + oauth: { useUpstreamRetryHints: true, baseCooldownMs: 500, maxBackoffSteps: 3 }, + }, + }), + signal: AbortSignal.timeout(10_000), + }); +}); + +test("same server failoverBeforeRetry disabled: first model 429 retried before trying second", async () => { + assert.ok(app); + // Full resilience reset — previous test restored defaults and may have left A cooldown + const patchRes = await fetch(`${app.baseUrl}/api/resilience`, { + method: "PATCH", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + connectionCooldown: { + apikey: { useUpstreamRetryHints: false, baseCooldownMs: 0, maxBackoffSteps: 0 }, + oauth: { useUpstreamRetryHints: false, baseCooldownMs: 0, maxBackoffSteps: 0 }, + }, + waitForCooldown: { enabled: false, maxRetries: 0, maxRetryWaitSec: 0 }, + }), + signal: AbortSignal.timeout(10_000), + }); + assert.equal(patchRes.status, 200); + await sleep(300); + await resetBreakers(app.baseUrl); + resetConnectionCooldowns(); + serverA.resetState(TOKEN_A, [buildError(429, "rate limited")]); + serverA.resetState(TOKEN_A2); + + const result = await postChat( + app.baseUrl, + "sys-same-server", + "test same server failover disabled" + ); + + assert.equal(result.response.status, 200, JSON.stringify(result.json)); + + // With maxRetries=0 the combo fails over to A2 rather than retrying A. + assert.equal(result.json.model, "sys-a2/test-model"); + assert.equal(serverA.getState(TOKEN_A).hits, 2); +}); + +test("same server failoverBeforeRetry enabled: first model 429 skipped to second immediately", async () => { + assert.ok(app); + await sleep(300); + await resetBreakers(app.baseUrl); + resetConnectionCooldowns(); + serverA.resetState(TOKEN_A, [buildError(429, "rate limited")]); + serverA.resetState(TOKEN_A2); + + const result = await postChat( + app.baseUrl, + "sys-same-server-fobr", + "test same server failover enabled" + ); + + assert.equal(result.response.status, 200, JSON.stringify(result.json)); + assert.equal(result.json.model, "sys-a2/test-model"); + assert.equal(serverA.getState(TOKEN_A).hits, 1); + assert.equal(serverA.getState(TOKEN_A2).hits, 1); +}); + +test("single provider, modelA 500: combo fails over to modelB", async () => { + assert.ok(app); + await resetBreakers(app.baseUrl); + resetConnectionCooldowns(); + serverA.resetState(TOKEN_A, [buildError(500, "model A error")]); + + const result = await postChat(app.baseUrl, "sys-single-provider", "test modelA 500"); + + assert.equal(result.response.status, 200, JSON.stringify(result.json)); + assert.equal(result.json.model, "sys-a/test-model"); + assert.equal(serverA.getState(TOKEN_A).hits, 2); +}); + +test("single provider, modelA 503: combo fails over to modelB", async () => { + assert.ok(app); + await resetBreakers(app.baseUrl); + resetConnectionCooldowns(); + serverA.resetState(TOKEN_A, [buildError(503, "Service Unavailable")]); + + const result = await postChat(app.baseUrl, "sys-single-provider", "test modelA 503"); + + assert.equal(result.response.status, 200, JSON.stringify(result.json)); + assert.equal(result.json.model, "sys-a/test-model"); + assert.equal(serverA.getState(TOKEN_A).hits, 2); +}); + +test("single provider, modelA 429 with fobr: immediate failover to modelB", async () => { + assert.ok(app); + await resetBreakers(app.baseUrl); + resetConnectionCooldowns(); + serverA.resetState(TOKEN_A, [buildError(429, "rate limited")]); + + const result = await postChat(app.baseUrl, "sys-single-provider-fobr", "test fobr"); + + assert.equal(result.response.status, 200, JSON.stringify(result.json)); + assert.equal(result.json.model, "sys-a/test-model"); + assert.equal(serverA.getState(TOKEN_A).hits, 2); +}); + +test("single provider, modelA 500 with fobr: modelA retry", async () => { + assert.ok(app); + await resetBreakers(app.baseUrl); + resetConnectionCooldowns(); + serverA.resetState(TOKEN_A, [buildError(500, "Oops!")]); + + const result = await postChat(app.baseUrl, "sys-single-provider-fobr", "test fobr"); + + assert.equal(result.response.status, 200, JSON.stringify(result.json)); + assert.equal(result.json.model, "sys-a/test-model"); + assert.equal(serverA.getState(TOKEN_A).hits, 2); +}); + +test("single provider, both models fail: request returns 5xx to client", async () => { + assert.ok(app); + await resetBreakers(app.baseUrl); + resetConnectionCooldowns(); + serverA.resetState(TOKEN_A, [buildError(500, "modelA down"), buildError(500, "modelB down")]); + + const result = await postChat(app.baseUrl, "sys-single-provider", "both down"); + + assert.ok(result.response.status >= 500, `expected 5xx, got ${result.response.status}`); + assert.equal(serverA.getState(TOKEN_A).hits, 2); +}); diff --git a/tests/unit/combo-config.test.ts b/tests/unit/combo-config.test.ts index abaa924f23..269bc690a0 100644 --- a/tests/unit/combo-config.test.ts +++ b/tests/unit/combo-config.test.ts @@ -20,6 +20,9 @@ test("getDefaultComboConfig returns a fresh copy of the defaults", () => { assert.equal(first.handoffThreshold, 0.85); assert.equal(first.maxMessagesForSummary, 30); assert.deepEqual(first.handoffProviders, ["codex"]); + assert.equal(first.failoverBeforeRetry, false); + assert.equal(first.maxSetRetries, 0); + assert.equal(first.setRetryDelayMs, 2000); first.strategy = "weighted"; assert.equal(second.strategy, "priority"); @@ -272,3 +275,94 @@ test("updateComboDefaultsSchema rejects composite tiers in global defaults and p ] ); }); + +test("createComboSchema accepts failoverBeforeRetry, maxSetRetries and setRetryDelayMs", () => { + const parsed = createComboSchema.parse({ + name: "failover-test", + models: ["openai/gpt-4"], + strategy: "priority", + config: { + failoverBeforeRetry: true, + maxSetRetries: 3, + setRetryDelayMs: 1500, + }, + }); + + assert.equal(parsed.config.failoverBeforeRetry, true); + assert.equal(parsed.config.maxSetRetries, 3); + assert.equal(parsed.config.setRetryDelayMs, 1500); +}); + +test("createComboSchema coerces string numbers for maxSetRetries and setRetryDelayMs", () => { + const parsed = createComboSchema.parse({ + name: "coerce-test", + models: ["openai/gpt-4"], + strategy: "priority", + config: { + maxSetRetries: "2", + setRetryDelayMs: "500", + }, + }); + + assert.equal(parsed.config.maxSetRetries, 2); + assert.equal(parsed.config.setRetryDelayMs, 500); +}); + +test("createComboSchema rejects maxSetRetries out of range", () => { + const tooHigh = createComboSchema.safeParse({ + name: "bad-max", + models: ["openai/gpt-4"], + strategy: "priority", + config: { maxSetRetries: 11 }, + }); + assert.equal(tooHigh.success, false); + + const negative = createComboSchema.safeParse({ + name: "bad-max", + models: ["openai/gpt-4"], + strategy: "priority", + config: { maxSetRetries: -1 }, + }); + assert.equal(negative.success, false); +}); + +test("createComboSchema rejects setRetryDelayMs out of range", () => { + const tooHigh = createComboSchema.safeParse({ + name: "bad-delay", + models: ["openai/gpt-4"], + strategy: "priority", + config: { setRetryDelayMs: 60001 }, + }); + assert.equal(tooHigh.success, false); + + const negative = createComboSchema.safeParse({ + name: "bad-delay", + models: ["openai/gpt-4"], + strategy: "priority", + config: { setRetryDelayMs: -1 }, + }); + assert.equal(negative.success, false); +}); + +test("resolveComboConfig cascades failoverBeforeRetry, maxSetRetries and setRetryDelayMs", () => { + const result = resolveComboConfig( + { + config: { + failoverBeforeRetry: true, + maxSetRetries: 2, + setRetryDelayMs: 3000, + }, + }, + { + comboDefaults: { + failoverBeforeRetry: false, + maxSetRetries: 0, + setRetryDelayMs: 2000, + }, + } + ); + + assert.equal(result.failoverBeforeRetry, true); + assert.equal(result.maxSetRetries, 2); + assert.equal(result.setRetryDelayMs, 3000); +});