chore(merge): resolve combo.ts conflict — preserve exhaustedProviders (#1731) + falloverBeforeRetry (#2417)

docs/AGENTROUTER.md

@@ -0,0 +1,149 @@
+# AgentRouter Setup Guide
+
+[AgentRouter](https://agentrouter.org) is an Anthropic-compatible relay that resells
+Claude and other models, often at lower prices than the direct Anthropic API. It is
+designed as a drop-in `ANTHROPIC_BASE_URL` replacement for the official Claude Code
+client, so it only accepts traffic that matches the Claude Code wire image (specific
+User-Agent, `anthropic-beta` flags, Stainless SDK headers, etc.).
+
+OmniRoute supports AgentRouter through the **Claude Code compatible** provider type
+(`anthropic-compatible-cc-*`), which speaks the Anthropic Messages API with the
+correct wire image. A generic `openai-compatible-chat` provider pointing at
+`https://agentrouter.org` will **not** work — the upstream WAF rejects requests that
+do not look like Claude Code.
+
+---
+
+## Prerequisites
+
+- An AgentRouter account and API key. New signups get free credits via the affiliate
+  link in the project [README](../README.md).
+- OmniRoute running with the `ENABLE_CC_COMPATIBLE_PROVIDER` feature flag enabled
+  (see below).
+
+## 1. Enable the CC-compatible provider type
+
+The Claude Code compatible provider type is gated behind a feature flag because it
+sends traffic that closely mirrors the official Claude Code client. Enable it by
+setting an environment variable before starting OmniRoute:
+
+```bash
+ENABLE_CC_COMPATIBLE_PROVIDER=true
+```
+
+Docker example:
+
+```bash
+docker run -d --name omniroute \
+  --restart unless-stopped \
+  -p 20128:20128 \
+  -v omniroute-data:/app/data \
+  -e ENABLE_CC_COMPATIBLE_PROVIDER=true \
+  diegosouzapw/omniroute:latest
+```
+
+After restarting, the dashboard exposes an **Add Claude Code Compatible** option in
+addition to the existing OpenAI-compatible and Anthropic-compatible flows.
+
+## 2. Create the provider in the dashboard
+
+1. Open **Dashboard → Providers → Add Provider**.
+2. Choose **Add Claude Code Compatible** (only visible when the flag above is set).
+3. Fill in the fields:
+
+| Field     | Value                                                          |
+| --------- | -------------------------------------------------------------- |
+| Name      | `AgentRouter` (or any label)                                   |
+| Prefix    | `agentrouter` (friendly alias shown in logs and the dashboard) |
+| Base URL  | `https://agentrouter.org`                                      |
+| Chat path | `/v1/messages?beta=true` (default — leave as-is)               |
+
+> The canonical model identifier still uses the full provider node ID
+> (`anthropic-compatible-cc-{uuid}/{model}`). The **Prefix** is just a display
+> alias resolved by `src/lib/usage/callLogs.ts` for friendlier log output.
+
+4. (Optional) Paste your API key in the **Validate** field and click **Check** to
+   confirm connectivity before saving.
+5. Click **Add**.
+
+Once created, open the provider and add a **Connection** with your AgentRouter API
+key (`sk-...`). The connection's `test_status` should turn `active`.
+
+## 3. Use it through a combo or directly
+
+Reference the model using your provider's prefix as the namespace:
+
+```bash
+curl -X POST http://localhost:20128/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "agentrouter/claude-opus-4-6",
+    "messages": [{"role": "user", "content": "hello"}],
+    "max_tokens": 100
+  }'
+```
+
+The canonical model ID `anthropic-compatible-cc-{uuid}/claude-opus-4-6` also works
+and is what shows up in the database and combo configuration.
+
+Or add it to a combo for routing, fallback, and quota management like any other
+provider.
+
+---
+
+## Wire image details
+
+For reference, the cc-compatible bridge sends the following on each upstream
+request (see `open-sse/services/claudeCodeCompatible.ts`):
+
+| Header                                      | Value                                                                    |
+| ------------------------------------------- | ------------------------------------------------------------------------ |
+| `Authorization`                             | `Bearer <api-key>`                                                       |
+| `User-Agent`                                | `claude-cli/2.1.137 (external, sdk-cli)`                                 |
+| `anthropic-version`                         | `2023-06-01`                                                             |
+| `anthropic-beta`                            | `claude-code-20250219,interleaved-thinking-2025-05-14,effort-2025-11-24` |
+| `anthropic-dangerous-direct-browser-access` | `true`                                                                   |
+| `x-app`                                     | `cli`                                                                    |
+| `X-Stainless-*`                             | Various Stainless SDK headers (lang, package version, OS, arch, etc.)    |
+
+This is what allows requests to pass the upstream WAF / client whitelist.
+
+---
+
+## Troubleshooting
+
+**`{"error":{"message":"unauthorized client detected, ..."}}`** — Your request did
+not match the Claude Code wire image. This happens when the provider is configured
+as `openai-compatible-chat` instead of `anthropic-compatible-cc`, or when the
+`ENABLE_CC_COMPATIBLE_PROVIDER=true` flag was not set at startup.
+
+**`{"error":{"message":"无效的令牌","type":"new_api_error"}}` (HTTP 401)** —
+"Invalid token". The wire image is correct but the API key is rejected. Generate a
+new key in the AgentRouter dashboard and update the connection.
+
+**`{"error":{"code":"content-blocked","type":"agent_router_api_error"}}`
+(HTTP 400)** — AgentRouter's moderation hook rejected the request content, or the
+key's plan does not permit the requested model. Try a different prompt or model;
+contact AgentRouter support if a benign prompt is consistently blocked.
+
+**`[400]: content-blocked` only on specific models** — Most AgentRouter plans only
+allow a subset of models (e.g. `claude-opus-4-6`). Other model IDs return
+`unauthorized_client_error` even though the key is valid. Check which models your
+plan covers in the AgentRouter dashboard.
+
+**`Invalid JSON response from provider (reset after Ns)` from the omniroute logs** —
+The upstream returned a non-JSON body (typically an HTML error page from the WAF).
+This usually means the request never reached the AgentRouter backend — recheck that
+the provider ID starts with `anthropic-compatible-cc-` (note the trailing dash —
+see `CLAUDE_CODE_COMPATIBLE_PREFIX` in `open-sse/services/claudeCodeCompatible.ts`)
+and the feature flag is enabled.
+
+---
+
+## See also
+
+- [`docs/PROVIDERS.md`](./PROVIDERS.md) — Other provider integration notes
+- [`docs/reference/FREE_TIERS.md`](./reference/FREE_TIERS.md) — Free-tier provider
+  catalog
+- [`open-sse/services/claudeCodeCompatible.ts`](../open-sse/services/claudeCodeCompatible.ts)
+  — Wire image implementation

open-sse/executors/base.ts

@@ -105,6 +105,8 @@ export type ExecuteInput = {
   clientHeaders?: Record<string, string> | null;
   /** Callback to persist tokens that are proactively refreshed during execution. */
   onCredentialsRefreshed?: (newCredentials: ProviderCredentials) => Promise<void> | void;
+  /** When true, skip the intra-URL 429 retry in execute() so the caller handles fallback. */
+  skipUpstreamRetry?: boolean;
 };
 
 export type CountTokensInput = {
@@ -526,6 +528,7 @@ export class BaseExecutor {
     extendedContext,
     upstreamExtraHeaders,
     clientHeaders,
+    skipUpstreamRetry = false,
   }: ExecuteInput) {
     const fallbackCount = this.getFallbackCount();
     let lastError: unknown = null;
@@ -936,6 +939,7 @@ export class BaseExecutor {
 
         // Intra-URL retry: if 429 and we haven't exhausted per-URL retries, wait and retry the same URL
         if (
+          !skipUpstreamRetry &&
           response.status === HTTP_STATUS.RATE_LIMITED &&
           (retryAttemptsByUrl[urlIndex] ?? 0) < BaseExecutor.RETRY_CONFIG.maxAttempts
         ) {
@@ -955,7 +959,7 @@ export class BaseExecutor {
           log?.warn?.("AUTH", `401 on ${url} - API key may be invalid`);
         }
 
-        if (this.shouldRetry(response.status, urlIndex)) {
+        if (!skipUpstreamRetry && this.shouldRetry(response.status, urlIndex)) {
           log?.debug?.("RETRY", `${response.status} on ${url}, trying fallback ${urlIndex + 1}`);
           lastStatus = response.status;
           continue;
@@ -969,7 +973,7 @@ export class BaseExecutor {
           log?.warn?.("TIMEOUT", `Fetch timeout after ${this.getTimeoutMs()}ms on ${url}`);
         }
         lastError = err;
-        if (urlIndex + 1 < fallbackCount) {
+        if (!skipUpstreamRetry && urlIndex + 1 < fallbackCount) {
           log?.debug?.("RETRY", `Error on ${url}, trying fallback ${urlIndex + 1}`);
           continue;
         }

open-sse/handlers/chatCore.ts

@@ -1259,6 +1259,7 @@ export async function handleChatCore({
   comboExecutionKey = null,
   disableEmergencyFallback = false,
   cachedSettings = null,
+  skipUpstreamRetry = false,
 }) {
   let { provider, model, extendedContext } = modelInfo;
   // apiFormat is an optional custom-model marker injected by getModelInfo for
@@ -3170,6 +3171,7 @@ export async function handleChatCore({
                 upstreamExtraHeaders: buildUpstreamHeadersForExecute(modelToCall),
                 clientHeaders: buildExecutorClientHeaders(clientRawRequest?.headers, userAgent),
                 onCredentialsRefreshed,
+                skipUpstreamRetry,
               });
               trace("post_executor", { status: res?.response?.status });
 
@@ -3661,6 +3663,7 @@ export async function handleChatCore({
           upstreamExtraHeaders: buildUpstreamHeadersForExecute(retryModelId),
           clientHeaders: buildExecutorClientHeaders(clientRawRequest?.headers, userAgent),
           onCredentialsRefreshed,
+          skipUpstreamRetry: isCombo,
         });
 
         if (retryResult.response.ok) {

open-sse/services/accountFallback.ts

@@ -488,6 +488,14 @@ export function shouldMarkAccountExhaustedFrom429(
   );
 }
 
+/**
+ * Clear all in-memory model lockouts and failure state (for tests / full reset).
+ */
+export function clearAllModelLockouts(): void {
+  modelLockouts.clear();
+  modelFailureState.clear();
+}
+
 /**
  * Check if a specific model on a specific account is locked
  * @returns {boolean}

open-sse/services/combo.ts

@@ -1690,6 +1690,8 @@ export async function handleComboChat({
   const maxRetries = config.maxRetries ?? 1;
   const retryDelayMs = resolveDelayMs(config.retryDelayMs, 2000);
   const fallbackDelayMs = resolveDelayMs(config.fallbackDelayMs, 0);
+  const maxSetRetries = config.maxSetRetries ?? 0;
+  const setRetryDelayMs = resolveDelayMs(config.setRetryDelayMs, 2000);
 
   let orderedTargets =
     strategy === "weighted"
@@ -1993,98 +1995,254 @@ export async function handleComboChat({
   }
 
   // #1731: Per-request in-memory set of providers whose quota is fully exhausted.
-  // When a target returns a quota-exhausted 429 (subscription/credits/daily),
-  // remaining targets from the same provider are skipped to avoid the
-  // 2–5 minute cascade through N same-provider targets.
+  // When a target returns a quota-exhausted 429, remaining same-provider targets are skipped.
   const exhaustedProviders = new Set<string>();
-
-  let lastError = null;
-  let earliestRetryAfter = null;
-  let lastStatus = null;
-  const startTime = Date.now();
   let globalAttempts = 0;
-  let fallbackCount = 0;
-  let recordedAttempts = 0;
 
-  for (let i = 0; i < orderedTargets.length; i++) {
-    const target = orderedTargets[i];
-    const modelStr = target.modelStr;
-    const provider = target.provider;
-    const profile = await getRuntimeProviderProfile(provider);
+  for (let setTry = 0; setTry <= maxSetRetries; setTry++) {
+    if (setTry > 0) {
+      log.info("COMBO", `All targets failed — retrying set (${setTry}/${maxSetRetries})`);
+      await new Promise((resolve) => {
+        const timer = setTimeout(resolve, setRetryDelayMs);
+        signal?.addEventListener(
+          "abort",
+          () => {
+            clearTimeout(timer);
+            resolve(undefined);
+          },
+          { once: true }
+        );
+      });
+      if (signal?.aborted) {
+        log.info("COMBO", "Client disconnected during set retry delay — aborting");
+        return errorResponse(499, "Client disconnected");
+      }
+    }
 
-    // Pre-check: skip models where all accounts are in cooldown
-    if (isModelAvailable) {
-      const available = await isModelAvailable(modelStr, target);
-      if (!available) {
-        log.info("COMBO", `Skipping ${modelStr} (all accounts in cooldown)`);
+    let lastError = null;
+    let earliestRetryAfter = null;
+    let lastStatus = null;
+    const startTime = Date.now();
+    let fallbackCount = 0;
+    let recordedAttempts = 0;
+
+    for (let i = 0; i < orderedTargets.length; i++) {
+      const target = orderedTargets[i];
+      const modelStr = target.modelStr;
+      const provider = target.provider;
+      const profile = await getRuntimeProviderProfile(provider);
+
+      // #1731: Skip targets from a provider that already signaled full quota exhaustion this request.
+      if (provider && exhaustedProviders.has(provider)) {
+        log.info(
+          "COMBO",
+          `Skipping ${modelStr} — provider ${provider} marked exhausted this request (#1731)`
+        );
         if (i > 0) fallbackCount++;
         continue;
       }
-    }
 
-    // #1731: Skip targets from a provider that already signaled full quota exhaustion
-    // this request.
-    if (provider && exhaustedProviders.has(provider)) {
-      log.info(
-        "COMBO",
-        `Skipping ${modelStr} — provider ${provider} marked exhausted this request (#1731)`
-      );
-      if (i > 0) fallbackCount++;
-      continue;
-    }
-
-    // Retry loop for transient errors
-    for (let retry = 0; retry <= maxRetries; retry++) {
-      // Fix #1681: Bail out immediately if the client has disconnected
-      if (signal?.aborted) {
-        log.info("COMBO", `Client disconnected — aborting combo loop before model ${modelStr}`);
-        return errorResponse(499, "Client disconnected");
-      }
-      globalAttempts++;
-      if (globalAttempts > MAX_GLOBAL_ATTEMPTS) {
-        log.warn(
-          "COMBO",
-          `Maximum combo attempts (${MAX_GLOBAL_ATTEMPTS}) exceeded across all targets and fallbacks. Terminating loop to prevent runaway background requests.`
-        );
-        return errorResponse(503, "Maximum combo retry limit reached");
-      }
-      if (retry > 0) {
-        log.info(
-          "COMBO",
-          `Retrying ${modelStr} in ${retryDelayMs}ms (attempt ${retry + 1}/${maxRetries + 1})`
-        );
-        await new Promise((resolve) => {
-          const timer = setTimeout(resolve, retryDelayMs);
-          signal?.addEventListener(
-            "abort",
-            () => {
-              clearTimeout(timer);
-              resolve(undefined);
-            },
-            { once: true }
-          );
-        });
-        if (signal?.aborted) {
-          log.info("COMBO", `Client disconnected during retry delay — aborting`);
-          return errorResponse(499, "Client disconnected");
+      // Pre-check: skip models where all accounts are in cooldown
+      if (isModelAvailable) {
+        const available = await isModelAvailable(modelStr, target);
+        if (!available) {
+          log.info("COMBO", `Skipping ${modelStr} (all accounts in cooldown)`);
+          if (i > 0) fallbackCount++;
+          continue;
         }
       }
 
-      log.info(
-        "COMBO",
-        `Trying model ${i + 1}/${orderedTargets.length}: ${modelStr}${retry > 0 ? ` (retry ${retry})` : ""}`
-      );
-
-      const result = await handleSingleModelWrapped(body, modelStr, target);
-
-      // Success — validate response quality before returning
-      if (result.ok) {
-        const quality = await validateResponseQuality(result, clientRequestedStream, log);
-        if (!quality.valid) {
+      // Retry loop for transient errors
+      for (let retry = 0; retry <= maxRetries; retry++) {
+        // Fix #1681: Bail out immediately if the client has disconnected
+        if (signal?.aborted) {
+          log.info("COMBO", `Client disconnected — aborting combo loop before model ${modelStr}`);
+          return errorResponse(499, "Client disconnected");
+        }
+        globalAttempts++;
+        if (globalAttempts > MAX_GLOBAL_ATTEMPTS) {
           log.warn(
             "COMBO",
-            `Model ${modelStr} returned 200 but failed quality check: ${quality.reason}`
+            `Maximum combo attempts (${MAX_GLOBAL_ATTEMPTS}) exceeded across all targets and fallbacks. Terminating loop to prevent runaway background requests.`
           );
+          return errorResponse(503, "Maximum combo retry limit reached");
+        }
+        if (retry > 0) {
+          log.info(
+            "COMBO",
+            `Retrying ${modelStr} in ${retryDelayMs}ms (attempt ${retry + 1}/${maxRetries + 1})`
+          );
+          await new Promise((resolve) => {
+            const timer = setTimeout(resolve, retryDelayMs);
+            signal?.addEventListener(
+              "abort",
+              () => {
+                clearTimeout(timer);
+                resolve(undefined);
+              },
+              { once: true }
+            );
+          });
+          if (signal?.aborted) {
+            log.info("COMBO", `Client disconnected during retry delay — aborting`);
+            return errorResponse(499, "Client disconnected");
+          }
+        }
+
+        log.info(
+          "COMBO",
+          `Trying model ${i + 1}/${orderedTargets.length}: ${modelStr}${retry > 0 ? ` (retry ${retry})` : ""}`
+        );
+
+        const result = await handleSingleModelWrapped(body, modelStr, {
+          ...target,
+          failoverBeforeRetry: config.failoverBeforeRetry,
+        });
+
+        // Success — validate response quality before returning
+        if (result.ok) {
+          const quality = await validateResponseQuality(result, clientRequestedStream, log);
+          if (!quality.valid) {
+            log.warn(
+              "COMBO",
+              `Model ${modelStr} returned 200 but failed quality check: ${quality.reason}`
+            );
+            recordComboRequest(combo.name, modelStr, {
+              success: false,
+              latencyMs: Date.now() - startTime,
+              fallbackCount,
+              strategy,
+              target: toRecordedTarget(target),
+            });
+            recordedAttempts++;
+            // Fix #1707: Set terminal state so the fallback doesn't emit
+            // misleading ALL_ACCOUNTS_INACTIVE when the real issue is quality.
+            lastError = `Upstream response failed quality validation: ${quality.reason}`;
+            if (!lastStatus) lastStatus = 502;
+            if (i > 0) fallbackCount++;
+            break; // move to next model
+          }
+          const latencyMs = Date.now() - startTime;
+          log.info(
+            "COMBO",
+            `Model ${modelStr} succeeded (${latencyMs}ms, ${fallbackCount} fallbacks)`
+          );
+          recordComboRequest(combo.name, modelStr, {
+            success: true,
+            latencyMs,
+            fallbackCount,
+            strategy,
+            target: toRecordedTarget(target),
+          });
+          recordedAttempts++;
+
+          // Context-relay intentionally splits responsibilities:
+          // combo.ts decides whether a successful turn should generate a handoff,
+          // while chat.ts injects the handoff after the real connectionId is resolved.
+          if (
+            strategy === "context-relay" &&
+            relayOptions?.sessionId &&
+            relayConfig &&
+            relayConfig.handoffProviders.includes(provider) &&
+            provider === "codex"
+          ) {
+            const connectionId = getSessionConnection(relayOptions.sessionId);
+            if (connectionId) {
+              const quotaInfo = await fetchCodexQuota(connectionId).catch(() => null);
+              if (quotaInfo) {
+                const resetCandidates = [quotaInfo.window5h?.resetAt, quotaInfo.window7d?.resetAt]
+                  .filter((value): value is string => typeof value === "string" && value.length > 0)
+                  .sort((a, b) => a.localeCompare(b));
+                const handoffSourceMessages =
+                  Array.isArray(body?.messages) && body.messages.length > 0
+                    ? body.messages
+                    : Array.isArray(body?.input)
+                      ? body.input
+                      : [];
+
+                maybeGenerateHandoff({
+                  sessionId: relayOptions.sessionId,
+                  comboName: combo.name,
+                  connectionId,
+                  percentUsed: quotaInfo.percentUsed,
+                  messages: handoffSourceMessages,
+                  model: modelStr,
+                  expiresAt: resetCandidates[0] || null,
+                  config: relayConfig,
+                  handleSingleModel: handleSingleModelWrapped,
+                });
+              }
+            }
+          }
+
+          // Record last known good provider (LKGP) for this combo/model (#919)
+          if (provider) {
+            const connId = target.connectionId || undefined;
+            void (async () => {
+              try {
+                const { setLKGP } = await import("../../src/lib/localDb");
+                await Promise.all([
+                  setLKGP(combo.name, target.executionKey, provider, connId),
+                  setLKGP(combo.name, combo.id || combo.name, provider, connId),
+                ]);
+              } catch (err) {
+                log.warn("COMBO", "Failed to record Last Known Good Provider. This is non-fatal.", {
+                  err,
+                });
+              }
+            })();
+          }
+
+          return quality.clonedResponse ?? result;
+        }
+
+        // Extract error info from response
+        let errorText = result.statusText || "";
+        let errorBody = null;
+        let retryAfter = null;
+        try {
+          const cloned = result.clone();
+          try {
+            const text = await cloned.text();
+            if (text) {
+              errorText = text.substring(0, 500);
+              errorBody = JSON.parse(text);
+              errorText =
+                errorBody?.error?.message || errorBody?.error || errorBody?.message || errorText;
+              retryAfter = errorBody?.retryAfter || null;
+            }
+          } catch {
+            /* Clone parse failed */
+          }
+        } catch {
+          /* Clone failed */
+        }
+
+        // Track earliest retryAfter
+        if (
+          retryAfter &&
+          (!earliestRetryAfter || new Date(retryAfter) < new Date(earliestRetryAfter))
+        ) {
+          earliestRetryAfter = retryAfter;
+        }
+
+        // Normalize error text
+        if (typeof errorText !== "string") {
+          try {
+            errorText = JSON.stringify(errorText);
+          } catch {
+            errorText = String(errorText);
+          }
+        }
+
+        const isStreamReadinessFailure =
+          (result.status === 502 || result.status === 504) &&
+          isStreamReadinessFailureErrorBody(errorBody);
+
+        // Fix #1681: Status 499 means client disconnected — stop combo loop immediately.
+        // There is no point trying fallback models when nobody is listening.
+        if (result.status === 499) {
+          log.info("COMBO", `Client disconnected (499) during ${modelStr} — stopping combo loop`);
           recordComboRequest(combo.name, modelStr, {
             success: false,
             latencyMs: Date.now() - startTime,
@@ -2093,134 +2251,56 @@ export async function handleComboChat({
             target: toRecordedTarget(target),
           });
           recordedAttempts++;
-          // Fix #1707: Set terminal state so the fallback doesn't emit
-          // misleading ALL_ACCOUNTS_INACTIVE when the real issue is quality.
-          lastError = `Upstream response failed quality validation: ${quality.reason}`;
-          if (!lastStatus) lastStatus = 502;
-          if (i > 0) fallbackCount++;
-          break; // move to next model
+          return result;
         }
-        const latencyMs = Date.now() - startTime;
-        log.info(
-          "COMBO",
-          `Model ${modelStr} succeeded (${latencyMs}ms, ${fallbackCount} fallbacks)`
+
+        // Combo fallback is target-level orchestration: a non-ok target response is
+        // treated as local to that target and the combo continues to the next target.
+        // Error classification is retained only for retry/cooldown pacing; it must
+        // not decide whether fallback happens, including for generic 400 responses.
+        const fallbackResult = checkFallbackError(
+          result.status,
+          errorText,
+          0,
+          null,
+          provider,
+          result.headers,
+          profile
         );
-        recordComboRequest(combo.name, modelStr, {
-          success: true,
-          latencyMs,
-          fallbackCount,
-          strategy,
-          target: toRecordedTarget(target),
-        });
-        recordedAttempts++;
+        const { cooldownMs } = fallbackResult;
 
-        // Context-relay intentionally splits responsibilities:
-        // combo.ts decides whether a successful turn should generate a handoff,
-        // while chat.ts injects the handoff after the real connectionId is resolved.
+        // #1731: If the entire provider quota is exhausted, mark it so subsequent
+        // same-provider targets are skipped immediately.
+        if (provider && isProviderExhaustedReason(fallbackResult)) {
+          exhaustedProviders.add(provider);
+          log.info(
+            "COMBO",
+            `Provider ${provider} quota exhausted — marking for skip on remaining targets (#1731)`
+          );
+        }
+
+        // Trigger shared provider circuit breaker for 5xx errors and connection failures.
+        // If the next target in the combo is on the same provider, don't mark the provider
+        // as failed — different models on the same provider may still succeed.
+        const nextTarget = orderedTargets[i + 1];
+        const sameProviderNext =
+          typeof nextTarget?.provider === "string" && nextTarget.provider === provider;
         if (
-          strategy === "context-relay" &&
-          relayOptions?.sessionId &&
-          relayConfig &&
-          relayConfig.handoffProviders.includes(provider) &&
-          provider === "codex"
+          !isStreamReadinessFailure &&
+          isProviderFailureCode(result.status) &&
+          !sameProviderNext
         ) {
-          const connectionId = getSessionConnection(relayOptions.sessionId);
-          if (connectionId) {
-            const quotaInfo = await fetchCodexQuota(connectionId).catch(() => null);
-            if (quotaInfo) {
-              const resetCandidates = [quotaInfo.window5h?.resetAt, quotaInfo.window7d?.resetAt]
-                .filter((value): value is string => typeof value === "string" && value.length > 0)
-                .sort((a, b) => a.localeCompare(b));
-              const handoffSourceMessages =
-                Array.isArray(body?.messages) && body.messages.length > 0
-                  ? body.messages
-                  : Array.isArray(body?.input)
-                    ? body.input
-                    : [];
-
-              maybeGenerateHandoff({
-                sessionId: relayOptions.sessionId,
-                comboName: combo.name,
-                connectionId,
-                percentUsed: quotaInfo.percentUsed,
-                messages: handoffSourceMessages,
-                model: modelStr,
-                expiresAt: resetCandidates[0] || null,
-                config: relayConfig,
-                handleSingleModel: handleSingleModelWrapped,
-              });
-            }
-          }
+          recordProviderFailure(provider, log, target.connectionId, profile);
         }
 
-        // Record last known good provider (LKGP) for this combo/model (#919)
-        if (provider) {
-          const connId = target.connectionId || undefined;
-          void (async () => {
-            try {
-              const { setLKGP } = await import("../../src/lib/localDb");
-              await Promise.all([
-                setLKGP(combo.name, target.executionKey, provider, connId),
-                setLKGP(combo.name, combo.id || combo.name, provider, connId),
-              ]);
-            } catch (err) {
-              log.warn("COMBO", "Failed to record Last Known Good Provider. This is non-fatal.", {
-                err,
-              });
-            }
-          })();
+        // Check if this is a transient error worth retrying on same model
+        const isTransient =
+          !isStreamReadinessFailure && [408, 429, 500, 502, 503, 504].includes(result.status);
+        if (retry < maxRetries && isTransient) {
+          continue; // Retry same model
         }
 
-        return quality.clonedResponse ?? result;
-      }
-
-      // Extract error info from response
-      let errorText = result.statusText || "";
-      let errorBody = null;
-      let retryAfter = null;
-      try {
-        const cloned = result.clone();
-        try {
-          const text = await cloned.text();
-          if (text) {
-            errorText = text.substring(0, 500);
-            errorBody = JSON.parse(text);
-            errorText =
-              errorBody?.error?.message || errorBody?.error || errorBody?.message || errorText;
-            retryAfter = errorBody?.retryAfter || null;
-          }
-        } catch {
-          /* Clone parse failed */
-        }
-      } catch {
-        /* Clone failed */
-      }
-
-      // Track earliest retryAfter
-      if (
-        retryAfter &&
-        (!earliestRetryAfter || new Date(retryAfter) < new Date(earliestRetryAfter))
-      ) {
-        earliestRetryAfter = retryAfter;
-      }
-
-      // Normalize error text
-      if (typeof errorText !== "string") {
-        try {
-          errorText = JSON.stringify(errorText);
-        } catch {
-          errorText = String(errorText);
-        }
-      }
-
-      const isStreamReadinessFailure =
-        (result.status === 502 || result.status === 504) &&
-        isStreamReadinessFailureErrorBody(errorBody);
-
-      // Fix #1681: Status 499 means client disconnected — stop combo loop immediately.
-      // There is no point trying fallback models when nobody is listening.
-      if (result.status === 499) {
-        log.info("COMBO", `Client disconnected (499) during ${modelStr} — stopping combo loop`);
+        // Done retrying this model
         recordComboRequest(combo.name, modelStr, {
           success: false,
           latencyMs: Date.now() - startTime,
@@ -2229,120 +2309,76 @@ export async function handleComboChat({
           target: toRecordedTarget(target),
         });
         recordedAttempts++;
-        return result;
-      }
+        lastError = errorText || String(result.status);
+        if (!lastStatus) lastStatus = result.status;
+        if (i > 0) fallbackCount++;
+        log.warn("COMBO", `Model ${modelStr} failed, trying next`, { status: result.status });
 
-      // Combo fallback is target-level orchestration: a non-ok target response is
-      // treated as local to that target and the combo continues to the next target.
-      // Error classification is retained only for retry/cooldown pacing; it must
-      // not decide whether fallback happens, including for generic 400 responses.
-      const fallbackResult = checkFallbackError(
-        result.status,
-        errorText,
-        0,
-        null,
-        provider,
-        result.headers,
-        profile
-      );
-      const { cooldownMs } = fallbackResult;
-
-      // #1731: If the entire provider quota is exhausted, mark it so subsequent
-      // same-provider targets are skipped immediately.
-      if (provider && isProviderExhaustedReason(fallbackResult)) {
-        exhaustedProviders.add(provider);
-        log.info(
-          "COMBO",
-          `Provider ${provider} quota exhausted — marking for skip on remaining targets (#1731)`
-        );
-      }
-
-      // Trigger shared provider circuit breaker for 5xx errors and connection failures
-      if (!isStreamReadinessFailure && isProviderFailureCode(result.status)) {
-        recordProviderFailure(provider, log, target.connectionId, profile);
-      }
-
-      // Check if this is a transient error worth retrying on same model
-      const isTransient =
-        !isStreamReadinessFailure && [408, 429, 500, 502, 503, 504].includes(result.status);
-      if (retry < maxRetries && isTransient) {
-        continue; // Retry same model
-      }
-
-      // Done retrying this model
-      recordComboRequest(combo.name, modelStr, {
-        success: false,
-        latencyMs: Date.now() - startTime,
-        fallbackCount,
-        strategy,
-        target: toRecordedTarget(target),
-      });
-      recordedAttempts++;
-      lastError = errorText || String(result.status);
-      if (!lastStatus) lastStatus = result.status;
-      if (i > 0) fallbackCount++;
-      log.warn("COMBO", `Model ${modelStr} failed, trying next`, { status: result.status });
-
-      const fallbackWaitMs =
-        fallbackDelayMs > 0 && cooldownMs > 0 && cooldownMs <= MAX_FALLBACK_WAIT_MS
-          ? Math.min(cooldownMs, fallbackDelayMs)
-          : 0;
-      if ([502, 503, 504].includes(result.status) && fallbackWaitMs > 0) {
-        log.info("COMBO", `Waiting ${fallbackWaitMs}ms before fallback to next model`);
-        await new Promise((resolve) => {
-          const timer = setTimeout(resolve, fallbackWaitMs);
-          signal?.addEventListener(
-            "abort",
-            () => {
-              clearTimeout(timer);
-              resolve(undefined);
-            },
-            { once: true }
-          );
-        });
-        if (signal?.aborted) {
-          log.info("COMBO", `Client disconnected during fallback wait — aborting`);
-          return errorResponse(499, "Client disconnected");
+        const fallbackWaitMs =
+          fallbackDelayMs > 0 && cooldownMs > 0 && cooldownMs <= MAX_FALLBACK_WAIT_MS
+            ? Math.min(cooldownMs, fallbackDelayMs)
+            : 0;
+        if ([502, 503, 504].includes(result.status) && fallbackWaitMs > 0) {
+          log.info("COMBO", `Waiting ${fallbackWaitMs}ms before fallback to next model`);
+          await new Promise((resolve) => {
+            const timer = setTimeout(resolve, fallbackWaitMs);
+            signal?.addEventListener(
+              "abort",
+              () => {
+                clearTimeout(timer);
+                resolve(undefined);
+              },
+              { once: true }
+            );
+          });
+          if (signal?.aborted) {
+            log.info("COMBO", `Client disconnected during fallback wait — aborting`);
+            return errorResponse(499, "Client disconnected");
+          }
         }
+
+        break; // Move to next model
       }
-
-      break; // Move to next model
     }
+
+    // All models failed in this set try
+    const latencyMs = Date.now() - startTime;
+    if (recordedAttempts === 0) {
+      recordComboRequest(combo.name, null, { success: false, latencyMs, fallbackCount, strategy });
+    }
+
+    // Retry the entire set if more attempts remain
+    if (setTry < maxSetRetries) continue;
+
+    // All set retries exhausted — return the final error
+    if (!lastStatus) {
+      return new Response(
+        JSON.stringify({
+          error: {
+            message: "Service temporarily unavailable: all upstream accounts are inactive",
+            type: "service_unavailable",
+            code: "ALL_ACCOUNTS_INACTIVE",
+          },
+        }),
+        { status: 503, headers: { "Content-Type": "application/json" } }
+      );
+    }
+
+    const status = lastStatus;
+    const msg = lastError || "All combo models unavailable";
+
+    if (earliestRetryAfter) {
+      const retryHuman = formatRetryAfter(earliestRetryAfter);
+      log.warn("COMBO", `All models failed | ${msg} (${retryHuman})`);
+      return unavailableResponse(status, msg, earliestRetryAfter, retryHuman);
+    }
+
+    log.warn("COMBO", `All models failed | ${msg}`);
+    return new Response(JSON.stringify({ error: { message: msg } }), {
+      status,
+      headers: { "Content-Type": "application/json" },
+    });
   }
-
-  // All models failed
-  const latencyMs = Date.now() - startTime;
-  if (recordedAttempts === 0) {
-    recordComboRequest(combo.name, null, { success: false, latencyMs, fallbackCount, strategy });
-  }
-
-  if (!lastStatus) {
-    return new Response(
-      JSON.stringify({
-        error: {
-          message: "Service temporarily unavailable: all upstream accounts are inactive",
-          type: "service_unavailable",
-          code: "ALL_ACCOUNTS_INACTIVE",
-        },
-      }),
-      { status: 503, headers: { "Content-Type": "application/json" } }
-    );
-  }
-
-  const status = lastStatus;
-  const msg = lastError || "All combo models unavailable";
-
-  if (earliestRetryAfter) {
-    const retryHuman = formatRetryAfter(earliestRetryAfter);
-    log.warn("COMBO", `All models failed | ${msg} (${retryHuman})`);
-    return unavailableResponse(status, msg, earliestRetryAfter, retryHuman);
-  }
-
-  log.warn("COMBO", `All models failed | ${msg}`);
-  return new Response(JSON.stringify({ error: { message: msg } }), {
-    status,
-    headers: { "Content-Type": "application/json" },
-  });
 }
 
 /**
@@ -2474,7 +2510,10 @@ async function handleRoundRobinCombo({
           `[RR #${counter}] → ${modelStr}${offset > 0 ? ` (fallback +${offset})` : ""}${retry > 0 ? ` (retry ${retry})` : ""}`
         );
 
-        const result = await handleSingleModel(body, modelStr, target);
+        const result = await handleSingleModel(body, modelStr, {
+          ...target,
+          failoverBeforeRetry: config.failoverBeforeRetry,
+        });
 
         // Success — validate response quality before returning
         if (result.ok) {

open-sse/services/comboConfig.ts

@@ -23,6 +23,9 @@ const DEFAULT_COMBO_CONFIG = {
   resetAwareWeeklyWeight: 0.65,
   resetAwareTieBandPercent: 5,
   resetAwareExhaustionGuardPercent: 10,
+  failoverBeforeRetry: false,
+  maxSetRetries: 0,
+  setRetryDelayMs: 2000,
 };
 
 const LEGACY_COMBO_RESILIENCE_KEYS = new Set([

package.json

@@ -113,6 +113,7 @@
     "test:protocols:e2e": "node scripts/dev/run-protocol-clients-tests.mjs",
     "test:vitest": "vitest run --config vitest.mcp.config.ts",
     "test:ecosystem": "node scripts/dev/run-ecosystem-tests.mjs",
+    "test:system": "cross-env DISABLE_SQLITE_AUTO_BACKUP=true node --import tsx --test --test-force-exit --test-concurrency=1 tests/e2e/system-failover.test.ts",
     "test:coverage": "cross-env DISABLE_SQLITE_AUTO_BACKUP=true c8 --output-dir coverage --exclude=tests/** --exclude=**/*.test.* --reporter=text-summary --reporter=html --reporter=json-summary --reporter=lcov --check-coverage --statements 75 --lines 75 --functions 75 --branches 70 node --import tsx --test --test-concurrency=1 tests/unit/*.test.ts",
     "test:coverage:legacy": "c8 --output-dir coverage --exclude=open-sse --check-coverage --lines 50 --functions 50 --branches 50 node --import tsx --test tests/unit/*.test.ts",
     "coverage:report": "c8 report --output-dir coverage --exclude=tests/** --exclude=**/*.test.* --reporter=text --reporter=text-summary --reporter=html --reporter=json-summary --reporter=lcov",

src/app/(dashboard)/dashboard/cli-tools/CLIToolsPageClient.tsx

@@ -180,8 +180,9 @@ export default function CLIToolsPageClient({ machineId: _machineId }) {
       activeProviders.map((c) => PROVIDER_ID_TO_ALIAS[c.provider] || c.provider)
     );
     dynamicModels.forEach((dm) => {
-      const modelId = dm.id || dm;
-      if (seenModels.has(modelId)) return;
+      const rawId = dm?.id ?? dm;
+      const modelId = typeof rawId === "string" ? rawId : "";
+      if (!modelId || seenModels.has(modelId)) return;
       // Parse alias/model format
       const slashIdx = modelId.indexOf("/");
       if (slashIdx === -1) return;

src/app/(dashboard)/dashboard/combos/page.tsx

@@ -156,6 +156,12 @@ const ADVANCED_FIELD_HELP_FALLBACK = {
     "Round-robin combo/model limit: max simultaneous requests sent to each model target. This is separate from any provider account-only cap.",
   queueTimeout:
     "How long a request can wait for a round-robin model slot before timing out. This queue is separate from any account-only concurrency cap.",
+  failoverBeforeRetry:
+    "When enabled, a 429 from the upstream triggers immediate target failover instead of retrying the same URL first.",
+  maxSetRetries:
+    "Number of times to retry the full target set when every target fails. 0 = no set-level retry.",
+  setRetryDelayMs:
+    "Delay between set-level retry attempts, giving transient issues time to resolve.",
 };
 
 const LEGACY_COMBO_RESILIENCE_KEYS = new Set([
@@ -1427,7 +1433,7 @@ function FieldLabelWithHelp({ label, help, showHelp = true }) {
     <div className="flex items-center gap-1 mb-0.5">
       <label className="text-[10px] text-text-muted">{label}</label>
       {showHelp && (
-        <Tooltip content={help}>
+        <Tooltip position="bottom" content={help}>
           <span className="material-symbols-outlined text-[12px] text-text-muted cursor-help">
             help
           </span>
@@ -3540,6 +3546,94 @@ function ComboFormModal({ isOpen, combo, onClose, onSave, activeProviders, combo
                       />
                     </div>
                   </div>
+                  {/* failoverBeforeRetry + maxSetRetries + setRetryDelayMs */}
+                  <div className="grid grid-cols-2 gap-2 pt-2 border-t border-black/5 dark:border-white/5">
+                    <div className="col-span-2">
+                      <div className="flex items-center gap-2 py-1">
+                        <input
+                          type="checkbox"
+                          id="failoverBeforeRetry"
+                          checked={!!config.failoverBeforeRetry}
+                          onChange={(e) =>
+                            setConfig({
+                              ...config,
+                              failoverBeforeRetry: e.target.checked || undefined,
+                            })
+                          }
+                          className="w-3.5 h-3.5 rounded border border-black/20 dark:border-white/20 accent-primary cursor-pointer"
+                        />
+                        <label
+                          htmlFor="failoverBeforeRetry"
+                          className="text-xs text-text-muted cursor-pointer select-none"
+                        >
+                          {t("failoverBeforeRetry")}
+                        </label>
+                        <Tooltip
+                          position="bottom"
+                          content={getI18nOrFallback(
+                            t,
+                            "advancedHelp.failoverBeforeRetry",
+                            ADVANCED_FIELD_HELP_FALLBACK.failoverBeforeRetry
+                          )}
+                        >
+                          <span className="material-symbols-outlined text-[12px] text-text-muted cursor-help">
+                            help
+                          </span>
+                        </Tooltip>
+                      </div>
+                    </div>
+                    <div>
+                      <FieldLabelWithHelp
+                        label={t("maxSetRetries")}
+                        help={getI18nOrFallback(
+                          t,
+                          "advancedHelp.maxSetRetries",
+                          ADVANCED_FIELD_HELP_FALLBACK.maxSetRetries
+                        )}
+                        showHelp={!isExpertMode}
+                      />
+                      <input
+                        type="number"
+                        min="0"
+                        max="10"
+                        value={config.maxSetRetries ?? ""}
+                        placeholder="0"
+                        onChange={(e) =>
+                          setConfig({
+                            ...config,
+                            maxSetRetries: e.target.value ? Number(e.target.value) : undefined,
+                          })
+                        }
+                        className="w-full text-xs py-1.5 px-2 rounded border border-black/10 dark:border-white/10 bg-transparent focus:border-primary focus:outline-none"
+                      />
+                    </div>
+                    <div>
+                      <FieldLabelWithHelp
+                        label={t("setRetryDelayMs")}
+                        help={getI18nOrFallback(
+                          t,
+                          "advancedHelp.setRetryDelayMs",
+                          ADVANCED_FIELD_HELP_FALLBACK.setRetryDelayMs
+                        )}
+                        showHelp={!isExpertMode}
+                      />
+                      <input
+                        type="number"
+                        min="0"
+                        max="60000"
+                        step="500"
+                        value={config.setRetryDelayMs ?? ""}
+                        placeholder="2000"
+                        onChange={(e) =>
+                          setConfig({
+                            ...config,
+                            setRetryDelayMs: e.target.value ? Number(e.target.value) : undefined,
+                          })
+                        }
+                        className="w-full text-xs py-1.5 px-2 rounded border border-black/10 dark:border-white/10 bg-transparent focus:border-primary focus:outline-none"
+                      />
+                    </div>
+                  </div>
                   {strategy === "round-robin" && (
                     <div className="grid grid-cols-2 gap-2 pt-2 border-t border-black/5 dark:border-white/5">
                       <div>

src/app/api/resilience/reset/route.ts

@@ -1,7 +1,7 @@
 import { NextResponse } from "next/server";
 
 /**
- * POST /api/resilience/reset — Reset all provider circuit breakers
+ * POST /api/resilience/reset — Reset all provider circuit breakers and model lockouts
  */
 export async function POST() {
   try {
@@ -17,10 +17,15 @@ export async function POST() {
       resetCount++;
     }
 
+    // Also clear in-memory model lockouts (per-model quota cooldowns)
+    const { clearAllModelLockouts } =
+      await import("@omniroute/open-sse/services/accountFallback.ts");
+    clearAllModelLockouts();
+
     return NextResponse.json({
       ok: true,
       resetCount,
-      message: `Reset ${resetCount} circuit breaker(s)`,
+      message: `Reset ${resetCount} circuit breaker(s) and model lockouts`,
     });
   } catch (err: unknown) {
     const message = err instanceof Error ? err.message : "Failed to reset resilience state";

src/app/offline/page.tsx

@@ -1,25 +1,24 @@
 "use client";
 
-import { useEffect, useState } from "react";
+import { useSyncExternalStore } from "react";
 import Link from "next/link";
 
+function subscribeToOnline(callback: () => void) {
+  window.addEventListener("online", callback);
+  window.addEventListener("offline", callback);
+  return () => {
+    window.removeEventListener("online", callback);
+    window.removeEventListener("offline", callback);
+  };
+}
+
 export default function OfflinePage() {
-  const [isOnline, setIsOnline] = useState<boolean>(() =>
-    typeof navigator !== "undefined" ? navigator.onLine : true
+  const isOnline = useSyncExternalStore(
+    subscribeToOnline,
+    () => navigator.onLine,
+    () => false
   );
 
-  useEffect(() => {
-    const onOnline = () => setIsOnline(true);
-    const onOffline = () => setIsOnline(false);
-
-    window.addEventListener("online", onOnline);
-    window.addEventListener("offline", onOffline);
-    return () => {
-      window.removeEventListener("online", onOnline);
-      window.removeEventListener("offline", onOffline);
-    };
-  }, []);
-
   return (
     <main className="min-h-screen bg-bg text-text-main flex items-center justify-center p-6">
       <section className="w-full max-w-xl rounded-2xl border border-border bg-surface p-8 shadow-soft text-center">

src/i18n/messages/en.json

@@ -1721,6 +1721,9 @@
     "contextRelaySummaryModelHelp": "Optional override model used only for generating the handoff summary. Leave empty to reuse the active combo model.",
     "contextRelayProviderNote": "Context Relay currently generates handoffs for Codex account rotation. Pair it with multiple accounts of the same provider for the best continuity.",
     "advancedHint": "Leave empty to use global defaults. These override per-provider settings.",
+    "failoverBeforeRetry": "Failover Before Retry",
+    "maxSetRetries": "Max Set Retries",
+    "setRetryDelayMs": "Set Retry Delay (ms)",
     "moveUp": "Move up",
     "moveDown": "Move down",
     "removeModel": "Remove",
@@ -1810,7 +1813,10 @@
       "timeout": "Maximum request duration before aborting.",
       "healthcheck": "Skips unhealthy models/providers from routing decisions.",
       "concurrencyPerModel": "Max simultaneous requests allowed per model in round-robin.",
-      "queueTimeout": "How long a request can wait in queue before timing out."
+      "queueTimeout": "How long a request can wait in queue before timing out.",
+      "failoverBeforeRetry": "When enabled, any upstream error triggers immediate failover to the next combo target, skipping all retries and fallback URLs.",
+      "maxSetRetries": "Number of times to retry the full target set when every target fails. 0 = no set-level retry.",
+      "setRetryDelayMs": "Delay between set-level retry attempts, giving transient issues time to resolve."
     },
     "templatesTitle": "Quick templates",
     "templatesDescription": "Apply a starting profile, then adjust models and config.",

src/shared/validation/schemas.ts

@@ -517,6 +517,9 @@ const comboRuntimeConfigSchema = z
     maxComboDepth: z.coerce.number().int().min(1).max(10).optional(),
     trackMetrics: z.boolean().optional(),
     compressionMode: compressionModeSchema.optional(),
+    failoverBeforeRetry: z.boolean().optional(),
+    maxSetRetries: z.coerce.number().int().min(0).max(10).optional(),
+    setRetryDelayMs: z.coerce.number().int().min(0).max(60000).optional(),
     // Auto-Combo / LKGP Extensions
     candidatePool: z.array(z.string().min(1)).optional(),
     weights: scoringWeightsSchema.optional(),

src/sse/handlers/chat.ts

@@ -463,6 +463,7 @@ export async function handleChat(request: any, clientRawRequest: any = null) {
           executionKey?: string | null;
           stepId?: string | null;
           allowedConnectionIds?: string[] | null;
+          failoverBeforeRetry?: boolean;
         }
       ) =>
         handleSingleModelChat(
@@ -481,6 +482,7 @@ export async function handleChat(request: any, clientRawRequest: any = null) {
             allowedConnectionIds: target?.allowedConnectionIds ?? null,
             comboStepId: target?.stepId || null,
             comboExecutionKey: target?.executionKey || target?.stepId || null,
+            skipUpstreamRetry: target?.failoverBeforeRetry ?? false,
             preselectedCredentials: comboPreselectedCredentials.get(
               getComboCredentialCacheKey(m, target)
             ),
@@ -611,6 +613,7 @@ async function handleSingleModelChat(
     allowedConnectionIds?: string[] | null;
     comboStepId?: string | null;
     comboExecutionKey?: string | null;
+    skipUpstreamRetry?: boolean;
     preselectedCredentials?: any;
     cachedSettings?: any;
   } = {},
@@ -643,6 +646,7 @@ async function handleSingleModelChat(
           connectionId?: string | null;
           executionKey?: string | null;
           stepId?: string | null;
+          failoverBeforeRetry?: boolean;
         }
       ) =>
         handleSingleModelChat(
@@ -660,6 +664,7 @@ async function handleSingleModelChat(
             allowedConnectionIds: null,
             comboStepId: null,
             comboExecutionKey: null,
+            skipUpstreamRetry: target?.failoverBeforeRetry ?? false,
           },
           redirectCombo.strategy ?? "priority",
           false
@@ -916,6 +921,7 @@ async function handleSingleModelChat(
         modelApiFormat: apiFormat,
         providerProfile,
         cachedSettings: runtimeOptions.cachedSettings,
+        skipUpstreamRetry: runtimeOptions.skipUpstreamRetry ?? false,
       });
       if (telemetry) telemetry.endPhase();
 
@@ -1129,7 +1135,11 @@ async function handleSingleModelChat(
         continue;
       }
 
-      if (!forceLiveComboTest && PROVIDER_BREAKER_FAILURE_STATUSES.has(Number(result.status))) {
+      if (
+        !forceLiveComboTest &&
+        !isCombo &&
+        PROVIDER_BREAKER_FAILURE_STATUSES.has(Number(result.status))
+      ) {
         breaker._onFailure();
       }
 

src/sse/handlers/chatHelpers.ts

@@ -303,6 +303,7 @@ export async function executeChatWithBreaker({
   modelApiFormat,
   providerProfile,
   cachedSettings,
+  skipUpstreamRetry = false,
 }: any): Promise<{ result: any; tlsFingerprintUsed: boolean }> {
   let tlsFingerprintUsed = false;
 
@@ -324,6 +325,7 @@ export async function executeChatWithBreaker({
           comboStepId,
           comboExecutionKey,
           cachedSettings,
+          skipUpstreamRetry,
           onCredentialsRefreshed: async (newCreds: any) => {
             await updateProviderCredentials(credentials.connectionId, {
               accessToken: newCreds.accessToken,

src/sse/services/auth.ts

@@ -1567,8 +1567,14 @@ export async function markAccountUnavailable(
       | undefined;
 
     const isPerModelQuotaProvider = hasPerModelQuota(provider, model, connectionPassthroughModels);
-    if (isPerModelQuotaProvider && provider && model && (status === 404 || status === 429)) {
-      const reason = status === 404 ? "not_found" : "rate_limited";
+    if (
+      isPerModelQuotaProvider &&
+      provider &&
+      model &&
+      (status === 404 || status === 429 || status >= 500)
+    ) {
+      const reason =
+        status === 404 ? "not_found" : status === 429 ? "rate_limited" : "server_error";
       const lockout = recordModelLockoutFailure(
         provider,
         connectionId,

tests/e2e/combos-flow.spec.ts

@@ -456,6 +456,13 @@ test.describe("Combos flow", () => {
     await comboDialog.locator('[data-testid="combo-manual-model-add"]').click();
     await expect(comboDialog.locator('[data-testid="combo-readiness-panel"]')).toHaveCount(0);
 
+    // New advanced settings: failoverBeforeRetry, maxSetRetries, setRetryDelayMs
+    await comboDialog.locator("#failoverBeforeRetry").check();
+    // maxSetRetries: placeholder "0" is unique (maxRetries uses "1")
+    await comboDialog.locator('input[placeholder="0"]').fill("2");
+    // setRetryDelayMs: placeholder "2000" — last occurrence is the new field
+    await comboDialog.locator('input[placeholder="2000"]').last().fill("1500");
+
     await comboDialog
       .getByRole("button", { name: /create combo|criar combo/i })
       .last()
@@ -476,6 +483,9 @@ test.describe("Combos flow", () => {
         weight: 0,
       },
     ]);
+    expect(state.lastPayload?.config?.failoverBeforeRetry).toBe(true);
+    expect(state.lastPayload?.config?.maxSetRetries).toBe(2);
+    expect(state.lastPayload?.config?.setRetryDelayMs).toBe(1500);
   });
 
   test("allows dragging combo cards to persist manual order", async ({ page }) => {

tests/e2e/helpers/mockUpstreamServer.ts

@@ -0,0 +1,175 @@
+import http from "node:http";
+import net from "node:net";
+
+export interface PlannedResponse {
+  status: number;
+  body: Record<string, unknown>;
+  headers?: Record<string, string>;
+  delayMs?: number;
+}
+
+export interface TokenState {
+  defaultResponse: PlannedResponse;
+  queue: PlannedResponse[];
+  hits: number;
+  startedAt: number[];
+  bodies: Array<Record<string, unknown>>;
+}
+
+function getFreePort(): Promise<number> {
+  return new Promise<number>((resolve, reject) => {
+    const server = net.createServer();
+    server.once("error", reject);
+    server.listen(0, "127.0.0.1", () => {
+      const address = server.address();
+      if (!address || typeof address === "string") {
+        server.close();
+        reject(new Error("Failed to allocate a free port"));
+        return;
+      }
+      const { port } = address;
+      server.close((err) => {
+        if (err) reject(err);
+        else resolve(port);
+      });
+    });
+  });
+}
+
+export function buildCompletion(
+  text: string,
+  overrides: Partial<PlannedResponse> & { model?: string } = {}
+) {
+  return {
+    status: overrides.status ?? 200,
+    delayMs: overrides.delayMs,
+    headers: overrides.headers,
+    body: overrides.body ?? {
+      id: `chatcmpl_${Math.random().toString(16).slice(2, 8)}`,
+      object: "chat.completion",
+      model: overrides.model ?? "test-model",
+      choices: [{ index: 0, message: { role: "assistant", content: text }, finish_reason: "stop" }],
+      usage: { prompt_tokens: 4, completion_tokens: 2, total_tokens: 6 },
+    },
+  };
+}
+
+export function buildError(status: number, message: string, headers: Record<string, string> = {}) {
+  return {
+    status,
+    headers,
+    body: { error: { message } },
+  };
+}
+
+export class MockUpstreamServer {
+  private behaviors = new Map<string, TokenState>();
+  private server: http.Server | null = null;
+  private _baseUrl = "";
+
+  get baseUrl(): string {
+    if (!this._baseUrl) throw new Error("Server not started yet");
+    return this._baseUrl;
+  }
+
+  get isRunning(): boolean {
+    return this.server !== null;
+  }
+
+  async start(): Promise<string> {
+    const port = await getFreePort();
+    this.server = http.createServer((req, res) => {
+      void this.handleRequest(req, res);
+    });
+    await new Promise<void>((resolve, reject) => {
+      this.server!.once("error", reject);
+      this.server!.listen(port, "127.0.0.1", () => resolve());
+    });
+    this._baseUrl = `http://127.0.0.1:${port}/v1`;
+    return this._baseUrl;
+  }
+
+  configureToken(
+    token: string,
+    config: { defaultResponse: PlannedResponse; queue?: PlannedResponse[] }
+  ): void {
+    this.behaviors.set(token, {
+      defaultResponse: config.defaultResponse,
+      queue: [...(config.queue || [])],
+      hits: 0,
+      startedAt: [],
+      bodies: [],
+    });
+  }
+
+  getState(token: string): TokenState {
+    const state = this.behaviors.get(token);
+    if (!state) throw new Error(`Unknown token: ${token}`);
+    return state;
+  }
+
+  resetState(token: string, queue?: PlannedResponse[]): void {
+    const state = this.behaviors.get(token);
+    if (!state) throw new Error(`Unknown token: ${token}`);
+    state.hits = 0;
+    state.startedAt = [];
+    state.bodies = [];
+    state.queue = [...(queue || [])];
+  }
+
+  async stop(): Promise<void> {
+    if (!this.server) return;
+    await new Promise<void>((resolve) => this.server?.close(() => resolve()));
+    this.server = null;
+  }
+
+  private async handleRequest(req: http.IncomingMessage, res: http.ServerResponse): Promise<void> {
+    const chunks: Buffer[] = [];
+    for await (const chunk of req) {
+      chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk));
+    }
+
+    const authHeader = String(req.headers.authorization || "");
+    const token = authHeader.replace(/^Bearer\s+/i, "").trim();
+    const rawBody = Buffer.concat(chunks).toString("utf8");
+    const parsedBody = rawBody ? JSON.parse(rawBody) : {};
+
+    if (req.method === "GET" && req.url?.startsWith("/v1/models")) {
+      res.writeHead(200, { "Content-Type": "application/json" });
+      res.end(JSON.stringify({ data: [{ id: "test-model", object: "model" }] }));
+      return;
+    }
+
+    if (req.method !== "POST" || !req.url?.startsWith("/v1/chat/completions")) {
+      res.writeHead(404, { "Content-Type": "application/json" });
+      res.end(JSON.stringify({ error: { message: `Unhandled: ${req.method} ${req.url}` } }));
+      return;
+    }
+
+    const behavior = this.behaviors.get(token);
+    if (!behavior) {
+      res.writeHead(401, { "Content-Type": "application/json" });
+      res.end(JSON.stringify({ error: { message: `Unknown token: ${token || "missing"}` } }));
+      return;
+    }
+
+    behavior.hits += 1;
+    behavior.startedAt.push(Date.now());
+    behavior.bodies.push(parsedBody as Record<string, unknown>);
+
+    const planned = behavior.queue.shift() || behavior.defaultResponse;
+    process.stderr.write(
+      `[MOCK] ${token.slice(0, 8)} hit=${behavior.hits} status=${planned.status}\n`
+    );
+    if (planned.delayMs && planned.delayMs > 0) {
+      await new Promise((r) => setTimeout(r, planned.delayMs));
+    }
+
+    const headers: Record<string, string> = {
+      "Content-Type": "application/json",
+      ...(planned.headers || {}),
+    };
+    res.writeHead(planned.status, headers);
+    res.end(JSON.stringify(planned.body));
+  }
+}

tests/e2e/system-failover.test.ts

@@ -0,0 +1,699 @@
+import test from "node:test";
+import assert from "node:assert/strict";
+import fs from "node:fs";
+import fsp from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import net from "node:net";
+import { spawn } from "node:child_process";
+import { fileURLToPath } from "node:url";
+import { MockUpstreamServer, buildCompletion, buildError } from "./helpers/mockUpstreamServer.ts";
+
+const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-system-failover-"));
+const DASHBOARD_PORT = await getFreePort();
+const REPO_ROOT = fileURLToPath(new URL("../..", import.meta.url));
+
+process.env.DATA_DIR = TEST_DATA_DIR;
+process.env.DISABLE_SQLITE_AUTO_BACKUP = "true";
+process.env.API_KEY_SECRET = process.env.API_KEY_SECRET || "system-failover-secret-123456";
+process.env.REQUIRE_API_KEY = "false";
+
+const core = await import("../../src/lib/db/core.ts");
+const providersDb = await import("../../src/lib/db/providers.ts");
+const combosDb = await import("../../src/lib/db/combos.ts");
+const settingsDb = await import("../../src/lib/db/settings.ts");
+const accountFallback = await import("../../open-sse/services/accountFallback.ts");
+
+function resetConnectionCooldowns() {
+  accountFallback.clearAllModelLockouts();
+  const db = core.getDbInstance() as any;
+  db.prepare(
+    `UPDATE provider_connections
+     SET rate_limited_until = NULL,
+         test_status = 'active',
+         backoff_level = 0,
+         last_error = NULL,
+         last_error_type = NULL,
+         last_error_source = NULL,
+         error_code = NULL,
+         last_error_at = NULL
+     WHERE rate_limited_until IS NOT NULL
+        OR test_status != 'active'`
+  ).run();
+  db.pragma("wal_checkpoint(TRUNCATE)");
+}
+
+function getFreePort() {
+  return new Promise<number>((resolve, reject) => {
+    const server = net.createServer();
+    server.once("error", reject);
+    server.listen(0, "127.0.0.1", () => {
+      const address = server.address();
+      if (!address || typeof address === "string") {
+        server.close();
+        reject(new Error("Failed to allocate a free port"));
+        return;
+      }
+      const { port } = address;
+      server.close((closeError) => {
+        if (closeError) reject(closeError);
+        else resolve(port);
+      });
+    });
+  });
+}
+
+function sleep(ms: number) {
+  return new Promise((resolve) => setTimeout(resolve, ms));
+}
+
+async function seedProvider(label: string, apiKey: string, baseUrl: string) {
+  const providerId = `openai-compatible-sys-${label}`;
+  await providersDb.createProviderNode({
+    id: providerId,
+    type: "openai-compatible",
+    name: `System ${label}`,
+    prefix: label,
+    apiType: "chat",
+    baseUrl,
+  });
+  await providersDb.createProviderConnection({
+    provider: providerId,
+    authType: "apikey",
+    name: `conn-${label}`,
+    apiKey,
+    isActive: true,
+    testStatus: "active",
+    providerSpecificData: { baseUrl, apiType: "chat" },
+  });
+  return { providerId, model: `${label}/test-model`, apiKey };
+}
+
+function createServerProcess(dataDir: string, port: number) {
+  const stdoutLines: string[] = [];
+  const stderrLines: string[] = [];
+  let exitInfo: { code: number | null; signal: NodeJS.Signals | null } | null = null;
+  const child = spawn(process.execPath, ["scripts/dev/run-next-playwright.mjs", "dev"], {
+    cwd: REPO_ROOT,
+    env: {
+      ...process.env,
+      DATA_DIR: dataDir,
+      PORT: String(port),
+      DASHBOARD_PORT: String(port),
+      API_PORT: String(port),
+      HOST: "127.0.0.1",
+      REQUIRE_API_KEY: "false",
+      API_KEY_SECRET: process.env.API_KEY_SECRET || "system-failover-secret-123456",
+      DISABLE_SQLITE_AUTO_BACKUP: "true",
+      INITIAL_PASSWORD: "",
+      NEXT_TELEMETRY_DISABLED: "1",
+      OMNIROUTE_DISABLE_BACKGROUND_SERVICES: "true",
+      OMNIROUTE_DISABLE_TOKEN_HEALTHCHECK: "true",
+      OMNIROUTE_DISABLE_LOCAL_HEALTHCHECK: "true",
+      OMNIROUTE_HIDE_HEALTHCHECK_LOGS: "true",
+      OMNIROUTE_E2E_BOOTSTRAP_MODE: "open",
+    },
+    stdio: ["ignore", "pipe", "pipe"],
+  });
+
+  child.once("exit", (code, signal) => {
+    exitInfo = { code, signal };
+  });
+  child.stdout.on("data", (chunk) => {
+    const lines = String(chunk).split(/\r?\n/).filter(Boolean);
+    stdoutLines.push(...lines);
+    if (stdoutLines.length > 200) stdoutLines.splice(0, stdoutLines.length - 200);
+  });
+  child.stderr.on("data", (chunk) => {
+    const lines = String(chunk).split(/\r?\n/).filter(Boolean);
+    stderrLines.push(...lines);
+    if (stderrLines.length > 200) stderrLines.splice(0, stderrLines.length - 200);
+  });
+
+  return {
+    child,
+    stdoutLines,
+    stderrLines,
+    baseUrl: `http://127.0.0.1:${port}`,
+    get exitInfo() {
+      return exitInfo;
+    },
+  };
+}
+
+async function waitForServer(
+  baseUrl: string,
+  logs: {
+    stdoutLines: string[];
+    stderrLines: string[];
+    exitInfo?: { code: number | null; signal: NodeJS.Signals | null } | null;
+  }
+) {
+  const startedAt = Date.now();
+  let lastError = "";
+  while (Date.now() - startedAt < 120_000) {
+    if (logs.exitInfo) {
+      throw new Error(
+        [
+          `OmniRoute exited before it became ready (code=${logs.exitInfo.code}, signal=${logs.exitInfo.signal})`,
+          "--- stdout ---",
+          ...logs.stdoutLines.slice(-40),
+          "--- stderr ---",
+          ...logs.stderrLines.slice(-40),
+        ].join("\n")
+      );
+    }
+
+    try {
+      const response = await fetch(`${baseUrl}/api/monitoring/health`, {
+        signal: AbortSignal.timeout(5_000),
+      });
+      if (response.ok) return;
+      lastError = `HTTP ${response.status}`;
+    } catch (error: any) {
+      lastError = error instanceof Error ? error.message : String(error);
+    }
+    await sleep(500);
+  }
+
+  throw new Error(
+    [
+      `Timed out waiting for OmniRoute to start: ${lastError}`,
+      "--- stdout ---",
+      ...logs.stdoutLines.slice(-40),
+      "--- stderr ---",
+      ...logs.stderrLines.slice(-40),
+    ].join("\n")
+  );
+}
+
+async function stopProcess(child: ReturnType<typeof spawn>) {
+  if (child.killed) return;
+  child.kill("SIGTERM");
+  const exited = await Promise.race([
+    new Promise<boolean>((resolve) => child.once("exit", () => resolve(true))),
+    sleep(5_000).then(() => false),
+  ]);
+  if (!exited && !child.killed) {
+    child.kill("SIGKILL");
+    await new Promise<void>((resolve) => child.once("exit", () => resolve()));
+  }
+}
+
+async function postChat(
+  baseUrl: string,
+  model: string,
+  content: string,
+  extraHeaders?: Record<string, string>
+) {
+  const response = await fetch(`${baseUrl}/api/v1/chat/completions`, {
+    method: "POST",
+    headers: { "Content-Type": "application/json", ...extraHeaders },
+    body: JSON.stringify({
+      model,
+      stream: false,
+      messages: [{ role: "user", content }],
+    }),
+    signal: AbortSignal.timeout(30_000),
+  });
+  const text = await response.text();
+  const json = text ? JSON.parse(text) : {};
+  return { response, json };
+}
+
+async function resetBreakers(url: string) {
+  await fetch(`${url}/api/resilience/reset`, {
+    method: "POST",
+    signal: AbortSignal.timeout(5_000),
+  });
+}
+
+const serverA = new MockUpstreamServer();
+const serverB = new MockUpstreamServer();
+let app:
+  | {
+      child: ReturnType<typeof spawn>;
+      stdoutLines: string[];
+      stderrLines: string[];
+      baseUrl: string;
+    }
+  | undefined;
+
+const TOKEN_A = "sk-sys-a";
+const TOKEN_B = "sk-sys-b";
+const TOKEN_A2 = "sk-sys-a2";
+const TOKEN_B2 = "sk-sys-b2";
+
+test.before(async () => {
+  const baseUrlA = await serverA.start();
+  const baseUrlB = await serverB.start();
+
+  serverA.configureToken(TOKEN_A, {
+    defaultResponse: buildCompletion("server A ok", { model: "sys-a/test-model" }),
+  });
+  serverA.configureToken(TOKEN_A2, {
+    defaultResponse: buildCompletion("server A2 ok", { model: "sys-a2/test-model" }),
+  });
+  serverB.configureToken(TOKEN_B, {
+    defaultResponse: buildCompletion("server B ok", { model: "sys-b/test-model" }),
+  });
+  serverB.configureToken(TOKEN_B2, {
+    defaultResponse: buildCompletion("server B2 ok", { model: "sys-b2/test-model" }),
+  });
+
+  const provA = await seedProvider("sys-a", TOKEN_A, baseUrlA);
+  const provB = await seedProvider("sys-b", TOKEN_B, baseUrlB);
+  const provA2 = await seedProvider("sys-a2", TOKEN_A2, baseUrlA);
+  const provB2 = await seedProvider("sys-b2", TOKEN_B2, baseUrlB);
+
+  await combosDb.createCombo({
+    name: "sys-priority",
+    strategy: "priority",
+    config: { maxRetries: 0, retryDelayMs: 0 },
+    models: [provA.model, provB.model],
+  });
+  await combosDb.createCombo({
+    name: "sys-priority-v2",
+    strategy: "priority",
+    config: { maxRetries: 0, retryDelayMs: 0 },
+    models: [provA2.model, provB2.model],
+  });
+  await combosDb.createCombo({
+    name: "sys-priority-fobr",
+    strategy: "priority",
+    config: { maxRetries: 0, retryDelayMs: 0, failoverBeforeRetry: true },
+    models: [provA.model, provB.model],
+  });
+  await combosDb.createCombo({
+    name: "sys-priority-setretry",
+    strategy: "priority",
+    config: {
+      maxRetries: 0,
+      retryDelayMs: 0,
+      failoverBeforeRetry: true,
+      maxSetRetries: 1,
+      setRetryDelayMs: 500,
+    },
+    models: [provA.model, provB.model],
+  });
+  await combosDb.createCombo({
+    name: "sys-same-server",
+    strategy: "priority",
+    config: { maxRetries: 0, retryDelayMs: 0 },
+    models: [provA.model, provA2.model],
+  });
+  await combosDb.createCombo({
+    name: "sys-same-server-fobr",
+    strategy: "priority",
+    config: { maxRetries: 0, retryDelayMs: 0, failoverBeforeRetry: true },
+    models: [provA.model, provA2.model],
+  });
+
+  await combosDb.createCombo({
+    name: "sys-single-provider",
+    strategy: "priority",
+    config: { maxRetries: 0, retryDelayMs: 0 },
+    models: ["sys-a/modelA", "sys-a/modelB"],
+  });
+  await combosDb.createCombo({
+    name: "sys-single-provider-fobr",
+    strategy: "priority",
+    config: { maxRetries: 0, retryDelayMs: 0, failoverBeforeRetry: true },
+    models: ["sys-a/modelA", "sys-a/modelB"],
+  });
+
+  await settingsDb.updateSettings({
+    resilienceSettings: {
+      requestQueue: {
+        autoEnableApiKeyProviders: true,
+        requestsPerMinute: 120,
+        minTimeBetweenRequestsMs: 0,
+        concurrentRequests: 4,
+        maxWaitMs: 2_000,
+      },
+      connectionCooldown: {
+        oauth: { baseCooldownMs: 500, useUpstreamRetryHints: true, maxBackoffSteps: 3 },
+        apikey: { baseCooldownMs: 200, useUpstreamRetryHints: false, maxBackoffSteps: 0 },
+      },
+      providerBreaker: {
+        oauth: { failureThreshold: 3, resetTimeoutMs: 2_000 },
+        apikey: { failureThreshold: 2, resetTimeoutMs: 1_500 },
+      },
+      waitForCooldown: {
+        enabled: false,
+        maxRetries: 0,
+        maxRetryWaitSec: 0,
+      },
+    },
+    requestRetry: 0,
+    maxRetryIntervalSec: 0,
+    requireLogin: false,
+    setupComplete: true,
+  });
+
+  core.closeDbInstance();
+
+  app = createServerProcess(TEST_DATA_DIR, DASHBOARD_PORT);
+  await waitForServer(app.baseUrl, app);
+
+  const warmup = await postChat(app.baseUrl, "sys-b/test-model", "warm up");
+  assert.equal(warmup.response.status, 200, JSON.stringify(warmup.json));
+  serverB.resetState(TOKEN_B);
+});
+
+test.after(async () => {
+  if (app) await stopProcess(app.child);
+  await serverA.stop();
+  await serverB.stop();
+  core.closeDbInstance();
+  await fsp.rm(TEST_DATA_DIR, { recursive: true, force: true });
+});
+
+test("primary healthy: request routes to Server A only", async () => {
+  assert.ok(app);
+  serverA.resetState(TOKEN_A);
+  serverB.resetState(TOKEN_B);
+
+  const result = await postChat(app.baseUrl, "sys-priority", "healthy primary");
+
+  assert.equal(result.response.status, 200, JSON.stringify(result.json));
+  assert.equal(result.json.choices[0].message.content, "server A ok");
+  assert.equal(result.json.model, "sys-a/test-model");
+  assert.equal(serverA.getState(TOKEN_A).hits, 1);
+  assert.equal(serverB.getState(TOKEN_B).hits, 0);
+});
+
+test("500 Internal Server Error: combo falls back to Server B", async () => {
+  assert.ok(app);
+  serverA.resetState(TOKEN_A, [buildError(500, "Internal Server Error")]);
+  serverB.resetState(TOKEN_B);
+
+  const result = await postChat(app.baseUrl, "sys-priority", "500 fallback");
+
+  assert.equal(result.response.status, 200, JSON.stringify(result.json));
+  assert.equal(result.json.choices[0].message.content, "server B ok");
+  assert.equal(result.json.model, "sys-b/test-model");
+  assert.equal(serverA.getState(TOKEN_A).hits, 1);
+  assert.equal(serverB.getState(TOKEN_B).hits, 1);
+});
+
+test("503 Service Unavailable: combo falls back to Server B", async () => {
+  assert.ok(app);
+  await resetBreakers(app.baseUrl);
+  resetConnectionCooldowns();
+  serverA.resetState(TOKEN_A, [buildError(503, "Service Unavailable")]);
+  serverB.resetState(TOKEN_B);
+
+  const result = await postChat(app.baseUrl, "sys-priority", "503 fallback");
+
+  assert.equal(result.response.status, 200, JSON.stringify(result.json));
+  assert.equal(result.json.choices[0].message.content, "server B ok");
+  assert.equal(result.json.model, "sys-b/test-model");
+  assert.equal(serverA.getState(TOKEN_A).hits, 1);
+  assert.equal(serverB.getState(TOKEN_B).hits, 1);
+});
+
+test("both servers fail (500): request returns a 5xx error to the client", async () => {
+  assert.ok(app);
+  await resetBreakers(app.baseUrl);
+  resetConnectionCooldowns();
+  serverA.resetState(TOKEN_A, [buildError(500, "A is down")]);
+  serverB.resetState(TOKEN_B, [buildError(500, "B is down")]);
+
+  const result = await postChat(app.baseUrl, "sys-priority", "both down");
+
+  assert.ok(result.response.status >= 500, `expected 5xx, got ${result.response.status}`);
+});
+
+test("combo fallback to Server B survives sequential 503 failures from Server A", async () => {
+  assert.ok(app);
+  await resetBreakers(app.baseUrl);
+  resetConnectionCooldowns();
+  serverA.resetState(TOKEN_A, [buildError(503, "transient blip"), buildError(503, "second blip")]);
+  serverB.resetState(TOKEN_B);
+
+  const first = await postChat(app.baseUrl, "sys-priority", "seq 503 attempt 1");
+  assert.equal(first.response.status, 200, JSON.stringify(first.json));
+  assert.equal(first.json.choices[0].message.content, "server B ok");
+  assert.equal(first.json.model, "sys-b/test-model");
+
+  // Wait for the 200ms apikey cooldown to expire so the second request also
+  // goes through the full A→B fallback path rather than skipping A entirely.
+  await sleep(250);
+
+  const second = await postChat(app.baseUrl, "sys-priority", "seq 503 attempt 2");
+  assert.equal(second.response.status, 200, JSON.stringify(second.json));
+  assert.equal(second.json.choices[0].message.content, "server B ok");
+  assert.equal(second.json.model, "sys-b/test-model");
+
+  assert.equal(serverA.getState(TOKEN_A).hits, 2);
+  assert.equal(serverB.getState(TOKEN_B).hits, 2);
+});
+
+test("429 with Retry-After and wait-for-cooldown: primary retries then falls back to B", async () => {
+  assert.ok(app);
+  await resetBreakers(app.baseUrl);
+  resetConnectionCooldowns();
+  serverA.resetState(TOKEN_A, [
+    buildError(429, "rate limited, retry after 1s", { "Retry-After": "1" }),
+  ]);
+  serverB.resetState(TOKEN_B);
+
+  const patchRes = await fetch(`${app.baseUrl}/api/resilience`, {
+    method: "PATCH",
+    headers: { "Content-Type": "application/json" },
+    body: JSON.stringify({
+      connectionCooldown: {
+        apikey: { useUpstreamRetryHints: true, baseCooldownMs: 200 },
+      },
+      waitForCooldown: { enabled: true, maxRetries: 1, maxRetryWaitSec: 2 },
+    }),
+    signal: AbortSignal.timeout(10_000),
+  });
+});
+
+test("failoverBeforeRetry enabled: upstream error triggers immediate failover to next target", async () => {
+  assert.ok(app);
+  await resetBreakers(app.baseUrl);
+  resetConnectionCooldowns();
+  serverA.resetState(TOKEN_A, [buildError(429, "rate limited")]);
+  serverB.resetState(TOKEN_B);
+
+  const result = await postChat(app.baseUrl, "sys-priority-fobr", "test failover before retry");
+
+  assert.equal(result.response.status, 200, JSON.stringify(result.json));
+  assert.equal(result.json.choices[0].message.content, "server B ok");
+  assert.equal(result.json.model, "sys-b/test-model");
+
+  // With failoverBeforeRetry=true, A should be hit exactly ONCE (no intra-URL retry)
+  assert.equal(serverA.getState(TOKEN_A).hits, 1);
+  assert.equal(serverB.getState(TOKEN_B).hits, 1);
+});
+
+test("failoverBeforeRetry disabled: 429 triggers executor intra-URL retry, succeeds on retry", async () => {
+  assert.ok(app);
+  // Full resilience reset so A isn't blocked by residual breaker/cooldown state
+  const patchRes = await fetch(`${app.baseUrl}/api/resilience`, {
+    method: "PATCH",
+    headers: { "Content-Type": "application/json" },
+    body: JSON.stringify({
+      connectionCooldown: {
+        apikey: { useUpstreamRetryHints: false, baseCooldownMs: 0, maxBackoffSteps: 0 },
+        oauth: { useUpstreamRetryHints: false, baseCooldownMs: 0, maxBackoffSteps: 0 },
+      },
+      waitForCooldown: { enabled: false, maxRetries: 0, maxRetryWaitSec: 0 },
+    }),
+    signal: AbortSignal.timeout(10_000),
+  });
+  assert.equal(patchRes.status, 200);
+  await resetBreakers(app.baseUrl);
+  resetConnectionCooldowns();
+  // Wait for any residual cooldowns to expire
+  await sleep(300);
+  // One 429, then the default (200) on retry
+  serverA.resetState(TOKEN_A, [buildError(429, "rate limited")]);
+  serverB.resetState(TOKEN_B);
+
+  const result = await postChat(app.baseUrl, "sys-priority", "test failover before retry disabled");
+
+  assert.equal(result.response.status, 200, JSON.stringify(result.json));
+
+  // With maxRetries=0 the combo does not retry A — it fails over to B immediately.
+  assert.equal(result.json.model, "sys-b/test-model");
+
+  // With failoverBeforeRetry=false, A should be hit TWICE (initial + 1 intra-URL retry).
+  // This contrasts with failoverBeforeRetry=true where A is hit exactly ONCE.
+  assert.equal(serverA.getState(TOKEN_A).hits, 2);
+});
+
+test("maxSetRetries: both A and B fail first pass, A 429 again, B 200 on retry", async () => {
+  assert.ok(app);
+  // Reset to a clean resilience slate: disable cooldowns, disable waitForCooldown,
+  // reset breakers, and clear connection rate_limited_until from previous tests.
+  const patchRes = await fetch(`${app.baseUrl}/api/resilience`, {
+    method: "PATCH",
+    headers: { "Content-Type": "application/json" },
+    body: JSON.stringify({
+      connectionCooldown: {
+        apikey: { useUpstreamRetryHints: false, baseCooldownMs: 0, maxBackoffSteps: 0 },
+        oauth: { useUpstreamRetryHints: false, baseCooldownMs: 0, maxBackoffSteps: 0 },
+      },
+      waitForCooldown: { enabled: false, maxRetries: 0, maxRetryWaitSec: 0 },
+    }),
+    signal: AbortSignal.timeout(10_000),
+  });
+  assert.equal(patchRes.status, 200);
+  await resetBreakers(app.baseUrl);
+  resetConnectionCooldowns();
+  // Set try 0: A 429, B 500 — both fail
+  // Set try 1: A 429, B 200 — B succeeds
+  serverA.resetState(TOKEN_A, [buildError(429, "rate limited"), buildError(429, "rate limited")]);
+  serverB.resetState(TOKEN_B, [
+    buildError(500, "server error"),
+    buildCompletion("server B ok on retry", { model: "sys-b/test-model" }),
+  ]);
+
+  const result = await postChat(app.baseUrl, "sys-priority-setretry", "test max set retries", {
+    "x-internal-test": "combo-health-check",
+  });
+
+  // Set try 0: A 429, B 500 → both fail
+  // Set try 1: A 429, B 200 → B succeeds
+  assert.equal(result.response.status, 200, JSON.stringify(result.json));
+  assert.equal(result.json.choices[0].message.content, "server B ok on retry");
+  assert.equal(result.json.model, "sys-b/test-model");
+  assert.equal(serverA.getState(TOKEN_A).hits, 2);
+  assert.equal(serverB.getState(TOKEN_B).hits, 2);
+
+  // Restore defaults so other tests are not affected
+  await fetch(`${app.baseUrl}/api/resilience`, {
+    method: "PATCH",
+    headers: { "Content-Type": "application/json" },
+    body: JSON.stringify({
+      connectionCooldown: {
+        apikey: { useUpstreamRetryHints: false, baseCooldownMs: 200, maxBackoffSteps: 0 },
+        oauth: { useUpstreamRetryHints: true, baseCooldownMs: 500, maxBackoffSteps: 3 },
+      },
+    }),
+    signal: AbortSignal.timeout(10_000),
+  });
+});
+
+test("same server failoverBeforeRetry disabled: first model 429 retried before trying second", async () => {
+  assert.ok(app);
+  // Full resilience reset — previous test restored defaults and may have left A cooldown
+  const patchRes = await fetch(`${app.baseUrl}/api/resilience`, {
+    method: "PATCH",
+    headers: { "Content-Type": "application/json" },
+    body: JSON.stringify({
+      connectionCooldown: {
+        apikey: { useUpstreamRetryHints: false, baseCooldownMs: 0, maxBackoffSteps: 0 },
+        oauth: { useUpstreamRetryHints: false, baseCooldownMs: 0, maxBackoffSteps: 0 },
+      },
+      waitForCooldown: { enabled: false, maxRetries: 0, maxRetryWaitSec: 0 },
+    }),
+    signal: AbortSignal.timeout(10_000),
+  });
+  assert.equal(patchRes.status, 200);
+  await sleep(300);
+  await resetBreakers(app.baseUrl);
+  resetConnectionCooldowns();
+  serverA.resetState(TOKEN_A, [buildError(429, "rate limited")]);
+  serverA.resetState(TOKEN_A2);
+
+  const result = await postChat(
+    app.baseUrl,
+    "sys-same-server",
+    "test same server failover disabled"
+  );
+
+  assert.equal(result.response.status, 200, JSON.stringify(result.json));
+
+  // With maxRetries=0 the combo fails over to A2 rather than retrying A.
+  assert.equal(result.json.model, "sys-a2/test-model");
+  assert.equal(serverA.getState(TOKEN_A).hits, 2);
+});
+
+test("same server failoverBeforeRetry enabled: first model 429 skipped to second immediately", async () => {
+  assert.ok(app);
+  await sleep(300);
+  await resetBreakers(app.baseUrl);
+  resetConnectionCooldowns();
+  serverA.resetState(TOKEN_A, [buildError(429, "rate limited")]);
+  serverA.resetState(TOKEN_A2);
+
+  const result = await postChat(
+    app.baseUrl,
+    "sys-same-server-fobr",
+    "test same server failover enabled"
+  );
+
+  assert.equal(result.response.status, 200, JSON.stringify(result.json));
+  assert.equal(result.json.model, "sys-a2/test-model");
+  assert.equal(serverA.getState(TOKEN_A).hits, 1);
+  assert.equal(serverA.getState(TOKEN_A2).hits, 1);
+});
+
+test("single provider, modelA 500: combo fails over to modelB", async () => {
+  assert.ok(app);
+  await resetBreakers(app.baseUrl);
+  resetConnectionCooldowns();
+  serverA.resetState(TOKEN_A, [buildError(500, "model A error")]);
+
+  const result = await postChat(app.baseUrl, "sys-single-provider", "test modelA 500");
+
+  assert.equal(result.response.status, 200, JSON.stringify(result.json));
+  assert.equal(result.json.model, "sys-a/test-model");
+  assert.equal(serverA.getState(TOKEN_A).hits, 2);
+});
+
+test("single provider, modelA 503: combo fails over to modelB", async () => {
+  assert.ok(app);
+  await resetBreakers(app.baseUrl);
+  resetConnectionCooldowns();
+  serverA.resetState(TOKEN_A, [buildError(503, "Service Unavailable")]);
+
+  const result = await postChat(app.baseUrl, "sys-single-provider", "test modelA 503");
+
+  assert.equal(result.response.status, 200, JSON.stringify(result.json));
+  assert.equal(result.json.model, "sys-a/test-model");
+  assert.equal(serverA.getState(TOKEN_A).hits, 2);
+});
+
+test("single provider, modelA 429 with fobr: immediate failover to modelB", async () => {
+  assert.ok(app);
+  await resetBreakers(app.baseUrl);
+  resetConnectionCooldowns();
+  serverA.resetState(TOKEN_A, [buildError(429, "rate limited")]);
+
+  const result = await postChat(app.baseUrl, "sys-single-provider-fobr", "test fobr");
+
+  assert.equal(result.response.status, 200, JSON.stringify(result.json));
+  assert.equal(result.json.model, "sys-a/test-model");
+  assert.equal(serverA.getState(TOKEN_A).hits, 2);
+});
+
+test("single provider, modelA 500 with fobr: modelA retry", async () => {
+  assert.ok(app);
+  await resetBreakers(app.baseUrl);
+  resetConnectionCooldowns();
+  serverA.resetState(TOKEN_A, [buildError(500, "Oops!")]);
+
+  const result = await postChat(app.baseUrl, "sys-single-provider-fobr", "test fobr");
+
+  assert.equal(result.response.status, 200, JSON.stringify(result.json));
+  assert.equal(result.json.model, "sys-a/test-model");
+  assert.equal(serverA.getState(TOKEN_A).hits, 2);
+});
+
+test("single provider, both models fail: request returns 5xx to client", async () => {
+  assert.ok(app);
+  await resetBreakers(app.baseUrl);
+  resetConnectionCooldowns();
+  serverA.resetState(TOKEN_A, [buildError(500, "modelA down"), buildError(500, "modelB down")]);
+
+  const result = await postChat(app.baseUrl, "sys-single-provider", "both down");
+
+  assert.ok(result.response.status >= 500, `expected 5xx, got ${result.response.status}`);
+  assert.equal(serverA.getState(TOKEN_A).hits, 2);
+});

tests/unit/combo-config.test.ts

@@ -20,6 +20,9 @@ test("getDefaultComboConfig returns a fresh copy of the defaults", () => {
   assert.equal(first.handoffThreshold, 0.85);
   assert.equal(first.maxMessagesForSummary, 30);
   assert.deepEqual(first.handoffProviders, ["codex"]);
+  assert.equal(first.failoverBeforeRetry, false);
+  assert.equal(first.maxSetRetries, 0);
+  assert.equal(first.setRetryDelayMs, 2000);
 
   first.strategy = "weighted";
   assert.equal(second.strategy, "priority");
@@ -272,3 +275,94 @@ test("updateComboDefaultsSchema rejects composite tiers in global defaults and p
     ]
   );
 });
+
+test("createComboSchema accepts failoverBeforeRetry, maxSetRetries and setRetryDelayMs", () => {
+  const parsed = createComboSchema.parse({
+    name: "failover-test",
+    models: ["openai/gpt-4"],
+    strategy: "priority",
+    config: {
+      failoverBeforeRetry: true,
+      maxSetRetries: 3,
+      setRetryDelayMs: 1500,
+    },
+  });
+
+  assert.equal(parsed.config.failoverBeforeRetry, true);
+  assert.equal(parsed.config.maxSetRetries, 3);
+  assert.equal(parsed.config.setRetryDelayMs, 1500);
+});
+
+test("createComboSchema coerces string numbers for maxSetRetries and setRetryDelayMs", () => {
+  const parsed = createComboSchema.parse({
+    name: "coerce-test",
+    models: ["openai/gpt-4"],
+    strategy: "priority",
+    config: {
+      maxSetRetries: "2",
+      setRetryDelayMs: "500",
+    },
+  });
+
+  assert.equal(parsed.config.maxSetRetries, 2);
+  assert.equal(parsed.config.setRetryDelayMs, 500);
+});
+
+test("createComboSchema rejects maxSetRetries out of range", () => {
+  const tooHigh = createComboSchema.safeParse({
+    name: "bad-max",
+    models: ["openai/gpt-4"],
+    strategy: "priority",
+    config: { maxSetRetries: 11 },
+  });
+  assert.equal(tooHigh.success, false);
+
+  const negative = createComboSchema.safeParse({
+    name: "bad-max",
+    models: ["openai/gpt-4"],
+    strategy: "priority",
+    config: { maxSetRetries: -1 },
+  });
+  assert.equal(negative.success, false);
+});
+
+test("createComboSchema rejects setRetryDelayMs out of range", () => {
+  const tooHigh = createComboSchema.safeParse({
+    name: "bad-delay",
+    models: ["openai/gpt-4"],
+    strategy: "priority",
+    config: { setRetryDelayMs: 60001 },
+  });
+  assert.equal(tooHigh.success, false);
+
+  const negative = createComboSchema.safeParse({
+    name: "bad-delay",
+    models: ["openai/gpt-4"],
+    strategy: "priority",
+    config: { setRetryDelayMs: -1 },
+  });
+  assert.equal(negative.success, false);
+});
+
+test("resolveComboConfig cascades failoverBeforeRetry, maxSetRetries and setRetryDelayMs", () => {
+  const result = resolveComboConfig(
+    {
+      config: {
+        failoverBeforeRetry: true,
+        maxSetRetries: 2,
+        setRetryDelayMs: 3000,
+      },
+    },
+    {
+      comboDefaults: {
+        failoverBeforeRetry: false,
+        maxSetRetries: 0,
+        setRetryDelayMs: 2000,
+      },
+    }
+  );
+
+  assert.equal(result.failoverBeforeRetry, true);
+  assert.equal(result.maxSetRetries, 2);
+  assert.equal(result.setRetryDelayMs, 3000);
+});