fix(resilience): route-restriction 403 must not mark a connection unavailable (#2929)

Fireworks Fire Pass (fpk_*) keys return 403 "Fire Pass API keys are not
authorized for this route." on /models while still serving chat. Two paths
wrongly penalized the key:

- validateOpenAILikeProvider returned "Invalid API key" for any 403 on the
  models endpoint without trying the chat probe.
- checkFallbackError classified the 403 as AUTH_ERROR (retryable cooldown), and
  even a generic fall-through hit the transient-cooldown default — marking the
  connection unavailable.

Fix: validateOpenAILikeProvider now inspects the 403 body and falls through to
the chat probe for "not authorized for this route" responses (401 and generic
403 still fail fast). checkFallbackError short-circuits such route-restriction
403s to { shouldFallback: false, cooldownMs: 0 } so the connection is not cooled
down. Per CLAUDE.md, a generic api-key 403 should be recoverable unless terminal.

Closes #2929

CHANGELOG.md

@@ -88,6 +88,12 @@
   resolves to the `opencode-zen` api-key tier (via a manual `ALIAS_TO_PROVIDER_ID`
   override), so combos built with the bare provider id misrouted away from the
   no-auth `opencode` provider; `oc/<model>` resolves correctly. (#2901)
+- **resilience/providers:** a route-restriction `403` (e.g. Fireworks Fire Pass
+  `fpk_*` keys returning "…not authorized for this route." on `/models`, while
+  chat still works) no longer marks the connection unavailable. Provider
+  validation falls through to the chat probe for such 403s instead of returning
+  "Invalid API key", and `checkFallbackError` short-circuits them to no cooldown.
+  Genuine auth failures (401 / generic 403) still fail fast. (#2929)
 
 ### ✨ New Features
 

open-sse/services/accountFallback.ts

@@ -1323,6 +1323,18 @@ export function checkFallbackError(
       };
     }
 
+    // #2929: A route-restriction 403 (e.g. Fireworks Fire Pass keys returning
+    // "Fire Pass API keys are not authorized for this route." on the /models
+    // endpoint) means the key is valid but lacks access to THIS route — it still
+    // serves chat. It must NOT cool down the connection or be classified as an
+    // auth error, otherwise a single model-listing 403 marks the key unavailable.
+    if (
+      status === HTTP_STATUS.FORBIDDEN &&
+      errorStr.toLowerCase().includes("not authorized for this route")
+    ) {
+      return { shouldFallback: false, cooldownMs: 0, reason: RateLimitReason.UNKNOWN };
+    }
+
     if (
       status === HTTP_STATUS.FORBIDDEN &&
       provider &&

src/lib/providers/validation.ts

@@ -382,10 +382,21 @@ async function validateOpenAILikeProvider({
       return { valid: true, error: null };
     }
 
-    if (response.status === 401 || response.status === 403) {
+    if (response.status === 401) {
       return { valid: false, error: "Invalid API key" };
     }
 
+    // #2929: A 403 on the models endpoint is not always a bad key. Some providers
+    // (e.g. Fireworks Fire Pass `fpk_*` keys) return "...not authorized for this
+    // route." on /models while still serving chat. Fall through to the chat probe
+    // for such route-restriction 403s instead of declaring the key invalid.
+    if (response.status === 403) {
+      const forbiddenBody = await response.text().catch(() => "");
+      if (!/not authorized for this route/i.test(forbiddenBody)) {
+        return { valid: false, error: "Invalid API key" };
+      }
+    }
+
     const chatUrl = resolveChatUrl(provider, baseUrl, providerSpecificData);
     if (!chatUrl) {
       return { valid: false, error: `Validation failed: ${response.status}` };

tests/unit/account-fallback-route-restriction-403.test.ts

@@ -0,0 +1,52 @@
+/**
+ * Issue #2929 — Fire Pass (fpk_*) API keys incorrectly marked as unavailable
+ * when the models endpoint returns 403.
+ *
+ * Fireworks Fire Pass keys return `403 "Fire Pass API keys are not authorized
+ * for this route."` on /models while still serving chat. That route-restriction
+ * 403 used to fall into the api-key-403 branch (→ AUTH_ERROR retryable fallback)
+ * or the generic "all other errors" default (→ transient cooldown), either of
+ * which cools down / marks the connection unavailable.
+ *
+ * A route-restriction 403 must be treated as benign for connection health:
+ * shouldFallback=false, no cooldown.
+ */
+import test from "node:test";
+import assert from "node:assert/strict";
+
+const { checkFallbackError } = await import("../../open-sse/services/accountFallback.ts");
+
+test("#2929 route-restriction 403 does NOT cool down the connection", () => {
+  const result = checkFallbackError(
+    403,
+    "Fire Pass API keys are not authorized for this route.",
+    0,
+    null,
+    "fireworks"
+  );
+  assert.equal(result.shouldFallback, false, "route-restriction 403 must not trigger fallback/cooldown");
+  assert.equal(result.cooldownMs, 0, "route-restriction 403 must not impose a cooldown");
+});
+
+test("#2929 a genuine api-key 403 still triggers fallback (no over-broadening)", () => {
+  // A 403 whose body does NOT indicate a route restriction must keep the
+  // existing behavior (auth-error / transient fallback), so real bad keys still
+  // get cooled down.
+  const result = checkFallbackError(403, "invalid api key", 0, null, "fireworks");
+  assert.equal(
+    result.shouldFallback,
+    true,
+    "a non-route-restriction 403 must still be treated as fallback-worthy"
+  );
+});
+
+test("#2929 case-insensitive match on the route-restriction phrase", () => {
+  const result = checkFallbackError(
+    403,
+    "ERROR: Not Authorized For This Route",
+    0,
+    null,
+    "fireworks"
+  );
+  assert.equal(result.shouldFallback, false);
+});

tests/unit/provider-validation-firepass-403.test.ts

@@ -0,0 +1,66 @@
+/**
+ * Issue #2929 — Fire Pass (fpk_*) keys marked invalid when /models returns 403.
+ *
+ * Fireworks Fire Pass keys return `403 "...not authorized for this route."` on
+ * the /models endpoint while still serving chat. validateOpenAILikeProvider (the
+ * default validator for registered OpenAI-format providers like `fireworks`)
+ * used to declare any 403 on /models as "Invalid API key" without trying the
+ * chat probe. For a route-restriction 403 it must now fall through to the chat
+ * probe; a generic 403 must still short-circuit as invalid.
+ */
+import test from "node:test";
+import assert from "node:assert/strict";
+
+const { validateProviderApiKey } = await import("../../src/lib/providers/validation.ts");
+
+test("#2929 route-restriction 403 on /models falls through to the chat probe (valid)", async () => {
+  const originalFetch = globalThis.fetch;
+  let callCount = 0;
+  globalThis.fetch = (async () => {
+    callCount += 1;
+    if (callCount === 1) {
+      // models endpoint: route-restriction 403
+      return new Response("Fire Pass API keys are not authorized for this route.", {
+        status: 403,
+      });
+    }
+    // chat probe succeeds
+    return new Response(JSON.stringify({ choices: [{ message: { content: "ok" } }] }), {
+      status: 200,
+    });
+  }) as typeof fetch;
+
+  try {
+    const result = await validateProviderApiKey({
+      provider: "fireworks",
+      apiKey: "fpk_test",
+      providerSpecificData: {},
+    });
+    assert.equal(result.valid, true, "Fire Pass key valid for chat must validate via the chat probe");
+    assert.equal(callCount, 2, "the chat probe must run after the route-restriction 403 on /models");
+  } finally {
+    globalThis.fetch = originalFetch;
+  }
+});
+
+test("#2929 a generic 403 on /models is still 'Invalid API key' (no chat probe)", async () => {
+  const originalFetch = globalThis.fetch;
+  let callCount = 0;
+  globalThis.fetch = (async () => {
+    callCount += 1;
+    return new Response(JSON.stringify({ error: "invalid api key" }), { status: 403 });
+  }) as typeof fetch;
+
+  try {
+    const result = await validateProviderApiKey({
+      provider: "fireworks",
+      apiKey: "bad-key",
+      providerSpecificData: {},
+    });
+    assert.equal(result.valid, false);
+    assert.equal(result.error, "Invalid API key");
+    assert.equal(callCount, 1, "a non-route-restriction 403 must short-circuit without a chat probe");
+  } finally {
+    globalThis.fetch = originalFetch;
+  }
+});