From dd70dbdaa0e7f304f57d4f12c7c4b04389112077 Mon Sep 17 00:00:00 2001 From: Innokentiy Solntsev Date: Thu, 17 Sep 2026 21:26:08 +0200 Subject: [PATCH] =?UTF-8?q?fix(sse):=20Anthropic=20OAuth=20403=20"Request?= =?UTF-8?q?=20not=20allowed"=20is=20a=20per-request=20refusal=20=E2=80=94?= =?UTF-8?q?=20cooldown=20with=20backoff=20instead=20of=20an=20instant=20ba?= =?UTF-8?q?n=20(#12859)=20(#12864)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(sse): Anthropic OAuth 403 "Request not allowed" is a per-request refusal, not a ban A single upstream 403 on the `claude` OAuth connection was classified FORBIDDEN and written as the terminal `banned` connection state (chatCore -> writeTerminalStatus). From then on every request to that provider was short-circuited with "All 1 connection(s) banned by upstream - please reconnect in the dashboard" without touching Anthropic, until an operator reconnected. Anthropic's OAuth surface answers a small fraction of otherwise-valid requests with 403 {"type":"permission_error","message":"Request not allowed"}. On the reporting install the same token returned 200 forty seconds before the 403 and again right after the connection was re-enabled; a revoked or expired token is a 401 authentication_error, not this. It is a refusal of one request, not of the credential. Classify it as the new non-terminal PROVIDER_ERROR_TYPES.REQUEST_REJECTED (scoped to provider `claude` and the "Request not allowed" body) and list that type in authTerminalStatus.isNonTerminalProviderError, mirroring the Cloudflare FINGERPRINT_REJECTION precedent. The combo layer still falls through to the next target for the failing request; the connection stays active for the next one. Any other claude 403 keeps its previous classification. Tests: error-classifier.test.ts covers the Anthropic body, the gateway-flattened "[403]: Request not allowed" message, the same body from a non-Anthropic provider (still FORBIDDEN), other claude 403s (unchanged), and the helpers; anthropic-request-not-allowed-not-a-ban.test.ts pins resolveTerminalConnectionStatus() -> null for the new type even with a `permanent` fallback verdict, and `banned` for a generic claude 403. * fix(sse): cooldown with backoff and streak escalation for REQUEST_REJECTED (#12859) Not "ignore the 403" either: if Anthropic ever made "Request not allowed" systematic, re-sending every request into it would be the wrong thing to do to an OAuth account. chatCore now handles REQUEST_REJECTED explicitly: - exclude the connection via setConnectionRateLimitUntil for a growing cooldown (5 -> 15 -> 45 min) so a sporadic refusal costs minutes, not a reconnect, and a systematic one cannot become a stream of 403s; - escalate to the terminal `banned` state only for 3 refusals within a 60-minute window (services/requestRejectedStreak.ts, in-memory per connection; a restart forgets the streak, erring towards more cooldowns rather than an operator-undone ban), with a last_error that says so; - probe-origin failures record but never cool down or ban (#9817). The existing "request not allowed" text rule (5 s) is unaffected: markAccountUnavailable skips a connection that already has a future rateLimitedUntil, so the minute-scale cooldown written here wins. Tests: request-rejected-streak.test.ts pins the window/threshold/backoff arithmetic; anthropic-request-not-allowed-cooldown-escalation.test.ts drives the real chat route against a mocked 403 upstream on a `claude` OAuth connection: 300 s cooldown, then 900 s, then banned on the third refusal; a different claude 403 body still bans on the first response. * chore(changelog): name the #12859 fragment after its PR (#12864) * refactor(sse): move the REQUEST_REJECTED branch into a chatCore leaf; register its tests for mutation coverage chatCore.ts is frozen at 5984 lines by the file-size ratchet; the branch body now lives in open-sse/handlers/chatCore/requestRejectedFailure.ts (chatCore: 5974 -> 5983). stryker.conf.json tap.testFiles gains the two new DB-backed tests so their mutant kills count (check:mutation-test-coverage). * fix(sse): count refusal episodes, reset on success, keep the dashboard honest (#12859 review) Review findings on the first cut of the REQUEST_REJECTED handling: - A burst of in-flight requests that all got the 403 within seconds produced streak 1, 2, 3 and a ban from one upstream event. The streak now counts cooldown *episodes*: a refusal that lands while the connection is already excluded is the same event and is not counted. - Nothing reset the streak on a healthy response, so sporadic refusals on a busy install could still accumulate to a ban. chatHelpers' onRequestSuccess now clears it (only a real success does - the recovery tick's clearAccountError is an elapsed cooldown, not a success). Clearing the cooldown by hand in the dashboard clears it too. - The third rung of the ladder was unreachable (the third refusal escalates): the ladder is now 5 -> 15 min, sourced from COOLDOWN_MS next to the existing 5 s "request not allowed" rule, with a note on why that rule is superseded for claude. The 60-min window becomes a 24 h staleness bound - "consecutive" is defined by successes, not by time. - Probe-origin refusals no longer touch the streak (#9817). - The cooldown is written like every other connection-level cooldown: ISO rateLimitedUntil + testStatus "unavailable" (+ lastErrorAt), so the dashboard shows the countdown and the recovery tick restores "active". - One refusal is re-seeded from the persisted row after a restart so a crash loop cannot reset the count on every boot. Docs: RESILIENCE_GUIDE terminal states + CODEBASE_DOCUMENTATION resilience row mention the streak module. Tests cover the burst, the success reset, the seed, and the ISO/unavailable shape end-to-end through the chat route. * chore(sse): drop unrelated Prettier churn in auth.ts / providers route * style(api): keep providers route Prettier-clean * refactor(sse): share the "exclude connection for a cooldown" leaf between GEO_BLOCKED, GCP_PROJECT_REQUIRED and the new branch The release tip moved chatCore.ts to its frozen 5984 lines, so the REQUEST_REJECTED branch cannot add a single net line. The GEO_BLOCKED and GCP_PROJECT_REQUIRED branches were the same eight statements with different constants and log wording; both now call open-sse/handlers/chatCore/connectionCooldown.ts::excludeConnectionForCooldown (behaviour, probe guard and log lines preserved verbatim). chatCore.ts ends 9 lines below the base it branched from. * chore(chatCore): tighten the cooldown comments to keep the file under its size ceiling After merging release/v3.8.51, chatCore.ts sat at 6150 lines against a frozen ceiling of 6146. Condense the explanatory comments this PR added to the GEO_BLOCKED and GCP_PROJECT_REQUIRED branches; no code change. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> --------- Co-authored-by: insoln Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> --- ...anthropic-request-not-allowed-not-a-ban.md | 1 + docs/architecture/CODEBASE_DOCUMENTATION.md | 2 +- docs/architecture/RESILIENCE_GUIDE.md | 2 +- open-sse/config/errorConfig.ts | 10 + open-sse/handlers/chatCore.ts | 56 ++-- .../handlers/chatCore/connectionCooldown.ts | 42 +++ .../chatCore/requestRejectedFailure.ts | 102 ++++++++ open-sse/services/errorClassifier.ts | 36 +++ open-sse/services/requestRejectedStreak.ts | 140 ++++++++++ src/app/api/providers/[id]/route.ts | 6 +- src/sse/handlers/chatHelpers.ts | 4 + src/sse/services/authTerminalStatus.ts | 5 +- stryker.conf.json | 2 + ...st-not-allowed-cooldown-escalation.test.ts | 241 ++++++++++++++++++ ...opic-request-not-allowed-not-a-ban.test.ts | 56 ++++ tests/unit/error-classifier.test.ts | 55 ++++ tests/unit/request-rejected-streak.test.ts | 116 +++++++++ 17 files changed, 846 insertions(+), 30 deletions(-) create mode 100644 changelog.d/fixes/12864-anthropic-request-not-allowed-not-a-ban.md create mode 100644 open-sse/handlers/chatCore/connectionCooldown.ts create mode 100644 open-sse/handlers/chatCore/requestRejectedFailure.ts create mode 100644 open-sse/services/requestRejectedStreak.ts create mode 100644 tests/unit/anthropic-request-not-allowed-cooldown-escalation.test.ts create mode 100644 tests/unit/anthropic-request-not-allowed-not-a-ban.test.ts create mode 100644 tests/unit/request-rejected-streak.test.ts diff --git a/changelog.d/fixes/12864-anthropic-request-not-allowed-not-a-ban.md b/changelog.d/fixes/12864-anthropic-request-not-allowed-not-a-ban.md new file mode 100644 index 0000000000..41f9af91db --- /dev/null +++ b/changelog.d/fixes/12864-anthropic-request-not-allowed-not-a-ban.md @@ -0,0 +1 @@ +- **fix(sse):** An Anthropic OAuth `403 "Request not allowed"` no longer bans the Claude connection on the first response — it is a per-request refusal on an otherwise healthy token, so it is now classified as the non-terminal `request_rejected` type, the connection is excluded for a growing cooldown (5 min, then 15 min) and only three consecutive refusals with no success in between escalate to `banned`; previously a single such response flipped the only Claude connection to `banned` and every later request was short-circuited with "All 1 connection(s) banned by upstream" until an operator reconnected ([#12859](https://github.com/diegosouzapw/OmniRoute/issues/12859), [#12864](https://github.com/diegosouzapw/OmniRoute/pull/12864) — thanks @insoln) diff --git a/docs/architecture/CODEBASE_DOCUMENTATION.md b/docs/architecture/CODEBASE_DOCUMENTATION.md index 0f60d2fdcc..a3df7cc0e1 100644 --- a/docs/architecture/CODEBASE_DOCUMENTATION.md +++ b/docs/architecture/CODEBASE_DOCUMENTATION.md @@ -523,7 +523,7 @@ Highlights (full list under `open-sse/services/`): | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Combo routing | `combo.ts` (19 strategies), `comboConfig.ts`, `comboMetrics.ts`, `comboManifestMetrics.ts`, `comboAgentMiddleware.ts` | | Auto Combo engine | `autoCombo/` — `engine.ts`, `scoring.ts`, `taskFitness.ts`, `virtualFactory.ts`, `modePacks.ts`, `autoPrefix.ts`, `persistence.ts`, `providerDiversity.ts`, `providerRegistryAccessor.ts`, `routerStrategy.ts`, `selfHealing.ts`, `index.ts` | -| Resilience | `accountFallback.ts` (cooldown + lockout), `errorClassifier.ts`, `emergencyFallback.ts`, `rateLimitManager.ts`, `rateLimitSemaphore.ts`, `accountSemaphore.ts`, `accountSelector.ts` | +| Resilience | `accountFallback.ts` (cooldown + lockout), `errorClassifier.ts`, `requestRejectedStreak.ts`, `emergencyFallback.ts`, `rateLimitManager.ts`, `rateLimitSemaphore.ts`, `accountSemaphore.ts`, `accountSelector.ts` | | Quotas | `quotaMonitor.ts`, `quotaPreflight.ts`, `bailianQuotaFetcher.ts`, `codexQuotaFetcher.ts`, `deepseekQuotaFetcher.ts`, `openrouterQuotaFetcher.ts`, `openrouterFreeWindow.ts`, `crofUsageFetcher.ts`, `antigravityCredits.ts` | | Caching | `reasoningCache.ts`, `searchCache.ts`, `signatureCache.ts`, `requestDedup.ts` | | Routing intelligence | `intentClassifier.ts`, `taskAwareRouter.ts`, `backgroundTaskDetector.ts`, `volumeDetector.ts`, `wildcardRouter.ts`, `workflowFSM.ts`, `specificityDetector.ts`, `specificityRules.ts`, `specificityTypes.ts` | diff --git a/docs/architecture/RESILIENCE_GUIDE.md b/docs/architecture/RESILIENCE_GUIDE.md index cc49c443b2..7ce95a00e2 100644 --- a/docs/architecture/RESILIENCE_GUIDE.md +++ b/docs/architecture/RESILIENCE_GUIDE.md @@ -100,7 +100,7 @@ Regression guard: `tests/unit/provider-cooldown-window-gate.test.ts`. **Terminal states (NOT cooldowns):** -- `banned` — set by banned-keyword / account-ban detection (see [BAN_DETECTION](../security/BAN_DETECTION.md)) +- `banned` — set by banned-keyword / account-ban detection (see [BAN_DETECTION](../security/BAN_DETECTION.md)), and by three consecutive upstream per-request refusals (`request_rejected`, e.g. Anthropic OAuth 403 "Request not allowed" — `open-sse/services/requestRejectedStreak.ts`); a single refusal only cools the connection down - `expired` (transitions to terminal after bounded retries — `EXPIRED_RETRY_MAX = 3` with exponential backoff — so transient OAuth errors can self-heal before the account is permanently deactivated) - `credits_exhausted` diff --git a/open-sse/config/errorConfig.ts b/open-sse/config/errorConfig.ts index 7e7355948f..6464a6eab1 100644 --- a/open-sse/config/errorConfig.ts +++ b/open-sse/config/errorConfig.ts @@ -74,6 +74,12 @@ export const COOLDOWN_MS = { transientMax: 60 * 1000, transient: TRANSIENT_COOLDOWN_MS, requestNotAllowed: 5 * 1000, + // Anthropic OAuth 403 "Request not allowed" (#12859): a per-request refusal + // on a healthy token. chatCore excludes the connection for requestRejected + // after the first refusal, requestRejectedRepeat after the second, and bans + // it on the third consecutive one (services/requestRejectedStreak.ts). + requestRejected: 5 * 60 * 1000, + requestRejectedRepeat: 15 * 60 * 1000, rateLimit: 2 * 60 * 1000, serviceUnavailable: 2 * 1000, authExpired: 2 * 60 * 1000, @@ -99,6 +105,10 @@ export const ERROR_RULES: ErrorRule[] = [ reason: "auth_error", }, { + // For provider `claude` this text is classified REQUEST_REJECTED and the + // connection-level cooldown is written by chatCore before the fallback + // layer runs (#12859); markAccountUnavailable then keeps the longer + // cooldown. This 5 s rule still serves every other provider. id: "request_not_allowed", text: "request not allowed", cooldownMs: COOLDOWN_MS.requestNotAllowed, diff --git a/open-sse/handlers/chatCore.ts b/open-sse/handlers/chatCore.ts index 506f14f58e..e1570743f0 100644 --- a/open-sse/handlers/chatCore.ts +++ b/open-sse/handlers/chatCore.ts @@ -275,6 +275,8 @@ import { stageTrace } from "./chatCore/stageTrace.ts"; import { attachCompressionUsageReceiptAfterAnalytics as attachCompressionUsageReceiptAfterAnalyticsFor } from "./chatCore/compressionUsageReceipt.ts"; import { prepareUpstreamBody } from "./chatCore/upstreamBody.ts"; import { getQuotaScopeLabelForProvider } from "../services/antigravityQuotaFamily.ts"; +import { excludeConnectionForCooldown } from "./chatCore/connectionCooldown.ts"; +import { handleRequestRejectedFailure } from "./chatCore/requestRejectedFailure.ts"; import { getKimiTemporaryRateLimitResetAt } from "./chatCore/kimiQuotaRecovery.ts"; import { getCallLogPipelineCaptureStreamChunks, @@ -4049,35 +4051,37 @@ export async function handleChatCore({ `[provider] Node ${errorConnectionId} project routing error (${statusCode}) -- not banning` ); } else if (errorType === PROVIDER_ERROR_TYPES.GEO_BLOCKED) { - const geoCooldownMs = COOLDOWN_MS.geoBlocked ?? 24 * 60 * 60 * 1000; - await updateProviderConnection(errorConnectionId, { - lastErrorType: errorType, - lastError: persistentMessage, - errorCode: statusCode, + // Google regional refusal: account-independent, non-terminal; park the connection + // until egress uses a supported region; probes skip the day-long cooldown (#9817). + await excludeConnectionForCooldown({ + connectionId: errorConnectionId, + errorType, + message: persistentMessage, + statusCode, + cooldownMs: COOLDOWN_MS.geoBlocked ?? 24 * 60 * 60 * 1000, + skipCooldownForProbe: true, + label: "geo-blocked", + suffix: "trying other accounts", + }); + } else if (errorType === PROVIDER_ERROR_TYPES.REQUEST_REJECTED) { + // Per-request refusal (#12859): growing cooldown, streak → banned. + await handleRequestRejectedFailure({ + connectionId: errorConnectionId, + statusCode, + message: persistentMessage, }); - if (!(await shouldIsolateProbeFailures())) { - try { - const { setConnectionRateLimitUntil } = await import("@/lib/db/providers"); - setConnectionRateLimitUntil(errorConnectionId, Date.now() + geoCooldownMs); - } catch {} - } - console.warn( - `[provider] Node ${errorConnectionId} geo-blocked (${statusCode}) -- excluded for ${Math.ceil(geoCooldownMs / 1000)}s, trying other accounts` - ); } else if (errorType === PROVIDER_ERROR_TYPES.GCP_PROJECT_REQUIRED) { - const byopCooldownMs = COOLDOWN_MS.gcpProjectRequired ?? 24 * 60 * 60 * 1000; - await updateProviderConnection(errorConnectionId, { - lastErrorType: errorType, - lastError: persistentMessage, - errorCode: statusCode, + // Antigravity BYOP: fixable via a Project ID; never a lockout/ban. Park the connection. + await excludeConnectionForCooldown({ + connectionId: errorConnectionId, + errorType, + message: persistentMessage, + statusCode, + cooldownMs: COOLDOWN_MS.gcpProjectRequired ?? 24 * 60 * 60 * 1000, + skipCooldownForProbe: false, + label: "GCP project required", + suffix: "routing to other accounts (enter a Project ID to restore)", }); - try { - const { setConnectionRateLimitUntil } = await import("@/lib/db/providers"); - setConnectionRateLimitUntil(errorConnectionId, Date.now() + byopCooldownMs); - } catch {} - console.warn( - `[provider] Node ${errorConnectionId} GCP project required (${statusCode}) -- excluded for ${Math.ceil(byopCooldownMs / 1000)}s, routing to other accounts (enter a Project ID to restore)` - ); } else if (errorType === PROVIDER_ERROR_TYPES.MODEL_NOT_FOUND) { const notFoundCooldownMs = COOLDOWN_MS.notFound; if (!(await shouldIsolateProbeFailures())) { diff --git a/open-sse/handlers/chatCore/connectionCooldown.ts b/open-sse/handlers/chatCore/connectionCooldown.ts new file mode 100644 index 0000000000..fe007ed7e4 --- /dev/null +++ b/open-sse/handlers/chatCore/connectionCooldown.ts @@ -0,0 +1,42 @@ +import { updateProviderConnection } from "@/lib/db/providers"; +import { shouldIsolateProbeFailures } from "@/shared/utils/probeOrigin"; + +/** + * chatCore leaf for the non-terminal "exclude this connection for a while" + * outcomes (GEO_BLOCKED, GCP_PROJECT_REQUIRED): record the error on the + * connection and park it via rateLimitedUntil so selection prefers sibling + * accounts, without ever marking it banned/expired. + */ +export async function excludeConnectionForCooldown(params: { + connectionId: string; + errorType: string; + message: string; + statusCode: number; + cooldownMs: number; + /** + * T-PROBE (#9817): when true, a probe-origin failure records the error but + * does not push the connection into the cooldown (routing state untouched). + */ + skipCooldownForProbe: boolean; + /** Log wording: `[provider] Node