mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-09-21 06:12:17 +03:00
fix(sse): Anthropic OAuth 403 "Request not allowed" is a per-request refusal — cooldown with backoff instead of an instant ban (#12859) (#12864)
* fix(sse): Anthropic OAuth 403 "Request not allowed" is a per-request refusal, not a ban
A single upstream 403 on the `claude` OAuth connection was classified
FORBIDDEN and written as the terminal `banned` connection state
(chatCore -> writeTerminalStatus). From then on every request to that
provider was short-circuited with "All 1 connection(s) banned by
upstream - please reconnect in the dashboard" without touching Anthropic,
until an operator reconnected.
Anthropic's OAuth surface answers a small fraction of otherwise-valid
requests with 403 {"type":"permission_error","message":"Request not
allowed"}. On the reporting install the same token returned 200 forty
seconds before the 403 and again right after the connection was
re-enabled; a revoked or expired token is a 401 authentication_error, not
this. It is a refusal of one request, not of the credential.
Classify it as the new non-terminal PROVIDER_ERROR_TYPES.REQUEST_REJECTED
(scoped to provider `claude` and the "Request not allowed" body) and list
that type in authTerminalStatus.isNonTerminalProviderError, mirroring the
Cloudflare FINGERPRINT_REJECTION precedent. The combo layer still falls
through to the next target for the failing request; the connection stays
active for the next one. Any other claude 403 keeps its previous
classification.
Tests: error-classifier.test.ts covers the Anthropic body, the
gateway-flattened "[403]: Request not allowed" message, the same body from
a non-Anthropic provider (still FORBIDDEN), other claude 403s (unchanged),
and the helpers; anthropic-request-not-allowed-not-a-ban.test.ts pins
resolveTerminalConnectionStatus() -> null for the new type even with a
`permanent` fallback verdict, and `banned` for a generic claude 403.
* fix(sse): cooldown with backoff and streak escalation for REQUEST_REJECTED (#12859)
Not "ignore the 403" either: if Anthropic ever made "Request not allowed"
systematic, re-sending every request into it would be the wrong thing to
do to an OAuth account. chatCore now handles REQUEST_REJECTED explicitly:
- exclude the connection via setConnectionRateLimitUntil for a growing
cooldown (5 -> 15 -> 45 min) so a sporadic refusal costs minutes, not a
reconnect, and a systematic one cannot become a stream of 403s;
- escalate to the terminal `banned` state only for 3 refusals within a
60-minute window (services/requestRejectedStreak.ts, in-memory per
connection; a restart forgets the streak, erring towards more cooldowns
rather than an operator-undone ban), with a last_error that says so;
- probe-origin failures record but never cool down or ban (#9817).
The existing "request not allowed" text rule (5 s) is unaffected:
markAccountUnavailable skips a connection that already has a future
rateLimitedUntil, so the minute-scale cooldown written here wins.
Tests: request-rejected-streak.test.ts pins the window/threshold/backoff
arithmetic; anthropic-request-not-allowed-cooldown-escalation.test.ts drives
the real chat route against a mocked 403 upstream on a `claude` OAuth
connection: 300 s cooldown, then 900 s, then banned on the third refusal;
a different claude 403 body still bans on the first response.
* chore(changelog): name the #12859 fragment after its PR (#12864)
* refactor(sse): move the REQUEST_REJECTED branch into a chatCore leaf; register its tests for mutation coverage
chatCore.ts is frozen at 5984 lines by the file-size ratchet; the branch
body now lives in open-sse/handlers/chatCore/requestRejectedFailure.ts
(chatCore: 5974 -> 5983). stryker.conf.json tap.testFiles gains the two new
DB-backed tests so their mutant kills count (check:mutation-test-coverage).
* fix(sse): count refusal episodes, reset on success, keep the dashboard honest (#12859 review)
Review findings on the first cut of the REQUEST_REJECTED handling:
- A burst of in-flight requests that all got the 403 within seconds
produced streak 1, 2, 3 and a ban from one upstream event. The streak
now counts cooldown *episodes*: a refusal that lands while the
connection is already excluded is the same event and is not counted.
- Nothing reset the streak on a healthy response, so sporadic refusals
on a busy install could still accumulate to a ban. chatHelpers'
onRequestSuccess now clears it (only a real success does - the recovery
tick's clearAccountError is an elapsed cooldown, not a success).
Clearing the cooldown by hand in the dashboard clears it too.
- The third rung of the ladder was unreachable (the third refusal
escalates): the ladder is now 5 -> 15 min, sourced from COOLDOWN_MS next
to the existing 5 s "request not allowed" rule, with a note on why that
rule is superseded for claude. The 60-min window becomes a 24 h
staleness bound - "consecutive" is defined by successes, not by time.
- Probe-origin refusals no longer touch the streak (#9817).
- The cooldown is written like every other connection-level cooldown:
ISO rateLimitedUntil + testStatus "unavailable" (+ lastErrorAt), so the
dashboard shows the countdown and the recovery tick restores "active".
- One refusal is re-seeded from the persisted row after a restart so a
crash loop cannot reset the count on every boot.
Docs: RESILIENCE_GUIDE terminal states + CODEBASE_DOCUMENTATION resilience
row mention the streak module. Tests cover the burst, the success reset,
the seed, and the ISO/unavailable shape end-to-end through the chat route.
* chore(sse): drop unrelated Prettier churn in auth.ts / providers route
* style(api): keep providers route Prettier-clean
* refactor(sse): share the "exclude connection for a cooldown" leaf between GEO_BLOCKED, GCP_PROJECT_REQUIRED and the new branch
The release tip moved chatCore.ts to its frozen 5984 lines, so the
REQUEST_REJECTED branch cannot add a single net line. The GEO_BLOCKED and
GCP_PROJECT_REQUIRED branches were the same eight statements with different
constants and log wording; both now call
open-sse/handlers/chatCore/connectionCooldown.ts::excludeConnectionForCooldown
(behaviour, probe guard and log lines preserved verbatim). chatCore.ts ends
9 lines below the base it branched from.
* chore(chatCore): tighten the cooldown comments to keep the file under its size ceiling
After merging release/v3.8.51, chatCore.ts sat at 6150 lines against a
frozen ceiling of 6146. Condense the explanatory comments this PR added
to the GEO_BLOCKED and GCP_PROJECT_REQUIRED branches; no code change.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: insoln <is@careerum.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
This commit is contained in:
committed by
GitHub
parent
47159ed56b
commit
dd70dbdaa0
@@ -0,0 +1 @@
|
||||
- **fix(sse):** An Anthropic OAuth `403 "Request not allowed"` no longer bans the Claude connection on the first response — it is a per-request refusal on an otherwise healthy token, so it is now classified as the non-terminal `request_rejected` type, the connection is excluded for a growing cooldown (5 min, then 15 min) and only three consecutive refusals with no success in between escalate to `banned`; previously a single such response flipped the only Claude connection to `banned` and every later request was short-circuited with "All 1 connection(s) banned by upstream" until an operator reconnected ([#12859](https://github.com/diegosouzapw/OmniRoute/issues/12859), [#12864](https://github.com/diegosouzapw/OmniRoute/pull/12864) — thanks @insoln)
|
||||
@@ -523,7 +523,7 @@ Highlights (full list under `open-sse/services/`):
|
||||
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Combo routing | `combo.ts` (19 strategies), `comboConfig.ts`, `comboMetrics.ts`, `comboManifestMetrics.ts`, `comboAgentMiddleware.ts` |
|
||||
| Auto Combo engine | `autoCombo/` — `engine.ts`, `scoring.ts`, `taskFitness.ts`, `virtualFactory.ts`, `modePacks.ts`, `autoPrefix.ts`, `persistence.ts`, `providerDiversity.ts`, `providerRegistryAccessor.ts`, `routerStrategy.ts`, `selfHealing.ts`, `index.ts` |
|
||||
| Resilience | `accountFallback.ts` (cooldown + lockout), `errorClassifier.ts`, `emergencyFallback.ts`, `rateLimitManager.ts`, `rateLimitSemaphore.ts`, `accountSemaphore.ts`, `accountSelector.ts` |
|
||||
| Resilience | `accountFallback.ts` (cooldown + lockout), `errorClassifier.ts`, `requestRejectedStreak.ts`, `emergencyFallback.ts`, `rateLimitManager.ts`, `rateLimitSemaphore.ts`, `accountSemaphore.ts`, `accountSelector.ts` |
|
||||
| Quotas | `quotaMonitor.ts`, `quotaPreflight.ts`, `bailianQuotaFetcher.ts`, `codexQuotaFetcher.ts`, `deepseekQuotaFetcher.ts`, `openrouterQuotaFetcher.ts`, `openrouterFreeWindow.ts`, `crofUsageFetcher.ts`, `antigravityCredits.ts` |
|
||||
| Caching | `reasoningCache.ts`, `searchCache.ts`, `signatureCache.ts`, `requestDedup.ts` |
|
||||
| Routing intelligence | `intentClassifier.ts`, `taskAwareRouter.ts`, `backgroundTaskDetector.ts`, `volumeDetector.ts`, `wildcardRouter.ts`, `workflowFSM.ts`, `specificityDetector.ts`, `specificityRules.ts`, `specificityTypes.ts` |
|
||||
|
||||
@@ -100,7 +100,7 @@ Regression guard: `tests/unit/provider-cooldown-window-gate.test.ts`.
|
||||
|
||||
**Terminal states (NOT cooldowns):**
|
||||
|
||||
- `banned` — set by banned-keyword / account-ban detection (see [BAN_DETECTION](../security/BAN_DETECTION.md))
|
||||
- `banned` — set by banned-keyword / account-ban detection (see [BAN_DETECTION](../security/BAN_DETECTION.md)), and by three consecutive upstream per-request refusals (`request_rejected`, e.g. Anthropic OAuth 403 "Request not allowed" — `open-sse/services/requestRejectedStreak.ts`); a single refusal only cools the connection down
|
||||
- `expired` (transitions to terminal after bounded retries — `EXPIRED_RETRY_MAX = 3` with exponential backoff — so transient OAuth errors can self-heal before the account is permanently deactivated)
|
||||
- `credits_exhausted`
|
||||
|
||||
|
||||
@@ -74,6 +74,12 @@ export const COOLDOWN_MS = {
|
||||
transientMax: 60 * 1000,
|
||||
transient: TRANSIENT_COOLDOWN_MS,
|
||||
requestNotAllowed: 5 * 1000,
|
||||
// Anthropic OAuth 403 "Request not allowed" (#12859): a per-request refusal
|
||||
// on a healthy token. chatCore excludes the connection for requestRejected
|
||||
// after the first refusal, requestRejectedRepeat after the second, and bans
|
||||
// it on the third consecutive one (services/requestRejectedStreak.ts).
|
||||
requestRejected: 5 * 60 * 1000,
|
||||
requestRejectedRepeat: 15 * 60 * 1000,
|
||||
rateLimit: 2 * 60 * 1000,
|
||||
serviceUnavailable: 2 * 1000,
|
||||
authExpired: 2 * 60 * 1000,
|
||||
@@ -99,6 +105,10 @@ export const ERROR_RULES: ErrorRule[] = [
|
||||
reason: "auth_error",
|
||||
},
|
||||
{
|
||||
// For provider `claude` this text is classified REQUEST_REJECTED and the
|
||||
// connection-level cooldown is written by chatCore before the fallback
|
||||
// layer runs (#12859); markAccountUnavailable then keeps the longer
|
||||
// cooldown. This 5 s rule still serves every other provider.
|
||||
id: "request_not_allowed",
|
||||
text: "request not allowed",
|
||||
cooldownMs: COOLDOWN_MS.requestNotAllowed,
|
||||
|
||||
@@ -275,6 +275,8 @@ import { stageTrace } from "./chatCore/stageTrace.ts";
|
||||
import { attachCompressionUsageReceiptAfterAnalytics as attachCompressionUsageReceiptAfterAnalyticsFor } from "./chatCore/compressionUsageReceipt.ts";
|
||||
import { prepareUpstreamBody } from "./chatCore/upstreamBody.ts";
|
||||
import { getQuotaScopeLabelForProvider } from "../services/antigravityQuotaFamily.ts";
|
||||
import { excludeConnectionForCooldown } from "./chatCore/connectionCooldown.ts";
|
||||
import { handleRequestRejectedFailure } from "./chatCore/requestRejectedFailure.ts";
|
||||
import { getKimiTemporaryRateLimitResetAt } from "./chatCore/kimiQuotaRecovery.ts";
|
||||
import {
|
||||
getCallLogPipelineCaptureStreamChunks,
|
||||
@@ -4049,35 +4051,37 @@ export async function handleChatCore({
|
||||
`[provider] Node ${errorConnectionId} project routing error (${statusCode}) -- not banning`
|
||||
);
|
||||
} else if (errorType === PROVIDER_ERROR_TYPES.GEO_BLOCKED) {
|
||||
const geoCooldownMs = COOLDOWN_MS.geoBlocked ?? 24 * 60 * 60 * 1000;
|
||||
await updateProviderConnection(errorConnectionId, {
|
||||
lastErrorType: errorType,
|
||||
lastError: persistentMessage,
|
||||
errorCode: statusCode,
|
||||
// Google regional refusal: account-independent, non-terminal; park the connection
|
||||
// until egress uses a supported region; probes skip the day-long cooldown (#9817).
|
||||
await excludeConnectionForCooldown({
|
||||
connectionId: errorConnectionId,
|
||||
errorType,
|
||||
message: persistentMessage,
|
||||
statusCode,
|
||||
cooldownMs: COOLDOWN_MS.geoBlocked ?? 24 * 60 * 60 * 1000,
|
||||
skipCooldownForProbe: true,
|
||||
label: "geo-blocked",
|
||||
suffix: "trying other accounts",
|
||||
});
|
||||
} else if (errorType === PROVIDER_ERROR_TYPES.REQUEST_REJECTED) {
|
||||
// Per-request refusal (#12859): growing cooldown, streak → banned.
|
||||
await handleRequestRejectedFailure({
|
||||
connectionId: errorConnectionId,
|
||||
statusCode,
|
||||
message: persistentMessage,
|
||||
});
|
||||
if (!(await shouldIsolateProbeFailures())) {
|
||||
try {
|
||||
const { setConnectionRateLimitUntil } = await import("@/lib/db/providers");
|
||||
setConnectionRateLimitUntil(errorConnectionId, Date.now() + geoCooldownMs);
|
||||
} catch {}
|
||||
}
|
||||
console.warn(
|
||||
`[provider] Node ${errorConnectionId} geo-blocked (${statusCode}) -- excluded for ${Math.ceil(geoCooldownMs / 1000)}s, trying other accounts`
|
||||
);
|
||||
} else if (errorType === PROVIDER_ERROR_TYPES.GCP_PROJECT_REQUIRED) {
|
||||
const byopCooldownMs = COOLDOWN_MS.gcpProjectRequired ?? 24 * 60 * 60 * 1000;
|
||||
await updateProviderConnection(errorConnectionId, {
|
||||
lastErrorType: errorType,
|
||||
lastError: persistentMessage,
|
||||
errorCode: statusCode,
|
||||
// Antigravity BYOP: fixable via a Project ID; never a lockout/ban. Park the connection.
|
||||
await excludeConnectionForCooldown({
|
||||
connectionId: errorConnectionId,
|
||||
errorType,
|
||||
message: persistentMessage,
|
||||
statusCode,
|
||||
cooldownMs: COOLDOWN_MS.gcpProjectRequired ?? 24 * 60 * 60 * 1000,
|
||||
skipCooldownForProbe: false,
|
||||
label: "GCP project required",
|
||||
suffix: "routing to other accounts (enter a Project ID to restore)",
|
||||
});
|
||||
try {
|
||||
const { setConnectionRateLimitUntil } = await import("@/lib/db/providers");
|
||||
setConnectionRateLimitUntil(errorConnectionId, Date.now() + byopCooldownMs);
|
||||
} catch {}
|
||||
console.warn(
|
||||
`[provider] Node ${errorConnectionId} GCP project required (${statusCode}) -- excluded for ${Math.ceil(byopCooldownMs / 1000)}s, routing to other accounts (enter a Project ID to restore)`
|
||||
);
|
||||
} else if (errorType === PROVIDER_ERROR_TYPES.MODEL_NOT_FOUND) {
|
||||
const notFoundCooldownMs = COOLDOWN_MS.notFound;
|
||||
if (!(await shouldIsolateProbeFailures())) {
|
||||
|
||||
42
open-sse/handlers/chatCore/connectionCooldown.ts
Normal file
42
open-sse/handlers/chatCore/connectionCooldown.ts
Normal file
@@ -0,0 +1,42 @@
|
||||
import { updateProviderConnection } from "@/lib/db/providers";
|
||||
import { shouldIsolateProbeFailures } from "@/shared/utils/probeOrigin";
|
||||
|
||||
/**
|
||||
* chatCore leaf for the non-terminal "exclude this connection for a while"
|
||||
* outcomes (GEO_BLOCKED, GCP_PROJECT_REQUIRED): record the error on the
|
||||
* connection and park it via rateLimitedUntil so selection prefers sibling
|
||||
* accounts, without ever marking it banned/expired.
|
||||
*/
|
||||
export async function excludeConnectionForCooldown(params: {
|
||||
connectionId: string;
|
||||
errorType: string;
|
||||
message: string;
|
||||
statusCode: number;
|
||||
cooldownMs: number;
|
||||
/**
|
||||
* T-PROBE (#9817): when true, a probe-origin failure records the error but
|
||||
* does not push the connection into the cooldown (routing state untouched).
|
||||
*/
|
||||
skipCooldownForProbe: boolean;
|
||||
/** Log wording: `[provider] Node <id> <label> (<status>) — excluded for <s>s, <suffix>` */
|
||||
label: string;
|
||||
suffix: string;
|
||||
}): Promise<void> {
|
||||
const { connectionId, errorType, message, statusCode, cooldownMs, label, suffix } = params;
|
||||
await updateProviderConnection(connectionId, {
|
||||
lastErrorType: errorType,
|
||||
lastError: message,
|
||||
errorCode: statusCode,
|
||||
});
|
||||
if (!(params.skipCooldownForProbe && (await shouldIsolateProbeFailures()))) {
|
||||
try {
|
||||
const { setConnectionRateLimitUntil } = await import("@/lib/db/providers");
|
||||
setConnectionRateLimitUntil(connectionId, Date.now() + cooldownMs);
|
||||
} catch {
|
||||
// DB write failure must never break the fallback loop
|
||||
}
|
||||
}
|
||||
console.warn(
|
||||
`[provider] Node ${connectionId} ${label} (${statusCode}) — excluded for ${Math.ceil(cooldownMs / 1000)}s, ${suffix}`
|
||||
);
|
||||
}
|
||||
102
open-sse/handlers/chatCore/requestRejectedFailure.ts
Normal file
102
open-sse/handlers/chatCore/requestRejectedFailure.ts
Normal file
@@ -0,0 +1,102 @@
|
||||
import { getProviderConnectionById, updateProviderConnection } from "@/lib/db/providers";
|
||||
import { shouldIsolateProbeFailures } from "@/shared/utils/probeOrigin";
|
||||
import { writeTerminalStatus } from "@/shared/utils/terminalStatus";
|
||||
|
||||
import { PROVIDER_ERROR_TYPES } from "../../services/errorClassifier.ts";
|
||||
import {
|
||||
hasRequestRejectedStreak,
|
||||
recordRequestRejected,
|
||||
seedRequestRejectedStreak,
|
||||
} from "../../services/requestRejectedStreak.ts";
|
||||
|
||||
/**
|
||||
* chatCore leaf for PROVIDER_ERROR_TYPES.REQUEST_REJECTED (#12859 — Anthropic
|
||||
* OAuth 403 "Request not allowed").
|
||||
*
|
||||
* The upstream refused THIS request, not the credential: the same token serves
|
||||
* the next request. One refusal must not ban the connection — but a run of
|
||||
* them is enforcement, and re-sending every request into it would be wrong.
|
||||
* So: exclude the connection for a short, growing cooldown and escalate to
|
||||
* `banned` only for consecutive refusals (services/requestRejectedStreak).
|
||||
*
|
||||
* Probe-origin failures (dashboard test-all) are recorded on the connection
|
||||
* but never touch the streak, cool it down or ban it (#9817).
|
||||
*/
|
||||
export async function handleRequestRejectedFailure(params: {
|
||||
connectionId: string;
|
||||
statusCode: number;
|
||||
message: string;
|
||||
}): Promise<void> {
|
||||
const { connectionId, statusCode, message } = params;
|
||||
const nowIso = new Date().toISOString();
|
||||
|
||||
if (await shouldIsolateProbeFailures()) {
|
||||
await updateProviderConnection(connectionId, {
|
||||
lastErrorType: PROVIDER_ERROR_TYPES.REQUEST_REJECTED,
|
||||
lastError: message,
|
||||
lastErrorAt: nowIso,
|
||||
errorCode: statusCode,
|
||||
});
|
||||
console.warn(
|
||||
`[provider] Node ${connectionId} probe refused by upstream (${statusCode}) — connection stays active`
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
// First refusal seen by this process: pick up one that happened before a
|
||||
// restart so a crash loop cannot reset the count on every boot.
|
||||
if (!hasRequestRejectedStreak(connectionId)) {
|
||||
try {
|
||||
const row = await getProviderConnectionById(connectionId);
|
||||
if (row?.lastErrorType === PROVIDER_ERROR_TYPES.REQUEST_REJECTED && row.lastErrorAt) {
|
||||
seedRequestRejectedStreak(connectionId, Date.parse(String(row.lastErrorAt)));
|
||||
}
|
||||
} catch {
|
||||
// best-effort — the in-memory streak still works without the seed
|
||||
}
|
||||
}
|
||||
|
||||
const verdict = recordRequestRejected(connectionId);
|
||||
const windowH = Math.round(verdict.windowMs / 3_600_000);
|
||||
|
||||
if (!verdict.counted) {
|
||||
console.warn(
|
||||
`[provider] Node ${connectionId} request refused by upstream (${statusCode}) while already excluded — same episode, not counted (refusal ${verdict.streak}/${verdict.threshold})`
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
if (verdict.escalate) {
|
||||
await writeTerminalStatus(
|
||||
connectionId,
|
||||
{
|
||||
testStatus: "banned",
|
||||
isActive: false,
|
||||
lastError: `${message} (${verdict.streak} consecutive refusals within ${windowH}h — treated as upstream enforcement)`,
|
||||
lastErrorType: PROVIDER_ERROR_TYPES.FORBIDDEN,
|
||||
errorCode: String(statusCode),
|
||||
},
|
||||
"production"
|
||||
);
|
||||
console.warn(
|
||||
`[provider] Node ${connectionId} refused ${verdict.streak}x in a row (${statusCode}) — disabling, reconnect required`
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
const until = new Date(Date.now() + verdict.cooldownMs).toISOString();
|
||||
// Same shape as the other connection-level cooldowns (testStatus
|
||||
// "unavailable" + ISO rateLimitedUntil): the dashboard shows the countdown
|
||||
// and the recovery tick restores "active" once the window has elapsed.
|
||||
await updateProviderConnection(connectionId, {
|
||||
testStatus: "unavailable",
|
||||
rateLimitedUntil: until,
|
||||
lastErrorType: PROVIDER_ERROR_TYPES.REQUEST_REJECTED,
|
||||
lastError: message,
|
||||
lastErrorAt: nowIso,
|
||||
errorCode: statusCode,
|
||||
});
|
||||
console.warn(
|
||||
`[provider] Node ${connectionId} request refused by upstream (${statusCode}) — excluded until ${until} (refusal ${verdict.streak}/${verdict.threshold} in a row), trying other accounts`
|
||||
);
|
||||
}
|
||||
@@ -90,6 +90,13 @@ export const PROVIDER_ERROR_TYPES = {
|
||||
// Google account must Bring Its Own GCP Project. Account-specific and
|
||||
// fixable by entering a Project ID — never a model lockout and never a ban.
|
||||
GCP_PROJECT_REQUIRED: "gcp_project_required",
|
||||
// The upstream refused THIS request (policy / request shape), not the
|
||||
// credential: the same connection serves the next request. Not terminal on
|
||||
// its own — chatCore excludes the connection for a growing cooldown and only
|
||||
// a streak of refusals escalates to `banned` (services/requestRejectedStreak).
|
||||
// First case: Anthropic's OAuth 403 "Request not allowed" (#12859), which
|
||||
// lands on a handful of requests between thousands of 200s on the same token.
|
||||
REQUEST_REJECTED: "request_rejected",
|
||||
} as const;
|
||||
|
||||
export type ProviderErrorType = (typeof PROVIDER_ERROR_TYPES)[keyof typeof PROVIDER_ERROR_TYPES];
|
||||
@@ -218,6 +225,25 @@ export function isCloudflareFingerprintRejection(errorText: string): boolean {
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Anthropic's OAuth (Claude subscription) surface answers a small fraction of
|
||||
* otherwise-valid requests with `403 {"type":"permission_error","message":
|
||||
* "Request not allowed"}`. Observed on one install: 200 on the same token 40 s
|
||||
* earlier, 200 on the next request after the connection was re-enabled — it is
|
||||
* a per-request refusal, not an account ban or a revoked token (a revoked token
|
||||
* is a 401 `authentication_error`). Classifying it FORBIDDEN flipped the only
|
||||
* Claude connection to the terminal `banned` state on a single response, and
|
||||
* every later request was short-circuited with "All 1 connection(s) banned by
|
||||
* upstream" until an operator reconnected in the dashboard.
|
||||
*/
|
||||
export function isAnthropicOAuthProvider(provider?: string | null): boolean {
|
||||
return String(provider || "").toLowerCase() === "claude";
|
||||
}
|
||||
|
||||
export function isAnthropicRequestNotAllowed(errorText: string): boolean {
|
||||
return /\brequest not allowed\b/i.test(String(errorText || ""));
|
||||
}
|
||||
|
||||
function responseBodyToString(responseBody: unknown): string {
|
||||
if (typeof responseBody === "string") return responseBody;
|
||||
if (responseBody !== null && typeof responseBody === "object") {
|
||||
@@ -355,6 +381,16 @@ export function classifyProviderError(
|
||||
if (statusCode === 403 && accountDeactivated) {
|
||||
return PROVIDER_ERROR_TYPES.ACCOUNT_DEACTIVATED;
|
||||
}
|
||||
if (
|
||||
statusCode === 403 &&
|
||||
isAnthropicOAuthProvider(provider) &&
|
||||
isAnthropicRequestNotAllowed(bodyStr)
|
||||
) {
|
||||
// Per-request refusal on an otherwise healthy Claude OAuth token — see
|
||||
// isAnthropicRequestNotAllowed. Must be checked BEFORE the generic 403 →
|
||||
// FORBIDDEN fall-through, which bans the connection permanently.
|
||||
return PROVIDER_ERROR_TYPES.REQUEST_REJECTED;
|
||||
}
|
||||
if (statusCode === 403) {
|
||||
// Cloud Code / Antigravity (Gemini Code Assist) 403s are almost always a
|
||||
// RECOVERABLE project-config issue — the Cloud AI Companion API not enabled
|
||||
|
||||
140
open-sse/services/requestRejectedStreak.ts
Normal file
140
open-sse/services/requestRejectedStreak.ts
Normal file
@@ -0,0 +1,140 @@
|
||||
import { COOLDOWN_MS } from "../config/errorConfig.ts";
|
||||
|
||||
/**
|
||||
* Per-connection streak of upstream per-request refusals (REQUEST_REJECTED —
|
||||
* today Anthropic's OAuth `403 "Request not allowed"`, #12859).
|
||||
*
|
||||
* A single refusal is a hiccup: the same token serves the next request. A run
|
||||
* of them is enforcement, and re-sending every request into it is the wrong
|
||||
* thing to do to an OAuth account. So the connection is excluded for a short,
|
||||
* growing cooldown after each refusal, and only a streak of *consecutive*
|
||||
* refusals escalates to the terminal `banned` state that used to fire on the
|
||||
* first one.
|
||||
*
|
||||
* What counts as one refusal: one cooldown *episode*. Responses that land while
|
||||
* the connection is already excluded were in flight before the cooldown was
|
||||
* written (parallel sessions, fan-out) — they are the same event, not new
|
||||
* evidence, and are ignored. A successful response on the connection clears
|
||||
* the streak (chatHelpers onRequestSuccess); the window below is only a
|
||||
* staleness bound so an ancient refusal can never be the first of a streak.
|
||||
*
|
||||
* State is in-memory on purpose. A restart forgets the streak, which errs on
|
||||
* the side of a few more cooldowns before escalation — never on the side of a
|
||||
* ban the operator has to undo by hand. The caller may re-seed one refusal
|
||||
* from the persisted row (`seedRequestRejectedStreak`) to bound a crash loop.
|
||||
*/
|
||||
|
||||
/** A refusal older than this cannot start or continue a streak. */
|
||||
export const REQUEST_REJECTED_WINDOW_MS = 24 * 60 * 60 * 1000;
|
||||
/** Consecutive refusals at which the connection is written as `banned`. */
|
||||
export const REQUEST_REJECTED_ESCALATION_THRESHOLD = 3;
|
||||
/** Cooldown after the 1st and 2nd refusal of a streak; the 3rd escalates. */
|
||||
export const REQUEST_REJECTED_COOLDOWNS_MS = [
|
||||
COOLDOWN_MS.requestRejected,
|
||||
COOLDOWN_MS.requestRejectedRepeat,
|
||||
];
|
||||
|
||||
export interface RequestRejectedVerdict {
|
||||
/** 1-based position of this refusal in the current streak. */
|
||||
streak: number;
|
||||
/** How long to exclude the connection from selection; 0 when escalating. */
|
||||
cooldownMs: number;
|
||||
/** True when the streak reached the threshold — caller writes the terminal state. */
|
||||
escalate: boolean;
|
||||
/**
|
||||
* False when the refusal landed while the connection was already in a
|
||||
* cooldown written by an earlier refusal: same event, nothing to persist.
|
||||
*/
|
||||
counted: boolean;
|
||||
windowMs: number;
|
||||
threshold: number;
|
||||
}
|
||||
|
||||
interface StreakState {
|
||||
count: number;
|
||||
lastRefusalAt: number;
|
||||
cooldownUntil: number;
|
||||
}
|
||||
|
||||
const streaks = new Map<string, StreakState>();
|
||||
let clock: () => number = () => Date.now();
|
||||
|
||||
function cooldownForStreak(streak: number): number {
|
||||
const step = Math.min(streak, REQUEST_REJECTED_COOLDOWNS_MS.length) - 1;
|
||||
return REQUEST_REJECTED_COOLDOWNS_MS[step];
|
||||
}
|
||||
|
||||
export function recordRequestRejected(
|
||||
connectionId: string,
|
||||
now: number = clock()
|
||||
): RequestRejectedVerdict {
|
||||
const existing = streaks.get(connectionId);
|
||||
const base = {
|
||||
windowMs: REQUEST_REJECTED_WINDOW_MS,
|
||||
threshold: REQUEST_REJECTED_ESCALATION_THRESHOLD,
|
||||
};
|
||||
|
||||
if (existing && now < existing.cooldownUntil) {
|
||||
// In flight before the cooldown was written — same episode.
|
||||
return {
|
||||
streak: existing.count,
|
||||
cooldownMs: Math.max(0, existing.cooldownUntil - now),
|
||||
escalate: false,
|
||||
counted: false,
|
||||
...base,
|
||||
};
|
||||
}
|
||||
|
||||
const continues = !!existing && now - existing.lastRefusalAt < REQUEST_REJECTED_WINDOW_MS;
|
||||
const streak = continues ? existing.count + 1 : 1;
|
||||
const escalate = streak >= REQUEST_REJECTED_ESCALATION_THRESHOLD;
|
||||
|
||||
if (escalate) {
|
||||
// The terminal state takes over; a reconnect starts from a clean slate.
|
||||
streaks.delete(connectionId);
|
||||
return { streak, cooldownMs: 0, escalate: true, counted: true, ...base };
|
||||
}
|
||||
|
||||
const cooldownMs = cooldownForStreak(streak);
|
||||
streaks.set(connectionId, { count: streak, lastRefusalAt: now, cooldownUntil: now + cooldownMs });
|
||||
return { streak, cooldownMs, escalate: false, counted: true, ...base };
|
||||
}
|
||||
|
||||
/**
|
||||
* Restore one refusal that happened before this process started (read from
|
||||
* the persisted connection row), so a crash loop cannot reset the count on
|
||||
* every boot. No-op when the connection already has in-memory state or the
|
||||
* refusal is outside the window.
|
||||
*/
|
||||
export function seedRequestRejectedStreak(
|
||||
connectionId: string,
|
||||
lastRefusalAt: number,
|
||||
now: number = clock()
|
||||
): boolean {
|
||||
if (streaks.has(connectionId)) return false;
|
||||
if (!Number.isFinite(lastRefusalAt) || now - lastRefusalAt >= REQUEST_REJECTED_WINDOW_MS) {
|
||||
return false;
|
||||
}
|
||||
streaks.set(connectionId, { count: 1, lastRefusalAt, cooldownUntil: lastRefusalAt });
|
||||
return true;
|
||||
}
|
||||
|
||||
export function hasRequestRejectedStreak(connectionId: string): boolean {
|
||||
return streaks.has(connectionId);
|
||||
}
|
||||
|
||||
/** Forget a connection's streak — on a successful response or an operator reset. */
|
||||
export function clearRequestRejectedStreak(connectionId: string): void {
|
||||
streaks.delete(connectionId);
|
||||
}
|
||||
|
||||
/** Test-only: wipe all streaks and restore the real clock. */
|
||||
export function __resetRequestRejectedStreaksForTests(): void {
|
||||
streaks.clear();
|
||||
clock = () => Date.now();
|
||||
}
|
||||
|
||||
/** Test-only: replace the clock the streak reads when no `now` is passed. */
|
||||
export function __setRequestRejectedClockForTests(fn: (() => number) | null): void {
|
||||
clock = fn ?? (() => Date.now());
|
||||
}
|
||||
@@ -6,6 +6,7 @@ import {
|
||||
} from "@/lib/compliance/providerAudit";
|
||||
import { getCachedProviderConnectionById } from "@/lib/db/readCache";
|
||||
import { updateProviderConnection } from "@/lib/db/providers";
|
||||
import { clearRequestRejectedStreak } from "@omniroute/open-sse/services/requestRejectedStreak.ts";
|
||||
import { deleteProviderConnection } from "@/lib/db/providers/deletion";
|
||||
import { isCloudEnabled } from "@/lib/db/settings";
|
||||
import { getConsistentMachineId } from "@/shared/utils/machineId";
|
||||
@@ -220,12 +221,15 @@ export async function PUT(request: Request, { params }: { params: Promise<{ id:
|
||||
if (lastErrorSource !== undefined) updateData.lastErrorSource = lastErrorSource;
|
||||
if (errorCode !== undefined) updateData.errorCode = errorCode;
|
||||
if (rateLimitedUntil !== undefined) updateData.rateLimitedUntil = rateLimitedUntil;
|
||||
// Clearing the cooldown by hand also forgets the refusal streak (#12859).
|
||||
if (rateLimitedUntil === null || testStatus === "active") clearRequestRejectedStreak(id);
|
||||
if (lastTested !== undefined) updateData.lastTested = lastTested;
|
||||
// healthCheckInterval PATCH semantics: undefined = leave as-is; null = clear
|
||||
// the override (connection follows the global default); 0-1440 = explicit
|
||||
// per-connection minutes (0 opts this connection out of the sweep).
|
||||
if (healthCheckInterval === null) updateData.healthCheckInterval = null;
|
||||
else if (healthCheckInterval !== undefined) updateData.healthCheckInterval = healthCheckInterval;
|
||||
else if (healthCheckInterval !== undefined)
|
||||
updateData.healthCheckInterval = healthCheckInterval;
|
||||
if (group !== undefined) updateData.group = group;
|
||||
if (maxConcurrent !== undefined) updateData.maxConcurrent = maxConcurrent;
|
||||
if (incomingWindowThresholds !== undefined) {
|
||||
|
||||
@@ -10,6 +10,7 @@ import {
|
||||
} from "../services/auth";
|
||||
import { maybeReactivateAfterExplicitProbe } from "../services/explicitInactiveProbe";
|
||||
import { connectionHasExtraKeys } from "@omniroute/open-sse/services/apiKeyRotator.ts";
|
||||
import { clearRequestRejectedStreak } from "@omniroute/open-sse/services/requestRejectedStreak.ts";
|
||||
import { createBuiltinAutoCombo } from "@omniroute/open-sse/services/autoCombo/builtinCatalog.ts";
|
||||
import * as log from "../utils/logger";
|
||||
import { updateProviderCredentials } from "../services/tokenRefresh";
|
||||
@@ -537,6 +538,9 @@ export async function executeChatWithBreaker({
|
||||
},
|
||||
onRequestSuccess: async () => {
|
||||
if (isShadowTraffic) return;
|
||||
// A healthy response ends any run of per-request refusals
|
||||
// (#12859) — only a real success does, not an elapsed cooldown.
|
||||
if (credentials.connectionId) clearRequestRejectedStreak(credentials.connectionId);
|
||||
await clearAccountError(credentials.connectionId, credentials);
|
||||
await maybeReactivateAfterExplicitProbe({
|
||||
connectionId: credentials.connectionId,
|
||||
|
||||
@@ -53,7 +53,10 @@ function isNonTerminalProviderError(providerErrorType: string | null): boolean {
|
||||
providerErrorType === PROVIDER_ERROR_TYPES.OAUTH_INVALID_TOKEN ||
|
||||
// #1010: Cloudflare fingerprint rejection is the CDN refusing the CLIENT's
|
||||
// signature, not the account's credentials — never a terminal account state.
|
||||
providerErrorType === PROVIDER_ERROR_TYPES.FINGERPRINT_REJECTION
|
||||
providerErrorType === PROVIDER_ERROR_TYPES.FINGERPRINT_REJECTION ||
|
||||
// Anthropic OAuth 403 "Request not allowed" refuses ONE request; the token
|
||||
// keeps serving the next one — never a terminal account state.
|
||||
providerErrorType === PROVIDER_ERROR_TYPES.REQUEST_REJECTED
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
@@ -72,6 +72,8 @@
|
||||
"tests/unit/agentrouter-lock-scope-10334.test.ts",
|
||||
"tests/unit/alibaba-free-tier-exhaustion.test.ts",
|
||||
"tests/unit/anthropic-thinking-signature-recovery.test.ts",
|
||||
"tests/unit/anthropic-request-not-allowed-cooldown-escalation.test.ts",
|
||||
"tests/unit/anthropic-request-not-allowed-not-a-ban.test.ts",
|
||||
"tests/unit/agy-family-not-connection-cooldown.test.ts",
|
||||
"tests/unit/agy-quota-exhaustion-threshold.test.ts",
|
||||
"tests/unit/antigravity-429-quota-cooldown.test.ts",
|
||||
|
||||
@@ -0,0 +1,241 @@
|
||||
/**
|
||||
* #12859 end-to-end through the real chat route: an Anthropic OAuth
|
||||
* `403 "Request not allowed"` must not ban the `claude` connection on the
|
||||
* first response — it excludes the connection for a short, growing cooldown
|
||||
* and only three consecutive refusals (no success in between) escalate to
|
||||
* `banned`. A burst of requests while the connection is already excluded is
|
||||
* one episode, and a healthy response resets the streak. Any other claude 403
|
||||
* keeps the previous behaviour (banned on the first one).
|
||||
*
|
||||
* Pattern mirrors tests/unit/probe-gate-autodisable.test.ts: temp DATA_DIR,
|
||||
* real SQLite, `globalThis.fetch` mocked as the upstream. The model id must
|
||||
* exist in the catalog (src/shared/constants/modelSpecs.ts).
|
||||
*/
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import fs from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
|
||||
const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-rr-escalation-"));
|
||||
process.env.DATA_DIR = TEST_DATA_DIR;
|
||||
|
||||
const core = await import("../../src/lib/db/core.ts");
|
||||
const { createProviderConnection, updateProviderConnection } =
|
||||
await import("../../src/lib/db/providers.ts");
|
||||
const { buildInternalChatRequest } = await import("../../src/lib/api/modelTestRunner.ts");
|
||||
const chatRouteModule = await import("../../src/app/api/v1/chat/completions/route.ts");
|
||||
const postChatCompletion = chatRouteModule.POST;
|
||||
const { resetAllCircuitBreakers } = await import("../../src/shared/utils/circuitBreaker.ts");
|
||||
const { invalidateDbCache } = await import("../../src/lib/db/readCache.ts");
|
||||
const {
|
||||
__resetRequestRejectedStreaksForTests,
|
||||
__setRequestRejectedClockForTests,
|
||||
hasRequestRejectedStreak,
|
||||
} = await import("../../open-sse/services/requestRejectedStreak.ts");
|
||||
const { COOLDOWN_MS } = await import("../../open-sse/config/errorConfig.ts");
|
||||
|
||||
const originalFetch = globalThis.fetch;
|
||||
const MODEL = "claude/claude-sonnet-5";
|
||||
|
||||
// The streak's own clock is advanced when a test "elapses" a cooldown, so the
|
||||
// in-memory episode boundary moves together with the DB row.
|
||||
let clockOffsetMs = 0;
|
||||
|
||||
test.beforeEach(() => {
|
||||
resetAllCircuitBreakers();
|
||||
__resetRequestRejectedStreaksForTests();
|
||||
clockOffsetMs = 0;
|
||||
__setRequestRejectedClockForTests(() => Date.now() + clockOffsetMs);
|
||||
invalidateDbCache("connections");
|
||||
});
|
||||
|
||||
test.after(() => {
|
||||
globalThis.fetch = originalFetch;
|
||||
core.resetDbInstance();
|
||||
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });
|
||||
});
|
||||
|
||||
type Row = Record<string, unknown> | undefined;
|
||||
|
||||
function readRow(connId: string): Row {
|
||||
const db = core.getDbInstance() as unknown as {
|
||||
prepare: (sql: string) => { get: (id: string) => Record<string, unknown> | undefined };
|
||||
};
|
||||
return db
|
||||
.prepare(
|
||||
"SELECT is_active, test_status, rate_limited_until, last_error, last_error_type, last_error_at, error_code FROM provider_connections WHERE id = ?"
|
||||
)
|
||||
.get(connId);
|
||||
}
|
||||
|
||||
async function createClaudeConnection(): Promise<string> {
|
||||
const conn = await createProviderConnection({
|
||||
provider: "claude",
|
||||
authType: "oauth",
|
||||
name: "rr-escalation",
|
||||
accessToken: "claude-oauth-access", // pragma: allowlist secret
|
||||
refreshToken: "claude-oauth-refresh", // pragma: allowlist secret
|
||||
expiresAt: new Date(Date.now() + 60 * 60 * 1000).toISOString(),
|
||||
isActive: true,
|
||||
testStatus: "active",
|
||||
});
|
||||
return String((conn as { id: string }).id);
|
||||
}
|
||||
|
||||
let upstreamCalls = 0;
|
||||
|
||||
function mockAnthropic403(message: string): void {
|
||||
globalThis.fetch = (async () => {
|
||||
upstreamCalls += 1;
|
||||
return new Response(
|
||||
JSON.stringify({ type: "error", error: { type: "permission_error", message } }),
|
||||
{ status: 403, headers: { "content-type": "application/json" } }
|
||||
);
|
||||
}) as typeof fetch;
|
||||
}
|
||||
|
||||
function mockAnthropic200(): void {
|
||||
globalThis.fetch = (async () => {
|
||||
upstreamCalls += 1;
|
||||
return new Response(
|
||||
JSON.stringify({
|
||||
id: "msg_test",
|
||||
type: "message",
|
||||
role: "assistant",
|
||||
model: "claude-sonnet-5",
|
||||
content: [{ type: "text", text: "OK" }],
|
||||
stop_reason: "end_turn",
|
||||
stop_sequence: null,
|
||||
usage: { input_tokens: 1, output_tokens: 1 },
|
||||
}),
|
||||
{ status: 200, headers: { "content-type": "application/json" } }
|
||||
);
|
||||
}) as typeof fetch;
|
||||
}
|
||||
|
||||
async function sendOnce(connId: string): Promise<Response> {
|
||||
return postChatCompletion(
|
||||
buildInternalChatRequest(
|
||||
{ model: MODEL, messages: [{ role: "user", content: "hi" }], stream: false },
|
||||
new AbortController().signal,
|
||||
connId
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
function cooldownMsOf(row: Row): number {
|
||||
const until = row?.rate_limited_until;
|
||||
assert.equal(typeof until, "string", "a cooldown must be persisted");
|
||||
const untilMs = Date.parse(until as string);
|
||||
assert.ok(Number.isFinite(untilMs), `rate_limited_until must be ISO, got ${String(until)}`);
|
||||
return untilMs - Date.now();
|
||||
}
|
||||
|
||||
function assertCoolingDown(row: Row, expectedMs: number, label: string) {
|
||||
assert.equal(row?.is_active, 1, `${label}: connection stays active`);
|
||||
assert.equal(row?.test_status, "unavailable", `${label}: cooling down, not banned`);
|
||||
assert.equal(row?.last_error_type, "request_rejected", `${label}: error type`);
|
||||
assert.equal(Number(row?.error_code), 403, `${label}: error code`);
|
||||
assert.equal(typeof row?.last_error_at, "string", `${label}: last_error_at written`);
|
||||
const ms = cooldownMsOf(row);
|
||||
assert.ok(
|
||||
Math.abs(ms - expectedMs) < 30_000,
|
||||
`${label}: cooldown ≈ ${expectedMs / 60000}min expected, got ${Math.round(ms / 1000)}s`
|
||||
);
|
||||
}
|
||||
|
||||
/** What the recovery tick does once the window has elapsed — not a success. */
|
||||
async function elapseCooldown(connId: string, cooldownMs: number) {
|
||||
clockOffsetMs += cooldownMs + 1000;
|
||||
await updateProviderConnection(connId, { testStatus: "active", rateLimitedUntil: null });
|
||||
invalidateDbCache("connections");
|
||||
}
|
||||
|
||||
test("claude 403 'Request not allowed': 5 min, then 15 min, banned on the 3rd consecutive refusal", async () => {
|
||||
const connId = await createClaudeConnection();
|
||||
mockAnthropic403("Request not allowed");
|
||||
|
||||
const first = await sendOnce(connId);
|
||||
assert.notEqual(first.status, 200);
|
||||
assertCoolingDown(readRow(connId), COOLDOWN_MS.requestRejected, "1st refusal");
|
||||
|
||||
await elapseCooldown(connId, COOLDOWN_MS.requestRejected);
|
||||
await sendOnce(connId);
|
||||
assertCoolingDown(readRow(connId), COOLDOWN_MS.requestRejectedRepeat, "2nd refusal");
|
||||
|
||||
await elapseCooldown(connId, COOLDOWN_MS.requestRejectedRepeat);
|
||||
await sendOnce(connId);
|
||||
const row = readRow(connId);
|
||||
assert.equal(row?.test_status, "banned", "3rd consecutive refusal: terminal");
|
||||
assert.equal(row?.is_active, 0);
|
||||
assert.match(String(row?.last_error), /3 consecutive refusals/);
|
||||
assert.equal(hasRequestRejectedStreak(connId), false, "escalation clears the streak");
|
||||
});
|
||||
|
||||
test("requests arriving while the connection is already excluded are one episode, not new refusals", async () => {
|
||||
const connId = await createClaudeConnection();
|
||||
mockAnthropic403("Request not allowed");
|
||||
upstreamCalls = 0;
|
||||
|
||||
await sendOnce(connId);
|
||||
const afterFirst = readRow(connId);
|
||||
assertCoolingDown(afterFirst, COOLDOWN_MS.requestRejected, "1st refusal");
|
||||
// The mocked fetch also serves auxiliary calls (identity bootstrap), so only
|
||||
// the delta matters below.
|
||||
const callsAfterFirst = upstreamCalls;
|
||||
|
||||
// Follow-ups pinned to the same connection (the internal request carries the
|
||||
// connection id, so selection does not filter it) all get the 403 while the
|
||||
// cooldown is running — like parallel sessions that were in flight when the
|
||||
// first 403 came back. They are one episode: nothing is re-persisted and,
|
||||
// above all, three of them must not ban.
|
||||
for (let i = 0; i < 3; i += 1) await sendOnce(connId);
|
||||
const afterBurst = readRow(connId);
|
||||
assert.ok(
|
||||
upstreamCalls > callsAfterFirst,
|
||||
"the burst did reach the upstream (pinned connection)"
|
||||
);
|
||||
assert.equal(afterBurst?.test_status, "unavailable", "still just cooling down, not banned");
|
||||
assert.equal(afterBurst?.is_active, 1);
|
||||
assert.equal(
|
||||
afterBurst?.rate_limited_until,
|
||||
afterFirst?.rate_limited_until,
|
||||
"cooldown unchanged by the burst"
|
||||
);
|
||||
assert.equal(hasRequestRejectedStreak(connId), true, "streak still open at 1");
|
||||
|
||||
// Once the cooldown has elapsed, the next refusal is the 2nd of the streak.
|
||||
await elapseCooldown(connId, COOLDOWN_MS.requestRejected);
|
||||
await sendOnce(connId);
|
||||
assertCoolingDown(readRow(connId), COOLDOWN_MS.requestRejectedRepeat, "2nd episode");
|
||||
});
|
||||
|
||||
test("a successful response resets the streak — sporadic refusals never accumulate", async () => {
|
||||
const connId = await createClaudeConnection();
|
||||
mockAnthropic403("Request not allowed");
|
||||
await sendOnce(connId);
|
||||
assertCoolingDown(readRow(connId), COOLDOWN_MS.requestRejected, "1st refusal");
|
||||
|
||||
await elapseCooldown(connId, COOLDOWN_MS.requestRejected);
|
||||
mockAnthropic200();
|
||||
const ok = await sendOnce(connId);
|
||||
assert.equal(ok.status, 200, "healthy response goes through");
|
||||
assert.equal(hasRequestRejectedStreak(connId), false, "success clears the streak");
|
||||
const clean = readRow(connId);
|
||||
assert.equal(clean?.test_status, "active");
|
||||
assert.equal(clean?.rate_limited_until, null);
|
||||
|
||||
mockAnthropic403("Request not allowed");
|
||||
await sendOnce(connId);
|
||||
assertCoolingDown(readRow(connId), COOLDOWN_MS.requestRejected, "refusal after success");
|
||||
});
|
||||
|
||||
test("any other claude 403 still bans on the first response (regression guard)", async () => {
|
||||
const connId = await createClaudeConnection();
|
||||
mockAnthropic403("Your organization has been disabled.");
|
||||
await sendOnce(connId);
|
||||
const row = readRow(connId);
|
||||
assert.equal(row?.test_status, "banned");
|
||||
assert.equal(row?.is_active, 0);
|
||||
});
|
||||
56
tests/unit/anthropic-request-not-allowed-not-a-ban.test.ts
Normal file
56
tests/unit/anthropic-request-not-allowed-not-a-ban.test.ts
Normal file
@@ -0,0 +1,56 @@
|
||||
/**
|
||||
* Anthropic OAuth 403 "Request not allowed" must never make the connection terminal.
|
||||
*
|
||||
* Reproduction of the incident behind this test: a single 403 with that body on
|
||||
* the only `claude` OAuth connection flipped it to `test_status = banned`
|
||||
* (chatCore FORBIDDEN → writeTerminalStatus), after which every request was
|
||||
* short-circuited with "All 1 connection(s) banned by upstream — please
|
||||
* reconnect in the dashboard" although the same token had returned 200 forty
|
||||
* seconds earlier and did so again once re-enabled.
|
||||
*
|
||||
* Two layers are pinned here:
|
||||
* 1. classifyProviderError() yields the non-terminal REQUEST_REJECTED type.
|
||||
* 2. resolveTerminalConnectionStatus() maps that type to `null` (no terminal
|
||||
* state), even when the fallback analysis flags the failure `permanent`.
|
||||
*/
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
const { classifyProviderError, PROVIDER_ERROR_TYPES } =
|
||||
await import("../../open-sse/services/errorClassifier.ts");
|
||||
const { resolveTerminalConnectionStatus } =
|
||||
await import("../../src/sse/services/authTerminalStatus.ts");
|
||||
|
||||
const ANTHROPIC_BODY = JSON.stringify({
|
||||
type: "error",
|
||||
error: { type: "permission_error", message: "Request not allowed" },
|
||||
});
|
||||
|
||||
test("claude 403 'Request not allowed' → REQUEST_REJECTED → no terminal connection status", () => {
|
||||
const errorType = classifyProviderError(403, ANTHROPIC_BODY, "claude");
|
||||
assert.equal(errorType, PROVIDER_ERROR_TYPES.REQUEST_REJECTED);
|
||||
|
||||
assert.equal(
|
||||
resolveTerminalConnectionStatus(403, { permanent: false }, errorType, "claude"),
|
||||
null,
|
||||
"a per-request refusal must not park the connection"
|
||||
);
|
||||
assert.equal(
|
||||
resolveTerminalConnectionStatus(403, { permanent: true }, errorType, "claude"),
|
||||
null,
|
||||
"non-terminal classification wins over a `permanent` fallback verdict"
|
||||
);
|
||||
});
|
||||
|
||||
test("regression guard: a generic claude 403 still resolves to banned", () => {
|
||||
const errorType = classifyProviderError(
|
||||
403,
|
||||
JSON.stringify({ error: { message: "you do not have permission" } }),
|
||||
"claude"
|
||||
);
|
||||
assert.equal(errorType, PROVIDER_ERROR_TYPES.FORBIDDEN);
|
||||
assert.equal(
|
||||
resolveTerminalConnectionStatus(403, { permanent: false }, errorType, "claude"),
|
||||
"banned"
|
||||
);
|
||||
});
|
||||
@@ -5,6 +5,8 @@ const {
|
||||
classifyProviderError,
|
||||
isResourceNotFoundResponse,
|
||||
isCloudflareFingerprintRejection,
|
||||
isAnthropicOAuthProvider,
|
||||
isAnthropicRequestNotAllowed,
|
||||
PROVIDER_ERROR_TYPES,
|
||||
} = await import("../../open-sse/services/errorClassifier.ts");
|
||||
|
||||
@@ -401,3 +403,56 @@ test("classifyProviderError: 422 without the BYOP code stays unclassified (no mo
|
||||
);
|
||||
assert.equal(classifyProviderError(422, "some other body", "antigravity"), null);
|
||||
});
|
||||
|
||||
// ── Anthropic OAuth 403 "Request not allowed" is a per-request refusal, not a ban ──
|
||||
|
||||
test("classifyProviderError: claude 403 'Request not allowed' (Anthropic body) => REQUEST_REJECTED, never FORBIDDEN", () => {
|
||||
// Verbatim Anthropic shape. On the reporting install this landed once between
|
||||
// hundreds of 200s on the same OAuth token and permanently banned the only
|
||||
// Claude connection.
|
||||
const body = JSON.stringify({
|
||||
type: "error",
|
||||
error: { type: "permission_error", message: "Request not allowed" },
|
||||
});
|
||||
const result = classifyProviderError(403, body, "claude");
|
||||
assert.equal(result, PROVIDER_ERROR_TYPES.REQUEST_REJECTED);
|
||||
assert.notEqual(result, PROVIDER_ERROR_TYPES.FORBIDDEN, "must not ban the connection");
|
||||
});
|
||||
|
||||
test("classifyProviderError: claude 403 'Request not allowed' via gateway-wrapped message => REQUEST_REJECTED", () => {
|
||||
// Shape as it reaches the classifier after the executor flattens the body.
|
||||
const result = classifyProviderError(403, "[403]: Request not allowed", "claude");
|
||||
assert.equal(result, PROVIDER_ERROR_TYPES.REQUEST_REJECTED);
|
||||
});
|
||||
|
||||
test("classifyProviderError: 'Request not allowed' from a non-Anthropic OAuth provider keeps its 403 semantics", () => {
|
||||
const body = JSON.stringify({ error: { message: "Request not allowed" } });
|
||||
assert.equal(classifyProviderError(403, body, "codex"), PROVIDER_ERROR_TYPES.FORBIDDEN);
|
||||
assert.equal(classifyProviderError(403, body, undefined), PROVIDER_ERROR_TYPES.FORBIDDEN);
|
||||
});
|
||||
|
||||
test("classifyProviderError: other claude 403 bodies still classify as before", () => {
|
||||
assert.equal(
|
||||
classifyProviderError(403, { error: { message: "you do not have permission" } }, "claude"),
|
||||
PROVIDER_ERROR_TYPES.FORBIDDEN
|
||||
);
|
||||
assert.equal(
|
||||
classifyProviderError(
|
||||
403,
|
||||
JSON.stringify({ error: { message: "account_deactivated: this account has been disabled" } }),
|
||||
"claude"
|
||||
),
|
||||
PROVIDER_ERROR_TYPES.ACCOUNT_DEACTIVATED
|
||||
);
|
||||
});
|
||||
|
||||
test("isAnthropicRequestNotAllowed / isAnthropicOAuthProvider helpers", () => {
|
||||
assert.equal(isAnthropicRequestNotAllowed("Request not allowed"), true);
|
||||
assert.equal(isAnthropicRequestNotAllowed('{"message":"request NOT allowed"}'), true);
|
||||
assert.equal(isAnthropicRequestNotAllowed("requests not allowed here"), false);
|
||||
assert.equal(isAnthropicRequestNotAllowed(""), false);
|
||||
assert.equal(isAnthropicOAuthProvider("claude"), true);
|
||||
assert.equal(isAnthropicOAuthProvider("Claude"), true);
|
||||
assert.equal(isAnthropicOAuthProvider("anthropic"), false);
|
||||
assert.equal(isAnthropicOAuthProvider(null), false);
|
||||
});
|
||||
|
||||
116
tests/unit/request-rejected-streak.test.ts
Normal file
116
tests/unit/request-rejected-streak.test.ts
Normal file
@@ -0,0 +1,116 @@
|
||||
/**
|
||||
* open-sse/services/requestRejectedStreak.ts (#12859): per-connection streak
|
||||
* of upstream per-request refusals → growing cooldown, escalation only for
|
||||
* consecutive refusals. One cooldown episode counts once: refusals landing
|
||||
* while the connection is already excluded were in flight before the cooldown
|
||||
* and are not new evidence.
|
||||
*/
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
const {
|
||||
recordRequestRejected,
|
||||
seedRequestRejectedStreak,
|
||||
hasRequestRejectedStreak,
|
||||
clearRequestRejectedStreak,
|
||||
__resetRequestRejectedStreaksForTests,
|
||||
REQUEST_REJECTED_WINDOW_MS,
|
||||
REQUEST_REJECTED_ESCALATION_THRESHOLD,
|
||||
REQUEST_REJECTED_COOLDOWNS_MS,
|
||||
} = await import("../../open-sse/services/requestRejectedStreak.ts");
|
||||
const { COOLDOWN_MS } = await import("../../open-sse/config/errorConfig.ts");
|
||||
|
||||
const MIN = 60 * 1000;
|
||||
const T0 = 1_000_000;
|
||||
|
||||
test.beforeEach(() => __resetRequestRejectedStreaksForTests());
|
||||
|
||||
test("ladder comes from COOLDOWN_MS and has one rung per non-escalating refusal", () => {
|
||||
assert.deepEqual(REQUEST_REJECTED_COOLDOWNS_MS, [
|
||||
COOLDOWN_MS.requestRejected,
|
||||
COOLDOWN_MS.requestRejectedRepeat,
|
||||
]);
|
||||
assert.equal(REQUEST_REJECTED_COOLDOWNS_MS.length, REQUEST_REJECTED_ESCALATION_THRESHOLD - 1);
|
||||
});
|
||||
|
||||
test("first refusal: first cooldown rung, counted, no escalation", () => {
|
||||
const v = recordRequestRejected("conn-a", T0);
|
||||
assert.equal(v.streak, 1);
|
||||
assert.equal(v.counted, true);
|
||||
assert.equal(v.escalate, false);
|
||||
assert.equal(v.cooldownMs, COOLDOWN_MS.requestRejected);
|
||||
assert.equal(v.threshold, REQUEST_REJECTED_ESCALATION_THRESHOLD);
|
||||
assert.equal(v.windowMs, REQUEST_REJECTED_WINDOW_MS);
|
||||
});
|
||||
|
||||
test("refusals that land while the connection is already excluded are the same episode", () => {
|
||||
const first = recordRequestRejected("conn-a", T0);
|
||||
// Three parallel requests that were in flight when the first 403 came back.
|
||||
const inFlight = [1, 2, 3].map((s) => recordRequestRejected("conn-a", T0 + s * 1000));
|
||||
for (const v of inFlight) {
|
||||
assert.equal(v.counted, false, "not new evidence");
|
||||
assert.equal(v.streak, 1, "streak stays at 1");
|
||||
assert.equal(v.escalate, false, "a burst must never ban");
|
||||
assert.ok(v.cooldownMs > 0 && v.cooldownMs <= first.cooldownMs, "remaining cooldown");
|
||||
}
|
||||
});
|
||||
|
||||
test("consecutive episodes grow the cooldown and escalate at the threshold", () => {
|
||||
const first = recordRequestRejected("conn-a", T0);
|
||||
const second = recordRequestRejected("conn-a", T0 + first.cooldownMs + 1);
|
||||
assert.equal(second.streak, 2);
|
||||
assert.equal(second.counted, true);
|
||||
assert.equal(second.cooldownMs, COOLDOWN_MS.requestRejectedRepeat);
|
||||
assert.equal(second.escalate, false);
|
||||
|
||||
const third = recordRequestRejected("conn-a", T0 + first.cooldownMs + second.cooldownMs + 2);
|
||||
assert.equal(third.streak, 3);
|
||||
assert.equal(third.escalate, true, "third consecutive refusal escalates");
|
||||
assert.equal(third.cooldownMs, 0, "terminal state replaces the cooldown");
|
||||
assert.equal(hasRequestRejectedStreak("conn-a"), false, "escalation clears the streak");
|
||||
|
||||
// A reconnected connection starts from 1 again.
|
||||
const afterBan = recordRequestRejected("conn-a", T0 + 60 * MIN);
|
||||
assert.equal(afterBan.streak, 1);
|
||||
assert.equal(afterBan.escalate, false);
|
||||
});
|
||||
|
||||
test("a refusal outside the staleness window starts a new streak", () => {
|
||||
recordRequestRejected("conn-a", T0);
|
||||
const late = recordRequestRejected("conn-a", T0 + REQUEST_REJECTED_WINDOW_MS + 1);
|
||||
assert.equal(late.streak, 1);
|
||||
assert.equal(late.cooldownMs, COOLDOWN_MS.requestRejected);
|
||||
});
|
||||
|
||||
test("a success (clearRequestRejectedStreak) resets the count", () => {
|
||||
const first = recordRequestRejected("conn-a", T0);
|
||||
clearRequestRejectedStreak("conn-a");
|
||||
const next = recordRequestRejected("conn-a", T0 + first.cooldownMs + 1);
|
||||
assert.equal(next.streak, 1, "sporadic refusals separated by successes never accumulate");
|
||||
assert.equal(next.cooldownMs, COOLDOWN_MS.requestRejected);
|
||||
});
|
||||
|
||||
test("streaks are per connection", () => {
|
||||
const a1 = recordRequestRejected("conn-a", T0);
|
||||
recordRequestRejected("conn-a", T0 + a1.cooldownMs + 1);
|
||||
const other = recordRequestRejected("conn-b", T0 + a1.cooldownMs + 2);
|
||||
assert.equal(other.streak, 1);
|
||||
});
|
||||
|
||||
test("seedRequestRejectedStreak restores one refusal from before a restart", () => {
|
||||
const lastRefusalAt = T0 - 10 * MIN;
|
||||
assert.equal(seedRequestRejectedStreak("conn-a", lastRefusalAt, T0), true);
|
||||
assert.equal(hasRequestRejectedStreak("conn-a"), true);
|
||||
const v = recordRequestRejected("conn-a", T0);
|
||||
assert.equal(v.streak, 2, "the seeded refusal counts as the first of the streak");
|
||||
assert.equal(v.counted, true, "the seeded refusal's cooldown is not re-applied");
|
||||
});
|
||||
|
||||
test("seedRequestRejectedStreak ignores stale rows and existing state", () => {
|
||||
assert.equal(seedRequestRejectedStreak("conn-a", T0 - REQUEST_REJECTED_WINDOW_MS, T0), false);
|
||||
assert.equal(seedRequestRejectedStreak("conn-a", Number.NaN, T0), false);
|
||||
assert.equal(hasRequestRejectedStreak("conn-a"), false);
|
||||
|
||||
recordRequestRejected("conn-b", T0);
|
||||
assert.equal(seedRequestRejectedStreak("conn-b", T0 - MIN, T0), false, "already tracked");
|
||||
});
|
||||
Reference in New Issue
Block a user