Files
OmniRoute/open-sse/config/errorConfig.ts
Innokentiy Solntsev dd70dbdaa0 fix(sse): Anthropic OAuth 403 "Request not allowed" is a per-request refusal — cooldown with backoff instead of an instant ban (#12859) (#12864)
* fix(sse): Anthropic OAuth 403 "Request not allowed" is a per-request refusal, not a ban

A single upstream 403 on the `claude` OAuth connection was classified
FORBIDDEN and written as the terminal `banned` connection state
(chatCore -> writeTerminalStatus). From then on every request to that
provider was short-circuited with "All 1 connection(s) banned by
upstream - please reconnect in the dashboard" without touching Anthropic,
until an operator reconnected.

Anthropic's OAuth surface answers a small fraction of otherwise-valid
requests with 403 {"type":"permission_error","message":"Request not
allowed"}. On the reporting install the same token returned 200 forty
seconds before the 403 and again right after the connection was
re-enabled; a revoked or expired token is a 401 authentication_error, not
this. It is a refusal of one request, not of the credential.

Classify it as the new non-terminal PROVIDER_ERROR_TYPES.REQUEST_REJECTED
(scoped to provider `claude` and the "Request not allowed" body) and list
that type in authTerminalStatus.isNonTerminalProviderError, mirroring the
Cloudflare FINGERPRINT_REJECTION precedent. The combo layer still falls
through to the next target for the failing request; the connection stays
active for the next one. Any other claude 403 keeps its previous
classification.

Tests: error-classifier.test.ts covers the Anthropic body, the
gateway-flattened "[403]: Request not allowed" message, the same body from
a non-Anthropic provider (still FORBIDDEN), other claude 403s (unchanged),
and the helpers; anthropic-request-not-allowed-not-a-ban.test.ts pins
resolveTerminalConnectionStatus() -> null for the new type even with a
`permanent` fallback verdict, and `banned` for a generic claude 403.

* fix(sse): cooldown with backoff and streak escalation for REQUEST_REJECTED (#12859)

Not "ignore the 403" either: if Anthropic ever made "Request not allowed"
systematic, re-sending every request into it would be the wrong thing to
do to an OAuth account. chatCore now handles REQUEST_REJECTED explicitly:

- exclude the connection via setConnectionRateLimitUntil for a growing
  cooldown (5 -> 15 -> 45 min) so a sporadic refusal costs minutes, not a
  reconnect, and a systematic one cannot become a stream of 403s;
- escalate to the terminal `banned` state only for 3 refusals within a
  60-minute window (services/requestRejectedStreak.ts, in-memory per
  connection; a restart forgets the streak, erring towards more cooldowns
  rather than an operator-undone ban), with a last_error that says so;
- probe-origin failures record but never cool down or ban (#9817).

The existing "request not allowed" text rule (5 s) is unaffected:
markAccountUnavailable skips a connection that already has a future
rateLimitedUntil, so the minute-scale cooldown written here wins.

Tests: request-rejected-streak.test.ts pins the window/threshold/backoff
arithmetic; anthropic-request-not-allowed-cooldown-escalation.test.ts drives
the real chat route against a mocked 403 upstream on a `claude` OAuth
connection: 300 s cooldown, then 900 s, then banned on the third refusal;
a different claude 403 body still bans on the first response.

* chore(changelog): name the #12859 fragment after its PR (#12864)

* refactor(sse): move the REQUEST_REJECTED branch into a chatCore leaf; register its tests for mutation coverage

chatCore.ts is frozen at 5984 lines by the file-size ratchet; the branch
body now lives in open-sse/handlers/chatCore/requestRejectedFailure.ts
(chatCore: 5974 -> 5983). stryker.conf.json tap.testFiles gains the two new
DB-backed tests so their mutant kills count (check:mutation-test-coverage).

* fix(sse): count refusal episodes, reset on success, keep the dashboard honest (#12859 review)

Review findings on the first cut of the REQUEST_REJECTED handling:

- A burst of in-flight requests that all got the 403 within seconds
  produced streak 1, 2, 3 and a ban from one upstream event. The streak
  now counts cooldown *episodes*: a refusal that lands while the
  connection is already excluded is the same event and is not counted.
- Nothing reset the streak on a healthy response, so sporadic refusals
  on a busy install could still accumulate to a ban. chatHelpers'
  onRequestSuccess now clears it (only a real success does - the recovery
  tick's clearAccountError is an elapsed cooldown, not a success).
  Clearing the cooldown by hand in the dashboard clears it too.
- The third rung of the ladder was unreachable (the third refusal
  escalates): the ladder is now 5 -> 15 min, sourced from COOLDOWN_MS next
  to the existing 5 s "request not allowed" rule, with a note on why that
  rule is superseded for claude. The 60-min window becomes a 24 h
  staleness bound - "consecutive" is defined by successes, not by time.
- Probe-origin refusals no longer touch the streak (#9817).
- The cooldown is written like every other connection-level cooldown:
  ISO rateLimitedUntil + testStatus "unavailable" (+ lastErrorAt), so the
  dashboard shows the countdown and the recovery tick restores "active".
- One refusal is re-seeded from the persisted row after a restart so a
  crash loop cannot reset the count on every boot.

Docs: RESILIENCE_GUIDE terminal states + CODEBASE_DOCUMENTATION resilience
row mention the streak module. Tests cover the burst, the success reset,
the seed, and the ISO/unavailable shape end-to-end through the chat route.

* chore(sse): drop unrelated Prettier churn in auth.ts / providers route

* style(api): keep providers route Prettier-clean

* refactor(sse): share the "exclude connection for a cooldown" leaf between GEO_BLOCKED, GCP_PROJECT_REQUIRED and the new branch

The release tip moved chatCore.ts to its frozen 5984 lines, so the
REQUEST_REJECTED branch cannot add a single net line. The GEO_BLOCKED and
GCP_PROJECT_REQUIRED branches were the same eight statements with different
constants and log wording; both now call
open-sse/handlers/chatCore/connectionCooldown.ts::excludeConnectionForCooldown
(behaviour, probe guard and log lines preserved verbatim). chatCore.ts ends
9 lines below the base it branched from.

* chore(chatCore): tighten the cooldown comments to keep the file under its size ceiling

After merging release/v3.8.51, chatCore.ts sat at 6150 lines against a
frozen ceiling of 6146. Condense the explanatory comments this PR added
to the GEO_BLOCKED and GCP_PROJECT_REQUIRED branches; no code change.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Co-authored-by: insoln <is@careerum.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
2026-09-17 16:26:08 -03:00

287 lines
9.7 KiB
TypeScript

export type ErrorInfo = {
type: string;
code: string;
};
export type ConfiguredErrorReason =
| "auth_error"
| "quota_exhausted"
| "rate_limit_exceeded"
| "model_capacity"
| "server_error"
| "unknown";
export type ErrorRule = {
id: string;
text?: string;
status?: number;
reason?: ConfiguredErrorReason;
cooldownMs?: number;
backoff?: boolean;
};
// OpenAI-compatible error types mapping (client-facing)
export const ERROR_TYPES: Record<number, ErrorInfo> = {
400: { type: "invalid_request_error", code: "bad_request" },
401: { type: "authentication_error", code: "invalid_api_key" },
402: { type: "billing_error", code: "payment_required" },
403: { type: "permission_error", code: "insufficient_quota" },
404: { type: "invalid_request_error", code: "model_not_found" },
406: { type: "invalid_request_error", code: "model_not_supported" },
410: { type: "invalid_request_error", code: "model_shutdown" },
429: { type: "rate_limit_error", code: "rate_limit_exceeded" },
499: { type: "client_disconnected", code: "client_disconnected" },
500: { type: "server_error", code: "internal_server_error" },
502: { type: "server_error", code: "bad_gateway" },
503: { type: "server_error", code: "service_unavailable" },
504: { type: "server_error", code: "gateway_timeout" },
};
// Default error messages per status code (client-facing)
export const DEFAULT_ERROR_MESSAGES: Record<number, string> = {
400: "Bad request",
401: "Invalid API key provided",
402: "Payment required",
403: "You exceeded your current quota",
404: "Model not found",
406: "Model not supported",
410: "Model has been shut down",
429: "Rate limit exceeded",
499: "Client disconnected",
500: "Internal server error",
502: "Bad gateway - upstream provider error",
503: "Service temporarily unavailable",
504: "Gateway timeout",
};
// Exponential backoff config for rate limits.
// Preserve OmniRoute's existing 2-minute cap to avoid changing runtime behavior.
export const BACKOFF_CONFIG = {
base: 1000,
max: 2 * 60 * 1000,
maxLevel: 15,
};
export const TRANSIENT_COOLDOWN_MS = 5 * 1000;
// Cooldown durations (ms)
export const COOLDOWN_MS = {
unauthorized: 2 * 60 * 1000,
paymentRequired: 2 * 60 * 1000,
notFound: 2 * 60 * 1000,
notFoundLocal: 5 * 1000,
transientInitial: TRANSIENT_COOLDOWN_MS,
transientMax: 60 * 1000,
transient: TRANSIENT_COOLDOWN_MS,
requestNotAllowed: 5 * 1000,
// Anthropic OAuth 403 "Request not allowed" (#12859): a per-request refusal
// on a healthy token. chatCore excludes the connection for requestRejected
// after the first refusal, requestRejectedRepeat after the second, and bans
// it on the third consecutive one (services/requestRejectedStreak.ts).
requestRejected: 5 * 60 * 1000,
requestRejectedRepeat: 15 * 60 * 1000,
rateLimit: 2 * 60 * 1000,
serviceUnavailable: 2 * 1000,
authExpired: 2 * 60 * 1000,
// Google regional-availability refusal: nothing changes region-wise on the
// account, so re-probe only after a long window (or when the operator routes
// egress through a supported-region proxy).
geoBlocked: 24 * 60 * 60 * 1000,
// Antigravity BYOP (GCP_PROJECT_REQUIRED): nothing changes on the account
// until the operator enters a Project ID, so keep the connection excluded
// from selection for a long window (mirrors the geo-blocked treatment).
gcpProjectRequired: 24 * 60 * 60 * 1000,
};
/**
* Shared rules for account fallback classification.
* Checked top-to-bottom: text rules first, then status rules.
*/
export const ERROR_RULES: ErrorRule[] = [
{
id: "no_credentials",
text: "no credentials",
cooldownMs: COOLDOWN_MS.notFound,
reason: "auth_error",
},
{
// For provider `claude` this text is classified REQUEST_REJECTED and the
// connection-level cooldown is written by chatCore before the fallback
// layer runs (#12859); markAccountUnavailable then keeps the longer
// cooldown. This 5 s rule still serves every other provider.
id: "request_not_allowed",
text: "request not allowed",
cooldownMs: COOLDOWN_MS.requestNotAllowed,
reason: "rate_limit_exceeded",
},
{
id: "improperly_formed_request",
text: "improperly formed request",
cooldownMs: 0,
reason: "model_capacity",
},
{ id: "rate_limit", text: "rate limit", backoff: true, reason: "rate_limit_exceeded" },
{
id: "too_many_requests",
text: "too many requests",
backoff: true,
reason: "rate_limit_exceeded",
},
{
id: "hour_quota_exceeded",
text: "hour quota",
backoff: true,
reason: "quota_exhausted",
},
{
id: "quota_has_been_exceeded",
text: "quota has been exceeded",
backoff: true,
reason: "quota_exhausted",
},
{
id: "quota_exceeded",
text: "quota exceeded",
backoff: true,
reason: "quota_exhausted",
},
{
id: "quota_will_reset",
text: "quota will reset",
backoff: true,
reason: "quota_exhausted",
},
{
id: "capacity_exhausted",
text: "exhausted your capacity",
backoff: true,
reason: "quota_exhausted",
},
{
id: "quota_exhausted",
text: "quota exhausted",
backoff: true,
reason: "quota_exhausted",
},
{
id: "free_tier_exhausted",
text: "free tier of the model has been exhausted",
backoff: true,
reason: "quota_exhausted",
},
{
id: "out_of_extra_usage",
text: "out of extra usage",
backoff: true,
reason: "quota_exhausted",
},
{
id: "extra_usage_required",
text: "extra usage required",
backoff: true,
reason: "quota_exhausted",
},
{ id: "capacity", text: "capacity", backoff: true, reason: "model_capacity" },
{ id: "overloaded", text: "overloaded", backoff: true, reason: "model_capacity" },
{ id: "high_demand", text: "high demand", backoff: true, reason: "model_capacity" },
{ id: "status_401", status: 401, cooldownMs: 0, reason: "auth_error" },
{ id: "status_402", status: 402, cooldownMs: 0, reason: "quota_exhausted" },
{ id: "status_403", status: 403, cooldownMs: 0, reason: "quota_exhausted" },
{ id: "status_404", status: 404, cooldownMs: COOLDOWN_MS.notFound, reason: "unknown" },
{ id: "status_406", status: 406, backoff: true, reason: "server_error" },
{ id: "status_408", status: 408, backoff: true, reason: "server_error" },
{ id: "status_429", status: 429, backoff: true, reason: "rate_limit_exceeded" },
{ id: "status_500", status: 500, backoff: true, reason: "server_error" },
{ id: "status_502", status: 502, backoff: true, reason: "server_error" },
{ id: "status_503", status: 503, backoff: true, reason: "server_error" },
{ id: "status_504", status: 504, backoff: true, reason: "server_error" },
];
function normalizeErrorMessage(message: unknown): string {
return String(message || "").toLowerCase();
}
export function getErrorInfo(statusCode: number): ErrorInfo {
return (
ERROR_TYPES[statusCode] ||
(statusCode >= 500
? { type: "server_error", code: "internal_server_error" }
: { type: "invalid_request_error", code: "" })
);
}
export function getDefaultErrorMessage(statusCode: number): string {
return DEFAULT_ERROR_MESSAGES[statusCode] || "An error occurred";
}
export function calculateBackoffCooldown(level = 0): number {
const safeLevel = Math.max(0, Math.floor(level));
const cooldown = BACKOFF_CONFIG.base * Math.pow(2, safeLevel);
return Math.min(cooldown, BACKOFF_CONFIG.max);
}
export function matchErrorRuleByText(message: unknown): ErrorRule | null {
const lower = normalizeErrorMessage(message);
if (!lower) return null;
return ERROR_RULES.find((rule) => rule.text && lower.includes(rule.text)) || null;
}
export function matchErrorRuleByStatus(statusCode: number): ErrorRule | null {
return ERROR_RULES.find((rule) => rule.status === statusCode) || null;
}
export function findMatchingErrorRule(statusCode: number, message: unknown): ErrorRule | null {
return matchErrorRuleByText(message) || matchErrorRuleByStatus(statusCode);
}
// #8248: NVIDIA NIM function-state DEGRADED — some NIM deployments signal a non-standard
// HTTP 400 whose body reports the backing "function" is DEGRADED (e.g. `Function id "<uuid>"
// submitted for inference is DEGRADED`) instead of a clean model-not-found/5xx. Bounded
// lookahead ({0,80}) — ReDoS-safe, no nested quantifiers.
const NIM_FUNCTION_DEGRADED_PATTERNS = [
/\bfunction\b[\s\S]{0,80}?\bDEGRADED\b/i,
/\bDEGRADED\b[\s\S]{0,80}?\bfunction\b/i,
];
export function isNimFunctionDegraded(errorText: string): boolean {
return NIM_FUNCTION_DEGRADED_PATTERNS.some((p) => p.test(errorText));
}
export interface ServiceSupervisorCooldown {
shouldFallback: true;
cooldownMs: number;
baseCooldownMs: number;
newBackoffLevel: 0;
reason: string;
skipProviderBreaker: true;
}
/**
* G-02: detect embedded service supervisor failures (X-Omni-Fallback-Hint: connection_cooldown).
* These are NOT upstream AI provider failures — they are local supervisor state changes. Returns
* a short 5s connection-cooldown decision (no provider circuit-breaker trip), or null when the
* status/header don't match.
*/
export function serviceSupervisorCooldown(
status: number,
headers: Headers | Record<string, string> | null
): ServiceSupervisorCooldown | null {
if (status !== 503 || !headers) return null;
const hintValue =
typeof (headers as Headers).get === "function"
? (headers as Headers).get("x-omni-fallback-hint")
: (headers as Record<string, string>)["x-omni-fallback-hint"] ||
(headers as Record<string, string>)["X-Omni-Fallback-Hint"];
if (typeof hintValue !== "string" || hintValue.toLowerCase() !== "connection_cooldown") {
return null;
}
return {
shouldFallback: true,
cooldownMs: 5_000,
baseCooldownMs: 5_000,
newBackoffLevel: 0,
reason: "service_not_running",
skipProviderBreaker: true,
};
}