mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-09-19 13:23:50 +03:00
* fix(sse): Anthropic OAuth 403 "Request not allowed" is a per-request refusal, not a ban
A single upstream 403 on the `claude` OAuth connection was classified
FORBIDDEN and written as the terminal `banned` connection state
(chatCore -> writeTerminalStatus). From then on every request to that
provider was short-circuited with "All 1 connection(s) banned by
upstream - please reconnect in the dashboard" without touching Anthropic,
until an operator reconnected.
Anthropic's OAuth surface answers a small fraction of otherwise-valid
requests with 403 {"type":"permission_error","message":"Request not
allowed"}. On the reporting install the same token returned 200 forty
seconds before the 403 and again right after the connection was
re-enabled; a revoked or expired token is a 401 authentication_error, not
this. It is a refusal of one request, not of the credential.
Classify it as the new non-terminal PROVIDER_ERROR_TYPES.REQUEST_REJECTED
(scoped to provider `claude` and the "Request not allowed" body) and list
that type in authTerminalStatus.isNonTerminalProviderError, mirroring the
Cloudflare FINGERPRINT_REJECTION precedent. The combo layer still falls
through to the next target for the failing request; the connection stays
active for the next one. Any other claude 403 keeps its previous
classification.
Tests: error-classifier.test.ts covers the Anthropic body, the
gateway-flattened "[403]: Request not allowed" message, the same body from
a non-Anthropic provider (still FORBIDDEN), other claude 403s (unchanged),
and the helpers; anthropic-request-not-allowed-not-a-ban.test.ts pins
resolveTerminalConnectionStatus() -> null for the new type even with a
`permanent` fallback verdict, and `banned` for a generic claude 403.
* fix(sse): cooldown with backoff and streak escalation for REQUEST_REJECTED (#12859)
Not "ignore the 403" either: if Anthropic ever made "Request not allowed"
systematic, re-sending every request into it would be the wrong thing to
do to an OAuth account. chatCore now handles REQUEST_REJECTED explicitly:
- exclude the connection via setConnectionRateLimitUntil for a growing
cooldown (5 -> 15 -> 45 min) so a sporadic refusal costs minutes, not a
reconnect, and a systematic one cannot become a stream of 403s;
- escalate to the terminal `banned` state only for 3 refusals within a
60-minute window (services/requestRejectedStreak.ts, in-memory per
connection; a restart forgets the streak, erring towards more cooldowns
rather than an operator-undone ban), with a last_error that says so;
- probe-origin failures record but never cool down or ban (#9817).
The existing "request not allowed" text rule (5 s) is unaffected:
markAccountUnavailable skips a connection that already has a future
rateLimitedUntil, so the minute-scale cooldown written here wins.
Tests: request-rejected-streak.test.ts pins the window/threshold/backoff
arithmetic; anthropic-request-not-allowed-cooldown-escalation.test.ts drives
the real chat route against a mocked 403 upstream on a `claude` OAuth
connection: 300 s cooldown, then 900 s, then banned on the third refusal;
a different claude 403 body still bans on the first response.
* chore(changelog): name the #12859 fragment after its PR (#12864)
* refactor(sse): move the REQUEST_REJECTED branch into a chatCore leaf; register its tests for mutation coverage
chatCore.ts is frozen at 5984 lines by the file-size ratchet; the branch
body now lives in open-sse/handlers/chatCore/requestRejectedFailure.ts
(chatCore: 5974 -> 5983). stryker.conf.json tap.testFiles gains the two new
DB-backed tests so their mutant kills count (check:mutation-test-coverage).
* fix(sse): count refusal episodes, reset on success, keep the dashboard honest (#12859 review)
Review findings on the first cut of the REQUEST_REJECTED handling:
- A burst of in-flight requests that all got the 403 within seconds
produced streak 1, 2, 3 and a ban from one upstream event. The streak
now counts cooldown *episodes*: a refusal that lands while the
connection is already excluded is the same event and is not counted.
- Nothing reset the streak on a healthy response, so sporadic refusals
on a busy install could still accumulate to a ban. chatHelpers'
onRequestSuccess now clears it (only a real success does - the recovery
tick's clearAccountError is an elapsed cooldown, not a success).
Clearing the cooldown by hand in the dashboard clears it too.
- The third rung of the ladder was unreachable (the third refusal
escalates): the ladder is now 5 -> 15 min, sourced from COOLDOWN_MS next
to the existing 5 s "request not allowed" rule, with a note on why that
rule is superseded for claude. The 60-min window becomes a 24 h
staleness bound - "consecutive" is defined by successes, not by time.
- Probe-origin refusals no longer touch the streak (#9817).
- The cooldown is written like every other connection-level cooldown:
ISO rateLimitedUntil + testStatus "unavailable" (+ lastErrorAt), so the
dashboard shows the countdown and the recovery tick restores "active".
- One refusal is re-seeded from the persisted row after a restart so a
crash loop cannot reset the count on every boot.
Docs: RESILIENCE_GUIDE terminal states + CODEBASE_DOCUMENTATION resilience
row mention the streak module. Tests cover the burst, the success reset,
the seed, and the ISO/unavailable shape end-to-end through the chat route.
* chore(sse): drop unrelated Prettier churn in auth.ts / providers route
* style(api): keep providers route Prettier-clean
* refactor(sse): share the "exclude connection for a cooldown" leaf between GEO_BLOCKED, GCP_PROJECT_REQUIRED and the new branch
The release tip moved chatCore.ts to its frozen 5984 lines, so the
REQUEST_REJECTED branch cannot add a single net line. The GEO_BLOCKED and
GCP_PROJECT_REQUIRED branches were the same eight statements with different
constants and log wording; both now call
open-sse/handlers/chatCore/connectionCooldown.ts::excludeConnectionForCooldown
(behaviour, probe guard and log lines preserved verbatim). chatCore.ts ends
9 lines below the base it branched from.
* chore(chatCore): tighten the cooldown comments to keep the file under its size ceiling
After merging release/v3.8.51, chatCore.ts sat at 6150 lines against a
frozen ceiling of 6146. Condense the explanatory comments this PR added
to the GEO_BLOCKED and GCP_PROJECT_REQUIRED branches; no code change.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: insoln <is@careerum.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
287 lines
9.7 KiB
TypeScript
287 lines
9.7 KiB
TypeScript
export type ErrorInfo = {
|
|
type: string;
|
|
code: string;
|
|
};
|
|
|
|
export type ConfiguredErrorReason =
|
|
| "auth_error"
|
|
| "quota_exhausted"
|
|
| "rate_limit_exceeded"
|
|
| "model_capacity"
|
|
| "server_error"
|
|
| "unknown";
|
|
|
|
export type ErrorRule = {
|
|
id: string;
|
|
text?: string;
|
|
status?: number;
|
|
reason?: ConfiguredErrorReason;
|
|
cooldownMs?: number;
|
|
backoff?: boolean;
|
|
};
|
|
|
|
// OpenAI-compatible error types mapping (client-facing)
|
|
export const ERROR_TYPES: Record<number, ErrorInfo> = {
|
|
400: { type: "invalid_request_error", code: "bad_request" },
|
|
401: { type: "authentication_error", code: "invalid_api_key" },
|
|
402: { type: "billing_error", code: "payment_required" },
|
|
403: { type: "permission_error", code: "insufficient_quota" },
|
|
404: { type: "invalid_request_error", code: "model_not_found" },
|
|
406: { type: "invalid_request_error", code: "model_not_supported" },
|
|
410: { type: "invalid_request_error", code: "model_shutdown" },
|
|
429: { type: "rate_limit_error", code: "rate_limit_exceeded" },
|
|
499: { type: "client_disconnected", code: "client_disconnected" },
|
|
500: { type: "server_error", code: "internal_server_error" },
|
|
502: { type: "server_error", code: "bad_gateway" },
|
|
503: { type: "server_error", code: "service_unavailable" },
|
|
504: { type: "server_error", code: "gateway_timeout" },
|
|
};
|
|
|
|
// Default error messages per status code (client-facing)
|
|
export const DEFAULT_ERROR_MESSAGES: Record<number, string> = {
|
|
400: "Bad request",
|
|
401: "Invalid API key provided",
|
|
402: "Payment required",
|
|
403: "You exceeded your current quota",
|
|
404: "Model not found",
|
|
406: "Model not supported",
|
|
410: "Model has been shut down",
|
|
429: "Rate limit exceeded",
|
|
499: "Client disconnected",
|
|
500: "Internal server error",
|
|
502: "Bad gateway - upstream provider error",
|
|
503: "Service temporarily unavailable",
|
|
504: "Gateway timeout",
|
|
};
|
|
|
|
// Exponential backoff config for rate limits.
|
|
// Preserve OmniRoute's existing 2-minute cap to avoid changing runtime behavior.
|
|
export const BACKOFF_CONFIG = {
|
|
base: 1000,
|
|
max: 2 * 60 * 1000,
|
|
maxLevel: 15,
|
|
};
|
|
|
|
export const TRANSIENT_COOLDOWN_MS = 5 * 1000;
|
|
|
|
// Cooldown durations (ms)
|
|
export const COOLDOWN_MS = {
|
|
unauthorized: 2 * 60 * 1000,
|
|
paymentRequired: 2 * 60 * 1000,
|
|
notFound: 2 * 60 * 1000,
|
|
notFoundLocal: 5 * 1000,
|
|
transientInitial: TRANSIENT_COOLDOWN_MS,
|
|
transientMax: 60 * 1000,
|
|
transient: TRANSIENT_COOLDOWN_MS,
|
|
requestNotAllowed: 5 * 1000,
|
|
// Anthropic OAuth 403 "Request not allowed" (#12859): a per-request refusal
|
|
// on a healthy token. chatCore excludes the connection for requestRejected
|
|
// after the first refusal, requestRejectedRepeat after the second, and bans
|
|
// it on the third consecutive one (services/requestRejectedStreak.ts).
|
|
requestRejected: 5 * 60 * 1000,
|
|
requestRejectedRepeat: 15 * 60 * 1000,
|
|
rateLimit: 2 * 60 * 1000,
|
|
serviceUnavailable: 2 * 1000,
|
|
authExpired: 2 * 60 * 1000,
|
|
// Google regional-availability refusal: nothing changes region-wise on the
|
|
// account, so re-probe only after a long window (or when the operator routes
|
|
// egress through a supported-region proxy).
|
|
geoBlocked: 24 * 60 * 60 * 1000,
|
|
// Antigravity BYOP (GCP_PROJECT_REQUIRED): nothing changes on the account
|
|
// until the operator enters a Project ID, so keep the connection excluded
|
|
// from selection for a long window (mirrors the geo-blocked treatment).
|
|
gcpProjectRequired: 24 * 60 * 60 * 1000,
|
|
};
|
|
|
|
/**
|
|
* Shared rules for account fallback classification.
|
|
* Checked top-to-bottom: text rules first, then status rules.
|
|
*/
|
|
export const ERROR_RULES: ErrorRule[] = [
|
|
{
|
|
id: "no_credentials",
|
|
text: "no credentials",
|
|
cooldownMs: COOLDOWN_MS.notFound,
|
|
reason: "auth_error",
|
|
},
|
|
{
|
|
// For provider `claude` this text is classified REQUEST_REJECTED and the
|
|
// connection-level cooldown is written by chatCore before the fallback
|
|
// layer runs (#12859); markAccountUnavailable then keeps the longer
|
|
// cooldown. This 5 s rule still serves every other provider.
|
|
id: "request_not_allowed",
|
|
text: "request not allowed",
|
|
cooldownMs: COOLDOWN_MS.requestNotAllowed,
|
|
reason: "rate_limit_exceeded",
|
|
},
|
|
{
|
|
id: "improperly_formed_request",
|
|
text: "improperly formed request",
|
|
cooldownMs: 0,
|
|
reason: "model_capacity",
|
|
},
|
|
{ id: "rate_limit", text: "rate limit", backoff: true, reason: "rate_limit_exceeded" },
|
|
{
|
|
id: "too_many_requests",
|
|
text: "too many requests",
|
|
backoff: true,
|
|
reason: "rate_limit_exceeded",
|
|
},
|
|
{
|
|
id: "hour_quota_exceeded",
|
|
text: "hour quota",
|
|
backoff: true,
|
|
reason: "quota_exhausted",
|
|
},
|
|
{
|
|
id: "quota_has_been_exceeded",
|
|
text: "quota has been exceeded",
|
|
backoff: true,
|
|
reason: "quota_exhausted",
|
|
},
|
|
{
|
|
id: "quota_exceeded",
|
|
text: "quota exceeded",
|
|
backoff: true,
|
|
reason: "quota_exhausted",
|
|
},
|
|
{
|
|
id: "quota_will_reset",
|
|
text: "quota will reset",
|
|
backoff: true,
|
|
reason: "quota_exhausted",
|
|
},
|
|
{
|
|
id: "capacity_exhausted",
|
|
text: "exhausted your capacity",
|
|
backoff: true,
|
|
reason: "quota_exhausted",
|
|
},
|
|
{
|
|
id: "quota_exhausted",
|
|
text: "quota exhausted",
|
|
backoff: true,
|
|
reason: "quota_exhausted",
|
|
},
|
|
{
|
|
id: "free_tier_exhausted",
|
|
text: "free tier of the model has been exhausted",
|
|
backoff: true,
|
|
reason: "quota_exhausted",
|
|
},
|
|
{
|
|
id: "out_of_extra_usage",
|
|
text: "out of extra usage",
|
|
backoff: true,
|
|
reason: "quota_exhausted",
|
|
},
|
|
{
|
|
id: "extra_usage_required",
|
|
text: "extra usage required",
|
|
backoff: true,
|
|
reason: "quota_exhausted",
|
|
},
|
|
{ id: "capacity", text: "capacity", backoff: true, reason: "model_capacity" },
|
|
{ id: "overloaded", text: "overloaded", backoff: true, reason: "model_capacity" },
|
|
{ id: "high_demand", text: "high demand", backoff: true, reason: "model_capacity" },
|
|
{ id: "status_401", status: 401, cooldownMs: 0, reason: "auth_error" },
|
|
{ id: "status_402", status: 402, cooldownMs: 0, reason: "quota_exhausted" },
|
|
{ id: "status_403", status: 403, cooldownMs: 0, reason: "quota_exhausted" },
|
|
{ id: "status_404", status: 404, cooldownMs: COOLDOWN_MS.notFound, reason: "unknown" },
|
|
{ id: "status_406", status: 406, backoff: true, reason: "server_error" },
|
|
{ id: "status_408", status: 408, backoff: true, reason: "server_error" },
|
|
{ id: "status_429", status: 429, backoff: true, reason: "rate_limit_exceeded" },
|
|
{ id: "status_500", status: 500, backoff: true, reason: "server_error" },
|
|
{ id: "status_502", status: 502, backoff: true, reason: "server_error" },
|
|
{ id: "status_503", status: 503, backoff: true, reason: "server_error" },
|
|
{ id: "status_504", status: 504, backoff: true, reason: "server_error" },
|
|
];
|
|
|
|
function normalizeErrorMessage(message: unknown): string {
|
|
return String(message || "").toLowerCase();
|
|
}
|
|
|
|
export function getErrorInfo(statusCode: number): ErrorInfo {
|
|
return (
|
|
ERROR_TYPES[statusCode] ||
|
|
(statusCode >= 500
|
|
? { type: "server_error", code: "internal_server_error" }
|
|
: { type: "invalid_request_error", code: "" })
|
|
);
|
|
}
|
|
|
|
export function getDefaultErrorMessage(statusCode: number): string {
|
|
return DEFAULT_ERROR_MESSAGES[statusCode] || "An error occurred";
|
|
}
|
|
|
|
export function calculateBackoffCooldown(level = 0): number {
|
|
const safeLevel = Math.max(0, Math.floor(level));
|
|
const cooldown = BACKOFF_CONFIG.base * Math.pow(2, safeLevel);
|
|
return Math.min(cooldown, BACKOFF_CONFIG.max);
|
|
}
|
|
|
|
export function matchErrorRuleByText(message: unknown): ErrorRule | null {
|
|
const lower = normalizeErrorMessage(message);
|
|
if (!lower) return null;
|
|
return ERROR_RULES.find((rule) => rule.text && lower.includes(rule.text)) || null;
|
|
}
|
|
|
|
export function matchErrorRuleByStatus(statusCode: number): ErrorRule | null {
|
|
return ERROR_RULES.find((rule) => rule.status === statusCode) || null;
|
|
}
|
|
|
|
export function findMatchingErrorRule(statusCode: number, message: unknown): ErrorRule | null {
|
|
return matchErrorRuleByText(message) || matchErrorRuleByStatus(statusCode);
|
|
}
|
|
|
|
// #8248: NVIDIA NIM function-state DEGRADED — some NIM deployments signal a non-standard
|
|
// HTTP 400 whose body reports the backing "function" is DEGRADED (e.g. `Function id "<uuid>"
|
|
// submitted for inference is DEGRADED`) instead of a clean model-not-found/5xx. Bounded
|
|
// lookahead ({0,80}) — ReDoS-safe, no nested quantifiers.
|
|
const NIM_FUNCTION_DEGRADED_PATTERNS = [
|
|
/\bfunction\b[\s\S]{0,80}?\bDEGRADED\b/i,
|
|
/\bDEGRADED\b[\s\S]{0,80}?\bfunction\b/i,
|
|
];
|
|
|
|
export function isNimFunctionDegraded(errorText: string): boolean {
|
|
return NIM_FUNCTION_DEGRADED_PATTERNS.some((p) => p.test(errorText));
|
|
}
|
|
|
|
export interface ServiceSupervisorCooldown {
|
|
shouldFallback: true;
|
|
cooldownMs: number;
|
|
baseCooldownMs: number;
|
|
newBackoffLevel: 0;
|
|
reason: string;
|
|
skipProviderBreaker: true;
|
|
}
|
|
|
|
/**
|
|
* G-02: detect embedded service supervisor failures (X-Omni-Fallback-Hint: connection_cooldown).
|
|
* These are NOT upstream AI provider failures — they are local supervisor state changes. Returns
|
|
* a short 5s connection-cooldown decision (no provider circuit-breaker trip), or null when the
|
|
* status/header don't match.
|
|
*/
|
|
export function serviceSupervisorCooldown(
|
|
status: number,
|
|
headers: Headers | Record<string, string> | null
|
|
): ServiceSupervisorCooldown | null {
|
|
if (status !== 503 || !headers) return null;
|
|
const hintValue =
|
|
typeof (headers as Headers).get === "function"
|
|
? (headers as Headers).get("x-omni-fallback-hint")
|
|
: (headers as Record<string, string>)["x-omni-fallback-hint"] ||
|
|
(headers as Record<string, string>)["X-Omni-Fallback-Hint"];
|
|
if (typeof hintValue !== "string" || hintValue.toLowerCase() !== "connection_cooldown") {
|
|
return null;
|
|
}
|
|
return {
|
|
shouldFallback: true,
|
|
cooldownMs: 5_000,
|
|
baseCooldownMs: 5_000,
|
|
newBackoffLevel: 0,
|
|
reason: "service_not_running",
|
|
skipProviderBreaker: true,
|
|
};
|
|
}
|