mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-09-16 03:42:21 +03:00
fix(providers): stop parking Mistral connections on a bare 401 with no clear auth failure (#13609)
Behind the new `MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT` flag (default off), a bare Mistral 401 (`{"detail":"Unauthorized"}`, identical for a revoked key and an exhausted quota) gets a retryable cooldown instead of parking the connection as `expired`; after three soft strikes within an hour the next bare 401 parks it, so revocation still converges.
Maintainer rework before merge (kept the idea, no default behavior change):
- The predicate is shared with the connection-test module instead of duplicated; the squeezed 139-char line that dodged the file-size gate is formatted normally and the growth is rebaselined honestly with an annotation.
Validated first on the combined board of all 38 PRs of this batch (10 merged as-is, 28 after the maintainer rework) on top of release/v3.8.51 c0f92ec: typecheck:core, check:open-sse-typecheck and check:dashboard-typecheck clean; ESLint clean on every changed file; file-size (rebaselined for the combined growth), complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync, migration-numbering and i18n new-key gates green; 735 focused node:test cases with the only batch-caused failure (a flag-count assertion) fixed. Then re-validated alone on the fresh release tip right before this merge: ESLint on the changed files, typecheck:core, check:open-sse-typecheck, the file-size/complexity/changelog gates and this PR's own tests.
Thanks @maxmad64bis!
This commit is contained in:
1
changelog.d/fixes/13609-mistral-401-ambiguous-auth.md
Normal file
1
changelog.d/fixes/13609-mistral-401-ambiguous-auth.md
Normal file
@@ -0,0 +1 @@
|
||||
- **fix(providers):** opt-in `MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT` flag (default off): a bare Mistral 401 with no explicit auth signal (identical for a revoked key and an exhausted quota) cools the connection down instead of parking it as expired, at most 3 times per hour per connection before it parks, so a revoked key still converges; the ambiguity check is now one implementation shared by the connection test and the runtime ([#13609](https://github.com/diegosouzapw/OmniRoute/pull/13609)) — thanks @maxmad64bis
|
||||
@@ -1,4 +1,5 @@
|
||||
{
|
||||
"_rebaseline_2026_09_15_13609_combined_growth": "Combined growth of the 2026-09-15 maxmad64bis uplift batch (each PR rebaselined its own growth; the merged sum is larger): open-sse/services/accountFallback.ts->2507. Every hunk is flag-gated or a verified fix covered by that PR's tests; see the batch report.",
|
||||
"_rebaseline_2026_09_15_13602_combined_growth": "Combined growth of the 2026-09-15 maxmad64bis uplift batch (each PR rebaselined its own growth; the merged sum is larger): src/sse/handlers/chatHelpers.ts->1214. Every hunk is flag-gated or a verified fix covered by that PR's tests; see the batch report.",
|
||||
"_rebaseline_2026_09_15_13580_combined_growth": "Combined growth of the 2026-09-15 maxmad64bis uplift batch (each PR rebaselined its own growth; the merged sum is larger): src/sse/handlers/chatHelpers.ts->1202. Every hunk is flag-gated or a verified fix covered by that PR's tests; see the batch report.",
|
||||
"_rebaseline_2026_09_13_13581_pool_egress_observation": "PR #13581 own growth: src/app/(dashboard)/dashboard/settings/components/ProxyRegistryManager.tsx 1475->1477 (+2 = the PoolEgressObservation import and its one-line mount under the pool members label). The observation itself lives outside the frozen file, all under cap: PoolEgressObservation.tsx, the dedicated GET /api/settings/proxies/pool/egress-observation route, src/lib/proxyPoolEgressObservation.ts and getPoolEgressObservation in src/lib/db/proxyLogs.ts. Only the mount point is irreducible. Covered by tests/unit/proxy-pool-egress-observation.test.ts, tests/unit/proxy-pool-egress-observation-route.test.ts and tests/unit/ui/PoolEgressObservation.test.tsx.",
|
||||
@@ -345,6 +346,7 @@
|
||||
"_rebaseline_2026_07_27_v3849_train1h": "Merge-train 1H (31 PRs) — owner-approved 2026-07-27. Two distinct causes, kept separate on purpose: (1) GENUINE irreducible growth at existing chokepoints — providerLimits/auth (#8632 Kimi quota-reset recovery), rateLimitManager (#8616 idle wedged limiters), models-catalog-route.test (#8610 OpenCode Go effort aliases); (2) COLLISION with #8585, which banked shrinks measured on the pre-train release tip while 30 sibling PRs in the SAME train grew those files again — chat/accountFallback (#8628), chatCore (#8613), videoGeneration (#8581), imageGeneration. The zero-headroom frozen entries cannot absorb either. Ceilings re-pinned to the post-merge tip; #8612 (also in this train) automates shrink-banking so this self-inflicted drift stops recurring. Detail: src/lib/usage/providerLimits.ts 1006->1013 (#8632); src/sse/services/auth.ts 2492->2508 (#8632); open-sse/services/rateLimitManager.ts 1014->1060 (#8616); src/sse/handlers/chat.ts 1842->1845 (#8628); open-sse/handlers/chatCore.ts 4939->4955 (#8613); open-sse/handlers/imageGeneration.ts 3100->3101 ((sem PR — teto do #8585)); open-sse/handlers/videoGeneration.ts 1038->1063 (#8581); open-sse/services/accountFallback.ts 1965->1966 (#8628); tests/unit/models-catalog-route.test.ts 1608->1636 (#8610)",
|
||||
"frozen": {
|
||||
"src/sse/handlers/chatHelpers.ts": 1214,
|
||||
"_rebaseline_2026_09_15_13609_mistral_ambiguous_401": "PR #13609 rework (maxmad64bis, bare Mistral 401 soft lockout behind MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT, default off). open-sse/services/accountFallback.ts 2469->2501 (+32): +14 are the change itself (shared-predicate + flag imports, the documented ambiguousAuth field on the checkFallbackError return type, and the flag-gated 401 branch formatted normally instead of the PR's 139-char squeezed configuredRule line); +18 are the lint-staged prettier pass normalizing lines that were already unformatted on the release tip (multi-import, ISO_RETRY_RE, two regex arrays, persistAntigravityFamilyCooldownIfQuota call, applyErrorState guard, trailing commas) — pure formatting, no logic. src/sse/services/auth.ts 3556->3557 (+1): markAccountUnavailable passes connectionId to resolveTerminalConnectionStatus so the soft-strike bound is per connection. The predicate and strike tracker live in the leaf open-sse/services/accountFallback/mistralAmbiguousAuth.ts (under cap). Covered by tests/unit/provider-401-ambiguous-runtime.test.ts (flag off/on, end-to-end through markAccountUnavailable).",
|
||||
"_rebaseline_2026_06_22_4644_deepseek_web_tools": "PR #4644 (BugsBag/robust deepseek-web tool-call parsing): open-sse/executors/deepseek-web.ts 1117->1125 (+8). The new agentic tool-call path emits surrounding text + reasoning before tool_calls and swaps to the dedicated deepseekWebTools.ts parser; the +8 lines are cohesive wiring at the existing transformSSE chokepoint (the parser itself lives in the new deepseekWebTools.ts file, already under cap). The PR's own fast-gate (PR->release) does not run check:file-size, so this surfaced only at release reconcile. Covered by tests/unit/deepseek-web-tools-variants.test.ts + deepseek-web-tools-execute.test.ts.",
|
||||
"_rebaseline_2026_06_23_4712_deepseek_web_tool_results": "PR for #4712 (deepseek-web drops role:tool): open-sse/executors/deepseek-web.ts 1125->1148 (+23). messagesToPrompt() now folds role:\"tool\" results into the single-prompt transcript (recovering the tool name from the preceding assistant tool_calls by tool_call_id) instead of silently dropping them; the lines are cohesive wiring inside the existing function. Covered by tests/unit/deepseek-web-tool-result-prompt-4712.test.ts.",
|
||||
"_rebaseline_2026_06_24_headroom_strategy": "Headroom-aware connection selection (dario technique): combo.ts 3168->3180 (+12 = a new `else if (strategy === \"headroom\")` dispatch branch in handleComboChat that delegates to orderTargetsByHeadroom + its log line, plus the import). The actual logic lives OUT of the god-file: the pure ranker rankByHeadroom/computeHeadroom is the new leaf open-sse/services/combo/headroomRanking.ts (91 LOC, <cap) and the async orderer orderTargetsByHeadroom is appended to the existing open-sse/services/combo/quotaStrategies.ts (<cap) next to its sibling reset-aware/reset-window orderers (reuses their connection-expansion machinery). headroom = 1 - max(util_5h, util_7d) from getSaturation (src/lib/quota/saturationSignals.ts), prefers the connection with the most free capacity. Only the dispatch wiring is irreducible at the existing combo strategy chokepoint (mirrors the reset-aware/reset-window/context-optimized branches); not extractable without hiding the call site. fill-first stays default; all existing strategies untouched. Covered by tests/unit/combo-headroom-ranking.test.ts (pure helper) + tests/unit/combo-headroom-strategy.test.ts (orderer, saturation injected). Structural shrink of combo.ts tracked in #3501.",
|
||||
@@ -455,7 +457,7 @@
|
||||
"open-sse/handlers/search.ts": 1789,
|
||||
"open-sse/mcp-server/schemas/tools.ts": 1621,
|
||||
"open-sse/mcp-server/server.ts": 1572,
|
||||
"open-sse/services/accountFallback.ts": 2493,
|
||||
"open-sse/services/accountFallback.ts": 2507,
|
||||
"open-sse/services/adobeFireflyBrowserLogin.ts": 1401,
|
||||
"open-sse/services/combo.ts": 4080,
|
||||
"open-sse/services/combo/executeTargetAttempt.ts": 1228,
|
||||
@@ -491,7 +493,7 @@
|
||||
"src/shared/constants/providers/apikey/gateways.ts": 1502,
|
||||
"src/shared/services/cliRuntime.ts": 1296,
|
||||
"src/sse/handlers/chat.ts": 2490,
|
||||
"src/sse/services/auth.ts": 3556,
|
||||
"src/sse/services/auth.ts": 3557,
|
||||
"tests/unit/account-fallback-service.test.ts": 2453,
|
||||
"tests/unit/provider-validation-specialty.test.ts": 4656,
|
||||
"open-sse/services/autoCombo/virtualFactory.ts": 1230,
|
||||
|
||||
@@ -46,7 +46,7 @@ A boolean flag is considered **enabled** when its effective value is `"true"`,
|
||||
|
||||
## Flag Catalog
|
||||
|
||||
67 flags across 6 categories. **Default** is the definition default — the value
|
||||
68 flags across 6 categories. **Default** is the definition default — the value
|
||||
used when neither a DB override nor an environment variable is present.
|
||||
|
||||
### Security (10)
|
||||
@@ -93,7 +93,7 @@ used when neither a DB override nor an environment variable is present.
|
||||
| `CAPABILITY_FILTER_ENABLED` | boolean | `false` | Reject requests before dispatch when the target model lacks required capabilities (vision, tools, structured output, context window). Protects direct single-provider requests that bypass the combo-layer compatibility filter. |
|
||||
| `RADAR_ENABLED` | boolean | `false` | Enable the OmniRoute Radar module (catalog feed screens and sync). Off by default; enabling only unlocks the UI — data sync remains a separate opt-in. |
|
||||
|
||||
### Runtime (29)
|
||||
### Runtime (30)
|
||||
|
||||
| Key | Type | Default | Restart | Description |
|
||||
| ------------------------------------------- | ------- | ------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
@@ -126,6 +126,7 @@ used when neither a DB override nor an environment variable is present.
|
||||
| `FREE_BADGE_REQUIRES_PROVIDER_FREE_TIER` | boolean | `false` | | Dashboard provider pages: show the Free badge only on signals the provider honors — drops the display-name heuristic, non-boolean free fields and :free suffixes on registered providers without a documented free tier. Off keeps the historical badge rule. |
|
||||
| `RETRY_AFTER_PROVENANCE_ENABLED` | boolean | `false` | | On aggregated 429/503 unavailable responses, omit `Retry-After` when no concrete future retry time is known (instead of a synthetic 1s), add `error.retry_after_provenance` (`signal` \| `none`), and let combo drain paths read prose retry hints from JSON and plain-text upstream bodies. The field only appears on responses built by `unavailableResponse()`; other 429/503 bodies are unchanged. |
|
||||
| `PROTECTED_PRIORITY_INFRA_502_ENABLED` | boolean | `false` | | When a `priority` combo target marked fallback-only-on-quota-exhaustion stops the combo for a cause that is provably not quota (provider circuit breaker open, predictive latency skip), answer 502 instead of the quota-looking 503. Lockout, cooldown, unavailable, exhaustion and concurrency-cap stops keep 503. |
|
||||
| `MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT` | boolean | `false` | | A bare Mistral 401 (`{"detail":"Unauthorized"}`, no explicit auth signal) is identical for a revoked key and for exhausted quota. When on, it cools the connection down instead of parking it as `expired`, at most 3 times per hour per connection; the next one parks it, so a revoked key still converges. Off by default: every bare Mistral 401 parks the connection as before. |
|
||||
|
||||
### CLI (5)
|
||||
|
||||
@@ -207,7 +208,7 @@ Returns every flag with its effective value, source, and a summary.
|
||||
"requiresRestart": false,
|
||||
"warningLevel": "caution",
|
||||
},
|
||||
// ... all 67 flags
|
||||
// ... all 68 flags
|
||||
],
|
||||
"summary": {
|
||||
"total": 56,
|
||||
|
||||
@@ -104,6 +104,8 @@ import { capScaledCooldownMs } from "./accountFallback/cooldownCap.ts";
|
||||
import { resolveApiKeyForbiddenFallback } from "./accountFallback/nonRetryableUpstream.ts";
|
||||
import * as exactModelLock from "./accountFallback/exactModelLock.ts";
|
||||
import { isCreditsExhaustedWithSharedWallet } from "./accountFallback/sharedWalletCredits.ts";
|
||||
import { isMistralAmbiguous401 } from "./accountFallback/mistralAmbiguousAuth.ts";
|
||||
import { isMistralAmbiguous401SoftLockoutEnabled } from "@/shared/utils/featureFlags";
|
||||
export type ProviderProfile = {
|
||||
baseCooldownMs: number;
|
||||
useUpstreamRetryHints: boolean;
|
||||
@@ -1685,6 +1687,8 @@ export function checkFallbackError(
|
||||
permanent?: boolean;
|
||||
creditsExhausted?: boolean;
|
||||
dailyQuotaExhausted?: boolean;
|
||||
/** #13609: bare Mistral 401 softened to a cooldown (MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT). */
|
||||
ambiguousAuth?: boolean;
|
||||
/** G-02: true when the error originates from an embedded service supervisor (not the upstream AI
|
||||
* provider itself). Callers should apply connection cooldown only — do NOT record a provider
|
||||
* circuit-breaker failure when this flag is set. */
|
||||
@@ -2179,6 +2183,16 @@ export function checkFallbackError(
|
||||
resolveRuleMatchBody(provider, structuredError ?? null, errorStr)
|
||||
)
|
||||
: null;
|
||||
// #13609 (opt-in): a bare Mistral 401 is not proof of a dead key — back off
|
||||
// instead; resolveTerminalConnectionStatus bounds how often (ambiguousAuth).
|
||||
if (
|
||||
status === HTTP_STATUS.UNAUTHORIZED &&
|
||||
!providerMatch &&
|
||||
isMistralAmbiguous401(provider, errorStr) &&
|
||||
isMistralAmbiguous401SoftLockoutEnabled()
|
||||
) {
|
||||
return { ...buildRetryableFallback(RateLimitReason.UNKNOWN), ambiguousAuth: true };
|
||||
}
|
||||
const cooldownMs = providerMatch?.cooldownMs ?? configuredRule.cooldownMs ?? 0;
|
||||
const ruleScope =
|
||||
providerMatch && honorsRuleLockScope(provider) ? providerMatch.scope : undefined;
|
||||
|
||||
58
open-sse/services/accountFallback/mistralAmbiguousAuth.ts
Normal file
58
open-sse/services/accountFallback/mistralAmbiguousAuth.ts
Normal file
@@ -0,0 +1,58 @@
|
||||
/**
|
||||
* Mistral ambiguous 401 (#7638 / #7718 / #13609).
|
||||
*
|
||||
* Mistral answers a revoked key and an exhausted quota with the same bare
|
||||
* `401 {"detail":"Unauthorized"}`. This leaf module (zero imports) is the ONE
|
||||
* implementation of that ambiguity check, shared by the connection-test
|
||||
* diagnosis (`src/app/api/providers/[id]/test/mistralAmbiguousAuth.ts`) and the
|
||||
* runtime fallback path (`checkFallbackError` + `resolveTerminalConnectionStatus`).
|
||||
*
|
||||
* It also holds the bound for the opt-in runtime soft lockout
|
||||
* (MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT): a connection gets at most
|
||||
* MISTRAL_AMBIGUOUS_401_MAX_SOFT_STRIKES soft cooldowns inside the strike window;
|
||||
* the next bare 401 parks it as expired, so a genuinely revoked key converges.
|
||||
*/
|
||||
|
||||
const AUTH_SIGNALS = ["invalid api key", "token invalid", "revoked", "access denied"];
|
||||
|
||||
/** Soft cooldowns allowed per connection inside the strike window. */
|
||||
export const MISTRAL_AMBIGUOUS_401_MAX_SOFT_STRIKES = 3;
|
||||
/** Window, measured from the first strike, in which soft strikes accumulate. */
|
||||
export const MISTRAL_AMBIGUOUS_401_STRIKE_WINDOW_MS = 60 * 60 * 1000;
|
||||
|
||||
/** True for a Mistral 401 body that carries no explicit auth signal (callers check the status). */
|
||||
export function isMistralAmbiguous401(
|
||||
provider: string | null | undefined,
|
||||
errorText: string | null | undefined
|
||||
): boolean {
|
||||
if (provider !== "mistral") return false;
|
||||
const normalized = String(errorText || "").toLowerCase();
|
||||
return !AUTH_SIGNALS.some((signal) => normalized.includes(signal));
|
||||
}
|
||||
|
||||
const strikes = new Map<string, { count: number; firstAt: number }>();
|
||||
|
||||
/**
|
||||
* Record one ambiguous bare 401 for `connectionId` and say whether it may still
|
||||
* be softened (true) or must park the connection (false). Crossing the bound
|
||||
* clears the entry, so a re-authenticated connection starts a fresh count.
|
||||
*/
|
||||
export function takeMistralAmbiguous401SoftStrike(connectionId: string, now = Date.now()): boolean {
|
||||
const entry = strikes.get(connectionId);
|
||||
const current =
|
||||
entry && now - entry.firstAt < MISTRAL_AMBIGUOUS_401_STRIKE_WINDOW_MS
|
||||
? entry
|
||||
: { count: 0, firstAt: now };
|
||||
current.count += 1;
|
||||
if (current.count > MISTRAL_AMBIGUOUS_401_MAX_SOFT_STRIKES) {
|
||||
strikes.delete(connectionId);
|
||||
return false;
|
||||
}
|
||||
strikes.set(connectionId, current);
|
||||
return true;
|
||||
}
|
||||
|
||||
/** Test hook: forget every recorded strike. */
|
||||
export function resetMistralAmbiguous401Strikes(): void {
|
||||
strikes.clear();
|
||||
}
|
||||
@@ -1,3 +1,5 @@
|
||||
import { isMistralAmbiguous401 } from "@omniroute/open-sse/services/accountFallback/mistralAmbiguousAuth.ts";
|
||||
|
||||
/**
|
||||
* #7638: Mistral's quota-exhausted response is `401 {"detail":"Unauthorized"}` — byte-identical
|
||||
* to a genuinely revoked key. Unlike other providers, a bare Mistral 401 with no auth-specific
|
||||
@@ -23,16 +25,6 @@ export interface ClassifyFailureArgs {
|
||||
provider?: string;
|
||||
}
|
||||
|
||||
function isMistralAmbiguous401(provider: string | undefined, normalized: string): boolean {
|
||||
if (provider !== "mistral") return false;
|
||||
const hasAuthSignal =
|
||||
normalized.includes("invalid api key") ||
|
||||
normalized.includes("token invalid") ||
|
||||
normalized.includes("revoked") ||
|
||||
normalized.includes("access denied");
|
||||
return !hasAuthSignal;
|
||||
}
|
||||
|
||||
/** Decides the diagnosis for a 401/403 status: ambiguous (Mistral-only) or the generic auth error. */
|
||||
export function classifyAmbiguousOrAuthError(
|
||||
provider: string | undefined,
|
||||
|
||||
@@ -702,6 +702,18 @@ export const FEATURE_FLAG_DEFINITIONS: FeatureFlagDefinition[] = [
|
||||
requiresRestart: false,
|
||||
warningLevel: "caution",
|
||||
},
|
||||
{
|
||||
key: "MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT",
|
||||
label: "Mistral Ambiguous 401 Soft Lockout",
|
||||
description:
|
||||
'A bare Mistral 401 ({"detail":"Unauthorized"}, no explicit auth signal) is byte-identical for a revoked key and for exhausted quota. When enabled, such a 401 cools the connection down instead of parking it as expired, up to 3 times within an hour; the next one still parks it as expired, so a revoked key converges. Off by default: every bare Mistral 401 parks the connection as expired, as before.',
|
||||
descriptionI18nKey: "featureFlagMistralAmbiguous401SoftLockoutDescription",
|
||||
category: "runtime",
|
||||
defaultValue: "false",
|
||||
type: "boolean",
|
||||
requiresRestart: false,
|
||||
warningLevel: "caution",
|
||||
},
|
||||
|
||||
// ──────────────── CLI (5) ────────────────
|
||||
{
|
||||
|
||||
@@ -272,6 +272,23 @@ export function isOpencodeTransientFailoverBackoffEnabled(): boolean {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Mistral bare-401 bounded soft lockout (#13609). Opt-in: when off, a bare Mistral 401 parks
|
||||
* the connection as expired exactly as before.
|
||||
* Fail closed: an unreadable flag store keeps the pre-flag behavior (disabled).
|
||||
*/
|
||||
export function isMistralAmbiguous401SoftLockoutEnabled(): boolean {
|
||||
try {
|
||||
return isFeatureFlagEnabled("MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT");
|
||||
} catch (error) {
|
||||
console.error(
|
||||
"[featureFlags] Failed to resolve MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT, defaulting to disabled:",
|
||||
error instanceof Error ? error.message : error
|
||||
);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
export function isServerOwnedToolLoopEnabled(
|
||||
reader: (key: string) => boolean = isFeatureFlagEnabled
|
||||
): boolean {
|
||||
|
||||
@@ -3157,11 +3157,12 @@ export async function markAccountUnavailable(
|
||||
|
||||
let terminalStatus = resolveTerminalConnectionStatus(
|
||||
status,
|
||||
result as { permanent?: boolean; creditsExhausted?: boolean },
|
||||
result as { permanent?: boolean; creditsExhausted?: boolean; ambiguousAuth?: boolean },
|
||||
providerErrorType,
|
||||
provider,
|
||||
isPerModelQuotaProvider,
|
||||
errorText
|
||||
errorText,
|
||||
connectionId
|
||||
);
|
||||
// A still-valid access token after a successful refresh is not "expired".
|
||||
// A follow-up 401 (timeout, hop, race) must cooldown, not park the account.
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { PROVIDER_ERROR_TYPES } from "@omniroute/open-sse/services/errorClassifier.ts";
|
||||
import { isCreditsExhausted } from "@omniroute/open-sse/services/accountFallback.ts";
|
||||
import { takeMistralAmbiguous401SoftStrike } from "@omniroute/open-sse/services/accountFallback/mistralAmbiguousAuth.ts";
|
||||
import { resolveProviderId, WEB_COOKIE_PROVIDERS } from "@/shared/constants/providers";
|
||||
|
||||
// #8200: cookie-auth providers (perplexity-web, grok-web, ...) use a rotating browser
|
||||
@@ -71,11 +72,12 @@ function isExpiredAuthFailure(
|
||||
|
||||
export function resolveTerminalConnectionStatus(
|
||||
status: number,
|
||||
result: { permanent?: boolean; creditsExhausted?: boolean },
|
||||
result: { permanent?: boolean; creditsExhausted?: boolean; ambiguousAuth?: boolean },
|
||||
providerErrorType: string | null = null,
|
||||
provider: string | null = null,
|
||||
isPerModelQuotaProvider = false,
|
||||
errorText: string = ""
|
||||
errorText: string = "",
|
||||
connectionId: string | null = null
|
||||
): string | null {
|
||||
if (shouldParkCreditsExhausted(status, result, isPerModelQuotaProvider, errorText)) {
|
||||
return "credits_exhausted";
|
||||
@@ -87,6 +89,12 @@ export function resolveTerminalConnectionStatus(
|
||||
return "banned";
|
||||
}
|
||||
if (isExpiredAuthFailure(status, providerErrorType, provider)) {
|
||||
// #13609: checkFallbackError only sets ambiguousAuth for a bare Mistral 401
|
||||
// with MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT on. Bounded per connection: past
|
||||
// the strike limit the connection parks as expired like any other 401.
|
||||
if (status === 401 && result.ambiguousAuth && connectionId) {
|
||||
if (takeMistralAmbiguous401SoftStrike(connectionId)) return null;
|
||||
}
|
||||
return "expired";
|
||||
}
|
||||
return null;
|
||||
|
||||
@@ -40,7 +40,7 @@ const {
|
||||
// the dead ONEPROXY_ENABLED (readerless since the 1proxy purge, #12091)
|
||||
// brought it back to 53. UNIVERSAL_CONTEXT_HANDOFF_ENABLED bumped it to 54.
|
||||
// #13641 added SEARCH_STATS_HIDE_DELETED_CONNECTIONS, bumping the count to 56.
|
||||
const EXPECTED_FEATURE_FLAG_COUNT = 67;
|
||||
const EXPECTED_FEATURE_FLAG_COUNT = 68;
|
||||
|
||||
// ──────────────────────────────────────────────────────
|
||||
// Test group 1 — Flag definitions registry
|
||||
@@ -231,6 +231,17 @@ describe("featureFlagDefinitions", () => {
|
||||
assert.strictEqual(def.requiresRestart, false);
|
||||
});
|
||||
|
||||
it("defines MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT as an opt-in runtime boolean flag disabled by default", () => {
|
||||
const def = FEATURE_FLAG_DEFINITIONS.find(
|
||||
(d) => d.key === "MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT"
|
||||
);
|
||||
assert.ok(def, "MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT should exist");
|
||||
assert.strictEqual(def.category, "runtime");
|
||||
assert.strictEqual(def.type, "boolean");
|
||||
assert.strictEqual(def.defaultValue, "false");
|
||||
assert.strictEqual(def.requiresRestart, false);
|
||||
});
|
||||
|
||||
it("defines network rotation shared-egress guard as a network boolean flag enabled by default", () => {
|
||||
const def = FEATURE_FLAG_DEFINITIONS.find(
|
||||
(d) => d.key === "NETWORK_ROTATION_SHARED_EGRESS_GUARD"
|
||||
|
||||
227
tests/unit/provider-401-ambiguous-runtime.test.ts
Normal file
227
tests/unit/provider-401-ambiguous-runtime.test.ts
Normal file
@@ -0,0 +1,227 @@
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import fs from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
|
||||
// #13609 rework: a bare Mistral 401 is byte-identical for a revoked key and an
|
||||
// exhausted quota (#7638). With MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT on, it cools the
|
||||
// connection down instead of parking it as expired — at most 3 times per hour per
|
||||
// connection, then it parks, so a revoked key still converges. Flag off (default):
|
||||
// every bare Mistral 401 parks the connection as before.
|
||||
|
||||
const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-13609-mistral-401-"));
|
||||
process.env.DATA_DIR = TEST_DATA_DIR;
|
||||
|
||||
const FLAG = "MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT";
|
||||
const BARE = '{"detail":"Unauthorized"}';
|
||||
|
||||
const core = await import("../../src/lib/db/core.ts");
|
||||
const providersDb = await import("../../src/lib/db/providers.ts");
|
||||
const auth = await import("../../src/sse/services/auth.ts");
|
||||
const { checkFallbackError } = await import("../../open-sse/services/accountFallback.ts");
|
||||
const { resolveTerminalConnectionStatus } =
|
||||
await import("../../src/sse/services/authTerminalStatus.ts");
|
||||
const { classifyProviderError, PROVIDER_ERROR_TYPES } =
|
||||
await import("../../open-sse/services/errorClassifier.ts");
|
||||
const { setOperatorProviderErrorRules } =
|
||||
await import("../../open-sse/config/providerErrorRules.ts");
|
||||
const {
|
||||
isMistralAmbiguous401,
|
||||
takeMistralAmbiguous401SoftStrike,
|
||||
resetMistralAmbiguous401Strikes,
|
||||
MISTRAL_AMBIGUOUS_401_MAX_SOFT_STRIKES,
|
||||
MISTRAL_AMBIGUOUS_401_STRIKE_WINDOW_MS,
|
||||
} = await import("../../open-sse/services/accountFallback/mistralAmbiguousAuth.ts");
|
||||
const { classifyFailure } = await import("../../src/app/api/providers/[id]/test/route.ts");
|
||||
|
||||
const priorFlag = process.env[FLAG];
|
||||
|
||||
function setFlag(value: string | undefined) {
|
||||
if (value === undefined) delete process.env[FLAG];
|
||||
else process.env[FLAG] = value;
|
||||
}
|
||||
|
||||
test.beforeEach(() => {
|
||||
resetMistralAmbiguous401Strikes();
|
||||
setFlag(undefined);
|
||||
});
|
||||
|
||||
test.after(() => {
|
||||
setFlag(priorFlag);
|
||||
core.resetDbInstance();
|
||||
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });
|
||||
});
|
||||
|
||||
function bareMistral401() {
|
||||
return checkFallbackError(401, BARE, 0, null, "mistral", null, null, null);
|
||||
}
|
||||
|
||||
test("one shared predicate: the connection-test diagnosis and the runtime agree", () => {
|
||||
assert.equal(isMistralAmbiguous401("mistral", BARE), true);
|
||||
assert.equal(isMistralAmbiguous401("mistral", "Invalid API key"), false);
|
||||
assert.equal(isMistralAmbiguous401("openai", BARE), false);
|
||||
assert.equal(
|
||||
classifyFailure({ error: BARE, statusCode: 401, provider: "mistral" }).type,
|
||||
"upstream_ambiguous_auth_or_quota"
|
||||
);
|
||||
assert.equal(
|
||||
classifyFailure({ error: "Token invalid", statusCode: 401, provider: "mistral" }).type,
|
||||
"upstream_auth_error"
|
||||
);
|
||||
});
|
||||
|
||||
test("flag off: a bare Mistral 401 stays an auth_error and resolves expired", () => {
|
||||
const r = bareMistral401();
|
||||
assert.equal(r.reason, "auth_error");
|
||||
assert.equal(r.ambiguousAuth, undefined);
|
||||
const type = classifyProviderError(401, BARE, "mistral");
|
||||
assert.equal(
|
||||
resolveTerminalConnectionStatus(401, r, type, "mistral", false, BARE, "c1"),
|
||||
"expired"
|
||||
);
|
||||
});
|
||||
|
||||
test("flag on: a bare Mistral 401 backs off instead of asserting an auth failure", () => {
|
||||
setFlag("true");
|
||||
const r = bareMistral401();
|
||||
assert.notEqual(r.reason, "auth_error");
|
||||
assert.equal(r.ambiguousAuth, true);
|
||||
assert.equal(r.shouldFallback, true);
|
||||
assert.ok(!r.permanent);
|
||||
assert.ok(r.cooldownMs > 0, "a real cooldown, not an immediate reselect");
|
||||
});
|
||||
|
||||
test("flag on: explicit auth signals, other providers and operator rules are unchanged", () => {
|
||||
setFlag("true");
|
||||
for (const body of ["Invalid API key", "token invalid", "revoked", "access denied"]) {
|
||||
const r = checkFallbackError(401, body, 0, null, "mistral", null, null, null);
|
||||
assert.equal(r.reason, "auth_error", body);
|
||||
assert.equal(r.ambiguousAuth, undefined, body);
|
||||
}
|
||||
assert.equal(
|
||||
checkFallbackError(401, BARE, 0, null, "openai", null, null, null).reason,
|
||||
"auth_error"
|
||||
);
|
||||
setOperatorProviderErrorRules({
|
||||
mistral: [{ status: 401, match: "unauthorized", scope: "connection", cooldownMs: 99999 }],
|
||||
});
|
||||
try {
|
||||
const r = bareMistral401();
|
||||
assert.equal(r.reason, "quota_exhausted");
|
||||
assert.equal(r.cooldownMs, 99999);
|
||||
assert.equal(r.ambiguousAuth, undefined);
|
||||
} finally {
|
||||
setOperatorProviderErrorRules({});
|
||||
}
|
||||
});
|
||||
|
||||
test("strike bound: 3 soft strikes per window, the 4th parks and restarts the count", () => {
|
||||
const t0 = 1_000_000;
|
||||
for (let i = 1; i <= MISTRAL_AMBIGUOUS_401_MAX_SOFT_STRIKES; i++) {
|
||||
assert.equal(takeMistralAmbiguous401SoftStrike("conn", t0 + i), true, `strike ${i}`);
|
||||
}
|
||||
assert.equal(takeMistralAmbiguous401SoftStrike("conn", t0 + 10), false, "bound reached");
|
||||
assert.equal(takeMistralAmbiguous401SoftStrike("conn", t0 + 11), true, "fresh count after park");
|
||||
assert.equal(takeMistralAmbiguous401SoftStrike("other", t0 + 12), true, "per connection");
|
||||
// Strikes older than the window do not accumulate.
|
||||
resetMistralAmbiguous401Strikes();
|
||||
for (let i = 0; i < MISTRAL_AMBIGUOUS_401_MAX_SOFT_STRIKES; i++) {
|
||||
takeMistralAmbiguous401SoftStrike("slow", t0);
|
||||
}
|
||||
assert.equal(
|
||||
takeMistralAmbiguous401SoftStrike("slow", t0 + MISTRAL_AMBIGUOUS_401_STRIKE_WINDOW_MS),
|
||||
true
|
||||
);
|
||||
});
|
||||
|
||||
test("resolveTerminalConnectionStatus ignores ambiguousAuth without a connection id or for other types", () => {
|
||||
const r = { ambiguousAuth: true };
|
||||
assert.equal(
|
||||
resolveTerminalConnectionStatus(
|
||||
401,
|
||||
r,
|
||||
PROVIDER_ERROR_TYPES.UNAUTHORIZED,
|
||||
"mistral",
|
||||
false,
|
||||
BARE
|
||||
),
|
||||
"expired"
|
||||
);
|
||||
assert.equal(
|
||||
resolveTerminalConnectionStatus(
|
||||
401,
|
||||
{},
|
||||
PROVIDER_ERROR_TYPES.UNAUTHORIZED,
|
||||
"mistral",
|
||||
false,
|
||||
BARE,
|
||||
"c"
|
||||
),
|
||||
"expired"
|
||||
);
|
||||
});
|
||||
|
||||
async function createMistralConnection() {
|
||||
const conn = await providersDb.createProviderConnection({
|
||||
provider: "mistral",
|
||||
authType: "apikey",
|
||||
apiKey: "mistral-test-key",
|
||||
isActive: true,
|
||||
testStatus: "active",
|
||||
});
|
||||
return String(conn.id);
|
||||
}
|
||||
|
||||
async function expireCooldown(connId: string) {
|
||||
// What selection-time auto-decay does once rateLimitedUntil has passed.
|
||||
await providersDb.updateProviderConnection(connId, {
|
||||
rateLimitedUntil: null,
|
||||
testStatus: "active",
|
||||
});
|
||||
}
|
||||
|
||||
test("markAccountUnavailable, flag off: one bare Mistral 401 parks the connection as expired", async () => {
|
||||
const connId = await createMistralConnection();
|
||||
|
||||
await auth.markAccountUnavailable(connId, 401, BARE, "mistral", "mistral-large-latest");
|
||||
|
||||
const after = await providersDb.getProviderConnectionById(connId);
|
||||
assert.equal(after.testStatus, "expired");
|
||||
});
|
||||
|
||||
test("markAccountUnavailable, flag on: cooldown for 3 bare 401s, then expired", async () => {
|
||||
setFlag("true");
|
||||
const connId = await createMistralConnection();
|
||||
|
||||
for (let strike = 1; strike <= MISTRAL_AMBIGUOUS_401_MAX_SOFT_STRIKES; strike++) {
|
||||
await auth.markAccountUnavailable(connId, 401, BARE, "mistral", "mistral-large-latest");
|
||||
const cooling = await providersDb.getProviderConnectionById(connId);
|
||||
assert.equal(cooling.testStatus, "unavailable", `strike ${strike} cools down`);
|
||||
assert.ok(
|
||||
new Date(String(cooling.rateLimitedUntil)).getTime() > Date.now(),
|
||||
`strike ${strike} sets a future rateLimitedUntil`
|
||||
);
|
||||
await expireCooldown(connId);
|
||||
}
|
||||
|
||||
await auth.markAccountUnavailable(connId, 401, BARE, "mistral", "mistral-large-latest");
|
||||
const parked = await providersDb.getProviderConnectionById(connId);
|
||||
assert.equal(parked.testStatus, "expired", "a persistent bare 401 still converges");
|
||||
});
|
||||
|
||||
test("markAccountUnavailable, flag on: an explicit auth signal parks on the first 401", async () => {
|
||||
setFlag("true");
|
||||
const connId = await createMistralConnection();
|
||||
|
||||
await auth.markAccountUnavailable(
|
||||
connId,
|
||||
401,
|
||||
"Invalid API key",
|
||||
"mistral",
|
||||
"mistral-large-latest"
|
||||
);
|
||||
|
||||
const after = await providersDb.getProviderConnectionById(connId);
|
||||
assert.equal(after.testStatus, "expired");
|
||||
});
|
||||
@@ -68,7 +68,7 @@ describe("isServerOwnedToolLoopEnabled wrapper", () => {
|
||||
|
||||
describe("feature-flags-settings count update", () => {
|
||||
it("flag count matches updated expected value", () => {
|
||||
assert.equal(FEATURE_FLAG_DEFINITIONS.length, 67);
|
||||
assert.equal(FEATURE_FLAG_DEFINITIONS.length, 68);
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
Reference in New Issue
Block a user