fix(providers): stop parking Mistral connections on a bare 401 with no clear auth failure (#13609)

Behind the new `MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT` flag (default off), a bare Mistral 401 (`{"detail":"Unauthorized"}`, identical for a revoked key and an exhausted quota) gets a retryable cooldown instead of parking the connection as `expired`; after three soft strikes within an hour the next bare 401 parks it, so revocation still converges.

Maintainer rework before merge (kept the idea, no default behavior change):
- The predicate is shared with the connection-test module instead of duplicated; the squeezed 139-char line that dodged the file-size gate is formatted normally and the growth is rebaselined honestly with an annotation.

Validated first on the combined board of all 38 PRs of this batch (10 merged as-is, 28 after the maintainer rework) on top of release/v3.8.51 c0f92ec: typecheck:core, check:open-sse-typecheck and check:dashboard-typecheck clean; ESLint clean on every changed file; file-size (rebaselined for the combined growth), complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync, migration-numbering and i18n new-key gates green; 735 focused node:test cases with the only batch-caused failure (a flag-count assertion) fixed. Then re-validated alone on the fresh release tip right before this merge: ESLint on the changed files, typecheck:core, check:open-sse-typecheck, the file-size/complexity/changelog gates and this PR's own tests.

Thanks @maxmad64bis!
This commit is contained in:
Dizzle
2026-09-16 02:19:47 +02:00
committed by GitHub
parent 13f44af6e5
commit 94d27e44fe
13 changed files with 365 additions and 21 deletions

View File

@@ -0,0 +1 @@
- **fix(providers):** opt-in `MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT` flag (default off): a bare Mistral 401 with no explicit auth signal (identical for a revoked key and an exhausted quota) cools the connection down instead of parking it as expired, at most 3 times per hour per connection before it parks, so a revoked key still converges; the ambiguity check is now one implementation shared by the connection test and the runtime ([#13609](https://github.com/diegosouzapw/OmniRoute/pull/13609)) — thanks @maxmad64bis

View File

@@ -1,4 +1,5 @@
{
"_rebaseline_2026_09_15_13609_combined_growth": "Combined growth of the 2026-09-15 maxmad64bis uplift batch (each PR rebaselined its own growth; the merged sum is larger): open-sse/services/accountFallback.ts->2507. Every hunk is flag-gated or a verified fix covered by that PR's tests; see the batch report.",
"_rebaseline_2026_09_15_13602_combined_growth": "Combined growth of the 2026-09-15 maxmad64bis uplift batch (each PR rebaselined its own growth; the merged sum is larger): src/sse/handlers/chatHelpers.ts->1214. Every hunk is flag-gated or a verified fix covered by that PR's tests; see the batch report.",
"_rebaseline_2026_09_15_13580_combined_growth": "Combined growth of the 2026-09-15 maxmad64bis uplift batch (each PR rebaselined its own growth; the merged sum is larger): src/sse/handlers/chatHelpers.ts->1202. Every hunk is flag-gated or a verified fix covered by that PR's tests; see the batch report.",
"_rebaseline_2026_09_13_13581_pool_egress_observation": "PR #13581 own growth: src/app/(dashboard)/dashboard/settings/components/ProxyRegistryManager.tsx 1475->1477 (+2 = the PoolEgressObservation import and its one-line mount under the pool members label). The observation itself lives outside the frozen file, all under cap: PoolEgressObservation.tsx, the dedicated GET /api/settings/proxies/pool/egress-observation route, src/lib/proxyPoolEgressObservation.ts and getPoolEgressObservation in src/lib/db/proxyLogs.ts. Only the mount point is irreducible. Covered by tests/unit/proxy-pool-egress-observation.test.ts, tests/unit/proxy-pool-egress-observation-route.test.ts and tests/unit/ui/PoolEgressObservation.test.tsx.",
@@ -345,6 +346,7 @@
"_rebaseline_2026_07_27_v3849_train1h": "Merge-train 1H (31 PRs) — owner-approved 2026-07-27. Two distinct causes, kept separate on purpose: (1) GENUINE irreducible growth at existing chokepoints — providerLimits/auth (#8632 Kimi quota-reset recovery), rateLimitManager (#8616 idle wedged limiters), models-catalog-route.test (#8610 OpenCode Go effort aliases); (2) COLLISION with #8585, which banked shrinks measured on the pre-train release tip while 30 sibling PRs in the SAME train grew those files again — chat/accountFallback (#8628), chatCore (#8613), videoGeneration (#8581), imageGeneration. The zero-headroom frozen entries cannot absorb either. Ceilings re-pinned to the post-merge tip; #8612 (also in this train) automates shrink-banking so this self-inflicted drift stops recurring. Detail: src/lib/usage/providerLimits.ts 1006->1013 (#8632); src/sse/services/auth.ts 2492->2508 (#8632); open-sse/services/rateLimitManager.ts 1014->1060 (#8616); src/sse/handlers/chat.ts 1842->1845 (#8628); open-sse/handlers/chatCore.ts 4939->4955 (#8613); open-sse/handlers/imageGeneration.ts 3100->3101 ((sem PR — teto do #8585)); open-sse/handlers/videoGeneration.ts 1038->1063 (#8581); open-sse/services/accountFallback.ts 1965->1966 (#8628); tests/unit/models-catalog-route.test.ts 1608->1636 (#8610)",
"frozen": {
"src/sse/handlers/chatHelpers.ts": 1214,
"_rebaseline_2026_09_15_13609_mistral_ambiguous_401": "PR #13609 rework (maxmad64bis, bare Mistral 401 soft lockout behind MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT, default off). open-sse/services/accountFallback.ts 2469->2501 (+32): +14 are the change itself (shared-predicate + flag imports, the documented ambiguousAuth field on the checkFallbackError return type, and the flag-gated 401 branch formatted normally instead of the PR's 139-char squeezed configuredRule line); +18 are the lint-staged prettier pass normalizing lines that were already unformatted on the release tip (multi-import, ISO_RETRY_RE, two regex arrays, persistAntigravityFamilyCooldownIfQuota call, applyErrorState guard, trailing commas) — pure formatting, no logic. src/sse/services/auth.ts 3556->3557 (+1): markAccountUnavailable passes connectionId to resolveTerminalConnectionStatus so the soft-strike bound is per connection. The predicate and strike tracker live in the leaf open-sse/services/accountFallback/mistralAmbiguousAuth.ts (under cap). Covered by tests/unit/provider-401-ambiguous-runtime.test.ts (flag off/on, end-to-end through markAccountUnavailable).",
"_rebaseline_2026_06_22_4644_deepseek_web_tools": "PR #4644 (BugsBag/robust deepseek-web tool-call parsing): open-sse/executors/deepseek-web.ts 1117->1125 (+8). The new agentic tool-call path emits surrounding text + reasoning before tool_calls and swaps to the dedicated deepseekWebTools.ts parser; the +8 lines are cohesive wiring at the existing transformSSE chokepoint (the parser itself lives in the new deepseekWebTools.ts file, already under cap). The PR's own fast-gate (PR->release) does not run check:file-size, so this surfaced only at release reconcile. Covered by tests/unit/deepseek-web-tools-variants.test.ts + deepseek-web-tools-execute.test.ts.",
"_rebaseline_2026_06_23_4712_deepseek_web_tool_results": "PR for #4712 (deepseek-web drops role:tool): open-sse/executors/deepseek-web.ts 1125->1148 (+23). messagesToPrompt() now folds role:\"tool\" results into the single-prompt transcript (recovering the tool name from the preceding assistant tool_calls by tool_call_id) instead of silently dropping them; the lines are cohesive wiring inside the existing function. Covered by tests/unit/deepseek-web-tool-result-prompt-4712.test.ts.",
"_rebaseline_2026_06_24_headroom_strategy": "Headroom-aware connection selection (dario technique): combo.ts 3168->3180 (+12 = a new `else if (strategy === \"headroom\")` dispatch branch in handleComboChat that delegates to orderTargetsByHeadroom + its log line, plus the import). The actual logic lives OUT of the god-file: the pure ranker rankByHeadroom/computeHeadroom is the new leaf open-sse/services/combo/headroomRanking.ts (91 LOC, <cap) and the async orderer orderTargetsByHeadroom is appended to the existing open-sse/services/combo/quotaStrategies.ts (<cap) next to its sibling reset-aware/reset-window orderers (reuses their connection-expansion machinery). headroom = 1 - max(util_5h, util_7d) from getSaturation (src/lib/quota/saturationSignals.ts), prefers the connection with the most free capacity. Only the dispatch wiring is irreducible at the existing combo strategy chokepoint (mirrors the reset-aware/reset-window/context-optimized branches); not extractable without hiding the call site. fill-first stays default; all existing strategies untouched. Covered by tests/unit/combo-headroom-ranking.test.ts (pure helper) + tests/unit/combo-headroom-strategy.test.ts (orderer, saturation injected). Structural shrink of combo.ts tracked in #3501.",
@@ -455,7 +457,7 @@
"open-sse/handlers/search.ts": 1789,
"open-sse/mcp-server/schemas/tools.ts": 1621,
"open-sse/mcp-server/server.ts": 1572,
"open-sse/services/accountFallback.ts": 2493,
"open-sse/services/accountFallback.ts": 2507,
"open-sse/services/adobeFireflyBrowserLogin.ts": 1401,
"open-sse/services/combo.ts": 4080,
"open-sse/services/combo/executeTargetAttempt.ts": 1228,
@@ -491,7 +493,7 @@
"src/shared/constants/providers/apikey/gateways.ts": 1502,
"src/shared/services/cliRuntime.ts": 1296,
"src/sse/handlers/chat.ts": 2490,
"src/sse/services/auth.ts": 3556,
"src/sse/services/auth.ts": 3557,
"tests/unit/account-fallback-service.test.ts": 2453,
"tests/unit/provider-validation-specialty.test.ts": 4656,
"open-sse/services/autoCombo/virtualFactory.ts": 1230,

View File

@@ -46,7 +46,7 @@ A boolean flag is considered **enabled** when its effective value is `"true"`,
## Flag Catalog
67 flags across 6 categories. **Default** is the definition default — the value
68 flags across 6 categories. **Default** is the definition default — the value
used when neither a DB override nor an environment variable is present.
### Security (10)
@@ -93,7 +93,7 @@ used when neither a DB override nor an environment variable is present.
| `CAPABILITY_FILTER_ENABLED` | boolean | `false` | Reject requests before dispatch when the target model lacks required capabilities (vision, tools, structured output, context window). Protects direct single-provider requests that bypass the combo-layer compatibility filter. |
| `RADAR_ENABLED` | boolean | `false` | Enable the OmniRoute Radar module (catalog feed screens and sync). Off by default; enabling only unlocks the UI — data sync remains a separate opt-in. |
### Runtime (29)
### Runtime (30)
| Key | Type | Default | Restart | Description |
| ------------------------------------------- | ------- | ------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
@@ -126,6 +126,7 @@ used when neither a DB override nor an environment variable is present.
| `FREE_BADGE_REQUIRES_PROVIDER_FREE_TIER` | boolean | `false` | | Dashboard provider pages: show the Free badge only on signals the provider honors — drops the display-name heuristic, non-boolean free fields and :free suffixes on registered providers without a documented free tier. Off keeps the historical badge rule. |
| `RETRY_AFTER_PROVENANCE_ENABLED` | boolean | `false` | | On aggregated 429/503 unavailable responses, omit `Retry-After` when no concrete future retry time is known (instead of a synthetic 1s), add `error.retry_after_provenance` (`signal` \| `none`), and let combo drain paths read prose retry hints from JSON and plain-text upstream bodies. The field only appears on responses built by `unavailableResponse()`; other 429/503 bodies are unchanged. |
| `PROTECTED_PRIORITY_INFRA_502_ENABLED` | boolean | `false` | | When a `priority` combo target marked fallback-only-on-quota-exhaustion stops the combo for a cause that is provably not quota (provider circuit breaker open, predictive latency skip), answer 502 instead of the quota-looking 503. Lockout, cooldown, unavailable, exhaustion and concurrency-cap stops keep 503. |
| `MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT` | boolean | `false` | | A bare Mistral 401 (`{"detail":"Unauthorized"}`, no explicit auth signal) is identical for a revoked key and for exhausted quota. When on, it cools the connection down instead of parking it as `expired`, at most 3 times per hour per connection; the next one parks it, so a revoked key still converges. Off by default: every bare Mistral 401 parks the connection as before. |
### CLI (5)
@@ -207,7 +208,7 @@ Returns every flag with its effective value, source, and a summary.
"requiresRestart": false,
"warningLevel": "caution",
},
// ... all 67 flags
// ... all 68 flags
],
"summary": {
"total": 56,

View File

@@ -104,6 +104,8 @@ import { capScaledCooldownMs } from "./accountFallback/cooldownCap.ts";
import { resolveApiKeyForbiddenFallback } from "./accountFallback/nonRetryableUpstream.ts";
import * as exactModelLock from "./accountFallback/exactModelLock.ts";
import { isCreditsExhaustedWithSharedWallet } from "./accountFallback/sharedWalletCredits.ts";
import { isMistralAmbiguous401 } from "./accountFallback/mistralAmbiguousAuth.ts";
import { isMistralAmbiguous401SoftLockoutEnabled } from "@/shared/utils/featureFlags";
export type ProviderProfile = {
baseCooldownMs: number;
useUpstreamRetryHints: boolean;
@@ -1685,6 +1687,8 @@ export function checkFallbackError(
permanent?: boolean;
creditsExhausted?: boolean;
dailyQuotaExhausted?: boolean;
/** #13609: bare Mistral 401 softened to a cooldown (MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT). */
ambiguousAuth?: boolean;
/** G-02: true when the error originates from an embedded service supervisor (not the upstream AI
* provider itself). Callers should apply connection cooldown only — do NOT record a provider
* circuit-breaker failure when this flag is set. */
@@ -2179,6 +2183,16 @@ export function checkFallbackError(
resolveRuleMatchBody(provider, structuredError ?? null, errorStr)
)
: null;
// #13609 (opt-in): a bare Mistral 401 is not proof of a dead key — back off
// instead; resolveTerminalConnectionStatus bounds how often (ambiguousAuth).
if (
status === HTTP_STATUS.UNAUTHORIZED &&
!providerMatch &&
isMistralAmbiguous401(provider, errorStr) &&
isMistralAmbiguous401SoftLockoutEnabled()
) {
return { ...buildRetryableFallback(RateLimitReason.UNKNOWN), ambiguousAuth: true };
}
const cooldownMs = providerMatch?.cooldownMs ?? configuredRule.cooldownMs ?? 0;
const ruleScope =
providerMatch && honorsRuleLockScope(provider) ? providerMatch.scope : undefined;

View File

@@ -0,0 +1,58 @@
/**
* Mistral ambiguous 401 (#7638 / #7718 / #13609).
*
* Mistral answers a revoked key and an exhausted quota with the same bare
* `401 {"detail":"Unauthorized"}`. This leaf module (zero imports) is the ONE
* implementation of that ambiguity check, shared by the connection-test
* diagnosis (`src/app/api/providers/[id]/test/mistralAmbiguousAuth.ts`) and the
* runtime fallback path (`checkFallbackError` + `resolveTerminalConnectionStatus`).
*
* It also holds the bound for the opt-in runtime soft lockout
* (MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT): a connection gets at most
* MISTRAL_AMBIGUOUS_401_MAX_SOFT_STRIKES soft cooldowns inside the strike window;
* the next bare 401 parks it as expired, so a genuinely revoked key converges.
*/
const AUTH_SIGNALS = ["invalid api key", "token invalid", "revoked", "access denied"];
/** Soft cooldowns allowed per connection inside the strike window. */
export const MISTRAL_AMBIGUOUS_401_MAX_SOFT_STRIKES = 3;
/** Window, measured from the first strike, in which soft strikes accumulate. */
export const MISTRAL_AMBIGUOUS_401_STRIKE_WINDOW_MS = 60 * 60 * 1000;
/** True for a Mistral 401 body that carries no explicit auth signal (callers check the status). */
export function isMistralAmbiguous401(
provider: string | null | undefined,
errorText: string | null | undefined
): boolean {
if (provider !== "mistral") return false;
const normalized = String(errorText || "").toLowerCase();
return !AUTH_SIGNALS.some((signal) => normalized.includes(signal));
}
const strikes = new Map<string, { count: number; firstAt: number }>();
/**
* Record one ambiguous bare 401 for `connectionId` and say whether it may still
* be softened (true) or must park the connection (false). Crossing the bound
* clears the entry, so a re-authenticated connection starts a fresh count.
*/
export function takeMistralAmbiguous401SoftStrike(connectionId: string, now = Date.now()): boolean {
const entry = strikes.get(connectionId);
const current =
entry && now - entry.firstAt < MISTRAL_AMBIGUOUS_401_STRIKE_WINDOW_MS
? entry
: { count: 0, firstAt: now };
current.count += 1;
if (current.count > MISTRAL_AMBIGUOUS_401_MAX_SOFT_STRIKES) {
strikes.delete(connectionId);
return false;
}
strikes.set(connectionId, current);
return true;
}
/** Test hook: forget every recorded strike. */
export function resetMistralAmbiguous401Strikes(): void {
strikes.clear();
}

View File

@@ -1,3 +1,5 @@
import { isMistralAmbiguous401 } from "@omniroute/open-sse/services/accountFallback/mistralAmbiguousAuth.ts";
/**
* #7638: Mistral's quota-exhausted response is `401 {"detail":"Unauthorized"}` — byte-identical
* to a genuinely revoked key. Unlike other providers, a bare Mistral 401 with no auth-specific
@@ -23,16 +25,6 @@ export interface ClassifyFailureArgs {
provider?: string;
}
function isMistralAmbiguous401(provider: string | undefined, normalized: string): boolean {
if (provider !== "mistral") return false;
const hasAuthSignal =
normalized.includes("invalid api key") ||
normalized.includes("token invalid") ||
normalized.includes("revoked") ||
normalized.includes("access denied");
return !hasAuthSignal;
}
/** Decides the diagnosis for a 401/403 status: ambiguous (Mistral-only) or the generic auth error. */
export function classifyAmbiguousOrAuthError(
provider: string | undefined,

View File

@@ -702,6 +702,18 @@ export const FEATURE_FLAG_DEFINITIONS: FeatureFlagDefinition[] = [
requiresRestart: false,
warningLevel: "caution",
},
{
key: "MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT",
label: "Mistral Ambiguous 401 Soft Lockout",
description:
'A bare Mistral 401 ({"detail":"Unauthorized"}, no explicit auth signal) is byte-identical for a revoked key and for exhausted quota. When enabled, such a 401 cools the connection down instead of parking it as expired, up to 3 times within an hour; the next one still parks it as expired, so a revoked key converges. Off by default: every bare Mistral 401 parks the connection as expired, as before.',
descriptionI18nKey: "featureFlagMistralAmbiguous401SoftLockoutDescription",
category: "runtime",
defaultValue: "false",
type: "boolean",
requiresRestart: false,
warningLevel: "caution",
},
// ──────────────── CLI (5) ────────────────
{

View File

@@ -272,6 +272,23 @@ export function isOpencodeTransientFailoverBackoffEnabled(): boolean {
}
}
/**
* Mistral bare-401 bounded soft lockout (#13609). Opt-in: when off, a bare Mistral 401 parks
* the connection as expired exactly as before.
* Fail closed: an unreadable flag store keeps the pre-flag behavior (disabled).
*/
export function isMistralAmbiguous401SoftLockoutEnabled(): boolean {
try {
return isFeatureFlagEnabled("MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT");
} catch (error) {
console.error(
"[featureFlags] Failed to resolve MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT, defaulting to disabled:",
error instanceof Error ? error.message : error
);
return false;
}
}
export function isServerOwnedToolLoopEnabled(
reader: (key: string) => boolean = isFeatureFlagEnabled
): boolean {

View File

@@ -3157,11 +3157,12 @@ export async function markAccountUnavailable(
let terminalStatus = resolveTerminalConnectionStatus(
status,
result as { permanent?: boolean; creditsExhausted?: boolean },
result as { permanent?: boolean; creditsExhausted?: boolean; ambiguousAuth?: boolean },
providerErrorType,
provider,
isPerModelQuotaProvider,
errorText
errorText,
connectionId
);
// A still-valid access token after a successful refresh is not "expired".
// A follow-up 401 (timeout, hop, race) must cooldown, not park the account.

View File

@@ -1,5 +1,6 @@
import { PROVIDER_ERROR_TYPES } from "@omniroute/open-sse/services/errorClassifier.ts";
import { isCreditsExhausted } from "@omniroute/open-sse/services/accountFallback.ts";
import { takeMistralAmbiguous401SoftStrike } from "@omniroute/open-sse/services/accountFallback/mistralAmbiguousAuth.ts";
import { resolveProviderId, WEB_COOKIE_PROVIDERS } from "@/shared/constants/providers";
// #8200: cookie-auth providers (perplexity-web, grok-web, ...) use a rotating browser
@@ -71,11 +72,12 @@ function isExpiredAuthFailure(
export function resolveTerminalConnectionStatus(
status: number,
result: { permanent?: boolean; creditsExhausted?: boolean },
result: { permanent?: boolean; creditsExhausted?: boolean; ambiguousAuth?: boolean },
providerErrorType: string | null = null,
provider: string | null = null,
isPerModelQuotaProvider = false,
errorText: string = ""
errorText: string = "",
connectionId: string | null = null
): string | null {
if (shouldParkCreditsExhausted(status, result, isPerModelQuotaProvider, errorText)) {
return "credits_exhausted";
@@ -87,6 +89,12 @@ export function resolveTerminalConnectionStatus(
return "banned";
}
if (isExpiredAuthFailure(status, providerErrorType, provider)) {
// #13609: checkFallbackError only sets ambiguousAuth for a bare Mistral 401
// with MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT on. Bounded per connection: past
// the strike limit the connection parks as expired like any other 401.
if (status === 401 && result.ambiguousAuth && connectionId) {
if (takeMistralAmbiguous401SoftStrike(connectionId)) return null;
}
return "expired";
}
return null;

View File

@@ -40,7 +40,7 @@ const {
// the dead ONEPROXY_ENABLED (readerless since the 1proxy purge, #12091)
// brought it back to 53. UNIVERSAL_CONTEXT_HANDOFF_ENABLED bumped it to 54.
// #13641 added SEARCH_STATS_HIDE_DELETED_CONNECTIONS, bumping the count to 56.
const EXPECTED_FEATURE_FLAG_COUNT = 67;
const EXPECTED_FEATURE_FLAG_COUNT = 68;
// ──────────────────────────────────────────────────────
// Test group 1 — Flag definitions registry
@@ -231,6 +231,17 @@ describe("featureFlagDefinitions", () => {
assert.strictEqual(def.requiresRestart, false);
});
it("defines MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT as an opt-in runtime boolean flag disabled by default", () => {
const def = FEATURE_FLAG_DEFINITIONS.find(
(d) => d.key === "MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT"
);
assert.ok(def, "MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT should exist");
assert.strictEqual(def.category, "runtime");
assert.strictEqual(def.type, "boolean");
assert.strictEqual(def.defaultValue, "false");
assert.strictEqual(def.requiresRestart, false);
});
it("defines network rotation shared-egress guard as a network boolean flag enabled by default", () => {
const def = FEATURE_FLAG_DEFINITIONS.find(
(d) => d.key === "NETWORK_ROTATION_SHARED_EGRESS_GUARD"

View File

@@ -0,0 +1,227 @@
import test from "node:test";
import assert from "node:assert/strict";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
// #13609 rework: a bare Mistral 401 is byte-identical for a revoked key and an
// exhausted quota (#7638). With MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT on, it cools the
// connection down instead of parking it as expired — at most 3 times per hour per
// connection, then it parks, so a revoked key still converges. Flag off (default):
// every bare Mistral 401 parks the connection as before.
const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-13609-mistral-401-"));
process.env.DATA_DIR = TEST_DATA_DIR;
const FLAG = "MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT";
const BARE = '{"detail":"Unauthorized"}';
const core = await import("../../src/lib/db/core.ts");
const providersDb = await import("../../src/lib/db/providers.ts");
const auth = await import("../../src/sse/services/auth.ts");
const { checkFallbackError } = await import("../../open-sse/services/accountFallback.ts");
const { resolveTerminalConnectionStatus } =
await import("../../src/sse/services/authTerminalStatus.ts");
const { classifyProviderError, PROVIDER_ERROR_TYPES } =
await import("../../open-sse/services/errorClassifier.ts");
const { setOperatorProviderErrorRules } =
await import("../../open-sse/config/providerErrorRules.ts");
const {
isMistralAmbiguous401,
takeMistralAmbiguous401SoftStrike,
resetMistralAmbiguous401Strikes,
MISTRAL_AMBIGUOUS_401_MAX_SOFT_STRIKES,
MISTRAL_AMBIGUOUS_401_STRIKE_WINDOW_MS,
} = await import("../../open-sse/services/accountFallback/mistralAmbiguousAuth.ts");
const { classifyFailure } = await import("../../src/app/api/providers/[id]/test/route.ts");
const priorFlag = process.env[FLAG];
function setFlag(value: string | undefined) {
if (value === undefined) delete process.env[FLAG];
else process.env[FLAG] = value;
}
test.beforeEach(() => {
resetMistralAmbiguous401Strikes();
setFlag(undefined);
});
test.after(() => {
setFlag(priorFlag);
core.resetDbInstance();
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });
});
function bareMistral401() {
return checkFallbackError(401, BARE, 0, null, "mistral", null, null, null);
}
test("one shared predicate: the connection-test diagnosis and the runtime agree", () => {
assert.equal(isMistralAmbiguous401("mistral", BARE), true);
assert.equal(isMistralAmbiguous401("mistral", "Invalid API key"), false);
assert.equal(isMistralAmbiguous401("openai", BARE), false);
assert.equal(
classifyFailure({ error: BARE, statusCode: 401, provider: "mistral" }).type,
"upstream_ambiguous_auth_or_quota"
);
assert.equal(
classifyFailure({ error: "Token invalid", statusCode: 401, provider: "mistral" }).type,
"upstream_auth_error"
);
});
test("flag off: a bare Mistral 401 stays an auth_error and resolves expired", () => {
const r = bareMistral401();
assert.equal(r.reason, "auth_error");
assert.equal(r.ambiguousAuth, undefined);
const type = classifyProviderError(401, BARE, "mistral");
assert.equal(
resolveTerminalConnectionStatus(401, r, type, "mistral", false, BARE, "c1"),
"expired"
);
});
test("flag on: a bare Mistral 401 backs off instead of asserting an auth failure", () => {
setFlag("true");
const r = bareMistral401();
assert.notEqual(r.reason, "auth_error");
assert.equal(r.ambiguousAuth, true);
assert.equal(r.shouldFallback, true);
assert.ok(!r.permanent);
assert.ok(r.cooldownMs > 0, "a real cooldown, not an immediate reselect");
});
test("flag on: explicit auth signals, other providers and operator rules are unchanged", () => {
setFlag("true");
for (const body of ["Invalid API key", "token invalid", "revoked", "access denied"]) {
const r = checkFallbackError(401, body, 0, null, "mistral", null, null, null);
assert.equal(r.reason, "auth_error", body);
assert.equal(r.ambiguousAuth, undefined, body);
}
assert.equal(
checkFallbackError(401, BARE, 0, null, "openai", null, null, null).reason,
"auth_error"
);
setOperatorProviderErrorRules({
mistral: [{ status: 401, match: "unauthorized", scope: "connection", cooldownMs: 99999 }],
});
try {
const r = bareMistral401();
assert.equal(r.reason, "quota_exhausted");
assert.equal(r.cooldownMs, 99999);
assert.equal(r.ambiguousAuth, undefined);
} finally {
setOperatorProviderErrorRules({});
}
});
test("strike bound: 3 soft strikes per window, the 4th parks and restarts the count", () => {
const t0 = 1_000_000;
for (let i = 1; i <= MISTRAL_AMBIGUOUS_401_MAX_SOFT_STRIKES; i++) {
assert.equal(takeMistralAmbiguous401SoftStrike("conn", t0 + i), true, `strike ${i}`);
}
assert.equal(takeMistralAmbiguous401SoftStrike("conn", t0 + 10), false, "bound reached");
assert.equal(takeMistralAmbiguous401SoftStrike("conn", t0 + 11), true, "fresh count after park");
assert.equal(takeMistralAmbiguous401SoftStrike("other", t0 + 12), true, "per connection");
// Strikes older than the window do not accumulate.
resetMistralAmbiguous401Strikes();
for (let i = 0; i < MISTRAL_AMBIGUOUS_401_MAX_SOFT_STRIKES; i++) {
takeMistralAmbiguous401SoftStrike("slow", t0);
}
assert.equal(
takeMistralAmbiguous401SoftStrike("slow", t0 + MISTRAL_AMBIGUOUS_401_STRIKE_WINDOW_MS),
true
);
});
test("resolveTerminalConnectionStatus ignores ambiguousAuth without a connection id or for other types", () => {
const r = { ambiguousAuth: true };
assert.equal(
resolveTerminalConnectionStatus(
401,
r,
PROVIDER_ERROR_TYPES.UNAUTHORIZED,
"mistral",
false,
BARE
),
"expired"
);
assert.equal(
resolveTerminalConnectionStatus(
401,
{},
PROVIDER_ERROR_TYPES.UNAUTHORIZED,
"mistral",
false,
BARE,
"c"
),
"expired"
);
});
async function createMistralConnection() {
const conn = await providersDb.createProviderConnection({
provider: "mistral",
authType: "apikey",
apiKey: "mistral-test-key",
isActive: true,
testStatus: "active",
});
return String(conn.id);
}
async function expireCooldown(connId: string) {
// What selection-time auto-decay does once rateLimitedUntil has passed.
await providersDb.updateProviderConnection(connId, {
rateLimitedUntil: null,
testStatus: "active",
});
}
test("markAccountUnavailable, flag off: one bare Mistral 401 parks the connection as expired", async () => {
const connId = await createMistralConnection();
await auth.markAccountUnavailable(connId, 401, BARE, "mistral", "mistral-large-latest");
const after = await providersDb.getProviderConnectionById(connId);
assert.equal(after.testStatus, "expired");
});
test("markAccountUnavailable, flag on: cooldown for 3 bare 401s, then expired", async () => {
setFlag("true");
const connId = await createMistralConnection();
for (let strike = 1; strike <= MISTRAL_AMBIGUOUS_401_MAX_SOFT_STRIKES; strike++) {
await auth.markAccountUnavailable(connId, 401, BARE, "mistral", "mistral-large-latest");
const cooling = await providersDb.getProviderConnectionById(connId);
assert.equal(cooling.testStatus, "unavailable", `strike ${strike} cools down`);
assert.ok(
new Date(String(cooling.rateLimitedUntil)).getTime() > Date.now(),
`strike ${strike} sets a future rateLimitedUntil`
);
await expireCooldown(connId);
}
await auth.markAccountUnavailable(connId, 401, BARE, "mistral", "mistral-large-latest");
const parked = await providersDb.getProviderConnectionById(connId);
assert.equal(parked.testStatus, "expired", "a persistent bare 401 still converges");
});
test("markAccountUnavailable, flag on: an explicit auth signal parks on the first 401", async () => {
setFlag("true");
const connId = await createMistralConnection();
await auth.markAccountUnavailable(
connId,
401,
"Invalid API key",
"mistral",
"mistral-large-latest"
);
const after = await providersDb.getProviderConnectionById(connId);
assert.equal(after.testStatus, "expired");
});

View File

@@ -68,7 +68,7 @@ describe("isServerOwnedToolLoopEnabled wrapper", () => {
describe("feature-flags-settings count update", () => {
it("flag count matches updated expected value", () => {
assert.equal(FEATURE_FLAG_DEFINITIONS.length, 67);
assert.equal(FEATURE_FLAG_DEFINITIONS.length, 68);
});
});