From c75e293a474c7a71d597e543c97a1577dbad0ac0 Mon Sep 17 00:00:00 2001 From: "Bob.Hou" Date: Fri, 11 Sep 2026 18:27:33 -0400 Subject: [PATCH 01/77] fix(api): thread X-OmniRoute-Fallback-Attempts through combo chat (#12339) (#13038) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Right call not to reuse the combo loop's `fallbackCount`: it only increments after a leg fails or is skipped, so the second target would still report 0 at dispatch. Stamping the ordered index (or round-robin offset) at the gate is the only place the number is actually known. This PR also carries the batch's file-size rebaseline, since it merges first and the ceiling has to cover every intermediate state. --- Validated in one consolidated worktree cut from `release/v3.8.51`, boarded with the other 19 PRs of this batch. Two in-batch conflicts, both additive and resolved by keeping each side: the `ENVIRONMENT.md` table (#13035 + #13011) and the `chatHelpers.ts` import block (#12975 on the tip + #13017). - `typecheck:core` clean; `check:dashboard-typecheck` OK (206 pre-existing, within baseline); `check:changelog-integrity` OK; `check:docs-counts` migrations ✓ - complexity 2816 / baseline 3218 and cognitive-complexity 1271 / baseline 1437 — both under baseline - 531 of 532 focused assertions green across the batch's 46 test files - `check-file-size` rebaselined for the batch's real growth (annotation `_rebaseline_2026_09_11_mergebatch_v3851_houminxi`, landed on #13038), attributed per PR The single red is **not this batch**: `tests/unit/combo/quota-weighted-strategy.test.ts` → "A/B isolation: 7 hard-empty + 2 at 0.5% + 1 at 40%, floor=1" asserts an order between two connections of identical weight and flakes on the pure tip too — 2 failures in 4 runs at `origin/release/v3.8.51` with nothing from this batch applied. ⚠️ base-red inherited: #12732 — `Docs Gates`, `Merge integrity`, `No new ESLint warnings`, `Unit Tests fast-path` and `Fast Quality Gates` reproduce on the pure tip (provider count 356 vs the 358 the modules define, SKILL.md drift, and `open-sse/utils/stream.ts` at 3115 > frozen 3098, untouched here). Thanks @HouMinXi — the live evidence on these (X500 logs, `storage.sqlite` state, real `/v1/models` probes, the 36-minute outage write-up) is what let a 20-PR batch be reviewed as a unit. --- .../fixes/13038-fallback-attempts-chat.md | 1 + config/quality/file-size-baseline.json | 12 +-- open-sse/handlers/chatCore.ts | 3 + .../chatCore/nonStreamingResponseHeaders.ts | 2 + .../chatCore/streamingResponseHeaders.ts | 2 + .../services/combo/comboCompatFallback.ts | 11 ++- open-sse/services/combo/executeTargetGates.ts | 3 +- open-sse/services/combo/roundRobinCombo.ts | 4 +- open-sse/services/combo/runtimeUnits.ts | 5 ++ open-sse/services/combo/types.ts | 4 +- src/sse/handlers/chat.ts | 5 ++ src/sse/handlers/chatHelpers.ts | 2 + ...core-nonstreaming-response-headers.test.ts | 13 ++++ ...hatcore-streaming-response-headers.test.ts | 23 +++++- ...mbo-compat-fallback-attempts-12339.test.ts | 54 ++++++++++++++ ...mbo-runtimeunits-diagnostics-11462.test.ts | 73 ++++++++++++++++++- .../unit/combo/execute-target-attempt.test.ts | 67 +++++++++++++++++ tests/unit/combo/execute-target-gates.test.ts | 66 +++++++++++++++++ tests/unit/omniroute-decision-header.test.ts | 62 +++++++++++++++- 19 files changed, 392 insertions(+), 20 deletions(-) create mode 100644 changelog.d/fixes/13038-fallback-attempts-chat.md create mode 100644 tests/unit/combo-compat-fallback-attempts-12339.test.ts diff --git a/changelog.d/fixes/13038-fallback-attempts-chat.md b/changelog.d/fixes/13038-fallback-attempts-chat.md new file mode 100644 index 0000000000..2b93d009b5 --- /dev/null +++ b/changelog.d/fixes/13038-fallback-attempts-chat.md @@ -0,0 +1 @@ +- **fix(api):** thread `X-OmniRoute-Fallback-Attempts` through combo chat completions so streaming and non-streaming responses report how many prior legs were attempted ([#13038](https://github.com/diegosouzapw/OmniRoute/pull/13038)) diff --git a/config/quality/file-size-baseline.json b/config/quality/file-size-baseline.json index 0b46fbf4a6..20b6c9f0aa 100644 --- a/config/quality/file-size-baseline.json +++ b/config/quality/file-size-baseline.json @@ -1,4 +1,5 @@ { + "_rebaseline_2026_09_11_mergebatch_v3851_houminxi": "/merge-batch 2026-09-11 (v3.8.51), batch by HouMinXi. Final combined values, set on the first PR merged so every intermediate state is covered. open-sse/handlers/chatCore.ts 6036->6144: #13069 routes the non-streaming leg through the same provider-failure classification, model lockout and credential-refresh path the streaming leg already used (+443/-340 = +103 net; it extracts applyProviderFailureClassification and wires both legs to it, which is what #13043 reported missing), plus #13050 stamping that the client asked for SSE before the web_search fallback flips stream off (+6) and #13038 threading the dispatched target index (+3). src/sse/services/auth.ts 3488->3542: #13017 adds the explicit-pin one-shot probe for a recoverable inactive row with its 60s storm gate (+42 net) and #13061 makes a grok-cli 402 a connection-wide shared-wallet signal instead of a per-model billing miss (+12 net). src/sse/handlers/chat.ts 2458->2462: #13038 (+5). open-sse/services/combo/executeTargetAttempt.ts 1205->1212: #13006 feeds the 402 it already classified into the quota cache instead of dropping it (+7). open-sse/services/accountFallback.ts 2468->2469: #13060 adds the Cline re-auth phrase to OAUTH_INVALID_TOKEN_SIGNALS (+1). open-sse/utils/stream.ts is deliberately NOT rebaselined: already 3115 > 3098 on the pure tip with zero contribution from this batch (base-red #12732, owned by /sweep-reds). The file also carried \"open-sse/handlers/chatCore.ts\" twice (6026 and 6036); JSON keeps the last, so the first was dead weight any writer could have picked instead. Collapsed to one entry at the live value. Covered by 531 focused assertions across the batch's 46 test files.", "_rebaseline_2026_09_11_12358_chat_pipeline_custom_node": "PR #12358 own test growth: tests/integration/chat-pipeline.test.ts 1648->1736 (+88). One new integration case, \"#11884 chat pipeline sends a custom node's edited Chat API type upstream\": it seeds a custom OpenAI-compatible node with an edited Chat/Responses API type, stubs fetch, drives handleChatCore and asserts the upstream request carries the live connection setting rather than the format baked into the node id at creation. Irreducible at this layer — the point of the test is the full route-to-upstream path, which is what #11884 regressed. Nothing else in the file changed. Covered by the case itself plus tests/unit/chat-helpers.test.ts (28/28).", "_rebaseline_2026_09_10_12975_rotation_correlation_id": "PR #12975 own growth: open-sse/executors/base.ts 1751->1753 (+2) and open-sse/handlers/chatCore.ts 6021->6024 (+3). The opencode rotation lines carry the request correlationId: one optional ExecuteInput field and one correlationId argument at each of the three executor.execute call sites in handleChatCore. Irreducible plumbing at existing call sites; the rotation logic itself lives in open-sse/executors/opencode.ts and the new leaf predicates (under cap). Covered by tests/unit/opencode-transient-rotation.test.ts and tests/unit/chat-correlation-id-exhaustion.test.ts.", "_rebaseline_2026_09_11_mergebatch_v3851_maxmad_opencode": "/merge-batch 2026-09-11 (v3.8.51), PRs #13141, #13146 and #12975 by maxmad64bis. src/sse/services/auth.ts 3450->3488 (+38): #13146 adds the narrow ruleScope===model branch to markAccountUnavailable (gated on status 400; every other status keeps its path) plus the HONORS_RULE_LOCK_SCOPE_PROVIDERS opencode entry, taking it to 3464; #12975 then adds buildExhaustionOptions so the exhaustion log lines carry the request correlationId (+24). open-sse/services/accountFallback.ts 2467->2468 (+1): #13141 routes hasFutureRateLimitUntil through the tolerant epoch normalizer; #13146 is net zero there (+16/-16). open-sse/executors/base.ts 1751->1753 (+2): #12975 adds the optional ExecuteInput.correlationId field with its doc comment. src/sse/handlers/chat.ts is NOT rebaselined: #12975 threads correlationId through the three executor call sites (+2) but the file lands at 2452, still under its existing 2458 freeze. open-sse/utils/stream.ts is deliberately NOT rebaselined either: it is already 3115 > 3098 on the pure tip with zero contribution from this batch (base-red #12732, owned by /sweep-reds). No new branching beyond the two guarded branches named above. Covered by tests/unit/combo-predicates-epoch-cooldown.test.ts, opencode-400-model-unavailable.test.ts, agentrouter-error-rules.test.ts, opencode-transient-rotation.test.ts and chat-correlation-id-exhaustion.test.ts.", @@ -430,16 +431,15 @@ "open-sse/executors/codex.ts": 1505, "open-sse/executors/cursor.ts": 1759, "open-sse/executors/muse-spark-web.ts": 1405, - "open-sse/handlers/chatCore.ts": 6026, - "open-sse/handlers/chatCore.ts": 6036, + "open-sse/handlers/chatCore.ts": 6144, "open-sse/handlers/imageGeneration.ts": 3259, "open-sse/handlers/search.ts": 1789, "open-sse/mcp-server/schemas/tools.ts": 1621, "open-sse/mcp-server/server.ts": 1572, - "open-sse/services/accountFallback.ts": 2468, + "open-sse/services/accountFallback.ts": 2469, "open-sse/services/adobeFireflyBrowserLogin.ts": 1401, "open-sse/services/combo.ts": 4080, - "open-sse/services/combo/executeTargetAttempt.ts": 1205, + "open-sse/services/combo/executeTargetAttempt.ts": 1212, "open-sse/translator/response/openai-responses.ts": 1466, "open-sse/utils/cursorAgentProtobuf.ts": 1547, "open-sse/utils/proxyFetch.ts": 1271, @@ -471,8 +471,8 @@ "src/shared/components/RequestLoggerV2.tsx": 1718, "src/shared/constants/providers/apikey/gateways.ts": 1502, "src/shared/services/cliRuntime.ts": 1296, - "src/sse/handlers/chat.ts": 2458, - "src/sse/services/auth.ts": 3488, + "src/sse/handlers/chat.ts": 2462, + "src/sse/services/auth.ts": 3542, "tests/unit/account-fallback-service.test.ts": 2453, "tests/unit/provider-validation-specialty.test.ts": 4656, "open-sse/services/autoCombo/virtualFactory.ts": 1219, diff --git a/open-sse/handlers/chatCore.ts b/open-sse/handlers/chatCore.ts index 5d1206f541..20078c522d 100644 --- a/open-sse/handlers/chatCore.ts +++ b/open-sse/handlers/chatCore.ts @@ -495,6 +495,7 @@ export async function handleChatCore({ // applied to a CLONE of `body` at the persistAttemptLogs sink (surface 1) — // the model-bound `body` itself is never touched. videoBridgeLog = undefined, + fallbackAttempts = undefined, }) { let { provider, model, extendedContext } = modelInfo; // #12150 P1b: true iff the video-bridge guardrail rendered >=1 transcript @@ -5433,6 +5434,7 @@ export async function handleChatCore({ requestId: skillRequestId, compressionResponseMeta, comboStrategy, + fallbackAttempts, }); // #6426: align response body `model` with the `X-OmniRoute-Model` header // (both must be the resolved backend model). Some upstreams (notably legacy @@ -5612,6 +5614,7 @@ export async function handleChatCore({ pendingRequestId, compressionResponseMeta, comboStrategy, + fallbackAttempts, }); // The streaming headers (turn-state included, when present) are committed to diff --git a/open-sse/handlers/chatCore/nonStreamingResponseHeaders.ts b/open-sse/handlers/chatCore/nonStreamingResponseHeaders.ts index 1a58391806..1991b97ee0 100644 --- a/open-sse/handlers/chatCore/nonStreamingResponseHeaders.ts +++ b/open-sse/handlers/chatCore/nonStreamingResponseHeaders.ts @@ -21,6 +21,7 @@ export function buildNonStreamingResponseHeaders( requestId: string | null | undefined; compressionResponseMeta?: string | null | undefined; comboStrategy?: string | null | undefined; + fallbackAttempts?: number; }, deps: { attachOmniRouteMetaHeaders: typeof defaultAttachMeta; now: () => number } = { attachOmniRouteMetaHeaders: defaultAttachMeta, @@ -40,6 +41,7 @@ export function buildNonStreamingResponseHeaders( costUsd: args.estimatedCost, requestId: args.requestId, strategy: args.comboStrategy ?? "single", + ...(args.fallbackAttempts !== undefined ? { fallbackAttempts: args.fallbackAttempts } : {}), }); if (args.compressionResponseMeta) { responseHeaders[OMNIROUTE_RESPONSE_HEADERS.compression] = args.compressionResponseMeta; diff --git a/open-sse/handlers/chatCore/streamingResponseHeaders.ts b/open-sse/handlers/chatCore/streamingResponseHeaders.ts index d9ba555fc9..811ac0945e 100644 --- a/open-sse/handlers/chatCore/streamingResponseHeaders.ts +++ b/open-sse/handlers/chatCore/streamingResponseHeaders.ts @@ -19,6 +19,7 @@ export function assembleStreamingResponseHeaders( pendingRequestId: string; compressionResponseMeta?: string | null | undefined; comboStrategy?: string | null | undefined; + fallbackAttempts?: number; }, buildStreamingResponseHeaders: typeof defaultBuildStreaming = defaultBuildStreaming ): Record { @@ -31,6 +32,7 @@ export function assembleStreamingResponseHeaders( usage: null, costUsd: 0, strategy: args.comboStrategy ?? "single", + ...(args.fallbackAttempts !== undefined ? { fallbackAttempts: args.fallbackAttempts } : {}), }), "x-omniroute-request-id": args.pendingRequestId, }; diff --git a/open-sse/services/combo/comboCompatFallback.ts b/open-sse/services/combo/comboCompatFallback.ts index 127f2c354c..0344c844ce 100644 --- a/open-sse/services/combo/comboCompatFallback.ts +++ b/open-sse/services/combo/comboCompatFallback.ts @@ -1,4 +1,9 @@ -import type { ComboLogger, HandleSingleModel, IsModelAvailable, ResolvedComboTarget } from "./types"; +import type { + ComboLogger, + HandleSingleModel, + IsModelAvailable, + ResolvedComboTarget, +} from "./types"; /** * Last-resort fallback tier for combo routing (#6238). @@ -40,7 +45,8 @@ export async function attemptCompatRejectedFallback( ): Promise { if (rejectedTargets.length === 0) return null; - for (const target of rejectedTargets) { + for (let i = 0; i < rejectedTargets.length; i++) { + const target = rejectedTargets[i]; if (ctx.isModelAvailable) { const available = await ctx.isModelAvailable(target.modelStr, target); if (!available) { @@ -67,6 +73,7 @@ export async function attemptCompatRejectedFallback( const result = await ctx.handleSingleModel(body, target.modelStr, { ...target, effectiveComboStrategy: ctx.strategy, + fallbackAttempts: i, }); if (result.ok) { ctx.log.info("COMBO", `Last-resort compat fallback succeeded via ${target.modelStr}`); diff --git a/open-sse/services/combo/executeTargetGates.ts b/open-sse/services/combo/executeTargetGates.ts index b0d5f3edf4..0a7ada3a24 100644 --- a/open-sse/services/combo/executeTargetGates.ts +++ b/open-sse/services/combo/executeTargetGates.ts @@ -129,8 +129,9 @@ export async function evaluateExecuteTargetGates(opts: { ...target, allowRateLimitedConnection: true, modelAbortSignal: abortSignal, + fallbackAttempts: i, } - : { ...target, modelAbortSignal: abortSignal }; + : { ...target, modelAbortSignal: abortSignal, fallbackAttempts: i }; if (target.connectionId && !allowRateLimitedConnection) { const persistedSkip = await resolvePersistedConnectionCooldownSkipReason( diff --git a/open-sse/services/combo/roundRobinCombo.ts b/open-sse/services/combo/roundRobinCombo.ts index 8547784a30..2cc3c6895f 100644 --- a/open-sse/services/combo/roundRobinCombo.ts +++ b/open-sse/services/combo/roundRobinCombo.ts @@ -487,8 +487,8 @@ export async function handleRoundRobinCombo({ const allowRateLimitedConnection = Boolean(provider && provider !== "unknown") && transientRateLimitedProviders.has(provider); const targetForAttempt = allowRateLimitedConnection - ? { ...target, allowRateLimitedConnection: true } - : target; + ? { ...target, allowRateLimitedConnection: true, fallbackAttempts: offset } + : { ...target, fallbackAttempts: offset }; // Pre-check availability if (isModelAvailable) { diff --git a/open-sse/services/combo/runtimeUnits.ts b/open-sse/services/combo/runtimeUnits.ts index f6321a68d2..632aad4bb5 100644 --- a/open-sse/services/combo/runtimeUnits.ts +++ b/open-sse/services/combo/runtimeUnits.ts @@ -79,6 +79,7 @@ async function executeModelUnit(args: { isModelAvailable?: IsModelAvailable; failoverBeforeRetry: unknown; effectiveComboStrategy: string; + fallbackAttempts: number; }): Promise { if (args.isModelAvailable) { const available = await args.isModelAvailable(args.unit.modelStr, args.unit); @@ -88,6 +89,7 @@ async function executeModelUnit(args: { ...args.unit, effectiveComboStrategy: args.effectiveComboStrategy, failoverBeforeRetry: args.failoverBeforeRetry, + fallbackAttempts: args.fallbackAttempts, }); } @@ -142,6 +144,7 @@ async function executeRuntimeUnit(args: { nesting: ComboNestingContext; failoverBeforeRetry: unknown; effectiveComboStrategy: string; + fallbackAttempts: number; }): Promise { if (args.unit.kind === "model") { return executeModelUnit({ @@ -151,6 +154,7 @@ async function executeRuntimeUnit(args: { isModelAvailable: args.isModelAvailable, failoverBeforeRetry: args.failoverBeforeRetry, effectiveComboStrategy: args.effectiveComboStrategy, + fallbackAttempts: args.fallbackAttempts, }); } return executeComboRefUnit({ @@ -289,6 +293,7 @@ export async function executeRuntimeUnitCombo(args: { nesting: args.nesting, failoverBeforeRetry: args.config.failoverBeforeRetry, effectiveComboStrategy: effectiveStrategy, + fallbackAttempts: fallbackCount, }); lastResponse = response; if (response.ok) { diff --git a/open-sse/services/combo/types.ts b/open-sse/services/combo/types.ts index 15800b25b5..04ba20e8aa 100644 --- a/open-sse/services/combo/types.ts +++ b/open-sse/services/combo/types.ts @@ -60,8 +60,10 @@ export type SingleModelTarget = modelAbortSignal?: AbortSignal | null; /** True when this target was selected via context-cache session pinning. */ modelPinned?: boolean; + /** Prior combo legs already attempted before this dispatch (#12339). */ + fallbackAttempts?: number; }) - | { modelAbortSignal: AbortSignal }; + | { modelAbortSignal: AbortSignal; fallbackAttempts?: number }; export type HandleSingleModel = ( body: Record, diff --git a/src/sse/handlers/chat.ts b/src/sse/handlers/chat.ts index fcd565133a..d3ff15d444 100644 --- a/src/sse/handlers/chat.ts +++ b/src/sse/handlers/chat.ts @@ -1133,6 +1133,7 @@ async function handleChatImplementation( providerId?: string | null; effectiveComboStrategy?: string | null; modelAbortSignal?: AbortSignal | null; + fallbackAttempts?: number; } ) => handleSingleModelChat( @@ -1180,6 +1181,7 @@ async function handleChatImplementation( // entry (trackPendingRequest(false) never runs) — live incident, // log id 1784418258231-14961a. modelAbortSignal: target?.modelAbortSignal ?? null, + fallbackAttempts: target?.fallbackAttempts, }, target?.effectiveComboStrategy ?? combo.strategy, true @@ -1392,6 +1394,7 @@ async function handleSingleModelChat( * the signal used for the actual dispatch, not left unused. */ modelAbortSignal?: AbortSignal | null; + fallbackAttempts?: number; } = {}, comboStrategy: string | null = null, isCombo: boolean = false @@ -1465,6 +1468,7 @@ async function handleSingleModelChat( videoBridgeLog: runtimeOptions.videoBridgeLog, // #7360 follow-up — see the primary handleSingleModel closure above. modelAbortSignal: target?.modelAbortSignal ?? null, + fallbackAttempts: target?.fallbackAttempts, }, resolvedTarget?.effectiveComboStrategy ?? redirectCombo.strategy ?? "priority", false @@ -1957,6 +1961,7 @@ async function handleSingleModelChat( reasoningTransportFallback: runtimeOptions.reasoningTransportFallback ?? "drop", managedLease: runtimeOptions.managedLease ?? null, videoBridgeLog: runtimeOptions.videoBridgeLog, + fallbackAttempts: runtimeOptions.fallbackAttempts, }, runtimeOptions ); diff --git a/src/sse/handlers/chatHelpers.ts b/src/sse/handlers/chatHelpers.ts index c29300b30d..1a6b293ddf 100644 --- a/src/sse/handlers/chatHelpers.ts +++ b/src/sse/handlers/chatHelpers.ts @@ -455,6 +455,7 @@ export async function executeChatWithBreaker({ // for every non-video request. Passed straight through to handleChatCore; // see its own destructure default for the shape and consumers. videoBridgeLog = undefined, + fallbackAttempts = undefined, }: ExecuteChatWithBreakerOptions): Promise { let tlsFingerprintUsed = false; const normalizedTrafficType: TrafficType = @@ -515,6 +516,7 @@ export async function executeChatWithBreaker({ reasoningTransportFallback, managedLease, videoBridgeLog, + fallbackAttempts, skipResourcePressureGuard: true, onCredentialsRefreshed: async (newCreds: any) => { await updateProviderCredentials(credentials.connectionId, { diff --git a/tests/unit/chatcore-nonstreaming-response-headers.test.ts b/tests/unit/chatcore-nonstreaming-response-headers.test.ts index b0ba09b59d..a8511e2a18 100644 --- a/tests/unit/chatcore-nonstreaming-response-headers.test.ts +++ b/tests/unit/chatcore-nonstreaming-response-headers.test.ts @@ -87,3 +87,16 @@ test("compression meta present → compression header set to that value", () => ); assert.ok(Object.values(h).includes("engine:x; source=header")); }); + +test("forwards fallbackAttempts into the non-streaming meta payload", () => { + const { deps, metaCalls } = makeDeps(); + buildNonStreamingResponseHeaders(baseArgs({ fallbackAttempts: 3 }), deps); + assert.equal(metaCalls.length, 1); + assert.equal(metaCalls[0].meta.fallbackAttempts, 3); +}); + +test("omitted fallbackAttempts does not invent a count", () => { + const { deps, metaCalls } = makeDeps(); + buildNonStreamingResponseHeaders(baseArgs(), deps); + assert.equal("fallbackAttempts" in metaCalls[0].meta, false); +}); diff --git a/tests/unit/chatcore-streaming-response-headers.test.ts b/tests/unit/chatcore-streaming-response-headers.test.ts index 8b714ca4eb..52168cd281 100644 --- a/tests/unit/chatcore-streaming-response-headers.test.ts +++ b/tests/unit/chatcore-streaming-response-headers.test.ts @@ -6,9 +6,8 @@ import { test } from "node:test"; import assert from "node:assert/strict"; -const { assembleStreamingResponseHeaders } = await import( - "../../open-sse/handlers/chatCore/streamingResponseHeaders.ts" -); +const { assembleStreamingResponseHeaders } = + await import("../../open-sse/handlers/chatCore/streamingResponseHeaders.ts"); function makeBuild() { const calls: Array<{ headers: unknown; meta: Record }> = []; @@ -51,7 +50,10 @@ test("buildStreamingResponseHeaders receives zeroed latency/usage/cost and cache test("no compression meta → no compression header", () => { const { build } = makeBuild(); - const h = assembleStreamingResponseHeaders(baseArgs({ compressionResponseMeta: undefined }), build); + const h = assembleStreamingResponseHeaders( + baseArgs({ compressionResponseMeta: undefined }), + build + ); assert.ok(!Object.values(h).includes("engine:z")); }); @@ -63,3 +65,16 @@ test("compression meta present → compression header set", () => { ); assert.ok(Object.values(h).includes("engine:z; source=routing")); }); + +test("forwards fallbackAttempts into the streaming meta payload", () => { + const { build, calls } = makeBuild(); + assembleStreamingResponseHeaders(baseArgs({ fallbackAttempts: 2 }), build); + assert.equal(calls.length, 1); + assert.equal(calls[0].meta.fallbackAttempts, 2); +}); + +test("omitted fallbackAttempts does not invent a count", () => { + const { build, calls } = makeBuild(); + assembleStreamingResponseHeaders(baseArgs(), build); + assert.equal("fallbackAttempts" in calls[0].meta, false); +}); diff --git a/tests/unit/combo-compat-fallback-attempts-12339.test.ts b/tests/unit/combo-compat-fallback-attempts-12339.test.ts new file mode 100644 index 0000000000..debbbefc5d --- /dev/null +++ b/tests/unit/combo-compat-fallback-attempts-12339.test.ts @@ -0,0 +1,54 @@ +import test from "node:test"; +import assert from "node:assert/strict"; +import { attemptCompatRejectedFallback } from "../../open-sse/services/combo/comboCompatFallback.ts"; +import type { ResolvedComboTarget } from "../../open-sse/services/combo/types.ts"; + +function modelTarget(overrides: Partial = {}): ResolvedComboTarget { + return { + kind: "model", + stepId: "s1", + executionKey: "ek-1", + modelStr: "openai/compat-b", + provider: "openai", + providerId: null, + connectionId: "c1", + weight: 1, + label: null, + ...overrides, + }; +} + +test("compat fallback stamps fallbackAttempts from the rejected-target index", async () => { + const seen: Array<{ model: string; fallbackAttempts?: number }> = []; + const targets = [ + modelTarget({ executionKey: "ek-0", stepId: "s0", modelStr: "openai/compat-a" }), + modelTarget({ executionKey: "ek-1", stepId: "s1", modelStr: "openai/compat-b" }), + ]; + const ok = () => + new Response(JSON.stringify({ choices: [{ message: { content: "ok" } }] }), { + status: 200, + headers: { "content-type": "application/json" }, + }); + const result = await attemptCompatRejectedFallback( + targets, + { messages: [] }, + { + handleSingleModel: async (_body, modelStr, target) => { + seen.push({ + model: modelStr, + fallbackAttempts: (target as { fallbackAttempts?: number } | undefined)?.fallbackAttempts, + }); + if (modelStr === "openai/compat-a") { + return new Response("fail", { status: 500 }); + } + return ok(); + }, + log: { info() {}, warn() {}, debug() {}, error() {} }, + strategy: "round-robin", + } + ); + assert.equal(result?.ok, true); + assert.equal(seen.length, 2); + assert.equal(seen[0].fallbackAttempts, 0); + assert.equal(seen[1].fallbackAttempts, 1); +}); diff --git a/tests/unit/combo-runtimeunits-diagnostics-11462.test.ts b/tests/unit/combo-runtimeunits-diagnostics-11462.test.ts index 6612a25ca8..900ae0b7b9 100644 --- a/tests/unit/combo-runtimeunits-diagnostics-11462.test.ts +++ b/tests/unit/combo-runtimeunits-diagnostics-11462.test.ts @@ -8,7 +8,10 @@ import test from "node:test"; import assert from "node:assert/strict"; import { executeRuntimeUnitCombo } from "../../open-sse/services/combo/runtimeUnits.ts"; -import type { ResolvedComboUnit, ComboNestingContext } from "../../open-sse/services/combo/types.ts"; +import type { + ResolvedComboUnit, + ComboNestingContext, +} from "../../open-sse/services/combo/types.ts"; function noopLog() { return { info: () => {}, warn: () => {}, error: () => {}, debug: () => {} }; @@ -86,3 +89,71 @@ test( assert.equal(body.diagnostics?.terminalReason, "max_attempts_exceeded"); } ); + +test("nested runtime-unit dispatch stamps fallbackAttempts from the unit index", async () => { + const units: ResolvedComboUnit[] = [ + { + kind: "model", + stepId: "step-a", + executionKey: "a", + modelStr: "openai/ru-a", + provider: "openai", + providerId: null, + connectionId: null, + weight: 1, + label: null, + }, + { + kind: "model", + stepId: "step-b", + executionKey: "b", + modelStr: "anthropic/ru-b", + provider: "anthropic", + providerId: null, + connectionId: null, + weight: 1, + label: null, + }, + ]; + const nesting: ComboNestingContext = { + depth: 0, + maxDepth: 5, + visitedComboNames: [], + rootComboName: "ru-fallback-12339", + attemptBudget: { count: 0, limit: 8 }, + }; + const seen: Array<{ model: string; fallbackAttempts?: number }> = []; + const ok = () => + new Response(JSON.stringify({ choices: [{ message: { content: "ok" } }] }), { + status: 200, + headers: { "content-type": "application/json" }, + }); + await executeRuntimeUnitCombo({ + body: { messages: [{ role: "user", content: "hi" }] }, + combo: { name: "ru-fallback-12339", strategy: "pipeline" }, + strategy: "pipeline", + units, + handleSingleModel: async (_body, modelStr, target) => { + seen.push({ + model: modelStr, + fallbackAttempts: (target as { fallbackAttempts?: number } | undefined)?.fallbackAttempts, + }); + if (modelStr === "openai/ru-a") { + return new Response(JSON.stringify({ error: { message: "upstream 500" } }), { + status: 500, + headers: { "content-type": "application/json" }, + }); + } + return ok(); + }, + log: noopLog() as never, + config: { maxRetries: 0, retryDelayMs: 0 }, + allCombos: [], + nesting, + baseOptions: {} as never, + runCombo: async () => failResponse(), + }); + assert.equal(seen.length, 2); + assert.equal(seen[0].fallbackAttempts, 0); + assert.equal(seen[1].fallbackAttempts, 1); +}); diff --git a/tests/unit/combo/execute-target-attempt.test.ts b/tests/unit/combo/execute-target-attempt.test.ts index 1609f53f80..e2ef819a19 100644 --- a/tests/unit/combo/execute-target-attempt.test.ts +++ b/tests/unit/combo/execute-target-attempt.test.ts @@ -286,3 +286,70 @@ test("body-specific 400 surfaces via {ok,response} not null", async () => { assert.equal(result?.ok, false); assert.equal(result?.response?.status, 400); }); + +test("spreads stamped fallbackAttempts onto the handleSingleModel target", async () => { + const { executeTargetAttempt } = + await import("../../../open-sse/services/combo/executeTargetAttempt.ts"); + let seen: unknown; + const target = { + ...modelTarget({ connectionId: "c1" }), + fallbackAttempts: 2, + } as ResolvedComboTarget & { fallbackAttempts: number }; + const deps = baseDeps({ + maxRetries: 0, + handleSingleModelWithTimeout: async (_body, _model, dispatched) => { + seen = dispatched; + return new Response(JSON.stringify({ choices: [{ message: { content: "ok" } }] }), { + status: 200, + headers: { "content-type": "application/json" }, + }); + }, + }); + const state = emptyState({ + orderedTargets: [target], + abortControllers: new Map([[0, new AbortController()]]), + }); + const result = await executeTargetAttempt({ + index: 0, + state, + deps, + targetForAttempt: target, + profile: {}, + protectedPriorityTarget: false, + }); + assert.equal(result?.ok, true); + assert.equal((seen as { fallbackAttempts?: number } | undefined)?.fallbackAttempts, 2); +}); + +test("injection: dropping fallbackAttempts from the dispatch target goes red", async () => { + const { executeTargetAttempt } = + await import("../../../open-sse/services/combo/executeTargetAttempt.ts"); + let seen: unknown; + const target = { + ...modelTarget({ connectionId: "c1" }), + fallbackAttempts: 2, + } as ResolvedComboTarget & { fallbackAttempts: number }; + const deps = baseDeps({ + maxRetries: 0, + handleSingleModelWithTimeout: async (_body, _model, dispatched) => { + seen = dispatched; + return new Response(JSON.stringify({ choices: [{ message: { content: "ok" } }] }), { + status: 200, + headers: { "content-type": "application/json" }, + }); + }, + }); + const state = emptyState({ + orderedTargets: [target], + abortControllers: new Map([[0, new AbortController()]]), + }); + await executeTargetAttempt({ + index: 0, + state, + deps, + targetForAttempt: target, + profile: {}, + protectedPriorityTarget: false, + }); + assert.equal(Object.prototype.hasOwnProperty.call(seen as object, "fallbackAttempts"), true); +}); diff --git a/tests/unit/combo/execute-target-gates.test.ts b/tests/unit/combo/execute-target-gates.test.ts index 4d121cdb13..062909cbff 100644 --- a/tests/unit/combo/execute-target-gates.test.ts +++ b/tests/unit/combo/execute-target-gates.test.ts @@ -156,3 +156,69 @@ test("protected priority non-quota skip returns 503 response not null", async () assert.equal(decision.result?.response?.status, 503); } }); + +test("proceed stamps fallbackAttempts from the ordered-target index", async () => { + const { evaluateExecuteTargetGates } = + await import("../../../open-sse/services/combo/executeTargetGates.ts"); + const first = modelTarget({ executionKey: "ek-0", stepId: "s0" }); + const second = modelTarget({ executionKey: "ek-1", stepId: "s1" }); + const state = emptyState({ + orderedTargets: [first, second], + abortControllers: new Map([ + [0, new AbortController()], + [1, new AbortController()], + ]), + }); + const firstDecision = await evaluateExecuteTargetGates({ + index: 0, + state, + deps: baseDeps(), + }); + const secondDecision = await evaluateExecuteTargetGates({ + index: 1, + state, + deps: baseDeps(), + }); + assert.equal(firstDecision.kind, "proceed"); + assert.equal(secondDecision.kind, "proceed"); + if (firstDecision.kind === "proceed") { + assert.equal( + (firstDecision.targetForAttempt as ResolvedComboTarget & { fallbackAttempts?: number }) + .fallbackAttempts, + 0 + ); + } + if (secondDecision.kind === "proceed") { + assert.equal( + (secondDecision.targetForAttempt as ResolvedComboTarget & { fallbackAttempts?: number }) + .fallbackAttempts, + 1 + ); + } +}); + +test("injection: dropping fallbackAttempts from targetForAttempt goes red", async () => { + const { evaluateExecuteTargetGates } = + await import("../../../open-sse/services/combo/executeTargetGates.ts"); + const first = modelTarget({ executionKey: "ek-0", stepId: "s0" }); + const second = modelTarget({ executionKey: "ek-1", stepId: "s1" }); + const state = emptyState({ + orderedTargets: [first, second], + abortControllers: new Map([ + [0, new AbortController()], + [1, new AbortController()], + ]), + }); + const decision = await evaluateExecuteTargetGates({ + index: 1, + state, + deps: baseDeps(), + }); + assert.equal(decision.kind, "proceed"); + if (decision.kind === "proceed") { + assert.equal( + Object.prototype.hasOwnProperty.call(decision.targetForAttempt, "fallbackAttempts"), + true + ); + } +}); diff --git a/tests/unit/omniroute-decision-header.test.ts b/tests/unit/omniroute-decision-header.test.ts index b83bed165c..587ec2fa0e 100644 --- a/tests/unit/omniroute-decision-header.test.ts +++ b/tests/unit/omniroute-decision-header.test.ts @@ -19,7 +19,10 @@ test("buildOmniRouteResponseMetaHeaders emits X-OmniRoute-Decision for a combo s model: "gpt-4o", latencyMs: 42, }); - assert.equal(headers["X-OmniRoute-Decision"], "strategy=priority; provider=openai; latency_ms=42"); + assert.equal( + headers["X-OmniRoute-Decision"], + "strategy=priority; provider=openai; latency_ms=42" + ); }); test("strategy: single (non-combo request) still emits the header", () => { @@ -28,7 +31,10 @@ test("strategy: single (non-combo request) still emits the header", () => { provider: "anthropic", latencyMs: 10, }); - assert.equal(headers["X-OmniRoute-Decision"], "strategy=single; provider=anthropic; latency_ms=10"); + assert.equal( + headers["X-OmniRoute-Decision"], + "strategy=single; provider=anthropic; latency_ms=10" + ); }); test("omitted strategy AND provider -> header absent entirely", () => { @@ -70,5 +76,55 @@ test("buildNonStreamingResponseHeaders falls back to strategy=single when comboS requestId: "req-2", comboStrategy: null, }); - assert.match(headers["X-OmniRoute-Decision"], /^strategy=single; provider=openai; latency_ms=\d+$/); + assert.match( + headers["X-OmniRoute-Decision"], + /^strategy=single; provider=openai; latency_ms=\d+$/ + ); +}); + +test("assembleStreamingResponseHeaders emits X-OmniRoute-Fallback-Attempts when count > 0", () => { + const headers = assembleStreamingResponseHeaders({ + providerHeaders: new Headers(), + provider: "openai", + model: "gpt-4o", + pendingRequestId: "req-3", + comboStrategy: "priority", + fallbackAttempts: 2, + }); + assert.equal(headers["X-OmniRoute-Fallback-Attempts"], "2"); +}); + +test("buildNonStreamingResponseHeaders emits X-OmniRoute-Fallback-Attempts when count > 0", () => { + const headers = buildNonStreamingResponseHeaders({ + provider: "openai", + model: "gpt-4o", + startTime: Date.now(), + responseUsage: null, + estimatedCost: 0, + requestId: "req-4", + comboStrategy: "priority", + fallbackAttempts: 1, + }); + assert.equal(headers["X-OmniRoute-Fallback-Attempts"], "1"); +}); + +test("builders omit X-OmniRoute-Fallback-Attempts when count is 0", () => { + const streaming = assembleStreamingResponseHeaders({ + providerHeaders: new Headers(), + provider: "openai", + model: "gpt-4o", + pendingRequestId: "req-5", + fallbackAttempts: 0, + }); + const nonStreaming = buildNonStreamingResponseHeaders({ + provider: "openai", + model: "gpt-4o", + startTime: Date.now(), + responseUsage: null, + estimatedCost: 0, + requestId: "req-6", + fallbackAttempts: 0, + }); + assert.equal(streaming["X-OmniRoute-Fallback-Attempts"], undefined); + assert.equal(nonStreaming["X-OmniRoute-Fallback-Attempts"], undefined); }); From a31c7880c49aa433661e5553f20e493bb0cd81f4 Mon Sep 17 00:00:00 2001 From: "Bob.Hou" Date: Fri, 11 Sep 2026 18:27:36 -0400 Subject: [PATCH 02/77] fix(chatcore): restore provider failure classification and credential refresh on non-streaming leg (#13043) (#13069) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This is the one that mattered most. The non-streaming leg returning error outcomes without classification meant 402/429 never reached model lockout or rate-limit bookkeeping, and a 401 on a refreshable provider failed instead of refreshing — on a leg that serves real traffic. Extracting `applyProviderFailureClassification` and wiring both legs through it is the right shape: the asymmetry was the bug, so the fix has to remove the asymmetry rather than patch one side. --- Validated in one consolidated worktree cut from `release/v3.8.51`, boarded with the other 19 PRs of this batch. Two in-batch conflicts, both additive and resolved by keeping each side: the `ENVIRONMENT.md` table (#13035 + #13011) and the `chatHelpers.ts` import block (#12975 on the tip + #13017). - `typecheck:core` clean; `check:dashboard-typecheck` OK (206 pre-existing, within baseline); `check:changelog-integrity` OK; `check:docs-counts` migrations ✓ - complexity 2816 / baseline 3218 and cognitive-complexity 1271 / baseline 1437 — both under baseline - 531 of 532 focused assertions green across the batch's 46 test files - `check-file-size` rebaselined for the batch's real growth (annotation `_rebaseline_2026_09_11_mergebatch_v3851_houminxi`, landed on #13038), attributed per PR The single red is **not this batch**: `tests/unit/combo/quota-weighted-strategy.test.ts` → "A/B isolation: 7 hard-empty + 2 at 0.5% + 1 at 40%, floor=1" asserts an order between two connections of identical weight and flakes on the pure tip too — 2 failures in 4 runs at `origin/release/v3.8.51` with nothing from this batch applied. ⚠️ base-red inherited: #12732 — `Docs Gates`, `Merge integrity`, `No new ESLint warnings`, `Unit Tests fast-path` and `Fast Quality Gates` reproduce on the pure tip (provider count 356 vs the 358 the modules define, SKILL.md drift, and `open-sse/utils/stream.ts` at 3115 > frozen 3098, untouched here). Thanks @HouMinXi — the live evidence on these (X500 logs, `storage.sqlite` state, real `/v1/models` probes, the 36-minute outage write-up) is what let a 20-PR batch be reviewed as a unit. --- ...43-non-streaming-failure-classification.md | 1 + open-sse/handlers/chatCore.ts | 783 ++++++++++-------- .../chatCore/nonStreamingProviderLeg.ts | 6 + .../chatCore/providerExecutionPipeline.ts | 48 +- src/lib/skills/toolLoopTypes.ts | 2 + 5 files changed, 488 insertions(+), 352 deletions(-) create mode 100644 changelog.d/fixes/13043-non-streaming-failure-classification.md diff --git a/changelog.d/fixes/13043-non-streaming-failure-classification.md b/changelog.d/fixes/13043-non-streaming-failure-classification.md new file mode 100644 index 0000000000..e6774f9825 --- /dev/null +++ b/changelog.d/fixes/13043-non-streaming-failure-classification.md @@ -0,0 +1 @@ +- Restore provider failure classification and credential refresh on non-streaming requests: classify non-2xx failures to lock models on per-model quota exhaustion, update connection rate limits from headers and body, and pass credential refresh handlers to pipeline execution so 401 tokens can be refreshed and retried (#13043). diff --git a/open-sse/handlers/chatCore.ts b/open-sse/handlers/chatCore.ts index 20078c522d..d14035a49e 100644 --- a/open-sse/handlers/chatCore.ts +++ b/open-sse/handlers/chatCore.ts @@ -3615,6 +3615,418 @@ export async function handleChatCore({ let finalBody; let claudePromptCacheLogMeta = null; + let credentialRefreshPersistRan = false; + const hadStreamOptions = + targetFormat === FORMATS.OPENAI_RESPONSES && + translatedBody && + typeof translatedBody === "object" && + "stream_options" in translatedBody; + if (hadStreamOptions) { + delete (translatedBody as Record).stream_options; + } + + const executeRefreshCredentials = async ( + currentCreds: Record + ): Promise | null> => { + if (typeof executor.refreshCredentials !== "function") { + return null; + } + if (hadStreamOptions) { + return null; + } + if (await shouldIsolateProbeFailures()) { + return null; + } + + const targetCredentials = (currentCreds || credentials || {}) as Record; + const attemptedRefreshToken = + typeof targetCredentials?.refreshToken === "string" ? targetCredentials.refreshToken : null; + credentialRefreshPersistRan = false; + const persistFn = onCredentialsRefreshed + ? async (refreshResult: Record) => { + credentialRefreshPersistRan = true; + Object.assign(targetCredentials, refreshResult); + Object.assign(credentials, refreshResult); + await onCredentialsRefreshed(refreshResult); + } + : undefined; + + const casConnectionId = + typeof targetCredentials?.connectionId === "string" + ? targetCredentials.connectionId.trim() + : ""; + const casReread = casConnectionId + ? async () => { + const latest = await getProviderConnectionById(casConnectionId); + return typeof latest?.refreshToken === "string" ? latest.refreshToken : null; + } + : null; + + const newCredentials = (await refreshWithRetry( + () => + runWithCasGuard( + casReread ? { expectedRefreshToken: attemptedRefreshToken, reread: casReread } : null, + () => + runWithOnPersist(persistFn, () => executor.refreshCredentials(targetCredentials, log)) + ), + 3, + log, + provider + )) as null | Record; + + if (newCredentials?.accessToken || newCredentials?.copilotToken) { + log?.info?.("TOKEN", `${provider?.toUpperCase()} | refreshed`); + if (!credentialRefreshPersistRan) { + Object.assign(targetCredentials, newCredentials); + Object.assign(credentials, newCredentials); + } + const errorConnectionId = String(getCurrentConnectionId() || connectionId || ""); + if (errorConnectionId) { + updateProviderConnection(errorConnectionId, newCredentials).catch(() => {}); + } + return newCredentials; + } + return null; + }; + + const handleCredentialsRefreshed = async (refreshed: Record) => { + Object.assign(credentials, refreshed); + if (!credentialRefreshPersistRan && onCredentialsRefreshed) { + credentialRefreshPersistRan = true; + const targetConnectionId = + (credentials as { connectionId?: string })?.connectionId || + (credentials as { id?: string })?.id || + getCurrentConnectionId() || + connectionId; + try { + await onCredentialsRefreshed({ + ...refreshed, + provider, + connectionId: targetConnectionId, + }); + } catch (refreshErr) { + log?.warn?.( + "REFRESH", + `onCredentialsRefreshed persistence callback failed for connection ${targetConnectionId}: ${refreshErr}` + ); + } + } + }; + + const applyProviderFailureClassification = async ({ + statusCode, + message, + headers, + upstreamErrorBody, + retryAfterMs, + targetModel, + }: { + statusCode: number; + message: string; + headers?: Headers | null; + upstreamErrorBody?: unknown; + retryAfterMs?: number | null; + targetModel: string; + }) => { + let errorType = classifyProviderError(statusCode, message, provider); + if (statusCode === 429 && isModelScope()) { + const decision = classifyModelScope429(message, normalizeHeaders(headers)); + errorType = + decision.kind === "quota_exhausted" + ? PROVIDER_ERROR_TYPES.QUOTA_EXHAUSTED + : PROVIDER_ERROR_TYPES.RATE_LIMITED; + log?.warn?.( + "MODELSCOPE_429", + `${decision.kind} (model remaining: ${decision.snapshot.modelRemaining ?? "unknown"}, total remaining: ${decision.snapshot.totalRemaining ?? "unknown"})` + ); + } + const persistentMessage = sanitizeErrorMessage(message) || "Provider request failed"; + const errorConnectionId = getCurrentConnectionId() || connectionId; + if (errorConnectionId && errorType) { + try { + if (errorType === PROVIDER_ERROR_TYPES.FORBIDDEN) { + const probeIsolated = await shouldIsolateProbeFailures(); + await writeTerminalStatus( + errorConnectionId, + { + testStatus: "banned", + isActive: false, + lastError: persistentMessage, + lastErrorType: errorType, + errorCode: String(statusCode), + }, + probeIsolated ? "probe" : "production" + ); + if (probeIsolated) { + console.warn( + `[provider] Node ${errorConnectionId} probe ${errorType} (${statusCode}) -- connection stays active` + ); + } else { + console.warn( + `[provider] Node ${errorConnectionId} banned (${statusCode}) -- disabling permanently` + ); + } + } else if (errorType === PROVIDER_ERROR_TYPES.ACCOUNT_DEACTIVATED) { + if ( + connectionHasExtraKeys( + errorConnectionId, + (credentials?.providerSpecificData as Record | undefined) + ?.extraApiKeys as string[] | undefined + ) + ) { + await updateProviderConnection(errorConnectionId, { + lastErrorType: errorType, + lastError: persistentMessage, + errorCode: statusCode, + }); + console.warn( + `[provider] Node ${errorConnectionId} account deactivated (${statusCode}) -- has extra keys, keeping connection active` + ); + } else { + const probeIsolated2 = await shouldIsolateProbeFailures(); + await writeTerminalStatus( + errorConnectionId, + { + testStatus: "deactivated", + isActive: false, + lastError: persistentMessage, + lastErrorType: errorType, + errorCode: String(statusCode), + }, + probeIsolated2 ? "probe" : "production" + ); + if (probeIsolated2) { + console.warn( + `[provider] Node ${errorConnectionId} probe ${errorType} (${statusCode}) -- connection stays active` + ); + } else { + console.warn( + `[provider] Node ${errorConnectionId} account deactivated (${statusCode}) -- disabling permanently` + ); + } + } + } else if (errorType === PROVIDER_ERROR_TYPES.QUOTA_EXHAUSTED) { + const probeIsolated3 = await shouldIsolateProbeFailures(); + if (probeIsolated3) { + await writeTerminalStatus( + errorConnectionId, + { + testStatus: "credits_exhausted", + lastError: persistentMessage, + lastErrorType: errorType, + errorCode: String(statusCode), + }, + "probe" + ); + console.warn( + `[provider] Node ${errorConnectionId} probe ${errorType} (${statusCode}) -- connection stays active` + ); + } else { + let kimiRateLimitResetAt: string | null = null; + if (provider === "kimi-coding") { + try { + const { fetchAndPersistProviderLimits } = + await import("@/lib/usage/providerLimits"); + const { usage } = await fetchAndPersistProviderLimits(errorConnectionId, "manual"); + kimiRateLimitResetAt = getKimiTemporaryRateLimitResetAt(usage); + } catch {} + } + + let quotaCooldownMs = kimiRateLimitResetAt + ? Math.max(new Date(kimiRateLimitResetAt).getTime() - Date.now(), 0) + : retryAfterMs || COOLDOWN_MS.rateLimit; + const deferAntigravityQuotaStateToCaller = shouldDeferAntigravityQuotaStateToCaller( + provider, + typeof onStreamFailure === "function" + ); + const isAntigravityQuotaFamily = shouldDeferAntigravityQuotaStateToCaller( + provider, + true + ); + let coreOwnedAntigravityLockout: { + cooldownMs: number; + failureCount: number; + } | null = null; + if (isAntigravityQuotaFamily && !deferAntigravityQuotaStateToCaller) { + const quotaErrorText = + typeof upstreamErrorBody === "string" + ? upstreamErrorBody + : upstreamErrorBody == null + ? message + : JSON.stringify(upstreamErrorBody); + coreOwnedAntigravityLockout = await recordCoreOwnedAntigravityQuotaState({ + provider, + connectionId: errorConnectionId, + model, + status: statusCode, + errorText: quotaErrorText, + headers: headers ?? undefined, + }); + quotaCooldownMs = coreOwnedAntigravityLockout.cooldownMs; + } + const accountSemaphoreKey = resolveAccountSemaphoreKey({ + provider, + model: targetModel, + connectionId: errorConnectionId, + credentials, + }); + if (accountSemaphoreKey && !deferAntigravityQuotaStateToCaller) { + markAccountSemaphoreBlocked(accountSemaphoreKey, quotaCooldownMs); + } + if (deferAntigravityQuotaStateToCaller) { + } else if (coreOwnedAntigravityLockout) { + console.warn( + `[provider] Node ${errorConnectionId} Antigravity model quota exhausted (${statusCode}) for ${model} - ${Math.ceil(coreOwnedAntigravityLockout.cooldownMs / 1000)}s (failureCount=${coreOwnedAntigravityLockout.failureCount}, owner=core)` + ); + } else if (kimiRateLimitResetAt) { + await updateProviderConnection(errorConnectionId, { + testStatus: "unavailable", + rateLimitedUntil: kimiRateLimitResetAt, + backoffLevel: 0, + lastErrorType: PROVIDER_ERROR_TYPES.RATE_LIMITED, + lastError: persistentMessage, + errorCode: statusCode, + }); + console.warn( + `[provider] Node ${errorConnectionId} Kimi request window exhausted (${statusCode}) -- retrying after ${kimiRateLimitResetAt}` + ); + } else if (isModelScope() && errorConnectionId) { + lockModel(provider, errorConnectionId, model, "quota_exhausted", quotaCooldownMs); + if (targetModel && targetModel !== model) { + lockModel( + provider, + errorConnectionId, + targetModel, + "quota_exhausted", + quotaCooldownMs + ); + } + console.warn( + `[provider] Node ${errorConnectionId} ModelScope model quota exhausted (${statusCode}) for ${targetModel} - ${Math.ceil(quotaCooldownMs / 1000)}s (connection stays active)` + ); + } else if ( + lockModelIfPerModelQuota( + provider, + errorConnectionId, + model, + "quota_exhausted", + quotaCooldownMs + ) || + (targetModel && + targetModel !== model && + lockModelIfPerModelQuota( + provider, + errorConnectionId, + targetModel, + "quota_exhausted", + quotaCooldownMs + )) + ) { + const quotaScope = getQuotaScopeLabelForProvider(provider, targetModel); + console.warn( + `[provider] Node ${errorConnectionId} ${quotaScope}-only quota exhausted (${statusCode}) for ${targetModel} - ${Math.ceil(quotaCooldownMs / 1000)}s (cooldown_scope=${quotaScope}, ttl_source=${retryAfterMs ? "upstream" : "inferred"}, connection stays active)` + ); + } else { + await writeTerminalStatus( + errorConnectionId, + { + testStatus: "credits_exhausted", + lastError: persistentMessage, + lastErrorType: errorType, + errorCode: String(statusCode), + }, + "production" + ); + console.warn(`[provider] Node ${errorConnectionId} exhausted quota (${statusCode})`); + } + } + } else if (errorType === PROVIDER_ERROR_TYPES.UNAUTHORIZED) { + await updateProviderConnection(errorConnectionId, { + lastErrorType: errorType, + lastError: persistentMessage, + errorCode: statusCode, + }); + } else if (errorType === PROVIDER_ERROR_TYPES.OAUTH_INVALID_TOKEN) { + await updateProviderConnection(errorConnectionId, { + lastErrorType: errorType, + lastError: persistentMessage, + errorCode: statusCode, + }); + console.warn( + `[provider] Node ${errorConnectionId} OAuth token invalid (${statusCode}) -- token refresh available` + ); + } else if (errorType === PROVIDER_ERROR_TYPES.PROJECT_ROUTE_ERROR) { + await updateProviderConnection(errorConnectionId, { + lastErrorType: errorType, + lastError: persistentMessage, + errorCode: statusCode, + }); + console.warn( + `[provider] Node ${errorConnectionId} project routing error (${statusCode}) -- not banning` + ); + } else if (errorType === PROVIDER_ERROR_TYPES.GEO_BLOCKED) { + const geoCooldownMs = COOLDOWN_MS.geoBlocked ?? 24 * 60 * 60 * 1000; + await updateProviderConnection(errorConnectionId, { + lastErrorType: errorType, + lastError: persistentMessage, + errorCode: statusCode, + }); + if (!(await shouldIsolateProbeFailures())) { + try { + const { setConnectionRateLimitUntil } = await import("@/lib/db/providers"); + setConnectionRateLimitUntil(errorConnectionId, Date.now() + geoCooldownMs); + } catch {} + } + console.warn( + `[provider] Node ${errorConnectionId} geo-blocked (${statusCode}) -- excluded for ${Math.ceil(geoCooldownMs / 1000)}s, trying other accounts` + ); + } else if (errorType === PROVIDER_ERROR_TYPES.GCP_PROJECT_REQUIRED) { + const byopCooldownMs = COOLDOWN_MS.gcpProjectRequired ?? 24 * 60 * 60 * 1000; + await updateProviderConnection(errorConnectionId, { + lastErrorType: errorType, + lastError: persistentMessage, + errorCode: statusCode, + }); + try { + const { setConnectionRateLimitUntil } = await import("@/lib/db/providers"); + setConnectionRateLimitUntil(errorConnectionId, Date.now() + byopCooldownMs); + } catch {} + console.warn( + `[provider] Node ${errorConnectionId} GCP project required (${statusCode}) -- excluded for ${Math.ceil(byopCooldownMs / 1000)}s, routing to other accounts (enter a Project ID to restore)` + ); + } else if (errorType === PROVIDER_ERROR_TYPES.MODEL_NOT_FOUND) { + const notFoundCooldownMs = COOLDOWN_MS.notFound; + if (!(await shouldIsolateProbeFailures())) { + const modelToLock = targetModel || model; + lockModel( + provider, + errorConnectionId, + modelToLock, + "model_not_found", + notFoundCooldownMs + ); + console.warn( + `[provider] Node ${errorConnectionId} model not found (${statusCode}) for ${modelToLock} - locking model for ${Math.ceil(notFoundCooldownMs / 1000)}s (connection stays active)` + ); + } + } + } catch {} + } + + if (headers) { + updateFromHeaders(provider, errorConnectionId, headers, statusCode, targetModel); + } + if (errorConnectionId && upstreamErrorBody !== null && upstreamErrorBody !== undefined) { + updateFromResponseBody( + provider, + errorConnectionId, + upstreamErrorBody, + statusCode, + targetModel + ); + } + }; + let pipelineRecovered = false; if (stream) { try { @@ -3640,7 +4052,8 @@ export async function handleChatCore({ replaceCredentials: (next) => { Object.assign(credentials, next); }, - onCredentialsRefreshed: async () => {}, + onCredentialsRefreshed: handleCredentialsRefreshed, + refreshCredentials: executeRefreshCredentials, assertManagedLeaseFence: (id) => { assertManagedLeaseFence(id); }, @@ -4187,339 +4600,15 @@ export async function handleChatCore({ break providerFailure; } - // T06/T10/T36: classify provider errors and persist terminal account states. - let errorType = classifyProviderError(statusCode, message, provider); - if (statusCode === 429 && isModelScope()) { - const decision = classifyModelScope429(message, normalizeHeaders(providerResponse.headers)); - errorType = - decision.kind === "quota_exhausted" - ? PROVIDER_ERROR_TYPES.QUOTA_EXHAUSTED - : PROVIDER_ERROR_TYPES.RATE_LIMITED; - log?.warn?.( - "MODELSCOPE_429", - `${decision.kind} (model remaining: ${decision.snapshot.modelRemaining ?? "unknown"}, total remaining: ${decision.snapshot.totalRemaining ?? "unknown"})` - ); - } - // Classifiers and recovery paths above consume the raw provider wording. - // Project a separate value only at persistent connection-state boundaries. - const persistentMessage = sanitizeErrorMessage(message) || "Provider request failed"; - const errorConnectionId = getCurrentConnectionId(); - if (errorConnectionId && errorType) { - try { - if (errorType === PROVIDER_ERROR_TYPES.FORBIDDEN) { - { - const probeIsolated = await shouldIsolateProbeFailures(); - await writeTerminalStatus( - errorConnectionId, - { - testStatus: "banned", - isActive: false, - lastError: persistentMessage, - lastErrorType: errorType, - errorCode: String(statusCode), - }, - probeIsolated ? "probe" : "production" - ); - if (probeIsolated) { - console.warn( - `[provider] Node ${errorConnectionId} probe ${errorType} (${statusCode}) — connection stays active` - ); - } else if (hasPerModelQuota(provider, model)) { - // Compatible / passthrough gateways: a 402 without a model id - // still must not terminalize the whole connection. Record the - // error for operators; sibling models stay selectable. - await updateProviderConnection(errorConnectionId, { - lastErrorType: errorType, - lastError: persistentMessage, - errorCode: statusCode, - }); - console.warn( - `[provider] Node ${errorConnectionId} per-model quota exhausted (${statusCode}) — connection stays active` - ); - } else { - console.warn( - `[provider] Node ${errorConnectionId} banned (${statusCode}) — disabling permanently` - ); - } - } - } else if (errorType === PROVIDER_ERROR_TYPES.ACCOUNT_DEACTIVATED) { - // T-PROBE: probe-origin failures (test-all) never deactivate — - // record but stay active; Plan A (extra keys) stays first so the - // real path keeps its existing priority (#9817). - // Plan A: if connection has extra API keys, don't disable — only the failing key is affected. - // Single-key connections still get disabled as before. - if ( - connectionHasExtraKeys( - errorConnectionId, - (credentials?.providerSpecificData as Record | undefined) - ?.extraApiKeys as string[] | undefined - ) - ) { - await updateProviderConnection(errorConnectionId, { - lastErrorType: errorType, - lastError: persistentMessage, - errorCode: statusCode, - }); - console.warn( - `[provider] Node ${errorConnectionId} account deactivated (${statusCode}) — has extra keys, keeping connection active` - ); - } else { - const probeIsolated2 = await shouldIsolateProbeFailures(); - await writeTerminalStatus( - errorConnectionId, - { - testStatus: "deactivated", - isActive: false, - lastError: persistentMessage, - lastErrorType: errorType, - errorCode: String(statusCode), - }, - probeIsolated2 ? "probe" : "production" - ); - if (probeIsolated2) { - console.warn( - `[provider] Node ${errorConnectionId} probe ${errorType} (${statusCode}) — connection stays active` - ); - } else { - console.warn( - `[provider] Node ${errorConnectionId} account deactivated (${statusCode}) — disabling permanently` - ); - } - } - } else if (errorType === PROVIDER_ERROR_TYPES.QUOTA_EXHAUSTED) { - { - const probeIsolated3 = await shouldIsolateProbeFailures(); - if (probeIsolated3) { - await writeTerminalStatus( - errorConnectionId, - { - testStatus: "credits_exhausted", - lastError: persistentMessage, - lastErrorType: errorType, - errorCode: String(statusCode), - }, - "probe" - ); - console.warn( - `[provider] Node ${errorConnectionId} probe ${errorType} (${statusCode}) — connection stays active` - ); - } else { - // Kimi's 403 says "billing cycle" for both an exhausted subscription and a - // temporary request window. Read its official usage endpoint before making - // the connection terminal: a non-zero Weekly quota plus an empty Ratelimit - // window must recover automatically at the reported reset time. - let kimiRateLimitResetAt: string | null = null; - if (provider === "kimi-coding") { - try { - const { fetchAndPersistProviderLimits } = - await import("@/lib/usage/providerLimits"); - const { usage } = await fetchAndPersistProviderLimits( - errorConnectionId, - "manual" - ); - kimiRateLimitResetAt = getKimiTemporaryRateLimitResetAt(usage); - } catch { - // Preserve the existing quota handling when Kimi's usage endpoint is unavailable. - } - } - - // Providers with per-model quotas — lock the model only, not the connection - let quotaCooldownMs = kimiRateLimitResetAt - ? Math.max(new Date(kimiRateLimitResetAt).getTime() - Date.now(), 0) - : retryAfterMs || COOLDOWN_MS.rateLimit; - const deferAntigravityQuotaStateToCaller = shouldDeferAntigravityQuotaStateToCaller( - provider, - typeof onStreamFailure === "function" - ); - const isAntigravityQuotaFamily = shouldDeferAntigravityQuotaStateToCaller( - provider, - true - ); - let coreOwnedAntigravityLockout: { - cooldownMs: number; - failureCount: number; - } | null = null; - if (isAntigravityQuotaFamily && !deferAntigravityQuotaStateToCaller) { - const quotaErrorText = - typeof upstreamErrorBody === "string" - ? upstreamErrorBody - : upstreamErrorBody == null - ? message - : JSON.stringify(upstreamErrorBody); - coreOwnedAntigravityLockout = await recordCoreOwnedAntigravityQuotaState({ - provider, - connectionId: errorConnectionId, - model, - status: statusCode, - errorText: quotaErrorText, - headers: providerResponse.headers, - }); - quotaCooldownMs = coreOwnedAntigravityLockout.cooldownMs; - } - const accountSemaphoreKey = resolveAccountSemaphoreKey({ - provider, - model: currentModel, - connectionId: errorConnectionId, - credentials, - }); - if (accountSemaphoreKey && !deferAntigravityQuotaStateToCaller) { - markAccountSemaphoreBlocked(accountSemaphoreKey, quotaCooldownMs); - } - if (deferAntigravityQuotaStateToCaller) { - // Defer both model and account-semaphore cooldowns to - // markAccountUnavailable, where header/body provenance and the - // configured maxCooldownMs are available. Direct consumers such - // as Responses pass no owner callback and retain core ownership. - } else if (coreOwnedAntigravityLockout) { - console.warn( - `[provider] Node ${errorConnectionId} Antigravity model quota exhausted (${statusCode}) for ${model} - ${Math.ceil(coreOwnedAntigravityLockout.cooldownMs / 1000)}s (failureCount=${coreOwnedAntigravityLockout.failureCount}, owner=core)` - ); - } else if (kimiRateLimitResetAt) { - await updateProviderConnection(errorConnectionId, { - testStatus: "unavailable", - rateLimitedUntil: kimiRateLimitResetAt, - backoffLevel: 0, - lastErrorType: PROVIDER_ERROR_TYPES.RATE_LIMITED, - lastError: persistentMessage, - errorCode: statusCode, - }); - console.warn( - `[provider] Node ${errorConnectionId} Kimi request window exhausted (${statusCode}) — retrying after ${kimiRateLimitResetAt}` - ); - } else if (isModelScope() && errorConnectionId) { - lockModel(provider, errorConnectionId, model, "quota_exhausted", quotaCooldownMs); - console.warn( - `[provider] Node ${errorConnectionId} ModelScope model quota exhausted (${statusCode}) for ${model} - ${Math.ceil(quotaCooldownMs / 1000)}s (connection stays active)` - ); - } else if ( - lockModelIfPerModelQuota( - provider, - errorConnectionId, - model, - "quota_exhausted", - quotaCooldownMs - ) - ) { - const quotaScope = getQuotaScopeLabelForProvider(provider, model); - console.warn( - `[provider] Node ${errorConnectionId} ${quotaScope}-only quota exhausted (${statusCode}) for ${model} - ${Math.ceil(quotaCooldownMs / 1000)}s (cooldown_scope=${quotaScope}, ttl_source=${retryAfterMs ? "upstream" : "inferred"}, connection stays active)` - ); - } else { - await writeTerminalStatus( - errorConnectionId, - { - testStatus: "credits_exhausted", - lastError: persistentMessage, - lastErrorType: errorType, - errorCode: String(statusCode), - }, - "production" - ); - console.warn( - `[provider] Node ${errorConnectionId} exhausted quota (${statusCode})` - ); - } - } // close probeIsolated3 else - } - } else if (errorType === PROVIDER_ERROR_TYPES.UNAUTHORIZED) { - // Normal 401 (token/session auth issue): keep account active for refresh/re-auth. - await updateProviderConnection(errorConnectionId, { - lastErrorType: errorType, - lastError: persistentMessage, - errorCode: statusCode, - }); - } else if (errorType === PROVIDER_ERROR_TYPES.OAUTH_INVALID_TOKEN) { - // OAuth 401 with invalid credentials - token refresh can recover - await updateProviderConnection(errorConnectionId, { - lastErrorType: errorType, - lastError: persistentMessage, - errorCode: statusCode, - }); - console.warn( - `[provider] Node ${errorConnectionId} OAuth token invalid (${statusCode}) — token refresh available` - ); - } else if (errorType === PROVIDER_ERROR_TYPES.PROJECT_ROUTE_ERROR) { - // Cloud Code 403 with stale project: not a ban, keep account active. - await updateProviderConnection(errorConnectionId, { - lastErrorType: errorType, - lastError: persistentMessage, - errorCode: statusCode, - }); - console.warn( - `[provider] Node ${errorConnectionId} project routing error (${statusCode}) — not banning` - ); - } else if (errorType === PROVIDER_ERROR_TYPES.GEO_BLOCKED) { - // Google regional-availability refusal (e.g. "User location is not - // supported for the API use."). Account-independent and non-terminal: - // exclude the connection for the cooldown window so routing moves to - // other accounts instead of re-selecting this one on every request, - // and never mark it banned/expired. It becomes usable again once - // egress is routed through a supported-region proxy. - const geoCooldownMs = COOLDOWN_MS.geoBlocked ?? 24 * 60 * 60 * 1000; - await updateProviderConnection(errorConnectionId, { - lastErrorType: errorType, - lastError: persistentMessage, - errorCode: statusCode, - }); - // T-PROBE: the 24h exclusion is a routing mutation — a probe must - // not push a connection into a day-long cooldown (#9817). - if (!(await shouldIsolateProbeFailures())) { - try { - const { setConnectionRateLimitUntil } = await import("@/lib/db/providers"); - setConnectionRateLimitUntil(errorConnectionId, Date.now() + geoCooldownMs); - } catch { - // DB write failure must never break the fallback loop - } - } - console.warn( - `[provider] Node ${errorConnectionId} geo-blocked (${statusCode}) — excluded for ${Math.ceil(geoCooldownMs / 1000)}s, trying other accounts` - ); - } else if (errorType === PROVIDER_ERROR_TYPES.GCP_PROJECT_REQUIRED) { - // Antigravity BYOP: the account must Bring Its Own GCP Project. - // Account-specific and fixable by entering a Project ID — never a - // model lockout, never a ban. Exclude the connection for the - // cooldown window so selection prefers sibling accounts; the 422 - // body carries the actionable message when no sibling is available. - const byopCooldownMs = COOLDOWN_MS.gcpProjectRequired ?? 24 * 60 * 60 * 1000; - await updateProviderConnection(errorConnectionId, { - lastErrorType: errorType, - lastError: persistentMessage, - errorCode: statusCode, - }); - try { - const { setConnectionRateLimitUntil } = await import("@/lib/db/providers"); - setConnectionRateLimitUntil(errorConnectionId, Date.now() + byopCooldownMs); - } catch { - // best-effort — never break the error path - } - console.warn( - `[provider] Node ${errorConnectionId} GCP project required (${statusCode}) — excluded for ${Math.ceil(byopCooldownMs / 1000)}s, routing to other accounts (enter a Project ID to restore)` - ); - } else if (errorType === PROVIDER_ERROR_TYPES.MODEL_NOT_FOUND) { - // 404 — model/endpoint does not exist upstream. Lock the model so the - // retry/backoff loop stops hammering the dead endpoint (which would - // otherwise degenerate into a 429 rate-limit storm). Connection stays - // active since only the specific model is unavailable. (#6827) - const notFoundCooldownMs = COOLDOWN_MS.notFound; - // T-PROBE: the model lockout is a routing mutation — a probe must - // not lock a model for the cooldown window (#9817). - if (!(await shouldIsolateProbeFailures())) { - lockModel( - provider, - errorConnectionId, - currentModel, - "model_not_found", - notFoundCooldownMs - ); - console.warn( - `[provider] Node ${errorConnectionId} model not found (${statusCode}) for ${currentModel} - locking model for ${Math.ceil(notFoundCooldownMs / 1000)}s (connection stays active)` - ); - } - } - } catch { - // Best-effort state update; request flow should continue with fallback handling. - } - } + const errorConnectionId = getCurrentConnectionId() || connectionId; + await applyProviderFailureClassification({ + statusCode, + message, + headers: providerResponse.headers, + upstreamErrorBody, + retryAfterMs, + targetModel: currentModel, + }); appendRequestLog({ model, @@ -4546,11 +4635,7 @@ export async function handleChatCore({ upstreamErrorBody ); - // Update rate limiter from error response headers - updateFromHeaders(provider, errorConnectionId, providerResponse.headers, statusCode, model); - if (errorConnectionId && upstreamErrorBody !== null && upstreamErrorBody !== undefined) { - updateFromResponseBody(provider, errorConnectionId, upstreamErrorBody, statusCode, model); - } + // Rate limiter updated in applyProviderFailureClassification // ── T5: Intra-family model fallback ────────────────────────────────────── // Before returning a model-unavailable error upstream, try sibling models @@ -4795,7 +4880,8 @@ export async function handleChatCore({ replaceCredentials: (next) => { Object.assign(credentials, next); }, - onCredentialsRefreshed: async () => {}, + onCredentialsRefreshed: handleCredentialsRefreshed, + refreshCredentials: executeRefreshCredentials, assertManagedLeaseFence: (id) => { assertManagedLeaseFence(id); }, @@ -4908,6 +4994,23 @@ export async function handleChatCore({ if (legResult.kind === "error") { const err = legResult.result; + const errMessage = + err?.rawMessage || + (err?.originalError instanceof Error ? err.originalError.message : err?.error) || + ""; + const errHeaders = err?.upstreamHeaders || err?.response?.headers; + const errUpstreamBody = err?.upstreamErrorBody; + if (err) { + await applyProviderFailureClassification({ + statusCode: err.status, + message: errMessage, + headers: errHeaders, + upstreamErrorBody: errUpstreamBody, + retryAfterMs: err.retryAfterMs ?? null, + targetModel: currentModel, + }); + } + const captured = providerRequestCapture.latest?.() ?? null; finalBody = captured?.body ?? finalBody ?? translatedBody; if (captured) { diff --git a/open-sse/handlers/chatCore/nonStreamingProviderLeg.ts b/open-sse/handlers/chatCore/nonStreamingProviderLeg.ts index 7656939533..ba17639af9 100644 --- a/open-sse/handlers/chatCore/nonStreamingProviderLeg.ts +++ b/open-sse/handlers/chatCore/nonStreamingProviderLeg.ts @@ -435,6 +435,9 @@ export async function runNonStreamingProviderLeg( { passthrough: input.sourceFormat === "claude" } ), response: outcome.result.response, + rawMessage: outcome.result.rawMessage || outcome.result.error, + upstreamErrorBody: outcome.result.upstreamErrorBody, + upstreamHeaders: outcome.result.upstreamHeaders ?? outcome.result.response?.headers, }, receipt, usage: outcome.providerUsage, @@ -758,6 +761,9 @@ export async function runNonStreamingProviderLeg( upstreamErrorType, { passthrough: sourceFormat === FORMATS.CLAUDE } ); + errorResult.rawMessage = message; + errorResult.upstreamHeaders = providerResponse.headers; + errorResult.upstreamErrorBody = parsedErrorBody; return { kind: "error", result: errorResult as ChatCoreErrorResult, diff --git a/open-sse/handlers/chatCore/providerExecutionPipeline.ts b/open-sse/handlers/chatCore/providerExecutionPipeline.ts index 077ff7cfe1..b058d8b54c 100644 --- a/open-sse/handlers/chatCore/providerExecutionPipeline.ts +++ b/open-sse/handlers/chatCore/providerExecutionPipeline.ts @@ -3,11 +3,17 @@ import type { getProviderCredentials } from "@/sse/services/auth.ts"; import type { updateFromHeaders, updateFromResponseBody } from "../../services/rateLimitManager.ts"; import type { writeTerminalStatus } from "@/shared/utils/terminalStatus.ts"; import type { updateProviderConnection } from "@/lib/db/providers.ts"; -import type { lockModel, recordCoreOwnedAntigravityQuotaState } from "../../services/accountFallback.ts"; +import type { + lockModel, + recordCoreOwnedAntigravityQuotaState, +} from "../../services/accountFallback.ts"; import { createErrorResult } from "../../utils/error.ts"; import { applyStatusRestatement } from "../../config/upstreamStatusRestatement.ts"; import { recoverAnthropicThinkingSignature } from "./thinkingSignatureRecovery.ts"; -import { isModelUnavailableError, getNextFamilyFallback as defaultGetNextFamilyFallback } from "../../services/modelFamilyFallback.ts"; +import { + isModelUnavailableError, + getNextFamilyFallback as defaultGetNextFamilyFallback, +} from "../../services/modelFamilyFallback.ts"; import { COOLDOWN_MS } from "../../config/errorConfig.ts"; import { normalizeHeaders } from "../../utils/headers.ts"; @@ -196,11 +202,7 @@ async function toOutcome( body, retryAfterMs: null, }); - const result = createErrorResult( - restatement.status, - message, - restatement.retryAfterMs - ); + const result = createErrorResult(restatement.status, message, restatement.retryAfterMs); return { kind: "error", result: { @@ -210,6 +212,9 @@ async function toOutcome( error: result.error, errorCode: result.errorCode, errorType: result.errorType, + rawMessage: message, + upstreamErrorBody: body, + upstreamHeaders: attempt.response.headers, }, providerUsage: null, model, @@ -273,7 +278,12 @@ export async function runProviderExecutionPipeline( const status = attempt.response.status; if (status >= 200 && status < 300) { - return toOutcome(attempt, wire.currentModel, currentConnectionId(connection), target.provider); + return toOutcome( + attempt, + wire.currentModel, + currentConnectionId(connection), + target.provider + ); } const isolateProbe = await state.isolateProbeFailures(); @@ -401,11 +411,16 @@ export async function runProviderExecutionPipeline( }; }, }); - if (signatureRecovery.attempted && signatureRecovery.succeeded && signatureRecovery.execution) { + if ( + signatureRecovery.attempted && + signatureRecovery.succeeded && + signatureRecovery.execution + ) { lastAttempt = { response: signatureRecovery.execution.response, url: signatureRecovery.execution.url ?? attempt.url, - headers: (signatureRecovery.execution.headers as Record) ?? attempt.headers, + headers: + (signatureRecovery.execution.headers as Record) ?? attempt.headers, transformedBody: signatureRecovery.execution.transformedBody ?? attempt.transformedBody, }; return toOutcome( @@ -430,7 +445,11 @@ export async function runProviderExecutionPipeline( // keep statusText } if (isModelUnavailableError(status, fallbackMessage, target.provider)) { - const nextModel = resolveFamilyFallback(wire.currentModel, wire.triedModels, target.provider); + const nextModel = resolveFamilyFallback( + wire.currentModel, + wire.triedModels, + target.provider + ); if (nextModel) { wire.setBodyAndModel({ ...wire.body, model: nextModel }, nextModel); modelFallbackPending = true; @@ -443,7 +462,12 @@ export async function runProviderExecutionPipeline( } if (lastAttempt) { - return toOutcome(lastAttempt, wire.currentModel, currentConnectionId(connection), target.provider); + return toOutcome( + lastAttempt, + wire.currentModel, + currentConnectionId(connection), + target.provider + ); } return leaseMismatch(wire.currentModel, currentConnectionId(connection)); } diff --git a/src/lib/skills/toolLoopTypes.ts b/src/lib/skills/toolLoopTypes.ts index 5ebf575518..f3af5a3871 100644 --- a/src/lib/skills/toolLoopTypes.ts +++ b/src/lib/skills/toolLoopTypes.ts @@ -44,6 +44,8 @@ export interface ChatCoreErrorResult { retryAfterMs?: number; originalError?: unknown; rawMessage?: string; + upstreamHeaders?: Headers; + upstreamErrorBody?: unknown; } export type NonStreamingProviderLegResult = From f0b83b86ba284a817bdfa153ecef5a225275444b Mon Sep 17 00:00:00 2001 From: "Bob.Hou" Date: Fri, 11 Sep 2026 18:27:40 -0400 Subject: [PATCH 03/77] fix(grok-cli): park a 402 on the empty Grok Build login, not grok-4.6 (#13061) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The live evidence makes this: four Grok Build logins at 32-93% weekly remaining, skipped as "model locked by resilience" because the one login at 1% answered 402 and `passthroughModels: true` made that read as a per-model billing miss. A shared weekly wallet is a connection-wide signal, not a model verdict — `isSharedWalletCredits402` puts the scope where the biller put it. --- Validated in one consolidated worktree cut from `release/v3.8.51`, boarded with the other 19 PRs of this batch. Two in-batch conflicts, both additive and resolved by keeping each side: the `ENVIRONMENT.md` table (#13035 + #13011) and the `chatHelpers.ts` import block (#12975 on the tip + #13017). - `typecheck:core` clean; `check:dashboard-typecheck` OK (206 pre-existing, within baseline); `check:changelog-integrity` OK; `check:docs-counts` migrations ✓ - complexity 2816 / baseline 3218 and cognitive-complexity 1271 / baseline 1437 — both under baseline - 531 of 532 focused assertions green across the batch's 46 test files - `check-file-size` rebaselined for the batch's real growth (annotation `_rebaseline_2026_09_11_mergebatch_v3851_houminxi`, landed on #13038), attributed per PR The single red is **not this batch**: `tests/unit/combo/quota-weighted-strategy.test.ts` → "A/B isolation: 7 hard-empty + 2 at 0.5% + 1 at 40%, floor=1" asserts an order between two connections of identical weight and flakes on the pure tip too — 2 failures in 4 runs at `origin/release/v3.8.51` with nothing from this batch applied. ⚠️ base-red inherited: #12732 — `Docs Gates`, `Merge integrity`, `No new ESLint warnings`, `Unit Tests fast-path` and `Fast Quality Gates` reproduce on the pure tip (provider count 356 vs the 358 the modules define, SKILL.md drift, and `open-sse/utils/stream.ts` at 3115 > frozen 3098, untouched here). Thanks @HouMinXi — the live evidence on these (X500 logs, `storage.sqlite` state, real `/v1/models` probes, the 36-minute outage write-up) is what let a 20-PR batch be reviewed as a unit. --- .../fixes/grok-cli-shared-wallet-402.md | 1 + open-sse/services/accountFallback.ts | 4 +- .../accountFallback/sharedWalletCredits.ts | 39 ++++ open-sse/services/combo/targetExhaustion.ts | 28 +++ src/sse/services/auth.ts | 14 +- .../auth-grok-cli-402-shared-wallet.test.ts | 186 ++++++++++++++++++ .../combo/combo-target-exhaustion.test.ts | 56 ++++++ 7 files changed, 325 insertions(+), 3 deletions(-) create mode 100644 changelog.d/fixes/grok-cli-shared-wallet-402.md create mode 100644 open-sse/services/accountFallback/sharedWalletCredits.ts create mode 100644 tests/unit/auth-grok-cli-402-shared-wallet.test.ts diff --git a/changelog.d/fixes/grok-cli-shared-wallet-402.md b/changelog.d/fixes/grok-cli-shared-wallet-402.md new file mode 100644 index 0000000000..5559902c6b --- /dev/null +++ b/changelog.d/fixes/grok-cli-shared-wallet-402.md @@ -0,0 +1 @@ +- **fix(grok-cli):** a 402 "Grok Build usage balance exhausted" parks that Grok login as out of credit (Grok Build CLI, grok.com cookie, and xAI OAuth share the weekly pool). Combo routing then tries the next login instead of locking the model for every account in the pool diff --git a/open-sse/services/accountFallback.ts b/open-sse/services/accountFallback.ts index bf66815a5d..d48bc88716 100644 --- a/open-sse/services/accountFallback.ts +++ b/open-sse/services/accountFallback.ts @@ -99,6 +99,7 @@ export { MODEL_LOCKOUT_EVICTION_CAP } from "./accountFallback/lockoutEviction.ts import { capScaledCooldownMs } from "./accountFallback/cooldownCap.ts"; import { resolveApiKeyForbiddenFallback } from "./accountFallback/nonRetryableUpstream.ts"; import * as exactModelLock from "./accountFallback/exactModelLock.ts"; +import { isCreditsExhaustedWithSharedWallet } from "./accountFallback/sharedWalletCredits.ts"; export type ProviderProfile = { baseCooldownMs: number; useUpstreamRetryHints: boolean; @@ -485,8 +486,7 @@ export function isAccountDeactivated(errorText: string): boolean { * T10: Returns true if response body indicates credits/quota are permanently exhausted. */ export function isCreditsExhausted(errorText: string): boolean { - const lower = String(errorText || "").toLowerCase(); - return CREDITS_EXHAUSTED_SIGNALS.some((sig) => lower.includes(sig)); + return isCreditsExhaustedWithSharedWallet(errorText, CREDITS_EXHAUSTED_SIGNALS); } /** diff --git a/open-sse/services/accountFallback/sharedWalletCredits.ts b/open-sse/services/accountFallback/sharedWalletCredits.ts new file mode 100644 index 0000000000..d24341bfc0 --- /dev/null +++ b/open-sse/services/accountFallback/sharedWalletCredits.ts @@ -0,0 +1,39 @@ +/** + * Providers whose 402 is a shared account wallet, not a per-model billing miss. + * + * Grok Build (`grok-cli`), grok.com cookie sessions (`grok-web`), and xAI + * OAuth (`xai-oauth`) bill Chat/Imagine/Voice/Build/API against one weekly + * percent pool. `passthroughModels: true` still stands for catalog/404 + * behaviour; it must not send this 402 through the #12242 model-only lockout, + * or a combo of five grok-4.6 steps parks the empty account and then skips the + * remaining live accounts as "model locked". + * + * `matchesSharedWalletCreditsBody` expects a pre-lowercased string. + */ +const SHARED_WALLET_402_PROVIDERS = new Set(["grok-cli", "grok-web", "xai-oauth"]); + +export const GROK_BUILD_USAGE_BALANCE_SIGNAL = "usage balance exhausted"; + +export function matchesSharedWalletCreditsBody(loweredErrorText: string): boolean { + return loweredErrorText.includes(GROK_BUILD_USAGE_BALANCE_SIGNAL); +} + +export function isSharedWalletCredits402( + provider: string | null | undefined, + status: number, + errorText?: string | null +): boolean { + if (status !== 402 || typeof provider !== "string" || !SHARED_WALLET_402_PROVIDERS.has(provider)) { + return false; + } + if (errorText == null || String(errorText).trim() === "") return true; + return matchesSharedWalletCreditsBody(String(errorText).toLowerCase()); +} + +export function isCreditsExhaustedWithSharedWallet( + errorText: string, + signals: readonly string[] +): boolean { + const lower = String(errorText || "").toLowerCase(); + return signals.some((sig) => lower.includes(sig)) || matchesSharedWalletCreditsBody(lower); +} diff --git a/open-sse/services/combo/targetExhaustion.ts b/open-sse/services/combo/targetExhaustion.ts index 7e636e69d0..ca0023bb36 100644 --- a/open-sse/services/combo/targetExhaustion.ts +++ b/open-sse/services/combo/targetExhaustion.ts @@ -33,6 +33,7 @@ import { isCloudflareFingerprintRejection } from "../errorClassifier.ts"; // Exclusive in practice to agentrouter's "额度不足" rule: no opencode-family // rule matches 403 today, so only agentrouter reaches this predicate via 403. import { isAgentrouterConnectionQuotaScope } from "@/sse/services/auth"; +import { isSharedWalletCredits402 } from "../accountFallback/sharedWalletCredits.ts"; import type { ComboLogger, ResolvedComboTarget } from "./types.ts"; // Connection-level failure statuses: the provider connection itself is likely bad (upstream @@ -168,6 +169,11 @@ export function applyComboTargetExhaustion( return true; } + if (isSharedWalletCredits402(provider, result.status, opts.errorText)) { + markSharedWalletCreditsExhaustion(target, { sets, log, tag }); + return true; + } + // #8133/#8137: auth-level failures (401/403) mean that connection's credentials are bad. // Split out to keep applyComboTargetExhaustion under the complexity ceiling. // Cloudflare 1010 (a 403 carrying error_code 1010 / browser_signature_banned) is NOT an @@ -343,6 +349,28 @@ function markAuthLevelExhaustion( } } +function markSharedWalletCreditsExhaustion( + target: ResolvedComboTarget, + opts: Pick +): void { + const { sets, log, tag } = opts; + const provider = target.provider; + const connId = target.connectionId ?? undefined; + if (connId) { + sets.exhaustedConnections.add(`${provider}:${connId}`); + log.info( + tag, + `Provider ${provider} connection ${connId} shared-wallet 402 — marking for skip on remaining targets` + ); + } else { + sets.exhaustedProviders.add(provider as string); + log.info( + tag, + `Provider ${provider} shared-wallet 402 (no connectionId) — marking for skip on remaining targets` + ); + } +} + /** * #10334: connection-scope account quota exhaustion (agentrouter-exclusive in * practice — see above). Mirrors diff --git a/src/sse/services/auth.ts b/src/sse/services/auth.ts index d0fff53479..999674480b 100644 --- a/src/sse/services/auth.ts +++ b/src/sse/services/auth.ts @@ -75,6 +75,7 @@ import { retryHintBypassesMaxCooldownMs, isProviderModelUnsupported400, } from "@omniroute/open-sse/services/accountFallback.ts"; +import { isSharedWalletCredits402 } from "@omniroute/open-sse/services/accountFallback/sharedWalletCredits.ts"; import { isLocalProvider } from "@omniroute/open-sse/config/providerRegistry.ts"; import { COOLDOWN_MS, RateLimitReason } from "@omniroute/open-sse/config/constants.ts"; import { sanitizeErrorMessage } from "@omniroute/open-sse/utils/errorSanitization.ts"; @@ -3020,6 +3021,11 @@ export async function markAccountUnavailable( return { shouldFallback: true, cooldownMs: lockout.cooldownMs }; } const result = fallbackResult; + if (isSharedWalletCredits402(provider, status, errorText)) { + result.creditsExhausted = true; + result.reason = result.reason || RateLimitReason.QUOTA_EXHAUSTED; + result.shouldFallback = true; + } const { shouldFallback, cooldownMs: rawCooldownMs, newBackoffLevel, reason } = result; if (!shouldFallback) return { shouldFallback: false, cooldownMs: 0 }; const providerErrorType = classifyProviderError(status, errorText, provider); @@ -3138,6 +3144,7 @@ export async function markAccountUnavailable( provider && model && !terminalStatus && + !isSharedWalletCredits402(provider, status, errorText) && !(provider === "vertex" && isVertexConnectionWidePermissionDenied(errorText)) ) { const lockoutReason = status === 402 ? "credits" : "forbidden"; @@ -3259,7 +3266,12 @@ export async function markAccountUnavailable( // the DB, but record an in-memory model lockout so credential selection // skips this exact provider+connection+model while it cools down — other // models on the same connection stay usable. - if (provider && model && cooldownMs > 0) { + if ( + provider && + model && + cooldownMs > 0 && + !isSharedWalletCredits402(provider, status, errorText) + ) { lockModel(provider, connectionId, model, reason || "unknown", cooldownMs); } await updateProviderConnection(connectionId, { diff --git a/tests/unit/auth-grok-cli-402-shared-wallet.test.ts b/tests/unit/auth-grok-cli-402-shared-wallet.test.ts new file mode 100644 index 0000000000..0f3fcf5eca --- /dev/null +++ b/tests/unit/auth-grok-cli-402-shared-wallet.test.ts @@ -0,0 +1,186 @@ +// Grok Build (`grok-cli`) bills Chat/Imagine/Voice/Build/API against one +// weekly credit pool. A 402 "Grok Build usage balance exhausted" is therefore +// a connection-wide wallet signal, not a per-model billing miss. The +// passthroughModels flag still stands for catalog/404 behaviour; it must not +// route this 402 through the #12242 model-only lockout, or a combo of five +// grok-4.6 steps parks the first empty account and then skips the four +// remaining live accounts as "model locked". +import test from "node:test"; +import assert from "node:assert/strict"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; + +const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-grok-cli-402-")); +process.env.DATA_DIR = TEST_DATA_DIR; + +const core = await import("../../src/lib/db/core.ts"); +const providersDb = await import("../../src/lib/db/providers.ts"); +const auth = await import("../../src/sse/services/auth.ts"); +const accountFallback = await import("../../open-sse/services/accountFallback.ts"); + +const GROK_BUILD_402 = "Grok Build usage balance exhausted"; + +async function resetStorage() { + core.resetDbInstance(); + fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true, maxRetries: 5, retryDelay: 50 }); + fs.mkdirSync(TEST_DATA_DIR, { recursive: true }); +} + +async function seedGrokCli(name: string) { + return seedSharedWallet("grok-cli", name); +} + +async function seedSharedWallet(provider: string, name: string) { + const oauth = provider === "grok-cli" || provider === "xai-oauth"; + return providersDb.createProviderConnection({ + provider, + authType: oauth ? "oauth" : "apikey", + name, + email: name, + ...(oauth + ? { accessToken: `${provider}-${name}` } + : { apiKey: `${provider}-${name}` }), + isActive: true, + testStatus: "active", + }); +} + +test.after(() => { + core.resetDbInstance(); + fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true }); +}); + +test("grok-cli 402 parks the connection as credits_exhausted, not a model lock", async () => { + await resetStorage(); + const conn = await seedGrokCli("empty@qq.com"); + const id = (conn as { id: string }).id; + + const result = await auth.markAccountUnavailable(id, 402, GROK_BUILD_402, "grok-cli", "grok-4.6"); + + assert.equal(result.shouldFallback, true); + + const after = await providersDb.getProviderConnectionById(id); + assert.equal(after.testStatus, "credits_exhausted"); + + const lockout = accountFallback.getModelLockoutInfo("grok-cli", id, "grok-4.6"); + assert.equal(lockout, null, "shared-wallet 402 must not lock grok-4.6 on this account"); +}); + +test("a sibling grok-cli account stays eligible after another account's 402", async () => { + await resetStorage(); + const empty = await seedGrokCli("empty@qq.com"); + const live = await seedGrokCli("live@hotmail.com"); + const emptyId = (empty as { id: string }).id; + const liveId = (live as { id: string }).id; + + await auth.markAccountUnavailable(emptyId, 402, GROK_BUILD_402, "grok-cli", "grok-4.6"); + + assert.equal( + accountFallback.isModelLocked("grok-cli", liveId, "grok-4.6"), + false, + "sibling account must not inherit the empty account's model lock" + ); + + const selected = await auth.getProviderCredentials("grok-cli"); + assert.ok(selected); + assert.equal(selected.connectionId, liveId); + assert.notEqual(selected.connectionId, emptyId); +}); + +test("grok-cli 402 still parks the connection when disableCooling is set", async () => { + await resetStorage(); + const conn = await providersDb.createProviderConnection({ + provider: "grok-cli", + authType: "oauth", + accessToken: "gcli-disabled-cooling", + isActive: true, + testStatus: "active", + providerSpecificData: { disableCooling: true }, + }); + const id = (conn as { id: string }).id; + + await auth.markAccountUnavailable(id, 402, GROK_BUILD_402, "grok-cli", "grok-4.6"); + + const after = await providersDb.getProviderConnectionById(id); + assert.equal(after.testStatus, "credits_exhausted"); +}); + +test("passthrough 402 on ollama-cloud still locks only the paid model (#12242)", async () => { + await resetStorage(); + const conn = await providersDb.createProviderConnection({ + provider: "ollama-cloud", + authType: "apikey", + apiKey: "ollama-cloud-test-key", + isActive: true, + testStatus: "active", + }); + const id = (conn as { id: string }).id; + + await auth.markAccountUnavailable( + id, + 402, + "Add credits to continue, or switch to a free model", + "ollama-cloud", + "gpt-chat-latest" + ); + + const after = await providersDb.getProviderConnectionById(id); + assert.equal(after.testStatus, "active"); + assert.equal( + accountFallback.getModelLockoutInfo("ollama-cloud", id, "gpt-chat-latest")?.reason, + "credits" + ); +}); + +test("Grok Build usage balance exhausted matches the credits-exhausted signal", () => { + assert.equal(accountFallback.isCreditsExhausted(GROK_BUILD_402), true); +}); + +test("a grok-cli 402 with an unrelated body does not park the connection", async () => { + await resetStorage(); + const conn = await seedGrokCli("empty-unrelated@qq.com"); + const id = (conn as { id: string }).id; + await auth.markAccountUnavailable( + id, + 402, + "Add credits to continue, or switch to a free model", + "grok-cli", + "grok-4.6" + ); + const after = await providersDb.getProviderConnectionById(id); + assert.equal(after.testStatus, "active"); + assert.equal( + accountFallback.getModelLockoutInfo("grok-cli", id, "grok-4.6")?.reason, + "credits" + ); +}); + +for (const provider of ["grok-web", "xai-oauth"] as const) { + test(`${provider} 402 parks the connection as credits_exhausted, not a model lock`, async () => { + await resetStorage(); + const conn = await seedSharedWallet(provider, `empty@${provider}.example`); + const id = (conn as { id: string }).id; + const model = provider === "grok-web" ? "fast" : "grok-4.5"; + + await auth.markAccountUnavailable(id, 402, GROK_BUILD_402, provider, model); + + const after = await providersDb.getProviderConnectionById(id); + assert.equal(after.testStatus, "credits_exhausted"); + assert.equal( + accountFallback.getModelLockoutInfo(provider, id, model), + null, + `${provider} shares the Grok weekly wallet` + ); + }); +} + +test("a grok-cli 402 with empty body parks the connection as credits_exhausted", async () => { + await resetStorage(); + const conn = await seedGrokCli("empty-nobody@qq.com"); + const id = (conn as { id: string }).id; + await auth.markAccountUnavailable(id, 402, "", "grok-cli", "grok-4.6"); + const after = await providersDb.getProviderConnectionById(id); + assert.equal(after.testStatus, "credits_exhausted"); + assert.equal(accountFallback.getModelLockoutInfo("grok-cli", id, "grok-4.6"), null); +}); diff --git a/tests/unit/combo/combo-target-exhaustion.test.ts b/tests/unit/combo/combo-target-exhaustion.test.ts index 8d4e3104d1..5f8f98501a 100644 --- a/tests/unit/combo/combo-target-exhaustion.test.ts +++ b/tests/unit/combo/combo-target-exhaustion.test.ts @@ -36,6 +36,7 @@ const baseOpts = { rawModel: "m1", isTokenLimitBreach: false, allAccountsRateLimited: false, + requestScopedFailure: false, log, tag: "COMBO", exhaustedLogLevel: "info" as const, @@ -683,6 +684,61 @@ test("sibling connection on the same provider is NOT skipped after a different c assert.ok(s.exhaustedConnections.has(`${failingTarget.provider}:${failingTarget.connectionId}`)); }); +test("grok-cli 402 marks only the empty connection, not the whole provider", () => { + const s = sets(); + const empty = target({ + provider: "grok-cli", + connectionId: "qq-empty", + modelStr: "grok-cli/grok-4.6", + }); + const sibling = target({ + provider: "grok-cli", + connectionId: "hotmail-live", + modelStr: "grok-cli/grok-4.6", + }); + + const exhausted = applyComboTargetExhaustion(empty, { + ...baseOpts, + result: { status: 402 }, + fallbackResult: { creditsExhausted: true, reason: "quota_exhausted" }, + errorText: "Grok Build usage balance exhausted", + rawModel: "grok-4.6", + sets: s, + }); + + assert.equal(exhausted, true); + assert.ok(s.exhaustedConnections.has("grok-cli:qq-empty")); + assert.equal( + s.exhaustedProviders.has("grok-cli"), + false, + "sibling grok-cli accounts still have weekly credits" + ); + assert.equal(s.exhaustedConnections.has("grok-cli:hotmail-live"), false); + void sibling; +}); + +for (const provider of ["grok-web", "xai-oauth"] as const) { + test(`${provider} 402 with empty body marks only that connection`, () => { + const s = sets(); + const empty = target({ + provider, + connectionId: "empty", + modelStr: `${provider}/m`, + }); + const exhausted = applyComboTargetExhaustion(empty, { + ...baseOpts, + result: { status: 402 }, + fallbackResult: {}, + errorText: "", + rawModel: "m", + sets: s, + }); + assert.equal(exhausted, true); + assert.ok(s.exhaustedConnections.has(`${provider}:empty`)); + assert.equal(s.exhaustedProviders.has(provider), false); + }); +} + test("401 carrying a real fingerprint signal still marks auth-level (exemption is 403-only)", () => { // Round 4 finding: Cloudflare 1010 is a 403-only CDN signal. A 401 invalid-credential // whose errorText carries a genuinely Cloudflare-keyed 1010 (error_code: 1010) must still From 541a6481a90c431398736d4f922343f9837d0ed9 Mon Sep 17 00:00:00 2001 From: "Bob.Hou" Date: Fri, 11 Sep 2026 18:27:45 -0400 Subject: [PATCH 04/77] fix(resilience): classify Cline 401 as refreshable OAuth (#12594) (#13060) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two real defects in one report, both correctly separated: the missing phrase in `OAUTH_INVALID_TOKEN_SIGNALS` made a recoverable Cline 401 terminal even though `refreshClineToken` already existed, and the cooling panel printing "429 (rate-limit)" for every future cooldown hid what actually happened. --- Validated in one consolidated worktree cut from `release/v3.8.51`, boarded with the other 19 PRs of this batch. Two in-batch conflicts, both additive and resolved by keeping each side: the `ENVIRONMENT.md` table (#13035 + #13011) and the `chatHelpers.ts` import block (#12975 on the tip + #13017). - `typecheck:core` clean; `check:dashboard-typecheck` OK (206 pre-existing, within baseline); `check:changelog-integrity` OK; `check:docs-counts` migrations ✓ - complexity 2816 / baseline 3218 and cognitive-complexity 1271 / baseline 1437 — both under baseline - 531 of 532 focused assertions green across the batch's 46 test files - `check-file-size` rebaselined for the batch's real growth (annotation `_rebaseline_2026_09_11_mergebatch_v3851_houminxi`, landed on #13038), attributed per PR The single red is **not this batch**: `tests/unit/combo/quota-weighted-strategy.test.ts` → "A/B isolation: 7 hard-empty + 2 at 0.5% + 1 at 40%, floor=1" asserts an order between two connections of identical weight and flakes on the pure tip too — 2 failures in 4 runs at `origin/release/v3.8.51` with nothing from this batch applied. ⚠️ base-red inherited: #12732 — `Docs Gates`, `Merge integrity`, `No new ESLint warnings`, `Unit Tests fast-path` and `Fast Quality Gates` reproduce on the pure tip (provider count 356 vs the 358 the modules define, SKILL.md drift, and `open-sse/utils/stream.ts` at 3115 > frozen 3098, untouched here). Thanks @HouMinXi — the live evidence on these (X500 logs, `storage.sqlite` state, real `/v1/models` probes, the 36-minute outage write-up) is what let a 20-PR batch be reviewed as a unit. --- changelog.d/fixes/12594-cline-401-oauth.md | 1 + open-sse/services/accountFallback.ts | 1 + .../components/CoolingConnectionsPanel.tsx | 29 ++++++- .../CoolingConnectionsPanel.test.tsx | 29 +++++++ src/i18n/messages/ar.json | 2 +- src/i18n/messages/az.json | 2 +- src/i18n/messages/bg.json | 2 +- src/i18n/messages/bn.json | 2 +- src/i18n/messages/cs.json | 2 +- src/i18n/messages/da.json | 2 +- src/i18n/messages/de.json | 2 +- src/i18n/messages/en.json | 2 +- src/i18n/messages/es.json | 2 +- src/i18n/messages/fa.json | 2 +- src/i18n/messages/fi.json | 2 +- src/i18n/messages/fr.json | 2 +- src/i18n/messages/gu.json | 2 +- src/i18n/messages/he.json | 2 +- src/i18n/messages/hi.json | 2 +- src/i18n/messages/hu.json | 2 +- src/i18n/messages/id.json | 2 +- src/i18n/messages/it.json | 2 +- src/i18n/messages/ja.json | 2 +- src/i18n/messages/ko.json | 2 +- src/i18n/messages/mr.json | 2 +- src/i18n/messages/ms.json | 2 +- src/i18n/messages/nl.json | 2 +- src/i18n/messages/no.json | 2 +- src/i18n/messages/phi.json | 2 +- src/i18n/messages/pl.json | 2 +- src/i18n/messages/pt-BR.json | 2 +- src/i18n/messages/pt.json | 2 +- src/i18n/messages/ro.json | 2 +- src/i18n/messages/ru.json | 2 +- src/i18n/messages/sk.json | 2 +- src/i18n/messages/sv.json | 2 +- src/i18n/messages/sw.json | 2 +- src/i18n/messages/ta.json | 2 +- src/i18n/messages/te.json | 2 +- src/i18n/messages/th.json | 2 +- src/i18n/messages/tr.json | 2 +- src/i18n/messages/uk-UA.json | 2 +- src/i18n/messages/ur.json | 2 +- src/i18n/messages/vi.json | 2 +- src/i18n/messages/zh-CN.json | 2 +- src/i18n/messages/zh-TW.json | 2 +- tests/unit/cline-401-oauth-12594.test.ts | 86 +++++++++++++++++++ 47 files changed, 186 insertions(+), 44 deletions(-) create mode 100644 changelog.d/fixes/12594-cline-401-oauth.md create mode 100644 tests/unit/cline-401-oauth-12594.test.ts diff --git a/changelog.d/fixes/12594-cline-401-oauth.md b/changelog.d/fixes/12594-cline-401-oauth.md new file mode 100644 index 0000000000..02643a94ae --- /dev/null +++ b/changelog.d/fixes/12594-cline-401-oauth.md @@ -0,0 +1 @@ +- Cline 401 bodies that say "re-authenticate your Cline account" classify as a refreshable OAuth token, not a terminal expired key. The cooling panel no longer labels every cooldown as a 429; it shows the recorded last error instead. (#12594) diff --git a/open-sse/services/accountFallback.ts b/open-sse/services/accountFallback.ts index d48bc88716..1d9e79cbb1 100644 --- a/open-sse/services/accountFallback.ts +++ b/open-sse/services/accountFallback.ts @@ -263,6 +263,7 @@ export const OAUTH_INVALID_TOKEN_SIGNALS = [ "login cookie", "valid authentication credential", "invalid credentials", + "re-authenticate your cline account", ]; // A model that upstream has permanently retired — Gemini's deprecated-model 404 diff --git a/src/app/(dashboard)/dashboard/providers/[id]/components/CoolingConnectionsPanel.tsx b/src/app/(dashboard)/dashboard/providers/[id]/components/CoolingConnectionsPanel.tsx index 48e391e652..97343fba3d 100644 --- a/src/app/(dashboard)/dashboard/providers/[id]/components/CoolingConnectionsPanel.tsx +++ b/src/app/(dashboard)/dashboard/providers/[id]/components/CoolingConnectionsPanel.tsx @@ -56,6 +56,19 @@ function isCoolingNow(connection: ConnectionRowConnection, now: number): boolean return Number.isFinite(until) && until > now; } +/** Visible last-error text (truncated) plus the full string for the tooltip. */ +function coolingRecordedError(connection: ConnectionRowConnection): { + display: string; + title: string; +} | null { + const raw = typeof connection.lastError === "string" ? connection.lastError.trim() : ""; + if (!raw) return null; + return { + display: raw.length > 160 ? `${raw.slice(0, 157)}...` : raw, + title: raw, + }; +} + interface ClearCooldownButtonProps { /** Row's connection id — without one there is nothing to PUT, so no button. */ readonly connectionId: string | undefined; @@ -126,7 +139,7 @@ export default function CoolingConnectionsPanel(props: CoolingConnectionsPanelPr {providerText( t, "coolingConnectionsDescription", - "These connections returned a 429 (rate-limit) on their last request. OmniRoute will skip them until the timer expires — no manual disable required." + "These connections are cooling after their last request. OmniRoute will skip them until the timer expires — no manual disable required." )}