From 6b587d0046454cc7f645b9f2daebec6bbeee979b Mon Sep 17 00:00:00 2001 From: "Bob.Hou" Date: Mon, 7 Sep 2026 08:07:19 -0400 Subject: [PATCH] refactor(combo): split executeTarget into gates, attempt, and loop (#12746) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Validado numa worktree combinada com as 16 PRs desta leva sobre `release/v3.8.51`: typecheck:core limpo, check-file-size e check-changelog-integrity OK, complexity 2788/3218 e cognitive 1261/1437, ESLint 0 erros nos 152 arquivos alterados, 771 testes unitários focados, 49 de integração e a suíte vitest:ui completa (2149) verdes. Guardar os contratos com testes ANTES de levantar o bloco (`05059880` no irmão, e o `287658f5` marcando os `executeTargetGates` como lift-as-is) é o que torna um refactor deste tamanho auditável. Sem essa ordem, um extract de 3 mil linhas é indistinguível de uma reescrita. Revalidei depois do merge da base: `combo-attempt-loop`, `execute-target-attempt`, `execute-target-gates` e `combo-loop-safety-timer-leak-11804` — 20/20 — mais typecheck:core limpo e o cap de arquivo OK. O #12811 entra na sequência logo em seguida, com os três commits do round-robin sobre este. --- config/quality/file-size-baseline.json | 3 + open-sse/services/combo.ts | 2141 +---------------- open-sse/services/combo/attemptLoopTypes.ts | 117 + open-sse/services/combo/comboAttemptLoop.ts | 575 +++++ .../services/combo/executeTargetAttempt.ts | 1204 +++++++++ .../services/combo/executeTargetClassify.ts | 54 + open-sse/services/combo/executeTargetGates.ts | 329 +++ ...combo-loop-safety-timer-leak-11804.test.ts | 7 +- tests/unit/combo/combo-attempt-loop.test.ts | 66 + .../unit/combo/execute-target-attempt.test.ts | 288 +++ tests/unit/combo/execute-target-gates.test.ts | 158 ++ 11 files changed, 2912 insertions(+), 2030 deletions(-) create mode 100644 open-sse/services/combo/attemptLoopTypes.ts create mode 100644 open-sse/services/combo/comboAttemptLoop.ts create mode 100644 open-sse/services/combo/executeTargetAttempt.ts create mode 100644 open-sse/services/combo/executeTargetClassify.ts create mode 100644 open-sse/services/combo/executeTargetGates.ts create mode 100644 tests/unit/combo/combo-attempt-loop.test.ts create mode 100644 tests/unit/combo/execute-target-attempt.test.ts create mode 100644 tests/unit/combo/execute-target-gates.test.ts diff --git a/config/quality/file-size-baseline.json b/config/quality/file-size-baseline.json index e8b11b616b..de4d73ff11 100644 --- a/config/quality/file-size-baseline.json +++ b/config/quality/file-size-baseline.json @@ -3,6 +3,7 @@ "_rebaseline_2026_09_05_claude_extra_usage_preflight": "Own growth: open-sse/services/combo.ts 4080->4084 (+4). buildAutoCandidates now forwards connection.providerSpecificData into evaluateQuotaCutoff so a Claude account with blockExtraUsage=false is not dropped at the 5h bar. Irreducible at the existing cutoff call site; the helper lives in claudeExtraUsage.ts (under cap). Covered by tests/unit/quota-preflight.test.ts.", "_rebaseline_2026_09_04_12697_combo_pin_allowlist": "PR #12697 own growth: src/sse/handlers/chat.ts 2454->2458 (+4). checkModelAvailable preflight and handleSingleModelChat now call comboPinAllowlist so a pin-only combo step cannot scan the provider pool after 502/429. Helper lives in src/lib/combos/steps.ts under cap. Covered by tests/unit/combo-pin-implicit-allowlist.test.ts (11/11).", "_rebaseline_2026_09_05_quota_weighted": "feat/quota-weighted-routing own growth: src/app/(dashboard)/dashboard/combos/page.tsx 5066->5080 (+14 = STRATEGY_GUIDANCE_FALLBACK + STRATEGY_RECOMMENDATIONS_FALLBACK entries for quota-weighted; copy is the spec-mandated when/avoid/example and tips, irreducible at the existing fallback maps). Rebased onto 9d1a896c6 where #12671 already grew the same file 5018->5066. Covered by tests/unit/combo/quota-weighted-strategy.test.ts + autocombo-unification.test.ts.", + "_rebaseline_2026_09_03_combo_execute_target_attempt": "Task 3 of handleComboChat split: new leaf open-sse/services/combo/executeTargetAttempt.ts lands at 1205 (check-file-size split-newline; wc -l 1204) above new-file cap 1200. Lift-as-is from combo.ts:1533-2616 retry loop. Pure classify predicates already extracted to executeTargetClassify.ts (54 LOC). Remaining growth is I/O + side effects (handleSingleModel, quality, pin/LKGP, cooldown, lockout) that cannot leave this file without splitting the retry loop mid-request. Frozen at exact LOC so it can only shrink. Covered by tests/unit/combo/execute-target-attempt.test.ts (7/7).", "_rebaseline_2026_09_03_reset_aware_model_family": "Own growth: open-sse/services/combo.ts 4036->4041 (+5). buildAutoCandidates now keys the reset-aware quota cache by getQuotaFetchScope and spreads requestedModel onto the connection so Gemini windows stay off a Claude-empty Antigravity account. Irreducible wiring at the existing fetchResetAwareQuotaWithCache call site; the family helper itself lives in antigravityQuotaFamily.ts. Covered by tests/unit/reset-aware-request-scope-12600.test.ts.", "_rebaseline_2026_09_03_overloaded_not_provider_breaker": "fix/overloaded-not-provider-breaker own growth: open-sse/services/combo.ts 4036->4075 (check-file-size split-newline, +39). Circuit-open pre-skip now records the breaker retryAfter and, when every target was skipped that way, waits the short reset via resolveCircuitOpenWaitDecision (new leaf in comboCooldownRetry.ts) instead of crystallizing ALL_TARGETS_SKIPPED in ~43ms. skippedForCircuitOpen / earliestCircuitOpenRetryMs reset each setTry so a later iteration cannot inherit a stale retryAfter. Irreducible at the existing ALL_TARGETS_SKIPPED chokepoint (same pattern as #7301/#8213 cooldown-wait). Predicate itself lives in circuitBreaker.ts / comboPredicates.ts / chatPredicates.ts, all under cap. Covered by tests/unit/overloaded-not-provider-breaker.test.ts + combo-cooldown-retry.test.ts.", "_rebaseline_2026_09_03_12649_free_tier_reaudit_gateways": "PR #12649 (fix/free-tier-quota-reaudit) own growth: src/shared/constants/providers/apikey/gateways.ts 1459->1462 (+3 = the nara authHint rewritten for the re-audited 7M/day plan now wraps to two lines, plus the Prettier reflow of two pre-existing >100-col authHint lines (oneminai, freebuff) that lint-staged enforces on any touch of the file; additive text at the existing registry chokepoint, same god-file no-split rationale as prior gateways.ts rebaselines: #11786 seekai, #10987 logfare, #10531 freebuff). Covered by tests/unit/free-tier-reaudit-2026-09.test.ts and tests/unit/free-providers-batch-2026-07.test.ts.", @@ -430,6 +431,8 @@ "open-sse/services/accountFallback.ts": 2467, "open-sse/services/adobeFireflyBrowserLogin.ts": 1401, "open-sse/services/combo.ts": 4084, + "open-sse/services/combo.ts": 4080, + "open-sse/services/combo/executeTargetAttempt.ts": 1205, "open-sse/translator/response/openai-responses.ts": 1466, "open-sse/utils/cursorAgentProtobuf.ts": 1547, "open-sse/utils/proxyFetch.ts": 1271, diff --git a/open-sse/services/combo.ts b/open-sse/services/combo.ts index 8e783f1f90..b63aee95ee 100644 --- a/open-sse/services/combo.ts +++ b/open-sse/services/combo.ts @@ -7,38 +7,24 @@ import { checkFallbackError, - classifyLockoutReason, - CONTEXT_OVERFLOW_PATTERNS, decayModelFailureCount, formatRetryAfter, - getModelLockoutInfo, getRuntimeProviderProfile, hasPerModelQuota, - isAccountSemaphoreFull, isModelLocked, - lockModelIfPerModelQuota, - MODEL_ACCESS_DENIED_PATTERNS, - recordModelLockoutFailure, - recordProviderFailure, recordProviderSuccess, - retryHintBypassesMaxCooldownMs, - selectLockoutCooldownMs, } from "./accountFallback.ts"; import { errorResponse, unavailableResponse, errorResponseWithComboDiagnostics, } from "../utils/error.ts"; -import type { ComboDiagnostics } from "../utils/error.ts"; -import { - COMBO_FAILURE_THRESHOLD, - clearComboFailureTracking, - recordComboFailure, -} from "./combo/failureTracker.ts"; -import { buildNoUpstreamResponseDiagnostics, buildRecoveryHint } from "./combo/pinRecovery.ts"; + +import { recordComboFailure } from "./combo/failureTracker.ts"; +import { buildRecoveryHint } from "./combo/pinRecovery.ts"; import { formatExhaustedConnectionKey } from "./combo/comboDiagFormat.ts"; import { buildTargetTimeoutRunner } from "./combo/targetTimeoutRunner.ts"; -import { recordComboRequest, recordComboShadowRequest, getComboMetrics } from "./comboMetrics.ts"; +import { recordComboRequest, getComboMetrics } from "./comboMetrics.ts"; import { qualityScoreFor } from "./routing/index.ts"; import { expandComboSystemPromptIfPresent, @@ -51,22 +37,9 @@ import { isComboCooldownWaitEligible, resolveComboTargetTimeoutMsForCombo, } from "./comboConfig.ts"; -import { - maybeGenerateHandoff, - maybeGenerateUniversalHandoff, - injectUniversalHandoffBody, - SKIP_UNIVERSAL_HANDOFF_FLAG, - type MessageLike, -} from "./contextHandoff.ts"; -import { - recordSessionModelUsage, - getLastSessionModel, - getHandoff, -} from "../../src/lib/db/contextHandoffs.ts"; -import { extractSessionAffinityKey } from "@/sse/services/auth"; + import { getHiddenModelsByProvider } from "@/models"; -import { resolveModelLockoutSettings } from "../../src/lib/resilience/modelLockoutSettings"; -import { fetchCodexQuota } from "./codexQuotaFetcher.ts"; + import { evaluateQuotaCutoff, getQuotaFetcher, type QuotaInfo } from "./quotaPreflight.ts"; import { resolveProviderId } from "../../src/shared/constants/providers.ts"; import { getQuotaFetchScope } from "./antigravityQuotaFamily.ts"; @@ -76,11 +49,9 @@ import { parseModel } from "./model.ts"; import { rejectRetiredAutoComboCandidates } from "./modelLifecycle.ts"; import { createComboContext } from "./combo/context.ts"; import { phaseComboSetup } from "./combo/comboSetup.ts"; -import { checkCredentialGate, logCredentialSkip } from "./credentialGate.ts"; -import { emit } from "../../src/lib/events/eventBus"; -import { notifyWebhookEvent } from "../../src/lib/webhookDispatcher"; + import { type ProviderCandidate } from "./autoCombo/scoring.ts"; -import { estimateTokens } from "./contextManager.ts"; + import { getSessionConnection } from "./sessionManager.ts"; import { getOAuthSessionAvailability } from "./oauthSessionOccupancy.ts"; import { @@ -92,10 +63,10 @@ import { peekStickyConnectionId, resolveDisableSessionStickiness, } from "./combo/sessionStickiness.ts"; -import { selectQuotaShareTarget } from "./combo/quotaShareStrategy.ts"; + import { makeConnectionConcurrencyResolver, lookupPositiveCap } from "./combo/concurrencyCaps.ts"; import { acquireQuotaShareConcurrencySlot } from "./combo/quotaShareConcurrency.ts"; -import { canAffordRequest } from "../../src/lib/quota/quotaScheduler.ts"; + import { resolveConnectionTimeoutMs } from "../handlers/chatCore/upstreamTimeouts.ts"; import { getCachedProviderConnectionById } from "../../src/lib/db/readCache.ts"; import { orderTargetsByEvalScores } from "./evalRouting.ts"; @@ -131,11 +102,10 @@ import { classifyComboOutcome, formatComboOutcomes, redactConnectionLabel, - buildRedactedSummary, resolveComboTerminalStatus, } from "./combo/comboErrorAggregation.ts"; import type { ComboErrorEntry } from "./combo/comboErrorAggregation.ts"; -import type { CompressionMode } from "./compression/types.ts"; + import { getCachedProviderConnections } from "../../src/lib/db/readCache"; import { isProviderInCooldown, recordProviderCooldown } from "./providerCooldownTracker.ts"; import { @@ -162,12 +132,8 @@ import { MAX_RR_COUNTERS, rrCounters, rrStickyTargets, - clampStickyRoundRobinTargetLimit, - clampStickyWeightedTargetLimit, getStickyRoundRobinStartIndex, recordStickyRoundRobinSuccess, - getStickyWeightedExecutionKey, - recordStickyWeightedSuccess, resolveComboStickyRoundRobinLimit, } from "./combo/rrState.ts"; import { expandTargetsForAllStrategies } from "./combo/connectionAwareExpansion.ts"; @@ -177,23 +143,13 @@ import { releaseRejectedQualityResponse, toRetryAfterDisplayValue, } from "./combo/validateQuality.ts"; -import { - resolveComboCooldownWaitDecision, - resolveCircuitOpenWaitDecision, - ResolveComboCooldownDecisionResult, -} from "./combo/comboCooldownRetry.ts"; -import { - computeClosestRetryAfter, - waitForCooldownAwareRetry, -} from "../../src/sse/services/cooldownAwareRetry.ts"; -import { dispatchChaosFromCombo, type ChaosTuning } from "./autoCombo/chaosEngine.ts"; +import { dispatchChaosFromCombo } from "./autoCombo/chaosEngine.ts"; import { TRANSIENT_FOR_SEMAPHORE, MAX_FALLBACK_WAIT_MS, MAX_GLOBAL_ATTEMPTS, MAX_GLOBAL_ATTEMPTS_HARD_CAP, COMBO_LOOP_SAFETY_TIMEOUT_MS, - COMBO_SAFETY_DRAIN_MS, isAllAccountsRateLimitedResponse, clampComboDepth, clampGlobalAttempts, @@ -201,20 +157,15 @@ import { shouldRecordProviderBreakerFailure, isComboRequestScopedFailure as isScopedFailure, isRequestScopedUpstreamFailure, - isInputBoundRequestFailure, shouldSkipConnDisable, resolveDelayMs, comboModelNotFoundResponse, isStreamReadinessFailureErrorBody, - isStreamEarlyEofErrorBody, isTokenLimitBreachErrorBody, isLocalQueueCapacityErrorBody, toRecordedTarget, getExhaustedTargetSkipReason, - clampPercent, quotaRemainingPercentFromQuota, - normalizeConnectionStatus, - hasFutureRateLimitUntil, getConnectionStatusQuotaCutoffReason, getPersistedConnectionCooldownSkipReason, resolvePersistedConnectionCooldownSkipReason, @@ -236,7 +187,6 @@ import { areAllPinnedTargetsModelScopedUnusable, createPinnedModelUnavailableResponse, getNativeCodexTurnPin, - pinNativeCodexTurn, } from "./combo/nativeCodexTurnPin.ts"; import { pinIsDurablyUnhealthy, @@ -260,14 +210,7 @@ import { resolveComboRuntimeUnits, resolveComboTargets, } from "./combo/comboStructure.ts"; -import { - createInvocationId, - finalizeComboTrace, - finishComboTrace, - getComboTrace, - recordComboDecision, - startComboTrace, -} from "./combo/decisionTrace.ts"; +import { createInvocationId, getComboTrace, startComboTrace } from "./combo/decisionTrace.ts"; import { QUOTA_SOFT_DEPRIORITIZE_FACTOR, setCandidateQuotaSoftPenalty, @@ -284,17 +227,13 @@ import { type ResetWindowConfig, } from "./combo/quotaScoring.ts"; import { fetchResetAwareQuotaWithCache, preScreenTargets } from "./combo/quotaStrategies.ts"; -import { - buildAutoQuotaThresholds, - resolveQuotaExhaustionCutoffForTarget, -} from "./combo/quotaExhaustionCutoff.ts"; +import { buildAutoQuotaThresholds } from "./combo/quotaExhaustionCutoff.ts"; import { expandTargetsByFingerprints } from "./combo/fingerprintExpansion.ts"; import { resolveComboTargetPipeline } from "./combo/targetResolution.ts"; -import { - isQuotaExhaustionResponse, - recordQuotaExhaustionClassification, - withQuotaExhaustionClassification, -} from "./combo/quotaExhaustion.ts"; +import { dispatchWithCooldownRetry } from "./combo/comboAttemptLoop.ts"; +import { evaluateExecuteTargetGates } from "./combo/executeTargetGates.ts"; +import { executeTargetAttempt } from "./combo/executeTargetAttempt.ts"; +import type { AttemptLoopDeps, AttemptLoopState } from "./combo/attemptLoopTypes.ts"; export { RESET_WINDOW_NAMES, QUOTA_SOFT_DEPRIORITIZE_FACTOR, setCandidateQuotaSoftPenalty }; export { scoreAutoTargets, expandAutoComboCandidatePool }; @@ -332,26 +271,6 @@ export { * peekStickyConnectionId guards against clearing an unrelated pin when the * failing target isn't actually the currently sticky-bound connection. */ -/** - * Connection read for the pre-dispatch persisted-cooldown gate. - * - * `fresh: false` (first attempt) uses the shared 5s readCache — the row was just - * read by the surrounding target resolution, so a second uncached hit is pure cost. - * `fresh: true` (every retry) goes straight to SQLite: during a burst a sibling - * request routinely writes `rate_limited_until` while this attempt is sleeping out - * its retry delay, so the cached snapshot would still say "no cooldown" — which is - * exactly how a retry ended up dispatching into a real upstream 429 on a connection - * the engine had already marked unavailable. - */ -async function readConnectionForCooldownGate( - connectionId: string, - fresh: boolean -): Promise | null | undefined> { - if (!fresh) return getCachedProviderConnectionById(connectionId); - const { getProviderConnectionById } = await import("@/lib/db/providers"); - return (await getProviderConnectionById(connectionId)) as Record | null; -} - export function releaseStickyPinOnFailure( messageHash: string | null | undefined, failedConnectionId: string | null | undefined @@ -618,7 +537,10 @@ export async function buildAutoCandidates( const quota = await quotaPromises.get(quotaKey)!; resetWindowAffinity = calculateResetWindowAffinity(quota, resetWindowConfig); if (!quotaCutoffBlocked) { - quotaRemaining = quotaRemainingPercentFromQuota(quota, { provider, requestedModel: modelStr }); + quotaRemaining = quotaRemainingPercentFromQuota(quota, { + provider, + requestedModel: modelStr, + }); } if (!quotaCutoffBlocked && quotaCutoffEnabled) { const cutoffDecision = evaluateQuotaCutoff( @@ -946,8 +868,6 @@ async function handleComboChatInner({ } const maxRetries = activeNativeTurnPin ? 0 : (config.maxRetries ?? 1); - const retryDelayMs = resolveDelayMs(config.retryDelayMs, 2000); - const fallbackDelayMs = resolveDelayMs(config.fallbackDelayMs, 0); const maxSetRetries = activeNativeTurnPin ? 0 : (config.maxSetRetries ?? 0); const setRetryDelayMs = resolveDelayMs(config.setRetryDelayMs, 2000); @@ -1056,1934 +976,96 @@ async function handleComboChatInner({ // We snapshot them now so cleanup can happen after the attempt loop finishes. const _registeredExecutionKeys = orderedTargets.map((t) => t.executionKey).filter(Boolean); - let globalAttempts = 0; - // #11134: operator-configurable shared attempt budget (clamped to the hard - // cap). Defaults to MAX_GLOBAL_ATTEMPTS when unset. - const maxGlobalAttempts = clampGlobalAttempts(config.maxGlobalAttempts); - - // Cooldown-aware retry (Variante A). Originally quota-share (qtSd/) only; - // extended to "auto" combos too (#7360 — a 2-model "default" auto combo - // hitting Gemini TPM/RPM on both targets was crystallizing a 503 "all - // targets exhausted" after ~6s instead of waiting out the ~60s TPM window): - // when the set loop would crystallize a 429 model_cooldown because the - // target hit a SHORT transient cooldown, we wait it out and re-run the - // whole set loop instead of propagating the 429. `globalAttempts` persists - // across these waits so MAX_GLOBAL_ATTEMPTS still bounds total work. The - // wait happens at the crystallization point. The only semaphore slot the - // quota-share path may hold is the FASE 2.1 per-connection concurrency slot - // (acquired once around dispatchWithCooldownRetry below); it is intentionally - // kept across the wait so the account stays "busy", and is released by the - // outer finally — not here. - // - // The set loop is wrapped in a small recursive closure rather than an extra - // labelled `while (true)` so the loop body keeps its original indentation; a - // wait+redispatch is a tail `return dispatchWithCooldownRetry()`, which - // re-runs ONLY the set loop (selection / shadow routing / setup above stay - // untouched), preserving the pre-existing `continue`-to-top-of-set-loop - // semantics exactly. const comboCooldownWaitEnabled = isComboCooldownWaitEligible( strategy, resilienceSettings.comboCooldownWait ); - let comboCooldownAttempt = 0; - let comboCooldownBudgetLeftMs = resilienceSettings.comboCooldownWait.budgetMs; - - // Global combo timeout: when set (>0), limits total wall-clock time the combo - // spends iterating through targets. After each target completes, if elapsed time - // exceeds comboTimeoutMs, remaining targets are skipped and a 504 with aggregated - // error diagnostics is returned. 0 = disabled (backward-compatible, unlimited). + const comboCooldownAttempt = { current: 0 }; + const comboCooldownBudgetLeftMs = { current: resilienceSettings.comboCooldownWait.budgetMs }; const comboTimeoutMs = config.comboTimeoutMs || 0; const comboStartTime = Date.now(); - let comboExpired = false; - // Accumulator for per-model error details across targets in the current set try. - // Reset at the start of each set retry (same lifecycle as lastError/recordedAttempts). - let comboErrors: Array = []; - // Quota trust spans set retries and recursive cooldown re-dispatches. Once any - // failure is non-quota, a nested caller must never treat this dispatch as quota-only. - let observedFailure = false; - let allObservedFailuresQuota = true; - const targetFailureTrust = new Map< - string, - { observedFailure: boolean; allObservedFailuresQuota: boolean } - >(); - const observeFailure = (quotaExhausted: boolean, targetExecutionKey?: string) => { - observedFailure = true; - allObservedFailuresQuota &&= quotaExhausted; - if (!targetExecutionKey) return; - const trust = targetFailureTrust.get(targetExecutionKey) ?? { - observedFailure: false, - allObservedFailuresQuota: true, - }; - trust.observedFailure = true; - trust.allObservedFailuresQuota &&= quotaExhausted; - targetFailureTrust.set(targetExecutionKey, trust); + + const state: AttemptLoopState = { + orderedTargets, + fallbackCount: 0, + recordedAttempts: 0, + comboErrors: [], + lastError: null, + lastStatus: null, + earliestRetryAfter: null, + comboExpired: false, + exhaustedProviders: new Set(), + exhaustedConnections: new Set(), + transientRateLimitedProviders: new Set(), + abortControllers: new Map(), + dispatchedTargets: new Set(), + targetFailureTrust: new Map(), + comboAttemptOrder, + skippedForCircuitOpen: false, + earliestCircuitOpenRetryMs: 0, + globalAttempts: 0, + observedFailure: false, + allObservedFailuresQuota: true, + observeFailure(quotaExhausted, targetExecutionKey) { + this.observedFailure = true; + this.allObservedFailuresQuota &&= quotaExhausted; + if (!targetExecutionKey) return; + const trust = this.targetFailureTrust.get(targetExecutionKey) ?? { + observedFailure: false, + allObservedFailuresQuota: true, + }; + trust.observedFailure = true; + trust.allObservedFailuresQuota &&= quotaExhausted; + this.targetFailureTrust.set(targetExecutionKey, trust); + }, + }; + + const deps: AttemptLoopDeps = { + strategy, + combo, + config: config as AttemptLoopDeps["config"], + log, + settings: settings ?? null, + resilienceSettings, + sticky: _sticky, + effectiveSessionId, + preScreenMap, + quotaCutoffResetWindowConfig, + maxRetries, + traceInvocationId, + clientRequestedStream, + handleSingleModelWithTimeout, + isModelAvailable, + perTargetAdmission, + signal, + body: body as Record, + startTime: comboStartTime, + releaseStickyPinOnFailure, + clearStaleLKGP, + clientManagedResponsesContext, + reasoningTokenBufferEnabled, + stickyWeightedLimit, + getWeightedStepKeyForTarget, + universalHandoffConfig, + relayOptions, + relayConfig, + }; + + const extra = { + maxSetRetries, + setRetryDelayMs, + comboTimeoutMs, + comboStartTime, + comboCooldownWaitEnabled, + comboCooldownAttempt, + comboCooldownBudgetLeftMs, + evaluateGates: evaluateExecuteTargetGates, + executeAttempt: executeTargetAttempt, }; - // FASE 2.1: per-connection concurrency limit for quota-share. The gating in - // selectQuotaShareTarget is fail-open and cannot hard-limit a single-connection - // pool, so we serialize concurrent requests to the selected account through a - // per-connection semaphore. Enabled only for quota-share combos (the cap is the - // account's) and gated by the kill-switch; the slot wraps the whole dispatch. const quotaShareConcurrencyEnabled = strategy === "quota-share" && resilienceSettings.quotaShareConcurrencyLimit.enabled; - const dispatchWithCooldownRetry = async (): Promise => { - // #7360: hoisted OUTSIDE the setTry loop (not reset each iteration) so they - // persist across set retries. Without this, a combo whose targets all get - // locked out on setTry 0 would have every SUBSEQUENT setTry pre-skip both - // targets via the isModelLocked check (no real dispatch, so these are never - // touched) — and the wait/crystallize decision below only runs on the FINAL - // setTry, which would see lastStatus/earliestRetryAfter reset to null and - // wrongly fall into the generic "all accounts inactive" 503 instead of ever - // reaching the cooldown-aware wait, even though a real 429 with a known - // retry-after WAS observed earlier in the same dispatch (live incident, - // #7360 follow-up: log id 1784416706646-51 — 6.9s to a 503 despite both - // targets reporting a clean 40s rate_limit lockout on the first attempt). - let lastError: string | null = null; - let earliestRetryAfter: ComboRetryAfter | null = null; - let lastStatus: number | null = null; - let skippedForCircuitOpen = false; - let earliestCircuitOpenRetryMs = 0; - // #11804: the loop-safety timer is armed per setTry iteration but must be - // cleared on EVERY exit path, not just the happy one. Hoisted to function - // scope so the `finally` at the end of this function always reaches it — - // otherwise each error path (all_targets_skipped / all_accounts_inactive / - // aggregated status / final fallback / global timeout) returned to the client - // leaving a 10-minute timer pending, whose closure retains orderedTargets and - // the exhausted provider/connection sets. Field evidence on the issue: the - // client got a 502 immediately, and "Combo loop safety timeout ... - // force-terminating" was logged exactly 600s later. - let activeLoopSafetyTimer: ReturnType | null = null; - - try { - for (let setTry = 0; setTry <= maxSetRetries; setTry++) { - // #1731: Per-set-iteration set of providers whose quota is fully exhausted. - // Reset each retry so providers excluded in a previous attempt get another chance. - const exhaustedProviders = new Set(); - const exhaustedConnections = new Set(); - const transientRateLimitedProviders = new Set(); - skippedForCircuitOpen = false; - earliestCircuitOpenRetryMs = 0; - if (setTry > 0) { - log.info("COMBO", `All targets failed — retrying set (${setTry}/${maxSetRetries})`); - await new Promise((resolve) => { - const timer = setTimeout(resolve, setRetryDelayMs); - signal?.addEventListener( - "abort", - () => { - clearTimeout(timer); - resolve(undefined); - }, - { once: true } - ); - }); - if (signal?.aborted) { - log.info("COMBO", "Client disconnected during set retry delay — aborting"); - return errorResponse(499, "Client disconnected"); - } - } - - const startTime = Date.now(); - let fallbackCount = 0; - let recordedAttempts = 0; - comboErrors = []; - - // QA P0: assemble a sanitized diagnostic trace from the state already in scope - // (pool size + this set-try's exhausted providers/connections + attempt order + - // a terminal-reason code). Never touches keys/tokens — provider/model ids only. - // Silent-stop fix: include a `recovery` hint (action verb + human next-step) so the - // OC plugin + non-header-aware clients can render an actionable error instead of an - // opaque 5xx. The optional `retryAfterSeconds` carries the upstream Retry-After hint. - const buildComboDiag = ( - terminalReason: string, - retryAfterSeconds?: number - ): ComboDiagnostics => ({ - poolSize: orderedTargets.length, - attempted: recordedAttempts, - excluded: [ - ...[...exhaustedProviders].map((p) => ({ provider: p, reason: "exhausted" })), - ...[...exhaustedConnections].map((c) => formatExhaustedConnectionKey(String(c))), - ], - attemptOrder: comboAttemptOrder, - terminalReason, - recovery: buildRecoveryHint(terminalReason, retryAfterSeconds), - }); - - let globalResolve: ((res: Response) => void) | null = null; - const globalPromise = new Promise((res) => { - globalResolve = res; - }); - - // G1 (silent-stop fix): the speculative loop's `Promise.race` waits on - // `globalPromise`, which is ONLY resolved from inside a task (success or - // fatal error). If a target hangs — e.g. the operator disabled the per-model - // timeout (`targetTimeoutMs: 0`) and the upstream never settles — the race - // never resolves and the request hangs forever with no response. This safety - // promise force-resolves after the combo budget (comboTimeoutMs when set, - // otherwise a hard ceiling) so the request ALWAYS terminates with an - // actionable 504 instead of dying silently. `comboExpired` is flipped so the - // target loop stops launching new work; the existing comboExpired branch - // returns the aggregated 504. - const loopSafetyMs = comboTimeoutMs > 0 ? comboTimeoutMs : COMBO_LOOP_SAFETY_TIMEOUT_MS; - let loopSafetyFired = false; - let loopSafetyTimer: ReturnType | null = null; - const loopSafetyPromise = new Promise((resolve) => { - loopSafetyTimer = setTimeout(() => { - loopSafetyFired = true; - log.warn( - "COMBO", - `Combo loop safety timeout (${loopSafetyMs}ms) reached without a terminal response — force-terminating` - ); - resolve( - errorResponseWithComboDiagnostics( - 504, - `Combo global timeout (${loopSafetyMs}ms) without a terminal response`, - buildComboDiag("combo_timeout"), - { code: "COMBO_TIMEOUT", type: "server_error" } - ) - ); - }, loopSafetyMs); - loopSafetyTimer.unref?.(); - activeLoopSafetyTimer = loopSafetyTimer; - }); - const runningTasks = new Set>(); - let anySuccess = false; - // #10681: steps already recorded as dispatched (so per-target retries do not - // duplicate the decision). - const dispatchedTargets = new Set(); - // G1: flip comboExpired as soon as the safety timer fires so the next loop - // iteration breaks instead of launching more targets after the budget, and - // abort every in-flight target so a hung upstream actually gets cancelled - // (not just "response stops"). - const markLoopExpiredIfSafetyFired = () => { - if (loopSafetyFired) { - comboExpired = true; - for (const [, ac] of abortControllers.entries()) ac.abort(); - } - }; - const abortControllers = new Map(); - const zeroLatencyOptimizationsEnabled = config.zeroLatencyOptimizationsEnabled === true; - const hasProtectedPriorityTarget = - strategy === "priority" && - orderedTargets.some((target) => target.fallbackOnlyOnQuotaExhaustion === true); - - const executeTarget = async ( - i: number - ): Promise<{ ok: boolean; response?: Response } | null> => { - const target = orderedTargets[i]; - const modelStr = target.modelStr; - const rawModel = parseModel(modelStr).model || modelStr; - const provider = target.provider; - const protectedPriorityTarget = - strategy === "priority" && target.fallbackOnlyOnQuotaExhaustion === true; - const stopProtectedPriorityTarget = (message: string) => { - observeFailure(false, target.executionKey); - clearStaleLKGP(combo.name, target.executionKey, combo.id, log, "COMBO"); - return protectedPriorityTarget - ? { ok: false, response: errorResponse(503, message) } - : null; - }; - - const cb = getCircuitBreaker(provider); - const cbStatus = cb.getStatus(); - if (cbStatus.state === "OPEN") { - skippedForCircuitOpen = true; - if ( - cbStatus.retryAfterMs > 0 && - (earliestCircuitOpenRetryMs === 0 || cbStatus.retryAfterMs < earliestCircuitOpenRetryMs) - ) { - earliestCircuitOpenRetryMs = cbStatus.retryAfterMs; - } - log.info("COMBO", `Skipping ${modelStr} — circuit breaker OPEN for ${provider}`); - recordComboDecision(traceInvocationId, { - step: target.executionKey, - target: modelStr, - decision: "skipped_before_dispatch", - reason: "circuit_open", - }); - if (i > 0) fallbackCount++; - return stopProtectedPriorityTarget(`Provider ${provider} circuit breaker is open`); - } - - if ( - resilienceSettings.providerCooldown.enabled && - Boolean(provider && provider !== "unknown") && - (isProviderInCooldown(provider, target.connectionId ?? undefined, resilienceSettings) || - isProviderInCooldown(provider, undefined, resilienceSettings)) - ) { - log.info("COMBO", `Skipping ${modelStr} — provider ${provider} in global cooldown`); - recordComboDecision(traceInvocationId, { - step: target.executionKey, - target: modelStr, - decision: "skipped_before_dispatch", - reason: "provider_cooldown", - }); - if (i > 0) fallbackCount++; - return stopProtectedPriorityTarget(`Provider ${provider} is in cooldown`); - } - - // Use pre-screened profile if available, otherwise fetch on demand - const preScreenEntry = preScreenMap.get(target.executionKey); - const profile = preScreenEntry?.profile ?? (await getRuntimeProviderProfile(provider)); - - const allowRateLimitedConnection = - Boolean(provider && provider !== "unknown") && - transientRateLimitedProviders.has(provider); - const targetForAttempt = allowRateLimitedConnection - ? { - ...target, - allowRateLimitedConnection: true, - modelAbortSignal: abortControllers.get(i)!.signal, - } - : { ...target, modelAbortSignal: abortControllers.get(i)!.signal }; - - // Persist the connection cooldown before dispatch. AUTH only learns - // unavailable during credential lookup, so a burst would otherwise - // burn max_concurrent slots on real upstream calls against a row - // SQLite already locked until the reset. - if (target.connectionId && !allowRateLimitedConnection) { - const persistedSkip = await resolvePersistedConnectionCooldownSkipReason( - target, - (id) => readConnectionForCooldownGate(id, false), - allowRateLimitedConnection - ); - if (persistedSkip) { - log.info("COMBO", persistedSkip); - clearStaleLKGP(combo.name, target.executionKey, combo.id, log, "COMBO"); - if (i > 0) fallbackCount++; - return null; - } - } - - // #1731 / #1731v2: skip targets already known-exhausted this request (shared predicate). - const exhaustedSkip = getExhaustedTargetSkipReason( - target, - exhaustedProviders, - exhaustedConnections - ); - if (exhaustedSkip) { - log.info("COMBO", exhaustedSkip); - recordComboDecision(traceInvocationId, { - step: target.executionKey, - target: modelStr, - decision: "skipped_before_dispatch", - reason: "request_exhaustion", - }); - if (i > 0) fallbackCount++; - return stopProtectedPriorityTarget(`Target ${modelStr} is unavailable`); - } - - // Pre-check: skip models locked by the resilience system (model-level lockout) - if (provider && rawModel && isModelLocked(provider, target.connectionId || "", rawModel)) { - log.info("COMBO", `Skipping ${modelStr} — model locked by resilience (cooldown active)`); - recordComboDecision(traceInvocationId, { - step: target.executionKey, - target: modelStr, - decision: "skipped_before_dispatch", - reason: "model_lockout", - }); - if (i > 0) fallbackCount++; - return stopProtectedPriorityTarget(`Model ${modelStr} is locked`); - } - - // #5923 (Finding #4) — honor the same opt-in quota-exhaustion cutoff the - // "auto" strategy already applies (buildAutoCandidates), for every other - // strategy (priority, weighted, etc.). Strictly scoped per (provider, - // connectionId): a 0%-remaining connection is skipped here, but sibling - // connections/models on the same provider are untouched — the provider - // circuit breaker is never touched by this check. The "auto" strategy is - // excluded to avoid a redundant duplicate fetch — it already filtered its - // candidate pool via `routableCandidates` before reaching this loop. - if (strategy !== "auto" && provider && target.connectionId) { - const quotaCutoff = await resolveQuotaExhaustionCutoffForTarget( - provider, - target.connectionId, - resilienceSettings, - quotaCutoffResetWindowConfig, - combo.name, - log, - modelStr - ); - if (quotaCutoff.blocked) { - log.info( - "COMBO", - `Skipping ${modelStr} — quota exhaustion cutoff (${quotaCutoff.reason || "quota_exhausted"})` - ); - clearStaleLKGP(combo.name, target.executionKey, combo.id, log, "COMBO"); - recordComboDecision(traceInvocationId, { - step: target.executionKey, - target: modelStr, - decision: "skipped_before_dispatch", - reason: "quota_cutoff", - }); - if (i > 0) fallbackCount++; - observeFailure(true, target.executionKey); - if (protectedPriorityTarget) { - const protectedTargetTrust = targetFailureTrust.get(target.executionKey); - if (!protectedTargetTrust?.allObservedFailuresQuota) { - return { - ok: false, - response: errorResponse(503, `Target ${modelStr} is unavailable`), - }; - } - } - return null; - } - } - - // Quota-aware scheduling (opt-in, OMNIROUTE_QUOTA_AWARE_ROUTING=1): - // when a per-connection token budget is configured (provider_quota_state), - // skip targets whose remaining budget cannot afford this request — - // BEFORE dispatching — instead of waiting for a 429. Fails open: when - // no budget is configured the decision is always affordable. - if (process.env.OMNIROUTE_QUOTA_AWARE_ROUTING === "1" && provider && target.connectionId) { - const quotaDecision = canAffordRequest( - target.connectionId, - modelStr, - body as Record | null | undefined - ); - if (!quotaDecision.affordable) { - log.info( - "COMBO", - `Skipping ${modelStr} — quota budget ${quotaDecision.reason} (remaining ${quotaDecision.tokensRemaining ?? 0}, cost ${quotaDecision.estimatedCost ?? 0})` - ); - clearStaleLKGP(combo.name, target.executionKey, combo.id, log, "COMBO"); - if (i > 0) fallbackCount++; - return null; - } - } - - // Pre-screen snapshot is NOT used as a permanent skip — availability - // is always re-checked via isModelAvailable below because connection - // cooldowns can expire between setTry retries, making a previously - // unavailable target available again. Circuit-breaker-OPEN providers - // are already caught by the dedicated breaker check above. - if (isModelAvailable) { - const available = await isModelAvailable(modelStr, targetForAttempt); - if (!available) { - log.debug?.( - "COMBO", - `Skipping ${modelStr} — no credentials available or model excluded` - ); - clearStaleLKGP(combo.name, target.executionKey, combo.id, log, "COMBO"); - recordComboDecision(traceInvocationId, { - step: target.executionKey, - target: modelStr, - decision: "skipped_before_dispatch", - reason: "availability", - }); - if (i > 0) fallbackCount++; - return stopProtectedPriorityTarget(`Model ${modelStr} is unavailable`); - } - } - - // Credential gate: skip targets with known-bad credentials (fail-fast) - const connectionId = target.connectionId as string | undefined; - if (connectionId) { - const gateResult = checkCredentialGate(connectionId, provider, modelStr); - if (gateResult.allowed === false) { - logCredentialSkip(log, modelStr, gateResult.reason || "Credential gate blocked"); - recordComboDecision(traceInvocationId, { - step: target.executionKey, - target: modelStr, - decision: "skipped_before_dispatch", - reason: "credential_gate", - }); - if (i > 0) fallbackCount++; - return stopProtectedPriorityTarget(`Credential gate blocked ${modelStr}`); - } - - // Concurrency gate: fail-fast skip when connection is at max_concurrent capacity (e.g. Featherless 1/1) - const maxConcurrentCap = await lookupPositiveCap(connectionId); - if ( - maxConcurrentCap && - isAccountSemaphoreFull(provider, connectionId, maxConcurrentCap) - ) { - log.info( - "COMBO", - `Skipping ${modelStr} — connection ${connectionId} is at max concurrency cap (${maxConcurrentCap})` - ); - recordComboDecision(traceInvocationId, { - step: target.executionKey, - target: modelStr, - decision: "skipped_before_dispatch", - reason: "concurrency_cap", - }); - if (i > 0) fallbackCount++; - return stopProtectedPriorityTarget(`Connection capacity reached for ${modelStr}`); - } - } - - // #9654 Wave 2: per-target lane-aware admission probe. With virtual - // lanes on, a tenant whose lane queue is full should skip extra - // fan-out targets instead of piling more queued work onto the lane. - // Strictly non-blocking (maxWaitMs 0) and a no-op when lanes are off — - // see createPerTargetAdmissionHook for the full contract. - if ( - perTargetAdmission && - !(await perTargetAdmission({ modelStr, executionKey: target.executionKey, body })) - ) { - log.info("COMBO", `Skipping ${modelStr} — admission lane full (#9654)`); - recordComboDecision(traceInvocationId, { - step: target.executionKey, - target: modelStr, - decision: "skipped_before_dispatch", - reason: "admission_lane", - }); - if (i > 0) fallbackCount++; - return null; - } - - // Retry loop for transient errors - for (let retry = 0; retry <= maxRetries; retry++) { - // Fix #1681: Bail out immediately if the client has disconnected - if (signal?.aborted) { - log.info("COMBO", `Client disconnected — aborting combo loop before model ${modelStr}`); - return { ok: false, response: errorResponse(499, "Client disconnected") }; - } - globalAttempts++; - if (globalAttempts > maxGlobalAttempts) { - log.warn( - "COMBO", - `Maximum combo attempts (${maxGlobalAttempts}) exceeded across all targets and fallbacks. Terminating loop to prevent runaway background requests.` - ); - // Actionable failure instead of an opaque 503 when every candidate - // failed the same recoverable way. If the dominant cause was reasoning - // models exhausting a too-small max_tokens budget (no content output), - // retrying other models can't help — tell the caller to raise max_tokens. - // Silent-stop fix: bump the consecutive-failure counter for this session-combo pair - // so the pin gets cleared on the 3rd attempt (recovery.next_step tells the client). - const reasoningExhausted = /reasoning consumed \d+\/\d+ tokens/.test(lastError || ""); - const failureReason = reasoningExhausted - ? "reasoning_budget_exhausted" - : "max_attempts_exceeded"; - recordComboFailure(effectiveSessionId, combo.name); - return { - ok: false, - response: errorResponseWithComboDiagnostics( - 503, - reasoningExhausted - ? "All combo candidates exhausted their token budget on reasoning without producing content. Increase max_tokens — reasoning models need a larger budget to emit content." - : "Maximum combo retry limit reached", - buildComboDiag(failureReason) - ), - }; - } - // Predictive TTFT Circuit Breaker (skip slow models) - if ( - zeroLatencyOptimizationsEnabled && - config.predictiveTtftMs && - config.predictiveTtftMs > 0 && - retry === 0 - ) { - const cMetrics = getComboMetrics(combo.name); - if (cMetrics) { - const targetKey = orderedTargets[i].executionKey || modelStr; - const m = cMetrics.byTarget[targetKey] || cMetrics.byModel[modelStr]; - if (shouldSkipForPredictedTtft(m, config.predictiveTtftMs)) { - log.warn( - "COMBO", - `Predictive TTFT Circuit Breaker: skipping ${modelStr} (avg ${m.avgLatencyMs}ms > max ${config.predictiveTtftMs}ms)` - ); - recordComboDecision(traceInvocationId, { - step: target.executionKey, - target: modelStr, - decision: "skipped_before_dispatch", - reason: "predictive_ttft", - }); - return stopProtectedPriorityTarget(`Predictive latency check rejected ${modelStr}`); - } - } - } - - if (retry > 0) { - log.info( - "COMBO", - `Retrying ${modelStr} in ${retryDelayMs}ms (attempt ${retry + 1}/${maxRetries + 1})` - ); - await new Promise((resolve) => { - const timer = setTimeout(resolve, retryDelayMs); - signal?.addEventListener( - "abort", - () => { - clearTimeout(timer); - resolve(undefined); - }, - { once: true } - ); - }); - if (signal?.aborted) { - log.info("COMBO", `Client disconnected during retry delay — aborting`); - return { ok: false, response: errorResponse(499, "Client disconnected") }; - } - - // Retry re-check: a sibling attempt (or attempt 1) may have persisted - // a quota cooldown while this attempt was sleeping out its retry delay - // ("Trying model 1/7: zai/glm-5.3 (retry 1)" after "already marked - // unavailable until …"). Reads fresh, not cached: see readConnectionForCooldownGate. - const persistedRetrySkip = await resolvePersistedConnectionCooldownSkipReason( - target, - (id) => readConnectionForCooldownGate(id, true), - allowRateLimitedConnection - ); - if (persistedRetrySkip) { - log.info("COMBO", persistedRetrySkip); - if (i > 0) fallbackCount++; - return null; - } - } - - log.info( - "COMBO", - `Trying model ${i + 1}/${orderedTargets.length}: ${modelStr}${retry > 0 ? ` (retry ${retry})` : ""}` - ); - emit("combo.target.attempt", { - comboName: combo.name, - targetIndex: i, - provider, - model: modelStr, - timestamp: Date.now(), - strategy, - }); - // QA P0 diagnostics: capture the attempt order (provider/model ids only). - comboAttemptOrder.push({ provider: provider ?? "unknown", model: modelStr }); - - // Copy-on-write, not a deep clone (#7847 — 9.53 MiB at 3 targets). Writes here are - // top-level scalars. Invariant: tests/unit/combo-attempt-body-isolation-7847.test.ts. - let attemptBody = { ...(body as Record) } as typeof body; - - // Proactive Context Compression for fallbacks (Zero-Latency optimization) - if ( - zeroLatencyOptimizationsEnabled && - i > 0 && - config.fallbackCompressionMode && - config.fallbackCompressionMode !== "off" - ) { - const { estimateTokens } = await import("./contextManager.ts"); - // #7847: object, not JSON.stringify — the string branch mis-counts inline images. - const estimatedTokens = estimateTokens(attemptBody); - if (estimatedTokens > (config.fallbackCompressionThreshold ?? 1000)) { - const { applyCompression } = await import("./compression/strategySelector.ts"); - const compressionResult = applyCompression( - attemptBody, - config.fallbackCompressionMode as CompressionMode, - // Opt into the TV1 bail-out so a throwing fallback engine is SKIPPED rather than - // propagating out of executeTarget and being swallowed as a "Speculative task - // error" (which silently drops this combo target). minGainPercent:0 keeps the - // advance behavior identical to the default path — this only adds skip-on-throw. - { model: modelStr, bailout: { enabled: true, minGainPercent: 0 } } - ); - if (compressionResult.compressed) { - log.info( - "COMBO", - `Proactive fallback compression applied (${config.fallbackCompressionMode}): ${estimatedTokens} -> ${compressionResult.stats?.compressedTokens} tokens` - ); - attemptBody = compressionResult.body; - } - } - } - - // Universal handoff: inject existing handoff if model changed. i === 0 - // only: a fallback target (i > 0) serves the SAME client request the - // failed primary target would have served, with the original messages - // already intact -- there's nothing to hand off, since the client never - // saw the earlier target fail. Injecting a handoff note there replaces - // real context with a context-free note, which weaker fallback models - // have been observed treating as license to fabricate content instead - // of just answering the actual request (#12227 follow-up). - if ( - i === 0 && - universalHandoffConfig.enabled && - relayOptions?.sessionId && - !(body as Record)?.[SKIP_UNIVERSAL_HANDOFF_FLAG] - ) { - const lastModel = getLastSessionModel(relayOptions.sessionId, combo.name); - if (lastModel && lastModel !== modelStr) { - const existingHandoff = getHandoff(relayOptions.sessionId, combo.name); - attemptBody = injectUniversalHandoffBody( - attemptBody, // Use the cloned body to maintain isolation - lastModel, - modelStr, - `Model routing: ${lastModel} → ${modelStr}`, - existingHandoff, - universalHandoffConfig.relayMode - ); - } - } - - // Issue #3587: Reasoning models can spend the whole output budget on - // reasoning. Only add headroom when the complete buffer fits inside the - // model's known output cap; otherwise preserve the client's explicit limit. - { - const bodyRecord = attemptBody as Record; - const currentMaxTokens = toPositiveInteger(bodyRecord.max_tokens); - const bufferedMaxTokens = resolveReasoningBufferedMaxTokens( - modelStr, - bodyRecord.max_tokens, - { enabled: reasoningTokenBufferEnabled } - ); - if (currentMaxTokens !== null && bufferedMaxTokens !== null) { - bodyRecord.max_tokens = bufferedMaxTokens; - if (bufferedMaxTokens !== currentMaxTokens) { - log.info( - "COMBO", - `Reasoning model ${modelStr}: adjusted max_tokens ${currentMaxTokens} -> ${bufferedMaxTokens}` - ); - } - } - } - // #5501: server-side template expansion for the combo system_message — - // resolved per-target, scoped to combo-injected content only (never - // client-owned system messages). Gate: a non-empty combo system_message. - attemptBody = expandComboSystemPromptIfPresent(attemptBody, combo, { - modelId: modelStr, - providerId: provider !== "unknown" ? provider : "", - account: - typeof target.label === "string" && target.label.trim().length > 0 - ? target.label.trim() - : "", - fingerprint: resolveTargetFingerprint(target) ?? "", - }); - // #10681: record dispatch once per target (retries keep the first decision). - if (!dispatchedTargets.has(target.executionKey)) { - dispatchedTargets.add(target.executionKey); - recordComboDecision(traceInvocationId, { - step: target.executionKey, - target: modelStr, - decision: "dispatched", - }); - } - const result = await handleSingleModelWithTimeout(attemptBody, modelStr, { - ...targetForAttempt, - effectiveComboStrategy: strategy, - failoverBeforeRetry: config.failoverBeforeRetry, - }); - - // Success — validate response quality before returning - if (result.ok) { - const selectedConnectionId = - result.headers?.get("X-OmniRoute-Selected-Connection-Id") || - result.headers?.get("x-omniroute-selected-connection-id") || - undefined; - const effectiveConnectionId = selectedConnectionId || target.connectionId || ""; - - // Clone BEFORE quality check — validateResponseQuality reads the body - // via getReader() which locks the stream. The clone's body is consumed - // by the quality check; the original stays unlocked for piping. - let qualityClone: Response; - try { - qualityClone = result.clone(); - } catch { - qualityClone = result; - } - const quality = await validateResponseQuality( - qualityClone, - clientRequestedStream, - log, - config.responseValidation - ); - releaseQualityClone(qualityClone, result, quality); - if (!quality.valid) { - releaseRejectedQualityResponse(qualityClone, result); - log.warn( - "COMBO", - `Model ${modelStr} returned 200 but failed quality check: ${quality.reason}` - ); - // #6692: a quality-rejected 200 never marks the connection row - // unhealthy, so the sticky pin's lazy headroom recheck would never - // catch it either — release it here, on the failing response. - releaseStickyPinOnFailure(_sticky.messageHash, effectiveConnectionId); - recordComboRequest(combo.name, modelStr, { - success: false, - latencyMs: Date.now() - startTime, - fallbackCount, - strategy, - target: toRecordedTarget(target), - }); - recordedAttempts++; - // Fix #1707: Set terminal state so the fallback doesn't emit - // misleading ALL_ACCOUNTS_INACTIVE when the real issue is quality. - lastError = `Upstream response failed quality validation: ${quality.reason}`; - lastStatus = 502; - // #10314: record quality failures as a FIRST-CLASS per-target outcome - // so a quality reason is never silently dropped from the aggregated - // terminal message when a later sibling overwrites lastError. - comboErrors.push({ - model: modelStr, - status: 502, - error: quality.reason || "upstream response failed quality validation", - kind: "quality", - }); - if (i > 0) fallbackCount++; - if (provider && rawModel) { - const mlSettings = resolveModelLockoutSettings(settings); - if (mlSettings.enabled && mlSettings.errorCodes.includes(502)) { - recordModelLockoutFailure( - provider, - target.connectionId || "", - rawModel, - "quality_failure", - 502, - mlSettings.baseCooldownMs, - profile, - { - exactCooldownMs: mlSettings.useExponentialBackoff - ? 0 - : mlSettings.baseCooldownMs, - maxCooldownMs: mlSettings.maxCooldownMs, - } - ); - } - } - emit("combo.target.failed", { - comboName: combo.name, - targetIndex: i, - provider, - model: modelStr, - error: `Quality: ${quality.reason}`, - latencyMs: Date.now() - startTime, - }); - observeFailure(false, target.executionKey); - return protectedPriorityTarget - ? { - ok: false, - response: errorResponse(502, "Upstream response failed quality validation"), - } - : null; - } - - if (clientManagedResponsesContext && effectiveConnectionId) { - pinNativeCodexTurn({ - body, - comboName: combo.name, - target, - connectionId: effectiveConnectionId, - }); - } - - // Success decay: a healthy response walks the model's lockout failure - // count back down (and eventually clears an expired lockout entirely). - if (provider && rawModel) { - const dcResult = decayModelFailureCount(provider, effectiveConnectionId, rawModel); - if (dcResult.cleared) { - log.info("COMBO", `Model ${modelStr} fully recovered — lockout cleared`); - } else if (dcResult.newFailureCount > 0) { - log.debug( - "COMBO", - `Model ${modelStr} decayed to failureCount=${dcResult.newFailureCount}` - ); - } - } - - const latencyMs = Date.now() - startTime; - emit("combo.target.succeeded", { - comboName: combo.name, - targetIndex: i, - provider, - model: modelStr, - latencyMs, - }); - log.info( - "COMBO", - `Model ${modelStr} succeeded (${latencyMs}ms, ${fallbackCount} fallbacks)` - ); - recordComboRequest(combo.name, modelStr, { - success: true, - latencyMs, - fallbackCount, - strategy, - target: toRecordedTarget(target), - }); - recordedAttempts++; - - // Reset cooldown on success - if (provider && provider !== "unknown") { - recordProviderSuccess(provider, effectiveConnectionId || undefined); - } - if (strategy === "weighted" && stickyWeightedLimit > 1) { - const stickySuccessKey = getWeightedStepKeyForTarget(target); - if (stickySuccessKey) { - recordStickyWeightedSuccess(combo.name, stickySuccessKey, stickyWeightedLimit); - } - } - // Webhook fan-out: best-effort, never blocks the response stream. - notifyWebhookEvent("request.completed", { - combo: combo.name, - provider, - model: modelStr, - account: - typeof target.label === "string" && target.label.trim().length > 0 - ? target.label.trim() - : "", - accountId: effectiveConnectionId ?? "", - latencyMs, - fallbackCount, - }); - - // Silent-stop fix: reset the consecutive-failure counter for this session-combo pair - // on every successful dispatch so a transient recovery doesn't get "credited" against - // the threshold the user already paid through to clear the stale pin. - if (effectiveSessionId) { - clearComboFailureTracking(effectiveSessionId, combo.name); - } - // Context cache pinning: record model usage for session-based pinning - // (independent of universal handoff — always fires when context_cache_protection is on) - // #3825: write under the SAME effectiveSessionId used by the read site so a - // sessionless conversation re-pins to this model on its next turn. - if ( - combo.context_cache_protection && - effectiveSessionId && - !(body as Record)?.[SKIP_UNIVERSAL_HANDOFF_FLAG] - ) { - recordSessionModelUsage( - effectiveSessionId, - combo.name, - modelStr, - provider, - target.connectionId ?? undefined - ); - } - - // Universal handoff: record model usage for session - if ( - universalHandoffConfig.enabled && - relayOptions?.sessionId && - !(body as Record)?.[SKIP_UNIVERSAL_HANDOFF_FLAG] - ) { - const prevModel = getLastSessionModel(relayOptions.sessionId, combo.name); - recordSessionModelUsage( - relayOptions.sessionId, - combo.name, - modelStr, - provider, - target.connectionId ?? undefined - ); - // i === 0 only: a same-request fallback target (i > 0) never - // needs a summary generated for it -- see the injection-site - // comment above. recordSessionModelUsage above stays - // unconditional regardless of i: it must reflect whichever - // model actually served THIS response, since the next - // request's i === 0 comparison depends on that being - // accurate even when this response came from a fallback. - if (i === 0 && prevModel && prevModel !== modelStr) { - const handoffSourceMessages = - Array.isArray(body?.messages) && body.messages.length > 0 - ? body.messages - : Array.isArray(body?.input) - ? body.input - : []; - - maybeGenerateUniversalHandoff({ - sessionId: relayOptions.sessionId, - comboName: combo.name, - messages: handoffSourceMessages as MessageLike[], - prevModel, - currModel: modelStr, - universalConfig: universalHandoffConfig, - handleSingleModel: handleSingleModelWithTimeout, - }); - } - - recordSessionModelUsage( - relayOptions.sessionId, - combo.name, - modelStr, - provider, - target.connectionId ?? undefined - ); - } - // Context-relay intentionally splits responsibilities: - // combo.ts decides whether a successful turn should generate a handoff, - // while chat.ts injects the handoff after the real connectionId is resolved. - if ( - strategy === "context-relay" && - relayOptions?.sessionId && - relayConfig && - relayConfig.handoffProviders.includes(provider) && - provider === "codex" - ) { - const connectionId = getSessionConnection(relayOptions.sessionId); - if (connectionId) { - const quotaInfo = await fetchCodexQuota(connectionId).catch(() => null); - if (quotaInfo) { - const resetCandidates = [ - quotaInfo.windows?.session?.resetAt, - quotaInfo.windows?.weekly?.resetAt, - quotaInfo.resetAt, - ] - .filter( - (value): value is string => typeof value === "string" && value.length > 0 - ) - .sort((a, b) => a.localeCompare(b)); - const handoffSourceMessages = - Array.isArray(body?.messages) && body.messages.length > 0 - ? body.messages - : Array.isArray(body?.input) - ? body.input - : []; - - maybeGenerateHandoff({ - sessionId: relayOptions.sessionId, - comboName: combo.name, - connectionId, - percentUsed: quotaInfo.percentUsed, - messages: handoffSourceMessages, - model: modelStr, - expiresAt: resetCandidates[0] || null, - config: relayConfig, - handleSingleModel: handleSingleModelWithTimeout, - }); - } - } - } - if (_sticky.messageHash && target.connectionId) - recordStickyBinding(_sticky.messageHash, target.connectionId); // LKGP (#919): - if (provider) { - const connId = effectiveConnectionId || undefined; - void (async () => { - try { - const { setLKGP } = await import("@/lib/db/settings"); - await Promise.all([ - setLKGP(combo.name, target.executionKey, provider, connId), - setLKGP(combo.name, combo.id || combo.name, provider, connId), - ]); - } catch (err) { - log.warn( - "COMBO", - "Failed to record Last Known Good Provider. This is non-fatal.", - { - err, - } - ); - } - })(); - } - - return { ok: true, response: result }; - } - - // Extract error info from response - let errorText = result.statusText || ""; - let errorBody: ComboErrorBody = null; - let retryAfter: ComboRetryAfter | null = null; - try { - const cloned = result.clone(); - try { - const text = await cloned.text(); - if (text) { - errorText = text.substring(0, 500); - errorBody = JSON.parse(text); - const parsedError = errorBody?.error; - errorText = - (typeof parsedError === "object" && parsedError?.message) || - (typeof parsedError === "string" ? parsedError : null) || - errorBody?.message || - errorText; - // Live incident (log id 1784457764961-73 follow-up): the pre-dispatch - // "all credentials cooling down" rejection (buildModelCooldownBody / - // handleNoCredentials in src/sse/handlers/chatHelpers.ts) nests its - // retry hint as error.retry_after (ISO string) / error.reset_seconds - // (seconds), not the top-level `retryAfter` every other 429 shape - // uses. Without this fallback, lastStatus gets recorded (fixed above) - // but earliestRetryAfter stays null, so the final check falls through - // to the generic "all combo models unavailable" error instead of ever - // reaching the cooldown-wait decision — same class of bug, different - // response shape. - const nestedRetryAfter = - typeof parsedError === "object" ? (parsedError?.retry_after ?? null) : null; - const nestedResetSeconds = - typeof parsedError === "object" ? (parsedError?.reset_seconds ?? null) : null; - retryAfter = - errorBody?.retryAfter || - nestedRetryAfter || - (typeof nestedResetSeconds === "number" && nestedResetSeconds > 0 - ? new Date(Date.now() + nestedResetSeconds * 1000).toISOString() - : null); - } - } catch { - /* Clone parse failed */ - } - } catch { - /* Clone failed */ - } - - // Track earliest retryAfter - if ( - retryAfter && - (!earliestRetryAfter || new Date(retryAfter) < new Date(earliestRetryAfter)) - ) { - earliestRetryAfter = retryAfter; - } - - // Normalize error text - if (typeof errorText !== "string") { - try { - errorText = JSON.stringify(errorText); - } catch { - errorText = String(errorText); - } - } - - const isStreamReadinessFailure = - (result.status === 502 || result.status === 504) && - isStreamReadinessFailureErrorBody(errorBody); - // An early EOF is an upstream failure, not a readiness probe — the breaker must - // see it even though the transient-retry path below treats both codes alike. - const isStreamEarlyEof = - (result.status === 502 || result.status === 504) && - isStreamEarlyEofErrorBody(errorBody); - - // FIX 5: a local per-API-key token-limit 429 must not cool shared accounts. - const isTokenLimitBreach = - result.status === 429 && isTokenLimitBreachErrorBody(errorBody); - const isLocalQueueCapacity = isLocalQueueCapacityErrorBody(errorBody); - - // Fix #1681: Status 499 means client disconnected — stop combo loop immediately. - // There is no point trying fallback models when nobody is listening. - if (result.status === 499) { - log.info("COMBO", `Client disconnected (499) during ${modelStr} — stopping combo loop`); - recordComboRequest(combo.name, modelStr, { - success: false, - latencyMs: Date.now() - startTime, - fallbackCount, - strategy, - target: toRecordedTarget(target), - }); - recordedAttempts++; - // executeTarget must return the {ok,response} contract — a raw Response - // here makes the speculative loop's res.ok/res.response checks both miss, - // so the combo would wrongly fall through to the next model after a 499. - return { ok: false, response: result }; - } - if (isLocalQueueCapacity) { - log.info( - "COMBO", - `Local rate-limit queue capacity reached for ${modelStr} — returning without upstream fallback` - ); - recordComboRequest(combo.name, modelStr, { - success: false, - latencyMs: Date.now() - startTime, - fallbackCount, - strategy, - target: toRecordedTarget(target), - }); - recordedAttempts++; - if (i > 0) fallbackCount++; - return { ok: false, response: result }; - } - - // Combo fallback is target-level orchestration: a non-ok target response is - // treated as local to that target and the combo continues to the next target. - // Error classification is retained only for retry/cooldown pacing; it must - // not decide whether fallback happens, including for generic 400 responses. - const rawError = errorBody?.error; - const structuredError = - rawError && typeof rawError === "object" - ? { - // Upstream JSON may carry a numeric `code`/`type` (e.g. {"code":40001}). - // Coerce to string if present instead of discarding, so downstream string - // ops (.toLowerCase, .startsWith) can run safely without type crashes. - code: - (rawError as Record).code !== undefined && - (rawError as Record).code !== null - ? String((rawError as Record).code) - : undefined, - type: - (rawError as Record).type !== undefined && - (rawError as Record).type !== null - ? String((rawError as Record).type) - : undefined, - } - : undefined; - const scopedFailure = isScopedFailure(result, errorText, structuredError); - - // #8375: input-bound request-scoped failures (context_length_exceeded) are - // deterministic for the same input — retrying on other accounts of the same - // model will fail identically. Short-circuit the combo immediately with the - // original error instead of burning MAX_GLOBAL_ATTEMPTS. - // Scoped to homogeneous remainders only: a heterogeneous combo (#6637) may - // have a later target with a different, larger context window that would - // NOT reject the same input — isContextOverflow400 below exists precisely to - // let that case fall through, so only short-circuit when every remaining - // target is the same model (the "retrying will fail identically" premise - // only holds within a homogeneous same-model pool). - const remainingTargets = orderedTargets.slice(i + 1); - const remainderIsHomogeneous = remainingTargets.every( - (nextInPool) => nextInPool.modelStr === modelStr - ); - const isInputBoundFailure = - isInputBoundRequestFailure(structuredError) && remainderIsHomogeneous; - if (isInputBoundFailure) { - log.warn( - "COMBO", - `Input-bound request failure from ${modelStr} — aborting combo (same input will fail identically on every account)` - ); - recordComboRequest(combo.name, modelStr, { - success: false, - latencyMs: Date.now() - startTime, - fallbackCount, - strategy, - target: toRecordedTarget(target), - }); - recordedAttempts++; - if (i > 0) fallbackCount++; - return { ok: false, response: result }; - } - const fallbackResult = checkFallbackError( - result.status, - errorText, - 0, - protectedPriorityTarget ? rawModel : null, - provider, - result.headers, - profile, - structuredError - ); - const { cooldownMs } = fallbackResult; - // #6863: a parsed upstream quota reset (e.g. Antigravity "Resets in 92h27m28s") - // arrives in `quotaResetHintMs` — it bypasses the operator-gated - // `useUpstreamRetryHints` connection-cooldown setting. Mirror the - // single-model path (src/sse/services/auth.ts): when the retry hint was - // already honored, `cooldownMs` IS the upstream value; otherwise prefer the - // parsed quota reset — even when it is SHORTER than the fallback cooldown - // (e.g. subscription-quota 1h default vs a real "resets in 10m"). - // `selectLockoutCooldownMs` still ignores hints at/below the base cooldown, - // so absent/tiny hints keep the #1308 exponential-backoff behavior. - const lockoutHintMs = - fallbackResult.usedUpstreamRetryHint === true - ? cooldownMs - : (fallbackResult.quotaResetHintMs ?? 0); - // Only a transport header or google.rpc.RetryInfo is authoritative enough - // to bypass maxCooldownMs. Prose and generic JSON remain useful exact hints, - // but the operator cap still bounds them. - const lockoutHintVerified = retryHintBypassesMaxCooldownMs( - fallbackResult.retryHintSource - ); - const selectedConnectionId = - result.headers?.get("X-OmniRoute-Selected-Connection-Id") || - result.headers?.get("x-omniroute-selected-connection-id") || - undefined; - const targetWithConnection = selectedConnectionId - ? { ...target, connectionId: selectedConnectionId } - : target; - - // #1731 / #1731v2: classify the upstream error and update the exhaustion sets - // (shared with handleRoundRobinCombo). Returns whether the provider is fully exhausted. - const providerExhausted = applyComboTargetExhaustion(targetWithConnection, { - result, - fallbackResult, - errorText, - rawModel, - isTokenLimitBreach, - allAccountsRateLimited: false, - requestScopedFailure: scopedFailure, - sets: { exhaustedProviders, exhaustedConnections, transientRateLimitedProviders }, - log, - tag: "COMBO", - exhaustedLogLevel: "info", - structuredError, - }); - // #6692: this connection was just classified as provider/connection-level - // exhausted — if it's the currently sticky-bound one, release the pin now - // rather than waiting for the next turn's lazy headroom/status recheck. - releaseStickyPinOnFailure(_sticky.messageHash, targetWithConnection.connectionId); - if ( - providerExhausted || - exhaustedConnections.has(`${provider}:${targetWithConnection.connectionId}`) || - (provider && exhaustedProviders.has(provider)) - ) { - clearStaleLKGP(combo.name, target.executionKey, combo.id, log, "COMBO"); - } - - // #2101: Prevent infinite fallback loops with 400 Bad Request errors that are genuinely - // body-specific (malformed JSON, bad format, missing required fields). - // These should NOT stop the combo: - // - Context overflow: different models have different context windows - // - Max_tokens / param errors: different models have different output limits - // - Model access denied / "not supported": different providers serve different - // model sets — keep the model in the combo and try the next target (#5249). - // Wrapper words like "invalid" / "bad request" still stop only when the text is - // NOT model-scoped (e.g. "invalid message format"). - if ( - result.status === 400 && - fallbackResult.shouldFallback && - !isContextOverflow400(errorText) && - !isParamValidation400(errorText) && - !isModelScoped400(errorText) && - (errorText.toLowerCase().includes("context") || - errorText.toLowerCase().includes("prompt") || - errorText.toLowerCase().includes("token") || - errorText.toLowerCase().includes("malformed") || - errorText.toLowerCase().includes("invalid") || - errorText.toLowerCase().includes("bad request")) - ) { - log.warn( - "COMBO", - `400 Bad Request with body-specific error detected on ${modelStr} — skipping fallback to other targets to prevent infinite loop` - ); - // Record the failure and break to avoid trying other targets with the same bad request - recordComboRequest(combo.name, modelStr, { - success: false, - latencyMs: Date.now() - startTime, - fallbackCount, - strategy, - target: toRecordedTarget(target), - }); - recordedAttempts++; - lastError = errorText || String(result.status); - comboErrors.push({ - model: modelStr, - status: result.status, - error: errorText || String(result.status), - kind: classifyComboOutcome(result.status, errorText), - }); - lastStatus = result.status; - if (i > 0) fallbackCount++; - log.warn("COMBO", `Model ${modelStr} failed with body-specific error, stopping combo`); - clearStaleLKGP(combo.name, target.executionKey, combo.id, log, "COMBO"); - // #4279: surface the 400 via the {ok,response} contract so the OUTER - // target loop resolves the combo and stops. A bare `break` here only - // exits the inner retry loop; executeTarget then returns null, which - // the outer loop treats as "this target produced nothing" and advances - // to the next model — so the guard failed to stop fallback and a combo - // of N body-rejecting targets tried all N. Mirrors the 499 path above. - return { ok: false, response: result }; - } - - // A model-scoped 400 ("The requested model is not supported" / "not - // available for integrator") is permanent for THIS connection — the - // account/integration will not gain support for the model mid-session. - // Combo still advances to the next target immediately (unchanged, - // preserves #5249's cross-provider fallback), but without a lockout - // here the SAME dead model gets retried on every future, separate - // request forever (observed: every auto-combo request wasted several - // upstream 400s on the same GitHub models, all day). isModelLocked() - // is checked before dispatch (see the pre-check above this loop), so - // this lockout is honored on the next request. - if (result.status === 400 && isModelScoped400(errorText) && provider && rawModel) { - lockModelIfPerModelQuota( - provider, - targetWithConnection.connectionId || "", - rawModel, - "model_capacity", - 60 * 60 * 1000 // 1h - ); - } - - // Trigger shared provider circuit breaker for 5xx errors and connection failures. If the - // next target is on the same provider, don't mark it failed (a different model may still - // succeed) — #8376: EXCEPT a proxy-unreachable failure, which poisons every model alike. - // G-02: when fallbackResult.skipProviderBreaker is set (embedded service supervisor outage - // signalled via X-Omni-Fallback-Hint: connection_cooldown) apply cooldown only — never trip. - const nextTarget = orderedTargets[i + 1]; - const sameProviderNext = - typeof nextTarget?.provider === "string" && nextTarget.provider === provider; - if ( - shouldRecordProviderBreakerFailure({ - isStreamReadinessFailure, - isStreamEarlyEof, - status: result.status, - sameProviderNext, - skipProviderBreaker: fallbackResult.skipProviderBreaker, - requestScopedFailure: scopedFailure, - error: errorText, - isProxyUnreachable: structuredError?.code === "proxy_unreachable", - }) - ) { - const isQueueTimeout = - errorText.includes("RATE_LIMIT_QUEUE_TIMEOUT") || - errorText.includes("RATE_LIMIT_QUEUE_WEDGED"); - recordProviderFailure(provider, log, targetWithConnection.connectionId, profile, { - isQueueTimeout, - isNetworkError: structuredError?.code === "proxy_unreachable", - }); - } - - const quotaExhausted = await isQuotaExhaustionResponse( - result, - provider, - rawModel, - profile - ); - recordQuotaExhaustionClassification(result, quotaExhausted); - observeFailure(quotaExhausted, target.executionKey); - - // Check if this is a transient error worth retrying on same model. - // A token-limit 429 is terminal for the client — never retry it. - const isTransient = - !isStreamReadinessFailure && - !isTokenLimitBreach && - !scopedFailure && - [408, 429, 500, 502, 503, 504].includes(result.status); - // failoverBeforeRetry means what it says: prefer the next sibling - // target over hammering this one again. Without this check, a - // transient error always re-hit the SAME model up to maxRetries - // times regardless of the setting — config.failoverBeforeRetry was - // threaded through to skipUpstreamRetry (a different, lower-level - // retry mechanism) but never consulted here, so a rate-limited - // model got maxRetries+1 back-to-back attempts on itself before - // this loop's own fallback-to-next-target ever ran (#2417). Only - // skip the same-model retry when `nextTarget` (computed above) - // actually gives us somewhere to fail over to — with no sibling - // left, skipping just burns the last attempt for nothing. - // - // #10217 round-4 fix: this guard reads `failoverBeforeRetryExplicit` - // (opt-in only), NOT `config.failoverBeforeRetry` — that field - // defaults to true for the separate skipUpstreamRetry mechanism - // (see DEFAULT_COMBO_CONFIG comment in comboConfig.ts) and reading - // it here would silently skip the same-model retry for every combo, - // not just ones that explicitly opted in. - if ( - retry < maxRetries && - isTransient && - !providerExhausted && - (!config.failoverBeforeRetryExplicit || !nextTarget) - ) { - if ( - !protectedPriorityTarget && - provider && - rawModel && - isModelLocked(provider, targetWithConnection.connectionId || "", rawModel) - ) { - log.info("COMBO", `Skipping retry for ${modelStr} — model lockout active`); - // Live incident (log id 1784457764961-73): earliestRetryAfter is already - // captured above from THIS dispatch's own response, but lastStatus was - // never recorded on this bail-out path — so once every target in the set - // hit an existing lockout, lastStatus stayed null and the final `if - // (!lastStatus)` check crystallized an immediate ALL_ACCOUNTS_INACTIVE 503 - // instead of ever reaching the `if (earliestRetryAfter)` cooldown-wait - // decision below, even though a real 429 with a short (~1min) retry-after - // was just observed. Recording it here mirrors the "done retrying" path. - lastError = errorText || String(result.status); - lastStatus = result.status; - if (i > 0) fallbackCount++; - return null; - } - // Record model lockout immediately on the first transient failure — - // once the model is cooling down, retrying it would waste an upstream - // call and extend the cooldown via exponential backoff. - let lockoutRecorded = false; - if (!protectedPriorityTarget && provider && rawModel && retry === 0 && !scopedFailure) { - const mlSettings = resolveModelLockoutSettings(settings); - if (mlSettings.enabled && mlSettings.errorCodes.includes(result.status)) { - recordModelLockoutFailure( - provider, - targetWithConnection.connectionId || "", - rawModel, - classifyLockoutReason(result.status), - result.status, - mlSettings.baseCooldownMs, - profile, - { - // #1308/#6863: honor a long upstream reset (e.g. "Resets in 160h") over - // the short base cooldown / exponential backoff when present. #7940's - // maxCooldownMs cap only applies to synthetic values — a verified - // upstream reset (lockoutHintVerified) bypasses it. - exactCooldownMs: selectLockoutCooldownMs(lockoutHintMs, mlSettings), - maxCooldownMs: mlSettings.maxCooldownMs, - // Preserve authoritative structured/header resets; clamp body prose. - exactCooldownIsUpstreamReset: lockoutHintVerified, - } - ); - lockoutRecorded = true; - } - } - if (lockoutRecorded) { - log.info("COMBO", `Skipping retry for ${modelStr} — model lockout active`); - // Same fix as the already-locked branch above — this is the - // first-failure lockout path, so lastStatus needs recording here too. - lastError = errorText || String(result.status); - lastStatus = result.status; - if (i > 0) fallbackCount++; - return null; - } - continue; // Retry same model (transient error, no lockout recorded) - } - - // Done retrying this model - const protectedTargetTrust = targetFailureTrust.get(target.executionKey); - if ( - protectedPriorityTarget && - (!protectedTargetTrust?.observedFailure || - !protectedTargetTrust.allObservedFailuresQuota) - ) { - recordComboRequest(combo.name, modelStr, { - success: false, - latencyMs: Date.now() - startTime, - fallbackCount, - strategy, - target: toRecordedTarget(target), - }); - recordedAttempts++; - return { ok: false, response: result }; - } - recordComboRequest(combo.name, modelStr, { - success: false, - latencyMs: Date.now() - startTime, - fallbackCount, - strategy, - target: toRecordedTarget(target), - }); - // LKGP (#919) mirror of the success-path set below: a just-failed target - // must not keep re-pinning itself as the "last known good" choice for the - // *next* separate request. Circuit breaker / model lockout deliberately - // don't react to request-scoped failure classes (see scopedFailure below), - // so nothing else clears this stale pin. - clearStaleLKGP(combo.name, target.executionKey, combo.id, log, "COMBO"); - recordedAttempts++; - lastError = errorText || String(result.status); - comboErrors.push({ - model: modelStr, - status: result.status, - error: errorText || String(result.status), - kind: classifyComboOutcome(result.status, errorText), - }); - lastStatus = result.status; - if (i > 0) fallbackCount++; - // Wire combo failures into the resilience dashboard (model-level lockout) - // alongside the provider-level cooldown below — they govern different scopes. - if (provider && rawModel && !scopedFailure) { - const mlSettings = resolveModelLockoutSettings(settings); - if (mlSettings.enabled && mlSettings.errorCodes.includes(result.status)) { - recordModelLockoutFailure( - provider, - targetWithConnection.connectionId || "", - rawModel, - classifyLockoutReason(result.status), - result.status, - mlSettings.baseCooldownMs, - profile, - { - // #1308/#6863: honor a long upstream reset over base/exponential cooldown. - // #7940's maxCooldownMs cap only applies to synthetic values — a verified - // upstream reset (lockoutHintVerified) bypasses it. - exactCooldownMs: selectLockoutCooldownMs(lockoutHintMs, mlSettings), - maxCooldownMs: mlSettings.maxCooldownMs, - // Preserve authoritative structured/header resets; clamp body prose. - exactCooldownIsUpstreamReset: lockoutHintVerified, - } - ); - } - } - log.warn("COMBO", `Model ${modelStr} failed, trying next`, { - status: result.status, - errorBody: redactConnectionLabel(errorText), - }); - - // #5976: per-model-quota providers (Gemini, GitHub, etc.) multiplex models - // behind one connection. A model-level 500 or 429 (RPM) must NOT cool down - // the entire provider — sibling models may still succeed. Skip cooldown - // recording for these providers on 500/429 errors so the next target can try. - if ( - resilienceSettings.providerCooldown.enabled && - provider && - provider !== "unknown" && - !scopedFailure && - !( - (result.status === 500 || result.status === 429) && - hasPerModelQuota(provider, rawModel) - ) - ) { - recordProviderCooldown( - provider, - targetWithConnection.connectionId ?? undefined, - resilienceSettings - ); - } - - const fallbackWaitMs = - fallbackDelayMs > 0 && cooldownMs > 0 && cooldownMs <= MAX_FALLBACK_WAIT_MS - ? Math.min(cooldownMs, fallbackDelayMs) - : 0; - if ([502, 503, 504].includes(result.status) && fallbackWaitMs > 0) { - log.debug?.("COMBO", `Waiting ${fallbackWaitMs}ms before fallback to next model`); - await new Promise((resolve) => { - const timer = setTimeout(resolve, fallbackWaitMs); - signal?.addEventListener( - "abort", - () => { - clearTimeout(timer); - resolve(undefined); - }, - { once: true } - ); - }); - if (signal?.aborted) { - log.info("COMBO", `Client disconnected during fallback wait — aborting`); - return { ok: false, response: errorResponse(499, "Client disconnected") }; - } - } - - return null; - } - return null; - }; - - for (let i = 0; i < orderedTargets.length; i++) { - if (anySuccess || comboExpired) break; - - const abortController = new AbortController(); - abortControllers.set(i, abortController); - const onClientAbort = () => abortController.abort(); - signal?.addEventListener("abort", onClientAbort); - - const task = (async () => { - try { - const res = await executeTarget(i); - if (res && !anySuccess) { - if (res.ok) { - anySuccess = true; - globalResolve!(res.response!); - for (const [idx, ac] of abortControllers.entries()) { - if (idx !== i) ac.abort(); - } - } else if (res.response) { - // Fatal error, abort combo - anySuccess = true; - globalResolve!(res.response); - } - } - } finally { - signal?.removeEventListener("abort", onClientAbort); - } - })().catch((err) => { - const logError = log.error ?? log.warn; - logError("COMBO", `Speculative task error for target ${i}`, err); - // G2 (silent-stop fix): never leave the speculative loop waiting on an - // unresolved globalPromise. If a task throws unexpectedly (outside - // executeTarget's error handling) and no other task succeeds, the post-loop - // `Promise.race([globalPromise, ...])` would hang forever. Resolve with a - // 502 so the request terminates with an actionable error. - if (!anySuccess && globalResolve) { - anySuccess = true; - globalResolve(errorResponse(502, `Combo target ${i} failed with an unexpected error`)); - } - }); - - runningTasks.add(task); - task.finally(() => runningTasks.delete(task)); - - if ( - zeroLatencyOptimizationsEnabled && - config.hedging && - !hasProtectedPriorityTarget && - i + 1 < orderedTargets.length - ) { - const hedgeDelay = resolveDelayMs(config.hedgeDelayMs, 500); - let timeoutResolve: () => void; - const timeoutPromise = new Promise((r) => { - timeoutResolve = r; - setTimeout(r, hedgeDelay); - }); - await Promise.race([task, globalPromise, timeoutPromise, loopSafetyPromise]); - } else { - await Promise.race([task, globalPromise, loopSafetyPromise]); - } - markLoopExpiredIfSafetyFired(); - - // Global combo timeout check: after each target completes, stop trying - // further targets if the total elapsed time exceeds comboTimeoutMs. - if (!anySuccess && comboTimeoutMs > 0 && Date.now() - comboStartTime >= comboTimeoutMs) { - comboExpired = true; - log.info( - "COMBO", - `Combo global timeout (${comboTimeoutMs}ms) reached after ` + - `${i + 1}/${orderedTargets.length} targets (${recordedAttempts} attempted) — stopping` - ); - } - } - - if (!anySuccess && runningTasks.size > 0) { - // G1: include loopSafetyPromise so a hung last task (per-model timeout - // disabled) cannot freeze this post-loop race forever. - await Promise.race([globalPromise, Promise.all([...runningTasks]), loopSafetyPromise]); - markLoopExpiredIfSafetyFired(); - } - - // G1: if the safety timer won the race (request would otherwise hang), give - // in-flight tasks a short drain window to land their per-model errors into - // comboErrors so the 504 carries the same "tried: a (500)" summary the - // regular comboExpired branch produces — then return the safety 504. - if (loopSafetyFired && !anySuccess) { - if (runningTasks.size > 0) { - await Promise.race([ - Promise.allSettled([...runningTasks]), - new Promise((resolve) => setTimeout(resolve, COMBO_SAFETY_DRAIN_MS)), - ]); - } - const summary = comboErrors - .slice(0, 5) - .map((e) => `${e.model} (${e.status})`) - .join(", "); - const msg = - `Combo global timeout (${loopSafetyMs}ms) after ${recordedAttempts}/${orderedTargets.length} targets` + - (comboErrors.length > 0 - ? ` | tried: ${summary}${comboErrors.length > 5 ? `... (+${comboErrors.length - 5})` : ""}` - : "") + - " without a terminal response"; - return errorResponseWithComboDiagnostics(504, msg, buildComboDiag("combo_timeout"), { - code: "COMBO_TIMEOUT", - type: "server_error", - }); - } - - // #10681: finalize the decision trace (success). - finalizeComboTrace(traceInvocationId, orderedTargets); - finishComboTrace(traceInvocationId, { status: 200 }); - if (anySuccess) { - // G1: clear the safety timer on the happy path so a successful combo does - // not leave a 10-minute timer alive per request. - if (loopSafetyTimer) { - clearTimeout(loopSafetyTimer); - loopSafetyTimer = null; - } - return await globalPromise; - } - - // #10681: finalize the decision trace (global timeout). - finalizeComboTrace(traceInvocationId, orderedTargets); - finishComboTrace(traceInvocationId, { status: 504 }); - // Global combo timeout: return aggregated error immediately, skipping set retries. - if (comboExpired) { - const summary = buildRedactedSummary(comboErrors); - const msg = - `Combo global timeout (${comboTimeoutMs}ms) after ${recordedAttempts}/${orderedTargets.length} targets` + - (comboErrors.length > 0 ? ` | tried: ${summary}` : ""); - const latencyMs = Date.now() - startTime; - if (recordedAttempts === 0) { - recordComboRequest(combo.name, null, { - success: false, - latencyMs, - fallbackCount, - strategy, - }); - } - notifyWebhookEvent("request.failed", { - combo: combo.name, - reason: "COMBO_TIMEOUT", - latencyMs, - fallbackCount, - }); - return errorResponseWithComboDiagnostics(504, msg, buildComboDiag("combo_timeout"), { - code: "COMBO_TIMEOUT", - type: "server_error", - }); - } - - // All models failed in this set try - const latencyMs = Date.now() - startTime; - if (recordedAttempts === 0) { - recordComboRequest(combo.name, null, { - success: false, - latencyMs, - fallbackCount, - strategy, - }); - } - - // Retry the entire set if more attempts remain - if (setTry < maxSetRetries) continue; - - if (!lastStatus && recordedAttempts === 0 && comboCooldownWaitEnabled) { - const circuitOpenWait = resolveCircuitOpenWaitDecision({ - skippedForCircuitOpen, - retryAfterMs: earliestCircuitOpenRetryMs, - attempt: comboCooldownAttempt, - budgetLeftMs: comboCooldownBudgetLeftMs, - settings: resilienceSettings.comboCooldownWait, - }); - if (circuitOpenWait.wait) { - log.info( - "COMBO", - `${strategy} circuit-open wait: waiting ${Math.ceil(circuitOpenWait.waitMs / 1000)}s (reason=${circuitOpenWait.reason ?? "circuit_open"}) then retrying (attempt ${comboCooldownAttempt + 1}/${resilienceSettings.comboCooldownWait.maxAttempts})` - ); - const completed = await waitForCooldownAwareRetry(circuitOpenWait.waitMs, signal); - if (!completed) { - return errorResponse(499, "Request aborted"); - } - comboCooldownAttempt += 1; - comboCooldownBudgetLeftMs = Math.max( - 0, - comboCooldownBudgetLeftMs - circuitOpenWait.waitMs - ); - return dispatchWithCooldownRetry(); - } - } - - // All set retries exhausted — return the final error - // #10681: finalize the decision trace (all targets failed or skipped). - finalizeComboTrace(traceInvocationId, orderedTargets); - finishComboTrace(traceInvocationId, { status: 503 }); - if (!lastStatus) { - if (recordedAttempts === 0) { - notifyWebhookEvent("request.failed", { - combo: combo.name, - reason: "ALL_TARGETS_SKIPPED", - latencyMs, - fallbackCount, - }); - return withQuotaExhaustionClassification( - errorResponseWithComboDiagnostics( - 503, - "Service temporarily unavailable: all targets were skipped by pre-dispatch filters", - buildComboDiag("all_targets_skipped"), - { code: "ALL_TARGETS_SKIPPED", type: "service_unavailable" } - ), - observedFailure ? allObservedFailuresQuota : null - ); - } - notifyWebhookEvent("request.failed", { - combo: combo.name, - reason: "ALL_ACCOUNTS_INACTIVE", - latencyMs, - fallbackCount, - }); - recordComboFailure(effectiveSessionId, combo.name); - return errorResponseWithComboDiagnostics( - 503, - "Service temporarily unavailable: all upstream accounts are inactive", - buildComboDiag("all_accounts_inactive"), - { code: "ALL_ACCOUNTS_INACTIVE", type: "service_unavailable" } - ); - } - - // #10501: derive the terminal HTTP status from the structured per-target - // outcomes instead of `lastStatus` (whichever target happened to fail - // LAST). A 4xx is preserved only when the request itself is genuinely - // invalid across every eligible target; a heterogeneous mix of failure - // classes (e.g. a quality failure + a sibling's 401) normalizes to a - // 5xx-class status reflecting an infra/provider problem, not a client - // error. See comboErrorAggregation.ts::resolveComboTerminalStatus. - const status = resolveComboTerminalStatus(comboErrors, lastStatus); - // #10314: build the terminal message from the structured per-target - // outcomes (each distinct class+reason listed separately) instead of - // mashing a single lastError with raw `[model (status)]` markers. Connection - // identifiers are redacted. Falls back to lastError when no target recorded - // a structured outcome. - const msg = formatComboOutcomes(comboErrors) || lastError || "All combo models unavailable"; - - // Cooldown-aware retry: instead of crystallizing a transient failure, wait - // out a SHORT cooldown and re-run the whole set loop. Guarded by the helper - // (quota_exhausted/auth/not-found excluded, ceiling, attempts, budget). - // MAX_GLOBAL_ATTEMPTS still bounds total dispatches. Available to ALL combo - // strategies when enabled — entry is driven by earliestRetryAfter + the - // real model-lockout reason, NOT by whichever target last overwrote - // `status` (a later 403 must not skip the allow-list check for an earlier - // 429's retry-after hint). SECURITY (see comboCooldownRetry.ts header): the - // allow-list is the PRIMARY barrier and `maxWaitMs` only the SECOND one. - // Hardcoding reason:"rate_limit" would drop the primary barrier and leave - // only the ceiling — which does NOT cover a quota_exhausted lock carrying a - // SHORT upstream retry-after. Model lockouts are recorded for all strategies, - // so the real reason is always available. - if (comboCooldownWaitEnabled && earliestRetryAfter) { - const decision: ResolveComboCooldownDecisionResult = resolveComboCooldownWaitDecision({ - targets: orderedTargets, - earliestRetryAfter, - attempt: comboCooldownAttempt, - budgetLeftMs: comboCooldownBudgetLeftMs, - settings: resilienceSettings.comboCooldownWait, - // Key each lookup on the TARGET's own model: quota-share combos are - // single-model/multi-account (so this is identical to the previous - // orderedTargets[0] behavior), but heterogeneous combos carry a - // different model per target. - lookupLock: (provider, connectionId, target) => { - const rawModel = parseModel(target?.modelStr ?? "").model || ""; - if (!rawModel) return null; - return getModelLockoutInfo(provider, connectionId, rawModel); - }, - computeWaitMs: (retryAfter) => computeClosestRetryAfter(retryAfter).waitMs, - }); - - if (decision.wait) { - log.info( - "COMBO", - `${strategy} cooldown wait: ${msg} — waiting ${Math.ceil( - decision.waitMs / 1000 - )}s (reason=${decision.reason ?? "?"}) then retrying (attempt ${ - comboCooldownAttempt + 1 - }/${resilienceSettings.comboCooldownWait.maxAttempts})` - ); - const completed = await waitForCooldownAwareRetry(decision.waitMs, signal); - if (!completed) { - log.info("COMBO", `${strategy} cooldown wait aborted by client disconnect`); - return errorResponse(499, "Request aborted"); - } - comboCooldownAttempt += 1; - comboCooldownBudgetLeftMs = Math.max(0, comboCooldownBudgetLeftMs - decision.waitMs); - return dispatchWithCooldownRetry(); - } - } - - // #10681: finalize the decision trace with the aggregated terminal status. - finalizeComboTrace(traceInvocationId, orderedTargets); - finishComboTrace(traceInvocationId, { status }); - // Retry-after decoration is separate from the wait decision above: only - // rate-limit-class final statuses may carry a `(reset after ...)` suffix - // (see unavailableRetryGate.ts — do not stitch a peer target's window onto - // a config-class status like 403/422). - if (earliestRetryAfter && isRetryAfterEligibleStatus(status)) { - const retryHuman = formatRetryAfter(toRetryAfterDisplayValue(earliestRetryAfter)); - log.warn("COMBO", `All models failed | ${msg} (${retryHuman})`); - return withQuotaExhaustionClassification( - unavailableResponse(status, msg, earliestRetryAfter, retryHuman), - observedFailure ? allObservedFailuresQuota : null - ); - } - - // Silent-stop fix: bump the failure counter (pin clears on 3rd consecutive) and emit - // `try-auto` recovery action via buildRecoveryHint so the OC plugin can show "→ Try - // model: auto" instead of an opaque 5xx. We pass the upstream retry-after seconds to - // the hint so the client can render a precise "wait Ns and retry" message. - log.warn("COMBO", `All models failed | ${msg}`); - const { pinClearedNow } = recordComboFailure(effectiveSessionId, combo.name); - if (pinClearedNow) { - log.info( - "COMBO", - `Auto-cleared session_model_history pin for combo "${combo.name}" after ${COMBO_FAILURE_THRESHOLD} consecutive failures to break the silent-stop loop` - ); - } - const retryAfterSeconds = undefined; - // #10966: when every observed failure was independently classified as quota/ - // balance exhaustion (isQuotaExhaustionResponse, tracked via observeFailure's - // allObservedFailuresQuota accumulator), stamp a stable `quota_exhausted` - // terminalReason instead of forwarding the raw upstream error string. The raw - // string falls through buildRecoveryHint's default branch ("retry" / "failed - // transiently"), which is actively misleading for a durable wallet/quota - // exhaustion — retrying the same combo will never refill it. - const terminalReason = - observedFailure && allObservedFailuresQuota - ? "quota_exhausted" - : (lastError ?? "all_models_failed"); - return withQuotaExhaustionClassification( - errorResponseWithComboDiagnostics( - status, - msg, - buildComboDiag(terminalReason, retryAfterSeconds) - ), - observedFailure ? allObservedFailuresQuota : null - ); - } - - // Final fallback — when the dispatch returned without crystallizing a status (rare). - // Surface the recovery hint with a generic retry recommendation so the client at least - // gets a non-opaque message instead of "Combo routing completed without an upstream response". - recordComboFailure(effectiveSessionId, combo.name); - return errorResponseWithComboDiagnostics( - 503, - "Combo routing completed without an upstream response", - buildNoUpstreamResponseDiagnostics(orderedTargets.length) - ); - } finally { - // #11804: always release the loop-safety timer. Covering every exit path by - // construction here means a future `return` added to this function cannot - // silently reintroduce the leak. - if (activeLoopSafetyTimer) { - clearTimeout(activeLoopSafetyTimer); - activeLoopSafetyTimer = null; - } - } - }; - // FASE 2.1: acquire the per-connection concurrency slot for the selected // quota-share target once, around the whole dispatch (including any // cooldown-aware re-dispatch), so concurrent requests to one subscription @@ -3006,7 +1088,7 @@ async function handleComboChatInner({ } try { - return await dispatchWithCooldownRetry(); + return await dispatchWithCooldownRetry({ state, deps, extra }); } finally { quotaShareConcurrencyRelease?.(); // #11371: release the in-flight slot quota-share ordering reserved for its @@ -3038,14 +1120,14 @@ async function handleRoundRobinCombo({ allCombos, signal, apiKeyAllowedConnections = null, - nesting = null, + nesting: _nesting = null, hiddenModelsByProvider = getHiddenModelsByProvider(), - clientManagedResponsesContext, - deferContextOverflowWhenCompressible = false, - compressionExclusions, - sourceFormat = null, - endpointPath = null, - requestHeaders = null, + clientManagedResponsesContext: _clientManagedResponsesContext, + deferContextOverflowWhenCompressible: _deferContextOverflowWhenCompressible = false, + compressionExclusions: _compressionExclusions, + sourceFormat: _sourceFormat = null, + endpointPath: _endpointPath = null, + requestHeaders: _requestHeaders = null, relayOptions, perTargetAdmission = null, }: HandleRoundRobinOptions): Promise { @@ -4079,5 +2161,8 @@ async function handleRoundRobinCombo({ } log.warn("COMBO-RR", `All models failed | ${msg}`); - return new Response(JSON.stringify({ error: { message: msg } }), { status, headers: { "Content-Type": "application/json" } }); + return new Response(JSON.stringify({ error: { message: msg } }), { + status, + headers: { "Content-Type": "application/json" }, + }); } diff --git a/open-sse/services/combo/attemptLoopTypes.ts b/open-sse/services/combo/attemptLoopTypes.ts new file mode 100644 index 0000000000..7fff71b3d5 --- /dev/null +++ b/open-sse/services/combo/attemptLoopTypes.ts @@ -0,0 +1,117 @@ +/** + * Shared types for the handleComboChat attempt-loop split (ROADMAP 3.8.52). + * + * Mutable loop state lives on AttemptLoopState. Read-only dependencies live on + * AttemptLoopDeps. Do not merge the two into ComboContext. + * + * @internal — not part of the public combo.ts barrel. + */ +import type { PerTargetAdmissionHook } from "../admission/types.ts"; +import type { ResilienceSettings } from "../../../src/lib/resilience/settings"; +import type { ContextRelayConfig, UniversalHandoffConfig } from "../contextHandoff.ts"; +import type { ComboErrorEntry } from "./comboErrorAggregation.ts"; +import type { ResetWindowConfig } from "./quotaScoring.ts"; +import type { ResponseValidationConfig } from "./responseValidation.ts"; +import type { ApplyStickinessResult } from "./sessionStickiness.ts"; +import type { + ComboLike, + ComboLogger, + ComboRetryAfter, + HandleSingleModel, + IsModelAvailable, + ResolvedComboTarget, +} from "./types.ts"; + +export type ExecuteTargetResult = { ok: boolean; response?: Response } | null; + +export type AttemptLoopState = { + orderedTargets: ResolvedComboTarget[]; + fallbackCount: number; + recordedAttempts: number; + comboErrors: ComboErrorEntry[]; + lastError: string | null; + lastStatus: number | null; + earliestRetryAfter: ComboRetryAfter | null; + comboExpired: boolean; + exhaustedProviders: Set; + exhaustedConnections: Set; + transientRateLimitedProviders: Set; + abortControllers: Map; + dispatchedTargets: Set; + targetFailureTrust: Map; + comboAttemptOrder: Array<{ provider: string; model: string }>; + skippedForCircuitOpen: boolean; + earliestCircuitOpenRetryMs: number; + /** Mutable attempt budget shared with dispatchWithCooldownRetry (Task 4). */ + globalAttempts: number; + /** Quota-trust accumulators; persist across set retries and cooldown re-dispatch. */ + observedFailure: boolean; + allObservedFailuresQuota: boolean; + observeFailure(quotaExhausted: boolean, targetExecutionKey?: string): void; +}; + +export type AttemptLoopDeps = { + strategy: string; + combo: ComboLike; + config: Record & { + zeroLatencyOptimizationsEnabled?: boolean; + responseValidation?: ResponseValidationConfig | null; + failoverBeforeRetryExplicit?: boolean; + failoverBeforeRetry?: boolean; + predictiveTtftMs?: number; + fallbackCompressionMode?: string; + fallbackCompressionThreshold?: number; + retryDelayMs?: number; + fallbackDelayMs?: number; + maxGlobalAttempts?: unknown; + hedging?: boolean; + hedgeDelayMs?: unknown; + }; + log: ComboLogger; + settings: Record | null; + resilienceSettings: ResilienceSettings; + sticky: ApplyStickinessResult; + effectiveSessionId: string | null; + preScreenMap: Map; + quotaCutoffResetWindowConfig: ResetWindowConfig; + maxRetries: number; + traceInvocationId: string; + clientRequestedStream: boolean; + handleSingleModelWithTimeout: HandleSingleModel; + isModelAvailable?: IsModelAvailable; + perTargetAdmission?: PerTargetAdmissionHook | null; + signal?: AbortSignal | null; + body: Record; + startTime: number; + releaseStickyPinOnFailure: ( + messageHash: string | null | undefined, + failedConnectionId: string | null | undefined + ) => void; + clearStaleLKGP: ( + comboName: string, + executionKey: string | undefined, + comboId: string | undefined, + log: ComboLogger, + tag: string + ) => void; + /** + * Closed-over setup values from handleComboChatInner. Optional so Task 2 + * gate tests keep compiling; attempt uses defaults when absent. + */ + clientManagedResponsesContext?: boolean; + reasoningTokenBufferEnabled?: boolean; + stickyWeightedLimit?: number; + getWeightedStepKeyForTarget?: (target: ResolvedComboTarget) => string | null; + universalHandoffConfig?: UniversalHandoffConfig; + relayOptions?: { sessionId?: string | null } | null; + relayConfig?: ContextRelayConfig | null; +}; + +export type GateDecision = + | { kind: "skip"; result: ExecuteTargetResult } + | { + kind: "proceed"; + targetForAttempt: ResolvedComboTarget; + profile: unknown; + protectedPriorityTarget: boolean; + }; diff --git a/open-sse/services/combo/comboAttemptLoop.ts b/open-sse/services/combo/comboAttemptLoop.ts new file mode 100644 index 0000000000..f1785454ae --- /dev/null +++ b/open-sse/services/combo/comboAttemptLoop.ts @@ -0,0 +1,575 @@ +/** + * Set-try + speculative dispatch loop for handleComboChatInner. + * Extracted from combo.ts dispatchWithCooldownRetry (#11804 finally lives here). + * + * @internal — not part of the public combo.ts barrel. + */ +import { formatRetryAfter, getModelLockoutInfo } from "../accountFallback.ts"; +import { + errorResponse, + errorResponseWithComboDiagnostics, + unavailableResponse, +} from "../../utils/error.ts"; +import type { ComboDiagnostics } from "../../utils/error.ts"; +import { COMBO_FAILURE_THRESHOLD, recordComboFailure } from "./failureTracker.ts"; +import { buildNoUpstreamResponseDiagnostics, buildRecoveryHint } from "./pinRecovery.ts"; +import { formatExhaustedConnectionKey } from "./comboDiagFormat.ts"; +import { recordComboRequest } from "../comboMetrics.ts"; +import { notifyWebhookEvent } from "../../../src/lib/webhookDispatcher.ts"; +import { parseModel } from "../model.ts"; +import { + formatComboOutcomes, + buildRedactedSummary, + resolveComboTerminalStatus, +} from "./comboErrorAggregation.ts"; +import { + resolveComboCooldownWaitDecision, + resolveCircuitOpenWaitDecision, + type ResolveComboCooldownDecisionResult, +} from "./comboCooldownRetry.ts"; +import { + computeClosestRetryAfter, + waitForCooldownAwareRetry, +} from "../../../src/sse/services/cooldownAwareRetry.ts"; +import { toRetryAfterDisplayValue } from "./validateQuality.ts"; +import { finalizeComboTrace, finishComboTrace } from "./decisionTrace.ts"; +import { isRetryAfterEligibleStatus } from "./unavailableRetryGate.ts"; +import { withQuotaExhaustionClassification } from "./quotaExhaustion.ts"; +import { + COMBO_LOOP_SAFETY_TIMEOUT_MS, + COMBO_SAFETY_DRAIN_MS, + resolveDelayMs, +} from "./comboPredicates.ts"; +import { evaluateExecuteTargetGates } from "./executeTargetGates.ts"; +import { executeTargetAttempt } from "./executeTargetAttempt.ts"; +import type { AttemptLoopDeps, AttemptLoopState, ExecuteTargetResult } from "./attemptLoopTypes.ts"; + +export type DispatchWithCooldownRetryExtra = { + maxSetRetries: number; + setRetryDelayMs: number; + comboTimeoutMs: number; + comboStartTime: number; + comboCooldownWaitEnabled: boolean; + comboCooldownAttempt: { current: number }; + comboCooldownBudgetLeftMs: { current: number }; + evaluateGates: typeof evaluateExecuteTargetGates; + executeAttempt: typeof executeTargetAttempt; +}; + +export async function dispatchWithCooldownRetry(opts: { + state: AttemptLoopState; + deps: AttemptLoopDeps; + extra: DispatchWithCooldownRetryExtra; +}): Promise { + const { state, deps, extra } = opts; + // #7360: persist lastStatus/earliestRetryAfter across set retries; reset + // only on a fresh dispatch (including cooldown-aware re-dispatch). + state.lastError = null; + state.earliestRetryAfter = null; + state.lastStatus = null; + state.skippedForCircuitOpen = false; + state.earliestCircuitOpenRetryMs = 0; + // #11804: the loop-safety timer is armed per setTry iteration but must be + // cleared on EVERY exit path, not just the happy one. Hoisted to function + // scope so the `finally` at the end of this function always reaches it. + let activeLoopSafetyTimer: ReturnType | null = null; + + try { + for (let setTry = 0; setTry <= extra.maxSetRetries; setTry++) { + // #1731: Per-set-iteration set of providers whose quota is fully exhausted. + // Reset each retry so providers excluded in a previous attempt get another chance. + state.exhaustedProviders = new Set(); + state.exhaustedConnections = new Set(); + state.transientRateLimitedProviders = new Set(); + state.skippedForCircuitOpen = false; + state.earliestCircuitOpenRetryMs = 0; + if (setTry > 0) { + deps.log.info( + "COMBO", + `All targets failed — retrying set (${setTry}/${extra.maxSetRetries})` + ); + await new Promise((resolve) => { + const timer = setTimeout(resolve, extra.setRetryDelayMs); + deps.signal?.addEventListener( + "abort", + () => { + clearTimeout(timer); + resolve(undefined); + }, + { once: true } + ); + }); + if (deps.signal?.aborted) { + deps.log.info("COMBO", "Client disconnected during set retry delay — aborting"); + return errorResponse(499, "Client disconnected"); + } + } + + deps.startTime = Date.now(); + state.fallbackCount = 0; + state.recordedAttempts = 0; + state.comboErrors = []; + + // QA P0: assemble a sanitized diagnostic trace from the state already in scope + // (pool size + this set-try's exhausted providers/connections + attempt order + + // a terminal-reason code). Never touches keys/tokens — provider/model ids only. + // Silent-stop fix: include a `recovery` hint (action verb + human next-step) so the + // OC plugin + non-header-aware clients can render an actionable error instead of an + // opaque 5xx. The optional `retryAfterSeconds` carries the upstream Retry-After hint. + const buildComboDiag = ( + terminalReason: string, + retryAfterSeconds?: number + ): ComboDiagnostics => ({ + poolSize: state.orderedTargets.length, + attempted: state.recordedAttempts, + excluded: [ + ...[...state.exhaustedProviders].map((p) => ({ provider: p, reason: "exhausted" })), + ...[...state.exhaustedConnections].map((c) => formatExhaustedConnectionKey(String(c))), + ], + attemptOrder: state.comboAttemptOrder, + terminalReason, + recovery: buildRecoveryHint(terminalReason, retryAfterSeconds), + }); + + let globalResolve: ((res: Response) => void) | null = null; + const globalPromise = new Promise((res) => { + globalResolve = res; + }); + + // G1 (silent-stop fix): the speculative loop's `Promise.race` waits on + // `globalPromise`, which is ONLY resolved from inside a task (success or + // fatal error). If a target hangs — e.g. the operator disabled the per-model + // timeout (`targetTimeoutMs: 0`) and the upstream never settles — the race + // never resolves and the request hangs forever with no response. This safety + // promise force-resolves after the combo budget (extra.comboTimeoutMs when set, + // otherwise a hard ceiling) so the request ALWAYS terminates with an + // actionable 504 instead of dying silently. `state.comboExpired` is flipped so the + // target loop stops launching new work; the existing state.comboExpired branch + // returns the aggregated 504. + const loopSafetyMs = + extra.comboTimeoutMs > 0 ? extra.comboTimeoutMs : COMBO_LOOP_SAFETY_TIMEOUT_MS; + let loopSafetyFired = false; + let loopSafetyTimer: ReturnType | null = null; + const loopSafetyPromise = new Promise((resolve) => { + loopSafetyTimer = setTimeout(() => { + loopSafetyFired = true; + deps.log.warn( + "COMBO", + `Combo loop safety timeout (${loopSafetyMs}ms) reached without a terminal response — force-terminating` + ); + resolve( + errorResponseWithComboDiagnostics( + 504, + `Combo global timeout (${loopSafetyMs}ms) without a terminal response`, + buildComboDiag("combo_timeout"), + { code: "COMBO_TIMEOUT", type: "server_error" } + ) + ); + }, loopSafetyMs); + loopSafetyTimer.unref?.(); + activeLoopSafetyTimer = loopSafetyTimer; + }); + const runningTasks = new Set>(); + let anySuccess = false; + // #10681: steps already recorded as dispatched (so per-target retries do not + // duplicate the decision). + state.dispatchedTargets = new Set(); + // G1: flip state.comboExpired as soon as the safety timer fires so the next loop + // iteration breaks instead of launching more targets after the budget, and + // abort every in-flight target so a hung upstream actually gets cancelled + // (not just "response stops"). + const markLoopExpiredIfSafetyFired = () => { + if (loopSafetyFired) { + state.comboExpired = true; + for (const [, ac] of state.abortControllers.entries()) ac.abort(); + } + }; + state.abortControllers = new Map(); + const zeroLatencyOptimizationsEnabled = deps.config.zeroLatencyOptimizationsEnabled === true; + const hasProtectedPriorityTarget = + deps.strategy === "priority" && + state.orderedTargets.some((target) => target.fallbackOnlyOnQuotaExhaustion === true); + + const executeTarget = async (i: number): Promise => { + const gate = await extra.evaluateGates({ index: i, state, deps }); + if (gate.kind === "skip") return gate.result; + return extra.executeAttempt({ + index: i, + state, + deps, + targetForAttempt: gate.targetForAttempt, + profile: gate.profile, + protectedPriorityTarget: gate.protectedPriorityTarget, + }); + }; + + for (let i = 0; i < state.orderedTargets.length; i++) { + if (anySuccess || state.comboExpired) break; + + const abortController = new AbortController(); + state.abortControllers.set(i, abortController); + const onClientAbort = () => abortController.abort(); + deps.signal?.addEventListener("abort", onClientAbort); + + const task = (async () => { + try { + const res = await executeTarget(i); + if (res && !anySuccess) { + if (res.ok) { + anySuccess = true; + globalResolve!(res.response!); + for (const [idx, ac] of state.abortControllers.entries()) { + if (idx !== i) ac.abort(); + } + } else if (res.response) { + // Fatal error, abort combo + anySuccess = true; + globalResolve!(res.response); + } + } + } finally { + deps.signal?.removeEventListener("abort", onClientAbort); + } + })().catch((err) => { + const logError = deps.log.error ?? deps.log.warn; + logError("COMBO", `Speculative task error for target ${i}`, err); + // G2 (silent-stop fix): never leave the speculative loop waiting on an + // unresolved globalPromise. If a task throws unexpectedly (outside + // executeTarget's error handling) and no other task succeeds, the post-loop + // `Promise.race([globalPromise, ...])` would hang forever. Resolve with a + // 502 so the request terminates with an actionable error. + if (!anySuccess && globalResolve) { + anySuccess = true; + globalResolve(errorResponse(502, `Combo target ${i} failed with an unexpected error`)); + } + }); + + runningTasks.add(task); + task.finally(() => runningTasks.delete(task)); + + if ( + zeroLatencyOptimizationsEnabled && + deps.config.hedging && + !hasProtectedPriorityTarget && + i + 1 < state.orderedTargets.length + ) { + const hedgeDelay = resolveDelayMs(deps.config.hedgeDelayMs, 500); + const timeoutPromise = new Promise((r) => { + setTimeout(r, hedgeDelay); + }); + await Promise.race([task, globalPromise, timeoutPromise, loopSafetyPromise]); + } else { + await Promise.race([task, globalPromise, loopSafetyPromise]); + } + markLoopExpiredIfSafetyFired(); + + // Global combo timeout check: after each target completes, stop trying + // further targets if the total elapsed time exceeds extra.comboTimeoutMs. + if ( + !anySuccess && + extra.comboTimeoutMs > 0 && + Date.now() - extra.comboStartTime >= extra.comboTimeoutMs + ) { + state.comboExpired = true; + deps.log.info( + "COMBO", + `Combo global timeout (${extra.comboTimeoutMs}ms) reached after ` + + `${i + 1}/${state.orderedTargets.length} targets (${state.recordedAttempts} attempted) — stopping` + ); + } + } + + if (!anySuccess && runningTasks.size > 0) { + // G1: include loopSafetyPromise so a hung last task (per-model timeout + // disabled) cannot freeze this post-loop race forever. + await Promise.race([globalPromise, Promise.all([...runningTasks]), loopSafetyPromise]); + markLoopExpiredIfSafetyFired(); + } + + // G1: if the safety timer won the race (request would otherwise hang), give + // in-flight tasks a short drain window to land their per-model errors into + // state.comboErrors so the 504 carries the same "tried: a (500)" summary the + // regular state.comboExpired branch produces — then return the safety 504. + if (loopSafetyFired && !anySuccess) { + if (runningTasks.size > 0) { + await Promise.race([ + Promise.allSettled([...runningTasks]), + new Promise((resolve) => setTimeout(resolve, COMBO_SAFETY_DRAIN_MS)), + ]); + } + const summary = state.comboErrors + .slice(0, 5) + .map((e) => `${e.model} (${e.status})`) + .join(", "); + const msg = + `Combo global timeout (${loopSafetyMs}ms) after ${state.recordedAttempts}/${state.orderedTargets.length} targets` + + (state.comboErrors.length > 0 + ? ` | tried: ${summary}${state.comboErrors.length > 5 ? `... (+${state.comboErrors.length - 5})` : ""}` + : "") + + " without a terminal response"; + return errorResponseWithComboDiagnostics(504, msg, buildComboDiag("combo_timeout"), { + code: "COMBO_TIMEOUT", + type: "server_error", + }); + } + + // #10681: finalize the decision trace (success). + finalizeComboTrace(deps.traceInvocationId, state.orderedTargets); + finishComboTrace(deps.traceInvocationId, { status: 200 }); + if (anySuccess) { + // G1: clear the safety timer on the happy path so a successful combo does + // not leave a 10-minute timer alive per request. + if (loopSafetyTimer) { + clearTimeout(loopSafetyTimer); + loopSafetyTimer = null; + } + return await globalPromise; + } + + // #10681: finalize the decision trace (global timeout). + finalizeComboTrace(deps.traceInvocationId, state.orderedTargets); + finishComboTrace(deps.traceInvocationId, { status: 504 }); + // Global combo timeout: return aggregated error immediately, skipping set retries. + if (state.comboExpired) { + const summary = buildRedactedSummary(state.comboErrors); + const msg = + `Combo global timeout (${extra.comboTimeoutMs}ms) after ${state.recordedAttempts}/${state.orderedTargets.length} targets` + + (state.comboErrors.length > 0 ? ` | tried: ${summary}` : ""); + const latencyMs = Date.now() - deps.startTime; + if (state.recordedAttempts === 0) { + recordComboRequest(deps.combo.name, null, { + success: false, + latencyMs, + fallbackCount: state.fallbackCount, + strategy: deps.strategy, + }); + } + notifyWebhookEvent("request.failed", { + combo: deps.combo.name, + reason: "COMBO_TIMEOUT", + latencyMs, + fallbackCount: state.fallbackCount, + }); + return errorResponseWithComboDiagnostics(504, msg, buildComboDiag("combo_timeout"), { + code: "COMBO_TIMEOUT", + type: "server_error", + }); + } + + // All models failed in this set try + const latencyMs = Date.now() - deps.startTime; + if (state.recordedAttempts === 0) { + recordComboRequest(deps.combo.name, null, { + success: false, + latencyMs, + fallbackCount: state.fallbackCount, + strategy: deps.strategy, + }); + } + + // Retry the entire set if more attempts remain + if (setTry < extra.maxSetRetries) continue; + + if (!state.lastStatus && state.recordedAttempts === 0 && extra.comboCooldownWaitEnabled) { + const circuitOpenWait = resolveCircuitOpenWaitDecision({ + skippedForCircuitOpen: state.skippedForCircuitOpen, + retryAfterMs: state.earliestCircuitOpenRetryMs, + attempt: extra.comboCooldownAttempt.current, + budgetLeftMs: extra.comboCooldownBudgetLeftMs.current, + settings: deps.resilienceSettings.comboCooldownWait, + }); + if (circuitOpenWait.wait) { + deps.log.info( + "COMBO", + `${deps.strategy} circuit-open wait: waiting ${Math.ceil(circuitOpenWait.waitMs / 1000)}s (reason=${circuitOpenWait.reason ?? "circuit_open"}) then retrying (attempt ${extra.comboCooldownAttempt.current + 1}/${deps.resilienceSettings.comboCooldownWait.maxAttempts})` + ); + const completed = await waitForCooldownAwareRetry(circuitOpenWait.waitMs, deps.signal); + if (!completed) { + return errorResponse(499, "Request aborted"); + } + extra.comboCooldownAttempt.current += 1; + extra.comboCooldownBudgetLeftMs.current = Math.max( + 0, + extra.comboCooldownBudgetLeftMs.current - circuitOpenWait.waitMs + ); + return dispatchWithCooldownRetry({ state, deps, extra }); + } + } + + // All set retries exhausted — return the final error + // #10681: finalize the decision trace (all targets failed or skipped). + finalizeComboTrace(deps.traceInvocationId, state.orderedTargets); + finishComboTrace(deps.traceInvocationId, { status: 503 }); + if (!state.lastStatus) { + if (state.recordedAttempts === 0) { + notifyWebhookEvent("request.failed", { + combo: deps.combo.name, + reason: "ALL_TARGETS_SKIPPED", + latencyMs, + fallbackCount: state.fallbackCount, + }); + return withQuotaExhaustionClassification( + errorResponseWithComboDiagnostics( + 503, + "Service temporarily unavailable: all targets were skipped by pre-dispatch filters", + buildComboDiag("all_targets_skipped"), + { code: "ALL_TARGETS_SKIPPED", type: "service_unavailable" } + ), + state.observedFailure ? state.allObservedFailuresQuota : null + ); + } + notifyWebhookEvent("request.failed", { + combo: deps.combo.name, + reason: "ALL_ACCOUNTS_INACTIVE", + latencyMs, + fallbackCount: state.fallbackCount, + }); + recordComboFailure(deps.effectiveSessionId, deps.combo.name); + return errorResponseWithComboDiagnostics( + 503, + "Service temporarily unavailable: all upstream accounts are inactive", + buildComboDiag("all_accounts_inactive"), + { code: "ALL_ACCOUNTS_INACTIVE", type: "service_unavailable" } + ); + } + + // #10501: derive the terminal HTTP status from the structured per-target + // outcomes instead of `state.lastStatus` (whichever target happened to fail + // LAST). A 4xx is preserved only when the request itself is genuinely + // invalid across every eligible target; a heterogeneous mix of failure + // classes (e.g. a quality failure + a sibling's 401) normalizes to a + // 5xx-class status reflecting an infra/provider problem, not a client + // error. See comboErrorAggregation.ts::resolveComboTerminalStatus. + const status = resolveComboTerminalStatus(state.comboErrors, state.lastStatus); + // #10314: build the terminal message from the structured per-target + // outcomes (each distinct class+reason listed separately) instead of + // mashing a single state.lastError with raw `[model (status)]` markers. Connection + // identifiers are redacted. Falls back to state.lastError when no target recorded + // a structured outcome. + const msg = + formatComboOutcomes(state.comboErrors) || state.lastError || "All combo models unavailable"; + + // Cooldown-aware retry: instead of crystallizing a transient failure, wait + // out a SHORT cooldown and re-run the whole set loop. Guarded by the helper + // (quota_exhausted/auth/not-found excluded, ceiling, attempts, budget). + // MAX_GLOBAL_ATTEMPTS still bounds total dispatches. Available to ALL combo + // strategies when enabled — entry is driven by earliestRetryAfter + the + // real model-lockout reason, NOT by whichever target last overwrote + // `status` (a later 403 must not skip the allow-list check for an earlier + // 429's retry-after hint). SECURITY (see comboCooldownRetry.ts header): the + // allow-list is the PRIMARY barrier and `maxWaitMs` only the SECOND one. + // Hardcoding reason:"rate_limit" would drop the primary barrier and leave + // only the ceiling — which does NOT cover a quota_exhausted lock carrying a + // SHORT upstream retry-after. Model lockouts are recorded for all strategies, + // so the real reason is always available. + if (extra.comboCooldownWaitEnabled && state.earliestRetryAfter) { + const decision: ResolveComboCooldownDecisionResult = resolveComboCooldownWaitDecision({ + targets: state.orderedTargets, + earliestRetryAfter: state.earliestRetryAfter, + attempt: extra.comboCooldownAttempt.current, + budgetLeftMs: extra.comboCooldownBudgetLeftMs.current, + settings: deps.resilienceSettings.comboCooldownWait, + // Key each lookup on the TARGET's own model: quota-share combos are + // single-model/multi-account (so this is identical to the previous + // state.orderedTargets[0] behavior), but heterogeneous combos carry a + // different model per target. + lookupLock: (provider, connectionId, target) => { + const rawModel = parseModel(target?.modelStr ?? "").model || ""; + if (!rawModel) return null; + return getModelLockoutInfo(provider, connectionId, rawModel); + }, + computeWaitMs: (retryAfter) => computeClosestRetryAfter(retryAfter).waitMs, + }); + + if (decision.wait) { + deps.log.info( + "COMBO", + `${deps.strategy} cooldown wait: ${msg} — waiting ${Math.ceil( + decision.waitMs / 1000 + )}s (reason=${decision.reason ?? "?"}) then retrying (attempt ${ + extra.comboCooldownAttempt.current + 1 + }/${deps.resilienceSettings.comboCooldownWait.maxAttempts})` + ); + const completed = await waitForCooldownAwareRetry(decision.waitMs, deps.signal); + if (!completed) { + deps.log.info("COMBO", `${deps.strategy} cooldown wait aborted by client disconnect`); + return errorResponse(499, "Request aborted"); + } + extra.comboCooldownAttempt.current += 1; + extra.comboCooldownBudgetLeftMs.current = Math.max( + 0, + extra.comboCooldownBudgetLeftMs.current - decision.waitMs + ); + return dispatchWithCooldownRetry({ state, deps, extra }); + } + } + + // #10681: finalize the decision trace with the aggregated terminal status. + finalizeComboTrace(deps.traceInvocationId, state.orderedTargets); + finishComboTrace(deps.traceInvocationId, { status }); + // Retry-after decoration is separate from the wait decision above: only + // rate-limit-class final statuses may carry a `(reset after ...)` suffix + // (see unavailableRetryGate.ts — do not stitch a peer target's window onto + // a config-class status like 403/422). + if (state.earliestRetryAfter && isRetryAfterEligibleStatus(status)) { + const retryHuman = formatRetryAfter(toRetryAfterDisplayValue(state.earliestRetryAfter)); + deps.log.warn("COMBO", `All models failed | ${msg} (${retryHuman})`); + return withQuotaExhaustionClassification( + unavailableResponse(status, msg, state.earliestRetryAfter, retryHuman), + state.observedFailure ? state.allObservedFailuresQuota : null + ); + } + + // Silent-stop fix: bump the failure counter (pin clears on 3rd consecutive) and emit + // `try-auto` recovery action via buildRecoveryHint so the OC plugin can show "→ Try + // model: auto" instead of an opaque 5xx. We pass the upstream retry-after seconds to + // the hint so the client can render a precise "wait Ns and retry" message. + deps.log.warn("COMBO", `All models failed | ${msg}`); + const { pinClearedNow } = recordComboFailure(deps.effectiveSessionId, deps.combo.name); + if (pinClearedNow) { + deps.log.info( + "COMBO", + `Auto-cleared session_model_history pin for combo "${deps.combo.name}" after ${COMBO_FAILURE_THRESHOLD} consecutive failures to break the silent-stop loop` + ); + } + const retryAfterSeconds = undefined; + // #10966: when every observed failure was independently classified as quota/ + // balance exhaustion (isQuotaExhaustionResponse, tracked via state.observeFailure's + // state.allObservedFailuresQuota accumulator), stamp a stable `quota_exhausted` + // terminalReason instead of forwarding the raw upstream error string. The raw + // string falls through buildRecoveryHint's default branch ("retry" / "failed + // transiently"), which is actively misleading for a durable wallet/quota + // exhaustion — retrying the same combo will never refill it. + const terminalReason = + state.observedFailure && state.allObservedFailuresQuota + ? "quota_exhausted" + : (state.lastError ?? "all_models_failed"); + return withQuotaExhaustionClassification( + errorResponseWithComboDiagnostics( + status, + msg, + buildComboDiag(terminalReason, retryAfterSeconds) + ), + state.observedFailure ? state.allObservedFailuresQuota : null + ); + } + + // Final fallback — when the dispatch returned without crystallizing a status (rare). + // Surface the recovery hint with a generic retry recommendation so the client at least + // gets a non-opaque message instead of "Combo routing completed without an upstream response". + recordComboFailure(deps.effectiveSessionId, deps.combo.name); + return errorResponseWithComboDiagnostics( + 503, + "Combo routing completed without an upstream response", + buildNoUpstreamResponseDiagnostics(state.orderedTargets.length) + ); + } finally { + // #11804: always release the loop-safety timer. Covering every exit path by + // construction here means a future `return` added to this function cannot + // silently reintroduce the leak. + if (activeLoopSafetyTimer) { + clearTimeout(activeLoopSafetyTimer); + activeLoopSafetyTimer = null; + } + } +} diff --git a/open-sse/services/combo/executeTargetAttempt.ts b/open-sse/services/combo/executeTargetAttempt.ts new file mode 100644 index 0000000000..75f2771ed1 --- /dev/null +++ b/open-sse/services/combo/executeTargetAttempt.ts @@ -0,0 +1,1204 @@ +/** + * Retry loop for a combo target that already passed pre-dispatch gates. + * Lift-as-is from combo.ts:1533–2616. Classify helpers live in + * executeTargetClassify.ts. Pin/LKGP side effects go through deps. + * + * @internal — not part of the public combo.ts barrel. + */ +import { + checkFallbackError, + classifyLockoutReason, + decayModelFailureCount, + hasPerModelQuota, + isModelLocked, + lockModelIfPerModelQuota, + recordModelLockoutFailure, + recordProviderFailure, + recordProviderSuccess, + retryHintBypassesMaxCooldownMs, + selectLockoutCooldownMs, +} from "../accountFallback.ts"; +import { errorResponse, errorResponseWithComboDiagnostics } from "../../utils/error.ts"; +import { recordComboFailure, clearComboFailureTracking } from "./failureTracker.ts"; +import { buildRecoveryHint } from "./pinRecovery.ts"; +import { formatExhaustedConnectionKey } from "./comboDiagFormat.ts"; +import { recordComboRequest, getComboMetrics } from "../comboMetrics.ts"; +import { + expandComboSystemPromptIfPresent, + resolveTargetFingerprint, +} from "../comboAgentMiddleware.ts"; +import { + maybeGenerateHandoff, + maybeGenerateUniversalHandoff, + injectUniversalHandoffBody, + SKIP_UNIVERSAL_HANDOFF_FLAG, + DEFAULT_UNIVERSAL_HANDOFF_CONFIG, + type MessageLike, +} from "../contextHandoff.ts"; +import { + recordSessionModelUsage, + getLastSessionModel, + getHandoff, +} from "../../../src/lib/db/contextHandoffs.ts"; +import { resolveModelLockoutSettings } from "../../../src/lib/resilience/modelLockoutSettings"; +import { fetchCodexQuota } from "../codexQuotaFetcher.ts"; +import { emit } from "../../../src/lib/events/eventBus"; +import { notifyWebhookEvent } from "../../../src/lib/webhookDispatcher"; +import { getSessionConnection } from "../sessionManager.ts"; +import { recordStickyBinding } from "./sessionStickiness.ts"; +import { recordStickyWeightedSuccess } from "./rrState.ts"; +import { resolveReasoningBufferedMaxTokens, toPositiveInteger } from "../reasoningTokenBuffer.ts"; +import { parseModel } from "../model.ts"; +import type { ProviderProfile } from "../accountFallback.ts"; +import { + MAX_FALLBACK_WAIT_MS, + clampGlobalAttempts, + shouldSkipForPredictedTtft, + shouldRecordProviderBreakerFailure, + isComboRequestScopedFailure as isScopedFailure, + isStreamReadinessFailureErrorBody, + isStreamEarlyEofErrorBody, + isTokenLimitBreachErrorBody, + isLocalQueueCapacityErrorBody, + toRecordedTarget, + resolveDelayMs, + resolvePersistedConnectionCooldownSkipReason, + isModelScoped400, +} from "./comboPredicates.ts"; +import { applyComboTargetExhaustion } from "./targetExhaustion.ts"; +import { pinNativeCodexTurn } from "./nativeCodexTurnPin.ts"; +import { recordComboDecision } from "./decisionTrace.ts"; +import { recordProviderCooldown } from "../providerCooldownTracker.ts"; +import { + validateResponseQuality, + releaseQualityClone, + releaseRejectedQualityResponse, +} from "./validateQuality.ts"; +import { + isQuotaExhaustionResponse, + recordQuotaExhaustionClassification, +} from "./quotaExhaustion.ts"; +import { classifyComboOutcome, redactConnectionLabel } from "./comboErrorAggregation.ts"; +import { readConnectionForCooldownGate } from "./executeTargetGates.ts"; +import { + remainderIsHomogeneous, + shouldAbortOnInputBoundFailure, + shouldSurfaceBodySpecific400, +} from "./executeTargetClassify.ts"; +import type { CompressionMode } from "../compression/types.ts"; +import type { AttemptLoopDeps, AttemptLoopState, ExecuteTargetResult } from "./attemptLoopTypes.ts"; +import type { ComboDiagnostics } from "../../utils/error.ts"; +import type { ComboErrorBody, ComboRetryAfter, ResolvedComboTarget } from "./types.ts"; +import type { ResponseValidationConfig } from "./responseValidation.ts"; + +export async function executeTargetAttempt(opts: { + index: number; + state: AttemptLoopState; + deps: AttemptLoopDeps; + targetForAttempt: ResolvedComboTarget; + profile: unknown; + protectedPriorityTarget: boolean; +}): Promise { + const { index: i, state, deps, targetForAttempt, protectedPriorityTarget } = opts; + const profile = opts.profile as ProviderProfile | undefined; + const target = state.orderedTargets[i]; + const modelStr = target.modelStr; + const rawModel = parseModel(modelStr).model || modelStr; + const provider = target.provider; + const allowRateLimitedConnection = + Boolean(provider && provider !== "unknown") && + state.transientRateLimitedProviders.has(provider); + const maxGlobalAttempts = clampGlobalAttempts(deps.config.maxGlobalAttempts); + const retryDelayMs = resolveDelayMs(deps.config.retryDelayMs, 2000); + const fallbackDelayMs = resolveDelayMs(deps.config.fallbackDelayMs, 0); + const universalHandoffConfig = deps.universalHandoffConfig ?? DEFAULT_UNIVERSAL_HANDOFF_CONFIG; + + const stopProtectedPriorityTarget = (message: string) => { + state.observeFailure(false, target.executionKey); + deps.clearStaleLKGP(deps.combo.name, target.executionKey, deps.combo.id, deps.log, "COMBO"); + return protectedPriorityTarget + ? { ok: false as const, response: errorResponse(503, message) } + : null; + }; + + const buildComboDiag = ( + terminalReason: string, + retryAfterSeconds?: number + ): ComboDiagnostics => ({ + poolSize: state.orderedTargets.length, + attempted: state.recordedAttempts, + excluded: [ + ...[...state.exhaustedProviders].map((p) => ({ provider: p, reason: "exhausted" })), + ...[...state.exhaustedConnections].map((c) => formatExhaustedConnectionKey(String(c))), + ], + attemptOrder: state.comboAttemptOrder, + terminalReason, + recovery: buildRecoveryHint(terminalReason, retryAfterSeconds), + }); + + // Retry loop for transient errors + for (let retry = 0; retry <= deps.maxRetries; retry++) { + // Fix #1681: Bail out immediately if the client has disconnected + if (deps.signal?.aborted) { + deps.log.info("COMBO", `Client disconnected — aborting combo loop before model ${modelStr}`); + return { ok: false, response: errorResponse(499, "Client disconnected") }; + } + state.globalAttempts++; + if (state.globalAttempts > maxGlobalAttempts) { + deps.log.warn( + "COMBO", + `Maximum combo attempts (${maxGlobalAttempts}) exceeded across all targets and fallbacks. Terminating loop to prevent runaway background requests.` + ); + // Actionable failure instead of an opaque 503 when every candidate + // failed the same recoverable way. If the dominant cause was reasoning + // models exhausting a too-small max_tokens budget (no content output), + // retrying other models can't help — tell the caller to raise max_tokens. + // Silent-stop fix: bump the consecutive-failure counter for this session-combo pair + // so the pin gets cleared on the 3rd attempt (recovery.next_step tells the client). + const reasoningExhausted = /reasoning consumed \d+\/\d+ tokens/.test(state.lastError || ""); + const failureReason = reasoningExhausted + ? "reasoning_budget_exhausted" + : "max_attempts_exceeded"; + recordComboFailure(deps.effectiveSessionId, deps.combo.name); + return { + ok: false, + response: errorResponseWithComboDiagnostics( + 503, + reasoningExhausted + ? "All combo candidates exhausted their token budget on reasoning without producing content. Increase max_tokens — reasoning models need a larger budget to emit content." + : "Maximum combo retry limit reached", + buildComboDiag(failureReason) + ), + }; + } + // Predictive TTFT Circuit Breaker (skip slow models) + if ( + deps.config.zeroLatencyOptimizationsEnabled === true && + deps.config.predictiveTtftMs && + deps.config.predictiveTtftMs > 0 && + retry === 0 + ) { + const cMetrics = getComboMetrics(deps.combo.name); + if (cMetrics) { + const targetKey = state.orderedTargets[i].executionKey || modelStr; + const m = cMetrics.byTarget[targetKey] || cMetrics.byModel[modelStr]; + if (shouldSkipForPredictedTtft(m, deps.config.predictiveTtftMs)) { + deps.log.warn( + "COMBO", + `Predictive TTFT Circuit Breaker: skipping ${modelStr} (avg ${m.avgLatencyMs}ms > max ${deps.config.predictiveTtftMs}ms)` + ); + recordComboDecision(deps.traceInvocationId, { + step: target.executionKey, + target: modelStr, + decision: "skipped_before_dispatch", + reason: "predictive_ttft", + }); + return stopProtectedPriorityTarget(`Predictive latency check rejected ${modelStr}`); + } + } + } + + if (retry > 0) { + deps.log.info( + "COMBO", + `Retrying ${modelStr} in ${retryDelayMs}ms (attempt ${retry + 1}/${deps.maxRetries + 1})` + ); + await new Promise((resolve) => { + const timer = setTimeout(resolve, retryDelayMs); + deps.signal?.addEventListener( + "abort", + () => { + clearTimeout(timer); + resolve(undefined); + }, + { once: true } + ); + }); + if (deps.signal?.aborted) { + deps.log.info("COMBO", `Client disconnected during retry delay — aborting`); + return { ok: false, response: errorResponse(499, "Client disconnected") }; + } + + // Retry re-check: a sibling attempt (or attempt 1) may have persisted + // a quota cooldown while this attempt was sleeping out its retry delay + // ("Trying model 1/7: zai/glm-5.3 (retry 1)" after "already marked + // unavailable until …"). Reads fresh, not cached: see readConnectionForCooldownGate. + const persistedRetrySkip = await resolvePersistedConnectionCooldownSkipReason( + target, + (id) => readConnectionForCooldownGate(id, true), + allowRateLimitedConnection + ); + if (persistedRetrySkip) { + deps.log.info("COMBO", persistedRetrySkip); + if (i > 0) state.fallbackCount++; + return null; + } + } + + deps.log.info( + "COMBO", + `Trying model ${i + 1}/${state.orderedTargets.length}: ${modelStr}${retry > 0 ? ` (retry ${retry})` : ""}` + ); + emit("combo.target.attempt", { + comboName: deps.combo.name, + targetIndex: i, + provider, + model: modelStr, + timestamp: Date.now(), + strategy: deps.strategy, + }); + // QA P0 diagnostics: capture the attempt order (provider/model ids only). + state.comboAttemptOrder.push({ provider: provider ?? "unknown", model: modelStr }); + + // Copy-on-write, not a deep clone (#7847 — 9.53 MiB at 3 targets). Writes here are + // top-level scalars. Invariant: tests/unit/combo-attempt-body-isolation-7847.test.ts. + let attemptBody = { ...(deps.body as Record) } as typeof deps.body; + + // Proactive Context Compression for fallbacks (Zero-Latency optimization) + if ( + deps.config.zeroLatencyOptimizationsEnabled === true && + i > 0 && + deps.config.fallbackCompressionMode && + deps.config.fallbackCompressionMode !== "off" + ) { + const { estimateTokens } = await import("../contextManager.ts"); + // #7847: object, not JSON.stringify — the string branch mis-counts inline images. + const estimatedTokens = estimateTokens(attemptBody); + if (estimatedTokens > (deps.config.fallbackCompressionThreshold ?? 1000)) { + const { applyCompression } = await import("../compression/strategySelector.ts"); + const compressionResult = applyCompression( + attemptBody, + deps.config.fallbackCompressionMode as CompressionMode, + // Opt into the TV1 bail-out so a throwing fallback engine is SKIPPED rather than + // propagating out of executeTarget and being swallowed as a "Speculative task + // error" (which silently drops this combo target). minGainPercent:0 keeps the + // advance behavior identical to the default path — this only adds skip-on-throw. + { model: modelStr, bailout: { enabled: true, minGainPercent: 0 } } + ); + if (compressionResult.compressed) { + deps.log.info( + "COMBO", + `Proactive fallback compression applied (${deps.config.fallbackCompressionMode}): ${estimatedTokens} -> ${compressionResult.stats?.compressedTokens} tokens` + ); + attemptBody = compressionResult.body; + } + } + } + + // Universal handoff: inject existing handoff if model changed. i === 0 + // only: a fallback target (i > 0) serves the SAME client request the + // failed primary target would have served, with the original messages + // already intact -- there's nothing to hand off, since the client never + // saw the earlier target fail. Injecting a handoff note there replaces + // real context with a context-free note, which weaker fallback models + // have been observed treating as license to fabricate content instead + // of just answering the actual request (#12227 follow-up). + if ( + i === 0 && + universalHandoffConfig.enabled && + deps.relayOptions?.sessionId && + !(deps.body as Record)?.[SKIP_UNIVERSAL_HANDOFF_FLAG] + ) { + const lastModel = getLastSessionModel(deps.relayOptions.sessionId, deps.combo.name); + if (lastModel && lastModel !== modelStr) { + const existingHandoff = getHandoff(deps.relayOptions.sessionId, deps.combo.name); + attemptBody = injectUniversalHandoffBody( + attemptBody, // Use the cloned body to maintain isolation + lastModel, + modelStr, + `Model routing: ${lastModel} → ${modelStr}`, + existingHandoff, + universalHandoffConfig.relayMode + ); + } + } + + // Issue #3587: Reasoning models can spend the whole output budget on + // reasoning. Only add headroom when the complete buffer fits inside the + // model's known output cap; otherwise preserve the client's explicit limit. + { + const bodyRecord = attemptBody as Record; + const currentMaxTokens = toPositiveInteger(bodyRecord.max_tokens); + const bufferedMaxTokens = resolveReasoningBufferedMaxTokens(modelStr, bodyRecord.max_tokens, { + enabled: deps.reasoningTokenBufferEnabled !== false, + }); + if (currentMaxTokens !== null && bufferedMaxTokens !== null) { + bodyRecord.max_tokens = bufferedMaxTokens; + if (bufferedMaxTokens !== currentMaxTokens) { + deps.log.info( + "COMBO", + `Reasoning model ${modelStr}: adjusted max_tokens ${currentMaxTokens} -> ${bufferedMaxTokens}` + ); + } + } + } + // #5501: server-side template expansion for the combo system_message — + // resolved per-target, scoped to combo-injected content only (never + // client-owned system messages). Gate: a non-empty combo system_message. + attemptBody = expandComboSystemPromptIfPresent(attemptBody, deps.combo, { + modelId: modelStr, + providerId: provider !== "unknown" ? provider : "", + account: + typeof target.label === "string" && target.label.trim().length > 0 + ? target.label.trim() + : "", + fingerprint: resolveTargetFingerprint(target) ?? "", + }); + // #10681: record dispatch once per target (retries keep the first decision). + if (!state.dispatchedTargets.has(target.executionKey)) { + state.dispatchedTargets.add(target.executionKey); + recordComboDecision(deps.traceInvocationId, { + step: target.executionKey, + target: modelStr, + decision: "dispatched", + }); + } + const result = await deps.handleSingleModelWithTimeout(attemptBody, modelStr, { + ...targetForAttempt, + effectiveComboStrategy: deps.strategy, + failoverBeforeRetry: deps.config.failoverBeforeRetry, + }); + + // Success — validate response quality before returning + if (result.ok) { + const selectedConnectionId = + result.headers?.get("X-OmniRoute-Selected-Connection-Id") || + result.headers?.get("x-omniroute-selected-connection-id") || + undefined; + const effectiveConnectionId = selectedConnectionId || target.connectionId || ""; + + // Clone BEFORE quality check — validateResponseQuality reads the body + // via getReader() which locks the stream. The clone's body is consumed + // by the quality check; the original stays unlocked for piping. + let qualityClone: Response; + try { + qualityClone = result.clone(); + } catch { + qualityClone = result; + } + const quality = await validateResponseQuality( + qualityClone, + deps.clientRequestedStream, + deps.log, + deps.config.responseValidation as ResponseValidationConfig | null | undefined + ); + releaseQualityClone(qualityClone, result, quality); + if (!quality.valid) { + releaseRejectedQualityResponse(qualityClone, result); + deps.log.warn( + "COMBO", + `Model ${modelStr} returned 200 but failed quality check: ${quality.reason}` + ); + // #6692: a quality-rejected 200 never marks the connection row + // unhealthy, so the sticky pin's lazy headroom recheck would never + // catch it either — release it here, on the failing response. + deps.releaseStickyPinOnFailure(deps.sticky.messageHash, effectiveConnectionId); + recordComboRequest(deps.combo.name, modelStr, { + success: false, + latencyMs: Date.now() - deps.startTime, + fallbackCount: state.fallbackCount, + strategy: deps.strategy, + target: toRecordedTarget(target), + }); + state.recordedAttempts++; + // Fix #1707: Set terminal state so the fallback doesn't emit + // misleading ALL_ACCOUNTS_INACTIVE when the real issue is quality. + state.lastError = `Upstream response failed quality validation: ${quality.reason}`; + state.lastStatus = 502; + // #10314: record quality failures as a FIRST-CLASS per-target outcome + // so a quality reason is never silently dropped from the aggregated + // terminal message when a later sibling overwrites lastError. + state.comboErrors.push({ + model: modelStr, + status: 502, + error: quality.reason || "upstream response failed quality validation", + kind: "quality", + }); + if (i > 0) state.fallbackCount++; + if (provider && rawModel) { + const mlSettings = resolveModelLockoutSettings(deps.settings); + if (mlSettings.enabled && mlSettings.errorCodes.includes(502)) { + recordModelLockoutFailure( + provider, + target.connectionId || "", + rawModel, + "quality_failure", + 502, + mlSettings.baseCooldownMs, + profile, + { + exactCooldownMs: mlSettings.useExponentialBackoff ? 0 : mlSettings.baseCooldownMs, + maxCooldownMs: mlSettings.maxCooldownMs, + } + ); + } + } + emit("combo.target.failed", { + comboName: deps.combo.name, + targetIndex: i, + provider, + model: modelStr, + error: `Quality: ${quality.reason}`, + latencyMs: Date.now() - deps.startTime, + }); + state.observeFailure(false, target.executionKey); + return protectedPriorityTarget + ? { + ok: false, + response: errorResponse(502, "Upstream response failed quality validation"), + } + : null; + } + + if (Boolean(deps.clientManagedResponsesContext) && effectiveConnectionId) { + pinNativeCodexTurn({ + body: deps.body, + comboName: deps.combo.name, + target, + connectionId: effectiveConnectionId, + }); + } + + // Success decay: a healthy response walks the model's lockout failure + // count back down (and eventually clears an expired lockout entirely). + if (provider && rawModel) { + const dcResult = decayModelFailureCount(provider, effectiveConnectionId, rawModel); + if (dcResult.cleared) { + deps.log.info("COMBO", `Model ${modelStr} fully recovered — lockout cleared`); + } else if (dcResult.newFailureCount > 0) { + deps.log.debug( + "COMBO", + `Model ${modelStr} decayed to failureCount=${dcResult.newFailureCount}` + ); + } + } + + const latencyMs = Date.now() - deps.startTime; + emit("combo.target.succeeded", { + comboName: deps.combo.name, + targetIndex: i, + provider, + model: modelStr, + latencyMs, + }); + deps.log.info( + "COMBO", + `Model ${modelStr} succeeded (${latencyMs}ms, ${state.fallbackCount} fallbacks)` + ); + recordComboRequest(deps.combo.name, modelStr, { + success: true, + latencyMs, + fallbackCount: state.fallbackCount, + strategy: deps.strategy, + target: toRecordedTarget(target), + }); + state.recordedAttempts++; + + // Reset cooldown on success + if (provider && provider !== "unknown") { + recordProviderSuccess(provider, effectiveConnectionId || undefined); + } + if (deps.strategy === "weighted" && (deps.stickyWeightedLimit ?? 0) > 1) { + const stickySuccessKey = deps.getWeightedStepKeyForTarget?.(target); + if (stickySuccessKey) { + recordStickyWeightedSuccess( + deps.combo.name, + stickySuccessKey, + deps.stickyWeightedLimit ?? 0 + ); + } + } + // Webhook fan-out: best-effort, never blocks the response stream. + notifyWebhookEvent("request.completed", { + combo: deps.combo.name, + provider, + model: modelStr, + account: + typeof target.label === "string" && target.label.trim().length > 0 + ? target.label.trim() + : "", + accountId: effectiveConnectionId ?? "", + latencyMs, + fallbackCount: state.fallbackCount, + }); + + // Silent-stop fix: reset the consecutive-failure counter for this session-combo pair + // on every successful dispatch so a transient recovery doesn't get "credited" against + // the threshold the user already paid through to clear the stale pin. + if (deps.effectiveSessionId) { + clearComboFailureTracking(deps.effectiveSessionId, deps.combo.name); + } + // Context cache pinning: record model usage for session-based pinning + // (independent of universal handoff — always fires when context_cache_protection is on) + // #3825: write under the SAME effectiveSessionId used by the read site so a + // sessionless conversation re-pins to this model on its next turn. + if ( + deps.combo.context_cache_protection && + deps.effectiveSessionId && + !(deps.body as Record)?.[SKIP_UNIVERSAL_HANDOFF_FLAG] + ) { + recordSessionModelUsage( + deps.effectiveSessionId, + deps.combo.name, + modelStr, + provider, + target.connectionId ?? undefined + ); + } + + // Universal handoff: record model usage for session + if ( + universalHandoffConfig.enabled && + deps.relayOptions?.sessionId && + !(deps.body as Record)?.[SKIP_UNIVERSAL_HANDOFF_FLAG] + ) { + const prevModel = getLastSessionModel(deps.relayOptions.sessionId, deps.combo.name); + recordSessionModelUsage( + deps.relayOptions.sessionId, + deps.combo.name, + modelStr, + provider, + target.connectionId ?? undefined + ); + // i === 0 only: a same-request fallback target (i > 0) never + // needs a summary generated for it -- see the injection-site + // comment above. recordSessionModelUsage above stays + // unconditional regardless of i: it must reflect whichever + // model actually served THIS response, since the next + // request's i === 0 comparison depends on that being + // accurate even when this response came from a fallback. + if (i === 0 && prevModel && prevModel !== modelStr) { + const handoffSourceMessages = + Array.isArray(deps.body?.messages) && deps.body.messages.length > 0 + ? deps.body.messages + : Array.isArray(deps.body?.input) + ? deps.body.input + : []; + + maybeGenerateUniversalHandoff({ + sessionId: deps.relayOptions.sessionId, + comboName: deps.combo.name, + messages: handoffSourceMessages as MessageLike[], + prevModel, + currModel: modelStr, + universalConfig: universalHandoffConfig, + handleSingleModel: deps.handleSingleModelWithTimeout, + }); + } + + recordSessionModelUsage( + deps.relayOptions.sessionId, + deps.combo.name, + modelStr, + provider, + target.connectionId ?? undefined + ); + } + // Context-relay intentionally splits responsibilities: + // combo.ts decides whether a successful turn should generate a handoff, + // while chat.ts injects the handoff after the real connectionId is resolved. + if ( + deps.strategy === "context-relay" && + deps.relayOptions?.sessionId && + deps.relayConfig && + (deps.relayConfig.handoffProviders ?? []).includes(provider) && + provider === "codex" + ) { + const connectionId = getSessionConnection(deps.relayOptions.sessionId); + if (connectionId) { + const quotaInfo = await fetchCodexQuota(connectionId).catch(() => null); + if (quotaInfo) { + const resetCandidates = [ + quotaInfo.windows?.session?.resetAt, + quotaInfo.windows?.weekly?.resetAt, + quotaInfo.resetAt, + ] + .filter((value): value is string => typeof value === "string" && value.length > 0) + .sort((a, b) => a.localeCompare(b)); + const handoffSourceMessages = + Array.isArray(deps.body?.messages) && deps.body.messages.length > 0 + ? deps.body.messages + : Array.isArray(deps.body?.input) + ? deps.body.input + : []; + + maybeGenerateHandoff({ + sessionId: deps.relayOptions.sessionId, + comboName: deps.combo.name, + connectionId, + percentUsed: quotaInfo.percentUsed, + messages: handoffSourceMessages, + model: modelStr, + expiresAt: resetCandidates[0] || null, + config: deps.relayConfig, + handleSingleModel: deps.handleSingleModelWithTimeout, + }); + } + } + } + if (deps.sticky.messageHash && target.connectionId) + recordStickyBinding(deps.sticky.messageHash, target.connectionId); // LKGP (#919): + if (provider) { + const connId = effectiveConnectionId || undefined; + void (async () => { + try { + const { setLKGP } = await import("@/lib/db/settings"); + await Promise.all([ + setLKGP(deps.combo.name, target.executionKey, provider, connId), + setLKGP(deps.combo.name, deps.combo.id || deps.combo.name, provider, connId), + ]); + } catch (err) { + deps.log.warn( + "COMBO", + "Failed to record Last Known Good Provider. This is non-fatal.", + { + err, + } + ); + } + })(); + } + + return { ok: true, response: result }; + } + + // Extract error info from response + let errorText = result.statusText || ""; + let errorBody: ComboErrorBody = null; + let retryAfter: ComboRetryAfter | null = null; + try { + const cloned = result.clone(); + try { + const text = await cloned.text(); + if (text) { + errorText = text.substring(0, 500); + errorBody = JSON.parse(text); + const parsedError = errorBody?.error; + errorText = + (typeof parsedError === "object" && parsedError?.message) || + (typeof parsedError === "string" ? parsedError : null) || + errorBody?.message || + errorText; + // Live incident (log id 1784457764961-73 follow-up): the pre-dispatch + // "all credentials cooling down" rejection (buildModelCooldownBody / + // handleNoCredentials in src/sse/handlers/chatHelpers.ts) nests its + // retry hint as error.retry_after (ISO string) / error.reset_seconds + // (seconds), not the top-level `retryAfter` every other 429 shape + // uses. Without this fallback, lastStatus gets recorded (fixed above) + // but earliestRetryAfter stays null, so the final check falls through + // to the generic "all combo models unavailable" error instead of ever + // reaching the cooldown-wait decision — same class of bug, different + // response shape. + const nestedRetryAfter = + typeof parsedError === "object" ? (parsedError?.retry_after ?? null) : null; + const nestedResetSeconds = + typeof parsedError === "object" ? (parsedError?.reset_seconds ?? null) : null; + retryAfter = + errorBody?.retryAfter || + nestedRetryAfter || + (typeof nestedResetSeconds === "number" && nestedResetSeconds > 0 + ? new Date(Date.now() + nestedResetSeconds * 1000).toISOString() + : null); + } + } catch { + /* Clone parse failed */ + } + } catch { + /* Clone failed */ + } + + // Track earliest retryAfter + if ( + retryAfter && + (!state.earliestRetryAfter || new Date(retryAfter) < new Date(state.earliestRetryAfter)) + ) { + state.earliestRetryAfter = retryAfter; + } + + // Normalize error text + if (typeof errorText !== "string") { + try { + errorText = JSON.stringify(errorText); + } catch { + errorText = String(errorText); + } + } + + const isStreamReadinessFailure = + (result.status === 502 || result.status === 504) && + isStreamReadinessFailureErrorBody(errorBody); + // An early EOF is an upstream failure, not a readiness probe — the breaker must + // see it even though the transient-retry path below treats both codes alike. + const isStreamEarlyEof = + (result.status === 502 || result.status === 504) && isStreamEarlyEofErrorBody(errorBody); + + // FIX 5: a local per-API-key token-limit 429 must not cool shared accounts. + const isTokenLimitBreach = result.status === 429 && isTokenLimitBreachErrorBody(errorBody); + const isLocalQueueCapacity = isLocalQueueCapacityErrorBody(errorBody); + + // Fix #1681: Status 499 means client disconnected — stop combo loop immediately. + // There is no point trying fallback models when nobody is listening. + if (result.status === 499) { + deps.log.info("COMBO", `Client disconnected (499) during ${modelStr} — stopping combo loop`); + recordComboRequest(deps.combo.name, modelStr, { + success: false, + latencyMs: Date.now() - deps.startTime, + fallbackCount: state.fallbackCount, + strategy: deps.strategy, + target: toRecordedTarget(target), + }); + state.recordedAttempts++; + // executeTarget must return the {ok,response} contract — a raw Response + // here makes the speculative loop's res.ok/res.response checks both miss, + // so the combo would wrongly fall through to the next model after a 499. + return { ok: false, response: result }; + } + if (isLocalQueueCapacity) { + deps.log.info( + "COMBO", + `Local rate-limit queue capacity reached for ${modelStr} — returning without upstream fallback` + ); + recordComboRequest(deps.combo.name, modelStr, { + success: false, + latencyMs: Date.now() - deps.startTime, + fallbackCount: state.fallbackCount, + strategy: deps.strategy, + target: toRecordedTarget(target), + }); + state.recordedAttempts++; + if (i > 0) state.fallbackCount++; + return { ok: false, response: result }; + } + + // Combo fallback is target-level orchestration: a non-ok target response is + // treated as local to that target and the combo continues to the next target. + // Error classification is retained only for retry/cooldown pacing; it must + // not decide whether fallback happens, including for generic 400 responses. + const rawError = errorBody?.error; + const structuredError = + rawError && typeof rawError === "object" + ? { + // Upstream JSON may carry a numeric `code`/`type` (e.g. {"code":40001}). + // Coerce to string if present instead of discarding, so downstream string + // ops (.toLowerCase, .startsWith) can run safely without type crashes. + code: + (rawError as Record).code !== undefined && + (rawError as Record).code !== null + ? String((rawError as Record).code) + : undefined, + type: + (rawError as Record).type !== undefined && + (rawError as Record).type !== null + ? String((rawError as Record).type) + : undefined, + } + : undefined; + const scopedFailure = isScopedFailure(result, errorText, structuredError); + + // #8375: input-bound request-scoped failures (context_length_exceeded) are + // deterministic for the same input — retrying on other accounts of the same + // model will fail identically. Short-circuit the combo immediately with the + // original error instead of burning MAX_GLOBAL_ATTEMPTS. + // Scoped to homogeneous remainders only: a heterogeneous combo (#6637) may + // have a later target with a different, larger context window that would + // NOT reject the same input — isContextOverflow400 below exists precisely to + // let that case fall through, so only short-circuit when every remaining + // target is the same model (the "retrying will fail identically" premise + // only holds within a homogeneous same-model pool). + const remainderHomogeneous = remainderIsHomogeneous(state.orderedTargets, i, modelStr); + const isInputBoundFailure = shouldAbortOnInputBoundFailure({ + structuredError, + remainderIsHomogeneous: remainderHomogeneous, + }); + if (isInputBoundFailure) { + deps.log.warn( + "COMBO", + `Input-bound request failure from ${modelStr} — aborting combo (same input will fail identically on every account)` + ); + recordComboRequest(deps.combo.name, modelStr, { + success: false, + latencyMs: Date.now() - deps.startTime, + fallbackCount: state.fallbackCount, + strategy: deps.strategy, + target: toRecordedTarget(target), + }); + state.recordedAttempts++; + if (i > 0) state.fallbackCount++; + return { ok: false, response: result }; + } + const fallbackResult = checkFallbackError( + result.status, + errorText, + 0, + protectedPriorityTarget ? rawModel : null, + provider, + result.headers, + profile, + structuredError + ); + const { cooldownMs } = fallbackResult; + // #6863: a parsed upstream quota reset (e.g. Antigravity "Resets in 92h27m28s") + // arrives in `quotaResetHintMs` — it bypasses the operator-gated + // `useUpstreamRetryHints` connection-cooldown setting. Mirror the + // single-model path (src/sse/services/auth.ts): when the retry hint was + // already honored, `cooldownMs` IS the upstream value; otherwise prefer the + // parsed quota reset — even when it is SHORTER than the fallback cooldown + // (e.g. subscription-quota 1h default vs a real "resets in 10m"). + // `selectLockoutCooldownMs` still ignores hints at/below the base cooldown, + // so absent/tiny hints keep the #1308 exponential-backoff behavior. + const lockoutHintMs = + fallbackResult.usedUpstreamRetryHint === true + ? cooldownMs + : (fallbackResult.quotaResetHintMs ?? 0); + // Only a transport header or google.rpc.RetryInfo is authoritative enough + // to bypass maxCooldownMs. Prose and generic JSON remain useful exact hints, + // but the operator cap still bounds them. + const lockoutHintVerified = retryHintBypassesMaxCooldownMs(fallbackResult.retryHintSource); + const selectedConnectionId = + result.headers?.get("X-OmniRoute-Selected-Connection-Id") || + result.headers?.get("x-omniroute-selected-connection-id") || + undefined; + const targetWithConnection = selectedConnectionId + ? { ...target, connectionId: selectedConnectionId } + : target; + + // #1731 / #1731v2: classify the upstream error and update the exhaustion sets + // (shared with handleRoundRobinCombo). Returns whether the provider is fully exhausted. + const providerExhausted = applyComboTargetExhaustion(targetWithConnection, { + result, + fallbackResult, + errorText, + rawModel, + isTokenLimitBreach, + allAccountsRateLimited: false, + requestScopedFailure: scopedFailure, + sets: { + exhaustedProviders: state.exhaustedProviders, + exhaustedConnections: state.exhaustedConnections, + transientRateLimitedProviders: state.transientRateLimitedProviders, + }, + log: deps.log, + tag: "COMBO", + exhaustedLogLevel: "info", + structuredError, + }); + // #6692: this connection was just classified as provider/connection-level + // exhausted — if it's the currently sticky-bound one, release the pin now + // rather than waiting for the next turn's lazy headroom/status recheck. + deps.releaseStickyPinOnFailure(deps.sticky.messageHash, targetWithConnection.connectionId); + if ( + providerExhausted || + state.exhaustedConnections.has(`${provider}:${targetWithConnection.connectionId}`) || + (provider && state.exhaustedProviders.has(provider)) + ) { + deps.clearStaleLKGP(deps.combo.name, target.executionKey, deps.combo.id, deps.log, "COMBO"); + } + + // #2101: Prevent infinite fallback loops with 400 Bad Request errors that are genuinely + // body-specific (malformed JSON, bad format, missing required fields). + // These should NOT stop the combo: + // - Context overflow: different models have different context windows + // - Max_tokens / param errors: different models have different output limits + // - Model access denied / "not supported": different providers serve different + // model sets — keep the model in the combo and try the next target (#5249). + // Wrapper words like "invalid" / "bad request" still stop only when the text is + // NOT model-scoped (e.g. "invalid message format"). + if ( + shouldSurfaceBodySpecific400({ + status: result.status, + errorText, + shouldFallback: fallbackResult.shouldFallback, + }) + ) { + deps.log.warn( + "COMBO", + `400 Bad Request with body-specific error detected on ${modelStr} — skipping fallback to other targets to prevent infinite loop` + ); + // Record the failure and break to avoid trying other targets with the same bad request + recordComboRequest(deps.combo.name, modelStr, { + success: false, + latencyMs: Date.now() - deps.startTime, + fallbackCount: state.fallbackCount, + strategy: deps.strategy, + target: toRecordedTarget(target), + }); + state.recordedAttempts++; + state.lastError = errorText || String(result.status); + state.comboErrors.push({ + model: modelStr, + status: result.status, + error: errorText || String(result.status), + kind: classifyComboOutcome(result.status, errorText), + }); + state.lastStatus = result.status; + if (i > 0) state.fallbackCount++; + deps.log.warn("COMBO", `Model ${modelStr} failed with body-specific error, stopping combo`); + deps.clearStaleLKGP(deps.combo.name, target.executionKey, deps.combo.id, deps.log, "COMBO"); + // #4279: surface the 400 via the {ok,response} contract so the OUTER + // target loop resolves the combo and stops. A bare `break` here only + // exits the inner retry loop; executeTarget then returns null, which + // the outer loop treats as "this target produced nothing" and advances + // to the next model — so the guard failed to stop fallback and a combo + // of N body-rejecting targets tried all N. Mirrors the 499 path above. + return { ok: false, response: result }; + } + + // A model-scoped 400 ("The requested model is not supported" / "not + // available for integrator") is permanent for THIS connection — the + // account/integration will not gain support for the model mid-session. + // Combo still advances to the next target immediately (unchanged, + // preserves #5249's cross-provider fallback), but without a lockout + // here the SAME dead model gets retried on every future, separate + // request forever (observed: every auto-combo request wasted several + // upstream 400s on the same GitHub models, all day). isModelLocked() + // is checked before dispatch (see the pre-check above this loop), so + // this lockout is honored on the next request. + if (result.status === 400 && isModelScoped400(errorText) && provider && rawModel) { + lockModelIfPerModelQuota( + provider, + targetWithConnection.connectionId || "", + rawModel, + "model_capacity", + 60 * 60 * 1000 // 1h + ); + } + + // Trigger shared provider circuit breaker for 5xx errors and connection failures. If the + // next target is on the same provider, don't mark it failed (a different model may still + // succeed) — #8376: EXCEPT a proxy-unreachable failure, which poisons every model alike. + // G-02: when fallbackResult.skipProviderBreaker is set (embedded service supervisor outage + // signalled via X-Omni-Fallback-Hint: connection_cooldown) apply cooldown only — never trip. + const nextTarget = state.orderedTargets[i + 1]; + const sameProviderNext = + typeof nextTarget?.provider === "string" && nextTarget.provider === provider; + if ( + shouldRecordProviderBreakerFailure({ + isStreamReadinessFailure, + isStreamEarlyEof, + status: result.status, + sameProviderNext, + skipProviderBreaker: fallbackResult.skipProviderBreaker, + requestScopedFailure: scopedFailure, + error: errorText, + isProxyUnreachable: structuredError?.code === "proxy_unreachable", + }) + ) { + const isQueueTimeout = + errorText.includes("RATE_LIMIT_QUEUE_TIMEOUT") || + errorText.includes("RATE_LIMIT_QUEUE_WEDGED"); + recordProviderFailure(provider, deps.log, targetWithConnection.connectionId, profile, { + isQueueTimeout, + isNetworkError: structuredError?.code === "proxy_unreachable", + }); + } + + const quotaExhausted = await isQuotaExhaustionResponse(result, provider, rawModel, profile); + recordQuotaExhaustionClassification(result, quotaExhausted); + state.observeFailure(quotaExhausted, target.executionKey); + + // Check if this is a transient error worth retrying on same model. + // A token-limit 429 is terminal for the client — never retry it. + const isTransient = + !isStreamReadinessFailure && + !isTokenLimitBreach && + !scopedFailure && + [408, 429, 500, 502, 503, 504].includes(result.status); + // failoverBeforeRetry means what it says: prefer the next sibling + // target over hammering this one again. Without this check, a + // transient error always re-hit the SAME model up to maxRetries + // times regardless of the setting — config.failoverBeforeRetry was + // threaded through to skipUpstreamRetry (a different, lower-level + // retry mechanism) but never consulted here, so a rate-limited + // model got maxRetries+1 back-to-back attempts on itself before + // this loop's own fallback-to-next-target ever ran (#2417). Only + // skip the same-model retry when `nextTarget` (computed above) + // actually gives us somewhere to fail over to — with no sibling + // left, skipping just burns the last attempt for nothing. + // + // #10217 round-4 fix: this guard reads `failoverBeforeRetryExplicit` + // (opt-in only), NOT `config.failoverBeforeRetry` — that field + // defaults to true for the separate skipUpstreamRetry mechanism + // (see DEFAULT_COMBO_CONFIG comment in comboConfig.ts) and reading + // it here would silently skip the same-model retry for every combo, + // not just ones that explicitly opted in. + if ( + retry < deps.maxRetries && + isTransient && + !providerExhausted && + (!deps.config.failoverBeforeRetryExplicit || !nextTarget) + ) { + if ( + !protectedPriorityTarget && + provider && + rawModel && + isModelLocked(provider, targetWithConnection.connectionId || "", rawModel) + ) { + deps.log.info("COMBO", `Skipping retry for ${modelStr} — model lockout active`); + // Live incident (log id 1784457764961-73): earliestRetryAfter is already + // captured above from THIS dispatch's own response, but lastStatus was + // never recorded on this bail-out path — so once every target in the set + // hit an existing lockout, lastStatus stayed null and the final `if + // (!lastStatus)` check crystallized an immediate ALL_ACCOUNTS_INACTIVE 503 + // instead of ever reaching the `if (earliestRetryAfter)` cooldown-wait + // decision below, even though a real 429 with a short (~1min) retry-after + // was just observed. Recording it here mirrors the "done retrying" path. + state.lastError = errorText || String(result.status); + state.lastStatus = result.status; + if (i > 0) state.fallbackCount++; + return null; + } + // Record model lockout immediately on the first transient failure — + // once the model is cooling down, retrying it would waste an upstream + // call and extend the cooldown via exponential backoff. + let lockoutRecorded = false; + if (!protectedPriorityTarget && provider && rawModel && retry === 0 && !scopedFailure) { + const mlSettings = resolveModelLockoutSettings(deps.settings); + if (mlSettings.enabled && mlSettings.errorCodes.includes(result.status)) { + recordModelLockoutFailure( + provider, + targetWithConnection.connectionId || "", + rawModel, + classifyLockoutReason(result.status), + result.status, + mlSettings.baseCooldownMs, + profile, + { + // #1308/#6863: honor a long upstream reset (e.g. "Resets in 160h") over + // the short base cooldown / exponential backoff when present. #7940's + // maxCooldownMs cap only applies to synthetic values — a verified + // upstream reset (lockoutHintVerified) bypasses it. + exactCooldownMs: selectLockoutCooldownMs(lockoutHintMs, mlSettings), + maxCooldownMs: mlSettings.maxCooldownMs, + // Preserve authoritative structured/header resets; clamp body prose. + exactCooldownIsUpstreamReset: lockoutHintVerified, + } + ); + lockoutRecorded = true; + } + } + if (lockoutRecorded) { + deps.log.info("COMBO", `Skipping retry for ${modelStr} — model lockout active`); + // Same fix as the already-locked branch above — this is the + // first-failure lockout path, so lastStatus needs recording here too. + state.lastError = errorText || String(result.status); + state.lastStatus = result.status; + if (i > 0) state.fallbackCount++; + return null; + } + continue; // Retry same model (transient error, no lockout recorded) + } + + // Done retrying this model + const protectedTargetTrust = state.targetFailureTrust.get(target.executionKey); + if ( + protectedPriorityTarget && + (!protectedTargetTrust?.observedFailure || !protectedTargetTrust.allObservedFailuresQuota) + ) { + recordComboRequest(deps.combo.name, modelStr, { + success: false, + latencyMs: Date.now() - deps.startTime, + fallbackCount: state.fallbackCount, + strategy: deps.strategy, + target: toRecordedTarget(target), + }); + state.recordedAttempts++; + return { ok: false, response: result }; + } + recordComboRequest(deps.combo.name, modelStr, { + success: false, + latencyMs: Date.now() - deps.startTime, + fallbackCount: state.fallbackCount, + strategy: deps.strategy, + target: toRecordedTarget(target), + }); + // LKGP (#919) mirror of the success-path set below: a just-failed target + // must not keep re-pinning itself as the "last known good" choice for the + // *next* separate request. Circuit breaker / model lockout deliberately + // don't react to request-scoped failure classes (see scopedFailure below), + // so nothing else clears this stale pin. + deps.clearStaleLKGP(deps.combo.name, target.executionKey, deps.combo.id, deps.log, "COMBO"); + state.recordedAttempts++; + state.lastError = errorText || String(result.status); + state.comboErrors.push({ + model: modelStr, + status: result.status, + error: errorText || String(result.status), + kind: classifyComboOutcome(result.status, errorText), + }); + state.lastStatus = result.status; + if (i > 0) state.fallbackCount++; + // Wire combo failures into the resilience dashboard (model-level lockout) + // alongside the provider-level cooldown below — they govern different scopes. + if (provider && rawModel && !scopedFailure) { + const mlSettings = resolveModelLockoutSettings(deps.settings); + if (mlSettings.enabled && mlSettings.errorCodes.includes(result.status)) { + recordModelLockoutFailure( + provider, + targetWithConnection.connectionId || "", + rawModel, + classifyLockoutReason(result.status), + result.status, + mlSettings.baseCooldownMs, + profile, + { + // #1308/#6863: honor a long upstream reset over base/exponential cooldown. + // #7940's maxCooldownMs cap only applies to synthetic values — a verified + // upstream reset (lockoutHintVerified) bypasses it. + exactCooldownMs: selectLockoutCooldownMs(lockoutHintMs, mlSettings), + maxCooldownMs: mlSettings.maxCooldownMs, + // Preserve authoritative structured/header resets; clamp body prose. + exactCooldownIsUpstreamReset: lockoutHintVerified, + } + ); + } + } + deps.log.warn("COMBO", `Model ${modelStr} failed, trying next`, { + status: result.status, + errorBody: redactConnectionLabel(errorText), + }); + + // #5976: per-model-quota providers (Gemini, GitHub, etc.) multiplex models + // behind one connection. A model-level 500 or 429 (RPM) must NOT cool down + // the entire provider — sibling models may still succeed. Skip cooldown + // recording for these providers on 500/429 errors so the next target can try. + if ( + deps.resilienceSettings.providerCooldown.enabled && + provider && + provider !== "unknown" && + !scopedFailure && + !((result.status === 500 || result.status === 429) && hasPerModelQuota(provider, rawModel)) + ) { + recordProviderCooldown( + provider, + targetWithConnection.connectionId ?? undefined, + deps.resilienceSettings + ); + } + + const fallbackWaitMs = + fallbackDelayMs > 0 && cooldownMs > 0 && cooldownMs <= MAX_FALLBACK_WAIT_MS + ? Math.min(cooldownMs, fallbackDelayMs) + : 0; + if ([502, 503, 504].includes(result.status) && fallbackWaitMs > 0) { + deps.log.debug?.("COMBO", `Waiting ${fallbackWaitMs}ms before fallback to next model`); + await new Promise((resolve) => { + const timer = setTimeout(resolve, fallbackWaitMs); + deps.signal?.addEventListener( + "abort", + () => { + clearTimeout(timer); + resolve(undefined); + }, + { once: true } + ); + }); + if (deps.signal?.aborted) { + deps.log.info("COMBO", `Client disconnected during fallback wait — aborting`); + return { ok: false, response: errorResponse(499, "Client disconnected") }; + } + } + + return null; + } + return null; +} diff --git a/open-sse/services/combo/executeTargetClassify.ts b/open-sse/services/combo/executeTargetClassify.ts new file mode 100644 index 0000000000..ffb60db2cc --- /dev/null +++ b/open-sse/services/combo/executeTargetClassify.ts @@ -0,0 +1,54 @@ +/** + * Pure classify helpers for executeTarget's retry loop. + * Lift-as-is from combo.ts #8375 / #2101 / #4279. No I/O. + * + * @internal — not part of the public combo.ts barrel. + */ +import { + isContextOverflow400, + isInputBoundRequestFailure, + isModelScoped400, + isParamValidation400, +} from "./comboPredicates.ts"; + +export function remainderIsHomogeneous( + orderedTargets: { modelStr: string }[], + index: number, + modelStr: string +): boolean { + return orderedTargets.slice(index + 1).every((nextInPool) => nextInPool.modelStr === modelStr); +} + +export function shouldAbortOnInputBoundFailure(opts: { + structuredError: unknown; + remainderIsHomogeneous: boolean; +}): boolean { + const structured = opts.structuredError as + { code?: string | null; type?: string | null } | undefined; + return isInputBoundRequestFailure(structured) && opts.remainderIsHomogeneous; +} + +/** + * #2101 / #4279: body-specific 400 must surface via {ok,response}, not null. + * Same predicate chain as combo.ts (overflow / param / model-scoped excluded). + */ +export function shouldSurfaceBodySpecific400(opts: { + status: number; + errorText: string; + shouldFallback: boolean; +}): boolean { + const errorText = opts.errorText; + return ( + opts.status === 400 && + opts.shouldFallback && + !isContextOverflow400(errorText) && + !isParamValidation400(errorText) && + !isModelScoped400(errorText) && + (errorText.toLowerCase().includes("context") || + errorText.toLowerCase().includes("prompt") || + errorText.toLowerCase().includes("token") || + errorText.toLowerCase().includes("malformed") || + errorText.toLowerCase().includes("invalid") || + errorText.toLowerCase().includes("bad request")) + ); +} diff --git a/open-sse/services/combo/executeTargetGates.ts b/open-sse/services/combo/executeTargetGates.ts new file mode 100644 index 0000000000..b0d5f3edf4 --- /dev/null +++ b/open-sse/services/combo/executeTargetGates.ts @@ -0,0 +1,329 @@ +/** + * Pre-dispatch skip gates for handleComboChat's executeTarget. + * Order is locked (spec §4.1). Do not reorder. + * + * Extracted from combo.ts executeTarget entry through the retry loop. + * + * @internal — not part of the public combo.ts barrel. + */ +import { + getRuntimeProviderProfile, + isAccountSemaphoreFull, + isModelLocked, +} from "../accountFallback.ts"; +import { isProviderInCooldown } from "../providerCooldownTracker.ts"; +import { checkCredentialGate, logCredentialSkip } from "../credentialGate.ts"; +import { errorResponse } from "../../utils/error.ts"; +import { getCircuitBreaker } from "../../../src/shared/utils/circuitBreaker"; +import { parseModel } from "../model.ts"; +import { canAffordRequest } from "../../../src/lib/quota/quotaScheduler.ts"; +import { getCachedProviderConnectionById } from "../../../src/lib/db/readCache.ts"; +import { lookupPositiveCap } from "./concurrencyCaps.ts"; +import { recordComboDecision } from "./decisionTrace.ts"; +import { + getExhaustedTargetSkipReason, + resolvePersistedConnectionCooldownSkipReason, +} from "./comboPredicates.ts"; +import { resolveQuotaExhaustionCutoffForTarget } from "./quotaExhaustionCutoff.ts"; +import type { AttemptLoopDeps, AttemptLoopState, GateDecision } from "./attemptLoopTypes.ts"; +import type { ResolvedComboTarget } from "./types.ts"; + +/** + * Cached vs fresh connection read for the persisted-cooldown gate. + * `fresh: false` (first attempt) uses the 5s readCache. `fresh: true` + * (every retry) goes straight to SQLite. + * + * Task 2 call sites pass `false` — same as combo.ts executeTarget today. + * Retry-path `fresh: true` is Task 4 wiring, not this extract. + */ +export async function readConnectionForCooldownGate( + connectionId: string, + fresh: boolean +): Promise | null | undefined> { + if (!fresh) return getCachedProviderConnectionById(connectionId); + const { getProviderConnectionById } = await import("@/lib/db/providers"); + return (await getProviderConnectionById(connectionId)) as Record | null; +} + +export async function evaluateExecuteTargetGates(opts: { + index: number; + state: AttemptLoopState; + deps: AttemptLoopDeps; +}): Promise { + const { index: i, state, deps } = opts; + const target = state.orderedTargets[i]; + const modelStr = target.modelStr; + const rawModel = parseModel(modelStr).model || modelStr; + const provider = target.provider; + const protectedPriorityTarget = + deps.strategy === "priority" && target.fallbackOnlyOnQuotaExhaustion === true; + + const stopProtectedPriorityTarget = (message: string) => { + state.observeFailure(false, target.executionKey); + deps.clearStaleLKGP(deps.combo.name, target.executionKey, deps.combo.id, deps.log, "COMBO"); + return protectedPriorityTarget + ? { ok: false as const, response: errorResponse(503, message) } + : null; + }; + + // Lift-as-is from combo.ts executeTarget: only count a fallback when + // this is not the first ordered target. Do not change the condition. + const bumpFallback = () => { + if (i > 0) state.fallbackCount++; + }; + + const cb = getCircuitBreaker(provider); + const cbStatus = cb.getStatus(); + if (cbStatus.state === "OPEN") { + state.skippedForCircuitOpen = true; + if ( + cbStatus.retryAfterMs > 0 && + (state.earliestCircuitOpenRetryMs === 0 || + cbStatus.retryAfterMs < state.earliestCircuitOpenRetryMs) + ) { + state.earliestCircuitOpenRetryMs = cbStatus.retryAfterMs; + } + deps.log.info("COMBO", `Skipping ${modelStr} — circuit breaker OPEN for ${provider}`); + recordComboDecision(deps.traceInvocationId, { + step: target.executionKey, + target: modelStr, + decision: "skipped_before_dispatch", + reason: "circuit_open", + }); + bumpFallback(); + return { + kind: "skip", + result: stopProtectedPriorityTarget(`Provider ${provider} circuit breaker is open`), + }; + } + + if ( + deps.resilienceSettings.providerCooldown.enabled && + Boolean(provider && provider !== "unknown") && + (isProviderInCooldown(provider, target.connectionId ?? undefined, deps.resilienceSettings) || + isProviderInCooldown(provider, undefined, deps.resilienceSettings)) + ) { + deps.log.info("COMBO", `Skipping ${modelStr} — provider ${provider} in global cooldown`); + recordComboDecision(deps.traceInvocationId, { + step: target.executionKey, + target: modelStr, + decision: "skipped_before_dispatch", + reason: "provider_cooldown", + }); + bumpFallback(); + return { + kind: "skip", + result: stopProtectedPriorityTarget(`Provider ${provider} is in cooldown`), + }; + } + + const preScreenEntry = deps.preScreenMap.get(target.executionKey); + const profile = preScreenEntry?.profile ?? (await getRuntimeProviderProfile(provider)); + + const allowRateLimitedConnection = + Boolean(provider && provider !== "unknown") && + state.transientRateLimitedProviders.has(provider); + const abortSignal = state.abortControllers.get(i)?.signal; + const targetForAttempt = allowRateLimitedConnection + ? { + ...target, + allowRateLimitedConnection: true, + modelAbortSignal: abortSignal, + } + : { ...target, modelAbortSignal: abortSignal }; + + if (target.connectionId && !allowRateLimitedConnection) { + const persistedSkip = await resolvePersistedConnectionCooldownSkipReason( + target, + (id) => readConnectionForCooldownGate(id, false), + allowRateLimitedConnection + ); + if (persistedSkip) { + // Lift-as-is: combo.ts skips without observeFailure / stopProtectedPriorityTarget. + deps.log.info("COMBO", persistedSkip); + deps.clearStaleLKGP(deps.combo.name, target.executionKey, deps.combo.id, deps.log, "COMBO"); + bumpFallback(); + return { kind: "skip", result: null }; + } + } + + const exhaustedSkip = getExhaustedTargetSkipReason( + target, + state.exhaustedProviders, + state.exhaustedConnections + ); + if (exhaustedSkip) { + deps.log.info("COMBO", exhaustedSkip); + recordComboDecision(deps.traceInvocationId, { + step: target.executionKey, + target: modelStr, + decision: "skipped_before_dispatch", + reason: "request_exhaustion", + }); + bumpFallback(); + return { + kind: "skip", + result: stopProtectedPriorityTarget(`Target ${modelStr} is unavailable`), + }; + } + + if (provider && rawModel && isModelLocked(provider, target.connectionId || "", rawModel)) { + deps.log.info("COMBO", `Skipping ${modelStr} — model locked by resilience (cooldown active)`); + recordComboDecision(deps.traceInvocationId, { + step: target.executionKey, + target: modelStr, + decision: "skipped_before_dispatch", + reason: "model_lockout", + }); + bumpFallback(); + return { + kind: "skip", + result: stopProtectedPriorityTarget(`Model ${modelStr} is locked`), + }; + } + + if (deps.strategy !== "auto" && provider && target.connectionId) { + const quotaCutoff = await resolveQuotaExhaustionCutoffForTarget( + provider, + target.connectionId, + deps.resilienceSettings, + deps.quotaCutoffResetWindowConfig, + deps.combo.name, + deps.log, + modelStr + ); + if (quotaCutoff.blocked) { + deps.log.info( + "COMBO", + `Skipping ${modelStr} — quota exhaustion cutoff (${quotaCutoff.reason || "quota_exhausted"})` + ); + deps.clearStaleLKGP(deps.combo.name, target.executionKey, deps.combo.id, deps.log, "COMBO"); + recordComboDecision(deps.traceInvocationId, { + step: target.executionKey, + target: modelStr, + decision: "skipped_before_dispatch", + reason: "quota_cutoff", + }); + bumpFallback(); + state.observeFailure(true, target.executionKey); + if (protectedPriorityTarget) { + const protectedTargetTrust = state.targetFailureTrust.get(target.executionKey); + if (!protectedTargetTrust?.allObservedFailuresQuota) { + return { + kind: "skip", + result: { + ok: false, + response: errorResponse(503, `Target ${modelStr} is unavailable`), + }, + }; + } + } + return { kind: "skip", result: null }; + } + } + + // Lift-as-is: combo.ts reads the env flag inline, not via AttemptLoopDeps. + if (process.env.OMNIROUTE_QUOTA_AWARE_ROUTING === "1" && provider && target.connectionId) { + const quotaDecision = canAffordRequest( + target.connectionId, + modelStr, + deps.body as Record | null | undefined + ); + if (!quotaDecision.affordable) { + deps.log.info( + "COMBO", + `Skipping ${modelStr} — quota budget ${quotaDecision.reason} (remaining ${quotaDecision.tokensRemaining ?? 0}, cost ${quotaDecision.estimatedCost ?? 0})` + ); + deps.clearStaleLKGP(deps.combo.name, target.executionKey, deps.combo.id, deps.log, "COMBO"); + bumpFallback(); + return { kind: "skip", result: null }; + } + } + + if (deps.isModelAvailable) { + const available = await deps.isModelAvailable(modelStr, targetForAttempt); + if (!available) { + deps.log.debug?.( + "COMBO", + `Skipping ${modelStr} — no credentials available or model excluded` + ); + deps.clearStaleLKGP(deps.combo.name, target.executionKey, deps.combo.id, deps.log, "COMBO"); + recordComboDecision(deps.traceInvocationId, { + step: target.executionKey, + target: modelStr, + decision: "skipped_before_dispatch", + reason: "availability", + }); + bumpFallback(); + return { + kind: "skip", + result: stopProtectedPriorityTarget(`Model ${modelStr} is unavailable`), + }; + } + } + + // Lift-as-is: combo.ts uses the same `as string | undefined` cast. + const connectionId = target.connectionId as string | undefined; + if (connectionId) { + const gateResult = checkCredentialGate(connectionId, provider, modelStr); + if (gateResult.allowed === false) { + logCredentialSkip(deps.log, modelStr, gateResult.reason || "Credential gate blocked"); + recordComboDecision(deps.traceInvocationId, { + step: target.executionKey, + target: modelStr, + decision: "skipped_before_dispatch", + reason: "credential_gate", + }); + bumpFallback(); + return { + kind: "skip", + result: stopProtectedPriorityTarget(`Credential gate blocked ${modelStr}`), + }; + } + + const maxConcurrentCap = await lookupPositiveCap(connectionId); + if (maxConcurrentCap && isAccountSemaphoreFull(provider, connectionId, maxConcurrentCap)) { + deps.log.info( + "COMBO", + `Skipping ${modelStr} — connection ${connectionId} is at max concurrency cap (${maxConcurrentCap})` + ); + recordComboDecision(deps.traceInvocationId, { + step: target.executionKey, + target: modelStr, + decision: "skipped_before_dispatch", + reason: "concurrency_cap", + }); + bumpFallback(); + return { + kind: "skip", + result: stopProtectedPriorityTarget(`Connection capacity reached for ${modelStr}`), + }; + } + } + + if ( + deps.perTargetAdmission && + !(await deps.perTargetAdmission({ + modelStr, + executionKey: target.executionKey, + body: deps.body, + })) + ) { + deps.log.info("COMBO", `Skipping ${modelStr} — admission lane full (#9654)`); + recordComboDecision(deps.traceInvocationId, { + step: target.executionKey, + target: modelStr, + decision: "skipped_before_dispatch", + reason: "admission_lane", + }); + bumpFallback(); + return { kind: "skip", result: null }; + } + + return { + kind: "proceed", + targetForAttempt: targetForAttempt as ResolvedComboTarget, + profile, + protectedPriorityTarget, + }; +} diff --git a/tests/unit/combo-loop-safety-timer-leak-11804.test.ts b/tests/unit/combo-loop-safety-timer-leak-11804.test.ts index 14c1031068..69e629f1d2 100644 --- a/tests/unit/combo-loop-safety-timer-leak-11804.test.ts +++ b/tests/unit/combo-loop-safety-timer-leak-11804.test.ts @@ -1,7 +1,7 @@ /** * #11804 — the combo loop-safety timer must be cleared on EVERY exit path. * - * `dispatchWithCooldownRetry` (open-sse/services/combo.ts) arms a + * `dispatchWithCooldownRetry` (open-sse/services/combo/comboAttemptLoop.ts) arms a * `setTimeout(..., loopSafetyMs)` — 10 minutes by default — once per `setTry` * iteration, so a combo that never produces a terminal response still answers * the client with a 504 instead of hanging forever. @@ -29,7 +29,10 @@ import { fileURLToPath } from "node:url"; import { dirname, resolve } from "node:path"; const here = dirname(fileURLToPath(import.meta.url)); -const comboSrc = readFileSync(resolve(here, "../../open-sse/services/combo.ts"), "utf8"); +const comboSrc = readFileSync( + resolve(here, "../../open-sse/services/combo/comboAttemptLoop.ts"), + "utf8" +); test("#11804: the loop-safety timer is released in a finally, not only on success", () => { assert.match( diff --git a/tests/unit/combo/combo-attempt-loop.test.ts b/tests/unit/combo/combo-attempt-loop.test.ts new file mode 100644 index 0000000000..eb6ab6774f --- /dev/null +++ b/tests/unit/combo/combo-attempt-loop.test.ts @@ -0,0 +1,66 @@ +/** + * Characterization for comboAttemptLoop.ts (#11804 finally + gates/attempt wiring). + * Plan Task 4. RED until that module exists. + */ +import test from "node:test"; +import assert from "node:assert/strict"; +import { readFileSync } from "node:fs"; +import { fileURLToPath } from "node:url"; +import { dirname, resolve } from "node:path"; + +const here = dirname(fileURLToPath(import.meta.url)); + +test("dispatchWithCooldownRetry clears activeLoopSafetyTimer in finally", async () => { + const src = readFileSync( + resolve(here, "../../../open-sse/services/combo/comboAttemptLoop.ts"), + "utf8" + ); + assert.match(src, /finally\s*\{[^}]*clearTimeout\(activeLoopSafetyTimer\)/s); + assert.match(src, /activeLoopSafetyTimer = loopSafetyTimer/); +}); + +test("dispatchWithCooldownRetry calls evaluateGates then executeAttempt, not inline executeTarget", async () => { + const src = readFileSync( + resolve(here, "../../../open-sse/services/combo/comboAttemptLoop.ts"), + "utf8" + ); + assert.match(src, /extra\.evaluateGates/); + assert.match(src, /extra\.executeAttempt/); + // Thin wrapper may keep the local name; the old inline retry/gate body must not. + assert.doesNotMatch(src, /getCircuitBreaker\(provider\)/); + assert.doesNotMatch(src, /for \(let retry = 0; retry <= deps\.maxRetries/); +}); + +test("attempt budget lives on state.globalAttempts, not extra.globalAttempts box", async () => { + const loopSrc = readFileSync( + resolve(here, "../../../open-sse/services/combo/comboAttemptLoop.ts"), + "utf8" + ); + const comboSrc = readFileSync(resolve(here, "../../../open-sse/services/combo.ts"), "utf8"); + const attemptSrc = readFileSync( + resolve(here, "../../../open-sse/services/combo/executeTargetAttempt.ts"), + "utf8" + ); + assert.match(attemptSrc, /state\.globalAttempts\+\+/); + assert.doesNotMatch(loopSrc, /globalAttempts:\s*\{\s*current:\s*number\s*\}/); + assert.doesNotMatch(comboSrc, /globalAttempts:\s*\{\s*current:\s*0\s*\}/); +}); + +test("handleComboChatInner does not leave unused delay locals or unused failureTracker import", async () => { + const comboSrc = readFileSync(resolve(here, "../../../open-sse/services/combo.ts"), "utf8"); + const inner = comboSrc.slice( + comboSrc.indexOf("async function handleComboChatInner"), + comboSrc.indexOf("async function handleRoundRobinCombo") + ); + assert.doesNotMatch(inner, /const retryDelayMs = resolveDelayMs/); + assert.doesNotMatch(inner, /const fallbackDelayMs = resolveDelayMs/); + assert.doesNotMatch(comboSrc, /clearComboFailureTracking/); +}); + +test("hedge delay does not declare unused timeoutResolve", async () => { + const loopSrc = readFileSync( + resolve(here, "../../../open-sse/services/combo/comboAttemptLoop.ts"), + "utf8" + ); + assert.doesNotMatch(loopSrc, /let timeoutResolve/); +}); diff --git a/tests/unit/combo/execute-target-attempt.test.ts b/tests/unit/combo/execute-target-attempt.test.ts new file mode 100644 index 0000000000..1609f53f80 --- /dev/null +++ b/tests/unit/combo/execute-target-attempt.test.ts @@ -0,0 +1,288 @@ +/** + * Characterization for executeTarget retry loop + classify + * (open-sse/services/combo/executeTargetAttempt.ts, + * open-sse/services/combo/executeTargetClassify.ts). + * + * Plan Task 3. RED until those modules exist. + */ +import test from "node:test"; +import assert from "node:assert/strict"; +import type { + AttemptLoopDeps, + AttemptLoopState, +} from "../../../open-sse/services/combo/attemptLoopTypes.ts"; +import type { ResolvedComboTarget } from "../../../open-sse/services/combo/types.ts"; + +function emptyState(overrides: Partial = {}): AttemptLoopState { + return { + orderedTargets: [], + fallbackCount: 0, + recordedAttempts: 0, + comboErrors: [], + lastError: null, + lastStatus: null, + earliestRetryAfter: null, + comboExpired: false, + exhaustedProviders: new Set(), + exhaustedConnections: new Set(), + transientRateLimitedProviders: new Set(), + abortControllers: new Map([[0, new AbortController()]]), + dispatchedTargets: new Set(), + targetFailureTrust: new Map(), + comboAttemptOrder: [], + skippedForCircuitOpen: false, + earliestCircuitOpenRetryMs: 0, + globalAttempts: 0, + observedFailure: false, + allObservedFailuresQuota: true, + observeFailure() {}, + ...overrides, + }; +} + +function baseDeps(overrides: Partial = {}): AttemptLoopDeps { + const handleSingleModelWithTimeout = async () => { + throw new Error("handleSingleModel must be stubbed"); + }; + return { + strategy: "priority", + combo: { name: "t", models: [] }, + config: {}, + log: { info() {}, warn() {}, debug() {}, error() {} }, + settings: null, + resilienceSettings: { + providerCooldown: { enabled: false }, + } as AttemptLoopDeps["resilienceSettings"], + sticky: { targets: [], messageHash: null, stuck: false }, + effectiveSessionId: null, + preScreenMap: new Map(), + quotaCutoffResetWindowConfig: {} as AttemptLoopDeps["quotaCutoffResetWindowConfig"], + maxRetries: 0, + traceInvocationId: "inv-attempt", + clientRequestedStream: false, + handleSingleModelWithTimeout, + body: { messages: [{ role: "user", content: "hi" }] }, + startTime: Date.now(), + releaseStickyPinOnFailure() {}, + clearStaleLKGP() {}, + ...overrides, + }; +} + +function modelTarget(overrides: Partial = {}): ResolvedComboTarget { + return { + kind: "model", + stepId: "s1", + executionKey: "ek-1", + modelStr: "openai/gpt-4o", + provider: "openai", + providerId: null, + connectionId: "c-fail", + weight: 1, + label: null, + ...overrides, + }; +} + +function emptyContent200(connectionId = "c-fail"): Response { + return new Response( + JSON.stringify({ choices: [{ message: { role: "assistant", content: "" } }] }), + { + status: 200, + headers: { + "content-type": "application/json", + "x-omniroute-selected-connection-id": connectionId, + }, + } + ); +} + +// "invalid message format" — same fixture as combo-body-specific-400-stop-4279.test.ts +function bodySpecific400(): Response { + return new Response( + JSON.stringify({ + detail: "Invalid message format: the request body is malformed.", + }), + { status: 400, headers: { "Content-Type": "application/json" } } + ); +} + +test("remainderIsHomogeneous is true only when remaining targets share modelStr", async () => { + const { remainderIsHomogeneous } = + await import("../../../open-sse/services/combo/executeTargetClassify.ts"); + const same = [ + { modelStr: "openai/gpt-4o" }, + { modelStr: "openai/gpt-4o" }, + { modelStr: "openai/gpt-4o" }, + ]; + assert.equal(remainderIsHomogeneous(same, 0, "openai/gpt-4o"), true); + const mixed = [{ modelStr: "openai/gpt-4o" }, { modelStr: "anthropic/claude" }]; + assert.equal(remainderIsHomogeneous(mixed, 0, "openai/gpt-4o"), false); + assert.equal(remainderIsHomogeneous(same, 2, "openai/gpt-4o"), true); +}); + +test("shouldAbortOnInputBoundFailure requires homogeneous remainder", async () => { + const { shouldAbortOnInputBoundFailure } = + await import("../../../open-sse/services/combo/executeTargetClassify.ts"); + const structured = { code: "context_length_exceeded" }; + assert.equal( + shouldAbortOnInputBoundFailure({ structuredError: structured, remainderIsHomogeneous: true }), + true + ); + assert.equal( + shouldAbortOnInputBoundFailure({ structuredError: structured, remainderIsHomogeneous: false }), + false + ); + assert.equal( + shouldAbortOnInputBoundFailure({ + structuredError: { code: "rate_limit" }, + remainderIsHomogeneous: true, + }), + false + ); +}); + +test("shouldSurfaceBodySpecific400 matches #4279 invalid-format 400, not model-scoped", async () => { + const { shouldSurfaceBodySpecific400 } = + await import("../../../open-sse/services/combo/executeTargetClassify.ts"); + assert.equal( + shouldSurfaceBodySpecific400({ + status: 400, + errorText: "Invalid message format: the request body is malformed.", + shouldFallback: true, + }), + true + ); + assert.equal( + shouldSurfaceBodySpecific400({ + status: 400, + errorText: "The requested model is not supported", + shouldFallback: true, + }), + false + ); + assert.equal( + shouldSurfaceBodySpecific400({ + status: 429, + errorText: "Invalid message format: the request body is malformed.", + shouldFallback: true, + }), + false + ); +}); + +test("quality-rejected 200 calls releaseStickyPinOnFailure and records kind quality", async () => { + const { executeTargetAttempt } = + await import("../../../open-sse/services/combo/executeTargetAttempt.ts"); + let released: string | null = null; + const target = modelTarget({ connectionId: "c-fail" }); + const deps = baseDeps({ + maxRetries: 0, + clientRequestedStream: false, + releaseStickyPinOnFailure(_hash, id) { + released = String(id); + }, + handleSingleModelWithTimeout: async () => emptyContent200("c-fail"), + sticky: { targets: [], messageHash: "h", stuck: true }, + }); + const state = emptyState({ + orderedTargets: [target], + abortControllers: new Map([[0, new AbortController()]]), + }); + const result = await executeTargetAttempt({ + index: 0, + state, + deps, + targetForAttempt: target, + profile: {}, + protectedPriorityTarget: false, + }); + assert.equal(released, "c-fail"); + assert.equal( + state.comboErrors.some((e) => e.kind === "quality"), + true + ); + assert.equal(result, null); +}); + +test("injection: missing releaseStickyPinOnFailure forwarding goes red on quality fail", async () => { + const { executeTargetAttempt } = + await import("../../../open-sse/services/combo/executeTargetAttempt.ts"); + let callCount = 0; + const target = modelTarget({ connectionId: "c-fail" }); + const deps = baseDeps({ + maxRetries: 0, + clientRequestedStream: false, + releaseStickyPinOnFailure() { + callCount += 1; + }, + handleSingleModelWithTimeout: async () => emptyContent200("c-fail"), + sticky: { targets: [], messageHash: "h", stuck: true }, + }); + const state = emptyState({ + orderedTargets: [target], + abortControllers: new Map([[0, new AbortController()]]), + }); + await executeTargetAttempt({ + index: 0, + state, + deps, + targetForAttempt: target, + profile: {}, + protectedPriorityTarget: false, + }); + assert.equal(callCount, 1); +}); + +test("499 surfaces {ok:false,response} and does not continue retries", async () => { + const { executeTargetAttempt } = + await import("../../../open-sse/services/combo/executeTargetAttempt.ts"); + let calls = 0; + const target = modelTarget({ connectionId: "c1" }); + const deps = baseDeps({ + maxRetries: 3, + handleSingleModelWithTimeout: async () => { + calls += 1; + return new Response("disconnected", { status: 499 }); + }, + }); + const state = emptyState({ + orderedTargets: [target], + abortControllers: new Map([[0, new AbortController()]]), + }); + const result = await executeTargetAttempt({ + index: 0, + state, + deps, + targetForAttempt: target, + profile: {}, + protectedPriorityTarget: false, + }); + assert.equal(result?.ok, false); + assert.equal(result?.response?.status, 499); + assert.equal(calls, 1); +}); + +test("body-specific 400 surfaces via {ok,response} not null", async () => { + const { executeTargetAttempt } = + await import("../../../open-sse/services/combo/executeTargetAttempt.ts"); + const target = modelTarget({ connectionId: "c1", modelStr: "codex/gpt-5.2" }); + const deps = baseDeps({ + maxRetries: 0, + handleSingleModelWithTimeout: async () => bodySpecific400(), + }); + const state = emptyState({ + orderedTargets: [target], + abortControllers: new Map([[0, new AbortController()]]), + }); + const result = await executeTargetAttempt({ + index: 0, + state, + deps, + targetForAttempt: target, + profile: {}, + protectedPriorityTarget: false, + }); + assert.equal(result?.ok, false); + assert.equal(result?.response?.status, 400); +}); diff --git a/tests/unit/combo/execute-target-gates.test.ts b/tests/unit/combo/execute-target-gates.test.ts new file mode 100644 index 0000000000..4d121cdb13 --- /dev/null +++ b/tests/unit/combo/execute-target-gates.test.ts @@ -0,0 +1,158 @@ +/** + * Characterization for executeTarget pre-dispatch gates + * (open-sse/services/combo/executeTargetGates.ts). + */ +import test from "node:test"; +import assert from "node:assert/strict"; +import { getCircuitBreaker, STATE } from "../../../src/shared/utils/circuitBreaker.ts"; +import type { + AttemptLoopDeps, + AttemptLoopState, + GateDecision, +} from "../../../open-sse/services/combo/attemptLoopTypes.ts"; +import type { ResolvedComboTarget } from "../../../open-sse/services/combo/types.ts"; + +test("attemptLoopTypes exports GateDecision discriminant", async () => { + const mod = await import("../../../open-sse/services/combo/attemptLoopTypes.ts"); + assert.equal(typeof mod, "object"); +}); + +function emptyState(overrides: Partial = {}): AttemptLoopState { + return { + orderedTargets: [], + fallbackCount: 0, + recordedAttempts: 0, + comboErrors: [], + lastError: null, + lastStatus: null, + earliestRetryAfter: null, + comboExpired: false, + exhaustedProviders: new Set(), + exhaustedConnections: new Set(), + transientRateLimitedProviders: new Set(), + abortControllers: new Map([[0, new AbortController()]]), + dispatchedTargets: new Set(), + targetFailureTrust: new Map(), + comboAttemptOrder: [], + skippedForCircuitOpen: false, + earliestCircuitOpenRetryMs: 0, + globalAttempts: 0, + observedFailure: false, + allObservedFailuresQuota: true, + observeFailure() {}, + ...overrides, + }; +} + +function baseDeps(overrides: Partial = {}): AttemptLoopDeps { + const handleSingleModelWithTimeout = async () => { + throw new Error("handleSingleModel must not be called from gates"); + }; + return { + strategy: "priority", + combo: { name: "t", models: [] }, + config: {}, + log: { info() {}, warn() {}, debug() {}, error() {} }, + settings: null, + resilienceSettings: { + providerCooldown: { enabled: false }, + } as AttemptLoopDeps["resilienceSettings"], + sticky: { targets: [], messageHash: null, stuck: false }, + effectiveSessionId: null, + preScreenMap: new Map(), + quotaCutoffResetWindowConfig: {} as AttemptLoopDeps["quotaCutoffResetWindowConfig"], + maxRetries: 0, + traceInvocationId: "inv-test", + clientRequestedStream: false, + handleSingleModelWithTimeout, + body: { messages: [{ role: "user", content: "hi" }] }, + startTime: Date.now(), + releaseStickyPinOnFailure() {}, + clearStaleLKGP() {}, + ...overrides, + }; +} + +function modelTarget(overrides: Partial = {}): ResolvedComboTarget { + return { + kind: "model", + stepId: "s1", + executionKey: "ek-1", + modelStr: "openai/gpt-4o", + provider: "openai", + providerId: null, + connectionId: "c1", + weight: 1, + label: null, + ...overrides, + }; +} + +test("breaker OPEN skips and does not call handleSingleModel", async () => { + const { evaluateExecuteTargetGates } = + await import("../../../open-sse/services/combo/executeTargetGates.ts"); + const provider = `openai-gates-test-${Date.now()}`; + const cb = getCircuitBreaker(provider, { failureThreshold: 1, resetTimeout: 60_000 }); + cb._onFailure("transient"); + assert.equal(cb.getStatus().state, STATE.OPEN); + const target = modelTarget({ provider, modelStr: `${provider}/gpt-4o-mini` }); + const state = emptyState({ orderedTargets: [target] }); + const decision: GateDecision = await evaluateExecuteTargetGates({ + index: 0, + state, + deps: baseDeps(), + }); + assert.equal(decision.kind, "skip"); + assert.equal(state.skippedForCircuitOpen, true); +}); + +test("exhausted connection skip uses getExhaustedTargetSkipReason", async () => { + const { evaluateExecuteTargetGates } = + await import("../../../open-sse/services/combo/executeTargetGates.ts"); + const target = modelTarget({ connectionId: "conn-1", provider: "openai" }); + const state = emptyState({ + orderedTargets: [target], + exhaustedConnections: new Set(["openai:conn-1"]), + }); + const decision = await evaluateExecuteTargetGates({ index: 0, state, deps: baseDeps() }); + assert.equal(decision.kind, "skip"); + if (decision.kind === "skip") { + assert.equal(decision.result, null); + } +}); + +test("quota cutoff skipped for strategy auto", async () => { + const { evaluateExecuteTargetGates } = + await import("../../../open-sse/services/combo/executeTargetGates.ts"); + const target = modelTarget(); + const state = emptyState({ orderedTargets: [target] }); + const decision = await evaluateExecuteTargetGates({ + index: 0, + state, + deps: baseDeps({ strategy: "auto" }), + }); + assert.equal(decision.kind, "proceed"); +}); + +test("protected priority non-quota skip returns 503 response not null", async () => { + const { evaluateExecuteTargetGates } = + await import("../../../open-sse/services/combo/executeTargetGates.ts"); + const target = modelTarget({ + connectionId: "c1", + fallbackOnlyOnQuotaExhaustion: true, + }); + const state = emptyState({ + orderedTargets: [target], + exhaustedConnections: new Set(["openai:c1"]), + }); + const decision = await evaluateExecuteTargetGates({ + index: 0, + state, + deps: baseDeps({ strategy: "priority" }), + }); + assert.equal(decision.kind, "skip"); + if (decision.kind === "skip") { + assert.equal(decision.result?.ok, false); + assert.equal(decision.result?.response?.status, 503); + } +});