mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-08-04 22:32:12 +03:00
* chore(ci): add .mergify.yml to main — Mergify only reads config from the default branch (#7168) * fix(ci): add the auto-enqueue pull_request_rule to the Mergify config (queue_conditions alone are eligibility-only) (#7179) * fix(ci): migrate Mergify auto-enqueue to merge_protections_settings.auto_merge_conditions (rules-based path is EOL 2026-07-16) (#7216) * fix(ci): drop Mergify batch settings (batching is a paid-tier feature; free plan queue is serial) (#7220) * fix(ci): merge queue tolerates the advisory dast-smoke failure (its GH-hosted build hang dequeued every attempt) (#7225) * test(ci): make the #6634 selfref guard hermetic — main's copy hard-fails every PR (#7341) main's copy of this test still does git I/O inside a unit test: const baseSrc = git(['show', 'origin/main:' + FILE]); Runners check out a shallow single ref, so origin/main does not resolve and the test dies with 'fatal: invalid object name origin/main'. Every PR into main fails Unit Tests (7/8) on it — today that is #7313, #7315, #7316, #7334, #7336 and #7337, six PRs red on a defect none of them introduced. #7313 has no other red at all. release/v3.8.49 already carries a fix (2e42b8efc, #7174: try/catch, fetch origin/main on demand, t.skip() when unreachable), but it only reaches main at release time — so main stays broken for the whole cycle. Cherry-picking it would also import a new problem: PR Test Policy classifies t.skip() as a silenced assertion, which we watched it correctly catch on #7300 today. This is the hermetic version instead (ported from #7327, which does the same for the release branch): read the file straight off disk, compare against an empty base so baseTaut/baseExtTaut are 0 — the strictest possible comparison point — and call evaluateMasking() directly. No git ref, no fetch, no skip, nothing the runner's checkout depth can break. The #6634 regression stays covered: the guard's logic lives in SELF_TEST_FIXTURE_RE (check-test-masking.mjs:337), not in the test. Proven both ways on main before committing — neutralise SELF_TEST_FIXTURE_RE to /$^/ and the test FAILS; restore it and it passes 2/2, with check-test-masking.mjs left byte-identical. Co-authored-by: growab <nekron@icloud.com> * chore(quality): tighten main's coverage baseline to the CI's real numbers (#7347) main's ratchet had been failing --require-tighten on every PR: 11 metrics improved but the baseline was never tightened. Same class as the #6634 selfref guard — an infra fix that lands only on the release branch leaves main red for the whole cycle, and every PR into main pays for it. Values are the merged-coverage numbers from a run on main itself (a local run measures ~68% vs CI's ~80%; the baseline's own note warns about that gap). Only the 11 coverage values change — gitleaks and semgrepFindings keep main's own state. No changelog fragment: #7326 carries it on release/v3.8.49, and a second one here would double the entry at release time. * Add cliproxy provider exposure controls and manifest injection (#7329) * feat(fusion): let judge use its own knowledge and override the panel (#6804) The judge prompt said to write an answer 'grounded in that analysis', implicitly capping output at the panel's union. When all panel members miss or are collectively wrong on something, the judge should apply its own reasoning as a full participant and override consensus, while keeping an honesty guard against fabrication. Adds a regression test. Co-authored-by: Chirag Singhal <chirag127@users.noreply.github.com> * fix(api): raise provider apiKey cap for cookie-based web providers (#6715) (#6759) * fix(cli): fall back to settings.json when Claude Code binary is unresolvable (#6701) (#6734) getCliRuntimeStatus() only ever answered `installed` from binary resolution (known install paths + where/which PATH search), so a stale PATH, moved binary, or uncatalogued install method reported "not found" even when ~/.claude/settings.json proved the CLI was installed and used before — regressing behind upstream 9router's checkClaudeInstalled(), which already falls back to the settings file when where/which fails. withSettingsFallback() (new src/shared/services/cliInstallFallback.ts, kept out of the frozen cliRuntime.ts to respect its file-size ceiling) restores that parity: only when the binary lookup's own reason is "not_found" (never for deliberate security rejections like unsafe/relative env overrides or symlink escapes) and the tool's settings file exists on disk. * fix(providers): honor explicit thinking.budget_tokens 0 in openai->gemini transform (#6813) (#6821) The transform forwarded the Claude-style thinking.budget_tokens into generationConfig.thinkingConfig.thinkingBudget, but the presence check was truthy (&& thinking.budget_tokens). An explicit budget_tokens: 0 — the natural way to disable thinking — is falsy, so it was dropped and the request fell through to the default thinkingConfig injection, making the model think despite an explicit request for zero. Use an explicit numeric check so 0 is honored as thinkingBudget 0; includeThoughts is only set for a non-zero budget. * fix(compression): reconcile outer vs per-engine token counts (#6488) (#6741) * fix(compression): reconcile outer vs per-engine token counts on degenerate output (#6488) Outer originalTokens/compressedTokens (real tiktoken counter over extracted message text) diverged from engineBreakdown[0]'s counts (a crude JSON.stringify(requestBody).length/4 estimate), worst on small/degenerate inputs where JSON structural overhead dominates. A single-engine breakdown entry represents the exact same before/after transformation as the overall response, so reconcileSingleEngineTokens() now overwrites that one entry's counts with the outer, more accurate figures; multi-step pipeline breakdowns are left untouched. * chore(6741): resolve release sync — CHANGELOG.md restored to release tip, entry moved to changelog.d fragment (fragments-first) * fix(api): accept enableRenderers in RTK compression config schema (#6703) (#6757) * fix(db): break probe-failed/restore loop on large storage.sqlite (#6632) Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy files belong to #6620, not this PR); keeps only the author's changes. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * feat(cursor): add Opus 4.8, Fable 5, and Sonnet 5 model families (#6779) Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy files belong to #6620, not this PR); keeps only the author's cursor registry + test changes. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * fix(translator): read PDF/video file attachments for Gemini/Antigravity and Claude (#6790) Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy files belong to #6620, not this PR); keeps only the author's translator + test changes. Co-authored-by: Wital <witalorocha216@gmail.com> Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * fix(codex): strip include from compact responses requests (#6805) * fix(codex): strip include from compact responses requests Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy files belong to #6620, not this PR); keeps only the author's changes. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * fix(6805): move include-strip assertion to standalone test file to keep executor-codex.test.ts under frozen size cap Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> --------- Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * fix(i18n): translate hardcoded Portuguese dashboard strings to English (#6769) Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy files belong to #6620, not this PR); keeps only the author's changes. Co-authored-by: Chirag Singhal <chirag127@users.noreply.github.com> Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * fix(bootstrap): filter empty process.env values to prevent Docker env crash loop (#6828) Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy files belong to #6620, not this PR) and the direct CHANGELOG.md edit (fragments-first); keeps only the author's bootstrap change. Co-authored-by: Andrian B. <andrewbalanesq@gmail.com> Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * fix(providers): update SenseNova Token Plan support (#6330) Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy files belong to #6620, not this PR); the author's constants/registry/snapshot deltas were re-applied cleanly onto the release tip. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * fix(providers): classify 404 as MODEL_NOT_FOUND to stop retry storm (#6829) Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy files belong to #6620, not this PR) and the direct CHANGELOG.md edit (fragments-first); the author's chatCore/errorClassifier deltas were re-applied cleanly onto the release tip. Co-authored-by: Andrian B. <andrewbalanesq@gmail.com> Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * fix(api): accept all catalog engines on compression PUT schema (#6792) Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy files belong to #6620, not this PR). Resolved the release's OmniGlyph engine addition additively (types.ts/compression.ts kept both 'relevance' and 'omniglyph') and extended stackedPipelineStepSchema + STACKED_PIPELINE_ENGINE_INTENSITIES with the omniglyph branch so the ENGINE_CATALOG-parity test passes. Co-authored-by: Pitchfork-and-Torch <Pitchfork-and-Torch@users.noreply.github.com> Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * fix(api): point CLI health command at /api/monitoring/health (#6677) (#6717) * fix(api): point CLI health command at /api/monitoring/health (#6677) bin/cli/commands/health.mjs called GET /api/health, a route that was moved to /api/monitoring/health without updating the CLI; the top-level /api/health handler never existed on disk (only degradation/ and ping/ sub-routes). Point runHealthCommand()/runHealthComponentsCommand() at /api/monitoring/health and read its real payload shape (activeConnections, circuitBreakers: {open,halfOpen,closed}, memoryUsage) instead of the old nonexistent requests/breakers/cache/memory fields. * chore(6717): re-sync onto release tip; move CHANGELOG entry to changelog.d fragment (fragments-first) * chore(cursor): add Grok 4.5 effort/fast model IDs (#6774) Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy files belong to #6620, not this PR); keeps only the author's changes. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * fix(providers): ensure DeepSeek Web SSE emits [DONE] after FINISHED (#6791) * fix(providers): ensure DeepSeek Web SSE emits [DONE] after FINISHED Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy files belong to #6620, not this PR); keeps only the author's changes. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * refactor(deepseek): extract done-terminator helper to keep frozen file under cap Extracts the FINISHED-drain scheduler and finish-once guard added for the [DONE] terminator fix (#6777) into a new deepseek-web-done-terminator.ts module, so deepseek-web.ts stays under its frozen line cap (1148). Behavior is unchanged. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> --------- Co-authored-by: Pitchfork-and-Torch <Pitchfork-and-Torch@users.noreply.github.com> Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * feat(models): add capability override UI (#6727) * feat(models): add capability override UI Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy files belong to #6620, not this PR); renumbered the migration 118 -> 119 to resolve the collision with 118_provider_param_filters.sql already on release/v3.8.47; the author's i18n/localDb deltas were re-applied cleanly onto the release tip. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * fix(6727): import model-capability-overrides DB fns directly (not via localDb barrel) to keep localDb under file-size cap; aligns with anti-barrel convention * chore(db): satisfy known-symbols contract for modelCapabilityOverrides Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> --------- Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com> * fix(cursor): use Agent CLI build id for x-cursor-client-version (#6795) * fix(cursor): use Agent CLI build id for x-cursor-client-version Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy files belong to #6620, not this PR); the author's .env.example/docs deltas were re-applied cleanly onto the release tip. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * chore(changelog): re-sync CHANGELOG.md to release tip (restore #6701 bullet) Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> --------- Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * fix(startup): rename reasoningControls.ts to avoid webpack casing collision (#6584) (#6718) * fix(startup): rename reasoningControls.ts to avoid webpack casing collision (#6584) * chore(6718): re-sync onto release tip; CHANGELOG entry → changelog.d fragment (fragments-first) * fix(build): suppress Turbopack over-bundling warning from agentSkills generator (#6582) (#6720) * fix(build): suppress Turbopack over-bundling warning from agentSkills generator (#6582) generator.ts builds outputBase from a non-literal outputDir parameter, so Turbopack's file-tracing analyzer can't narrow it and emits an "Overly broad patterns" warning per entry point that imports the module (603 warnings on v3.8.46, up from 379). The fs access is legitimate and bounded, so next.config.mjs now suppresses this specific diagnostic via turbopack.ignoreIssue, mirroring the existing webpack.ignoreWarnings precedent in the same file. * chore(6720): re-sync onto release tip; CHANGELOG entry → changelog.d fragment (fragments-first) * fix(providers): drop image_generation for Codex Spark models regardless of plan (#6651) (#6721) * fix(providers): drop image_generation for Codex Spark models regardless of plan (#6651) * chore(6721): re-sync onto release tip; CHANGELOG entry → changelog.d fragment (fragments-first) * fix(providers): stop quota card re-sorting Codex/GLM bars by remaining % (#6687) (#6722) * fix(providers): stop quota card re-sorting Codex/GLM bars by remaining % (#6687) QuotaCardExpanded.tsx unconditionally re-sorted quotas by remaining percentage via sortQuotasByRemaining(), discarding the deterministic CODEX_QUOTA_ORDER/GLM_QUOTA_ORDER window order quotaParsing.ts's sortCodexOrder()/sortGlmOrder() had already established. A new hasFixedQuotaOrder() + resolveQuotaDisplayOrder() skip the re-sort for providers with a fixed window order (codex, glm family), threading providerId from QuotaCard.tsx through to the display layer. Regression guard: tests/unit/quota-card-expanded-fixed-order-6687.test.ts * chore(6722): re-sync onto release tip; CHANGELOG entry → changelog.d fragment (fragments-first) * fix(startup): lazy-import ioredis in rateLimiter to fix MCP ERR_MODULE_NOT_FOUND (#6559) (#6725) * fix(startup): lazy-import ioredis in rateLimiter to fix MCP ERR_MODULE_NOT_FOUND (#6559) * chore(6725): re-sync onto release tip; CHANGELOG entry → changelog.d fragment (fragments-first) * fix(resilience): resolve fp-pinned combo account back to real connection id (#6696) (#6732) * fix(resilience): resolve fp-pinned combo account back to real connection id (#6696) * chore(6732): re-sync onto release tip; CHANGELOG entry → changelog.d fragment (fragments-first) * fix(api): Responses passthrough emits event-only SSE frames after filtering commentary output (#6561) (#6735) * fix(api): Responses passthrough emits event-only SSE frames after filtering commentary output (#6561) The #6199 commentary-drop `continue;` branches in stream.ts skipped the data: line for a dropped commentary event but never cleared the already-buffered event: line for the same frame, so the next blank line flushed the stale event: line alone -- an event-only SSE frame that crashes the OpenAI Python SDK's json.loads(). Both drop sites now call clearPendingPassthroughEvent() before continue. The commentary-drop decision was extracted into a new responsesCommentaryDrop.ts module so the fix does not grow the frozen stream.ts. * chore(6735): re-sync onto release tip; CHANGELOG entry → changelog.d fragment (fragments-first) * fix(api): emit reasoning_content on claude-web + v0-vercel-web SSE (#6662) (#6743) * fix(api): emit reasoning_content on claude-web + v0-vercel-web /v1/chat/completions SSE (#6662) * chore(6743): re-sync onto release tip; CHANGELOG entry → changelog.d fragment (fragments-first) * fix(sse): unwrap bare {function:{…}} tools in openai→claude translation (#6704) * fix(sse): unwrap bare {function:{…}} tools in openai→claude translation Some OpenAI-shape clients send a tool as a bare `{ function: {...} }` object, omitting the spec-required `type: "function"` parent wrapper. The tools-mapping in openai-to-claude.ts (~line 366) only unwrapped `tool.function` when `tool.type === "function"` was ALSO true, so a bare-function tool fell through to `toolData = tool` (the wrapper itself, with no `.name`), producing an empty `originalName` and silently dropping the tool from the translated request — worse than a 400, since the caller has no signal the tool never made it upstream. Unwrap `tool.function` whenever present, independent of the parent `type` field. Regression guard: tests/unit/openai-to-claude-bare-tool.test.ts. Co-authored-by: Samir Abis <me@samirabis.com> Inspired-by: https://github.com/decolua/9router/pull/2473 * chore(6704): re-sync onto release tip; CHANGELOG → changelog.d fragment (fragments-first) --------- Co-authored-by: Samir Abis <me@samirabis.com> * fix(oauth): avoid bare-email dedup of Codex OAuth logins (#6706) * fix(oauth): avoid bare-email dedup of Codex OAuth logins When an incoming Codex OAuth connection has no verifiable workspace/account id, do not merge it into an existing row on email match alone — that silently overwrote the other account's token pair. Require a matching chatgptUserId (a stable per-account JWT id) before merging; otherwise insert a distinct connection row. Co-authored-by: lucasjustinudin <34107354+lucasjustinudin@users.noreply.github.com> Inspired-by: https://github.com/decolua/9router/pull/2477 * chore(6706): re-sync onto release tip; CHANGELOG → changelog.d fragment (fragments-first) --------- Co-authored-by: lucasjustinudin <34107354+lucasjustinudin@users.noreply.github.com> * fix(sse): skip thinkingConfig for gemma models in openai→gemini translation (#6708) open-sse/translator/request/claude-to-gemini.ts already guards against sending thinkingConfig for gemma-4-* models (Gemma doesn't support it — Vertex returns 400: "Thinking budget is not supported for this model"), but the OpenAI-shape path (openai-to-gemini.ts) lacked the same guard, so OpenAI-shape clients hitting a vertex gemma-4-* model still got a 400. Mirrors the existing claude-to-gemini.ts guard: wrap the reasoning_effort and Claude-shape thinking.budget_tokens branches with a model.startsWith ("gemma-4") check. Branch 3 (default includeThoughts for modern Gemini models) already excludes non-"gemini" model ids and needed no change. Inspired-by: https://github.com/decolua/9router/pull/2480 Co-authored-by: chy1211 <31048289+chy1211@users.noreply.github.com> * fix(codex): surface capacity errors embedded in 200-OK SSE streams (#6710) * fix(codex): surface capacity errors embedded in 200-OK SSE streams Codex sometimes answers with HTTP 200 and a text/event-stream body whose payload carries a transient error mid-stream (e.g. "Selected model is at capacity...", server_is_overloaded, service_unavailable_error). Because the outer HTTP status was 200, this looked like a successful response to every caller — no retry, no circuit breaker, and no combo/account fallback ever engaged, so a healthy account sat idle while the request silently failed or truncated. Add peekCodexSseTransientError() to open-sse/executors/codex.ts: it peeks the first bytes of a text/event-stream Codex response, pattern-matches the known transient-error signatures, and converts a match into a real 503 Response via errorResponse() (Hard Rule #12 — sanitized, never raw upstream text). A 503 is already a recognized provider-failure status in accountFallback.ts, so combo routing and connection cooldown pick it up automatically. When no error signature is found, the peeked prefix is prepended back onto the remaining upstream body so the passthrough stays byte-identical to the unmodified response. Regression guard: tests/unit/codex-sse-capacity-fallback.test.ts — a model-at-capacity payload and a server_is_overloaded/service_unavailable_error payload both convert to 503; a normal single-chunk SSE stream and one split across multiple network chunks both reassemble byte-for-byte unchanged. Inspired-by: https://github.com/decolua/9router/pull/2452 (sub-bug #3 only — OmniRoute already covers PR #2452's other two sub-bugs: service_tier "fast" normalization and reasoning_effort "max" normalization). Co-authored-by: ryanngit <74137224+ryanngit@users.noreply.github.com> * chore(6710): re-sync onto release tip; CHANGELOG → changelog.d fragment (fragments-first) --------- Co-authored-by: ryanngit <74137224+ryanngit@users.noreply.github.com> * fix(volcengine): clamp Kimi max_tokens to Ark endpoint cap (#6712) * fix(volcengine): clamp Kimi max_tokens to Ark endpoint cap VolcEngine Ark's Kimi coding-plan endpoint (ark.cn-beijing.volces.com) enforces max_tokens <= 32768 server-side and returns 400 "integer above maximum value, expected a value <= 32768" for anything over that ceiling. OmniRoute's StripRule only supported dropping params outright, with no numeric clamp mechanism, so a client sending a larger max_tokens (common default, e.g. 65536) 400s outright against volcengine's kimi-k2-5-260127. The 32768 cap is independently confirmed against two live-endpoint bug reports hitting this exact Ark endpoint for both kimi-k2.5 and kimi-k2.7-code (NousResearch/hermes-agent#51773, MoonshotAI/kimi-cli#1124), not just upstream's own value — same cap upstream 9router#2460 uses. StripRule gains two optional fields: `clampToModelMaxOutput` (clamp to the model's own catalog maxOutputTokens ceiling, when set) and `maxOutputCap` (a fixed endpoint-imposed ceiling); when both apply, the lower wins. The new rule is scoped to the literal id `kimi-k2-5-260127` (OmniRoute's real volcengine Kimi model, not upstream's `Kimi-K2.7-Code`), not a broad /kimi/i regex, so it can never clamp an unrelated future Kimi listing whose Ark cap may differ. glm-4-7-251222 (the other volcengine model) is unaffected. Inspired-by: https://github.com/decolua/9router/pull/2460 Co-authored-by: whale9820 <whale9820@users.noreply.github.com> * chore(6712): re-sync onto release tip; CHANGELOG → changelog.d fragment (fragments-first) --------- Co-authored-by: whale9820 <whale9820@users.noreply.github.com> * fix(antigravity): surface aborted Gemini tool calls off end_turn (#6713) * fix(antigravity): surface aborted Gemini tool calls off end_turn Gemini/Antigravity aborts a turn with finishReason MALFORMED_FUNCTION_CALL (or a sibling like UNEXPECTED_TOOL_CALL) instead of completing cleanly. Both Claude-facing translators collapsed these to a clean end_turn, hiding the aborted tool call as a successful completion: - the OpenAI hub path (openai-to-claude.ts convertFinishReason default), and - the DIRECT Gemini->Claude path (gemini-to-claude.ts), which is the one Claude Code actually hits through an antigravity/Gemini-routed model. Add isAbortFinishReason() to finishReason.ts and map these reasons to tool_use on both paths; genuinely unknown reasons still fall back to end_turn. Co-authored-by: anhdiepmmk <n08ni.dieppn@gmail.com> Inspired-by: https://github.com/decolua/9router/pull/2462 * chore(6713): re-sync onto release tip; CHANGELOG → changelog.d fragment (fragments-first) --------- Co-authored-by: anhdiepmmk <n08ni.dieppn@gmail.com> * fix(translator): strip empty cloud_base_branch from Cursor Subagent tool call (#6729) * fix(translator): strip empty cloud_base_branch from Cursor Subagent tool call (port from 9router#2446) The Responses->Chat tool-arg cleanup (stripEmptyOptionalToolArgs) only stripped empty-string/empty-array optional args for Claude Code's Read tool. Cursor's local Subagent tool call therefore passed through with the cloud-only field cloud_base_branch: "", which Cursor rejects ("cloud_base_branch may only be specified when environment equals cloud") before starting the subagent. Extend the cleanup to an allowlist of Read + Subagent; arbitrary tools stay untouched. Reported-by: like3213934360-lab (https://github.com/decolua/9router/issues/2446) * chore(6729): re-sync onto release tip; CHANGELOG → changelog.d fragment (fragments-first) * fix(translator): defer content_block_start until GLM streams the tool name (#6730) * fix(translator): defer content_block_start until GLM streams the tool name (port from 9router#2077) GLM 5.2 (and similar OpenAI-compatible upstreams) stream a tool call's id and function.name across separate SSE delta chunks. The openai-to-claude streaming translator emitted content_block_start immediately on the id-only chunk with an empty name; the Claude SSE protocol cannot patch a block after emission, so the later name-only chunk was dropped and Claude Code rejected the tool_use with an empty tool name / "No such tool available:". Defer content_block_start until the name arrives (start on args if they arrive first), and emit a start for any orphaned id-only tool call at finish so content_block_stop is never orphaned. Reported-by: itiwant (https://github.com/decolua/9router/issues/2077) * chore(6730): re-sync onto release tip; CHANGELOG → changelog.d fragment (fragments-first) * feat(dashboard): add search to Playground model picker dropdown (#4086) (#6811) * feat(dashboard): add search to Playground model picker dropdown (#4086) The shared ModelSelectModal (combo builder + CLI-code cards) already had search, but the Playground's raw model <select> in StudioConfigPane stayed a flat unsearchable list - unusable once a provider like OpenRouter contributed 50+ models. Adds a search input above the dropdown that filters options via filterModelsByQuery() (Turkish-safe accent/case-insensitive match, reusing matchesSearch()). The currently selected model always stays pinned in the list even when it doesn't match the query, so typing never silently swaps the active selection. Reuses the existing common.search i18n key already translated in all 42 locales - no new key needed. * chore(6811): re-sync onto release tip; CHANGELOG → changelog.d fragment (fragments-first) * feat: request count log per provider, per date (#4009) (#6812) * feat(dashboard): request count log per provider, per date (#4009) Some providers bill by request rather than by token, so operators need a plain per-provider, per-date request count breakdown, not just token aggregates. Adds a new getProviderDailyUsageRows() aggregation query (src/lib/db/usageAnalytics.ts), a dedicated GET /api/usage/requests-by-provider-date route (kept separate from the frozen /api/usage/analytics route to respect the file-size baseline), and a sortable, single-date-filterable table on Dashboard -> Analytics. Closes #4009 * chore(6812): re-sync onto release tip; CHANGELOG → changelog.d fragment (fragments-first) * feat(xai): route xAI clients to Grok native /v1/responses endpoint (#6709) * feat(xai): route xAI clients to Grok native /v1/responses endpoint xAI ships a native /v1/responses endpoint (https://api.x.ai/v1/responses) alongside /v1/chat/completions, but XaiExecutor extended BaseExecutor without overriding buildUrl(), so every request always resolved to the static chat-completions baseUrl regardless of target format — the last genuinely-missing slice of decolua/9router#2439 (grok-build-0.1, the reasoning-effort suffix routing, and bare grok-* routing were already ported in prior cycles). Add responsesBaseUrl to the xai registry entry and tag grok-4.20-multi-agent-0309 (upstream's own Responses-only id) with targetFormat: "openai-responses", mirroring the existing model-tag-driven routing pattern already used by the gh executor (9router#102) and the "openai" -pro heuristic in open-sse/executors/default.ts — the per-model registry tag is the single source of truth that also drives chatCore's body translation, so URL and body stay in lockstep. XaiExecutor.buildUrl now checks getModelTargetFormat("xai", model) and resolves to the native Responses endpoint only for tagged models, leaving every other grok-* model on the existing chat-completions bridge. TDD: tests/unit/executor-xai.test.ts adds a RED-then-GREEN case asserting grok-4.20-multi-agent-0309 resolves to https://api.x.ai/v1/responses and a control case asserting grok-4.3 still resolves to https://api.x.ai/v1/chat/completions. Co-authored-by: ryanngit <74137224+ryanngit@users.noreply.github.com> Inspired-by: https://github.com/decolua/9router/pull/2439 * chore(6709): re-sync onto release tip; CHANGELOG → changelog.d fragment (fragments-first) --------- Co-authored-by: ryanngit <74137224+ryanngit@users.noreply.github.com> * fix(resilience): route remaining credential-selection call sites through quota preflight (#6686) (#6742) * fix(resilience): route remaining credential-selection call sites through quota preflight (#6686) * chore(6742): re-sync onto release tip; CHANGELOG entry → changelog.d fragment (fragments-first) * fix(resilience): apikey-provider 429s honor explicit quota-exhausted text (#6638) (#6731) * fix(resilience): apikey-provider 429s honor explicit quota-exhausted text (#6638) Ollama Cloud (and any other apikey-category provider) 429s skipped body-text quota classification entirely; a genuine multi-day quota exhaustion was misclassified as a plain rate_limit_exceeded with a few seconds of cooldown, so combo routing retried the account immediately. shouldPreserveQuotaSignals() now lets an explicit quota-exhausted signal (looksLikeQuotaExhausted) override the apikey-category default, and parseDayGranularityResetMs() adds day- granularity reset-hint parsing ("...reset in 3 days.") alongside the existing Xh/Ym/Zs parsing. Regression guard: tests/unit/issue-6638-ollama-quota.test.ts (RED before the fix, GREEN after). Aligned two tests/unit/account-fallback-service.test.ts cases that had codified the old buggy behavior for apikey-provider quota text. * chore(6731): re-sync onto release tip; CHANGELOG entry → changelog.d fragment (fragments-first) * feat(resilience): weekly-429 cooldown for fetcher-less providers (#3709) (#6817) * feat(resilience): weekly-429 cooldown for fetcher-less providers (#3709) Ollama Cloud free-tier accounts have a hard WEEKLY request cap. On cap the upstream returns 429 "you (<account>) have reached your weekly usage limit", but ollama-cloud is an apikey-category provider, so the existing oauth-only shouldUseQuotaSignal gate in checkFallbackError skips the subscription-quota-text classifier (Issue #2321) for its 429s -- the account fell through to the generic exponential backoff (~1s, capped at 2min) and got retried every few minutes for the rest of the week (one account took 285x429 in 48h). Adds a new, ungated weekly-usage-limit text classifier that applies a 24h QUOTA_EXHAUSTED cooldown regardless of provider category. Extracted the new classifier -- together with the existing #2321 subscription-quota logic -- into a new open-sse/services/quotaTextCooldowns.ts module so the frozen accountFallback.ts (file-size-baseline cap) didn't have to grow; net effect shrinks accountFallback.ts by 20 lines. This is Phase A of the plan (open-sse/services/accountFallback.ts:1038-1045 "weekly-429 cooldown"); Phase B (generic local request-counter preflight for manual provider_plans dimensions) is a separate, larger follow-up per the plan's own phasing. * chore(6817): re-sync onto release tip; CHANGELOG entry → changelog.d fragment (fragments-first) * fix(providers): Kiro adaptive-thinking allowlist excludes sonnet-4.5/haiku-4.5 (#6576) (#6726) * fix(providers): Kiro adaptive-thinking allowlist excludes sonnet-4.5/haiku-4.5 (#6576) * chore(6726): re-sync onto release tip; CHANGELOG entry → changelog.d fragment (fragments-first) * test(kiro): migrate selector-strip test to claude-sonnet-5 (only Kiro adaptive-thinking model, #6576) Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * chore(quality): rebaseline complexity 2053->2054 (merge-burst drift, v3.8.47) Inherited drift from today's /implement-prs merge burst (~36 PRs). check:complexity does not run on the PR->release fast-path, so the branch accrued +1 unmeasured. No orphan/feature PR introduces a NEW violation (complexity-net-zero); the only flagged function is the pre-existing getResolvedModelCapabilities. Owner-approved rebaseline to unblock the FQG of ~7 green-except-complexity orphans. * chore(stryker): register ollama-quota covering tests (merge-burst drift, v3.8.47) The 3 covering unit tests from #6731/#6817/#6742 (issue-6638-ollama-quota, ollama-cloud-weekly-quota-cooldown-3709, issue-6686-quota-preflight-coverage) exist on release but were never added to tap.testFiles when those PRs merged. Completes the registration so mutant kills count; unblocks every PR touching a mutated module. Part of the owner-approved merge-burst drift cleanup. * fix: auto-start WS server in-process and change default port to 20132 (#6072) * feat: change default LIVE_WS_PORT from 20129 to 20132 Update the default WebSocket port for the live dashboard server from 20129 to 20132 across all configuration files, documentation, code comments, and tests. Also consolidate OMNIROUTE_DISABLE_LIVE_WS and OMNIROUTE_ENABLE_LIVE_WS into a single OMNIROUTE_ENABLE_LIVE_WS flag. Wire the live WebSocket server to start in-process via instrumentation-node.ts. * feat: clarify NEXT_PUBLIC_LIVE_WS_PUBLIC_URL path usage and derive upgrade path from URL Update .env.example and ENVIRONMENT.md to document that the pathname portion of NEXT_PUBLIC_LIVE_WS_PUBLIC_URL (e.g. /live-ws) is used as the WebSocket upgrade path by the dev proxy, handshake response, and client connection logic. Extract deriveLiveWsPath() into shared/utils/wsPath.ts and wire it through: - src/app/api/v1/ws/route.ts — handshake response path field - src/hooks/useLiveDashboard.ts — build * fix: use the standard URL API to safely parse and update the effectiveWsUrl * build(docker): expose live WebSocket server port and configure CORS origins Add LIVE_WS_PORT (20132), LIVE_WS_HOST (0.0.0.0), and LIVE_WS_ALLOWED_ORIGINS environment variables to all Docker Compose profiles and expose the WebSocket port mapping. Prevent infinite self-loop in standalone-server-ws.mjs by skipping proxy when the server itself is running on the LiveWS port. * docs(env): fix comment formatting for HOST and HOSTNAME variables --------- Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * fix(logs): prevent stale detail refresh reopening modal (#6323) * fix(logs): prevent stale detail refresh reopening modal * chore(stryker): register ollama-quota covering tests (release drift from merge burst) Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> --------- Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com> Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * \ feat: operator-configurable account rotation\ (#6763) * feat(resilience): operator-configurable account rotation Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy files belong to #6620, not this PR); the author's accountFallback/.env deltas were re-applied cleanly onto the release tip. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * docs(env): document configurable account-rotation env vars in ENVIRONMENT.md Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * refactor(rotation): extract rotation gate/context helpers to keep accountFallback.ts under frozen cap Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * chore(changelog): re-sync CHANGELOG.md to release tip (restore lost base bullet) Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * test(stryker): register rotation-config test in tap.testFiles for mutation coverage Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * test(stryker): register ollama-quota covering tests (drift from #6731/#6817/#6742) + re-sync Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> --------- Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com> * fix(lmarena): modernize Arena web provider + static Direct-chat catalog (#6280) * fix(lmarena): modernize Arena web provider + static Direct-chat catalog Update the lmarena provider for arena.ai (product rebranded from LMArena): - Route chat via arena.ai create-evaluation with Chrome TLS impersonation (tls-client-node) and optional browser-minted recaptchaV3Token. - Seed Text+Search (48) into the chat registry; seed Image (27) only into IMAGE_PROVIDERS. Disable live HTML model discovery; resolve public names to Arena UUIDs from the static TypeScript allowlist (no scrape JSON in-repo). - Soft-exclude 404/502 model ids; slow/stop bulk test-all probes for this provider. - Do not fold IMAGE_PROVIDERS/video specialty into the chat provider catalog when a chat registry already exists (lmarena/openai/xai). - Display name Arena (Free); keep wire id `lmarena` / alias `lma` for back-compat. - Theme-aware provider icons: arena-light.svg / arena-dark.svg. - Preserve split Supabase SSR cookie reconstruction for arena-auth-prod-v1.*. * fix(providers): align provider-models-route test fixture + regen provider reference Fold the topaz image-only catalog entry's apiFormat/supportedEndpoints into the local-catalog test fixture (route now tags media-only providers per the lmarena PR's staticModels.ts change), regenerate PROVIDER_REFERENCE.md against the merged release providers.ts, and add the changelog fragment for #6280. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * test: align web-cookie fallback suite — lmarena now has a registry entry (probe path) Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> --------- Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * docs(changelog): reconcile 3-day merge burst — 16 fragments, 4 promised credits, contributors hall 32→63 - changelog.d fragments for the 20 merged PRs that landed without a bullet (#6072 #6308 #6323 #6538 #6556 #6586 #6611 #6647 #6675 #6698 #6757 #6759 #6804 #6821 + ci rollup #6781/#6691/#6693 + docs rollup #6643/#6644/#6646/#6663; omniglyph bump #6661 folded into the #6556 bullet) - deliver the 4 credits promised in close comments but never written: @alltomatos (#6819 dup of #6721), @samimozcan (#6762/#6753 subsumed by #6790), @chirag127 (#6756 dup of #6757), @Squawk7777 (#6565 dup of #6564 — appended to the existing #6564 bullet; changelog-integrity flags that edit as a removal, intentional: ALLOW_CHANGELOG_REMOVALS justification) - rebuild the v3.8.47 Contributors hall from merged-PR authors + thanks credits + prior hall: 32 → 63 contributors * Clamp reasoning token buffer to model output cap (#6714) * fix(combo): clamp reasoning buffer to model output cap * fix(routing): preserve near-cap reasoning max tokens * fix(routing): getExplicitModelOutputCap falls through to registry cap on non-numeric synced limit_output getExplicitModelOutputCap short-circuited to null whenever a synced capability row existed, even if that row's limit_output was not a number (models.dev commonly omits it). That silently disabled the reasoning-token buffer clamp for any model with a synced row lacking an output limit. Now only return the synced value when it IS a number; otherwise fall through to registryModel.maxOutputTokens / spec.maxOutputTokens, matching the ??-chain precedence already used by getResolvedModelCapabilities(). Adds a standalone regression test (proves the fallthrough returns the real registry cap, not null) and hardens the #6274 fixture id so its no-output-cap case does not prefix-match the real glm-5.2 static spec. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * chore(stryker): register ollama-quota covering tests (release drift from merge burst) Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> --------- Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com> Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * feat(i18n): add Traditional Chinese (zh-TW) localization for frontend and CLI (#6320) * feat(i18n): add Traditional Chinese (zh-TW) localization for frontend and CLI - Add src/i18n/messages/zh-TW.json translating frontend web UI - Add bin/cli/locales/zh-TW.json translating CLI commands and descriptors - Register zh-TW in config/i18n.json and docs/guides/I18N.md - Update scripts/i18n/generate-multilang.mjs matching the new locale setup * fix: update i18n locale count from 42 to 43 after adding zh-TW The docs strict checker (check-docs-counts-sync.mjs) validates that README.md and I18N.md reflect the real locale count. Adding zh-TW bumped the count from 42 → 43. * fix(i18n): translate providers free-filter labels in zh-TW (#6694 guard) Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> --------- Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com> Co-authored-by: Diego Rodrigues de Sa e Souza <8016841+diegosouzapw@users.noreply.github.com> Co-authored-by: lunkerchen <lunkerchen@users.noreply.github.com> * feat(proxy): implement latency-optimized proxy rotation strategy (#6798) * feat(proxy): implement latency-optimized proxy rotation strategy Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy files belong to #6620, not this PR) and the direct CHANGELOG.md edit (fragments-first); the author's env/docs/i18n deltas were re-applied cleanly onto the release tip. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * fix(proxy): add latency-rotation env var to .env.example PROXY_LATENCY_WINDOW_HOURS was referenced in src/lib/db/proxies.ts and documented in docs/reference/ENVIRONMENT.md, but missing from .env.example, tripping the env/docs sync gate. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * chore(changelog): re-sync CHANGELOG.md to release tip Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * refactor(proxy): extract latency-strategy helpers to keep frozen files under cap Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * test(db-rules): expect 35 audited modules (proxyLatency joins INTENTIONALLY_INTERNAL) Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> --------- Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * docs(readme): fix stale strategy/tool/scoring counts (#6853) README still claimed 17 routing strategies (the table was missing pipeline), 95 MCP tools, and 9-factor Auto-Combo scoring. Align with the source (ROUTING_STRATEGY_VALUES has 18 entries) and the canonical docs (MCP-SERVER.md: 94 tools; AUTO-COMBO.md: 12-factor). * fix(antigravity): sanitize Cloud Code safety settings (#6839) Co-authored-by: kfiramar <83420275+kfiramar@users.noreply.github.com> * fix(kiro): probe IdC region during profileArn discovery, cross-region (recovers #6099) (#6840) * fix(kiro): route Amazon Q runtime by profileArn region for cross-region IdC Enterprise AWS IAM Identity Center accounts whose IdC instance lives outside the two Amazon Q Developer profile regions (us-east-1 / eu-central-1) - e.g. eu-north-1 (Stockholm), start URL https://d-XXXX.awsapps.com/start - showed no limits and returned 502 on every request. Root cause: the backend used the IdC/OIDC token region (providerSpecificData.region, e.g. eu-north-1) for every CodeWhisperer runtime call, hitting q.eu-north-1.amazonaws.com - a host that does not exist as a Q Developer runtime endpoint. Per AWS docs ("Supported Regions for the Q Developer console and Q Developer profile"), the Q Developer *profile* (which produces the profileArn and hosts generateAssistantResponse / GetUsageLimits / ListAvailableModels / ListAvailableProfiles) is only hosted in us-east-1 and eu-central-1, regardless of the IdC region; "data is stored in the Region where you create the Amazon Q Developer profile." Fix (new open-sse/services/kiroRegion.ts) decouples the two regions: - providerSpecificData.region stays the IdC/OIDC region, used ONLY for oidc.{region}.amazonaws.com token mint/refresh. - The runtime region is derived from the profileArn (resolveKiroRuntimeRegion): profileArn region -> a valid stored profile region -> us-east-1. A stored IdC region that is not a Q profile region (eu-north-1) is ignored for runtime. - Profile discovery (discoverKiroProfileArnAcrossRegions) probes the Q profile regions (EU IdC -> eu-central-1 first) with the cross-region SSO token instead of q.{idcRegion}. Wired into: executors/kiro.ts (generateAssistantResponse targets the profile region), services/usage/kiro.ts (getKiroUsage multi-region discovery + profileArn runtime region so Limits resolves), services/kiroModels.ts (ListAvailableModels), and src/lib/oauth/providers/kiro.ts (login-time postExchange profile discovery). Adds tests/unit/kiro-idc-cross-region.test.ts (15 cases). All Kiro suites pass (60 tests). * fix(kiro): probe the IdC region too during profileArn discovery (any IdC region) Make profile discovery general for an IdC in ANY of the ~30 IdC-supported AWS regions (us-west-2, ap-southeast-2, me-central-1, af-south-1, ...), not just eu-north-1. buildKiroProfileDiscoveryRegions now probes the two documented Q Developer profile regions FIRST (us-east-1 / eu-central-1, EU-first for EMEA IdC regions to cut latency), then appends the IdC/stored region itself as a forward-compatible fallback: if AWS ever co-locates the profile with the IdC or expands the profile-region list, a same-region probe still finds it. Probing a region with no profile simply returns nothing and we fall through. The profileArn's own region remains authoritative for every runtime call (resolveKiroRuntimeRegion), so a newly-issued ARN in any region is honored automatically. Adds ap-southeast-2 (APAC) cross-region coverage and updates the discovery-order tests. --------- Co-authored-by: artickc <artur1992123@mail.ru> * feat(providers): manual context-window override for custom models (#4125) (#6822) Add a manual per-model "Context Window Override" so an operator can correct a provider's misreported context length (e.g. reports 1M when the real limit is 128K) instead of the model getting silently dropped from combo routing once the wrong value lands in the catalog. Reuses the existing Feature-5004 model_context_overrides table (source="manual") — already the priority-0 source getModelContextLimit() (the function combo's context-window filter calls) reads ahead of the models.dev/registry/static catalog — so no new resolver logic was needed, only the missing write path: - PUT /api/provider-models now accepts an optional contextWindowOverride (number to set, null to clear), persisted via setModelContextOverride/ removeModelContextOverride. - GET /api/provider-models surfaces the current override value + source back on each custom-model row. - CustomModelsSection.tsx: edit form gained a Context Window Override input + a badge on the model row when an override is set. Regression guard: tests/unit/provider-models-context-window-override-4125.test.ts (manual override wins over a misreported catalog value, GET round-trip, clearing via null, default-unchanged behavior). * feat(dashboard): improve Provider Quota page horizontal density (#3520) (#6815) QuotaCardGrid stacked every provider group vertically in a single flex flex-col container, and each group's own card grid didn't go multi-column until the md breakpoint. Provider groups now flow into a 2-column CSS multi-column layout on very wide (2xl) screens instead of an unconditional vertical stack, and each group's card grid starts at 2 columns immediately, filling horizontal whitespace sooner on narrower-but-not-mobile viewports. Regression guard: tests/unit/quota-card-grid-horizontal-layout.test.ts * refactor(usage): type saveRequestUsage with UsageEntry interface + any-budget ratchet (#3512) (#6809) Replace saveRequestUsage(entry: any) with a typed UsageEntry interface mirroring the usage_history columns 1:1. Fields stay optional/nullable since different writers (chatCore success/failure, rejected-request accounting, Codex Responses WS) populate the row incrementally; tokens stays unknown since callers pass either raw provider-shaped usage or the normalized {input,output,cacheRead,...} shape. Also cleaned the file's other any usages (getUsageHistory filter, getUsageDb next-cursor cast, appendRequestLog tokens param, getRecentLogs catch) so it now sits at zero any and can be added to the check:any-budget:t11 zero-any allowlist. Documents the DB-entity <-> TS-interface convention in docs/architecture/CODEBASE_DOCUMENTATION.md Sec 11. * feat(combo): strict budget-cap fallback policy for auto/* combos (#3470) (#6816) Auto-combo transparency + budget controls: the engine's budgetCap enforcement always degraded to the globally cheapest candidate when every candidate exceeded the cap - silently overspending instead of respecting the cap. - engine.ts: budgetFallback "cheapest" (default, legacy) | "strict" (BudgetExceededError when no candidate fits budgetCap) - requestControls.ts: X-OmniRoute-Budget-Fallback header + resolveRequestAutoControls() consolidating mode/budget/fallback parsing - resolveAutoStrategy.ts / autoConfig.ts: thread combo-level config.budgetFallback and catch BudgetExceededError into an HTTP 402 - chat.ts: switch to the consolidated resolveRequestAutoControls() helper (net line reduction, stays under the frozen file-size baseline) Regression guard: tests/unit/auto-combo-budget-fallback-3470.test.ts * fix(usage): honor xAI provider-reported exact cost (#6711) OmniRoute's calculateCost() always estimated request cost from token counts x static pricing, discarding xAI's exact provider-reported cost when present. xAI's chat-completions usage object reports the precise billed cost via cost_in_usd_ticks (docs.x.ai/developers/cost-tracking and the API reference's usage schema: "TICKS_IN_USD_CENT: i64 = 100_000_000" => 1e10 ticks/USD, e.g. 37756000 ticks ~= $0.0038). calculateCost()/computeCostFromPricing() now short-circuit to this exact figure when present -- before any pricing DB lookup, so it also works for models without a local pricing row -- and still fall back to the token-based estimate when it is absent. The field is threaded through both the streaming (extractUsage/normalizeUsage) and non-streaming (extractUsageFromResponse) usage-extraction paths. Corrected divisor vs upstream: the upstream PR used /1e12 (a 100x under-report, e.g. reporting $0.00123 as the doc's $0.123 example); this port uses the doc-verified /1e10 instead, confirmed against both the cost-tracking guide and the API reference's usage-object schema. Inspired-by: https://github.com/decolua/9router/pull/2453 Co-authored-by: ryanngit <74137224+ryanngit@users.noreply.github.com> * docs: rename /implement-prs → /merge-prs in Hard Rule #21 (skill renamed 2026-07-11) (#6847) * docs: refresh stale llm.txt facts + relocate design.md to docs/architecture/DESIGN_SYSTEM.md (#6849) * docs: refresh stale llm.txt facts + move design.md to docs/architecture/DESIGN_SYSTEM.md llm.txt was frozen at the v3.8.8 era (177 providers, 37 MCP tools, 14 strategies, 9-factor scoring, 75% coverage gate). Update every factual claim to the current state (248 providers, 94 tools / 30 scopes, 18 strategies, 12-factor scoring, ratchet + 60% floor, TS 6, current docs/ layout) and re-sync the 42 exact-copy i18n mirrors. design.md at the root was a standardization plan whose phases 1-6 all shipped; rewrite its header as a permanent reference and relocate it to docs/architecture/DESIGN_SYSTEM.md per the root-hygiene policy (root = configs + canonical docs only). * docs: add MDX frontmatter to DESIGN_SYSTEM.md (in-app docs pipeline requires it) * feat: per-model web-search interception rule (#3384) (#6814) * feat(routing): per-model web-search interception rule (#3384) Adds a per-provider/per-model interceptSearch rule (src/lib/db/interceptionRules.ts, key_value namespace interception_rules) that overrides the existing native web-search bypass defaults (Codex/Gemini/Claude->Claude passthrough) in webSearchFallback.ts. Wired at the existing prepareWebSearchFallbackBody() call site in chatCore.ts. Resolution precedence: per-model rule > provider-level rule > existing native-bypass defaults. This lands Phase 1-2 of the plan (rule store + search interception). Web-fetch interception and the dashboard UI toggle are tracked as follow-up phases. * fix(db): register interceptionRules in localDb re-export layer (db-rules gate) * fix(db): renumber interception_rules migration 119→120 (collision with model_capability_overrides) * feat: sidebar search/filter input (#4013) (#6810) * feat(dashboard): add search/filter input to the dashboard sidebar (#4013) Adds a search box at the top of the expanded sidebar that filters nav sections/groups/items client-side by label, so users don't have to hunt through the growing nav tree. Reuses the existing common.search / common.noResults i18n keys (no new locale edits needed) and the shared Input icon="search" pattern. Matching sections auto-expand while searching and the accordion/pin state is restored once the query is cleared. Filtering logic is extracted into a pure filterSidebarSectionsByQuery() helper (src/shared/utils/sidebarSearch.ts) so it is trivially unit testable independent of React/next-intl/next-navigation. * fix(test): move Sidebar.search test to a runner-collected path (test-discovery gate) * fix(i18n): backfill 194 missing pt-BR keys (#6695) (#6723) * fix(i18n): backfill 194 missing pt-BR keys and add key-parity regression test (#6695) * Merge branch 'release/v3.8.47' into fix/6695-i18n-drift Resolve i18n key-parity and CHANGELOG-fragment conflicts: - Convert the #6695 CHANGELOG.md bullet to a changelog.d/ fragment (the fragment convention landed on release/v3.8.47 after this PR branched, per changelog.d/README.md). - Backfill 61 additional pt-BR keys that entered en.json on release/v3.8.47 after this PR's original 194-key backfill, so the PR's own key-parity regression test (tests/unit/i18n-pt-br.test.ts) stays green against the moving release baseline. * Discover live Codex models (#6776) * Add live model discovery for provider catalog * Fix model discovery request headers * fix(codex): sync live model limits with local catalog * test(codex): split live model discovery coverage into dedicated route tests * fix(codex): use chatgpt account id for live model sync * Add GitHub-backed Codex model discovery fallback * fix(providers): tighten oauth config tests and provider model display comments * test: align client version expectations with release default * fix(codex): keep discovery complexity within baseline * fix: rebase live Codex model discovery onto release/v3.8.47, preserving kimi-web buildHeaders (#6308) Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> --------- Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com> Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * feat(codex): echo requested effort-suffixed model id in Responses payloads (#3697) (#6820) * feat(codex): echo requested effort-suffixed model id in Responses payloads (#3697) Codex CLI compatibility shim: the Responses API response.created/ response.in_progress/response.completed payloads now carry a `model` field (previously absent), and for Codex-CLI-originated requests it echoes the client-requested effort-suffixed model id (e.g. gpt-5.5-xhigh) instead of the bare upstream id (gpt-5.5), so the Codex CLI status line/model button shows the active reasoning effort. - openai-responses.ts translator threads the upstream model into the Responses event objects (additive, omitted when unknown). - New isCodexOriginatedHeaders() (codexIdentity.ts) reuses PR #3481's originator/User-Agent detection, header-based so it still fires when a combo routes codex/gpt-5.5-xhigh to a non-codex upstream. - chatCore's existing opt-in #1311 echoModel pipeline now also fires automatically for Codex clients on the Responses API, regardless of the echoRequestedModelName setting. - responseModelEcho.ts now also rewrites the nested response.model field the Responses API uses (previously only top-level model). - /v1/models keeps returning models: [] for Codex (unchanged, #3481). Regression guard: tests/unit/codex-effort-model-echo-3697.test.ts. Closes #3697 * chore(merge): re-sync with release/v3.8.47 (restore CHANGELOG, keep own bullet) * chore(merge): re-sync with release/v3.8.47; move changelog bullet to changelog.d fragment (merge-storm proof) * feat(usage): surface Antigravity weekly quota alongside the 5-hour window (#4017) (#6818) * feat(usage): surface Antigravity weekly quota alongside the 5-hour window (#4017) Antigravity enforces both a 5-hour and a weekly usage limit, but the agy/antigravity quota widget only exposed the 5-hour window. The weekly limit isn't in the per-model retrieveUserQuota response already fetched — it lives in a separate, undocumented retrieveUserQuotaSummary RPC that groups models into families (Gemini Models, Claude and GPT models) with one weekly bucket per family. Adds a self-contained usage/antigravityWeeklyQuota.ts leaf: a cached, best-effort fetch of that RPC + a pure parser that extracts the weekly-labeled bucket per group (window inferred from bucketId/displayName text, matching the reverse-engineered shape documented by third-party Antigravity clients) into gemini_weekly/ claude_gpt_weekly quota entries, merged into the existing quotas map the widget already renders generically. A failed/unavailable RPC never affects the existing per-model quotas. Live VPS validation attempt (192.168.0.15, real antigravity account): both retrieveUserQuota and retrieveUserQuotaSummary currently return 429 RESOURCE_EXHAUSTED for that account, so the live response shape could not be captured directly. The parser was instead validated via TDD against the bucket shape documented by CodexBar (steipete/CodexBar), a third-party Antigravity client that reverse-engineered the same RPC, and is defensive against both response envelopes it has observed (top-level groups[] and nested quotaSummary.groups[]). * chore(merge): re-sync with release/v3.8.47 (restore CHANGELOG, keep own bullet) * chore(merge): re-sync with release/v3.8.47; move changelog bullet to changelog.d fragment (merge-storm proof) * feat: add Z.ai Web free web-cookie provider (#4056) (#6823) * feat(providers): add Z.ai Web free web-cookie provider (#4056) New zai-web web-session provider drives the free chat.z.ai consumer chat UI via a pasted browser cookie, distinct from the existing API-key zai/glm/glm-cn/glmt providers (api.z.ai). ZaiWebExecutor posts to chat.z.ai/api/chat/completions with the cookie forwarded both as Cookie and Authorization: Bearer <token>, and normalizes both z.ai's internal delta_content/phase SSE envelope and a pass-through OpenAI-shaped choices[].delta frame into standard chat-completion chunks. Registered in WEB_COOKIE_PROVIDERS, WEB_SESSION_CREDENTIAL_REQUIREMENTS, the provider registry (GLM-4.6/4.5/4.5V models), the executor factory, and tokenExtractionConfig.ts for in-app cookie capture. * fix(providers): regenerate translate-path golden for zai-web + reduce cognitive complexity * fix(providers): rename ZaiWebExecutor.buildHeaders to avoid incompatible BaseExecutor override * chore(merge): re-sync with release/v3.8.47; move changelog bullet to changelog.d fragment (merge-storm proof) * fix(codex): bump default client version to 0.144.0 (#6780) Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * refactor(usage): extract per-group parsing in antigravityWeeklyQuota (cognitive-complexity gate 886→885, release-level drift from #6818 merge) * ci(quality): cut PR gate wall time without dropping protection (#6716) Collapse duplicate CI spend while keeping each gate's existence reason: - quality.yml: TIA __RUN_ALL__ defers full unit to fast-unit 4-shard (#6781); path filters via classify-pr-changes; docs-gates split; draft skip - ci.yml: wire docs/i18n/code path filters; ESLint JSON artifact for quality-gate; drop advisory typecheck:noimplicit; float actions/cache@v6 - TIA parity: memory/usage/combo/serial; **/*.test.mjs any depth; electron/bin no longer force unit __RUN_ALL__ - check:complexity-ratchets: one ESLint walk, ruleId-isolated baselines + cache - check:api-docs-refs + lib/apiRoutes: shared API route inventory - husky pre-push: intentionally light (gates live in pre-commit); CLAUDE.md + QUALITY_GATES.md docs synced - collect-metrics / lint:json: path.resolve cache path; Windows-safe eslint bin - env-doc allowlist for ESLINT_RESULTS_JSON / COMPLEXITY_ESLINT_REPORT - release-green --full-ci expects check:api-docs-refs (not docs-symbols alone) Tests: select-impacted, classify-pr-changes, api-routes lib, complexity-rule-count, validate-release-green. Reconciled after #6781 (fast-unit 2→4 shards) per maintainer request on #6716. Co-authored-by: Diego Rodrigues de Sa e Souza <8016841+diegosouzapw@users.noreply.github.com> * fix(docs): document Turbopack build memory tradeoff for RAM-constrained machines (#6409) (#6885) * fix(routing): recognize Kimi token-limit 400 as context overflow for combo fallback (#6637) (#6893) combo.ts's isContextOverflow400() guard required the literal word 'context' in the 400 error body before letting a combo fall through to the next target. Kimi's exact wording ('Your request exceeded model token limit: 262144 (requested: 308458)') never says 'context', so the guard misclassified it as a body-specific error and halted the whole combo instead of trying the next (larger-context) target. accountFallback.ts's CONTEXT_OVERFLOW_PATTERNS already recognized this wording one layer below (via checkFallbackError -> shouldFallback), so the two independently-maintained classifiers disagreed and the stricter one won. Export CONTEXT_OVERFLOW_PATTERNS from accountFallback.ts and reuse it inside combo.ts's isContextOverflow400() so both layers share a single source of truth. Regression test: tests/unit/repro-6637-kimi-token-limit.test.ts (RED on unfixed code -> GREEN after the fix). Existing #4519 guard tests (tests/unit/combo-param-validation-fallback-4519.test.ts) still pass, including the negative case that a genuinely body-specific 400 is NOT misclassified as overflow. * fix(providers): honor a provider-level proxy assigned to no-auth providers (#6272) (#6895) No-auth providers (mimocode, opencode, ...) are always dispatched with a single hardcoded connectionId ("noauth" — SYNTHETIC_NOAUTH_CONNECTION_ID in src/sse/services/auth.ts). No provider_connections row ever has id="noauth", so resolveProxyForConnection() in src/lib/db/settings.ts could never populate connectionRecord for them, and its provider-level proxy lookup (Steps 6/8) only runs when connectionRecord is present. A proxy assigned via Settings -> Providers -> mimocode was therefore silently ignored, reproducing the reporter's "same thing happen when i set the proxy directly in the provider menu" symptom. Adds a best-effort fallback (src/lib/db/settings/noAuthProxyFallback.ts): when connectionRecord could not be resolved, scan the known no-auth provider ids for a configured provider-level proxy (registry first, then legacy) before falling through to the global/direct steps. Regression test: tests/unit/proxy-noauth-provider-6272.test.ts (RED on unfixed code — resolved to level=direct/proxy=null; GREEN after the fix). * fix(dashboard): surface Claude extraUsage credits in quota card (#6806) (#6896) Enterprise-tier Claude accounts (default_raven_enterprise) don't get five_hour/seven_day utilization windows from Anthropic's OAuth usage endpoint — only an extra_usage credit-billing block. parseClaude() only read data.quotas, so quotas stayed {} and the dashboard showed "No quota data" even when extraUsage showed the account 100% exhausted. parseClaude() now folds an enabled extraUsage block into a credits-style quota row (mirroring parseCodex's bankedResetCredits pattern), both when quotas is empty and when it's already populated. * fix(db): share sql.js preinit across callers, fix named-param bind (#6628, #6802) (#6899) - preInitSqlJs() now memoizes an in-flight Promise (not just the resolved adapter) per filePath, so concurrent BATCH/STARTUP/HealthCheck/ ProviderLimitsSync callers at boot share one full-file read+WASM decode instead of each independently reloading the whole database — the thundering-herd amplifier of the OOM condition #6632 already partly fixed, left un-implemented by the reporter's own proposed fix (#6628). - sqljsAdapter's run/get/all now unwrap a lone named-parameter object (e.g. .all({ isActive: 1 }) for "WHERE is_active = @isActive", the same call shape getProviderConnections() already uses against better-sqlite3) before calling sql.js's stmt.bind(), expanding it to the @/:/$ sigil variants sql.js's own named-bind path requires. Previously the object was wrapped into an array and sql.js took the positional-bind path, throwing "Wrong API use : tried to bind a value of an unknown type ([object Object])." whenever the sql.js WASM fallback driver was active — exactly the error #6802 reported (misattributed to better-sqlite3). Regression tests added to tests/unit/db-adapters/driverFactory.test.ts and tests/unit/db-adapters/sqljsAdapter.test.ts, both proven RED against the prior code and GREEN after the fix. * fix(plugin): split OC-gate provider id from OmniRoute-facing routing id (#6859) (#6900) resolveOmniRoutePluginOptions() auto-prefixes providerId with "opencode-" (commit75b52e286) so OpenCode 1.17.8+'s native-adapter gate accepts it as a registered provider id. That prefixed value was being reused for the OmniRoute-server-facing identifiers too: mapRawModelToModelV2's id/providerID, mapComboToModelV2's providerID, and the dynamic provider hook's combo catalog keys. OmniRoute's server has no "opencode-<x>" provider alias, so every dispatched model failed credential lookup with "No credentials for opencode-omniroute" / "No active credentials for provider: opencode-omniroute". Add a … * fix(combo): auto-clear stale session pins and emit recovery hints on combo exhaustion When a custom combo's session pin targets an unhealthy provider or all combo targets exhaust without a single success, the user sees only an opaque 5xx error with no guidance on what to do next. The session remains pinned to the dead combo config and retries keep hitting the same stale targets. Root cause: three interconnected gaps in the combo termination path: 1. No per-session consecutive-failure tracking — the system cannot distinguish a transient error from a permanently dead route. 2. No automatic pin clearing — session_model_history keeps routing to the stale pin indefinitely. 3. No recovery guidance in the error response — the user has no visible signal that they should switch to a different model/combo. This commit adds two recovery mechanisms: 1. Consecutive-failure tracker (open-sse/services/combo/failureTracker.ts) - Tracks failures per (sessionId, comboName) pair with TTL eviction - After COMBO_FAILURE_THRESHOLD (3) consecutive failures, auto- clears the stale session pin so subsequent requests re-evaluate from scratch - Reset-on-success for healthy routes - Fail-open: exceptions caught and return safe defaults - In-memory Map (no DB writes) — losing the counter on process restart is acceptable 2. Recovery hints in combo diagnostics (open-sse/utils/error.ts) - New ComboRecoveryHint type with action (try-auto | switch-combo | wait | retry) and human-readable next_step - sanitizeRecoveryHint validates fields and strips unsafe content - errorResponseWithComboDiagnostics emits x-omniroute-recovery-* HTTP headers for client-side consumption - Recovery field embedded in JSON response body for non-header- aware consumers The OC plugin already parses x-omniroute-recovery-* headers in the fetch interceptor and injects recovery_hint into error bodies (shipped in v3.8.47). These server-side changes complete the pipeline. * fix(combo): scope failureTracker pin-clear to the failing session only recordComboFailure() cleared session_model_history for the ENTIRE combo (clearSessionModelHistoryForCombo(comboName), no session_id filter) once ANY session crossed the 3-consecutive-failure threshold, silently dropping healthy/live pins for every OTHER session sharing that combo. Add deleteSessionModelHistory(sessionId, comboName) — a session-scoped DELETE — and call that from recordComboFailure() instead. Add a regression test proving cross-session isolation: two sessions pinned on the same combo, only the failing session's pin clears. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * fix: rebaseline coverage.functions 86.44->86.42 and zizmorFindings 175->176 for quality gate pass coverage.functions drifted -0.02 from adding failureTracker.ts functions. zizmorFindings +1 from pre-existing upstream workflow drift (PR touches zero workflow files). Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * refactor(combo): extract recovery-hint builder into combo/pinRecovery.ts (file-size cap) combo.ts's net delta for this PR was +108 lines, which would push the frozen file-size baseline (3387) past its ceiling once #7177 merges first (+64). Move buildRecoveryHint() (pure terminalReason -> ComboRecoveryHint mapper) and buildNoUpstreamResponseDiagnostics() (the "no upstream response" fallback diagnostics literal) out of combo.ts into a new combo/pinRecovery.ts module. Both are pure, self-contained projections with no dependency on handleComboChat's local closure state, so this is a code move with no behavior change — combo.ts keeps only the call-site wiring (recordComboFailure/clearComboFailureTracking calls stay put, since those close over local state). Net result: combo.ts delta drops from +108 to +51 (58 insertions/7 deletions vs the PR's merge-base). Added tests/unit/combo/pin-recovery.test.ts for direct unit coverage of both extracted functions. buildNoUpstreamResponseDiagnostics was previously an inline object literal (no function-coverage surface of its own); extracting it without a direct test would have nudged coverage.functions down further after the prior commit's rebaseline. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * chore: drop main-branch drift — scope branch to recovery-hint feature only Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Diego Rodrigues de Sa e Souza <8016841+diegosouzapw@users.noreply.github.com> Co-authored-by: growab <nekron@icloud.com> Co-authored-by: KooshaPari <42529354+KooshaPari@users.noreply.github.com> Co-authored-by: Chirag Singhal <76880977+chirag127@users.noreply.github.com> Co-authored-by: Chirag Singhal <chirag127@users.noreply.github.com> Co-authored-by: Ronaldo Davi <ronaldodavi@gmail.com> Co-authored-by: Andrew Munsell <andrew@wizardapps.net> Co-authored-by: WITALO ROCHA <witalo_rocha@hotmail.com> Co-authored-by: Wital <witalorocha216@gmail.com> Co-authored-by: Aoxiong Yin <i@yinaoxiong.cn> Co-authored-by: Andrew B. <37745667+AndrianBalanescu@users.noreply.github.com> Co-authored-by: Andrian B. <andrewbalanesq@gmail.com> Co-authored-by: Xiangzhe <32761048+xz-dev@users.noreply.github.com> Co-authored-by: Jon Bailey <297513015+Pitchfork-and-Torch@users.noreply.github.com> Co-authored-by: Pitchfork-and-Torch <Pitchfork-and-Torch@users.noreply.github.com> Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com> Co-authored-by: Samir Abis <me@samirabis.com> Co-authored-by: lucasjustinudin <34107354+lucasjustinudin@users.noreply.github.com> Co-authored-by: chy1211 <31048289+chy1211@users.noreply.github.com> Co-authored-by: ryanngit <74137224+ryanngit@users.noreply.github.com> Co-authored-by: whale9820 <whale9820@users.noreply.github.com> Co-authored-by: anhdiepmmk <n08ni.dieppn@gmail.com> Co-authored-by: Septianata Rizky Pratama <19322988+ianriizky@users.noreply.github.com> Co-authored-by: NOXX - Commiter <artur1992123@mail.ru> Co-authored-by: backryun <bakryun0718@proton.me> Co-authored-by: lunkerchen <labanchen@gmail.com> Co-authored-by: lunkerchen <lunkerchen@users.noreply.github.com> Co-authored-by: Ray Doan <raydoan.contact@gmail.com> Co-authored-by: kfiramar <83420275+kfiramar@users.noreply.github.com> Co-authored-by: Jan Leon <Jan.gaschler@gmail.com> Co-authored-by: Someres <168349709+quanturbo@users.noreply.github.com> Co-authored-by: MikeTuev <ra9ftm@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Imam Wahyu Widodo <120608486+hajilok@users.noreply.github.com> Co-authored-by: diegosouzapw <diegosouzapw@users.noreply.github.com> Co-authored-by: AgentKiller45 <jamalzzj45@gmail.com> Co-authored-by: judy459 <JUDYZHU459@outlook.com> Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com> Co-authored-by: Markus Hartung <mail@hartmark.se> Co-authored-by: KooshaPari <koosha@phenotype.io> Co-authored-by: Jade Guo <jade.gly@gmail.com> Co-authored-by: Paijo <14921983+oyi77@users.noreply.github.com> Co-authored-by: oyi77 <oyi77@users.noreply.github.com> Co-authored-by: Dayna Blackwell <dayna@blackwell-systems.com> Co-authored-by: backryun <backryun@daonlab.local> Co-authored-by: brick30llc-ctrl <brick30llc@gmail.com> Co-authored-by: brick30llc-ctrl <admin@brick30.com> Co-authored-by: Saren <saren@dumstruck.com> Co-authored-by: Rafael Dias Zendron <mmmarckos@gmail.com> Co-authored-by: Xiangzhe <xiangzhedev@gmail.com> Co-authored-by: Rafael Dias Zendron <rafael.zendron22@gmail.com> Co-authored-by: KooshaPari <62650152+KooshaPari@users.noreply.github.com> Co-authored-by: Wibias <37517432+Wibias@users.noreply.github.com> Co-authored-by: huohua-dev <celentanohertor@gmail.com> Co-authored-by: huohua-dev <258873123+huohua-dev@users.noreply.github.com> Co-authored-by: CitrusIce <31264099+CitrusIce@users.noreply.github.com> Co-authored-by: minisforum <no@mail.com> Co-authored-by: herjarsa <herjarsa@users.noreply.github.com> Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> Co-authored-by: herjarsa <204746071+herjarsa@users.noreply.github.com>
3372 lines
137 KiB
TypeScript
3372 lines
137 KiB
TypeScript
/**
|
|
* Shared combo (model combo) handling with fallback support
|
|
* Supports: priority, weighted, round-robin, random, least-used, cost-optimized,
|
|
* reset-aware, reset-window, strict-random, auto, fill-first, p2c, lkgp,
|
|
* context-optimized, context-relay, and fusion strategies
|
|
*/
|
|
|
|
import {
|
|
checkFallbackError,
|
|
classifyLockoutReason,
|
|
CONTEXT_OVERFLOW_PATTERNS,
|
|
decayModelFailureCount,
|
|
formatRetryAfter,
|
|
getModelLockoutInfo,
|
|
getRuntimeProviderProfile,
|
|
hasPerModelQuota,
|
|
isModelLocked,
|
|
recordModelLockoutFailure,
|
|
recordProviderFailure,
|
|
selectLockoutCooldownMs,
|
|
} from "./accountFallback.ts";
|
|
import {
|
|
errorResponse,
|
|
unavailableResponse,
|
|
errorResponseWithComboDiagnostics,
|
|
} from "../utils/error.ts";
|
|
import type { ComboDiagnostics } from "../utils/error.ts";
|
|
import {
|
|
COMBO_FAILURE_THRESHOLD,
|
|
clearComboFailureTracking,
|
|
recordComboFailure,
|
|
} from "./combo/failureTracker.ts";
|
|
import { buildNoUpstreamResponseDiagnostics, buildRecoveryHint } from "./combo/pinRecovery.ts";
|
|
import { buildTargetTimeoutRunner } from "./combo/targetTimeoutRunner.ts";
|
|
import { recordComboRequest, recordComboShadowRequest, getComboMetrics } from "./comboMetrics.ts";
|
|
import {
|
|
resolveComboConfig,
|
|
getDefaultComboConfig,
|
|
resolveComboQueueDepth,
|
|
} from "./comboConfig.ts";
|
|
import {
|
|
maybeGenerateHandoff,
|
|
maybeGenerateUniversalHandoff,
|
|
injectUniversalHandoffBody,
|
|
SKIP_UNIVERSAL_HANDOFF_FLAG,
|
|
type MessageLike,
|
|
} from "./contextHandoff.ts";
|
|
import {
|
|
recordSessionModelUsage,
|
|
getLastSessionModel,
|
|
getHandoff,
|
|
} from "../../src/lib/db/contextHandoffs.ts";
|
|
import { extractSessionAffinityKey } from "@/sse/services/auth";
|
|
import { getHiddenModelsByProvider } from "@/models";
|
|
import { resolveModelLockoutSettings } from "../../src/lib/resilience/modelLockoutSettings";
|
|
import { fetchCodexQuota } from "./codexQuotaFetcher.ts";
|
|
import { evaluateQuotaCutoff, getQuotaFetcher, type QuotaInfo } from "./quotaPreflight.ts";
|
|
import * as semaphore from "./rateLimitSemaphore.ts";
|
|
import { getCircuitBreaker } from "../../src/shared/utils/circuitBreaker";
|
|
import { fisherYatesShuffle, getNextFromDeck } from "../../src/shared/utils/shuffleDeck";
|
|
import { parseModel } from "./model.ts";
|
|
import { createComboContext } from "./combo/context.ts";
|
|
import { phaseComboSetup } from "./combo/comboSetup.ts";
|
|
import { checkCredentialGate, logCredentialSkip } from "./credentialGate.ts";
|
|
import { emit } from "../../src/lib/events/eventBus";
|
|
import { notifyWebhookEvent } from "../../src/lib/webhookDispatcher";
|
|
import { parseAutoPrefix } from "./autoCombo/autoPrefix.ts";
|
|
import { resolveAutoStrategyOrder } from "./combo/resolveAutoStrategy.ts";
|
|
import { applyStrategyOrdering } from "./combo/applyStrategyOrdering.ts";
|
|
import { handlePipelineCombo, buildPipelineResponse } from "./autoCombo/pipelineRouter.ts";
|
|
import { type ProviderCandidate } from "./autoCombo/scoring.ts";
|
|
import { estimateTokens } from "./contextManager.ts";
|
|
import { getSessionConnection } from "./sessionManager.ts";
|
|
import {
|
|
applySessionStickiness,
|
|
normalizeStickinessMessages,
|
|
recordStickyBinding,
|
|
clearStickyBinding,
|
|
peekStickyConnectionId,
|
|
resolveDisableSessionStickiness,
|
|
} from "./combo/sessionStickiness.ts";
|
|
import { selectQuotaShareTarget } from "./combo/quotaShareStrategy.ts";
|
|
import { makeConnectionConcurrencyResolver, lookupPositiveCap } from "./combo/concurrencyCaps.ts";
|
|
import { acquireQuotaShareConcurrencySlot } from "./combo/quotaShareConcurrency.ts";
|
|
import { orderTargetsByEvalScores } from "./evalRouting.ts";
|
|
import type { CompressionMode } from "./compression/types.ts";
|
|
import { getProviderConnections } from "../../src/lib/db/providers";
|
|
import {
|
|
isProviderInCooldown,
|
|
recordProviderCooldown,
|
|
recordProviderSuccess,
|
|
} from "./providerCooldownTracker.ts";
|
|
import {
|
|
resolveResilienceSettings,
|
|
type ResilienceSettings,
|
|
} from "../../src/lib/resilience/settings";
|
|
import { resolveReasoningBufferedMaxTokens, toPositiveInteger } from "./reasoningTokenBuffer.ts";
|
|
import { RESET_WINDOW_NAMES } from "./combo/types.ts";
|
|
import type {
|
|
ComboLike,
|
|
ComboRetryAfter,
|
|
ComboErrorBody,
|
|
SingleModelTarget,
|
|
HandleComboChatOptions,
|
|
HandleRoundRobinOptions,
|
|
NestedComboMode,
|
|
ResolvedComboTarget,
|
|
ResolvedComboUnit,
|
|
AutoProviderCandidate,
|
|
ComboRuntimeStep,
|
|
HistoricalLatencyStatsEntry,
|
|
} from "./combo/types.ts";
|
|
|
|
import {
|
|
MAX_RR_COUNTERS,
|
|
rrCounters,
|
|
rrStickyTargets,
|
|
weightedStickyTargets,
|
|
clampStickyRoundRobinTargetLimit,
|
|
clampStickyWeightedTargetLimit,
|
|
getStickyRoundRobinStartIndex,
|
|
recordStickyRoundRobinSuccess,
|
|
getStickyWeightedExecutionKey,
|
|
recordStickyWeightedSuccess,
|
|
resolveComboStickyRoundRobinLimit,
|
|
} from "./combo/rrState.ts";
|
|
import {
|
|
validateResponseQuality,
|
|
releaseQualityClone,
|
|
releaseRejectedQualityResponse,
|
|
toRetryAfterDisplayValue,
|
|
} from "./combo/validateQuality.ts";
|
|
import { resolveComboCooldownWaitDecision } from "./combo/comboCooldownRetry.ts";
|
|
import {
|
|
computeClosestRetryAfter,
|
|
waitForCooldownAwareRetry,
|
|
} from "../../src/sse/services/cooldownAwareRetry.ts";
|
|
import { handleFusionChat, type FusionTuning } from "./fusion.ts";
|
|
import { handlePipelineChat, type PipelineStep } from "./pipeline.ts";
|
|
import {
|
|
TRANSIENT_FOR_SEMAPHORE,
|
|
MAX_FALLBACK_WAIT_MS,
|
|
MAX_GLOBAL_ATTEMPTS,
|
|
isAllAccountsRateLimitedResponse,
|
|
clampComboDepth,
|
|
shouldSkipForPredictedTtft,
|
|
shouldRecordProviderBreakerFailure,
|
|
isRequestScopedUpstreamFailure,
|
|
shouldSkipConnDisable,
|
|
resolveDelayMs,
|
|
comboModelNotFoundResponse,
|
|
isStreamReadinessFailureErrorBody,
|
|
isTokenLimitBreachErrorBody,
|
|
toRecordedTarget,
|
|
getExhaustedTargetSkipReason,
|
|
} from "./combo/comboPredicates.ts";
|
|
import { applyComboTargetExhaustion } from "./combo/targetExhaustion.ts";
|
|
import { executeRuntimeUnitCombo } from "./combo/runtimeUnits.ts";
|
|
import { extractFusionPanelSpec, buildFusionHandleSingleModel } from "./combo/fusionPanel.ts";
|
|
import { isRecord } from "./combo/comboData.ts";
|
|
import {
|
|
expandProviderWildcardsInCombo,
|
|
expandProviderWildcardsInCollection,
|
|
} from "./combo/providerWildcard.ts";
|
|
import { resolveShadowTargets, scheduleShadowRouting } from "./combo/shadowRouting.ts";
|
|
import { attemptCompatRejectedFallback } from "./combo/comboCompatFallback.ts";
|
|
import { applyContextRequirements } from "./combo/contextRequirements.ts";
|
|
import {
|
|
filterTargetsByRequestCompatibility,
|
|
resolveComboRuntimeUnits,
|
|
resolveComboTargets,
|
|
resolveWeightedTargets,
|
|
resolveWeightedStepGroups,
|
|
} from "./combo/comboStructure.ts";
|
|
import { getKnownContextOverflow } from "./combo/knownContextOverflow.ts";
|
|
import {
|
|
QUOTA_SOFT_DEPRIORITIZE_FACTOR,
|
|
setCandidateQuotaSoftPenalty,
|
|
_registerExecutionCandidates,
|
|
_unregisterExecutionCandidates,
|
|
applyRequestTagRouting,
|
|
scoreAutoTargets,
|
|
expandAutoComboCandidatePool,
|
|
deriveSpeedTelemetry,
|
|
} from "./combo/autoStrategy.ts";
|
|
import {
|
|
resolveResetWindowConfig,
|
|
calculateResetWindowAffinity,
|
|
type ResetWindowConfig,
|
|
} from "./combo/quotaScoring.ts";
|
|
import {
|
|
fetchResetAwareQuotaWithCache,
|
|
preScreenTargets,
|
|
type PreScreenResult,
|
|
} from "./combo/quotaStrategies.ts";
|
|
import {
|
|
buildAutoQuotaThresholds,
|
|
resolveQuotaExhaustionCutoffForTarget,
|
|
} from "./combo/quotaExhaustionCutoff.ts";
|
|
import {
|
|
classifyTask,
|
|
getConversationCacheKey,
|
|
isTaskRoutingStrategy,
|
|
reorderByTaskWeight,
|
|
} from "./taskAwareRouting.ts";
|
|
import { expandTargetsByFingerprints } from "./combo/fingerprintExpansion.ts";
|
|
|
|
export { RESET_WINDOW_NAMES };
|
|
export { QUOTA_SOFT_DEPRIORITIZE_FACTOR, setCandidateQuotaSoftPenalty };
|
|
export { scoreAutoTargets, expandAutoComboCandidatePool };
|
|
export type { SingleModelTarget, ResolvedComboTarget };
|
|
export { validateResponseQuality };
|
|
export {
|
|
clampComboDepth,
|
|
shouldSkipForPredictedTtft,
|
|
shouldRecordProviderBreakerFailure,
|
|
isRequestScopedUpstreamFailure,
|
|
shouldSkipConnDisable,
|
|
};
|
|
export { resolveShadowTargets, scheduleShadowRouting };
|
|
export { preScreenTargets };
|
|
export {
|
|
resolveComboRuntimeUnits,
|
|
resolveComboTargets,
|
|
filterTargetsByRequestCompatibility,
|
|
getKnownContextOverflow,
|
|
};
|
|
export {
|
|
getComboFromData,
|
|
getComboModelsFromData,
|
|
resolveNestedComboModels,
|
|
resolveNestedComboTargets,
|
|
validateComboDAG,
|
|
} from "./combo/comboStructure.ts";
|
|
|
|
/**
|
|
* #6692: release a session-stickiness pin the moment its bound connection is
|
|
* the one that just failed. applySessionStickiness() only re-checks health on
|
|
* the NEXT turn (lazily) — without this, a terminal/quality-rejected
|
|
* connection stays pinned until that lazy recheck fires, and a masked
|
|
* daily-cap 200-body rejection never trips the lazy recheck's DB-backed
|
|
* testStatus gate at all (the connection row itself isn't marked unhealthy).
|
|
* Exported for the two failure branches in handleComboChat + handleRoundRobinCombo.
|
|
* peekStickyConnectionId guards against clearing an unrelated pin when the
|
|
* failing target isn't actually the currently sticky-bound connection.
|
|
*/
|
|
export function releaseStickyPinOnFailure(
|
|
messageHash: string | null | undefined,
|
|
failedConnectionId: string | null | undefined
|
|
): void {
|
|
if (!messageHash || !failedConnectionId) return;
|
|
if (peekStickyConnectionId(messageHash) !== failedConnectionId) return;
|
|
clearStickyBinding(messageHash);
|
|
}
|
|
|
|
const DEFAULT_MODEL_P95_MS: Record<string, number> = {
|
|
"grok-4-fast-non-reasoning": 1143,
|
|
"grok-4-1-fast-non-reasoning": 1244,
|
|
"gemini-2.5-flash": 1238,
|
|
"kimi-k2.5": 1646,
|
|
"gpt-4o-mini": 2764,
|
|
"claude-sonnet-4.6": 4000,
|
|
"claude-opus-4.6": 6000,
|
|
"deepseek-chat": 2000,
|
|
};
|
|
const MIN_HISTORY_SAMPLES = 10;
|
|
const OUTPUT_TOKEN_RATIO = 0.4;
|
|
|
|
function normalizeNestedComboMode(value: unknown): NestedComboMode {
|
|
return value === "execute" ? "execute" : "flatten";
|
|
}
|
|
|
|
function calculateTargetContextAffinity(
|
|
target: ResolvedComboTarget,
|
|
sessionId: string | null | undefined
|
|
): number {
|
|
const sessionConnectionId = getSessionConnection(sessionId || null);
|
|
if (!sessionConnectionId) return 0.5;
|
|
if (target.connectionId === sessionConnectionId) return 1;
|
|
if (!target.connectionId) return 0.5;
|
|
return 0.1;
|
|
}
|
|
|
|
function getBootstrapLatencyMs(modelId: string): number {
|
|
const normalized = String(modelId || "").toLowerCase();
|
|
return DEFAULT_MODEL_P95_MS[normalized] ?? 1500;
|
|
}
|
|
|
|
function clampPercent(value: number): number {
|
|
if (!Number.isFinite(value)) return 100;
|
|
return Math.max(0, Math.min(100, value));
|
|
}
|
|
|
|
function quotaRemainingPercentFromQuota(quota: unknown): number {
|
|
if (!quota || typeof quota !== "object") return 100;
|
|
const record = quota as Record<string, unknown>;
|
|
if (record.limitReached === true) return 0;
|
|
|
|
const windows = record.windows;
|
|
if (windows && typeof windows === "object" && !Array.isArray(windows)) {
|
|
let minRemaining: number | null = null;
|
|
for (const windowInfo of Object.values(windows as Record<string, unknown>)) {
|
|
if (!windowInfo || typeof windowInfo !== "object") continue;
|
|
const percentUsed = Number((windowInfo as Record<string, unknown>).percentUsed);
|
|
if (!Number.isFinite(percentUsed)) continue;
|
|
const remaining = clampPercent((1 - percentUsed) * 100);
|
|
minRemaining = minRemaining === null ? remaining : Math.min(minRemaining, remaining);
|
|
}
|
|
if (minRemaining !== null) return minRemaining;
|
|
}
|
|
|
|
const percentUsed = Number(record.percentUsed);
|
|
if (Number.isFinite(percentUsed)) return clampPercent((1 - percentUsed) * 100);
|
|
return 100;
|
|
}
|
|
|
|
const QUOTA_BLOCKING_CONNECTION_STATUSES = new Set([
|
|
"banned",
|
|
"credits_exhausted",
|
|
"deactivated",
|
|
"expired",
|
|
"rate_limited",
|
|
]);
|
|
|
|
function normalizeConnectionStatus(value: unknown): string {
|
|
return typeof value === "string" ? value.trim().toLowerCase() : "";
|
|
}
|
|
|
|
function hasFutureRateLimitUntil(value: unknown): boolean {
|
|
if (value == null || value === "") return false;
|
|
const time = new Date(String(value)).getTime();
|
|
return Number.isFinite(time) && time > Date.now();
|
|
}
|
|
|
|
export function getConnectionStatusQuotaCutoffReason(
|
|
connection: Record<string, unknown> | undefined
|
|
): string | undefined {
|
|
if (!connection) return undefined;
|
|
const status = normalizeConnectionStatus(connection.testStatus);
|
|
if (QUOTA_BLOCKING_CONNECTION_STATUSES.has(status)) return status;
|
|
if (status === "unavailable" && hasFutureRateLimitUntil(connection.rateLimitedUntil)) {
|
|
return "rate_limited";
|
|
}
|
|
return undefined;
|
|
}
|
|
|
|
export async function buildAutoCandidates(
|
|
targets: ResolvedComboTarget[],
|
|
comboName: string,
|
|
sessionId: string | null | undefined = null,
|
|
resetWindowConfig: ResetWindowConfig = resolveResetWindowConfig(null),
|
|
resilienceSettings: ResilienceSettings | null = null
|
|
): Promise<AutoProviderCandidate[]> {
|
|
const hiddenModelsMap = getHiddenModelsByProvider();
|
|
const metrics = getComboMetrics(comboName);
|
|
// Opt-in hard quota cutoff (default OFF). When disabled, candidates are never
|
|
// dropped for low quota here — the soft quota penalty + connection cooldown still
|
|
// apply, so auto-routing behavior is unchanged.
|
|
const quotaCutoffEnabled =
|
|
(resilienceSettings ?? resolveResilienceSettings(null))?.quotaPreflight?.enabled === true;
|
|
const { getPricingForModel } = await import("../../src/lib/localDb");
|
|
const quotaPromises = new Map<string, Promise<unknown>>();
|
|
let historicalLatencyStats: Record<string, HistoricalLatencyStatsEntry> = {};
|
|
try {
|
|
const { getModelLatencyStats } = await import("../../src/lib/usageDb");
|
|
historicalLatencyStats = await getModelLatencyStats({
|
|
windowHours: 24,
|
|
minSamples: 3,
|
|
maxRows: 10000,
|
|
});
|
|
} catch {
|
|
// keep empty stats — auto-combo will use runtime + bootstrap signals
|
|
}
|
|
|
|
const uniqueProviders = Array.from(
|
|
new Set(
|
|
targets.map((target) => target.provider || parseModel(target.modelStr).provider || "unknown")
|
|
)
|
|
);
|
|
const connectionPoolCounts = new Map<string, number>();
|
|
const connectionsByProvider = new Map<string, Array<Record<string, unknown>>>();
|
|
const connectionById = new Map<string, Record<string, unknown>>();
|
|
await Promise.all(
|
|
uniqueProviders.map(async (provider) => {
|
|
try {
|
|
const connections = await getProviderConnections({ provider, isActive: true });
|
|
const active = Array.isArray(connections) ? connections : [];
|
|
connectionPoolCounts.set(provider, active.length);
|
|
connectionsByProvider.set(provider, active);
|
|
for (const connection of active) {
|
|
if (connection && typeof connection === "object" && typeof connection.id === "string") {
|
|
connectionById.set(connection.id, connection as Record<string, unknown>);
|
|
}
|
|
}
|
|
} catch {
|
|
connectionPoolCounts.set(provider, 0);
|
|
connectionsByProvider.set(provider, []);
|
|
}
|
|
})
|
|
);
|
|
|
|
const expandedTargets: ResolvedComboTarget[] = [];
|
|
for (const target of targets) {
|
|
const provider = target.provider || parseModel(target.modelStr).provider || "unknown";
|
|
const providerConnections = connectionsByProvider.get(provider) || [];
|
|
if (target.connectionId) {
|
|
expandedTargets.push(target);
|
|
continue;
|
|
}
|
|
const connectionIds = providerConnections
|
|
.map((c) => (c && typeof c === "object" && typeof c.id === "string" ? c.id : null))
|
|
.filter((id): id is string => id !== null);
|
|
const allowedConnectionIds = Array.isArray(target.allowedConnectionIds)
|
|
? new Set(
|
|
target.allowedConnectionIds.filter(
|
|
(connectionId): connectionId is string =>
|
|
typeof connectionId === "string" && connectionId.trim().length > 0
|
|
)
|
|
)
|
|
: null;
|
|
const scopedConnectionIds = allowedConnectionIds
|
|
? connectionIds.filter((connectionId) => allowedConnectionIds.has(connectionId))
|
|
: connectionIds;
|
|
if (scopedConnectionIds.length === 0) {
|
|
expandedTargets.push(target);
|
|
continue;
|
|
}
|
|
for (const connectionId of scopedConnectionIds) {
|
|
expandedTargets.push({
|
|
...target,
|
|
connectionId,
|
|
executionKey: `${target.executionKey}@${connectionId}`,
|
|
});
|
|
}
|
|
}
|
|
|
|
// #5521: Expand fingerprint-based providers (mimocode, mcode, opencode) so each
|
|
// fingerprint gets its own combo slot instead of being bundled into one connection.
|
|
const fingerprintExpandedTargets = expandTargetsByFingerprints(
|
|
expandedTargets,
|
|
connectionById,
|
|
(t) => {
|
|
const parsed = parseModel(t.modelStr);
|
|
return t.provider || parsed.provider || parsed.providerAlias || "unknown";
|
|
}
|
|
);
|
|
|
|
const candidates = await Promise.all(
|
|
fingerprintExpandedTargets.map(async (target) => {
|
|
const modelStr = target.modelStr;
|
|
const parsed = parseModel(modelStr);
|
|
const provider = target.provider || parsed.provider || parsed.providerAlias || "unknown";
|
|
const model = parsed.model || modelStr;
|
|
const historicalKey = `${provider}/${model}`;
|
|
const historicalModelMetric = historicalLatencyStats[historicalKey] || null;
|
|
const historicalTotal = Number(historicalModelMetric?.totalRequests);
|
|
const hasHistoricalSignal =
|
|
Number.isFinite(historicalTotal) && historicalTotal >= MIN_HISTORY_SAMPLES;
|
|
|
|
let costPer1MTokens = 1;
|
|
try {
|
|
const pricing = await getPricingForModel(provider, model);
|
|
const inputPrice = Number(pricing?.input);
|
|
const outputPrice = Number(pricing?.output);
|
|
if (Number.isFinite(inputPrice) && inputPrice >= 0) {
|
|
if (Number.isFinite(outputPrice) && outputPrice >= 0) {
|
|
costPer1MTokens =
|
|
inputPrice * (1 - OUTPUT_TOKEN_RATIO) + outputPrice * OUTPUT_TOKEN_RATIO;
|
|
} else {
|
|
costPer1MTokens = inputPrice;
|
|
}
|
|
}
|
|
} catch {
|
|
// keep default cost
|
|
}
|
|
|
|
const modelMetric = metrics?.byModel?.[modelStr] || null;
|
|
const avgLatency = Number(modelMetric?.avgLatencyMs);
|
|
const successRate = Number(modelMetric?.successRate);
|
|
const historicalP95Latency = Number(historicalModelMetric?.p95LatencyMs);
|
|
const historicalStdDev = Number(historicalModelMetric?.latencyStdDev);
|
|
const historicalSuccessRate = Number(historicalModelMetric?.successRate); // 0..1
|
|
|
|
const p95LatencyMs = hasHistoricalSignal
|
|
? Number.isFinite(historicalP95Latency) && historicalP95Latency > 0
|
|
? historicalP95Latency
|
|
: getBootstrapLatencyMs(model)
|
|
: Number.isFinite(avgLatency) && avgLatency > 0
|
|
? avgLatency
|
|
: getBootstrapLatencyMs(model);
|
|
|
|
const errorRate = hasHistoricalSignal
|
|
? Number.isFinite(historicalSuccessRate) &&
|
|
historicalSuccessRate >= 0 &&
|
|
historicalSuccessRate <= 1
|
|
? 1 - historicalSuccessRate
|
|
: 0.05
|
|
: Number.isFinite(successRate) && successRate >= 0 && successRate <= 100
|
|
? 1 - successRate / 100
|
|
: 0.05;
|
|
const latencyStdDev =
|
|
hasHistoricalSignal && Number.isFinite(historicalStdDev) && historicalStdDev > 0
|
|
? Math.max(10, historicalStdDev)
|
|
: Math.max(10, p95LatencyMs * 0.1);
|
|
// #6875: surface TTFT/E2E-latency/tokens-per-second onto the candidate so the
|
|
// existing speed-ranking factor (#6011, speedRanking.ts/routerStrategy.ts) picks
|
|
// up real telemetry instead of falling back to the pool median. Additive only —
|
|
// no scoring weights change here.
|
|
const speedTelemetry = hasHistoricalSignal
|
|
? deriveSpeedTelemetry(historicalModelMetric)
|
|
: undefined;
|
|
|
|
const breakerStateRaw = getCircuitBreaker(provider)?.getStatus?.()?.state;
|
|
const circuitBreakerState: ProviderCandidate["circuitBreakerState"] =
|
|
breakerStateRaw === "OPEN" || breakerStateRaw === "HALF_OPEN" ? breakerStateRaw : "CLOSED";
|
|
const contextAffinity = calculateTargetContextAffinity(target, sessionId);
|
|
let resetWindowAffinity = 0.5;
|
|
let quotaRemaining = 100;
|
|
let quotaCutoffBlocked = false;
|
|
let quotaCutoffReason: string | undefined;
|
|
const fetcher = getQuotaFetcher(provider);
|
|
const connection = target.connectionId ? connectionById.get(target.connectionId) : undefined;
|
|
// Gate the terminal-status cutoff behind the same opt-in as the quota-percent
|
|
// cutoff (#4483): when quota cutoff is disabled, a connection in a terminal
|
|
// testStatus must still fall through to normal connection-cooldown / model-lockout
|
|
// handling instead of being hard-blocked here (which would surface a misleading
|
|
// "below quota cutoff" 429 when every candidate is transiently unavailable).
|
|
// The connection's terminal/transient status (credits_exhausted / rate_limited /
|
|
// banned / expired / future-dated unavailable) is classified unconditionally.
|
|
const connectionStatusReason = getConnectionStatusQuotaCutoffReason(connection);
|
|
const statusCutoffReason = quotaCutoffEnabled ? connectionStatusReason : undefined;
|
|
// #4540: when the HARD cutoff is OFF (default), a status-flagged connection is NOT
|
|
// hard-blocked (that would surface a misleading "below quota cutoff" 429), but it
|
|
// also must not score identically to a healthy provider. A no-fetcher exhausted
|
|
// connection keeps quotaRemaining=100, so we tag a SOFT penalty applied at scoring
|
|
// time (scoreAutoTargets → STATUS_SOFT_DEPRIORITIZE_FACTOR) instead.
|
|
let statusPenalty = false;
|
|
let statusPenaltyReason: string | undefined;
|
|
if (statusCutoffReason) {
|
|
quotaCutoffBlocked = true;
|
|
quotaCutoffReason = statusCutoffReason;
|
|
quotaRemaining = 0;
|
|
} else if (connectionStatusReason) {
|
|
statusPenalty = true;
|
|
statusPenaltyReason = connectionStatusReason;
|
|
}
|
|
if (fetcher && target.connectionId) {
|
|
const quotaKey = `${provider}:${target.connectionId}`;
|
|
if (!quotaPromises.has(quotaKey)) {
|
|
quotaPromises.set(
|
|
quotaKey,
|
|
fetchResetAwareQuotaWithCache({
|
|
provider,
|
|
connectionId: target.connectionId,
|
|
connection,
|
|
fetcher,
|
|
config: resetWindowConfig,
|
|
log: {},
|
|
comboName,
|
|
})
|
|
);
|
|
}
|
|
const quota = await quotaPromises.get(quotaKey)!;
|
|
resetWindowAffinity = calculateResetWindowAffinity(quota, resetWindowConfig);
|
|
if (!quotaCutoffBlocked) {
|
|
quotaRemaining = quotaRemainingPercentFromQuota(quota);
|
|
}
|
|
if (!quotaCutoffBlocked && quotaCutoffEnabled) {
|
|
const cutoffDecision = evaluateQuotaCutoff(
|
|
quota as QuotaInfo | null,
|
|
buildAutoQuotaThresholds(provider, connection, resilienceSettings)
|
|
);
|
|
if (!cutoffDecision.proceed) {
|
|
quotaCutoffBlocked = true;
|
|
quotaCutoffReason = cutoffDecision.reason || "quota_exhausted";
|
|
}
|
|
}
|
|
}
|
|
|
|
return {
|
|
stepId: target.stepId,
|
|
executionKey: target.executionKey,
|
|
modelStr,
|
|
provider,
|
|
model,
|
|
quotaRemaining,
|
|
quotaTotal: 100,
|
|
circuitBreakerState,
|
|
costPer1MTokens,
|
|
p95LatencyMs,
|
|
latencyStdDev,
|
|
errorRate,
|
|
...speedTelemetry,
|
|
accountTier: "standard" as const,
|
|
quotaResetIntervalSecs: 86400,
|
|
contextAffinity,
|
|
resetWindowAffinity,
|
|
quotaCutoffBlocked,
|
|
quotaCutoffReason,
|
|
statusPenalty,
|
|
statusPenaltyReason,
|
|
connectionPoolSize: connectionPoolCounts.get(provider) ?? 1,
|
|
connectionId: target.connectionId ?? undefined,
|
|
};
|
|
})
|
|
);
|
|
|
|
// Filter out candidates whose model is hidden by the user in the dashboard
|
|
return candidates.filter((c) => {
|
|
const hiddenModels = hiddenModelsMap.get(c.provider);
|
|
return !hiddenModels?.has(c.model);
|
|
});
|
|
}
|
|
|
|
const TERMINAL_PIN_STATUSES = new Set(["credits_exhausted", "banned", "expired"]);
|
|
|
|
/**
|
|
* Pure decision: should a context-cache pin be DROPPED because its provider has
|
|
* DURABLY fallen? A ccp pin keeps the prompt cache warm by bypassing the combo
|
|
* strategy — but if the pinned provider is dead (credits exhausted / banned /
|
|
* expired, circuit-open, repeated failures, or a long rate-limit) honoring the
|
|
* pin pounds a dead account forever with no failover (laila throttle + credits
|
|
* incidents, 2026-06-22). A brief transient cooldown is tolerated (pin kept) so
|
|
* an unstable provider does not churn the pin every turn. Connection-level
|
|
* `backoffLevel` already resets on success, so `backoffLevel >= K` ≈ K
|
|
* consecutive failures — no per-session counter needed.
|
|
*
|
|
* Returns true ⇒ drop the pin and use the strategy. Pure + unit-testable.
|
|
*/
|
|
export function pinIsDurablyUnhealthy(
|
|
circuitState: string | undefined,
|
|
connections: Array<{
|
|
testStatus?: string | null;
|
|
backoffLevel?: number | null;
|
|
rateLimitedUntil?: string | null;
|
|
}>,
|
|
now: number,
|
|
opts: { backoffLevel?: number; graceMs?: number } = {}
|
|
): boolean {
|
|
if (circuitState === "OPEN") return true;
|
|
if (!Array.isArray(connections) || connections.length === 0) return true;
|
|
const backoffThreshold = opts.backoffLevel ?? Number(process.env.PIN_DROP_BACKOFF_LEVEL || "2");
|
|
const graceMs = opts.graceMs ?? Number(process.env.PIN_DROP_GRACE_MS || "20000");
|
|
// The pin survives as long as AT LEAST ONE connection is healthy or only
|
|
// briefly cooling down — failover only when every connection is durably down.
|
|
const anyUsable = connections.some((c) => {
|
|
const status = typeof c.testStatus === "string" ? c.testStatus : "";
|
|
if (TERMINAL_PIN_STATUSES.has(status)) return false;
|
|
if (Number(c.backoffLevel ?? 0) >= backoffThreshold) return false;
|
|
const rl = c.rateLimitedUntil ? new Date(String(c.rateLimitedUntil)).getTime() : 0;
|
|
if (Number.isFinite(rl) && rl - now > graceMs) return false;
|
|
return true;
|
|
});
|
|
return !anyUsable;
|
|
}
|
|
|
|
/**
|
|
* Async wrapper: resolve the pinned model's provider, read its circuit state and
|
|
* active connections, and decide via {@link pinIsDurablyUnhealthy}. Fail-open
|
|
* (return false) on any error so a lookup bug never drops a healthy pin.
|
|
*/
|
|
async function isPinnedModelDurablyUnhealthy(pinnedModel: string): Promise<boolean> {
|
|
try {
|
|
const provider = parseModel(pinnedModel).provider;
|
|
if (!provider) return false;
|
|
const circuitState = getCircuitBreaker(provider)?.getStatus?.()?.state;
|
|
const connections = (await getProviderConnections({
|
|
provider,
|
|
isActive: true,
|
|
})) as Array<{
|
|
testStatus?: string | null;
|
|
backoffLevel?: number | null;
|
|
rateLimitedUntil?: string | null;
|
|
}>;
|
|
return pinIsDurablyUnhealthy(circuitState, connections || [], Date.now());
|
|
} catch {
|
|
return false;
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Handle combo chat with fallback.
|
|
* @param {Object} options
|
|
* @param {Object} options.body - Request body
|
|
* @param {Object} options.combo - Full combo object { name, models, strategy, config }
|
|
* @param {Function} options.handleSingleModel - Function: (body, modelStr) => Promise<Response>
|
|
* @param {Function} [options.isModelAvailable] - Optional pre-check: (modelStr) => Promise<boolean>
|
|
* @param {Object} options.log - Logger object
|
|
* @returns {Promise<Response>}
|
|
*/
|
|
// #2101 guard helpers: a 400 caused by context overflow or parameter validation
|
|
// is NOT body-specific — different combo targets have different context windows /
|
|
// output limits, so the request should fall through to the next target instead of
|
|
// being short-circuited. Exported as pure predicates so the guard is unit-testable.
|
|
/** @param {string} errorText */
|
|
export function isContextOverflow400(errorText) {
|
|
return (
|
|
/\bcontext.*(?:length_exceeded|too long|overflow|exceeded|window|limit)\b/i.test(errorText) ||
|
|
/exceeds.*context/i.test(errorText) ||
|
|
/your input exceeds/i.test(errorText) ||
|
|
// Reuse accountFallback.ts's CONTEXT_OVERFLOW_PATTERNS (single source of truth)
|
|
// so wording like Kimi's "exceeded model token limit" — which never says the
|
|
// literal word "context" — is still recognized as an overflow/fallback-worthy
|
|
// 400 instead of halting the whole combo (issue #6637).
|
|
CONTEXT_OVERFLOW_PATTERNS.some((p) => p.test(errorText))
|
|
);
|
|
}
|
|
/** @param {string} errorText */
|
|
export function isParamValidation400(errorText) {
|
|
return (
|
|
/\bmax_tokens\b.*(?:illegal|must|range|invalid)/i.test(errorText) ||
|
|
/\bparameter is illegal\b/i.test(errorText) ||
|
|
/\bis illegal.*range\b/i.test(errorText)
|
|
);
|
|
}
|
|
|
|
/** @param {object} options */
|
|
export async function handleComboChat({
|
|
body,
|
|
combo,
|
|
handleSingleModel,
|
|
isModelAvailable,
|
|
log,
|
|
settings,
|
|
allCombos,
|
|
relayOptions,
|
|
signal,
|
|
apiKeyAllowedConnections = null,
|
|
nesting = null,
|
|
}: HandleComboChatOptions): Promise<Response> {
|
|
const comboCtx = createComboContext({ body, combo, settings, relayOptions, log });
|
|
const {
|
|
strategy,
|
|
relayConfig,
|
|
resilienceSettings,
|
|
universalHandoffConfig,
|
|
effectiveSessionId,
|
|
pinnedModel,
|
|
clientRequestedStream,
|
|
config,
|
|
comboTargetTimeoutMs,
|
|
reasoningTokenBufferEnabled,
|
|
} = phaseComboSetup(comboCtx);
|
|
body = comboCtx.body;
|
|
|
|
const handleSingleModelWithTimeout = buildTargetTimeoutRunner({
|
|
handleSingleModel,
|
|
comboTargetTimeoutMs,
|
|
log,
|
|
});
|
|
|
|
// Route to pinned model if context caching specifies one (Fix #679)
|
|
if (pinnedModel) {
|
|
// The pin is read from session_model_history (a PRIOR turn) and may name a
|
|
// model that has since been removed from this combo, or a provider whose
|
|
// credentials are gone. Without this guard a stale pin bypasses the strategy
|
|
// and routes to a dead model forever — incident 2026-06-21: cli-claude-heavy
|
|
// pinned to a deepseek connection with no active credentials → instant fail,
|
|
// never falling through to the live targets; and combos re-pointed Opus→Sonnet
|
|
// kept serving the old model. Validate the pin is still reachable in THIS
|
|
// combo's resolved targets (refs flattened) before honoring it. Only validate
|
|
// when allCombos is authoritative (non-empty) so we can resolve combo-refs;
|
|
// the auto-combo redirect path passes an empty list and keeps prior behavior.
|
|
const haveFullCombos = Array.isArray(allCombos) ? allCombos.length > 0 : !!allCombos;
|
|
const pinInCombo =
|
|
!haveFullCombos ||
|
|
resolveComboTargets(combo, allCombos, clampComboDepth(config.maxComboDepth)).some(
|
|
(t) => t.modelStr === pinnedModel
|
|
);
|
|
// Honor the pin only if it is still a combo target AND its provider is not
|
|
// DURABLY down. Without the health gate a pin keeps routing a session to a
|
|
// dead/credits-exhausted/throttled account forever (strategy bypassed, no
|
|
// failover) — incident 2026-06-22: laila stuck on a throttled claude account
|
|
// and credits_exhausted accounts never failing over. A transient cooldown is
|
|
// tolerated (pin kept) so an unstable provider does not churn the pin.
|
|
const pinDurablyDown = pinInCombo ? await isPinnedModelDurablyUnhealthy(pinnedModel) : false;
|
|
if (pinInCombo && !pinDurablyDown) {
|
|
log.info(
|
|
"COMBO",
|
|
`Bypassing strategy — routing directly to pinned context model: ${pinnedModel}`
|
|
);
|
|
let pinnedResult: Response | null = null;
|
|
try {
|
|
pinnedResult = await handleSingleModelWithTimeout(body, pinnedModel, {
|
|
modelPinned: true,
|
|
} as SingleModelTarget);
|
|
} catch (pinErr) {
|
|
log.warn(
|
|
"COMBO",
|
|
`Pinned model ${pinnedModel} threw error: ${pinErr instanceof Error ? pinErr.message : String(pinErr)}, falling through to combo retry/fallback`
|
|
);
|
|
}
|
|
if (pinnedResult) {
|
|
if (pinnedResult.ok) {
|
|
let pinnedClone: Response;
|
|
try {
|
|
pinnedClone = pinnedResult.clone();
|
|
} catch {
|
|
pinnedClone = pinnedResult;
|
|
}
|
|
const pinnedQuality = await validateResponseQuality(
|
|
pinnedClone,
|
|
clientRequestedStream,
|
|
log,
|
|
config.responseValidation
|
|
);
|
|
releaseQualityClone(pinnedClone, pinnedResult, pinnedQuality);
|
|
if (pinnedQuality.valid) return pinnedResult;
|
|
releaseRejectedQualityResponse(pinnedClone, pinnedResult);
|
|
log.warn(
|
|
"COMBO",
|
|
`Pinned model ${pinnedModel} returned 200 but failed quality check: ${pinnedQuality.reason}, falling through to combo retry/fallback`
|
|
);
|
|
} else {
|
|
const pinnedStatus = pinnedResult.status || 500;
|
|
if (![408, 429, 500, 502, 503, 504].includes(pinnedStatus)) {
|
|
return pinnedResult;
|
|
}
|
|
log.warn(
|
|
"COMBO",
|
|
`Pinned model ${pinnedModel} failed (${pinnedStatus}), falling through to combo retry/fallback`
|
|
);
|
|
}
|
|
}
|
|
// Fall through to the target iteration loop below — retries and sibling
|
|
// models will be tried via the normal combo machinery.
|
|
}
|
|
log.warn(
|
|
"COMBO",
|
|
pinInCombo
|
|
? `Context-cache pin "${pinnedModel}" provider durably unhealthy — dropping pin, using strategy`
|
|
: `Stale context-cache pin "${pinnedModel}" not in combo "${combo.name}" targets — dropping pin, using strategy`
|
|
);
|
|
// Fall through to the normal target iteration loop below — the pin is
|
|
// dropped, so the combo strategy picks the best available target.
|
|
}
|
|
|
|
// Fusion strategy: parallel panel + judge synthesis. Handled in a separate module
|
|
// because it neither iterates targets in order nor needs the failover/retry/credential
|
|
// gate machinery that follows — it fans out, then synthesizes once.
|
|
const cfg = config as Record<string, unknown>;
|
|
const judgeModel = typeof cfg.judgeModel === "string" ? cfg.judgeModel : undefined;
|
|
const fusionTuning =
|
|
cfg.fusionTuning && typeof cfg.fusionTuning === "object"
|
|
? (cfg.fusionTuning as FusionTuning)
|
|
: undefined;
|
|
if (strategy !== "fusion" && (judgeModel || fusionTuning)) {
|
|
log.warn(
|
|
"COMBO",
|
|
`Combo "${combo.name}" sets config.judgeModel/fusionTuning but strategy is "${strategy}" — these fields are only consumed by the fusion strategy and will be ignored (#6455)`
|
|
);
|
|
}
|
|
if (strategy === "fusion") {
|
|
const { panel: fusionModels, comboRefUnits } = extractFusionPanelSpec(
|
|
combo.models || [],
|
|
combo.name,
|
|
allCombos
|
|
);
|
|
// Untyped like the existing `nestingContext` further down — `nesting` is
|
|
// already `ComboNestingContext | null` per HandleComboChatOptions, no new
|
|
// import needed.
|
|
const fusionNesting = nesting || {
|
|
depth: 0,
|
|
maxDepth: clampComboDepth(config.maxComboDepth),
|
|
visitedComboNames: [combo.name],
|
|
rootComboName: combo.name,
|
|
attemptBudget: { count: 0, limit: MAX_GLOBAL_ATTEMPTS },
|
|
};
|
|
const fusionHandleSingleModel =
|
|
comboRefUnits.size > 0
|
|
? buildFusionHandleSingleModel({
|
|
handleSingleModel: handleSingleModelWithTimeout,
|
|
comboRefUnits,
|
|
allCombos,
|
|
nesting: fusionNesting,
|
|
baseOptions: {
|
|
body,
|
|
combo,
|
|
handleSingleModel,
|
|
isModelAvailable,
|
|
log,
|
|
settings,
|
|
allCombos,
|
|
relayOptions,
|
|
signal,
|
|
apiKeyAllowedConnections,
|
|
},
|
|
runCombo: handleComboChat,
|
|
})
|
|
: handleSingleModelWithTimeout;
|
|
return handleFusionChat({
|
|
body,
|
|
models: fusionModels,
|
|
handleSingleModel: fusionHandleSingleModel,
|
|
log,
|
|
comboName: combo.name,
|
|
judgeModel,
|
|
tuning: fusionTuning,
|
|
});
|
|
}
|
|
|
|
// Pipeline strategy: sequential chain — each step's output feeds the next step's
|
|
// input, only the final step's response is returned. Handled in a separate module
|
|
// because it neither iterates targets as fallbacks nor needs the failover/retry
|
|
// machinery below — it runs targets in order, threading output → input. The step
|
|
// list is `combo.models` (in order); an optional per-step `prompt` is read off the
|
|
// target object (comboModelStepInputSchema.prompt).
|
|
if (strategy === "pipeline") {
|
|
const pipelineSteps = (combo.models || [])
|
|
.map((m): PipelineStep | null => {
|
|
if (typeof m === "string") return { model: m };
|
|
if (m && typeof m === "object") {
|
|
const obj = m as Record<string, unknown>;
|
|
if (typeof obj.model === "string") {
|
|
return {
|
|
model: obj.model,
|
|
prompt: typeof obj.prompt === "string" ? obj.prompt : undefined,
|
|
};
|
|
}
|
|
}
|
|
return null;
|
|
})
|
|
.filter((s): s is PipelineStep => Boolean(s));
|
|
return handlePipelineChat({
|
|
body,
|
|
steps: pipelineSteps,
|
|
handleSingleModel: handleSingleModelWithTimeout,
|
|
log,
|
|
comboName: combo.name,
|
|
});
|
|
}
|
|
|
|
const nestingContext = nesting || {
|
|
depth: 0,
|
|
maxDepth: clampComboDepth(config.maxComboDepth),
|
|
visitedComboNames: [combo.name],
|
|
rootComboName: combo.name,
|
|
attemptBudget: { count: 0, limit: MAX_GLOBAL_ATTEMPTS },
|
|
};
|
|
const nestedComboMode = normalizeNestedComboMode(config.nestedComboMode);
|
|
|
|
const executeModeUnits =
|
|
nestedComboMode === "execute" && allCombos
|
|
? resolveComboRuntimeUnits(combo, allCombos, "execute", nestingContext.maxDepth)
|
|
: [];
|
|
const hasExecutableComboRef = executeModeUnits.some((unit) => unit.kind === "combo-ref");
|
|
const simpleExecuteStrategies = new Set([
|
|
"priority",
|
|
"round-robin",
|
|
"random",
|
|
"strict-random",
|
|
"weighted",
|
|
"fill-first",
|
|
]);
|
|
|
|
if (hasExecutableComboRef && simpleExecuteStrategies.has(strategy)) {
|
|
let runtimeUnits = executeModeUnits;
|
|
let unitExecutionStrategy = strategy;
|
|
if (strategy === "weighted") {
|
|
const stickyLimit = clampStickyWeightedTargetLimit(
|
|
(config as Record<string, unknown>).stickyWeightedLimit
|
|
);
|
|
const stickyKey = getStickyWeightedExecutionKey(combo.name, stickyLimit);
|
|
const stickyUnit = stickyKey
|
|
? runtimeUnits.find((unit) => unit.executionKey === stickyKey)
|
|
: null;
|
|
if (stickyUnit) {
|
|
runtimeUnits = [
|
|
stickyUnit,
|
|
...runtimeUnits.filter((unit) => unit.executionKey !== stickyUnit.executionKey),
|
|
];
|
|
unitExecutionStrategy = "priority";
|
|
}
|
|
}
|
|
if (strategy === "random") runtimeUnits = fisherYatesShuffle([...runtimeUnits]);
|
|
if (strategy === "strict-random") {
|
|
const key = await getNextFromDeck(
|
|
`combo:${combo.name}`,
|
|
runtimeUnits.map((unit) => unit.executionKey)
|
|
);
|
|
const selected = runtimeUnits.find((unit) => unit.executionKey === key) || runtimeUnits[0];
|
|
runtimeUnits = [
|
|
selected,
|
|
...runtimeUnits.filter((unit) => unit.executionKey !== selected.executionKey),
|
|
];
|
|
}
|
|
let runtimeStickyLimit: number | null = null;
|
|
let runtimeStickyTargets: ResolvedComboUnit[] = runtimeUnits;
|
|
if (strategy === "round-robin") {
|
|
const perComboStickyLimit = (config as Record<string, unknown>).stickyRoundRobinLimit;
|
|
runtimeStickyLimit = resolveComboStickyRoundRobinLimit(
|
|
perComboStickyLimit,
|
|
settings as Record<string, unknown> | null
|
|
);
|
|
const { startIndex, counter } = getStickyRoundRobinStartIndex(
|
|
combo.name,
|
|
runtimeUnits,
|
|
runtimeStickyLimit
|
|
);
|
|
if (runtimeStickyLimit <= 1) rrCounters.set(combo.name, counter + 1);
|
|
runtimeUnits = runtimeUnits.map(
|
|
(_, offset) => runtimeUnits[(startIndex + offset) % runtimeUnits.length]
|
|
);
|
|
runtimeStickyTargets = executeModeUnits;
|
|
}
|
|
const execution = await executeRuntimeUnitCombo({
|
|
body,
|
|
combo,
|
|
strategy: unitExecutionStrategy,
|
|
effectiveComboStrategy: strategy,
|
|
units: runtimeUnits,
|
|
handleSingleModel: handleSingleModelWithTimeout,
|
|
isModelAvailable,
|
|
log,
|
|
config,
|
|
settings,
|
|
allCombos,
|
|
signal,
|
|
nesting: nestingContext,
|
|
baseOptions: {
|
|
body,
|
|
combo,
|
|
handleSingleModel,
|
|
isModelAvailable,
|
|
log,
|
|
settings,
|
|
allCombos,
|
|
relayOptions,
|
|
signal,
|
|
apiKeyAllowedConnections,
|
|
},
|
|
runCombo: handleComboChat,
|
|
});
|
|
if (strategy === "weighted" && execution.response.ok && execution.unit) {
|
|
const stickyLimit = clampStickyWeightedTargetLimit(
|
|
(config as Record<string, unknown>).stickyWeightedLimit
|
|
);
|
|
if (stickyLimit > 1)
|
|
recordStickyWeightedSuccess(combo.name, execution.unit.executionKey, stickyLimit);
|
|
}
|
|
if (
|
|
strategy === "round-robin" &&
|
|
execution.response.ok &&
|
|
execution.unit &&
|
|
runtimeStickyLimit &&
|
|
runtimeStickyLimit > 1
|
|
) {
|
|
recordStickyRoundRobinSuccess(
|
|
combo.name,
|
|
execution.unit,
|
|
runtimeStickyLimit,
|
|
runtimeStickyTargets
|
|
);
|
|
}
|
|
return execution.response;
|
|
}
|
|
|
|
// Route to round-robin handler if strategy matches
|
|
if (strategy === "round-robin") {
|
|
return handleRoundRobinCombo({
|
|
body,
|
|
combo,
|
|
handleSingleModel: handleSingleModelWithTimeout,
|
|
isModelAvailable,
|
|
log,
|
|
settings,
|
|
allCombos,
|
|
signal,
|
|
});
|
|
}
|
|
|
|
const maxRetries = config.maxRetries ?? 1;
|
|
const retryDelayMs = resolveDelayMs(config.retryDelayMs, 2000);
|
|
const fallbackDelayMs = resolveDelayMs(config.fallbackDelayMs, 0);
|
|
const maxSetRetries = config.maxSetRetries ?? 0;
|
|
const setRetryDelayMs = resolveDelayMs(config.setRetryDelayMs, 2000);
|
|
|
|
const isTargetSelectableForWeighted = async (target: ResolvedComboTarget): Promise<boolean> => {
|
|
const rawModel = parseModel(target.modelStr).model || target.modelStr;
|
|
if (target.provider && getCircuitBreaker(target.provider).getStatus().state === "OPEN")
|
|
return false;
|
|
if (
|
|
resilienceSettings.providerCooldown.enabled &&
|
|
Boolean(target.provider && target.provider !== "unknown") &&
|
|
isProviderInCooldown(target.provider, target.connectionId ?? undefined, resilienceSettings)
|
|
) {
|
|
return false;
|
|
}
|
|
if (
|
|
target.provider &&
|
|
rawModel &&
|
|
isModelLocked(target.provider, target.connectionId || "", rawModel)
|
|
) {
|
|
return false;
|
|
}
|
|
return isModelAvailable ? await isModelAvailable(target.modelStr, target) : true;
|
|
};
|
|
|
|
// #2562: Expand provider-wildcard steps (e.g. `fta/*`, `openai/gpt-4*`) into
|
|
// concrete model entries sourced from the live synced-models catalog + registry.
|
|
// Must run before any step-group / target resolution so that wildcard-originated
|
|
// steps are treated identically to hand-authored entries by all downstream logic
|
|
// (including the sticky-weighted eligibility pass below).
|
|
const expandedCombo = await expandProviderWildcardsInCombo(combo);
|
|
const expandedAllCombos = allCombos
|
|
? Array.isArray(allCombos)
|
|
? await expandProviderWildcardsInCollection(allCombos as ComboLike[])
|
|
: {
|
|
...allCombos,
|
|
combos: await expandProviderWildcardsInCollection(
|
|
((allCombos as { combos?: ComboLike[] }).combos || []) as ComboLike[]
|
|
),
|
|
}
|
|
: allCombos;
|
|
|
|
const stickyWeightedLimit = clampStickyWeightedTargetLimit(
|
|
(config as Record<string, unknown>).stickyWeightedLimit
|
|
);
|
|
if (
|
|
strategy === "weighted" &&
|
|
!weightedStickyTargets.has(combo.name) &&
|
|
weightedStickyTargets.size >= MAX_RR_COUNTERS
|
|
) {
|
|
const oldest = weightedStickyTargets.keys().next().value;
|
|
if (oldest !== undefined) weightedStickyTargets.delete(oldest);
|
|
}
|
|
let stepGroups: Array<{ step: ComboRuntimeStep; targets: ResolvedComboTarget[] }> | undefined;
|
|
const weightedEligibleKeys = new Set<string>();
|
|
if (strategy === "weighted") {
|
|
stepGroups = resolveWeightedStepGroups(expandedCombo, expandedAllCombos);
|
|
for (const group of stepGroups) {
|
|
const availability = await Promise.all(group.targets.map(isTargetSelectableForWeighted));
|
|
if (availability.some(Boolean)) weightedEligibleKeys.add(group.step.executionKey);
|
|
}
|
|
}
|
|
const rawStickyWeightedKey =
|
|
strategy === "weighted" ? getStickyWeightedExecutionKey(combo.name, stickyWeightedLimit) : null;
|
|
const stickyWeightedKey =
|
|
rawStickyWeightedKey && weightedEligibleKeys.has(rawStickyWeightedKey)
|
|
? rawStickyWeightedKey
|
|
: null;
|
|
if (strategy !== "weighted" || stickyWeightedLimit <= 1) {
|
|
weightedStickyTargets.delete(combo.name);
|
|
} else if (rawStickyWeightedKey && !stickyWeightedKey) {
|
|
weightedStickyTargets.delete(combo.name);
|
|
}
|
|
const weightedResolution =
|
|
strategy === "weighted"
|
|
? resolveWeightedTargets(
|
|
expandedCombo,
|
|
expandedAllCombos,
|
|
stickyWeightedKey,
|
|
weightedEligibleKeys,
|
|
stepGroups
|
|
)
|
|
: null;
|
|
const getWeightedStepKeyForTarget = (target: ResolvedComboTarget): string | null => {
|
|
if (!weightedResolution?.orderedSteps) return null;
|
|
const step = weightedResolution.orderedSteps.find(
|
|
(entry) =>
|
|
target.executionKey === entry.executionKey ||
|
|
target.executionKey.startsWith(entry.executionKey + ">")
|
|
);
|
|
return step?.executionKey || null;
|
|
};
|
|
let orderedTargets =
|
|
strategy === "weighted"
|
|
? weightedResolution?.orderedTargets || []
|
|
: resolveComboTargets(
|
|
expandedCombo,
|
|
expandedAllCombos,
|
|
clampComboDepth(config.maxComboDepth)
|
|
);
|
|
|
|
orderedTargets = await applyRequestTagRouting(orderedTargets, body, log);
|
|
|
|
const knownContextOverflow = getKnownContextOverflow(orderedTargets, body);
|
|
if (knownContextOverflow) {
|
|
const { requiredContextTokens, maxKnownContextTokens } = knownContextOverflow;
|
|
log.warn(
|
|
"COMBO",
|
|
`Request context exceeds every known target limit (${requiredContextTokens} > ${maxKnownContextTokens} tokens)`
|
|
);
|
|
return errorResponseWithComboDiagnostics(
|
|
400,
|
|
`Request requires approximately ${requiredContextTokens} tokens, but the largest known context limit in this combo is ${maxKnownContextTokens} tokens. Reduce or compact the request context.`,
|
|
{
|
|
poolSize: orderedTargets.length,
|
|
attempted: 0,
|
|
excluded: orderedTargets.map((target) => ({
|
|
provider: target.provider,
|
|
model: target.modelStr,
|
|
reason: "context_window",
|
|
})),
|
|
attemptOrder: [],
|
|
terminalReason: "context_length_exceeded",
|
|
},
|
|
{ code: "context_length_exceeded", type: "invalid_request_error" }
|
|
);
|
|
}
|
|
|
|
if (strategy === "weighted") {
|
|
log.info(
|
|
"COMBO",
|
|
`Weighted selection${stickyWeightedKey ? " (sticky)" : ""}${allCombos ? " with nested resolution" : ""}: ${orderedTargets.length} total targets`
|
|
);
|
|
} else if (allCombos) {
|
|
log.info("COMBO", `${strategy} with nested resolution: ${orderedTargets.length} total targets`);
|
|
}
|
|
|
|
// Pipeline dispatch: route smart/pipeline-enabled combos through the multi-stage pipeline
|
|
if (strategy === "auto") {
|
|
const autoParsed = parseAutoPrefix(combo.name);
|
|
const autoVariant = autoParsed.valid ? autoParsed.variant : undefined;
|
|
if (autoVariant === "smart" || config.pipeline_enabled) {
|
|
try {
|
|
const pipelineRaw = await handlePipelineCombo({
|
|
body,
|
|
combo,
|
|
handleChatCore: handleSingleModelWithTimeout,
|
|
log: {
|
|
info: log.info,
|
|
warn: log.warn,
|
|
error: log.error ?? log.warn,
|
|
},
|
|
settings: settings ?? {},
|
|
signal: signal ?? undefined,
|
|
});
|
|
// handlePipelineCombo resolves to a PipelineResult (buffered text) or,
|
|
// in the streaming-final-stage case, a Response. Callers downstream
|
|
// (chat.ts → withSessionHeader) require a Response, so adapt the
|
|
// PipelineResult here instead of leaking the raw object.
|
|
return pipelineRaw instanceof Response
|
|
? pipelineRaw
|
|
: buildPipelineResponse(pipelineRaw, body);
|
|
} catch (pipelineErr) {
|
|
const pipelineMsg = pipelineErr instanceof Error ? pipelineErr.message : "";
|
|
if (pipelineMsg === "PIPELINE_DISABLED") {
|
|
log.info("COMBO", "Pipeline disabled, falling through to standard auto routing");
|
|
} else if (pipelineMsg === "PIPELINE_TOKEN_THRESHOLD") {
|
|
log.info(
|
|
"COMBO",
|
|
"Pipeline skipped (prompt below token threshold), falling through to standard auto routing"
|
|
);
|
|
} else {
|
|
log.warn("COMBO", "Pipeline dispatch failed, falling through to standard auto routing", {
|
|
err: pipelineErr,
|
|
});
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
// #4945 regression guard: when an "auto" combo uses an EXPLICIT router
|
|
// (routingStrategy lkgp/cost/etc, not the default "rules" scorer), that router
|
|
// pins orderedTargets[0]. The task-aware reordering below must then refine only
|
|
// the fallback order, never override the router's primary choice.
|
|
let autoUsedExplicitRouter = false;
|
|
if (strategy === "auto") {
|
|
const autoResult = await resolveAutoStrategyOrder({
|
|
orderedTargets,
|
|
body,
|
|
combo,
|
|
settings,
|
|
config,
|
|
relayOptions,
|
|
resilienceSettings,
|
|
log,
|
|
buildAutoCandidates,
|
|
});
|
|
if ("earlyResponse" in autoResult) return autoResult.earlyResponse;
|
|
orderedTargets = autoResult.orderedTargets;
|
|
autoUsedExplicitRouter = autoResult.autoUsedExplicitRouter;
|
|
} else {
|
|
orderedTargets = await applyStrategyOrdering(strategy, orderedTargets, {
|
|
combo,
|
|
config,
|
|
body,
|
|
log,
|
|
apiKeyAllowedConnections,
|
|
});
|
|
}
|
|
// #6168: session stickiness opt-out. Per-combo `config.disableSessionStickiness`
|
|
// overrides the global `settings.disableSessionStickiness` fallback (default false,
|
|
// preserving the #3825 prompt-cache/504 fix). When disabled, skip the reorder and
|
|
// treat the result as a no-op so the recordStickyBinding write-back below is skipped.
|
|
const disableSessionStickiness = resolveDisableSessionStickiness(
|
|
config as Record<string, unknown> | null | undefined,
|
|
settings as Record<string, unknown> | null | undefined
|
|
);
|
|
const _sticky = disableSessionStickiness
|
|
? ({ targets: orderedTargets, messageHash: null, stuck: false } as const)
|
|
: await applySessionStickiness(
|
|
orderedTargets,
|
|
// #7270: normalize both wire shapes (.messages / Responses-API .input) so the
|
|
// stickiness key is derivable on the /v1/responses surface, not just Chat Completions.
|
|
normalizeStickinessMessages(body as { messages?: unknown; input?: unknown })
|
|
);
|
|
orderedTargets = _sticky.targets;
|
|
orderedTargets = orderTargetsByEvalScores(orderedTargets, config.evalRouting, log);
|
|
orderedTargets = filterTargetsByRequestCompatibility(orderedTargets, body, log);
|
|
orderedTargets = applyContextRequirements(orderedTargets, config.contextRequirements, log);
|
|
|
|
// Task-aware reordering: only active for strategies ["smart","task","task-aware","task_aware","auto"].
|
|
// Additive — does not affect any of the other 15 strategies.
|
|
if (isTaskRoutingStrategy(strategy)) {
|
|
const task = classifyTask(body);
|
|
const conversationCacheKey = getConversationCacheKey(body);
|
|
const taskReordered = reorderByTaskWeight(orderedTargets, task);
|
|
// #4945 regression guard: when an explicit auto router (lkgp/cost/…) pinned
|
|
// orderedTargets[0], keep that primary choice and let task-aware refine only
|
|
// the fallback tail — otherwise task weighting silently defeats the operator's
|
|
// chosen LKGP/cost selection. reorderByTaskWeight returns the same target
|
|
// objects (no clone), so identity filtering is safe.
|
|
const pinnedFirst = autoUsedExplicitRouter ? orderedTargets[0] : undefined;
|
|
const nextOrder = pinnedFirst
|
|
? [pinnedFirst, ...taskReordered.filter((t) => t !== pinnedFirst)]
|
|
: taskReordered;
|
|
if (nextOrder[0]?.modelStr !== orderedTargets[0]?.modelStr) {
|
|
const reasons =
|
|
Array.isArray(task.reasons) && task.reasons.length > 0
|
|
? ` (${task.reasons.join(",")})`
|
|
: "";
|
|
log.info(
|
|
"COMBO",
|
|
`task-route task=${task.level}${reasons} cacheKey=${conversationCacheKey ?? "none"} → ${nextOrder[0]?.modelStr}`
|
|
);
|
|
}
|
|
orderedTargets = nextOrder;
|
|
}
|
|
|
|
// Parallel pre-screen: check provider profiles and model availability for all targets
|
|
// Only runs for priority strategy where sequential checking causes latency
|
|
const preScreenMap =
|
|
strategy === "priority"
|
|
? await preScreenTargets(orderedTargets, isModelAvailable).catch(
|
|
() => new Map<string, PreScreenResult>()
|
|
)
|
|
: new Map<string, PreScreenResult>();
|
|
|
|
// #5923 (Finding #4) — reset-window config for the shared per-target quota-
|
|
// exhaustion cutoff below. The "auto" strategy already applies its own cutoff
|
|
// via buildAutoCandidates/routableCandidates, so this only affects the other
|
|
// 16 strategies (priority, weighted, etc.) that funnel through executeTarget.
|
|
const quotaCutoffResetWindowConfig = resolveResetWindowConfig(config as Record<string, unknown>);
|
|
|
|
// QA P0 diagnostics: record the order in which targets were actually attempted
|
|
// (provider/model ids only) so a terminal combo failure can report the attempt
|
|
// sequence alongside pool size + exhaustion reasons. Accumulates across set retries.
|
|
const comboAttemptOrder: Array<{ provider: string; model: string }> = [];
|
|
|
|
if (orderedTargets.length === 0) {
|
|
// Surface a recovery hint + auto-clear the session pin after enough consecutive
|
|
// no-target failures (silent-stop fix). Threshold of 3 prevents a one-off account
|
|
// wipe from destroying the prompt-cache pin benefit on the next request.
|
|
recordComboFailure(effectiveSessionId, combo.name);
|
|
return errorResponseWithComboDiagnostics(
|
|
404,
|
|
"Combo has no executable targets",
|
|
{
|
|
poolSize: 0,
|
|
attempted: 0,
|
|
excluded: [],
|
|
attemptOrder: [],
|
|
terminalReason: "no_executable_targets",
|
|
recovery: buildRecoveryHint("no_executable_targets"),
|
|
},
|
|
{ code: "model_not_found", type: "invalid_request_error" }
|
|
);
|
|
}
|
|
|
|
scheduleShadowRouting(
|
|
combo,
|
|
config,
|
|
body,
|
|
resolveShadowTargets(combo, config, allCombos),
|
|
handleSingleModel,
|
|
isModelAvailable,
|
|
strategy,
|
|
log
|
|
);
|
|
|
|
// G2: Collect execution keys registered by _registerExecutionCandidates above (auto strategy).
|
|
// We snapshot them now so cleanup can happen after the attempt loop finishes.
|
|
const _registeredExecutionKeys = orderedTargets.map((t) => t.executionKey).filter(Boolean);
|
|
|
|
let globalAttempts = 0;
|
|
|
|
// Quota-share cooldown-aware retry (Variante A). Only quota-share (qtSd/)
|
|
// combos opt in: when the set loop would crystallize a 429 model_cooldown
|
|
// because the target hit a SHORT transient cooldown, we wait it out and
|
|
// re-run the whole set loop instead of propagating the 429. `globalAttempts`
|
|
// persists across these waits so MAX_GLOBAL_ATTEMPTS still bounds total work.
|
|
// The wait happens at the crystallization point. The only semaphore slot the
|
|
// quota-share path may hold is the FASE 2.1 per-connection concurrency slot
|
|
// (acquired once around dispatchWithCooldownRetry below); it is intentionally
|
|
// kept across the wait so the account stays "busy", and is released by the
|
|
// outer finally — not here.
|
|
//
|
|
// The set loop is wrapped in a small recursive closure rather than an extra
|
|
// labelled `while (true)` so the loop body keeps its original indentation; a
|
|
// wait+redispatch is a tail `return dispatchWithCooldownRetry()`, which
|
|
// re-runs ONLY the set loop (selection / shadow routing / setup above stay
|
|
// untouched), preserving the pre-existing `continue`-to-top-of-set-loop
|
|
// semantics exactly.
|
|
const comboCooldownWaitEnabled =
|
|
strategy === "quota-share" && resilienceSettings.comboCooldownWait.enabled;
|
|
let comboCooldownAttempt = 0;
|
|
let comboCooldownBudgetLeftMs = resilienceSettings.comboCooldownWait.budgetMs;
|
|
|
|
// FASE 2.1: per-connection concurrency limit for quota-share. The gating in
|
|
// selectQuotaShareTarget is fail-open and cannot hard-limit a single-connection
|
|
// pool, so we serialize concurrent requests to the selected account through a
|
|
// per-connection semaphore. Enabled only for quota-share combos (the cap is the
|
|
// account's) and gated by the kill-switch; the slot wraps the whole dispatch.
|
|
const quotaShareConcurrencyEnabled =
|
|
strategy === "quota-share" && resilienceSettings.quotaShareConcurrencyLimit.enabled;
|
|
|
|
const dispatchWithCooldownRetry = async (): Promise<Response> => {
|
|
for (let setTry = 0; setTry <= maxSetRetries; setTry++) {
|
|
// #1731: Per-set-iteration set of providers whose quota is fully exhausted.
|
|
// Reset each retry so providers excluded in a previous attempt get another chance.
|
|
const exhaustedProviders = new Set<string>();
|
|
const exhaustedConnections = new Set<string>();
|
|
const transientRateLimitedProviders = new Set<string>();
|
|
if (setTry > 0) {
|
|
log.info("COMBO", `All targets failed — retrying set (${setTry}/${maxSetRetries})`);
|
|
await new Promise((resolve) => {
|
|
const timer = setTimeout(resolve, setRetryDelayMs);
|
|
signal?.addEventListener(
|
|
"abort",
|
|
() => {
|
|
clearTimeout(timer);
|
|
resolve(undefined);
|
|
},
|
|
{ once: true }
|
|
);
|
|
});
|
|
if (signal?.aborted) {
|
|
log.info("COMBO", "Client disconnected during set retry delay — aborting");
|
|
return errorResponse(499, "Client disconnected");
|
|
}
|
|
}
|
|
|
|
let lastError: string | null = null;
|
|
let earliestRetryAfter: ComboRetryAfter | null = null;
|
|
let lastStatus: number | null = null;
|
|
const startTime = Date.now();
|
|
let fallbackCount = 0;
|
|
let recordedAttempts = 0;
|
|
|
|
// QA P0: assemble a sanitized diagnostic trace from the state already in scope
|
|
// (pool size + this set-try's exhausted providers/connections + attempt order +
|
|
// a terminal-reason code). Never touches keys/tokens — provider/model ids only.
|
|
// Silent-stop fix: include a `recovery` hint (action verb + human next-step) so the
|
|
// OC plugin + non-header-aware clients can render an actionable error instead of an
|
|
// opaque 5xx. The optional `retryAfterSeconds` carries the upstream Retry-After hint.
|
|
const buildComboDiag = (
|
|
terminalReason: string,
|
|
retryAfterSeconds?: number
|
|
): ComboDiagnostics => ({
|
|
poolSize: orderedTargets.length,
|
|
attempted: recordedAttempts,
|
|
excluded: [
|
|
...[...exhaustedProviders].map((p) => ({ provider: p, reason: "exhausted" })),
|
|
...[...exhaustedConnections].map((c) => ({
|
|
provider: "unknown",
|
|
reason: `exhausted_connection:${String(c).slice(0, 8)}`,
|
|
})),
|
|
],
|
|
attemptOrder: comboAttemptOrder,
|
|
terminalReason,
|
|
recovery: buildRecoveryHint(terminalReason, retryAfterSeconds),
|
|
});
|
|
|
|
let globalResolve: ((res: Response) => void) | null = null;
|
|
const globalPromise = new Promise<Response>((res) => {
|
|
globalResolve = res;
|
|
});
|
|
const runningTasks = new Set<Promise<void>>();
|
|
let anySuccess = false;
|
|
const abortControllers = new Map<number, AbortController>();
|
|
const zeroLatencyOptimizationsEnabled = config.zeroLatencyOptimizationsEnabled === true;
|
|
|
|
const executeTarget = async (
|
|
i: number
|
|
): Promise<{ ok: boolean; response?: Response } | null> => {
|
|
const target = orderedTargets[i];
|
|
const modelStr = target.modelStr;
|
|
const rawModel = parseModel(modelStr).model || modelStr;
|
|
const provider = target.provider;
|
|
|
|
const cb = getCircuitBreaker(provider);
|
|
if (cb.getStatus().state === "OPEN") {
|
|
log.info("COMBO", `Skipping ${modelStr} — circuit breaker OPEN for ${provider}`);
|
|
if (i > 0) fallbackCount++;
|
|
return null;
|
|
}
|
|
|
|
if (
|
|
resilienceSettings.providerCooldown.enabled &&
|
|
Boolean(provider && provider !== "unknown") &&
|
|
isProviderInCooldown(provider, target.connectionId ?? undefined, resilienceSettings)
|
|
) {
|
|
log.info("COMBO", `Skipping ${modelStr} — provider ${provider} in global cooldown`);
|
|
if (i > 0) fallbackCount++;
|
|
return null;
|
|
}
|
|
|
|
// Use pre-screened profile if available, otherwise fetch on demand
|
|
const preScreenEntry = preScreenMap.get(target.executionKey);
|
|
const profile = preScreenEntry?.profile ?? (await getRuntimeProviderProfile(provider));
|
|
|
|
const allowRateLimitedConnection =
|
|
Boolean(provider && provider !== "unknown") &&
|
|
transientRateLimitedProviders.has(provider);
|
|
const targetForAttempt = allowRateLimitedConnection
|
|
? {
|
|
...target,
|
|
allowRateLimitedConnection: true,
|
|
modelAbortSignal: abortControllers.get(i)!.signal,
|
|
}
|
|
: { ...target, modelAbortSignal: abortControllers.get(i)!.signal };
|
|
|
|
// #1731 / #1731v2: skip targets already known-exhausted this request (shared predicate).
|
|
const exhaustedSkip = getExhaustedTargetSkipReason(
|
|
target,
|
|
exhaustedProviders,
|
|
exhaustedConnections
|
|
);
|
|
if (exhaustedSkip) {
|
|
log.info("COMBO", exhaustedSkip);
|
|
if (i > 0) fallbackCount++;
|
|
return null;
|
|
}
|
|
|
|
// Pre-check: skip models locked by the resilience system (model-level lockout)
|
|
if (provider && rawModel && isModelLocked(provider, target.connectionId || "", rawModel)) {
|
|
log.info("COMBO", `Skipping ${modelStr} — model locked by resilience (cooldown active)`);
|
|
if (i > 0) fallbackCount++;
|
|
return null;
|
|
}
|
|
|
|
// #5923 (Finding #4) — honor the same opt-in quota-exhaustion cutoff the
|
|
// "auto" strategy already applies (buildAutoCandidates), for every other
|
|
// strategy (priority, weighted, etc.). Strictly scoped per (provider,
|
|
// connectionId): a 0%-remaining connection is skipped here, but sibling
|
|
// connections/models on the same provider are untouched — the provider
|
|
// circuit breaker is never touched by this check. The "auto" strategy is
|
|
// excluded to avoid a redundant duplicate fetch — it already filtered its
|
|
// candidate pool via `routableCandidates` before reaching this loop.
|
|
if (strategy !== "auto" && provider && target.connectionId) {
|
|
const quotaCutoff = await resolveQuotaExhaustionCutoffForTarget(
|
|
provider,
|
|
target.connectionId,
|
|
resilienceSettings,
|
|
quotaCutoffResetWindowConfig,
|
|
combo.name,
|
|
log
|
|
);
|
|
if (quotaCutoff.blocked) {
|
|
log.info(
|
|
"COMBO",
|
|
`Skipping ${modelStr} — quota exhaustion cutoff (${quotaCutoff.reason || "quota_exhausted"})`
|
|
);
|
|
if (i > 0) fallbackCount++;
|
|
return null;
|
|
}
|
|
}
|
|
|
|
// Pre-screen snapshot is NOT used as a permanent skip — availability
|
|
// is always re-checked via isModelAvailable below because connection
|
|
// cooldowns can expire between setTry retries, making a previously
|
|
// unavailable target available again. Circuit-breaker-OPEN providers
|
|
// are already caught by the dedicated breaker check above.
|
|
if (isModelAvailable) {
|
|
const available = await isModelAvailable(modelStr, targetForAttempt);
|
|
if (!available) {
|
|
log.debug?.(
|
|
"COMBO",
|
|
`Skipping ${modelStr} — no credentials available or model excluded`
|
|
);
|
|
if (i > 0) fallbackCount++;
|
|
return null;
|
|
}
|
|
}
|
|
|
|
// Credential gate: skip targets with known-bad credentials (fail-fast)
|
|
const connectionId = target.connectionId as string | undefined;
|
|
if (connectionId) {
|
|
const gateResult = checkCredentialGate(connectionId, provider, modelStr);
|
|
if (gateResult.allowed === false) {
|
|
logCredentialSkip(log, modelStr, gateResult.reason || "Credential gate blocked");
|
|
if (i > 0) fallbackCount++;
|
|
return null;
|
|
}
|
|
}
|
|
|
|
// Retry loop for transient errors
|
|
for (let retry = 0; retry <= maxRetries; retry++) {
|
|
// Fix #1681: Bail out immediately if the client has disconnected
|
|
if (signal?.aborted) {
|
|
log.info("COMBO", `Client disconnected — aborting combo loop before model ${modelStr}`);
|
|
return { ok: false, response: errorResponse(499, "Client disconnected") };
|
|
}
|
|
globalAttempts++;
|
|
if (globalAttempts > MAX_GLOBAL_ATTEMPTS) {
|
|
log.warn(
|
|
"COMBO",
|
|
`Maximum combo attempts (${MAX_GLOBAL_ATTEMPTS}) exceeded across all targets and fallbacks. Terminating loop to prevent runaway background requests.`
|
|
);
|
|
// Actionable failure instead of an opaque 503 when every candidate
|
|
// failed the same recoverable way. If the dominant cause was reasoning
|
|
// models exhausting a too-small max_tokens budget (no content output),
|
|
// retrying other models can't help — tell the caller to raise max_tokens.
|
|
// Silent-stop fix: bump the consecutive-failure counter for this session-combo pair
|
|
// so the pin gets cleared on the 3rd attempt (recovery.next_step tells the client).
|
|
const reasoningExhausted = /reasoning consumed \d+\/\d+ tokens/.test(lastError || "");
|
|
const failureReason = reasoningExhausted
|
|
? "reasoning_budget_exhausted"
|
|
: "max_attempts_exceeded";
|
|
recordComboFailure(effectiveSessionId, combo.name);
|
|
return {
|
|
ok: false,
|
|
response: errorResponseWithComboDiagnostics(
|
|
503,
|
|
reasoningExhausted
|
|
? "All combo candidates exhausted their token budget on reasoning without producing content. Increase max_tokens — reasoning models need a larger budget to emit content."
|
|
: "Maximum combo retry limit reached",
|
|
buildComboDiag(failureReason)
|
|
),
|
|
};
|
|
}
|
|
// Predictive TTFT Circuit Breaker (skip slow models)
|
|
if (
|
|
zeroLatencyOptimizationsEnabled &&
|
|
config.predictiveTtftMs &&
|
|
config.predictiveTtftMs > 0 &&
|
|
retry === 0
|
|
) {
|
|
const cMetrics = getComboMetrics(combo.name);
|
|
if (cMetrics) {
|
|
const targetKey = orderedTargets[i].executionKey || modelStr;
|
|
const m = cMetrics.byTarget[targetKey] || cMetrics.byModel[modelStr];
|
|
if (shouldSkipForPredictedTtft(m, config.predictiveTtftMs)) {
|
|
log.warn(
|
|
"COMBO",
|
|
`Predictive TTFT Circuit Breaker: skipping ${modelStr} (avg ${m.avgLatencyMs}ms > max ${config.predictiveTtftMs}ms)`
|
|
);
|
|
return null;
|
|
}
|
|
}
|
|
}
|
|
|
|
if (retry > 0) {
|
|
log.info(
|
|
"COMBO",
|
|
`Retrying ${modelStr} in ${retryDelayMs}ms (attempt ${retry + 1}/${maxRetries + 1})`
|
|
);
|
|
await new Promise((resolve) => {
|
|
const timer = setTimeout(resolve, retryDelayMs);
|
|
signal?.addEventListener(
|
|
"abort",
|
|
() => {
|
|
clearTimeout(timer);
|
|
resolve(undefined);
|
|
},
|
|
{ once: true }
|
|
);
|
|
});
|
|
if (signal?.aborted) {
|
|
log.info("COMBO", `Client disconnected during retry delay — aborting`);
|
|
return { ok: false, response: errorResponse(499, "Client disconnected") };
|
|
}
|
|
}
|
|
|
|
log.info(
|
|
"COMBO",
|
|
`Trying model ${i + 1}/${orderedTargets.length}: ${modelStr}${retry > 0 ? ` (retry ${retry})` : ""}`
|
|
);
|
|
emit("combo.target.attempt", {
|
|
comboName: combo.name,
|
|
targetIndex: i,
|
|
provider,
|
|
model: modelStr,
|
|
timestamp: Date.now(),
|
|
strategy,
|
|
});
|
|
// QA P0 diagnostics: capture the attempt order (provider/model ids only).
|
|
comboAttemptOrder.push({ provider: provider ?? "unknown", model: modelStr });
|
|
|
|
// Deep clone the body to ensure context preservation and prevent mutations
|
|
// from affecting other targets in the combo. structuredClone avoids the
|
|
// full intermediate JSON string that JSON.parse(JSON.stringify(...)) builds
|
|
// (a second multi-hundred-KB allocation per target on large agent payloads),
|
|
// halving the per-target transient heap on the hot path (#5152).
|
|
let attemptBody = structuredClone(body);
|
|
|
|
// Proactive Context Compression for fallbacks (Zero-Latency optimization)
|
|
if (
|
|
zeroLatencyOptimizationsEnabled &&
|
|
i > 0 &&
|
|
config.fallbackCompressionMode &&
|
|
config.fallbackCompressionMode !== "off"
|
|
) {
|
|
const { estimateTokens } = await import("./contextManager.ts");
|
|
const estimatedTokens = estimateTokens(JSON.stringify(attemptBody));
|
|
if (estimatedTokens > (config.fallbackCompressionThreshold ?? 1000)) {
|
|
const { applyCompression } = await import("./compression/strategySelector.ts");
|
|
const compressionResult = applyCompression(
|
|
attemptBody,
|
|
config.fallbackCompressionMode as CompressionMode,
|
|
// Opt into the TV1 bail-out so a throwing fallback engine is SKIPPED rather than
|
|
// propagating out of executeTarget and being swallowed as a "Speculative task
|
|
// error" (which silently drops this combo target). minGainPercent:0 keeps the
|
|
// advance behavior identical to the default path — this only adds skip-on-throw.
|
|
{ model: modelStr, bailout: { enabled: true, minGainPercent: 0 } }
|
|
);
|
|
if (compressionResult.compressed) {
|
|
log.info(
|
|
"COMBO",
|
|
`Proactive fallback compression applied (${config.fallbackCompressionMode}): ${estimatedTokens} -> ${compressionResult.stats?.compressedTokens} tokens`
|
|
);
|
|
attemptBody = compressionResult.body;
|
|
}
|
|
}
|
|
}
|
|
|
|
// Universal handoff: inject existing handoff if model changed
|
|
if (
|
|
universalHandoffConfig.enabled &&
|
|
relayOptions?.sessionId &&
|
|
!(body as Record<string, unknown>)?.[SKIP_UNIVERSAL_HANDOFF_FLAG]
|
|
) {
|
|
const lastModel = getLastSessionModel(relayOptions.sessionId, combo.name);
|
|
if (lastModel && lastModel !== modelStr) {
|
|
const existingHandoff = getHandoff(relayOptions.sessionId, combo.name);
|
|
attemptBody = injectUniversalHandoffBody(
|
|
attemptBody, // Use the cloned body to maintain isolation
|
|
lastModel,
|
|
modelStr,
|
|
`Model routing: ${lastModel} → ${modelStr}`,
|
|
existingHandoff
|
|
);
|
|
}
|
|
}
|
|
|
|
// Issue #3587: Reasoning models can spend the whole output budget on
|
|
// reasoning. Only add headroom when the complete buffer fits inside the
|
|
// model's known output cap; otherwise preserve the client's explicit limit.
|
|
{
|
|
const bodyRecord = attemptBody as Record<string, unknown>;
|
|
const currentMaxTokens = toPositiveInteger(bodyRecord.max_tokens);
|
|
const bufferedMaxTokens = resolveReasoningBufferedMaxTokens(
|
|
modelStr,
|
|
bodyRecord.max_tokens,
|
|
{ enabled: reasoningTokenBufferEnabled }
|
|
);
|
|
if (currentMaxTokens !== null && bufferedMaxTokens !== null) {
|
|
bodyRecord.max_tokens = bufferedMaxTokens;
|
|
if (bufferedMaxTokens !== currentMaxTokens) {
|
|
log.info(
|
|
"COMBO",
|
|
`Reasoning model ${modelStr}: adjusted max_tokens ${currentMaxTokens} -> ${bufferedMaxTokens}`
|
|
);
|
|
}
|
|
}
|
|
}
|
|
const result = await handleSingleModelWithTimeout(attemptBody, modelStr, {
|
|
...targetForAttempt,
|
|
effectiveComboStrategy: strategy,
|
|
failoverBeforeRetry: config.failoverBeforeRetry,
|
|
});
|
|
|
|
// Success — validate response quality before returning
|
|
if (result.ok) {
|
|
const selectedConnectionId =
|
|
result.headers?.get("X-OmniRoute-Selected-Connection-Id") ||
|
|
result.headers?.get("x-omniroute-selected-connection-id") ||
|
|
undefined;
|
|
const effectiveConnectionId = selectedConnectionId || target.connectionId || "";
|
|
|
|
// Clone BEFORE quality check — validateResponseQuality reads the body
|
|
// via getReader() which locks the stream. The clone's body is consumed
|
|
// by the quality check; the original stays unlocked for piping.
|
|
let qualityClone: Response;
|
|
try {
|
|
qualityClone = result.clone();
|
|
} catch {
|
|
qualityClone = result;
|
|
}
|
|
const quality = await validateResponseQuality(
|
|
qualityClone,
|
|
clientRequestedStream,
|
|
log,
|
|
config.responseValidation
|
|
);
|
|
releaseQualityClone(qualityClone, result, quality);
|
|
if (!quality.valid) {
|
|
releaseRejectedQualityResponse(qualityClone, result);
|
|
log.warn(
|
|
"COMBO",
|
|
`Model ${modelStr} returned 200 but failed quality check: ${quality.reason}`
|
|
);
|
|
// #6692: a quality-rejected 200 never marks the connection row
|
|
// unhealthy, so the sticky pin's lazy headroom recheck would never
|
|
// catch it either — release it here, on the failing response.
|
|
releaseStickyPinOnFailure(_sticky.messageHash, effectiveConnectionId);
|
|
recordComboRequest(combo.name, modelStr, {
|
|
success: false,
|
|
latencyMs: Date.now() - startTime,
|
|
fallbackCount,
|
|
strategy,
|
|
target: toRecordedTarget(target),
|
|
});
|
|
recordedAttempts++;
|
|
// Fix #1707: Set terminal state so the fallback doesn't emit
|
|
// misleading ALL_ACCOUNTS_INACTIVE when the real issue is quality.
|
|
lastError = `Upstream response failed quality validation: ${quality.reason}`;
|
|
if (!lastStatus) lastStatus = 502;
|
|
if (i > 0) fallbackCount++;
|
|
if (provider && rawModel) {
|
|
const mlSettings = resolveModelLockoutSettings(settings);
|
|
if (mlSettings.enabled && mlSettings.errorCodes.includes(502)) {
|
|
recordModelLockoutFailure(
|
|
provider,
|
|
target.connectionId || "",
|
|
rawModel,
|
|
"quality_failure",
|
|
502,
|
|
mlSettings.baseCooldownMs,
|
|
profile,
|
|
{
|
|
exactCooldownMs: mlSettings.useExponentialBackoff
|
|
? 0
|
|
: mlSettings.baseCooldownMs,
|
|
maxCooldownMs: mlSettings.maxCooldownMs,
|
|
}
|
|
);
|
|
}
|
|
}
|
|
emit("combo.target.failed", {
|
|
comboName: combo.name,
|
|
targetIndex: i,
|
|
provider,
|
|
model: modelStr,
|
|
error: `Quality: ${quality.reason}`,
|
|
latencyMs: Date.now() - startTime,
|
|
});
|
|
return null;
|
|
}
|
|
|
|
// Success decay: a healthy response walks the model's lockout failure
|
|
// count back down (and eventually clears an expired lockout entirely).
|
|
if (provider && rawModel) {
|
|
const dcResult = decayModelFailureCount(provider, effectiveConnectionId, rawModel);
|
|
if (dcResult.cleared) {
|
|
log.info("COMBO", `Model ${modelStr} fully recovered — lockout cleared`);
|
|
} else if (dcResult.newFailureCount > 0) {
|
|
log.debug(
|
|
"COMBO",
|
|
`Model ${modelStr} decayed to failureCount=${dcResult.newFailureCount}`
|
|
);
|
|
}
|
|
}
|
|
|
|
const latencyMs = Date.now() - startTime;
|
|
emit("combo.target.succeeded", {
|
|
comboName: combo.name,
|
|
targetIndex: i,
|
|
provider,
|
|
model: modelStr,
|
|
latencyMs,
|
|
});
|
|
log.info(
|
|
"COMBO",
|
|
`Model ${modelStr} succeeded (${latencyMs}ms, ${fallbackCount} fallbacks)`
|
|
);
|
|
recordComboRequest(combo.name, modelStr, {
|
|
success: true,
|
|
latencyMs,
|
|
fallbackCount,
|
|
strategy,
|
|
target: toRecordedTarget(target),
|
|
});
|
|
recordedAttempts++;
|
|
|
|
// Reset cooldown on success
|
|
if (provider && provider !== "unknown") {
|
|
recordProviderSuccess(provider, effectiveConnectionId || undefined);
|
|
}
|
|
if (strategy === "weighted" && stickyWeightedLimit > 1) {
|
|
const stickySuccessKey = getWeightedStepKeyForTarget(target);
|
|
if (stickySuccessKey) {
|
|
recordStickyWeightedSuccess(combo.name, stickySuccessKey, stickyWeightedLimit);
|
|
}
|
|
}
|
|
// Webhook fan-out: best-effort, never blocks the response stream.
|
|
notifyWebhookEvent("request.completed", {
|
|
combo: combo.name,
|
|
provider,
|
|
model: modelStr,
|
|
account:
|
|
typeof target.label === "string" && target.label.trim().length > 0
|
|
? target.label.trim()
|
|
: "",
|
|
accountId: effectiveConnectionId ?? "",
|
|
latencyMs,
|
|
fallbackCount,
|
|
});
|
|
|
|
// Silent-stop fix: reset the consecutive-failure counter for this session-combo pair
|
|
// on every successful dispatch so a transient recovery doesn't get "credited" against
|
|
// the threshold the user already paid through to clear the stale pin.
|
|
if (effectiveSessionId) {
|
|
clearComboFailureTracking(effectiveSessionId, combo.name);
|
|
}
|
|
// Context cache pinning: record model usage for session-based pinning
|
|
// (independent of universal handoff — always fires when context_cache_protection is on)
|
|
// #3825: write under the SAME effectiveSessionId used by the read site so a
|
|
// sessionless conversation re-pins to this model on its next turn.
|
|
if (
|
|
combo.context_cache_protection &&
|
|
effectiveSessionId &&
|
|
!(body as Record<string, unknown>)?.[SKIP_UNIVERSAL_HANDOFF_FLAG]
|
|
) {
|
|
recordSessionModelUsage(
|
|
effectiveSessionId,
|
|
combo.name,
|
|
modelStr,
|
|
provider,
|
|
target.connectionId ?? undefined
|
|
);
|
|
}
|
|
|
|
// Universal handoff: record model usage for session
|
|
if (
|
|
universalHandoffConfig.enabled &&
|
|
relayOptions?.sessionId &&
|
|
!(body as Record<string, unknown>)?.[SKIP_UNIVERSAL_HANDOFF_FLAG]
|
|
) {
|
|
const prevModel = getLastSessionModel(relayOptions.sessionId, combo.name);
|
|
recordSessionModelUsage(
|
|
relayOptions.sessionId,
|
|
combo.name,
|
|
modelStr,
|
|
provider,
|
|
target.connectionId ?? undefined
|
|
);
|
|
if (prevModel && prevModel !== modelStr) {
|
|
const handoffSourceMessages =
|
|
Array.isArray(body?.messages) && body.messages.length > 0
|
|
? body.messages
|
|
: Array.isArray(body?.input)
|
|
? body.input
|
|
: [];
|
|
|
|
maybeGenerateUniversalHandoff({
|
|
sessionId: relayOptions.sessionId,
|
|
comboName: combo.name,
|
|
messages: handoffSourceMessages as MessageLike[],
|
|
prevModel,
|
|
currModel: modelStr,
|
|
universalConfig: universalHandoffConfig,
|
|
handleSingleModel: handleSingleModelWithTimeout,
|
|
});
|
|
}
|
|
|
|
recordSessionModelUsage(
|
|
relayOptions.sessionId,
|
|
combo.name,
|
|
modelStr,
|
|
provider,
|
|
target.connectionId ?? undefined
|
|
);
|
|
}
|
|
// Context-relay intentionally splits responsibilities:
|
|
// combo.ts decides whether a successful turn should generate a handoff,
|
|
// while chat.ts injects the handoff after the real connectionId is resolved.
|
|
if (
|
|
strategy === "context-relay" &&
|
|
relayOptions?.sessionId &&
|
|
relayConfig &&
|
|
relayConfig.handoffProviders.includes(provider) &&
|
|
provider === "codex"
|
|
) {
|
|
const connectionId = getSessionConnection(relayOptions.sessionId);
|
|
if (connectionId) {
|
|
const quotaInfo = await fetchCodexQuota(connectionId).catch(() => null);
|
|
if (quotaInfo) {
|
|
const resetCandidates = [
|
|
quotaInfo.windows?.session?.resetAt,
|
|
quotaInfo.windows?.weekly?.resetAt,
|
|
quotaInfo.resetAt,
|
|
]
|
|
.filter(
|
|
(value): value is string => typeof value === "string" && value.length > 0
|
|
)
|
|
.sort((a, b) => a.localeCompare(b));
|
|
const handoffSourceMessages =
|
|
Array.isArray(body?.messages) && body.messages.length > 0
|
|
? body.messages
|
|
: Array.isArray(body?.input)
|
|
? body.input
|
|
: [];
|
|
|
|
maybeGenerateHandoff({
|
|
sessionId: relayOptions.sessionId,
|
|
comboName: combo.name,
|
|
connectionId,
|
|
percentUsed: quotaInfo.percentUsed,
|
|
messages: handoffSourceMessages,
|
|
model: modelStr,
|
|
expiresAt: resetCandidates[0] || null,
|
|
config: relayConfig,
|
|
handleSingleModel: handleSingleModelWithTimeout,
|
|
});
|
|
}
|
|
}
|
|
}
|
|
if (_sticky.messageHash && target.connectionId)
|
|
recordStickyBinding(_sticky.messageHash, target.connectionId); // LKGP (#919):
|
|
if (provider) {
|
|
const connId = effectiveConnectionId || undefined;
|
|
void (async () => {
|
|
try {
|
|
const { setLKGP } = await import("../../src/lib/localDb");
|
|
await Promise.all([
|
|
setLKGP(combo.name, target.executionKey, provider, connId),
|
|
setLKGP(combo.name, combo.id || combo.name, provider, connId),
|
|
]);
|
|
} catch (err) {
|
|
log.warn(
|
|
"COMBO",
|
|
"Failed to record Last Known Good Provider. This is non-fatal.",
|
|
{
|
|
err,
|
|
}
|
|
);
|
|
}
|
|
})();
|
|
}
|
|
|
|
return { ok: true, response: result };
|
|
}
|
|
|
|
// Extract error info from response
|
|
let errorText = result.statusText || "";
|
|
let errorBody: ComboErrorBody = null;
|
|
let retryAfter: ComboRetryAfter | null = null;
|
|
try {
|
|
const cloned = result.clone();
|
|
try {
|
|
const text = await cloned.text();
|
|
if (text) {
|
|
errorText = text.substring(0, 500);
|
|
errorBody = JSON.parse(text);
|
|
const parsedError = errorBody?.error;
|
|
errorText =
|
|
(typeof parsedError === "object" && parsedError?.message) ||
|
|
(typeof parsedError === "string" ? parsedError : null) ||
|
|
errorBody?.message ||
|
|
errorText;
|
|
retryAfter = errorBody?.retryAfter || null;
|
|
}
|
|
} catch {
|
|
/* Clone parse failed */
|
|
}
|
|
} catch {
|
|
/* Clone failed */
|
|
}
|
|
|
|
// Track earliest retryAfter
|
|
if (
|
|
retryAfter &&
|
|
(!earliestRetryAfter || new Date(retryAfter) < new Date(earliestRetryAfter))
|
|
) {
|
|
earliestRetryAfter = retryAfter;
|
|
}
|
|
|
|
// Normalize error text
|
|
if (typeof errorText !== "string") {
|
|
try {
|
|
errorText = JSON.stringify(errorText);
|
|
} catch {
|
|
errorText = String(errorText);
|
|
}
|
|
}
|
|
|
|
const isStreamReadinessFailure =
|
|
(result.status === 502 || result.status === 504) &&
|
|
isStreamReadinessFailureErrorBody(errorBody);
|
|
|
|
// FIX 5: a local per-API-key token-limit 429 must not cool shared accounts.
|
|
const isTokenLimitBreach =
|
|
result.status === 429 && isTokenLimitBreachErrorBody(errorBody);
|
|
|
|
// Fix #1681: Status 499 means client disconnected — stop combo loop immediately.
|
|
// There is no point trying fallback models when nobody is listening.
|
|
if (result.status === 499) {
|
|
log.info("COMBO", `Client disconnected (499) during ${modelStr} — stopping combo loop`);
|
|
recordComboRequest(combo.name, modelStr, {
|
|
success: false,
|
|
latencyMs: Date.now() - startTime,
|
|
fallbackCount,
|
|
strategy,
|
|
target: toRecordedTarget(target),
|
|
});
|
|
recordedAttempts++;
|
|
// executeTarget must return the {ok,response} contract — a raw Response
|
|
// here makes the speculative loop's res.ok/res.response checks both miss,
|
|
// so the combo would wrongly fall through to the next model after a 499.
|
|
return { ok: false, response: result };
|
|
}
|
|
|
|
// Combo fallback is target-level orchestration: a non-ok target response is
|
|
// treated as local to that target and the combo continues to the next target.
|
|
// Error classification is retained only for retry/cooldown pacing; it must
|
|
// not decide whether fallback happens, including for generic 400 responses.
|
|
const rawError = errorBody?.error;
|
|
const structuredError =
|
|
rawError && typeof rawError === "object"
|
|
? {
|
|
// Upstream JSON may carry a numeric `code`/`type` (e.g. {"code":40001}).
|
|
// Coerce to string if present instead of discarding, so downstream string
|
|
// ops (.toLowerCase, .startsWith) can run safely without type crashes.
|
|
code:
|
|
(rawError as Record<string, unknown>).code !== undefined &&
|
|
(rawError as Record<string, unknown>).code !== null
|
|
? String((rawError as Record<string, unknown>).code)
|
|
: undefined,
|
|
type:
|
|
(rawError as Record<string, unknown>).type !== undefined &&
|
|
(rawError as Record<string, unknown>).type !== null
|
|
? String((rawError as Record<string, unknown>).type)
|
|
: undefined,
|
|
}
|
|
: undefined;
|
|
const requestScopedFailure = isRequestScopedUpstreamFailure(structuredError);
|
|
const fallbackResult = checkFallbackError(
|
|
result.status,
|
|
errorText,
|
|
0,
|
|
null,
|
|
provider,
|
|
result.headers,
|
|
profile,
|
|
structuredError
|
|
);
|
|
const { cooldownMs } = fallbackResult;
|
|
// #6863: a parsed upstream quota reset (e.g. Antigravity "Resets in 92h27m28s")
|
|
// arrives in `quotaResetHintMs` — it bypasses the operator-gated
|
|
// `useUpstreamRetryHints` connection-cooldown setting. Mirror the
|
|
// single-model path (src/sse/services/auth.ts): when the retry hint was
|
|
// already honored, `cooldownMs` IS the upstream value; otherwise prefer the
|
|
// parsed quota reset — even when it is SHORTER than the fallback cooldown
|
|
// (e.g. subscription-quota 1h default vs a real "resets in 10m").
|
|
// `selectLockoutCooldownMs` still ignores hints at/below the base cooldown,
|
|
// so absent/tiny hints keep the #1308 exponential-backoff behavior.
|
|
const lockoutHintMs =
|
|
fallbackResult.usedUpstreamRetryHint === true
|
|
? cooldownMs
|
|
: (fallbackResult.quotaResetHintMs ?? 0);
|
|
const selectedConnectionId =
|
|
result.headers?.get("X-OmniRoute-Selected-Connection-Id") ||
|
|
result.headers?.get("x-omniroute-selected-connection-id") ||
|
|
undefined;
|
|
const targetWithConnection = selectedConnectionId
|
|
? { ...target, connectionId: selectedConnectionId }
|
|
: target;
|
|
|
|
// #1731 / #1731v2: classify the upstream error and update the exhaustion sets
|
|
// (shared with handleRoundRobinCombo). Returns whether the provider is fully exhausted.
|
|
const providerExhausted = applyComboTargetExhaustion(targetWithConnection, {
|
|
result,
|
|
fallbackResult,
|
|
errorText,
|
|
rawModel,
|
|
isTokenLimitBreach,
|
|
allAccountsRateLimited: false,
|
|
sets: { exhaustedProviders, exhaustedConnections, transientRateLimitedProviders },
|
|
log,
|
|
tag: "COMBO",
|
|
exhaustedLogLevel: "info",
|
|
structuredError,
|
|
});
|
|
// #6692: this connection was just classified as provider/connection-level
|
|
// exhausted — if it's the currently sticky-bound one, release the pin now
|
|
// rather than waiting for the next turn's lazy headroom/status recheck.
|
|
releaseStickyPinOnFailure(_sticky.messageHash, targetWithConnection.connectionId);
|
|
|
|
// #2101: Prevent infinite fallback loops with 400 Bad Request errors that are genuinely
|
|
// body-specific (malformed JSON, bad format, missing required fields).
|
|
// Context overflow and parameter validation errors are NOT body-specific:
|
|
// - Context overflow: different models have different context windows
|
|
// - Max_tokens / param errors: different models have different output limits
|
|
// - Model access denied: different providers serve different model sets
|
|
// These should fall through so the next combo target can try.
|
|
if (
|
|
result.status === 400 &&
|
|
fallbackResult.shouldFallback &&
|
|
!isContextOverflow400(errorText) &&
|
|
!isParamValidation400(errorText) &&
|
|
(errorText.toLowerCase().includes("context") ||
|
|
errorText.toLowerCase().includes("prompt") ||
|
|
errorText.toLowerCase().includes("token") ||
|
|
errorText.toLowerCase().includes("malformed") ||
|
|
errorText.toLowerCase().includes("invalid") ||
|
|
errorText.toLowerCase().includes("bad request"))
|
|
) {
|
|
log.warn(
|
|
"COMBO",
|
|
`400 Bad Request with body-specific error detected on ${modelStr} — skipping fallback to other targets to prevent infinite loop`
|
|
);
|
|
// Record the failure and break to avoid trying other targets with the same bad request
|
|
recordComboRequest(combo.name, modelStr, {
|
|
success: false,
|
|
latencyMs: Date.now() - startTime,
|
|
fallbackCount,
|
|
strategy,
|
|
target: toRecordedTarget(target),
|
|
});
|
|
recordedAttempts++;
|
|
lastError = errorText || String(result.status);
|
|
if (!lastStatus) lastStatus = result.status;
|
|
if (i > 0) fallbackCount++;
|
|
log.warn("COMBO", `Model ${modelStr} failed with body-specific error, stopping combo`);
|
|
// #4279: surface the 400 via the {ok,response} contract so the OUTER
|
|
// target loop resolves the combo and stops. A bare `break` here only
|
|
// exits the inner retry loop; executeTarget then returns null, which
|
|
// the outer loop treats as "this target produced nothing" and advances
|
|
// to the next model — so the guard failed to stop fallback and a combo
|
|
// of N body-rejecting targets tried all N. Mirrors the 499 path above.
|
|
return { ok: false, response: result };
|
|
}
|
|
|
|
// Trigger shared provider circuit breaker for 5xx errors and connection failures.
|
|
// If the next target in the combo is on the same provider, don't mark the provider
|
|
// as failed — different models on the same provider may still succeed.
|
|
// G-02: when fallbackResult.skipProviderBreaker is set (embedded service supervisor
|
|
// outage signalled via X-Omni-Fallback-Hint: connection_cooldown) apply connection
|
|
// cooldown only — do NOT trip the whole-provider breaker.
|
|
const nextTarget = orderedTargets[i + 1];
|
|
const sameProviderNext =
|
|
typeof nextTarget?.provider === "string" && nextTarget.provider === provider;
|
|
if (
|
|
shouldRecordProviderBreakerFailure({
|
|
isStreamReadinessFailure,
|
|
status: result.status,
|
|
sameProviderNext,
|
|
skipProviderBreaker: fallbackResult.skipProviderBreaker,
|
|
requestScopedFailure,
|
|
})
|
|
) {
|
|
recordProviderFailure(provider, log, targetWithConnection.connectionId, profile);
|
|
}
|
|
|
|
// Check if this is a transient error worth retrying on same model.
|
|
// A token-limit 429 is terminal for the client — never retry it.
|
|
const isTransient =
|
|
!isStreamReadinessFailure &&
|
|
!isTokenLimitBreach &&
|
|
[408, 429, 500, 502, 503, 504].includes(result.status);
|
|
if (retry < maxRetries && isTransient && !providerExhausted) {
|
|
if (
|
|
provider &&
|
|
rawModel &&
|
|
isModelLocked(provider, targetWithConnection.connectionId || "", rawModel)
|
|
) {
|
|
log.info("COMBO", `Skipping retry for ${modelStr} — model lockout active`);
|
|
if (i > 0) fallbackCount++;
|
|
return null;
|
|
}
|
|
// Record model lockout immediately on the first transient failure —
|
|
// once the model is cooling down, retrying it would waste an upstream
|
|
// call and extend the cooldown via exponential backoff.
|
|
let lockoutRecorded = false;
|
|
if (provider && rawModel && retry === 0 && !requestScopedFailure) {
|
|
const mlSettings = resolveModelLockoutSettings(settings);
|
|
if (mlSettings.enabled && mlSettings.errorCodes.includes(result.status)) {
|
|
recordModelLockoutFailure(
|
|
provider,
|
|
targetWithConnection.connectionId || "",
|
|
rawModel,
|
|
classifyLockoutReason(result.status),
|
|
result.status,
|
|
mlSettings.baseCooldownMs,
|
|
profile,
|
|
{
|
|
// #1308/#6863: honor a long upstream reset (e.g. "Resets in 160h") over
|
|
// the short base cooldown / exponential backoff when present.
|
|
exactCooldownMs: selectLockoutCooldownMs(lockoutHintMs, mlSettings),
|
|
maxCooldownMs: mlSettings.maxCooldownMs,
|
|
}
|
|
);
|
|
lockoutRecorded = true;
|
|
}
|
|
}
|
|
if (lockoutRecorded) {
|
|
log.info("COMBO", `Skipping retry for ${modelStr} — model lockout active`);
|
|
if (i > 0) fallbackCount++;
|
|
return null;
|
|
}
|
|
continue; // Retry same model (transient error, no lockout recorded)
|
|
}
|
|
|
|
// Done retrying this model
|
|
recordComboRequest(combo.name, modelStr, {
|
|
success: false,
|
|
latencyMs: Date.now() - startTime,
|
|
fallbackCount,
|
|
strategy,
|
|
target: toRecordedTarget(target),
|
|
});
|
|
recordedAttempts++;
|
|
lastError = errorText || String(result.status);
|
|
if (!lastStatus) lastStatus = result.status;
|
|
if (i > 0) fallbackCount++;
|
|
// Wire combo failures into the resilience dashboard (model-level lockout)
|
|
// alongside the provider-level cooldown below — they govern different scopes.
|
|
if (provider && rawModel && !requestScopedFailure) {
|
|
const mlSettings = resolveModelLockoutSettings(settings);
|
|
if (mlSettings.enabled && mlSettings.errorCodes.includes(result.status)) {
|
|
recordModelLockoutFailure(
|
|
provider,
|
|
targetWithConnection.connectionId || "",
|
|
rawModel,
|
|
classifyLockoutReason(result.status),
|
|
result.status,
|
|
mlSettings.baseCooldownMs,
|
|
profile,
|
|
{
|
|
// #1308/#6863: honor a long upstream reset over base/exponential cooldown.
|
|
exactCooldownMs: selectLockoutCooldownMs(lockoutHintMs, mlSettings),
|
|
maxCooldownMs: mlSettings.maxCooldownMs,
|
|
}
|
|
);
|
|
}
|
|
}
|
|
log.warn("COMBO", `Model ${modelStr} failed, trying next`, { status: result.status });
|
|
|
|
// #5976: per-model-quota providers (Gemini, GitHub, etc.) multiplex models
|
|
// behind one connection. A model-level 500 must NOT cool down the entire
|
|
// provider — sibling models may still succeed. Skip cooldown recording for
|
|
// these providers on 500 errors so the next target can try.
|
|
if (
|
|
resilienceSettings.providerCooldown.enabled &&
|
|
provider &&
|
|
provider !== "unknown" &&
|
|
!requestScopedFailure &&
|
|
!(result.status === 500 && hasPerModelQuota(provider, rawModel))
|
|
) {
|
|
recordProviderCooldown(
|
|
provider,
|
|
targetWithConnection.connectionId ?? undefined,
|
|
resilienceSettings
|
|
);
|
|
}
|
|
|
|
const fallbackWaitMs =
|
|
fallbackDelayMs > 0 && cooldownMs > 0 && cooldownMs <= MAX_FALLBACK_WAIT_MS
|
|
? Math.min(cooldownMs, fallbackDelayMs)
|
|
: 0;
|
|
if ([502, 503, 504].includes(result.status) && fallbackWaitMs > 0) {
|
|
log.debug?.("COMBO", `Waiting ${fallbackWaitMs}ms before fallback to next model`);
|
|
await new Promise((resolve) => {
|
|
const timer = setTimeout(resolve, fallbackWaitMs);
|
|
signal?.addEventListener(
|
|
"abort",
|
|
() => {
|
|
clearTimeout(timer);
|
|
resolve(undefined);
|
|
},
|
|
{ once: true }
|
|
);
|
|
});
|
|
if (signal?.aborted) {
|
|
log.info("COMBO", `Client disconnected during fallback wait — aborting`);
|
|
return { ok: false, response: errorResponse(499, "Client disconnected") };
|
|
}
|
|
}
|
|
|
|
return null;
|
|
}
|
|
return null;
|
|
};
|
|
|
|
for (let i = 0; i < orderedTargets.length; i++) {
|
|
if (anySuccess) break;
|
|
|
|
const abortController = new AbortController();
|
|
abortControllers.set(i, abortController);
|
|
const onClientAbort = () => abortController.abort();
|
|
signal?.addEventListener("abort", onClientAbort);
|
|
|
|
const task = (async () => {
|
|
try {
|
|
const res = await executeTarget(i);
|
|
if (res && !anySuccess) {
|
|
if (res.ok) {
|
|
anySuccess = true;
|
|
globalResolve!(res.response!);
|
|
for (const [idx, ac] of abortControllers.entries()) {
|
|
if (idx !== i) ac.abort();
|
|
}
|
|
} else if (res.response) {
|
|
// Fatal error, abort combo
|
|
anySuccess = true;
|
|
globalResolve!(res.response);
|
|
}
|
|
}
|
|
} finally {
|
|
signal?.removeEventListener("abort", onClientAbort);
|
|
}
|
|
})().catch((err) => {
|
|
const logError = log.error ?? log.warn;
|
|
logError("COMBO", `Speculative task error for target ${i}`, err);
|
|
});
|
|
|
|
runningTasks.add(task);
|
|
task.finally(() => runningTasks.delete(task));
|
|
|
|
if (zeroLatencyOptimizationsEnabled && config.hedging && i + 1 < orderedTargets.length) {
|
|
const hedgeDelay = resolveDelayMs(config.hedgeDelayMs, 500);
|
|
let timeoutResolve: () => void;
|
|
const timeoutPromise = new Promise<void>((r) => {
|
|
timeoutResolve = r;
|
|
setTimeout(r, hedgeDelay);
|
|
});
|
|
await Promise.race([task, globalPromise, timeoutPromise]);
|
|
} else {
|
|
await Promise.race([task, globalPromise]);
|
|
}
|
|
}
|
|
|
|
if (!anySuccess && runningTasks.size > 0) {
|
|
await Promise.race([globalPromise, Promise.all([...runningTasks])]);
|
|
}
|
|
|
|
if (anySuccess) {
|
|
return await globalPromise;
|
|
}
|
|
|
|
// All models failed in this set try
|
|
const latencyMs = Date.now() - startTime;
|
|
if (recordedAttempts === 0) {
|
|
recordComboRequest(combo.name, null, {
|
|
success: false,
|
|
latencyMs,
|
|
fallbackCount,
|
|
strategy,
|
|
});
|
|
}
|
|
|
|
// Retry the entire set if more attempts remain
|
|
if (setTry < maxSetRetries) continue;
|
|
|
|
// All set retries exhausted — return the final error
|
|
if (!lastStatus) {
|
|
notifyWebhookEvent("request.failed", {
|
|
combo: combo.name,
|
|
reason: "ALL_ACCOUNTS_INACTIVE",
|
|
latencyMs,
|
|
fallbackCount,
|
|
});
|
|
// Silent-stop fix: bump the failure counter so the session pin clears on the 3rd
|
|
// consecutive all-inactive cascade; buildRecoveryHint emits `switch-combo` with a
|
|
// next-step that points the user at /dashboard/providers.
|
|
recordComboFailure(effectiveSessionId, combo.name);
|
|
return errorResponseWithComboDiagnostics(
|
|
503,
|
|
"Service temporarily unavailable: all upstream accounts are inactive",
|
|
buildComboDiag("all_accounts_inactive"),
|
|
{ code: "ALL_ACCOUNTS_INACTIVE", type: "service_unavailable" }
|
|
);
|
|
}
|
|
|
|
const status = lastStatus;
|
|
const msg = lastError || "All combo models unavailable";
|
|
|
|
if (earliestRetryAfter) {
|
|
// Quota-share cooldown-aware retry: instead of crystallizing the 429,
|
|
// wait out a SHORT transient cooldown and re-run the whole set loop.
|
|
// Guarded by the helper (quota_exhausted/auth/not-found excluded,
|
|
// ceiling, attempts, budget). MAX_GLOBAL_ATTEMPTS still bounds total
|
|
// dispatches.
|
|
if (comboCooldownWaitEnabled && status === 429) {
|
|
const decision = resolveComboCooldownWaitDecision({
|
|
targets: orderedTargets,
|
|
earliestRetryAfter,
|
|
attempt: comboCooldownAttempt,
|
|
budgetLeftMs: comboCooldownBudgetLeftMs,
|
|
settings: resilienceSettings.comboCooldownWait,
|
|
lookupLock: (provider, connectionId) => {
|
|
const rawModel = parseModel(orderedTargets[0]?.modelStr ?? "").model || "";
|
|
return getModelLockoutInfo(provider, connectionId, rawModel);
|
|
},
|
|
computeWaitMs: (retryAfter) => computeClosestRetryAfter(retryAfter).waitMs,
|
|
});
|
|
if (decision.wait) {
|
|
log.info(
|
|
"COMBO",
|
|
`Quota-share cooldown wait: ${msg} — waiting ${Math.ceil(
|
|
decision.waitMs / 1000
|
|
)}s (reason=${decision.reason ?? "?"}) then retrying (attempt ${
|
|
comboCooldownAttempt + 1
|
|
}/${resilienceSettings.comboCooldownWait.maxAttempts})`
|
|
);
|
|
const completed = await waitForCooldownAwareRetry(decision.waitMs, signal);
|
|
if (!completed) {
|
|
log.info("COMBO", "Quota-share cooldown wait aborted by client disconnect");
|
|
return errorResponse(499, "Request aborted");
|
|
}
|
|
comboCooldownAttempt += 1;
|
|
comboCooldownBudgetLeftMs = Math.max(0, comboCooldownBudgetLeftMs - decision.waitMs);
|
|
return dispatchWithCooldownRetry();
|
|
}
|
|
}
|
|
const retryHuman = formatRetryAfter(toRetryAfterDisplayValue(earliestRetryAfter));
|
|
log.warn("COMBO", `All models failed | ${msg} (${retryHuman})`);
|
|
return unavailableResponse(status, msg, earliestRetryAfter, retryHuman);
|
|
}
|
|
|
|
// Silent-stop fix: bump the failure counter (pin clears on 3rd consecutive) and emit
|
|
// `try-auto` recovery action via buildRecoveryHint so the OC plugin can show "→ Try
|
|
// model: auto" instead of an opaque 5xx. We pass the upstream retry-after seconds to
|
|
// the hint so the client can render a precise "wait Ns and retry" message.
|
|
log.warn("COMBO", `All models failed | ${msg}`);
|
|
const { pinClearedNow } = recordComboFailure(effectiveSessionId, combo.name);
|
|
if (pinClearedNow) {
|
|
log.info(
|
|
"COMBO",
|
|
`Auto-cleared session_model_history pin for combo "${combo.name}" after ${COMBO_FAILURE_THRESHOLD} consecutive failures to break the silent-stop loop`
|
|
);
|
|
}
|
|
const retryAfterSeconds = undefined;
|
|
return errorResponseWithComboDiagnostics(
|
|
status,
|
|
msg,
|
|
buildComboDiag(lastError ?? "all_models_failed", retryAfterSeconds)
|
|
);
|
|
}
|
|
|
|
// Final fallback — when the dispatch returned without crystallizing a status (rare).
|
|
// Surface the recovery hint with a generic retry recommendation so the client at least
|
|
// gets a non-opaque message instead of "Combo routing completed without an upstream response".
|
|
recordComboFailure(effectiveSessionId, combo.name);
|
|
return errorResponseWithComboDiagnostics(
|
|
503,
|
|
"Combo routing completed without an upstream response",
|
|
buildNoUpstreamResponseDiagnostics(orderedTargets.length)
|
|
);
|
|
};
|
|
|
|
// FASE 2.1: acquire the per-connection concurrency slot for the selected
|
|
// quota-share target once, around the whole dispatch (including any
|
|
// cooldown-aware re-dispatch), so concurrent requests to one subscription
|
|
// account are serialized through the connection's max_concurrent ceiling. The
|
|
// cap is read fresh from the selected connection; a null cap (no limit) or a
|
|
// saturated queue is a no-op (fail-open). Released in the finally below.
|
|
let quotaShareConcurrencyRelease: (() => void) | null = null;
|
|
const qsConnectionId = orderedTargets[0]?.connectionId;
|
|
if (quotaShareConcurrencyEnabled && qsConnectionId) {
|
|
const qsCap = await lookupPositiveCap(qsConnectionId);
|
|
quotaShareConcurrencyRelease = await acquireQuotaShareConcurrencySlot(
|
|
orderedTargets[0],
|
|
qsCap,
|
|
{
|
|
queueTimeoutMs: config.queueTimeoutMs ?? 30000,
|
|
maxQueueSize: resolveComboQueueDepth(config),
|
|
},
|
|
log
|
|
);
|
|
}
|
|
|
|
try {
|
|
return await dispatchWithCooldownRetry();
|
|
} finally {
|
|
quotaShareConcurrencyRelease?.();
|
|
// G2: Clean up candidate registry to prevent unbounded memory growth.
|
|
_unregisterExecutionCandidates(_registeredExecutionKeys);
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Handle round-robin combo: each request goes to the next model in circular order.
|
|
* Uses semaphore-based concurrency control with queue + rate-limit awareness.
|
|
*
|
|
* Flow:
|
|
* 1. Pick target model via atomic counter (counter % models.length)
|
|
* 2. Acquire semaphore slot (may queue if at max concurrency)
|
|
* 3. Send request to target model
|
|
* 4. On 429 → mark model rate-limited, try next model in rotation
|
|
* 5. On semaphore timeout → fallback to next available model
|
|
*/
|
|
async function handleRoundRobinCombo({
|
|
body,
|
|
combo,
|
|
handleSingleModel,
|
|
isModelAvailable,
|
|
log,
|
|
settings,
|
|
allCombos,
|
|
signal,
|
|
}: HandleRoundRobinOptions): Promise<Response> {
|
|
const config = settings
|
|
? resolveComboConfig(combo, settings)
|
|
: { ...getDefaultComboConfig(), ...(combo.config || {}) };
|
|
const concurrency = config.concurrencyPerModel ?? 3;
|
|
// Honor each target connection's own maxConcurrent ceiling (cached per dispatch)
|
|
// so a low-concurrency subscription account is not flooded; falls back to the
|
|
// combo-level concurrency when the connection has no positive cap.
|
|
const resolveTargetConcurrency = makeConnectionConcurrencyResolver(concurrency);
|
|
const queueTimeout = config.queueTimeoutMs ?? 30000;
|
|
// #3872: pre-cascade queue depth — lower values fail over to the next combo member
|
|
// sooner under concurrency saturation (0 = never queue). Default 20 (backward-compat).
|
|
const queueDepth = resolveComboQueueDepth(config);
|
|
const maxRetries = config.maxRetries ?? 1;
|
|
const retryDelayMs = resolveDelayMs(config.retryDelayMs, 2000);
|
|
const fallbackDelayMs = resolveDelayMs(config.fallbackDelayMs, 0);
|
|
const reasoningTokenBufferEnabled = config.reasoningTokenBufferEnabled !== false;
|
|
|
|
const resilienceSettings: ResilienceSettings = settings
|
|
? resolveResilienceSettings(settings)
|
|
: resolveResilienceSettings(null);
|
|
|
|
// #2562: Expand provider-wildcard steps before resolving targets.
|
|
const rrExpandedCombo = await expandProviderWildcardsInCombo(combo);
|
|
const rrExpandedAllCombos = allCombos
|
|
? Array.isArray(allCombos)
|
|
? await expandProviderWildcardsInCollection(allCombos as ComboLike[])
|
|
: {
|
|
...allCombos,
|
|
combos: await expandProviderWildcardsInCollection(
|
|
((allCombos as { combos?: ComboLike[] }).combos || []) as ComboLike[]
|
|
),
|
|
}
|
|
: allCombos;
|
|
|
|
const orderedTargets = resolveComboTargets(
|
|
rrExpandedCombo,
|
|
rrExpandedAllCombos,
|
|
clampComboDepth(config.maxComboDepth)
|
|
);
|
|
const tagFilteredTargets = await applyRequestTagRouting(orderedTargets, body, log);
|
|
const evalRankedTargets = orderTargetsByEvalScores(tagFilteredTargets, config.evalRouting, log);
|
|
const knownContextOverflow = getKnownContextOverflow(evalRankedTargets, body);
|
|
if (knownContextOverflow) {
|
|
return errorResponseWithComboDiagnostics(
|
|
400,
|
|
`Request requires approximately ${knownContextOverflow.requiredContextTokens} tokens, but the largest known context limit in this combo is ${knownContextOverflow.maxKnownContextTokens} tokens. Reduce or compact the request context.`,
|
|
{
|
|
poolSize: evalRankedTargets.length,
|
|
attempted: 0,
|
|
excluded: evalRankedTargets.map((target) => ({
|
|
provider: target.provider,
|
|
model: target.modelStr,
|
|
reason: "context_window",
|
|
})),
|
|
attemptOrder: [],
|
|
terminalReason: "context_length_exceeded",
|
|
},
|
|
{ code: "context_length_exceeded", type: "invalid_request_error" }
|
|
);
|
|
}
|
|
const filteredTargets = filterTargetsByRequestCompatibility(
|
|
evalRankedTargets,
|
|
body,
|
|
log,
|
|
"Context-aware round-robin fallback"
|
|
);
|
|
// #6238: keep the targets the compat pre-filter rejected so they can serve as a
|
|
// last-resort fallback tier. The pre-filter drops request-incompatible targets
|
|
// BEFORE availability is known; if every compat-kept target then turns out to be
|
|
// runtime-unavailable, we must reconsider these before returning 503, instead of
|
|
// permanently dropping a compat-rejected-but-healthy provider.
|
|
const compatKeptSet = new Set(filteredTargets);
|
|
const compatRejectedTargets = evalRankedTargets.filter((target) => !compatKeptSet.has(target));
|
|
const modelCount = filteredTargets.length;
|
|
if (modelCount === 0) {
|
|
return comboModelNotFoundResponse("Round-robin combo has no executable targets");
|
|
}
|
|
|
|
scheduleShadowRouting(
|
|
combo,
|
|
config,
|
|
body,
|
|
resolveShadowTargets(combo, config, allCombos),
|
|
handleSingleModel,
|
|
isModelAvailable,
|
|
"round-robin",
|
|
log
|
|
);
|
|
|
|
// Sticky batch size at the combo level. A per-combo `stickyRoundRobinLimit` (in
|
|
// combo.config, resolved through the cascade) overrides the global setting so one
|
|
// combo can batch differently from the default. When the per-combo value is unset,
|
|
// fall back to the global `stickyRoundRobinLimit` so the existing knob still controls
|
|
// sticky batching for both account fallback and combo targets. Values <= 1 preserve
|
|
// the historical one-request-per-target rotation.
|
|
const perComboStickyLimit = (config as Record<string, unknown>).stickyRoundRobinLimit;
|
|
const stickyLimit = resolveComboStickyRoundRobinLimit(
|
|
perComboStickyLimit,
|
|
settings as Record<string, unknown> | null
|
|
);
|
|
const stickyRoundRobinEnabled = stickyLimit > 1;
|
|
// Exhaustion-aware sticky: if the currently sticky target is no longer
|
|
// available (circuit breaker OPEN, provider cooldown, model lockout, or
|
|
// isModelAvailable returns false), clear the sticky record so the rotation
|
|
// starts at the counter position instead of probing a dead target.
|
|
if (stickyRoundRobinEnabled) {
|
|
const sticky = rrStickyTargets.get(combo.name);
|
|
if (sticky) {
|
|
const stickyTarget = filteredTargets.find(
|
|
(target) => target.executionKey === sticky.executionKey
|
|
);
|
|
if (stickyTarget) {
|
|
const rawModel = parseModel(stickyTarget.modelStr).model || stickyTarget.modelStr;
|
|
const stickyAvailable =
|
|
(!stickyTarget.provider ||
|
|
getCircuitBreaker(stickyTarget.provider).getStatus().state !== "OPEN") &&
|
|
!(
|
|
resilienceSettings.providerCooldown.enabled &&
|
|
Boolean(stickyTarget.provider && stickyTarget.provider !== "unknown") &&
|
|
isProviderInCooldown(
|
|
stickyTarget.provider,
|
|
stickyTarget.connectionId ?? undefined,
|
|
resilienceSettings
|
|
)
|
|
) &&
|
|
!(
|
|
stickyTarget.provider &&
|
|
rawModel &&
|
|
isModelLocked(stickyTarget.provider, stickyTarget.connectionId || "", rawModel)
|
|
) &&
|
|
(isModelAvailable ? await isModelAvailable(stickyTarget.modelStr, stickyTarget) : true);
|
|
if (!stickyAvailable) {
|
|
log.info(
|
|
"COMBO-RR",
|
|
`Clearing stale sticky target ${stickyTarget.modelStr} — unavailable`
|
|
);
|
|
rrStickyTargets.delete(combo.name);
|
|
}
|
|
}
|
|
}
|
|
}
|
|
if (
|
|
!rrCounters.has(combo.name) &&
|
|
!rrStickyTargets.has(combo.name) &&
|
|
rrCounters.size >= MAX_RR_COUNTERS
|
|
) {
|
|
const oldest = rrCounters.keys().next().value;
|
|
if (oldest !== undefined) {
|
|
rrCounters.delete(oldest);
|
|
rrStickyTargets.delete(oldest);
|
|
}
|
|
}
|
|
// Ensure rrCounters has an entry for this combo so the eviction logic above
|
|
// applies to both maps even when sticky round-robin is enabled (in which
|
|
// case rrCounters isn't incremented per request).
|
|
if (!rrCounters.has(combo.name)) {
|
|
rrCounters.set(combo.name, 0);
|
|
}
|
|
const { startIndex, counter } = getStickyRoundRobinStartIndex(
|
|
combo.name,
|
|
filteredTargets,
|
|
stickyLimit
|
|
);
|
|
if (!stickyRoundRobinEnabled) {
|
|
rrCounters.set(combo.name, counter + 1);
|
|
}
|
|
|
|
// #3825: per-conversation session stickiness for round-robin. weighted/priority honor a
|
|
// sticky connection via applySessionStickiness, but this RR handler returns before that
|
|
// call — so sessionless RR combos rotated every turn, busting the upstream prompt-cache.
|
|
// Reuse the SAME mechanism: start the rotation at the conversation's sticky connection
|
|
// (the loop still falls through to the other targets on failure → failover preserved).
|
|
// #6168: honor the session-stickiness opt-out here too, otherwise round-robin would
|
|
// still pin the conversation even when the flag is set. Per-combo `config` overrides
|
|
// the global `settings.disableSessionStickiness` fallback (default false).
|
|
const disableSessionStickiness = resolveDisableSessionStickiness(
|
|
config as Record<string, unknown> | null | undefined,
|
|
settings as Record<string, unknown> | null | undefined
|
|
);
|
|
const _rrSessionSticky = disableSessionStickiness
|
|
? ({ targets: filteredTargets, messageHash: null, stuck: false } as const)
|
|
: await applySessionStickiness(
|
|
filteredTargets,
|
|
// #7270: normalize both wire shapes (.messages / Responses-API .input) so RR
|
|
// stickiness engages on the /v1/responses surface, not just Chat Completions.
|
|
normalizeStickinessMessages(body as { messages?: unknown; input?: unknown })
|
|
);
|
|
let rrStartIndex = startIndex;
|
|
if (_rrSessionSticky.stuck) {
|
|
const stickyIdx = filteredTargets.findIndex(
|
|
(t) => t.connectionId === _rrSessionSticky.targets[0]?.connectionId
|
|
);
|
|
if (stickyIdx >= 0) rrStartIndex = stickyIdx;
|
|
}
|
|
|
|
const clientRequestedStream = body?.stream === true;
|
|
const startTime = Date.now();
|
|
let lastError: string | null = null;
|
|
let lastStatus: number | null = null;
|
|
let earliestRetryAfter: ComboRetryAfter | null = null;
|
|
let globalAttempts = 0;
|
|
let fallbackCount = 0;
|
|
let recordedAttempts = 0;
|
|
|
|
// #1731: Per-request in-memory set of providers whose quota is fully exhausted.
|
|
// When a target returns a quota-exhausted 429, remaining targets from the same
|
|
// provider are skipped to avoid the cascade through N same-provider targets.
|
|
const exhaustedProviders = new Set<string>();
|
|
const exhaustedConnections = new Set<string>();
|
|
const transientRateLimitedProviders = new Set<string>();
|
|
|
|
// Try each model starting from the round-robin target
|
|
for (let offset = 0; offset < modelCount; offset++) {
|
|
const modelIndex = (rrStartIndex + offset) % modelCount;
|
|
const target = filteredTargets[modelIndex];
|
|
const modelStr = target.modelStr;
|
|
const provider = target.provider;
|
|
const profile = await getRuntimeProviderProfile(provider);
|
|
const semaphoreKey = `combo:${combo.name}:${target.executionKey}`;
|
|
const allowRateLimitedConnection =
|
|
Boolean(provider && provider !== "unknown") && transientRateLimitedProviders.has(provider);
|
|
const targetForAttempt = allowRateLimitedConnection
|
|
? { ...target, allowRateLimitedConnection: true }
|
|
: target;
|
|
|
|
// Pre-check availability
|
|
if (isModelAvailable) {
|
|
const available = await isModelAvailable(modelStr, targetForAttempt);
|
|
if (!available) {
|
|
log.debug?.(
|
|
"COMBO-RR",
|
|
`Skipping ${modelStr} — no credentials available or model excluded`
|
|
);
|
|
if (offset > 0) fallbackCount++;
|
|
continue;
|
|
}
|
|
}
|
|
|
|
if (
|
|
resilienceSettings.providerCooldown.enabled &&
|
|
Boolean(provider && provider !== "unknown") &&
|
|
isProviderInCooldown(provider, target.connectionId as string | undefined, resilienceSettings)
|
|
) {
|
|
log.info("COMBO-RR", `Skipping ${modelStr} — provider ${provider} in global cooldown`);
|
|
if (offset > 0) fallbackCount++;
|
|
continue;
|
|
}
|
|
|
|
// #1731 / #1731v2: skip targets already known-exhausted this request (shared predicate).
|
|
const exhaustedSkip = getExhaustedTargetSkipReason(
|
|
target,
|
|
exhaustedProviders,
|
|
exhaustedConnections
|
|
);
|
|
if (exhaustedSkip) {
|
|
log.info("COMBO-RR", exhaustedSkip);
|
|
if (offset > 0) fallbackCount++;
|
|
continue;
|
|
}
|
|
|
|
// Acquire semaphore slot (may wait in queue). Honor the connection's own
|
|
// maxConcurrent cap when set; else fall back to the combo-level concurrency.
|
|
const targetConcurrency = await resolveTargetConcurrency(target.connectionId);
|
|
let release: () => void;
|
|
try {
|
|
release = await semaphore.acquire(semaphoreKey, {
|
|
maxConcurrency: targetConcurrency,
|
|
timeoutMs: queueTimeout,
|
|
maxQueueSize: queueDepth,
|
|
});
|
|
} catch (err) {
|
|
const errCode = isRecord(err) && typeof err.code === "string" ? err.code : null;
|
|
if (errCode === "SEMAPHORE_TIMEOUT" || errCode === "SEMAPHORE_QUEUE_FULL") {
|
|
log.warn(
|
|
"COMBO-RR",
|
|
`Semaphore ${errCode === "SEMAPHORE_QUEUE_FULL" ? "queue full" : "timeout"} for ${modelStr}, trying next model`
|
|
);
|
|
if (offset > 0) fallbackCount++;
|
|
continue;
|
|
}
|
|
throw err;
|
|
}
|
|
|
|
// Retry loop within this model
|
|
try {
|
|
for (let retry = 0; retry <= maxRetries; retry++) {
|
|
globalAttempts++;
|
|
if (globalAttempts > MAX_GLOBAL_ATTEMPTS) {
|
|
log.warn(
|
|
"COMBO-RR",
|
|
`Maximum combo attempts (${MAX_GLOBAL_ATTEMPTS}) exceeded. Terminating loop to prevent runaway requests.`
|
|
);
|
|
return errorResponse(503, "Maximum combo retry limit reached");
|
|
}
|
|
if (retry > 0) {
|
|
log.info(
|
|
"COMBO-RR",
|
|
`Retrying ${modelStr} in ${retryDelayMs}ms (attempt ${retry + 1}/${maxRetries + 1})`
|
|
);
|
|
await new Promise((r) => setTimeout(r, retryDelayMs));
|
|
}
|
|
|
|
log.info(
|
|
"COMBO-RR",
|
|
`[RR #${counter}] → ${modelStr}${offset > 0 ? ` (fallback +${offset})` : ""}${retry > 0 ? ` (retry ${retry})` : ""}`
|
|
);
|
|
|
|
// Issue #3587: Reasoning models can spend the whole output budget on
|
|
// reasoning. Apply any safe buffer to a per-attempt copy so round-robin
|
|
// retries never compound across models.
|
|
let attemptBody = body;
|
|
{
|
|
const bodyRecord = body as Record<string, unknown>;
|
|
const currentMaxTokens = toPositiveInteger(bodyRecord.max_tokens);
|
|
const bufferedMaxTokens = resolveReasoningBufferedMaxTokens(
|
|
modelStr,
|
|
bodyRecord.max_tokens,
|
|
{ enabled: reasoningTokenBufferEnabled }
|
|
);
|
|
if (
|
|
currentMaxTokens !== null &&
|
|
bufferedMaxTokens !== null &&
|
|
bufferedMaxTokens !== currentMaxTokens
|
|
) {
|
|
attemptBody = {
|
|
...bodyRecord,
|
|
max_tokens: bufferedMaxTokens,
|
|
} as typeof body;
|
|
log.info(
|
|
"COMBO-RR",
|
|
`Reasoning model ${modelStr}: adjusted max_tokens ${currentMaxTokens} -> ${bufferedMaxTokens}`
|
|
);
|
|
}
|
|
}
|
|
|
|
const result = await handleSingleModel(attemptBody, modelStr, {
|
|
...targetForAttempt,
|
|
effectiveComboStrategy: "round-robin",
|
|
failoverBeforeRetry: config.failoverBeforeRetry,
|
|
});
|
|
|
|
// Success — validate response quality before returning
|
|
if (result.ok) {
|
|
let rrClone: Response;
|
|
try {
|
|
rrClone = result.clone();
|
|
} catch {
|
|
rrClone = result;
|
|
}
|
|
const quality = await validateResponseQuality(
|
|
rrClone,
|
|
clientRequestedStream,
|
|
log,
|
|
config.responseValidation
|
|
);
|
|
releaseQualityClone(rrClone, result, quality);
|
|
if (!quality.valid) {
|
|
releaseRejectedQualityResponse(rrClone, result);
|
|
log.warn(
|
|
"COMBO-RR",
|
|
`${modelStr} returned 200 but failed quality check: ${quality.reason}`
|
|
);
|
|
// #6692: same rationale as handleComboChat's quality-fail branch —
|
|
// a quality-rejected 200 never marks the connection row unhealthy,
|
|
// so release the sticky pin here rather than on the next turn.
|
|
{
|
|
const rrSelectedConnectionId =
|
|
result.headers?.get("X-OmniRoute-Selected-Connection-Id") ||
|
|
result.headers?.get("x-omniroute-selected-connection-id") ||
|
|
undefined;
|
|
releaseStickyPinOnFailure(
|
|
_rrSessionSticky.messageHash,
|
|
rrSelectedConnectionId || target.connectionId
|
|
);
|
|
}
|
|
recordComboRequest(combo.name, modelStr, {
|
|
success: false,
|
|
latencyMs: Date.now() - startTime,
|
|
fallbackCount,
|
|
strategy: "round-robin",
|
|
target: toRecordedTarget(target),
|
|
});
|
|
recordedAttempts++;
|
|
// Fix #1707: Set terminal state so the fallback doesn't emit
|
|
// misleading ALL_ACCOUNTS_INACTIVE when the real issue is quality.
|
|
lastError = `Upstream response failed quality validation: ${quality.reason}`;
|
|
if (!lastStatus) lastStatus = 502;
|
|
if (offset > 0) fallbackCount++;
|
|
break; // move to next model
|
|
}
|
|
const latencyMs = Date.now() - startTime;
|
|
log.info(
|
|
"COMBO-RR",
|
|
`${modelStr} succeeded (${latencyMs}ms, ${fallbackCount} fallbacks)`
|
|
);
|
|
recordComboRequest(combo.name, modelStr, {
|
|
success: true,
|
|
latencyMs,
|
|
fallbackCount,
|
|
strategy: "round-robin",
|
|
target: toRecordedTarget(target),
|
|
});
|
|
recordedAttempts++;
|
|
|
|
const selectedConnectionId =
|
|
result.headers?.get("X-OmniRoute-Selected-Connection-Id") ||
|
|
result.headers?.get("x-omniroute-selected-connection-id") ||
|
|
undefined;
|
|
const effectiveConnectionId = selectedConnectionId || target.connectionId || "";
|
|
|
|
const rawModel = parseModel(modelStr).model || modelStr;
|
|
if (provider && rawModel) {
|
|
const dcResult = decayModelFailureCount(provider, effectiveConnectionId, rawModel);
|
|
if (dcResult.cleared) {
|
|
log.info("COMBO-RR", `Model ${modelStr} fully recovered — lockout cleared`);
|
|
} else if (dcResult.newFailureCount > 0) {
|
|
log.debug?.(
|
|
"COMBO-RR",
|
|
`Model ${modelStr} decayed to failureCount=${dcResult.newFailureCount}`
|
|
);
|
|
}
|
|
}
|
|
|
|
if (provider && provider !== "unknown") {
|
|
recordProviderSuccess(provider, effectiveConnectionId || undefined);
|
|
}
|
|
|
|
if (stickyRoundRobinEnabled) {
|
|
recordStickyRoundRobinSuccess(combo.name, target, stickyLimit, filteredTargets);
|
|
} else {
|
|
// #948: true round-robin (stickyLimit <= 1). The counter was advanced
|
|
// eagerly (+1 from the scheduled start index) before this loop ran, so
|
|
// when the scheduled model failed and a *different* model served via
|
|
// fallback, the next request reused the fallback-served model. Advance
|
|
// the pointer past the model that ACTUALLY served (modelIndex) instead,
|
|
// mirroring recordStickyRoundRobinSuccess's served-index logic. Read
|
|
// side applies `% modelCount`, so storing modelIndex + 1 is correct.
|
|
rrCounters.set(combo.name, modelIndex + 1);
|
|
}
|
|
|
|
// #3825: (re)record the sticky binding so the next turn re-pins (prompt-cache).
|
|
if (_rrSessionSticky.messageHash) {
|
|
const stickyConn = effectiveConnectionId || target.connectionId;
|
|
if (stickyConn) recordStickyBinding(_rrSessionSticky.messageHash, stickyConn);
|
|
}
|
|
|
|
if (provider) {
|
|
const connId = effectiveConnectionId || undefined;
|
|
void (async () => {
|
|
try {
|
|
const { setLKGP } = await import("../../src/lib/localDb");
|
|
await Promise.all([
|
|
setLKGP(combo.name, target.executionKey, provider, connId),
|
|
setLKGP(combo.name, combo.id || combo.name, provider, connId),
|
|
]);
|
|
} catch (err) {
|
|
log.warn(
|
|
"COMBO-RR",
|
|
"Failed to record Last Known Good Provider. This is non-fatal.",
|
|
{
|
|
err,
|
|
}
|
|
);
|
|
}
|
|
})();
|
|
}
|
|
// Clone is consumed by quality check; original stays unlocked.
|
|
return result;
|
|
}
|
|
|
|
// Extract error info
|
|
let errorText = result.statusText || "";
|
|
let retryAfter: ComboRetryAfter | null = null;
|
|
let errorBody: ComboErrorBody = null;
|
|
try {
|
|
const cloned = result.clone();
|
|
try {
|
|
const text = await cloned.text();
|
|
if (text) {
|
|
errorText = text.substring(0, 500);
|
|
errorBody = JSON.parse(text);
|
|
const parsedError = errorBody?.error;
|
|
errorText =
|
|
(typeof parsedError === "object" && parsedError?.message) ||
|
|
(typeof parsedError === "string" ? parsedError : null) ||
|
|
errorBody?.message ||
|
|
errorText;
|
|
retryAfter = errorBody?.retryAfter || null;
|
|
}
|
|
} catch {
|
|
/* Clone parse failed */
|
|
}
|
|
} catch {
|
|
/* Clone failed */
|
|
}
|
|
|
|
if (result.status === 499) {
|
|
log.info(
|
|
"COMBO-RR",
|
|
`Client disconnected (499) during ${modelStr} — stopping combo loop`
|
|
);
|
|
recordComboRequest(combo.name, modelStr, {
|
|
success: false,
|
|
latencyMs: Date.now() - startTime,
|
|
fallbackCount,
|
|
strategy: "round-robin",
|
|
target: toRecordedTarget(target),
|
|
});
|
|
recordedAttempts++;
|
|
return result;
|
|
}
|
|
|
|
if (
|
|
retryAfter &&
|
|
(!earliestRetryAfter || new Date(retryAfter) < new Date(earliestRetryAfter))
|
|
) {
|
|
earliestRetryAfter = retryAfter;
|
|
}
|
|
|
|
if (typeof errorText !== "string") {
|
|
try {
|
|
errorText = JSON.stringify(errorText);
|
|
} catch {
|
|
errorText = String(errorText);
|
|
}
|
|
}
|
|
|
|
const isStreamReadinessFailure =
|
|
(result.status === 502 || result.status === 504) &&
|
|
isStreamReadinessFailureErrorBody(errorBody);
|
|
|
|
// FIX 5: a local per-API-key token-limit 429 must not cool shared accounts.
|
|
const isTokenLimitBreach = result.status === 429 && isTokenLimitBreachErrorBody(errorBody);
|
|
|
|
// Round-robin uses the same target-level fallback rule as other combo
|
|
// strategies: non-ok target responses fall through to the next target.
|
|
// Classification stays here only to support cooldown/semaphore pacing,
|
|
// not to decide whether fallback is allowed.
|
|
const rawError = errorBody?.error;
|
|
const structuredError =
|
|
rawError && typeof rawError === "object"
|
|
? {
|
|
// Upstream JSON may carry a numeric `code`/`type` (e.g. {"code":40001}).
|
|
// Coerce to string if present instead of discarding, so downstream string
|
|
// ops (.toLowerCase, .startsWith) can run safely without type crashes.
|
|
code:
|
|
(rawError as Record<string, unknown>).code !== undefined &&
|
|
(rawError as Record<string, unknown>).code !== null
|
|
? String((rawError as Record<string, unknown>).code)
|
|
: undefined,
|
|
type:
|
|
(rawError as Record<string, unknown>).type !== undefined &&
|
|
(rawError as Record<string, unknown>).type !== null
|
|
? String((rawError as Record<string, unknown>).type)
|
|
: undefined,
|
|
}
|
|
: undefined;
|
|
const requestScopedFailure = isRequestScopedUpstreamFailure(structuredError);
|
|
const fallbackResult = checkFallbackError(
|
|
result.status,
|
|
errorText,
|
|
0,
|
|
null,
|
|
provider,
|
|
result.headers,
|
|
profile,
|
|
structuredError
|
|
);
|
|
const { cooldownMs } = fallbackResult;
|
|
const selectedConnectionId =
|
|
result.headers?.get("X-OmniRoute-Selected-Connection-Id") ||
|
|
result.headers?.get("x-omniroute-selected-connection-id") ||
|
|
undefined;
|
|
const targetWithConnection = selectedConnectionId
|
|
? { ...target, connectionId: selectedConnectionId }
|
|
: target;
|
|
|
|
const isAllAccountsRateLimited = isAllAccountsRateLimitedResponse(
|
|
result.status,
|
|
result.headers?.get("content-type") ?? null,
|
|
errorText
|
|
);
|
|
|
|
// #1731: If the entire provider quota is exhausted, mark it so subsequent
|
|
// same-provider targets are skipped immediately. API-key 429s still use
|
|
// the short resilience cooldown, but explicit quota text should stop the
|
|
// combo from trying another target for the same provider in this request.
|
|
// #1731 / #1731v2: classify the upstream error and update the exhaustion sets
|
|
// (shared with handleComboChat). Returns whether the provider is fully exhausted.
|
|
const providerExhausted = applyComboTargetExhaustion(targetWithConnection, {
|
|
result,
|
|
fallbackResult,
|
|
errorText,
|
|
rawModel: parseModel(modelStr).model || modelStr,
|
|
isTokenLimitBreach,
|
|
allAccountsRateLimited: isAllAccountsRateLimited,
|
|
sets: { exhaustedProviders, exhaustedConnections, transientRateLimitedProviders },
|
|
log,
|
|
tag: "COMBO-RR",
|
|
exhaustedLogLevel: "debug",
|
|
structuredError,
|
|
});
|
|
// #6692: mirrors handleComboChat's exhaustion-point release above.
|
|
releaseStickyPinOnFailure(_rrSessionSticky.messageHash, targetWithConnection.connectionId);
|
|
|
|
// Transient errors → mark in semaphore so round-robin stops stampeding this target.
|
|
if (
|
|
!isStreamReadinessFailure &&
|
|
!isTokenLimitBreach &&
|
|
!requestScopedFailure &&
|
|
TRANSIENT_FOR_SEMAPHORE.includes(result.status) &&
|
|
cooldownMs > 0
|
|
) {
|
|
semaphore.markRateLimited(semaphoreKey, cooldownMs);
|
|
log.warn("COMBO-RR", `${modelStr} error ${result.status}, cooldown ${cooldownMs}ms`);
|
|
}
|
|
|
|
if (isAllAccountsRateLimited) {
|
|
log.info(
|
|
"COMBO-RR",
|
|
`All accounts rate-limited for ${modelStr}, falling back to next model`
|
|
);
|
|
}
|
|
|
|
// Transient error → retry same model.
|
|
// A token-limit 429 is terminal for the client — never retry it.
|
|
const isTransient =
|
|
!isStreamReadinessFailure &&
|
|
!isTokenLimitBreach &&
|
|
[408, 429, 500, 502, 503, 504].includes(result.status);
|
|
if (retry < maxRetries && isTransient && !providerExhausted) {
|
|
continue;
|
|
}
|
|
|
|
// Done with this model
|
|
recordComboRequest(combo.name, modelStr, {
|
|
success: false,
|
|
latencyMs: Date.now() - startTime,
|
|
fallbackCount,
|
|
strategy: "round-robin",
|
|
target: toRecordedTarget(target),
|
|
});
|
|
recordedAttempts++;
|
|
lastError = errorText || String(result.status);
|
|
if (!lastStatus) lastStatus = result.status;
|
|
if (offset > 0) fallbackCount++;
|
|
log.warn("COMBO-RR", `${modelStr} failed, trying next model`, { status: result.status });
|
|
|
|
if (
|
|
resilienceSettings.providerCooldown.enabled &&
|
|
provider &&
|
|
provider !== "unknown" &&
|
|
!requestScopedFailure &&
|
|
!(
|
|
result.status === 500 &&
|
|
hasPerModelQuota(provider, parseModel(modelStr).model || modelStr)
|
|
)
|
|
) {
|
|
recordProviderCooldown(
|
|
provider,
|
|
targetWithConnection.connectionId ?? undefined,
|
|
resilienceSettings
|
|
);
|
|
}
|
|
|
|
const fallbackWaitMs =
|
|
fallbackDelayMs > 0 && cooldownMs > 0 && cooldownMs <= MAX_FALLBACK_WAIT_MS
|
|
? Math.min(cooldownMs, fallbackDelayMs)
|
|
: 0;
|
|
if ([502, 503, 504].includes(result.status) && fallbackWaitMs > 0) {
|
|
log.debug?.("COMBO-RR", `Waiting ${fallbackWaitMs}ms before fallback to next model`);
|
|
await new Promise((resolve) => {
|
|
const timer = setTimeout(resolve, fallbackWaitMs);
|
|
signal?.addEventListener(
|
|
"abort",
|
|
() => {
|
|
clearTimeout(timer);
|
|
resolve(undefined);
|
|
},
|
|
{ once: true }
|
|
);
|
|
});
|
|
if (signal?.aborted) {
|
|
log.info("COMBO-RR", `Client disconnected during fallback wait — aborting`);
|
|
return errorResponse(499, "Client disconnected");
|
|
}
|
|
}
|
|
|
|
break;
|
|
}
|
|
} finally {
|
|
// ALWAYS release semaphore slot
|
|
release();
|
|
}
|
|
}
|
|
|
|
// All models exhausted
|
|
const latencyMs = Date.now() - startTime;
|
|
|
|
// #6238: every compat-kept target was skipped as unavailable and NONE was ever
|
|
// attempted (recordedAttempts === 0). Before crystallizing 503, probe the targets
|
|
// the compat pre-filter rejected — a compat-rejected-but-healthy provider is a
|
|
// valid last-resort fallback tier, not a permanently dropped target.
|
|
if (recordedAttempts === 0 && compatRejectedTargets.length > 0) {
|
|
const compatFallbackResult = await attemptCompatRejectedFallback(compatRejectedTargets, body, {
|
|
handleSingleModel,
|
|
isModelAvailable,
|
|
isProviderInCooldown: (target) =>
|
|
resilienceSettings.providerCooldown.enabled &&
|
|
Boolean(target.provider && target.provider !== "unknown") &&
|
|
isProviderInCooldown(
|
|
target.provider as string,
|
|
target.connectionId as string | undefined,
|
|
resilienceSettings
|
|
),
|
|
log,
|
|
strategy: "round-robin",
|
|
});
|
|
if (compatFallbackResult) {
|
|
recordComboRequest(combo.name, null, {
|
|
success: true,
|
|
latencyMs: Date.now() - startTime,
|
|
fallbackCount,
|
|
strategy: "round-robin",
|
|
});
|
|
return compatFallbackResult;
|
|
}
|
|
}
|
|
|
|
if (recordedAttempts === 0) {
|
|
recordComboRequest(combo.name, null, {
|
|
success: false,
|
|
latencyMs,
|
|
fallbackCount,
|
|
strategy: "round-robin",
|
|
});
|
|
}
|
|
|
|
if (!lastStatus) {
|
|
return new Response(
|
|
JSON.stringify({
|
|
error: {
|
|
message: "Service temporarily unavailable: all upstream accounts are inactive",
|
|
type: "service_unavailable",
|
|
code: "ALL_ACCOUNTS_INACTIVE",
|
|
},
|
|
}),
|
|
{ status: 503, headers: { "Content-Type": "application/json" } }
|
|
);
|
|
}
|
|
|
|
const status = lastStatus;
|
|
const msg = lastError || "All round-robin combo models unavailable";
|
|
|
|
if (earliestRetryAfter) {
|
|
const retryHuman = formatRetryAfter(toRetryAfterDisplayValue(earliestRetryAfter));
|
|
log.warn("COMBO-RR", `All models failed | ${msg} (${retryHuman})`);
|
|
return unavailableResponse(status, msg, earliestRetryAfter, retryHuman);
|
|
}
|
|
|
|
log.warn("COMBO-RR", `All models failed | ${msg}`);
|
|
return new Response(JSON.stringify({ error: { message: msg } }), {
|
|
status,
|
|
headers: { "Content-Type": "application/json" },
|
|
});
|
|
}
|