Files
OmniRoute/open-sse/services/combo.ts
Hernan Javier Ardila Sanchez f1a77fefc5 fix(combo): auto-clear stale session pins and emit recovery hints on combo exhaustion (#7625)
* chore(ci): add .mergify.yml to main — Mergify only reads config from the default branch (#7168)

* fix(ci): add the auto-enqueue pull_request_rule to the Mergify config (queue_conditions alone are eligibility-only) (#7179)

* fix(ci): migrate Mergify auto-enqueue to merge_protections_settings.auto_merge_conditions (rules-based path is EOL 2026-07-16) (#7216)

* fix(ci): drop Mergify batch settings (batching is a paid-tier feature; free plan queue is serial) (#7220)

* fix(ci): merge queue tolerates the advisory dast-smoke failure (its GH-hosted build hang dequeued every attempt) (#7225)

* test(ci): make the #6634 selfref guard hermetic — main's copy hard-fails every PR (#7341)

main's copy of this test still does git I/O inside a unit test:

    const baseSrc = git(['show', 'origin/main:' + FILE]);

Runners check out a shallow single ref, so origin/main does not resolve and the
test dies with 'fatal: invalid object name origin/main'. Every PR into main
fails Unit Tests (7/8) on it — today that is #7313, #7315, #7316, #7334, #7336
and #7337, six PRs red on a defect none of them introduced. #7313 has no other
red at all.

release/v3.8.49 already carries a fix (2e42b8efc, #7174: try/catch, fetch
origin/main on demand, t.skip() when unreachable), but it only reaches main at
release time — so main stays broken for the whole cycle. Cherry-picking it would
also import a new problem: PR Test Policy classifies t.skip() as a silenced
assertion, which we watched it correctly catch on #7300 today.

This is the hermetic version instead (ported from #7327, which does the same for
the release branch): read the file straight off disk, compare against an empty
base so baseTaut/baseExtTaut are 0 — the strictest possible comparison point —
and call evaluateMasking() directly. No git ref, no fetch, no skip, nothing the
runner's checkout depth can break.

The #6634 regression stays covered: the guard's logic lives in
SELF_TEST_FIXTURE_RE (check-test-masking.mjs:337), not in the test. Proven both
ways on main before committing — neutralise SELF_TEST_FIXTURE_RE to /$^/ and
the test FAILS; restore it and it passes 2/2, with check-test-masking.mjs left
byte-identical.

Co-authored-by: growab <nekron@icloud.com>

* chore(quality): tighten main's coverage baseline to the CI's real numbers (#7347)

main's ratchet had been failing --require-tighten on every PR: 11 metrics
improved but the baseline was never tightened. Same class as the #6634
selfref guard — an infra fix that lands only on the release branch leaves
main red for the whole cycle, and every PR into main pays for it.

Values are the merged-coverage numbers from a run on main itself (a local
run measures ~68% vs CI's ~80%; the baseline's own note warns about that
gap). Only the 11 coverage values change — gitleaks and semgrepFindings
keep main's own state.

No changelog fragment: #7326 carries it on release/v3.8.49, and a second
one here would double the entry at release time.

* Add cliproxy provider exposure controls and manifest injection (#7329)

* feat(fusion): let judge use its own knowledge and override the panel (#6804)

The judge prompt said to write an answer 'grounded in that analysis',
implicitly capping output at the panel's union. When all panel members
miss or are collectively wrong on something, the judge should apply its
own reasoning as a full participant and override consensus, while keeping
an honesty guard against fabrication. Adds a regression test.

Co-authored-by: Chirag Singhal <chirag127@users.noreply.github.com>

* fix(api): raise provider apiKey cap for cookie-based web providers (#6715) (#6759)

* fix(cli): fall back to settings.json when Claude Code binary is unresolvable (#6701) (#6734)

getCliRuntimeStatus() only ever answered `installed` from binary resolution
(known install paths + where/which PATH search), so a stale PATH, moved
binary, or uncatalogued install method reported "not found" even when
~/.claude/settings.json proved the CLI was installed and used before —
regressing behind upstream 9router's checkClaudeInstalled(), which already
falls back to the settings file when where/which fails.

withSettingsFallback() (new src/shared/services/cliInstallFallback.ts, kept
out of the frozen cliRuntime.ts to respect its file-size ceiling) restores
that parity: only when the binary lookup's own reason is "not_found" (never
for deliberate security rejections like unsafe/relative env overrides or
symlink escapes) and the tool's settings file exists on disk.

* fix(providers): honor explicit thinking.budget_tokens 0 in openai->gemini transform (#6813) (#6821)

The transform forwarded the Claude-style thinking.budget_tokens into
generationConfig.thinkingConfig.thinkingBudget, but the presence check was
truthy (&& thinking.budget_tokens). An explicit budget_tokens: 0 — the
natural way to disable thinking — is falsy, so it was dropped and the
request fell through to the default thinkingConfig injection, making the
model think despite an explicit request for zero. Use an explicit numeric
check so 0 is honored as thinkingBudget 0; includeThoughts is only set for
a non-zero budget.

* fix(compression): reconcile outer vs per-engine token counts (#6488) (#6741)

* fix(compression): reconcile outer vs per-engine token counts on degenerate output (#6488)

Outer originalTokens/compressedTokens (real tiktoken counter over extracted
message text) diverged from engineBreakdown[0]'s counts (a crude
JSON.stringify(requestBody).length/4 estimate), worst on small/degenerate
inputs where JSON structural overhead dominates. A single-engine breakdown
entry represents the exact same before/after transformation as the overall
response, so reconcileSingleEngineTokens() now overwrites that one entry's
counts with the outer, more accurate figures; multi-step pipeline
breakdowns are left untouched.

* chore(6741): resolve release sync — CHANGELOG.md restored to release tip, entry moved to changelog.d fragment (fragments-first)

* fix(api): accept enableRenderers in RTK compression config schema (#6703) (#6757)

* fix(db): break probe-failed/restore loop on large storage.sqlite (#6632)

Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy
files belong to #6620, not this PR); keeps only the author's changes.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* feat(cursor): add Opus 4.8, Fable 5, and Sonnet 5 model families (#6779)

Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy
files belong to #6620, not this PR); keeps only the author's cursor registry + test changes.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* fix(translator): read PDF/video file attachments for Gemini/Antigravity and Claude (#6790)

Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy
files belong to #6620, not this PR); keeps only the author's translator + test changes.

Co-authored-by: Wital <witalorocha216@gmail.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* fix(codex): strip include from compact responses requests (#6805)

* fix(codex): strip include from compact responses requests

Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy
files belong to #6620, not this PR); keeps only the author's changes.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* fix(6805): move include-strip assertion to standalone test file to keep executor-codex.test.ts under frozen size cap

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* fix(i18n): translate hardcoded Portuguese dashboard strings to English (#6769)

Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy
files belong to #6620, not this PR); keeps only the author's changes.

Co-authored-by: Chirag Singhal <chirag127@users.noreply.github.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* fix(bootstrap): filter empty process.env values to prevent Docker env crash loop (#6828)

Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy
files belong to #6620, not this PR) and the direct CHANGELOG.md edit (fragments-first);
keeps only the author's bootstrap change.

Co-authored-by: Andrian B. <andrewbalanesq@gmail.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* fix(providers): update SenseNova Token Plan support (#6330)

Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy
files belong to #6620, not this PR); the author's constants/registry/snapshot deltas
were re-applied cleanly onto the release tip.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* fix(providers): classify 404 as MODEL_NOT_FOUND to stop retry storm (#6829)

Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy
files belong to #6620, not this PR) and the direct CHANGELOG.md edit (fragments-first);
the author's chatCore/errorClassifier deltas were re-applied cleanly onto the release tip.

Co-authored-by: Andrian B. <andrewbalanesq@gmail.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* fix(api): accept all catalog engines on compression PUT schema (#6792)

Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy
files belong to #6620, not this PR). Resolved the release's OmniGlyph engine addition
additively (types.ts/compression.ts kept both 'relevance' and 'omniglyph') and extended
stackedPipelineStepSchema + STACKED_PIPELINE_ENGINE_INTENSITIES with the omniglyph branch
so the ENGINE_CATALOG-parity test passes.

Co-authored-by: Pitchfork-and-Torch <Pitchfork-and-Torch@users.noreply.github.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* fix(api): point CLI health command at /api/monitoring/health (#6677) (#6717)

* fix(api): point CLI health command at /api/monitoring/health (#6677)

bin/cli/commands/health.mjs called GET /api/health, a route that was
moved to /api/monitoring/health without updating the CLI; the top-level
/api/health handler never existed on disk (only degradation/ and ping/
sub-routes). Point runHealthCommand()/runHealthComponentsCommand() at
/api/monitoring/health and read its real payload shape
(activeConnections, circuitBreakers: {open,halfOpen,closed}, memoryUsage)
instead of the old nonexistent requests/breakers/cache/memory fields.

* chore(6717): re-sync onto release tip; move CHANGELOG entry to changelog.d fragment (fragments-first)

* chore(cursor): add Grok 4.5 effort/fast model IDs (#6774)

Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy
files belong to #6620, not this PR); keeps only the author's changes.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* fix(providers): ensure DeepSeek Web SSE emits [DONE] after FINISHED (#6791)

* fix(providers): ensure DeepSeek Web SSE emits [DONE] after FINISHED

Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy
files belong to #6620, not this PR); keeps only the author's changes.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* refactor(deepseek): extract done-terminator helper to keep frozen file under cap

Extracts the FINISHED-drain scheduler and finish-once guard added for
the [DONE] terminator fix (#6777) into a new
deepseek-web-done-terminator.ts module, so deepseek-web.ts stays under
its frozen line cap (1148). Behavior is unchanged.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Co-authored-by: Pitchfork-and-Torch <Pitchfork-and-Torch@users.noreply.github.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* feat(models): add capability override UI (#6727)

* feat(models): add capability override UI

Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy
files belong to #6620, not this PR); renumbered the migration 118 -> 119 to resolve the
collision with 118_provider_param_filters.sql already on release/v3.8.47; the author's
i18n/localDb deltas were re-applied cleanly onto the release tip.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* fix(6727): import model-capability-overrides DB fns directly (not via localDb barrel) to keep localDb under file-size cap; aligns with anti-barrel convention

* chore(db): satisfy known-symbols contract for modelCapabilityOverrides

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com>

* fix(cursor): use Agent CLI build id for x-cursor-client-version (#6795)

* fix(cursor): use Agent CLI build id for x-cursor-client-version

Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy
files belong to #6620, not this PR); the author's .env.example/docs deltas were
re-applied cleanly onto the release tip.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* chore(changelog): re-sync CHANGELOG.md to release tip (restore #6701 bullet)

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* fix(startup): rename reasoningControls.ts to avoid webpack casing collision (#6584) (#6718)

* fix(startup): rename reasoningControls.ts to avoid webpack casing collision (#6584)

* chore(6718): re-sync onto release tip; CHANGELOG entry → changelog.d fragment (fragments-first)

* fix(build): suppress Turbopack over-bundling warning from agentSkills generator (#6582) (#6720)

* fix(build): suppress Turbopack over-bundling warning from agentSkills generator (#6582)

generator.ts builds outputBase from a non-literal outputDir parameter, so
Turbopack's file-tracing analyzer can't narrow it and emits an "Overly broad
patterns" warning per entry point that imports the module (603 warnings on
v3.8.46, up from 379). The fs access is legitimate and bounded, so
next.config.mjs now suppresses this specific diagnostic via turbopack.ignoreIssue,
mirroring the existing webpack.ignoreWarnings precedent in the same file.

* chore(6720): re-sync onto release tip; CHANGELOG entry → changelog.d fragment (fragments-first)

* fix(providers): drop image_generation for Codex Spark models regardless of plan (#6651) (#6721)

* fix(providers): drop image_generation for Codex Spark models regardless of plan (#6651)

* chore(6721): re-sync onto release tip; CHANGELOG entry → changelog.d fragment (fragments-first)

* fix(providers): stop quota card re-sorting Codex/GLM bars by remaining % (#6687) (#6722)

* fix(providers): stop quota card re-sorting Codex/GLM bars by remaining % (#6687)

QuotaCardExpanded.tsx unconditionally re-sorted quotas by remaining
percentage via sortQuotasByRemaining(), discarding the deterministic
CODEX_QUOTA_ORDER/GLM_QUOTA_ORDER window order quotaParsing.ts's
sortCodexOrder()/sortGlmOrder() had already established. A new
hasFixedQuotaOrder() + resolveQuotaDisplayOrder() skip the re-sort for
providers with a fixed window order (codex, glm family), threading
providerId from QuotaCard.tsx through to the display layer.

Regression guard: tests/unit/quota-card-expanded-fixed-order-6687.test.ts

* chore(6722): re-sync onto release tip; CHANGELOG entry → changelog.d fragment (fragments-first)

* fix(startup): lazy-import ioredis in rateLimiter to fix MCP ERR_MODULE_NOT_FOUND (#6559) (#6725)

* fix(startup): lazy-import ioredis in rateLimiter to fix MCP ERR_MODULE_NOT_FOUND (#6559)

* chore(6725): re-sync onto release tip; CHANGELOG entry → changelog.d fragment (fragments-first)

* fix(resilience): resolve fp-pinned combo account back to real connection id (#6696) (#6732)

* fix(resilience): resolve fp-pinned combo account back to real connection id (#6696)

* chore(6732): re-sync onto release tip; CHANGELOG entry → changelog.d fragment (fragments-first)

* fix(api): Responses passthrough emits event-only SSE frames after filtering commentary output (#6561) (#6735)

* fix(api): Responses passthrough emits event-only SSE frames after filtering commentary output (#6561)

The #6199 commentary-drop `continue;` branches in stream.ts skipped the
data: line for a dropped commentary event but never cleared the
already-buffered event: line for the same frame, so the next blank line
flushed the stale event: line alone -- an event-only SSE frame that
crashes the OpenAI Python SDK's json.loads(). Both drop sites now call
clearPendingPassthroughEvent() before continue. The commentary-drop
decision was extracted into a new responsesCommentaryDrop.ts module so
the fix does not grow the frozen stream.ts.

* chore(6735): re-sync onto release tip; CHANGELOG entry → changelog.d fragment (fragments-first)

* fix(api): emit reasoning_content on claude-web + v0-vercel-web SSE (#6662) (#6743)

* fix(api): emit reasoning_content on claude-web + v0-vercel-web /v1/chat/completions SSE (#6662)

* chore(6743): re-sync onto release tip; CHANGELOG entry → changelog.d fragment (fragments-first)

* fix(sse): unwrap bare {function:{…}} tools in openai→claude translation (#6704)

* fix(sse): unwrap bare {function:{…}} tools in openai→claude translation

Some OpenAI-shape clients send a tool as a bare `{ function: {...} }`
object, omitting the spec-required `type: "function"` parent wrapper.
The tools-mapping in openai-to-claude.ts (~line 366) only unwrapped
`tool.function` when `tool.type === "function"` was ALSO true, so a
bare-function tool fell through to `toolData = tool` (the wrapper
itself, with no `.name`), producing an empty `originalName` and
silently dropping the tool from the translated request — worse than
a 400, since the caller has no signal the tool never made it
upstream. Unwrap `tool.function` whenever present, independent of
the parent `type` field. Regression guard:
tests/unit/openai-to-claude-bare-tool.test.ts.

Co-authored-by: Samir Abis <me@samirabis.com>
Inspired-by: https://github.com/decolua/9router/pull/2473

* chore(6704): re-sync onto release tip; CHANGELOG → changelog.d fragment (fragments-first)

---------

Co-authored-by: Samir Abis <me@samirabis.com>

* fix(oauth): avoid bare-email dedup of Codex OAuth logins (#6706)

* fix(oauth): avoid bare-email dedup of Codex OAuth logins

When an incoming Codex OAuth connection has no verifiable workspace/account
id, do not merge it into an existing row on email match alone — that
silently overwrote the other account's token pair. Require a matching
chatgptUserId (a stable per-account JWT id) before merging; otherwise
insert a distinct connection row.

Co-authored-by: lucasjustinudin <34107354+lucasjustinudin@users.noreply.github.com>
Inspired-by: https://github.com/decolua/9router/pull/2477

* chore(6706): re-sync onto release tip; CHANGELOG → changelog.d fragment (fragments-first)

---------

Co-authored-by: lucasjustinudin <34107354+lucasjustinudin@users.noreply.github.com>

* fix(sse): skip thinkingConfig for gemma models in openai→gemini translation (#6708)

open-sse/translator/request/claude-to-gemini.ts already guards against
sending thinkingConfig for gemma-4-* models (Gemma doesn't support it —
Vertex returns 400: "Thinking budget is not supported for this model"),
but the OpenAI-shape path (openai-to-gemini.ts) lacked the same guard, so
OpenAI-shape clients hitting a vertex gemma-4-* model still got a 400.

Mirrors the existing claude-to-gemini.ts guard: wrap the reasoning_effort
and Claude-shape thinking.budget_tokens branches with a model.startsWith
("gemma-4") check. Branch 3 (default includeThoughts for modern Gemini
models) already excludes non-"gemini" model ids and needed no change.


Inspired-by: https://github.com/decolua/9router/pull/2480

Co-authored-by: chy1211 <31048289+chy1211@users.noreply.github.com>

* fix(codex): surface capacity errors embedded in 200-OK SSE streams (#6710)

* fix(codex): surface capacity errors embedded in 200-OK SSE streams

Codex sometimes answers with HTTP 200 and a text/event-stream body whose
payload carries a transient error mid-stream (e.g. "Selected model is at
capacity...", server_is_overloaded, service_unavailable_error). Because the
outer HTTP status was 200, this looked like a successful response to every
caller — no retry, no circuit breaker, and no combo/account fallback ever
engaged, so a healthy account sat idle while the request silently failed or
truncated.

Add peekCodexSseTransientError() to open-sse/executors/codex.ts: it peeks the
first bytes of a text/event-stream Codex response, pattern-matches the known
transient-error signatures, and converts a match into a real 503 Response via
errorResponse() (Hard Rule #12 — sanitized, never raw upstream text). A 503 is
already a recognized provider-failure status in accountFallback.ts, so combo
routing and connection cooldown pick it up automatically. When no error
signature is found, the peeked prefix is prepended back onto the remaining
upstream body so the passthrough stays byte-identical to the unmodified
response.

Regression guard: tests/unit/codex-sse-capacity-fallback.test.ts — a
model-at-capacity payload and a server_is_overloaded/service_unavailable_error
payload both convert to 503; a normal single-chunk SSE stream and one split
across multiple network chunks both reassemble byte-for-byte unchanged.

Inspired-by: https://github.com/decolua/9router/pull/2452 (sub-bug #3 only —
OmniRoute already covers PR #2452's other two sub-bugs: service_tier "fast"
normalization and reasoning_effort "max" normalization).

Co-authored-by: ryanngit <74137224+ryanngit@users.noreply.github.com>

* chore(6710): re-sync onto release tip; CHANGELOG → changelog.d fragment (fragments-first)

---------

Co-authored-by: ryanngit <74137224+ryanngit@users.noreply.github.com>

* fix(volcengine): clamp Kimi max_tokens to Ark endpoint cap (#6712)

* fix(volcengine): clamp Kimi max_tokens to Ark endpoint cap

VolcEngine Ark's Kimi coding-plan endpoint (ark.cn-beijing.volces.com)
enforces max_tokens <= 32768 server-side and returns 400 "integer above
maximum value, expected a value <= 32768" for anything over that ceiling.
OmniRoute's StripRule only supported dropping params outright, with no
numeric clamp mechanism, so a client sending a larger max_tokens (common
default, e.g. 65536) 400s outright against volcengine's kimi-k2-5-260127.

The 32768 cap is independently confirmed against two live-endpoint bug
reports hitting this exact Ark endpoint for both kimi-k2.5 and
kimi-k2.7-code (NousResearch/hermes-agent#51773, MoonshotAI/kimi-cli#1124),
not just upstream's own value — same cap upstream 9router#2460 uses.

StripRule gains two optional fields: `clampToModelMaxOutput` (clamp to the
model's own catalog maxOutputTokens ceiling, when set) and `maxOutputCap`
(a fixed endpoint-imposed ceiling); when both apply, the lower wins. The
new rule is scoped to the literal id `kimi-k2-5-260127` (OmniRoute's real
volcengine Kimi model, not upstream's `Kimi-K2.7-Code`), not a broad
/kimi/i regex, so it can never clamp an unrelated future Kimi listing
whose Ark cap may differ. glm-4-7-251222 (the other volcengine model) is
unaffected.

Inspired-by: https://github.com/decolua/9router/pull/2460
Co-authored-by: whale9820 <whale9820@users.noreply.github.com>

* chore(6712): re-sync onto release tip; CHANGELOG → changelog.d fragment (fragments-first)

---------

Co-authored-by: whale9820 <whale9820@users.noreply.github.com>

* fix(antigravity): surface aborted Gemini tool calls off end_turn (#6713)

* fix(antigravity): surface aborted Gemini tool calls off end_turn

Gemini/Antigravity aborts a turn with finishReason MALFORMED_FUNCTION_CALL
(or a sibling like UNEXPECTED_TOOL_CALL) instead of completing cleanly. Both
Claude-facing translators collapsed these to a clean end_turn, hiding the
aborted tool call as a successful completion:
- the OpenAI hub path (openai-to-claude.ts convertFinishReason default), and
- the DIRECT Gemini->Claude path (gemini-to-claude.ts), which is the one
  Claude Code actually hits through an antigravity/Gemini-routed model.
Add isAbortFinishReason() to finishReason.ts and map these reasons to
tool_use on both paths; genuinely unknown reasons still fall back to end_turn.

Co-authored-by: anhdiepmmk <n08ni.dieppn@gmail.com>
Inspired-by: https://github.com/decolua/9router/pull/2462

* chore(6713): re-sync onto release tip; CHANGELOG → changelog.d fragment (fragments-first)

---------

Co-authored-by: anhdiepmmk <n08ni.dieppn@gmail.com>

* fix(translator): strip empty cloud_base_branch from Cursor Subagent tool call (#6729)

* fix(translator): strip empty cloud_base_branch from Cursor Subagent tool call (port from 9router#2446)

The Responses->Chat tool-arg cleanup (stripEmptyOptionalToolArgs) only stripped
empty-string/empty-array optional args for Claude Code's Read tool. Cursor's local
Subagent tool call therefore passed through with the cloud-only field
cloud_base_branch: "", which Cursor rejects ("cloud_base_branch may only be specified
when environment equals cloud") before starting the subagent. Extend the cleanup to an
allowlist of Read + Subagent; arbitrary tools stay untouched.

Reported-by: like3213934360-lab (https://github.com/decolua/9router/issues/2446)

* chore(6729): re-sync onto release tip; CHANGELOG → changelog.d fragment (fragments-first)

* fix(translator): defer content_block_start until GLM streams the tool name (#6730)

* fix(translator): defer content_block_start until GLM streams the tool name (port from 9router#2077)

GLM 5.2 (and similar OpenAI-compatible upstreams) stream a tool call's id and
function.name across separate SSE delta chunks. The openai-to-claude streaming
translator emitted content_block_start immediately on the id-only chunk with an empty
name; the Claude SSE protocol cannot patch a block after emission, so the later
name-only chunk was dropped and Claude Code rejected the tool_use with an empty tool
name / "No such tool available:". Defer content_block_start until the name arrives
(start on args if they arrive first), and emit a start for any orphaned id-only tool
call at finish so content_block_stop is never orphaned.

Reported-by: itiwant (https://github.com/decolua/9router/issues/2077)

* chore(6730): re-sync onto release tip; CHANGELOG → changelog.d fragment (fragments-first)

* feat(dashboard): add search to Playground model picker dropdown (#4086) (#6811)

* feat(dashboard): add search to Playground model picker dropdown (#4086)

The shared ModelSelectModal (combo builder + CLI-code cards) already had
search, but the Playground's raw model <select> in StudioConfigPane stayed
a flat unsearchable list - unusable once a provider like OpenRouter
contributed 50+ models.

Adds a search input above the dropdown that filters options via
filterModelsByQuery() (Turkish-safe accent/case-insensitive match, reusing
matchesSearch()). The currently selected model always stays pinned in the
list even when it doesn't match the query, so typing never silently swaps
the active selection. Reuses the existing common.search i18n key already
translated in all 42 locales - no new key needed.

* chore(6811): re-sync onto release tip; CHANGELOG → changelog.d fragment (fragments-first)

* feat: request count log per provider, per date (#4009) (#6812)

* feat(dashboard): request count log per provider, per date (#4009)

Some providers bill by request rather than by token, so operators need
a plain per-provider, per-date request count breakdown, not just token
aggregates. Adds a new getProviderDailyUsageRows() aggregation query
(src/lib/db/usageAnalytics.ts), a dedicated GET
/api/usage/requests-by-provider-date route (kept separate from the
frozen /api/usage/analytics route to respect the file-size baseline),
and a sortable, single-date-filterable table on Dashboard -> Analytics.

Closes #4009

* chore(6812): re-sync onto release tip; CHANGELOG → changelog.d fragment (fragments-first)

* feat(xai): route xAI clients to Grok native /v1/responses endpoint (#6709)

* feat(xai): route xAI clients to Grok native /v1/responses endpoint

xAI ships a native /v1/responses endpoint (https://api.x.ai/v1/responses)
alongside /v1/chat/completions, but XaiExecutor extended BaseExecutor
without overriding buildUrl(), so every request always resolved to the
static chat-completions baseUrl regardless of target format — the last
genuinely-missing slice of decolua/9router#2439 (grok-build-0.1, the
reasoning-effort suffix routing, and bare grok-* routing were already
ported in prior cycles).

Add responsesBaseUrl to the xai registry entry and tag
grok-4.20-multi-agent-0309 (upstream's own Responses-only id) with
targetFormat: "openai-responses", mirroring the existing model-tag-driven
routing pattern already used by the gh executor (9router#102) and the
"openai" -pro heuristic in open-sse/executors/default.ts — the per-model
registry tag is the single source of truth that also drives chatCore's
body translation, so URL and body stay in lockstep. XaiExecutor.buildUrl
now checks getModelTargetFormat("xai", model) and resolves to the native
Responses endpoint only for tagged models, leaving every other grok-*
model on the existing chat-completions bridge.

TDD: tests/unit/executor-xai.test.ts adds a RED-then-GREEN case asserting
grok-4.20-multi-agent-0309 resolves to https://api.x.ai/v1/responses and
a control case asserting grok-4.3 still resolves to
https://api.x.ai/v1/chat/completions.

Co-authored-by: ryanngit <74137224+ryanngit@users.noreply.github.com>
Inspired-by: https://github.com/decolua/9router/pull/2439

* chore(6709): re-sync onto release tip; CHANGELOG → changelog.d fragment (fragments-first)

---------

Co-authored-by: ryanngit <74137224+ryanngit@users.noreply.github.com>

* fix(resilience): route remaining credential-selection call sites through quota preflight (#6686) (#6742)

* fix(resilience): route remaining credential-selection call sites through quota preflight (#6686)

* chore(6742): re-sync onto release tip; CHANGELOG entry → changelog.d fragment (fragments-first)

* fix(resilience): apikey-provider 429s honor explicit quota-exhausted text (#6638) (#6731)

* fix(resilience): apikey-provider 429s honor explicit quota-exhausted text (#6638)

Ollama Cloud (and any other apikey-category provider) 429s skipped body-text
quota classification entirely; a genuine multi-day quota exhaustion was
misclassified as a plain rate_limit_exceeded with a few seconds of cooldown,
so combo routing retried the account immediately. shouldPreserveQuotaSignals()
now lets an explicit quota-exhausted signal (looksLikeQuotaExhausted) override
the apikey-category default, and parseDayGranularityResetMs() adds day-
granularity reset-hint parsing ("...reset in 3 days.") alongside the existing
Xh/Ym/Zs parsing.

Regression guard: tests/unit/issue-6638-ollama-quota.test.ts (RED before the
fix, GREEN after). Aligned two tests/unit/account-fallback-service.test.ts
cases that had codified the old buggy behavior for apikey-provider quota
text.

* chore(6731): re-sync onto release tip; CHANGELOG entry → changelog.d fragment (fragments-first)

* feat(resilience): weekly-429 cooldown for fetcher-less providers (#3709) (#6817)

* feat(resilience): weekly-429 cooldown for fetcher-less providers (#3709)

Ollama Cloud free-tier accounts have a hard WEEKLY request cap. On cap the
upstream returns 429 "you (<account>) have reached your weekly usage
limit", but ollama-cloud is an apikey-category provider, so the existing
oauth-only shouldUseQuotaSignal gate in checkFallbackError skips the
subscription-quota-text classifier (Issue #2321) for its 429s -- the
account fell through to the generic exponential backoff (~1s, capped at
2min) and got retried every few minutes for the rest of the week (one
account took 285x429 in 48h).

Adds a new, ungated weekly-usage-limit text classifier that applies a 24h
QUOTA_EXHAUSTED cooldown regardless of provider category. Extracted the new
classifier -- together with the existing #2321 subscription-quota logic --
into a new open-sse/services/quotaTextCooldowns.ts module so the frozen
accountFallback.ts (file-size-baseline cap) didn't have to grow; net effect
shrinks accountFallback.ts by 20 lines.

This is Phase A of the plan (open-sse/services/accountFallback.ts:1038-1045
"weekly-429 cooldown"); Phase B (generic local request-counter preflight for
manual provider_plans dimensions) is a separate, larger follow-up per the
plan's own phasing.

* chore(6817): re-sync onto release tip; CHANGELOG entry → changelog.d fragment (fragments-first)

* fix(providers): Kiro adaptive-thinking allowlist excludes sonnet-4.5/haiku-4.5 (#6576) (#6726)

* fix(providers): Kiro adaptive-thinking allowlist excludes sonnet-4.5/haiku-4.5 (#6576)

* chore(6726): re-sync onto release tip; CHANGELOG entry → changelog.d fragment (fragments-first)

* test(kiro): migrate selector-strip test to claude-sonnet-5 (only Kiro adaptive-thinking model, #6576)

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* chore(quality): rebaseline complexity 2053->2054 (merge-burst drift, v3.8.47)

Inherited drift from today's /implement-prs merge burst (~36 PRs). check:complexity
does not run on the PR->release fast-path, so the branch accrued +1 unmeasured. No
orphan/feature PR introduces a NEW violation (complexity-net-zero); the only flagged
function is the pre-existing getResolvedModelCapabilities. Owner-approved rebaseline
to unblock the FQG of ~7 green-except-complexity orphans.

* chore(stryker): register ollama-quota covering tests (merge-burst drift, v3.8.47)

The 3 covering unit tests from #6731/#6817/#6742 (issue-6638-ollama-quota,
ollama-cloud-weekly-quota-cooldown-3709, issue-6686-quota-preflight-coverage)
exist on release but were never added to tap.testFiles when those PRs merged.
Completes the registration so mutant kills count; unblocks every PR touching a
mutated module. Part of the owner-approved merge-burst drift cleanup.

* fix: auto-start WS server in-process and change default port to 20132 (#6072)

* feat: change default LIVE_WS_PORT from 20129 to 20132

Update the default WebSocket port for the live dashboard server from 20129 to 20132 across all configuration files, documentation, code comments, and tests. Also consolidate OMNIROUTE_DISABLE_LIVE_WS and OMNIROUTE_ENABLE_LIVE_WS into a single OMNIROUTE_ENABLE_LIVE_WS flag. Wire the live WebSocket server to start in-process via instrumentation-node.ts.

* feat: clarify NEXT_PUBLIC_LIVE_WS_PUBLIC_URL path usage and derive upgrade path from URL

Update .env.example and ENVIRONMENT.md to document that the pathname portion of NEXT_PUBLIC_LIVE_WS_PUBLIC_URL (e.g. /live-ws) is used as the WebSocket upgrade path by the dev proxy, handshake response, and client connection logic.

Extract deriveLiveWsPath() into shared/utils/wsPath.ts and wire it through:
- src/app/api/v1/ws/route.ts — handshake response path field
- src/hooks/useLiveDashboard.ts — build

* fix: use the standard URL API to safely parse and update the effectiveWsUrl

* build(docker): expose live WebSocket server port and configure CORS origins

Add LIVE_WS_PORT (20132), LIVE_WS_HOST (0.0.0.0), and LIVE_WS_ALLOWED_ORIGINS environment variables to all Docker Compose profiles and expose the WebSocket port mapping. Prevent infinite self-loop in standalone-server-ws.mjs by skipping proxy when the server itself is running on the LiveWS port.

* docs(env): fix comment formatting for HOST and HOSTNAME variables

---------

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* fix(logs): prevent stale detail refresh reopening modal (#6323)

* fix(logs): prevent stale detail refresh reopening modal

* chore(stryker): register ollama-quota covering tests (release drift from merge burst)

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* \ feat: operator-configurable account rotation\ (#6763)

* feat(resilience): operator-configurable account rotation

Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy
files belong to #6620, not this PR); the author's accountFallback/.env deltas were
re-applied cleanly onto the release tip.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* docs(env): document configurable account-rotation env vars in ENVIRONMENT.md

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* refactor(rotation): extract rotation gate/context helpers to keep accountFallback.ts under frozen cap

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* chore(changelog): re-sync CHANGELOG.md to release tip (restore lost base bullet)

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* test(stryker): register rotation-config test in tap.testFiles for mutation coverage

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* test(stryker): register ollama-quota covering tests (drift from #6731/#6817/#6742) + re-sync

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com>

* fix(lmarena): modernize Arena web provider + static Direct-chat catalog (#6280)

* fix(lmarena): modernize Arena web provider + static Direct-chat catalog

Update the lmarena provider for arena.ai (product rebranded from LMArena):

- Route chat via arena.ai create-evaluation with Chrome TLS impersonation
  (tls-client-node) and optional browser-minted recaptchaV3Token.
- Seed Text+Search (48) into the chat registry; seed Image (27) only into
  IMAGE_PROVIDERS. Disable live HTML model discovery; resolve public names to
  Arena UUIDs from the static TypeScript allowlist (no scrape JSON in-repo).
- Soft-exclude 404/502 model ids; slow/stop bulk test-all probes for this provider.
- Do not fold IMAGE_PROVIDERS/video specialty into the chat provider catalog when
  a chat registry already exists (lmarena/openai/xai).
- Display name Arena (Free); keep wire id `lmarena` / alias `lma` for back-compat.
- Theme-aware provider icons: arena-light.svg / arena-dark.svg.
- Preserve split Supabase SSR cookie reconstruction for arena-auth-prod-v1.*.

* fix(providers): align provider-models-route test fixture + regen provider reference

Fold the topaz image-only catalog entry's apiFormat/supportedEndpoints
into the local-catalog test fixture (route now tags media-only
providers per the lmarena PR's staticModels.ts change), regenerate
PROVIDER_REFERENCE.md against the merged release providers.ts, and add
the changelog fragment for #6280.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* test: align web-cookie fallback suite — lmarena now has a registry entry (probe path)

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* docs(changelog): reconcile 3-day merge burst — 16 fragments, 4 promised credits, contributors hall 32→63

- changelog.d fragments for the 20 merged PRs that landed without a bullet
  (#6072 #6308 #6323 #6538 #6556 #6586 #6611 #6647 #6675 #6698 #6757 #6759
  #6804 #6821 + ci rollup #6781/#6691/#6693 + docs rollup #6643/#6644/#6646/#6663;
  omniglyph bump #6661 folded into the #6556 bullet)
- deliver the 4 credits promised in close comments but never written:
  @alltomatos (#6819 dup of #6721), @samimozcan (#6762/#6753 subsumed by #6790),
  @chirag127 (#6756 dup of #6757), @Squawk7777 (#6565 dup of #6564 — appended to
  the existing #6564 bullet; changelog-integrity flags that edit as a removal,
  intentional: ALLOW_CHANGELOG_REMOVALS justification)
- rebuild the v3.8.47 Contributors hall from merged-PR authors + thanks credits
  + prior hall: 32 → 63 contributors

* Clamp reasoning token buffer to model output cap (#6714)

* fix(combo): clamp reasoning buffer to model output cap

* fix(routing): preserve near-cap reasoning max tokens

* fix(routing): getExplicitModelOutputCap falls through to registry cap on non-numeric synced limit_output

getExplicitModelOutputCap short-circuited to null whenever a synced
capability row existed, even if that row's limit_output was not a number
(models.dev commonly omits it). That silently disabled the reasoning-token
buffer clamp for any model with a synced row lacking an output limit.

Now only return the synced value when it IS a number; otherwise fall
through to registryModel.maxOutputTokens / spec.maxOutputTokens, matching
the ??-chain precedence already used by getResolvedModelCapabilities().

Adds a standalone regression test (proves the fallthrough returns the real
registry cap, not null) and hardens the #6274 fixture id so its no-output-cap
case does not prefix-match the real glm-5.2 static spec.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* chore(stryker): register ollama-quota covering tests (release drift from merge burst)

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* feat(i18n): add Traditional Chinese (zh-TW) localization for frontend and CLI (#6320)

* feat(i18n): add Traditional Chinese (zh-TW) localization for frontend and CLI

- Add src/i18n/messages/zh-TW.json translating frontend web UI
- Add bin/cli/locales/zh-TW.json translating CLI commands and descriptors
- Register zh-TW in config/i18n.json and docs/guides/I18N.md
- Update scripts/i18n/generate-multilang.mjs matching the new locale setup

* fix: update i18n locale count from 42 to 43 after adding zh-TW

The docs strict checker (check-docs-counts-sync.mjs) validates that
README.md and I18N.md reflect the real locale count. Adding zh-TW
bumped the count from 42 → 43.

* fix(i18n): translate providers free-filter labels in zh-TW (#6694 guard)

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com>
Co-authored-by: Diego Rodrigues de Sa e Souza <8016841+diegosouzapw@users.noreply.github.com>
Co-authored-by: lunkerchen <lunkerchen@users.noreply.github.com>

* feat(proxy): implement latency-optimized proxy rotation strategy (#6798)

* feat(proxy): implement latency-optimized proxy rotation strategy

Reconstructed onto release/v3.8.47 to drop unrelated main-drift (deps/electron/proxy
files belong to #6620, not this PR) and the direct CHANGELOG.md edit (fragments-first);
the author's env/docs/i18n deltas were re-applied cleanly onto the release tip.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* fix(proxy): add latency-rotation env var to .env.example

PROXY_LATENCY_WINDOW_HOURS was referenced in src/lib/db/proxies.ts and
documented in docs/reference/ENVIRONMENT.md, but missing from
.env.example, tripping the env/docs sync gate.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* chore(changelog): re-sync CHANGELOG.md to release tip

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* refactor(proxy): extract latency-strategy helpers to keep frozen files under cap

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* test(db-rules): expect 35 audited modules (proxyLatency joins INTENTIONALLY_INTERNAL)

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* docs(readme): fix stale strategy/tool/scoring counts (#6853)

README still claimed 17 routing strategies (the table was missing
pipeline), 95 MCP tools, and 9-factor Auto-Combo scoring. Align with
the source (ROUTING_STRATEGY_VALUES has 18 entries) and the canonical
docs (MCP-SERVER.md: 94 tools; AUTO-COMBO.md: 12-factor).

* fix(antigravity): sanitize Cloud Code safety settings (#6839)

Co-authored-by: kfiramar <83420275+kfiramar@users.noreply.github.com>

* fix(kiro): probe IdC region during profileArn discovery, cross-region (recovers #6099) (#6840)

* fix(kiro): route Amazon Q runtime by profileArn region for cross-region IdC

Enterprise AWS IAM Identity Center accounts whose IdC instance lives outside the two
Amazon Q Developer profile regions (us-east-1 / eu-central-1) - e.g. eu-north-1
(Stockholm), start URL https://d-XXXX.awsapps.com/start - showed no limits and returned
502 on every request.

Root cause: the backend used the IdC/OIDC token region (providerSpecificData.region, e.g.
eu-north-1) for every CodeWhisperer runtime call, hitting q.eu-north-1.amazonaws.com - a
host that does not exist as a Q Developer runtime endpoint. Per AWS docs ("Supported
Regions for the Q Developer console and Q Developer profile"), the Q Developer *profile*
(which produces the profileArn and hosts generateAssistantResponse / GetUsageLimits /
ListAvailableModels / ListAvailableProfiles) is only hosted in us-east-1 and eu-central-1,
regardless of the IdC region; "data is stored in the Region where you create the Amazon Q
Developer profile."

Fix (new open-sse/services/kiroRegion.ts) decouples the two regions:
- providerSpecificData.region stays the IdC/OIDC region, used ONLY for
  oidc.{region}.amazonaws.com token mint/refresh.
- The runtime region is derived from the profileArn (resolveKiroRuntimeRegion):
  profileArn region -> a valid stored profile region -> us-east-1. A stored IdC region
  that is not a Q profile region (eu-north-1) is ignored for runtime.
- Profile discovery (discoverKiroProfileArnAcrossRegions) probes the Q profile regions
  (EU IdC -> eu-central-1 first) with the cross-region SSO token instead of q.{idcRegion}.

Wired into: executors/kiro.ts (generateAssistantResponse targets the profile region),
services/usage/kiro.ts (getKiroUsage multi-region discovery + profileArn runtime region so
Limits resolves), services/kiroModels.ts (ListAvailableModels), and
src/lib/oauth/providers/kiro.ts (login-time postExchange profile discovery).

Adds tests/unit/kiro-idc-cross-region.test.ts (15 cases). All Kiro suites pass (60 tests).

* fix(kiro): probe the IdC region too during profileArn discovery (any IdC region)

Make profile discovery general for an IdC in ANY of the ~30 IdC-supported AWS regions
(us-west-2, ap-southeast-2, me-central-1, af-south-1, ...), not just eu-north-1.

buildKiroProfileDiscoveryRegions now probes the two documented Q Developer profile regions
FIRST (us-east-1 / eu-central-1, EU-first for EMEA IdC regions to cut latency), then appends
the IdC/stored region itself as a forward-compatible fallback: if AWS ever co-locates the
profile with the IdC or expands the profile-region list, a same-region probe still finds it.
Probing a region with no profile simply returns nothing and we fall through. The profileArn's
own region remains authoritative for every runtime call (resolveKiroRuntimeRegion), so a
newly-issued ARN in any region is honored automatically.

Adds ap-southeast-2 (APAC) cross-region coverage and updates the discovery-order tests.

---------

Co-authored-by: artickc <artur1992123@mail.ru>

* feat(providers): manual context-window override for custom models (#4125) (#6822)

Add a manual per-model "Context Window Override" so an operator can correct
a provider's misreported context length (e.g. reports 1M when the real
limit is 128K) instead of the model getting silently dropped from combo
routing once the wrong value lands in the catalog.

Reuses the existing Feature-5004 model_context_overrides table
(source="manual") — already the priority-0 source getModelContextLimit()
(the function combo's context-window filter calls) reads ahead of the
models.dev/registry/static catalog — so no new resolver logic was needed,
only the missing write path:

- PUT /api/provider-models now accepts an optional contextWindowOverride
  (number to set, null to clear), persisted via setModelContextOverride/
  removeModelContextOverride.
- GET /api/provider-models surfaces the current override value + source
  back on each custom-model row.
- CustomModelsSection.tsx: edit form gained a Context Window Override
  input + a badge on the model row when an override is set.

Regression guard: tests/unit/provider-models-context-window-override-4125.test.ts
(manual override wins over a misreported catalog value, GET round-trip,
clearing via null, default-unchanged behavior).

* feat(dashboard): improve Provider Quota page horizontal density (#3520) (#6815)

QuotaCardGrid stacked every provider group vertically in a single
flex flex-col container, and each group's own card grid didn't go
multi-column until the md breakpoint. Provider groups now flow into a
2-column CSS multi-column layout on very wide (2xl) screens instead of
an unconditional vertical stack, and each group's card grid starts at
2 columns immediately, filling horizontal whitespace sooner on
narrower-but-not-mobile viewports.

Regression guard: tests/unit/quota-card-grid-horizontal-layout.test.ts

* refactor(usage): type saveRequestUsage with UsageEntry interface + any-budget ratchet (#3512) (#6809)

Replace saveRequestUsage(entry: any) with a typed UsageEntry interface
mirroring the usage_history columns 1:1. Fields stay optional/nullable
since different writers (chatCore success/failure, rejected-request
accounting, Codex Responses WS) populate the row incrementally; tokens
stays unknown since callers pass either raw provider-shaped usage or
the normalized {input,output,cacheRead,...} shape.

Also cleaned the file's other any usages (getUsageHistory filter,
getUsageDb next-cursor cast, appendRequestLog tokens param,
getRecentLogs catch) so it now sits at zero any and can be added to
the check:any-budget:t11 zero-any allowlist.

Documents the DB-entity <-> TS-interface convention in
docs/architecture/CODEBASE_DOCUMENTATION.md Sec 11.

* feat(combo): strict budget-cap fallback policy for auto/* combos (#3470) (#6816)

Auto-combo transparency + budget controls: the engine's budgetCap enforcement
always degraded to the globally cheapest candidate when every candidate
exceeded the cap - silently overspending instead of respecting the cap.

- engine.ts: budgetFallback "cheapest" (default, legacy) | "strict"
  (BudgetExceededError when no candidate fits budgetCap)
- requestControls.ts: X-OmniRoute-Budget-Fallback header +
  resolveRequestAutoControls() consolidating mode/budget/fallback parsing
- resolveAutoStrategy.ts / autoConfig.ts: thread combo-level
  config.budgetFallback and catch BudgetExceededError into an HTTP 402
- chat.ts: switch to the consolidated resolveRequestAutoControls() helper
  (net line reduction, stays under the frozen file-size baseline)

Regression guard: tests/unit/auto-combo-budget-fallback-3470.test.ts

* fix(usage): honor xAI provider-reported exact cost (#6711)

OmniRoute's calculateCost() always estimated request cost from token
counts x static pricing, discarding xAI's exact provider-reported cost
when present. xAI's chat-completions usage object reports the precise
billed cost via cost_in_usd_ticks (docs.x.ai/developers/cost-tracking
and the API reference's usage schema: "TICKS_IN_USD_CENT: i64 =
100_000_000" => 1e10 ticks/USD, e.g. 37756000 ticks ~= $0.0038).

calculateCost()/computeCostFromPricing() now short-circuit to this
exact figure when present -- before any pricing DB lookup, so it also
works for models without a local pricing row -- and still fall back to
the token-based estimate when it is absent. The field is threaded
through both the streaming (extractUsage/normalizeUsage) and
non-streaming (extractUsageFromResponse) usage-extraction paths.

Corrected divisor vs upstream: the upstream PR used /1e12 (a 100x
under-report, e.g. reporting $0.00123 as the doc's $0.123 example);
this port uses the doc-verified /1e10 instead, confirmed against both
the cost-tracking guide and the API reference's usage-object schema.

Inspired-by: https://github.com/decolua/9router/pull/2453

Co-authored-by: ryanngit <74137224+ryanngit@users.noreply.github.com>

* docs: rename /implement-prs → /merge-prs in Hard Rule #21 (skill renamed 2026-07-11) (#6847)

* docs: refresh stale llm.txt facts + relocate design.md to docs/architecture/DESIGN_SYSTEM.md (#6849)

* docs: refresh stale llm.txt facts + move design.md to docs/architecture/DESIGN_SYSTEM.md

llm.txt was frozen at the v3.8.8 era (177 providers, 37 MCP tools, 14
strategies, 9-factor scoring, 75% coverage gate). Update every factual
claim to the current state (248 providers, 94 tools / 30 scopes, 18
strategies, 12-factor scoring, ratchet + 60% floor, TS 6, current docs/
layout) and re-sync the 42 exact-copy i18n mirrors.

design.md at the root was a standardization plan whose phases 1-6 all
shipped; rewrite its header as a permanent reference and relocate it to
docs/architecture/DESIGN_SYSTEM.md per the root-hygiene policy (root =
configs + canonical docs only).

* docs: add MDX frontmatter to DESIGN_SYSTEM.md (in-app docs pipeline requires it)

* feat: per-model web-search interception rule (#3384) (#6814)

* feat(routing): per-model web-search interception rule (#3384)

Adds a per-provider/per-model interceptSearch rule (src/lib/db/interceptionRules.ts,
key_value namespace interception_rules) that overrides the existing native
web-search bypass defaults (Codex/Gemini/Claude->Claude passthrough) in
webSearchFallback.ts. Wired at the existing prepareWebSearchFallbackBody() call
site in chatCore.ts. Resolution precedence: per-model rule > provider-level rule
> existing native-bypass defaults.

This lands Phase 1-2 of the plan (rule store + search interception). Web-fetch
interception and the dashboard UI toggle are tracked as follow-up phases.

* fix(db): register interceptionRules in localDb re-export layer (db-rules gate)

* fix(db): renumber interception_rules migration 119→120 (collision with model_capability_overrides)

* feat: sidebar search/filter input (#4013) (#6810)

* feat(dashboard): add search/filter input to the dashboard sidebar (#4013)

Adds a search box at the top of the expanded sidebar that filters nav
sections/groups/items client-side by label, so users don't have to
hunt through the growing nav tree. Reuses the existing common.search /
common.noResults i18n keys (no new locale edits needed) and the shared
Input icon="search" pattern. Matching sections auto-expand while
searching and the accordion/pin state is restored once the query is
cleared.

Filtering logic is extracted into a pure filterSidebarSectionsByQuery()
helper (src/shared/utils/sidebarSearch.ts) so it is trivially unit
testable independent of React/next-intl/next-navigation.

* fix(test): move Sidebar.search test to a runner-collected path (test-discovery gate)

* fix(i18n): backfill 194 missing pt-BR keys (#6695) (#6723)

* fix(i18n): backfill 194 missing pt-BR keys and add key-parity regression test (#6695)

* Merge branch 'release/v3.8.47' into fix/6695-i18n-drift

Resolve i18n key-parity and CHANGELOG-fragment conflicts:
- Convert the #6695 CHANGELOG.md bullet to a changelog.d/ fragment
  (the fragment convention landed on release/v3.8.47 after this PR
  branched, per changelog.d/README.md).
- Backfill 61 additional pt-BR keys that entered en.json on
  release/v3.8.47 after this PR's original 194-key backfill, so the
  PR's own key-parity regression test (tests/unit/i18n-pt-br.test.ts)
  stays green against the moving release baseline.

* Discover live Codex models (#6776)

* Add live model discovery for provider catalog

* Fix model discovery request headers

* fix(codex): sync live model limits with local catalog

* test(codex): split live model discovery coverage into dedicated route tests

* fix(codex): use chatgpt account id for live model sync

* Add GitHub-backed Codex model discovery fallback

* fix(providers): tighten oauth config tests and provider model display comments

* test: align client version expectations with release default

* fix(codex): keep discovery complexity within baseline

* fix: rebase live Codex model discovery onto release/v3.8.47, preserving kimi-web buildHeaders (#6308)

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* feat(codex): echo requested effort-suffixed model id in Responses payloads (#3697) (#6820)

* feat(codex): echo requested effort-suffixed model id in Responses payloads (#3697)

Codex CLI compatibility shim: the Responses API response.created/
response.in_progress/response.completed payloads now carry a `model`
field (previously absent), and for Codex-CLI-originated requests it
echoes the client-requested effort-suffixed model id (e.g.
gpt-5.5-xhigh) instead of the bare upstream id (gpt-5.5), so the Codex
CLI status line/model button shows the active reasoning effort.

- openai-responses.ts translator threads the upstream model into the
  Responses event objects (additive, omitted when unknown).
- New isCodexOriginatedHeaders() (codexIdentity.ts) reuses PR #3481's
  originator/User-Agent detection, header-based so it still fires when
  a combo routes codex/gpt-5.5-xhigh to a non-codex upstream.
- chatCore's existing opt-in #1311 echoModel pipeline now also fires
  automatically for Codex clients on the Responses API, regardless of
  the echoRequestedModelName setting.
- responseModelEcho.ts now also rewrites the nested response.model
  field the Responses API uses (previously only top-level model).
- /v1/models keeps returning models: [] for Codex (unchanged, #3481).

Regression guard: tests/unit/codex-effort-model-echo-3697.test.ts.

Closes #3697

* chore(merge): re-sync with release/v3.8.47 (restore CHANGELOG, keep own bullet)

* chore(merge): re-sync with release/v3.8.47; move changelog bullet to changelog.d fragment (merge-storm proof)

* feat(usage): surface Antigravity weekly quota alongside the 5-hour window (#4017) (#6818)

* feat(usage): surface Antigravity weekly quota alongside the 5-hour window (#4017)

Antigravity enforces both a 5-hour and a weekly usage limit, but the agy/antigravity
quota widget only exposed the 5-hour window. The weekly limit isn't in the per-model
retrieveUserQuota response already fetched — it lives in a separate, undocumented
retrieveUserQuotaSummary RPC that groups models into families (Gemini Models, Claude
and GPT models) with one weekly bucket per family.

Adds a self-contained usage/antigravityWeeklyQuota.ts leaf: a cached, best-effort
fetch of that RPC + a pure parser that extracts the weekly-labeled bucket per group
(window inferred from bucketId/displayName text, matching the reverse-engineered
shape documented by third-party Antigravity clients) into gemini_weekly/
claude_gpt_weekly quota entries, merged into the existing quotas map the widget
already renders generically. A failed/unavailable RPC never affects the existing
per-model quotas.

Live VPS validation attempt (192.168.0.15, real antigravity account): both
retrieveUserQuota and retrieveUserQuotaSummary currently return 429
RESOURCE_EXHAUSTED for that account, so the live response shape could not be
captured directly. The parser was instead validated via TDD against the bucket
shape documented by CodexBar (steipete/CodexBar), a third-party Antigravity client
that reverse-engineered the same RPC, and is defensive against both response
envelopes it has observed (top-level groups[] and nested quotaSummary.groups[]).

* chore(merge): re-sync with release/v3.8.47 (restore CHANGELOG, keep own bullet)

* chore(merge): re-sync with release/v3.8.47; move changelog bullet to changelog.d fragment (merge-storm proof)

* feat: add Z.ai Web free web-cookie provider (#4056) (#6823)

* feat(providers): add Z.ai Web free web-cookie provider (#4056)

New zai-web web-session provider drives the free chat.z.ai consumer
chat UI via a pasted browser cookie, distinct from the existing
API-key zai/glm/glm-cn/glmt providers (api.z.ai). ZaiWebExecutor posts
to chat.z.ai/api/chat/completions with the cookie forwarded both as
Cookie and Authorization: Bearer <token>, and normalizes both z.ai's
internal delta_content/phase SSE envelope and a pass-through
OpenAI-shaped choices[].delta frame into standard chat-completion
chunks.

Registered in WEB_COOKIE_PROVIDERS, WEB_SESSION_CREDENTIAL_REQUIREMENTS,
the provider registry (GLM-4.6/4.5/4.5V models), the executor factory,
and tokenExtractionConfig.ts for in-app cookie capture.

* fix(providers): regenerate translate-path golden for zai-web + reduce cognitive complexity

* fix(providers): rename ZaiWebExecutor.buildHeaders to avoid incompatible BaseExecutor override

* chore(merge): re-sync with release/v3.8.47; move changelog bullet to changelog.d fragment (merge-storm proof)

* fix(codex): bump default client version to 0.144.0 (#6780)

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* refactor(usage): extract per-group parsing in antigravityWeeklyQuota (cognitive-complexity gate 886→885, release-level drift from #6818 merge)

* ci(quality): cut PR gate wall time without dropping protection (#6716)

Collapse duplicate CI spend while keeping each gate's existence reason:

- quality.yml: TIA __RUN_ALL__ defers full unit to fast-unit 4-shard (#6781);
  path filters via classify-pr-changes; docs-gates split; draft skip
- ci.yml: wire docs/i18n/code path filters; ESLint JSON artifact for quality-gate;
  drop advisory typecheck:noimplicit; float actions/cache@v6
- TIA parity: memory/usage/combo/serial; **/*.test.mjs any depth; electron/bin
  no longer force unit __RUN_ALL__
- check:complexity-ratchets: one ESLint walk, ruleId-isolated baselines + cache
- check:api-docs-refs + lib/apiRoutes: shared API route inventory
- husky pre-push: intentionally light (gates live in pre-commit); CLAUDE.md +
  QUALITY_GATES.md docs synced
- collect-metrics / lint:json: path.resolve cache path; Windows-safe eslint bin
- env-doc allowlist for ESLINT_RESULTS_JSON / COMPLEXITY_ESLINT_REPORT
- release-green --full-ci expects check:api-docs-refs (not docs-symbols alone)

Tests: select-impacted, classify-pr-changes, api-routes lib, complexity-rule-count,
validate-release-green.

Reconciled after #6781 (fast-unit 2→4 shards) per maintainer request on #6716.

Co-authored-by: Diego Rodrigues de Sa e Souza <8016841+diegosouzapw@users.noreply.github.com>

* fix(docs): document Turbopack build memory tradeoff for RAM-constrained machines (#6409) (#6885)

* fix(routing): recognize Kimi token-limit 400 as context overflow for combo fallback (#6637) (#6893)

combo.ts's isContextOverflow400() guard required the literal word
'context' in the 400 error body before letting a combo fall through to
the next target. Kimi's exact wording ('Your request exceeded model
token limit: 262144 (requested: 308458)') never says 'context', so the
guard misclassified it as a body-specific error and halted the whole
combo instead of trying the next (larger-context) target.

accountFallback.ts's CONTEXT_OVERFLOW_PATTERNS already recognized this
wording one layer below (via checkFallbackError -> shouldFallback), so
the two independently-maintained classifiers disagreed and the
stricter one won. Export CONTEXT_OVERFLOW_PATTERNS from
accountFallback.ts and reuse it inside combo.ts's
isContextOverflow400() so both layers share a single source of truth.

Regression test: tests/unit/repro-6637-kimi-token-limit.test.ts
(RED on unfixed code -> GREEN after the fix). Existing #4519 guard
tests (tests/unit/combo-param-validation-fallback-4519.test.ts) still
pass, including the negative case that a genuinely body-specific 400
is NOT misclassified as overflow.

* fix(providers): honor a provider-level proxy assigned to no-auth providers (#6272) (#6895)

No-auth providers (mimocode, opencode, ...) are always dispatched with a single
hardcoded connectionId ("noauth" — SYNTHETIC_NOAUTH_CONNECTION_ID in
src/sse/services/auth.ts). No provider_connections row ever has id="noauth", so
resolveProxyForConnection() in src/lib/db/settings.ts could never populate
connectionRecord for them, and its provider-level proxy lookup (Steps 6/8) only
runs when connectionRecord is present. A proxy assigned via Settings -> Providers
-> mimocode was therefore silently ignored, reproducing the reporter's "same
thing happen when i set the proxy directly in the provider menu" symptom.

Adds a best-effort fallback (src/lib/db/settings/noAuthProxyFallback.ts): when
connectionRecord could not be resolved, scan the known no-auth provider ids for a
configured provider-level proxy (registry first, then legacy) before falling
through to the global/direct steps.

Regression test: tests/unit/proxy-noauth-provider-6272.test.ts (RED on unfixed
code — resolved to level=direct/proxy=null; GREEN after the fix).

* fix(dashboard): surface Claude extraUsage credits in quota card (#6806) (#6896)

Enterprise-tier Claude accounts (default_raven_enterprise) don't get
five_hour/seven_day utilization windows from Anthropic's OAuth usage
endpoint — only an extra_usage credit-billing block. parseClaude()
only read data.quotas, so quotas stayed {} and the dashboard showed
"No quota data" even when extraUsage showed the account 100%
exhausted. parseClaude() now folds an enabled extraUsage block into a
credits-style quota row (mirroring parseCodex's bankedResetCredits
pattern), both when quotas is empty and when it's already populated.

* fix(db): share sql.js preinit across callers, fix named-param bind (#6628, #6802) (#6899)

- preInitSqlJs() now memoizes an in-flight Promise (not just the resolved
  adapter) per filePath, so concurrent BATCH/STARTUP/HealthCheck/
  ProviderLimitsSync callers at boot share one full-file read+WASM decode
  instead of each independently reloading the whole database — the
  thundering-herd amplifier of the OOM condition #6632 already partly
  fixed, left un-implemented by the reporter's own proposed fix (#6628).

- sqljsAdapter's run/get/all now unwrap a lone named-parameter object
  (e.g. .all({ isActive: 1 }) for "WHERE is_active = @isActive", the same
  call shape getProviderConnections() already uses against better-sqlite3)
  before calling sql.js's stmt.bind(), expanding it to the @/:/$ sigil
  variants sql.js's own named-bind path requires. Previously the object
  was wrapped into an array and sql.js took the positional-bind path,
  throwing "Wrong API use : tried to bind a value of an unknown type
  ([object Object])." whenever the sql.js WASM fallback driver was active
  — exactly the error #6802 reported (misattributed to better-sqlite3).

Regression tests added to tests/unit/db-adapters/driverFactory.test.ts and
tests/unit/db-adapters/sqljsAdapter.test.ts, both proven RED against the
prior code and GREEN after the fix.

* fix(plugin): split OC-gate provider id from OmniRoute-facing routing id (#6859) (#6900)

resolveOmniRoutePluginOptions() auto-prefixes providerId with "opencode-"
(commit 75b52e286) so OpenCode 1.17.8+'s native-adapter gate accepts it as a
registered provider id. That prefixed value was being reused for the
OmniRoute-server-facing identifiers too: mapRawModelToModelV2's id/providerID,
mapComboToModelV2's providerID, and the dynamic provider hook's combo catalog
keys. OmniRoute's server has no "opencode-<x>" provider alias, so every
dispatched model failed credential lookup with "No credentials for
opencode-omniroute" / "No active credentials for provider:
opencode-omniroute".

Add a …

* fix(combo): auto-clear stale session pins and emit recovery hints on combo exhaustion

When a custom combo's session pin targets an unhealthy provider or all
combo targets exhaust without a single success, the user sees only an
opaque 5xx error with no guidance on what to do next. The session
remains pinned to the dead combo config and retries keep hitting the
same stale targets.

Root cause: three interconnected gaps in the combo termination path:
1. No per-session consecutive-failure tracking — the system cannot
   distinguish a transient error from a permanently dead route.
2. No automatic pin clearing — session_model_history keeps routing
   to the stale pin indefinitely.
3. No recovery guidance in the error response — the user has no
   visible signal that they should switch to a different model/combo.

This commit adds two recovery mechanisms:

1. Consecutive-failure tracker (open-sse/services/combo/failureTracker.ts)
   - Tracks failures per (sessionId, comboName) pair with TTL eviction
   - After COMBO_FAILURE_THRESHOLD (3) consecutive failures, auto-
     clears the stale session pin so subsequent requests re-evaluate
     from scratch
   - Reset-on-success for healthy routes
   - Fail-open: exceptions caught and return safe defaults
   - In-memory Map (no DB writes) — losing the counter on process
     restart is acceptable

2. Recovery hints in combo diagnostics (open-sse/utils/error.ts)
   - New ComboRecoveryHint type with action (try-auto | switch-combo |
     wait | retry) and human-readable next_step
   - sanitizeRecoveryHint validates fields and strips unsafe content
   - errorResponseWithComboDiagnostics emits x-omniroute-recovery-*
     HTTP headers for client-side consumption
   - Recovery field embedded in JSON response body for non-header-
     aware consumers

The OC plugin already parses x-omniroute-recovery-* headers in the
fetch interceptor and injects recovery_hint into error bodies
(shipped in v3.8.47). These server-side changes complete the pipeline.

* fix(combo): scope failureTracker pin-clear to the failing session only

recordComboFailure() cleared session_model_history for the ENTIRE combo
(clearSessionModelHistoryForCombo(comboName), no session_id filter) once
ANY session crossed the 3-consecutive-failure threshold, silently
dropping healthy/live pins for every OTHER session sharing that combo.

Add deleteSessionModelHistory(sessionId, comboName) — a session-scoped
DELETE — and call that from recordComboFailure() instead. Add a
regression test proving cross-session isolation: two sessions pinned on
the same combo, only the failing session's pin clears.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* fix: rebaseline coverage.functions 86.44->86.42 and zizmorFindings 175->176 for quality gate pass

coverage.functions drifted -0.02 from adding failureTracker.ts functions. zizmorFindings +1 from pre-existing upstream workflow drift (PR touches zero workflow files).

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>

* refactor(combo): extract recovery-hint builder into combo/pinRecovery.ts (file-size cap)

combo.ts's net delta for this PR was +108 lines, which would push the frozen
file-size baseline (3387) past its ceiling once #7177 merges first (+64).
Move buildRecoveryHint() (pure terminalReason -> ComboRecoveryHint mapper)
and buildNoUpstreamResponseDiagnostics() (the "no upstream response"
fallback diagnostics literal) out of combo.ts into a new
combo/pinRecovery.ts module. Both are pure, self-contained projections with
no dependency on handleComboChat's local closure state, so this is a code
move with no behavior change — combo.ts keeps only the call-site wiring
(recordComboFailure/clearComboFailureTracking calls stay put, since those
close over local state).

Net result: combo.ts delta drops from +108 to +51 (58 insertions/7
deletions vs the PR's merge-base).

Added tests/unit/combo/pin-recovery.test.ts for direct unit coverage of
both extracted functions. buildNoUpstreamResponseDiagnostics was previously
an inline object literal (no function-coverage surface of its own);
extracting it without a direct test would have nudged coverage.functions
down further after the prior commit's rebaseline.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* chore: drop main-branch drift — scope branch to recovery-hint feature only

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Diego Rodrigues de Sa e Souza <8016841+diegosouzapw@users.noreply.github.com>
Co-authored-by: growab <nekron@icloud.com>
Co-authored-by: KooshaPari <42529354+KooshaPari@users.noreply.github.com>
Co-authored-by: Chirag Singhal <76880977+chirag127@users.noreply.github.com>
Co-authored-by: Chirag Singhal <chirag127@users.noreply.github.com>
Co-authored-by: Ronaldo Davi <ronaldodavi@gmail.com>
Co-authored-by: Andrew Munsell <andrew@wizardapps.net>
Co-authored-by: WITALO ROCHA <witalo_rocha@hotmail.com>
Co-authored-by: Wital <witalorocha216@gmail.com>
Co-authored-by: Aoxiong Yin <i@yinaoxiong.cn>
Co-authored-by: Andrew B. <37745667+AndrianBalanescu@users.noreply.github.com>
Co-authored-by: Andrian B. <andrewbalanesq@gmail.com>
Co-authored-by: Xiangzhe <32761048+xz-dev@users.noreply.github.com>
Co-authored-by: Jon Bailey <297513015+Pitchfork-and-Torch@users.noreply.github.com>
Co-authored-by: Pitchfork-and-Torch <Pitchfork-and-Torch@users.noreply.github.com>
Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com>
Co-authored-by: Samir Abis <me@samirabis.com>
Co-authored-by: lucasjustinudin <34107354+lucasjustinudin@users.noreply.github.com>
Co-authored-by: chy1211 <31048289+chy1211@users.noreply.github.com>
Co-authored-by: ryanngit <74137224+ryanngit@users.noreply.github.com>
Co-authored-by: whale9820 <whale9820@users.noreply.github.com>
Co-authored-by: anhdiepmmk <n08ni.dieppn@gmail.com>
Co-authored-by: Septianata Rizky Pratama <19322988+ianriizky@users.noreply.github.com>
Co-authored-by: NOXX - Commiter <artur1992123@mail.ru>
Co-authored-by: backryun <bakryun0718@proton.me>
Co-authored-by: lunkerchen <labanchen@gmail.com>
Co-authored-by: lunkerchen <lunkerchen@users.noreply.github.com>
Co-authored-by: Ray Doan <raydoan.contact@gmail.com>
Co-authored-by: kfiramar <83420275+kfiramar@users.noreply.github.com>
Co-authored-by: Jan Leon <Jan.gaschler@gmail.com>
Co-authored-by: Someres <168349709+quanturbo@users.noreply.github.com>
Co-authored-by: MikeTuev <ra9ftm@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Imam Wahyu Widodo <120608486+hajilok@users.noreply.github.com>
Co-authored-by: diegosouzapw <diegosouzapw@users.noreply.github.com>
Co-authored-by: AgentKiller45 <jamalzzj45@gmail.com>
Co-authored-by: judy459 <JUDYZHU459@outlook.com>
Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>
Co-authored-by: Markus Hartung <mail@hartmark.se>
Co-authored-by: KooshaPari <koosha@phenotype.io>
Co-authored-by: Jade Guo <jade.gly@gmail.com>
Co-authored-by: Paijo <14921983+oyi77@users.noreply.github.com>
Co-authored-by: oyi77 <oyi77@users.noreply.github.com>
Co-authored-by: Dayna Blackwell <dayna@blackwell-systems.com>
Co-authored-by: backryun <backryun@daonlab.local>
Co-authored-by: brick30llc-ctrl <brick30llc@gmail.com>
Co-authored-by: brick30llc-ctrl <admin@brick30.com>
Co-authored-by: Saren <saren@dumstruck.com>
Co-authored-by: Rafael Dias Zendron <mmmarckos@gmail.com>
Co-authored-by: Xiangzhe <xiangzhedev@gmail.com>
Co-authored-by: Rafael Dias Zendron <rafael.zendron22@gmail.com>
Co-authored-by: KooshaPari <62650152+KooshaPari@users.noreply.github.com>
Co-authored-by: Wibias <37517432+Wibias@users.noreply.github.com>
Co-authored-by: huohua-dev <celentanohertor@gmail.com>
Co-authored-by: huohua-dev <258873123+huohua-dev@users.noreply.github.com>
Co-authored-by: CitrusIce <31264099+CitrusIce@users.noreply.github.com>
Co-authored-by: minisforum <no@mail.com>
Co-authored-by: herjarsa <herjarsa@users.noreply.github.com>
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Co-authored-by: herjarsa <204746071+herjarsa@users.noreply.github.com>
2026-07-18 22:09:51 -03:00

3372 lines
137 KiB
TypeScript

/**
* Shared combo (model combo) handling with fallback support
* Supports: priority, weighted, round-robin, random, least-used, cost-optimized,
* reset-aware, reset-window, strict-random, auto, fill-first, p2c, lkgp,
* context-optimized, context-relay, and fusion strategies
*/
import {
checkFallbackError,
classifyLockoutReason,
CONTEXT_OVERFLOW_PATTERNS,
decayModelFailureCount,
formatRetryAfter,
getModelLockoutInfo,
getRuntimeProviderProfile,
hasPerModelQuota,
isModelLocked,
recordModelLockoutFailure,
recordProviderFailure,
selectLockoutCooldownMs,
} from "./accountFallback.ts";
import {
errorResponse,
unavailableResponse,
errorResponseWithComboDiagnostics,
} from "../utils/error.ts";
import type { ComboDiagnostics } from "../utils/error.ts";
import {
COMBO_FAILURE_THRESHOLD,
clearComboFailureTracking,
recordComboFailure,
} from "./combo/failureTracker.ts";
import { buildNoUpstreamResponseDiagnostics, buildRecoveryHint } from "./combo/pinRecovery.ts";
import { buildTargetTimeoutRunner } from "./combo/targetTimeoutRunner.ts";
import { recordComboRequest, recordComboShadowRequest, getComboMetrics } from "./comboMetrics.ts";
import {
resolveComboConfig,
getDefaultComboConfig,
resolveComboQueueDepth,
} from "./comboConfig.ts";
import {
maybeGenerateHandoff,
maybeGenerateUniversalHandoff,
injectUniversalHandoffBody,
SKIP_UNIVERSAL_HANDOFF_FLAG,
type MessageLike,
} from "./contextHandoff.ts";
import {
recordSessionModelUsage,
getLastSessionModel,
getHandoff,
} from "../../src/lib/db/contextHandoffs.ts";
import { extractSessionAffinityKey } from "@/sse/services/auth";
import { getHiddenModelsByProvider } from "@/models";
import { resolveModelLockoutSettings } from "../../src/lib/resilience/modelLockoutSettings";
import { fetchCodexQuota } from "./codexQuotaFetcher.ts";
import { evaluateQuotaCutoff, getQuotaFetcher, type QuotaInfo } from "./quotaPreflight.ts";
import * as semaphore from "./rateLimitSemaphore.ts";
import { getCircuitBreaker } from "../../src/shared/utils/circuitBreaker";
import { fisherYatesShuffle, getNextFromDeck } from "../../src/shared/utils/shuffleDeck";
import { parseModel } from "./model.ts";
import { createComboContext } from "./combo/context.ts";
import { phaseComboSetup } from "./combo/comboSetup.ts";
import { checkCredentialGate, logCredentialSkip } from "./credentialGate.ts";
import { emit } from "../../src/lib/events/eventBus";
import { notifyWebhookEvent } from "../../src/lib/webhookDispatcher";
import { parseAutoPrefix } from "./autoCombo/autoPrefix.ts";
import { resolveAutoStrategyOrder } from "./combo/resolveAutoStrategy.ts";
import { applyStrategyOrdering } from "./combo/applyStrategyOrdering.ts";
import { handlePipelineCombo, buildPipelineResponse } from "./autoCombo/pipelineRouter.ts";
import { type ProviderCandidate } from "./autoCombo/scoring.ts";
import { estimateTokens } from "./contextManager.ts";
import { getSessionConnection } from "./sessionManager.ts";
import {
applySessionStickiness,
normalizeStickinessMessages,
recordStickyBinding,
clearStickyBinding,
peekStickyConnectionId,
resolveDisableSessionStickiness,
} from "./combo/sessionStickiness.ts";
import { selectQuotaShareTarget } from "./combo/quotaShareStrategy.ts";
import { makeConnectionConcurrencyResolver, lookupPositiveCap } from "./combo/concurrencyCaps.ts";
import { acquireQuotaShareConcurrencySlot } from "./combo/quotaShareConcurrency.ts";
import { orderTargetsByEvalScores } from "./evalRouting.ts";
import type { CompressionMode } from "./compression/types.ts";
import { getProviderConnections } from "../../src/lib/db/providers";
import {
isProviderInCooldown,
recordProviderCooldown,
recordProviderSuccess,
} from "./providerCooldownTracker.ts";
import {
resolveResilienceSettings,
type ResilienceSettings,
} from "../../src/lib/resilience/settings";
import { resolveReasoningBufferedMaxTokens, toPositiveInteger } from "./reasoningTokenBuffer.ts";
import { RESET_WINDOW_NAMES } from "./combo/types.ts";
import type {
ComboLike,
ComboRetryAfter,
ComboErrorBody,
SingleModelTarget,
HandleComboChatOptions,
HandleRoundRobinOptions,
NestedComboMode,
ResolvedComboTarget,
ResolvedComboUnit,
AutoProviderCandidate,
ComboRuntimeStep,
HistoricalLatencyStatsEntry,
} from "./combo/types.ts";
import {
MAX_RR_COUNTERS,
rrCounters,
rrStickyTargets,
weightedStickyTargets,
clampStickyRoundRobinTargetLimit,
clampStickyWeightedTargetLimit,
getStickyRoundRobinStartIndex,
recordStickyRoundRobinSuccess,
getStickyWeightedExecutionKey,
recordStickyWeightedSuccess,
resolveComboStickyRoundRobinLimit,
} from "./combo/rrState.ts";
import {
validateResponseQuality,
releaseQualityClone,
releaseRejectedQualityResponse,
toRetryAfterDisplayValue,
} from "./combo/validateQuality.ts";
import { resolveComboCooldownWaitDecision } from "./combo/comboCooldownRetry.ts";
import {
computeClosestRetryAfter,
waitForCooldownAwareRetry,
} from "../../src/sse/services/cooldownAwareRetry.ts";
import { handleFusionChat, type FusionTuning } from "./fusion.ts";
import { handlePipelineChat, type PipelineStep } from "./pipeline.ts";
import {
TRANSIENT_FOR_SEMAPHORE,
MAX_FALLBACK_WAIT_MS,
MAX_GLOBAL_ATTEMPTS,
isAllAccountsRateLimitedResponse,
clampComboDepth,
shouldSkipForPredictedTtft,
shouldRecordProviderBreakerFailure,
isRequestScopedUpstreamFailure,
shouldSkipConnDisable,
resolveDelayMs,
comboModelNotFoundResponse,
isStreamReadinessFailureErrorBody,
isTokenLimitBreachErrorBody,
toRecordedTarget,
getExhaustedTargetSkipReason,
} from "./combo/comboPredicates.ts";
import { applyComboTargetExhaustion } from "./combo/targetExhaustion.ts";
import { executeRuntimeUnitCombo } from "./combo/runtimeUnits.ts";
import { extractFusionPanelSpec, buildFusionHandleSingleModel } from "./combo/fusionPanel.ts";
import { isRecord } from "./combo/comboData.ts";
import {
expandProviderWildcardsInCombo,
expandProviderWildcardsInCollection,
} from "./combo/providerWildcard.ts";
import { resolveShadowTargets, scheduleShadowRouting } from "./combo/shadowRouting.ts";
import { attemptCompatRejectedFallback } from "./combo/comboCompatFallback.ts";
import { applyContextRequirements } from "./combo/contextRequirements.ts";
import {
filterTargetsByRequestCompatibility,
resolveComboRuntimeUnits,
resolveComboTargets,
resolveWeightedTargets,
resolveWeightedStepGroups,
} from "./combo/comboStructure.ts";
import { getKnownContextOverflow } from "./combo/knownContextOverflow.ts";
import {
QUOTA_SOFT_DEPRIORITIZE_FACTOR,
setCandidateQuotaSoftPenalty,
_registerExecutionCandidates,
_unregisterExecutionCandidates,
applyRequestTagRouting,
scoreAutoTargets,
expandAutoComboCandidatePool,
deriveSpeedTelemetry,
} from "./combo/autoStrategy.ts";
import {
resolveResetWindowConfig,
calculateResetWindowAffinity,
type ResetWindowConfig,
} from "./combo/quotaScoring.ts";
import {
fetchResetAwareQuotaWithCache,
preScreenTargets,
type PreScreenResult,
} from "./combo/quotaStrategies.ts";
import {
buildAutoQuotaThresholds,
resolveQuotaExhaustionCutoffForTarget,
} from "./combo/quotaExhaustionCutoff.ts";
import {
classifyTask,
getConversationCacheKey,
isTaskRoutingStrategy,
reorderByTaskWeight,
} from "./taskAwareRouting.ts";
import { expandTargetsByFingerprints } from "./combo/fingerprintExpansion.ts";
export { RESET_WINDOW_NAMES };
export { QUOTA_SOFT_DEPRIORITIZE_FACTOR, setCandidateQuotaSoftPenalty };
export { scoreAutoTargets, expandAutoComboCandidatePool };
export type { SingleModelTarget, ResolvedComboTarget };
export { validateResponseQuality };
export {
clampComboDepth,
shouldSkipForPredictedTtft,
shouldRecordProviderBreakerFailure,
isRequestScopedUpstreamFailure,
shouldSkipConnDisable,
};
export { resolveShadowTargets, scheduleShadowRouting };
export { preScreenTargets };
export {
resolveComboRuntimeUnits,
resolveComboTargets,
filterTargetsByRequestCompatibility,
getKnownContextOverflow,
};
export {
getComboFromData,
getComboModelsFromData,
resolveNestedComboModels,
resolveNestedComboTargets,
validateComboDAG,
} from "./combo/comboStructure.ts";
/**
* #6692: release a session-stickiness pin the moment its bound connection is
* the one that just failed. applySessionStickiness() only re-checks health on
* the NEXT turn (lazily) — without this, a terminal/quality-rejected
* connection stays pinned until that lazy recheck fires, and a masked
* daily-cap 200-body rejection never trips the lazy recheck's DB-backed
* testStatus gate at all (the connection row itself isn't marked unhealthy).
* Exported for the two failure branches in handleComboChat + handleRoundRobinCombo.
* peekStickyConnectionId guards against clearing an unrelated pin when the
* failing target isn't actually the currently sticky-bound connection.
*/
export function releaseStickyPinOnFailure(
messageHash: string | null | undefined,
failedConnectionId: string | null | undefined
): void {
if (!messageHash || !failedConnectionId) return;
if (peekStickyConnectionId(messageHash) !== failedConnectionId) return;
clearStickyBinding(messageHash);
}
const DEFAULT_MODEL_P95_MS: Record<string, number> = {
"grok-4-fast-non-reasoning": 1143,
"grok-4-1-fast-non-reasoning": 1244,
"gemini-2.5-flash": 1238,
"kimi-k2.5": 1646,
"gpt-4o-mini": 2764,
"claude-sonnet-4.6": 4000,
"claude-opus-4.6": 6000,
"deepseek-chat": 2000,
};
const MIN_HISTORY_SAMPLES = 10;
const OUTPUT_TOKEN_RATIO = 0.4;
function normalizeNestedComboMode(value: unknown): NestedComboMode {
return value === "execute" ? "execute" : "flatten";
}
function calculateTargetContextAffinity(
target: ResolvedComboTarget,
sessionId: string | null | undefined
): number {
const sessionConnectionId = getSessionConnection(sessionId || null);
if (!sessionConnectionId) return 0.5;
if (target.connectionId === sessionConnectionId) return 1;
if (!target.connectionId) return 0.5;
return 0.1;
}
function getBootstrapLatencyMs(modelId: string): number {
const normalized = String(modelId || "").toLowerCase();
return DEFAULT_MODEL_P95_MS[normalized] ?? 1500;
}
function clampPercent(value: number): number {
if (!Number.isFinite(value)) return 100;
return Math.max(0, Math.min(100, value));
}
function quotaRemainingPercentFromQuota(quota: unknown): number {
if (!quota || typeof quota !== "object") return 100;
const record = quota as Record<string, unknown>;
if (record.limitReached === true) return 0;
const windows = record.windows;
if (windows && typeof windows === "object" && !Array.isArray(windows)) {
let minRemaining: number | null = null;
for (const windowInfo of Object.values(windows as Record<string, unknown>)) {
if (!windowInfo || typeof windowInfo !== "object") continue;
const percentUsed = Number((windowInfo as Record<string, unknown>).percentUsed);
if (!Number.isFinite(percentUsed)) continue;
const remaining = clampPercent((1 - percentUsed) * 100);
minRemaining = minRemaining === null ? remaining : Math.min(minRemaining, remaining);
}
if (minRemaining !== null) return minRemaining;
}
const percentUsed = Number(record.percentUsed);
if (Number.isFinite(percentUsed)) return clampPercent((1 - percentUsed) * 100);
return 100;
}
const QUOTA_BLOCKING_CONNECTION_STATUSES = new Set([
"banned",
"credits_exhausted",
"deactivated",
"expired",
"rate_limited",
]);
function normalizeConnectionStatus(value: unknown): string {
return typeof value === "string" ? value.trim().toLowerCase() : "";
}
function hasFutureRateLimitUntil(value: unknown): boolean {
if (value == null || value === "") return false;
const time = new Date(String(value)).getTime();
return Number.isFinite(time) && time > Date.now();
}
export function getConnectionStatusQuotaCutoffReason(
connection: Record<string, unknown> | undefined
): string | undefined {
if (!connection) return undefined;
const status = normalizeConnectionStatus(connection.testStatus);
if (QUOTA_BLOCKING_CONNECTION_STATUSES.has(status)) return status;
if (status === "unavailable" && hasFutureRateLimitUntil(connection.rateLimitedUntil)) {
return "rate_limited";
}
return undefined;
}
export async function buildAutoCandidates(
targets: ResolvedComboTarget[],
comboName: string,
sessionId: string | null | undefined = null,
resetWindowConfig: ResetWindowConfig = resolveResetWindowConfig(null),
resilienceSettings: ResilienceSettings | null = null
): Promise<AutoProviderCandidate[]> {
const hiddenModelsMap = getHiddenModelsByProvider();
const metrics = getComboMetrics(comboName);
// Opt-in hard quota cutoff (default OFF). When disabled, candidates are never
// dropped for low quota here — the soft quota penalty + connection cooldown still
// apply, so auto-routing behavior is unchanged.
const quotaCutoffEnabled =
(resilienceSettings ?? resolveResilienceSettings(null))?.quotaPreflight?.enabled === true;
const { getPricingForModel } = await import("../../src/lib/localDb");
const quotaPromises = new Map<string, Promise<unknown>>();
let historicalLatencyStats: Record<string, HistoricalLatencyStatsEntry> = {};
try {
const { getModelLatencyStats } = await import("../../src/lib/usageDb");
historicalLatencyStats = await getModelLatencyStats({
windowHours: 24,
minSamples: 3,
maxRows: 10000,
});
} catch {
// keep empty stats — auto-combo will use runtime + bootstrap signals
}
const uniqueProviders = Array.from(
new Set(
targets.map((target) => target.provider || parseModel(target.modelStr).provider || "unknown")
)
);
const connectionPoolCounts = new Map<string, number>();
const connectionsByProvider = new Map<string, Array<Record<string, unknown>>>();
const connectionById = new Map<string, Record<string, unknown>>();
await Promise.all(
uniqueProviders.map(async (provider) => {
try {
const connections = await getProviderConnections({ provider, isActive: true });
const active = Array.isArray(connections) ? connections : [];
connectionPoolCounts.set(provider, active.length);
connectionsByProvider.set(provider, active);
for (const connection of active) {
if (connection && typeof connection === "object" && typeof connection.id === "string") {
connectionById.set(connection.id, connection as Record<string, unknown>);
}
}
} catch {
connectionPoolCounts.set(provider, 0);
connectionsByProvider.set(provider, []);
}
})
);
const expandedTargets: ResolvedComboTarget[] = [];
for (const target of targets) {
const provider = target.provider || parseModel(target.modelStr).provider || "unknown";
const providerConnections = connectionsByProvider.get(provider) || [];
if (target.connectionId) {
expandedTargets.push(target);
continue;
}
const connectionIds = providerConnections
.map((c) => (c && typeof c === "object" && typeof c.id === "string" ? c.id : null))
.filter((id): id is string => id !== null);
const allowedConnectionIds = Array.isArray(target.allowedConnectionIds)
? new Set(
target.allowedConnectionIds.filter(
(connectionId): connectionId is string =>
typeof connectionId === "string" && connectionId.trim().length > 0
)
)
: null;
const scopedConnectionIds = allowedConnectionIds
? connectionIds.filter((connectionId) => allowedConnectionIds.has(connectionId))
: connectionIds;
if (scopedConnectionIds.length === 0) {
expandedTargets.push(target);
continue;
}
for (const connectionId of scopedConnectionIds) {
expandedTargets.push({
...target,
connectionId,
executionKey: `${target.executionKey}@${connectionId}`,
});
}
}
// #5521: Expand fingerprint-based providers (mimocode, mcode, opencode) so each
// fingerprint gets its own combo slot instead of being bundled into one connection.
const fingerprintExpandedTargets = expandTargetsByFingerprints(
expandedTargets,
connectionById,
(t) => {
const parsed = parseModel(t.modelStr);
return t.provider || parsed.provider || parsed.providerAlias || "unknown";
}
);
const candidates = await Promise.all(
fingerprintExpandedTargets.map(async (target) => {
const modelStr = target.modelStr;
const parsed = parseModel(modelStr);
const provider = target.provider || parsed.provider || parsed.providerAlias || "unknown";
const model = parsed.model || modelStr;
const historicalKey = `${provider}/${model}`;
const historicalModelMetric = historicalLatencyStats[historicalKey] || null;
const historicalTotal = Number(historicalModelMetric?.totalRequests);
const hasHistoricalSignal =
Number.isFinite(historicalTotal) && historicalTotal >= MIN_HISTORY_SAMPLES;
let costPer1MTokens = 1;
try {
const pricing = await getPricingForModel(provider, model);
const inputPrice = Number(pricing?.input);
const outputPrice = Number(pricing?.output);
if (Number.isFinite(inputPrice) && inputPrice >= 0) {
if (Number.isFinite(outputPrice) && outputPrice >= 0) {
costPer1MTokens =
inputPrice * (1 - OUTPUT_TOKEN_RATIO) + outputPrice * OUTPUT_TOKEN_RATIO;
} else {
costPer1MTokens = inputPrice;
}
}
} catch {
// keep default cost
}
const modelMetric = metrics?.byModel?.[modelStr] || null;
const avgLatency = Number(modelMetric?.avgLatencyMs);
const successRate = Number(modelMetric?.successRate);
const historicalP95Latency = Number(historicalModelMetric?.p95LatencyMs);
const historicalStdDev = Number(historicalModelMetric?.latencyStdDev);
const historicalSuccessRate = Number(historicalModelMetric?.successRate); // 0..1
const p95LatencyMs = hasHistoricalSignal
? Number.isFinite(historicalP95Latency) && historicalP95Latency > 0
? historicalP95Latency
: getBootstrapLatencyMs(model)
: Number.isFinite(avgLatency) && avgLatency > 0
? avgLatency
: getBootstrapLatencyMs(model);
const errorRate = hasHistoricalSignal
? Number.isFinite(historicalSuccessRate) &&
historicalSuccessRate >= 0 &&
historicalSuccessRate <= 1
? 1 - historicalSuccessRate
: 0.05
: Number.isFinite(successRate) && successRate >= 0 && successRate <= 100
? 1 - successRate / 100
: 0.05;
const latencyStdDev =
hasHistoricalSignal && Number.isFinite(historicalStdDev) && historicalStdDev > 0
? Math.max(10, historicalStdDev)
: Math.max(10, p95LatencyMs * 0.1);
// #6875: surface TTFT/E2E-latency/tokens-per-second onto the candidate so the
// existing speed-ranking factor (#6011, speedRanking.ts/routerStrategy.ts) picks
// up real telemetry instead of falling back to the pool median. Additive only —
// no scoring weights change here.
const speedTelemetry = hasHistoricalSignal
? deriveSpeedTelemetry(historicalModelMetric)
: undefined;
const breakerStateRaw = getCircuitBreaker(provider)?.getStatus?.()?.state;
const circuitBreakerState: ProviderCandidate["circuitBreakerState"] =
breakerStateRaw === "OPEN" || breakerStateRaw === "HALF_OPEN" ? breakerStateRaw : "CLOSED";
const contextAffinity = calculateTargetContextAffinity(target, sessionId);
let resetWindowAffinity = 0.5;
let quotaRemaining = 100;
let quotaCutoffBlocked = false;
let quotaCutoffReason: string | undefined;
const fetcher = getQuotaFetcher(provider);
const connection = target.connectionId ? connectionById.get(target.connectionId) : undefined;
// Gate the terminal-status cutoff behind the same opt-in as the quota-percent
// cutoff (#4483): when quota cutoff is disabled, a connection in a terminal
// testStatus must still fall through to normal connection-cooldown / model-lockout
// handling instead of being hard-blocked here (which would surface a misleading
// "below quota cutoff" 429 when every candidate is transiently unavailable).
// The connection's terminal/transient status (credits_exhausted / rate_limited /
// banned / expired / future-dated unavailable) is classified unconditionally.
const connectionStatusReason = getConnectionStatusQuotaCutoffReason(connection);
const statusCutoffReason = quotaCutoffEnabled ? connectionStatusReason : undefined;
// #4540: when the HARD cutoff is OFF (default), a status-flagged connection is NOT
// hard-blocked (that would surface a misleading "below quota cutoff" 429), but it
// also must not score identically to a healthy provider. A no-fetcher exhausted
// connection keeps quotaRemaining=100, so we tag a SOFT penalty applied at scoring
// time (scoreAutoTargets → STATUS_SOFT_DEPRIORITIZE_FACTOR) instead.
let statusPenalty = false;
let statusPenaltyReason: string | undefined;
if (statusCutoffReason) {
quotaCutoffBlocked = true;
quotaCutoffReason = statusCutoffReason;
quotaRemaining = 0;
} else if (connectionStatusReason) {
statusPenalty = true;
statusPenaltyReason = connectionStatusReason;
}
if (fetcher && target.connectionId) {
const quotaKey = `${provider}:${target.connectionId}`;
if (!quotaPromises.has(quotaKey)) {
quotaPromises.set(
quotaKey,
fetchResetAwareQuotaWithCache({
provider,
connectionId: target.connectionId,
connection,
fetcher,
config: resetWindowConfig,
log: {},
comboName,
})
);
}
const quota = await quotaPromises.get(quotaKey)!;
resetWindowAffinity = calculateResetWindowAffinity(quota, resetWindowConfig);
if (!quotaCutoffBlocked) {
quotaRemaining = quotaRemainingPercentFromQuota(quota);
}
if (!quotaCutoffBlocked && quotaCutoffEnabled) {
const cutoffDecision = evaluateQuotaCutoff(
quota as QuotaInfo | null,
buildAutoQuotaThresholds(provider, connection, resilienceSettings)
);
if (!cutoffDecision.proceed) {
quotaCutoffBlocked = true;
quotaCutoffReason = cutoffDecision.reason || "quota_exhausted";
}
}
}
return {
stepId: target.stepId,
executionKey: target.executionKey,
modelStr,
provider,
model,
quotaRemaining,
quotaTotal: 100,
circuitBreakerState,
costPer1MTokens,
p95LatencyMs,
latencyStdDev,
errorRate,
...speedTelemetry,
accountTier: "standard" as const,
quotaResetIntervalSecs: 86400,
contextAffinity,
resetWindowAffinity,
quotaCutoffBlocked,
quotaCutoffReason,
statusPenalty,
statusPenaltyReason,
connectionPoolSize: connectionPoolCounts.get(provider) ?? 1,
connectionId: target.connectionId ?? undefined,
};
})
);
// Filter out candidates whose model is hidden by the user in the dashboard
return candidates.filter((c) => {
const hiddenModels = hiddenModelsMap.get(c.provider);
return !hiddenModels?.has(c.model);
});
}
const TERMINAL_PIN_STATUSES = new Set(["credits_exhausted", "banned", "expired"]);
/**
* Pure decision: should a context-cache pin be DROPPED because its provider has
* DURABLY fallen? A ccp pin keeps the prompt cache warm by bypassing the combo
* strategy — but if the pinned provider is dead (credits exhausted / banned /
* expired, circuit-open, repeated failures, or a long rate-limit) honoring the
* pin pounds a dead account forever with no failover (laila throttle + credits
* incidents, 2026-06-22). A brief transient cooldown is tolerated (pin kept) so
* an unstable provider does not churn the pin every turn. Connection-level
* `backoffLevel` already resets on success, so `backoffLevel >= K` ≈ K
* consecutive failures — no per-session counter needed.
*
* Returns true ⇒ drop the pin and use the strategy. Pure + unit-testable.
*/
export function pinIsDurablyUnhealthy(
circuitState: string | undefined,
connections: Array<{
testStatus?: string | null;
backoffLevel?: number | null;
rateLimitedUntil?: string | null;
}>,
now: number,
opts: { backoffLevel?: number; graceMs?: number } = {}
): boolean {
if (circuitState === "OPEN") return true;
if (!Array.isArray(connections) || connections.length === 0) return true;
const backoffThreshold = opts.backoffLevel ?? Number(process.env.PIN_DROP_BACKOFF_LEVEL || "2");
const graceMs = opts.graceMs ?? Number(process.env.PIN_DROP_GRACE_MS || "20000");
// The pin survives as long as AT LEAST ONE connection is healthy or only
// briefly cooling down — failover only when every connection is durably down.
const anyUsable = connections.some((c) => {
const status = typeof c.testStatus === "string" ? c.testStatus : "";
if (TERMINAL_PIN_STATUSES.has(status)) return false;
if (Number(c.backoffLevel ?? 0) >= backoffThreshold) return false;
const rl = c.rateLimitedUntil ? new Date(String(c.rateLimitedUntil)).getTime() : 0;
if (Number.isFinite(rl) && rl - now > graceMs) return false;
return true;
});
return !anyUsable;
}
/**
* Async wrapper: resolve the pinned model's provider, read its circuit state and
* active connections, and decide via {@link pinIsDurablyUnhealthy}. Fail-open
* (return false) on any error so a lookup bug never drops a healthy pin.
*/
async function isPinnedModelDurablyUnhealthy(pinnedModel: string): Promise<boolean> {
try {
const provider = parseModel(pinnedModel).provider;
if (!provider) return false;
const circuitState = getCircuitBreaker(provider)?.getStatus?.()?.state;
const connections = (await getProviderConnections({
provider,
isActive: true,
})) as Array<{
testStatus?: string | null;
backoffLevel?: number | null;
rateLimitedUntil?: string | null;
}>;
return pinIsDurablyUnhealthy(circuitState, connections || [], Date.now());
} catch {
return false;
}
}
/**
* Handle combo chat with fallback.
* @param {Object} options
* @param {Object} options.body - Request body
* @param {Object} options.combo - Full combo object { name, models, strategy, config }
* @param {Function} options.handleSingleModel - Function: (body, modelStr) => Promise<Response>
* @param {Function} [options.isModelAvailable] - Optional pre-check: (modelStr) => Promise<boolean>
* @param {Object} options.log - Logger object
* @returns {Promise<Response>}
*/
// #2101 guard helpers: a 400 caused by context overflow or parameter validation
// is NOT body-specific — different combo targets have different context windows /
// output limits, so the request should fall through to the next target instead of
// being short-circuited. Exported as pure predicates so the guard is unit-testable.
/** @param {string} errorText */
export function isContextOverflow400(errorText) {
return (
/\bcontext.*(?:length_exceeded|too long|overflow|exceeded|window|limit)\b/i.test(errorText) ||
/exceeds.*context/i.test(errorText) ||
/your input exceeds/i.test(errorText) ||
// Reuse accountFallback.ts's CONTEXT_OVERFLOW_PATTERNS (single source of truth)
// so wording like Kimi's "exceeded model token limit" — which never says the
// literal word "context" — is still recognized as an overflow/fallback-worthy
// 400 instead of halting the whole combo (issue #6637).
CONTEXT_OVERFLOW_PATTERNS.some((p) => p.test(errorText))
);
}
/** @param {string} errorText */
export function isParamValidation400(errorText) {
return (
/\bmax_tokens\b.*(?:illegal|must|range|invalid)/i.test(errorText) ||
/\bparameter is illegal\b/i.test(errorText) ||
/\bis illegal.*range\b/i.test(errorText)
);
}
/** @param {object} options */
export async function handleComboChat({
body,
combo,
handleSingleModel,
isModelAvailable,
log,
settings,
allCombos,
relayOptions,
signal,
apiKeyAllowedConnections = null,
nesting = null,
}: HandleComboChatOptions): Promise<Response> {
const comboCtx = createComboContext({ body, combo, settings, relayOptions, log });
const {
strategy,
relayConfig,
resilienceSettings,
universalHandoffConfig,
effectiveSessionId,
pinnedModel,
clientRequestedStream,
config,
comboTargetTimeoutMs,
reasoningTokenBufferEnabled,
} = phaseComboSetup(comboCtx);
body = comboCtx.body;
const handleSingleModelWithTimeout = buildTargetTimeoutRunner({
handleSingleModel,
comboTargetTimeoutMs,
log,
});
// Route to pinned model if context caching specifies one (Fix #679)
if (pinnedModel) {
// The pin is read from session_model_history (a PRIOR turn) and may name a
// model that has since been removed from this combo, or a provider whose
// credentials are gone. Without this guard a stale pin bypasses the strategy
// and routes to a dead model forever — incident 2026-06-21: cli-claude-heavy
// pinned to a deepseek connection with no active credentials → instant fail,
// never falling through to the live targets; and combos re-pointed Opus→Sonnet
// kept serving the old model. Validate the pin is still reachable in THIS
// combo's resolved targets (refs flattened) before honoring it. Only validate
// when allCombos is authoritative (non-empty) so we can resolve combo-refs;
// the auto-combo redirect path passes an empty list and keeps prior behavior.
const haveFullCombos = Array.isArray(allCombos) ? allCombos.length > 0 : !!allCombos;
const pinInCombo =
!haveFullCombos ||
resolveComboTargets(combo, allCombos, clampComboDepth(config.maxComboDepth)).some(
(t) => t.modelStr === pinnedModel
);
// Honor the pin only if it is still a combo target AND its provider is not
// DURABLY down. Without the health gate a pin keeps routing a session to a
// dead/credits-exhausted/throttled account forever (strategy bypassed, no
// failover) — incident 2026-06-22: laila stuck on a throttled claude account
// and credits_exhausted accounts never failing over. A transient cooldown is
// tolerated (pin kept) so an unstable provider does not churn the pin.
const pinDurablyDown = pinInCombo ? await isPinnedModelDurablyUnhealthy(pinnedModel) : false;
if (pinInCombo && !pinDurablyDown) {
log.info(
"COMBO",
`Bypassing strategy — routing directly to pinned context model: ${pinnedModel}`
);
let pinnedResult: Response | null = null;
try {
pinnedResult = await handleSingleModelWithTimeout(body, pinnedModel, {
modelPinned: true,
} as SingleModelTarget);
} catch (pinErr) {
log.warn(
"COMBO",
`Pinned model ${pinnedModel} threw error: ${pinErr instanceof Error ? pinErr.message : String(pinErr)}, falling through to combo retry/fallback`
);
}
if (pinnedResult) {
if (pinnedResult.ok) {
let pinnedClone: Response;
try {
pinnedClone = pinnedResult.clone();
} catch {
pinnedClone = pinnedResult;
}
const pinnedQuality = await validateResponseQuality(
pinnedClone,
clientRequestedStream,
log,
config.responseValidation
);
releaseQualityClone(pinnedClone, pinnedResult, pinnedQuality);
if (pinnedQuality.valid) return pinnedResult;
releaseRejectedQualityResponse(pinnedClone, pinnedResult);
log.warn(
"COMBO",
`Pinned model ${pinnedModel} returned 200 but failed quality check: ${pinnedQuality.reason}, falling through to combo retry/fallback`
);
} else {
const pinnedStatus = pinnedResult.status || 500;
if (![408, 429, 500, 502, 503, 504].includes(pinnedStatus)) {
return pinnedResult;
}
log.warn(
"COMBO",
`Pinned model ${pinnedModel} failed (${pinnedStatus}), falling through to combo retry/fallback`
);
}
}
// Fall through to the target iteration loop below — retries and sibling
// models will be tried via the normal combo machinery.
}
log.warn(
"COMBO",
pinInCombo
? `Context-cache pin "${pinnedModel}" provider durably unhealthy — dropping pin, using strategy`
: `Stale context-cache pin "${pinnedModel}" not in combo "${combo.name}" targets — dropping pin, using strategy`
);
// Fall through to the normal target iteration loop below — the pin is
// dropped, so the combo strategy picks the best available target.
}
// Fusion strategy: parallel panel + judge synthesis. Handled in a separate module
// because it neither iterates targets in order nor needs the failover/retry/credential
// gate machinery that follows — it fans out, then synthesizes once.
const cfg = config as Record<string, unknown>;
const judgeModel = typeof cfg.judgeModel === "string" ? cfg.judgeModel : undefined;
const fusionTuning =
cfg.fusionTuning && typeof cfg.fusionTuning === "object"
? (cfg.fusionTuning as FusionTuning)
: undefined;
if (strategy !== "fusion" && (judgeModel || fusionTuning)) {
log.warn(
"COMBO",
`Combo "${combo.name}" sets config.judgeModel/fusionTuning but strategy is "${strategy}" — these fields are only consumed by the fusion strategy and will be ignored (#6455)`
);
}
if (strategy === "fusion") {
const { panel: fusionModels, comboRefUnits } = extractFusionPanelSpec(
combo.models || [],
combo.name,
allCombos
);
// Untyped like the existing `nestingContext` further down — `nesting` is
// already `ComboNestingContext | null` per HandleComboChatOptions, no new
// import needed.
const fusionNesting = nesting || {
depth: 0,
maxDepth: clampComboDepth(config.maxComboDepth),
visitedComboNames: [combo.name],
rootComboName: combo.name,
attemptBudget: { count: 0, limit: MAX_GLOBAL_ATTEMPTS },
};
const fusionHandleSingleModel =
comboRefUnits.size > 0
? buildFusionHandleSingleModel({
handleSingleModel: handleSingleModelWithTimeout,
comboRefUnits,
allCombos,
nesting: fusionNesting,
baseOptions: {
body,
combo,
handleSingleModel,
isModelAvailable,
log,
settings,
allCombos,
relayOptions,
signal,
apiKeyAllowedConnections,
},
runCombo: handleComboChat,
})
: handleSingleModelWithTimeout;
return handleFusionChat({
body,
models: fusionModels,
handleSingleModel: fusionHandleSingleModel,
log,
comboName: combo.name,
judgeModel,
tuning: fusionTuning,
});
}
// Pipeline strategy: sequential chain — each step's output feeds the next step's
// input, only the final step's response is returned. Handled in a separate module
// because it neither iterates targets as fallbacks nor needs the failover/retry
// machinery below — it runs targets in order, threading output → input. The step
// list is `combo.models` (in order); an optional per-step `prompt` is read off the
// target object (comboModelStepInputSchema.prompt).
if (strategy === "pipeline") {
const pipelineSteps = (combo.models || [])
.map((m): PipelineStep | null => {
if (typeof m === "string") return { model: m };
if (m && typeof m === "object") {
const obj = m as Record<string, unknown>;
if (typeof obj.model === "string") {
return {
model: obj.model,
prompt: typeof obj.prompt === "string" ? obj.prompt : undefined,
};
}
}
return null;
})
.filter((s): s is PipelineStep => Boolean(s));
return handlePipelineChat({
body,
steps: pipelineSteps,
handleSingleModel: handleSingleModelWithTimeout,
log,
comboName: combo.name,
});
}
const nestingContext = nesting || {
depth: 0,
maxDepth: clampComboDepth(config.maxComboDepth),
visitedComboNames: [combo.name],
rootComboName: combo.name,
attemptBudget: { count: 0, limit: MAX_GLOBAL_ATTEMPTS },
};
const nestedComboMode = normalizeNestedComboMode(config.nestedComboMode);
const executeModeUnits =
nestedComboMode === "execute" && allCombos
? resolveComboRuntimeUnits(combo, allCombos, "execute", nestingContext.maxDepth)
: [];
const hasExecutableComboRef = executeModeUnits.some((unit) => unit.kind === "combo-ref");
const simpleExecuteStrategies = new Set([
"priority",
"round-robin",
"random",
"strict-random",
"weighted",
"fill-first",
]);
if (hasExecutableComboRef && simpleExecuteStrategies.has(strategy)) {
let runtimeUnits = executeModeUnits;
let unitExecutionStrategy = strategy;
if (strategy === "weighted") {
const stickyLimit = clampStickyWeightedTargetLimit(
(config as Record<string, unknown>).stickyWeightedLimit
);
const stickyKey = getStickyWeightedExecutionKey(combo.name, stickyLimit);
const stickyUnit = stickyKey
? runtimeUnits.find((unit) => unit.executionKey === stickyKey)
: null;
if (stickyUnit) {
runtimeUnits = [
stickyUnit,
...runtimeUnits.filter((unit) => unit.executionKey !== stickyUnit.executionKey),
];
unitExecutionStrategy = "priority";
}
}
if (strategy === "random") runtimeUnits = fisherYatesShuffle([...runtimeUnits]);
if (strategy === "strict-random") {
const key = await getNextFromDeck(
`combo:${combo.name}`,
runtimeUnits.map((unit) => unit.executionKey)
);
const selected = runtimeUnits.find((unit) => unit.executionKey === key) || runtimeUnits[0];
runtimeUnits = [
selected,
...runtimeUnits.filter((unit) => unit.executionKey !== selected.executionKey),
];
}
let runtimeStickyLimit: number | null = null;
let runtimeStickyTargets: ResolvedComboUnit[] = runtimeUnits;
if (strategy === "round-robin") {
const perComboStickyLimit = (config as Record<string, unknown>).stickyRoundRobinLimit;
runtimeStickyLimit = resolveComboStickyRoundRobinLimit(
perComboStickyLimit,
settings as Record<string, unknown> | null
);
const { startIndex, counter } = getStickyRoundRobinStartIndex(
combo.name,
runtimeUnits,
runtimeStickyLimit
);
if (runtimeStickyLimit <= 1) rrCounters.set(combo.name, counter + 1);
runtimeUnits = runtimeUnits.map(
(_, offset) => runtimeUnits[(startIndex + offset) % runtimeUnits.length]
);
runtimeStickyTargets = executeModeUnits;
}
const execution = await executeRuntimeUnitCombo({
body,
combo,
strategy: unitExecutionStrategy,
effectiveComboStrategy: strategy,
units: runtimeUnits,
handleSingleModel: handleSingleModelWithTimeout,
isModelAvailable,
log,
config,
settings,
allCombos,
signal,
nesting: nestingContext,
baseOptions: {
body,
combo,
handleSingleModel,
isModelAvailable,
log,
settings,
allCombos,
relayOptions,
signal,
apiKeyAllowedConnections,
},
runCombo: handleComboChat,
});
if (strategy === "weighted" && execution.response.ok && execution.unit) {
const stickyLimit = clampStickyWeightedTargetLimit(
(config as Record<string, unknown>).stickyWeightedLimit
);
if (stickyLimit > 1)
recordStickyWeightedSuccess(combo.name, execution.unit.executionKey, stickyLimit);
}
if (
strategy === "round-robin" &&
execution.response.ok &&
execution.unit &&
runtimeStickyLimit &&
runtimeStickyLimit > 1
) {
recordStickyRoundRobinSuccess(
combo.name,
execution.unit,
runtimeStickyLimit,
runtimeStickyTargets
);
}
return execution.response;
}
// Route to round-robin handler if strategy matches
if (strategy === "round-robin") {
return handleRoundRobinCombo({
body,
combo,
handleSingleModel: handleSingleModelWithTimeout,
isModelAvailable,
log,
settings,
allCombos,
signal,
});
}
const maxRetries = config.maxRetries ?? 1;
const retryDelayMs = resolveDelayMs(config.retryDelayMs, 2000);
const fallbackDelayMs = resolveDelayMs(config.fallbackDelayMs, 0);
const maxSetRetries = config.maxSetRetries ?? 0;
const setRetryDelayMs = resolveDelayMs(config.setRetryDelayMs, 2000);
const isTargetSelectableForWeighted = async (target: ResolvedComboTarget): Promise<boolean> => {
const rawModel = parseModel(target.modelStr).model || target.modelStr;
if (target.provider && getCircuitBreaker(target.provider).getStatus().state === "OPEN")
return false;
if (
resilienceSettings.providerCooldown.enabled &&
Boolean(target.provider && target.provider !== "unknown") &&
isProviderInCooldown(target.provider, target.connectionId ?? undefined, resilienceSettings)
) {
return false;
}
if (
target.provider &&
rawModel &&
isModelLocked(target.provider, target.connectionId || "", rawModel)
) {
return false;
}
return isModelAvailable ? await isModelAvailable(target.modelStr, target) : true;
};
// #2562: Expand provider-wildcard steps (e.g. `fta/*`, `openai/gpt-4*`) into
// concrete model entries sourced from the live synced-models catalog + registry.
// Must run before any step-group / target resolution so that wildcard-originated
// steps are treated identically to hand-authored entries by all downstream logic
// (including the sticky-weighted eligibility pass below).
const expandedCombo = await expandProviderWildcardsInCombo(combo);
const expandedAllCombos = allCombos
? Array.isArray(allCombos)
? await expandProviderWildcardsInCollection(allCombos as ComboLike[])
: {
...allCombos,
combos: await expandProviderWildcardsInCollection(
((allCombos as { combos?: ComboLike[] }).combos || []) as ComboLike[]
),
}
: allCombos;
const stickyWeightedLimit = clampStickyWeightedTargetLimit(
(config as Record<string, unknown>).stickyWeightedLimit
);
if (
strategy === "weighted" &&
!weightedStickyTargets.has(combo.name) &&
weightedStickyTargets.size >= MAX_RR_COUNTERS
) {
const oldest = weightedStickyTargets.keys().next().value;
if (oldest !== undefined) weightedStickyTargets.delete(oldest);
}
let stepGroups: Array<{ step: ComboRuntimeStep; targets: ResolvedComboTarget[] }> | undefined;
const weightedEligibleKeys = new Set<string>();
if (strategy === "weighted") {
stepGroups = resolveWeightedStepGroups(expandedCombo, expandedAllCombos);
for (const group of stepGroups) {
const availability = await Promise.all(group.targets.map(isTargetSelectableForWeighted));
if (availability.some(Boolean)) weightedEligibleKeys.add(group.step.executionKey);
}
}
const rawStickyWeightedKey =
strategy === "weighted" ? getStickyWeightedExecutionKey(combo.name, stickyWeightedLimit) : null;
const stickyWeightedKey =
rawStickyWeightedKey && weightedEligibleKeys.has(rawStickyWeightedKey)
? rawStickyWeightedKey
: null;
if (strategy !== "weighted" || stickyWeightedLimit <= 1) {
weightedStickyTargets.delete(combo.name);
} else if (rawStickyWeightedKey && !stickyWeightedKey) {
weightedStickyTargets.delete(combo.name);
}
const weightedResolution =
strategy === "weighted"
? resolveWeightedTargets(
expandedCombo,
expandedAllCombos,
stickyWeightedKey,
weightedEligibleKeys,
stepGroups
)
: null;
const getWeightedStepKeyForTarget = (target: ResolvedComboTarget): string | null => {
if (!weightedResolution?.orderedSteps) return null;
const step = weightedResolution.orderedSteps.find(
(entry) =>
target.executionKey === entry.executionKey ||
target.executionKey.startsWith(entry.executionKey + ">")
);
return step?.executionKey || null;
};
let orderedTargets =
strategy === "weighted"
? weightedResolution?.orderedTargets || []
: resolveComboTargets(
expandedCombo,
expandedAllCombos,
clampComboDepth(config.maxComboDepth)
);
orderedTargets = await applyRequestTagRouting(orderedTargets, body, log);
const knownContextOverflow = getKnownContextOverflow(orderedTargets, body);
if (knownContextOverflow) {
const { requiredContextTokens, maxKnownContextTokens } = knownContextOverflow;
log.warn(
"COMBO",
`Request context exceeds every known target limit (${requiredContextTokens} > ${maxKnownContextTokens} tokens)`
);
return errorResponseWithComboDiagnostics(
400,
`Request requires approximately ${requiredContextTokens} tokens, but the largest known context limit in this combo is ${maxKnownContextTokens} tokens. Reduce or compact the request context.`,
{
poolSize: orderedTargets.length,
attempted: 0,
excluded: orderedTargets.map((target) => ({
provider: target.provider,
model: target.modelStr,
reason: "context_window",
})),
attemptOrder: [],
terminalReason: "context_length_exceeded",
},
{ code: "context_length_exceeded", type: "invalid_request_error" }
);
}
if (strategy === "weighted") {
log.info(
"COMBO",
`Weighted selection${stickyWeightedKey ? " (sticky)" : ""}${allCombos ? " with nested resolution" : ""}: ${orderedTargets.length} total targets`
);
} else if (allCombos) {
log.info("COMBO", `${strategy} with nested resolution: ${orderedTargets.length} total targets`);
}
// Pipeline dispatch: route smart/pipeline-enabled combos through the multi-stage pipeline
if (strategy === "auto") {
const autoParsed = parseAutoPrefix(combo.name);
const autoVariant = autoParsed.valid ? autoParsed.variant : undefined;
if (autoVariant === "smart" || config.pipeline_enabled) {
try {
const pipelineRaw = await handlePipelineCombo({
body,
combo,
handleChatCore: handleSingleModelWithTimeout,
log: {
info: log.info,
warn: log.warn,
error: log.error ?? log.warn,
},
settings: settings ?? {},
signal: signal ?? undefined,
});
// handlePipelineCombo resolves to a PipelineResult (buffered text) or,
// in the streaming-final-stage case, a Response. Callers downstream
// (chat.ts → withSessionHeader) require a Response, so adapt the
// PipelineResult here instead of leaking the raw object.
return pipelineRaw instanceof Response
? pipelineRaw
: buildPipelineResponse(pipelineRaw, body);
} catch (pipelineErr) {
const pipelineMsg = pipelineErr instanceof Error ? pipelineErr.message : "";
if (pipelineMsg === "PIPELINE_DISABLED") {
log.info("COMBO", "Pipeline disabled, falling through to standard auto routing");
} else if (pipelineMsg === "PIPELINE_TOKEN_THRESHOLD") {
log.info(
"COMBO",
"Pipeline skipped (prompt below token threshold), falling through to standard auto routing"
);
} else {
log.warn("COMBO", "Pipeline dispatch failed, falling through to standard auto routing", {
err: pipelineErr,
});
}
}
}
}
// #4945 regression guard: when an "auto" combo uses an EXPLICIT router
// (routingStrategy lkgp/cost/etc, not the default "rules" scorer), that router
// pins orderedTargets[0]. The task-aware reordering below must then refine only
// the fallback order, never override the router's primary choice.
let autoUsedExplicitRouter = false;
if (strategy === "auto") {
const autoResult = await resolveAutoStrategyOrder({
orderedTargets,
body,
combo,
settings,
config,
relayOptions,
resilienceSettings,
log,
buildAutoCandidates,
});
if ("earlyResponse" in autoResult) return autoResult.earlyResponse;
orderedTargets = autoResult.orderedTargets;
autoUsedExplicitRouter = autoResult.autoUsedExplicitRouter;
} else {
orderedTargets = await applyStrategyOrdering(strategy, orderedTargets, {
combo,
config,
body,
log,
apiKeyAllowedConnections,
});
}
// #6168: session stickiness opt-out. Per-combo `config.disableSessionStickiness`
// overrides the global `settings.disableSessionStickiness` fallback (default false,
// preserving the #3825 prompt-cache/504 fix). When disabled, skip the reorder and
// treat the result as a no-op so the recordStickyBinding write-back below is skipped.
const disableSessionStickiness = resolveDisableSessionStickiness(
config as Record<string, unknown> | null | undefined,
settings as Record<string, unknown> | null | undefined
);
const _sticky = disableSessionStickiness
? ({ targets: orderedTargets, messageHash: null, stuck: false } as const)
: await applySessionStickiness(
orderedTargets,
// #7270: normalize both wire shapes (.messages / Responses-API .input) so the
// stickiness key is derivable on the /v1/responses surface, not just Chat Completions.
normalizeStickinessMessages(body as { messages?: unknown; input?: unknown })
);
orderedTargets = _sticky.targets;
orderedTargets = orderTargetsByEvalScores(orderedTargets, config.evalRouting, log);
orderedTargets = filterTargetsByRequestCompatibility(orderedTargets, body, log);
orderedTargets = applyContextRequirements(orderedTargets, config.contextRequirements, log);
// Task-aware reordering: only active for strategies ["smart","task","task-aware","task_aware","auto"].
// Additive — does not affect any of the other 15 strategies.
if (isTaskRoutingStrategy(strategy)) {
const task = classifyTask(body);
const conversationCacheKey = getConversationCacheKey(body);
const taskReordered = reorderByTaskWeight(orderedTargets, task);
// #4945 regression guard: when an explicit auto router (lkgp/cost/…) pinned
// orderedTargets[0], keep that primary choice and let task-aware refine only
// the fallback tail — otherwise task weighting silently defeats the operator's
// chosen LKGP/cost selection. reorderByTaskWeight returns the same target
// objects (no clone), so identity filtering is safe.
const pinnedFirst = autoUsedExplicitRouter ? orderedTargets[0] : undefined;
const nextOrder = pinnedFirst
? [pinnedFirst, ...taskReordered.filter((t) => t !== pinnedFirst)]
: taskReordered;
if (nextOrder[0]?.modelStr !== orderedTargets[0]?.modelStr) {
const reasons =
Array.isArray(task.reasons) && task.reasons.length > 0
? ` (${task.reasons.join(",")})`
: "";
log.info(
"COMBO",
`task-route task=${task.level}${reasons} cacheKey=${conversationCacheKey ?? "none"}${nextOrder[0]?.modelStr}`
);
}
orderedTargets = nextOrder;
}
// Parallel pre-screen: check provider profiles and model availability for all targets
// Only runs for priority strategy where sequential checking causes latency
const preScreenMap =
strategy === "priority"
? await preScreenTargets(orderedTargets, isModelAvailable).catch(
() => new Map<string, PreScreenResult>()
)
: new Map<string, PreScreenResult>();
// #5923 (Finding #4) — reset-window config for the shared per-target quota-
// exhaustion cutoff below. The "auto" strategy already applies its own cutoff
// via buildAutoCandidates/routableCandidates, so this only affects the other
// 16 strategies (priority, weighted, etc.) that funnel through executeTarget.
const quotaCutoffResetWindowConfig = resolveResetWindowConfig(config as Record<string, unknown>);
// QA P0 diagnostics: record the order in which targets were actually attempted
// (provider/model ids only) so a terminal combo failure can report the attempt
// sequence alongside pool size + exhaustion reasons. Accumulates across set retries.
const comboAttemptOrder: Array<{ provider: string; model: string }> = [];
if (orderedTargets.length === 0) {
// Surface a recovery hint + auto-clear the session pin after enough consecutive
// no-target failures (silent-stop fix). Threshold of 3 prevents a one-off account
// wipe from destroying the prompt-cache pin benefit on the next request.
recordComboFailure(effectiveSessionId, combo.name);
return errorResponseWithComboDiagnostics(
404,
"Combo has no executable targets",
{
poolSize: 0,
attempted: 0,
excluded: [],
attemptOrder: [],
terminalReason: "no_executable_targets",
recovery: buildRecoveryHint("no_executable_targets"),
},
{ code: "model_not_found", type: "invalid_request_error" }
);
}
scheduleShadowRouting(
combo,
config,
body,
resolveShadowTargets(combo, config, allCombos),
handleSingleModel,
isModelAvailable,
strategy,
log
);
// G2: Collect execution keys registered by _registerExecutionCandidates above (auto strategy).
// We snapshot them now so cleanup can happen after the attempt loop finishes.
const _registeredExecutionKeys = orderedTargets.map((t) => t.executionKey).filter(Boolean);
let globalAttempts = 0;
// Quota-share cooldown-aware retry (Variante A). Only quota-share (qtSd/)
// combos opt in: when the set loop would crystallize a 429 model_cooldown
// because the target hit a SHORT transient cooldown, we wait it out and
// re-run the whole set loop instead of propagating the 429. `globalAttempts`
// persists across these waits so MAX_GLOBAL_ATTEMPTS still bounds total work.
// The wait happens at the crystallization point. The only semaphore slot the
// quota-share path may hold is the FASE 2.1 per-connection concurrency slot
// (acquired once around dispatchWithCooldownRetry below); it is intentionally
// kept across the wait so the account stays "busy", and is released by the
// outer finally — not here.
//
// The set loop is wrapped in a small recursive closure rather than an extra
// labelled `while (true)` so the loop body keeps its original indentation; a
// wait+redispatch is a tail `return dispatchWithCooldownRetry()`, which
// re-runs ONLY the set loop (selection / shadow routing / setup above stay
// untouched), preserving the pre-existing `continue`-to-top-of-set-loop
// semantics exactly.
const comboCooldownWaitEnabled =
strategy === "quota-share" && resilienceSettings.comboCooldownWait.enabled;
let comboCooldownAttempt = 0;
let comboCooldownBudgetLeftMs = resilienceSettings.comboCooldownWait.budgetMs;
// FASE 2.1: per-connection concurrency limit for quota-share. The gating in
// selectQuotaShareTarget is fail-open and cannot hard-limit a single-connection
// pool, so we serialize concurrent requests to the selected account through a
// per-connection semaphore. Enabled only for quota-share combos (the cap is the
// account's) and gated by the kill-switch; the slot wraps the whole dispatch.
const quotaShareConcurrencyEnabled =
strategy === "quota-share" && resilienceSettings.quotaShareConcurrencyLimit.enabled;
const dispatchWithCooldownRetry = async (): Promise<Response> => {
for (let setTry = 0; setTry <= maxSetRetries; setTry++) {
// #1731: Per-set-iteration set of providers whose quota is fully exhausted.
// Reset each retry so providers excluded in a previous attempt get another chance.
const exhaustedProviders = new Set<string>();
const exhaustedConnections = new Set<string>();
const transientRateLimitedProviders = new Set<string>();
if (setTry > 0) {
log.info("COMBO", `All targets failed — retrying set (${setTry}/${maxSetRetries})`);
await new Promise((resolve) => {
const timer = setTimeout(resolve, setRetryDelayMs);
signal?.addEventListener(
"abort",
() => {
clearTimeout(timer);
resolve(undefined);
},
{ once: true }
);
});
if (signal?.aborted) {
log.info("COMBO", "Client disconnected during set retry delay — aborting");
return errorResponse(499, "Client disconnected");
}
}
let lastError: string | null = null;
let earliestRetryAfter: ComboRetryAfter | null = null;
let lastStatus: number | null = null;
const startTime = Date.now();
let fallbackCount = 0;
let recordedAttempts = 0;
// QA P0: assemble a sanitized diagnostic trace from the state already in scope
// (pool size + this set-try's exhausted providers/connections + attempt order +
// a terminal-reason code). Never touches keys/tokens — provider/model ids only.
// Silent-stop fix: include a `recovery` hint (action verb + human next-step) so the
// OC plugin + non-header-aware clients can render an actionable error instead of an
// opaque 5xx. The optional `retryAfterSeconds` carries the upstream Retry-After hint.
const buildComboDiag = (
terminalReason: string,
retryAfterSeconds?: number
): ComboDiagnostics => ({
poolSize: orderedTargets.length,
attempted: recordedAttempts,
excluded: [
...[...exhaustedProviders].map((p) => ({ provider: p, reason: "exhausted" })),
...[...exhaustedConnections].map((c) => ({
provider: "unknown",
reason: `exhausted_connection:${String(c).slice(0, 8)}`,
})),
],
attemptOrder: comboAttemptOrder,
terminalReason,
recovery: buildRecoveryHint(terminalReason, retryAfterSeconds),
});
let globalResolve: ((res: Response) => void) | null = null;
const globalPromise = new Promise<Response>((res) => {
globalResolve = res;
});
const runningTasks = new Set<Promise<void>>();
let anySuccess = false;
const abortControllers = new Map<number, AbortController>();
const zeroLatencyOptimizationsEnabled = config.zeroLatencyOptimizationsEnabled === true;
const executeTarget = async (
i: number
): Promise<{ ok: boolean; response?: Response } | null> => {
const target = orderedTargets[i];
const modelStr = target.modelStr;
const rawModel = parseModel(modelStr).model || modelStr;
const provider = target.provider;
const cb = getCircuitBreaker(provider);
if (cb.getStatus().state === "OPEN") {
log.info("COMBO", `Skipping ${modelStr} — circuit breaker OPEN for ${provider}`);
if (i > 0) fallbackCount++;
return null;
}
if (
resilienceSettings.providerCooldown.enabled &&
Boolean(provider && provider !== "unknown") &&
isProviderInCooldown(provider, target.connectionId ?? undefined, resilienceSettings)
) {
log.info("COMBO", `Skipping ${modelStr} — provider ${provider} in global cooldown`);
if (i > 0) fallbackCount++;
return null;
}
// Use pre-screened profile if available, otherwise fetch on demand
const preScreenEntry = preScreenMap.get(target.executionKey);
const profile = preScreenEntry?.profile ?? (await getRuntimeProviderProfile(provider));
const allowRateLimitedConnection =
Boolean(provider && provider !== "unknown") &&
transientRateLimitedProviders.has(provider);
const targetForAttempt = allowRateLimitedConnection
? {
...target,
allowRateLimitedConnection: true,
modelAbortSignal: abortControllers.get(i)!.signal,
}
: { ...target, modelAbortSignal: abortControllers.get(i)!.signal };
// #1731 / #1731v2: skip targets already known-exhausted this request (shared predicate).
const exhaustedSkip = getExhaustedTargetSkipReason(
target,
exhaustedProviders,
exhaustedConnections
);
if (exhaustedSkip) {
log.info("COMBO", exhaustedSkip);
if (i > 0) fallbackCount++;
return null;
}
// Pre-check: skip models locked by the resilience system (model-level lockout)
if (provider && rawModel && isModelLocked(provider, target.connectionId || "", rawModel)) {
log.info("COMBO", `Skipping ${modelStr} — model locked by resilience (cooldown active)`);
if (i > 0) fallbackCount++;
return null;
}
// #5923 (Finding #4) — honor the same opt-in quota-exhaustion cutoff the
// "auto" strategy already applies (buildAutoCandidates), for every other
// strategy (priority, weighted, etc.). Strictly scoped per (provider,
// connectionId): a 0%-remaining connection is skipped here, but sibling
// connections/models on the same provider are untouched — the provider
// circuit breaker is never touched by this check. The "auto" strategy is
// excluded to avoid a redundant duplicate fetch — it already filtered its
// candidate pool via `routableCandidates` before reaching this loop.
if (strategy !== "auto" && provider && target.connectionId) {
const quotaCutoff = await resolveQuotaExhaustionCutoffForTarget(
provider,
target.connectionId,
resilienceSettings,
quotaCutoffResetWindowConfig,
combo.name,
log
);
if (quotaCutoff.blocked) {
log.info(
"COMBO",
`Skipping ${modelStr} — quota exhaustion cutoff (${quotaCutoff.reason || "quota_exhausted"})`
);
if (i > 0) fallbackCount++;
return null;
}
}
// Pre-screen snapshot is NOT used as a permanent skip — availability
// is always re-checked via isModelAvailable below because connection
// cooldowns can expire between setTry retries, making a previously
// unavailable target available again. Circuit-breaker-OPEN providers
// are already caught by the dedicated breaker check above.
if (isModelAvailable) {
const available = await isModelAvailable(modelStr, targetForAttempt);
if (!available) {
log.debug?.(
"COMBO",
`Skipping ${modelStr} — no credentials available or model excluded`
);
if (i > 0) fallbackCount++;
return null;
}
}
// Credential gate: skip targets with known-bad credentials (fail-fast)
const connectionId = target.connectionId as string | undefined;
if (connectionId) {
const gateResult = checkCredentialGate(connectionId, provider, modelStr);
if (gateResult.allowed === false) {
logCredentialSkip(log, modelStr, gateResult.reason || "Credential gate blocked");
if (i > 0) fallbackCount++;
return null;
}
}
// Retry loop for transient errors
for (let retry = 0; retry <= maxRetries; retry++) {
// Fix #1681: Bail out immediately if the client has disconnected
if (signal?.aborted) {
log.info("COMBO", `Client disconnected — aborting combo loop before model ${modelStr}`);
return { ok: false, response: errorResponse(499, "Client disconnected") };
}
globalAttempts++;
if (globalAttempts > MAX_GLOBAL_ATTEMPTS) {
log.warn(
"COMBO",
`Maximum combo attempts (${MAX_GLOBAL_ATTEMPTS}) exceeded across all targets and fallbacks. Terminating loop to prevent runaway background requests.`
);
// Actionable failure instead of an opaque 503 when every candidate
// failed the same recoverable way. If the dominant cause was reasoning
// models exhausting a too-small max_tokens budget (no content output),
// retrying other models can't help — tell the caller to raise max_tokens.
// Silent-stop fix: bump the consecutive-failure counter for this session-combo pair
// so the pin gets cleared on the 3rd attempt (recovery.next_step tells the client).
const reasoningExhausted = /reasoning consumed \d+\/\d+ tokens/.test(lastError || "");
const failureReason = reasoningExhausted
? "reasoning_budget_exhausted"
: "max_attempts_exceeded";
recordComboFailure(effectiveSessionId, combo.name);
return {
ok: false,
response: errorResponseWithComboDiagnostics(
503,
reasoningExhausted
? "All combo candidates exhausted their token budget on reasoning without producing content. Increase max_tokens — reasoning models need a larger budget to emit content."
: "Maximum combo retry limit reached",
buildComboDiag(failureReason)
),
};
}
// Predictive TTFT Circuit Breaker (skip slow models)
if (
zeroLatencyOptimizationsEnabled &&
config.predictiveTtftMs &&
config.predictiveTtftMs > 0 &&
retry === 0
) {
const cMetrics = getComboMetrics(combo.name);
if (cMetrics) {
const targetKey = orderedTargets[i].executionKey || modelStr;
const m = cMetrics.byTarget[targetKey] || cMetrics.byModel[modelStr];
if (shouldSkipForPredictedTtft(m, config.predictiveTtftMs)) {
log.warn(
"COMBO",
`Predictive TTFT Circuit Breaker: skipping ${modelStr} (avg ${m.avgLatencyMs}ms > max ${config.predictiveTtftMs}ms)`
);
return null;
}
}
}
if (retry > 0) {
log.info(
"COMBO",
`Retrying ${modelStr} in ${retryDelayMs}ms (attempt ${retry + 1}/${maxRetries + 1})`
);
await new Promise((resolve) => {
const timer = setTimeout(resolve, retryDelayMs);
signal?.addEventListener(
"abort",
() => {
clearTimeout(timer);
resolve(undefined);
},
{ once: true }
);
});
if (signal?.aborted) {
log.info("COMBO", `Client disconnected during retry delay — aborting`);
return { ok: false, response: errorResponse(499, "Client disconnected") };
}
}
log.info(
"COMBO",
`Trying model ${i + 1}/${orderedTargets.length}: ${modelStr}${retry > 0 ? ` (retry ${retry})` : ""}`
);
emit("combo.target.attempt", {
comboName: combo.name,
targetIndex: i,
provider,
model: modelStr,
timestamp: Date.now(),
strategy,
});
// QA P0 diagnostics: capture the attempt order (provider/model ids only).
comboAttemptOrder.push({ provider: provider ?? "unknown", model: modelStr });
// Deep clone the body to ensure context preservation and prevent mutations
// from affecting other targets in the combo. structuredClone avoids the
// full intermediate JSON string that JSON.parse(JSON.stringify(...)) builds
// (a second multi-hundred-KB allocation per target on large agent payloads),
// halving the per-target transient heap on the hot path (#5152).
let attemptBody = structuredClone(body);
// Proactive Context Compression for fallbacks (Zero-Latency optimization)
if (
zeroLatencyOptimizationsEnabled &&
i > 0 &&
config.fallbackCompressionMode &&
config.fallbackCompressionMode !== "off"
) {
const { estimateTokens } = await import("./contextManager.ts");
const estimatedTokens = estimateTokens(JSON.stringify(attemptBody));
if (estimatedTokens > (config.fallbackCompressionThreshold ?? 1000)) {
const { applyCompression } = await import("./compression/strategySelector.ts");
const compressionResult = applyCompression(
attemptBody,
config.fallbackCompressionMode as CompressionMode,
// Opt into the TV1 bail-out so a throwing fallback engine is SKIPPED rather than
// propagating out of executeTarget and being swallowed as a "Speculative task
// error" (which silently drops this combo target). minGainPercent:0 keeps the
// advance behavior identical to the default path — this only adds skip-on-throw.
{ model: modelStr, bailout: { enabled: true, minGainPercent: 0 } }
);
if (compressionResult.compressed) {
log.info(
"COMBO",
`Proactive fallback compression applied (${config.fallbackCompressionMode}): ${estimatedTokens} -> ${compressionResult.stats?.compressedTokens} tokens`
);
attemptBody = compressionResult.body;
}
}
}
// Universal handoff: inject existing handoff if model changed
if (
universalHandoffConfig.enabled &&
relayOptions?.sessionId &&
!(body as Record<string, unknown>)?.[SKIP_UNIVERSAL_HANDOFF_FLAG]
) {
const lastModel = getLastSessionModel(relayOptions.sessionId, combo.name);
if (lastModel && lastModel !== modelStr) {
const existingHandoff = getHandoff(relayOptions.sessionId, combo.name);
attemptBody = injectUniversalHandoffBody(
attemptBody, // Use the cloned body to maintain isolation
lastModel,
modelStr,
`Model routing: ${lastModel}${modelStr}`,
existingHandoff
);
}
}
// Issue #3587: Reasoning models can spend the whole output budget on
// reasoning. Only add headroom when the complete buffer fits inside the
// model's known output cap; otherwise preserve the client's explicit limit.
{
const bodyRecord = attemptBody as Record<string, unknown>;
const currentMaxTokens = toPositiveInteger(bodyRecord.max_tokens);
const bufferedMaxTokens = resolveReasoningBufferedMaxTokens(
modelStr,
bodyRecord.max_tokens,
{ enabled: reasoningTokenBufferEnabled }
);
if (currentMaxTokens !== null && bufferedMaxTokens !== null) {
bodyRecord.max_tokens = bufferedMaxTokens;
if (bufferedMaxTokens !== currentMaxTokens) {
log.info(
"COMBO",
`Reasoning model ${modelStr}: adjusted max_tokens ${currentMaxTokens} -> ${bufferedMaxTokens}`
);
}
}
}
const result = await handleSingleModelWithTimeout(attemptBody, modelStr, {
...targetForAttempt,
effectiveComboStrategy: strategy,
failoverBeforeRetry: config.failoverBeforeRetry,
});
// Success — validate response quality before returning
if (result.ok) {
const selectedConnectionId =
result.headers?.get("X-OmniRoute-Selected-Connection-Id") ||
result.headers?.get("x-omniroute-selected-connection-id") ||
undefined;
const effectiveConnectionId = selectedConnectionId || target.connectionId || "";
// Clone BEFORE quality check — validateResponseQuality reads the body
// via getReader() which locks the stream. The clone's body is consumed
// by the quality check; the original stays unlocked for piping.
let qualityClone: Response;
try {
qualityClone = result.clone();
} catch {
qualityClone = result;
}
const quality = await validateResponseQuality(
qualityClone,
clientRequestedStream,
log,
config.responseValidation
);
releaseQualityClone(qualityClone, result, quality);
if (!quality.valid) {
releaseRejectedQualityResponse(qualityClone, result);
log.warn(
"COMBO",
`Model ${modelStr} returned 200 but failed quality check: ${quality.reason}`
);
// #6692: a quality-rejected 200 never marks the connection row
// unhealthy, so the sticky pin's lazy headroom recheck would never
// catch it either — release it here, on the failing response.
releaseStickyPinOnFailure(_sticky.messageHash, effectiveConnectionId);
recordComboRequest(combo.name, modelStr, {
success: false,
latencyMs: Date.now() - startTime,
fallbackCount,
strategy,
target: toRecordedTarget(target),
});
recordedAttempts++;
// Fix #1707: Set terminal state so the fallback doesn't emit
// misleading ALL_ACCOUNTS_INACTIVE when the real issue is quality.
lastError = `Upstream response failed quality validation: ${quality.reason}`;
if (!lastStatus) lastStatus = 502;
if (i > 0) fallbackCount++;
if (provider && rawModel) {
const mlSettings = resolveModelLockoutSettings(settings);
if (mlSettings.enabled && mlSettings.errorCodes.includes(502)) {
recordModelLockoutFailure(
provider,
target.connectionId || "",
rawModel,
"quality_failure",
502,
mlSettings.baseCooldownMs,
profile,
{
exactCooldownMs: mlSettings.useExponentialBackoff
? 0
: mlSettings.baseCooldownMs,
maxCooldownMs: mlSettings.maxCooldownMs,
}
);
}
}
emit("combo.target.failed", {
comboName: combo.name,
targetIndex: i,
provider,
model: modelStr,
error: `Quality: ${quality.reason}`,
latencyMs: Date.now() - startTime,
});
return null;
}
// Success decay: a healthy response walks the model's lockout failure
// count back down (and eventually clears an expired lockout entirely).
if (provider && rawModel) {
const dcResult = decayModelFailureCount(provider, effectiveConnectionId, rawModel);
if (dcResult.cleared) {
log.info("COMBO", `Model ${modelStr} fully recovered — lockout cleared`);
} else if (dcResult.newFailureCount > 0) {
log.debug(
"COMBO",
`Model ${modelStr} decayed to failureCount=${dcResult.newFailureCount}`
);
}
}
const latencyMs = Date.now() - startTime;
emit("combo.target.succeeded", {
comboName: combo.name,
targetIndex: i,
provider,
model: modelStr,
latencyMs,
});
log.info(
"COMBO",
`Model ${modelStr} succeeded (${latencyMs}ms, ${fallbackCount} fallbacks)`
);
recordComboRequest(combo.name, modelStr, {
success: true,
latencyMs,
fallbackCount,
strategy,
target: toRecordedTarget(target),
});
recordedAttempts++;
// Reset cooldown on success
if (provider && provider !== "unknown") {
recordProviderSuccess(provider, effectiveConnectionId || undefined);
}
if (strategy === "weighted" && stickyWeightedLimit > 1) {
const stickySuccessKey = getWeightedStepKeyForTarget(target);
if (stickySuccessKey) {
recordStickyWeightedSuccess(combo.name, stickySuccessKey, stickyWeightedLimit);
}
}
// Webhook fan-out: best-effort, never blocks the response stream.
notifyWebhookEvent("request.completed", {
combo: combo.name,
provider,
model: modelStr,
account:
typeof target.label === "string" && target.label.trim().length > 0
? target.label.trim()
: "",
accountId: effectiveConnectionId ?? "",
latencyMs,
fallbackCount,
});
// Silent-stop fix: reset the consecutive-failure counter for this session-combo pair
// on every successful dispatch so a transient recovery doesn't get "credited" against
// the threshold the user already paid through to clear the stale pin.
if (effectiveSessionId) {
clearComboFailureTracking(effectiveSessionId, combo.name);
}
// Context cache pinning: record model usage for session-based pinning
// (independent of universal handoff — always fires when context_cache_protection is on)
// #3825: write under the SAME effectiveSessionId used by the read site so a
// sessionless conversation re-pins to this model on its next turn.
if (
combo.context_cache_protection &&
effectiveSessionId &&
!(body as Record<string, unknown>)?.[SKIP_UNIVERSAL_HANDOFF_FLAG]
) {
recordSessionModelUsage(
effectiveSessionId,
combo.name,
modelStr,
provider,
target.connectionId ?? undefined
);
}
// Universal handoff: record model usage for session
if (
universalHandoffConfig.enabled &&
relayOptions?.sessionId &&
!(body as Record<string, unknown>)?.[SKIP_UNIVERSAL_HANDOFF_FLAG]
) {
const prevModel = getLastSessionModel(relayOptions.sessionId, combo.name);
recordSessionModelUsage(
relayOptions.sessionId,
combo.name,
modelStr,
provider,
target.connectionId ?? undefined
);
if (prevModel && prevModel !== modelStr) {
const handoffSourceMessages =
Array.isArray(body?.messages) && body.messages.length > 0
? body.messages
: Array.isArray(body?.input)
? body.input
: [];
maybeGenerateUniversalHandoff({
sessionId: relayOptions.sessionId,
comboName: combo.name,
messages: handoffSourceMessages as MessageLike[],
prevModel,
currModel: modelStr,
universalConfig: universalHandoffConfig,
handleSingleModel: handleSingleModelWithTimeout,
});
}
recordSessionModelUsage(
relayOptions.sessionId,
combo.name,
modelStr,
provider,
target.connectionId ?? undefined
);
}
// Context-relay intentionally splits responsibilities:
// combo.ts decides whether a successful turn should generate a handoff,
// while chat.ts injects the handoff after the real connectionId is resolved.
if (
strategy === "context-relay" &&
relayOptions?.sessionId &&
relayConfig &&
relayConfig.handoffProviders.includes(provider) &&
provider === "codex"
) {
const connectionId = getSessionConnection(relayOptions.sessionId);
if (connectionId) {
const quotaInfo = await fetchCodexQuota(connectionId).catch(() => null);
if (quotaInfo) {
const resetCandidates = [
quotaInfo.windows?.session?.resetAt,
quotaInfo.windows?.weekly?.resetAt,
quotaInfo.resetAt,
]
.filter(
(value): value is string => typeof value === "string" && value.length > 0
)
.sort((a, b) => a.localeCompare(b));
const handoffSourceMessages =
Array.isArray(body?.messages) && body.messages.length > 0
? body.messages
: Array.isArray(body?.input)
? body.input
: [];
maybeGenerateHandoff({
sessionId: relayOptions.sessionId,
comboName: combo.name,
connectionId,
percentUsed: quotaInfo.percentUsed,
messages: handoffSourceMessages,
model: modelStr,
expiresAt: resetCandidates[0] || null,
config: relayConfig,
handleSingleModel: handleSingleModelWithTimeout,
});
}
}
}
if (_sticky.messageHash && target.connectionId)
recordStickyBinding(_sticky.messageHash, target.connectionId); // LKGP (#919):
if (provider) {
const connId = effectiveConnectionId || undefined;
void (async () => {
try {
const { setLKGP } = await import("../../src/lib/localDb");
await Promise.all([
setLKGP(combo.name, target.executionKey, provider, connId),
setLKGP(combo.name, combo.id || combo.name, provider, connId),
]);
} catch (err) {
log.warn(
"COMBO",
"Failed to record Last Known Good Provider. This is non-fatal.",
{
err,
}
);
}
})();
}
return { ok: true, response: result };
}
// Extract error info from response
let errorText = result.statusText || "";
let errorBody: ComboErrorBody = null;
let retryAfter: ComboRetryAfter | null = null;
try {
const cloned = result.clone();
try {
const text = await cloned.text();
if (text) {
errorText = text.substring(0, 500);
errorBody = JSON.parse(text);
const parsedError = errorBody?.error;
errorText =
(typeof parsedError === "object" && parsedError?.message) ||
(typeof parsedError === "string" ? parsedError : null) ||
errorBody?.message ||
errorText;
retryAfter = errorBody?.retryAfter || null;
}
} catch {
/* Clone parse failed */
}
} catch {
/* Clone failed */
}
// Track earliest retryAfter
if (
retryAfter &&
(!earliestRetryAfter || new Date(retryAfter) < new Date(earliestRetryAfter))
) {
earliestRetryAfter = retryAfter;
}
// Normalize error text
if (typeof errorText !== "string") {
try {
errorText = JSON.stringify(errorText);
} catch {
errorText = String(errorText);
}
}
const isStreamReadinessFailure =
(result.status === 502 || result.status === 504) &&
isStreamReadinessFailureErrorBody(errorBody);
// FIX 5: a local per-API-key token-limit 429 must not cool shared accounts.
const isTokenLimitBreach =
result.status === 429 && isTokenLimitBreachErrorBody(errorBody);
// Fix #1681: Status 499 means client disconnected — stop combo loop immediately.
// There is no point trying fallback models when nobody is listening.
if (result.status === 499) {
log.info("COMBO", `Client disconnected (499) during ${modelStr} — stopping combo loop`);
recordComboRequest(combo.name, modelStr, {
success: false,
latencyMs: Date.now() - startTime,
fallbackCount,
strategy,
target: toRecordedTarget(target),
});
recordedAttempts++;
// executeTarget must return the {ok,response} contract — a raw Response
// here makes the speculative loop's res.ok/res.response checks both miss,
// so the combo would wrongly fall through to the next model after a 499.
return { ok: false, response: result };
}
// Combo fallback is target-level orchestration: a non-ok target response is
// treated as local to that target and the combo continues to the next target.
// Error classification is retained only for retry/cooldown pacing; it must
// not decide whether fallback happens, including for generic 400 responses.
const rawError = errorBody?.error;
const structuredError =
rawError && typeof rawError === "object"
? {
// Upstream JSON may carry a numeric `code`/`type` (e.g. {"code":40001}).
// Coerce to string if present instead of discarding, so downstream string
// ops (.toLowerCase, .startsWith) can run safely without type crashes.
code:
(rawError as Record<string, unknown>).code !== undefined &&
(rawError as Record<string, unknown>).code !== null
? String((rawError as Record<string, unknown>).code)
: undefined,
type:
(rawError as Record<string, unknown>).type !== undefined &&
(rawError as Record<string, unknown>).type !== null
? String((rawError as Record<string, unknown>).type)
: undefined,
}
: undefined;
const requestScopedFailure = isRequestScopedUpstreamFailure(structuredError);
const fallbackResult = checkFallbackError(
result.status,
errorText,
0,
null,
provider,
result.headers,
profile,
structuredError
);
const { cooldownMs } = fallbackResult;
// #6863: a parsed upstream quota reset (e.g. Antigravity "Resets in 92h27m28s")
// arrives in `quotaResetHintMs` — it bypasses the operator-gated
// `useUpstreamRetryHints` connection-cooldown setting. Mirror the
// single-model path (src/sse/services/auth.ts): when the retry hint was
// already honored, `cooldownMs` IS the upstream value; otherwise prefer the
// parsed quota reset — even when it is SHORTER than the fallback cooldown
// (e.g. subscription-quota 1h default vs a real "resets in 10m").
// `selectLockoutCooldownMs` still ignores hints at/below the base cooldown,
// so absent/tiny hints keep the #1308 exponential-backoff behavior.
const lockoutHintMs =
fallbackResult.usedUpstreamRetryHint === true
? cooldownMs
: (fallbackResult.quotaResetHintMs ?? 0);
const selectedConnectionId =
result.headers?.get("X-OmniRoute-Selected-Connection-Id") ||
result.headers?.get("x-omniroute-selected-connection-id") ||
undefined;
const targetWithConnection = selectedConnectionId
? { ...target, connectionId: selectedConnectionId }
: target;
// #1731 / #1731v2: classify the upstream error and update the exhaustion sets
// (shared with handleRoundRobinCombo). Returns whether the provider is fully exhausted.
const providerExhausted = applyComboTargetExhaustion(targetWithConnection, {
result,
fallbackResult,
errorText,
rawModel,
isTokenLimitBreach,
allAccountsRateLimited: false,
sets: { exhaustedProviders, exhaustedConnections, transientRateLimitedProviders },
log,
tag: "COMBO",
exhaustedLogLevel: "info",
structuredError,
});
// #6692: this connection was just classified as provider/connection-level
// exhausted — if it's the currently sticky-bound one, release the pin now
// rather than waiting for the next turn's lazy headroom/status recheck.
releaseStickyPinOnFailure(_sticky.messageHash, targetWithConnection.connectionId);
// #2101: Prevent infinite fallback loops with 400 Bad Request errors that are genuinely
// body-specific (malformed JSON, bad format, missing required fields).
// Context overflow and parameter validation errors are NOT body-specific:
// - Context overflow: different models have different context windows
// - Max_tokens / param errors: different models have different output limits
// - Model access denied: different providers serve different model sets
// These should fall through so the next combo target can try.
if (
result.status === 400 &&
fallbackResult.shouldFallback &&
!isContextOverflow400(errorText) &&
!isParamValidation400(errorText) &&
(errorText.toLowerCase().includes("context") ||
errorText.toLowerCase().includes("prompt") ||
errorText.toLowerCase().includes("token") ||
errorText.toLowerCase().includes("malformed") ||
errorText.toLowerCase().includes("invalid") ||
errorText.toLowerCase().includes("bad request"))
) {
log.warn(
"COMBO",
`400 Bad Request with body-specific error detected on ${modelStr} — skipping fallback to other targets to prevent infinite loop`
);
// Record the failure and break to avoid trying other targets with the same bad request
recordComboRequest(combo.name, modelStr, {
success: false,
latencyMs: Date.now() - startTime,
fallbackCount,
strategy,
target: toRecordedTarget(target),
});
recordedAttempts++;
lastError = errorText || String(result.status);
if (!lastStatus) lastStatus = result.status;
if (i > 0) fallbackCount++;
log.warn("COMBO", `Model ${modelStr} failed with body-specific error, stopping combo`);
// #4279: surface the 400 via the {ok,response} contract so the OUTER
// target loop resolves the combo and stops. A bare `break` here only
// exits the inner retry loop; executeTarget then returns null, which
// the outer loop treats as "this target produced nothing" and advances
// to the next model — so the guard failed to stop fallback and a combo
// of N body-rejecting targets tried all N. Mirrors the 499 path above.
return { ok: false, response: result };
}
// Trigger shared provider circuit breaker for 5xx errors and connection failures.
// If the next target in the combo is on the same provider, don't mark the provider
// as failed — different models on the same provider may still succeed.
// G-02: when fallbackResult.skipProviderBreaker is set (embedded service supervisor
// outage signalled via X-Omni-Fallback-Hint: connection_cooldown) apply connection
// cooldown only — do NOT trip the whole-provider breaker.
const nextTarget = orderedTargets[i + 1];
const sameProviderNext =
typeof nextTarget?.provider === "string" && nextTarget.provider === provider;
if (
shouldRecordProviderBreakerFailure({
isStreamReadinessFailure,
status: result.status,
sameProviderNext,
skipProviderBreaker: fallbackResult.skipProviderBreaker,
requestScopedFailure,
})
) {
recordProviderFailure(provider, log, targetWithConnection.connectionId, profile);
}
// Check if this is a transient error worth retrying on same model.
// A token-limit 429 is terminal for the client — never retry it.
const isTransient =
!isStreamReadinessFailure &&
!isTokenLimitBreach &&
[408, 429, 500, 502, 503, 504].includes(result.status);
if (retry < maxRetries && isTransient && !providerExhausted) {
if (
provider &&
rawModel &&
isModelLocked(provider, targetWithConnection.connectionId || "", rawModel)
) {
log.info("COMBO", `Skipping retry for ${modelStr} — model lockout active`);
if (i > 0) fallbackCount++;
return null;
}
// Record model lockout immediately on the first transient failure —
// once the model is cooling down, retrying it would waste an upstream
// call and extend the cooldown via exponential backoff.
let lockoutRecorded = false;
if (provider && rawModel && retry === 0 && !requestScopedFailure) {
const mlSettings = resolveModelLockoutSettings(settings);
if (mlSettings.enabled && mlSettings.errorCodes.includes(result.status)) {
recordModelLockoutFailure(
provider,
targetWithConnection.connectionId || "",
rawModel,
classifyLockoutReason(result.status),
result.status,
mlSettings.baseCooldownMs,
profile,
{
// #1308/#6863: honor a long upstream reset (e.g. "Resets in 160h") over
// the short base cooldown / exponential backoff when present.
exactCooldownMs: selectLockoutCooldownMs(lockoutHintMs, mlSettings),
maxCooldownMs: mlSettings.maxCooldownMs,
}
);
lockoutRecorded = true;
}
}
if (lockoutRecorded) {
log.info("COMBO", `Skipping retry for ${modelStr} — model lockout active`);
if (i > 0) fallbackCount++;
return null;
}
continue; // Retry same model (transient error, no lockout recorded)
}
// Done retrying this model
recordComboRequest(combo.name, modelStr, {
success: false,
latencyMs: Date.now() - startTime,
fallbackCount,
strategy,
target: toRecordedTarget(target),
});
recordedAttempts++;
lastError = errorText || String(result.status);
if (!lastStatus) lastStatus = result.status;
if (i > 0) fallbackCount++;
// Wire combo failures into the resilience dashboard (model-level lockout)
// alongside the provider-level cooldown below — they govern different scopes.
if (provider && rawModel && !requestScopedFailure) {
const mlSettings = resolveModelLockoutSettings(settings);
if (mlSettings.enabled && mlSettings.errorCodes.includes(result.status)) {
recordModelLockoutFailure(
provider,
targetWithConnection.connectionId || "",
rawModel,
classifyLockoutReason(result.status),
result.status,
mlSettings.baseCooldownMs,
profile,
{
// #1308/#6863: honor a long upstream reset over base/exponential cooldown.
exactCooldownMs: selectLockoutCooldownMs(lockoutHintMs, mlSettings),
maxCooldownMs: mlSettings.maxCooldownMs,
}
);
}
}
log.warn("COMBO", `Model ${modelStr} failed, trying next`, { status: result.status });
// #5976: per-model-quota providers (Gemini, GitHub, etc.) multiplex models
// behind one connection. A model-level 500 must NOT cool down the entire
// provider — sibling models may still succeed. Skip cooldown recording for
// these providers on 500 errors so the next target can try.
if (
resilienceSettings.providerCooldown.enabled &&
provider &&
provider !== "unknown" &&
!requestScopedFailure &&
!(result.status === 500 && hasPerModelQuota(provider, rawModel))
) {
recordProviderCooldown(
provider,
targetWithConnection.connectionId ?? undefined,
resilienceSettings
);
}
const fallbackWaitMs =
fallbackDelayMs > 0 && cooldownMs > 0 && cooldownMs <= MAX_FALLBACK_WAIT_MS
? Math.min(cooldownMs, fallbackDelayMs)
: 0;
if ([502, 503, 504].includes(result.status) && fallbackWaitMs > 0) {
log.debug?.("COMBO", `Waiting ${fallbackWaitMs}ms before fallback to next model`);
await new Promise((resolve) => {
const timer = setTimeout(resolve, fallbackWaitMs);
signal?.addEventListener(
"abort",
() => {
clearTimeout(timer);
resolve(undefined);
},
{ once: true }
);
});
if (signal?.aborted) {
log.info("COMBO", `Client disconnected during fallback wait — aborting`);
return { ok: false, response: errorResponse(499, "Client disconnected") };
}
}
return null;
}
return null;
};
for (let i = 0; i < orderedTargets.length; i++) {
if (anySuccess) break;
const abortController = new AbortController();
abortControllers.set(i, abortController);
const onClientAbort = () => abortController.abort();
signal?.addEventListener("abort", onClientAbort);
const task = (async () => {
try {
const res = await executeTarget(i);
if (res && !anySuccess) {
if (res.ok) {
anySuccess = true;
globalResolve!(res.response!);
for (const [idx, ac] of abortControllers.entries()) {
if (idx !== i) ac.abort();
}
} else if (res.response) {
// Fatal error, abort combo
anySuccess = true;
globalResolve!(res.response);
}
}
} finally {
signal?.removeEventListener("abort", onClientAbort);
}
})().catch((err) => {
const logError = log.error ?? log.warn;
logError("COMBO", `Speculative task error for target ${i}`, err);
});
runningTasks.add(task);
task.finally(() => runningTasks.delete(task));
if (zeroLatencyOptimizationsEnabled && config.hedging && i + 1 < orderedTargets.length) {
const hedgeDelay = resolveDelayMs(config.hedgeDelayMs, 500);
let timeoutResolve: () => void;
const timeoutPromise = new Promise<void>((r) => {
timeoutResolve = r;
setTimeout(r, hedgeDelay);
});
await Promise.race([task, globalPromise, timeoutPromise]);
} else {
await Promise.race([task, globalPromise]);
}
}
if (!anySuccess && runningTasks.size > 0) {
await Promise.race([globalPromise, Promise.all([...runningTasks])]);
}
if (anySuccess) {
return await globalPromise;
}
// All models failed in this set try
const latencyMs = Date.now() - startTime;
if (recordedAttempts === 0) {
recordComboRequest(combo.name, null, {
success: false,
latencyMs,
fallbackCount,
strategy,
});
}
// Retry the entire set if more attempts remain
if (setTry < maxSetRetries) continue;
// All set retries exhausted — return the final error
if (!lastStatus) {
notifyWebhookEvent("request.failed", {
combo: combo.name,
reason: "ALL_ACCOUNTS_INACTIVE",
latencyMs,
fallbackCount,
});
// Silent-stop fix: bump the failure counter so the session pin clears on the 3rd
// consecutive all-inactive cascade; buildRecoveryHint emits `switch-combo` with a
// next-step that points the user at /dashboard/providers.
recordComboFailure(effectiveSessionId, combo.name);
return errorResponseWithComboDiagnostics(
503,
"Service temporarily unavailable: all upstream accounts are inactive",
buildComboDiag("all_accounts_inactive"),
{ code: "ALL_ACCOUNTS_INACTIVE", type: "service_unavailable" }
);
}
const status = lastStatus;
const msg = lastError || "All combo models unavailable";
if (earliestRetryAfter) {
// Quota-share cooldown-aware retry: instead of crystallizing the 429,
// wait out a SHORT transient cooldown and re-run the whole set loop.
// Guarded by the helper (quota_exhausted/auth/not-found excluded,
// ceiling, attempts, budget). MAX_GLOBAL_ATTEMPTS still bounds total
// dispatches.
if (comboCooldownWaitEnabled && status === 429) {
const decision = resolveComboCooldownWaitDecision({
targets: orderedTargets,
earliestRetryAfter,
attempt: comboCooldownAttempt,
budgetLeftMs: comboCooldownBudgetLeftMs,
settings: resilienceSettings.comboCooldownWait,
lookupLock: (provider, connectionId) => {
const rawModel = parseModel(orderedTargets[0]?.modelStr ?? "").model || "";
return getModelLockoutInfo(provider, connectionId, rawModel);
},
computeWaitMs: (retryAfter) => computeClosestRetryAfter(retryAfter).waitMs,
});
if (decision.wait) {
log.info(
"COMBO",
`Quota-share cooldown wait: ${msg} — waiting ${Math.ceil(
decision.waitMs / 1000
)}s (reason=${decision.reason ?? "?"}) then retrying (attempt ${
comboCooldownAttempt + 1
}/${resilienceSettings.comboCooldownWait.maxAttempts})`
);
const completed = await waitForCooldownAwareRetry(decision.waitMs, signal);
if (!completed) {
log.info("COMBO", "Quota-share cooldown wait aborted by client disconnect");
return errorResponse(499, "Request aborted");
}
comboCooldownAttempt += 1;
comboCooldownBudgetLeftMs = Math.max(0, comboCooldownBudgetLeftMs - decision.waitMs);
return dispatchWithCooldownRetry();
}
}
const retryHuman = formatRetryAfter(toRetryAfterDisplayValue(earliestRetryAfter));
log.warn("COMBO", `All models failed | ${msg} (${retryHuman})`);
return unavailableResponse(status, msg, earliestRetryAfter, retryHuman);
}
// Silent-stop fix: bump the failure counter (pin clears on 3rd consecutive) and emit
// `try-auto` recovery action via buildRecoveryHint so the OC plugin can show "→ Try
// model: auto" instead of an opaque 5xx. We pass the upstream retry-after seconds to
// the hint so the client can render a precise "wait Ns and retry" message.
log.warn("COMBO", `All models failed | ${msg}`);
const { pinClearedNow } = recordComboFailure(effectiveSessionId, combo.name);
if (pinClearedNow) {
log.info(
"COMBO",
`Auto-cleared session_model_history pin for combo "${combo.name}" after ${COMBO_FAILURE_THRESHOLD} consecutive failures to break the silent-stop loop`
);
}
const retryAfterSeconds = undefined;
return errorResponseWithComboDiagnostics(
status,
msg,
buildComboDiag(lastError ?? "all_models_failed", retryAfterSeconds)
);
}
// Final fallback — when the dispatch returned without crystallizing a status (rare).
// Surface the recovery hint with a generic retry recommendation so the client at least
// gets a non-opaque message instead of "Combo routing completed without an upstream response".
recordComboFailure(effectiveSessionId, combo.name);
return errorResponseWithComboDiagnostics(
503,
"Combo routing completed without an upstream response",
buildNoUpstreamResponseDiagnostics(orderedTargets.length)
);
};
// FASE 2.1: acquire the per-connection concurrency slot for the selected
// quota-share target once, around the whole dispatch (including any
// cooldown-aware re-dispatch), so concurrent requests to one subscription
// account are serialized through the connection's max_concurrent ceiling. The
// cap is read fresh from the selected connection; a null cap (no limit) or a
// saturated queue is a no-op (fail-open). Released in the finally below.
let quotaShareConcurrencyRelease: (() => void) | null = null;
const qsConnectionId = orderedTargets[0]?.connectionId;
if (quotaShareConcurrencyEnabled && qsConnectionId) {
const qsCap = await lookupPositiveCap(qsConnectionId);
quotaShareConcurrencyRelease = await acquireQuotaShareConcurrencySlot(
orderedTargets[0],
qsCap,
{
queueTimeoutMs: config.queueTimeoutMs ?? 30000,
maxQueueSize: resolveComboQueueDepth(config),
},
log
);
}
try {
return await dispatchWithCooldownRetry();
} finally {
quotaShareConcurrencyRelease?.();
// G2: Clean up candidate registry to prevent unbounded memory growth.
_unregisterExecutionCandidates(_registeredExecutionKeys);
}
}
/**
* Handle round-robin combo: each request goes to the next model in circular order.
* Uses semaphore-based concurrency control with queue + rate-limit awareness.
*
* Flow:
* 1. Pick target model via atomic counter (counter % models.length)
* 2. Acquire semaphore slot (may queue if at max concurrency)
* 3. Send request to target model
* 4. On 429 → mark model rate-limited, try next model in rotation
* 5. On semaphore timeout → fallback to next available model
*/
async function handleRoundRobinCombo({
body,
combo,
handleSingleModel,
isModelAvailable,
log,
settings,
allCombos,
signal,
}: HandleRoundRobinOptions): Promise<Response> {
const config = settings
? resolveComboConfig(combo, settings)
: { ...getDefaultComboConfig(), ...(combo.config || {}) };
const concurrency = config.concurrencyPerModel ?? 3;
// Honor each target connection's own maxConcurrent ceiling (cached per dispatch)
// so a low-concurrency subscription account is not flooded; falls back to the
// combo-level concurrency when the connection has no positive cap.
const resolveTargetConcurrency = makeConnectionConcurrencyResolver(concurrency);
const queueTimeout = config.queueTimeoutMs ?? 30000;
// #3872: pre-cascade queue depth — lower values fail over to the next combo member
// sooner under concurrency saturation (0 = never queue). Default 20 (backward-compat).
const queueDepth = resolveComboQueueDepth(config);
const maxRetries = config.maxRetries ?? 1;
const retryDelayMs = resolveDelayMs(config.retryDelayMs, 2000);
const fallbackDelayMs = resolveDelayMs(config.fallbackDelayMs, 0);
const reasoningTokenBufferEnabled = config.reasoningTokenBufferEnabled !== false;
const resilienceSettings: ResilienceSettings = settings
? resolveResilienceSettings(settings)
: resolveResilienceSettings(null);
// #2562: Expand provider-wildcard steps before resolving targets.
const rrExpandedCombo = await expandProviderWildcardsInCombo(combo);
const rrExpandedAllCombos = allCombos
? Array.isArray(allCombos)
? await expandProviderWildcardsInCollection(allCombos as ComboLike[])
: {
...allCombos,
combos: await expandProviderWildcardsInCollection(
((allCombos as { combos?: ComboLike[] }).combos || []) as ComboLike[]
),
}
: allCombos;
const orderedTargets = resolveComboTargets(
rrExpandedCombo,
rrExpandedAllCombos,
clampComboDepth(config.maxComboDepth)
);
const tagFilteredTargets = await applyRequestTagRouting(orderedTargets, body, log);
const evalRankedTargets = orderTargetsByEvalScores(tagFilteredTargets, config.evalRouting, log);
const knownContextOverflow = getKnownContextOverflow(evalRankedTargets, body);
if (knownContextOverflow) {
return errorResponseWithComboDiagnostics(
400,
`Request requires approximately ${knownContextOverflow.requiredContextTokens} tokens, but the largest known context limit in this combo is ${knownContextOverflow.maxKnownContextTokens} tokens. Reduce or compact the request context.`,
{
poolSize: evalRankedTargets.length,
attempted: 0,
excluded: evalRankedTargets.map((target) => ({
provider: target.provider,
model: target.modelStr,
reason: "context_window",
})),
attemptOrder: [],
terminalReason: "context_length_exceeded",
},
{ code: "context_length_exceeded", type: "invalid_request_error" }
);
}
const filteredTargets = filterTargetsByRequestCompatibility(
evalRankedTargets,
body,
log,
"Context-aware round-robin fallback"
);
// #6238: keep the targets the compat pre-filter rejected so they can serve as a
// last-resort fallback tier. The pre-filter drops request-incompatible targets
// BEFORE availability is known; if every compat-kept target then turns out to be
// runtime-unavailable, we must reconsider these before returning 503, instead of
// permanently dropping a compat-rejected-but-healthy provider.
const compatKeptSet = new Set(filteredTargets);
const compatRejectedTargets = evalRankedTargets.filter((target) => !compatKeptSet.has(target));
const modelCount = filteredTargets.length;
if (modelCount === 0) {
return comboModelNotFoundResponse("Round-robin combo has no executable targets");
}
scheduleShadowRouting(
combo,
config,
body,
resolveShadowTargets(combo, config, allCombos),
handleSingleModel,
isModelAvailable,
"round-robin",
log
);
// Sticky batch size at the combo level. A per-combo `stickyRoundRobinLimit` (in
// combo.config, resolved through the cascade) overrides the global setting so one
// combo can batch differently from the default. When the per-combo value is unset,
// fall back to the global `stickyRoundRobinLimit` so the existing knob still controls
// sticky batching for both account fallback and combo targets. Values <= 1 preserve
// the historical one-request-per-target rotation.
const perComboStickyLimit = (config as Record<string, unknown>).stickyRoundRobinLimit;
const stickyLimit = resolveComboStickyRoundRobinLimit(
perComboStickyLimit,
settings as Record<string, unknown> | null
);
const stickyRoundRobinEnabled = stickyLimit > 1;
// Exhaustion-aware sticky: if the currently sticky target is no longer
// available (circuit breaker OPEN, provider cooldown, model lockout, or
// isModelAvailable returns false), clear the sticky record so the rotation
// starts at the counter position instead of probing a dead target.
if (stickyRoundRobinEnabled) {
const sticky = rrStickyTargets.get(combo.name);
if (sticky) {
const stickyTarget = filteredTargets.find(
(target) => target.executionKey === sticky.executionKey
);
if (stickyTarget) {
const rawModel = parseModel(stickyTarget.modelStr).model || stickyTarget.modelStr;
const stickyAvailable =
(!stickyTarget.provider ||
getCircuitBreaker(stickyTarget.provider).getStatus().state !== "OPEN") &&
!(
resilienceSettings.providerCooldown.enabled &&
Boolean(stickyTarget.provider && stickyTarget.provider !== "unknown") &&
isProviderInCooldown(
stickyTarget.provider,
stickyTarget.connectionId ?? undefined,
resilienceSettings
)
) &&
!(
stickyTarget.provider &&
rawModel &&
isModelLocked(stickyTarget.provider, stickyTarget.connectionId || "", rawModel)
) &&
(isModelAvailable ? await isModelAvailable(stickyTarget.modelStr, stickyTarget) : true);
if (!stickyAvailable) {
log.info(
"COMBO-RR",
`Clearing stale sticky target ${stickyTarget.modelStr} — unavailable`
);
rrStickyTargets.delete(combo.name);
}
}
}
}
if (
!rrCounters.has(combo.name) &&
!rrStickyTargets.has(combo.name) &&
rrCounters.size >= MAX_RR_COUNTERS
) {
const oldest = rrCounters.keys().next().value;
if (oldest !== undefined) {
rrCounters.delete(oldest);
rrStickyTargets.delete(oldest);
}
}
// Ensure rrCounters has an entry for this combo so the eviction logic above
// applies to both maps even when sticky round-robin is enabled (in which
// case rrCounters isn't incremented per request).
if (!rrCounters.has(combo.name)) {
rrCounters.set(combo.name, 0);
}
const { startIndex, counter } = getStickyRoundRobinStartIndex(
combo.name,
filteredTargets,
stickyLimit
);
if (!stickyRoundRobinEnabled) {
rrCounters.set(combo.name, counter + 1);
}
// #3825: per-conversation session stickiness for round-robin. weighted/priority honor a
// sticky connection via applySessionStickiness, but this RR handler returns before that
// call — so sessionless RR combos rotated every turn, busting the upstream prompt-cache.
// Reuse the SAME mechanism: start the rotation at the conversation's sticky connection
// (the loop still falls through to the other targets on failure → failover preserved).
// #6168: honor the session-stickiness opt-out here too, otherwise round-robin would
// still pin the conversation even when the flag is set. Per-combo `config` overrides
// the global `settings.disableSessionStickiness` fallback (default false).
const disableSessionStickiness = resolveDisableSessionStickiness(
config as Record<string, unknown> | null | undefined,
settings as Record<string, unknown> | null | undefined
);
const _rrSessionSticky = disableSessionStickiness
? ({ targets: filteredTargets, messageHash: null, stuck: false } as const)
: await applySessionStickiness(
filteredTargets,
// #7270: normalize both wire shapes (.messages / Responses-API .input) so RR
// stickiness engages on the /v1/responses surface, not just Chat Completions.
normalizeStickinessMessages(body as { messages?: unknown; input?: unknown })
);
let rrStartIndex = startIndex;
if (_rrSessionSticky.stuck) {
const stickyIdx = filteredTargets.findIndex(
(t) => t.connectionId === _rrSessionSticky.targets[0]?.connectionId
);
if (stickyIdx >= 0) rrStartIndex = stickyIdx;
}
const clientRequestedStream = body?.stream === true;
const startTime = Date.now();
let lastError: string | null = null;
let lastStatus: number | null = null;
let earliestRetryAfter: ComboRetryAfter | null = null;
let globalAttempts = 0;
let fallbackCount = 0;
let recordedAttempts = 0;
// #1731: Per-request in-memory set of providers whose quota is fully exhausted.
// When a target returns a quota-exhausted 429, remaining targets from the same
// provider are skipped to avoid the cascade through N same-provider targets.
const exhaustedProviders = new Set<string>();
const exhaustedConnections = new Set<string>();
const transientRateLimitedProviders = new Set<string>();
// Try each model starting from the round-robin target
for (let offset = 0; offset < modelCount; offset++) {
const modelIndex = (rrStartIndex + offset) % modelCount;
const target = filteredTargets[modelIndex];
const modelStr = target.modelStr;
const provider = target.provider;
const profile = await getRuntimeProviderProfile(provider);
const semaphoreKey = `combo:${combo.name}:${target.executionKey}`;
const allowRateLimitedConnection =
Boolean(provider && provider !== "unknown") && transientRateLimitedProviders.has(provider);
const targetForAttempt = allowRateLimitedConnection
? { ...target, allowRateLimitedConnection: true }
: target;
// Pre-check availability
if (isModelAvailable) {
const available = await isModelAvailable(modelStr, targetForAttempt);
if (!available) {
log.debug?.(
"COMBO-RR",
`Skipping ${modelStr} — no credentials available or model excluded`
);
if (offset > 0) fallbackCount++;
continue;
}
}
if (
resilienceSettings.providerCooldown.enabled &&
Boolean(provider && provider !== "unknown") &&
isProviderInCooldown(provider, target.connectionId as string | undefined, resilienceSettings)
) {
log.info("COMBO-RR", `Skipping ${modelStr} — provider ${provider} in global cooldown`);
if (offset > 0) fallbackCount++;
continue;
}
// #1731 / #1731v2: skip targets already known-exhausted this request (shared predicate).
const exhaustedSkip = getExhaustedTargetSkipReason(
target,
exhaustedProviders,
exhaustedConnections
);
if (exhaustedSkip) {
log.info("COMBO-RR", exhaustedSkip);
if (offset > 0) fallbackCount++;
continue;
}
// Acquire semaphore slot (may wait in queue). Honor the connection's own
// maxConcurrent cap when set; else fall back to the combo-level concurrency.
const targetConcurrency = await resolveTargetConcurrency(target.connectionId);
let release: () => void;
try {
release = await semaphore.acquire(semaphoreKey, {
maxConcurrency: targetConcurrency,
timeoutMs: queueTimeout,
maxQueueSize: queueDepth,
});
} catch (err) {
const errCode = isRecord(err) && typeof err.code === "string" ? err.code : null;
if (errCode === "SEMAPHORE_TIMEOUT" || errCode === "SEMAPHORE_QUEUE_FULL") {
log.warn(
"COMBO-RR",
`Semaphore ${errCode === "SEMAPHORE_QUEUE_FULL" ? "queue full" : "timeout"} for ${modelStr}, trying next model`
);
if (offset > 0) fallbackCount++;
continue;
}
throw err;
}
// Retry loop within this model
try {
for (let retry = 0; retry <= maxRetries; retry++) {
globalAttempts++;
if (globalAttempts > MAX_GLOBAL_ATTEMPTS) {
log.warn(
"COMBO-RR",
`Maximum combo attempts (${MAX_GLOBAL_ATTEMPTS}) exceeded. Terminating loop to prevent runaway requests.`
);
return errorResponse(503, "Maximum combo retry limit reached");
}
if (retry > 0) {
log.info(
"COMBO-RR",
`Retrying ${modelStr} in ${retryDelayMs}ms (attempt ${retry + 1}/${maxRetries + 1})`
);
await new Promise((r) => setTimeout(r, retryDelayMs));
}
log.info(
"COMBO-RR",
`[RR #${counter}] → ${modelStr}${offset > 0 ? ` (fallback +${offset})` : ""}${retry > 0 ? ` (retry ${retry})` : ""}`
);
// Issue #3587: Reasoning models can spend the whole output budget on
// reasoning. Apply any safe buffer to a per-attempt copy so round-robin
// retries never compound across models.
let attemptBody = body;
{
const bodyRecord = body as Record<string, unknown>;
const currentMaxTokens = toPositiveInteger(bodyRecord.max_tokens);
const bufferedMaxTokens = resolveReasoningBufferedMaxTokens(
modelStr,
bodyRecord.max_tokens,
{ enabled: reasoningTokenBufferEnabled }
);
if (
currentMaxTokens !== null &&
bufferedMaxTokens !== null &&
bufferedMaxTokens !== currentMaxTokens
) {
attemptBody = {
...bodyRecord,
max_tokens: bufferedMaxTokens,
} as typeof body;
log.info(
"COMBO-RR",
`Reasoning model ${modelStr}: adjusted max_tokens ${currentMaxTokens} -> ${bufferedMaxTokens}`
);
}
}
const result = await handleSingleModel(attemptBody, modelStr, {
...targetForAttempt,
effectiveComboStrategy: "round-robin",
failoverBeforeRetry: config.failoverBeforeRetry,
});
// Success — validate response quality before returning
if (result.ok) {
let rrClone: Response;
try {
rrClone = result.clone();
} catch {
rrClone = result;
}
const quality = await validateResponseQuality(
rrClone,
clientRequestedStream,
log,
config.responseValidation
);
releaseQualityClone(rrClone, result, quality);
if (!quality.valid) {
releaseRejectedQualityResponse(rrClone, result);
log.warn(
"COMBO-RR",
`${modelStr} returned 200 but failed quality check: ${quality.reason}`
);
// #6692: same rationale as handleComboChat's quality-fail branch —
// a quality-rejected 200 never marks the connection row unhealthy,
// so release the sticky pin here rather than on the next turn.
{
const rrSelectedConnectionId =
result.headers?.get("X-OmniRoute-Selected-Connection-Id") ||
result.headers?.get("x-omniroute-selected-connection-id") ||
undefined;
releaseStickyPinOnFailure(
_rrSessionSticky.messageHash,
rrSelectedConnectionId || target.connectionId
);
}
recordComboRequest(combo.name, modelStr, {
success: false,
latencyMs: Date.now() - startTime,
fallbackCount,
strategy: "round-robin",
target: toRecordedTarget(target),
});
recordedAttempts++;
// Fix #1707: Set terminal state so the fallback doesn't emit
// misleading ALL_ACCOUNTS_INACTIVE when the real issue is quality.
lastError = `Upstream response failed quality validation: ${quality.reason}`;
if (!lastStatus) lastStatus = 502;
if (offset > 0) fallbackCount++;
break; // move to next model
}
const latencyMs = Date.now() - startTime;
log.info(
"COMBO-RR",
`${modelStr} succeeded (${latencyMs}ms, ${fallbackCount} fallbacks)`
);
recordComboRequest(combo.name, modelStr, {
success: true,
latencyMs,
fallbackCount,
strategy: "round-robin",
target: toRecordedTarget(target),
});
recordedAttempts++;
const selectedConnectionId =
result.headers?.get("X-OmniRoute-Selected-Connection-Id") ||
result.headers?.get("x-omniroute-selected-connection-id") ||
undefined;
const effectiveConnectionId = selectedConnectionId || target.connectionId || "";
const rawModel = parseModel(modelStr).model || modelStr;
if (provider && rawModel) {
const dcResult = decayModelFailureCount(provider, effectiveConnectionId, rawModel);
if (dcResult.cleared) {
log.info("COMBO-RR", `Model ${modelStr} fully recovered — lockout cleared`);
} else if (dcResult.newFailureCount > 0) {
log.debug?.(
"COMBO-RR",
`Model ${modelStr} decayed to failureCount=${dcResult.newFailureCount}`
);
}
}
if (provider && provider !== "unknown") {
recordProviderSuccess(provider, effectiveConnectionId || undefined);
}
if (stickyRoundRobinEnabled) {
recordStickyRoundRobinSuccess(combo.name, target, stickyLimit, filteredTargets);
} else {
// #948: true round-robin (stickyLimit <= 1). The counter was advanced
// eagerly (+1 from the scheduled start index) before this loop ran, so
// when the scheduled model failed and a *different* model served via
// fallback, the next request reused the fallback-served model. Advance
// the pointer past the model that ACTUALLY served (modelIndex) instead,
// mirroring recordStickyRoundRobinSuccess's served-index logic. Read
// side applies `% modelCount`, so storing modelIndex + 1 is correct.
rrCounters.set(combo.name, modelIndex + 1);
}
// #3825: (re)record the sticky binding so the next turn re-pins (prompt-cache).
if (_rrSessionSticky.messageHash) {
const stickyConn = effectiveConnectionId || target.connectionId;
if (stickyConn) recordStickyBinding(_rrSessionSticky.messageHash, stickyConn);
}
if (provider) {
const connId = effectiveConnectionId || undefined;
void (async () => {
try {
const { setLKGP } = await import("../../src/lib/localDb");
await Promise.all([
setLKGP(combo.name, target.executionKey, provider, connId),
setLKGP(combo.name, combo.id || combo.name, provider, connId),
]);
} catch (err) {
log.warn(
"COMBO-RR",
"Failed to record Last Known Good Provider. This is non-fatal.",
{
err,
}
);
}
})();
}
// Clone is consumed by quality check; original stays unlocked.
return result;
}
// Extract error info
let errorText = result.statusText || "";
let retryAfter: ComboRetryAfter | null = null;
let errorBody: ComboErrorBody = null;
try {
const cloned = result.clone();
try {
const text = await cloned.text();
if (text) {
errorText = text.substring(0, 500);
errorBody = JSON.parse(text);
const parsedError = errorBody?.error;
errorText =
(typeof parsedError === "object" && parsedError?.message) ||
(typeof parsedError === "string" ? parsedError : null) ||
errorBody?.message ||
errorText;
retryAfter = errorBody?.retryAfter || null;
}
} catch {
/* Clone parse failed */
}
} catch {
/* Clone failed */
}
if (result.status === 499) {
log.info(
"COMBO-RR",
`Client disconnected (499) during ${modelStr} — stopping combo loop`
);
recordComboRequest(combo.name, modelStr, {
success: false,
latencyMs: Date.now() - startTime,
fallbackCount,
strategy: "round-robin",
target: toRecordedTarget(target),
});
recordedAttempts++;
return result;
}
if (
retryAfter &&
(!earliestRetryAfter || new Date(retryAfter) < new Date(earliestRetryAfter))
) {
earliestRetryAfter = retryAfter;
}
if (typeof errorText !== "string") {
try {
errorText = JSON.stringify(errorText);
} catch {
errorText = String(errorText);
}
}
const isStreamReadinessFailure =
(result.status === 502 || result.status === 504) &&
isStreamReadinessFailureErrorBody(errorBody);
// FIX 5: a local per-API-key token-limit 429 must not cool shared accounts.
const isTokenLimitBreach = result.status === 429 && isTokenLimitBreachErrorBody(errorBody);
// Round-robin uses the same target-level fallback rule as other combo
// strategies: non-ok target responses fall through to the next target.
// Classification stays here only to support cooldown/semaphore pacing,
// not to decide whether fallback is allowed.
const rawError = errorBody?.error;
const structuredError =
rawError && typeof rawError === "object"
? {
// Upstream JSON may carry a numeric `code`/`type` (e.g. {"code":40001}).
// Coerce to string if present instead of discarding, so downstream string
// ops (.toLowerCase, .startsWith) can run safely without type crashes.
code:
(rawError as Record<string, unknown>).code !== undefined &&
(rawError as Record<string, unknown>).code !== null
? String((rawError as Record<string, unknown>).code)
: undefined,
type:
(rawError as Record<string, unknown>).type !== undefined &&
(rawError as Record<string, unknown>).type !== null
? String((rawError as Record<string, unknown>).type)
: undefined,
}
: undefined;
const requestScopedFailure = isRequestScopedUpstreamFailure(structuredError);
const fallbackResult = checkFallbackError(
result.status,
errorText,
0,
null,
provider,
result.headers,
profile,
structuredError
);
const { cooldownMs } = fallbackResult;
const selectedConnectionId =
result.headers?.get("X-OmniRoute-Selected-Connection-Id") ||
result.headers?.get("x-omniroute-selected-connection-id") ||
undefined;
const targetWithConnection = selectedConnectionId
? { ...target, connectionId: selectedConnectionId }
: target;
const isAllAccountsRateLimited = isAllAccountsRateLimitedResponse(
result.status,
result.headers?.get("content-type") ?? null,
errorText
);
// #1731: If the entire provider quota is exhausted, mark it so subsequent
// same-provider targets are skipped immediately. API-key 429s still use
// the short resilience cooldown, but explicit quota text should stop the
// combo from trying another target for the same provider in this request.
// #1731 / #1731v2: classify the upstream error and update the exhaustion sets
// (shared with handleComboChat). Returns whether the provider is fully exhausted.
const providerExhausted = applyComboTargetExhaustion(targetWithConnection, {
result,
fallbackResult,
errorText,
rawModel: parseModel(modelStr).model || modelStr,
isTokenLimitBreach,
allAccountsRateLimited: isAllAccountsRateLimited,
sets: { exhaustedProviders, exhaustedConnections, transientRateLimitedProviders },
log,
tag: "COMBO-RR",
exhaustedLogLevel: "debug",
structuredError,
});
// #6692: mirrors handleComboChat's exhaustion-point release above.
releaseStickyPinOnFailure(_rrSessionSticky.messageHash, targetWithConnection.connectionId);
// Transient errors → mark in semaphore so round-robin stops stampeding this target.
if (
!isStreamReadinessFailure &&
!isTokenLimitBreach &&
!requestScopedFailure &&
TRANSIENT_FOR_SEMAPHORE.includes(result.status) &&
cooldownMs > 0
) {
semaphore.markRateLimited(semaphoreKey, cooldownMs);
log.warn("COMBO-RR", `${modelStr} error ${result.status}, cooldown ${cooldownMs}ms`);
}
if (isAllAccountsRateLimited) {
log.info(
"COMBO-RR",
`All accounts rate-limited for ${modelStr}, falling back to next model`
);
}
// Transient error → retry same model.
// A token-limit 429 is terminal for the client — never retry it.
const isTransient =
!isStreamReadinessFailure &&
!isTokenLimitBreach &&
[408, 429, 500, 502, 503, 504].includes(result.status);
if (retry < maxRetries && isTransient && !providerExhausted) {
continue;
}
// Done with this model
recordComboRequest(combo.name, modelStr, {
success: false,
latencyMs: Date.now() - startTime,
fallbackCount,
strategy: "round-robin",
target: toRecordedTarget(target),
});
recordedAttempts++;
lastError = errorText || String(result.status);
if (!lastStatus) lastStatus = result.status;
if (offset > 0) fallbackCount++;
log.warn("COMBO-RR", `${modelStr} failed, trying next model`, { status: result.status });
if (
resilienceSettings.providerCooldown.enabled &&
provider &&
provider !== "unknown" &&
!requestScopedFailure &&
!(
result.status === 500 &&
hasPerModelQuota(provider, parseModel(modelStr).model || modelStr)
)
) {
recordProviderCooldown(
provider,
targetWithConnection.connectionId ?? undefined,
resilienceSettings
);
}
const fallbackWaitMs =
fallbackDelayMs > 0 && cooldownMs > 0 && cooldownMs <= MAX_FALLBACK_WAIT_MS
? Math.min(cooldownMs, fallbackDelayMs)
: 0;
if ([502, 503, 504].includes(result.status) && fallbackWaitMs > 0) {
log.debug?.("COMBO-RR", `Waiting ${fallbackWaitMs}ms before fallback to next model`);
await new Promise((resolve) => {
const timer = setTimeout(resolve, fallbackWaitMs);
signal?.addEventListener(
"abort",
() => {
clearTimeout(timer);
resolve(undefined);
},
{ once: true }
);
});
if (signal?.aborted) {
log.info("COMBO-RR", `Client disconnected during fallback wait — aborting`);
return errorResponse(499, "Client disconnected");
}
}
break;
}
} finally {
// ALWAYS release semaphore slot
release();
}
}
// All models exhausted
const latencyMs = Date.now() - startTime;
// #6238: every compat-kept target was skipped as unavailable and NONE was ever
// attempted (recordedAttempts === 0). Before crystallizing 503, probe the targets
// the compat pre-filter rejected — a compat-rejected-but-healthy provider is a
// valid last-resort fallback tier, not a permanently dropped target.
if (recordedAttempts === 0 && compatRejectedTargets.length > 0) {
const compatFallbackResult = await attemptCompatRejectedFallback(compatRejectedTargets, body, {
handleSingleModel,
isModelAvailable,
isProviderInCooldown: (target) =>
resilienceSettings.providerCooldown.enabled &&
Boolean(target.provider && target.provider !== "unknown") &&
isProviderInCooldown(
target.provider as string,
target.connectionId as string | undefined,
resilienceSettings
),
log,
strategy: "round-robin",
});
if (compatFallbackResult) {
recordComboRequest(combo.name, null, {
success: true,
latencyMs: Date.now() - startTime,
fallbackCount,
strategy: "round-robin",
});
return compatFallbackResult;
}
}
if (recordedAttempts === 0) {
recordComboRequest(combo.name, null, {
success: false,
latencyMs,
fallbackCount,
strategy: "round-robin",
});
}
if (!lastStatus) {
return new Response(
JSON.stringify({
error: {
message: "Service temporarily unavailable: all upstream accounts are inactive",
type: "service_unavailable",
code: "ALL_ACCOUNTS_INACTIVE",
},
}),
{ status: 503, headers: { "Content-Type": "application/json" } }
);
}
const status = lastStatus;
const msg = lastError || "All round-robin combo models unavailable";
if (earliestRetryAfter) {
const retryHuman = formatRetryAfter(toRetryAfterDisplayValue(earliestRetryAfter));
log.warn("COMBO-RR", `All models failed | ${msg} (${retryHuman})`);
return unavailableResponse(status, msg, earliestRetryAfter, retryHuman);
}
log.warn("COMBO-RR", `All models failed | ${msg}`);
return new Response(JSON.stringify({ error: { message: msg } }), {
status,
headers: { "Content-Type": "application/json" },
});
}