pjandro/OmniRoute
1
0
Fork 0
mirror of https://github.com/diegosouzapw/OmniRoute.git synced 2026-09-18 21:02:50 +03:00
Code Issues Packages Projects Releases Wiki Activity

fix(resilience): lock the exact model, not the quota family, on 5xx model-lockout failures (#12957)

Browse Source 
* fix(resilience): lock the exact model, not the quota family, on 5xx model-lockout failures

A 5xx model-lockout failure — a transport error (terminated, EHOSTUNREACH,
connect timeout), an upstream server error, or OmniRoute's own synthesized
502 from quality validation — is evidence about one model endpoint at that
moment, not about the account's quota family. recordModelLockoutFailure()
wrote it under the quota-family key regardless, so for codex (whose family
key is the whole `codex` scope, i.e. every gpt-5* model) one empty stream
on gpt-5.6-luna removed gpt-5.6-sol and gpt-5.6-terra from routing too,
for 2–30 min with exponential escalation, while the quota was untouched.

- exactModelLock.ts: resolveLockoutScope(status, explicit) — 429/403/402
  (and 404, already narrowed by getModelLockKey) keep the family key; any
  other status uses the exact provider/connection/model key. An explicit
  `scope` option still wins.
- recordModelLockoutFailure() resolves the scope once for key + lock fn.
- decayModelFailureCount() now walks every key shape (family, not_found,
  exact) so success-decay reaches exact-scope locks; null model stays a
  no-op.
- getAllModelLockouts() parses the `exact:` marker out of the key so the
  Model Cooldowns card lists the bare model and can clear it by that name.
- docs: RESILIENCE_GUIDE §3 key-scope-by-status; changelog fragment.

* chore(changelog): name the fragment after PR #12957 and link issue #12955

---------

Co-authored-by: insoln <is@careerum.com>
This commit is contained in:
Innokentiy Solntsev
2026-09-17 21:25:32 +02:00
committed by GitHub
parent 21d756d7f0
commit bc7f68fb91
6 changed files with 288 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
docs/architecture/RESILIENCE_GUIDE.md
View File

@@ -166,6 +166,21 @@ Related mechanisms remain separate:
**Scope:** provider + connection + model triple.
**Key scope by status:** the failing status decides which key a lockout writes
to (`resolveLockoutScope()` in `open-sse/services/accountFallback/exactModelLock.ts`):
- `429` / `403` / `402` — a quota or entitlement signal — lock the **quota family**:
for codex the whole `codex` / `spark` scope (every `gpt-5*` model of the
connection), for other providers `getQuotaScopedModelForProvider()`.
- `404` locks the bare model (`getModelLockKey()` narrows `not_found`).
- Any other status — `5xx` transport/server failures and OmniRoute's own
synthesized `502` from quality validation — locks the **exact**
provider/connection/model tuple only. A bad stream on one model is not evidence
about the account's quota; before this rule one empty response on
`codex/gpt-5.6-luna` removed every `gpt-5*` model of that connection from
routing for 2–30 min (escalating) while its quota was untouched.
- A caller's explicit `scope` option always wins (Antigravity passes `"exact"`).
**Purpose:** avoid disabling a whole connection when only one model is unavailable or quota-limited.
**Examples:**
@@ -224,7 +239,8 @@ escalation window. This success-decay is in addition to plain timer expiry —
either path can re-enable a model.
**State:** lockouts are held **in-memory** (per-process `Map`s of
`ModelLockoutEntry` keyed by `provider:connectionId:model`), not persisted to
`ModelLockoutEntry` keyed by `provider:connectionId:model`, exact-scope locks by
`provider:connectionId:exact:model`), not persisted to
the DB — they are lost on restart. The _settings_ are persisted; the active
lockout _state_ is ephemeral.