OmniRoute/docs/architecture/RESILIENCE_GUIDE.md

title, version, lastUpdated

title	version	lastUpdated
Resilience Guide	3.8.40	2026-06-28

Resilience Guide

OmniRoute has three distinct but related resilience mechanisms. Each has a different scope and purpose. Keep them separate when debugging routing behavior.

Source: diagrams/resilience-3layers.mmd

1. Provider Circuit Breaker

Scope: entire provider (e.g., glm, openai, anthropic).

Purpose: stop sending traffic to a provider that is repeatedly failing at the upstream/service level.

Implementation:

• Core class: src/shared/utils/circuitBreaker.ts
• Wiring: src/sse/handlers/chatHelpers.ts, src/sse/handlers/chat.ts
• Status API: GET /api/monitoring/health
• Reset API: POST /api/resilience/reset
• Wrappers: open-sse/services/accountFallback.ts
• DB table: domain_circuit_breakers

States:

• CLOSED — normal traffic allowed
• DEGRADED — traffic still allowed, but elevated provider failures are being tracked
• OPEN — provider temporarily blocked; combo routing skips it
• HALF_OPEN — reset timeout elapsed; probe request allowed

Configurable defaults (open-sse/config/constants.ts, exposed in Dashboard → Settings → Resilience):

Class	Degraded at	Opens at	Reset timeout
OAuth	5 failures	8 failures	60s
API-key	7 failures	12 failures	30s
Local	derived	2 failures	15s

degradationThreshold controls when a provider enters DEGRADED; failureThreshold controls when it opens and is skipped. Local provider profiles are not exposed on the Resilience settings page yet.

Trip codes: only provider-level statuses [408, 500, 502, 503, 504]. Do NOT trip for account-level errors (most 401/403/429 — those belong to cooldown or lockout).

Lazy recovery: when OPEN expires, getStatus(), canExecute(), getRetryAfterMs() refresh state to HALF_OPEN. No background timer needed.

---

Opt-in global Provider Cooldown (window gate)

A fourth, opt-in layer (PROVIDER_COOLDOWN_ENABLED, default off) keeps a cross-request memory of failing providers in open-sse/services/providerCooldownTracker.ts, consulted by combo target resolution so consecutive combo requests stop re-walking a provider that just failed. Provider-level entries honor the PROVIDER_PROFILES window gate:

Profile	trips after (providerFailureThreshold)	inside (providerFailureWindowMs)	cools for (providerCooldownMs)
OAuth	10	15min	5min
API key	15	30min	10min

Below the threshold the provider is not considered cooling; a success clears the window. Connection-level entries (provider:connectionId) keep the exponential minRetryCooldownMs → maxRetryCooldownMs backoff instead. Overrides: OMNIROUTE_PROVIDER_BREAKER_{OAUTH,API_KEY}_{FAILURE_THRESHOLD,FAILURE_WINDOW_MS,COOLDOWN_MS}. Regression guard: tests/unit/provider-cooldown-window-gate.test.ts.

2. Connection Cooldown

Scope: single provider connection/account/key.

Purpose: skip one bad key while other connections for the same provider keep serving.

Implementation:

• Mark unavailable: src/sse/services/auth.ts::markAccountUnavailable()
• Selection: getProviderCredentials* in same file
• Cooldown calc: open-sse/services/accountFallback.ts::checkFallbackError()
• Settings: src/lib/resilience/settings.ts

Fields per connection:

• rateLimitedUntil — timestamp until cooldown expires
• testStatus: "unavailable"
• lastError, lastErrorType, errorCode
• backoffLevel — exponential backoff counter

Default cooldowns:

• OAuth base: 5s
• API-key base: 3s
• API-key 429: prefers upstream Retry-After/reset headers/parseable reset text
• Backoff: baseCooldownMs * 2 ** failureIndex

Anti-thundering-herd guard: prevents concurrent failures from over-extending cooldown or double-incrementing backoffLevel.

Terminal states (NOT cooldowns):

• banned — set by banned-keyword / account-ban detection (see BAN_DETECTION), and by three consecutive upstream per-request refusals (request_rejected, e.g. Anthropic OAuth 403 "Request not allowed" — open-sse/services/requestRejectedStreak.ts); a single refusal only cools the connection down
• expired (transitions to terminal after bounded retries — EXPIRED_RETRY_MAX = 3 with exponential backoff — so transient OAuth errors can self-heal before the account is permanently deactivated)
• credits_exhausted

These persist until credentials change or an operator resets them. Do not overwrite terminal states with transient cooldown state.

Lazy recovery: when rateLimitedUntil is past, connection becomes eligible again. On successful use, clearAccountError() clears all error fields.

Claude OAuth usage wall: lower-priority lane + session-limit reset

Scope: one Claude subscription (OAuth) connection. Both features are opt-in per connection (Edit connection → Claude section → lowPriorityMode / autoLimitReset in providerSpecificData, both default off) and mirror Claude Code's /low-priority and /limit-reset commands (wire contract captured from Claude Code 2.1.263).

Implementation:

• State machine + response classification: open-sse/services/claudeLowPriority.ts
• Reset status/claim client: open-sse/services/claudeLimitReset.ts
• Executor hook (header injection + same-account retry): open-sse/executors/base.ts::execute()
• Opt-in persistence: src/lib/providers/requestDefaults.ts::normalizeProviderSpecificData()

Trigger: the 5-hour usage wall — a 429 whose headers carry anthropic-ratelimit-unified-status: rejected and, when the account is eligible, anthropic-ratelimit-unified-slow-offer: treatment. Nothing is sent before that first wall 429; a burst 429 without unified headers goes through the normal cooldown path.

Lower-priority lane (lowPriorityMode):

• On the wall 429 the executor accepts the offer and immediately retries the same account with anthropic-usage-limit: slow; the lane stays active until the announced anthropic-ratelimit-unified-reset (+60s grace) and every request in that window carries the header. The intercepted 429 never reaches handleChatCore, so the connection is not put in cooldown and is not rotated away.
• anthropic-ratelimit-unified-slow-status on later responses: active / not_needed keep the lane; slot_busy (429) or a 529 wait the server's anthropic-ratelimit-unified-slow-retry-after (default 20s, clamp 5–600s, ±30% jitter) and retry, bounded by anthropic-ratelimit-unified-slow-max-wait (default 20 min, clamp 1 min–6 h) — past that the lane ends and a 10-minute cool-off blocks re-acceptance. The wait is additionally capped by what is left of the request's own upstream-start timeout (resolveFetchStartTimeout, 10 min by default) minus a 5 s margin: without that cap the 20-minute default max-wait would outlive the request and the sleep would be aborted mid-wait, surfacing a TimeoutError instead of the graceful max_wait end + cool-off.
• weekly_limit / budget_exhausted / off / ineligible, a 5h-window rollover, or ineligible + anthropic-ratelimit-unified-overage-in-use: true (which ends it as extra_usage on any status, since paid overage now covers the wall) end the lane; the response then flows to the normal cooldown path. budget_exhausted is remembered until the announced budget reset (≤ 8 days).
• The wall check runs after the executor's own 400-driven intra-attempt retries (context editing, thinking/effort clamps, param auto-learn), so a wall 429 that only surfaces on one of those retries is still intercepted instead of reaching the cooldown path.
• State is in-memory per connection (a restart costs one extra wall 429 to re-accept).

Session-limit reset (autoLimitReset, tried before the lane when both are on):

• GET https://api.anthropic.com/api/oauth/usage?at_wall=1&skip_spend=1 → juniper_tide block; when arm: "reset" and available: true, POST https://api.anthropic.com/api/organizations/{orgUUID}/reset_rate_limits with { "program": "juniper_tide" } (organization UUID from providerSpecificData.organizationUUID, bootstrap fallback).
• result: reset|not_limited → the request is retried at full speed (no slow header). already_used / not_offered memoise next_available_at (default one week); any failure backs off 15 minutes. The reset is once a week and still counts toward the weekly limit.

Regression guards: tests/unit/claude-low-priority-mode.test.ts, tests/unit/claude-limit-reset.test.ts, tests/unit/claude-low-priority-executor.test.ts.

Session affinity (#7274)

Scope: one client session (X-Session-Id / x-codex-session-id / x-omniroute-session header) pinned to one connection, for any provider.

Purpose: keep a multi-turn agent (Claude Code, aider, custom agents) on the same account across requests, reducing cross-account context loss and repeated cold-start 429s on providers with per-account session state.

Implementation:

• TTL resolution: src/sse/services/sessionAffinityPin.ts::resolveSessionAffinityTtlMs()
• Pin selection/creation: src/sse/services/sessionAffinityPin.ts::selectSessionAffinityConnection()
• Header extraction (generic, any provider): src/sse/services/auth.ts::extractSessionAffinityKey()
• Persisted pin table: sessionAccountAffinity (src/lib/db/sessionAccountAffinity.ts)
• Setting: sessionAffinityTtlMs (global TTL in ms, 0 disables) — src/lib/db/settings.ts. Renamed from the Codex-only codexSessionAffinityTtlMs by migration 124_generic_session_affinity_ttl.sql, which carries over any previously-configured Codex TTL as the new default.

Before #7274, resolveSessionAffinityTtlMs() hard-bailed to 0 for every provider except codex, so the TTL setting (and the session headers) had no effect anywhere else even though the pinning mechanism and header extraction were already provider-agnostic. The fix removed that early-return; the TTL now applies uniformly to every provider once set globally above 0.

The three session-affinity headers are never forwarded upstream — executors build their own upstream headers from scratch rather than passing client headers through, so this stays an internal correlation id only.

Exclusive managed session connection leases

Scope: one active managed HTTP client/session owns one eligible OmniRoute connection.

Purpose: provide durable exclusive connection ownership for clients that need a hard routing fence across requests. This differs from session affinity, which is a soft continuity preference: an exclusive lease persists lifecycle state in SQLite, enforces global active-owner and active-connection uniqueness, and rejects a stale generation before provider dispatch.

The feature is opt-in per API key. A managed key must have the lease:exclusive scope and an explicit non-empty allowedConnections list. Any HTTP client can use the lifecycle endpoint; no client name, user-agent, provider, OAuth method, or model is required. The lease owns a connection, not a model, so a model change retains the binding while the connection remains ordinarily eligible. Normal model, quota, health, cooldown, and allowlist rules remain authoritative and may transition the same generation to another free eligible connection.

The lifecycle is POST /api/v1/session-leases with JSON actions acquire, renew, and release. Managed inference requests present the opaque X-OmniRoute-Lease-Owner value and exact X-OmniRoute-Lease-Generation. The owner uses vlo_ followed by 43 base64url characters; only its SHA-256 hash is stored. Every final dispatch fence also binds the authenticated API key ID and active connection ID. Lease control headers are removed from logs, retained request snapshots, and upstream executor headers.

If ordinary routing has eligible managed candidates but every free candidate is occupied by a foreign active lease, OmniRoute returns HTTP 429, lease-capacity-unavailable code, a waiting-for-capacity state, and a bounded Retry-After derived from the earliest relevant expiry. Ordinary empty eligibility is not lease contention and keeps its existing routing error semantics.

Related mechanisms remain separate:

• OAuth session occupancy is process-local soft distribution for OAuth accounts.
• Account semaphores grant request-concurrency permits and end when a request completes.
• Exclusive managed session leases are durable lifecycle ownership with a generation fence.

---

3. Model Lockout

Scope: provider + connection + model triple.

Key scope by status: the failing status decides which key a lockout writes to (resolveLockoutScope() in open-sse/services/accountFallback/exactModelLock.ts):

• 429 / 403 / 402 — a quota or entitlement signal — lock the quota family: for codex the whole codex / spark scope (every gpt-5* model of the connection), for other providers getQuotaScopedModelForProvider().
• 404 locks the bare model (getModelLockKey() narrows not_found).
• Any other status — 5xx transport/server failures and OmniRoute's own synthesized 502 from quality validation — locks the exact provider/connection/model tuple only. A bad stream on one model is not evidence about the account's quota; before this rule one empty response on codex/gpt-5.6-luna removed every gpt-5* model of that connection from routing for 2–30 min (escalating) while its quota was untouched.
• A caller's explicit scope option always wins (Antigravity passes "exact").

Purpose: avoid disabling a whole connection when only one model is unavailable or quota-limited.

Examples:

• Per-model quota providers returning 429
• Local providers returning 404 for one missing model
• Provider-specific mode/model permission failures (e.g., Grok modes)

Implementation: open-sse/services/accountFallback.ts — lockModel(), clearModelLock(), getAllModelLockouts().

Model Cooldowns Dashboard (v3.8.0)

UI: Settings → Model Cooldowns (src/app/(dashboard)/dashboard/settings/components/ModelCooldownsCard.tsx)

Lists active lockouts with: provider, connection, model, reason, expiresAt. Operators can manually re-enable a model from the card.

REST API:

• GET /api/resilience/model-cooldowns — list active lockouts
• DELETE /api/resilience/model-cooldowns — manual re-enable. Body: {provider, connection, model}. Auth: management.

Lockout settings UI + success-decay recovery (v3.8.23)

Model lockout went from always-on hardcoded behavior to a fully configurable, opt-in feature with its own settings card and a self-healing recovery path.

Settings card: Settings → Model Lockout (src/app/(dashboard)/dashboard/settings/components/ModelLockoutCard.tsx). This is distinct from the read-only ModelCooldownsCard above (which only lists active lockouts) — the new card configures the parameters. Defaults live in DEFAULT_MODEL_LOCKOUT_SETTINGS (src/lib/resilience/modelLockoutSettings.ts):

Setting	Default	Meaning
enabled	false	Master toggle — model lockout is off by default.
errorCodes	[403, 404, 429, 502, 503, 504]	Upstream statuses that count as a model-scoped failure.
baseCooldownMs	120_000 (120 s)	Initial lockout duration for the first failure.
maxCooldownMs	1_800_000 (30 min)	Cap on the escalated cooldown.
maxBackoffSteps	10	Max exponential-backoff escalation steps.
useExponentialBackoff	true	Whether repeated failures escalate the cooldown exponentially.

Settings persist through the normal settings store and validate via the resilience settings schema; the card clamps baseCooldownMs/maxCooldownMs (with maxCooldownMs ≥ baseCooldownMs) and maxBackoffSteps.

Success-decay recovery: recovery is not purely timer expiry. A healthy response walks the model's failure count back down so a model that recovered mid-window stops escalating (and clears) before its timer would. On a successful combo target, open-sse/services/combo.ts calls decayModelFailureCount() (open-sse/services/accountFallback.ts), which halves the stored failureCount (Math.floor(failureCount / 2)); when it reaches 0 the lockout entry is deleted entirely. The counterpart recordModelLockoutFailure() increments the count (and escalates the cooldown) on failures within the escalation window. This success-decay is in addition to plain timer expiry — either path can re-enable a model.

State: lockouts are held in-memory (per-process Maps of ModelLockoutEntry keyed by provider:connectionId:model, exact-scope locks by provider:connectionId:exact:model), not persisted to the DB — they are lost on restart. The settings are persisted; the active lockout state is ephemeral.

---

4. Quota-Share Concurrency Control (v3.8.36)

Subscription accounts (GLM, MiniMax, etc.) often accept only ~1–3 concurrent requests; exceeding that triggers 429s and cooldowns. This is acute under quota-share (qtSd/…) combos, where several API keys share one upstream account. Three layers keep a shared account from being flooded.

Per-connection concurrency cap (max_concurrent)

Each provider connection can declare a max_concurrent ceiling (provider_connections.max_concurrent, set in the connection modal / API / DB). Leave it empty for no limit. This is the single knob that drives the serialization layer below — set it to the account's real concurrency (e.g. GLM ~1, MiniMax ~2).

Quota-share request serialization

When a quota-share dispatch targets a connection that declares a positive max_concurrent, concurrent requests to that account are serialized through a per-connection semaphore (key qsconn:<connectionId>): excess requests wait in the queue instead of flooding the account. It is fail-open — a saturated queue or timeout proceeds without a slot rather than ever rejecting a dispatchable request. Toggle in Settings → Resilience → Quota-share per-connection concurrency (resilienceSettings.quotaShareConcurrencyLimit.enabled, default on). Without a max_concurrent cap the behavior is unchanged.

The quota-share routing gate (selectQuotaShareTarget, DRR + P2C) is itself fail-open and only deprioritizes an at-cap connection — with a single-connection pool it cannot hard-limit, so this semaphore is what actually contains the flood.

Combo cooldown-aware retry

For every combo strategy (when enabled), a request that would crystallize a 429 for a SHORT transient cooldown waits it out and re-dispatches instead of returning the 429 — this covers Gemini-class TPM/RPM windows (~60s retry-after) on multi-model combos, e.g. both targets of a 2-model combo hitting a per-model rate limit. Bounded by comboCooldownWait (enabled, maxWaitMs, maxAttempts, budgetMs) in Settings → Resilience. It never waits on quota_exhausted (locked until midnight) or auth/not-found reasons.

---

5. Request Queue Admission Control (v3.8.49 · issue #6593)

Scope: the local per-provider+connection rate-limit queue (open-sse/services/rateLimitManager.ts, backed by Bottleneck), one layer below the three mechanisms above.

maxWaitMs bounds queue wait; executionMaxWaitMs bounds execution. The two are deliberately separate, and neither feeds the other.

resilienceSettings.requestQueue.maxWaitMs is the queue-wait budget: it covers waiting for a provider slot and then sitting QUEUED, and its timer is cleared the moment the job leaves QUEUED and starts executing (rateLimitManager.ts, wrappedFn). A request that exceeds it never reaches the upstream. Default 30000ms, supplied by DEFAULT_REQUEST_QUEUE_MAX_WAIT_MS in src/lib/resilience/settings.ts and pinned by tests/unit/ratelimit-admission-control-6593.test.ts, so a change to it turns that test red rather than leaving this paragraph quietly stale.

resilienceSettings.requestQueue.executionMaxWaitMs is what Bottleneck receives as the job expiration, whose timer starts only after dispatch. It is a backstop for executors without an upstream timeout of their own, and it is raised to the executor's own fetch-start timeout when that is longer, so it cannot cut off a healthy in-flight response. Default 600000ms (10 min).

Feeding the queue budget into expiration is what used to kill non-incremental gateways mid-flight — they legitimately run for minutes before first bytes — and it is why an expiration is surfaced as code: "RATE_LIMIT_EXECUTION_TIMEOUT" (HTTP 504) while the queue budget carries the queue-timeout code. Override either via RATE_LIMIT_MAX_WAIT_MS / RATE_LIMIT_EXECUTION_MAX_WAIT_MS (env) or the dashboard (Settings → Resilience). Both are clamped to 1ms–24h when normalised.

Precedence, for both: the env var only supplies the default. A value persisted in resilienceSettings.requestQueue (dashboard / API patch, stored in key_value) wins over it, and a per-connection rateLimitOverrides.maxWaitMs / .executionMaxWaitMs wins over that. Setting the env var on a deployment that already has a persisted value therefore changes nothing — clear or update the persisted setting instead.

Queue residence is bounded by maxWaitMs; maxQueueDepth below bounds how many callers may be queued at once.

maxQueueDepth — opt-in admission cap (new). resilienceSettings.requestQueue.maxQueueDepth bounds how many requests may sit queued (not yet dispatched) for one provider+connection at once. When the queue already holds maxQueueDepth requests, a new request is fast-rejected with a typed code: "RATE_LIMIT_QUEUE_FULL" error before it ever reaches limiter.schedule() — so the rejection is cheap and happens ahead of any downstream prompt-compression / translation work for that request. Default 0 = disabled, preserving the existing unbounded-queue behavior; bounded 0–100000. Override via RATE_LIMIT_MAX_QUEUE_DEPTH (env) or resilienceSettings.requestQueue.maxQueueDepth (dashboard/API patch).

The admission check itself is a pure function (open-sse/services/rateLimitManager/admission.ts::checkQueueAdmission) so it is unit-testable without a real Bottleneck limiter.

The RFC that opened #6593 also proposed a bypassCompressionOnRateLimit flag. This repo's open-sse/services/compression/ pipeline is prompt/context compression on the outbound LLM request (chatCore.ts, around the resolveCompressionSettings/selectCompressionStrategy block), not HTTP response compression on synthesized 429 bodies — there is no matching code path for a literal bypass flag. That prompt-compression step also currently runs before withRateLimit() in the request pipeline, so reordering to skip it on a queue-full rejection is a separate, larger change than this issue's scope; it was intentionally not implemented here and is left as a follow-up if the CPU-saving win is worth the reordering risk.

---

6. Slow-stream throughput watchdog (#9709)

The optional resilienceSettings.streamRecovery.throughputWatchdog guard detects an upstream that is still sending chunks but producing assistant output below the configured useful-output rate. It is deliberately distinct from the idle timeout: heartbeats and metadata reset neither timer and do not count as progress. It is also distinct from the hard attempt deadline (#9153), which remains an absolute safety ceiling regardless of output quality.

The watchdog requires a warm-up period followed by a complete rolling window before it can abort. It counts text deltas from Chat Completions and Responses API output events (a conservative UTF-8 byte proxy), ignores usage-only and empty events, and suspends judgement while tool-call or reasoning events are in flight. It is disabled by default and can be enabled with STREAM_THROUGHPUT_WATCHDOG_ENABLED=true; the window, warm-up, minimum rate, and minimum measurable output are bounded by the normal resilience-settings normalization layer.

When enabled, a watchdog abort is applied only to the active upstream attempt. Before any client-visible bytes, the existing same-account early-recovery path may reopen the attempt. After commit, the stream is never blindly replayed; only the existing safe mid-stream continuation contract can stitch a suffix. Finalization remains single-shot, so usage accounting and semaphore release are not duplicated.

---

7. Upstream Status Restatement (misstated quota errors)

Scope: one upstream gateway that reports temporary quota exhaustion with the wrong HTTP status.

Purpose: correct a misleading status BEFORE classification, so downstream consumers (fallback engine, combo aggregation, the client-facing response) see the true retryable nature of the failure.

Some gateways signal TEMPORARY quota exhaustion with a non-retryable HTTP status. agentrouter.org returns 403 (sometimes 400) with a Chinese body (用户额度不足 / 额度不足) instead of the standard 429. Clients like Claude Code treat 403 as permanent and abort the session, and without correction the fallback engine would classify it as AUTH_ERROR instead of a quota event.

Implementation:

• Registry + matcher: open-sse/config/upstreamStatusRestatement.ts — a per-provider list of rules ({id, fromStatuses, toStatus, textMarkers, excludeMarkers, defaultRetryAfterMs}), matched via applyStatusRestatement().
• Call site: the providerFailure: block in open-sse/handlers/chatCore.ts (around line 3654), right after parseUpstreamError() parses an upstream response with an error HTTP status (!providerResponse.ok), and before any classification runs, so every downstream consumer sees the corrected status. Errors embedded inside a 200 SSE stream follow a separate, later stream-parsing path and are not covered by this hook today — a known limitation, not yet needed for agentrouter's misstatus (which surfaces as an error HTTP status).
• Retry eligibility: 429 is in RETRY_AFTER_ELIGIBLE_STATUSES (open-sse/services/combo/unavailableRetryGate.ts), so a restated error carries a real retry window instead of surfacing as a dead 403.
• The synthetic 60s defaultRetryAfterMs (upstreamStatusRestatement.ts) is only what the restated response tells the client; it is not itself the connection's internal cooldown/lockout duration — that is governed separately by whichever mechanism actually handles the restated error (Connection Cooldown's escalating backoff, §2, base 3s for API-key providers; or Model Lockout, §3, for per-model-quota providers like agentrouter). The router can become eligible to retry internally sooner than the 60s window it advertises to the client — intentional headroom, not a bug.

Permanent errors (agentrouter's 无权访问模型 — no access to this model) are NEVER restated: excludeMarkers vetoes the rule even when textMarkers hit, so the error keeps its original status and nothing retries it forever. The matching provider classification rule (agentrouter-model-access-denied in open-sse/config/providerErrorRules.ts: reason: "auth_error", scope: "model", a 6h declared base cooldown) is consulted by checkFallbackError (open-sse/services/accountFallback.ts) before the generic apikey-category FORBIDDEN early-return, gated on honorsRuleLockScope(provider) (#10334 — currently agentrouter-exclusive via the HONORS_RULE_LOCK_SCOPE_PROVIDERS allowlist in providerErrorRules.ts). The rule's declared 6h cooldown flows through as fallbackResult.baseCooldownMs, but it still feeds the pre-existing per-model-quota lockout path (lockModelIfPerModelQuota() / recordModelLockoutFailure(), unchanged by #10334 except for the cooldown source): it is clamped down to the operator's mlSettings.maxCooldownMs (default 1_800_000ms / 30min), like every other model lockout, and the persisted lockout reason stays the pre-existing hardcoded "forbidden", not the rule's "auth_error" — only the cooldown duration is honored end-to-end, not the reason string. The connection itself stays active; sibling models on the same connection are unaffected.

Restated quota errors (额度不足) reach a provider rule in production (agentrouter-user-quota-exhausted: reason: "quota_exhausted", scope: "connection", no declared cooldown of its own — the persistence layer's scaled backoff default applies). Since #10334, scope on ProviderErrorRuleMatch IS consumed end-to-end, but only for providers in the HONORS_RULE_LOCK_SCOPE_PROVIDERS allowlist (providerErrorRules.ts — today only "agentrouter", gated via honorsRuleLockScope()). For every other provider scope remains informational, exactly as before #10334. checkFallbackError surfaces the matched rule's scope as fallbackResult.ruleScope; isAgentrouterConnectionQuotaScope() (src/sse/services/auth.ts) is the shared guard that confirms a ruleScope is genuinely safe to honor as a connection-wide, self-recovering signal (scope "connection", reason quota_exhausted, never permanent, never creditsExhausted — a defense against a future rule pairing scope "connection" with a permanent account state). Two consumers call it:

• Persistence (markAccountUnavailable(), src/sse/services/auth.ts): instead of falling into the passthrough-provider per-model lockout branch (agentrouter is passthroughModels: true → hasPerModelQuota() returns true), it applies a temporary connection cooldown — testStatus: "unavailable" + rateLimitedUntil, never a terminal status (credits_exhausted/banned/expired) — so the connection self-recovers once the cooldown lapses instead of requiring a manual credential reset. Skipped for connections with disableCooling: true (#2997): that opt-out falls through to the per-model lockout instead (a documented trade-off — see the code comment above the branch).
• Same-request combo routing (applyComboTargetExhaustion(), open-sse/services/combo/targetExhaustion.ts): the same guard marks the connection into the in-memory exhaustedConnections set, keyed ${provider}:${connectionId}. This only skips a remaining SAME-REQUEST target that itself already carries that exact connectionId on its own target object (getExhaustedTargetSkipReason(), open-sse/services/combo/comboPredicates.ts, if (provider && connectionId) before the exhaustedConnections lookup) — a plain model-list combo, where sibling targets carry no pinned connectionId of their own and one is only resolved per-dispatch from the response's X-OmniRoute-Selected-Connection-Id header, never hits that key match. For that common case, the real protection against a remaining leg reusing the just-exhausted account is NOT this Set — it is the persistence layer above (the connection's rateLimitedUntil is now in the future) combined with this same guard suppressing transientRateLimitedProviders for the failure (see "Two-stage design" and the code comment on the isAgentrouterConnectionQuotaScope branch in targetExhaustion.ts): with that Set left unmarked, combo.ts's allowRateLimitedConnection force-allow (open-sse/services/combo.ts:1005-1013, :2734-2738) does NOT kick in for the provider's remaining legs, so credential selection's rateLimitedUntil filter (src/sse/services/auth.ts:1238) is honored normally and a remaining leg either picks a different, still-eligible agentrouter connection or fails with no credentials available — it does not force its way back onto the connection this branch just cooled down.

Two-stage design: status restatement, then classification

Status restatement (upstreamStatusRestatement.ts) and provider classification rules (open-sse/config/providerErrorRules.ts, providerRuleRegistry) are separate registries that both key on provider id and text markers, but they run in different places and serve different purposes: restatement rewrites the HTTP status early in chatCore.ts; classification rules pick the fallback reason and lock scope (model / provider / connection) inside checkFallbackError() (open-sse/services/accountFallback.ts).

Classification rules only see full error text (needed to match body markers like 额度不足) for providers listed in the FULL_TEXT_RULE_PROVIDERS allowlist in providerErrorRules.ts — currently only "agentrouter". For every other built-in catalog provider, checkFallbackError hands getProviderErrorRuleMatch only the structured error ({code, type}), which is enough for header/status/code-based rules but blind to body-text markers. The helper resolveRuleMatchBody() performs this selection: full error text for allowlisted providers, the structured error otherwise. Adding a built-in provider to FULL_TEXT_RULE_PROVIDERS is an explicit per-provider opt-in — it exists so that the default path for every provider not on the list stays byte-for-byte unchanged.

A rule's scope (model / provider / connection) is a separate opt-in from FULL_TEXT_RULE_PROVIDERS: checkFallbackError only surfaces it as fallbackResult.ruleScope, and downstream consumers only honor it as anything other than an informational label, for providers in the HONORS_RULE_LOCK_SCOPE_PROVIDERS allowlist in the same file (gated via honorsRuleLockScope() — today only "agentrouter"). See "Restated quota errors" above for what a scope: "connection" match actually does once a provider is on that allowlist.

#11104 — operator-declared rules bypass both allowlists. An operator can declare a per-provider rule at runtime via settings.providerErrorRules (open-sse/config/providerErrorRules.ts::setOperatorProviderErrorRules) without editing this file. Gating an operator rule behind FULL_TEXT_RULE_PROVIDERS/HONORS_RULE_LOCK_SCOPE_PROVIDERS — allowlists meant to protect the default behavior of built-in catalog rules — would make the settings mechanism inert for every provider except the ones already listed there, since declaring the rule is already the operator's explicit opt-in. resolveRuleMatchBody() and honorsRuleLockScope() both check hasOperatorRuleForProvider() first: a provider with an operator rule gets the raw error text and has its declared scope honored, regardless of whether it also appears in either allowlist.

Known gap — providerRuleRegistry is never consulted for HTTP 400. checkFallbackError's BAD_REQUEST branch classifies status 400 entirely through its own pattern arrays (MODEL_ACCESS_DENIED_PATTERNS, CONTEXT_OVERFLOW_PATTERNS, etc. in accountFallback.ts) and returns before the configuredRule/getProviderErrorRuleMatch branch above it is reached. A built-in catalog rule (or an operator rule) with status: 400 is syntactically valid but will never fire. No existing rule targets 400 today, so nothing in production is affected — but a future 400 rule needs this branch touched first, which is a larger change than adding a rule (it reclassifies 400 for every provider already relying on the pattern-array behavior) and is out of scope for a single-provider rule addition.

Adding a new quota-misstating gateway

1. Register one rule array in statusRestatementRegistry (open-sse/config/upstreamStatusRestatement.ts). Keep textMarkers provider-specific; never reuse generic English phrases that collide with CREDITS_EXHAUSTED_SIGNALS (open-sse/services/accountFallback.ts).
2. Optionally register classification rules in open-sse/config/providerErrorRules.ts (providerRuleRegistry) to pick the right lock scope (connection for account-wide quota, model for per-model errors). This step only takes effect in production for providers whose rules need the full error text (body markers): add the provider id to FULL_TEXT_RULE_PROVIDERS in the same file — otherwise checkFallbackError only ever hands the rule the structured {code, type} error and a body-text rule will never match live traffic. Rules that match purely on status/headers (like Opencode's or Minimax's) do not need this opt-in. Separately, if the rule declares scope: "connection" and the intent is an actual connection-wide cooldown plus same-request combo skip (not just an informational label), add the provider id to HONORS_RULE_LOCK_SCOPE_PROVIDERS in the same file — this is what gates isAgentrouterConnectionQuotaScope()-style consumption in markAccountUnavailable() (src/sse/services/auth.ts) and applyComboTargetExhaustion() (open-sse/services/combo/targetExhaustion.ts); without it, scope still flows through fallbackResult.ruleScope but nothing acts on it.
3. Add unit tests mirroring tests/unit/upstream-status-restatement.test.ts and tests/unit/agentrouter-error-rules.test.ts (including the not-permanent / not-creditsExhausted guards, and — if the provider needs the allowlist — a test asserting resolveRuleMatchBody() returns the full text only for that provider).

No changes to chatCore.ts, classifyError, or combo are needed.

Egress-bucketed lock (#10880)

Providers in EGRESS_BUCKETED_LOCK_PROVIDERS (opencode family) are treated as IP-bucketed upstream (the opencode free tier is IP-bucketed, not account-bucketed — see #9611): a status-429 classified quota_exhausted or rate_limit_exceeded cools down every allowlisted-family connection whose last known egress IP matches the failing connection's, before the rotation can try them — avoiding N-1 guaranteed-failed upstream calls (same shape as #10460/#10525). rate_limit_exceeded is included deliberately: on the markAccountUnavailable path the opencode-specific rules never match (no headers/body handed to checkFallbackError, opencode not in FULL_TEXT_RULE_PROVIDERS), so a 429 whose body carries the subscription-quota text ("monthly usage limit reached") is classified quota_exhausted by the quota-text fallback (buildSubscriptionQuotaFallback, accountFallback.ts; 1h cooldown) before the status_429 rule is ever reached — while a quota-text-free 429 (plain rate limiting) classifies via the status_429 rule as rate_limit_exceeded and still cools the IP family down. For an allowlisted provider an IP-bucketed rate limit is the same signal as an exhausted quota. Honest limits:

• Best-effort: the lock resolves the connection's last known egress_ip from proxy_logs (24h window, synchronous, no cache). Cold cache (egress IP never probed) or no row → the failing connection is still cooled by the branch (recorded like today), only no sibling is locked.
• Never terminal: the cooldown is a renewing quota window (testStatus: "unavailable"); a permanent state is never derived from an IP-level signal. disableCooling connections skip the branch entirely.
• Lock granularity changes for the allowlisted family: this is a scope change, not only a sibling optimization. opencode is a passthroughModels provider, so before this branch a 429 produced a per-MODEL lockout; it now produces a connection cooldown — including for an operator running a single connection with no sibling at all. That is the granularity the opencode rule table already declares correct (scope: "connection", providerErrorRules.ts), never honored so far because opencode is not in HONORS_RULE_LOCK_SCOPE_PROVIDERS. The branch writes the failing connection's cooldown + backoffLevel itself, mirroring the connection-scoped agentrouter branch, and returns — the per-model block and the generic path below are never reached.
• Combo included: like the agentrouter branch, the scope deliberately ignores the persistUnavailableState/isCombo downgrade a combo caller applies to a 429. A per-model lockout is not a weaker form of this scope, it is the wrong unit: it says nothing about the exhausted IP, so the combo rotation would keep burning one guaranteed-failed call per sibling.
• Sibling safety: a sibling already terminal (banned/credits_exhausted) or already in a longer cooldown is never overwritten.
• Exclusive allowlist: widening EGRESS_BUCKETED_LOCK_PROVIDERS is an explicit owner decision; no generic wiring (pattern #10334/#10419). The sibling query binds that same allowlist rather than repeating it as a SQL literal, so widening it stays a one-line change.
• Egress IP rotation, both directions: the lookup window (24h) is far wider than the egress-IP cache TTL (5 min), so "last known IP" is history, not current state. If a connection's proxy rotated within the window the lock may miss a genuinely shared IP (the recorded IP is the new, unexhausted one) — and symmetrically it may cool a sibling that has since rotated away from the exhausted IP. The second case costs that sibling one cooldown window; both are accepted best-effort limits of a history-based lookup.
• Cost: two bounded scans of proxy_logs (window-filtered via idx_pl_timestamp), only at 429 frequency. No new index (migration 134 YAGNI). Measured on a real-traffic DB copy of moderate size; a high-throughput instance holds proportionally more rows in the same window.

---

Other Resilience Features

• 19 routing strategies (priority, weighted, round-robin, context-relay, fill-first, p2c, random, least-used, cost-optimized, reset-aware, reset-window, headroom, strict-random, auto, lkgp, context-optimized, cache-optimized, fusion, pipeline) — see AUTO-COMBO.md.
• Reset-aware routing (v3.8.0) — prioritizes connections by quota reset time.
• Background mode degradation — Responses API background: true degraded to sync with warning.
• Dynamic tool limit detection — backs off providers when tool count limits hit.
• Emergency fallback — controlled by OMNIROUTE_EMERGENCY_FALLBACK; operators can override it from the Feature Flags page without a restart.

---

Debugging

• Weighted combo answers 503 all_targets_cooling_down (Retry-After set, diagnostics.excluded lists every target with model_lockout / circuit_open / provider_cooldown / unavailable) → the pool is configured and connected, every target is just excluded by a resilience timer; the [COMBO] Weighted selection: every target excluded before dispatch — … warning names the reasons and remaining seconds. A 404 no_executable_targets from the same combo means no resilience timer was involved (nothing to run, or every account failed the availability probe). Built in open-sse/services/combo/pinRecovery.ts from the exclusions collected in targetResolution.ts.
• All keys for a provider skipped → check both circuit breaker state AND each connection's rateLimitedUntil/testStatus.
• Provider permanently excluded after reset window → code reading raw state instead of getStatus()/canExecute().
• One key fails, others should work → prefer connection cooldown over circuit breaker.
• Only one model fails → prefer model lockout over connection cooldown.
• State should self-recover but doesn't → check for future timestamp + read path that refreshes expired state. Permanent statuses require manual changes.

---

TLS Fingerprinting & Stealth

Provider-specific stealth (JA3/JA4, CCH, obfuscation) is separately documented — see docs/security/STEALTH_GUIDE.md (git; not compiled into /docs).

---

Resilience testing (Phase 8 · Block C)

Beyond unit tests for resilience logic, three tests exercise the runtime under real stress/failure conditions (all integration/nightly — none block PRs):

Test	What	Run
Chaos	Fake-upstream node injects real latency/reset/timeout/503; validates that the circuit breaker opens/recovers and checkFallbackError classifies 503 as recoverable fallback.	RUN_CHAOS_INT=1 npm run test:chaos
Heap-growth	~500 streams per createSSEStream under --expose-gc; fails if the heap grows beyond the ceiling (OOM guard #3069).	npm run test:heap
k6 soak	Sustained load against /api/monitoring/health; p95/error thresholds.	k6 run tests/load/k6-soak.js (nightly)

Orchestrated by .github/workflows/nightly-resilience.yml (cron + dispatch). In the default test:integration, chaos and heap self-skip (without RUN_CHAOS_INT/--expose-gc).

---

See Also

• Architecture Guide — System architecture and internals
• User Guide — Providers, combos, CLI integration
• Auto-Combo Engine — 16-factor scoring, mode packs