pjandro/OmniRoute
1
0
Fork 0
mirror of https://github.com/diegosouzapw/OmniRoute.git synced 2026-07-26 09:52:11 +03:00
Code Issues Packages Projects Releases Wiki Activity

refactor: unify resilience controls (#1449)

Browse Source 
Integrated into release/v3.7.0
This commit is contained in:
Randi
2026-04-21 03:13:28 -04:00
committed by GitHub
parent 9311f2a627
commit b38e57d452
68 changed files with 3778 additions and 3832 deletions
Download Patch File Download Diff File Expand all files Collapse all files

68
README.md
View File

@@ -324,12 +324,13 @@ AI providers can become unstable, return 5xx errors, or hit temporary rate limit
**How OmniRoute solves it:**
- **Settings-Driven Lock Hierarchy** — Provider profiles control default account/model lockouts, global model quarantine, and provider circuit breakers from one control surface, while explicit upstream `Retry-After` windows still take priority
- **Exponential Backoff** — Progressive retry delays for both account/model lockouts and higher-level quarantine
- **Request Queue & Pacing** — Per-connection request buckets smooth bursts before they hit upstream rate caps
- **Connection Cooldown** — A single connection cools down after retryable failures with optional upstream `Retry-After` hints and exponential backoff
- **Provider Circuit Breaker** — The provider only trips after fallback is exhausted and the provider request still fails with provider-wide transient errors; connection-scoped `429` rate limits stay in Connection Cooldown
- **Wait For Cooldown** — The server can wait for the earliest connection cooldown to expire and retry the same client request automatically
- **Anti-Thundering Herd** — Mutex + semaphore protection against concurrent retry storms
- **Combo Fallback Chains** — If the primary provider fails, automatically falls through the chain with no intervention
- **Combo Circuit Breaker** — Auto-disables failing providers within a combo chain
- **Health Dashboard** — Uptime monitoring, circuit breaker states, lockouts, cache stats, p50/p95/p99 latency
- **Health Dashboard** — Uptime monitoring, provider circuit breaker states, cooldowns, cache stats, p50/p95/p99 latency
</details>
@@ -470,7 +471,7 @@ As request volume grows, without caching the same questions generate duplicate c
- **Semantic Cache** — Two-tier cache (signature + semantic) reduces cost and latency
- **Request Idempotency** — 5s deduplication window for identical requests
- **Rate Limit Detection** — Per-provider RPM, min gap, and max concurrent tracking
- **Editable Rate Limits** — Configurable defaults in Settings → Resilience with persistence
- **Request Queue & Pacing** — Configurable queue, pacing, and concurrency defaults in Settings → Resilience
- **API Key Validation Cache** — 3-tier cache for production performance
- **Health Dashboard with Telemetry** — p50/p95/p99 latency, cache stats, uptime
@@ -567,8 +568,8 @@ Teams need quick runtime changes during incidents or cost events.
**How OmniRoute solves it:**
- Switch combo activation directly from MCP dashboard
- Apply resilience profiles from pre-defined policy packs
- Reset circuit breaker state from the same operations panel
- Tune queue, cooldown, breaker, and wait settings from the dedicated Resilience page
- Review live provider breaker state from the Health dashboard
</details>
@@ -1352,7 +1353,7 @@ OmniRoute v3.6 is built as an operational platform, not just a relay proxy.
| 🔢 **Hybrid Token Counting** | Uses provider-side `/messages/count_tokens` when available; falls back to estimation — accurate usage tracking without guessing |
| 🌱 **Model Alias Auto-Seed** | 30+ cross-proxy dialect aliases normalised at startup — no more routing mismatches |
| 🛡️ **Safe Outbound Fetch** | All provider validation and model discovery go through a guarded fetch layer blocking private/local URLs with retry, timeout, and SSRF protection |
| 🔄 **Cooldown-Aware Retries** | Chat requests auto-retry on model-scoped cooldowns with configurable `requestRetry` and `maxRetryIntervalSec` |
| ⏳ **Wait For Cooldown** | Server-side chat retries when every candidate connection is cooling down; configurable `enabled`, `maxRetries`, and `maxRetryWaitSec` |
| 🔍 **Runtime Env Validation** | Startup validates all env vars with Zod schemas — clear errors for missing secrets, invalid URLs, or wrong types |
| 📋 **Compliance Audit Expansion** | Structured audit logs with pagination, request context, auth events, provider CRUD events, and SSRF-blocked validation logging |
| 🔐 **TPS Log Metric** | Log details modal shows Tokens Per Second (TPS) — quick performance at-a-glance for every request |
@@ -1406,7 +1407,7 @@ OmniRoute v3.6 is built as an operational platform, not just a relay proxy.
| 📡 **A2A Task Lifecycle Management** | List/filter tasks, inspect events/artifacts, cancel running tasks |
| 📋 **Agent Card Discovery** | `/.well-known/agent.json` for client auto-discovery |
| 🧪 **Protocol E2E Test Harness** | Real MCP SDK + A2A client flows in `test:protocols:e2e` |
| ⚙️ **Operational Controls** | Switch combo, apply resilience profiles, reset breakers from one control surface |
| ⚙️ **Operational Controls** | Switch combos, tune resilience settings, and review breaker state from dedicated Health and Settings surfaces |
### 🧠 Routing & Intelligence
@@ -1446,29 +1447,30 @@ OmniRoute v3.6 is built as an operational platform, not just a relay proxy.
### 🛡️ Resilience, Security & Governance
| Feature | What It Does |
| ----------------------------------- | --------------------------------------------------------------------------------------- |
| 🔌 **Circuit Breakers** | Per-provider and per-model trip/recover with 10-minute cooldowns |
| 🔒 **Daily Quota Lock** 🆕 | Detects exhaustion signals and locks routing for the specific model until midnight |
| 🎯 **Endpoint-Aware Models** | Custom models declare supported endpoints + API format |
| 🛡️ **Anti-Thundering Herd** | Mutex + semaphore protections on retry/rate events |
| 🧠 **Semantic + Signature Cache** | Cost/latency reduction with two cache layers |
| ⚡ **Request Idempotency** | Duplicate protection window |
| 🔒 **TLS Fingerprint Spoofing** | Browser-like TLS fingerprint — **reduces bot detection and account flagging** |
| 🔏 **CLI Fingerprint Matching** | Matches native CLI request signatures — **reduces ban risk while preserving proxy IP** |
| 🌐 **IP Filtering** | Allowlist/blocklist control for exposed deployments |
| 📊 **Editable Rate Limits** | Configurable global/provider-level limits with persistence |
| 📉 **Graceful Degradation** | Multi-layer capability fallbacks protecting core gateway operations |
| 📜 **Config Audit Trail** | Diff-based change tracking preventing operational drift with simple rollbacks |
| ⏳ **Provider Health Sync** | Proactive token expiration monitoring triggering alerts before authorization failures |
| 🚪 **Auto-Disable Banned Accounts** | Operational circuit breaker sealing permanently blocked token accounts automatically |
| 🔑 **API Key Management + Scoping** | Secure key issuance/rotation and model/provider controls |
| 👁️ **Scoped API Key Reveal** 🆕 | Opt-in recovery of API keys via `ALLOW_API_KEY_REVEAL` |
| 🛡️ **Protected `/models`** | Optional auth gating and provider hiding for model catalog |
| 🛡️ **Safe Outbound Fetch** 🆕 | Guarded fetch for provider calls — blocks private/local URLs, retries, SSRF protection |
| 🔄 **Cooldown-Aware Retries** 🆕 | Auto-retry chat on model cooldowns; configurable `requestRetry` / `maxRetryIntervalSec` |
| 🔍 **Runtime Env Validation** 🆕 | Zod-based env schema validation at startup with actionable error messages |
| 📋 **Compliance Audit v2** 🆕 | Pagination, request context, auth events, provider CRUD, and SSRF-blocked logging |
| Feature | What It Does |
| ----------------------------------- | ------------------------------------------------------------------------------------------------------- |
| 🔌 **Provider Circuit Breakers** | Provider-wide trip/recover after fallback exhaustion with configurable thresholds |
| 🔒 **Daily Quota Lock** 🆕 | Detects exhaustion signals and locks routing for the specific model until midnight |
| 🎯 **Endpoint-Aware Models** | Custom models declare supported endpoints + API format |
| 🛡️ **Anti-Thundering Herd** | Mutex + semaphore protections on retry/rate events |
| 🧠 **Semantic + Signature Cache** | Cost/latency reduction with two cache layers |
| ⚡ **Request Idempotency** | Duplicate protection window |
| 🔒 **TLS Fingerprint Spoofing** | Browser-like TLS fingerprint — **reduces bot detection and account flagging** |
| 🔏 **CLI Fingerprint Matching** | Matches native CLI request signatures — **reduces ban risk while preserving proxy IP** |
| 🌐 **IP Filtering** | Allowlist/blocklist control for exposed deployments |
| 🚦 **Request Queue & Pacing** | Configurable per-connection request buckets for RPM, spacing, concurrency, and max wait |
| 📉 **Graceful Degradation** | Multi-layer capability fallbacks protecting core gateway operations |
| 📜 **Config Audit Trail** | Diff-based change tracking preventing operational drift with simple rollbacks |
| ⏳ **Provider Health Sync** | Proactive token expiration monitoring triggering alerts before authorization failures |
| ❄️ **Connection Cooldown** | Retryable 408/429/5xx failures cool down a single connection with optional upstream hints |
| 🚪 **Auto-Disable Banned Accounts** | Permanently blocked token accounts can be disabled automatically |
| 🔑 **API Key Management + Scoping** | Secure key issuance/rotation and model/provider controls |
| 👁️ **Scoped API Key Reveal** 🆕 | Opt-in recovery of API keys via `ALLOW_API_KEY_REVEAL` |
| 🛡️ **Protected `/models`** | Optional auth gating and provider hiding for model catalog |
| 🛡️ **Safe Outbound Fetch** 🆕 | Guarded fetch for provider calls — blocks private/local URLs, retries, SSRF protection |
| ⏳ **Wait For Cooldown** 🆕 | Auto-retry chat after connection cooldowns; configurable `enabled`, `maxRetries`, and `maxRetryWaitSec` |
| 🔍 **Runtime Env Validation** 🆕 | Zod-based env schema validation at startup with actionable error messages |
| 📋 **Compliance Audit v2** 🆕 | Pagination, request context, auth events, provider CRUD, and SSRF-blocked logging |
### 📊 Observability & Analytics
@@ -2283,7 +2285,7 @@ OmniRoute has **218+ features planned** across multiple development phases. Here
| 🧠 **Routing & Intelligence** | 25+ | Lowest-latency routing, tag-based routing, quota preflight, quota-aware P2C, step-based combo routing |
| 🔒 **Security & Compliance** | 20+ | SSRF hardening, credential cloaking, rate-limit per endpoint, management key scoping |
| 📊 **Observability** | 15+ | OpenTelemetry integration, real-time quota monitoring, combo target health, cost tracking per model |
| 🔄 **Provider Integrations** | 20+ | Dynamic model registry, provider cooldowns, multi-account Codex, Copilot quota parsing |
| 🔄 **Provider Integrations** | 20+ | Dynamic model registry, connection cooldowns, multi-account Codex, Copilot quota parsing |
| ⚡ **Performance** | 15+ | Dual cache layer, prompt cache, response cache, streaming keepalive, batch API |
| 🌐 **Ecosystem** | 10+ | WebSocket API, config hot-reload, distributed config store, commercial mode |

31
docs/API_REFERENCE.md
View File

@@ -284,12 +284,12 @@ GET response includes `agents[]` (id, name, binary, version, installed, protocol
### Resilience & Rate Limits
| Endpoint | Method | Description |
| ----------------------- | --------- | ------------------------------- |
| `/api/resilience` | GET/PATCH | Get/update resilience profiles |
| `/api/resilience/reset` | POST | Reset circuit breakers |
| `/api/rate-limits` | GET | Per-account rate limit status |
| `/api/rate-limit` | GET | Global rate limit configuration |
| Endpoint | Method | Description |
| ----------------------- | --------- | ---------------------------------------------------------------------------------- |
| `/api/resilience` | GET/PATCH | Get/update request queue, connection cooldown, provider breaker, and wait settings |
| `/api/resilience/reset` | POST | Reset provider circuit breakers |
| `/api/rate-limits` | GET | Per-account rate limit status |
| `/api/rate-limit` | GET | Global rate limit configuration |
### Evals
@@ -443,25 +443,6 @@ Content-Type: application/json
}
```
---
## Model Availability
```bash
# Get real-time model availability across all providers
GET /api/models/availability
# Check availability for a specific model
POST /api/models/availability
Content-Type: application/json
{
"model": "claude-sonnet-4-5-20250929"
}
```
---
## Request Processing
1. Client sends request to `/v1/*`

14
docs/ARCHITECTURE.md
View File

@@ -42,7 +42,7 @@ Core capabilities:
- Circuit breaker pattern for provider resilience
- Anti-thundering herd protection with mutex locking
- Signature-based request deduplication cache
- Domain layer: model availability, cost rules, fallback policy, lockout policy
- Domain layer: cost rules, fallback policy, lockout policy
- Context Relay: session handoff summaries for account rotation continuity
- Domain state persistence (SQLite write-through cache for fallbacks, budgets, lockouts, circuit breakers)
- Policy engine for centralized request evaluation (lockout → budget → fallback)
@@ -51,7 +51,7 @@ Core capabilities:
- Correlation ID (X-Request-Id) for end-to-end tracing
- Compliance audit logging with opt-out per API key
- Eval framework for LLM quality assurance
- Resilience UI dashboard with real-time circuit breaker status
- Health dashboard with real-time provider circuit breaker status
- MCP Server (25 tools) with 3 transports (stdio/SSE/Streamable HTTP)
- A2A Server (JSON-RPC 2.0 + SSE) with skills and task lifecycle
- Memory system (extraction, injection, retrieval, summarization)
@@ -205,10 +205,9 @@ Management domains:
- System prompt: `src/app/api/settings/system-prompt` (GET/PUT)
- Sessions: `src/app/api/sessions` (GET)
- Rate limits: `src/app/api/rate-limits` (GET)
- Resilience: `src/app/api/resilience` (GET/PATCH) — provider profiles, circuit breaker, rate limit state
- Resilience reset: `src/app/api/resilience/reset` (POST) — reset breakers + cooldowns
- Resilience: `src/app/api/resilience` (GET/PATCH) — request queue, connection cooldown, provider breaker, wait-for-cooldown config
- Resilience reset: `src/app/api/resilience/reset` (POST) — reset provider breakers
- Cache stats: `src/app/api/cache/stats` (GET/DELETE)
- Model availability: `src/app/api/models/availability` (GET/POST)
- Telemetry: `src/app/api/telemetry/summary` (GET)
- Budget: `src/app/api/usage/budget` (GET/POST)
- Fallback chains: `src/app/api/fallback/chains` (GET/POST/DELETE)
@@ -265,7 +264,6 @@ Services (business logic):
Domain layer modules:
- Model availability: `src/lib/domain/modelAvailability.ts`
- Cost rules/budgets: `src/lib/domain/costRules.ts`
- Fallback policy: `src/lib/domain/fallbackPolicy.ts`
- Combo resolver: `src/lib/domain/comboResolver.ts`
@@ -794,7 +792,7 @@ legacy compatibility. The current runtime contract uses:
## 1) Account/Provider Availability
- provider account cooldown on transient/rate/auth errors
- connection cooldown on retryable upstream failures
- account fallback before failing request
- combo model fallback when current model/provider path is exhausted
@@ -876,7 +874,7 @@ Environment variables actively used by code:
5. The `open-sse/` directory is published as the `@omniroute/open-sse` **npm workspace package**. Source code imports it via `@omniroute/open-sse/...` (resolved by Next.js `transpilePackages`). File paths in this document still use the directory name `open-sse/` for consistency.
6. Charts in the dashboard use **Recharts** (SVG-based) for accessible, interactive analytics visualizations (model usage bar charts, provider breakdown tables with success rates).
7. E2E tests use **Playwright** (`tests/e2e/`), run via `npm run test:e2e`. Unit tests use **Node.js test runner** (`tests/unit/`), run via `npm run test:unit`. Source code under `src/` is **TypeScript** (`.ts`/`.tsx`); the `open-sse/` workspace remains JavaScript (`.js`).
8. Settings page is organized into 5 tabs: Security, Routing (6 global strategies: fill-first, round-robin, p2c, random, least-used, cost-optimized), Resilience (editable rate limits, circuit breaker, policies, **Context Relay** handoff config), AI (thinking budget, system prompt, prompt cache), Advanced (proxy).
8. Settings page is organized into 7 tabs: General, Appearance, AI, Security, Routing, Resilience, Advanced. The Resilience page only configures request queue, connection cooldown, provider breaker, and wait-for-cooldown behavior; live breaker runtime state is shown on the Health page.
9. **Context Relay** strategy (`context-relay`) is split across two layers: `combo.ts` decides if a handoff should be generated, `chat.ts` injects the handoff after account resolution. Handoff data lives in `context_handoffs` SQLite table. This split is intentional because only `chat.ts` knows whether the actual account changed.
10. **Proxy enforcement** is now comprehensive: `tokenHealthCheck.ts` resolves proxy per connection, `/api/providers/validate` uses `runWithProxyContext`, and `proxyFetch.ts` uses `undici.fetch()` to maintain dispatcher compatibility on Node 22.
11. **Node.js runtime policy detection**: `/api/settings/require-login` returns `nodeVersion` and `nodeCompatible` fields. The login page renders a warning banner when the runtime falls outside the supported secure Node.js lines.

55
docs/USER_GUIDE.md
View File

@@ -756,35 +756,34 @@ Chain: production-fallback
Configure via **Dashboard → Settings → Resilience**.
OmniRoute implements provider-level resilience with four components:
OmniRoute implements provider-level resilience with five components:
1. **Provider Profiles** — Per-provider configuration for:
- **Transient Cooldown** — Base cooldown for transient upstream failures
- **Rate Limit Cooldown** — Base cooldown for `429`-driven lockouts
- **Max Backoff Level** — Maximum exponential backoff level for repeated failures
- **CB Threshold** — Failure count before model quarantine / provider circuit breaker escalates
- **CB Reset Time** — Failure counting window and breaker reset timer
2. **Editable Rate Limits** — System-level defaults configurable in the dashboard:
1. **Request Queue & Pacing** — System-level request shaping:
- **Requests Per Minute (RPM)** — Maximum requests per minute per account
- **Min Time Between Requests** — Minimum gap in milliseconds between requests
- **Max Concurrent Requests** — Maximum simultaneous requests per account
- Click **Edit** to modify, then **Save** or **Cancel**. Values persist via the resilience API.
3. **Circuit Breaker** — Tracks failures per provider and automatically opens the circuit when the configured threshold is reached:
2. **Connection Cooldown** — Per-auth-type configuration for a single connection after retryable failures:
- **Base Cooldown** — Default cooldown window for retryable upstream failures
- **Use Upstream Retry Hints** — Honors authoritative `Retry-After` or reset hints when provided
- **Max Backoff Steps** — Maximum exponential backoff level for repeated failures
3. **Provider Circuit Breaker** — Tracks end-to-end provider failures and automatically opens the breaker when the configured threshold is reached:
- **Failure Threshold** — Consecutive provider failures before opening the breaker
- **Reset Timeout** — Time window before the provider is tested again
- **CLOSED** (Healthy) — Requests flow normally
- **OPEN** — Provider is temporarily blocked after repeated failures
- **HALF_OPEN** — Testing if provider has recovered
The same provider profile also drives model-scoped lockouts:
- Account/model lockouts react immediately to authoritative `429` / `404` signals and use the configured cooldown + backoff values
- Global provider/model quarantine only activates after repeated exhaustion hits the configured **CB Threshold** within **CB Reset Time**
Connection-scoped `429` rate limits stay in **Connection Cooldown** and do not count toward the provider breaker.
4. **Policies & Locked Identifiers** — Shows circuit breaker status and locked identifiers with force-unlock capability.
The provider breaker runtime state is shown on **Dashboard → Health** only.
5. **Rate Limit Auto-Detection** — Monitors `429` and `Retry-After` headers to proactively avoid hitting provider rate limits. When an upstream provider returns an explicit wait window, that authoritative `Retry-After` value overrides the base cooldown from the provider profile.
4. **Wait For Cooldown** — If every candidate connection is already cooling down, OmniRoute can wait for the earliest cooldown and retry the same client request automatically.
**Pro Tip:** Use **Reset All** button to clear all circuit breakers and cooldowns when a provider recovers from an outage.
5. **Rate Limit Auto-Detection** — When upstream providers return explicit wait windows, those hints override the local connection cooldown when the setting is enabled.
**Pro Tip:** Use the **Health** page to inspect and reset live provider breakers after an outage. The Resilience page only changes configuration.
---
@@ -824,14 +823,14 @@ curl -X POST http://localhost:20128/api/db-backups/import \
The settings page is organized into 6 tabs for easy navigation:
| Tab | Contents |
| -------------- | ---------------------------------------------------------------------------------------------- |
| **General** | System storage tools, appearance settings, theme controls, and per-item sidebar visibility |
| **Security** | Login/Password settings, IP Access Control, API auth for `/models`, and Provider Blocking |
| **Routing** | Global routing strategy (6 options), wildcard model aliases, fallback chains, combo defaults |
| **Resilience** | Provider profiles, editable rate limits, circuit breaker status, policies & locked identifiers |
| **AI** | Thinking budget configuration, global system prompt injection, prompt cache stats |
| **Advanced** | Global proxy configuration (HTTP/SOCKS5) |
| Tab | Contents |
| -------------- | -------------------------------------------------------------------------------------------- |
| **General** | System storage tools, appearance settings, theme controls, and per-item sidebar visibility |
| **Security** | Login/Password settings, IP Access Control, API auth for `/models`, and Provider Blocking |
| **Routing** | Global routing strategy (6 options), wildcard model aliases, fallback chains, combo defaults |
| **Resilience** | Request queue, connection cooldown, provider breaker config, and wait-for-cooldown behavior |
| **AI** | Thinking budget configuration, global system prompt injection, prompt cache stats |
| **Advanced** | Global proxy configuration (HTTP/SOCKS5) |
---
@@ -904,9 +903,9 @@ Access via **Dashboard → Health**. Real-time system health overview with 6 car
| Card | What It Shows |
| --------------------- | ----------------------------------------------------------- |
| **System Status** | Uptime, version, memory usage, data directory |
| **Provider Health** | Per-provider circuit breaker state (Closed/Open/Half-Open) |
| **Rate Limits** | Active rate limit cooldowns per account with remaining time |
| **Active Lockouts** | Providers temporarily blocked by the lockout policy |
| **Provider Health** | Global provider circuit breaker runtime state |
| **Rate Limits** | Active connection cooldowns per account with remaining time |
| **Active Lockouts** | Active model-scoped lockouts and temporary exclusions |
| **Signature Cache** | Deduplication cache stats (active keys, hit rate) |
| **Latency Telemetry** | p50/p95/p99 latency aggregation per provider |

26
docs/openapi.yaml
View File

@@ -460,22 +460,6 @@ paths:
"200":
description: Complete catalog with all providers
/api/models/availability:
get:
tags: [Models]
summary: Get model availability status
description: Returns availability data for all configured models across providers.
responses:
"200":
description: Model availability map
post:
tags: [Models]
summary: Refresh model availability
description: Triggers a re-check of model availability across all providers.
responses:
"200":
description: Availability refreshed
# ─── Management Endpoints ──────────────────────────────────────
/api/providers:
@@ -1844,13 +1828,13 @@ paths:
/api/resilience:
get:
tags: [System]
summary: Get resilience profiles and circuit breaker states
summary: Get resilience configuration
responses:
"200":
description: Resilience configuration and current states
put:
description: Request queue, connection cooldown, provider breaker, and wait settings
patch:
tags: [System]
summary: Update resilience profiles
summary: Update resilience configuration
requestBody:
required: true
content:
@@ -1859,7 +1843,7 @@ paths:
type: object
responses:
"200":
description: Updated resilience profiles
description: Updated resilience configuration
/api/resilience/reset:
post:

67
open-sse/handlers/chatCore.ts
View File

@@ -14,13 +14,6 @@ import { createRequestLogger } from "../utils/requestLogger.ts";
import { getModelTargetFormat, PROVIDER_ID_TO_ALIAS } from "../config/providerModels.ts";
import { resolveModelAlias } from "../services/modelDeprecation.ts";
import { getUnsupportedParams } from "../config/providerRegistry.ts";
import {
hasPerModelQuota,
lockModelIfPerModelQuota,
isDailyQuotaExhausted,
getMsUntilTomorrow,
} from "../services/accountFallback.ts";
import { COOLDOWN_MS } from "../config/constants.ts";
import {
buildErrorBody,
createErrorResult,
@@ -2080,66 +2073,6 @@ export async function handleChatCore({
console.warn(
`[provider] Node ${connectionId} account deactivated (${statusCode}) — disabling permanently`
);
} else if (errorType === PROVIDER_ERROR_TYPES.RATE_LIMITED) {
// For providers with per-model quotas (passthrough providers, Gemini),
// each model has independent quota. A 429 on one model must NOT lock out
// the entire connection — other models may still have quota available.
const rateLimitCooldownMs = retryAfterMs || COOLDOWN_MS.rateLimit;
if (
lockModelIfPerModelQuota(
provider,
connectionId,
model,
"rate_limited",
rateLimitCooldownMs
)
) {
console.warn(
`[provider] Node ${connectionId} model-only rate limited (${statusCode}) for ${model} - ${Math.ceil(rateLimitCooldownMs / 1000)}s (connection stays active)`
);
} else {
const rateLimitedUntil = new Date(Date.now() + rateLimitCooldownMs).toISOString();
await updateProviderConnection(connectionId, {
rateLimitedUntil: rateLimitedUntil,
testStatus: "unavailable",
lastErrorType: errorType,
lastError: message,
errorCode: statusCode,
healthCheckInterval: null,
lastHealthCheckAt: null,
});
console.warn(
`[provider] Node ${connectionId} rate limited (${statusCode}) - Next available at ${rateLimitedUntil}`
);
}
} else if (errorType === PROVIDER_ERROR_TYPES.QUOTA_EXHAUSTED) {
// Providers with per-model quotas — lock the model only, not the connection
// Daily quota exhausted: lock until tomorrow; otherwise use default cooldown
const isDailyQuota = isDailyQuotaExhausted(message);
const quotaCooldownMs = isDailyQuota
? getMsUntilTomorrow()
: retryAfterMs || COOLDOWN_MS.rateLimit;
if (
lockModelIfPerModelQuota(
provider,
connectionId,
model,
isDailyQuota ? "daily_quota_exhausted" : "quota_exhausted",
quotaCooldownMs
)
) {
console.warn(
`[provider] Node ${connectionId} ${isDailyQuota ? "daily " : ""}quota exhausted (${statusCode}) for ${model} - ${Math.ceil(quotaCooldownMs / 1000)}s (connection stays active)`
);
} else {
await updateProviderConnection(connectionId, {
testStatus: "credits_exhausted",
lastErrorType: errorType,
lastError: message,
errorCode: statusCode,
});
console.warn(`[provider] Node ${connectionId} exhausted quota (${statusCode})`);
}
} else if (errorType === PROVIDER_ERROR_TYPES.ACCOUNT_DEACTIVATED) {
await updateProviderConnection(connectionId, {
isActive: false,

151
open-sse/mcp-server/tools/advancedTools.ts
View File

@@ -112,97 +112,121 @@ interface BudgetGuardState {
let activeBudgetGuard: BudgetGuardState | null = null;
type ResilienceProfileConfig = {
profiles: {
requestQueue: {
requestsPerMinute: number;
minTimeBetweenRequestsMs: number;
concurrentRequests: number;
};
connectionCooldown: {
oauth: {
transientCooldown: number;
rateLimitCooldown: number;
maxBackoffLevel: number;
circuitBreakerThreshold: number;
circuitBreakerReset: number;
baseCooldownMs: number;
useUpstreamRetryHints: boolean;
maxBackoffSteps: number;
};
apikey: {
transientCooldown: number;
rateLimitCooldown: number;
maxBackoffLevel: number;
circuitBreakerThreshold: number;
circuitBreakerReset: number;
baseCooldownMs: number;
useUpstreamRetryHints: boolean;
maxBackoffSteps: number;
};
};
defaults: {
requestsPerMinute: number;
minTimeBetweenRequests: number;
concurrentRequests: number;
providerBreaker: {
oauth: {
failureThreshold: number;
resetTimeoutMs: number;
};
apikey: {
failureThreshold: number;
resetTimeoutMs: number;
};
};
};
const RESILIENCE_PROFILES = {
aggressive: {
profiles: {
requestQueue: {
requestsPerMinute: 180,
minTimeBetweenRequestsMs: 100,
concurrentRequests: 16,
},
connectionCooldown: {
oauth: {
transientCooldown: 3000,
rateLimitCooldown: 30000,
maxBackoffLevel: 4,
circuitBreakerThreshold: 2,
circuitBreakerReset: 30000,
baseCooldownMs: 30000,
useUpstreamRetryHints: false,
maxBackoffSteps: 4,
},
apikey: {
transientCooldown: 2000,
rateLimitCooldown: 0,
maxBackoffLevel: 3,
circuitBreakerThreshold: 3,
circuitBreakerReset: 15000,
baseCooldownMs: 2000,
useUpstreamRetryHints: true,
maxBackoffSteps: 3,
},
},
defaults: {
requestsPerMinute: 180,
minTimeBetweenRequests: 100,
concurrentRequests: 16,
providerBreaker: {
oauth: {
failureThreshold: 2,
resetTimeoutMs: 30000,
},
apikey: {
failureThreshold: 3,
resetTimeoutMs: 15000,
},
},
},
balanced: {
profiles: {
requestQueue: {
requestsPerMinute: 100,
minTimeBetweenRequestsMs: 200,
concurrentRequests: 10,
},
connectionCooldown: {
oauth: {
transientCooldown: 5000,
rateLimitCooldown: 60000,
maxBackoffLevel: 8,
circuitBreakerThreshold: 3,
circuitBreakerReset: 60000,
baseCooldownMs: 60000,
useUpstreamRetryHints: false,
maxBackoffSteps: 8,
},
apikey: {
transientCooldown: 3000,
rateLimitCooldown: 0,
maxBackoffLevel: 5,
circuitBreakerThreshold: 5,
circuitBreakerReset: 30000,
baseCooldownMs: 3000,
useUpstreamRetryHints: true,
maxBackoffSteps: 5,
},
},
defaults: {
requestsPerMinute: 100,
minTimeBetweenRequests: 200,
concurrentRequests: 10,
providerBreaker: {
oauth: {
failureThreshold: 3,
resetTimeoutMs: 60000,
},
apikey: {
failureThreshold: 5,
resetTimeoutMs: 30000,
},
},
},
conservative: {
profiles: {
requestQueue: {
requestsPerMinute: 60,
minTimeBetweenRequestsMs: 350,
concurrentRequests: 6,
},
connectionCooldown: {
oauth: {
transientCooldown: 8000,
rateLimitCooldown: 120000,
maxBackoffLevel: 10,
circuitBreakerThreshold: 8,
circuitBreakerReset: 120000,
baseCooldownMs: 120000,
useUpstreamRetryHints: false,
maxBackoffSteps: 10,
},
apikey: {
transientCooldown: 5000,
rateLimitCooldown: 30000,
maxBackoffLevel: 8,
circuitBreakerThreshold: 8,
circuitBreakerReset: 60000,
baseCooldownMs: 30000,
useUpstreamRetryHints: false,
maxBackoffSteps: 8,
},
},
defaults: {
requestsPerMinute: 60,
minTimeBetweenRequests: 350,
concurrentRequests: 6,
providerBreaker: {
oauth: {
failureThreshold: 8,
resetTimeoutMs: 120000,
},
apikey: {
failureThreshold: 8,
resetTimeoutMs: 60000,
},
},
},
} satisfies Record<"aggressive" | "balanced" | "conservative", ResilienceProfileConfig>;
@@ -460,13 +484,10 @@ export async function handleSetResilienceProfile(args: {
};
}
// Apply to OmniRoute via API (contract: PATCH + { profiles, defaults })
// Apply to OmniRoute via API using the plan-aligned resilience structure.
await apiFetch("/api/resilience", {
method: "PATCH",
body: JSON.stringify({
profiles: settings.profiles,
defaults: settings.defaults,
}),
body: JSON.stringify(settings),
});
const result = { applied: true, profile: args.profile, settings };

8
open-sse/services/AGENTS.md
View File

@@ -8,9 +8,9 @@
### Combo Routing Engine
- **`combo.ts`** (800 LOC) — Entry point for multi-model routing. **`handleComboChat()`** iterates through targets in order until success or all fail. **`resolveComboTargets()`** expands combo config into ordered `ResolvedComboTarget[]` (provider + model + account + credentials). Enforces per-target circuit breaker and fallback logic.
- **`combo.ts`** (800 LOC) — Entry point for multi-model routing. **`handleComboChat()`** iterates through targets in order until success or all fail. **`resolveComboTargets()`** expands combo config into ordered `ResolvedComboTarget[]` (provider + model + account + credentials). Enforces target retry, round-robin slot control, and provider-level resilience gates.
- **Strategies** (13 total): `priority` (ordered list), `weighted` (probabilistic), `fill-first` (fill quota first), `round-robin`, `P2C` (power of two choices), `random`, `least-used`, `cost-optimized`, `strict-random`, `auto`, `lkgp` (last known good provider), `context-optimized`, `context-relay`.
- **Circuit Breaker**: Per target, tracks consecutive failures; breaks after threshold, reopens on success or timeout.
- **Provider Breaker Integration**: Combo targets respect the global provider circuit breaker and skip to the next target when a provider is already open.
### Quota & Rate Limiting
@@ -120,7 +120,7 @@ Each service requires unit and integration tests. For authoritative coverage req
- **Combo-first design**: All routing decisions go through combo engine; fallback strategies are combo targets, not ad-hoc logic
- **Service composition**: Small focused modules; combo.ts orchestrates them, not monolithic routing
- **Circuit breaker per-target**: Failures isolated to specific provider+account combo; other targets unaffected
- **Provider breaker is global**: Combo targets respect the shared provider circuit breaker; combo does not maintain a second target-local breaker
- **Caching everywhere**: Models, providers, quotas, family fallbacks all pre-cached; invalidated on write
- **13 strategies** over hardcoded logic: Strategy pattern allows new routing logic without touching combo.ts core
@@ -130,6 +130,6 @@ Each service requires unit and integration tests. For authoritative coverage req
- New services must not add blocking I/O to routing hot path
- Combo target resolution under 10ms (measure with benchmarks)
- Circuit breaker state per-target, not global
- Combo should not reintroduce a second breaker layer on top of the global provider breaker
- All fallback chains tested (no infinite loops)
- Coverage requirements: See [`CONTRIBUTING.md#running-tests`](../../CONTRIBUTING.md#running-tests) (60% gate enforced in CI)

493
open-sse/services/accountFallback.ts
View File

@@ -4,11 +4,30 @@ import {
BACKOFF_STEPS_MS,
RateLimitReason,
HTTP_STATUS,
PROVIDER_PROFILES,
} from "../config/constants.ts";
import { getPassthroughProviders, getProviderCategory } from "../config/providerRegistry.ts";
import {
DEFAULT_RESILIENCE_SETTINGS,
resolveResilienceSettings,
} from "../../src/lib/resilience/settings";
import {
getAllCircuitBreakerStatuses,
getCircuitBreaker,
STATE,
} from "../../src/shared/utils/circuitBreaker";
type ProviderProfile = (typeof PROVIDER_PROFILES)["oauth"];
type ProviderProfile = {
baseCooldownMs: number;
useUpstreamRetryHints: boolean;
maxBackoffSteps: number;
failureThreshold: number;
resetTimeoutMs: number;
transientCooldown: number;
rateLimitCooldown: number;
maxBackoffLevel: number;
circuitBreakerThreshold: number;
circuitBreakerReset: number;
};
type JsonRecord = Record<string, unknown>;
type ModelLockoutEntry = {
reason: string;
@@ -24,29 +43,10 @@ type ModelFailureState = {
resetAfterMs: number;
};
// Provider-level failure tracking for circuit breaker behavior
type ProviderFailureEntry = {
failureCount: number;
lastFailureAt: number;
resetAfterMs: number;
cooldownUntil: number | null;
};
// Error codes that count toward provider-level failure threshold
const PROVIDER_FAILURE_ERROR_CODES = new Set([429, 408, 500, 502, 503, 504]);
// Configuration for provider-level failure tracking
const PROVIDER_FAILURE_THRESHOLD = 5;
const PROVIDER_FAILURE_WINDOW_MS = 20 * 60 * 1000; // 20 minutes
const PROVIDER_COOLDOWN_MS = 10 * 60 * 1000; // 10 minutes cooling
// Provider-level failure state map: providerId -> failure entry
const providerFailureState = new Map<string, ProviderFailureEntry>();
// Guard against synchronous re-entrant calls within the same event-loop tick.
// NOT a true mutex — Node.js is single-threaded, so different SSE streams
// can interleave across ticks. This Set prevents a single call from recursively
// re-entering recordProviderFailure within the same synchronous call stack.
const providerFailureLocks = new Set<string>();
// Error codes that count toward provider-level failure threshold.
// Connection-scoped 429 rate limits stay in connection cooldown handling and
// do not contribute to the shared provider breaker.
const PROVIDER_FAILURE_ERROR_CODES = new Set([408, 500, 502, 503, 504]);
// T06 (sub2api PR #1037): Signals that indicate permanent account deactivation.
// When a 401 body contains these strings, the account is permanently dead
@@ -139,17 +139,7 @@ export function isOAuthInvalidToken(errorText: string): boolean {
return OAUTH_INVALID_TOKEN_SIGNALS.some((sig) => lower.includes(sig));
}
// ─── Provider Profile Helper ────────────────────────────────────────────────
/**
* Get the resilience profile for a provider (oauth or apikey).
* @param {string} provider - Provider ID or alias
* @returns {import('../config/constants.js').PROVIDER_PROFILES['oauth']}
*/
export function getProviderProfile(provider) {
const category = getProviderCategory(provider);
return PROVIDER_PROFILES[category] ?? PROVIDER_PROFILES.apikey;
}
// ─── Resilience Profile Helper ──────────────────────────────────────────────
function asRecord(value: unknown): JsonRecord {
return value && typeof value === "object" && !Array.isArray(value) ? (value as JsonRecord) : {};
@@ -162,42 +152,47 @@ function isCompatibleProvider(provider: string | null | undefined): boolean {
);
}
function mergeProviderProfile(fallback: ProviderProfile, overrides: unknown): ProviderProfile {
const record = asRecord(overrides);
function buildProviderProfile(
category: "oauth" | "apikey",
settings?: Record<string, unknown> | null
) {
const resilience = settings ? resolveResilienceSettings(settings) : DEFAULT_RESILIENCE_SETTINGS;
const connectionCooldown = resilience.connectionCooldown[category];
const providerBreaker = resilience.providerBreaker[category];
return {
transientCooldown:
typeof record.transientCooldown === "number"
? record.transientCooldown
: fallback.transientCooldown,
rateLimitCooldown:
typeof record.rateLimitCooldown === "number"
? record.rateLimitCooldown
: fallback.rateLimitCooldown,
maxBackoffLevel:
typeof record.maxBackoffLevel === "number"
? record.maxBackoffLevel
: fallback.maxBackoffLevel,
circuitBreakerThreshold:
typeof record.circuitBreakerThreshold === "number"
? record.circuitBreakerThreshold
: fallback.circuitBreakerThreshold,
circuitBreakerReset:
typeof record.circuitBreakerReset === "number"
? record.circuitBreakerReset
: fallback.circuitBreakerReset,
};
baseCooldownMs: connectionCooldown.baseCooldownMs,
useUpstreamRetryHints: connectionCooldown.useUpstreamRetryHints,
maxBackoffSteps: connectionCooldown.maxBackoffSteps,
failureThreshold: providerBreaker.failureThreshold,
resetTimeoutMs: providerBreaker.resetTimeoutMs,
transientCooldown: connectionCooldown.baseCooldownMs,
rateLimitCooldown: connectionCooldown.useUpstreamRetryHints
? 0
: connectionCooldown.baseCooldownMs,
maxBackoffLevel: connectionCooldown.maxBackoffSteps,
circuitBreakerThreshold: providerBreaker.failureThreshold,
circuitBreakerReset: providerBreaker.resetTimeoutMs,
} satisfies ProviderProfile;
}
/**
* Get the resilience profile for a provider (oauth or apikey).
* @param {string} provider - Provider ID or alias
*/
export function getProviderProfile(provider) {
const category = getProviderCategory(provider);
return buildProviderProfile(category);
}
export async function getRuntimeProviderProfile(provider: string | null | undefined) {
const fallback = getProviderProfile(provider);
try {
const { getCachedSettings } = await import("@/lib/db/readCache");
const settings = await getCachedSettings();
const profiles = asRecord(settings.providerProfiles);
const category = getProviderCategory(provider);
return mergeProviderProfile(fallback, profiles[category]);
return buildProviderProfile(category, settings);
} catch {
return fallback;
return getProviderProfile(provider);
}
}
@@ -211,7 +206,7 @@ function getModelLockKey(provider: string, connectionId: string, model: string)
}
function getFailureWindowMs(profile: ProviderProfile | null = null, fallbackMs = 30 * 60 * 1000) {
const configured = profile?.circuitBreakerReset;
const configured = profile?.resetTimeoutMs;
return typeof configured === "number" && configured > 0 ? configured : fallbackMs;
}
@@ -233,23 +228,13 @@ function getModelLockBaseCooldown(
fallbackCooldownMs: number,
profile: ProviderProfile | null = null
) {
if (status === HTTP_STATUS.RATE_LIMITED) {
if (typeof profile?.rateLimitCooldown === "number" && profile.rateLimitCooldown > 0) {
return profile.rateLimitCooldown;
}
if (Number.isFinite(fallbackCooldownMs) && fallbackCooldownMs > 0) {
return fallbackCooldownMs;
}
return getQuotaCooldown(0);
}
if (typeof profile?.transientCooldown === "number" && profile.transientCooldown > 0) {
return profile.transientCooldown;
}
if (Number.isFinite(fallbackCooldownMs) && fallbackCooldownMs > 0) {
return fallbackCooldownMs;
}
return COOLDOWN_MS.transientInitial;
if (typeof profile?.baseCooldownMs === "number" && profile.baseCooldownMs >= 0) {
return profile.baseCooldownMs;
}
return status === HTTP_STATUS.RATE_LIMITED ? getQuotaCooldown(0) : COOLDOWN_MS.transientInitial;
}
function getScaledCooldown(
@@ -333,7 +318,8 @@ export function recordModelLockoutFailure(
reason: string,
status: number,
fallbackCooldownMs: number,
profile: ProviderProfile | null = null
profile: ProviderProfile | null = null,
options: { exactCooldownMs?: number | null } = {}
) {
ensureCleanupTimer();
const key = getModelLockKey(provider, connectionId, model);
@@ -351,11 +337,14 @@ export function recordModelLockoutFailure(
});
const baseCooldownMs = getModelLockBaseCooldown(status, fallbackCooldownMs, profile);
const cooldownMs = getScaledCooldown(
baseCooldownMs,
failureCount,
profile?.maxBackoffLevel ?? BACKOFF_CONFIG.maxLevel
);
const cooldownMs =
typeof options.exactCooldownMs === "number" && options.exactCooldownMs > 0
? options.exactCooldownMs
: getScaledCooldown(
baseCooldownMs,
failureCount,
profile?.maxBackoffSteps ?? BACKOFF_CONFIG.maxLevel
);
lockModel(provider, connectionId, model, reason, cooldownMs, {
failureCount,
@@ -485,113 +474,62 @@ export function getAllModelLockouts() {
return active;
}
// ─── Provider-Level Failure Tracking ─────────────────────────────────────────
// Track failures at provider level: when a provider has too many transient failures
// across all its connections, cooldown the entire provider temporarily.
// ─── Provider Breaker Compatibility Wrappers ────────────────────────────────
// Legacy helpers now delegate to the shared provider circuit breaker.
/**
* Check if a provider is currently in cooldown due to too many failures
*/
export function isProviderInCooldown(provider: string | null | undefined): boolean {
if (!provider) return false;
const entry = providerFailureState.get(provider);
if (!entry) return false;
// If in cooldown, check if it has expired
if (entry.cooldownUntil !== null && Date.now() >= entry.cooldownUntil) {
providerFailureState.delete(provider);
return false;
}
return entry.cooldownUntil !== null;
function getProviderBreaker(provider: string | null | undefined) {
if (!provider) return null;
const profile = getProviderProfile(provider);
return getCircuitBreaker(provider, {
failureThreshold: profile.failureThreshold ?? profile.circuitBreakerThreshold,
resetTimeout: profile.resetTimeoutMs ?? profile.circuitBreakerReset,
});
}
/**
* Get remaining cooldown time for a provider
* Check if a provider is currently blocked by the shared circuit breaker.
*/
export function isProviderInCooldown(provider: string | null | undefined): boolean {
const breaker = getProviderBreaker(provider);
return breaker ? !breaker.canExecute() : false;
}
/**
* Get remaining retry-after time for a provider breaker.
*/
export function getProviderCooldownRemainingMs(provider: string | null | undefined): number | null {
if (!provider) return null;
const entry = providerFailureState.get(provider);
if (!entry || entry.cooldownUntil === null) return null;
const remaining = entry.cooldownUntil - Date.now();
const breaker = getProviderBreaker(provider);
if (!breaker || breaker.canExecute()) return null;
const remaining = breaker.getRetryAfterMs();
return remaining > 0 ? remaining : null;
}
/**
* Record a failure for a provider. When threshold is reached within the window,
* the provider enters cooldown.
* Record a provider failure against the shared circuit breaker.
*/
export function recordProviderFailure(
provider: string | null | undefined,
log?: { warn?: (...args: unknown[]) => void }
): void {
if (!provider) return;
// Guard against concurrent re-entrant calls within the same tick
if (providerFailureLocks.has(provider)) return;
providerFailureLocks.add(provider);
try {
const now = Date.now();
const entry = providerFailureState.get(provider);
// Check if we're in cooldown period
if (entry && entry.cooldownUntil !== null && now < entry.cooldownUntil) {
return; // Already in cooldown, don't record
}
// Check if failure window has expired
if (entry && now - entry.lastFailureAt > entry.resetAfterMs) {
// Window expired, reset count
providerFailureState.set(provider, {
failureCount: 1,
lastFailureAt: now,
resetAfterMs: PROVIDER_FAILURE_WINDOW_MS,
cooldownUntil: null,
});
return;
}
// Increment failure count
const newCount = entry ? entry.failureCount + 1 : 1;
if (newCount >= PROVIDER_FAILURE_THRESHOLD) {
// Threshold reached, enter cooldown
const cooldownUntil = now + PROVIDER_COOLDOWN_MS;
providerFailureState.set(provider, {
failureCount: newCount,
lastFailureAt: now,
resetAfterMs: PROVIDER_FAILURE_WINDOW_MS,
cooldownUntil,
});
log?.warn?.(
`[ProviderFailure] ${provider}: ${newCount} failures in ${PROVIDER_FAILURE_WINDOW_MS / 1000}s — entering ${PROVIDER_COOLDOWN_MS / 1000}s cooldown`
);
} else {
// Just increment counter
providerFailureState.set(provider, {
failureCount: newCount,
lastFailureAt: now,
resetAfterMs: PROVIDER_FAILURE_WINDOW_MS,
cooldownUntil: null,
});
}
} finally {
providerFailureLocks.delete(provider);
const breaker = getProviderBreaker(provider);
if (!breaker || !provider) return;
breaker._onFailure();
const status = breaker.getStatus();
if (status.state === STATE.OPEN) {
log?.warn?.(`[ProviderBreaker] ${provider}: OPEN after ${status.failureCount} final failures`);
}
}
/**
* Clear provider failure state (e.g., after successful request)
* Reset the shared provider breaker.
*/
export function clearProviderFailure(provider: string | null | undefined): void {
if (!provider) return;
providerFailureState.delete(provider);
const breaker = getProviderBreaker(provider);
breaker?.reset();
}
/**
* Get all providers currently in cooldown (for debugging/dashboard)
* Get all providers currently blocked by the shared breaker.
*/
export function getProvidersInCooldown(): Array<{
provider: string;
@@ -599,22 +537,17 @@ export function getProvidersInCooldown(): Array<{
cooldownRemainingMs: number | null;
lastFailureAt: number;
}> {
const result = [];
for (const [provider, entry] of providerFailureState) {
if (entry.cooldownUntil === null) continue;
const remaining = entry.cooldownUntil - Date.now();
if (remaining <= 0) {
providerFailureState.delete(provider);
continue;
}
result.push({
provider,
failureCount: entry.failureCount,
cooldownRemainingMs: remaining,
lastFailureAt: entry.lastFailureAt,
});
}
return result;
return getAllCircuitBreakerStatuses()
.filter((status) => {
const breaker = getProviderBreaker(status.name);
return Boolean(breaker && !breaker.canExecute());
})
.map((status) => ({
provider: status.name,
failureCount: status.failureCount,
cooldownRemainingMs: status.retryAfterMs || null,
lastFailureAt: status.lastFailureTime,
}));
}
/**
@@ -866,16 +799,6 @@ export function getQuotaCooldown(backoffLevel = 0) {
return Math.min(cooldown, BACKOFF_CONFIG.max);
}
function getRateLimitCooldown(backoffLevel = 0, profile: ProviderProfile | null = null) {
const maxLevel = profile?.maxBackoffLevel ?? BACKOFF_CONFIG.maxLevel;
const cappedLevel = Math.min(Math.max(0, backoffLevel), maxLevel);
const configuredBase = profile?.rateLimitCooldown;
if (typeof configuredBase === "number" && configuredBase > 0) {
return configuredBase * Math.pow(2, cappedLevel);
}
return getQuotaCooldown(cappedLevel);
}
/**
* Check if error should trigger account fallback (switch to next account)
* @param {number} status - HTTP status code
@@ -889,21 +812,24 @@ export function checkFallbackError(
status,
errorText,
backoffLevel = 0,
model = null,
_model = null,
provider = null,
headers = null,
profileOverride: ProviderProfile | null = null
) {
const errorStr = (errorText || "").toString();
const profile = profileOverride ?? (provider ? getProviderProfile(provider) : null);
const maxBackoffSteps = profile?.maxBackoffSteps ?? BACKOFF_CONFIG.maxLevel;
const retryableStatuses = new Set([
HTTP_STATUS.REQUEST_TIMEOUT,
HTTP_STATUS.RATE_LIMITED,
HTTP_STATUS.SERVER_ERROR,
HTTP_STATUS.BAD_GATEWAY,
HTTP_STATUS.SERVICE_UNAVAILABLE,
HTTP_STATUS.GATEWAY_TIMEOUT,
]);
// Track provider-level failures for circuit breaker behavior
// Only count transient errors that are likely to recover
if (isProviderFailureCode(status)) {
recordProviderFailure(provider);
}
function parseResetFromHeaders(headers, errorStr = "") {
function parseResetFromHeaders(headers) {
if (!headers) return null;
// Retry-After header
@@ -935,6 +861,59 @@ export function checkFallbackError(
}
return null;
}
function getUpstreamRetryHintMs() {
if (!profile?.useUpstreamRetryHints) return null;
const resetTime = parseResetFromHeaders(headers);
if (resetTime) {
const waitMs = Math.max(resetTime - Date.now(), 0);
if (waitMs > 0) return waitMs;
}
const retryFromErrorText = parseRetryFromErrorText(errorStr);
if (retryFromErrorText && retryFromErrorText > 0) {
return retryFromErrorText;
}
return null;
}
function getScaledBaseCooldown(reason, level = backoffLevel) {
const baseCooldownMs =
typeof profile?.baseCooldownMs === "number" && profile.baseCooldownMs >= 0
? profile.baseCooldownMs
: COOLDOWN_MS.transientInitial;
return {
baseCooldownMs,
cooldownMs: getScaledCooldown(baseCooldownMs, level + 1, maxBackoffSteps),
newBackoffLevel: Math.min(level + 1, maxBackoffSteps),
};
}
function buildRetryableFallback(reason) {
const upstreamRetryHintMs = getUpstreamRetryHintMs();
if (typeof upstreamRetryHintMs === "number" && upstreamRetryHintMs > 0) {
return {
shouldFallback: true,
cooldownMs: upstreamRetryHintMs,
baseCooldownMs: upstreamRetryHintMs,
newBackoffLevel: 0,
usedUpstreamRetryHint: true,
reason,
};
}
const scaled = getScaledBaseCooldown(reason, backoffLevel);
return {
shouldFallback: true,
cooldownMs: scaled.cooldownMs,
baseCooldownMs: scaled.baseCooldownMs,
newBackoffLevel: scaled.newBackoffLevel,
usedUpstreamRetryHint: false,
reason,
};
}
// Check error message FIRST - specific patterns take priority over status codes
if (errorText) {
const lowerError = errorStr.toLowerCase();
@@ -984,6 +963,7 @@ export function checkFallbackError(
return {
shouldFallback: true,
cooldownMs: COOLDOWN_MS.requestNotAllowed,
baseCooldownMs: COOLDOWN_MS.requestNotAllowed,
reason: RateLimitReason.RATE_LIMIT_EXCEEDED,
};
}
@@ -999,45 +979,16 @@ export function checkFallbackError(
lowerError.includes("capacity") ||
lowerError.includes("overloaded")
) {
const resetTime = parseResetFromHeaders(headers);
if (resetTime) {
const waitMs = resetTime - Date.now();
if (waitMs > 60_000) {
return {
shouldFallback: true,
cooldownMs: waitMs,
newBackoffLevel: 0,
reason: RateLimitReason.RATE_LIMIT_EXCEEDED,
};
}
}
const retryFromBody = parseRetryFromErrorText(errorStr);
if (retryFromBody && retryFromBody > 60_000) {
return {
shouldFallback: true,
cooldownMs: retryFromBody,
newBackoffLevel: 0,
reason: RateLimitReason.RATE_LIMIT_EXCEEDED,
};
}
const newLevel = Math.min(
backoffLevel + 1,
profile?.maxBackoffLevel ?? BACKOFF_CONFIG.maxLevel
);
const reason = classifyErrorText(errorStr);
return {
shouldFallback: true,
cooldownMs: getRateLimitCooldown(backoffLevel, profile),
newBackoffLevel: newLevel,
reason,
};
return buildRetryableFallback(reason);
}
}
if (status === HTTP_STATUS.UNAUTHORIZED) {
return {
shouldFallback: true,
cooldownMs: COOLDOWN_MS.unauthorized,
cooldownMs: 0,
baseCooldownMs: 0,
reason: RateLimitReason.AUTH_ERROR,
};
}
@@ -1045,7 +996,8 @@ export function checkFallbackError(
if (status === HTTP_STATUS.PAYMENT_REQUIRED || status === HTTP_STATUS.FORBIDDEN) {
return {
shouldFallback: true,
cooldownMs: COOLDOWN_MS.paymentRequired,
cooldownMs: 0,
baseCooldownMs: 0,
reason: RateLimitReason.QUOTA_EXHAUSTED,
};
}
@@ -1060,64 +1012,11 @@ export function checkFallbackError(
// 429 - Rate limit with exponential backoff
if (status === HTTP_STATUS.RATE_LIMITED) {
const resetTime = parseResetFromHeaders(headers);
if (resetTime) {
const waitMs = resetTime - Date.now();
if (waitMs > 60_000) {
return {
shouldFallback: true,
cooldownMs: waitMs,
newBackoffLevel: 0,
reason: RateLimitReason.RATE_LIMIT_EXCEEDED,
};
}
}
const newLevel = Math.min(
backoffLevel + 1,
profile?.maxBackoffLevel ?? BACKOFF_CONFIG.maxLevel
);
return {
shouldFallback: true,
cooldownMs: getRateLimitCooldown(backoffLevel, profile),
newBackoffLevel: newLevel,
reason: RateLimitReason.RATE_LIMIT_EXCEEDED,
};
return buildRetryableFallback(RateLimitReason.RATE_LIMIT_EXCEEDED);
}
// Transient / server errors — exponential backoff with provider profile
const transientStatuses = [
HTTP_STATUS.NOT_ACCEPTABLE,
HTTP_STATUS.REQUEST_TIMEOUT,
HTTP_STATUS.SERVER_ERROR,
HTTP_STATUS.BAD_GATEWAY,
HTTP_STATUS.SERVICE_UNAVAILABLE,
HTTP_STATUS.GATEWAY_TIMEOUT,
];
if (transientStatuses.includes(status)) {
const resetTime = parseResetFromHeaders(headers, errorStr);
if (resetTime) {
const waitMs = resetTime - Date.now();
if (waitMs > 60_000) {
return {
shouldFallback: true,
cooldownMs: waitMs,
newBackoffLevel: 0,
reason: RateLimitReason.SERVER_ERROR,
};
}
}
const baseCooldown = profile?.transientCooldown ?? COOLDOWN_MS.transientInitial;
const maxLevel = profile?.maxBackoffLevel ?? BACKOFF_CONFIG.maxLevel;
const cooldownMs = Math.min(baseCooldown * Math.pow(2, backoffLevel), COOLDOWN_MS.transientMax);
const newLevel = Math.min(backoffLevel + 1, maxLevel);
return {
shouldFallback: true,
cooldownMs,
newBackoffLevel: newLevel,
reason: RateLimitReason.SERVER_ERROR,
};
if (status === HTTP_STATUS.NOT_ACCEPTABLE || retryableStatuses.has(status)) {
return buildRetryableFallback(RateLimitReason.SERVER_ERROR);
}
// 400 — context overflow / malformed request may succeed on another model in the combo
@@ -1140,7 +1039,8 @@ export function checkFallbackError(
// All other errors - fallback with transient cooldown
return {
shouldFallback: true,
cooldownMs: COOLDOWN_MS.transient,
cooldownMs: profile?.baseCooldownMs ?? COOLDOWN_MS.transient,
baseCooldownMs: profile?.baseCooldownMs ?? COOLDOWN_MS.transient,
reason: RateLimitReason.UNKNOWN,
};
}
@@ -1232,13 +1132,10 @@ export function applyErrorState(account, status, errorText, provider = null) {
if (!account) return account;
const backoffLevel = account.backoffLevel || 0;
const { cooldownMs, newBackoffLevel, reason } = checkFallbackError(
status,
errorText,
backoffLevel,
null,
provider
);
const fallbackDecision = checkFallbackError(status, errorText, backoffLevel, null, provider);
const { cooldownMs, reason } = fallbackDecision;
const newBackoffLevel =
"newBackoffLevel" in fallbackDecision ? fallbackDecision.newBackoffLevel : undefined;
return {
...account,

127
open-sse/services/combo.ts
View File

@@ -40,6 +40,16 @@ import {
getComboStepWeight,
normalizeComboStep,
} from "../../src/lib/combos/steps.ts";
function isProviderBreakerOpenResponse(
result: Response,
errorBody?: { error?: { code?: string | null } } | null
) {
return (
result.headers.get("x-omniroute-provider-breaker") === "open" ||
errorBody?.error?.code === "provider_circuit_open"
);
}
import {
getConnectionRoutingTags,
matchesRoutingTags,
@@ -47,9 +57,8 @@ import {
type RoutingTagMatchMode,
} from "../../src/domain/tagRouter.ts";
// Status codes that should mark semaphore + record circuit breaker failures
// 401, 403 added so Auth errors quickly open the circuit breaker to prevent background request leaks
const TRANSIENT_FOR_BREAKER = [401, 403, 429, 500, 502, 503, 504];
// Status codes that should mark round-robin target semaphores as cooling down.
const TRANSIENT_FOR_SEMAPHORE = [429, 502, 503, 504];
const COMBO_BAD_REQUEST_FALLBACK_PATTERNS = [
/\bprohibited_content\b/i,
/request blocked by .*api/i,
@@ -269,10 +278,6 @@ function getTargetProvider(modelStr: string, providerId?: string | null): string
return providerId || parsed.provider || parsed.providerAlias || "unknown";
}
function getComboBreakerKey(comboName: string, executionKey: string): string {
return `combo:${comboName}:${executionKey}`;
}
function toRecordedTarget(target: ResolvedComboTarget) {
return {
executionKey: target.executionKey,
@@ -868,9 +873,7 @@ async function buildAutoCandidates(targets, comboName) {
? Math.max(10, historicalStdDev)
: Math.max(10, p95LatencyMs * 0.1);
const breakerStateRaw = getCircuitBreaker(
getComboBreakerKey(comboName, target.executionKey)
)?.getStatus?.()?.state;
const breakerStateRaw = getCircuitBreaker(provider)?.getStatus?.()?.state;
const circuitBreakerState =
breakerStateRaw === "OPEN" || breakerStateRaw === "HALF_OPEN" ? breakerStateRaw : "CLOSED";
@@ -1502,18 +1505,6 @@ export async function handleComboChat({
const modelStr = target.modelStr;
const provider = target.provider;
const profile = await getRuntimeProviderProfile(provider);
const breakerKey = getComboBreakerKey(combo.name, target.executionKey);
const breaker = getCircuitBreaker(breakerKey, {
failureThreshold: profile.circuitBreakerThreshold,
resetTimeout: profile.circuitBreakerReset,
});
// Skip model if circuit breaker is OPEN
if (!breaker.canExecute()) {
log.info("COMBO", `Skipping ${modelStr}: circuit breaker OPEN for ${provider}`);
if (i > 0) fallbackCount++;
continue;
}
// Pre-check: skip models where all accounts are in cooldown
if (isModelAvailable) {
@@ -1558,7 +1549,6 @@ export async function handleComboChat({
"COMBO",
`Model ${modelStr} returned 200 but failed quality check: ${quality.reason}`
);
breaker._onFailure();
recordComboRequest(combo.name, modelStr, {
success: false,
latencyMs: Date.now() - startTime,
@@ -1575,7 +1565,6 @@ export async function handleComboChat({
"COMBO",
`Model ${modelStr} succeeded (${latencyMs}ms, ${fallbackCount} fallbacks)`
);
breaker._onSuccess();
recordComboRequest(combo.name, modelStr, {
success: true,
latencyMs,
@@ -1644,6 +1633,7 @@ export async function handleComboChat({
// Extract error info from response
let errorText = result.statusText || "";
let errorBody = null;
let retryAfter = null;
try {
const cloned = result.clone();
@@ -1651,7 +1641,7 @@ export async function handleComboChat({
const text = await cloned.text();
if (text) {
errorText = text.substring(0, 500);
const errorBody = JSON.parse(text);
errorBody = JSON.parse(text);
errorText =
errorBody?.error?.message || errorBody?.error || errorBody?.message || errorText;
retryAfter = errorBody?.retryAfter || null;
@@ -1680,11 +1670,15 @@ export async function handleComboChat({
}
}
const isAllAccountsRateLimited = isAllAccountsRateLimitedResponse(
result.status,
result.headers?.get("content-type") ?? null,
errorText
);
const providerBreakerOpen = isProviderBreakerOpenResponse(result, errorBody);
if (providerBreakerOpen) {
lastError = errorText || String(result.status);
if (!lastStatus) lastStatus = result.status;
if (i > 0) fallbackCount++;
log.info("COMBO", `Skipping ${modelStr}: provider circuit breaker OPEN for ${provider}`);
break;
}
const { shouldFallback, cooldownMs } = checkFallbackError(
result.status,
@@ -1697,14 +1691,7 @@ export async function handleComboChat({
);
const comboBadRequestFallback = shouldFallbackComboBadRequest(result.status, errorText);
// Record failure in circuit breaker for transient errors
if (TRANSIENT_FOR_BREAKER.includes(result.status)) {
breaker._onFailure();
}
if (isAllAccountsRateLimited) {
log.info("COMBO", `All accounts rate-limited for ${modelStr}, falling back to next model`);
} else if (!shouldFallback && !comboBadRequestFallback) {
if (!shouldFallback && !comboBadRequestFallback) {
log.warn("COMBO", `Model ${modelStr} failed (no fallback)`, { status: result.status });
recordComboRequest(combo.name, modelStr, {
success: false,
@@ -1757,25 +1744,12 @@ export async function handleComboChat({
}
}
// Early exit: check if all models have breaker OPEN
const allBreakersOpen = orderedTargets.every((target) => {
return !getCircuitBreaker(getComboBreakerKey(combo.name, target.executionKey)).canExecute();
});
// All models failed
const latencyMs = Date.now() - startTime;
if (recordedAttempts === 0) {
recordComboRequest(combo.name, null, { success: false, latencyMs, fallbackCount, strategy });
}
if (allBreakersOpen) {
log.warn("COMBO", "All models have circuit breaker OPEN — aborting");
return unavailableResponse(
503,
"All providers temporarily unavailable (circuit breakers open)"
);
}
if (!lastStatus) {
return new Response(
JSON.stringify({
@@ -1861,19 +1835,7 @@ async function handleRoundRobinCombo({
const modelStr = target.modelStr;
const provider = target.provider;
const profile = await getRuntimeProviderProfile(provider);
const breakerKey = getComboBreakerKey(combo.name, target.executionKey);
const semaphoreKey = `combo:${combo.name}:${target.executionKey}`;
const breaker = getCircuitBreaker(breakerKey, {
failureThreshold: profile.circuitBreakerThreshold,
resetTimeout: profile.circuitBreakerReset,
});
// Skip model if circuit breaker is OPEN
if (!breaker.canExecute()) {
log.info("COMBO-RR", `Skipping ${modelStr}: circuit breaker OPEN for ${provider}`);
if (offset > 0) fallbackCount++;
continue;
}
// Pre-check availability
if (isModelAvailable) {
@@ -1935,7 +1897,6 @@ async function handleRoundRobinCombo({
"COMBO-RR",
`${modelStr} returned 200 but failed quality check: ${quality.reason}`
);
breaker._onFailure();
recordComboRequest(combo.name, modelStr, {
success: false,
latencyMs: Date.now() - startTime,
@@ -1952,7 +1913,6 @@ async function handleRoundRobinCombo({
"COMBO-RR",
`${modelStr} succeeded (${latencyMs}ms, ${fallbackCount} fallbacks)`
);
breaker._onSuccess();
recordComboRequest(combo.name, modelStr, {
success: true,
latencyMs,
@@ -1984,13 +1944,14 @@ async function handleRoundRobinCombo({
// Extract error info
let errorText = result.statusText || "";
let retryAfter = null;
let errorBody: { error?: { code?: string | null; message?: string | null } } | null = null;
try {
const cloned = result.clone();
try {
const text = await cloned.text();
if (text) {
errorText = text.substring(0, 500);
const errorBody = JSON.parse(text);
errorBody = JSON.parse(text);
errorText =
errorBody?.error?.message || errorBody?.error || errorBody?.message || errorText;
retryAfter = errorBody?.retryAfter || null;
@@ -2017,6 +1978,17 @@ async function handleRoundRobinCombo({
}
}
if (isProviderBreakerOpenResponse(result, errorBody)) {
lastError = errorText || String(result.status);
if (!lastStatus) lastStatus = result.status;
if (offset > 0) fallbackCount++;
log.info(
"COMBO-RR",
`Skipping ${modelStr}: provider circuit breaker OPEN for ${provider}`
);
break;
}
const { shouldFallback, cooldownMs } = checkFallbackError(
result.status,
errorText,
@@ -2034,14 +2006,10 @@ async function handleRoundRobinCombo({
errorText
);
// Transient errors → mark in semaphore AND record circuit breaker failure
if (TRANSIENT_FOR_BREAKER.includes(result.status) && cooldownMs > 0) {
// Transient errors → mark in semaphore so round-robin stops stampeding this target.
if (TRANSIENT_FOR_SEMAPHORE.includes(result.status) && cooldownMs > 0) {
semaphore.markRateLimited(semaphoreKey, cooldownMs);
breaker._onFailure();
log.warn(
"COMBO-RR",
`${modelStr} error ${result.status}, cooldown ${cooldownMs}ms (breaker: ${breaker.getStatus().failureCount}/${profile.circuitBreakerThreshold})`
);
log.warn("COMBO-RR", `${modelStr} error ${result.status}, cooldown ${cooldownMs}ms`);
}
if (isAllAccountsRateLimited) {
@@ -2117,19 +2085,6 @@ async function handleRoundRobinCombo({
});
}
// Early exit: check if all models have breaker OPEN
const allBreakersOpen = orderedTargets.every((target) => {
return !getCircuitBreaker(getComboBreakerKey(combo.name, target.executionKey)).canExecute();
});
if (allBreakersOpen) {
log.warn("COMBO-RR", "All models have circuit breaker OPEN — aborting");
return unavailableResponse(
503,
"All providers temporarily unavailable (circuit breakers open)"
);
}
if (!lastStatus) {
return new Response(
JSON.stringify({

16
open-sse/services/comboConfig.ts
View File

@@ -9,11 +9,8 @@ const DEFAULT_COMBO_CONFIG = {
strategy: "priority",
maxRetries: 1,
retryDelayMs: 2000,
timeoutMs: 600000,
concurrencyPerModel: 3, // max simultaneous requests per model (round-robin)
queueTimeoutMs: 30000, // max wait time in semaphore queue (round-robin)
healthCheckEnabled: true,
healthCheckTimeoutMs: 3000,
handoffThreshold: 0.85,
handoffModel: "",
handoffProviders: ["codex"],
@@ -22,6 +19,12 @@ const DEFAULT_COMBO_CONFIG = {
trackMetrics: true,
};
const LEGACY_COMBO_RESILIENCE_KEYS = new Set([
"timeoutMs",
"healthCheckEnabled",
"healthCheckTimeoutMs",
]);
/**
* Resolve effective config for a combo, applying cascade:
* DEFAULT_COMBO_CONFIG → settings.comboDefaults → settings.providerOverrides[provider] → combo.config
@@ -38,7 +41,12 @@ export function resolveComboConfig(combo, settings, provider?: string | null) {
// Clean undefined values before spreading
const clean = (obj) =>
Object.fromEntries(Object.entries(obj).filter(([, v]) => v !== undefined && v !== null));
Object.fromEntries(
Object.entries(obj).filter(
([key, value]) =>
value !== undefined && value !== null && !LEGACY_COMBO_RESILIENCE_KEYS.has(key)
)
);
return {
...DEFAULT_COMBO_CONFIG,

107
open-sse/services/rateLimitManager.ts
View File

@@ -9,10 +9,14 @@
*/
import Bottleneck from "bottleneck";
import { parseRetryAfterFromBody, lockModel } from "./accountFallback.ts";
import { parseRetryAfterFromBody } from "./accountFallback.ts";
import { getProviderCategory } from "../config/providerRegistry.ts";
import { DEFAULT_API_LIMITS } from "../config/constants.ts";
import { getCodexRateLimitKey } from "../executors/codex.ts";
import {
DEFAULT_RESILIENCE_SETTINGS,
resolveResilienceSettings,
type RequestQueueSettings,
} from "../../src/lib/resilience/settings";
interface LearnedLimitEntry {
provider: string;
@@ -24,7 +28,9 @@ interface LearnedLimitEntry {
}
interface LimiterUpdateSettings {
maxConcurrent?: number | null;
minTime: number;
maxWait?: number | null;
reservoir?: number | null;
reservoirRefreshAmount?: number | null;
reservoirRefreshInterval?: number | null;
@@ -61,20 +67,18 @@ const PERSIST_DEBOUNCE_MS = 60_000; // Debounce persistence to every 60s max
// Track initialization
let initialized = false;
// Max time (ms) a job can wait in queue before failing with a timeout error.
// Prevents infinite queuing when all providers are exhausted after a 429.
// Configurable via RATE_LIMIT_MAX_WAIT_MS env var (default: 2 minutes).
const MAX_WAIT_MS = parseInt(process.env.RATE_LIMIT_MAX_WAIT_MS || "120000", 10);
let currentRequestQueueSettings: RequestQueueSettings = DEFAULT_RESILIENCE_SETTINGS.requestQueue;
// Default conservative settings (before we learn from headers)
const DEFAULT_SETTINGS = {
maxConcurrent: 10,
minTime: 0, // No throttle by default — let headers teach us
reservoir: null, // No initial reservoir — unlimited until we learn
reservoirRefreshAmount: null,
reservoirRefreshInterval: null,
maxWait: MAX_WAIT_MS, // Fail-fast: don't queue forever on 429 exhaustion
};
function buildLimiterDefaults() {
return {
maxConcurrent: currentRequestQueueSettings.concurrentRequests,
minTime: currentRequestQueueSettings.minTimeBetweenRequestsMs,
reservoir: currentRequestQueueSettings.requestsPerMinute,
reservoirRefreshAmount: currentRequestQueueSettings.requestsPerMinute,
reservoirRefreshInterval: 60 * 1000,
maxWait: currentRequestQueueSettings.maxWaitMs,
};
}
function trackAsyncOperation<T>(promise: Promise<T>): Promise<T> {
pendingAsyncOperations.add(promise);
@@ -93,11 +97,12 @@ export async function initializeRateLimits() {
initialized = true;
try {
const { getProviderConnections } = await import("@/lib/localDb");
const connections = await getProviderConnections();
const { getProviderConnections, getSettings } = await import("@/lib/localDb");
const [connections, settings] = await Promise.all([getProviderConnections(), getSettings()]);
const resilience = resolveResilienceSettings(settings);
applyRequestQueueSettings(resilience.requestQueue);
let explicitCount = 0;
let autoCount = 0;
let customCount = 0;
for (const connRaw of connections as unknown[]) {
const conn = toRecord(connRaw);
@@ -105,38 +110,17 @@ export async function initializeRateLimits() {
const provider = typeof conn.provider === "string" ? conn.provider : "";
const isActive = conn.isActive === true;
const rateLimitProtection = conn.rateLimitProtection === true;
const customRpm = toNumber(conn.customRpm, 0);
const customTpm = toNumber(conn.customTpm, 0);
if (!connectionId || !provider) continue;
// Custom rpm/tpm configured — enable rate limiting with user-defined values (#198)
if (customRpm > 0 || customTpm > 0) {
enabledConnections.add(connectionId);
customCount++;
const key = `${provider}:${connectionId}`;
const rpm = customRpm > 0 ? customRpm : DEFAULT_API_LIMITS.requestsPerMinute;
const minTime = Math.max(0, Math.floor(60000 / rpm) - 10);
if (!limiters.has(key)) {
limiters.set(
key,
new Bottleneck({
maxConcurrent: DEFAULT_API_LIMITS.concurrentRequests,
minTime,
reservoir: rpm,
reservoirRefreshAmount: rpm,
reservoirRefreshInterval: 60 * 1000,
maxWait: MAX_WAIT_MS,
id: key,
})
);
}
} else if (rateLimitProtection) {
if (rateLimitProtection) {
// Explicitly enabled by user
enabledConnections.add(connectionId);
explicitCount++;
} else if (getProviderCategory(provider) === "apikey" && isActive) {
} else if (
resilience.requestQueue.autoEnableApiKeyProviders &&
getProviderCategory(provider) === "apikey" &&
isActive
) {
// Auto-enable for API key providers (safety net)
enabledConnections.add(connectionId);
autoCount++;
@@ -147,12 +131,7 @@ export async function initializeRateLimits() {
limiters.set(
key,
new Bottleneck({
maxConcurrent: DEFAULT_API_LIMITS.concurrentRequests,
minTime: DEFAULT_API_LIMITS.minTimeBetweenRequests,
reservoir: DEFAULT_API_LIMITS.requestsPerMinute,
reservoirRefreshAmount: DEFAULT_API_LIMITS.requestsPerMinute,
reservoirRefreshInterval: 60 * 1000, // Refresh every minute
maxWait: MAX_WAIT_MS,
...buildLimiterDefaults(),
id: key,
})
);
@@ -160,9 +139,9 @@ export async function initializeRateLimits() {
}
}
if (explicitCount > 0 || autoCount > 0 || customCount > 0) {
if (explicitCount > 0 || autoCount > 0) {
console.log(
`🛡️ [RATE-LIMIT] Loaded ${explicitCount} explicit + ${autoCount} auto-enabled + ${customCount} custom rpm/tpm protection(s)`
`🛡️ [RATE-LIMIT] Loaded ${explicitCount} explicit + ${autoCount} auto-enabled protection(s)`
);
}
@@ -173,6 +152,21 @@ export async function initializeRateLimits() {
}
}
export function applyRequestQueueSettings(nextSettings: RequestQueueSettings) {
currentRequestQueueSettings = { ...nextSettings };
for (const limiter of limiters.values()) {
limiter.updateSettings({
maxConcurrent: currentRequestQueueSettings.concurrentRequests,
minTime: currentRequestQueueSettings.minTimeBetweenRequestsMs,
maxWait: currentRequestQueueSettings.maxWaitMs,
reservoir: currentRequestQueueSettings.requestsPerMinute,
reservoirRefreshAmount: currentRequestQueueSettings.requestsPerMinute,
reservoirRefreshInterval: 60 * 1000,
});
}
}
/**
* Enable rate limit protection for a connection
*/
@@ -222,7 +216,7 @@ function getLimiter(provider, connectionId, model = null) {
if (!limiters.has(key)) {
const limiter = new Bottleneck({
...DEFAULT_SETTINGS,
...buildLimiterDefaults(),
id: key,
});
@@ -648,10 +642,5 @@ export function updateFromResponseBody(provider, connectionId, responseBody, sta
reservoirRefreshAmount: 60,
reservoirRefreshInterval: retryAfterMs,
});
// Also apply model-level lockout if model is known
if (model) {
lockModel(provider, connectionId, model, reason, retryAfterMs);
}
}
}

27
open-sse/utils/error.ts
View File

@@ -243,6 +243,33 @@ export function unavailableResponse(
});
}
export function providerCircuitOpenResponse(
provider: string,
retryAfter?: string | number | Date | null
) {
const retryAfterSec = normalizeRetryAfterSeconds(retryAfter);
return new Response(
JSON.stringify({
error: {
message: `Provider ${provider} circuit breaker is open`,
type: "server_error",
code: "provider_circuit_open",
provider,
retry_after: retryAfterSec,
},
}),
{
status: 503,
headers: {
"Content-Type": "application/json",
"Access-Control-Allow-Origin": getCorsOrigin(),
"Retry-After": String(retryAfterSec),
"X-OmniRoute-Provider-Breaker": "open",
},
}
);
}
export function buildModelCooldownBody({
model,
retryAfterSec,

131
src/app/(dashboard)/dashboard/combos/IntelligentComboPanel.tsx
View File

@@ -1,6 +1,6 @@
"use client";
import { useCallback, useEffect, useMemo, useState } from "react";
import { useMemo, useState } from "react";
import Button from "@/shared/components/Button";
import Card from "@/shared/components/Card";
import { useNotificationStore } from "@/store/notificationStore";
@@ -8,10 +8,8 @@ import {
FACTOR_LABELS,
MODE_PACK_OPTIONS,
buildIntelligentProviderScores,
extractIntelligentHealthState,
normalizeIntelligentRoutingConfig,
} from "@/lib/combos/intelligentRouting";
import ModelStatusBadge from "@/app/(dashboard)/dashboard/providers/components/ModelStatusBadge";
function getI18nOrFallback(t: any, key: string, fallback: string) {
if (typeof t?.has === "function" && t.has(key)) return t(key);
@@ -47,8 +45,6 @@ export default function IntelligentComboPanel({
onComboUpdated?: (combo: any) => void;
}) {
const notify = useNotificationStore();
const [incidentMode, setIncidentMode] = useState(false);
const [exclusions, setExclusions] = useState<any[]>([]);
const [savingModePack, setSavingModePack] = useState<string | null>(null);
const normalizedConfig = useMemo(
() => normalizeIntelligentRoutingConfig(combo?.config),
@@ -56,35 +52,6 @@ export default function IntelligentComboPanel({
);
const providerScores = useMemo(() => buildIntelligentProviderScores(combo), [combo]);
const fetchHealth = useCallback(async () => {
try {
const response = await fetch("/api/monitoring/health");
if (!response.ok) {
setIncidentMode(false);
setExclusions([]);
return;
}
const health = await response.json();
const nextState = extractIntelligentHealthState(health);
setIncidentMode(nextState.incidentMode);
setExclusions(nextState.exclusions);
} catch {
setIncidentMode(false);
setExclusions([]);
}
}, []);
useEffect(() => {
const timeoutId = setTimeout(fetchHealth, 0);
const intervalId = setInterval(fetchHealth, 30_000);
return () => {
clearTimeout(timeoutId);
clearInterval(intervalId);
};
}, [fetchHealth]);
const handleModePackChange = async (modePackId: string) => {
if (!combo?.id || modePackId === normalizedConfig.modePack) return;
setSavingModePack(modePackId);
@@ -157,19 +124,9 @@ export default function IntelligentComboPanel({
</div>
</div>
<div
className={`inline-flex items-center gap-2 rounded-full px-3 py-1.5 text-xs font-medium ${
incidentMode
? "bg-amber-500/15 text-amber-600 dark:text-amber-300"
: "bg-emerald-500/15 text-emerald-600 dark:text-emerald-300"
}`}
>
<span className="material-symbols-outlined text-[14px]">
{incidentMode ? "warning" : "verified"}
</span>
{incidentMode
? getI18nOrFallback(t, "incidentMode", "Incident Mode")
: getI18nOrFallback(t, "normalOperation", "Normal Operation")}
<div className="inline-flex items-center gap-2 rounded-full bg-blue-500/15 px-3 py-1.5 text-xs font-medium text-blue-600 dark:text-blue-300">
<span className="material-symbols-outlined text-[14px]">tune</span>
{getI18nOrFallback(t, "configOnlyStatus", "Configuration View")}
</div>
</div>
@@ -181,22 +138,20 @@ export default function IntelligentComboPanel({
{getI18nOrFallback(t, "statusOverview", "Status Overview")}
</p>
<p className="text-[11px] text-text-muted mt-1">
{incidentMode
? getI18nOrFallback(
t,
"highCircuitBreakerRate",
"High circuit breaker trip rate detected."
)
: getI18nOrFallback(
t,
"allProvidersHealthy",
"Providers are reporting healthy routing conditions."
)}
{getI18nOrFallback(
t,
"configOnlyHint",
"This panel shows routing inputs only. Live breaker state is available on the Health page."
)}
</p>
</div>
<div className="rounded-lg bg-black/5 dark:bg-white/5 px-3 py-2 text-right">
<p className="text-[10px] uppercase tracking-wide text-text-muted">Exclusions</p>
<p className="text-lg font-semibold text-text-main">{exclusions.length}</p>
<p className="text-[10px] uppercase tracking-wide text-text-muted">
Candidate Pool
</p>
<p className="text-lg font-semibold text-text-main">
{normalizedConfig.candidatePool.length || activeProviders.length}
</p>
</div>
</div>
</Card.Section>
@@ -284,8 +239,6 @@ export default function IntelligentComboPanel({
</p>
<p className="text-[11px] text-text-muted mt-0.5">{entry.model}</p>
</div>
{/* Model status badge - shows cooldown/error state */}
<ModelStatusBadge provider={entry.provider} model={entry.model} size="sm" />
<span className="rounded-full bg-primary/10 px-2 py-1 text-[11px] font-semibold text-primary">
{percentage}%
</span>
@@ -320,51 +273,33 @@ export default function IntelligentComboPanel({
<div className="flex items-center justify-between gap-2">
<div>
<p className="text-sm font-semibold text-text-main">
{getI18nOrFallback(t, "excludedProviders", "Excluded Providers")}
{getI18nOrFallback(t, "routingInputs", "Routing Inputs")}
</p>
<p className="text-[11px] text-text-muted mt-1">
{getI18nOrFallback(
t,
"excludedProvidersHint",
"Providers with an OPEN circuit breaker are temporarily excluded from routing."
"routingInputsHint",
"Mode pack and weighting stay here; breaker runtime state stays on the Health page."
)}
</p>
</div>
</div>
<div className="mt-3 flex flex-col gap-2">
{exclusions.length === 0 ? (
<div className="rounded-lg border border-dashed border-emerald-500/20 bg-emerald-500/5 p-3 text-[11px] text-emerald-700 dark:text-emerald-300">
<div className="flex items-center gap-2">
<span className="material-symbols-outlined text-[14px]">verified</span>
{getI18nOrFallback(
t,
"noExcludedProviders",
"No providers are currently excluded."
)}
</div>
</div>
) : (
exclusions.map((exclusion) => (
<div
key={`${exclusion.provider}-${exclusion.excludedAt}`}
className="rounded-lg border border-amber-500/20 bg-amber-500/10 p-3"
>
<div className="flex items-start justify-between gap-2">
<div>
<p className="text-xs font-semibold text-text-main">{exclusion.provider}</p>
<p className="text-[11px] text-text-muted mt-1">{exclusion.reason}</p>
</div>
<span className="rounded-full bg-amber-500/15 px-2 py-1 text-[10px] font-semibold text-amber-700 dark:text-amber-300">
{getI18nOrFallback(t, "cooldownMinutes", "Cooldown: {minutes}m").replace(
"{minutes}",
`${Math.ceil(exclusion.cooldownMs / 60000)}`
)}
</span>
</div>
</div>
))
)}
<div className="mt-3 grid grid-cols-1 gap-2 sm:grid-cols-2">
<div className="rounded-lg border border-black/8 bg-white/60 p-3 dark:border-white/8 dark:bg-white/[0.03]">
<p className="text-[11px] uppercase tracking-wide text-text-muted">Mode Pack</p>
<p className="mt-1 text-sm font-semibold text-text-main">
{normalizedConfig.modePack}
</p>
</div>
<div className="rounded-lg border border-black/8 bg-white/60 p-3 dark:border-white/8 dark:bg-white/[0.03]">
<p className="text-[11px] uppercase tracking-wide text-text-muted">
Exploration Rate
</p>
<p className="mt-1 text-sm font-semibold text-text-main">
{Math.round(normalizedConfig.explorationRate * 100)}%
</p>
</div>
</div>
</Card.Section>
</div>

89
src/app/(dashboard)/dashboard/combos/page.tsx
View File

@@ -140,12 +140,26 @@ const STRATEGY_GUIDANCE_FALLBACK = {
const ADVANCED_FIELD_HELP_FALLBACK = {
maxRetries: "How many retries are attempted before failing the request.",
retryDelay: "Initial delay between retries. Higher values reduce burst pressure.",
timeout: "Maximum request time before aborting. Set higher for long generations.",
healthcheck: "Skips unhealthy models/providers from routing decisions when enabled.",
concurrencyPerModel: "Max simultaneous requests sent to each model in round-robin.",
queueTimeout: "How long a request can wait in queue before timeout in round-robin.",
};
const LEGACY_COMBO_RESILIENCE_KEYS = new Set([
"timeoutMs",
"healthCheckEnabled",
"healthCheckTimeoutMs",
]);
function sanitizeComboRuntimeConfig(config) {
if (!config || typeof config !== "object") return {};
return Object.fromEntries(
Object.entries(config).filter(
([key, value]) =>
value !== undefined && value !== null && !LEGACY_COMBO_RESILIENCE_KEYS.has(key)
)
);
}
const STRATEGY_RECOMMENDATIONS_FALLBACK = {
priority: {
title: "Fail-safe baseline",
@@ -334,7 +348,6 @@ const COMBO_TEMPLATES = [
config: {
maxRetries: 3,
retryDelayMs: 500,
healthCheckEnabled: true,
},
},
{
@@ -349,7 +362,6 @@ const COMBO_TEMPLATES = [
config: {
maxRetries: 2,
retryDelayMs: 1500,
healthCheckEnabled: true,
},
},
{
@@ -364,7 +376,6 @@ const COMBO_TEMPLATES = [
config: {
maxRetries: 1,
retryDelayMs: 500,
healthCheckEnabled: true,
},
},
{
@@ -379,7 +390,6 @@ const COMBO_TEMPLATES = [
config: {
maxRetries: 1,
retryDelayMs: 1000,
healthCheckEnabled: true,
},
},
{
@@ -394,7 +404,6 @@ const COMBO_TEMPLATES = [
config: {
maxRetries: 2,
retryDelayMs: 1000,
healthCheckEnabled: true,
},
},
];
@@ -740,7 +749,7 @@ export default function CombosPage() {
name: newName,
models: combo.models,
strategy: combo.strategy || "priority",
config: combo.config || {},
config: sanitizeComboRuntimeConfig(combo.config),
};
await handleCreate(data);
@@ -1814,7 +1823,7 @@ function ComboFormModal({ isOpen, combo, onClose, onSave, activeProviders }) {
const [builderError, setBuilderError] = useState("");
const [builderStage, setBuilderStage] = useState<string>(COMBO_BUILDER_STAGES[0]);
const [showAdvanced, setShowAdvanced] = useState(false);
const [config, setConfig] = useState(combo?.config || {});
const [config, setConfig] = useState(sanitizeComboRuntimeConfig(combo?.config));
const [showStrategyNudge, setShowStrategyNudge] = useState(false);
const strategyChangeMountedRef = useRef(false);
// Agent features (#399 / #401 / #454)
@@ -1840,8 +1849,10 @@ function ComboFormModal({ isOpen, combo, onClose, onSave, activeProviders }) {
}
: {};
const nextConfig = nextCombo?.config
? { ...nextCombo.config }
: Object.fromEntries(Object.entries(nextDefaults).filter(([key]) => key !== "strategy"));
? sanitizeComboRuntimeConfig(nextCombo.config)
: sanitizeComboRuntimeConfig(
Object.fromEntries(Object.entries(nextDefaults).filter(([key]) => key !== "strategy"))
);
setName(nextCombo?.name || "");
setModels((nextCombo?.models || []).map((m) => normalizeModelEntry(m)));
@@ -2317,25 +2328,23 @@ function ComboFormModal({ isOpen, combo, onClose, onSave, activeProviders }) {
const applyStrategyRecommendations = () => {
const strategyDefaults = {
priority: { maxRetries: 2, retryDelayMs: 1500, healthCheckEnabled: true },
weighted: { maxRetries: 1, retryDelayMs: 1000, healthCheckEnabled: true },
priority: { maxRetries: 2, retryDelayMs: 1500 },
weighted: { maxRetries: 1, retryDelayMs: 1000 },
"round-robin": {
maxRetries: 1,
retryDelayMs: 750,
healthCheckEnabled: true,
concurrencyPerModel: 3,
queueTimeoutMs: 30000,
},
"context-relay": {
maxRetries: 1,
retryDelayMs: 750,
healthCheckEnabled: true,
handoffThreshold: 0.85,
maxMessagesForSummary: 30,
},
random: { maxRetries: 1, retryDelayMs: 1000, healthCheckEnabled: true },
"least-used": { maxRetries: 1, retryDelayMs: 1000, healthCheckEnabled: true },
"cost-optimized": { maxRetries: 1, retryDelayMs: 500, healthCheckEnabled: true },
random: { maxRetries: 1, retryDelayMs: 1000 },
"least-used": { maxRetries: 1, retryDelayMs: 1000 },
"cost-optimized": { maxRetries: 1, retryDelayMs: 500 },
};
const defaults = strategyDefaults[strategy] || strategyDefaults.priority;
@@ -2466,7 +2475,7 @@ function ComboFormModal({ isOpen, combo, onClose, onSave, activeProviders }) {
};
// Include config only if any values are set
const configToSave = { ...config };
const configToSave = sanitizeComboRuntimeConfig(config);
// Add round-robin specific fields to config
if (strategy === "round-robin") {
if (config.concurrencyPerModel !== undefined)
@@ -3225,48 +3234,6 @@ function ComboFormModal({ isOpen, combo, onClose, onSave, activeProviders }) {
className="w-full text-xs py-1.5 px-2 rounded border border-black/10 dark:border-white/10 bg-transparent focus:border-primary focus:outline-none"
/>
</div>
<div>
<FieldLabelWithHelp
label={t("timeout")}
help={getI18nOrFallback(
t,
"advancedHelp.timeout",
ADVANCED_FIELD_HELP_FALLBACK.timeout
)}
/>
<input
type="number"
min="1000"
step="1000"
value={config.timeoutMs ?? ""}
placeholder="120000"
onChange={(e) =>
setConfig({
...config,
timeoutMs: e.target.value ? Number(e.target.value) : undefined,
})
}
className="w-full text-xs py-1.5 px-2 rounded border border-black/10 dark:border-white/10 bg-transparent focus:border-primary focus:outline-none"
/>
</div>
<div className="flex items-center justify-between gap-2">
<FieldLabelWithHelp
label={t("healthcheck")}
help={getI18nOrFallback(
t,
"advancedHelp.healthcheck",
ADVANCED_FIELD_HELP_FALLBACK.healthcheck
)}
/>
<input
type="checkbox"
checked={config.healthCheckEnabled !== false}
onChange={(e) =>
setConfig({ ...config, healthCheckEnabled: e.target.checked })
}
className="accent-primary"
/>
</div>
</div>
{strategy === "round-robin" && (
<div className="grid grid-cols-2 gap-2 pt-2 border-t border-black/5 dark:border-white/5">

108
src/app/(dashboard)/dashboard/endpoint/components/MCPDashboard.tsx
View File

@@ -69,72 +69,90 @@ const AUDIT_PAGE_SIZE = 20;
const RESILIENCE_PRESETS = {
aggressive: {
profiles: {
requestQueue: {
requestsPerMinute: 180,
minTimeBetweenRequestsMs: 100,
concurrentRequests: 16,
},
connectionCooldown: {
oauth: {
transientCooldown: 3000,
rateLimitCooldown: 30000,
maxBackoffLevel: 4,
circuitBreakerThreshold: 2,
circuitBreakerReset: 30000,
baseCooldownMs: 30000,
useUpstreamRetryHints: false,
maxBackoffSteps: 4,
},
apikey: {
transientCooldown: 2000,
rateLimitCooldown: 0,
maxBackoffLevel: 3,
circuitBreakerThreshold: 3,
circuitBreakerReset: 15000,
baseCooldownMs: 2000,
useUpstreamRetryHints: true,
maxBackoffSteps: 3,
},
},
defaults: {
requestsPerMinute: 180,
minTimeBetweenRequests: 100,
concurrentRequests: 16,
providerBreaker: {
oauth: {
failureThreshold: 2,
resetTimeoutMs: 30000,
},
apikey: {
failureThreshold: 3,
resetTimeoutMs: 15000,
},
},
},
balanced: {
profiles: {
requestQueue: {
requestsPerMinute: 100,
minTimeBetweenRequestsMs: 200,
concurrentRequests: 10,
},
connectionCooldown: {
oauth: {
transientCooldown: 5000,
rateLimitCooldown: 60000,
maxBackoffLevel: 8,
circuitBreakerThreshold: 3,
circuitBreakerReset: 60000,
baseCooldownMs: 60000,
useUpstreamRetryHints: false,
maxBackoffSteps: 8,
},
apikey: {
transientCooldown: 3000,
rateLimitCooldown: 0,
maxBackoffLevel: 5,
circuitBreakerThreshold: 5,
circuitBreakerReset: 30000,
baseCooldownMs: 3000,
useUpstreamRetryHints: true,
maxBackoffSteps: 5,
},
},
defaults: {
requestsPerMinute: 100,
minTimeBetweenRequests: 200,
concurrentRequests: 10,
providerBreaker: {
oauth: {
failureThreshold: 3,
resetTimeoutMs: 60000,
},
apikey: {
failureThreshold: 5,
resetTimeoutMs: 30000,
},
},
},
conservative: {
profiles: {
requestQueue: {
requestsPerMinute: 60,
minTimeBetweenRequestsMs: 350,
concurrentRequests: 6,
},
connectionCooldown: {
oauth: {
transientCooldown: 8000,
rateLimitCooldown: 120000,
maxBackoffLevel: 10,
circuitBreakerThreshold: 8,
circuitBreakerReset: 120000,
baseCooldownMs: 120000,
useUpstreamRetryHints: false,
maxBackoffSteps: 10,
},
apikey: {
transientCooldown: 5000,
rateLimitCooldown: 30000,
maxBackoffLevel: 8,
circuitBreakerThreshold: 8,
circuitBreakerReset: 60000,
baseCooldownMs: 30000,
useUpstreamRetryHints: false,
maxBackoffSteps: 8,
},
},
defaults: {
requestsPerMinute: 60,
minTimeBetweenRequests: 350,
concurrentRequests: 6,
providerBreaker: {
oauth: {
failureThreshold: 8,
resetTimeoutMs: 120000,
},
apikey: {
failureThreshold: 8,
resetTimeoutMs: 60000,
},
},
},
} as const;

3
src/app/(dashboard)/dashboard/health/page.tsx
View File

@@ -809,6 +809,9 @@ export default function HealthPage() {
{cb.failures === 1
? t("failures", { count: cb.failures })
: t("failuresPlural", { count: cb.failures })}
{Number(cb.retryAfterMs) > 0 && (
<span className="ml-2">· retry in {fmtMs(cb.retryAfterMs)}</span>
)}
{cb.lastFailure && (
<span className="ml-2">
· {t("lastFailure")}:{" "}

2
src/app/(dashboard)/dashboard/providers/[id]/page.tsx
View File

@@ -53,7 +53,6 @@ import {
getCodexRequestDefaults as _getCodexRequestDefaults,
} from "@/lib/providers/requestDefaults";
import { resolveDashboardProviderInfo } from "../providerPageUtils";
import ModelStatusBadge from "@/app/(dashboard)/dashboard/providers/components/ModelStatusBadge";
type CompatByProtocolMap = Partial<
Record<
@@ -3362,7 +3361,6 @@ function ModelRow({
<code className="rounded bg-sidebar px-1.5 py-0.5 font-mono text-xs text-text-muted">
{fullModel}
</code>
<ModelStatusBadge provider={provider} model={model.id} size="sm" />
<ModelSourceBadge source={model.source} />
<button
onClick={() => onCopy(fullModel, `model-${model.id}`)}

200
src/app/(dashboard)/dashboard/providers/components/ModelAvailabilityBadge.tsx
View File

@@ -1,200 +0,0 @@
"use client";
/**
* ModelAvailabilityBadge — compact inline status indicator
*
* Replaces the full ModelAvailabilityPanel card with a small badge
* that shows green when all models are operational, or amber/red
* when there are issues, with a hover popover for details.
*/
import { useState, useEffect, useCallback, useRef } from "react";
import { useTranslations } from "next-intl";
import { Button } from "@/shared/components";
import { useNotificationStore } from "@/store/notificationStore";
export default function ModelAvailabilityBadge() {
const t = useTranslations("providers");
const tc = useTranslations("common");
const STATUS_CONFIG = {
available: { icon: "check_circle", color: "#22c55e", label: t("available") },
cooldown: { icon: "schedule", color: "#f59e0b", label: t("cooldown") },
unavailable: { icon: "error", color: "#ef4444", label: t("unavailable") },
unknown: { icon: "help", color: "#6b7280", label: t("unknown") },
};
const [data, setData] = useState<any>(null);
const [loading, setLoading] = useState(true);
const [expanded, setExpanded] = useState(false);
const [clearing, setClearing] = useState<string | null>(null);
const ref = useRef<HTMLDivElement>(null);
const notify = useNotificationStore();
const fetchStatus = useCallback(async () => {
try {
const res = await fetch("/api/models/availability");
if (res.ok) {
const json = await res.json();
setData(json);
}
} catch {
// silent fail — will retry
} finally {
setLoading(false);
}
}, []);
useEffect(() => {
fetchStatus();
const interval = setInterval(fetchStatus, 30000);
return () => clearInterval(interval);
}, [fetchStatus]);
// Close popover on outside click
useEffect(() => {
const handleClick = (e: MouseEvent) => {
if (ref.current && !ref.current.contains(e.target as Node)) {
setExpanded(false);
}
};
if (expanded) document.addEventListener("mousedown", handleClick);
return () => document.removeEventListener("mousedown", handleClick);
}, [expanded]);
const handleClearCooldown = async (provider: string, model: string) => {
setClearing(`${provider}:${model}`);
try {
const res = await fetch("/api/models/availability", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ action: "clearCooldown", provider, model }),
});
if (res.ok) {
notify.success(t("cooldownCleared", { model }));
await fetchStatus();
} else {
notify.error(t("failedClearCooldown"));
}
} catch {
notify.error(t("failedClearCooldown"));
} finally {
setClearing(null);
}
};
if (loading) return null;
const models = data?.models || [];
const unavailableCount =
data?.unavailableCount || models.filter((m: any) => m.status !== "available").length;
const isHealthy = unavailableCount === 0;
// Group unhealthy models by provider
const byProvider: Record<string, any[]> = {};
models.forEach((m: any) => {
if (m.status === "available") return;
const key = m.provider || "unknown";
if (!byProvider[key]) byProvider[key] = [];
byProvider[key].push(m);
});
return (
<div className="relative" ref={ref}>
<button
onClick={() => setExpanded(!expanded)}
className={`inline-flex items-center gap-1.5 px-3 py-1.5 rounded-lg text-xs font-medium border transition-all ${
isHealthy
? "bg-emerald-500/10 border-emerald-500/20 text-emerald-500 hover:bg-emerald-500/15"
: "bg-amber-500/10 border-amber-500/20 text-amber-500 hover:bg-amber-500/15"
}`}
>
<span className="material-symbols-outlined text-[14px]" aria-hidden="true">
{isHealthy ? "verified" : "warning"}
</span>
{isHealthy ? t("allModelsOperational") : t("modelsWithIssues", { count: unavailableCount })}
</button>
{/* Expanded popover */}
{expanded && (
<div className="absolute top-full right-0 mt-2 w-80 bg-surface border border-border rounded-xl shadow-2xl z-50 overflow-hidden">
<div className="flex items-center justify-between px-4 py-3 border-b border-border bg-bg">
<div className="flex items-center gap-2">
<span
className="material-symbols-outlined text-[16px]"
style={{ color: isHealthy ? "#22c55e" : "#f59e0b" }}
aria-hidden="true"
>
{isHealthy ? "verified" : "warning"}
</span>
<span className="text-sm font-semibold text-text-main">{t("modelStatus")}</span>
</div>
<button
onClick={fetchStatus}
className="p-1 rounded-lg hover:bg-surface text-text-muted hover:text-text-main transition-colors"
title={tc("refresh")}
>
<span className="material-symbols-outlined text-[14px]" aria-hidden="true">
refresh
</span>
</button>
</div>
<div className="px-4 py-3 max-h-60 overflow-y-auto">
{isHealthy ? (
<p className="text-sm text-text-muted text-center py-2">{t("allModelsNormal")}</p>
) : (
<div className="flex flex-col gap-2.5">
{Object.entries(byProvider).map(([provider, provModels]) => (
<div key={provider}>
<p className="text-xs font-semibold text-text-main mb-1.5 capitalize">
{provider}
</p>
<div className="flex flex-col gap-1">
{provModels.map((m) => {
const status =
STATUS_CONFIG[m.status as keyof typeof STATUS_CONFIG] ||
STATUS_CONFIG.unknown;
const isClearing = clearing === `${m.provider}:${m.model}`;
return (
<div
key={`${m.provider}-${m.model}`}
className="flex items-center justify-between px-2.5 py-1.5 rounded-lg bg-surface/30"
>
<div className="flex items-center gap-1.5 min-w-0">
<span
className="material-symbols-outlined text-[14px] shrink-0"
style={{ color: status.color }}
aria-hidden="true"
>
{status.icon}
</span>
<span className="font-mono text-xs text-text-main truncate">
{m.model}
</span>
</div>
{m.status === "cooldown" && (
<Button
size="sm"
variant="ghost"
onClick={() => handleClearCooldown(m.provider, m.model)}
disabled={isClearing}
className="text-[10px] px-1.5! py-0.5! ml-2"
>
{isClearing ? t("clearing") : t("clearCooldown")}
</Button>
)}
</div>
);
})}
</div>
</div>
))}
</div>
)}
</div>
</div>
)}
</div>
);
}

203
src/app/(dashboard)/dashboard/providers/components/ModelAvailabilityPanel.tsx
View File

@@ -1,203 +0,0 @@
"use client";
/**
* ModelAvailabilityPanel — Batch B
*
* Shows real-time model availability and cooldown status.
* Fetched from /api/models/availability.
*/
import { useState, useEffect, useCallback } from "react";
import { useTranslations } from "next-intl";
import { Card, Button } from "@/shared/components";
import { useNotificationStore } from "@/store/notificationStore";
export default function ModelAvailabilityPanel() {
const t = useTranslations("providers");
const tc = useTranslations("common");
const STATUS_CONFIG = {
available: { icon: "check_circle", color: "#22c55e", label: t("available") },
cooldown: { icon: "schedule", color: "#f59e0b", label: t("cooldown") },
unavailable: { icon: "error", color: "#ef4444", label: t("unavailable") },
unknown: { icon: "help", color: "#6b7280", label: t("unknown") },
};
const [data, setData] = useState<any>(null);
const [loading, setLoading] = useState(true);
const [clearing, setClearing] = useState<string | null>(null);
const notify = useNotificationStore();
const fetchStatus = useCallback(async () => {
try {
const res = await fetch("/api/models/availability");
if (res.ok) {
const json = await res.json();
setData(json);
}
} catch {
// silent fail — will retry
} finally {
setLoading(false);
}
}, []);
useEffect(() => {
fetchStatus();
const interval = setInterval(fetchStatus, 30000);
return () => clearInterval(interval);
}, [fetchStatus]);
const handleClearCooldown = async (provider: string, model: string) => {
setClearing(`${provider}:${model}`);
try {
const res = await fetch("/api/models/availability", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ action: "clearCooldown", provider, model }),
});
if (res.ok) {
notify.success(t("cooldownCleared", { model }));
await fetchStatus();
} else {
notify.error(t("failedClearCooldown"));
}
} catch {
notify.error(t("failedClearCooldown"));
} finally {
setClearing(null);
}
};
if (loading) {
return (
<Card className="p-6 mt-6">
<div className="flex items-center gap-2 text-text-muted animate-pulse">
<span className="material-symbols-outlined text-[20px]" aria-hidden="true">
monitoring
</span>
{t("loadingAvailability")}
</div>
</Card>
);
}
const models = data?.models || [];
const unavailableCount =
data?.unavailableCount || models.filter((m: any) => m.status !== "available").length;
if (models.length === 0 || unavailableCount === 0) {
return (
<Card className="p-6 mt-6">
<div className="flex items-center gap-3 mb-2">
<div className="p-2 rounded-lg bg-emerald-500/10 text-emerald-500">
<span className="material-symbols-outlined text-[20px]" aria-hidden="true">
verified
</span>
</div>
<div>
<h3 className="text-lg font-semibold text-text-main">{t("modelAvailability")}</h3>
<p className="text-sm text-text-muted">{t("allModelsOperational")}</p>
</div>
</div>
</Card>
);
}
// Group by provider
const byProvider: Record<string, any[]> = {};
models.forEach((m: any) => {
if (m.status === "available") return;
const key = m.provider || "unknown";
if (!byProvider[key]) byProvider[key] = [];
byProvider[key].push(m);
});
return (
<Card className="p-6 mt-6">
<div className="flex items-center justify-between mb-4">
<div className="flex items-center gap-3">
<div className="p-2 rounded-lg bg-amber-500/10 text-amber-500">
<span className="material-symbols-outlined text-[20px]" aria-hidden="true">
warning
</span>
</div>
<div>
<h3 className="text-lg font-semibold text-text-main">{t("modelAvailability")}</h3>
<p className="text-sm text-text-muted">
{t("modelsWithIssues", { count: unavailableCount })}
</p>
</div>
</div>
<Button
size="sm"
variant="ghost"
onClick={fetchStatus}
className="text-text-muted"
title={tc("refresh")}
>
<span className="material-symbols-outlined text-[16px]" aria-hidden="true">
refresh
</span>
</Button>
</div>
<div className="flex flex-col gap-3">
{Object.entries(byProvider).map(([provider, provModels]) => (
<div key={provider} className="border border-border/30 rounded-lg p-3">
<p className="text-sm font-medium text-text-main mb-2 capitalize">{provider}</p>
<div className="flex flex-col gap-1.5">
{provModels.map((m) => {
const status =
STATUS_CONFIG[m.status as keyof typeof STATUS_CONFIG] || STATUS_CONFIG.unknown;
const isClearing = clearing === `${m.provider}:${m.model}`;
return (
<div
key={`${m.provider}-${m.model}`}
className="flex items-center justify-between px-3 py-2 rounded-lg bg-surface/30"
>
<div className="flex items-center gap-2">
<span
className="material-symbols-outlined text-[16px]"
style={{ color: status.color }}
aria-hidden="true"
>
{status.icon}
</span>
<span className="font-mono text-sm text-text-main">{m.model}</span>
<span
className="text-xs px-1.5 py-0.5 rounded-full"
style={{
backgroundColor: `${status.color}15`,
color: status.color,
}}
>
{status.label}
</span>
{m.cooldownUntil && (
<span className="text-xs text-text-muted">
{t("until", { time: new Date(m.cooldownUntil).toLocaleTimeString() })}
</span>
)}
</div>
{m.status === "cooldown" && (
<Button
size="sm"
variant="ghost"
onClick={() => handleClearCooldown(m.provider, m.model)}
disabled={isClearing}
className="text-xs"
>
{isClearing ? t("clearing") : t("clearCooldown")}
</Button>
)}
</div>
);
})}
</div>
</div>
))}
</div>
</Card>
);
}

162
src/app/(dashboard)/dashboard/providers/components/ModelStatusBadge.tsx
View File

@@ -1,162 +0,0 @@
"use client";
/**
* ModelStatusBadge — compact single-model status indicator
*
* Shows a small badge with status icon (cooldown/unavailable/error)
* with a tooltip containing additional details like remaining cooldown time
* or last error message.
* Only renders for non-available models to keep the UI clean.
*
* Uses shared polling via ModelStatusContext to avoid redundant API calls.
*/
import { useState, useEffect, useRef } from "react";
import { useTranslations } from "next-intl";
import Tooltip from "@/shared/components/Tooltip";
import { useModelStatus } from "./ModelStatusContext";
interface ModelStatusBadgeProps {
provider: string;
model: string;
size?: "sm" | "md";
className?: string;
}
function formatRemainingTime(ms: number): string {
if (ms <= 0) return "";
const seconds = Math.ceil(ms / 1000);
if (seconds < 60) return `${seconds}s`;
const minutes = Math.floor(seconds / 60);
const remainingSeconds = seconds % 60;
if (minutes < 60) {
return remainingSeconds > 0 ? `${minutes}m ${remainingSeconds}s` : `${minutes}m`;
}
const hours = Math.floor(minutes / 60);
const remainingMinutes = minutes % 60;
return remainingMinutes > 0 ? `${hours}h ${remainingMinutes}m` : `${hours}h`;
}
export default function ModelStatusBadge({
provider,
model,
size = "sm",
className = "",
}: ModelStatusBadgeProps) {
const t = useTranslations("providers");
const status = useModelStatus(provider, model);
// Store the latest remaining time calculated by the interval
const [displayRemainingMs, setDisplayRemainingMs] = useState<number | null>(null);
// Use ref to track server-provided initial cooldown duration for countdown calculation
const initialCooldownMsRef = useRef<number | null>(null);
// Track when we started counting down
const countdownStartRef = useRef<number | null>(null);
// Set up countdown timer when cooldown begins
useEffect(() => {
if (status?.status !== "cooldown" || !status.remainingMs) {
// Reset refs when not in cooldown (no setState here)
initialCooldownMsRef.current = null;
countdownStartRef.current = null;
return;
}
// Initialize timing refs (no setState here)
initialCooldownMsRef.current = status.remainingMs;
countdownStartRef.current = Date.now();
// Update countdown every second via interval callback (setState allowed here)
const interval = setInterval(() => {
if (!initialCooldownMsRef.current || !countdownStartRef.current) {
return;
}
const elapsed = Date.now() - countdownStartRef.current;
const newRemaining = Math.max(0, initialCooldownMsRef.current - elapsed);
setDisplayRemainingMs(newRemaining);
// Stop updating when countdown reaches zero
if (newRemaining === 0) {
clearInterval(interval);
}
}, 1000);
return () => clearInterval(interval);
}, [status?.status, status?.remainingMs]);
// Derive the displayed remaining time: use countdown value if active, else use status value
const remainingMs =
status?.status === "cooldown" ? (displayRemainingMs ?? status.remainingMs ?? null) : null;
// Don't render badge for available models (keep UI clean)
if (!status || status.status === "available" || status.status === "unknown") {
return null;
}
const getStatusColor = () => {
switch (status.status) {
case "cooldown":
return "#f59e0b";
case "unavailable":
case "error":
return "#ef4444";
default:
return "#6b7280";
}
};
const getStatusIcon = () => {
switch (status.status) {
case "cooldown":
return "schedule";
case "unavailable":
case "error":
return "error";
default:
return "help";
}
};
const getTooltipText = () => {
switch (status.status) {
case "cooldown": {
const remaining = remainingMs !== null ? formatRemainingTime(remainingMs) : "";
const reason = status.reason ? ` (${status.reason})` : "";
const remainingText = remaining ? ` - ${remaining}` : "";
return `${t("cooldown")}${reason}${remainingText}`;
}
case "unavailable":
return `${t("unavailable")}${status.reason ? `: ${status.reason}` : ""}`;
case "error":
return `${t("error")}${status.lastError ? `: ${status.lastError}` : ""}`;
default:
return "";
}
};
const color = getStatusColor();
const sizeClasses = size === "sm" ? "px-1.5 py-0.5" : "px-2 py-1";
const iconSize = size === "sm" ? "text-[12px]" : "text-[14px]";
return (
<Tooltip content={getTooltipText()} position="top" delayMs={200}>
<span
className={`inline-flex items-center gap-1 rounded-full text-[10px] font-semibold ${sizeClasses} ${className}`}
style={{
backgroundColor: `${color}15`,
color: color,
}}
>
<span className={`material-symbols-outlined ${iconSize}`} aria-hidden="true">
{getStatusIcon()}
</span>
{status.status === "cooldown" && remainingMs !== null && (
<span>{formatRemainingTime(remainingMs)}</span>
)}
</span>
</Tooltip>
);
}

186
src/app/(dashboard)/dashboard/providers/components/ModelStatusContext.tsx
View File

@@ -1,186 +0,0 @@
"use client";
/**
* ModelStatusContext — shared polling for model availability
*
* Prevents redundant API calls by having all ModelStatusBadge components
* share a single polling interval. Only one request is made every 15 seconds
* regardless of how many badges are on the page.
*/
import React, { createContext, useContext, useEffect, useRef, useCallback, useMemo } from "react";
export interface ModelStatus {
status: "available" | "cooldown" | "unavailable" | "error" | "unknown";
reason?: string;
remainingMs?: number;
lastError?: string;
}
interface ModelStatusContextValue {
getStatus: (provider: string, model: string) => ModelStatus | null;
registerModel: (key: string, provider: string, model: string) => void;
unregisterModel: (key: string) => void;
}
const ModelStatusContext = createContext<ModelStatusContextValue | null>(null);
// Global map of model key -> status
let modelStatusMap = new Map<string, ModelStatus>();
// Global set of registered model keys
let registeredModels = new Set<string>();
// Polling interval ref (singleton)
let pollIntervalRef: NodeJS.Timeout | null = null;
function getModelKey(provider: string, model: string): string {
return `${provider}/${model}`;
}
async function fetchModelStatus(): Promise<void> {
try {
const res = await fetch("/api/models/availability");
if (!res.ok) return;
const json = await res.json();
const models = json?.models || [];
// Update all registered models with fresh data
const now = Date.now();
registeredModels.forEach((key) => {
const [provider, model] = key.split("/");
// Use exact matching first to avoid gpt-4 matching gpt-4-turbo incorrectly
const modelEntry =
models.find((m: any) => m.provider === provider && m.model === model) ||
models.find(
// Fallback to prefix matching only for models that contain the registered key
// This handles cases like "gpt-4o" matching badge for "gpt-4"
(m: any) =>
m.provider === provider &&
m.model &&
model &&
(m.model.startsWith(model + "-") || model.startsWith(m.model + "-"))
);
if (modelEntry) {
const newStatus: ModelStatus = {
status: modelEntry.status || "unknown",
reason: modelEntry.reason,
remainingMs: modelEntry.remainingMs,
lastError: modelEntry.lastError,
};
// For cooldown status, calculate remaining time based on server-provided value
if (modelEntry.status === "cooldown" && modelEntry.remainingMs) {
newStatus.remainingMs = modelEntry.remainingMs;
}
modelStatusMap.set(key, newStatus);
} else {
modelStatusMap.set(key, { status: "available" });
}
});
// Trigger re-render by dispatching custom event
window.dispatchEvent(new CustomEvent("model-status-update"));
} catch {
// Best-effort polling
}
}
function ensurePolling(): void {
if (pollIntervalRef) return;
// Initial fetch
fetchModelStatus();
// Poll every 15 seconds
pollIntervalRef = setInterval(fetchModelStatus, 15000);
}
function stopPolling(): void {
if (pollIntervalRef) {
clearInterval(pollIntervalRef);
pollIntervalRef = null;
}
}
export function ModelStatusProvider({ children }: { children: React.ReactNode }) {
const [, forceUpdate] = React.useState(0);
// Listen for status updates from the global poller
useEffect(() => {
const handleUpdate = () => forceUpdate((n) => n + 1);
window.addEventListener("model-status-update", handleUpdate);
return () => window.removeEventListener("model-status-update", handleUpdate);
}, []);
// Cleanup on unmount — stop polling only when no models remain registered
useEffect(() => {
return () => {
if (registeredModels.size === 0) {
stopPolling();
}
};
}, []);
const getStatus = useCallback((provider: string, model: string): ModelStatus | null => {
const key = getModelKey(provider, model);
return modelStatusMap.get(key) || null;
}, []);
const registerModel = useCallback((key: string, provider: string, model: string): void => {
const wasEmpty = registeredModels.size === 0;
registeredModels.add(key);
// Start polling when first model registers
if (wasEmpty) {
ensurePolling();
}
// Immediately fetch if no data yet
if (!modelStatusMap.has(key)) {
fetchModelStatus();
}
}, []);
const unregisterModel = useCallback((key: string): void => {
registeredModels.delete(key);
modelStatusMap.delete(key);
// Stop polling when last model unregisters
if (registeredModels.size === 0) {
stopPolling();
}
}, []);
const value = useMemo(
() => ({
getStatus,
registerModel,
unregisterModel,
}),
[getStatus, registerModel, unregisterModel]
);
return <ModelStatusContext.Provider value={value}>{children}</ModelStatusContext.Provider>;
}
export function useModelStatus(provider: string, model: string): ModelStatus | null {
const context = useContext(ModelStatusContext);
if (!context) {
throw new Error("useModelStatus must be used within a ModelStatusProvider");
}
const key = useMemo(() => getModelKey(provider, model), [provider, model]);
// Register/unregister on mount/unmount
useEffect(() => {
context.registerModel(key, provider, model);
return () => context.unregisterModel(key);
}, [context, key, provider, model]);
return context.getStatus(provider, model);
}
export default ModelStatusContext;

2
src/app/(dashboard)/dashboard/providers/page.tsx
View File

@@ -26,7 +26,6 @@ import { getErrorCode, getRelativeTime } from "@/shared/utils";
import { pickDisplayValue } from "@/shared/utils/maskEmail";
import useEmailPrivacyStore from "@/store/emailPrivacyStore";
import { useNotificationStore } from "@/store/notificationStore";
import ModelAvailabilityBadge from "./components/ModelAvailabilityBadge";
import { useTranslations } from "next-intl";
import {
buildMergedOAuthProviderEntries,
@@ -496,7 +495,6 @@ export default function ProvidersPage() {
<span className="size-2.5 rounded-full bg-blue-500" title={t("oauthLabel")} />
</h2>
<div className="flex items-center gap-2">
<ModelAvailabilityBadge />
<Toggle
size="sm"
checked={showConfiguredOnly}

76
src/app/(dashboard)/dashboard/settings/components/ComboDefaultsTab.tsx
View File

@@ -10,6 +10,12 @@ const STRATEGY_LABEL_FALLBACKS: Record<string, string> = {
"context-relay": "Context Relay",
};
const LEGACY_COMBO_RESILIENCE_KEYS = new Set([
"timeoutMs",
"healthCheckEnabled",
"healthCheckTimeoutMs",
]);
function translateOrFallback(
t: ReturnType<typeof useTranslations>,
key: string,
@@ -18,14 +24,31 @@ function translateOrFallback(
return typeof t.has === "function" && t.has(key) ? t(key) : fallback;
}
function sanitizeComboRuntimeConfig(config?: Record<string, any> | null) {
if (!config || typeof config !== "object") return {};
return Object.fromEntries(
Object.entries(config).filter(
([key, value]) =>
value !== undefined && value !== null && !LEGACY_COMBO_RESILIENCE_KEYS.has(key)
)
);
}
function sanitizeProviderOverrides(overrides?: Record<string, any> | null) {
if (!overrides || typeof overrides !== "object") return {};
return Object.fromEntries(
Object.entries(overrides).map(([providerId, config]) => [
providerId,
sanitizeComboRuntimeConfig(config),
])
);
}
export default function ComboDefaultsTab() {
const [comboDefaults, setComboDefaults] = useState<any>({
strategy: "priority",
maxRetries: 1,
retryDelayMs: 2000,
timeoutMs: 120000,
healthCheckEnabled: true,
healthCheckTimeoutMs: 3000,
maxComboDepth: 3,
trackMetrics: true,
handoffThreshold: 0.85,
@@ -54,7 +77,6 @@ export default function ComboDefaultsTab() {
const numericSettings = [
{ key: "maxRetries", label: t("maxRetriesLabel"), min: 0, max: 5 },
{ key: "retryDelayMs", label: t("retryDelayLabel"), min: 500, max: 10000, step: 500 },
{ key: "timeoutMs", label: t("timeoutLabel"), min: 5000, step: 5000 },
{ key: "maxComboDepth", label: t("maxNestingDepth"), min: 1, max: 10 },
];
@@ -66,7 +88,7 @@ export default function ComboDefaultsTab() {
.then(([comboData, settingsData]) => {
setComboDefaults((prev) => ({
...prev,
...(comboData.comboDefaults || {}),
...sanitizeComboRuntimeConfig(comboData.comboDefaults),
strategy:
settingsData.fallbackStrategy ?? comboData.comboDefaults?.strategy ?? prev.strategy,
stickyRoundRobinLimit:
@@ -74,7 +96,9 @@ export default function ComboDefaultsTab() {
comboData.comboDefaults?.stickyRoundRobinLimit ??
prev.stickyRoundRobinLimit,
}));
if (comboData.providerOverrides) setProviderOverrides(comboData.providerOverrides);
if (comboData.providerOverrides) {
setProviderOverrides(sanitizeProviderOverrides(comboData.providerOverrides));
}
})
.catch((err) => console.error("Failed to fetch combo defaults:", err));
}, []);
@@ -116,7 +140,10 @@ export default function ComboDefaultsTab() {
const comboDefaultsRes = await fetch("/api/settings/combo-defaults", {
method: "PATCH",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ comboDefaults: comboDefaultsPayload, providerOverrides }),
body: JSON.stringify({
comboDefaults: sanitizeComboRuntimeConfig(comboDefaultsPayload),
providerOverrides: sanitizeProviderOverrides(providerOverrides),
}),
});
if (!comboDefaultsRes.ok) {
@@ -136,7 +163,7 @@ export default function ComboDefaultsTab() {
const addProviderOverride = () => {
const name = newOverrideProvider.trim().toLowerCase();
if (!name || providerOverrides[name]) return;
setProviderOverrides((prev) => ({ ...prev, [name]: { maxRetries: 1, timeoutMs: 120000 } }));
setProviderOverrides((prev) => ({ ...prev, [name]: { maxRetries: 1 } }));
setNewOverrideProvider("");
};
@@ -381,21 +408,6 @@ export default function ComboDefaultsTab() {
{/* Toggles */}
<div className="flex flex-col gap-3 pt-3 border-t border-border/50">
<div className="flex items-center justify-between">
<div>
<p className="font-medium text-sm">{t("healthCheck")}</p>
<p className="text-xs text-text-muted">{t("healthCheckDesc")}</p>
</div>
<Toggle
checked={comboDefaults.healthCheckEnabled !== false}
onChange={() =>
setComboDefaults((prev) => ({
...prev,
healthCheckEnabled: !prev.healthCheckEnabled,
}))
}
/>
</div>
<div className="flex items-center justify-between">
<div>
<p className="font-medium text-sm">{t("trackMetrics")}</p>
@@ -436,24 +448,6 @@ export default function ComboDefaultsTab() {
aria-label={t("providerMaxRetriesAria", { provider })}
/>
<span className="text-[10px] text-text-muted">{t("retries")}</span>
<Input
type="number"
min="5000"
step="5000"
value={config.timeoutMs ?? 120000}
onChange={(e) =>
setProviderOverrides((prev) => ({
...prev,
[provider]: {
...prev[provider],
timeoutMs: parseInt(e.target.value) || 120000,
},
}))
}
className="text-xs w-24"
aria-label={t("providerTimeoutAria", { provider })}
/>
<span className="text-[10px] text-text-muted">{t("ms")}</span>
<button
onClick={() => removeProviderOverride(provider)}
className="ml-auto text-red-400 hover:text-red-500 transition-colors"

62
src/app/(dashboard)/dashboard/settings/components/PoliciesPanel.tsx
View File

@@ -3,22 +3,16 @@
/**
* PoliciesPanel — Batch E
*
* Shows circuit breaker states and locked identifiers.
* Shows locked identifiers managed by policy-related safeguards.
* Allows force-unlocking locked identifiers.
* API: /api/policies
*/
import { useState, useEffect, useCallback } from "react";
import { Card, Button, EmptyState } from "@/shared/components";
import { Card, Button } from "@/shared/components";
import { useNotificationStore } from "@/store/notificationStore";
import { useTranslations } from "next-intl";
const CB_STATUS = {
closed: { icon: "check_circle", color: "#22c55e", label: "Closed" },
"half-open": { icon: "pending", color: "#f59e0b", label: "Half-Open" },
open: { icon: "error", color: "#ef4444", label: "Open" },
};
export default function PoliciesPanel() {
const [data, setData] = useState(null);
const [loading, setLoading] = useState(true);
@@ -78,11 +72,9 @@ export default function PoliciesPanel() {
);
}
const circuitBreakers = data?.circuitBreakers || [];
const lockedIds = data?.lockedIdentifiers || [];
const hasIssues = circuitBreakers.some((cb) => cb.state !== "closed") || lockedIds.length > 0;
if (!hasIssues) {
if (lockedIds.length === 0) {
return (
<Card className="p-6 mt-6">
<div className="flex items-center gap-3">
@@ -90,7 +82,7 @@ export default function PoliciesPanel() {
<span className="material-symbols-outlined text-[20px]">verified_user</span>
</div>
<div>
<h3 className="text-lg font-semibold text-text-main">{t("policiesCircuitBreakers")}</h3>
<h3 className="text-lg font-semibold text-text-main">{t("policiesLocked")}</h3>
<p className="text-sm text-text-muted">{t("allOperational")}</p>
</div>
</div>
@@ -106,7 +98,7 @@ export default function PoliciesPanel() {
<span className="material-symbols-outlined text-[20px]">gpp_maybe</span>
</div>
<div>
<h3 className="text-lg font-semibold text-text-main">{t("policiesCircuitBreakers")}</h3>
<h3 className="text-lg font-semibold text-text-main">{t("policiesLocked")}</h3>
<p className="text-sm text-text-muted">{t("activeIssuesDetected")}</p>
</div>
</div>
@@ -115,50 +107,6 @@ export default function PoliciesPanel() {
</Button>
</div>
{/* Circuit Breakers */}
{circuitBreakers.filter((cb) => cb.state !== "closed").length > 0 && (
<div className="mb-4">
<p className="text-sm font-medium text-text-muted mb-2">{t("circuitBreakers")}</p>
<div className="flex flex-col gap-1.5">
{circuitBreakers
.filter((cb) => cb.state !== "closed")
.map((cb, i) => {
const status = CB_STATUS[cb.state] || CB_STATUS.open;
return (
<div
key={cb.name || i}
className="flex items-center justify-between px-3 py-2 rounded-lg bg-surface/30 border border-border/20"
>
<div className="flex items-center gap-2">
<span
className="material-symbols-outlined text-[16px]"
style={{ color: status.color }}
>
{status.icon}
</span>
<span className="text-sm text-text-main font-medium">
{cb.name || cb.provider || "Unknown"}
</span>
<span
className="text-xs px-1.5 py-0.5 rounded-full"
style={{
backgroundColor: `${status.color}15`,
color: status.color,
}}
>
{status.label}
</span>
{cb.failures > 0 && (
<span className="text-xs text-text-muted">{cb.failures} failures</span>
)}
</div>
</div>
);
})}
</div>
</div>
)}
{/* Locked Identifiers */}
{lockedIds.length > 0 && (
<div>

1301
src/app/(dashboard)/dashboard/settings/components/ResilienceTab.tsx
View File

File diff suppressed because it is too large Load Diff

7
src/app/(dashboard)/layout.tsx
View File

@@ -1,10 +1,5 @@
import { DashboardLayout } from "@/shared/components";
import { ModelStatusProvider } from "./dashboard/providers/components/ModelStatusContext";
export default function DashboardRootLayout({ children }) {
return (
<ModelStatusProvider>
<DashboardLayout>{children}</DashboardLayout>
</ModelStatusProvider>
);
return <DashboardLayout>{children}</DashboardLayout>;
}

50
src/app/api/models/availability/route.ts
View File

@@ -1,50 +0,0 @@
import { NextResponse } from "next/server";
import {
getAvailabilityReport,
clearModelUnavailability,
getUnavailableCount,
} from "@/domain/modelAvailability";
import { clearModelAvailabilitySchema } from "@/shared/validation/schemas";
import { isValidationFailure, validateBody } from "@/shared/validation/helpers";
export async function GET() {
try {
const report = getAvailabilityReport();
const count = getUnavailableCount();
return NextResponse.json({ unavailableCount: count, models: report });
} catch (error) {
console.error("Error getting model availability:", error);
return NextResponse.json({ error: "Failed to get model availability" }, { status: 500 });
}
}
export async function POST(request) {
let rawBody;
try {
rawBody = await request.json();
} catch {
return NextResponse.json(
{
error: {
message: "Invalid request",
details: [{ field: "body", message: "Invalid JSON body" }],
},
},
{ status: 400 }
);
}
try {
const validation = validateBody(clearModelAvailabilitySchema, rawBody);
if (isValidationFailure(validation)) {
return NextResponse.json({ error: validation.error }, { status: 400 });
}
const { provider, model } = validation.data;
const removed = clearModelUnavailability(provider, model);
return NextResponse.json({ success: true, removed });
} catch (error) {
console.error("Error clearing model availability:", error);
return NextResponse.json({ error: "Failed to clear model availability" }, { status: 500 });
}
}

4
src/app/api/policies/route.ts
View File

@@ -1,14 +1,12 @@
import { NextResponse } from "next/server";
import { getAllCircuitBreakerStatuses } from "@/shared/utils/circuitBreaker";
import { getLockedIdentifiers, forceUnlock } from "@/domain/lockoutPolicy";
import { policyActionSchema } from "@/shared/validation/schemas";
import { isValidationFailure, validateBody } from "@/shared/validation/helpers";
export async function GET() {
try {
const circuitBreakers = getAllCircuitBreakerStatuses();
const lockedIdentifiers = getLockedIdentifiers();
return NextResponse.json({ circuitBreakers, lockedIdentifiers });
return NextResponse.json({ lockedIdentifiers });
} catch (error) {
console.error("Error loading policies:", error);
return NextResponse.json({ error: "Failed to load policies" }, { status: 500 });

3
src/app/api/resilience/reset/route.ts
View File

@@ -1,11 +1,10 @@
import { NextResponse } from "next/server";
/**
* POST /api/resilience/reset — Reset all circuit breakers and clear cooldowns
* POST /api/resilience/reset — Reset all provider circuit breakers
*/
export async function POST() {
try {
// Reset all circuit breakers
const { getAllCircuitBreakerStatuses, getCircuitBreaker } =
await import("@/shared/utils/circuitBreaker");

173
src/app/api/resilience/route.ts
View File

@@ -1,5 +1,12 @@
import { NextResponse } from "next/server";
import { getSettings, updateSettings } from "@/lib/localDb";
import {
buildLegacyResilienceCompat,
mergeResilienceSettings,
resolveResilienceSettings,
type ResilienceSettings,
type ResilienceSettingsPatch,
} from "@/lib/resilience/settings";
import { updateResilienceSchema } from "@/shared/validation/schemas";
import { isValidationFailure, validateBody } from "@/shared/validation/helpers";
@@ -13,41 +20,130 @@ function getErrorMessage(error: unknown, fallback: string): string {
return error instanceof Error && error.message ? error.message : fallback;
}
function normalizeLegacyPatch(body: JsonRecord): ResilienceSettingsPatch {
const profiles = asRecord(body.profiles);
const defaults = asRecord(body.defaults);
const oauth = asRecord(profiles.oauth);
const apikey = asRecord(profiles.apikey);
const patch: ResilienceSettingsPatch = {};
if (Object.keys(defaults).length > 0) {
patch.requestQueue = {
...(typeof defaults.requestsPerMinute === "number"
? { requestsPerMinute: defaults.requestsPerMinute }
: {}),
...(typeof defaults.minTimeBetweenRequests === "number"
? { minTimeBetweenRequestsMs: defaults.minTimeBetweenRequests }
: {}),
...(typeof defaults.concurrentRequests === "number"
? { concurrentRequests: defaults.concurrentRequests }
: {}),
};
}
if (Object.keys(oauth).length > 0 || Object.keys(apikey).length > 0) {
const buildLegacyCooldownPatch = (profile: JsonRecord) => {
const cooldownCandidates = [
typeof profile.transientCooldown === "number" ? profile.transientCooldown : null,
typeof profile.rateLimitCooldown === "number" && profile.rateLimitCooldown > 0
? profile.rateLimitCooldown
: null,
].filter((value): value is number => typeof value === "number");
return {
...(cooldownCandidates.length > 0
? { baseCooldownMs: Math.max(...cooldownCandidates) }
: {}),
...(typeof profile.rateLimitCooldown === "number"
? { useUpstreamRetryHints: profile.rateLimitCooldown === 0 }
: {}),
...(typeof profile.maxBackoffLevel === "number"
? { maxBackoffSteps: profile.maxBackoffLevel }
: {}),
};
};
patch.connectionCooldown = {
...(Object.keys(oauth).length > 0
? {
oauth: buildLegacyCooldownPatch(oauth),
}
: {}),
...(Object.keys(apikey).length > 0
? {
apikey: buildLegacyCooldownPatch(apikey),
}
: {}),
};
patch.providerBreaker = {
...(Object.keys(oauth).length > 0
? {
oauth: {
...(typeof oauth.circuitBreakerThreshold === "number"
? { failureThreshold: oauth.circuitBreakerThreshold }
: {}),
...(typeof oauth.circuitBreakerReset === "number"
? { resetTimeoutMs: oauth.circuitBreakerReset }
: {}),
},
}
: {}),
...(Object.keys(apikey).length > 0
? {
apikey: {
...(typeof apikey.circuitBreakerThreshold === "number"
? { failureThreshold: apikey.circuitBreakerThreshold }
: {}),
...(typeof apikey.circuitBreakerReset === "number"
? { resetTimeoutMs: apikey.circuitBreakerReset }
: {}),
},
}
: {}),
};
}
return patch;
}
async function syncRuntimeSettings(resilienceSettings: ResilienceSettings) {
const { applyRequestQueueSettings } =
await import("@omniroute/open-sse/services/rateLimitManager");
applyRequestQueueSettings(resilienceSettings.requestQueue);
}
/**
* GET /api/resilience — Get current resilience configuration and status
* GET /api/resilience — Get current resilience configuration
*/
export async function GET() {
try {
// Dynamic imports for open-sse modules
const { getAllCircuitBreakerStatuses } = await import("@/shared/utils/circuitBreaker");
const { getAllRateLimitStatus } = await import("@omniroute/open-sse/services/rateLimitManager");
const { PROVIDER_PROFILES, DEFAULT_API_LIMITS } =
await import("@omniroute/open-sse/config/constants");
const settings = await getSettings();
const circuitBreakers = getAllCircuitBreakerStatuses();
const rateLimitStatus = getAllRateLimitStatus();
const resilience = resolveResilienceSettings(settings);
return NextResponse.json({
profiles: settings.providerProfiles || PROVIDER_PROFILES,
defaults: {
...DEFAULT_API_LIMITS,
...asRecord(settings.rateLimitDefaults),
requestQueue: resilience.requestQueue,
connectionCooldown: resilience.connectionCooldown,
providerBreaker: resilience.providerBreaker,
waitForCooldown: {
enabled: resilience.waitForCooldown.enabled,
maxRetries: resilience.waitForCooldown.maxRetries,
maxRetryWaitSec: resilience.waitForCooldown.maxRetryWaitSec,
},
circuitBreakers,
rateLimitStatus,
legacy: buildLegacyResilienceCompat(resilience),
});
} catch (err: unknown) {
console.error("[API] GET /api/resilience error:", err);
return NextResponse.json(
{ error: getErrorMessage(err, "Failed to load resilience status") },
{ error: getErrorMessage(err, "Failed to load resilience settings") },
{ status: 500 }
);
}
}
/**
* PATCH /api/resilience — Update provider resilience profiles and/or rate limit defaults
* PATCH /api/resilience — Update resilience configuration
*/
export async function PATCH(request) {
let rawBody;
@@ -70,18 +166,47 @@ export async function PATCH(request) {
if (isValidationFailure(validation)) {
return NextResponse.json({ error: validation.error }, { status: 400 });
}
const { profiles, defaults } = validation.data;
const updates: Record<string, any> = {};
if (profiles) updates.providerProfiles = profiles;
if (defaults) updates.rateLimitDefaults = defaults;
const body = validation.data as JsonRecord;
const currentSettings = await getSettings();
const currentResilience = resolveResilienceSettings(currentSettings);
const nextResilience = mergeResilienceSettings(currentResilience, {
...(body.requestQueue
? { requestQueue: body.requestQueue as ResilienceSettingsPatch["requestQueue"] }
: {}),
...(body.connectionCooldown
? {
connectionCooldown:
body.connectionCooldown as ResilienceSettingsPatch["connectionCooldown"],
}
: {}),
...(body.providerBreaker
? { providerBreaker: body.providerBreaker as ResilienceSettingsPatch["providerBreaker"] }
: {}),
...(body.waitForCooldown
? { waitForCooldown: body.waitForCooldown as ResilienceSettingsPatch["waitForCooldown"] }
: {}),
...normalizeLegacyPatch(body),
});
await updateSettings(updates);
await updateSettings({
resilienceSettings: nextResilience,
requestRetry: nextResilience.waitForCooldown.maxRetries,
maxRetryIntervalSec: nextResilience.waitForCooldown.maxRetryWaitSec,
});
await syncRuntimeSettings(nextResilience);
return NextResponse.json({
ok: true,
...(profiles ? { profiles } : {}),
...(defaults ? { defaults } : {}),
requestQueue: nextResilience.requestQueue,
connectionCooldown: nextResilience.connectionCooldown,
providerBreaker: nextResilience.providerBreaker,
waitForCooldown: {
enabled: nextResilience.waitForCooldown.enabled,
maxRetries: nextResilience.waitForCooldown.maxRetries,
maxRetryWaitSec: nextResilience.waitForCooldown.maxRetryWaitSec,
},
legacy: buildLegacyResilienceCompat(nextResilience),
});
} catch (err: unknown) {
console.error("[API] PATCH /api/resilience error:", err);

64
src/app/api/settings/combo-defaults/route.ts
View File

@@ -3,6 +3,32 @@ import { getSettings, updateSettings } from "@/lib/localDb";
import { updateComboDefaultsSchema } from "@/shared/validation/schemas";
import { isValidationFailure, validateBody } from "@/shared/validation/helpers";
const LEGACY_COMBO_RESILIENCE_KEYS = new Set([
"timeoutMs",
"healthCheckEnabled",
"healthCheckTimeoutMs",
]);
function sanitizeComboRuntimeConfig(config?: Record<string, any> | null) {
if (!config || typeof config !== "object") return {};
return Object.fromEntries(
Object.entries(config).filter(
([key, value]) =>
value !== undefined && value !== null && !LEGACY_COMBO_RESILIENCE_KEYS.has(key)
)
);
}
function sanitizeProviderOverrides(overrides?: Record<string, any> | null) {
if (!overrides || typeof overrides !== "object") return {};
return Object.fromEntries(
Object.entries(overrides).map(([providerId, config]) => [
providerId,
sanitizeComboRuntimeConfig(config),
])
);
}
/**
* GET /api/settings/combo-defaults
* Returns the current combo global defaults and provider overrides
@@ -10,21 +36,23 @@ import { isValidationFailure, validateBody } from "@/shared/validation/helpers";
export async function GET() {
try {
const settings: any = await getSettings();
const comboDefaults = sanitizeComboRuntimeConfig(settings.comboDefaults);
const providerOverrides = sanitizeProviderOverrides(settings.providerOverrides);
return NextResponse.json({
comboDefaults: settings.comboDefaults || {
strategy: "priority",
maxRetries: 1,
retryDelayMs: 2000,
timeoutMs: 120000,
healthCheckEnabled: true,
healthCheckTimeoutMs: 3000,
handoffThreshold: 0.85,
handoffModel: "",
maxMessagesForSummary: 30,
maxComboDepth: 3,
trackMetrics: true,
},
providerOverrides: settings.providerOverrides || {},
comboDefaults:
Object.keys(comboDefaults).length > 0
? comboDefaults
: {
strategy: "priority",
maxRetries: 1,
retryDelayMs: 2000,
handoffThreshold: 0.85,
handoffModel: "",
maxMessagesForSummary: 30,
maxComboDepth: 3,
trackMetrics: true,
},
providerOverrides,
});
} catch (error) {
console.log("Error fetching combo defaults:", error);
@@ -63,16 +91,16 @@ export async function PATCH(request) {
const updates: Record<string, any> = {};
if (body.comboDefaults) {
updates.comboDefaults = body.comboDefaults;
updates.comboDefaults = sanitizeComboRuntimeConfig(body.comboDefaults);
}
if (body.providerOverrides) {
updates.providerOverrides = body.providerOverrides;
updates.providerOverrides = sanitizeProviderOverrides(body.providerOverrides);
}
const settings: any = await updateSettings(updates);
return NextResponse.json({
comboDefaults: settings.comboDefaults || {},
providerOverrides: settings.providerOverrides || {},
comboDefaults: sanitizeComboRuntimeConfig(settings.comboDefaults),
providerOverrides: sanitizeProviderOverrides(settings.providerOverrides),
});
} catch (error) {
console.log("Error updating combo defaults:", error);

319
src/domain/modelAvailability.ts
View File

@@ -1,319 +0,0 @@
/**
* Model Availability — Domain Layer (T-19)
*
* Tracks model availability per provider with TTL-based cooldowns.
* When a model becomes unavailable (rate-limited, erroring), it is
* marked with a cooldown period. The availability report powers
* the dashboard health view.
*
* @module domain/modelAvailability
*/
/**
* @typedef {Object} UnavailableEntry
* @property {string} provider
* @property {string} model
* @property {number} unavailableSince - timestamp
* @property {number} cooldownMs
* @property {string} [reason]
*/
/** @type {Map<string, UnavailableEntry>} */
const unavailable = new Map();
/**
* @typedef {Object} FailureState
* @property {number} failureCount
* @property {number} lastFailureAt
* @property {number} resetAfterMs
*/
/**
* @typedef {Object} ProviderProfile
* @property {number} [transientCooldown]
* @property {number} [rateLimitCooldown]
* @property {number} [maxBackoffLevel]
* @property {number} [circuitBreakerThreshold]
* @property {number} [circuitBreakerReset]
*/
/** @type {Map<string, FailureState>} */
const failureState = new Map();
const FAILURE_WINDOW_MS = 30 * 60 * 1000;
const PROBLEMATIC_STATUS_COOLDOWNS = {
429: 5 * 60 * 1000,
408: 60 * 1000,
500: 2 * 60 * 1000,
502: 2 * 60 * 1000,
503: 2 * 60 * 1000,
504: 2 * 60 * 1000,
};
const MIN_PROBLEMATIC_COOLDOWN_MS = 60 * 1000;
const MAX_PROBLEMATIC_COOLDOWN_MS = 30 * 60 * 1000;
function toPositiveNumber(value) {
return Number.isFinite(value) && Number(value) > 0 ? Number(value) : null;
}
function toNonNegativeNumber(value) {
return Number.isFinite(value) && Number(value) >= 0 ? Number(value) : null;
}
/**
* The first layer already reacts immediately to authoritative model/account failures.
* Global provider/model quarantine is the escalation layer, so its failure window and
* threshold are only customized when a runtime provider profile is supplied.
*
* @param {ProviderProfile | null | undefined} profile
* @returns {number}
*/
function getFailureWindowMs(profile) {
return toPositiveNumber(profile?.circuitBreakerReset) ?? FAILURE_WINDOW_MS;
}
/**
* Without a runtime profile we preserve legacy behavior: quarantine on the first failure.
*
* @param {ProviderProfile | null | undefined} profile
* @returns {number}
*/
function getFailureThreshold(profile) {
return toPositiveNumber(profile?.circuitBreakerThreshold) ?? 1;
}
function getLegacyStatusCooldown(status) {
return status && Object.prototype.hasOwnProperty.call(PROBLEMATIC_STATUS_COOLDOWNS, status)
? PROBLEMATIC_STATUS_COOLDOWNS[status]
: 0;
}
/**
* @param {number | null} status
* @param {ProviderProfile | null | undefined} profile
* @returns {number}
*/
function getProfileStatusCooldown(status, profile) {
if (!profile) return 0;
if (status === 429) {
return toPositiveNumber(profile.rateLimitCooldown) ?? 0;
}
return toPositiveNumber(profile.transientCooldown) ?? 0;
}
/**
* @param {number} baseCooldownMs
* @param {number} failureCount
* @param {ProviderProfile | null | undefined} profile
* @returns {number}
*/
function getScaledCooldown(baseCooldownMs, failureCount, profile) {
const safeBase = toPositiveNumber(baseCooldownMs) ?? 1000;
if (!profile) {
return Math.min(
Math.max(safeBase, MIN_PROBLEMATIC_COOLDOWN_MS) * Math.pow(2, Math.max(0, failureCount - 1)),
MAX_PROBLEMATIC_COOLDOWN_MS
);
}
const maxBackoffLevel = Math.max(
0,
Math.trunc(toNonNegativeNumber(profile.maxBackoffLevel) ?? 0)
);
const exponent = Math.min(Math.max(0, failureCount - 1), maxBackoffLevel);
return safeBase * Math.pow(2, exponent);
}
/**
* Build a composite key for provider+model.
* @param {string} provider
* @param {string} model
* @returns {string}
*/
function makeKey(provider, model) {
return `${provider}::${model}`;
}
/**
* Check if a model is currently available.
*
* @param {string} provider - Provider ID (e.g. "openai", "anthropic")
* @param {string} model - Model ID (e.g. "gpt-4o", "claude-sonnet-4-20250514")
* @returns {boolean} true if model is available (not in cooldown)
*/
export function isModelAvailable(provider, model) {
const key = makeKey(provider, model);
const entry = unavailable.get(key);
if (!entry) return true;
// Check if cooldown has expired
if (Date.now() - entry.unavailableSince >= entry.cooldownMs) {
unavailable.delete(key);
return true;
}
return false;
}
/**
* Get remaining cooldown information for a model, if it is currently unavailable.
*
* @param {string} provider
* @param {string} model
* @returns {{ provider: string, model: string, reason: string, remainingMs: number, unavailableSince: string } | null}
*/
export function getModelCooldownInfo(provider, model) {
const key = makeKey(provider, model);
const entry = unavailable.get(key);
if (!entry) return null;
const elapsed = Date.now() - entry.unavailableSince;
if (elapsed >= entry.cooldownMs) {
unavailable.delete(key);
return null;
}
return {
provider: entry.provider,
model: entry.model,
reason: entry.reason || "unknown",
remainingMs: entry.cooldownMs - elapsed,
unavailableSince: new Date(entry.unavailableSince).toISOString(),
};
}
/**
* Mark a model as temporarily unavailable.
*
* @param {string} provider
* @param {string} model
* @param {number} [cooldownMs=60000] - Cooldown in milliseconds (default 60s)
* @param {string} [reason] - Optional reason for unavailability
*/
export function setModelUnavailable(provider, model, cooldownMs = 60000, reason) {
const key = makeKey(provider, model);
const now = Date.now();
const safeCooldownMs = Number.isFinite(cooldownMs) && cooldownMs > 0 ? cooldownMs : 60000;
const existing = unavailable.get(key);
const existingRemainingMs =
existing && Date.now() - existing.unavailableSince < existing.cooldownMs
? existing.cooldownMs - (Date.now() - existing.unavailableSince)
: 0;
const effectiveCooldownMs = Math.max(safeCooldownMs, existingRemainingMs);
unavailable.set(key, {
provider,
model,
unavailableSince: now,
cooldownMs: effectiveCooldownMs,
reason: reason || "unknown",
});
}
/**
* Marca provider/model como problemático com cooldown adaptativo.
* Mantém retrocompatibilidade: não altera o comportamento de setModelUnavailable,
* apenas oferece uma estratégia mais agressiva para falhas recorrentes.
*
* @param {string} provider
* @param {string} model
* @param {{ status?: number, baseCooldownMs?: number, reason?: string, profile?: ProviderProfile | null }} [options]
* @returns {{ cooldownMs: number, failureCount: number, quarantined: boolean, threshold: number, resetAfterMs: number }}
*/
export function markModelAsProblematic(provider, model, options = {}) {
const key = makeKey(provider, model);
const now = Date.now();
const status = Number.isFinite(options.status) ? Number(options.status) : null;
const profile = options.profile || null;
const explicitBaseCooldownMs =
Number.isFinite(options.baseCooldownMs) && Number(options.baseCooldownMs) > 0
? Number(options.baseCooldownMs)
: 0;
const statusBaseCooldown = profile
? getProfileStatusCooldown(status, profile)
: getLegacyStatusCooldown(status);
const baseCooldownMs = Math.max(explicitBaseCooldownMs, statusBaseCooldown);
const prev = failureState.get(key);
const resetAfterMs = getFailureWindowMs(profile);
const withinFailureWindow = prev && now - prev.lastFailureAt <= prev.resetAfterMs;
const failureCount = withinFailureWindow ? prev.failureCount + 1 : 1;
failureState.set(key, { failureCount, lastFailureAt: now, resetAfterMs });
const threshold = getFailureThreshold(profile);
const cooldownMs = getScaledCooldown(baseCooldownMs, failureCount, profile);
const quarantined = failureCount >= threshold;
if (quarantined) {
setModelUnavailable(provider, model, cooldownMs, options.reason || "problematic_model");
}
return {
cooldownMs,
failureCount,
quarantined,
threshold,
resetAfterMs,
};
}
/**
* Clear unavailability for a model (e.g. after manual reset).
*
* @param {string} provider
* @param {string} model
* @returns {boolean} true if entry existed and was removed
*/
export function clearModelUnavailability(provider, model) {
const key = makeKey(provider, model);
failureState.delete(key);
return unavailable.delete(key);
}
/**
* Get a report of all currently unavailable models.
*
* @returns {Array<{ provider: string, model: string, reason: string, remainingMs: number, unavailableSince: string }>}
*/
export function getAvailabilityReport() {
const now = Date.now();
const report = [];
for (const [key, entry] of unavailable.entries()) {
const elapsed = now - entry.unavailableSince;
if (elapsed >= entry.cooldownMs) {
unavailable.delete(key);
continue;
}
report.push({
provider: entry.provider,
model: entry.model,
reason: entry.reason || "unknown",
remainingMs: entry.cooldownMs - elapsed,
unavailableSince: new Date(entry.unavailableSince).toISOString(),
});
}
return report;
}
/**
* Get total count of unavailable models.
* @returns {number}
*/
export function getUnavailableCount() {
// Prune expired entries first
getAvailabilityReport();
return unavailable.size;
}
/**
* Reset all availability states (for testing or admin).
*/
export function resetAllAvailability() {
unavailable.clear();
failureState.clear();
}

54
src/lib/combos/intelligentRouting.ts
View File

@@ -31,13 +31,6 @@ export type IntelligentProviderScore = {
factors: IntelligentRoutingWeights;
};
export type IntelligentExclusionEntry = {
provider: string;
excludedAt: string;
cooldownMs: number;
reason: string;
};
export const DEFAULT_INTELLIGENT_WEIGHTS: IntelligentRoutingWeights = {
quota: 0.2,
health: 0.25,
@@ -161,50 +154,3 @@ export function buildIntelligentProviderScores(combo: {
factors: weights,
}));
}
export function extractIntelligentHealthState(health: unknown): {
incidentMode: boolean;
exclusions: IntelligentExclusionEntry[];
} {
const healthRecord = isRecord(health) ? health : {};
const providerHealth = isRecord(healthRecord.providerHealth) ? healthRecord.providerHealth : {};
const providerBreakers = Object.entries(providerHealth).map(([provider, status]) => {
const statusRecord = isRecord(status) ? status : {};
return {
provider,
state: typeof statusRecord.state === "string" ? statusRecord.state : "CLOSED",
lastFailure: typeof statusRecord.lastFailure === "string" ? statusRecord.lastFailure : null,
};
});
const breakersFromArray = Array.isArray(healthRecord.circuitBreakers)
? healthRecord.circuitBreakers
.map((entry) => {
const breaker = isRecord(entry) ? entry : {};
const provider =
typeof breaker.provider === "string"
? breaker.provider
: typeof breaker.name === "string"
? breaker.name
: "unknown";
return {
provider,
state: typeof breaker.state === "string" ? breaker.state : "CLOSED",
lastFailure: typeof breaker.lastFailure === "string" ? breaker.lastFailure : null,
};
})
.filter((entry) => typeof entry.provider === "string")
: [];
const breakers = breakersFromArray.length > 0 ? breakersFromArray : providerBreakers;
const openBreakers = breakers.filter((breaker) => breaker.state === "OPEN");
return {
incidentMode: openBreakers.length / Math.max(breakers.length, 1) > 0.5,
exclusions: openBreakers.map((breaker) => ({
provider: breaker.provider,
excludedAt: breaker.lastFailure || new Date().toISOString(),
cooldownMs: 5 * 60 * 1000,
reason: "Circuit breaker OPEN",
})),
};
}

70
src/lib/dataPaths.js
View File

@@ -1,70 +0,0 @@
"use strict";
var __importDefault = (this && this.__importDefault) || function (mod) {
return (mod && mod.__esModule) ? mod : { "default": mod };
};
Object.defineProperty(exports, "__esModule", { value: true });
exports.APP_NAME = void 0;
exports.getLegacyDotDataDir = getLegacyDotDataDir;
exports.getDefaultDataDir = getDefaultDataDir;
exports.resolveDataDir = resolveDataDir;
exports.isSamePath = isSamePath;
const path_1 = __importDefault(require("path"));
const os_1 = __importDefault(require("os"));
exports.APP_NAME = "omniroute";
function fallbackHomeDir() {
const envHome = process.env.HOME || process.env.USERPROFILE;
if (typeof envHome === "string" && envHome.trim().length > 0) {
return path_1.default.resolve(envHome);
}
return os_1.default.tmpdir();
}
function safeHomeDir() {
try {
return os_1.default.homedir();
}
catch {
return fallbackHomeDir();
}
}
function normalizeConfiguredPath(dir) {
if (typeof dir !== "string")
return null;
const trimmed = dir.trim();
if (!trimmed)
return null;
return path_1.default.resolve(trimmed);
}
function getLegacyDotDataDir() {
return path_1.default.join(safeHomeDir(), `.${exports.APP_NAME}`);
}
function getDefaultDataDir() {
const homeDir = safeHomeDir();
if (process.platform === "win32") {
const appData = process.env.APPDATA || path_1.default.join(homeDir, "AppData", "Roaming");
return path_1.default.join(appData, exports.APP_NAME);
}
// Support XDG on Linux/macOS when explicitly configured.
const xdgConfigHome = normalizeConfiguredPath(process.env.XDG_CONFIG_HOME);
if (xdgConfigHome) {
return path_1.default.join(xdgConfigHome, exports.APP_NAME);
}
return getLegacyDotDataDir();
}
function resolveDataDir({ isCloud = false } = {}) {
if (isCloud)
return "/tmp";
const configured = normalizeConfiguredPath(process.env.DATA_DIR);
if (configured)
return configured;
return getDefaultDataDir();
}
function isSamePath(a, b) {
if (!a || !b)
return false;
const normalizedA = path_1.default.resolve(a);
const normalizedB = path_1.default.resolve(b);
if (process.platform === "win32") {
return normalizedA.toLowerCase() === normalizedB.toLowerCase();
}
return normalizedA === normalizedB;
}

34
src/lib/monitoring/observability.ts
View File

@@ -4,7 +4,8 @@ interface CircuitBreakerStatus {
name: string;
state: string;
failureCount?: number;
lastFailureTime?: string | null;
lastFailureTime?: number | string | null;
retryAfterMs?: number;
}
interface SessionSnapshot {
@@ -155,13 +156,31 @@ export function buildHealthPayload({
platform: process.platform,
};
const providerBreakers = circuitBreakers
.filter((cb) => !cb.name.startsWith("test-") && !cb.name.startsWith("test_"))
.map((cb) => {
const lastFailure =
typeof cb.lastFailureTime === "number" && Number.isFinite(cb.lastFailureTime)
? new Date(cb.lastFailureTime).toISOString()
: typeof cb.lastFailureTime === "string"
? cb.lastFailureTime
: null;
return {
provider: cb.name,
state: cb.state,
failureCount: cb.failureCount || 0,
lastFailure,
retryAfterMs: cb.retryAfterMs || 0,
};
});
const providerHealth: Record<string, JsonRecord> = {};
for (const cb of circuitBreakers) {
if (cb.name.startsWith("test-") || cb.name.startsWith("test_")) continue;
providerHealth[cb.name] = {
state: cb.state,
failures: cb.failureCount || 0,
lastFailure: cb.lastFailureTime || null,
for (const breaker of providerBreakers) {
providerHealth[breaker.provider] = {
state: breaker.state,
failures: breaker.failureCount,
lastFailure: breaker.lastFailure,
retryAfterMs: breaker.retryAfterMs,
};
}
@@ -197,6 +216,7 @@ export function buildHealthPayload({
...breakerCounts,
total: breakerCounts.open + breakerCounts.halfOpen + breakerCounts.closed,
},
providerBreakers,
providerHealth,
providerSummary: {
catalogCount,

419
src/lib/resilience/settings.ts Normal file
View File

@@ -0,0 +1,419 @@
import { DEFAULT_API_LIMITS, PROVIDER_PROFILES } from "@omniroute/open-sse/config/constants";
type JsonRecord = Record<string, unknown>;
type AuthCategory = "oauth" | "apikey";
export interface RequestQueueSettings {
autoEnableApiKeyProviders: boolean;
requestsPerMinute: number;
minTimeBetweenRequestsMs: number;
concurrentRequests: number;
maxWaitMs: number;
}
export interface ConnectionCooldownProfileSettings {
baseCooldownMs: number;
useUpstreamRetryHints: boolean;
maxBackoffSteps: number;
}
export interface ProviderBreakerProfileSettings {
failureThreshold: number;
resetTimeoutMs: number;
}
export interface WaitForCooldownSettings {
enabled: boolean;
maxRetries: number;
maxRetryWaitSec: number;
maxRetryWaitMs: number;
}
export interface ResilienceSettings {
requestQueue: RequestQueueSettings;
connectionCooldown: Record<AuthCategory, ConnectionCooldownProfileSettings>;
providerBreaker: Record<AuthCategory, ProviderBreakerProfileSettings>;
waitForCooldown: WaitForCooldownSettings;
}
export interface ResilienceSettingsPatch {
requestQueue?: Partial<RequestQueueSettings>;
connectionCooldown?: Partial<Record<AuthCategory, Partial<ConnectionCooldownProfileSettings>>>;
providerBreaker?: Partial<Record<AuthCategory, Partial<ProviderBreakerProfileSettings>>>;
waitForCooldown?: Partial<WaitForCooldownSettings>;
}
function asRecord(value: unknown): JsonRecord {
return value && typeof value === "object" && !Array.isArray(value) ? (value as JsonRecord) : {};
}
function toInteger(
value: unknown,
fallback: number,
options: { min?: number; max?: number } = {}
): number {
const min = options.min ?? 0;
const max = options.max ?? Number.MAX_SAFE_INTEGER;
const parsed =
typeof value === "number"
? value
: typeof value === "string" && value.trim().length > 0
? Number(value)
: Number.NaN;
if (!Number.isFinite(parsed)) {
return fallback;
}
return Math.min(max, Math.max(min, Math.trunc(parsed)));
}
function toBoolean(value: unknown, fallback: boolean): boolean {
return typeof value === "boolean" ? value : fallback;
}
export const DEFAULT_REQUEST_QUEUE_MAX_WAIT_MS = (() => {
const parsed = Number(process.env.RATE_LIMIT_MAX_WAIT_MS || "120000");
return Number.isFinite(parsed) && parsed > 0 ? Math.trunc(parsed) : 120000;
})();
export const DEFAULT_RESILIENCE_SETTINGS: ResilienceSettings = {
requestQueue: {
autoEnableApiKeyProviders: true,
requestsPerMinute: DEFAULT_API_LIMITS.requestsPerMinute,
minTimeBetweenRequestsMs: DEFAULT_API_LIMITS.minTimeBetweenRequests,
concurrentRequests: DEFAULT_API_LIMITS.concurrentRequests,
maxWaitMs: DEFAULT_REQUEST_QUEUE_MAX_WAIT_MS,
},
connectionCooldown: {
oauth: {
baseCooldownMs: PROVIDER_PROFILES.oauth.transientCooldown,
useUpstreamRetryHints: PROVIDER_PROFILES.oauth.rateLimitCooldown === 0,
maxBackoffSteps: PROVIDER_PROFILES.oauth.maxBackoffLevel,
},
apikey: {
baseCooldownMs: PROVIDER_PROFILES.apikey.transientCooldown,
useUpstreamRetryHints: PROVIDER_PROFILES.apikey.rateLimitCooldown === 0,
maxBackoffSteps: PROVIDER_PROFILES.apikey.maxBackoffLevel,
},
},
providerBreaker: {
oauth: {
failureThreshold: PROVIDER_PROFILES.oauth.circuitBreakerThreshold,
resetTimeoutMs: PROVIDER_PROFILES.oauth.circuitBreakerReset,
},
apikey: {
failureThreshold: PROVIDER_PROFILES.apikey.circuitBreakerThreshold,
resetTimeoutMs: PROVIDER_PROFILES.apikey.circuitBreakerReset,
},
},
waitForCooldown: {
enabled: true,
maxRetries: 3,
maxRetryWaitSec: 30,
maxRetryWaitMs: 30000,
},
};
function normalizeRequestQueueSettings(
next: unknown,
fallback: RequestQueueSettings
): RequestQueueSettings {
const record = asRecord(next);
const requestsPerMinute = toInteger(record.requestsPerMinute, fallback.requestsPerMinute, {
min: 1,
max: 1_000_000,
});
const minTimeBetweenRequestsMs = toInteger(
record.minTimeBetweenRequestsMs,
fallback.minTimeBetweenRequestsMs,
{ min: 0, max: 60 * 60 * 1000 }
);
const concurrentRequests = toInteger(record.concurrentRequests, fallback.concurrentRequests, {
min: 1,
max: 10_000,
});
const maxWaitMs = toInteger(record.maxWaitMs, fallback.maxWaitMs, {
min: 1,
max: 24 * 60 * 60 * 1000,
});
return {
autoEnableApiKeyProviders: toBoolean(
record.autoEnableApiKeyProviders,
fallback.autoEnableApiKeyProviders
),
requestsPerMinute,
minTimeBetweenRequestsMs,
concurrentRequests,
maxWaitMs,
};
}
function normalizeConnectionCooldownProfile(
next: unknown,
fallback: ConnectionCooldownProfileSettings
): ConnectionCooldownProfileSettings {
const record = asRecord(next);
return {
baseCooldownMs: toInteger(record.baseCooldownMs, fallback.baseCooldownMs, {
min: 0,
max: 24 * 60 * 60 * 1000,
}),
useUpstreamRetryHints: toBoolean(record.useUpstreamRetryHints, fallback.useUpstreamRetryHints),
maxBackoffSteps: toInteger(record.maxBackoffSteps, fallback.maxBackoffSteps, {
min: 0,
max: 32,
}),
};
}
function normalizeLegacyConnectionCooldownProfile(
next: unknown,
fallback: ConnectionCooldownProfileSettings
): ConnectionCooldownProfileSettings {
const record = asRecord(next);
const transientCooldown = toInteger(record.transientCooldown, fallback.baseCooldownMs, {
min: 0,
max: 24 * 60 * 60 * 1000,
});
const legacyRateLimitCooldown = toInteger(record.rateLimitCooldown, transientCooldown, {
min: 0,
max: 24 * 60 * 60 * 1000,
});
const useUpstreamRetryHints =
typeof record.rateLimitCooldown === "number"
? record.rateLimitCooldown === 0
: fallback.useUpstreamRetryHints;
return {
baseCooldownMs: useUpstreamRetryHints
? transientCooldown
: Math.max(transientCooldown, legacyRateLimitCooldown),
useUpstreamRetryHints,
maxBackoffSteps: toInteger(record.maxBackoffLevel, fallback.maxBackoffSteps, {
min: 0,
max: 32,
}),
};
}
function normalizeProviderBreakerProfile(
next: unknown,
fallback: ProviderBreakerProfileSettings
): ProviderBreakerProfileSettings {
const record = asRecord(next);
return {
failureThreshold: toInteger(record.failureThreshold, fallback.failureThreshold, {
min: 1,
max: 1000,
}),
resetTimeoutMs: toInteger(record.resetTimeoutMs, fallback.resetTimeoutMs, {
min: 1000,
max: 24 * 60 * 60 * 1000,
}),
};
}
function normalizeWaitForCooldownSettings(
next: unknown,
fallback: WaitForCooldownSettings
): WaitForCooldownSettings {
const record = asRecord(next);
const maxRetryWaitSec = toInteger(record.maxRetryWaitSec, fallback.maxRetryWaitSec, {
min: 0,
max: 300,
});
const maxRetries = toInteger(record.maxRetries, fallback.maxRetries, { min: 0, max: 10 });
const enabled =
toBoolean(record.enabled, fallback.enabled) && maxRetries > 0 && maxRetryWaitSec > 0;
return {
enabled,
maxRetries,
maxRetryWaitSec,
maxRetryWaitMs: maxRetryWaitSec * 1000,
};
}
function buildLegacyFallback(settings: JsonRecord): ResilienceSettings {
const profiles = asRecord(settings.providerProfiles);
const defaults = asRecord(settings.rateLimitDefaults);
const oauthLegacy = asRecord(profiles.oauth);
const apikeyLegacy = asRecord(profiles.apikey);
const waitMaxRetrySec = toInteger(
settings.maxRetryIntervalSec,
DEFAULT_RESILIENCE_SETTINGS.waitForCooldown.maxRetryWaitSec,
{ min: 0, max: 300 }
);
const waitMaxRetries = toInteger(
settings.requestRetry,
DEFAULT_RESILIENCE_SETTINGS.waitForCooldown.maxRetries,
{ min: 0, max: 10 }
);
return {
requestQueue: {
autoEnableApiKeyProviders: DEFAULT_RESILIENCE_SETTINGS.requestQueue.autoEnableApiKeyProviders,
requestsPerMinute: toInteger(
defaults.requestsPerMinute,
DEFAULT_RESILIENCE_SETTINGS.requestQueue.requestsPerMinute,
{ min: 1, max: 1_000_000 }
),
minTimeBetweenRequestsMs: toInteger(
defaults.minTimeBetweenRequests,
DEFAULT_RESILIENCE_SETTINGS.requestQueue.minTimeBetweenRequestsMs,
{ min: 0, max: 60 * 60 * 1000 }
),
concurrentRequests: toInteger(
defaults.concurrentRequests,
DEFAULT_RESILIENCE_SETTINGS.requestQueue.concurrentRequests,
{ min: 1, max: 10_000 }
),
maxWaitMs: DEFAULT_RESILIENCE_SETTINGS.requestQueue.maxWaitMs,
},
connectionCooldown: {
oauth: normalizeLegacyConnectionCooldownProfile(
oauthLegacy,
DEFAULT_RESILIENCE_SETTINGS.connectionCooldown.oauth
),
apikey: normalizeLegacyConnectionCooldownProfile(
apikeyLegacy,
DEFAULT_RESILIENCE_SETTINGS.connectionCooldown.apikey
),
},
providerBreaker: {
oauth: {
failureThreshold: toInteger(
oauthLegacy.circuitBreakerThreshold,
DEFAULT_RESILIENCE_SETTINGS.providerBreaker.oauth.failureThreshold,
{ min: 1, max: 1000 }
),
resetTimeoutMs: toInteger(
oauthLegacy.circuitBreakerReset,
DEFAULT_RESILIENCE_SETTINGS.providerBreaker.oauth.resetTimeoutMs,
{ min: 1000, max: 24 * 60 * 60 * 1000 }
),
},
apikey: {
failureThreshold: toInteger(
apikeyLegacy.circuitBreakerThreshold,
DEFAULT_RESILIENCE_SETTINGS.providerBreaker.apikey.failureThreshold,
{ min: 1, max: 1000 }
),
resetTimeoutMs: toInteger(
apikeyLegacy.circuitBreakerReset,
DEFAULT_RESILIENCE_SETTINGS.providerBreaker.apikey.resetTimeoutMs,
{ min: 1000, max: 24 * 60 * 60 * 1000 }
),
},
},
waitForCooldown: {
enabled: waitMaxRetries > 0 && waitMaxRetrySec > 0,
maxRetries: waitMaxRetries,
maxRetryWaitSec: waitMaxRetrySec,
maxRetryWaitMs: waitMaxRetrySec * 1000,
},
};
}
export function resolveResilienceSettings(
settings: Record<string, unknown> | null | undefined
): ResilienceSettings {
const record = asRecord(settings);
const current = asRecord(record.resilienceSettings);
const fallback = buildLegacyFallback(record);
return {
requestQueue: normalizeRequestQueueSettings(current.requestQueue, fallback.requestQueue),
connectionCooldown: {
oauth: normalizeConnectionCooldownProfile(
asRecord(current.connectionCooldown).oauth,
fallback.connectionCooldown.oauth
),
apikey: normalizeConnectionCooldownProfile(
asRecord(current.connectionCooldown).apikey,
fallback.connectionCooldown.apikey
),
},
providerBreaker: {
oauth: normalizeProviderBreakerProfile(
asRecord(current.providerBreaker).oauth,
fallback.providerBreaker.oauth
),
apikey: normalizeProviderBreakerProfile(
asRecord(current.providerBreaker).apikey,
fallback.providerBreaker.apikey
),
},
waitForCooldown: normalizeWaitForCooldownSettings(
current.waitForCooldown,
fallback.waitForCooldown
),
};
}
export function mergeResilienceSettings(
current: ResilienceSettings,
updates: ResilienceSettingsPatch
): ResilienceSettings {
return {
requestQueue: normalizeRequestQueueSettings(updates.requestQueue, current.requestQueue),
connectionCooldown: {
oauth: normalizeConnectionCooldownProfile(
updates.connectionCooldown?.oauth,
current.connectionCooldown.oauth
),
apikey: normalizeConnectionCooldownProfile(
updates.connectionCooldown?.apikey,
current.connectionCooldown.apikey
),
},
providerBreaker: {
oauth: normalizeProviderBreakerProfile(
updates.providerBreaker?.oauth,
current.providerBreaker.oauth
),
apikey: normalizeProviderBreakerProfile(
updates.providerBreaker?.apikey,
current.providerBreaker.apikey
),
},
waitForCooldown: normalizeWaitForCooldownSettings(
updates.waitForCooldown,
current.waitForCooldown
),
};
}
export function buildLegacyResilienceCompat(settings: ResilienceSettings) {
return {
profiles: {
oauth: {
transientCooldown: settings.connectionCooldown.oauth.baseCooldownMs,
rateLimitCooldown: settings.connectionCooldown.oauth.useUpstreamRetryHints
? 0
: settings.connectionCooldown.oauth.baseCooldownMs,
maxBackoffLevel: settings.connectionCooldown.oauth.maxBackoffSteps,
circuitBreakerThreshold: settings.providerBreaker.oauth.failureThreshold,
circuitBreakerReset: settings.providerBreaker.oauth.resetTimeoutMs,
},
apikey: {
transientCooldown: settings.connectionCooldown.apikey.baseCooldownMs,
rateLimitCooldown: settings.connectionCooldown.apikey.useUpstreamRetryHints
? 0
: settings.connectionCooldown.apikey.baseCooldownMs,
maxBackoffLevel: settings.connectionCooldown.apikey.maxBackoffSteps,
circuitBreakerThreshold: settings.providerBreaker.apikey.failureThreshold,
circuitBreakerReset: settings.providerBreaker.apikey.resetTimeoutMs,
},
},
defaults: {
requestsPerMinute: settings.requestQueue.requestsPerMinute,
minTimeBetweenRequests: settings.requestQueue.minTimeBetweenRequestsMs,
concurrentRequests: settings.requestQueue.concurrentRequests,
},
};
}

1
src/shared/utils/circuitBreaker.ts
View File

@@ -180,6 +180,7 @@ export class CircuitBreaker {
state: this.state,
failureCount: this.failureCount,
lastFailureTime: this.lastFailureTime,
retryAfterMs: this.getRetryAfterMs(),
};
}

80
src/shared/validation/schemas.ts
View File

@@ -616,11 +616,6 @@ export const updateModelAliasSchema = z.object({
alias: z.string().trim().min(1, "Alias is required").max(200),
});
export const clearModelAvailabilitySchema = z.object({
provider: z.string().trim().min(1, "provider is required").max(120),
model: modelIdSchema,
});
/** Align with `sanitizeUpstreamHeadersMap` — allow non-ASCII names; reject Host / hop-by-hop / whitespace / ":". */
const upstreamHeaderNameSchema = z
.string()
@@ -709,7 +704,7 @@ export const toggleRateLimitSchema = z.object({
enabled: z.boolean(),
});
const resilienceProfileSchema = z.object({
const legacyResilienceProfileSchema = z.object({
transientCooldown: z.number().min(0),
rateLimitCooldown: z.number().min(0),
maxBackoffLevel: z.number().int().min(0),
@@ -717,30 +712,87 @@ const resilienceProfileSchema = z.object({
circuitBreakerReset: z.number().min(0),
});
const resilienceDefaultsSchema = z
const legacyResilienceDefaultsSchema = z
.object({
requestsPerMinute: z.number().int().min(1).optional(),
minTimeBetweenRequests: z.number().int().min(1).optional(),
minTimeBetweenRequests: z.number().int().min(0).optional(),
concurrentRequests: z.number().int().min(1).optional(),
})
.strict();
const requestQueueSettingsSchema = z
.object({
autoEnableApiKeyProviders: z.boolean().optional(),
requestsPerMinute: z.number().int().min(1).optional(),
minTimeBetweenRequestsMs: z.number().int().min(0).optional(),
concurrentRequests: z.number().int().min(1).optional(),
maxWaitMs: z.number().int().min(1).optional(),
})
.strict();
const connectionCooldownProfileSchema = z
.object({
baseCooldownMs: z.number().int().min(0).optional(),
useUpstreamRetryHints: z.boolean().optional(),
maxBackoffSteps: z.number().int().min(0).optional(),
})
.strict();
const providerBreakerProfileSchema = z
.object({
failureThreshold: z.number().int().min(1).optional(),
resetTimeoutMs: z.number().int().min(1000).optional(),
})
.strict();
const waitForCooldownSettingsSchema = z
.object({
enabled: z.boolean().optional(),
maxRetries: z.number().int().min(0).max(10).optional(),
maxRetryWaitSec: z.number().int().min(0).max(300).optional(),
})
.strict();
export const updateResilienceSchema = z
.object({
profiles: z
requestQueue: requestQueueSettingsSchema.optional(),
connectionCooldown: z
.object({
oauth: resilienceProfileSchema.optional(),
apikey: resilienceProfileSchema.optional(),
oauth: connectionCooldownProfileSchema.optional(),
apikey: connectionCooldownProfileSchema.optional(),
})
.strict()
.optional(),
defaults: resilienceDefaultsSchema.optional(),
providerBreaker: z
.object({
oauth: providerBreakerProfileSchema.optional(),
apikey: providerBreakerProfileSchema.optional(),
})
.strict()
.optional(),
waitForCooldown: waitForCooldownSettingsSchema.optional(),
profiles: z
.object({
oauth: legacyResilienceProfileSchema.optional(),
apikey: legacyResilienceProfileSchema.optional(),
})
.strict()
.optional(),
defaults: legacyResilienceDefaultsSchema.optional(),
})
.strict()
.superRefine((value, ctx) => {
if (!value.profiles && !value.defaults) {
if (
!value.requestQueue &&
!value.connectionCooldown &&
!value.providerBreaker &&
!value.waitForCooldown &&
!value.profiles &&
!value.defaults
) {
ctx.addIssue({
code: z.ZodIssueCode.custom,
message: "Must provide profiles or defaults",
message: "Must provide resilience settings to update",
path: [],
});
}

73
src/sse/handlers/chat.ts
View File

@@ -42,11 +42,6 @@ import {
// Pipeline integration — wired modules
import { getCircuitBreaker } from "../../shared/utils/circuitBreaker";
import {
isModelAvailable,
markModelAsProblematic,
clearModelUnavailability,
} from "../../domain/modelAvailability";
import { markAccountExhaustedFrom429 } from "../../domain/quotaCache";
import { RequestTelemetry, recordTelemetry } from "../../shared/utils/requestTelemetry";
import { generateRequestId } from "../../shared/utils/requestId";
@@ -107,6 +102,8 @@ function intersectAllowedConnectionIds(primary: unknown, secondary: unknown): st
return first || second || null;
}
const PROVIDER_BREAKER_FAILURE_STATUSES = new Set([408, 500, 502, 503, 504]);
/**
* Handle chat completion request
* Supports: OpenAI, Claude, Gemini, OpenAI Responses API formats
@@ -324,14 +321,6 @@ export async function handleChat(request: any, clientRawRequest: any = null) {
return false;
}
// Fixed-account combo steps must bypass the provider/model cooldown gate here.
// A previous account failure can quarantine the model globally, but the next
// step may intentionally pin a different connection for the same model.
if (!hasForcedConnection && !isModelAvailable(provider, resolvedModel)) {
log.debug("AVAILABILITY", `${provider}/${modelInfo.model} in cooldown, skipping`);
return false;
}
const creds = await getProviderCredentialsWithQuotaPreflight(
provider,
null,
@@ -524,7 +513,7 @@ async function handleSingleModelChat(
? "fixed combo step connection"
: undefined;
// 2. Pipeline gates (availability + circuit breaker)
// 2. Pipeline gates (availability + provider circuit breaker)
const providerProfile = await getRuntimeProviderProfile(provider);
const gate = await checkPipelineGates(provider, model, {
ignoreCircuitBreaker: forceLiveComboTest || hasForcedConnection,
@@ -535,8 +524,8 @@ async function handleSingleModelChat(
if (gate) return gate;
const breaker = getCircuitBreaker(provider, {
failureThreshold: providerProfile.circuitBreakerThreshold,
resetTimeout: providerProfile.circuitBreakerReset,
failureThreshold: providerProfile.failureThreshold,
resetTimeout: providerProfile.resetTimeoutMs,
onStateChange: (name: string, from: string, to: string) =>
log.info("CIRCUIT", `${name}: ${from} → ${to}`),
});
@@ -549,9 +538,10 @@ async function handleSingleModelChat(
const retrySettings = disableCooldownAwareRetry
? {
...baseRetrySettings,
requestRetry: 0,
maxRetryIntervalSec: 0,
maxRetryIntervalMs: 0,
enabled: false,
maxRetries: 0,
maxRetryWaitSec: 0,
maxRetryWaitMs: 0,
}
: baseRetrySettings;
const requestSignal = request?.signal ?? null;
@@ -597,26 +587,6 @@ async function handleSingleModelChat(
);
if (!credentials || "allRateLimited" in credentials) {
if ([408, 429, 500, 502, 503, 504].includes(Number(lastStatus))) {
const quarantine = markModelAsProblematic(provider, model, {
status: Number(lastStatus),
baseCooldownMs: lastCooldownMs,
reason: `HTTP ${lastStatus}`,
profile: providerProfile,
});
if (quarantine.quarantined) {
log.info(
"AVAILABILITY",
`${provider}/${model} marked unavailable — all accounts exhausted (HTTP ${lastStatus}, cooldown ${Math.ceil(quarantine.cooldownMs / 1000)}s, failureCount ${quarantine.failureCount}/${quarantine.threshold})`
);
} else {
log.info(
"AVAILABILITY",
`${provider}/${model} recorded exhaustion failure ${quarantine.failureCount}/${quarantine.threshold} (HTTP ${lastStatus}, cooldown basis ${Math.ceil(quarantine.cooldownMs / 1000)}s)`
);
}
}
if (credentials?.allRateLimited) {
const retryDecision = getCooldownAwareRetryDecision({
retryAfter: credentials.retryAfter,
@@ -628,7 +598,7 @@ async function handleSingleModelChat(
const waitSec = Math.max(Math.ceil(retryDecision.waitMs / 1000), 0);
log.info(
"COOLDOWN_RETRY",
`${provider}/${model} all accounts cooling down (${retryDecision.retryAfterHuman || `retry in ${waitSec}s`}) — waiting ${waitSec}s before retry ${requestRetryAttempt + 1}/${retrySettings.requestRetry}`
`${provider}/${model} all connections cooling down (${retryDecision.retryAfterHuman || `retry in ${waitSec}s`}) — waiting ${waitSec}s before retry ${requestRetryAttempt + 1}/${retrySettings.maxRetries}`
);
const completed = await waitForCooldownAwareRetry(retryDecision.waitMs, requestSignal);
@@ -643,12 +613,21 @@ async function handleSingleModelChat(
requestRetryAttempt += 1;
log.info(
"COOLDOWN_RETRY",
`${provider}/${model} cooldown elapsed — restarting request attempt ${requestRetryAttempt}/${retrySettings.requestRetry}`
`${provider}/${model} cooldown elapsed — restarting request attempt ${requestRetryAttempt}/${retrySettings.maxRetries}`
);
continue requestAttemptLoop;
}
}
const breakerFailureStatus = Number(lastStatus ?? credentials?.lastErrorCode);
if (
!forceLiveComboTest &&
credentials?.allRateLimited &&
PROVIDER_BREAKER_FAILURE_STATUSES.has(breakerFailureStatus)
) {
breaker._onFailure();
}
return handleNoCredentials(
credentials,
excludedConnectionIds.size > 0 ? Array.from(excludedConnectionIds)[0] : null,
@@ -717,11 +696,9 @@ async function handleSingleModelChat(
const proxyInfo = await safeResolveProxy(credentials.connectionId);
const proxyStartTime = Date.now();
// 4. Execute chat via core (with circuit breaker + optional TLS)
// 4. Execute chat via core after breaker gate checks (with optional TLS tracking)
if (telemetry) telemetry.startPhase("connect");
const { result, tlsFingerprintUsed } = await executeChatWithBreaker({
bypassCircuitBreaker: forceLiveComboTest,
breaker,
body: requestBody,
provider,
model,
@@ -765,7 +742,9 @@ async function handleSingleModelChat(
if (result.success) {
clearModelLock(provider, credentials.connectionId, model);
clearModelUnavailability(provider, model);
if (!forceLiveComboTest) {
breaker._onSuccess();
}
if (injectedHandoff && runtimeOptions.sessionId && comboName) {
deleteHandoff(runtimeOptions.sessionId, comboName);
}
@@ -873,6 +852,10 @@ async function handleSingleModelChat(
continue;
}
if (!forceLiveComboTest && PROVIDER_BREAKER_FAILURE_STATUSES.has(Number(result.status))) {
breaker._onFailure();
}
return result.response;
}
}

71
src/sse/handlers/chatHelpers.ts
View File

@@ -14,6 +14,7 @@ import { handleChatCore } from "@omniroute/open-sse/handlers/chatCore.ts";
import {
errorResponse,
modelCooldownResponse,
providerCircuitOpenResponse,
unavailableResponse,
} from "@omniroute/open-sse/utils/error.ts";
import { HTTP_STATUS } from "@omniroute/open-sse/config/constants.ts";
@@ -23,14 +24,10 @@ import {
isTlsFingerprintActive,
} from "@omniroute/open-sse/utils/proxyFetch.ts";
import { resolveProxyForConnection } from "@/lib/localDb";
import { getCircuitBreaker, CircuitBreakerOpenError } from "../../shared/utils/circuitBreaker";
import { getModelCooldownInfo, isModelAvailable } from "../../domain/modelAvailability";
import { getCircuitBreaker } from "../../shared/utils/circuitBreaker";
import { logProxyEvent } from "../../lib/proxyLogger";
import { logTranslationEvent } from "../../lib/translatorEvents";
import {
getRuntimeProviderProfile,
clearProviderFailure,
} from "@omniroute/open-sse/services/accountFallback.ts";
import { getRuntimeProviderProfile } from "@omniroute/open-sse/services/accountFallback.ts";
export async function resolveModelOrError(modelStr: string, body: any, endpointPath: string = "") {
const modelInfo = await getModelInfo(modelStr);
@@ -79,30 +76,16 @@ export async function checkPipelineGates(
providerProfile?: {
circuitBreakerThreshold?: number;
circuitBreakerReset?: number;
failureThreshold?: number;
resetTimeoutMs?: number;
} | null;
} = {}
) {
const bypassReason = options.bypassReason || "pipeline override";
const modelAvailable = isModelAvailable(provider, model);
if (!modelAvailable && options.ignoreModelCooldown) {
log.info("AVAILABILITY", `${provider}/${model} cooldown bypassed (${bypassReason})`);
} else if (!modelAvailable) {
const cooldownInfo = getModelCooldownInfo(provider, model);
const retryAfterSec = cooldownInfo
? Math.max(Math.ceil(cooldownInfo.remainingMs / 1000), 1)
: 1;
log.warn("AVAILABILITY", `${provider}/${model} is in cooldown, rejecting request`);
return unavailableResponse(
HTTP_STATUS.SERVICE_UNAVAILABLE,
`Model ${provider}/${model} is temporarily unavailable (cooldown)`,
retryAfterSec
);
}
const providerProfile = options.providerProfile ?? (await getRuntimeProviderProfile(provider));
const breaker = getCircuitBreaker(provider, {
failureThreshold: providerProfile.circuitBreakerThreshold,
resetTimeout: providerProfile.circuitBreakerReset,
failureThreshold: providerProfile.failureThreshold ?? providerProfile.circuitBreakerThreshold,
resetTimeout: providerProfile.resetTimeoutMs ?? providerProfile.circuitBreakerReset,
onStateChange: (name: string, from: string, to: string) =>
log.info("CIRCUIT", `${name}: ${from} → ${to}`),
});
@@ -112,19 +95,13 @@ export async function checkPipelineGates(
const retryAfterMs = breaker.getRetryAfterMs();
const retryAfterSec = Math.max(Math.ceil(retryAfterMs / 1000), 1);
log.warn("CIRCUIT", `Circuit breaker OPEN for ${provider}, rejecting request`);
return unavailableResponse(
HTTP_STATUS.SERVICE_UNAVAILABLE,
`Provider ${provider} circuit breaker is open`,
retryAfterSec
);
return providerCircuitOpenResponse(provider, retryAfterSec);
}
return null;
}
export async function executeChatWithBreaker({
bypassCircuitBreaker,
breaker,
body,
provider,
model,
@@ -173,46 +150,18 @@ export async function executeChatWithBreaker({
},
onRequestSuccess: async () => {
await clearAccountError(credentials.connectionId, credentials);
// Clear provider-level failure state on successful request
clearProviderFailure(provider);
},
})
);
if (bypassCircuitBreaker) {
if (!proxyInfo?.proxy && isTlsFingerprintActive()) {
const tracked = await runWithTlsTracking(chatFn);
return { result: tracked.result, tlsFingerprintUsed: tracked.tlsFingerprintUsed };
}
const result = await chatFn();
return { result, tlsFingerprintUsed: false };
}
if (!proxyInfo?.proxy && isTlsFingerprintActive()) {
const tracked = await breaker.execute(async () => runWithTlsTracking(chatFn));
const tracked = await runWithTlsTracking(chatFn);
return { result: tracked.result, tlsFingerprintUsed: tracked.tlsFingerprintUsed };
}
const result = await breaker.execute(chatFn);
const result = await chatFn();
return { result, tlsFingerprintUsed: false };
} catch (cbErr: any) {
if (cbErr instanceof CircuitBreakerOpenError) {
log.warn("CIRCUIT", `${provider} circuit open during retry: ${cbErr.message}`);
return {
result: {
success: false,
response: unavailableResponse(
HTTP_STATUS.SERVICE_UNAVAILABLE,
`Provider ${provider} circuit breaker is open`,
Math.ceil(cbErr.retryAfterMs / 1000)
),
status: HTTP_STATUS.SERVICE_UNAVAILABLE,
},
tlsFingerprintUsed: false,
};
}
if (cbErr?.code === "PROXY_UNREACHABLE" || /proxy unreachable/i.test(cbErr?.message || "")) {
const detail = cbErr?.message || "Proxy unreachable";
log.warn("PROXY", detail);

115
src/sse/services/auth.ts
View File

@@ -19,12 +19,14 @@ import {
hasPerModelQuota,
getRuntimeProviderProfile,
recordModelLockoutFailure,
isProviderInCooldown,
getProviderCooldownRemainingMs,
} from "@omniroute/open-sse/services/accountFallback.ts";
import { isLocalProvider } from "@omniroute/open-sse/config/providerRegistry.ts";
import { COOLDOWN_MS } from "@omniroute/open-sse/config/constants.ts";
import { preflightQuota } from "@omniroute/open-sse/services/quotaPreflight.ts";
import {
classifyProviderError,
PROVIDER_ERROR_TYPES,
} from "@omniroute/open-sse/services/errorClassifier.ts";
import { getCodexModelScope } from "@omniroute/open-sse/executors/codex.ts";
import { getProviderAlias, resolveProviderId } from "@/shared/constants/providers";
import { isModelExcludedByConnection } from "@/domain/connectionModelRules";
@@ -253,6 +255,31 @@ function isTerminalConnectionStatus(connection: ProviderConnectionView): boolean
return status === "credits_exhausted" || status === "banned" || status === "expired";
}
function resolveTerminalConnectionStatus(
status: number,
result: { permanent?: boolean; creditsExhausted?: boolean },
providerErrorType: string | null = null
): string | null {
if (result.creditsExhausted || status === 402) return "credits_exhausted";
if (
providerErrorType === PROVIDER_ERROR_TYPES.PROJECT_ROUTE_ERROR ||
providerErrorType === PROVIDER_ERROR_TYPES.OAUTH_INVALID_TOKEN
) {
return null;
}
if (result.permanent || providerErrorType === PROVIDER_ERROR_TYPES.FORBIDDEN || status === 403) {
return "banned";
}
if (
providerErrorType === PROVIDER_ERROR_TYPES.ACCOUNT_DEACTIVATED ||
providerErrorType === PROVIDER_ERROR_TYPES.UNAUTHORIZED ||
status === 401
) {
return "expired";
}
return null;
}
export function resolveQuotaLimitPolicy(
provider: string,
providerSpecificData: JsonRecord
@@ -684,22 +711,6 @@ export async function getProviderCredentials(
return true;
});
// Check if the entire provider is in cooldown (too many transient failures)
if (isProviderInCooldown(provider)) {
const cooldownRemaining = getProviderCooldownRemainingMs(provider);
log.warn(
"AUTH",
`${provider} | provider in cooldown for ${Math.ceil((cooldownRemaining || 0) / 1000)}s (${availableConnections.length} connections bypassed)`
);
return {
allRateLimited: true,
retryAfter: new Date(Date.now() + (cooldownRemaining || 0)).toISOString(),
retryAfterHuman: formatRetryAfter(
new Date(Date.now() + (cooldownRemaining || 0)).toISOString()
),
};
}
log.debug(
"AUTH",
`${provider} | available: ${availableConnections.length}/${connections.length}`
@@ -1206,6 +1217,15 @@ export async function markAccountUnavailable(
const effectiveProviderProfile =
providerProfile || (provider ? await getRuntimeProviderProfile(provider) : null);
const fallbackResult = checkFallbackError(
status,
errorText,
backoffLevel,
model,
provider,
null,
effectiveProviderProfile
);
// Read passthroughModels from connection config (user-configured per-model quota)
const connProviderSpecificData = (conn?.providerSpecificData as Record<string, unknown>) || {};
@@ -1216,18 +1236,20 @@ export async function markAccountUnavailable(
const isPerModelQuotaProvider = hasPerModelQuota(provider, model, connectionPassthroughModels);
if (isPerModelQuotaProvider && provider && model && (status === 404 || status === 429)) {
const reason = status === 404 ? "not_found" : "rate_limited";
const fallbackCooldown =
status === 404
? (effectiveProviderProfile?.transientCooldown ?? COOLDOWN_MS.notFoundLocal)
: 0;
const lockout = recordModelLockoutFailure(
provider,
connectionId,
model,
reason,
status,
fallbackCooldown,
effectiveProviderProfile
status === 404
? (effectiveProviderProfile?.baseCooldownMs ?? COOLDOWN_MS.notFoundLocal)
: (fallbackResult.baseCooldownMs ?? effectiveProviderProfile?.baseCooldownMs ?? 0),
effectiveProviderProfile,
{
exactCooldownMs:
fallbackResult.usedUpstreamRetryHint === true ? fallbackResult.cooldownMs : null,
}
);
// Update last error for observability (without changing terminal status)
updateProviderConnection(connectionId, {
@@ -1242,18 +1264,12 @@ export async function markAccountUnavailable(
);
return { shouldFallback: true, cooldownMs: lockout.cooldownMs };
}
const result = checkFallbackError(
status,
errorText,
backoffLevel,
model,
provider,
null,
effectiveProviderProfile
);
const { shouldFallback, cooldownMs, newBackoffLevel, reason } = result;
const result = fallbackResult;
const { shouldFallback, cooldownMs: rawCooldownMs, newBackoffLevel, reason } = result;
if (!shouldFallback) return { shouldFallback: false, cooldownMs: 0 };
const providerErrorType = classifyProviderError(status, errorText);
const terminalStatus = resolveTerminalConnectionStatus(status, result, providerErrorType);
const cooldownMs = terminalStatus ? 0 : rawCooldownMs;
// ── 404 model-only lockout: connection stays active ──
// For local providers (detected by URL), a 404 means the specific model
@@ -1286,7 +1302,6 @@ export async function markAccountUnavailable(
return { shouldFallback: true, cooldownMs: lockout.cooldownMs };
}
const rateLimitedUntil = getUnavailableUntil(cooldownMs);
const errorMsg = typeof errorText === "string" ? errorText.slice(0, 100) : "Provider error";
// T09: Codex per-scope lockout (do not block the whole account globally).
@@ -1294,7 +1309,7 @@ export async function markAccountUnavailable(
const scope = getCodexModelScope(model);
const existingScopeMap = asRecord(conn.providerSpecificData.codexScopeRateLimitedUntil);
const persistedScopeUntil = getCodexScopeRateLimitedUntil(conn.providerSpecificData, model);
const scopeRateLimitedUntil = persistedScopeUntil || rateLimitedUntil;
const scopeRateLimitedUntil = persistedScopeUntil || getUnavailableUntil(cooldownMs);
const scopeCooldownMs = Math.max(new Date(scopeRateLimitedUntil).getTime() - Date.now(), 0);
await updateProviderConnection(connectionId, {
@@ -1323,14 +1338,27 @@ export async function markAccountUnavailable(
return { shouldFallback: true, cooldownMs: scopeCooldownMs };
}
await updateProviderConnection(connectionId, {
rateLimitedUntil,
testStatus: "unavailable",
const baseUpdate = {
lastError: errorMsg,
lastErrorType: providerErrorType,
errorCode: status,
lastErrorAt: new Date().toISOString(),
backoffLevel: newBackoffLevel ?? backoffLevel,
});
};
if (cooldownMs > 0) {
await updateProviderConnection(connectionId, {
...baseUpdate,
rateLimitedUntil: getUnavailableUntil(cooldownMs),
testStatus: "unavailable",
});
} else {
await updateProviderConnection(connectionId, {
...baseUpdate,
rateLimitedUntil: null,
...(terminalStatus ? { testStatus: terminalStatus } : {}),
});
}
// T-AUTODISABLE: If auto-disable setting is enabled and error is permanent/terminal,
// mark account as inactive so it is never retried again.
@@ -1354,11 +1382,6 @@ export async function markAccountUnavailable(
}
}
// Per-model lockout: lock the specific model if known
if (provider && model && cooldownMs > 0) {
lockModel(provider, connectionId, model, reason || "unknown", cooldownMs);
}
if (provider && status && errorMsg) {
console.error(`❌ ${provider} [${status}]: ${errorMsg}`);
}

41
src/sse/services/cooldownAwareRetry.ts
View File

@@ -1,14 +1,14 @@
import { formatRetryAfter } from "@omniroute/open-sse/services/accountFallback.ts";
import { resolveResilienceSettings } from "@/lib/resilience/settings";
const DEFAULT_REQUEST_RETRY = 3;
const DEFAULT_MAX_RETRY_INTERVAL_SEC = 30;
const MAX_REQUEST_RETRY = 10;
const MAX_RETRY_INTERVAL_SEC = 300;
export interface CooldownAwareRetrySettings {
requestRetry: number;
maxRetryIntervalSec: number;
maxRetryIntervalMs: number;
enabled: boolean;
maxRetries: number;
maxRetryWaitSec: number;
maxRetryWaitMs: number;
}
function normalizeInteger(
@@ -35,20 +35,26 @@ function normalizeInteger(
export function resolveCooldownAwareRetrySettings(
settings: Record<string, unknown> | null | undefined
): CooldownAwareRetrySettings {
const requestRetry = normalizeInteger(settings?.requestRetry, DEFAULT_REQUEST_RETRY, {
const waitForCooldown = resolveResilienceSettings(settings).waitForCooldown;
const maxRetries = normalizeInteger(waitForCooldown.maxRetries, waitForCooldown.maxRetries, {
min: 0,
max: MAX_REQUEST_RETRY,
});
const maxRetryIntervalSec = normalizeInteger(
settings?.maxRetryIntervalSec,
DEFAULT_MAX_RETRY_INTERVAL_SEC,
{ min: 0, max: MAX_RETRY_INTERVAL_SEC }
const maxRetryWaitSec = normalizeInteger(
waitForCooldown.maxRetryWaitSec,
waitForCooldown.maxRetryWaitSec,
{
min: 0,
max: MAX_RETRY_INTERVAL_SEC,
}
);
const enabled = Boolean(waitForCooldown.enabled) && maxRetries > 0 && maxRetryWaitSec > 0;
return {
requestRetry,
maxRetryIntervalSec,
maxRetryIntervalMs: maxRetryIntervalSec * 1000,
enabled,
maxRetries,
maxRetryWaitSec,
maxRetryWaitMs: maxRetryWaitSec * 1000,
};
}
@@ -90,9 +96,10 @@ export function getCooldownAwareRetryDecision({
} {
const closest = computeClosestRetryAfter(retryAfter);
if (
settings.requestRetry <= 0 ||
settings.maxRetryIntervalMs <= 0 ||
attempt >= settings.requestRetry ||
!settings.enabled ||
settings.maxRetries <= 0 ||
settings.maxRetryWaitMs <= 0 ||
attempt >= settings.maxRetries ||
closest.waitMs === null
) {
return {
@@ -103,7 +110,7 @@ export function getCooldownAwareRetryDecision({
};
}
if (closest.waitMs > settings.maxRetryIntervalMs) {
if (closest.waitMs > settings.maxRetryWaitMs) {
return {
shouldRetry: false,
retryAfter: closest.retryAfter,

5
src/types/settings.ts
View File

@@ -1,4 +1,5 @@
import type { HideableSidebarItemId } from "@/shared/constants/sidebarVisibility";
import type { ResilienceSettings } from "@/lib/resilience/settings";
/**
* Application settings stored in SQLite key-value pairs.
@@ -20,15 +21,13 @@ export interface Settings {
jwtSecret?: string;
hideHealthCheckLogs?: boolean;
hiddenSidebarItems?: HideableSidebarItemId[];
resilienceSettings?: ResilienceSettings;
}
export interface ComboDefaults {
strategy: "priority" | "weighted" | "round-robin" | "context-relay";
maxRetries: number;
retryDelayMs: number;
timeoutMs: number;
healthCheckEnabled: boolean;
healthCheckTimeoutMs: number;
maxComboDepth: number;
trackMetrics: boolean;
concurrencyPerModel?: number;

374
tests/e2e/resilience-plan-alignment.spec.ts Normal file
View File

@@ -0,0 +1,374 @@
import { expect, test, type Page } from "@playwright/test";
import { gotoDashboardRoute } from "./helpers/dashboardAuth";
const resilienceSettings = {
requestQueue: {
autoEnableApiKeyProviders: true,
requestsPerMinute: 100,
minTimeBetweenRequestsMs: 200,
concurrentRequests: 10,
maxWaitMs: 120000,
},
connectionCooldown: {
oauth: {
baseCooldownMs: 60000,
useUpstreamRetryHints: false,
maxBackoffSteps: 8,
},
apikey: {
baseCooldownMs: 3000,
useUpstreamRetryHints: true,
maxBackoffSteps: 5,
},
},
providerBreaker: {
oauth: {
failureThreshold: 3,
resetTimeoutMs: 60000,
},
apikey: {
failureThreshold: 5,
resetTimeoutMs: 30000,
},
},
waitForCooldown: {
enabled: true,
maxRetries: 3,
maxRetryWaitSec: 30,
},
};
async function mockResilienceSettings(page: Page) {
await page.route("**/api/resilience", async (route) => {
const method = route.request().method();
if (method === "GET") {
await route.fulfill({
status: 200,
contentType: "application/json",
body: JSON.stringify(resilienceSettings),
});
return;
}
await route.fulfill({
status: 200,
contentType: "application/json",
body: JSON.stringify({
ok: true,
...resilienceSettings,
}),
});
});
}
async function mockHealthPageApis(page: Page) {
const now = new Date().toISOString();
await page.route("**/api/monitoring/health", async (route) => {
await route.fulfill({
status: 200,
contentType: "application/json",
body: JSON.stringify({
status: "error",
system: {
uptime: 3723,
version: "3.7.0",
nodeVersion: "22.12.0",
memoryUsage: {
rss: 64 * 1024 * 1024,
heapUsed: 24 * 1024 * 1024,
heapTotal: 48 * 1024 * 1024,
},
},
providerHealth: {
openai: {
state: "OPEN",
failures: 3,
retryAfterMs: 15000,
lastFailure: now,
},
gemini: {
state: "HALF_OPEN",
failures: 1,
retryAfterMs: 5000,
lastFailure: now,
},
groq: {
state: "CLOSED",
failures: 0,
retryAfterMs: 0,
lastFailure: null,
},
},
providerSummary: {
configuredCount: 3,
activeCount: 2,
monitoredCount: 3,
},
rateLimitStatus: {},
lockouts: {},
sessions: {
activeCount: 0,
stickyBoundCount: 0,
byApiKey: {},
top: [],
},
quotaMonitor: {
active: 0,
alerting: 0,
exhausted: 0,
errors: 0,
byProvider: {},
monitors: [],
},
}),
});
});
await page.route("**/api/telemetry/summary", async (route) => {
await route.fulfill({
status: 200,
contentType: "application/json",
body: JSON.stringify({ p50: 120, p95: 240, p99: 450, totalRequests: 18 }),
});
});
await page.route("**/api/cache/stats", async (route) => {
await route.fulfill({
status: 200,
contentType: "application/json",
body: JSON.stringify({ size: 3, maxSize: 100, hitRate: 50, hits: 2, misses: 2 }),
});
});
await page.route("**/api/rate-limits", async (route) => {
await route.fulfill({
status: 200,
contentType: "application/json",
body: JSON.stringify({
cacheStats: {
defaultCount: 0,
tool: { entries: 0, patterns: 0 },
family: { entries: 0, patterns: 0 },
session: { entries: 0, patterns: 0 },
},
}),
});
});
await page.route("**/api/health/degradation", async (route) => {
await route.fulfill({
status: 200,
contentType: "application/json",
body: JSON.stringify({
summary: { full: 0, reduced: 0, minimal: 0, default: 0 },
features: [],
}),
});
});
await page.route("**/api/v1/db/health", async (route) => {
await route.fulfill({
status: 200,
contentType: "application/json",
body: JSON.stringify({
isHealthy: true,
issues: [],
repairedCount: 0,
backupCreated: false,
}),
});
});
}
async function mockProvidersPageApis(page: Page) {
await page.route("**/api/providers", async (route) => {
await route.fulfill({
status: 200,
contentType: "application/json",
body: JSON.stringify({
connections: [
{
id: "conn-openai-main",
provider: "openai",
authType: "apikey",
name: "OpenAI Main",
testStatus: "active",
},
{
id: "conn-gemini-main",
provider: "gemini",
authType: "apikey",
name: "Gemini Main",
testStatus: "active",
},
],
}),
});
});
await page.route("**/api/provider-nodes", async (route) => {
await route.fulfill({
status: 200,
contentType: "application/json",
body: JSON.stringify({ nodes: [], ccCompatibleProviderEnabled: false }),
});
});
await page.route("**/api/providers/expiration", async (route) => {
await route.fulfill({
status: 200,
contentType: "application/json",
body: JSON.stringify({}),
});
});
await page.route("**/api/system/env/repair", async (route) => {
await route.fulfill({
status: 200,
contentType: "application/json",
body: JSON.stringify({ available: false, missingCount: 0 }),
});
});
}
async function mockIntelligentCombosPageApis(page: Page) {
await page.route("**/api/combos", async (route) => {
await route.fulfill({
status: 200,
contentType: "application/json",
body: JSON.stringify({
combos: [
{
id: "combo-auto",
name: "combo-auto",
models: ["openai/gpt-4o-mini", "gemini/gemini-2.5-pro"],
strategy: "auto",
config: {
candidatePool: ["openai", "gemini", "anthropic"],
modePack: "ship-fast",
explorationRate: 0.15,
},
isActive: true,
},
],
}),
});
});
await page.route("**/api/combos/metrics", async (route) => {
await route.fulfill({
status: 200,
contentType: "application/json",
body: JSON.stringify({ metrics: {} }),
});
});
await page.route("**/api/providers", async (route) => {
await route.fulfill({
status: 200,
contentType: "application/json",
body: JSON.stringify({
connections: [
{ id: "conn-openai", provider: "openai", name: "OpenAI", testStatus: "active" },
{ id: "conn-gemini", provider: "gemini", name: "Gemini", testStatus: "active" },
{
id: "conn-anthropic",
provider: "anthropic",
name: "Anthropic",
testStatus: "active",
},
],
}),
});
});
await page.route("**/api/provider-nodes", async (route) => {
await route.fulfill({
status: 200,
contentType: "application/json",
body: JSON.stringify({ nodes: [] }),
});
});
await page.route("**/api/settings/proxy", async (route) => {
await route.fulfill({
status: 200,
contentType: "application/json",
body: JSON.stringify({ combos: {} }),
});
});
}
test.describe("Resilience Plan Alignment", () => {
test("resilience settings page only shows the plan-aligned cooldown fields", async ({ page }) => {
await mockResilienceSettings(page);
await gotoDashboardRoute(page, "/dashboard/settings?tab=resilience");
await expect(page.getByText("Connection Cooldown")).toBeVisible({ timeout: 15000 });
await expect(page.getByText("Base cooldown", { exact: true }).first()).toBeVisible();
await expect(page.getByText("Use upstream retry hints", { exact: true }).first()).toBeVisible();
await expect(page.getByText("Max backoff steps", { exact: true }).first()).toBeVisible();
await expect(page.getByText(/Rate-limit fallback/i)).toHaveCount(0);
});
test("health page renders provider breaker runtime state for multiple providers", async ({
page,
}) => {
await mockHealthPageApis(page);
await gotoDashboardRoute(page, "/dashboard/health");
await expect(page.getByText("Provider Health")).toBeVisible({ timeout: 15000 });
await expect(page.getByText("OpenAI")).toBeVisible();
await expect(page.getByText("Gemini")).toBeVisible();
await expect(page.getByText("Groq")).toBeVisible();
await expect(page.getByText("Recovering", { exact: true }).first()).toBeVisible();
await expect(page.getByText("Down", { exact: true }).first()).toBeVisible();
});
test("providers page no longer requests legacy model availability data", async ({ page }) => {
let availabilityRequests = 0;
await mockProvidersPageApis(page);
await page.route("**/api/models/availability", async (route) => {
availabilityRequests += 1;
await route.fulfill({
status: 410,
contentType: "application/json",
body: JSON.stringify({ error: "removed" }),
});
});
await gotoDashboardRoute(page, "/dashboard/providers");
await page.waitForLoadState("networkidle");
expect(availabilityRequests).toBe(0);
await expect(page.getByText("OpenAI").first()).toBeVisible();
await expect(page.getByText(/Model Availability/i)).toHaveCount(0);
});
test("intelligent combo panel stays config-only and does not fetch breaker runtime state", async ({
page,
}) => {
let monitoringHealthRequests = 0;
await mockIntelligentCombosPageApis(page);
await page.route("**/api/monitoring/health", async (route) => {
monitoringHealthRequests += 1;
await route.fulfill({
status: 200,
contentType: "application/json",
body: JSON.stringify({ providerHealth: {} }),
});
});
await gotoDashboardRoute(page, "/dashboard/combos?filter=intelligent");
await page.waitForLoadState("networkidle");
expect(monitoringHealthRequests).toBe(0);
await expect(page.getByText("Intelligent Routing Dashboard")).toBeVisible({ timeout: 15000 });
await expect(page.getByText("Routing Inputs", { exact: true })).toBeVisible();
await expect(page.getByText(/Excluded Providers/i)).toHaveCount(0);
await expect(page.getByText(/Incident Mode/i)).toHaveCount(0);
});
});

5
tests/integration/_chatPipelineHarness.ts
View File

@@ -33,8 +33,6 @@ export async function createChatPipelineHarness(prefix) {
const { initTranslators } = await import("../../open-sse/translator/index.ts");
const { clearInflight } = await import("../../open-sse/services/requestDedup.ts");
const { BaseExecutor } = await import("../../open-sse/executors/base.ts");
const { resetAllAvailability, setModelUnavailable } =
await import("../../src/domain/modelAvailability.ts");
const { resetAllCircuitBreakers } = await import("../../src/shared/utils/circuitBreaker.ts");
const originalFetch = globalThis.fetch;
@@ -209,7 +207,6 @@ export async function createChatPipelineHarness(prefix) {
clearInflight();
idempotencyLayerModule.clearIdempotency();
semanticCacheModule.clearCache();
resetAllAvailability();
resetAllCircuitBreakers();
apiKeysDb.resetApiKeyState();
readCacheDb.invalidateDbCache();
@@ -229,7 +226,6 @@ export async function createChatPipelineHarness(prefix) {
idempotencyLayerModule.clearIdempotency();
semanticCacheModule.clearCache();
clearSkillState();
resetAllAvailability();
resetAllCircuitBreakers();
core.resetDbInstance();
fs.rmSync(testDataDir, { recursive: true, force: true });
@@ -292,7 +288,6 @@ export async function createChatPipelineHarness(prefix) {
semanticCacheModule,
seedApiKey,
seedConnection,
setModelUnavailable,
settingsDb,
skillByIdRouteModule,
skillExecutor,

67
tests/integration/chat-pipeline.test.ts
View File

@@ -23,9 +23,8 @@ const { handleChat } = await import("../../src/sse/handlers/chat.ts");
const { initTranslators } = await import("../../open-sse/translator/index.ts");
const { clearInflight } = await import("../../open-sse/services/requestDedup.ts");
const { BaseExecutor } = await import("../../open-sse/executors/base.ts");
const { resetAllAvailability, setModelUnavailable } =
await import("../../src/domain/modelAvailability.ts");
const { resetAllCircuitBreakers } = await import("../../src/shared/utils/circuitBreaker.ts");
const { getCircuitBreaker, resetAllCircuitBreakers } =
await import("../../src/shared/utils/circuitBreaker.ts");
const { clearProviderFailure } = await import("../../open-sse/services/accountFallback.ts");
const originalFetch = globalThis.fetch;
@@ -295,7 +294,6 @@ async function resetStorage() {
globalThis.fetch = originalFetch;
process.env.REQUIRE_API_KEY = "false";
clearInflight();
resetAllAvailability();
resetAllCircuitBreakers();
apiKeysDb.resetApiKeyState();
readCacheDb.invalidateDbCache();
@@ -442,7 +440,6 @@ test.after(async () => {
BaseExecutor.RETRY_CONFIG.delayMs = originalRetryDelayMs;
globalThis.fetch = originalFetch;
clearInflight();
resetAllAvailability();
resetAllCircuitBreakers();
core.resetDbInstance();
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
@@ -909,26 +906,6 @@ test("chat pipeline returns current no-credentials contract when no provider con
assert.match(json.error.message, /No credentials for provider: openai/);
});
test("chat pipeline returns 503 when the requested model is temporarily unavailable", async () => {
await seedConnection("openai", { apiKey: "sk-openai-unavailable" });
setModelUnavailable("openai", "gpt-4o-mini", 60000, "test cooldown");
const response = await handleChat(
buildRequest({
body: {
model: "openai/gpt-4o-mini",
stream: false,
messages: [{ role: "user", content: "Provider unavailable" }],
},
})
);
const json = await response.json();
assert.equal(response.status, 503);
assert.ok(Number(response.headers.get("Retry-After")) >= 1);
assert.match(json.error.message, /temporarily unavailable/i);
});
test("chat pipeline surfaces upstream 500 responses as structured errors", async () => {
await seedConnection("openai", { apiKey: "sk-openai-500" });
@@ -993,6 +970,46 @@ test("chat pipeline returns 429 with Retry-After when the upstream rate-limits t
assert.match(json.error.message, /\[openai\/gpt-4o-mini\]/);
});
test("chat pipeline keeps provider breaker closed for repeated connection-scoped 429s", async () => {
await seedConnection("openai", { apiKey: "sk-openai-429-breaker" });
await settingsDb.updateSettings({
requestRetry: 0,
maxRetryIntervalSec: 0,
});
globalThis.fetch = async () =>
new Response(
JSON.stringify({
error: {
message: "Rate limit exceeded. Your quota will reset after 30s.",
},
}),
{
status: 429,
headers: { "Content-Type": "application/json" },
}
);
for (let i = 0; i < 3; i += 1) {
const response = await handleChat(
buildRequest({
body: {
model: "openai/gpt-4o-mini",
stream: false,
messages: [{ role: "user", content: `Trigger 429 #${i + 1}` }],
},
})
);
assert.equal(response.status, 429);
}
const breaker = getCircuitBreaker("openai");
const status = breaker.getStatus();
assert.equal(status.state, "CLOSED");
assert.equal(status.failureCount, 0);
});
test("chat pipeline maps upstream timeouts to 504 responses", async () => {
await seedConnection("openai", { apiKey: "sk-openai-timeout" });

2
tests/integration/chatcore-compression-integration.test.ts
View File

@@ -15,14 +15,12 @@ const readCacheDb = await import("../../src/lib/db/readCache.ts");
const combosDb = await import("../../src/lib/db/combos.ts");
const { handleChatCore } = await import("../../open-sse/handlers/chatCore.ts");
const { estimateTokens, getTokenLimit } = await import("../../open-sse/services/contextManager.ts");
const { resetAllAvailability } = await import("../../src/domain/modelAvailability.ts");
const { resetAllCircuitBreakers } = await import("../../src/shared/utils/circuitBreaker.ts");
const originalFetch = globalThis.fetch;
async function resetStorage() {
globalThis.fetch = originalFetch;
resetAllAvailability();
resetAllCircuitBreakers();
readCacheDb.invalidateDbCache();
await new Promise((resolve) => setTimeout(resolve, 20));

9
tests/integration/integration-wiring.test.ts
View File

@@ -84,8 +84,8 @@ describe("Pipeline Wiring — sse chat handler", () => {
assert.match(src, /getCircuitBreaker|CircuitBreakerOpenError/);
});
it("should import model availability integration", () => {
assert.match(src, /isModelAvailable|setModelUnavailable/);
it("should use credential preflight instead of global model quarantine gates", () => {
assert.match(src, /getProviderCredentialsWithQuotaPreflight/);
});
it("should import request telemetry integration", () => {
@@ -129,7 +129,6 @@ describe("Pipeline Wiring — middleware proxy", () => {
describe("API Routes — existence check", () => {
const routes = [
"src/app/api/cache/stats/route.ts",
"src/app/api/models/availability/route.ts",
"src/app/api/telemetry/summary/route.ts",
"src/app/api/usage/budget/route.ts",
"src/app/api/usage/quota/route.ts",
@@ -152,10 +151,6 @@ describe("API Routes — export HTTP methods", () => {
assertRouteMethods("src/app/api/cache/stats/route.ts", ["GET", "DELETE"]);
});
it("/api/models/availability should export GET and POST", () => {
assertRouteMethods("src/app/api/models/availability/route.ts", ["GET", "POST"]);
});
it("/api/telemetry/summary should export GET", () => {
assertRouteMethods("src/app/api/telemetry/summary/route.ts", ["GET"]);
});

14
tests/integration/proxy-pipeline.test.ts
View File

@@ -88,25 +88,25 @@ describe("Chat Pipeline — combo fallback support", () => {
assert.match(src, /handleSingleModel.*handleSingleModelChat/s);
});
it("should check model availability before attempting combo models", () => {
assert.match(src, /isModelAvailable/);
it("should preflight provider credentials before attempting combo models", () => {
assert.match(src, /getProviderCredentialsWithQuotaPreflight/);
});
});
describe("Chat Pipeline — circuit breaker integration", () => {
const helpersSrc = readSrc("sse/handlers/chatHelpers.ts");
it("should import CircuitBreakerOpenError", () => {
it("should import providerCircuitOpenResponse", () => {
assert.ok(helpersSrc, "chatHelpers.ts should exist");
assert.match(helpersSrc, /CircuitBreakerOpenError/);
assert.match(helpersSrc, /providerCircuitOpenResponse/);
});
it("should handle CircuitBreakerOpenError with retry-after", () => {
it("should handle circuit-open responses with retry-after", () => {
assert.match(helpersSrc, /retryAfterMs/);
});
it("should reject requests when circuit is open", () => {
assert.match(helpersSrc, /circuit breaker is open/i);
it("should reject requests when circuit is open via structured provider breaker response", () => {
assert.match(helpersSrc, /providerCircuitOpenResponse\(provider,\s*retryAfterSec\)/);
});
});

746
tests/integration/resilience-http-e2e.test.ts Normal file
View File

@@ -0,0 +1,746 @@
import test from "node:test";
import assert from "node:assert/strict";
import fs from "node:fs";
import fsp from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import http from "node:http";
import net from "node:net";
import { spawn } from "node:child_process";
import { fileURLToPath } from "node:url";
const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-resilience-http-e2e-"));
const DASHBOARD_PORT = await getFreePort();
const REPO_ROOT = fileURLToPath(new URL("../..", import.meta.url));
process.env.DATA_DIR = TEST_DATA_DIR;
process.env.DISABLE_SQLITE_AUTO_BACKUP = "true";
process.env.API_KEY_SECRET = process.env.API_KEY_SECRET || "resilience-http-e2e-secret-123456";
process.env.REQUIRE_API_KEY = "false";
const core = await import("../../src/lib/db/core.ts");
const providersDb = await import("../../src/lib/db/providers.ts");
const combosDb = await import("../../src/lib/db/combos.ts");
const settingsDb = await import("../../src/lib/db/settings.ts");
function getFreePort() {
return new Promise<number>((resolve, reject) => {
const server = net.createServer();
server.once("error", reject);
server.listen(0, "127.0.0.1", () => {
const address = server.address();
if (!address || typeof address === "string") {
server.close();
reject(new Error("Failed to allocate a free port"));
return;
}
const { port } = address;
server.close((closeError) => {
if (closeError) reject(closeError);
else resolve(port);
});
});
});
}
function sleep(ms: number) {
return new Promise((resolve) => setTimeout(resolve, ms));
}
function buildCompletion(content: string) {
return {
status: 200,
body: {
id: `chatcmpl_${Math.random().toString(16).slice(2, 8)}`,
object: "chat.completion",
choices: [{ index: 0, message: { role: "assistant", content }, finish_reason: "stop" }],
usage: { prompt_tokens: 4, completion_tokens: 2, total_tokens: 6 },
},
};
}
function buildError(status: number, message: string, headers: Record<string, string> = {}) {
return {
status,
headers,
body: { error: { message } },
};
}
type PlannedResponse = {
status: number;
body: Record<string, unknown>;
headers?: Record<string, string>;
delayMs?: number;
};
type TokenBehavior = {
defaultResponse: PlannedResponse;
queue: PlannedResponse[];
hits: number;
startedAt: number[];
bodies: Array<Record<string, unknown>>;
};
function createFakeOpenAiRelay() {
const behaviors = new Map<string, TokenBehavior>();
let server: http.Server | null = null;
let baseUrl = "";
const handleRequest = async (req: http.IncomingMessage, res: http.ServerResponse) => {
const chunks: Buffer[] = [];
req.on("data", (chunk) => chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk)));
req.on("end", async () => {
const authHeader = String(req.headers.authorization || "");
const token = authHeader.replace(/^Bearer\s+/i, "").trim();
const rawBody = Buffer.concat(chunks).toString("utf8");
const parsedBody = rawBody ? JSON.parse(rawBody) : {};
if (req.method === "GET" && req.url?.startsWith("/v1/models")) {
res.writeHead(200, { "Content-Type": "application/json" });
res.end(JSON.stringify({ data: [{ id: "test-model", object: "model" }] }));
return;
}
if (req.method !== "POST" || !req.url?.startsWith("/v1/chat/completions")) {
res.writeHead(404, { "Content-Type": "application/json" });
res.end(JSON.stringify({ error: { message: `Unhandled path: ${req.method} ${req.url}` } }));
return;
}
const behavior = behaviors.get(token);
if (!behavior) {
res.writeHead(401, { "Content-Type": "application/json" });
res.end(JSON.stringify({ error: { message: `Unknown token: ${token || "missing"}` } }));
return;
}
behavior.hits += 1;
behavior.startedAt.push(Date.now());
behavior.bodies.push(parsedBody as Record<string, unknown>);
const planned = behavior.queue.shift() || behavior.defaultResponse;
if (planned.delayMs && planned.delayMs > 0) {
await sleep(planned.delayMs);
}
const headers = { "Content-Type": "application/json", ...(planned.headers || {}) };
res.writeHead(planned.status, headers);
res.end(JSON.stringify(planned.body));
});
};
return {
async start() {
const port = await getFreePort();
await new Promise<void>((resolve, reject) => {
server = http.createServer((req, res) => {
void handleRequest(req, res);
});
server.once("error", reject);
server.listen(port, "127.0.0.1", () => resolve());
});
baseUrl = `http://127.0.0.1:${port}/v1`;
return baseUrl;
},
getBaseUrl() {
if (!baseUrl) throw new Error("Fake relay has not started yet");
return baseUrl;
},
configureToken(
token: string,
config: { defaultResponse: PlannedResponse; queue?: PlannedResponse[] }
) {
behaviors.set(token, {
defaultResponse: config.defaultResponse,
queue: [...(config.queue || [])],
hits: 0,
startedAt: [],
bodies: [],
});
},
getState(token: string) {
const state = behaviors.get(token);
if (!state) throw new Error(`Unknown token state for ${token}`);
return state;
},
resetState(token: string, queue?: PlannedResponse[]) {
const state = behaviors.get(token);
if (!state) throw new Error(`Unknown token state for ${token}`);
state.hits = 0;
state.startedAt = [];
state.bodies = [];
state.queue = [...(queue || [])];
},
async stop() {
if (!server) return;
await new Promise<void>((resolve) => server?.close(() => resolve()));
server = null;
},
};
}
function createServerProcess(dataDir: string, port: number) {
const stdoutLines: string[] = [];
const stderrLines: string[] = [];
const child = spawn(process.execPath, ["scripts/run-next.mjs", "dev"], {
cwd: REPO_ROOT,
env: {
...process.env,
DATA_DIR: dataDir,
PORT: String(port),
DASHBOARD_PORT: String(port),
API_PORT: String(port),
HOST: "127.0.0.1",
REQUIRE_API_KEY: "false",
API_KEY_SECRET: process.env.API_KEY_SECRET || "resilience-http-e2e-secret-123456",
DISABLE_SQLITE_AUTO_BACKUP: "true",
INITIAL_PASSWORD: "",
NEXT_TELEMETRY_DISABLED: "1",
OMNIROUTE_DISABLE_BACKGROUND_SERVICES: "true",
OMNIROUTE_DISABLE_TOKEN_HEALTHCHECK: "true",
OMNIROUTE_DISABLE_LOCAL_HEALTHCHECK: "true",
OMNIROUTE_HIDE_HEALTHCHECK_LOGS: "true",
},
stdio: ["ignore", "pipe", "pipe"],
});
child.stdout.on("data", (chunk) => {
const lines = String(chunk).split(/\r?\n/).filter(Boolean);
stdoutLines.push(...lines);
if (stdoutLines.length > 200) stdoutLines.splice(0, stdoutLines.length - 200);
});
child.stderr.on("data", (chunk) => {
const lines = String(chunk).split(/\r?\n/).filter(Boolean);
stderrLines.push(...lines);
if (stderrLines.length > 200) stderrLines.splice(0, stderrLines.length - 200);
});
return {
child,
stdoutLines,
stderrLines,
baseUrl: `http://127.0.0.1:${port}`,
};
}
async function waitForServer(
baseUrl: string,
logs: { stdoutLines: string[]; stderrLines: string[] }
) {
const startedAt = Date.now();
let lastError = "";
while (Date.now() - startedAt < 120_000) {
try {
const response = await fetch(`${baseUrl}/api/monitoring/health`, {
signal: AbortSignal.timeout(5_000),
});
if (response.ok) return;
lastError = `HTTP ${response.status}`;
} catch (error) {
lastError = error instanceof Error ? error.message : String(error);
}
await sleep(500);
}
throw new Error(
[
`Timed out waiting for OmniRoute to start: ${lastError}`,
"--- stdout ---",
...logs.stdoutLines.slice(-40),
"--- stderr ---",
...logs.stderrLines.slice(-40),
].join("\n")
);
}
async function stopProcess(child: ReturnType<typeof spawn>) {
if (child.killed) return;
child.kill("SIGTERM");
const exited = await Promise.race([
new Promise<boolean>((resolve) => child.once("exit", () => resolve(true))),
sleep(5_000).then(() => false),
]);
if (!exited && !child.killed) {
child.kill("SIGKILL");
await new Promise<void>((resolve) => child.once("exit", () => resolve()));
}
}
async function seedCompatibleProvider(prefix: string, apiKey: string, baseUrl: string) {
const providerId = `openai-compatible-chat-e2e-${prefix}`;
await providersDb.createProviderNode({
id: providerId,
type: "openai-compatible",
name: `E2E ${prefix}`,
prefix,
apiType: "chat",
baseUrl,
});
await providersDb.createProviderConnection({
provider: providerId,
authType: "apikey",
name: `conn-${prefix}`,
apiKey,
isActive: true,
testStatus: "active",
providerSpecificData: {
baseUrl,
apiType: "chat",
},
});
return { providerId, model: `${prefix}/test-model`, apiKey };
}
function buildResilienceConfig(overrides: Record<string, unknown> = {}) {
const base = {
requestQueue: {
autoEnableApiKeyProviders: true,
requestsPerMinute: 120,
minTimeBetweenRequestsMs: 0,
concurrentRequests: 4,
maxWaitMs: 2_000,
},
connectionCooldown: {
oauth: {
baseCooldownMs: 500,
useUpstreamRetryHints: true,
maxBackoffSteps: 3,
},
apikey: {
baseCooldownMs: 300,
useUpstreamRetryHints: false,
maxBackoffSteps: 0,
},
},
providerBreaker: {
oauth: {
failureThreshold: 3,
resetTimeoutMs: 2_000,
},
apikey: {
failureThreshold: 2,
resetTimeoutMs: 1_500,
},
},
waitForCooldown: {
enabled: false,
maxRetries: 0,
maxRetryWaitSec: 0,
},
};
return {
...base,
...overrides,
requestQueue: {
...base.requestQueue,
...((overrides.requestQueue as Record<string, unknown>) || {}),
},
connectionCooldown: {
oauth: {
...base.connectionCooldown.oauth,
...(((overrides.connectionCooldown as Record<string, unknown>)?.oauth as Record<
string,
unknown
>) || {}),
},
apikey: {
...base.connectionCooldown.apikey,
...(((overrides.connectionCooldown as Record<string, unknown>)?.apikey as Record<
string,
unknown
>) || {}),
},
},
providerBreaker: {
oauth: {
...base.providerBreaker.oauth,
...(((overrides.providerBreaker as Record<string, unknown>)?.oauth as Record<
string,
unknown
>) || {}),
},
apikey: {
...base.providerBreaker.apikey,
...(((overrides.providerBreaker as Record<string, unknown>)?.apikey as Record<
string,
unknown
>) || {}),
},
},
waitForCooldown: {
...base.waitForCooldown,
...((overrides.waitForCooldown as Record<string, unknown>) || {}),
},
};
}
async function patchResilience(baseUrl: string, config: Record<string, unknown>) {
const response = await fetch(`${baseUrl}/api/resilience`, {
method: "PATCH",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(config),
signal: AbortSignal.timeout(10_000),
});
const payload = await response.json();
assert.equal(response.status, 200, JSON.stringify(payload));
return payload;
}
async function getJson(url: string) {
const response = await fetch(url, { signal: AbortSignal.timeout(10_000) });
const json = await response.json();
return { response, json };
}
async function postChat(baseUrl: string, model: string, content: string) {
const response = await fetch(`${baseUrl}/api/v1/chat/completions`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
model,
stream: false,
messages: [{ role: "user", content }],
}),
signal: AbortSignal.timeout(20_000),
});
const text = await response.text();
const json = text ? JSON.parse(text) : {};
return { response, json };
}
const relay = createFakeOpenAiRelay();
let app:
| {
child: ReturnType<typeof spawn>;
stdoutLines: string[];
stderrLines: string[];
baseUrl: string;
}
| undefined;
const TOKENS = {
p1: "sk-e2e-p1",
p2: "sk-e2e-p2",
p3: "sk-e2e-p3",
p4: "sk-e2e-p4",
p5: "sk-e2e-p5",
p6: "sk-e2e-p6",
p7: "sk-e2e-p7",
p8: "sk-e2e-p8",
};
test.before(async () => {
const fakeBaseUrl = await relay.start();
relay.configureToken(TOKENS.p1, {
defaultResponse: buildCompletion("primary healthy again"),
queue: [buildError(503, "primary transient failure")],
});
relay.configureToken(TOKENS.p2, {
defaultResponse: buildCompletion("secondary stable"),
});
relay.configureToken(TOKENS.p3, {
defaultResponse: buildCompletion("wait-for-cooldown via upstream hint"),
queue: [buildError(429, "rate limited, retry after 1 second", { "Retry-After": "1" })],
});
relay.configureToken(TOKENS.p4, {
defaultResponse: buildCompletion("ignored upstream retry hint"),
queue: [buildError(429, "rate limited, retry after 5 seconds", { "Retry-After": "5" })],
});
relay.configureToken(TOKENS.p5, {
defaultResponse: buildCompletion("breaker target recovered"),
queue: [buildError(503, "breaker failure #1"), buildError(503, "breaker failure #2")],
});
relay.configureToken(TOKENS.p6, {
defaultResponse: buildCompletion("round robin A"),
});
relay.configureToken(TOKENS.p7, {
defaultResponse: buildCompletion("round robin B"),
});
relay.configureToken(TOKENS.p8, {
defaultResponse: {
...buildCompletion("queued connection request"),
delayMs: 250,
},
});
await seedCompatibleProvider("p1", TOKENS.p1, fakeBaseUrl);
await seedCompatibleProvider("p2", TOKENS.p2, fakeBaseUrl);
await seedCompatibleProvider("p3", TOKENS.p3, fakeBaseUrl);
await seedCompatibleProvider("p4", TOKENS.p4, fakeBaseUrl);
await seedCompatibleProvider("p5", TOKENS.p5, fakeBaseUrl);
await seedCompatibleProvider("p6", TOKENS.p6, fakeBaseUrl);
await seedCompatibleProvider("p7", TOKENS.p7, fakeBaseUrl);
await seedCompatibleProvider("p8", TOKENS.p8, fakeBaseUrl);
await combosDb.createCombo({
name: "res-priority-fallback",
strategy: "priority",
config: { maxRetries: 0, retryDelayMs: 0 },
models: ["p1/test-model", "p2/test-model"],
});
await combosDb.createCombo({
name: "res-breaker-combo",
strategy: "priority",
config: { maxRetries: 0, retryDelayMs: 0 },
models: ["p5/test-model", "p2/test-model"],
});
await combosDb.createCombo({
name: "res-rr",
strategy: "round-robin",
config: { maxRetries: 0, retryDelayMs: 0, concurrencyPerModel: 1, queueTimeoutMs: 800 },
models: ["p6/test-model", "p7/test-model"],
});
await settingsDb.updateSettings({
resilienceSettings: buildResilienceConfig(),
requestRetry: 0,
maxRetryIntervalSec: 0,
requireLogin: false,
setupComplete: true,
});
core.closeDbInstance();
app = createServerProcess(TEST_DATA_DIR, DASHBOARD_PORT);
await waitForServer(app.baseUrl, app);
await patchResilience(app.baseUrl, buildResilienceConfig());
const warmup = await postChat(app.baseUrl, "p2/test-model", "warm up chat route");
assert.equal(warmup.response.status, 200, JSON.stringify(warmup.json));
relay.resetState(TOKENS.p2);
});
test.after(async () => {
if (app) {
await stopProcess(app.child);
}
await relay.stop();
core.closeDbInstance();
await fsp.rm(TEST_DATA_DIR, { recursive: true, force: true });
});
test("resilience API only exposes configuration, not runtime breaker state", async () => {
assert.ok(app);
const { response, json } = await getJson(`${app.baseUrl}/api/resilience`);
assert.equal(response.status, 200);
assert.deepEqual(Object.keys(json).sort(), [
"connectionCooldown",
"legacy",
"providerBreaker",
"requestQueue",
"waitForCooldown",
]);
assert.equal("providerBreakers" in json, false);
assert.equal("runtime" in json, false);
});
test("request queue serializes concurrent requests on the same connection", async () => {
assert.ok(app);
await patchResilience(
app.baseUrl,
buildResilienceConfig({
requestQueue: {
concurrentRequests: 1,
maxWaitMs: 1_500,
},
})
);
relay.resetState(TOKENS.p8);
const startedAt = Date.now();
const [first, second] = await Promise.all([
postChat(app.baseUrl, "p8/test-model", "queue-one"),
postChat(app.baseUrl, "p8/test-model", "queue-two"),
]);
const elapsed = Date.now() - startedAt;
const state = relay.getState(TOKENS.p8);
assert.equal(first.response.status, 200, JSON.stringify(first.json));
assert.equal(second.response.status, 200, JSON.stringify(second.json));
assert.equal(state.hits, 2);
assert.ok(elapsed >= 450, `expected queued elapsed >= 450ms, got ${elapsed}ms`);
assert.ok(
state.startedAt[1] - state.startedAt[0] >= 180,
`expected second request to be delayed by queue, got ${state.startedAt[1] - state.startedAt[0]}ms`
);
});
test("priority combo falls back on 503 and skips the cooled-down primary on the next request", async () => {
assert.ok(app);
await patchResilience(app.baseUrl, buildResilienceConfig());
relay.resetState(TOKENS.p1, [buildError(503, "primary transient failure")]);
relay.resetState(TOKENS.p2);
const first = await postChat(app.baseUrl, "res-priority-fallback", "priority fallback request");
assert.equal(first.response.status, 200, JSON.stringify(first.json));
assert.equal(first.json.choices[0].message.content, "secondary stable");
assert.equal(relay.getState(TOKENS.p1).hits, 1);
assert.equal(relay.getState(TOKENS.p2).hits, 1);
const second = await postChat(app.baseUrl, "res-priority-fallback", "priority fallback again");
assert.equal(second.response.status, 200, JSON.stringify(second.json));
assert.equal(second.json.choices[0].message.content, "secondary stable");
assert.equal(relay.getState(TOKENS.p1).hits, 1);
assert.equal(relay.getState(TOKENS.p2).hits, 2);
});
test("wait-for-cooldown honors upstream Retry-After when enabled", async () => {
assert.ok(app);
await patchResilience(
app.baseUrl,
buildResilienceConfig({
connectionCooldown: {
apikey: {
useUpstreamRetryHints: true,
baseCooldownMs: 200,
},
},
waitForCooldown: {
enabled: true,
maxRetries: 1,
maxRetryWaitSec: 2,
},
})
);
relay.resetState(TOKENS.p3);
const warmup = await postChat(app.baseUrl, "p3/test-model", "warm provider-specific route");
assert.equal(warmup.response.status, 200, JSON.stringify(warmup.json));
relay.resetState(TOKENS.p3, [
buildError(429, "rate limited, retry after 1 second", { "Retry-After": "1" }),
]);
const startedAt = Date.now();
const result = await postChat(app.baseUrl, "p3/test-model", "wait for cooldown via upstream");
const elapsed = Date.now() - startedAt;
assert.equal(result.response.status, 200, JSON.stringify(result.json));
assert.equal(result.json.choices[0].message.content, "wait-for-cooldown via upstream hint");
assert.equal(relay.getState(TOKENS.p3).hits, 2);
assert.ok(elapsed >= 800, `expected upstream wait >= 800ms, got ${elapsed}ms`);
});
test("connection cooldown can ignore upstream Retry-After and use the configured local cooldown", async () => {
assert.ok(app);
await patchResilience(
app.baseUrl,
buildResilienceConfig({
connectionCooldown: {
apikey: {
useUpstreamRetryHints: false,
baseCooldownMs: 200,
},
},
waitForCooldown: {
enabled: true,
maxRetries: 1,
maxRetryWaitSec: 2,
},
})
);
relay.resetState(TOKENS.p4);
const warmup = await postChat(app.baseUrl, "p4/test-model", "warm provider-specific route");
assert.equal(warmup.response.status, 200, JSON.stringify(warmup.json));
relay.resetState(TOKENS.p4, [
buildError(429, "rate limited, retry after 30 seconds", { "Retry-After": "30" }),
]);
const startedAt = Date.now();
const result = await postChat(app.baseUrl, "p4/test-model", "ignore upstream retry-after");
const elapsed = Date.now() - startedAt;
assert.equal(result.response.status, 200, JSON.stringify(result.json));
assert.equal(result.json.choices[0].message.content, "ignored upstream retry hint");
assert.equal(relay.getState(TOKENS.p4).hits, 2);
assert.ok(
elapsed < 5_000,
`expected ignored upstream hint to avoid a 30s wait, got ${elapsed}ms`
);
});
test("provider circuit breaker opens after repeated final failures and Health reports it", async () => {
assert.ok(app);
await patchResilience(
app.baseUrl,
buildResilienceConfig({
waitForCooldown: {
enabled: false,
maxRetries: 0,
maxRetryWaitSec: 0,
},
providerBreaker: {
apikey: {
failureThreshold: 2,
resetTimeoutMs: 1_500,
},
},
})
);
relay.resetState(TOKENS.p5, [
buildError(503, "breaker failure #1"),
buildError(503, "breaker failure #2"),
]);
const first = await postChat(app.baseUrl, "p5/test-model", "breaker first failure");
const second = await postChat(app.baseUrl, "p5/test-model", "breaker second failure");
const third = await postChat(app.baseUrl, "p5/test-model", "breaker should now be open");
assert.equal(first.response.status, 503);
assert.equal(second.response.status, 503);
assert.equal(third.response.status, 503);
assert.match(String(second.json.error?.message || ""), /reset after/i);
assert.match(String(third.json.error?.message || ""), /circuit breaker is open/i);
assert.equal(relay.getState(TOKENS.p5).hits, 1);
const health = await getJson(`${app.baseUrl}/api/monitoring/health`);
assert.equal(health.response.status, 200);
const breakerEntry = (health.json.providerBreakers || []).find(
(entry: Record<string, unknown>) => entry.provider === "openai-compatible-chat-e2e-p5"
);
assert.ok(breakerEntry, "expected provider breaker entry for p5");
assert.equal(breakerEntry.state, "OPEN");
assert.ok(Number(breakerEntry.failureCount) >= 2);
assert.ok(Number(breakerEntry.retryAfterMs) > 0);
assert.ok(
(health.json.providerBreakers || []).every(
(entry: Record<string, unknown>) => !String(entry.provider || "").includes(":")
)
);
});
test("combo respects the global provider breaker and falls through without a combo-local breaker", async () => {
assert.ok(app);
relay.resetState(TOKENS.p2);
const result = await postChat(
app.baseUrl,
"res-breaker-combo",
"combo should skip broken provider"
);
assert.equal(result.response.status, 200, JSON.stringify(result.json));
assert.equal(result.json.choices[0].message.content, "secondary stable");
assert.equal(relay.getState(TOKENS.p5).hits, 1);
assert.equal(relay.getState(TOKENS.p2).hits, 1);
});
test("round-robin combo still alternates healthy providers after combo breaker removal", async () => {
assert.ok(app);
await patchResilience(app.baseUrl, buildResilienceConfig());
relay.resetState(TOKENS.p6);
relay.resetState(TOKENS.p7);
const first = await postChat(app.baseUrl, "res-rr", "round robin one");
const second = await postChat(app.baseUrl, "res-rr", "round robin two");
assert.equal(first.response.status, 200, JSON.stringify(first.json));
assert.equal(second.response.status, 200, JSON.stringify(second.json));
assert.equal(first.json.choices[0].message.content, "round robin A");
assert.equal(second.json.choices[0].message.content, "round robin B");
assert.equal(relay.getState(TOKENS.p6).hits, 1);
assert.equal(relay.getState(TOKENS.p7).hits, 1);
});

1
tests/integration/security-hardening.test.ts
View File

@@ -232,7 +232,6 @@ test("T06 route payload validation uses validateBody in critical endpoints", ()
"src/app/api/policies/route.ts",
"src/app/api/fallback/chains/route.ts",
"src/app/api/models/route.ts",
"src/app/api/models/availability/route.ts",
"src/app/api/provider-models/route.ts",
"src/app/api/pricing/route.ts",
"src/app/api/rate-limits/route.ts",

107
tests/unit/account-fallback-service.test.ts
View File

@@ -207,8 +207,37 @@ test("lockModelIfPerModelQuota only locks supported providers and real models",
});
test("getProviderProfile differentiates oauth and api-key providers", () => {
assert.deepEqual(getProviderProfile("claude"), PROVIDER_PROFILES.oauth);
assert.deepEqual(getProviderProfile("openai"), PROVIDER_PROFILES.apikey);
const oauthProfile = getProviderProfile("claude");
assert.equal(oauthProfile.transientCooldown, PROVIDER_PROFILES.oauth.transientCooldown);
assert.equal(
oauthProfile.rateLimitCooldown,
oauthProfile.useUpstreamRetryHints ? 0 : oauthProfile.baseCooldownMs
);
assert.equal(oauthProfile.maxBackoffLevel, PROVIDER_PROFILES.oauth.maxBackoffLevel);
assert.equal(
oauthProfile.circuitBreakerThreshold,
PROVIDER_PROFILES.oauth.circuitBreakerThreshold
);
assert.equal(oauthProfile.circuitBreakerReset, PROVIDER_PROFILES.oauth.circuitBreakerReset);
assert.equal(oauthProfile.baseCooldownMs, PROVIDER_PROFILES.oauth.transientCooldown);
assert.equal(oauthProfile.failureThreshold, PROVIDER_PROFILES.oauth.circuitBreakerThreshold);
assert.equal(oauthProfile.resetTimeoutMs, PROVIDER_PROFILES.oauth.circuitBreakerReset);
const apiKeyProfile = getProviderProfile("openai");
assert.equal(apiKeyProfile.transientCooldown, PROVIDER_PROFILES.apikey.transientCooldown);
assert.equal(
apiKeyProfile.rateLimitCooldown,
apiKeyProfile.useUpstreamRetryHints ? 0 : apiKeyProfile.baseCooldownMs
);
assert.equal(apiKeyProfile.maxBackoffLevel, PROVIDER_PROFILES.apikey.maxBackoffLevel);
assert.equal(
apiKeyProfile.circuitBreakerThreshold,
PROVIDER_PROFILES.apikey.circuitBreakerThreshold
);
assert.equal(apiKeyProfile.circuitBreakerReset, PROVIDER_PROFILES.apikey.circuitBreakerReset);
assert.equal(apiKeyProfile.baseCooldownMs, PROVIDER_PROFILES.apikey.transientCooldown);
assert.equal(apiKeyProfile.failureThreshold, PROVIDER_PROFILES.apikey.circuitBreakerThreshold);
assert.equal(apiKeyProfile.resetTimeoutMs, PROVIDER_PROFILES.apikey.circuitBreakerReset);
});
test("shouldMarkAccountExhaustedFrom429 skips connection poisoning for compatible providers", () => {
@@ -229,11 +258,11 @@ test("recordModelLockoutFailure uses provider profile cooldowns, backoff, and re
const compatibleProvider = "openai-compatible-custom-node";
const compatibleModel = "custom-model-a";
const profile = {
transientCooldown: 250,
rateLimitCooldown: 125,
maxBackoffLevel: 2,
circuitBreakerThreshold: 60,
circuitBreakerReset: 500,
baseCooldownMs: 125,
useUpstreamRetryHints: false,
maxBackoffSteps: 2,
failureThreshold: 60,
resetTimeoutMs: 500,
};
const first = recordModelLockoutFailure(
@@ -298,8 +327,8 @@ test("recordModelLockoutFailure uses provider profile cooldowns, backoff, and re
});
// Provider-level failure circuit breaker tests
test("isProviderFailureCode correctly identifies transient error codes", () => {
assert.equal(isProviderFailureCode(429), true);
test("isProviderFailureCode correctly identifies provider-wide transient error codes", () => {
assert.equal(isProviderFailureCode(429), false);
assert.equal(isProviderFailureCode(408), true);
assert.equal(isProviderFailureCode(500), true);
assert.equal(isProviderFailureCode(502), true);
@@ -360,62 +389,16 @@ test("recordProviderFailure tracks failures and triggers cooldown after threshol
}
});
test("recordProviderFailure resets counter after failure window expires", () => {
const originalNow = Date.now;
let now = 1_700_000_000_000;
Date.now = () => now;
test("checkFallbackError no longer mutates provider breaker state on per-connection failures", () => {
const provider = "test-provider-check";
clearProviderFailure(provider);
try {
const provider = "test-provider-window";
clearProviderFailure(provider);
// Record 3 failures
for (let i = 0; i < 3; i++) {
recordProviderFailure(provider);
now += 1000;
}
assert.equal(isProviderInCooldown(provider), false);
// Wait for failure window to expire (20 minutes + 1 second)
now += 20 * 60 * 1000 + 1000;
// Next failure should reset counter, not trigger cooldown
recordProviderFailure(provider);
assert.equal(isProviderInCooldown(provider), false);
// Need 4 more failures to trigger cooldown
for (let i = 0; i < 4; i++) {
recordProviderFailure(provider);
now += 1000;
}
assert.equal(isProviderInCooldown(provider), true);
} finally {
Date.now = originalNow;
clearProviderFailure("test-provider-window");
for (let i = 0; i < 5; i++) {
checkFallbackError(429, "rate limited", 0, null, provider);
}
});
test("checkFallbackError records provider failure for transient errors", () => {
const originalNow = Date.now;
let now = 1_700_000_000_000;
Date.now = () => now;
try {
const provider = "test-provider-check";
clearProviderFailure(provider);
// Simulate 5 transient errors through checkFallbackError
for (let i = 0; i < 5; i++) {
checkFallbackError(429, "rate limited", 0, null, provider);
now += 1000;
}
// Provider should now be in cooldown
assert.equal(isProviderInCooldown(provider), true);
} finally {
Date.now = originalNow;
clearProviderFailure("test-provider-check");
}
assert.equal(isProviderInCooldown(provider), false);
clearProviderFailure(provider);
});
test("checkFallbackError does not record provider failure for non-transient errors", () => {

126
tests/unit/auth-terminal-status.test.ts
View File

@@ -109,3 +109,129 @@ test("markAccountUnavailable does not overwrite terminal status", async () => {
const after = await providersDb.getProviderConnectionById(conn.id);
assert.equal(after.testStatus, "credits_exhausted");
});
test("markAccountUnavailable marks 401 connections as expired without adding cooldown", async () => {
await resetStorage();
const conn = await providersDb.createProviderConnection({
provider: "openai",
authType: "apikey",
apiKey: "sk-expired",
isActive: true,
testStatus: "active",
});
const result = await auth.markAccountUnavailable(
conn.id,
401,
"unauthorized",
"openai",
"gpt-4.1"
);
const after = await providersDb.getProviderConnectionById(conn.id);
assert.equal(result.shouldFallback, true);
assert.equal(result.cooldownMs, 0);
assert.equal(after.testStatus, "expired");
assert.ok(!after.rateLimitedUntil);
});
test("markAccountUnavailable marks 402 connections as credits_exhausted without adding cooldown", async () => {
await resetStorage();
const conn = await providersDb.createProviderConnection({
provider: "openai",
authType: "apikey",
apiKey: "sk-credits",
isActive: true,
testStatus: "active",
});
const result = await auth.markAccountUnavailable(
conn.id,
402,
"payment required",
"openai",
"gpt-4.1"
);
const after = await providersDb.getProviderConnectionById(conn.id);
assert.equal(result.shouldFallback, true);
assert.equal(result.cooldownMs, 0);
assert.equal(after.testStatus, "credits_exhausted");
assert.ok(!after.rateLimitedUntil);
});
test("markAccountUnavailable marks 403 connections as banned without adding cooldown", async () => {
await resetStorage();
const conn = await providersDb.createProviderConnection({
provider: "openai",
authType: "apikey",
apiKey: "sk-banned",
isActive: true,
testStatus: "active",
});
const result = await auth.markAccountUnavailable(conn.id, 403, "forbidden", "openai", "gpt-4.1");
const after = await providersDb.getProviderConnectionById(conn.id);
assert.equal(result.shouldFallback, true);
assert.equal(result.cooldownMs, 0);
assert.equal(after.testStatus, "banned");
assert.ok(!after.rateLimitedUntil);
});
test("markAccountUnavailable keeps project-route 403 errors non-terminal", async () => {
await resetStorage();
const conn = await providersDb.createProviderConnection({
provider: "openai",
authType: "apikey",
apiKey: "sk-project-route",
isActive: true,
testStatus: "active",
});
const result = await auth.markAccountUnavailable(
conn.id,
403,
"The service has not been used in project",
"openai",
"gpt-4.1"
);
const after = await providersDb.getProviderConnectionById(conn.id);
assert.equal(result.shouldFallback, true);
assert.equal(result.cooldownMs, 0);
assert.equal(after.testStatus, "active");
assert.equal(after.lastErrorType, "project_route_error");
assert.ok(!after.rateLimitedUntil);
});
test("markAccountUnavailable keeps oauth-invalid 401 errors non-terminal", async () => {
await resetStorage();
const conn = await providersDb.createProviderConnection({
provider: "openai",
authType: "apikey",
apiKey: "sk-oauth-invalid",
isActive: true,
testStatus: "active",
});
const result = await auth.markAccountUnavailable(
conn.id,
401,
"Invalid authentication credentials provided",
"openai",
"gpt-4.1"
);
const after = await providersDb.getProviderConnectionById(conn.id);
assert.equal(result.shouldFallback, true);
assert.equal(result.cooldownMs, 0);
assert.equal(after.testStatus, "active");
assert.equal(after.lastErrorType, "oauth_invalid_token");
assert.ok(!after.rateLimitedUntil);
});

29
tests/unit/autocombo-unification.test.ts
View File

@@ -128,29 +128,40 @@ test("sidebar visibility excludes the removed auto-combo item", async () => {
assert.deepEqual(sidebarVisibility.normalizeHiddenSidebarItems(["auto-combo", "home"]), ["home"]);
});
test("intelligent routing helpers normalize config and health state", () => {
test("intelligent routing helpers normalize config and build provider scores", () => {
const normalizedConfig = intelligentRouting.normalizeIntelligentRoutingConfig({
candidatePool: ["openai", "anthropic"],
explorationRate: "0.25",
modePack: "",
routerStrategy: "",
weights: { quota: 0.4 },
});
assert.deepEqual(normalizedConfig.candidatePool, ["openai", "anthropic"]);
assert.equal(normalizedConfig.explorationRate, 0.25);
assert.equal(normalizedConfig.modePack, "ship-fast");
assert.equal(normalizedConfig.routerStrategy, "rules");
assert.equal(normalizedConfig.weights.quota, 0.4);
assert.equal(
normalizedConfig.weights.health,
intelligentRouting.DEFAULT_INTELLIGENT_WEIGHTS.health
);
const healthState = intelligentRouting.extractIntelligentHealthState({
circuitBreakers: [
{ provider: "openai", state: "OPEN", lastFailure: "2026-04-12T12:00:00Z" },
{ provider: "anthropic", state: "OPEN", lastFailure: "2026-04-12T12:01:00Z" },
{ provider: "google", state: "CLOSED" },
],
const providerScores = intelligentRouting.buildIntelligentProviderScores({
config: normalizedConfig,
});
assert.equal(healthState.incidentMode, true);
assert.equal(healthState.exclusions.length, 2);
assert.equal(providerScores.length, 2);
assert.deepEqual(
providerScores.map((entry) => ({
provider: entry.provider,
model: entry.model,
score: entry.score,
quotaWeight: entry.factors.quota,
})),
[
{ provider: "openai", model: "auto", score: 0.5, quotaWeight: 0.4 },
{ provider: "anthropic", model: "auto", score: 0.5, quotaWeight: 0.4 },
]
);
});

155
tests/unit/batch-a-domain.test.ts
View File

@@ -1,166 +1,13 @@
/**
* Batch A — Domain Layer + Infrastructure Tests
*
* Tests for: modelAvailability, costRules, fallbackPolicy,
* Tests for: costRules, fallbackPolicy,
* errorCodes, requestId, fetchTimeout
*/
import { describe, it, before, after } from "node:test";
import assert from "node:assert/strict";
// ──────────────── T-19: Model Availability ────────────────
import {
isModelAvailable,
setModelUnavailable,
markModelAsProblematic,
clearModelUnavailability,
getAvailabilityReport,
getUnavailableCount,
resetAllAvailability,
} from "../../src/domain/modelAvailability.ts";
describe("modelAvailability", () => {
before(() => resetAllAvailability());
after(() => resetAllAvailability());
it("should report model as available by default", () => {
assert.equal(isModelAvailable("openai", "gpt-4o"), true);
});
it("should mark model as unavailable", () => {
setModelUnavailable("openai", "gpt-4o", 60000, "rate limited");
assert.equal(isModelAvailable("openai", "gpt-4o"), false);
});
it("should report unavailable models", () => {
const report = getAvailabilityReport();
assert.equal(report.length, 1);
assert.equal(report[0].provider, "openai");
assert.equal(report[0].model, "gpt-4o");
assert.equal(report[0].reason, "rate limited");
assert.ok(report[0].remainingMs > 0);
});
it("should count unavailable models", () => {
assert.equal(getUnavailableCount(), 1);
});
it("should clear model unavailability", () => {
clearModelUnavailability("openai", "gpt-4o");
assert.equal(isModelAvailable("openai", "gpt-4o"), true);
assert.equal(getUnavailableCount(), 0);
});
it("should auto-expire after cooldown", () => {
setModelUnavailable("anthropic", "claude-sonnet-4-20250514", 1, "test");
// Wait 2ms for expiry
const start = Date.now();
while (Date.now() - start < 5) {} // spin wait
assert.equal(isModelAvailable("anthropic", "claude-sonnet-4-20250514"), true);
});
it("should apply adaptive quarantine for repeated problematic failures", () => {
const first = markModelAsProblematic("nvidia", "z-ai/glm4.7", {
status: 502,
baseCooldownMs: 3000,
reason: "HTTP 502",
});
const second = markModelAsProblematic("nvidia", "z-ai/glm4.7", {
status: 502,
baseCooldownMs: 3000,
reason: "HTTP 502",
});
assert.equal(first.failureCount, 1);
assert.equal(first.cooldownMs, 120000);
assert.equal(second.failureCount, 2);
assert.equal(second.cooldownMs, 240000);
const report = getAvailabilityReport();
const entry = report.find((r) => r.provider === "nvidia" && r.model === "z-ai/glm4.7");
assert.ok(entry, "nvidia model should be quarantined");
assert.ok(entry.remainingMs >= 200000, "cooldown should be preserved/escalated");
});
it("should reset failure history after model recovery", () => {
clearModelUnavailability("nvidia", "z-ai/glm4.7");
const afterReset = markModelAsProblematic("nvidia", "z-ai/glm4.7", {
status: 502,
baseCooldownMs: 3000,
reason: "HTTP 502",
});
assert.equal(afterReset.failureCount, 1);
assert.equal(afterReset.cooldownMs, 120000);
});
it("should use provider profile threshold, cooldowns, and reset window for global quarantine", () => {
const originalNow = Date.now;
let now = 10_000;
Date.now = () => now;
try {
const profile = {
transientCooldown: 200,
rateLimitCooldown: 100,
maxBackoffLevel: 2,
circuitBreakerThreshold: 3,
circuitBreakerReset: 500,
};
const first = markModelAsProblematic("openai", "gpt-4o-mini", {
status: 429,
reason: "HTTP 429",
profile,
});
const second = markModelAsProblematic("openai", "gpt-4o-mini", {
status: 429,
reason: "HTTP 429",
profile,
});
assert.equal(first.failureCount, 1);
assert.equal(first.cooldownMs, 100);
assert.equal(first.quarantined, false);
assert.equal(isModelAvailable("openai", "gpt-4o-mini"), true);
assert.equal(second.failureCount, 2);
assert.equal(second.cooldownMs, 200);
assert.equal(second.quarantined, false);
assert.equal(isModelAvailable("openai", "gpt-4o-mini"), true);
now += 10;
const third = markModelAsProblematic("openai", "gpt-4o-mini", {
status: 429,
reason: "HTTP 429",
profile,
});
assert.equal(third.failureCount, 3);
assert.equal(third.cooldownMs, 400);
assert.equal(third.quarantined, true);
assert.equal(isModelAvailable("openai", "gpt-4o-mini"), false);
now += 600;
assert.equal(isModelAvailable("openai", "gpt-4o-mini"), true);
const afterWindow = markModelAsProblematic("openai", "gpt-4o-mini", {
status: 429,
reason: "HTTP 429",
profile,
});
assert.equal(afterWindow.failureCount, 1);
assert.equal(afterWindow.cooldownMs, 100);
assert.equal(afterWindow.quarantined, false);
} finally {
Date.now = originalNow;
clearModelUnavailability("openai", "gpt-4o-mini");
}
});
});
// ──────────────── T-19: Cost Rules ────────────────
import {

17
tests/unit/chat-combo-live-test.test.ts
View File

@@ -12,8 +12,6 @@ const providersDb = await import("../../src/lib/db/providers.ts");
const chatRoute = await import("../../src/app/api/v1/chat/completions/route.ts");
const { generateSignature, invalidateBySignature, setCachedResponse } =
await import("../../src/lib/semanticCache.ts");
const { clearModelUnavailability, resetAllAvailability, setModelUnavailable } =
await import("../../src/domain/modelAvailability.ts");
const { getCircuitBreaker, resetAllCircuitBreakers, STATE } =
await import("../../src/shared/utils/circuitBreaker.ts");
@@ -28,7 +26,6 @@ async function resetStorage() {
core.resetDbInstance();
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
fs.mkdirSync(TEST_DATA_DIR, { recursive: true });
resetAllAvailability();
resetAllCircuitBreakers();
}
@@ -79,27 +76,20 @@ test.beforeEach(async () => {
test.afterEach(async () => {
await flushBackgroundWork();
globalThis.fetch = originalFetch;
resetAllAvailability();
resetAllCircuitBreakers();
});
test.after(async () => {
await flushBackgroundWork();
globalThis.fetch = originalFetch;
resetAllAvailability();
resetAllCircuitBreakers();
core.resetDbInstance();
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
});
test("combo live test bypasses local cooldown and breaker state to perform a real upstream request", async () => {
test("combo live test bypasses connection cooldown and breaker state to perform a real upstream request", async () => {
const created = await seedSuppressedConnection();
setModelUnavailable("openai", "gpt-4o-mini", 60_000, "test cooldown");
const breaker = getCircuitBreaker("openai");
breaker.state = STATE.OPEN;
breaker.lastFailureTime = Date.now();
const fetchCalls = [];
globalThis.fetch = async (url, init = {}) => {
fetchCalls.push({ url: String(url), init });
@@ -120,7 +110,10 @@ test("combo live test bypasses local cooldown and breaker state to perform a rea
assert.equal(blockedByCooldown.status, 503);
assert.equal(fetchCalls.length, 0);
clearModelUnavailability("openai", "gpt-4o-mini");
const breaker = getCircuitBreaker("openai");
breaker.state = STATE.OPEN;
breaker.lastFailureTime = Date.now();
breaker.resetTimeout = 60_000;
const blockedByBreaker = await chatRoute.POST(makeRequest());
assert.equal(blockedByBreaker.status, 503);

119
tests/unit/chat-helpers.test.ts
View File

@@ -18,13 +18,10 @@ const {
safeLogEvents,
withSessionHeader,
} = await import("../../src/sse/handlers/chatHelpers.ts");
const { getModelCooldownInfo, setModelUnavailable, resetAllAvailability } =
await import("../../src/domain/modelAvailability.ts");
const { getCircuitBreaker, resetAllCircuitBreakers, CircuitBreakerOpenError, STATE } =
const { getCircuitBreaker, resetAllCircuitBreakers, STATE } =
await import("../../src/shared/utils/circuitBreaker.ts");
async function resetStorage() {
resetAllAvailability();
resetAllCircuitBreakers();
core.resetDbInstance();
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
@@ -92,22 +89,6 @@ test("resolveModelOrError rejects malformed model strings", async () => {
assert.match(json.error.message, /Invalid model format/i);
});
test("checkPipelineGates blocks models in cooldown", async () => {
setModelUnavailable("openai", "gpt-4o-mini", 12_000, "cooldown");
const response = await checkPipelineGates("openai", "gpt-4o-mini");
const json = await response.json();
const cooldownInfo = getModelCooldownInfo("openai", "gpt-4o-mini");
const retryAfter = Number(response.headers.get("Retry-After"));
assert.equal(response.status, 503);
assert.ok(cooldownInfo);
assert.ok(retryAfter >= 1);
assert.ok(retryAfter <= 12);
assert.ok(retryAfter >= Math.ceil((cooldownInfo?.remainingMs || 0) / 1000) - 1);
assert.match(json.error.message, /temporarily unavailable/i);
});
test("checkPipelineGates blocks providers with an open circuit breaker", async () => {
const breaker = getCircuitBreaker("openai");
breaker.state = STATE.OPEN;
@@ -116,8 +97,8 @@ test("checkPipelineGates blocks providers with an open circuit breaker", async (
const response = await checkPipelineGates("openai", "gpt-4o-mini", {
providerProfile: {
circuitBreakerThreshold: 5,
circuitBreakerReset: 5_000,
failureThreshold: 5,
resetTimeoutMs: 5_000,
},
});
const json = await response.json();
@@ -127,6 +108,8 @@ test("checkPipelineGates blocks providers with an open circuit breaker", async (
assert.ok(retryAfter >= 4);
assert.ok(retryAfter <= 5);
assert.match(json.error.message, /circuit breaker is open/i);
assert.equal(json.error.code, "provider_circuit_open");
assert.equal(response.headers.get("X-OmniRoute-Provider-Breaker"), "open");
});
test("checkPipelineGates reapplies runtime breaker settings to existing breakers", async () => {
@@ -139,8 +122,8 @@ test("checkPipelineGates reapplies runtime breaker settings to existing breakers
const response = await checkPipelineGates("openai", "gpt-4o-mini", {
providerProfile: {
circuitBreakerThreshold: 60,
circuitBreakerReset: 5_000,
failureThreshold: 60,
resetTimeoutMs: 5_000,
},
});
@@ -233,60 +216,44 @@ test("safeResolveProxy returns the direct route when no proxy config is present"
});
});
test("executeChatWithBreaker converts circuit-open and proxy-fast-fail errors", async () => {
const credentials = { connectionId: "conn_helper" };
const openResult = await executeChatWithBreaker({
bypassCircuitBreaker: false,
breaker: {
execute: async () => {
throw new CircuitBreakerOpenError("already open", "openai", 5_000);
},
},
body: { model: "openai/gpt-4o-mini" },
provider: "openai",
model: "gpt-4o-mini",
refreshedCredentials: credentials,
proxyInfo: null,
log: console,
clientRawRequest: null,
credentials,
apiKeyInfo: null,
userAgent: "",
comboName: null,
comboStrategy: null,
isCombo: false,
extendedContext: false,
});
test("executeChatWithBreaker converts proxy fast-fail errors", async () => {
const originalFetch = globalThis.fetch;
globalThis.fetch = async () => {
const error = new Error("Proxy unreachable");
(error as Error & { code?: string }).code = "PROXY_UNREACHABLE";
throw error;
};
const proxyResult = await executeChatWithBreaker({
bypassCircuitBreaker: false,
breaker: {
execute: async () => {
const error = new Error("Proxy unreachable");
error.code = "PROXY_UNREACHABLE";
throw error;
},
},
body: { model: "openai/gpt-4o-mini" },
provider: "openai",
model: "gpt-4o-mini",
refreshedCredentials: credentials,
proxyInfo: null,
log: console,
clientRawRequest: null,
credentials,
apiKeyInfo: null,
userAgent: "",
comboName: null,
comboStrategy: null,
isCombo: false,
extendedContext: false,
});
try {
const credentials = {
connectionId: "conn_helper",
apiKey: "sk-openai-helper",
providerSpecificData: {},
};
const proxyResult = await executeChatWithBreaker({
body: { model: "openai/gpt-4o-mini" },
provider: "openai",
model: "gpt-4o-mini",
refreshedCredentials: credentials,
proxyInfo: null,
log: console,
clientRawRequest: null,
credentials,
apiKeyInfo: null,
userAgent: "",
comboName: null,
comboStrategy: null,
isCombo: false,
extendedContext: false,
comboStepId: null,
comboExecutionKey: null,
});
assert.equal(openResult.result.status, 503);
assert.equal(openResult.result.response.status, 503);
assert.equal(proxyResult.result.status, 503);
assert.equal(proxyResult.result.error, "Proxy unreachable");
assert.equal(proxyResult.result.status, 502);
assert.match(String(proxyResult.result.error || ""), /Proxy unreachable/);
} finally {
globalThis.fetch = originalFetch;
}
});
test("safeLogEvents tolerates success and timeout payloads", () => {

20
tests/unit/chat-route-coverage.test.ts
View File

@@ -14,13 +14,11 @@ const {
resetStorage,
seedApiKey,
seedConnection,
setModelUnavailable,
settingsDb,
toPlainHeaders,
} = harness;
const { getCircuitBreaker, STATE } = await import("../../src/shared/utils/circuitBreaker.ts");
const { clearModelUnavailability } = await import("../../src/domain/modelAvailability.ts");
const { clearProviderFailure } = await import("../../open-sse/services/accountFallback.ts");
const { getDefaultTaskModelMap, resetTaskRoutingStats, setTaskRoutingConfig } =
await import("../../open-sse/services/taskAwareRouter.ts");
@@ -345,9 +343,11 @@ test("handleChat returns 400 when no provider credentials exist", async () => {
assert.match(json.error.message, /No credentials for provider: openai/);
});
test("handleChat returns 503 for cooled-down models and open circuit breakers", async () => {
await seedConnection("openai", { apiKey: "sk-openai-breaker" });
setModelUnavailable("openai", "gpt-4o-mini", 60_000, "test cooldown");
test("handleChat returns 503 for cooled-down connections and 503 for open circuit breakers", async () => {
await seedConnection("openai", {
apiKey: "sk-openai-breaker",
rateLimitedUntil: new Date(Date.now() + 60_000).toISOString(),
});
const cooldownResponse = await handleChat(
buildRequest({
@@ -360,15 +360,13 @@ test("handleChat returns 503 for cooled-down models and open circuit breakers",
);
const cooldownJson = await cooldownResponse.json();
assert.equal(cooldownResponse.status, 503);
assert.match(cooldownJson.error.message, /temporarily unavailable/i);
clearModelUnavailability("openai", "gpt-4o-mini");
const freshBreaker = getCircuitBreaker("openai");
freshBreaker.reset();
assert.ok(Number(cooldownResponse.headers.get("Retry-After")) >= 1);
assert.match(cooldownJson.error.message, /\[openai\/gpt-4o-mini\]/i);
const breaker = getCircuitBreaker("openai");
breaker.state = STATE.OPEN;
breaker.lastFailureTime = Date.now();
breaker.resetTimeout = 60_000;
const breakerBlocked = await handleChat(
buildRequest({
@@ -382,6 +380,8 @@ test("handleChat returns 503 for cooled-down models and open circuit breakers",
const breakerJson = await breakerBlocked.json();
assert.equal(breakerBlocked.status, 503);
assert.equal(breakerBlocked.headers.get("X-OmniRoute-Provider-Breaker"), "open");
assert.equal(breakerJson.error.code, "provider_circuit_open");
assert.match(breakerJson.error.message, /circuit breaker is open/i);
});

230
tests/unit/combo-circuit-breaker.test.ts
View File

@@ -1,230 +0,0 @@
import test from "node:test";
import assert from "node:assert/strict";
// Reset circuit breaker registry between tests
const { CircuitBreaker, getCircuitBreaker, getAllCircuitBreakerStatuses, STATE } =
await import("../../src/shared/utils/circuitBreaker.ts");
const { handleComboChat, getComboFromData } = await import("../../open-sse/services/combo.ts");
const { normalizeComboStep } = await import("../../src/lib/combos/steps.ts");
const { PROVIDER_PROFILES } = await import("../../open-sse/config/constants.ts");
// ─── Helpers ────────────────────────────────────────────────────────────────
/** Create a mock logger */
function mockLog() {
const entries = [];
return {
info: (tag, msg) => entries.push({ level: "info", tag, msg }),
warn: (tag, msg) => entries.push({ level: "warn", tag, msg }),
error: (tag, msg) => entries.push({ level: "error", tag, msg }),
entries,
};
}
/** Create a handleSingleModel that returns given status codes in sequence */
function mockHandler(statusSequence) {
let callIndex = 0;
return async (body, modelStr) => {
const status = statusSequence[callIndex] ?? statusSequence[statusSequence.length - 1] ?? 200;
callIndex++;
if (status === 200) {
return new Response(JSON.stringify({ ok: true }), { status: 200 });
}
return new Response(JSON.stringify({ error: { message: `Error ${status}` } }), {
status,
statusText: `Error ${status}`,
});
};
}
function getComboTargetBreakerKey(combo, index = 0) {
const step = normalizeComboStep(combo.models[index], {
comboName: combo.name,
index,
});
return `combo:${combo.name}:${step.id}`;
}
// ─── Circuit Breaker Integration Tests ──────────────────────────────────────
// NOTE: combo.ts uses the full model string (e.g. "combo:groq/llama-3.3-70b")
// as the circuit breaker key, not just the provider prefix.
test("handleComboChat: circuit breaker opens after repeated 502 errors", async () => {
const combo = {
name: "test-combo",
models: [{ model: "groq/llama-3.3-70b", weight: 0 }],
strategy: "priority",
config: { maxRetries: 0 },
};
const breakerKey = getComboTargetBreakerKey(combo);
const breaker = getCircuitBreaker(breakerKey, {
failureThreshold: PROVIDER_PROFILES.apikey.circuitBreakerThreshold,
resetTimeout: 60000,
});
breaker.reset();
const log = mockLog();
// Send requests that all fail with 502 until the API-key profile threshold is reached.
for (let i = 0; i < PROVIDER_PROFILES.apikey.circuitBreakerThreshold; i++) {
await handleComboChat({
body: {},
combo,
handleSingleModel: mockHandler([502]),
isModelAvailable: () => true,
log,
settings: null,
allCombos: null,
});
}
// Breaker should now be OPEN
const status = breaker.getStatus();
console.log("=== BREAKER STATUS AFTER THRESHOLD CALLS ===", status);
assert.equal(
status.state,
STATE.OPEN,
"Breaker should be OPEN after the API-key failure threshold is reached"
);
assert.equal(
status.failureCount,
PROVIDER_PROFILES.apikey.circuitBreakerThreshold,
"Failure count should match the API-key profile threshold"
);
});
test("handleComboChat: skips models with open circuit breaker", async () => {
// Set up: groq breaker is OPEN, fireworks breaker is CLOSED
const combo = {
name: "test-skip-combo",
models: [
{ model: "groq/llama-3.3-70b", weight: 0 },
{ model: "fireworks/deepseek-v3p1", weight: 0 },
],
strategy: "priority",
};
const groqBreakerKey = getComboTargetBreakerKey(combo, 0);
const groqBreaker = getCircuitBreaker(groqBreakerKey, {
failureThreshold: 3,
resetTimeout: 60000,
});
groqBreaker.reset();
// Force open the breaker
groqBreaker._onFailure();
groqBreaker._onFailure();
groqBreaker._onFailure();
assert.equal(groqBreaker.getStatus().state, STATE.OPEN);
const fireworksBreakerKey = getComboTargetBreakerKey(combo, 1);
const fireworksBreaker = getCircuitBreaker(fireworksBreakerKey, {
failureThreshold: 5,
resetTimeout: 30000,
});
fireworksBreaker.reset();
const log = mockLog();
const result = await handleComboChat({
body: {},
combo,
handleSingleModel: mockHandler([200]), // fireworks will succeed
isModelAvailable: () => true,
log,
settings: null,
allCombos: null,
});
assert.equal(result.ok, true, "Should succeed via fireworks fallback");
// Check logs show groq was skipped
const skipLog = log.entries.find(
(e) => e.msg.includes("circuit breaker OPEN") && e.msg.includes("groq")
);
assert.ok(skipLog, "Should log that groq was skipped due to breaker");
});
test("handleComboChat: returns 503 when all breakers are open", async () => {
const combo = {
name: "test-all-open",
models: [
{ model: "groq/llama-3.3-70b", weight: 0 },
{ model: "fireworks/deepseek-v3p1", weight: 0 },
],
strategy: "priority",
};
// Open both breakers explicitly before invoking the combo.
const groqBreaker = getCircuitBreaker(getComboTargetBreakerKey(combo, 0), {
failureThreshold: 3,
resetTimeout: 60000,
});
groqBreaker.reset();
groqBreaker._onFailure();
groqBreaker._onFailure();
groqBreaker._onFailure();
assert.equal(groqBreaker.getStatus().state, STATE.OPEN);
const fireworksBreaker = getCircuitBreaker(getComboTargetBreakerKey(combo, 1), {
failureThreshold: 5,
resetTimeout: 30000,
});
fireworksBreaker.reset();
for (let i = 0; i < 5; i++) fireworksBreaker._onFailure();
assert.equal(fireworksBreaker.getStatus().state, STATE.OPEN);
const log = mockLog();
const result = await handleComboChat({
body: {},
combo,
handleSingleModel: mockHandler([200]), // Won't be called
isModelAvailable: () => true,
log,
settings: null,
allCombos: null,
});
assert.equal(result.status, 503, "Should return 503");
const body = await result.json();
assert.ok(body.error.message.includes("circuit breakers open"), "Should mention breakers");
});
test("handleComboChat: 429 errors also trigger circuit breaker", async () => {
const combo = {
name: "test-429",
models: [{ model: "cerebras/llama-3.3-70b", weight: 0 }],
strategy: "priority",
};
const breakerKey = getComboTargetBreakerKey(combo);
const breaker = getCircuitBreaker(breakerKey, {
failureThreshold: 5,
resetTimeout: 30000,
});
breaker.reset();
const log = mockLog();
// 5 x 429 should open breaker
for (let i = 0; i < 5; i++) {
await handleComboChat({
body: {},
combo,
handleSingleModel: mockHandler([429]),
isModelAvailable: () => true,
log,
settings: null,
allCombos: null,
});
}
assert.equal(breaker.getStatus().state, STATE.OPEN, "429s should open breaker");
});
test("circuit breaker uses provider profile thresholds", () => {
// OAuth providers (e.g. claude) should have lower threshold
assert.equal(PROVIDER_PROFILES.oauth.circuitBreakerThreshold, 3);
// API providers (e.g. groq) should have higher threshold
assert.equal(PROVIDER_PROFILES.apikey.circuitBreakerThreshold, 5);
});

13
tests/unit/combo-config.test.ts
View File

@@ -13,7 +13,9 @@ test("getDefaultComboConfig returns a fresh copy of the defaults", () => {
assert.notEqual(first, second);
assert.equal(first.strategy, "priority");
assert.equal(first.maxRetries, 1);
assert.equal(first.timeoutMs, 600000);
assert.equal(first.retryDelayMs, 2000);
assert.ok(!("timeoutMs" in first));
assert.ok(!("healthCheckEnabled" in first));
assert.equal(first.handoffThreshold, 0.85);
assert.equal(first.maxMessagesForSummary, 30);
assert.deepEqual(first.handoffProviders, ["codex"]);
@@ -27,7 +29,6 @@ test("resolveComboConfig applies the full cascade from defaults to combo overrid
{
config: {
maxRetries: 4,
timeoutMs: 45000,
},
},
{
@@ -47,12 +48,12 @@ test("resolveComboConfig applies the full cascade from defaults to combo overrid
assert.equal(result.strategy, "round-robin");
assert.equal(result.retryDelayMs, 500);
assert.equal(result.timeoutMs, 45000);
assert.equal(result.maxRetries, 4);
assert.equal(result.healthCheckEnabled, true);
assert.ok(!("timeoutMs" in result));
assert.ok(!("healthCheckEnabled" in result));
});
test("resolveComboConfig ignores null and undefined overrides", () => {
test("resolveComboConfig ignores null, undefined, and legacy resilience overrides", () => {
const result = resolveComboConfig(
{
config: {
@@ -75,7 +76,7 @@ test("resolveComboConfig ignores null and undefined overrides", () => {
"openai"
);
assert.equal(result.timeoutMs, 600000);
assert.ok(!("timeoutMs" in result));
assert.equal(result.queueTimeoutMs, 15000);
assert.equal(result.concurrencyPerModel, 9);
assert.equal(result.trackMetrics, false);

43
tests/unit/combo-context-relay.test.ts
View File

@@ -13,12 +13,10 @@ const handoffDb = await import("../../src/lib/db/contextHandoffs.ts");
const { registerCodexConnection } = await import("../../open-sse/services/codexQuotaFetcher.ts");
const { clearSessions, touchSession } = await import("../../open-sse/services/sessionManager.ts");
const { resetAllComboMetrics } = await import("../../open-sse/services/comboMetrics.ts");
const { resetAllCircuitBreakers, getCircuitBreaker } =
await import("../../src/shared/utils/circuitBreaker.ts");
const { resetAllCircuitBreakers } = await import("../../src/shared/utils/circuitBreaker.ts");
const { resetAll: resetAllSemaphores } =
await import("../../open-sse/services/rateLimitSemaphore.ts");
const { _resetAllDecks } = await import("../../src/shared/utils/shuffleDeck.ts");
const { normalizeComboStep } = await import("../../src/lib/combos/steps.ts");
const originalFetch = globalThis.fetch;
@@ -39,6 +37,24 @@ function okResponse(body = { choices: [{ message: { content: "ok" } }] }) {
});
}
function providerBreakerOpenResponse() {
return new Response(
JSON.stringify({
error: {
message: "Provider circuit breaker is open",
code: "provider_circuit_open",
},
}),
{
status: 503,
headers: {
"content-type": "application/json",
"x-omniroute-provider-breaker": "open",
},
}
);
}
function buildQuotaResponse(usedPercent, resetAfterSeconds = 3600) {
return new Response(
JSON.stringify({
@@ -161,23 +177,13 @@ test("handleComboChat context-relay skips unavailable models and falls through t
assert.deepEqual(calls, ["openai/gpt-4o-mini"]);
});
test("handleComboChat context-relay skips models with an open circuit breaker", async () => {
test("handleComboChat context-relay skips targets that report an open provider circuit breaker", async () => {
const combo = {
name: "relay-breaker",
strategy: "context-relay",
models: ["codex/gpt-5.4", "openai/gpt-4o-mini"],
config: { maxRetries: 0 },
};
const firstStep = normalizeComboStep(combo.models[0], {
comboName: combo.name,
index: 0,
});
const breaker = getCircuitBreaker(`combo:${combo.name}:${firstStep.id}`, {
failureThreshold: 1,
resetTimeout: 60000,
});
breaker._onFailure();
const log = createLog();
const calls = [];
@@ -188,6 +194,9 @@ test("handleComboChat context-relay skips models with an open circuit breaker",
combo,
handleSingleModel: async (_body, modelStr) => {
calls.push(modelStr);
if (modelStr === "codex/gpt-5.4") {
return providerBreakerOpenResponse();
}
return okResponse();
},
isModelAvailable: async () => true,
@@ -197,8 +206,10 @@ test("handleComboChat context-relay skips models with an open circuit breaker",
});
assert.equal(result.ok, true);
assert.deepEqual(calls, ["openai/gpt-4o-mini"]);
assert.ok(log.entries.some((entry) => entry.msg.includes("circuit breaker OPEN")));
assert.deepEqual(calls, ["codex/gpt-5.4", "openai/gpt-4o-mini"]);
assert.ok(
log.entries.some((entry) => String(entry.msg).includes("provider circuit breaker OPEN"))
);
});
test("handleComboChat context-relay persists a handoff when codex quota reaches the warning threshold", async () => {

93
tests/unit/combo-routing-engine.test.ts
View File

@@ -24,8 +24,7 @@ const { saveModelsDevCapabilities, clearModelsDevCapabilities } =
await import("../../src/lib/modelsDevSync.ts");
const { getComboMetrics, recordComboRequest, resetAllComboMetrics } =
await import("../../open-sse/services/comboMetrics.ts");
const { getCircuitBreaker, resetAllCircuitBreakers } =
await import("../../src/shared/utils/circuitBreaker.ts");
const { resetAllCircuitBreakers } = await import("../../src/shared/utils/circuitBreaker.ts");
const { acquire: acquireSemaphore, resetAll: resetAllSemaphores } =
await import("../../open-sse/services/rateLimitSemaphore.ts");
const { _resetAllDecks } = await import("../../src/shared/utils/shuffleDeck.ts");
@@ -54,6 +53,24 @@ function errorResponse(status: number, message: string = `Error ${status}`) {
});
}
function providerBreakerOpenResponse() {
return new Response(
JSON.stringify({
error: {
message: "Provider circuit breaker is open",
code: "provider_circuit_open",
},
}),
{
status: 503,
headers: {
"content-type": "application/json",
"x-omniroute-provider-breaker": "open",
},
}
);
}
function streamResponse(chunks: any[]) {
return new Response(chunks.join(""), {
status: 200,
@@ -83,7 +100,7 @@ function capabilityEntry(limitContext: any) {
};
}
function getComboTargetBreakerKey(comboName: string, index: number, stepInput: any) {
function getComboTargetExecutionKey(comboName: string, index: number, stepInput: any) {
const step = normalizeComboStep(stepInput, { comboName, index });
if (!step) throw new Error(`Failed to normalize combo step for ${comboName}#${index}`);
return `combo:${comboName}:${step.id}`;
@@ -1038,7 +1055,7 @@ test("handleComboChat round-robin returns 404 when no models are configured", as
test("handleComboChat round-robin falls through semaphore timeouts and malformed success payloads", async () => {
const release = await acquireSemaphore(
getComboTargetBreakerKey("rr-timeout-fallback", 0, "model-a"),
getComboTargetExecutionKey("rr-timeout-fallback", 0, "model-a"),
{
maxConcurrency: 1,
timeoutMs: 100,
@@ -1376,38 +1393,36 @@ test("handleComboChat returns a 503 when every model is unavailable before execu
assert.equal(payload.error.code, "ALL_ACCOUNTS_INACTIVE");
});
test("handleComboChat returns the circuit-breaker unavailable response when all breakers are open", async () => {
for (const [index, modelStr] of ["openai/model-a", "openai/model-b"].entries()) {
const breaker = getCircuitBreaker(
getComboTargetBreakerKey("all-breakers-open", index, modelStr),
{
failureThreshold: 1,
resetTimeout: 60000,
}
);
breaker._onFailure();
}
test("handleComboChat falls through targets that return provider circuit breaker open responses", async () => {
const calls = [];
const log = createLog();
const result = await handleComboChat({
body: {},
combo: {
name: "all-breakers-open",
name: "provider-breaker-open",
strategy: "priority",
models: ["openai/model-a", "openai/model-b"],
},
handleSingleModel: async () => {
throw new Error("handleSingleModel should not run when all breakers are open");
handleSingleModel: async (_body, modelStr) => {
calls.push(modelStr);
if (modelStr === "openai/model-a") {
return providerBreakerOpenResponse();
}
return okResponse();
},
isModelAvailable: async () => true,
log: createLog(),
log,
settings: null,
relayOptions: null as any,
allCombos: null,
relayOptions: null,
});
assert.equal(result.status, 503);
assert.match((await result.json()).error.message, /circuit breakers open/);
assert.equal(result.ok, true);
assert.deepEqual(calls, ["openai/model-a", "openai/model-b"]);
assert.ok(
log.entries.some((entry) => String(entry.msg).includes("provider circuit breaker OPEN"))
);
});
test("handleComboChat auto strategy honors LKGP after filtering to tool-capable models", async () => {
@@ -1799,39 +1814,37 @@ test("handleComboChat round-robin resolves nested combos and returns inactive wh
assert.equal(payload.error.code, "ALL_ACCOUNTS_INACTIVE");
});
test("handleComboChat round-robin returns circuit-breaker unavailable when every model is open", async () => {
for (const [index, modelStr] of ["openai/model-a", "openai/model-b"].entries()) {
const breaker = getCircuitBreaker(
getComboTargetBreakerKey("rr-breakers-open", index, modelStr),
{
failureThreshold: 1,
resetTimeout: 60000,
}
);
breaker._onFailure();
}
test("handleComboChat round-robin skips targets that return provider circuit breaker open responses", async () => {
const calls = [];
const log = createLog();
const result = await handleComboChat({
body: {},
combo: {
name: "rr-breakers-open",
name: "rr-provider-breaker-open",
strategy: "round-robin",
models: ["openai/model-a", "openai/model-b"],
config: { maxRetries: 0 },
},
handleSingleModel: async () => {
throw new Error("round-robin should not execute when all breakers are open");
handleSingleModel: async (_body, modelStr) => {
calls.push(modelStr);
if (modelStr === "openai/model-a") {
return providerBreakerOpenResponse();
}
return okResponse();
},
isModelAvailable: async () => true,
log: createLog(),
log,
settings: null,
relayOptions: null as any,
allCombos: null,
relayOptions: null,
});
assert.equal(result.status, 503);
assert.match((await result.json()).error.message, /circuit breakers open/);
assert.equal(result.ok, true);
assert.deepEqual(calls, ["openai/model-a", "openai/model-b"]);
assert.ok(
log.entries.some((entry) => String(entry.msg).includes("provider circuit breaker OPEN"))
);
});
test("handleComboChat round-robin retries a transient failure on the same model before succeeding", async () => {

27
tests/unit/domain-branch-hardening.test.ts
View File

@@ -11,7 +11,6 @@ const core = await import("../../src/lib/db/core.ts");
const costRules = await import("../../src/domain/costRules.ts");
const fallbackPolicy = await import("../../src/domain/fallbackPolicy.ts");
const lockoutPolicy = await import("../../src/domain/lockoutPolicy.ts");
const modelAvailability = await import("../../src/domain/modelAvailability.ts");
const providerExpiration = await import("../../src/domain/providerExpiration.ts");
const quotaCache = await import("../../src/domain/quotaCache.ts");
const comboResolver = await import("../../src/domain/comboResolver.ts");
@@ -28,7 +27,6 @@ function isoFromNow(offsetMs) {
async function resetStorage() {
costRules.resetCostData();
fallbackPolicy.resetAllFallbacks();
modelAvailability.resetAllAvailability();
providerExpiration.resetExpirations();
quotaCache.stopBackgroundRefresh();
core.resetDbInstance();
@@ -63,7 +61,6 @@ test.after(async () => {
quotaCache.stopBackgroundRefresh();
costRules.resetCostData();
fallbackPolicy.resetAllFallbacks();
modelAvailability.resetAllAvailability();
providerExpiration.resetExpirations();
core.resetDbInstance();
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
@@ -192,30 +189,6 @@ test("resolveComboModel also covers implicit defaults and missing optional field
assert.deepEqual(comboResolver.getComboFallbacks({}, 0), []);
});
test("modelAvailability tracks missing, active and expired cooldowns", () => {
let now = 1_000;
Date.now = () => now;
assert.equal(modelAvailability.isModelAvailable("openai", "gpt-4o"), true);
modelAvailability.setModelUnavailable("openai", "gpt-4o", 100, undefined);
modelAvailability.setModelUnavailable("anthropic", "claude-sonnet", 500, "capacity");
assert.equal(modelAvailability.isModelAvailable("openai", "gpt-4o"), false);
assert.equal(modelAvailability.getUnavailableCount(), 2);
const report = modelAvailability.getAvailabilityReport();
assert.equal(report.length, 2);
assert.equal(report[0].reason, "unknown");
assert.equal(report[1].reason, "capacity");
now += 150;
assert.equal(modelAvailability.isModelAvailable("openai", "gpt-4o"), true);
assert.equal(modelAvailability.clearModelUnavailability("openai", "gpt-4o"), false);
assert.equal(modelAvailability.clearModelUnavailability("anthropic", "claude-sonnet"), true);
assert.equal(modelAvailability.getUnavailableCount(), 0);
});
test("providerExpiration derives status, sorting, summary and header-based expiration hints", () => {
const expired = providerExpiration.setExpiration(
"conn-expired",

78
tests/unit/error-classification.test.ts
View File

@@ -6,7 +6,7 @@ const { checkFallbackError, getProviderProfile, parseRetryFromErrorText } =
const { getProviderCategory } = await import("../../open-sse/config/providerRegistry.ts");
const { COOLDOWN_MS, PROVIDER_PROFILES, RateLimitReason } =
const { BACKOFF_CONFIG, COOLDOWN_MS, PROVIDER_PROFILES, RateLimitReason } =
await import("../../open-sse/config/constants.ts");
// ─── Provider Category Tests ────────────────────────────────────────────────
@@ -41,12 +41,36 @@ test("getProviderCategory: unknown provider defaults to 'apikey'", () => {
test("getProviderProfile: OAuth provider returns oauth profile", () => {
const profile = getProviderProfile("claude");
assert.deepEqual(profile, PROVIDER_PROFILES.oauth);
assert.equal(profile.baseCooldownMs, PROVIDER_PROFILES.oauth.transientCooldown);
assert.equal(profile.useUpstreamRetryHints, false);
assert.equal(profile.maxBackoffSteps, PROVIDER_PROFILES.oauth.maxBackoffLevel);
assert.equal(profile.failureThreshold, PROVIDER_PROFILES.oauth.circuitBreakerThreshold);
assert.equal(profile.resetTimeoutMs, PROVIDER_PROFILES.oauth.circuitBreakerReset);
assert.equal(profile.transientCooldown, PROVIDER_PROFILES.oauth.transientCooldown);
assert.equal(
profile.rateLimitCooldown,
profile.useUpstreamRetryHints ? 0 : profile.baseCooldownMs
);
assert.equal(profile.maxBackoffLevel, PROVIDER_PROFILES.oauth.maxBackoffLevel);
assert.equal(profile.circuitBreakerThreshold, PROVIDER_PROFILES.oauth.circuitBreakerThreshold);
assert.equal(profile.circuitBreakerReset, PROVIDER_PROFILES.oauth.circuitBreakerReset);
});
test("getProviderProfile: API provider returns apikey profile", () => {
const profile = getProviderProfile("groq");
assert.deepEqual(profile, PROVIDER_PROFILES.apikey);
assert.equal(profile.baseCooldownMs, PROVIDER_PROFILES.apikey.transientCooldown);
assert.equal(profile.useUpstreamRetryHints, true);
assert.equal(profile.maxBackoffSteps, PROVIDER_PROFILES.apikey.maxBackoffLevel);
assert.equal(profile.failureThreshold, PROVIDER_PROFILES.apikey.circuitBreakerThreshold);
assert.equal(profile.resetTimeoutMs, PROVIDER_PROFILES.apikey.circuitBreakerReset);
assert.equal(profile.transientCooldown, PROVIDER_PROFILES.apikey.transientCooldown);
assert.equal(
profile.rateLimitCooldown,
profile.useUpstreamRetryHints ? 0 : profile.baseCooldownMs
);
assert.equal(profile.maxBackoffLevel, PROVIDER_PROFILES.apikey.maxBackoffLevel);
assert.equal(profile.circuitBreakerThreshold, PROVIDER_PROFILES.apikey.circuitBreakerThreshold);
assert.equal(profile.circuitBreakerReset, PROVIDER_PROFILES.apikey.circuitBreakerReset);
});
test("getProviderProfile: profiles have different thresholds", () => {
@@ -64,7 +88,7 @@ test("getProviderProfile: profiles have different thresholds", () => {
// ─── Exponential Backoff for Transient Errors ───────────────────────────────
test("502 transient: exponential backoff 5s → 10s → 20s → 40s → 60s (capped)", () => {
test("502 transient: exponential backoff doubles until the configured max backoff step", () => {
const cooldowns = [];
for (let level = 0; level < 6; level++) {
const result = checkFallbackError(502, "", level, null, null);
@@ -73,14 +97,14 @@ test("502 transient: exponential backoff 5s → 10s → 20s → 40s → 60s (cap
assert.equal(result.newBackoffLevel, level + 1);
assert.equal(result.reason, RateLimitReason.SERVER_ERROR);
}
// Without provider: uses COOLDOWN_MS.transientInitial (5s) as base
assert.equal(cooldowns[0], COOLDOWN_MS.transientInitial); // 5s
assert.equal(cooldowns[1], COOLDOWN_MS.transientInitial * 2); // 10s
assert.equal(cooldowns[2], COOLDOWN_MS.transientInitial * 4); // 20s
assert.equal(cooldowns[3], COOLDOWN_MS.transientInitial * 8); // 40s
// Level 4: 5s * 16 = 80s → capped at 60s
assert.equal(cooldowns[4], COOLDOWN_MS.transientMax); // 60s
assert.equal(cooldowns[5], COOLDOWN_MS.transientMax); // 60s (stays capped)
assert.deepEqual(cooldowns, [
COOLDOWN_MS.transientInitial,
COOLDOWN_MS.transientInitial * 2,
COOLDOWN_MS.transientInitial * 4,
COOLDOWN_MS.transientInitial * 8,
COOLDOWN_MS.transientInitial * 16,
COOLDOWN_MS.transientInitial * 32,
]);
});
test("502 with OAuth provider: uses oauth profile transientCooldown", () => {
@@ -116,11 +140,13 @@ test("429 rate limit: still uses quota-based exponential backoff", () => {
assert.equal(result.reason, RateLimitReason.RATE_LIMIT_EXCEEDED);
});
test("401 auth error: still uses flat cooldown, no backoff", () => {
test("401 auth error: returns terminal auth semantics without connection cooldown", () => {
const result = checkFallbackError(401, "", 0, null, "groq");
assert.equal(result.shouldFallback, true);
assert.equal(result.cooldownMs, COOLDOWN_MS.unauthorized);
assert.equal(result.cooldownMs, 0);
assert.equal(result.baseCooldownMs, 0);
assert.equal(result.newBackoffLevel, undefined);
assert.equal(result.reason, RateLimitReason.AUTH_ERROR);
});
test("400 bad request: still returns shouldFallback false", () => {
@@ -162,7 +188,7 @@ test("parseRetryFromErrorText: parses will reset after variant", () => {
// ─── T06: Keyword Matching for Long Cooldowns ────────────────────────────────
test("quota will reset keyword triggers long cooldown from body", () => {
test("quota reset text is ignored when upstream retry hints are disabled", () => {
const result = checkFallbackError(
429,
"Your quota will reset after 27h41m36s",
@@ -172,19 +198,31 @@ test("quota will reset keyword triggers long cooldown from body", () => {
null
);
assert.equal(result.shouldFallback, true);
assert.ok(result.cooldownMs > 60_000, "cooldownMs should be > 60s");
assert.equal(result.newBackoffLevel, 0, "backoffLevel should reset to 0");
assert.equal(result.cooldownMs, PROVIDER_PROFILES.oauth.transientCooldown);
assert.equal(result.newBackoffLevel, 1);
assert.equal(result.usedUpstreamRetryHint, false);
});
test("exhausted your capacity keyword triggers long cooldown", () => {
test("quota reset text is honored when upstream retry hints are enabled", () => {
const result = checkFallbackError(
429,
"You have exhausted your capacity. Your quota will reset after 2h",
0,
null,
"antigravity",
"groq",
null
);
assert.equal(result.shouldFallback, true);
assert.ok(result.cooldownMs > 60_000);
assert.equal(result.cooldownMs, 2 * 60 * 60 * 1000);
assert.equal(result.newBackoffLevel, 0);
assert.equal(result.usedUpstreamRetryHint, true);
});
test("high transient backoff levels clamp to the configured maxBackoffSteps", () => {
const result = checkFallbackError(502, "", BACKOFF_CONFIG.maxLevel + 5, null, null);
assert.equal(result.newBackoffLevel, BACKOFF_CONFIG.maxLevel);
assert.equal(
result.cooldownMs,
COOLDOWN_MS.transientInitial * Math.pow(2, BACKOFF_CONFIG.maxLevel)
);
});

11
tests/unit/rate-limit-manager.test.ts
View File

@@ -157,9 +157,14 @@ test("rate limit manager parses retry hints from response bodies and locks model
"gpt-4o"
);
const lockout = accountFallback.getModelLockoutInfo("openai", "conn-body", "gpt-4o");
assert.equal(lockout.reason, "rate_limit_exceeded");
assert.ok(lockout.remainingMs > 0);
assert.equal(accountFallback.getModelLockoutInfo("openai", "conn-body", "gpt-4o"), null);
const limiterState = await rateLimitManager.__getLimiterStateForTests(
"openai",
"conn-body",
"gpt-4o"
);
assert.equal(limiterState?.key, "openai:conn-body");
assert.equal(rateLimitManager.getRateLimitStatus("openai", "conn-body").active, true);
rateLimitManager.updateFromResponseBody(
"openai",

15
tests/unit/sse-auth.test.ts
View File

@@ -14,6 +14,7 @@ const settingsDb = await import("../../src/lib/db/settings.ts");
const apiKeysDb = await import("../../src/lib/db/apiKeys.ts");
const auth = await import("../../src/sse/services/auth.ts");
const quotaCache = await import("../../src/domain/quotaCache.ts");
const { COOLDOWN_MS } = await import("../../open-sse/config/constants.ts");
async function resetStorage() {
core.resetDbInstance();
@@ -760,7 +761,7 @@ test("getProviderCredentials exposes copilotToken when present in providerSpecif
assert.equal(selected.copilotToken, "copilot-token-value");
});
test("markAccountUnavailable uses configured cooldowns for local 404 model lockouts", async () => {
test("markAccountUnavailable keeps local 404 failures model-scoped with the local not-found cooldown", async () => {
await settingsDb.updateSettings({
providerProfiles: {
apikey: {
@@ -790,7 +791,7 @@ test("markAccountUnavailable uses configured cooldowns for local 404 model locko
const updated = await providersDb.getProviderConnectionById(connection.id);
assert.equal(result.shouldFallback, true);
assert.equal(result.cooldownMs, 250);
assert.equal(result.cooldownMs, COOLDOWN_MS.notFoundLocal);
assert.equal(updated.testStatus, "active");
assert.equal(updated.rateLimitedUntil, undefined);
assert.equal(updated.lastErrorType, "not_found");
@@ -843,7 +844,7 @@ test("markAccountUnavailable applies a model-only lockout for compatible provide
assert.equal(Number(updated.errorCode), 429);
});
test("markAccountUnavailable honors configured api-key rate-limit cooldowns", async () => {
test("markAccountUnavailable uses the unified configured api-key connection cooldown", async () => {
await settingsDb.updateSettings({
providerProfiles: {
apikey: {
@@ -869,7 +870,7 @@ test("markAccountUnavailable honors configured api-key rate-limit cooldowns", as
);
assert.equal(result.shouldFallback, true);
assert.equal(result.cooldownMs, 125);
assert.equal(result.cooldownMs, 200);
});
test("markAccountUnavailable stores Codex scope-specific cooldowns without a global rate limit", async () => {
@@ -1021,7 +1022,7 @@ test("markAccountUnavailable auto-disables permanently banned accounts when the
assert.equal(result.shouldFallback, true);
assert.equal(updated.isActive, false);
assert.equal(updated.testStatus, "unavailable");
assert.equal(updated.testStatus, "banned");
});
test("markAccountUnavailable leaves permanently banned accounts active when auto-disable is disabled", async () => {
@@ -1041,7 +1042,7 @@ test("markAccountUnavailable leaves permanently banned accounts active when auto
assert.equal(result.shouldFallback, true);
assert.equal(updated.isActive, true);
assert.equal(updated.testStatus, "unavailable");
assert.equal(updated.testStatus, "banned");
});
test("markAccountUnavailable swallows auto-disable persistence errors", async () => {
@@ -1085,7 +1086,7 @@ test("markAccountUnavailable swallows auto-disable persistence errors", async ()
assert.equal(result.shouldFallback, true);
assert.equal(updated.isActive, true);
assert.equal(updated.testStatus, "unavailable");
assert.equal(updated.testStatus, "banned");
} finally {
db.prepare = originalPrepare;
}

11
tests/unit/thundering-herd.test.ts
View File

@@ -9,7 +9,7 @@ import assert from "node:assert/strict";
const { checkFallbackError, getProviderProfile } =
await import("../../open-sse/services/accountFallback.ts");
const { PROVIDER_PROFILES, DEFAULT_API_LIMITS, COOLDOWN_MS, RateLimitReason } =
const { BACKOFF_CONFIG, PROVIDER_PROFILES, DEFAULT_API_LIMITS, COOLDOWN_MS, RateLimitReason } =
await import("../../open-sse/config/constants.ts");
const { getProviderCategory } = await import("../../open-sse/config/providerRegistry.ts");
@@ -36,10 +36,13 @@ test("API profile has shorter transient cooldown", () => {
// ─── Backoff Ceiling Tests (prevents infinite growth) ───────────────────────
test("Exponential backoff is capped at transientMax for high backoff levels", () => {
// Level 20 → 5s * 2^20 = 5.2M ms, but capped at 60s
test("Exponential backoff clamps to the configured maxBackoffLevel", () => {
const result = checkFallbackError(502, "", 20, null, null);
assert.equal(result.cooldownMs, COOLDOWN_MS.transientMax);
assert.equal(result.newBackoffLevel, BACKOFF_CONFIG.maxLevel);
assert.equal(
result.cooldownMs,
COOLDOWN_MS.transientInitial * Math.pow(2, BACKOFF_CONFIG.maxLevel)
);
});
test("API provider backoff level caps at profile maxBackoffLevel", () => {