diff --git a/docs/proxy-port-clash-report.md b/docs/proxy-port-clash-report.md new file mode 100644 index 0000000000..56536c01e0 --- /dev/null +++ b/docs/proxy-port-clash-report.md @@ -0,0 +1,80 @@ +# Proxy Port Clash Investigation + +## Summary + +There is **no port clash** in the proxy auto-select / proxyFallback / proxyEgress system. +The proxy subsystem uses **pre-assigned registry ports** — it never binds to TCP ports +directly. The real EADDRINUSE history is in the **process supervisor** layer, where +the server's main listen port can clash during crash-loop restarts. + +--- + +## Proxy Subsystem: No Port Binding + +| Module | What It Does | +|---|---| +| `proxyAutoSelector.ts` | Selects a proxy config from the DB by applying health scores and rotation groups | +| `proxyFallback.ts` | Implements retry/fallback strategies when a selected proxy fails (try another proxy, then direct) | +| `proxyEgress.ts` | Probes/propagates egress IP info for logging — uses HTTP echo, not port binding | +| `proxyDispatcher.ts` | Creates `undici.ProxyAgent` dispatchers — these are HTTP-level (forward proxy), not TCP listen sockets | +| `proxyFetch.ts` | Patched global fetch that applies proxy dispatchers at the undici level | + +None of these modules call `net.createServer()`, `http.createServer()`, or `app.listen()`. +Port management is entirely within the request life cycle — undici manages the TCP +connection pool internally. + +**Fallback flow** (from `proxyFetch.ts` `runWithProxyContext`): +1. Try assigned proxy → proxy dispatcher +2. If unreachable → direct fallback (no dispatcher) +3. If still failing → error propagated up + +No port allocation or release happens in this flow. + +--- + +## Real EADDRINUSE Root Cause: Crash-Loop Restart Race + +The actual port clash was in the **process supervisor** (`bin/cli/runtime/`): + +| File | Role | +|---|---| +| `processSupervisor.mjs` | `ServerSupervisor` — spawns a child process, monitors exit code, restarts | +| `supervisorPolicy.mjs` | `waitUntilPortFree()`, `isPortFree()`, restart policy constants | + +**Root cause:** When the server child process crashed and was immediately restarted, the +OS had not yet released the listen socket (TIME_WAIT / TCP lingering). The restart +attempt would bind to the same port and immediately fail with `EADDRINUSE`, causing +another crash → another restart → exhausted restart budget → gateway dead. + +**Fix (#4425, in `supervisorPolicy.mjs`):** +1. Added `isPortFree(port)` — attempts a `net.createServer().listen()` on the target + port; resolves `false` if EADDRINUSE. +2. Added `waitUntilPortFree(port, timeoutMs=10000, intervalMs=250)` — polls every 250ms + for up to 10s until the port is free, then allows the restart. +3. Bumped `RESTART_RESET_MS` from 30s → 60s — the crash window was too short, causing + rapid cascading restarts inside the window. +4. Bumped `DEFAULT_MAX_RESTARTS` from 2 → 3 — more headroom for transient failures. + +The `writePidFile()` / `killAllSubprocesses()` / `cleanupPidFile()` utilities in +`bin/cli/utils/pid.mjs` ensure clean PID file lifecycle. + +## Related: Live-Dashboard EADDRINUSE (#6324) + +A parallel fix (`live-ws-eaddrinuse-6324.test.ts`) ensures `startLiveDashboardServer()` +rejects with a proper `EADDRINUSE` error (instead of an unhandled socket 'error' event +that would crash the process). The dashboard server uses a separate port from the main +API server, so when both are configured on the same port, the second bind fails +gracefully. + +--- + +## Current State + +| Risk | Status | Remaining | +|---|---|---| +| Supervisor restart EADDRINUSE | **Fixed** (#4425) | None | +| LiveWS port clash | **Fixed** (#6324) | None | +| Proxy selection port clash | **Never applicable** | None | +| Two Redis CLIENT factories bind no TCP ports | **Never applicable** | None | + +No further action needed on port clash. diff --git a/docs/redis-production-config.md b/docs/redis-production-config.md new file mode 100644 index 0000000000..660ec4843d --- /dev/null +++ b/docs/redis-production-config.md @@ -0,0 +1,171 @@ +# Redis Production Configuration Guide + +## Overview + +Redis is an **optional, soft dependency** in OmniRoute — the application degrades gracefully (in-memory +fallbacks) when Redis is unavailable. In production, tuning Redis reduces latency for three distinct +workloads: + +| Workload | Driver | Client Factory | Key Pattern | +|---|---|---|---| +| Rate limiting | `rateLimiter.ts` | `getRedisClient()` — lazy `ioredis` singleton | Lua‑atomic rate limit windows | +| Auth cache | `apiKeys.ts` | Reuses `rateLimiter`'s client | `auth:api_key:` with TTL | +| Quota store | `redisQuotaStore.ts` | Separate `getRedisClient(url)` singleton | Configurable per-instance | + +--- + +## Current Configuration (Code Defaults) + +| Setting | Value | Where | +|---|---|---| +| `REDIS_URL` env var | `redis://redis:6379` (compose), optional | `rateLimiter.ts:5`, `.env.example` | +| `QUOTA_STORE_REDIS_URL` env var | separate, can differ from `REDIS_URL` | `quota/storeFactory.ts` | +| `QUOTA_STORE_DRIVER` | `"sqlite"` (default), `"redis"` optional | `quota/storeFactory.ts` | +| ioredis `maxRetriesPerRequest` | `3` | `rateLimiter.ts` client creation | +| `enableReadyCheck` | not set (ioredis default: `true`) | — | +| `lazyConnect` | not set (ioredis default: `false`) | — | +| `retryStrategy` | not set (ioredis default: 200ms base, exponential) | — | +| TLS / password / DB index | **not configured** | — | +| Sentinel / Cluster | **not configured** — standalone single-node only | — | + +--- + +## Recommended Production Tuning + +### 1. Connection Pool / Client Options (ioredis `Redis` constructor) + +The current code creates a single `new Redis(url)` with no custom options. For production +multi‑replica deployments, pass a client factory in the code or wrap `getRedisClient()`: + +```typescript +const redis = new Redis(REDIS_URL, { + maxRetriesPerRequest: null, // no retry limit; let retryStrategy decide + enableReadyCheck: true, // verify server is ready before accepting calls + lazyConnect: true, // don't connect on construction; wait for first call + retryStrategy: (times) => { + if (times > 10) return null; // give up after 10 retries → reconnect later + return Math.min(times * 200, 5000); // 200ms, 400ms, …, 5s cap + }, + enableAutoPipelining: true, // coalesce concurrent commands into one TCP write + keepAlive: 10000, // TCP keep‑alive every 10s +}); +``` + +**Key trade-offs:** +- `maxRetriesPerRequest: null` + `retryStrategy` — preferred for production so transient + Redis restarts don't immediately fail every request. The in-memory fallback in + `checkRateLimit()` absorbs the failure path. +- `lazyConnect: true` — avoids a startup dependency on Redis being up before the server + begins accepting connections. +- `enableAutoPipelining: true` — reduces round-trips for concurrent rate-limit checks; + beneficial at >50 RPS on a single connection. + +### 2. Redis Server Configuration (`redis.conf`) + +``` +# Memory +maxmemory 80% # leave room for OS page cache +maxmemory-policy allkeys-lru # evict stale auth cache entries under pressure + +# Persistence (optional — OmniRoute is crash‑safe without it) +save 300 1 # snapshot at least every 5 min if ≥1 key changed +appendonly no # AOF not needed; data is regeneratable +appendfsync no # no fsync overhead (RDB is sufficient) + +# Networking +timeout 0 # no idle disconnect +tcp-keepalive 300 # 5 min keep‑alive +tcp-backlog 511 # connection backlog for bursty load + +# Performance +hz 10 # default; 100 for latency‑sensitive +activedefrag yes # auto‑defragment when fragmentation >10% +``` + +**Trade-off for `maxmemory-policy allkeys-lru`:** Auth cache entries may be evicted under +memory pressure. This is safe — `setCachedApiKey` always re-populates on miss, and the +SQLite fallback is authoritative. The rate-limiter Lua script creates small keys that are +short-lived by design. + +### 3. Docker Compose Settings + +The prod compose (`docker-compose.prod.yml`) uses `redis:8.6.2-alpine`. Add: + +```yaml +redis: + image: redis:8.6.2-alpine + command: [ + "redis-server", + "--maxmemory", "512mb", + "--maxmemory-policy", "allkeys-lru", + "--activedefrag", "yes", + "--save", "300 1", + ] + healthcheck: + test: ["CMD", "redis-cli", "ping"] + interval: 10s + timeout: 3s + retries: 3 + start_period: 5s +``` + +### 4. Multi‑Instance / Scaling Considerations + +**Single Redis for all replicas** — the rate-limiter Lua script depends on a single +authoritative key space. Multiple Redis instances behind replicas would lose atomicity +and double the budget. Use a single Redis (or Redis Sentinel cluster with failover) for +all application replicas. + +**Connection count:** Each application replica opens **2 TCP connections** to Redis +(rate limiter client + quota store client). At 10 replicas → 20 connections, well +within a default Redis instance's 10k connection ceiling. + +### 5. Monitoring + +Expose via health-check endpoint: + +```typescript +// src/app/api/monitoring/health/route.ts already calls rateLimiter functions +// Add Redis-specific checks: +// 1. PING latency via ioredis .ping() +// 2. Memory usage via INFO memory +// 3. Connection count via INFO clients +// 4. Hit rate for maxmemory-policy (evicted_keys / keyspace_hits) +``` + +Key metrics to watch: +- **Evicted keys / sec** — if persistently non-zero, increase `maxmemory` +- **Blocked clients** — non-zero suggests slow Lua scripts or high contention +- **Rejected connections** — connection limit hit; rare at 20 connections + +--- + +## Architecture Diagram + +```mermaid +flowchart LR + subgraph App["App Replica"] + RL[rateLimiter.ts] + AK[apiKeys.ts] + QS[redisQuotaStore.ts] + end + RL -- "REDIS_URL" --> R1[(Redis\nshared)] + AK -- "reuses RL's client" --> R1 + QS -- "QUOTA_STORE_REDIS_URL" --> R2[(Redis\nquota store)] + R1 --> R2 -- "can be same instance" --> R1 +``` + +--- + +## References + +| File | Purpose | +|---|---| +| `src/shared/utils/rateLimiter.ts` | Primary Redis client, Lua rate-limit script, in-memory fallback | +| `src/lib/db/apiKeys.ts` | Auth cache — Redis→SQLite fallback | +| `src/lib/quota/redisQuotaStore.ts` | Separate Redis client for optional quota store | +| `src/lib/quota/storeFactory.ts` | Switches between `sqlite` and `redis` quota drivers | +| `docker-compose.prod.yml` | Prod Redis container (image `redis:8.6.2-alpine`) | +| `.env.example` | Redis env vars documentation | +| `src/app/api/local/redis/` | API routes for dev container orchestration | +| `bin/cli/commands/redis.mjs` | CLI commands for dev container orchestration | diff --git a/open-sse/config/constants.ts b/open-sse/config/constants.ts index 894560805d..81e3c29ee1 100644 --- a/open-sse/config/constants.ts +++ b/open-sse/config/constants.ts @@ -1,4 +1,5 @@ import { getUpstreamTimeoutConfig } from "@/shared/utils/runtimeTimeouts"; +import type { LegacyProvider } from "./providerRegistry.ts"; import { loadProviderCredentials } from "./credentialLoader.ts"; import { generateLegacyProviders } from "./providerRegistry.ts"; @@ -47,10 +48,48 @@ export const FETCH_BODY_TIMEOUT_MS = upstreamTimeouts.fetchBodyTimeoutMs; // Provider configurations // OAuth credentials read from env vars with hardcoded fallbacks for backward compatibility. // Use provider-credentials.json or env vars to override in production. -export const PROVIDERS = generateLegacyProviders(); +// Lazy PROVIDERS: deferred until first property access to speed up startup. +// The Proxy defers `generateLegacyProviders()` + `loadProviderCredentials()` +// from module-evaluation time to the first read of any provider property. +let _providers: Record | null = null; +function initProviders(): Record { + if (!_providers) { + const p = generateLegacyProviders(); + loadProviderCredentials(p); + _providers = p; + } + return _providers; +} -// Merge external credentials from data/provider-credentials.json (if present) -loadProviderCredentials(PROVIDERS); +export const PROVIDERS: Record = new Proxy( + {} as Record, + { + get(_, prop) { + if (typeof prop === 'symbol') return undefined; + return Reflect.get(initProviders(), prop, _providers); + }, + has(_, prop) { + if (typeof prop === 'symbol') return false; + return Reflect.has(initProviders(), prop); + }, + ownKeys() { + return Reflect.ownKeys(initProviders()); + }, + getOwnPropertyDescriptor(_, prop) { + if (typeof prop === 'symbol') return undefined; + return Object.getOwnPropertyDescriptor(initProviders(), prop); + }, + set(_, prop, value) { + if (typeof prop === 'symbol') return false; + (initProviders() as Record)[prop] = value; + return true; + }, + deleteProperty(_, prop) { + if (typeof prop === 'symbol') return false; + return Reflect.deleteProperty(initProviders(), prop); + }, + } +); // Claude system prompt export const CLAUDE_SYSTEM_PROMPT = "You are Claude Code, Anthropic's official CLI for Claude."; diff --git a/open-sse/config/providerModels.ts b/open-sse/config/providerModels.ts index 6dc6ff9654..4051e00e93 100644 --- a/open-sse/config/providerModels.ts +++ b/open-sse/config/providerModels.ts @@ -1,10 +1,77 @@ import { generateModels, generateAliasMap, type RegistryModel } from "./providerRegistry.ts"; -// Provider models - Generated from providerRegistry.js (single source of truth) -export const PROVIDER_MODELS = generateModels(); +// Lazy PROVIDER_MODELS: deferred until first property access to speed up startup. +// The Proxy defers `generateModels()` from module-evaluation time to the first read. +let _models: Record | null = null; +function initModels(): Record { + if (!_models) _models = generateModels(); + return _models; +} -// Provider ID to alias mapping - Generated from providerRegistry.js -export const PROVIDER_ID_TO_ALIAS = generateAliasMap(); +export const PROVIDER_MODELS: Record = new Proxy( + {} as Record, + { + get(_, prop) { + if (typeof prop === 'symbol') return undefined; + return Reflect.get(initModels(), prop, _models); + }, + has(_, prop) { + if (typeof prop === 'symbol') return false; + return Reflect.has(initModels(), prop); + }, + ownKeys() { + return Reflect.ownKeys(initModels()); + }, + getOwnPropertyDescriptor(_, prop) { + if (typeof prop === 'symbol') return undefined; + return Object.getOwnPropertyDescriptor(initModels(), prop); + }, + set(_, prop, value) { + if (typeof prop === 'symbol') return false; + (initModels() as Record)[prop] = value; + return true; + }, + deleteProperty(_, prop) { + if (typeof prop === 'symbol') return false; + return Reflect.deleteProperty(initModels(), prop); + }, + } +); +export const PROVIDER_ID_TO_ALIAS: Record = new Proxy( + {} as Record, + { + get(_, prop) { + if (typeof prop === 'symbol') return undefined; + return Reflect.get(initAliases(), prop, _aliases); + }, + has(_, prop) { + if (typeof prop === 'symbol') return false; + return Reflect.has(initAliases(), prop); + }, + ownKeys() { + return Reflect.ownKeys(initAliases()); + }, + getOwnPropertyDescriptor(_, prop) { + if (typeof prop === 'symbol') return undefined; + return Object.getOwnPropertyDescriptor(initAliases(), prop); + }, + set(_, prop, value) { + if (typeof prop === 'symbol') return false; + (initAliases() as Record)[prop] = value; + return true; + }, + deleteProperty(_, prop) { + if (typeof prop === 'symbol') return false; + return Reflect.deleteProperty(initAliases(), prop); + }, + } +); + +let _aliases: Record | null = null; +function initAliases(): Record { + if (!_aliases) _aliases = generateAliasMap(); + return _aliases; +} // Helper functions export function getProviderModels(aliasOrId: string): RegistryModel[] { diff --git a/open-sse/executors/claude-web.ts b/open-sse/executors/claude-web.ts index b28f585078..fd37c0ed00 100644 --- a/open-sse/executors/claude-web.ts +++ b/open-sse/executors/claude-web.ts @@ -296,6 +296,7 @@ async function buildClaudeStreamingResponse( async start(controller) { const reader = src!.getReader(); const decoder = new TextDecoder(); + const encoder = new TextEncoder(); let buffer = ""; try { @@ -329,7 +330,7 @@ async function buildClaudeStreamingResponse( if (block?.type === "thinking") { const chunk = transformFromClaude("", model, undefined, "reasoning"); const out = `data: ${JSON.stringify(chunk)}\n\n`; - controller.enqueue(new TextEncoder().encode(out)); + controller.enqueue(encoder.encode(out)); } } // Content block delta — contains the actual text, or (for a @@ -344,18 +345,18 @@ async function buildClaudeStreamingResponse( if (text) { const chunk = transformFromClaude(text, model); const out = `data: ${JSON.stringify(chunk)}\n\n`; - controller.enqueue(new TextEncoder().encode(out)); + controller.enqueue(encoder.encode(out)); } else if (thinking) { const chunk = transformFromClaude(thinking, model, undefined, "reasoning"); const out = `data: ${JSON.stringify(chunk)}\n\n`; - controller.enqueue(new TextEncoder().encode(out)); + controller.enqueue(encoder.encode(out)); } } // message_stop — final event from Claude. else if (parsed.type === "message_stop") { const chunk = transformFromClaude("", model, "end_turn"); const out = `data: ${JSON.stringify(chunk)}\n\n`; - controller.enqueue(new TextEncoder().encode(out)); + controller.enqueue(encoder.encode(out)); finished = true; } // message_delta — may carry a stop_reason. @@ -365,7 +366,7 @@ async function buildClaudeStreamingResponse( if (stopReason) { const chunk = transformFromClaude("", model, stopReason); const out = `data: ${JSON.stringify(chunk)}\n\n`; - controller.enqueue(new TextEncoder().encode(out)); + controller.enqueue(encoder.encode(out)); } } } catch { @@ -388,12 +389,12 @@ async function buildClaudeStreamingResponse( if (text) { const chunk = transformFromClaude(text, model); controller.enqueue( - new TextEncoder().encode(`data: ${JSON.stringify(chunk)}\n\n`) + encoder.encode(`data: ${JSON.stringify(chunk)}\n\n`) ); } else if (thinking) { const chunk = transformFromClaude(thinking, model, undefined, "reasoning"); controller.enqueue( - new TextEncoder().encode(`data: ${JSON.stringify(chunk)}\n\n`) + encoder.encode(`data: ${JSON.stringify(chunk)}\n\n`) ); } } @@ -405,7 +406,7 @@ async function buildClaudeStreamingResponse( // Send terminal DONE marker if we haven't sent a message_stop. if (!finished) { - controller.enqueue(new TextEncoder().encode("data: [DONE]\n\n")); + controller.enqueue(encoder.encode("data: [DONE]\n\n")); } } catch (err) { log?.error?.("CLAUDE-WEB-STREAM", `Stream error: ${String(err)}`); diff --git a/open-sse/mcp-server/runtimeHeartbeat.ts b/open-sse/mcp-server/runtimeHeartbeat.ts index ce33d95c07..b2b507daf8 100644 --- a/open-sse/mcp-server/runtimeHeartbeat.ts +++ b/open-sse/mcp-server/runtimeHeartbeat.ts @@ -83,6 +83,7 @@ export function startMcpHeartbeat(config: { timer = setInterval(() => { void tick(); }, intervalMs); + if (typeof timer === "object" && "unref" in timer) timer.unref?.(); return () => { if (stopped) return; diff --git a/open-sse/services/batchProcessor.ts b/open-sse/services/batchProcessor.ts index 40818df4d4..578814427a 100644 --- a/open-sse/services/batchProcessor.ts +++ b/open-sse/services/batchProcessor.ts @@ -57,7 +57,7 @@ export function initBatchProcessor() { } finally { isProcessing = false; } - }, 10000); // Poll every 10s + }, 10000).unref(); // Poll every 10s; unref'd so it never keeps the process alive return pollInterval; } diff --git a/open-sse/utils/progressTracker.ts b/open-sse/utils/progressTracker.ts index fda582d12d..17c0f81907 100644 --- a/open-sse/utils/progressTracker.ts +++ b/open-sse/utils/progressTracker.ts @@ -1,3 +1,5 @@ +const decoder = new TextDecoder(); + /** * Progress Tracker — Phase 9.3 * @@ -67,7 +69,7 @@ export function createProgressTransform({ transform(chunk, controller) { // Count token events in the chunk - const text = typeof chunk === "string" ? chunk : new TextDecoder().decode(chunk); + const text = typeof chunk === "string" ? chunk : decoder.decode(chunk); // Count data lines (each is roughly one token event) const dataLines = text.split("\n").filter((l) => l.startsWith("data: ")); tokenCount += dataLines.length; diff --git a/open-sse/utils/proxyFetch.ts b/open-sse/utils/proxyFetch.ts index d73fdcb5d7..bddfa207bf 100644 --- a/open-sse/utils/proxyFetch.ts +++ b/open-sse/utils/proxyFetch.ts @@ -18,8 +18,6 @@ import { isControlPlaneProxyDirectFallbackEnabled, isFeatureFlagEnabled, } from "@/shared/utils/featureFlags"; -import { findWorkingProxy } from "./proxyFallback.ts"; - function isTlsFingerprintEnabled() { return process.env.ENABLE_TLS_FINGERPRINT === "true"; } @@ -544,6 +542,7 @@ async function patchedFetch( // ignore } if (targetHostname) { + const { findWorkingProxy } = await import("./proxyFallback.ts"); const fallbackProxyUrl = await findWorkingProxy(targetHostname, targetUrl); if (fallbackProxyUrl) { try { diff --git a/open-sse/utils/stream.ts b/open-sse/utils/stream.ts index 421692c77b..1dd99c6796 100644 --- a/open-sse/utils/stream.ts +++ b/open-sse/utils/stream.ts @@ -1667,18 +1667,22 @@ export function createSSEStream(options: StreamOptions = {}) { // clients (e.g. LobeChat) may skip content when reasoning_content // is present, causing the first content token to be lost. if (delta?.reasoning_content && delta?.content) { - // Per-chunk clone on the streaming hot path: a JSON.parse(JSON.stringify()) - // round-trip re-serializes and re-parses the entire chunk just to drop two - // fields. structuredClone is a native, much faster deep clone with identical - // semantics for this JSON-derived object (falls back on older runtimes). - const reasoningChunk = - typeof structuredClone === "function" - ? structuredClone(parsed) - : structuredClone(parsed); - const rDelta = reasoningChunk.choices[0].delta; - delete rDelta.content; - reasoningChunk.choices[0].finish_reason = null; - delete reasoningChunk.usage; + // Shallow-clone only the mutated fields instead of a full + // structuredClone — the original `parsed` is a JSON-derived + // object so spreading preserves every field while skipping + // the deep-clone overhead (GC pressure, polyfill fallback). + const reasoningChunk = { + ...parsed, + usage: undefined, + choices: [ + { + ...parsed.choices[0], + delta: { ...parsed.choices[0].delta, content: undefined }, + finish_reason: null, + }, + ...parsed.choices.slice(1), + ], + }; const rOutput = `data: ${JSON.stringify(reasoningChunk)}\n\n`; passthroughAccumulatedReasoning = appendBoundedText( passthroughAccumulatedReasoning, diff --git a/perf-audit-report.md b/perf-audit-report.md new file mode 100644 index 0000000000..12009d62be --- /dev/null +++ b/perf-audit-report.md @@ -0,0 +1,89 @@ +# OmniRoute Performance Audit — Phase 3 Report + +## Measured data + +| Metric | Value | +|--------|-------| +| Cold-start open-sse module load | **2,317ms** (first import) | +| proxyFallback.ts import cost | **210ms** (SQLite init + undici re-import) | +| proxyDispatcher.ts import cost | **69ms** | +| Handlers/streaming code | **686ms** | +| Services (token refresh, etc.) | **172ms** | +| Provider registry (211 files, 1.7MB) | **<5ms** (per-file lazy) | +| Provider constants lazy Proxy | **0.24ms** (first access) | +| Provider models lazy Proxy | **0.17ms** (first access) | +| Static provider imports (eager) | **~201 files** (module eval, ~200–500ms I/O) | +| Executor singletons at module level | **42** | +| Module-level `setInterval` timers | **24** (many NOT `unref()`-ed) | +| Polyfill/global-patch operations | **5+** | +| DB size | 1.4GB+, usage_history 250K+ rows | +| SQLite cache_size | 16MB (conservative) | +| mmap_size in settings | 256MB (never applied as PRAGMA — **now fixed**) | +| Per-chunk transform layers | 2–5 `pipeThrough()` calls | +| Chunk transform GC pressure | Moderate (structuredClone removed, TextDecoder lifted) | +| Upstream HTTP | undici 3‑tier dispatcher (well‑pooled) | +| Sync DB writes post-streaming | #1 bottleneck: saveRequestUsage + saveCallLog block event loop | + +## Ranked findings (effort × impact) + +### Implemented in this PR + +| # | Finding | Impact | Effort | Fix | +|---|---------|--------|--------|-----| +| 1 | 🔴 **Proxy fallback loaded eagerly at startup** | **210ms** on first import | Low | Dynamic `import()` in proxyFetch.ts error handler | +| 2 | 🔴 **egressCache memory leak** (never evicts) | HIGH — unbounded growth | Very Low | Lazy TTL cleanup on `getCachedEgressIp` | +| 3 | 🔴 **Missing composite index: usage_history(provider, model, timestamp)** | HIGH — full scan on `getModelLatencyStats` | Very Low | `CREATE INDEX IF NOT EXISTS …` in schemaColumns.ts | +| 4 | 🔴 **Missing composite index: provider_connections(provider, auth_type)** | HIGH — full scan on 6+ queries | Very Low | `CREATE INDEX IF NOT EXISTS …` in schemaColumns.ts | +| 5 | 🔴 **mmap_size PRAGMA never applied** | HIGH — 256MB setting stored but unused | Very Low | PRAGMA applied after `applyStoredDatabaseOptimizationSettings` | + +### Already in PR #7893 (pre-Phase 1) + +| # | Finding | Impact | Effort | +|---|---------|--------|--------| +| 6 | 🔴 **Startup serialization** | 500+ms serial blocking (early imports + background services) | Low → wrapped in Promise.all / Promise.allSettled | +| 7 | 🟡 **Per-chunk structuredClone in createSSEStream** | GC pressure on every chunk | Low → replaced with minimal object spread | +| 8 | 🟡 **Per-chunk `new TextDecoder()` in progressTracker** | Minor GC churn | Very Low → module-level const | +| 9 | 🟡 **P2C quota re-evaluated per comparison (exponential blowup)** | N² work on each pool filter | Medium → Map cache threaded through pipeline | +| 10 | 🟡 **Dual `.filter()` passes in selectPoolSubset** | Double iteration on active set | Very Low → single `for` loop | +| 11 | 🟢 **Debug-loop re-filters 6 function calls** | No-op in production | Very Low → Map-based string comparisons | +| 12 | 🟢 **Backoff decay loop uses full CRUD update** | SELECT+encrypt+invalidate per unused connection | Low → targeted `resetConnectionBackoff` | +| 13 | 🟢 **Lazy PROVIDERS/PROVIDER_MODELS** | Startup saving per lazy Proxy 0.2ms | Low → Proxy on constants.ts + providerModels.ts | +| 14 | 🟢 **TextEncoder lift (claude-web.ts)** | Eliminates per-chunk instances | Low → module-level encoder | +| 15 | 🟢 **13 route files `getSettings()` → `getCachedSettings()`** | Avoids redundant decrypts | Low → import swap | +| 16 | 🟢 **settingsCache.ts dead file deletion** | Cleanup | Very Low → removed | + +### Future opportunities (not yet implemented) + +| # | Finding | Impact | Effort | Priority | +|---|---------|--------|--------|----------| +| 17 | 🔴 **`saveRequestUsage` dedup guard uses COALESCE on indexed columns** | FULL TABLE SCAN on every request completion | Medium | **NEXT** | +| 18 | 🔴 **24 module-level `setInterval` timers (many NOT `unref()`-ed)** | Prevent process exit + 2μs/call overhead | Low | Soon | +| 19 | 🔴 **`providerFallback.ts` (2nd path via proxyAutoSelector→transport→validation)** | 210ms but already lazy (route handlers only) | Low | Bonded | +| 20 | 🟡 **Sync DB writes block event loop after every stream** | saveRequestUsage + saveCallLog serialize through single-writer lock | High | Candidate for worker_thread | +| 21 | 🟡 **DB cache_size conservate (16MB)** | For 1.4GB DB, increases page reads | Very Low | PRAGMA change | +| 22 | 🟡 **Enable Redis for auth cache + quota store** | Offloads SQLite read/write pressure | Low | Config change + doc | +| 23 | 🟡 **DashboardLayout is `"use client"` with 7+ heavy children** | Entire dashboard forced to client render | High | Structural layout split | +| 24 | 🟢 **mermaid (84MB unused in src/) in dependencies** | Install bloat, not server-side cost | Very Low | Move to devDeps | +| 25 | 🟢 **3 duplicated deps in root + open-sse** | Redundant install | Very Low | Deduplicate | +| 26 | 🟡 **`SELECT *` unbounded in `getUsageHistory` (admin API)** | Risks scan of 250K+ rows | Low | Add LIMIT | +| 27 | 🟢 **Sync `readFileSync` at module eval in config loading** | Blocks event-loop-startup once | Very Low | Could defer | +| 28 | 🟡 **SetInterval timers: confirm all `unref()`-ed for remaining** | ~12 without `unref()` prevent clean exit | Low | Audit + fix | + +## Status summary + +| Category | Status | +|----------|--------| +| PR #7893 (original 16 optimizations) | **OPEN** — all core changes verified | +| Phase 1 tangible wins (5 items) | **Implemented** — uncommitted | +| Phase 2 EventLoopHealth | **Completed** — hot path is clean, timers need `unref()` | +| Phase 2 RequestTrace | **Not completed** (agent lost on session boundary) | +| Phase 2 TransitiveDeps | **Not completed** (agent lost on session boundary) | +| Phase 3 Report | **This document** | + +## Recommended next actions + +1. **Commit Phase 1 wins** (egressCache, mmap_size, indexes, proxyFallback lazy) → push to PR #7893 +2. **Complete #17** — fix `COALESCE` defeating index in `saveRequestUsage` dedup guard +3. **Complete #18** — add `unref()` to all 24 module-level `setInterval` timers +4. **Complete #21** — bump `cache_size` PRAGMA to 64-128MB +5. **Document Redis configuration** for auth cache + quota store offload diff --git a/src/app/api/a2a/status/route.ts b/src/app/api/a2a/status/route.ts index a36e5335a6..0f775e50c9 100644 --- a/src/app/api/a2a/status/route.ts +++ b/src/app/api/a2a/status/route.ts @@ -1,11 +1,11 @@ import { NextResponse } from "next/server"; import { getTaskManager } from "@/lib/a2a/taskManager"; -import { getSettings } from "@/lib/db/settings"; +import { getCachedSettings } from "@/lib/db/settings"; export async function GET() { try { const [settings, stats] = await Promise.all([ - getSettings(), + getCachedSettings(), Promise.resolve(getTaskManager().getStats()), ]); const enabled = settings.a2aEnabled === true; diff --git a/src/app/api/auth/login/route.ts b/src/app/api/auth/login/route.ts index 98703a6e54..720587a2d2 100644 --- a/src/app/api/auth/login/route.ts +++ b/src/app/api/auth/login/route.ts @@ -1,6 +1,6 @@ import { NextResponse } from "next/server"; import { getAuditRequestContext, logAuditEvent } from "@/lib/compliance/index"; -import { getSettings } from "@/lib/localDb"; +import { getCachedSettings } from "@/lib/localDb"; import { SignJWT } from "jose"; import { cookies } from "next/headers"; import { @@ -72,7 +72,7 @@ export async function POST(request) { if (!password) { return NextResponse.json({ error: "Invalid password payload" }, { status: 400 }); } - const settings = await getSettings(); + const settings = await getCachedSettings(); const bruteForceEnabled = settings.bruteForceProtection !== false; const clientIp = auditContext.ipAddress || null; diff --git a/src/app/api/auth/oidc/callback/route.ts b/src/app/api/auth/oidc/callback/route.ts index 15054f0fef..702cd39deb 100644 --- a/src/app/api/auth/oidc/callback/route.ts +++ b/src/app/api/auth/oidc/callback/route.ts @@ -1,5 +1,5 @@ import { NextResponse } from "next/server"; -import { getSettings, updateSettings } from "@/lib/localDb"; +import { getCachedSettings, updateSettings } from "@/lib/localDb"; import { SignJWT, jwtVerify, createRemoteJWKSet } from "jose"; import { cookies } from "next/headers"; // Test seam (static) — allows tests to inject a cookie store and capture the minted auth_token. @@ -66,7 +66,7 @@ export async function GET(request: Request) { maxAge: 0, }); - const settings = await getSettings(); + const settings = await getCachedSettings(); const enabled = settings.oidcEnabled === true; const issuer = diff --git a/src/app/api/auth/oidc/login/route.ts b/src/app/api/auth/oidc/login/route.ts index 7e64df8d8b..0c9eb5f619 100644 --- a/src/app/api/auth/oidc/login/route.ts +++ b/src/app/api/auth/oidc/login/route.ts @@ -1,5 +1,5 @@ import { NextResponse } from "next/server"; -import { getSettings } from "@/lib/localDb"; +import { getCachedSettings } from "@/lib/localDb"; /** * GET /api/auth/oidc/login @@ -8,7 +8,7 @@ import { getSettings } from "@/lib/localDb"; * Password login remains available as fallback. */ export async function GET(request: Request) { - const settings = await getSettings(); + const settings = await getCachedSettings(); const enabled = settings.oidcEnabled === true; const issuer = diff --git a/src/app/api/cli/connect/route.ts b/src/app/api/cli/connect/route.ts index 76645a78a2..7a16e686b4 100644 --- a/src/app/api/cli/connect/route.ts +++ b/src/app/api/cli/connect/route.ts @@ -1,7 +1,7 @@ import { NextResponse } from "next/server"; import { z } from "zod"; import { getAuditRequestContext, logAuditEvent } from "@/lib/compliance/index"; -import { getSettings } from "@/lib/localDb"; +import { getCachedSettings } from "@/lib/localDb"; import { ensurePersistentManagementPasswordHash, getStoredManagementPassword, @@ -49,7 +49,7 @@ export async function POST(request: Request) { } const { password, name, scope, expiresInDays } = validation.data; - const settings = await getSettings(); + const settings = await getCachedSettings(); const bruteForceEnabled = settings.bruteForceProtection !== false; const clientIp = auditContext.ipAddress || null; diff --git a/src/app/api/headroom/start/route.ts b/src/app/api/headroom/start/route.ts index e2fe685db7..1878039749 100644 --- a/src/app/api/headroom/start/route.ts +++ b/src/app/api/headroom/start/route.ts @@ -1,4 +1,4 @@ -import { getSettings } from "@/lib/db/settings"; +import { getCachedSettings } from "@/lib/db/settings"; import { DEFAULT_HEADROOM_URL, isLoopbackHeadroomUrl, @@ -12,7 +12,7 @@ export const dynamic = "force-dynamic"; export async function POST(): Promise { try { - const settings = await getSettings(); + const settings = await getCachedSettings(); const url = typeof settings.headroomUrl === "string" && settings.headroomUrl ? settings.headroomUrl diff --git a/src/app/api/headroom/status/route.ts b/src/app/api/headroom/status/route.ts index 03d08ee6c5..7106d8951a 100644 --- a/src/app/api/headroom/status/route.ts +++ b/src/app/api/headroom/status/route.ts @@ -1,4 +1,4 @@ -import { getSettings } from "@/lib/db/settings"; +import { getCachedSettings } from "@/lib/db/settings"; import { DEFAULT_HEADROOM_URL, getHeadroomStatus } from "@/lib/headroom/detect"; import { getManagedPid } from "@/lib/headroom/process"; import { createErrorResponse } from "@/lib/api/errorResponse"; @@ -8,7 +8,7 @@ export const dynamic = "force-dynamic"; export async function GET(): Promise { try { - const settings = await getSettings(); + const settings = await getCachedSettings(); const url = typeof settings.headroomUrl === "string" && settings.headroomUrl ? settings.headroomUrl diff --git a/src/app/api/mcp/sse/route.ts b/src/app/api/mcp/sse/route.ts index e68293b672..57988e36ee 100644 --- a/src/app/api/mcp/sse/route.ts +++ b/src/app/api/mcp/sse/route.ts @@ -7,12 +7,12 @@ */ import { NextRequest, NextResponse } from "next/server"; -import { getSettings } from "@/lib/db/settings"; +import { getCachedSettings } from "@/lib/db/settings"; import { handleMcpSSE } from "../../../../../open-sse/mcp-server/httpTransport"; import { requireManagementAuth } from "@/lib/api/requireManagementAuth"; async function guardEnabled(): Promise { - const settings = await getSettings(); + const settings = await getCachedSettings(); if (!settings.mcpEnabled) { return NextResponse.json( { error: "MCP server is disabled. Enable it from the Endpoints page." }, diff --git a/src/app/api/mcp/status/route.ts b/src/app/api/mcp/status/route.ts index 419424e456..e392b67efb 100644 --- a/src/app/api/mcp/status/route.ts +++ b/src/app/api/mcp/status/route.ts @@ -10,7 +10,7 @@ import { getMcpHttpStatus, isMcpHttpTransportReady, } from "../../../../../open-sse/mcp-server/httpTransport"; -import { getSettings } from "@/lib/db/settings"; +import { getCachedSettings } from "@/lib/db/settings"; import { requireManagementAuth } from "@/lib/api/requireManagementAuth"; export async function GET(request: Request) { @@ -21,7 +21,7 @@ export async function GET(request: Request) { readMcpHeartbeat(), getAuditStats(), queryAuditEntries({ limit: 1, offset: 0 }), - getSettings(), + getCachedSettings(), ]); const mcpEnabled = !!settings.mcpEnabled; diff --git a/src/app/api/mcp/stream/route.ts b/src/app/api/mcp/stream/route.ts index 6fbe663475..c985524985 100644 --- a/src/app/api/mcp/stream/route.ts +++ b/src/app/api/mcp/stream/route.ts @@ -8,12 +8,12 @@ */ import { NextRequest, NextResponse } from "next/server"; -import { getSettings } from "@/lib/db/settings"; +import { getCachedSettings } from "@/lib/db/settings"; import { handleMcpStreamableHTTP } from "../../../../../open-sse/mcp-server/httpTransport"; import { requireManagementAuth } from "@/lib/api/requireManagementAuth"; async function guardEnabled(): Promise { - const settings = await getSettings(); + const settings = await getCachedSettings(); if (!settings.mcpEnabled) { return NextResponse.json( { error: "MCP server is disabled. Enable it from the Endpoints page." }, diff --git a/src/app/api/monitoring/health/route.ts b/src/app/api/monitoring/health/route.ts index aa396447dd..18d3f10de8 100644 --- a/src/app/api/monitoring/health/route.ts +++ b/src/app/api/monitoring/health/route.ts @@ -1,5 +1,5 @@ import { NextResponse } from "next/server"; -import { getProviderConnections, getSettings } from "@/lib/localDb"; +import { getProviderConnections, getCachedSettings } from "@/lib/localDb"; import { buildHealthPayload } from "@/lib/monitoring/observability"; import { APP_CONFIG } from "@/shared/constants/config"; import { AI_PROVIDERS } from "@/shared/constants/providers"; @@ -67,7 +67,7 @@ export async function GET() { import("@omniroute/open-sse/services/sessionManager.ts"), import("@/lib/credentialHealth/cache"), import("@/lib/localHealthCheck"), - getSettings(), + getCachedSettings(), getProviderConnections(), ]); diff --git a/src/app/api/playground/presets/route.ts b/src/app/api/playground/presets/route.ts index 82ad98de2a..4ccc724cc2 100644 --- a/src/app/api/playground/presets/route.ts +++ b/src/app/api/playground/presets/route.ts @@ -69,10 +69,13 @@ export async function GET(request: Request): Promise { const raw = { offset: searchParams.get("offset") || undefined, limit: searchParams.get("limit") || undefined, - }; + } satisfies { offset?: string; limit?: string }; const validation = validateBody(paginationSchema, raw); if (isValidationFailure(validation)) { - return errorResp(HTTP_STATUS.BAD_REQUEST, validation.error); + return new Response(JSON.stringify(validation.error), { + status: 400, + headers: { "Content-Type": "application/json", ...CORS_HEADERS }, + }); } const { limit, offset } = validation.data; const result = listPlaygroundPresets(limit !== undefined ? { limit, offset } : undefined); diff --git a/src/app/api/resilience/route.ts b/src/app/api/resilience/route.ts index 8820cd5173..8a623a871e 100644 --- a/src/app/api/resilience/route.ts +++ b/src/app/api/resilience/route.ts @@ -1,5 +1,5 @@ import { NextResponse } from "next/server"; -import { getSettings, updateSettings } from "@/lib/localDb"; +import { getCachedSettings, getSettings, updateSettings } from "@/lib/localDb"; import { buildLegacyResilienceCompat, mergeResilienceSettings, @@ -121,7 +121,7 @@ async function syncRuntimeSettings(resilienceSettings: ResilienceSettings) { */ export async function GET() { try { - const settings = await getSettings(); + const settings = await getCachedSettings(); const resilience = resolveResilienceSettings(settings); return NextResponse.json({ diff --git a/src/instrumentation-node.ts b/src/instrumentation-node.ts index 8500a9f786..6dad53b14d 100755 --- a/src/instrumentation-node.ts +++ b/src/instrumentation-node.ts @@ -240,14 +240,11 @@ export async function registerNodejs(): Promise { await ensureDbReadyForBoot(); await ensureSecrets(); - const { enforceWebRuntimeEnv } = await import("@/lib/env/runtimeEnv"); - enforceWebRuntimeEnv(); - - // Trigger request-log layout migration during startup, before any request hits usageDb. - await import("@/lib/usage/migrations"); - - const { initConsoleInterceptor } = await import("@/lib/consoleInterceptor"); - initConsoleInterceptor(); + await Promise.all([ + import("@/lib/env/runtimeEnv").then(({ enforceWebRuntimeEnv }) => enforceWebRuntimeEnv()), + import("@/lib/usage/migrations"), + import("@/lib/consoleInterceptor").then(({ initConsoleInterceptor }) => initConsoleInterceptor()), + ]); // Clear stale transient connection cooldowns persisted from an unclean crash. // A crash mid-burst can leave far-future `rate_limited_until` values in the DB @@ -456,118 +453,90 @@ export async function registerNodejs(): Promise { void warmModelCatalogCache(); if (!isBackgroundServicesDisabled()) { - try { - const { bootstrapEmbeddedServices } = await import("@/lib/services/bootstrap"); - await bootstrapEmbeddedServices(); - console.log("[STARTUP] Embedded services bootstrap complete"); - } catch (err: unknown) { - const msg = err instanceof Error ? err.message : String(err); - console.warn("[STARTUP] Embedded services bootstrap failed (non-fatal):", msg); - } + // All services are independent — run in parallel for faster cold start. + await Promise.allSettled([ + import("@/lib/services/bootstrap").then(async (m) => { + await m.bootstrapEmbeddedServices(); + console.log("[STARTUP] Embedded services bootstrap complete"); + }).catch((err: unknown) => { + const msg = err instanceof Error ? err.message : String(err); + console.warn("[STARTUP] Embedded services bootstrap failed (non-fatal):", msg); + }), - try { - const { initEmbedWsProxy } = await import("@/lib/services/embedWsProxy"); - initEmbedWsProxy(); - } catch (err: unknown) { - const msg = err instanceof Error ? err.message : String(err); - console.warn("[STARTUP] Embed WS proxy failed to start (non-fatal):", msg); - } + import("@/lib/services/embedWsProxy").then((m) => m.initEmbedWsProxy()) + .catch((err: unknown) => { + const msg = err instanceof Error ? err.message : String(err); + console.warn("[STARTUP] Embed WS proxy failed to start (non-fatal):", msg); + }), - try { - const { autoRefreshDaemon } = await import("@omniroute/open-sse/services/autoRefreshDaemon"); - autoRefreshDaemon.start(); - } catch (err: unknown) { - const msg = err instanceof Error ? err.message : String(err); - console.warn("[STARTUP] Auto-refresh daemon failed to start (non-fatal):", msg); - } + import("@omniroute/open-sse/services/autoRefreshDaemon").then((m) => m.autoRefreshDaemon.start()) + .catch((err: unknown) => { + const msg = err instanceof Error ? err.message : String(err); + console.warn("[STARTUP] Auto-refresh daemon failed to start (non-fatal):", msg); + }), - // Proactive connection-cooldown recovery (#8): re-validate connections whose - // transient `rate_limited_until` window has elapsed OUTSIDE the request hot - // path, so the first request after a cooldown does not pay the probe latency. - // Lazy/self-recovery still happens in getProviderCredentials; this front-runs it. - try { - const { initConnectionRecoveryScheduler } = await import("@/lib/quota/connectionRecovery"); - initConnectionRecoveryScheduler(); - } catch (err: unknown) { - const msg = err instanceof Error ? err.message : String(err); - console.warn("[STARTUP] Connection recovery scheduler failed to start (non-fatal):", msg); - } + // Proactive connection-cooldown recovery (#8): re-validate connections whose + // transient `rate_limited_until` window has elapsed OUTSIDE the request hot path, + // so the first request after a cooldown does not pay the probe latency. + import("@/lib/quota/connectionRecovery").then((m) => m.initConnectionRecoveryScheduler()) + .catch((err: unknown) => { + const msg = err instanceof Error ? err.message : String(err); + console.warn("[STARTUP] Connection recovery scheduler failed to start (non-fatal):", msg); + }), - try { // Arena ELO sync: model intelligence from the Arena AI leaderboard, powering the - // Free Provider Rankings page. On by default; configurable from Dashboard Feature Flags. - // Non-blocking — the initial sync is fire-and-forget and never fatal. - const { initArenaEloSync } = await import("@/lib/arenaEloSync"); - const started = await initArenaEloSync(); - if (started) { - console.log("[STARTUP] Arena ELO sync initialized"); - } - } catch (err: unknown) { - const msg = err instanceof Error ? err.message : String(err); - console.warn("[STARTUP] Arena ELO sync failed to start (non-fatal):", msg); - } + // Free Provider Rankings page. On by default; non-blocking, never fatal. + import("@/lib/arenaEloSync").then(async (m) => { + const started = await m.initArenaEloSync(); + if (started) console.log("[STARTUP] Arena ELO sync initialized"); + }).catch((err: unknown) => { + const msg = err instanceof Error ? err.message : String(err); + console.warn("[STARTUP] Arena ELO sync failed to start (non-fatal):", msg); + }), - // Pricing sync: opt-in external pricing data (self-gated by PRICING_SYNC_ENABLED inside - // initPricingSync). Was only wired into the unused server-init.ts, so it never ran in the - // standalone runtime even when enabled. Non-blocking, never fatal. - try { - const { initPricingSync } = await import("@/lib/pricingSync"); - await initPricingSync(); - } catch (err: unknown) { - const msg = err instanceof Error ? err.message : String(err); - console.warn("[STARTUP] Pricing sync failed to start (non-fatal):", msg); - } + // Pricing sync: opt-in external pricing data (self-gated by PRICING_SYNC_ENABLED inside + // initPricingSync). Non-blocking, never fatal. + import("@/lib/pricingSync").then((m) => m.initPricingSync()) + .catch((err: unknown) => { + const msg = err instanceof Error ? err.message : String(err); + console.warn("[STARTUP] Pricing sync failed to start (non-fatal):", msg); + }), - // models.dev capability sync: opt-in via Settings > AI (self-gated by - // settings.modelsDevSyncEnabled inside initModelsDevSync). Previously had no caller at all, - // so the toggle was inert. Non-blocking, never fatal. - try { - const { initModelsDevSync } = await import("@/lib/modelsDevSync"); - await initModelsDevSync(); - } catch (err: unknown) { - const msg = err instanceof Error ? err.message : String(err); - console.warn("[STARTUP] models.dev sync failed to start (non-fatal):", msg); - } + // models.dev capability sync: opt-in via Settings > AI (self-gated by + // settings.modelsDevSyncEnabled inside initModelsDevSync). Non-blocking, never fatal. + import("@/lib/modelsDevSync").then((m) => m.initModelsDevSync()) + .catch((err: unknown) => { + const msg = err instanceof Error ? err.message : String(err); + console.warn("[STARTUP] models.dev sync failed to start (non-fatal):", msg); + }), - // Context-window self-correction (5004): periodically reconcile provider-declared - // windows (from /models discovery) into auto:discovery overrides. Reuses already-synced - // data (no new fetch); disable via CONTEXT_WINDOW_RECONCILE_INTERVAL=0. Never fatal. - try { - const { startContextWindowReconcile } = await import("@/lib/contextWindowResolver"); - startContextWindowReconcile(); - } catch (err: unknown) { - const msg = err instanceof Error ? err.message : String(err); - console.warn("[STARTUP] context-window reconcile failed to start (non-fatal):", msg); - } + // Context-window self-correction (5004): periodically reconcile provider-declared + // windows (from /models discovery) into auto:discovery overrides. Never fatal. + import("@/lib/contextWindowResolver").then((m) => m.startContextWindowReconcile()) + .catch((err: unknown) => { + const msg = err instanceof Error ? err.message : String(err); + console.warn("[STARTUP] context-window reconcile failed to start (non-fatal):", msg); + }), - // TV6 typed memory decay: optional periodic sweep of decayed episodic memories. Doubly - // opt-in (no-op unless MEMORY_TYPED_DECAY_ENABLED=true AND - // MEMORY_TYPED_DECAY_SWEEP_INTERVAL>0). Never deletes by default. Never fatal. - try { - const { startMemoryDecaySweep } = await import("@/lib/memory/typedDecay"); - startMemoryDecaySweep(); - } catch (err: unknown) { - const msg = err instanceof Error ? err.message : String(err); - console.warn("[STARTUP] memory decay sweep failed to start (non-fatal):", msg); - } + // TV6 typed memory decay: optional periodic sweep of decayed episodic memories. + // Doubly opt-in (no-op unless MEMORY_TYPED_DECAY_ENABLED=true AND + // MEMORY_TYPED_DECAY_SWEEP_INTERVAL>0). Never deletes by default. Never fatal. + import("@/lib/memory/typedDecay").then((m) => m.startMemoryDecaySweep()) + .catch((err: unknown) => { + const msg = err instanceof Error ? err.message : String(err); + console.warn("[STARTUP] memory decay sweep failed to start (non-fatal):", msg); + }), - // Real-time dashboard WebSocket daemon (port 20132): powers Combo Studio Live, - // the Home live-pulse, and Live Compression. liveServer.ts auto-starts the - // daemon on import (gated by OMNIROUTE_ENABLE_LIVE_WS, default ON) — but NOTHING - // imported it in the packaged standalone/PM2 runtime. Only the unused - // `server-init.ts` and a dev-only helper script (`scripts/start-ws-server.mjs`) - // ever pulled it into a module graph, so in the published `omniroute` bin the - // daemon never bound its port and every live dashboard reported "Live disabled — - // WebSocket disconnected". Importing it here (the instrumentation hook that DOES - // run in standalone) fires that flag-gated auto-start. Side-effect import + the - // module's own `.catch` keep it non-fatal. - try { - await import("@/server/ws/liveServer"); - console.log("[STARTUP] Live dashboard WebSocket daemon bootstrap invoked"); - } catch (err: unknown) { - const msg = err instanceof Error ? err.message : String(err); - console.warn("[STARTUP] Live dashboard WebSocket daemon failed to start (non-fatal):", msg); - } + // Real-time dashboard WebSocket daemon (port 20132): powers Combo Studio Live, + // the Home live-pulse, and Live Compression. Side-effect import triggers the + // flag-gated auto-start (OMNIROUTE_ENABLE_LIVE_WS, default ON). + import("@/server/ws/liveServer").then(() => { + console.log("[STARTUP] Live dashboard WebSocket daemon bootstrap invoked"); + }).catch((err: unknown) => { + const msg = err instanceof Error ? err.message : String(err); + console.warn("[STARTUP] Live dashboard WebSocket daemon failed to start (non-fatal):", msg); + }), + ]); } markServerReady(); diff --git a/src/lib/db/core.ts b/src/lib/db/core.ts index d958ead15b..24703387df 100644 --- a/src/lib/db/core.ts +++ b/src/lib/db/core.ts @@ -1190,6 +1190,19 @@ export function getDbInstance(): SqliteDatabase { applyStoredDatabaseOptimizationSettings(db); + // Apply mmap_size from stored settings (migration 046), fallback to 256MiB + try { + const mmapRow = db + .prepare("SELECT value FROM key_value WHERE namespace = ? AND key = ?") + .get("databaseSettings", "mmapSize") as { value: string } | undefined; + const mmapSize = mmapRow ? Math.max(0, parseInt(mmapRow.value, 10) || 0) : 268435456; + if (mmapSize > 0) { + db.pragma(`mmap_size = ${mmapSize}`); + } + } catch { + // mmap_size is best-effort; not available in all runtimes (e.g. web) + } + offloadLegacyCallLogDetails(db); // Auto-migrate from db.json if exists diff --git a/src/lib/db/providers.ts b/src/lib/db/providers.ts index 3c37e2040e..f5cd1c3c6f 100644 --- a/src/lib/db/providers.ts +++ b/src/lib/db/providers.ts @@ -868,6 +868,36 @@ export async function touchConnectionLastUsed( }); } +/** + * Lightweight backoff reset — runs a targeted UPDATE without SELECT or re-encrypt. + * Follows the `clearConnectionErrorIfUnchanged` pattern but without the CAS check, + * since the caller already verified the connection is eligible for reset. + * Resets all backoff/error columns so the connection re-enters the selection pool. + * Does invalidateDbCache + bumpProxyConfigGeneration since backoff affects priority. + */ +export async function resetConnectionBackoff(id: string): Promise { + if (!id) return; + const db = getDbInstance() as unknown as DbLike; + const now = new Date().toISOString(); + db.prepare( + `UPDATE provider_connections SET + backoff_level = 0, + test_status = 'active', + last_error = NULL, + last_error_at = NULL, + last_error_type = NULL, + last_error_source = NULL, + error_code = NULL, + updated_at = @updatedAt + WHERE id = @id` + ).run({ + updatedAt: now, + id, + }); + invalidateDbCache("connections"); + bumpProxyConfigGeneration(); +} + export async function deleteProviderConnection(id: string) { const db = getDbInstance() as unknown as DbLike; const existing = db.prepare("SELECT provider FROM provider_connections WHERE id = ?").get(id); @@ -975,102 +1005,10 @@ export async function getDistinctGroups(): Promise { return rows.map((r) => String(r.group ?? "")).filter(Boolean); } -// ──────────────── Auto Migration ──────────────── - -/** - * Scans all connections and re-encrypts any fields using the old dynamic salt - * so they use the new canonical static salt. - */ -export function autoMigrateLegacyEncryptedConnections(): number { - const db = getDbInstance() as unknown as DbLike; - const rows = db.prepare("SELECT * FROM provider_connections").all(); - let migratedCount = 0; - - for (const row of rows) { - const camelRow = rowToCamel(row); - if (!camelRow) continue; - - let updatedRow = false; - - const encryptedFields = ["apiKey", "idToken", "accessToken", "refreshToken"]; - for (const field of encryptedFields) { - if (typeof camelRow[field] === "string") { - const { updated, value } = migrateLegacyEncryptedString(camelRow[field] as string); - if (updated) { - camelRow[field] = value; - updatedRow = true; - } - } - } - - if (updatedRow) { - // camelRow[field] is already re-encrypted! - // But _updateConnectionRow does not re-encrypt automatically, so we pass it safely. - // Wait, _updateConnectionRow runs the full data through `encryptConnectionFields`, - // but `encryptConnectionFields` will re-encrypt plain text. - // BUT `migrateLegacyEncryptedString` returns ALREADY ENCRYPTED ciphertext! - // Wait... if we pass ALREADY ENCRYPTED text to `_updateConnectionRow`, - // `encryptConnectionFields` in `_updateConnectionRow` will encrypt it AGAIN! - // Let's modify the DB directly so we don't double encrypt. - - db.prepare( - "UPDATE provider_connections SET api_key = @apiKey, id_token = @idToken, access_token = @accessToken, refresh_token = @refreshToken, updated_at = @updatedAt WHERE id = @id" - ).run({ - id: camelRow.id, - apiKey: camelRow.apiKey ?? null, - idToken: camelRow.idToken ?? null, - accessToken: camelRow.accessToken ?? null, - refreshToken: camelRow.refreshToken ?? null, - updatedAt: new Date().toISOString(), - }); - migratedCount++; - } - } - - if (migratedCount > 0) { - backupDbFile("pre-write"); - invalidateDbCache("connections"); - console.log(`[DB] Auto-migrated ${migratedCount} connection(s) to new static-salt encryption.`); - } - - return migratedCount; -} - -export function getGheCopilotHosts(): string[] { - const hosts = new Set(); - try { - const db = getDbInstance(); - const rows = db - .prepare( - "SELECT provider_specific_data FROM provider_connections WHERE provider = 'ghe-copilot' AND is_active = 1" - ) - .all() as { provider_specific_data: string | null }[]; - for (const row of rows) { - if (!row.provider_specific_data) continue; - try { - const psd = JSON.parse(row.provider_specific_data); - const urls = [psd.gheUrl, psd.copilotApiUrl, psd.copilotProxyUrl]; - for (const urlStr of urls) { - if (typeof urlStr === "string" && urlStr.trim()) { - try { - const url = new URL(urlStr); - if (url.hostname) { - hosts.add(url.hostname.toLowerCase()); - } - } catch { - // Ignore invalid URLs - } - } - } - } catch { - // Ignore JSON parse errors - } - } - } catch (err) { - console.error("[DB] getGheCopilotHosts: failed to read GHE Copilot connections", err); - } - return [...hosts]; -} +export { + autoMigrateLegacyEncryptedConnections, + getGheCopilotHosts, +} from "./providers/migrations"; // ──────────────── Re-exports from leaf modules ──────────────── diff --git a/src/lib/db/providers/migrations.ts b/src/lib/db/providers/migrations.ts new file mode 100644 index 0000000000..ac216a5545 --- /dev/null +++ b/src/lib/db/providers/migrations.ts @@ -0,0 +1,120 @@ +/** + * db/providers/migrations — Connection-level migration utilities and GHE Copilot host discovery. + * + * Split from provider.ts to keep the main CRUD file under the size ratchet. + */ + +import { getDbInstance, rowToCamel } from "../core"; +import { backupDbFile } from "../backup"; +import { migrateLegacyEncryptedString } from "../encryption"; +import { invalidateDbCache } from "../readCache"; + +interface StatementLike { + all: (...params: unknown[]) => TRow[]; + get: (...params: unknown[]) => TRow | undefined; + run: (...params: unknown[]) => { changes?: number }; +} + +interface DbLike { + prepare: (sql: string) => StatementLike; + transaction: (fn: () => T) => () => T; +} + +// ──────────────── Auto Migration ──────────────── + +/** + * Scans all connections and re-encrypts any fields using the old dynamic salt + * so they use the new canonical static salt. + */ +export function autoMigrateLegacyEncryptedConnections(): number { + const db = getDbInstance() as unknown as DbLike; + const rows = db.prepare("SELECT * FROM provider_connections").all(); + let migratedCount = 0; + + for (const row of rows) { + const camelRow = rowToCamel(row); + if (!camelRow) continue; + + let updatedRow = false; + + const encryptedFields = ["apiKey", "idToken", "accessToken", "refreshToken"]; + for (const field of encryptedFields) { + if (typeof camelRow[field] === "string") { + const { updated, value } = migrateLegacyEncryptedString(camelRow[field] as string); + if (updated) { + camelRow[field] = value; + updatedRow = true; + } + } + } + + if (updatedRow) { + // camelRow[field] is already re-encrypted! + // But _updateConnectionRow does not re-encrypt automatically, so we pass it safely. + // Wait, _updateConnectionRow runs the full data through `encryptConnectionFields`, + // but `encryptConnectionFields` will re-encrypt plain text. + // BUT `migrateLegacyEncryptedString` returns ALREADY ENCRYPTED ciphertext! + // Wait... if we pass ALREADY ENCRYPTED text to `_updateConnectionRow`, + // `encryptConnectionFields` in `_updateConnectionRow` will encrypt it AGAIN! + // Let's modify the DB directly so we don't double encrypt. + + db.prepare( + "UPDATE provider_connections SET api_key = @apiKey, id_token = @idToken, access_token = @accessToken, refresh_token = @refreshToken, updated_at = @updatedAt WHERE id = @id" + ).run({ + id: camelRow.id, + apiKey: camelRow.apiKey ?? null, + idToken: camelRow.idToken ?? null, + accessToken: camelRow.accessToken ?? null, + refreshToken: camelRow.refreshToken ?? null, + updatedAt: new Date().toISOString(), + }); + migratedCount++; + } + } + + if (migratedCount > 0) { + backupDbFile("pre-write"); + invalidateDbCache("connections"); + console.log(`[DB] Auto-migrated ${migratedCount} connection(s) to new static-salt encryption.`); + } + + return migratedCount; +} + +// ──────────────── GHE Copilot ──────────────── + +export function getGheCopilotHosts(): string[] { + const hosts = new Set(); + try { + const db = getDbInstance(); + const rows = db + .prepare( + "SELECT provider_specific_data FROM provider_connections WHERE provider = 'ghe-copilot' AND is_active = 1" + ) + .all() as { provider_specific_data: string | null }[]; + for (const row of rows) { + if (!row.provider_specific_data) continue; + try { + const psd = JSON.parse(row.provider_specific_data); + const urls = [psd.gheUrl, psd.copilotApiUrl, psd.copilotProxyUrl]; + for (const urlStr of urls) { + if (typeof urlStr === "string" && urlStr.trim()) { + try { + const url = new URL(urlStr); + if (url.hostname) { + hosts.add(url.hostname.toLowerCase()); + } + } catch { + // Ignore invalid URLs + } + } + } + } catch { + // Ignore JSON parse errors + } + } + } catch (err) { + console.error("[DB] getGheCopilotHosts: failed to read GHE Copilot connections", err); + } + return [...hosts]; +} diff --git a/src/lib/db/schemaColumns.ts b/src/lib/db/schemaColumns.ts index 04898d5d4f..d9334f05f5 100644 --- a/src/lib/db/schemaColumns.ts +++ b/src/lib/db/schemaColumns.ts @@ -87,6 +87,9 @@ export function ensureProviderConnectionsColumns(db: SqliteDatabase) { db.exec( "CREATE INDEX IF NOT EXISTS idx_pc_auth_active_refresh ON provider_connections(auth_type, is_active, refresh_token)" ); + db.exec( + "CREATE INDEX IF NOT EXISTS idx_pc_provider_auth_type ON provider_connections(provider, auth_type)" + ); } catch (error: unknown) { const message = error instanceof Error ? error.message : String(error); console.warn("[DB] Failed to verify provider_connections schema:", message); @@ -147,6 +150,20 @@ export function ensureUsageHistoryColumns(db: SqliteDatabase) { db.exec("ALTER TABLE usage_history ADD COLUMN account_label_priority INTEGER DEFAULT 0"); console.log("[DB] Added usage_history.account_label_priority column"); } + db.exec( + "CREATE INDEX IF NOT EXISTS idx_uh_provider_model_timestamp ON usage_history(provider, model, timestamp)" + ); + db.exec( + `CREATE INDEX IF NOT EXISTS idx_uh_dedup ON usage_history( + timestamp, + COALESCE(provider, ''), + COALESCE(model, ''), + COALESCE(connection_id, ''), + COALESCE(api_key_id, ''), + tokens_input, + tokens_output + )` + ); } catch (error: unknown) { const message = error instanceof Error ? error.message : String(error); console.warn("[DB] Failed to verify usage_history schema:", message); diff --git a/src/lib/db/settings.ts b/src/lib/db/settings.ts index 9889998c73..7cc6f499eb 100644 --- a/src/lib/db/settings.ts +++ b/src/lib/db/settings.ts @@ -768,3 +768,4 @@ export { getCacheTrend, resetCacheMetrics, } from "./settings/cacheMetrics"; +export { getCachedSettings } from "./readCache"; diff --git a/src/lib/localDb.ts b/src/lib/localDb.ts index 4ec1a7e18b..82b041f621 100755 --- a/src/lib/localDb.ts +++ b/src/lib/localDb.ts @@ -13,6 +13,7 @@ export { getProviderConnectionById, createProviderConnection, updateProviderConnection, + resetConnectionBackoff, clearConnectionErrorIfUnchanged, touchConnectionLastUsed, deleteProviderConnection, @@ -32,7 +33,6 @@ export { isConnectionRateLimited, getRateLimitedConnections, clearStaleCrashCooldowns, - // T13: Stale quota display fix (zero out usage after window resets) getEffectiveQuotaUsage, formatResetCountdown, diff --git a/src/lib/proxyEgress.ts b/src/lib/proxyEgress.ts index b679de521d..6f8b10e81a 100644 --- a/src/lib/proxyEgress.ts +++ b/src/lib/proxyEgress.ts @@ -79,11 +79,15 @@ export function clearEgressCache(): void { * echo-IP round-trip. Returns null if not yet probed. */ export function getCachedEgressIp(proxyUrl: string | null): string | null { - const cached = egressCache.get(proxyUrl ?? "__direct__"); + const key = proxyUrl ?? "__direct__"; + const cached = egressCache.get(key); if (!cached) return null; - if (Date.now() - cached.at >= EGRESS_CACHE_TTL_MS) return null; + if (Date.now() - cached.at >= EGRESS_CACHE_TTL_MS) { + egressCache.delete(key); + return null; + } return cached.ip; -} + } const warmingInFlight = new Set(); diff --git a/src/lib/settingsCache.ts b/src/lib/settingsCache.ts deleted file mode 100644 index 9a88251809..0000000000 --- a/src/lib/settingsCache.ts +++ /dev/null @@ -1,62 +0,0 @@ -/** - * Settings Cache — FASE-03 Architecture Refactoring - * - * In-memory cache for settings to eliminate self-fetch anti-pattern in middleware. - * The middleware was making HTTP requests to its own /api/settings endpoint, - * which caused circular dependencies and performance issues. - * - * @module settingsCache - */ - -import { getSettings } from "@/lib/localDb"; - -/** @type {{ data: object|null, lastFetch: number, ttl: number }} */ -const cache = { - data: null, - lastFetch: 0, - ttl: 5000, // 5 seconds TTL -}; - -/** - * Get settings from cache (or refresh if stale). - * This replaces the self-fetch pattern in middleware. - * - * @returns {Promise} Settings object - */ -export async function getCachedSettings() { - const now = Date.now(); - - if (cache.data && now - cache.lastFetch < cache.ttl) { - return cache.data; - } - - try { - const settings = await getSettings(); - cache.data = settings; - cache.lastFetch = now; - return settings; - } catch (err) { - // If fetch fails but we have stale data, return it - if (cache.data) { - console.error("[SettingsCache] Failed to refresh, using stale data:", err.message); - return cache.data; - } - throw err; - } -} - -/** - * Invalidate the cache (e.g. after settings update). - */ -export function invalidateSettingsCache() { - cache.data = null; - cache.lastFetch = 0; -} - -/** - * Set the cache TTL in milliseconds. - * @param {number} ms - */ -export function setSettingsCacheTTL(ms) { - cache.ttl = ms; -} diff --git a/src/sse/services/auth.ts b/src/sse/services/auth.ts index dca2d10ad1..1f685cc261 100644 --- a/src/sse/services/auth.ts +++ b/src/sse/services/auth.ts @@ -6,10 +6,11 @@ import { getCachedProviderNodes, validateApiKey, updateProviderConnection, - touchConnectionLastUsed, - clearConnectionErrorIfUnchanged, + resetConnectionBackoff, getSettings, getCachedSettings, + touchConnectionLastUsed, + clearConnectionErrorIfUnchanged, } from "@/lib/localDb"; import { createLazyConnectionView, @@ -594,15 +595,22 @@ function getConnectionRecencyPenalty(connection: ProviderConnectionView): number function getP2CConnectionScore( provider: string, connection: ProviderConnectionView, - requestedModel: string | null = null + requestedModel: string | null = null, + quotaResults?: Map ): { score: number; quotaHeadroomPercent: number | null } { - const quotaBlocked = evaluateQuotaLimitPolicy(provider, connection, requestedModel).blocked; - const quotaExhausted = isQuotaExhaustedForRequest(connection.id, provider, requestedModel); - const quotaHeadroomPercent = getConnectionQuotaHeadroomPercent( - provider, - connection, - requestedModel - ); + let quotaBlocked: boolean; + let quotaExhausted: boolean; + + if (connection.id && quotaResults?.has(connection.id)) { + const cached = quotaResults.get(connection.id)!; + quotaBlocked = cached.blocked; + quotaExhausted = cached.exhausted; + } else { + quotaBlocked = evaluateQuotaLimitPolicy(provider, connection, requestedModel).blocked; + quotaExhausted = isQuotaExhaustedForRequest(connection.id, provider, requestedModel); + } + + const quotaHeadroomPercent = getConnectionQuotaHeadroomPercent(provider, connection, requestedModel); let quotaPenalty = 0; if (quotaHeadroomPercent !== null) { @@ -630,10 +638,11 @@ function compareP2CConnections( provider: string, a: ProviderConnectionView, b: ProviderConnectionView, - requestedModel: string | null = null + requestedModel: string | null = null, + quotaResults?: Map ): number { - const aScore = getP2CConnectionScore(provider, a, requestedModel); - const bScore = getP2CConnectionScore(provider, b, requestedModel); + const aScore = getP2CConnectionScore(provider, a, requestedModel, quotaResults); + const bScore = getP2CConnectionScore(provider, b, requestedModel, quotaResults); if (aScore.score !== bScore.score) { return aScore.score - bScore.score; } @@ -1146,32 +1155,39 @@ export async function getProviderCredentials( !isAccountUnavailable(c.rateLimitedUntil) ) { c.backoffLevel = 0; - updateProviderConnection(c.id, { - backoffLevel: 0, - testStatus: "active", - lastError: null, - lastErrorAt: null, - lastErrorType: null, - lastErrorSource: null, - errorCode: null, - }).catch(() => {}); + resetConnectionBackoff(c.id).catch(() => {}); } } let modelLockedCount = 0; let familyLockedCount = 0; + const connectionFilterStatus = new Map(); // Filter out unavailable accounts and excluded connection const availableConnections = connections.filter((c) => { - if (excludedConnectionIds.has(c.id)) return false; + if (excludedConnectionIds.has(c.id)) { + connectionFilterStatus.set(c.id, "excluded"); + return false; + } if (requestedModel && isModelExcludedByConnection(requestedModel, c.providerSpecificData)) { + connectionFilterStatus.set(c.id, "modelExcluded"); return false; } if (!allowSuppressedConnections) { - if (!allowRateLimitedConnections && isAccountUnavailable(c.rateLimitedUntil)) return false; - if (isTerminalConnectionStatus(c)) return false; - if (provider === "codex" && isCodexScopeUnavailable(c, requestedModel)) return false; + if (!allowRateLimitedConnections && isAccountUnavailable(c.rateLimitedUntil)) { + connectionFilterStatus.set(c.id, "rateLimited"); + return false; + } + if (isTerminalConnectionStatus(c)) { + connectionFilterStatus.set(c.id, "terminalStatus"); + return false; + } + if (provider === "codex" && isCodexScopeUnavailable(c, requestedModel)) { + connectionFilterStatus.set(c.id, "codexScopeLimited"); + return false; + } // Per-model lockout: if this specific model/family is locked on this connection, skip it if (requestedModel && isModelLocked(provider, c.id, requestedModel)) { + connectionFilterStatus.set(c.id, "modelLocked"); if ( provider === "antigravity" && getQuotaScopeLabelForProvider(provider, requestedModel) === "family" @@ -1183,6 +1199,7 @@ export async function getProviderCredentials( return false; } } + connectionFilterStatus.set(c.id, "available"); return true; }); @@ -1197,15 +1214,13 @@ export async function getProviderCredentials( ); } connections.forEach((c) => { - const excluded = excludedConnectionIds.has(c.id); - const rateLimited = isAccountUnavailable(c.rateLimitedUntil); - const terminalStatus = isTerminalConnectionStatus(c); - const codexScopeLimited = provider === "codex" && isCodexScopeUnavailable(c, requestedModel); - const modelLocked = - Boolean(requestedModel) && isModelLocked(provider, c.id, requestedModel as string); - const modelExcluded = - Boolean(requestedModel) && - isModelExcludedByConnection(requestedModel as string, c.providerSpecificData); + const status = connectionFilterStatus.get(c.id); + const excluded = status === "excluded"; + const rateLimited = status === "rateLimited"; + const terminalStatus = status === "terminalStatus"; + const codexScopeLimited = status === "codexScopeLimited"; + const modelLocked = status === "modelLocked"; + const modelExcluded = status === "modelExcluded"; if (excluded || rateLimited) { log.debug( "AUTH", @@ -1334,10 +1349,12 @@ export async function getProviderCredentials( reasons: string[]; resetAt: string | null; }> = []; + const quotaResults = new Map(); if (!bypassQuotaPolicy) { policyEligibleConnections = availableConnections.filter((connection) => { const evaluation = evaluateQuotaLimitPolicy(provider, connection, requestedModel); + quotaResults.set(connection.id, { blocked: evaluation.blocked, exhausted: false }); if (!evaluation.blocked) return true; blockedByPolicy.push({ @@ -1377,13 +1394,19 @@ export async function getProviderCredentials( }; } - // Quota-aware: filter out accounts with exhausted quota for the requested scope. - const withQuota = policyEligibleConnections.filter( - (c) => !isQuotaExhaustedForRequest(c.id, provider, requestedModel) - ); - const exhaustedQuota = policyEligibleConnections.filter((c) => - isQuotaExhaustedForRequest(c.id, provider, requestedModel) - ); + // Quota-aware: partition accounts with and without quota for the requested scope. + const withQuota: typeof policyEligibleConnections = []; + const exhaustedQuota: typeof policyEligibleConnections = []; + for (const c of policyEligibleConnections) { + const exhausted = isQuotaExhaustedForRequest(c.id, provider, requestedModel); + const existing = quotaResults.get(c.id); + if (existing) existing.exhausted = exhausted; + if (!exhausted) { + withQuota.push(c); + } else { + exhaustedQuota.push(c); + } + } if (exhaustedQuota.length > 0) { log.info( @@ -1551,7 +1574,7 @@ export async function getProviderCredentials( // health instead of defaulting to random-first selection. if (candidatePool.length <= 2) { connection = [...candidatePool].sort((a, b) => - compareP2CConnections(provider, a, b, requestedModel) + compareP2CConnections(provider, a, b, requestedModel, quotaResults) )[0]; } else { const i = @@ -1561,7 +1584,7 @@ export async function getProviderCredentials( if (j >= i) j++; const a = candidatePool[i]; const b = candidatePool[j]; - connection = compareP2CConnections(provider, a, b, requestedModel) <= 0 ? a : b; + connection = compareP2CConnections(provider, a, b, requestedModel, quotaResults) <= 0 ? a : b; } } else if (strategy === "random") { // Random: Fisher-Yates-inspired random pick diff --git a/src/types/databaseSettings.ts b/src/types/databaseSettings.ts index 459f580334..bb661147eb 100644 --- a/src/types/databaseSettings.ts +++ b/src/types/databaseSettings.ts @@ -126,7 +126,7 @@ export const DEFAULT_DATABASE_SETTINGS: Omit { // or it never runs in production. initArenaEloSync self-gates through the feature flag // resolver so ARENA_ELO_SYNC_ENABLED and dashboard overrides still apply. assert.match(src, /initArenaEloSync/); - assert.match(src, /const started = await initArenaEloSync\(\)/); + assert.match(src, /m\.initArenaEloSync\(\)/); }); it("should initialize pricing + models.dev sync on the live startup path (self-gated, opt-in)", () => { diff --git a/tests/unit/database-settings-maintenance.test.ts b/tests/unit/database-settings-maintenance.test.ts index b8dbf20d6f..a6c3d69e83 100644 --- a/tests/unit/database-settings-maintenance.test.ts +++ b/tests/unit/database-settings-maintenance.test.ts @@ -218,7 +218,7 @@ test("database settings reader normalizes legacy negative cache size to the posi "INSERT OR REPLACE INTO key_value (namespace, key, value) VALUES ('databaseSettings', ?, ?)" ).run("optimization.cacheSize", JSON.stringify(-2000)); - assert.equal(databaseSettings.getUserDatabaseSettings().optimization.cacheSize, 16384); + assert.equal(databaseSettings.getUserDatabaseSettings().optimization.cacheSize, 65536); }); test("purgeDetailedLogs deletes request_detail_logs", async () => { diff --git a/tests/unit/reset-connection-backoff.test.ts b/tests/unit/reset-connection-backoff.test.ts new file mode 100644 index 0000000000..0fce46fb03 --- /dev/null +++ b/tests/unit/reset-connection-backoff.test.ts @@ -0,0 +1,105 @@ +import test from "node:test"; +import assert from "node:assert/strict"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; + +// resetConnectionBackoff (open-sse perf PR #7893) is a lightweight-UPDATE variant +// of the CAS-based clearConnectionErrorIfUnchanged pattern: it resets the backoff +// and error columns without a preceding SELECT/re-encrypt/backup pass. It had zero +// direct test coverage prior to this file — regression guard added per Hard Rule #8. + +const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-reset-backoff-")); +process.env.DATA_DIR = TEST_DATA_DIR; +process.env.API_KEY_SECRET = "test-reset-connection-backoff-secret"; + +const core = await import("../../src/lib/db/core.ts"); +const providersDb = await import("../../src/lib/db/providers.ts"); + +test.after(() => { + core.resetDbInstance(); + fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true }); +}); + +async function createBackedOffConnection() { + const created = await providersDb.createProviderConnection({ + provider: "glm", + authType: "apikey", + name: `GLM Backoff ${Date.now()}-${Math.random()}`, + apiKey: "glm-test-key", + }); + const connectionId = (created as { id: string }).id; + // backoffLevel/error fields aren't accepted at creation time — set them via + // updateProviderConnection to reach the "backed off" state under test, mirroring + // the pattern in tests/unit/provider-limits-recovery.test.ts. + await providersDb.updateProviderConnection(connectionId, { + testStatus: "unavailable", + lastError: "rate limit exceeded", + lastErrorType: "rate_limited", + lastErrorSource: "executor", + errorCode: 429, + backoffLevel: 3, + }); + return created; +} + +test("resetConnectionBackoff clears backoff/error columns and re-activates the connection", async () => { + const created = await createBackedOffConnection(); + const connectionId = (created as { id: string }).id; + + const before = (await providersDb.getProviderConnectionById(connectionId)) as Record< + string, + unknown + >; + assert.equal(before.testStatus, "unavailable"); + assert.equal(before.backoffLevel, 3); + + await providersDb.resetConnectionBackoff(connectionId); + + const after = (await providersDb.getProviderConnectionById(connectionId)) as Record< + string, + unknown + >; + assert.equal(after.testStatus, "active"); + assert.equal(after.backoffLevel, 0); + assert.equal(after.lastError, undefined); + assert.equal(after.lastErrorType, undefined); + assert.equal(after.errorCode, undefined); +}); + +test("resetConnectionBackoff does not clear a terminal status (e.g. banned)", async () => { + const created = await providersDb.createProviderConnection({ + provider: "glm", + authType: "apikey", + name: `GLM Banned ${Date.now()}-${Math.random()}`, + apiKey: "glm-test-key", + }); + const connectionId = (created as { id: string }).id; + await providersDb.updateProviderConnection(connectionId, { + testStatus: "banned", + lastError: "account banned", + lastErrorType: "banned", + backoffLevel: 5, + }); + + // resetConnectionBackoff is a targeted UPDATE with no CAS/status guard — callers + // are responsible for only invoking it on connections eligible for reset. This + // test documents that unconditional behavior: it WILL flip a terminal status too, + // which is why callers must gate the call themselves (see comment on the fn). + await providersDb.resetConnectionBackoff(connectionId); + + const after = (await providersDb.getProviderConnectionById(connectionId)) as Record< + string, + unknown + >; + assert.equal(after.testStatus, "active"); + assert.equal(after.backoffLevel, 0); +}); + +test("resetConnectionBackoff is a no-op for an empty id", async () => { + await assert.doesNotReject(() => providersDb.resetConnectionBackoff("")); +}); + +test("resetConnectionBackoff is a no-op for an unknown id (no throw)", async () => { + await assert.doesNotReject(() => providersDb.resetConnectionBackoff("nonexistent-connection-id")); +});