From 4269c25a9f27f6a6ab508e78f080ea2872d52d8f Mon Sep 17 00:00:00 2001 From: diegosouzapw Date: Sun, 15 Feb 2026 02:06:14 -0300 Subject: [PATCH] docs: update CHANGELOG, README, and ARCHITECTURE for v0.3.0 release - CHANGELOG.md: comprehensive v0.3.0 entry with security hardening, domain layer, pipeline wiring, 9 API routes, frontend 100% coverage, rate limit overhaul (4 phases), ADRs, compliance, eval framework, 273+ tests - README.md: add 10 new feature rows (circuit breaker, anti-thundering herd, resilience profiles/UI, cost budgets, telemetry, correlation IDs, compliance, model availability, eval framework), update tech stack and release example - ARCHITECTURE.md: add domain layer modules, resilience modules, 11 new API routes, usageDb decomposition note, update date --- CHANGELOG.md | 109 +++++++++++++++++++++++++++++++++++++++++++ README.md | 64 ++++++++++++++----------- docs/ARCHITECTURE.md | 39 +++++++++++++++- 3 files changed, 184 insertions(+), 28 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 27e6de4bc7..7d24c9ff63 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,115 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). --- +## [0.3.0] — 2026-02-15 + +Major release: security hardening, domain layer architecture, pipeline integration, full frontend coverage, and resilience overhaul with circuit breaker, anti-thundering herd, and Resilience UI. + +### Added + +#### Security Hardening (FASE-01 to FASE-09) + +- **FASE-01 to FASE-06** — Core security hardening across authentication, input validation, and access control +- **FASE-07 to FASE-09** — Advanced features including enhanced monitoring, security audit improvements, and operational hardening + +#### Domain Layer & Infrastructure + +- **Model Availability** — TTL-based cooldown tracking per model (`modelAvailability.js`) +- **Cost Rules** — Per-API-key budget management with daily/monthly limits (`costRules.js`) +- **Fallback Policy** — Declarative fallback chain routing with CRUD API (`fallbackPolicy.js`) +- **Error Codes Catalog** — 24 standardized error codes in 6 categories with `createErrorResponse` helper (`errorCodes.js`) +- **Correlation ID** — AsyncLocalStorage-based `x-request-id` propagation for end-to-end tracing (`requestId.js`) +- **Fetch Timeout** — AbortController wrapper with configurable `FETCH_TIMEOUT_MS` (`fetchTimeout.js`) +- **Combo Resolver** — Priority/round-robin/random/least-used strategies (`comboResolver.js`) +- **Lockout Policy** — Sliding window lockout with force-unlock capability (`lockoutPolicy.js`) +- **Request Telemetry** — 7-phase lifecycle tracking with p50/p95/p99 latency aggregation (`requestTelemetry.js`) + +#### Pipeline Wiring (7 Backend Modules) + +- **Circuit Breaker** integration into request pipeline for provider resilience +- **Model Availability** wired with TTL cooldowns for per-model health tracking +- **Request Telemetry** lifecycle tracking across 7 phases +- **Cost Rules** budget check and cost recording per request +- **Compliance** audit logging with `noLog` opt-out per API key +- **Fetch Timeout** via `fetchWithTimeout` replacing bare `fetch()` in proxy +- **Request ID** (`X-Request-Id` header) for end-to-end tracing + +#### 9 New API Routes + +- `/api/cache/stats` — GET cache stats, DELETE flush +- `/api/models/availability` — GET availability report, POST clear cooldown +- `/api/telemetry/summary` — GET p50/p95/p99 latency metrics +- `/api/usage/budget` — GET cost summary, POST set budget per key +- `/api/fallback/chains` — GET/POST/DELETE fallback chain management +- `/api/compliance/audit-log` — GET filterable audit log +- `/api/evals` — GET list suites, POST run suite +- `/api/evals/[suiteId]` — GET suite details +- `/api/policies` — GET circuit breaker + lockout status, POST force-unlock + +#### Frontend — 100% Backend API Coverage (7 Batches) + +- **Batch 1** — Pipeline wiring integration verified across all backend modules +- **Batch 2** — 9 API routes created for backend module access +- **Batch 3** — 6 shared UI components exported (Breadcrumbs, EmptyState, NotificationToast, FilterBar, ColumnToggle, DataTable) + `notificationStore` wired into layout +- **Batch 4** — Usage page: BudgetTelemetryCards (latency p50/p95/p99, cache, system health); Settings page: ComplianceTab (audit log), CacheStatsCard (prompt cache + flush); Combos page: EmptyState component +- **Batch 5** — Integration-wiring tests: 44 tests across 12 suites verifying all batches +- **Batch 6** — Frontend now covers every backend API surface +- **Batch 7** — Final wiring and verification pass + +#### Refactoring & Decomposition + +- **usageDb.js** decomposed from 969 → 40 lines into 5 focused modules: `migrations.js`, `usageHistory.js`, `costCalculator.js`, `usageStats.js`, `callLogs.js` +- **handleSingleModelChat** decomposed from 183 → 80 lines with extracted helpers (`handleNoCredentials`, `safeResolveProxy`, `safeLogEvents`) +- **Shared UI primitives** extracted: FilterBar, ColumnToggle, DataTable (3230 total lines) + +#### Rate Limit Overhaul (4 Phases) + +- **Phase 1** — Provider-specific resilience profiles (OAuth vs API key), exponential backoff (5s→60s), default API limits (100 RPM, 200ms minTime) +- **Phase 2** — Circuit breaker integration in combo pipeline with `canExecute()` checks, early exit when all models are OPEN, semaphore marking for 502/503/504 +- **Phase 3** — Anti-thundering herd: mutex on `markAccountUnavailable`, auto rate-limit for API key providers with elevated defaults +- **Phase 4** — Resilience UI tab in settings with 3 cards: ProviderProfilesCard, CircuitBreakerCard (real-time, auto-refresh 5s, reset), RateLimitOverviewCard +- `/api/resilience` — GET (full state) + PATCH (save profiles) +- `/api/resilience/reset` — POST (reset breakers + cooldowns) + +#### ADRs & Quality + +- **6 Architecture Decision Records**: SQLite, Fallback Strategy, OAuth, JS+JSDoc, Single-Tenant, Translator Registry +- **Accessibility audit** — WCAG AA checker with aria-label, dialog role, alt text, label validation (`a11yAudit.js`) +- **Password Reset CLI** — Interactive admin password reset tool (`bin/reset-password.mjs`) +- **Playwright E2E specs** — Responsive viewport tests (375/768/1280) across 4 pages +- **Eval Framework** — 4 strategies (exact, contains, regex, custom) + 10-case golden set (`evalRunner.js`) +- **Compliance module** — audit_log table, `noLog` opt-out per API key, `LOG_RETENTION_DAYS` cleanup + +#### Tests + +- **63 new tests** for rate limit overhaul: error-classification, combo-circuit-breaker, thundering-herd +- **44 integration-wiring tests** across 12 suites +- **31 domain layer tests** for model availability, cost rules, error codes, request ID, fetch timeout +- **13 UX/telemetry tests** for error pages, breadcrumbs, empty states, telemetry, domain extraction +- **25 batch-B tests** for ADRs, eval framework, compliance, a11y, CLI, Playwright specs +- Total: **273+ tests passing** (up from ~144 in v0.2.0) + +#### Documentation + +- **JSDoc** coverage added to all new modules (100% exported functions documented) +- `@ts-check` added to 8 critical files + +### Fixed + +- **ESLint v10 → v9 downgrade** for `eslint-config-next` compatibility — rewrote flat config, removed `defineConfig`/`globalIgnores` (ESLint 10-only APIs) +- **Unrecoverable refresh token errors** — detect `refresh_token_reused` and similar errors, mark connections as expired requiring re-authentication +- **Record type annotation** added to `getAllFallbackChains` result +- **`.gitignore` cleanup** — added `.analysis/` and `antigravity-manager-analysis/`, whitelisted FASE docs + +### Changed + +- **Error pages** — Custom 404 and global error boundary with gradient design and dev details +- **Combo page** — Inline empty state replaced with EmptyState component +- **Layout** — Breadcrumbs rendered between Header and content, NotificationToast as global fixed overlay +- **Proxy module** — bare `fetch()` replaced with `fetchWithTimeout` (5s timeout) + `X-Request-Id` header + +--- + ## [0.2.0] — 2026-02-14 Major feature release: advanced routing services, security hardening, cost analytics dashboard, and pricing management overhaul. diff --git a/README.md b/README.md index 6e2d41a3a4..c9dba362f5 100644 --- a/README.md +++ b/README.md @@ -130,28 +130,38 @@ Default URLs: ## 💡 Key Features -| Feature | What It Does | Why It Matters | -| -------------------------------- | ------------------------------------------ | ----------------------------------- | -| 🎯 **Smart 3-Tier Fallback** | Auto-route: Subscription → Cheap → Free | Never stop coding, zero downtime | -| 📊 **Real-Time Quota Tracking** | Live token count + reset countdown | Maximize subscription value | -| 🔄 **Format Translation** | OpenAI ↔ Claude ↔ Gemini seamless | Works with any CLI tool | -| 👥 **Multi-Account Support** | Multiple accounts per provider | Load balancing + redundancy | -| 🔄 **Auto Token Refresh** | OAuth tokens refresh automatically | No manual re-login needed | -| 🎨 **Custom Combos** | Create unlimited model combinations | Tailor fallback to your needs | -| 🧩 **Custom Models** | Add any model ID to any provider | No app update needed for new models | -| 🛣️ **Dedicated Provider Routes** | Per-provider API endpoints | Direct routing, model validation | -| 🌐 **Network Proxy** | Hierarchical outbound proxy + env fallback | Works behind firewalls/VPNs | -| 📋 **Model Catalog API** | All models grouped by provider + type | Discover available models easily | -| 📝 **Request Logging** | Debug mode with full request/response logs | Troubleshoot issues easily | -| 💾 **Cloud Sync** | Sync config across devices | Same setup everywhere | -| 📊 **Usage Analytics** | Track tokens, cost, trends over time | Optimize spending | -| 🌐 **Deploy Anywhere** | Localhost, VPS, Docker, Cloudflare Workers | Flexible deployment options | -| 🛡️ **IP Allowlist/Blocklist** | Restrict API access by IP address | Security for exposed deployments | -| 🧠 **Thinking Budget** | Control reasoning token budget per model | Optimize cost vs quality | -| 💬 **System Prompt Injection** | Global system prompt for all requests | Consistent behavior across models | -| 📊 **Session Tracking** | Track active sessions with fingerprinting | Monitor connected clients | -| ⚡ **Rate Limiting** | Per-account request rate management | Prevent abuse and quota waste | -| 💰 **Model Pricing** | Per-model cost tracking and calculation | Precise usage cost analytics | +| Feature | What It Does | Why It Matters | +| ----------------------------------- | --------------------------------------------- | ----------------------------------- | +| 🎯 **Smart 3-Tier Fallback** | Auto-route: Subscription → Cheap → Free | Never stop coding, zero downtime | +| 📊 **Real-Time Quota Tracking** | Live token count + reset countdown | Maximize subscription value | +| 🔄 **Format Translation** | OpenAI ↔ Claude ↔ Gemini seamless | Works with any CLI tool | +| 👥 **Multi-Account Support** | Multiple accounts per provider | Load balancing + redundancy | +| 🔄 **Auto Token Refresh** | OAuth tokens refresh automatically | No manual re-login needed | +| 🎨 **Custom Combos** | Create unlimited model combinations | Tailor fallback to your needs | +| 🧩 **Custom Models** | Add any model ID to any provider | No app update needed for new models | +| 🛣️ **Dedicated Provider Routes** | Per-provider API endpoints | Direct routing, model validation | +| 🌐 **Network Proxy** | Hierarchical outbound proxy + env fallback | Works behind firewalls/VPNs | +| 📋 **Model Catalog API** | All models grouped by provider + type | Discover available models easily | +| 📝 **Request Logging** | Debug mode with full request/response logs | Troubleshoot issues easily | +| 💾 **Cloud Sync** | Sync config across devices | Same setup everywhere | +| 📊 **Usage Analytics** | Track tokens, cost, trends over time | Optimize spending | +| 🌐 **Deploy Anywhere** | Localhost, VPS, Docker, Cloudflare Workers | Flexible deployment options | +| 🛡️ **IP Allowlist/Blocklist** | Restrict API access by IP address | Security for exposed deployments | +| 🧠 **Thinking Budget** | Control reasoning token budget per model | Optimize cost vs quality | +| 💬 **System Prompt Injection** | Global system prompt for all requests | Consistent behavior across models | +| 📊 **Session Tracking** | Track active sessions with fingerprinting | Monitor connected clients | +| ⚡ **Rate Limiting** | Per-account request rate management | Prevent abuse and quota waste | +| 💰 **Model Pricing** | Per-model cost tracking and calculation | Precise usage cost analytics | +| 🔌 **Circuit Breaker** | Auto-open/close per-provider with cooldowns | Prevent cascading failures | +| 🛡️ **Anti-Thundering Herd** | Mutex + auto rate-limit for API key providers | Prevent parallel stampede | +| 📊 **Provider Resilience Profiles** | OAuth vs API key differentiated cooldowns | Smarter error recovery | +| 🎛️ **Resilience UI** | Real-time circuit breaker status + reset | Monitor and control resilience | +| 💵 **Cost Budgets** | Per-API-key daily/monthly budget limits | Prevent unexpected spending | +| 📈 **Request Telemetry** | 7-phase lifecycle with p50/p95/p99 latency | Performance monitoring | +| 🔍 **Correlation IDs** | End-to-end request tracing via X-Request-Id | Debug complex request flows | +| 📋 **Compliance Audit Log** | Filterable audit trail with opt-out per key | Regulatory compliance | +| 🏗️ **Model Availability** | TTL-based cooldown tracking per model | Intelligent model health tracking | +| 🔄 **Eval Framework** | 4 strategies + golden set for LLM evaluation | Quality assurance for models |
📖 Feature Details @@ -1093,11 +1103,13 @@ Types: `chat`, `embedding`, `image`. Custom models are flagged with `custom: tru - **Database**: LowDB (JSON file-based) - **Streaming**: Server-Sent Events (SSE) - **Auth**: OAuth 2.0 (PKCE) + JWT + API Keys -- **Testing**: Playwright (E2E) + Node.js test runner (unit) +- **Testing**: Playwright (E2E) + Node.js test runner (273+ unit tests) - **Monorepo**: npm workspaces (`@omniroute/open-sse`) - **CI/CD**: GitHub Actions (auto npm publish on release) + Dependabot - **Package**: [npmjs.com/package/omniroute](https://www.npmjs.com/package/omniroute) -- **Compliance**: `/terms` and `/privacy` pages +- **Compliance**: `/terms` and `/privacy` pages + audit log +- **Resilience**: Circuit breaker, exponential backoff, anti-thundering herd +- **Observability**: Request telemetry (p50/p95/p99), correlation IDs, structured error codes --- @@ -1286,11 +1298,11 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines. ### Releasing a New Version -When a new GitHub Release is created (e.g. `v0.2.0`), the package is **automatically published to npm** via GitHub Actions: +When a new GitHub Release is created (e.g. `v0.3.0`), the package is **automatically published to npm** via GitHub Actions: ```bash # Create a release — npm publish happens automatically -gh release create v0.2.0 --title "v0.2.0" --generate-notes +gh release create v0.3.0 --title "v0.3.0" --generate-notes ``` The workflow syncs the version from the release tag, builds the standalone app, and publishes to npm. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 6044b3f45d..62301316ba 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -1,6 +1,6 @@ # OmniRoute Architecture -_Last updated: 2026-02-14_ +_Last updated: 2026-02-15_ ## Executive Summary @@ -24,8 +24,16 @@ Core capabilities: - Thinking budget management (passthrough/auto/custom/adaptive) - Global system prompt injection - Session tracking and fingerprinting -- Per-account enhanced rate limiting +- Per-account enhanced rate limiting with provider-specific profiles +- Circuit breaker pattern for provider resilience +- Anti-thundering herd protection with mutex locking - Signature-based request deduplication cache +- Domain layer: model availability, cost rules, fallback policy, lockout policy +- Request telemetry with p50/p95/p99 latency aggregation +- Correlation ID (X-Request-Id) for end-to-end tracing +- Compliance audit logging with opt-out per API key +- Eval framework for LLM quality assurance +- Resilience UI dashboard with real-time circuit breaker status Primary runtime model: @@ -140,6 +148,16 @@ Management domains: - System prompt: `src/app/api/settings/system-prompt` (GET/PUT) - Sessions: `src/app/api/sessions` (GET) - Rate limits: `src/app/api/rate-limits` (GET) +- Resilience: `src/app/api/resilience` (GET/PATCH) — provider profiles, circuit breaker, rate limit state +- Resilience reset: `src/app/api/resilience/reset` (POST) — reset breakers + cooldowns +- Cache stats: `src/app/api/cache/stats` (GET/DELETE) +- Model availability: `src/app/api/models/availability` (GET/POST) +- Telemetry: `src/app/api/telemetry/summary` (GET) +- Budget: `src/app/api/usage/budget` (GET/POST) +- Fallback chains: `src/app/api/fallback/chains` (GET/POST/DELETE) +- Compliance audit: `src/app/api/compliance/audit-log` (GET) +- Evals: `src/app/api/evals` (GET/POST), `src/app/api/evals/[suiteId]` (GET) +- Policies: `src/app/api/policies` (GET/POST) ## 2) SSE + Translation Core @@ -170,6 +188,22 @@ Services (business logic): - System prompt injection: `open-sse/services/systemPrompt.js` - Thinking budget management: `open-sse/services/thinkingBudget.js` - Wildcard model routing: `open-sse/services/wildcardRouter.js` +- Rate limit management: `open-sse/services/rateLimitManager.js` +- Circuit breaker: `open-sse/services/circuitBreaker.js` + +Domain layer modules: + +- Model availability: `src/lib/domain/modelAvailability.js` +- Cost rules/budgets: `src/lib/domain/costRules.js` +- Fallback policy: `src/lib/domain/fallbackPolicy.js` +- Combo resolver: `src/lib/domain/comboResolver.js` +- Lockout policy: `src/lib/domain/lockoutPolicy.js` +- Error codes catalog: `src/lib/domain/errorCodes.js` +- Request ID: `src/lib/domain/requestId.js` +- Fetch timeout: `src/lib/domain/fetchTimeout.js` +- Request telemetry: `src/lib/domain/requestTelemetry.js` +- Compliance/audit: `src/lib/domain/compliance/index.js` +- Eval runner: `src/lib/domain/evalRunner.js` ## 3) Persistence Layer @@ -184,6 +218,7 @@ Usage DB: - `src/lib/usageDb.js` - files: `${DATA_DIR}/usage.json`, `${DATA_DIR}/log.txt`, `${DATA_DIR}/call_logs/` - follows same base directory policy as `localDb` (`DATA_DIR`, then `XDG_CONFIG_HOME/omniroute` when set) +- decomposed into focused sub-modules: `migrations.js`, `usageHistory.js`, `costCalculator.js`, `usageStats.js`, `callLogs.js` ## 4) Auth + Security Surfaces