pjandro/OmniRoute
1
0
Fork 0
mirror of https://github.com/diegosouzapw/OmniRoute.git synced 2026-08-01 21:02:12 +03:00
Code Issues Packages Projects Releases Wiki Activity
1,280 Commits 100 Branches 313 Tags
6afb314d2624f689e084bcf711eb9007c869be21
Commit Graph

3 Commits

Author SHA1 Message Date
oyi77
6780485051 feat(cache): persistent metrics, cache entry browser, settings UI, MCP tools, prefix analyzer 
Implements remaining features from #813:

Phase 1 - Persistent Metrics:
- Add cache_metrics table for persistent hit/miss tracking
- Semantic cache stats now survive server restarts

Phase 2 - Cache Entry Browser:
- /api/cache/entries endpoint with search, pagination, delete
- CacheEntriesTab component for browsing cached entries

Phase 3 - Settings UI:
- CacheSettingsTab for semantic/prompt cache configuration
- /api/settings/cache-config endpoint

Phase 4 - Prefix Analyzer:
- src/lib/promptCache/prefixAnalyzer.ts for intelligent caching
- Analyzes message arrays to find stable prefixes

Phase 5 - Provider Support:
- Added deepseek to CACHING_PROVIDERS

Phase 6 - MCP Tools:
- omniroute_cache_stats tool
- omniroute_cache_flush tool

Phase 7 - Retention:
- cleanOldMetrics() for auto-purge of old entries

Closes #813
 2026-03-30 17:32:45 -03:00
tombii
26f7b36ce4 feat: add cache control settings and token-based metrics 
Settings:
- Add `alwaysPreserveClientCache` setting with modes: auto/always/never
- UI toggle in Dashboard > Settings > Routing tab
- Auto mode preserves cache_control for Claude Code clients with deterministic routing

Metrics:
- Track prompt cache token usage (input, cached, creation)
- Display cache reuse ratio (cached/input tokens)
- Breakdown by provider and routing strategy
- Shows tokens saved and estimated cost savings

API Endpoints:
- GET /api/settings/cache-metrics - retrieve metrics
- DELETE /api/settings/cache-metrics - reset metrics

Files:
- open-sse/utils/cacheControlPolicy.ts: CacheControlMetrics interface, trackCacheMetrics, updateCacheTokenMetrics
- open-sse/handlers/chatCore.ts: Track cache tokens from provider responses
- src/lib/db/settings.ts: Database functions for metrics persistence
- src/lib/cacheControlSettings.ts: Cached settings accessor
- src/app/(dashboard)/dashboard/settings/components/CacheStatsCard.tsx: Metrics dashboard UI
- tests/unit/*.test.mjs: Unit tests (41 tests pass)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-29 14:37:55 +02:00
tombii
0bc557fb8b feat(sse): preserve client cache_control for Claude Code with deterministic routing 
Adds intelligent cache control preservation for Claude Code clients:

- New cacheControlPolicy.ts module with detection logic:
  - isClaudeCodeClient(): Detects Claude Code via User-Agent
  - providerSupportsCaching(): Checks provider (claude, anthropic, zai, qwen)
  - isDeterministicStrategy(): Identifies priority/cost-optimized strategies
  - shouldPreserveCacheControl(): Main policy decision

- Cache control is preserved when:
  1. Client is Claude Code (detected via User-Agent)
  2. Provider supports prompt caching
  3. Request routing is deterministic:
     - Single model requests (always)
     - Combo with priority or cost-optimized strategy only

- Updated translator to accept preserveCacheControl option
- Updated chatCore and chat handler to propagate combo strategy
- Added comprehensive unit tests (24 tests)

Non-deterministic combo strategies (weighted, round-robin, random, etc.)
continue to use OmniRoute's managed caching strategy.

Refs: #cache-control-preservation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-29 12:24:44 +02:00