mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-09-21 14:22:14 +03:00
d8c69b87e3
* feat(cache): implement dual-layer semantic caching with in-memory and redis vector stores Implements production-grade, configurable semantic caching for OmniRoute. - Dual-layer architecture: Layer 1 exact hash match (0 embedding latency) + Layer 2 vector cosine similarity search. - Backends: in-memory vector store with L2 normalization, LRU and TTL + Redis vector store adapter with fail-open fallback. - Embedding generation: conversation history normalization, system prompt exclusion, timeout protection. - Streaming support: serializable SSE stream synthesis ending in data: [DONE]\n\n. - Request overrides and telemetry headers: X-OmniRoute-Cache (HIT (exact) | HIT (semantic) | MISS), X-OmniRoute-Cache-Similarity, X-OmniRoute-Savings-Tokens, Cache-Control: no-cache, x-omniroute-no-cache, x-omniroute-cache-threshold, x-omniroute-cache-type, x-omniroute-cache-no-store, x-omniroute-cache-key. - Unit test coverage across dual-layer search, eviction, redis resilience, and streaming replay. * feat(cache): add default embedding client and live integration test for Lemonade server and Redis - Add embeddingBaseUrl and embeddingApiKey configuration options to SemanticCacheConfig. - Implement createDefaultEmbeddingGenerator for automatic OpenAI-compatible embedding integration. - Add live verification script scripts/ad-hoc/test-semantic-cache-lemonade.ts. - Add network-aware integration test tests/integration/semantic-cache-lemonade.test.ts for Lemonade harrier-oss-v1-0.6b and Redis vector store. * fix(cache): address CodeRabbit review recommendations on PR #1 - Multi-tenant partition isolation: support null sentinel in StoreFilter so anonymous requests cannot match authenticated entries. - Provider propagation: pass routed provider to non-streaming and streaming cache writes. - Embedding resilience: race generator with timeout promise in generateEmbeddingWithTimeout to guard against uncooperative generators. - Memory store consistency: replace older entries with identical hash on insert, and safe-guard hashToId deletion in removeEntry. - Redis store consistency: prune expired/missing entries from candidate sets during search/stats, and delete hash mapping conditionally. - Token telemetry: use managerResult.tokensSaved when cached response lacks usage data. - Clova batch timeout: add AbortSignal.timeout(FETCH_TIMEOUT_MS) to fetchClovaEmbeddingBatch. --------- Co-authored-by: John Smith <you@example.com>