mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-08-26 00:52:18 +03:00
fix(cache): add latency marker + per-key bypass for semantic cache (#8984)
* fix(cache): add latency marker + per-key bypass for semantic cache
Semantic cache silently corrupts latency measurements: a 10s upstream
call served from cache looks like 19ms. Three fixes:
A. Latency marker: cache HIT responses now carry
X-OmniRoute-Cache-Latency: synthetic so measurement tools can
distinguish real vs cached latency.
B. Per-key bypass: new apiKeys.cacheDefaultMode ('legacy' | 'bypass')
lets latency-sensitive clients opt out of cache reads entirely.
- DB column + migration (134)
- rowParser parseCacheDefaultMode
- API create default + PATCH update
- checkSemanticCache returns null on bypass
C. Type safety: ApiKeyRow/ApiKeyView/params updated, superRefine
guard includes cacheDefaultMode.
Cache write path intentionally unchanged: apiKeyId is already in the
cache signature (semanticCache.ts:140), so per-key isolation prevents
cross-key pollution.
Changed test files:
- tests/unit/chatcore-semantic-cache.test.ts (3 new tests)
Signed-off-by: Minxi Hou <houminxi@gmail.com>
* docs: document semantic cache latency impact + bypass configuration
---------
Signed-off-by: Minxi Hou <houminxi@gmail.com>
This commit is contained in:
@@ -24,6 +24,7 @@ export const API_KEY_COLUMN_FALLBACKS = [
|
||||
{ name: "allowed_endpoints", definition: "allowed_endpoints TEXT" },
|
||||
{ name: "allowed_quotas", definition: "allowed_quotas TEXT NOT NULL DEFAULT '[]'" },
|
||||
{ name: "stream_default_mode", definition: "stream_default_mode TEXT NOT NULL DEFAULT 'legacy'" },
|
||||
{ name: "cache_default_mode", definition: "cache_default_mode TEXT NOT NULL DEFAULT 'legacy'" },
|
||||
{
|
||||
name: "disable_non_public_models",
|
||||
definition: "disable_non_public_models INTEGER NOT NULL DEFAULT 0",
|
||||
|
||||
Reference in New Issue
Block a user