fix(cache): add latency marker + per-key bypass for semantic cache (#8984)

* fix(cache): add latency marker + per-key bypass for semantic cache

Semantic cache silently corrupts latency measurements: a 10s upstream
call served from cache looks like 19ms. Three fixes:

A. Latency marker: cache HIT responses now carry
   X-OmniRoute-Cache-Latency: synthetic so measurement tools can
   distinguish real vs cached latency.

B. Per-key bypass: new apiKeys.cacheDefaultMode ('legacy' | 'bypass')
   lets latency-sensitive clients opt out of cache reads entirely.
   - DB column + migration (134)
   - rowParser parseCacheDefaultMode
   - API create default + PATCH update
   - checkSemanticCache returns null on bypass

C. Type safety: ApiKeyRow/ApiKeyView/params updated, superRefine
   guard includes cacheDefaultMode.

Cache write path intentionally unchanged: apiKeyId is already in the
cache signature (semanticCache.ts:140), so per-key isolation prevents
cross-key pollution.

Changed test files:
- tests/unit/chatcore-semantic-cache.test.ts (3 new tests)

Signed-off-by: Minxi Hou <houminxi@gmail.com>

* docs: document semantic cache latency impact + bypass configuration

---------

Signed-off-by: Minxi Hou <houminxi@gmail.com>
This commit is contained in:
Bob.Hou
2026-08-11 08:55:07 -04:00
committed by GitHub
parent 16b67f5f68
commit 2b2d947faf
13 changed files with 193 additions and 8 deletions

View File

@@ -24,6 +24,7 @@ export const API_KEY_COLUMN_FALLBACKS = [
{ name: "allowed_endpoints", definition: "allowed_endpoints TEXT" },
{ name: "allowed_quotas", definition: "allowed_quotas TEXT NOT NULL DEFAULT '[]'" },
{ name: "stream_default_mode", definition: "stream_default_mode TEXT NOT NULL DEFAULT 'legacy'" },
{ name: "cache_default_mode", definition: "cache_default_mode TEXT NOT NULL DEFAULT 'legacy'" },
{
name: "disable_non_public_models",
definition: "disable_non_public_models INTEGER NOT NULL DEFAULT 0",