diff --git a/CHANGELOG.md b/CHANGELOG.md index 30be1ab7b4..bd4c496659 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,7 @@ _In development — bullets added per PR; finalized at release._ ### 🐛 Fixed +- **fix(api): cache-HIT `X-OmniRoute-Response-Cost` now reports the incremental cost (≈0), not the original** — on a semantic-cache HIT the gateway serves the stored response **without** an upstream call, but `X-OmniRoute-Response-Cost` was reporting the original call's full cost (recomputed from the cached `usage`). A consumer summing `response-cost` for billing was therefore charging for responses that cost ≈$0 to serve (and stale entries could inflate it). Cache hits now bill `X-OmniRoute-Response-Cost: 0.0000000000` (the real incremental cost), and the avoided cost is surfaced in a new **`X-OmniRoute-Cost-Saved`** header for cache analytics — mirroring the existing `tokens_saved` concept. The MISS path is unchanged. (PRD-2026-06-19-cache-hit-cost-reporting) - **fix(models): imported vision-capable models keep their vision capability** — after importing a provider key, vision-capable models (e.g. OpenRouter models whose `architecture` declares image input, and other synced providers) were listed as text-only in `/v1/models` and the dashboard — even though image requests actually worked. Synced model records never captured the vision flag, and the catalog's OpenRouter live-enrichment (which derives vision from `architecture.input_modalities`) is skipped once a provider has synced models. Discovery now captures `supportsVision` at sync time (from `architecture.input_modalities`, the string `architecture.modality`, or a top-level `input_modalities`), mirroring the existing `supportsThinking` capture, and the catalog surfaces `capabilities.vision` for synced models. ([#4264](https://github.com/diegosouzapw/OmniRoute/issues/4264) — thanks @FerLuisxd) - **fix(providers): Cloudflare Workers AI model discovery shows model names, not UUIDs** — importing a Cloudflare Workers AI key listed models with internal UUID identifiers (e.g. `429b9e8b-d99e-…`) instead of their usable slugs (`@cf/meta/llama-3.1-8b-instruct`). Cloudflare's `/ai/models/search` returns `{ id: "", name: "@cf/…" }`, and discovery was passing the raw objects through — so the UUID `id` became the callable model id. The `cloudflare-ai` discovery now maps each result's `name` → id, surfacing the real `@cf/…` model ids. ([#4259](https://github.com/diegosouzapw/OmniRoute/issues/4259) — thanks @FerLuisxd) diff --git a/docs/reference/API_REFERENCE.md b/docs/reference/API_REFERENCE.md index 7dd25281d4..833f8dd015 100644 --- a/docs/reference/API_REFERENCE.md +++ b/docs/reference/API_REFERENCE.md @@ -74,11 +74,14 @@ Content-Type: application/json | `X-OmniRoute-Session-Id` | Response | Effective session ID used by OmniRoute | | `X-OmniRoute-Request-Id` | Response | Request correlation id (when known) | | `X-OmniRoute-Version` | Response | OmniRoute build version (always present) | +| `X-OmniRoute-Cost-Saved` | Response | USD the cache avoided on a HIT (cache hits only) | > Nginx note: if you rely on underscore headers (for example `x_session_id`), enable `underscores_in_headers on;`. > **Cost telemetry headers:** non-streaming success responses also carry the `X-OmniRoute-*` cost-telemetry set — `X-OmniRoute-Response-Cost` (USD, fixed 10 decimals; `0.0000000000` for free/unpriced), `X-OmniRoute-Tokens-In` / `X-OmniRoute-Tokens-Out`, `X-OmniRoute-Model`, `X-OmniRoute-Provider`, `X-OmniRoute-Latency-Ms`, `X-OmniRoute-Cache-Hit`, and `X-OmniRoute-Fallback-Attempts` (only when > 0), plus `X-OmniRoute-Request-Id` and `X-OmniRoute-Version`. These are emitted by chat completions, `/v1/responses`, `/v1/messages`, **and the media endpoints** — `/v1/embeddings`, `/v1/images/generations`, `/v1/audio/speech`, `/v1/audio/transcriptions`, `/v1/rerank`, `/v1/videos/generations`, `/v1/music/generations`, and `/v1/moderations` (always cost `0`). Media cost is computed per modality (per-image, per-second, per-character, per search-unit) when pricing is available, otherwise `0` (fail-open). +> **Cache-hit cost semantics:** on a semantic-cache HIT (`X-OmniRoute-Cache-Hit: true`) no upstream call is made, so `X-OmniRoute-Response-Cost` is `0.0000000000` (the **incremental** cost of serving the hit). The original/would-have-been cost is reported separately in `X-OmniRoute-Cost-Saved`. Billing consumers should sum `X-OmniRoute-Response-Cost` (hits cost nothing); cache analytics can aggregate `X-OmniRoute-Cost-Saved`. + --- ## Embeddings diff --git a/docs/reference/openapi.yaml b/docs/reference/openapi.yaml index 6f59c19624..ee68c297fe 100644 --- a/docs/reference/openapi.yaml +++ b/docs/reference/openapi.yaml @@ -1134,6 +1134,13 @@ paths: schema: type: string description: OmniRoute build version (always present). + X-OmniRoute-Cost-Saved: + schema: + type: string + description: >- + On a semantic-cache HIT, the original (would-have-been) cost in USD that + the cache avoided (fixed 10 decimals). Present only on cache hits; + X-OmniRoute-Response-Cost is 0 for the same response (incremental cost). content: application/json: schema: diff --git a/open-sse/handlers/chatCore/semanticCache.ts b/open-sse/handlers/chatCore/semanticCache.ts index 9a27a75a94..bb247b085f 100644 --- a/open-sse/handlers/chatCore/semanticCache.ts +++ b/open-sse/handlers/chatCore/semanticCache.ts @@ -74,13 +74,17 @@ export async function checkSemanticCache({ "Content-Type": cachedSse ? "text/event-stream" : "application/json", [OMNIROUTE_RESPONSE_HEADERS.cache]: "HIT", }; + // A cache HIT serves WITHOUT an upstream call, so the incremental cost billed to + // the client is 0 (consumers that sum X-OmniRoute-Response-Cost must not charge for + // hits). The original/would-have-been cost is surfaced via X-OmniRoute-Cost-Saved. attachOmniRouteMetaHeaders(headers, { provider, model, cacheHit: true, latencyMs: Date.now() - startTime, usage: cachedUsage, - costUsd: cachedCost, + costUsd: 0, + costSavedUsd: cachedCost, }); return { success: true, diff --git a/src/domain/omnirouteResponseMeta.ts b/src/domain/omnirouteResponseMeta.ts index b22bed59b7..9abe49f327 100644 --- a/src/domain/omnirouteResponseMeta.ts +++ b/src/domain/omnirouteResponseMeta.ts @@ -48,6 +48,7 @@ export function formatOmniRouteCost(costUsd: unknown): string { export function buildOmniRouteResponseMetaHeaders({ cacheHit = false, costUsd = 0, + costSavedUsd = undefined, fallbackAttempts = 0, latencyMs = 0, model = null, @@ -57,6 +58,14 @@ export function buildOmniRouteResponseMetaHeaders({ }: { cacheHit?: boolean; costUsd?: unknown; + /** + * Cost the cache AVOIDED. A semantic-cache HIT serves at ≈0 incremental cost + * (`costUsd: 0`) but saved the original call's cost — surface it here so billing + * consumers don't charge for hits while analytics can still see what was saved. + * Emitted as `X-OmniRoute-Cost-Saved` only when provided (omitted on normal + * responses); pass `0` to explicitly mark a free-model HIT that saved nothing. + */ + costSavedUsd?: unknown; fallbackAttempts?: number; latencyMs?: unknown; model?: string | null; @@ -86,6 +95,12 @@ export function buildOmniRouteResponseMetaHeaders({ headers[OMNIROUTE_RESPONSE_HEADERS.provider] = getProviderAlias(provider); } + // Cache-saved cost: emitted only when the caller passes a value (cache HITs), so + // non-cache responses keep their existing header shape. `0` is a valid saved cost. + if (costSavedUsd != null) { + headers[OMNIROUTE_RESPONSE_HEADERS.costSaved] = formatOmniRouteCost(costSavedUsd); + } + const attempts = toNonNegativeInteger(fallbackAttempts); if (attempts > 0) { headers[OMNIROUTE_RESPONSE_HEADERS.fallbackAttempts] = String(attempts); diff --git a/src/shared/constants/headers.ts b/src/shared/constants/headers.ts index f5be0c303e..9044575fe1 100644 --- a/src/shared/constants/headers.ts +++ b/src/shared/constants/headers.ts @@ -1,6 +1,7 @@ export const OMNIROUTE_RESPONSE_HEADERS = { cache: "X-OmniRoute-Cache", cacheHit: "X-OmniRoute-Cache-Hit", + costSaved: "X-OmniRoute-Cost-Saved", fallbackAttempts: "X-OmniRoute-Fallback-Attempts", latencyMs: "X-OmniRoute-Latency-Ms", model: "X-OmniRoute-Model", diff --git a/tests/unit/chatcore-semantic-cache.test.ts b/tests/unit/chatcore-semantic-cache.test.ts index 9eae53e253..1ca36cc072 100644 --- a/tests/unit/chatcore-semantic-cache.test.ts +++ b/tests/unit/chatcore-semantic-cache.test.ts @@ -18,6 +18,8 @@ const { generateSignature, setCachedResponse, clearCache } = await import( "../../src/lib/semanticCache.ts" ); const { OMNIROUTE_RESPONSE_HEADERS } = await import("../../src/shared/constants/headers.ts"); +const { calculateCost } = await import("../../src/lib/usage/costCalculator.ts"); +const { formatOmniRouteCost } = await import("../../src/domain/omnirouteResponseMeta.ts"); test.after(() => { core.resetDbInstance(); @@ -305,3 +307,88 @@ test("checkSemanticCache HITs even when the cached body has no usage (cost falls assert.equal(persistCalls.length, 1); assert.equal(persistCalls[0].tokens, undefined, "no usage -> persisted tokens is undefined"); }); + +// ─── Cache-HIT cost reporting (PRD-2026-06-19-cache-hit-cost-reporting) ─────── +// A HIT does NOT call upstream, so the INCREMENTAL cost of serving it is ≈0. The +// X-OmniRoute-Response-Cost must therefore be 0 (so billing consumers don't charge +// for cache hits), while the original cost is surfaced via X-OmniRoute-Cost-Saved. + +test("checkSemanticCache HIT bills 0 incremental cost and reports the original cost in X-OmniRoute-Cost-Saved", async () => { + clearCache(); + const usage = { prompt_tokens: 1000, completion_tokens: 1000, total_tokens: 2000 }; + const cached = { + id: "chatcmpl-cached-cost-saved", + choices: [ + { index: 0, message: { role: "assistant", content: "cached answer" }, finish_reason: "stop" }, + ], + usage, + }; + const { args } = makeHitArgs({ + body: { + model: "gpt-4o", + messages: [{ role: "user", content: "hit query cost-saved" }], + temperature: 0, + }, + stream: false, + }); + seedHit(args, cached); + + // The original (would-have-been) cost — computed with the SAME calculator the handler + // uses, against the same fresh DATA_DIR, so the values match deterministically. + const expectedSaved = formatOmniRouteCost( + await calculateCost(args.provider, args.model, usage as Record) + ); + assert.notEqual( + expectedSaved, + "0.0000000000", + "sanity: gpt-4o must be priced for this regression to be meaningful" + ); + + const result = await checkSemanticCache(args as Parameters[0]); + assert.ok(result, "HIT -> non-null result"); + const res = result.response as Response; + + assert.equal(res.headers.get(OMNIROUTE_RESPONSE_HEADERS.cache), "HIT"); + // Incremental cost billed to the client on a HIT is 0 (no upstream call happened). + assert.equal( + res.headers.get(OMNIROUTE_RESPONSE_HEADERS.responseCost), + "0.0000000000", + "cache HIT must bill 0 incremental cost" + ); + // The avoided cost is surfaced for cache analytics. + assert.equal( + res.headers.get(OMNIROUTE_RESPONSE_HEADERS.costSaved), + expectedSaved, + "X-OmniRoute-Cost-Saved reflects the original cost the cache avoided" + ); +}); + +test("checkSemanticCache isolates HITs per apiKeyId (no cross-key cache sharing) [#3740 edge]", async () => { + clearCache(); + const cached = { + id: "chatcmpl-cached-isolation", + choices: [ + { index: 0, message: { role: "assistant", content: "key A answer" }, finish_reason: "stop" }, + ], + usage: { prompt_tokens: 5, completion_tokens: 5, total_tokens: 10 }, + }; + const sharedBody = () => ({ + model: "gpt-4o", + messages: [{ role: "user", content: "isolation probe" }], + temperature: 0, + }); + + // Seed the cache under apiKeyId "keyA". + const { args: argsA } = makeHitArgs({ body: sharedBody(), apiKeyId: "keyA" }); + seedHit(argsA, cached); + + // A different key with an IDENTICAL body must NOT see keyA's entry (namespaced signature). + const { args: argsB } = makeHitArgs({ body: sharedBody(), apiKeyId: "keyB" }); + const missB = await checkSemanticCache(argsB as Parameters[0]); + assert.equal(missB, null, "keyB must NOT resolve keyA's cached entry (per-key isolation)"); + + // keyA itself still gets a HIT. + const { args: argsA2 } = makeHitArgs({ body: sharedBody(), apiKeyId: "keyA" }); + const hitA = await checkSemanticCache(argsA2 as Parameters[0]); + assert.ok(hitA, "keyA must resolve its own cached entry"); +}); diff --git a/tests/unit/omniroute-response-meta.test.ts b/tests/unit/omniroute-response-meta.test.ts index fb1cdcea0a..5956411e8d 100644 --- a/tests/unit/omniroute-response-meta.test.ts +++ b/tests/unit/omniroute-response-meta.test.ts @@ -120,3 +120,46 @@ test("buildOmniRouteSseMetadataComment emits comment lines compatible with SSE", assert.match(comment, /^: x-omniroute-tokens-out=2/m); assert.match(comment, /^: x-omniroute-response-cost=0\.0000000000/m); }); + +test("buildOmniRouteResponseMetaHeaders emits X-OmniRoute-Cost-Saved only when costSavedUsd is provided", () => { + // Cache HIT: the incremental cost of serving the hit is 0, but the cache saved the + // original (would-have-been) cost — surfaced via the Cost-Saved header for analytics. + const hit = buildOmniRouteResponseMetaHeaders({ + provider: "openai", + model: "gpt-4o", + cacheHit: true, + costUsd: 0, + costSavedUsd: 0.0125, + }); + assert.equal(hit[OMNIROUTE_RESPONSE_HEADERS.responseCost], "0.0000000000"); + assert.equal(hit[OMNIROUTE_RESPONSE_HEADERS.costSaved], "0.0125000000"); + + // A normal response (no costSavedUsd) omits the Cost-Saved header entirely. + const miss = buildOmniRouteResponseMetaHeaders({ + provider: "openai", + model: "gpt-4o", + costUsd: 0.0125, + }); + assert.equal(miss[OMNIROUTE_RESPONSE_HEADERS.costSaved], undefined); + + // A free-model HIT still emits Cost-Saved (= 0) — it explicitly passed costSavedUsd. + const freeHit = buildOmniRouteResponseMetaHeaders({ + cacheHit: true, + costUsd: 0, + costSavedUsd: 0, + }); + assert.equal(freeHit[OMNIROUTE_RESPONSE_HEADERS.costSaved], "0.0000000000"); +}); + +test("attachOmniRouteMetaHeaders forwards costSavedUsd onto a Headers bag", () => { + const headers = new Headers({ "Content-Type": "application/json" }); + attachOmniRouteMetaHeaders(headers, { + provider: "openai", + model: "gpt-4o", + cacheHit: true, + costUsd: 0, + costSavedUsd: 0.0125, + }); + assert.equal(headers.get(OMNIROUTE_RESPONSE_HEADERS.responseCost), "0.0000000000"); + assert.equal(headers.get(OMNIROUTE_RESPONSE_HEADERS.costSaved), "0.0125000000"); +});