mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-08-12 10:12:11 +03:00
feat(api): prompt-cache health summary endpoint and analytics tab (#8827)
Adds GET /api/usage/cache-health and a Cache Health tab under /dashboard/analytics, both backed by a pure summary over the cache columns already present in call_logs. Motivated by a production diagnosis where the aggregate ratio was actively misleading. The window read 24.8M cached tokens and wrote 9.5M — a write/read of 0.385, which reads as merely mediocre. The actual shape was very different: the median call wrote 848 tokens while 18% of the calls carried 94% of every written token, and two models in the same window sat at 0.13 (Sonnet) and 0.51 (Opus). Averaging hid all three facts. So the summary reports what the average cannot: the distribution (p50/p90/p99), the concentration (how few calls carry how much of the write), and the per-model split. The heavy-write threshold is relative to the window (10x the median, floored at 1024) because a cutoff tuned for 130k-token conversations reports nothing at all on 2k-token ones; 1024 is the minimum Anthropic bills for cache creation, below which a write carries no signal. Calls that neither read nor wrote are counted separately from thrash — a route that does not cache is an absence of caching, not a sick cache — and only successful calls are summarized, since a 4xx/5xx never reached the provider cache and would dilute the ratio. Tests cover the summary (8) and the route (6, against a real SQLite so the WHERE clause itself is exercised), including that an internal failure answers 500 without leaking the stack trace, the SQL text or a table name.
This commit is contained in:
committed by
GitHub
parent
c8f1d62de5
commit
aa85fa02bb
33
src/app/api/usage/cache-health/route.ts
Normal file
33
src/app/api/usage/cache-health/route.ts
Normal file
@@ -0,0 +1,33 @@
|
||||
import { NextResponse } from "next/server";
|
||||
import { z } from "zod";
|
||||
|
||||
import { buildCacheHealthResponse } from "@/lib/usage/cacheHealth";
|
||||
|
||||
const querySchema = z.object({
|
||||
range: z.enum(["1h", "24h", "7d", "30d"]).default("24h"),
|
||||
model: z.string().min(1).max(200).optional(),
|
||||
});
|
||||
|
||||
export async function GET(request: Request) {
|
||||
try {
|
||||
const { searchParams } = new URL(request.url);
|
||||
const parsed = querySchema.safeParse({
|
||||
range: searchParams.get("range") ?? undefined,
|
||||
model: searchParams.get("model") || undefined,
|
||||
});
|
||||
|
||||
if (!parsed.success) {
|
||||
return NextResponse.json(
|
||||
{ error: parsed.error.issues[0]?.message ?? "Invalid query parameters" },
|
||||
{ status: 400 }
|
||||
);
|
||||
}
|
||||
|
||||
return NextResponse.json(buildCacheHealthResponse(parsed.data));
|
||||
} catch (error) {
|
||||
// Never echo the error: the message can carry the DATA_DIR path and the
|
||||
// SQL text. Details go to the server log, the caller gets a fixed string.
|
||||
console.error("Error building cache health summary:", error);
|
||||
return NextResponse.json({ error: "Failed to build cache health summary" }, { status: 500 });
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user