feat(api): prompt-cache health summary endpoint and analytics tab (#8827)

Adds GET /api/usage/cache-health and a Cache Health tab under
/dashboard/analytics, both backed by a pure summary over the cache columns
already present in call_logs.

Motivated by a production diagnosis where the aggregate ratio was actively
misleading. The window read 24.8M cached tokens and wrote 9.5M — a
write/read of 0.385, which reads as merely mediocre. The actual shape was
very different: the median call wrote 848 tokens while 18% of the calls
carried 94% of every written token, and two models in the same window sat
at 0.13 (Sonnet) and 0.51 (Opus). Averaging hid all three facts.

So the summary reports what the average cannot: the distribution
(p50/p90/p99), the concentration (how few calls carry how much of the
write), and the per-model split. The heavy-write threshold is relative to
the window (10x the median, floored at 1024) because a cutoff tuned for
130k-token conversations reports nothing at all on 2k-token ones; 1024 is
the minimum Anthropic bills for cache creation, below which a write carries
no signal.

Calls that neither read nor wrote are counted separately from thrash — a
route that does not cache is an absence of caching, not a sick cache — and
only successful calls are summarized, since a 4xx/5xx never reached the
provider cache and would dilute the ratio.

Tests cover the summary (8) and the route (6, against a real SQLite so the
WHERE clause itself is exercised), including that an internal failure
answers 500 without leaking the stack trace, the SQL text or a table name.
This commit is contained in:
Diego Rodrigues de Sa e Souza
2026-07-28 03:53:28 -03:00
committed by GitHub
parent c8f1d62de5
commit aa85fa02bb
6 changed files with 886 additions and 2 deletions

View File

@@ -0,0 +1,33 @@
import { NextResponse } from "next/server";
import { z } from "zod";
import { buildCacheHealthResponse } from "@/lib/usage/cacheHealth";
const querySchema = z.object({
range: z.enum(["1h", "24h", "7d", "30d"]).default("24h"),
model: z.string().min(1).max(200).optional(),
});
export async function GET(request: Request) {
try {
const { searchParams } = new URL(request.url);
const parsed = querySchema.safeParse({
range: searchParams.get("range") ?? undefined,
model: searchParams.get("model") || undefined,
});
if (!parsed.success) {
return NextResponse.json(
{ error: parsed.error.issues[0]?.message ?? "Invalid query parameters" },
{ status: 400 }
);
}
return NextResponse.json(buildCacheHealthResponse(parsed.data));
} catch (error) {
// Never echo the error: the message can carry the DATA_DIR path and the
// SQL text. Details go to the server log, the caller gets a fixed string.
console.error("Error building cache health summary:", error);
return NextResponse.json({ error: "Failed to build cache health summary" }, { status: 500 });
}
}