mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-07-26 09:52:11 +03:00
fix(usage): correct token/request counting for 30D/90D/YTD/ALL ranges (#7300)
* chore(ci): add .mergify.yml to main — Mergify only reads config from the default branch (#7168)
* fix(ci): add the auto-enqueue pull_request_rule to the Mergify config (queue_conditions alone are eligibility-only) (#7179)
* fix(ci): migrate Mergify auto-enqueue to merge_protections_settings.auto_merge_conditions (rules-based path is EOL 2026-07-16) (#7216)
* fix(ci): drop Mergify batch settings (batching is a paid-tier feature; free plan queue is serial) (#7220)
* fix(ci): merge queue tolerates the advisory dast-smoke failure (its GH-hosted build hang dequeued every attempt) (#7225)
* fix(usage): correct token/request counting for 30D/90D/YTD/ALL ranges
Two bugs caused incorrect usage statistics for date ranges beyond the
raw data retention window:
1. Cutoff mismatch: the analytics route computed rawCutoffDate from
aggregation.rawDataRetentionDays (migration 046 seeds =7) while
cleanupUsageHistory rolls up and deletes at retention.usageHistory
(=30). The window [day-30, day-7) existed in usage_history but was
excluded from BOTH UNION legs — raw leg floored at day-7, aggregated
leg ended at day-7 — producing undercounted token sums for 30D,
90D, YTD, and ALL ranges.
Fix: use dbSettings.retention.usageHistory for the raw cutoff in
both route.ts and getRawDataCutoffDate() (aggregateHistory.ts),
matching the actual cleanup boundary.
2. Request undercount: COUNT(*) on the unified source counted each
daily_usage_summary row as 1, not total_requests. A day with 50
rolled-up requests counted as 1.
Fix: add a 'requests' column to both UNION legs (raw: 1, aggregated:
total_requests), change COUNT(*) to SUM(requests) in 6 query
functions, and change successfulRequests from
SUM(CASE WHEN success=1 THEN 1 ELSE 0 END) to
SUM(CASE WHEN success=1 THEN requests ELSE 0 END). Also set
agg leg latency_ms to NULL so AVG(latency_ms) is not skewed.
Tests: 33/33 source-level tests pass (db-usageanalytics-split.test.ts),
verifying 'requests' column presence and SUM(requests) usage in all
affected queries. DB-level integration test added to
usage-analytics.test.ts (requires node + better-sqlite3).
* fix(ci): green CI reds on #7300 — file-size ratchet, stale test fixture, shallow-checkout selfref test
- src/app/api/usage/analytics/route.ts: trim the new comment to keep the file
at the frozen file-size baseline (942 lines) after the retention.usageHistory
cutoff fix — no logic change.
- tests/unit/usage-analytics-route.test.ts: the pre-existing "does not
double-count raw and aggregated rows" test hardcoded a 30-day cutoff that
matched the OLD (buggy) aggregation.rawDataRetentionDays default. Now that
the raw/aggregated boundary correctly uses retention.usageHistory (365 days
by default, matching cleanupUsageHistory's actual rollup/delete boundary),
the fixture's synthetic "old" row was within the raw window and got
excluded from the aggregated leg. Read the real retention setting instead
of hardcoding 30 so the fixture reflects the corrected boundary. Same
assertions (still expects no double-counting, totalRequests=2,
totalTokens=185) — only the fixture dates change.
- tests/unit/check-test-masking-selfref-6634.test.ts: tolerate the shallow/
single-ref checkout used by GitHub-hosted Unit Tests runners (no local
origin/main ref) by fetching it on demand and skipping (never failing) when
unreachable offline. Matches the fix already applied on another branch
(2e42b8efc/#7174) for the same root cause, not yet on main.
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
* test(ci): make the #6634 selfref test checkout-independent (read the real file, no git ref)
The previous on-demand `git fetch origin main` + t.skip() fallback cleared the
shallow-checkout failure but tripped the PR Test Policy's test-masking gate
(a new .skip counts as a silenced assert — correctly so).
Drop the git dependency entirely instead: read the REAL current source of
tests/unit/check-test-masking.test.ts from disk (so the actual #6404 fixture
literals stay under test) and model the pre-#6404 state with an empty base,
which maximizes headTaut - baseTaut — the strictest input for the exclusion
this test asserts. No skip, no weakened assertion, same deepEqual guarantee.
Verified non-vacuous: neutralizing SELF_TEST_FIXTURE_RE in
scripts/check/check-test-masking.mjs makes this test fail; restoring it makes
it pass.
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
* test(ci): make the #6634 selfref guard hermetic — main's copy hard-fails every PR (#7341)
main's copy of this test still does git I/O inside a unit test:
const baseSrc = git(['show', 'origin/main:' + FILE]);
Runners check out a shallow single ref, so origin/main does not resolve and the
test dies with 'fatal: invalid object name origin/main'. Every PR into main
fails Unit Tests (7/8) on it — today that is #7313, #7315, #7316, #7334, #7336
and #7337, six PRs red on a defect none of them introduced. #7313 has no other
red at all.
release/v3.8.49 already carries a fix (2e42b8efc, #7174: try/catch, fetch
origin/main on demand, t.skip() when unreachable), but it only reaches main at
release time — so main stays broken for the whole cycle. Cherry-picking it would
also import a new problem: PR Test Policy classifies t.skip() as a silenced
assertion, which we watched it correctly catch on #7300 today.
This is the hermetic version instead (ported from #7327, which does the same for
the release branch): read the file straight off disk, compare against an empty
base so baseTaut/baseExtTaut are 0 — the strictest possible comparison point —
and call evaluateMasking() directly. No git ref, no fetch, no skip, nothing the
runner's checkout depth can break.
The #6634 regression stays covered: the guard's logic lives in
SELF_TEST_FIXTURE_RE (check-test-masking.mjs:337), not in the test. Proven both
ways on main before committing — neutralise SELF_TEST_FIXTURE_RE to /$^/ and
the test FAILS; restore it and it passes 2/2, with check-test-masking.mjs left
byte-identical.
Co-authored-by: growab <nekron@icloud.com>
* chore(quality): tighten main's coverage baseline to the CI's real numbers (#7347)
main's ratchet had been failing --require-tighten on every PR: 11 metrics
improved but the baseline was never tightened. Same class as the #6634
selfref guard — an infra fix that lands only on the release branch leaves
main red for the whole cycle, and every PR into main pays for it.
Values are the merged-coverage numbers from a run on main itself (a local
run measures ~68% vs CI's ~80%; the baseline's own note warns about that
gap). Only the 11 coverage values change — gitleaks and semgrepFindings
keep main's own state.
No changelog fragment: #7326 carries it on release/v3.8.49, and a second
one here would double the entry at release time.
---------
Co-authored-by: Diego Rodrigues de Sa e Souza <8016841+diegosouzapw@users.noreply.github.com>
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
This commit is contained in:
growab
@@ -347,10 +347,10 @@ export async function GET(request: Request) {
|
||||
}
|
||||
}
|
||||
|
||||
// Compute the raw-data cutoff: rows older than this may have been rolled up to
|
||||
// daily_usage_summary and deleted from usage_history.
|
||||
// Raw-data cutoff: must match cleanupUsageHistory's rollup/delete boundary —
|
||||
// retention.usageHistory (src/lib/db/cleanup.ts), NOT aggregation.rawDataRetentionDays.
|
||||
const dbSettings = getUserDatabaseSettings();
|
||||
const rawRetentionDays = dbSettings.aggregation?.rawDataRetentionDays ?? 30;
|
||||
const rawRetentionDays = dbSettings.retention?.usageHistory ?? 30;
|
||||
const rawCutoff = new Date();
|
||||
rawCutoff.setDate(rawCutoff.getDate() - rawRetentionDays);
|
||||
const rawCutoffIso = rawCutoff.toISOString();
|
||||
|
||||
@@ -49,14 +49,14 @@ export function getUsageSummary(unifiedSource: string, params: AnalyticsParams):
|
||||
.prepare(
|
||||
`
|
||||
SELECT
|
||||
COUNT(*) as totalRequests,
|
||||
COALESCE(SUM(requests), 0) as totalRequests,
|
||||
COALESCE(SUM(tokens_input), 0) as promptTokens,
|
||||
COALESCE(SUM(tokens_output), 0) as completionTokens,
|
||||
COALESCE(SUM(tokens_input + tokens_output), 0) as totalTokens,
|
||||
COUNT(DISTINCT model) as uniqueModels,
|
||||
COUNT(DISTINCT COALESCE(NULLIF(account_key, ''), NULLIF(connection_id, ''))) as uniqueAccounts,
|
||||
COUNT(DISTINCT COALESCE(NULLIF(api_key_id, ''), NULLIF(api_key_name, ''))) as uniqueApiKeys,
|
||||
COALESCE(SUM(CASE WHEN success = 1 THEN 1 ELSE 0 END), 0) as successfulRequests,
|
||||
COALESCE(SUM(CASE WHEN success = 1 THEN requests ELSE 0 END), 0) as successfulRequests,
|
||||
COALESCE(AVG(latency_ms), 0) as avgLatencyMs,
|
||||
COALESCE(MIN(timestamp), '') as firstRequest,
|
||||
COALESCE(MAX(timestamp), '') as lastRequest
|
||||
@@ -101,7 +101,7 @@ export function getDailyUsage(unifiedSource: string, params: AnalyticsParams): D
|
||||
`
|
||||
SELECT
|
||||
DATE(timestamp) as date,
|
||||
COUNT(*) as requests,
|
||||
COALESCE(SUM(requests), 0) as requests,
|
||||
COALESCE(SUM(tokens_input), 0) as promptTokens,
|
||||
COALESCE(SUM(tokens_output), 0) as completionTokens,
|
||||
COALESCE(SUM(tokens_input + tokens_output), 0) as totalTokens
|
||||
@@ -215,7 +215,7 @@ export function getModelUsageRows(unifiedSource: string, params: AnalyticsParams
|
||||
LOWER(model) as model,
|
||||
LOWER(provider) as provider,
|
||||
COALESCE(NULLIF(service_tier, ''), 'standard') as serviceTier,
|
||||
COUNT(*) as requests,
|
||||
COALESCE(SUM(requests), 0) as requests,
|
||||
COALESCE(SUM(tokens_input), 0) as promptTokens,
|
||||
COALESCE(SUM(tokens_output), 0) as completionTokens,
|
||||
COALESCE(SUM(tokens_cache_read), 0) as cacheReadTokens,
|
||||
@@ -223,7 +223,7 @@ export function getModelUsageRows(unifiedSource: string, params: AnalyticsParams
|
||||
COALESCE(SUM(tokens_reasoning), 0) as reasoningTokens,
|
||||
COALESCE(SUM(tokens_input + tokens_output), 0) as totalTokens,
|
||||
COALESCE(AVG(latency_ms), 0) as avgLatencyMs,
|
||||
COALESCE(SUM(CASE WHEN success = 1 THEN 1 ELSE 0 END), 0) as successfulRequests,
|
||||
COALESCE(SUM(CASE WHEN success = 1 THEN requests ELSE 0 END), 0) as successfulRequests,
|
||||
COALESCE(MAX(timestamp), '') as lastUsed
|
||||
FROM ${unifiedSource} AS _u
|
||||
GROUP BY LOWER(model), LOWER(provider), serviceTier
|
||||
@@ -298,12 +298,12 @@ export function getProviderUsageRows(
|
||||
`
|
||||
SELECT
|
||||
LOWER(provider) as provider,
|
||||
COUNT(*) as requests,
|
||||
COALESCE(SUM(requests), 0) as requests,
|
||||
COALESCE(SUM(tokens_input), 0) as promptTokens,
|
||||
COALESCE(SUM(tokens_output), 0) as completionTokens,
|
||||
COALESCE(SUM(tokens_input + tokens_output), 0) as totalTokens,
|
||||
COALESCE(AVG(latency_ms), 0) as avgLatencyMs,
|
||||
COALESCE(SUM(CASE WHEN success = 1 THEN 1 ELSE 0 END), 0) as successfulRequests
|
||||
COALESCE(SUM(CASE WHEN success = 1 THEN requests ELSE 0 END), 0) as successfulRequests
|
||||
FROM ${unifiedSource} AS _u
|
||||
GROUP BY LOWER(provider)
|
||||
ORDER BY requests DESC
|
||||
@@ -553,7 +553,7 @@ export function getServiceTierUsageRows(
|
||||
LOWER(provider) as provider,
|
||||
LOWER(model) as model,
|
||||
COALESCE(NULLIF(service_tier, ''), 'standard') as serviceTier,
|
||||
COUNT(*) as requests,
|
||||
COALESCE(SUM(requests), 0) as requests,
|
||||
COALESCE(SUM(tokens_input), 0) as promptTokens,
|
||||
COALESCE(SUM(tokens_output), 0) as completionTokens,
|
||||
COALESCE(SUM(tokens_cache_read), 0) as cacheReadTokens,
|
||||
@@ -633,7 +633,7 @@ export function getWeeklyPatternRows(
|
||||
SELECT
|
||||
DATE(timestamp) as date,
|
||||
strftime('%w', timestamp) as dayOfWeek,
|
||||
COUNT(*) as requests,
|
||||
COALESCE(SUM(requests), 0) as requests,
|
||||
COALESCE(SUM(tokens_input + tokens_output), 0) as totalTokens
|
||||
FROM ${unifiedSource} AS _u
|
||||
GROUP BY DATE(timestamp), strftime('%w', timestamp)
|
||||
|
||||
@@ -88,7 +88,6 @@ export function buildUnifiedSource(opts: BuildUnifiedSourceOptions): UnifiedSour
|
||||
unifiedParams.rawCutoffDate = rawCutoffDate;
|
||||
}
|
||||
const aggWhere = aggConditions.length > 0 ? `WHERE ${aggConditions.join(" AND ")}` : "";
|
||||
|
||||
const unifiedSource = needsAggregated
|
||||
? `(
|
||||
SELECT
|
||||
@@ -106,7 +105,8 @@ export function buildUnifiedSource(opts: BuildUnifiedSourceOptions): UnifiedSour
|
||||
connection_id,
|
||||
account_key,
|
||||
api_key_id,
|
||||
api_key_name
|
||||
api_key_name,
|
||||
1 as requests
|
||||
FROM usage_history
|
||||
${rawWhere}
|
||||
UNION ALL
|
||||
@@ -121,11 +121,12 @@ export function buildUnifiedSource(opts: BuildUnifiedSourceOptions): UnifiedSour
|
||||
0 as tokens_reasoning,
|
||||
'standard' as service_tier,
|
||||
1 as success,
|
||||
0 as latency_ms,
|
||||
NULL as latency_ms,
|
||||
NULL as connection_id,
|
||||
NULL as account_key,
|
||||
NULL as api_key_id,
|
||||
NULL as api_key_name
|
||||
NULL as api_key_name,
|
||||
total_requests as requests
|
||||
FROM daily_usage_summary
|
||||
${aggWhere}
|
||||
)`
|
||||
@@ -134,10 +135,11 @@ export function buildUnifiedSource(opts: BuildUnifiedSourceOptions): UnifiedSour
|
||||
tokens_input, tokens_output,
|
||||
tokens_cache_read, tokens_cache_creation, tokens_reasoning,
|
||||
service_tier, success, latency_ms,
|
||||
connection_id, account_key, api_key_id, api_key_name
|
||||
connection_id, account_key, api_key_id, api_key_name,
|
||||
1 as requests
|
||||
FROM usage_history
|
||||
${rawWhere}
|
||||
)`;
|
||||
)`;
|
||||
|
||||
return { unifiedSource, unifiedParams };
|
||||
}
|
||||
|
||||
@@ -196,7 +196,13 @@ export async function rollupUsageHistoryBeforeDate(beforeDate: string): Promise<
|
||||
* @returns ISO date string (YYYY-MM-DD)
|
||||
*/
|
||||
export async function getRawDataCutoffDate(): Promise<string> {
|
||||
const rawDataRetentionDays = getUserDatabaseSettings().aggregation.rawDataRetentionDays;
|
||||
// The raw-data cutoff MUST match the actual rollup/delete boundary used by
|
||||
// cleanupUsageHistory (src/lib/db/cleanup.ts), which is driven by
|
||||
// retention.usageHistory — NOT aggregation.rawDataRetentionDays.
|
||||
// Using rawDataRetentionDays (default 7 per migration 046) creates a gap:
|
||||
// analytics floors raw data at day-7 while cleanup doesn't roll up until
|
||||
// day-30, so the window [day-30, day-7) is excluded from BOTH UNION legs.
|
||||
const rawDataRetentionDays = getUserDatabaseSettings().retention.usageHistory;
|
||||
|
||||
const cutoffDate = new Date();
|
||||
cutoffDate.setDate(cutoffDate.getDate() - rawDataRetentionDays);
|
||||
|
||||
@@ -42,6 +42,10 @@ describe("usageAnalytics/sources — buildUnifiedSource", () => {
|
||||
);
|
||||
assert.equal(unifiedParams.rawCutoff, "2024-06-15");
|
||||
assert.equal(unifiedParams.rawCutoffDate, "2024-06-15");
|
||||
// Regression: unified source must surface a `requests` column in both legs
|
||||
// so COUNT(*)→SUM(requests) counts actual requests, not row count.
|
||||
assert.ok(unifiedSource.includes("1 as requests"), "raw leg must surface requests column");
|
||||
assert.ok(unifiedSource.includes("total_requests as requests"), "aggregated leg must surface total_requests as requests");
|
||||
});
|
||||
|
||||
it("drops the aggregated leg when an api-key filter is active (raw-only)", () => {
|
||||
@@ -57,10 +61,11 @@ describe("usageAnalytics/sources — buildUnifiedSource", () => {
|
||||
!unifiedSource.includes("daily_usage_summary"),
|
||||
"aggregated leg must be absent once an api-key filter scopes the query to raw rows"
|
||||
);
|
||||
// raw leg floors at @since (not @rawCutoff) and carries the api-key params
|
||||
// Regression: raw-only fallback must also surface the requests column.
|
||||
assert.ok(unifiedSource.includes("1 as requests"), "raw-only fallback must surface requests column");
|
||||
assert.equal(unifiedParams.rawCutoff, undefined);
|
||||
assert.equal(unifiedParams.since, "2024-06-01T00:00:00.000Z");
|
||||
assert.equal(unifiedParams.apiKey0, "k1");
|
||||
assert.equal(unifiedParams.rawCutoff, undefined);
|
||||
});
|
||||
|
||||
it("buildPresetUnifiedSource returns the unifiedSource/unifiedParams shape", () => {
|
||||
@@ -119,6 +124,50 @@ describe("usageAnalytics.ts public API surface", () => {
|
||||
});
|
||||
});
|
||||
|
||||
// ── 4. Regression: COUNT(*) → SUM(requests) in all query functions ──────────
|
||||
// Verifies the fix without needing a DB driver — the function source must
|
||||
// contain SUM(requests), not COUNT(*), so aggregated rows with total_requests>1
|
||||
// are counted correctly instead of as 1-per-row.
|
||||
|
||||
describe("usageAnalytics queries use SUM(requests) not COUNT(*)", () => {
|
||||
const queriesUsingCount = [
|
||||
"getUsageSummary",
|
||||
"getDailyUsage",
|
||||
"getModelUsageRows",
|
||||
"getProviderUsageRows",
|
||||
"getServiceTierUsageRows",
|
||||
];
|
||||
|
||||
for (const fn of queriesUsingCount) {
|
||||
it(`${fn} uses SUM(requests) not COUNT(*)`, () => {
|
||||
const src = host[fn].toString();
|
||||
assert.ok(src.includes("SUM(requests)"), `${fn} must use SUM(requests) for correct aggregated row counting`);
|
||||
assert.ok(!src.match(/COUNT\(\*\)/), `${fn} must not use COUNT(*) which undercounts aggregated rows`);
|
||||
});
|
||||
}
|
||||
|
||||
it("getWeeklyPatternRows inner subquery uses SUM(requests)", () => {
|
||||
const src = host.getWeeklyPatternRows.toString();
|
||||
assert.ok(src.includes("SUM(requests)"), "getWeeklyPatternRows inner subquery must use SUM(requests)");
|
||||
});
|
||||
|
||||
it("getUsageSummary uses SUM(CASE WHEN success=1 THEN requests ELSE 0 END) for successfulRequests", () => {
|
||||
const src = host.getUsageSummary.toString();
|
||||
assert.ok(
|
||||
src.includes("SUM(CASE WHEN success = 1 THEN requests ELSE 0 END)"),
|
||||
"successfulRequests must weight by requests, not count 1-per-row"
|
||||
);
|
||||
});
|
||||
|
||||
it("getModelUsageRows uses SUM(CASE WHEN success=1 THEN requests ELSE 0 END) for successfulRequests", () => {
|
||||
const src = host.getModelUsageRows.toString();
|
||||
assert.ok(
|
||||
src.includes("SUM(CASE WHEN success = 1 THEN requests ELSE 0 END)"),
|
||||
"successfulRequests must weight by requests, not count 1-per-row"
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
// ── 3. sources leaf exports the builders directly ────────────────────────────
|
||||
|
||||
describe("sources.ts exports the builders directly", () => {
|
||||
|
||||
@@ -333,3 +333,260 @@ test("GET /api/usage/analytics includes byAccount array with cost data", async (
|
||||
);
|
||||
});
|
||||
|
||||
test("GET /api/usage/analytics includes cost by API key", async () => {
|
||||
await seedAnalyticsData();
|
||||
|
||||
const response = await analyticsRoute.GET(makeRequest("http://localhost/api/usage/analytics"));
|
||||
const body = await response.json();
|
||||
|
||||
assert.equal(response.status, 200);
|
||||
assert.ok(Array.isArray(body.byApiKey));
|
||||
assert.equal(body.byApiKey.length, 1);
|
||||
assert.equal(body.byApiKey[0].apiKeyId, "test-key");
|
||||
assert.equal(body.byApiKey[0].apiKeyName, "Primary Key");
|
||||
assertClose(body.byApiKey[0].cost, body.summary.totalCost);
|
||||
});
|
||||
|
||||
test("GET /api/usage/analytics does not double-count raw and aggregated rows", async () => {
|
||||
const db = core.getDbInstance();
|
||||
const today = new Date();
|
||||
const todayStr = today.toISOString().split("T")[0];
|
||||
// The raw/aggregated split boundary is retention.usageHistory (365 days by
|
||||
// default — see cleanupUsageHistory in src/lib/db/cleanup.ts, which rolls up
|
||||
// and deletes usage_history rows using that exact setting). Read it live
|
||||
// instead of hardcoding 30 days so this fixture stays valid regardless of
|
||||
// the configured retention window.
|
||||
const { getUserDatabaseSettings } = await import("../../src/lib/db/databaseSettings.ts");
|
||||
const rawRetentionDays = getUserDatabaseSettings().retention.usageHistory;
|
||||
const cutoffDate = new Date();
|
||||
cutoffDate.setDate(cutoffDate.getDate() - rawRetentionDays);
|
||||
const olderDate = new Date(cutoffDate);
|
||||
olderDate.setDate(olderDate.getDate() - 1);
|
||||
const olderDateStr = olderDate.toISOString().split("T")[0];
|
||||
|
||||
db.prepare(
|
||||
`INSERT INTO usage_history (provider, model, connection_id, tokens_input, tokens_output, success, latency_ms, timestamp)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?)`
|
||||
).run("openai", "gpt-4o", "raw-current", 100, 50, 1, 200, today.toISOString());
|
||||
|
||||
const insertSummary = db.prepare(
|
||||
`INSERT INTO daily_usage_summary (provider, model, date, total_requests, total_input_tokens, total_output_tokens, total_cost)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?)`
|
||||
);
|
||||
insertSummary.run("openai", "gpt-4o", todayStr, 99, 9900, 9900, 0);
|
||||
insertSummary.run("openai", "gpt-4o", olderDateStr, 1, 25, 10, 0);
|
||||
|
||||
const response = await analyticsRoute.GET(
|
||||
makeRequest("http://localhost/api/usage/analytics?range=all")
|
||||
);
|
||||
const body = await response.json();
|
||||
|
||||
assert.equal(response.status, 200);
|
||||
assert.equal(body.summary.totalRequests, 2);
|
||||
assert.equal(body.summary.totalTokens, 185);
|
||||
});
|
||||
|
||||
test("GET /api/usage/analytics omits global aggregates when filtering by API key", async () => {
|
||||
const apiKey = await apiKeysDb.createApiKey("Scoped Key", "machine1234567890");
|
||||
const db = core.getDbInstance();
|
||||
|
||||
db.prepare(
|
||||
`INSERT INTO usage_history (provider, model, connection_id, api_key_id, api_key_name, tokens_input, tokens_output, success, latency_ms, timestamp)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`
|
||||
).run(
|
||||
"openai",
|
||||
"gpt-4o",
|
||||
"scoped-conn",
|
||||
apiKey.id,
|
||||
"Scoped Key",
|
||||
100,
|
||||
50,
|
||||
1,
|
||||
200,
|
||||
new Date().toISOString()
|
||||
);
|
||||
|
||||
db.prepare(
|
||||
`INSERT INTO daily_usage_summary (provider, model, date, total_requests, total_input_tokens, total_output_tokens, total_cost)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?)`
|
||||
).run("openai", "gpt-4o", "2024-01-01", 99, 9900, 9900, 0);
|
||||
|
||||
const response = await analyticsRoute.GET(
|
||||
makeRequest(`http://localhost/api/usage/analytics?range=all&apiKeyIds=${apiKey.id}`)
|
||||
);
|
||||
const body = await response.json();
|
||||
|
||||
assert.equal(response.status, 200);
|
||||
assert.equal(body.summary.totalRequests, 1);
|
||||
assert.equal(body.summary.totalTokens, 150);
|
||||
assert.equal(body.byApiKey.length, 1);
|
||||
assert.equal(body.byApiKey[0].apiKeyId, apiKey.id);
|
||||
});
|
||||
|
||||
test("GET /api/usage/analytics groups renamed API key usage by stable ID", async () => {
|
||||
const apiKey = await apiKeysDb.createApiKey("Averyanov", "machine1234567890");
|
||||
await apiKeysDb.updateApiKeyPermissions(apiKey.id, { name: "Alexander Averyanov" });
|
||||
|
||||
const db = core.getDbInstance();
|
||||
const now = Date.now();
|
||||
const insertUsage = db.prepare(
|
||||
`INSERT INTO usage_history (provider, model, connection_id, api_key_id, api_key_name, tokens_input, tokens_output, success, latency_ms, timestamp)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`
|
||||
);
|
||||
insertUsage.run(
|
||||
"openai",
|
||||
"gpt-4o",
|
||||
"test-conn",
|
||||
apiKey.id,
|
||||
"Averyanov",
|
||||
100,
|
||||
50,
|
||||
1,
|
||||
200,
|
||||
new Date(now - 60_000).toISOString()
|
||||
);
|
||||
insertUsage.run(
|
||||
"openai",
|
||||
"gpt-4o",
|
||||
"test-conn",
|
||||
apiKey.id,
|
||||
"Desktop",
|
||||
200,
|
||||
100,
|
||||
1,
|
||||
250,
|
||||
new Date(now).toISOString()
|
||||
);
|
||||
|
||||
const response = await analyticsRoute.GET(makeRequest("http://localhost/api/usage/analytics"));
|
||||
const body = await response.json();
|
||||
|
||||
assert.equal(response.status, 200);
|
||||
assert.equal(body.summary.uniqueApiKeys, 1);
|
||||
assert.equal(body.byApiKey.length, 1);
|
||||
assert.equal(body.byApiKey[0].apiKeyId, apiKey.id);
|
||||
assert.equal(body.byApiKey[0].apiKeyName, "Alexander Averyanov");
|
||||
assert.deepEqual(body.byApiKey[0].historicalApiKeyNames.sort(), ["Averyanov", "Desktop"]);
|
||||
assert.equal(body.byApiKey[0].requests, 2);
|
||||
assert.equal(body.byApiKey[0].promptTokens, 300);
|
||||
assert.equal(body.byApiKey[0].completionTokens, 150);
|
||||
|
||||
const filteredResponse = await analyticsRoute.GET(
|
||||
makeRequest(`http://localhost/api/usage/analytics?apiKeyIds=${apiKey.id}`)
|
||||
);
|
||||
const filteredBody = await filteredResponse.json();
|
||||
|
||||
assert.equal(filteredResponse.status, 200);
|
||||
assert.equal(filteredBody.summary.totalRequests, 2);
|
||||
assert.equal(filteredBody.byApiKey.length, 1);
|
||||
assert.equal(filteredBody.byApiKey[0].apiKeyId, apiKey.id);
|
||||
});
|
||||
|
||||
test("GET /api/usage/analytics does not persist guessed API key attribution", async () => {
|
||||
await localDb.updatePricing({
|
||||
openai: { "gpt-4o": { input: 2.5, output: 10 } },
|
||||
});
|
||||
await apiKeysDb.createApiKey("Unrestricted Key", "machine1234567890");
|
||||
|
||||
const db = core.getDbInstance();
|
||||
db.prepare(
|
||||
`INSERT INTO usage_history (provider, model, connection_id, api_key_id, api_key_name, tokens_input, tokens_output, success, latency_ms, timestamp)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`
|
||||
).run("openai", "gpt-4o", "legacy-conn", null, null, 100, 50, 1, 200, new Date().toISOString());
|
||||
|
||||
const response = await analyticsRoute.GET(makeRequest("http://localhost/api/usage/analytics"));
|
||||
const body = await response.json();
|
||||
|
||||
assert.equal(response.status, 200);
|
||||
assert.equal(body.byApiKey.length, 0);
|
||||
|
||||
const row = db
|
||||
.prepare("SELECT api_key_id, api_key_name FROM usage_history WHERE connection_id = ?")
|
||||
.get("legacy-conn") as { api_key_id: string | null; api_key_name: string | null };
|
||||
assert.equal(row.api_key_id, null);
|
||||
assert.equal(row.api_key_name, null);
|
||||
});
|
||||
|
||||
test("GET /api/usage/analytics returns weeklyPattern for the costs dashboard", async () => {
|
||||
await seedAnalyticsData();
|
||||
|
||||
const response = await analyticsRoute.GET(makeRequest("http://localhost/api/usage/analytics"));
|
||||
const body = await response.json();
|
||||
|
||||
assert.equal(response.status, 200);
|
||||
assert.ok(Array.isArray(body.weeklyPattern));
|
||||
assert.equal(body.weeklyPattern.length, 7);
|
||||
assert.deepEqual(
|
||||
body.weeklyPattern.map((row) => row.day),
|
||||
["Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat"]
|
||||
);
|
||||
assert.ok(body.weeklyPattern.some((row) => row.totalTokens > 0 && row.avgTokens > 0));
|
||||
});
|
||||
|
||||
test("GET /api/usage/analytics includes activityMap for heatmap", async () => {
|
||||
await seedAnalyticsData();
|
||||
|
||||
const response = await analyticsRoute.GET(makeRequest("http://localhost/api/usage/analytics"));
|
||||
const body = await response.json();
|
||||
|
||||
assert.equal(response.status, 200);
|
||||
assert.ok(typeof body.activityMap === "object");
|
||||
assert.ok(Object.keys(body.activityMap).length > 0);
|
||||
});
|
||||
|
||||
test("GET /api/usage/analytics returns 500 on database errors", async () => {
|
||||
const response = await analyticsRoute.GET(makeRequest("http://localhost/api/usage/analytics"));
|
||||
const body = await response.json();
|
||||
|
||||
assert.equal(response.status, 200);
|
||||
assert.ok(body.summary.totalRequests === 0);
|
||||
});
|
||||
|
||||
test("GET /api/usage/analytics does not throw Unknown named parameter on short range (needsAggregated=false)", async () => {
|
||||
// Regression: shared params object leaked agg-only bindings (@sinceDate, @rawCutoffDate)
|
||||
// into queries that don't reference them, causing better-sqlite3 to throw.
|
||||
// A short range (1h) triggers needsAggregated=false because the entire window
|
||||
// falls within the raw-data-only period.
|
||||
const db = core.getDbInstance();
|
||||
const now = new Date();
|
||||
db.prepare(
|
||||
`INSERT INTO usage_history (provider, model, connection_id, tokens_input, tokens_output, success, latency_ms, timestamp)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?)`
|
||||
).run("openai", "gpt-4o", "test-conn", 100, 50, 1, 200, now.toISOString());
|
||||
|
||||
const response = await analyticsRoute.GET(
|
||||
makeRequest("http://localhost/api/usage/analytics?range=1h")
|
||||
);
|
||||
const body = await response.json();
|
||||
|
||||
assert.equal(response.status, 200);
|
||||
assert.equal(body.summary.totalRequests, 1);
|
||||
});
|
||||
|
||||
test("GET /api/usage/analytics does not throw Unknown named parameter with apiKey filter on long range", async () => {
|
||||
// Regression: Object.assign(presetParams, params) leaked all main-query bindings
|
||||
// into preset queries that only reference preset-prefixed placeholders.
|
||||
const apiKey = await apiKeysDb.createApiKey("Preset Key", "machine-preset1234");
|
||||
const db = core.getDbInstance();
|
||||
const now = new Date();
|
||||
|
||||
// Seed data old enough to trigger aggregated + preset path
|
||||
for (let i = 0; i < 5; i++) {
|
||||
const ts = new Date(now.getTime() - (35 + i) * 24 * 60 * 60 * 1000).toISOString();
|
||||
db.prepare(
|
||||
`INSERT INTO usage_history (provider, model, connection_id, api_key_id, api_key_name, tokens_input, tokens_output, success, latency_ms, timestamp)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`
|
||||
).run("openai", "gpt-4o", "test-conn", apiKey.id, apiKey.name, 100, 50, 1, 200, ts);
|
||||
}
|
||||
|
||||
const response = await analyticsRoute.GET(
|
||||
makeRequest(`http://localhost/api/usage/analytics?range=60d&apiKeyId=${apiKey.id}`)
|
||||
);
|
||||
const body = await response.json();
|
||||
|
||||
assert.equal(response.status, 200);
|
||||
// Core regression check: no "Unknown named parameter" error.
|
||||
// The exact count depends on raw-vs-aggregated boundary; we only need to
|
||||
// confirm the endpoint returns 200 without throwing.
|
||||
assert.ok(typeof body.summary.totalRequests === "number");
|
||||
});
|
||||
|
||||
@@ -522,3 +522,44 @@ test("clearPendingRequests allows fresh tracking after clearing", () => {
|
||||
assert.equal(pending.byModel["model-d (provider-w)"], 1);
|
||||
assert.ok(pending.details["conn-4"]);
|
||||
});
|
||||
|
||||
test("getUsageSummary counts total_requests from daily_usage_summary, not 1-per-row (#usage-stats-fix)", async () => {
|
||||
const db = core.getDbInstance();
|
||||
|
||||
// Insert a daily_usage_summary row with total_requests=50, 1000 input, 500 output.
|
||||
// With the old COUNT(*) query this would count as 1 request; with SUM(requests)
|
||||
// it must count as 50.
|
||||
db.prepare(`
|
||||
INSERT INTO daily_usage_summary (provider, model, date, total_requests, total_input_tokens, total_output_tokens, total_cost)
|
||||
VALUES ('openai', 'gpt-4', '2024-01-10', 50, 1000, 500, 0.02)
|
||||
`).run();
|
||||
|
||||
// Also insert one raw row so we can verify the UNION merges both legs.
|
||||
db.prepare(`
|
||||
INSERT INTO usage_history (timestamp, provider, model, tokens_input, tokens_output, success, latency_ms, service_tier)
|
||||
VALUES ('2024-01-20T10:00:00.000Z', 'openai', 'gpt-4', 100, 50, 1, 200, 'standard')
|
||||
`).run();
|
||||
|
||||
// Build a unified source with rawCutoffDate BETWEEN the two rows so both
|
||||
// legs are exercised: aggregated leg gets the Jan 10 row, raw leg gets the Jan 20 row.
|
||||
const { buildUnifiedSource } = await import("../../src/lib/db/usageAnalytics/sources.ts");
|
||||
const { getUsageSummary } = await import("../../src/lib/db/usageAnalytics.ts");
|
||||
const { unifiedSource, unifiedParams } = buildUnifiedSource({
|
||||
sinceIso: "2024-01-01T00:00:00.000Z",
|
||||
untilIso: null,
|
||||
rawCutoffDate: "2024-01-15", // raw leg starts at Jan 15; summary leg is before Jan 15
|
||||
apiKeyWhere: "",
|
||||
apiKeyParams: {},
|
||||
});
|
||||
|
||||
const summary = getUsageSummary(unifiedSource, unifiedParams);
|
||||
|
||||
// 50 from daily_usage_summary + 1 from raw usage_history = 51
|
||||
assert.equal(summary.totalRequests, 51, "totalRequests must be 50 (aggregated) + 1 (raw), not 1+1");
|
||||
// 1000 from daily_usage_summary + 100 from raw = 1100
|
||||
assert.equal(summary.promptTokens, 1100, "promptTokens must merge aggregated + raw token sums");
|
||||
// 500 from daily_usage_summary + 50 from raw = 550
|
||||
assert.equal(summary.completionTokens, 550, "completionTokens must merge aggregated + raw token sums");
|
||||
// All 51 requests are successful (aggregated leg hardcodes success=1, raw has success=1)
|
||||
assert.equal(summary.successfulRequests, 51, "successfulRequests must count all rolled-up requests as successful");
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user