From e9f297021ec00975ccfe85aebc3d56d26350cff6 Mon Sep 17 00:00:00 2001 From: growab Date: Thu, 23 Jul 2026 17:15:18 +0300 Subject: [PATCH] fix(usage): correct token/request counting for 30D/90D/YTD/ALL ranges (#7300) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore(ci): add .mergify.yml to main — Mergify only reads config from the default branch (#7168) * fix(ci): add the auto-enqueue pull_request_rule to the Mergify config (queue_conditions alone are eligibility-only) (#7179) * fix(ci): migrate Mergify auto-enqueue to merge_protections_settings.auto_merge_conditions (rules-based path is EOL 2026-07-16) (#7216) * fix(ci): drop Mergify batch settings (batching is a paid-tier feature; free plan queue is serial) (#7220) * fix(ci): merge queue tolerates the advisory dast-smoke failure (its GH-hosted build hang dequeued every attempt) (#7225) * fix(usage): correct token/request counting for 30D/90D/YTD/ALL ranges Two bugs caused incorrect usage statistics for date ranges beyond the raw data retention window: 1. Cutoff mismatch: the analytics route computed rawCutoffDate from aggregation.rawDataRetentionDays (migration 046 seeds =7) while cleanupUsageHistory rolls up and deletes at retention.usageHistory (=30). The window [day-30, day-7) existed in usage_history but was excluded from BOTH UNION legs — raw leg floored at day-7, aggregated leg ended at day-7 — producing undercounted token sums for 30D, 90D, YTD, and ALL ranges. Fix: use dbSettings.retention.usageHistory for the raw cutoff in both route.ts and getRawDataCutoffDate() (aggregateHistory.ts), matching the actual cleanup boundary. 2. Request undercount: COUNT(*) on the unified source counted each daily_usage_summary row as 1, not total_requests. A day with 50 rolled-up requests counted as 1. Fix: add a 'requests' column to both UNION legs (raw: 1, aggregated: total_requests), change COUNT(*) to SUM(requests) in 6 query functions, and change successfulRequests from SUM(CASE WHEN success=1 THEN 1 ELSE 0 END) to SUM(CASE WHEN success=1 THEN requests ELSE 0 END). Also set agg leg latency_ms to NULL so AVG(latency_ms) is not skewed. Tests: 33/33 source-level tests pass (db-usageanalytics-split.test.ts), verifying 'requests' column presence and SUM(requests) usage in all affected queries. DB-level integration test added to usage-analytics.test.ts (requires node + better-sqlite3). * fix(ci): green CI reds on #7300 — file-size ratchet, stale test fixture, shallow-checkout selfref test - src/app/api/usage/analytics/route.ts: trim the new comment to keep the file at the frozen file-size baseline (942 lines) after the retention.usageHistory cutoff fix — no logic change. - tests/unit/usage-analytics-route.test.ts: the pre-existing "does not double-count raw and aggregated rows" test hardcoded a 30-day cutoff that matched the OLD (buggy) aggregation.rawDataRetentionDays default. Now that the raw/aggregated boundary correctly uses retention.usageHistory (365 days by default, matching cleanupUsageHistory's actual rollup/delete boundary), the fixture's synthetic "old" row was within the raw window and got excluded from the aggregated leg. Read the real retention setting instead of hardcoding 30 so the fixture reflects the corrected boundary. Same assertions (still expects no double-counting, totalRequests=2, totalTokens=185) — only the fixture dates change. - tests/unit/check-test-masking-selfref-6634.test.ts: tolerate the shallow/ single-ref checkout used by GitHub-hosted Unit Tests runners (no local origin/main ref) by fetching it on demand and skipping (never failing) when unreachable offline. Matches the fix already applied on another branch (2e42b8efc/#7174) for the same root cause, not yet on main. Co-authored-by: diegosouzapw * test(ci): make the #6634 selfref test checkout-independent (read the real file, no git ref) The previous on-demand `git fetch origin main` + t.skip() fallback cleared the shallow-checkout failure but tripped the PR Test Policy's test-masking gate (a new .skip counts as a silenced assert — correctly so). Drop the git dependency entirely instead: read the REAL current source of tests/unit/check-test-masking.test.ts from disk (so the actual #6404 fixture literals stay under test) and model the pre-#6404 state with an empty base, which maximizes headTaut - baseTaut — the strictest input for the exclusion this test asserts. No skip, no weakened assertion, same deepEqual guarantee. Verified non-vacuous: neutralizing SELF_TEST_FIXTURE_RE in scripts/check/check-test-masking.mjs makes this test fail; restoring it makes it pass. Co-authored-by: diegosouzapw * test(ci): make the #6634 selfref guard hermetic — main's copy hard-fails every PR (#7341) main's copy of this test still does git I/O inside a unit test: const baseSrc = git(['show', 'origin/main:' + FILE]); Runners check out a shallow single ref, so origin/main does not resolve and the test dies with 'fatal: invalid object name origin/main'. Every PR into main fails Unit Tests (7/8) on it — today that is #7313, #7315, #7316, #7334, #7336 and #7337, six PRs red on a defect none of them introduced. #7313 has no other red at all. release/v3.8.49 already carries a fix (2e42b8efc, #7174: try/catch, fetch origin/main on demand, t.skip() when unreachable), but it only reaches main at release time — so main stays broken for the whole cycle. Cherry-picking it would also import a new problem: PR Test Policy classifies t.skip() as a silenced assertion, which we watched it correctly catch on #7300 today. This is the hermetic version instead (ported from #7327, which does the same for the release branch): read the file straight off disk, compare against an empty base so baseTaut/baseExtTaut are 0 — the strictest possible comparison point — and call evaluateMasking() directly. No git ref, no fetch, no skip, nothing the runner's checkout depth can break. The #6634 regression stays covered: the guard's logic lives in SELF_TEST_FIXTURE_RE (check-test-masking.mjs:337), not in the test. Proven both ways on main before committing — neutralise SELF_TEST_FIXTURE_RE to /$^/ and the test FAILS; restore it and it passes 2/2, with check-test-masking.mjs left byte-identical. Co-authored-by: growab * chore(quality): tighten main's coverage baseline to the CI's real numbers (#7347) main's ratchet had been failing --require-tighten on every PR: 11 metrics improved but the baseline was never tightened. Same class as the #6634 selfref guard — an infra fix that lands only on the release branch leaves main red for the whole cycle, and every PR into main pays for it. Values are the merged-coverage numbers from a run on main itself (a local run measures ~68% vs CI's ~80%; the baseline's own note warns about that gap). Only the 11 coverage values change — gitleaks and semgrepFindings keep main's own state. No changelog fragment: #7326 carries it on release/v3.8.49, and a second one here would double the entry at release time. --------- Co-authored-by: Diego Rodrigues de Sa e Souza <8016841+diegosouzapw@users.noreply.github.com> Co-authored-by: diegosouzapw --- src/app/api/usage/analytics/route.ts | 6 +- src/lib/db/usageAnalytics.ts | 18 +- src/lib/db/usageAnalytics/sources.ts | 14 +- src/lib/usage/aggregateHistory.ts | 8 +- tests/unit/db-usageanalytics-split.test.ts | 53 ++++- tests/unit/usage-analytics-route.test.ts | 257 +++++++++++++++++++++ tests/unit/usage-analytics.test.ts | 41 ++++ 7 files changed, 376 insertions(+), 21 deletions(-) diff --git a/src/app/api/usage/analytics/route.ts b/src/app/api/usage/analytics/route.ts index ae2c2e9d96..f26ef58a9a 100644 --- a/src/app/api/usage/analytics/route.ts +++ b/src/app/api/usage/analytics/route.ts @@ -347,10 +347,10 @@ export async function GET(request: Request) { } } - // Compute the raw-data cutoff: rows older than this may have been rolled up to - // daily_usage_summary and deleted from usage_history. + // Raw-data cutoff: must match cleanupUsageHistory's rollup/delete boundary — + // retention.usageHistory (src/lib/db/cleanup.ts), NOT aggregation.rawDataRetentionDays. const dbSettings = getUserDatabaseSettings(); - const rawRetentionDays = dbSettings.aggregation?.rawDataRetentionDays ?? 30; + const rawRetentionDays = dbSettings.retention?.usageHistory ?? 30; const rawCutoff = new Date(); rawCutoff.setDate(rawCutoff.getDate() - rawRetentionDays); const rawCutoffIso = rawCutoff.toISOString(); diff --git a/src/lib/db/usageAnalytics.ts b/src/lib/db/usageAnalytics.ts index 3e66795425..932001b771 100644 --- a/src/lib/db/usageAnalytics.ts +++ b/src/lib/db/usageAnalytics.ts @@ -49,14 +49,14 @@ export function getUsageSummary(unifiedSource: string, params: AnalyticsParams): .prepare( ` SELECT - COUNT(*) as totalRequests, + COALESCE(SUM(requests), 0) as totalRequests, COALESCE(SUM(tokens_input), 0) as promptTokens, COALESCE(SUM(tokens_output), 0) as completionTokens, COALESCE(SUM(tokens_input + tokens_output), 0) as totalTokens, COUNT(DISTINCT model) as uniqueModels, COUNT(DISTINCT COALESCE(NULLIF(account_key, ''), NULLIF(connection_id, ''))) as uniqueAccounts, COUNT(DISTINCT COALESCE(NULLIF(api_key_id, ''), NULLIF(api_key_name, ''))) as uniqueApiKeys, - COALESCE(SUM(CASE WHEN success = 1 THEN 1 ELSE 0 END), 0) as successfulRequests, + COALESCE(SUM(CASE WHEN success = 1 THEN requests ELSE 0 END), 0) as successfulRequests, COALESCE(AVG(latency_ms), 0) as avgLatencyMs, COALESCE(MIN(timestamp), '') as firstRequest, COALESCE(MAX(timestamp), '') as lastRequest @@ -101,7 +101,7 @@ export function getDailyUsage(unifiedSource: string, params: AnalyticsParams): D ` SELECT DATE(timestamp) as date, - COUNT(*) as requests, + COALESCE(SUM(requests), 0) as requests, COALESCE(SUM(tokens_input), 0) as promptTokens, COALESCE(SUM(tokens_output), 0) as completionTokens, COALESCE(SUM(tokens_input + tokens_output), 0) as totalTokens @@ -215,7 +215,7 @@ export function getModelUsageRows(unifiedSource: string, params: AnalyticsParams LOWER(model) as model, LOWER(provider) as provider, COALESCE(NULLIF(service_tier, ''), 'standard') as serviceTier, - COUNT(*) as requests, + COALESCE(SUM(requests), 0) as requests, COALESCE(SUM(tokens_input), 0) as promptTokens, COALESCE(SUM(tokens_output), 0) as completionTokens, COALESCE(SUM(tokens_cache_read), 0) as cacheReadTokens, @@ -223,7 +223,7 @@ export function getModelUsageRows(unifiedSource: string, params: AnalyticsParams COALESCE(SUM(tokens_reasoning), 0) as reasoningTokens, COALESCE(SUM(tokens_input + tokens_output), 0) as totalTokens, COALESCE(AVG(latency_ms), 0) as avgLatencyMs, - COALESCE(SUM(CASE WHEN success = 1 THEN 1 ELSE 0 END), 0) as successfulRequests, + COALESCE(SUM(CASE WHEN success = 1 THEN requests ELSE 0 END), 0) as successfulRequests, COALESCE(MAX(timestamp), '') as lastUsed FROM ${unifiedSource} AS _u GROUP BY LOWER(model), LOWER(provider), serviceTier @@ -298,12 +298,12 @@ export function getProviderUsageRows( ` SELECT LOWER(provider) as provider, - COUNT(*) as requests, + COALESCE(SUM(requests), 0) as requests, COALESCE(SUM(tokens_input), 0) as promptTokens, COALESCE(SUM(tokens_output), 0) as completionTokens, COALESCE(SUM(tokens_input + tokens_output), 0) as totalTokens, COALESCE(AVG(latency_ms), 0) as avgLatencyMs, - COALESCE(SUM(CASE WHEN success = 1 THEN 1 ELSE 0 END), 0) as successfulRequests + COALESCE(SUM(CASE WHEN success = 1 THEN requests ELSE 0 END), 0) as successfulRequests FROM ${unifiedSource} AS _u GROUP BY LOWER(provider) ORDER BY requests DESC @@ -553,7 +553,7 @@ export function getServiceTierUsageRows( LOWER(provider) as provider, LOWER(model) as model, COALESCE(NULLIF(service_tier, ''), 'standard') as serviceTier, - COUNT(*) as requests, + COALESCE(SUM(requests), 0) as requests, COALESCE(SUM(tokens_input), 0) as promptTokens, COALESCE(SUM(tokens_output), 0) as completionTokens, COALESCE(SUM(tokens_cache_read), 0) as cacheReadTokens, @@ -633,7 +633,7 @@ export function getWeeklyPatternRows( SELECT DATE(timestamp) as date, strftime('%w', timestamp) as dayOfWeek, - COUNT(*) as requests, + COALESCE(SUM(requests), 0) as requests, COALESCE(SUM(tokens_input + tokens_output), 0) as totalTokens FROM ${unifiedSource} AS _u GROUP BY DATE(timestamp), strftime('%w', timestamp) diff --git a/src/lib/db/usageAnalytics/sources.ts b/src/lib/db/usageAnalytics/sources.ts index d3a16da688..23ef171e9f 100644 --- a/src/lib/db/usageAnalytics/sources.ts +++ b/src/lib/db/usageAnalytics/sources.ts @@ -88,7 +88,6 @@ export function buildUnifiedSource(opts: BuildUnifiedSourceOptions): UnifiedSour unifiedParams.rawCutoffDate = rawCutoffDate; } const aggWhere = aggConditions.length > 0 ? `WHERE ${aggConditions.join(" AND ")}` : ""; - const unifiedSource = needsAggregated ? `( SELECT @@ -106,7 +105,8 @@ export function buildUnifiedSource(opts: BuildUnifiedSourceOptions): UnifiedSour connection_id, account_key, api_key_id, - api_key_name + api_key_name, + 1 as requests FROM usage_history ${rawWhere} UNION ALL @@ -121,11 +121,12 @@ export function buildUnifiedSource(opts: BuildUnifiedSourceOptions): UnifiedSour 0 as tokens_reasoning, 'standard' as service_tier, 1 as success, - 0 as latency_ms, + NULL as latency_ms, NULL as connection_id, NULL as account_key, NULL as api_key_id, - NULL as api_key_name + NULL as api_key_name, + total_requests as requests FROM daily_usage_summary ${aggWhere} )` @@ -134,10 +135,11 @@ export function buildUnifiedSource(opts: BuildUnifiedSourceOptions): UnifiedSour tokens_input, tokens_output, tokens_cache_read, tokens_cache_creation, tokens_reasoning, service_tier, success, latency_ms, - connection_id, account_key, api_key_id, api_key_name + connection_id, account_key, api_key_id, api_key_name, + 1 as requests FROM usage_history ${rawWhere} - )`; + )`; return { unifiedSource, unifiedParams }; } diff --git a/src/lib/usage/aggregateHistory.ts b/src/lib/usage/aggregateHistory.ts index b541540016..8e543d5ef8 100644 --- a/src/lib/usage/aggregateHistory.ts +++ b/src/lib/usage/aggregateHistory.ts @@ -196,7 +196,13 @@ export async function rollupUsageHistoryBeforeDate(beforeDate: string): Promise< * @returns ISO date string (YYYY-MM-DD) */ export async function getRawDataCutoffDate(): Promise { - const rawDataRetentionDays = getUserDatabaseSettings().aggregation.rawDataRetentionDays; + // The raw-data cutoff MUST match the actual rollup/delete boundary used by + // cleanupUsageHistory (src/lib/db/cleanup.ts), which is driven by + // retention.usageHistory — NOT aggregation.rawDataRetentionDays. + // Using rawDataRetentionDays (default 7 per migration 046) creates a gap: + // analytics floors raw data at day-7 while cleanup doesn't roll up until + // day-30, so the window [day-30, day-7) is excluded from BOTH UNION legs. + const rawDataRetentionDays = getUserDatabaseSettings().retention.usageHistory; const cutoffDate = new Date(); cutoffDate.setDate(cutoffDate.getDate() - rawDataRetentionDays); diff --git a/tests/unit/db-usageanalytics-split.test.ts b/tests/unit/db-usageanalytics-split.test.ts index 70271c71c9..095055fd3b 100644 --- a/tests/unit/db-usageanalytics-split.test.ts +++ b/tests/unit/db-usageanalytics-split.test.ts @@ -42,6 +42,10 @@ describe("usageAnalytics/sources — buildUnifiedSource", () => { ); assert.equal(unifiedParams.rawCutoff, "2024-06-15"); assert.equal(unifiedParams.rawCutoffDate, "2024-06-15"); + // Regression: unified source must surface a `requests` column in both legs + // so COUNT(*)→SUM(requests) counts actual requests, not row count. + assert.ok(unifiedSource.includes("1 as requests"), "raw leg must surface requests column"); + assert.ok(unifiedSource.includes("total_requests as requests"), "aggregated leg must surface total_requests as requests"); }); it("drops the aggregated leg when an api-key filter is active (raw-only)", () => { @@ -57,10 +61,11 @@ describe("usageAnalytics/sources — buildUnifiedSource", () => { !unifiedSource.includes("daily_usage_summary"), "aggregated leg must be absent once an api-key filter scopes the query to raw rows" ); - // raw leg floors at @since (not @rawCutoff) and carries the api-key params + // Regression: raw-only fallback must also surface the requests column. + assert.ok(unifiedSource.includes("1 as requests"), "raw-only fallback must surface requests column"); + assert.equal(unifiedParams.rawCutoff, undefined); assert.equal(unifiedParams.since, "2024-06-01T00:00:00.000Z"); assert.equal(unifiedParams.apiKey0, "k1"); - assert.equal(unifiedParams.rawCutoff, undefined); }); it("buildPresetUnifiedSource returns the unifiedSource/unifiedParams shape", () => { @@ -119,6 +124,50 @@ describe("usageAnalytics.ts public API surface", () => { }); }); +// ── 4. Regression: COUNT(*) → SUM(requests) in all query functions ────────── +// Verifies the fix without needing a DB driver — the function source must +// contain SUM(requests), not COUNT(*), so aggregated rows with total_requests>1 +// are counted correctly instead of as 1-per-row. + +describe("usageAnalytics queries use SUM(requests) not COUNT(*)", () => { + const queriesUsingCount = [ + "getUsageSummary", + "getDailyUsage", + "getModelUsageRows", + "getProviderUsageRows", + "getServiceTierUsageRows", + ]; + + for (const fn of queriesUsingCount) { + it(`${fn} uses SUM(requests) not COUNT(*)`, () => { + const src = host[fn].toString(); + assert.ok(src.includes("SUM(requests)"), `${fn} must use SUM(requests) for correct aggregated row counting`); + assert.ok(!src.match(/COUNT\(\*\)/), `${fn} must not use COUNT(*) which undercounts aggregated rows`); + }); + } + + it("getWeeklyPatternRows inner subquery uses SUM(requests)", () => { + const src = host.getWeeklyPatternRows.toString(); + assert.ok(src.includes("SUM(requests)"), "getWeeklyPatternRows inner subquery must use SUM(requests)"); + }); + + it("getUsageSummary uses SUM(CASE WHEN success=1 THEN requests ELSE 0 END) for successfulRequests", () => { + const src = host.getUsageSummary.toString(); + assert.ok( + src.includes("SUM(CASE WHEN success = 1 THEN requests ELSE 0 END)"), + "successfulRequests must weight by requests, not count 1-per-row" + ); + }); + + it("getModelUsageRows uses SUM(CASE WHEN success=1 THEN requests ELSE 0 END) for successfulRequests", () => { + const src = host.getModelUsageRows.toString(); + assert.ok( + src.includes("SUM(CASE WHEN success = 1 THEN requests ELSE 0 END)"), + "successfulRequests must weight by requests, not count 1-per-row" + ); + }); +}); + // ── 3. sources leaf exports the builders directly ──────────────────────────── describe("sources.ts exports the builders directly", () => { diff --git a/tests/unit/usage-analytics-route.test.ts b/tests/unit/usage-analytics-route.test.ts index 016947e483..8e57e7e88b 100644 --- a/tests/unit/usage-analytics-route.test.ts +++ b/tests/unit/usage-analytics-route.test.ts @@ -333,3 +333,260 @@ test("GET /api/usage/analytics includes byAccount array with cost data", async ( ); }); +test("GET /api/usage/analytics includes cost by API key", async () => { + await seedAnalyticsData(); + + const response = await analyticsRoute.GET(makeRequest("http://localhost/api/usage/analytics")); + const body = await response.json(); + + assert.equal(response.status, 200); + assert.ok(Array.isArray(body.byApiKey)); + assert.equal(body.byApiKey.length, 1); + assert.equal(body.byApiKey[0].apiKeyId, "test-key"); + assert.equal(body.byApiKey[0].apiKeyName, "Primary Key"); + assertClose(body.byApiKey[0].cost, body.summary.totalCost); +}); + +test("GET /api/usage/analytics does not double-count raw and aggregated rows", async () => { + const db = core.getDbInstance(); + const today = new Date(); + const todayStr = today.toISOString().split("T")[0]; + // The raw/aggregated split boundary is retention.usageHistory (365 days by + // default — see cleanupUsageHistory in src/lib/db/cleanup.ts, which rolls up + // and deletes usage_history rows using that exact setting). Read it live + // instead of hardcoding 30 days so this fixture stays valid regardless of + // the configured retention window. + const { getUserDatabaseSettings } = await import("../../src/lib/db/databaseSettings.ts"); + const rawRetentionDays = getUserDatabaseSettings().retention.usageHistory; + const cutoffDate = new Date(); + cutoffDate.setDate(cutoffDate.getDate() - rawRetentionDays); + const olderDate = new Date(cutoffDate); + olderDate.setDate(olderDate.getDate() - 1); + const olderDateStr = olderDate.toISOString().split("T")[0]; + + db.prepare( + `INSERT INTO usage_history (provider, model, connection_id, tokens_input, tokens_output, success, latency_ms, timestamp) + VALUES (?, ?, ?, ?, ?, ?, ?, ?)` + ).run("openai", "gpt-4o", "raw-current", 100, 50, 1, 200, today.toISOString()); + + const insertSummary = db.prepare( + `INSERT INTO daily_usage_summary (provider, model, date, total_requests, total_input_tokens, total_output_tokens, total_cost) + VALUES (?, ?, ?, ?, ?, ?, ?)` + ); + insertSummary.run("openai", "gpt-4o", todayStr, 99, 9900, 9900, 0); + insertSummary.run("openai", "gpt-4o", olderDateStr, 1, 25, 10, 0); + + const response = await analyticsRoute.GET( + makeRequest("http://localhost/api/usage/analytics?range=all") + ); + const body = await response.json(); + + assert.equal(response.status, 200); + assert.equal(body.summary.totalRequests, 2); + assert.equal(body.summary.totalTokens, 185); +}); + +test("GET /api/usage/analytics omits global aggregates when filtering by API key", async () => { + const apiKey = await apiKeysDb.createApiKey("Scoped Key", "machine1234567890"); + const db = core.getDbInstance(); + + db.prepare( + `INSERT INTO usage_history (provider, model, connection_id, api_key_id, api_key_name, tokens_input, tokens_output, success, latency_ms, timestamp) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)` + ).run( + "openai", + "gpt-4o", + "scoped-conn", + apiKey.id, + "Scoped Key", + 100, + 50, + 1, + 200, + new Date().toISOString() + ); + + db.prepare( + `INSERT INTO daily_usage_summary (provider, model, date, total_requests, total_input_tokens, total_output_tokens, total_cost) + VALUES (?, ?, ?, ?, ?, ?, ?)` + ).run("openai", "gpt-4o", "2024-01-01", 99, 9900, 9900, 0); + + const response = await analyticsRoute.GET( + makeRequest(`http://localhost/api/usage/analytics?range=all&apiKeyIds=${apiKey.id}`) + ); + const body = await response.json(); + + assert.equal(response.status, 200); + assert.equal(body.summary.totalRequests, 1); + assert.equal(body.summary.totalTokens, 150); + assert.equal(body.byApiKey.length, 1); + assert.equal(body.byApiKey[0].apiKeyId, apiKey.id); +}); + +test("GET /api/usage/analytics groups renamed API key usage by stable ID", async () => { + const apiKey = await apiKeysDb.createApiKey("Averyanov", "machine1234567890"); + await apiKeysDb.updateApiKeyPermissions(apiKey.id, { name: "Alexander Averyanov" }); + + const db = core.getDbInstance(); + const now = Date.now(); + const insertUsage = db.prepare( + `INSERT INTO usage_history (provider, model, connection_id, api_key_id, api_key_name, tokens_input, tokens_output, success, latency_ms, timestamp) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)` + ); + insertUsage.run( + "openai", + "gpt-4o", + "test-conn", + apiKey.id, + "Averyanov", + 100, + 50, + 1, + 200, + new Date(now - 60_000).toISOString() + ); + insertUsage.run( + "openai", + "gpt-4o", + "test-conn", + apiKey.id, + "Desktop", + 200, + 100, + 1, + 250, + new Date(now).toISOString() + ); + + const response = await analyticsRoute.GET(makeRequest("http://localhost/api/usage/analytics")); + const body = await response.json(); + + assert.equal(response.status, 200); + assert.equal(body.summary.uniqueApiKeys, 1); + assert.equal(body.byApiKey.length, 1); + assert.equal(body.byApiKey[0].apiKeyId, apiKey.id); + assert.equal(body.byApiKey[0].apiKeyName, "Alexander Averyanov"); + assert.deepEqual(body.byApiKey[0].historicalApiKeyNames.sort(), ["Averyanov", "Desktop"]); + assert.equal(body.byApiKey[0].requests, 2); + assert.equal(body.byApiKey[0].promptTokens, 300); + assert.equal(body.byApiKey[0].completionTokens, 150); + + const filteredResponse = await analyticsRoute.GET( + makeRequest(`http://localhost/api/usage/analytics?apiKeyIds=${apiKey.id}`) + ); + const filteredBody = await filteredResponse.json(); + + assert.equal(filteredResponse.status, 200); + assert.equal(filteredBody.summary.totalRequests, 2); + assert.equal(filteredBody.byApiKey.length, 1); + assert.equal(filteredBody.byApiKey[0].apiKeyId, apiKey.id); +}); + +test("GET /api/usage/analytics does not persist guessed API key attribution", async () => { + await localDb.updatePricing({ + openai: { "gpt-4o": { input: 2.5, output: 10 } }, + }); + await apiKeysDb.createApiKey("Unrestricted Key", "machine1234567890"); + + const db = core.getDbInstance(); + db.prepare( + `INSERT INTO usage_history (provider, model, connection_id, api_key_id, api_key_name, tokens_input, tokens_output, success, latency_ms, timestamp) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)` + ).run("openai", "gpt-4o", "legacy-conn", null, null, 100, 50, 1, 200, new Date().toISOString()); + + const response = await analyticsRoute.GET(makeRequest("http://localhost/api/usage/analytics")); + const body = await response.json(); + + assert.equal(response.status, 200); + assert.equal(body.byApiKey.length, 0); + + const row = db + .prepare("SELECT api_key_id, api_key_name FROM usage_history WHERE connection_id = ?") + .get("legacy-conn") as { api_key_id: string | null; api_key_name: string | null }; + assert.equal(row.api_key_id, null); + assert.equal(row.api_key_name, null); +}); + +test("GET /api/usage/analytics returns weeklyPattern for the costs dashboard", async () => { + await seedAnalyticsData(); + + const response = await analyticsRoute.GET(makeRequest("http://localhost/api/usage/analytics")); + const body = await response.json(); + + assert.equal(response.status, 200); + assert.ok(Array.isArray(body.weeklyPattern)); + assert.equal(body.weeklyPattern.length, 7); + assert.deepEqual( + body.weeklyPattern.map((row) => row.day), + ["Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat"] + ); + assert.ok(body.weeklyPattern.some((row) => row.totalTokens > 0 && row.avgTokens > 0)); +}); + +test("GET /api/usage/analytics includes activityMap for heatmap", async () => { + await seedAnalyticsData(); + + const response = await analyticsRoute.GET(makeRequest("http://localhost/api/usage/analytics")); + const body = await response.json(); + + assert.equal(response.status, 200); + assert.ok(typeof body.activityMap === "object"); + assert.ok(Object.keys(body.activityMap).length > 0); +}); + +test("GET /api/usage/analytics returns 500 on database errors", async () => { + const response = await analyticsRoute.GET(makeRequest("http://localhost/api/usage/analytics")); + const body = await response.json(); + + assert.equal(response.status, 200); + assert.ok(body.summary.totalRequests === 0); +}); + +test("GET /api/usage/analytics does not throw Unknown named parameter on short range (needsAggregated=false)", async () => { + // Regression: shared params object leaked agg-only bindings (@sinceDate, @rawCutoffDate) + // into queries that don't reference them, causing better-sqlite3 to throw. + // A short range (1h) triggers needsAggregated=false because the entire window + // falls within the raw-data-only period. + const db = core.getDbInstance(); + const now = new Date(); + db.prepare( + `INSERT INTO usage_history (provider, model, connection_id, tokens_input, tokens_output, success, latency_ms, timestamp) + VALUES (?, ?, ?, ?, ?, ?, ?, ?)` + ).run("openai", "gpt-4o", "test-conn", 100, 50, 1, 200, now.toISOString()); + + const response = await analyticsRoute.GET( + makeRequest("http://localhost/api/usage/analytics?range=1h") + ); + const body = await response.json(); + + assert.equal(response.status, 200); + assert.equal(body.summary.totalRequests, 1); +}); + +test("GET /api/usage/analytics does not throw Unknown named parameter with apiKey filter on long range", async () => { + // Regression: Object.assign(presetParams, params) leaked all main-query bindings + // into preset queries that only reference preset-prefixed placeholders. + const apiKey = await apiKeysDb.createApiKey("Preset Key", "machine-preset1234"); + const db = core.getDbInstance(); + const now = new Date(); + + // Seed data old enough to trigger aggregated + preset path + for (let i = 0; i < 5; i++) { + const ts = new Date(now.getTime() - (35 + i) * 24 * 60 * 60 * 1000).toISOString(); + db.prepare( + `INSERT INTO usage_history (provider, model, connection_id, api_key_id, api_key_name, tokens_input, tokens_output, success, latency_ms, timestamp) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)` + ).run("openai", "gpt-4o", "test-conn", apiKey.id, apiKey.name, 100, 50, 1, 200, ts); + } + + const response = await analyticsRoute.GET( + makeRequest(`http://localhost/api/usage/analytics?range=60d&apiKeyId=${apiKey.id}`) + ); + const body = await response.json(); + + assert.equal(response.status, 200); + // Core regression check: no "Unknown named parameter" error. + // The exact count depends on raw-vs-aggregated boundary; we only need to + // confirm the endpoint returns 200 without throwing. + assert.ok(typeof body.summary.totalRequests === "number"); +}); diff --git a/tests/unit/usage-analytics.test.ts b/tests/unit/usage-analytics.test.ts index ff7e486ccc..fa6c969649 100644 --- a/tests/unit/usage-analytics.test.ts +++ b/tests/unit/usage-analytics.test.ts @@ -522,3 +522,44 @@ test("clearPendingRequests allows fresh tracking after clearing", () => { assert.equal(pending.byModel["model-d (provider-w)"], 1); assert.ok(pending.details["conn-4"]); }); + +test("getUsageSummary counts total_requests from daily_usage_summary, not 1-per-row (#usage-stats-fix)", async () => { + const db = core.getDbInstance(); + + // Insert a daily_usage_summary row with total_requests=50, 1000 input, 500 output. + // With the old COUNT(*) query this would count as 1 request; with SUM(requests) + // it must count as 50. + db.prepare(` + INSERT INTO daily_usage_summary (provider, model, date, total_requests, total_input_tokens, total_output_tokens, total_cost) + VALUES ('openai', 'gpt-4', '2024-01-10', 50, 1000, 500, 0.02) + `).run(); + + // Also insert one raw row so we can verify the UNION merges both legs. + db.prepare(` + INSERT INTO usage_history (timestamp, provider, model, tokens_input, tokens_output, success, latency_ms, service_tier) + VALUES ('2024-01-20T10:00:00.000Z', 'openai', 'gpt-4', 100, 50, 1, 200, 'standard') + `).run(); + + // Build a unified source with rawCutoffDate BETWEEN the two rows so both + // legs are exercised: aggregated leg gets the Jan 10 row, raw leg gets the Jan 20 row. + const { buildUnifiedSource } = await import("../../src/lib/db/usageAnalytics/sources.ts"); + const { getUsageSummary } = await import("../../src/lib/db/usageAnalytics.ts"); + const { unifiedSource, unifiedParams } = buildUnifiedSource({ + sinceIso: "2024-01-01T00:00:00.000Z", + untilIso: null, + rawCutoffDate: "2024-01-15", // raw leg starts at Jan 15; summary leg is before Jan 15 + apiKeyWhere: "", + apiKeyParams: {}, + }); + + const summary = getUsageSummary(unifiedSource, unifiedParams); + + // 50 from daily_usage_summary + 1 from raw usage_history = 51 + assert.equal(summary.totalRequests, 51, "totalRequests must be 50 (aggregated) + 1 (raw), not 1+1"); + // 1000 from daily_usage_summary + 100 from raw = 1100 + assert.equal(summary.promptTokens, 1100, "promptTokens must merge aggregated + raw token sums"); + // 500 from daily_usage_summary + 50 from raw = 550 + assert.equal(summary.completionTokens, 550, "completionTokens must merge aggregated + raw token sums"); + // All 51 requests are successful (aggregated leg hardcodes success=1, raw has success=1) + assert.equal(summary.successfulRequests, 51, "successfulRequests must count all rolled-up requests as successful"); +});