pjandro/OmniRoute
1
0
Fork 0
mirror of https://github.com/diegosouzapw/OmniRoute.git synced 2026-07-31 04:12:10 +03:00
Code Issues Packages Projects Releases Wiki Activity

fix(6848): auto-cleanup for telemetry tables causing OOM (#6988)

Browse Source 
* fix(6848): add auto-cleanup for telemetry tables that grow without bound

Add retention-based cleanup for 4 tables that had no prune policy:
- domain_cost_history (timestamp INTEGER, unix epoch)
- compression_cache_stats (created_at DATETIME)
- xp_audit_log (created_at TEXT)
- compression_run_telemetry (timestamp INTEGER, unix epoch)

All default to 30-day retention, integrated into runAutoCleanup()
which runs on startup + every 6h via startCleanupScheduler().
Also runs VACUUM after startup cleanup to reclaim disk space.

6 unit tests covering retention boundary, no-op on recent data,
and DEFAULT_DATABASE_SETTINGS key existence.

Closes #6848

* ci: retrigger CI for Electron Package Smoke flaky test

* ci: retrigger flaky integration test (batch-e2e timeout)

* test(#6848): rewrite tests to call real cleanup functions with seeded DB data

Replace mock-only assertions with integration-style tests that seed data
into the isolated test DB, call the actual cleanup functions from
src/lib/db/cleanup.ts, and verify rows are correctly deleted.

All 6 tests now exercise real code paths:
- cleanupDomainCostHistory: verify old rows deleted, recent preserved
- cleanupCompressionCacheStats: verify old rows deleted, recent preserved
- cleanupXpAuditLog: verify old rows deleted, recent preserved
- cleanupCompressionRunTelemetry: ensure table + verify cleanup
- Combined: all 4 functions return 0 deletions when data is within retention
- DEFAULT_DATABASE_SETTINGS: verify new retention keys exist with value 30

* test(#6848): self-contained DATA_DIR isolation for the cleanup test

Builds on the existing rewrite (already correctly importing and calling
the real cleanupDomainCostHistory/cleanupCompressionCacheStats/
cleanupXpAuditLog/cleanupCompressionRunTelemetry from src/lib/db/cleanup.ts
with real seeded-row assertions instead of re-implementing the DELETE
inline) and closes the remaining gap: DATA_DIR isolation relied entirely
on the test:unit harness's `--import ./tests/_setup/isolateDataDir.ts`,
which is invisible from the test file itself.

Per CONTRIBUTING.md/CLAUDE.md, a single test file is documented to run
directly as `node --import tsx/esm --test tests/unit/<file>.test.ts` —
without the harness's isolation import, getDbInstance() resolves to the
developer's real ~/.omniroute/storage.sqlite, and this file's DELETE-based
cleanup calls operate on real rows, not test rows. Confirmed by running it
that way before this fix: it deleted 238 real compression_cache_stats rows
and 53 real xp_audit_log rows (assertions failed on the row counts, which
is how the gap surfaced) instead of the 2/3 rows the test itself inserted.

Fix: mkdtempSync + DATA_DIR override before the first src/lib/db/* import
(self-contained, matches the pattern in
tests/unit/duckduckgo-vqd-429-misclassification-6996.test.ts), plus
test.after() calling resetDbInstance() and removing the temp dir — the
repo's DB-test-cleanup rule (a dangling handle can hang the native test
runner). Re-ran after the fix: same 6/6 pass, now against an isolated
/tmp DB with the expected 3/2/3/2 row counts, in ~3s instead of ~15s.

Red-first proof: reset cleanupDomainCostHistory's cutoff to a no-op
(`cutoffEpoch = 0`, never matches a real row) — the dedicated
"cleanupDomainCostHistory: deletes rows older than retention window" test
failed as expected; restored and reran clean (6/6).

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
This commit is contained in:
Rafael Dias Zendron
2026-07-18 15:14:41 -03:00
committed by GitHub
parent 1843b34866
commit d415baa026
4 changed files with 318 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

104
src/lib/db/cleanup.ts
View File

@@ -244,6 +244,66 @@ export async function cleanupMemoryEntries(): Promise<CleanupResult> {
return result;
}
/**
* Clean up old domain_cost_history based on retention settings. (#6848)
* Uses unix-epoch `timestamp` column (INTEGER).
*/
export async function cleanupDomainCostHistory(): Promise<CleanupResult> {
const db = getDbInstance();
const retention = getRetentionSettings();
const retentionDays = retention.domainCostHistory;
const cutoffEpoch = Math.floor(Date.now() / 1000) - retentionDays * 86_400;
const result: CleanupResult = { deleted: 0, errors: 0 };
try {
const stmt = db.prepare("DELETE FROM domain_cost_history WHERE timestamp < ?");
const runResult = stmt.run(cutoffEpoch);
result.deleted = runResult.changes;
console.log(
`[Cleanup] Deleted ${result.deleted} domain_cost_history older than ${retentionDays} days`
);
} catch (err: unknown) {
console.error("[Cleanup] Error cleaning domain_cost_history:", err);
result.errors++;
}
return result;
}
/**
* Clean up old compression_cache_stats based on retention settings. (#6848)
* Uses `created_at` column (DATETIME string).
*/
export async function cleanupCompressionCacheStats(): Promise<CleanupResult> {
const db = getDbInstance();
const retention = getRetentionSettings();
const retentionDays = retention.compressionCacheStats;
const cutoffDate = new Date();
cutoffDate.setDate(cutoffDate.getDate() - retentionDays);
const cutoffISO = cutoffDate.toISOString();
const result: CleanupResult = { deleted: 0, errors: 0 };
try {
const stmt = db.prepare("DELETE FROM compression_cache_stats WHERE created_at < ?");
const runResult = stmt.run(cutoffISO);
result.deleted = runResult.changes;
console.log(
`[Cleanup] Deleted ${result.deleted} compression_cache_stats older than ${retentionDays} days`
);
} catch (err: unknown) {
console.error("[Cleanup] Error cleaning compression_cache_stats:", err);
result.errors++;
}
return result;
}
/**
* Clean up old xp_audit_log based on retention settings.
*/
@@ -274,6 +334,35 @@ export async function cleanupXpAuditLog(): Promise<CleanupResult> {
return result;
}
/**
* Clean up old compression_run_telemetry based on retention settings. (#6848)
* Uses unix-epoch `timestamp` column (INTEGER).
*/
export async function cleanupCompressionRunTelemetry(): Promise<CleanupResult> {
const db = getDbInstance();
const retention = getRetentionSettings();
const retentionDays = retention.compressionRunTelemetry;
const cutoffEpoch = Math.floor(Date.now() / 1000) - retentionDays * 86_400;
const result: CleanupResult = { deleted: 0, errors: 0 };
try {
const stmt = db.prepare("DELETE FROM compression_run_telemetry WHERE timestamp < ?");
const runResult = stmt.run(cutoffEpoch);
result.deleted = runResult.changes;
console.log(
`[Cleanup] Deleted ${result.deleted} compression_run_telemetry older than ${retentionDays} days`
);
} catch (err: unknown) {
console.error("[Cleanup] Error cleaning compression_run_telemetry:", err);
result.errors++;
}
return result;
}
/**
* Run all cleanup functions if auto-cleanup is enabled.
*/
@@ -300,7 +389,10 @@ export async function runAutoCleanup(): Promise<{
mcpAudit: await cleanupMcpAudit(),
a2aEvents: await cleanupA2aEvents(),
memoryEntries: await cleanupMemoryEntries(),
domainCostHistory: await cleanupDomainCostHistory(),
compressionCacheStats: await cleanupCompressionCacheStats(),
xpAuditLog: await cleanupXpAuditLog(),
compressionRunTelemetry: await cleanupCompressionRunTelemetry(),
proxyLogs: await cleanupProxyLogs(),
};
@@ -523,9 +615,7 @@ export async function cleanupProxyLogs(): Promise<CleanupResult> {
const runResult = stmt.run(cutoffISO);
result.deleted = runResult.changes;
console.log(
`[Cleanup] Deleted ${result.deleted} proxy_logs older than ${retentionDays} days`
);
console.log(`[Cleanup] Deleted ${result.deleted} proxy_logs older than ${retentionDays} days`);
} catch (err: unknown) {
console.error("[Cleanup] Error cleaning proxy_logs:", err);
result.errors++;
@@ -557,9 +647,7 @@ export function startCleanupScheduler(): void {
const proxyResult = await cleanupProxyLogs();
const totalDeleted = result.totalDeleted + proxyResult.deleted;
if (totalDeleted > 0) {
console.log(
`[Cleanup] Startup cleanup freed ${totalDeleted} rows. Running VACUUM...`
);
console.log(`[Cleanup] Startup cleanup freed ${totalDeleted} rows. Running VACUUM...`);
try {
const db = getDbInstance();
db.exec("VACUUM");
@@ -580,9 +668,7 @@ export function startCleanupScheduler(): void {
const proxyResult = await cleanupProxyLogs();
const totalDeleted = result.totalDeleted + proxyResult.deleted;
if (totalDeleted > 0) {
console.log(
`[Cleanup] Periodic cleanup freed ${totalDeleted} rows. Running VACUUM...`
);
console.log(`[Cleanup] Periodic cleanup freed ${totalDeleted} rows. Running VACUUM...`);
try {
const db = getDbInstance();
db.exec("VACUUM");

3
src/lib/db/databaseSettings.ts
View File

@@ -49,7 +49,10 @@ const LEGACY_FLAT_KEYS: {
callLogs: ["callLogs"],
usageHistory: ["usageHistory"],
memoryEntries: ["memoryEntries"],
domainCostHistory: ["domainCostHistory"],
compressionCacheStats: ["compressionCacheStats"],
xpAuditLog: ["xpAuditLog"],
compressionRunTelemetry: ["compressionRunTelemetry"],
autoCleanupEnabled: ["autoCleanupEnabled"],
},
aggregation: {

6
src/types/databaseSettings.ts
View File

@@ -46,7 +46,10 @@ export interface DatabaseSettings {
callLogs: number;
usageHistory: number;
memoryEntries: number;
domainCostHistory: number;
compressionCacheStats: number;
xpAuditLog: number;
compressionRunTelemetry: number;
autoCleanupEnabled: boolean;
};
@@ -107,7 +110,10 @@ export const DEFAULT_DATABASE_SETTINGS: Omit<DatabaseSettings, "location" | "sta
callLogs: 30,
usageHistory: 30,
memoryEntries: 30,
domainCostHistory: 30,
compressionCacheStats: 30,
xpAuditLog: 30,
compressionRunTelemetry: 30,
autoCleanupEnabled: true,
},
aggregation: {

214
tests/unit/telemetry-auto-cleanup-6848.test.ts Normal file
View File

@@ -0,0 +1,214 @@
/**
* Issue #6848 — Auto-cleanup for telemetry tables that grow without bound.
*
* domain_cost_history, compression_cache_stats, xp_audit_log, and
* compression_run_telemetry had no retention cleanup, causing unbounded
* DB growth and OOM crashes on relays with heavy traffic.
*
* These tests call the REAL cleanup functions against a real SQLite adapter
* (seeded with test data). If the production DELETE logic breaks, these
* tests WILL fail.
*
* DATA_DIR isolation is self-contained (mkdtempSync below), not dependent on
* the test:unit harness's `--import ./tests/_setup/isolateDataDir.ts`. This
* file calls real DELETE-based cleanup functions against getDbInstance(),
* which resolves to the developer's real ~/.omniroute/storage.sqlite when
* DATA_DIR is unset — running this file directly (as documented under
* "Running Tests": `node --import tsx/esm --test tests/unit/<file>.test.ts`)
* would otherwise delete real rows from that database, not test rows. Do NOT
* remove the DATA_DIR override below.
*/
import test from "node:test";
import assert from "node:assert/strict";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-6848-"));
process.env.DATA_DIR = TEST_DATA_DIR;
// Import the real functions — they use getDbInstance(), which resolves to the
// isolated temp DB created above (set before this import so the module's
// first getDbInstance() call already sees the isolated DATA_DIR).
const {
cleanupDomainCostHistory,
cleanupCompressionCacheStats,
cleanupXpAuditLog,
cleanupCompressionRunTelemetry,
} = await import("../../src/lib/db/cleanup.ts");
const { getDbInstance, resetDbInstance } = await import("../../src/lib/db/core.ts");
// Repo test rule: DB-touching tests must close the handle in test.after(),
// or the native test runner can hang indefinitely on a dangling connection.
test.after(() => {
resetDbInstance();
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
});
const DAY = 86_400; // seconds
/** Ensure compression_run_telemetry table exists (created lazily in production). */
function ensureTelemetryTable(): void {
const db = getDbInstance()!;
db.exec(`
CREATE TABLE IF NOT EXISTS compression_run_telemetry (
id INTEGER PRIMARY KEY AUTOINCREMENT,
timestamp INTEGER NOT NULL,
request_id TEXT,
model TEXT,
provider TEXT,
source TEXT,
tokens_before INTEGER NOT NULL,
tokens_after INTEGER NOT NULL,
ratio REAL,
cost_delta REAL,
output_styles TEXT,
output_style_bypass TEXT,
output_tokens INTEGER
)
`);
}
// ─── Tests ───────────────────────────────────────────────────────────────
test("#6848 cleanupDomainCostHistory: deletes rows older than retention window", async () => {
const db = getDbInstance()!;
const now = Math.floor(Date.now() / 1000);
const insert = db.prepare(
"INSERT INTO domain_cost_history (api_key_id, cost, timestamp) VALUES (?, ?, ?)"
);
// 3 old (40 days ago), 2 recent (5 days ago)
insert.run("key1", 1.0, now - 40 * DAY);
insert.run("key1", 2.0, now - 40 * DAY);
insert.run("key1", 3.0, now - 40 * DAY);
insert.run("key1", 4.0, now - 5 * DAY);
insert.run("key1", 5.0, now - 5 * DAY);
const result = await cleanupDomainCostHistory();
assert.strictEqual(result.deleted, 3);
assert.strictEqual(result.errors, 0);
const remaining = db.prepare("SELECT COUNT(*) as cnt FROM domain_cost_history").get() as {
cnt: number;
};
assert.strictEqual(remaining.cnt, 2);
});
test("#6848 cleanupCompressionCacheStats: deletes rows older than retention window", async () => {
const db = getDbInstance()!;
const oldDate = new Date(Date.now() - 40 * DAY * 1000).toISOString();
const recentDate = new Date(Date.now() - 5 * DAY * 1000).toISOString();
const insert = db.prepare(
"INSERT INTO compression_cache_stats (provider, compression_mode, created_at) VALUES (?, ?, ?)"
);
insert.run("openai", "auto", oldDate);
insert.run("openai", "auto", oldDate);
insert.run("anthropic", "auto", recentDate);
const result = await cleanupCompressionCacheStats();
assert.strictEqual(result.deleted, 2);
assert.strictEqual(result.errors, 0);
const remaining = db.prepare("SELECT COUNT(*) as cnt FROM compression_cache_stats").get() as {
cnt: number;
};
assert.strictEqual(remaining.cnt, 1);
});
test("#6848 cleanupXpAuditLog: deletes rows older than retention window", async () => {
const db = getDbInstance()!;
const oldDate = new Date(Date.now() - 40 * DAY * 1000).toISOString();
const recentDate = new Date(Date.now() - 5 * DAY * 1000).toISOString();
const insert = db.prepare(
"INSERT INTO xp_audit_log (api_key_id, action, xp_earned, created_at) VALUES (?, ?, ?, ?)"
);
insert.run("key1", "login", 10, oldDate);
insert.run("key1", "login", 10, oldDate);
insert.run("key1", "login", 10, oldDate);
insert.run("key1", "login", 10, recentDate);
const result = await cleanupXpAuditLog();
assert.strictEqual(result.deleted, 3);
assert.strictEqual(result.errors, 0);
const remaining = db.prepare("SELECT COUNT(*) as cnt FROM xp_audit_log").get() as { cnt: number };
assert.strictEqual(remaining.cnt, 1);
});
test("#6848 cleanupCompressionRunTelemetry: deletes rows older than retention window", async () => {
ensureTelemetryTable();
const db = getDbInstance()!;
const now = Math.floor(Date.now() / 1000);
const insert = db.prepare(
"INSERT INTO compression_run_telemetry (timestamp, tokens_before, tokens_after) VALUES (?, ?, ?)"
);
insert.run(now - 40 * DAY, 1000, 500);
insert.run(now - 40 * DAY, 2000, 800);
insert.run(now - 5 * DAY, 1500, 600);
const result = await cleanupCompressionRunTelemetry();
assert.strictEqual(result.deleted, 2);
assert.strictEqual(result.errors, 0);
const remaining = db.prepare("SELECT COUNT(*) as cnt FROM compression_run_telemetry").get() as {
cnt: number;
};
assert.strictEqual(remaining.cnt, 1);
});
test("#6848 no rows deleted when all data is within retention window (calls all 4 real functions)", async () => {
ensureTelemetryTable();
const db = getDbInstance()!;
const now = Math.floor(Date.now() / 1000);
const recentISO = new Date().toISOString();
db.prepare("INSERT INTO domain_cost_history (api_key_id, cost, timestamp) VALUES (?, ?, ?)").run(
"k",
1,
now - DAY
);
db.prepare(
"INSERT INTO compression_cache_stats (provider, compression_mode, created_at) VALUES (?, ?, ?)"
).run("p", "auto", recentISO);
db.prepare(
"INSERT INTO xp_audit_log (api_key_id, action, xp_earned, created_at) VALUES (?, ?, ?, ?)"
).run("k", "a", 5, recentISO);
db.prepare(
"INSERT INTO compression_run_telemetry (timestamp, tokens_before, tokens_after) VALUES (?, ?, ?)"
).run(now - DAY, 100, 50);
const r1 = await cleanupDomainCostHistory();
const r2 = await cleanupCompressionCacheStats();
const r3 = await cleanupXpAuditLog();
const r4 = await cleanupCompressionRunTelemetry();
assert.strictEqual(r1.deleted, 0);
assert.strictEqual(r2.deleted, 0);
assert.strictEqual(r3.deleted, 0);
assert.strictEqual(r4.deleted, 0);
});
test("#6848 DEFAULT_DATABASE_SETTINGS has new retention keys", async () => {
const mod = await import("../../src/types/databaseSettings.ts");
const defaults = mod.DEFAULT_DATABASE_SETTINGS.retention;
assert.ok(typeof defaults.domainCostHistory === "number");
assert.ok(typeof defaults.compressionCacheStats === "number");
assert.ok(typeof defaults.xpAuditLog === "number");
assert.ok(typeof defaults.compressionRunTelemetry === "number");
assert.strictEqual(defaults.domainCostHistory, 30);
assert.strictEqual(defaults.compressionCacheStats, 30);
assert.strictEqual(defaults.xpAuditLog, 30);
assert.strictEqual(defaults.compressionRunTelemetry, 30);
});