pjandro/OmniRoute
1
0
Fork 0
mirror of https://github.com/diegosouzapw/OmniRoute.git synced 2026-08-15 03:32:21 +03:00
Code Issues Packages Projects Releases Wiki Activity

fix(db): prune pre-migration backups so db_backups stops growing unbounded (#10423)

Browse Source 
* fix(db): prune pre-migration backups so db_backups stops growing unbounded

createPreMigrationBackup() wrote a VACUUM INTO snapshot on every migration run
and never pruned. On a long-lived instance db_backups/ reached 48.999 files /
204 GB against a 5,3 MB live database; a second devbox showed the same shape
(5.711 files / 24 GB).

The retention policy already existed in cleanupDbBackups() but nothing on the
migration path reached it — its only callers are backup.ts and the
/api/db-backups route, neither of which runs during a migration.

migrationRunner.ts cannot import backup.ts: core.ts imports migrationRunner.ts
and backup.ts imports core.ts, so that edge would close a cycle. The policy
therefore moves to a new core-free module, backupRetention.ts, which both call
sites share — cleanupDbBackups() now delegates to it rather than duplicating it.

At the migration call site the operator's maxFiles/retentionDays are read
through the adapter already open for the run; going through getDbInstance()
would re-enter database initialization. Pruning never throws, so housekeeping
cannot fail a migration.

Closes #10421

* chore(db): declare backupRetention as an intentionally-internal db module

check:db-rules requires every src/lib/db/ module to be either re-exported by
localDb.ts or listed in INTENTIONALLY_INTERNAL. backupRetention.ts is a shared
primitive consumed only by db/backup.ts and db/migrationRunner.ts — the same
category as the migrationRunner entry — so it belongs in the allowlist rather
than in the public re-export surface.

* test(db): include backupRetention in the audited INTENTIONALLY_INTERNAL list

check-db-rules-classification.test.ts freezes the exact membership of
INTENTIONALLY_INTERNAL, so adding the 40th entry has to be reflected there too
— the gate script and this test pin the same contract from opposite sides.

---------

Co-authored-by: Xiangzhe <bakryun0718@proton.me>
This commit is contained in:
Diego Rodrigues de Sa e Souza
2026-08-14 21:27:16 -03:00
committed by GitHub
parent 4b914ad297
commit 4a53a53277
7 changed files with 502 additions and 118 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/reference/ENVIRONMENT.md
View File

@@ -1195,8 +1195,8 @@ Provider quota endpoints, network tunnels (Tailscale, Ngrok, MITM debug proxy),
| `TAILSCALED_BIN` | _(auto-detect)_ | `src/lib/tailscaleTunnel.ts` | Explicit path to the `tailscaled` daemon binary. |
| `TAILSCALE_AUTHKEY` | _(unset)_ | `src/lib/tailscaleTunnel.ts` | Pre-shared Tailscale auth key for non-interactive / headless `tailscale up` (passed via `--auth-key=`). When unset, login falls back to the interactive browser auth URL. |
| `NGROK_AUTHTOKEN` | _(unset)_ | `src/lib/ngrokTunnel.ts` | Authenticates outbound ngrok tunnels. |
| `DB_BACKUP_MAX_FILES` | `20` | `src/lib/db/backup.ts` | Maximum SQLite backup files retained on disk. Overrides the value saved from Settings → Database backup retention. |
| `DB_BACKUP_RETENTION_DAYS` | `0` | `src/lib/db/backup.ts` | Maximum age (days) of retained backups. `0` disables age-based pruning. Overrides the value saved from Settings → Database backup retention. |
| `DB_BACKUP_MAX_FILES` | `20` | `src/lib/db/backup.ts`, `src/lib/db/migrationRunner.ts` | Maximum SQLite backup files retained on disk. Applies to manual/scheduled backups and to pre-migration snapshots. Overrides the value saved from Settings → Database backup retention. |
| `DB_BACKUP_RETENTION_DAYS` | `0` | `src/lib/db/backup.ts`, `src/lib/db/migrationRunner.ts` | Maximum age (days) of retained backups. `0` disables age-based pruning. Applies to manual/scheduled backups and to pre-migration snapshots. Overrides the value saved from Settings → Database backup retention. |
| `OMNIROUTE_BACKUP_SCHEDULE_JOB_INTERVAL_MS` | `30000` | `src/lib/jobs/backupScheduleJob.ts` | Tick interval (ms) of the server-side job that executes `backup-schedule.json`. Must stay well under the 1-minute cron granularity; values below `5000` or unparseable fall back to `30000`. |
| `OMNIROUTE_TLS_PROXY_URL` | _(unset)_ | `open-sse/services/chatgptTlsClient.ts` | Override the TLS sidecar URL for tests. Production should leave unset. |
| `CONTAINER_HOST` | `docker` | `scripts/check-permissions.sh` | Container runtime hint for the entrypoint permission check. Set to `podman` for any Podman topology. Because the container cannot determine whether the engine is local or reached through Podman Machine, the warning stays topology-neutral and points to `contrib/podman/README.md`. |

1
scripts/check/check-db-rules.mjs
View File

@@ -42,6 +42,7 @@ export const INTENTIONALLY_INTERNAL = new Set([
"accessTokens", // intentionally-internal: 4 rotas /api/cli/* (connect, whoami, tokens, tokens/[id]) + server/authz/accessTokenAuth.ts via import direto "@/lib/db/accessTokens" (Rule #2)
"apiKeyColumnFallbacks", // db-internal: importado só por db/apiKeys.ts (API_KEY_COLUMN_FALLBACKS — fallbacks de coluna split do apiKeys.ts)
"apiKeyUsageLimitFields", // db-internal: importado só por db/apiKeys.ts (helpers de campo de limite de uso split do apiKeys.ts; mig 101)
"backupRetention", // db-internal: importado só por db/backup.ts e db/migrationRunner.ts (política de retenção compartilhada; mora fora de backup.ts porque core.ts importa migrationRunner.ts — importar backup.ts de lá fecharia um ciclo, #10421)
"caseMapping", // db-internal: importado só por db/core.ts (toSnakeCase/toCamelCase/objToSnake — column-mapping snake↔camel split do core.ts, #4947)
"cleanup", // intentionally-internal: 3 API routes (purge-quota-snapshots, purge-call-logs, purge-detailed-logs)
"cliToolState", // intentionally-internal: 14+ API routes em /api/cli-tools/*-settings

131
src/lib/db/backup.ts
View File

@@ -14,6 +14,13 @@ import {
DATA_DIR,
} from "./core";
import { resetAllDbModuleState } from "./stateReset";
import {
MAX_DB_BACKUPS,
DEFAULT_DB_BACKUP_RETENTION_DAYS,
parsePositiveInt,
parseNonNegativeInt,
pruneBackupDirectory,
} from "./backupRetention";
import { isAutomatedTestProcess } from "@/shared/utils/testProcess";
type CountRow = { cnt?: number };
@@ -22,22 +29,8 @@ type CountRow = { cnt?: number };
let _lastBackupAt = 0;
const BACKUP_THROTTLE_MS = 60 * 60 * 1000; // 60 minutes
const MAX_DB_BACKUPS = 20;
const DEFAULT_DB_BACKUP_RETENTION_DAYS = 0;
const TRUE_ENV_VALUES = new Set(["1", "true", "yes", "on"]);
function parsePositiveInt(value: string | undefined, fallback: number) {
if (!value) return fallback;
const parsed = Number.parseInt(value, 10);
return Number.isInteger(parsed) && parsed > 0 ? parsed : fallback;
}
function parseNonNegativeInt(value: string | undefined, fallback: number) {
if (value === undefined) return fallback;
const parsed = Number.parseInt(value, 10);
return Number.isInteger(parsed) && parsed >= 0 ? parsed : fallback;
}
// #3834: the "Keep latest backups" UI value is persisted here so it survives a page
// refresh / the loadStorageHealth() refetch. A dedicated namespace avoids any
// cross-talk with the databaseSettings key_value store (which rewrites all of its own
@@ -108,108 +101,16 @@ function getBackupDir() {
return DB_BACKUPS_DIR || path.join(DATA_DIR, "db_backups");
}
function getBackupFamilyBase(filename: string) {
if (filename.endsWith("-wal") || filename.endsWith("-shm")) return filename.slice(0, -4);
if (filename.endsWith("-journal")) return filename.slice(0, -8);
return filename;
}
export function cleanupDbBackups(options?: {
maxFiles?: number;
retentionDays?: number;
backupDir?: string;
}) {
const backupDir = options?.backupDir ?? getBackupDir();
const maxFiles = options?.maxFiles ?? getDbBackupMaxFiles();
const retentionDays = options?.retentionDays ?? getDbBackupRetentionDays();
function collectBackupFamilies(backupDir: string) {
if (!fs.existsSync(backupDir)) return [];
const families = new Map<
string,
{
base: string;
hasPrimary: boolean;
primaryMtimeMs: number;
latestMtimeMs: number;
files: string[];
}
>();
for (const name of fs.readdirSync(backupDir)) {
if (!name.startsWith("db_")) continue;
const base = getBackupFamilyBase(name);
const filePath = path.join(backupDir, name);
let stat;
try {
stat = fs.statSync(filePath);
} catch {
continue;
}
const family = families.get(base) || {
base,
hasPrimary: false,
primaryMtimeMs: 0,
latestMtimeMs: 0,
files: [],
};
family.files.push(name);
family.latestMtimeMs = Math.max(family.latestMtimeMs, stat.mtimeMs);
if (name === base && name.endsWith(".sqlite")) {
family.hasPrimary = true;
family.primaryMtimeMs = stat.mtimeMs;
}
families.set(base, family);
}
return [...families.values()];
}
export function cleanupDbBackups(options?: { maxFiles?: number; retentionDays?: number }) {
const backupDir = getBackupDir();
if (!fs.existsSync(backupDir)) {
return {
deletedBackupFamilies: 0,
deletedFiles: 0,
keptBackupFamilies: 0,
maxFiles: options?.maxFiles ?? getDbBackupMaxFiles(),
retentionDays: options?.retentionDays ?? getDbBackupRetentionDays(),
};
}
const maxFiles = Math.max(1, options?.maxFiles ?? getDbBackupMaxFiles());
const retentionDays = Math.max(0, options?.retentionDays ?? getDbBackupRetentionDays());
const cutoffMs = retentionDays > 0 ? Date.now() - retentionDays * 24 * 60 * 60 * 1000 : 0;
const families = collectBackupFamilies(backupDir);
const primaryFamilies = families
.filter((family) => family.hasPrimary)
.sort((a, b) => b.primaryMtimeMs - a.primaryMtimeMs);
const keepPrimaryBases = new Set(primaryFamilies.slice(0, maxFiles).map((family) => family.base));
let deletedBackupFamilies = 0;
let deletedFiles = 0;
for (const family of families) {
const isOverflowPrimary = family.hasPrimary && !keepPrimaryBases.has(family.base);
const isExpired = retentionDays > 0 && family.latestMtimeMs < cutoffMs;
const isOrphan = !family.hasPrimary;
if (!isOverflowPrimary && !isExpired && !isOrphan) continue;
deletedBackupFamilies += 1;
for (const name of family.files) {
try {
fs.unlinkSync(path.join(backupDir, name));
deletedFiles += 1;
} catch {
/* ignore */
}
}
}
return {
deletedBackupFamilies,
deletedFiles,
keptBackupFamilies: collectBackupFamilies(backupDir).filter((family) => family.hasPrimary)
.length,
maxFiles,
retentionDays,
};
return pruneBackupDirectory({ backupDir, maxFiles, retentionDays });
}
function coerceBoolean(value: unknown): boolean | null {

158
src/lib/db/backupRetention.ts Normal file
View File

@@ -0,0 +1,158 @@
/**
* Backup retention primitives — pure filesystem work, no `core.ts` dependency.
*
* This module exists so BOTH backup call sites can share one retention policy:
*
* - `backup.ts` (manual/API/auto backups) — resolves the operator's settings from the
* database and delegates here.
* - `migrationRunner.ts` (pre-migration snapshots) — cannot import `backup.ts`, because
* `core.ts` already imports `migrationRunner.ts` and `backup.ts` imports `core.ts`;
* that edge would close a cycle. Keeping the policy here, free of `core`, lets the
* migration path prune without one.
*
* Before #10421 the migration path had no retention at all and `db_backups/` grew
* without bound (observed: 48.999 files / 204 GB against a 5,3 MB live database).
*/
import fs from "fs";
import path from "path";
export const MAX_DB_BACKUPS = 20;
export const DEFAULT_DB_BACKUP_RETENTION_DAYS = 0;
export function parsePositiveInt(value: string | undefined, fallback: number) {
if (!value) return fallback;
const parsed = Number.parseInt(value, 10);
return Number.isInteger(parsed) && parsed > 0 ? parsed : fallback;
}
export function parseNonNegativeInt(value: string | undefined, fallback: number) {
if (value === undefined) return fallback;
const parsed = Number.parseInt(value, 10);
return Number.isInteger(parsed) && parsed >= 0 ? parsed : fallback;
}
/**
* A backup "family" is the primary `.sqlite` file plus its SQLite sidecars
* (`-wal` / `-shm` / `-journal`). Retention operates on families so a sidecar is never
* orphaned from — or outlives — the snapshot it belongs to.
*/
export function getBackupFamilyBase(filename: string) {
if (filename.endsWith("-wal") || filename.endsWith("-shm")) return filename.slice(0, -4);
if (filename.endsWith("-journal")) return filename.slice(0, -8);
return filename;
}
export type BackupFamily = {
base: string;
hasPrimary: boolean;
primaryMtimeMs: number;
latestMtimeMs: number;
files: string[];
};
export function collectBackupFamilies(backupDir: string): BackupFamily[] {
if (!fs.existsSync(backupDir)) return [];
const families = new Map<string, BackupFamily>();
for (const name of fs.readdirSync(backupDir)) {
if (!name.startsWith("db_")) continue;
const base = getBackupFamilyBase(name);
const filePath = path.join(backupDir, name);
let stat;
try {
stat = fs.statSync(filePath);
} catch {
continue;
}
const family = families.get(base) || {
base,
hasPrimary: false,
primaryMtimeMs: 0,
latestMtimeMs: 0,
files: [],
};
family.files.push(name);
family.latestMtimeMs = Math.max(family.latestMtimeMs, stat.mtimeMs);
if (name === base && name.endsWith(".sqlite")) {
family.hasPrimary = true;
family.primaryMtimeMs = stat.mtimeMs;
}
families.set(base, family);
}
return [...families.values()];
}
export type PruneResult = {
deletedBackupFamilies: number;
deletedFiles: number;
keptBackupFamilies: number;
maxFiles: number;
retentionDays: number;
};
/**
* Delete backup families beyond `maxFiles` (newest kept), older than `retentionDays`
* (0 disables the age rule), or orphaned (sidecars whose primary is already gone).
*/
export function pruneBackupDirectory(options: {
backupDir: string;
maxFiles: number;
retentionDays: number;
}): PruneResult {
const { backupDir } = options;
const maxFiles = Math.max(1, options.maxFiles);
const retentionDays = Math.max(0, options.retentionDays);
if (!fs.existsSync(backupDir)) {
return {
deletedBackupFamilies: 0,
deletedFiles: 0,
keptBackupFamilies: 0,
maxFiles,
retentionDays,
};
}
const cutoffMs = retentionDays > 0 ? Date.now() - retentionDays * 24 * 60 * 60 * 1000 : 0;
const families = collectBackupFamilies(backupDir);
const primaryFamilies = families
.filter((family) => family.hasPrimary)
.sort((a, b) => b.primaryMtimeMs - a.primaryMtimeMs);
const keepPrimaryBases = new Set(primaryFamilies.slice(0, maxFiles).map((family) => family.base));
let deletedBackupFamilies = 0;
let deletedFiles = 0;
for (const family of families) {
const isOverflowPrimary = family.hasPrimary && !keepPrimaryBases.has(family.base);
const isExpired = retentionDays > 0 && family.latestMtimeMs < cutoffMs;
const isOrphan = !family.hasPrimary;
if (!isOverflowPrimary && !isExpired && !isOrphan) continue;
deletedBackupFamilies += 1;
for (const name of family.files) {
try {
fs.unlinkSync(path.join(backupDir, name));
deletedFiles += 1;
} catch {
/* ignore */
}
}
}
return {
deletedBackupFamilies,
deletedFiles,
keptBackupFamilies: collectBackupFamilies(backupDir).filter((family) => family.hasPrimary)
.length,
maxFiles,
retentionDays,
};
}

65
src/lib/db/migrationRunner.ts
View File

@@ -29,6 +29,15 @@ import {
OPTIONAL_FTS5_MIGRATION_VERSIONS,
} from "./migrationRunner/constants";
import { getExtraMigrationFiles } from "./migrationRunner/extraDirs";
// Retention primitives live in their own `core`-free module: `core.ts` imports this file,
// so importing `backup.ts` (which imports `core.ts`) here would close a dependency cycle.
import {
MAX_DB_BACKUPS,
DEFAULT_DB_BACKUP_RETENTION_DAYS,
parsePositiveInt,
parseNonNegativeInt,
pruneBackupDirectory,
} from "./backupRetention";
const isNodeTestRunnerChild = typeof process.env.NODE_TEST_CONTEXT === "string";
@@ -805,6 +814,56 @@ function rehomeLegacyVersionSlotMigrations(
return repaired;
}
/**
* Read a persisted `dbBackup` retention setting through the adapter that is ALREADY open
* for this migration run.
*
* `backup.ts`'s equivalent goes through `getDbInstance()`, which is unsafe here: this
* code runs from inside database initialization, so asking for the singleton would
* re-enter it. Reading off `db` keeps the same stored values without that risk. A DB too
* old to have `key_value` yet simply falls back to the default.
*/
function readStoredBackupSetting(db: SqliteAdapter, key: string, min: number): number | undefined {
try {
const row = db
.prepare("SELECT value FROM key_value WHERE namespace = ? AND key = ?")
.get("dbBackup", key) as { value?: string } | undefined;
if (!row?.value) return undefined;
const parsed = JSON.parse(row.value);
return Number.isInteger(parsed) && parsed >= min ? parsed : undefined;
} catch {
return undefined;
}
}
/**
* Enforce the backup retention budget after a pre-migration snapshot (#10421).
*
* Precedence matches `backup.ts`: env override → persisted operator setting → default.
* Never throws: a migration must not fail because housekeeping did.
*/
function pruneMigrationBackups(db: SqliteAdapter, backupDir: string): void {
try {
const maxFiles = process.env.DB_BACKUP_MAX_FILES
? parsePositiveInt(process.env.DB_BACKUP_MAX_FILES, MAX_DB_BACKUPS)
: (readStoredBackupSetting(db, "maxFiles", 1) ?? MAX_DB_BACKUPS);
const retentionDays = process.env.DB_BACKUP_RETENTION_DAYS
? parseNonNegativeInt(process.env.DB_BACKUP_RETENTION_DAYS, DEFAULT_DB_BACKUP_RETENTION_DAYS)
: (readStoredBackupSetting(db, "retentionDays", 0) ?? DEFAULT_DB_BACKUP_RETENTION_DAYS);
const result = pruneBackupDirectory({ backupDir, maxFiles, retentionDays });
if (result.deletedFiles > 0) {
console.log(
`[Migration] Pruned ${result.deletedFiles} old backup file(s) ` +
`(${result.keptBackupFamilies} kept, maxFiles=${maxFiles}, retentionDays=${retentionDays}).`
);
}
} catch (err: unknown) {
const message = err instanceof Error ? err.message : String(err);
console.warn(`[Migration] Failed to prune old backups: ${message}`);
}
}
/**
* Create a pre-migration backup of the SQLite database using VACUUM INTO.
* Returns the backup path on success, null on failure.
@@ -825,6 +884,12 @@ function createPreMigrationBackup(db: SqliteAdapter): string | null {
db.exec(`VACUUM INTO '${escapedBackupPath}'`);
console.log(`[Migration] Pre-migration backup created: ${backupPath}`);
// #10421: apply the operator's retention budget right here. Without this the
// migration path was the one backup producer that never pruned, so every process
// start with a pending migration added ~5 MB forever (observed: 49k files / 204 GB).
pruneMigrationBackups(db, backupDir);
return backupPath;
} catch (err: unknown) {
const message = err instanceof Error ? err.message : String(err);

3
tests/unit/check-db-rules-classification.test.ts
View File

@@ -121,12 +121,13 @@ test("INTENTIONALLY_INTERNAL is exported from check-db-rules.mjs", () => {
assert.ok(INTENTIONALLY_INTERNAL.size > 0, "INTENTIONALLY_INTERNAL must not be empty");
});
test("INTENTIONALLY_INTERNAL contains the expected 39 audited modules", () => {
test("INTENTIONALLY_INTERNAL contains the expected 40 audited modules", () => {
const expected = [
"_rowTypes",
"accessTokens",
"apiKeyColumnFallbacks",
"apiKeyUsageLimitFields",
"backupRetention",
"caseMapping",
"cleanup",
"cliToolState",

258
tests/unit/db-pre-migration-backup-retention-10421.test.ts Normal file
View File

@@ -0,0 +1,258 @@
// #10421 — pre-migration backups were created on every migration run and never pruned,
// so `db_backups/` grew without bound (observed: 48.999 files / 204 GB against a 5,3 MB
// live database). The pruning logic already existed in `cleanupDbBackups()` but nothing
// on the migration path ever reached it. These tests pin the retention step to the
// backup call site so the operator's maxFiles/retentionDays budget is honored there too.
import test from "node:test";
import assert from "node:assert/strict";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { pathToFileURL } from "node:url";
import Database from "better-sqlite3";
const serial = { concurrency: false };
async function importFresh(modulePath: string) {
const url = pathToFileURL(path.resolve(modulePath)).href;
return import(`${url}?test=${Date.now()}-${Math.random().toString(16).slice(2)}`);
}
function withMockedMigrationFs(files: Record<string, string>, fn: () => void) {
const originalExistsSync = fs.existsSync;
const originalReaddirSync = fs.readdirSync;
const originalReadFileSync = fs.readFileSync;
const isMigrationDir = (target: unknown) =>
String(target).replaceAll("\\", "/").endsWith("/src/lib/db/migrations") ||
String(target).replaceAll("\\", "/").endsWith("/migrations");
fs.existsSync = ((target: unknown) => {
if (isMigrationDir(target)) return true;
const fileName = path.basename(String(target));
if (Object.hasOwn(files, fileName)) return true;
return originalExistsSync(target as string);
}) as typeof fs.existsSync;
fs.readdirSync = ((target: string, options?: unknown) => {
if (isMigrationDir(target)) return Object.keys(files);
return originalReaddirSync(target, options as never);
}) as typeof fs.readdirSync;
fs.readFileSync = ((target: unknown, options?: unknown) => {
const fileName = path.basename(String(target));
if (Object.hasOwn(files, fileName)) return files[fileName];
return originalReadFileSync(target as string, options as never);
}) as typeof fs.readFileSync;
try {
return fn();
} finally {
fs.existsSync = originalExistsSync;
fs.readdirSync = originalReaddirSync;
fs.readFileSync = originalReadFileSync;
}
}
/** Minimal SqliteAdapter over a real on-disk file (VACUUM INTO needs a file, not :memory:). */
function createFileDb(sqlitePath: string) {
const db = new Database(sqlitePath);
return {
driver: "better-sqlite3",
get open() {
return db.open;
},
get name() {
return db.name;
},
prepare: (sql: string) => db.prepare(sql),
exec: (sql: string) => db.exec(sql),
pragma: (str: string, options?: unknown) => db.pragma(str, options as never),
transaction: (fn: (...args: unknown[]) => unknown) => {
const tx = db.transaction((...args: unknown[]) => fn(...args));
return (...args: unknown[]) => tx(...args);
},
immediate: (fn: () => void) => fn(),
async backup() {},
checkpoint() {},
close: () => db.close(),
get raw() {
return db;
},
};
}
/**
* Build a DB that already has migrations applied (so the pre-migration backup path is
* reached: it requires `applied.size > 0`) plus one pending migration to trigger a run.
*/
function seedAppliedDb(db: ReturnType<typeof createFileDb>) {
db.exec(`
CREATE TABLE provider_connections (id TEXT PRIMARY KEY);
CREATE TABLE combos (id TEXT PRIMARY KEY);
CREATE TABLE call_logs (id TEXT PRIMARY KEY);
`);
}
/**
* Record 001 as applied in the runner's own ledger table. `runMigrations` only takes a
* pre-migration backup when `applied.size > 0`, so this is what puts the test on the
* code path under exercise.
*/
function seedAppliedMigration(db: ReturnType<typeof createFileDb>) {
db.exec(`
CREATE TABLE IF NOT EXISTS _omniroute_migrations (
version TEXT PRIMARY KEY,
name TEXT NOT NULL,
applied_at TEXT NOT NULL DEFAULT (datetime('now'))
);
`);
db.prepare(
"INSERT OR REPLACE INTO _omniroute_migrations (version, name, applied_at) VALUES (?, ?, ?)"
).run("001", "initial_schema", new Date().toISOString());
}
function makeTempDataDir() {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-backup-retention-"));
fs.mkdirSync(path.join(dir, "db_backups"), { recursive: true });
return dir;
}
/** Pre-existing backups, oldest first, with distinct mtimes so retention ordering is stable. */
function seedBackups(backupDir: string, count: number) {
const names: string[] = [];
for (let i = 0; i < count; i++) {
const name = `db_2026-08-${String(i + 1).padStart(2, "0")}T00-00-00-000Z_pre-migration.sqlite`;
const filePath = path.join(backupDir, name);
fs.writeFileSync(filePath, "x");
const t = new Date(2026, 7, i + 1).getTime() / 1000;
fs.utimesSync(filePath, t, t);
names.push(name);
}
return names;
}
function countBackups(backupDir: string) {
return fs.readdirSync(backupDir).filter((n) => n.startsWith("db_")).length;
}
function withEnv(vars: Record<string, string | undefined>, fn: () => void) {
const saved: Record<string, string | undefined> = {};
for (const [k, v] of Object.entries(vars)) {
saved[k] = process.env[k];
if (v === undefined) delete process.env[k];
else process.env[k] = v;
}
try {
return fn();
} finally {
for (const [k, v] of Object.entries(saved)) {
if (v === undefined) delete process.env[k];
else process.env[k] = v;
}
}
}
test(
"#10421 runMigrations prunes pre-migration backups to the configured maxFiles",
serial,
async () => {
const dataDir = makeTempDataDir();
const backupDir = path.join(dataDir, "db_backups");
const sqlitePath = path.join(dataDir, "storage.sqlite");
const db = createFileDb(sqlitePath);
try {
seedAppliedDb(db);
seedBackups(backupDir, 30);
assert.equal(countBackups(backupDir), 30, "precondition: 30 stale backups on disk");
const { runMigrations } = await importFresh("src/lib/db/migrationRunner.ts");
withEnv(
{
DB_BACKUP_MAX_FILES: "5",
DB_BACKUP_RETENTION_DAYS: "0",
DISABLE_SQLITE_AUTO_BACKUP: undefined,
},
() => {
withMockedMigrationFs(
{
"001_initial_schema.sql": "SELECT 1;",
"002_retention_probe.sql": "CREATE TABLE retention_probe_10421 (id INTEGER);",
},
() => {
// Mark 001 as applied so `applied.size > 0` and the backup path is reached.
seedAppliedMigration(db);
runMigrations(db);
}
);
}
);
const remaining = countBackups(backupDir);
assert.ok(
remaining <= 5,
`expected retention to cap db_backups at 5 files, found ${remaining} — ` +
`pre-migration backups are accumulating unbounded (#10421)`
);
} finally {
db.close();
fs.rmSync(dataDir, { recursive: true, force: true });
}
}
);
test("#10421 the newest pre-migration backup survives pruning", serial, async () => {
const dataDir = makeTempDataDir();
const backupDir = path.join(dataDir, "db_backups");
const sqlitePath = path.join(dataDir, "storage.sqlite");
const db = createFileDb(sqlitePath);
try {
seedAppliedDb(db);
seedBackups(backupDir, 10);
const { runMigrations } = await importFresh("src/lib/db/migrationRunner.ts");
withEnv(
{
DB_BACKUP_MAX_FILES: "3",
DB_BACKUP_RETENTION_DAYS: "0",
DISABLE_SQLITE_AUTO_BACKUP: undefined,
},
() => {
withMockedMigrationFs(
{
"001_initial_schema.sql": "SELECT 1;",
"002_retention_probe.sql": "CREATE TABLE retention_probe_10421b (id INTEGER);",
},
() => {
seedAppliedMigration(db);
runMigrations(db);
}
);
}
);
const remaining = fs.readdirSync(backupDir).filter((n) => n.startsWith("db_"));
assert.ok(remaining.length <= 3, `expected <=3 backups, found ${remaining.length}`);
// The backup written by THIS run must be among the survivors — pruning must never
// discard the snapshot that protects the migration it was taken for.
const seededNames = new Set(
Array.from({ length: 10 }, (_, i) => {
return `db_2026-08-${String(i + 1).padStart(2, "0")}T00-00-00-000Z_pre-migration.sqlite`;
})
);
const fresh = remaining.filter((n) => !seededNames.has(n));
assert.equal(fresh.length, 1, `expected the run's own backup to survive, got ${fresh.length}`);
} finally {
db.close();
fs.rmSync(dataDir, { recursive: true, force: true });
}
});