pjandro/OmniRoute
1
0
Fork 0
mirror of https://github.com/diegosouzapw/OmniRoute.git synced 2026-09-16 11:52:26 +03:00
Code Issues Packages Projects Releases Wiki Activity

fix(db): harden migration recovery snapshots

Browse Source
This commit is contained in:
diegosouzapw
2026-09-02 04:29:27 -03:00
parent 713440be0a
commit 4700ae4068
14 changed files with 1879 additions and 391 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
.env.example
View File

@@ -55,10 +55,11 @@ INITIAL_PASSWORD=CHANGEME
# loader (bin/cli/plugins.mjs) at a package tree — this one drives the server-side scanner.
# OMNIROUTE_PLUGINS_DIR=/opt/omniroute/plugins
# Escape hatch for the test-context DATA_DIR guard (#10428). A test run that never
# chose a DATA_DIR is redirected to a throwaway temp dir so it cannot open the
# operator's real database. Set to 1 only for a deliberate run against the real
# DATA_DIR — never for CI. Used by: src/lib/dataPaths.ts
# Escape hatch for the test/eval DATA_DIR guard (#10428). A test or node eval/print
# probe (-e/--eval/-p/--print, including --eval=/--print=) that never chose a DATA_DIR
# is redirected to a throwaway temp dir so it cannot open the operator's real database.
# Set to 1 only for a deliberate run against the real DATA_DIR — never for CI.
# Used by: src/lib/dataPaths.ts
# OMNIROUTE_ALLOW_DEFAULT_DATA_DIR=1
# Build provenance (#10427). OMNIROUTE_BUILD_SHA lets a container inject the artifact's git
@@ -96,9 +97,11 @@ STORAGE_ENCRYPTION_KEY=
# Default: v1 | Increment when rotating STORAGE_ENCRYPTION_KEY.
STORAGE_ENCRYPTION_KEY_VERSION=v1
# Automatic SQLite backup on startup.
# Used by: src/lib/db/backup.ts — creates a timestamped backup before migrations.
# Default: false (backups enabled) | Set true to skip backup on every restart.
# Routine/pre-write SQLite backups.
# Used by: src/lib/db/backup.ts. Set true only when those backups are managed externally.
# This never disables the migration runner's mandatory, content-addressed safety snapshot
# or its mass-migration guard for an existing persistent database.
# Default: false (routine backups enabled).
DISABLE_SQLITE_AUTO_BACKUP=false
# ── Redis (Rate Limiting) ──

1
changelog.d/fixes/migration-151-152-safety.md Normal file
View File

@@ -0,0 +1 @@
- Harden SQLite upgrades around the historical migration-074 version collision: missing discovery tables are replayed atomically before migrations 151/152, existing databases receive reusable content-addressed safety snapshots, and Node test/eval probes without `DATA_DIR` are isolated from the operator database.

2
docs/guides/DOCKER_GUIDE.md
View File

@@ -613,7 +613,7 @@ In-process density (compression off the HTTP isolate) is [#11023](https://github
## Important Notes
- **SQLite WAL Mode:** `docker stop` should be allowed to finish so OmniRoute can checkpoint the latest changes back into `storage.sqlite`. The bundled Compose files already set a 40s stop grace period. If you run the image directly, keep `--stop-timeout 40`.
- **`DISABLE_SQLITE_AUTO_BACKUP`:** Set to `true` if backups are managed externally.
- **`DISABLE_SQLITE_AUTO_BACKUP`:** Set to `true` if routine/pre-write backups are managed externally. Existing-database migrations still require their own durable safety snapshot and mass-migration guard.
- **Data Persistence:** Always mount a volume to `/app/data` to persist your database, keys, and configurations across container restarts.
- **Port Configuration:** Override `PORT` environment variable to change the default `20128` port.

18
docs/reference/ENVIRONMENT.md
View File

@@ -86,7 +86,7 @@ OmniRoute uses **SQLite** (via `better-sqlite3`) for all persistence. These vari
| Variable | Default | Source File | Description |
| -------------------------------------- | -------------------- | ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `DATA_DIR` | `~/.omniroute/` | `src/lib/db/core.ts` | Root directory for SQLite DB, backups, and data files. Override for Docker volumes or custom paths. |
| `OMNIROUTE_ALLOW_DEFAULT_DATA_DIR` | _(unset)_ | `src/lib/dataPaths.ts` | Escape hatch for the test-context DATA_DIR guard (#10428). Test runs with no `DATA_DIR` are redirected to a throwaway temp dir so they cannot open the operator's real database; set to `1` to opt back in to the real directory. |
| `OMNIROUTE_ALLOW_DEFAULT_DATA_DIR` | _(unset)_ | `src/lib/dataPaths.ts` | Escape hatch for the test/eval DATA_DIR guard (#10428). Tests and Node eval/print probes (`-e`/`--eval`/`-p`/`--print`, including `--eval=`/`--print=` forms) with no `DATA_DIR` are redirected to a throwaway temp dir so they cannot open the operator's real database; set to `1` to opt back in to the real directory. |
| `OMNIROUTE_BUILD_SHA` | _(unset)_ | `src/lib/monitoring/buildSha.ts` | Git SHA of the running artifact. Stamped by `npm run build:release`; injectable in containers that ship without the `dist/BUILD_SHA` sentinel. Surfaced as `system.buildSha` on `/api/monitoring/health`. |
| `OMNIROUTE_RELEASE_REF` | `origin/main` | `scripts/build/buildProvenance.ts` | Ref the pack-artifact provenance gate checks the build SHA against (#10427). |
| `OMNIROUTE_ALLOW_CANARY_BUILD` | _(unset)_ | `scripts/build/buildProvenance.ts` | Set to `1` to allow packing a build whose SHA is not on the release line, recording it as a deliberate canary instead of failing the gate (#10427). |
@@ -97,7 +97,7 @@ OmniRoute uses **SQLite** (via `better-sqlite3`) for all persistence. These vari
| `OMNIROUTE_PLUGINS_DIR` | _(unset)_ | `src/lib/plugins/scanner.ts` | Directory the **runtime plugin scanner** reads — and the root the plugin manager installs into — overriding the home-derived default (#11827). Point it at the bind-mounted plugin tree in Docker/K8s instead of moving HOME just to relocate the scan path (HOME governs every other home-relative behaviour too). Unset = `~/.omniroute/plugins`, or `/tmp/.omniroute/plugins` when the process exports no home at all — the silent non-discovery this variable removes. The resolved directory is logged once at startup as `scanner.dir_resolved` with the input that won. Server-side only: CLI command plugins keep their own `OMNIROUTE_PLUGIN_PATH` (section 9). |
| `STORAGE_ENCRYPTION_KEY` | _(empty = disabled)_ | `src/lib/db/encryption.ts` | AES key for full SQLite database encryption at rest. Generate with `openssl rand -hex 32`. |
| `STORAGE_ENCRYPTION_KEY_VERSION` | `v1` | `scripts/build/bootstrap-env.mjs`, `electron/main.js` | Version label for the encryption key. Increment when performing key rotation to support decryption of old backups. |
| `DISABLE_SQLITE_AUTO_BACKUP` | `false` | `src/lib/db/backup.ts` | When `true`, skips automatic + pre-write SQLite file backups (startup, models.dev pricing save/clear, settings writes). Manual and pre-restore backups still run. Non-manual backups are also **throttled to at most once per 60 minutes** so hourly models.dev sync does not copy the whole DB on every pricing write. Dashboard **Settings → Storage** can disable auto-backup independently. |
| `DISABLE_SQLITE_AUTO_BACKUP` | `false` | `src/lib/db/backup.ts` | When `true`, skips routine/pre-write SQLite file backups (models.dev pricing save/clear, settings writes). Manual and pre-restore backups still run. It does **not** disable the migration runner's mandatory durable safety snapshot or mass-migration guard for an existing persistent DB. Non-manual backups are throttled to at most once per 60 minutes. Dashboard **Settings → Storage** can disable routine auto-backup independently. |
| `OMNIROUTE_CRYPT_KEY` | _(unset)_ | `src/lib/db/encryption.ts` | **Legacy alias** for `STORAGE_ENCRYPTION_KEY`. Accepted as a fallback when the primary variable is absent. |
| `OMNIROUTE_API_KEY_BASE64` | _(unset)_ | `src/lib/db/encryption.ts` | **Legacy alias** (Base64-encoded form) accepted as a fallback. Decoded automatically before use. |
| `OMNIROUTE_DB_HEALTHCHECK_INTERVAL_MS` | _(unset)_ | `src/lib/db/core.ts` | Override the periodic SQLite healthcheck interval (ms). When unset, defaults are derived from `NODE_ENV`. |
@@ -121,6 +121,16 @@ OmniRoute uses **SQLite** (via `better-sqlite3`) for all persistence. These vari
| `BATCH_BACKOFF_MAX_MS` | `3600000` (1h) | `open-sse/services/batchProcessor.ts` | Cap (ms) for exponential backoff between batch item retries. |
| `BATCH_MAX_CONCURRENT` | `1` | `open-sse/services/batchProcessor.ts` | Maximum number of batches processed concurrently. Raise to increase throughput; keep low to avoid rate-limit storms. |
> [!IMPORTANT]
> Before changing an existing persistent database, the migration runner publishes a complete,
> content-addressed snapshot under `DATA_DIR/db_backups/`. Publication requires a filesystem
> that supports same-filesystem, no-overwrite hard links plus durable file sync. POSIX hosts also
> require directory sync; on Windows, Node may reject directory handles, so OmniRoute flushes the
> published file and treats directory-entry sync as best effort.
> If the mounted `DATA_DIR` cannot provide those guarantees, startup fails closed before applying
> a migration. Move `DATA_DIR` to a volume with those primitives; do not use
> `DISABLE_SQLITE_AUTO_BACKUP` to bypass migration safety.
### Scenarios
| Scenario | Configuration |
@@ -1313,8 +1323,8 @@ Provider quota endpoints, network tunnels (Tailscale, Ngrok, MITM debug proxy),
| `TAILSCALED_BIN` | _(auto-detect)_ | `src/lib/tailscaleTunnel.ts` | Explicit path to the `tailscaled` daemon binary. |
| `TAILSCALE_AUTHKEY` | _(unset)_ | `src/lib/tailscaleTunnel.ts` | Pre-shared Tailscale auth key for non-interactive / headless `tailscale up` (passed via `--auth-key=`). When unset, login falls back to the interactive browser auth URL. |
| `NGROK_AUTHTOKEN` | _(unset)_ | `src/lib/ngrokTunnel.ts` | Authenticates outbound ngrok tunnels. |
| `DB_BACKUP_MAX_FILES` | `20` | `src/lib/db/backup.ts`, `src/lib/db/migrationRunner.ts` | Maximum SQLite backup files retained on disk. Applies to manual/scheduled backups and to pre-migration snapshots. Overrides the value saved from Settings → Database backup retention. |
| `DB_BACKUP_RETENTION_DAYS` | `0` | `src/lib/db/backup.ts`, `src/lib/db/migrationRunner.ts` | Maximum age (days) of retained backups. `0` disables age-based pruning. Applies to manual/scheduled backups and to pre-migration snapshots. Overrides the value saved from Settings → Database backup retention. |
| `DB_BACKUP_MAX_FILES` | `20` | `src/lib/db/backup.ts` | Maximum SQLite backup files retained by manual/scheduled backup cleanup. Migration snapshots are content-addressed and reused for an identical DB state; they are not pruned inside the concurrent migration window. Overrides the value saved from Settings → Database backup retention. |
| `DB_BACKUP_RETENTION_DAYS` | `0` | `src/lib/db/backup.ts` | Maximum age (days) retained by manual/scheduled backup cleanup. `0` disables age-based pruning. Migration snapshots are not pruned inside the concurrent migration window. Overrides the value saved from Settings → Database backup retention. |
| `OMNIROUTE_BACKUP_SCHEDULE_JOB_INTERVAL_MS` | `30000` | `src/lib/jobs/backupScheduleJob.ts` | Tick interval (ms) of the server-side job that executes `backup-schedule.json`. Must stay well under the 1-minute cron granularity; values below `5000` or unparseable fall back to `30000`. |
| `CONTAINER_HOST` | `docker` | `scripts/check-permissions.sh` | Container runtime hint for the entrypoint permission check. Set to `podman` for any Podman topology. Because the container cannot determine whether the engine is local or reached through Podman Machine, the warning stays topology-neutral and points to `contrib/podman/README.md`. |
| `QUOTA_STORE_DRIVER` | `sqlite` | `src/lib/quota/storeFactory.ts` | Quota-share consumption store backend: `sqlite` (default) or `redis`. |

49
src/lib/dataPaths.ts
View File

@@ -101,30 +101,70 @@ export function isTestContext(): boolean {
);
}
/**
* `node --eval` / `node -e` (and their print variants) are common shapes used by
* one-off import probes.
* Such a process has no application entry point from which to establish storage intent,
* so defaulting it to the operator's durable database is unsafe. A deliberate production
* inspection can still opt in with an explicit DATA_DIR (preferred) or
* OMNIROUTE_ALLOW_DEFAULT_DATA_DIR=1.
*/
function isEvalProbeContext(): boolean {
return process.execArgv.some(
(arg) =>
arg === "--eval" ||
arg === "-e" ||
arg === "-pe" ||
arg === "-ep" ||
arg.startsWith("--eval=") ||
arg === "--print" ||
arg === "-p" ||
arg.startsWith("--print=")
);
}
/** Process-wide redirect target, so repeated calls share one DB instead of one per call. */
let testContextDataDir: string | null = null;
let testContextCleanupRegistered = false;
export function resolveWritableDataDir({ isCloud = false }: { isCloud?: boolean } = {}): string {
const resolved = resolveDataDir({ isCloud });
const configured = normalizeConfiguredPath(process.env.DATA_DIR);
// Cloud/serverless never owns a writable home dir; leave its sentinel alone.
if (isCloud) return resolved;
// #10428: a test/ad-hoc run that never chose a DATA_DIR would otherwise open the
// #10428: a test/eval-probe run that never chose a DATA_DIR would otherwise open the
// OPERATOR'S REAL database (~/.omniroute/storage.sqlite — live provider credentials).
// Redirect to a throwaway dir instead of throwing: the documented single-file command
// (`node --import tsx/esm --test tests/unit/x.test.ts`) does not load the isolation
// setup, and a hard failure there would only teach people to disable the guard.
// `OMNIROUTE_ALLOW_DEFAULT_DATA_DIR=1` opts back in, so the intent is recorded.
if (
!process.env.DATA_DIR &&
isTestContext() &&
!configured &&
(isTestContext() || isEvalProbeContext()) &&
process.env.OMNIROUTE_ALLOW_DEFAULT_DATA_DIR !== "1"
) {
if (!testContextDataDir) {
testContextDataDir = fs.mkdtempSync(path.join(os.tmpdir(), `${APP_NAME}-testctx-`));
if (!testContextCleanupRegistered) {
testContextCleanupRegistered = true;
process.once("exit", () => {
if (!testContextDataDir) return;
try {
fs.rmSync(testContextDataDir, {
recursive: true,
force: true,
maxRetries: 5,
retryDelay: 25,
});
} catch {
// An unclean exit is left to the operating system's temp-directory policy.
}
});
}
console.warn(
`[DATA_DIR] test context without DATA_DIR → using '${testContextDataDir}' instead of ` +
`[DATA_DIR] test/eval context without DATA_DIR → using '${testContextDataDir}' instead of ` +
`'${resolved}'. Set DATA_DIR explicitly (or load tests/_setup/isolateDataDir.ts) to silence this.`
);
}
@@ -132,7 +172,6 @@ export function resolveWritableDataDir({ isCloud = false }: { isCloud?: boolean
}
// No explicit override → already the default user dir; nothing to fall back to.
const configured = normalizeConfiguredPath(process.env.DATA_DIR);
if (!configured) return resolved;
try {

49
src/lib/db/backup.ts
View File

@@ -101,6 +101,24 @@ function getBackupDir() {
return DB_BACKUPS_DIR || path.join(DATA_DIR, "db_backups");
}
function listBackupFilesNewestFirst(backupDir: string) {
return fs
.readdirSync(backupDir)
.filter((filename) => filename.startsWith("db_") && filename.endsWith(".sqlite"))
.flatMap((filename) => {
try {
return [{ filename, stat: fs.statSync(path.join(backupDir, filename)) }];
} catch {
// A concurrent retention pass may remove an entry after readdir.
return [];
}
})
.sort(
(left, right) =>
right.stat.mtimeMs - left.stat.mtimeMs || right.filename.localeCompare(left.filename)
);
}
export function cleanupDbBackups(options?: {
maxFiles?: number;
retentionDays?: number;
@@ -272,16 +290,26 @@ export function backupDbFile(reason = "auto") {
if (reason !== "manual" && reason !== "pre-restore") {
// Shrink detection is useful for automatic safety backups, but it should
// never block an explicit operator action like manual backup or pre-restore.
// Only timestamp-named automatic/manual backups are shrink baselines. The
// content-addressed migration snapshots are restore points, not periodic size
// samples; excluding them also keeps this lookup to names only with a single stat
// even in legacy directories containing tens of thousands of timestamp backups.
const existingBackups = fs
.readdirSync(backupDir)
.filter((f) => f.startsWith("db_") && f.endsWith(".sqlite"))
.filter((filename) => /^db_\d{4}-.*\.sqlite$/.test(filename))
.sort();
if (existingBackups.length > 0) {
const latestBackup = existingBackups[existingBackups.length - 1];
const latestStat = fs.statSync(path.join(backupDir, latestBackup));
if (latestStat.size > 4096 && stat.size < latestStat.size * 0.5) {
console.warn(`[DB] Backup SKIPPED — DB shrank from ${latestStat.size}B to ${stat.size}B`);
return null;
const latestBackup = existingBackups.at(-1)!;
try {
const latestStat = fs.statSync(path.join(backupDir, latestBackup));
if (latestStat.size > 4096 && stat.size < latestStat.size * 0.5) {
console.warn(
`[DB] Backup SKIPPED — DB shrank from ${latestStat.size}B to ${stat.size}B`
);
return null;
}
} catch (error: unknown) {
if ((error as NodeJS.ErrnoException | null)?.code !== "ENOENT") throw error;
}
}
}
@@ -316,16 +344,11 @@ export async function listDbBackups() {
try {
if (!fs.existsSync(backupDir)) return [];
const entries = fs
.readdirSync(backupDir)
.filter((f) => f.startsWith("db_") && f.endsWith(".sqlite"))
.sort()
.reverse();
const entries = listBackupFilesNewestFirst(backupDir);
const { tryOpenSync } = await import("@/lib/db/adapters/driverFactory");
return entries.map((filename) => {
return entries.map(({ filename, stat }) => {
const filePath = path.join(backupDir, filename);
const stat = fs.statSync(filePath);
const match = filename.match(/^db_(.+?)_([^.]+)\.sqlite$/);
const reason = match ? match[2] : "unknown";

17
src/lib/db/backupRetention.ts
View File

@@ -1,17 +1,12 @@
/**
* Backup retention primitives — pure filesystem work, no `core.ts` dependency.
*
* This module exists so BOTH backup call sites can share one retention policy:
*
* - `backup.ts` (manual/API/auto backups) — resolves the operator's settings from the
* database and delegates here.
* - `migrationRunner.ts` (pre-migration snapshots) — cannot import `backup.ts`, because
* `core.ts` already imports `migrationRunner.ts` and `backup.ts` imports `core.ts`;
* that edge would close a cycle. Keeping the policy here, free of `core`, lets the
* migration path prune without one.
*
* Before #10421 the migration path had no retention at all and `db_backups/` grew
* without bound (observed: 48.999 files / 204 GB against a 5,3 MB live database).
* `backup.ts` (manual/API/auto backups) resolves the operator's settings from the
* database and delegates pure family pruning here. The migration runner deliberately
* does not prune during its concurrent safety window: its snapshots are content-addressed
* and reused for an identical DB state, while manual/scheduled cleanup remains the single
* retention boundary. Before #10421, repeated failed starts created distinct timestamped
* snapshots and `db_backups/` grew without bound (observed: 48,999 files / 204 GB).
*/
import fs from "fs";

889
src/lib/db/migrationRunner.ts
View File

File diff suppressed because it is too large Load Diff

8
src/lib/db/migrationRunner/constants.ts
View File

@@ -158,6 +158,14 @@ export const RENAMED_MIGRATION_COMPATIBILITY = [
toVersion: "151",
toName: "windsurf_to_devin_desktop",
},
{
// inspector_custom_hosts was once published in slot 074, now occupied by
// discovery_results. Its canonical idempotent migration lives at 081.
fromVersion: "074",
fromName: "inspector_custom_hosts",
toVersion: "081",
toName: "inspector_custom_hosts",
},
{
fromVersion: "134",
fromName: "ccr_blocks",

96
tests/unit/datadir-test-context-guard-10428.test.ts
View File

@@ -1,5 +1,6 @@
import test from "node:test";
import assert from "node:assert/strict";
import { spawnSync } from "node:child_process";
import os from "node:os";
import path from "node:path";
import fs from "node:fs";
@@ -21,6 +22,30 @@ import fs from "node:fs";
*/
const { resolveWritableDataDir, getDefaultDataDir } = await import("../../src/lib/dataPaths.ts");
const redirectedDirs = new Set<string>();
function assertOwnedRedirectDir(candidate: string): string {
const resolved = path.resolve(candidate);
const tempRoot = path.resolve(os.tmpdir());
assert.ok(
resolved.startsWith(`${tempRoot}${path.sep}`) &&
path.basename(resolved).startsWith("omniroute-testctx-"),
`refusing to treat a non-owned path as a test redirect: ${resolved}`
);
return resolved;
}
function rememberRedirectDir(candidate: string): string {
const resolved = assertOwnedRedirectDir(candidate);
redirectedDirs.add(resolved);
return resolved;
}
test.after(() => {
for (const redirected of redirectedDirs) {
fs.rmSync(redirected, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });
}
});
function withEnv(overrides: Record<string, string | undefined>, run: () => void) {
const saved: Record<string, string | undefined> = {};
@@ -39,11 +64,58 @@ function withEnv(overrides: Record<string, string | undefined>, run: () => void)
}
}
const EVAL_PROBE_SCRIPT =
"import('./src/lib/dataPaths.ts').then(({ resolveWritableDataDir }) => " +
"console.log('OMNIROUTE_TEST_DATA_DIR=' + resolveWritableDataDir()))";
function assertEvalProbeIsIsolated(evalArgs: string[], configuredDataDir = "") {
const result = spawnSync(process.execPath, ["--import", "tsx/esm", ...evalArgs], {
cwd: process.cwd(),
encoding: "utf8",
env: {
...process.env,
DATA_DIR: configuredDataDir,
XDG_CONFIG_HOME: "",
NODE_ENV: "production",
NODE_TEST_CONTEXT: "",
VITEST: "",
OMNIROUTE_ALLOW_DEFAULT_DATA_DIR: "",
},
});
assert.equal(result.status, 0, result.stderr);
const outputLine = result.stdout
.trim()
.split("\n")
.find((line) => line.startsWith("OMNIROUTE_TEST_DATA_DIR="));
const resolved = outputLine?.slice("OMNIROUTE_TEST_DATA_DIR=".length) ?? "";
const ownedRedirect = assertOwnedRedirectDir(resolved);
try {
assert.notEqual(
ownedRedirect,
path.join(os.homedir(), ".omniroute"),
"an eval/import probe must not inherit the normal server's default database"
);
assert.equal(
fs.existsSync(ownedRedirect),
false,
"the child exit handler must remove its exact redirected DATA_DIR"
);
} finally {
fs.rmSync(ownedRedirect, {
recursive: true,
force: true,
maxRetries: 5,
retryDelay: 100,
});
}
}
test("G1: a test context with no DATA_DIR never resolves to the operator's real data dir", () => {
withEnv(
{ DATA_DIR: undefined, NODE_ENV: "test", OMNIROUTE_ALLOW_DEFAULT_DATA_DIR: undefined },
() => {
const resolved = resolveWritableDataDir();
const resolved = rememberRedirectDir(resolveWritableDataDir());
assert.notEqual(
resolved,
getDefaultDataDir(),
@@ -101,7 +173,7 @@ test("G5: node:test subprocesses are detected through NODE_TEST_CONTEXT too", ()
OMNIROUTE_ALLOW_DEFAULT_DATA_DIR: undefined,
},
() => {
const resolved = resolveWritableDataDir();
const resolved = rememberRedirectDir(resolveWritableDataDir());
assert.notEqual(resolved, getDefaultDataDir());
assert.ok(resolved.startsWith(os.tmpdir()));
}
@@ -110,8 +182,24 @@ test("G5: node:test subprocesses are detected through NODE_TEST_CONTEXT too", ()
test("G6: the redirect is stable within a process (same dir on repeated calls)", () => {
withEnv({ DATA_DIR: undefined, NODE_ENV: "test" }, () => {
const first = resolveWritableDataDir();
const second = resolveWritableDataDir();
const first = rememberRedirectDir(resolveWritableDataDir());
const second = rememberRedirectDir(resolveWritableDataDir());
assert.equal(first, second, "a per-call temp dir would split the DB across handles");
});
});
test("G7: a node --eval probe without DATA_DIR is isolated from the operator home", () => {
assertEvalProbeIsIsolated(["--eval", EVAL_PROBE_SCRIPT]);
});
test("G8: the single-argument --eval= form is isolated too", () => {
assertEvalProbeIsIsolated([`--eval=${EVAL_PROBE_SCRIPT}`]);
});
test("G9: whitespace DATA_DIR is absent for a node -e probe", () => {
assertEvalProbeIsIsolated(["-e", EVAL_PROBE_SCRIPT], " ");
});
test("G10: a combined node -pe probe is isolated too", () => {
assertEvalProbeIsIsolated(["-pe", EVAL_PROBE_SCRIPT]);
});

26
tests/unit/db-backup-extended.test.ts
View File

@@ -97,6 +97,32 @@ test("backupDbFile creates manual backups and listDbBackups returns metadata", a
assert.equal(fs.existsSync(backupPath), true);
});
test("listDbBackups orders mixed timestamp and content-addressed names by mtime", async () => {
seedConnections(2);
fs.mkdirSync(core.DB_BACKUPS_DIR, { recursive: true });
const lexicallyFutureButOld = "db_2099-01-01T00-00-00-000Z_manual.sqlite";
const timestampMiddle = "db_2026-09-02T00-00-00-000Z_manual.sqlite";
const contentAddressedNewest = `db_state-${"a".repeat(64)}_pre-migration.sqlite`;
for (const filename of [lexicallyFutureButOld, timestampMiddle, contentAddressedNewest]) {
await core.getDbInstance().backup(path.join(core.DB_BACKUPS_DIR, filename));
}
const now = Date.now() / 1000;
fs.utimesSync(path.join(core.DB_BACKUPS_DIR, lexicallyFutureButOld), now - 120, now - 120);
fs.utimesSync(path.join(core.DB_BACKUPS_DIR, timestampMiddle), now - 60, now - 60);
fs.utimesSync(path.join(core.DB_BACKUPS_DIR, contentAddressedNewest), now, now);
const backups = await backupDb.listDbBackups();
assert.deepEqual(
backups.map((backup) => backup.id),
[contentAddressedNewest, timestampMiddle, lexicallyFutureButOld],
"content-addressed migration snapshots must not make filename order masquerade as recency"
);
assert.equal(backups[0]?.reason, "pre-migration");
assert.equal(backups[0]?.connectionCount, 2);
});
test("listDbBackups returns an empty list when the backup directory is missing", async () => {
fs.rmSync(core.DB_BACKUPS_DIR, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });
const backups = await backupDb.listDbBackups();

754
tests/unit/db-migration-missing-physical-schema.test.ts Normal file
View File

@@ -0,0 +1,754 @@
// ENVIRONMENT NOTE (sandbox better-sqlite3 / glibc limitation, not a code defect):
// This test constructs a real better-sqlite3 database. Production and CI load the
// native addon normally; see tests/unit/_helpers/betterSqlite3Availability.ts for
// the documented fallback context on older sandboxes.
import assert from "node:assert/strict";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { spawnSync } from "node:child_process";
import test from "node:test";
import { fileURLToPath } from "node:url";
import Database from "better-sqlite3";
const isIsolatedChild = process.env.OMNIROUTE_DB_MIGRATION_SAFETY_CHILD === "1";
if (!isIsolatedChild) {
test("historical migration repair scenarios pass in an isolated process", () => {
const dataDir = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-schema-repair-data-"));
const migrationsDir = fs.mkdtempSync(
path.join(os.tmpdir(), "omniroute-schema-repair-migrations-")
);
try {
const childEnv = {
...process.env,
DATA_DIR: dataDir,
OMNIROUTE_DB_MIGRATION_SAFETY_CHILD: "1",
OMNIROUTE_MAX_PENDING_MIGRATIONS: "",
OMNIROUTE_MIGRATIONS_DIR: migrationsDir,
};
// Node's test runner exports this only to the current test worker. Passing it into
// another `node --test` process makes Node classify the nested file as recursive and
// skip every subtest while returning exit 0 — a dangerous false green.
delete childEnv.NODE_TEST_CONTEXT;
const result = spawnSync(
process.execPath,
["--import", "tsx/esm", "--test", fileURLToPath(import.meta.url)],
{
cwd: process.cwd(),
encoding: "utf8",
env: childEnv,
}
);
assert.equal(
result.status,
0,
`isolated migration regressions failed\nstdout:\n${result.stdout}\nstderr:\n${result.stderr}`
);
assert.match(result.stdout, /\btests 11\b/, "the isolated child must execute all subtests");
assert.match(result.stdout, /\bpass 11\b/, "the isolated child must pass all subtests");
} finally {
fs.rmSync(dataDir, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });
fs.rmSync(migrationsDir, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });
}
});
} else {
const dataDir = process.env.DATA_DIR;
const migrationsDir = process.env.OMNIROUTE_MIGRATIONS_DIR;
assert.ok(dataDir, "isolated child requires an explicit DATA_DIR");
assert.ok(migrationsDir, "isolated child requires an explicit migrations directory");
const discoveryMigrationSql = fs.readFileSync(
path.resolve("src/lib/db/migrations/074_discovery_results.sql"),
"utf8"
);
fs.writeFileSync(
path.join(migrationsDir, "074_discovery_results.sql"),
discoveryMigrationSql,
"utf8"
);
fs.writeFileSync(
path.join(migrationsDir, "081_inspector_custom_hosts.sql"),
`
CREATE TABLE IF NOT EXISTS inspector_custom_hosts (
host TEXT PRIMARY KEY,
enabled INTEGER NOT NULL DEFAULT 1
);
CREATE INDEX IF NOT EXISTS idx_inspector_custom_hosts_enabled
ON inspector_custom_hosts(enabled);
`,
"utf8"
);
fs.writeFileSync(
path.join(migrationsDir, "151_windsurf_to_devin_desktop.sql"),
"UPDATE discovery_results SET provider_id = 'devin-desktop' WHERE provider_id = 'windsurf';",
"utf8"
);
fs.writeFileSync(
path.join(migrationsDir, "152_remove_puter_provider.sql"),
"DELETE FROM discovery_results WHERE provider_id = 'puter';",
"utf8"
);
const { runMigrations } = await import("../../src/lib/db/migrationRunner.ts");
function listPreMigrationBackups(): string[] {
const backupDir = path.join(dataDir, "db_backups");
if (!fs.existsSync(backupDir)) return [];
return fs
.readdirSync(backupDir)
.filter((name) => name.endsWith("_pre-migration.sqlite"))
.sort();
}
function withNonTestEnvironment<T>(fn: () => T): T {
const previousNodeEnv = process.env.NODE_ENV;
const previousVitest = process.env.VITEST;
const previousArgv = [...process.argv];
const previousExecArgv = [...process.execArgv];
delete process.env.NODE_ENV;
delete process.env.VITEST;
process.argv = process.argv.filter((arg) => !arg.includes("test"));
process.execArgv = process.execArgv.filter((arg) => !arg.includes("test"));
try {
return fn();
} finally {
process.argv = previousArgv;
process.execArgv = previousExecArgv;
if (previousNodeEnv === undefined) delete process.env.NODE_ENV;
else process.env.NODE_ENV = previousNodeEnv;
if (previousVitest === undefined) delete process.env.VITEST;
else process.env.VITEST = previousVitest;
}
}
test.after(() => {
// The parent owns both explicit temp directories and removes them after this
// process exits. Keeping ownership there also covers child startup failures.
});
test("runner repairs the 074 inspector collision before migrations 151 and 152", () => {
const db = new Database(":memory:");
try {
db.exec(`
CREATE TABLE _omniroute_migrations (
version TEXT PRIMARY KEY,
name TEXT NOT NULL,
applied_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE TABLE inspector_custom_hosts (
host TEXT PRIMARY KEY,
enabled INTEGER NOT NULL DEFAULT 1
);
INSERT INTO inspector_custom_hosts (host, enabled)
VALUES ('api.example.test', 1);
INSERT INTO _omniroute_migrations (version, name)
VALUES ('074', 'inspector_custom_hosts');
`);
assert.equal(
db
.prepare(
"SELECT name FROM sqlite_master WHERE type = 'table' AND name = 'discovery_results'"
)
.get(),
undefined,
"precondition: the collided 074 marker hides the missing discovery_results table"
);
assert.equal(runMigrations(db as never), 3);
assert.ok(
db
.prepare(
"SELECT name FROM sqlite_master WHERE type = 'table' AND name = 'discovery_results'"
)
.get(),
"074 must be replayed before migrations 151 and 152 reference discovery_results"
);
assert.deepEqual(
db.prepare("SELECT version, name FROM _omniroute_migrations ORDER BY version").all(),
[
{ version: "074", name: "discovery_results" },
{ version: "081", name: "inspector_custom_hosts" },
{ version: "151", name: "windsurf_to_devin_desktop" },
{ version: "152", name: "remove_puter_provider" },
]
);
assert.deepEqual(
db.prepare("SELECT host, enabled FROM inspector_custom_hosts").get(),
{ host: "api.example.test", enabled: 1 },
"re-homing the inspector marker to 081 must preserve the existing table data"
);
assert.equal(runMigrations(db as never), 0, "the repaired state must be idempotent");
} finally {
db.close();
}
});
test("runner rehomes a collided 074 inspector marker even when both tables exist", () => {
const db = new Database(":memory:");
try {
db.exec(discoveryMigrationSql);
db.exec(`
CREATE TABLE _omniroute_migrations (
version TEXT PRIMARY KEY,
name TEXT NOT NULL,
applied_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE TABLE inspector_custom_hosts (
host TEXT PRIMARY KEY,
enabled INTEGER NOT NULL DEFAULT 1
);
INSERT INTO inspector_custom_hosts (host, enabled)
VALUES ('api.example.test', 1);
INSERT INTO _omniroute_migrations (version, name)
VALUES ('074', 'inspector_custom_hosts');
`);
assert.equal(runMigrations(db as never), 3);
assert.deepEqual(
db.prepare("SELECT version, name FROM _omniroute_migrations ORDER BY version").all(),
[
{ version: "074", name: "discovery_results" },
{ version: "081", name: "inspector_custom_hosts" },
{ version: "151", name: "windsurf_to_devin_desktop" },
{ version: "152", name: "remove_puter_provider" },
],
"the old 074 name must not remain as a permanent CRITICAL mismatch"
);
assert.deepEqual(db.prepare("SELECT host FROM inspector_custom_hosts").get(), {
host: "api.example.test",
});
} finally {
db.close();
}
});
test("runner fails closed when target 081 has unknown provenance", () => {
const db = new Database(":memory:");
try {
db.exec(`
CREATE TABLE inspector_custom_hosts (
host TEXT PRIMARY KEY,
enabled INTEGER NOT NULL DEFAULT 1
);
CREATE TABLE _omniroute_migrations (
version TEXT PRIMARY KEY,
name TEXT NOT NULL,
applied_at TEXT NOT NULL DEFAULT (datetime('now'))
);
INSERT INTO _omniroute_migrations (version, name)
VALUES ('074', 'inspector_custom_hosts');
INSERT INTO _omniroute_migrations (version, name)
VALUES ('081', 'unknown_historical_migration');
`);
assert.throws(
() => runMigrations(db as never),
/target version 081 is occupied by unknown migration/i
);
assert.deepEqual(
db.prepare("SELECT version, name FROM _omniroute_migrations ORDER BY version").all(),
[
{ version: "074", name: "inspector_custom_hosts" },
{ version: "081", name: "unknown_historical_migration" },
],
"a target collision must preserve both provenance records"
);
} finally {
db.close();
}
});
test("runner rejects an unknown 074 marker even when all later migrations are marked", () => {
const db = new Database(":memory:");
try {
db.exec(`
CREATE TABLE _omniroute_migrations (
version TEXT PRIMARY KEY,
name TEXT NOT NULL,
applied_at TEXT NOT NULL DEFAULT (datetime('now'))
);
INSERT INTO _omniroute_migrations (version, name)
VALUES ('074', 'unknown_historical_migration');
INSERT INTO _omniroute_migrations (version, name)
VALUES ('081', 'inspector_custom_hosts');
INSERT INTO _omniroute_migrations (version, name)
VALUES ('151', 'windsurf_to_devin_desktop');
INSERT INTO _omniroute_migrations (version, name)
VALUES ('152', 'remove_puter_provider');
`);
assert.throws(
() => runMigrations(db as never),
/required table "discovery_results" is missing.*unknown migration/i,
"unknown provenance must fail closed instead of being silently rewritten"
);
assert.deepEqual(
db.prepare("SELECT version, name FROM _omniroute_migrations ORDER BY version").all(),
[
{ version: "074", name: "unknown_historical_migration" },
{ version: "081", name: "inspector_custom_hosts" },
{ version: "151", name: "windsurf_to_devin_desktop" },
{ version: "152", name: "remove_puter_provider" },
]
);
} finally {
db.close();
}
});
test("runner backs up an existing DB before reopening its only applied marker", () => {
const sqlitePath = path.join(dataDir, "only-marker.sqlite");
const db = new Database(sqlitePath);
const previousDisableBackup = process.env.DISABLE_SQLITE_AUTO_BACKUP;
try {
delete process.env.DISABLE_SQLITE_AUTO_BACKUP;
db.exec(`
CREATE TABLE provider_connections (id TEXT PRIMARY KEY);
CREATE TABLE _omniroute_migrations (
version TEXT PRIMARY KEY,
name TEXT NOT NULL,
applied_at TEXT NOT NULL DEFAULT (datetime('now'))
);
INSERT INTO provider_connections (id) VALUES ('existing-data');
INSERT INTO _omniroute_migrations (version, name)
VALUES ('074', 'discovery_results');
`);
assert.equal(runMigrations(db as never), 4);
const backupDir = path.join(dataDir, "db_backups");
const backups = fs
.readdirSync(backupDir)
.filter((name) => name.endsWith("_pre-migration.sqlite"));
assert.equal(
backups.length,
1,
"removing the only marker must not make an existing DB look fresh and skip its snapshot"
);
} finally {
db.close();
if (previousDisableBackup === undefined) delete process.env.DISABLE_SQLITE_AUTO_BACKUP;
else process.env.DISABLE_SQLITE_AUTO_BACKUP = previousDisableBackup;
}
});
test("snapshot publication never deletes a raced final path", () => {
const sqlitePath = path.join(dataDir, "snapshot-publish-race.sqlite");
const db = new Database(sqlitePath);
const previousDisableBackup = process.env.DISABLE_SQLITE_AUTO_BACKUP;
const originalLinkSync = fs.linkSync;
let racedFinalPath: string | null = null;
try {
delete process.env.DISABLE_SQLITE_AUTO_BACKUP;
db.exec(`
CREATE TABLE _omniroute_migrations (
version TEXT PRIMARY KEY,
name TEXT NOT NULL,
applied_at TEXT NOT NULL DEFAULT (datetime('now'))
);
INSERT INTO _omniroute_migrations (version, name)
VALUES ('074', 'discovery_results');
`);
fs.linkSync = ((_existingPath: fs.PathLike, newPath: fs.PathLike) => {
racedFinalPath = String(newPath);
fs.writeFileSync(racedFinalPath, "third-party-sentinel");
throw Object.assign(new Error("destination already exists"), { code: "EEXIST" });
}) as typeof fs.linkSync;
assert.throws(
() => runMigrations(db as never),
/without a durable snapshot/,
"a raced final name must fail closed before atomic replay"
);
assert.ok(racedFinalPath);
assert.equal(
fs.readFileSync(racedFinalPath, "utf8"),
"third-party-sentinel",
"snapshot failure cleanup must never unlink another actor's final path"
);
assert.deepEqual(db.prepare("SELECT version, name FROM _omniroute_migrations").all(), [
{ version: "074", name: "discovery_results" },
]);
} finally {
fs.linkSync = originalLinkSync;
if (racedFinalPath && fs.existsSync(racedFinalPath)) fs.unlinkSync(racedFinalPath);
db.close();
if (previousDisableBackup === undefined) delete process.env.DISABLE_SQLITE_AUTO_BACKUP;
else process.env.DISABLE_SQLITE_AUTO_BACKUP = previousDisableBackup;
}
});
test("snapshot publication fails closed when hard links are unsupported", () => {
const sqlitePath = path.join(dataDir, "snapshot-publish-fallback.sqlite");
const db = new Database(sqlitePath);
const previousDisableBackup = process.env.DISABLE_SQLITE_AUTO_BACKUP;
const originalLinkSync = fs.linkSync;
const backupsBefore = listPreMigrationBackups();
try {
delete process.env.DISABLE_SQLITE_AUTO_BACKUP;
db.exec(`
CREATE TABLE _omniroute_migrations (
version TEXT PRIMARY KEY,
name TEXT NOT NULL,
applied_at TEXT NOT NULL DEFAULT (datetime('now'))
);
INSERT INTO _omniroute_migrations (version, name)
VALUES ('074', 'discovery_results');
`);
fs.linkSync = (() => {
throw Object.assign(new Error("hard links unsupported"), { code: "ENOTSUP" });
}) as typeof fs.linkSync;
assert.throws(
() => runMigrations(db as never),
/durable snapshot.*hard links unsupported.*hard links.*synchronization/is
);
assert.deepEqual(db.prepare("SELECT version, name FROM _omniroute_migrations").all(), [
{ version: "074", name: "discovery_results" },
]);
assert.equal(
db
.prepare(
"SELECT name FROM sqlite_master WHERE type = 'table' AND name = 'discovery_results'"
)
.get(),
undefined
);
assert.deepEqual(listPreMigrationBackups(), backupsBefore);
} finally {
fs.linkSync = originalLinkSync;
db.close();
if (previousDisableBackup === undefined) delete process.env.DISABLE_SQLITE_AUTO_BACKUP;
else process.env.DISABLE_SQLITE_AUTO_BACKUP = previousDisableBackup;
}
});
test("an only-marker repair cannot disarm the mass-migration barrier on retry", () => {
const sqlitePath = path.join(dataDir, "only-marker-mass-safety.sqlite");
const db = new Database(sqlitePath);
const previousDisableBackup = process.env.DISABLE_SQLITE_AUTO_BACKUP;
const previousMaxPending = process.env.OMNIROUTE_MAX_PENDING_MIGRATIONS;
try {
process.env.DISABLE_SQLITE_AUTO_BACKUP = "true";
process.env.OMNIROUTE_MAX_PENDING_MIGRATIONS = "1";
db.exec(`
CREATE TABLE provider_connections (id TEXT PRIMARY KEY);
CREATE TABLE inspector_custom_hosts (
host TEXT PRIMARY KEY,
enabled INTEGER NOT NULL DEFAULT 1
);
CREATE TABLE _omniroute_migrations (
version TEXT PRIMARY KEY,
name TEXT NOT NULL,
applied_at TEXT NOT NULL DEFAULT (datetime('now'))
);
INSERT INTO provider_connections (id) VALUES ('existing-data');
INSERT INTO _omniroute_migrations (version, name)
VALUES ('074', 'inspector_custom_hosts');
`);
const runOnce = () => withNonTestEnvironment(() => runMigrations(db as never));
const backupsBefore = listPreMigrationBackups();
assert.throws(runOnce, /threshold is 1/i);
assert.deepEqual(
db.prepare("SELECT version, name FROM _omniroute_migrations").all(),
[{ version: "074", name: "inspector_custom_hosts" }],
"an abort must restore the marker that was rehomed to calculate the real pending set"
);
const afterFirstAbort = listPreMigrationBackups();
const created = afterFirstAbort.filter((name) => !backupsBefore.includes(name));
assert.equal(created.length, 1, "the first abort must retain one restore point");
assert.match(created[0]!, /^db_state-[a-f0-9]{64}_pre-migration\.sqlite$/);
assert.throws(runOnce, /threshold is 1/i);
assert.deepEqual(
db.prepare("SELECT version, name FROM _omniroute_migrations").all(),
[{ version: "074", name: "inspector_custom_hosts" }],
"the second startup must hit the same barrier instead of treating the DB as fresh"
);
assert.deepEqual(
listPreMigrationBackups(),
afterFirstAbort,
"the identical retry must reuse the first content-addressed snapshot"
);
} finally {
db.close();
if (previousDisableBackup === undefined) delete process.env.DISABLE_SQLITE_AUTO_BACKUP;
else process.env.DISABLE_SQLITE_AUTO_BACKUP = previousDisableBackup;
if (previousMaxPending === undefined) delete process.env.OMNIROUTE_MAX_PENDING_MIGRATIONS;
else process.env.OMNIROUTE_MAX_PENDING_MIGRATIONS = previousMaxPending;
}
});
test("a failed atomic 074 replay restores its marker and does not churn snapshots", () => {
const sqlitePath = path.join(dataDir, "failed-atomic-replay.sqlite");
const db = new Database(sqlitePath);
const previousDisableBackup = process.env.DISABLE_SQLITE_AUTO_BACKUP;
try {
delete process.env.DISABLE_SQLITE_AUTO_BACKUP;
db.exec(`
CREATE TABLE _omniroute_migrations (
version TEXT PRIMARY KEY,
name TEXT NOT NULL,
applied_at TEXT NOT NULL DEFAULT (datetime('now'))
);
INSERT INTO _omniroute_migrations (version, name)
VALUES ('074', 'discovery_results');
CREATE TRIGGER block_migration_ledger_replay
BEFORE INSERT ON _omniroute_migrations
WHEN NEW.version = '074'
BEGIN
SELECT RAISE(ABORT, 'ledger replay blocked');
END;
`);
const runOnce = () => runMigrations(db as never);
const backupsBefore = listPreMigrationBackups();
assert.throws(runOnce, /ledger replay blocked/);
assert.deepEqual(db.prepare("SELECT version, name FROM _omniroute_migrations").all(), [
{ version: "074", name: "discovery_results" },
]);
assert.equal(
db
.prepare(
"SELECT name FROM sqlite_master WHERE type = 'table' AND name = 'discovery_results'"
)
.get(),
undefined,
"the table creation and marker replacement must roll back together"
);
const afterFirstFailure = listPreMigrationBackups();
const created = afterFirstFailure.filter((name) => !backupsBefore.includes(name));
assert.equal(created.length, 1, "the first failed replay must retain one restore point");
assert.match(created[0]!, /^db_state-[a-f0-9]{64}_pre-migration\.sqlite$/);
assert.throws(runOnce, /ledger replay blocked/);
assert.deepEqual(
listPreMigrationBackups(),
afterFirstFailure,
"the identical failed replay must reuse its content-addressed restore point"
);
} finally {
db.close();
if (previousDisableBackup === undefined) delete process.env.DISABLE_SQLITE_AUTO_BACKUP;
else process.env.DISABLE_SQLITE_AUTO_BACKUP = previousDisableBackup;
}
});
test("sql.js rolls ledger repairs back when the mass-migration barrier aborts", async () => {
const sqlitePath = path.join(dataDir, "sqljs-mass-safety.sqlite");
const { createSqlJsAdapter } = await import("../../src/lib/db/adapters/sqljsAdapter.ts");
const db = await createSqlJsAdapter(sqlitePath);
const previousMaxPending = process.env.OMNIROUTE_MAX_PENDING_MIGRATIONS;
const backupsBefore = listPreMigrationBackups();
try {
process.env.OMNIROUTE_MAX_PENDING_MIGRATIONS = "1";
db.exec(`
CREATE TABLE provider_connections (id TEXT PRIMARY KEY);
CREATE TABLE inspector_custom_hosts (
host TEXT PRIMARY KEY,
enabled INTEGER NOT NULL DEFAULT 1
);
CREATE TABLE _omniroute_migrations (
version TEXT PRIMARY KEY,
name TEXT NOT NULL,
applied_at TEXT NOT NULL DEFAULT (datetime('now'))
);
INSERT INTO provider_connections (id) VALUES ('existing-data');
INSERT INTO inspector_custom_hosts (host) VALUES ('api.example.test');
INSERT INTO _omniroute_migrations (version, name)
VALUES ('074', 'inspector_custom_hosts');
`);
const runOnce = () => withNonTestEnvironment(() => runMigrations(db));
const expectedLedger = [{ version: "074", name: "inspector_custom_hosts" }];
assert.throws(runOnce, /threshold is 1/i);
assert.deepEqual(
db.prepare("SELECT version, name FROM _omniroute_migrations ORDER BY version").all(),
expectedLedger,
"sql.js must roll the compatibility repair back with the safety savepoint"
);
const afterFirstAbort = listPreMigrationBackups();
const created = afterFirstAbort.filter((name) => !backupsBefore.includes(name));
assert.equal(created.length, 1, "the first sql.js abort must retain one host snapshot");
assert.match(created[0]!, /^db_state-[a-f0-9]{64}_pre-migration\.sqlite$/);
assert.throws(runOnce, /threshold is 1/i);
assert.deepEqual(
db.prepare("SELECT version, name FROM _omniroute_migrations ORDER BY version").all(),
expectedLedger,
"a retry must see the same original ledger rather than committed repair residue"
);
assert.deepEqual(
listPreMigrationBackups(),
afterFirstAbort,
"the identical sql.js abort must reuse its content-addressed snapshot"
);
} finally {
db.close();
if (previousMaxPending === undefined) delete process.env.OMNIROUTE_MAX_PENDING_MIGRATIONS;
else process.env.OMNIROUTE_MAX_PENDING_MIGRATIONS = previousMaxPending;
}
});
test("sql.js exports a real host snapshot before replaying 074", async () => {
const sqlitePath = path.join(dataDir, "sqljs-physical-replay.sqlite");
const { createSqlJsAdapter } = await import("../../src/lib/db/adapters/sqljsAdapter.ts");
const db = await createSqlJsAdapter(sqlitePath);
const previousDisableBackup = process.env.DISABLE_SQLITE_AUTO_BACKUP;
const backupsBefore = listPreMigrationBackups();
try {
delete process.env.DISABLE_SQLITE_AUTO_BACKUP;
db.exec(`
PRAGMA user_version = 42;
PRAGMA application_id = 1337;
CREATE TABLE _omniroute_migrations (
version TEXT PRIMARY KEY,
name TEXT NOT NULL,
applied_at TEXT NOT NULL DEFAULT (datetime('now'))
);
INSERT INTO _omniroute_migrations (version, name)
VALUES ('074', 'discovery_results');
CREATE TRIGGER block_sqljs_ledger_replay
BEFORE INSERT ON _omniroute_migrations
WHEN NEW.version = '074'
BEGIN
SELECT RAISE(ABORT, 'sqljs ledger replay blocked');
END;
`);
assert.throws(() => runMigrations(db), /sqljs ledger replay blocked/);
assert.deepEqual(db.prepare("SELECT version, name FROM _omniroute_migrations").all(), [
{ version: "074", name: "discovery_results" },
]);
assert.equal(
db
.prepare(
"SELECT name FROM sqlite_master WHERE type = 'table' AND name = 'discovery_results'"
)
.get(),
undefined,
"sql.js must roll back the table and marker replacement together"
);
const afterFirstFailure = listPreMigrationBackups();
const firstCreated = afterFirstFailure.filter((name) => !backupsBefore.includes(name));
assert.equal(firstCreated.length, 1, "sql.js must retain one host restore point");
assert.throws(() => runMigrations(db), /sqljs ledger replay blocked/);
assert.deepEqual(
listPreMigrationBackups(),
afterFirstFailure,
"the identical sql.js failure must reuse its content-addressed snapshot"
);
db.exec("DROP TRIGGER block_sqljs_ledger_replay");
assert.equal(runMigrations(db), 4);
const created = listPreMigrationBackups().filter((name) => !backupsBefore.includes(name));
assert.equal(
created.length,
2,
`dropping the trigger changes the DB state and must create a second snapshot: ${created}`
);
const snapshot = new Database(path.join(dataDir, "db_backups", created[0]!), {
readonly: true,
});
try {
assert.equal(snapshot.pragma("integrity_check", { simple: true }), "ok");
assert.equal(snapshot.pragma("user_version", { simple: true }), 42);
assert.equal(snapshot.pragma("application_id", { simple: true }), 1337);
const snapshotBytes = fs.readFileSync(path.join(dataDir, "db_backups", created[0]!));
assert.equal(snapshotBytes.readUInt32BE(24), 1);
assert.equal(
snapshotBytes.readUInt32BE(92),
snapshotBytes.readUInt32BE(24),
"the normalized SQLite change counter and version-valid-for fields must agree"
);
assert.deepEqual(
snapshot.prepare("SELECT version, name FROM _omniroute_migrations").all(),
[{ version: "074", name: "discovery_results" }]
);
assert.equal(
snapshot
.prepare(
"SELECT name FROM sqlite_master WHERE type = 'table' AND name = 'discovery_results'"
)
.get(),
undefined,
"the snapshot must contain the complete pre-replay image"
);
} finally {
snapshot.close();
}
const { listDbBackups } = await import("../../src/lib/db/backup.ts");
const listed = await listDbBackups();
assert.equal(
listed.find((backup) => backup.id === created[0])?.reason,
"pre-migration",
"the content address must not change the public backup reason"
);
db.close();
const reopened = await createSqlJsAdapter(sqlitePath);
try {
assert.deepEqual(
reopened
.prepare("SELECT version, name FROM _omniroute_migrations ORDER BY version")
.all(),
[
{ version: "074", name: "discovery_results" },
{ version: "081", name: "inspector_custom_hosts" },
{ version: "151", name: "windsurf_to_devin_desktop" },
{ version: "152", name: "remove_puter_provider" },
]
);
assert.ok(
reopened
.prepare(
"SELECT name FROM sqlite_master WHERE type = 'index' AND name = 'idx_discovery_results_provider'"
)
.get()
);
assert.ok(
reopened
.prepare(
"SELECT name FROM sqlite_master WHERE type = 'index' AND name = 'idx_discovery_results_status'"
)
.get()
);
} finally {
reopened.close();
}
} finally {
if (db.open) db.close();
if (previousDisableBackup === undefined) delete process.env.DISABLE_SQLITE_AUTO_BACKUP;
else process.env.DISABLE_SQLITE_AUTO_BACKUP = previousDisableBackup;
}
});
}

15
tests/unit/db-migrationrunner-constants-split.test.ts
View File

@@ -70,8 +70,8 @@ describe("migrationRunner/constants — exact small-table snapshots", () => {
// ── large tables — count + shape + spot-checks (corruption guard) ─────────────
describe("migrationRunner/constants — large-table integrity", () => {
it("RENAMED_MIGRATION_COMPATIBILITY has 31 well-formed entries", () => {
assert.equal(RENAMED_MIGRATION_COMPATIBILITY.length, 31);
it("RENAMED_MIGRATION_COMPATIBILITY has 32 well-formed entries", () => {
assert.equal(RENAMED_MIGRATION_COMPATIBILITY.length, 32);
for (const e of RENAMED_MIGRATION_COMPATIBILITY) {
assert.equal(typeof e.fromVersion, "string");
assert.equal(typeof e.fromName, "string");
@@ -113,6 +113,17 @@ describe("migrationRunner/constants — large-table integrity", () => {
"144",
]
);
assert.deepEqual(
RENAMED_MIGRATION_COMPATIBILITY.find(
(e) => e.fromVersion === "074" && e.fromName === "inspector_custom_hosts"
),
{
fromVersion: "074",
fromName: "inspector_custom_hosts",
toVersion: "081",
toName: "inspector_custom_hosts",
}
);
assert.deepEqual(RENAMED_MIGRATION_COMPATIBILITY.at(-7), {
fromVersion: "134",
fromName: "ccr_blocks",

329
tests/unit/db-pre-migration-backup-retention-10421.test.ts
View File

@@ -1,27 +1,24 @@
// ENVIRONMENT NOTE (sandbox better-sqlite3 / glibc limitation, not a code defect):
// This test constructs or exercises a real better-sqlite3-backed SQLite database.
// better-sqlite3 is a native addon; production and CI load it normally, but some
// sandboxes/dev boxes ship a system glibc older than the prebuilt binary requires
// ("GLIBC_2.29 not found"), so the native module fails to dlopen and any test that
// reaches better-sqlite3 directly (or asserts stdout that the load-failure warning
// would pollute) fails HERE while passing in CI. This is a known environment
// limitation, not a defect in the code under test: the OmniRoute runtime itself
// cascades to node:sqlite/sql.js when better-sqlite3 is unavailable. See
// tests/unit/_helpers/betterSqlite3Availability.ts for a guard helper.
// #10421 — pre-migration backups were created on every migration run and never pruned,
// so `db_backups/` grew without bound (observed: 48.999 files / 204 GB against a 5,3 MB
// live database). The pruning logic already existed in `cleanupDbBackups()` but nothing
// on the migration path ever reached it. These tests pin the retention step to the
// backup call site so the operator's maxFiles/retentionDays budget is honored there too.
// This suite uses a real on-disk better-sqlite3 database because migration snapshots
// must exercise SQLite's native read-only VACUUM path. Production and CI load the native
// addon normally; see tests/unit/_helpers/betterSqlite3Availability.ts for older sandboxes.
//
// #10421 — repeated failed startups once created a fresh timestamped snapshot every time
// and pruned unrelated restore points. Migration safety now publishes a content-addressed
// snapshot once per database state, never deletes a published snapshot, and leaves retention
// to the manual/scheduled backup paths outside the migration window.
import test from "node:test";
import assert from "node:assert/strict";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import test from "node:test";
import { pathToFileURL } from "node:url";
import Database from "better-sqlite3";
import { createBetterSqliteAdapter } from "../../src/lib/db/adapters/betterSqliteAdapter.ts";
const serial = { concurrency: false };
async function importFresh(modulePath: string) {
@@ -29,27 +26,23 @@ async function importFresh(modulePath: string) {
return import(`${url}?test=${Date.now()}-${Math.random().toString(16).slice(2)}`);
}
function withMockedMigrationFs(files: Record<string, string>, fn: () => void) {
function withMockedMigrationFs<T>(files: Record<string, string>, fn: () => T): T {
const originalExistsSync = fs.existsSync;
const originalReaddirSync = fs.readdirSync;
const originalReadFileSync = fs.readFileSync;
const isMigrationDir = (target: unknown) =>
String(target).replaceAll("\\", "/").endsWith("/src/lib/db/migrations") ||
String(target).replaceAll("\\", "/").endsWith("/migrations");
fs.existsSync = ((target: unknown) => {
if (isMigrationDir(target)) return true;
const fileName = path.basename(String(target));
if (Object.hasOwn(files, fileName)) return true;
if (Object.hasOwn(files, path.basename(String(target)))) return true;
return originalExistsSync(target as string);
}) as typeof fs.existsSync;
fs.readdirSync = ((target: string, options?: unknown) => {
if (isMigrationDir(target)) return Object.keys(files);
return originalReaddirSync(target, options as never);
}) as typeof fs.readdirSync;
fs.readFileSync = ((target: unknown, options?: unknown) => {
const fileName = path.basename(String(target));
if (Object.hasOwn(files, fileName)) return files[fileName];
@@ -65,149 +58,198 @@ function withMockedMigrationFs(files: Record<string, string>, fn: () => void) {
}
}
/** Minimal SqliteAdapter over a real on-disk file (VACUUM INTO needs a file, not :memory:). */
function createFileDb(sqlitePath: string) {
const db = new Database(sqlitePath);
return {
driver: "better-sqlite3",
get open() {
return db.open;
},
get name() {
return db.name;
},
prepare: (sql: string) => db.prepare(sql),
exec: (sql: string) => db.exec(sql),
pragma: (str: string, options?: unknown) => db.pragma(str, options as never),
transaction: (fn: (...args: unknown[]) => unknown) => {
const tx = db.transaction((...args: unknown[]) => fn(...args));
return (...args: unknown[]) => tx(...args);
},
immediate: (fn: () => void) => fn(),
async backup() {},
checkpoint() {},
close: () => db.close(),
get raw() {
return db;
},
};
return createBetterSqliteAdapter(new Database(sqlitePath));
}
/**
* Build a DB that already has migrations applied (so the pre-migration backup path is
* reached: it requires `applied.size > 0`) plus one pending migration to trigger a run.
*/
function seedAppliedDb(db: ReturnType<typeof createFileDb>) {
function seedExistingDb(db: ReturnType<typeof createFileDb>): void {
db.exec(`
CREATE TABLE provider_connections (id TEXT PRIMARY KEY);
CREATE TABLE combos (id TEXT PRIMARY KEY);
CREATE TABLE call_logs (id TEXT PRIMARY KEY);
`);
}
/**
* Record 001 as applied in the runner's own ledger table. `runMigrations` only takes a
* pre-migration backup when `applied.size > 0`, so this is what puts the test on the
* code path under exercise.
*/
function seedAppliedMigration(db: ReturnType<typeof createFileDb>) {
db.exec(`
CREATE TABLE IF NOT EXISTS _omniroute_migrations (
CREATE TABLE _omniroute_migrations (
version TEXT PRIMARY KEY,
name TEXT NOT NULL,
applied_at TEXT NOT NULL DEFAULT (datetime('now'))
);
INSERT INTO provider_connections (id) VALUES ('existing-data');
INSERT INTO _omniroute_migrations (version, name) VALUES ('001', 'initial_schema');
`);
db.prepare(
"INSERT OR REPLACE INTO _omniroute_migrations (version, name, applied_at) VALUES (?, ?, ?)"
).run("001", "initial_schema", new Date().toISOString());
}
function makeTempDataDir() {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-backup-retention-"));
function makeTempDataDir(): string {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-migration-snapshot-"));
fs.mkdirSync(path.join(dir, "db_backups"), { recursive: true });
return dir;
}
/** Pre-existing backups, oldest first, with distinct mtimes so retention ordering is stable. */
function seedBackups(backupDir: string, count: number) {
function seedTraditionalBackups(backupDir: string, count: number): string[] {
const names: string[] = [];
for (let i = 0; i < count; i++) {
const name = `db_2026-08-${String(i + 1).padStart(2, "0")}T00-00-00-000Z_pre-migration.sqlite`;
const filePath = path.join(backupDir, name);
fs.writeFileSync(filePath, "x");
const t = new Date(2026, 7, i + 1).getTime() / 1000;
fs.utimesSync(filePath, t, t);
for (let index = 0; index < count; index += 1) {
const name =
`db_2026-08-${String(index + 1).padStart(2, "0")}` + "T00-00-00-000Z_pre-migration.sqlite";
fs.writeFileSync(path.join(backupDir, name), `seed-${index}`);
names.push(name);
}
return names;
}
function countBackups(backupDir: string) {
return fs.readdirSync(backupDir).filter((n) => n.startsWith("db_")).length;
function listCanonicalBackups(backupDir: string): string[] {
if (!fs.existsSync(backupDir)) return [];
return fs
.readdirSync(backupDir)
.filter((name) => name.startsWith("db_") && name.endsWith(".sqlite"))
.sort();
}
function withEnv(vars: Record<string, string | undefined>, fn: () => void) {
const saved: Record<string, string | undefined> = {};
for (const [k, v] of Object.entries(vars)) {
saved[k] = process.env[k];
if (v === undefined) delete process.env[k];
else process.env[k] = v;
function listOwnedTempDirs(backupDir: string): string[] {
if (!fs.existsSync(backupDir)) return [];
return fs.readdirSync(backupDir).filter((name) => name.startsWith(".migration-snapshot-"));
}
function withEnv<T>(vars: Record<string, string | undefined>, fn: () => T): T {
const saved = new Map<string, string | undefined>();
for (const [key, value] of Object.entries(vars)) {
saved.set(key, process.env[key]);
if (value === undefined) delete process.env[key];
else process.env[key] = value;
}
try {
return fn();
} finally {
for (const [k, v] of Object.entries(saved)) {
if (v === undefined) delete process.env[k];
else process.env[k] = v;
for (const [key, value] of saved) {
if (value === undefined) delete process.env[key];
else process.env[key] = value;
}
}
}
test(
"#10421 runMigrations prunes pre-migration backups to the configured maxFiles",
"repeated zero-progress failures reuse one content-addressed snapshot without pruning",
serial,
async () => {
const dataDir = makeTempDataDir();
const backupDir = path.join(dataDir, "db_backups");
const sqlitePath = path.join(dataDir, "storage.sqlite");
const db = createFileDb(sqlitePath);
const db = createFileDb(path.join(dataDir, "storage.sqlite"));
try {
seedAppliedDb(db);
seedBackups(backupDir, 30);
assert.equal(countBackups(backupDir), 30, "precondition: 30 stale backups on disk");
seedExistingDb(db);
const seeded = seedTraditionalBackups(backupDir, 6);
const { runMigrations } = await importFresh("src/lib/db/migrationRunner.ts");
const files = {
"001_initial_schema.sql": "SELECT 1;",
"002_broken_probe.sql": "INSERT INTO table_that_does_not_exist VALUES (1);",
};
const fail = () => withMockedMigrationFs(files, () => runMigrations(db));
withEnv(
{
DB_BACKUP_MAX_FILES: "5",
DB_BACKUP_RETENTION_DAYS: "0",
DISABLE_SQLITE_AUTO_BACKUP: undefined,
},
() => {
withMockedMigrationFs(
{
"001_initial_schema.sql": "SELECT 1;",
"002_retention_probe.sql": "CREATE TABLE retention_probe_10421 (id INTEGER);",
},
() => {
// Mark 001 as applied so `applied.size > 0` and the backup path is reached.
seedAppliedMigration(db);
runMigrations(db);
}
);
}
assert.throws(fail, /table_that_does_not_exist/);
const afterFirst = listCanonicalBackups(backupDir);
const contentAddressed = afterFirst.filter((name) => name.startsWith("db_state-"));
assert.equal(contentAddressed.length, 1);
assert.match(contentAddressed[0]!, /^db_state-[a-f0-9]{64}_pre-migration\.sqlite$/);
assert.equal(
seeded.every((name) => afterFirst.includes(name)),
true,
"migration failure must not prune pre-existing restore points"
);
const remaining = countBackups(backupDir);
assert.throws(fail, /table_that_does_not_exist/);
assert.deepEqual(
listCanonicalBackups(backupDir),
afterFirst,
"an unchanged failed startup must reuse the exact content-addressed snapshot"
);
assert.deepEqual(listOwnedTempDirs(backupDir), []);
} finally {
db.close();
fs.rmSync(dataDir, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });
}
}
);
test(
"an existing DB fails closed when hard-link publication is unavailable even with auto backup disabled",
serial,
async () => {
const dataDir = makeTempDataDir();
const backupDir = path.join(dataDir, "db_backups");
const db = createFileDb(path.join(dataDir, "storage.sqlite"));
const originalLinkSync = fs.linkSync;
try {
seedExistingDb(db);
const { runMigrations } = await importFresh("src/lib/db/migrationRunner.ts");
fs.linkSync = (() => {
throw Object.assign(new Error("hard links unsupported by this filesystem"), {
code: "ENOTSUP",
});
}) as typeof fs.linkSync;
assert.throws(
() =>
withEnv({ DISABLE_SQLITE_AUTO_BACKUP: "true" }, () =>
withMockedMigrationFs(
{
"001_initial_schema.sql": "SELECT 1;",
"002_ordinary_pending.sql": "CREATE TABLE must_not_apply (id INTEGER);",
},
() => runMigrations(db)
)
),
/durable snapshot.*hard links unsupported.*hard links.*synchronization/is
);
assert.equal(
db.prepare("SELECT name FROM sqlite_master WHERE name = 'must_not_apply'").get(),
undefined,
"an ordinary pending migration must not run without its mandatory snapshot"
);
assert.deepEqual(
db.prepare("SELECT version, name FROM _omniroute_migrations ORDER BY version").all(),
[{ version: "001", name: "initial_schema" }]
);
assert.deepEqual(listCanonicalBackups(backupDir), []);
assert.deepEqual(listOwnedTempDirs(backupDir), []);
} finally {
fs.linkSync = originalLinkSync;
db.close();
fs.rmSync(dataDir, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });
}
}
);
test(
"successful migrations retain existing backups and do not prune inside the migration window",
serial,
async () => {
const dataDir = makeTempDataDir();
const backupDir = path.join(dataDir, "db_backups");
const db = createFileDb(path.join(dataDir, "storage.sqlite"));
try {
seedExistingDb(db);
const seeded = seedTraditionalBackups(backupDir, 6);
const { runMigrations } = await importFresh("src/lib/db/migrationRunner.ts");
const count = withEnv({ DB_BACKUP_MAX_FILES: "1", DB_BACKUP_RETENTION_DAYS: "0" }, () =>
withMockedMigrationFs(
{
"001_initial_schema.sql": "SELECT 1;",
"002_success.sql": "CREATE TABLE migration_success (id INTEGER);",
},
() => runMigrations(db)
)
);
assert.equal(count, 1);
assert.ok(
remaining <= 5,
`expected retention to cap db_backups at 5 files, found ${remaining} — ` +
`pre-migration backups are accumulating unbounded (#10421)`
db.prepare("SELECT name FROM sqlite_master WHERE name = 'migration_success'").get()
);
const after = listCanonicalBackups(backupDir);
assert.equal(after.filter((name) => name.startsWith("db_state-")).length, 1);
assert.equal(
seeded.every((name) => after.includes(name)),
true,
"retention must remain outside the concurrent migration window"
);
} finally {
db.close();
@@ -216,51 +258,26 @@ test(
}
);
test("#10421 the newest pre-migration backup survives pruning", serial, async () => {
test("an already-current DB does not acquire an IMMEDIATE writer lock", serial, async () => {
const dataDir = makeTempDataDir();
const backupDir = path.join(dataDir, "db_backups");
const sqlitePath = path.join(dataDir, "storage.sqlite");
const db = createFileDb(sqlitePath);
const db = createFileDb(path.join(dataDir, "storage.sqlite"));
try {
seedAppliedDb(db);
seedBackups(backupDir, 10);
seedExistingDb(db);
const { runMigrations } = await importFresh("src/lib/db/migrationRunner.ts");
withEnv(
{
DB_BACKUP_MAX_FILES: "3",
DB_BACKUP_RETENTION_DAYS: "0",
DISABLE_SQLITE_AUTO_BACKUP: undefined,
const noWriterAdapter = {
...db,
immediate: () => {
throw new Error("unexpected IMMEDIATE writer lock");
},
() => {
withMockedMigrationFs(
{
"001_initial_schema.sql": "SELECT 1;",
"002_retention_probe.sql": "CREATE TABLE retention_probe_10421b (id INTEGER);",
},
() => {
seedAppliedMigration(db);
};
runMigrations(db);
}
);
}
assert.equal(
withMockedMigrationFs({ "001_initial_schema.sql": "SELECT 1;" }, () =>
runMigrations(noWriterAdapter)
),
0
);
const remaining = fs.readdirSync(backupDir).filter((n) => n.startsWith("db_"));
assert.ok(remaining.length <= 3, `expected <=3 backups, found ${remaining.length}`);
// The backup written by THIS run must be among the survivors — pruning must never
// discard the snapshot that protects the migration it was taken for.
const seededNames = new Set(
Array.from({ length: 10 }, (_, i) => {
return `db_2026-08-${String(i + 1).padStart(2, "0")}T00-00-00-000Z_pre-migration.sqlite`;
})
);
const fresh = remaining.filter((n) => !seededNames.has(n));
assert.equal(fresh.length, 1, `expected the run's own backup to survive, got ${fresh.length}`);
} finally {
db.close();
fs.rmSync(dataDir, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });