docs(db): add reproducible SQLite coupling inventory (#8262)

Co-authored-by: 千乘妍 (Xiaoyaner) <xiaoyaner0201@users.noreply.github.com>
This commit is contained in:
小妍儿 ✨
2026-07-23 20:51:06 +08:00
committed by GitHub
parent da6d72e26b
commit fbb8d45757
3 changed files with 580 additions and 0 deletions

View File

@@ -0,0 +1,226 @@
---
title: "SQLite coupling inventory"
status: measured-snapshot
lastUpdated: 2026-07-23
---
# SQLite coupling inventory
- **Tracking issue:** [#8075](https://github.com/diegosouzapw/OmniRoute/issues/8075)
- **Snapshot revision:** `9a3b605f3420ae3ab08bd93d6443034f03a1bcbc`
- **Scanned-corpus SHA-256:** `72334620a7a18a42bcede1643fb2fdf95da6eae9ffa66a891ae14ed633ad43f6`
- **Purpose:** Measure the current persistence cut lines before proposing repository interfaces
- **Runtime impact:** None; this document and its audit script do not change database behavior
## How to reproduce
From the repository root:
```bash
node scripts/check/audit-sqlite-coupling.mjs
node scripts/check/audit-sqlite-coupling.mjs --json
node --test scripts/check/audit-sqlite-coupling.test.mjs
```
The script reads tracked files from Git, scans non-test source under `src/`, `open-sse/`,
`electron/`, and `bin/`, and scans migration SQL under `src/lib/db/migrations/`. It excludes the
top-level test tree, co-located test directories, test/spec source files, and paths outside those
configured source roots (including documentation and scripts).
The script refuses to run if tracked files in those source roots differ from `HEAD`. It reports
both the audit-tool revision and a SHA-256 over the ordered path/content corpus. The snapshot above
was taken from the listed source revision; this PR changes only excluded documentation and script
paths, so rerunning from the clean PR branch produces the same corpus digest.
This is a **lexical inventory**, not a TypeScript or SQL semantic analysis:
- counts are occurrences of defined patterns, not counts of distinct SQL statements;
- adapter-call and direct-singleton patterns mask comments and literal contents first;
- template-literal contents, including embedded expressions, are excluded from those code-syntax
counts;
- the lightweight masker is not a JavaScript parser, so unusual regular-expression literal syntax
can still require manual review;
- comments and string literals can contribute to dialect-signal counts, which intentionally search
raw text for embedded SQL;
- a `.prepare()` match outside `src/lib/db/` is a review lead, not proof that the call should move;
- calls hidden behind a differently named wrapper may not be counted;
- file counts are deduplicated, while occurrence counts are not.
The JSON output includes every matching path so reviewers can inspect or reclassify individual
results rather than trusting totals alone.
## Snapshot scope
At the recorded revision, the script scanned:
- 3,830 tracked non-test source files;
- 129 migration SQL files.
The source-file count is intentionally broad because the goal is to find persistence coupling that
has escaped the nominal database directory, including CLI and proxy/runtime code.
## Boundary signals
| Signal | Files | Occurrences |
| -------------------------------------------------------------------------------- | ----: | ----------: |
| Direct `getDbInstance()` call syntax outside comments/literals and `src/lib/db/` | 45 | 150 |
| `localDb` import consumers | 211 | — |
| `SqliteAdapter` type consumers outside comments/literals and `src/lib/db/` | 3 | — |
The `localDb` barrel already gives many callers a domain-function seam, but
`src/lib/localDb.ts` remains a re-export layer rather than a backend contract. The 45 direct
singleton consumers are the clearest first review set because they bypass that logical seam and
hold an adapter-shaped handle directly.
The three non-test source files outside `src/lib/db/` that mention the `SqliteAdapter` type in code
syntax are:
- `src/app/api/db-backups/import/route.ts`;
- `src/lib/compliance/index.ts`;
- `src/lib/compliance/noLog.ts`.
These are not equivalent migration tasks. Backup import is capability-specific; compliance
persistence may be portable domain state. The future boundary should classify them rather than
moving all three mechanically.
## Adapter-shaped call syntax
| Signal | Occurrences | Files | Outside `src/lib/db/` occurrences | Outside files |
| ----------------- | ----------: | ----: | --------------------------------: | ------------: |
| `.prepare()` | 1,219 | 163 | 252 | 52 |
| `.transaction()` | 62 | 40 | 12 | 10 |
| `.immediate()` | 3 | 3 | 0 | 0 |
| `.pragma()` | 39 | 11 | 6 | 4 |
| `.backup()` | 6 | 5 | 3 | 3 |
| `.checkpoint()` | 0 | 0 | 0 | 0 |
| `lastInsertRowid` | 15 | 7 | 1 | 1 |
This table shows why `SqliteAdapter` is a SQLite runtime compatibility layer rather than a portable
backend abstraction. Its synchronous statement and transaction shape is widely used, and some of
that shape is visible outside the nominal database layer.
The top direct `getDbInstance()` consumers outside `src/lib/db/` at this revision are:
| File | Occurrences |
| -------------------------------------------------- | ----------: |
| `src/lib/proxySubscription/subscriptionService.ts` | 12 |
| `src/lib/semanticCache.ts` | 10 |
| `src/lib/usage/callLogs.ts` | 9 |
| `src/lib/cloudAgent/db.ts` | 8 |
| `src/lib/memory/store.ts` | 8 |
| `src/lib/memory/vectorStore.ts` | 8 |
| `src/lib/modelsDevSync.ts` | 8 |
| `src/lib/gamification/badges.ts` | 5 |
| `src/lib/memory/retrieval.ts` | 5 |
| `src/lib/pricingSync.ts` | 5 |
| `src/lib/skills/registry.ts` | 5 |
| `src/lib/usage/usageHistory.ts` | 5 |
The list spans control-plane configuration, usage/audit data, cache, memory/vector search, skills,
gamification, and CLI/provider support. A single generic SQL adapter would preserve this spread;
domain repositories provide a way to reduce it slice by slice.
## SQLite dialect and lifecycle signals
| Signal | Occurrences | Files |
| --------------------- | ----------: | ----: |
| `PRAGMA` text | 97 | 41 |
| `sqlite_master` | 14 | 11 |
| `BEGIN IMMEDIATE` | 2 | 2 |
| `INSERT OR REPLACE` | 83 | 45 |
| `AUTOINCREMENT` | 34 | 24 |
| `datetime('now')` | 171 | 68 |
| `VACUUM` | 39 | 10 |
| `wal_checkpoint` | 13 | 7 |
| `fts5` | 43 | 8 |
| `vec0` | 7 | 1 |
| `last_insert_rowid()` | 1 | 1 |
These values are text signals and include comments where present. They are useful for locating
portability work, not for estimating implementation effort by multiplication.
Verified high-coupling areas include:
- `src/lib/db/core.ts`: singleton lifecycle, SQLite file paths, WAL checkpoint, recovery, schema,
compaction, and backup creation;
- `src/lib/db/migrationRunner.ts`: numbered SQL migration execution, `sqlite_master`,
`PRAGMA table_info`, transaction behavior, and optional FTS5 handling;
- `src/lib/db/optimizationSettings.ts`: page/cache settings, auto-vacuum, WAL transitions, and
`VACUUM`;
- `src/lib/db/backup.ts`: database backup and restore lifecycle;
- `src/lib/db/schemaColumns.ts`: SQLite schema introspection and compatibility columns;
- `src/lib/memory/vectorStore.ts` and `src/lib/memory/retrieval.ts`: `vec0` and FTS5 behavior;
- `src/lib/db/adapters/`: compatibility implementations for the supported SQLite runtimes.
These areas should not be forced through a lowest-common-denominator repository interface. They
need explicit SQLite capabilities or separate backend implementations.
## Migration coupling
The snapshot contains 129 tracked migration SQL files. `src/lib/db/migrationRunner.ts` does more
than execute ordered files: it owns migration discovery, version history, duplicate-version safety,
schema probes, FTS5 capability checks, pre-migration safety, and SQLite transaction execution.
Consequently:
- another SQL dialect cannot safely reuse the migration files unchanged;
- external backends need their own migration implementation and schema history;
- logical migration milestones may be shared, but physical SQL and capability probes remain
backend-specific;
- multi-replica operation requires migration ownership or locking before an external backend is
considered ready.
## Recommended cut lines
### 1. Keep SQLite runtime compatibility intact
Do not replace `SqliteAdapter` or the driver cascade in the first repository PR. Keep file recovery,
WAL, backup, optimization, FTS5, and vector behavior behind the current SQLite implementation.
### 2. Start with direct singleton consumers
Use the 45-file direct-consumer list as the initial review queue. Classify each file as:
- portable domain state;
- backend-specific maintenance or search;
- process-local or rebuildable state;
- legacy access that should call an existing domain module.
Classification must precede interface design. A path appearing in the inventory is not, by itself,
a mandate to create a repository.
### 3. Prove repositories with SQLite first
For one bounded domain:
1. define behavior-oriented repository operations;
2. adapt current SQLite queries behind that repository;
3. run behavior and transaction conformance tests against SQLite;
4. migrate callers without changing the default runtime;
5. only then implement the same repository for an external backend.
### 4. Separate portable control-plane state from capability-specific data
Provider connections, API keys, combos, and routing configuration are candidates for the first
portable slice, subject to maintainer approval and a table-ownership review. Memory vector search,
SQLite file backup/recovery, and database optimization are poor first slices because their behavior
is deliberately SQLite-specific.
### 5. Treat usage, quota, affinity, and audit as a later coordination slice
These domains have concurrency and volume semantics beyond CRUD. Their repository contracts should
be designed together with multi-replica transaction, lease, retention, and failure-mode tests rather
than copied mechanically from current SQL.
## What this inventory does not decide
This inventory does not:
- approve PostgreSQL or MySQL support;
- define repository TypeScript interfaces;
- choose the first table or domain to migrate;
- claim every lexical match is a defect;
- claim the current module boundaries are ineffective;
- change SQLite, migrations, backup, search, or runtime behavior.
Its purpose is to make the next design discussion evidence-based and reproducible.

View File

@@ -0,0 +1,277 @@
#!/usr/bin/env node
import { execFileSync } from "node:child_process";
import { createHash } from "node:crypto";
import { readFileSync } from "node:fs";
import path from "node:path";
const root = process.cwd();
const toolRevision = execFileSync("git", ["rev-parse", "HEAD"], {
cwd: root,
encoding: "utf8",
}).trim();
const sourceRoots = ["src/", "open-sse/", "electron/", "bin/"];
try {
execFileSync("git", ["diff", "--quiet", "HEAD", "--", ...sourceRoots], {
cwd: root,
stdio: "ignore",
});
} catch {
throw new Error(
"Refusing to audit: tracked files in the configured source roots differ from HEAD. Commit or restore them first."
);
}
const trackedFiles = execFileSync("git", ["ls-files", "-z"], {
cwd: root,
encoding: "utf8",
})
.split("\0")
.filter(Boolean);
const sourceExtensions = new Set([".ts", ".tsx", ".js", ".mjs", ".cjs"]);
const isTestFile = (file) =>
file.includes("/__tests__/") || /\.(?:test|spec)\.[cm]?[jt]sx?$/.test(file);
const sourceFiles = trackedFiles.filter(
(file) =>
sourceRoots.some((prefix) => file.startsWith(prefix)) &&
sourceExtensions.has(path.extname(file)) &&
!isTestFile(file)
);
const migrationFiles = trackedFiles.filter(
(file) => file.startsWith("src/lib/db/migrations/") && file.endsWith(".sql")
);
const scannedFiles = [...sourceFiles, ...migrationFiles];
const contents = new Map(
scannedFiles.map((file) => [file, readFileSync(path.join(root, file), "utf8")])
);
const corpusSha256 = createHash("sha256")
.update(
scannedFiles
.sort()
.map((file) => `${file}\0${contents.get(file)}\0`)
.join("")
)
.digest("hex");
function countMatches(text, regex) {
return [
...text.matchAll(
new RegExp(regex.source, regex.flags.includes("g") ? regex.flags : `${regex.flags}g`)
),
].length;
}
function maskCommentsAndLiterals(text) {
const output = [...text];
let state = "code";
for (let index = 0; index < text.length; index += 1) {
const char = text[index];
const next = text[index + 1];
if (state === "code") {
if (char === "/" && next === "/") {
output[index] = output[index + 1] = " ";
index += 1;
state = "line-comment";
} else if (char === "/" && next === "*") {
output[index] = output[index + 1] = " ";
index += 1;
state = "block-comment";
} else if (char === "'") {
output[index] = " ";
state = "single-quote";
} else if (char === '"') {
output[index] = " ";
state = "double-quote";
} else if (char === "`") {
output[index] = " ";
state = "template";
}
continue;
}
if (char !== "\n" && char !== "\r") output[index] = " ";
if (state === "line-comment" && (char === "\n" || char === "\r")) state = "code";
if (state === "block-comment" && char === "*" && next === "/") {
output[index + 1] = " ";
index += 1;
state = "code";
} else if (state === "single-quote" && char === "\\") {
if (index + 1 < text.length) output[index + 1] = " ";
index += 1;
} else if (state === "single-quote" && char === "'") {
state = "code";
} else if (state === "double-quote" && char === "\\") {
if (index + 1 < text.length) output[index + 1] = " ";
index += 1;
} else if (state === "double-quote" && char === '"') {
state = "code";
} else if (state === "template" && char === "\\") {
if (index + 1 < text.length) output[index + 1] = " ";
index += 1;
} else if (state === "template" && char === "`") {
state = "code";
}
}
return output.join("");
}
const codeContents = new Map(
sourceFiles.map((file) => [file, maskCommentsAndLiterals(contents.get(file) ?? "")])
);
function collect(regex, files = sourceFiles, source = codeContents) {
const matches = [];
let occurrences = 0;
for (const file of files) {
const count = countMatches(source.get(file) ?? "", regex);
if (count === 0) continue;
occurrences += count;
matches.push({ file, occurrences: count });
}
matches.sort(
(left, right) => right.occurrences - left.occurrences || left.file.localeCompare(right.file)
);
return { occurrences, files: matches.length, matches };
}
function splitDbLayer(result) {
const inside = result.matches.filter((entry) => entry.file.startsWith("src/lib/db/"));
const outside = result.matches.filter((entry) => !entry.file.startsWith("src/lib/db/"));
return {
total: { occurrences: result.occurrences, files: result.files, matches: result.matches },
insideDbLayer: {
occurrences: inside.reduce((sum, entry) => sum + entry.occurrences, 0),
files: inside.length,
matches: inside,
},
outsideDbLayer: {
occurrences: outside.reduce((sum, entry) => sum + entry.occurrences, 0),
files: outside.length,
matches: outside,
},
};
}
const adapterSurfacePatterns = {
prepare: /\.prepare(?:<[^>]+>)?\s*\(/g,
transaction: /\.transaction\s*\(/g,
immediate: /\.immediate\s*\(/g,
pragma: /\.pragma\s*\(/g,
backup: /\.backup\s*\(/g,
checkpoint: /\.checkpoint\s*\(/g,
lastInsertRowid: /\blastInsertRowid\b/g,
};
const sqliteDialectPatterns = {
pragma: /\bPRAGMA\b/gi,
sqliteMaster: /\bsqlite_master\b/gi,
beginImmediate: /\bBEGIN\s+IMMEDIATE\b/gi,
insertOrReplace: /\bINSERT\s+OR\s+REPLACE\b/gi,
autoincrement: /\bAUTOINCREMENT\b/gi,
datetimeNow: /datetime\s*\(\s*['"]now['"]\s*\)/gi,
vacuum: /\bVACUUM\b/gi,
walCheckpoint: /\bwal_checkpoint\b/gi,
fts5: /\bfts5\b/gi,
vec0: /\bvec0\b/gi,
lastInsertRowidFunction: /\blast_insert_rowid\s*\(/gi,
};
const adapterSurface = Object.fromEntries(
Object.entries(adapterSurfacePatterns).map(([name, regex]) => [
name,
splitDbLayer(collect(regex)),
])
);
const sqliteDialect = Object.fromEntries(
Object.entries(sqliteDialectPatterns).map(([name, regex]) => [
name,
collect(regex, scannedFiles, contents),
])
);
const directCoreConsumers = collect(/\bgetDbInstance\s*\(/g).matches.filter(
(entry) => !entry.file.startsWith("src/lib/db/")
);
const localDbImportConsumers = sourceFiles
.filter((file) =>
/(?:from|import\s*\()\s*["'][^"']*(?:\/lib\/localDb|\.\/localDb)["']/.test(
contents.get(file) ?? ""
)
)
.sort();
const sqliteAdapterTypeConsumers = sourceFiles
.filter(
(file) =>
!file.startsWith("src/lib/db/") && /\bSqliteAdapter\b/.test(codeContents.get(file) ?? "")
)
.sort();
const inventory = {
toolRevision,
corpusSha256,
scope: {
sourceRoots,
sourceFiles: sourceFiles.length,
migrationFiles: migrationFiles.length,
exclusions: [
"tests/",
"co-located __tests__ directories",
"*.test.* and *.spec.* source files",
"paths outside the configured source roots (including docs/ and scripts/)",
],
},
boundaries: {
directGetDbInstanceOutsideDbLayer: {
files: directCoreConsumers.length,
occurrences: directCoreConsumers.reduce((sum, entry) => sum + entry.occurrences, 0),
matches: directCoreConsumers,
},
localDbImportConsumers: {
files: localDbImportConsumers.length,
paths: localDbImportConsumers,
},
sqliteAdapterTypeConsumersOutsideDbLayer: {
files: sqliteAdapterTypeConsumers.length,
paths: sqliteAdapterTypeConsumers,
},
},
adapterSurface,
sqliteDialect,
};
function markdownTable(rows) {
return [
"| Signal | Occurrences | Files | Outside `src/lib/db/` occurrences | Outside files |",
"| --- | ---: | ---: | ---: | ---: |",
...rows.map(
([name, value]) =>
`| \`${name}\` | ${value.total.occurrences} | ${value.total.files} | ${value.outsideDbLayer.occurrences} | ${value.outsideDbLayer.files} |`
),
].join("\n");
}
function renderTopMatches(matches, limit = 12) {
if (matches.length === 0) return "- None";
return matches
.slice(0, limit)
.map((entry) => `- \`${entry.file}\`: ${entry.occurrences}`)
.join("\n");
}
function renderMarkdown() {
const dialectRows = Object.entries(inventory.sqliteDialect).map(
([name, value]) => `| \`${name}\` | ${value.occurrences} | ${value.files} |`
);
return `# SQLite coupling audit output\n\n- Audit-tool revision: \`${inventory.toolRevision}\`\n- Scanned-corpus SHA-256: \`${inventory.corpusSha256}\`\n- Tracked non-test source files scanned: ${inventory.scope.sourceFiles}\n- Migration SQL files scanned: ${inventory.scope.migrationFiles}\n\n## Boundary signals\n\n- Direct \`getDbInstance()\` call syntax outside comments and literals, excluding \`src/lib/db/\`: ${inventory.boundaries.directGetDbInstanceOutsideDbLayer.files} files / ${inventory.boundaries.directGetDbInstanceOutsideDbLayer.occurrences} occurrences\n- \`@/lib/localDb\` or relative \`localDb\` import consumers: ${inventory.boundaries.localDbImportConsumers.files} files\n- \`SqliteAdapter\` type consumers outside comments and literals, excluding \`src/lib/db/\`: ${inventory.boundaries.sqliteAdapterTypeConsumersOutsideDbLayer.files} files\n\n## Adapter-shaped call syntax\n\n${markdownTable(Object.entries(inventory.adapterSurface))}\n\n## SQLite dialect and lifecycle text\n\n| Signal | Occurrences | Files |\n| --- | ---: | ---: |\n${dialectRows.join("\n")}\n\n## Top direct \`getDbInstance()\` consumers outside the DB layer\n\n${renderTopMatches(inventory.boundaries.directGetDbInstanceOutsideDbLayer.matches)}\n\n## Top \`.prepare()\` files outside the DB layer\n\n${renderTopMatches(inventory.adapterSurface.prepare.outsideDbLayer.matches)}\n`;
}
if (process.argv.includes("--json")) {
process.stdout.write(`${JSON.stringify(inventory, null, 2)}\n`);
} else {
process.stdout.write(renderMarkdown());
}

View File

@@ -0,0 +1,77 @@
import assert from "node:assert/strict";
import { execFileSync, spawnSync } from "node:child_process";
import { cpSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import test from "node:test";
import { fileURLToPath } from "node:url";
const scriptPath = path.join(
path.dirname(fileURLToPath(import.meta.url)),
"audit-sqlite-coupling.mjs"
);
function git(cwd, ...args) {
return execFileSync("git", args, { cwd, encoding: "utf8" });
}
test("audit separates code syntax from comments/literals and rejects a dirty corpus", () => {
const root = mkdtempSync(path.join(tmpdir(), "omniroute-sqlite-audit-"));
try {
mkdirSync(path.join(root, "scripts/check"), { recursive: true });
mkdirSync(path.join(root, "src/lib/db/migrations"), { recursive: true });
cpSync(scriptPath, path.join(root, "scripts/check/audit-sqlite-coupling.mjs"));
writeFileSync(
path.join(root, "src/lib/db/core.ts"),
`export function internal(db) { return db.prepare("SELECT 1"); }\n`
);
writeFileSync(
path.join(root, "src/consumer.ts"),
`import { getDbInstance } from "./lib/db/core";\n` +
`import { getSettings } from "@/lib/localDb";\n` +
`// getDbInstance().prepare("comment only")\n` +
`const fixture = "getDbInstance().prepare('literal only')";\n` +
"const template = `getDbInstance().prepare('template only')`;\n" +
`export const value = getDbInstance().prepare("SELECT 1");\n`
);
writeFileSync(
path.join(root, "src/lib/db/migrations/001_fixture.sql"),
"PRAGMA foreign_keys = ON;\nCREATE TABLE fixture (id INTEGER PRIMARY KEY AUTOINCREMENT);\n"
);
git(root, "init", "-q");
git(root, "config", "user.email", "audit@example.invalid");
git(root, "config", "user.name", "Audit Test");
git(root, "add", ".");
git(root, "commit", "-qm", "fixture");
const result = JSON.parse(
execFileSync("node", ["scripts/check/audit-sqlite-coupling.mjs", "--json"], {
cwd: root,
encoding: "utf8",
})
);
assert.equal(result.boundaries.directGetDbInstanceOutsideDbLayer.files, 1);
assert.equal(result.boundaries.directGetDbInstanceOutsideDbLayer.occurrences, 1);
assert.equal(result.boundaries.localDbImportConsumers.files, 1);
assert.equal(result.adapterSurface.prepare.total.occurrences, 2);
assert.equal(result.adapterSurface.prepare.insideDbLayer.occurrences, 1);
assert.equal(result.adapterSurface.prepare.outsideDbLayer.occurrences, 1);
assert.equal(result.adapterSurface.prepare.total.matches.length, 2);
assert.equal(result.sqliteDialect.pragma.occurrences, 1);
assert.match(result.corpusSha256, /^[a-f0-9]{64}$/);
writeFileSync(path.join(root, "src/consumer.ts"), "// dirty tracked source\n", { flag: "a" });
const dirtyRun = spawnSync("node", ["scripts/check/audit-sqlite-coupling.mjs", "--json"], {
cwd: root,
encoding: "utf8",
});
assert.notEqual(dirtyRun.status, 0);
assert.match(dirtyRun.stderr, /Refusing to audit/);
} finally {
rmSync(root, { recursive: true, force: true });
}
});