Compare commits

..

4 Commits

Author SHA1 Message Date
diegosouzapw
8506966d7e docs(i18n): re-sync the 42 llm.txt mirrors after the migration-count bump 2026-08-27 17:25:11 -03:00
diegosouzapw
b95fedfc2a docs: bump the migration count to 160 after 163_model_capabilities
check:docs-counts-sync enforces the shipped migration count as a STRICT claim in README.md,
AGENTS.md and llm.txt.
2026-08-27 17:24:39 -03:00
diegosouzapw
eafca19b37 docs(db): record the real cause of the cache_metrics residual in the allowlist
The allowlist described every residual as "a CREATE that left the migration set in some
past cycle". cache_metrics never was in the migration set: it is created lazily by
ensureCacheMetricsTable() (src/lib/semanticCache.ts:34) the first time the semantic cache
runs, which is the same class as the model_capabilities divergence that blocked the v3.8.50
publish. Document both causes so the next residual is fixed with a migration where that is
the right answer, instead of reflexively allowlisted.
2026-08-27 17:17:37 -03:00
diegosouzapw
6a23c1b645 fix(db): converge the install and upgrade schemas; stop ENOSPC from faking a divergence
The v3.8.50 publish run failed `check:install-upgrade` with "15 tables a CLEAN install
creates but an UPGRADE does not" (agentic_conversations, ccr_blocks, the whole Radar set,
jobs/job_runs, exclusive_connection_leases, …). None of them was missing.

Root cause, from the CI log (run 33104507735): the Phase B upgrade `npm install` hit
`npm warn tar TAR_ENTRY_ERROR ENOSPC: no space left on device` 5611 times, npm still exited
0, and the resulting truncated package made `omniroute serve` "exit with code 0 before
serving". No migration ever ran, so the database still held the 3.8.49 schema (115 tables)
and every post-133 migration table read as a divergence.

Verified against the real thing: booting the published omniroute@3.8.49 and replaying that
database through the current runner applies exactly 29 migrations and lands on the same
table set a clean install produces — the migration set was never at fault.

What changes:

- `163_model_capabilities.sql` — the one genuine convergence defect. The table was only
  ever created by `ensureCapabilitiesTable()` on the first models.dev sync, so whether a
  database has it depends on timing, not on the schema version. It is the residual the
  gate reported. A migration makes both install paths deterministic.
- `check:install-upgrade` now fails on an ENOSPC-truncated install instead of measuring a
  broken tree; authenticates its health probe with a minted internal-service token, so the
  version assertion works against the health payload hardened by GHSA-mvf8-qc78-5mxm
  (an anonymous caller gets no version — the same run also failed with "health reports
  version undefined"); frees the ~3 GB clean-install tree before the upgrade phase; warns
  when the temp filesystem cannot hold the run; prints the failing server's output; and
  skips the convergence verdict when a phase never served, so a broken boot can no longer
  manufacture a schema divergence on top of the real failure.

Tests: `tests/unit/db-install-upgrade-schema-parity.test.ts` pins the deterministic half of
the gate in milliseconds (every migration reachable on a clean install; model_capabilities
comes from the migration set; its DDL does not drift from the runtime helper), and the
ENOSPC guard is covered in the existing gate test.
2026-08-27 17:16:14 -03:00
5 changed files with 4 additions and 41 deletions

View File

@@ -1023,13 +1023,6 @@ PROVIDER_LIMITS_SYNC_SPACING_MS=1500
# to disable the check. Used by: src/lib/db/migrationRunner.ts. Default: 50.
#OMNIROUTE_MAX_PENDING_MIGRATIONS=50
# Working directory for the check:install-upgrade release gate. It builds two ~3 GB
# install trees plus a ~275 MB tarball, so it needs roughly 12 GB — more than the
# 12 GB RAM-backed tmpfs that /tmp is on the self-hosted runner, where it exhausted
# the tmpfs and npm silently truncated the package. Defaults to <repo>/.install-upgrade
# on real disk. Used by: scripts/check/check-install-upgrade.mjs. Default: <repo>/.install-upgrade.
#OMNIROUTE_INSTALL_UPGRADE_WORKDIR=/var/tmp/omniroute-install-upgrade
# Trust user-managed RTK project filter rules without strict signature checks.
# Used by: open-sse/services/compression/engines/rtk/filterLoader.ts. Default: 0.
#OMNIROUTE_RTK_TRUST_PROJECT_FILTERS=0

View File

@@ -287,12 +287,7 @@ jobs:
# a staged package that is never approved simply expires, with no `npm deprecate` needed.
- name: Prove clean-install AND upgrade-over-previous both boot
if: steps.resolve.outputs.skip != 'true'
# 60, not 30. This gate was added in #8953 and the 2026-08-27 v3.8.50 publish
# was the FIRST run to ever reach it — every earlier attempt died upstream, so
# its budget had never been measured against a real run. It then blew the limit
# on its debut: `npm pack` alone took 24m37s, leaving 5 minutes for two installs
# and two boots. 30 was a guess; 60 is sized to the one measurement we have.
timeout-minutes: 60
timeout-minutes: 30
run: npm run check:install-upgrade
# WS1.3 (D2, v3.8.49 plan): STAGED publishing by default — `npm stage publish`

3
.gitignore vendored
View File

@@ -293,6 +293,3 @@ docker-compose.yml.bak
# Ad-hoc test sandboxes (never tracked — may contain local DBs)
/.sandbox/
.aider*
# check:install-upgrade work trees (~12 GB, disposable)
/.install-upgrade/

View File

@@ -103,7 +103,6 @@ OmniRoute uses **SQLite** (via `better-sqlite3`) for all persistence. These vari
| `OMNIROUTE_MIGRATIONS_DIR` | _(auto-detect)_ | `src/lib/db/migrationRunner.ts` | Override the directory that the migration runner scans. Useful when shipping bundled migrations in custom builds. |
| `OMNIROUTE_EXTRA_MIGRATIONS_DIRS` | _(unset)_ | `src/lib/db/migrationRunner/extraDirs.ts` | Additional migration directories as `namespace=dir` entries separated by the platform path delimiter (e.g. `ee=/opt/app/enterprise/db/migrations`). Files found there are recorded as `<namespace>-<number>`, so a distribution shipping its own migrations never collides with the upstream numeric slots. A malformed entry, an invalid namespace or a missing directory throws at startup instead of silently skipping the schema. |
| `OMNIROUTE_MAX_PENDING_MIGRATIONS` | `50` | `src/lib/db/migrationRunner.ts` | Mass-pending-migrations safety threshold (#3416). Startup aborts if more than this many migrations are pending on an existing DB (guards against a wiped tracking table). Raise it to restore an older backup; set to `0` to disable the check. |
| `OMNIROUTE_INSTALL_UPGRADE_WORKDIR` | _(`<repo>/.install-upgrade`)_ | `scripts/check/check-install-upgrade.mjs` | Working directory for the `check:install-upgrade` release gate. It needs roughly 12 GB (two ~3 GB install trees plus the tarball), so it must not run on a small tmpfs — on the self-hosted runner `/tmp` is a 12 GB RAM-backed tmpfs and the gate exhausted it, truncating the package. |
| `OMNIROUTE_SPEND_FLUSH_INTERVAL_MS` | _(default in code)_ | `src/lib/spend/batchWriter.ts` | Flush interval (ms) for the batched spend/cost writer. Lower values reduce write coalescing; higher values reduce DB contention. |
| `OMNIROUTE_SPEND_MAX_BUFFER_SIZE` | _(default in code)_ | `src/lib/spend/batchWriter.ts` | Max buffered spend entries before a forced flush. Raise on high-QPS deployments; lower when bounded memory matters more. |
| `OMNIROUTE_PROXY_FETCH_DEBUG` | _(unset)_ | `open-sse/utils/proxyFetch.ts` | Set to `"true"` to emit `[ProxyFetch]` debug logs on the Vercel relay path. Off by default to avoid leaking routing hints. |

View File

@@ -302,7 +302,7 @@ export function assertNoDiskExhaustion(output, label) {
throw new Error(
`${label}: the install ran out of disk space (${count} ENOSPC error(s) from npm). ` +
`The package tree is truncated, so anything measured from it — boot, schema, ` +
`migrations — is meaningless. Free space in ${workDirForMessages} (each install tree is ` +
`migrations — is meaningless. Free space in ${os.tmpdir()} (each install tree is ` +
`~3 GB) and re-run. This is an environment failure, NOT a schema divergence.`
);
}
@@ -318,11 +318,6 @@ function freeBytes(dir) {
const GB = 1024 ** 3;
// Set once the work directory exists, so the ENOSPC message names the filesystem that
// actually ran out — pointing at /tmp when the gate works elsewhere sends the reader to
// free space on the wrong volume (which is what happened during the v3.8.50 publish).
let workDirForMessages = os.tmpdir();
function resolvePreviousVersion(current, explicit) {
if (explicit) return explicit;
const out = execFileSync("npm", ["view", "omniroute", "dist-tags.latest"], { encoding: "utf8" });
@@ -353,25 +348,11 @@ async function main() {
}
const version = JSON.parse(fs.readFileSync(path.join(ROOT, "package.json"), "utf8")).version;
const allowlist = loadAllowlist(ROOT);
// NOT os.tmpdir(): on the self-hosted runner /tmp is a 12 GB tmpfs backed by RAM, while
// the root filesystem has ~66 GB free. This gate needs ~12 GB, so it exhausted the tmpfs
// and npm truncated the package — 58269 ENOSPC errors on the v3.8.50 publish, which the
// previous code could only report as a crash. Freeing disk did not help because the disk
// was never the constraint. Work on real disk beside the repo instead.
const workRoot =
process.env.OMNIROUTE_INSTALL_UPGRADE_WORKDIR || path.join(ROOT, ".install-upgrade");
fs.mkdirSync(workRoot, { recursive: true });
const tmp = fs.mkdtempSync(path.join(workRoot, "omniroute-install-upgrade-"));
workDirForMessages = tmp;
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-install-upgrade-"));
const failures = [];
const warnings = [];
try {
// Timed, because this turned out to be the expensive part: on the 2026-08-27
// v3.8.50 publish `npm pack` alone took 24m37s, leaving 5 of the step's 30-minute
// budget for two installs and two boots. Without a duration here the log showed
// only "packing…" then a timeout, which reads like a hang and is not.
const packStarted = Date.now();
log(`packing v${version}`);
const packOut = execFileSync("npm", ["pack", "--json", "--pack-destination", tmp], {
cwd: ROOT,
@@ -379,8 +360,6 @@ async function main() {
maxBuffer: 128 * 1024 * 1024,
});
const tarball = path.join(tmp, pickTarball(packOut));
const packMb = (fs.statSync(tarball).size / 1024 / 1024).toFixed(1);
log(`packed in ${Math.round((Date.now() - packStarted) / 1000)}s (${packMb} MB)`);
// Each install tree is ~3 GB and this run builds two of them, side by side, plus the
// ~275 MB tarball. On the v3.8.50 publish run that overflowed the runner disk mid-way
@@ -388,7 +367,7 @@ async function main() {
// exited 0, and every later measurement was taken from a broken tree.
const availableBytes = freeBytes(tmp);
if (availableBytes !== null) {
log(`free space in ${tmp}: ${(availableBytes / GB).toFixed(1)} GB`);
log(`free space in ${os.tmpdir()}: ${(availableBytes / GB).toFixed(1)} GB`);
if (availableBytes < 12 * GB) {
warn(
`only ${(availableBytes / GB).toFixed(1)} GB free — this gate needs roughly 12 GB ` +