fix(ci): run the install-upgrade gate on disk, not on the /tmp tmpfs (#11855)

* fix(ci): run the install-upgrade gate on disk, not on the /tmp tmpfs

The v3.8.50 publish failed this gate again, and this time it said why:

  free space in /tmp: 2.9 GB
  ⚠️  only 2.9 GB free — this gate needs roughly 12 GB
  crashed: upgrade install ran out of disk space (58269 ENOSPC errors)

On the self-hosted runner `/tmp` is a **12 GB tmpfs backed by RAM**, while the
root filesystem had 66 GB free. The gate builds two ~3 GB install trees, installs
the second one over twice, and packs a 275 MB tarball — roughly 12 GB, all of it
demanded from the wrong filesystem.

This is why freeing disk never fixed it: 84 GB were freed on `/`, and none of it
ever reached the volume the gate was using. The check even measured the right
number and reported it against the wrong path, so the warning read as "the disk
is full" when the disk was fine.

- work in `<repo>/.install-upgrade/` (gitignored) instead of `os.tmpdir()`,
  overridable with `OMNIROUTE_INSTALL_UPGRADE_WORKDIR`
- the free-space log and the ENOSPC crash message now name the directory the run
  actually uses, so the next reader is sent to the filesystem that ran out

Phase A already passes on the current main: clean install healthy, version
reported correctly, 130 tables — the authentication fix and migration 163 from
#11845 both hold. Only Phase B was starved.

* docs(env): document OMNIROUTE_INSTALL_UPGRADE_WORKDIR

The workdir override introduced in this branch is a new `process.env.*` read, and
two gates caught it immediately: `issue #7793: real .env.example is in sync with
process.env.* reads in code` and `check:env-doc-sync` (Docs Sync STRICT).

Both were right — an env var that exists only in code is an env var nobody can
find. Documented in `.env.example` and `docs/reference/ENVIRONMENT.md` with the
reason it exists: the gate needs ~12 GB and must not land on a small tmpfs.

.env.example

@@ -1023,6 +1023,13 @@ PROVIDER_LIMITS_SYNC_SPACING_MS=1500
 # to disable the check. Used by: src/lib/db/migrationRunner.ts.  Default: 50.
 #OMNIROUTE_MAX_PENDING_MIGRATIONS=50
 
+# Working directory for the check:install-upgrade release gate. It builds two ~3 GB
+# install trees plus a ~275 MB tarball, so it needs roughly 12 GB — more than the
+# 12 GB RAM-backed tmpfs that /tmp is on the self-hosted runner, where it exhausted
+# the tmpfs and npm silently truncated the package. Defaults to <repo>/.install-upgrade
+# on real disk. Used by: scripts/check/check-install-upgrade.mjs.  Default: <repo>/.install-upgrade.
+#OMNIROUTE_INSTALL_UPGRADE_WORKDIR=/var/tmp/omniroute-install-upgrade
+
 # Trust user-managed RTK project filter rules without strict signature checks.
 # Used by: open-sse/services/compression/engines/rtk/filterLoader.ts.  Default: 0.
 #OMNIROUTE_RTK_TRUST_PROJECT_FILTERS=0

.github/workflows/npm-publish.yml

@@ -287,7 +287,12 @@ jobs:
       # a staged package that is never approved simply expires, with no `npm deprecate` needed.
       - name: Prove clean-install AND upgrade-over-previous both boot
         if: steps.resolve.outputs.skip != 'true'
-        timeout-minutes: 30
+        # 60, not 30. This gate was added in #8953 and the 2026-08-27 v3.8.50 publish
+        # was the FIRST run to ever reach it — every earlier attempt died upstream, so
+        # its budget had never been measured against a real run. It then blew the limit
+        # on its debut: `npm pack` alone took 24m37s, leaving 5 minutes for two installs
+        # and two boots. 30 was a guess; 60 is sized to the one measurement we have.
+        timeout-minutes: 60
         run: npm run check:install-upgrade
 
       # WS1.3 (D2, v3.8.49 plan): STAGED publishing by default — `npm stage publish`

.gitignore

@@ -293,3 +293,6 @@ docker-compose.yml.bak
 # Ad-hoc test sandboxes (never tracked — may contain local DBs)
 /.sandbox/
 .aider*
+
+# check:install-upgrade work trees (~12 GB, disposable)
+/.install-upgrade/

docs/reference/ENVIRONMENT.md

@@ -103,6 +103,7 @@ OmniRoute uses **SQLite** (via `better-sqlite3`) for all persistence. These vari
 | `OMNIROUTE_MIGRATIONS_DIR`             | _(auto-detect)_      | `src/lib/db/migrationRunner.ts`                       | Override the directory that the migration runner scans. Useful when shipping bundled migrations in custom builds.                                                                                                                               |
 | `OMNIROUTE_EXTRA_MIGRATIONS_DIRS`      | _(unset)_            | `src/lib/db/migrationRunner/extraDirs.ts`             | Additional migration directories as `namespace=dir` entries separated by the platform path delimiter (e.g. `ee=/opt/app/enterprise/db/migrations`). Files found there are recorded as `<namespace>-<number>`, so a distribution shipping its own migrations never collides with the upstream numeric slots. A malformed entry, an invalid namespace or a missing directory throws at startup instead of silently skipping the schema. |
 | `OMNIROUTE_MAX_PENDING_MIGRATIONS`     | `50`                 | `src/lib/db/migrationRunner.ts`                       | Mass-pending-migrations safety threshold (#3416). Startup aborts if more than this many migrations are pending on an existing DB (guards against a wiped tracking table). Raise it to restore an older backup; set to `0` to disable the check. |
+| `OMNIROUTE_INSTALL_UPGRADE_WORKDIR`     | _(`<repo>/.install-upgrade`)_ | `scripts/check/check-install-upgrade.mjs`             | Working directory for the `check:install-upgrade` release gate. It needs roughly 12 GB (two ~3 GB install trees plus the tarball), so it must not run on a small tmpfs — on the self-hosted runner `/tmp` is a 12 GB RAM-backed tmpfs and the gate exhausted it, truncating the package. |
 | `OMNIROUTE_SPEND_FLUSH_INTERVAL_MS`    | _(default in code)_  | `src/lib/spend/batchWriter.ts`                        | Flush interval (ms) for the batched spend/cost writer. Lower values reduce write coalescing; higher values reduce DB contention.                                                                                                                |
 | `OMNIROUTE_SPEND_MAX_BUFFER_SIZE`      | _(default in code)_  | `src/lib/spend/batchWriter.ts`                        | Max buffered spend entries before a forced flush. Raise on high-QPS deployments; lower when bounded memory matters more.                                                                                                                        |
 | `OMNIROUTE_PROXY_FETCH_DEBUG`          | _(unset)_            | `open-sse/utils/proxyFetch.ts`                        | Set to `"true"` to emit `[ProxyFetch]` debug logs on the Vercel relay path. Off by default to avoid leaking routing hints.                                                                                                                      |

scripts/check/check-install-upgrade.mjs

@@ -302,7 +302,7 @@ export function assertNoDiskExhaustion(output, label) {
   throw new Error(
     `${label}: the install ran out of disk space (${count} ENOSPC error(s) from npm). ` +
       `The package tree is truncated, so anything measured from it — boot, schema, ` +
-      `migrations — is meaningless. Free space in ${os.tmpdir()} (each install tree is ` +
+      `migrations — is meaningless. Free space in ${workDirForMessages} (each install tree is ` +
       `~3 GB) and re-run. This is an environment failure, NOT a schema divergence.`
   );
 }
@@ -318,6 +318,11 @@ function freeBytes(dir) {
 
 const GB = 1024 ** 3;
 
+// Set once the work directory exists, so the ENOSPC message names the filesystem that
+// actually ran out — pointing at /tmp when the gate works elsewhere sends the reader to
+// free space on the wrong volume (which is what happened during the v3.8.50 publish).
+let workDirForMessages = os.tmpdir();
+
 function resolvePreviousVersion(current, explicit) {
   if (explicit) return explicit;
   const out = execFileSync("npm", ["view", "omniroute", "dist-tags.latest"], { encoding: "utf8" });
@@ -348,11 +353,25 @@ async function main() {
   }
   const version = JSON.parse(fs.readFileSync(path.join(ROOT, "package.json"), "utf8")).version;
   const allowlist = loadAllowlist(ROOT);
-  const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-install-upgrade-"));
+  // NOT os.tmpdir(): on the self-hosted runner /tmp is a 12 GB tmpfs backed by RAM, while
+  // the root filesystem has ~66 GB free. This gate needs ~12 GB, so it exhausted the tmpfs
+  // and npm truncated the package — 58269 ENOSPC errors on the v3.8.50 publish, which the
+  // previous code could only report as a crash. Freeing disk did not help because the disk
+  // was never the constraint. Work on real disk beside the repo instead.
+  const workRoot =
+    process.env.OMNIROUTE_INSTALL_UPGRADE_WORKDIR || path.join(ROOT, ".install-upgrade");
+  fs.mkdirSync(workRoot, { recursive: true });
+  const tmp = fs.mkdtempSync(path.join(workRoot, "omniroute-install-upgrade-"));
+  workDirForMessages = tmp;
   const failures = [];
   const warnings = [];
 
   try {
+    // Timed, because this turned out to be the expensive part: on the 2026-08-27
+    // v3.8.50 publish `npm pack` alone took 24m37s, leaving 5 of the step's 30-minute
+    // budget for two installs and two boots. Without a duration here the log showed
+    // only "packing…" then a timeout, which reads like a hang and is not.
+    const packStarted = Date.now();
     log(`packing v${version}…`);
     const packOut = execFileSync("npm", ["pack", "--json", "--pack-destination", tmp], {
       cwd: ROOT,
@@ -360,6 +379,8 @@ async function main() {
       maxBuffer: 128 * 1024 * 1024,
     });
     const tarball = path.join(tmp, pickTarball(packOut));
+    const packMb = (fs.statSync(tarball).size / 1024 / 1024).toFixed(1);
+    log(`packed in ${Math.round((Date.now() - packStarted) / 1000)}s (${packMb} MB)`);
 
     // Each install tree is ~3 GB and this run builds two of them, side by side, plus the
     // ~275 MB tarball. On the v3.8.50 publish run that overflowed the runner disk mid-way
@@ -367,7 +388,7 @@ async function main() {
     // exited 0, and every later measurement was taken from a broken tree.
     const availableBytes = freeBytes(tmp);
     if (availableBytes !== null) {
-      log(`free space in ${os.tmpdir()}: ${(availableBytes / GB).toFixed(1)} GB`);
+      log(`free space in ${tmp}: ${(availableBytes / GB).toFixed(1)} GB`);
       if (availableBytes < 12 * GB) {
         warn(
           `only ${(availableBytes / GB).toFixed(1)} GB free — this gate needs roughly 12 GB ` +