Files
OmniRoute/scripts/dev/healthcheck.mjs
Ravi Tharuma 4c7b902257 fix(ops): Docker HEALTHCHECK probes /healthz not deep monitoring (#10307)
* fix(deps): bump nanoid, dompurify for 2 new Dependabot alerts (#189, #190)

Bumps: nanoid ^3.3.17 (was transitive, now overridden), dompurify ^3.4.13
(with monaco-editor scoped override). Closes Dependabot #189, #190.

Remaining #182-#188 (js-yaml + mermaid) already closed by #9651 merge —
awaiting Dependabot re-scan.

npm audit → 0 vulnerabilities.

* fix(repo): harden .gitignore to also ignore a _tasks symlink (/_tasks)

_tasks is a SEPARATE nested git repo (gitignored). The pattern _tasks/ (trailing
slash) ignores only a directory, not a SYMLINK named _tasks. A self-referential
_tasks symlink can slip in via git add -A and, once pulled, checkout materializes
it over the real _tasks repo (destroying plans/specs/hands-off). Anchored /_tasks
ignores the symlink too, preventing re-capture.

* Hide health-check excluded models from /v1/models catalog (#10026)

Mirror the request-time exclusion rule (provider_specific_data.excludedModels)
in the unified catalog builder: a model is hidden when its provider has
connections but none of them is eligible for it. Applied across the
PROVIDER_MODELS, synced, custom, alias-backed, and managed-fallback loops
so ghost models no longer appear as available.

Co-authored-by: ritheshcn25 <ritheshcn25@users.noreply.github.com>

* fix(models): memoize getModelsDevPricing (event loop / healthz) (#10055)

* fix(models): memoize getModelsDevPricing for /v1/models catalog

resolveCatalogPricing called getModelsDevPricing once per model while
building GET /v1/models. Each call re-scanned models_dev_pricing and
JSON.parsed every row (~10k SQL scans + multi-GB parse work), pegging
the event loop so even /healthz timed out (#9685, #10052).

Memoize the parsed map until saveModelsDevPricing / clearModelsDevPricing
and add a unit test for invalidation.

Signed-off-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>

* fix(db): invalidate modelsDevPricing cache on DB reset (#10055)

Copilot review fixes:
1. Register invalidateModelsDevPricingCache() with DB state reset system
   so resetDbInstance() clears the process-local memo, preventing stale
   pricing data from surviving across DB reset/restore operations.
2. Add test assertion verifying DB reset bypasses the memo (Copilot #10055).

The process-local memo at modelsDevSync.ts:204 caches getModelsDevPricing()
results until saveModelsDevPricing()/clearModelsDevPricing() to avoid
re-scanning all pricing rows on every /v1/models request. Without this hook,
backup restore and test DB resets would serve stale cached data from the
previous connection.

Tests: npm run test:unit:serial -- tests/unit/modelsDevSync-extended.test.ts

---------

Signed-off-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>
Co-authored-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>

* fix(ops): Docker HEALTHCHECK probes /healthz not deep monitoring

/api/monitoring/health does a SQLite ping and more. When the event loop
is busy the official image HEALTHCHECK (5s timeout) marks the container
Unhealthy and orchestrators restart the only replica mid-session.

* fix(ops): keep healthcheck PR scoped to the /healthz probe

Drop the stray catalog ghost-model exclusion that leaked into this branch
from main (already covered upstream). Restore catalog.ts to the release
version so the PR contains only the Docker HEALTHCHECK /healthz fix, its
tests, and the changelog entry.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Signed-off-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>
Co-authored-by: diegosouzapw <diegosouzapw@users.noreply.github.com>
Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@outlook.com>
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
Co-authored-by: ritheshcn25 <rithesh.chandran@snb.ca>
Co-authored-by: ritheshcn25 <ritheshcn25@users.noreply.github.com>
Co-authored-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: adevwithpurpose <adevwithpurpose@users.noreply.github.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
2026-08-16 00:42:36 -03:00

139 lines
5.0 KiB
JavaScript

#!/usr/bin/env node
/**
* Docker healthcheck script for OmniRoute.
* Probes the lightweight /healthz endpoint on the dashboard port.
* /api/monitoring/health is the deep human/dashboard check (SQLite ping);
* using it as Docker HEALTHCHECK marks the container Unhealthy whenever the
* event loop is busy (#10052) and can restart the only replica mid-session.
* Used by Dockerfile and docker-compose files.
*
* #3151 — in some Docker network setups the server binds to a container IP and
* a probe against `127.0.0.1` is not reachable, while `localhost`/`::1` (or vice
* versa) is. The previous version probed ONLY `127.0.0.1` and swallowed every
* error, so the container was reported `unhealthy` with an empty, undiagnosable
* `State.Health[].Output`. We now try an ordered list of hosts and surface the
* last error on total failure.
*
* Bridge Network Fix: Also probes the container's internal bridge IP (e.g., 172.17.0.2)
* to handle Docker network setups that isolate loopback interfaces.
*/
import { pathToFileURL } from "node:url";
import { networkInterfaces } from "node:os";
const DEFAULT_HOSTS = ["127.0.0.1", "localhost", "::1"];
const DEFAULT_TIMEOUT_MS = 4000;
const DEFAULT_HEALTH_PATH = "/healthz";
function normalizeBasePath(value) {
const trimmed = typeof value === "string" ? value.trim() : "";
if (!trimmed || trimmed === "/") return "";
if (!trimmed.startsWith("/") || /[?#\\]/.test(trimmed)) return "";
const segments = trimmed.split("/").filter(Boolean);
if (segments.some((segment) => segment === "." || segment === "..")) return "";
return `/${segments.join("/")}`;
}
/** Prefixes the health route with the configured Next.js basePath. */
export function resolveHealthPath(basePathValue) {
const basePath = normalizeBasePath(basePathValue);
return basePath ? `${basePath}${DEFAULT_HEALTH_PATH}` : DEFAULT_HEALTH_PATH;
}
/**
* Get the primary non-loopback IPv4 address (container internal IP).
* Falls back to null if unable to determine.
*/
function getContainerInternalIP() {
try {
const interfaces = networkInterfaces();
for (const [name, addrs] of Object.entries(interfaces)) {
// Skip loopback and docker0, prioritize eth0/veth interfaces
if (name.startsWith("lo") || name === "docker0") continue;
const ipv4 = addrs?.find((a) => a.family === "IPv4" && !a.internal);
if (ipv4) return ipv4.address;
}
} catch {
// silently ignore if unable to read interfaces
}
return null;
}
/**
* Build the health URL for a host, bracketing IPv6 literals (e.g. `::1`).
* @param {string} host
* @param {string|number} port
* @param {string} healthPath path to probe, including any basePath prefix
*/
function healthUrl(host, port, healthPath = DEFAULT_HEALTH_PATH) {
const hostPart = host.includes(":") ? `[${host}]` : host;
return `http://${hostPart}:${port}${healthPath}`;
}
/**
* Probe the health endpoint across an ordered list of hosts. Resolves with the
* first host that returns a 2xx response; rejects with the last error if every
* host fails. Each attempt is bounded by a per-host timeout so one unreachable
* host cannot hang the whole probe.
*
* @param {object} opts
* @param {string|number} opts.port
* @param {string[]} [opts.hosts]
* @param {typeof fetch} [opts.fetchImpl]
* @param {number} [opts.timeoutMs]
* @param {string} [opts.healthPath]
* @returns {Promise<string>} the host that succeeded
*/
export async function probeHealth({
port,
hosts = DEFAULT_HOSTS,
fetchImpl = fetch,
timeoutMs = DEFAULT_TIMEOUT_MS,
healthPath = DEFAULT_HEALTH_PATH,
} = {}) {
let lastError = new Error("no hosts to probe");
for (const host of hosts) {
try {
const res = await fetchImpl(healthUrl(host, port, healthPath), {
signal: AbortSignal.timeout(timeoutMs),
});
if (res.ok) return host;
lastError = new Error(`${host}: HTTP ${res.status}`);
} catch (err) {
lastError = new Error(`${host}: ${err instanceof Error ? err.message : String(err)}`);
}
}
throw lastError;
}
async function main() {
const port = process.env.DASHBOARD_PORT || process.env.PORT || "20128";
// Build host list: defaults + detected container bridge IP
const hosts = [...DEFAULT_HOSTS];
const containerIP = getContainerInternalIP();
if (containerIP && !hosts.includes(containerIP)) {
hosts.push(containerIP);
}
try {
const healthPath = resolveHealthPath(process.env.OMNIROUTE_BASE_PATH);
await probeHealth({ port, hosts, healthPath });
process.exit(0);
} catch (err) {
// Surface the failure so `docker inspect ... .State.Health[].Output` is
// diagnostic instead of empty (#3151).
process.stderr.write(`healthcheck failed: ${err instanceof Error ? err.message : err}\n`);
process.exit(1);
}
}
// Only auto-run when invoked as the entrypoint (so importing the helper in
// tests does not trigger a real probe + process.exit).
const isEntrypoint =
Boolean(process.argv[1]) && import.meta.url === pathToFileURL(process.argv[1]).href;
if (isEntrypoint) {
main();
}