feat(health): background health check for local provider_nodes

Local inference backends (oMLX, Ollama, LM Studio) configured as
provider_nodes have no health monitoring. When a local provider is
down, OmniRoute waits the full timeout before failing.

This adds a background health check that polls local provider_nodes:
- GET /models with 5s timeout for each local node (localhost only)
- In-memory health cache (no DB migration needed)
- Promise.allSettled for parallel checks (one slow node doesn't block)
- Exponential backoff on failures: 30s → 60s → 120s → 300s max
- Reset to 30s on first success after failure
- State transition logging (healthy ↔ unhealthy)
- Expose health status via GET /api/monitoring/health (localProviders)
- Auto-init on first import (same pattern as tokenHealthCheck)
- 401 treated as healthy (server up, auth required)
- isNodeHealthy() returns true if never checked (optimistic default)
This commit is contained in:
Regis
2026-03-16 22:26:37 +01:00
parent 9b255e643a
commit 0aede2ef63
2 changed files with 220 additions and 0 deletions

View File

@@ -18,6 +18,7 @@ export async function GET() {
const circuitBreakers = getAllCircuitBreakerStatuses();
const rateLimitStatus = getAllRateLimitStatus();
const lockouts = getAllModelLockouts();
const { getAllHealthStatuses } = await import("@/lib/localHealthCheck");
// System info
const system = {
@@ -46,6 +47,7 @@ export async function GET() {
timestamp: new Date().toISOString(),
system,
providerHealth,
localProviders: getAllHealthStatuses(),
rateLimitStatus,
lockouts,
setupComplete: settings?.setupComplete || false,