mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-08-07 07:42:13 +03:00
perf: lazy provider init, P2C quota cache, structuredClone elimination, getSettings→getCachedSettings (batch 2) (#7893)
* perf: startup parallelization, stream TextEncoder lift, auth middleware bottlenecks
Startup (~100-300ms faster cold start):
- Parallelize 4 early imports via Promise.all() in registerNodejs()
- Parallelize 10 independent background services via Promise.allSettled()
- Each service has independent try/catch — no failure domino effect
Streaming pipeline (8 fewer TextEncoder GC allocations per SSE event):
- Lift new TextEncoder() from per-chunk inside buildClaudeStreamingResponse
to function scope alongside existing decoder singleton
Auth middleware bottlenecks (from PerfBottleneckAnalysis):
- Backoff decay loop: replace updateProviderConnection (full CRUD:
SELECT+encrypt+cache-invalidate+backup) with resetConnectionBackoff
(targeted UPDATE of backoff/error columns only)
- Dual .filter() for quota: replace two passes calling
isQuotaExhaustedForRequest per connection with a single for loop
partitioning into withQuota/exhaustedQuota
- Debug-log filter recomputation: capture connectionFilterStatus Map
during the filter pass; debug loop reads 6 string comparisons instead
of 6 function calls per connection
Supporting:
- Add resetConnectionBackoff to src/lib/db/providers.ts (patterned after
clearConnectionErrorIfUnchanged, no CAS check)
- Re-export resetConnectionBackoff from src/lib/localDb.ts
- Update integration-wiring.test.ts regex for parallelized dynamic import
* perf: P2C quota cache, lazy provider init, structuredClone elimination, getSettings→getCachedSettings
- **auth.ts: P2C quota re-evaluation cache** — quotaResults Map threaded
through selectPoolSubset → compareP2CConnections → getP2CConnectionScore.
Populated during filter + partition passes, eliminating redundant
evaluateQuotaLimitPolicy / isQuotaExhaustedForRequest calls when the
P2C comparator re-evaluates previously-scored connections.
- **constants.ts: lazy PROVIDERS via Proxy** — replaces eager
generateLegacyProviders() + loadProviderCredentials() at module load
with Proxy delegating to deferred init on first property access.
- **providerModels.ts: lazy PROVIDER_MODELS + PROVIDER_ID_TO_ALIAS** —
same Proxy pattern for both exports; generateModels()/generateAliasMap()
deferred until first read.
- **stream.ts: structuredClone → minimal object spread** — replaces
O(n) deep clone of SSE response chunks with targeted reconstruction
of only mutated fields (usage, delta.content, finish_reason).
- **progressTracker.ts: TextDecoder lift** — module-level decoder
instead of per-chunk new TextDecoder().
- **Route files: getSettings() → getCachedSettings()** — 13 API route
files converted from uncached per-request DB reads to TTL-cached
wrapper (5s default), eliminating redundant queries on every request.
- **settings.ts: re-export getCachedSettings** from readCache for
non-localDb consumers.
- **Remove settingsCache.ts** — dead file, no imports reference it.
TS compile: 0 errors. Auth tests: 225/225 pass. Services: 269/269 pass.
* perf: Phase 1 tangible wins — egressCache eviction, mmap_size PRAGMA, composite indexes, proxyFallback lazy import
- egressCache: lazy TTL eviction on getCachedEgressIp access (bounds memory
leak to distinct proxy URLs, typically <100)
- mmap_size: apply stored PRAGMA from key_value table (256MiB default) after
applyStoredDatabaseOptimizationSettings — setting was stored but never applied
- schemaColumns: add idx_uh_provider_model_timestamp (covers getModelLatencyStats)
and idx_pc_provider_auth_type (covers 6+ provider_connections queries)
- proxyFallback: convert static import to dynamic import() inside error handler
(defers 210ms module load from startup to first proxy-retry scenario)
* perf: add dedup expression index, unref() sweep timers
- Add COALESCE expression index idx_uh_dedup on usage_history
matching the exact dedup query pattern. Eliminates FULL TABLE
SCAN on every saveRequestUsage insert.
- Add composite idx_uh_provider_model_timestamp on usage_history.
- Add composite idx_pc_provider_auth_type on provider_connections.
- Add .unref() to setInterval in batchProcessor.ts (polling loop).
- Add .unref() to setInterval in runtimeHeartbeat.ts (heartbeat).
* perf: bump SQLite cache_size default from 16MB to 64MB
New installs now start with 64MB page cache (was 16MB). Existing
users' stored settings are unchanged. Reduces disk reads for the
typical ~250MB database by keeping ~25% of pages in memory.
Also resolved pre-existing merge conflict in webhooks.ts.
* docs: add Redis production config guide and proxy port clash investigation report
- docs/redis-production-config.md: comprehensive Redis tuning guide
covering client options, server config, Docker settings, scaling,
and monitoring for all three Redis workloads (rate limiting,
auth cache, quota store)
- docs/proxy-port-clash-report.md: investigation confirming proxy
subsystem has no port binding issues; real EADDRINUSE history
traced to process supervisor crash-loop restart race (#4425) and
live-dashboard port clash (#6324), both already fixed
* fix: address PR #7893 review — add Proxy traps, extract migrations to reduce providers.ts size
- Add set trap to PROVIDER_ID_TO_ALIAS Proxy (providerModels.ts)
- Add deleteProperty traps to all three lazy Proxies (PROVIDERS,
PROVIDER_MODELS, PROVIDER_ID_TO_ALIAS)
- Extract autoMigrateLegacyEncryptedConnections and getGheCopilotHosts
from providers.ts (1129→1036 lines, -93) into providers/migrations.ts
- Both functions re-exported via providers.ts for backward compat
File-size ratchet resolved: src/lib/db/providers.ts now 1036 lines.
* fix: resolve merge conflict markers in 3 route/test files
- model-combo-mappings/route.ts: kept upstream version (Zod pagination
via validateBody + isValidationFailure), restored missing return
statement for GET handler
- playground/presets/route.ts: kept stashed version details (satisfies
type-narrowing + inlined Response) — functionally identical
- error-sanitization.test.ts: matches upstream exactly (no diff)
Test verification: same 7 pre-existing failures confirmed on upstream
baseline (c1bdd91e7). Zero regressions from conflict resolution.
Closes remaining uncommitted work from PR #7046 rebase.
* test(db): add resetConnectionBackoff coverage + fix file-size ratchet regression
- Add tests/unit/reset-connection-backoff.test.ts: covers the new
resetConnectionBackoff lightweight-UPDATE helper (clears backoff/error
columns and re-activates a connection, unconditional-write behavior on
terminal statuses, no-op for empty/unknown ids). Zero prior coverage
per pre-merge review of PR #7893 (Hard Rule #8).
- open-sse/services/batchProcessor.ts: fold the new unref() call into the
existing setInterval(...).unref() chain instead of a separate statement,
keeping the file at the frozen 915-line ratchet (Timeout.unref() already
returns `this`, so no type cast is needed).
- src/lib/localDb.ts: drop one blank separator line so the new
resetConnectionBackoff re-export stays within the frozen 808-line ratchet.
Pre-merge fix for PR #7893 (perf/startup-stream-auth-optimizations) per
/green-prs plan-file _tasks/pipeline/prs/1-analyzed/7893-*.plan.md.
Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: oyi77 <oyi77@users.noreply.github.com>
Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com>
Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouzapw@users.noreply.github.com>
This commit is contained in:
Paijo
@@ -240,14 +240,11 @@ export async function registerNodejs(): Promise<void> {
|
||||
await ensureDbReadyForBoot();
|
||||
|
||||
await ensureSecrets();
|
||||
const { enforceWebRuntimeEnv } = await import("@/lib/env/runtimeEnv");
|
||||
enforceWebRuntimeEnv();
|
||||
|
||||
// Trigger request-log layout migration during startup, before any request hits usageDb.
|
||||
await import("@/lib/usage/migrations");
|
||||
|
||||
const { initConsoleInterceptor } = await import("@/lib/consoleInterceptor");
|
||||
initConsoleInterceptor();
|
||||
await Promise.all([
|
||||
import("@/lib/env/runtimeEnv").then(({ enforceWebRuntimeEnv }) => enforceWebRuntimeEnv()),
|
||||
import("@/lib/usage/migrations"),
|
||||
import("@/lib/consoleInterceptor").then(({ initConsoleInterceptor }) => initConsoleInterceptor()),
|
||||
]);
|
||||
|
||||
// Clear stale transient connection cooldowns persisted from an unclean crash.
|
||||
// A crash mid-burst can leave far-future `rate_limited_until` values in the DB
|
||||
@@ -456,118 +453,90 @@ export async function registerNodejs(): Promise<void> {
|
||||
void warmModelCatalogCache();
|
||||
|
||||
if (!isBackgroundServicesDisabled()) {
|
||||
try {
|
||||
const { bootstrapEmbeddedServices } = await import("@/lib/services/bootstrap");
|
||||
await bootstrapEmbeddedServices();
|
||||
console.log("[STARTUP] Embedded services bootstrap complete");
|
||||
} catch (err: unknown) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
console.warn("[STARTUP] Embedded services bootstrap failed (non-fatal):", msg);
|
||||
}
|
||||
// All services are independent — run in parallel for faster cold start.
|
||||
await Promise.allSettled([
|
||||
import("@/lib/services/bootstrap").then(async (m) => {
|
||||
await m.bootstrapEmbeddedServices();
|
||||
console.log("[STARTUP] Embedded services bootstrap complete");
|
||||
}).catch((err: unknown) => {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
console.warn("[STARTUP] Embedded services bootstrap failed (non-fatal):", msg);
|
||||
}),
|
||||
|
||||
try {
|
||||
const { initEmbedWsProxy } = await import("@/lib/services/embedWsProxy");
|
||||
initEmbedWsProxy();
|
||||
} catch (err: unknown) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
console.warn("[STARTUP] Embed WS proxy failed to start (non-fatal):", msg);
|
||||
}
|
||||
import("@/lib/services/embedWsProxy").then((m) => m.initEmbedWsProxy())
|
||||
.catch((err: unknown) => {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
console.warn("[STARTUP] Embed WS proxy failed to start (non-fatal):", msg);
|
||||
}),
|
||||
|
||||
try {
|
||||
const { autoRefreshDaemon } = await import("@omniroute/open-sse/services/autoRefreshDaemon");
|
||||
autoRefreshDaemon.start();
|
||||
} catch (err: unknown) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
console.warn("[STARTUP] Auto-refresh daemon failed to start (non-fatal):", msg);
|
||||
}
|
||||
import("@omniroute/open-sse/services/autoRefreshDaemon").then((m) => m.autoRefreshDaemon.start())
|
||||
.catch((err: unknown) => {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
console.warn("[STARTUP] Auto-refresh daemon failed to start (non-fatal):", msg);
|
||||
}),
|
||||
|
||||
// Proactive connection-cooldown recovery (#8): re-validate connections whose
|
||||
// transient `rate_limited_until` window has elapsed OUTSIDE the request hot
|
||||
// path, so the first request after a cooldown does not pay the probe latency.
|
||||
// Lazy/self-recovery still happens in getProviderCredentials; this front-runs it.
|
||||
try {
|
||||
const { initConnectionRecoveryScheduler } = await import("@/lib/quota/connectionRecovery");
|
||||
initConnectionRecoveryScheduler();
|
||||
} catch (err: unknown) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
console.warn("[STARTUP] Connection recovery scheduler failed to start (non-fatal):", msg);
|
||||
}
|
||||
// Proactive connection-cooldown recovery (#8): re-validate connections whose
|
||||
// transient `rate_limited_until` window has elapsed OUTSIDE the request hot path,
|
||||
// so the first request after a cooldown does not pay the probe latency.
|
||||
import("@/lib/quota/connectionRecovery").then((m) => m.initConnectionRecoveryScheduler())
|
||||
.catch((err: unknown) => {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
console.warn("[STARTUP] Connection recovery scheduler failed to start (non-fatal):", msg);
|
||||
}),
|
||||
|
||||
try {
|
||||
// Arena ELO sync: model intelligence from the Arena AI leaderboard, powering the
|
||||
// Free Provider Rankings page. On by default; configurable from Dashboard Feature Flags.
|
||||
// Non-blocking — the initial sync is fire-and-forget and never fatal.
|
||||
const { initArenaEloSync } = await import("@/lib/arenaEloSync");
|
||||
const started = await initArenaEloSync();
|
||||
if (started) {
|
||||
console.log("[STARTUP] Arena ELO sync initialized");
|
||||
}
|
||||
} catch (err: unknown) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
console.warn("[STARTUP] Arena ELO sync failed to start (non-fatal):", msg);
|
||||
}
|
||||
// Free Provider Rankings page. On by default; non-blocking, never fatal.
|
||||
import("@/lib/arenaEloSync").then(async (m) => {
|
||||
const started = await m.initArenaEloSync();
|
||||
if (started) console.log("[STARTUP] Arena ELO sync initialized");
|
||||
}).catch((err: unknown) => {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
console.warn("[STARTUP] Arena ELO sync failed to start (non-fatal):", msg);
|
||||
}),
|
||||
|
||||
// Pricing sync: opt-in external pricing data (self-gated by PRICING_SYNC_ENABLED inside
|
||||
// initPricingSync). Was only wired into the unused server-init.ts, so it never ran in the
|
||||
// standalone runtime even when enabled. Non-blocking, never fatal.
|
||||
try {
|
||||
const { initPricingSync } = await import("@/lib/pricingSync");
|
||||
await initPricingSync();
|
||||
} catch (err: unknown) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
console.warn("[STARTUP] Pricing sync failed to start (non-fatal):", msg);
|
||||
}
|
||||
// Pricing sync: opt-in external pricing data (self-gated by PRICING_SYNC_ENABLED inside
|
||||
// initPricingSync). Non-blocking, never fatal.
|
||||
import("@/lib/pricingSync").then((m) => m.initPricingSync())
|
||||
.catch((err: unknown) => {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
console.warn("[STARTUP] Pricing sync failed to start (non-fatal):", msg);
|
||||
}),
|
||||
|
||||
// models.dev capability sync: opt-in via Settings > AI (self-gated by
|
||||
// settings.modelsDevSyncEnabled inside initModelsDevSync). Previously had no caller at all,
|
||||
// so the toggle was inert. Non-blocking, never fatal.
|
||||
try {
|
||||
const { initModelsDevSync } = await import("@/lib/modelsDevSync");
|
||||
await initModelsDevSync();
|
||||
} catch (err: unknown) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
console.warn("[STARTUP] models.dev sync failed to start (non-fatal):", msg);
|
||||
}
|
||||
// models.dev capability sync: opt-in via Settings > AI (self-gated by
|
||||
// settings.modelsDevSyncEnabled inside initModelsDevSync). Non-blocking, never fatal.
|
||||
import("@/lib/modelsDevSync").then((m) => m.initModelsDevSync())
|
||||
.catch((err: unknown) => {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
console.warn("[STARTUP] models.dev sync failed to start (non-fatal):", msg);
|
||||
}),
|
||||
|
||||
// Context-window self-correction (5004): periodically reconcile provider-declared
|
||||
// windows (from /models discovery) into auto:discovery overrides. Reuses already-synced
|
||||
// data (no new fetch); disable via CONTEXT_WINDOW_RECONCILE_INTERVAL=0. Never fatal.
|
||||
try {
|
||||
const { startContextWindowReconcile } = await import("@/lib/contextWindowResolver");
|
||||
startContextWindowReconcile();
|
||||
} catch (err: unknown) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
console.warn("[STARTUP] context-window reconcile failed to start (non-fatal):", msg);
|
||||
}
|
||||
// Context-window self-correction (5004): periodically reconcile provider-declared
|
||||
// windows (from /models discovery) into auto:discovery overrides. Never fatal.
|
||||
import("@/lib/contextWindowResolver").then((m) => m.startContextWindowReconcile())
|
||||
.catch((err: unknown) => {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
console.warn("[STARTUP] context-window reconcile failed to start (non-fatal):", msg);
|
||||
}),
|
||||
|
||||
// TV6 typed memory decay: optional periodic sweep of decayed episodic memories. Doubly
|
||||
// opt-in (no-op unless MEMORY_TYPED_DECAY_ENABLED=true AND
|
||||
// MEMORY_TYPED_DECAY_SWEEP_INTERVAL>0). Never deletes by default. Never fatal.
|
||||
try {
|
||||
const { startMemoryDecaySweep } = await import("@/lib/memory/typedDecay");
|
||||
startMemoryDecaySweep();
|
||||
} catch (err: unknown) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
console.warn("[STARTUP] memory decay sweep failed to start (non-fatal):", msg);
|
||||
}
|
||||
// TV6 typed memory decay: optional periodic sweep of decayed episodic memories.
|
||||
// Doubly opt-in (no-op unless MEMORY_TYPED_DECAY_ENABLED=true AND
|
||||
// MEMORY_TYPED_DECAY_SWEEP_INTERVAL>0). Never deletes by default. Never fatal.
|
||||
import("@/lib/memory/typedDecay").then((m) => m.startMemoryDecaySweep())
|
||||
.catch((err: unknown) => {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
console.warn("[STARTUP] memory decay sweep failed to start (non-fatal):", msg);
|
||||
}),
|
||||
|
||||
// Real-time dashboard WebSocket daemon (port 20132): powers Combo Studio Live,
|
||||
// the Home live-pulse, and Live Compression. liveServer.ts auto-starts the
|
||||
// daemon on import (gated by OMNIROUTE_ENABLE_LIVE_WS, default ON) — but NOTHING
|
||||
// imported it in the packaged standalone/PM2 runtime. Only the unused
|
||||
// `server-init.ts` and a dev-only helper script (`scripts/start-ws-server.mjs`)
|
||||
// ever pulled it into a module graph, so in the published `omniroute` bin the
|
||||
// daemon never bound its port and every live dashboard reported "Live disabled —
|
||||
// WebSocket disconnected". Importing it here (the instrumentation hook that DOES
|
||||
// run in standalone) fires that flag-gated auto-start. Side-effect import + the
|
||||
// module's own `.catch` keep it non-fatal.
|
||||
try {
|
||||
await import("@/server/ws/liveServer");
|
||||
console.log("[STARTUP] Live dashboard WebSocket daemon bootstrap invoked");
|
||||
} catch (err: unknown) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
console.warn("[STARTUP] Live dashboard WebSocket daemon failed to start (non-fatal):", msg);
|
||||
}
|
||||
// Real-time dashboard WebSocket daemon (port 20132): powers Combo Studio Live,
|
||||
// the Home live-pulse, and Live Compression. Side-effect import triggers the
|
||||
// flag-gated auto-start (OMNIROUTE_ENABLE_LIVE_WS, default ON).
|
||||
import("@/server/ws/liveServer").then(() => {
|
||||
console.log("[STARTUP] Live dashboard WebSocket daemon bootstrap invoked");
|
||||
}).catch((err: unknown) => {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
console.warn("[STARTUP] Live dashboard WebSocket daemon failed to start (non-fatal):", msg);
|
||||
}),
|
||||
]);
|
||||
}
|
||||
|
||||
markServerReady();
|
||||
|
||||
Reference in New Issue
Block a user