mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-08-19 05:32:19 +03:00
perf: lazy provider init, P2C quota cache, structuredClone elimination, getSettings→getCachedSettings (batch 2) (#7893)
* perf: startup parallelization, stream TextEncoder lift, auth middleware bottlenecks
Startup (~100-300ms faster cold start):
- Parallelize 4 early imports via Promise.all() in registerNodejs()
- Parallelize 10 independent background services via Promise.allSettled()
- Each service has independent try/catch — no failure domino effect
Streaming pipeline (8 fewer TextEncoder GC allocations per SSE event):
- Lift new TextEncoder() from per-chunk inside buildClaudeStreamingResponse
to function scope alongside existing decoder singleton
Auth middleware bottlenecks (from PerfBottleneckAnalysis):
- Backoff decay loop: replace updateProviderConnection (full CRUD:
SELECT+encrypt+cache-invalidate+backup) with resetConnectionBackoff
(targeted UPDATE of backoff/error columns only)
- Dual .filter() for quota: replace two passes calling
isQuotaExhaustedForRequest per connection with a single for loop
partitioning into withQuota/exhaustedQuota
- Debug-log filter recomputation: capture connectionFilterStatus Map
during the filter pass; debug loop reads 6 string comparisons instead
of 6 function calls per connection
Supporting:
- Add resetConnectionBackoff to src/lib/db/providers.ts (patterned after
clearConnectionErrorIfUnchanged, no CAS check)
- Re-export resetConnectionBackoff from src/lib/localDb.ts
- Update integration-wiring.test.ts regex for parallelized dynamic import
* perf: P2C quota cache, lazy provider init, structuredClone elimination, getSettings→getCachedSettings
- **auth.ts: P2C quota re-evaluation cache** — quotaResults Map threaded
through selectPoolSubset → compareP2CConnections → getP2CConnectionScore.
Populated during filter + partition passes, eliminating redundant
evaluateQuotaLimitPolicy / isQuotaExhaustedForRequest calls when the
P2C comparator re-evaluates previously-scored connections.
- **constants.ts: lazy PROVIDERS via Proxy** — replaces eager
generateLegacyProviders() + loadProviderCredentials() at module load
with Proxy delegating to deferred init on first property access.
- **providerModels.ts: lazy PROVIDER_MODELS + PROVIDER_ID_TO_ALIAS** —
same Proxy pattern for both exports; generateModels()/generateAliasMap()
deferred until first read.
- **stream.ts: structuredClone → minimal object spread** — replaces
O(n) deep clone of SSE response chunks with targeted reconstruction
of only mutated fields (usage, delta.content, finish_reason).
- **progressTracker.ts: TextDecoder lift** — module-level decoder
instead of per-chunk new TextDecoder().
- **Route files: getSettings() → getCachedSettings()** — 13 API route
files converted from uncached per-request DB reads to TTL-cached
wrapper (5s default), eliminating redundant queries on every request.
- **settings.ts: re-export getCachedSettings** from readCache for
non-localDb consumers.
- **Remove settingsCache.ts** — dead file, no imports reference it.
TS compile: 0 errors. Auth tests: 225/225 pass. Services: 269/269 pass.
* perf: Phase 1 tangible wins — egressCache eviction, mmap_size PRAGMA, composite indexes, proxyFallback lazy import
- egressCache: lazy TTL eviction on getCachedEgressIp access (bounds memory
leak to distinct proxy URLs, typically <100)
- mmap_size: apply stored PRAGMA from key_value table (256MiB default) after
applyStoredDatabaseOptimizationSettings — setting was stored but never applied
- schemaColumns: add idx_uh_provider_model_timestamp (covers getModelLatencyStats)
and idx_pc_provider_auth_type (covers 6+ provider_connections queries)
- proxyFallback: convert static import to dynamic import() inside error handler
(defers 210ms module load from startup to first proxy-retry scenario)
* perf: add dedup expression index, unref() sweep timers
- Add COALESCE expression index idx_uh_dedup on usage_history
matching the exact dedup query pattern. Eliminates FULL TABLE
SCAN on every saveRequestUsage insert.
- Add composite idx_uh_provider_model_timestamp on usage_history.
- Add composite idx_pc_provider_auth_type on provider_connections.
- Add .unref() to setInterval in batchProcessor.ts (polling loop).
- Add .unref() to setInterval in runtimeHeartbeat.ts (heartbeat).
* perf: bump SQLite cache_size default from 16MB to 64MB
New installs now start with 64MB page cache (was 16MB). Existing
users' stored settings are unchanged. Reduces disk reads for the
typical ~250MB database by keeping ~25% of pages in memory.
Also resolved pre-existing merge conflict in webhooks.ts.
* docs: add Redis production config guide and proxy port clash investigation report
- docs/redis-production-config.md: comprehensive Redis tuning guide
covering client options, server config, Docker settings, scaling,
and monitoring for all three Redis workloads (rate limiting,
auth cache, quota store)
- docs/proxy-port-clash-report.md: investigation confirming proxy
subsystem has no port binding issues; real EADDRINUSE history
traced to process supervisor crash-loop restart race (#4425) and
live-dashboard port clash (#6324), both already fixed
* fix: address PR #7893 review — add Proxy traps, extract migrations to reduce providers.ts size
- Add set trap to PROVIDER_ID_TO_ALIAS Proxy (providerModels.ts)
- Add deleteProperty traps to all three lazy Proxies (PROVIDERS,
PROVIDER_MODELS, PROVIDER_ID_TO_ALIAS)
- Extract autoMigrateLegacyEncryptedConnections and getGheCopilotHosts
from providers.ts (1129→1036 lines, -93) into providers/migrations.ts
- Both functions re-exported via providers.ts for backward compat
File-size ratchet resolved: src/lib/db/providers.ts now 1036 lines.
* fix: resolve merge conflict markers in 3 route/test files
- model-combo-mappings/route.ts: kept upstream version (Zod pagination
via validateBody + isValidationFailure), restored missing return
statement for GET handler
- playground/presets/route.ts: kept stashed version details (satisfies
type-narrowing + inlined Response) — functionally identical
- error-sanitization.test.ts: matches upstream exactly (no diff)
Test verification: same 7 pre-existing failures confirmed on upstream
baseline (c1bdd91e7). Zero regressions from conflict resolution.
Closes remaining uncommitted work from PR #7046 rebase.
* test(db): add resetConnectionBackoff coverage + fix file-size ratchet regression
- Add tests/unit/reset-connection-backoff.test.ts: covers the new
resetConnectionBackoff lightweight-UPDATE helper (clears backoff/error
columns and re-activates a connection, unconditional-write behavior on
terminal statuses, no-op for empty/unknown ids). Zero prior coverage
per pre-merge review of PR #7893 (Hard Rule #8).
- open-sse/services/batchProcessor.ts: fold the new unref() call into the
existing setInterval(...).unref() chain instead of a separate statement,
keeping the file at the frozen 915-line ratchet (Timeout.unref() already
returns `this`, so no type cast is needed).
- src/lib/localDb.ts: drop one blank separator line so the new
resetConnectionBackoff re-export stays within the frozen 808-line ratchet.
Pre-merge fix for PR #7893 (perf/startup-stream-auth-optimizations) per
/green-prs plan-file _tasks/pipeline/prs/1-analyzed/7893-*.plan.md.
Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: oyi77 <oyi77@users.noreply.github.com>
Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com>
Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouzapw@users.noreply.github.com>
This commit is contained in:
Paijo
@@ -6,10 +6,11 @@ import {
|
||||
getCachedProviderNodes,
|
||||
validateApiKey,
|
||||
updateProviderConnection,
|
||||
touchConnectionLastUsed,
|
||||
clearConnectionErrorIfUnchanged,
|
||||
resetConnectionBackoff,
|
||||
getSettings,
|
||||
getCachedSettings,
|
||||
touchConnectionLastUsed,
|
||||
clearConnectionErrorIfUnchanged,
|
||||
} from "@/lib/localDb";
|
||||
import {
|
||||
createLazyConnectionView,
|
||||
@@ -594,15 +595,22 @@ function getConnectionRecencyPenalty(connection: ProviderConnectionView): number
|
||||
function getP2CConnectionScore(
|
||||
provider: string,
|
||||
connection: ProviderConnectionView,
|
||||
requestedModel: string | null = null
|
||||
requestedModel: string | null = null,
|
||||
quotaResults?: Map<string, { blocked: boolean; exhausted: boolean }>
|
||||
): { score: number; quotaHeadroomPercent: number | null } {
|
||||
const quotaBlocked = evaluateQuotaLimitPolicy(provider, connection, requestedModel).blocked;
|
||||
const quotaExhausted = isQuotaExhaustedForRequest(connection.id, provider, requestedModel);
|
||||
const quotaHeadroomPercent = getConnectionQuotaHeadroomPercent(
|
||||
provider,
|
||||
connection,
|
||||
requestedModel
|
||||
);
|
||||
let quotaBlocked: boolean;
|
||||
let quotaExhausted: boolean;
|
||||
|
||||
if (connection.id && quotaResults?.has(connection.id)) {
|
||||
const cached = quotaResults.get(connection.id)!;
|
||||
quotaBlocked = cached.blocked;
|
||||
quotaExhausted = cached.exhausted;
|
||||
} else {
|
||||
quotaBlocked = evaluateQuotaLimitPolicy(provider, connection, requestedModel).blocked;
|
||||
quotaExhausted = isQuotaExhaustedForRequest(connection.id, provider, requestedModel);
|
||||
}
|
||||
|
||||
const quotaHeadroomPercent = getConnectionQuotaHeadroomPercent(provider, connection, requestedModel);
|
||||
|
||||
let quotaPenalty = 0;
|
||||
if (quotaHeadroomPercent !== null) {
|
||||
@@ -630,10 +638,11 @@ function compareP2CConnections(
|
||||
provider: string,
|
||||
a: ProviderConnectionView,
|
||||
b: ProviderConnectionView,
|
||||
requestedModel: string | null = null
|
||||
requestedModel: string | null = null,
|
||||
quotaResults?: Map<string, { blocked: boolean; exhausted: boolean }>
|
||||
): number {
|
||||
const aScore = getP2CConnectionScore(provider, a, requestedModel);
|
||||
const bScore = getP2CConnectionScore(provider, b, requestedModel);
|
||||
const aScore = getP2CConnectionScore(provider, a, requestedModel, quotaResults);
|
||||
const bScore = getP2CConnectionScore(provider, b, requestedModel, quotaResults);
|
||||
if (aScore.score !== bScore.score) {
|
||||
return aScore.score - bScore.score;
|
||||
}
|
||||
@@ -1146,32 +1155,39 @@ export async function getProviderCredentials(
|
||||
!isAccountUnavailable(c.rateLimitedUntil)
|
||||
) {
|
||||
c.backoffLevel = 0;
|
||||
updateProviderConnection(c.id, {
|
||||
backoffLevel: 0,
|
||||
testStatus: "active",
|
||||
lastError: null,
|
||||
lastErrorAt: null,
|
||||
lastErrorType: null,
|
||||
lastErrorSource: null,
|
||||
errorCode: null,
|
||||
}).catch(() => {});
|
||||
resetConnectionBackoff(c.id).catch(() => {});
|
||||
}
|
||||
}
|
||||
|
||||
let modelLockedCount = 0;
|
||||
let familyLockedCount = 0;
|
||||
const connectionFilterStatus = new Map<string, string>();
|
||||
// Filter out unavailable accounts and excluded connection
|
||||
const availableConnections = connections.filter((c) => {
|
||||
if (excludedConnectionIds.has(c.id)) return false;
|
||||
if (excludedConnectionIds.has(c.id)) {
|
||||
connectionFilterStatus.set(c.id, "excluded");
|
||||
return false;
|
||||
}
|
||||
if (requestedModel && isModelExcludedByConnection(requestedModel, c.providerSpecificData)) {
|
||||
connectionFilterStatus.set(c.id, "modelExcluded");
|
||||
return false;
|
||||
}
|
||||
if (!allowSuppressedConnections) {
|
||||
if (!allowRateLimitedConnections && isAccountUnavailable(c.rateLimitedUntil)) return false;
|
||||
if (isTerminalConnectionStatus(c)) return false;
|
||||
if (provider === "codex" && isCodexScopeUnavailable(c, requestedModel)) return false;
|
||||
if (!allowRateLimitedConnections && isAccountUnavailable(c.rateLimitedUntil)) {
|
||||
connectionFilterStatus.set(c.id, "rateLimited");
|
||||
return false;
|
||||
}
|
||||
if (isTerminalConnectionStatus(c)) {
|
||||
connectionFilterStatus.set(c.id, "terminalStatus");
|
||||
return false;
|
||||
}
|
||||
if (provider === "codex" && isCodexScopeUnavailable(c, requestedModel)) {
|
||||
connectionFilterStatus.set(c.id, "codexScopeLimited");
|
||||
return false;
|
||||
}
|
||||
// Per-model lockout: if this specific model/family is locked on this connection, skip it
|
||||
if (requestedModel && isModelLocked(provider, c.id, requestedModel)) {
|
||||
connectionFilterStatus.set(c.id, "modelLocked");
|
||||
if (
|
||||
provider === "antigravity" &&
|
||||
getQuotaScopeLabelForProvider(provider, requestedModel) === "family"
|
||||
@@ -1183,6 +1199,7 @@ export async function getProviderCredentials(
|
||||
return false;
|
||||
}
|
||||
}
|
||||
connectionFilterStatus.set(c.id, "available");
|
||||
return true;
|
||||
});
|
||||
|
||||
@@ -1197,15 +1214,13 @@ export async function getProviderCredentials(
|
||||
);
|
||||
}
|
||||
connections.forEach((c) => {
|
||||
const excluded = excludedConnectionIds.has(c.id);
|
||||
const rateLimited = isAccountUnavailable(c.rateLimitedUntil);
|
||||
const terminalStatus = isTerminalConnectionStatus(c);
|
||||
const codexScopeLimited = provider === "codex" && isCodexScopeUnavailable(c, requestedModel);
|
||||
const modelLocked =
|
||||
Boolean(requestedModel) && isModelLocked(provider, c.id, requestedModel as string);
|
||||
const modelExcluded =
|
||||
Boolean(requestedModel) &&
|
||||
isModelExcludedByConnection(requestedModel as string, c.providerSpecificData);
|
||||
const status = connectionFilterStatus.get(c.id);
|
||||
const excluded = status === "excluded";
|
||||
const rateLimited = status === "rateLimited";
|
||||
const terminalStatus = status === "terminalStatus";
|
||||
const codexScopeLimited = status === "codexScopeLimited";
|
||||
const modelLocked = status === "modelLocked";
|
||||
const modelExcluded = status === "modelExcluded";
|
||||
if (excluded || rateLimited) {
|
||||
log.debug(
|
||||
"AUTH",
|
||||
@@ -1334,10 +1349,12 @@ export async function getProviderCredentials(
|
||||
reasons: string[];
|
||||
resetAt: string | null;
|
||||
}> = [];
|
||||
const quotaResults = new Map<string, { blocked: boolean; exhausted: boolean }>();
|
||||
|
||||
if (!bypassQuotaPolicy) {
|
||||
policyEligibleConnections = availableConnections.filter((connection) => {
|
||||
const evaluation = evaluateQuotaLimitPolicy(provider, connection, requestedModel);
|
||||
quotaResults.set(connection.id, { blocked: evaluation.blocked, exhausted: false });
|
||||
if (!evaluation.blocked) return true;
|
||||
|
||||
blockedByPolicy.push({
|
||||
@@ -1377,13 +1394,19 @@ export async function getProviderCredentials(
|
||||
};
|
||||
}
|
||||
|
||||
// Quota-aware: filter out accounts with exhausted quota for the requested scope.
|
||||
const withQuota = policyEligibleConnections.filter(
|
||||
(c) => !isQuotaExhaustedForRequest(c.id, provider, requestedModel)
|
||||
);
|
||||
const exhaustedQuota = policyEligibleConnections.filter((c) =>
|
||||
isQuotaExhaustedForRequest(c.id, provider, requestedModel)
|
||||
);
|
||||
// Quota-aware: partition accounts with and without quota for the requested scope.
|
||||
const withQuota: typeof policyEligibleConnections = [];
|
||||
const exhaustedQuota: typeof policyEligibleConnections = [];
|
||||
for (const c of policyEligibleConnections) {
|
||||
const exhausted = isQuotaExhaustedForRequest(c.id, provider, requestedModel);
|
||||
const existing = quotaResults.get(c.id);
|
||||
if (existing) existing.exhausted = exhausted;
|
||||
if (!exhausted) {
|
||||
withQuota.push(c);
|
||||
} else {
|
||||
exhaustedQuota.push(c);
|
||||
}
|
||||
}
|
||||
|
||||
if (exhaustedQuota.length > 0) {
|
||||
log.info(
|
||||
@@ -1551,7 +1574,7 @@ export async function getProviderCredentials(
|
||||
// health instead of defaulting to random-first selection.
|
||||
if (candidatePool.length <= 2) {
|
||||
connection = [...candidatePool].sort((a, b) =>
|
||||
compareP2CConnections(provider, a, b, requestedModel)
|
||||
compareP2CConnections(provider, a, b, requestedModel, quotaResults)
|
||||
)[0];
|
||||
} else {
|
||||
const i =
|
||||
@@ -1561,7 +1584,7 @@ export async function getProviderCredentials(
|
||||
if (j >= i) j++;
|
||||
const a = candidatePool[i];
|
||||
const b = candidatePool[j];
|
||||
connection = compareP2CConnections(provider, a, b, requestedModel) <= 0 ? a : b;
|
||||
connection = compareP2CConnections(provider, a, b, requestedModel, quotaResults) <= 0 ? a : b;
|
||||
}
|
||||
} else if (strategy === "random") {
|
||||
// Random: Fisher-Yates-inspired random pick
|
||||
|
||||
Reference in New Issue
Block a user