pjandro/OmniRoute
1
0
Fork 0
mirror of https://github.com/diegosouzapw/OmniRoute.git synced 2026-08-18 21:22:28 +03:00
Code Issues Packages Projects Releases Wiki Activity

perf(api): memoize getConnectionsForProvider in catalog builder

Browse Source 
Combining this PR's own bulk hidden-model optimization with the
already-merged isExcludedByProviderConnections() check (from a
different PR) reintroduced an O(connections) scan per model inside
the catalog builder's hot loop, regressing the exact single-stretch
event-loop budget tests/unit/9147-catalog-eventloop-yield.test.ts
enforces (was passing on this PR's own commit before the merge).

Memoizing getConnectionsForProvider() by its (unordered) key-set
substantially reduces the redundant per-model connection scans
(measured ~497ms -> ~210-300ms worst single stretch across repeated
runs), but does NOT fully close the gap to the 150ms budget — still
red. Committing this as a real, safe improvement; flagging for
further investigation (likely getConnectionsForProvider's first-call
cost per provider, or hasEligibleConnectionForModel) before this PR
merges. NOT deciding to relax the test threshold myself.
This commit is contained in:
adevwithpurpose
2026-08-18 12:27:20 -03:00
parent 92cd0d2ff9
commit 2e2301a5e4
1 changed files with 10 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
src/app/api/v1/models/catalog.ts
View File

@@ -389,7 +389,16 @@ async function buildUnifiedModelsResponseCore(
if ("alias" in p && typeof p.alias === "string") activeAliases.add(p.alias);
}
// #9147 follow-up: this is called ~1-3x per model at catalog scale (providerSupportsModel,
// isExcludedByProviderConnections). Connections do not change mid-build, so memoize per
// unique (unordered) key-set instead of rescanning connectionsByProvider on every call —
// otherwise the O(models) hot loop regains an O(connections) cost per model and blows the
// single-stretch event-loop budget this file's own yield mechanism is meant to protect.
const connectionsForProviderCache = new Map<string, typeof connections>();
const getConnectionsForProvider = (...keys: Array<string | null | undefined>) => {
const cacheKey = keys.filter((k): k is string => Boolean(k)).sort().join("");
const cached = connectionsForProviderCache.get(cacheKey);
if (cached) return cached;
const seen = new Set<string>();
const collected: typeof connections = [];
for (const key of keys) {
@@ -400,6 +409,7 @@ async function buildUnifiedModelsResponseCore(
collected.push(connection);
}
}
connectionsForProviderCache.set(cacheKey, collected);
return collected;
};