mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-07-31 12:22:14 +03:00
0ecc380928
Phase 1 of client-side quota tracking for NVIDIA NIM (no rate-limit headers, no usage API): - Register nvidia in PROVIDER_DEFAULT_RATE_LIMITS (40 RPM sliding window, matching the documented free-tier note), operator-overridable via a new ResilienceSettings.providerQuotaOverrides map. - Per-connection concurrency cap (default 6) via a new nvidiaConcurrencyGate leaf module wrapping rateLimitSemaphore, wired into DefaultExecutor.execute(). - Per-model 429 lockout: confirmed already satisfied by #6773's passthroughModels flag on the nvidia registry entry (no new code needed) — added as a regression-guard test instead. Phase 2 (AIMD adaptive ceiling learning) and Phase 3 (dashboard quota card + combo-routing headroom preference) are explicitly deferred to follow-up issues, per the plan's own scope note.
40 lines
1.9 KiB
TypeScript
40 lines
1.9 KiB
TypeScript
/**
|
|
* nvidiaConcurrencyGate — per-connection concurrency cap for the `nvidia`
|
|
* provider (#6846 Phase 1).
|
|
*
|
|
* NVIDIA NIM's free tier sends no rate-limit headers and has no usage API
|
|
* (see docs/reference/PROVIDER_REFERENCE.md `nvidia` note), so most 429 storms
|
|
* come from parallel bursts against one connection rather than sustained total
|
|
* volume. This reuses the generic `rateLimitSemaphore` FIFO queue (already used
|
|
* by combo round-robin/quota-share dispatch) to serialize concurrent requests
|
|
* per `nvidia:<connectionId>` — excess requests wait in the queue instead of
|
|
* firing doomed parallel calls.
|
|
*
|
|
* No-op for every other provider and for a missing `connectionId` — returns
|
|
* `null`, and callers should skip the `finally` release when the return value
|
|
* is `null`.
|
|
*/
|
|
import * as semaphore from "../../services/rateLimitSemaphore.ts";
|
|
import { getProviderConcurrencyCap } from "../../services/providerDefaultRateLimit.ts";
|
|
|
|
const NVIDIA_DEFAULT_CONCURRENCY_CAP = 6;
|
|
const NVIDIA_ACQUIRE_TIMEOUT_MS = 30_000;
|
|
|
|
/**
|
|
* Acquire a concurrency slot for an nvidia request. Resolves to a release
|
|
* function that MUST be called (typically in a `finally`) once the request
|
|
* completes, or `null` when the gate does not apply (non-nvidia provider, or
|
|
* no connection id to scope the gate to). Rejects with a
|
|
* `SEMAPHORE_TIMEOUT`-coded error if the queue wait exceeds the timeout —
|
|
* propagates like any other executor failure (caller/combo fallback handles it).
|
|
*/
|
|
export function acquireNvidiaConcurrencySlot(
|
|
provider: string,
|
|
connectionId: string | null | undefined
|
|
): Promise<(() => void) | null> {
|
|
if (provider !== "nvidia" || !connectionId) return Promise.resolve(null);
|
|
const maxConcurrency = getProviderConcurrencyCap(provider, NVIDIA_DEFAULT_CONCURRENCY_CAP);
|
|
const key = `${provider}:${connectionId}`;
|
|
return semaphore.acquire(key, { maxConcurrency, timeoutMs: NVIDIA_ACQUIRE_TIMEOUT_MS });
|
|
}
|