mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-09-19 05:12:16 +03:00
bdf218387b
* fix(resilience): per-model 402 on a passthrough gateway no longer terminalizes the whole connection 402 variant of #3027. Passthrough/gateway providers that multiplex many models behind one credential (kilo-gateway, ollama-cloud, etc.) can 402 on a single PAID model while free models on the same key remain perfectly usable. Previously any 402 unconditionally set the connection to a terminal `credits_exhausted` status, which is never auto-recovered without an operator reset — taking out every remaining model on that provider, amplified further inside combo routing (measured: one 402 removed 9 of 14 fallback targets in a real combo, dropping success rate from 98.3% to 74.2% on a fixed load test per the issue report). Root cause (matches the issue's own analysis): 1. resolveTerminalConnectionStatus() returned "credits_exhausted" for ANY status === 402, with no per-model/passthrough check. 2. The generic per-model lockout gate (404/429/>=500) excluded 402. 3. The #3027 403-branch is gated on `!terminalStatus` — since (1) already resolves a terminal status for any 402 before that branch runs, simply adding 402 to its condition alone would not have fired. Fix: - resolveTerminalConnectionStatus() now takes isPerModelQuotaProvider and skips the connection-wide terminal path for a bare `status === 402` when true, letting it fall through to the per-model lockout branch instead. An explicit result.creditsExhausted (a provider's own classification, independent of HTTP status) is untouched and remains unconditionally terminal. - Extended the existing #3027 per-model lockout branch to also handle 402 (reason "credits" vs "forbidden" for 403), reusing the same cooldown/lockout machinery and log format. - Single-credential (non-passthrough) providers are unaffected: isPerModelQuotaProvider is false there, so a 402 still terminalizes the connection as before — that behavior is deliberate for prepaid API keys (#5239 / #10616). Also checked the issue's 4th root cause (terminal statuses never auto-recovering) against the current codebase: connectionRecovery.ts already has a 30-minute credits_exhausted reprobe (isCreditsExhaustedReprobeCandidate) that the issue's report — filed against v3.8.49 — didn't account for. The other two files it names (rateLimit.ts's clearStaleCrashCooldowns, tokenHealthCheck.ts's OAuth-refresh skip) legitimately exclude credits_exhausted for unrelated reasons and are not bugs. Moot regardless: this fix prevents credits_exhausted from being set at all for the passthrough case, so no recovery wait is needed in the first place. Tests: tests/unit/auth-passthrough-per-model-402-12242.test.ts, modeled on the existing #3027 precedent test (real DB-backed integration test via auth.markAccountUnavailable). Covers: paid-model-only lockout with free model unaffected, a subsequent free-model request succeeding after a sibling paid-model 402, single-credential 402 still fully terminal, and no connection-wide backoff escalation on repeated 402s. Verified: - node --import tsx/esm --test tests/unit/auth-passthrough-per-model-402-12242.test.ts: 4/4 pass - All related pre-existing tests (auth-ollama-cloud-per-model-403-3027, auth-terminal-status, openrouter-free-model-credits-exhausted, vertex-passthrough-model-lockout, 10347-embed-402-cooldown): 27/27 pass, no regressions - npm run typecheck:core: 0 errors - npm run check:cycles: no cycles - eslint (auth.ts + new test file, with project suppressions): 0 errors Fixes #12242 * chore(quality): register 402 per-model test in stryker tap and de-ratchet auth.ts - stryker.conf.json: add tests/unit/auth-passthrough-per-model-402-12242.test.ts to tap.testFiles in its alphabetical slot - auth.ts: extract the #12242 connection-wide 402 decision into the pure helper isConnectionWideCreditsExhausted() so resolveTerminalConnectionStatus stays within the cyclomatic ratchet (file back to the base's 11 violations) --------- Co-authored-by: OmniRoute Dev <dev@local> Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>