fix: drop Bottleneck queue on 429 instead of waiting for reservoir refresh

When a provider returns 429 (rate limit exceeded), the rate limit manager
was setting reservoir=0 and waiting for reservoirRefreshInterval before
releasing queued requests. For providers with long rate limit windows
(e.g. Codex with hours-long resets), this caused all queued requests to
hang indefinitely — they never timed out or returned an error.

This prevented upstream callers (e.g. LiteLLM) from triggering fallback
to alternative providers, effectively making the entire model unavailable
until the rate limit window expired.

Fix: on 429, call limiter.stop({ dropWaitingJobs: true }) to immediately
fail all queued requests, then delete the limiter from the Map so
getLimiter() creates a fresh instance for subsequent requests.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Oleg Saprykin
2026-03-20 18:07:56 +03:00
parent b62e6c5a69
commit 64f040bddd

View File

@@ -339,14 +339,19 @@ export function updateFromHeaders(provider, connectionId, headers, status, model
// Handle 429 — rate limited
if (status === 429) {
const retryAfterMs = parseResetTime(retryAfterStr) || 60000; // Default 60s
const counts = limiter.counts();
const limiterKey = `${provider}:${connectionId}`;
console.log(
`🚫 [RATE-LIMIT] ${provider}:${connectionId.slice(0, 8)} — 429 received, pausing for ${Math.ceil(retryAfterMs / 1000)}s`
`🚫 [RATE-LIMIT] ${provider}:${connectionId.slice(0, 8)} — 429 received, pausing for ${Math.ceil(retryAfterMs / 1000)}s, dropping ${counts.QUEUED} queued request(s)`
);
limiter.updateSettings({
reservoir: 0,
reservoirRefreshAmount: limit || 60,
reservoirRefreshInterval: retryAfterMs,
// Stop the limiter and drop all waiting jobs so they fail immediately
// instead of hanging in the queue until reservoir refreshes (which can
// be hours for providers like Codex with long rate limit windows).
// This lets upstream callers (e.g. LiteLLM) trigger fallback to other providers.
// After stop, delete from Map so getLimiter() creates a fresh instance.
limiter.stop({ dropWaitingJobs: true }).then(() => {
limiters.delete(limiterKey);
});
return;
}