fix(sse): Anthropic OAuth 403 "Request not allowed" is a per-request refusal — cooldown with backoff instead of an instant ban (#12859) (#12864)

* fix(sse): Anthropic OAuth 403 "Request not allowed" is a per-request refusal, not a ban

A single upstream 403 on the `claude` OAuth connection was classified
FORBIDDEN and written as the terminal `banned` connection state
(chatCore -> writeTerminalStatus). From then on every request to that
provider was short-circuited with "All 1 connection(s) banned by
upstream - please reconnect in the dashboard" without touching Anthropic,
until an operator reconnected.

Anthropic's OAuth surface answers a small fraction of otherwise-valid
requests with 403 {"type":"permission_error","message":"Request not
allowed"}. On the reporting install the same token returned 200 forty
seconds before the 403 and again right after the connection was
re-enabled; a revoked or expired token is a 401 authentication_error, not
this. It is a refusal of one request, not of the credential.

Classify it as the new non-terminal PROVIDER_ERROR_TYPES.REQUEST_REJECTED
(scoped to provider `claude` and the "Request not allowed" body) and list
that type in authTerminalStatus.isNonTerminalProviderError, mirroring the
Cloudflare FINGERPRINT_REJECTION precedent. The combo layer still falls
through to the next target for the failing request; the connection stays
active for the next one. Any other claude 403 keeps its previous
classification.

Tests: error-classifier.test.ts covers the Anthropic body, the
gateway-flattened "[403]: Request not allowed" message, the same body from
a non-Anthropic provider (still FORBIDDEN), other claude 403s (unchanged),
and the helpers; anthropic-request-not-allowed-not-a-ban.test.ts pins
resolveTerminalConnectionStatus() -> null for the new type even with a
`permanent` fallback verdict, and `banned` for a generic claude 403.

* fix(sse): cooldown with backoff and streak escalation for REQUEST_REJECTED (#12859)

Not "ignore the 403" either: if Anthropic ever made "Request not allowed"
systematic, re-sending every request into it would be the wrong thing to
do to an OAuth account. chatCore now handles REQUEST_REJECTED explicitly:

- exclude the connection via setConnectionRateLimitUntil for a growing
  cooldown (5 -> 15 -> 45 min) so a sporadic refusal costs minutes, not a
  reconnect, and a systematic one cannot become a stream of 403s;
- escalate to the terminal `banned` state only for 3 refusals within a
  60-minute window (services/requestRejectedStreak.ts, in-memory per
  connection; a restart forgets the streak, erring towards more cooldowns
  rather than an operator-undone ban), with a last_error that says so;
- probe-origin failures record but never cool down or ban (#9817).

The existing "request not allowed" text rule (5 s) is unaffected:
markAccountUnavailable skips a connection that already has a future
rateLimitedUntil, so the minute-scale cooldown written here wins.

Tests: request-rejected-streak.test.ts pins the window/threshold/backoff
arithmetic; anthropic-request-not-allowed-cooldown-escalation.test.ts drives
the real chat route against a mocked 403 upstream on a `claude` OAuth
connection: 300 s cooldown, then 900 s, then banned on the third refusal;
a different claude 403 body still bans on the first response.

* chore(changelog): name the #12859 fragment after its PR (#12864)

* refactor(sse): move the REQUEST_REJECTED branch into a chatCore leaf; register its tests for mutation coverage

chatCore.ts is frozen at 5984 lines by the file-size ratchet; the branch
body now lives in open-sse/handlers/chatCore/requestRejectedFailure.ts
(chatCore: 5974 -> 5983). stryker.conf.json tap.testFiles gains the two new
DB-backed tests so their mutant kills count (check:mutation-test-coverage).

* fix(sse): count refusal episodes, reset on success, keep the dashboard honest (#12859 review)

Review findings on the first cut of the REQUEST_REJECTED handling:

- A burst of in-flight requests that all got the 403 within seconds
  produced streak 1, 2, 3 and a ban from one upstream event. The streak
  now counts cooldown *episodes*: a refusal that lands while the
  connection is already excluded is the same event and is not counted.
- Nothing reset the streak on a healthy response, so sporadic refusals
  on a busy install could still accumulate to a ban. chatHelpers'
  onRequestSuccess now clears it (only a real success does - the recovery
  tick's clearAccountError is an elapsed cooldown, not a success).
  Clearing the cooldown by hand in the dashboard clears it too.
- The third rung of the ladder was unreachable (the third refusal
  escalates): the ladder is now 5 -> 15 min, sourced from COOLDOWN_MS next
  to the existing 5 s "request not allowed" rule, with a note on why that
  rule is superseded for claude. The 60-min window becomes a 24 h
  staleness bound - "consecutive" is defined by successes, not by time.
- Probe-origin refusals no longer touch the streak (#9817).
- The cooldown is written like every other connection-level cooldown:
  ISO rateLimitedUntil + testStatus "unavailable" (+ lastErrorAt), so the
  dashboard shows the countdown and the recovery tick restores "active".
- One refusal is re-seeded from the persisted row after a restart so a
  crash loop cannot reset the count on every boot.

Docs: RESILIENCE_GUIDE terminal states + CODEBASE_DOCUMENTATION resilience
row mention the streak module. Tests cover the burst, the success reset,
the seed, and the ISO/unavailable shape end-to-end through the chat route.

* chore(sse): drop unrelated Prettier churn in auth.ts / providers route

* style(api): keep providers route Prettier-clean

* refactor(sse): share the "exclude connection for a cooldown" leaf between GEO_BLOCKED, GCP_PROJECT_REQUIRED and the new branch

The release tip moved chatCore.ts to its frozen 5984 lines, so the
REQUEST_REJECTED branch cannot add a single net line. The GEO_BLOCKED and
GCP_PROJECT_REQUIRED branches were the same eight statements with different
constants and log wording; both now call
open-sse/handlers/chatCore/connectionCooldown.ts::excludeConnectionForCooldown
(behaviour, probe guard and log lines preserved verbatim). chatCore.ts ends
9 lines below the base it branched from.

* chore(chatCore): tighten the cooldown comments to keep the file under its size ceiling

After merging release/v3.8.51, chatCore.ts sat at 6150 lines against a
frozen ceiling of 6146. Condense the explanatory comments this PR added
to the GEO_BLOCKED and GCP_PROJECT_REQUIRED branches; no code change.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Co-authored-by: insoln <is@careerum.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
This commit is contained in:
Innokentiy Solntsev
2026-09-17 21:26:08 +02:00
committed by GitHub
parent 47159ed56b
commit dd70dbdaa0
17 changed files with 846 additions and 30 deletions

View File

@@ -0,0 +1 @@
- **fix(sse):** An Anthropic OAuth `403 "Request not allowed"` no longer bans the Claude connection on the first response — it is a per-request refusal on an otherwise healthy token, so it is now classified as the non-terminal `request_rejected` type, the connection is excluded for a growing cooldown (5 min, then 15 min) and only three consecutive refusals with no success in between escalate to `banned`; previously a single such response flipped the only Claude connection to `banned` and every later request was short-circuited with "All 1 connection(s) banned by upstream" until an operator reconnected ([#12859](https://github.com/diegosouzapw/OmniRoute/issues/12859), [#12864](https://github.com/diegosouzapw/OmniRoute/pull/12864) — thanks @insoln)

View File

@@ -523,7 +523,7 @@ Highlights (full list under `open-sse/services/`):
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Combo routing | `combo.ts` (19 strategies), `comboConfig.ts`, `comboMetrics.ts`, `comboManifestMetrics.ts`, `comboAgentMiddleware.ts` |
| Auto Combo engine | `autoCombo/``engine.ts`, `scoring.ts`, `taskFitness.ts`, `virtualFactory.ts`, `modePacks.ts`, `autoPrefix.ts`, `persistence.ts`, `providerDiversity.ts`, `providerRegistryAccessor.ts`, `routerStrategy.ts`, `selfHealing.ts`, `index.ts` |
| Resilience | `accountFallback.ts` (cooldown + lockout), `errorClassifier.ts`, `emergencyFallback.ts`, `rateLimitManager.ts`, `rateLimitSemaphore.ts`, `accountSemaphore.ts`, `accountSelector.ts` |
| Resilience | `accountFallback.ts` (cooldown + lockout), `errorClassifier.ts`, `requestRejectedStreak.ts`, `emergencyFallback.ts`, `rateLimitManager.ts`, `rateLimitSemaphore.ts`, `accountSemaphore.ts`, `accountSelector.ts` |
| Quotas | `quotaMonitor.ts`, `quotaPreflight.ts`, `bailianQuotaFetcher.ts`, `codexQuotaFetcher.ts`, `deepseekQuotaFetcher.ts`, `openrouterQuotaFetcher.ts`, `openrouterFreeWindow.ts`, `crofUsageFetcher.ts`, `antigravityCredits.ts` |
| Caching | `reasoningCache.ts`, `searchCache.ts`, `signatureCache.ts`, `requestDedup.ts` |
| Routing intelligence | `intentClassifier.ts`, `taskAwareRouter.ts`, `backgroundTaskDetector.ts`, `volumeDetector.ts`, `wildcardRouter.ts`, `workflowFSM.ts`, `specificityDetector.ts`, `specificityRules.ts`, `specificityTypes.ts` |

View File

@@ -100,7 +100,7 @@ Regression guard: `tests/unit/provider-cooldown-window-gate.test.ts`.
**Terminal states (NOT cooldowns):**
- `banned` — set by banned-keyword / account-ban detection (see [BAN_DETECTION](../security/BAN_DETECTION.md))
- `banned` — set by banned-keyword / account-ban detection (see [BAN_DETECTION](../security/BAN_DETECTION.md)), and by three consecutive upstream per-request refusals (`request_rejected`, e.g. Anthropic OAuth 403 "Request not allowed" — `open-sse/services/requestRejectedStreak.ts`); a single refusal only cools the connection down
- `expired` (transitions to terminal after bounded retries — `EXPIRED_RETRY_MAX = 3` with exponential backoff — so transient OAuth errors can self-heal before the account is permanently deactivated)
- `credits_exhausted`

View File

@@ -74,6 +74,12 @@ export const COOLDOWN_MS = {
transientMax: 60 * 1000,
transient: TRANSIENT_COOLDOWN_MS,
requestNotAllowed: 5 * 1000,
// Anthropic OAuth 403 "Request not allowed" (#12859): a per-request refusal
// on a healthy token. chatCore excludes the connection for requestRejected
// after the first refusal, requestRejectedRepeat after the second, and bans
// it on the third consecutive one (services/requestRejectedStreak.ts).
requestRejected: 5 * 60 * 1000,
requestRejectedRepeat: 15 * 60 * 1000,
rateLimit: 2 * 60 * 1000,
serviceUnavailable: 2 * 1000,
authExpired: 2 * 60 * 1000,
@@ -99,6 +105,10 @@ export const ERROR_RULES: ErrorRule[] = [
reason: "auth_error",
},
{
// For provider `claude` this text is classified REQUEST_REJECTED and the
// connection-level cooldown is written by chatCore before the fallback
// layer runs (#12859); markAccountUnavailable then keeps the longer
// cooldown. This 5 s rule still serves every other provider.
id: "request_not_allowed",
text: "request not allowed",
cooldownMs: COOLDOWN_MS.requestNotAllowed,

View File

@@ -275,6 +275,8 @@ import { stageTrace } from "./chatCore/stageTrace.ts";
import { attachCompressionUsageReceiptAfterAnalytics as attachCompressionUsageReceiptAfterAnalyticsFor } from "./chatCore/compressionUsageReceipt.ts";
import { prepareUpstreamBody } from "./chatCore/upstreamBody.ts";
import { getQuotaScopeLabelForProvider } from "../services/antigravityQuotaFamily.ts";
import { excludeConnectionForCooldown } from "./chatCore/connectionCooldown.ts";
import { handleRequestRejectedFailure } from "./chatCore/requestRejectedFailure.ts";
import { getKimiTemporaryRateLimitResetAt } from "./chatCore/kimiQuotaRecovery.ts";
import {
getCallLogPipelineCaptureStreamChunks,
@@ -4049,35 +4051,37 @@ export async function handleChatCore({
`[provider] Node ${errorConnectionId} project routing error (${statusCode}) -- not banning`
);
} else if (errorType === PROVIDER_ERROR_TYPES.GEO_BLOCKED) {
const geoCooldownMs = COOLDOWN_MS.geoBlocked ?? 24 * 60 * 60 * 1000;
await updateProviderConnection(errorConnectionId, {
lastErrorType: errorType,
lastError: persistentMessage,
errorCode: statusCode,
// Google regional refusal: account-independent, non-terminal; park the connection
// until egress uses a supported region; probes skip the day-long cooldown (#9817).
await excludeConnectionForCooldown({
connectionId: errorConnectionId,
errorType,
message: persistentMessage,
statusCode,
cooldownMs: COOLDOWN_MS.geoBlocked ?? 24 * 60 * 60 * 1000,
skipCooldownForProbe: true,
label: "geo-blocked",
suffix: "trying other accounts",
});
} else if (errorType === PROVIDER_ERROR_TYPES.REQUEST_REJECTED) {
// Per-request refusal (#12859): growing cooldown, streak → banned.
await handleRequestRejectedFailure({
connectionId: errorConnectionId,
statusCode,
message: persistentMessage,
});
if (!(await shouldIsolateProbeFailures())) {
try {
const { setConnectionRateLimitUntil } = await import("@/lib/db/providers");
setConnectionRateLimitUntil(errorConnectionId, Date.now() + geoCooldownMs);
} catch {}
}
console.warn(
`[provider] Node ${errorConnectionId} geo-blocked (${statusCode}) -- excluded for ${Math.ceil(geoCooldownMs / 1000)}s, trying other accounts`
);
} else if (errorType === PROVIDER_ERROR_TYPES.GCP_PROJECT_REQUIRED) {
const byopCooldownMs = COOLDOWN_MS.gcpProjectRequired ?? 24 * 60 * 60 * 1000;
await updateProviderConnection(errorConnectionId, {
lastErrorType: errorType,
lastError: persistentMessage,
errorCode: statusCode,
// Antigravity BYOP: fixable via a Project ID; never a lockout/ban. Park the connection.
await excludeConnectionForCooldown({
connectionId: errorConnectionId,
errorType,
message: persistentMessage,
statusCode,
cooldownMs: COOLDOWN_MS.gcpProjectRequired ?? 24 * 60 * 60 * 1000,
skipCooldownForProbe: false,
label: "GCP project required",
suffix: "routing to other accounts (enter a Project ID to restore)",
});
try {
const { setConnectionRateLimitUntil } = await import("@/lib/db/providers");
setConnectionRateLimitUntil(errorConnectionId, Date.now() + byopCooldownMs);
} catch {}
console.warn(
`[provider] Node ${errorConnectionId} GCP project required (${statusCode}) -- excluded for ${Math.ceil(byopCooldownMs / 1000)}s, routing to other accounts (enter a Project ID to restore)`
);
} else if (errorType === PROVIDER_ERROR_TYPES.MODEL_NOT_FOUND) {
const notFoundCooldownMs = COOLDOWN_MS.notFound;
if (!(await shouldIsolateProbeFailures())) {

View File

@@ -0,0 +1,42 @@
import { updateProviderConnection } from "@/lib/db/providers";
import { shouldIsolateProbeFailures } from "@/shared/utils/probeOrigin";
/**
* chatCore leaf for the non-terminal "exclude this connection for a while"
* outcomes (GEO_BLOCKED, GCP_PROJECT_REQUIRED): record the error on the
* connection and park it via rateLimitedUntil so selection prefers sibling
* accounts, without ever marking it banned/expired.
*/
export async function excludeConnectionForCooldown(params: {
connectionId: string;
errorType: string;
message: string;
statusCode: number;
cooldownMs: number;
/**
* T-PROBE (#9817): when true, a probe-origin failure records the error but
* does not push the connection into the cooldown (routing state untouched).
*/
skipCooldownForProbe: boolean;
/** Log wording: `[provider] Node <id> <label> (<status>) — excluded for <s>s, <suffix>` */
label: string;
suffix: string;
}): Promise<void> {
const { connectionId, errorType, message, statusCode, cooldownMs, label, suffix } = params;
await updateProviderConnection(connectionId, {
lastErrorType: errorType,
lastError: message,
errorCode: statusCode,
});
if (!(params.skipCooldownForProbe && (await shouldIsolateProbeFailures()))) {
try {
const { setConnectionRateLimitUntil } = await import("@/lib/db/providers");
setConnectionRateLimitUntil(connectionId, Date.now() + cooldownMs);
} catch {
// DB write failure must never break the fallback loop
}
}
console.warn(
`[provider] Node ${connectionId} ${label} (${statusCode}) — excluded for ${Math.ceil(cooldownMs / 1000)}s, ${suffix}`
);
}

View File

@@ -0,0 +1,102 @@
import { getProviderConnectionById, updateProviderConnection } from "@/lib/db/providers";
import { shouldIsolateProbeFailures } from "@/shared/utils/probeOrigin";
import { writeTerminalStatus } from "@/shared/utils/terminalStatus";
import { PROVIDER_ERROR_TYPES } from "../../services/errorClassifier.ts";
import {
hasRequestRejectedStreak,
recordRequestRejected,
seedRequestRejectedStreak,
} from "../../services/requestRejectedStreak.ts";
/**
* chatCore leaf for PROVIDER_ERROR_TYPES.REQUEST_REJECTED (#12859 — Anthropic
* OAuth 403 "Request not allowed").
*
* The upstream refused THIS request, not the credential: the same token serves
* the next request. One refusal must not ban the connection — but a run of
* them is enforcement, and re-sending every request into it would be wrong.
* So: exclude the connection for a short, growing cooldown and escalate to
* `banned` only for consecutive refusals (services/requestRejectedStreak).
*
* Probe-origin failures (dashboard test-all) are recorded on the connection
* but never touch the streak, cool it down or ban it (#9817).
*/
export async function handleRequestRejectedFailure(params: {
connectionId: string;
statusCode: number;
message: string;
}): Promise<void> {
const { connectionId, statusCode, message } = params;
const nowIso = new Date().toISOString();
if (await shouldIsolateProbeFailures()) {
await updateProviderConnection(connectionId, {
lastErrorType: PROVIDER_ERROR_TYPES.REQUEST_REJECTED,
lastError: message,
lastErrorAt: nowIso,
errorCode: statusCode,
});
console.warn(
`[provider] Node ${connectionId} probe refused by upstream (${statusCode}) — connection stays active`
);
return;
}
// First refusal seen by this process: pick up one that happened before a
// restart so a crash loop cannot reset the count on every boot.
if (!hasRequestRejectedStreak(connectionId)) {
try {
const row = await getProviderConnectionById(connectionId);
if (row?.lastErrorType === PROVIDER_ERROR_TYPES.REQUEST_REJECTED && row.lastErrorAt) {
seedRequestRejectedStreak(connectionId, Date.parse(String(row.lastErrorAt)));
}
} catch {
// best-effort — the in-memory streak still works without the seed
}
}
const verdict = recordRequestRejected(connectionId);
const windowH = Math.round(verdict.windowMs / 3_600_000);
if (!verdict.counted) {
console.warn(
`[provider] Node ${connectionId} request refused by upstream (${statusCode}) while already excluded — same episode, not counted (refusal ${verdict.streak}/${verdict.threshold})`
);
return;
}
if (verdict.escalate) {
await writeTerminalStatus(
connectionId,
{
testStatus: "banned",
isActive: false,
lastError: `${message} (${verdict.streak} consecutive refusals within ${windowH}h — treated as upstream enforcement)`,
lastErrorType: PROVIDER_ERROR_TYPES.FORBIDDEN,
errorCode: String(statusCode),
},
"production"
);
console.warn(
`[provider] Node ${connectionId} refused ${verdict.streak}x in a row (${statusCode}) — disabling, reconnect required`
);
return;
}
const until = new Date(Date.now() + verdict.cooldownMs).toISOString();
// Same shape as the other connection-level cooldowns (testStatus
// "unavailable" + ISO rateLimitedUntil): the dashboard shows the countdown
// and the recovery tick restores "active" once the window has elapsed.
await updateProviderConnection(connectionId, {
testStatus: "unavailable",
rateLimitedUntil: until,
lastErrorType: PROVIDER_ERROR_TYPES.REQUEST_REJECTED,
lastError: message,
lastErrorAt: nowIso,
errorCode: statusCode,
});
console.warn(
`[provider] Node ${connectionId} request refused by upstream (${statusCode}) — excluded until ${until} (refusal ${verdict.streak}/${verdict.threshold} in a row), trying other accounts`
);
}

View File

@@ -90,6 +90,13 @@ export const PROVIDER_ERROR_TYPES = {
// Google account must Bring Its Own GCP Project. Account-specific and
// fixable by entering a Project ID — never a model lockout and never a ban.
GCP_PROJECT_REQUIRED: "gcp_project_required",
// The upstream refused THIS request (policy / request shape), not the
// credential: the same connection serves the next request. Not terminal on
// its own — chatCore excludes the connection for a growing cooldown and only
// a streak of refusals escalates to `banned` (services/requestRejectedStreak).
// First case: Anthropic's OAuth 403 "Request not allowed" (#12859), which
// lands on a handful of requests between thousands of 200s on the same token.
REQUEST_REJECTED: "request_rejected",
} as const;
export type ProviderErrorType = (typeof PROVIDER_ERROR_TYPES)[keyof typeof PROVIDER_ERROR_TYPES];
@@ -218,6 +225,25 @@ export function isCloudflareFingerprintRejection(errorText: string): boolean {
);
}
/**
* Anthropic's OAuth (Claude subscription) surface answers a small fraction of
* otherwise-valid requests with `403 {"type":"permission_error","message":
* "Request not allowed"}`. Observed on one install: 200 on the same token 40 s
* earlier, 200 on the next request after the connection was re-enabled — it is
* a per-request refusal, not an account ban or a revoked token (a revoked token
* is a 401 `authentication_error`). Classifying it FORBIDDEN flipped the only
* Claude connection to the terminal `banned` state on a single response, and
* every later request was short-circuited with "All 1 connection(s) banned by
* upstream" until an operator reconnected in the dashboard.
*/
export function isAnthropicOAuthProvider(provider?: string | null): boolean {
return String(provider || "").toLowerCase() === "claude";
}
export function isAnthropicRequestNotAllowed(errorText: string): boolean {
return /\brequest not allowed\b/i.test(String(errorText || ""));
}
function responseBodyToString(responseBody: unknown): string {
if (typeof responseBody === "string") return responseBody;
if (responseBody !== null && typeof responseBody === "object") {
@@ -355,6 +381,16 @@ export function classifyProviderError(
if (statusCode === 403 && accountDeactivated) {
return PROVIDER_ERROR_TYPES.ACCOUNT_DEACTIVATED;
}
if (
statusCode === 403 &&
isAnthropicOAuthProvider(provider) &&
isAnthropicRequestNotAllowed(bodyStr)
) {
// Per-request refusal on an otherwise healthy Claude OAuth token — see
// isAnthropicRequestNotAllowed. Must be checked BEFORE the generic 403 →
// FORBIDDEN fall-through, which bans the connection permanently.
return PROVIDER_ERROR_TYPES.REQUEST_REJECTED;
}
if (statusCode === 403) {
// Cloud Code / Antigravity (Gemini Code Assist) 403s are almost always a
// RECOVERABLE project-config issue — the Cloud AI Companion API not enabled

View File

@@ -0,0 +1,140 @@
import { COOLDOWN_MS } from "../config/errorConfig.ts";
/**
* Per-connection streak of upstream per-request refusals (REQUEST_REJECTED —
* today Anthropic's OAuth `403 "Request not allowed"`, #12859).
*
* A single refusal is a hiccup: the same token serves the next request. A run
* of them is enforcement, and re-sending every request into it is the wrong
* thing to do to an OAuth account. So the connection is excluded for a short,
* growing cooldown after each refusal, and only a streak of *consecutive*
* refusals escalates to the terminal `banned` state that used to fire on the
* first one.
*
* What counts as one refusal: one cooldown *episode*. Responses that land while
* the connection is already excluded were in flight before the cooldown was
* written (parallel sessions, fan-out) — they are the same event, not new
* evidence, and are ignored. A successful response on the connection clears
* the streak (chatHelpers onRequestSuccess); the window below is only a
* staleness bound so an ancient refusal can never be the first of a streak.
*
* State is in-memory on purpose. A restart forgets the streak, which errs on
* the side of a few more cooldowns before escalation — never on the side of a
* ban the operator has to undo by hand. The caller may re-seed one refusal
* from the persisted row (`seedRequestRejectedStreak`) to bound a crash loop.
*/
/** A refusal older than this cannot start or continue a streak. */
export const REQUEST_REJECTED_WINDOW_MS = 24 * 60 * 60 * 1000;
/** Consecutive refusals at which the connection is written as `banned`. */
export const REQUEST_REJECTED_ESCALATION_THRESHOLD = 3;
/** Cooldown after the 1st and 2nd refusal of a streak; the 3rd escalates. */
export const REQUEST_REJECTED_COOLDOWNS_MS = [
COOLDOWN_MS.requestRejected,
COOLDOWN_MS.requestRejectedRepeat,
];
export interface RequestRejectedVerdict {
/** 1-based position of this refusal in the current streak. */
streak: number;
/** How long to exclude the connection from selection; 0 when escalating. */
cooldownMs: number;
/** True when the streak reached the threshold — caller writes the terminal state. */
escalate: boolean;
/**
* False when the refusal landed while the connection was already in a
* cooldown written by an earlier refusal: same event, nothing to persist.
*/
counted: boolean;
windowMs: number;
threshold: number;
}
interface StreakState {
count: number;
lastRefusalAt: number;
cooldownUntil: number;
}
const streaks = new Map<string, StreakState>();
let clock: () => number = () => Date.now();
function cooldownForStreak(streak: number): number {
const step = Math.min(streak, REQUEST_REJECTED_COOLDOWNS_MS.length) - 1;
return REQUEST_REJECTED_COOLDOWNS_MS[step];
}
export function recordRequestRejected(
connectionId: string,
now: number = clock()
): RequestRejectedVerdict {
const existing = streaks.get(connectionId);
const base = {
windowMs: REQUEST_REJECTED_WINDOW_MS,
threshold: REQUEST_REJECTED_ESCALATION_THRESHOLD,
};
if (existing && now < existing.cooldownUntil) {
// In flight before the cooldown was written — same episode.
return {
streak: existing.count,
cooldownMs: Math.max(0, existing.cooldownUntil - now),
escalate: false,
counted: false,
...base,
};
}
const continues = !!existing && now - existing.lastRefusalAt < REQUEST_REJECTED_WINDOW_MS;
const streak = continues ? existing.count + 1 : 1;
const escalate = streak >= REQUEST_REJECTED_ESCALATION_THRESHOLD;
if (escalate) {
// The terminal state takes over; a reconnect starts from a clean slate.
streaks.delete(connectionId);
return { streak, cooldownMs: 0, escalate: true, counted: true, ...base };
}
const cooldownMs = cooldownForStreak(streak);
streaks.set(connectionId, { count: streak, lastRefusalAt: now, cooldownUntil: now + cooldownMs });
return { streak, cooldownMs, escalate: false, counted: true, ...base };
}
/**
* Restore one refusal that happened before this process started (read from
* the persisted connection row), so a crash loop cannot reset the count on
* every boot. No-op when the connection already has in-memory state or the
* refusal is outside the window.
*/
export function seedRequestRejectedStreak(
connectionId: string,
lastRefusalAt: number,
now: number = clock()
): boolean {
if (streaks.has(connectionId)) return false;
if (!Number.isFinite(lastRefusalAt) || now - lastRefusalAt >= REQUEST_REJECTED_WINDOW_MS) {
return false;
}
streaks.set(connectionId, { count: 1, lastRefusalAt, cooldownUntil: lastRefusalAt });
return true;
}
export function hasRequestRejectedStreak(connectionId: string): boolean {
return streaks.has(connectionId);
}
/** Forget a connection's streak — on a successful response or an operator reset. */
export function clearRequestRejectedStreak(connectionId: string): void {
streaks.delete(connectionId);
}
/** Test-only: wipe all streaks and restore the real clock. */
export function __resetRequestRejectedStreaksForTests(): void {
streaks.clear();
clock = () => Date.now();
}
/** Test-only: replace the clock the streak reads when no `now` is passed. */
export function __setRequestRejectedClockForTests(fn: (() => number) | null): void {
clock = fn ?? (() => Date.now());
}

View File

@@ -6,6 +6,7 @@ import {
} from "@/lib/compliance/providerAudit";
import { getCachedProviderConnectionById } from "@/lib/db/readCache";
import { updateProviderConnection } from "@/lib/db/providers";
import { clearRequestRejectedStreak } from "@omniroute/open-sse/services/requestRejectedStreak.ts";
import { deleteProviderConnection } from "@/lib/db/providers/deletion";
import { isCloudEnabled } from "@/lib/db/settings";
import { getConsistentMachineId } from "@/shared/utils/machineId";
@@ -220,12 +221,15 @@ export async function PUT(request: Request, { params }: { params: Promise<{ id:
if (lastErrorSource !== undefined) updateData.lastErrorSource = lastErrorSource;
if (errorCode !== undefined) updateData.errorCode = errorCode;
if (rateLimitedUntil !== undefined) updateData.rateLimitedUntil = rateLimitedUntil;
// Clearing the cooldown by hand also forgets the refusal streak (#12859).
if (rateLimitedUntil === null || testStatus === "active") clearRequestRejectedStreak(id);
if (lastTested !== undefined) updateData.lastTested = lastTested;
// healthCheckInterval PATCH semantics: undefined = leave as-is; null = clear
// the override (connection follows the global default); 0-1440 = explicit
// per-connection minutes (0 opts this connection out of the sweep).
if (healthCheckInterval === null) updateData.healthCheckInterval = null;
else if (healthCheckInterval !== undefined) updateData.healthCheckInterval = healthCheckInterval;
else if (healthCheckInterval !== undefined)
updateData.healthCheckInterval = healthCheckInterval;
if (group !== undefined) updateData.group = group;
if (maxConcurrent !== undefined) updateData.maxConcurrent = maxConcurrent;
if (incomingWindowThresholds !== undefined) {

View File

@@ -10,6 +10,7 @@ import {
} from "../services/auth";
import { maybeReactivateAfterExplicitProbe } from "../services/explicitInactiveProbe";
import { connectionHasExtraKeys } from "@omniroute/open-sse/services/apiKeyRotator.ts";
import { clearRequestRejectedStreak } from "@omniroute/open-sse/services/requestRejectedStreak.ts";
import { createBuiltinAutoCombo } from "@omniroute/open-sse/services/autoCombo/builtinCatalog.ts";
import * as log from "../utils/logger";
import { updateProviderCredentials } from "../services/tokenRefresh";
@@ -537,6 +538,9 @@ export async function executeChatWithBreaker({
},
onRequestSuccess: async () => {
if (isShadowTraffic) return;
// A healthy response ends any run of per-request refusals
// (#12859) — only a real success does, not an elapsed cooldown.
if (credentials.connectionId) clearRequestRejectedStreak(credentials.connectionId);
await clearAccountError(credentials.connectionId, credentials);
await maybeReactivateAfterExplicitProbe({
connectionId: credentials.connectionId,

View File

@@ -53,7 +53,10 @@ function isNonTerminalProviderError(providerErrorType: string | null): boolean {
providerErrorType === PROVIDER_ERROR_TYPES.OAUTH_INVALID_TOKEN ||
// #1010: Cloudflare fingerprint rejection is the CDN refusing the CLIENT's
// signature, not the account's credentials — never a terminal account state.
providerErrorType === PROVIDER_ERROR_TYPES.FINGERPRINT_REJECTION
providerErrorType === PROVIDER_ERROR_TYPES.FINGERPRINT_REJECTION ||
// Anthropic OAuth 403 "Request not allowed" refuses ONE request; the token
// keeps serving the next one — never a terminal account state.
providerErrorType === PROVIDER_ERROR_TYPES.REQUEST_REJECTED
);
}

View File

@@ -72,6 +72,8 @@
"tests/unit/agentrouter-lock-scope-10334.test.ts",
"tests/unit/alibaba-free-tier-exhaustion.test.ts",
"tests/unit/anthropic-thinking-signature-recovery.test.ts",
"tests/unit/anthropic-request-not-allowed-cooldown-escalation.test.ts",
"tests/unit/anthropic-request-not-allowed-not-a-ban.test.ts",
"tests/unit/agy-family-not-connection-cooldown.test.ts",
"tests/unit/agy-quota-exhaustion-threshold.test.ts",
"tests/unit/antigravity-429-quota-cooldown.test.ts",

View File

@@ -0,0 +1,241 @@
/**
* #12859 end-to-end through the real chat route: an Anthropic OAuth
* `403 "Request not allowed"` must not ban the `claude` connection on the
* first response — it excludes the connection for a short, growing cooldown
* and only three consecutive refusals (no success in between) escalate to
* `banned`. A burst of requests while the connection is already excluded is
* one episode, and a healthy response resets the streak. Any other claude 403
* keeps the previous behaviour (banned on the first one).
*
* Pattern mirrors tests/unit/probe-gate-autodisable.test.ts: temp DATA_DIR,
* real SQLite, `globalThis.fetch` mocked as the upstream. The model id must
* exist in the catalog (src/shared/constants/modelSpecs.ts).
*/
import { test } from "node:test";
import assert from "node:assert/strict";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-rr-escalation-"));
process.env.DATA_DIR = TEST_DATA_DIR;
const core = await import("../../src/lib/db/core.ts");
const { createProviderConnection, updateProviderConnection } =
await import("../../src/lib/db/providers.ts");
const { buildInternalChatRequest } = await import("../../src/lib/api/modelTestRunner.ts");
const chatRouteModule = await import("../../src/app/api/v1/chat/completions/route.ts");
const postChatCompletion = chatRouteModule.POST;
const { resetAllCircuitBreakers } = await import("../../src/shared/utils/circuitBreaker.ts");
const { invalidateDbCache } = await import("../../src/lib/db/readCache.ts");
const {
__resetRequestRejectedStreaksForTests,
__setRequestRejectedClockForTests,
hasRequestRejectedStreak,
} = await import("../../open-sse/services/requestRejectedStreak.ts");
const { COOLDOWN_MS } = await import("../../open-sse/config/errorConfig.ts");
const originalFetch = globalThis.fetch;
const MODEL = "claude/claude-sonnet-5";
// The streak's own clock is advanced when a test "elapses" a cooldown, so the
// in-memory episode boundary moves together with the DB row.
let clockOffsetMs = 0;
test.beforeEach(() => {
resetAllCircuitBreakers();
__resetRequestRejectedStreaksForTests();
clockOffsetMs = 0;
__setRequestRejectedClockForTests(() => Date.now() + clockOffsetMs);
invalidateDbCache("connections");
});
test.after(() => {
globalThis.fetch = originalFetch;
core.resetDbInstance();
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });
});
type Row = Record<string, unknown> | undefined;
function readRow(connId: string): Row {
const db = core.getDbInstance() as unknown as {
prepare: (sql: string) => { get: (id: string) => Record<string, unknown> | undefined };
};
return db
.prepare(
"SELECT is_active, test_status, rate_limited_until, last_error, last_error_type, last_error_at, error_code FROM provider_connections WHERE id = ?"
)
.get(connId);
}
async function createClaudeConnection(): Promise<string> {
const conn = await createProviderConnection({
provider: "claude",
authType: "oauth",
name: "rr-escalation",
accessToken: "claude-oauth-access", // pragma: allowlist secret
refreshToken: "claude-oauth-refresh", // pragma: allowlist secret
expiresAt: new Date(Date.now() + 60 * 60 * 1000).toISOString(),
isActive: true,
testStatus: "active",
});
return String((conn as { id: string }).id);
}
let upstreamCalls = 0;
function mockAnthropic403(message: string): void {
globalThis.fetch = (async () => {
upstreamCalls += 1;
return new Response(
JSON.stringify({ type: "error", error: { type: "permission_error", message } }),
{ status: 403, headers: { "content-type": "application/json" } }
);
}) as typeof fetch;
}
function mockAnthropic200(): void {
globalThis.fetch = (async () => {
upstreamCalls += 1;
return new Response(
JSON.stringify({
id: "msg_test",
type: "message",
role: "assistant",
model: "claude-sonnet-5",
content: [{ type: "text", text: "OK" }],
stop_reason: "end_turn",
stop_sequence: null,
usage: { input_tokens: 1, output_tokens: 1 },
}),
{ status: 200, headers: { "content-type": "application/json" } }
);
}) as typeof fetch;
}
async function sendOnce(connId: string): Promise<Response> {
return postChatCompletion(
buildInternalChatRequest(
{ model: MODEL, messages: [{ role: "user", content: "hi" }], stream: false },
new AbortController().signal,
connId
)
);
}
function cooldownMsOf(row: Row): number {
const until = row?.rate_limited_until;
assert.equal(typeof until, "string", "a cooldown must be persisted");
const untilMs = Date.parse(until as string);
assert.ok(Number.isFinite(untilMs), `rate_limited_until must be ISO, got ${String(until)}`);
return untilMs - Date.now();
}
function assertCoolingDown(row: Row, expectedMs: number, label: string) {
assert.equal(row?.is_active, 1, `${label}: connection stays active`);
assert.equal(row?.test_status, "unavailable", `${label}: cooling down, not banned`);
assert.equal(row?.last_error_type, "request_rejected", `${label}: error type`);
assert.equal(Number(row?.error_code), 403, `${label}: error code`);
assert.equal(typeof row?.last_error_at, "string", `${label}: last_error_at written`);
const ms = cooldownMsOf(row);
assert.ok(
Math.abs(ms - expectedMs) < 30_000,
`${label}: cooldown ≈ ${expectedMs / 60000}min expected, got ${Math.round(ms / 1000)}s`
);
}
/** What the recovery tick does once the window has elapsed — not a success. */
async function elapseCooldown(connId: string, cooldownMs: number) {
clockOffsetMs += cooldownMs + 1000;
await updateProviderConnection(connId, { testStatus: "active", rateLimitedUntil: null });
invalidateDbCache("connections");
}
test("claude 403 'Request not allowed': 5 min, then 15 min, banned on the 3rd consecutive refusal", async () => {
const connId = await createClaudeConnection();
mockAnthropic403("Request not allowed");
const first = await sendOnce(connId);
assert.notEqual(first.status, 200);
assertCoolingDown(readRow(connId), COOLDOWN_MS.requestRejected, "1st refusal");
await elapseCooldown(connId, COOLDOWN_MS.requestRejected);
await sendOnce(connId);
assertCoolingDown(readRow(connId), COOLDOWN_MS.requestRejectedRepeat, "2nd refusal");
await elapseCooldown(connId, COOLDOWN_MS.requestRejectedRepeat);
await sendOnce(connId);
const row = readRow(connId);
assert.equal(row?.test_status, "banned", "3rd consecutive refusal: terminal");
assert.equal(row?.is_active, 0);
assert.match(String(row?.last_error), /3 consecutive refusals/);
assert.equal(hasRequestRejectedStreak(connId), false, "escalation clears the streak");
});
test("requests arriving while the connection is already excluded are one episode, not new refusals", async () => {
const connId = await createClaudeConnection();
mockAnthropic403("Request not allowed");
upstreamCalls = 0;
await sendOnce(connId);
const afterFirst = readRow(connId);
assertCoolingDown(afterFirst, COOLDOWN_MS.requestRejected, "1st refusal");
// The mocked fetch also serves auxiliary calls (identity bootstrap), so only
// the delta matters below.
const callsAfterFirst = upstreamCalls;
// Follow-ups pinned to the same connection (the internal request carries the
// connection id, so selection does not filter it) all get the 403 while the
// cooldown is running — like parallel sessions that were in flight when the
// first 403 came back. They are one episode: nothing is re-persisted and,
// above all, three of them must not ban.
for (let i = 0; i < 3; i += 1) await sendOnce(connId);
const afterBurst = readRow(connId);
assert.ok(
upstreamCalls > callsAfterFirst,
"the burst did reach the upstream (pinned connection)"
);
assert.equal(afterBurst?.test_status, "unavailable", "still just cooling down, not banned");
assert.equal(afterBurst?.is_active, 1);
assert.equal(
afterBurst?.rate_limited_until,
afterFirst?.rate_limited_until,
"cooldown unchanged by the burst"
);
assert.equal(hasRequestRejectedStreak(connId), true, "streak still open at 1");
// Once the cooldown has elapsed, the next refusal is the 2nd of the streak.
await elapseCooldown(connId, COOLDOWN_MS.requestRejected);
await sendOnce(connId);
assertCoolingDown(readRow(connId), COOLDOWN_MS.requestRejectedRepeat, "2nd episode");
});
test("a successful response resets the streak — sporadic refusals never accumulate", async () => {
const connId = await createClaudeConnection();
mockAnthropic403("Request not allowed");
await sendOnce(connId);
assertCoolingDown(readRow(connId), COOLDOWN_MS.requestRejected, "1st refusal");
await elapseCooldown(connId, COOLDOWN_MS.requestRejected);
mockAnthropic200();
const ok = await sendOnce(connId);
assert.equal(ok.status, 200, "healthy response goes through");
assert.equal(hasRequestRejectedStreak(connId), false, "success clears the streak");
const clean = readRow(connId);
assert.equal(clean?.test_status, "active");
assert.equal(clean?.rate_limited_until, null);
mockAnthropic403("Request not allowed");
await sendOnce(connId);
assertCoolingDown(readRow(connId), COOLDOWN_MS.requestRejected, "refusal after success");
});
test("any other claude 403 still bans on the first response (regression guard)", async () => {
const connId = await createClaudeConnection();
mockAnthropic403("Your organization has been disabled.");
await sendOnce(connId);
const row = readRow(connId);
assert.equal(row?.test_status, "banned");
assert.equal(row?.is_active, 0);
});

View File

@@ -0,0 +1,56 @@
/**
* Anthropic OAuth 403 "Request not allowed" must never make the connection terminal.
*
* Reproduction of the incident behind this test: a single 403 with that body on
* the only `claude` OAuth connection flipped it to `test_status = banned`
* (chatCore FORBIDDEN → writeTerminalStatus), after which every request was
* short-circuited with "All 1 connection(s) banned by upstream — please
* reconnect in the dashboard" although the same token had returned 200 forty
* seconds earlier and did so again once re-enabled.
*
* Two layers are pinned here:
* 1. classifyProviderError() yields the non-terminal REQUEST_REJECTED type.
* 2. resolveTerminalConnectionStatus() maps that type to `null` (no terminal
* state), even when the fallback analysis flags the failure `permanent`.
*/
import test from "node:test";
import assert from "node:assert/strict";
const { classifyProviderError, PROVIDER_ERROR_TYPES } =
await import("../../open-sse/services/errorClassifier.ts");
const { resolveTerminalConnectionStatus } =
await import("../../src/sse/services/authTerminalStatus.ts");
const ANTHROPIC_BODY = JSON.stringify({
type: "error",
error: { type: "permission_error", message: "Request not allowed" },
});
test("claude 403 'Request not allowed' → REQUEST_REJECTED → no terminal connection status", () => {
const errorType = classifyProviderError(403, ANTHROPIC_BODY, "claude");
assert.equal(errorType, PROVIDER_ERROR_TYPES.REQUEST_REJECTED);
assert.equal(
resolveTerminalConnectionStatus(403, { permanent: false }, errorType, "claude"),
null,
"a per-request refusal must not park the connection"
);
assert.equal(
resolveTerminalConnectionStatus(403, { permanent: true }, errorType, "claude"),
null,
"non-terminal classification wins over a `permanent` fallback verdict"
);
});
test("regression guard: a generic claude 403 still resolves to banned", () => {
const errorType = classifyProviderError(
403,
JSON.stringify({ error: { message: "you do not have permission" } }),
"claude"
);
assert.equal(errorType, PROVIDER_ERROR_TYPES.FORBIDDEN);
assert.equal(
resolveTerminalConnectionStatus(403, { permanent: false }, errorType, "claude"),
"banned"
);
});

View File

@@ -5,6 +5,8 @@ const {
classifyProviderError,
isResourceNotFoundResponse,
isCloudflareFingerprintRejection,
isAnthropicOAuthProvider,
isAnthropicRequestNotAllowed,
PROVIDER_ERROR_TYPES,
} = await import("../../open-sse/services/errorClassifier.ts");
@@ -401,3 +403,56 @@ test("classifyProviderError: 422 without the BYOP code stays unclassified (no mo
);
assert.equal(classifyProviderError(422, "some other body", "antigravity"), null);
});
// ── Anthropic OAuth 403 "Request not allowed" is a per-request refusal, not a ban ──
test("classifyProviderError: claude 403 'Request not allowed' (Anthropic body) => REQUEST_REJECTED, never FORBIDDEN", () => {
// Verbatim Anthropic shape. On the reporting install this landed once between
// hundreds of 200s on the same OAuth token and permanently banned the only
// Claude connection.
const body = JSON.stringify({
type: "error",
error: { type: "permission_error", message: "Request not allowed" },
});
const result = classifyProviderError(403, body, "claude");
assert.equal(result, PROVIDER_ERROR_TYPES.REQUEST_REJECTED);
assert.notEqual(result, PROVIDER_ERROR_TYPES.FORBIDDEN, "must not ban the connection");
});
test("classifyProviderError: claude 403 'Request not allowed' via gateway-wrapped message => REQUEST_REJECTED", () => {
// Shape as it reaches the classifier after the executor flattens the body.
const result = classifyProviderError(403, "[403]: Request not allowed", "claude");
assert.equal(result, PROVIDER_ERROR_TYPES.REQUEST_REJECTED);
});
test("classifyProviderError: 'Request not allowed' from a non-Anthropic OAuth provider keeps its 403 semantics", () => {
const body = JSON.stringify({ error: { message: "Request not allowed" } });
assert.equal(classifyProviderError(403, body, "codex"), PROVIDER_ERROR_TYPES.FORBIDDEN);
assert.equal(classifyProviderError(403, body, undefined), PROVIDER_ERROR_TYPES.FORBIDDEN);
});
test("classifyProviderError: other claude 403 bodies still classify as before", () => {
assert.equal(
classifyProviderError(403, { error: { message: "you do not have permission" } }, "claude"),
PROVIDER_ERROR_TYPES.FORBIDDEN
);
assert.equal(
classifyProviderError(
403,
JSON.stringify({ error: { message: "account_deactivated: this account has been disabled" } }),
"claude"
),
PROVIDER_ERROR_TYPES.ACCOUNT_DEACTIVATED
);
});
test("isAnthropicRequestNotAllowed / isAnthropicOAuthProvider helpers", () => {
assert.equal(isAnthropicRequestNotAllowed("Request not allowed"), true);
assert.equal(isAnthropicRequestNotAllowed('{"message":"request NOT allowed"}'), true);
assert.equal(isAnthropicRequestNotAllowed("requests not allowed here"), false);
assert.equal(isAnthropicRequestNotAllowed(""), false);
assert.equal(isAnthropicOAuthProvider("claude"), true);
assert.equal(isAnthropicOAuthProvider("Claude"), true);
assert.equal(isAnthropicOAuthProvider("anthropic"), false);
assert.equal(isAnthropicOAuthProvider(null), false);
});

View File

@@ -0,0 +1,116 @@
/**
* open-sse/services/requestRejectedStreak.ts (#12859): per-connection streak
* of upstream per-request refusals → growing cooldown, escalation only for
* consecutive refusals. One cooldown episode counts once: refusals landing
* while the connection is already excluded were in flight before the cooldown
* and are not new evidence.
*/
import test from "node:test";
import assert from "node:assert/strict";
const {
recordRequestRejected,
seedRequestRejectedStreak,
hasRequestRejectedStreak,
clearRequestRejectedStreak,
__resetRequestRejectedStreaksForTests,
REQUEST_REJECTED_WINDOW_MS,
REQUEST_REJECTED_ESCALATION_THRESHOLD,
REQUEST_REJECTED_COOLDOWNS_MS,
} = await import("../../open-sse/services/requestRejectedStreak.ts");
const { COOLDOWN_MS } = await import("../../open-sse/config/errorConfig.ts");
const MIN = 60 * 1000;
const T0 = 1_000_000;
test.beforeEach(() => __resetRequestRejectedStreaksForTests());
test("ladder comes from COOLDOWN_MS and has one rung per non-escalating refusal", () => {
assert.deepEqual(REQUEST_REJECTED_COOLDOWNS_MS, [
COOLDOWN_MS.requestRejected,
COOLDOWN_MS.requestRejectedRepeat,
]);
assert.equal(REQUEST_REJECTED_COOLDOWNS_MS.length, REQUEST_REJECTED_ESCALATION_THRESHOLD - 1);
});
test("first refusal: first cooldown rung, counted, no escalation", () => {
const v = recordRequestRejected("conn-a", T0);
assert.equal(v.streak, 1);
assert.equal(v.counted, true);
assert.equal(v.escalate, false);
assert.equal(v.cooldownMs, COOLDOWN_MS.requestRejected);
assert.equal(v.threshold, REQUEST_REJECTED_ESCALATION_THRESHOLD);
assert.equal(v.windowMs, REQUEST_REJECTED_WINDOW_MS);
});
test("refusals that land while the connection is already excluded are the same episode", () => {
const first = recordRequestRejected("conn-a", T0);
// Three parallel requests that were in flight when the first 403 came back.
const inFlight = [1, 2, 3].map((s) => recordRequestRejected("conn-a", T0 + s * 1000));
for (const v of inFlight) {
assert.equal(v.counted, false, "not new evidence");
assert.equal(v.streak, 1, "streak stays at 1");
assert.equal(v.escalate, false, "a burst must never ban");
assert.ok(v.cooldownMs > 0 && v.cooldownMs <= first.cooldownMs, "remaining cooldown");
}
});
test("consecutive episodes grow the cooldown and escalate at the threshold", () => {
const first = recordRequestRejected("conn-a", T0);
const second = recordRequestRejected("conn-a", T0 + first.cooldownMs + 1);
assert.equal(second.streak, 2);
assert.equal(second.counted, true);
assert.equal(second.cooldownMs, COOLDOWN_MS.requestRejectedRepeat);
assert.equal(second.escalate, false);
const third = recordRequestRejected("conn-a", T0 + first.cooldownMs + second.cooldownMs + 2);
assert.equal(third.streak, 3);
assert.equal(third.escalate, true, "third consecutive refusal escalates");
assert.equal(third.cooldownMs, 0, "terminal state replaces the cooldown");
assert.equal(hasRequestRejectedStreak("conn-a"), false, "escalation clears the streak");
// A reconnected connection starts from 1 again.
const afterBan = recordRequestRejected("conn-a", T0 + 60 * MIN);
assert.equal(afterBan.streak, 1);
assert.equal(afterBan.escalate, false);
});
test("a refusal outside the staleness window starts a new streak", () => {
recordRequestRejected("conn-a", T0);
const late = recordRequestRejected("conn-a", T0 + REQUEST_REJECTED_WINDOW_MS + 1);
assert.equal(late.streak, 1);
assert.equal(late.cooldownMs, COOLDOWN_MS.requestRejected);
});
test("a success (clearRequestRejectedStreak) resets the count", () => {
const first = recordRequestRejected("conn-a", T0);
clearRequestRejectedStreak("conn-a");
const next = recordRequestRejected("conn-a", T0 + first.cooldownMs + 1);
assert.equal(next.streak, 1, "sporadic refusals separated by successes never accumulate");
assert.equal(next.cooldownMs, COOLDOWN_MS.requestRejected);
});
test("streaks are per connection", () => {
const a1 = recordRequestRejected("conn-a", T0);
recordRequestRejected("conn-a", T0 + a1.cooldownMs + 1);
const other = recordRequestRejected("conn-b", T0 + a1.cooldownMs + 2);
assert.equal(other.streak, 1);
});
test("seedRequestRejectedStreak restores one refusal from before a restart", () => {
const lastRefusalAt = T0 - 10 * MIN;
assert.equal(seedRequestRejectedStreak("conn-a", lastRefusalAt, T0), true);
assert.equal(hasRequestRejectedStreak("conn-a"), true);
const v = recordRequestRejected("conn-a", T0);
assert.equal(v.streak, 2, "the seeded refusal counts as the first of the streak");
assert.equal(v.counted, true, "the seeded refusal's cooldown is not re-applied");
});
test("seedRequestRejectedStreak ignores stale rows and existing state", () => {
assert.equal(seedRequestRejectedStreak("conn-a", T0 - REQUEST_REJECTED_WINDOW_MS, T0), false);
assert.equal(seedRequestRejectedStreak("conn-a", Number.NaN, T0), false);
assert.equal(hasRequestRejectedStreak("conn-a"), false);
recordRequestRejected("conn-b", T0);
assert.equal(seedRequestRejectedStreak("conn-b", T0 - MIN, T0), false, "already tracked");
});