diff --git a/docs/architecture/RESILIENCE_GUIDE.md b/docs/architecture/RESILIENCE_GUIDE.md index 98629da7db..325058940f 100644 --- a/docs/architecture/RESILIENCE_GUIDE.md +++ b/docs/architecture/RESILIENCE_GUIDE.md @@ -330,32 +330,59 @@ excludeMarkers, defaultRetryAfterMs}`), matched via `applyStatusRestatement()`. Permanent errors (agentrouter's `无权访问模型` — no access to this model) are NEVER restated: `excludeMarkers` vetoes the rule even when `textMarkers` hit, -so the error keeps its original status and nothing retries it forever. A -separate provider classification rule -(`agentrouter-model-access-denied` in `open-sse/config/providerErrorRules.ts`) -declares an `auth_error`/scope-`model` match for this text, but it does not -fire on the live production path today: the rule only matches `status === -403`, and `checkFallbackError`'s apikey-category `FORBIDDEN` branch -(`open-sse/services/accountFallback.ts`) returns early for a plain 403 -*before* the provider-rule lookup ever runs. In practice a `无权访问模型` 403 -is handled the same way as the base apikey-provider 403 path (see Connection -Cooldown, §2), not as a 6h model lockout. The rule still exists as a -declarative classification consumable by future callers of `classifyError` -with context — wiring it into the production `checkFallbackError` path is -tracked as a follow-up, not yet done. +so the error keeps its original status and nothing retries it forever. The +matching provider classification rule +(`agentrouter-model-access-denied` in `open-sse/config/providerErrorRules.ts`: +`reason: "auth_error"`, `scope: "model"`, a `6h` declared base cooldown) is +consulted by `checkFallbackError` (`open-sse/services/accountFallback.ts`) +*before* the generic apikey-category `FORBIDDEN` early-return, gated on +`honorsRuleLockScope(provider)` (#10334 — currently agentrouter-exclusive via +the `HONORS_RULE_LOCK_SCOPE_PROVIDERS` allowlist in +`providerErrorRules.ts`). The rule's declared 6h cooldown flows through as +`fallbackResult.baseCooldownMs`, but it still feeds the pre-existing +per-model-quota lockout path (`lockModelIfPerModelQuota()` / +`recordModelLockoutFailure()`, unchanged by #10334 except for the cooldown +source): it is clamped down to the operator's `mlSettings.maxCooldownMs` +(default `1_800_000ms` / 30min), like every other model lockout, and the +*persisted lockout reason* stays the pre-existing hardcoded `"forbidden"`, +not the rule's `"auth_error"` — only the cooldown duration is honored +end-to-end, not the reason string. The connection itself stays active; +sibling models on the same connection are unaffected. -Restated quota errors (`额度不足`) do reach a provider rule in production -(`agentrouter-user-quota-exhausted`, scope `"connection"`), but `scope` on -`ProviderErrorRuleMatch` is currently informational — the persistence path -(`checkFallbackError` → `combo.ts`) only consumes `reason` and `cooldownMs`, -never `scope`. What actually happens for agentrouter (`passthroughModels: -true` → `hasPerModelQuota()` returns `true`) is a **per-model** lockout via -`recordModelLockoutFailure()`: the connection itself is never cooled down for -this error (`combo.ts` skips `recordProviderCooldown` for 429 when -`hasPerModelQuota` is true), so other models on the same account keep being -tried — each one burns one call and its own lockout before combo routing -moves on. Honoring `scope` end-to-end (so a `"connection"` match actually -locks the connection) is tracked as a follow-up. +Restated quota errors (`额度不足`) reach a provider rule in production +(`agentrouter-user-quota-exhausted`: `reason: "quota_exhausted"`, `scope: +"connection"`, no declared cooldown of its own — the persistence layer's +scaled backoff default applies). Since #10334, `scope` on +`ProviderErrorRuleMatch` IS consumed end-to-end, but **only** for providers in +the `HONORS_RULE_LOCK_SCOPE_PROVIDERS` allowlist (`providerErrorRules.ts` — +today only `"agentrouter"`, gated via `honorsRuleLockScope()`). For every +other provider `scope` remains informational, exactly as before #10334. +`checkFallbackError` surfaces the matched rule's scope as +`fallbackResult.ruleScope`; `isAgentrouterConnectionQuotaScope()` +(`src/sse/services/auth.ts`) is the shared guard that confirms a +`ruleScope` is genuinely safe to honor as a connection-wide, self-recovering +signal (scope `"connection"`, reason `quota_exhausted`, never `permanent`, +never `creditsExhausted` — a defense against a future rule pairing scope +`"connection"` with a permanent account state). Two consumers call it: + +- **Persistence** (`markAccountUnavailable()`, `src/sse/services/auth.ts`): + instead of falling into the passthrough-provider **per-model** lockout + branch (agentrouter is `passthroughModels: true` → `hasPerModelQuota()` + returns `true`), it applies a **temporary connection cooldown** — + `testStatus: "unavailable"` + `rateLimitedUntil`, never a terminal status + (`credits_exhausted`/`banned`/`expired`) — so the connection self-recovers + once the cooldown lapses instead of requiring a manual credential reset. + Skipped for connections with `disableCooling: true` (#2997): that opt-out + falls through to the per-model lockout instead (a documented trade-off — + see the code comment above the branch). +- **Same-request combo routing** (`applyComboTargetExhaustion()`, + `open-sse/services/combo/targetExhaustion.ts`): the same guard marks the + connection into the in-memory `exhaustedConnections` set so remaining + same-connection targets are skipped for the rest of the *current* request — + not just future requests. Without this, a combo with several legs on the + same exhausted agentrouter account would burn one upstream call per + remaining leg before the persisted cooldown above ever took effect on the + next request. ### Two-stage design: status restatement, then classification @@ -380,6 +407,15 @@ allowlisted providers, the structured error otherwise. Adding a provider to that the default path for every provider not on the list stays byte-for-byte unchanged. +A rule's `scope` (`model` / `provider` / `connection`) is a separate opt-in +from `FULL_TEXT_RULE_PROVIDERS`: `checkFallbackError` only surfaces it as +`fallbackResult.ruleScope`, and downstream consumers only honor it as +anything other than an informational label, for providers in the +`HONORS_RULE_LOCK_SCOPE_PROVIDERS` allowlist in the same file (`gated via +honorsRuleLockScope()` — today only `"agentrouter"`). See "Restated quota +errors" above for what a `scope: "connection"` match actually does once a +provider is on that allowlist. + ### Adding a new quota-misstating gateway 1. Register one rule array in `statusRestatementRegistry` @@ -395,7 +431,15 @@ byte-for-byte unchanged. `checkFallbackError` only ever hands the rule the structured `{code, type}` error and a body-text rule will never match live traffic. Rules that match purely on `status`/`headers` (like Opencode's or - Minimax's) do not need this opt-in. + Minimax's) do not need this opt-in. Separately, if the rule declares + `scope: "connection"` and the intent is an actual connection-wide cooldown + plus same-request combo skip (not just an informational label), add the + provider id to `HONORS_RULE_LOCK_SCOPE_PROVIDERS` in the same file — this + is what gates `isAgentrouterConnectionQuotaScope()`-style consumption in + `markAccountUnavailable()` (`src/sse/services/auth.ts`) and + `applyComboTargetExhaustion()` + (`open-sse/services/combo/targetExhaustion.ts`); without it, `scope` + still flows through `fallbackResult.ruleScope` but nothing acts on it. 3. Add unit tests mirroring `tests/unit/upstream-status-restatement.test.ts` and `tests/unit/agentrouter-error-rules.test.ts` (including the not-permanent / not-creditsExhausted guards, and — if the provider needs diff --git a/open-sse/services/combo/targetExhaustion.ts b/open-sse/services/combo/targetExhaustion.ts index c4a880e79a..3d7af3afd7 100644 --- a/open-sse/services/combo/targetExhaustion.ts +++ b/open-sse/services/combo/targetExhaustion.ts @@ -27,6 +27,10 @@ import { import { RateLimitReason } from "../../config/constants.ts"; import { isProviderCircuitOpenResult, isRequestScopedUpstreamFailure } from "./comboPredicates.ts"; import { isCloudflareFingerprintRejection } from "../errorClassifier.ts"; +// #10334 — agentrouter-exclusive predicate shared with the persistence layer +// (markAccountUnavailable) so the same-request combo skip and the persisted +// connection cooldown agree on exactly which fallbackResult shapes qualify. +import { isAgentrouterConnectionQuotaScope } from "@/sse/services/auth"; import type { ComboLogger, ResolvedComboTarget } from "./types.ts"; // Connection-level failure statuses: the provider connection itself is likely bad (upstream @@ -60,7 +64,13 @@ export type ComboExhaustionSets = { export type ApplyComboTargetExhaustionOptions = { result: { status: number; headers?: Headers | null }; - fallbackResult: Parameters[0]; + fallbackResult: Parameters[0] & { + /** #10334 — agentrouter-exclusive; see isAgentrouterConnectionQuotaScope + * (src/sse/services/auth.ts). Populated only for providers in + * HONORS_RULE_LOCK_SCOPE_PROVIDERS (today: agentrouter only). */ + ruleScope?: "model" | "provider" | "connection"; + permanent?: boolean; + }; errorText: string; rawModel: string; isTokenLimitBreach: boolean; @@ -86,6 +96,26 @@ export function applyComboTargetExhaustion( const { result, sets, log, tag, errorText, structuredError } = opts; const provider = target.provider; + // #10334: agentrouter-exclusive account-wide quota exhaustion ("额度不足", + // restated to 429) must skip remaining SAME-CONNECTION targets within THIS + // request too, not just via the persisted cooldown markAccountUnavailable + // applies for the NEXT request. agentrouter is a passthroughModels provider + // (hasPerModelQuota() === true), so without this branch the classification + // below would fall straight through isProviderQuotaExhausted's + // !hasPerModelQuota() guard and markConnectionLevelExhaustion's 429-is-not- + // connection-level guard (CONNECTION_LEVEL_ERROR_STATUSES excludes 429), + // marking nothing — combo would keep burning one upstream call per + // remaining model of the same exhausted account. isAgentrouterConnectionQuotaScope + // is the same guard markAccountUnavailable uses, so both consumers agree on + // exactly which fallbackResult shapes qualify (never a permanent/credits- + // exhausted result, even one carrying ruleScope "connection"). Runs before + // the auth-level (401/403) branch below since it is unaffected by it (this + // path is 429-only) but keeps the diff to a single early return. + if (isAgentrouterConnectionQuotaScope(provider, opts.fallbackResult)) { + markAgentrouterConnectionQuotaExhaustion(target, { sets, log, tag }); + return true; + } + // #8133/#8137: auth-level failures (401/403) mean that connection's credentials are bad. // Split out to keep applyComboTargetExhaustion under the complexity ceiling. // Cloudflare 1010 (a 403 carrying error_code 1010 / browser_signature_banned) is NOT an @@ -259,6 +289,35 @@ function markAuthLevelExhaustion( } } +/** + * #10334: agentrouter-exclusive connection-scope account quota exhaustion. Mirrors + * markAuthLevelExhaustion's connectionId-present/absent split — when the target carries a + * connectionId, only that connection's account is exhausted (sibling agentrouter connections + * for the same user may still have quota); fall back to whole-provider exhaustion only when no + * connectionId is available. + */ +function markAgentrouterConnectionQuotaExhaustion( + target: ResolvedComboTarget, + opts: Pick +): void { + const { sets, log, tag } = opts; + const provider = target.provider; + const connId = target.connectionId ?? undefined; + if (connId) { + sets.exhaustedConnections.add(`${provider}:${connId}`); + log.info( + tag, + `Provider ${provider} connection ${connId} account quota exhausted (rule scope=connection) — marking for skip on remaining targets (#10334)` + ); + } else { + sets.exhaustedProviders.add(provider as string); + log.info( + tag, + `Provider ${provider} account quota exhausted (rule scope=connection, no connectionId) — marking for skip on remaining targets (#10334)` + ); + } +} + /** * #1731v2: connection-level errors (408/5xx, excluding the OmniRoute circuit-open signal) suggest * the provider connection itself is bad → skip remaining same-connection (or same-provider, when diff --git a/tests/unit/agentrouter-lock-scope-10334.test.ts b/tests/unit/agentrouter-lock-scope-10334.test.ts index fd538b762e..8a86e04bf1 100644 --- a/tests/unit/agentrouter-lock-scope-10334.test.ts +++ b/tests/unit/agentrouter-lock-scope-10334.test.ts @@ -21,6 +21,10 @@ const core = await import("../../src/lib/db/core.ts"); const providersDb = await import("../../src/lib/db/providers.ts"); const auth = await import("../../src/sse/services/auth.ts"); const accountFallback = await import("../../open-sse/services/accountFallback.ts"); +const { applyComboTargetExhaustion } = await import( + "../../open-sse/services/combo/targetExhaustion.ts" +); +const { classifyProviderError } = await import("../../open-sse/services/errorClassifier.ts"); const QUOTA_EXHAUSTED_429 = '{"error":{"message":"账户额度不足，请充值后重试"}}'; const MODEL_ACCESS_DENIED_403 = '{"error":{"message":"无权访问模型 claude-opus-5"}}'; @@ -354,3 +358,189 @@ test("disableCooling=true skips the connection-scope branch and falls back to pe "expected a per-model lockout when disableCooling bypasses the connection branch" ); }); + +// ─── Task 3 (#10334): combo skips the exhausted agentrouter connection +// WITHIN THE SAME REQUEST ────────────────────────────────────────────────── +// The tests above pin markAccountUnavailable's PERSISTED connection cooldown +// — that only protects the NEXT request. applyComboTargetExhaustion (the +// #1731/#1731v2 shared classifier both combo dispatchers call after every +// target's upstream error — open-sse/services/combo/targetExhaustion.ts) is +// what decides whether remaining targets of the CURRENT request are skipped. +// Without a matching gate there, a combo with 5 legs on the same exhausted +// agentrouter account would still burn all 5 upstream calls before the +// persisted cooldown from the tests above ever kicks in. + +function comboSets() { + return { + exhaustedProviders: new Set(), + exhaustedConnections: new Set(), + transientRateLimitedProviders: new Set(), + }; +} + +function comboTarget(overrides: Record = {}) { + return { + kind: "model", + executionKey: "ek", + modelStr: "agentrouter/claude-opus-5", + provider: "agentrouter", + providerId: null, + connectionId: "conn-agentrouter-1", + ...overrides, + } as Parameters[0]; +} + +const comboLog = { info() {}, warn() {}, error() {}, debug() {} }; + +const comboBaseOpts = { + errorText: QUOTA_EXHAUSTED_429, + rawModel: "claude-opus-5", + isTokenLimitBreach: false, + allAccountsRateLimited: false, + requestScopedFailure: false, + log: comboLog, + tag: "COMBO", + exhaustedLogLevel: "info" as const, +}; + +// The real shape checkFallbackError surfaces for agentrouter's restated 429 +// (open-sse/config/providerErrorRules.ts's "agentrouter-user-quota-exhausted" +// rule: reason "quota_exhausted", scope "connection") — same shape pinned by +// isAgentrouterConnectionQuotaScope's own tests above. +const CONNECTION_SCOPE_FALLBACK_RESULT = { + ruleScope: "connection" as const, + reason: "quota_exhausted", +}; + +test("combo in-request skip: agentrouter connection-scope quota marks exhaustedConnections (#10334)", () => { + const sets = comboSets(); + const exhausted = applyComboTargetExhaustion(comboTarget(), { + ...comboBaseOpts, + result: { status: 429 }, + fallbackResult: CONNECTION_SCOPE_FALLBACK_RESULT, + sets, + }); + assert.equal( + exhausted, + true, + "combo must treat this like an exhausted target — no same-target retry" + ); + assert.ok( + sets.exhaustedConnections.has("agentrouter:conn-agentrouter-1"), + "the exhausted account's connection must be marked so remaining same-connection targets are skipped this request" + ); + assert.equal( + sets.exhaustedProviders.size, + 0, + "must NOT exhaust the whole provider — sibling agentrouter connections keep their own quota" + ); +}); + +test("combo in-request skip: no connectionId falls back to whole-provider exhaustion", () => { + const sets = comboSets(); + const exhausted = applyComboTargetExhaustion(comboTarget({ connectionId: null }), { + ...comboBaseOpts, + result: { status: 429 }, + fallbackResult: CONNECTION_SCOPE_FALLBACK_RESULT, + sets, + }); + assert.equal(exhausted, true); + assert.ok( + sets.exhaustedProviders.has("agentrouter"), + "no connectionId to scope to — must fall back to whole-provider, mirroring markAuthLevelExhaustion" + ); + assert.equal(sets.exhaustedConnections.size, 0); +}); + +test("exclusivity: an equivalent connection-scope-shaped result for ollama-cloud marks nothing (#10334 is agentrouter-only)", () => { + const sets = comboSets(); + // Synthetic: production never actually produces ruleScope for a + // non-allowlisted provider (honorsRuleLockScope gates it upstream inside + // checkFallbackError) — feeding it here directly proves + // applyComboTargetExhaustion ALSO re-checks the provider via + // isAgentrouterConnectionQuotaScope rather than trusting whatever shape + // it is handed. + const exhausted = applyComboTargetExhaustion( + comboTarget({ provider: "ollama-cloud", connectionId: "conn-ollama-1" }), + { + ...comboBaseOpts, + result: { status: 429 }, + fallbackResult: CONNECTION_SCOPE_FALLBACK_RESULT, + sets, + } + ); + assert.equal( + exhausted, + false, + "ollama-cloud must fall through to today's per-model-quota behavior unchanged" + ); + assert.equal(sets.exhaustedConnections.size, 0); + assert.equal(sets.exhaustedProviders.size, 0); +}); + +test("exclusivity: vertex with the same synthetic connection-scope result marks nothing", () => { + const sets = comboSets(); + const exhausted = applyComboTargetExhaustion( + comboTarget({ provider: "vertex", connectionId: "conn-vertex-1" }), + { + ...comboBaseOpts, + result: { status: 429 }, + fallbackResult: CONNECTION_SCOPE_FALLBACK_RESULT, + sets, + } + ); + assert.equal(exhausted, false); + assert.equal(sets.exhaustedConnections.size, 0); + assert.equal(sets.exhaustedProviders.size, 0); +}); + +test("guard: a permanent agentrouter fallbackResult with scope connection does NOT mark the connection exhausted here either", () => { + const sets = comboSets(); + const exhausted = applyComboTargetExhaustion(comboTarget(), { + ...comboBaseOpts, + result: { status: 429 }, + fallbackResult: { ruleScope: "connection" as const, reason: "auth_error", permanent: true }, + sets, + }); + assert.equal(exhausted, false); + assert.equal(sets.exhaustedConnections.has("agentrouter:conn-agentrouter-1"), false); + assert.equal(sets.exhaustedProviders.size, 0); +}); + +test("guard: a credits-exhausted agentrouter fallbackResult with scope connection does NOT mark the connection exhausted here either", () => { + const sets = comboSets(); + const exhausted = applyComboTargetExhaustion(comboTarget(), { + ...comboBaseOpts, + result: { status: 429 }, + fallbackResult: { + ruleScope: "connection" as const, + reason: "quota_exhausted", + creditsExhausted: true, + }, + sets, + }); + assert.equal(exhausted, false); + assert.equal(sets.exhaustedConnections.has("agentrouter:conn-agentrouter-1"), false); + assert.equal(sets.exhaustedProviders.size, 0); +}); + +// ─── Invariant sentinel ───────────────────────────────────────────────── +// classifyProviderError (open-sse/services/errorClassifier.ts) must NEVER +// classify agentrouter's restated 429 body ("用户额度不足") as quota_exhausted. +// If it ever does, open-sse/handlers/chatCore.ts's providerFailure handling +// (~line 3835-3856) can reach the terminal `else` branch +// (`testStatus: "credits_exhausted"`) for agentrouter whenever +// lockModelIfPerModelQuota does not itself claim the failure — turning a +// transient, self-recovering account-quota window into a connection that +// requires a manual operator reset. agentrouter is an apikey-category +// provider (not oauth), so shouldPreserveQuotaSignalsFor429 in +// errorClassifier.ts returns false for it and the 429 branch falls through +// to RATE_LIMITED instead — pin that this stays true. +test("sentinel: classifyProviderError never returns quota_exhausted for agentrouter's restated 429 body", () => { + const classification = classifyProviderError(429, "用户额度不足", "agentrouter"); + assert.notEqual( + classification, + "quota_exhausted", + "a quota_exhausted classification here would route agentrouter's transient account quota into chatCore's terminal credits_exhausted branch (~chatCore.ts:3849)" + ); +});