feat(sse): skip exhausted agentrouter account within the same combo request

applyComboTargetExhaustion now marks an agentrouter connection into the in-memory exhaustedConnections set when checkFallbackError reports a connection-scope quota result (isAgentrouterConnectionQuotaScope, reused from the persistence layer), so remaining same-connection targets in the SAME combo request are skipped instead of each burning its own upstream call before the persisted cooldown takes effect on the next request. Gated strictly to the agentrouter allowlist — every other provider is unaffected. Also updates RESILIENCE_GUIDE.md §7 to correct two stale claims: the agentrouter-model-access-denied rule does fire in production now (feeds the per-model lockout's cooldown), and rule scope is consumed end-to-end for providers in HONORS_RULE_LOCK_SCOPE_PROVIDERS instead of staying purely informational.
2026-08-15 19:52:50 +03:00 · 2026-08-14 14:15:25 -03:00
parent a13869364e
commit c7b521dfb3
3 changed files with 320 additions and 27 deletions
--- a/docs/architecture/RESILIENCE_GUIDE.md
+++ b/docs/architecture/RESILIENCE_GUIDE.md
@@ -330,32 +330,59 @@ excludeMarkers, defaultRetryAfterMs}`), matched via `applyStatusRestatement()`.

 Permanent errors (agentrouter's `无权访问模型` — no access to this model) are
 NEVER restated: `excludeMarkers` vetoes the rule even when `textMarkers` hit,
-so the error keeps its original status and nothing retries it forever. A
-separate provider classification rule
-(`agentrouter-model-access-denied` in `open-sse/config/providerErrorRules.ts`)
-declares an `auth_error`/scope-`model` match for this text, but it does not
-fire on the live production path today: the rule only matches `status ===
-403`, and `checkFallbackError`'s apikey-category `FORBIDDEN` branch
-(`open-sse/services/accountFallback.ts`) returns early for a plain 403
-*before* the provider-rule lookup ever runs. In practice a `无权访问模型` 403
-is handled the same way as the base apikey-provider 403 path (see Connection
-Cooldown, §2), not as a 6h model lockout. The rule still exists as a
-declarative classification consumable by future callers of `classifyError`
-with context — wiring it into the production `checkFallbackError` path is
-tracked as a follow-up, not yet done.
+so the error keeps its original status and nothing retries it forever. The
+matching provider classification rule
+(`agentrouter-model-access-denied` in `open-sse/config/providerErrorRules.ts`:
+`reason: "auth_error"`, `scope: "model"`, a `6h` declared base cooldown) is
+consulted by `checkFallbackError` (`open-sse/services/accountFallback.ts`)
+*before* the generic apikey-category `FORBIDDEN` early-return, gated on
+`honorsRuleLockScope(provider)` (#10334 — currently agentrouter-exclusive via
+the `HONORS_RULE_LOCK_SCOPE_PROVIDERS` allowlist in
+`providerErrorRules.ts`). The rule's declared 6h cooldown flows through as
+`fallbackResult.baseCooldownMs`, but it still feeds the pre-existing
+per-model-quota lockout path (`lockModelIfPerModelQuota()` /
+`recordModelLockoutFailure()`, unchanged by #10334 except for the cooldown
+source): it is clamped down to the operator's `mlSettings.maxCooldownMs`
+(default `1_800_000ms` / 30min), like every other model lockout, and the
+*persisted lockout reason* stays the pre-existing hardcoded `"forbidden"`,
+not the rule's `"auth_error"` — only the cooldown duration is honored
+end-to-end, not the reason string. The connection itself stays active;
+sibling models on the same connection are unaffected.

-Restated quota errors (`额度不足`) do reach a provider rule in production
-(`agentrouter-user-quota-exhausted`, scope `"connection"`), but `scope` on
-`ProviderErrorRuleMatch` is currently informational — the persistence path
-(`checkFallbackError` → `combo.ts`) only consumes `reason` and `cooldownMs`,
-never `scope`. What actually happens for agentrouter (`passthroughModels:
-true` → `hasPerModelQuota()` returns `true`) is a **per-model** lockout via
-`recordModelLockoutFailure()`: the connection itself is never cooled down for
-this error (`combo.ts` skips `recordProviderCooldown` for 429 when
-`hasPerModelQuota` is true), so other models on the same account keep being
-tried — each one burns one call and its own lockout before combo routing
-moves on. Honoring `scope` end-to-end (so a `"connection"` match actually
-locks the connection) is tracked as a follow-up.
+Restated quota errors (`额度不足`) reach a provider rule in production
+(`agentrouter-user-quota-exhausted`: `reason: "quota_exhausted"`, `scope:
+"connection"`, no declared cooldown of its own — the persistence layer's
+scaled backoff default applies). Since #10334, `scope` on
+`ProviderErrorRuleMatch` IS consumed end-to-end, but **only** for providers in
+the `HONORS_RULE_LOCK_SCOPE_PROVIDERS` allowlist (`providerErrorRules.ts` —
+today only `"agentrouter"`, gated via `honorsRuleLockScope()`). For every
+other provider `scope` remains informational, exactly as before #10334.
+`checkFallbackError` surfaces the matched rule's scope as
+`fallbackResult.ruleScope`; `isAgentrouterConnectionQuotaScope()`
+(`src/sse/services/auth.ts`) is the shared guard that confirms a
+`ruleScope` is genuinely safe to honor as a connection-wide, self-recovering
+signal (scope `"connection"`, reason `quota_exhausted`, never `permanent`,
+never `creditsExhausted` — a defense against a future rule pairing scope
+`"connection"` with a permanent account state). Two consumers call it:
+
+- **Persistence** (`markAccountUnavailable()`, `src/sse/services/auth.ts`):
+  instead of falling into the passthrough-provider **per-model** lockout
+  branch (agentrouter is `passthroughModels: true` → `hasPerModelQuota()`
+  returns `true`), it applies a **temporary connection cooldown** —
+  `testStatus: "unavailable"` + `rateLimitedUntil`, never a terminal status
+  (`credits_exhausted`/`banned`/`expired`) — so the connection self-recovers
+  once the cooldown lapses instead of requiring a manual credential reset.
+  Skipped for connections with `disableCooling: true` (#2997): that opt-out
+  falls through to the per-model lockout instead (a documented trade-off —
+  see the code comment above the branch).
+- **Same-request combo routing** (`applyComboTargetExhaustion()`,
+  `open-sse/services/combo/targetExhaustion.ts`): the same guard marks the
+  connection into the in-memory `exhaustedConnections` set so remaining
+  same-connection targets are skipped for the rest of the *current* request —
+  not just future requests. Without this, a combo with several legs on the
+  same exhausted agentrouter account would burn one upstream call per
+  remaining leg before the persisted cooldown above ever took effect on the
+  next request.

 ### Two-stage design: status restatement, then classification

@@ -380,6 +407,15 @@ allowlisted providers, the structured error otherwise. Adding a provider to
 that the default path for every provider not on the list stays
 byte-for-byte unchanged.

+A rule's `scope` (`model` / `provider` / `connection`) is a separate opt-in
+from `FULL_TEXT_RULE_PROVIDERS`: `checkFallbackError` only surfaces it as
+`fallbackResult.ruleScope`, and downstream consumers only honor it as
+anything other than an informational label, for providers in the
+`HONORS_RULE_LOCK_SCOPE_PROVIDERS` allowlist in the same file (`gated via
+honorsRuleLockScope()` — today only `"agentrouter"`). See "Restated quota
+errors" above for what a `scope: "connection"` match actually does once a
+provider is on that allowlist.
+
 ### Adding a new quota-misstating gateway

 1. Register one rule array in `statusRestatementRegistry`
@@ -395,7 +431,15 @@ byte-for-byte unchanged.
   `checkFallbackError` only ever hands the rule the structured
   `{code, type}` error and a body-text rule will never match live traffic.
   Rules that match purely on `status`/`headers` (like Opencode's or
-   Minimax's) do not need this opt-in.
+   Minimax's) do not need this opt-in. Separately, if the rule declares
+   `scope: "connection"` and the intent is an actual connection-wide cooldown
+   plus same-request combo skip (not just an informational label), add the
+   provider id to `HONORS_RULE_LOCK_SCOPE_PROVIDERS` in the same file — this
+   is what gates `isAgentrouterConnectionQuotaScope()`-style consumption in
+   `markAccountUnavailable()` (`src/sse/services/auth.ts`) and
+   `applyComboTargetExhaustion()`
+   (`open-sse/services/combo/targetExhaustion.ts`); without it, `scope`
+   still flows through `fallbackResult.ruleScope` but nothing acts on it.
 3. Add unit tests mirroring `tests/unit/upstream-status-restatement.test.ts`
   and `tests/unit/agentrouter-error-rules.test.ts` (including the
   not-permanent / not-creditsExhausted guards, and — if the provider needs
--- a/open-sse/services/combo/targetExhaustion.ts
+++ b/open-sse/services/combo/targetExhaustion.ts
@@ -27,6 +27,10 @@ import {
 import { RateLimitReason } from "../../config/constants.ts";
 import { isProviderCircuitOpenResult, isRequestScopedUpstreamFailure } from "./comboPredicates.ts";
 import { isCloudflareFingerprintRejection } from "../errorClassifier.ts";
+// #10334 — agentrouter-exclusive predicate shared with the persistence layer
+// (markAccountUnavailable) so the same-request combo skip and the persisted
+// connection cooldown agree on exactly which fallbackResult shapes qualify.
+import { isAgentrouterConnectionQuotaScope } from "@/sse/services/auth";
 import type { ComboLogger, ResolvedComboTarget } from "./types.ts";

 // Connection-level failure statuses: the provider connection itself is likely bad (upstream
@@ -60,7 +64,13 @@ export type ComboExhaustionSets = {

 export type ApplyComboTargetExhaustionOptions = {
  result: { status: number; headers?: Headers | null };
-  fallbackResult: Parameters<typeof isProviderExhaustedReason>[0];
+  fallbackResult: Parameters<typeof isProviderExhaustedReason>[0] & {
+    /** #10334 — agentrouter-exclusive; see isAgentrouterConnectionQuotaScope
+     * (src/sse/services/auth.ts). Populated only for providers in
+     * HONORS_RULE_LOCK_SCOPE_PROVIDERS (today: agentrouter only). */
+    ruleScope?: "model" | "provider" | "connection";
+    permanent?: boolean;
+  };
  errorText: string;
  rawModel: string;
  isTokenLimitBreach: boolean;
@@ -86,6 +96,26 @@ export function applyComboTargetExhaustion(
  const { result, sets, log, tag, errorText, structuredError } = opts;
  const provider = target.provider;

+  // #10334: agentrouter-exclusive account-wide quota exhaustion ("额度不足",
+  // restated to 429) must skip remaining SAME-CONNECTION targets within THIS
+  // request too, not just via the persisted cooldown markAccountUnavailable
+  // applies for the NEXT request. agentrouter is a passthroughModels provider
+  // (hasPerModelQuota() === true), so without this branch the classification
+  // below would fall straight through isProviderQuotaExhausted's
+  // !hasPerModelQuota() guard and markConnectionLevelExhaustion's 429-is-not-
+  // connection-level guard (CONNECTION_LEVEL_ERROR_STATUSES excludes 429),
+  // marking nothing — combo would keep burning one upstream call per
+  // remaining model of the same exhausted account. isAgentrouterConnectionQuotaScope
+  // is the same guard markAccountUnavailable uses, so both consumers agree on
+  // exactly which fallbackResult shapes qualify (never a permanent/credits-
+  // exhausted result, even one carrying ruleScope "connection"). Runs before
+  // the auth-level (401/403) branch below since it is unaffected by it (this
+  // path is 429-only) but keeps the diff to a single early return.
+  if (isAgentrouterConnectionQuotaScope(provider, opts.fallbackResult)) {
+    markAgentrouterConnectionQuotaExhaustion(target, { sets, log, tag });
+    return true;
+  }
+
  // #8133/#8137: auth-level failures (401/403) mean that connection's credentials are bad.
  // Split out to keep applyComboTargetExhaustion under the complexity ceiling.
  // Cloudflare 1010 (a 403 carrying error_code 1010 / browser_signature_banned) is NOT an
@@ -259,6 +289,35 @@ function markAuthLevelExhaustion(
  }
 }

+/**
+ * #10334: agentrouter-exclusive connection-scope account quota exhaustion. Mirrors
+ * markAuthLevelExhaustion's connectionId-present/absent split — when the target carries a
+ * connectionId, only that connection's account is exhausted (sibling agentrouter connections
+ * for the same user may still have quota); fall back to whole-provider exhaustion only when no
+ * connectionId is available.
+ */
+function markAgentrouterConnectionQuotaExhaustion(
+  target: ResolvedComboTarget,
+  opts: Pick<ApplyComboTargetExhaustionOptions, "sets" | "log" | "tag">
+): void {
+  const { sets, log, tag } = opts;
+  const provider = target.provider;
+  const connId = target.connectionId ?? undefined;
+  if (connId) {
+    sets.exhaustedConnections.add(`${provider}:${connId}`);
+    log.info(
+      tag,
+      `Provider ${provider} connection ${connId} account quota exhausted (rule scope=connection) — marking for skip on remaining targets (#10334)`
+    );
+  } else {
+    sets.exhaustedProviders.add(provider as string);
+    log.info(
+      tag,
+      `Provider ${provider} account quota exhausted (rule scope=connection, no connectionId) — marking for skip on remaining targets (#10334)`
+    );
+  }
+}
+
 /**
 * #1731v2: connection-level errors (408/5xx, excluding the OmniRoute circuit-open signal) suggest
 * the provider connection itself is bad → skip remaining same-connection (or same-provider, when
--- a/tests/unit/agentrouter-lock-scope-10334.test.ts
+++ b/tests/unit/agentrouter-lock-scope-10334.test.ts
@@ -21,6 +21,10 @@ const core = await import("../../src/lib/db/core.ts");
 const providersDb = await import("../../src/lib/db/providers.ts");
 const auth = await import("../../src/sse/services/auth.ts");
 const accountFallback = await import("../../open-sse/services/accountFallback.ts");
+const { applyComboTargetExhaustion } = await import(
+  "../../open-sse/services/combo/targetExhaustion.ts"
+);
+const { classifyProviderError } = await import("../../open-sse/services/errorClassifier.ts");

 const QUOTA_EXHAUSTED_429 = '{"error":{"message":"账户额度不足，请充值后重试"}}';
 const MODEL_ACCESS_DENIED_403 = '{"error":{"message":"无权访问模型 claude-opus-5"}}';
@@ -354,3 +358,189 @@ test("disableCooling=true skips the connection-scope branch and falls back to pe
    "expected a per-model lockout when disableCooling bypasses the connection branch"
  );
 });
+
+// ─── Task 3 (#10334): combo skips the exhausted agentrouter connection
+// WITHIN THE SAME REQUEST ──────────────────────────────────────────────────
+// The tests above pin markAccountUnavailable's PERSISTED connection cooldown
+// — that only protects the NEXT request. applyComboTargetExhaustion (the
+// #1731/#1731v2 shared classifier both combo dispatchers call after every
+// target's upstream error — open-sse/services/combo/targetExhaustion.ts) is
+// what decides whether remaining targets of the CURRENT request are skipped.
+// Without a matching gate there, a combo with 5 legs on the same exhausted
+// agentrouter account would still burn all 5 upstream calls before the
+// persisted cooldown from the tests above ever kicks in.
+
+function comboSets() {
+  return {
+    exhaustedProviders: new Set<string>(),
+    exhaustedConnections: new Set<string>(),
+    transientRateLimitedProviders: new Set<string>(),
+  };
+}
+
+function comboTarget(overrides: Record<string, unknown> = {}) {
+  return {
+    kind: "model",
+    executionKey: "ek",
+    modelStr: "agentrouter/claude-opus-5",
+    provider: "agentrouter",
+    providerId: null,
+    connectionId: "conn-agentrouter-1",
+    ...overrides,
+  } as Parameters<typeof applyComboTargetExhaustion>[0];
+}
+
+const comboLog = { info() {}, warn() {}, error() {}, debug() {} };
+
+const comboBaseOpts = {
+  errorText: QUOTA_EXHAUSTED_429,
+  rawModel: "claude-opus-5",
+  isTokenLimitBreach: false,
+  allAccountsRateLimited: false,
+  requestScopedFailure: false,
+  log: comboLog,
+  tag: "COMBO",
+  exhaustedLogLevel: "info" as const,
+};
+
+// The real shape checkFallbackError surfaces for agentrouter's restated 429
+// (open-sse/config/providerErrorRules.ts's "agentrouter-user-quota-exhausted"
+// rule: reason "quota_exhausted", scope "connection") — same shape pinned by
+// isAgentrouterConnectionQuotaScope's own tests above.
+const CONNECTION_SCOPE_FALLBACK_RESULT = {
+  ruleScope: "connection" as const,
+  reason: "quota_exhausted",
+};
+
+test("combo in-request skip: agentrouter connection-scope quota marks exhaustedConnections (#10334)", () => {
+  const sets = comboSets();
+  const exhausted = applyComboTargetExhaustion(comboTarget(), {
+    ...comboBaseOpts,
+    result: { status: 429 },
+    fallbackResult: CONNECTION_SCOPE_FALLBACK_RESULT,
+    sets,
+  });
+  assert.equal(
+    exhausted,
+    true,
+    "combo must treat this like an exhausted target — no same-target retry"
+  );
+  assert.ok(
+    sets.exhaustedConnections.has("agentrouter:conn-agentrouter-1"),
+    "the exhausted account's connection must be marked so remaining same-connection targets are skipped this request"
+  );
+  assert.equal(
+    sets.exhaustedProviders.size,
+    0,
+    "must NOT exhaust the whole provider — sibling agentrouter connections keep their own quota"
+  );
+});
+
+test("combo in-request skip: no connectionId falls back to whole-provider exhaustion", () => {
+  const sets = comboSets();
+  const exhausted = applyComboTargetExhaustion(comboTarget({ connectionId: null }), {
+    ...comboBaseOpts,
+    result: { status: 429 },
+    fallbackResult: CONNECTION_SCOPE_FALLBACK_RESULT,
+    sets,
+  });
+  assert.equal(exhausted, true);
+  assert.ok(
+    sets.exhaustedProviders.has("agentrouter"),
+    "no connectionId to scope to — must fall back to whole-provider, mirroring markAuthLevelExhaustion"
+  );
+  assert.equal(sets.exhaustedConnections.size, 0);
+});
+
+test("exclusivity: an equivalent connection-scope-shaped result for ollama-cloud marks nothing (#10334 is agentrouter-only)", () => {
+  const sets = comboSets();
+  // Synthetic: production never actually produces ruleScope for a
+  // non-allowlisted provider (honorsRuleLockScope gates it upstream inside
+  // checkFallbackError) — feeding it here directly proves
+  // applyComboTargetExhaustion ALSO re-checks the provider via
+  // isAgentrouterConnectionQuotaScope rather than trusting whatever shape
+  // it is handed.
+  const exhausted = applyComboTargetExhaustion(
+    comboTarget({ provider: "ollama-cloud", connectionId: "conn-ollama-1" }),
+    {
+      ...comboBaseOpts,
+      result: { status: 429 },
+      fallbackResult: CONNECTION_SCOPE_FALLBACK_RESULT,
+      sets,
+    }
+  );
+  assert.equal(
+    exhausted,
+    false,
+    "ollama-cloud must fall through to today's per-model-quota behavior unchanged"
+  );
+  assert.equal(sets.exhaustedConnections.size, 0);
+  assert.equal(sets.exhaustedProviders.size, 0);
+});
+
+test("exclusivity: vertex with the same synthetic connection-scope result marks nothing", () => {
+  const sets = comboSets();
+  const exhausted = applyComboTargetExhaustion(
+    comboTarget({ provider: "vertex", connectionId: "conn-vertex-1" }),
+    {
+      ...comboBaseOpts,
+      result: { status: 429 },
+      fallbackResult: CONNECTION_SCOPE_FALLBACK_RESULT,
+      sets,
+    }
+  );
+  assert.equal(exhausted, false);
+  assert.equal(sets.exhaustedConnections.size, 0);
+  assert.equal(sets.exhaustedProviders.size, 0);
+});
+
+test("guard: a permanent agentrouter fallbackResult with scope connection does NOT mark the connection exhausted here either", () => {
+  const sets = comboSets();
+  const exhausted = applyComboTargetExhaustion(comboTarget(), {
+    ...comboBaseOpts,
+    result: { status: 429 },
+    fallbackResult: { ruleScope: "connection" as const, reason: "auth_error", permanent: true },
+    sets,
+  });
+  assert.equal(exhausted, false);
+  assert.equal(sets.exhaustedConnections.has("agentrouter:conn-agentrouter-1"), false);
+  assert.equal(sets.exhaustedProviders.size, 0);
+});
+
+test("guard: a credits-exhausted agentrouter fallbackResult with scope connection does NOT mark the connection exhausted here either", () => {
+  const sets = comboSets();
+  const exhausted = applyComboTargetExhaustion(comboTarget(), {
+    ...comboBaseOpts,
+    result: { status: 429 },
+    fallbackResult: {
+      ruleScope: "connection" as const,
+      reason: "quota_exhausted",
+      creditsExhausted: true,
+    },
+    sets,
+  });
+  assert.equal(exhausted, false);
+  assert.equal(sets.exhaustedConnections.has("agentrouter:conn-agentrouter-1"), false);
+  assert.equal(sets.exhaustedProviders.size, 0);
+});
+
+// ─── Invariant sentinel ─────────────────────────────────────────────────
+// classifyProviderError (open-sse/services/errorClassifier.ts) must NEVER
+// classify agentrouter's restated 429 body ("用户额度不足") as quota_exhausted.
+// If it ever does, open-sse/handlers/chatCore.ts's providerFailure handling
+// (~line 3835-3856) can reach the terminal `else` branch
+// (`testStatus: "credits_exhausted"`) for agentrouter whenever
+// lockModelIfPerModelQuota does not itself claim the failure — turning a
+// transient, self-recovering account-quota window into a connection that
+// requires a manual operator reset. agentrouter is an apikey-category
+// provider (not oauth), so shouldPreserveQuotaSignalsFor429 in
+// errorClassifier.ts returns false for it and the 429 branch falls through
+// to RATE_LIMITED instead — pin that this stays true.
+test("sentinel: classifyProviderError never returns quota_exhausted for agentrouter's restated 429 body", () => {
+  const classification = classifyProviderError(429, "用户额度不足", "agentrouter");
+  assert.notEqual(
+    classification,
+    "quota_exhausted",
+    "a quota_exhausted classification here would route agentrouter's transient account quota into chatCore's terminal credits_exhausted branch (~chatCore.ts:3849)"
+  );
+});