fix(oauth): keep the Claude refresh token so the retry budget is reachable

An unrecoverable Claude OAuth refresh (invalid_grant / refresh_token_reused,
often a dual-consumer race on the same Claude Max account) left the connection
sticky-dead: active, expired and unable to recover without a manual re-auth.

#11414 already keeps such a connection retryable for EXPIRED_RETRY_MAX sweeps,
but the very same update ran `refreshToken: null` for every rotating provider,
Claude included. The next sweep then stops at the `!conn.refreshToken` guard,
whose self-heal branch only fires while testStatus is empty or "active" — the
row is already "expired", so checkConnection returns silently and the retry
budget is never spent. The #11414 regression test missed this because it drives
a synthetic provider that is not in ROTATING_REFRESH_PROVIDERS.

- Claude opts out of clearing the rotating refresh token on the unrecoverable
  path, for the same reason #3679 preserves it for non-rotating providers: it is
  the user's only recovery artifact. Codex and the other rotating providers keep
  clearing their genuinely consumed one-time-use tokens.
- CredentialHealth's sweep now honors the refresh circuit and parks the next
  attempt on the circuit deadline instead of re-probing a connection whose token
  refresh is already backing off.
- isInRefreshBackoff moves to src/lib/tokenRefreshCircuit.ts so CredentialHealth
  can use it without importing tokenHealthCheck's auto-starting scheduler;
  tokenHealthCheck re-exports it for existing callers.

Regression test drives the real `claude` provider through two consecutive
sweeps: without the fix the token is gone after the first and the second sweep
never spends retry 2. A Codex case guards the unchanged behavior.

Closes #13183
This commit is contained in:
Ravi Tharuma
2026-09-10 13:55:14 -03:00
committed by diegosouzapw
parent 81bf3cc36e
commit 1683d57749
4 changed files with 280 additions and 7 deletions

View File

@@ -27,6 +27,7 @@ import {
isCredentialProbeInconclusive,
resolveInconclusiveProbeRecheckDelayMs,
} from "@/lib/credentialHealth/probePolicy";
import { isInRefreshBackoff } from "@/lib/tokenRefreshCircuit";
import { emit } from "@/lib/events/eventBus";
import { isAutomatedTestProcess } from "@/shared/utils/testProcess";
import { SEARCH_VALIDATOR_CONFIGS } from "@/lib/providers/validation/searchProviders";
@@ -331,6 +332,7 @@ export async function sweep(): Promise<void> {
provider: string;
authType?: string;
healthCheckInterval?: number | null;
providerSpecificData?: { refreshCircuit?: { until?: string } } | null;
}>;
try {
@@ -348,6 +350,7 @@ export async function sweep(): Promise<void> {
provider: string;
authType?: string;
healthCheckInterval?: number | null;
providerSpecificData?: { refreshCircuit?: { until?: string } } | null;
}>;
} catch (err) {
console.error(LOG_PREFIX, "Failed to load provider connections:", err);
@@ -364,6 +367,20 @@ export async function sweep(): Promise<void> {
// Per-connection opt-out: never tested.
if (intervalMs === null) return false;
const state_ = getSchedulerState();
// Honor the OAuth refresh circuit (#13183): probing a connection whose token
// refresh is already in backoff just re-reports the same failure every sweep
// and keeps the dashboard red until the window expires or the user re-auths.
// Park the next attempt on the circuit's own deadline instead.
if (isInRefreshBackoff(conn, now)) {
const untilMs = new Date(
String(conn.providerSpecificData?.refreshCircuit?.until)
).getTime();
state_.perConnTiming.set(conn.id, {
lastAttemptAt: state_.perConnTiming.get(conn.id)?.lastAttemptAt ?? now,
nextAttemptAt: untilMs,
});
return false;
}
const timing = state_.perConnTiming.get(conn.id);
// No timing entry = never tested since boot → due now
if (!timing) return true;

View File

@@ -30,6 +30,10 @@ import {
checkWebCookieConnectionIfNeeded,
isWebCookieHealthProbeCandidate,
} from "@/lib/tokenHealthCheckWebCookie";
import {
isInRefreshBackoff,
preservesRefreshTokenOnUnrecoverable,
} from "@/lib/tokenRefreshCircuit";
const LOG_PREFIX = "[HealthCheck]";
const TRUE_ENV_VALUES = new Set(["1", "true", "yes", "on"]);
@@ -173,12 +177,10 @@ export function getRefreshBackoffUntil(streak: number, now: string): string {
return new Date(new Date(now).getTime() + backoffMin * 60 * 1000).toISOString();
}
export function isInRefreshBackoff(conn: any, nowMs: number): boolean {
const until = conn?.providerSpecificData?.refreshCircuit?.until;
if (typeof until !== "string") return false;
const untilMs = new Date(until).getTime();
return Number.isFinite(untilMs) && untilMs > nowMs;
}
// Both live in `@/lib/tokenRefreshCircuit` so CredentialHealth can import them
// without pulling this module's auto-starting scheduler. Re-exported for
// existing callers and tests.
export { isInRefreshBackoff, preservesRefreshTokenOnUnrecoverable };
export function buildRefreshFailureUpdate(
conn: any,
@@ -1126,7 +1128,11 @@ export async function checkConnection(conn) {
// gemini) the stored refresh_token is the user's only recovery
// artifact — nulling it caused #3679 (the connection reports "No valid refresh
// token available" and can never recover even after re-activation). Preserve it.
...(isRotatingProvider ? { refreshToken: null } : {}),
// PRESERVE_REFRESH_TOKEN_PROVIDERS (Claude) opt out too: nulling on the first
// failure makes the #11414 retry budget above unreachable (#13183).
...(isRotatingProvider && !preservesRefreshTokenOnUnrecoverable(conn.provider)
? { refreshToken: null }
: {}),
});
logError(
`${LOG_PREFIX}${conn.provider}/${getConnectionLogLabel(conn)}` +

View File

@@ -0,0 +1,40 @@
/**
* Shared refresh-policy helpers for Token Health Check + CredentialHealth.
*
* Kept tiny and dependency-free on purpose: CredentialHealth's sweep needs to
* honor the OAuth refresh backoff window, but importing `tokenHealthCheck`
* would pull in its module-level scheduler (which auto-starts timers).
*
* Moved verbatim out of `src/lib/tokenHealthCheck.ts`, which re-exports it for
* existing callers and tests.
*/
export function isInRefreshBackoff(conn: any, nowMs: number): boolean {
const until = conn?.providerSpecificData?.refreshCircuit?.until;
if (typeof until !== "string") return false;
const untilMs = new Date(until).getTime();
return Number.isFinite(untilMs) && untilMs > nowMs;
}
/**
* Rotating-refresh providers whose refresh token must survive an "unrecoverable"
* refresh error instead of being nulled on the first failure.
*
* #11414 keeps the connection active for EXPIRED_RETRY_MAX retries, but the same
* update also nulled the refresh token for every rotating provider. The next sweep
* then hits the `!conn.refreshToken` guard, whose self-heal branch only fires while
* testStatus is empty or "active" — the row is already "expired", so the sweep
* returns silently and the retry budget is never spent. The connection stays active,
* expired and unrecoverable until a manual re-auth (#13183).
*
* Claude access tokens are short-lived (~8h) and an invalid_grant /
* refresh_token_reused is frequently a dual-consumer race (the same Claude Max
* account refreshed by another OAuth client), not a confirmed revoke — so the token
* is worth keeping for the retries. Same reasoning as #3679 for non-rotating
* providers: the stored refresh token is the user's only recovery artifact.
*/
const PRESERVE_REFRESH_TOKEN_PROVIDERS = new Set(["claude"]);
export function preservesRefreshTokenOnUnrecoverable(provider: unknown): boolean {
return PRESERVE_REFRESH_TOKEN_PROVIDERS.has(String(provider || "").toLowerCase());
}