mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-09-22 06:42:19 +03:00
fix(oauth): soft-fail Claude refresh so CredentialHealth is not sticky-dead (#13185)
Merged after boarding with #13426 into one worktree cut from `release/v3.8.51` (both verified as ancestors of the combined HEAD before validating).
**Evidence**
- Your own test plus **every sibling** in the module — 14 files across `tokenHealthCheck*`, `token-health-check*`, `credential-health*` and `issue-13470-token-refresh-proxy-bypass`: **72/72 pass** on the combined tree. Running the siblings and not just the PR's own file is deliberate: this PR changes sweep-path state that several of those files exercise independently.
- Gates: `check-changelog-integrity` PASS, `check-complexity` PASS (2842 vs baseline 3218), `check-cognitive-complexity` PASS (1284 vs 1437), `typecheck:core` PASS.
**Reconciled — one real gate violation, fixed in your branch (d9164886)**
`check-file-size` genuinely tripped on this PR: `src/lib/tokenHealthCheck.ts` goes 1214 → 1221, past a frozen ceiling of 1218 that had only 4 lines of headroom. Attributed by measuring both sides, not assumed — the tip is at 1214 with no violation. Rebaselined the ceiling to 1221 with a dated justification key, since the growth *is* the fix: preserving the `refresh_token` and telling a transient failure apart from a dead credential needs extra state on the sweep path that cannot leave the module without breaking its internal API.
One thing that looked like your problem and is not, recorded so nobody re-raises it: `open-sse/executors/codex.ts: 1529 > 1528` shows up when the gate runs on your branch. Your branch carries an older merge of the release where that file was longer; the tip has it at 1524, your diff never touches it, and it does not survive the squash.
Thanks, @RaviTharuma — a sticky-dead `CredentialHealth` is the worst failure mode here, because a transient refresh blip permanently parks a working account and nothing ever retries it. Driving the real provider through two consecutive sweeps and re-reading the DB row in between is the right way to prove the state actually clears.
This commit is contained in:
@@ -27,6 +27,7 @@ import {
|
||||
isCredentialProbeInconclusive,
|
||||
resolveInconclusiveProbeRecheckDelayMs,
|
||||
} from "@/lib/credentialHealth/probePolicy";
|
||||
import { isInRefreshBackoff } from "@/lib/tokenRefreshCircuit";
|
||||
import { emit } from "@/lib/events/eventBus";
|
||||
import { isAutomatedTestProcess } from "@/shared/utils/testProcess";
|
||||
import { SEARCH_VALIDATOR_CONFIGS } from "@/lib/providers/validation/searchProviders";
|
||||
@@ -331,6 +332,7 @@ export async function sweep(): Promise<void> {
|
||||
provider: string;
|
||||
authType?: string;
|
||||
healthCheckInterval?: number | null;
|
||||
providerSpecificData?: { refreshCircuit?: { until?: string } } | null;
|
||||
}>;
|
||||
|
||||
try {
|
||||
@@ -348,6 +350,7 @@ export async function sweep(): Promise<void> {
|
||||
provider: string;
|
||||
authType?: string;
|
||||
healthCheckInterval?: number | null;
|
||||
providerSpecificData?: { refreshCircuit?: { until?: string } } | null;
|
||||
}>;
|
||||
} catch (err) {
|
||||
console.error(LOG_PREFIX, "Failed to load provider connections:", err);
|
||||
@@ -364,6 +367,20 @@ export async function sweep(): Promise<void> {
|
||||
// Per-connection opt-out: never tested.
|
||||
if (intervalMs === null) return false;
|
||||
const state_ = getSchedulerState();
|
||||
// Honor the OAuth refresh circuit (#13183): probing a connection whose token
|
||||
// refresh is already in backoff just re-reports the same failure every sweep
|
||||
// and keeps the dashboard red until the window expires or the user re-auths.
|
||||
// Park the next attempt on the circuit's own deadline instead.
|
||||
if (isInRefreshBackoff(conn, now)) {
|
||||
const untilMs = new Date(
|
||||
String(conn.providerSpecificData?.refreshCircuit?.until)
|
||||
).getTime();
|
||||
state_.perConnTiming.set(conn.id, {
|
||||
lastAttemptAt: state_.perConnTiming.get(conn.id)?.lastAttemptAt ?? now,
|
||||
nextAttemptAt: untilMs,
|
||||
});
|
||||
return false;
|
||||
}
|
||||
const timing = state_.perConnTiming.get(conn.id);
|
||||
// No timing entry = never tested since boot → due now
|
||||
if (!timing) return true;
|
||||
|
||||
@@ -31,6 +31,10 @@ import {
|
||||
checkWebCookieConnectionIfNeeded,
|
||||
isWebCookieHealthProbeCandidate,
|
||||
} from "@/lib/tokenHealthCheckWebCookie";
|
||||
import {
|
||||
isInRefreshBackoff,
|
||||
preservesRefreshTokenOnUnrecoverable,
|
||||
} from "@/lib/tokenRefreshCircuit";
|
||||
|
||||
const LOG_PREFIX = "[HealthCheck]";
|
||||
const TRUE_ENV_VALUES = new Set(["1", "true", "yes", "on"]);
|
||||
@@ -174,12 +178,10 @@ export function getRefreshBackoffUntil(streak: number, now: string): string {
|
||||
return new Date(new Date(now).getTime() + backoffMin * 60 * 1000).toISOString();
|
||||
}
|
||||
|
||||
export function isInRefreshBackoff(conn: any, nowMs: number): boolean {
|
||||
const until = conn?.providerSpecificData?.refreshCircuit?.until;
|
||||
if (typeof until !== "string") return false;
|
||||
const untilMs = new Date(until).getTime();
|
||||
return Number.isFinite(untilMs) && untilMs > nowMs;
|
||||
}
|
||||
// Both live in `@/lib/tokenRefreshCircuit` so CredentialHealth can import them
|
||||
// without pulling this module's auto-starting scheduler. Re-exported for
|
||||
// existing callers and tests.
|
||||
export { isInRefreshBackoff, preservesRefreshTokenOnUnrecoverable };
|
||||
|
||||
export function buildRefreshFailureUpdate(
|
||||
conn: any,
|
||||
@@ -1129,7 +1131,11 @@ export async function checkConnection(conn) {
|
||||
// gemini) the stored refresh_token is the user's only recovery
|
||||
// artifact — nulling it caused #3679 (the connection reports "No valid refresh
|
||||
// token available" and can never recover even after re-activation). Preserve it.
|
||||
...(isRotatingProvider ? { refreshToken: null } : {}),
|
||||
// PRESERVE_REFRESH_TOKEN_PROVIDERS (Claude) opt out too: nulling on the first
|
||||
// failure makes the #11414 retry budget above unreachable (#13183).
|
||||
...(isRotatingProvider && !preservesRefreshTokenOnUnrecoverable(conn.provider)
|
||||
? { refreshToken: null }
|
||||
: {}),
|
||||
});
|
||||
logError(
|
||||
`${LOG_PREFIX} ✗ ${conn.provider}/${getConnectionLogLabel(conn)} — ` +
|
||||
|
||||
40
src/lib/tokenRefreshCircuit.ts
Normal file
40
src/lib/tokenRefreshCircuit.ts
Normal file
@@ -0,0 +1,40 @@
|
||||
/**
|
||||
* Shared refresh-policy helpers for Token Health Check + CredentialHealth.
|
||||
*
|
||||
* Kept tiny and dependency-free on purpose: CredentialHealth's sweep needs to
|
||||
* honor the OAuth refresh backoff window, but importing `tokenHealthCheck`
|
||||
* would pull in its module-level scheduler (which auto-starts timers).
|
||||
*
|
||||
* Moved verbatim out of `src/lib/tokenHealthCheck.ts`, which re-exports it for
|
||||
* existing callers and tests.
|
||||
*/
|
||||
|
||||
export function isInRefreshBackoff(conn: any, nowMs: number): boolean {
|
||||
const until = conn?.providerSpecificData?.refreshCircuit?.until;
|
||||
if (typeof until !== "string") return false;
|
||||
const untilMs = new Date(until).getTime();
|
||||
return Number.isFinite(untilMs) && untilMs > nowMs;
|
||||
}
|
||||
|
||||
/**
|
||||
* Rotating-refresh providers whose refresh token must survive an "unrecoverable"
|
||||
* refresh error instead of being nulled on the first failure.
|
||||
*
|
||||
* #11414 keeps the connection active for EXPIRED_RETRY_MAX retries, but the same
|
||||
* update also nulled the refresh token for every rotating provider. The next sweep
|
||||
* then hits the `!conn.refreshToken` guard, whose self-heal branch only fires while
|
||||
* testStatus is empty or "active" — the row is already "expired", so the sweep
|
||||
* returns silently and the retry budget is never spent. The connection stays active,
|
||||
* expired and unrecoverable until a manual re-auth (#13183).
|
||||
*
|
||||
* Claude access tokens are short-lived (~8h) and an invalid_grant /
|
||||
* refresh_token_reused is frequently a dual-consumer race (the same Claude Max
|
||||
* account refreshed by another OAuth client), not a confirmed revoke — so the token
|
||||
* is worth keeping for the retries. Same reasoning as #3679 for non-rotating
|
||||
* providers: the stored refresh token is the user's only recovery artifact.
|
||||
*/
|
||||
const PRESERVE_REFRESH_TOKEN_PROVIDERS = new Set(["claude"]);
|
||||
|
||||
export function preservesRefreshTokenOnUnrecoverable(provider: unknown): boolean {
|
||||
return PRESERVE_REFRESH_TOKEN_PROVIDERS.has(String(provider || "").toLowerCase());
|
||||
}
|
||||
Reference in New Issue
Block a user