pjandro/OmniRoute
1
0
Fork 0
mirror of https://github.com/diegosouzapw/OmniRoute.git synced 2026-08-16 20:22:21 +03:00
Code Issues Packages Projects Releases Wiki Activity

fix(executors): rotate to the next account on network throws when the account has a dedicated proxy (#10402)

Browse Source 
OpencodeExecutor and MimocodeExecutor rotated to the next account only on
HTTP 429. A network exception (timeout, connection refused/reset) on one
account instead propagated out of execute() and failed the whole request,
even when other accounts remained available.

Both executors now rotate on a network exception only when the failed
account has its own dedicated proxy (account.proxy !== null) — a dead
proxy is genuinely account-scoped, so rotating away from it is safe.
Accounts sharing the default egress (no proxy configured) trigger the
same cooldown and are skipped for the rest of the request once the shared
egress is known down, but a later account with its own dedicated proxy is
still tried normally — a throw on a proxy-less account no longer strands
a proxied account further in the rotation. This behavior is gated behind
NETWORK_ROTATION_SHARED_EGRESS_GUARD (Feature Flag, default on); disabled,
it reproduces the immediate-propagation behavior this fix started from.

The shared rotation mechanics (pickAccount/markCooldown/markSuccess) are
extracted into executors/accountRotation.ts, used by both executors —
they had independently implemented the same round-robin+cooldown
skeleton. This also fixes an identical, pre-existing bug in
MimocodeExecutor that predates this PR: its catch block called
markCooldown unconditionally on any throw, with no proxy check and no
warn log (a silent exception swallow on a path that influences the
result).

The cooldown formula for both the proxy and shared-egress cases reuses
the repo's already-established "transient, not clearly attributable"
constants (errorConfig.ts TRANSIENT_COOLDOWN_MS/COOLDOWN_MS.transientMax,
already used by accountFallback.ts for network-error classification)
instead of introducing a separate value.

MimocodeExecutor's network-error 502 body also now goes through
buildErrorBody()/sanitizeErrorMessage() instead of embedding the raw
caught error message directly (Hard Rule #12), matching the sanitization
already used on its #2101 malformed-request path.

Validated by TDD (Hard Rule #18): tests/unit/account-rotation.test.ts
covers the shared module directly; opencode-proxy-rotation-4954.test.ts
and mimocode-executor.test.ts cover the proxy-configured rotation path,
the mixed-fleet case, the shared-egress single-network-call case, and the
NETWORK_ROTATION_SHARED_EGRESS_GUARD-disabled legacy path, for each
executor. tsc, lint, and the provider golden-path gates
(check:provider-consistency, check:provider-assets,
provider-translate-path-golden.test.ts) are clean on all touched files.

Co-authored-by: Max <maxmad64@gmail.com>
This commit is contained in:
Dizzle
2026-08-16 05:16:04 +02:00
committed by GitHub
parent 6e97fbf340
commit 94cf4c402a
10 changed files with 918 additions and 103 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
changelog.d/fixes/10393-opencode-rotate-network-throw.md Normal file
View File

@@ -0,0 +1 @@
- **fix(executors):** OpencodeExecutor and MimocodeExecutor now rotate to the next account on network exceptions (timeout, connection refused/reset) when the failed account has a dedicated proxy, not only on 429 — a throw on one account no longer fails the whole request when other accounts remain. Accounts sharing the default egress (no proxy) fail fast instead of retrying the same outage against every account. The shared rotation mechanics (`pickAccount`/`markCooldown`/`markSuccess`) are now extracted into `accountRotation.ts`, fixing an identical unconditional-cooldown gap that pre-dated this PR in MimocodeExecutor ([#10393](https://github.com/diegosouzapw/OmniRoute/pull/10393))

109
open-sse/executors/accountRotation.ts Normal file
View File

@@ -0,0 +1,109 @@
/**
* Shared multi-account rotation mechanics for noauth executors that round-robin
* across several "accounts" (fingerprints), each with an optional dedicated
* proxy — currently `OpencodeExecutor` and `MimocodeExecutor`.
*
* Extracted after both executors independently implemented the same
* pickAccount/markCooldown/markSuccess skeleton with the same exponential
* backoff, and independently needed the same fix for the same latent bug (a
* network exception was treated as account-scoped rotation fodder even for
* accounts sharing the default egress — see `isNetworkErrorRotatable`).
*/
// Reuses the repo's established "transient, not clearly attributable" failure
// cooldown (already used by accountFallback.ts for network-error dedup, see
// its "one transient blip opens the whole-provider breaker" comment) instead
// of inventing a separate constant — same magnitude the codebase already
// applies whether the failure is a 429 or a network-level throw.
import { TRANSIENT_COOLDOWN_MS, COOLDOWN_MS } from "../config/errorConfig.ts";
/** Per-account proxy configuration, persisted by NoAuthAccountCard under
* `providerSpecificData.accountProxies` (keyed by the account id, which the UI
* stores in `providerSpecificData.fingerprints`). */
export interface AccountProxyConfig {
fingerprint: string;
proxy: {
type: string;
host: string;
port: number;
username?: string;
password?: string;
relayAuth?: string;
} | null;
}
/** The subset of per-account state the rotation mechanics need. Executors may
* carry additional fields (e.g. mimocode's `jwt`/`expiresAt`) — this is the
* minimum shape `pickAccount`/`markCooldown`/`markSuccess` operate on. */
export interface RotatableAccount {
fingerprint: string;
cooldownUntil: number;
consecutiveFails: number;
proxy: AccountProxyConfig["proxy"];
}
const COOLDOWN_BASE_MS = TRANSIENT_COOLDOWN_MS;
const COOLDOWN_MAX_MS = COOLDOWN_MS.transientMax;
export function isAccountReady(account: RotatableAccount): boolean {
return account.cooldownUntil <= Date.now();
}
/** Round-robin pick, skipping accounts not `isReady`; falls back to the next
* index (even if not ready) so a caller always gets an account rather than
* hanging when every account is unavailable. Mutates `state.nextAccountIdx`.
*
* `isReady` defaults to the plain cooldown check (`isAccountReady`); pass a
* custom predicate when readiness depends on more than cooldown (e.g.
* mimocode's JWT-freshness-aware variant). */
export function pickAccount<T extends RotatableAccount>(
accounts: T[],
state: { nextAccountIdx: number },
isReady: (account: T) => boolean = isAccountReady
): T {
for (let i = 0; i < accounts.length; i++) {
const idx = (state.nextAccountIdx + i) % accounts.length;
const acct = accounts[idx];
if (isReady(acct)) {
state.nextAccountIdx = (idx + 1) % accounts.length;
return acct;
}
}
const fallbackIdx = state.nextAccountIdx % accounts.length;
state.nextAccountIdx = (state.nextAccountIdx + 1) % accounts.length;
return accounts[fallbackIdx];
}
export function markCooldown(account: RotatableAccount): void {
account.consecutiveFails++;
const backoff = Math.min(
COOLDOWN_BASE_MS * Math.pow(2, account.consecutiveFails - 1),
COOLDOWN_MAX_MS
);
account.cooldownUntil = Date.now() + backoff + Math.random() * 1000;
}
export function markSuccess(account: RotatableAccount): void {
account.consecutiveFails = 0;
}
/** Mask an account id for logs (UI calls it a fingerprint). */
export function maskAccountId(fingerprint: string): string {
if (!fingerprint) return "direct";
return `${fingerprint.slice(0, 8)}…`;
}
/**
* Whether a network exception (timeout, connection refused/reset) on this
* account should trigger rotation to the next account, vs propagating.
*
* Only true when the account has its own egress (a configured proxy) — that's
* the case a dead/unreachable proxy genuinely justifies rotating away from.
* Accounts sharing the default egress (no proxy) can all fail at once on a
* real network outage: rotating there would just retry the same failure
* against every account while poisoning each one's cooldown for a cause that
* isn't theirs.
*/
export function isNetworkErrorRotatable(account: RotatableAccount): boolean {
return account.proxy !== null;
}

122
open-sse/executors/mimocode.ts
View File

@@ -27,13 +27,21 @@ import { createProxyDispatcher } from "../utils/proxyDispatcher.ts";
import { RATE_LIMIT_TEXT_PATTERNS } from "../services/accountFallback.ts";
import { buildErrorBody, sanitizeErrorMessage } from "../utils/error.ts";
import { fetch as undiciFetch, type Dispatcher } from "undici";
import {
type AccountProxyConfig as SharedAccountProxyConfig,
type RotatableAccount,
pickAccount as pickRotatableAccount,
markCooldown as markAccountCooldown,
markSuccess as markAccountSuccess,
maskAccountId,
isNetworkErrorRotatable,
} from "./accountRotation.ts";
import { isNetworkRotationSharedEgressGuardEnabled } from "@/shared/utils/featureFlags";
const BOOTSTRAP_PATH = "/api/free-ai/bootstrap";
const CHAT_PATH = "/api/free-ai/openai/chat";
const JWT_REFRESH_BUFFER_MS = 5 * 60 * 1000;
const BOOTSTRAP_TIMEOUT_MS = 15_000;
const COOLDOWN_BASE_MS = 5_000;
const COOLDOWN_MAX_MS = 60_000;
const MIMO_SOURCE = "mimocode-cli-free";
@@ -82,24 +90,12 @@ const USER_AGENTS = [
// ── Account State ──────────────────────────────────────────────────────────
/** Per-account proxy configuration, passed through providerSpecificData.accountProxies. */
export interface AccountProxyConfig {
fingerprint: string;
proxy: {
type: string;
host: string;
port: number;
username?: string;
password?: string;
relayAuth?: string;
} | null;
}
export type AccountProxyConfig = SharedAccountProxyConfig;
interface AccountState {
interface AccountState extends RotatableAccount {
fingerprint: string;
jwt: string;
expiresAt: number;
cooldownUntil: number;
consecutiveFails: number;
/**
* #3837/#5521: the account's resolved proxy, or `null` when none is configured.
* Always present (never `undefined`) so callers can read `acct.proxy` directly —
@@ -223,7 +219,10 @@ function rewriteModelName(model: string): string {
export class MimocodeExecutor extends BaseExecutor {
private accounts: AccountState[] = [];
private nextAccountIdx = 0;
// Not `private`: passed as the mutable rotation cursor to the shared
// pickAccount() helper, which needs a plain `{ nextAccountIdx }` shape —
// TS's private-member nominal check rejects `this` there otherwise.
nextAccountIdx = 0;
private baseUrl: string;
private proxyUrlMap = new Map<string, string>();
private static encoder = new TextEncoder();
@@ -342,30 +341,15 @@ export class MimocodeExecutor extends BaseExecutor {
}
private pickAccount(): AccountState {
for (let i = 0; i < this.accounts.length; i++) {
const idx = (this.nextAccountIdx + i) % this.accounts.length;
const acct = this.accounts[idx];
if (isAccountReady(acct)) {
this.nextAccountIdx = (idx + 1) % this.accounts.length;
return acct;
}
}
const fallbackIdx = this.nextAccountIdx % this.accounts.length;
this.nextAccountIdx = (this.nextAccountIdx + 1) % this.accounts.length;
return this.accounts[fallbackIdx];
return pickRotatableAccount(this.accounts, this, isAccountReady);
}
private markCooldown(account: AccountState): void {
account.consecutiveFails++;
const backoff = Math.min(
COOLDOWN_BASE_MS * Math.pow(2, account.consecutiveFails - 1),
COOLDOWN_MAX_MS
);
account.cooldownUntil = Date.now() + backoff + Math.random() * 1000;
markAccountCooldown(account);
}
private markSuccess(account: AccountState): void {
account.consecutiveFails = 0;
markAccountSuccess(account);
}
/**
@@ -592,9 +576,25 @@ export class MimocodeExecutor extends BaseExecutor {
this.syncAccountsFromCredentials(input.credentials);
const sharedEgressGuardEnabled = isNetworkRotationSharedEgressGuardEnabled();
// Set once a proxy-less account's network throw reveals the shared egress
// is down — subsequent proxy-less accounts this request are skipped
// without a network call, but proxied accounts (independent egress) are
// still tried normally. See NETWORK_ROTATION_SHARED_EGRESS_GUARD.
let sharedEgressDown = false;
// Try each account, skip cooldown ones
for (let attempt = 0; attempt < this.accounts.length; attempt++) {
const account = this.pickAccount();
if (sharedEgressGuardEnabled && sharedEgressDown && !account.proxy) {
log?.warn?.(
"MIMOCODE",
`skipping account ${maskAccountId(account.fingerprint)} (no dedicated proxy, shared egress already down this request)`
);
continue;
}
try {
const headers = this.buildHeaders(input.credentials, stream);
const resp = await this.fetchWithAuthRetry(url, headers, reqBody, signal, account, log);
@@ -623,16 +623,60 @@ export class MimocodeExecutor extends BaseExecutor {
transformedBody: reqBody,
};
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
const masked = maskAccountId(account.fingerprint);
// Mirrors OpencodeExecutor's rotation guard: a network exception is only account-scoped
// when this account has its OWN egress (a configured proxy). Without
// one, accounts share the default egress — the failure isn't
// attributable to this account, and trying the next one would just
// retry the same outage while poisoning its cooldown for a cause
// that isn't theirs. Fail fast instead of exhausting every account.
if (!isNetworkErrorRotatable(account)) {
if (sharedEgressGuardEnabled) {
this.markCooldown(account);
sharedEgressDown = true;
log?.warn?.(
"MIMOCODE",
`network error on account ${masked} (no dedicated proxy, shared egress), cooldown applied — trying next available account… (${msg})`
);
continue;
}
log?.warn?.(
"MIMOCODE",
`network error on account ${masked} (no dedicated proxy, shared egress) — not rotating (${msg})`
);
return {
response: new Response(
encoder.encode(
JSON.stringify(
buildErrorBody(502, msg, undefined, {
type: "upstream_error",
code: "EXECUTOR_ERROR",
})
)
),
{ status: 502, headers: { "Content-Type": "application/json" } }
),
url,
headers: this.buildHeaders(input.credentials, stream),
transformedBody: body,
};
}
this.markCooldown(account);
log?.warn?.("MIMOCODE", `network error on account ${masked}, rotating to next… (${msg})`);
if (attempt === this.accounts.length - 1) {
const msg = err instanceof Error ? err.message : String(err);
log?.error?.("MIMOCODE", `Executor error: ${msg}`);
return {
response: new Response(
encoder.encode(
JSON.stringify({
error: { message: msg, type: "upstream_error", code: "EXECUTOR_ERROR" },
})
JSON.stringify(
buildErrorBody(502, msg, undefined, {
type: "upstream_error",
code: "EXECUTOR_ERROR",
})
)
),
{ status: 502, headers: { "Content-Type": "application/json" } }
),

148
open-sse/executors/opencode.ts
View File

@@ -7,37 +7,30 @@ import {
} from "../utils/reasoningContentInjector.ts";
import { runWithProxyContext } from "../utils/proxyFetch.ts";
import { forwardOpencodeClientHeaders } from "../utils/opencodeHeaders.ts";
import {
type AccountProxyConfig,
type RotatableAccount,
pickAccount as pickRotatableAccount,
markCooldown as markAccountCooldown,
markSuccess as markAccountSuccess,
maskAccountId,
isNetworkErrorRotatable,
} from "./accountRotation.ts";
import { isNetworkRotationSharedEgressGuardEnabled } from "@/shared/utils/featureFlags";
/**
* Per-account proxy configuration, persisted by NoAuthAccountCard under
* `providerSpecificData.accountProxies` (keyed by the account id, which the UI
* stores in `providerSpecificData.fingerprints`). Same shape mimocode uses.
*/
export interface OpencodeAccountProxyConfig {
fingerprint: string;
proxy: {
type: string;
host: string;
port: number;
username?: string;
password?: string;
relayAuth?: string;
} | null;
}
export type OpencodeAccountProxyConfig = AccountProxyConfig;
/** Runtime rotation/cooldown state for one "OpenCode Free" account. */
interface OpencodeAccountState {
interface OpencodeAccountState extends RotatableAccount {
/** Account id (UI: providerSpecificData.fingerprints[i]); "" for the default direct account. */
fingerprint: string;
cooldownUntil: number;
consecutiveFails: number;
/** Resolved proxy config for this account (null = direct egress). */
proxy: OpencodeAccountProxyConfig["proxy"];
}
const OPENCODE_COOLDOWN_BASE_MS = 5_000;
const OPENCODE_COOLDOWN_MAX_MS = 60_000;
const EFFORT_LEVELS = ["low", "medium", "high", "max"] as const;
/**
@@ -147,7 +140,10 @@ export class OpencodeExecutor extends BaseExecutor {
private accounts: OpencodeAccountState[] = [
{ fingerprint: "", cooldownUntil: 0, consecutiveFails: 0, proxy: null },
];
private nextAccountIdx = 0;
// Not `private`: passed as the mutable rotation cursor to the shared
// pickAccount() helper, which needs a plain `{ nextAccountIdx }` shape —
// TS's private-member nominal check rejects `this` there otherwise.
nextAccountIdx = 0;
constructor(provider: string) {
super(provider, PROVIDERS[provider] || PROVIDERS.openai);
@@ -190,42 +186,17 @@ export class OpencodeExecutor extends BaseExecutor {
if (this.nextAccountIdx >= this.accounts.length) this.nextAccountIdx = 0;
}
private isAccountReady(account: OpencodeAccountState): boolean {
return account.cooldownUntil <= Date.now();
}
/** Round-robin pick, skipping accounts in cooldown; falls back to the next index. */
private pickAccount(): OpencodeAccountState {
for (let i = 0; i < this.accounts.length; i++) {
const idx = (this.nextAccountIdx + i) % this.accounts.length;
const acct = this.accounts[idx];
if (this.isAccountReady(acct)) {
this.nextAccountIdx = (idx + 1) % this.accounts.length;
return acct;
}
}
const fallbackIdx = this.nextAccountIdx % this.accounts.length;
this.nextAccountIdx = (this.nextAccountIdx + 1) % this.accounts.length;
return this.accounts[fallbackIdx];
return pickRotatableAccount(this.accounts, this);
}
private markCooldown(account: OpencodeAccountState): void {
account.consecutiveFails++;
const backoff = Math.min(
OPENCODE_COOLDOWN_BASE_MS * Math.pow(2, account.consecutiveFails - 1),
OPENCODE_COOLDOWN_MAX_MS
);
account.cooldownUntil = Date.now() + backoff + Math.random() * 1000;
markAccountCooldown(account);
}
private markSuccess(account: OpencodeAccountState): void {
account.consecutiveFails = 0;
}
/** Mask an account id for logs (UI calls it a fingerprint). */
private static maskAccountId(fingerprint: string): string {
if (!fingerprint) return "direct";
return `${fingerprint.slice(0, 8)}…`;
markAccountSuccess(account);
}
async execute(input: ExecuteInput) {
@@ -267,11 +238,35 @@ export class OpencodeExecutor extends BaseExecutor {
}
const { log } = input;
let lastResult: Awaited<ReturnType<BaseExecutor["execute"]>> | null = null;
// This loop only ever dispatches through super.execute() (the HTTP request
// path), which always resolves the object-shaped arm of ExecutorExecuteResult
// — the bare-Response arm belongs to web/scraping executors only (base.ts:290).
type HttpExecuteResult = Extract<
Awaited<ReturnType<BaseExecutor["execute"]>>,
{ response: Response }
>;
let lastResult: HttpExecuteResult | null = null;
let lastSharedEgressError: unknown = null;
const sharedEgressGuardEnabled = isNetworkRotationSharedEgressGuardEnabled();
// Set once a proxy-less account's network throw reveals the shared
// egress is down (see NETWORK_ROTATION_SHARED_EGRESS_GUARD below) —
// subsequent proxy-less accounts this request are skipped without a
// network call, but proxied accounts (independent egress) are still
// tried normally.
let sharedEgressDown = false;
for (let attempt = 0; attempt < this.accounts.length; attempt++) {
const account = this.pickAccount();
const masked = OpencodeExecutor.maskAccountId(account.fingerprint);
const masked = maskAccountId(account.fingerprint);
if (sharedEgressGuardEnabled && sharedEgressDown && !account.proxy) {
log?.warn?.(
"OPENCODE",
`skipping account ${masked} (no dedicated proxy, shared egress already down this request)`
);
continue;
}
// #5217 (Gap 2): promoted debug→info so the per-request account/proxy
// rotation selection is visible in the Console log view at the default
// APP_LOG_LEVEL=info (users could not see which account/proxy was used).
@@ -287,9 +282,46 @@ export class OpencodeExecutor extends BaseExecutor {
// Pin egress to this account's proxy for the whole BaseExecutor dispatch
// (incl. its intra-URL 429 retries). skipUpstreamRetry lets THIS loop own
// the cross-account 429 fallback instead of BaseExecutor's same-key retry.
const result = await runWithProxyContext(account.proxy, () =>
super.execute({ ...input, skipUpstreamRetry: true })
);
let result: HttpExecuteResult;
try {
// super.execute() here always dispatches the HTTP path (opencode is an
// OpenAI-compatible API, never the web/scraping bare-Response arm) —
// see base.ts:290-294.
result = (await runWithProxyContext(account.proxy, () =>
super.execute({ ...input, skipUpstreamRetry: true })
)) as HttpExecuteResult;
} catch (err) {
const reason = err instanceof Error ? err.message : String(err);
// A network exception (timeout, connection refused/reset) is only
// account-scoped when this account has its OWN egress (a configured
// proxy) — that's the case a dead/unreachable proxy justifies rotating
// away from. Without a proxy, accounts share the same network egress:
// the failure isn't attributable to this account. Never swallowed
// silently either way: logged before rotating, skipping, or rethrowing.
if (!isNetworkErrorRotatable(account)) {
if (sharedEgressGuardEnabled) {
this.markCooldown(account);
sharedEgressDown = true;
lastSharedEgressError = err;
log?.warn?.(
"OPENCODE",
`network error on account ${masked} (no dedicated proxy, shared egress), cooldown applied — trying next available account… (${reason})`
);
continue;
}
log?.warn?.(
"OPENCODE",
`network error on account ${masked} (no dedicated proxy, shared egress) — not rotating (${reason})`
);
throw err;
}
this.markCooldown(account);
log?.warn?.(
"OPENCODE",
`network error on account ${masked}, rotating to next… (${reason})`
);
continue;
}
lastResult = result;
const status = result.response.status;
@@ -303,6 +335,16 @@ export class OpencodeExecutor extends BaseExecutor {
return result;
}
// The loop exhausted without a result. If it's because every remaining
// proxy-less account was skipped once the shared egress was known down
// (rather than actually tried), propagate that original throw — an
// extra direct call here would just be a second doomed attempt against
// the same dead path, which is exactly the latency this guard exists
// to avoid (see NETWORK_ROTATION_SHARED_EGRESS_GUARD).
if (sharedEgressDown && !lastResult && lastSharedEgressError !== null) {
throw lastSharedEgressError;
}
// All accounts returned 429 (or errored) — surface the last response.
return lastResult ?? (await super.execute(input));
} finally {

12
src/shared/constants/featureFlagDefinitions.ts
View File

@@ -164,6 +164,18 @@ export const FEATURE_FLAG_DEFINITIONS: FeatureFlagDefinition[] = [
requiresRestart: false,
warningLevel: "danger",
},
{
key: "NETWORK_ROTATION_SHARED_EGRESS_GUARD",
label: "Network Rotation Shared-Egress Guard",
description:
"On a network exception (timeout, connection refused/reset) for a multi-account rotation executor, when the failing account has no dedicated proxy, apply a short cooldown and skip other proxy-less accounts for the rest of the request instead of retrying each one. On by default (safe: no egress IP change, only reduces latency/cooldown risk on shared-egress accounts). Disable to restore immediate propagation on the first proxy-less throw.",
descriptionI18nKey: "featureFlagNetworkRotationSharedEgressGuardDescription",
category: "network",
defaultValue: "true",
type: "boolean",
requiresRestart: false,
warningLevel: "info",
},
{
key: "MITM_DISABLE_TLS_VERIFY",
label: "Disable TLS Verify (MITM)",

12
src/shared/utils/featureFlags.ts
View File

@@ -111,3 +111,15 @@ export function isControlPlaneProxyDirectFallbackEnabled(): boolean {
return false;
}
}
export function isNetworkRotationSharedEgressGuardEnabled(): boolean {
try {
return isFeatureFlagEnabled("NETWORK_ROTATION_SHARED_EGRESS_GUARD");
} catch (error) {
console.error(
"[featureFlags] Failed to resolve NETWORK_ROTATION_SHARED_EGRESS_GUARD, defaulting to enabled:",
error instanceof Error ? error.message : error
);
return true;
}
}

116
tests/unit/account-rotation.test.ts Normal file
View File

@@ -0,0 +1,116 @@
import { describe, it } from "node:test";
import assert from "node:assert";
import {
isAccountReady,
pickAccount,
markCooldown,
markSuccess,
maskAccountId,
isNetworkErrorRotatable,
type RotatableAccount,
} from "../../open-sse/executors/accountRotation.ts";
function account(overrides: Partial<RotatableAccount> = {}): RotatableAccount {
return {
fingerprint: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
cooldownUntil: 0,
consecutiveFails: 0,
proxy: null,
...overrides,
};
}
describe("accountRotation", () => {
it("isAccountReady is true when cooldownUntil is in the past", () => {
assert.strictEqual(isAccountReady(account({ cooldownUntil: Date.now() - 1000 })), true);
});
it("isAccountReady is false when cooldownUntil is in the future", () => {
assert.strictEqual(isAccountReady(account({ cooldownUntil: Date.now() + 60_000 })), false);
});
it("markCooldown increments consecutiveFails and sets a future cooldownUntil", () => {
const acct = account();
markCooldown(acct);
assert.strictEqual(acct.consecutiveFails, 1);
assert.ok(acct.cooldownUntil > Date.now());
});
it("markCooldown backs off exponentially with consecutive failures", () => {
const acct = account();
markCooldown(acct);
const firstCooldown = acct.cooldownUntil;
markCooldown(acct);
assert.strictEqual(acct.consecutiveFails, 2);
// Second backoff (base*2^1) must be strictly larger than the first
// (base*2^0), modulo the shared jitter window — compare the floor.
assert.ok(acct.cooldownUntil - Date.now() > firstCooldown - Date.now() - 1000);
});
it("markCooldown uses the same magnitude regardless of why it was called (429 or network throw)", () => {
// No `short`/severity parameter: proxy-attributable failures (429, dead
// proxy) and shared-egress network throws use the identical formula —
// the repo's own established "transient, not clearly attributable"
// cooldown (errorConfig.ts TRANSIENT_COOLDOWN_MS/transientMax) already
// covers both cases at the same magnitude. The behavioral fix for
// shared-egress accounts lives in the caller's skip logic, not here.
const a = account();
const b = account();
markCooldown(a);
markCooldown(b);
// Both draw from the same base backoff ± up to 1s jitter — same formula,
// no separate "short" magnitude for either call site.
assert.ok(
Math.abs(a.cooldownUntil - b.cooldownUntil) <= 1000,
"same account state must produce cooldowns within the shared jitter window"
);
});
it("markSuccess resets consecutiveFails to 0", () => {
const acct = account({ consecutiveFails: 5 });
markSuccess(acct);
assert.strictEqual(acct.consecutiveFails, 0);
});
it("maskAccountId masks a real fingerprint to its first 8 chars + ellipsis", () => {
assert.strictEqual(maskAccountId("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"), "aaaaaaaa…");
});
it("maskAccountId reports the empty/default fingerprint as 'direct'", () => {
assert.strictEqual(maskAccountId(""), "direct");
});
it("pickAccount skips accounts in cooldown and rotates nextAccountIdx", () => {
const a = account({ fingerprint: "a", cooldownUntil: Date.now() + 60_000 });
const b = account({ fingerprint: "b", cooldownUntil: 0 });
const state = { nextAccountIdx: 0 };
const picked = pickAccount([a, b], state);
assert.strictEqual(picked.fingerprint, "b", "must skip the account still in cooldown");
});
it("pickAccount falls back to the next index when every account is in cooldown", () => {
const a = account({ fingerprint: "a", cooldownUntil: Date.now() + 60_000 });
const b = account({ fingerprint: "b", cooldownUntil: Date.now() + 60_000 });
const state = { nextAccountIdx: 0 };
const picked = pickAccount([a, b], state);
assert.strictEqual(picked.fingerprint, "a", "must still return an account, not throw/hang");
});
it("pickAccount accepts a custom isReady predicate (e.g. JWT-freshness-aware)", () => {
const a = account({ fingerprint: "a", cooldownUntil: 0 });
const b = account({ fingerprint: "b", cooldownUntil: 0 });
const state = { nextAccountIdx: 0 };
// Custom predicate rejects "a" for a reason cooldown alone wouldn't catch.
const picked = pickAccount([a, b], state, (acct: RotatableAccount) => acct.fingerprint !== "a");
assert.strictEqual(picked.fingerprint, "b");
});
it("isNetworkErrorRotatable is true only when the account has a configured proxy", () => {
const withProxy = account({
proxy: { type: "http", host: "127.0.0.1", port: 8080 },
});
const withoutProxy = account({ proxy: null });
assert.strictEqual(isNetworkErrorRotatable(withProxy), true);
assert.strictEqual(isNetworkErrorRotatable(withoutProxy), false);
});
});

16
tests/unit/feature-flags-settings.test.ts
View File

@@ -1,4 +1,4 @@
import { describe, it, before, beforeEach, after } from "node:test";
import { describe, it, beforeEach, after } from "node:test";
import assert from "node:assert/strict";
import os from "node:os";
import path from "node:path";
@@ -30,7 +30,7 @@ const {
isControlPlaneProxyDirectFallbackEnabled,
} = await import("../../src/shared/utils/featureFlags.ts");
const EXPECTED_FEATURE_FLAG_COUNT = 47;
const EXPECTED_FEATURE_FLAG_COUNT = 48;
// ──────────────────────────────────────────────────────
// Test group 1 — Flag definitions registry
@@ -161,6 +161,18 @@ describe("featureFlagDefinitions", () => {
assert.strictEqual(def.warningLevel, "danger");
});
it("defines network rotation shared-egress guard as a network boolean flag enabled by default", () => {
const def = FEATURE_FLAG_DEFINITIONS.find(
(d) => d.key === "NETWORK_ROTATION_SHARED_EGRESS_GUARD"
);
assert.ok(def, "NETWORK_ROTATION_SHARED_EGRESS_GUARD should exist");
assert.strictEqual(def.category, "network");
assert.strictEqual(def.type, "boolean");
assert.strictEqual(def.defaultValue, "true");
assert.strictEqual(def.requiresRestart, false);
assert.strictEqual(def.warningLevel, "info");
});
it("defines remote audio provider nodes as a network boolean flag disabled by default", () => {
// Guards the egress default: with this on, /v1/audio/* may reach a provider node
// hosted outside localhost. It must never become an implicit default (cf. #3963).

252
tests/unit/mimocode-executor.test.ts
View File

@@ -1,4 +1,4 @@
import { describe, it } from "node:test";
import { describe, it, beforeEach, afterEach } from "node:test";
import assert from "node:assert";
import {
MimocodeExecutor,
@@ -254,8 +254,6 @@ describe("mimocode providerRegistry entry", () => {
});
describe("mimocode per-account proxy", () => {
const exec = new MimocodeExecutor();
it("AccountProxyConfig type has required fields", () => {
const config: AccountProxyConfig = {
fingerprint: "abc123",
@@ -498,6 +496,7 @@ interface TestAccountState {
expiresAt: number;
cooldownUntil: number;
consecutiveFails: number;
proxy?: unknown;
}
interface ExecutorAccountAccess {
@@ -625,3 +624,250 @@ describe("mimocode 400 classification (#2101/#4976)", () => {
}
});
});
describe("mimocode network-error rotation (parity with OpencodeExecutor)", () => {
function makeJwt(): string {
const header = Buffer.from(JSON.stringify({ alg: "none" })).toString("base64url");
const payload = Buffer.from(
JSON.stringify({ exp: Math.floor(Date.now() / 1000) + 3600 })
).toString("base64url");
return `${header}.${payload}.sig`;
}
function twoAccountExecutor(proxies: [unknown, unknown]): MimocodeExecutor {
const exec = new MimocodeExecutor();
const access = accountAccess(exec);
access.accounts = [
{
fingerprint: "acct-a",
jwt: "",
expiresAt: 0,
cooldownUntil: 0,
consecutiveFails: 0,
proxy: proxies[0],
},
{
fingerprint: "acct-b",
jwt: "",
expiresAt: 0,
cooldownUntil: 0,
consecutiveFails: 0,
proxy: proxies[1],
},
] as TestAccountState[];
access.nextAccountIdx = 0;
return exec;
}
const A_PROXY = { type: "http", host: "127.0.0.1", port: 8080 };
const B_PROXY = { type: "http", host: "127.0.0.1", port: 8081 };
it("rotates to the next account on a network throw when the failed account has a dedicated proxy", async () => {
const testExec = twoAccountExecutor([A_PROXY, B_PROXY]);
// Force both dispatch legs (bootstrap + chat) through the plain `fetch()`
// fallback instead of a real undici proxy dispatcher — this test exercises
// the rotation DECISION (account.proxy is configured → rotate), not actual
// proxy network I/O, which has its own dedicated dispatcher tests below.
(testExec as unknown as { getProxyDispatcher: () => undefined }).getProxyDispatcher = () =>
undefined;
let chatCalls = 0;
const originalFetch = globalThis.fetch;
globalThis.fetch = (async (url: unknown) => {
const urlStr = String(url);
if (urlStr.includes("/api/free-ai/bootstrap")) {
return new Response(JSON.stringify({ jwt: makeJwt() }), { status: 200 });
}
if (urlStr.includes("/api/free-ai/openai/chat")) {
chatCalls++;
if (chatCalls === 1) throw new Error("ECONNRESET");
return new Response(JSON.stringify({ id: "ok", choices: [] }), { status: 200 });
}
throw new Error(`unexpected fetch: ${urlStr}`);
}) as typeof fetch;
const warnCalls: string[] = [];
try {
const result = await testExec.execute({
model: "mimo-auto",
body: { messages: [{ role: "user", content: "hi" }], stream: false },
stream: false,
signal: null,
credentials: {
providerSpecificData: {
fingerprints: ["acct-a", "acct-b"],
accountProxies: [
{ fingerprint: "acct-a", proxy: A_PROXY },
{ fingerprint: "acct-b", proxy: B_PROXY },
],
},
},
log: {
debug: () => {},
info: () => {},
warn: (_tag: unknown, msg: string) => warnCalls.push(msg),
error: () => {},
},
});
assert.strictEqual(chatCalls, 2, "should retry on the next account after the throw");
assert.strictEqual(result.response.status, 200);
const acctA = accountAccess(testExec).accounts[0];
assert.ok(acctA.cooldownUntil > Date.now(), "account with a dedicated proxy must cool down");
assert.ok(
warnCalls.some((m) => /network error/i.test(m)),
`expected a "network error" warn log; got=${JSON.stringify(warnCalls)}`
);
} finally {
globalThis.fetch = originalFetch;
}
});
describe("NETWORK_ROTATION_SHARED_EGRESS_GUARD", () => {
const FLAG = "NETWORK_ROTATION_SHARED_EGRESS_GUARD";
let originalEnvValue: string | undefined;
beforeEach(() => {
originalEnvValue = process.env[FLAG];
});
afterEach(() => {
if (originalEnvValue === undefined) delete process.env[FLAG];
else process.env[FLAG] = originalEnvValue;
});
it("rotates to a proxied account after a proxy-less account throws (mixed fleet, guard on by default)", async () => {
const testExec = twoAccountExecutor([null, B_PROXY]);
// Force both dispatch legs through the plain `fetch()` fallback instead
// of a real undici proxy dispatcher — this test exercises the rotation
// DECISION, not actual proxy network I/O.
(testExec as unknown as { getProxyDispatcher: () => undefined }).getProxyDispatcher = () =>
undefined;
let chatCalls = 0;
const originalFetch = globalThis.fetch;
globalThis.fetch = (async (url: unknown) => {
const urlStr = String(url);
if (urlStr.includes("/api/free-ai/bootstrap")) {
return new Response(JSON.stringify({ jwt: makeJwt() }), { status: 200 });
}
if (urlStr.includes("/api/free-ai/openai/chat")) {
chatCalls++;
if (chatCalls === 1) throw new Error("ETIMEDOUT");
return new Response(JSON.stringify({ id: "ok", choices: [] }), { status: 200 });
}
throw new Error(`unexpected fetch: ${urlStr}`);
}) as typeof fetch;
try {
const result = await testExec.execute({
model: "mimo-auto",
body: { messages: [{ role: "user", content: "hi" }], stream: false },
stream: false,
signal: null,
credentials: {
providerSpecificData: {
fingerprints: ["acct-a", "acct-b"],
accountProxies: [{ fingerprint: "acct-b", proxy: B_PROXY }],
},
},
log: { debug: () => {}, info: () => {}, warn: () => {}, error: () => {} },
});
assert.strictEqual(chatCalls, 2, "the proxied account (B) must still be tried");
assert.strictEqual(result.response.status, 200);
} finally {
globalThis.fetch = originalFetch;
}
});
it("makes a single real network call when no account has a configured proxy (guard on by default)", async () => {
const testExec = twoAccountExecutor([null, null]);
let chatCalls = 0;
const originalFetch = globalThis.fetch;
globalThis.fetch = (async (url: unknown) => {
const urlStr = String(url);
if (urlStr.includes("/api/free-ai/bootstrap")) {
return new Response(JSON.stringify({ jwt: makeJwt() }), { status: 200 });
}
if (urlStr.includes("/api/free-ai/openai/chat")) {
chatCalls++;
throw new Error("ETIMEDOUT");
}
throw new Error(`unexpected fetch: ${urlStr}`);
}) as typeof fetch;
try {
const result = await testExec.execute({
model: "mimo-auto",
body: { messages: [{ role: "user", content: "hi" }], stream: false },
stream: false,
signal: null,
credentials: {},
log: { debug: () => {}, info: () => {}, warn: () => {}, error: () => {} },
});
assert.strictEqual(
chatCalls,
1,
"remaining proxy-less accounts must be skipped without a network call once the shared egress is known down"
);
assert.strictEqual(result.response.status, 502);
} finally {
globalThis.fetch = originalFetch;
}
});
it("fails fast without rotating when the guard is disabled (legacy behavior)", async () => {
process.env[FLAG] = "false";
const testExec = twoAccountExecutor([null, null]);
let chatCalls = 0;
const originalFetch = globalThis.fetch;
globalThis.fetch = (async (url: unknown) => {
const urlStr = String(url);
if (urlStr.includes("/api/free-ai/bootstrap")) {
return new Response(JSON.stringify({ jwt: makeJwt() }), { status: 200 });
}
if (urlStr.includes("/api/free-ai/openai/chat")) {
chatCalls++;
throw new Error("ETIMEDOUT");
}
throw new Error(`unexpected fetch: ${urlStr}`);
}) as typeof fetch;
const warnCalls: string[] = [];
try {
const result = await testExec.execute({
model: "mimo-auto",
body: { messages: [{ role: "user", content: "hi" }], stream: false },
stream: false,
signal: null,
credentials: {},
log: {
debug: () => {},
info: () => {},
warn: (_tag: unknown, msg: string) => warnCalls.push(msg),
error: () => {},
},
});
assert.strictEqual(
chatCalls,
1,
"must NOT retry against another account sharing the same egress"
);
const acctA = accountAccess(testExec).accounts[0];
assert.strictEqual(
acctA.cooldownUntil,
0,
"an account without a dedicated proxy must not be cooled down for a shared-egress failure"
);
assert.strictEqual(result.response.status, 502);
assert.ok(
warnCalls.some((m) => /network error/i.test(m) && /not rotating/i.test(m)),
`expected a "network error … not rotating" warn log; got=${JSON.stringify(warnCalls)}`
);
} finally {
globalThis.fetch = originalFetch;
}
});
});
});

233
tests/unit/opencode-proxy-rotation-4954.test.ts
View File

@@ -2,6 +2,7 @@ import { describe, it, beforeEach, afterEach, before, after } from "node:test";
import assert from "node:assert";
import net from "node:net";
import { OpencodeExecutor } from "../../open-sse/executors/opencode.ts";
import type { ExecutorLog } from "../../open-sse/executors/base.ts";
import {
resolveProxyForRequest,
runWithAppliedProxyCapture,
@@ -57,17 +58,21 @@ after(() => {
serverB?.close();
});
function credentialsWithProxies() {
/** Two fingerprints; `withProxies: false` omits accountProxies so both accounts
* share the default egress instead of each having a dedicated proxy. */
function credentialsWithProxies(withProxies = true) {
return {
apiKey: null,
accessToken: null,
connectionId: "noauth",
providerSpecificData: {
fingerprints: [ACCOUNT_A, ACCOUNT_B],
accountProxies: [
{ fingerprint: ACCOUNT_A, proxy: { type: "http", host: "127.0.0.1", port: portA } },
{ fingerprint: ACCOUNT_B, proxy: { type: "http", host: "127.0.0.1", port: portB } },
],
...(withProxies && {
accountProxies: [
{ fingerprint: ACCOUNT_A, proxy: { type: "http", host: "127.0.0.1", port: portA } },
{ fingerprint: ACCOUNT_B, proxy: { type: "http", host: "127.0.0.1", port: portB } },
],
}),
},
} as any;
}
@@ -89,7 +94,8 @@ describe("OpencodeExecutor per-account proxy + rotation (#4954)", () => {
function installFetchStub(statuses: number[]) {
let call = 0;
globalThis.fetch = (async (input: any) => {
const url = typeof input === "string" ? input : input?.url || String(input);
const url =
typeof input === "string" ? input : input instanceof URL ? input.toString() : input.url;
const resolved = resolveProxyForRequest(url);
let host: string | null = null;
let port: string | null = null;
@@ -169,6 +175,221 @@ describe("OpencodeExecutor per-account proxy + rotation (#4954)", () => {
}
});
it("rotates to the next account on a network throw (not just 429)", async () => {
const exec = new OpencodeExecutor("opencode-zen");
let call = 0;
const originalFetchForThrow = globalThis.fetch;
globalThis.fetch = (async (input: Parameters<typeof fetch>[0]) => {
const url =
typeof input === "string" ? input : input instanceof URL ? input.toString() : input.url;
const resolved = resolveProxyForRequest(url);
observed.push({
source: resolved.source,
host: resolved.proxyUrl ? new URL(resolved.proxyUrl).hostname : null,
port: resolved.proxyUrl ? new URL(resolved.proxyUrl).port : null,
});
call++;
if (call === 1) {
throw new Error("ECONNRESET: connection reset by peer");
}
return new Response(JSON.stringify({ ok: true }), {
status: 200,
headers: { "Content-Type": "application/json" },
});
}) as typeof globalThis.fetch;
try {
const result = await exec.execute({
model: "deepseek-v4-flash-free",
body: { messages: [{ role: "user", content: "hi" }], stream: false },
stream: false,
signal: null,
credentials: credentialsWithProxies(),
log,
});
assert.strictEqual(
(result as { response: { status: number } }).response.status,
200,
"a throw on account A must not abort the request — account B must be tried"
);
assert.ok(observed.length >= 2, "should have retried on a second account after the throw");
assert.notStrictEqual(
observed[0].port,
observed[1].port,
"rotation must switch to a different account/proxy after a throw"
);
} finally {
globalThis.fetch = originalFetchForThrow;
}
});
it("logs a network-error rotation and does not swallow it silently", async () => {
const exec = new OpencodeExecutor("opencode-zen");
let call = 0;
const originalFetchForThrow = globalThis.fetch;
globalThis.fetch = (async () => {
call++;
if (call === 1) throw new Error("ETIMEDOUT");
return new Response(JSON.stringify({ ok: true }), {
status: 200,
headers: { "Content-Type": "application/json" },
});
}) as typeof globalThis.fetch;
const warnCalls: Array<{ tag: unknown; msg: string }> = [];
const spyLog: ExecutorLog = {
debug() {},
info() {},
warn: (tag, msg) => {
warnCalls.push({ tag, msg });
},
error() {},
};
try {
await exec.execute({
model: "deepseek-v4-flash-free",
body: { messages: [{ role: "user", content: "hi" }], stream: false },
stream: false,
signal: null,
credentials: credentialsWithProxies(),
log: spyLog,
});
assert.ok(
warnCalls.some((c) => c.tag === "OPENCODE" && /network error/i.test(c.msg)),
`expected a warn-level "network error" log; got=${JSON.stringify(warnCalls)}`
);
} finally {
globalThis.fetch = originalFetchForThrow;
}
});
describe("NETWORK_ROTATION_SHARED_EGRESS_GUARD", () => {
const FLAG = "NETWORK_ROTATION_SHARED_EGRESS_GUARD";
let originalEnvValue: string | undefined;
beforeEach(() => {
originalEnvValue = process.env[FLAG];
});
afterEach(() => {
if (originalEnvValue === undefined) delete process.env[FLAG];
else process.env[FLAG] = originalEnvValue;
});
it("rotates to a proxied account after a proxy-less account throws (mixed fleet, guard on by default)", async () => {
const exec = new OpencodeExecutor("opencode-zen");
let call = 0;
const originalFetchForThrow = globalThis.fetch;
globalThis.fetch = (async () => {
call++;
if (call === 1) throw new Error("ETIMEDOUT");
return new Response(JSON.stringify({ ok: true }), {
status: 200,
headers: { "Content-Type": "application/json" },
});
}) as typeof globalThis.fetch;
try {
// ACCOUNT_A has no proxy, ACCOUNT_B does — credentialsWithProxies(true)
// only configures a proxy for accounts present in accountProxies; give
// A no entry so it stays proxy-less while B keeps its dedicated proxy.
const credentials = credentialsWithProxies();
credentials.providerSpecificData.accountProxies =
credentials.providerSpecificData.accountProxies.filter(
(ap: { fingerprint: string }) => ap.fingerprint !== ACCOUNT_A
);
const result = await exec.execute({
model: "deepseek-v4-flash-free",
body: { messages: [{ role: "user", content: "hi" }], stream: false },
stream: false,
signal: null,
credentials,
log,
});
assert.strictEqual(
(result as { response: { status: number } }).response.status,
200,
"the proxied account (B) must still be tried and must succeed the request"
);
assert.strictEqual(call, 2, "exactly one throw (A) then one success (B)");
} finally {
globalThis.fetch = originalFetchForThrow;
}
});
it("makes a single real network call when no account has a configured proxy (guard on by default)", async () => {
const exec = new OpencodeExecutor("opencode-zen");
let call = 0;
const originalFetchForThrow = globalThis.fetch;
globalThis.fetch = (async () => {
call++;
throw new Error("ETIMEDOUT");
}) as typeof globalThis.fetch;
try {
await assert.rejects(
() =>
exec.execute({
model: "deepseek-v4-flash-free",
body: { messages: [{ role: "user", content: "hi" }], stream: false },
stream: false,
signal: null,
credentials: credentialsWithProxies(false),
log,
}),
/ETIMEDOUT/,
"must ultimately propagate once no candidate account remains"
);
assert.strictEqual(
call,
1,
"remaining proxy-less accounts must be skipped without a network call once the shared egress is known down"
);
} finally {
globalThis.fetch = originalFetchForThrow;
}
});
it("propagates immediately on the first proxy-less throw when the guard is disabled", async () => {
process.env[FLAG] = "false";
const exec = new OpencodeExecutor("opencode-zen");
let call = 0;
const originalFetchForThrow = globalThis.fetch;
globalThis.fetch = (async () => {
call++;
throw new Error("ETIMEDOUT");
}) as typeof globalThis.fetch;
try {
await assert.rejects(
() =>
exec.execute({
model: "deepseek-v4-flash-free",
body: { messages: [{ role: "user", content: "hi" }], stream: false },
stream: false,
signal: null,
credentials: credentialsWithProxies(false),
log,
}),
/ETIMEDOUT/,
"a network throw on a proxy-less account must propagate, not be swallowed into rotation"
);
assert.strictEqual(
call,
1,
"must not retry against another account when the guard is disabled"
);
} finally {
globalThis.fetch = originalFetchForThrow;
}
});
});
// #5217 (Gap 2): the per-request account/proxy selection log was log.debug, which
// is hidden at the default APP_LOG_LEVEL=info — operators could not see which
// account/proxy a request rotated to. It must be emitted at info level.