pjandro/OmniRoute
1
0
Fork 0
mirror of https://github.com/diegosouzapw/OmniRoute.git synced 2026-09-14 19:02:17 +03:00
Code Issues Packages Projects Releases Wiki Activity

fix(sse): rotate opencode accounts on transient 5xx (#12975)

Browse Source 
Reconciled and merged. This branch was stacked on #12941, which has since landed, so it read as 1484 additions across 16 files and CONFLICTING. I merged the current `release/v3.8.51` into it rather than rewriting your branch: `open-sse/executors/opencode.ts` conflicted in six places where your side was a strict superset of the squashed #12941, and the tip had touched that file through nothing but #12941, so your side was taken whole. The PR now reads as its real 12 files, +839/-25.

The change itself is right: a flapping upstream 5xx aborting the whole agentic chain is exactly the case where rotating to the next healthy account is safe, and keeping it a separate arm from the 400-empty branch matters because that one has to clone-read the body while this one never touches it. Threading `correlationId` through so interleaved requests stay attributable — and never fabricating one when absent — is the right discipline.

Both geo-block regression suites pass alongside the new ones (43/43 across the five opencode test files), which is what proves the conflict resolution preserved #12941's behaviour.

I also tightened the batch's file-size rebaseline here: `src/sse/handlers/chat.ts` needed no bump at all (it lands at 2452, under its existing 2458 freeze) and `open-sse/executors/base.ts` needed only your +2. An earlier measurement had included a local prettier reformat that is not part of this branch.

---

Validated in one consolidated worktree cut from `release/v3.8.51`, boarded together with the rest of this batch — zero conflicts between them.

- `typecheck:core` clean; `check:changelog-integrity` OK
- complexity 2799 / baseline 3218 and cognitive-complexity 1265 / baseline 1437 — both under baseline
- 86 focused assertions green across the batch's 10 unit test files, plus 16/16 on the v1 plugin option schema and 16/16 on the v2 option tests
- `check-file-size` rebaselined for this batch's real growth (annotation `_rebaseline_2026_09_11_mergebatch_v3851_maxmad_opencode`, landed on #13141). `open-sse/utils/stream.ts` was deliberately left frozen: it is already 3115 > 3098 on the pure tip with zero contribution from this batch.

⚠️ base-red inherited: #12732 — `Docs Gates`, `Merge integrity`, `No new ESLint warnings`, `Unit Tests fast-path` and `Fast Quality Gates` all reproduce on the pure `release/v3.8.51` tip (provider count 356 vs the 358 the modules define, SKILL.md drift, and the `stream.ts` freeze above). None of them touch these diffs.

Thanks @maxmad64bis.
This commit is contained in:
Dizzle
2026-09-11 18:57:28 +02:00
committed by GitHub
parent a19bb2227f
commit cfa2fc7548
12 changed files with 839 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
changelog.d/fixes/12975-opencode-transient-5xx-rotation.md Normal file
View File

@@ -0,0 +1 @@
- **fix(sse):** transient opencode upstream failures rotate to the next account proxy instead of failing, so one flapping egress no longer aborts the whole chain ([#12975](https://github.com/diegosouzapw/OmniRoute/pull/12975)) — thanks @maxmad64bis

7
config/quality/file-size-baseline.json
View File

@@ -1,5 +1,6 @@
{
"_rebaseline_2026_09_11_mergebatch_v3851_maxmad_opencode": "/merge-batch 2026-09-11 (v3.8.51), PRs #13141, #13146 and #12975 by maxmad64bis. Final combined values for the batch, set here on the first PR merged so every intermediate merge state is covered too. src/sse/services/auth.ts 3450->3488 (+38): #13146 adds the narrow ruleScope===model branch to markAccountUnavailable (gated on status 400; every other status keeps its path) plus the HONORS_RULE_LOCK_SCOPE_PROVIDERS opencode entry, and #12975 adds buildExhaustionOptions so the exhaustion log lines carry the request correlationId. open-sse/executors/base.ts 1751->1757 (+6): #12975 adds the optional ExecuteInput.correlationId field with its doc comment (+2); the other +4 is prettier splitting the cliFingerprints import, a 103-char line the tip left unformatted, which lint-staged rewrites on any commit touching the file. src/sse/handlers/chat.ts 2458->2460 (+2): #12975 threads correlationId through the three executor call sites (+2) and prettier splits a 168-char comboTargetPassesKeyModelPolicy condition (+8), same unformatted-tip cause; the tip itself sits 9 lines under its own freeze, which absorbs the rest. open-sse/services/accountFallback.ts 2467->2468 (+1): #13141 routes hasFutureRateLimitUntil through the tolerant epoch normalizer; #13146 is net zero there (+16/-16). No new branching beyond the two guarded branches named above. open-sse/utils/stream.ts is deliberately NOT rebaselined: it is already 3115 > 3098 on the pure tip with zero contribution from this batch (base-red #12732, owned by /sweep-reds). Covered by tests/unit/combo-predicates-epoch-cooldown.test.ts, opencode-400-model-unavailable.test.ts, agentrouter-error-rules.test.ts, opencode-transient-rotation.test.ts and chat-correlation-id-exhaustion.test.ts.",
"_rebaseline_2026_09_10_12975_rotation_correlation_id": "PR #12975 own growth: open-sse/executors/base.ts 1751->1753 (+2) and open-sse/handlers/chatCore.ts 6021->6024 (+3). The opencode rotation lines carry the request correlationId: one optional ExecuteInput field and one correlationId argument at each of the three executor.execute call sites in handleChatCore. Irreducible plumbing at existing call sites; the rotation logic itself lives in open-sse/executors/opencode.ts and the new leaf predicates (under cap). Covered by tests/unit/opencode-transient-rotation.test.ts and tests/unit/chat-correlation-id-exhaustion.test.ts.",
"_rebaseline_2026_09_11_mergebatch_v3851_maxmad_opencode": "/merge-batch 2026-09-11 (v3.8.51), PRs #13141, #13146 and #12975 by maxmad64bis. src/sse/services/auth.ts 3450->3488 (+38): #13146 adds the narrow ruleScope===model branch to markAccountUnavailable (gated on status 400; every other status keeps its path) plus the HONORS_RULE_LOCK_SCOPE_PROVIDERS opencode entry, taking it to 3464; #12975 then adds buildExhaustionOptions so the exhaustion log lines carry the request correlationId (+24). open-sse/services/accountFallback.ts 2467->2468 (+1): #13141 routes hasFutureRateLimitUntil through the tolerant epoch normalizer; #13146 is net zero there (+16/-16). open-sse/executors/base.ts 1751->1753 (+2): #12975 adds the optional ExecuteInput.correlationId field with its doc comment. src/sse/handlers/chat.ts is NOT rebaselined: #12975 threads correlationId through the three executor call sites (+2) but the file lands at 2452, still under its existing 2458 freeze. open-sse/utils/stream.ts is deliberately NOT rebaselined either: it is already 3115 > 3098 on the pure tip with zero contribution from this batch (base-red #12732, owned by /sweep-reds). No new branching beyond the two guarded branches named above. Covered by tests/unit/combo-predicates-epoch-cooldown.test.ts, opencode-400-model-unavailable.test.ts, agentrouter-error-rules.test.ts, opencode-transient-rotation.test.ts and chat-correlation-id-exhaustion.test.ts.",
"_rebaseline_2026_09_10_mergebatch_v3851_greenpt_eurouter": "/merge-batch 2026-09-10 (v3.8.51), PRs #13024 (GreenPT, closes #12986) and #13025 (EURouter, closes #12985) by ntdatt812: src/shared/constants/providers/apikey/gateways.ts 1462->1502 (+40 = two APIKEY_PROVIDERS_GATEWAYS catalog entries, declarative data only: id/alias/name/icon/color/website plus the hasFree=false rationale comments and the apiHint copy each PR verified). No logic and no new branching. Same god-file no-split rationale as every prior gateways.ts rebaseline (#11786 seekai, #10987 logfare, #10668 tabitoken, #10531 freebuff, #11631 1min.ai): the file header says it is pure data merged by apikey/index.ts via spread, and it is already split into 6 family files under apikey/, so splitting a catalog for two entries would violate the semantic-families rule rather than help. Both entries are deliberately conservative (models: [] with passthroughModels, no tool/vision capability declared, hasFree false), so the growth is the entry itself, not claims. EURouter is in AGGREGATOR_PROVIDER_IDS because it routes to third-party upstreams; GreenPT is not because it serves its own inference. Covered by tests/unit/greenpt-provider.test.ts and tests/unit/eurouter-provider.test.ts.",
"_rebaseline_2026_09_10_12828_translate_usage_chunk": "PR #12828 own growth: open-sse/utils/stream.ts 3072->3080 (+8). Translate-mode streams now send the estimated usage as the canonical trailing usage-only chunk before [DONE] when the upstream stays silent (parity with the #12151 passthrough flush), with a latch so a finish chunk that already carried the estimate is not doubled. The chunk builder is shared with the passthrough flush in open-sse/utils/usageOnlyChunk.ts (under cap); what remains is the flush-site wiring. Covered by tests/unit/stream-translate-usage-trailing.test.ts.",
"_rebaseline_2026_09_10_12715_queue_budget": "PR #12715 own growth: open-sse/handlers/chatCore.ts 6021->6036 (+15). Hierarchical admission now resolves the per-connection queue budget before the gates and hands withRateLimit the remaining budget, the correlation id and the executor timeout context, so gate wait, provider slot and Bottleneck queue share one bound instead of stacking. Error shaping lives in open-sse/handlers/chatCore/queueBudget.ts (under cap); what remains is irreducible call-site wiring. Covered by tests/unit/rate-limit-remaining-budget.test.ts, rate-limit-manager-queue-bound.test.ts and chatcore-hierarchical-admission.test.ts.",
@@ -423,7 +424,7 @@
"_rebaseline_pr1043_minimax_tts": "Upstream port decolua/9router#1043 (toanalien) own growth: audioSpeech.ts 965->1061 (+96). Adds MiniMax T2A v2 TTS dispatch (handleMinimaxSpeech + hexToBytes helper) — provider entry was already in audioRegistry (format: minimax-tts) but no handler existed, falling through to the OpenAI-compatible default that fails (T2A has custom shape + hex-encoded audio + base_resp envelope). New branch sits next to the other inline provider branches (xiaomi-mimo, coqui, tortoise, aws-polly) — extracting would just create indirection. Covered by tests/unit/minimax-tts-1043.test.ts (3 tests, GREEN: success, base_resp error, invalid-hex).",
"_rebaseline_pr4592_exclude_exhausted_auto": "Reconcile #4592 already-merged growth: combo.ts 2991->3036 (+45, terminal-status quota-cutoff exclusion in buildAutoCandidates + opt-in gate). Fast-gate PR->release does not run check:file-size.",
"open-sse/executors/antigravity.ts": 1665,
"open-sse/executors/base.ts": 1757,
"open-sse/executors/base.ts": 1753,
"open-sse/executors/chatgpt-web.ts": 5056,
"open-sse/executors/codex.ts": 1505,
"open-sse/executors/cursor.ts": 1759,
@@ -469,7 +470,7 @@
"src/shared/components/RequestLoggerV2.tsx": 1718,
"src/shared/constants/providers/apikey/gateways.ts": 1502,
"src/shared/services/cliRuntime.ts": 1296,
"src/sse/handlers/chat.ts": 2460,
"src/sse/handlers/chat.ts": 2458,
"src/sse/services/auth.ts": 3488,
"tests/unit/account-fallback-service.test.ts": 2453,
"tests/unit/provider-validation-specialty.test.ts": 4656,

2
open-sse/executors/base.ts
View File

@@ -211,6 +211,8 @@ export type ExecuteInput = {
) => Promise<void> | void;
/** When true, skip the intra-URL 429 retry in execute() so the caller handles fallback. */
skipUpstreamRetry?: boolean;
/** Request-scoped id for log attribution; absent off the chat path, never fabricated. */
correlationId?: string | null;
/** Delegated Context Editing (Claude only): when enabled, attach the
* `context_management.clear_tool_uses` strategy so the provider clears stale
* tool-use blocks server-side. Honored only on the genuine `claude` path. */

57
open-sse/executors/opencode.ts
View File

@@ -29,6 +29,7 @@ import {
extractChatcmplId,
} from "./accountRotation.ts";
import { isOpencodeGeoBlocked, proxyKeyOf } from "./opencodeGeoBlock.ts";
import { isRetriableUpstreamFailure } from "./opencodeTransientFailure.ts";
import { isNetworkRotationSharedEgressGuardEnabled } from "@/shared/utils/featureFlags";
/**
@@ -504,6 +505,10 @@ export class OpencodeExecutor extends BaseExecutor {
this.syncAccountsFromCredentials(input.credentials);
const { log } = input;
// Request-scoped attribution prefix for rotation logs: message head,
// empty when absent (never n/a/none/fabricated). The existing motif
// stays byte-identical after the prefix.
const cid = input.correlationId ? `correlationId=${input.correlationId} ` : "";
const hasProxies = this.accounts.some((a) => a.proxy !== null);
// Fast path: no multi-account proxy wiring configured → original behavior,
@@ -533,7 +538,7 @@ export class OpencodeExecutor extends BaseExecutor {
const chatcmplId = extractChatcmplId(bodyText);
log?.warn?.(
"OPENCODE",
`upstream empty rejection on direct account (${chatcmplId}), retrying once…`
`${cid}upstream empty rejection on direct account (${chatcmplId}), retrying once…`
);
return this.normalizeMuseSparkResponse(input, await super.execute(input));
}
@@ -567,8 +572,9 @@ export class OpencodeExecutor extends BaseExecutor {
// through the accounts is the retry). Avoids an unbounded loop on a
// persistently malformed upstream.
const emptyRejectionBudget = this.accounts.length === 1 ? 1 : 0;
// 403-geo tried set: proxy keys already proven geo-blocked for this
// request's model. Request-local only — nothing persists past execute().
// Tried set: proxy keys already proven unusable for this request's
// model (geo-blocked, or transient 5xx). Request-local only — nothing
// persists past execute().
const geoTriedProxyKeys = new Set<string>();
let directTried = false;
@@ -594,15 +600,19 @@ export class OpencodeExecutor extends BaseExecutor {
}
const lastStatus = lastResult !== null ? lastResult.response.status : null;
const lastWasGeo = lastStatus === 403 || lastStatus === 451;
const lastWasTransient = lastStatus !== null && lastStatus >= 500 && lastStatus < 600;
const isMonoRetryOwed = this.accounts.length === 1 && lastWasTransient;
if (
!isMonoRetryOwed &&
lastResult !== null &&
geoTriedProxyKeys.size > 0 &&
!isProxiedCandidate(account) &&
!(account.proxy === null && !directTried)
) {
// Geo exhaustion (last was 403/451) → surface as-is, no success mark.
// Transient exhaustion (last was 5xx) → same: surface last as-is.
// Any other last status (e.g. 429 after 403s) → skip without a call.
if (lastWasGeo) break;
if (lastWasGeo || lastWasTransient) break;
continue;
}
// Commit the last-resort direct attempt so a later exclusion breaks
@@ -614,7 +624,7 @@ export class OpencodeExecutor extends BaseExecutor {
if (sharedEgressGuardEnabled && sharedEgressDown && !account.proxy) {
log?.warn?.(
"OPENCODE",
`skipping account ${masked} (no dedicated proxy, shared egress already down this request)`
`${cid}skipping account ${masked} (no dedicated proxy, shared egress already down this request)`
);
continue;
}
@@ -625,7 +635,7 @@ export class OpencodeExecutor extends BaseExecutor {
// Token stays masked — never log the full account id.
log?.info?.(
"OPENCODE",
`dispatch via account ${masked} (idx ${attempt + 1}/${this.accounts.length})` +
`${cid}dispatch via account ${masked} (idx ${attempt + 1}/${this.accounts.length})` +
(account.proxy
? ` through proxy ${account.proxy.host}:${account.proxy.port}`
: " direct")
@@ -657,20 +667,20 @@ export class OpencodeExecutor extends BaseExecutor {
lastSharedEgressError = err;
log?.warn?.(
"OPENCODE",
`network error on account ${masked} (no dedicated proxy, shared egress), cooldown applied — trying next available account… (${reason})`
`${cid}network error on account ${masked} (no dedicated proxy, shared egress), cooldown applied — trying next available account… (${reason})`
);
continue;
}
log?.warn?.(
"OPENCODE",
`network error on account ${masked} (no dedicated proxy, shared egress) — not rotating (${reason})`
`${cid}network error on account ${masked} (no dedicated proxy, shared egress) — not rotating (${reason})`
);
throw err;
}
this.markCooldown(account);
log?.warn?.(
"OPENCODE",
`network error on account ${masked}, rotating to next… (${reason})`
`${cid}network error on account ${masked}, rotating to next… (${reason})`
);
continue;
}
@@ -679,7 +689,28 @@ export class OpencodeExecutor extends BaseExecutor {
const status = result.response.status;
if (status === 429) {
this.markCooldown(account);
log?.warn?.("OPENCODE", `Rate limited (429) on account ${masked}, rotating to next…`);
log?.warn?.(
"OPENCODE",
`${cid}Rate limited (429) on account ${masked}, rotating to next…`
);
continue;
}
if (isRetriableUpstreamFailure(status)) {
const key = proxyKeyOf(account.proxy);
if (key !== null) geoTriedProxyKeys.add(key);
else directTried = true;
log?.warn?.(
"OPENCODE",
`${cid}transient upstream ${status} on account ${masked} (proxy ${key ?? "direct"}), rotating to next…`
);
// Deliberately a separate branch from the 400-empty arm below,
// not one merged `if`: this arm never touches the body, the 400
// arm must clone-read it. Both share the predicate + tried-set.
// Single proxied account: one retry via the existing budget (a
// proxy-less single account takes the fast path, never the loop).
// Transient is not deterministic like geo: upstream may recover.
// No 0-retry guard here (it stays geo-only).
continue;
}
@@ -696,7 +727,7 @@ export class OpencodeExecutor extends BaseExecutor {
else directTried = true;
log?.warn?.(
"OPENCODE",
`geo-blocked on account ${masked} (proxy ${key ?? "direct"}), rotating to next…`
`${cid}geo-blocked on account ${masked} (proxy ${key ?? "direct"}), rotating to next…`
);
// Single account with a proxy: 0 retries (same egress = dead latency).
// (The fast path above already covers single-without-proxy; here length===1 WITH proxy.)
@@ -719,11 +750,11 @@ export class OpencodeExecutor extends BaseExecutor {
} catch {
log?.debug?.("OPENCODE", "body read failed on empty rejection check");
}
if (bodyText !== null && isEmptyUpstreamRejection(400, bodyText)) {
if (bodyText !== null && isRetriableUpstreamFailure(400, bodyText)) {
const chatcmplId = extractChatcmplId(bodyText);
log?.warn?.(
"OPENCODE",
`upstream empty rejection on account ${masked} (${chatcmplId}), rotating to next…`
`${cid}upstream empty rejection on account ${masked} (${chatcmplId}), rotating to next…`
);
continue;
}

17
open-sse/executors/opencodeTransientFailure.ts Normal file
View File

@@ -0,0 +1,17 @@
/**
* opencodeTransientFailure.ts — retriable-upstream predicate for the opencode
* executor loop.
*
* Leaf module: one internal import only (isEmptyUpstreamRejection, same
* executors layer — no registry, no DB). 5xx short-circuits on status alone;
* the 400 arm delegates to the existing empty-rejection classifier.
*/
import { isEmptyUpstreamRejection } from "./accountRotation.ts";
export function isRetriableUpstreamFailure(status: number, bodyText?: string): boolean {
if (status >= 500 && status < 600) return true;
if (status !== 400) return false;
if (typeof bodyText !== "string" || bodyText === "") return false;
return isEmptyUpstreamRejection(status, bodyText);
}

3
open-sse/handlers/chatCore.ts
View File

@@ -3167,6 +3167,7 @@ export async function handleChatCore({
onCredentialsRefreshed,
skipUpstreamRetry,
contextEditing: { enabled: contextEditingEnabled },
correlationId,
})
),
});
@@ -3353,6 +3354,7 @@ export async function handleChatCore({
onCredentialsRefreshed,
skipUpstreamRetry,
contextEditing: { enabled: contextEditingEnabled },
correlationId,
})
),
});
@@ -4009,6 +4011,7 @@ export async function handleChatCore({
onCredentialsRefreshed,
skipUpstreamRetry: isCombo,
contextEditing: { enabled: contextEditingEnabled },
correlationId,
})
)
);

12
src/sse/handlers/chat.ts
View File

@@ -12,6 +12,7 @@ import { resolveRoutingModel, RoutingModelOps } from "./resolveRoutingModel";
import {
getProviderCredentialsWithQuotaPreflight,
markAccountUnavailable,
buildExhaustionOptions,
extractApiKey,
isValidApiKey,
extractSessionAffinityKey,
@@ -1781,7 +1782,8 @@ async function handleSingleModelChat(
lastStatus,
candidateAliases,
isCombo,
shadowedNode
shadowedNode,
runtimeOptions?.correlationId ?? null
);
const lastFailedConnectionId =
excludedConnectionIds.size > 0
@@ -2093,7 +2095,7 @@ async function handleSingleModelChat(
provider,
model,
providerProfile,
{ isCombo }
buildExhaustionOptions(runtimeOptions.correlationId ?? null, { isCombo })
);
if (shouldFallback && !hasForcedConnection) {
@@ -2142,7 +2144,7 @@ async function handleSingleModelChat(
provider,
model,
providerProfile,
{ isCombo }
buildExhaustionOptions(runtimeOptions.correlationId ?? null, { isCombo })
);
if (shouldFallback && !hasForcedConnection) {
@@ -2387,7 +2389,7 @@ async function handleSingleModelChat(
provider,
model,
providerProfile,
{
buildExhaustionOptions(runtimeOptions.correlationId ?? null, {
persistUnavailableState: !(
isCombo &&
result.status === 429 &&
@@ -2395,7 +2397,7 @@ async function handleSingleModelChat(
),
isCombo,
headers: result.response.headers,
}
})
);
// An explicit pin (combo step `connectionId` / `x-omniroute-connection`) is an

12
src/sse/handlers/chatHelpers.ts
View File

@@ -3,7 +3,11 @@ import {
getComboForModel,
getModelInfoOrRetirementResponse,
} from "../services/model";
import { clearAccountError, markAccountUnavailable } from "../services/auth";
import {
clearAccountError,
markAccountUnavailable,
buildExhaustionOptions,
} from "../services/auth";
import { connectionHasExtraKeys } from "@omniroute/open-sse/services/apiKeyRotator.ts";
import { createBuiltinAutoCombo } from "@omniroute/open-sse/services/autoCombo/builtinCatalog.ts";
import * as log from "../utils/logger";
@@ -555,7 +559,7 @@ export async function executeChatWithBreaker({
provider,
model,
providerProfile,
{ isCombo }
buildExhaustionOptions(correlationId ?? null, { isCombo })
);
},
})
@@ -731,7 +735,8 @@ export function handleNoCredentials(
lastStatus: number | null,
candidateAliases?: readonly string[],
isCombo: boolean = false,
shadowedNode: ShadowedProviderNode | null = null
shadowedNode: ShadowedProviderNode | null = null,
correlationId?: string | null
) {
if (credentials?.allRateLimited) {
const errorMsg = lastError || credentials.lastError || "Unavailable";
@@ -772,6 +777,7 @@ export function handleNoCredentials(
provider,
model,
lastStatus,
...(correlationId ? { correlationId } : {}),
});
return errorResponse(lastStatus, lastError);
}

26
src/sse/services/auth.ts
View File

@@ -2550,6 +2550,26 @@ async function applyEgressIpLockout(
}
}
/** Build the options for markAccountUnavailable on the chat exhaustion path.
* Single place that forwards the request id so no chat sender can forget it:
* every chat caller passes its in-scope id through here. */
export function buildExhaustionOptions(
correlationId: string | null,
rest: {
persistUnavailableState?: boolean;
/** Caller is the combo engine — it records its own model-level lockouts. */
isCombo?: boolean;
headers?: Headers | Record<string, string> | null;
} = {}
): {
persistUnavailableState?: boolean;
isCombo?: boolean;
headers?: Headers | Record<string, string> | null;
correlationId: string | null;
} {
return { ...rest, correlationId };
}
/** Persist exponential-backoff state for an unavailable provider connection. */
export async function markAccountUnavailable(
connectionId: string,
@@ -2563,6 +2583,7 @@ export async function markAccountUnavailable(
/** Caller is the combo engine — it records its own model-level lockouts. */
isCombo?: boolean;
headers?: Headers | Record<string, string> | null;
correlationId?: string | null;
} = {}
) {
const currentMutex = markMutexes.get(connectionId) || Promise.resolve();
@@ -2931,7 +2952,10 @@ export async function markAccountUnavailable(
}).catch(() => {});
log.info(
"AUTH",
`Server error for ${provider}:${model} — ${status} ${reason} (no model lockout, connection stays active for sibling models)`
`Server error for ${provider}:${model} — ${status} ${reason} (no model lockout, connection stays active for sibling models)`,
{
...(options.correlationId ? { correlationId: options.correlationId } : {}),
}
);
return { shouldFallback: true, cooldownMs: 0 };
}

177
tests/unit/chat-correlation-id-exhaustion.test.ts Normal file
View File

@@ -0,0 +1,177 @@
import test from "node:test";
import assert from "node:assert/strict";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-exhaustion-id-"));
process.env.DATA_DIR = TEST_DATA_DIR;
const core = await import("../../src/lib/db/core.ts");
const providersDb = await import("../../src/lib/db/providers.ts");
const auth = await import("../../src/sse/services/auth.ts");
const chatHelpers = await import("../../src/sse/handlers/chatHelpers.ts");
async function resetStorage() {
core.resetDbInstance();
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
fs.mkdirSync(TEST_DATA_DIR, { recursive: true });
}
test.after(() => {
core.resetDbInstance();
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
});
async function createConnection(provider = "opencode-test") {
const conn = await providersDb.createProviderConnection({
provider,
authType: "oauth",
accessToken: "access-token",
refreshToken: "refresh-token",
isActive: true,
testStatus: "active",
});
return String(conn.id);
}
async function driveExhaustionViaBare500(
connId: string,
options?: { correlationId?: string | null }
) {
// Bare 500 takes the status === 500 early branch: no model lockout, the
// request-scoped id lands on the exhaustion line.
return auth.markAccountUnavailable(
connId,
500,
"transient upstream 500",
"opencode-test",
"test-model",
null,
options ?? {}
);
}
function readSource(rel: string) {
return fs.readFileSync(new URL(rel, import.meta.url), "utf8");
}
test("exhaustion lines carry the request id", async (t) => {
await t.test("chat sender forwards the id (sender side)", async () => {
// A receiver-only test (options hand-set at the auth call) would
// still pass if a chat sender stopped forwarding the id. This test reads
// the sender call sites directly: every chat sender must pass its
// in-scope request id via options. If any of the four senders drops the
// field, the count/asserts below fail.
const chatSource = readSource("../../src/sse/handlers/chat.ts");
const helpersSource = readSource("../../src/sse/handlers/chatHelpers.ts");
const chatSenders = [
...chatSource.matchAll(/buildExhaustionOptions\(runtimeOptions\.correlationId \?\? null,/g),
];
assert.equal(
chatSenders.length,
3,
"chat.ts must pass runtimeOptions.correlationId at all three markAccountUnavailable senders (:2089/:2138/:2383)"
);
assert.match(
helpersSource,
/buildExhaustionOptions\(correlationId \?\? null,/,
"chatHelpers.ts onStreamFailure must pass its in-scope correlationId via options"
);
// The fallback-path sender (:2383) carries the full options literal —
// persist flag, combo flag, headers AND the id together.
assert.match(
chatSource,
/buildExhaustionOptions\(runtimeOptions\.correlationId \?\? null, \{\s*persistUnavailableState: !\([\s\S]*?headers: result\.response\.headers,\s*\}\)/,
"chat.ts:2383 fallback sender must forward the id alongside the existing options literal"
);
// The exhaustion caller passes the id positionally (10th arg), not a bare
// request id from another scope.
assert.match(
chatSource,
/handleNoCredentials\(\s*credentials,[\s\S]*?shadowedNode,\s*runtimeOptions\?\.correlationId \?\? null\s*\)/,
"chat.ts:1775 must pass runtimeOptions?.correlationId ?? null as the trailing handleNoCredentials arg"
);
// The pure helper itself forwards the exact id the sender passes in.
assert.deepEqual(auth.buildExhaustionOptions("trace-123", { isCombo: true }), {
isCombo: true,
correlationId: "trace-123",
});
assert.deepEqual(auth.buildExhaustionOptions(null, { isCombo: false }), {
isCombo: false,
correlationId: null,
});
});
await t.test("auth.ts emits structured id meta on the exhaustion line", async () => {
const authSource = readSource("../../src/sse/services/auth.ts");
assert.match(
authSource,
/\.\.\.\(options\.correlationId \? \{ correlationId: options\.correlationId \} : \{\}\)/,
"auth.ts:2868 must spread correlationId into the log meta only when truthy"
);
await resetStorage();
const withId = await createConnection();
const resWithId = await driveExhaustionViaBare500(
withId,
auth.buildExhaustionOptions("trace-123")
);
// Bare 500: no model lockout, connection stays active, fallback allowed.
assert.equal(resWithId.shouldFallback, true);
const withAfter = await providersDb.getProviderConnectionById(withId);
assert.equal(
(withAfter as unknown as { lastErrorType?: string })?.lastErrorType,
"server_error"
);
await resetStorage();
const withoutId = await createConnection();
const resWithoutId = await driveExhaustionViaBare500(
withoutId,
auth.buildExhaustionOptions(null)
);
assert.equal(resWithoutId.shouldFallback, true);
});
await t.test("handleNoCredentials emits structured id meta", async () => {
const helpersSource = readSource("../../src/sse/handlers/chatHelpers.ts");
assert.match(
helpersSource,
/\.\.\.\(correlationId \? \{ correlationId \} : \{\}\)/,
"chatHelpers.ts:771 must spread correlationId into the log meta only when truthy"
);
// Exhaustion with an id returns the upstream error; without an id the
// response shape is unchanged.
const withId = chatHelpers.handleNoCredentials(
null,
"conn-1",
"opencode-test",
"test-model",
"upstream 500",
500,
undefined,
false,
null,
"trace-123"
);
assert.equal(withId.status, 500);
const withBody = (await withId.json()) as { error?: { message?: string } };
assert.equal(withBody?.error?.message, "upstream 500");
const withoutId = chatHelpers.handleNoCredentials(
null,
"conn-1",
"opencode-test",
"test-model",
"upstream 500",
500
);
assert.equal(withoutId.status, 500);
const withoutBody = (await withoutId.json()) as { error?: { message?: string } };
assert.equal(withoutBody?.error?.message, "upstream 500");
});
});

36
tests/unit/opencode-transient-failure-predicate.test.ts Normal file
View File

@@ -0,0 +1,36 @@
import { describe, it } from "node:test";
import assert from "node:assert";
import { isRetriableUpstreamFailure } from "../../open-sse/executors/opencodeTransientFailure.ts";
const EMPTY_400_BODY = JSON.stringify({
id: "chatcmpl-abc123",
choices: [{ message: {}, finish_reason: null }],
});
const REAL_400_BODY = JSON.stringify({ error: { message: "bad request" } });
describe("isRetriableUpstreamFailure", () => {
it("matches 500/502/503/504 by status alone, no body needed", () => {
assert.strictEqual(isRetriableUpstreamFailure(500), true);
assert.strictEqual(isRetriableUpstreamFailure(502), true);
assert.strictEqual(isRetriableUpstreamFailure(503), true);
assert.strictEqual(isRetriableUpstreamFailure(504), true);
});
it("matches 500 even with a body present (status short-circuits first)", () => {
assert.strictEqual(isRetriableUpstreamFailure(500, "Internal server error"), true);
});
it("matches empty 400 with body", () => {
assert.strictEqual(isRetriableUpstreamFailure(400, EMPTY_400_BODY), true);
});
it("rejects real-error 400", () => {
assert.strictEqual(isRetriableUpstreamFailure(400, REAL_400_BODY), false);
});
it("rejects 400 without body (absent = non-empty = no retry)", () => {
assert.strictEqual(isRetriableUpstreamFailure(400), false);
assert.strictEqual(isRetriableUpstreamFailure(400, ""), false);
});
it("rejects 403/429/200", () => {
assert.strictEqual(isRetriableUpstreamFailure(403), false);
assert.strictEqual(isRetriableUpstreamFailure(429), false);
assert.strictEqual(isRetriableUpstreamFailure(200), false);
});
});

514
tests/unit/opencode-transient-rotation.test.ts Normal file
View File

@@ -0,0 +1,514 @@
import { describe, it, beforeEach, afterEach, before, after } from "node:test";
import assert from "node:assert";
import net from "node:net";
import { OpencodeExecutor } from "../../open-sse/executors/opencode.ts";
import type { ExecutorLog, ProviderCredentials } from "../../open-sse/executors/base.ts";
import { resolveProxyForRequest } from "../../open-sse/utils/proxyFetch.ts";
const log: ExecutorLog = { debug() {}, info() {}, warn() {}, error() {} };
const FP_A = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
const FP_B = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb";
const FP_C = "cccccccccccccccccccccccccccccccc";
let serverA: net.Server;
let serverB: net.Server;
let serverC: net.Server;
let portA = 0;
let portB = 0;
let portC = 0;
function listen(server: net.Server): Promise<number> {
return new Promise((resolve) => {
server.listen(0, "127.0.0.1", () => {
resolve((server.address() as net.AddressInfo).port);
});
});
}
before(async () => {
serverA = net.createServer((s) => s.destroy());
serverB = net.createServer((s) => s.destroy());
serverC = net.createServer((s) => s.destroy());
portA = await listen(serverA);
portB = await listen(serverB);
portC = await listen(serverC);
});
after(() => {
serverA?.close();
serverB?.close();
serverC?.close();
});
function portFor(fp: string): number {
if (fp === FP_A) return portA;
if (fp === FP_B) return portB;
return portC;
}
function credentialsFor(fingerprints: string[]): ProviderCredentials {
return {
apiKey: null,
accessToken: null,
connectionId: "noauth",
providerSpecificData: {
fingerprints,
accountProxies: fingerprints.map((fp) => ({
fingerprint: fp,
proxy: { type: "http", host: "127.0.0.1", port: portFor(fp) },
})),
},
};
}
describe("OpencodeExecutor transient-failure rotation", () => {
let originalFetch: typeof globalThis.fetch;
let observed: string[];
beforeEach(() => {
originalFetch = globalThis.fetch;
observed = [];
});
afterEach(() => {
globalThis.fetch = originalFetch;
});
class CloneCountingResponse extends Response {
static clones = 0;
clone(): Response {
CloneCountingResponse.clones++;
return super.clone();
}
}
function installFetch(plan: Array<{ status: number; body?: string }>) {
let call = 0;
CloneCountingResponse.clones = 0;
globalThis.fetch = (async (input: RequestInfo | URL) => {
const url =
typeof input === "string" ? input : input instanceof URL ? input.toString() : input.url;
const resolved = resolveProxyForRequest(url);
observed.push(resolved.proxyUrl ? new URL(resolved.proxyUrl).port : "direct");
const step = plan[Math.min(call, plan.length - 1)];
call++;
return new CloneCountingResponse(step.body ?? JSON.stringify({ ok: step.status === 200 }), {
status: step.status,
headers: { "Content-Type": "application/json" },
});
}) as typeof globalThis.fetch;
}
it("rotates past a 500 to the healthy proxy without cooldown", async () => {
const exec = new OpencodeExecutor("opencode-zen");
installFetch([{ status: 500 }, { status: 200 }]);
const result = await exec.execute({
model: "muse-spark-1.3-contributor-free",
body: { messages: [{ role: "user", content: "hi" }], stream: false },
stream: false,
signal: null,
credentials: credentialsFor([FP_A, FP_B]),
log,
});
assert.strictEqual((result as { response: Response }).response.status, 200);
assert.strictEqual(observed.length, 2);
assert.strictEqual(observed[0], String(portA));
assert.strictEqual(
CloneCountingResponse.clones,
1,
"only success-path normalize clones; 500 branch reads no body"
);
});
it("rotates on 502/503/504 like on 500", async () => {
for (const status of [502, 503, 504]) {
observed = [];
const exec = new OpencodeExecutor("opencode-zen");
installFetch([{ status }, { status: 200 }]);
const result = await exec.execute({
model: "muse-spark-1.3-contributor-free",
body: { messages: [{ role: "user", content: "hi" }], stream: false },
stream: false,
signal: null,
credentials: credentialsFor([FP_A, FP_B]),
log,
});
assert.strictEqual(
(result as { response: Response }).response.status,
200,
`status ${status} must rotate`
);
assert.strictEqual(observed.length, 2);
}
});
it("single account without proxy stays on fast path on 500 (propagates)", async () => {
const exec = new OpencodeExecutor("opencode-zen");
installFetch([{ status: 500 }]);
const creds = credentialsFor([FP_A]);
(creds.providerSpecificData as Record<string, unknown>).accountProxies = [];
const result = await exec.execute({
model: "muse-spark-1.3-contributor-free",
body: { messages: [{ role: "user", content: "hi" }], stream: false },
stream: false,
signal: null,
credentials: creds,
log,
});
assert.strictEqual((result as { response: Response }).response.status, 500);
assert.strictEqual(observed.length, 1);
});
it("true mono-direct (no fingerprints) propagates 500 without success mark", async () => {
const exec = new OpencodeExecutor("opencode-zen");
installFetch([{ status: 500 }]);
const creds: ProviderCredentials = {
apiKey: null,
accessToken: null,
connectionId: "noauth",
providerSpecificData: { fingerprints: [] },
};
const result = await exec.execute({
model: "muse-spark-1.3-contributor-free",
body: { messages: [{ role: "user", content: "hi" }], stream: false },
stream: false,
signal: null,
credentials: creds,
log,
});
assert.strictEqual((result as { response: Response }).response.status, 500);
assert.strictEqual(observed.length, 1, "fast path: single call, no loop");
});
it("propagates the last 500 after exhausting all proxies", async () => {
const exec = new OpencodeExecutor("opencode-zen");
installFetch([{ status: 200 }]);
await exec.execute({
model: "muse-spark-1.3-contributor-free",
body: { messages: [{ role: "user", content: "hi" }], stream: false },
stream: false,
signal: null,
credentials: credentialsFor([FP_A, FP_B, FP_C]),
log,
});
const warm = (
exec as unknown as { accounts: Array<{ cooldownUntil: number; consecutiveFails: number }> }
).accounts;
assert.strictEqual(warm.length, 3, "warm-up materialized all accounts");
for (const a of warm) a.consecutiveFails = 2;
installFetch([{ status: 500 }, { status: 500 }, { status: 500 }]);
observed = [];
const result = await exec.execute({
model: "muse-spark-1.3-contributor-free",
body: { messages: [{ role: "user", content: "hi" }], stream: false },
stream: false,
signal: null,
credentials: credentialsFor([FP_A, FP_B, FP_C]),
log,
});
assert.strictEqual((result as { response: Response }).response.status, 500);
assert.strictEqual(observed.length, 3, "every proxy tried exactly once");
for (const port of [portA, portB, portC]) {
assert.ok(observed.includes(String(port)), `proxy ${port} tried`);
}
const after = (
exec as unknown as { accounts: Array<{ cooldownUntil: number; consecutiveFails: number }> }
).accounts;
for (const a of after) {
assert.strictEqual(a.cooldownUntil, 0, "no cooldown from 500 exhaustion");
assert.strictEqual(a.consecutiveFails, 2, "500 exhaustion never marks success");
}
});
it("never re-touches a proxy tried by either 500 or geo-403", async () => {
const exec = new OpencodeExecutor("opencode-zen");
const GEO_BODY = JSON.stringify({
error: { type: "RegionError", message: "This model is not available in your country." },
});
installFetch([{ status: 500 }, { status: 403, body: GEO_BODY }, { status: 200 }]);
const result = await exec.execute({
model: "muse-spark-1.3-contributor-free",
body: { messages: [{ role: "user", content: "hi" }], stream: false },
stream: false,
signal: null,
credentials: credentialsFor([FP_A, FP_B, FP_C]),
log,
});
assert.strictEqual((result as { response: Response }).response.status, 200);
assert.strictEqual(observed.length, 3);
assert.strictEqual(
observed.filter((p) => p === String(portA)).length,
1,
"500-tried proxy A called exactly once"
);
});
it("a 429 still cools down while a 500 rotates cleanly", async () => {
const exec = new OpencodeExecutor("opencode-zen");
installFetch([{ status: 500 }, { status: 429 }, { status: 200 }]);
const result = await exec.execute({
model: "muse-spark-1.3-contributor-free",
body: { messages: [{ role: "user", content: "hi" }], stream: false },
stream: false,
signal: null,
credentials: credentialsFor([FP_A, FP_B, FP_C]),
log,
});
assert.strictEqual((result as { response: Response }).response.status, 200);
assert.strictEqual(observed.length, 3);
const state = (exec as unknown as { accounts: Array<{ cooldownUntil: number }> }).accounts;
const cooled = state.filter((a) => a.cooldownUntil > Date.now());
assert.strictEqual(cooled.length, 1, "exactly the 429 account cooled down");
});
it("single proxied account: one retry on 500, then last surfaces", async () => {
const exec = new OpencodeExecutor("opencode-zen");
const creds = credentialsFor([FP_A]);
installFetch([{ status: 500 }, { status: 500 }]);
const result = await exec.execute({
model: "muse-spark-1.3-contributor-free",
body: { messages: [{ role: "user", content: "hi" }], stream: false },
stream: false,
signal: null,
credentials: creds,
log,
});
assert.strictEqual((result as { response: Response }).response.status, 500);
assert.strictEqual(observed.length, 2, "one retry via the mono budget, then stop");
});
it("500 rotation never cools the account down", async () => {
const exec2 = new OpencodeExecutor("opencode-zen");
installFetch([{ status: 200 }]);
await exec2.execute({
model: "muse-spark-1.3-contributor-free",
body: { messages: [{ role: "user", content: "hi" }], stream: false },
stream: false,
signal: null,
credentials: credentialsFor([FP_A, FP_B]),
log,
});
const mid = (
exec2 as unknown as { accounts: Array<{ cooldownUntil: number; consecutiveFails: number }> }
).accounts;
assert.strictEqual(mid.length, 2, "warm-up materialized both accounts");
for (const a of mid) a.consecutiveFails = 2;
installFetch([{ status: 500 }, { status: 200 }]);
await exec2.execute({
model: "muse-spark-1.3-contributor-free",
body: { messages: [{ role: "user", content: "hi" }], stream: false },
stream: false,
signal: null,
credentials: credentialsFor([FP_A, FP_B]),
log,
});
const after = (
exec2 as unknown as { accounts: Array<{ cooldownUntil: number; consecutiveFails: number }> }
).accounts;
for (const a of after) {
assert.strictEqual(a.cooldownUntil, 0, "no cooldown from 500 rotation");
}
assert.strictEqual(
after.filter((a) => a.consecutiveFails === 0).length,
1,
"exactly the winning account resets via markSuccess"
);
assert.strictEqual(
after.filter((a) => a.consecutiveFails === 2).length,
after.length - 1,
"blocked accounts keep prior fails"
);
});
it("a 500 on the last-resort direct attempt surfaces cleanly", async () => {
const exec = new OpencodeExecutor("opencode-zen");
const creds = credentialsFor([FP_A, FP_B]);
(creds.providerSpecificData as Record<string, unknown>).accountProxies = [
{ fingerprint: FP_A, proxy: { type: "http", host: "127.0.0.1", port: portA } },
];
installFetch([{ status: 500 }, { status: 500 }]);
const result = await exec.execute({
model: "muse-spark-1.3-contributor-free",
body: { messages: [{ role: "user", content: "hi" }], stream: false },
stream: false,
signal: null,
credentials: creds,
log,
});
assert.strictEqual((result as { response: Response }).response.status, 500);
assert.strictEqual(observed.length, 2, "one proxied + one direct, direct last");
assert.strictEqual(observed[0], String(portA));
assert.strictEqual(observed[1], "direct");
});
it("executor rotation lines carry correlationId", async () => {
// Genuinely overlapped A/B: both execute() calls are in flight
// simultaneously on ONE shared executor (production shape — the registry
// caches one instance per provider). Each of the 4 upstream dispatches is
// a deferred promise resolved in a cross order (B1, A1, A2, B2), so a
// shared/module-level cid — or any cross-request bleed — would attribute
// at least one line to the wrong request and fail the per-id assertions.
const exec = new OpencodeExecutor("opencode-zen");
const gates: Array<{
resolve: (r: Response) => void;
url: string;
}> = [];
const gateFetchCalls: string[] = [];
globalThis.fetch = ((input: RequestInfo | URL) => {
const url =
typeof input === "string" ? input : input instanceof URL ? input.toString() : input.url;
const resolved = resolveProxyForRequest(url);
gateFetchCalls.push(resolved.proxyUrl ? new URL(resolved.proxyUrl).port : "direct");
return new Promise<Response>((resolve) => {
gates.push({ resolve, url });
});
}) as typeof globalThis.fetch;
const ok = () =>
new Response(JSON.stringify({ ok: true }), {
status: 200,
headers: { "Content-Type": "application/json" },
});
const fail500 = () =>
new Response(JSON.stringify({ ok: false }), {
status: 500,
headers: { "Content-Type": "application/json" },
});
function runWithLines(id: string) {
const lines: string[] = [];
const spyLog: ExecutorLog = {
debug() {},
info(tag, message) {
lines.push(`${tag} ${message}`);
},
warn(tag, message) {
lines.push(`${tag} ${message}`);
},
error() {},
};
const done = exec
.execute({
model: "muse-spark-1.3-contributor-free",
body: { messages: [{ role: "user", content: "hi" }], stream: false },
stream: false,
signal: null,
credentials: credentialsFor([FP_A, FP_B]),
log: spyLog,
correlationId: id,
})
.then((result) => {
assert.strictEqual(
(result as { response: Response }).response.status,
200,
`request ${id} must rotate past its 500`
);
return lines;
});
return { id, lines, done };
}
const reqA = runWithLines("A");
const reqB = runWithLines("B");
// Let both first dispatches land before resolving anything: proves both
// requests are in flight simultaneously (the cross-talk window).
for (let i = 0; i < 50 && gates.length < 2; i++) {
await new Promise((r) => setImmediate(r));
}
assert.strictEqual(gates.length, 2, "both requests must be in flight simultaneously");
// Controllable cross order: B's 500 first, then A's 500, then A's 200, B's 200.
gates[1].resolve(fail500());
for (let i = 0; i < 50 && gates.length < 3; i++) {
await new Promise((r) => setImmediate(r));
}
gates[0].resolve(fail500());
for (let i = 0; i < 50 && gates.length < 4; i++) {
await new Promise((r) => setImmediate(r));
}
assert.strictEqual(gates.length, 4, "both rotations must dispatch a second attempt");
gates[2].resolve(ok());
gates[3].resolve(ok());
const [linesA, linesB] = await Promise.all([reqA.done, reqB.done]);
for (const [lines, id] of [
[linesA, "A"],
[linesB, "B"],
] as const) {
const rotation = lines.filter((l) => /rotating to next|dispatch via account/.test(l));
assert.ok(rotation.length > 0, `request ${id} must emit rotation lines`);
for (const line of rotation) {
assert.ok(
line.startsWith(`OPENCODE correlationId=${id} `),
`line must start with correlationId=${id}: ${line}`
);
}
}
assert.ok(
linesA.every((l) => !l.includes("correlationId=B")),
"no cross-talk: A's lines must never carry B's id"
);
assert.ok(
linesB.every((l) => !l.includes("correlationId=A")),
"no cross-talk: B's lines must never carry A's id"
);
// Absent id leaves the line unchanged: no correlationId field, motif intact.
installFetch([{ status: 500 }, { status: 200 }]);
const plainExec = new OpencodeExecutor("opencode-zen");
const plain: string[] = [];
const plainLog: ExecutorLog = {
debug() {},
info(tag, message) {
plain.push(`${tag} ${message}`);
},
warn(tag, message) {
plain.push(`${tag} ${message}`);
},
error() {},
};
const plainResult = await plainExec.execute({
model: "muse-spark-1.3-contributor-free",
body: { messages: [{ role: "user", content: "hi" }], stream: false },
stream: false,
signal: null,
credentials: credentialsFor([FP_A, FP_B]),
log: plainLog,
});
assert.strictEqual((plainResult as { response: Response }).response.status, 200);
const plainRotation = plain.filter((l) => /rotating to next|dispatch via account/.test(l));
assert.ok(plainRotation.length > 0, "must emit rotation lines without an id");
for (const line of plainRotation) {
assert.ok(!line.includes("correlationId"), `no id field when absent: ${line}`);
}
assert.ok(
plainRotation.some((l) =>
/transient upstream 500 on account .* \(proxy .*\), rotating to next…/.test(l)
),
"existing 5xx rotation motif byte-identical when no id is present"
);
assert.ok(
plainRotation.some((l) => /dispatch via account .* \(idx \d+\/2\)/.test(l)),
"existing dispatch motif byte-identical when no id is present"
);
});
});