fix(combo): auto-resume pinned native Codex turns (re-land of #13180) (#14162)

Re-land of #13180 by @mdigitalbh81 (their commits carried with authorship intact), merged via /merge-batch (2026-09-19) on top of the current `release/v3.8.51` tip.

**Reconciled before landing (three maintainer commits on top of the re-land):**
- `81d770b3` — `check:api-typecheck` / `check:open-sse-typecheck` were red on the PR's own head with `TS2339 'reason' does not exist on type 'AutoResumeDecision'` ×3 in `open-sse/services/combo.ts` (green on the tip). Without `strictNullChecks` the truthiness test on the `eligible` discriminant does not narrow the else-branch; `=== true` does.
- `66e57cfe` — `tests/unit/native-codex-auto-resume.test.ts` was 1540 lines, above the 1200-line new-test cap. The three pure guard cases (`hasUnresolvedToolCalls`, `hasProviderSpecificUnsafeContinuationState`, `MAX_AUTORESUMES_PER_TURN`) moved verbatim to `native-codex-auto-resume-guards.test.ts`; the combo-flow file is now 1131 lines.
- `c2f1937f` — `executeTargetAttempt.ts` 1258→1273 rebaselined with a dated justification (own growth: generation advance + log at the pin site).

**Evidence on the merged tree:** `native-codex-auto-resume{,-guards}.test.ts` 15/15; `native-codex-turn-pin-model-scoped-fallback`, `native-codex-turn-pin-10379`, `chatgpt-web-codex-turn-pin` 33/33 total; broader combo sweep (`combo-responses-sse-failure-fallback`, `combo-context-window-filter`, `perf-waterfall-elimination`, `chatgpt-web-codex`, `combo-context-overflow-compression-probe`) 70/70; api/open-sse typecheck clean for the PR's files; file-size, changelog-integrity, complexity and cognitive-complexity gates OK.

**Inherited, not from this PR** (reproduced on the pure tip): the 23 unit reds in the fast-path shards (vi locale parity, pack-artifact allowlists, `.env.example` sync, casing, budget fallback, etc.), the 5 `no-unused-vars` lint errors (`cliRuntime.ts`, `arena-elo-sync-redesign`, `compressionAnalyticsWriterFlatRate`, `waitForServer-slow-first-response`), the `omni-version-manager` generated-skill drift, `tinycmsDomMocks.ts` / `rerankProviderNodes.ts` / `antigravity.ts` typecheck errors, and the 4 env vars missing from `.env.example`.

Supersedes #13180. Follow-up to #12240.
This commit is contained in:
Diego Rodrigues de Sa e Souza
2026-09-19 01:24:39 -03:00
committed by GitHub
parent 3fd1265d89
commit cd4c6f6947
8 changed files with 2131 additions and 36 deletions

View File

@@ -0,0 +1 @@
- **fix(combo):** auto-resume a pinned native Codex turn on a healthy sibling connection or model when the pinned provider becomes unavailable for a model-scoped reason (quota, model lockout) instead of failing the turn outright — provider-wide circuit-breaker/cooldown state, pending tool calls, opaque continuation state, and partial streams still block resume, and at most one auto-resume happens per logical turn ([#13180](https://github.com/diegosouzapw/OmniRoute/pull/13180)) — thanks @mdigitalbh81

View File

@@ -481,7 +481,7 @@
"open-sse/services/accountFallback.ts": 2517,
"open-sse/services/adobeFireflyBrowserLogin.ts": 1401,
"open-sse/services/combo.ts": 4080,
"open-sse/services/combo/executeTargetAttempt.ts": 1258,
"open-sse/services/combo/executeTargetAttempt.ts": 1273,
"open-sse/translator/response/openai-responses.ts": 1518,
"open-sse/utils/cursorAgentProtobuf.ts": 1588,
"open-sse/utils/proxyFetch.ts": 1276,
@@ -719,5 +719,6 @@
"_rebaseline_2026_09_17_11742_log_boundary_hardening": "PR #11742 (rebase para release/v3.8.51): open-sse/handlers/chatCore.ts 6219->6287. O crescimento e a unica parte da PR que sobreviveu ao tip: endurecimento da fronteira de LOG (mais amplo que a Hard Rule #12, que cobre respostas). Sao 136 linhas adicionadas, das quais ~40 sao chamadas diretas de sanitizacao — sanitizeErrorMessage em erro de plugin (onError), em timeout de semaforo e na failureMessage antes de ela chegar ao console.log e ao call-log; sanitizeUpstreamDetails no log de resposta malformada; getSafeErrorMetadata + try/catch nos pontos onde metadata hostil (Proxy) podia lançar. O resto da PR foi descartado por ja estar no tip (#12506/#12945/#13635 error boundaries, #12429 wreq-js, #11754 aposentadoria do ChatGPT Web) — open-sse/utils/ difere do tip por UMA linha (registro do identificador publico lmarena_stream_error).",
"_rebaseline_2026_09_17_13670_allow_auto_combos": "PR #13670 (@fouadSalkini): per-key allowAutoCombos para gatear os combos auto/* embutidos. src/app/api/v1/models/catalog.ts 2075->2117 e src/lib/db/apiKeys.ts 1625->1659. Crescimento e 100% proprio da PR, nao herdado: medido no tip puro, catalog.ts esta em 2074 (abaixo do teto 2075) e apiKeys.ts em 1620 (abaixo de 1625). O aumento e a propria feature — o campo de permissao por chave precisa ser lido, validado e propagado ate o filtro do catalogo, e cada ponto e chamada explicita, nao extraivel sem esconder o gate. Coberto pelos 25 testes da PR. As demais violacoes desta arvore (chatHelpers.ts, chatCore.ts e tests/unit/chatcore-translation-paths.test.ts) sao base-red herdado do tip e nao foram tocadas aqui.",
"_rebaseline_2026_09_17c_chatcore_translation_paths_test": "tests/unit/chatcore-translation-paths.test.ts 3447->3449 (#13173, prefixos de cache de meio de conversa do Fable — as assercoes novas do caso). Ultimo teto remanescente da leva de merges de 2026-09-17; os outros dois (chatHelpers.ts e chatCore.ts) foram absorvidos pelos rebaselines das proprias PRs que mergearam depois. Medido no tip limpo.",
"_rebaseline_2026_09_18_13929_antigravity_account_lease_merge": "PR #13929 (Re-land of #10011, @Ardem2025 via @diegosouzapw): the Antigravity account lease, merged onto the current release/v3.8.51 tip (which had independently moved chat.ts to 2520 and auth.ts to 3557 via unrelated PRs). Combined ceiling after merge: src/sse/handlers/chat.ts->2541, src/sse/services/auth.ts->3577. The lease registry, its lifecycle glue and its selection glue were extracted into three NEW modules (src/sse/services/antigravityRoutingState.ts, antigravityLeaseLifecycle.ts, antigravityLeaseSelection.ts) precisely to keep this growth to the call sites; what remains in chat.ts/auth.ts is the wiring itself, which cannot be moved out of the selection loop and the dispatch path. Every added hunk is inert unless ANTIGRAVITY_ACCOUNT_LEASE_ENABLED (default false) is on. Covered by tests/unit/antigravity-routing-state.test.ts, antigravity-lease-lifecycle.test.ts and antigravity-account-lease-flag.test.ts. UPDATE (re-sync 2026-09-18 after trains 3b/4d moved the tip): auth.ts 3577->3582 (same +29 own growth over a tip now at 3552). open-sse/executors/base.ts 1753->1754 is NOT this PR's growth — it is release-tip drift from train 3b (#13002 +5 / #13705 -4 net +1, both merged without a baseline entry); absorbed here by the captain session under the owner-approved train-rebaseline policy so the tip stops failing check:file-size for every PR boarding after it."
"_rebaseline_2026_09_18_13929_antigravity_account_lease_merge": "PR #13929 (Re-land of #10011, @Ardem2025 via @diegosouzapw): the Antigravity account lease, merged onto the current release/v3.8.51 tip (which had independently moved chat.ts to 2520 and auth.ts to 3557 via unrelated PRs). Combined ceiling after merge: src/sse/handlers/chat.ts->2541, src/sse/services/auth.ts->3577. The lease registry, its lifecycle glue and its selection glue were extracted into three NEW modules (src/sse/services/antigravityRoutingState.ts, antigravityLeaseLifecycle.ts, antigravityLeaseSelection.ts) precisely to keep this growth to the call sites; what remains in chat.ts/auth.ts is the wiring itself, which cannot be moved out of the selection loop and the dispatch path. Every added hunk is inert unless ANTIGRAVITY_ACCOUNT_LEASE_ENABLED (default false) is on. Covered by tests/unit/antigravity-routing-state.test.ts, antigravity-lease-lifecycle.test.ts and antigravity-account-lease-flag.test.ts. UPDATE (re-sync 2026-09-18 after trains 3b/4d moved the tip): auth.ts 3577->3582 (same +29 own growth over a tip now at 3552). open-sse/executors/base.ts 1753->1754 is NOT this PR's growth — it is release-tip drift from train 3b (#13002 +5 / #13705 -4 net +1, both merged without a baseline entry); absorbed here by the captain session under the owner-approved train-rebaseline policy so the tip stops failing check:file-size for every PR boarding after it.",
"_rebaseline_2026_09_19_14162_native_codex_auto_resume": "PR #14162 (re-land of #13180, @mdigitalbh81 via @diegosouzapw): native Codex turn auto-resume. open-sse/services/combo/executeTargetAttempt.ts 1258->1273 (+15). Growth is 100% the PR's own, measured against the clean tip (1258 there, gate green): the pin step now advances the logical turn generation and logs the resumed provider/model when the attempt is an auto-resume dispatch, and the generation is passed into pinNativeCodexTurn — the branch has to sit at the pin site because that is the only place the winning target and effective connection are known. Covered by tests/unit/native-codex-auto-resume.test.ts + native-codex-auto-resume-guards.test.ts (15/15) and #13564's native-codex-turn-pin-model-scoped-fallback.test.ts (7/7)."
}

View File

@@ -90,6 +90,8 @@ export {
import {
applyNativeCodexTurnPin,
areAllPinnedTargetsModelScopedUnusable,
canAutoResumeNativeCodexTurn,
createPinnedModelUnavailableResponse,
getNativeCodexTurnPin,
releaseNativeCodexTurnPin,
} from "./combo/nativeCodexTurnPin.ts";
@@ -131,6 +133,22 @@ import { executeTargetAttempt } from "./combo/executeTargetAttempt.ts";
import type { AttemptLoopDeps, AttemptLoopState } from "./combo/attemptLoopTypes.ts";
import { clearStaleLKGP } from "./combo/staleLkgpClear.ts";
// Native Codex auto-resume (#13180) rejection reasons that mean the turn either carries
// state unsafe to hand to an untested alternate model (pending tool calls, opaque
// provider-specific continuation state) or has already used its one allowed resume for
// this logical turn. These must terminate the turn rather than fall through to #13564's
// plain "release pin and route naturally" fallback. Every other reason (e.g. the request
// does not use the Responses-API input/messages shape #13180's eligibility check needs)
// falls through unchanged so non-native-turn-shaped Codex requests keep working exactly
// as before #13180.
const NATIVE_CODEX_AUTO_RESUME_UNSAFE_REASONS = new Set([
"pending_tool_call",
"unsafe_provider_state",
"no_alternate_target",
"no_healthy_alternate_target",
"max_resumes_exceeded",
]);
export { RESET_WINDOW_NAMES, QUOTA_SOFT_DEPRIORITIZE_FACTOR, setCandidateQuotaSoftPenalty };
export { scoreAutoTargets, expandAutoComboCandidatePool };
export type { SingleModelTarget, ResolvedComboTarget };
@@ -776,9 +794,10 @@ async function handleComboChatInner({
});
if (runtimeUnitDispatch) return runtimeUnitDispatch;
const activeNativeTurnPin = clientManagedResponsesContext
let activeNativeTurnPin = clientManagedResponsesContext
? getNativeCodexTurnPin(body, combo.name)
: null;
let isAutoResuming = false;
// Route new round-robin turns to the specialized handler. A native Codex
// continuation with an established provider/account pin must use the common
@@ -858,15 +877,60 @@ async function handleComboChatInner({
isModelAvailable,
});
if (allPinnedUnusable) {
// All pinned provider+model targets are model-scoped unusable — release
// the pin and fall through to full combo routing so the turn can try
// other models in the combo pool. This matches Claude Code's behavior
// where no turn pin allows natural multi-model fallback.
releaseNativeCodexTurnPin(body as Record<string, unknown>, combo.name);
log.warn(
"COMBO",
`Native Codex turn pin released: pinned model ${activeNativeTurnPin.modelStr} model-scoped unavailable; falling back to full combo routing`
);
const autoResumeEligibility = await canAutoResumeNativeCodexTurn({
body: body as Record<string, unknown>,
comboName: combo.name,
activePin: activeNativeTurnPin,
allTargets: orderedTargets,
resilienceSettings,
quotaCutoffResetWindowConfig,
isModelAvailable,
log,
});
if (autoResumeEligibility.eligible === true) {
const selectedAlternate = autoResumeEligibility.selectedTarget;
log.info(
"COMBO",
`Native Codex auto-resume eligible: previous provider/model=${activeNativeTurnPin.provider}/${activeNativeTurnPin.modelStr}, previous logical turn generation=${autoResumeEligibility.previousPin.generation ?? 0}, reason=model_scoped_unavailable`
);
log.info(
"COMBO",
`Native Codex auto-resume started: previous provider/model=${activeNativeTurnPin.provider}/${activeNativeTurnPin.modelStr}, target provider/model=${selectedAlternate.provider}/${selectedAlternate.modelStr}, target generation=${autoResumeEligibility.nextGeneration}`
);
const alternateTargets = orderedTargets.filter(
(t) =>
t.modelStr === selectedAlternate.modelStr && t.provider === selectedAlternate.provider
);
orderedTargets = alternateTargets;
activeNativeTurnPin = null;
isAutoResuming = true;
} else if (NATIVE_CODEX_AUTO_RESUME_UNSAFE_REASONS.has(autoResumeEligibility.reason)) {
// These specific rejection reasons mean the turn carries state (pending
// tool calls, opaque provider-specific continuation state) or has
// already exhausted its resume budget, so handing it to an untested
// alternate model via natural combo routing (#13564's plain fallback)
// would be unsafe or would violate #13180's "at most one auto-resume
// per logical turn" bound. Terminate instead of falling through.
targetResolution.quotaShareRelease?.();
log.warn(
"COMBO",
`Native Codex turn cannot continue: pinned model ${activeNativeTurnPin.modelStr} is unavailable (model-scoped); auto-resume rejected (${autoResumeEligibility.reason}); preserving turn pin and terminating turn`
);
return createPinnedModelUnavailableResponse();
} else {
// Every other rejection reason (e.g. the request body does not carry
// the Responses-API `input`/`messages` shape #13180's eligibility
// check needs) means auto-resume simply cannot be evaluated — it says
// nothing about the request being unsafe. Fall back to the plain
// release-and-route-naturally behavior (#13564) so non-native-turn or
// legacy-shaped Codex requests keep working exactly as before #13180.
releaseNativeCodexTurnPin(body as Record<string, unknown>, combo.name);
log.warn(
"COMBO",
`Native Codex turn pin released: pinned model ${activeNativeTurnPin.modelStr} model-scoped unavailable; auto-resume not eligible (${autoResumeEligibility.reason}); falling back to full combo routing`
);
}
} else {
orderedTargets = pinnedTargets;
log.info(
@@ -989,6 +1053,7 @@ async function handleComboChatInner({
releaseStickyPinOnFailure,
clearStaleLKGP,
clientManagedResponsesContext,
nativeCodexAutoResume: isAutoResuming,
reasoningTokenBufferEnabled,
stickyWeightedLimit,
getWeightedStepKeyForTarget,

View File

@@ -102,6 +102,7 @@ export type AttemptLoopDeps = {
* gate tests keep compiling; attempt uses defaults when absent.
*/
clientManagedResponsesContext?: boolean;
nativeCodexAutoResume?: boolean;
reasoningTokenBufferEnabled?: boolean;
stickyWeightedLimit?: number;
getWeightedStepKeyForTarget?: (target: ResolvedComboTarget) => string | null;

View File

@@ -71,7 +71,7 @@ import {
isModelScoped400,
} from "./comboPredicates.ts";
import { applyComboTargetExhaustion } from "./targetExhaustion.ts";
import { pinNativeCodexTurn } from "./nativeCodexTurnPin.ts";
import { advanceNativeCodexTurnGeneration, pinNativeCodexTurn } from "./nativeCodexTurnPin.ts";
import { recordComboDecision } from "./decisionTrace.ts";
import { recordProviderCooldown } from "../providerCooldownTracker.ts";
import {
@@ -469,12 +469,27 @@ export async function executeTargetAttempt(opts: {
}
if (Boolean(deps.clientManagedResponsesContext) && effectiveConnectionId) {
pinNativeCodexTurn({
body: deps.body,
comboName: deps.combo.name,
target,
connectionId: effectiveConnectionId,
});
if (deps.nativeCodexAutoResume) {
const nextGen = advanceNativeCodexTurnGeneration(deps.body, deps.combo.name);
deps.log.info(
"COMBO",
`Native Codex auto-resume routed: new provider/model=${target.modelStr} on connection ${effectiveConnectionId.slice(0, 8)} (logical turn generation ${nextGen})`
);
pinNativeCodexTurn({
body: deps.body,
comboName: deps.combo.name,
target,
connectionId: effectiveConnectionId,
generation: nextGen ?? undefined,
});
} else {
pinNativeCodexTurn({
body: deps.body,
comboName: deps.combo.name,
target,
connectionId: effectiveConnectionId,
});
}
}
// Success decay: a healthy response walks the model's lockout failure

View File

@@ -13,18 +13,30 @@ import type { ComboLogger, IsModelAvailable } from "./types.ts";
import type { ResolvedComboTarget } from "./types.ts";
type NativeTurnPin = {
export type NativeTurnPin = {
comboName: string;
modelStr: string;
provider: string;
connectionId: string;
createdAt: number;
expiresAt: number;
generation?: number;
};
export interface NativeCodexTurnRecord {
comboName: string;
threadId: string;
turnId: string;
activeGeneration: number;
pins: Map<number, NativeTurnPin>;
createdAt: number;
expiresAt: number;
}
export const MAX_AUTORESUMES_PER_TURN = 1;
const TTL_MS = 45 * 60_000;
const MAX_PINS = 1_000;
const pins = new Map<string, NativeTurnPin>();
const turns = new Map<string, NativeCodexTurnRecord>();
function record(value: unknown): Record<string, unknown> | undefined {
return value && typeof value === "object" && !Array.isArray(value)
@@ -57,21 +69,51 @@ export function nativeCodexTurnKey(
}
function prune(now = Date.now()): void {
for (const [key, pin] of pins) if (pin.expiresAt <= now) pins.delete(key);
while (pins.size > MAX_PINS) {
const oldest = pins.keys().next().value as string | undefined;
for (const [key, rec] of turns) if (rec.expiresAt <= now) turns.delete(key);
while (turns.size > MAX_PINS) {
const oldest = turns.keys().next().value as string | undefined;
if (!oldest) break;
pins.delete(oldest);
turns.delete(oldest);
}
}
export function getNativeCodexTurnPin(
body: Record<string, unknown>,
comboName: string
comboName: string,
generation?: number
): NativeTurnPin | null {
prune();
const key = nativeCodexTurnKey(body, comboName);
return key ? (pins.get(key) ?? null) : null;
if (!key) return null;
const rec = turns.get(key);
if (!rec) return null;
const gen = generation !== undefined ? generation : rec.activeGeneration;
return rec.pins.get(gen) ?? null;
}
export function getNativeCodexTurnActiveGeneration(
body: Record<string, unknown>,
comboName: string
): number {
prune();
const key = nativeCodexTurnKey(body, comboName);
if (!key) return 0;
return turns.get(key)?.activeGeneration ?? 0;
}
export function advanceNativeCodexTurnGeneration(
body: Record<string, unknown>,
comboName: string
): number | null {
prune();
const key = nativeCodexTurnKey(body, comboName);
if (!key) return null;
const rec = turns.get(key);
if (!rec) return null;
rec.activeGeneration += 1;
const now = Date.now();
rec.expiresAt = now + TTL_MS;
return rec.activeGeneration;
}
export function pinNativeCodexTurn(args: {
@@ -79,10 +121,27 @@ export function pinNativeCodexTurn(args: {
comboName: string;
target: ResolvedComboTarget;
connectionId: string;
generation?: number;
}): void {
const key = nativeCodexTurnKey(args.body, args.comboName);
if (!key || !args.connectionId) return;
const existing = pins.get(key);
let rec = turns.get(key);
const now = Date.now();
if (!rec) {
const metadata = turnMetadata(args.body);
rec = {
comboName: args.comboName,
threadId: typeof metadata?.thread_id === "string" ? metadata.thread_id : "",
turnId: typeof metadata?.turn_id === "string" ? metadata.turn_id : "",
activeGeneration: 0,
pins: new Map(),
createdAt: now,
expiresAt: now + TTL_MS,
};
turns.set(key, rec);
}
const gen = args.generation !== undefined ? args.generation : rec.activeGeneration;
const existing = rec.pins.get(gen);
if (
existing &&
(existing.modelStr !== args.target.modelStr || existing.provider !== args.target.provider)
@@ -91,15 +150,16 @@ export function pinNativeCodexTurn(args: {
}
// ConnectionId changes are allowed (failover to sibling connection)
// as long as provider + model stay the same.
const now = Date.now();
pins.set(key, {
rec.pins.set(gen, {
comboName: args.comboName,
modelStr: args.target.modelStr,
provider: args.target.provider,
connectionId: args.connectionId,
createdAt: existing?.createdAt ?? now,
expiresAt: now + TTL_MS,
generation: gen,
});
rec.expiresAt = now + TTL_MS;
prune(now);
}
@@ -153,10 +213,16 @@ export function applyNativeCodexTurnPin(
export function revokeNativeCodexTurnPinsForConnection(connectionId: string): number {
let revoked = 0;
for (const [key, pin] of pins) {
if (pin.connectionId !== connectionId) continue;
pins.delete(key);
revoked += 1;
for (const [key, rec] of turns) {
for (const [gen, pin] of rec.pins) {
if (pin.connectionId === connectionId) {
rec.pins.delete(gen);
revoked += 1;
}
}
if (rec.pins.size === 0) {
turns.delete(key);
}
}
return revoked;
}
@@ -282,9 +348,408 @@ export async function areAllPinnedTargetsModelScopedUnusable(
export function releaseNativeCodexTurnPin(body: Record<string, unknown>, comboName: string): void {
const key = nativeCodexTurnKey(body, comboName);
if (key) pins.delete(key);
if (key) turns.delete(key);
}
export function clearNativeCodexTurnPinsForTests(): void {
pins.clear();
turns.clear();
}
export function hasUnresolvedToolCalls(body: Record<string, unknown>): boolean {
const input: unknown[] = Array.isArray(body.input)
? body.input
: Array.isArray(body.messages)
? body.messages
: [];
if (input.length === 0) return false;
const callCounts = new Map<string, number>();
const outputCounts = new Map<string, number>();
const processPart = (part: unknown): boolean => {
if (!part || typeof part !== "object") return true;
const rec = part as Record<string, unknown>;
const type = typeof rec.type === "string" ? rec.type : "";
if (type === "function_call" || type === "custom_tool_call") {
const callId =
(typeof rec.call_id === "string" && rec.call_id.trim() ? rec.call_id.trim() : "") ||
(typeof rec.id === "string" && rec.id.trim() ? rec.id.trim() : "");
if (!callId) return false;
callCounts.set(callId, (callCounts.get(callId) ?? 0) + 1);
} else if (type === "function_call_output" || type === "custom_tool_call_output") {
const callId =
(typeof rec.call_id === "string" && rec.call_id.trim() ? rec.call_id.trim() : "") ||
(typeof rec.id === "string" && rec.id.trim() ? rec.id.trim() : "");
if (!callId) return false;
outputCounts.set(callId, (outputCounts.get(callId) ?? 0) + 1);
} else if (type === "tool_use") {
const callId = typeof rec.id === "string" && rec.id.trim() ? rec.id.trim() : "";
if (!callId) return false;
callCounts.set(callId, (callCounts.get(callId) ?? 0) + 1);
} else if (type === "tool_result") {
const callId =
(typeof rec.tool_use_id === "string" && rec.tool_use_id.trim()
? rec.tool_use_id.trim()
: "") || (typeof rec.id === "string" && rec.id.trim() ? rec.id.trim() : "");
if (!callId) return false;
outputCounts.set(callId, (outputCounts.get(callId) ?? 0) + 1);
}
return true;
};
for (const item of input) {
if (!item || typeof item !== "object") continue;
const rec = item as Record<string, unknown>;
const role = typeof rec.role === "string" ? rec.role : "";
if (rec.function_call && typeof rec.function_call === "object" && role === "assistant") {
return true;
}
if (role === "function") {
return true;
}
if (!processPart(rec)) return true;
if (role === "assistant" && Array.isArray(rec.tool_calls)) {
for (const tc of rec.tool_calls) {
if (!tc || typeof tc !== "object") return true;
const id =
typeof (tc as Record<string, unknown>).id === "string"
? ((tc as Record<string, unknown>).id as string).trim()
: "";
if (!id) return true;
callCounts.set(id, (callCounts.get(id) ?? 0) + 1);
}
} else if (role === "tool") {
const toolCallId =
(typeof rec.tool_call_id === "string" && rec.tool_call_id.trim()
? rec.tool_call_id.trim()
: "") ||
(typeof rec.call_id === "string" && rec.call_id.trim() ? rec.call_id.trim() : "") ||
(typeof rec.id === "string" && rec.id.trim() ? rec.id.trim() : "");
if (!toolCallId) return true;
outputCounts.set(toolCallId, (outputCounts.get(toolCallId) ?? 0) + 1);
}
if (Array.isArray(rec.content)) {
for (const part of rec.content) {
if (!processPart(part)) return true;
}
}
if (Array.isArray(rec.output)) {
for (const part of rec.output) {
if (!processPart(part)) return true;
}
}
}
if (callCounts.size === 0 && outputCounts.size === 0) return false;
if (callCounts.size !== outputCounts.size) return true;
for (const [id, count] of callCounts) {
if (count !== 1) return true;
if (outputCounts.get(id) !== 1) return true;
}
for (const [id, count] of outputCounts) {
if (count !== 1) return true;
if (!callCounts.has(id)) return true;
}
return false;
}
function isUnsafeItemOrPart(rec: Record<string, unknown>): boolean {
const type = typeof rec.type === "string" ? rec.type : "";
if (type === "item_reference" || type === "redacted_thinking") return true;
if (type === "encrypted_content") return true;
if (typeof rec.previous_response_id === "string" && rec.previous_response_id.trim() !== "")
return true;
if (typeof rec.previousResponseId === "string" && rec.previousResponseId.trim() !== "")
return true;
if (typeof rec.continuation_token === "string" && rec.continuation_token.trim() !== "")
return true;
if (typeof rec.continuationToken === "string" && rec.continuationToken.trim() !== "") return true;
if (typeof rec.encrypted_content === "string" && rec.encrypted_content.trim() !== "") return true;
if (typeof rec.encryptedContent === "string" && rec.encryptedContent.trim() !== "") return true;
if (typeof rec.encrypted_reasoning === "string" && rec.encrypted_reasoning.trim() !== "")
return true;
if (typeof rec.encryptedReasoning === "string" && rec.encryptedReasoning.trim() !== "")
return true;
if (typeof rec.thought_signature === "string" && rec.thought_signature.trim() !== "") return true;
if (typeof rec.thoughtSignature === "string" && rec.thoughtSignature.trim() !== "") return true;
if (typeof rec.signature === "string" && rec.signature.trim() !== "") return true;
if (
rec.provider_metadata &&
typeof rec.provider_metadata === "object" &&
Object.keys(rec.provider_metadata as object).length > 0
) {
return true;
}
if (
rec.providerMetadata &&
typeof rec.providerMetadata === "object" &&
Object.keys(rec.providerMetadata as object).length > 0
) {
return true;
}
if (
rec.provider_data &&
typeof rec.provider_data === "object" &&
Object.keys(rec.provider_data as object).length > 0
) {
return true;
}
if (
rec.providerData &&
typeof rec.providerData === "object" &&
Object.keys(rec.providerData as object).length > 0
) {
return true;
}
return false;
}
export function hasProviderSpecificUnsafeContinuationState(
body: Record<string, unknown>,
_activePin?: NativeTurnPin
): boolean {
if (typeof body.previous_response_id === "string" && body.previous_response_id.trim() !== "") {
return true;
}
if (typeof body.previousResponseId === "string" && body.previousResponseId.trim() !== "") {
return true;
}
if (typeof body.continuation_token === "string" && body.continuation_token.trim() !== "") {
return true;
}
if (typeof body.continuationToken === "string" && body.continuationToken.trim() !== "") {
return true;
}
if (typeof body.response_id === "string" && body.response_id.trim() !== "") {
return true;
}
if (typeof body.responseId === "string" && body.responseId.trim() !== "") {
return true;
}
if (typeof body.parent_response_id === "string" && body.parent_response_id.trim() !== "") {
return true;
}
if (typeof body.parentResponseId === "string" && body.parentResponseId.trim() !== "") {
return true;
}
if (typeof body.thought_signature === "string" && body.thought_signature.trim() !== "") {
return true;
}
if (typeof body.thoughtSignature === "string" && body.thoughtSignature.trim() !== "") {
return true;
}
if (typeof body.signature === "string" && body.signature.trim() !== "") {
return true;
}
if (typeof body.conversation_id === "string" && body.conversation_id.trim() !== "") {
return true;
}
if (typeof body.conversationId === "string" && body.conversationId.trim() !== "") {
return true;
}
if (
body.conversation &&
typeof body.conversation === "object" &&
Object.keys(body.conversation as object).length > 0
) {
return true;
}
if (
body.provider_metadata &&
typeof body.provider_metadata === "object" &&
Object.keys(body.provider_metadata as object).length > 0
) {
return true;
}
if (
body.providerMetadata &&
typeof body.providerMetadata === "object" &&
Object.keys(body.providerMetadata as object).length > 0
) {
return true;
}
if (
body.provider_data &&
typeof body.provider_data === "object" &&
Object.keys(body.provider_data as object).length > 0
) {
return true;
}
if (
body.providerData &&
typeof body.providerData === "object" &&
Object.keys(body.providerData as object).length > 0
) {
return true;
}
const input: unknown[] = Array.isArray(body.input)
? body.input
: Array.isArray(body.messages)
? body.messages
: [];
for (const item of input) {
if (!item || typeof item !== "object") continue;
const rec = item as Record<string, unknown>;
if (isUnsafeItemOrPart(rec)) return true;
if (Array.isArray(rec.content)) {
for (const part of rec.content) {
if (
part &&
typeof part === "object" &&
isUnsafeItemOrPart(part as Record<string, unknown>)
) {
return true;
}
}
}
if (Array.isArray(rec.output)) {
for (const outItem of rec.output) {
if (
outItem &&
typeof outItem === "object" &&
isUnsafeItemOrPart(outItem as Record<string, unknown>)
) {
return true;
}
}
}
if (Array.isArray(rec.summary)) {
for (const sumItem of rec.summary) {
if (
sumItem &&
typeof sumItem === "object" &&
isUnsafeItemOrPart(sumItem as Record<string, unknown>)
) {
return true;
}
}
}
if (Array.isArray(rec.tool_calls)) {
for (const tc of rec.tool_calls) {
if (tc && typeof tc === "object" && isUnsafeItemOrPart(tc as Record<string, unknown>)) {
return true;
}
}
}
}
return false;
}
export interface CanAutoResumeNativeCodexTurnOptions {
body: Record<string, unknown>;
comboName: string;
activePin: NativeTurnPin;
allTargets: ResolvedComboTarget[];
resilienceSettings?: ResilienceSettings | null;
quotaCutoffResetWindowConfig?: ResetWindowConfig;
isModelAvailable?: IsModelAvailable;
log?: ComboLogger;
}
export type AutoResumeDecision =
| {
eligible: true;
nextGeneration: number;
previousPin: NativeTurnPin;
selectedTarget: ResolvedComboTarget;
}
| {
eligible: false;
reason: string;
details?: Record<string, unknown>;
};
export async function canAutoResumeNativeCodexTurn(
options: CanAutoResumeNativeCodexTurnOptions
): Promise<AutoResumeDecision> {
const {
body,
comboName,
activePin,
allTargets,
resilienceSettings,
quotaCutoffResetWindowConfig,
isModelAvailable,
} = options;
const key = nativeCodexTurnKey(body, comboName);
if (!key) return { eligible: false, reason: "invalid_turn_key" };
const rec = turns.get(key);
const currentGen = rec?.activeGeneration ?? 0;
if (currentGen >= MAX_AUTORESUMES_PER_TURN) {
return { eligible: false, reason: "max_resumes_exceeded" };
}
const inputList = Array.isArray(body.input)
? body.input
: Array.isArray(body.messages)
? body.messages
: null;
if (!inputList || inputList.length === 0) {
return { eligible: false, reason: "missing_or_empty_input" };
}
if (hasUnresolvedToolCalls(body)) {
return { eligible: false, reason: "pending_tool_call" };
}
if (hasProviderSpecificUnsafeContinuationState(body, activePin)) {
return { eligible: false, reason: "unsafe_provider_state" };
}
const alternateTargets = allTargets.filter(
(t) => t.modelStr !== activePin.modelStr || t.provider !== activePin.provider
);
if (alternateTargets.length === 0) {
return { eligible: false, reason: "no_alternate_target" };
}
let selectedTarget: ResolvedComboTarget | null = null;
for (const alt of alternateTargets) {
if (alt.provider && alt.provider !== "unknown") {
const cb = getCircuitBreaker(alt.provider);
if (cb.getStatus().state === "OPEN") continue;
}
if (
resilienceSettings?.providerCooldown?.enabled &&
(isProviderInCooldown(alt.provider, alt.connectionId || undefined, resilienceSettings) ||
isProviderInCooldown(alt.provider, undefined, resilienceSettings))
) {
continue;
}
const unusable = await isPinnedTargetModelScopedUnusable({
target: alt,
resilienceSettings,
quotaCutoffResetWindowConfig,
comboName,
body,
isModelAvailable,
});
if (!unusable) {
selectedTarget = alt;
break;
}
}
if (!selectedTarget) {
return { eligible: false, reason: "no_healthy_alternate_target" };
}
return {
eligible: true,
nextGeneration: currentGen + 1,
previousPin: activePin,
selectedTarget,
};
}

View File

@@ -0,0 +1,416 @@
import test from "node:test";
import assert from "node:assert/strict";
// Pure guard predicates of the native Codex auto-resume path (#13180). Split out of
// tests/unit/native-codex-auto-resume.test.ts (which exercises the full combo flow
// against a scratch DB) so each file stays under the 1200-line test cap; these three
// cases need no DB, no combo config and no fixtures.
const {
hasUnresolvedToolCalls,
hasProviderSpecificUnsafeContinuationState,
MAX_AUTORESUMES_PER_TURN,
} = await import("../../open-sse/services/combo/nativeCodexTurnPin.ts");
test("MAX_AUTORESUMES_PER_TURN constant is 1", () => {
assert.equal(MAX_AUTORESUMES_PER_TURN, 1);
});
test("hasUnresolvedToolCalls correctly validates 1:1 call-output pairs and rejects duplicates/orphans/nested", () => {
// Empty input: no tool calls
assert.equal(hasUnresolvedToolCalls({}), false);
assert.equal(hasUnresolvedToolCalls({ input: [] }), false);
// Nested output array with unresolved tool_use is unsafe (true)
assert.equal(
hasUnresolvedToolCalls({
input: [
{
type: "function_call_output",
call_id: "c1",
output: [{ type: "tool_use", id: "tu-nested", name: "bash" }],
},
],
}),
true
);
// Two calls with same call_id and two outputs with same call_id (count=2 != 1) is unsafe (true)
assert.equal(
hasUnresolvedToolCalls({
input: [
{ type: "function_call", call_id: "c-dup2", name: "cat", arguments: "{}" },
{ type: "function_call", call_id: "c-dup2", name: "cat", arguments: "{}" },
{ type: "function_call_output", call_id: "c-dup2", output: "out1" },
{ type: "function_call_output", call_id: "c-dup2", output: "out2" },
],
}),
true
);
// Two distinct calls with two distinct matching outputs is safe (false)
assert.equal(
hasUnresolvedToolCalls({
input: [
{ type: "function_call", call_id: "c-1", name: "cat", arguments: "{}" },
{ type: "function_call", call_id: "c-2", name: "ls", arguments: "{}" },
{ type: "function_call_output", call_id: "c-1", output: "out1" },
{ type: "function_call_output", call_id: "c-2", output: "out2" },
],
}),
false
);
// Resolved function call (1 call, 1 matching output)
assert.equal(
hasUnresolvedToolCalls({
input: [
{ type: "message", role: "user", content: "read file" },
{ type: "function_call", call_id: "call-1", name: "cat", arguments: "{}" },
{ type: "function_call_output", call_id: "call-1", output: "hello world" },
],
}),
false
);
// Unresolved function call (call with no output)
assert.equal(
hasUnresolvedToolCalls({
input: [
{ type: "message", role: "user", content: "read file" },
{ type: "function_call", call_id: "call-1", name: "cat", arguments: "{}" },
],
}),
true
);
// Resolved custom tool call
assert.equal(
hasUnresolvedToolCalls({
input: [
{ type: "message", role: "user", content: "patch file" },
{ type: "custom_tool_call", call_id: "call-2", name: "apply_patch", input: "diff" },
{ type: "custom_tool_call_output", call_id: "call-2", output: "ok" },
],
}),
false
);
// Unresolved custom tool call
assert.equal(
hasUnresolvedToolCalls({
input: [
{ type: "message", role: "user", content: "patch file" },
{ type: "custom_tool_call", call_id: "call-2", name: "apply_patch", input: "diff" },
],
}),
true
);
// Anthropic tool_use and tool_result in content array (resolved)
assert.equal(
hasUnresolvedToolCalls({
messages: [
{
role: "assistant",
content: [{ type: "tool_use", id: "tu-1", name: "bash", input: {} }],
},
{
role: "user",
content: [{ type: "tool_result", tool_use_id: "tu-1", content: "done" }],
},
],
}),
false
);
// Anthropic tool_use in content array (unresolved)
assert.equal(
hasUnresolvedToolCalls({
messages: [
{
role: "assistant",
content: [{ type: "tool_use", id: "tu-1", name: "bash", input: {} }],
},
],
}),
true
);
// Assistant message tool_calls format (resolved)
assert.equal(
hasUnresolvedToolCalls({
input: [
{
type: "message",
role: "assistant",
tool_calls: [{ id: "call-3", type: "function", function: { name: "shell" } }],
},
{ type: "message", role: "tool", tool_call_id: "call-3", content: "done" },
],
}),
false
);
// Assistant message tool_calls format (unresolved)
assert.equal(
hasUnresolvedToolCalls({
input: [
{
type: "message",
role: "assistant",
tool_calls: [{ id: "call-3", type: "function", function: { name: "shell" } }],
},
],
}),
true
);
// Duplicate tool call ID: two calls with same ID, one output -> unsafe (true)
assert.equal(
hasUnresolvedToolCalls({
input: [
{ type: "function_call", call_id: "call-dup", name: "cat", arguments: "{}" },
{ type: "function_call", call_id: "call-dup", name: "cat", arguments: "{}" },
{ type: "function_call_output", call_id: "call-dup", output: "res" },
],
}),
true
);
// Duplicate tool output ID: one call, two outputs with same ID -> unsafe (true)
assert.equal(
hasUnresolvedToolCalls({
input: [
{ type: "function_call", call_id: "call-dup-out", name: "cat", arguments: "{}" },
{ type: "function_call_output", call_id: "call-dup-out", output: "res1" },
{ type: "function_call_output", call_id: "call-dup-out", output: "res2" },
],
}),
true
);
// Orphaned tool output: output without matching call -> unsafe (true)
assert.equal(
hasUnresolvedToolCalls({
input: [{ type: "function_call_output", call_id: "orphan-call", output: "res" }],
}),
true
);
// Malformed tool call with empty call_id -> unsafe (true)
assert.equal(
hasUnresolvedToolCalls({
input: [{ type: "function_call", call_id: "", name: "cat", arguments: "{}" }],
}),
true
);
// Legacy unidentifiable function_call -> unsafe (true)
assert.equal(
hasUnresolvedToolCalls({
input: [{ role: "assistant", function_call: { name: "test", arguments: "{}" } }],
}),
true
);
});
test("hasProviderSpecificUnsafeContinuationState detects opaque provider state at all levels", () => {
// Clean input: safe
assert.equal(
hasProviderSpecificUnsafeContinuationState({
input: [{ type: "message", role: "user", content: "hello" }],
}),
false
);
// conversation_id at root: unsafe
assert.equal(
hasProviderSpecificUnsafeContinuationState({
conversation_id: "conv_12345",
input: [{ type: "message", role: "user", content: "hello" }],
}),
true
);
// conversation object at root: unsafe
assert.equal(
hasProviderSpecificUnsafeContinuationState({
conversation: { id: "conv_67890" },
input: [{ type: "message", role: "user", content: "hello" }],
}),
true
);
// Item with item-level previous_response_id: unsafe
assert.equal(
hasProviderSpecificUnsafeContinuationState({
input: [
{
type: "message",
role: "assistant",
previous_response_id: "resp_nested_prev",
content: "hello",
},
],
}),
true
);
// Item with item-level continuation_token: unsafe
assert.equal(
hasProviderSpecificUnsafeContinuationState({
input: [
{
type: "message",
role: "assistant",
continuation_token: "tok_nested_cont",
content: "hello",
},
],
}),
true
);
// previous_response_id: unsafe (binds to upstream response store)
assert.equal(
hasProviderSpecificUnsafeContinuationState({
previous_response_id: "resp_12345_upstream",
input: [{ type: "message", role: "user", content: "hello" }],
}),
true
);
// continuation_token: unsafe
assert.equal(
hasProviderSpecificUnsafeContinuationState({
continuation_token: "tok_opaque_blob",
input: [{ type: "message", role: "user", content: "hello" }],
}),
true
);
// response_id: unsafe
assert.equal(
hasProviderSpecificUnsafeContinuationState({
response_id: "resp_999",
input: [{ type: "message", role: "user", content: "hello" }],
}),
true
);
// provider_metadata at root: unsafe
assert.equal(
hasProviderSpecificUnsafeContinuationState({
provider_metadata: { openai: { message_id: "m1" } },
input: [{ type: "message", role: "user", content: "hello" }],
}),
true
);
// item_reference: unsafe (server-side item ID)
assert.equal(
hasProviderSpecificUnsafeContinuationState({
input: [{ type: "item_reference", id: "item_abc123" }],
}),
true
);
// reasoning item with encrypted_content: unsafe
assert.equal(
hasProviderSpecificUnsafeContinuationState({
input: [
{ type: "reasoning", encrypted_content: "enc_blob_xyz" },
{ type: "message", role: "user", content: "hello" },
],
}),
true
);
// thinking item with thought_signature: unsafe
assert.equal(
hasProviderSpecificUnsafeContinuationState({
input: [
{ type: "thinking", thought_signature: "sig_gemini_blob" },
{ type: "message", role: "user", content: "hello" },
],
}),
true
);
// redacted_thinking item: unsafe
assert.equal(
hasProviderSpecificUnsafeContinuationState({
input: [{ type: "redacted_thinking", data: "redacted" }],
}),
true
);
// Nested thinking part inside content array with signature: unsafe
assert.equal(
hasProviderSpecificUnsafeContinuationState({
messages: [
{
role: "assistant",
content: [
{ type: "thinking", thinking: "deep thought", signature: "sig-xyz" },
{ type: "text", text: "hello" },
],
},
],
}),
true
);
// encrypted_content item: unsafe
assert.equal(
hasProviderSpecificUnsafeContinuationState({
input: [{ type: "encrypted_content", encrypted_content: "enc_123" }],
}),
true
);
// Root body thoughtSignature (camelCase): unsafe
assert.equal(
hasProviderSpecificUnsafeContinuationState({
thoughtSignature: "sig_camel_case",
input: [{ type: "message", role: "user", content: "hello" }],
}),
true
);
// Root body provider_data: unsafe
assert.equal(
hasProviderSpecificUnsafeContinuationState({
provider_data: { gemini: { candidate_token_count: 50 } },
input: [{ type: "message", role: "user", content: "hello" }],
}),
true
);
// Nested output array with encrypted_content: unsafe
assert.equal(
hasProviderSpecificUnsafeContinuationState({
input: [
{
type: "function_call_output",
call_id: "c1",
output: [{ type: "encrypted_content", encrypted_content: "enc_blob" }],
},
],
}),
true
);
// Nested summary array with thought_signature: unsafe
assert.equal(
hasProviderSpecificUnsafeContinuationState({
input: [
{
type: "reasoning",
summary: [{ type: "summary_text", text: "...", thought_signature: "sig" }],
},
],
}),
true
);
});

File diff suppressed because it is too large Load Diff