mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-09-22 06:42:19 +03:00
Re-land of #13180 by @mdigitalbh81 (their commits carried with authorship intact), merged via /merge-batch (2026-09-19) on top of the current `release/v3.8.51` tip. **Reconciled before landing (three maintainer commits on top of the re-land):** - `81d770b3` — `check:api-typecheck` / `check:open-sse-typecheck` were red on the PR's own head with `TS2339 'reason' does not exist on type 'AutoResumeDecision'` ×3 in `open-sse/services/combo.ts` (green on the tip). Without `strictNullChecks` the truthiness test on the `eligible` discriminant does not narrow the else-branch; `=== true` does. - `66e57cfe` — `tests/unit/native-codex-auto-resume.test.ts` was 1540 lines, above the 1200-line new-test cap. The three pure guard cases (`hasUnresolvedToolCalls`, `hasProviderSpecificUnsafeContinuationState`, `MAX_AUTORESUMES_PER_TURN`) moved verbatim to `native-codex-auto-resume-guards.test.ts`; the combo-flow file is now 1131 lines. - `c2f1937f` — `executeTargetAttempt.ts` 1258→1273 rebaselined with a dated justification (own growth: generation advance + log at the pin site). **Evidence on the merged tree:** `native-codex-auto-resume{,-guards}.test.ts` 15/15; `native-codex-turn-pin-model-scoped-fallback`, `native-codex-turn-pin-10379`, `chatgpt-web-codex-turn-pin` 33/33 total; broader combo sweep (`combo-responses-sse-failure-fallback`, `combo-context-window-filter`, `perf-waterfall-elimination`, `chatgpt-web-codex`, `combo-context-overflow-compression-probe`) 70/70; api/open-sse typecheck clean for the PR's files; file-size, changelog-integrity, complexity and cognitive-complexity gates OK. **Inherited, not from this PR** (reproduced on the pure tip): the 23 unit reds in the fast-path shards (vi locale parity, pack-artifact allowlists, `.env.example` sync, casing, budget fallback, etc.), the 5 `no-unused-vars` lint errors (`cliRuntime.ts`, `arena-elo-sync-redesign`, `compressionAnalyticsWriterFlatRate`, `waitForServer-slow-first-response`), the `omni-version-manager` generated-skill drift, `tinycmsDomMocks.ts` / `rerankProviderNodes.ts` / `antigravity.ts` typecheck errors, and the 4 env vars missing from `.env.example`. Supersedes #13180. Follow-up to #12240.
This commit is contained in:
committed by
GitHub
parent
3fd1265d89
commit
cd4c6f6947
1
changelog.d/fixes/13180-native-codex-turn-auto-resume.md
Normal file
1
changelog.d/fixes/13180-native-codex-turn-auto-resume.md
Normal file
@@ -0,0 +1 @@
|
|||||||
|
- **fix(combo):** auto-resume a pinned native Codex turn on a healthy sibling connection or model when the pinned provider becomes unavailable for a model-scoped reason (quota, model lockout) instead of failing the turn outright — provider-wide circuit-breaker/cooldown state, pending tool calls, opaque continuation state, and partial streams still block resume, and at most one auto-resume happens per logical turn ([#13180](https://github.com/diegosouzapw/OmniRoute/pull/13180)) — thanks @mdigitalbh81
|
||||||
@@ -481,7 +481,7 @@
|
|||||||
"open-sse/services/accountFallback.ts": 2517,
|
"open-sse/services/accountFallback.ts": 2517,
|
||||||
"open-sse/services/adobeFireflyBrowserLogin.ts": 1401,
|
"open-sse/services/adobeFireflyBrowserLogin.ts": 1401,
|
||||||
"open-sse/services/combo.ts": 4080,
|
"open-sse/services/combo.ts": 4080,
|
||||||
"open-sse/services/combo/executeTargetAttempt.ts": 1258,
|
"open-sse/services/combo/executeTargetAttempt.ts": 1273,
|
||||||
"open-sse/translator/response/openai-responses.ts": 1518,
|
"open-sse/translator/response/openai-responses.ts": 1518,
|
||||||
"open-sse/utils/cursorAgentProtobuf.ts": 1588,
|
"open-sse/utils/cursorAgentProtobuf.ts": 1588,
|
||||||
"open-sse/utils/proxyFetch.ts": 1276,
|
"open-sse/utils/proxyFetch.ts": 1276,
|
||||||
@@ -719,5 +719,6 @@
|
|||||||
"_rebaseline_2026_09_17_11742_log_boundary_hardening": "PR #11742 (rebase para release/v3.8.51): open-sse/handlers/chatCore.ts 6219->6287. O crescimento e a unica parte da PR que sobreviveu ao tip: endurecimento da fronteira de LOG (mais amplo que a Hard Rule #12, que cobre respostas). Sao 136 linhas adicionadas, das quais ~40 sao chamadas diretas de sanitizacao — sanitizeErrorMessage em erro de plugin (onError), em timeout de semaforo e na failureMessage antes de ela chegar ao console.log e ao call-log; sanitizeUpstreamDetails no log de resposta malformada; getSafeErrorMetadata + try/catch nos pontos onde metadata hostil (Proxy) podia lançar. O resto da PR foi descartado por ja estar no tip (#12506/#12945/#13635 error boundaries, #12429 wreq-js, #11754 aposentadoria do ChatGPT Web) — open-sse/utils/ difere do tip por UMA linha (registro do identificador publico lmarena_stream_error).",
|
"_rebaseline_2026_09_17_11742_log_boundary_hardening": "PR #11742 (rebase para release/v3.8.51): open-sse/handlers/chatCore.ts 6219->6287. O crescimento e a unica parte da PR que sobreviveu ao tip: endurecimento da fronteira de LOG (mais amplo que a Hard Rule #12, que cobre respostas). Sao 136 linhas adicionadas, das quais ~40 sao chamadas diretas de sanitizacao — sanitizeErrorMessage em erro de plugin (onError), em timeout de semaforo e na failureMessage antes de ela chegar ao console.log e ao call-log; sanitizeUpstreamDetails no log de resposta malformada; getSafeErrorMetadata + try/catch nos pontos onde metadata hostil (Proxy) podia lançar. O resto da PR foi descartado por ja estar no tip (#12506/#12945/#13635 error boundaries, #12429 wreq-js, #11754 aposentadoria do ChatGPT Web) — open-sse/utils/ difere do tip por UMA linha (registro do identificador publico lmarena_stream_error).",
|
||||||
"_rebaseline_2026_09_17_13670_allow_auto_combos": "PR #13670 (@fouadSalkini): per-key allowAutoCombos para gatear os combos auto/* embutidos. src/app/api/v1/models/catalog.ts 2075->2117 e src/lib/db/apiKeys.ts 1625->1659. Crescimento e 100% proprio da PR, nao herdado: medido no tip puro, catalog.ts esta em 2074 (abaixo do teto 2075) e apiKeys.ts em 1620 (abaixo de 1625). O aumento e a propria feature — o campo de permissao por chave precisa ser lido, validado e propagado ate o filtro do catalogo, e cada ponto e chamada explicita, nao extraivel sem esconder o gate. Coberto pelos 25 testes da PR. As demais violacoes desta arvore (chatHelpers.ts, chatCore.ts e tests/unit/chatcore-translation-paths.test.ts) sao base-red herdado do tip e nao foram tocadas aqui.",
|
"_rebaseline_2026_09_17_13670_allow_auto_combos": "PR #13670 (@fouadSalkini): per-key allowAutoCombos para gatear os combos auto/* embutidos. src/app/api/v1/models/catalog.ts 2075->2117 e src/lib/db/apiKeys.ts 1625->1659. Crescimento e 100% proprio da PR, nao herdado: medido no tip puro, catalog.ts esta em 2074 (abaixo do teto 2075) e apiKeys.ts em 1620 (abaixo de 1625). O aumento e a propria feature — o campo de permissao por chave precisa ser lido, validado e propagado ate o filtro do catalogo, e cada ponto e chamada explicita, nao extraivel sem esconder o gate. Coberto pelos 25 testes da PR. As demais violacoes desta arvore (chatHelpers.ts, chatCore.ts e tests/unit/chatcore-translation-paths.test.ts) sao base-red herdado do tip e nao foram tocadas aqui.",
|
||||||
"_rebaseline_2026_09_17c_chatcore_translation_paths_test": "tests/unit/chatcore-translation-paths.test.ts 3447->3449 (#13173, prefixos de cache de meio de conversa do Fable — as assercoes novas do caso). Ultimo teto remanescente da leva de merges de 2026-09-17; os outros dois (chatHelpers.ts e chatCore.ts) foram absorvidos pelos rebaselines das proprias PRs que mergearam depois. Medido no tip limpo.",
|
"_rebaseline_2026_09_17c_chatcore_translation_paths_test": "tests/unit/chatcore-translation-paths.test.ts 3447->3449 (#13173, prefixos de cache de meio de conversa do Fable — as assercoes novas do caso). Ultimo teto remanescente da leva de merges de 2026-09-17; os outros dois (chatHelpers.ts e chatCore.ts) foram absorvidos pelos rebaselines das proprias PRs que mergearam depois. Medido no tip limpo.",
|
||||||
"_rebaseline_2026_09_18_13929_antigravity_account_lease_merge": "PR #13929 (Re-land of #10011, @Ardem2025 via @diegosouzapw): the Antigravity account lease, merged onto the current release/v3.8.51 tip (which had independently moved chat.ts to 2520 and auth.ts to 3557 via unrelated PRs). Combined ceiling after merge: src/sse/handlers/chat.ts->2541, src/sse/services/auth.ts->3577. The lease registry, its lifecycle glue and its selection glue were extracted into three NEW modules (src/sse/services/antigravityRoutingState.ts, antigravityLeaseLifecycle.ts, antigravityLeaseSelection.ts) precisely to keep this growth to the call sites; what remains in chat.ts/auth.ts is the wiring itself, which cannot be moved out of the selection loop and the dispatch path. Every added hunk is inert unless ANTIGRAVITY_ACCOUNT_LEASE_ENABLED (default false) is on. Covered by tests/unit/antigravity-routing-state.test.ts, antigravity-lease-lifecycle.test.ts and antigravity-account-lease-flag.test.ts. UPDATE (re-sync 2026-09-18 after trains 3b/4d moved the tip): auth.ts 3577->3582 (same +29 own growth over a tip now at 3552). open-sse/executors/base.ts 1753->1754 is NOT this PR's growth — it is release-tip drift from train 3b (#13002 +5 / #13705 -4 net +1, both merged without a baseline entry); absorbed here by the captain session under the owner-approved train-rebaseline policy so the tip stops failing check:file-size for every PR boarding after it."
|
"_rebaseline_2026_09_18_13929_antigravity_account_lease_merge": "PR #13929 (Re-land of #10011, @Ardem2025 via @diegosouzapw): the Antigravity account lease, merged onto the current release/v3.8.51 tip (which had independently moved chat.ts to 2520 and auth.ts to 3557 via unrelated PRs). Combined ceiling after merge: src/sse/handlers/chat.ts->2541, src/sse/services/auth.ts->3577. The lease registry, its lifecycle glue and its selection glue were extracted into three NEW modules (src/sse/services/antigravityRoutingState.ts, antigravityLeaseLifecycle.ts, antigravityLeaseSelection.ts) precisely to keep this growth to the call sites; what remains in chat.ts/auth.ts is the wiring itself, which cannot be moved out of the selection loop and the dispatch path. Every added hunk is inert unless ANTIGRAVITY_ACCOUNT_LEASE_ENABLED (default false) is on. Covered by tests/unit/antigravity-routing-state.test.ts, antigravity-lease-lifecycle.test.ts and antigravity-account-lease-flag.test.ts. UPDATE (re-sync 2026-09-18 after trains 3b/4d moved the tip): auth.ts 3577->3582 (same +29 own growth over a tip now at 3552). open-sse/executors/base.ts 1753->1754 is NOT this PR's growth — it is release-tip drift from train 3b (#13002 +5 / #13705 -4 net +1, both merged without a baseline entry); absorbed here by the captain session under the owner-approved train-rebaseline policy so the tip stops failing check:file-size for every PR boarding after it.",
|
||||||
|
"_rebaseline_2026_09_19_14162_native_codex_auto_resume": "PR #14162 (re-land of #13180, @mdigitalbh81 via @diegosouzapw): native Codex turn auto-resume. open-sse/services/combo/executeTargetAttempt.ts 1258->1273 (+15). Growth is 100% the PR's own, measured against the clean tip (1258 there, gate green): the pin step now advances the logical turn generation and logs the resumed provider/model when the attempt is an auto-resume dispatch, and the generation is passed into pinNativeCodexTurn — the branch has to sit at the pin site because that is the only place the winning target and effective connection are known. Covered by tests/unit/native-codex-auto-resume.test.ts + native-codex-auto-resume-guards.test.ts (15/15) and #13564's native-codex-turn-pin-model-scoped-fallback.test.ts (7/7)."
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -90,6 +90,8 @@ export {
|
|||||||
import {
|
import {
|
||||||
applyNativeCodexTurnPin,
|
applyNativeCodexTurnPin,
|
||||||
areAllPinnedTargetsModelScopedUnusable,
|
areAllPinnedTargetsModelScopedUnusable,
|
||||||
|
canAutoResumeNativeCodexTurn,
|
||||||
|
createPinnedModelUnavailableResponse,
|
||||||
getNativeCodexTurnPin,
|
getNativeCodexTurnPin,
|
||||||
releaseNativeCodexTurnPin,
|
releaseNativeCodexTurnPin,
|
||||||
} from "./combo/nativeCodexTurnPin.ts";
|
} from "./combo/nativeCodexTurnPin.ts";
|
||||||
@@ -131,6 +133,22 @@ import { executeTargetAttempt } from "./combo/executeTargetAttempt.ts";
|
|||||||
import type { AttemptLoopDeps, AttemptLoopState } from "./combo/attemptLoopTypes.ts";
|
import type { AttemptLoopDeps, AttemptLoopState } from "./combo/attemptLoopTypes.ts";
|
||||||
import { clearStaleLKGP } from "./combo/staleLkgpClear.ts";
|
import { clearStaleLKGP } from "./combo/staleLkgpClear.ts";
|
||||||
|
|
||||||
|
// Native Codex auto-resume (#13180) rejection reasons that mean the turn either carries
|
||||||
|
// state unsafe to hand to an untested alternate model (pending tool calls, opaque
|
||||||
|
// provider-specific continuation state) or has already used its one allowed resume for
|
||||||
|
// this logical turn. These must terminate the turn rather than fall through to #13564's
|
||||||
|
// plain "release pin and route naturally" fallback. Every other reason (e.g. the request
|
||||||
|
// does not use the Responses-API input/messages shape #13180's eligibility check needs)
|
||||||
|
// falls through unchanged so non-native-turn-shaped Codex requests keep working exactly
|
||||||
|
// as before #13180.
|
||||||
|
const NATIVE_CODEX_AUTO_RESUME_UNSAFE_REASONS = new Set([
|
||||||
|
"pending_tool_call",
|
||||||
|
"unsafe_provider_state",
|
||||||
|
"no_alternate_target",
|
||||||
|
"no_healthy_alternate_target",
|
||||||
|
"max_resumes_exceeded",
|
||||||
|
]);
|
||||||
|
|
||||||
export { RESET_WINDOW_NAMES, QUOTA_SOFT_DEPRIORITIZE_FACTOR, setCandidateQuotaSoftPenalty };
|
export { RESET_WINDOW_NAMES, QUOTA_SOFT_DEPRIORITIZE_FACTOR, setCandidateQuotaSoftPenalty };
|
||||||
export { scoreAutoTargets, expandAutoComboCandidatePool };
|
export { scoreAutoTargets, expandAutoComboCandidatePool };
|
||||||
export type { SingleModelTarget, ResolvedComboTarget };
|
export type { SingleModelTarget, ResolvedComboTarget };
|
||||||
@@ -776,9 +794,10 @@ async function handleComboChatInner({
|
|||||||
});
|
});
|
||||||
if (runtimeUnitDispatch) return runtimeUnitDispatch;
|
if (runtimeUnitDispatch) return runtimeUnitDispatch;
|
||||||
|
|
||||||
const activeNativeTurnPin = clientManagedResponsesContext
|
let activeNativeTurnPin = clientManagedResponsesContext
|
||||||
? getNativeCodexTurnPin(body, combo.name)
|
? getNativeCodexTurnPin(body, combo.name)
|
||||||
: null;
|
: null;
|
||||||
|
let isAutoResuming = false;
|
||||||
|
|
||||||
// Route new round-robin turns to the specialized handler. A native Codex
|
// Route new round-robin turns to the specialized handler. A native Codex
|
||||||
// continuation with an established provider/account pin must use the common
|
// continuation with an established provider/account pin must use the common
|
||||||
@@ -858,15 +877,60 @@ async function handleComboChatInner({
|
|||||||
isModelAvailable,
|
isModelAvailable,
|
||||||
});
|
});
|
||||||
if (allPinnedUnusable) {
|
if (allPinnedUnusable) {
|
||||||
// All pinned provider+model targets are model-scoped unusable — release
|
const autoResumeEligibility = await canAutoResumeNativeCodexTurn({
|
||||||
// the pin and fall through to full combo routing so the turn can try
|
body: body as Record<string, unknown>,
|
||||||
// other models in the combo pool. This matches Claude Code's behavior
|
comboName: combo.name,
|
||||||
// where no turn pin allows natural multi-model fallback.
|
activePin: activeNativeTurnPin,
|
||||||
releaseNativeCodexTurnPin(body as Record<string, unknown>, combo.name);
|
allTargets: orderedTargets,
|
||||||
log.warn(
|
resilienceSettings,
|
||||||
"COMBO",
|
quotaCutoffResetWindowConfig,
|
||||||
`Native Codex turn pin released: pinned model ${activeNativeTurnPin.modelStr} model-scoped unavailable; falling back to full combo routing`
|
isModelAvailable,
|
||||||
);
|
log,
|
||||||
|
});
|
||||||
|
|
||||||
|
if (autoResumeEligibility.eligible === true) {
|
||||||
|
const selectedAlternate = autoResumeEligibility.selectedTarget;
|
||||||
|
log.info(
|
||||||
|
"COMBO",
|
||||||
|
`Native Codex auto-resume eligible: previous provider/model=${activeNativeTurnPin.provider}/${activeNativeTurnPin.modelStr}, previous logical turn generation=${autoResumeEligibility.previousPin.generation ?? 0}, reason=model_scoped_unavailable`
|
||||||
|
);
|
||||||
|
log.info(
|
||||||
|
"COMBO",
|
||||||
|
`Native Codex auto-resume started: previous provider/model=${activeNativeTurnPin.provider}/${activeNativeTurnPin.modelStr}, target provider/model=${selectedAlternate.provider}/${selectedAlternate.modelStr}, target generation=${autoResumeEligibility.nextGeneration}`
|
||||||
|
);
|
||||||
|
const alternateTargets = orderedTargets.filter(
|
||||||
|
(t) =>
|
||||||
|
t.modelStr === selectedAlternate.modelStr && t.provider === selectedAlternate.provider
|
||||||
|
);
|
||||||
|
orderedTargets = alternateTargets;
|
||||||
|
activeNativeTurnPin = null;
|
||||||
|
isAutoResuming = true;
|
||||||
|
} else if (NATIVE_CODEX_AUTO_RESUME_UNSAFE_REASONS.has(autoResumeEligibility.reason)) {
|
||||||
|
// These specific rejection reasons mean the turn carries state (pending
|
||||||
|
// tool calls, opaque provider-specific continuation state) or has
|
||||||
|
// already exhausted its resume budget, so handing it to an untested
|
||||||
|
// alternate model via natural combo routing (#13564's plain fallback)
|
||||||
|
// would be unsafe or would violate #13180's "at most one auto-resume
|
||||||
|
// per logical turn" bound. Terminate instead of falling through.
|
||||||
|
targetResolution.quotaShareRelease?.();
|
||||||
|
log.warn(
|
||||||
|
"COMBO",
|
||||||
|
`Native Codex turn cannot continue: pinned model ${activeNativeTurnPin.modelStr} is unavailable (model-scoped); auto-resume rejected (${autoResumeEligibility.reason}); preserving turn pin and terminating turn`
|
||||||
|
);
|
||||||
|
return createPinnedModelUnavailableResponse();
|
||||||
|
} else {
|
||||||
|
// Every other rejection reason (e.g. the request body does not carry
|
||||||
|
// the Responses-API `input`/`messages` shape #13180's eligibility
|
||||||
|
// check needs) means auto-resume simply cannot be evaluated — it says
|
||||||
|
// nothing about the request being unsafe. Fall back to the plain
|
||||||
|
// release-and-route-naturally behavior (#13564) so non-native-turn or
|
||||||
|
// legacy-shaped Codex requests keep working exactly as before #13180.
|
||||||
|
releaseNativeCodexTurnPin(body as Record<string, unknown>, combo.name);
|
||||||
|
log.warn(
|
||||||
|
"COMBO",
|
||||||
|
`Native Codex turn pin released: pinned model ${activeNativeTurnPin.modelStr} model-scoped unavailable; auto-resume not eligible (${autoResumeEligibility.reason}); falling back to full combo routing`
|
||||||
|
);
|
||||||
|
}
|
||||||
} else {
|
} else {
|
||||||
orderedTargets = pinnedTargets;
|
orderedTargets = pinnedTargets;
|
||||||
log.info(
|
log.info(
|
||||||
@@ -989,6 +1053,7 @@ async function handleComboChatInner({
|
|||||||
releaseStickyPinOnFailure,
|
releaseStickyPinOnFailure,
|
||||||
clearStaleLKGP,
|
clearStaleLKGP,
|
||||||
clientManagedResponsesContext,
|
clientManagedResponsesContext,
|
||||||
|
nativeCodexAutoResume: isAutoResuming,
|
||||||
reasoningTokenBufferEnabled,
|
reasoningTokenBufferEnabled,
|
||||||
stickyWeightedLimit,
|
stickyWeightedLimit,
|
||||||
getWeightedStepKeyForTarget,
|
getWeightedStepKeyForTarget,
|
||||||
|
|||||||
@@ -102,6 +102,7 @@ export type AttemptLoopDeps = {
|
|||||||
* gate tests keep compiling; attempt uses defaults when absent.
|
* gate tests keep compiling; attempt uses defaults when absent.
|
||||||
*/
|
*/
|
||||||
clientManagedResponsesContext?: boolean;
|
clientManagedResponsesContext?: boolean;
|
||||||
|
nativeCodexAutoResume?: boolean;
|
||||||
reasoningTokenBufferEnabled?: boolean;
|
reasoningTokenBufferEnabled?: boolean;
|
||||||
stickyWeightedLimit?: number;
|
stickyWeightedLimit?: number;
|
||||||
getWeightedStepKeyForTarget?: (target: ResolvedComboTarget) => string | null;
|
getWeightedStepKeyForTarget?: (target: ResolvedComboTarget) => string | null;
|
||||||
|
|||||||
@@ -71,7 +71,7 @@ import {
|
|||||||
isModelScoped400,
|
isModelScoped400,
|
||||||
} from "./comboPredicates.ts";
|
} from "./comboPredicates.ts";
|
||||||
import { applyComboTargetExhaustion } from "./targetExhaustion.ts";
|
import { applyComboTargetExhaustion } from "./targetExhaustion.ts";
|
||||||
import { pinNativeCodexTurn } from "./nativeCodexTurnPin.ts";
|
import { advanceNativeCodexTurnGeneration, pinNativeCodexTurn } from "./nativeCodexTurnPin.ts";
|
||||||
import { recordComboDecision } from "./decisionTrace.ts";
|
import { recordComboDecision } from "./decisionTrace.ts";
|
||||||
import { recordProviderCooldown } from "../providerCooldownTracker.ts";
|
import { recordProviderCooldown } from "../providerCooldownTracker.ts";
|
||||||
import {
|
import {
|
||||||
@@ -469,12 +469,27 @@ export async function executeTargetAttempt(opts: {
|
|||||||
}
|
}
|
||||||
|
|
||||||
if (Boolean(deps.clientManagedResponsesContext) && effectiveConnectionId) {
|
if (Boolean(deps.clientManagedResponsesContext) && effectiveConnectionId) {
|
||||||
pinNativeCodexTurn({
|
if (deps.nativeCodexAutoResume) {
|
||||||
body: deps.body,
|
const nextGen = advanceNativeCodexTurnGeneration(deps.body, deps.combo.name);
|
||||||
comboName: deps.combo.name,
|
deps.log.info(
|
||||||
target,
|
"COMBO",
|
||||||
connectionId: effectiveConnectionId,
|
`Native Codex auto-resume routed: new provider/model=${target.modelStr} on connection ${effectiveConnectionId.slice(0, 8)} (logical turn generation ${nextGen})`
|
||||||
});
|
);
|
||||||
|
pinNativeCodexTurn({
|
||||||
|
body: deps.body,
|
||||||
|
comboName: deps.combo.name,
|
||||||
|
target,
|
||||||
|
connectionId: effectiveConnectionId,
|
||||||
|
generation: nextGen ?? undefined,
|
||||||
|
});
|
||||||
|
} else {
|
||||||
|
pinNativeCodexTurn({
|
||||||
|
body: deps.body,
|
||||||
|
comboName: deps.combo.name,
|
||||||
|
target,
|
||||||
|
connectionId: effectiveConnectionId,
|
||||||
|
});
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Success decay: a healthy response walks the model's lockout failure
|
// Success decay: a healthy response walks the model's lockout failure
|
||||||
|
|||||||
@@ -13,18 +13,30 @@ import type { ComboLogger, IsModelAvailable } from "./types.ts";
|
|||||||
|
|
||||||
import type { ResolvedComboTarget } from "./types.ts";
|
import type { ResolvedComboTarget } from "./types.ts";
|
||||||
|
|
||||||
type NativeTurnPin = {
|
export type NativeTurnPin = {
|
||||||
comboName: string;
|
comboName: string;
|
||||||
modelStr: string;
|
modelStr: string;
|
||||||
provider: string;
|
provider: string;
|
||||||
connectionId: string;
|
connectionId: string;
|
||||||
createdAt: number;
|
createdAt: number;
|
||||||
expiresAt: number;
|
expiresAt: number;
|
||||||
|
generation?: number;
|
||||||
};
|
};
|
||||||
|
|
||||||
|
export interface NativeCodexTurnRecord {
|
||||||
|
comboName: string;
|
||||||
|
threadId: string;
|
||||||
|
turnId: string;
|
||||||
|
activeGeneration: number;
|
||||||
|
pins: Map<number, NativeTurnPin>;
|
||||||
|
createdAt: number;
|
||||||
|
expiresAt: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export const MAX_AUTORESUMES_PER_TURN = 1;
|
||||||
const TTL_MS = 45 * 60_000;
|
const TTL_MS = 45 * 60_000;
|
||||||
const MAX_PINS = 1_000;
|
const MAX_PINS = 1_000;
|
||||||
const pins = new Map<string, NativeTurnPin>();
|
const turns = new Map<string, NativeCodexTurnRecord>();
|
||||||
|
|
||||||
function record(value: unknown): Record<string, unknown> | undefined {
|
function record(value: unknown): Record<string, unknown> | undefined {
|
||||||
return value && typeof value === "object" && !Array.isArray(value)
|
return value && typeof value === "object" && !Array.isArray(value)
|
||||||
@@ -57,21 +69,51 @@ export function nativeCodexTurnKey(
|
|||||||
}
|
}
|
||||||
|
|
||||||
function prune(now = Date.now()): void {
|
function prune(now = Date.now()): void {
|
||||||
for (const [key, pin] of pins) if (pin.expiresAt <= now) pins.delete(key);
|
for (const [key, rec] of turns) if (rec.expiresAt <= now) turns.delete(key);
|
||||||
while (pins.size > MAX_PINS) {
|
while (turns.size > MAX_PINS) {
|
||||||
const oldest = pins.keys().next().value as string | undefined;
|
const oldest = turns.keys().next().value as string | undefined;
|
||||||
if (!oldest) break;
|
if (!oldest) break;
|
||||||
pins.delete(oldest);
|
turns.delete(oldest);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
export function getNativeCodexTurnPin(
|
export function getNativeCodexTurnPin(
|
||||||
body: Record<string, unknown>,
|
body: Record<string, unknown>,
|
||||||
comboName: string
|
comboName: string,
|
||||||
|
generation?: number
|
||||||
): NativeTurnPin | null {
|
): NativeTurnPin | null {
|
||||||
prune();
|
prune();
|
||||||
const key = nativeCodexTurnKey(body, comboName);
|
const key = nativeCodexTurnKey(body, comboName);
|
||||||
return key ? (pins.get(key) ?? null) : null;
|
if (!key) return null;
|
||||||
|
const rec = turns.get(key);
|
||||||
|
if (!rec) return null;
|
||||||
|
const gen = generation !== undefined ? generation : rec.activeGeneration;
|
||||||
|
return rec.pins.get(gen) ?? null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function getNativeCodexTurnActiveGeneration(
|
||||||
|
body: Record<string, unknown>,
|
||||||
|
comboName: string
|
||||||
|
): number {
|
||||||
|
prune();
|
||||||
|
const key = nativeCodexTurnKey(body, comboName);
|
||||||
|
if (!key) return 0;
|
||||||
|
return turns.get(key)?.activeGeneration ?? 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function advanceNativeCodexTurnGeneration(
|
||||||
|
body: Record<string, unknown>,
|
||||||
|
comboName: string
|
||||||
|
): number | null {
|
||||||
|
prune();
|
||||||
|
const key = nativeCodexTurnKey(body, comboName);
|
||||||
|
if (!key) return null;
|
||||||
|
const rec = turns.get(key);
|
||||||
|
if (!rec) return null;
|
||||||
|
rec.activeGeneration += 1;
|
||||||
|
const now = Date.now();
|
||||||
|
rec.expiresAt = now + TTL_MS;
|
||||||
|
return rec.activeGeneration;
|
||||||
}
|
}
|
||||||
|
|
||||||
export function pinNativeCodexTurn(args: {
|
export function pinNativeCodexTurn(args: {
|
||||||
@@ -79,10 +121,27 @@ export function pinNativeCodexTurn(args: {
|
|||||||
comboName: string;
|
comboName: string;
|
||||||
target: ResolvedComboTarget;
|
target: ResolvedComboTarget;
|
||||||
connectionId: string;
|
connectionId: string;
|
||||||
|
generation?: number;
|
||||||
}): void {
|
}): void {
|
||||||
const key = nativeCodexTurnKey(args.body, args.comboName);
|
const key = nativeCodexTurnKey(args.body, args.comboName);
|
||||||
if (!key || !args.connectionId) return;
|
if (!key || !args.connectionId) return;
|
||||||
const existing = pins.get(key);
|
let rec = turns.get(key);
|
||||||
|
const now = Date.now();
|
||||||
|
if (!rec) {
|
||||||
|
const metadata = turnMetadata(args.body);
|
||||||
|
rec = {
|
||||||
|
comboName: args.comboName,
|
||||||
|
threadId: typeof metadata?.thread_id === "string" ? metadata.thread_id : "",
|
||||||
|
turnId: typeof metadata?.turn_id === "string" ? metadata.turn_id : "",
|
||||||
|
activeGeneration: 0,
|
||||||
|
pins: new Map(),
|
||||||
|
createdAt: now,
|
||||||
|
expiresAt: now + TTL_MS,
|
||||||
|
};
|
||||||
|
turns.set(key, rec);
|
||||||
|
}
|
||||||
|
const gen = args.generation !== undefined ? args.generation : rec.activeGeneration;
|
||||||
|
const existing = rec.pins.get(gen);
|
||||||
if (
|
if (
|
||||||
existing &&
|
existing &&
|
||||||
(existing.modelStr !== args.target.modelStr || existing.provider !== args.target.provider)
|
(existing.modelStr !== args.target.modelStr || existing.provider !== args.target.provider)
|
||||||
@@ -91,15 +150,16 @@ export function pinNativeCodexTurn(args: {
|
|||||||
}
|
}
|
||||||
// ConnectionId changes are allowed (failover to sibling connection)
|
// ConnectionId changes are allowed (failover to sibling connection)
|
||||||
// as long as provider + model stay the same.
|
// as long as provider + model stay the same.
|
||||||
const now = Date.now();
|
rec.pins.set(gen, {
|
||||||
pins.set(key, {
|
|
||||||
comboName: args.comboName,
|
comboName: args.comboName,
|
||||||
modelStr: args.target.modelStr,
|
modelStr: args.target.modelStr,
|
||||||
provider: args.target.provider,
|
provider: args.target.provider,
|
||||||
connectionId: args.connectionId,
|
connectionId: args.connectionId,
|
||||||
createdAt: existing?.createdAt ?? now,
|
createdAt: existing?.createdAt ?? now,
|
||||||
expiresAt: now + TTL_MS,
|
expiresAt: now + TTL_MS,
|
||||||
|
generation: gen,
|
||||||
});
|
});
|
||||||
|
rec.expiresAt = now + TTL_MS;
|
||||||
prune(now);
|
prune(now);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -153,10 +213,16 @@ export function applyNativeCodexTurnPin(
|
|||||||
|
|
||||||
export function revokeNativeCodexTurnPinsForConnection(connectionId: string): number {
|
export function revokeNativeCodexTurnPinsForConnection(connectionId: string): number {
|
||||||
let revoked = 0;
|
let revoked = 0;
|
||||||
for (const [key, pin] of pins) {
|
for (const [key, rec] of turns) {
|
||||||
if (pin.connectionId !== connectionId) continue;
|
for (const [gen, pin] of rec.pins) {
|
||||||
pins.delete(key);
|
if (pin.connectionId === connectionId) {
|
||||||
revoked += 1;
|
rec.pins.delete(gen);
|
||||||
|
revoked += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (rec.pins.size === 0) {
|
||||||
|
turns.delete(key);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
return revoked;
|
return revoked;
|
||||||
}
|
}
|
||||||
@@ -282,9 +348,408 @@ export async function areAllPinnedTargetsModelScopedUnusable(
|
|||||||
|
|
||||||
export function releaseNativeCodexTurnPin(body: Record<string, unknown>, comboName: string): void {
|
export function releaseNativeCodexTurnPin(body: Record<string, unknown>, comboName: string): void {
|
||||||
const key = nativeCodexTurnKey(body, comboName);
|
const key = nativeCodexTurnKey(body, comboName);
|
||||||
if (key) pins.delete(key);
|
if (key) turns.delete(key);
|
||||||
}
|
}
|
||||||
|
|
||||||
export function clearNativeCodexTurnPinsForTests(): void {
|
export function clearNativeCodexTurnPinsForTests(): void {
|
||||||
pins.clear();
|
turns.clear();
|
||||||
|
}
|
||||||
|
|
||||||
|
export function hasUnresolvedToolCalls(body: Record<string, unknown>): boolean {
|
||||||
|
const input: unknown[] = Array.isArray(body.input)
|
||||||
|
? body.input
|
||||||
|
: Array.isArray(body.messages)
|
||||||
|
? body.messages
|
||||||
|
: [];
|
||||||
|
if (input.length === 0) return false;
|
||||||
|
|
||||||
|
const callCounts = new Map<string, number>();
|
||||||
|
const outputCounts = new Map<string, number>();
|
||||||
|
|
||||||
|
const processPart = (part: unknown): boolean => {
|
||||||
|
if (!part || typeof part !== "object") return true;
|
||||||
|
const rec = part as Record<string, unknown>;
|
||||||
|
const type = typeof rec.type === "string" ? rec.type : "";
|
||||||
|
|
||||||
|
if (type === "function_call" || type === "custom_tool_call") {
|
||||||
|
const callId =
|
||||||
|
(typeof rec.call_id === "string" && rec.call_id.trim() ? rec.call_id.trim() : "") ||
|
||||||
|
(typeof rec.id === "string" && rec.id.trim() ? rec.id.trim() : "");
|
||||||
|
if (!callId) return false;
|
||||||
|
callCounts.set(callId, (callCounts.get(callId) ?? 0) + 1);
|
||||||
|
} else if (type === "function_call_output" || type === "custom_tool_call_output") {
|
||||||
|
const callId =
|
||||||
|
(typeof rec.call_id === "string" && rec.call_id.trim() ? rec.call_id.trim() : "") ||
|
||||||
|
(typeof rec.id === "string" && rec.id.trim() ? rec.id.trim() : "");
|
||||||
|
if (!callId) return false;
|
||||||
|
outputCounts.set(callId, (outputCounts.get(callId) ?? 0) + 1);
|
||||||
|
} else if (type === "tool_use") {
|
||||||
|
const callId = typeof rec.id === "string" && rec.id.trim() ? rec.id.trim() : "";
|
||||||
|
if (!callId) return false;
|
||||||
|
callCounts.set(callId, (callCounts.get(callId) ?? 0) + 1);
|
||||||
|
} else if (type === "tool_result") {
|
||||||
|
const callId =
|
||||||
|
(typeof rec.tool_use_id === "string" && rec.tool_use_id.trim()
|
||||||
|
? rec.tool_use_id.trim()
|
||||||
|
: "") || (typeof rec.id === "string" && rec.id.trim() ? rec.id.trim() : "");
|
||||||
|
if (!callId) return false;
|
||||||
|
outputCounts.set(callId, (outputCounts.get(callId) ?? 0) + 1);
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
};
|
||||||
|
|
||||||
|
for (const item of input) {
|
||||||
|
if (!item || typeof item !== "object") continue;
|
||||||
|
const rec = item as Record<string, unknown>;
|
||||||
|
const role = typeof rec.role === "string" ? rec.role : "";
|
||||||
|
|
||||||
|
if (rec.function_call && typeof rec.function_call === "object" && role === "assistant") {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (role === "function") {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!processPart(rec)) return true;
|
||||||
|
|
||||||
|
if (role === "assistant" && Array.isArray(rec.tool_calls)) {
|
||||||
|
for (const tc of rec.tool_calls) {
|
||||||
|
if (!tc || typeof tc !== "object") return true;
|
||||||
|
const id =
|
||||||
|
typeof (tc as Record<string, unknown>).id === "string"
|
||||||
|
? ((tc as Record<string, unknown>).id as string).trim()
|
||||||
|
: "";
|
||||||
|
if (!id) return true;
|
||||||
|
callCounts.set(id, (callCounts.get(id) ?? 0) + 1);
|
||||||
|
}
|
||||||
|
} else if (role === "tool") {
|
||||||
|
const toolCallId =
|
||||||
|
(typeof rec.tool_call_id === "string" && rec.tool_call_id.trim()
|
||||||
|
? rec.tool_call_id.trim()
|
||||||
|
: "") ||
|
||||||
|
(typeof rec.call_id === "string" && rec.call_id.trim() ? rec.call_id.trim() : "") ||
|
||||||
|
(typeof rec.id === "string" && rec.id.trim() ? rec.id.trim() : "");
|
||||||
|
if (!toolCallId) return true;
|
||||||
|
outputCounts.set(toolCallId, (outputCounts.get(toolCallId) ?? 0) + 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (Array.isArray(rec.content)) {
|
||||||
|
for (const part of rec.content) {
|
||||||
|
if (!processPart(part)) return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (Array.isArray(rec.output)) {
|
||||||
|
for (const part of rec.output) {
|
||||||
|
if (!processPart(part)) return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (callCounts.size === 0 && outputCounts.size === 0) return false;
|
||||||
|
if (callCounts.size !== outputCounts.size) return true;
|
||||||
|
for (const [id, count] of callCounts) {
|
||||||
|
if (count !== 1) return true;
|
||||||
|
if (outputCounts.get(id) !== 1) return true;
|
||||||
|
}
|
||||||
|
for (const [id, count] of outputCounts) {
|
||||||
|
if (count !== 1) return true;
|
||||||
|
if (!callCounts.has(id)) return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
function isUnsafeItemOrPart(rec: Record<string, unknown>): boolean {
|
||||||
|
const type = typeof rec.type === "string" ? rec.type : "";
|
||||||
|
if (type === "item_reference" || type === "redacted_thinking") return true;
|
||||||
|
if (type === "encrypted_content") return true;
|
||||||
|
if (typeof rec.previous_response_id === "string" && rec.previous_response_id.trim() !== "")
|
||||||
|
return true;
|
||||||
|
if (typeof rec.previousResponseId === "string" && rec.previousResponseId.trim() !== "")
|
||||||
|
return true;
|
||||||
|
if (typeof rec.continuation_token === "string" && rec.continuation_token.trim() !== "")
|
||||||
|
return true;
|
||||||
|
if (typeof rec.continuationToken === "string" && rec.continuationToken.trim() !== "") return true;
|
||||||
|
if (typeof rec.encrypted_content === "string" && rec.encrypted_content.trim() !== "") return true;
|
||||||
|
if (typeof rec.encryptedContent === "string" && rec.encryptedContent.trim() !== "") return true;
|
||||||
|
if (typeof rec.encrypted_reasoning === "string" && rec.encrypted_reasoning.trim() !== "")
|
||||||
|
return true;
|
||||||
|
if (typeof rec.encryptedReasoning === "string" && rec.encryptedReasoning.trim() !== "")
|
||||||
|
return true;
|
||||||
|
if (typeof rec.thought_signature === "string" && rec.thought_signature.trim() !== "") return true;
|
||||||
|
if (typeof rec.thoughtSignature === "string" && rec.thoughtSignature.trim() !== "") return true;
|
||||||
|
if (typeof rec.signature === "string" && rec.signature.trim() !== "") return true;
|
||||||
|
if (
|
||||||
|
rec.provider_metadata &&
|
||||||
|
typeof rec.provider_metadata === "object" &&
|
||||||
|
Object.keys(rec.provider_metadata as object).length > 0
|
||||||
|
) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
rec.providerMetadata &&
|
||||||
|
typeof rec.providerMetadata === "object" &&
|
||||||
|
Object.keys(rec.providerMetadata as object).length > 0
|
||||||
|
) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
rec.provider_data &&
|
||||||
|
typeof rec.provider_data === "object" &&
|
||||||
|
Object.keys(rec.provider_data as object).length > 0
|
||||||
|
) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
rec.providerData &&
|
||||||
|
typeof rec.providerData === "object" &&
|
||||||
|
Object.keys(rec.providerData as object).length > 0
|
||||||
|
) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function hasProviderSpecificUnsafeContinuationState(
|
||||||
|
body: Record<string, unknown>,
|
||||||
|
_activePin?: NativeTurnPin
|
||||||
|
): boolean {
|
||||||
|
if (typeof body.previous_response_id === "string" && body.previous_response_id.trim() !== "") {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (typeof body.previousResponseId === "string" && body.previousResponseId.trim() !== "") {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (typeof body.continuation_token === "string" && body.continuation_token.trim() !== "") {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (typeof body.continuationToken === "string" && body.continuationToken.trim() !== "") {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (typeof body.response_id === "string" && body.response_id.trim() !== "") {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (typeof body.responseId === "string" && body.responseId.trim() !== "") {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (typeof body.parent_response_id === "string" && body.parent_response_id.trim() !== "") {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (typeof body.parentResponseId === "string" && body.parentResponseId.trim() !== "") {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (typeof body.thought_signature === "string" && body.thought_signature.trim() !== "") {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (typeof body.thoughtSignature === "string" && body.thoughtSignature.trim() !== "") {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (typeof body.signature === "string" && body.signature.trim() !== "") {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (typeof body.conversation_id === "string" && body.conversation_id.trim() !== "") {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (typeof body.conversationId === "string" && body.conversationId.trim() !== "") {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
body.conversation &&
|
||||||
|
typeof body.conversation === "object" &&
|
||||||
|
Object.keys(body.conversation as object).length > 0
|
||||||
|
) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
body.provider_metadata &&
|
||||||
|
typeof body.provider_metadata === "object" &&
|
||||||
|
Object.keys(body.provider_metadata as object).length > 0
|
||||||
|
) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
body.providerMetadata &&
|
||||||
|
typeof body.providerMetadata === "object" &&
|
||||||
|
Object.keys(body.providerMetadata as object).length > 0
|
||||||
|
) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
body.provider_data &&
|
||||||
|
typeof body.provider_data === "object" &&
|
||||||
|
Object.keys(body.provider_data as object).length > 0
|
||||||
|
) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
body.providerData &&
|
||||||
|
typeof body.providerData === "object" &&
|
||||||
|
Object.keys(body.providerData as object).length > 0
|
||||||
|
) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
const input: unknown[] = Array.isArray(body.input)
|
||||||
|
? body.input
|
||||||
|
: Array.isArray(body.messages)
|
||||||
|
? body.messages
|
||||||
|
: [];
|
||||||
|
|
||||||
|
for (const item of input) {
|
||||||
|
if (!item || typeof item !== "object") continue;
|
||||||
|
const rec = item as Record<string, unknown>;
|
||||||
|
if (isUnsafeItemOrPart(rec)) return true;
|
||||||
|
|
||||||
|
if (Array.isArray(rec.content)) {
|
||||||
|
for (const part of rec.content) {
|
||||||
|
if (
|
||||||
|
part &&
|
||||||
|
typeof part === "object" &&
|
||||||
|
isUnsafeItemOrPart(part as Record<string, unknown>)
|
||||||
|
) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (Array.isArray(rec.output)) {
|
||||||
|
for (const outItem of rec.output) {
|
||||||
|
if (
|
||||||
|
outItem &&
|
||||||
|
typeof outItem === "object" &&
|
||||||
|
isUnsafeItemOrPart(outItem as Record<string, unknown>)
|
||||||
|
) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (Array.isArray(rec.summary)) {
|
||||||
|
for (const sumItem of rec.summary) {
|
||||||
|
if (
|
||||||
|
sumItem &&
|
||||||
|
typeof sumItem === "object" &&
|
||||||
|
isUnsafeItemOrPart(sumItem as Record<string, unknown>)
|
||||||
|
) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (Array.isArray(rec.tool_calls)) {
|
||||||
|
for (const tc of rec.tool_calls) {
|
||||||
|
if (tc && typeof tc === "object" && isUnsafeItemOrPart(tc as Record<string, unknown>)) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CanAutoResumeNativeCodexTurnOptions {
|
||||||
|
body: Record<string, unknown>;
|
||||||
|
comboName: string;
|
||||||
|
activePin: NativeTurnPin;
|
||||||
|
allTargets: ResolvedComboTarget[];
|
||||||
|
resilienceSettings?: ResilienceSettings | null;
|
||||||
|
quotaCutoffResetWindowConfig?: ResetWindowConfig;
|
||||||
|
isModelAvailable?: IsModelAvailable;
|
||||||
|
log?: ComboLogger;
|
||||||
|
}
|
||||||
|
|
||||||
|
export type AutoResumeDecision =
|
||||||
|
| {
|
||||||
|
eligible: true;
|
||||||
|
nextGeneration: number;
|
||||||
|
previousPin: NativeTurnPin;
|
||||||
|
selectedTarget: ResolvedComboTarget;
|
||||||
|
}
|
||||||
|
| {
|
||||||
|
eligible: false;
|
||||||
|
reason: string;
|
||||||
|
details?: Record<string, unknown>;
|
||||||
|
};
|
||||||
|
|
||||||
|
export async function canAutoResumeNativeCodexTurn(
|
||||||
|
options: CanAutoResumeNativeCodexTurnOptions
|
||||||
|
): Promise<AutoResumeDecision> {
|
||||||
|
const {
|
||||||
|
body,
|
||||||
|
comboName,
|
||||||
|
activePin,
|
||||||
|
allTargets,
|
||||||
|
resilienceSettings,
|
||||||
|
quotaCutoffResetWindowConfig,
|
||||||
|
isModelAvailable,
|
||||||
|
} = options;
|
||||||
|
|
||||||
|
const key = nativeCodexTurnKey(body, comboName);
|
||||||
|
if (!key) return { eligible: false, reason: "invalid_turn_key" };
|
||||||
|
|
||||||
|
const rec = turns.get(key);
|
||||||
|
const currentGen = rec?.activeGeneration ?? 0;
|
||||||
|
if (currentGen >= MAX_AUTORESUMES_PER_TURN) {
|
||||||
|
return { eligible: false, reason: "max_resumes_exceeded" };
|
||||||
|
}
|
||||||
|
|
||||||
|
const inputList = Array.isArray(body.input)
|
||||||
|
? body.input
|
||||||
|
: Array.isArray(body.messages)
|
||||||
|
? body.messages
|
||||||
|
: null;
|
||||||
|
if (!inputList || inputList.length === 0) {
|
||||||
|
return { eligible: false, reason: "missing_or_empty_input" };
|
||||||
|
}
|
||||||
|
|
||||||
|
if (hasUnresolvedToolCalls(body)) {
|
||||||
|
return { eligible: false, reason: "pending_tool_call" };
|
||||||
|
}
|
||||||
|
|
||||||
|
if (hasProviderSpecificUnsafeContinuationState(body, activePin)) {
|
||||||
|
return { eligible: false, reason: "unsafe_provider_state" };
|
||||||
|
}
|
||||||
|
|
||||||
|
const alternateTargets = allTargets.filter(
|
||||||
|
(t) => t.modelStr !== activePin.modelStr || t.provider !== activePin.provider
|
||||||
|
);
|
||||||
|
if (alternateTargets.length === 0) {
|
||||||
|
return { eligible: false, reason: "no_alternate_target" };
|
||||||
|
}
|
||||||
|
|
||||||
|
let selectedTarget: ResolvedComboTarget | null = null;
|
||||||
|
for (const alt of alternateTargets) {
|
||||||
|
if (alt.provider && alt.provider !== "unknown") {
|
||||||
|
const cb = getCircuitBreaker(alt.provider);
|
||||||
|
if (cb.getStatus().state === "OPEN") continue;
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
resilienceSettings?.providerCooldown?.enabled &&
|
||||||
|
(isProviderInCooldown(alt.provider, alt.connectionId || undefined, resilienceSettings) ||
|
||||||
|
isProviderInCooldown(alt.provider, undefined, resilienceSettings))
|
||||||
|
) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
const unusable = await isPinnedTargetModelScopedUnusable({
|
||||||
|
target: alt,
|
||||||
|
resilienceSettings,
|
||||||
|
quotaCutoffResetWindowConfig,
|
||||||
|
comboName,
|
||||||
|
body,
|
||||||
|
isModelAvailable,
|
||||||
|
});
|
||||||
|
if (!unusable) {
|
||||||
|
selectedTarget = alt;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!selectedTarget) {
|
||||||
|
return { eligible: false, reason: "no_healthy_alternate_target" };
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
eligible: true,
|
||||||
|
nextGeneration: currentGen + 1,
|
||||||
|
previousPin: activePin,
|
||||||
|
selectedTarget,
|
||||||
|
};
|
||||||
}
|
}
|
||||||
|
|||||||
416
tests/unit/native-codex-auto-resume-guards.test.ts
Normal file
416
tests/unit/native-codex-auto-resume-guards.test.ts
Normal file
@@ -0,0 +1,416 @@
|
|||||||
|
import test from "node:test";
|
||||||
|
import assert from "node:assert/strict";
|
||||||
|
|
||||||
|
// Pure guard predicates of the native Codex auto-resume path (#13180). Split out of
|
||||||
|
// tests/unit/native-codex-auto-resume.test.ts (which exercises the full combo flow
|
||||||
|
// against a scratch DB) so each file stays under the 1200-line test cap; these three
|
||||||
|
// cases need no DB, no combo config and no fixtures.
|
||||||
|
const {
|
||||||
|
hasUnresolvedToolCalls,
|
||||||
|
hasProviderSpecificUnsafeContinuationState,
|
||||||
|
MAX_AUTORESUMES_PER_TURN,
|
||||||
|
} = await import("../../open-sse/services/combo/nativeCodexTurnPin.ts");
|
||||||
|
|
||||||
|
test("MAX_AUTORESUMES_PER_TURN constant is 1", () => {
|
||||||
|
assert.equal(MAX_AUTORESUMES_PER_TURN, 1);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("hasUnresolvedToolCalls correctly validates 1:1 call-output pairs and rejects duplicates/orphans/nested", () => {
|
||||||
|
// Empty input: no tool calls
|
||||||
|
assert.equal(hasUnresolvedToolCalls({}), false);
|
||||||
|
assert.equal(hasUnresolvedToolCalls({ input: [] }), false);
|
||||||
|
|
||||||
|
// Nested output array with unresolved tool_use is unsafe (true)
|
||||||
|
assert.equal(
|
||||||
|
hasUnresolvedToolCalls({
|
||||||
|
input: [
|
||||||
|
{
|
||||||
|
type: "function_call_output",
|
||||||
|
call_id: "c1",
|
||||||
|
output: [{ type: "tool_use", id: "tu-nested", name: "bash" }],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
|
||||||
|
// Two calls with same call_id and two outputs with same call_id (count=2 != 1) is unsafe (true)
|
||||||
|
assert.equal(
|
||||||
|
hasUnresolvedToolCalls({
|
||||||
|
input: [
|
||||||
|
{ type: "function_call", call_id: "c-dup2", name: "cat", arguments: "{}" },
|
||||||
|
{ type: "function_call", call_id: "c-dup2", name: "cat", arguments: "{}" },
|
||||||
|
{ type: "function_call_output", call_id: "c-dup2", output: "out1" },
|
||||||
|
{ type: "function_call_output", call_id: "c-dup2", output: "out2" },
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
|
||||||
|
// Two distinct calls with two distinct matching outputs is safe (false)
|
||||||
|
assert.equal(
|
||||||
|
hasUnresolvedToolCalls({
|
||||||
|
input: [
|
||||||
|
{ type: "function_call", call_id: "c-1", name: "cat", arguments: "{}" },
|
||||||
|
{ type: "function_call", call_id: "c-2", name: "ls", arguments: "{}" },
|
||||||
|
{ type: "function_call_output", call_id: "c-1", output: "out1" },
|
||||||
|
{ type: "function_call_output", call_id: "c-2", output: "out2" },
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
false
|
||||||
|
);
|
||||||
|
|
||||||
|
// Resolved function call (1 call, 1 matching output)
|
||||||
|
assert.equal(
|
||||||
|
hasUnresolvedToolCalls({
|
||||||
|
input: [
|
||||||
|
{ type: "message", role: "user", content: "read file" },
|
||||||
|
{ type: "function_call", call_id: "call-1", name: "cat", arguments: "{}" },
|
||||||
|
{ type: "function_call_output", call_id: "call-1", output: "hello world" },
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
false
|
||||||
|
);
|
||||||
|
|
||||||
|
// Unresolved function call (call with no output)
|
||||||
|
assert.equal(
|
||||||
|
hasUnresolvedToolCalls({
|
||||||
|
input: [
|
||||||
|
{ type: "message", role: "user", content: "read file" },
|
||||||
|
{ type: "function_call", call_id: "call-1", name: "cat", arguments: "{}" },
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
|
||||||
|
// Resolved custom tool call
|
||||||
|
assert.equal(
|
||||||
|
hasUnresolvedToolCalls({
|
||||||
|
input: [
|
||||||
|
{ type: "message", role: "user", content: "patch file" },
|
||||||
|
{ type: "custom_tool_call", call_id: "call-2", name: "apply_patch", input: "diff" },
|
||||||
|
{ type: "custom_tool_call_output", call_id: "call-2", output: "ok" },
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
false
|
||||||
|
);
|
||||||
|
|
||||||
|
// Unresolved custom tool call
|
||||||
|
assert.equal(
|
||||||
|
hasUnresolvedToolCalls({
|
||||||
|
input: [
|
||||||
|
{ type: "message", role: "user", content: "patch file" },
|
||||||
|
{ type: "custom_tool_call", call_id: "call-2", name: "apply_patch", input: "diff" },
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
|
||||||
|
// Anthropic tool_use and tool_result in content array (resolved)
|
||||||
|
assert.equal(
|
||||||
|
hasUnresolvedToolCalls({
|
||||||
|
messages: [
|
||||||
|
{
|
||||||
|
role: "assistant",
|
||||||
|
content: [{ type: "tool_use", id: "tu-1", name: "bash", input: {} }],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
role: "user",
|
||||||
|
content: [{ type: "tool_result", tool_use_id: "tu-1", content: "done" }],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
false
|
||||||
|
);
|
||||||
|
|
||||||
|
// Anthropic tool_use in content array (unresolved)
|
||||||
|
assert.equal(
|
||||||
|
hasUnresolvedToolCalls({
|
||||||
|
messages: [
|
||||||
|
{
|
||||||
|
role: "assistant",
|
||||||
|
content: [{ type: "tool_use", id: "tu-1", name: "bash", input: {} }],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
|
||||||
|
// Assistant message tool_calls format (resolved)
|
||||||
|
assert.equal(
|
||||||
|
hasUnresolvedToolCalls({
|
||||||
|
input: [
|
||||||
|
{
|
||||||
|
type: "message",
|
||||||
|
role: "assistant",
|
||||||
|
tool_calls: [{ id: "call-3", type: "function", function: { name: "shell" } }],
|
||||||
|
},
|
||||||
|
{ type: "message", role: "tool", tool_call_id: "call-3", content: "done" },
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
false
|
||||||
|
);
|
||||||
|
|
||||||
|
// Assistant message tool_calls format (unresolved)
|
||||||
|
assert.equal(
|
||||||
|
hasUnresolvedToolCalls({
|
||||||
|
input: [
|
||||||
|
{
|
||||||
|
type: "message",
|
||||||
|
role: "assistant",
|
||||||
|
tool_calls: [{ id: "call-3", type: "function", function: { name: "shell" } }],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
|
||||||
|
// Duplicate tool call ID: two calls with same ID, one output -> unsafe (true)
|
||||||
|
assert.equal(
|
||||||
|
hasUnresolvedToolCalls({
|
||||||
|
input: [
|
||||||
|
{ type: "function_call", call_id: "call-dup", name: "cat", arguments: "{}" },
|
||||||
|
{ type: "function_call", call_id: "call-dup", name: "cat", arguments: "{}" },
|
||||||
|
{ type: "function_call_output", call_id: "call-dup", output: "res" },
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
|
||||||
|
// Duplicate tool output ID: one call, two outputs with same ID -> unsafe (true)
|
||||||
|
assert.equal(
|
||||||
|
hasUnresolvedToolCalls({
|
||||||
|
input: [
|
||||||
|
{ type: "function_call", call_id: "call-dup-out", name: "cat", arguments: "{}" },
|
||||||
|
{ type: "function_call_output", call_id: "call-dup-out", output: "res1" },
|
||||||
|
{ type: "function_call_output", call_id: "call-dup-out", output: "res2" },
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
|
||||||
|
// Orphaned tool output: output without matching call -> unsafe (true)
|
||||||
|
assert.equal(
|
||||||
|
hasUnresolvedToolCalls({
|
||||||
|
input: [{ type: "function_call_output", call_id: "orphan-call", output: "res" }],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
|
||||||
|
// Malformed tool call with empty call_id -> unsafe (true)
|
||||||
|
assert.equal(
|
||||||
|
hasUnresolvedToolCalls({
|
||||||
|
input: [{ type: "function_call", call_id: "", name: "cat", arguments: "{}" }],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
|
||||||
|
// Legacy unidentifiable function_call -> unsafe (true)
|
||||||
|
assert.equal(
|
||||||
|
hasUnresolvedToolCalls({
|
||||||
|
input: [{ role: "assistant", function_call: { name: "test", arguments: "{}" } }],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("hasProviderSpecificUnsafeContinuationState detects opaque provider state at all levels", () => {
|
||||||
|
// Clean input: safe
|
||||||
|
assert.equal(
|
||||||
|
hasProviderSpecificUnsafeContinuationState({
|
||||||
|
input: [{ type: "message", role: "user", content: "hello" }],
|
||||||
|
}),
|
||||||
|
false
|
||||||
|
);
|
||||||
|
|
||||||
|
// conversation_id at root: unsafe
|
||||||
|
assert.equal(
|
||||||
|
hasProviderSpecificUnsafeContinuationState({
|
||||||
|
conversation_id: "conv_12345",
|
||||||
|
input: [{ type: "message", role: "user", content: "hello" }],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
|
||||||
|
// conversation object at root: unsafe
|
||||||
|
assert.equal(
|
||||||
|
hasProviderSpecificUnsafeContinuationState({
|
||||||
|
conversation: { id: "conv_67890" },
|
||||||
|
input: [{ type: "message", role: "user", content: "hello" }],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
|
||||||
|
// Item with item-level previous_response_id: unsafe
|
||||||
|
assert.equal(
|
||||||
|
hasProviderSpecificUnsafeContinuationState({
|
||||||
|
input: [
|
||||||
|
{
|
||||||
|
type: "message",
|
||||||
|
role: "assistant",
|
||||||
|
previous_response_id: "resp_nested_prev",
|
||||||
|
content: "hello",
|
||||||
|
},
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
|
||||||
|
// Item with item-level continuation_token: unsafe
|
||||||
|
assert.equal(
|
||||||
|
hasProviderSpecificUnsafeContinuationState({
|
||||||
|
input: [
|
||||||
|
{
|
||||||
|
type: "message",
|
||||||
|
role: "assistant",
|
||||||
|
continuation_token: "tok_nested_cont",
|
||||||
|
content: "hello",
|
||||||
|
},
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
|
||||||
|
// previous_response_id: unsafe (binds to upstream response store)
|
||||||
|
assert.equal(
|
||||||
|
hasProviderSpecificUnsafeContinuationState({
|
||||||
|
previous_response_id: "resp_12345_upstream",
|
||||||
|
input: [{ type: "message", role: "user", content: "hello" }],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
|
||||||
|
// continuation_token: unsafe
|
||||||
|
assert.equal(
|
||||||
|
hasProviderSpecificUnsafeContinuationState({
|
||||||
|
continuation_token: "tok_opaque_blob",
|
||||||
|
input: [{ type: "message", role: "user", content: "hello" }],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
|
||||||
|
// response_id: unsafe
|
||||||
|
assert.equal(
|
||||||
|
hasProviderSpecificUnsafeContinuationState({
|
||||||
|
response_id: "resp_999",
|
||||||
|
input: [{ type: "message", role: "user", content: "hello" }],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
|
||||||
|
// provider_metadata at root: unsafe
|
||||||
|
assert.equal(
|
||||||
|
hasProviderSpecificUnsafeContinuationState({
|
||||||
|
provider_metadata: { openai: { message_id: "m1" } },
|
||||||
|
input: [{ type: "message", role: "user", content: "hello" }],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
|
||||||
|
// item_reference: unsafe (server-side item ID)
|
||||||
|
assert.equal(
|
||||||
|
hasProviderSpecificUnsafeContinuationState({
|
||||||
|
input: [{ type: "item_reference", id: "item_abc123" }],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
|
||||||
|
// reasoning item with encrypted_content: unsafe
|
||||||
|
assert.equal(
|
||||||
|
hasProviderSpecificUnsafeContinuationState({
|
||||||
|
input: [
|
||||||
|
{ type: "reasoning", encrypted_content: "enc_blob_xyz" },
|
||||||
|
{ type: "message", role: "user", content: "hello" },
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
|
||||||
|
// thinking item with thought_signature: unsafe
|
||||||
|
assert.equal(
|
||||||
|
hasProviderSpecificUnsafeContinuationState({
|
||||||
|
input: [
|
||||||
|
{ type: "thinking", thought_signature: "sig_gemini_blob" },
|
||||||
|
{ type: "message", role: "user", content: "hello" },
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
|
||||||
|
// redacted_thinking item: unsafe
|
||||||
|
assert.equal(
|
||||||
|
hasProviderSpecificUnsafeContinuationState({
|
||||||
|
input: [{ type: "redacted_thinking", data: "redacted" }],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
|
||||||
|
// Nested thinking part inside content array with signature: unsafe
|
||||||
|
assert.equal(
|
||||||
|
hasProviderSpecificUnsafeContinuationState({
|
||||||
|
messages: [
|
||||||
|
{
|
||||||
|
role: "assistant",
|
||||||
|
content: [
|
||||||
|
{ type: "thinking", thinking: "deep thought", signature: "sig-xyz" },
|
||||||
|
{ type: "text", text: "hello" },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
|
||||||
|
// encrypted_content item: unsafe
|
||||||
|
assert.equal(
|
||||||
|
hasProviderSpecificUnsafeContinuationState({
|
||||||
|
input: [{ type: "encrypted_content", encrypted_content: "enc_123" }],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
|
||||||
|
// Root body thoughtSignature (camelCase): unsafe
|
||||||
|
assert.equal(
|
||||||
|
hasProviderSpecificUnsafeContinuationState({
|
||||||
|
thoughtSignature: "sig_camel_case",
|
||||||
|
input: [{ type: "message", role: "user", content: "hello" }],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
|
||||||
|
// Root body provider_data: unsafe
|
||||||
|
assert.equal(
|
||||||
|
hasProviderSpecificUnsafeContinuationState({
|
||||||
|
provider_data: { gemini: { candidate_token_count: 50 } },
|
||||||
|
input: [{ type: "message", role: "user", content: "hello" }],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
|
||||||
|
// Nested output array with encrypted_content: unsafe
|
||||||
|
assert.equal(
|
||||||
|
hasProviderSpecificUnsafeContinuationState({
|
||||||
|
input: [
|
||||||
|
{
|
||||||
|
type: "function_call_output",
|
||||||
|
call_id: "c1",
|
||||||
|
output: [{ type: "encrypted_content", encrypted_content: "enc_blob" }],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
|
||||||
|
// Nested summary array with thought_signature: unsafe
|
||||||
|
assert.equal(
|
||||||
|
hasProviderSpecificUnsafeContinuationState({
|
||||||
|
input: [
|
||||||
|
{
|
||||||
|
type: "reasoning",
|
||||||
|
summary: [{ type: "summary_text", text: "...", thought_signature: "sig" }],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
});
|
||||||
1131
tests/unit/native-codex-auto-resume.test.ts
Normal file
1131
tests/unit/native-codex-auto-resume.test.ts
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user