pjandro/OmniRoute
1
0
Fork 0
mirror of https://github.com/diegosouzapw/OmniRoute.git synced 2026-09-08 07:52:12 +03:00
Code Issues Packages Projects Releases Wiki Activity

fix(chat): continue after a server-owned tool on Chat Completions (#12867)

Browse Source 
Validado numa worktree combinada com as 16 PRs desta leva sobre `release/v3.8.51`: typecheck:core limpo, check-file-size e check-changelog-integrity OK, complexity 2788/3218 e cognitive 1261/1437, ESLint 0 erros nos 152 arquivos alterados, e a suíte vitest:ui completa (2149) verde.

Sobre esta PR especificamente: rodei os **23 arquivos de teste** que ela toca sobre o tip final, depois do merge da base — **392/392**. A migration `174_server_tool_executions.sql` não colide (o tip está em 173, e você já a renumerou em `c35f0fd7`).

O dono foi consultado antes do merge, porque o loop está atrás da flag `SERVER_OWNED_TOOL_LOOP_ENABLED` mas o primeiro send não-streaming mudou de dono sem flag, e a verificação manual em combo com Memory continuava desmarcada. A condição dele foi: entra se os testes focados passarem aqui. Passaram.

O lock de passthrough (`fetchCalls.length === 1`) é a parte que mais me convenceu — o double-dispatch que um `if (stream)` em volta do send existente causaria é exatamente o tipo de regressão que não aparece em teste de comportamento, só em contagem de chamada.

**Três ajustes meus na sua branch:**

1. `tests/unit/chatcore-stream-error-result.test.ts` procurava `"const legResult = await runNonStreamingProviderLeg"`, mas o seu commit final `6077b9dd` passou a reatribuir `legResult` e trocou para `let`. O guard falhava na sua própria branch (confirmei que o arquivo e o `chatCore.ts` eram byte-idênticos ao head da PR, então não era efeito da leva). Passou a aceitar `const|let` — a intenção do guard é o try/catch em volta da chamada, não a palavra-chave.

2. `tests/integration/skills-pipeline.test.ts` foi de 1156 para 1338 linhas e estourou o `testCap` de 1200. Segui o mesmo caminho que você já tinha tomado em `a1d2d20d` para os testes unitários: extraí os três casos do server-owned tool loop para `tests/integration/server-owned-tool-loop-pipeline.test.ts` (259 linhas), com instância própria do harness. O glob `tests/integration/*.test.ts` pega o arquivo novo sem registro adicional. 3/3 verdes isolados.

3. O arquivo novo herdou cinco `any` do original — que só passavam por estarem congelados no `eslint-suppressions.json` sob o nome antigo. Tipei como `Record<string, unknown>`. E `tests/unit/non-streaming-finalization.test.ts` tinha dois argumentos não usados em `trackPendingRequest`, agora prefixados com `_`.

Nada disso toca produção nem enfraquece asserção.
This commit is contained in:
Bob.Hou
2026-09-07 08:15:00 -04:00
committed by GitHub
parent ce49d969ca
commit d6f315018a
51 changed files with 12726 additions and 630 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
changelog.d/fixes/12696-server-owned-tool-loop.md Normal file
View File

@@ -0,0 +1 @@
- **fix(chat):** Chat Completions no longer return empty `content` after a server-owned memory or skills tool; the first provider send and account/model recovery now share one pipeline so a follow-up round-trip can fill the reply ([#12696](https://github.com/diegosouzapw/OmniRoute/issues/12696)) — thanks @HouMinXi

3
config/quality/file-size-baseline.json
View File

@@ -4,6 +4,7 @@
"_rebaseline_2026_09_04_12697_combo_pin_allowlist": "PR #12697 own growth: src/sse/handlers/chat.ts 2454->2458 (+4). checkModelAvailable preflight and handleSingleModelChat now call comboPinAllowlist so a pin-only combo step cannot scan the provider pool after 502/429. Helper lives in src/lib/combos/steps.ts under cap. Covered by tests/unit/combo-pin-implicit-allowlist.test.ts (11/11).",
"_rebaseline_2026_09_05_quota_weighted": "feat/quota-weighted-routing own growth: src/app/(dashboard)/dashboard/combos/page.tsx 5066->5080 (+14 = STRATEGY_GUIDANCE_FALLBACK + STRATEGY_RECOMMENDATIONS_FALLBACK entries for quota-weighted; copy is the spec-mandated when/avoid/example and tips, irreducible at the existing fallback maps). Rebased onto 9d1a896c6 where #12671 already grew the same file 5018->5066. Covered by tests/unit/combo/quota-weighted-strategy.test.ts + autocombo-unification.test.ts.",
"_rebaseline_2026_09_03_combo_execute_target_attempt": "Task 3 of handleComboChat split: new leaf open-sse/services/combo/executeTargetAttempt.ts lands at 1205 (check-file-size split-newline; wc -l 1204) above new-file cap 1200. Lift-as-is from combo.ts:1533-2616 retry loop. Pure classify predicates already extracted to executeTargetClassify.ts (54 LOC). Remaining growth is I/O + side effects (handleSingleModel, quality, pin/LKGP, cooldown, lockout) that cannot leave this file without splitting the retry loop mid-request. Frozen at exact LOC so it can only shrink. Covered by tests/unit/combo/execute-target-attempt.test.ts (7/7).",
"_rebaseline_2026_09_06_12696_chat_pipeline_retry_off": "PR #12696 own test growth: tests/integration/chat-pipeline.test.ts 1644->1648 (+4). The upstream-500 structured-error case now sets requestRetry/maxRetryIntervalSec to 0 so the new provider-execution pipeline cannot retry the mock 500 and double-count fetch. Irreducible at the existing seed+fetch mock; covered by the same test.",
"_rebaseline_2026_09_03_reset_aware_model_family": "Own growth: open-sse/services/combo.ts 4036->4041 (+5). buildAutoCandidates now keys the reset-aware quota cache by getQuotaFetchScope and spreads requestedModel onto the connection so Gemini windows stay off a Claude-empty Antigravity account. Irreducible wiring at the existing fetchResetAwareQuotaWithCache call site; the family helper itself lives in antigravityQuotaFamily.ts. Covered by tests/unit/reset-aware-request-scope-12600.test.ts.",
"_rebaseline_2026_09_03_overloaded_not_provider_breaker": "fix/overloaded-not-provider-breaker own growth: open-sse/services/combo.ts 4036->4075 (check-file-size split-newline, +39). Circuit-open pre-skip now records the breaker retryAfter and, when every target was skipped that way, waits the short reset via resolveCircuitOpenWaitDecision (new leaf in comboCooldownRetry.ts) instead of crystallizing ALL_TARGETS_SKIPPED in ~43ms. skippedForCircuitOpen / earliestCircuitOpenRetryMs reset each setTry so a later iteration cannot inherit a stale retryAfter. Irreducible at the existing ALL_TARGETS_SKIPPED chokepoint (same pattern as #7301/#8213 cooldown-wait). Predicate itself lives in circuitBreaker.ts / comboPredicates.ts / chatPredicates.ts, all under cap. Covered by tests/unit/overloaded-not-provider-breaker.test.ts + combo-cooldown-retry.test.ts.",
"_rebaseline_2026_09_03_12649_free_tier_reaudit_gateways": "PR #12649 (fix/free-tier-quota-reaudit) own growth: src/shared/constants/providers/apikey/gateways.ts 1459->1462 (+3 = the nara authHint rewritten for the re-audited 7M/day plan now wraps to two lines, plus the Prettier reflow of two pre-existing >100-col authHint lines (oneminai, freebuff) that lint-staged enforces on any touch of the file; additive text at the existing registry chokepoint, same god-file no-split rationale as prior gateways.ts rebaselines: #11786 seekai, #10987 logfare, #10531 freebuff). Covered by tests/unit/free-tier-reaudit-2026-09.test.ts and tests/unit/free-providers-batch-2026-07.test.ts.",
@@ -214,7 +215,7 @@
"_rebaseline_2026_08_24_video_bridge_fu01_fu03_fu04_result_cache_tests": "PRs #11362 (FU-01 cache hardening) + #11382 (FU-03 visual dedup policy identity) + #11383 (FU-04 focused analysis mode) own test growth: videoBridgeResultCache.test.ts <1000->1040, +40 (sum of three stacked PRs boarded together in the same merge-batch, each adding its own cache-identity assertions on the shared result-cache seam). Owner pre-authorized rebaseline for legitimate PR growth (2026-08-19 directive).",
"_rebaseline_basered_codebuddy_cn": "Base-red fix (#4664 CodeBuddy CN): oauth-providers-config.test.ts 867->870 (+3) to align the EXPECTED provider list/config with the codebuddy-cn provider that #4664 added to the registry without updating this test (it asserts 'exactly once').",
"_rebaseline_pr4613_compatible_provider_groups": "Reconcile #4613 already-merged growth: providers-page-utils.test.ts 1004->1052 (+48, buildCompatibleProviderGroups partition unit test). Fast-gate PR->release does not run check:file-size, so this surfaced post-merge.",
"tests/integration/chat-pipeline.test.ts": 1644,
"tests/integration/chat-pipeline.test.ts": 1648,
"tests/unit/account-fallback-service.test.ts": 2056,
"tests/unit/batch_api.test.ts": 1345,
"tests/unit/cc-compatible-provider.test.ts": 1225,

1079
open-sse/handlers/chatCore.ts
View File

File diff suppressed because it is too large Load Diff

39
open-sse/handlers/chatCore/failureUsage.ts
View File

@@ -23,6 +23,36 @@ export function projectFailureUsageErrorCode(opts: {
return errorBody.error.code || String(opts.statusCode);
}
export interface FailureUsageAggregate {
prompt_tokens?: number;
completion_tokens?: number;
cache_read_input_tokens?: number;
cache_creation_input_tokens?: number;
reasoning_tokens?: number;
}
export function toFailureUsageAggregate(
usage:
| {
prompt_tokens?: number;
completion_tokens?: number;
cache_read_input_tokens?: number;
cache_creation_input_tokens?: number;
reasoning_tokens?: number;
}
| null
| undefined
): FailureUsageAggregate | undefined {
if (!usage) return undefined;
return {
prompt_tokens: usage.prompt_tokens,
completion_tokens: usage.completion_tokens,
cache_read_input_tokens: usage.cache_read_input_tokens,
cache_creation_input_tokens: usage.cache_creation_input_tokens,
reasoning_tokens: usage.reasoning_tokens,
};
}
export function buildFailureUsageRecord(opts: {
provider: string | null | undefined;
model: string | null | undefined;
@@ -35,11 +65,18 @@ export function buildFailureUsageRecord(opts: {
errorCode: string | null | undefined;
latencyMs: number;
endpoint?: string | null | undefined;
aggregate?: FailureUsageAggregate | null;
}) {
return {
provider: opts.provider || "unknown",
model: opts.model || "unknown",
tokens: { input: 0, output: 0, cacheRead: 0, cacheCreation: 0, reasoning: 0 },
tokens: {
input: opts.aggregate?.prompt_tokens ?? 0,
output: opts.aggregate?.completion_tokens ?? 0,
cacheRead: opts.aggregate?.cache_read_input_tokens ?? 0,
cacheCreation: opts.aggregate?.cache_creation_input_tokens ?? 0,
reasoning: opts.aggregate?.reasoning_tokens ?? 0,
},
status: String(opts.statusCode),
success: false,
latencyMs: opts.latencyMs,

83
open-sse/handlers/chatCore/memorySkillsInjection.ts
View File

@@ -5,7 +5,7 @@ import {
toMemoryRetrievalConfig,
} from "@/lib/memory/settings";
import { injectMemory, shouldInjectMemory } from "@/lib/memory/injection";
import { injectSkills } from "@/lib/skills/injection";
import { injectSkillsWithMetadata } from "@/lib/skills/injection";
import { buildMemoryToolsForProvider } from "@/lib/skills/memoryBuiltins";
import { skillRegistry } from "@/lib/skills/registry";
import { FORMATS } from "../../translator/formats.ts";
@@ -13,6 +13,13 @@ import { detectCachingContext } from "../../services/compression/cachingAware.ts
type MemorySkillsLogger = { debug?: (...args: unknown[]) => void } | null | undefined;
export interface MemorySkillsInjectionResult {
body: Record<string, unknown>;
memorySettings: { enabled: boolean; skillsEnabled: boolean; maxTokens: number } | null;
builtinToolNames: string[];
injectedCustomSkillNames: string[];
}
function getToolName(tool: unknown): string {
if (!tool || typeof tool !== "object") return "";
const r = tool as Record<string, unknown>;
@@ -60,11 +67,14 @@ export async function injectMemoryAndSkills({
targetFormat: string;
backgroundReason: string | null;
log: MemorySkillsLogger;
}) {
}): Promise<MemorySkillsInjectionResult> {
const memorySettings = memoryOwnerId
? await getMemorySettings().catch(() => DEFAULT_MEMORY_SETTINGS)
: null;
const builtinOwnerSet: string[] = [];
const injectedCustomSkillNames: string[] = [];
if (
memoryOwnerId &&
memorySettings &&
@@ -178,34 +188,48 @@ export async function injectMemoryAndSkills({
return [];
})
);
const memoryTools = buildMemoryToolsForProvider(
const newMemoryTools = buildMemoryToolsForProvider(
getSkillsProviderForFormat(sourceFormat)
).filter((tool) => {
const record = tool as Record<string, unknown>;
const name = (record.function as Record<string, unknown> | undefined)?.name ?? record.name;
return typeof name === "string" && !existingToolNames.has(name);
});
if (memoryTools.length > 0) {
if (newMemoryTools.length > 0) {
body = {
...body,
tools: [...existingTools, ...memoryTools],
tools: [...existingTools, ...newMemoryTools],
};
// Track the names of newly injected memory tools for the owner set.
builtinOwnerSet.push(
...newMemoryTools
.map((tool) => {
const record = tool as Record<string, unknown>;
const name =
(record.function as Record<string, unknown> | undefined)?.name ?? record.name;
return typeof name === "string" ? name : "";
})
.filter(Boolean)
);
log?.debug?.(
"MEMORY",
`Injected ${memoryTools.length} memory tool(s) for key=${memoryOwnerId}`
`Injected ${newMemoryTools.length} memory tool(s) for key=${memoryOwnerId}`
);
}
}
if (memoryOwnerId && memorySettings?.skillsEnabled) {
if (memoryOwnerId && memorySettings?.skillsEnabled && body.stream !== true) {
// Ensure the registry cache is warm before listing: on a cold/fresh
// process skills that exist only in the DB would be missed (false
// negative -> silent skip). loadFromDatabase() is a no-op when the cache
// is already warm (TTL = 60 s), so repeated calls are cheap. Mirrors the
// pattern in src/lib/skills/interception.ts (#2815).
// Memory builtins and registered Skills are only executed by the
// non-streaming server-owned tool loop; stream clients execute tools
// client-side, so we skip injection for stream requests.
await skillRegistry.loadFromDatabase(memoryOwnerId);
const existingTools = Array.isArray(body.tools) ? body.tools : [];
const mergedTools = injectSkills({
const { tools: mergedTools, injectedNames } = injectSkillsWithMetadata({
provider: getSkillsProviderForFormat(sourceFormat),
existingTools,
apiKeyId: memoryOwnerId,
@@ -225,6 +249,7 @@ export async function injectMemoryAndSkills({
...body,
tools: mergedTools,
};
injectedCustomSkillNames.push(...injectedNames);
log?.debug?.("SKILLS", `Injected ${mergedTools.length - existingTools.length} skills`);
}
}
@@ -236,5 +261,45 @@ export async function injectMemoryAndSkills({
};
}
return { body, memorySettings };
return { body, memorySettings, builtinToolNames: builtinOwnerSet, injectedCustomSkillNames };
}
interface FallbackPlan {
enabled: boolean;
toolName: string | null;
convertedToolCount: number;
}
/**
* Pure helper: merge web-search/web-fetch fallback tool names into the
* builtin owner set. Adds a name only when plan.enabled===true,
* plan.convertedToolCount>0, plan.toolName is non-null, and that name
* did not already exist in the pre-conversion client tools (builtinToolNames)
* OR in the original client tool names captured before fallback injection.
* Does not mutate its input; returns a new result.
*/
export function mergeInjectedFallbackOwnerNames(
injectionResult: { builtinToolNames: string[] },
plans: FallbackPlan[],
preConversionClientToolNames?: string[]
): { builtinToolNames: string[] } {
const existing = new Set(injectionResult.builtinToolNames);
if (preConversionClientToolNames) {
for (const name of preConversionClientToolNames) {
existing.add(name);
}
}
const extraNames: string[] = [];
for (const plan of plans) {
if (
plan.enabled &&
plan.convertedToolCount > 0 &&
plan.toolName &&
!existing.has(plan.toolName)
) {
extraNames.push(plan.toolName);
existing.add(plan.toolName);
}
}
return { builtinToolNames: [...injectionResult.builtinToolNames, ...extraNames] };
}

159
open-sse/handlers/chatCore/nonStreamingClientTranslate.ts Normal file
View File

@@ -0,0 +1,159 @@
/**
* Client translation for non-streaming responses.
* Extracted from chatCore.ts (lines ~5098-5195) by symbol boundaries.
*
* Handles: translate, tool-name restore, finish-reason normalization, sanitize,
* reasoning replay capture, and client usage buffer application.
*
* Phase distinction:
* - "final": applies applyClientUsageBuffer (normalizes visible usage fields)
* - "intermediate": skips usage buffer (raw usage preserved for aggregation)
*/
import type {
NonStreamingClientTranslateInput,
NonStreamingClientTranslateResult,
} from "@/lib/skills/toolLoopTypes.ts";
import { needsTranslation } from "../../translator/index.ts";
import { FORMATS } from "../../translator/formats.ts";
import { translateNonStreamingResponse } from "../responseTranslator.ts";
import { extractToolSchemaMap } from "../../translator/response/openai-responses/toolSchemas.ts";
import { stripMarkdownCodeFence } from "../../utils/aiSdkCompat.ts";
import { normalizeOpenAIToolFinishReasons } from "./passthroughToolNames.ts";
import {
cacheReasoningFromAssistantMessage,
requiresReasoningReplay,
} from "../../services/reasoningCache.ts";
import {
sanitizeOpenAIResponse,
sanitizeResponsesApiResponse,
shouldParseTextualReasoningTags,
} from "../responseSanitizer.ts";
import { isStripReasoningRequested } from "./headers.ts";
import { applyClientUsageBuffer } from "./clientUsageBuffer.ts";
export type { NonStreamingClientTranslateInput, NonStreamingClientTranslateResult };
/**
* Translate a non-streaming provider response to the client's expected format.
*
* All of: translate, tool-name identity restore, finish-reason normalization,
* and sanitize are applied every round (both intermediate and final).
* `applyClientUsageBuffer` is applied only for `phase === "final"`.
* Reasoning replay capture runs every round.
*/
export function translateNonStreamingClientResponse(
input: NonStreamingClientTranslateInput
): NonStreamingClientTranslateResult {
const {
responseBody,
responsePayloadFormat,
clientResponseFormat,
sourceFormat,
provider,
model,
requestBody,
responseToolNameMap,
requestToolIdentityMap,
reasoningCacheScope,
clientHeaders,
isClaudeCodeCompatible,
phase,
} = input;
// ── Extract tool schemas for schema-aware translation ──────────────────────
const finalBody = requestBody as Record<string, unknown> | null;
const responseToolSchemas = extractToolSchemaMap(finalBody || responseBody);
// ── Translate response to client's expected format ─────────────────────────
let translatedResponse = needsTranslation(responsePayloadFormat, clientResponseFormat)
? translateNonStreamingResponse(
responseBody,
responsePayloadFormat,
clientResponseFormat,
responseToolNameMap,
responseToolSchemas
)
: responseBody;
const responseForMemoryExtraction = translatedResponse;
// ── T26: Strip markdown code blocks if provider format is Claude ───────────
if (sourceFormat === "claude") {
if (typeof translatedResponse?.choices?.[0]?.message?.content === "string") {
translatedResponse.choices[0].message.content = stripMarkdownCodeFence(
translatedResponse.choices[0].message.content
) as string;
}
}
// ── T18: Normalize finish_reason to 'tool_calls' if tool calls present ─────
normalizeOpenAIToolFinishReasons(translatedResponse);
// ── Reasoning Replay Cache (#1628) ────────────────────────────────────────
// Capture reasoning_content from non-streaming responses with tool_calls
// so it can be replayed on subsequent turns.
try {
const cacheResponse = translatedResponse?.choices?.[0]
? translatedResponse
: needsTranslation(responsePayloadFormat, FORMATS.OPENAI)
? translateNonStreamingResponse(
responseBody,
responsePayloadFormat,
FORMATS.OPENAI,
responseToolNameMap,
responseToolSchemas
)
: responseBody;
const firstChoice = cacheResponse?.choices?.[0];
const msg = firstChoice?.message;
// Prefer explicit historyMessages (parent: translatedBody.messages). Do not
// overload requestBody — Responses-shaped finalBody has `input`, not `messages`.
const historyMessages = Array.isArray(input.historyMessages)
? input.historyMessages
: (finalBody as { messages?: unknown[] } | null | undefined)?.messages;
if (requiresReasoningReplay({ provider, model })) {
cacheReasoningFromAssistantMessage(msg, provider, model, {
scope: reasoningCacheScope,
historyMessages: Array.isArray(historyMessages) ? historyMessages : [],
});
}
} catch {
// Cache capture is non-critical — never block the response
}
// ── Sanitize response for SDK compatibility ────────────────────────────────
if (clientResponseFormat === FORMATS.OPENAI_RESPONSES) {
translatedResponse = sanitizeResponsesApiResponse(translatedResponse);
// Restore {namespace, name} on function_call items for round-trip closure (#7936)
const responseOutput = translatedResponse?.output;
if (requestToolIdentityMap && Array.isArray(responseOutput)) {
for (const item of responseOutput) {
if (item?.type !== "function_call") continue;
const identity = requestToolIdentityMap.get(item.name);
if (identity) {
item.namespace = identity.namespace;
item.name = identity.name;
}
}
}
} else if (clientResponseFormat === FORMATS.OPENAI) {
const stripReasoning = isStripReasoningRequested(clientHeaders ?? null);
translatedResponse = sanitizeOpenAIResponse(translatedResponse, {
stripReasoning,
parseTextualReasoningTags: shouldParseTextualReasoningTags(provider, model),
});
}
// ── Client usage buffer (#8331) ───────────────────────────────────────────
// Only apply for final phase; intermediate preserves raw usage for aggregation.
if (phase === "final") {
applyClientUsageBuffer(translatedResponse, finalBody || responseBody, clientResponseFormat, {
preserveContextBudgetInVisibleUsage: isClaudeCodeCompatible,
});
}
return {
response: translatedResponse,
responseForMemoryExtraction,
};
}

133
open-sse/handlers/chatCore/nonStreamingFinalization.ts Normal file
View File

@@ -0,0 +1,133 @@
/**
* Request-level finalization for non-streaming chat.
* Success and failure each write usage/cost/quota/attempt/pending once.
*/
import type {
ChatCoreErrorResult,
ProviderLegUsage,
ServerOwnedToolLoopResult,
} from "@/lib/skills/toolLoopTypes.ts";
import type { PersistAttemptLogsArgs } from "./attemptLogging.ts";
import { type FailureUsageAggregate, toFailureUsageAggregate } from "./failureUsage.ts";
export type NonStreamingFinalizationPlan =
| {
kind: "success";
usage: ProviderLegUsage | null;
totalCostUsd: number;
receiptCount: number;
}
| {
kind: "failure";
error: ChatCoreErrorResult;
usage: ProviderLegUsage | null;
totalCostUsd: number;
receiptCount: number;
};
export interface NonStreamingFinalizationDeps {
writeUsage: (plan: NonStreamingFinalizationPlan) => void | Promise<void>;
writeCost: (totalCostUsd: number) => void;
scheduleQuota: (plan: NonStreamingFinalizationPlan) => void | Promise<void>;
writeAttempt: (plan: NonStreamingFinalizationPlan) => void;
finalizePending: (plan: NonStreamingFinalizationPlan) => void;
}
function missingError(): ChatCoreErrorResult {
return {
success: false,
status: 500,
response: new Response(null, { status: 500 }),
error: "Missing tool-loop error result",
errorCode: "internal_error",
};
}
export function buildNonStreamingFinalizationPlan(
loop: ServerOwnedToolLoopResult
): NonStreamingFinalizationPlan {
const usage = loop.cumulativeUsage;
const totalCostUsd = loop.totalCostUsd;
const receiptCount = loop.receipts.length;
if (loop.kind === "error") {
return {
kind: "failure",
error: loop.errorResult ?? missingError(),
usage,
totalCostUsd,
receiptCount,
};
}
return {
kind: "success",
usage,
totalCostUsd,
receiptCount,
};
}
export async function finalizeNonStreamingRequest(
plan: NonStreamingFinalizationPlan,
deps: NonStreamingFinalizationDeps
): Promise<void> {
await deps.writeUsage(plan);
deps.writeCost(plan.totalCostUsd);
if (plan.kind === "success") {
await deps.scheduleQuota(plan);
}
deps.writeAttempt(plan);
deps.finalizePending(plan);
}
export async function finalizeToolLoopError(input: {
loop: ServerOwnedToolLoopResult;
model: string;
provider: string;
connectionId?: string;
providerRequest?: Record<string, unknown>;
persistFailureUsage: (
status: number,
errorCode: string,
usage?: FailureUsageAggregate | null
) => void;
persistAttemptLogs: (params: PersistAttemptLogsArgs) => void;
trackPendingRequest: (
model: string,
provider: string,
connectionId?: string,
isPending?: boolean
) => void;
}): Promise<ChatCoreErrorResult> {
const plan = buildNonStreamingFinalizationPlan(input.loop);
const err = plan.kind === "failure" ? plan.error : missingError();
await finalizeNonStreamingRequest(plan, {
writeUsage: () => {
input.persistFailureUsage(
err.status,
err.errorCode || `upstream_${err.status}`,
toFailureUsageAggregate(plan.usage)
);
},
writeCost: () => {},
scheduleQuota: () => {},
writeAttempt: () => {
input.persistAttemptLogs({
status: err.status,
error: err.error || "Provider request failed",
providerRequest: input.providerRequest,
clientResponse: {
error: {
message: err.error || "Provider request failed",
type: err.errorType || "api_error",
},
},
cacheSource: "upstream",
});
},
finalizePending: () => {
input.trackPendingRequest(input.model, input.provider, input.connectionId, false);
},
});
return err;
}

1144
open-sse/handlers/chatCore/nonStreamingProviderLeg.ts Normal file
View File

File diff suppressed because it is too large Load Diff

449
open-sse/handlers/chatCore/providerExecutionPipeline.ts Normal file
View File

@@ -0,0 +1,449 @@
import type { ChatCoreErrorResult, ProviderLegUsage } from "@/lib/skills/toolLoopTypes.ts";
import type { getProviderCredentials } from "@/sse/services/auth.ts";
import type { updateFromHeaders, updateFromResponseBody } from "../../services/rateLimitManager.ts";
import type { writeTerminalStatus } from "@/shared/utils/terminalStatus.ts";
import type { updateProviderConnection } from "@/lib/db/providers.ts";
import type { lockModel, recordCoreOwnedAntigravityQuotaState } from "../../services/accountFallback.ts";
import { createErrorResult } from "../../utils/error.ts";
import { applyStatusRestatement } from "../../config/upstreamStatusRestatement.ts";
import { recoverAnthropicThinkingSignature } from "./thinkingSignatureRecovery.ts";
import { isModelUnavailableError, getNextFamilyFallback as defaultGetNextFamilyFallback } from "../../services/modelFamilyFallback.ts";
import { COOLDOWN_MS } from "../../config/errorConfig.ts";
import { normalizeHeaders } from "../../utils/headers.ts";
export interface ChatCoreExecutorResult {
response: Response;
url: string;
headers: Record<string, string>;
transformedBody: unknown;
transport?: string;
_executionCredentials?: Record<string, unknown>;
_accountSemaphoreRelease?: () => void;
}
export interface ProviderExecutionPolicy {
allowAccountRotation: boolean;
allowModelFallback: boolean;
expectedConnectionId?: string;
}
export type ProviderExecutionOutcome =
| {
kind: "response";
response: Response;
url: string;
headers: Record<string, string>;
transformedBody: unknown;
model: string;
connectionId: string;
}
| {
kind: "error";
result: ChatCoreErrorResult;
providerUsage: ProviderLegUsage | null;
model: string;
connectionId: string;
};
export interface PipelineTargetContext {
provider: string;
requestedModel: string;
sourceFormat: string;
targetFormat: string;
stream: boolean;
}
export interface PipelineConnectionContext {
initialConnectionId: string;
getCurrentConnectionId: () => string | undefined;
getCredentials: () => Record<string, unknown>;
replaceCredentials: (next: Record<string, unknown>) => void;
onCredentialsRefreshed: (next: Record<string, unknown>) => void | Promise<void>;
assertManagedLeaseFence: (connectionId: string) => void;
getProviderCredentials: typeof getProviderCredentials;
refreshCredentials?: (
credentials: Record<string, unknown>
) => Promise<Record<string, unknown> | null>;
}
export interface PipelineWireState {
body: Record<string, unknown>;
currentModel: string;
triedModels: Set<string>;
setBodyAndModel: (body: Record<string, unknown>, model: string) => void;
}
export interface PipelineStateHooks {
updatePendingStage: (stage: string, data?: Record<string, unknown>) => void;
recordRateLimitHeaders: typeof updateFromHeaders;
recordRateLimitBody: typeof updateFromResponseBody;
writeTerminalStatus: typeof writeTerminalStatus;
persistConnectionPatch: typeof updateProviderConnection;
setConnectionRateLimitedUntil: (
connectionId: string,
untilMs: number | null
) => void | Promise<void>;
lockModel: typeof lockModel;
recordAntigravityQuotaState: typeof recordCoreOwnedAntigravityQuotaState;
markAccountSemaphoreBlocked: (connectionId: string) => void;
isolateProbeFailures: () => boolean | Promise<boolean>;
onCodexScopeRateLimited?: (params: {
failedConnectionId: string;
model: string | null;
rateLimitedUntil: string;
credentials?: Record<string, unknown> | null;
}) => void | Promise<void>;
onClearSessionAffinity?: (params: { failedConnectionId: string }) => void | Promise<void>;
onAuditAccountRotation?: (params: {
action: "codex.account_rotation";
failedConnectionId: string;
newConnectionId: string;
attempt: number;
retryAfterMs: number | null;
}) => void | Promise<void>;
}
export interface ProviderExecutionPipelineInput {
policy: Readonly<ProviderExecutionPolicy>;
target: PipelineTargetContext;
connection: PipelineConnectionContext;
wire: PipelineWireState;
state: PipelineStateHooks;
sendProviderAttempt: (model: string, allowDedup: boolean) => Promise<ChatCoreExecutorResult>;
getNextFamilyFallback?: (
currentModel: string,
triedModels: Set<string>,
providerHint?: string | null
) => string | null;
}
const LEASE_MISMATCH_STATUS = 409;
const LEASE_MISMATCH_CODE = "LEASE_CONNECTION_MISMATCH";
function currentConnectionId(connection: PipelineConnectionContext): string {
return connection.getCurrentConnectionId() ?? connection.initialConnectionId;
}
function retryAfterMsFrom(attempt: ChatCoreExecutorResult): number | null {
// attempt.headers is the outbound request bag (BaseExecutor finalHeaders).
// Retry-After lives on the upstream Response — same source as the parent
// chatCore rotate path. normalizeHeaders lower-cases keys, so "Retry-After"
// is looked up as "retry-after"; it does not drop the field.
const raw = normalizeHeaders(attempt.response?.headers)["retry-after"];
if (raw == null || raw === "") return null;
const parsed = Number.parseFloat(String(raw));
if (!Number.isFinite(parsed) || parsed < 0) return null;
return parsed * 1000;
}
function leaseMismatch(model: string, connectionId: string): ProviderExecutionOutcome {
const result = createErrorResult(
LEASE_MISMATCH_STATUS,
"Managed lease connection mismatch",
null,
LEASE_MISMATCH_CODE,
"lease_error"
);
return {
kind: "error",
result: {
success: false,
status: result.status,
response: result.response,
error: result.error,
errorCode: LEASE_MISMATCH_CODE,
errorType: "lease_error",
},
providerUsage: null,
model,
connectionId,
};
}
async function toOutcome(
attempt: ChatCoreExecutorResult,
model: string,
connectionId: string,
provider: string
): Promise<ProviderExecutionOutcome> {
const status = attempt.response.status;
if (status >= 200 && status < 300) {
return {
kind: "response",
response: attempt.response,
url: attempt.url,
headers: attempt.headers,
transformedBody: attempt.transformedBody,
model,
connectionId,
};
}
let message = attempt.response.statusText || "upstream error";
let body: unknown = attempt.transformedBody;
try {
// clone() is the drain. sendProviderAttempt must not cancel() a streaming
// non-2xx body before we get here (BYOP 422 / Codex 429 Retry-After).
body = JSON.parse(await attempt.response.clone().text());
const err = (body as { error?: { message?: unknown } } | null)?.error;
if (err && typeof err.message === "string" && err.message) message = err.message;
} catch {
// keep statusText
}
const restatement = applyStatusRestatement({
provider,
status,
message,
body,
retryAfterMs: null,
});
const result = createErrorResult(
restatement.status,
message,
restatement.retryAfterMs
);
return {
kind: "error",
result: {
success: false,
status: result.status,
response: attempt.response,
error: result.error,
errorCode: result.errorCode,
errorType: result.errorType,
},
providerUsage: null,
model,
connectionId,
};
}
function assertLease(
policy: Readonly<ProviderExecutionPolicy>,
connection: PipelineConnectionContext,
model: string
): ProviderExecutionOutcome | null {
const expected = policy.expectedConnectionId;
if (!expected) return null;
const current = connection.getCurrentConnectionId();
if (current && current !== expected) {
return leaseMismatch(model, current);
}
return null;
}
function maxAttemptsFor(provider: string): number {
return provider === "codex" ? 3 : 1;
}
/**
* Shared first-send + provider recovery. Does not read a successful body.
* Account/model retries live here; sendProviderAttempt is one wire send.
*/
export async function runProviderExecutionPipeline(
input: ProviderExecutionPipelineInput
): Promise<ProviderExecutionOutcome> {
const { policy, target, connection, wire, state, sendProviderAttempt } = input;
const maxAttempts = maxAttemptsFor(target.provider);
const excludedIds: string[] = [];
let attempts = 0;
let lastAttempt: ChatCoreExecutorResult | null = null;
let antigravityByopRotationPending = false;
let authRefreshPending = false;
let authRefreshed = false;
let modelFallbackPending = false;
const resolveFamilyFallback = input.getNextFamilyFallback ?? defaultGetNextFamilyFallback;
while (
attempts < maxAttempts ||
antigravityByopRotationPending ||
authRefreshPending ||
modelFallbackPending
) {
antigravityByopRotationPending = false;
authRefreshPending = false;
modelFallbackPending = false;
const before = assertLease(policy, connection, wire.currentModel);
if (before) return before;
const attempt = await sendProviderAttempt(wire.currentModel, attempts === 0);
lastAttempt = attempt;
const after = assertLease(policy, connection, wire.currentModel);
if (after) return after;
const status = attempt.response.status;
if (status >= 200 && status < 300) {
return toOutcome(attempt, wire.currentModel, currentConnectionId(connection), target.provider);
}
const isolateProbe = await state.isolateProbeFailures();
const canRotateAccount = policy.allowAccountRotation && !isolateProbe;
if (
canRotateAccount &&
target.provider === "codex" &&
status === 429 &&
attempts < maxAttempts - 1
) {
const failedId = currentConnectionId(connection);
const retryAfterMs = retryAfterMsFrom(attempt);
if (failedId && !excludedIds.includes(failedId)) excludedIds.push(failedId);
if (failedId) {
await state.onCodexScopeRateLimited?.({
failedConnectionId: failedId,
model: wire.currentModel || target.requestedModel || null,
rateLimitedUntil: new Date(Date.now() + (retryAfterMs || 60_000)).toISOString(),
credentials: connection.getCredentials(),
});
await state.onClearSessionAffinity?.({ failedConnectionId: failedId });
}
const nextCreds = await connection
.getProviderCredentials("codex", null, null, wire.currentModel, {
excludeConnectionIds: [...excludedIds],
})
.catch(() => null);
if (nextCreds && !nextCreds.allRateLimited && nextCreds.connectionId) {
await state.onAuditAccountRotation?.({
action: "codex.account_rotation",
failedConnectionId: failedId,
newConnectionId: String(nextCreds.connectionId),
attempt: attempts + 1,
retryAfterMs,
});
connection.replaceCredentials(nextCreds as Record<string, unknown>);
attempts += 1;
continue;
}
}
if (canRotateAccount && target.provider === "antigravity" && status === 422) {
// Same drain as toOutcome: clone the Response. A prior body.cancel()
// makes this throw "Body has already been consumed" and skips rotate.
const byopBody = await attempt.response
.clone()
.text()
.catch(() => "");
if (byopBody.includes("gcp_project_required")) {
const failedId = currentConnectionId(connection);
if (failedId && !excludedIds.includes(failedId)) excludedIds.push(failedId);
if (failedId) {
await state.setConnectionRateLimitedUntil(
failedId,
Date.now() + (COOLDOWN_MS.gcpProjectRequired ?? 24 * 60 * 60 * 1000)
);
}
const nextCreds = await connection
.getProviderCredentials("antigravity", null, null, wire.currentModel, {
excludeConnectionIds: [...excludedIds],
})
.catch(() => null);
if (nextCreds && !nextCreds.allRateLimited && nextCreds.connectionId) {
connection.replaceCredentials(nextCreds as Record<string, unknown>);
antigravityByopRotationPending = true;
continue;
}
}
}
if (
!authRefreshed &&
(status === 401 || status === 403) &&
typeof connection.refreshCredentials === "function"
) {
const refreshed = await connection.refreshCredentials(connection.getCredentials());
if (refreshed && (refreshed.accessToken || refreshed.copilotToken)) {
connection.replaceCredentials({ ...connection.getCredentials(), ...refreshed });
await connection.onCredentialsRefreshed(refreshed);
authRefreshed = true;
authRefreshPending = true;
continue;
}
}
{
let signatureMessage = attempt.response.statusText || "upstream error";
try {
const parsed = JSON.parse(await attempt.response.clone().text()) as {
error?: { message?: unknown };
};
if (typeof parsed?.error?.message === "string" && parsed.error.message) {
signatureMessage = parsed.error.message;
}
} catch {
// keep statusText
}
const signatureRecovery = await recoverAnthropicThinkingSignature({
provider: target.provider,
statusCode: status,
message: signatureMessage,
body: wire.body,
execute: async (recoveryBody) => {
if (recoveryBody && typeof recoveryBody === "object" && !Array.isArray(recoveryBody)) {
wire.setBodyAndModel(recoveryBody as Record<string, unknown>, wire.currentModel);
}
return sendProviderAttempt(wire.currentModel, false);
},
parseError: async (response) => {
let message = response.statusText || "upstream error";
let responseBody: unknown = null;
try {
responseBody = JSON.parse(await response.clone().text());
const err = (responseBody as { error?: { message?: unknown } } | null)?.error;
if (typeof err?.message === "string" && err.message) message = err.message;
} catch {
// keep statusText
}
return {
statusCode: response.status,
message,
retryAfterMs: null,
responseBody,
};
},
});
if (signatureRecovery.attempted && signatureRecovery.succeeded && signatureRecovery.execution) {
lastAttempt = {
response: signatureRecovery.execution.response,
url: signatureRecovery.execution.url ?? attempt.url,
headers: (signatureRecovery.execution.headers as Record<string, string>) ?? attempt.headers,
transformedBody: signatureRecovery.execution.transformedBody ?? attempt.transformedBody,
};
return toOutcome(
lastAttempt,
wire.currentModel,
currentConnectionId(connection),
target.provider
);
}
}
if (policy.allowModelFallback) {
let fallbackMessage = attempt.response.statusText || "upstream error";
try {
const parsed = JSON.parse(await attempt.response.clone().text()) as {
error?: { message?: unknown };
};
if (typeof parsed?.error?.message === "string" && parsed.error.message) {
fallbackMessage = parsed.error.message;
}
} catch {
// keep statusText
}
if (isModelUnavailableError(status, fallbackMessage, target.provider)) {
const nextModel = resolveFamilyFallback(wire.currentModel, wire.triedModels, target.provider);
if (nextModel) {
wire.setBodyAndModel({ ...wire.body, model: nextModel }, nextModel);
modelFallbackPending = true;
continue;
}
}
}
return toOutcome(attempt, wire.currentModel, currentConnectionId(connection), target.provider);
}
if (lastAttempt) {
return toOutcome(lastAttempt, wire.currentModel, currentConnectionId(connection), target.provider);
}
return leaseMismatch(wire.currentModel, currentConnectionId(connection));
}

15
open-sse/handlers/chatCore/serverOwnedToolLoopGate.ts Normal file
View File

@@ -0,0 +1,15 @@
import { FORMATS } from "../../translator/formats.ts";
export function shouldRunServerOwnedToolLoop(input: {
enabled: boolean;
stream: boolean;
isResponsesEndpoint: boolean;
sourceFormat: string;
}): boolean {
if (!input.enabled) return false;
if (input.stream) return false;
if (input.isResponsesEndpoint) return false;
if (input.sourceFormat === FORMATS.OPENAI) return true;
if (input.sourceFormat === FORMATS.CLAUDE) return true;
return false;
}

175
open-sse/handlers/chatCore/serverOwnedToolLoopWire.ts Normal file
View File

@@ -0,0 +1,175 @@
import type {
ExecutionContext,
NonStreamingProviderLegResult,
ProviderLegUsage,
ServerOwnedToolLoopResult,
ToolCall,
} from "@/lib/skills/toolLoopTypes.ts";
import { executeServerOwned } from "@/lib/skills/interception";
import { runServerOwnedToolLoop, LOOP_BUDGET_MS } from "@/lib/skills/serverOwnedToolLoop.ts";
import { deriveToolRequestIdentity } from "@/lib/skills/stableJson.ts";
import { getIdempotencyKey } from "@/lib/idempotencyLayer";
import { runNonStreamingProviderLeg } from "./nonStreamingProviderLeg.ts";
import type { ProviderLegInput } from "./nonStreamingProviderLeg.ts";
import { shouldRunServerOwnedToolLoop } from "./serverOwnedToolLoopGate.ts";
import { FORMATS } from "../../translator/formats.ts";
export function derivePostInjectionRequestIdentity(input: {
apiKeyId: string;
headers: unknown;
skillRequestId: string;
postInjectionBody: Record<string, unknown>;
}): string {
const stableClientRequestId = getIdempotencyKey(input.headers as never);
return deriveToolRequestIdentity({
apiKeyId: input.apiKeyId,
stableClientRequestId,
skillRequestId: input.skillRequestId,
postInjectionBody: input.postInjectionBody,
});
}
export async function continueServerOwnedToolLoop(input: {
initialLeg: NonStreamingProviderLegResult & { kind: "ok" };
sourceBody: Record<string, unknown>;
sourceFormat: "openai" | "claude";
skillsModelId: string;
executionContext: ExecutionContext;
abortSignal?: AbortSignal;
deadlineAtMs: number;
expectedConnectionId?: string;
followUpLeg: (nextSourceBody: Record<string, unknown>) => Promise<NonStreamingProviderLegResult>;
executeServerOwned?: (
calls: ToolCall[],
context: ExecutionContext
) => Promise<import("@/lib/skills/toolLoopTypes.ts").ExecutedToolResult[]>;
}): Promise<ServerOwnedToolLoopResult> {
const runOwned = input.executeServerOwned ?? executeServerOwned;
return runServerOwnedToolLoop({
initialLeg: input.initialLeg,
sourceBody: input.sourceBody,
sourceFormat: input.sourceFormat,
skillsModelId: input.skillsModelId,
executionContext: input.executionContext,
abortSignal: input.abortSignal,
deadlineAtMs: input.deadlineAtMs,
executeServerOwned: (calls: ToolCall[], context: ExecutionContext) => runOwned(calls, context),
resumeUpstream: async (nextSourceBody, expectedConnectionId) => {
if (
expectedConnectionId &&
input.expectedConnectionId &&
expectedConnectionId !== input.expectedConnectionId
) {
return {
kind: "error",
result: {
success: false,
status: 409,
response: new Response(null, { status: 409 }),
error: "Follow-up connection mismatch",
errorCode: "LEASE_CONNECTION_MISMATCH",
},
receipt: input.initialLeg.receipt,
usage: null,
};
}
return input.followUpLeg(nextSourceBody);
},
});
}
export function followUpLegInput(
base: Omit<
ProviderLegInput,
"phase" | "allowAccountRotation" | "allowModelFallback" | "sourceBody"
>,
nextSourceBody: Record<string, unknown>,
expectedConnectionId?: string
): ProviderLegInput {
return {
...base,
phase: "follow-up",
sourceBody: nextSourceBody,
expectedConnectionId,
allowAccountRotation: false,
allowModelFallback: false,
};
}
export function mergeLoopIntoOkLeg(
leg: NonStreamingProviderLegResult & { kind: "ok" },
loop: ServerOwnedToolLoopResult
): NonStreamingProviderLegResult & { kind: "ok" } {
return {
...leg,
response: loop.response ?? leg.response,
responseForMemoryExtraction:
loop.responseForMemoryExtraction ?? leg.responseForMemoryExtraction,
providerBody: loop.finalProviderBody ?? leg.providerBody,
providerRequest: loop.finalProviderRequest ?? leg.providerRequest,
usage: loop.cumulativeUsage,
};
}
export type ToolLoopApplyResult =
| { kind: "skip" }
| {
kind: "ok";
leg: NonStreamingProviderLegResult & { kind: "ok" };
usage: ProviderLegUsage | null;
loop: ServerOwnedToolLoopResult;
}
| { kind: "error"; loop: ServerOwnedToolLoopResult };
export async function applyServerOwnedToolLoopIfNeeded(input: {
enabled: boolean;
stream: boolean;
isResponsesEndpoint: boolean;
sourceFormat: string;
initialLeg: NonStreamingProviderLegResult;
sourceBody: Record<string, unknown>;
skillsModelId: string;
executionContext: ExecutionContext;
abortSignal?: AbortSignal;
expectedConnectionId?: string;
followUpLeg: (nextSourceBody: Record<string, unknown>) => Promise<NonStreamingProviderLegResult>;
logReceipt: (receipt: ServerOwnedToolLoopResult["receipts"][number]) => void;
executeServerOwned?: (
calls: ToolCall[],
context: ExecutionContext
) => Promise<import("@/lib/skills/toolLoopTypes.ts").ExecutedToolResult[]>;
}): Promise<ToolLoopApplyResult> {
if (
input.initialLeg.kind !== "ok" ||
!shouldRunServerOwnedToolLoop({
enabled: input.enabled,
stream: input.stream,
isResponsesEndpoint: input.isResponsesEndpoint,
sourceFormat: input.sourceFormat,
})
) {
return { kind: "skip" };
}
const loop = await continueServerOwnedToolLoop({
initialLeg: input.initialLeg,
sourceBody: input.sourceBody,
sourceFormat: input.sourceFormat === FORMATS.CLAUDE ? "claude" : "openai",
skillsModelId: input.skillsModelId,
executionContext: input.executionContext,
abortSignal: input.abortSignal,
deadlineAtMs: Date.now() + LOOP_BUDGET_MS,
expectedConnectionId: input.expectedConnectionId,
followUpLeg: input.followUpLeg,
executeServerOwned: input.executeServerOwned,
});
for (const receipt of loop.receipts) input.logReceipt(receipt);
if (loop.kind === "error") return { kind: "error", loop };
return {
kind: "ok",
leg: mergeLoopIntoOkLeg(input.initialLeg, loop),
usage: loop.cumulativeUsage,
loop,
};
}
export { LOOP_BUDGET_MS, runNonStreamingProviderLeg };

23
open-sse/utils/requestLogger.ts
View File

@@ -19,6 +19,7 @@ export type RequestPipelinePayloads = {
providerResponse?: JsonRecord;
clientResponse?: JsonRecord;
error?: JsonRecord;
toolLoop?: { legs: JsonRecord[] };
streamChunks?: {
provider?: string[];
openai?: string[];
@@ -48,6 +49,7 @@ type RequestLogger = {
logConvertedResponse: (body: unknown) => void;
appendConvertedChunk: (chunk: string) => void;
logError: (error: unknown, requestBody?: unknown) => void;
logToolLoopReceipt: (receipt: unknown) => void;
getPipelinePayloads: () => RequestPipelinePayloads | null;
};
@@ -74,6 +76,7 @@ const MAX_LOG_STRING_LENGTH = 64 * 1024;
// existing plain-constant shape; CHAT_LOG_ARRAY_TAIL_ITEMS still overrides it.
export const MAX_LOG_ARRAY_ITEMS = getChatLogArrayTailItems();
const MAX_LOG_OBJECT_KEYS = 80;
const MAX_TOOL_LOOP_LEGS = 4;
function maskSensitiveHeaders(headers: HeaderInput): Record<string, unknown> {
if (!headers) return {};
@@ -282,7 +285,16 @@ function compactPipelinePayloads(
continue;
}
result[key as keyof RequestPipelinePayloads] = value;
if (key === "toolLoop" && value && typeof value === "object") {
const legs = (value as { legs?: unknown }).legs;
if (Array.isArray(legs) && legs.length > 0) {
result.toolLoop = { legs: legs as JsonRecord[] };
}
continue;
}
const payloadKey = key as Exclude<keyof RequestPipelinePayloads, "streamChunks" | "toolLoop">;
result[payloadKey] = value as JsonRecord;
}
return hasOwnValues(result) ? result : null;
@@ -384,6 +396,7 @@ export async function createRequestLogger(
logConvertedResponse() {},
appendConvertedChunk: chunkMethods.appendConvertedChunk,
logError() {},
logToolLoopReceipt() {},
getPipelinePayloads() {
return routeDecision ? { routeDecision } : null;
},
@@ -468,6 +481,14 @@ export async function createRequestLogger(
};
},
logToolLoopReceipt(receipt) {
const legs = payloads.toolLoop?.legs ?? [];
if (legs.length >= MAX_TOOL_LOOP_LEGS) return;
const cloned = cloneBoundedForLog(receipt);
if (!cloned || typeof cloned !== "object" || Array.isArray(cloned)) return;
payloads.toolLoop = { legs: [...legs, cloned as JsonRecord] };
},
getPipelinePayloads() {
return compactPipelinePayloads(payloads);
},

4
src/i18n/messages/en.json
View File

@@ -13006,6 +13006,10 @@
},
"SKILLS_SANDBOX_NETWORK_ENABLED": {
"description": "Enable network access in the skills sandbox."
},
"SERVER_OWNED_TOOL_LOOP_ENABLED": {
"label": "Server-Owned Tool Loop",
"description": "Continue non-streaming server-owned tool calls until the model returns a client-usable response."
}
}
},

4
src/i18n/messages/pt-BR.json
View File

@@ -13007,6 +13007,10 @@
},
"SKILLS_SANDBOX_NETWORK_ENABLED": {
"description": "Ativa o acesso à rede no sandbox de skills."
},
"SERVER_OWNED_TOOL_LOOP_ENABLED": {
"label": "Server-Owned Tool Loop",
"description": "Continue chamadas de ferramentas do servidor (server-owned) em não-streaming até que o modelo retorne uma resposta utilizável pelo cliente."
}
}
},

4
src/i18n/messages/vi.json
View File

@@ -13007,6 +13007,10 @@
},
"SKILLS_SANDBOX_NETWORK_ENABLED": {
"description": "Cho phép môi trường sandbox của kỹ năng truy cập mạng."
},
"SERVER_OWNED_TOOL_LOOP_ENABLED": {
"label": "Vòng lặp công cụ do máy chủ sở hữu",
"description": "Tiếp tục các lời gọi công cụ do máy chủ sở hữu ở chế độ không streaming cho đến khi mô hình trả về phản hồi mà máy khách dùng được."
}
}
},

25
src/lib/db/migrations/174_server_tool_executions.sql Normal file
View File

@@ -0,0 +1,25 @@
-- Migration 174: Durable server tool execution fence table.
-- Tracks claim/result state for server-owned tool calls to prevent duplicate execution
-- across retries and concurrent requests. Independent of skill_executions.
CREATE TABLE IF NOT EXISTS server_tool_executions (
id TEXT PRIMARY KEY,
api_key_id TEXT NOT NULL,
request_identity TEXT NOT NULL,
tool_call_id TEXT NOT NULL,
tool_name TEXT NOT NULL,
input_digest TEXT NOT NULL,
output TEXT,
status TEXT NOT NULL CHECK(status IN ('running', 'success', 'error', 'timeout')),
error_message TEXT,
duration_ms INTEGER,
claim_expires_at TEXT NOT NULL,
completed_at TEXT,
created_at TEXT NOT NULL DEFAULT (datetime('now')),
UNIQUE(api_key_id, request_identity, tool_call_id)
);
CREATE INDEX IF NOT EXISTS idx_server_tool_executions_status_expiry
ON server_tool_executions(status, claim_expires_at);
CREATE INDEX IF NOT EXISTS idx_server_tool_executions_created
ON server_tool_executions(created_at);

241
src/lib/db/skillExecutionFence.ts Normal file
View File

@@ -0,0 +1,241 @@
import { randomUUID } from "node:crypto";
import type { SqliteAdapter } from "./adapters/types";
import { getDbInstance } from "./core";
import {
sanitizeErrorMessage,
sanitizeUpstreamDetails,
} from "@omniroute/open-sse/utils/errorSanitization";
const MAX_PERSISTED_OUTPUT_CHARS = 32_768;
const MAX_PERSISTED_ERROR_CHARS = 4_096;
export type ServerToolClaim =
| { kind: "claimed"; executionId: string }
| {
kind: "replay";
executionId: string;
status: "success" | "error" | "timeout";
output: unknown;
errorMessage: string | null;
}
| { kind: "in_progress"; executionId: string }
| { kind: "unknown"; executionId: string }
| { kind: "identity_conflict"; executionId: string };
function isUniqueConstraintError(err: unknown): boolean {
if (!err || typeof err !== "object") return false;
const code = String((err as { code?: unknown }).code ?? "");
const msg = String((err as { message?: unknown }).message ?? "");
// Accept only extended UNIQUE/PRIMARYKEY codes — NOT generic SQLITE_CONSTRAINT
// (NOT NULL/CHECK constraints must NOT be swallowed)
if (/SQLITE_CONSTRAINT_PRIMARYKEY/i.test(code)) return true;
if (/SQLITE_CONSTRAINT_UNIQUE/i.test(code)) return true;
// Fallback: message must explicitly reference UNIQUE (not just CONSTRAINT)
if (/UNIQUE constraint failed/i.test(msg)) return true;
if (/UNIQUE constraint violation/i.test(msg)) return true;
return false;
}
function sanitizeAndBounded(value: unknown, maxChars: number): string | null {
if (value === null || value === undefined) return null;
// Strings: sanitize credentials/stack, store as bounded plain text (error_message is TEXT)
if (typeof value === "string") {
const sanitized = sanitizeErrorMessage(value);
return sanitized.length > maxChars ? sanitized.slice(0, maxChars) : sanitized;
}
// Objects/arrays: recursive sanitization, JSON.stringify, then guaranteed-valid truncation
const sanitized = sanitizeUpstreamDetails(value);
let text: string;
try {
text = JSON.stringify(sanitized);
} catch {
text = JSON.stringify({ error: "Value is not JSON-serializable" });
}
if (text.length <= maxChars) return text;
// Truncation envelope: progressively shorten preview until serialized envelope fits
const envelope = { truncated: true, preview: "" as string };
for (let len = maxChars - 40; len > 0; len -= 20) {
envelope.preview = text.slice(0, len);
const candidate = JSON.stringify(envelope);
if (candidate.length <= maxChars) return candidate;
}
envelope.preview = text.slice(0, 20);
return JSON.stringify(envelope);
}
export function readRow(
db: SqliteAdapter,
executionId: string
): {
id: string;
tool_name: string;
input_digest: string;
status: string;
output: string | null;
error_message: string | null;
} | null {
const row = db
.prepare(
"SELECT id, tool_name, input_digest, status, output, error_message FROM server_tool_executions WHERE id = ?"
)
.get(executionId) as
| {
id: string;
tool_name: string;
input_digest: string;
status: string;
output: string | null;
error_message: string | null;
}
| undefined;
return row ?? null;
}
function readExistingByIdentity(
db: SqliteAdapter,
apiKeyId: string,
requestIdentity: string,
toolCallId: string
): (ReturnType<typeof readRow> & { claim_expires_at: string }) | null {
const row = db
.prepare(
`SELECT id, tool_name, input_digest, status, output, error_message, claim_expires_at
FROM server_tool_executions
WHERE api_key_id = ? AND request_identity = ? AND tool_call_id = ?`
)
.get(apiKeyId, requestIdentity, toolCallId) as
| {
id: string;
tool_name: string;
input_digest: string;
status: string;
output: string | null;
error_message: string | null;
claim_expires_at: string;
}
| undefined;
return row ?? null;
}
function buildReplayClaim(row: {
id: string;
status: string;
output: string | null;
error_message: string | null;
}): ServerToolClaim {
let parsedOutput: unknown = null;
if (row.output !== null) {
try {
parsedOutput = JSON.parse(row.output);
} catch {
parsedOutput = row.output;
}
}
return {
kind: "replay",
executionId: row.id,
status: row.status as "success" | "error" | "timeout",
output: parsedOutput,
errorMessage: row.error_message,
};
}
export function claimServerToolExecution(
input: {
apiKeyId: string;
requestIdentity: string;
toolCallId: string;
toolName: string;
inputDigest: string;
leaseExpiresAt: string;
},
db: SqliteAdapter = getDbInstance(),
nowMs?: number
): ServerToolClaim {
const executionId = randomUUID();
const now = nowMs ?? Date.now();
// Try INSERT in a transaction
const tryInsert = db.transaction(() => {
db.prepare(
`INSERT INTO server_tool_executions
(id, api_key_id, request_identity, tool_call_id, tool_name, input_digest, status, claim_expires_at)
VALUES (?, ?, ?, ?, ?, ?, 'running', ?)`
).run(
executionId,
input.apiKeyId,
input.requestIdentity,
input.toolCallId,
input.toolName,
input.inputDigest,
input.leaseExpiresAt
);
});
try {
tryInsert();
return { kind: "claimed", executionId };
} catch (err: unknown) {
if (!isUniqueConstraintError(err)) {
throw err;
}
}
// UNIQUE conflict — re-read in a fresh transaction
const existing = db.transaction(() => {
return readExistingByIdentity(db, input.apiKeyId, input.requestIdentity, input.toolCallId);
})();
if (!existing) {
return { kind: "unknown", executionId };
}
// Identity conflict: different name or digest
if (existing.tool_name !== input.toolName || existing.input_digest !== input.inputDigest) {
return { kind: "identity_conflict", executionId: existing.id };
}
// Terminal status → replay
if (
existing.status === "success" ||
existing.status === "error" ||
existing.status === "timeout"
) {
return buildReplayClaim(existing);
}
// Running status — check STORED lease expiry (not input)
const storedExpiresAt = new Date(existing.claim_expires_at).getTime();
if (storedExpiresAt <= now) {
return { kind: "unknown", executionId: existing.id };
}
// Running + unexpired → in_progress
return { kind: "in_progress", executionId: existing.id };
}
export function finalizeServerToolExecution(
input: {
executionId: string;
status: "success" | "error" | "timeout";
output: unknown | null;
errorMessage: string | null;
durationMs: number;
},
db: SqliteAdapter = getDbInstance()
): boolean {
const safeOutput = sanitizeAndBounded(input.output, MAX_PERSISTED_OUTPUT_CHARS);
const safeError = sanitizeAndBounded(input.errorMessage, MAX_PERSISTED_ERROR_CHARS);
const result = db.transaction(() => {
return db
.prepare(
`UPDATE server_tool_executions
SET status = ?, output = ?, error_message = ?, duration_ms = ?, completed_at = datetime('now')
WHERE id = ? AND status = 'running'`
)
.run(input.status, safeOutput, safeError, input.durationMs, input.executionId);
})();
return result.changes > 0;
}

154
src/lib/skills/executor.ts
View File

@@ -218,6 +218,8 @@ class SkillExecutor {
throw new Error(`Skill not found: ${skillName}`);
}
// Check enabled/disabled BEFORE creating a DB row (preserves pre-Task-3
// behavior: disabled/missing skills never write to skill_executions).
if (!skill.enabled) {
throw new Error(`Skill is disabled: ${skillName}`);
}
@@ -242,41 +244,10 @@ class SkillExecutor {
new Date().toISOString()
);
let handler = this.handlers.get(skill.handler);
if (!handler) {
// Builtin handlers are registered by instrumentation-node at startup,
// but Next.js may compile this module into multiple chunks (each with
// its own SkillExecutor singleton). Fall back to the builtin registry
// so `POST /api/skills/executions` works regardless of which chunk the
// route is served from.
const builtin = builtinSkills[skill.handler];
if (builtin) {
this.handlers.set(skill.handler, builtin);
handler = builtin;
}
}
if (!handler) {
throw new Error(`Handler not found: ${skill.handler}`);
}
let output: Record<string, unknown> | null = null;
let errorMessage: string | null = null;
let status = SkillStatus.SUCCESS;
try {
const result = await this.executeWithTimeout(
handler(input, { apiKeyId: context.apiKeyId, sessionId: context.sessionId || "" })
);
const resultIsFailure = isSkillFailureOutput(result);
output = projectSkillOutputForBoundary(result);
if (resultIsFailure) {
errorMessage = skillFailureMessage(result);
status = SkillStatus.ERROR;
}
} catch (err) {
errorMessage = toSafeSkillErrorMessage(err);
status = SkillStatus.ERROR;
}
const { output, errorMessage, status } = await this.runHandler(skillName, input, {
apiKeyId: context.apiKeyId,
sessionId: context.sessionId || "",
});
const durationMs = Date.now() - startTime;
@@ -323,6 +294,119 @@ class SkillExecutor {
]);
}
/**
* Shared handler lookup + execute + output projection used by both
* `execute()` (which writes history) and `executeClaimed()` (which does not).
*/
private async runHandler(
skillName: string,
input: Record<string, unknown>,
context: { apiKeyId: string; sessionId: string }
): Promise<{
output: Record<string, unknown> | null;
errorMessage: string | null;
status: SkillStatus;
}> {
const skill = skillRegistry.getSkill(skillName, context.apiKeyId);
if (!skill) {
throw new Error(`Skill not found: ${skillName}`);
}
if (!skill.enabled) {
throw new Error(`Skill is disabled: ${skillName}`);
}
let handler = this.handlers.get(skill.handler);
if (!handler) {
const builtin = builtinSkills[skill.handler];
if (builtin) {
this.handlers.set(skill.handler, builtin);
handler = builtin;
}
}
if (!handler) {
throw new Error(`Handler not found: ${skill.handler}`);
}
let output: Record<string, unknown> | null = null;
let errorMessage: string | null = null;
let status = SkillStatus.SUCCESS;
try {
const result = await this.executeWithTimeout(
handler(input, { apiKeyId: context.apiKeyId, sessionId: context.sessionId || "" })
);
const resultIsFailure = isSkillFailureOutput(result);
output = projectSkillOutputForBoundary(result);
if (resultIsFailure) {
errorMessage = skillFailureMessage(result);
status = SkillStatus.ERROR;
}
} catch (err) {
errorMessage = toSafeSkillErrorMessage(err);
status = SkillStatus.ERROR;
}
return { output, errorMessage, status };
}
/**
* Execute a claimed skill call for the server-owned tool loop.
* Reuses the same handler lookup/timeout/projection as `execute()`
* but does NOT write to `skill_executions` — the fence table owns
* persistence for claimed executions.
*/
async executeClaimed(
skillName: string,
input: Record<string, unknown>,
context: { apiKeyId: string; sessionId: string },
executionId: string
): Promise<SkillExecution> {
const settings = await getSettings();
if (settings.skillsEnabled === false) {
throw new Error("Skills execution is disabled. Enable Skills in Settings > AI.");
}
const skill = skillRegistry.getSkill(skillName, context.apiKeyId);
if (!skill) {
throw new Error(`Skill not found: ${skillName}`);
}
const startTime = Date.now();
log.info("skills.executor.claimed_start", {
skillId: skill.id,
skillName,
apiKeyId: context.apiKeyId,
executionId,
});
const { output, errorMessage, status } = await this.runHandler(skillName, input, context);
const durationMs = Date.now() - startTime;
if (status !== SkillStatus.SUCCESS) {
throw new Error(`Skill execution failed: ${errorMessage ?? "unknown error"}`);
}
log.info("skills.executor.claimed_complete", {
skillId: skill.id,
success: status === SkillStatus.SUCCESS,
durationMs,
executionId,
});
return {
id: executionId,
skillId: skill.id,
apiKeyId: context.apiKeyId,
sessionId: context.sessionId || "",
input,
output,
status,
errorMessage,
durationMs,
createdAt: new Date(),
};
}
getExecution(executionId: string): SkillExecution | undefined {
const db = getDbInstance();
const row = db.prepare("SELECT * FROM skill_executions WHERE id = ?").get(executionId) as any;

412
src/lib/skills/followUpTranscript.ts Normal file
View File

@@ -0,0 +1,412 @@
import type {
BoundedToolResult,
BuildFollowUpTranscriptInput,
ExecutedToolResult,
ToolCall,
} from "./toolLoopTypes";
/**
* Pure transcript builder for the server-owned tool loop (spec §5.2).
*
* `serializeBoundedToolResult` serializes one executed tool result to JSON text
* and, when it exceeds a UTF-8 byte budget, truncates it at code-point
* boundaries with a `[TRUNCATED N BYTES BY OMNIROUTE]` marker whose own bytes
* count against the budget.
*
* `buildFollowUpSourceBody` appends the assistant tool-call turn and the
* bounded tool results to the source-format `messages` array. It never mutates
* its inputs, rejects orphan/mixed calls before building anything, and consumes
* the total output budget per-tool in result order.
*/
export const MAX_RESULT_BYTES_PER_TOOL = 32_768;
export const MAX_RESULT_BYTES_TOTAL = 65_536;
const NON_SERIALIZABLE_ERROR = "Tool result is not JSON-serializable";
function assertValidBudget(name: string, value: number): void {
if (!Number.isFinite(value) || !Number.isInteger(value) || value < 0) {
throw new RangeError(`${name} must be a non-negative finite integer, got ${value}`);
}
}
function markerFor(droppedBytes: number): string {
return `[TRUNCATED ${droppedBytes} BYTES BY OMNIROUTE]`;
}
/**
* Returns the longest code-point-aligned prefix of `text` whose UTF-8 byte
* length does not exceed `maxBytes`. Iterating `for...of` over a string yields
* full code points, so surrogate pairs (astral CJK, emoji) are never split and
* the result is always valid UTF-8.
*/
function truncateToCodePointBoundary(text: string, maxBytes: number): string {
let out = "";
let bytes = 0;
for (const codePoint of text) {
const codePointBytes = Buffer.byteLength(codePoint, "utf8");
if (bytes + codePointBytes > maxBytes) break;
out += codePoint;
bytes += codePointBytes;
}
return out;
}
function projectSerializable(value: unknown): unknown {
if (value === undefined) return null;
if (value instanceof Error) return { error: value.message };
if (typeof value === "bigint") return value.toString();
return value;
}
export function serializeBoundedToolResult(value: unknown, maxBytes: number): BoundedToolResult {
assertValidBudget("maxBytes", maxBytes);
const projected = projectSerializable(value);
let serialized: string | undefined;
try {
const raw = JSON.stringify(projected);
serialized = typeof raw === "string" ? raw : undefined;
} catch {
serialized = undefined;
}
// Top-level function/symbol (JSON.stringify returns undefined, not a string)
// and serialization exceptions (cycles, nested BigInt, exotic objects) all
// resolve to the fixed non-serializable error shape.
if (serialized === undefined) {
serialized = JSON.stringify({ error: NON_SERIALIZABLE_ERROR });
}
const originalBytes = Buffer.byteLength(serialized, "utf8");
if (originalBytes <= maxBytes) {
return { text: serialized, truncated: false, originalBytes };
}
const fullMarker = markerFor(originalBytes);
const fullMarkerBytes = Buffer.byteLength(fullMarker, "utf8");
// The full marker alone does not fit: return its code-point-safe UTF-8 prefix
// (maxBytes 0 yields the empty string).
if (maxBytes <= fullMarkerBytes) {
return {
text: truncateToCodePointBoundary(fullMarker, maxBytes),
truncated: true,
originalBytes,
};
}
// Reserve marker space first, then take the longest valid prefix of the text.
// `dropped <= originalBytes` so markerFor(dropped) is never longer than the
// reserved marker; the total therefore stays within maxBytes.
const prefixBudget = maxBytes - fullMarkerBytes;
const prefix = truncateToCodePointBoundary(serialized, prefixBudget);
const droppedBytes = originalBytes - Buffer.byteLength(prefix, "utf8");
return {
text: prefix + markerFor(droppedBytes),
truncated: true,
originalBytes,
};
}
function safeStringifyArguments(value: Record<string, unknown>): string {
try {
const raw = JSON.stringify(value);
return typeof raw === "string" ? raw : "{}";
} catch {
return "{}";
}
}
function validateCallsAndResults(toolCalls: ToolCall[], results: ExecutedToolResult[]): void {
if (toolCalls.length !== results.length) {
throw new Error(
`buildFollowUpSourceBody requires toolCalls (${toolCalls.length}) and results (${results.length}) to have the same length`
);
}
const callIds = toolCalls.map((call) => call.id);
const resultIds = results.map((result) => result.id);
if (new Set(callIds).size !== callIds.length) {
throw new Error("buildFollowUpSourceBody requires unique tool call ids");
}
if (new Set(resultIds).size !== resultIds.length) {
throw new Error("buildFollowUpSourceBody requires unique tool result ids");
}
const resultIdSet = new Set(resultIds);
for (const id of callIds) {
if (!resultIdSet.has(id)) {
throw new Error(
`buildFollowUpSourceBody requires every tool call to have a matching result (missing: ${id})`
);
}
}
for (const result of results) {
const call = toolCalls.find((c) => c.id === result.id);
if (call && call.name !== result.name) {
throw new Error(
`buildFollowUpSourceBody: result name "${result.name}" for id "${result.id}" does not match tool call name "${call.name}"`
);
}
}
}
function extractOpenAIMessage(response: Record<string, unknown>): Record<string, unknown> | null {
const choice = Array.isArray(response.choices) ? (response.choices[0] as unknown) : null;
if (choice && typeof choice === "object") {
const message = (choice as Record<string, unknown>).message;
if (message && typeof message === "object" && !Array.isArray(message)) {
return message as Record<string, unknown>;
}
}
if (
response.message &&
typeof response.message === "object" &&
!Array.isArray(response.message)
) {
return response.message as Record<string, unknown>;
}
return null;
}
/**
* Returns the assistant tool_calls exactly as the previous response carried
* them (provenance preserved), restricted to calls with a matching result.
* Falls back to reconstructing the OpenAI wire shape from the parsed
* `toolCalls` when the response has no tool_calls of its own.
*
* When the previous response does carry tool_calls, validates that each
* call ID appears exactly once with the correct name (fails closed on
* partial, duplicate, or mismatched entries).
*/
function resolveOpenAIAssistantToolCalls(
previousResponse: Record<string, unknown>,
toolCalls: ToolCall[],
matchedIds: Set<string>
): unknown[] {
let originalToolCalls: unknown[] = [];
const prevMessage = extractOpenAIMessage(previousResponse);
if (prevMessage && Array.isArray(prevMessage.tool_calls)) {
originalToolCalls = prevMessage.tool_calls as unknown[];
}
if (originalToolCalls.length === 0 && toolCalls.length > 0) {
originalToolCalls = toolCalls.map((call) => ({
id: call.id,
type: "function",
function: { name: call.name, arguments: safeStringifyArguments(call.arguments) },
}));
}
// Validate when previous response carries its own tool_calls.
const hasOwnToolCalls =
prevMessage && Array.isArray(prevMessage.tool_calls) && prevMessage.tool_calls.length > 0;
if (hasOwnToolCalls) {
const idCounts = new Map<string, number>();
const idNames = new Map<string, string>();
for (const raw of originalToolCalls) {
if (!raw || typeof raw !== "object") continue;
const rec = raw as Record<string, unknown>;
const id = typeof rec.id === "string" ? rec.id : undefined;
if (id === undefined) continue;
idCounts.set(id, (idCounts.get(id) ?? 0) + 1);
if (idNames.has(id)) continue;
const fn = rec.function;
if (
fn &&
typeof fn === "object" &&
typeof (fn as Record<string, unknown>).name === "string"
) {
idNames.set(id, (fn as Record<string, unknown>).name as string);
}
}
for (const call of toolCalls) {
if (!matchedIds.has(call.id)) continue;
const count = idCounts.get(call.id) ?? 0;
if (count !== 1) {
throw new Error(
`buildFollowUpSourceBody: previous response tool_calls must contain exactly one match per call ID (id "${call.id}" has ${count})`
);
}
const prevName = idNames.get(call.id);
if (prevName !== undefined && prevName !== call.name) {
throw new Error(
`buildFollowUpSourceBody: previous response tool_call "${call.id}" name "${prevName}" does not match tool call name "${call.name}"`
);
}
}
}
return originalToolCalls.filter((call) => {
if (!call || typeof call !== "object") return false;
const record = call as Record<string, unknown>;
const id = typeof record.id === "string" ? record.id : record.call_id;
return typeof id === "string" && matchedIds.has(id);
});
}
/**
* Resolves tool_use blocks from the Claude previous response. When content
* blocks are present, validates that each call ID appears exactly once with
* the correct name (fails closed on partial, duplicate, or mismatched).
* Non-tool content blocks (text, thinking, reasoning) are preserved in their
* original order; only tool_use blocks are filtered to matched IDs.
*/
function resolveClaudeToolUseBlocks(
previousResponse: Record<string, unknown>,
toolCalls: ToolCall[],
matchedIds: Set<string>
): unknown[] {
if (Array.isArray(previousResponse.content)) {
const blocks = previousResponse.content as unknown[];
// Validate when content has tool_use blocks for matched IDs.
const idCounts = new Map<string, number>();
const idNames = new Map<string, string>();
for (const block of blocks) {
if (!block || typeof block !== "object") continue;
const rec = block as Record<string, unknown>;
if (rec.type !== "tool_use" || typeof rec.id !== "string") continue;
if (!matchedIds.has(rec.id)) continue;
idCounts.set(rec.id, (idCounts.get(rec.id) ?? 0) + 1);
if (idNames.has(rec.id)) continue;
if (typeof rec.name === "string") {
idNames.set(rec.id, rec.name);
}
}
for (const call of toolCalls) {
if (!matchedIds.has(call.id)) continue;
const count = idCounts.get(call.id) ?? 0;
if (count !== 1) {
throw new Error(
`buildFollowUpSourceBody: previous response content must contain exactly one tool_use match per call ID (id "${call.id}" has ${count})`
);
}
const prevName = idNames.get(call.id);
if (prevName !== undefined && prevName !== call.name) {
throw new Error(
`buildFollowUpSourceBody: previous response tool_use "${call.id}" name "${prevName}" does not match tool call name "${call.name}"`
);
}
}
// Match prepareClaudeRequest: keep all thinking/signature blocks, keep
// ordinary content only before the first tool_use, and retain only the
// tool_use blocks whose results are being replayed.
let foundToolUse = false;
const replayBlocks: unknown[] = [];
for (const block of blocks) {
if (!block || typeof block !== "object") continue;
const rec = block as Record<string, unknown>;
if (rec.type === "tool_use") {
foundToolUse = true;
if (typeof rec.id === "string" && matchedIds.has(rec.id)) replayBlocks.push(block);
continue;
}
if (rec.type === "thinking" || rec.type === "redacted_thinking" || !foundToolUse) {
replayBlocks.push(block);
}
}
return replayBlocks;
}
return toolCalls.map((call) => ({
type: "tool_use",
id: call.id,
name: call.name,
input: call.arguments,
}));
}
export function buildFollowUpSourceBody(
input: BuildFollowUpTranscriptInput
): Record<string, unknown> {
const { sourceBody, previousResponse, toolCalls, results, sourceFormat } = input;
const maxResultBytes = input.maxResultBytes ?? MAX_RESULT_BYTES_PER_TOOL;
const maxTotalResultBytes = input.maxTotalResultBytes ?? MAX_RESULT_BYTES_TOTAL;
if (sourceFormat !== "openai" && sourceFormat !== "claude") {
throw new Error('sourceFormat must be "openai" or "claude"');
}
assertValidBudget("maxResultBytes", maxResultBytes);
assertValidBudget("maxTotalResultBytes", maxTotalResultBytes);
if (!Array.isArray(sourceBody.messages)) {
throw new Error("buildFollowUpSourceBody requires sourceBody.messages array");
}
validateCallsAndResults(toolCalls, results);
// When a pre-serialized map is provided (from the loop's own budget pass),
// use its text verbatim instead of re-serializing. Otherwise serialize with
// the standard budget logic.
const serializedResultTextById = input.serializedResultTextById;
let boundedResults: BoundedToolResult[];
if (serializedResultTextById) {
boundedResults = results.map((result) => {
const text = serializedResultTextById.get(result.id) ?? "";
return { text, truncated: false, originalBytes: Buffer.byteLength(text, "utf8") };
});
} else {
let remainingBytes = maxTotalResultBytes;
boundedResults = results.map((result) => {
const itemMaxBytes = Math.min(maxResultBytes, remainingBytes);
const bounded = serializeBoundedToolResult(result.result, itemMaxBytes);
remainingBytes -= Buffer.byteLength(bounded.text, "utf8");
return bounded;
});
}
const matchedIds = new Set(results.map((result) => result.id));
const messages = [...(sourceBody.messages as unknown[])];
if (sourceFormat === "openai") {
const prevMessage = extractOpenAIMessage(previousResponse);
const previousContent = prevMessage && "content" in prevMessage ? prevMessage.content : null;
const assistantMessage: Record<string, unknown> = {
role: "assistant",
content: previousContent ?? null,
};
const assistantToolCalls = resolveOpenAIAssistantToolCalls(
previousResponse,
toolCalls,
matchedIds
);
if (assistantToolCalls.length > 0) {
assistantMessage.tool_calls = assistantToolCalls;
}
messages.push(assistantMessage);
boundedResults.forEach((bounded, index) => {
messages.push({
role: "tool",
tool_call_id: results[index].id,
content: bounded.text,
});
});
return { ...sourceBody, messages, stream: false };
}
// Claude Messages: the original assistant tool_use turn, then a separate
// user tool_result message. A tool_result must never share the assistant
// content — Anthropic rejects it (openai-to-claude.ts:323).
const toolUseBlocks = resolveClaudeToolUseBlocks(previousResponse, toolCalls, matchedIds);
messages.push({ role: "assistant", content: toolUseBlocks });
messages.push({
role: "user",
content: boundedResults.map((bounded, index) => ({
type: "tool_result",
tool_use_id: results[index].id,
content: bounded.text,
})),
});
return { ...sourceBody, messages };
}

49
src/lib/skills/injection.ts
View File

@@ -253,6 +253,28 @@ function scoreAutoSkill(
}
export function injectSkills(options: InjectionOptions): unknown[] {
return injectSkillsWithMetadata(options).tools;
}
export interface InjectSkillsWithMetadataResult {
tools: unknown[];
injectedNames: string[];
}
function getToolNameFromDef(tool: unknown): string {
if (!tool || typeof tool !== "object") return "";
const r = tool as Record<string, unknown>;
if (typeof r.name === "string") return r.name;
if (r.function && typeof r.function === "object") {
const fn = r.function as Record<string, unknown>;
if (typeof fn.name === "string") return fn.name;
}
return "";
}
export function injectSkillsWithMetadata(
options: InjectionOptions
): InjectSkillsWithMetadataResult {
const contextText = buildContextText(options);
const contextTokens = extractTokens(contextText);
const backgroundTokens = extractTokens(toLowerText(options.backgroundReason));
@@ -295,7 +317,7 @@ export function injectSkills(options: InjectionOptions): unknown[] {
apiKeyId: options.apiKeyId,
reason: "no_enabled_skills",
});
return options.existingTools || [];
return { tools: options.existingTools || [], injectedNames: [] };
}
log.info("skills.injection.injected", {
@@ -317,11 +339,30 @@ export function injectSkills(options: InjectionOptions): unknown[] {
}
});
if (options.existingTools && options.existingTools.length > 0) {
return [...injectedTools, ...options.existingTools];
// Compute the set of existing tool names to exclude client collisions.
const existingToolNames = new Set(
(options.existingTools || []).map((t) => getToolNameFromDef(t)).filter(Boolean)
);
// Filter out skills whose encoded name collides with a client-declared tool.
const nonCollidingTools = injectedTools.filter((tool) => {
const name = getToolNameFromDef(tool);
return name && !existingToolNames.has(name);
});
const injectedNames: string[] = [];
for (const tool of nonCollidingTools) {
const name = getToolNameFromDef(tool);
if (name) {
injectedNames.push(name);
}
}
return injectedTools;
if (options.existingTools && options.existingTools.length > 0) {
return { tools: [...nonCollidingTools, ...options.existingTools], injectedNames };
}
return { tools: nonCollidingTools, injectedNames };
}
export function injectSkillTools(

399
src/lib/skills/interception.ts
View File

@@ -6,10 +6,29 @@ import { detectProvider, decodeSkillToolName } from "./injection";
import { OMNIROUTE_WEB_SEARCH_FALLBACK_TOOL_NAME } from "@omniroute/open-sse/services/webSearchFallback.ts";
import { OMNIROUTE_WEB_FETCH_FALLBACK_TOOL_NAME } from "@omniroute/open-sse/services/webFetchInterception.ts";
import { sanitizeErrorMessage } from "@omniroute/open-sse/utils/errorSanitization.ts";
import { runWithServerToolFence } from "./toolExecutionFence";
import type { ExecutedToolResult, ToolCall, ExecutionContext } from "./toolLoopTypes";
import { logger } from "../../../open-sse/utils/logger.ts";
const log = logger("SKILLS_INTERCEPTION");
/**
* Typed error for server-owned tool execution control-flow states
* (in_progress, unknown, identity_conflict). These must never be fed
* back to the model as tool results — they represent infrastructure
* conditions that should surface as HTTP-level errors.
*/
export class ServerOwnedExecutionError extends Error {
readonly code: string;
readonly httpStatus: number;
constructor(message: string, code: string, httpStatus: number) {
super(message);
this.name = "ServerOwnedExecutionError";
this.code = code;
this.httpStatus = httpStatus;
}
}
function toSafeSkillErrorMessage(value: unknown): string {
try {
const raw = value instanceof Error ? value.message : value;
@@ -24,24 +43,7 @@ function projectSkillResultForPublicResponse(result: unknown): unknown {
return projectSkillOutputForBoundary(result as Record<string, unknown>);
}
interface ToolCall {
id: string;
name: string;
arguments: Record<string, unknown>;
}
interface ExecutionContext {
apiKeyId: string;
sessionId: string;
requestId: string;
builtinToolNames?: string[];
customSkillExecutionEnabled?: boolean;
// #7339: threaded through to the web_fetch builtin so it can resolve a per-model
// pinned fetch backend (interceptionRules.fetchBackend). Optional — every other
// builtin/skill ignores these.
provider?: string;
model?: string;
}
// ToolCall and ExecutionContext types are imported from ./toolLoopTypes.ts
const BUILTIN_TOOL_ALIASES: Record<string, string> = {
[OMNIROUTE_WEB_SEARCH_FALLBACK_TOOL_NAME]: "web_search",
@@ -200,8 +202,19 @@ export async function interceptToolCalls(
return results;
}
export function extractToolCalls(response: any, modelId: string): ToolCall[] {
const provider = detectProvider(modelId);
export function extractToolCalls(response: any, modelIdOrSourceFormat: string): ToolCall[] {
// Accept either a sourceFormat ("openai" | "claude") or a model ID string.
// Map known sourceFormat values; fall back to detectProvider for model IDs.
const format =
modelIdOrSourceFormat === "openai" || modelIdOrSourceFormat === "claude"
? modelIdOrSourceFormat
: undefined;
const provider =
format === "claude"
? "anthropic"
: format === "openai"
? "openai"
: detectProvider(modelIdOrSourceFormat);
switch (provider) {
case "openai": {
@@ -446,3 +459,349 @@ export async function handleToolCallExecution(
return response;
}
}
// ─── Task 3: ownership classifier ────────────────────────────────────────────
export async function classifyServerOwnedCalls(
toolCalls: ToolCall[],
context: ExecutionContext
): Promise<{ serverOwned: ToolCall[]; clientNative: ToolCall[] }> {
const builtinSet = new Set(context.builtinToolNames || []);
const customSet = new Set(context.injectedCustomSkillNames || []);
const serverOwned: ToolCall[] = [];
const clientNative: ToolCall[] = [];
for (const call of toolCalls) {
if (builtinSet.has(call.name)) {
serverOwned.push(call);
} else if (context.customSkillExecutionEnabled && customSet.has(call.name)) {
serverOwned.push(call);
} else {
clientNative.push(call);
}
}
return { serverOwned, clientNative };
}
// ─── Task 3: executeServerOwned — fence-gated execution callback ────────────
const LEASE_DURATION_MS = 120_000;
/**
* Runtime seam: overridable fence function for testing.
* When null, the real `runWithServerToolFence` is used.
* Tests may set this to inject controlled fence outcomes.
*/
let _fenceFn: typeof runWithServerToolFence | null = null;
export function setFenceFnForTesting(fn: typeof runWithServerToolFence | null): void {
_fenceFn = fn;
}
export async function executeServerOwned(
calls: ToolCall[],
context: ExecutionContext,
fenceFn?: typeof runWithServerToolFence
): Promise<ExecutedToolResult[]> {
if (context.executionFenceEnabled && !context.requestIdentity) {
throw new Error(
"executeServerOwned requires context.requestIdentity when executionFenceEnabled is true"
);
}
const results: ExecutedToolResult[] = [];
for (const call of calls) {
const builtinHandlerName = resolveBuiltinHandlerName(call.name, context);
const isMemoryBuiltin = builtinHandlerName && MEMORY_TOOL_NAMES.has(builtinHandlerName);
const isOrdinaryBuiltin = builtinHandlerName && builtinHandlerName in builtinSkills;
const isCustomSkill =
!builtinHandlerName &&
context.customSkillExecutionEnabled &&
context.injectedCustomSkillNames?.includes(call.name);
const executeFn = async (executionId: string): Promise<unknown> => {
if (isMemoryBuiltin) {
const handlerName = builtinHandlerName as keyof typeof memoryBuiltinHandlers;
return memoryBuiltinHandlers[handlerName](call.arguments, {
apiKeyId: context.apiKeyId,
sessionId: context.sessionId,
});
}
if (isOrdinaryBuiltin) {
const handlerName = builtinHandlerName as keyof typeof builtinSkills;
return builtinSkills[handlerName](call.arguments, {
apiKeyId: context.apiKeyId,
sessionId: context.sessionId,
provider: context.provider,
model: context.model,
});
}
if (isCustomSkill) {
const decodedName = decodeSkillToolName(call.name);
const [name, version] = decodedName.includes("@")
? decodedName.split("@", 2)
: [decodedName, "latest"];
const skillName = version === "latest" ? name : `${name}@${version}`;
const execution = await skillExecutor.executeClaimed(
skillName,
call.arguments,
{
apiKeyId: context.apiKeyId,
sessionId: context.sessionId,
},
executionId
);
return (
execution.output ??
(execution.errorMessage
? { error: toSafeSkillErrorMessage(execution.errorMessage) }
: { error: "Skill execution returned no output" })
);
}
throw new Error(`No handler for tool: ${call.name}`);
};
if (context.executionFenceEnabled && context.requestIdentity) {
const activeFenceFn = fenceFn ?? _fenceFn ?? runWithServerToolFence;
const fenceResult = await activeFenceFn({
apiKeyId: context.apiKeyId,
requestIdentity: context.requestIdentity,
toolCallId: call.id,
toolName: call.name,
arguments: call.arguments,
leaseDurationMs: LEASE_DURATION_MS,
execute: executeFn,
});
switch (fenceResult.kind) {
case "executed":
results.push({
id: call.id,
name: call.name,
result: projectSkillResultForPublicResponse(fenceResult.value),
replayed: false,
});
break;
case "replayed": {
if (fenceResult.status === "error") {
throw new ServerOwnedExecutionError(
fenceResult.errorMessage ?? "Tool execution failed",
"TOOL_EXECUTION_ERROR",
500
);
}
if (fenceResult.status === "timeout") {
throw new ServerOwnedExecutionError(
fenceResult.errorMessage ?? "Tool execution timed out",
"TOOL_EXECUTION_TIMEOUT",
504
);
}
results.push({
id: call.id,
name: call.name,
result: projectSkillResultForPublicResponse(fenceResult.value),
replayed: true,
});
break;
}
case "in_progress":
throw new ServerOwnedExecutionError(
"Tool execution in progress",
"TOOL_IN_PROGRESS",
409
);
case "unknown":
throw new ServerOwnedExecutionError(
"Tool execution state unknown",
"TOOL_STATE_UNKNOWN",
500
);
case "identity_conflict":
throw new ServerOwnedExecutionError(
"Tool execution identity conflict",
"IDENTITY_CONFLICT",
409
);
}
} else {
// Flag-off path: no fence, dispatch directly.
try {
const value = await executeFn("");
results.push({
id: call.id,
name: call.name,
result: projectSkillResultForPublicResponse(value),
replayed: false,
});
} catch (err) {
results.push({
id: call.id,
name: call.name,
result: { error: toSafeSkillErrorMessage(err) },
replayed: false,
});
}
}
}
return results;
}
// ─── Task 3: pure escape formatter ───────────────────────────────────────────
function extractOpenAIToolCalls(
response: Record<string, unknown>
): Array<{ id: string; function: { name: string; arguments: string } }> {
const rootToolCalls = Array.isArray(response.tool_calls) ? response.tool_calls : [];
const choiceToolCalls = Array.isArray(response.choices)
? (response.choices as any[]).flatMap((choice: any) =>
Array.isArray(choice?.message?.tool_calls) ? choice.message.tool_calls : []
)
: [];
return rootToolCalls.length > 0 ? rootToolCalls : choiceToolCalls;
}
function getOpenAIResponseOutput(
response: Record<string, unknown>
): { target: Record<string, unknown>; output: unknown[] } | null {
if (Array.isArray(response.output)) {
return { target: response, output: response.output };
}
if (
response.response &&
typeof response.response === "object" &&
!Array.isArray(response.response) &&
Array.isArray((response.response as Record<string, unknown>).output)
) {
return {
target: response.response as Record<string, unknown>,
output: (response.response as Record<string, unknown>).output as unknown[],
};
}
return null;
}
export function formatEscapeHatchResponse(
response: Record<string, unknown>,
serverCalls: ToolCall[],
results: ExecutedToolResult[],
clientCalls: ToolCall[],
sourceFormat: "openai" | "claude",
serializedResultTextById?: Map<string, string>
): Record<string, unknown> {
const serverIds = new Set(serverCalls.map((c) => c.id));
if (sourceFormat === "openai") {
// Check for Responses API format.
const responsesOutput = getOpenAIResponseOutput(response);
if (responsesOutput) {
// For Responses, append function_call_output for server calls.
const functionOutputs = results
.filter((r) => serverIds.has(r.id))
.map((r) => ({
type: "function_call_output",
call_id: r.id,
output: serializedResultTextById?.get(r.id) ?? JSON.stringify(r.result),
}));
return {
...response,
response:
responsesOutput.target !== response
? {
...(response.response as Record<string, unknown>),
output: [...responsesOutput.output, ...functionOutputs],
}
: undefined,
output:
responsesOutput.target === response
? [...responsesOutput.output, ...functionOutputs]
: response.output,
};
}
// Chat Completions format.
const originalToolCalls = extractOpenAIToolCalls(response);
const remainingToolCalls = originalToolCalls.filter(
(tc: any) => !serverIds.has(tc.id || tc.call_id)
);
// Build result text from server results.
const resultTexts = results
.filter((r) => serverIds.has(r.id))
.map(
(r) =>
`[${r.name} result]\n${serializedResultTextById?.get(r.id) ?? JSON.stringify(r.result)}`
)
.join("\n\n");
const existingContent =
typeof response.choices?.[0]?.message?.content === "string"
? response.choices[0].message.content
: "";
const newContent = existingContent ? `${existingContent}\n\n${resultTexts}` : resultTexts;
// Clone response to avoid mutation.
const formatted = JSON.parse(JSON.stringify(response));
if (formatted.choices?.[0]?.message) {
formatted.choices[0].message.content = newContent;
formatted.choices[0].message.tool_calls =
remainingToolCalls.length > 0 ? remainingToolCalls : undefined;
}
// Mixed → keep tool_calls finish_reason; all-server → stop.
if (remainingToolCalls.length === 0 && clientCalls.length === 0) {
formatted.choices[0].finish_reason = "stop";
}
return formatted;
}
if (sourceFormat === "claude") {
const remainingContent = (Array.isArray(response.content) ? response.content : []).filter(
(block: any) => !(block?.type === "tool_use" && serverIds.has(block.id))
);
// Build result text blocks.
const resultTextBlocks = results
.filter((r) => serverIds.has(r.id))
.map((r) => ({
type: "text",
text: `[${r.name} result]\n${serializedResultTextById?.get(r.id) ?? JSON.stringify(r.result)}`,
}));
// Insert result text blocks before the first remaining tool_use.
const firstRemainingIndex = remainingContent.findIndex(
(block: any) => block?.type === "tool_use"
);
let newContent: unknown[];
if (firstRemainingIndex === -1) {
newContent = [...remainingContent, ...resultTextBlocks];
} else {
newContent = [
...remainingContent.slice(0, firstRemainingIndex),
...resultTextBlocks,
...remainingContent.slice(firstRemainingIndex),
];
}
const formatted: Record<string, unknown> = { ...response, content: newContent };
// All-server → end_turn; mixed → keep original stop_reason.
const remainingToolUseCount = remainingContent.filter(
(b: any) => b?.type === "tool_use"
).length;
if (remainingToolUseCount === 0 && clientCalls.length === 0) {
formatted.stop_reason = "end_turn";
formatted.stop_sequence = null;
}
return formatted;
}
return response;
}

466
src/lib/skills/serverOwnedToolLoop.ts Normal file
View File

@@ -0,0 +1,466 @@
/**
* Server-owned tool loop state machine (spec §5.5).
*
* Handles the complete lifecycle:
* 1. Classify ownership of tool calls
* 2. Execute server-owned tools
* 3. Serialize results within UTF-8 byte budgets
* 4. Build accumulated transcript and resume upstream
* 5. Terminate on any boundary condition
*/
import type {
ServerOwnedToolLoopOptions,
ServerOwnedToolLoopResult,
NonStreamingProviderLegResult,
ProviderLegUsage,
ProviderLegReceipt,
ChatCoreErrorResult,
ExecutedToolResult,
} from "./toolLoopTypes.ts";
import {
classifyServerOwnedCalls,
extractToolCalls,
formatEscapeHatchResponse,
ServerOwnedExecutionError,
} from "./interception.ts";
import {
buildFollowUpSourceBody,
serializeBoundedToolResult,
MAX_RESULT_BYTES_PER_TOOL,
MAX_RESULT_BYTES_TOTAL,
} from "./followUpTranscript.ts";
import { createErrorResult } from "@omniroute/open-sse/utils/error.ts";
// ─── Constants ────────────────────────────────────────────────────────────────
export const MAX_FOLLOW_UPS = 3;
export const LOOP_BUDGET_MS = 120_000;
export const MIN_REMAINING_FOR_FOLLOW_UP_MS = 10_000;
// ─── Usage aggregation ────────────────────────────────────────────────────────
export function aggregateProviderLegUsage(
usages: Array<ProviderLegUsage | null>
): ProviderLegUsage {
let prompt_tokens = 0;
let completion_tokens = 0;
let cached_tokens: number | undefined;
let cache_read_input_tokens: number | undefined;
let cache_creation_input_tokens: number | undefined;
let reasoning_tokens: number | undefined;
let cost_in_usd_ticks: number | undefined;
for (const u of usages) {
if (!u) continue;
prompt_tokens += u.prompt_tokens;
completion_tokens += u.completion_tokens;
if (u.cached_tokens !== undefined) {
cached_tokens = (cached_tokens ?? 0) + u.cached_tokens;
}
if (u.cache_read_input_tokens !== undefined) {
cache_read_input_tokens = (cache_read_input_tokens ?? 0) + u.cache_read_input_tokens;
}
if (u.cache_creation_input_tokens !== undefined) {
cache_creation_input_tokens =
(cache_creation_input_tokens ?? 0) + u.cache_creation_input_tokens;
}
if (u.reasoning_tokens !== undefined) {
reasoning_tokens = (reasoning_tokens ?? 0) + u.reasoning_tokens;
}
if (u.cost_in_usd_ticks !== undefined) {
cost_in_usd_ticks = (cost_in_usd_ticks ?? 0) + u.cost_in_usd_ticks;
}
}
return {
prompt_tokens,
completion_tokens,
total_tokens: prompt_tokens + completion_tokens,
...(cached_tokens !== undefined ? { cached_tokens } : {}),
...(cache_read_input_tokens !== undefined ? { cache_read_input_tokens } : {}),
...(cache_creation_input_tokens !== undefined ? { cache_creation_input_tokens } : {}),
...(reasoning_tokens !== undefined ? { reasoning_tokens } : {}),
...(cost_in_usd_ticks !== undefined ? { cost_in_usd_ticks } : {}),
};
}
function aggregateUsageOrNull(usages: Array<ProviderLegUsage | null>): ProviderLegUsage | null {
const hasNonNull = usages.some((u) => u !== null);
if (!hasNonNull) return null;
return aggregateProviderLegUsage(usages);
}
// ─── Execution error → errorResult mapping ────────────────────────────────────
function mapExecutionError(err: ServerOwnedExecutionError): ChatCoreErrorResult {
return createErrorResult(
err.httpStatus,
err.message,
null,
err.code,
"server_tool_execution_error"
) as unknown as ChatCoreErrorResult;
}
// ─── State machine ────────────────────────────────────────────────────────────
export async function runServerOwnedToolLoop(
options: ServerOwnedToolLoopOptions
): Promise<ServerOwnedToolLoopResult> {
const now = options.now ?? performance.now.bind(performance);
const loopStartedAtMs = now();
const loopDeadlineAtMs = Math.min(options.deadlineAtMs, loopStartedAtMs + LOOP_BUDGET_MS);
const maxFollowUps = options.maxFollowUps ?? MAX_FOLLOW_UPS;
const maxResultBytes = options.maxResultBytes ?? MAX_RESULT_BYTES_PER_TOOL;
const maxTotalResultBytes = options.maxTotalResultBytes ?? MAX_RESULT_BYTES_TOTAL;
// State
let currentSourceBody = options.sourceBody;
let currentLeg: NonStreamingProviderLegResult = options.initialLeg;
let cumulativeOutputBytes = 0;
let followUps = 0;
const receipts: ProviderLegReceipt[] = [options.initialLeg.receipt];
const usages: Array<ProviderLegUsage | null> = [options.initialLeg.usage];
// Accumulate initial leg receipt cost
let totalCostUsd = options.initialLeg.receipt.computedCostUsd ?? 0;
const errorResult: (msg: string, status?: number, code?: string) => ServerOwnedToolLoopResult = (
msg,
status = 500,
code = "internal_error"
) => ({
kind: "error",
errorResult: createErrorResult(status, msg, null, code) as unknown as ChatCoreErrorResult,
cumulativeUsage: aggregateUsageOrNull(usages),
totalCostUsd,
receipts,
followUps,
termination: "provider_error",
});
// Main loop
while (true) {
// Extract tool calls from current leg response
const response = currentLeg.kind === "ok" ? currentLeg.response : undefined;
if (!response) {
return errorResult("No response from provider leg");
}
// Extract tool calls by actual source format, not model alias heuristic
const toolCalls = extractToolCalls(response, options.sourceFormat);
// Classify ownership
const { serverOwned, clientNative } = await classifyServerOwnedCalls(
toolCalls,
options.executionContext
);
// No server-owned calls → done
if (serverOwned.length === 0) {
const termination = clientNative.length > 0 ? "client_tools" : "completed";
return {
kind: "ok",
response: currentLeg.kind === "ok" ? currentLeg.response : undefined,
responseForMemoryExtraction:
currentLeg.kind === "ok" ? currentLeg.responseForMemoryExtraction : undefined,
finalProviderBody: currentLeg.kind === "ok" ? currentLeg.providerBody : undefined,
finalProviderRequest: currentLeg.kind === "ok" ? currentLeg.providerRequest : undefined,
cumulativeUsage: aggregateUsageOrNull(usages),
totalCostUsd,
receipts,
followUps,
termination,
};
}
// Mixed tools → escape hatch, no follow-up
if (clientNative.length > 0) {
// Check abort before execute
if (options.abortSignal?.aborted) {
return {
kind: "error",
errorResult: createErrorResult(
499,
"Client closed request",
null,
"client_closed_request",
"invalid_request_error"
) as unknown as ChatCoreErrorResult,
cumulativeUsage: aggregateUsageOrNull(usages),
totalCostUsd,
receipts,
followUps,
termination: "client_abort",
};
}
// Execute server calls, then format with bounded serialization
const execResults = await options.executeServerOwned(serverOwned, options.executionContext);
// Build serialized text map using bounded serialization
const serMap = new Map<string, string>();
let remainingBytes = maxTotalResultBytes;
for (const r of execResults) {
const itemMaxBytes = Math.min(maxResultBytes, remainingBytes);
const bounded = serializeBoundedToolResult(r.result, itemMaxBytes);
serMap.set(r.id, bounded.text);
remainingBytes -= Buffer.byteLength(bounded.text, "utf8");
}
const formatted = formatEscapeHatchResponse(
response,
serverOwned,
execResults,
clientNative,
options.sourceFormat,
serMap
);
return {
kind: "ok",
response: formatted,
responseForMemoryExtraction:
currentLeg.kind === "ok" ? currentLeg.responseForMemoryExtraction : undefined,
finalProviderBody: currentLeg.kind === "ok" ? currentLeg.providerBody : undefined,
finalProviderRequest: currentLeg.kind === "ok" ? currentLeg.providerRequest : undefined,
cumulativeUsage: aggregateUsageOrNull(usages),
totalCostUsd,
receipts,
followUps,
termination: "mixed_tools",
};
}
// All server-owned: check abort before execute
if (options.abortSignal?.aborted) {
return {
kind: "error",
errorResult: createErrorResult(
499,
"Client closed request",
null,
"client_closed_request",
"invalid_request_error"
) as unknown as ChatCoreErrorResult,
cumulativeUsage: aggregateUsageOrNull(usages),
totalCostUsd,
receipts,
followUps,
termination: "client_abort",
};
}
// Execute server-owned calls
let execResults: ExecutedToolResult[];
try {
execResults = await options.executeServerOwned(serverOwned, options.executionContext);
} catch (err: unknown) {
if (err instanceof ServerOwnedExecutionError) {
const termMap: Record<string, ServerOwnedToolLoopResult["termination"]> = {
TOOL_IN_PROGRESS: "execution_in_progress",
TOOL_STATE_UNKNOWN: "execution_unknown",
IDENTITY_CONFLICT: "execution_identity_conflict",
TOOL_EXECUTION_ERROR: "execution_error",
TOOL_EXECUTION_TIMEOUT: "execution_timeout",
};
const termination = termMap[err.code] ?? "execution_error";
return {
kind: "error",
errorResult: mapExecutionError(err),
cumulativeUsage: aggregateUsageOrNull(usages),
totalCostUsd,
receipts,
followUps,
termination,
};
}
throw err;
}
// Serialize results with cumulative UTF-8 budget
const serMap = new Map<string, string>();
let anyTruncated = false;
let remainingBytes = maxTotalResultBytes - cumulativeOutputBytes;
if (remainingBytes < 0) remainingBytes = 0;
for (const r of execResults) {
const itemMaxBytes = Math.min(maxResultBytes, remainingBytes);
const bounded = serializeBoundedToolResult(r.result, itemMaxBytes);
serMap.set(r.id, bounded.text);
const bytes = Buffer.byteLength(bounded.text, "utf8");
cumulativeOutputBytes += bytes;
remainingBytes -= bytes;
if (bounded.truncated) {
anyTruncated = true;
}
}
// Check budget exhaustion — any truncated result terminates
if (anyTruncated || cumulativeOutputBytes >= maxTotalResultBytes) {
const formatted = formatEscapeHatchResponse(
response,
serverOwned,
execResults,
[],
options.sourceFormat,
serMap
);
return {
kind: "ok",
response: formatted,
responseForMemoryExtraction:
currentLeg.kind === "ok" ? currentLeg.responseForMemoryExtraction : undefined,
finalProviderBody: currentLeg.kind === "ok" ? currentLeg.providerBody : undefined,
finalProviderRequest: currentLeg.kind === "ok" ? currentLeg.providerRequest : undefined,
cumulativeUsage: aggregateUsageOrNull(usages),
totalCostUsd,
receipts,
followUps,
termination: "tool_output_budget",
};
}
// Check follow-up limit
if (followUps >= maxFollowUps) {
const formatted = formatEscapeHatchResponse(
response,
serverOwned,
execResults,
[],
options.sourceFormat,
serMap
);
return {
kind: "ok",
response: formatted,
responseForMemoryExtraction:
currentLeg.kind === "ok" ? currentLeg.responseForMemoryExtraction : undefined,
finalProviderBody: currentLeg.kind === "ok" ? currentLeg.providerBody : undefined,
finalProviderRequest: currentLeg.kind === "ok" ? currentLeg.providerRequest : undefined,
cumulativeUsage: aggregateUsageOrNull(usages),
totalCostUsd,
receipts,
followUps,
termination: "max_followups",
};
}
// Check deadline before resume
const remainingMs = loopDeadlineAtMs - now();
if (remainingMs < MIN_REMAINING_FOR_FOLLOW_UP_MS) {
const formatted = formatEscapeHatchResponse(
response,
serverOwned,
execResults,
[],
options.sourceFormat,
serMap
);
return {
kind: "ok",
response: formatted,
responseForMemoryExtraction:
currentLeg.kind === "ok" ? currentLeg.responseForMemoryExtraction : undefined,
finalProviderBody: currentLeg.kind === "ok" ? currentLeg.providerBody : undefined,
finalProviderRequest: currentLeg.kind === "ok" ? currentLeg.providerRequest : undefined,
cumulativeUsage: aggregateUsageOrNull(usages),
totalCostUsd,
receipts,
followUps,
termination: "deadline",
};
}
// Check abort before resume
if (options.abortSignal?.aborted) {
return {
kind: "error",
errorResult: createErrorResult(
499,
"Client closed request",
null,
"client_closed_request",
"invalid_request_error"
) as unknown as ChatCoreErrorResult,
cumulativeUsage: aggregateUsageOrNull(usages),
totalCostUsd,
receipts,
followUps,
termination: "client_abort",
};
}
// Build accumulated transcript
const nextSourceBody = buildFollowUpSourceBody({
sourceBody: currentSourceBody,
previousResponse: response,
toolCalls: serverOwned,
results: execResults,
sourceFormat: options.sourceFormat,
maxResultBytes,
maxTotalResultBytes: maxTotalResultBytes - cumulativeOutputBytes,
serializedResultTextById: serMap,
});
// Resume upstream
const nextLeg = await options.resumeUpstream(
nextSourceBody,
options.initialLeg.connectionId,
loopDeadlineAtMs
);
// Provider error → preserve identity
if (nextLeg.kind === "error") {
receipts.push(nextLeg.receipt);
usages.push(nextLeg.usage);
totalCostUsd += nextLeg.receipt.computedCostUsd ?? 0;
return {
kind: "error",
errorResult: nextLeg.result,
cumulativeUsage: aggregateUsageOrNull(usages),
totalCostUsd,
receipts,
followUps: followUps + 1,
termination: "provider_error",
};
}
// Connection mismatch check
if (nextLeg.connectionId !== options.initialLeg.connectionId) {
receipts.push(nextLeg.receipt);
usages.push(nextLeg.usage);
totalCostUsd += nextLeg.receipt.computedCostUsd ?? 0;
return {
kind: "error",
errorResult: createErrorResult(
409,
"Follow-up connection does not match initial connection",
null,
"LEASE_CONNECTION_MISMATCH",
"lease_error"
) as unknown as ChatCoreErrorResult,
cumulativeUsage: aggregateUsageOrNull(usages),
totalCostUsd,
receipts,
followUps: followUps + 1,
termination: "connection_mismatch",
};
}
// Update state for next iteration
currentSourceBody = nextSourceBody;
currentLeg = nextLeg;
followUps++;
receipts.push(nextLeg.receipt);
usages.push(nextLeg.usage);
totalCostUsd += nextLeg.receipt.computedCostUsd ?? 0;
}
}

121
src/lib/skills/stableJson.ts Normal file
View File

@@ -0,0 +1,121 @@
import { createHash } from "node:crypto";
const REJECT_MSG = "Value cannot be represented as canonical JSON";
function codePointCompare(a: string, b: string): number {
const aLen = a.length;
const bLen = b.length;
let i = 0;
let j = 0;
while (i < aLen && j < bLen) {
const aCode = a.charCodeAt(i);
const bCode = b.charCodeAt(j);
// Check if either is a surrogate pair
if (aCode >= 0xd800 && aCode <= 0xdbff && i + 1 < aLen) {
const aFull = (aCode - 0xd800) * 0x400 + (a.charCodeAt(i + 1) - 0xdc00) + 0x10000;
if (bCode >= 0xd800 && bCode <= 0xdbff && j + 1 < bLen) {
const bFull = (bCode - 0xd800) * 0x400 + (b.charCodeAt(j + 1) - 0xdc00) + 0x10000;
if (aFull !== bFull) return aFull - bFull;
i += 2;
j += 2;
} else {
// astral vs BMP
return 1;
}
} else if (bCode >= 0xd800 && bCode <= 0xdbff && j + 1 < bLen) {
return -1;
} else {
if (aCode !== bCode) return aCode - bCode;
i++;
j++;
}
}
return aLen - bLen;
}
function canonicalStringify(value: unknown, seen: Set<unknown>): string {
if (value === undefined) throw new TypeError(REJECT_MSG);
if (typeof value === "bigint") throw new TypeError(REJECT_MSG);
if (typeof value === "symbol") throw new TypeError(REJECT_MSG);
if (typeof value === "function") throw new TypeError(REJECT_MSG);
if (typeof value === "number") {
// Normalize -0 to 0
const normalized = Object.is(value, -0) ? 0 : value;
if (!Number.isFinite(normalized)) throw new TypeError(REJECT_MSG);
return String(normalized);
}
if (typeof value === "string") return JSON.stringify(value);
if (typeof value === "boolean") return String(value);
if (value === null) return "null";
if (typeof value === "object") {
if (seen.has(value)) throw new TypeError(REJECT_MSG);
seen.add(value);
if (Array.isArray(value)) {
// Reject sparse arrays
const len = (value as unknown[]).length;
for (let i = 0; i < len; i++) {
if (!(i in (value as unknown[]))) {
throw new TypeError(REJECT_MSG);
}
}
// Reject arrays with getters
for (let i = 0; i < len; i++) {
const desc = Object.getOwnPropertyDescriptor(value, i);
if (desc && (desc.get || desc.set)) {
throw new TypeError(REJECT_MSG);
}
}
const items = (value as unknown[]).map((v) => canonicalStringify(v, seen));
seen.delete(value);
return `[${items.join(",")}]`;
}
// Reject non-plain objects: Date, Map, Set, class instances, etc.
const proto = Object.getPrototypeOf(value);
if (proto !== Object.prototype && proto !== null) {
throw new TypeError(REJECT_MSG);
}
// Plain object: check for accessors on any own key, then sort by code point
const keys = Object.keys(value);
for (const k of keys) {
const desc = Object.getOwnPropertyDescriptor(value, k);
if (desc && (desc.get || desc.set)) {
throw new TypeError(REJECT_MSG);
}
}
keys.sort(codePointCompare);
const pairs = keys.map(
(k) =>
`${JSON.stringify(k)}:${canonicalStringify((value as Record<string, unknown>)[k], seen)}`
);
seen.delete(value);
return `{${pairs.join(",")}}`;
}
throw new TypeError(REJECT_MSG);
}
export function canonicalJson(value: unknown): string {
return canonicalStringify(value, new Set());
}
export function canonicalJsonSha256(value: unknown): string {
const json = canonicalJson(value);
return createHash("sha256").update(json, "utf8").digest("hex");
}
export function deriveToolRequestIdentity(input: {
apiKeyId: string;
stableClientRequestId: string | null;
skillRequestId: string;
postInjectionBody: Record<string, unknown>;
}): string {
const stableKey = input.stableClientRequestId ?? input.skillRequestId;
const bodyDigest = canonicalJsonSha256(input.postInjectionBody);
return `${input.apiKeyId}:${stableKey}:${bodyDigest}`;
}

241
src/lib/skills/toolExecutionFence.ts Normal file
View File

@@ -0,0 +1,241 @@
import { canonicalJsonSha256 } from "./stableJson";
import {
claimServerToolExecution,
finalizeServerToolExecution,
readRow,
} from "../db/skillExecutionFence";
import type { SqliteAdapter } from "../db/adapters/types";
import { getDbInstance } from "../db/core";
export type RunWithServerToolFenceResult<T> =
| { kind: "executed"; value: T }
| { kind: "replayed"; value: T; status: "success"; errorMessage: null }
| {
kind: "replayed";
value: unknown | null;
status: "error" | "timeout";
errorMessage: string | null;
}
| { kind: "in_progress" }
| { kind: "unknown" }
| { kind: "identity_conflict" };
type ExecutionKey = string;
const activePromises = new Map<
ExecutionKey,
{ promise: Promise<unknown>; status: "pending" | "resolved" | "rejected" }
>();
const POLL_INTERVAL_MS = 50;
const MAX_POLL_MS = 2_000;
function buildExecutionKey(
apiKeyId: string,
requestIdentity: string,
toolCallId: string
): ExecutionKey {
return `${apiKeyId}:${requestIdentity}:${toolCallId}`;
}
export interface RunWithServerToolFenceOptions<T> {
apiKeyId: string;
requestIdentity: string;
toolCallId: string;
toolName: string;
arguments: Record<string, unknown>;
leaseDurationMs: number;
execute: (executionId: string) => Promise<T>;
now?: () => number;
sleep?: (ms: number) => Promise<void>;
db?: SqliteAdapter;
}
export async function runWithServerToolFence<T>(
input: RunWithServerToolFenceOptions<T>
): Promise<RunWithServerToolFenceResult<T>> {
const db = input.db ?? getDbInstance();
const now = input.now ?? (() => Date.now());
const sleep = input.sleep ?? ((ms: number) => new Promise((resolve) => setTimeout(resolve, ms)));
const inputDigest = canonicalJsonSha256(input.arguments);
const leaseExpiresAt = new Date(now() + input.leaseDurationMs).toISOString();
const claim = claimServerToolExecution(
{
apiKeyId: input.apiKeyId,
requestIdentity: input.requestIdentity,
toolCallId: input.toolCallId,
toolName: input.toolName,
inputDigest,
leaseExpiresAt,
},
db,
now()
);
switch (claim.kind) {
case "claimed": {
const key = buildExecutionKey(input.apiKeyId, input.requestIdentity, input.toolCallId);
const claimStartTime = now();
const wrapperPromise = (async () => {
try {
const value = await input.execute(claim.executionId);
const durationMs = now() - claimStartTime;
finalizeServerToolExecution(
{
executionId: claim.executionId,
status: "success",
output: value,
errorMessage: null,
durationMs,
},
db
);
return { kind: "success" as const, value, errorMessage: null };
} catch (err: unknown) {
const message = err instanceof Error ? err.message : String(err);
const safeMessage = message.replace(/\bat\s+\/[^\s"']+/g, "[stack-redacted]");
const durationMs = now() - claimStartTime;
finalizeServerToolExecution(
{
executionId: claim.executionId,
status: "error",
output: null,
errorMessage: safeMessage,
durationMs,
},
db
);
return { kind: "error" as const, value: null, errorMessage: safeMessage };
}
})();
const entry: { promise: Promise<unknown>; status: "pending" | "resolved" | "rejected" } = {
promise: wrapperPromise as Promise<unknown>,
status: "pending",
};
activePromises.set(key, entry);
wrapperPromise.then(
() => {
entry.status = "resolved";
},
() => {
entry.status = "rejected";
}
);
try {
const result = await wrapperPromise;
if (result.kind === "success") {
return { kind: "executed", value: result.value as T };
}
// Handler errored — finalize already done, propagate
throw new Error(result.errorMessage ?? "tool execution failed");
} finally {
activePromises.delete(key);
}
}
case "replay": {
if (claim.status === "success") {
return {
kind: "replayed",
value: claim.output as T,
status: "success",
errorMessage: null,
};
}
return {
kind: "replayed",
value: claim.output ?? null,
status: claim.status,
errorMessage: claim.errorMessage,
};
}
case "in_progress": {
const key = buildExecutionKey(input.apiKeyId, input.requestIdentity, input.toolCallId);
const deadline = now() + MAX_POLL_MS;
while (now() < deadline) {
// Check process-internal promise first
const active = activePromises.get(key);
if (active) {
if (active.status === "resolved") {
try {
const result = await active.promise;
if (result && typeof result === "object" && "status" in result) {
const r = result as { value: unknown; status: string; errorMessage: string | null };
if (r.status === "success") {
return {
kind: "replayed",
value: r.value as T,
status: "success",
errorMessage: null,
};
}
return {
kind: "replayed",
value: r.value ?? null,
status: r.status as "error" | "timeout",
errorMessage: r.errorMessage,
};
}
return {
kind: "replayed",
value: result as T,
status: "success",
errorMessage: null,
};
} catch {
return { kind: "unknown" };
}
}
if (active.status === "rejected") {
// Rejected in-process promise — handler failed, return unknown
return { kind: "unknown" };
}
}
// Also check DB — another process may have finalized
const row = readRow(db, claim.executionId);
if (row && row.status !== "running") {
if (row.status === "success" || row.status === "error" || row.status === "timeout") {
let parsedOutput: unknown = null;
if (row.output !== null) {
try {
parsedOutput = JSON.parse(row.output);
} catch {
parsedOutput = row.output;
}
}
if (row.status === "success") {
return {
kind: "replayed",
value: parsedOutput as T,
status: "success",
errorMessage: null,
};
}
return {
kind: "replayed",
value: parsedOutput ?? null,
status: row.status as "error" | "timeout",
errorMessage: row.error_message,
};
}
return { kind: "unknown" };
}
await sleep(POLL_INTERVAL_MS);
}
return { kind: "in_progress" };
}
case "unknown":
return { kind: "unknown" };
case "identity_conflict":
return { kind: "identity_conflict" };
default:
return { kind: "unknown" };
}
}

200
src/lib/skills/toolLoopTypes.ts Normal file
View File

@@ -0,0 +1,200 @@
/**
* Shared types for the server-owned tool loop.
* All consumers use `import type` — no runtime imports.
*/
// ─── §5.4 Provider Leg ─────────────────────────────────────────────────────
export interface ProviderLegUsage {
prompt_tokens: number;
completion_tokens: number;
total_tokens: number;
cached_tokens?: number;
cache_read_input_tokens?: number;
cache_creation_input_tokens?: number;
reasoning_tokens?: number;
cost_in_usd_ticks?: number;
}
export interface ProviderLegReceipt {
index: number;
connectionId: string;
provider: string;
model: string;
startedAt: string;
endedAt: string;
latencyMs: number;
httpStatus: number;
errorType: string | null;
usage: ProviderLegUsage | null;
serviceTier: string | null;
computedCostUsd: number | null;
toolCalls: Array<{ id: string; name: string }>;
termination: string;
clientVisible: boolean;
}
export interface ChatCoreErrorResult {
success: false;
status: number;
response: Response;
error?: string;
errorCode?: string;
errorType?: string;
retryAfterMs?: number;
originalError?: unknown;
rawMessage?: string;
}
export type NonStreamingProviderLegResult =
| {
kind: "ok";
response: Record<string, unknown>;
responseForMemoryExtraction: Record<string, unknown>;
providerBody: Record<string, unknown>;
providerRequest: Record<string, unknown>;
usage: ProviderLegUsage | null;
responsePayloadFormat: string;
looksLikeSSE: boolean;
connectionId: string;
headers: Headers;
requestHeaders?: Record<string, string>;
requestUrl?: string;
upstreamResponse?: Response;
receipt: ProviderLegReceipt;
}
| {
kind: "error";
result: ChatCoreErrorResult;
receipt: ProviderLegReceipt;
usage: ProviderLegUsage | null;
};
// ─── §5.5 Tool Loop ────────────────────────────────────────────────────────
export interface ServerOwnedToolLoopOptions {
initialLeg: NonStreamingProviderLegResult & { kind: "ok" };
sourceBody: Record<string, unknown>;
sourceFormat: "openai" | "claude";
skillsModelId: string;
executionContext: ExecutionContext;
abortSignal?: AbortSignal;
now?: () => number;
executeServerOwned: (
calls: ToolCall[],
context: ExecutionContext
) => Promise<ExecutedToolResult[]>;
resumeUpstream: (
nextSourceBody: Record<string, unknown>,
expectedConnectionId: string,
deadlineAtMs: number
) => Promise<NonStreamingProviderLegResult>;
maxFollowUps?: number;
maxResultBytes?: number;
maxTotalResultBytes?: number;
deadlineAtMs: number;
}
export interface ServerOwnedToolLoopResult {
kind: "ok" | "error";
response?: Record<string, unknown>;
responseForMemoryExtraction?: Record<string, unknown>;
finalProviderBody?: Record<string, unknown>;
finalProviderRequest?: Record<string, unknown>;
errorResult?: ChatCoreErrorResult;
cumulativeUsage: ProviderLegUsage | null;
totalCostUsd: number;
receipts: ProviderLegReceipt[];
followUps: number;
termination:
| "completed"
| "client_tools"
| "mixed_tools"
| "max_followups"
| "tool_output_budget"
| "deadline"
| "client_abort"
| "provider_error"
| "connection_mismatch"
| "execution_in_progress"
| "execution_unknown"
| "execution_identity_conflict"
| "execution_error"
| "execution_timeout";
}
// ─── §5.1 Shared Context ───────────────────────────────────────────────────
export interface ToolCall {
id: string;
name: string;
arguments: Record<string, unknown>;
}
export interface ExecutionContext {
apiKeyId: string;
sessionId: string;
requestId: string;
requestIdentity?: string;
builtinToolNames?: string[];
injectedCustomSkillNames?: string[];
customSkillExecutionEnabled?: boolean;
executionFenceEnabled?: boolean;
provider?: string;
model?: string;
}
export interface ExecutedToolResult {
id: string;
name: string;
result: unknown;
replayed: boolean;
}
// ─── §5.2 Transcript Builder ───────────────────────────────────────────────
export interface BuildFollowUpTranscriptInput {
sourceBody: Record<string, unknown>;
previousResponse: Record<string, unknown>;
toolCalls: ToolCall[];
results: ExecutedToolResult[];
sourceFormat: "openai" | "claude";
maxResultBytes: number;
maxTotalResultBytes?: number;
serializedResultTextById?: Map<string, string>;
}
export interface BoundedToolResult {
text: string;
truncated: boolean;
originalBytes: number;
}
// ─── §5.3 Client Translate ─────────────────────────────────────────────────
export interface NonStreamingClientTranslateInput {
responseBody: Record<string, unknown>;
responsePayloadFormat: string;
clientResponseFormat: string;
sourceFormat: string;
provider: string;
model: string;
requestBody: Record<string, unknown>;
/**
* Transcript used for no-tool_calls reasoning replay (#1628).
* Must be the client-translated Chat `messages` (parent: `translatedBody.messages`),
* not `finalBody` — Responses-shaped `finalBody` has `input`, not `messages`.
*/
historyMessages?: unknown[] | null;
responseToolNameMap: Map<string, string> | null;
requestToolIdentityMap: Map<string, { namespace?: string; name: string }> | null;
reasoningCacheScope: string | null;
clientHeaders: Headers | Record<string, unknown> | null;
isClaudeCodeCompatible: boolean;
phase: "intermediate" | "final";
}
export interface NonStreamingClientTranslateResult {
response: Record<string, unknown>;
responseForMemoryExtraction: Record<string, unknown>;
}

12
src/shared/constants/featureFlagDefinitions.ts
View File

@@ -558,6 +558,18 @@ export const FEATURE_FLAG_DEFINITIONS: FeatureFlagDefinition[] = [
requiresRestart: false,
warningLevel: "info",
},
{
key: "SERVER_OWNED_TOOL_LOOP_ENABLED",
label: "Server-Owned Tool Loop",
description:
"Continue non-streaming server-owned tool calls until the model returns a client-usable response.",
descriptionI18nKey: "featureFlagServerOwnedToolLoopDescription",
category: "runtime",
defaultValue: "false",
type: "boolean",
requiresRestart: false,
warningLevel: "caution",
},
// ──────────────── CLI (5) ────────────────
{

14
src/shared/utils/featureFlags.ts
View File

@@ -172,3 +172,17 @@ export function isNetworkRotationSharedEgressGuardEnabled(): boolean {
return true;
}
}
export function isServerOwnedToolLoopEnabled(
reader: (key: string) => boolean = isFeatureFlagEnabled
): boolean {
try {
return reader("SERVER_OWNED_TOOL_LOOP_ENABLED");
} catch (error) {
console.error(
"[featureFlags] Failed to resolve SERVER_OWNED_TOOL_LOOP_ENABLED, defaulting to disabled:",
error instanceof Error ? error.message : error
);
return false;
}
}

11
stryker.conf.json
View File

@@ -238,6 +238,7 @@
"tests/unit/cursor-renewal.test.ts",
"tests/unit/custom-model-target-format.test.ts",
"tests/unit/db-reset-module-state.test.ts",
"tests/unit/db-server-tool-executions-migration.test.ts",
"tests/unit/db/stats-dbstat-optional.test.ts",
"tests/unit/ddg-circuit-breaker-null-content-6999-7000.test.ts",
"tests/unit/domain-persistence.test.ts",
@@ -254,6 +255,7 @@
"tests/unit/executor-devin-cli-agentic-acp.test.ts",
"tests/unit/executor-web-cookie-sweep.test.ts",
"tests/unit/false-terminal-401-quota.test.ts",
"tests/unit/follow-up-transcript.test.ts",
"tests/unit/format-provider-error-cause.test.ts",
"tests/unit/forwarded-header-budget.test.ts",
"tests/unit/fusion-vision-panel-3378.test.ts",
@@ -297,6 +299,8 @@
"tests/unit/no-memory-header.test.ts",
"tests/unit/noauth-autocombo-lockout-7623.test.ts",
"tests/unit/ollama-404-model-lockout-11071.test.ts",
"tests/unit/non-streaming-client-translate.test.ts",
"tests/unit/non-streaming-provider-leg.test.ts",
"tests/unit/non-streaming-sse-terminal-typescan-4459.test.ts",
"tests/unit/nvidia-410-model-scope.test.ts",
"tests/unit/nvidia-passthrough-models-6773.test.ts",
@@ -324,6 +328,7 @@
"tests/unit/probe-testall-isolation.test.ts",
"tests/unit/provider-breaker-halfopen-recovery.test.ts",
"tests/unit/provider-error-rules.test.ts",
"tests/unit/provider-execution-pipeline.test.ts",
"tests/unit/provider-health-matrix.test.ts",
"tests/unit/provider-request-failure-pipeline.test.ts",
"tests/unit/providers-route-codex-account-pool.test.ts",
@@ -373,6 +378,10 @@
"tests/unit/serial/combo-quota-share-cooldown-wait-timing.test.ts",
"tests/unit/serial/combo-strategy-fallbacks-half-open-timing.test.ts",
"tests/unit/serial/provider-health-autopilot.test.ts",
"tests/unit/server-owned-tool-loop-flag.test.ts",
"tests/unit/server-owned-tool-loop.test.ts",
"tests/unit/skill-execution-fence.test.ts",
"tests/unit/skills-interception-server-owned.test.ts",
"tests/unit/service-combo-metrics.test.ts",
"tests/unit/service-provider-plugin-registry.test.ts",
"tests/unit/services-branch-hardening.test.ts",
@@ -386,6 +395,7 @@
"tests/unit/sse-auth-exclusive-leases.test.ts",
"tests/unit/sse-auth-resource-404.test.ts",
"tests/unit/sse-auth.test.ts",
"tests/unit/stable-json.test.ts",
"tests/unit/stream-early-eof-breaker.test.ts",
"tests/unit/stream-readiness.test.ts",
"tests/unit/strict-random-deck.test.ts",
@@ -396,6 +406,7 @@
"tests/unit/thundering-herd.test.ts",
"tests/unit/token-refresh-race-comprehensive.test.ts",
"tests/unit/token-refresh-service.test.ts",
"tests/unit/tool-loop-usage.test.ts",
"tests/unit/tools-filter-anthropic-format.test.ts",
"tests/unit/tproxy-route.test.ts",
"tests/unit/trae-publiccred.test.ts",

4
tests/integration/chat-pipeline.test.ts
View File

@@ -1269,6 +1269,10 @@ test("chat pipeline returns current no-credentials contract when no provider con
test("chat pipeline surfaces upstream 500 responses as structured errors", async () => {
await seedConnection("openai", { apiKey: "sk-openai-500" });
await settingsDb.updateSettings({
requestRetry: 0,
maxRetryIntervalSec: 0,
});
globalThis.fetch = async () =>
new Response(JSON.stringify({ error: { message: "provider exploded" } }), {

259
tests/integration/server-owned-tool-loop-pipeline.test.ts Normal file
View File

@@ -0,0 +1,259 @@
import test from "node:test";
import assert from "node:assert/strict";
import { encodeSkillToolName } from "../../src/lib/skills/injection.ts";
import { createChatPipelineHarness } from "./_chatPipelineHarness.ts";
// Split out of skills-pipeline.test.ts (#12867): the three server-owned tool loop
// cases pushed that file to 1338 lines, past the 1200-line test cap. Same harness,
// its own instance so the two files stay independently runnable.
const harness = await createChatPipelineHarness("server-owned-tool-loop-pipeline");
const {
BaseExecutor,
buildOpenAIResponse,
buildOpenAIToolCallResponse,
buildRequest,
handleChat,
resetStorage,
seedApiKey,
seedConnection,
settingsDb,
skillExecutor,
skillRegistry,
} = harness;
test.beforeEach(async () => {
BaseExecutor.RETRY_CONFIG.delayMs = 0;
await resetStorage();
});
test.afterEach(async () => {
BaseExecutor.RETRY_CONFIG.delayMs = harness.originalRetryDelayMs;
await resetStorage();
});
test.after(async () => {
await harness.cleanup();
});
async function enableSkills() {
await settingsDb.updateSettings({ skillsEnabled: true });
}
async function registerSkill({
apiKeyId,
name,
version = "1.0.0",
handler,
enabled = true,
description = "Test skill",
mode,
tags,
installCount,
}) {
return skillRegistry.register({
apiKeyId,
name,
version,
description,
schema: {
input: {
type: "object",
properties: {
location: { type: "string" },
path: { type: "string" },
},
},
output: {
type: "object",
},
},
handler,
enabled,
mode,
tags,
installCount,
});
}
test("server-owned tool loop completes end-to-end for OpenAI without returning tool_results (issue #12696)", async () => {
await seedConnection("openai", { apiKey: "test-openai-key" });
const apiKey = await seedApiKey();
await enableSkills();
skillExecutor.registerHandler("weather-handler-loop-openai", async (input) => ({
forecast: `Sunny in ${input.location}`,
}));
await registerSkill({
apiKeyId: apiKey.id,
name: "lookupWeather",
handler: "weather-handler-loop-openai",
});
const prevFlag = process.env.SERVER_OWNED_TOOL_LOOP_ENABLED;
process.env.SERVER_OWNED_TOOL_LOOP_ENABLED = "true";
const fetchCalls: Array<{ url: string; body: Record<string, unknown> }> = [];
globalThis.fetch = async (input, init) => {
const bodyStr = typeof init?.body === "string" ? init.body : "{}";
const body = JSON.parse(bodyStr);
fetchCalls.push({ url: String(input), body });
if (fetchCalls.length === 1) {
return buildOpenAIToolCallResponse({
toolCallId: "call_weather_loop_1",
toolName: encodeSkillToolName("lookupWeather", "1.0.0"),
argumentsObject: { location: "Tokyo" },
});
}
return buildOpenAIResponse("The weather in Tokyo is 18C and sunny.", "gpt-4o-mini");
};
try {
const response = await handleChat(
buildRequest({
authKey: apiKey.key,
body: {
model: "openai/gpt-4o-mini",
stream: false,
messages: [{ role: "user", content: "What is the weather in Tokyo?" }],
},
})
);
assert.equal(response.status, 200);
const json = (await response.json()) as Record<string, unknown>;
assert.equal(fetchCalls.length, 2, "should have dispatched 2 provider legs");
assert.equal(
json.tool_results,
undefined,
"gateway should not return tool_results when loop is enabled"
);
assert.equal(json.choices[0].finish_reason, "stop");
assert.equal(json.choices[0].message.content, "The weather in Tokyo is 18C and sunny.");
assert.equal(fetchCalls[1].body.messages.length, 3);
assert.equal(fetchCalls[1].body.messages[1].role, "assistant");
assert.equal(fetchCalls[1].body.messages[2].role, "tool");
assert.equal(fetchCalls[1].body.messages[2].tool_call_id, "call_weather_loop_1");
} finally {
process.env.SERVER_OWNED_TOOL_LOOP_ENABLED = prevFlag;
}
});
test("server-owned tool loop follow-up failure propagates error cleanly", async () => {
await seedConnection("openai", { apiKey: "test-openai-key" });
const apiKey = await seedApiKey();
await enableSkills();
skillExecutor.registerHandler("weather-handler-loop-fail", async (input) => ({
forecast: `Sunny in ${input.location}`,
}));
await registerSkill({
apiKeyId: apiKey.id,
name: "lookupWeather",
handler: "weather-handler-loop-fail",
});
const prevFlag = process.env.SERVER_OWNED_TOOL_LOOP_ENABLED;
process.env.SERVER_OWNED_TOOL_LOOP_ENABLED = "true";
let callCount = 0;
globalThis.fetch = async () => {
callCount++;
if (callCount === 1) {
return buildOpenAIToolCallResponse({
callId: "call_weather_fail_1",
toolName: encodeSkillToolName("lookupWeather", "1.0.0"),
argumentsObject: { location: "Tokyo" },
});
}
return new Response(JSON.stringify({ error: { message: "Rate limit exceeded" } }), {
status: 429,
headers: { "content-type": "application/json" },
});
};
try {
const response = await handleChat(
buildRequest({
authKey: apiKey.key,
body: {
model: "openai/gpt-4o-mini",
stream: false,
messages: [{ role: "user", content: "What is the weather in Tokyo?" }],
},
})
);
assert.ok(callCount >= 2, `expected at least 2 fetch calls, got ${callCount}`);
assert.equal(response.status, 429);
} finally {
process.env.SERVER_OWNED_TOOL_LOOP_ENABLED = prevFlag;
}
});
test("server-owned tool loop completes end-to-end for Claude messages client without returning tool_results", async () => {
await seedConnection("openai", { apiKey: "test-openai-key" });
const apiKey = await seedApiKey();
await enableSkills();
skillExecutor.registerHandler("weather-handler-loop-claude-client", async (input) => ({
forecast: `Sunny in ${input.location}`,
}));
await registerSkill({
apiKeyId: apiKey.id,
name: "lookupWeather",
handler: "weather-handler-loop-claude-client",
});
const prevFlag = process.env.SERVER_OWNED_TOOL_LOOP_ENABLED;
process.env.SERVER_OWNED_TOOL_LOOP_ENABLED = "true";
const fetchCalls: Array<{ url: string; body: Record<string, unknown> }> = [];
globalThis.fetch = async (input, init) => {
const bodyStr = typeof init?.body === "string" ? init.body : "{}";
const body = JSON.parse(bodyStr);
fetchCalls.push({ url: String(input), body });
if (fetchCalls.length === 1) {
return buildOpenAIToolCallResponse({
toolCallId: "call_weather_claude_1",
toolName: encodeSkillToolName("lookupWeather", "1.0.0"),
argumentsObject: { location: "Tokyo" },
});
}
return buildOpenAIResponse("The weather in Tokyo is 18C and sunny.", "gpt-4o-mini");
};
try {
const response = await handleChat(
buildRequest({
url: "http://localhost/v1/messages",
authKey: apiKey.key,
body: {
model: "openai/gpt-4o-mini",
stream: false,
max_tokens: 256,
messages: [
{ role: "user", content: [{ type: "text", text: "What is the weather in Tokyo?" }] },
],
},
})
);
assert.equal(response.status, 200);
const json = (await response.json()) as Record<string, unknown>;
assert.equal(fetchCalls.length, 2, "should have dispatched 2 provider legs");
assert.equal(json.type, "message");
assert.equal(json.role, "assistant");
assert.equal(json.stop_reason, "end_turn");
assert.equal(json.content[0].text, "The weather in Tokyo is 18C and sunny.");
assert.equal(json.tool_results, undefined);
} finally {
process.env.SERVER_OWNED_TOOL_LOOP_ENABLED = prevFlag;
}
});

111
tests/unit/antigravity-byop-account-rotation.test.ts
View File

@@ -186,6 +186,117 @@ test("Antigravity BYOP 422 rotates to a sibling account and the request succeeds
}
});
test("streaming Antigravity BYOP 422 still rotates — execute must not cancel the error body", async () => {
const byopAccount = await createAntigravityAccount({
name: "antigravity-byop-stream-a",
email: "byop-stream-a@example.test",
accessToken: "fake-byop-account-a-token",
refreshToken: "fake-byop-account-a-refresh",
priority: 1,
});
const healthyAccount = await createAntigravityAccount({
name: "antigravity-healthy-stream-b",
email: "byop-stream-b@example.test",
accessToken: "fake-healthy-account-b-token",
refreshToken: "fake-healthy-account-b-refresh",
priority: 2,
});
let onboardCallsForA = 0;
const modelCalls: Array<{ token: string }> = [];
const originalFetch = globalThis.fetch;
globalThis.fetch = async (input, init) => {
const request = input instanceof Request ? input : new Request(input, init);
if (request.url.startsWith("https://oauth2.googleapis.com/token")) {
const form = await request.text().catch(() => "");
const refreshMatch = form.match(/refresh_token=([^&]+)/);
const refreshToken = refreshMatch ? decodeURIComponent(refreshMatch[1]) : "";
const accessToken =
refreshToken === "fake-byop-account-a-refresh"
? "fake-byop-account-a-token"
: "fake-healthy-account-b-token";
return new Response(JSON.stringify({ access_token: accessToken, expires_in: 3600 }), {
status: 200,
headers: { "Content-Type": "application/json" },
});
}
if (request.url.endsWith(":loadCodeAssist")) {
const token = (request.headers.get("authorization") || "").replace(/^Bearer\s+/i, "");
if (token === "fake-healthy-account-b-token") {
return new Response(
JSON.stringify({ cloudaicompanionProject: "projects/healthy-b-project" }),
{ status: 200, headers: { "Content-Type": "application/json" } }
);
}
return new Response("{}", {
status: 200,
headers: { "Content-Type": "application/json" },
});
}
if (request.url.endsWith(":onboardUser")) {
const token = (request.headers.get("authorization") || "").replace(/^Bearer\s+/i, "");
if (token === "fake-byop-account-a-token") {
onboardCallsForA += 1;
return new Response(JSON.stringify({ done: true }), {
status: 200,
headers: { "Content-Type": "application/json" },
});
}
return new Response(
JSON.stringify({
done: true,
cloudaicompanionProject: { name: "projects/healthy-b-project" },
}),
{ status: 200, headers: { "Content-Type": "application/json" } }
);
}
if (new URL(request.url).hostname === "cloudcode-pa.googleapis.com") {
modelCalls.push({
token: (request.headers.get("authorization") || "").replace(/^Bearer\s+/i, ""),
});
return new Response(
'data: {"response":{"candidates":[{"content":{"parts":[{"text":"ok from account B"}]},"finishReason":"STOP"}]}}\n\n',
{ status: 200, headers: { "Content-Type": "text/event-stream" } }
);
}
throw new Error(`Unexpected external fetch: ${request.url}`);
};
try {
const response = await handleChat(
buildRequest({
body: {
model: "antigravity/gemini-2.5-flash",
stream: true,
messages: [{ role: "user", content: "hello" }],
},
})
);
assert.equal(response.status, 200);
const bodyText = await response.text().catch(() => "");
assert.match(bodyText, /ok from account B/);
assert.ok(modelCalls.length >= 1, "model call should have been made");
assert.equal(modelCalls[0].token, "fake-healthy-account-b-token");
assert.equal(onboardCallsForA, 1);
const updatedA = await providersDb.getProviderConnectionById(byopAccount.id);
assert.ok(
updatedA && Number(updatedA.rateLimitedUntil) > Date.now(),
"BYOP account should be excluded from selection"
);
const updatedB = await providersDb.getProviderConnectionById(healthyAccount.id);
assert.ok(
!updatedB ||
!Number(updatedB.rateLimitedUntil) ||
Number(updatedB.rateLimitedUntil) <= Date.now(),
"healthy sibling account must not be excluded"
);
} finally {
globalThis.fetch = originalFetch;
clearAntigravityProjectCache();
}
});
test("Antigravity BYOP with no sibling account surfaces the actionable 422 and excludes the connection", async () => {
const byopAccount = await createAntigravityAccount({
name: "antigravity-byop-only",

70
tests/unit/chatcore-failure-usage.test.ts
View File

@@ -64,16 +64,74 @@ test("applies the unknown/undefined fallbacks", () => {
test("combo strategy is included only for combo requests", () => {
const combo = buildFailureUsageRecord({
provider: "x", model: "y", connectionId: null, apiKeyInfo: null,
effectiveServiceTier: "standard", isCombo: true, comboStrategy: "round-robin",
statusCode: 500, errorCode: "boom", latencyMs: 1,
provider: "x",
model: "y",
connectionId: null,
apiKeyInfo: null,
effectiveServiceTier: "standard",
isCombo: true,
comboStrategy: "round-robin",
statusCode: 500,
errorCode: "boom",
latencyMs: 1,
});
assert.equal(combo.comboStrategy, "round-robin");
const comboNoStrategy = buildFailureUsageRecord({
provider: "x", model: "y", connectionId: null, apiKeyInfo: null,
effectiveServiceTier: "standard", isCombo: true, comboStrategy: null,
statusCode: 500, errorCode: "boom", latencyMs: 1,
provider: "x",
model: "y",
connectionId: null,
apiKeyInfo: null,
effectiveServiceTier: "standard",
isCombo: true,
comboStrategy: null,
statusCode: 500,
errorCode: "boom",
latencyMs: 1,
});
assert.equal(comboNoStrategy.comboStrategy, undefined);
});
test("maps aggregate usage onto failure tokens instead of zeros", () => {
const r = buildFailureUsageRecord({
provider: "openai",
model: "gpt-4o",
connectionId: "conn-1",
apiKeyInfo: { id: "key-1", name: "My Key" },
effectiveServiceTier: "priority",
isCombo: false,
comboStrategy: null,
statusCode: 429,
errorCode: "rate_limited",
latencyMs: 50,
aggregate: {
prompt_tokens: 100,
completion_tokens: 20,
cache_read_input_tokens: 10,
reasoning_tokens: 5,
},
});
assert.deepEqual(r.tokens, {
input: 100,
output: 20,
cacheRead: 10,
cacheCreation: 0,
reasoning: 5,
});
});
test("keeps zeroed tokens when aggregate is absent", () => {
const r = buildFailureUsageRecord({
provider: "openai",
model: "gpt-4o",
connectionId: null,
apiKeyInfo: null,
effectiveServiceTier: "standard",
isCombo: false,
comboStrategy: null,
statusCode: 502,
errorCode: null,
latencyMs: 7,
});
assert.deepEqual(r.tokens, { input: 0, output: 0, cacheRead: 0, cacheCreation: 0, reasoning: 0 });
});

347
tests/unit/chatcore-memory-skills-injection.test.ts
View File

@@ -9,12 +9,24 @@ import path from "node:path";
const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-mem-skills-"));
process.env.DATA_DIR = TEST_DATA_DIR;
const { getSkillsProviderForFormat, injectMemoryAndSkills, sortToolsByName } =
await import("../../open-sse/handlers/chatCore/memorySkillsInjection.ts");
const {
getSkillsProviderForFormat,
injectMemoryAndSkills,
sortToolsByName,
mergeInjectedFallbackOwnerNames,
} = await import("../../open-sse/handlers/chatCore/memorySkillsInjection.ts");
const { FORMATS } = await import("../../open-sse/translator/formats.ts");
const core = await import("../../src/lib/db/core.ts");
const { skillRegistry } = await import("../../src/lib/skills/registry.ts");
function resetSkillsRegistry() {
skillRegistry["registeredSkills"].clear();
skillRegistry["versionCache"].clear();
skillRegistry.invalidateCache();
}
test.after(() => {
resetSkillsRegistry();
core.resetDbInstance();
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });
});
@@ -258,3 +270,334 @@ test("injectMemoryAndSkills does not inject memory tools when memory is disabled
invalidateMemorySettingsCache();
});
// ─── Task 3: owner-set provenance + stream gate RED tests ────────────────────
test("stream:true + skills enabled + registry has items → no custom skill tool injected, injectedCustomSkillNames=[]", async () => {
const { updateSettings } = await import("../../src/lib/db/settings.ts");
const { invalidateMemorySettingsCache: inv2 } = await import("../../src/lib/memory/settings.ts");
await updateSettings({ memoryEnabled: true, memoryMaxTokens: 2000, skillsEnabled: true });
inv2();
resetSkillsRegistry();
await skillRegistry.register({
name: "test-skill",
version: "1.0.0",
description: "test skill for stream gate",
schema: { input: {}, output: {} },
handler: "test-handler",
enabled: true,
apiKeyId: "owner-stream-skills",
mode: "on",
});
const body: Record<string, unknown> = {
model: "gpt-4o",
stream: true,
messages: [{ role: "user", content: "hello" }],
};
const result = await injectMemoryAndSkills({
body,
memoryOwnerId: "owner-stream-skills",
provider: "openai",
effectiveModel: "gpt-4o",
sourceFormat: FORMATS.OPENAI,
targetFormat: FORMATS.OPENAI,
backgroundReason: null,
log: { debug: () => {} },
});
const toolNames = (
(result.body.tools as { function?: { name?: string }; name?: string }[] | undefined) ?? []
).map((t) => t.function?.name ?? t.name);
const hasCustomSkill = toolNames.some(
(n) => typeof n === "string" && (n.includes("test-skill") || n.startsWith("omr_skill_"))
);
assert.equal(hasCustomSkill, false, "stream:true must not inject custom skill tools");
assert.deepEqual(
(result as Record<string, unknown>).injectedCustomSkillNames,
[],
"injectedCustomSkillNames must be empty for stream requests"
);
resetSkillsRegistry();
inv2();
});
test("memory actual injection → builtinToolNames equals exactly the newly added memory tool names", async () => {
const { updateSettings } = await import("../../src/lib/db/settings.ts");
const { invalidateMemorySettingsCache: inv3 } = await import("../../src/lib/memory/settings.ts");
const { MEMORY_BUILTIN_TOOL_NAMES } = await import("../../src/lib/skills/memoryBuiltins.ts");
await updateSettings({ memoryEnabled: true, memoryMaxTokens: 2000 });
inv3();
resetSkillsRegistry();
const body: Record<string, unknown> = {
model: "gpt-4o",
messages: [{ role: "user", content: "hello" }],
tools: [{ type: "function", function: { name: "some_client_tool", description: "x" } }],
};
const result = await injectMemoryAndSkills({
body,
memoryOwnerId: "owner-builtin-own",
provider: "openai",
effectiveModel: "gpt-4o",
sourceFormat: FORMATS.OPENAI,
targetFormat: FORMATS.OPENAI,
backgroundReason: null,
log: { debug: () => {} },
});
const builtinToolNames = (result as Record<string, unknown>).builtinToolNames as
string[] | undefined;
assert.ok(builtinToolNames, "builtinToolNames must be present in result");
const expectedNewMemoryNames = [...MEMORY_BUILTIN_TOOL_NAMES];
assert.deepEqual(
builtinToolNames.sort(),
expectedNewMemoryNames.sort(),
"builtinToolNames must equal exactly the newly added memory tool names"
);
resetSkillsRegistry();
inv3();
});
test("client already has memory_search → not injected, not in builtinToolNames", async () => {
const { updateSettings } = await import("../../src/lib/db/settings.ts");
const { invalidateMemorySettingsCache: inv4 } = await import("../../src/lib/memory/settings.ts");
const { MEMORY_SEARCH_TOOL_NAME } = await import("../../src/lib/skills/memoryBuiltins.ts");
await updateSettings({ memoryEnabled: true, memoryMaxTokens: 2000 });
inv4();
resetSkillsRegistry();
const body: Record<string, unknown> = {
model: "gpt-4o",
messages: [{ role: "user", content: "hello" }],
tools: [
{
type: "function",
function: { name: MEMORY_SEARCH_TOOL_NAME, description: "client memory" },
},
],
};
const result = await injectMemoryAndSkills({
body,
memoryOwnerId: "owner-client-mem",
provider: "openai",
effectiveModel: "gpt-4o",
sourceFormat: FORMATS.OPENAI,
targetFormat: FORMATS.OPENAI,
backgroundReason: null,
log: { debug: () => {} },
});
const toolNames = (
(result.body.tools as { function?: { name?: string }[] | undefined }) ?? []
).map((t: { function?: { name?: string } }) => t.function?.name);
const memorySearchCount = toolNames.filter((n) => n === MEMORY_SEARCH_TOOL_NAME).length;
assert.equal(memorySearchCount, 1, "only one memory_search (client's) must exist");
const builtinToolNames = (result as Record<string, unknown>).builtinToolNames as
string[] | undefined;
assert.ok(builtinToolNames, "builtinToolNames must be present");
assert.equal(
builtinToolNames.includes(MEMORY_SEARCH_TOOL_NAME),
false,
"client-owned memory_search must NOT be in builtinToolNames"
);
resetSkillsRegistry();
inv4();
});
test("custom skill client collision: client has same encoded skill name → not injected, not in injectedCustomSkillNames", async () => {
const { updateSettings } = await import("../../src/lib/db/settings.ts");
const { invalidateMemorySettingsCache: inv5 } = await import("../../src/lib/memory/settings.ts");
const { encodeSkillToolName } = await import("../../src/lib/skills/injection.ts");
await updateSettings({ memoryEnabled: true, memoryMaxTokens: 2000, skillsEnabled: true });
inv5();
resetSkillsRegistry();
await skillRegistry.register({
name: "collision-skill",
version: "1.0.0",
description: "skill that collides",
schema: { input: {}, output: {} },
handler: "collision-handler",
enabled: true,
apiKeyId: "owner-collision",
mode: "on",
});
const encodedName = encodeSkillToolName("collision-skill", "1.0.0");
const body: Record<string, unknown> = {
model: "gpt-4o",
messages: [{ role: "user", content: "hello" }],
tools: [{ type: "function", function: { name: encodedName, description: "client collision" } }],
};
const result = await injectMemoryAndSkills({
body,
memoryOwnerId: "owner-collision",
provider: "openai",
effectiveModel: "gpt-4o",
sourceFormat: FORMATS.OPENAI,
targetFormat: FORMATS.OPENAI,
backgroundReason: null,
log: { debug: () => {} },
});
const toolNames = (
(result.body.tools as { function?: { name?: string }[] | undefined }) ?? []
).map((t: { function?: { name?: string } }) => t.function?.name);
const count = toolNames.filter((n) => n === encodedName).length;
assert.equal(count, 1, "only one instance of encoded name must exist (client's)");
const injectedCustomSkillNames = (result as Record<string, unknown>).injectedCustomSkillNames as
string[] | undefined;
assert.ok(injectedCustomSkillNames, "injectedCustomSkillNames must be present");
assert.equal(
injectedCustomSkillNames.includes(encodedName),
false,
"client-owned skill name must NOT be in injectedCustomSkillNames"
);
resetSkillsRegistry();
inv5();
});
test("web-search fallback: client has same tool name → not added to builtinToolNames", async () => {
const { updateSettings } = await import("../../src/lib/db/settings.ts");
const { invalidateMemorySettingsCache: inv6 } = await import("../../src/lib/memory/settings.ts");
const { OMNIROUTE_WEB_SEARCH_FALLBACK_TOOL_NAME } =
await import("../../open-sse/services/webSearchFallback.ts");
await updateSettings({ memoryEnabled: true, memoryMaxTokens: 2000 });
inv6();
resetSkillsRegistry();
const body: Record<string, unknown> = {
model: "gpt-4o",
messages: [{ role: "user", content: "hello" }],
tools: [
{
type: "function",
function: { name: OMNIROUTE_WEB_SEARCH_FALLBACK_TOOL_NAME, description: "client search" },
},
],
};
const result = await injectMemoryAndSkills({
body,
memoryOwnerId: "owner-websearch",
provider: "openai",
effectiveModel: "gpt-4o",
sourceFormat: FORMATS.OPENAI,
targetFormat: FORMATS.OPENAI,
backgroundReason: null,
log: { debug: () => {} },
});
const builtinToolNames = (result as Record<string, unknown>).builtinToolNames as
string[] | undefined;
assert.ok(builtinToolNames, "builtinToolNames must be present");
assert.equal(
builtinToolNames.includes(OMNIROUTE_WEB_SEARCH_FALLBACK_TOOL_NAME),
false,
"client-owned web search tool must NOT be in builtinToolNames"
);
resetSkillsRegistry();
inv6();
});
// ─── Fix Round 2: Defect 5 — mergeInjectedFallbackOwnerNames + provenance ───
test("mergeInjectedFallbackOwnerNames: adds name only when enabled=true, convertedToolCount>0, toolName non-null, and not already in client tools", () => {
const result = mergeInjectedFallbackOwnerNames({ builtinToolNames: ["memory_search"] }, [
{ enabled: true, toolName: "omniroute_web_search", convertedToolCount: 2 },
{ enabled: true, toolName: null, convertedToolCount: 1 },
{ enabled: false, toolName: "omniroute_web_fetch", convertedToolCount: 3 },
{ enabled: true, toolName: "omniroute_web_fetch", convertedToolCount: 0 },
]);
assert.deepEqual(result.builtinToolNames, ["memory_search", "omniroute_web_search"]);
});
test("mergeInjectedFallbackOwnerNames: does not mutate input injectionResult", () => {
const input = { builtinToolNames: ["original"] };
const plans = [{ enabled: true, toolName: "omniroute_web_search", convertedToolCount: 1 }];
const result = mergeInjectedFallbackOwnerNames(input, plans);
// input must be unchanged
assert.deepEqual(input.builtinToolNames, ["original"]);
// result is a new object
assert.notEqual(result, input);
assert.deepEqual(result.builtinToolNames, ["original", "omniroute_web_search"]);
});
test("mergeInjectedFallbackOwnerNames: skips name already present in pre-conversion client tools", () => {
const result = mergeInjectedFallbackOwnerNames({ builtinToolNames: ["omniroute_web_search"] }, [
{ enabled: true, toolName: "omniroute_web_search", convertedToolCount: 2 },
]);
// Must not duplicate — omniroute_web_search already present
assert.deepEqual(result.builtinToolNames, ["omniroute_web_search"]);
});
// ─── Fix Round 3: Defect 3 — pre-conversion collision guard ─────────────────
test("mergeInjectedFallbackOwnerNames: client has omniroute_web_search → not added to builtinToolNames even if enabled=true", () => {
// Scenario: client sends {type:"web_search"} plus function named omniroute_web_search.
// prepareWebSearchFallbackBody emits enabled=true, convertedToolCount=2 (from the
// builtin conversion) but the synthetic tool was NOT added because client already has it.
// mergeInjectedFallbackOwnerNames must check pre-conversion client names.
const result = mergeInjectedFallbackOwnerNames(
{ builtinToolNames: [] },
[{ enabled: true, toolName: "omniroute_web_search", convertedToolCount: 2 }],
["omniroute_web_search"]
);
// Must NOT add omniroute_web_search — client already owns it
assert.deepEqual(result.builtinToolNames, []);
});
test("mergeInjectedFallbackOwnerNames: client has omniroute_web_fetch → not added to builtinToolNames", () => {
const result = mergeInjectedFallbackOwnerNames(
{ builtinToolNames: [] },
[{ enabled: true, toolName: "omniroute_web_fetch", convertedToolCount: 1 }],
["omniroute_web_fetch"]
);
assert.deepEqual(result.builtinToolNames, []);
});
test("mergeInjectedFallbackOwnerNames: client does NOT have the fallback name → added to builtinToolNames", () => {
const result = mergeInjectedFallbackOwnerNames(
{ builtinToolNames: [] },
[{ enabled: true, toolName: "omniroute_web_search", convertedToolCount: 2 }],
["some_other_tool"]
);
assert.deepEqual(result.builtinToolNames, ["omniroute_web_search"]);
});
test("mergeInjectedFallbackOwnerNames: no preConversionClientToolNames provided → falls back to existing behavior", () => {
const result = mergeInjectedFallbackOwnerNames({ builtinToolNames: [] }, [
{ enabled: true, toolName: "omniroute_web_search", convertedToolCount: 2 },
]);
assert.deepEqual(result.builtinToolNames, ["omniroute_web_search"]);
});

23
tests/unit/chatcore-stream-error-result.test.ts
View File

@@ -67,3 +67,26 @@ test("getUpstreamErrorIdentifier returns a non-empty string code or undefined",
assert.equal(getUpstreamErrorIdentifier(null), undefined);
assert.equal(getUpstreamErrorIdentifier("ECONNRESET"), undefined);
});
test("non-streaming runNonStreamingProviderLeg is inside a try that maps semaphore errors", async () => {
const fs = await import("node:fs");
const src = fs.readFileSync("open-sse/handlers/chatCore.ts", "utf8");
// `let`, not `const`, since 6077b9dd (#12867) made the finalization step reassign
// legResult. The guard is about the try/catch that wraps the call, not the keyword.
const idx = src.search(/(?:const|let) legResult = await runNonStreamingProviderLeg/);
assert.ok(idx >= 0, "non-streaming branch must exist");
const start = src.lastIndexOf("if (!stream)", idx);
const end = src.indexOf("// Streaming response", idx);
assert.ok(start >= 0 && end > start, "non-stream block bounds");
const block = src.slice(start, end);
assert.match(
block,
/try\s*\{[\s\S]*runNonStreamingProviderLeg/,
"non-stream leg must sit in a try so SEMAPHORE_TIMEOUT cannot escape handleChatCore"
);
assert.match(
block,
/isSemaphoreCapacityError/,
"same catch that maps stream semaphore errors must cover the non-stream leg"
);
});

242
tests/unit/db-server-tool-executions-migration.test.ts Normal file
View File

@@ -0,0 +1,242 @@
process.env.DISABLE_SQLITE_AUTO_BACKUP = "true";
import test from "node:test";
import assert from "node:assert/strict";
import fs from "node:fs";
import path from "node:path";
import os from "node:os";
import { createRequire } from "node:module";
const require_ = createRequire(import.meta.url);
const BetterSqlite3 = require_("better-sqlite3") as typeof import("better-sqlite3");
import { createBetterSqliteAdapter } from "../../src/lib/db/adapters/betterSqliteAdapter";
import type { SqliteAdapter } from "../../src/lib/db/adapters/types";
const MIGRATION_174_PATH = path.resolve(
import.meta.dirname ?? ".",
"../../src/lib/db/migrations/174_server_tool_executions.sql"
);
const MIGRATION_174_SQL = fs.readFileSync(MIGRATION_174_PATH, "utf8");
// Minimal pre-174 fixture: only skills + skill_executions tables.
// No SCHEMA_SQL import from core.ts, no runMigrations.
const PRE_174_FIXTURE = `
CREATE TABLE IF NOT EXISTS skills (
id TEXT PRIMARY KEY,
api_key_id TEXT NOT NULL,
name TEXT NOT NULL,
version TEXT NOT NULL DEFAULT '1.0.0',
description TEXT,
schema TEXT NOT NULL,
handler TEXT NOT NULL,
enabled INTEGER NOT NULL DEFAULT 1,
created_at TEXT NOT NULL DEFAULT (datetime('now')),
updated_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE TABLE IF NOT EXISTS skill_executions (
id TEXT PRIMARY KEY,
skill_id TEXT NOT NULL,
api_key_id TEXT NOT NULL,
session_id TEXT,
input TEXT NOT NULL,
output TEXT,
status TEXT NOT NULL CHECK(status IN ('pending', 'running', 'success', 'error', 'timeout')),
error_message TEXT,
duration_ms INTEGER,
created_at TEXT NOT NULL DEFAULT (datetime('now')),
FOREIGN KEY (skill_id) REFERENCES skills(id) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_skill_executions_skill ON skill_executions(skill_id);
CREATE INDEX IF NOT EXISTS idx_skill_executions_api_key ON skill_executions(api_key_id);
`;
function makeTempDb(): {
adapter: SqliteAdapter;
dir: string;
raw: import("better-sqlite3").Database;
} {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), "migration-174-test-"));
const dbPath = path.join(dir, "test.db");
const raw = new BetterSqlite3(dbPath);
raw.pragma("journal_mode = WAL");
raw.pragma("busy_timeout = 2000");
// Create migrations tracking table
raw.exec(`
CREATE TABLE IF NOT EXISTS _omniroute_migrations (
version TEXT PRIMARY KEY,
name TEXT NOT NULL,
applied_at TEXT NOT NULL DEFAULT (datetime('now'))
);
`);
// Apply minimal pre-174 fixture (skills + skill_executions only)
raw.exec(PRE_174_FIXTURE);
// Apply migration 174 via raw.exec (real SQL, twice for idempotency)
raw.exec(MIGRATION_174_SQL);
raw.exec(MIGRATION_174_SQL);
const adapter = createBetterSqliteAdapter(raw);
return { adapter, dir, raw };
}
function getColumnInfo(raw: import("better-sqlite3").Database, table: string) {
return raw.pragma(`table_info(${table})`) as Array<{
name: string;
type: string;
notnull: number;
pk: number;
}>;
}
function getIndexInfo(raw: import("better-sqlite3").Database, table: string) {
return raw.pragma(`index_list(${table})`) as Array<{
name: string;
unique: number;
}>;
}
// ── Migration 174 tests ──
test("migration 174: server_tool_executions table exists with correct columns", (_, done) => {
const { raw, dir } = makeTempDb();
try {
const tables = raw.pragma("table_list") as Array<{ name: string }>;
const names = tables.map((t) => t.name);
assert.ok(
names.includes("server_tool_executions"),
`Expected server_tool_executions in: ${names.join(", ")}`
);
const cols = getColumnInfo(raw, "server_tool_executions");
const colNames = cols.map((c) => c.name);
assert.ok(colNames.includes("id"), "must have id column");
assert.ok(colNames.includes("api_key_id"), "must have api_key_id column");
assert.ok(colNames.includes("request_identity"), "must have request_identity column");
assert.ok(colNames.includes("tool_call_id"), "must have tool_call_id column");
assert.ok(colNames.includes("tool_name"), "must have tool_name column");
assert.ok(colNames.includes("input_digest"), "must have input_digest column");
assert.ok(colNames.includes("status"), "must have status column");
assert.ok(colNames.includes("claim_expires_at"), "must have claim_expires_at column");
assert.ok(colNames.includes("duration_ms"), "must have duration_ms column");
} finally {
raw.close();
fs.rmSync(dir, { recursive: true, force: true });
done();
}
});
test("migration 174: UNIQUE constraint on (api_key_id, request_identity, tool_call_id)", (_, done) => {
const { raw, dir } = makeTempDb();
try {
raw.exec(`
INSERT INTO server_tool_executions
(id, api_key_id, request_identity, tool_call_id, tool_name, input_digest, status, claim_expires_at)
VALUES ('e1','k1','r1','c1','tool_a','d1','running',datetime('now'))
`);
assert.throws(() => {
raw.exec(`
INSERT INTO server_tool_executions
(id, api_key_id, request_identity, tool_call_id, tool_name, input_digest, status, claim_expires_at)
VALUES ('e2','k1','r1','c1','tool_a','d1','running',datetime('now'))
`);
}, /UNIQUE/i);
} finally {
raw.close();
fs.rmSync(dir, { recursive: true, force: true });
done();
}
});
test("migration 174: two indexes exist on server_tool_executions", (_, done) => {
const { raw, dir } = makeTempDb();
try {
const indexes = getIndexInfo(raw, "server_tool_executions");
const names = indexes.map((i) => i.name);
assert.ok(
names.some((n) => n.includes("status_expiry")),
`Expected status_expiry index, got: ${names.join(", ")}`
);
assert.ok(
names.some((n) => n.includes("created")),
`Expected created index, got: ${names.join(", ")}`
);
} finally {
raw.close();
fs.rmSync(dir, { recursive: true, force: true });
done();
}
});
test("migration 174: existing skill_executions data preserved after migration", (_, done) => {
const { raw, dir } = makeTempDb();
try {
// Insert a skill to satisfy FK
raw.exec(`
INSERT INTO skills (id, api_key_id, name, version, schema, handler)
VALUES ('s1','k1','test','1.0.0','{}','h.js')
`);
raw.exec(`
INSERT INTO skill_executions (id, skill_id, api_key_id, input, status)
VALUES ('old_exec','s1','k1','{"q":"test"}','success')
`);
const rows = raw.prepare("SELECT * FROM skill_executions WHERE id = 'old_exec'").all();
assert.equal(rows.length, 1, "old row should exist after migration 174");
} finally {
raw.close();
fs.rmSync(dir, { recursive: true, force: true });
done();
}
});
test("migration 174: skill_executions still enforces skill_id NOT NULL", (_, done) => {
const { raw, dir } = makeTempDb();
try {
assert.throws(() => {
raw.exec(
`INSERT INTO skill_executions (id, api_key_id, input, status)
VALUES ('bad','k1','{}','running')`
);
}, /NOT NULL/i);
} finally {
raw.close();
fs.rmSync(dir, { recursive: true, force: true });
done();
}
});
test("migration 174: custom skill execution write still works after migration", (_, done) => {
const { raw, dir } = makeTempDb();
try {
// Insert a skill to satisfy FK
raw.exec(`
INSERT INTO skills (id, api_key_id, name, version, schema, handler)
VALUES ('s2','k1','test2','1.0.0','{}','h.js')
`);
raw.exec(`
INSERT INTO skill_executions (id, skill_id, api_key_id, input, status)
VALUES ('new_exec','s2','k1','{"q":"test2"}','success')
`);
const allRows = raw.prepare("SELECT * FROM skill_executions").all();
assert.ok(allRows.length >= 1, "should read skill_executions after migration 174");
} finally {
raw.close();
fs.rmSync(dir, { recursive: true, force: true });
done();
}
});
test("migration 174: idempotent — running twice does not error or duplicate", (_, done) => {
const { raw, dir } = makeTempDb();
try {
// makeTempDb already runs the SQL twice; verify no error and table exists
const tables = raw.pragma("table_list") as Array<{ name: string }>;
const names = tables.map((t) => t.name);
assert.ok(names.includes("server_tool_executions"), "table must exist after double-apply");
// Verify no duplicate columns or indexes from double-apply
const indexes = getIndexInfo(raw, "server_tool_executions");
const statusIdx = indexes.filter((i) => i.name.includes("status_expiry"));
assert.equal(statusIdx.length, 1, "must have exactly one status_expiry index");
} finally {
raw.close();
fs.rmSync(dir, { recursive: true, force: true });
done();
}
});

2
tests/unit/feature-flags-settings.test.ts
View File

@@ -39,7 +39,7 @@ const {
// OMNIROUTE_DISABLE_THINKING_LEVEL_VARIANTS bumped it from 53 to 54;
// the dead ONEPROXY_ENABLED (readerless since the 1proxy purge, #12091)
// brought it back to 53. UNIVERSAL_CONTEXT_HANDOFF_ENABLED bumped it to 54.
const EXPECTED_FEATURE_FLAG_COUNT = 54;
const EXPECTED_FEATURE_FLAG_COUNT = 55;
// ──────────────────────────────────────────────────────
// Test group 1 — Flag definitions registry

705
tests/unit/follow-up-transcript.test.ts Normal file
View File

@@ -0,0 +1,705 @@
import test from "node:test";
import assert from "node:assert/strict";
import { Buffer } from "node:buffer";
import {
MAX_RESULT_BYTES_PER_TOOL,
MAX_RESULT_BYTES_TOTAL,
serializeBoundedToolResult,
buildFollowUpSourceBody,
} from "../../src/lib/skills/followUpTranscript.ts";
import type {
ToolCall,
ExecutedToolResult,
BuildFollowUpTranscriptInput,
BoundedToolResult,
} from "../../src/lib/skills/toolLoopTypes.ts";
type UnknownRecord = Record<string, unknown>;
const MESSAGES_REQUIRED = "buildFollowUpSourceBody requires sourceBody.messages array";
const NON_SERIALIZABLE_JSON = '{"error":"Tool result is not JSON-serializable"}';
// ─── OpenAI Chat Completions ─────────────────────────────────────────────────
function openaiInput(overrides: Partial<BuildFollowUpTranscriptInput> = {}) {
const sourceBody = {
model: "gpt-4o",
messages: [
{ role: "system", content: "You are a helpful assistant." },
{ role: "user", content: "remember foo" },
],
tools: [
{
type: "function",
function: {
name: "memory_search",
description: "search memory",
parameters: { type: "object", properties: { query: { type: "string" } } },
},
},
],
tool_choice: "auto",
stream: false,
} as UnknownRecord;
const previousResponse = {
id: "chatcmpl-abc",
object: "chat.completion",
choices: [
{
index: 0,
message: {
role: "assistant",
content: "I will look that up.",
tool_calls: [
{
id: "call_1",
type: "function",
function: { name: "memory_search", arguments: '{"query":"foo"}' },
},
{
id: "call_2",
type: "function",
function: { name: "memory_save", arguments: '{"key":"k","value":"v"}' },
},
],
},
finish_reason: "tool_calls",
},
],
} as UnknownRecord;
const toolCalls: ToolCall[] = [
{ id: "call_1", name: "memory_search", arguments: { query: "foo" } },
{ id: "call_2", name: "memory_save", arguments: { key: "k", value: "v" } },
];
const results: ExecutedToolResult[] = [
{ id: "call_1", name: "memory_search", result: { hits: ["a", "b"] }, replayed: false },
{ id: "call_2", name: "memory_save", result: { ok: true }, replayed: true },
];
return {
sourceBody,
previousResponse,
toolCalls,
results,
sourceFormat: "openai",
maxResultBytes: MAX_RESULT_BYTES_PER_TOOL,
...overrides,
} as BuildFollowUpTranscriptInput & {
sourceBody: UnknownRecord;
previousResponse: UnknownRecord;
toolCalls: ToolCall[];
results: ExecutedToolResult[];
};
}
test("OpenAI: messages array required — Responses-only body throws", () => {
const responsesBody = {
model: "gpt-4o",
input: [{ role: "user", content: "hi" }],
} as UnknownRecord;
assert.throws(
() => buildFollowUpSourceBody(openaiInput({ sourceBody: responsesBody })),
(err: unknown) => err instanceof Error && err.message === MESSAGES_REQUIRED
);
assert.throws(
() =>
buildFollowUpSourceBody(
openaiInput({ sourceBody: { model: "gpt-4o", messages: "not-an-array" } as UnknownRecord })
),
(err: unknown) => err instanceof Error && err.message === MESSAGES_REQUIRED
);
});
test("OpenAI: calls/results length, duplicates and ID set mismatch all fail closed", () => {
const missingResult = openaiInput({
toolCalls: [
{ id: "call_1", name: "memory_search", arguments: { query: "foo" } },
{ id: "call_9", name: "client_tool", arguments: {} },
],
});
assert.throws(() => buildFollowUpSourceBody(missingResult), /matching result/);
const extraResult = openaiInput({
results: [
{ id: "call_1", name: "memory_search", result: { hits: ["a"] }, replayed: false },
{ id: "call_2", name: "memory_save", result: { ok: true }, replayed: false },
{ id: "call_9", name: "ghost", result: "no call", replayed: false },
],
});
assert.throws(() => buildFollowUpSourceBody(extraResult), /same length/);
const duplicateCallId = openaiInput({
toolCalls: [
{ id: "call_1", name: "memory_search", arguments: {} },
{ id: "call_1", name: "memory_save", arguments: {} },
],
});
assert.throws(() => buildFollowUpSourceBody(duplicateCallId), /unique tool call ids/);
const duplicateResultId = openaiInput({
results: [
{ id: "call_1", name: "memory_search", result: "a", replayed: false },
{ id: "call_1", name: "memory_search", result: "b", replayed: false },
],
});
assert.throws(() => buildFollowUpSourceBody(duplicateResultId), /unique tool result ids/);
});
test("OpenAI: appends assistant turn and tool messages, preserves tools/tool_choice/model", () => {
const input = openaiInput();
const sourceBodySnapshot = JSON.parse(JSON.stringify(input.sourceBody));
const responseSnapshot = JSON.parse(JSON.stringify(input.previousResponse));
const out = buildFollowUpSourceBody(input);
assert.notStrictEqual(out, input.sourceBody);
assert.deepEqual(input.sourceBody, sourceBodySnapshot);
assert.deepEqual(input.previousResponse, responseSnapshot);
assert.strictEqual(out.model, "gpt-4o");
assert.deepEqual(out.tools, input.sourceBody.tools);
assert.deepEqual(out.tool_choice, "auto");
assert.strictEqual(out.stream, false);
const messages = out.messages as UnknownRecord[];
assert.strictEqual(messages.length, 5);
assert.deepEqual(messages.slice(0, 2), input.sourceBody.messages);
const assistant = messages[2];
assert.strictEqual(assistant.role, "assistant");
assert.strictEqual(assistant.content, "I will look that up.");
assert.deepEqual(assistant.tool_calls, input.previousResponse.choices[0].message.tool_calls);
for (let i = 0; i < input.results.length; i++) {
const toolMessage = messages[3 + i];
assert.strictEqual(toolMessage.role, "tool");
assert.strictEqual(toolMessage.tool_call_id, input.results[i].id);
assert.strictEqual(toolMessage.content, JSON.stringify(input.results[i].result));
}
});
test("OpenAI: reconstructs assistant tool_calls from parsed calls when response carries none", () => {
const input = openaiInput();
(input.previousResponse.choices[0].message as UnknownRecord).tool_calls = undefined;
const out = buildFollowUpSourceBody(input);
const assistant = (out.messages as UnknownRecord[])[2];
assert.deepEqual(assistant.tool_calls, [
{
id: "call_1",
type: "function",
function: { name: "memory_search", arguments: '{"query":"foo"}' },
},
{
id: "call_2",
type: "function",
function: { name: "memory_save", arguments: '{"key":"k","value":"v"}' },
},
]);
});
test("OpenAI: bounds each tool result to 32768 UTF-8 bytes and total to 65536", () => {
const big = "x".repeat(40_000);
const input = openaiInput({
results: [
{ id: "call_1", name: "memory_search", result: { payload: big }, replayed: false },
{ id: "call_2", name: "memory_save", result: { payload: big }, replayed: false },
],
});
const out = buildFollowUpSourceBody(input);
const messages = out.messages as UnknownRecord[];
const firstContent = String(messages[3].content);
const secondContent = String(messages[4].content);
const firstBytes = Buffer.byteLength(firstContent, "utf8");
const secondBytes = Buffer.byteLength(secondContent, "utf8");
assert.strictEqual(MAX_RESULT_BYTES_PER_TOOL, 32_768);
assert.strictEqual(MAX_RESULT_BYTES_TOTAL, 65_536);
assert.ok(firstBytes <= MAX_RESULT_BYTES_PER_TOOL, `first ${firstBytes}`);
assert.ok(secondBytes <= MAX_RESULT_BYTES_PER_TOOL, `second ${secondBytes}`);
assert.ok(firstBytes + secondBytes <= MAX_RESULT_BYTES_TOTAL, "total byte bound");
assert.ok(firstContent.includes("[TRUNCATED"));
assert.ok(secondContent.includes("[TRUNCATED"));
});
test("OpenAI: maxTotalResultBytes=20 exhausts budget, second tool message is empty and total stays <=20", () => {
const input = openaiInput({
results: [
{ id: "call_1", name: "memory_search", result: "a".repeat(240), replayed: false },
{ id: "call_2", name: "memory_save", result: "z".repeat(240), replayed: false },
],
maxTotalResultBytes: 20,
});
const out = buildFollowUpSourceBody(input);
const messages = out.messages as UnknownRecord[];
const firstBytes = Buffer.byteLength(String(messages[3].content), "utf8");
const secondContent = messages[4].content;
assert.ok(firstBytes <= 20, `first ${firstBytes}`);
assert.strictEqual(secondContent, "");
assert.ok(firstBytes + Buffer.byteLength(String(secondContent), "utf8") <= 20);
assert.ok(String(messages[3].content).includes("[TRUNCATED"));
});
// ─── serializeBoundedToolResult ──────────────────────────────────────────────
function assertValidUtf8(text: string): void {
assert.strictEqual(Buffer.from(text, "utf8").toString("utf8"), text);
}
function hasLoneSurrogate(text: string): boolean {
for (let i = 0; i < text.length; i++) {
const code = text.charCodeAt(i);
if (code >= 0xd800 && code <= 0xdbff) {
const next = text.charCodeAt(i + 1);
if (next < 0xdc00 || next > 0xdfff) return true;
} else if (code >= 0xdc00 && code <= 0xdfff) {
const prev = text.charCodeAt(i - 1);
if (prev < 0xd800 || prev > 0xdbff) return true;
}
}
return false;
}
test("serializeBoundedToolResult: CJK + astral cut lands on code-point boundaries", () => {
const raw = "你好🌍".repeat(9);
const serialized = JSON.stringify(raw);
const originalBytes = Buffer.byteLength(serialized, "utf8");
const maxBytes = 50;
const bounded = serializeBoundedToolResult(raw, maxBytes);
assert.strictEqual(bounded.originalBytes, originalBytes);
assert.ok(bounded.truncated);
assert.ok(
Buffer.byteLength(bounded.text, "utf8") <= maxBytes,
`byte bound: ${Buffer.byteLength(bounded.text, "utf8")} <= ${maxBytes}`
);
assert.ok(bounded.text.includes("[TRUNCATED"), "marker present");
const match = bounded.text.match(/\[TRUNCATED (\d+) BYTES BY OMNIROUTE\]/);
assert.ok(match, "marker format valid");
const droppedBytes = Number(match![1]);
assert.ok(droppedBytes > 0, "dropped bytes > 0");
assert.ok(droppedBytes <= originalBytes, "dropped <= original");
assertValidUtf8(bounded.text);
assert.ok(!hasLoneSurrogate(bounded.text), "no orphan surrogate");
});
test("serializeBoundedToolResult: marker-only UTF-8-safe prefix when maxBytes <= markerBytes, empty at 0", () => {
const text = "你好世界🌍🌎";
const markerOnly = serializeBoundedToolResult(text, 20);
assert.ok(markerOnly.truncated);
assert.strictEqual(markerOnly.text, "[TRUNCATED 22 BYTES ");
assert.strictEqual(Buffer.byteLength(markerOnly.text, "utf8"), 20);
assertValidUtf8(markerOnly.text);
const empty = serializeBoundedToolResult(text, 0);
assert.strictEqual(empty.text, "");
assertValidUtf8(empty.text);
});
test("serializeBoundedToolResult: unfits pass through untruncated with byte counts", () => {
const bounded = serializeBoundedToolResult({ hits: ["a"] }, 1024);
assert.strictEqual(bounded.truncated, false);
assert.strictEqual(bounded.text, '{"hits":["a"]}');
assert.strictEqual(bounded.originalBytes, Buffer.byteLength('{"hits":["a"]}', "utf8"));
});
test("serializeBoundedToolResult: projects undefined/Error/bigint and rejects non-JSON values", () => {
assert.strictEqual(serializeBoundedToolResult(undefined, 1024).text, "null");
assert.strictEqual(serializeBoundedToolResult(new Error("x"), 1024).text, '{"error":"x"}');
assert.strictEqual(serializeBoundedToolResult(10n, 1024).text, '"10"');
assert.strictEqual(serializeBoundedToolResult(() => undefined, 1024).text, NON_SERIALIZABLE_JSON);
assert.strictEqual(serializeBoundedToolResult(Symbol("s"), 1024).text, NON_SERIALIZABLE_JSON);
const cyclic: UnknownRecord = {};
cyclic.self = cyclic;
assert.strictEqual(serializeBoundedToolResult(cyclic, 1024).text, NON_SERIALIZABLE_JSON);
assert.strictEqual(serializeBoundedToolResult({ a: 10n }, 1024).text, NON_SERIALIZABLE_JSON);
const boundedResult: BoundedToolResult = serializeBoundedToolResult(10n, 1024);
assert.deepEqual(Object.keys(boundedResult).sort(), ["originalBytes", "text", "truncated"]);
});
// ─── Task 4: Budget validation (Req 1) ─────────────────────────────────────
test("serializeBoundedToolResult: NaN budget throws RangeError", () => {
assert.throws(() => serializeBoundedToolResult("hello", NaN), {
name: "RangeError",
message: /maxBytes must be a non-negative finite integer/,
});
});
test("serializeBoundedToolResult: Infinity budget throws RangeError", () => {
assert.throws(() => serializeBoundedToolResult("hello", Infinity), {
name: "RangeError",
message: /maxBytes must be a non-negative finite integer/,
});
});
test("serializeBoundedToolResult: -Infinity budget throws RangeError", () => {
assert.throws(() => serializeBoundedToolResult("hello", -Infinity), {
name: "RangeError",
message: /maxBytes must be a non-negative finite integer/,
});
});
test("serializeBoundedToolResult: negative budget throws RangeError", () => {
assert.throws(() => serializeBoundedToolResult("hello", -1), {
name: "RangeError",
message: /maxBytes must be a non-negative finite integer/,
});
});
test("serializeBoundedToolResult: non-integer budget throws RangeError", () => {
assert.throws(() => serializeBoundedToolResult("hello", 1.5), {
name: "RangeError",
message: /maxBytes must be a non-negative finite integer/,
});
});
test("buildFollowUpSourceBody: NaN maxResultBytes throws RangeError", () => {
const input = openaiInput({ maxResultBytes: NaN });
assert.throws(() => buildFollowUpSourceBody(input), {
name: "RangeError",
message: /must be a non-negative finite integer/,
});
});
test("buildFollowUpSourceBody: negative maxTotalResultBytes throws RangeError", () => {
const input = openaiInput({ maxTotalResultBytes: -1 });
assert.throws(() => buildFollowUpSourceBody(input), {
name: "RangeError",
message: /must be a non-negative finite integer/,
});
});
// ─── Task 4: sourceFormat validation (Req 2) ───────────────────────────────
test("buildFollowUpSourceBody: invalid sourceFormat 'anthropic' fails closed", () => {
const input = openaiInput({ sourceFormat: "anthropic" } as unknown as {
sourceFormat: "openai" | "claude";
});
assert.throws(() => buildFollowUpSourceBody(input), /sourceFormat must be "openai" or "claude"/);
});
test("buildFollowUpSourceBody: empty sourceFormat fails closed", () => {
const input = openaiInput({ sourceFormat: "" } as unknown as {
sourceFormat: "openai" | "claude";
});
assert.throws(() => buildFollowUpSourceBody(input), /sourceFormat must be "openai" or "claude"/);
});
// ─── Task 4: calls/results name match (Req 3) ──────────────────────────────
test("calls/results name mismatch on same ID fails closed", () => {
const input = openaiInput({
toolCalls: [
{ id: "call_1", name: "memory_search", arguments: { query: "foo" } },
{ id: "call_2", name: "memory_save", arguments: { key: "k", value: "v" } },
],
results: [
{ id: "call_1", name: "WRONG_NAME", result: { hits: ["a"] }, replayed: false },
{ id: "call_2", name: "memory_save", result: { ok: true }, replayed: false },
],
});
assert.throws(
() => buildFollowUpSourceBody(input),
/result name .* does not match tool call name/
);
});
// ─── Task 4: OpenAI previous response tool_call match (Req 4) ───────────────
test("OpenAI: partial tool_calls in previous response fails closed (missing call)", () => {
const input = openaiInput();
// Only include call_1 in the previous response, omit call_2
(input.previousResponse.choices[0].message as UnknownRecord).tool_calls = [
{
id: "call_1",
type: "function",
function: { name: "memory_search", arguments: '{"query":"foo"}' },
},
];
assert.throws(
() => buildFollowUpSourceBody(input),
/previous response tool_calls must contain exactly one match per call ID/
);
});
test("OpenAI: duplicate tool_calls for same ID in previous response fails closed", () => {
const input = openaiInput();
// Duplicate call_1 in the previous response
(input.previousResponse.choices[0].message as UnknownRecord).tool_calls = [
{
id: "call_1",
type: "function",
function: { name: "memory_search", arguments: '{"query":"foo"}' },
},
{
id: "call_1",
type: "function",
function: { name: "memory_search", arguments: '{"query":"foo"}' },
},
{
id: "call_2",
type: "function",
function: { name: "memory_save", arguments: '{"key":"k","value":"v"}' },
},
];
assert.throws(
() => buildFollowUpSourceBody(input),
/previous response tool_calls must contain exactly one match per call ID/
);
});
test("OpenAI: mismatched name in previous response tool_call fails closed", () => {
const input = openaiInput();
// call_1 has a different name in the previous response
(input.previousResponse.choices[0].message as UnknownRecord).tool_calls = [
{
id: "call_1",
type: "function",
function: { name: "WRONG_NAME", arguments: '{"query":"foo"}' },
},
{
id: "call_2",
type: "function",
function: { name: "memory_save", arguments: '{"key":"k","value":"v"}' },
},
];
assert.throws(
() => buildFollowUpSourceBody(input),
/previous response tool_call .* name .* does not match/
);
});
// ─── Task 4: Claude previousResponse tool_use match (Req 5) ─────────────────
test("Claude: partial tool_use blocks in previous response fails closed", () => {
const input = claudeInput();
// Only include toolu_1, omit toolu_2
(input.previousResponse as UnknownRecord).content = [
{ type: "text", text: "Let me look." },
{ type: "tool_use", id: "toolu_1", name: "memory_search", input: { query: "foo" } },
];
assert.throws(
() => buildFollowUpSourceBody(input),
/previous response content must contain exactly one tool_use match per call ID/
);
});
test("Claude: duplicate tool_use blocks for same ID in previous response fails closed", () => {
const input = claudeInput();
// Duplicate toolu_1
(input.previousResponse as UnknownRecord).content = [
{ type: "text", text: "Let me look." },
{ type: "tool_use", id: "toolu_1", name: "memory_search", input: { query: "foo" } },
{ type: "tool_use", id: "toolu_1", name: "memory_search", input: { query: "foo" } },
{ type: "tool_use", id: "toolu_2", name: "memory_save", input: { key: "k" } },
];
assert.throws(
() => buildFollowUpSourceBody(input),
/previous response content must contain exactly one tool_use match per call ID/
);
});
test("Claude: mismatched name in previous response tool_use fails closed", () => {
const input = claudeInput();
// toolu_1 has a different name
(input.previousResponse as UnknownRecord).content = [
{ type: "text", text: "Let me look." },
{ type: "tool_use", id: "toolu_1", name: "WRONG_NAME", input: { query: "foo" } },
{ type: "tool_use", id: "toolu_2", name: "memory_save", input: { key: "k" } },
];
assert.throws(
() => buildFollowUpSourceBody(input),
/previous response tool_use .* name .* does not match/
);
});
// ─── Claude Messages ─────────────────────────────────────────────────────────
function claudeInput(overrides: Partial<BuildFollowUpTranscriptInput> = {}) {
const sourceBody = {
model: "claude-sonnet-4-5",
max_tokens: 1024,
stream: true,
messages: [{ role: "user", content: "remember foo" }],
tools: [
{
name: "memory_search",
description: "search memory",
input_schema: { type: "object", properties: { query: { type: "string" } } },
},
],
} as UnknownRecord;
const previousResponse = {
id: "msg_1",
type: "message",
role: "assistant",
content: [
{ type: "text", text: "Let me look." },
{ type: "tool_use", id: "toolu_1", name: "memory_search", input: { query: "foo" } },
{ type: "tool_use", id: "toolu_2", name: "memory_save", input: { key: "k" } },
],
stop_reason: "tool_use",
} as UnknownRecord;
const toolCalls: ToolCall[] = [
{ id: "toolu_1", name: "memory_search", arguments: { query: "foo" } },
{ id: "toolu_2", name: "memory_save", arguments: { key: "k" } },
];
const results: ExecutedToolResult[] = [
{ id: "toolu_1", name: "memory_search", result: { hits: ["a"] }, replayed: false },
{ id: "toolu_2", name: "memory_save", result: { stored: true }, replayed: true },
];
return {
sourceBody,
previousResponse,
toolCalls,
results,
sourceFormat: "claude",
maxResultBytes: MAX_RESULT_BYTES_PER_TOOL,
...overrides,
} as BuildFollowUpTranscriptInput & {
sourceBody: UnknownRecord;
previousResponse: UnknownRecord;
toolCalls: ToolCall[];
results: ExecutedToolResult[];
};
}
test("Claude: appends assistant tool_use turn, then a separate user tool_result message", () => {
const input = claudeInput();
const sourceBodySnapshot = JSON.parse(JSON.stringify(input.sourceBody));
const out = buildFollowUpSourceBody(input);
assert.notStrictEqual(out, input.sourceBody);
assert.deepEqual(input.sourceBody, sourceBodySnapshot);
assert.strictEqual(out.stream, true, "stream is not forced for Claude bodies");
const messages = out.messages as UnknownRecord[];
assert.strictEqual(messages.length, 3);
assert.deepEqual(messages[0], input.sourceBody.messages[0]);
const assistant = messages[1];
assert.strictEqual(assistant.role, "assistant");
const assistantBlocks = assistant.content as UnknownRecord[];
assert.strictEqual(assistantBlocks.length, 3, "text + 2 tool_use blocks preserved");
assert.strictEqual(assistantBlocks[0].type, "text");
assert.strictEqual(assistantBlocks[0].text, "Let me look.");
assert.ok(assistantBlocks.slice(1).every((block) => block.type === "tool_use"));
assert.deepEqual(
assistantBlocks.slice(1),
input.previousResponse.content.filter((block: UnknownRecord) => block.type === "tool_use")
);
const user = messages[2];
assert.strictEqual(user.role, "user");
const resultBlocks = user.content as UnknownRecord[];
assert.strictEqual(resultBlocks.length, 2);
assert.ok(resultBlocks.every((block) => block.type === "tool_result"));
assert.deepEqual(
resultBlocks.map((block) => block.tool_use_id),
["toolu_1", "toolu_2"]
);
assert.strictEqual(resultBlocks[0].content, JSON.stringify({ hits: ["a"] }));
assert.strictEqual(resultBlocks[1].content, JSON.stringify({ stored: true }));
assert.deepEqual(out.tools, input.sourceBody.tools);
});
// ─── Task 4 Fix R1: Claude text/thinking block preservation ─────────────────
test("Claude: assistant content preserves original text blocks in original order", () => {
const input = claudeInput();
const out = buildFollowUpSourceBody(input);
const messages = out.messages as UnknownRecord[];
const assistant = messages[1];
const assistantBlocks = assistant.content as UnknownRecord[];
// Must include the text block (index 0) AND the two tool_use blocks
assert.strictEqual(assistantBlocks.length, 3, "should have text + 2 tool_use blocks");
assert.strictEqual(assistantBlocks[0].type, "text");
assert.strictEqual(assistantBlocks[0].text, "Let me look.");
assert.strictEqual(assistantBlocks[1].type, "tool_use");
assert.strictEqual(assistantBlocks[1].id, "toolu_1");
assert.strictEqual(assistantBlocks[2].type, "tool_use");
assert.strictEqual(assistantBlocks[2].id, "toolu_2");
});
test("Claude: assistant content preserves thinking blocks alongside tool_use", () => {
const input = claudeInput();
(input.previousResponse as UnknownRecord).content = [
{ type: "thinking", thinking: "Let me reason about this.", signature: "sig_1" },
{ type: "text", text: "I'll search now." },
{ type: "tool_use", id: "toolu_1", name: "memory_search", input: { query: "foo" } },
{ type: "tool_use", id: "toolu_2", name: "memory_save", input: { key: "k" } },
];
const out = buildFollowUpSourceBody(input);
const messages = out.messages as UnknownRecord[];
const assistantBlocks = messages[1].content as UnknownRecord[];
assert.strictEqual(assistantBlocks.length, 4, "should have thinking + text + 2 tool_use");
assert.strictEqual(assistantBlocks[0].type, "thinking");
assert.strictEqual(assistantBlocks[0].thinking, "Let me reason about this.");
assert.strictEqual(assistantBlocks[1].type, "text");
assert.strictEqual(assistantBlocks[1].text, "I'll search now.");
assert.strictEqual(assistantBlocks[2].type, "tool_use");
assert.strictEqual(assistantBlocks[3].type, "tool_use");
});
test("Claude: assistant content filters unmatched tool_use but keeps text", () => {
const input = claudeInput();
(input.previousResponse as UnknownRecord).content = [
{ type: "text", text: "Looking..." },
{ type: "tool_use", id: "toolu_1", name: "memory_search", input: { query: "foo" } },
{ type: "tool_use", id: "toolu_UNMATCHED", name: "other_tool", input: {} },
{ type: "tool_use", id: "toolu_2", name: "memory_save", input: { key: "k" } },
];
const out = buildFollowUpSourceBody(input);
const messages = out.messages as UnknownRecord[];
const assistantBlocks = messages[1].content as UnknownRecord[];
// text preserved, unmatched tool_use filtered out, matched tool_use kept
assert.strictEqual(assistantBlocks.length, 3);
assert.strictEqual(assistantBlocks[0].type, "text");
assert.strictEqual(assistantBlocks[1].id, "toolu_1");
assert.strictEqual(assistantBlocks[2].id, "toolu_2");
});
test("Claude: strips text after first tool_use but preserves later thinking blocks", () => {
const input = claudeInput();
(input.previousResponse as UnknownRecord).content = [
{ type: "text", text: "before" },
{ type: "tool_use", id: "toolu_1", name: "memory_search", input: { query: "foo" } },
{ type: "text", text: "after — must be stripped" },
{ type: "thinking", thinking: "signed thought", signature: "sig_after" },
{ type: "tool_use", id: "toolu_2", name: "memory_save", input: { key: "k" } },
];
const out = buildFollowUpSourceBody(input);
const assistantBlocks = (out.messages as UnknownRecord[])[1].content as UnknownRecord[];
assert.deepEqual(
assistantBlocks.map((block) => block.type),
["text", "tool_use", "thinking", "tool_use"]
);
assert.equal(
assistantBlocks.some(
(block) => block.type === "text" && block.text === "after — must be stripped"
),
false
);
});
test("Claude: mismatched IDs fail closed even when response content would filter them", () => {
const input = claudeInput({
toolCalls: [
{ id: "toolu_1", name: "memory_search", arguments: { query: "foo" } },
{ id: "toolu_9", name: "client_tool", arguments: {} },
],
});
assert.throws(() => buildFollowUpSourceBody(input), /matching result/);
});

322
tests/unit/non-streaming-client-translate.test.ts Normal file
View File

@@ -0,0 +1,322 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import {
translateNonStreamingClientResponse,
type NonStreamingClientTranslateInput,
} from "../../open-sse/handlers/chatCore/nonStreamingClientTranslate.ts";
import { FORMATS } from "../../open-sse/translator/formats.ts";
import {
buildAssistantMessageCacheKey,
clearReasoningCacheAll,
lookupReasoning,
} from "../../open-sse/services/reasoningCache.ts";
import { invalidateBufferTokensCache } from "../../open-sse/utils/usageTracking.ts";
/* ── helpers ─────────────────────────────────────────────────────────────── */
function baseInput(
overrides: Partial<NonStreamingClientTranslateInput> = {}
): NonStreamingClientTranslateInput {
return {
responseBody: {
id: "chatcmpl-test",
object: "chat.completion",
choices: [
{
index: 0,
message: { role: "assistant", content: "Hello!" },
finish_reason: "stop",
},
],
usage: { prompt_tokens: 10, completion_tokens: 5, total_tokens: 15 },
},
responsePayloadFormat: "openai",
clientResponseFormat: "openai",
sourceFormat: "openai",
provider: "openai",
model: "gpt-4o",
requestBody: { messages: [{ role: "user", content: "hi" }] },
responseToolNameMap: null,
requestToolIdentityMap: null,
reasoningCacheScope: null,
clientHeaders: null,
isClaudeCodeCompatible: false,
phase: "final",
...overrides,
};
}
/* ── characterization tests ──────────────────────────────────────────────── */
test("basic translate: same-format passthrough returns responseBody", () => {
const input = baseInput();
const result = translateNonStreamingClientResponse(input);
assert.equal(result.response.choices[0].message.content, "Hello!");
assert.ok(result.responseForMemoryExtraction);
});
test("translate from claude to openai format", () => {
const input = baseInput({
responseBody: {
id: "msg-123",
content: [{ type: "text", text: "Hi there" }],
stop_reason: "end_turn",
usage: { input_tokens: 10, output_tokens: 5 },
},
responsePayloadFormat: "claude",
clientResponseFormat: "openai",
sourceFormat: "claude",
provider: "anthropic",
model: "claude-sonnet-4-20250514",
});
const result = translateNonStreamingClientResponse(input);
const msg = result.response.choices?.[0]?.message;
assert.ok(msg, "should have choices[0].message");
assert.equal((msg as { content: string }).content, "Hi there");
});
test("claude source strips markdown code fence", () => {
const input = baseInput({
responseBody: {
id: "msg-123",
content: [
{
type: "text",
text: '```json\n{"key": "value"}\n```',
},
],
stop_reason: "end_turn",
usage: { input_tokens: 10, output_tokens: 5 },
},
responsePayloadFormat: "claude",
clientResponseFormat: "openai",
sourceFormat: "claude",
provider: "anthropic",
model: "claude-sonnet-4-20250514",
});
const result = translateNonStreamingClientResponse(input);
const content = result.response.choices?.[0]?.message?.content;
assert.ok(typeof content === "string");
// After stripping, the content should not have the markdown fence wrapper
assert.ok(!content.startsWith("```json"), "markdown fence should be stripped");
});
test("normalizeOpenAIToolFinishReasons: tool_calls present → finish_reason tool_calls", () => {
const input = baseInput({
responseBody: {
id: "chatcmpl-test",
choices: [
{
index: 0,
message: {
role: "assistant",
content: null,
tool_calls: [
{
id: "call_1",
type: "function",
function: { name: "get_weather", arguments: "{}" },
},
],
},
finish_reason: "stop",
},
],
usage: { prompt_tokens: 10, completion_tokens: 5, total_tokens: 15 },
},
});
const result = translateNonStreamingClientResponse(input);
assert.equal(result.response.choices[0].finish_reason, "tool_calls");
});
test("reasoning replay: no-tool history comes from historyMessages, not requestBody.input", () => {
clearReasoningCacheAll();
const scope = "api-key:test:s...6a";
const historyMessages = [{ role: "user", content: "hi from translatedBody" }];
const assistantMessage = {
role: "assistant",
content: "thinking result",
reasoning_content: "let me think...",
};
const input = baseInput({
responseBody: {
id: "chatcmpl-test",
choices: [
{
index: 0,
message: assistantMessage,
finish_reason: "stop",
},
],
usage: { prompt_tokens: 10, completion_tokens: 5, total_tokens: 15 },
},
// Responses-shaped finalBody: input, no messages. Parent used translatedBody.messages.
requestBody: { input: [{ role: "user", content: "wrong body" }] },
historyMessages,
provider: "deepseek",
model: "deepseek-v4-pro",
reasoningCacheScope: scope,
phase: "intermediate",
});
const result = translateNonStreamingClientResponse(input);
assert.ok(result.response);
const cacheKey = buildAssistantMessageCacheKey(
scope,
[...historyMessages, assistantMessage],
historyMessages.length
);
assert.equal(
lookupReasoning(cacheKey),
"let me think...",
"must cache against translatedBody.messages, not finalBody.input"
);
});
test("phase=final applies client usage buffer", () => {
// Gemini format skips OpenAI/Responses sanitize, so extra usage fields
// only disappear if applyClientUsageBuffer → filterUsageForFormat runs.
const input = baseInput({
phase: "final",
clientResponseFormat: FORMATS.GEMINI,
responsePayloadFormat: FORMATS.GEMINI,
sourceFormat: FORMATS.GEMINI,
responseBody: {
id: "chatcmpl-test",
choices: [
{
index: 0,
message: { role: "assistant", content: "Hello!" },
finish_reason: "stop",
},
],
usage: {
prompt_tokens: 10,
completion_tokens: 5,
total_tokens: 15,
x_provider_extra: 99,
},
},
});
const result = translateNonStreamingClientResponse(input);
const usage = (result.response as { usage: Record<string, unknown> }).usage;
assert.equal(usage.x_provider_extra, undefined, "final phase must filter extra usage fields");
assert.equal(
usage.prompt_tokens,
undefined,
"final Gemini filter must drop OpenAI-shaped prompt_tokens"
);
});
test("phase=intermediate skips applyClientUsageBuffer", () => {
const input = baseInput({
phase: "intermediate",
clientResponseFormat: FORMATS.GEMINI,
responsePayloadFormat: FORMATS.GEMINI,
sourceFormat: FORMATS.GEMINI,
responseBody: {
id: "chatcmpl-test",
choices: [
{
index: 0,
message: { role: "assistant", content: "partial" },
finish_reason: null,
},
],
usage: {
prompt_tokens: 10,
completion_tokens: 5,
total_tokens: 15,
x_provider_extra: 99,
},
},
});
const result = translateNonStreamingClientResponse(input);
const usage = (result.response as { usage: Record<string, unknown> }).usage;
assert.equal(
usage.x_provider_extra,
99,
"intermediate must keep raw extra usage fields (buffer not applied)"
);
assert.equal(usage.prompt_tokens, 10);
});
test("Responses API format: sanitizeResponsesApiResponse is applied", () => {
const input = baseInput({
clientResponseFormat: FORMATS.OPENAI_RESPONSES,
responseBody: {
id: "resp_123",
object: "response",
output: [
{
type: "function_call",
name: "ns__get_weather",
arguments: "{}",
},
],
usage: { input_tokens: 10, output_tokens: 5 },
illegal_top_level: "drop-me",
},
requestToolIdentityMap: new Map([
["ns__get_weather", { namespace: "ns", name: "get_weather" }],
]),
});
const result = translateNonStreamingClientResponse(input);
assert.equal(result.response.object, "response");
assert.equal(result.response.illegal_top_level, undefined, "sanitizer must drop illegal fields");
const output = result.response.output as Array<Record<string, unknown>>;
assert.equal(output[0]?.type, "function_call");
assert.equal(output[0]?.namespace, "ns", "#7936 restore namespace");
assert.equal(output[0]?.name, "get_weather", "#7936 restore original name");
});
test("empty content response: passthrough without crash", () => {
const input = baseInput({
responseBody: {},
});
const result = translateNonStreamingClientResponse(input);
assert.ok(result.response);
assert.ok(result.responseForMemoryExtraction);
});
test("isClaudeCodeCompatible preserves context budget usage", () => {
const saved = process.env.USAGE_TOKEN_BUFFER;
process.env.USAGE_TOKEN_BUFFER = "2000";
invalidateBufferTokensCache();
try {
const input = baseInput({
isClaudeCodeCompatible: true,
clientResponseFormat: FORMATS.OPENAI,
phase: "final",
responseBody: {
id: "chatcmpl-test",
object: "chat.completion",
choices: [
{
index: 0,
message: { role: "assistant", content: "test" },
finish_reason: "stop",
},
],
usage: {
prompt_tokens: 100,
completion_tokens: 50,
total_tokens: 150,
},
},
});
const result = translateNonStreamingClientResponse(input);
const usage = (result.response as { usage: Record<string, unknown> }).usage;
assert.equal(
usage.prompt_tokens,
2100,
"Claude Code path must fold context_budget_prompt_tokens (100+2000) into visible prompt_tokens"
);
assert.equal(usage.total_tokens, 2150);
assert.equal("context_budget_prompt_tokens" in usage, false);
} finally {
if (saved === undefined) delete process.env.USAGE_TOKEN_BUFFER;
else process.env.USAGE_TOKEN_BUFFER = saved;
invalidateBufferTokensCache();
}
});

219
tests/unit/non-streaming-finalization.test.ts Normal file
View File

@@ -0,0 +1,219 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import type {
ChatCoreErrorResult,
ProviderLegReceipt,
ProviderLegUsage,
ServerOwnedToolLoopResult,
} from "../../src/lib/skills/toolLoopTypes.ts";
import {
buildNonStreamingFinalizationPlan,
finalizeNonStreamingRequest,
finalizeToolLoopError,
type NonStreamingFinalizationDeps,
} from "../../open-sse/handlers/chatCore/nonStreamingFinalization.ts";
function receipt(index: number, overrides: Partial<ProviderLegReceipt> = {}): ProviderLegReceipt {
return {
index,
connectionId: "conn-1",
provider: "openai",
model: "gpt-4o",
startedAt: "2026-01-01T00:00:00.000Z",
endedAt: "2026-01-01T00:00:01.000Z",
latencyMs: 100,
httpStatus: 200,
errorType: null,
usage: { prompt_tokens: 10, completion_tokens: 5, total_tokens: 15 },
serviceTier: null,
computedCostUsd: 0.01,
toolCalls: [],
termination: "completed",
clientVisible: true,
...overrides,
};
}
function usage(overrides: Partial<ProviderLegUsage> = {}): ProviderLegUsage {
return {
prompt_tokens: 100,
completion_tokens: 20,
total_tokens: 120,
cache_read_input_tokens: 10,
reasoning_tokens: 5,
...overrides,
};
}
function errorResult(): ChatCoreErrorResult {
return {
success: false,
status: 429,
response: new Response(JSON.stringify({ error: { message: "rate limited" } }), {
status: 429,
}),
error: "rate limited",
errorCode: "rate_limited",
errorType: "rate_limit_error",
retryAfterMs: 1000,
};
}
function spyDeps(): NonStreamingFinalizationDeps & { calls: Record<string, number> } {
const calls = {
writeUsage: 0,
writeCost: 0,
scheduleQuota: 0,
writeAttempt: 0,
finalizePending: 0,
};
return {
calls,
writeUsage: () => {
calls.writeUsage += 1;
},
writeCost: () => {
calls.writeCost += 1;
},
scheduleQuota: () => {
calls.scheduleQuota += 1;
},
writeAttempt: () => {
calls.writeAttempt += 1;
},
finalizePending: () => {
calls.finalizePending += 1;
},
};
}
test("failure plan maps loop aggregate and 429 error", () => {
const loop: ServerOwnedToolLoopResult = {
kind: "error",
errorResult: errorResult(),
cumulativeUsage: usage(),
totalCostUsd: 0.03,
receipts: [
receipt(0),
receipt(1, {
httpStatus: 429,
errorType: "rate_limit_error",
termination: "provider_error",
computedCostUsd: 0.02,
}),
],
followUps: 1,
termination: "provider_error",
};
const plan = buildNonStreamingFinalizationPlan(loop);
assert.equal(plan.kind, "failure");
if (plan.kind !== "failure") return;
assert.equal(plan.error.status, 429);
assert.equal(plan.error.errorCode, "rate_limited");
assert.deepEqual(plan.usage, usage());
assert.equal(plan.totalCostUsd, 0.03);
assert.equal(plan.receiptCount, 2);
});
test("success plan maps loop usage and receipt count", () => {
const loop: ServerOwnedToolLoopResult = {
kind: "ok",
response: { choices: [{ message: { content: "done" }, finish_reason: "stop" }] },
cumulativeUsage: usage({ prompt_tokens: 11, completion_tokens: 7, total_tokens: 18 }),
totalCostUsd: 0.02,
receipts: [receipt(0)],
followUps: 0,
termination: "completed",
};
const plan = buildNonStreamingFinalizationPlan(loop);
assert.equal(plan.kind, "success");
if (plan.kind !== "success") return;
assert.equal(plan.usage?.prompt_tokens, 11);
assert.equal(plan.totalCostUsd, 0.02);
assert.equal(plan.receiptCount, 1);
});
test("failure finalizer writes usage/cost/attempt/pending once and skips quota", async () => {
const loop: ServerOwnedToolLoopResult = {
kind: "error",
errorResult: errorResult(),
cumulativeUsage: usage(),
totalCostUsd: 0.03,
receipts: [receipt(0), receipt(1, { httpStatus: 429 })],
followUps: 1,
termination: "provider_error",
};
const deps = spyDeps();
await finalizeNonStreamingRequest(buildNonStreamingFinalizationPlan(loop), deps);
assert.equal(deps.calls.writeUsage, 1);
assert.equal(deps.calls.writeCost, 1);
assert.equal(deps.calls.scheduleQuota, 0);
assert.equal(deps.calls.writeAttempt, 1);
assert.equal(deps.calls.finalizePending, 1);
});
test("success finalizer writes usage/cost/quota/attempt/pending once", async () => {
const loop: ServerOwnedToolLoopResult = {
kind: "ok",
response: { choices: [{ message: { content: "done" }, finish_reason: "stop" }] },
cumulativeUsage: usage(),
totalCostUsd: 0.02,
receipts: [receipt(0)],
followUps: 0,
termination: "completed",
};
const deps = spyDeps();
await finalizeNonStreamingRequest(buildNonStreamingFinalizationPlan(loop), deps);
assert.equal(deps.calls.writeUsage, 1);
assert.equal(deps.calls.writeCost, 1);
assert.equal(deps.calls.scheduleQuota, 1);
assert.equal(deps.calls.writeAttempt, 1);
assert.equal(deps.calls.finalizePending, 1);
});
test("finalizeToolLoopError delegates through finalization plan and deps", async () => {
const loop: ServerOwnedToolLoopResult = {
kind: "error",
errorResult: errorResult(),
cumulativeUsage: usage(),
totalCostUsd: 0.05,
receipts: [receipt(0), receipt(1, { httpStatus: 429 })],
followUps: 1,
termination: "provider_error",
};
let usageSaved = false;
let attemptLogged = false;
let pendingTracked = false;
const res = await finalizeToolLoopError({
loop,
model: "gpt-4o",
provider: "openai",
connectionId: "conn-1",
providerRequest: { messages: [] },
persistFailureUsage: (status, code, u) => {
assert.equal(status, 429);
assert.equal(code, "rate_limited");
assert.equal(u?.prompt_tokens, 100);
assert.equal(u?.completion_tokens, 20);
assert.equal(u?.cache_read_input_tokens, 10);
assert.equal(u?.reasoning_tokens, 5);
usageSaved = true;
},
persistAttemptLogs: (params) => {
assert.equal(params.status, 429);
attemptLogged = true;
},
trackPendingRequest: (m, _p, _conn, pending) => {
assert.equal(m, "gpt-4o");
assert.equal(pending, false);
pendingTracked = true;
},
});
assert.equal(res.status, 429);
assert.equal(usageSaved, true);
assert.equal(attemptLogged, true);
assert.equal(pendingTracked, true);
});

1129
tests/unit/non-streaming-provider-leg.test.ts Normal file
View File

File diff suppressed because it is too large Load Diff

632
tests/unit/provider-execution-pipeline.test.ts Normal file
View File

@@ -0,0 +1,632 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import type {
ChatCoreExecutorResult,
PipelineConnectionContext,
PipelineStateHooks,
PipelineTargetContext,
PipelineWireState,
ProviderExecutionPipelineInput,
ProviderExecutionPolicy,
} from "../../open-sse/handlers/chatCore/providerExecutionPipeline.ts";
test("runProviderExecutionPipeline is importable", async () => {
const mod = await import("../../open-sse/handlers/chatCore/providerExecutionPipeline.ts");
assert.equal(typeof mod.runProviderExecutionPipeline, "function");
});
function jsonResponse(body: unknown, status: number, extraHeaders: Record<string, string> = {}) {
return new Response(JSON.stringify(body), {
status,
headers: { "content-type": "application/json", ...extraHeaders },
});
}
function makeAttempt(
body: unknown,
status: number,
extra: Partial<ChatCoreExecutorResult> = {}
): ChatCoreExecutorResult {
const response = jsonResponse(body, status, extra.headers as Record<string, string> | undefined);
return {
response,
url: extra.url ?? "https://upstream.test/v1/chat/completions",
headers: extra.headers ?? { "content-type": "application/json" },
transformedBody: extra.transformedBody ?? { model: "gpt-5" },
...extra,
};
}
function noopState(): PipelineStateHooks {
return {
updatePendingStage: () => {},
recordRateLimitHeaders: () => {},
recordRateLimitBody: () => {},
writeTerminalStatus: async () => {},
persistConnectionPatch: () => {},
setConnectionRateLimitedUntil: () => {},
lockModel: () => {},
recordAntigravityQuotaState: async () => {},
markAccountSemaphoreBlocked: () => {},
isolateProbeFailures: () => false,
};
}
function makeInput(opts: {
policy: ProviderExecutionPolicy;
provider: string;
model?: string;
stream?: boolean;
connectionId?: string;
send: (model: string, allowDedup: boolean) => Promise<ChatCoreExecutorResult>;
getProviderCredentials?: PipelineConnectionContext["getProviderCredentials"];
replaceCredentials?: PipelineConnectionContext["replaceCredentials"];
getCurrentConnectionId?: () => string | undefined;
refreshCredentials?: PipelineConnectionContext["refreshCredentials"];
onCredentialsRefreshed?: PipelineConnectionContext["onCredentialsRefreshed"];
getNextFamilyFallback?: ProviderExecutionPipelineInput["getNextFamilyFallback"];
state?: Partial<PipelineStateHooks>;
}): ProviderExecutionPipelineInput {
const model = opts.model ?? "gpt-5";
const connectionId = opts.connectionId ?? "conn-a";
let currentId: string | undefined = connectionId;
let credentials: Record<string, unknown> = { connectionId };
const target: PipelineTargetContext = {
provider: opts.provider,
requestedModel: model,
sourceFormat: "openai",
targetFormat: "openai",
stream: opts.stream ?? false,
};
const wire: PipelineWireState = {
body: { model, messages: [{ role: "user", content: "hi" }] },
currentModel: model,
triedModels: new Set([model]),
setBodyAndModel: (body, nextModel) => {
wire.body = body;
wire.currentModel = nextModel;
wire.triedModels.add(nextModel);
},
};
const connection: PipelineConnectionContext = {
initialConnectionId: connectionId,
getCurrentConnectionId: opts.getCurrentConnectionId ?? (() => currentId),
getCredentials: () => credentials,
replaceCredentials:
opts.replaceCredentials ??
((next) => {
credentials = next;
currentId = typeof next.connectionId === "string" ? next.connectionId : currentId;
}),
onCredentialsRefreshed: opts.onCredentialsRefreshed ?? (() => {}),
assertManagedLeaseFence: () => {},
refreshCredentials: opts.refreshCredentials,
getProviderCredentials:
opts.getProviderCredentials ??
(async () => {
throw new Error("getProviderCredentials must not be called in this fixture");
}),
};
return {
policy: opts.policy,
target,
connection,
wire,
state: { ...noopState(), ...(opts.state || {}) },
sendProviderAttempt: opts.send,
getNextFamilyFallback: opts.getNextFamilyFallback,
};
}
test("initial Codex 429: rotation resolver>=1 and successful retry", async () => {
const { runProviderExecutionPipeline } = await import(
"../../open-sse/handlers/chatCore/providerExecutionPipeline.ts"
);
let sendCount = 0;
let resolverCallCount = 0;
const input = makeInput({
policy: { allowAccountRotation: true, allowModelFallback: true, expectedConnectionId: undefined },
provider: "codex",
connectionId: "conn-a",
send: async () => {
sendCount += 1;
if (sendCount === 1) {
return makeAttempt({ error: { message: "rate limited", type: "rate_limit_error" } }, 429, {
headers: { "retry-after": "1" },
});
}
return makeAttempt({
id: "chatcmpl-ok",
choices: [{ message: { role: "assistant", content: "rotated" }, finish_reason: "stop" }],
}, 200);
},
getProviderCredentials: (async () => {
resolverCallCount += 1;
return { connectionId: "conn-b", allRateLimited: false };
}) as PipelineConnectionContext["getProviderCredentials"],
});
const outcome = await runProviderExecutionPipeline(input);
assert.equal(resolverCallCount >= 1, true, "resolver must run on initial Codex 429");
assert.equal(sendCount, 2, "second send after rotation");
assert.equal(outcome.kind, "response");
if (outcome.kind === "response") {
assert.equal(outcome.connectionId, "conn-b");
assert.equal(outcome.response.status, 200);
}
});
test("initial Antigravity 422 gcp_project_required: rotation resolver>=1 and successful retry", async () => {
const { runProviderExecutionPipeline } = await import(
"../../open-sse/handlers/chatCore/providerExecutionPipeline.ts"
);
let sendCount = 0;
let resolverCallCount = 0;
const input = makeInput({
policy: { allowAccountRotation: true, allowModelFallback: true },
provider: "antigravity",
connectionId: "agy-a",
send: async () => {
sendCount += 1;
if (sendCount === 1) {
return makeAttempt({ error: { message: "gcp_project_required", type: "invalid_request" } }, 422);
}
return makeAttempt(
{
id: "chatcmpl-ok",
choices: [{ message: { role: "assistant", content: "rotated" }, finish_reason: "stop" }],
},
200
);
},
getProviderCredentials: (async () => {
resolverCallCount += 1;
return { connectionId: "agy-b", allRateLimited: false };
}) as PipelineConnectionContext["getProviderCredentials"],
});
const outcome = await runProviderExecutionPipeline(input);
assert.equal(resolverCallCount >= 1, true, "resolver must run on initial Antigravity BYOP 422");
assert.equal(sendCount, 2, "second send after BYOP rotation");
assert.equal(outcome.kind, "response");
if (outcome.kind === "response") {
assert.equal(outcome.connectionId, "agy-b");
assert.equal(outcome.response.status, 200);
}
});
test("follow-up rotation blocks resolver on Antigravity 422", async () => {
const { runProviderExecutionPipeline } = await import(
"../../open-sse/handlers/chatCore/providerExecutionPipeline.ts"
);
let sendCount = 0;
let resolverCallCount = 0;
const input = makeInput({
policy: {
allowAccountRotation: false,
allowModelFallback: false,
expectedConnectionId: "agy-a",
},
provider: "antigravity",
connectionId: "agy-a",
send: async () => {
sendCount += 1;
return makeAttempt({ error: { message: "gcp_project_required", type: "invalid_request" } }, 422);
},
getProviderCredentials: (async () => {
resolverCallCount += 1;
return { connectionId: "agy-b", allRateLimited: false };
}) as PipelineConnectionContext["getProviderCredentials"],
});
const outcome = await runProviderExecutionPipeline(input);
assert.equal(resolverCallCount, 0, "follow-up must not call credentials resolver");
assert.equal(sendCount, 1, "follow-up sends once");
assert.equal(outcome.kind, "error");
if (outcome.kind === "error") {
assert.equal(outcome.result.status, 422);
assert.equal(outcome.connectionId, "agy-a");
}
});
test("follow-up rotation blocks resolver on Codex 429", async () => {
const { runProviderExecutionPipeline } = await import(
"../../open-sse/handlers/chatCore/providerExecutionPipeline.ts"
);
let sendCount = 0;
let resolverCallCount = 0;
const input = makeInput({
policy: {
allowAccountRotation: false,
allowModelFallback: false,
expectedConnectionId: "conn-a",
},
provider: "codex",
connectionId: "conn-a",
send: async () => {
sendCount += 1;
return makeAttempt({ error: { message: "rate limited", type: "rate_limit_error" } }, 429);
},
getProviderCredentials: (async () => {
resolverCallCount += 1;
return { connectionId: "conn-b", allRateLimited: false };
}) as PipelineConnectionContext["getProviderCredentials"],
});
const outcome = await runProviderExecutionPipeline(input);
assert.equal(resolverCallCount, 0, "follow-up must not call credentials resolver");
assert.equal(sendCount, 1, "follow-up sends once");
assert.equal(outcome.kind, "error");
if (outcome.kind === "error") {
assert.equal(outcome.result.status, 429);
assert.equal(outcome.connectionId, "conn-a");
}
});
test("401 refresh succeeds then retries once on same connection", async () => {
const { runProviderExecutionPipeline } = await import(
"../../open-sse/handlers/chatCore/providerExecutionPipeline.ts"
);
let sendCount = 0;
let refreshCount = 0;
let persistCount = 0;
let resolverCallCount = 0;
const input = makeInput({
policy: { allowAccountRotation: true, allowModelFallback: true },
provider: "openai",
connectionId: "conn-a",
send: async () => {
sendCount += 1;
if (sendCount === 1) {
return makeAttempt({ error: { message: "invalid_api_key", type: "authentication_error" } }, 401);
}
return makeAttempt(
{
id: "chatcmpl-ok",
choices: [{ message: { role: "assistant", content: "refreshed" }, finish_reason: "stop" }],
},
200
);
},
refreshCredentials: async (creds) => {
refreshCount += 1;
return { ...creds, accessToken: "new-token" };
},
onCredentialsRefreshed: async () => {
persistCount += 1;
},
getProviderCredentials: (async () => {
resolverCallCount += 1;
return { connectionId: "conn-b", allRateLimited: false };
}) as PipelineConnectionContext["getProviderCredentials"],
});
const outcome = await runProviderExecutionPipeline(input);
assert.equal(refreshCount, 1, "refresh once");
assert.equal(persistCount, 1, "onCredentialsRefreshed once");
assert.equal(resolverCallCount, 0, "401 refresh must not rotate accounts");
assert.equal(sendCount, 2, "retry once after refresh");
assert.equal(outcome.kind, "response");
if (outcome.kind === "response") {
assert.equal(outcome.connectionId, "conn-a");
assert.equal(outcome.response.status, 200);
}
});
test("status restatement rewrites agentrouter 403 quota exhaustion to 429 before classification", async () => {
const { runProviderExecutionPipeline } = await import(
"../../open-sse/handlers/chatCore/providerExecutionPipeline.ts"
);
let sendCount = 0;
const input = makeInput({
policy: { allowAccountRotation: true, allowModelFallback: true },
provider: "agentrouter",
connectionId: "ar-a",
send: async () => {
sendCount += 1;
return makeAttempt({ error: { message: "用户额度不足", type: "forbidden" } }, 403);
},
});
const outcome = await runProviderExecutionPipeline(input);
assert.equal(sendCount, 1);
assert.equal(outcome.kind, "error");
if (outcome.kind === "error") {
assert.equal(outcome.result.status, 429, "restated before classification");
assert.equal(outcome.connectionId, "ar-a");
}
});
test("thinking-signature recovery returns winning response", async () => {
const { runProviderExecutionPipeline } = await import(
"../../open-sse/handlers/chatCore/providerExecutionPipeline.ts"
);
let sendCount = 0;
const input = makeInput({
policy: { allowAccountRotation: true, allowModelFallback: true },
provider: "claude",
connectionId: "cl-a",
send: async () => {
sendCount += 1;
if (sendCount === 1) {
return makeAttempt(
{ error: { message: "invalid signature in thinking block", type: "invalid_request_error" } },
400
);
}
return makeAttempt(
{
id: "msg-ok",
type: "message",
role: "assistant",
content: [{ type: "text", text: "recovered" }],
},
200
);
},
});
input.wire.body = {
model: "gpt-5",
messages: [
{ role: "user", content: "q1" },
{
role: "assistant",
content: [
{ type: "thinking", thinking: "old" },
{ type: "text", text: "a1" },
],
},
{ role: "user", content: "q2" },
],
};
const outcome = await runProviderExecutionPipeline(input);
assert.equal(sendCount, 2, "one recovery send after signature error");
assert.equal(outcome.kind, "response");
if (outcome.kind === "response") {
assert.equal(outcome.response.status, 200);
assert.equal(outcome.connectionId, "cl-a");
}
});
test("initial model-unavailable falls back to sibling model", async () => {
const { runProviderExecutionPipeline } = await import(
"../../open-sse/handlers/chatCore/providerExecutionPipeline.ts"
);
let sendCount = 0;
let fallbackLookupCount = 0;
const sentModels: string[] = [];
const input = makeInput({
policy: { allowAccountRotation: true, allowModelFallback: true },
provider: "openai",
model: "gpt-5",
connectionId: "conn-a",
send: async (model) => {
sendCount += 1;
sentModels.push(model);
if (model === "gpt-5") {
return makeAttempt(
{ error: { message: "model is not available", type: "invalid_request_error" } },
404
);
}
return makeAttempt(
{
id: "chatcmpl-ok",
choices: [{ message: { role: "assistant", content: "fallback" }, finish_reason: "stop" }],
},
200
);
},
getNextFamilyFallback: (current) => {
fallbackLookupCount += 1;
return current === "gpt-5" ? "gpt-5-mini" : null;
},
});
const outcome = await runProviderExecutionPipeline(input);
assert.equal(fallbackLookupCount >= 1, true, "family fallback consulted");
assert.deepEqual(sentModels, ["gpt-5", "gpt-5-mini"]);
assert.equal(sendCount, 2);
assert.equal(outcome.kind, "response");
if (outcome.kind === "response") {
assert.equal(outcome.model, "gpt-5-mini");
assert.equal(outcome.response.status, 200);
}
});
test("follow-up allowModelFallback=false blocks model-unavailable fallback", async () => {
const { runProviderExecutionPipeline } = await import(
"../../open-sse/handlers/chatCore/providerExecutionPipeline.ts"
);
let sendCount = 0;
let fallbackLookupCount = 0;
const input = makeInput({
policy: {
allowAccountRotation: false,
allowModelFallback: false,
expectedConnectionId: "conn-a",
},
provider: "openai",
model: "gpt-5",
connectionId: "conn-a",
send: async () => {
sendCount += 1;
return makeAttempt(
{ error: { message: "model is not available", type: "invalid_request_error" } },
404
);
},
getNextFamilyFallback: () => {
fallbackLookupCount += 1;
return "gpt-5-mini";
},
});
const outcome = await runProviderExecutionPipeline(input);
assert.equal(fallbackLookupCount, 0, "follow-up must not consult family fallback");
assert.equal(sendCount, 1);
assert.equal(outcome.kind, "error");
if (outcome.kind === "error") {
assert.equal(outcome.result.status, 404);
assert.equal(outcome.model, "gpt-5");
}
});
test("Codex 429 rotation calls scope-rate-limit, affinity-clear, and audit hooks", async () => {
const { runProviderExecutionPipeline } = await import(
"../../open-sse/handlers/chatCore/providerExecutionPipeline.ts"
);
const rateLimited: Array<Record<string, unknown>> = [];
const affinityCleared: string[] = [];
const audits: Array<Record<string, unknown>> = [];
let sendCount = 0;
const input = makeInput({
policy: { allowAccountRotation: true, allowModelFallback: true },
provider: "codex",
connectionId: "conn-a",
send: async () => {
sendCount += 1;
if (sendCount === 1) {
return makeAttempt({ error: { message: "rate limited", type: "rate_limit_error" } }, 429, {
headers: { "retry-after": "2" },
});
}
return makeAttempt(
{
id: "chatcmpl-ok",
choices: [{ message: { role: "assistant", content: "rotated" }, finish_reason: "stop" }],
},
200
);
},
getProviderCredentials: (async () => ({
connectionId: "conn-b",
allRateLimited: false,
})) as PipelineConnectionContext["getProviderCredentials"],
state: {
onCodexScopeRateLimited: (params) => {
rateLimited.push(params as unknown as Record<string, unknown>);
},
onClearSessionAffinity: (params) => {
affinityCleared.push(params.failedConnectionId);
},
onAuditAccountRotation: (params) => {
audits.push(params as unknown as Record<string, unknown>);
},
},
});
const outcome = await runProviderExecutionPipeline(input);
assert.equal(outcome.kind, "response");
assert.equal(rateLimited.length, 1, "must persist Codex model-scope cooldown");
assert.equal(rateLimited[0]?.failedConnectionId, "conn-a");
assert.deepEqual(affinityCleared, ["conn-a"]);
assert.equal(audits.length, 1);
assert.equal(audits[0]?.action, "codex.account_rotation");
assert.equal(audits[0]?.failedConnectionId, "conn-a");
assert.equal(audits[0]?.newConnectionId, "conn-b");
});
test("Codex 429 cooldown reads Retry-After from the response, not request headers", async () => {
const { runProviderExecutionPipeline } = await import(
"../../open-sse/handlers/chatCore/providerExecutionPipeline.ts"
);
const rateLimited: Array<Record<string, unknown>> = [];
let sendCount = 0;
const input = makeInput({
policy: { allowAccountRotation: true, allowModelFallback: true },
provider: "codex",
connectionId: "conn-a",
send: async () => {
sendCount += 1;
if (sendCount === 1) {
// BaseExecutor puts REQUEST headers on attempt.headers (Authorization).
// Upstream Retry-After lives on the Response. Mixing the two bags is the
// extract regression: cooldown silently falls back to 60s.
return {
response: jsonResponse(
{ error: { message: "rate limited", type: "rate_limit_error" } },
429,
{ "Retry-After": "5" }
),
url: "https://upstream.test/v1/chat/completions",
headers: { Authorization: "Bearer request-token", "content-type": "application/json" },
transformedBody: { model: "gpt-5" },
};
}
return makeAttempt(
{
id: "chatcmpl-ok",
choices: [{ message: { role: "assistant", content: "rotated" }, finish_reason: "stop" }],
},
200
);
},
getProviderCredentials: (async () => ({
connectionId: "conn-b",
allRateLimited: false,
})) as PipelineConnectionContext["getProviderCredentials"],
state: {
onCodexScopeRateLimited: (params) => {
rateLimited.push(params as unknown as Record<string, unknown>);
},
},
});
const outcome = await runProviderExecutionPipeline(input);
assert.equal(outcome.kind, "response");
assert.equal(rateLimited.length, 1, "must persist Codex model-scope cooldown");
const until = new Date(String(rateLimited[0]?.rateLimitedUntil)).getTime();
const delta = until - Date.now();
assert.ok(
delta > 4_000 && delta < 8_000,
`Retry-After: 5 must yield ~5s cooldown, got ${delta}ms (60s = still reading request headers)`
);
});
test("Antigravity BYOP 422 rotation persists cooldown via setConnectionRateLimitedUntil", async () => {
const { runProviderExecutionPipeline } = await import(
"../../open-sse/handlers/chatCore/providerExecutionPipeline.ts"
);
const cooldowns: Array<{ id: string; untilMs: number | null }> = [];
let sendCount = 0;
const input = makeInput({
policy: { allowAccountRotation: true, allowModelFallback: true },
provider: "antigravity",
connectionId: "agy-a",
send: async () => {
sendCount += 1;
if (sendCount === 1) {
return makeAttempt(
{ error: { message: "gcp_project_required", type: "invalid_request" } },
422
);
}
return makeAttempt(
{
id: "chatcmpl-ok",
choices: [{ message: { role: "assistant", content: "rotated" }, finish_reason: "stop" }],
},
200
);
},
getProviderCredentials: (async () => ({
connectionId: "agy-b",
allRateLimited: false,
})) as PipelineConnectionContext["getProviderCredentials"],
state: {
setConnectionRateLimitedUntil: (id, untilMs) => {
cooldowns.push({ id, untilMs });
},
},
});
const outcome = await runProviderExecutionPipeline(input);
assert.equal(outcome.kind, "response");
assert.equal(sendCount, 2);
assert.equal(cooldowns.length, 1, "BYOP rotate must persist cooldown before picking sibling");
assert.equal(cooldowns[0]?.id, "agy-a");
assert.equal(typeof cooldowns[0]?.untilMs, "number");
assert.equal((cooldowns[0]?.untilMs ?? 0) > Date.now(), true);
});

49
tests/unit/request-logger-endpoints.test.ts
View File

@@ -738,3 +738,52 @@ test("generic client snapshots exclude hard-lease control headers", async () =>
assert.equal(out.headers["x-omniroute-lease-generation"], undefined);
assert.equal(out.headers["x-session-id"], "independent-routing-session");
});
function syntheticReceipt(index: number) {
return {
index,
connectionId: "conn-1",
provider: "openai",
model: "gpt-4o",
startedAt: "2026-01-01T00:00:00.000Z",
endedAt: "2026-01-01T00:00:01.000Z",
latencyMs: 10 + index,
httpStatus: 200,
errorType: null,
usage: { prompt_tokens: 1, completion_tokens: 1, total_tokens: 2 },
serviceTier: null,
computedCostUsd: 0.001,
toolCalls: [{ id: `call-${index}`, name: "memory_search" }],
termination: "completed",
clientVisible: true,
};
}
test("logToolLoopReceipt keeps first 4 receipts and clones them", async () => {
const { createRequestLogger } = await import("../../open-sse/utils/requestLogger.ts");
const logger = await createRequestLogger("openai", "openai", "gpt-4o", {
enabled: true,
captureStreamChunks: false,
});
for (let i = 0; i < 5; i++) {
logger.logToolLoopReceipt(syntheticReceipt(i));
}
const payloads = logger.getPipelinePayloads();
assert.ok(payloads?.toolLoop);
assert.equal(payloads.toolLoop.legs.length, 4);
assert.deepEqual(
payloads.toolLoop.legs.map((leg) => (leg as { index: number }).index),
[0, 1, 2, 3]
);
assert.equal("arguments" in (payloads.toolLoop.legs[0] as object), false);
});
test("logToolLoopReceipt is a no-op when logger is disabled", async () => {
const { createRequestLogger } = await import("../../open-sse/utils/requestLogger.ts");
const logger = await createRequestLogger("openai", "openai", "gpt-4o", {
enabled: false,
captureStreamChunks: false,
});
logger.logToolLoopReceipt(syntheticReceipt(0));
assert.equal(logger.getPipelinePayloads(), null);
});

140
tests/unit/server-owned-tool-loop-flag.test.ts Normal file
View File

@@ -0,0 +1,140 @@
import { describe, it, before, after, beforeEach } from "node:test";
import assert from "node:assert/strict";
import os from "node:os";
import path from "node:path";
import fs from "node:fs";
import { fileURLToPath } from "node:url";
const __dirname = path.dirname(fileURLToPath(import.meta.url));
const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-test-flag-loop-"));
process.env.DATA_DIR = tmpDir;
const { FEATURE_FLAG_DEFINITIONS } =
await import("../../src/shared/constants/featureFlagDefinitions.ts");
const { setFeatureFlagOverride, clearAllFeatureFlagOverrides } =
await import("../../src/lib/db/featureFlags.ts");
const { isServerOwnedToolLoopEnabled } = await import("../../src/shared/utils/featureFlags.ts");
describe("SERVER_OWNED_TOOL_LOOP_ENABLED flag definition", () => {
it("exists in FEATURE_FLAG_DEFINITIONS", () => {
const def = FEATURE_FLAG_DEFINITIONS.find((d) => d.key === "SERVER_OWNED_TOOL_LOOP_ENABLED");
assert.ok(def, "SERVER_OWNED_TOOL_LOOP_ENABLED should exist");
assert.equal(def.category, "runtime");
assert.equal(def.defaultValue, "false");
assert.equal(def.requiresRestart, false);
assert.equal(def.descriptionI18nKey, "featureFlagServerOwnedToolLoopDescription");
});
});
describe("isServerOwnedToolLoopEnabled wrapper", () => {
beforeEach(() => {
clearAllFeatureFlagOverrides();
});
it("returns false when no override is set (default)", () => {
assert.equal(isServerOwnedToolLoopEnabled(), false);
});
it("returns true when DB override is set to true", () => {
setFeatureFlagOverride("SERVER_OWNED_TOOL_LOOP_ENABLED", "true");
assert.equal(isServerOwnedToolLoopEnabled(), true);
});
it("returns false and logs when injected reader throws", () => {
const logs: unknown[] = [];
const origError = console.error;
console.error = (...args: unknown[]) => {
logs.push(args);
};
try {
const throwingReader = () => {
throw new Error("flag read failed");
};
const result = isServerOwnedToolLoopEnabled(throwingReader);
assert.equal(result, false);
assert.ok(logs.length >= 1, "console.error should be called at least once");
assert.ok(
logs.some((args) =>
String(args).includes("Failed to resolve SERVER_OWNED_TOOL_LOOP_ENABLED")
),
"error log should mention the flag key"
);
} finally {
console.error = origError;
}
});
});
describe("feature-flags-settings count update", () => {
it("flag count matches updated expected value", () => {
assert.equal(FEATURE_FLAG_DEFINITIONS.length, 55);
});
});
describe("i18n key parity for SERVER_OWNED_TOOL_LOOP_ENABLED", () => {
let enMessages: Record<string, unknown>;
let ptBrMessages: Record<string, unknown>;
before(async () => {
const enRaw = fs.readFileSync(
path.resolve(__dirname, "../../src/i18n/messages/en.json"),
"utf8"
);
enMessages = JSON.parse(enRaw);
const ptBrRaw = fs.readFileSync(
path.resolve(__dirname, "../../src/i18n/messages/pt-BR.json"),
"utf8"
);
ptBrMessages = JSON.parse(ptBrRaw);
});
it("en.json has nested featureFlags.definitions.SERVER_OWNED_TOOL_LOOP_ENABLED.label", () => {
const defs = enMessages.featureFlags as Record<string, unknown> | undefined;
assert.ok(defs, "en.json should have featureFlags section");
const definitions = (defs as Record<string, unknown>).definitions as
Record<string, unknown> | undefined;
assert.ok(definitions, "en.json featureFlags should have definitions");
const flagDef = definitions.SERVER_OWNED_TOOL_LOOP_ENABLED as
Record<string, unknown> | undefined;
assert.ok(flagDef, "definitions should contain SERVER_OWNED_TOOL_LOOP_ENABLED");
assert.equal(flagDef.label, "Server-Owned Tool Loop");
assert.equal(
flagDef.description,
"Continue non-streaming server-owned tool calls until the model returns a client-usable response."
);
});
it("pt-BR.json has nested featureFlags.definitions.SERVER_OWNED_TOOL_LOOP_ENABLED.label", () => {
const defs = ptBrMessages.featureFlags as Record<string, unknown> | undefined;
assert.ok(defs, "pt-BR.json should have featureFlags section");
const definitions = (defs as Record<string, unknown>).definitions as
Record<string, unknown> | undefined;
assert.ok(definitions, "pt-BR.json featureFlags should have definitions");
const flagDef = definitions.SERVER_OWNED_TOOL_LOOP_ENABLED as
Record<string, unknown> | undefined;
assert.ok(flagDef, "definitions should contain SERVER_OWNED_TOOL_LOOP_ENABLED");
assert.equal(flagDef.label, "Server-Owned Tool Loop");
assert.equal(typeof flagDef.description, "string");
assert.ok(
((flagDef.description as string) || "").length > 0,
"description should be non-empty"
);
});
it("en.json does NOT have stale top-level featureFlagServerOwnedToolLoopDescription", () => {
assert.equal(
(enMessages as Record<string, unknown>).featureFlagServerOwnedToolLoopDescription,
undefined,
"top-level key should be removed"
);
});
});
after(() => {
try {
fs.rmSync(tmpDir, { recursive: true, force: true });
} catch {
// ignore
}
});

82
tests/unit/server-owned-tool-loop-gate.test.ts Normal file
View File

@@ -0,0 +1,82 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { shouldRunServerOwnedToolLoop } from "../../open-sse/handlers/chatCore/serverOwnedToolLoopGate.ts";
import { FORMATS } from "../../open-sse/translator/formats.ts";
test("flag-off never runs the loop", () => {
assert.equal(
shouldRunServerOwnedToolLoop({
enabled: false,
stream: false,
isResponsesEndpoint: false,
sourceFormat: FORMATS.OPENAI,
}),
false
);
});
test("streaming never runs the loop", () => {
assert.equal(
shouldRunServerOwnedToolLoop({
enabled: true,
stream: true,
isResponsesEndpoint: false,
sourceFormat: FORMATS.OPENAI,
}),
false
);
});
test("Responses endpoint and format keep the old path", () => {
assert.equal(
shouldRunServerOwnedToolLoop({
enabled: true,
stream: false,
isResponsesEndpoint: true,
sourceFormat: FORMATS.OPENAI,
}),
false
);
assert.equal(
shouldRunServerOwnedToolLoop({
enabled: true,
stream: false,
isResponsesEndpoint: false,
sourceFormat: FORMATS.OPENAI_RESPONSES,
}),
false
);
});
test("non-streaming Chat and Claude run the loop when enabled", () => {
assert.equal(
shouldRunServerOwnedToolLoop({
enabled: true,
stream: false,
isResponsesEndpoint: false,
sourceFormat: FORMATS.OPENAI,
}),
true
);
assert.equal(
shouldRunServerOwnedToolLoop({
enabled: true,
stream: false,
isResponsesEndpoint: false,
sourceFormat: FORMATS.CLAUDE,
}),
true
);
});
test("gemini and other source formats keep the old path", () => {
assert.equal(
shouldRunServerOwnedToolLoop({
enabled: true,
stream: false,
isResponsesEndpoint: false,
sourceFormat: FORMATS.GEMINI,
}),
false
);
});

246
tests/unit/server-owned-tool-loop-wire.test.ts Normal file
View File

@@ -0,0 +1,246 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import {
applyServerOwnedToolLoopIfNeeded,
derivePostInjectionRequestIdentity,
} from "../../open-sse/handlers/chatCore/serverOwnedToolLoopWire.ts";
import type {
NonStreamingProviderLegResult,
ProviderLegReceipt,
} from "../../src/lib/skills/toolLoopTypes.ts";
import { FORMATS } from "../../open-sse/translator/formats.ts";
function receipt(index: number): ProviderLegReceipt {
return {
index,
connectionId: "conn-1",
provider: "openai",
model: "gpt-4o",
startedAt: "2026-01-01T00:00:00.000Z",
endedAt: "2026-01-01T00:00:01.000Z",
latencyMs: 10,
httpStatus: 200,
errorType: null,
usage: { prompt_tokens: 10, completion_tokens: 5, total_tokens: 15 },
serviceTier: null,
computedCostUsd: 0.01,
toolCalls: [],
termination: "completed",
clientVisible: true,
};
}
function okLeg(overrides: Partial<NonStreamingProviderLegResult & { kind: "ok" }> = {}) {
return {
kind: "ok" as const,
response: {
choices: [
{
message: {
role: "assistant",
content: null,
tool_calls: [
{
id: "call_1",
type: "function",
function: { name: "memory_search", arguments: '{"q":"x"}' },
},
],
},
finish_reason: "tool_calls",
},
],
},
responseForMemoryExtraction: {},
providerBody: {},
providerRequest: {},
usage: { prompt_tokens: 10, completion_tokens: 5, total_tokens: 15 },
responsePayloadFormat: "openai",
looksLikeSSE: false,
connectionId: "conn-1",
headers: new Headers(),
receipt: receipt(0),
...overrides,
};
}
test("flag-off skips the loop and does not resume", async () => {
let followUps = 0;
const result = await applyServerOwnedToolLoopIfNeeded({
enabled: false,
stream: false,
isResponsesEndpoint: false,
sourceFormat: FORMATS.OPENAI,
initialLeg: okLeg(),
sourceBody: { model: "gpt-4o", messages: [{ role: "user", content: "hi" }] },
skillsModelId: "openai",
executionContext: {
apiKeyId: "k",
sessionId: "s",
requestId: "r",
builtinToolNames: ["memory_search"],
},
expectedConnectionId: "conn-1",
followUpLeg: async () => {
followUps += 1;
throw new Error("must not resume");
},
logReceipt: () => {},
});
assert.equal(result.kind, "skip");
assert.equal(followUps, 0);
});
test("Responses format skips even when enabled", async () => {
const result = await applyServerOwnedToolLoopIfNeeded({
enabled: true,
stream: false,
isResponsesEndpoint: true,
sourceFormat: FORMATS.OPENAI,
initialLeg: okLeg(),
sourceBody: { model: "gpt-4o", messages: [] },
skillsModelId: "openai",
executionContext: { apiKeyId: "k", sessionId: "s", requestId: "r" },
expectedConnectionId: "conn-1",
followUpLeg: async () => {
throw new Error("must not resume");
},
logReceipt: () => {},
});
assert.equal(result.kind, "skip");
});
test("enabled Chat loop resumes once and logs receipts", async () => {
const logged: number[] = [];
let followUps = 0;
const result = await applyServerOwnedToolLoopIfNeeded({
enabled: true,
stream: false,
isResponsesEndpoint: false,
sourceFormat: FORMATS.OPENAI,
initialLeg: okLeg(),
sourceBody: {
model: "gpt-4o",
messages: [{ role: "user", content: "hi" }],
},
skillsModelId: "openai",
executionContext: {
apiKeyId: "k",
sessionId: "s",
requestId: "r",
builtinToolNames: ["memory_search"],
},
expectedConnectionId: "conn-1",
followUpLeg: async () => {
followUps += 1;
return {
kind: "ok",
response: {
choices: [{ message: { role: "assistant", content: "done" }, finish_reason: "stop" }],
},
responseForMemoryExtraction: { text: "done" },
providerBody: { id: "2" },
providerRequest: { messages: [] },
usage: { prompt_tokens: 4, completion_tokens: 2, total_tokens: 6 },
responsePayloadFormat: "openai",
looksLikeSSE: false,
connectionId: "conn-1",
headers: new Headers(),
receipt: receipt(1),
};
},
logReceipt: (r) => logged.push(r.index),
executeServerOwned: async (calls) =>
calls.map((c) => ({ id: c.id, name: c.name, result: { hits: [] }, replayed: false })),
});
assert.equal(result.kind, "ok");
if (result.kind !== "ok") return;
assert.equal(followUps, 1);
assert.deepEqual(logged, [0, 1]);
assert.equal(
(result.leg.response as { choices: Array<{ message: { content: string } }> }).choices[0].message
.content,
"done"
);
assert.equal(result.usage?.prompt_tokens, 14);
});
test("provider error from follow-up is returned as error", async () => {
const result = await applyServerOwnedToolLoopIfNeeded({
enabled: true,
stream: false,
isResponsesEndpoint: false,
sourceFormat: FORMATS.OPENAI,
initialLeg: okLeg(),
sourceBody: { model: "gpt-4o", messages: [{ role: "user", content: "hi" }] },
skillsModelId: "openai",
executionContext: {
apiKeyId: "k",
sessionId: "s",
requestId: "r",
builtinToolNames: ["memory_search"],
},
expectedConnectionId: "conn-1",
followUpLeg: async () => ({
kind: "error",
result: {
success: false,
status: 429,
response: new Response(null, { status: 429 }),
error: "rate limited",
errorCode: "rate_limited",
},
receipt: receipt(1),
usage: { prompt_tokens: 1, completion_tokens: 0, total_tokens: 1 },
}),
logReceipt: () => {},
executeServerOwned: async (calls) =>
calls.map((c) => ({ id: c.id, name: c.name, result: { hits: [] }, replayed: false })),
});
assert.equal(result.kind, "error");
if (result.kind !== "error") return;
assert.equal(result.loop.errorResult?.status, 429);
});
test("derivePostInjectionRequestIdentity uses the client idempotency header", () => {
const a = derivePostInjectionRequestIdentity({
apiKeyId: "k",
headers: { "idempotency-key": "tool-loop-retry-1" },
skillRequestId: "internal-1",
postInjectionBody: { messages: [{ role: "user", content: "hi" }] },
});
const b = derivePostInjectionRequestIdentity({
apiKeyId: "k",
headers: { "idempotency-key": "tool-loop-retry-1" },
skillRequestId: "internal-2",
postInjectionBody: { messages: [{ role: "user", content: "hi" }] },
});
const c = derivePostInjectionRequestIdentity({
apiKeyId: "k",
headers: { "idempotency-key": "other" },
skillRequestId: "internal-1",
postInjectionBody: { messages: [{ role: "user", content: "hi" }] },
});
assert.equal(a, b);
assert.notEqual(a, c);
});
test("applyServerOwnedToolLoopIfNeeded accepts undefined expectedConnectionId for unmanaged leases", async () => {
const result = await applyServerOwnedToolLoopIfNeeded({
enabled: false,
stream: false,
isResponsesEndpoint: false,
sourceFormat: FORMATS.OPENAI,
initialLeg: okLeg(),
sourceBody: { model: "gpt-4o", messages: [] },
skillsModelId: "openai",
executionContext: { apiKeyId: "k", sessionId: "s", requestId: "r" },
expectedConnectionId: undefined,
followUpLeg: async () => {
throw new Error("must not resume");
},
logReceipt: () => {},
});
assert.equal(result.kind, "skip");
});

1104
tests/unit/server-owned-tool-loop.test.ts Normal file
View File

File diff suppressed because it is too large Load Diff

786
tests/unit/skill-execution-fence.test.ts Normal file
View File

@@ -0,0 +1,786 @@
process.env.DISABLE_SQLITE_AUTO_BACKUP = "true";
import test from "node:test";
import assert from "node:assert/strict";
import fs from "node:fs";
import path from "node:path";
import os from "node:os";
import { createRequire } from "node:module";
const require_ = createRequire(import.meta.url);
const BetterSqlite3 = require_("better-sqlite3") as typeof import("better-sqlite3");
import { createBetterSqliteAdapter } from "../../src/lib/db/adapters/betterSqliteAdapter";
import type { SqliteAdapter } from "../../src/lib/db/adapters/types";
import {
claimServerToolExecution,
finalizeServerToolExecution,
} from "../../src/lib/db/skillExecutionFence";
import { runWithServerToolFence } from "../../src/lib/skills/toolExecutionFence";
// Minimal fixture schema — only what tests need; no SCHEMA_SQL import from core.ts
const FIXTURE_SCHEMA = `
CREATE TABLE IF NOT EXISTS server_tool_executions (
id TEXT PRIMARY KEY,
api_key_id TEXT NOT NULL,
request_identity TEXT NOT NULL,
tool_call_id TEXT NOT NULL,
tool_name TEXT NOT NULL,
input_digest TEXT NOT NULL,
output TEXT,
status TEXT NOT NULL CHECK(status IN ('running', 'success', 'error', 'timeout')),
error_message TEXT,
duration_ms INTEGER,
claim_expires_at TEXT NOT NULL,
completed_at TEXT,
created_at TEXT NOT NULL DEFAULT (datetime('now')),
UNIQUE(api_key_id, request_identity, tool_call_id)
);
CREATE INDEX IF NOT EXISTS idx_server_tool_executions_status_expiry
ON server_tool_executions(status, claim_expires_at);
CREATE INDEX IF NOT EXISTS idx_server_tool_executions_created
ON server_tool_executions(created_at);
`;
function makeTempDb(): {
adapter: SqliteAdapter;
dir: string;
raw: import("better-sqlite3").Database;
} {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), "fence-test-"));
const dbPath = path.join(dir, "test.db");
const raw = new BetterSqlite3(dbPath);
raw.pragma("journal_mode = WAL");
raw.pragma("busy_timeout = 2000");
raw.exec(FIXTURE_SCHEMA);
const adapter = createBetterSqliteAdapter(raw);
return { adapter, dir, raw };
}
function makeSecondAdapter(dir: string): {
adapter: SqliteAdapter;
raw: import("better-sqlite3").Database;
} {
const dbPath = path.join(dir, "test.db");
const raw = new BetterSqlite3(dbPath);
raw.pragma("journal_mode = WAL");
raw.pragma("busy_timeout = 2000");
const adapter = createBetterSqliteAdapter(raw);
return { adapter, raw };
}
function cleanup(raw: import("better-sqlite3").Database, dir: string) {
raw.close();
fs.rmSync(dir, { recursive: true, force: true });
}
const BASE_INPUT = {
apiKeyId: "key-1",
requestIdentity: "key-1:req-id:body-hash",
toolCallId: "call-1",
toolName: "memory_search",
inputDigest: "abc123",
leaseExpiresAt: new Date(Date.now() + 60_000).toISOString(),
};
const BASE_FINALIZE = {
executionId: "",
status: "success" as const,
output: { result: "ok" },
errorMessage: null,
durationMs: 100,
};
// ── Defect 1: Lease expiry compares STORED claim_expires_at, not input ──
test("claim: stored expired lease → unknown even if retry supplies future lease", () => {
const { adapter, dir, raw } = makeTempDb();
try {
const pastLease = new Date(Date.now() - 60_000).toISOString();
const claim1 = claimServerToolExecution(
{ ...BASE_INPUT, leaseExpiresAt: pastLease },
adapter,
Date.now()
);
assert.equal(claim1.kind, "claimed");
const futureLease = new Date(Date.now() + 300_000).toISOString();
const claim2 = claimServerToolExecution(
{ ...BASE_INPUT, leaseExpiresAt: futureLease },
adapter,
Date.now()
);
assert.equal(claim2.kind, "unknown", "must compare stored lease, not input lease");
} finally {
cleanup(raw, dir);
}
});
test("claim: stored future lease with expired retry → in_progress (not unknown)", () => {
const { adapter, dir, raw } = makeTempDb();
try {
const futureLease = new Date(Date.now() + 300_000).toISOString();
const claim1 = claimServerToolExecution(
{ ...BASE_INPUT, leaseExpiresAt: futureLease },
adapter,
Date.now()
);
assert.equal(claim1.kind, "claimed");
const pastLease = new Date(Date.now() - 10_000).toISOString();
const claim2 = claimServerToolExecution(
{ ...BASE_INPUT, leaseExpiresAt: pastLease },
adapter,
Date.now()
);
assert.equal(claim2.kind, "in_progress", "stored lease is future, so should be in_progress");
} finally {
cleanup(raw, dir);
}
});
// ── Basic claim/replay tests ──
test("claim: first claim returns claimed", () => {
const { adapter, dir, raw } = makeTempDb();
try {
const result = claimServerToolExecution(BASE_INPUT, adapter);
assert.equal(result.kind, "claimed");
assert.ok(typeof result.executionId === "string" && result.executionId.length > 0);
} finally {
cleanup(raw, dir);
}
});
test("claim: terminal row (success) with same identity returns replay", () => {
const { adapter, dir, raw } = makeTempDb();
try {
const claim1 = claimServerToolExecution(BASE_INPUT, adapter);
assert.equal(claim1.kind, "claimed");
finalizeServerToolExecution({ ...BASE_FINALIZE, executionId: claim1.executionId }, adapter);
const claim2 = claimServerToolExecution(BASE_INPUT, adapter);
assert.equal(claim2.kind, "replay");
if (claim2.kind === "replay") {
assert.equal(claim2.status, "success");
assert.deepEqual(claim2.output, { result: "ok" });
}
} finally {
cleanup(raw, dir);
}
});
test("claim: terminal row (error) replay preserves error status and message", () => {
const { adapter, dir, raw } = makeTempDb();
try {
const claim1 = claimServerToolExecution(BASE_INPUT, adapter);
assert.equal(claim1.kind, "claimed");
finalizeServerToolExecution(
{
...BASE_FINALIZE,
executionId: claim1.executionId,
status: "error",
output: null,
errorMessage: "tool failed",
durationMs: 42,
},
adapter
);
const claim2 = claimServerToolExecution(BASE_INPUT, adapter);
assert.equal(claim2.kind, "replay");
if (claim2.kind === "replay") {
assert.equal(claim2.status, "error");
assert.equal(claim2.errorMessage, "tool failed");
}
} finally {
cleanup(raw, dir);
}
});
test("claim: same key but different name returns identity_conflict", () => {
const { adapter, dir, raw } = makeTempDb();
try {
claimServerToolExecution(BASE_INPUT, adapter);
const claim2 = claimServerToolExecution({ ...BASE_INPUT, toolName: "different_tool" }, adapter);
assert.equal(claim2.kind, "identity_conflict");
} finally {
cleanup(raw, dir);
}
});
test("claim: same key but different inputDigest returns identity_conflict", () => {
const { adapter, dir, raw } = makeTempDb();
try {
claimServerToolExecution(BASE_INPUT, adapter);
const claim2 = claimServerToolExecution(
{ ...BASE_INPUT, inputDigest: "different_digest" },
adapter
);
assert.equal(claim2.kind, "identity_conflict");
} finally {
cleanup(raw, dir);
}
});
// ── Defect 2: Cross-handle poll test with gate-based concurrency ──
test("fence: cross-handle poll — B sees in_progress, then A finalizes, then B replays", async () => {
const { adapter: adapterA, dir, raw: rawA } = makeTempDb();
const { adapter: adapterB, raw: rawB } = makeSecondAdapter(dir);
let fakeTime = 1_000_000;
const now = () => fakeTime;
const sleep = async (ms: number) => {
fakeTime += ms;
await new Promise<void>((r) => setTimeout(r, 0));
};
// Gate: A signals when it starts, B signals when it observes in_progress
let aStartedResolve!: () => void;
const aStarted = new Promise<void>((r) => {
aStartedResolve = r;
});
let bObservedResolve!: () => void;
const bObserved = new Promise<void>((r) => {
bObservedResolve = r;
});
let handlerCallCount = 0;
const handler = async () => {
handlerCallCount++;
aStartedResolve();
// Wait until B has observed in_progress before finalizing
await bObserved;
return "handler-result";
};
const sharedArgs = { q: "test-query" };
// Handle A claims via fence wrapper
const pA = runWithServerToolFence({
apiKeyId: "key-1",
requestIdentity: "key-1:req-id:body-hash",
toolCallId: "call-1",
toolName: "memory_search",
arguments: sharedArgs,
leaseDurationMs: 60_000,
execute: handler,
now,
sleep,
db: adapterA,
});
// Wait for A to start executing
await aStarted;
// Handle B calls the fence wrapper — should find running row, observe in_progress
const pB = runWithServerToolFence({
apiKeyId: "key-1",
requestIdentity: "key-1:req-id:body-hash",
toolCallId: "call-1",
toolName: "memory_search",
arguments: sharedArgs,
leaseDurationMs: 60_000,
execute: async () => "should-not-run",
now,
sleep,
db: adapterB,
});
// Injected sleep proves B entered polling before A may finalize.
await new Promise<void>((resolve) => setImmediate(resolve));
assert.ok(fakeTime > 1_000_000, "B must poll before A finalizes");
bObservedResolve();
const [, fenceResult] = await Promise.all([pA, pB]);
assert.equal(handlerCallCount, 1, "handler executes exactly once");
assert.equal(fenceResult.kind, "replayed");
if (fenceResult.kind === "replayed") {
assert.equal(fenceResult.status, "success");
}
rawA.close();
rawB.close();
fs.rmSync(dir, { recursive: true, force: true });
});
test("fence: in_progress poll returns in_progress when not finalized within deadline", async () => {
const { adapter, dir, raw } = makeTempDb();
let fakeTime = 1_000_000;
const now = () => fakeTime;
const sleep = async (ms: number) => {
fakeTime += ms;
await new Promise<void>((r) => setTimeout(r, 0));
};
const sharedArgs = { q: "test" };
const digest = require_("crypto")
.createHash("sha256")
.update(JSON.stringify(sharedArgs))
.digest("hex");
// Insert a running row directly (simulates another process that claimed but hasn't finalized)
raw.exec(`
INSERT INTO server_tool_executions
(id, api_key_id, request_identity, tool_call_id, tool_name, input_digest, status, claim_expires_at)
VALUES ('ext-id', 'key-1', 'key-1:req:body', 'call-1', 'tool_a', '${digest}', 'running', datetime('now', '+300 seconds'))
`);
const result = await runWithServerToolFence({
apiKeyId: "key-1",
requestIdentity: "key-1:req:body",
toolCallId: "call-1",
toolName: "tool_a",
arguments: sharedArgs,
leaseDurationMs: 60_000,
execute: async () => "should-not-run",
now,
sleep,
db: adapter,
});
assert.equal(result.kind, "in_progress");
assert.ok(fakeTime >= 1_000_000 + 2_000, "time should have advanced by poll budget");
cleanup(raw, dir);
});
// ── Defect 3: Duration test — drives runWithServerToolFence, advances injected clock ──
test("finalize: duration_ms persisted exactly via fence wrapper (success)", async () => {
const { adapter, dir, raw } = makeTempDb();
let fakeTime = 10_000;
const now = () => fakeTime;
const sleep = async (ms: number) => {
fakeTime += ms;
};
const result = await runWithServerToolFence({
apiKeyId: "key-1",
requestIdentity: "key-1:req:body",
toolCallId: "call-1",
toolName: "tool_a",
arguments: { x: 1 },
leaseDurationMs: 60_000,
execute: async (_execId) => {
fakeTime += 42;
return "done";
},
now,
sleep,
db: adapter,
});
assert.equal(result.kind, "executed");
// Query duration_ms from DB — wrapper must have calculated it, not hardcoded 0
const row = raw
.prepare("SELECT duration_ms, status FROM server_tool_executions WHERE status = 'success'")
.get() as { duration_ms: number | null; status: string };
assert.equal(row.duration_ms, 42, "duration_ms must be exactly 42 via injected clock");
assert.equal(row.status, "success");
cleanup(raw, dir);
});
test("finalize: duration_ms persisted exactly via fence wrapper (error)", async () => {
const { adapter, dir, raw } = makeTempDb();
let fakeTime = 10_000;
const now = () => fakeTime;
const sleep = async (ms: number) => {
fakeTime += ms;
};
await assert.rejects(
runWithServerToolFence({
apiKeyId: "key-1",
requestIdentity: "key-1:req:body",
toolCallId: "call-1",
toolName: "tool_a",
arguments: { x: 1 },
leaseDurationMs: 60_000,
execute: async () => {
fakeTime += 17;
throw new Error("boom");
},
now,
sleep,
db: adapter,
}),
/boom/,
"handler error must propagate"
);
const row = raw
.prepare("SELECT duration_ms, status FROM server_tool_executions WHERE status = 'error'")
.get() as { duration_ms: number | null; status: string };
assert.equal(row.duration_ms, 17, "duration_ms must be exactly 17 for error via injected clock");
assert.equal(row.status, "error");
cleanup(raw, dir);
});
// ── Defect 4: Rejected in-process active Promise → joiner gets replayed ──
test("fence: rejected in-process promise — joiner gets replayed with error status", async () => {
const { adapter, dir, raw } = makeTempDb();
let fakeTime = 1_000_000;
const now = () => fakeTime;
const sleep = async (ms: number) => {
fakeTime += ms;
await new Promise<void>((r) => setTimeout(r, 0));
};
// Gate for concurrency: A signals started, B signals it observed in_progress
let aStartedResolve!: () => void;
const aStarted = new Promise<void>((r) => {
aStartedResolve = r;
});
let bObservedResolve!: () => void;
const bObserved = new Promise<void>((r) => {
bObservedResolve = r;
});
let handlerCalled = false;
const failingHandler = async () => {
handlerCalled = true;
aStartedResolve();
await bObserved;
throw new Error("handler crashed");
};
// Handle A claims via fence — handler will throw, wrapper resolves with error status
const pA = runWithServerToolFence({
apiKeyId: "key-2",
requestIdentity: "key-2:req:body",
toolCallId: "call-x",
toolName: "tool_a",
arguments: {},
leaseDurationMs: 60_000,
execute: failingHandler,
now,
sleep,
db: adapter,
});
await aStarted;
// Handle B immediately tries same key — gets in_progress, polls the active promise
const pB = runWithServerToolFence({
apiKeyId: "key-2",
requestIdentity: "key-2:req:body",
toolCallId: "call-x",
toolName: "tool_a",
arguments: {},
leaseDurationMs: 60_000,
execute: async () => {
throw new Error("should-not-run");
},
now,
sleep,
db: adapter,
});
await new Promise((r) => setTimeout(r, 5));
bObservedResolve();
const [rA, rB] = await Promise.allSettled([pA, pB]);
assert.ok(handlerCalled, "handler should have been called");
assert.equal(rA.status, "rejected", "handle A should reject because handler threw");
assert.equal(rB.status, "fulfilled");
if (rB.status === "fulfilled") {
assert.equal(rB.value.kind, "replayed", "joiner must get replayed for terminal error");
if (rB.value.kind === "replayed") {
assert.equal(rB.value.status, "error", "replayed status must be error");
assert.equal(rB.value.errorMessage, "handler crashed");
}
}
cleanup(raw, dir);
});
// ── Defect 6: isUniqueConstraintError rejects non-UNIQUE constraints ──
test("isUniqueConstraintError: NOT NULL error propagates through claimServerToolExecution", () => {
const { dir, raw } = makeTempDb();
try {
// Build a fake adapter that wraps the real one but makes INSERT throw NOT NULL
const realDir = fs.mkdtempSync(path.join(os.tmpdir(), "fence-fake-"));
const fakeRaw = new BetterSqlite3(path.join(realDir, "test.db"));
fakeRaw.pragma("journal_mode = WAL");
fakeRaw.exec(FIXTURE_SCHEMA);
const fake = createBetterSqliteAdapter(fakeRaw);
// Proxy: intercept INSERT INTO server_tool_executions and inject NOT NULL error
const proxyAdapter = new Proxy(fake, {
get(target, prop) {
if (prop === "prepare") {
return (sql: string) => {
const stmt = (target.prepare as Function)(sql);
if (/INSERT INTO server_tool_executions/i.test(sql)) {
return {
...stmt,
run: (..._args: unknown[]) => {
const err = new Error("NOT NULL constraint failed") as Error & { code: string };
err.code = "SQLITE_CONSTRAINT_NOTNULL";
throw err;
},
};
}
return stmt;
};
}
return (target as Record<string, unknown>)[prop as string];
},
}) as SqliteAdapter;
assert.throws(
() => {
claimServerToolExecution(BASE_INPUT, proxyAdapter);
},
(err: unknown) => {
return (
err instanceof Error &&
err.message.includes("NOT NULL") &&
(err as { code?: string }).code === "SQLITE_CONSTRAINT_NOTNULL"
);
},
"NOT NULL constraint must propagate, not be swallowed by isUniqueConstraintError"
);
fakeRaw.close();
fs.rmSync(realDir, { recursive: true, force: true });
} finally {
cleanup(raw, dir);
}
});
test("isUniqueConstraintError: CHECK error propagates through claimServerToolExecution", () => {
const { dir, raw } = makeTempDb();
try {
const realDir = fs.mkdtempSync(path.join(os.tmpdir(), "fence-fake-"));
const fakeRaw = new BetterSqlite3(path.join(realDir, "test.db"));
fakeRaw.pragma("journal_mode = WAL");
fakeRaw.exec(FIXTURE_SCHEMA);
const fake = createBetterSqliteAdapter(fakeRaw);
const proxyAdapter = new Proxy(fake, {
get(target, prop) {
if (prop === "prepare") {
return (sql: string) => {
const stmt = (target.prepare as Function)(sql);
if (/INSERT INTO server_tool_executions/i.test(sql)) {
return {
...stmt,
run: (..._args: unknown[]) => {
const err = new Error("CHECK constraint failed") as Error & { code: string };
err.code = "SQLITE_CONSTRAINT_CHECK";
throw err;
},
};
}
return stmt;
};
}
return (target as Record<string, unknown>)[prop as string];
},
}) as SqliteAdapter;
assert.throws(
() => {
claimServerToolExecution(BASE_INPUT, proxyAdapter);
},
(err: unknown) => {
return (
err instanceof Error &&
err.message.includes("CHECK") &&
(err as { code?: string }).code === "SQLITE_CONSTRAINT_CHECK"
);
},
"CHECK constraint must propagate, not be swallowed by isUniqueConstraintError"
);
fakeRaw.close();
fs.rmSync(realDir, { recursive: true, force: true });
} finally {
cleanup(raw, dir);
}
});
// ── Existing tests (cleaned up, no done callbacks) ──
test("finalize: only updates running rows, second finalize returns false", () => {
const { adapter, dir, raw } = makeTempDb();
try {
const claim = claimServerToolExecution(BASE_INPUT, adapter);
assert.equal(claim.kind, "claimed");
const firstFinalize = finalizeServerToolExecution(
{ ...BASE_FINALIZE, executionId: claim.executionId },
adapter
);
assert.equal(firstFinalize, true);
const secondFinalize = finalizeServerToolExecution(
{ ...BASE_FINALIZE, executionId: claim.executionId },
adapter
);
assert.equal(secondFinalize, false);
} finally {
cleanup(raw, dir);
}
});
test("finalize: output sanitized — nested credentials stripped, valid JSON preserved", () => {
const { adapter, dir, raw } = makeTempDb();
try {
const claim = claimServerToolExecution(BASE_INPUT, adapter);
assert.equal(claim.kind, "claimed");
const sensitiveOutput = {
token: "sk-live-secret123",
nested: { bearer: "Bearer abcdefghijklmnop", deep: { key: "sk-test-abcdefghijklmnop" } },
data: "normal text",
};
finalizeServerToolExecution(
{ ...BASE_FINALIZE, executionId: claim.executionId, output: sensitiveOutput },
adapter
);
const row = raw
.prepare("SELECT output FROM server_tool_executions WHERE id = ?")
.get(claim.executionId) as { output: string | null };
assert.ok(row.output, "output should be stored");
assert.ok(
!row.output.includes("sk-live-secret123"),
"output must not contain raw sk- credential"
);
assert.ok(
!row.output.includes("Bearer abcdefghijklmnop"),
"output must not contain raw Bearer token"
);
const parsed = JSON.parse(row.output);
assert.ok(typeof parsed === "object", "sanitized output must be parseable JSON");
} finally {
cleanup(raw, dir);
}
});
test("finalize: error message sanitized — no raw stack in error_message", () => {
const { adapter, dir, raw } = makeTempDb();
try {
const claim = claimServerToolExecution(BASE_INPUT, adapter);
assert.equal(claim.kind, "claimed");
finalizeServerToolExecution(
{
executionId: claim.executionId,
status: "error",
output: null,
errorMessage: "Something failed\n at /internal/path.js:42\n at processTicksAndRejections",
durationMs: 50,
},
adapter
);
const row = raw
.prepare("SELECT error_message FROM server_tool_executions WHERE id = ?")
.get(claim.executionId) as { error_message: string | null };
assert.ok(row.error_message, "error_message should be stored");
assert.ok(!row.error_message.includes("at /internal/"), "error must not contain raw stack");
} finally {
cleanup(raw, dir);
}
});
test("finalize: output truncated to 32KB with valid JSON envelope", () => {
const { adapter, dir, raw } = makeTempDb();
try {
const claim = claimServerToolExecution(BASE_INPUT, adapter);
assert.equal(claim.kind, "claimed");
// Build a nested object that serializes to >32KB after sanitization
// (sanitizeUpstreamDetails truncates arrays to 32 elements, and strips 'key'-like names)
const bigObj: Record<string, string> = {};
for (let i = 0; i < 2000; i++) {
bigObj[`field_${i}_padding`] = "x".repeat(20);
}
finalizeServerToolExecution(
{ ...BASE_FINALIZE, executionId: claim.executionId, output: bigObj },
adapter
);
const row = raw
.prepare("SELECT output FROM server_tool_executions WHERE id = ?")
.get(claim.executionId) as { output: string | null };
assert.ok(row.output, "output must be stored");
assert.ok(row.output.length <= 32768, `output length ${row.output.length} must be <=32768`);
const parsed = JSON.parse(row.output);
assert.ok(parsed.truncated === true, "truncated envelope must have truncated:true");
assert.ok(typeof parsed.preview === "string", "truncated envelope must have preview string");
} finally {
cleanup(raw, dir);
}
});
test("finalize: raw tool arguments are never stored in the execution row", () => {
const { adapter, dir, raw } = makeTempDb();
try {
const claim = claimServerToolExecution(BASE_INPUT, adapter);
assert.equal(claim.kind, "claimed");
finalizeServerToolExecution(
{ ...BASE_FINALIZE, executionId: claim.executionId, output: { key: "value" } },
adapter
);
const row = raw
.prepare("SELECT output, input_digest FROM server_tool_executions WHERE id = ?")
.get(claim.executionId) as { output: string | null; input_digest: string };
assert.ok(row.output, "output should be stored");
assert.ok(row.input_digest, "input_digest must be digest only, not raw args");
} finally {
cleanup(raw, dir);
}
});
test("contention: two independent SQLite handles on same file — loser UNIQUE caught cleanly", () => {
const { adapter: adapter1, dir, raw: raw1 } = makeTempDb();
const { adapter: adapter2, raw: raw2 } = makeSecondAdapter(dir);
try {
const claim1 = claimServerToolExecution(BASE_INPUT, adapter1);
assert.equal(claim1.kind, "claimed");
const claim2 = claimServerToolExecution(BASE_INPUT, adapter2);
assert.ok(
claim2.kind === "replay" || claim2.kind === "in_progress",
`Expected replay or in_progress, got ${claim2.kind}`
);
} finally {
raw1.close();
raw2.close();
fs.rmSync(dir, { recursive: true, force: true });
}
});
test("fence: in-process promise joiner replays success result", async () => {
const { adapter, dir, raw } = makeTempDb();
let handlerCallCount = 0;
const handler = async () => {
handlerCallCount++;
return "handler-result";
};
const p1 = runWithServerToolFence(
{
apiKeyId: "key-1",
requestIdentity: "key-1:req:body",
toolCallId: "call-1",
toolName: "tool_a",
arguments: { q: "test" },
leaseDurationMs: 60_000,
execute: handler,
},
adapter
);
const p2 = runWithServerToolFence(
{
apiKeyId: "key-1",
requestIdentity: "key-1:req:body",
toolCallId: "call-1",
toolName: "tool_a",
arguments: { q: "test" },
leaseDurationMs: 60_000,
execute: handler,
},
adapter
);
const [r1, r2] = await Promise.all([p1, p2]);
assert.equal(handlerCallCount, 1, "handler should execute exactly once");
assert.equal(r1.kind, "executed");
assert.equal(r2.kind, "replayed");
if (r2.kind === "replayed") {
assert.equal(r2.status, "success");
assert.equal(r2.value, "handler-result");
}
cleanup(raw, dir);
});

41
tests/unit/skills-executor.test.ts
View File

@@ -293,3 +293,44 @@ test("skillExecutor turns handler errors and timeouts into error executions", as
assert.equal(timedOut.output, null);
assert.match(timedOut.errorMessage, /timed out/i);
});
// ─── Task 3: executeClaimed separation from execute RED tests ─────────────────
test("executeClaimed executes registered handler and returns SkillExecution without writing skill_executions row", async () => {
await registerEchoSkill();
skillExecutor.registerHandler("echo-handler", async (input, context) => ({
echoed: `${input.value}:${context.apiKeyId}`,
}));
const execution = await skillExecutor.executeClaimed(
"echo@1.0.0",
{ value: "claimed" },
{ apiKeyId: "key-a", sessionId: "session-claimed" },
"test-execution-id"
);
assert.equal(execution.status, "success");
assert.deepEqual(execution.output, { echoed: "claimed:key-a" });
// Must NOT write to skill_executions (only execute() does).
const count = skillExecutor.countExecutions("key-a");
assert.equal(count, 0, "executeClaimed must not write skill_executions row");
});
test("execute still writes exactly 1 skill_executions row (existing contract preserved)", async () => {
await registerEchoSkill();
skillExecutor.registerHandler("echo-handler", async (input) => ({
echoed: input.value,
}));
await skillExecutor.execute(
"echo@1.0.0",
{ value: "persist" },
{ apiKeyId: "key-a", sessionId: "session-persist" }
);
const count = skillExecutor.countExecutions("key-a");
assert.equal(count, 1, "execute must write exactly 1 skill_executions row");
});

795
tests/unit/skills-interception-server-owned.test.ts Normal file
View File

@@ -0,0 +1,795 @@
import test from "node:test";
import assert from "node:assert/strict";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
// Local shape for formatter return values — avoids explicit any in assertions.
type FormattedResponse = {
choices?: Array<{
message: {
content: string | null;
tool_calls?: Array<{ id: string; function: { name: string; arguments: string } }>;
};
finish_reason: string;
}>;
content?: Array<{ type: string; id?: string; text?: string }>;
stop_reason?: string;
stop_sequence?: string | null;
output?: Array<{ type: string; call_id?: string; name?: string; arguments?: string }>;
response?: {
output?: Array<{ type: string; call_id?: string; name?: string; arguments?: string }>;
};
};
const TEST_ROOT = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-skills-interception-owned-"));
const TEST_DATA_DIR = path.join(TEST_ROOT, "data");
const TEST_PLUGINS_DIR = path.join(TEST_ROOT, "plugins");
const ORIGINAL_DATA_DIR = process.env.DATA_DIR;
const ORIGINAL_PLUGINS_DIR = process.env.OMNIROUTE_PLUGINS_DIR;
fs.mkdirSync(TEST_DATA_DIR, { recursive: true });
fs.mkdirSync(TEST_PLUGINS_DIR, { recursive: true });
process.env.DATA_DIR = TEST_DATA_DIR;
process.env.OMNIROUTE_PLUGINS_DIR = TEST_PLUGINS_DIR;
const coreDb = await import("../../src/lib/db/core.ts");
const { skillRegistry } = await import("../../src/lib/skills/registry.ts");
const { skillExecutor } = await import("../../src/lib/skills/executor.ts");
const {
classifyServerOwnedCalls,
formatEscapeHatchResponse,
executeServerOwned,
ServerOwnedExecutionError,
} = await import("../../src/lib/skills/interception.ts");
function resetRuntime() {
skillRegistry["registeredSkills"].clear();
skillRegistry["versionCache"].clear();
skillExecutor["handlers"].clear();
skillExecutor.setTimeout(50);
}
async function resetStorage() {
resetRuntime();
coreDb.resetDbInstance();
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });
fs.mkdirSync(TEST_DATA_DIR, { recursive: true });
}
async function registerRuntimeSkills() {
await skillRegistry.register({
name: "lookup",
version: "1.0.0",
description: "lookup records",
schema: { input: { id: "string" }, output: { record: "string" } },
handler: "lookup-handler",
enabled: true,
apiKeyId: "key-a",
});
await skillRegistry.register({
name: "broken",
version: "1.0.0",
description: "always fails",
schema: { input: {}, output: {} },
handler: "broken-handler",
enabled: true,
apiKeyId: "key-a",
});
skillExecutor.registerHandler("lookup-handler", async (input) => ({
record: `resolved:${input.id}`,
}));
skillExecutor.registerHandler("broken-handler", async () => {
throw new Error("skill failure");
});
}
test.beforeEach(async () => {
process.env.DATA_DIR = TEST_DATA_DIR;
process.env.OMNIROUTE_PLUGINS_DIR = TEST_PLUGINS_DIR;
await resetStorage();
await registerRuntimeSkills();
});
test.after(() => {
resetRuntime();
coreDb.resetDbInstance();
if (ORIGINAL_DATA_DIR === undefined) delete process.env.DATA_DIR;
else process.env.DATA_DIR = ORIGINAL_DATA_DIR;
if (ORIGINAL_PLUGINS_DIR === undefined) delete process.env.OMNIROUTE_PLUGINS_DIR;
else process.env.OMNIROUTE_PLUGINS_DIR = ORIGINAL_PLUGINS_DIR;
fs.rmSync(TEST_ROOT, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });
});
// ─── Task 3: classifyServerOwnedCalls + formatEscapeHatchResponse RED tests ──
test("classifyServerOwnedCalls: owner set builtin/custom → serverOwned; registry-registered but not in owner set → clientNative", async () => {
const calls = [
{ id: "c1", name: "http_request", arguments: {} },
{ id: "c2", name: "lookup@1.0.0", arguments: {} },
{ id: "c3", name: "Bash", arguments: {} },
];
const result = await classifyServerOwnedCalls(calls, {
apiKeyId: "key-a",
sessionId: "s1",
requestId: "r1",
builtinToolNames: ["http_request"],
injectedCustomSkillNames: ["lookup@1.0.0"],
customSkillExecutionEnabled: true,
});
assert.equal(result.serverOwned.length, 2);
assert.equal(result.serverOwned[0].id, "c1");
assert.equal(result.serverOwned[1].id, "c2");
assert.equal(result.clientNative.length, 1);
assert.equal(result.clientNative[0].id, "c3");
});
test("classifyServerOwnedCalls: client same-name memory_search → not server-owned", async () => {
const calls = [
{ id: "c1", name: "memory_search", arguments: {} },
{ id: "c2", name: "http_request", arguments: {} },
];
const result = await classifyServerOwnedCalls(calls, {
apiKeyId: "key-a",
sessionId: "s1",
requestId: "r1",
builtinToolNames: ["http_request"],
// memory_search NOT in builtinToolNames (client owns it)
injectedCustomSkillNames: [],
customSkillExecutionEnabled: true,
});
// memory_search is client-native because it's not in any owner set.
assert.equal(result.clientNative.length, 1);
assert.equal(result.clientNative[0].id, "c1");
assert.equal(result.serverOwned.length, 1);
assert.equal(result.serverOwned[0].id, "c2");
});
test("classifyServerOwnedCalls: registered skill with client same-name → client-native (not server-owned)", async () => {
// Register a skill that the client also declares with the same encoded name.
await skillRegistry.register({
name: "collision-check",
version: "1.0.0",
description: "collision test",
schema: { input: {}, output: {} },
handler: "collision-handler",
enabled: true,
apiKeyId: "key-a",
mode: "on",
});
const encodedName = (await import("../../src/lib/skills/injection.ts")).encodeSkillToolName(
"collision-check",
"1.0.0"
);
// Client declares a tool with the same encoded name.
const calls = [
{ id: "c1", name: encodedName, arguments: {} },
{ id: "c2", name: "http_request", arguments: {} },
];
const result = await classifyServerOwnedCalls(calls, {
apiKeyId: "key-a",
sessionId: "s1",
requestId: "r1",
builtinToolNames: ["http_request"],
injectedCustomSkillNames: [], // empty: client collision prevented injection
customSkillExecutionEnabled: true,
});
// The registered skill with client same-name must be client-native.
assert.equal(result.clientNative.length, 1);
assert.equal(result.clientNative[0].id, "c1");
assert.equal(result.serverOwned.length, 1);
assert.equal(result.serverOwned[0].id, "c2");
skillRegistry["registeredSkills"].clear();
skillRegistry["versionCache"].clear();
});
test("formatEscapeHatchResponse: mixed OpenAI — strip server calls, append results to content, keep client calls, finish_reason:tool_calls", async () => {
const response = {
choices: [
{
message: {
content: null,
tool_calls: [
{ id: "srv1", function: { name: "http_request", arguments: "{}" } },
{ id: "cli1", function: { name: "Bash", arguments: '{"cmd":"ls"}' } },
],
},
finish_reason: "tool_calls",
},
],
};
const serverCalls = [{ id: "srv1", name: "http_request", arguments: {} }];
const clientCalls = [{ id: "cli1", name: "Bash", arguments: { cmd: "ls" } }];
const results = [{ id: "srv1", name: "http_request", result: { status: 200 }, replayed: false }];
const formatted = formatEscapeHatchResponse(
response,
serverCalls,
results,
clientCalls,
"openai"
);
const choice = (formatted as FormattedResponse).choices[0];
// Server call stripped from tool_calls, client call kept.
assert.equal(choice.message.tool_calls.length, 1);
assert.equal(choice.message.tool_calls[0].id, "cli1");
// Server result appended to content.
assert.ok(typeof choice.message.content === "string");
assert.ok(choice.message.content.includes("200"));
// finish_reason stays tool_calls (mixed).
assert.equal(choice.finish_reason, "tool_calls");
});
test("formatEscapeHatchResponse: all-server OpenAI — strip tool calls, append results, finish_reason:stop", async () => {
const response = {
choices: [
{
message: {
content: null,
tool_calls: [{ id: "srv1", function: { name: "http_request", arguments: "{}" } }],
},
finish_reason: "tool_calls",
},
],
};
const serverCalls = [{ id: "srv1", name: "http_request", arguments: {} }];
const results = [{ id: "srv1", name: "http_request", result: { ok: true }, replayed: false }];
const formatted = formatEscapeHatchResponse(response, serverCalls, results, [], "openai");
const choice = (formatted as FormattedResponse).choices[0];
assert.ok(
!choice.message.tool_calls || choice.message.tool_calls.length === 0,
"all-server must have no remaining tool_calls"
);
assert.ok(typeof choice.message.content === "string");
assert.ok(choice.message.content.includes("ok"));
assert.equal(choice.finish_reason, "stop");
});
test("formatEscapeHatchResponse: Claude mixed — strip server tool_use, keep client tool_use, end_turn stays", async () => {
const response = {
content: [
{ type: "tool_use", id: "srv1", name: "http_request", input: {} },
{ type: "tool_use", id: "cli1", name: "Bash", input: { cmd: "ls" } },
],
stop_reason: "tool_use",
};
const serverCalls = [{ id: "srv1", name: "http_request", arguments: {} }];
const clientCalls = [{ id: "cli1", name: "Bash", arguments: { cmd: "ls" } }];
const results = [{ id: "srv1", name: "http_request", result: { ok: true }, replayed: false }];
const formatted = formatEscapeHatchResponse(
response,
serverCalls,
results,
clientCalls,
"claude"
);
const content = (formatted as FormattedResponse).content as Array<{ type: string; id?: string }>;
// Server tool_use removed, client tool_use kept.
const toolUses = content.filter((b) => b.type === "tool_use");
assert.equal(toolUses.length, 1);
assert.equal(toolUses[0].id, "cli1");
// stop_reason stays tool_use (mixed).
assert.equal((formatted as FormattedResponse).stop_reason, "tool_use");
});
test("formatEscapeHatchResponse: Claude all-server — strip tool_use, append text, end_turn", async () => {
const response = {
content: [{ type: "tool_use", id: "srv1", name: "http_request", input: {} }],
stop_reason: "tool_use",
};
const serverCalls = [{ id: "srv1", name: "http_request", arguments: {} }];
const results = [{ id: "srv1", name: "http_request", result: { ok: true }, replayed: false }];
const formatted = formatEscapeHatchResponse(response, serverCalls, results, [], "claude");
const content = (formatted as FormattedResponse).content as Array<{ type: string }>;
const toolUses = content.filter((b) => b.type === "tool_use");
assert.equal(toolUses.length, 0);
const textBlocks = content.filter((b) => b.type === "text");
assert.ok(textBlocks.length > 0);
assert.equal((formatted as FormattedResponse).stop_reason, "end_turn");
});
test("formatEscapeHatchResponse: formatter does not call interceptToolCalls or any handler (purity)", async () => {
// Purity proof: the formatter is synchronous and its source must not contain
// calls to interceptToolCalls, skillExecutor, or handler invocations.
const fnSource = formatEscapeHatchResponse.toString();
assert.ok(
!fnSource.includes("interceptToolCalls"),
"formatter source must not reference interceptToolCalls"
);
assert.ok(
!fnSource.includes("skillExecutor"),
"formatter source must not reference skillExecutor"
);
assert.ok(!fnSource.includes("await"), "formatter must be synchronous (no await)");
// Also verify it returns immediately without side effects.
const response = {
choices: [
{
message: {
content: null,
tool_calls: [{ id: "srv1", function: { name: "http_request", arguments: "{}" } }],
},
finish_reason: "tool_calls",
},
],
};
const result = formatEscapeHatchResponse(
response,
[{ id: "srv1", name: "http_request", arguments: {} }],
[{ id: "srv1", name: "http_request", result: { ok: true }, replayed: false }],
[],
"openai"
);
assert.ok(result, "formatter returns a result");
assert.ok(result.choices[0].message.content, "formatter populates content");
});
test("formatEscapeHatchResponse: Responses wrapper is byte-identical for function_call_output", async () => {
const response = {
object: "response",
output: [{ type: "function_call", call_id: "call1", name: "lookup@1.0.0", arguments: "{}" }],
};
const serverCalls = [{ id: "call1", name: "lookup@1.0.0", arguments: {} }];
const results = [
{ id: "call1", name: "lookup@1.0.0", result: { record: "42" }, replayed: false },
];
const formatted = formatEscapeHatchResponse(response, serverCalls, results, [], "openai");
// Responses format: original output + function_call_output appended.
const output = (formatted as FormattedResponse).output;
assert.equal(output.length, 2);
assert.equal(output[0].type, "function_call");
assert.equal(output[1].type, "function_call_output");
assert.equal(output[1].call_id, "call1");
});
// ─── F3: nested Responses output formatter ──────────────────────────────────
test("formatEscapeHatchResponse: nested {response:{output}} appends function_call_output to nested output, not top-level", async () => {
const nestedResponse = {
object: "response",
response: {
output: [{ type: "function_call", call_id: "nc1", name: "lookup@1.0.0", arguments: "{}" }],
},
};
const serverCalls = [{ id: "nc1", name: "lookup@1.0.0", arguments: {} }];
const results = [
{ id: "nc1", name: "lookup@1.0.0", result: { record: "nested-42" }, replayed: false },
];
const formatted = formatEscapeHatchResponse(nestedResponse, serverCalls, results, [], "openai");
// Must append to nested response.output, not top-level output.
const nestedOutput = (formatted as { response?: { output?: unknown[] } }).response?.output;
assert.ok(Array.isArray(nestedOutput), "nested response.output must be an array");
assert.equal(nestedOutput.length, 2, "nested output must have original + appended");
assert.equal(nestedOutput[0].type, "function_call");
assert.equal(nestedOutput[1].type, "function_call_output");
assert.equal((nestedOutput[1] as { call_id: string }).call_id, "nc1");
// Top-level must NOT have an output array.
assert.equal(
Array.isArray((formatted as { output?: unknown[] }).output),
false,
"top-level output must not exist"
);
});
// ─── F1: executeServerOwned RED tests ───────────────────────────────────────
test("executeServerOwned: requires requestIdentity when executionFenceEnabled", async () => {
const calls = [{ id: "c1", name: "http_request", arguments: { url: "https://example.com" } }];
const context = {
apiKeyId: "key-fence",
sessionId: "s1",
requestId: "r1",
builtinToolNames: ["http_request"],
executionFenceEnabled: true,
// requestIdentity is deliberately missing
};
await assert.rejects(
() => executeServerOwned(calls, context),
/requestIdentity/,
"must require requestIdentity when executionFenceEnabled"
);
});
test("executeServerOwned: dispatches memory builtin and returns ExecutedToolResult", async () => {
const calls = [{ id: "c1", name: "memory_search", arguments: { query: "test" } }];
const context = {
apiKeyId: "key-mem",
sessionId: "s1",
requestId: "r1",
builtinToolNames: ["memory_search"],
executionFenceEnabled: false,
};
const results = await executeServerOwned(calls, context);
assert.equal(results.length, 1);
assert.equal(results[0].id, "c1");
assert.equal(results[0].name, "memory_search");
assert.equal(results[0].replayed, false);
assert.ok(results[0].result !== undefined, "result must be present");
});
test("executeServerOwned: dispatches ordinary builtin (http_request)", async () => {
const calls = [{ id: "c1", name: "http_request", arguments: { url: "https://example.com" } }];
const context = {
apiKeyId: "key-builtin",
sessionId: "s1",
requestId: "r1",
builtinToolNames: ["http_request"],
executionFenceEnabled: false,
};
const results = await executeServerOwned(calls, context);
assert.equal(results.length, 1);
assert.equal(results[0].id, "c1");
assert.equal(results[0].name, "http_request");
assert.equal(results[0].replayed, false);
});
test("executeServerOwned: surfaces identity_conflict as typed error, never feeds to model", async () => {
// Custom skill call with no matching handler — should surface as error
const calls = [{ id: "c1", name: "missing-skill@1.0.0", arguments: {} }];
const context = {
apiKeyId: "key-err",
sessionId: "s1",
requestId: "r1",
injectedCustomSkillNames: ["missing-skill@1.0.0"],
customSkillExecutionEnabled: true,
executionFenceEnabled: false,
};
const results = await executeServerOwned(calls, context);
assert.equal(results.length, 1);
assert.equal(results[0].id, "c1");
assert.equal(results[0].replayed, false);
// Result must contain an error indicator
const resultRecord = results[0].result as Record<string, unknown>;
assert.ok(
resultRecord && (resultRecord.error || resultRecord.status),
"result must contain error indicator"
);
});
// ─── Fix Round 2: Defect 1 — typed errors for fence control-flow states ────
test("ServerOwnedExecutionError is an exported class with code and httpStatus", () => {
const err = new ServerOwnedExecutionError("test", "TEST_CODE", 409);
assert.ok(err instanceof Error);
assert.equal(err.code, "TEST_CODE");
assert.equal(err.httpStatus, 409);
assert.equal(err.message, "test");
});
test("executeServerOwned: in_progress fence state → throws ServerOwnedExecutionError with TOOL_IN_PROGRESS (409)", async () => {
const { setFenceFnForTesting } = await import("../../src/lib/skills/interception.ts");
const mockFence = async () => ({ kind: "in_progress" as const });
setFenceFnForTesting(mockFence);
try {
const calls = [{ id: "c1", name: "http_request", arguments: {} }];
const context = {
apiKeyId: "key-fence",
sessionId: "s1",
requestId: "r1",
builtinToolNames: ["http_request"],
executionFenceEnabled: true,
requestIdentity: "identity-1",
};
try {
await executeServerOwned(calls, context);
assert.fail("must throw");
} catch (e: unknown) {
assert.ok(e instanceof ServerOwnedExecutionError);
assert.equal(e.code, "TOOL_IN_PROGRESS");
assert.equal(e.httpStatus, 409);
}
} finally {
setFenceFnForTesting(null);
}
});
test("executeServerOwned: unknown fence state → throws ServerOwnedExecutionError with TOOL_STATE_UNKNOWN (500)", async () => {
const { setFenceFnForTesting } = await import("../../src/lib/skills/interception.ts");
const mockFence = async () => ({ kind: "unknown" as const });
setFenceFnForTesting(mockFence);
try {
const calls = [{ id: "c1", name: "http_request", arguments: {} }];
const context = {
apiKeyId: "key-fence",
sessionId: "s1",
requestId: "r1",
builtinToolNames: ["http_request"],
executionFenceEnabled: true,
requestIdentity: "identity-1",
};
try {
await executeServerOwned(calls, context);
assert.fail("must throw");
} catch (e: unknown) {
assert.ok(e instanceof ServerOwnedExecutionError);
assert.equal(e.code, "TOOL_STATE_UNKNOWN");
assert.equal(e.httpStatus, 500);
}
} finally {
setFenceFnForTesting(null);
}
});
test("executeServerOwned: identity_conflict fence state → throws ServerOwnedExecutionError with IDENTITY_CONFLICT (409)", async () => {
const { setFenceFnForTesting } = await import("../../src/lib/skills/interception.ts");
const mockFence = async () => ({ kind: "identity_conflict" as const });
setFenceFnForTesting(mockFence);
try {
const calls = [{ id: "c1", name: "http_request", arguments: {} }];
const context = {
apiKeyId: "key-fence",
sessionId: "s1",
requestId: "r1",
builtinToolNames: ["http_request"],
executionFenceEnabled: true,
requestIdentity: "identity-1",
};
try {
await executeServerOwned(calls, context);
assert.fail("must throw");
} catch (e: unknown) {
assert.ok(e instanceof ServerOwnedExecutionError);
assert.equal(e.code, "IDENTITY_CONFLICT");
assert.equal(e.httpStatus, 409);
}
} finally {
setFenceFnForTesting(null);
}
});
// ─── Fix Round 3: Defect 1 — replay error/timeout detection ───────────────
test("executeServerOwned: error replay → throws ServerOwnedExecutionError with TOOL_EXECUTION_ERROR (500)", async () => {
const { setFenceFnForTesting } = await import("../../src/lib/skills/interception.ts");
const mockFence = async () => ({
kind: "replayed" as const,
value: null,
status: "error" as const,
errorMessage: "handler crashed",
});
setFenceFnForTesting(mockFence);
try {
const calls = [{ id: "c1", name: "http_request", arguments: {} }];
const context = {
apiKeyId: "key-fence",
sessionId: "s1",
requestId: "r1",
builtinToolNames: ["http_request"],
executionFenceEnabled: true,
requestIdentity: "identity-1",
};
try {
await executeServerOwned(calls, context);
assert.fail("must throw");
} catch (e: unknown) {
assert.ok(e instanceof ServerOwnedExecutionError);
assert.equal(e.code, "TOOL_EXECUTION_ERROR");
assert.equal(e.httpStatus, 500);
assert.equal(e.message, "handler crashed");
}
} finally {
setFenceFnForTesting(null);
}
});
test("executeServerOwned: timeout replay → throws ServerOwnedExecutionError with TOOL_EXECUTION_TIMEOUT (504)", async () => {
const { setFenceFnForTesting } = await import("../../src/lib/skills/interception.ts");
const mockFence = async () => ({
kind: "replayed" as const,
value: null,
status: "timeout" as const,
errorMessage: "execution exceeded deadline",
});
setFenceFnForTesting(mockFence);
try {
const calls = [{ id: "c1", name: "http_request", arguments: {} }];
const context = {
apiKeyId: "key-fence",
sessionId: "s1",
requestId: "r1",
builtinToolNames: ["http_request"],
executionFenceEnabled: true,
requestIdentity: "identity-1",
};
try {
await executeServerOwned(calls, context);
assert.fail("must throw");
} catch (e: unknown) {
assert.ok(e instanceof ServerOwnedExecutionError);
assert.equal(e.code, "TOOL_EXECUTION_TIMEOUT");
assert.equal(e.httpStatus, 504);
assert.equal(e.message, "execution exceeded deadline");
}
} finally {
setFenceFnForTesting(null);
}
});
test("executeServerOwned: success replay → returns ExecutedToolResult with replayed:true", async () => {
const { setFenceFnForTesting } = await import("../../src/lib/skills/interception.ts");
const mockFence = async () => ({
kind: "replayed" as const,
value: { cached: true },
status: "success" as const,
errorMessage: null,
});
setFenceFnForTesting(mockFence);
try {
const calls = [{ id: "c1", name: "http_request", arguments: {} }];
const context = {
apiKeyId: "key-fence",
sessionId: "s1",
requestId: "r1",
builtinToolNames: ["http_request"],
executionFenceEnabled: true,
requestIdentity: "identity-1",
};
const results = await executeServerOwned(calls, context);
assert.equal(results.length, 1);
assert.equal(results[0].id, "c1");
assert.equal(results[0].replayed, true);
assert.deepEqual(results[0].result, { cached: true });
} finally {
setFenceFnForTesting(null);
}
});
test("executeServerOwned: error replay with null errorMessage → uses default message", async () => {
const { setFenceFnForTesting } = await import("../../src/lib/skills/interception.ts");
const mockFence = async () => ({
kind: "replayed" as const,
value: null,
status: "error" as const,
errorMessage: null,
});
setFenceFnForTesting(mockFence);
try {
const calls = [{ id: "c1", name: "http_request", arguments: {} }];
const context = {
apiKeyId: "key-fence",
sessionId: "s1",
requestId: "r1",
builtinToolNames: ["http_request"],
executionFenceEnabled: true,
requestIdentity: "identity-1",
};
try {
await executeServerOwned(calls, context);
assert.fail("must throw");
} catch (e: unknown) {
assert.ok(e instanceof ServerOwnedExecutionError);
assert.equal(e.code, "TOOL_EXECUTION_ERROR");
assert.equal(e.message, "Tool execution failed");
}
} finally {
setFenceFnForTesting(null);
}
});
// ─── Fix Round 2: Defect 2 — executeClaimed non-success throws ──────────────
test("executeClaimed: custom skill handler returns non-success status → executeClaimed throws safe error", async () => {
// Register a custom skill whose handler returns a failure output
await skillRegistry.register({
name: "fail-skill",
version: "1.0.0",
description: "always returns failure status",
schema: { input: {}, output: {} },
handler: "fail-handler",
enabled: true,
apiKeyId: "key-a",
});
skillExecutor.registerHandler("fail-handler", async () => ({
status: "failed",
message: "something went wrong",
}));
await assert.rejects(
() =>
skillExecutor.executeClaimed(
"fail-skill",
{},
{ apiKeyId: "key-a", sessionId: "s1" },
"exec-fail"
),
/Skill execution failed/,
"executeClaimed must throw when handler returns non-success status"
);
skillRegistry["registeredSkills"].clear();
skillRegistry["versionCache"].clear();
});
// ─── Fix Round 2: Defect 3 — LEASE_DURATION_MS = 120000 ─────────────────────
test("LEASE_DURATION_MS is 120000 to match loop wall-clock upper bound", async () => {
const fs = await import("node:fs");
const sourceCode = fs.readFileSync(
new URL("../../src/lib/skills/interception.ts", import.meta.url),
"utf8"
);
const match = sourceCode.match(/const\s+LEASE_DURATION_MS\s*=\s*([\d_]+)/);
assert.ok(match, "LEASE_DURATION_MS must be defined in interception.ts");
const value = Number(match[1].replace(/_/g, ""));
assert.equal(value, 120_000, "LEASE_DURATION_MS must be 120000 (not 30000)");
});
// ─── Fix Round 2: Defect 4 — classifyServerOwnedCalls no DB load ─────────────
test("classifyServerOwnedCalls does not call skillRegistry.loadFromDatabase (ownership from owner sets only)", async () => {
let loadFromDatabaseCalled = false;
const origLoad = skillRegistry.loadFromDatabase.bind(skillRegistry);
skillRegistry.loadFromDatabase = async (..._args: unknown[]) => {
loadFromDatabaseCalled = true;
return origLoad(...(_args as [string]));
};
try {
const calls = [{ id: "c1", name: "http_request", arguments: {} }];
await classifyServerOwnedCalls(calls, {
apiKeyId: "key-a",
sessionId: "s1",
requestId: "r1",
builtinToolNames: ["http_request"],
injectedCustomSkillNames: [],
customSkillExecutionEnabled: false,
});
assert.equal(
loadFromDatabaseCalled,
false,
"classifyServerOwnedCalls must NOT call loadFromDatabase — owner sets are sufficient"
);
} finally {
skillRegistry.loadFromDatabase = origLoad;
}
});

239
tests/unit/stable-json.test.ts Normal file
View File

@@ -0,0 +1,239 @@
import test from "node:test";
import assert from "node:assert/strict";
import fs from "node:fs";
import path from "node:path";
import { fileURLToPath } from "node:url";
const __dirname = path.dirname(fileURLToPath(import.meta.url));
import {
canonicalJson,
canonicalJsonSha256,
deriveToolRequestIdentity,
} from "../../src/lib/skills/stableJson.ts";
test("canonicalJson sorts nested object keys and preserves array order", () => {
const a = { z: [{ b: 2, a: 1 }], a: true };
const b = { a: true, z: [{ a: 1, b: 2 }] };
assert.equal(canonicalJson(a), canonicalJson(b));
assert.equal(canonicalJsonSha256(a), canonicalJsonSha256(b));
assert.notEqual(canonicalJsonSha256({ a: [1, 2] }), canonicalJsonSha256({ a: [2, 1] }));
assert.equal(
deriveToolRequestIdentity({
apiKeyId: "key-a",
stableClientRequestId: "req-a",
skillRequestId: "internal-1",
postInjectionBody: a,
}),
deriveToolRequestIdentity({
apiKeyId: "key-a",
stableClientRequestId: "req-a",
skillRequestId: "internal-2",
postInjectionBody: b,
})
);
});
test("canonicalJson rejects values that cannot form an execution identity", () => {
const cyclic: Record<string, unknown> = {};
cyclic.self = cyclic;
for (const value of [undefined, 1n, Number.NaN, Number.POSITIVE_INFINITY, cyclic]) {
assert.throws(() => canonicalJson(value), /canonical JSON/i);
}
assert.throws(() => canonicalJson({ bad: () => 1 }), /canonical JSON/i);
assert.throws(() => canonicalJson({ bad: Symbol("x") }), /canonical JSON/i);
});
test("canonicalJson sorts keys by Unicode code point, not UTF-16 code unit", () => {
// \uE000 (BMP private-use, code point 57344) vs "a" (code point 97)
// Code-point sort: a=97 < \uE000=57344
const withBmp = { "\uE000": 1, a: 2 };
const withAstral = { "\u{10000}": 1, a: 2 };
const sorted1 = canonicalJson(withBmp);
const sorted2 = canonicalJson(withAstral);
// a < \uE000 in code-point order
const bmpKey = "\uE000";
assert.ok(
sorted1.indexOf('"a"') < sorted1.indexOf(bmpKey),
`expected "a" before BMP key in: ${sorted1}`
);
// a < \u{10000} in code-point order; astral is encoded as surrogate pair
const astralKey = "\u{10000}";
assert.ok(
sorted2.indexOf('"a"') < sorted2.indexOf(astralKey),
`expected "a" before astral key in: ${sorted2}`
);
});
test("canonicalJson normalizes -0 to 0", () => {
assert.equal(canonicalJson({ val: -0 }), '{"val":0}');
assert.equal(canonicalJson({ val: 0 }), '{"val":0}');
assert.equal(canonicalJsonSha256({ val: -0 }), canonicalJsonSha256({ val: 0 }));
});
test("canonicalJson rejects sparse arrays", () => {
const sparse = [1, , 3];
assert.throws(() => canonicalJson(sparse), /canonical JSON/i);
});
test("canonicalJson rejects objects with getters/accessors", () => {
const obj: Record<string, unknown> = {};
Object.defineProperty(obj, "hidden", {
get() {
return 42;
},
enumerable: true,
});
assert.throws(() => canonicalJson(obj), /canonical JSON/i);
});
test("canonicalJson rejects non-plain objects (Date, Map, Set, class instances)", () => {
assert.throws(() => canonicalJson(new Date()), /canonical JSON/i);
assert.throws(() => canonicalJson(new Map()), /canonical JSON/i);
assert.throws(() => canonicalJson(new Set()), /canonical JSON/i);
class Custom {}
assert.throws(() => canonicalJson(new Custom()), /canonical JSON/i);
});
test("deriveToolRequestIdentity uses stableClientRequestId when present", () => {
const body = { a: 1, b: 2 };
const withStable = deriveToolRequestIdentity({
apiKeyId: "key-1",
stableClientRequestId: "idempotent-req-1",
skillRequestId: "uuid-internal",
postInjectionBody: body,
});
const withDifferentInternal = deriveToolRequestIdentity({
apiKeyId: "key-1",
stableClientRequestId: "idempotent-req-1",
skillRequestId: "different-uuid",
postInjectionBody: body,
});
// stable key present => internal UUID ignored
assert.equal(withStable, withDifferentInternal);
});
test("deriveToolRequestIdentity uses skillRequestId when stableClientRequestId is null", () => {
const body = { a: 1 };
const r1 = deriveToolRequestIdentity({
apiKeyId: "key-1",
stableClientRequestId: null,
skillRequestId: "uuid-a",
postInjectionBody: body,
});
const r2 = deriveToolRequestIdentity({
apiKeyId: "key-1",
stableClientRequestId: null,
skillRequestId: "uuid-b",
postInjectionBody: body,
});
// Different skillRequestId => different identity
assert.notEqual(r1, r2);
});
test("deriveToolRequestIdentity includes apiKeyId and body digest", () => {
const body = { x: 1 };
const r1 = deriveToolRequestIdentity({
apiKeyId: "key-1",
stableClientRequestId: "req",
skillRequestId: "sr",
postInjectionBody: body,
});
const r2 = deriveToolRequestIdentity({
apiKeyId: "key-2",
stableClientRequestId: "req",
skillRequestId: "sr",
postInjectionBody: body,
});
assert.notEqual(r1, r2);
// Body change => different identity
const r3 = deriveToolRequestIdentity({
apiKeyId: "key-1",
stableClientRequestId: "req",
skillRequestId: "sr",
postInjectionBody: { x: 2 },
});
assert.notEqual(r1, r3);
});
test("canonicalJson sorts BMP PUA \uE000 before astral \u{10000} in same object", () => {
// U+E000 (BMP private-use, code point 57344) vs U+10000 (Linear B, code point 65536)
// code-point order: E000 < 10000
// UTF-16 code-unit order: \uD800 (surrogate of 10000) < \uE000 — reversed!
// So default .sort() would place \u{10000} before \uE000.
const obj = { "\u{10000}": 1, "\uE000": 2 };
const serialized = canonicalJson(obj);
const e000Pos = serialized.indexOf("\uE000");
const astralPos = serialized.indexOf("\u{10000}");
assert.ok(
e000Pos < astralPos,
`expected \\uE000 (pos ${e000Pos}) before \\u{10000} (pos ${astralPos}) in: ${serialized}`
);
});
test("canonicalJson rejects objects with getters without invoking the getter", () => {
let getterCallCount = 0;
const obj: Record<string, unknown> = {};
Object.defineProperty(obj, "hidden", {
get() {
getterCallCount++;
return 42;
},
enumerable: true,
});
assert.throws(() => canonicalJson(obj), /canonical JSON/i);
assert.equal(
getterCallCount,
0,
"getter must not be invoked when canonicalJson rejects the object"
);
});
test("termination union in toolLoopTypes.ts source matches expected set", () => {
const src = fs.readFileSync(
path.resolve(__dirname, "../../src/lib/skills/toolLoopTypes.ts"),
"utf8"
);
// Extract the termination union block from ServerOwnedToolLoopResult
const loopResultStart = src.indexOf("interface ServerOwnedToolLoopResult");
assert.ok(loopResultStart !== -1, "ServerOwnedToolLoopResult must exist in toolLoopTypes.ts");
const afterLoopResult = src.slice(loopResultStart);
const termStart = afterLoopResult.indexOf("termination:");
assert.ok(termStart !== -1, "termination must exist in ServerOwnedToolLoopResult");
const afterTerm = afterLoopResult.slice(termStart);
// Match until the closing brace of the interface
const closingBrace = afterTerm.indexOf("\n}");
assert.ok(closingBrace !== -1, "closing brace must follow termination union");
const unionText = afterTerm.slice(0, closingBrace);
const actual = new Set<string>();
for (const m of unionText.matchAll(/"([^"]+)"/g)) {
actual.add(m[1]);
}
const expected = new Set([
"completed",
"client_tools",
"mixed_tools",
"max_followups",
"tool_output_budget",
"deadline",
"client_abort",
"provider_error",
"connection_mismatch",
"execution_in_progress",
"execution_unknown",
"execution_identity_conflict",
"execution_error",
"execution_timeout",
]);
// Every expected member must appear in source
for (const val of expected) {
assert.ok(actual.has(val), `termination union in source must include "${val}"`);
}
// No extra unexpected members
for (const val of actual) {
assert.ok(expected.has(val), `unexpected termination member "${val}" in source`);
}
});

149
tests/unit/tool-loop-usage.test.ts Normal file
View File

@@ -0,0 +1,149 @@
import test from "node:test";
import assert from "node:assert/strict";
import { aggregateProviderLegUsage } from "../../src/lib/skills/serverOwnedToolLoop.ts";
import type { ProviderLegUsage } from "../../src/lib/skills/toolLoopTypes.ts";
// ─── Basic aggregation ────────────────────────────────────────────────────────
test("aggregateProviderLegUsage: sums prompt_tokens, completion_tokens, total_tokens", () => {
const a: ProviderLegUsage = {
prompt_tokens: 100,
completion_tokens: 20,
total_tokens: 120,
};
const b: ProviderLegUsage = {
prompt_tokens: 150,
completion_tokens: 80,
total_tokens: 230,
};
const result = aggregateProviderLegUsage([a, b]);
assert.strictEqual(result.prompt_tokens, 250);
assert.strictEqual(result.completion_tokens, 100);
assert.strictEqual(result.total_tokens, 350, "total_tokens rederived as prompt+completion");
});
// ─── Optional fields: present in at least one leg ────────────────────────────
test("aggregateProviderLegUsage: optional fields summed when present", () => {
const a: ProviderLegUsage = {
prompt_tokens: 10,
completion_tokens: 5,
total_tokens: 15,
cached_tokens: 3,
cache_read_input_tokens: 2,
cache_creation_input_tokens: 1,
reasoning_tokens: 4,
cost_in_usd_ticks: 100,
};
const b: ProviderLegUsage = {
prompt_tokens: 10,
completion_tokens: 5,
total_tokens: 15,
// no optional fields
};
const result = aggregateProviderLegUsage([a, b]);
assert.strictEqual(result.prompt_tokens, 20);
assert.strictEqual(result.completion_tokens, 10);
assert.strictEqual(result.total_tokens, 30);
assert.strictEqual(result.cached_tokens, 3);
assert.strictEqual(result.cache_read_input_tokens, 2);
assert.strictEqual(result.cache_creation_input_tokens, 1);
assert.strictEqual(result.reasoning_tokens, 4);
assert.strictEqual(result.cost_in_usd_ticks, 100);
});
// ─── All-null usage → null ───────────────────────────────────────────────────
test("aggregateProviderLegUsage: all null usages → returns zero usage (not null)", () => {
const result = aggregateProviderLegUsage([null, null]);
assert.strictEqual(result.prompt_tokens, 0);
assert.strictEqual(result.completion_tokens, 0);
assert.strictEqual(result.total_tokens, 0);
});
test("aggregateProviderLegUsage: mix of null and non-null → sums only non-null", () => {
const a: ProviderLegUsage = {
prompt_tokens: 100,
completion_tokens: 20,
total_tokens: 120,
};
const result = aggregateProviderLegUsage([a, null]);
assert.strictEqual(result.prompt_tokens, 100);
assert.strictEqual(result.completion_tokens, 20);
assert.strictEqual(result.total_tokens, 120);
});
// ─── total_tokens rederived ───────────────────────────────────────────────────
test("aggregateProviderLegUsage: total_tokens is always prompt+completion, not sum of raw total_tokens", () => {
const a: ProviderLegUsage = {
prompt_tokens: 100,
completion_tokens: 20,
total_tokens: 999, // wrong raw value
};
const b: ProviderLegUsage = {
prompt_tokens: 100,
completion_tokens: 20,
total_tokens: 999, // wrong raw value
};
const result = aggregateProviderLegUsage([a, b]);
assert.strictEqual(result.total_tokens, 240, "100+20 + 100+20 = 240, not 1998");
});
// ─── Optional fields absent when no leg defines them ──────────────────────────
test("aggregateProviderLegUsage: optional fields absent when no leg defines them", () => {
const a: ProviderLegUsage = {
prompt_tokens: 10,
completion_tokens: 5,
total_tokens: 15,
};
const result = aggregateProviderLegUsage([a]);
assert.strictEqual(result.cached_tokens, undefined);
assert.strictEqual(result.cache_read_input_tokens, undefined);
assert.strictEqual(result.cache_creation_input_tokens, undefined);
assert.strictEqual(result.reasoning_tokens, undefined);
assert.strictEqual(result.cost_in_usd_ticks, undefined);
});
// ─── Empty array ──────────────────────────────────────────────────────────────
test("aggregateProviderLegUsage: empty array → zeros", () => {
const result = aggregateProviderLegUsage([]);
assert.strictEqual(result.prompt_tokens, 0);
assert.strictEqual(result.completion_tokens, 0);
assert.strictEqual(result.total_tokens, 0);
});
// ─── Claude-style alias normalization is caller's responsibility ──────────────
test("aggregateProviderLegUsage: Claude input_tokens/output_tokens are NOT aliases — caller must normalize before aggregation", () => {
// Claude uses input_tokens/output_tokens, not prompt_tokens/completion_tokens.
// The caller must normalize before passing to aggregateProviderLegUsage.
// This test verifies that aggregateProviderLegUsage uses the standard field names.
const a: ProviderLegUsage = {
prompt_tokens: 100,
completion_tokens: 20,
total_tokens: 120,
};
const result = aggregateProviderLegUsage([a]);
assert.strictEqual(result.prompt_tokens, 100);
assert.strictEqual(result.completion_tokens, 20);
// If someone passes input_tokens (Claude alias), it would be ignored
const claudeStyle = {
prompt_tokens: 0,
completion_tokens: 0,
total_tokens: 0,
input_tokens: 50,
output_tokens: 10,
} as unknown as ProviderLegUsage;
const result2 = aggregateProviderLegUsage([claudeStyle]);
assert.strictEqual(result2.prompt_tokens, 0, "Claude alias not summed into prompt_tokens");
});