Files
OmniRoute/tests/unit/chatcore-claude-effort-variant.test.ts
Will Gordon 4795825513 fix(sse): make Claude effort/no-think catalog variants dispatchable on every provider (#9006)
* fix(executors): route Claude-via-Vertex through native rawPredict with real streaming

Claude models on Vertex AI were being sent through the generic OpenAI-
compatible partner endpoint, which 404s/errors for Claude on at least
some projects. Route them through Vertex's native Anthropic Messages
API (publishers/anthropic/.../rawPredict) instead, stripping the
body-level model field rawPredict rejects and injecting the required
anthropic_version field.

rawPredict only ever returns a complete JSON body, never real SSE
framing, so streaming requests now get a genuine Anthropic-format SSE
stream synthesized from that JSON (message_start/content_block_*/
message_delta/message_stop), which the existing claude-to-openai
response translator already knows how to parse.

Also fixes two response-format resolution bugs that silently dropped
a custom model's DB-stored targetFormat override whenever the model
id also existed in the static provider registry (as claude-sonnet-4-6
and claude-opus-4-7 do under vertex): resolveModelOrError had its own
ad-hoc resolution that never consulted the override, and even once
fixed, executeChatWithBreaker discarded the correctly-resolved format
before handleChatCore's own resolution ran a second time.

* docs: add changelog fragment for #8909

* refactor(sse): extract shared Claude effort-model predicate

* fix(sse): strip Claude effort-suffix ids for any provider serving a real Claude model

* fix(sse): keep no-think and CC-discovery catalog variant roots unprefixed

* fix(dashboard): re-qualify no-think playground model ids correctly

* fix(sse): scope Vertex 404s to a per-model lockout via passthroughModels

* docs: add changelog fragment for the Claude catalog/dispatch fix

* fix(sse): align regex naming and changelog formatting

* fix(sse): clarify effort-variant strip comment and add cross-module drift guard

* fix(sse): disambiguate Vertex connection-wide vs per-model 403s

* docs: document Vertex 403 disambiguation in changelog fragment

* fix(sse): correlate reason and resource within the same ErrorInfo detail

* fix(sse): extract Vertex error classifier and rebaseline frozen file sizes

* test: register vertex-passthrough-model-lockout in stryker tap.testFiles

* fix(sse): reconciles rebase-onto-tip drift for 9006

Two categories of inherited base-branch breakage surfaced when
rebasing onto release/v3.8.50's latest tip, both confirmed unrelated
to this PR's own diff:

- check:file-size: base.ts and chat.ts drifted further past their
  frozen caps via already-merged commits (7163081f5 and others) that
  didn't rebaseline after growing them. Documented and bumped in
  file-size-baseline.json.
- chat-helpers.test.ts: two gpt-5.5 routing assertions predate #9275
  (fix(routing): bare model ids route to codex first), which
  deliberately made gpt-5.5 route to codex unconditionally, regardless
  of which other providers are active. Confirmed via #9275's own
  commit message and code comments this is intentional, not a
  regression; verified reproducible on the raw base tip alone, with
  no changes from this PR involved. Updated both assertions and their
  names to match the new, intentional default.

* ci: re-trigger checks after GitHub Actions incident (2026-08-07, resolved)

* ci: re-trigger checks (previous push event was dropped)

* fix(quality): rebaseline combo-routing-engine.test.ts own-comment growth

The ALL_ACCOUNTS_INACTIVE->ALL_TARGETS_SKIPPED fix (a32aed738) added explanatory comments (+7 lines), pushing the file past its frozen 3457 cap. CI's PR-mode check:file-size caught it; local check-file-size.mjs was not re-run after that specific commit.
2026-08-11 10:02:48 -03:00

176 lines
6.9 KiB
TypeScript

// tests/unit/chatcore-claude-effort-variant.test.ts
// Characterization of applyClaudeEffortVariant — the Claude effort-suffix normalization extracted
// from handleChatCore (chatCore god-file decomposition, #3501). The VS Code "Effort" slider
// advertises claude-...-{low,medium,high,xhigh,max}; Anthropic has no such model, so the suffix is
// stripped to the base id and surfaced as reasoning_effort. Locks: the direct-Claude-lane
// unconditional strip (claude / claude-code-compatible), the predicate-gated strip for any other
// provider serving a real Claude model, the in-place body mutation (model + reasoning_effort), the
// sourceFormat==="claude" skip, the explicit-effort-wins rule, and the returned effectiveModel/log.
import { test } from "node:test";
import assert from "node:assert/strict";
import { applyClaudeEffortVariant } from "../../open-sse/handlers/chatCore/claudeEffortVariant.ts";
import { FORMATS } from "../../open-sse/translator/formats.ts";
test("claude provider + effort suffix → strips to base, mutates body model + reasoning_effort, returns log", () => {
const body: Record<string, unknown> = { model: "claude-sonnet-4-high", messages: [] };
const r = applyClaudeEffortVariant({
provider: "claude",
effectiveModel: "claude-sonnet-4-high",
body,
sourceFormat: FORMATS.OPENAI,
});
assert.equal(r.effectiveModel, "claude-sonnet-4");
assert.equal(body.model, "claude-sonnet-4");
assert.equal(body.reasoning_effort, "high");
assert.match(String(r.log), /stripped "-high" → claude-sonnet-4 \(reasoning_effort=high\)/);
});
test("claude-code-compatible provider triggers the same stripping", () => {
const body: Record<string, unknown> = { model: "claude-opus-4-xhigh", messages: [] };
const r = applyClaudeEffortVariant({
provider: "anthropic-compatible-cc-default",
effectiveModel: "claude-opus-4-xhigh",
body,
sourceFormat: FORMATS.OPENAI,
});
assert.equal(r.effectiveModel, "claude-opus-4");
assert.equal(body.model, "claude-opus-4");
assert.equal(body.reasoning_effort, "xhigh");
});
test("sourceFormat 'claude' strips the model but does NOT inject reasoning_effort", () => {
const body: Record<string, unknown> = { model: "claude-sonnet-4-medium", messages: [] };
const r = applyClaudeEffortVariant({
provider: "claude",
effectiveModel: "claude-sonnet-4-medium",
body,
sourceFormat: FORMATS.CLAUDE,
});
assert.equal(r.effectiveModel, "claude-sonnet-4");
assert.equal(body.model, "claude-sonnet-4");
assert.equal(body.reasoning_effort, undefined);
});
test("an explicit client reasoning_effort wins (not overwritten)", () => {
const body: Record<string, unknown> = {
model: "claude-sonnet-4-low",
reasoning_effort: "high",
messages: [],
};
const r = applyClaudeEffortVariant({
provider: "claude",
effectiveModel: "claude-sonnet-4-low",
body,
sourceFormat: FORMATS.OPENAI,
});
assert.equal(r.effectiveModel, "claude-sonnet-4");
assert.equal(body.reasoning_effort, "high"); // unchanged
});
test("explicit effort nested under reasoning.effort also wins", () => {
const body: Record<string, unknown> = {
model: "claude-sonnet-4-low",
reasoning: { effort: "medium" },
messages: [],
};
const r = applyClaudeEffortVariant({
provider: "claude",
effectiveModel: "claude-sonnet-4-low",
body,
sourceFormat: FORMATS.OPENAI,
});
assert.equal(body.reasoning_effort, undefined); // explicit reasoning.effort present → no injection
assert.equal(r.effectiveModel, "claude-sonnet-4");
});
test("no effort suffix → no change, no log", () => {
const body: Record<string, unknown> = { model: "claude-sonnet-4", messages: [] };
const r = applyClaudeEffortVariant({
provider: "claude",
effectiveModel: "claude-sonnet-4",
body,
sourceFormat: FORMATS.OPENAI,
});
assert.equal(r.effectiveModel, "claude-sonnet-4");
assert.equal(body.model, "claude-sonnet-4");
assert.equal(body.reasoning_effort, undefined);
assert.equal(r.log, null);
});
test("non-claude provider is a no-op even with an effort suffix", () => {
const body: Record<string, unknown> = { model: "gpt-5-high", messages: [] };
const r = applyClaudeEffortVariant({
provider: "openai",
effectiveModel: "gpt-5-high",
body,
sourceFormat: FORMATS.OPENAI,
});
assert.equal(r.effectiveModel, "gpt-5-high");
assert.equal(body.model, "gpt-5-high");
assert.equal(body.reasoning_effort, undefined);
assert.equal(r.log, null);
});
test("non-claude provider serving a real Claude model strips the effort suffix", () => {
const body: Record<string, unknown> = { model: "claude-sonnet-5-high", messages: [] };
const r = applyClaudeEffortVariant({
provider: "vertex",
effectiveModel: "claude-sonnet-5-high",
body,
sourceFormat: FORMATS.OPENAI,
});
assert.equal(r.effectiveModel, "claude-sonnet-5");
assert.equal(body.model, "claude-sonnet-5");
assert.equal(body.reasoning_effort, "high");
});
test("safety guard: non-claude provider with a non-Claude model ending in a suffix word is left unchanged", () => {
const body: Record<string, unknown> = { model: "custom-model-high", messages: [] };
const r = applyClaudeEffortVariant({
provider: "some-other-provider",
effectiveModel: "custom-model-high",
body,
sourceFormat: FORMATS.OPENAI,
});
assert.equal(r.effectiveModel, "custom-model-high");
assert.equal(body.model, "custom-model-high");
assert.equal(body.reasoning_effort, undefined);
assert.equal(r.log, null);
});
test("claude-code-compatible provider strips even an unregistered model id (direct lane short-circuits the predicate)", () => {
// Proves the "unconditional strip, zero regression" claim: isDirectClaudeLane short-circuits
// the `||`, so isKnownClaudeEffortBaseModel() is never consulted for claude/CC-compatible
// providers — unlike the safety-guard case above, which requires the predicate to pass.
const body: Record<string, unknown> = {
model: "totally-unregistered-model-xyz-high",
messages: [],
};
const r = applyClaudeEffortVariant({
provider: "anthropic-compatible-cc-default",
effectiveModel: "totally-unregistered-model-xyz-high",
body,
sourceFormat: FORMATS.OPENAI,
});
assert.equal(r.effectiveModel, "totally-unregistered-model-xyz");
assert.equal(body.model, "totally-unregistered-model-xyz");
assert.equal(body.reasoning_effort, "high");
});
test("no-think alias's explicit reasoning_effort:none is not overwritten by a stripped effort suffix", () => {
const body: Record<string, unknown> = {
model: "claude-sonnet-5-high",
reasoning_effort: "none",
messages: [],
};
const r = applyClaudeEffortVariant({
provider: "vertex",
effectiveModel: "claude-sonnet-5-high",
body,
sourceFormat: FORMATS.OPENAI,
});
assert.equal(r.effectiveModel, "claude-sonnet-5");
assert.equal(body.model, "claude-sonnet-5");
assert.equal(body.reasoning_effort, "none");
});