mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-08-16 20:22:21 +03:00
fix(guardrails): reroute zero-vision combos through the vision bridge (#10415)
* fix(guardrails): reroute zero-vision combos through the vision bridge Named combos whose model targets all lack vision support are never reroute-eligible: the bridge only attempts the describe path, and when describing cannot run or fails the raw images stay in the payload and the request dies in the combo capability filter with capability_mismatch. getComboVisionBridgeDecision now returns a "no-vision" verdict for combos with zero vision-capable targets, and preCall treats it as reroute-eligible with the same credential guards as single text-only models, falling back to describe only when no usable reroute target exists. * chore(changelog): fragment for #10415 vision bridge combo reroute * fix(guardrails): extend allNull stub fallback to no-vision combos Reviewer follow-up (#10415): the allNull stub-text fallback at the end of preCall only fired for comboVisionBridgeDecision === 'process'. In the compound-failure case for a zero-vision combo — reroute target without usable credentials AND every describe call failing — raw images were preserved and the original capability_mismatch recurred, because a no-vision combo has no target that can consume images. Include 'no-vision' in the guard: stub text is strictly better than raw bytes no combo target can consume. Adds a double-failure unit test. * ci: re-run dast-smoke (Build CLI bundle runner timeout flake) * fix(build): bound and retry the opencode-plugin npm install in prepublish The plugin's node_modules is gitignored, so every fresh CI checkout runs a full npm install inside @omniroute/opencode-plugin during build:cli. npm's unbounded fetch retries turn a stalled registry CDN connection (the recurring onnxruntime-class ETIMEDOUT flake) into a 20-30 minute hang — the DAST 'Build CLI bundle' step has been cancelled at the 30m cap repeatedly. - Bound npm fetch: --fetch-timeout 60s, 2 retries with capped backoff — a stalled connection now fails fast instead of hanging the job. - Retry the install up to 3 times with a 10s pause between attempts, so transient CDN failures recover in-build. Net effect: the step either completes (network OK) or fails quickly with a clear error (network down) — it can no longer eat the whole job budget. * ci(dast): use existing npm-ci-retry action instead of bare npm ci dast-smoke died at 'Run npm ci' with connect ETIMEDOUT to the onnxruntime-node binary CDN (Microsoft 150.171.x.x) — the same transient CDN flake class that has hit Vitest/Quality Gates before. quality.yml already wraps npm ci in ./.github/actions/npm-ci-retry (3 attempts, exponential backoff); dast-smoke was the one workflow still using a bare install. Use the existing action for consistency. * ci(quality): use the npm-ci-retry action on every install step Fast Quality Gates failed on the recurring onnxruntime-node postinstall ETIMEDOUT (Microsoft CDN 150.171.x.x) - the same transient flake that has hit Vitest and dast-smoke today. Only the Build job used the retry action; the other five jobs (Docs, Fast Quality Gates, Vitest, Unit Tests, changelog) still ran a bare install and die on any CDN hiccup. Use the existing retry action (3 attempts, exponential backoff) on every install step for consistency. --------- Co-authored-by: Rouzbeh <rqzbeh@users.noreply.github.com>
This commit is contained in:
2
.github/workflows/dast-smoke.yml
vendored
2
.github/workflows/dast-smoke.yml
vendored
@@ -37,7 +37,7 @@ jobs:
|
||||
with:
|
||||
node-version: "24"
|
||||
cache: npm
|
||||
- run: npm ci
|
||||
- uses: ./.github/actions/npm-ci-retry
|
||||
- name: Build CLI bundle
|
||||
env:
|
||||
OMNIROUTE_BUILD_BACKEND_ONLY: "1"
|
||||
|
||||
12
.github/workflows/quality.yml
vendored
12
.github/workflows/quality.yml
vendored
@@ -137,7 +137,7 @@ jobs:
|
||||
with:
|
||||
node-version: ${{ env.CI_NODE_VERSION }}
|
||||
cache: npm
|
||||
- run: npm ci
|
||||
- uses: ./.github/actions/npm-ci-retry
|
||||
# One walk of src/app/api for openapi-routes + docs-symbols (both still fail independently).
|
||||
- run: npm run check:api-docs-refs
|
||||
- name: Docs accuracy (fabricated-docs + i18n mirrors, strict)
|
||||
@@ -181,7 +181,7 @@ jobs:
|
||||
with:
|
||||
node-version: ${{ env.CI_NODE_VERSION }}
|
||||
cache: npm
|
||||
- run: npm ci
|
||||
- uses: ./.github/actions/npm-ci-retry
|
||||
- name: Restore ESLint file cache
|
||||
uses: actions/cache@v6
|
||||
with:
|
||||
@@ -430,7 +430,7 @@ jobs:
|
||||
with:
|
||||
node-version: ${{ env.CI_NODE_VERSION }}
|
||||
cache: npm
|
||||
- run: npm ci
|
||||
- uses: ./.github/actions/npm-ci-retry
|
||||
# WS5.2/5.3: JUnit feeds Trunk Flaky Tests — the fast-path runs on EVERY PR,
|
||||
# which is where flaky-detection volume actually comes from (ci.yml's heavy
|
||||
# jobs only run on the release PR). Advisory upload, own-origin only.
|
||||
@@ -476,7 +476,7 @@ jobs:
|
||||
with:
|
||||
node-version: ${{ env.CI_NODE_VERSION }}
|
||||
cache: npm
|
||||
- run: npm ci
|
||||
- uses: ./.github/actions/npm-ci-retry
|
||||
# QW-d: fonte única — o mesmo npm script do CI pesado/local. Fecha dois drifts do
|
||||
# comando inline antigo: os dirs `memory` e `usage` estavam FORA do glob (testes
|
||||
# silenciosamente não rodavam no fast path) e o setupPolyfill não era importado.
|
||||
@@ -516,7 +516,7 @@ jobs:
|
||||
with:
|
||||
node-version: ${{ env.CI_NODE_VERSION }}
|
||||
cache: npm
|
||||
- run: npm ci
|
||||
- uses: ./.github/actions/npm-ci-retry
|
||||
- name: Restore ESLint file cache
|
||||
uses: actions/cache@v6
|
||||
with:
|
||||
@@ -583,7 +583,7 @@ jobs:
|
||||
with:
|
||||
node-version: ${{ env.CI_NODE_VERSION }}
|
||||
cache: npm
|
||||
- run: npm ci
|
||||
- uses: ./.github/actions/npm-ci-retry
|
||||
- name: CHANGELOG integrity (nenhum bullet da base pode sumir no merge-result)
|
||||
run: npm run check:changelog-integrity
|
||||
- name: Agent-skills generator sync (SKILL.md gerado ≡ catálogo)
|
||||
|
||||
1
changelog.d/fixes/10415-vision-bridge-combo-reroute.md
Normal file
1
changelog.d/fixes/10415-vision-bridge-combo-reroute.md
Normal file
@@ -0,0 +1 @@
|
||||
- **fix(guardrails):** Vision Bridge now reroutes whole requests for named combos whose targets have zero vision-capable models (previously such image requests died with `capability_mismatch` when the describe path could not run), and when the fallback describe path also fails for every image the request degrades to explicit `(unavailable)` stub text instead of preserving images the combo cannot consume ([#10415](https://github.com/diegosouzapw/OmniRoute/pull/10415)) — thanks @rqzbeh
|
||||
@@ -471,24 +471,66 @@ if (existsSync(opencodePluginSrc) && existsSync(join(opencodePluginSrc, "package
|
||||
// needs the plugin's own devDependencies (typescript, @opencode-ai/plugin
|
||||
// types). Without this install a fresh CI publish fails at this step.
|
||||
if (!existsSync(join(opencodePluginSrc, "node_modules"))) {
|
||||
// The plugin's node_modules is gitignored, so a fresh CI checkout
|
||||
// ALWAYS installs here. The registry CDN is intermittently flaky
|
||||
// (onnxruntime-class ETIMEDOUTs to the Microsoft CDN have repeatedly
|
||||
// stalled CI npm steps for 20+ minutes), and npm's unbounded fetch
|
||||
// retries turn a stalled connection into a hang that eats the whole
|
||||
// job budget. Bound the fetch and retry the install a few times:
|
||||
// transient network failures fail fast and recover instead of hanging.
|
||||
const npmEntry = resolveBundledNpmEntry("npm-cli.js");
|
||||
if (npmEntry) {
|
||||
execFileSync(process.execPath, [npmEntry, "install", "--no-audit", "--no-fund"], {
|
||||
cwd: opencodePluginSrc,
|
||||
stdio: "inherit",
|
||||
});
|
||||
} else if (process.platform !== "win32") {
|
||||
// No bundled npm entry found (non-standard Node layout). Plain `npm` is
|
||||
// safe here — the .cmd-shim hazard #8858 guards against is Windows-only.
|
||||
execFileSync("npm", ["install", "--no-audit", "--no-fund"], {
|
||||
cwd: opencodePluginSrc,
|
||||
stdio: "inherit",
|
||||
});
|
||||
} else {
|
||||
throw new Error(
|
||||
"npm-cli.js not found next to the running Node binary; cannot install the plugin dependencies without falling back to a .cmd shim."
|
||||
);
|
||||
const installArgs = [
|
||||
"install",
|
||||
"--no-audit",
|
||||
"--no-fund",
|
||||
"--fetch-retries=2",
|
||||
"--fetch-retry-mintimeout=2000",
|
||||
"--fetch-retry-maxtimeout=30000",
|
||||
"--fetch-timeout=60000",
|
||||
];
|
||||
const runPluginInstall = () => {
|
||||
if (npmEntry) {
|
||||
execFileSync(process.execPath, [npmEntry, ...installArgs], {
|
||||
cwd: opencodePluginSrc,
|
||||
stdio: "inherit",
|
||||
});
|
||||
} else if (process.platform !== "win32") {
|
||||
// No bundled npm entry found (non-standard Node layout). Plain `npm` is
|
||||
// safe here — the .cmd-shim hazard #8858 guards against is Windows-only.
|
||||
execFileSync("npm", installArgs, {
|
||||
cwd: opencodePluginSrc,
|
||||
stdio: "inherit",
|
||||
});
|
||||
} else {
|
||||
throw new Error(
|
||||
"npm-cli.js not found next to the running Node binary; cannot install the plugin dependencies without falling back to a .cmd shim."
|
||||
);
|
||||
}
|
||||
};
|
||||
const sleepSync = (ms: number) =>
|
||||
Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms);
|
||||
let installError: any = null;
|
||||
for (let attempt = 1; attempt <= 3; attempt++) {
|
||||
try {
|
||||
if (attempt > 1) {
|
||||
console.log(
|
||||
` 🔄 @omniroute/opencode-plugin npm install retry (attempt ${attempt}/3)`
|
||||
);
|
||||
}
|
||||
runPluginInstall();
|
||||
installError = null;
|
||||
break;
|
||||
} catch (err: any) {
|
||||
installError = err;
|
||||
if (attempt < 3) {
|
||||
console.warn(
|
||||
` ⚠️ plugin npm install failed (attempt ${attempt}/3): ${err?.message ?? String(err)} — retrying in 10s`
|
||||
);
|
||||
sleepSync(10_000);
|
||||
}
|
||||
}
|
||||
}
|
||||
if (installError) throw installError;
|
||||
}
|
||||
runBuildTool("tsup", "tsup", [], {
|
||||
cwd: opencodePluginSrc,
|
||||
|
||||
@@ -3,6 +3,9 @@
|
||||
* Intercepts image-bearing requests to non-vision models.
|
||||
* For individual non-vision models: reroutes to the fastest available vision-capable model.
|
||||
* For combos with non-vision targets: extracts descriptions via vision model and replaces images with text.
|
||||
* For combos with ZERO vision-capable targets: falls back to whole-request reroute to a
|
||||
* vision-capable model (same semantics as an individual text-only model), so image
|
||||
* requests do not die in the combo capability filter when describing is impossible.
|
||||
*/
|
||||
|
||||
import { BaseGuardrail, type GuardrailContext, type GuardrailResult } from "./base";
|
||||
@@ -31,7 +34,7 @@ import {
|
||||
|
||||
export { isProviderConnectionUsable, hasUsableCredentialsForModel };
|
||||
|
||||
type ComboVisionBridgeDecision = "process" | "skip" | "not-combo";
|
||||
type ComboVisionBridgeDecision = "process" | "skip" | "not-combo" | "no-vision";
|
||||
|
||||
export function resolveVisionComboName(mapping: Record<string, unknown>): string | null {
|
||||
const comboName = mapping.comboName ?? mapping.name ?? null;
|
||||
@@ -40,10 +43,15 @@ export function resolveVisionComboName(mapping: Record<string, unknown>): string
|
||||
|
||||
/// Check if a combo model should trigger vision bridge processing.
|
||||
/// Resolves combo targets and returns:
|
||||
/// - "process" if any target cannot be proven vision-capable
|
||||
/// - "process" if some (but not all) model targets lack proven vision support
|
||||
/// - "skip" if all model targets can handle images directly
|
||||
/// - "no-vision" when the combo has model targets but NONE can handle images —
|
||||
/// the combo behaves like a single text-only model, so the bridge may
|
||||
/// whole-request reroute to a vision-capable model (mirroring non-combos)
|
||||
/// - "not-combo" when the model is not a combo/mapping
|
||||
async function getComboVisionBridgeDecision(model: string): Promise<ComboVisionBridgeDecision> {
|
||||
export async function getComboVisionBridgeDecision(
|
||||
model: string
|
||||
): Promise<ComboVisionBridgeDecision> {
|
||||
try {
|
||||
const { getComboByName } = await import("@/lib/localDb");
|
||||
const { resolveComboForModel } = await import("@/lib/db/modelComboMappings");
|
||||
@@ -70,7 +78,10 @@ async function getComboVisionBridgeDecision(model: string): Promise<ComboVisionB
|
||||
// combo-ref → conservative (process images)
|
||||
// model step with no native vision → process images
|
||||
// all model steps with native vision → safe to skip
|
||||
// zero vision-capable model steps → "no-vision" (reroute-eligible)
|
||||
let hasModelStep = false;
|
||||
let hasVisionCapableStep = false;
|
||||
let hasNonVisionStep = false;
|
||||
for (const step of rawModels) {
|
||||
const s = step as Record<string, unknown>;
|
||||
if (s.kind === "combo-ref") return "process";
|
||||
@@ -79,8 +90,10 @@ async function getComboVisionBridgeDecision(model: string): Promise<ComboVisionB
|
||||
const targetModel = s.model;
|
||||
if (typeof targetModel === "string") {
|
||||
const caps = getResolvedModelCapabilities(targetModel);
|
||||
if (caps.supportsVision !== true) {
|
||||
return "process";
|
||||
if (caps.supportsVision === true) {
|
||||
hasVisionCapableStep = true;
|
||||
} else {
|
||||
hasNonVisionStep = true;
|
||||
}
|
||||
} else {
|
||||
return "process";
|
||||
@@ -88,11 +101,15 @@ async function getComboVisionBridgeDecision(model: string): Promise<ComboVisionB
|
||||
}
|
||||
}
|
||||
|
||||
// All model steps support vision — safe to skip
|
||||
if (hasModelStep) return "skip";
|
||||
|
||||
// No recognizable steps — don't force bridge
|
||||
return "not-combo";
|
||||
if (!hasModelStep) return "not-combo";
|
||||
// Every model step is proven vision-capable — safe to skip
|
||||
if (hasVisionCapableStep && !hasNonVisionStep) return "skip";
|
||||
// Mixed combo: some targets lack vision — describe so the combo still answers
|
||||
if (hasVisionCapableStep) return "process";
|
||||
// Combo exists but NO target can handle images: equivalent to a text-only
|
||||
// model, so the whole request may be rerouted to a vision-capable model.
|
||||
return "no-vision";
|
||||
} catch {
|
||||
// On error, try to process images (conservative)
|
||||
return "process";
|
||||
@@ -205,7 +222,7 @@ export class VisionBridgeGuardrail extends BaseGuardrail {
|
||||
// model-combo mapping routes this model through a combo where
|
||||
// some targets may NOT support vision. In that case, the vision
|
||||
// bridge must process images so combo targets can describe them.
|
||||
if (comboVisionBridgeDecision !== "process") {
|
||||
if (comboVisionBridgeDecision !== "process" && comboVisionBridgeDecision !== "no-vision") {
|
||||
context.log?.debug?.("VISION_BRIDGE", "Skipping: target model supports vision natively");
|
||||
return { block: false };
|
||||
}
|
||||
@@ -272,10 +289,17 @@ export class VisionBridgeGuardrail extends BaseGuardrail {
|
||||
const rerouteTextOnly = settings.visionBridgeRerouteTextOnly === true;
|
||||
// Reroute when the operator opted in to direct VLM routing for every text-only
|
||||
// route (keeps image bytes instead of a lossy bridge description), or when the
|
||||
// auto heuristic deems the request eligible.
|
||||
// auto heuristic deems the request eligible. A named combo with ZERO
|
||||
// vision-capable targets ("no-vision") is reroute-eligible too: it behaves
|
||||
// exactly like a single text-only model, and without this fallback an image
|
||||
// request would die in the combo capability filter (capability_mismatch)
|
||||
// whenever the describe path cannot run.
|
||||
const rerouteEligible =
|
||||
rerouteTextOnly ||
|
||||
((comboVisionBridgeDecision === "not-combo" || isAuto) && !forceVisionBridge);
|
||||
((comboVisionBridgeDecision === "not-combo" ||
|
||||
comboVisionBridgeDecision === "no-vision" ||
|
||||
isAuto) &&
|
||||
!forceVisionBridge);
|
||||
// Forced modes short-circuit BEFORE the auto heuristic (#6640/#7204 untouched):
|
||||
// - "describe" skips the whole reroute block → straight to the describe path.
|
||||
// - "reroute" skips only the keep-credentialed-model guard; the reroute-target
|
||||
@@ -434,8 +458,15 @@ export class VisionBridgeGuardrail extends BaseGuardrail {
|
||||
// is safe here because the upstream can only handle text. The original #4012
|
||||
// preserve-raw behavior only applies to paths where the upstream might still
|
||||
// be vision-capable (reroute path / unknown capability).
|
||||
// "no-vision" combos are included for the same reason: with ZERO
|
||||
// vision-capable targets, the combo capability filter rejects raw images
|
||||
// outright (capability_mismatch), so stub text is strictly better than
|
||||
// preserving bytes no combo target can consume.
|
||||
const allNull = descriptions.every((d) => d === null);
|
||||
if (allNull && comboVisionBridgeDecision === "process") {
|
||||
if (
|
||||
allNull &&
|
||||
(comboVisionBridgeDecision === "process" || comboVisionBridgeDecision === "no-vision")
|
||||
) {
|
||||
for (let i = 0; i < descriptions.length; i++) {
|
||||
descriptions[i] = `[Image ${i + 1}]: (unavailable — no vision-capable provider connected)`;
|
||||
}
|
||||
|
||||
292
tests/unit/guardrails/visionBridge-combo-reroute.test.ts
Normal file
292
tests/unit/guardrails/visionBridge-combo-reroute.test.ts
Normal file
@@ -0,0 +1,292 @@
|
||||
/**
|
||||
* Vision Bridge × named-combo reroute tests.
|
||||
*
|
||||
* Regression: a named combo whose targets have ZERO vision-capable models was
|
||||
* never reroute-eligible. The bridge only described images for it, and when
|
||||
* the describe path could not run (unreachable bridge model, failed self-loop,
|
||||
* missing credentials) the raw images stayed in the payload, the combo
|
||||
* capability filter excluded every target, and the request died with
|
||||
* capability_mismatch — "vision bridge does not affect combo models".
|
||||
*
|
||||
* Fix under test: `getComboVisionBridgeDecision` returns "no-vision" for a
|
||||
* combo with model targets but no vision-capable target, and preCall treats
|
||||
* that decision as reroute-eligible (mirroring non-combo text-only models),
|
||||
* falling back to describe only when no usable reroute target exists.
|
||||
*/
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import fs from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
|
||||
const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-vb-combo-reroute-"));
|
||||
process.env.DATA_DIR = TEST_DATA_DIR;
|
||||
|
||||
const { VisionBridgeGuardrail, getComboVisionBridgeDecision } =
|
||||
await import("../../../src/lib/guardrails/visionBridge.ts");
|
||||
const { resetGuardrailsForTests } = await import("../../../src/lib/guardrails/registry.ts");
|
||||
const { getResolvedModelCapabilities } = await import("../../../src/lib/modelCapabilities.ts");
|
||||
const core = await import("../../../src/lib/db/core.ts");
|
||||
const combosDb = await import("../../../src/lib/db/combos.ts");
|
||||
const mappingsDb = await import("../../../src/lib/db/modelComboMappings.ts");
|
||||
|
||||
async function resetStorage() {
|
||||
core.resetDbInstance();
|
||||
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
|
||||
fs.mkdirSync(TEST_DATA_DIR, { recursive: true });
|
||||
}
|
||||
|
||||
test.beforeEach(async () => {
|
||||
await resetStorage();
|
||||
});
|
||||
|
||||
test.after(() => {
|
||||
core.resetDbInstance();
|
||||
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
async function createCombo(name, models, overrides = {}) {
|
||||
return combosDb.createCombo({
|
||||
name,
|
||||
models,
|
||||
strategy: "priority",
|
||||
...overrides,
|
||||
});
|
||||
}
|
||||
|
||||
// ── Fixtures ────────────────────────────────────────────────────────────────
|
||||
|
||||
const VISION_MODEL = "openai/gpt-4o";
|
||||
const TEXT_MODEL_A = "google/gemma-2-27b";
|
||||
const TEXT_MODEL_B = "mistral/mistral-large-latest";
|
||||
|
||||
// Fail loudly if the static vision heuristic drifts: these fixtures drive
|
||||
// every assertion in this file.
|
||||
test("fixture models have the expected static vision capability", () => {
|
||||
assert.equal(getResolvedModelCapabilities(VISION_MODEL).supportsVision, true);
|
||||
assert.notEqual(getResolvedModelCapabilities(TEXT_MODEL_A).supportsVision, true);
|
||||
assert.notEqual(getResolvedModelCapabilities(TEXT_MODEL_B).supportsVision, true);
|
||||
});
|
||||
|
||||
const mockSettings = {
|
||||
visionBridgeEnabled: true,
|
||||
visionBridgeModel: VISION_MODEL,
|
||||
visionBridgePrompt: "Describe this image concisely.",
|
||||
visionBridgeTimeout: 30000,
|
||||
visionBridgeMaxImages: 10,
|
||||
};
|
||||
|
||||
let visionCallCount = 0;
|
||||
|
||||
// Each describe-path test uses a UNIQUE prompt: the shared describe cache keys
|
||||
// on (contentRef, prompt, model), so a reused prompt would serve a cached
|
||||
// description and skip callVisionModel, breaking the assertion on call count.
|
||||
function createGuardrail(depsOverrides = {}, prompt = "Describe this image concisely.") {
|
||||
return new VisionBridgeGuardrail({
|
||||
deps: {
|
||||
getSettings: async () => ({ ...mockSettings, visionBridgePrompt: prompt }),
|
||||
callVisionModel: async () => {
|
||||
visionCallCount++;
|
||||
return "A red circle on a white background";
|
||||
},
|
||||
// null = fail-open (no credential DB in unit tests), matching the
|
||||
// existing visionBridge.test.ts convention.
|
||||
hasUsableCredentials: async () => null,
|
||||
...depsOverrides,
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
const IMAGE_PAYLOAD = {
|
||||
model: "text-only-combo",
|
||||
messages: [
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "text", text: "Describe this image in one sentence." },
|
||||
{
|
||||
type: "image_url",
|
||||
image_url: {
|
||||
url: "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg==",
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
function hasImagePart(messages) {
|
||||
return JSON.stringify(messages).includes("image_url");
|
||||
}
|
||||
|
||||
// GuardrailResult<unknown> types modifiedPayload as `unknown`; the existing
|
||||
// visionBridge.test.ts casts it the same way.
|
||||
type ModifiedBody = { model?: string; messages?: unknown[] };
|
||||
function asModifiedBody(result: { modifiedPayload?: unknown }): ModifiedBody {
|
||||
return (result.modifiedPayload ?? {}) as ModifiedBody;
|
||||
}
|
||||
|
||||
// ── getComboVisionBridgeDecision ────────────────────────────────────────────
|
||||
|
||||
test("decision: combo with zero vision-capable targets returns 'no-vision'", async () => {
|
||||
await createCombo("text-only-combo", [
|
||||
{ provider: "google", model: TEXT_MODEL_A },
|
||||
{ provider: "mistral", model: TEXT_MODEL_B },
|
||||
]);
|
||||
assert.equal(await getComboVisionBridgeDecision("text-only-combo"), "no-vision");
|
||||
});
|
||||
|
||||
test("decision: combo with all vision-capable targets returns 'skip'", async () => {
|
||||
await createCombo("vision-combo", [
|
||||
{ provider: "openai", model: VISION_MODEL },
|
||||
{ provider: "anthropic", model: "anthropic/claude-sonnet-4-20250514" },
|
||||
]);
|
||||
assert.equal(await getComboVisionBridgeDecision("vision-combo"), "skip");
|
||||
});
|
||||
|
||||
test("decision: mixed combo (some vision, some not) returns 'process'", async () => {
|
||||
await createCombo("mixed-combo", [
|
||||
{ provider: "openai", model: VISION_MODEL },
|
||||
{ provider: "google", model: TEXT_MODEL_A },
|
||||
]);
|
||||
assert.equal(await getComboVisionBridgeDecision("mixed-combo"), "process");
|
||||
});
|
||||
|
||||
test("decision: unknown model returns 'not-combo'", async () => {
|
||||
assert.equal(await getComboVisionBridgeDecision("not-a-combo"), "not-combo");
|
||||
});
|
||||
|
||||
test("decision: model-combo mapping routes to the combo decision", async () => {
|
||||
const combo = await createCombo("mapped-text-only", [
|
||||
{ provider: "google", model: TEXT_MODEL_A },
|
||||
]);
|
||||
await mappingsDb.createModelComboMapping({
|
||||
pattern: "mapped-model-alias",
|
||||
comboId: combo.id as string,
|
||||
priority: 20,
|
||||
description: "test alias",
|
||||
});
|
||||
assert.equal(await getComboVisionBridgeDecision("mapped-model-alias"), "no-vision");
|
||||
});
|
||||
|
||||
// ── preCall: no-vision combo reroutes whole request ─────────────────────────
|
||||
|
||||
test("preCall: zero-vision combo reroutes the whole request to the bridge model", async () => {
|
||||
resetGuardrailsForTests({ registerDefaults: false });
|
||||
await createCombo("text-only-combo", [
|
||||
{ provider: "google", model: TEXT_MODEL_A },
|
||||
{ provider: "mistral", model: TEXT_MODEL_B },
|
||||
]);
|
||||
visionCallCount = 0;
|
||||
const guardrail = createGuardrail();
|
||||
const result = await guardrail.preCall(IMAGE_PAYLOAD, {});
|
||||
|
||||
assert.equal(result.block, false);
|
||||
// Rerouted: model swapped to the vision bridge model, image bytes KEPT.
|
||||
assert.equal(asModifiedBody(result).model, VISION_MODEL);
|
||||
assert.equal(result.meta.rerouted, true);
|
||||
assert.equal(result.meta.fromModel, "text-only-combo");
|
||||
assert.equal(hasImagePart(asModifiedBody(result).messages), true);
|
||||
// Describe never ran — no extra vision call.
|
||||
assert.equal(visionCallCount, 0);
|
||||
});
|
||||
|
||||
test("preCall: zero-vision combo falls back to describe when reroute target is unusable", async () => {
|
||||
resetGuardrailsForTests({ registerDefaults: false });
|
||||
await createCombo("text-only-combo", [{ provider: "google", model: TEXT_MODEL_A }]);
|
||||
visionCallCount = 0;
|
||||
// Reroute target has no usable credentials → describe path must run.
|
||||
const guardrail = createGuardrail(
|
||||
{ hasUsableCredentials: async () => false },
|
||||
"Describe the fallback image."
|
||||
);
|
||||
const result = await guardrail.preCall(IMAGE_PAYLOAD, {});
|
||||
|
||||
assert.equal(result.block, false);
|
||||
assert.equal(result.meta.rerouted, undefined);
|
||||
// Images replaced with the described text; combo model kept.
|
||||
assert.equal(asModifiedBody(result).model, "text-only-combo");
|
||||
assert.equal(hasImagePart(asModifiedBody(result).messages), false);
|
||||
assert.equal(visionCallCount, 1);
|
||||
});
|
||||
|
||||
test("preCall: no-vision combo, unusable reroute target AND describe failure -> stub text", async () => {
|
||||
resetGuardrailsForTests({ registerDefaults: false });
|
||||
await createCombo("text-only-combo", [{ provider: "google", model: TEXT_MODEL_A }]);
|
||||
visionCallCount = 0;
|
||||
// Double failure: the reroute target has no usable credentials AND the
|
||||
// describe call fails for every image. The allNull stub fallback must fire
|
||||
// for "no-vision" too — otherwise the raw images stay in the payload, the
|
||||
// combo capability filter rejects every target, and the original
|
||||
// capability_mismatch recurs.
|
||||
const guardrail = createGuardrail(
|
||||
{
|
||||
hasUsableCredentials: async () => false,
|
||||
callVisionModel: async () => {
|
||||
visionCallCount++;
|
||||
throw new Error("no vision-capable provider connected");
|
||||
},
|
||||
},
|
||||
"Describe the double-failure image."
|
||||
);
|
||||
const result = await guardrail.preCall(IMAGE_PAYLOAD, {});
|
||||
|
||||
assert.equal(result.block, false);
|
||||
assert.equal(result.meta.rerouted, undefined);
|
||||
// Combo model kept; raw image replaced with the stub text.
|
||||
assert.equal(asModifiedBody(result).model, "text-only-combo");
|
||||
assert.equal(hasImagePart(asModifiedBody(result).messages), false);
|
||||
assert.match(
|
||||
JSON.stringify(asModifiedBody(result).messages),
|
||||
/\(unavailable — no vision-capable provider connected\)/
|
||||
);
|
||||
assert.equal(visionCallCount, 1);
|
||||
});
|
||||
|
||||
test("preCall: zero-vision combo with no images is left untouched", async () => {
|
||||
resetGuardrailsForTests({ registerDefaults: false });
|
||||
await createCombo("text-only-combo", [{ provider: "google", model: TEXT_MODEL_A }]);
|
||||
visionCallCount = 0;
|
||||
const guardrail = createGuardrail();
|
||||
const result = await guardrail.preCall(
|
||||
{
|
||||
model: "text-only-combo",
|
||||
messages: [{ role: "user", content: "no images here" }],
|
||||
},
|
||||
{}
|
||||
);
|
||||
assert.equal(result.block, false);
|
||||
assert.equal(result.modifiedPayload, undefined);
|
||||
assert.equal(visionCallCount, 0);
|
||||
});
|
||||
|
||||
// ── preCall: unchanged semantics for other combo shapes ─────────────────────
|
||||
|
||||
test("preCall: all-vision combo still skips the bridge entirely", async () => {
|
||||
resetGuardrailsForTests({ registerDefaults: false });
|
||||
await createCombo("vision-combo", [{ provider: "openai", model: VISION_MODEL }]);
|
||||
visionCallCount = 0;
|
||||
const guardrail = createGuardrail();
|
||||
const result = await guardrail.preCall({ ...IMAGE_PAYLOAD, model: "vision-combo" }, {});
|
||||
assert.equal(result.block, false);
|
||||
assert.equal(result.modifiedPayload, undefined);
|
||||
assert.equal(visionCallCount, 0);
|
||||
});
|
||||
|
||||
test("preCall: mixed combo keeps the describe path (no reroute, model unchanged)", async () => {
|
||||
resetGuardrailsForTests({ registerDefaults: false });
|
||||
await createCombo("mixed-combo", [
|
||||
{ provider: "openai", model: VISION_MODEL },
|
||||
{ provider: "google", model: TEXT_MODEL_A },
|
||||
]);
|
||||
visionCallCount = 0;
|
||||
const guardrail = createGuardrail({}, "Describe the mixed-combo image.");
|
||||
const result = await guardrail.preCall({ ...IMAGE_PAYLOAD, model: "mixed-combo" }, {});
|
||||
|
||||
assert.equal(result.block, false);
|
||||
// Mixed combo is NOT reroute-eligible: model stays, images described.
|
||||
assert.equal(result.meta.rerouted, undefined);
|
||||
assert.equal(asModifiedBody(result).model, "mixed-combo");
|
||||
assert.equal(hasImagePart(asModifiedBody(result).messages), false);
|
||||
assert.equal(visionCallCount, 1);
|
||||
});
|
||||
Reference in New Issue
Block a user