mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-08-15 19:52:50 +03:00
* fix(ci): clear base-reds on release/v3.8.50 (round 3) - CHANGELOG.md: restore the top [Unreleased] section dropped by the #10189 reconcile (docs-sync gate: first section must be Unreleased) - env-doc-sync: document CONDUCTOR_ORCHESTRATOR_TOKEN + CONDUCTOR_SPOKESPERSON_URL in .env.example/ENVIRONMENT.md; allowlist the CI-only GITHUB_STEP_SUMMARY and TS7_BASE_REF (ts7 ratchet signals); drop a stray merge artifact line - providers: restore the audited chatanywhere metadata entry that base-reds round 2 dropped together with its duplicate — the provider was half-wired (registry+endpoint without APIKEY metadata), which is what the wave3 test catches; re-pin providers-constants-split at the measured 228 - docs counts: 338 -> 339 (today's +2 void-ai/helixmind, -1 Puter) via gen:provider-reference + README/AGENTS/llm.txt/package.json/diagrams/i18n mirrors - file-size ratchet: annotated rebaseline for the two pre-existing drifts (ModelSelectModal 1138, gateways 1250) following the 2026-08-11 precedent Refs #9985 * fix(ci): base-reds round 3b — stale sibling tests + mode-pack weight contract - check-docs-counts-sync.test.ts: drop the imports/subtests of the four helpers #10196 removed from the gate script (readMcpFactsFromSource, listLocalizedDocs, makeRequiredCountsValidator, checkFreeTierInventory) — the new-API tests that #10196 added stay; the file now loads again under the node runner - quota-connection-recovery.test.ts: convert from vitest APIs to node:test — the file lives in tests/unit/*.test.ts (node-runner glob) and the vitest runtime crashes when imported outside vitest, killing the whole shard entry - modePacks.ts: re-normalize all six mode packs to sum 1.0 — #8940 added sessionAvailability: 0.05 to every pack without rebalancing (1.05 total); ratios preserved exactly (÷1.05), so post-normalizeScoringWeights behavior is unchanged; restores the declared sum-to-1.0 contract the 4235 test pins Refs #9985 * fix(ci): base-reds round 3c — vitest siblings, weights default, secrets FP, mutation tap - DistributeProxiesButton.test.tsx: wrap renders in NextIntlClientProvider — #9245 localized the component (useTranslations) and left the test without the intl context, failing all 14 cases - scoring.ts: re-normalize DEFAULT_WEIGHTS to sum 1.0 (same #8940 class as the mode packs — sessionAvailability added without rebalancing; ratios preserved) - .gitleaks.toml: generalize the kimi sponsor-banner localStorage-key allowlist to -v\d+ — #10200 bumped v1→v2 and the stale regex regressed the secrets ratchet with a false positive - stryker.conf.json: register 6 covering unit tests in tap.testFiles (4 modules) so their mutant kills count — unblocks check:mutation-test-coverage --strict Refs #9985 * fix(ci): base-reds round 3d — inspector factor gap, stale registry/gap tests, i18n key sync - comboScoringInspector: add cacheAffinity/sessionAvailability/connectionDensity to FACTOR_KEYS + the factor-key type — calculateScore() weighs them but the breakdown omitted them, so the explained contributions never summed to the reported score (inspector bug, red on the pure tip) - combo-scoring-inspector.test: make the explicit-weights override sum-neutral (±0.05 shift) so it stays valid for any DEFAULT_WEIGHTS values — the hardcoded override only summed to 1.0 against the pre-#8940 defaults, which is also why explicit weights silently fell back to 'default' on the tip - unorouter-registry.test: align to the canonical .com host (api.unorouter.ai 301-redirects there, verified live) and to wave4's live model discovery (passthrough, no static seed) — the .ai/auto-model expectations were stale - check-migration-numbering.test: 147 left KNOWN_GAPS when 147_api_keys_model_access_mode.sql landed — assert absent (same as 143) - i18n: sync-ui pass — 35,914 missing UI keys stamped as __MISSING__ placeholders across 42 locales (mechanical; greens the pt-BR key-presence integrity test; coverage pct unchanged by design — translation is a separate workstream) Refs #9985 * fix(ci): base-reds round 3e — 2 real defects + 14 stale sibling tests (waves A-E) Real defects fixed: - src/lib/db/apiKeys.ts: #9313's empty-allowlist early return bypassed the group permission check, silently disabling group deny rules (#8817) for every key without a per-key allowlist; fall-through restored, restricted+[] deny-all kept - open-sse/utils/proxyFetch.ts: #10032 re-appended the raw transport error to the propagated message, reintroducing the proxy user:password leak #9837 closed; new redactProxyDetailsInMessage() keeps the reason, redacts URL/credentials - .github/workflows/quality.yml: #10134 added the TS7 ratchet as a separate blocking step AFTER the aggregated gates — the exact #8542 masking mechanism; folded into the non-fail-fast loop (still blocking, still PR-only) ⚠️ CI edit, gate-strengthening — explicit owner sign-off requested on the PR - src/i18n/messages/ko.json: 3 machine-mistranslation regressions caught by the #8244 glossary checker (장애인→비활성화됨, 양말5://→socks5://, 비클로드→Claude가 아닌) Stale sibling tests aligned to deliberately-moved contracts (each cites its mover): request-log-detail-layout + -stream (#9245 intl provider), repro-8542 pin update, quality-rail-gate-membership (#10134 shape), agentSkills-routes 45→46 (#9058), cloudflare-ai-catalog-8717 (#8804 supersedes #8808), executor-xai (#9994), vision-bridge-claude-wire (#9463 minimax→openai), sse-auth forced-pin (#8893), tls-proxy-context (strengthened leak guards), rate-limit-local-error-classification (#9164/#9342), minimax-thinking-signature (#9463), codebuddy-cn (#9723 +1 test), github-copilot-custom-model (#9050), providers-g4f-batch3 (#9584), synced-capability-warmup (#9199, stricter), sidebar-tools-group (#8221), oauth-modal-grok-cli-paste (#9245); agentSkills/catalog.ts comment 45→46; file-size rebaseline for proxyFetch (+19, annotated) Refs #9985 * fix(ci): base-reds round 3f — waves F-J: 9 more real defects + stale sibling sweep Real production defects fixed (all red on the pure tip, each with its origin): - routeGuard.ts: #8949 accidentally DELETED the /api/providers/[id]/login local-only pattern — the route spawns a browser, so the loopback gate for a process-spawning route was gone (Hard Rules #15/#17); restored (314 guard tests green) - agentSkills generator: #9058's category dispatch gave the config category an empty body, wiping skills/config-codex-cli/SKILL.md at the #10131 sync; fixed + SKILL.md regenerated via the official generator - imageRegistry: #9982 broke same-provider bare aliasing (antigravity preview id sent upstream unresolved); new resolveSameProviderBareAlias() keeps the fal cross-provider fix intact - imageRegistry: #9982's prefix strip handed the bare nano-banana ids to fal-ai, violating the pinned 2026-07-31 operator decision (adobe-firefly owns them); fal entries made prefix-only (dispatch already re-prefixes) - mediaGeneration/fal.ts: the missing-credential 401 guard was lost when #10198 deleted the superseded falHandler — tests were hitting the live network - bottleneckPatch/rateLimitManager: #9041's merge clobbered #9604, resurrecting the Bottleneck v2.19.5 heartbeat bug (reservoir never refills); patched the library defect at the root and re-aligned chat-rate-limit-body-lock to the working reservoir contract - processSupervisor.mjs: #9761 regressed the Node spawn to bare "node" (the #9156 launchd bug) and dropped #9209's ipv4first args; both restored - openai-responses/pureHelpers: #9423's Agent null-sentinel was unreachable on the schemaless JSON-string path; gate extended - i18n en.json: #8222's regen reverted the #9976 unclosed-tag fix and #8559's combo-cooldown copy; #9038 shipped 40 t() calls with no messages (runtime MISSING_MESSAGE); all restored/added + official sync-ui stamps, and vi's zero-marker policy re-established via the sanctioned translation backend Stale sibling tests aligned (movers cited inline): chat-helpers (#9447), executor-antigravity (#9351), video-fal-grok (#9982), visionBridge (#9759), web-session-credentials (#8974), production-build-module-integrity (positive anchor added), agentSkills-generator/skillManifestsLint/skills-injection/ agentSkillTools-mcp/listCapabilities-a2a (#9058), memory-settings (#10010), model-catalog-policy-invalidation (#8906), model-alias-seed (#9485), reactive-context-compaction (#8949), combo-provider-wildcard (broken upsert helper), oauth-google-loopback (43-locale resurrected-key removal) Validation: 501/501 across the 47 touched test files; typecheck:core, lint, file-size, docs-sync all green. Refs #9985 * fix(ci): base-reds round 3g — wave K/L: 4 more real defects + stale alignments Real defects: - base/reasoningEffort.ts: the stale duplicate cherry-pick #9612 re-added the codex minimal→low rewrite that #9883 had deliberately removed (OMP minimal passthrough); block removed again - cursorImages.ts: #9840 wired prepareCursorImageForWire (sharp re-encode, fail-closed) into the SHARED resolveCursorImages, breaking zai-web and conol-web image uploads (HTTP 400 'undecodable'); new prepareForWire opt-out, Cursor default path unchanged (8 cursor suites green) - modelCapabilities/snapshot: catalog prepare still issued 323 per-model reads of model_context_overrides + max_input_tokens overrides, violating #9199's bulk-load contract; both now resolve from the snapshot single pass - v1-models-discovery-conformance: re-pinned to the bounded 30s SWR window (#9199/#10198) — the old 'stale-first regardless of age' contract is gone Stale tests aligned (movers cited inline): codex-tools-strict-default (#9828 redundant-oneOf strip), devin-providers (#9245 i18n), db-migrationrunner- constants-split (147→151 renumber #8228), gitlab-duo-oauth-setup (#9245), chatcore-extracted-modules (#9161 outbound-protocol keying) compression-api CI failures were cascade artifacts of codex-tools-strict-default failing in the same force-exit shard process — no own defect (171/171 local). Refs #9985 * fix(test): compression-api — register both describes before the runner starts The DATA_DIR setup + route/db top-level awaits sat BETWEEN the two describes; under --test-force-exit (the CI unit-runner flag) the process exits once the already-registered tests finish, so on slow CI machines the whole second describe died as 'Promise resolution is still pending' — the recurring CI-only shard-2 failure that never reproduced locally without the flag. Moved to the top of the file; 10/10 under --test-force-exit locally. Refs #9985 * fix(quality): freeze modelCapabilities.ts at 1006 (annotated) — snapshot routing growth Refs #9985 * fix(quality): move the modelCapabilities freeze into the frozen map (nested schema) Refs #9985 * fix(i18n): translate all 39,718 pending UI keys across 42 locales (owner-approved) Mass-translated every __MISSING__ placeholder via the official i18n:sync-ui --translate-markers pipeline (operator backend), restoring i18nUiCoverage to the 100 baseline (was 89.9 after the merge-storm UI landings + the 42 keys #9038 never shipped). Post-pass repairs, all caught by the existing gates: - glossary: retired renderings the machine reintroduced normalized again (提供商→提供者 zh-CN/zh-TW, 鏈接→連結, 文檔→文件, 調用→呼叫, 供應商→提供者, 響應→回應, 不活躍→未啟用 zh-TW; 클로드→Claude, 옴니루트→OmniRoute ko); DATA_DIR forbidden rendering avoided via 数据文件夹 rephrase - ICU integrity: 120 values with renamed/dropped {params} repaired (39 positional renames, 81 reset to the en source — functional over fluent) Validation: glossary/pt-BR/vi/deno-relay/settings-keys/value-drift/google- loopback suites 76/76; placeholder diff en×42 locales = 0; worst-locale coverage = 100.0%. Refs #9985 --------- Co-authored-by: backryun <bakryun0718@proton.me>
906 lines
31 KiB
TypeScript
906 lines
31 KiB
TypeScript
/**
|
|
* Tests for VisionBridgeGuardrail.
|
|
* Uses dependency injection to avoid SQLite dependency.
|
|
*/
|
|
|
|
import test from "node:test";
|
|
import assert from "node:assert/strict";
|
|
|
|
const { VisionBridgeGuardrail, resolveVisionComboName } =
|
|
await import("../../../src/lib/guardrails/visionBridge.ts");
|
|
const { resetGuardrailsForTests } = await import("../../../src/lib/guardrails/registry.ts");
|
|
const { getResolvedModelCapabilities } = await import("../../../src/lib/modelCapabilities.ts");
|
|
import type { GuardrailContext } from "../../../src/lib/guardrails/base.ts";
|
|
import type { VisionModelConfig } from "../../../src/lib/guardrails/visionBridgeHelpers.ts";
|
|
|
|
// ── Mock state ──────────────────────────────────────────────────────────────
|
|
|
|
let mockSettings: Record<string, unknown> = {
|
|
visionBridgeEnabled: true,
|
|
visionBridgeModel: "openai/gpt-4o-mini",
|
|
visionBridgePrompt: "Describe this image concisely.",
|
|
visionBridgeTimeout: 30000,
|
|
visionBridgeMaxImages: 10,
|
|
};
|
|
|
|
let mockVisionResponse = "A beautiful sunset over the ocean";
|
|
let shouldVisionFail = false;
|
|
let visionCallCount = 0;
|
|
|
|
function createGuardrail(options?: Parameters<typeof VisionBridgeGuardrail>[0]) {
|
|
return new VisionBridgeGuardrail({
|
|
...options,
|
|
deps: {
|
|
getSettings: async () => mockSettings,
|
|
callVisionModel: async (_imageDataUri: string, _config: VisionModelConfig) => {
|
|
visionCallCount++;
|
|
if (shouldVisionFail) {
|
|
throw new Error("Vision model failed");
|
|
}
|
|
return mockVisionResponse;
|
|
},
|
|
// Fail-open (null) so classic VB-S01/S07/S10 reroute tests keep working without a
|
|
// live credential DB. Credential-aware cases inject an explicit mock.
|
|
hasUsableCredentials: async () => null,
|
|
...(options?.deps ?? {}),
|
|
},
|
|
});
|
|
}
|
|
|
|
test.beforeEach(() => {
|
|
resetGuardrailsForTests({ registerDefaults: false });
|
|
visionCallCount = 0;
|
|
shouldVisionFail = false;
|
|
mockSettings = {
|
|
visionBridgeEnabled: true,
|
|
visionBridgeModel: "openai/gpt-4o-mini",
|
|
visionBridgePrompt: "Describe this image concisely.",
|
|
visionBridgeTimeout: 30000,
|
|
visionBridgeMaxImages: 10,
|
|
};
|
|
});
|
|
|
|
// ── Helpers ─────────────────────────────────────────────────────────────────
|
|
|
|
function createContext(overrides: Partial<GuardrailContext> = {}): GuardrailContext {
|
|
return {
|
|
model: "minimax/minimax-01",
|
|
log: console,
|
|
...overrides,
|
|
};
|
|
}
|
|
|
|
function createPayload(overrides: Record<string, unknown> = {}): Record<string, unknown> {
|
|
return {
|
|
model: "minimax/minimax-01",
|
|
messages: [{ role: "user", content: "Hello" }],
|
|
...overrides,
|
|
};
|
|
}
|
|
|
|
// ── Basic Properties ────────────────────────────────────────────────────────
|
|
|
|
test("VisionBridgeGuardrail has correct name and priority", () => {
|
|
const guardrail = createGuardrail();
|
|
assert.strictEqual(guardrail.name, "vision-bridge");
|
|
assert.strictEqual(guardrail.priority, 5);
|
|
});
|
|
|
|
test("VisionBridgeGuardrail is enabled by default", () => {
|
|
const guardrail = createGuardrail();
|
|
assert.strictEqual(guardrail.enabled, true);
|
|
});
|
|
|
|
test("VisionBridgeGuardrail can be disabled via constructor", () => {
|
|
const guardrail = createGuardrail({ enabled: false });
|
|
assert.strictEqual(guardrail.enabled, false);
|
|
});
|
|
|
|
test("resolveVisionComboName accepts only non-empty string mapping names", () => {
|
|
assert.equal(resolveVisionComboName({ comboName: "vision-fallback" }), "vision-fallback");
|
|
assert.equal(resolveVisionComboName({ name: "legacy-fallback" }), "legacy-fallback");
|
|
assert.equal(resolveVisionComboName({ comboName: { nested: true } }), null);
|
|
assert.equal(resolveVisionComboName({ comboName: 42 }), null);
|
|
assert.equal(resolveVisionComboName({ comboName: "" }), null);
|
|
});
|
|
|
|
// ── VB-S05: Vision Bridge disabled via settings ────────────────────────────
|
|
|
|
test("VB-S05: passthroughs when visionBridgeEnabled is false", async () => {
|
|
mockSettings.visionBridgeEnabled = false;
|
|
const guardrail = createGuardrail();
|
|
|
|
const payload = createPayload({
|
|
messages: [
|
|
{
|
|
role: "user",
|
|
content: [
|
|
{ type: "text", text: "What is in this image?" },
|
|
{ type: "image_url", image_url: { url: "https://example.com/image.png" } },
|
|
],
|
|
},
|
|
],
|
|
});
|
|
|
|
const result = await guardrail.preCall(payload, createContext());
|
|
assert.strictEqual(result.block, false);
|
|
assert.strictEqual(result.modifiedPayload, undefined);
|
|
});
|
|
|
|
// ── VB-S06: Disabled via context ────────────────────────────────────────────
|
|
|
|
test("VB-S06: skips when disabledGuardrails includes vision-bridge", async () => {
|
|
const guardrail = createGuardrail();
|
|
const payload = createPayload();
|
|
const context = createContext({ disabledGuardrails: ["vision-bridge"] });
|
|
|
|
const result = await guardrail.preCall(payload, context);
|
|
assert.strictEqual(result.block, false);
|
|
assert.strictEqual(result.modifiedPayload, undefined);
|
|
});
|
|
|
|
// ── VB-S02: Vision-capable model passthrough ────────────────────────────────
|
|
|
|
test("VB-S02: passthroughs for vision-capable model (gpt-4o)", async () => {
|
|
const guardrail = createGuardrail();
|
|
const payload = createPayload({
|
|
model: "openai/gpt-4o",
|
|
messages: [
|
|
{
|
|
role: "user",
|
|
content: [
|
|
{ type: "text", text: "What is this?" },
|
|
{
|
|
type: "image_url",
|
|
image_url: { url: "https://example.com/image.png" },
|
|
},
|
|
],
|
|
},
|
|
],
|
|
});
|
|
|
|
const result = await guardrail.preCall(payload, createContext({ model: "openai/gpt-4o" }));
|
|
|
|
// If supportsVision is true, it should passthrough (no modification)
|
|
// If supportsVision is null/undefined (no sync data), it will process — that's correct behavior
|
|
const capabilities = getResolvedModelCapabilities("openai/gpt-4o");
|
|
if (capabilities.supportsVision === true) {
|
|
assert.strictEqual(result.block, false);
|
|
assert.strictEqual(result.modifiedPayload, undefined);
|
|
} else {
|
|
// Without sync data, supportsVision is null — guardrail processes the image
|
|
// This is correct fail-open behavior for unknown model capabilities
|
|
assert.strictEqual(result.block, false);
|
|
}
|
|
});
|
|
|
|
test("VB-S02b: respects native vision support for GPT-family models", async () => {
|
|
const guardrail = createGuardrail();
|
|
|
|
for (const model of ["gpt-5.5", "gpt-5.5-high", "codex/gpt-5.5", "openai/gpt-4o-mini"]) {
|
|
visionCallCount = 0;
|
|
|
|
const payload = createPayload({
|
|
model,
|
|
messages: [
|
|
{
|
|
role: "user",
|
|
content: [
|
|
{ type: "text", text: "What is this?" },
|
|
{
|
|
type: "image_url",
|
|
image_url: { url: "https://example.com/image.png" },
|
|
},
|
|
],
|
|
},
|
|
],
|
|
});
|
|
|
|
const result = await guardrail.preCall(payload, createContext({ model }));
|
|
|
|
assert.strictEqual(result.block, false, `expected passthrough for ${model}`);
|
|
assert.strictEqual(visionCallCount, 0, `expected no bridge call for ${model}`);
|
|
|
|
// If supportsVision is true, payload should be unmodified.
|
|
// If supportsVision is null, the guardrail reroutes (modifiedPayload defined, model changed).
|
|
// Both are correct behavior — the key invariant is no describe call.
|
|
const caps = getResolvedModelCapabilities(model);
|
|
if (caps.supportsVision === true) {
|
|
assert.strictEqual(
|
|
result.modifiedPayload,
|
|
undefined,
|
|
`expected unmodified payload for ${model}`
|
|
);
|
|
}
|
|
}
|
|
});
|
|
|
|
test("VB-S02c: Conol multimodal models bypass the vision bridge", async () => {
|
|
const guardrail = createGuardrail();
|
|
const model = "conol-web/claude-fable-5-xhigh";
|
|
const payload = createPayload({
|
|
model,
|
|
messages: [
|
|
{
|
|
role: "user",
|
|
content: [
|
|
{ type: "text", text: "What is this?" },
|
|
{
|
|
type: "image_url",
|
|
image_url: { url: "data:image/png;base64,aW1hZ2U=" },
|
|
},
|
|
],
|
|
},
|
|
],
|
|
});
|
|
visionCallCount = 0;
|
|
|
|
const result = await guardrail.preCall(payload, createContext({ model }));
|
|
|
|
assert.equal(getResolvedModelCapabilities(model).supportsVision, true);
|
|
assert.strictEqual(result.block, false);
|
|
assert.strictEqual(result.modifiedPayload, undefined);
|
|
assert.strictEqual(visionCallCount, 0);
|
|
});
|
|
|
|
test("VB-S02d: Conol text-only models remain eligible for the vision bridge", async () => {
|
|
const guardrail = createGuardrail();
|
|
const model = "conol-web/deepseek/deepseek-v4-pro";
|
|
const payload = createPayload({
|
|
model,
|
|
messages: [
|
|
{
|
|
role: "user",
|
|
content: [
|
|
{ type: "text", text: "What is this?" },
|
|
{
|
|
type: "image_url",
|
|
image_url: { url: "data:image/png;base64,aW1hZ2U=" },
|
|
},
|
|
],
|
|
},
|
|
],
|
|
});
|
|
visionCallCount = 0;
|
|
|
|
const result = await guardrail.preCall(payload, createContext({ model }));
|
|
|
|
assert.equal(getResolvedModelCapabilities(model).supportsVision, false);
|
|
assert.strictEqual(result.block, false);
|
|
assert.notStrictEqual(result.modifiedPayload, undefined);
|
|
// #9759: with a configured vision model, individual text-only models REROUTE
|
|
// to it (images kept) instead of describing through an intermediate vision
|
|
// call — same contract as VB-S07. The point of this case is unchanged: conol
|
|
// text-only models must not be skipped by the bridge.
|
|
const modified = result.modifiedPayload as { model?: string };
|
|
assert.ok(modified.model, "rerouted model should be set");
|
|
assert.notStrictEqual(modified.model, model, "model should be different from original");
|
|
assert.strictEqual(visionCallCount, 0, "reroute keeps images; no describe call");
|
|
});
|
|
|
|
test("VB-S02: model capabilities returns supportsVision for known models", () => {
|
|
const gpt4oCaps = getResolvedModelCapabilities("openai/gpt-4o");
|
|
// supportsVision may be true (if sync data exists) or null (if not synced)
|
|
assert.ok(gpt4oCaps.supportsVision === true || gpt4oCaps.supportsVision === null);
|
|
});
|
|
|
|
// ── VB-S04: No images passthrough ──────────────────────────────────────────
|
|
|
|
test("VB-S04: passthroughs when no images in messages", async () => {
|
|
const guardrail = createGuardrail();
|
|
const payload = createPayload({
|
|
messages: [{ role: "user", content: "Hello, how are you?" }],
|
|
});
|
|
|
|
const result = await guardrail.preCall(payload, createContext());
|
|
assert.strictEqual(result.block, false);
|
|
assert.strictEqual(result.modifiedPayload, undefined);
|
|
});
|
|
|
|
test("VB-S04: passthroughs when messages array is empty", async () => {
|
|
const guardrail = createGuardrail();
|
|
const payload = createPayload({ messages: [] });
|
|
const result = await guardrail.preCall(payload, createContext());
|
|
assert.strictEqual(result.block, false);
|
|
});
|
|
|
|
// ── VB-S12: Auto-prefix skip ────────────────────────────────────────────────
|
|
|
|
test("VB-S12: reroutes auto/ prefix model to vision model (auto/vision)", async () => {
|
|
const guardrail = createGuardrail();
|
|
|
|
const payload = createPayload({
|
|
model: "auto/vision",
|
|
messages: [
|
|
{
|
|
role: "user",
|
|
content: [
|
|
{ type: "text", text: "What is in this image?" },
|
|
{ type: "image_url", image_url: { url: "https://example.com/image.png" } },
|
|
],
|
|
},
|
|
],
|
|
});
|
|
|
|
const result = await guardrail.preCall(payload, createContext({ model: "auto/vision" }));
|
|
assert.strictEqual(result.block, false);
|
|
assert.ok(result.modifiedPayload, "auto/vision should reroute to vision model");
|
|
assert.strictEqual(
|
|
result.modifiedPayload?.model,
|
|
"openai/gpt-4o-mini",
|
|
"should reroute to configured vision model"
|
|
);
|
|
assert.strictEqual(result.meta?.rerouted, true, "rerouted meta should be set");
|
|
assert.strictEqual(visionCallCount, 0, "should NOT call vision API (reroute, not describe)");
|
|
});
|
|
|
|
test("VB-S12b: reroutes auto prefix to best vision model when images present", async () => {
|
|
const guardrail = createGuardrail();
|
|
|
|
const payload = createPayload({
|
|
model: "auto",
|
|
messages: [
|
|
{
|
|
role: "user",
|
|
content: [
|
|
{ type: "text", text: "What is in this image?" },
|
|
{ type: "image_url", image_url: { url: "https://example.com/image.png" } },
|
|
],
|
|
},
|
|
],
|
|
});
|
|
|
|
const result = await guardrail.preCall(payload, createContext({ model: "auto" }));
|
|
assert.strictEqual(result.block, false);
|
|
assert.ok(result.modifiedPayload, "auto should reroute to vision model");
|
|
assert.strictEqual(
|
|
result.modifiedPayload?.model,
|
|
"openai/gpt-4o-mini",
|
|
"should reroute to configured vision model"
|
|
);
|
|
assert.strictEqual(result.meta?.rerouted, true, "rerouted meta should be set");
|
|
});
|
|
|
|
// ── VB-S01: Single image → reroute (individual non-vision model) ───────────
|
|
|
|
test("VB-S01: reroutes non-vision model with images to best vision model", async () => {
|
|
const guardrail = createGuardrail();
|
|
|
|
const payload = createPayload({
|
|
model: "minimax/minimax-01",
|
|
messages: [
|
|
{
|
|
role: "user",
|
|
content: [
|
|
{ type: "text", text: "What is in this image?" },
|
|
{ type: "image_url", image_url: { url: "https://example.com/image.png" } },
|
|
],
|
|
},
|
|
],
|
|
});
|
|
|
|
const result = await guardrail.preCall(payload, createContext({ model: "minimax/minimax-01" }));
|
|
|
|
assert.strictEqual(result.block, false);
|
|
assert.ok(result.modifiedPayload);
|
|
|
|
// Model should be rerouted to best vision-capable model (auto-selected from providers)
|
|
const modified = result.modifiedPayload as {
|
|
model?: string;
|
|
messages: Array<{ content: unknown[] }>;
|
|
};
|
|
assert.ok(modified.model, "rerouted model should be set");
|
|
assert.notStrictEqual(
|
|
modified.model,
|
|
"minimax/minimax-01",
|
|
"model should be different from original"
|
|
);
|
|
|
|
// Images should be KEPT since the vision model handles them natively
|
|
const content = modified.messages[0].content as Array<{ type: string; [key: string]: unknown }>;
|
|
const imagePart = content.find((p) => p.type === "image_url");
|
|
assert.ok(imagePart, "original image_url part must be preserved for rerouted vision model");
|
|
|
|
// Meta should indicate reroute occurred
|
|
const meta = result.meta as Record<string, unknown>;
|
|
assert.strictEqual(meta.rerouted, true);
|
|
assert.strictEqual(meta.fromModel, "minimax/minimax-01");
|
|
assert.ok(
|
|
typeof meta.toModel === "string" && meta.toModel.length > 0,
|
|
"toModel should be a non-empty string"
|
|
);
|
|
assert.notStrictEqual(meta.toModel, "minimax/minimax-01", "toModel should differ from original");
|
|
assert.strictEqual(meta.imagesKept, 1);
|
|
assert.strictEqual(visionCallCount, 0, "should NOT call vision API for description");
|
|
});
|
|
|
|
// ── VB-S13: Reroute preserves multiple images ──────────────────────────────
|
|
|
|
test("VB-S13: reroutes with multiple images, all preserved", async () => {
|
|
const guardrail = createGuardrail();
|
|
|
|
const payload = createPayload({
|
|
model: "minimax/minimax-01",
|
|
messages: [
|
|
{
|
|
role: "user",
|
|
content: [
|
|
{ type: "text", text: "Describe these images" },
|
|
{
|
|
type: "image_url",
|
|
image_url: { url: "https://example.com/cat.png" },
|
|
},
|
|
{
|
|
type: "image_url",
|
|
image_url: { url: "https://example.com/dog.png" },
|
|
},
|
|
{
|
|
type: "image_url",
|
|
image_url: { url: "https://example.com/bird.png" },
|
|
},
|
|
],
|
|
},
|
|
],
|
|
});
|
|
|
|
const result = await guardrail.preCall(payload, createContext({ model: "minimax/minimax-01" }));
|
|
|
|
assert.strictEqual(result.block, false);
|
|
assert.ok(result.modifiedPayload);
|
|
|
|
// All 3 images should be present in the rerouted payload
|
|
const modified = result.modifiedPayload as {
|
|
model?: string;
|
|
messages: Array<{ content: unknown[] }>;
|
|
};
|
|
const content = modified.messages[0].content as Array<{ type: string; [key: string]: unknown }>;
|
|
const images = content.filter((p) => p.type === "image_url");
|
|
assert.strictEqual(images.length, 3, "all 3 images should be preserved");
|
|
assert.strictEqual(visionCallCount, 0, "should NOT call vision API");
|
|
});
|
|
|
|
// ── VB-S07: Base64 image format → reroute ──────────────────────────────────
|
|
|
|
test("VB-S07: reroutes base64 image to vision model", async () => {
|
|
const guardrail = createGuardrail();
|
|
|
|
const payload = createPayload({
|
|
model: "minimax/minimax-01",
|
|
messages: [
|
|
{
|
|
role: "user",
|
|
content: [
|
|
{
|
|
type: "image",
|
|
source: {
|
|
type: "base64",
|
|
media_type: "image/png",
|
|
data: "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg==",
|
|
},
|
|
},
|
|
],
|
|
},
|
|
],
|
|
});
|
|
|
|
const result = await guardrail.preCall(payload, createContext({ model: "minimax/minimax-01" }));
|
|
|
|
assert.strictEqual(result.block, false);
|
|
assert.ok(result.modifiedPayload);
|
|
const modified = result.modifiedPayload as { model?: string };
|
|
assert.ok(modified.model, "rerouted model should be set");
|
|
assert.notStrictEqual(
|
|
modified.model,
|
|
"minimax/minimax-01",
|
|
"model should be different from original"
|
|
);
|
|
// Don't assert a specific model — auto-router picks the best available vision model
|
|
assert.strictEqual(visionCallCount, 0, "should NOT call vision API");
|
|
});
|
|
|
|
// ── VB-S03: Fail-open on vision error (via combo mapping path) ────────────
|
|
|
|
test("VB-S03/#8430: combo-mapping describe failure replaces the image with an error stub (not preserved)", async () => {
|
|
shouldVisionFail = true;
|
|
const guardrail = createGuardrail({
|
|
deps: {
|
|
checkModelHasComboMapping: async (_model: string) => true,
|
|
},
|
|
});
|
|
|
|
const payload = createPayload({
|
|
model: "openai/gpt-4o",
|
|
messages: [
|
|
{
|
|
role: "user",
|
|
content: [
|
|
{ type: "text", text: "What is this?" },
|
|
{
|
|
type: "image_url",
|
|
image_url: { url: "https://example.com/image.png" },
|
|
},
|
|
],
|
|
},
|
|
],
|
|
});
|
|
|
|
const result = await guardrail.preCall(payload, createContext({ model: "openai/gpt-4o" }));
|
|
|
|
assert.strictEqual(result.block, false);
|
|
|
|
const modified = (result.modifiedPayload ?? payload) as {
|
|
messages: Array<{ content: unknown[] }>;
|
|
};
|
|
const content = modified.messages[0].content as Array<{
|
|
type: string;
|
|
text?: string;
|
|
}>;
|
|
|
|
// SEMANTIC CHANGE (#8430): in the combo describe path (forced here via
|
|
// checkModelHasComboMapping), when EVERY describe call fails, the upstream is
|
|
// a confirmed non-vision model that cannot handle raw images — the raw
|
|
// image_url part is now replaced with an "(unavailable)" error stub instead
|
|
// of being preserved. The original #4012 preserve-raw behavior still applies
|
|
// to the reroute path, where the upstream model might still be vision-capable
|
|
// (see tests/unit/vision-bridge-preserve-on-failure-4012.test.ts, updated by
|
|
// the same #8430 commit).
|
|
const imagePart = content.find((p) => p.type === "image_url");
|
|
assert.strictEqual(
|
|
imagePart,
|
|
undefined,
|
|
"raw image_url must be replaced when every describe call fails in the combo path"
|
|
);
|
|
const unavailPart = content.find((p) => p.type === "text" && p.text?.includes("unavailable"));
|
|
assert.ok(unavailPart, "an 'unavailable' error stub should be present when describe fails");
|
|
});
|
|
|
|
test("VB-S03: logs warning when vision API fails (via combo mapping)", async () => {
|
|
shouldVisionFail = true;
|
|
let warningLogged = false;
|
|
const guardrail = createGuardrail({
|
|
deps: {
|
|
checkModelHasComboMapping: async (_model: string) => true,
|
|
},
|
|
});
|
|
|
|
const payload = createPayload({
|
|
model: "openai/gpt-4o",
|
|
messages: [
|
|
{
|
|
role: "user",
|
|
content: [
|
|
{
|
|
type: "image_url",
|
|
image_url: { url: "https://example.com/image.png" },
|
|
},
|
|
],
|
|
},
|
|
],
|
|
});
|
|
|
|
const mockLog = {
|
|
warn: (_tag: string, msg: string) => {
|
|
if (msg.includes("Failed to get description")) {
|
|
warningLogged = true;
|
|
}
|
|
},
|
|
};
|
|
|
|
await guardrail.preCall(
|
|
payload,
|
|
createContext({
|
|
model: "openai/gpt-4o",
|
|
log: mockLog as GuardrailContext["log"],
|
|
})
|
|
);
|
|
|
|
assert.strictEqual(warningLogged, true);
|
|
});
|
|
|
|
// ── VB-S09: Image count limit (via combo mapping) ──────────────────────────
|
|
|
|
test("VB-S09: respects maxImages setting in combo mapping path", async () => {
|
|
mockSettings.visionBridgeMaxImages = 2;
|
|
const guardrail = createGuardrail({
|
|
deps: {
|
|
checkModelHasComboMapping: async (_model: string) => true,
|
|
},
|
|
});
|
|
|
|
const images = Array.from({ length: 5 }, (_, i) => ({
|
|
type: "image_url" as const,
|
|
image_url: { url: `https://example.com/image${i}.png` },
|
|
}));
|
|
|
|
const payload = createPayload({
|
|
model: "openai/gpt-4o",
|
|
messages: [
|
|
{
|
|
role: "user",
|
|
content: [{ type: "text", text: "Describe these" }, ...images],
|
|
},
|
|
],
|
|
});
|
|
|
|
await guardrail.preCall(payload, createContext({ model: "openai/gpt-4o" }));
|
|
|
|
// Should only call vision API for 2 images (maxImages=2)
|
|
assert.strictEqual(visionCallCount, 2);
|
|
});
|
|
|
|
// ── VB-S10: Meta information returned (reroute path) ───────────────────────
|
|
|
|
test("VB-S10: returns meta with reroute info for individual non-vision model", async () => {
|
|
const guardrail = createGuardrail();
|
|
|
|
const payload = createPayload({
|
|
model: "minimax/minimax-01",
|
|
messages: [
|
|
{
|
|
role: "user",
|
|
content: [
|
|
{ type: "text", text: "What is this?" },
|
|
{
|
|
type: "image_url",
|
|
image_url: { url: "https://example.com/a.png" },
|
|
},
|
|
{
|
|
type: "image_url",
|
|
image_url: { url: "https://example.com/b.png" },
|
|
},
|
|
],
|
|
},
|
|
],
|
|
});
|
|
|
|
const result = await guardrail.preCall(payload, createContext({ model: "minimax/minimax-01" }));
|
|
|
|
assert.ok(result.meta);
|
|
assert.ok(typeof result.meta === "object");
|
|
|
|
const meta = result.meta as Record<string, unknown>;
|
|
assert.strictEqual(meta.rerouted, true);
|
|
assert.strictEqual(meta.fromModel, "minimax/minimax-01");
|
|
assert.ok(
|
|
typeof meta.toModel === "string" && meta.toModel.length > 0,
|
|
"toModel should be a non-empty string"
|
|
);
|
|
assert.notStrictEqual(meta.toModel, "minimax/minimax-01", "toModel should differ from original");
|
|
assert.strictEqual(meta.imagesKept, 2);
|
|
});
|
|
|
|
// ── VB-S01b: Describe images via combo mapping path ────────────────────────
|
|
|
|
test("VB-S01b: describes images when combo mapping forces process path", async () => {
|
|
mockVisionResponse = "A cat sitting on a windowsill";
|
|
const guardrail = createGuardrail({
|
|
deps: {
|
|
checkModelHasComboMapping: async (_model: string) => true,
|
|
},
|
|
});
|
|
|
|
const payload = createPayload({
|
|
model: "openai/gpt-4o",
|
|
messages: [
|
|
{
|
|
role: "user",
|
|
content: [
|
|
{ type: "text", text: "What is this?" },
|
|
{
|
|
type: "image_url",
|
|
image_url: { url: "https://example.com/cat.png" },
|
|
},
|
|
],
|
|
},
|
|
],
|
|
});
|
|
|
|
const result = await guardrail.preCall(payload, createContext({ model: "openai/gpt-4o" }));
|
|
|
|
assert.strictEqual(result.block, false);
|
|
assert.ok(result.modifiedPayload);
|
|
|
|
const modified = result.modifiedPayload as { messages: Array<{ content: unknown[] }> };
|
|
const content = modified.messages[0].content as Array<{ type: string; text?: string }>;
|
|
|
|
// Images should be replaced with text descriptions (combo path)
|
|
const imagePart = content.find((p) => p.type === "image_url");
|
|
assert.strictEqual(imagePart, undefined, "image should be replaced by description");
|
|
|
|
const descriptionPart = content.find((p) => p.type === "text" && p.text?.includes("cat"));
|
|
assert.ok(descriptionPart, "description should be present");
|
|
assert.ok(visionCallCount > 0, "vision API should have been called for description");
|
|
});
|
|
|
|
// ── VB-S11: Combo mapping forces vision processing despite vision-capable model ──
|
|
|
|
test("VB-S11: processes images when vision-capable model has combo mapping", async () => {
|
|
mockVisionResponse = "A description from combo-mapped vision bridge";
|
|
const guardrail = createGuardrail({
|
|
deps: {
|
|
checkModelHasComboMapping: async (_model: string) => true,
|
|
},
|
|
});
|
|
|
|
const payload = createPayload({
|
|
model: "openai/gpt-4o",
|
|
messages: [
|
|
{
|
|
role: "user",
|
|
content: [
|
|
{ type: "text", text: "What is this?" },
|
|
{
|
|
type: "image_url",
|
|
image_url: { url: "https://example.com/image.png" },
|
|
},
|
|
],
|
|
},
|
|
],
|
|
});
|
|
|
|
const startCallCount = visionCallCount;
|
|
const result = await guardrail.preCall(payload, createContext({ model: "openai/gpt-4o" }));
|
|
|
|
// Vision bridge should have processed the image
|
|
assert.strictEqual(result.block, false);
|
|
assert.ok(visionCallCount > startCallCount, "Expected vision model to be called");
|
|
assert.ok(
|
|
result.modifiedPayload !== undefined,
|
|
"Expected modifiedPayload when combo mapping forces vision bridge"
|
|
);
|
|
});
|
|
|
|
test("VB-S11b: passthroughs when vision-capable model has NO combo mapping", async () => {
|
|
const guardrail = createGuardrail({
|
|
deps: {
|
|
checkModelHasComboMapping: async (_model: string) => false,
|
|
},
|
|
});
|
|
|
|
const payload = createPayload({
|
|
model: "openai/gpt-4o",
|
|
messages: [
|
|
{
|
|
role: "user",
|
|
content: [
|
|
{ type: "text", text: "What is this?" },
|
|
{
|
|
type: "image_url",
|
|
image_url: { url: "https://example.com/image.png" },
|
|
},
|
|
],
|
|
},
|
|
],
|
|
});
|
|
|
|
const result = await guardrail.preCall(payload, createContext({ model: "openai/gpt-4o" }));
|
|
|
|
// Vision bridge should skip (passthrough) since model supports vision + no combo mapping
|
|
assert.strictEqual(result.block, false);
|
|
assert.strictEqual(result.modifiedPayload, undefined);
|
|
assert.strictEqual(visionCallCount, 0);
|
|
});
|
|
|
|
// ── Credential-aware whole-request reroute (combo zai hijack fix) ───────────
|
|
|
|
test("VB-CRED-01: does NOT whole-request-reroute when original model has usable credentials", async () => {
|
|
// Repro: OpenCode 94-msg body with images + combo target zai/glm-5.2 was
|
|
// hijacked to opencode-zen/gpt-5.4 (priority 0, noauth) → 401 Missing API key.
|
|
const guardrail = createGuardrail({
|
|
deps: {
|
|
hasUsableCredentials: async (m: string) =>
|
|
m.startsWith("zai/") ? true : m.startsWith("opencode-") ? false : null,
|
|
},
|
|
});
|
|
|
|
const payload = createPayload({
|
|
model: "zai/glm-5.2",
|
|
messages: [
|
|
{
|
|
role: "user",
|
|
content: [
|
|
{ type: "text", text: "What is in this screenshot?" },
|
|
{
|
|
type: "image_url",
|
|
image_url: { url: "https://example.com/shot.png" },
|
|
},
|
|
],
|
|
},
|
|
],
|
|
});
|
|
|
|
const result = await guardrail.preCall(payload, createContext({ model: "zai/glm-5.2" }));
|
|
assert.strictEqual(result.block, false);
|
|
// Must NOT swap model to opencode-zen / openai vision target
|
|
if (result.modifiedPayload) {
|
|
const modified = result.modifiedPayload as { model?: string };
|
|
assert.strictEqual(
|
|
modified.model,
|
|
"zai/glm-5.2",
|
|
"credentialed original model must not be whole-request-rerouted"
|
|
);
|
|
}
|
|
const meta = result.meta as Record<string, unknown> | undefined;
|
|
assert.notStrictEqual(meta?.rerouted, true, "must not set rerouted meta for credentialed model");
|
|
});
|
|
|
|
test("VB-CRED-01A: reroutes a credentialed text-only model when configured to preserve images", async () => {
|
|
mockSettings.visionBridgeRerouteTextOnly = true;
|
|
const guardrail = createGuardrail({
|
|
deps: {
|
|
hasUsableCredentials: async (m: string) =>
|
|
m === "zai/glm-5.2" || m === "openai/gpt-4o-mini" ? true : null,
|
|
},
|
|
});
|
|
|
|
const payload = createPayload({
|
|
model: "zai/glm-5.2",
|
|
messages: [
|
|
{
|
|
role: "user",
|
|
content: [
|
|
{ type: "text", text: "What is in this screenshot?" },
|
|
{
|
|
type: "image_url",
|
|
image_url: { url: "https://example.com/shot.png" },
|
|
},
|
|
],
|
|
},
|
|
],
|
|
});
|
|
|
|
const result = await guardrail.preCall(payload, createContext({ model: "zai/glm-5.2" }));
|
|
assert.strictEqual(result.block, false);
|
|
|
|
const modified = result.modifiedPayload as {
|
|
model?: string;
|
|
messages: Array<{ content: Array<{ type: string; image_url?: { url: string } }> }>;
|
|
};
|
|
assert.strictEqual(modified.model, "openai/gpt-4o-mini");
|
|
assert.deepStrictEqual(modified.messages[0].content[1], {
|
|
type: "image_url",
|
|
image_url: { url: "https://example.com/shot.png" },
|
|
});
|
|
assert.strictEqual(visionCallCount, 0, "the bridge must not replace the image with text");
|
|
});
|
|
|
|
test("VB-CRED-02: does NOT reroute to a vision model known to lack credentials", async () => {
|
|
mockSettings.visionBridgeModel = "opencode-zen/gpt-5.4";
|
|
const guardrail = createGuardrail({
|
|
deps: {
|
|
// Original unusable, best vision model also unusable
|
|
hasUsableCredentials: async () => false,
|
|
},
|
|
});
|
|
|
|
const payload = createPayload({
|
|
model: "minimax/minimax-01",
|
|
messages: [
|
|
{
|
|
role: "user",
|
|
content: [
|
|
{ type: "text", text: "Describe" },
|
|
{ type: "image_url", image_url: { url: "https://example.com/a.png" } },
|
|
],
|
|
},
|
|
],
|
|
});
|
|
|
|
const result = await guardrail.preCall(payload, createContext({ model: "minimax/minimax-01" }));
|
|
assert.strictEqual(result.block, false);
|
|
const meta = result.meta as Record<string, unknown> | undefined;
|
|
assert.notStrictEqual(meta?.rerouted, true, "must not reroute to unusable vision model");
|
|
});
|
|
|
|
test("isProviderConnectionUsable rejects noauth without api key", async () => {
|
|
const { isProviderConnectionUsable } =
|
|
await import("../../../src/lib/guardrails/visionBridge.ts");
|
|
assert.strictEqual(isProviderConnectionUsable({ authType: "noauth", apiKey: null }), false);
|
|
assert.strictEqual(isProviderConnectionUsable({ authType: "apikey", apiKey: "sk-real" }), true);
|
|
assert.strictEqual(isProviderConnectionUsable({ authType: "oauth", refreshToken: "rt" }), true);
|
|
assert.strictEqual(
|
|
isProviderConnectionUsable({ authType: "apikey", apiKey: "x", testStatus: "banned" }),
|
|
false
|
|
);
|
|
});
|