mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-08-14 03:02:14 +03:00
Compare commits
3 Commits
feat/i-hav
...
feat/bridg
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f139a107f2 | ||
|
|
89dee4a2df | ||
|
|
e04a91aa18 |
@@ -107,6 +107,26 @@ fragment the cache. Failed describes are never cached. Settings:
|
||||
| `modalityBridgeCacheTtlMinutes` | `60` | 1–1440 |
|
||||
| `modalityBridgeCacheMaxEntries` | `200` | 10–5000 |
|
||||
|
||||
#### Remote image normalization (self-loop describe/base64 fetch)
|
||||
|
||||
When the bridge fetches a **remote** image itself — the Anthropic describe
|
||||
self-call and the claude-wire-format base64 conversion
|
||||
(`ensureBase64ImagesForClaudeWire`), both via
|
||||
`fetchRemoteImageAsDataUri()` in `visionBridgeHelpers.ts` — the resulting data
|
||||
URI is passed through `normalizeDataUri()`
|
||||
(`open-sse/utils/imageNormalize.ts`) before being embedded in the vision-model
|
||||
request. Oversized images are downscaled to a **2048px long edge** (matching
|
||||
the resize cap OpenAI/Anthropic already apply server-side), which cuts
|
||||
upload bytes/latency without changing what the vision model sees. Resizing
|
||||
uses `sharp`, loaded via dynamic import: on a platform where its native
|
||||
binary fails to load, `normalizeDataUri()` **never throws** — it falls back
|
||||
to a passthrough of the original bytes, so the describe/base64-conversion
|
||||
path always keeps working. Non-image bytes (a fetch that did not return a
|
||||
decodable image) are also passed through untouched. This normalization is
|
||||
scoped to images the bridge fetches for its own self-call — it is never
|
||||
applied to the caller's raw passthrough payload, consistent with the
|
||||
opt-in-only mutation principle (Hard Rule #20).
|
||||
|
||||
#### Settings schema + migration
|
||||
|
||||
The new `modalityBridge*` keys are Zod-validated in `updateSettingsSchema`
|
||||
|
||||
@@ -95,30 +95,6 @@ export const OUTPUT_STYLE_CATALOG: Record<string, OutputStyle> = {
|
||||
},
|
||||
},
|
||||
},
|
||||
// i-have-adhd (action-first output) — integrated into the output-style registry
|
||||
// so it rides the existing production injector, like ponytail.
|
||||
// Source: https://github.com/ayghri/i-have-adhd (MIT). The upstream skill's 10
|
||||
// ADHD-friendly rules, adapted for proxy injection: agent-harness-specific rules
|
||||
// (restate plan state, time estimates) reworded as conditionals so they hold for
|
||||
// plain chat clients too.
|
||||
"i-have-adhd": {
|
||||
id: "i-have-adhd",
|
||||
label: "I have ADHD (action-first)",
|
||||
description:
|
||||
"Action-first output: next action leads, steps numbered, one concrete next step, no preamble.",
|
||||
levels: {
|
||||
lite: `# I have ADHD (lite)\nLead with the action: command, path, or snippet first, prose after. Number multi-step work; each step one bounded action. End with ONE concrete next step. No preamble, no recap, no closing pleasantries. ${SHARED_BOUNDARIES}`,
|
||||
full: `# I have ADHD — action-first output\n\nThe reader has ADHD. Shape output so an ADHD brain can act on it:\n1. Lead with the next action — command, path, or snippet first; context after, if at all.\n2. Number multi-step work; each step is one bounded action; use the fewest steps that work.\n3. End with ONE concrete next step doable in under two minutes.\n4. Suppress tangents: finish the first issue, offer the second as a separate question.\n5. In multi-turn work, restate where things stand ("step 3 of 5 done") — the reader cannot hold state between messages.\n6. When human effort is involved, estimate it in concrete units (minutes, an afternoon), never "some work".\n7. Make wins visible: state what now works and how to try it.\n8. Errors matter-of-fact: cause and fix; never "Uh oh".\n9. Cap lists at 5 items; split into "do now" vs "later" beyond that.\n10. No preamble, no recap, no closers ("Hope this helps").\nExceptions: an explicit "explain" request gets a full body (still no preamble/closer); destructive actions get confirmation first; real ambiguity gets one short clarifying question. ${SHARED_BOUNDARIES}`,
|
||||
ultra: `# I have ADHD (ultra)\nAction first: command/path/snippet, then prose if needed. Numbered bounded steps, fewest that work. One <2-min next step at the end. No tangents — separate question. Multi-turn: restate state. Human effort: concrete time units. Wins visible. Errors: cause + fix. Lists ≤5. Zero preamble/recap/closers. Explain-requests get full body; destructive actions get confirmation; real ambiguity gets one question. ${SHARED_BOUNDARIES}`,
|
||||
},
|
||||
i18n: {
|
||||
"pt-BR": {
|
||||
lite: `# Eu tenho TDAH (lite)\nComece pela ação: comando, path ou snippet primeiro, prosa depois. Numere trabalho multi-passo; cada passo é uma ação delimitada. Termine com UMA próxima ação concreta. Sem preâmbulo, sem recap, sem despedidas. ${SHARED_BOUNDARIES}`,
|
||||
full: `# Eu tenho TDAH — saída action-first\n\nO leitor tem TDAH. Molde a saída para que um cérebro TDAH consiga agir sobre ela:\n1. Comece pela próxima ação — comando, path ou snippet primeiro; contexto depois, se necessário.\n2. Numere trabalho multi-passo; cada passo é uma ação delimitada; use o menor número de passos que funcione.\n3. Termine com UMA próxima ação concreta executável em menos de dois minutos.\n4. Suprima tangentes: termine a primeira questão, ofereça a segunda como pergunta separada.\n5. Em trabalho multi-turno, reafirme onde as coisas estão ("passo 3 de 5 feito") — o leitor não guarda estado entre mensagens.\n6. Quando houver esforço humano, estime em unidades concretas (minutos, uma tarde), nunca "um pouco de trabalho".\n7. Torne vitórias visíveis: diga o que funciona agora e como testar.\n8. Erros de forma direta: causa e fix; nunca "Opa!".\n9. Listas com no máximo 5 itens; acima disso, divida em "agora" vs "depois".\n10. Sem preâmbulo, sem recap, sem despedidas ("Espero ter ajudado").\nExceções: pedido explícito de "explique" recebe corpo completo (ainda sem preâmbulo/despedida); ações destrutivas recebem confirmação antes; ambiguidade real recebe uma pergunta curta de esclarecimento. ${SHARED_BOUNDARIES}`,
|
||||
ultra: `# Eu tenho TDAH (ultra)\nAção primeiro: comando/path/snippet, prosa depois se precisar. Passos numerados e delimitados, o mínimo que funcione. UMA próxima ação <2 min no fim. Sem tangentes — pergunta separada. Multi-turno: reafirme o estado. Esforço humano: unidades concretas de tempo. Vitórias visíveis. Erros: causa + fix. Listas ≤5. Zero preâmbulo/recap/despedidas. "Explique" recebe corpo completo; ação destrutiva recebe confirmação; ambiguidade real recebe uma pergunta. ${SHARED_BOUNDARIES}`,
|
||||
},
|
||||
},
|
||||
},
|
||||
"terse-cjk": {
|
||||
id: "terse-cjk",
|
||||
label: "Terse CJK (文言)",
|
||||
|
||||
62
open-sse/utils/imageNormalize.ts
Normal file
62
open-sse/utils/imageNormalize.ts
Normal file
@@ -0,0 +1,62 @@
|
||||
/**
|
||||
* Optional-sharp image normalization.
|
||||
*
|
||||
* Rationale (migrated from freellmapi `server/src/lib/image-normalize.ts:40-58`):
|
||||
* OpenAI resizes images to a long-edge cap of 2048px server-side, Anthropic applies
|
||||
* a similar cap. Downscaling client-side before upload reduces tokens/latency without
|
||||
* changing model behavior. `sharp` is loaded via dynamic import so that a platform
|
||||
* where its native binary fails to load never crashes the request path — it just
|
||||
* falls back to a passthrough (original buffer, unresized).
|
||||
*/
|
||||
|
||||
const DEFAULT_MAX_LONG_EDGE = 2048;
|
||||
|
||||
type SharpModule = typeof import("sharp");
|
||||
let sharpPromise: Promise<SharpModule | null> | null = null;
|
||||
|
||||
async function loadSharp(): Promise<SharpModule | null> {
|
||||
if (!sharpPromise) {
|
||||
sharpPromise = import("sharp").then((m) => (m.default ?? m) as SharpModule).catch(() => null);
|
||||
}
|
||||
return sharpPromise;
|
||||
}
|
||||
|
||||
export async function normalizeImageBuffer(
|
||||
input: Buffer,
|
||||
opts?: { maxLongEdge?: number }
|
||||
): Promise<{ buffer: Buffer; mime: string | null; resized: boolean }> {
|
||||
const maxLongEdge = opts?.maxLongEdge ?? DEFAULT_MAX_LONG_EDGE;
|
||||
const sharp = await loadSharp();
|
||||
if (!sharp) return { buffer: input, mime: null, resized: false };
|
||||
try {
|
||||
const img = sharp(input, { failOn: "error" });
|
||||
const meta = await img.metadata();
|
||||
const long = Math.max(meta.width ?? 0, meta.height ?? 0);
|
||||
if (!long || long <= maxLongEdge) {
|
||||
return { buffer: input, mime: meta.format ? `image/${meta.format}` : null, resized: false };
|
||||
}
|
||||
const buffer = await img
|
||||
.resize({ width: maxLongEdge, height: maxLongEdge, fit: "inside", withoutEnlargement: true })
|
||||
.toBuffer();
|
||||
return { buffer, mime: meta.format ? `image/${meta.format}` : null, resized: true };
|
||||
} catch {
|
||||
return { buffer: input, mime: null, resized: false };
|
||||
}
|
||||
}
|
||||
|
||||
export async function normalizeDataUri(
|
||||
dataUri: string,
|
||||
opts?: { maxLongEdge?: number }
|
||||
): Promise<string> {
|
||||
try {
|
||||
const match = /^data:([^;,]+);base64,(.*)$/s.exec(dataUri);
|
||||
if (!match) return dataUri;
|
||||
const input = Buffer.from(match[2], "base64");
|
||||
if (!input.length) return dataUri;
|
||||
const out = await normalizeImageBuffer(input, opts);
|
||||
if (!out.resized) return dataUri;
|
||||
return `data:${match[1]};base64,${out.buffer.toString("base64")}`;
|
||||
} catch {
|
||||
return dataUri;
|
||||
}
|
||||
}
|
||||
@@ -2,6 +2,7 @@
|
||||
* Vision Bridge helper functions for image processing.
|
||||
*/
|
||||
import { detectMediaParts, type MediaPart } from "@omniroute/open-sse/utils/mediaParts";
|
||||
import { normalizeDataUri } from "@omniroute/open-sse/utils/imageNormalize";
|
||||
import { fetchRemoteImage } from "@/shared/network/remoteImageFetch";
|
||||
import { getRuntimePorts } from "@/lib/runtime/ports";
|
||||
import { resolveSelfLoopBearer } from "@/shared/middleware/chatBodyAdmission";
|
||||
@@ -314,7 +315,12 @@ async function fetchRemoteImageAsDataUri(
|
||||
fetchImpl,
|
||||
});
|
||||
const mediaType = remoteImage.contentType.split(";")[0]?.trim() || "image/png";
|
||||
return `data:${mediaType};base64,${remoteImage.buffer.toString("base64")}`;
|
||||
const dataUri = `data:${mediaType};base64,${remoteImage.buffer.toString("base64")}`;
|
||||
// Downscale to the long-edge cap before handing the image to the vision
|
||||
// model self-call — scoped to this bridge-fetched image only, never the
|
||||
// user's raw passthrough payload (opt-in principle, HR#20).
|
||||
// `normalizeDataUri` never throws and is a passthrough for non-image bytes.
|
||||
return normalizeDataUri(dataUri);
|
||||
}
|
||||
|
||||
async function normalizeVisionImageInput(
|
||||
|
||||
@@ -1,76 +0,0 @@
|
||||
/**
|
||||
* Tests for the i-have-adhd output style — action-first prompt injection.
|
||||
*
|
||||
* Verifies:
|
||||
* - i-have-adhd is registered with lite/full/ultra levels
|
||||
* - i18n map exists for pt-BR with all three levels
|
||||
* - Each level (en and pt-BR) contains the SHARED_BOUNDARIES suffix
|
||||
* - Core concepts present: action-first, numbered steps, no preamble
|
||||
* - No locale gate (style valid under every language)
|
||||
*/
|
||||
|
||||
import { describe, it } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import {
|
||||
OUTPUT_STYLE_CATALOG,
|
||||
outputStyleMeta,
|
||||
} from "../../../open-sse/services/compression/outputStyles/catalog.ts";
|
||||
|
||||
const ADHD = OUTPUT_STYLE_CATALOG["i-have-adhd"];
|
||||
|
||||
function assertString(v: unknown, label: string): asserts v is string {
|
||||
assert.equal(typeof v, "string", `${label} must be a string`);
|
||||
}
|
||||
|
||||
describe("i-have-adhd output style", () => {
|
||||
it("is registered in the catalog with lite/full/ultra levels", () => {
|
||||
assert.ok(ADHD, "i-have-adhd must be in catalog");
|
||||
assert.equal(ADHD.id, "i-have-adhd");
|
||||
assert.ok(ADHD.label.includes("ADHD"));
|
||||
assertString(ADHD.levels.lite, "lite");
|
||||
assertString(ADHD.levels.full, "full");
|
||||
assertString(ADHD.levels.ultra, "ultra");
|
||||
});
|
||||
|
||||
it("every level ends with the shared boundaries suffix", () => {
|
||||
const shared = "Code blocks, file paths";
|
||||
assert.ok(ADHD.levels.lite.includes(shared));
|
||||
assert.ok(ADHD.levels.full.includes(shared));
|
||||
assert.ok(ADHD.levels.ultra.includes(shared));
|
||||
});
|
||||
|
||||
it("the full level contains the action-first core concepts", () => {
|
||||
assert.ok(ADHD.levels.full.includes("Lead with the next action"));
|
||||
assert.ok(/[Nn]umber/.test(ADHD.levels.full), "full mentions numbered steps");
|
||||
assert.ok(ADHD.levels.full.includes("No preamble"));
|
||||
});
|
||||
|
||||
it("has an i18n map for pt-BR with all three intensity levels", () => {
|
||||
assert.ok(ADHD.i18n, "i18n must be defined");
|
||||
const pt = ADHD.i18n["pt-BR"];
|
||||
assert.ok(pt, "pt-BR must exist");
|
||||
assertString(pt.lite, "pt-BR.lite");
|
||||
assertString(pt.full, "pt-BR.full");
|
||||
assertString(pt.ultra, "pt-BR.ultra");
|
||||
});
|
||||
|
||||
it("each pt-BR level ends with shared boundaries", () => {
|
||||
const shared = "Code blocks";
|
||||
const pt = ADHD.i18n?.["pt-BR"];
|
||||
assert.ok(pt, "pt-BR i18n must exist");
|
||||
assert.ok(pt.lite.includes(shared), "pt-BR.lite contains shared boundaries");
|
||||
assert.ok(pt.full.includes(shared), "pt-BR.full contains shared boundaries");
|
||||
assert.ok(pt.ultra.includes(shared), "pt-BR.ultra contains shared boundaries");
|
||||
});
|
||||
|
||||
it("pt-BR full contains Portuguese action-first terminology", () => {
|
||||
const pt = ADHD.i18n?.["pt-BR"];
|
||||
assert.ok(pt, "pt-BR i18n must exist");
|
||||
assert.ok(/ação/.test(pt.full), "pt-BR.full mentions ação");
|
||||
assert.ok(/preâmbulo/.test(pt.full), "pt-BR.full mentions preâmbulo");
|
||||
});
|
||||
|
||||
it("carries no locale gate", () => {
|
||||
assert.equal(outputStyleMeta("i-have-adhd").locale, undefined);
|
||||
});
|
||||
});
|
||||
52
tests/unit/image-normalize.test.ts
Normal file
52
tests/unit/image-normalize.test.ts
Normal file
@@ -0,0 +1,52 @@
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { normalizeImageBuffer, normalizeDataUri } from "../../open-sse/utils/imageNormalize.ts";
|
||||
|
||||
test("passthrough when input is not a decodable image (sharp absent or garbage bytes)", async () => {
|
||||
const junk = Buffer.from("not-an-image");
|
||||
const out = await normalizeImageBuffer(junk);
|
||||
assert.equal(out.resized, false);
|
||||
assert.ok(out.buffer.equals(junk));
|
||||
});
|
||||
|
||||
test("normalizeDataUri never throws and preserves the uri on failure", async () => {
|
||||
const uri = "data:image/png;base64,%%%broken%%%";
|
||||
assert.equal(await normalizeDataUri(uri), uri);
|
||||
});
|
||||
|
||||
// Só roda quando sharp estiver instalado (optionalDependency presente no devbox):
|
||||
test("downscales a large PNG to the long-edge cap when sharp is available", async (t) => {
|
||||
let sharp: typeof import("sharp");
|
||||
try {
|
||||
sharp = (await import("sharp")).default as never;
|
||||
} catch {
|
||||
t.skip("sharp not installed");
|
||||
return;
|
||||
}
|
||||
const big = await sharp({ create: { width: 4096, height: 100, channels: 3, background: "#fff" } })
|
||||
.png()
|
||||
.toBuffer();
|
||||
const out = await normalizeImageBuffer(big, { maxLongEdge: 2048 });
|
||||
assert.equal(out.resized, true);
|
||||
const meta = await sharp(out.buffer).metadata();
|
||||
assert.equal(meta.width, 2048);
|
||||
});
|
||||
|
||||
test("downscales a height-dominant PNG to the long-edge cap on the height axis", async (t) => {
|
||||
let sharp: typeof import("sharp");
|
||||
try {
|
||||
sharp = (await import("sharp")).default as never;
|
||||
} catch {
|
||||
t.skip("sharp not installed");
|
||||
return;
|
||||
}
|
||||
const tall = await sharp({
|
||||
create: { width: 100, height: 4096, channels: 3, background: "#fff" },
|
||||
})
|
||||
.png()
|
||||
.toBuffer();
|
||||
const out = await normalizeImageBuffer(tall, { maxLongEdge: 2048 });
|
||||
assert.equal(out.resized, true);
|
||||
const meta = await sharp(out.buffer).metadata();
|
||||
assert.equal(meta.height, 2048);
|
||||
});
|
||||
95
tests/unit/vision-bridge-image-normalize.test.ts
Normal file
95
tests/unit/vision-bridge-image-normalize.test.ts
Normal file
@@ -0,0 +1,95 @@
|
||||
/**
|
||||
* Task B2: the vision bridge self-loop fetches a remote image and hands it
|
||||
* to the vision model as a data URI (`fetchRemoteImageAsDataUri`,
|
||||
* `src/lib/guardrails/visionBridgeHelpers.ts`). That fetched image must be
|
||||
* normalized (long-edge cap 2048, `@omniroute/open-sse/utils/imageNormalize`)
|
||||
* before being embedded — the same treatment `normalizeDataUri` already
|
||||
* gives any other image, now applied to remote fetches performed by the
|
||||
* bridge itself. Scope: ONLY this self-call path, never the user's raw
|
||||
* passthrough payload (HR#20 opt-in principle).
|
||||
*
|
||||
* `ensureBase64ImagesForClaudeWire` is the exported entry point that reaches
|
||||
* the private `fetchRemoteImageAsDataUri` — it resolves every non-data-URI
|
||||
* image part of a claude-wire-format request via that same fetch helper, so
|
||||
* it is the smallest public surface to exercise the fetch → normalize path
|
||||
* with dependency-injected `fetchImpl` (mirrors the DI pattern used by
|
||||
* `tests/unit/vision-bridge-describe-cache.test.ts` and
|
||||
* `tests/unit/remote-image-fetch.test.ts`).
|
||||
*/
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import { ensureBase64ImagesForClaudeWire } from "../../src/lib/guardrails/visionBridgeHelpers.ts";
|
||||
|
||||
// zai speaks the claude wire format (open-sse/config/providers/registry/zai/index.ts),
|
||||
// so `isClaudeWireFormatModel` routes it through the base64 self-fetch path.
|
||||
const CLAUDE_WIRE_MODEL = "zai/glm-4.6";
|
||||
|
||||
function bodyWithRemoteImage(url: string) {
|
||||
return {
|
||||
messages: [
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "text", text: "describe this" },
|
||||
{ type: "image_url", image_url: { url } },
|
||||
],
|
||||
},
|
||||
],
|
||||
};
|
||||
}
|
||||
|
||||
test("remote image fetched for the claude-wire self-call is downscaled to the long-edge cap", async (t) => {
|
||||
let sharp: typeof import("sharp");
|
||||
try {
|
||||
sharp = (await import("sharp")).default as never;
|
||||
} catch {
|
||||
t.skip("sharp not installed");
|
||||
return;
|
||||
}
|
||||
const big = await sharp({ create: { width: 4096, height: 100, channels: 3, background: "#fff" } })
|
||||
.png()
|
||||
.toBuffer();
|
||||
|
||||
const fetchImpl = (async () =>
|
||||
new Response(big, {
|
||||
status: 200,
|
||||
headers: { "content-type": "image/png" },
|
||||
})) as unknown as typeof fetch;
|
||||
|
||||
const result = await ensureBase64ImagesForClaudeWire(
|
||||
bodyWithRemoteImage("https://example.com/big.png"),
|
||||
CLAUDE_WIRE_MODEL,
|
||||
fetchImpl
|
||||
);
|
||||
|
||||
const imagePart = (result.messages?.[0]?.content as Array<{ image_url?: { url: string } }>)[1];
|
||||
const dataUri = imagePart?.image_url?.url ?? "";
|
||||
assert.match(dataUri, /^data:image\/png;base64,/);
|
||||
|
||||
const b64 = dataUri.split(",")[1] ?? "";
|
||||
const decoded = Buffer.from(b64, "base64");
|
||||
const meta = await sharp(decoded).metadata();
|
||||
assert.ok((meta.width ?? 0) <= 2048, `expected width <= 2048, got ${meta.width}`);
|
||||
assert.notEqual(meta.width, 4096, "image must have been downscaled, not left at 4096");
|
||||
});
|
||||
|
||||
test("remote non-image bytes pass through untouched (fail-open, no normalization)", async () => {
|
||||
const junk = Buffer.from("not-an-image-at-all");
|
||||
|
||||
const fetchImpl = (async () =>
|
||||
new Response(junk, {
|
||||
status: 200,
|
||||
headers: { "content-type": "application/octet-stream" },
|
||||
})) as unknown as typeof fetch;
|
||||
|
||||
const result = await ensureBase64ImagesForClaudeWire(
|
||||
bodyWithRemoteImage("https://example.com/junk.bin"),
|
||||
CLAUDE_WIRE_MODEL,
|
||||
fetchImpl
|
||||
);
|
||||
|
||||
const imagePart = (result.messages?.[0]?.content as Array<{ image_url?: { url: string } }>)[1];
|
||||
const dataUri = imagePart?.image_url?.url ?? "";
|
||||
assert.equal(dataUri, `data:application/octet-stream;base64,${junk.toString("base64")}`);
|
||||
});
|
||||
Reference in New Issue
Block a user