mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-08-09 16:53:13 +03:00
Compare commits
9 Commits
feat/9544-
...
maint/cher
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a1f2d98a29 | ||
|
|
a5305808e6 | ||
|
|
2b4cbfb018 | ||
|
|
c624581ee3 | ||
|
|
cb8d9ef833 | ||
|
|
aae408f585 | ||
|
|
3e1c31c606 | ||
|
|
2e12ee89f7 | ||
|
|
723ce0b166 |
@@ -1 +0,0 @@
|
||||
- feat(providers): add Muse Code CLI provider preset (#9544)
|
||||
@@ -1,4 +1,5 @@
|
||||
{
|
||||
"_rebaseline_2026_08_09_9296_adobe_media_capabilities": "PR #9296 (artickc, fix/adobe-firefly-model-capabilities) own growth: src/app/api/v1/models/catalog.ts 1590->1597 (+7). The image and video catalog serializers now expose the already-normalized Adobe Firefly discovery capability data (media_capabilities, plus the existing video modality/size fields) at their only response-emission chokepoints. The discovery parser and capability normalization remain in open-sse/services/adobeFireflyModels.ts; extracting these seven serialization fields would obscure the catalog contract. Covered by tests/unit/adobe-firefly.test.ts and tests/unit/image-upscale.test.ts.",
|
||||
"_rebaseline_2026_07_24_8470_hyperagent_sticky_thread": "PR #8470 (artickc, fix/hyperagent-tool-loop-thread-sticky) own growth: open-sse/executors/hyperagent.ts 936->1025 (wc -l; check-file-size.mjs counts via split(\"\\n\").length so the gate sees 937->1026, +89, crosses the 1000 cap). Fixes a real bug where a reverse-conversion proxy (text-Intent/JSON to Claude Code native tool_calls) rewrites assistant messages between agentic tool-loop turns, breaking HyperAgent’s conversation-prefix fingerprint and cold-starting the thread mid tool-loop. Adds Anthropic tool_use/tool_result flattening to extractMessageText() plus a new rootUserFingerprint()/root-key lookup tier in resolveHyperAgentThreadBinding()/storeHyperAgentThreadAfterTurn() so the thread stays sticky across the tool loop. Cohesive additions inside the existing single-file executor; not extractable without splitting the executor mid-request-flow. Covered by tests/unit/executor-hyperagent.test.ts (19/19, +5 new cases for tool_result/tool_use flattening + root-key stickiness). Pre-merge review flagged a cross-conversation root-key collision risk (tracked in the PR’s own mandatory pre-merge checklist, not yet addressed) — unrelated to this file-size ratchet, tracked separately by /fix-prs.",
|
||||
"_rebaseline_2026_07_25_8494_capability_filter_fail_closed": "PR #8494 (fix/capability-filters-fail-closed, #8488) own growth: open-sse/services/combo.ts 3640->3693 (+53) adds a fail-closed guard after filterTargetsByRequestCompatibility() — when every eligible target is excluded by request-capability filtering (vision/tools/etc) instead of quota/health, the combo now returns an explicit `capability_mismatch` 400 (describeCapabilityFilterExhaustion, imported from combo/comboStructure.ts) rather than silently falling through to a generic no-targets error, plus a `compatFilterFailOpen` escape hatch (combo config OR settings) mirrored at both the main/auto and round-robin call sites for symmetry. combo/comboStructure.ts (previously under cap, un-frozen) grows 794->918 (+124) — new home for describeCapabilityFilterExhaustion + providerSupportsEmulatedToolCalling (#5240 emulated tool-calling exemption so fail-closed does not regress prompt-emulation-only combos like all-chatgpt-web). Irreducible orchestration wiring at the existing filter chokepoint (same precedent as #7301's universal-cooldown-retry generalization). Companion test tests/unit/combo-routing-engine.test.ts 3409->3449 (+40, fail-closed/fail-open coverage across both call sites) also rebaselined. Covered by tests/unit/8488-capability-filter-fail-closed.test.ts (new) + 95/95 passing across both files. Structural shrink of combo.ts tracked in #3501.",
|
||||
"_rebaseline_2026_07_25_8499_ts7_result_union_predicates": "PR #8499 (backryun, chore/ts7-types-executor-scattered) own growth: muse-spark-web.ts 1396->1405 (+9, irreducible). Under this workspace's `strictNullChecks: false`, the boolean-literal discriminant on `GraphqlResult` (`{ ok: true } | { ok: false; error: string }`) narrows the positive `.ok===true` branch but leaves `!result.ok` at the full union under TS7, making `.error` unreachable to the checker at the two call sites (warmup, mode-switch). Fixed by adding a single `isGraphqlFailure()` type-predicate helper (doc comment + 3-line body) reused at both call sites instead of duplicating the predicate inline — not extractable to a shared module without splitting a single-file executor's local narrowing helper out of its own file. Covered by the existing muse-spark-web executor test suite (no behavior change, pure narrowing fix).",
|
||||
@@ -387,7 +388,7 @@
|
||||
"src/app/(dashboard)/dashboard/usage/components/EvalsTab.tsx": 2148,
|
||||
"src/app/(dashboard)/dashboard/usage/components/ProviderLimits/index.tsx": 1119,
|
||||
"src/app/api/providers/[id]/models/route.ts": 2361,
|
||||
"src/app/api/v1/models/catalog.ts": 1590,
|
||||
"src/app/api/v1/models/catalog.ts": 1597,
|
||||
"src/lib/db/apiKeys.ts": 1529,
|
||||
"src/lib/db/core.ts": 1639,
|
||||
"src/lib/db/migrationRunner.ts": 1094,
|
||||
@@ -536,7 +537,7 @@
|
||||
"src/app/(dashboard)/dashboard/usage/components/EvalsTab.tsx": "2148",
|
||||
"src/app/(dashboard)/dashboard/usage/components/ProviderLimits/index.tsx": "1119",
|
||||
"src/app/api/providers/[id]/models/route.ts": "2361",
|
||||
"src/app/api/v1/models/catalog.ts": "1590",
|
||||
"src/app/api/v1/models/catalog.ts": "1597",
|
||||
"src/lib/tokenHealthCheck.ts": "1053",
|
||||
"src/lib/db/apiKeys.ts": "1529",
|
||||
"src/lib/db/core.ts": "1639",
|
||||
|
||||
113
docs/video-preset-generation.md
Normal file
113
docs/video-preset-generation.md
Normal file
@@ -0,0 +1,113 @@
|
||||
# Video Generation Through Preset Jobs
|
||||
|
||||
Custom provider nodes whose `/videos` surface is an **async submit → poll → fetch-result API** (instead of a synchronous generation endpoint) can be wired into the `/api/v1/videos/generations` route without any new provider code. The model row carries a `generationConfig.preset`, and the dispatcher routes the request through a single job executor that is configured entirely by declarative preset data.
|
||||
|
||||
## How dispatch works
|
||||
|
||||
1. The route parses `model` as `provider/model` and resolves the provider node's credentials (`POST /api/v1/videos/generations`).
|
||||
2. `handleVideoGeneration` (in `open-sse/handlers/videoGeneration.ts`) checks whether the provider is a **custom provider node** (no entry in the static video registry).
|
||||
3. For custom nodes it reads the custom model row via `getCustomModelVideoPreset(provider, model)`:
|
||||
- The model row has `generationConfig.preset` set (e.g. `"agnes-video-job"`) → dispatch through the **job executor** (`open-sse/handlers/videoGeneration/job.ts`).
|
||||
- The preset name does not match any known preset → **502** `Unknown video job preset: <preset>` (server-side misconfiguration).
|
||||
- No preset configured → fall back to the generic OpenAI-compatible sync handler, mirroring the images route.
|
||||
4. The job executor runs the preset pipeline: **submit** the job, **poll** for terminal status, **read** the finished video URL, and return the standard OpenAI-compatible response shape.
|
||||
|
||||
The executor is one handler family; every provider-specific detail (paths, auth, body shape, status/result fields, poll cadence) is data in the preset definition.
|
||||
|
||||
## Response contract
|
||||
|
||||
Both the sync and job paths return the same shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"created": 1234567890,
|
||||
"data": [{ "url": "https://…", "format": "mp4" }]
|
||||
}
|
||||
```
|
||||
|
||||
This is the shape the media-generation consumer reads (`data.data[0].url`), so preset-job providers are drop-in replacements for sync providers.
|
||||
|
||||
## Presets
|
||||
|
||||
Presets live in `open-sse/handlers/videoGeneration/job.ts` (`VIDEO_JOB_PRESETS`). Each preset declares:
|
||||
|
||||
| Field | Meaning |
|
||||
| -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `authHeaderName` / `authScheme` | `x-api-key` with `raw` value (Agnes, muapi) or `Authorization` with `Bearer` prefix (Sora). Missing credentials → request goes out without an auth header. |
|
||||
| `baseUrlFallback` | Default base URL. Overridden by the provider connection's `providerSpecificData.baseUrl` (or top-level `baseUrl`), which wins when set. |
|
||||
| `submit.path` / `submit.buildBody` | Where and how the job is submitted. `{model}` in the path is substituted with the encoded model id; the body is built from `model`/`prompt`/`duration` plus pass-through of every other request field. |
|
||||
| `taskIdPath` | Dot path into the submit response identifying the job (e.g. `task_id`, `request_id`, `id`). Missing job id → **502**. |
|
||||
| `poll.pathTemplate` | Poll URL template; `{taskId}` is substituted. |
|
||||
| `statusPath` / `statusDone` / `statusFailed` | Where the job status lives and which values are terminal. |
|
||||
| `resultPath` | Dot path into the poll response holding the finished video URL: a string, a string array, or an array of `{ url }` objects are all accepted. Completed job with no readable URL → **502**. |
|
||||
| `maxPolls` / `pollIntervalMs` | Poll budget (default 60 polls × 2000 ms). Exhausted → **504** `Video job timed out`. |
|
||||
|
||||
### `agnes-video-job` — Agnes Video V2.0
|
||||
|
||||
- Auth: `x-api-key: <key>` (raw).
|
||||
- Base URL fallback: `https://apihub.agnes-ai.com`.
|
||||
- Submit: `POST /v1/videos` with `{ model, prompt, ...extras }` — image, mode, `num_frames`, `frame_rate` and other provider knobs pass through untouched.
|
||||
- Job id: `task_id` from the submit response.
|
||||
- Poll: `GET /v1/videos/{taskId}`; status at `status` (`completed` / `failed`).
|
||||
- Result: `metadata.url` — the completed video URL is returned as JSON metadata, not a binary body.
|
||||
|
||||
### `muapi-video-job` — muapi.ai
|
||||
|
||||
- Auth: `x-api-key: <key>` (raw).
|
||||
- Base URL fallback: `https://api.muapi.ai`.
|
||||
- Submit: `POST /api/v1/{model}` with `{ prompt, duration?, ...extras }`.
|
||||
- Job id: `request_id` from the submit response.
|
||||
- Poll: `GET /api/v1/predictions/{taskId}/result`; status at `status` (`completed` / `failed`).
|
||||
- Result: `outputs` — an array of video URLs.
|
||||
|
||||
### `sora-job` — OpenAI Sora
|
||||
|
||||
- Auth: `Authorization: Bearer <key>`.
|
||||
- Base URL fallback: `https://api.openai.com`.
|
||||
- Submit: `POST /v1/videos` with `{ model, prompt, seconds?, ...extras }`. `seconds` is a **string** enum (`"4" | "8" | "12"`) in the Sora API, so a numeric `duration` is stringified; size mapping is intentionally not forced.
|
||||
- Job id: `id` from the submit response.
|
||||
- Poll: `GET /v1/videos/{taskId}`; status at `status` (`completed` / `failed`).
|
||||
- Result: `data` — an array whose entries are either a URL string or `{ url: "…" }`.
|
||||
|
||||
## Setup
|
||||
|
||||
1. **Register the provider node** as an OpenAI-compatible custom provider (`providerSpecificData.baseUrl` optional — the preset's `baseUrlFallback` is used when absent).
|
||||
2. **Register a custom model** tagged with the `videos` endpoint and a `generationConfig`:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "super-video-v1",
|
||||
"name": "Super Video v1",
|
||||
"source": "manual",
|
||||
"apiFormat": "chat-completions",
|
||||
"supportedEndpoints": ["videos"],
|
||||
"generationConfig": { "preset": "agnes-video-job" }
|
||||
}
|
||||
```
|
||||
|
||||
`addCustomModel` (in `src/lib/db/models.ts`) accepts `generationConfig?: { preset: string }` as its final parameter and persists it on the model row; `updateCustomModel` forwards it the same way. The provider-models API accepts `generationConfig` on create and update.
|
||||
|
||||
3. **Call the route** as usual:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8787/api/v1/videos/generations \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer $API_KEY" \
|
||||
-d '{
|
||||
"model": "my-custom-provider/super-video-v1",
|
||||
"prompt": "a cat playing piano",
|
||||
"duration": 5
|
||||
}'
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Cause |
|
||||
| ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------ |
|
||||
| `400 Unknown video provider: …` | Non-custom provider not in the static registry; preset jobs only apply to custom provider nodes. |
|
||||
| `502 Unknown video job preset: …` | `generationConfig.preset` does not match any preset in `VIDEO_JOB_PRESETS`. Fix the model row. |
|
||||
| `502 Video provider did not return a job id (…)` | Submit succeeded but the response had no readable value at `taskIdPath`. |
|
||||
| `502 Video job failed (…)` / `Video job completed but no result URL found (…)` | Poll reached a terminal `statusFailed` state, or `resultPath` held no readable URL. |
|
||||
| `504 Video job timed out after 60 polls (…)` | Job never reached a terminal status within the poll budget. |
|
||||
| Upstream 4xx/5xx passthrough | `fetchJson` returns the upstream status when the submit/poll request itself is not OK. |
|
||||
| Requests go out without auth | No `apiKey`/`accessToken` on the provider connection; the executor sends `Content-Type` only. |
|
||||
@@ -12,6 +12,10 @@ import { FREEPIK_IMAGE_PROVIDER } from "./providers/registry/freepik/index.ts";
|
||||
import { STABILITY_AI_IMAGE_MODELS } from "./providers/registry/stability-ai/imageModels.ts";
|
||||
import { GEMINI_IMAGEN_PROVIDER } from "./providers/registry/gemini/imageModels.ts";
|
||||
import { CHEAPERINFERENCE_IMAGE_PROVIDER } from "./providers/registry/cheaperinference/imageModels.ts";
|
||||
import {
|
||||
ADOBE_FIREFLY_IMAGE_ROUTING_ALIASES,
|
||||
toRegistryImageModels,
|
||||
} from "../services/adobeFireflyModels.ts";
|
||||
|
||||
interface ImageModelEntry {
|
||||
id: string;
|
||||
@@ -22,6 +26,8 @@ interface ImageModelEntry {
|
||||
imageRequired?: boolean;
|
||||
description?: string;
|
||||
isMarket?: boolean;
|
||||
supportedSizes?: string[];
|
||||
mediaCapabilities?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
interface ImageProviderConfig {
|
||||
@@ -35,6 +41,7 @@ interface ImageProviderConfig {
|
||||
authHeader: string;
|
||||
format: string;
|
||||
models: ImageModelEntry[];
|
||||
routingAliases?: readonly string[];
|
||||
supportedSizes: string[];
|
||||
}
|
||||
|
||||
@@ -46,6 +53,7 @@ interface ImageModelAliasEntry {
|
||||
inputModalities?: string[];
|
||||
imageRequired?: boolean;
|
||||
description?: string;
|
||||
mediaCapabilities?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
interface ImageCatalogModelEntry {
|
||||
@@ -55,6 +63,7 @@ interface ImageCatalogModelEntry {
|
||||
supportedSizes: string[];
|
||||
inputModalities: string[];
|
||||
description?: string;
|
||||
mediaCapabilities?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
const IMAGE_MODEL_ALIASES: Record<string, ImageModelAliasEntry> = {
|
||||
@@ -678,55 +687,9 @@ export const IMAGE_PROVIDERS: Record<string, ImageProviderConfig> = {
|
||||
authType: "apikey",
|
||||
authHeader: "bearer",
|
||||
format: "adobe-firefly-image",
|
||||
models: [
|
||||
{
|
||||
id: "nano-banana-pro",
|
||||
name: "Firefly Gemini 3.0 (Nano Banana Pro)",
|
||||
inputModalities: ["text", "image"],
|
||||
},
|
||||
{
|
||||
id: "nano-banana",
|
||||
name: "Firefly Gemini 2.5 (Nano Banana)",
|
||||
inputModalities: ["text", "image"],
|
||||
},
|
||||
{
|
||||
id: "nano-banana-2",
|
||||
name: "Firefly Gemini 3.1 (Nano Banana 2)",
|
||||
inputModalities: ["text", "image"],
|
||||
},
|
||||
{ id: "gpt-image-2", name: "Firefly GPT Image 2", inputModalities: ["text", "image"] },
|
||||
{ id: "gpt-image", name: "Firefly GPT Image 2", inputModalities: ["text", "image"] },
|
||||
{ id: "gpt-image-1.5", name: "Firefly GPT Image 1.5", inputModalities: ["text", "image"] },
|
||||
{ id: "flux-2", name: "Firefly Flux 2", inputModalities: ["text", "image"] },
|
||||
{ id: "flux-pro", name: "Firefly Flux 1.1 Pro", inputModalities: ["text", "image"] },
|
||||
{ id: "flux-ultra", name: "Firefly Flux 1.1 Ultra", inputModalities: ["text", "image"] },
|
||||
{ id: "seedream-4", name: "Firefly Seedream 4.0", inputModalities: ["text", "image"] },
|
||||
{
|
||||
id: "seedream-5-lite",
|
||||
name: "Firefly Seedream 5.0 Lite",
|
||||
inputModalities: ["text", "image"],
|
||||
},
|
||||
{
|
||||
id: "runway-gen4-image",
|
||||
name: "Firefly Runway Gen-4 Image",
|
||||
inputModalities: ["text", "image"],
|
||||
},
|
||||
// Topaz Labs upscalers (inputMediaUseCase: ["upscaling"]).
|
||||
// Served by firefly-3p /v2/3p-images/upsample — see config/upscaleRegistry.ts.
|
||||
{
|
||||
id: "topaz-standard",
|
||||
name: "Firefly Topaz Upscale (Standard)",
|
||||
inputModalities: ["image"],
|
||||
imageRequired: true,
|
||||
},
|
||||
{
|
||||
id: "topaz-bloom",
|
||||
name: "Firefly Topaz Bloom (Creative Upscale)",
|
||||
inputModalities: ["image"],
|
||||
imageRequired: true,
|
||||
},
|
||||
],
|
||||
supportedSizes: ["1:1", "16:9", "9:16", "4:3", "3:4", "1024x1024", "1792x1024", "1024x1792"],
|
||||
models: toRegistryImageModels(),
|
||||
routingAliases: ADOBE_FIREFLY_IMAGE_ROUTING_ALIASES,
|
||||
supportedSizes: [],
|
||||
},
|
||||
|
||||
// Cheaper Inference (OSS-sponsor gateway). Declared AFTER adobe-firefly on
|
||||
@@ -887,7 +850,7 @@ export function parseImageModel(modelStr) {
|
||||
|
||||
// No provider prefix — try to find the model in every provider
|
||||
for (const [providerId, config] of Object.entries(IMAGE_PROVIDERS)) {
|
||||
if (config.models.some((m) => m.id === modelStr)) {
|
||||
if (config.routingAliases?.includes(modelStr) || config.models.some((m) => m.id === modelStr)) {
|
||||
return { provider: providerId, model: modelStr };
|
||||
}
|
||||
}
|
||||
@@ -906,9 +869,10 @@ function imageProviderCatalogEntries(
|
||||
id: `${providerId}/${model.id}`,
|
||||
name: model.name,
|
||||
provider: providerId,
|
||||
supportedSizes: config.supportedSizes,
|
||||
supportedSizes: model.supportedSizes || config.supportedSizes,
|
||||
inputModalities: model.inputModalities || ["text"],
|
||||
description: model.description || undefined,
|
||||
mediaCapabilities: model.mediaCapabilities,
|
||||
}));
|
||||
}
|
||||
|
||||
|
||||
@@ -225,7 +225,6 @@ import { digitaloceanProvider } from "./registry/digitalocean/index.ts";
|
||||
import { hcnsecProvider } from "./registry/hcnsec/index.ts";
|
||||
import { promptqlProvider } from "./registry/promptql/index.ts";
|
||||
import { hyperagentProvider } from "./registry/hyperagent/index.ts";
|
||||
import { muse_codeProvider } from "./registry/muse-code/index.ts";
|
||||
|
||||
export const REGISTRY: Record<string, RegistryEntry> = {
|
||||
aimlapi: aimlapiProvider,
|
||||
@@ -452,6 +451,5 @@ export const REGISTRY: Record<string, RegistryEntry> = {
|
||||
hcnsec: hcnsecProvider,
|
||||
promptql: promptqlProvider,
|
||||
hyperagent: hyperagentProvider,
|
||||
"muse-code": muse_codeProvider,
|
||||
unorouter: unorouterProvider,
|
||||
};
|
||||
|
||||
@@ -1,106 +0,0 @@
|
||||
import type { RegistryEntry } from "../../shared.ts";
|
||||
import { buildOpenAiCompatibleRegistryEntry } from "../../shared.ts";
|
||||
|
||||
/**
|
||||
* Muse Code CLI — Meta's agentic coding tool.
|
||||
*
|
||||
* Wire format: OpenAI Responses API (POST /responses).
|
||||
* Auth: Bearer token from META_API_KEY env var.
|
||||
* Reasoning efforts: xhigh/ultra -> high (handled generically).
|
||||
*
|
||||
* @see https://github.com/joymadhu49/muse-openrouter-shim
|
||||
*/
|
||||
export const muse_codeProvider: RegistryEntry = buildOpenAiCompatibleRegistryEntry({
|
||||
id: "muse-code",
|
||||
alias: "mc",
|
||||
passthroughModels: true,
|
||||
defaultContextLength: 200000,
|
||||
models: [
|
||||
{
|
||||
id: "llama-4-maverick",
|
||||
name: "Llama 4 Maverick",
|
||||
contextLength: 1048576,
|
||||
maxOutputTokens: 131072,
|
||||
supportsReasoning: true,
|
||||
supportsXHighEffort: true,
|
||||
toolCalling: true,
|
||||
supportsVision: true,
|
||||
targetFormat: "openai-responses",
|
||||
unsupportedParams: ["logprobs", "topLogprobs", "logitBias"],
|
||||
},
|
||||
{
|
||||
id: "llama-4-scout",
|
||||
name: "Llama 4 Scout",
|
||||
contextLength: 1048576,
|
||||
maxOutputTokens: 131072,
|
||||
supportsReasoning: true,
|
||||
supportsXHighEffort: true,
|
||||
toolCalling: true,
|
||||
supportsVision: true,
|
||||
targetFormat: "openai-responses",
|
||||
unsupportedParams: ["logprobs", "topLogprobs", "logitBias"],
|
||||
},
|
||||
{
|
||||
id: "llama-3.3-70b",
|
||||
name: "Llama 3.3 70B",
|
||||
contextLength: 131072,
|
||||
maxOutputTokens: 32768,
|
||||
supportsReasoning: false,
|
||||
toolCalling: true,
|
||||
targetFormat: "openai-responses",
|
||||
unsupportedParams: ["logprobs", "topLogprobs"],
|
||||
},
|
||||
{
|
||||
id: "llama-3.1-405b",
|
||||
name: "Llama 3.1 405B",
|
||||
contextLength: 131072,
|
||||
maxOutputTokens: 32768,
|
||||
supportsReasoning: false,
|
||||
toolCalling: true,
|
||||
targetFormat: "openai-responses",
|
||||
unsupportedParams: ["logprobs", "topLogprobs"],
|
||||
},
|
||||
{
|
||||
id: "llama-3.1-70b",
|
||||
name: "Llama 3.1 70B",
|
||||
contextLength: 131072,
|
||||
maxOutputTokens: 32768,
|
||||
supportsReasoning: false,
|
||||
toolCalling: true,
|
||||
targetFormat: "openai-responses",
|
||||
unsupportedParams: ["logprobs", "topLogprobs"],
|
||||
},
|
||||
{
|
||||
id: "llama-3.1-8b",
|
||||
name: "Llama 3.1 8B",
|
||||
contextLength: 131072,
|
||||
maxOutputTokens: 32768,
|
||||
supportsReasoning: false,
|
||||
toolCalling: true,
|
||||
targetFormat: "openai-responses",
|
||||
unsupportedParams: ["logprobs", "topLogprobs"],
|
||||
},
|
||||
{
|
||||
id: "llama-3.2-90b-vision",
|
||||
name: "Llama 3.2 90B Vision",
|
||||
contextLength: 131072,
|
||||
maxOutputTokens: 32768,
|
||||
supportsReasoning: false,
|
||||
toolCalling: true,
|
||||
supportsVision: true,
|
||||
targetFormat: "openai-responses",
|
||||
unsupportedParams: ["logprobs", "topLogprobs"],
|
||||
},
|
||||
{
|
||||
id: "llama-3.2-11b-vision",
|
||||
name: "Llama 3.2 11B Vision",
|
||||
contextLength: 131072,
|
||||
maxOutputTokens: 32768,
|
||||
supportsReasoning: false,
|
||||
toolCalling: true,
|
||||
supportsVision: true,
|
||||
targetFormat: "openai-responses",
|
||||
unsupportedParams: ["logprobs", "topLogprobs"],
|
||||
},
|
||||
],
|
||||
});
|
||||
@@ -5,14 +5,17 @@
|
||||
* Supports local providers plus hosted task-based APIs such as Runway.
|
||||
*/
|
||||
|
||||
import { parseModelFromRegistry, getAllModelsFromRegistry } from "./registryUtils.ts";
|
||||
import { parseModelFromRegistry } from "./registryUtils.ts";
|
||||
import { RUNWAYML_SUPPORTED_VIDEO_MODELS } from "./runway.ts";
|
||||
import { SEGMIND_VIDEO_MODELS } from "./providers/registry/segmind/videoModels.ts";
|
||||
import { toRegistryVideoModels } from "../services/adobeFireflyModels.ts";
|
||||
|
||||
interface VideoModel {
|
||||
id: string;
|
||||
name: string;
|
||||
isMarket?: boolean;
|
||||
supportedSizes?: string[];
|
||||
mediaCapabilities?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
interface VideoProvider {
|
||||
@@ -326,8 +329,7 @@ export const VIDEO_PROVIDERS: Record<string, VideoProvider> = {
|
||||
},
|
||||
|
||||
// Adobe Firefly (unofficial) — same IMS/cookie credential as the image entry.
|
||||
// Async 3P video generate + poll (Sora 2, Veo 3.1, Kling …). Fallback list
|
||||
// from models/discovery capture (adobe/get_models.txt).
|
||||
// Exact async video models and capabilities from the verified discovery snapshot.
|
||||
"adobe-firefly": {
|
||||
id: "adobe-firefly",
|
||||
alias: "firefly",
|
||||
@@ -335,18 +337,7 @@ export const VIDEO_PROVIDERS: Record<string, VideoProvider> = {
|
||||
authType: "apikey",
|
||||
authHeader: "bearer",
|
||||
format: "adobe-firefly-video",
|
||||
models: [
|
||||
{ id: "sora-2", name: "Firefly Sora 2" },
|
||||
{ id: "sora-2-pro", name: "Firefly Sora 2 Pro" },
|
||||
{ id: "veo-3.1", name: "Firefly Veo 3.1" },
|
||||
{ id: "veo-3.1-fast", name: "Firefly Veo 3.1 Fast" },
|
||||
{ id: "veo-3.1-ref", name: "Firefly Veo 3.1 Reference" },
|
||||
{ id: "kling-3", name: "Firefly Kling v3 Standard I2V" },
|
||||
{ id: "kling-v3-t2v", name: "Firefly Kling v3 Standard T2V" },
|
||||
{ id: "kling-v3-pro-i2v", name: "Firefly Kling v3 Pro I2V" },
|
||||
{ id: "luma-ray3", name: "Firefly Ray3" },
|
||||
{ id: "runway-gen4-turbo", name: "Firefly Runway Gen-4 Video" },
|
||||
],
|
||||
models: toRegistryVideoModels(),
|
||||
},
|
||||
};
|
||||
|
||||
@@ -368,5 +359,17 @@ export function parseVideoModel(modelStr: string | null) {
|
||||
* Get all video models as a flat list
|
||||
*/
|
||||
export function getAllVideoModels() {
|
||||
return getAllModelsFromRegistry(VIDEO_PROVIDERS);
|
||||
return Object.entries(VIDEO_PROVIDERS).flatMap(([providerId, config]) =>
|
||||
[providerId, config.alias]
|
||||
.filter((prefix): prefix is string => Boolean(prefix))
|
||||
.flatMap((prefix) =>
|
||||
config.models.map((model) => ({
|
||||
id: `${prefix}/${model.id}`,
|
||||
name: model.name,
|
||||
provider: providerId,
|
||||
supportedSizes: model.supportedSizes || [],
|
||||
mediaCapabilities: model.mediaCapabilities,
|
||||
}))
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
@@ -16,11 +16,11 @@ import {
|
||||
AdobeFireflyError,
|
||||
adobeFireflyGenerateImage,
|
||||
adobeFireflyImageTimeoutMs,
|
||||
adobeFireflyMaxImageRefs,
|
||||
resolveAdobeAccessToken,
|
||||
resolveAdobeSourceImageIds,
|
||||
resolveAdobeSourceImageReferences,
|
||||
resolveAdobeImageModel,
|
||||
} from "../../../services/adobeFireflyClient.ts";
|
||||
import { getAdobeReferenceUploadLimit } from "../../../services/adobeFireflyModels.ts";
|
||||
import { isAdobeFireflyUpscaleModel } from "../../../services/adobeFireflyUpscale.ts";
|
||||
import { handleAdobeFireflyImageUpscale } from "../../imageUpscale/adobeFirefly.ts";
|
||||
|
||||
@@ -90,7 +90,8 @@ export async function handleAdobeFireflyImageGeneration({
|
||||
|
||||
// Keep the raw credential blob for Cookie + sherlockToken (x-arp-session-id).
|
||||
// JWT may be embedded in the same paste as cookies (HAR / multi-line).
|
||||
const psd = (credentials as { providerSpecificData?: { cookie?: string } })?.providerSpecificData;
|
||||
const psd = (credentials as { providerSpecificData?: { cookie?: string } })
|
||||
?.providerSpecificData;
|
||||
const sessionCookie =
|
||||
(typeof psd?.cookie === "string" && psd.cookie.trim()) ||
|
||||
(typeof credentials?.apiKey === "string" && credentials.apiKey.trim()) ||
|
||||
@@ -98,15 +99,11 @@ export async function handleAdobeFireflyImageGeneration({
|
||||
? credentials.accessToken
|
||||
: undefined);
|
||||
|
||||
// Cap uploads by model family. gpt-image: 2 subject refs max (3–4+ stalls colligo → 504).
|
||||
// nano: 4 general refs for multi-panel composition.
|
||||
const { id: resolvedId } = resolveAdobeImageModel(model);
|
||||
const maxRefs = adobeFireflyMaxImageRefs(resolvedId);
|
||||
|
||||
const sourceImageIds = await resolveAdobeSourceImageIds({
|
||||
const { spec } = resolveAdobeImageModel(model);
|
||||
const references = await resolveAdobeSourceImageReferences({
|
||||
accessToken,
|
||||
body,
|
||||
max: maxRefs,
|
||||
max: getAdobeReferenceUploadLimit(spec, "image"),
|
||||
sessionCookie,
|
||||
prompt,
|
||||
fetchImpl,
|
||||
@@ -121,13 +118,13 @@ export async function handleAdobeFireflyImageGeneration({
|
||||
: undefined;
|
||||
const timeoutMs = adobeFireflyImageTimeoutMs({
|
||||
timeoutMs: explicitTimeout,
|
||||
refCount: sourceImageIds.length,
|
||||
refCount: references.length,
|
||||
});
|
||||
|
||||
log?.info?.(
|
||||
"IMAGE",
|
||||
`${provider}/${model} (adobe-firefly) | prompt: "${prompt.slice(0, 60)}${prompt.length > 60 ? "..." : ""}"` +
|
||||
(sourceImageIds.length ? ` | refs: ${sourceImageIds.length}/${maxRefs}` : "") +
|
||||
(references.length ? ` | refs: ${references.length}` : "") +
|
||||
` | pollTimeoutMs=${timeoutMs}`
|
||||
);
|
||||
|
||||
@@ -139,9 +136,8 @@ export async function handleAdobeFireflyImageGeneration({
|
||||
aspectRatio: body.aspect_ratio ?? body.aspectRatio ?? body.size,
|
||||
quality: body.quality,
|
||||
seed: Number.isFinite(seed as number) ? (seed as number) : undefined,
|
||||
negativePrompt:
|
||||
typeof body.negative_prompt === "string" ? body.negative_prompt : undefined,
|
||||
sourceImageIds: sourceImageIds.length ? sourceImageIds : undefined,
|
||||
negativePrompt: typeof body.negative_prompt === "string" ? body.negative_prompt : undefined,
|
||||
references: references.length ? references : undefined,
|
||||
sessionCookie,
|
||||
timeoutMs,
|
||||
fetchImpl,
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
* Handles POST /v1/videos/generations requests. Proxies to upstream video
|
||||
* generation providers (ComfyUI AnimateDiff/SVD, SD WebUI AnimateDiff, and
|
||||
* more — see the per-format handlers below). Response format (OpenAI-like):
|
||||
* { "created": 1234567890, "data": [{ "b64_json": "...", "format": "mp4" }] }
|
||||
* { "created": 1234567890, "data": [{ "url": "https://…", "format": "mp4" }] }
|
||||
*/
|
||||
|
||||
import { getVideoProvider, parseVideoModel } from "../config/videoRegistry.ts";
|
||||
@@ -18,6 +18,16 @@ import { handleNovitaVideoGeneration } from "./videoGeneration/novitaHandler.ts"
|
||||
import { handleXaiVideoGeneration } from "./videoGeneration/xaiGrokImagineHandler.ts";
|
||||
import { handleSegmindVideoGeneration } from "./videoGeneration/providers/segmind.ts";
|
||||
import { handleAdobeFireflyVideoGeneration } from "./videoGeneration/adobeFireflyHandler.ts";
|
||||
import { handleOpenAIVideoGeneration } from "./videoGeneration/openai.ts";
|
||||
import { getVideoJobPreset, handleVideoJobGeneration } from "./videoGeneration/job.ts";
|
||||
import {
|
||||
extractRunwayFailureMessage,
|
||||
normalizeRunwayVideoResult,
|
||||
resolvePositiveInteger,
|
||||
resolveRunwayDuration,
|
||||
resolveRunwayPromptImage,
|
||||
resolveRunwayRatio,
|
||||
} from "./videoGeneration/runwayHelpers.ts";
|
||||
import { getExecutor } from "../executors/index.ts";
|
||||
import { getKieTaskId, isJsonObject, parseKieResultJson } from "../utils/kieTask.ts";
|
||||
import {
|
||||
@@ -33,13 +43,94 @@ import {
|
||||
resolveComfyUiBaseUrl,
|
||||
} from "../utils/comfyuiClient.ts";
|
||||
import { saveCallLog } from "@/lib/usageDb";
|
||||
import { getAllCustomModels } from "@/lib/db/models";
|
||||
import { sanitizeErrorMessage } from "../utils/error.ts";
|
||||
import {
|
||||
FetchTimeoutError,
|
||||
fetchWithTimeout,
|
||||
getConfiguredTimeout,
|
||||
} from "@/shared/utils/fetchTimeout";
|
||||
|
||||
/**
|
||||
* Resolve the base URL for OpenAI-compatible video generation endpoints.
|
||||
* Prefers providerSpecificData.baseUrl (from custom node config), falls back to
|
||||
* top-level credentials.baseUrl, then to the provided fallback.
|
||||
*/
|
||||
export function resolveVideoBaseUrl(
|
||||
credentials:
|
||||
{ baseUrl?: unknown; providerSpecificData?: { baseUrl?: unknown } | null } | null | undefined,
|
||||
fallback: string
|
||||
): string {
|
||||
const psd = credentials?.providerSpecificData;
|
||||
const psdBaseUrl =
|
||||
psd && typeof psd === "object" && typeof psd.baseUrl === "string" && psd.baseUrl.trim()
|
||||
? psd.baseUrl.trim()
|
||||
: null;
|
||||
const topLevelBaseUrl =
|
||||
typeof credentials?.baseUrl === "string" && credentials.baseUrl.trim()
|
||||
? credentials.baseUrl.trim()
|
||||
: null;
|
||||
const nodeBaseUrl = psdBaseUrl || topLevelBaseUrl;
|
||||
|
||||
if (!nodeBaseUrl) return fallback;
|
||||
|
||||
// Trim trailing slashes
|
||||
let normalized = nodeBaseUrl;
|
||||
while (normalized.endsWith("/")) normalized = normalized.slice(0, -1);
|
||||
if (normalized.endsWith("/videos/generations")) return normalized;
|
||||
const stripped = normalized.replace(/\/videos\/generations$/, "");
|
||||
return `${stripped}/videos/generations`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Read generationConfig.preset from the custom model row for the given
|
||||
* provider/model id. Returns null when the model has no preset configured (or
|
||||
* the registry is unreadable), so callers can fall back to the sync path.
|
||||
*/
|
||||
async function getCustomModelVideoPreset(
|
||||
providerId: string,
|
||||
modelId: string
|
||||
): Promise<string | null> {
|
||||
try {
|
||||
const customModelsMap = (await getAllCustomModels()) as Record<
|
||||
string,
|
||||
Array<Record<string, unknown>>
|
||||
>;
|
||||
const models = customModelsMap[providerId];
|
||||
if (!Array.isArray(models)) return null;
|
||||
for (const model of models) {
|
||||
if (!model || typeof model !== "object" || model.id !== modelId) continue;
|
||||
const generationConfig = model.generationConfig;
|
||||
if (
|
||||
generationConfig &&
|
||||
typeof generationConfig === "object" &&
|
||||
typeof (generationConfig as Record<string, unknown>).preset === "string"
|
||||
) {
|
||||
return (generationConfig as Record<string, unknown>).preset as string;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
return null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Handle video generation request
|
||||
*/
|
||||
export async function handleVideoGeneration({ body, credentials, log }) {
|
||||
const { provider, model } = parseVideoModel(body.model);
|
||||
|
||||
/**
|
||||
* Handle video generation request
|
||||
*/
|
||||
export async function handleVideoGeneration({ body, credentials, log, resolvedProvider = null }) {
|
||||
let { provider, model } = parseVideoModel(body.model);
|
||||
if (resolvedProvider) {
|
||||
provider = resolvedProvider;
|
||||
model = body.model.startsWith(provider + "/")
|
||||
? body.model.slice(provider.length + 1)
|
||||
: body.model;
|
||||
}
|
||||
|
||||
if (!provider) {
|
||||
return {
|
||||
@@ -51,11 +142,59 @@ export async function handleVideoGeneration({ body, credentials, log }) {
|
||||
|
||||
const providerConfig = getVideoProvider(provider);
|
||||
if (!providerConfig) {
|
||||
return {
|
||||
success: false,
|
||||
status: 400,
|
||||
error: `Unknown video provider: ${provider}`,
|
||||
if (!resolvedProvider) {
|
||||
return {
|
||||
success: false,
|
||||
status: 400,
|
||||
error: `Unknown video provider: ${provider}`,
|
||||
};
|
||||
}
|
||||
// Custom provider node. When the custom model row carries a
|
||||
// generationConfig.preset (e.g. "agnes-video-job"), dispatch through the
|
||||
// submit → poll job pipeline; otherwise mirror the images route and use the
|
||||
// generic OpenAI-compatible handler with a synthetic config.
|
||||
const presetName = await getCustomModelVideoPreset(provider, model);
|
||||
if (presetName !== null) {
|
||||
if (!getVideoJobPreset(presetName)) {
|
||||
return {
|
||||
success: false,
|
||||
status: 502,
|
||||
error: `Unknown video job preset: ${presetName}`,
|
||||
};
|
||||
}
|
||||
if (log)
|
||||
log.info("VIDEO", `Custom model ${provider}/${model} — using job preset ${presetName}`);
|
||||
return handleVideoJobGeneration({
|
||||
model,
|
||||
presetName,
|
||||
body,
|
||||
credentials,
|
||||
log,
|
||||
});
|
||||
}
|
||||
if (log)
|
||||
log.info("VIDEO", `Custom model ${provider}/${model} — using OpenAI-compatible handler`);
|
||||
const syntheticConfig = {
|
||||
id: provider,
|
||||
baseUrl: resolveVideoBaseUrl(
|
||||
credentials,
|
||||
"http://generative.language.googleapis.com/v1beta/openai/videos/generations"
|
||||
),
|
||||
authType: "apikey",
|
||||
authHeader: "bearer",
|
||||
format: "openai-video",
|
||||
};
|
||||
return handleOpenAIVideoGeneration({
|
||||
model,
|
||||
body,
|
||||
credentials,
|
||||
provider,
|
||||
providerConfig: syntheticConfig,
|
||||
log,
|
||||
});
|
||||
}
|
||||
if (providerConfig.format === "openai-video") {
|
||||
return handleOpenAIVideoGeneration({ model, provider, providerConfig, body, credentials, log });
|
||||
}
|
||||
|
||||
if (providerConfig.format === "vertex-veo") {
|
||||
@@ -158,7 +297,10 @@ export async function handleVideoGeneration({ body, credentials, log }) {
|
||||
log,
|
||||
});
|
||||
}
|
||||
|
||||
if (resolvedProvider) {
|
||||
// Custom provider with no matching built-in format — use OpenAI-compatible fallback
|
||||
return handleOpenAIVideoGeneration({ model, provider, providerConfig, body, credentials, log });
|
||||
}
|
||||
return {
|
||||
success: false,
|
||||
status: 400,
|
||||
@@ -832,148 +974,6 @@ const RUNWAY_TERMINAL_FAILURE_STATUSES = new Set([
|
||||
"DELETED",
|
||||
]);
|
||||
|
||||
function resolveRunwayPromptImage(body) {
|
||||
const directCandidates = [
|
||||
body.promptImage,
|
||||
body.prompt_image,
|
||||
body.image,
|
||||
body.image_url,
|
||||
body.imageUrl,
|
||||
body.provider_options?.promptImage,
|
||||
body.provider_options?.prompt_image,
|
||||
];
|
||||
|
||||
for (const candidate of directCandidates) {
|
||||
if (typeof candidate === "string" && candidate.trim()) return candidate.trim();
|
||||
if (candidate && typeof candidate === "object") return candidate;
|
||||
if (Array.isArray(candidate) && candidate.length > 0) return candidate;
|
||||
}
|
||||
|
||||
const arrayCandidates = [
|
||||
body.imageUrls,
|
||||
body.image_urls,
|
||||
body.provider_options?.imageUrls,
|
||||
body.provider_options?.image_urls,
|
||||
];
|
||||
for (const candidate of arrayCandidates) {
|
||||
if (Array.isArray(candidate) && candidate.length > 0) return candidate;
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
function resolveRunwayRatio(body) {
|
||||
const aspectRatio = typeof body.aspect_ratio === "string" ? body.aspect_ratio : body.aspectRatio;
|
||||
if (aspectRatio === "1280:720" || aspectRatio === "720:1280") return aspectRatio;
|
||||
if (aspectRatio === "16:9") return "1280:720";
|
||||
if (aspectRatio === "9:16") return "720:1280";
|
||||
|
||||
const size = typeof body.size === "string" ? body.size : "";
|
||||
const [widthRaw, heightRaw] = size.split("x");
|
||||
const width = Number(widthRaw);
|
||||
const height = Number(heightRaw);
|
||||
if (Number.isFinite(width) && Number.isFinite(height) && width > 0 && height > 0) {
|
||||
return width >= height ? "1280:720" : "720:1280";
|
||||
}
|
||||
|
||||
return "1280:720";
|
||||
}
|
||||
|
||||
function resolveRunwayDuration(body) {
|
||||
if (Number.isFinite(body.duration)) {
|
||||
return clampRunwayDuration(body.duration);
|
||||
}
|
||||
|
||||
if (Number.isFinite(body.frames) && Number.isFinite(body.fps) && Number(body.fps) > 0) {
|
||||
return clampRunwayDuration(Number(body.frames) / Number(body.fps));
|
||||
}
|
||||
|
||||
return 5;
|
||||
}
|
||||
|
||||
function clampRunwayDuration(value) {
|
||||
const duration = Math.round(Number(value));
|
||||
if (!Number.isFinite(duration)) return 5;
|
||||
return Math.min(10, Math.max(2, duration));
|
||||
}
|
||||
|
||||
function resolvePositiveInteger(value, fallback) {
|
||||
const numeric = Number(value);
|
||||
if (!Number.isFinite(numeric) || numeric <= 0) return fallback;
|
||||
return Math.floor(numeric);
|
||||
}
|
||||
|
||||
function extractRunwayOutputUrls(task) {
|
||||
const rawOutput = Array.isArray(task?.output)
|
||||
? task.output
|
||||
: Array.isArray(task?.result)
|
||||
? task.result
|
||||
: [];
|
||||
|
||||
return rawOutput
|
||||
.map((entry) => {
|
||||
if (typeof entry === "string") return entry;
|
||||
if (!entry || typeof entry !== "object") return null;
|
||||
return entry.url || entry.uri || entry.videoUrl || entry.video_url || null;
|
||||
})
|
||||
.filter((value) => typeof value === "string" && value.length > 0);
|
||||
}
|
||||
|
||||
function extractRunwayFailureMessage(task) {
|
||||
const directCandidates = [
|
||||
task?.failure,
|
||||
task?.failureReason,
|
||||
task?.error,
|
||||
task?.errorMessage,
|
||||
task?.message,
|
||||
];
|
||||
for (const candidate of directCandidates) {
|
||||
if (typeof candidate === "string" && candidate.trim()) return candidate.trim();
|
||||
}
|
||||
|
||||
if (task?.failure && typeof task.failure === "object") {
|
||||
const nestedCandidates = [
|
||||
task.failure.message,
|
||||
task.failure.reason,
|
||||
task.failure.error,
|
||||
task.failure.code,
|
||||
];
|
||||
for (const candidate of nestedCandidates) {
|
||||
if (typeof candidate === "string" && candidate.trim()) return candidate.trim();
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
async function normalizeRunwayVideoResult(task, body) {
|
||||
const urls = extractRunwayOutputUrls(task);
|
||||
if (urls.length === 0) {
|
||||
throw new Error(
|
||||
`Runway task completed without output URLs: ${JSON.stringify(task).slice(0, 400)}`
|
||||
);
|
||||
}
|
||||
|
||||
if (body.response_format === "url") {
|
||||
return urls.map((url) => ({ url, format: "mp4" }));
|
||||
}
|
||||
|
||||
const videos = [];
|
||||
for (const url of urls) {
|
||||
const response = await fetch(url);
|
||||
if (!response.ok) {
|
||||
throw new Error(`Runway output fetch failed (${response.status})`);
|
||||
}
|
||||
const arrayBuffer = await response.arrayBuffer();
|
||||
videos.push({
|
||||
b64_json: Buffer.from(arrayBuffer).toString("base64"),
|
||||
format: "mp4",
|
||||
});
|
||||
}
|
||||
|
||||
return videos;
|
||||
}
|
||||
|
||||
async function handleHaiperVideoGeneration({
|
||||
model,
|
||||
provider,
|
||||
|
||||
@@ -10,9 +10,10 @@ import {
|
||||
AdobeFireflyError,
|
||||
adobeFireflyGenerateVideo,
|
||||
resolveAdobeAccessToken,
|
||||
resolveAdobeSourceImageIds,
|
||||
resolveAdobeSourceImageReferences,
|
||||
resolveAdobeVideoModel,
|
||||
} from "../../services/adobeFireflyClient.ts";
|
||||
import { getAdobeReferenceUploadLimit } from "../../services/adobeFireflyModels.ts";
|
||||
|
||||
function normalizePositiveNumber(value: unknown, fallback: number): number {
|
||||
const n = Number(value);
|
||||
@@ -55,7 +56,8 @@ export async function handleAdobeFireflyVideoGeneration({
|
||||
? Number(body.seed)
|
||||
: undefined;
|
||||
// Keep raw paste for Cookie + sherlockToken (x-arp-session-id).
|
||||
const psd = (credentials as { providerSpecificData?: { cookie?: string } })?.providerSpecificData;
|
||||
const psd = (credentials as { providerSpecificData?: { cookie?: string } })
|
||||
?.providerSpecificData;
|
||||
const sessionCookie =
|
||||
(typeof psd?.cookie === "string" && psd.cookie.trim()) ||
|
||||
(typeof credentials?.apiKey === "string" && credentials.apiKey.trim()) ||
|
||||
@@ -63,13 +65,11 @@ export async function handleAdobeFireflyVideoGeneration({
|
||||
? credentials.accessToken
|
||||
: undefined);
|
||||
|
||||
// Kling i2v / Veo ref / Sora frame: upload reference images first.
|
||||
const { id: videoModelId } = resolveAdobeVideoModel(String(model));
|
||||
const maxFrames = videoModelId.includes("kling") || videoModelId.includes("sora") ? 2 : 3;
|
||||
const sourceImageIds = await resolveAdobeSourceImageIds({
|
||||
const { spec } = resolveAdobeVideoModel(String(model));
|
||||
const references = await resolveAdobeSourceImageReferences({
|
||||
accessToken,
|
||||
body,
|
||||
max: maxFrames,
|
||||
max: getAdobeReferenceUploadLimit(spec, "image"),
|
||||
sessionCookie,
|
||||
prompt,
|
||||
fetchImpl,
|
||||
@@ -79,7 +79,7 @@ export async function handleAdobeFireflyVideoGeneration({
|
||||
log?.info?.(
|
||||
"VIDEO",
|
||||
`${provider}/${model} (adobe-firefly) | prompt: "${prompt.slice(0, 60)}${prompt.length > 60 ? "..." : ""}"` +
|
||||
(sourceImageIds.length ? ` | frames: ${sourceImageIds.length}` : "")
|
||||
(references.length ? ` | refs: ${references.length}` : "")
|
||||
);
|
||||
|
||||
const result = await adobeFireflyGenerateVideo({
|
||||
@@ -99,7 +99,7 @@ export async function handleAdobeFireflyVideoGeneration({
|
||||
? body.negativePrompt
|
||||
: undefined,
|
||||
generateAudio: body.generate_audio !== false && body.generateAudio !== false,
|
||||
sourceImageIds: sourceImageIds.length ? sourceImageIds : undefined,
|
||||
references: references.length ? references : undefined,
|
||||
sessionCookie,
|
||||
timeoutMs,
|
||||
fetchImpl,
|
||||
|
||||
418
open-sse/handlers/videoGeneration/job.ts
Normal file
418
open-sse/handlers/videoGeneration/job.ts
Normal file
@@ -0,0 +1,418 @@
|
||||
/**
|
||||
* Async job/poll video generation for custom OpenAI-compatible provider nodes
|
||||
* whose /videos surface is a submit → poll → fetch-result API (e.g. Agnes
|
||||
* Video V2.0, muapi.ai, OpenAI Sora). Presets are declarative data — the
|
||||
* handler here is one family; everything else is per-preset config.
|
||||
*
|
||||
* Response shape stays OpenAI-like: { created, data: [{ url, format: "mp4" }] } so the
|
||||
* /v1/videos/generations route returns the same contract as the synchronous
|
||||
* path.
|
||||
*/
|
||||
|
||||
import {
|
||||
fetchWithTimeout,
|
||||
FetchTimeoutError,
|
||||
getConfiguredTimeout,
|
||||
} from "@/shared/utils/fetchTimeout";
|
||||
import { sanitizeErrorMessage } from "../../utils/error.ts";
|
||||
import { sleep } from "../../utils/sleep.ts";
|
||||
|
||||
interface LogLike {
|
||||
info?: (tag: string, msg: string, meta?: unknown) => void;
|
||||
warn?: (tag: string, msg: string, meta?: unknown) => void;
|
||||
error?: (tag: string, msg: string, meta?: unknown) => void;
|
||||
}
|
||||
|
||||
interface CredentialsLike {
|
||||
providerSpecificData?: { baseUrl?: unknown } | null;
|
||||
baseUrl?: unknown;
|
||||
apiKey?: unknown;
|
||||
accessToken?: unknown;
|
||||
}
|
||||
|
||||
/** Dot-path reader restricted to plain objects/arrays (no prototypes). */
|
||||
function readPath(value: unknown, path: string): unknown {
|
||||
if (!path) return value;
|
||||
let current: unknown = value;
|
||||
for (const segment of path.split(".")) {
|
||||
if (current === null || current === undefined) return undefined;
|
||||
if (typeof current !== "object") return undefined;
|
||||
if (Array.isArray(current)) {
|
||||
const index = Number(segment);
|
||||
if (!Number.isInteger(index) || index < 0 || index >= current.length) return undefined;
|
||||
current = current[index];
|
||||
continue;
|
||||
}
|
||||
if (!Object.prototype.hasOwnProperty.call(current, segment)) return undefined;
|
||||
current = (current as Record<string, unknown>)[segment];
|
||||
}
|
||||
return current;
|
||||
}
|
||||
|
||||
/** Non-empty string from a dot path, or null. */
|
||||
function readStringPath(value: unknown, path: string): string | null {
|
||||
const found = readPath(value, path);
|
||||
return typeof found === "string" && found.trim() ? found : null;
|
||||
}
|
||||
|
||||
function isDoneStatus(
|
||||
status: unknown,
|
||||
done: string[],
|
||||
failed: string[]
|
||||
): "done" | "failed" | "pending" {
|
||||
if (typeof status !== "string") return "pending";
|
||||
if (failed.includes(status)) return "failed";
|
||||
if (done.includes(status)) return "done";
|
||||
return "pending";
|
||||
}
|
||||
|
||||
export type VideoJobPreset = {
|
||||
id: string;
|
||||
displayName: string;
|
||||
/** auth header name plus value scheme */
|
||||
authHeaderName: "x-api-key" | "Authorization";
|
||||
authScheme: "bearer" | "raw";
|
||||
baseUrlFallback: string;
|
||||
submit: {
|
||||
method: "POST";
|
||||
/** may contain {model} — substituted before POST */
|
||||
path: string;
|
||||
buildBody: (params: {
|
||||
model?: string;
|
||||
prompt?: string;
|
||||
duration?: number;
|
||||
extras: Record<string, unknown>;
|
||||
}) => Record<string, unknown>;
|
||||
};
|
||||
/** dot path into the submit response identifying the job */
|
||||
taskIdPath: string;
|
||||
poll: {
|
||||
/** contains {taskId} */
|
||||
pathTemplate: string;
|
||||
};
|
||||
statusPath: string;
|
||||
statusDone: string[];
|
||||
statusFailed: string[];
|
||||
/** dot path into the poll response holding the finished video URL/array */
|
||||
resultPath: string;
|
||||
maxPolls: number;
|
||||
pollIntervalMs: number;
|
||||
};
|
||||
|
||||
// #9820: declarative presets for the shipping async job/poll video providers.
|
||||
const VIDEO_JOB_PRESETS: Record<string, VideoJobPreset> = {
|
||||
"agnes-video-job": {
|
||||
id: "agnes-video-job",
|
||||
displayName: "Agnes Video V2.0",
|
||||
authHeaderName: "x-api-key",
|
||||
authScheme: "raw",
|
||||
// Real default, matching the Agnes Video V2.0 reference: POST /v1/videos with
|
||||
// x-api-key auth; GET /v1/videos/{task_id} returns status/progress/metadata.
|
||||
baseUrlFallback: "https://apihub.agnes-ai.com",
|
||||
submit: {
|
||||
method: "POST",
|
||||
path: "/v1/videos",
|
||||
buildBody: ({ model, prompt, extras }) => ({
|
||||
model,
|
||||
prompt,
|
||||
// passthrough of image/mode/num_frames/frame_rate/… — the generic
|
||||
// route body uses .catchall, so provider-specific knobs survive.
|
||||
...extras,
|
||||
}),
|
||||
},
|
||||
taskIdPath: "task_id",
|
||||
poll: { pathTemplate: "/v1/videos/{taskId}" },
|
||||
statusPath: "status",
|
||||
statusDone: ["completed"],
|
||||
statusFailed: ["failed"],
|
||||
resultPath: "metadata.url",
|
||||
maxPolls: 60,
|
||||
pollIntervalMs: 2000,
|
||||
},
|
||||
"muapi-video-job": {
|
||||
id: "muapi-video-job",
|
||||
displayName: "muapi.ai",
|
||||
authHeaderName: "x-api-key",
|
||||
authScheme: "raw",
|
||||
// muapi.ai video/audio surface is Replicate-style: POST /api/v1/{model}
|
||||
// returns { request_id }; poll GET /api/v1/predictions/{id}/result.
|
||||
baseUrlFallback: "https://api.muapi.ai",
|
||||
submit: {
|
||||
method: "POST",
|
||||
path: "/api/v1/{model}",
|
||||
buildBody: (params) => {
|
||||
const { prompt, duration, extras } = params;
|
||||
return {
|
||||
prompt,
|
||||
...(typeof duration === "number" ? { duration } : {}),
|
||||
...extras,
|
||||
};
|
||||
},
|
||||
},
|
||||
taskIdPath: "request_id",
|
||||
poll: { pathTemplate: "/api/v1/predictions/{taskId}/result" },
|
||||
statusPath: "status",
|
||||
statusDone: ["completed"],
|
||||
statusFailed: ["failed"],
|
||||
resultPath: "outputs",
|
||||
maxPolls: 60,
|
||||
pollIntervalMs: 2000,
|
||||
},
|
||||
"sora-job": {
|
||||
id: "sora-job",
|
||||
displayName: "OpenAI Sora",
|
||||
authHeaderName: "Authorization",
|
||||
authScheme: "bearer",
|
||||
baseUrlFallback: "https://api.openai.com",
|
||||
submit: {
|
||||
method: "POST",
|
||||
path: "/v1/videos",
|
||||
buildBody: (params) => {
|
||||
const { model, prompt, duration, extras } = params;
|
||||
// seconds is a STRING enum ("4"|"8"|"12") in the Sora API; absolute
|
||||
// size mapping is intentionally not forced here.
|
||||
return {
|
||||
model,
|
||||
prompt,
|
||||
...(typeof duration === "number" ? { seconds: String(duration) } : {}),
|
||||
...extras,
|
||||
};
|
||||
},
|
||||
},
|
||||
taskIdPath: "id",
|
||||
poll: { pathTemplate: "/v1/videos/{taskId}" },
|
||||
statusPath: "status",
|
||||
statusDone: ["completed"],
|
||||
statusFailed: ["failed"],
|
||||
resultPath: "data",
|
||||
maxPolls: 60,
|
||||
pollIntervalMs: 2000,
|
||||
},
|
||||
};
|
||||
|
||||
/** Resolve a configured job preset; null when the preset is unknown/none. */
|
||||
export function getVideoJobPreset(presetName: unknown): VideoJobPreset | null {
|
||||
if (typeof presetName !== "string") return null;
|
||||
const preset = VIDEO_JOB_PRESETS[presetName];
|
||||
return preset ?? null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Handle a video-generation job via the submit→poll preset pipeline.
|
||||
* Returns the same shape as the sync handlers: { success, data?: …, status?, error? }.
|
||||
*/
|
||||
export async function handleVideoJobGeneration({
|
||||
model,
|
||||
presetName,
|
||||
body,
|
||||
credentials,
|
||||
log,
|
||||
maxPolls: maxPollsOverride,
|
||||
pollIntervalMs: pollIntervalOverride,
|
||||
}: {
|
||||
model: string;
|
||||
presetName: string;
|
||||
body: Record<string, unknown>;
|
||||
credentials?: unknown;
|
||||
log?: {
|
||||
info?: (tag: string, msg: string, meta?: unknown) => void;
|
||||
error?: (tag: string, msg: string) => void;
|
||||
};
|
||||
maxPolls?: number;
|
||||
pollIntervalMs?: number;
|
||||
}) {
|
||||
const preset = getVideoJobPreset(presetName);
|
||||
if (!preset) {
|
||||
return {
|
||||
success: false,
|
||||
status: 400,
|
||||
error: `Unknown video job preset: ${presetName}`,
|
||||
};
|
||||
}
|
||||
|
||||
const baseUrl = resolveJobBaseUrl(credentials, preset.baseUrlFallback);
|
||||
log?.info?.("VIDEO", `Job preset ${presetName} submitting ${model}`);
|
||||
log?.info?.("VIDEO", JSON.stringify({ baseUrl }));
|
||||
|
||||
const bodyForPreset = preset.submit.buildBody({
|
||||
model: model,
|
||||
prompt: typeof body.prompt === "string" ? body.prompt : undefined,
|
||||
duration: typeof body.duration === "number" ? body.duration : undefined,
|
||||
// passthrough of the remainder — the API keeps catchall extras
|
||||
extras: Object.fromEntries(
|
||||
Object.entries(body ?? {}).filter(
|
||||
([key]) => key !== "model" && key !== "prompt" && key !== "duration"
|
||||
)
|
||||
),
|
||||
});
|
||||
|
||||
const submitPath = preset.submit.path.replace("{model}", encodeURIComponent(model));
|
||||
const submitUrl = `${baseUrl}${submitPath}`; // baseUrl never ends with "/"
|
||||
const submitResult = await fetchJson(submitUrl, {
|
||||
method: preset.submit.method,
|
||||
headers: buildJobHeaders(preset, credentials),
|
||||
body: JSON.stringify(bodyForPreset),
|
||||
log,
|
||||
});
|
||||
if (!submitResult.ok) {
|
||||
return { success: false, status: submitResult.status, error: submitResult.error };
|
||||
}
|
||||
|
||||
const taskId = readStringPath(submitResult.data, preset.taskIdPath);
|
||||
if (!taskId) {
|
||||
return {
|
||||
success: false,
|
||||
status: 502,
|
||||
error: `Video provider did not return a job id (${presetName})`,
|
||||
};
|
||||
}
|
||||
|
||||
// Poll loop.
|
||||
const maxPolls = maxPollsOverride ?? preset.maxPolls;
|
||||
const pollInterval = pollIntervalOverride ?? preset.pollIntervalMs;
|
||||
|
||||
for (let attempt = 1; attempt <= maxPolls; attempt += 1) {
|
||||
await sleep(pollInterval);
|
||||
const pollUrl = `${baseUrl}${preset.poll.pathTemplate.replace("{taskId}", encodeURIComponent(taskId))}`;
|
||||
const pollResult = await fetchJson(pollUrl, {
|
||||
method: "GET",
|
||||
headers: buildJobHeaders(preset, credentials),
|
||||
log,
|
||||
});
|
||||
if (!pollResult.ok) {
|
||||
return { success: false, status: pollResult.status, error: pollResult.error };
|
||||
}
|
||||
|
||||
const status = readPath(pollResult.data, preset.statusPath);
|
||||
const jobState = isDoneStatus(status, preset.statusDone, preset.statusFailed);
|
||||
if (jobState === "done") {
|
||||
const url = readResultUrl(pollResult.data, preset.resultPath);
|
||||
if (!url) {
|
||||
return {
|
||||
success: false,
|
||||
status: 502,
|
||||
error: `Video job completed but no result URL found (${presetName})`,
|
||||
};
|
||||
}
|
||||
log?.info?.("VIDEO", `Job completed after ${attempt} poll(s)`);
|
||||
return {
|
||||
success: true,
|
||||
data: {
|
||||
created: Math.floor(Date.now() / 1000),
|
||||
data: [{ url, format: "mp4" }],
|
||||
},
|
||||
};
|
||||
}
|
||||
if (jobState === "failed") {
|
||||
return {
|
||||
success: false,
|
||||
status: 502,
|
||||
error: `Video job failed (${presetName})`,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
success: false,
|
||||
status: 504,
|
||||
error: `Video job timed out after ${maxPolls} polls (${presetName})`,
|
||||
};
|
||||
}
|
||||
|
||||
function buildJobHeaders(preset: VideoJobPreset, credentials?: unknown): Record<string, string> {
|
||||
const creds = credentials as CredentialsLike | null | undefined;
|
||||
const apiKey =
|
||||
typeof creds?.apiKey === "string" && creds.apiKey
|
||||
? creds.apiKey
|
||||
: typeof creds?.accessToken === "string" && creds.accessToken
|
||||
? creds.accessToken
|
||||
: "";
|
||||
const headers: Record<string, string> = { "Content-Type": "application/json" };
|
||||
if (!apiKey) return headers;
|
||||
if (preset.authScheme === "raw") {
|
||||
headers[preset.authHeaderName] = apiKey;
|
||||
} else {
|
||||
headers[preset.authHeaderName] = `Bearer ${apiKey}`;
|
||||
}
|
||||
return headers;
|
||||
}
|
||||
|
||||
function resolveJobBaseUrl(credentials: unknown, fallback: string): string {
|
||||
const creds = credentials as CredentialsLike | null | undefined;
|
||||
const psdBaseUrl =
|
||||
creds?.providerSpecificData?.baseUrl != null &&
|
||||
typeof creds.providerSpecificData.baseUrl === "string" &&
|
||||
creds.providerSpecificData.baseUrl.trim()
|
||||
? (creds.providerSpecificData.baseUrl as string).trim()
|
||||
: null;
|
||||
const topLevelBaseUrl =
|
||||
creds?.baseUrl != null && typeof creds.baseUrl === "string" && creds.baseUrl.trim()
|
||||
? (creds.baseUrl as string).trim()
|
||||
: null;
|
||||
const nodeBaseUrl = psdBaseUrl || topLevelBaseUrl;
|
||||
if (!nodeBaseUrl) return fallback.replace(/\/+$/, "");
|
||||
let normalized = nodeBaseUrl;
|
||||
while (normalized.endsWith("/")) normalized = normalized.slice(0, -1);
|
||||
return normalized;
|
||||
}
|
||||
|
||||
async function fetchJson(
|
||||
url: string,
|
||||
{
|
||||
method,
|
||||
headers,
|
||||
body,
|
||||
log,
|
||||
}: {
|
||||
method: string;
|
||||
headers: Record<string, string>;
|
||||
body?: string;
|
||||
log?: LogLike;
|
||||
}
|
||||
): Promise<{ ok: true; data: unknown } | { ok: false; status: number; error: string }> {
|
||||
try {
|
||||
const response = await fetchWithTimeout(url, {
|
||||
method,
|
||||
headers,
|
||||
...(body !== undefined ? { body } : {}),
|
||||
timeoutMs: getConfiguredTimeout(),
|
||||
});
|
||||
if (!response.ok) {
|
||||
const errorText = await response.text();
|
||||
log?.error?.("VIDEO", `Upstream ${response.status} for ${url}: ${errorText.slice(0, 200)}`);
|
||||
return { ok: false, status: response.status, error: errorText };
|
||||
}
|
||||
const data = await response.json();
|
||||
return { ok: true, data };
|
||||
} catch (err: unknown) {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
const isTimeout =
|
||||
err instanceof FetchTimeoutError || (err instanceof Error && err.name === "AbortError");
|
||||
log?.error?.(
|
||||
"VIDEO",
|
||||
`${isTimeout ? "Timeout" : "Request error"} for ${url}: ${sanitizeErrorMessage(message)}`
|
||||
);
|
||||
return {
|
||||
ok: false,
|
||||
status: isTimeout ? 504 : 502,
|
||||
error: `Video provider error: ${sanitizeErrorMessage(message)}`,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
function readResultUrl(data: unknown, resultPath: string): string | null {
|
||||
const found = readPath(data, resultPath);
|
||||
if (typeof found === "string" && found.trim()) return found.trim();
|
||||
if (Array.isArray(found)) {
|
||||
const first = found[0];
|
||||
// muapi-style: resultPath "outputs" resolves to ["https://…"].
|
||||
if (typeof first === "string" && first.trim()) return first.trim();
|
||||
// sora-style: resultPath "data" resolves to [{ url: "https://…" }].
|
||||
if (first && typeof first === "object" && !Array.isArray(first)) {
|
||||
const urlEntry = (first as Record<string, unknown>).url;
|
||||
if (typeof urlEntry === "string" && urlEntry.trim()) return urlEntry.trim();
|
||||
}
|
||||
return null;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
156
open-sse/handlers/videoGeneration/openai.ts
Normal file
156
open-sse/handlers/videoGeneration/openai.ts
Normal file
@@ -0,0 +1,156 @@
|
||||
import {
|
||||
fetchWithTimeout,
|
||||
FetchTimeoutError,
|
||||
getConfiguredTimeout,
|
||||
} from "@/shared/utils/fetchTimeout";
|
||||
import { saveCallLog } from "@/lib/usageDb";
|
||||
import { sanitizeErrorMessage } from "../../utils/error.ts";
|
||||
|
||||
interface LogLike {
|
||||
info?: (tag: string, msg: string, meta?: unknown) => void;
|
||||
error?: (tag: string, msg: string) => void;
|
||||
}
|
||||
|
||||
interface CredentialsLike {
|
||||
providerSpecificData?: { baseUrl?: unknown } | null;
|
||||
baseUrl?: unknown;
|
||||
apiKey?: unknown;
|
||||
accessToken?: unknown;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the video generation endpoint URL from credentials and fallback.
|
||||
* Handles baseUrl from providerSpecificData or top-level credentials.
|
||||
*/
|
||||
function resolveVideoEndpoint(credentials: unknown, fallback: string): string {
|
||||
const creds = credentials as CredentialsLike | null | undefined;
|
||||
const psdBaseUrl =
|
||||
creds?.providerSpecificData?.baseUrl != null &&
|
||||
typeof creds.providerSpecificData.baseUrl === "string" &&
|
||||
creds.providerSpecificData.baseUrl.trim()
|
||||
? creds.providerSpecificData.baseUrl.trim()
|
||||
: null;
|
||||
const topLevelBaseUrl =
|
||||
creds?.baseUrl != null && typeof creds.baseUrl === "string" && creds.baseUrl.trim()
|
||||
? creds.baseUrl.trim()
|
||||
: null;
|
||||
const nodeBaseUrl = psdBaseUrl || topLevelBaseUrl;
|
||||
let n = nodeBaseUrl;
|
||||
while (n.endsWith("/")) n = n.slice(0, -1);
|
||||
if (n.endsWith("/videos/generations")) return n;
|
||||
return `${n}/videos/generations`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch the video generation endpoint with timeout and error handling.
|
||||
*/
|
||||
async function fetchVideoEndpoint(
|
||||
url: string,
|
||||
{ headers, body, log }: { headers: Record<string, string>; body: string; log?: LogLike }
|
||||
) {
|
||||
try {
|
||||
const response = await fetchWithTimeout(url, {
|
||||
method: "POST",
|
||||
headers,
|
||||
body,
|
||||
timeoutMs: getConfiguredTimeout(),
|
||||
});
|
||||
if (!response.ok) {
|
||||
const errorText = await response.text();
|
||||
log?.error?.("VIDEO", `Upstream ${response.status} for ${url}: ${errorText}`);
|
||||
return { success: false, status: response.status, error: errorText };
|
||||
}
|
||||
const data = await response.json();
|
||||
return {
|
||||
success: true,
|
||||
data: { created: data.created || Math.floor(Date.now() / 1000), data: data.data || [] },
|
||||
};
|
||||
} catch (err) {
|
||||
const message = err?.message;
|
||||
const isTimeout = err instanceof FetchTimeoutError || err?.name === "AbortError";
|
||||
log?.error?.(
|
||||
"VIDEO",
|
||||
`${isTimeout ? "Timeout" : "Request error"} for ${url}: ${sanitizeErrorMessage(message || err)}`
|
||||
);
|
||||
return {
|
||||
success: false,
|
||||
status: isTimeout ? 504 : 502,
|
||||
error: `Video provider error: ${sanitizeErrorMessage(message || err)}`,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Handle OpenAI-compatible video generation.
|
||||
* This handler is dispatched for custom providers with format "openai-video".
|
||||
*/
|
||||
export async function handleOpenAIVideoGeneration({
|
||||
model,
|
||||
provider,
|
||||
providerConfig,
|
||||
body,
|
||||
credentials,
|
||||
log,
|
||||
}: {
|
||||
model: string;
|
||||
provider: string;
|
||||
providerConfig: { baseUrl: string; authHeader: string };
|
||||
body: unknown;
|
||||
credentials: unknown;
|
||||
log?: LogLike;
|
||||
}) {
|
||||
const startTime = Date.now();
|
||||
const creds = credentials as CredentialsLike | null | undefined;
|
||||
const apiToken = creds?.apiKey || creds?.accessToken;
|
||||
const endpoint = resolveVideoEndpoint(credentials, providerConfig.baseUrl);
|
||||
const headers = {
|
||||
"Content-Type": "application/json",
|
||||
...(providerConfig.authHeader === "x-api-key"
|
||||
? { "x-api-key": String(apiToken) }
|
||||
: { Authorization: `Bearer ${apiToken}` }),
|
||||
};
|
||||
const bodyObj = body as Record<string, unknown>;
|
||||
const upstreamBody = {
|
||||
model,
|
||||
prompt: (bodyObj.prompt ?? "") as string,
|
||||
...(typeof bodyObj.duration === "number" && { duration: bodyObj.duration }),
|
||||
};
|
||||
const logRequestBody = {
|
||||
model: bodyObj.model,
|
||||
prompt:
|
||||
typeof bodyObj.prompt === "string"
|
||||
? bodyObj.prompt.slice(0, 200)
|
||||
: String(bodyObj.prompt ?? ""),
|
||||
duration: bodyObj.duration,
|
||||
};
|
||||
log?.info?.("VIDEO", `OpenAI-compatible video generation: ${provider}/${model} -> ${endpoint}`, {
|
||||
body: logRequestBody,
|
||||
});
|
||||
|
||||
const fetchResult = await fetchVideoEndpoint(endpoint, {
|
||||
headers,
|
||||
body: JSON.stringify(upstreamBody),
|
||||
log,
|
||||
});
|
||||
|
||||
if (!fetchResult.success) {
|
||||
return { success: false, status: fetchResult.status, error: fetchResult.error };
|
||||
}
|
||||
|
||||
// Save call log for billing/tracking
|
||||
await saveCallLog({
|
||||
provider,
|
||||
model: String(bodyObj.model),
|
||||
endpoint: "video",
|
||||
status: fetchResult.status,
|
||||
durationMs: Date.now() - startTime,
|
||||
tokensIn: 0,
|
||||
tokensOut: 0,
|
||||
requestId: null,
|
||||
});
|
||||
|
||||
return {
|
||||
success: true,
|
||||
data: fetchResult.data,
|
||||
};
|
||||
}
|
||||
125
open-sse/handlers/videoGeneration/runwayHelpers.ts
Normal file
125
open-sse/handlers/videoGeneration/runwayHelpers.ts
Normal file
@@ -0,0 +1,125 @@
|
||||
export function resolveRunwayPromptImage(body) {
|
||||
const directCandidates = [
|
||||
body.promptImage,
|
||||
body.prompt_image,
|
||||
body.image,
|
||||
body.image_url,
|
||||
body.imageUrl,
|
||||
body.provider_options?.promptImage,
|
||||
body.provider_options?.prompt_image,
|
||||
];
|
||||
|
||||
for (const candidate of directCandidates) {
|
||||
if (typeof candidate === "string" && candidate.trim()) return candidate.trim();
|
||||
if (candidate && typeof candidate === "object") return candidate;
|
||||
if (Array.isArray(candidate) && candidate.length > 0) return candidate;
|
||||
}
|
||||
|
||||
const arrayCandidates = [
|
||||
body.imageUrls,
|
||||
body.image_urls,
|
||||
body.provider_options?.imageUrls,
|
||||
body.provider_options?.image_urls,
|
||||
];
|
||||
for (const candidate of arrayCandidates) {
|
||||
if (Array.isArray(candidate) && candidate.length > 0) return candidate;
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
export function resolveRunwayRatio(body) {
|
||||
const aspectRatio = typeof body.aspect_ratio === "string" ? body.aspect_ratio : body.aspectRatio;
|
||||
if (aspectRatio === "1280:720" || aspectRatio === "720:1280") return aspectRatio;
|
||||
if (aspectRatio === "16:9") return "1280:720";
|
||||
if (aspectRatio === "9:16") return "720:1280";
|
||||
|
||||
const size = typeof body.size === "string" ? body.size : "";
|
||||
const [widthRaw, heightRaw] = size.split("x");
|
||||
const width = Number(widthRaw);
|
||||
const height = Number(heightRaw);
|
||||
if (Number.isFinite(width) && Number.isFinite(height) && width > 0 && height > 0) {
|
||||
return width >= height ? "1280:720" : "720:1280";
|
||||
}
|
||||
|
||||
return "1280:720";
|
||||
}
|
||||
|
||||
export function resolveRunwayDuration(body) {
|
||||
if (Number.isFinite(body.duration)) return clampRunwayDuration(body.duration);
|
||||
if (Number.isFinite(body.frames) && Number.isFinite(body.fps) && Number(body.fps) > 0) {
|
||||
return clampRunwayDuration(Number(body.frames) / Number(body.fps));
|
||||
}
|
||||
return 5;
|
||||
}
|
||||
|
||||
function clampRunwayDuration(value) {
|
||||
const duration = Math.round(Number(value));
|
||||
if (!Number.isFinite(duration)) return 5;
|
||||
return Math.min(10, Math.max(2, duration));
|
||||
}
|
||||
|
||||
export function resolvePositiveInteger(value, fallback) {
|
||||
const numeric = Number(value);
|
||||
if (!Number.isFinite(numeric) || numeric <= 0) return fallback;
|
||||
return Math.floor(numeric);
|
||||
}
|
||||
|
||||
function extractRunwayOutputUrls(task) {
|
||||
const rawOutput = Array.isArray(task?.output)
|
||||
? task.output
|
||||
: Array.isArray(task?.result)
|
||||
? task.result
|
||||
: [];
|
||||
return rawOutput
|
||||
.map((entry) => {
|
||||
if (typeof entry === "string") return entry;
|
||||
if (!entry || typeof entry !== "object") return null;
|
||||
return entry.url || entry.uri || entry.videoUrl || entry.video_url || null;
|
||||
})
|
||||
.filter((value) => typeof value === "string" && value.length > 0);
|
||||
}
|
||||
|
||||
export function extractRunwayFailureMessage(task) {
|
||||
const directCandidates = [
|
||||
task?.failure,
|
||||
task?.failureReason,
|
||||
task?.error,
|
||||
task?.errorMessage,
|
||||
task?.message,
|
||||
];
|
||||
for (const candidate of directCandidates) {
|
||||
if (typeof candidate === "string" && candidate.trim()) return candidate.trim();
|
||||
}
|
||||
if (task?.failure && typeof task.failure === "object") {
|
||||
const nestedCandidates = [
|
||||
task.failure.message,
|
||||
task.failure.reason,
|
||||
task.failure.error,
|
||||
task.failure.code,
|
||||
];
|
||||
for (const candidate of nestedCandidates) {
|
||||
if (typeof candidate === "string" && candidate.trim()) return candidate.trim();
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
export async function normalizeRunwayVideoResult(task, body) {
|
||||
const urls = extractRunwayOutputUrls(task);
|
||||
if (urls.length === 0) {
|
||||
throw new Error(
|
||||
`Runway task completed without output URLs: ${JSON.stringify(task).slice(0, 400)}`
|
||||
);
|
||||
}
|
||||
if (body.response_format === "url") return urls.map((url) => ({ url, format: "mp4" }));
|
||||
|
||||
const videos = [];
|
||||
for (const url of urls) {
|
||||
const response = await fetch(url);
|
||||
if (!response.ok) throw new Error(`Runway output fetch failed (${response.status})`);
|
||||
const arrayBuffer = await response.arrayBuffer();
|
||||
videos.push({ b64_json: Buffer.from(arrayBuffer).toString("base64"), format: "mp4" });
|
||||
}
|
||||
return videos;
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
8
open-sse/services/adobeFireflyModelSnapshot.ts
Normal file
8
open-sse/services/adobeFireflyModelSnapshot.ts
Normal file
File diff suppressed because one or more lines are too long
@@ -1,328 +1,590 @@
|
||||
/**
|
||||
* Adobe Firefly model catalog: live discovery + static fallback from browser capture.
|
||||
* Adobe Firefly model discovery and normalized media capabilities.
|
||||
*
|
||||
* Live: POST firefly-3p.ff.adobe.io/v2/models/discovery (needs valid IMS token).
|
||||
* Fallback: curated rows from adobe/get_models.txt (2026-07 Firefly SPA capture) so
|
||||
* Media/Models still list usable ids when discovery fails or credentials are missing.
|
||||
* The live discovery schema is authoritative. The generated snapshot is used only
|
||||
* when a request cannot perform authenticated discovery (for example /v1/models).
|
||||
*/
|
||||
|
||||
import {
|
||||
type AdobeFireflyDiscoveredModel,
|
||||
discoverAdobeFireflyModels,
|
||||
resolveAdobeAccessToken,
|
||||
} from "./adobeFireflyClient.ts";
|
||||
import { ADOBE_FIREFLY_DISCOVERY_SNAPSHOT } from "./adobeFireflyModelSnapshot.ts";
|
||||
|
||||
export type AdobeFireflyModality = "image" | "video" | "audio" | "unknown";
|
||||
|
||||
export interface AdobeFireflyDiscoveredModel {
|
||||
modelId: string;
|
||||
modelVersion: string;
|
||||
displayName: string;
|
||||
modality: AdobeFireflyModality;
|
||||
enabled: boolean;
|
||||
providerName?: string;
|
||||
releaseReadiness?: string;
|
||||
healthStatus?: string;
|
||||
inputMediaUseCases: string[];
|
||||
requestSchema?: Record<string, unknown>;
|
||||
backingModel?: string;
|
||||
}
|
||||
|
||||
export interface AdobeFireflyReferenceInputCapability {
|
||||
mediaType: string;
|
||||
usageType: string;
|
||||
minItems: number;
|
||||
maxItems: number | null;
|
||||
maxFileSizeBytes: number | null;
|
||||
}
|
||||
|
||||
export interface AdobeFireflyMediaCapabilities {
|
||||
inputMediaUseCases: string[];
|
||||
schemaProperties: string[];
|
||||
requiredProperties: string[];
|
||||
referenceInputs: AdobeFireflyReferenceInputCapability[];
|
||||
maxReferenceItems: number | null;
|
||||
supportedSizes: string[];
|
||||
supportedAspectRatios: string[];
|
||||
supportedResolutions: string[];
|
||||
supportedDurations: number[];
|
||||
durationMin: number | null;
|
||||
durationMax: number | null;
|
||||
durationDefault: number | null;
|
||||
outputCountMin: number | null;
|
||||
outputCountMax: number | null;
|
||||
promptMaxLength: number | null;
|
||||
releaseReadiness: string;
|
||||
healthStatus: string;
|
||||
}
|
||||
|
||||
export interface AdobeFireflyCatalogModel {
|
||||
/** OpenAI-style id without provider prefix, e.g. nano-banana-pro or flux-fluxPro */
|
||||
/** Stable API id without the provider prefix. */
|
||||
id: string;
|
||||
name: string;
|
||||
modality: "image" | "video";
|
||||
/** Upstream wire modelId for generate-async */
|
||||
upstreamModelId: string;
|
||||
/** Upstream wire modelVersion for generate-async */
|
||||
upstreamModelVersion: string;
|
||||
inputModalities?: string[];
|
||||
providerName: string;
|
||||
backingModel: string;
|
||||
inputModalities: string[];
|
||||
capabilities: AdobeFireflyMediaCapabilities;
|
||||
}
|
||||
|
||||
/**
|
||||
* Static fallback built from adobe/get_models.txt discovery response.
|
||||
* Friendly aliases first (Media page defaults), then popular upstream families.
|
||||
*/
|
||||
export const ADOBE_FIREFLY_FALLBACK_MODELS: AdobeFireflyCatalogModel[] = [
|
||||
// ── Friendly aliases (handler resolveAdobeImageModel / resolveAdobeVideoModel) ──
|
||||
{
|
||||
id: "nano-banana-pro",
|
||||
name: "Gemini 3.0 (Nano Banana Pro)",
|
||||
modality: "image",
|
||||
upstreamModelId: "gemini-flash",
|
||||
upstreamModelVersion: "nano-banana-2",
|
||||
inputModalities: ["text", "image"],
|
||||
},
|
||||
{
|
||||
id: "nano-banana",
|
||||
name: "Gemini 2.5 (Nano Banana)",
|
||||
modality: "image",
|
||||
upstreamModelId: "gemini-flash",
|
||||
upstreamModelVersion: "nano-banana",
|
||||
inputModalities: ["text", "image"],
|
||||
},
|
||||
{
|
||||
id: "nano-banana-2",
|
||||
name: "Gemini 3.1 (Nano Banana 2)",
|
||||
modality: "image",
|
||||
upstreamModelId: "gemini-flash",
|
||||
upstreamModelVersion: "nano-banana-3",
|
||||
inputModalities: ["text", "image"],
|
||||
},
|
||||
{
|
||||
id: "gpt-image-2",
|
||||
name: "GPT Image 2",
|
||||
modality: "image",
|
||||
upstreamModelId: "gpt-image",
|
||||
upstreamModelVersion: "2",
|
||||
inputModalities: ["text", "image"],
|
||||
},
|
||||
{
|
||||
id: "gpt-image",
|
||||
name: "GPT Image 2",
|
||||
modality: "image",
|
||||
upstreamModelId: "gpt-image",
|
||||
upstreamModelVersion: "2",
|
||||
inputModalities: ["text", "image"],
|
||||
},
|
||||
{
|
||||
id: "gpt-image-1.5",
|
||||
name: "GPT Image 1.5",
|
||||
modality: "image",
|
||||
upstreamModelId: "gpt-image",
|
||||
upstreamModelVersion: "1.5",
|
||||
inputModalities: ["text", "image"],
|
||||
},
|
||||
{
|
||||
id: "sora-2",
|
||||
name: "Sora 2",
|
||||
modality: "video",
|
||||
upstreamModelId: "sora",
|
||||
upstreamModelVersion: "sora-2",
|
||||
},
|
||||
{
|
||||
id: "sora-2-pro",
|
||||
name: "Sora 2 Pro",
|
||||
modality: "video",
|
||||
upstreamModelId: "sora",
|
||||
upstreamModelVersion: "sora-2-pro",
|
||||
},
|
||||
{
|
||||
id: "veo-3.1",
|
||||
name: "Veo 3.1",
|
||||
modality: "video",
|
||||
upstreamModelId: "veo",
|
||||
upstreamModelVersion: "3.1-generate",
|
||||
},
|
||||
{
|
||||
id: "veo-3.1-fast",
|
||||
name: "Veo 3.1 Fast",
|
||||
modality: "video",
|
||||
upstreamModelId: "veo",
|
||||
upstreamModelVersion: "3.1-fast-generate",
|
||||
},
|
||||
{
|
||||
id: "veo-3.1-ref",
|
||||
name: "Veo 3.1 Reference",
|
||||
modality: "video",
|
||||
upstreamModelId: "veo",
|
||||
upstreamModelVersion: "3.1-generate",
|
||||
},
|
||||
{
|
||||
id: "kling-3",
|
||||
name: "Kling Video v3 Standard Image to Video",
|
||||
modality: "video",
|
||||
upstreamModelId: "kling",
|
||||
upstreamModelVersion: "kling_v3_standard_i2v",
|
||||
},
|
||||
// ── Additional image families from discovery capture ──
|
||||
{
|
||||
id: "flux-2",
|
||||
name: "Flux 2",
|
||||
modality: "image",
|
||||
upstreamModelId: "flux",
|
||||
upstreamModelVersion: "2",
|
||||
inputModalities: ["text", "image"],
|
||||
},
|
||||
{
|
||||
id: "flux-pro",
|
||||
name: "Flux 1.1 Pro",
|
||||
modality: "image",
|
||||
upstreamModelId: "flux",
|
||||
upstreamModelVersion: "fluxPro",
|
||||
inputModalities: ["text", "image"],
|
||||
},
|
||||
{
|
||||
id: "flux-ultra",
|
||||
name: "Flux 1.1 Ultra",
|
||||
modality: "image",
|
||||
upstreamModelId: "flux",
|
||||
upstreamModelVersion: "fluxUltra",
|
||||
inputModalities: ["text", "image"],
|
||||
},
|
||||
{
|
||||
id: "seedream-4",
|
||||
name: "Seedream 4.0",
|
||||
modality: "image",
|
||||
upstreamModelId: "seedream",
|
||||
upstreamModelVersion: "seedream_v4",
|
||||
inputModalities: ["text", "image"],
|
||||
},
|
||||
{
|
||||
id: "seedream-5-lite",
|
||||
name: "Seedream 5.0 Lite",
|
||||
modality: "image",
|
||||
upstreamModelId: "seedream",
|
||||
upstreamModelVersion: "seedream_v5_lite",
|
||||
inputModalities: ["text", "image"],
|
||||
},
|
||||
{
|
||||
id: "runway-gen4-image",
|
||||
name: "Runway Gen-4 Image",
|
||||
modality: "image",
|
||||
upstreamModelId: "runway-gen4-image",
|
||||
upstreamModelVersion: "gen4_image",
|
||||
inputModalities: ["text", "image"],
|
||||
},
|
||||
// ── Additional video families ──
|
||||
{
|
||||
id: "kling-v3-t2v",
|
||||
name: "Kling Video v3 Standard Text to Video",
|
||||
modality: "video",
|
||||
upstreamModelId: "kling",
|
||||
upstreamModelVersion: "kling_v3_standard_t2v",
|
||||
},
|
||||
{
|
||||
id: "kling-v3-pro-i2v",
|
||||
name: "Kling Video v3 Pro Image to Video",
|
||||
modality: "video",
|
||||
upstreamModelId: "kling",
|
||||
upstreamModelVersion: "kling_v3_pro_i2v",
|
||||
},
|
||||
{
|
||||
id: "luma-ray3",
|
||||
name: "Ray3",
|
||||
modality: "video",
|
||||
upstreamModelId: "luma",
|
||||
upstreamModelVersion: "3.0-ray",
|
||||
},
|
||||
{
|
||||
id: "runway-gen4-turbo",
|
||||
name: "Runway Gen-4 Video",
|
||||
modality: "video",
|
||||
upstreamModelId: "runway",
|
||||
upstreamModelVersion: "gen4_turbo",
|
||||
},
|
||||
];
|
||||
export interface AdobeFireflyImageModelSpec extends AdobeFireflyCatalogModel {
|
||||
modality: "image";
|
||||
/** Payload dialect observed for this model family. */
|
||||
family: "gemini" | "gpt-image" | "generic";
|
||||
}
|
||||
|
||||
/** Stable slug for upstream modelId + modelVersion (catalog id when not a friendly alias). */
|
||||
export interface AdobeFireflyVideoModelSpec extends AdobeFireflyCatalogModel {
|
||||
modality: "video";
|
||||
defaultDuration: number;
|
||||
defaultResolution: string;
|
||||
}
|
||||
|
||||
interface MergedObjectSchema {
|
||||
properties: Record<string, Record<string, unknown>>;
|
||||
required: string[];
|
||||
}
|
||||
|
||||
function asRecord(value: unknown): Record<string, unknown> {
|
||||
return value && typeof value === "object" && !Array.isArray(value)
|
||||
? (value as Record<string, unknown>)
|
||||
: {};
|
||||
}
|
||||
|
||||
function asStringArray(value: unknown): string[] {
|
||||
return Array.isArray(value)
|
||||
? value.map((item) => String(item)).filter((item) => item.length > 0)
|
||||
: [];
|
||||
}
|
||||
|
||||
function finiteInteger(value: unknown): number | null {
|
||||
return Number.isInteger(value) ? (value as number) : null;
|
||||
}
|
||||
|
||||
/** Merge object properties/required keys contributed through JSON Schema allOf. */
|
||||
export function mergeAdobeObjectSchema(schema: unknown): MergedObjectSchema {
|
||||
const merged: MergedObjectSchema = { properties: {}, required: [] };
|
||||
const visit = (value: unknown) => {
|
||||
const node = asRecord(value);
|
||||
const properties = asRecord(node.properties);
|
||||
for (const [key, property] of Object.entries(properties)) {
|
||||
merged.properties[key] = asRecord(property);
|
||||
}
|
||||
merged.required.push(...asStringArray(node.required));
|
||||
if (Array.isArray(node.allOf)) node.allOf.forEach(visit);
|
||||
};
|
||||
visit(schema);
|
||||
merged.required = [...new Set(merged.required)];
|
||||
return merged;
|
||||
}
|
||||
|
||||
function schemaBranches(schema: unknown): Record<string, unknown>[] {
|
||||
const root = asRecord(schema);
|
||||
if (Object.keys(root).length === 0) return [];
|
||||
return [
|
||||
root,
|
||||
...(Array.isArray(root.anyOf) ? root.anyOf.map(asRecord) : []),
|
||||
...(Array.isArray(root.oneOf) ? root.oneOf.map(asRecord) : []),
|
||||
];
|
||||
}
|
||||
|
||||
function enumStrings(schema: unknown): string[] {
|
||||
return [
|
||||
...new Set(
|
||||
schemaBranches(schema)
|
||||
.flatMap((branch) => (Array.isArray(branch.enum) ? branch.enum : []))
|
||||
.filter((value): value is string => typeof value === "string")
|
||||
),
|
||||
];
|
||||
}
|
||||
|
||||
function integerBranch(schema: unknown): Record<string, unknown> {
|
||||
return schemaBranches(schema).find((branch) => branch.type === "integer") || {};
|
||||
}
|
||||
|
||||
/** Stable, collision-resistant public id for an exact upstream model/version pair. */
|
||||
export function slugifyAdobeModel(modelId: string, modelVersion: string): string {
|
||||
const mid = String(modelId || "")
|
||||
.trim()
|
||||
.toLowerCase()
|
||||
.replace(/[^a-z0-9]+/g, "-")
|
||||
.replace(/^-|-$/g, "");
|
||||
const ver = String(modelVersion || "")
|
||||
.trim()
|
||||
.toLowerCase()
|
||||
.replace(/[^a-z0-9.]+/g, "-")
|
||||
.replace(/^-|-$/g, "");
|
||||
if (!ver || ver === "default" || ver === mid) return mid || "model";
|
||||
return `${mid}-${ver}`;
|
||||
const slug = (value: string, allowDot = false) =>
|
||||
String(value || "")
|
||||
.trim()
|
||||
.toLowerCase()
|
||||
.replace(allowDot ? /[^a-z0-9.]+/g : /[^a-z0-9]+/g, "-")
|
||||
.replace(/^-|-$/g, "");
|
||||
const family = slug(modelId);
|
||||
// Adobe still uses `kling_v3_omni*` internally, while discovery exposes these
|
||||
// products to users as Kling O3. Never leak the obsolete/internal "omni" name
|
||||
// into the public API catalog; the untouched upstream version stays in the spec.
|
||||
const publicVersion =
|
||||
family === "kling" ? modelVersion.replace(/^kling_v3_omni/i, "kling_o3") : modelVersion;
|
||||
const version = slug(publicVersion, true);
|
||||
if (!version || version === "default" || version === family) return family || "model";
|
||||
return `${family}-${version}`;
|
||||
}
|
||||
|
||||
/** Map discovery rows → catalog entries (image/video only). */
|
||||
export function mapDiscoveredToCatalog(
|
||||
rows: AdobeFireflyDiscoveredModel[]
|
||||
): AdobeFireflyCatalogModel[] {
|
||||
const out: AdobeFireflyCatalogModel[] = [];
|
||||
const seen = new Set<string>();
|
||||
/** Parse POST /v2/models/discovery without discarding its resolved request schema. */
|
||||
export function parseAdobeModelsDiscovery(body: unknown): AdobeFireflyDiscoveredModel[] {
|
||||
const root = asRecord(body);
|
||||
const families = Array.isArray(root.models) ? root.models : [];
|
||||
const rows: AdobeFireflyDiscoveredModel[] = [];
|
||||
|
||||
// Prefer friendly aliases when upstream matches known fallback rows.
|
||||
for (const fb of ADOBE_FIREFLY_FALLBACK_MODELS) {
|
||||
const hit = rows.find(
|
||||
(r) =>
|
||||
r.modelId === fb.upstreamModelId &&
|
||||
r.modelVersion === fb.upstreamModelVersion &&
|
||||
(r.modality === fb.modality || r.modality === "unknown")
|
||||
);
|
||||
if (hit && !seen.has(fb.id)) {
|
||||
seen.add(fb.id);
|
||||
out.push({
|
||||
...fb,
|
||||
name: hit.displayName || fb.name,
|
||||
for (const familyValue of families) {
|
||||
const family = asRecord(familyValue);
|
||||
const modelId = String(family.modelId || "").trim();
|
||||
if (!modelId) continue;
|
||||
for (const [modelVersion, versionValue] of Object.entries(asRecord(family.modelVersions))) {
|
||||
const version = asRecord(versionValue);
|
||||
if (version.enabled === false) continue;
|
||||
const outputModalities = asStringArray(version.outputModality).map((item) =>
|
||||
item.toLowerCase()
|
||||
);
|
||||
const modality: AdobeFireflyModality = outputModalities.includes("image")
|
||||
? "image"
|
||||
: outputModalities.includes("video")
|
||||
? "video"
|
||||
: outputModalities.includes("audio")
|
||||
? "audio"
|
||||
: "unknown";
|
||||
rows.push({
|
||||
modelId,
|
||||
modelVersion,
|
||||
displayName: String(
|
||||
version.modelDisplayName || version.modelCaiDisplayName || modelVersion
|
||||
),
|
||||
modality,
|
||||
enabled: version.enabled !== false,
|
||||
providerName:
|
||||
typeof family.acModelFamilyProviderDisplayName === "string"
|
||||
? family.acModelFamilyProviderDisplayName
|
||||
: undefined,
|
||||
releaseReadiness:
|
||||
typeof version.releaseReadiness === "string" ? version.releaseReadiness : undefined,
|
||||
healthStatus: typeof version.healthStatus === "string" ? version.healthStatus : undefined,
|
||||
inputMediaUseCases: asStringArray(version.inputMediaUseCase),
|
||||
requestSchema: asRecord(version.requestSchema),
|
||||
backingModel:
|
||||
typeof version.bksGenerationModel === "string" ? version.bksGenerationModel : undefined,
|
||||
});
|
||||
}
|
||||
}
|
||||
return rows;
|
||||
}
|
||||
|
||||
function normalizeCapabilities(row: AdobeFireflyDiscoveredModel): AdobeFireflyMediaCapabilities {
|
||||
const schema = mergeAdobeObjectSchema(row.requestSchema);
|
||||
const referenceSchema = asRecord(schema.properties.referenceBlobs);
|
||||
const referenceInputs: AdobeFireflyReferenceInputCapability[] = [];
|
||||
const mediaCapabilities = Array.isArray(referenceSchema["x-capabilities"])
|
||||
? referenceSchema["x-capabilities"]
|
||||
: [];
|
||||
for (const mediaValue of mediaCapabilities) {
|
||||
const media = asRecord(mediaValue);
|
||||
const maxFileSizeBytes = finiteInteger(media.maxFileSizeBytes);
|
||||
const usageConstraints = Array.isArray(media.usageConstraints) ? media.usageConstraints : [];
|
||||
for (const usageValue of usageConstraints) {
|
||||
const usage = asRecord(usageValue);
|
||||
if (usage.deprecated === true) continue;
|
||||
const usageType = String(usage.usageType || "");
|
||||
const mediaType = String(media.mediaType || "");
|
||||
if (!usageType || !mediaType) continue;
|
||||
referenceInputs.push({
|
||||
mediaType,
|
||||
usageType,
|
||||
minItems: finiteInteger(usage.minItems) ?? 0,
|
||||
maxItems: finiteInteger(usage.maxItems),
|
||||
maxFileSizeBytes,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
for (const r of rows) {
|
||||
if (r.modality !== "image" && r.modality !== "video") continue;
|
||||
const id = slugifyAdobeModel(r.modelId, r.modelVersion);
|
||||
if (seen.has(id)) continue;
|
||||
// Skip if already covered by a friendly alias with same upstream
|
||||
if (
|
||||
out.some(
|
||||
(o) =>
|
||||
o.upstreamModelId === r.modelId && o.upstreamModelVersion === r.modelVersion
|
||||
const supportedSizes = [
|
||||
...new Set(
|
||||
schemaBranches(schema.properties.size)
|
||||
.flatMap((branch) => (Array.isArray(branch.enum) ? branch.enum : []))
|
||||
.map(asRecord)
|
||||
.filter((size) => finiteInteger(size.width) !== null && finiteInteger(size.height) !== null)
|
||||
.map((size) => `${size.width}x${size.height}`)
|
||||
),
|
||||
];
|
||||
const supportedAspectRatios = [
|
||||
...new Set(
|
||||
schemaBranches(schema.properties.generationSettings).flatMap((branch) =>
|
||||
enumStrings(asRecord(asRecord(branch.properties).aspectRatio))
|
||||
)
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
seen.add(id);
|
||||
out.push({
|
||||
id,
|
||||
name: r.displayName || id,
|
||||
modality: r.modality,
|
||||
upstreamModelId: r.modelId,
|
||||
upstreamModelVersion: r.modelVersion,
|
||||
inputModalities: r.modality === "image" ? ["text", "image"] : ["text"],
|
||||
});
|
||||
}
|
||||
|
||||
return out;
|
||||
}
|
||||
|
||||
export function getAdobeFireflyFallbackCatalog(modality?: "image" | "video"): AdobeFireflyCatalogModel[] {
|
||||
if (!modality) return [...ADOBE_FIREFLY_FALLBACK_MODELS];
|
||||
return ADOBE_FIREFLY_FALLBACK_MODELS.filter((m) => m.modality === modality);
|
||||
}
|
||||
|
||||
/**
|
||||
* Live discovery when credentials resolve; otherwise static fallback from get_models capture.
|
||||
*/
|
||||
export async function resolveAdobeFireflyCatalog(opts: {
|
||||
credentials?: {
|
||||
apiKey?: string;
|
||||
accessToken?: string;
|
||||
providerSpecificData?: Record<string, unknown> | null;
|
||||
} | null;
|
||||
modality?: "image" | "video";
|
||||
fetchImpl?: typeof fetch;
|
||||
}): Promise<{ models: AdobeFireflyCatalogModel[]; source: "api" | "fallback" }> {
|
||||
const fetchImpl = opts.fetchImpl || fetch;
|
||||
try {
|
||||
if (opts.credentials) {
|
||||
const token = await resolveAdobeAccessToken(opts.credentials, fetchImpl);
|
||||
const discovered = await discoverAdobeFireflyModels(token, fetchImpl);
|
||||
let catalog = mapDiscoveredToCatalog(discovered);
|
||||
if (opts.modality) catalog = catalog.filter((m) => m.modality === opts.modality);
|
||||
if (catalog.length > 0) return { models: catalog, source: "api" };
|
||||
}
|
||||
} catch {
|
||||
// fall through to static catalog
|
||||
}
|
||||
),
|
||||
];
|
||||
const duration = integerBranch(schema.properties.duration);
|
||||
const outputCount = integerBranch(schema.properties.n);
|
||||
const prompt =
|
||||
schemaBranches(schema.properties.prompt).find((branch) => branch.type === "string") || {};
|
||||
|
||||
return {
|
||||
models: getAdobeFireflyFallbackCatalog(opts.modality),
|
||||
source: "fallback",
|
||||
inputMediaUseCases: [...row.inputMediaUseCases],
|
||||
schemaProperties: Object.keys(schema.properties),
|
||||
requiredProperties: [...schema.required],
|
||||
referenceInputs,
|
||||
maxReferenceItems: finiteInteger(referenceSchema.maxItems),
|
||||
supportedSizes,
|
||||
supportedAspectRatios,
|
||||
supportedResolutions: enumStrings(schema.properties.resolution),
|
||||
supportedDurations: [
|
||||
...new Set(
|
||||
schemaBranches(schema.properties.duration)
|
||||
.flatMap((branch) => (Array.isArray(branch.enum) ? branch.enum : []))
|
||||
.filter((value): value is number => Number.isInteger(value))
|
||||
),
|
||||
],
|
||||
durationMin: finiteInteger(duration.minimum),
|
||||
durationMax: finiteInteger(duration.maximum),
|
||||
durationDefault: finiteInteger(duration.default),
|
||||
outputCountMin: finiteInteger(outputCount.minimum),
|
||||
outputCountMax: finiteInteger(outputCount.maximum),
|
||||
promptMaxLength: finiteInteger(prompt.maxLength),
|
||||
releaseReadiness: row.releaseReadiness || "",
|
||||
healthStatus: row.healthStatus || "",
|
||||
};
|
||||
}
|
||||
|
||||
/** Registry-shaped models for imageRegistry / videoRegistry. */
|
||||
export function toRegistryImageModels(
|
||||
models: AdobeFireflyCatalogModel[] = getAdobeFireflyFallbackCatalog("image")
|
||||
): Array<{ id: string; name: string; inputModalities?: string[] }> {
|
||||
return models
|
||||
.filter((m) => m.modality === "image")
|
||||
.map((m) => ({
|
||||
id: m.id,
|
||||
name: m.name.startsWith("Firefly ") ? m.name : `Firefly ${m.name}`,
|
||||
inputModalities: m.inputModalities || ["text", "image"],
|
||||
}));
|
||||
function isCallableGenerationModel(row: AdobeFireflyDiscoveredModel): boolean {
|
||||
if (row.modality !== "image" && row.modality !== "video") return false;
|
||||
if (!mergeAdobeObjectSchema(row.requestSchema).properties.prompt) return false;
|
||||
const excluded = new Set(["upscaling", "sharpening", "denoising"]);
|
||||
return !row.inputMediaUseCases.some((value) => excluded.has(value.toLowerCase()));
|
||||
}
|
||||
|
||||
export function toRegistryVideoModels(
|
||||
models: AdobeFireflyCatalogModel[] = getAdobeFireflyFallbackCatalog("video")
|
||||
): Array<{ id: string; name: string }> {
|
||||
return models
|
||||
.filter((m) => m.modality === "video")
|
||||
.map((m) => ({
|
||||
id: m.id,
|
||||
name: m.name.startsWith("Firefly ") ? m.name : `Firefly ${m.name}`,
|
||||
}));
|
||||
function deriveInputModalities(capabilities: AdobeFireflyMediaCapabilities): string[] {
|
||||
return ["text", ...new Set(capabilities.referenceInputs.map((reference) => reference.mediaType))];
|
||||
}
|
||||
|
||||
function semanticCatalogKey(model: AdobeFireflyCatalogModel): string {
|
||||
return JSON.stringify({
|
||||
backingModel: model.backingModel,
|
||||
name: model.name,
|
||||
modality: model.modality,
|
||||
capabilities: model.capabilities,
|
||||
});
|
||||
}
|
||||
|
||||
/** Normalize and de-duplicate callable image/video rows from live discovery. */
|
||||
export function mapDiscoveredToCatalog(
|
||||
rows: AdobeFireflyDiscoveredModel[]
|
||||
): AdobeFireflyCatalogModel[] {
|
||||
const output: AdobeFireflyCatalogModel[] = [];
|
||||
const seen = new Set<string>();
|
||||
for (const row of rows) {
|
||||
if (!isCallableGenerationModel(row)) continue;
|
||||
const capabilities = normalizeCapabilities(row);
|
||||
const model: AdobeFireflyCatalogModel = {
|
||||
id: slugifyAdobeModel(row.modelId, row.modelVersion),
|
||||
name: row.displayName,
|
||||
modality: row.modality as "image" | "video",
|
||||
upstreamModelId: row.modelId,
|
||||
upstreamModelVersion: row.modelVersion,
|
||||
providerName: row.providerName || "",
|
||||
backingModel: row.backingModel || "",
|
||||
inputModalities: deriveInputModalities(capabilities),
|
||||
capabilities,
|
||||
};
|
||||
const key = semanticCatalogKey(model);
|
||||
if (seen.has(key)) continue;
|
||||
seen.add(key);
|
||||
output.push(model);
|
||||
}
|
||||
return output;
|
||||
}
|
||||
|
||||
function snapshotCatalog(): AdobeFireflyCatalogModel[] {
|
||||
return ADOBE_FIREFLY_DISCOVERY_SNAPSHOT.map((model) => {
|
||||
const capabilities: AdobeFireflyMediaCapabilities = {
|
||||
inputMediaUseCases: [...model.inputMediaUseCases],
|
||||
schemaProperties: [...model.schemaProperties],
|
||||
requiredProperties: [...model.requiredProperties],
|
||||
referenceInputs: model.referenceInputs.map((reference) => ({ ...reference })),
|
||||
maxReferenceItems: model.maxReferenceItems,
|
||||
supportedSizes: [...model.supportedSizes],
|
||||
supportedAspectRatios: [...model.supportedAspectRatios],
|
||||
supportedResolutions: [...model.supportedResolutions],
|
||||
supportedDurations: [...model.supportedDurations],
|
||||
durationMin: model.durationMin,
|
||||
durationMax: model.durationMax,
|
||||
durationDefault: model.durationDefault,
|
||||
outputCountMin: model.outputCountMin,
|
||||
outputCountMax: model.outputCountMax,
|
||||
promptMaxLength: model.promptMaxLength,
|
||||
releaseReadiness: model.releaseReadiness,
|
||||
healthStatus: model.healthStatus,
|
||||
};
|
||||
return {
|
||||
id: model.id,
|
||||
name: model.name,
|
||||
modality: model.modality,
|
||||
upstreamModelId: model.upstreamModelId,
|
||||
upstreamModelVersion: model.upstreamModelVersion,
|
||||
providerName: model.providerName,
|
||||
backingModel: model.backingModel,
|
||||
inputModalities: deriveInputModalities(capabilities),
|
||||
capabilities,
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
export const ADOBE_FIREFLY_FALLBACK_MODELS: AdobeFireflyCatalogModel[] = snapshotCatalog();
|
||||
|
||||
export function getAdobeFireflyFallbackCatalog(
|
||||
modality?: "image" | "video"
|
||||
): AdobeFireflyCatalogModel[] {
|
||||
return ADOBE_FIREFLY_FALLBACK_MODELS.filter((model) => !modality || model.modality === modality);
|
||||
}
|
||||
|
||||
function imageFamily(model: AdobeFireflyCatalogModel): AdobeFireflyImageModelSpec["family"] {
|
||||
if (model.upstreamModelId === "gemini-flash") return "gemini";
|
||||
if (model.upstreamModelId === "gpt-image" || model.upstreamModelId === "gpt-4o-image") {
|
||||
return "gpt-image";
|
||||
}
|
||||
return "generic";
|
||||
}
|
||||
|
||||
export const ADOBE_FIREFLY_IMAGE_MODELS: Record<string, AdobeFireflyImageModelSpec> =
|
||||
Object.fromEntries(
|
||||
getAdobeFireflyFallbackCatalog("image").map((model) => [
|
||||
model.id,
|
||||
{ ...model, modality: "image" as const, family: imageFamily(model) },
|
||||
])
|
||||
);
|
||||
|
||||
function defaultDuration(model: AdobeFireflyCatalogModel): number {
|
||||
const caps = model.capabilities;
|
||||
return caps.durationDefault ?? caps.supportedDurations[0] ?? caps.durationMin ?? 5;
|
||||
}
|
||||
|
||||
function defaultResolution(model: AdobeFireflyCatalogModel): string {
|
||||
if (model.capabilities.supportedSizes.some((value) => value.includes("1920x1080"))) {
|
||||
return "1080p";
|
||||
}
|
||||
return "720p";
|
||||
}
|
||||
|
||||
export const ADOBE_FIREFLY_VIDEO_MODELS: Record<string, AdobeFireflyVideoModelSpec> =
|
||||
Object.fromEntries(
|
||||
getAdobeFireflyFallbackCatalog("video").map((model) => [
|
||||
model.id,
|
||||
{
|
||||
...model,
|
||||
modality: "video" as const,
|
||||
defaultDuration: defaultDuration(model),
|
||||
defaultResolution: defaultResolution(model),
|
||||
},
|
||||
])
|
||||
);
|
||||
|
||||
const LEGACY_MODEL_ALIASES: Record<string, string> = {
|
||||
"nano-banana": "gemini-flash-nano-banana",
|
||||
"nano-banana-pro": "gemini-flash-nano-banana-2",
|
||||
"nano-banana-2": "gemini-flash-nano-banana-3",
|
||||
"gpt-image": "gpt-image-2",
|
||||
"gpt-image-2": "gpt-image-2",
|
||||
"gpt-image-1.5": "gpt-image-1.5",
|
||||
"flux-2": "flux-2",
|
||||
"flux-pro": "flux-fluxpro",
|
||||
"flux-ultra": "flux-fluxultra",
|
||||
"seedream-4": "seedream-seedream-v4",
|
||||
"seedream-5-lite": "seedream-seedream-v5-lite",
|
||||
"runway-gen4-image": "runway-gen4-image",
|
||||
"veo-3.1": "veo-3.1-generate",
|
||||
"veo-3.1-fast": "veo-3.1-fast-generate",
|
||||
"luma-ray3": "luma-3.0-ray",
|
||||
"runway-gen4-turbo": "runway-gen4-turbo",
|
||||
// Backward compatibility only; the catalog advertises the exact discovered id.
|
||||
"kling-3": "kling-kling-v3-standard-i2v",
|
||||
};
|
||||
|
||||
// Preserve established API aliases when (and only when) they resolve to a model
|
||||
// that is present in the verified discovery snapshot. These keys are not listed.
|
||||
for (const [alias, target] of Object.entries(LEGACY_MODEL_ALIASES)) {
|
||||
const imageTarget = ADOBE_FIREFLY_IMAGE_MODELS[target];
|
||||
if (imageTarget) ADOBE_FIREFLY_IMAGE_MODELS[alias] = imageTarget;
|
||||
const videoTarget = ADOBE_FIREFLY_VIDEO_MODELS[target];
|
||||
if (videoTarget) ADOBE_FIREFLY_VIDEO_MODELS[alias] = videoTarget;
|
||||
}
|
||||
|
||||
/** Backward-compatible request ids. Kept out of every advertised model catalog. */
|
||||
export const ADOBE_FIREFLY_IMAGE_ROUTING_ALIASES = Object.freeze(
|
||||
Object.entries(LEGACY_MODEL_ALIASES)
|
||||
.filter(([, target]) => Boolean(ADOBE_FIREFLY_IMAGE_MODELS[target]))
|
||||
.map(([alias]) => alias)
|
||||
);
|
||||
|
||||
function normalizeRequestedId(model: string): string {
|
||||
return String(model || "")
|
||||
.trim()
|
||||
.toLowerCase()
|
||||
.replace(/^adobe-firefly\//, "")
|
||||
.replace(/^firefly\//, "");
|
||||
}
|
||||
|
||||
function resolveCatalogId(model: string): string {
|
||||
const requested = normalizeRequestedId(model);
|
||||
return LEGACY_MODEL_ALIASES[requested] || requested;
|
||||
}
|
||||
|
||||
export function resolveAdobeImageModel(model: string): {
|
||||
id: string;
|
||||
spec: AdobeFireflyImageModelSpec;
|
||||
} {
|
||||
const id = resolveCatalogId(model);
|
||||
const spec = ADOBE_FIREFLY_IMAGE_MODELS[id];
|
||||
if (!spec) {
|
||||
throw new Error(
|
||||
`Unknown Adobe Firefly image model: ${normalizeRequestedId(model) || "(empty)"}`
|
||||
);
|
||||
}
|
||||
return { id, spec };
|
||||
}
|
||||
|
||||
export function resolveAdobeVideoModel(model: string): {
|
||||
id: string;
|
||||
spec: AdobeFireflyVideoModelSpec;
|
||||
} {
|
||||
const id = resolveCatalogId(model);
|
||||
const spec = ADOBE_FIREFLY_VIDEO_MODELS[id];
|
||||
if (!spec) {
|
||||
throw new Error(
|
||||
`Unknown Adobe Firefly video model: ${normalizeRequestedId(model) || "(empty)"}`
|
||||
);
|
||||
}
|
||||
return { id, spec };
|
||||
}
|
||||
|
||||
export function toRegistryImageModels(): Array<{
|
||||
id: string;
|
||||
name: string;
|
||||
inputModalities: string[];
|
||||
imageRequired?: boolean;
|
||||
supportedSizes: string[];
|
||||
mediaCapabilities: Record<string, unknown>;
|
||||
}> {
|
||||
const generated = getAdobeFireflyFallbackCatalog("image").map((model) => ({
|
||||
id: model.id,
|
||||
name: `Firefly ${model.name}`,
|
||||
inputModalities: model.inputModalities,
|
||||
supportedSizes: model.capabilities.supportedSizes,
|
||||
mediaCapabilities: toAdobeMediaCapabilitiesApi(model),
|
||||
}));
|
||||
// Upscaling uses a distinct Firefly endpoint and is not returned by the image
|
||||
// generation discovery schema. Keep its two supported Topaz models visible in
|
||||
// the same provider catalog so image clients can select them deliberately.
|
||||
return [
|
||||
...generated,
|
||||
{
|
||||
id: "topaz-standard",
|
||||
name: "Firefly Topaz Upscale (Standard)",
|
||||
inputModalities: ["image"],
|
||||
imageRequired: true,
|
||||
supportedSizes: [],
|
||||
mediaCapabilities: { input_media_use_cases: ["upscaling"] },
|
||||
},
|
||||
{
|
||||
id: "topaz-bloom",
|
||||
name: "Firefly Topaz Bloom (Creative Upscale)",
|
||||
inputModalities: ["image"],
|
||||
imageRequired: true,
|
||||
supportedSizes: [],
|
||||
mediaCapabilities: { input_media_use_cases: ["upscaling"] },
|
||||
},
|
||||
];
|
||||
}
|
||||
|
||||
export function toRegistryVideoModels(): Array<{
|
||||
id: string;
|
||||
name: string;
|
||||
supportedSizes: string[];
|
||||
mediaCapabilities: Record<string, unknown>;
|
||||
}> {
|
||||
return getAdobeFireflyFallbackCatalog("video").map((model) => ({
|
||||
id: model.id,
|
||||
name: `Firefly ${model.name}`,
|
||||
supportedSizes: model.capabilities.supportedSizes,
|
||||
mediaCapabilities: toAdobeMediaCapabilitiesApi(model),
|
||||
}));
|
||||
}
|
||||
|
||||
/** JSON-safe extension emitted by /v1/models. */
|
||||
export function toAdobeMediaCapabilitiesApi(
|
||||
model: AdobeFireflyCatalogModel
|
||||
): Record<string, unknown> {
|
||||
const caps = model.capabilities;
|
||||
return {
|
||||
upstream_model_id: model.upstreamModelId,
|
||||
upstream_model_version: model.upstreamModelVersion,
|
||||
provider_name: model.providerName,
|
||||
release_readiness: caps.releaseReadiness,
|
||||
health_status: caps.healthStatus,
|
||||
input_media_use_cases: caps.inputMediaUseCases,
|
||||
reference_inputs: caps.referenceInputs.map((reference) => ({
|
||||
media_type: reference.mediaType,
|
||||
usage_type: reference.usageType,
|
||||
min_items: reference.minItems,
|
||||
max_items: reference.maxItems,
|
||||
max_file_size_bytes: reference.maxFileSizeBytes,
|
||||
})),
|
||||
max_reference_items: caps.maxReferenceItems,
|
||||
supported_sizes: caps.supportedSizes,
|
||||
supported_aspect_ratios: caps.supportedAspectRatios,
|
||||
supported_resolutions: caps.supportedResolutions,
|
||||
supported_durations: caps.supportedDurations,
|
||||
duration_min: caps.durationMin,
|
||||
duration_max: caps.durationMax,
|
||||
duration_default: caps.durationDefault,
|
||||
output_count_min: caps.outputCountMin,
|
||||
output_count_max: caps.outputCountMax,
|
||||
prompt_max_length: caps.promptMaxLength,
|
||||
};
|
||||
}
|
||||
|
||||
export function getAdobeReferenceUploadLimit(
|
||||
model: AdobeFireflyCatalogModel,
|
||||
mediaType: string
|
||||
): number {
|
||||
if (model.capabilities.maxReferenceItems !== null) {
|
||||
return Math.max(1, Math.min(32, model.capabilities.maxReferenceItems));
|
||||
}
|
||||
const declaredTotal = model.capabilities.referenceInputs
|
||||
.filter((reference) => reference.mediaType === mediaType)
|
||||
.reduce((total, reference) => total + (reference.maxItems ?? 0), 0);
|
||||
return Math.max(1, Math.min(32, declaredTotal || 1));
|
||||
}
|
||||
|
||||
207
scripts/dev/generate-adobe-firefly-snapshot.mjs
Normal file
207
scripts/dev/generate-adobe-firefly-snapshot.mjs
Normal file
@@ -0,0 +1,207 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { createHash } from "node:crypto";
|
||||
|
||||
function usage() {
|
||||
console.error(
|
||||
"Usage: node scripts/dev/generate-adobe-firefly-snapshot.mjs <discovery.json> <output.ts>"
|
||||
);
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const [, , inputArg, outputArg] = process.argv;
|
||||
if (!inputArg || !outputArg) usage();
|
||||
|
||||
const inputPath = path.resolve(inputArg);
|
||||
const outputPath = path.resolve(outputArg);
|
||||
const inputBytes = fs.readFileSync(inputPath);
|
||||
const sourceHash = createHash("sha256").update(inputBytes).digest("hex");
|
||||
const root = JSON.parse(inputBytes.toString("utf8"));
|
||||
|
||||
function mergeObjectSchema(schema) {
|
||||
const merged = { properties: {}, required: [] };
|
||||
const visit = (node) => {
|
||||
if (!node || typeof node !== "object") return;
|
||||
if (node.properties && typeof node.properties === "object") {
|
||||
Object.assign(merged.properties, node.properties);
|
||||
}
|
||||
if (Array.isArray(node.required)) merged.required.push(...node.required);
|
||||
if (Array.isArray(node.allOf)) node.allOf.forEach(visit);
|
||||
};
|
||||
visit(schema);
|
||||
merged.required = [...new Set(merged.required)];
|
||||
return merged;
|
||||
}
|
||||
|
||||
function branches(schema) {
|
||||
if (!schema || typeof schema !== "object") return [];
|
||||
return [schema, ...(schema.anyOf || []), ...(schema.oneOf || [])];
|
||||
}
|
||||
|
||||
function stringEnums(schema) {
|
||||
return [
|
||||
...new Set(
|
||||
branches(schema)
|
||||
.flatMap((branch) => (Array.isArray(branch.enum) ? branch.enum : []))
|
||||
.filter((value) => typeof value === "string")
|
||||
),
|
||||
];
|
||||
}
|
||||
|
||||
function integerSchema(schema) {
|
||||
return branches(schema).find((branch) => branch.type === "integer") || {};
|
||||
}
|
||||
|
||||
function publicModelId(modelId, modelVersion) {
|
||||
const slug = (value, allowDot = false) =>
|
||||
String(value || "")
|
||||
.trim()
|
||||
.toLowerCase()
|
||||
.replace(allowDot ? /[^a-z0-9.]+/g : /[^a-z0-9]+/g, "-")
|
||||
.replace(/^-|-$/g, "");
|
||||
const family = slug(modelId);
|
||||
const publicVersion =
|
||||
family === "kling" ? String(modelVersion).replace(/^kling_v3_omni/i, "kling_o3") : modelVersion;
|
||||
const version = slug(publicVersion, true);
|
||||
if (!version || version === "default" || version === family) return family || "model";
|
||||
return `${family}-${version}`;
|
||||
}
|
||||
|
||||
function normalizeModel(family, modelVersion, version) {
|
||||
const schema = mergeObjectSchema(version.requestSchema);
|
||||
const properties = schema.properties;
|
||||
const referenceSchema = properties.referenceBlobs || {};
|
||||
const referenceInputs = [];
|
||||
for (const media of referenceSchema["x-capabilities"] || []) {
|
||||
for (const usage of media.usageConstraints || []) {
|
||||
if (usage.deprecated === true) continue;
|
||||
referenceInputs.push({
|
||||
mediaType: String(media.mediaType || ""),
|
||||
usageType: String(usage.usageType || ""),
|
||||
minItems: Number.isInteger(usage.minItems) ? usage.minItems : 0,
|
||||
maxItems: Number.isInteger(usage.maxItems) ? usage.maxItems : null,
|
||||
maxFileSizeBytes: Number.isInteger(media.maxFileSizeBytes) ? media.maxFileSizeBytes : null,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
const supportedSizes = [
|
||||
...new Set(
|
||||
branches(properties.size)
|
||||
.flatMap((branch) => (Array.isArray(branch.enum) ? branch.enum : []))
|
||||
.filter(
|
||||
(size) =>
|
||||
size &&
|
||||
Number.isInteger(size.width) &&
|
||||
size.width > 0 &&
|
||||
Number.isInteger(size.height) &&
|
||||
size.height > 0
|
||||
)
|
||||
.map((size) => `${size.width}x${size.height}`)
|
||||
),
|
||||
];
|
||||
const supportedAspectRatios = [
|
||||
...new Set(
|
||||
branches(properties.generationSettings).flatMap((branch) =>
|
||||
stringEnums(branch?.properties?.aspectRatio)
|
||||
)
|
||||
),
|
||||
];
|
||||
const duration = integerSchema(properties.duration);
|
||||
const supportedDurations = [
|
||||
...new Set(
|
||||
branches(properties.duration)
|
||||
.flatMap((branch) => (Array.isArray(branch.enum) ? branch.enum : []))
|
||||
.filter(Number.isInteger)
|
||||
),
|
||||
];
|
||||
const prompt = branches(properties.prompt).find((branch) => branch.type === "string") || {};
|
||||
const outputCount = integerSchema(properties.n);
|
||||
|
||||
return {
|
||||
id: publicModelId(family.modelId, modelVersion),
|
||||
name: String(version.modelDisplayName || version.modelCaiDisplayName || modelVersion),
|
||||
modality: version.outputModality[0],
|
||||
upstreamModelId: family.modelId,
|
||||
upstreamModelVersion: modelVersion,
|
||||
providerName: String(family.acModelFamilyProviderDisplayName || ""),
|
||||
releaseReadiness: String(version.releaseReadiness || ""),
|
||||
healthStatus: String(version.healthStatus || ""),
|
||||
inputMediaUseCases: (version.inputMediaUseCase || []).map(String),
|
||||
schemaProperties: Object.keys(properties),
|
||||
requiredProperties: schema.required,
|
||||
referenceInputs,
|
||||
maxReferenceItems: Number.isInteger(referenceSchema.maxItems) ? referenceSchema.maxItems : null,
|
||||
supportedSizes,
|
||||
supportedAspectRatios,
|
||||
supportedResolutions: stringEnums(properties.resolution),
|
||||
supportedDurations,
|
||||
durationMin: Number.isInteger(duration.minimum) ? duration.minimum : null,
|
||||
durationMax: Number.isInteger(duration.maximum) ? duration.maximum : null,
|
||||
durationDefault: Number.isInteger(duration.default) ? duration.default : null,
|
||||
outputCountMin: Number.isInteger(outputCount.minimum) ? outputCount.minimum : null,
|
||||
outputCountMax: Number.isInteger(outputCount.maximum) ? outputCount.maximum : null,
|
||||
promptMaxLength: Number.isInteger(prompt.maxLength) ? prompt.maxLength : null,
|
||||
backingModel: String(version.bksGenerationModel || ""),
|
||||
};
|
||||
}
|
||||
|
||||
const rawModels = [];
|
||||
for (const family of Array.isArray(root.models) ? root.models : []) {
|
||||
for (const [modelVersion, version] of Object.entries(family.modelVersions || {})) {
|
||||
if (!version || version.enabled === false) continue;
|
||||
const modality = Array.isArray(version.outputModality)
|
||||
? version.outputModality.map((value) => String(value).toLowerCase())[0]
|
||||
: "";
|
||||
if (modality !== "image" && modality !== "video") continue;
|
||||
|
||||
const schema = mergeObjectSchema(version.requestSchema);
|
||||
if (!schema.properties.prompt) continue;
|
||||
const useCases = (version.inputMediaUseCase || []).map((value) => String(value).toLowerCase());
|
||||
if (useCases.some((value) => ["upscaling", "sharpening", "denoising"].includes(value))) {
|
||||
continue;
|
||||
}
|
||||
rawModels.push(normalizeModel(family, modelVersion, version));
|
||||
}
|
||||
}
|
||||
|
||||
// Discovery currently repeats a few exact aliases (for example flux/fluxPro and
|
||||
// fluxPro/1.1). Keep the first canonical wire pair and suppress duplicate cards.
|
||||
const seen = new Set();
|
||||
const models = [];
|
||||
for (const model of rawModels) {
|
||||
const semanticKey = JSON.stringify({
|
||||
backingModel: model.backingModel,
|
||||
name: model.name,
|
||||
modality: model.modality,
|
||||
schemaProperties: model.schemaProperties,
|
||||
requiredProperties: model.requiredProperties,
|
||||
referenceInputs: model.referenceInputs,
|
||||
maxReferenceItems: model.maxReferenceItems,
|
||||
supportedSizes: model.supportedSizes,
|
||||
supportedAspectRatios: model.supportedAspectRatios,
|
||||
supportedResolutions: model.supportedResolutions,
|
||||
supportedDurations: model.supportedDurations,
|
||||
durationMin: model.durationMin,
|
||||
durationMax: model.durationMax,
|
||||
});
|
||||
if (seen.has(semanticKey)) continue;
|
||||
seen.add(semanticKey);
|
||||
models.push(model);
|
||||
}
|
||||
|
||||
const source = `/**
|
||||
* Generated from Adobe Firefly POST /v2/models/discovery with resolveSchema=true.
|
||||
* Source SHA-256: ${sourceHash}
|
||||
* Regenerate with scripts/dev/generate-adobe-firefly-snapshot.mjs; do not edit by hand.
|
||||
* The generated literal stays compact to satisfy the repository's line-count gate.
|
||||
*/
|
||||
// prettier-ignore
|
||||
export const ADOBE_FIREFLY_DISCOVERY_SNAPSHOT = ${JSON.stringify(models)} as const;
|
||||
`;
|
||||
|
||||
fs.mkdirSync(path.dirname(outputPath), { recursive: true });
|
||||
fs.writeFileSync(outputPath, source, "utf8");
|
||||
console.log(`Wrote ${models.length} models to ${outputPath}`);
|
||||
@@ -146,6 +146,8 @@ export async function POST(request) {
|
||||
max_output_tokens: maxOutputTokens,
|
||||
// #1904: manual vision-capability override set in the add-model form.
|
||||
supportsVision,
|
||||
// #9820: optional video-generation job preset (job/poll path).
|
||||
generationConfig,
|
||||
} = validation.data;
|
||||
|
||||
const model = await addCustomModel(
|
||||
@@ -160,7 +162,8 @@ export async function POST(request) {
|
||||
...(maxInputTokens != null ? { inputTokenLimit: maxInputTokens } : {}),
|
||||
...(maxOutputTokens != null ? { outputTokenLimit: maxOutputTokens } : {}),
|
||||
},
|
||||
typeof supportsVision === "boolean" ? supportsVision : undefined
|
||||
typeof supportsVision === "boolean" ? supportsVision : undefined,
|
||||
generationConfig
|
||||
);
|
||||
return Response.json({ model });
|
||||
} catch (error) {
|
||||
@@ -213,6 +216,7 @@ export async function PUT(request) {
|
||||
compatByProtocol,
|
||||
contextWindowOverride,
|
||||
supportsVision,
|
||||
generationConfig,
|
||||
} = validation.data;
|
||||
|
||||
const raw = rawBody as Record<string, unknown>;
|
||||
@@ -227,6 +231,11 @@ export async function PUT(request) {
|
||||
if ("upstreamHeaders" in raw) updates.upstreamHeaders = upstreamHeaders;
|
||||
// #1904: manual vision-capability override — null clears back to heuristic.
|
||||
if ("supportsVision" in raw) updates.supportsVision = supportsVision;
|
||||
// #9820: video-generation job preset — schema is non-nullable optional, so
|
||||
// presence implies a well-formed { preset } object; null is rejected by Zod.
|
||||
if ("generationConfig" in raw && generationConfig !== undefined) {
|
||||
updates.generationConfig = generationConfig;
|
||||
}
|
||||
if ("compatByProtocol" in raw && compatByProtocol !== undefined) {
|
||||
updates.compatByProtocol = compatByProtocol;
|
||||
}
|
||||
|
||||
73
src/app/api/providers/[id]/models/adobeFireflyDiscovery.ts
Normal file
73
src/app/api/providers/[id]/models/adobeFireflyDiscovery.ts
Normal file
@@ -0,0 +1,73 @@
|
||||
import {
|
||||
discoverAdobeFireflyModels,
|
||||
resolveAdobeAccessToken,
|
||||
} from "@omniroute/open-sse/services/adobeFireflyClient.ts";
|
||||
import {
|
||||
getAdobeFireflyFallbackCatalog,
|
||||
mapDiscoveredToCatalog,
|
||||
toAdobeMediaCapabilitiesApi,
|
||||
type AdobeFireflyCatalogModel,
|
||||
} from "@omniroute/open-sse/services/adobeFireflyModels.ts";
|
||||
import { sanitizeErrorMessage } from "@omniroute/open-sse/utils/error";
|
||||
|
||||
type AdobeProviderData = { cookie?: unknown; access_token?: unknown; accessToken?: unknown };
|
||||
|
||||
interface AdobeProviderModelsResult {
|
||||
models: Array<Record<string, unknown>>;
|
||||
source: "api" | "local_catalog";
|
||||
warning?: string;
|
||||
}
|
||||
|
||||
function toModelResponse(model: AdobeFireflyCatalogModel): Record<string, unknown> {
|
||||
const endpoint = model.modality === "image" ? "images" : "videos";
|
||||
return {
|
||||
id: model.id,
|
||||
name: model.name,
|
||||
owned_by: "adobe-firefly",
|
||||
apiFormat: endpoint,
|
||||
supportedEndpoints: [endpoint],
|
||||
type: model.modality,
|
||||
input_modalities: model.inputModalities,
|
||||
output_modalities: [model.modality],
|
||||
supported_sizes: model.capabilities.supportedSizes,
|
||||
media_capabilities: toAdobeMediaCapabilitiesApi(model),
|
||||
};
|
||||
}
|
||||
|
||||
function fallback(warning: string): AdobeProviderModelsResult {
|
||||
return {
|
||||
models: getAdobeFireflyFallbackCatalog().map(toModelResponse),
|
||||
source: "local_catalog",
|
||||
warning,
|
||||
};
|
||||
}
|
||||
|
||||
export async function getAdobeModels(
|
||||
apiKey: string | undefined,
|
||||
accessToken: string | undefined,
|
||||
providerData: unknown,
|
||||
fetchImpl: typeof fetch = fetch
|
||||
): Promise<AdobeProviderModelsResult> {
|
||||
const providerSpecificData =
|
||||
providerData && typeof providerData === "object" ? (providerData as AdobeProviderData) : {};
|
||||
try {
|
||||
const token = await resolveAdobeAccessToken(
|
||||
{
|
||||
apiKey,
|
||||
accessToken,
|
||||
providerSpecificData,
|
||||
},
|
||||
fetchImpl
|
||||
);
|
||||
const models = mapDiscoveredToCatalog(await discoverAdobeFireflyModels(token, fetchImpl));
|
||||
return models.length > 0
|
||||
? { models: models.map(toModelResponse), source: "api" }
|
||||
: fallback("Adobe Firefly discovery returned no callable image or video models");
|
||||
} catch (error) {
|
||||
return fallback(
|
||||
`Adobe Firefly discovery unavailable: ${sanitizeErrorMessage(
|
||||
error instanceof Error ? error.message : String(error)
|
||||
)}`
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -84,10 +84,8 @@ import {
|
||||
isAutoFetchModelsEnabled,
|
||||
persistDiscoveredModels,
|
||||
} from "@/lib/providerModels/modelDiscovery";
|
||||
import {
|
||||
buildProviderModelsUrl,
|
||||
getDiscoveryClientVersionOptions,
|
||||
} from "./discoveryClientVersion";
|
||||
import { buildProviderModelsUrl, getDiscoveryClientVersionOptions } from "./discoveryClientVersion";
|
||||
import { getAdobeModels } from "./adobeFireflyDiscovery";
|
||||
import {
|
||||
parseGeminiModelsList,
|
||||
type GeminiDiscoveryModel,
|
||||
@@ -422,10 +420,7 @@ export async function GET(
|
||||
// #6267 — a models-endpoint redirect (307/308) is not a fixable-config
|
||||
// error. safeOutboundFetch throws REDIRECT_BLOCKED which
|
||||
// getSafeOutboundFetchErrorStatus maps to 503, but unlike the other 503
|
||||
// cases (URL_GUARD_BLOCKED / INVALID_URL, which are genuinely
|
||||
// unrecoverable and stay hard errors) a blocked redirect should degrade to
|
||||
// the local/cached catalog OmniRoute ships instead of surfacing a raw 503.
|
||||
// General fix — covers any config-driven provider that 307s (e.g. qwen-web).
|
||||
// Redirect blocks degrade to the local/cached catalog; invalid URLs remain hard errors.
|
||||
if (error instanceof SafeOutboundFetchError && error.code === "REDIRECT_BLOCKED") {
|
||||
return buildDiscoveryFallbackResponse(warnings);
|
||||
}
|
||||
@@ -434,6 +429,11 @@ export async function GET(
|
||||
return buildDiscoveryFallbackResponse(warnings);
|
||||
};
|
||||
|
||||
if (provider === "adobe-firefly") {
|
||||
const discovery = await getAdobeModels(apiKey, accessToken, connection.providerSpecificData);
|
||||
return buildResponse({ provider, connectionId, ...discovery });
|
||||
}
|
||||
|
||||
const maybeReturnCachedDiscovery = () => {
|
||||
if (!refresh && cachedDiscoveryModels.length > 0) {
|
||||
return buildCachedDiscoveryResponse();
|
||||
|
||||
@@ -1113,6 +1113,7 @@ async function buildUnifiedModelsResponseCore(
|
||||
input_modalities: imgModel.inputModalities || ["text"],
|
||||
output_modalities: ["image"],
|
||||
...(imgModel.description ? { description: imgModel.description } : {}),
|
||||
...(imgModel.mediaCapabilities ? { media_capabilities: imgModel.mediaCapabilities } : {}),
|
||||
});
|
||||
}
|
||||
|
||||
@@ -1178,6 +1179,12 @@ async function buildUnifiedModelsResponseCore(
|
||||
created: timestamp,
|
||||
owned_by: videoModel.provider,
|
||||
type: "video",
|
||||
supported_sizes: videoModel.supportedSizes,
|
||||
input_modalities: ["text"],
|
||||
output_modalities: ["video"],
|
||||
...(videoModel.mediaCapabilities
|
||||
? { media_capabilities: videoModel.mediaCapabilities }
|
||||
: {}),
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
@@ -1,87 +0,0 @@
|
||||
/**
|
||||
* Muse Code CLI proprietary model catalog endpoint.
|
||||
*
|
||||
* Muse CLI calls GET /muse-code/models (or --base-url/muse-code/models)
|
||||
* to discover available models. Returns the proprietary Muse format:
|
||||
*
|
||||
* { object: "list", data: [{ id, object, created, owned_by, metadata }] }
|
||||
*
|
||||
* Each model's metadata includes: name, family, reasoning, tool_call,
|
||||
* modalities, limit, cost.
|
||||
*/
|
||||
|
||||
import { muse_codeProvider } from "@omniroute/open-sse/config/providers/registry/muse-code/index.ts";
|
||||
|
||||
const MUSECODE_TIMESTAMP = Math.floor(Date.now() / 1000);
|
||||
|
||||
interface MuseCodeModel {
|
||||
id: string;
|
||||
object: "model";
|
||||
created: number;
|
||||
owned_by: string;
|
||||
metadata: {
|
||||
name: string;
|
||||
family: string;
|
||||
reasoning: boolean;
|
||||
tool_call: boolean;
|
||||
modalities: string[];
|
||||
limit: number;
|
||||
cost: number;
|
||||
};
|
||||
}
|
||||
|
||||
function buildModelCatalog(): MuseCodeModel[] {
|
||||
const data: MuseCodeModel[] = [];
|
||||
|
||||
for (const model of muse_codeProvider.models) {
|
||||
let family = "llama";
|
||||
if (model.id.includes("llama-4")) family = "llama-4";
|
||||
else if (model.id.includes("llama-3.3")) family = "llama-3.3";
|
||||
else if (model.id.includes("llama-3.2")) family = "llama-3.2";
|
||||
else if (model.id.includes("llama-3.1")) family = "llama-3.1";
|
||||
|
||||
const modalities: string[] = ["text"];
|
||||
if (model.supportsVision) modalities.push("image");
|
||||
|
||||
data.push({
|
||||
id: model.id,
|
||||
object: "model",
|
||||
created: MUSECODE_TIMESTAMP,
|
||||
owned_by: "meta",
|
||||
metadata: {
|
||||
name: model.name,
|
||||
family,
|
||||
reasoning: !!model.supportsReasoning,
|
||||
tool_call: !!model.toolCalling,
|
||||
modalities,
|
||||
limit: model.contextLength ?? 200_000,
|
||||
cost: model.id.includes("maverick") || model.id.includes("405b") ? 3 : 1,
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
return data;
|
||||
}
|
||||
|
||||
// Cache the catalog for the lifetime of the process — model list is static.
|
||||
const CATALOG = buildModelCatalog();
|
||||
const CATALOG_PAYLOAD = JSON.stringify({ object: "list", data: CATALOG }, null, 2);
|
||||
|
||||
export async function OPTIONS() {
|
||||
return new Response(null, {
|
||||
headers: {
|
||||
"Access-Control-Allow-Methods": "GET, OPTIONS",
|
||||
"Access-Control-Allow-Headers": "*",
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
export async function GET() {
|
||||
return new Response(CATALOG_PAYLOAD, {
|
||||
status: 200,
|
||||
headers: {
|
||||
"content-type": "application/json",
|
||||
"cache-control": "public, max-age=3600",
|
||||
},
|
||||
});
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
import { handleVideoGeneration } from "@omniroute/open-sse/handlers/videoGeneration.ts";
|
||||
import { resolveVideoCredentialProvider } from "@omniroute/open-sse/handlers/videoGeneration/googleFlow.ts";
|
||||
import { withInjectionGuard } from "@/middleware/promptInjectionGuard";
|
||||
import { getAllCustomModels } from "@/lib/db/models";
|
||||
import {
|
||||
getProviderCredentialsWithQuotaPreflight,
|
||||
clearRecoveredProviderState,
|
||||
@@ -88,7 +89,31 @@ async function postHandler(request, context) {
|
||||
if (policy.rejection) return policy.rejection;
|
||||
|
||||
// Parse model to get provider
|
||||
const { provider } = parsedModel;
|
||||
let { provider, model: requestedModel } = parsedModel;
|
||||
let isCustomModel = false;
|
||||
if (!provider) {
|
||||
// Custom OpenAI-compatible provider nodes (mirrors images route): scan the
|
||||
// dynamic model registry for a matching `${nodeId}/${modelId}` entry.
|
||||
try {
|
||||
const customModelsMap = (await getAllCustomModels()) as Record<string, any>;
|
||||
for (const [providerId, models] of Object.entries(customModelsMap)) {
|
||||
if (!Array.isArray(models)) continue;
|
||||
for (const model of models) {
|
||||
if (!model?.id || !Array.isArray(model.supportedEndpoints)) continue;
|
||||
if (!model.supportedEndpoints.includes("videos")) continue;
|
||||
const fullId = `${providerId}/${model.id}`;
|
||||
if (fullId === body.model) {
|
||||
provider = providerId;
|
||||
requestedModel = model.id;
|
||||
isCustomModel = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// registry read failure — fall through to invalid-model error below
|
||||
}
|
||||
}
|
||||
if (!provider) {
|
||||
return errorResponse(
|
||||
HTTP_STATUS.BAD_REQUEST,
|
||||
@@ -116,11 +141,32 @@ async function postHandler(request, context) {
|
||||
if (isAllRateLimitedCredentials(credentials)) {
|
||||
return rateLimitedProviderResponse(provider, credentials);
|
||||
}
|
||||
} else if (isCustomModel) {
|
||||
credentials = await getProviderCredentialsWithQuotaPreflight(
|
||||
provider,
|
||||
null,
|
||||
null,
|
||||
requestedModel
|
||||
);
|
||||
if (!credentials) {
|
||||
return errorResponse(
|
||||
HTTP_STATUS.BAD_REQUEST,
|
||||
`No credentials for custom video provider: ${provider}`
|
||||
);
|
||||
}
|
||||
if (isAllRateLimitedCredentials(credentials)) {
|
||||
return rateLimitedProviderResponse(provider, credentials);
|
||||
}
|
||||
} else if (providerConfig?.authType === "none") {
|
||||
credentials = await resolveLocalOverrideCredentials(provider);
|
||||
}
|
||||
|
||||
const result: MediaGenerationResultLike = await handleVideoGeneration({ body, credentials, log });
|
||||
const result: MediaGenerationResultLike = await handleVideoGeneration({
|
||||
body,
|
||||
credentials,
|
||||
log,
|
||||
...(isCustomModel && { resolvedProvider: provider }),
|
||||
});
|
||||
|
||||
if (isMediaGenerationFailure(result)) {
|
||||
return failedMediaGenerationResponse(result, "Video generation provider error");
|
||||
|
||||
@@ -109,7 +109,11 @@ export async function addCustomModel(
|
||||
tokenLimits: { inputTokenLimit?: number; outputTokenLimit?: number } = {},
|
||||
// #1904: optional manual vision-capability override for the "add custom model"
|
||||
// form — read back by getCustomVisionCapabilityFields() in the /v1/models catalog.
|
||||
supportsVision?: boolean
|
||||
supportsVision?: boolean,
|
||||
// #9820: optional video-generation job preset (e.g. "agnes-video-job") for
|
||||
// custom OpenAI-compatible video models. Persisted on the model row; the
|
||||
// /v1/videos/generations handler reads it back to pick the job/poll path.
|
||||
generationConfig?: { preset: string }
|
||||
) {
|
||||
const db = getDbInstance();
|
||||
const row = db
|
||||
@@ -135,6 +139,7 @@ export async function addCustomModel(
|
||||
? { outputTokenLimit: tokenLimits.outputTokenLimit }
|
||||
: {}),
|
||||
...(typeof supportsVision === "boolean" ? { supportsVision } : {}),
|
||||
...(generationConfig && generationConfig.preset ? { generationConfig } : {}),
|
||||
};
|
||||
models.push(model);
|
||||
db.prepare(
|
||||
@@ -161,6 +166,7 @@ export async function replaceCustomModels(
|
||||
description?: string;
|
||||
supportsThinking?: boolean;
|
||||
targetFormat?: string;
|
||||
generationConfig?: { preset?: string };
|
||||
}>,
|
||||
{ allowEmpty = false }: { allowEmpty?: boolean } = {}
|
||||
) {
|
||||
@@ -196,6 +202,13 @@ export async function replaceCustomModels(
|
||||
: (prev as any)?.targetFormat
|
||||
? { targetFormat: (prev as any).targetFormat }
|
||||
: {}),
|
||||
// #9820: preserve a video job preset across auto-sync (new value wins,
|
||||
// else prev — so sync overwrites don't drop a job-config model).
|
||||
...(m.generationConfig?.preset
|
||||
? { generationConfig: { preset: m.generationConfig.preset } }
|
||||
: (prev as any)?.generationConfig?.preset
|
||||
? { generationConfig: { preset: (prev as any).generationConfig.preset } }
|
||||
: {}),
|
||||
// Preserve metadata from provider API (or previous sync)
|
||||
...(m.inputTokenLimit != null
|
||||
? { inputTokenLimit: m.inputTokenLimit }
|
||||
@@ -722,6 +735,18 @@ export async function updateCustomModel(
|
||||
}
|
||||
}
|
||||
|
||||
// #9820: optional video-generation job preset. Mirrors the upstreamHeaders
|
||||
// pattern: `null`/`undefined` clears a previously set preset; a well-formed
|
||||
// object replaces it verbatim.
|
||||
if (Object.prototype.hasOwnProperty.call(updates, "generationConfig")) {
|
||||
const gc = updates.generationConfig;
|
||||
if (gc === null || gc === undefined) {
|
||||
delete next.generationConfig;
|
||||
} else if (typeof gc === "object" && !Array.isArray(gc)) {
|
||||
next.generationConfig = gc;
|
||||
}
|
||||
}
|
||||
|
||||
models[index] = next;
|
||||
|
||||
db.prepare("UPDATE key_value SET value = ? WHERE namespace = 'customModels' AND key = ?").run(
|
||||
|
||||
@@ -51,13 +51,6 @@ const GEMINI_CLI_PROFILE: ClientIdentityProfile = Object.freeze({
|
||||
"User-Agent": "GeminiCLI/0.1.0 (linux; x64)",
|
||||
}),
|
||||
});
|
||||
const MUSE_CLI_PROFILE: ClientIdentityProfile = Object.freeze({
|
||||
id: "muse-cli",
|
||||
label: "Muse Code CLI",
|
||||
headers: Object.freeze({
|
||||
"User-Agent": "MuseCodeCLI/0.1.0 (linux; x64)",
|
||||
}),
|
||||
});
|
||||
|
||||
/** Ordered so `CLIENT_IDENTITY_PROFILE_OPTIONS` renders "Default" first. */
|
||||
export const CLIENT_IDENTITY_PROFILES: Readonly<Record<string, ClientIdentityProfile>> =
|
||||
@@ -66,7 +59,6 @@ export const CLIENT_IDENTITY_PROFILES: Readonly<Record<string, ClientIdentityPro
|
||||
"claude-cli": CLAUDE_CLI_PROFILE,
|
||||
"codex-cli": CODEX_CLI_PROFILE,
|
||||
"gemini-cli": GEMINI_CLI_PROFILE,
|
||||
"muse-cli": MUSE_CLI_PROFILE,
|
||||
});
|
||||
|
||||
export const CLIENT_IDENTITY_PROFILE_IDS: readonly string[] = Object.keys(CLIENT_IDENTITY_PROFILES);
|
||||
|
||||
@@ -275,19 +275,4 @@ export const APIKEY_PROVIDERS_FRONTIER = {
|
||||
"Writer Palmyra is OpenAI-compatible at https://api.writer.com/v1. palmyra-x5 offers a 1M-token context window.",
|
||||
hasFree: false,
|
||||
},
|
||||
"muse-code": {
|
||||
id: "muse-code",
|
||||
alias: "mc",
|
||||
name: "Muse Code (Meta)",
|
||||
icon: "auto_awesome",
|
||||
color: "#0866FF",
|
||||
textIcon: "MC",
|
||||
website: "https://github.com/meta-llama/llama-stack",
|
||||
authHint:
|
||||
"Use your META_API_KEY env var as a Bearer token. Muse Code CLI uses the OpenAI Responses API wire format (POST /responses).",
|
||||
apiHint:
|
||||
"Muse Code is OpenAI-compatible. OmniRoute routes chat traffic through the Responses API and exposes the proprietary model catalog at /v1/muse-code/models.",
|
||||
passthroughModels: true,
|
||||
hasFree: false,
|
||||
},
|
||||
};
|
||||
|
||||
@@ -249,6 +249,7 @@ export const providerModelMutationSchema = z.object({
|
||||
"audio-transcriptions",
|
||||
"audio-speech",
|
||||
"images-generations",
|
||||
"videos",
|
||||
])
|
||||
)
|
||||
.default(["chat"]),
|
||||
@@ -281,6 +282,17 @@ export const providerModelMutationSchema = z.object({
|
||||
compatByProtocol: z
|
||||
.partialRecord(z.enum(["openai", "openai-responses", "claude"]), modelCompatPerProtocolSchema)
|
||||
.optional(),
|
||||
// #9820: optional async video-generation job preset for a custom
|
||||
// OpenAI-compatible provider whose /videos surface is a submit→poll API
|
||||
// (agnes-video-job, muapi-video-job, sora-job). Persisted on the custom model
|
||||
// row; the /v1/videos/generations handler branches on it between the
|
||||
// synchronous OpenAI-compatible path and the job/poll path. `"openai-video"`
|
||||
// is a legacy no-op value that keeps the sync handler selected.
|
||||
generationConfig: z
|
||||
.object({
|
||||
preset: z.enum(["agnes-video-job", "muapi-video-job", "sora-job", "openai-video"]),
|
||||
})
|
||||
.optional(),
|
||||
});
|
||||
|
||||
export const updateModelAliasesSchema = z.object({
|
||||
|
||||
@@ -3405,26 +3405,6 @@
|
||||
"stream": "https://api.morphllm.com/v1/chat/completions"
|
||||
}
|
||||
},
|
||||
"muse-code": {
|
||||
"format": "openai",
|
||||
"headers": {
|
||||
"apiKey": {
|
||||
"Accept": "text/event-stream",
|
||||
"Authorization": "Bearer <TOK>",
|
||||
"Content-Type": "application/json"
|
||||
},
|
||||
"nonStream": {
|
||||
"Authorization": "Bearer <TOK>",
|
||||
"Content-Type": "application/json"
|
||||
},
|
||||
"oauth": {
|
||||
"Accept": "text/event-stream",
|
||||
"Authorization": "Bearer <TOK>",
|
||||
"Content-Type": "application/json"
|
||||
}
|
||||
},
|
||||
"url": {}
|
||||
},
|
||||
"muse-spark-web": {
|
||||
"format": "openai",
|
||||
"headers": {
|
||||
|
||||
90
tests/unit/adobe-firefly-references.test.ts
Normal file
90
tests/unit/adobe-firefly-references.test.ts
Normal file
@@ -0,0 +1,90 @@
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert";
|
||||
import {
|
||||
ADOBE_FIREFLY_VIDEO_MODELS,
|
||||
extractAdobeSourceImageReferences,
|
||||
normalizeAdobeReferenceBlobs,
|
||||
} from "../../open-sse/services/adobeFireflyClient.ts";
|
||||
import { getAdobeModels } from "../../src/app/api/providers/[id]/models/adobeFireflyDiscovery.ts";
|
||||
|
||||
function userImsJwt(): string {
|
||||
const payload = Buffer.from(
|
||||
JSON.stringify({
|
||||
user_id: "test@AdobeID",
|
||||
type: "access_token",
|
||||
client_id: "clio-playground-web",
|
||||
})
|
||||
).toString("base64url");
|
||||
return `eyJhbGciOiJSUzI1NiJ9.${payload}.${"sig".padEnd(40, "x")}`;
|
||||
}
|
||||
|
||||
test("reference validation enforces discovered roles, counts, and frame order", () => {
|
||||
const kling = ADOBE_FIREFLY_VIDEO_MODELS["kling-3"];
|
||||
assert.deepEqual(
|
||||
normalizeAdobeReferenceBlobs(kling, [
|
||||
{ id: "frame-a", mediaType: "image", usage: "frame" },
|
||||
{ id: "frame-b", mediaType: "image", usage: "frame" },
|
||||
]),
|
||||
[
|
||||
{ id: "frame-a", usage: "frame", order: 1 },
|
||||
{ id: "frame-b", usage: "frame", order: 2 },
|
||||
]
|
||||
);
|
||||
assert.throws(
|
||||
() => normalizeAdobeReferenceBlobs(kling, [{ id: "bad", mediaType: "image", usage: "mask" }]),
|
||||
/does not support image references with usage 'mask'/
|
||||
);
|
||||
assert.throws(
|
||||
() =>
|
||||
normalizeAdobeReferenceBlobs(kling, [
|
||||
{ id: "frame-a", usage: "frame" },
|
||||
{ id: "frame-b", usage: "frame" },
|
||||
{ id: "frame-c", usage: "frame" },
|
||||
]),
|
||||
/at most 2 frame image reference/
|
||||
);
|
||||
});
|
||||
|
||||
test("structured references skip malformed entries and preserve explicit roles", () => {
|
||||
assert.deepEqual(
|
||||
extractAdobeSourceImageReferences({
|
||||
adobe_reference_inputs: [
|
||||
null,
|
||||
{ media_type: "video", source: "ignored" },
|
||||
{ media_type: "image", source: "data:image/png;base64,AAAA", usage: "frame", order: 2 },
|
||||
],
|
||||
}),
|
||||
[{ source: "data:image/png;base64,AAAA", usage: "frame", order: 2 }]
|
||||
);
|
||||
});
|
||||
|
||||
test("provider discovery adapter returns live capabilities and verified fallback", async () => {
|
||||
const live = await getAdobeModels(undefined, userImsJwt(), {}, async () =>
|
||||
Response.json({
|
||||
models: [
|
||||
{
|
||||
modelId: "firefly-image",
|
||||
acModelFamilyProviderDisplayName: "Adobe",
|
||||
modelVersions: {
|
||||
image5: {
|
||||
enabled: true,
|
||||
outputModality: ["image"],
|
||||
modelDisplayName: "Firefly Image 5",
|
||||
requestSchema: { type: "object", properties: { prompt: { type: "string" } } },
|
||||
},
|
||||
},
|
||||
},
|
||||
],
|
||||
})
|
||||
);
|
||||
assert.equal(live.source, "api");
|
||||
assert.equal(live.models[0].id, "firefly-image-image5");
|
||||
assert.ok(live.models[0].media_capabilities);
|
||||
|
||||
const fallback = await getAdobeModels(undefined, userImsJwt(), {}, async () => {
|
||||
throw new Error("offline");
|
||||
});
|
||||
assert.equal(fallback.source, "local_catalog");
|
||||
assert.equal(fallback.models.length, 52);
|
||||
assert.match(fallback.warning || "", /discovery unavailable/);
|
||||
});
|
||||
@@ -78,6 +78,11 @@ test("adobe-firefly is registered in IMAGE_PROVIDERS with adobe-firefly-image fo
|
||||
assert.equal(entry.format, "adobe-firefly-image");
|
||||
assert.match(entry.baseUrl, /firefly-3p\.ff\.adobe\.io/);
|
||||
assert.ok(Array.isArray(entry.models) && entry.models.length >= 4);
|
||||
assert.equal(
|
||||
entry.models.some((model: { id: string }) => model.id === "nano-banana-pro"),
|
||||
false,
|
||||
"routing-only compatibility aliases must not be advertised as discovered models"
|
||||
);
|
||||
});
|
||||
|
||||
test("adobe-firefly is registered in VIDEO_PROVIDERS with adobe-firefly-video format", () => {
|
||||
@@ -154,20 +159,25 @@ test("normalizeAdobeOutputResolution maps quality tiers", () => {
|
||||
assert.equal(normalizeAdobeOutputResolution(undefined, undefined), "2K");
|
||||
});
|
||||
|
||||
test("resolveAdobeImageModel maps catalog and long model ids", () => {
|
||||
assert.equal(resolveAdobeImageModel("nano-banana-pro").id, "nano-banana-pro");
|
||||
assert.equal(resolveAdobeImageModel("adobe-firefly/nano-banana-2").id, "nano-banana-2");
|
||||
assert.equal(resolveAdobeImageModel("firefly-nano-banana-pro-2k-16x9").id, "nano-banana-pro");
|
||||
assert.equal(resolveAdobeImageModel("gpt-image").id, "gpt-image");
|
||||
test("resolveAdobeImageModel maps valid aliases to exact discovery ids", () => {
|
||||
assert.equal(resolveAdobeImageModel("nano-banana-pro").id, "gemini-flash-nano-banana-2");
|
||||
assert.equal(
|
||||
resolveAdobeImageModel("adobe-firefly/nano-banana-2").id,
|
||||
"gemini-flash-nano-banana-3"
|
||||
);
|
||||
assert.equal(resolveAdobeImageModel("gpt-image").id, "gpt-image-2");
|
||||
assert.throws(
|
||||
() => resolveAdobeImageModel("invented-image-model"),
|
||||
/Unknown Adobe Firefly image model/
|
||||
);
|
||||
assert.ok(ADOBE_FIREFLY_IMAGE_MODELS["nano-banana-pro"].upstreamModelVersion);
|
||||
});
|
||||
|
||||
test("resolveAdobeVideoModel maps sora/veo/kling families", () => {
|
||||
assert.equal(resolveAdobeVideoModel("sora-2").id, "sora-2");
|
||||
assert.equal(resolveAdobeVideoModel("firefly-sora2-pro-8s-16x9").id, "sora-2-pro");
|
||||
assert.equal(resolveAdobeVideoModel("veo-3.1-fast").id, "veo-3.1-fast");
|
||||
assert.equal(resolveAdobeVideoModel("kling-3").id, "kling-3");
|
||||
assert.ok(ADOBE_FIREFLY_VIDEO_MODELS["sora-2"].defaultDuration > 0);
|
||||
test("resolveAdobeVideoModel maps only discovered video models", () => {
|
||||
assert.equal(resolveAdobeVideoModel("veo-3.1-fast").id, "veo-3.1-fast-generate");
|
||||
assert.equal(resolveAdobeVideoModel("kling-3").id, "kling-kling-v3-standard-i2v");
|
||||
assert.throws(() => resolveAdobeVideoModel("sora-2"), /Unknown Adobe Firefly video model/);
|
||||
assert.ok(ADOBE_FIREFLY_VIDEO_MODELS["veo-3.1"].defaultDuration > 0);
|
||||
});
|
||||
|
||||
test("buildAdobeImagePayload produces nano and gpt-image shapes", () => {
|
||||
@@ -265,41 +275,12 @@ test("buildAdobeImagePayload attaches referenceBlobs like live adobe_atach_image
|
||||
sourceImageIds: ["aaaaaaaa-bbbb-4ccc-8ddd-eeeeeeeeeeee"],
|
||||
});
|
||||
assert.deepEqual(gpt.referenceBlobs, [
|
||||
{ id: "aaaaaaaa-bbbb-4ccc-8ddd-eeeeeeeeeeee", usage: "subject" },
|
||||
{ id: "aaaaaaaa-bbbb-4ccc-8ddd-eeeeeeeeeeee", usage: "source" },
|
||||
]);
|
||||
assert.equal((gpt.generationMetadata as Record<string, unknown>).module, "image2image");
|
||||
|
||||
// gpt-image: only first 2 subject refs survive (extra screenshots hang colligo).
|
||||
const gptMany = buildAdobeImagePayload({
|
||||
prompt: "edit me",
|
||||
aspectRatio: "1:1",
|
||||
outputResolution: "1K",
|
||||
modelSpec: ADOBE_FIREFLY_IMAGE_MODELS["gpt-image-2"],
|
||||
sourceImageIds: ["id-1", "id-2", "id-3", "id-4", "id-5"],
|
||||
});
|
||||
assert.deepEqual(gptMany.referenceBlobs, [
|
||||
{ id: "id-1", usage: "subject" },
|
||||
{ id: "id-2", usage: "subject" },
|
||||
]);
|
||||
|
||||
// nano keeps up to 4 general refs for multi-panel composition.
|
||||
const nanoMany = buildAdobeImagePayload({
|
||||
prompt: "compose",
|
||||
aspectRatio: "16:9",
|
||||
outputResolution: "2K",
|
||||
modelSpec: ADOBE_FIREFLY_IMAGE_MODELS["nano-banana-2"],
|
||||
sourceImageIds: ["a", "b", "c", "d", "e"],
|
||||
});
|
||||
assert.equal((nanoMany.referenceBlobs as unknown[]).length, 4);
|
||||
assert.equal((nanoMany.referenceBlobs as Array<{ usage: string }>)[0].usage, "general");
|
||||
});
|
||||
|
||||
test("adobeFireflyMaxImageRefs + adaptive image timeout", () => {
|
||||
assert.equal(adobeFireflyMaxImageRefs("gpt-image-2"), 2);
|
||||
assert.equal(adobeFireflyMaxImageRefs("adobe-firefly/gpt-image"), 2);
|
||||
assert.equal(adobeFireflyMaxImageRefs("nano-banana-2"), 4);
|
||||
assert.equal(adobeFireflyMaxImageRefs("flux-2"), 2);
|
||||
|
||||
test("adobeFireflyImageTimeoutMs scales boundedly with reference count", () => {
|
||||
assert.equal(adobeFireflyImageTimeoutMs({ refCount: 0 }), DEFAULT_IMAGE_TIMEOUT_MS);
|
||||
assert.equal(
|
||||
adobeFireflyImageTimeoutMs({ refCount: 2 }),
|
||||
@@ -381,16 +362,7 @@ test("resolveAdobeSourceImageIds uploads data URLs then returns blob ids", async
|
||||
assert.equal(ADOBE_FIREFLY_IMAGE_UPLOAD_URL.includes("storage/image"), true);
|
||||
});
|
||||
|
||||
test("buildAdobeVideoPayload produces sora and veo shapes", () => {
|
||||
const sora = buildAdobeVideoPayload({
|
||||
prompt: "ocean waves",
|
||||
aspectRatio: "16:9",
|
||||
duration: 8,
|
||||
modelSpec: ADOBE_FIREFLY_VIDEO_MODELS["sora-2"],
|
||||
});
|
||||
assert.equal(sora.modelId, "sora");
|
||||
assert.equal(sora.duration, 8);
|
||||
|
||||
test("buildAdobeVideoPayload follows discovered fields and reference roles", () => {
|
||||
const veo = buildAdobeVideoPayload({
|
||||
prompt: "city flyover",
|
||||
aspectRatio: "9:16",
|
||||
@@ -399,12 +371,30 @@ test("buildAdobeVideoPayload produces sora and veo shapes", () => {
|
||||
});
|
||||
assert.equal(veo.modelId, "veo");
|
||||
assert.equal(veo.modelVersion, "3.1-generate");
|
||||
assert.equal(
|
||||
(veo.modelSpecificPayload as Record<string, Record<string, unknown>>).parameters
|
||||
.durationSeconds,
|
||||
6
|
||||
);
|
||||
assert.equal(veo.duration, 6);
|
||||
assert.equal(veo.generateAudio, true);
|
||||
|
||||
const kling = buildAdobeVideoPayload({
|
||||
prompt: "ocean waves",
|
||||
aspectRatio: "16:9",
|
||||
duration: 5,
|
||||
modelSpec: ADOBE_FIREFLY_VIDEO_MODELS["kling-3"],
|
||||
sourceImageIds: ["aaaaaaaa-bbbb-4ccc-8ddd-eeeeeeeeeeee"],
|
||||
});
|
||||
assert.equal(kling.modelVersion, "kling_v3_standard_i2v");
|
||||
assert.deepEqual(kling.referenceBlobs, [
|
||||
{ id: "aaaaaaaa-bbbb-4ccc-8ddd-eeeeeeeeeeee", usage: "frame", order: 1 },
|
||||
]);
|
||||
assert.throws(
|
||||
() =>
|
||||
buildAdobeVideoPayload({
|
||||
prompt: "bad duration",
|
||||
aspectRatio: "16:9",
|
||||
duration: 5,
|
||||
modelSpec: ADOBE_FIREFLY_VIDEO_MODELS["veo-3.1"],
|
||||
}),
|
||||
/supports duration/
|
||||
);
|
||||
});
|
||||
|
||||
test("extractAdobeResultLink prefers x-override-status-link then links.result", () => {
|
||||
@@ -539,7 +529,7 @@ test("adobe-firefly is in USAGE_SUPPORTED_PROVIDERS for Limits", () => {
|
||||
assert.ok(USAGE_SUPPORTED_PROVIDERS.includes("firefly"));
|
||||
});
|
||||
|
||||
test("parseAdobeModelsDiscovery extracts image/video versions", () => {
|
||||
test("parseAdobeModelsDiscovery preserves schemas and maps exact ids", () => {
|
||||
const rows = parseAdobeModelsDiscovery({
|
||||
models: [
|
||||
{
|
||||
@@ -550,16 +540,44 @@ test("parseAdobeModelsDiscovery extracts image/video versions", () => {
|
||||
outputModality: ["image"],
|
||||
modelDisplayName: "Gemini 3.0 (Nano Banana Pro)",
|
||||
healthStatus: "HEALTHY",
|
||||
inputMediaUseCase: ["editing"],
|
||||
bksGenerationModel: "firefly_3p:external:gemini_flash_2",
|
||||
requestSchema: {
|
||||
type: "object",
|
||||
properties: {
|
||||
prompt: { type: "string" },
|
||||
referenceBlobs: {
|
||||
maxItems: 14,
|
||||
"x-capabilities": [
|
||||
{
|
||||
mediaType: "image",
|
||||
usageConstraints: [{ usageType: "general", minItems: 0, maxItems: 14 }],
|
||||
maxFileSizeBytes: 104857600,
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
modelId: "sora",
|
||||
modelId: "veo",
|
||||
modelVersions: {
|
||||
"sora-2": {
|
||||
"3.1-generate": {
|
||||
enabled: true,
|
||||
outputModality: ["video"],
|
||||
modelDisplayName: "Sora 2",
|
||||
modelDisplayName: "Veo 3.1",
|
||||
requestSchema: {
|
||||
allOf: [
|
||||
{
|
||||
properties: {
|
||||
prompt: { type: "string" },
|
||||
duration: { anyOf: [{ type: "integer", enum: [4, 6, 8] }] },
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
@@ -569,14 +587,35 @@ test("parseAdobeModelsDiscovery extracts image/video versions", () => {
|
||||
assert.equal(rows[0].modality, "image");
|
||||
assert.equal(rows[1].modality, "video");
|
||||
const catalog = mapDiscoveredToCatalog(rows);
|
||||
assert.ok(catalog.some((m) => m.id === "nano-banana-pro"));
|
||||
assert.ok(catalog.some((m) => m.id === "sora-2"));
|
||||
assert.ok(catalog.some((m) => m.id === "gemini-flash-nano-banana-2"));
|
||||
assert.ok(catalog.some((m) => m.id === "veo-3.1-generate"));
|
||||
assert.equal(catalog[0].capabilities.referenceInputs[0].maxItems, 14);
|
||||
assert.deepEqual(catalog[1].capabilities.supportedDurations, [4, 6, 8]);
|
||||
});
|
||||
|
||||
test("fallback catalog has image and video entries from get_models capture", () => {
|
||||
assert.ok(ADOBE_FIREFLY_FALLBACK_MODELS.length >= 10);
|
||||
assert.ok(getAdobeFireflyFallbackCatalog("image").length >= 4);
|
||||
assert.ok(getAdobeFireflyFallbackCatalog("video").length >= 4);
|
||||
test("fallback catalog is the verified discovery snapshot without invented Sora", () => {
|
||||
assert.equal(ADOBE_FIREFLY_FALLBACK_MODELS.length, 52);
|
||||
assert.equal(getAdobeFireflyFallbackCatalog("image").length, 17);
|
||||
assert.equal(getAdobeFireflyFallbackCatalog("video").length, 35);
|
||||
assert.equal(
|
||||
ADOBE_FIREFLY_FALLBACK_MODELS.some((model) => model.id.includes("sora")),
|
||||
false
|
||||
);
|
||||
assert.equal(
|
||||
ADOBE_FIREFLY_FALLBACK_MODELS.some(
|
||||
(model) => model.id.includes("kling") && model.id.includes("omni")
|
||||
),
|
||||
false
|
||||
);
|
||||
assert.ok(ADOBE_FIREFLY_FALLBACK_MODELS.some((model) => model.id === "kling-kling-o3"));
|
||||
assert.equal(
|
||||
ADOBE_FIREFLY_IMAGE_MODELS["nano-banana-pro"].capabilities.referenceInputs[0].maxItems,
|
||||
14
|
||||
);
|
||||
assert.equal(
|
||||
ADOBE_FIREFLY_IMAGE_MODELS["gpt-image"].capabilities.referenceInputs[0].maxItems,
|
||||
16
|
||||
);
|
||||
});
|
||||
|
||||
test("extractAdobeAccountIdFromToken reads user_id claim", () => {
|
||||
@@ -716,7 +755,7 @@ test("adobeFireflyGenerateVideo submit+poll happy path (mocked)", async () => {
|
||||
const result = await adobeFireflyGenerateVideo({
|
||||
accessToken: "tok",
|
||||
prompt: "drone over forest",
|
||||
model: "sora-2",
|
||||
model: "veo-3.1",
|
||||
duration: 4,
|
||||
aspectRatio: "16:9",
|
||||
fetchImpl: fetchImpl as typeof fetch,
|
||||
@@ -727,7 +766,7 @@ test("adobeFireflyGenerateVideo submit+poll happy path (mocked)", async () => {
|
||||
|
||||
test("handleAdobeFireflyVideoGeneration returns 400 without prompt", async () => {
|
||||
const result = await handleAdobeFireflyVideoGeneration({
|
||||
model: "sora-2",
|
||||
model: "veo-3.1",
|
||||
provider: "adobe-firefly",
|
||||
body: {},
|
||||
credentials: { apiKey: "aaa.bbb.ccc" },
|
||||
|
||||
@@ -1,81 +0,0 @@
|
||||
/**
|
||||
* Tests for Muse Code CLI model catalog endpoint.
|
||||
*
|
||||
* Verifies GET /v1/muse-code/models returns the proprietary Muse format.
|
||||
*/
|
||||
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import { muse_codeProvider } from "../../open-sse/config/providers/registry/muse-code/index.ts";
|
||||
|
||||
// ── Model catalog shape ─────────────────────────────────────────────────────
|
||||
|
||||
test("muse-code provider has at least one model", () => {
|
||||
assert.ok(muse_codeProvider.models.length >= 1);
|
||||
});
|
||||
|
||||
test("muse-code models have unique ids", () => {
|
||||
const ids = muse_codeProvider.models.map((m) => m.id);
|
||||
const unique = new Set(ids);
|
||||
assert.equal(unique.size, ids.length, "model IDs must be unique");
|
||||
});
|
||||
|
||||
test("muse-code models include llama-4-maverick", () => {
|
||||
const ids = muse_codeProvider.models.map((m) => m.id);
|
||||
assert.ok(ids.includes("llama-4-maverick"), "must include llama-4-maverick");
|
||||
});
|
||||
|
||||
test("muse-code models include llama-4-scout", () => {
|
||||
const ids = muse_codeProvider.models.map((m) => m.id);
|
||||
assert.ok(ids.includes("llama-4-scout"), "must include llama-4-scout");
|
||||
});
|
||||
|
||||
test("muse-code models include llama-3.3-70b", () => {
|
||||
const ids = muse_codeProvider.models.map((m) => m.id);
|
||||
assert.ok(ids.includes("llama-3.3-70b"), "must include llama-3.3-70b");
|
||||
});
|
||||
|
||||
test("llama-4 models have supportsXHighEffort", () => {
|
||||
const maverick = muse_codeProvider.models.find((m) => m.id === "llama-4-maverick");
|
||||
assert.ok(maverick, "llama-4-maverick must exist");
|
||||
assert.equal(maverick.supportsXHighEffort, true);
|
||||
|
||||
const scout = muse_codeProvider.models.find((m) => m.id === "llama-4-scout");
|
||||
assert.ok(scout, "llama-4-scout must exist");
|
||||
assert.equal(scout.supportsXHighEffort, true);
|
||||
});
|
||||
|
||||
test("llama-3.3-70b does not support reasoning", () => {
|
||||
const model = muse_codeProvider.models.find((m) => m.id === "llama-3.3-70b");
|
||||
assert.ok(model, "llama-3.3-70b must exist");
|
||||
assert.equal(model.supportsReasoning, false);
|
||||
});
|
||||
|
||||
test("non-reasoning models do not declare supportsXHighEffort", () => {
|
||||
for (const model of muse_codeProvider.models) {
|
||||
if (!model.supportsReasoning) {
|
||||
assert.equal(
|
||||
model.supportsXHighEffort,
|
||||
undefined,
|
||||
`${model.id} is not a reasoning model but has supportsXHighEffort`
|
||||
);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
// ── Vision models ───────────────────────────────────────────────────────────
|
||||
|
||||
test("vision models have supportsVision: true", () => {
|
||||
const expectedVision = [
|
||||
"llama-4-maverick",
|
||||
"llama-4-scout",
|
||||
"llama-3.2-90b-vision",
|
||||
"llama-3.2-11b-vision",
|
||||
];
|
||||
for (const model of muse_codeProvider.models) {
|
||||
if (expectedVision.includes(model.id)) {
|
||||
assert.equal(model.supportsVision, true, `${model.id} should have supportsVision`);
|
||||
}
|
||||
}
|
||||
});
|
||||
@@ -1,91 +0,0 @@
|
||||
/**
|
||||
* Tests for Muse Code CLI provider registry entry.
|
||||
*
|
||||
* Verifies the provider entry loads correctly with expected config.
|
||||
*/
|
||||
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import { muse_codeProvider } from "../../open-sse/config/providers/registry/muse-code/index.ts";
|
||||
import { getRegistryEntry } from "../../open-sse/config/providerRegistry.ts";
|
||||
|
||||
// ── Registry entry structure ────────────────────────────────────────────────
|
||||
|
||||
test("muse-code provider entry has id", () => {
|
||||
assert.equal(muse_codeProvider.id, "muse-code");
|
||||
});
|
||||
|
||||
test("muse-code provider entry has alias", () => {
|
||||
assert.equal(muse_codeProvider.alias, "mc");
|
||||
});
|
||||
|
||||
test("muse-code provider uses openai format", () => {
|
||||
assert.equal(muse_codeProvider.format, "openai");
|
||||
});
|
||||
|
||||
test("muse-code provider uses apikey auth", () => {
|
||||
assert.equal(muse_codeProvider.authType, "apikey");
|
||||
assert.equal(muse_codeProvider.authHeader, "bearer");
|
||||
});
|
||||
|
||||
test("muse-code provider has passthroughModels enabled", () => {
|
||||
assert.equal(muse_codeProvider.passthroughModels, true);
|
||||
});
|
||||
|
||||
// ── Model entries ───────────────────────────────────────────────────────────
|
||||
|
||||
test("muse-code provider has curated models", () => {
|
||||
assert.ok(muse_codeProvider.models.length > 0);
|
||||
});
|
||||
|
||||
test("all muse-code models have contextLength", () => {
|
||||
for (const model of muse_codeProvider.models) {
|
||||
assert.ok(
|
||||
typeof model.contextLength === "number" && model.contextLength > 0,
|
||||
`${model.id} must have positive contextLength`
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
test("all muse-code models have toolCalling: true", () => {
|
||||
for (const model of muse_codeProvider.models) {
|
||||
assert.equal(model.toolCalling, true, `${model.id} must have toolCalling enabled`);
|
||||
}
|
||||
});
|
||||
|
||||
test("all muse-code models have targetFormat: openai-responses", () => {
|
||||
for (const model of muse_codeProvider.models) {
|
||||
assert.equal(
|
||||
model.targetFormat,
|
||||
"openai-responses",
|
||||
`${model.id} must use openai-responses target format`
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
test("reasoning models have supportsXHighEffort", () => {
|
||||
for (const model of muse_codeProvider.models) {
|
||||
if (model.supportsReasoning) {
|
||||
assert.equal(
|
||||
model.supportsXHighEffort,
|
||||
true,
|
||||
`${model.id} is a reasoning model but missing supportsXHighEffort`
|
||||
);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
// ── Registry discovery ──────────────────────────────────────────────────────
|
||||
|
||||
test("muse-code is discoverable via getRegistryEntry", () => {
|
||||
const entry = getRegistryEntry("muse-code");
|
||||
assert.ok(entry, "getRegistryEntry must return muse-code entry");
|
||||
assert.equal(entry.id, "muse-code");
|
||||
});
|
||||
|
||||
test("muse-code is discoverable via alias", () => {
|
||||
const entry = getRegistryEntry("mc");
|
||||
assert.ok(entry, "getRegistryEntry must find muse-code by alias mc");
|
||||
assert.equal(entry.id, "muse-code");
|
||||
});
|
||||
351
tests/unit/video-custom-provider-route.test.ts
Normal file
351
tests/unit/video-custom-provider-route.test.ts
Normal file
@@ -0,0 +1,351 @@
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import fs from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
|
||||
const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-video-custom-route-"));
|
||||
process.env.DATA_DIR = TEST_DATA_DIR;
|
||||
process.env.API_KEY_SECRET = process.env.API_KEY_SECRET || "video-custom-route-test-secret";
|
||||
|
||||
const core = await import("../../src/lib/db/core.ts");
|
||||
const modelsDb = await import("../../src/lib/db/models.ts");
|
||||
const providersDb = await import("../../src/lib/db/providers.ts");
|
||||
const videoRoute = await import("../../src/app/api/v1/videos/generations/route.ts");
|
||||
|
||||
const originalFetch = globalThis.fetch;
|
||||
const originalSetTimeout = globalThis.setTimeout;
|
||||
|
||||
function createResponse(body: BodyInit | null, init?: ResponseInit & { setCookies?: string[] }) {
|
||||
const response = new Response(body, init);
|
||||
if (init?.setCookies) {
|
||||
const cookies = init.setCookies.map((c) => c).join("; ");
|
||||
response.headers.set("set-cookie", cookies);
|
||||
}
|
||||
return response;
|
||||
}
|
||||
|
||||
function immediateButSafeTimeout(
|
||||
callback: (...args: unknown[]) => void,
|
||||
ms?: number,
|
||||
...args: unknown[]
|
||||
) {
|
||||
if (ms === 20_000 || ms === 5_000) {
|
||||
return originalSetTimeout(callback as TimerHandler, 0, ...args);
|
||||
}
|
||||
return originalSetTimeout(callback as TimerHandler, ms, ...args);
|
||||
}
|
||||
|
||||
test.afterEach(() => {
|
||||
globalThis.fetch = originalFetch;
|
||||
globalThis.setTimeout = originalSetTimeout;
|
||||
});
|
||||
|
||||
test.after(() => {
|
||||
core.closeDbInstance();
|
||||
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
test("video route uses OpenAI-compatible handler for custom provider with videos endpoint", async () => {
|
||||
globalThis.setTimeout = immediateButSafeTimeout as typeof setTimeout;
|
||||
|
||||
// Seed a custom model tagged with "videos" endpoint
|
||||
await modelsDb.addCustomModel(
|
||||
"custom-video-provider",
|
||||
"super-video-v1",
|
||||
"Super Video v1",
|
||||
"manual",
|
||||
"chat-completions",
|
||||
["videos"]
|
||||
);
|
||||
|
||||
// Create a provider connection with the custom base URL
|
||||
await providersDb.createProviderConnection({
|
||||
provider: "custom-video-provider",
|
||||
authType: "apikey",
|
||||
apiKey: "custom-key",
|
||||
providerSpecificData: { baseUrl: "https://custom.example.com/v1/videos/generations" },
|
||||
});
|
||||
|
||||
let captured: { url: string; body: unknown; headers: unknown } | null = null;
|
||||
|
||||
globalThis.fetch = (async (url: unknown, init?: RequestInit) => {
|
||||
const stringUrl = String(url);
|
||||
const requestBody = init?.body ? JSON.parse(String(init.body)) : {};
|
||||
|
||||
captured = {
|
||||
url: stringUrl,
|
||||
body: requestBody,
|
||||
headers: init?.headers,
|
||||
};
|
||||
|
||||
// Return a valid OpenAI-like video generation response
|
||||
return createResponse(
|
||||
JSON.stringify({
|
||||
created: Math.floor(Date.now() / 1000),
|
||||
data: [{ url: "https://custom.example.com/generated.mp4", format: "mp4" }],
|
||||
}),
|
||||
{ status: 200, headers: { "content-type": "application/json" } }
|
||||
);
|
||||
}) as typeof fetch;
|
||||
|
||||
const response = await videoRoute.POST(
|
||||
new Request("http://localhost/api/v1/videos/generations", {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
model: "custom-video-provider/super-video-v1",
|
||||
prompt: "a cat playing piano",
|
||||
duration: 5,
|
||||
}),
|
||||
})
|
||||
);
|
||||
|
||||
const payload = (await response.json()) as {
|
||||
data: Array<{ b64_json?: string; url?: string; format?: string }>;
|
||||
};
|
||||
|
||||
assert.equal(response.status, 200);
|
||||
assert.equal(payload.data.length, 1);
|
||||
assert.equal(payload.data[0].url, "https://custom.example.com/generated.mp4");
|
||||
assert.equal(payload.data[0].format, "mp4");
|
||||
|
||||
// Verify the upstream call went to the custom provider's base URL
|
||||
assert.ok(captured, "fetch should have been called");
|
||||
assert.equal(captured!.url, "https://custom.example.com/v1/videos/generations");
|
||||
assert.equal(captured!.headers.Authorization, "Bearer custom-key");
|
||||
assert.deepEqual(captured!.body, {
|
||||
model: "super-video-v1",
|
||||
prompt: "a cat playing piano",
|
||||
duration: 5,
|
||||
});
|
||||
});
|
||||
|
||||
test("video route returns 400 for custom provider without videos endpoint", async () => {
|
||||
globalThis.setTimeout = immediateButSafeTimeout as typeof setTimeout;
|
||||
|
||||
// Seed a custom model WITHOUT "videos" endpoint
|
||||
await modelsDb.addCustomModel(
|
||||
"custom-no-video-provider",
|
||||
"text-only-model",
|
||||
"Text Only Model",
|
||||
"manual",
|
||||
"chat-completions",
|
||||
["chat", "embeddings"]
|
||||
);
|
||||
|
||||
const response = await videoRoute.POST(
|
||||
new Request("http://localhost/api/v1/videos/generations", {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
model: "custom-no-video-provider/text-only-model",
|
||||
prompt: "this should fail",
|
||||
}),
|
||||
})
|
||||
);
|
||||
|
||||
assert.equal(response.status, 400);
|
||||
const payload = await response.json();
|
||||
assert.match(payload.error.message, /Invalid video model/);
|
||||
});
|
||||
|
||||
test("video route returns 400 for unknown custom provider", async () => {
|
||||
globalThis.setTimeout = immediateButSafeTimeout as typeof setTimeout;
|
||||
|
||||
const response = await videoRoute.POST(
|
||||
new Request("http://localhost/api/v1/videos/generations", {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
model: "unknown-provider/unknown-model",
|
||||
prompt: "this should fail",
|
||||
}),
|
||||
})
|
||||
);
|
||||
|
||||
assert.equal(response.status, 400);
|
||||
const payload = await response.json();
|
||||
assert.match(payload.error.message, /Invalid video model/);
|
||||
});
|
||||
|
||||
test("video route dispatches submit→poll job flow for custom model with agnes-video-job preset", async () => {
|
||||
globalThis.setTimeout = immediateButSafeTimeout as typeof setTimeout;
|
||||
|
||||
await modelsDb.addCustomModel(
|
||||
"custom-job-provider",
|
||||
"job-video-v1",
|
||||
"Job Video v1",
|
||||
"manual",
|
||||
"chat-completions",
|
||||
["videos"],
|
||||
undefined,
|
||||
{},
|
||||
undefined,
|
||||
{ preset: "agnes-video-job" }
|
||||
);
|
||||
|
||||
await providersDb.createProviderConnection({
|
||||
provider: "custom-job-provider",
|
||||
authType: "apikey",
|
||||
apiKey: "custom-key",
|
||||
providerSpecificData: { baseUrl: "https://custom.example.com" },
|
||||
});
|
||||
|
||||
const calls: Array<{
|
||||
url: string;
|
||||
method: string;
|
||||
body: unknown;
|
||||
headers: Record<string, string>;
|
||||
}> = [];
|
||||
|
||||
globalThis.fetch = (async (url: unknown, init?: RequestInit) => {
|
||||
const stringUrl = String(url);
|
||||
const method = init?.method || "GET";
|
||||
const requestBody = init?.body ? JSON.parse(String(init.body)) : {};
|
||||
const headers = (init?.headers || {}) as Record<string, string>;
|
||||
|
||||
calls.push({ url: stringUrl, method, body: requestBody, headers });
|
||||
|
||||
if (stringUrl === "https://custom.example.com/v1/videos") {
|
||||
return createResponse(JSON.stringify({ task_id: "task-123" }), {
|
||||
status: 200,
|
||||
headers: { "content-type": "application/json" },
|
||||
});
|
||||
}
|
||||
if (stringUrl === "https://custom.example.com/v1/videos/task-123") {
|
||||
return createResponse(
|
||||
JSON.stringify({
|
||||
status: "completed",
|
||||
metadata: { url: "https://custom.example.com/job-out.mp4" },
|
||||
}),
|
||||
{ status: 200, headers: { "content-type": "application/json" } }
|
||||
);
|
||||
}
|
||||
return createResponse(JSON.stringify({ error: "unexpected fetch" }), { status: 500 });
|
||||
}) as typeof fetch;
|
||||
|
||||
const response = await videoRoute.POST(
|
||||
new Request("http://localhost/api/v1/videos/generations", {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
model: "custom-job-provider/job-video-v1",
|
||||
prompt: "a cat playing piano",
|
||||
}),
|
||||
})
|
||||
);
|
||||
|
||||
const payload = (await response.json()) as {
|
||||
created: number;
|
||||
data: Array<{ url?: string; format?: string }>;
|
||||
};
|
||||
|
||||
assert.equal(response.status, 200);
|
||||
assert.equal(payload.data.length, 1);
|
||||
assert.equal(payload.data[0].url, "https://custom.example.com/job-out.mp4");
|
||||
assert.equal(payload.data[0].format, "mp4");
|
||||
assert.ok(payload.created > 0);
|
||||
|
||||
assert.equal(calls.length, 2);
|
||||
assert.equal(calls[0].method, "POST");
|
||||
assert.equal(calls[0].url, "https://custom.example.com/v1/videos");
|
||||
assert.equal(calls[0].headers["x-api-key"], "custom-key");
|
||||
assert.deepEqual(calls[0].body, {
|
||||
model: "job-video-v1",
|
||||
prompt: "a cat playing piano",
|
||||
});
|
||||
assert.equal(calls[1].method, "GET");
|
||||
assert.equal(calls[1].url, "https://custom.example.com/v1/videos/task-123");
|
||||
});
|
||||
|
||||
test("video route returns 502 when job preset reports failed status", async () => {
|
||||
globalThis.setTimeout = immediateButSafeTimeout as typeof setTimeout;
|
||||
|
||||
await modelsDb.addCustomModel(
|
||||
"custom-job-provider-fail",
|
||||
"job-video-fail-v1",
|
||||
"Job Video Fail v1",
|
||||
"manual",
|
||||
"chat-completions",
|
||||
["videos"],
|
||||
undefined,
|
||||
{},
|
||||
undefined,
|
||||
{ preset: "agnes-video-job" }
|
||||
);
|
||||
|
||||
await providersDb.createProviderConnection({
|
||||
provider: "custom-job-provider-fail",
|
||||
authType: "apikey",
|
||||
apiKey: "custom-fail-key",
|
||||
providerSpecificData: { baseUrl: "https://custom.example.com" },
|
||||
});
|
||||
|
||||
globalThis.fetch = (async (url: unknown, init?: RequestInit) => {
|
||||
if (String(url).endsWith("/v1/videos")) {
|
||||
return createResponse(JSON.stringify({ task_id: "task-fail" }), {
|
||||
status: 200,
|
||||
headers: { "content-type": "application/json" },
|
||||
});
|
||||
}
|
||||
return createResponse(JSON.stringify({ status: "failed" }), {
|
||||
status: 200,
|
||||
headers: { "content-type": "application/json" },
|
||||
});
|
||||
}) as typeof fetch;
|
||||
|
||||
const response = await videoRoute.POST(
|
||||
new Request("http://localhost/api/v1/videos/generations", {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
model: "custom-job-provider-fail/job-video-fail-v1",
|
||||
prompt: "this should fail",
|
||||
}),
|
||||
})
|
||||
);
|
||||
|
||||
assert.equal(response.status, 502);
|
||||
const payload = (await response.json()) as { error: { message?: string } };
|
||||
assert.equal(payload.error?.message, "Video job failed (agnes-video-job)");
|
||||
});
|
||||
|
||||
test("video route returns 502 for unknown generationConfig preset", async () => {
|
||||
globalThis.setTimeout = immediateButSafeTimeout as typeof setTimeout;
|
||||
|
||||
await modelsDb.addCustomModel(
|
||||
"custom-job-provider-bad",
|
||||
"job-video-bad-v1",
|
||||
"Job Video Bad v1",
|
||||
"manual",
|
||||
"chat-completions",
|
||||
["videos"],
|
||||
undefined,
|
||||
{},
|
||||
undefined,
|
||||
{ preset: "no-such-preset" }
|
||||
);
|
||||
|
||||
await providersDb.createProviderConnection({
|
||||
provider: "custom-job-provider-bad",
|
||||
authType: "apikey",
|
||||
apiKey: "custom-bad-key",
|
||||
providerSpecificData: { baseUrl: "https://custom.example.com" },
|
||||
});
|
||||
|
||||
const response = await videoRoute.POST(
|
||||
new Request("http://localhost/api/v1/videos/generations", {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
model: "custom-job-provider-bad/job-video-bad-v1",
|
||||
prompt: "bad preset",
|
||||
}),
|
||||
})
|
||||
);
|
||||
|
||||
assert.equal(response.status, 502);
|
||||
const payload = (await response.json()) as { error: { message?: string } };
|
||||
assert.equal(payload.error.message, "Unknown video job preset: no-such-preset");
|
||||
});
|
||||
@@ -581,3 +581,108 @@ test("handleVideoGeneration rejects Runway models that require promptImage", asy
|
||||
assert.equal(result.status, 400);
|
||||
assert.match(result.error, /requires promptImage/i);
|
||||
});
|
||||
test("handleVideoGeneration uses OpenAI-compatible handler for resolved custom video providers", async () => {
|
||||
const originalFetch = globalThis.fetch;
|
||||
let captured;
|
||||
|
||||
globalThis.fetch = async (url, options = {}) => {
|
||||
captured = {
|
||||
url: String(url),
|
||||
body: JSON.parse(String(options.body || "{}")),
|
||||
headers: options.headers,
|
||||
};
|
||||
|
||||
return new Response(
|
||||
JSON.stringify({
|
||||
created: Math.floor(Date.now() / 1000),
|
||||
data: [{ url: "https://custom.example.com/video.mp4", format: "mp4" }],
|
||||
}),
|
||||
{ status: 200, headers: { "content-type": "application/json" } }
|
||||
);
|
||||
};
|
||||
|
||||
try {
|
||||
const result = await handleVideoGeneration({
|
||||
body: {
|
||||
model: "custom-provider/super-video",
|
||||
prompt: "a cat playing piano",
|
||||
duration: 5,
|
||||
},
|
||||
credentials: {
|
||||
apiKey: "custom-video-key",
|
||||
providerSpecificData: {
|
||||
baseUrl: "https://custom.example.com/v1/videos/generations",
|
||||
},
|
||||
},
|
||||
resolvedProvider: "custom-provider",
|
||||
log: null,
|
||||
});
|
||||
|
||||
assert.equal(result.success, true);
|
||||
assert.equal(captured.url, "https://custom.example.com/v1/videos/generations");
|
||||
assert.equal(captured.headers.Authorization, "Bearer custom-video-key");
|
||||
assert.deepEqual(captured.body, {
|
||||
model: "super-video",
|
||||
prompt: "a cat playing piano",
|
||||
duration: 5,
|
||||
});
|
||||
assert.deepEqual(result.data.data, [
|
||||
{ url: "https://custom.example.com/video.mp4", format: "mp4" },
|
||||
]);
|
||||
} finally {
|
||||
globalThis.fetch = originalFetch;
|
||||
}
|
||||
});
|
||||
|
||||
test("handleVideoGeneration honors resolvedProvider for bare (prefix-less) custom video models", async () => {
|
||||
const originalFetch = globalThis.fetch;
|
||||
let captured;
|
||||
|
||||
globalThis.fetch = async (url, options = {}) => {
|
||||
captured = {
|
||||
url: String(url),
|
||||
body: JSON.parse(String(options.body || "{}")),
|
||||
headers: options.headers,
|
||||
};
|
||||
|
||||
return new Response(
|
||||
JSON.stringify({
|
||||
created: Math.floor(Date.now() / 1000),
|
||||
data: [{ url: "https://custom.example.com/bare.mp4", format: "mp4" }],
|
||||
}),
|
||||
{ status: 200, headers: { "content-type": "application/json" } }
|
||||
);
|
||||
};
|
||||
|
||||
try {
|
||||
const result = await handleVideoGeneration({
|
||||
body: {
|
||||
model: "super-video",
|
||||
prompt: "a cat playing piano",
|
||||
duration: 5,
|
||||
},
|
||||
credentials: {
|
||||
apiKey: "custom-video-key",
|
||||
providerSpecificData: {
|
||||
baseUrl: "https://custom.example.com/v1/videos/generations",
|
||||
},
|
||||
},
|
||||
resolvedProvider: "custom-provider",
|
||||
log: null,
|
||||
});
|
||||
|
||||
assert.equal(result.success, true);
|
||||
assert.equal(captured.url, "https://custom.example.com/v1/videos/generations");
|
||||
assert.equal(captured.headers.Authorization, "Bearer custom-video-key");
|
||||
assert.deepEqual(captured.body, {
|
||||
model: "super-video",
|
||||
prompt: "a cat playing piano",
|
||||
duration: 5,
|
||||
});
|
||||
assert.deepEqual(result.data.data, [
|
||||
{ url: "https://custom.example.com/bare.mp4", format: "mp4" },
|
||||
]);
|
||||
} finally {
|
||||
globalThis.fetch = originalFetch;
|
||||
}
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user