mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-08-18 21:22:28 +03:00
feat(providers): complete Jina + Gemini Embedding 2 multimodal via OmniRoute (#10581)
* feat(providers): complete Jina AI via OmniRoute including Omni multimodal
Dashboard and env keys share one Jina credential pool, native v5 Omni
{text}/{image}/{content} docs pass through /v1/embeddings intact, and
classify/segment/search are proxied without a third unused Jina card.
* chore(changelog): name Jina complete-provider fragment for #10581
* feat(providers): make Gemini Embedding 2 multimodal work via OmniRoute
Route gemini-embedding-2 through embedContent/batchEmbedContents so N
OpenAI input items become N vectors, pass through native multimodal
parts, and use dashboard Gemini keys (GEMINI_API_KEY only as fallback).
* fix(providers): resolve rebase fallout for Jina/Gemini embeddings
- narrow the two new no-explicit-any violations introduced by this PR
(validateJinaFoundationProvider's params + catch, search.ts's
normalizeJinaSearchResponse data param)
- cast credentials to Record<string, unknown> at the two quota-preflight
call sites in src/sse/services/auth.ts so the new JinaEnvCredentials /
GeminiEnvCredentials union members type-check without loosening the
allRateLimited narrowing used elsewhere in the same function
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
This commit is contained in:
Ravi Tharuma
@@ -129,18 +129,43 @@ Content-Type: application/json
|
||||
}
|
||||
```
|
||||
|
||||
Available providers: Nebius, OpenAI, Mistral, Together AI, Fireworks, NVIDIA, **OpenRouter**.
|
||||
Available providers: Nebius, OpenAI, Mistral, Together AI, Fireworks, NVIDIA, **OpenRouter**, Jina AI.
|
||||
|
||||
Catalog ids are `provider/model` (example: `jina-ai/jina-embeddings-v5-omni-small`). Bare Jina model ids that appear in the registry (for example `jina-embeddings-v5-text-small`, `jina-reranker-v3.5`) also resolve. Jina embed/rerank/classify/segment use dashboard `jina-ai` credentials first; `JINA_AI_API_KEY` is a fallback only when no dashboard key exists. The `jina-reader` card is Reader / `r.jina.ai` only (`POST /v1/web/fetch`) and never serves embeddings or rerank.
|
||||
|
||||
Registry models that advertise multimodal support also accept up to 32 provider-neutral structured
|
||||
items. Media item types are `text`, `image`, `audio`, `video`, and `document`. Their media `source`
|
||||
is either `{"type":"url","url":"https://..."}` or
|
||||
`{"type":"base64","data":"...","media_type":"..."}`.
|
||||
|
||||
Jina v5 Omni (`jina-ai/jina-embeddings-v5-omni-small`, `jina-ai/jina-embeddings-v5-omni-nano`,
|
||||
and the family alias `jina-ai/jina-embeddings-v5-omni` → omni-small) also accepts Jina's native
|
||||
EmbeddingsV5Request docs and **forwards them intact** to `https://api.jina.ai/v1/embeddings`:
|
||||
|
||||
```json
|
||||
{
|
||||
"model": "jina-ai/jina-embeddings-v5-omni-small",
|
||||
"task": "retrieval.query",
|
||||
"normalized": true,
|
||||
"input": [
|
||||
{ "text": "a red bicycle" },
|
||||
{ "image": "https://example.com/bike.png" },
|
||||
{ "content": [{ "text": "caption" }, { "image": "data:image/png;base64,..." }] }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Native `{ image | audio | video | pdf }` values may be a public HTTPS URL, a `data:` URI, or raw
|
||||
base64. OmniRoute does not stringify those objects or fetch native image URLs — Jina retrieves
|
||||
public media itself. Extra Jina fields (`task`, `normalized`, `truncate`, `embedding_type`) are
|
||||
forwarded. Text-only Jina SKUs still reject non-text docs.
|
||||
|
||||
Security and transport bounds:
|
||||
|
||||
- Remote media URLs must be public HTTPS. OmniRoute fetches them server-side with redirect
|
||||
revalidation, timeout, decoded size limits, public DNS checks, and connection pinning to a
|
||||
validated answer before the provider call. Providers never receive the original remote URL.
|
||||
- Remote media URLs must be public HTTPS. Canonical `{type,source:url}` items are fetched
|
||||
server-side (redirect revalidation, timeout, size limits, public DNS, connection pinning) and
|
||||
inlined before the provider call. Jina-native `{image:"https://..."}` items are forwarded as-is
|
||||
after the same public-HTTPS check; Jina fetches the URL.
|
||||
- Inline base64 media is limited to 8 MiB decoded per item and 16 MiB decoded across the request.
|
||||
|
||||
Provider translation (canonical items are never forwarded unchanged):
|
||||
@@ -338,6 +363,8 @@ Use this endpoint when a sidecar runs out-of-process and cannot import
|
||||
| POST | `/v1/audio/transcriptions` | OpenAI Audio (STT) |
|
||||
| POST | `/v1/audio/speech` | OpenAI TTS (returns audio body) |
|
||||
| POST | `/v1/rerank` | Cohere/Voyage-style rerank |
|
||||
| POST | `/v1/classify` | Jina classify (`api.jina.ai`) |
|
||||
| POST | `/v1/segment` | Jina segmenter (`segment.jina.ai`) |
|
||||
| POST | `/v1/moderations` | OpenAI Moderations |
|
||||
| GET | `/v1/models` | OpenAI |
|
||||
| POST | `/v1/messages/count_tokens` | Anthropic |
|
||||
@@ -357,7 +384,16 @@ For clients that cannot attach `Authorization: Bearer ...`, OmniRoute also accep
|
||||
|
||||
```bash
|
||||
# Rerank
|
||||
POST /v1/rerank { "model": "cohere/rerank-3", "query": "...", "documents": ["..."] }
|
||||
POST /v1/rerank { "model": "jina-ai/jina-reranker-v3.5", "query": "...", "documents": ["..."] }
|
||||
|
||||
# Jina classify (Foundation API credentials)
|
||||
POST /v1/classify { "model": "jina-embeddings-v5-text-small", "input": ["..."], "labels": ["a", "b"] }
|
||||
|
||||
# Jina segmenter
|
||||
POST /v1/segment { "content": "...", "return_chunks": true }
|
||||
|
||||
# Jina search (s.jina.ai; provider aliases: jina-search, jina-ai, jina)
|
||||
POST /v1/search { "query": "...", "provider": "jina-search" }
|
||||
|
||||
# Moderations
|
||||
POST /v1/moderations { "model": "omni-moderation-latest", "input": "..." }
|
||||
|
||||
@@ -656,12 +656,20 @@ Recognized pattern: `{PROVIDER_ID}_API_KEY`
|
||||
| ------------------ | ---------- |
|
||||
| `DEEPSEEK_API_KEY` | DeepSeek |
|
||||
| `NVIDIA_API_KEY` | NVIDIA NIM |
|
||||
| `JINA_AI_API_KEY` | Jina AI (Foundation API + Reader fallback) |
|
||||
| `JINA_API_KEY` | Jina AI (alias for `JINA_AI_API_KEY`) |
|
||||
| `GEMINI_API_KEY` | Gemini (Google AI Studio) embeddings + chat fallback |
|
||||
| `GOOGLE_API_KEY` | Gemini (alias for `GEMINI_API_KEY`) |
|
||||
|
||||
> [!NOTE]
|
||||
> Static `${PROVIDER}_API_KEY` entries for Groq, xAI, Mistral, Perplexity, Together AI, Fireworks, Cerebras, Cohere, Nebius, and Qianfan were removed in v3.8.0 because the runtime no longer reads them — those providers rely exclusively on Dashboard / `data/provider-credentials.json` / the encrypted DB. See the _Audit: Removed / Dead Variables_ section at the bottom of this document for the migration path.
|
||||
|
||||
> [!TIP]
|
||||
> Keys set via the Dashboard are stored encrypted in SQLite and take precedence over environment variables.
|
||||
>
|
||||
> **Jina:** `jina-ai/…` embeddings, rerank, classify, segment, and `jina-search` do **not** bill a cluster env key when a dashboard `jina-ai` (or shared `jina-reader`) connection exists — `getProviderCredentials` is fill-first. `JINA_AI_API_KEY` / `JINA_API_KEY` are used only when no usable dashboard key exists. Call logs attribute the env fallback as `connection_id=env:JINA_AI_API_KEY`. The Reader card (`jina-reader`, `r.jina.ai`) never serves `/v1/embeddings` or `/v1/rerank`.
|
||||
>
|
||||
> **Gemini:** `gemini/gemini-embedding-2` (alias `google/gemini-embedding-2`) uses the dashboard `gemini` connection first. `GEMINI_API_KEY` / `GOOGLE_API_KEY` are used only when no usable dashboard key exists. Call logs attribute the env fallback as `connection_id=env:GEMINI_API_KEY`. Native multimodal traffic uses `x-goog-api-key` against `:embedContent` / `:batchEmbedContents` — N OpenAI `input` items become N vectors.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -224,8 +224,8 @@ Use the dashboard at `/dashboard/providers` to enable, configure, and test each
|
||||
| `inception` | `inception` | Inception | API key | [link](https://docs.inceptionlabs.ai) | 10M free tokens on signup, no credit card required. |
|
||||
| `inference-net` | `inet` | Inference.net | API key | [link](https://inference.net) | $25 free credits on signup plus research grants available |
|
||||
| `internlm` | `internlm` | InternLM (Intern-S1) | API key | [link](https://internlm.intern-ai.org.cn/) | Free monthly quota ~1M input / 3M output tokens (~10 RPM) |
|
||||
| `jina-ai` | `jina` | Jina AI | API key, embed/rerank | [link](https://jina.ai) | Bearer API key for the Jina AI rerank API. |
|
||||
| `jina-reader` | `jr` | Jina Reader | API key | [link](https://jina.ai/reader) | — |
|
||||
| `jina-ai` | `jina` | Jina AI (Foundation API) | API key, embed/rerank | [link](https://jina.ai) | Bearer API key for api.jina.ai — embeddings, rerank, classify, segment, and search. Dashboard keys take precedence over JINA_AI_API_KEY. This is not the Reader / r.jina.ai card and does not fetch URLs. |
|
||||
| `jina-reader` | `jr` | Jina Reader (r.jina.ai) | API key | [link](https://jina.ai/reader) | Bearer API key for r.jina.ai URL-to-markdown (/v1/web/fetch only). Does not serve /v1/embeddings or /v1/rerank. The same Jina token as Foundation API works; OmniRoute reuses a jina-ai dashboard key or JINA_AI_API_KEY when this card is empty. |
|
||||
| `kenari` | `kenari` | Kenari | API key | [link](https://kenari.id) | Use your Kenari API key (kn-...) in Authorization: Bearer <key>. Fully OpenAI-compatible. API base URL: https://kenari.id/v1. |
|
||||
| `kie` | `kie` | KIE.AI | API key | [link](https://kie.ai) | — |
|
||||
| `kilo-gateway` | `kg` | Kilo Gateway | API key, aggregator | [link](https://kilo.ai) | — |
|
||||
|
||||
Reference in New Issue
Block a user