pjandro/OmniRoute
1
0
Fork 0
mirror of https://github.com/diegosouzapw/OmniRoute.git synced 2026-08-18 21:22:28 +03:00
Code Issues Packages Projects Releases Wiki Activity

feat(providers): complete Jina + Gemini Embedding 2 multimodal via OmniRoute (#10581)

Browse Source 
* feat(providers): complete Jina AI via OmniRoute including Omni multimodal

Dashboard and env keys share one Jina credential pool, native v5 Omni
{text}/{image}/{content} docs pass through /v1/embeddings intact, and
classify/segment/search are proxied without a third unused Jina card.

* chore(changelog): name Jina complete-provider fragment for #10581

* feat(providers): make Gemini Embedding 2 multimodal work via OmniRoute

Route gemini-embedding-2 through embedContent/batchEmbedContents so N
OpenAI input items become N vectors, pass through native multimodal
parts, and use dashboard Gemini keys (GEMINI_API_KEY only as fallback).

* fix(providers): resolve rebase fallout for Jina/Gemini embeddings

- narrow the two new no-explicit-any violations introduced by this PR
  (validateJinaFoundationProvider's params + catch, search.ts's
  normalizeJinaSearchResponse data param)
- cast credentials to Record<string, unknown> at the two quota-preflight
  call sites in src/sse/services/auth.ts so the new JinaEnvCredentials /
  GeminiEnvCredentials union members type-check without loosening the
  allRateLimited narrowing used elsewhere in the same function

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Co-authored-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
This commit is contained in:
Ravi Tharuma
2026-08-18 15:52:43 +02:00
committed by GitHub
parent 228ef6fba9
commit 3d0ffb49a4
41 changed files with 2416 additions and 91 deletions
Download Patch File Download Diff File Expand all files Collapse all files

46
docs/reference/API_REFERENCE.md
View File

@@ -129,18 +129,43 @@ Content-Type: application/json
}
```
Available providers: Nebius, OpenAI, Mistral, Together AI, Fireworks, NVIDIA, **OpenRouter**.
Available providers: Nebius, OpenAI, Mistral, Together AI, Fireworks, NVIDIA, **OpenRouter**, Jina AI.
Catalog ids are `provider/model` (example: `jina-ai/jina-embeddings-v5-omni-small`). Bare Jina model ids that appear in the registry (for example `jina-embeddings-v5-text-small`, `jina-reranker-v3.5`) also resolve. Jina embed/rerank/classify/segment use dashboard `jina-ai` credentials first; `JINA_AI_API_KEY` is a fallback only when no dashboard key exists. The `jina-reader` card is Reader / `r.jina.ai` only (`POST /v1/web/fetch`) and never serves embeddings or rerank.
Registry models that advertise multimodal support also accept up to 32 provider-neutral structured
items. Media item types are `text`, `image`, `audio`, `video`, and `document`. Their media `source`
is either `{"type":"url","url":"https://..."}` or
`{"type":"base64","data":"...","media_type":"..."}`.
Jina v5 Omni (`jina-ai/jina-embeddings-v5-omni-small`, `jina-ai/jina-embeddings-v5-omni-nano`,
and the family alias `jina-ai/jina-embeddings-v5-omni` → omni-small) also accepts Jina's native
EmbeddingsV5Request docs and **forwards them intact** to `https://api.jina.ai/v1/embeddings`:
```json
{
"model": "jina-ai/jina-embeddings-v5-omni-small",
"task": "retrieval.query",
"normalized": true,
"input": [
{ "text": "a red bicycle" },
{ "image": "https://example.com/bike.png" },
{ "content": [{ "text": "caption" }, { "image": "data:image/png;base64,..." }] }
]
}
```
Native `{ image | audio | video | pdf }` values may be a public HTTPS URL, a `data:` URI, or raw
base64. OmniRoute does not stringify those objects or fetch native image URLs — Jina retrieves
public media itself. Extra Jina fields (`task`, `normalized`, `truncate`, `embedding_type`) are
forwarded. Text-only Jina SKUs still reject non-text docs.
Security and transport bounds:
- Remote media URLs must be public HTTPS. OmniRoute fetches them server-side with redirect
revalidation, timeout, decoded size limits, public DNS checks, and connection pinning to a
validated answer before the provider call. Providers never receive the original remote URL.
- Remote media URLs must be public HTTPS. Canonical `{type,source:url}` items are fetched
server-side (redirect revalidation, timeout, size limits, public DNS, connection pinning) and
inlined before the provider call. Jina-native `{image:"https://..."}` items are forwarded as-is
after the same public-HTTPS check; Jina fetches the URL.
- Inline base64 media is limited to 8 MiB decoded per item and 16 MiB decoded across the request.
Provider translation (canonical items are never forwarded unchanged):
@@ -338,6 +363,8 @@ Use this endpoint when a sidecar runs out-of-process and cannot import
| POST | `/v1/audio/transcriptions` | OpenAI Audio (STT) |
| POST | `/v1/audio/speech` | OpenAI TTS (returns audio body) |
| POST | `/v1/rerank` | Cohere/Voyage-style rerank |
| POST | `/v1/classify` | Jina classify (`api.jina.ai`) |
| POST | `/v1/segment` | Jina segmenter (`segment.jina.ai`) |
| POST | `/v1/moderations` | OpenAI Moderations |
| GET | `/v1/models` | OpenAI |
| POST | `/v1/messages/count_tokens` | Anthropic |
@@ -357,7 +384,16 @@ For clients that cannot attach `Authorization: Bearer ...`, OmniRoute also accep
```bash
# Rerank
POST /v1/rerank { "model": "cohere/rerank-3", "query": "...", "documents": ["..."] }
POST /v1/rerank { "model": "jina-ai/jina-reranker-v3.5", "query": "...", "documents": ["..."] }
# Jina classify (Foundation API credentials)
POST /v1/classify { "model": "jina-embeddings-v5-text-small", "input": ["..."], "labels": ["a", "b"] }
# Jina segmenter
POST /v1/segment { "content": "...", "return_chunks": true }
# Jina search (s.jina.ai; provider aliases: jina-search, jina-ai, jina)
POST /v1/search { "query": "...", "provider": "jina-search" }
# Moderations
POST /v1/moderations { "model": "omni-moderation-latest", "input": "..." }