mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-07-26 09:52:11 +03:00
docs(docs): expand v3.8.0 guides and add provider reference generator
Refresh the documentation set for v3.8.0 with new guides covering authz, agent protocols, cloud agents, compliance, electron, evals, guardrails, memory, skills, stealth, tunnels, webhooks, and more. Add an auto-generated provider reference plus a `gen:provider-reference` script to keep provider catalog docs aligned with `src/shared/constants/providers.ts`.
This commit is contained in:
diegosouzapw
@@ -2,6 +2,13 @@
|
||||
|
||||
> Agent-to-Agent Protocol v0.3 — OmniRoute as an intelligent routing agent
|
||||
|
||||
The A2A surface has two faces:
|
||||
|
||||
- **JSON-RPC 2.0** at `POST /a2a` (canonical entry point, defined in `src/app/a2a/route.ts`).
|
||||
- **REST** under `/api/a2a/*` for dashboards and tooling (status, task list, cancel).
|
||||
|
||||
Tasks are tracked by `A2ATaskManager` (`src/lib/a2a/taskManager.ts`, default 5-minute TTL). Skills are dispatched via `A2A_SKILL_HANDLERS` in `src/lib/a2a/taskExecution.ts`.
|
||||
|
||||
## Agent Discovery
|
||||
|
||||
```bash
|
||||
@@ -10,6 +17,8 @@ curl http://localhost:20128/.well-known/agent.json
|
||||
|
||||
Returns the Agent Card describing OmniRoute's capabilities, skills, and authentication requirements.
|
||||
|
||||
The Agent Card's `version` field is sourced from `process.env.npm_package_version` (see `src/app/.well-known/agent.json/route.ts:13`), so it stays auto-synced with `package.json` on every release.
|
||||
|
||||
---
|
||||
|
||||
## Authentication
|
||||
@@ -124,10 +133,73 @@ curl -X POST http://localhost:20128/a2a \
|
||||
|
||||
## Available Skills
|
||||
|
||||
| Skill | Description |
|
||||
| :----------------- | :------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `smart-routing` | Routes prompts through OmniRoute's intelligent pipeline. Returns response with routing explanation, cost, and resilience trace. |
|
||||
| `quota-management` | Answers natural-language queries about provider quotas, suggests free combos, and provides quota rankings. |
|
||||
OmniRoute exposes 5 A2A skills wired in `src/lib/a2a/taskExecution.ts::A2A_SKILL_HANDLERS`. Each skill module lives in `src/lib/a2a/skills/`.
|
||||
|
||||
| Skill | ID | Description |
|
||||
| :----------------- | :------------------- | :------------------------------------------------------------------------------------------ |
|
||||
| Smart Routing | `smart-routing` | Routes a prompt through the optimal provider/combo using OmniRoute's combo engine + scoring |
|
||||
| Quota Management | `quota-management` | Reports per-provider quota state, helps callers decide when to throttle/switch |
|
||||
| Provider Discovery | `provider-discovery` | Lists installed providers with capabilities, free-tier flags, OAuth status |
|
||||
| Cost Analysis | `cost-analysis` | Estimates cost of a request/conversation given the catalog + recent usage |
|
||||
| Health Report | `health-report` | Aggregates circuit breaker, cooldown, lockout state per provider |
|
||||
|
||||
> Note: the Agent Card description currently advertises "36+ providers" (`src/app/.well-known/agent.json/route.ts:26` and `:55`). The actual catalog has grown to 180+ providers — the string should be updated in a follow-up change (tracked as a separate doc/code TODO; not modified here).
|
||||
|
||||
---
|
||||
|
||||
## REST API (auxiliary)
|
||||
|
||||
The JSON-RPC endpoint `/a2a` is the canonical A2A entry point. The REST endpoints below provide auxiliary access for dashboards and external tooling:
|
||||
|
||||
| Endpoint | Method | Description | Auth |
|
||||
| :--------------------------- | :----- | :------------------------------- | :--------------------- |
|
||||
| `/api/a2a/status` | GET | Server status, registered skills | (public) |
|
||||
| `/api/a2a/tasks` | GET | List tasks with filters | management |
|
||||
| `/api/a2a/tasks/[id]` | GET | Get task by ID | management |
|
||||
| `/api/a2a/tasks/[id]/cancel` | POST | Cancel running task | management |
|
||||
| `/.well-known/agent.json` | GET | Agent Card (A2A discovery) | (public, cached 3600s) |
|
||||
|
||||
---
|
||||
|
||||
## Adding a New Skill
|
||||
|
||||
1. **Create skill file:** `src/lib/a2a/skills/<your-skill>.ts`
|
||||
|
||||
Export an async function `(task: A2ATask) => Promise<{ artifacts, metadata }>`. Follow the shape of existing skills such as `smartRouting.ts`.
|
||||
|
||||
2. **Register handler:** in `src/lib/a2a/taskExecution.ts`, add an entry to `A2A_SKILL_HANDLERS`:
|
||||
|
||||
```typescript
|
||||
export const A2A_SKILL_HANDLERS = {
|
||||
// ...existing skills
|
||||
"your-skill": async (task) => {
|
||||
const skillModule = await import("./skills/yourSkill");
|
||||
return skillModule.executeYourSkill(task);
|
||||
},
|
||||
};
|
||||
```
|
||||
|
||||
3. **Expose in Agent Card:** in `src/app/.well-known/agent.json/route.ts`, append to the `skills` array:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "your-skill",
|
||||
"name": "Your Skill",
|
||||
"description": "Brief, intent-focused description",
|
||||
"tags": ["routing", "quota"],
|
||||
"examples": ["Sample natural-language invocation"]
|
||||
}
|
||||
```
|
||||
|
||||
4. **Write tests:** `tests/unit/a2a-<your-skill>.test.ts`. Cover happy path + error path.
|
||||
|
||||
5. **Document** the new skill in this file's `Available Skills` table.
|
||||
|
||||
---
|
||||
|
||||
## Task TTL
|
||||
|
||||
Tasks expire after `ttlMinutes` (default 5 min) — configured in the `A2ATaskManager` constructor at `src/lib/a2a/taskManager.ts:82`. To customize, fork the `A2ATaskManager` instantiation and pass a different value (e.g., `new A2ATaskManager(15)` for 15-minute TTL). A background interval sweeps expired tasks every 60 seconds.
|
||||
|
||||
---
|
||||
|
||||
@@ -139,7 +211,7 @@ submitted → working → completed
|
||||
→ cancelled
|
||||
```
|
||||
|
||||
- Tasks expire after 5 minutes (configurable)
|
||||
- Tasks expire after 5 minutes by default (see [Task TTL](#task-ttl))
|
||||
- Terminal states: `completed`, `failed`, `cancelled`
|
||||
- Event log tracks every state transition
|
||||
|
||||
|
||||
276
docs/AGENT_PROTOCOLS_GUIDE.md
Normal file
276
docs/AGENT_PROTOCOLS_GUIDE.md
Normal file
@@ -0,0 +1,276 @@
|
||||
# Agent Protocols Guide
|
||||
|
||||
> **Source:** `src/lib/{a2a,acp,cloudAgent}/`, `src/app/api/{a2a,acp,cloud}/`, `src/app/api/v1/agents/`
|
||||
> **Last updated:** 2026-05-13 — v3.8.0
|
||||
|
||||
OmniRoute exposes three different agent-related surfaces. They look similar at first glance but solve different problems. Use this page to pick the right one.
|
||||
|
||||
## TL;DR
|
||||
|
||||
| Surface | Best for | Transport | Standard |
|
||||
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------- | -------------------- |
|
||||
| **A2A — Agent-to-Agent** | Cross-agent collaboration with peer agents that speak the A2A protocol | JSON-RPC 2.0 over HTTP | A2A v0.3 (open spec) |
|
||||
| **ACP — CLI Agents Registry** | Detecting / registering / launching CLI coding agents installed on the user's machine (Cursor, Cline, Codex CLI, Claude Code, Aider, etc.) | HTTP REST | OmniRoute-specific |
|
||||
| **Cloud Agents** | Submitting long-running coding tasks to external cloud services (Codex Cloud, Devin, Jules) | HTTP REST + DB-backed tasks | OmniRoute-specific |
|
||||
|
||||
The three are independent — pick any subset.
|
||||
|
||||
## Decision Tree
|
||||
|
||||
```
|
||||
Do you need a cloud service to do work outside this machine (Codex Cloud / Devin / Jules)?
|
||||
├─ YES → Cloud Agents (POST /api/v1/agents/tasks)
|
||||
└─ NO → Continue
|
||||
│
|
||||
Do you have a peer agent that speaks A2A and wants to collaborate?
|
||||
├─ YES → A2A (POST /a2a)
|
||||
└─ NO → Continue
|
||||
│
|
||||
Do you need to list / configure CLI coding agents installed locally?
|
||||
├─ YES → ACP (GET /api/acp/agents)
|
||||
└─ NO → Use plain /v1/chat/completions
|
||||
```
|
||||
|
||||
## 1. A2A — Agent-to-Agent
|
||||
|
||||
**Spec:** [A2A v0.3](https://a2a-protocol.org)
|
||||
**OmniRoute endpoint:** `POST /a2a` (JSON-RPC 2.0)
|
||||
**Agent Card:** `GET /.well-known/agent.json`
|
||||
|
||||
### When to use
|
||||
|
||||
- Building a multi-agent system where OmniRoute is one of the peers
|
||||
- Exposing OmniRoute's routing intelligence (smart-routing, quota-management, etc.) to agents in frameworks like Google ADK or generic agent meshes
|
||||
- Wrapping OmniRoute behind a standard discovery + invocation surface
|
||||
|
||||
### Methods
|
||||
|
||||
- `message/send` — submit a message, receive sync response
|
||||
- `message/stream` — submit + receive SSE-streamed progress events
|
||||
- `tasks/get` — read task by ID
|
||||
- `tasks/cancel` — cancel a running task
|
||||
|
||||
### Built-in skills (5)
|
||||
|
||||
- `smart-routing` — route a prompt through the optimal combo
|
||||
- `quota-management` — report per-provider quota state
|
||||
- `provider-discovery` — list installed providers with capabilities
|
||||
- `cost-analysis` — estimate cost of a request/conversation
|
||||
- `health-report` — aggregate breaker/cooldown/lockout state per provider
|
||||
|
||||
### Deep dive
|
||||
|
||||
See [A2A-SERVER.md](./A2A-SERVER.md) for transport details, agent card structure, task TTL config, and the template for adding new skills.
|
||||
|
||||
## 2. ACP — CLI Agents Registry
|
||||
|
||||
**OmniRoute endpoint:** `GET /api/acp/agents`
|
||||
**Source:** `src/lib/acp/{index,manager,registry}.ts`
|
||||
|
||||
### What it is
|
||||
|
||||
ACP is OmniRoute's **local CLI agent inventory**. It detects which coding CLIs are installed on the host (Cursor, Cline, Claude Code, Codex CLI, Continue, etc.), resolves their versions, and surfaces them to the dashboard so the user can wire each CLI to point at OmniRoute.
|
||||
|
||||
This is NOT an external protocol — it's an internal registry that powers the "CLI Tools" UI and the CLI fingerprint tracking (see [CLI-TOOLS.md](./CLI-TOOLS.md)).
|
||||
|
||||
### What it does
|
||||
|
||||
- Probes the host for installed CLI binaries (uses `which` / `where` per OS)
|
||||
- Reads each CLI's version (calls `<bin> --version`)
|
||||
- Optionally accepts user-defined custom agents (binary path + version probe + spawn args)
|
||||
- Persists custom agents in settings
|
||||
- Returns the unified list to the dashboard
|
||||
|
||||
### REST API
|
||||
|
||||
| Endpoint | Method | Description | Auth |
|
||||
| ----------------- | ------ | ------------------------------------------------------------- | ------- |
|
||||
| `/api/acp/agents` | GET | List detected + custom agents (installed/total counts) | API key |
|
||||
| `/api/acp/agents` | POST | Add/update/remove custom agent (action discriminator in body) | API key |
|
||||
|
||||
Body shape for POST (`customAgentBodySchema` in `src/app/api/acp/agents/route.ts`):
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "add|update|remove",
|
||||
"id": "cursor",
|
||||
"name": "Cursor",
|
||||
"binary": "/usr/local/bin/cursor",
|
||||
"versionCommand": "--version",
|
||||
"providerAlias": "cursor",
|
||||
"spawnArgs": ["--api-base", "http://localhost:20128"],
|
||||
"protocol": "stdio"
|
||||
}
|
||||
```
|
||||
|
||||
### Use cases
|
||||
|
||||
- Dashboard "CLI Tools" page lists what's installed and helps you point each at OmniRoute
|
||||
- Custom agents let power users register internal/proprietary CLIs that OmniRoute doesn't know about by default
|
||||
- Detection result fuels the `cli-tools` fingerprint matrix
|
||||
|
||||
### When NOT to use ACP
|
||||
|
||||
- ACP doesn't _run_ tasks. It only detects + configures CLIs. To actually invoke a CLI, you launch it yourself with the env vars OmniRoute provides (`OPENAI_BASE_URL`, `OPENAI_API_KEY`, etc.).
|
||||
|
||||
## 3. Cloud Agents
|
||||
|
||||
**OmniRoute endpoints:** `/api/v1/agents/tasks/*` (lifecycle) + `/api/cloud/*` (plumbing)
|
||||
**Source:** `src/lib/cloudAgent/`
|
||||
|
||||
### What it is
|
||||
|
||||
A uniform interface over third-party cloud coding agents. You submit a prompt + repo URL, OmniRoute dispatches to the right cloud agent, polls status, returns results.
|
||||
|
||||
### Supported agents (3, all confirmed in `src/lib/cloudAgent/agents/`)
|
||||
|
||||
- `codex-cloud` — OpenAI Codex Cloud
|
||||
- `devin` — Cognition Devin
|
||||
- `jules` — Google Jules
|
||||
|
||||
### Lifecycle
|
||||
|
||||
```
|
||||
POST /api/v1/agents/tasks
|
||||
→ BaseAgent.createTask() per agent class
|
||||
→ external service starts work
|
||||
→ task row created in DB (cloud_agent_tasks)
|
||||
↓
|
||||
GET /api/v1/agents/tasks/[id]
|
||||
→ lazy status sync from provider
|
||||
→ returns current status + plan + activity log
|
||||
↓
|
||||
POST /api/v1/agents/tasks/[id] (action: "approve" | "message" | "cancel")
|
||||
→ forwards to provider (or marks cancelled locally)
|
||||
↓
|
||||
DELETE /api/v1/agents/tasks/[id]
|
||||
→ local cancel
|
||||
```
|
||||
|
||||
### Auth
|
||||
|
||||
⚠️ **All `/api/v1/agents/tasks/*` endpoints require management auth** (commit `588a0333`). Bearer-only callers receive 401 since v3.8.0.
|
||||
|
||||
### Deep dive
|
||||
|
||||
See [CLOUD_AGENT.md](./CLOUD_AGENT.md) for the `CloudAgentBase` contract, per-agent specifics, schema details, and credential plumbing endpoints.
|
||||
|
||||
## Comparison: A2A vs Cloud Agents
|
||||
|
||||
Both have "long-running tasks" but at different layers:
|
||||
|
||||
| Aspect | A2A | Cloud Agents |
|
||||
| ------------------ | --------------------------------------------------------------------------------- | ---------------------------------------- |
|
||||
| Standard | Open A2A v0.3 | OmniRoute-specific |
|
||||
| Where compute runs | Inside OmniRoute (uses configured combos) | External (Codex / Devin / Jules servers) |
|
||||
| Task duration | Default TTL 5 min (configurable in `TaskManager`) | Minutes to hours |
|
||||
| Repo-aware | No (passes prompts only) | Yes (repo URL + branch) |
|
||||
| Use case | Cross-agent collab, smart routing as a service | Delegate "implement feature X in repo Y" |
|
||||
| Auth | Optional `OMNIROUTE_API_KEY` for `/a2a`; management for `/api/a2a/*` REST helpers | Always management |
|
||||
|
||||
## Integration Examples
|
||||
|
||||
### Discover OmniRoute's A2A capabilities
|
||||
|
||||
```bash
|
||||
curl http://localhost:20128/.well-known/agent.json
|
||||
```
|
||||
|
||||
Returns the Agent Card with all 5 skills, transports, and version.
|
||||
|
||||
### Call OmniRoute as an A2A agent
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/a2a \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"jsonrpc": "2.0",
|
||||
"method": "message/send",
|
||||
"params": {
|
||||
"messages": [{"role": "user", "content": "Route this prompt"}],
|
||||
"skillId": "smart-routing"
|
||||
},
|
||||
"id": 1
|
||||
}'
|
||||
```
|
||||
|
||||
### List installed CLI agents via ACP
|
||||
|
||||
```bash
|
||||
curl http://localhost:20128/api/acp/agents \
|
||||
-H "Authorization: Bearer <api-key>"
|
||||
```
|
||||
|
||||
### Add a custom CLI agent
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/api/acp/agents \
|
||||
-H "Authorization: Bearer <api-key>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"action": "add",
|
||||
"id": "my-custom-cli",
|
||||
"name": "My Custom CLI",
|
||||
"binary": "/opt/mycli/bin/mycli",
|
||||
"versionCommand": "--version",
|
||||
"providerAlias": "openai"
|
||||
}'
|
||||
```
|
||||
|
||||
### Submit a Cloud Agent task
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/api/v1/agents/tasks \
|
||||
-H "Cookie: auth_token=..." \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"agentId": "devin",
|
||||
"prompt": "Implement feature X in repo Y",
|
||||
"repo": "https://github.com/user/repo",
|
||||
"branch": "main"
|
||||
}'
|
||||
```
|
||||
|
||||
### Poll cloud task status
|
||||
|
||||
```bash
|
||||
curl http://localhost:20128/api/v1/agents/tasks/<task-id> \
|
||||
-H "Cookie: auth_token=..."
|
||||
```
|
||||
|
||||
## When to Use What
|
||||
|
||||
- **Chatbot / copilot frontend** → `/v1/chat/completions` (OpenAI-compat — not an agent protocol)
|
||||
- **Multi-agent collaboration** → A2A
|
||||
- **Listing local CLIs in the dashboard** → ACP
|
||||
- **Delegating long-running coding tasks to cloud services** → Cloud Agents
|
||||
|
||||
## Internal Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────┐
|
||||
│ OmniRoute Core │
|
||||
└─────────────────────┘
|
||||
↑ ↑ ↑
|
||||
┌─────────┘ │ └─────────┐
|
||||
│ │ │
|
||||
┌───────┐ ┌─────────┐ ┌────────────┐
|
||||
│ A2A │ │ ACP │ │ Cloud │
|
||||
│ (/a2a)│ │ (/acp) │ │ Agents │
|
||||
└───────┘ └─────────┘ │ (/v1/agents│
|
||||
│ │ │ /tasks) │
|
||||
↓ ↓ └────────────┘
|
||||
External peer Local CLI │
|
||||
agents that binaries on ↓
|
||||
speak A2A v0.3 the host Codex Cloud,
|
||||
Devin, Jules
|
||||
```
|
||||
|
||||
## See Also
|
||||
|
||||
- [A2A-SERVER.md](./A2A-SERVER.md) — A2A deep dive
|
||||
- [CLOUD_AGENT.md](./CLOUD_AGENT.md) — Cloud Agents deep dive
|
||||
- [CLI-TOOLS.md](./CLI-TOOLS.md) — External CLI integrations (uses ACP)
|
||||
- [SKILLS.md](./SKILLS.md) — Skills framework (different from A2A skills — local execution sandbox)
|
||||
- [API_REFERENCE.md](./API_REFERENCE.md#agents-protocol) — endpoint reference
|
||||
- Source: `src/lib/{a2a,acp,cloudAgent}/`
|
||||
@@ -13,8 +13,24 @@ Complete reference for all OmniRoute API endpoints.
|
||||
- [Image Generation](#image-generation)
|
||||
- [List Models](#list-models)
|
||||
- [Compatibility Endpoints](#compatibility-endpoints)
|
||||
- [Files API](#files-api)
|
||||
- [Batches API](#batches-api)
|
||||
- [Search API](#search-api)
|
||||
- [WebSocket Streaming](#websocket-streaming)
|
||||
- [Quotas & Issues Reporting](#quotas--issues-reporting)
|
||||
- [Semantic Cache](#semantic-cache)
|
||||
- [Dashboard & Management](#dashboard--management)
|
||||
- [Combo Management](#combo-management)
|
||||
- [Webhooks](#webhooks)
|
||||
- [Registered Keys (Auto-Management)](#registered-keys-auto-management)
|
||||
- [Agents Protocol](#agents-protocol)
|
||||
- [Management Proxies](#management-proxies)
|
||||
- [Resilience (extended)](#resilience-extended)
|
||||
- [Skills](#skills)
|
||||
- [Memory](#memory)
|
||||
- [MCP Server](#mcp-server)
|
||||
- [A2A Server](#a2a-server)
|
||||
- [Cloud, Evals & Assess](#cloud-evals--assess)
|
||||
- [Request Processing](#request-processing)
|
||||
- [Authentication](#authentication)
|
||||
|
||||
@@ -113,18 +129,45 @@ Authorization: Bearer your-api-key
|
||||
|
||||
## Compatibility Endpoints
|
||||
|
||||
| Method | Path | Format |
|
||||
| ------ | --------------------------- | ---------------------- |
|
||||
| POST | `/v1/chat/completions` | OpenAI |
|
||||
| POST | `/v1/messages` | Anthropic |
|
||||
| POST | `/v1/responses` | OpenAI Responses |
|
||||
| POST | `/v1/embeddings` | OpenAI |
|
||||
| POST | `/v1/images/generations` | OpenAI |
|
||||
| GET | `/v1/models` | OpenAI |
|
||||
| POST | `/v1/messages/count_tokens` | Anthropic |
|
||||
| GET | `/v1beta/models` | Gemini |
|
||||
| POST | `/v1beta/models/{...path}` | Gemini generateContent |
|
||||
| POST | `/v1/api/chat` | Ollama |
|
||||
| Method | Path | Format |
|
||||
| ------ | --------------------------- | ------------------------------- |
|
||||
| POST | `/v1/chat/completions` | OpenAI |
|
||||
| POST | `/v1/messages` | Anthropic |
|
||||
| POST | `/v1/responses` | OpenAI Responses |
|
||||
| POST | `/v1/embeddings` | OpenAI |
|
||||
| POST | `/v1/images/generations` | OpenAI Images |
|
||||
| POST | `/v1/images/edits` | OpenAI Images (edit/inpaint) |
|
||||
| POST | `/v1/videos/generations` | OpenAI-style video generation |
|
||||
| POST | `/v1/music/generations` | OpenAI-style music generation |
|
||||
| POST | `/v1/audio/transcriptions` | OpenAI Audio (STT) |
|
||||
| POST | `/v1/audio/speech` | OpenAI TTS (returns audio body) |
|
||||
| POST | `/v1/rerank` | Cohere/Voyage-style rerank |
|
||||
| POST | `/v1/moderations` | OpenAI Moderations |
|
||||
| GET | `/v1/models` | OpenAI |
|
||||
| POST | `/v1/messages/count_tokens` | Anthropic |
|
||||
| GET | `/v1beta/models` | Gemini |
|
||||
| POST | `/v1beta/models/{...path}` | Gemini generateContent |
|
||||
| POST | `/v1/api/chat` | Ollama |
|
||||
|
||||
All POST routes follow the same shape: `Bearer your-api-key` + Zod-validated JSON body (`v1RerankSchema`, `v1ModerationSchema`, `v1AudioSpeechSchema`, etc., see `src/shared/validation/schemas.ts`). 4xx is returned on schema failure.
|
||||
|
||||
```bash
|
||||
# Rerank
|
||||
POST /v1/rerank { "model": "cohere/rerank-3", "query": "...", "documents": ["..."] }
|
||||
|
||||
# Moderations
|
||||
POST /v1/moderations { "model": "omni-moderation-latest", "input": "..." }
|
||||
|
||||
# TTS — returns audio/mpeg (or requested format) body
|
||||
POST /v1/audio/speech { "model": "openai/tts-1", "input": "Hello", "voice": "alloy" }
|
||||
|
||||
# Image edit (multipart)
|
||||
POST /v1/images/edits -F image=@input.png -F prompt="..." -F mask=@mask.png
|
||||
|
||||
# Video / music generation (provider-prefixed model id)
|
||||
POST /v1/videos/generations { "model": "runway/gen-3", "prompt": "..." }
|
||||
POST /v1/music/generations { "model": "suno/v3.5", "prompt": "..." }
|
||||
```
|
||||
|
||||
### Dedicated Provider Routes
|
||||
|
||||
@@ -138,6 +181,75 @@ The provider prefix is auto-added if missing. Mismatched models return `400`.
|
||||
|
||||
---
|
||||
|
||||
## Files API
|
||||
|
||||
OpenAI-compatible files endpoint for batch input/output and file-purpose uploads.
|
||||
|
||||
| Method | Path | Description |
|
||||
| ------ | ------------------------ | ------------------------------------------------------------------------------------------------------------- |
|
||||
| POST | `/v1/files` | Upload a file (multipart: `file`, `purpose`, `expires_after[anchor]`, `expires_after[seconds]`) — 512 MiB max |
|
||||
| GET | `/v1/files` | List files for the authenticated API key |
|
||||
| GET | `/v1/files/[id]` | Retrieve a file's metadata |
|
||||
| DELETE | `/v1/files/[id]` | Delete a file |
|
||||
| GET | `/v1/files/[id]/content` | Stream the raw file body back |
|
||||
|
||||
**Auth:** Bearer API key — files are scoped per-API-key via `getApiKeyRequestScope`.
|
||||
|
||||
---
|
||||
|
||||
## Batches API
|
||||
|
||||
OpenAI-compatible batch processing.
|
||||
|
||||
| Method | Path | Description |
|
||||
| ------ | ------------------------- | --------------------------------------------------------------------------------------------------------- |
|
||||
| POST | `/v1/batches` | Create batch — body validated by `v1BatchCreateSchema` (`input_file_id`, `endpoint`, `completion_window`) |
|
||||
| GET | `/v1/batches` | List batches |
|
||||
| GET | `/v1/batches/[id]` | Retrieve batch status + `request_counts` |
|
||||
| DELETE | `/v1/batches/[id]` | Delete a finished/failed batch |
|
||||
| POST | `/v1/batches/[id]/cancel` | Cancel an in-progress batch |
|
||||
|
||||
**Auth:** Bearer API key. Batches are scoped per-API-key.
|
||||
|
||||
---
|
||||
|
||||
## Search API
|
||||
|
||||
Web/search provider abstraction (Tavily, Brave, Exa, Serper, etc.).
|
||||
|
||||
| Method | Path | Description |
|
||||
| ------ | ---------------------- | ------------------------------------------------------------------------------------ |
|
||||
| GET | `/v1/search` | List configured search providers + capabilities |
|
||||
| POST | `/v1/search` | Run a search query — body validated by `v1SearchSchema`, supports caching/coalescing |
|
||||
| GET | `/v1/search/analytics` | Per-provider hit/latency/cache stats |
|
||||
|
||||
**Auth:** Bearer API key (`extractApiKey` + `isValidApiKey`). Search policy enforced via `enforceApiKeyPolicy`.
|
||||
|
||||
---
|
||||
|
||||
## WebSocket Streaming
|
||||
|
||||
```bash
|
||||
GET /v1/ws?handshake=1
|
||||
```
|
||||
|
||||
Validates a WebSocket upgrade handshake and returns the wire protocol example messages (`request`, `cancel`). Actual WS frames are handled by the bundled WS server outside the Next.js route table.
|
||||
|
||||
**Auth:** Bearer API key during handshake.
|
||||
|
||||
---
|
||||
|
||||
## Quotas & Issues Reporting
|
||||
|
||||
| Method | Path | Description |
|
||||
| ------ | ------------------- | ------------------------------------------------------------------------------------- |
|
||||
| GET | `/v1/quotas/check` | Pre-validate quota for a `provider` + `accountId` before issuing a registered key |
|
||||
| POST | `/v1/issues/report` | Report a quota/key issuance failure to GitHub (requires `GITHUB_ISSUES_REPO` + token) |
|
||||
|
||||
**Auth:** Bearer API key (`isAuthenticated`).
|
||||
|
||||
---
|
||||
|
||||
## Semantic Cache
|
||||
|
||||
```bash
|
||||
@@ -304,12 +416,16 @@ GET response includes `agents[]` (id, name, binary, version, installed, protocol
|
||||
|
||||
### Resilience & Rate Limits
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
| ----------------------- | --------- | ---------------------------------------------------------------------------------- |
|
||||
| `/api/resilience` | GET/PATCH | Get/update request queue, connection cooldown, provider breaker, and wait settings |
|
||||
| `/api/resilience/reset` | POST | Reset provider circuit breakers |
|
||||
| `/api/rate-limits` | GET | Per-account rate limit status |
|
||||
| `/api/rate-limit` | GET | Global rate limit configuration |
|
||||
| Endpoint | Method | Description |
|
||||
| --------------------------------- | --------- | ------------------------------------------------------------------------------------ |
|
||||
| `/api/resilience` | GET/PATCH | Get/update request queue, connection cooldown, provider breaker, and wait settings |
|
||||
| `/api/resilience/reset` | POST | Reset provider circuit breakers |
|
||||
| `/api/resilience/model-cooldowns` | GET | List active per-(provider, connection, model) lockouts, sorted by remaining time |
|
||||
| `/api/resilience/model-cooldowns` | DELETE | Clear a model lockout — body `{provider, model}` or `{all: true}` to wipe everything |
|
||||
| `/api/rate-limits` | GET | Per-account rate limit status |
|
||||
| `/api/rate-limit` | GET | Global rate limit configuration |
|
||||
|
||||
> All four `/api/resilience/*` routes require **management auth** (`requireManagementAuth`). See [Resilience (extended)](#resilience-extended) for a full breakdown of provider breaker vs connection cooldown vs model lockout.
|
||||
|
||||
### Evals
|
||||
|
||||
@@ -485,6 +601,261 @@ Full architecture reference: [`ARCHITECTURE.md`](ARCHITECTURE.md)
|
||||
|
||||
---
|
||||
|
||||
## Combo Management
|
||||
|
||||
Higher-level routing combos (already summarized under `/api/combos*`) can also be mapped 1:1 from a model id pattern, allowing transparent redirection of an OpenAI-style model id to a combo.
|
||||
|
||||
| Method | Path | Description |
|
||||
| ------ | -------------------------------- | ------------------------------------------------------------------------------ |
|
||||
| GET | `/api/model-combo-mappings` | List all model→combo mappings |
|
||||
| POST | `/api/model-combo-mappings` | Create mapping — body: `{pattern, comboId, priority?, enabled?, description?}` |
|
||||
| GET | `/api/model-combo-mappings/[id]` | Retrieve a single mapping |
|
||||
| PUT | `/api/model-combo-mappings/[id]` | Update fields of an existing mapping |
|
||||
| DELETE | `/api/model-combo-mappings/[id]` | Remove a mapping |
|
||||
|
||||
**Auth:** management session/API key (`requireManagementAuth`).
|
||||
|
||||
---
|
||||
|
||||
## Webhooks
|
||||
|
||||
Outbound webhook subscriptions for OmniRoute events (request completion, quota exhaustion, key rotation, etc.).
|
||||
|
||||
| Method | Path | Description |
|
||||
| ------ | ------------------------- | --------------------------------------------------------------------- |
|
||||
| GET | `/api/webhooks` | List webhooks (secrets are masked to `<prefix>...`) |
|
||||
| POST | `/api/webhooks` | Create webhook — body: `{url, events?: ["*"], secret?, description?}` |
|
||||
| GET | `/api/webhooks/[id]` | Retrieve a webhook |
|
||||
| PUT | `/api/webhooks/[id]` | Update url/events/secret/description |
|
||||
| DELETE | `/api/webhooks/[id]` | Remove a webhook |
|
||||
| POST | `/api/webhooks/[id]/test` | Send a test payload to the webhook URL and return delivery status |
|
||||
|
||||
**Auth:** management session/API key (`requireManagementAuth`).
|
||||
|
||||
---
|
||||
|
||||
## Registered Keys (Auto-Management)
|
||||
|
||||
Used by the auto-key management subsystem to issue and rotate API keys against a backing provider/account, with daily/hourly quotas.
|
||||
|
||||
| Method | Path | Description |
|
||||
| ------ | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| GET | `/api/v1/registered-keys` | List registered keys (masked prefix only) |
|
||||
| POST | `/api/v1/registered-keys` | Issue a new registered key — body: `{name, provider?, accountId?, idempotencyKey?, expiresAt?, dailyBudget?, hourlyBudget?}`. Returns the raw key **once**. Returns `429` on quota refusal. |
|
||||
| GET | `/api/v1/registered-keys/[id]` | Retrieve a registered key's metadata (no raw material) |
|
||||
| DELETE | `/api/v1/registered-keys/[id]` | Revoke a registered key |
|
||||
| POST | `/api/v1/registered-keys/[id]/revoke` | Explicit revoke endpoint (same effect as DELETE) |
|
||||
|
||||
**Auth:** Bearer API key (`isAuthenticated`). See also `/v1/quotas/check` and `/v1/issues/report`.
|
||||
|
||||
---
|
||||
|
||||
## Agents Protocol
|
||||
|
||||
Cloud agent tasks (Claude Code, Codex Cloud, OpenHands, etc.) executed remotely on behalf of OmniRoute users.
|
||||
|
||||
| Method | Path | Description |
|
||||
| ------ | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| GET | `/api/v1/agents/tasks` | List tasks — optional `?provider=`, `?status=`, `?limit=` (1–500, default 50) |
|
||||
| POST | `/api/v1/agents/tasks` | Create task — body validated by `CreateCloudAgentTaskSchema` (`providerId`, `prompt`, `source`, `options?`). Returns `201` with task envelope |
|
||||
| DELETE | `/api/v1/agents/tasks?id=...` | Delete a task |
|
||||
| GET | `/api/v1/agents/tasks/[id]` | Read task — synchronously refreshes status from the upstream cloud agent when an `external_id` is set |
|
||||
| POST | `/api/v1/agents/tasks/[id]` | Discriminated action: `{action: "approve"}`, `{action: "message", message}`, or `{action: "cancel"}` |
|
||||
| DELETE | `/api/v1/agents/tasks/[id]` | Delete a specific task by id |
|
||||
|
||||
> **Auth:** management auth required on every method (`requireCloudAgentManagementAuth`). Prior to v3.8.0 these were unauthenticated — see commit `588a0333` for the breaking change.
|
||||
|
||||
```bash
|
||||
# Create a Claude Code cloud task
|
||||
curl -X POST http://localhost:20128/api/v1/agents/tasks \
|
||||
-H "Authorization: Bearer your-management-key" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"providerId":"claude-code-cloud","prompt":"Fix the failing test","source":{"repo":"...","branch":"..."}}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Management Proxies
|
||||
|
||||
Outbound HTTP(S)/SOCKS proxies that can be assigned to providers, accounts, or globally.
|
||||
|
||||
| Method | Path | Description |
|
||||
| ------ | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| GET | `/api/v1/management/proxies` | List proxies (with `?id=` returns one; with `?id=&where_used=1` returns the assignment graph) |
|
||||
| POST | `/api/v1/management/proxies` | Create proxy — body validated by `createProxyRegistrySchema` |
|
||||
| PATCH | `/api/v1/management/proxies` | Update proxy — body validated by `updateProxyRegistrySchema` (requires `id`) |
|
||||
| DELETE | `/api/v1/management/proxies?id=...&force=1` | Delete proxy (use `force=1` to detach assignments) |
|
||||
| GET | `/api/v1/management/proxies/assignments` | List assignments — filterable by `proxy_id`, `scope`, `scope_id`; pass `resolve_connection_id=<id>` to resolve the active proxy for a connection |
|
||||
| PUT | `/api/v1/management/proxies/assignments` | Assign — body validated by `proxyAssignmentSchema` (`{scope, scopeId?, proxyId?}`). Clears dispatcher cache |
|
||||
| PUT | `/api/v1/management/proxies/bulk-assign` | Bulk-assign — body validated by `bulkProxyAssignmentSchema` (`{scope, scopeIds[], proxyId?}`) |
|
||||
| GET | `/api/v1/management/proxies/health?hours=24` | Aggregate proxy health (success/fail counts, latency) over a window |
|
||||
|
||||
**Auth:** management session/API key on every route (`requireManagementAuth`).
|
||||
|
||||
> The task description's `POST /api/v1/management/proxies/[id]/assignments` and `POST /api/v1/management/proxies/[id]/health` are served by the flat `/assignments` and `/health` routes shown above — there are no per-id subroutes in the codebase.
|
||||
|
||||
---
|
||||
|
||||
## Resilience (extended)
|
||||
|
||||
OmniRoute exposes three independent temporary-failure mechanisms; the management endpoints below let operators read and override them:
|
||||
|
||||
| Scope | State storage | Read | Reset / clear |
|
||||
| ------------------- | ------------------------------------------ | ----------------------------------------- | ------------------------------------------- |
|
||||
| Provider breaker | `domain_circuit_breakers` + in-memory | `/api/monitoring/health` | `POST /api/resilience/reset` |
|
||||
| Connection cooldown | `rateLimitedUntil` on provider connections | `/api/rate-limits`, `/api/providers/[id]` | (re-enables lazily; clear via provider PUT) |
|
||||
| Model lockout | In-memory model-availability registry | `GET /api/resilience/model-cooldowns` | `DELETE /api/resilience/model-cooldowns` |
|
||||
|
||||
```bash
|
||||
# Clear a single model lockout
|
||||
curl -X DELETE http://localhost:20128/api/resilience/model-cooldowns \
|
||||
-H "Cookie: auth_token=..." \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"provider":"openai","model":"gpt-4o-mini"}'
|
||||
|
||||
# Wipe every lockout
|
||||
curl -X DELETE http://localhost:20128/api/resilience/model-cooldowns \
|
||||
-H "Cookie: auth_token=..." \
|
||||
-d '{"all":true}'
|
||||
```
|
||||
|
||||
Full conceptual reference and breaker defaults: see [`CLAUDE.md`](../CLAUDE.md) → "Resilience Runtime State".
|
||||
|
||||
---
|
||||
|
||||
## Skills
|
||||
|
||||
Skill framework for extending OmniRoute with custom executable handlers, plus marketplace integrations.
|
||||
|
||||
| Method | Path | Description |
|
||||
| ------ | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
|
||||
| GET | `/api/skills` | List installed skills — filterable by `?q=`, `?mode=on\|off\|auto`, `?source=skillsmp\|skillssh\|local`, paginated |
|
||||
| GET | `/api/skills/[id]` | Retrieve one skill |
|
||||
| PUT | `/api/skills/[id]` | Update skill (name, description, mode, schema, handler, tags) |
|
||||
| DELETE | `/api/skills/[id]` | Uninstall a skill |
|
||||
| POST | `/api/skills/install` | Install a skill from a raw manifest — body: `{name, version, description, schema:{input, output}, handlerCode, apiKeyId?}` |
|
||||
| GET | `/api/skills/executions` | List recent skill executions (audit trail with inputs/outputs/duration) |
|
||||
| GET | `/api/skills/marketplace?q=...` | Search/popular list from the SkillsMP marketplace (requires `skillsmpApiKey` setting) |
|
||||
| POST | `/api/skills/marketplace/install` | Install a skill by id from SkillsMP |
|
||||
| GET | `/api/skills/skillssh?q=&limit=` | Search the skills.sh registry |
|
||||
| POST | `/api/skills/skillssh/install` | Install a skill by id from skills.sh |
|
||||
|
||||
**Auth:** management session/API key. Marketplace search routes accept either management auth or a Bearer API key (`isAuthenticated`).
|
||||
|
||||
---
|
||||
|
||||
## Memory
|
||||
|
||||
Persistent conversational/factual memory store, scoped per API key / session.
|
||||
|
||||
| Method | Path | Description |
|
||||
| ------ | -------------------- | ------------------------------------------------------------------------------------------------------------ |
|
||||
| GET | `/api/memory` | List memories — `?apiKeyId=`, `?type=`, `?sessionId=`, `?q=`, with `offset/limit` or `page/limit` pagination |
|
||||
| POST | `/api/memory` | Create memory — body validated by Zod: `{content, key, type?, sessionId?, apiKeyId?, metadata?, expiresAt?}` |
|
||||
| GET | `/api/memory/[id]` | Retrieve one memory |
|
||||
| DELETE | `/api/memory/[id]` | Delete a memory |
|
||||
| GET | `/api/memory/health` | Memory subsystem health (DB connectivity, embeddings backend, vector index status) |
|
||||
|
||||
**Auth:** management session/API key (`requireManagementAuth`). `type` enum: `FACTUAL`, `EPISODIC`, `SEMANTIC`, `PROCEDURAL` (see `MemoryType` in `src/lib/memory/types.ts`).
|
||||
|
||||
---
|
||||
|
||||
## MCP Server
|
||||
|
||||
OmniRoute ships an embedded Model Context Protocol server with 3 transports (stdio, SSE, streamable-http) and scoped tools. The dashboard endpoints below read status/audit data and proxy the HTTP transports.
|
||||
|
||||
| Method | Path | Description |
|
||||
| ------ | ---------------------- | ------------------------------------------------------------------------------------------------ | -------------------- |
|
||||
| GET | `/api/mcp/status` | Heartbeat, transport, online state, last call, top tools, 24h success rate |
|
||||
| GET | `/api/mcp/tools` | List of MCP tools with `name`, `description`, `scopes`, `phase`, `auditLevel`, `sourceEndpoints` |
|
||||
| GET | `/api/mcp/sse` | Open SSE stream for the SSE transport (returns `503` if MCP disabled or transport mismatch) |
|
||||
| POST | `/api/mcp/sse` | Send JSON-RPC frame on the SSE transport |
|
||||
| GET | `/api/mcp/stream` | Open SSE side of the Streamable HTTP transport (server-initiated messages) |
|
||||
| POST | `/api/mcp/stream` | Send JSON-RPC frame on the Streamable HTTP transport |
|
||||
| DELETE | `/api/mcp/stream` | End a Streamable HTTP session |
|
||||
| GET | `/api/mcp/audit` | Query audit log — `?limit=`, `?offset=`, `?tool=`, `?success=true | false`, `?apiKeyId=` |
|
||||
| GET | `/api/mcp/audit/stats` | Aggregate audit stats (totals, success rate, avg duration, top tools) |
|
||||
|
||||
**Auth:** the `sse`/`stream` transports honor the MCP-specific auth surface (Bearer API key with `mcp` scope); the `status`/`tools`/`audit*` routes are readable from the dashboard (no extra auth required beyond reaching the dashboard host).
|
||||
|
||||
> Both HTTP transports are gated by `settings.mcpEnabled` and `settings.mcpTransport` — a transport mismatch returns `400`, an MCP disabled state returns `503`.
|
||||
|
||||
---
|
||||
|
||||
## A2A Server
|
||||
|
||||
OmniRoute exposes an A2A (Agent-to-Agent) JSON-RPC 2.0 endpoint plus a REST wrapper for inspection/dashboard use.
|
||||
|
||||
### JSON-RPC
|
||||
|
||||
```bash
|
||||
POST /a2a
|
||||
Authorization: Bearer your-api-key # optional unless OMNIROUTE_API_KEY is set
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 1,
|
||||
"method": "message/send",
|
||||
"params": {
|
||||
"skill": "smart-routing",
|
||||
"messages": [{"role": "user", "content": "Route this coding task"}]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Supported methods (all gated on `settings.a2aEnabled`):
|
||||
|
||||
| Method | Description |
|
||||
| ---------------- | ------------------------------------------------------------------ |
|
||||
| `message/send` | Synchronous skill execution; returns `{task, artifacts, metadata}` |
|
||||
| `message/stream` | Streaming SSE execution of the same skill set |
|
||||
| `tasks/get` | Fetch a task by `taskId` |
|
||||
| `tasks/cancel` | Cancel a task by `taskId` |
|
||||
|
||||
Built-in skills: `smart-routing`, `quota-management`, `provider-discovery`, `cost-analysis`, `health-report`.
|
||||
|
||||
### Agent Card
|
||||
|
||||
```bash
|
||||
GET /.well-known/agent.json
|
||||
```
|
||||
|
||||
Returns the public A2A agent card (name, description, capabilities, skill catalog, auth scheme) — cached publicly for 1h. No auth required.
|
||||
|
||||
### REST helpers
|
||||
|
||||
| Method | Path | Description |
|
||||
| ------ | ---------------------------- | --------------------------------------------------------------------------------------------------------------- |
|
||||
| GET | `/api/a2a/status` | A2A enabled + task stats + cached agent card summary |
|
||||
| GET | `/api/a2a/tasks` | List tasks — `?state=submitted\|working\|completed\|failed\|cancelled`, `?skill=`, `?limit=` (≤200), `?offset=` |
|
||||
| POST | `/api/a2a/tasks` | (Not implemented as a REST helper — create via JSON-RPC `message/send`) |
|
||||
| GET | `/api/a2a/tasks/[id]` | Retrieve one task |
|
||||
| POST | `/api/a2a/tasks/[id]/cancel` | Cancel a task |
|
||||
|
||||
**Auth:** the REST helpers run without management auth (dashboard-readable); the JSON-RPC `/a2a` route uses Bearer `OMNIROUTE_API_KEY` if configured.
|
||||
|
||||
---
|
||||
|
||||
## Cloud, Evals & Assess
|
||||
|
||||
| Method | Path | Description |
|
||||
| ------ | ------------------------------- | ------------------------------------------------------------------------------------------------- | ----------------------------- | ----------------------------------- |
|
||||
| POST | `/api/cloud/auth` | Verify a Bearer key and return masked provider connections + model aliases for cloud sync clients |
|
||||
| POST | `/api/cloud/credentials/update` | Update encrypted credentials for a cloud-synced provider |
|
||||
| POST | `/api/cloud/model/resolve` | Resolve a logical model id to a concrete provider/model using the local routing table |
|
||||
| GET | `/api/cloud/models/alias` | List model aliases as exposed to cloud sync |
|
||||
| GET | `/api/assess` | Read latest assessment categorizations (per-provider/model) |
|
||||
| POST | `/api/assess` | Run an assessment — body: `{scope: {type:"all"} | {type:"provider", providerId} | {type:"model", modelId}, trigger?}` |
|
||||
| GET | `/api/evals` | List built-in eval suites + most recent runs |
|
||||
| POST | `/api/evals` | Trigger an eval run |
|
||||
| POST | `/api/evals/suites` | Create a custom eval suite — body validated by `evalSuiteSaveSchema` |
|
||||
| GET | `/api/evals/suites/[id]` | Retrieve a custom eval suite |
|
||||
|
||||
**Auth:** `/api/cloud/auth` validates a Bearer key directly; the other `/api/cloud/*`, `/api/evals/*`, and `/api/assess` routes require management session/API key. `/api/assess` POST uses `validateBody` with a discriminated-union scope schema.
|
||||
|
||||
---
|
||||
|
||||
## Authentication
|
||||
|
||||
- Dashboard routes (`/dashboard/*`) use `auth_token` cookie
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
🌐 **Languages:** 🇺🇸 [English](ARCHITECTURE.md) | 🇧🇷 [Português (Brasil)](i18n/pt-BR/ARCHITECTURE.md) | 🇪🇸 [Español](i18n/es/ARCHITECTURE.md) | 🇫🇷 [Français](i18n/fr/ARCHITECTURE.md) | 🇮🇹 [Italiano](i18n/it/ARCHITECTURE.md) | 🇷🇺 [Русский](i18n/ru/ARCHITECTURE.md) | 🇨🇳 [中文 (简体)](i18n/zh-CN/ARCHITECTURE.md) | 🇩🇪 [Deutsch](i18n/de/ARCHITECTURE.md) | 🇮🇳 [हिन्दी](i18n/in/ARCHITECTURE.md) | 🇹🇭 [ไทย](i18n/th/ARCHITECTURE.md) | 🇺🇦 [Українська](i18n/uk-UA/ARCHITECTURE.md) | 🇸🇦 [العربية](i18n/ar/ARCHITECTURE.md) | 🇯🇵 [日本語](i18n/ja/ARCHITECTURE.md) | 🇻🇳 [Tiếng Việt](i18n/vi/ARCHITECTURE.md) | 🇧🇬 [Български](i18n/bg/ARCHITECTURE.md) | 🇩🇰 [Dansk](i18n/da/ARCHITECTURE.md) | 🇫🇮 [Suomi](i18n/fi/ARCHITECTURE.md) | 🇮🇱 [עברית](i18n/he/ARCHITECTURE.md) | 🇭🇺 [Magyar](i18n/hu/ARCHITECTURE.md) | 🇮🇩 [Bahasa Indonesia](i18n/id/ARCHITECTURE.md) | 🇰🇷 [한국어](i18n/ko/ARCHITECTURE.md) | 🇲🇾 [Bahasa Melayu](i18n/ms/ARCHITECTURE.md) | 🇳🇱 [Nederlands](i18n/nl/ARCHITECTURE.md) | 🇳🇴 [Norsk](i18n/no/ARCHITECTURE.md) | 🇵🇹 [Português (Portugal)](i18n/pt/ARCHITECTURE.md) | 🇷🇴 [Română](i18n/ro/ARCHITECTURE.md) | 🇵🇱 [Polski](i18n/pl/ARCHITECTURE.md) | 🇸🇰 [Slovenčina](i18n/sk/ARCHITECTURE.md) | 🇸🇪 [Svenska](i18n/sv/ARCHITECTURE.md) | 🇵🇭 [Filipino](i18n/phi/ARCHITECTURE.md) | 🇨🇿 [Čeština](i18n/cs/ARCHITECTURE.md)
|
||||
|
||||
_Last updated: 2026-05-02_
|
||||
_Last updated: 2026-05-13_
|
||||
|
||||
## Executive Summary
|
||||
|
||||
@@ -11,13 +11,13 @@ It provides a single OpenAI-compatible endpoint (`/v1/*`) and routes traffic acr
|
||||
|
||||
Core capabilities:
|
||||
|
||||
- OpenAI-compatible API surface for CLI/tools (160+ providers, 16 executors)
|
||||
- OpenAI-compatible API surface for CLI/tools (179 providers, 31 executors)
|
||||
- Request/response translation across provider formats
|
||||
- Model combo fallback (multi-model sequence)
|
||||
- Structured combo steps (`provider + model + connection`) with runtime ordering by `compositeTiers`
|
||||
- Account-level fallback (multi-account per provider)
|
||||
- Quota preflight and quota-aware P2C account selection in the main chat path
|
||||
- OAuth + API-key provider connection management (13 OAuth modules)
|
||||
- OAuth + API-key provider connection management (14 OAuth modules)
|
||||
- Embedding generation via `/v1/embeddings` (6 providers, 9 models)
|
||||
- Image generation via `/v1/images/generations` (10+ providers, 20+ models)
|
||||
- Audio transcription via `/v1/audio/transcriptions` (7 providers)
|
||||
@@ -60,7 +60,7 @@ Core capabilities:
|
||||
- Prompt injection guard middleware
|
||||
- Prompt compression pipeline with Caveman, RTK, stacked pipelines, compression combos, language packs, and analytics
|
||||
- ACP (Agent Communication Protocol) registry
|
||||
- Modular OAuth providers (13 individual modules under `src/lib/oauth/providers/`)
|
||||
- Modular OAuth providers (14 individual modules under `src/lib/oauth/providers/`)
|
||||
- Uninstall/full-uninstall scripts
|
||||
- OAuth environment repair action
|
||||
- WebSocket bridge for OpenAI-compatible WS clients (`/v1/ws`)
|
||||
@@ -103,11 +103,22 @@ Main pages under `src/app/(dashboard)/dashboard/`:
|
||||
- `/dashboard/endpoint` — endpoint proxy + MCP + A2A + API endpoint tabs
|
||||
- `/dashboard/providers` — provider connections and credentials
|
||||
- `/dashboard/combos` — combo strategies, templates, step-based builder, model routing rules, manual persisted ordering
|
||||
- `/dashboard/auto-combo` — Auto Combo Engine: scoring weights, mode packs, virtual factory presets, telemetry
|
||||
- `/dashboard/costs` — cost aggregation and pricing visibility
|
||||
- `/dashboard/analytics` — usage analytics, evaluations, combo target health
|
||||
- `/dashboard/limits` — quota/rate controls
|
||||
- `/dashboard/cli-tools` — CLI onboarding, runtime detection, config generation
|
||||
- `/dashboard/agents` — detected ACP agents + custom agent registration
|
||||
- `/dashboard/cloud-agents` — cloud-hosted agent tasks (Codex Cloud, Devin, Jules) and task lifecycle
|
||||
- `/dashboard/skills` — A2A skill registry, sandbox execution, built-in skill catalog
|
||||
- `/dashboard/memory` — persistent conversational memory inspection and retrieval
|
||||
- `/dashboard/webhooks` — outbound webhook subscriptions, secret rotation, retry stats
|
||||
- `/dashboard/batch` — batch job submission and progress
|
||||
- `/dashboard/cache` — read-through and reasoning cache statistics, eviction controls
|
||||
- `/dashboard/playground` — interactive chat playground against any configured combo/model
|
||||
- `/dashboard/changelog` — in-app changelog viewer (renders `CHANGELOG.md`)
|
||||
- `/dashboard/system` — runtime diagnostics, version info, environment validation surface
|
||||
- `/dashboard/onboarding` — first-run setup wizard for new installations
|
||||
- `/dashboard/media` — image/video/music playground
|
||||
- `/dashboard/search-tools` — search provider testing and history
|
||||
- `/dashboard/health` — uptime, circuit breakers, rate limits, quota-monitored sessions
|
||||
@@ -116,6 +127,10 @@ Main pages under `src/app/(dashboard)/dashboard/`:
|
||||
- `/dashboard/context/caveman` — Caveman compression rules, language packs, preview, and output mode
|
||||
- `/dashboard/context/rtk` — RTK command-output filters, preview, and runtime safety settings
|
||||
- `/dashboard/context/combos` — named compression pipelines assigned to routing combos
|
||||
- `/dashboard/translator` — translator inspection and request format conversion preview
|
||||
- `/dashboard/audit` — compliance audit log browser with pagination and structured metadata
|
||||
- `/dashboard/usage` — per-request usage browser tied to `usage_history`
|
||||
- `/dashboard/compression` — compression analytics, statistics, and pipeline assignment
|
||||
- `/dashboard/api-manager` — API key lifecycle and model permissions
|
||||
|
||||
## High-Level System Context
|
||||
@@ -285,12 +300,172 @@ Domain layer modules:
|
||||
- Eval runner: `src/lib/domain/evalRunner.ts`
|
||||
- Domain state persistence: `src/lib/db/domainState.ts` — SQLite CRUD for fallback chains, budgets, cost history, lockout state, circuit breakers
|
||||
|
||||
OAuth provider modules (13 individual files under `src/lib/oauth/providers/`):
|
||||
OAuth provider modules (14 individual files under `src/lib/oauth/providers/`):
|
||||
|
||||
- Registry index: `src/lib/oauth/providers/index.ts`
|
||||
- Individual providers: `claude.ts`, `codex.ts`, `gemini.ts`, `antigravity.ts`, `qoder.ts`, `qwen.ts`, `kimi-coding.ts`, `github.ts`, `kiro.ts`, `cursor.ts`, `kilocode.ts`, `cline.ts`
|
||||
- Individual providers: `claude.ts`, `codex.ts`, `gemini.ts`, `antigravity.ts`, `qoder.ts`, `qwen.ts`, `kimi-coding.ts`, `github.ts`, `kiro.ts`, `cursor.ts`, `kilocode.ts`, `cline.ts`, `windsurf.ts`, `gitlab-duo.ts`
|
||||
- Thin wrapper: `src/lib/oauth/providers.ts` — re-exports from individual modules
|
||||
|
||||
## Major Subsystems (v3.8.0)
|
||||
|
||||
### A. Auto Combo Engine
|
||||
|
||||
Auto Combo dynamically scores and picks routing targets at request time, rather than
|
||||
relying on a static combo definition. It powers the `auto/*` model prefix family.
|
||||
|
||||
- Engine entry: `open-sse/services/autoCombo/` (`autoComboEngine.ts`,
|
||||
`scoringEngine.ts`, `virtualFactory.ts`, `modePacks.ts`)
|
||||
- Resolver: `src/domain/comboResolver.ts` (auto-detection of `auto/` prefix)
|
||||
- Dashboard: `/dashboard/auto-combo`
|
||||
- Telemetry: `auto_combo_decisions` SQLite table
|
||||
|
||||
Key capabilities:
|
||||
|
||||
- **14 routing strategies** (priority, weighted, fill-first, round-robin, P2C, random,
|
||||
least-used, cost-optimized, strict-random, **auto**, lkgp, context-optimized,
|
||||
context-relay, plus a fallback path) — auto is the headline addition in v3.8.0.
|
||||
- **9-factor scoring**: cost, latency p95, success rate, quota headroom, lockout
|
||||
proximity, breaker state, recent failures, model availability, and tag affinity.
|
||||
- **Virtual factory** materializes ephemeral combos when no matching named combo
|
||||
exists, sourcing candidates from healthy active provider connections.
|
||||
- **Auto prefixes**: `auto/coding`, `auto/cheap`, `auto/fast`, `auto/offline`,
|
||||
`auto/smart`, `auto/lkgp` — each backed by a tuned weight profile.
|
||||
- **4 mode packs**: coding, fast, cheap, smart — shipped as preset weight
|
||||
configurations callable from the dashboard.
|
||||
|
||||
For full algorithmic detail (factor formulas, weight tuning), see
|
||||
[`docs/AUTO-COMBO.md`](AUTO-COMBO.md).
|
||||
|
||||
### B. Cloud Agents
|
||||
|
||||
Cloud Agents wraps third-party hosted code-agent platforms (Codex Cloud, Devin,
|
||||
Jules) behind a uniform DB-backed task lifecycle. All task creation/inspection
|
||||
endpoints require management authentication.
|
||||
|
||||
- Module root: `src/lib/cloudAgent/` (`baseAgent.ts`, `registry.ts`, `api.ts`,
|
||||
`types.ts`, `db.ts`, plus per-agent subdirectories under `agents/`)
|
||||
- Per-agent implementations: `agents/codex/`, `agents/devin/`, `agents/jules/`
|
||||
- Public endpoints: `/api/v1/agents/tasks/*` (list/create/get/cancel)
|
||||
- Management endpoints: `/api/cloud/*` (provisioning, status, batch)
|
||||
- Dashboard: `/dashboard/cloud-agents`
|
||||
- Storage: `cloud_agent_tasks` table
|
||||
|
||||
For per-agent provisioning and OAuth specifics, see
|
||||
[`docs/CLOUD_AGENT.md`](CLOUD_AGENT.md).
|
||||
|
||||
### C. Guardrails
|
||||
|
||||
The guardrails module is a hot-reloadable middleware layer that inspects requests
|
||||
and responses for PII, prompt injection, and unsafe vision content. Violations
|
||||
short-circuit the request with HTTP **503** plus a structured error code, allowing
|
||||
downstream callers to retry or branch.
|
||||
|
||||
- Module root: `src/lib/guardrails/` (`base.ts`, `registry.ts`, `piiMasker.ts`,
|
||||
`promptInjection.ts`, `visionBridge.ts`, `visionBridgeHelpers.ts`)
|
||||
- Hot reload: registry watches for config changes and rebuilds the chain in place
|
||||
- Wire-in points: chat handler entry, image generation handler, response sanitizer
|
||||
- HTTP contract: violations surface as `503` with `error.code = "GUARDRAIL_VIOLATION"`
|
||||
|
||||
For ruleset authoring and threshold tuning, see
|
||||
[`docs/GUARDRAILS.md`](GUARDRAILS.md).
|
||||
|
||||
### D. Domain Layer
|
||||
|
||||
The `src/domain/` namespace centralizes policy decisions so route handlers do not
|
||||
have to assemble lockout/budget/fallback logic themselves.
|
||||
|
||||
- Policy engine: `src/domain/policyEngine.ts` — single entry point for
|
||||
pre-execution evaluation (lockout → budget → fallback ordering)
|
||||
- Cost rules: `src/domain/costRules.ts`
|
||||
- Fallback policy: `src/domain/fallbackPolicy.ts`
|
||||
- Lockout policy: `src/domain/lockoutPolicy.ts`
|
||||
- Tag-based routing: `src/domain/tagRouter.ts`
|
||||
- Combo resolver: `src/domain/comboResolver.ts` — resolves combo names, auto/\*
|
||||
prefixes, and wildcard model targets to concrete execution plans
|
||||
- Connection/model rule joiner: `src/domain/connectionModelRules.ts`
|
||||
- Model availability snapshots: `src/domain/modelAvailability.ts`
|
||||
- Provider expiration tracking: `src/domain/providerExpiration.ts`
|
||||
- Quota cache: `src/domain/quotaCache.ts`
|
||||
- Degradation state: `src/domain/degradation.ts`
|
||||
- Configuration audit: `src/domain/configAudit.ts`
|
||||
- OmniRoute response metadata builder: `src/domain/omnirouteResponseMeta.ts`
|
||||
- Assessment subsystem: `src/domain/assessment/` — periodic evaluation jobs
|
||||
|
||||
### E. Authorization Pipeline
|
||||
|
||||
The authorization pipeline classifies every incoming request and applies the
|
||||
appropriate policy chain before dispatch.
|
||||
|
||||
- Pipeline entry: `src/server/authz/pipeline.ts`
|
||||
- Request classifier: `src/server/authz/classify.ts` — distinguishes public
|
||||
compatibility routes from management routes
|
||||
- Public route inventory: `src/shared/constants/publicApiRoutes.ts`
|
||||
- Policies: `src/server/authz/policies/` — composable predicates
|
||||
(`requireApiKey`, `requireManagement`, `requireFreshAuth`, etc.)
|
||||
- Header utilities: `src/server/authz/headers.ts`
|
||||
- Assertion helper: `src/server/authz/assertAuth.ts`
|
||||
- Request context: `src/server/authz/context.ts`
|
||||
|
||||
Public vs management routes are a hard boundary: agent/cooldown APIs and
|
||||
provider mutations require management auth (HTTP 401 if missing).
|
||||
|
||||
For the full route classification rules, see
|
||||
[`docs/AUTHZ_GUIDE.md`](AUTHZ_GUIDE.md).
|
||||
|
||||
### F. Workflow FSM and Task-Aware Router
|
||||
|
||||
A finite-state-machine driven router layered above combo selection to direct
|
||||
traffic based on the detected workflow stage (planning, execution,
|
||||
review) and background-task affinity.
|
||||
|
||||
- Workflow FSM: `open-sse/services/workflowFSM.ts`
|
||||
- Task-aware router: `open-sse/services/taskAwareRouter.ts`
|
||||
- Background task detector: `open-sse/services/backgroundTaskDetector.ts`
|
||||
- Intent classifier: `open-sse/services/intentClassifier.ts`
|
||||
|
||||
The FSM transitions feed into Auto Combo's scoring, biasing toward cheaper models
|
||||
for background/automation tasks and toward stronger models for interactive
|
||||
planning/review turns.
|
||||
|
||||
### G. Provider-Specific Resilience
|
||||
|
||||
Several providers ship dedicated resilience and stealth modules that piggy-back on
|
||||
the global circuit breaker / connection cooldown / model lockout layers:
|
||||
|
||||
- Antigravity 429 engine: `open-sse/services/antigravity429Engine.ts` (rotates
|
||||
identity, scrubs response headers, drives credits/version tracking via
|
||||
`antigravityCredits.ts`, `antigravityHeaderScrub.ts`, `antigravityHeaders.ts`,
|
||||
`antigravityIdentity.ts`, `antigravityObfuscation.ts`, `antigravityVersion.ts`)
|
||||
- ModelScope quota policy: `open-sse/services/modelscopePolicy.ts`
|
||||
- Claude Code CCH (Compatibility Channel Handshake): `open-sse/services/claudeCodeCCH.ts`,
|
||||
plus `claudeCodeCompatible.ts`, `claudeCodeConstraints.ts`, `claudeCodeExtraRemap.ts`,
|
||||
`claudeCodeToolRemapper.ts`
|
||||
- Claude Code fingerprint shaping: `open-sse/services/claudeCodeFingerprint.ts`
|
||||
- Claude Code obfuscation: `open-sse/services/claudeCodeObfuscation.ts`
|
||||
- ChatGPT TLS client: `open-sse/services/chatgptTlsClient.ts` (curl-impersonate
|
||||
style for ChatGPT-Web sessions)
|
||||
- ChatGPT image cache: `open-sse/services/chatgptImageCache.ts`
|
||||
|
||||
For the full stealth playbook and operational guidance, see
|
||||
[`docs/STEALTH_GUIDE.md`](STEALTH_GUIDE.md).
|
||||
|
||||
### H. Webhooks, Reasoning Cache, Read Cache
|
||||
|
||||
- **Webhooks** — outbound dispatch for provider/account/task events.
|
||||
- Dispatcher: `src/lib/webhookDispatcher.ts`
|
||||
- Storage: `webhooks` SQLite table (via `src/lib/db/webhooks.ts`)
|
||||
- Dashboard: `/dashboard/webhooks` (subscriptions, secrets, retry history)
|
||||
- For event taxonomy and retry semantics, see [`docs/WEBHOOKS.md`](WEBHOOKS.md).
|
||||
- **Reasoning Cache** — replayable reasoning blocks for providers that emit
|
||||
thinking tokens (Claude, GLMT, etc.) so consecutive turns can skip re-thinking.
|
||||
- DB layer: `src/lib/db/reasoningCache.ts`
|
||||
- Service layer: `open-sse/services/reasoningCache.ts`
|
||||
- For replay semantics, see [`docs/REASONING_REPLAY.md`](REASONING_REPLAY.md).
|
||||
- **Read Cache** — short-lived response cache keyed by signature and used to
|
||||
collapse identical retries from broken upstream SDKs.
|
||||
- DB layer: `src/lib/db/readCache.ts`
|
||||
- Stats endpoint: `GET /api/cache/stats`, dashboard at `/dashboard/cache`
|
||||
|
||||
## 3) Persistence Layer
|
||||
|
||||
Primary state DB (SQLite):
|
||||
@@ -660,9 +835,12 @@ flowchart LR
|
||||
### Translation Registry and Format Converters
|
||||
|
||||
- `open-sse/translator/index.ts`: translator registry and orchestration
|
||||
- Request translators: `open-sse/translator/request/*`
|
||||
- Response translators: `open-sse/translator/response/*`
|
||||
- Request translators: `open-sse/translator/request/*` (9 modules — `antigravity-to-openai`, `claude-to-gemini`, `claude-to-openai`, `gemini-to-openai`, `openai-responses`, `openai-to-claude`, `openai-to-cursor`, `openai-to-gemini`, `openai-to-kiro`)
|
||||
- Response translators: `open-sse/translator/response/*` (8 modules — `claude-to-openai`, `cursor-to-openai`, `gemini-to-claude`, `gemini-to-openai`, `kiro-to-openai`, `openai-responses`, `openai-to-antigravity`, `openai-to-claude`)
|
||||
- Helpers: `open-sse/translator/helpers/*` (8 modules — `claudeHelper`, `geminiHelper`, `geminiToolsSanitizer`, `maxTokensHelper`, `openaiHelper`, `responsesApiHelper`, `schemaCoercion`, `toolCallHelper`)
|
||||
- Format constants: `open-sse/translator/formats.ts`
|
||||
- Bootstrap and registry: `open-sse/translator/bootstrap.ts`, `open-sse/translator/registry.ts`
|
||||
- Image-format helpers: `open-sse/translator/image/`
|
||||
|
||||
### Persistence
|
||||
|
||||
@@ -674,66 +852,108 @@ flowchart LR
|
||||
|
||||
Each provider has a specialized executor extending `BaseExecutor` (in `open-sse/executors/base.ts`), which provides URL building, header construction, retry with exponential backoff, credential refresh hooks, and the `execute()` orchestration method.
|
||||
|
||||
| Executor | Provider(s) | Special Handling |
|
||||
| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
|
||||
| `DefaultExecutor` | OpenAI, Claude, Gemini, Qwen, OpenRouter, GLM, Kimi, MiniMax, DeepSeek, Groq, xAI, Mistral, Perplexity, Together, Fireworks, Cerebras, Cohere, NVIDIA, etc. | Dynamic URL/header config per provider |
|
||||
| `AntigravityExecutor` | Google Antigravity | Custom project/session IDs, Retry-After parsing |
|
||||
| `CliProxyApiExecutor` | CLIProxyAPI-compatible providers | Custom auth and protocol handling |
|
||||
| `CloudflareAiExecutor` | Cloudflare Workers AI | Account ID injection, Neurons-based usage tracking |
|
||||
| `CodexExecutor` | OpenAI Codex | Injects system instructions, forces reasoning effort |
|
||||
| `CursorExecutor` | Cursor IDE | ConnectRPC protocol, Protobuf encoding, request signing via checksum |
|
||||
| `GithubExecutor` | GitHub Copilot | Copilot token refresh, VSCode-mimicking headers |
|
||||
| `GeminiCLIExecutor` | Gemini CLI | Google OAuth token refresh cycle |
|
||||
| `KiroExecutor` | AWS CodeWhisperer/Kiro | AWS EventStream binary format → SSE conversion |
|
||||
| `OpenCodeExecutor` | OpenCode | AI SDK compatible provider setup |
|
||||
| `PollinationsExecutor` | Pollinations AI | No API key required, rate-limited requests |
|
||||
| `PuterExecutor` | Puter | Browser-based provider integration |
|
||||
| `QoderExecutor` | Qoder AI | PAT and OAuth support, multi-model free tier |
|
||||
| `VertexExecutor` | Google Vertex AI | Service account auth, region-based endpoints |
|
||||
| Executor | Provider(s) | Special Handling |
|
||||
| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
|
||||
| `DefaultExecutor` | OpenAI, Claude, Gemini, Qwen, OpenRouter, GLM, Kimi, MiniMax, DeepSeek, Groq, xAI, Mistral, Perplexity, Together, Fireworks, Cerebras, Cohere, NVIDIA, etc. | Dynamic URL/header config per provider |
|
||||
| `AntigravityExecutor` | Google Antigravity | Custom project/session IDs, Retry-After parsing, 429 obfuscation |
|
||||
| `AzureOpenAIExecutor` | Azure OpenAI | Deployment-based routing, api-version query enforcement |
|
||||
| `BlackboxWebExecutor` | Blackbox AI (web-mode) | Web-session reverse with TLS fingerprint emulation |
|
||||
| `ChatGPTWebExecutor` | ChatGPT web | TLS client + session cookie management (`chatgptTlsClient.ts`) |
|
||||
| `ClaudeIdentityExecutor` | Claude.ai (CCH path) | Constraint + tool-remap pipelines, fingerprint shaping |
|
||||
| `CliProxyApiExecutor` | CLIProxyAPI-compatible providers | Custom auth and protocol handling |
|
||||
| `CloudflareAiExecutor` | Cloudflare Workers AI | Account ID injection, Neurons-based usage tracking |
|
||||
| `CodexExecutor` | OpenAI Codex | Injects system instructions, forces reasoning effort |
|
||||
| `CommandCodeExecutor` | Command Code | OAuth + per-session header rotation |
|
||||
| `CursorExecutor` | Cursor IDE | ConnectRPC protocol, Protobuf encoding, request signing via checksum |
|
||||
| `DevinCliExecutor` | Devin CLI | Devin task lifecycle bridging via cloud agent module |
|
||||
| `GeminiCLIExecutor` | Gemini CLI | Google OAuth token refresh cycle |
|
||||
| `GithubExecutor` | GitHub Copilot | Copilot token refresh, VSCode-mimicking headers |
|
||||
| `GitlabExecutor` | GitLab Duo | GitLab OAuth + project-scoped routing |
|
||||
| `GlmExecutor` | Z.AI GLM (incl. `glmt` preset) | Thinking-budget aware, GLMT preset constants |
|
||||
| `GrokWebExecutor` | xAI Grok web | Web-session reverse, mode selection (think/standard) |
|
||||
| `KieExecutor` | KIE | Custom token issuance with rotating session anchors |
|
||||
| `KiroExecutor` | AWS CodeWhisperer/Kiro | AWS EventStream binary format → SSE conversion |
|
||||
| `MuseSparkWebExecutor` | Muse Spark (web) | Web-session reverse with image-message bridging |
|
||||
| `NlpCloudExecutor` | NLP Cloud | Provider-specific request body shape |
|
||||
| `OpenCodeExecutor` | OpenCode | AI SDK compatible provider setup |
|
||||
| `PerplexityWebExecutor` | Perplexity web | Web-session reverse for chat continuation |
|
||||
| `PetalsExecutor` | Petals distributed inference | Decentralized swarm routing |
|
||||
| `PollinationsExecutor` | Pollinations AI | No API key required, rate-limited requests |
|
||||
| `PuterExecutor` | Puter | Browser-based provider integration |
|
||||
| `QoderExecutor` | Qoder AI | PAT and OAuth support, multi-model free tier |
|
||||
| `VertexExecutor` | Google Vertex AI | Service account auth, region-based endpoints |
|
||||
| `WindsurfExecutor` | Windsurf (Codeium) | Codeium OAuth + session token refresh |
|
||||
|
||||
All other providers (including custom compatible nodes) use the `DefaultExecutor`.
|
||||
|
||||
## Provider Compatibility Matrix
|
||||
|
||||
| Provider | Format | Auth | Stream | Non-Stream | Token Refresh | Usage API |
|
||||
| ---------------- | ---------------- | --------------------- | ---------------- | ---------- | ------------- | ------------------ |
|
||||
| Claude | claude | API Key / OAuth | ✅ | ✅ | ✅ | ⚠️ Admin only |
|
||||
| Gemini | gemini | API Key / OAuth | ✅ | ✅ | ✅ | ⚠️ Cloud Console |
|
||||
| Gemini CLI | gemini-cli | OAuth | ✅ | ✅ | ✅ | ⚠️ Cloud Console |
|
||||
| Antigravity | antigravity | OAuth | ✅ | ✅ | ✅ | ✅ Full quota API |
|
||||
| OpenAI | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Codex | openai-responses | OAuth | ✅ forced | ❌ | ✅ | ✅ Rate limits |
|
||||
| GitHub Copilot | openai | OAuth + Copilot Token | ✅ | ✅ | ✅ | ✅ Quota snapshots |
|
||||
| Cursor | cursor | Custom checksum | ✅ | ✅ | ❌ | ❌ |
|
||||
| Kiro | kiro | AWS SSO OIDC | ✅ (EventStream) | ❌ | ✅ | ✅ Usage limits |
|
||||
| Qwen | openai | OAuth | ✅ | ✅ | ✅ | ⚠️ Per request |
|
||||
| Qoder | openai | OAuth / PAT | ✅ | ✅ | ✅ | ⚠️ Per request |
|
||||
| Kilo Code | openai | OAuth | ✅ | ✅ | ✅ | ❌ |
|
||||
| Cline | openai | OAuth | ✅ | ✅ | ✅ | ❌ |
|
||||
| Kimi Coding | openai | OAuth | ✅ | ✅ | ✅ | ❌ |
|
||||
| OpenRouter | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| GLM/Kimi/MiniMax | claude | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| DeepSeek | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Groq | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| xAI (Grok) | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Mistral | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Perplexity | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Together AI | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Fireworks AI | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Cerebras | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Cohere | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| NVIDIA NIM | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Cloudflare AI | openai | API Token + Acct ID | ✅ | ✅ | ❌ | ❌ |
|
||||
| Pollinations | openai | None (no key) | ✅ | ✅ | ❌ | ❌ |
|
||||
| Scaleway AI | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| LongCat | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Ollama Cloud | openai | API Key (optional) | ✅ | ✅ | ❌ | ❌ |
|
||||
| HuggingFace | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Nebius | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| SiliconFlow | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Hyperbolic | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Vertex AI | gemini | Service Account | ✅ | ✅ | ✅ | ⚠️ Cloud Console |
|
||||
| Puter | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
> **Note:** The matrix below is a representative sample of the 179 registered providers in
|
||||
> OmniRoute v3.8.0. For the canonical and continuously-updated list, refer to
|
||||
> [`docs/PROVIDER_REFERENCE.md`](PROVIDER_REFERENCE.md) (auto-generated) or the source of
|
||||
> truth at `src/shared/constants/providers.ts` (Zod-validated at load).
|
||||
|
||||
| Provider | Format | Auth | Stream | Non-Stream | Token Refresh | Usage API |
|
||||
| ----------------- | ---------------- | --------------------- | ---------------- | ---------- | ------------- | ------------------ |
|
||||
| Claude | claude | API Key / OAuth | ✅ | ✅ | ✅ | ⚠️ Admin only |
|
||||
| Gemini | gemini | API Key / OAuth | ✅ | ✅ | ✅ | ⚠️ Cloud Console |
|
||||
| Gemini CLI | gemini-cli | OAuth | ✅ | ✅ | ✅ | ⚠️ Cloud Console |
|
||||
| Antigravity | antigravity | OAuth | ✅ | ✅ | ✅ | ✅ Full quota API |
|
||||
| OpenAI | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Codex | openai-responses | OAuth | ✅ forced | ❌ | ✅ | ✅ Rate limits |
|
||||
| GitHub Copilot | openai | OAuth + Copilot Token | ✅ | ✅ | ✅ | ✅ Quota snapshots |
|
||||
| Cursor | cursor | Custom checksum | ✅ | ✅ | ❌ | ❌ |
|
||||
| Kiro | kiro | AWS SSO OIDC | ✅ (EventStream) | ❌ | ✅ | ✅ Usage limits |
|
||||
| Qwen | openai | OAuth | ✅ | ✅ | ✅ | ⚠️ Per request |
|
||||
| Qoder | openai | OAuth / PAT | ✅ | ✅ | ✅ | ⚠️ Per request |
|
||||
| Kilo Code | openai | OAuth | ✅ | ✅ | ✅ | ❌ |
|
||||
| Cline | openai | OAuth | ✅ | ✅ | ✅ | ❌ |
|
||||
| Kimi Coding | openai | OAuth | ✅ | ✅ | ✅ | ❌ |
|
||||
| OpenRouter | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| GLM/Kimi/MiniMax | claude | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| DeepSeek | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Groq | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| xAI (Grok) | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Mistral | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Perplexity | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Together AI | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Fireworks AI | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Cerebras | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Cohere | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| NVIDIA NIM | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Cloudflare AI | openai | API Token + Acct ID | ✅ | ✅ | ❌ | ❌ |
|
||||
| Pollinations | openai | None (no key) | ✅ | ✅ | ❌ | ❌ |
|
||||
| Scaleway AI | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| LongCat | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Ollama Cloud | openai | API Key (optional) | ✅ | ✅ | ❌ | ❌ |
|
||||
| HuggingFace | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Nebius | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| SiliconFlow | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Hyperbolic | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Vertex AI | gemini | Service Account | ✅ | ✅ | ✅ | ⚠️ Cloud Console |
|
||||
| Puter | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Command Code | openai | OAuth | ✅ | ✅ | ✅ | ⚠️ Per request |
|
||||
| Z.AI / GLM | openai | API Key / OAuth | ✅ | ✅ | ❌ | ❌ |
|
||||
| GLMT (preset) | claude | API Key | ✅ | ✅ | ❌ | ⚠️ Per request |
|
||||
| Kimi Coding | openai | OAuth / API Key | ✅ | ✅ | ✅ | ❌ |
|
||||
| KIE | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Windsurf | openai | OAuth (Codeium) | ✅ | ✅ | ✅ | ⚠️ Per request |
|
||||
| GitLab Duo | openai | OAuth (GitLab) | ✅ | ✅ | ✅ | ❌ |
|
||||
| Devin CLI | openai | OAuth | ✅ | ✅ | ✅ | ✅ Task API |
|
||||
| Codex Cloud | openai-responses | OAuth | ✅ | ❌ | ✅ | ✅ Rate limits |
|
||||
| Jules | openai | OAuth | ✅ | ✅ | ✅ | ✅ Task API |
|
||||
| AgentRouter | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| ChatGPT-Web | openai | Session cookie + TLS | ✅ | ✅ | ❌ | ❌ |
|
||||
| Grok-Web | openai | Session cookie | ✅ | ✅ | ❌ | ❌ |
|
||||
| Perplexity-Web | openai | Session cookie | ✅ | ✅ | ❌ | ❌ |
|
||||
| BlackBox-Web | openai | Session cookie + TLS | ✅ | ✅ | ❌ | ❌ |
|
||||
| Muse-Spark-Web | openai | Session cookie | ✅ | ✅ | ❌ | ❌ |
|
||||
| ModelScope | openai | API Key | ✅ | ✅ | ❌ | ⚠️ Quota policy |
|
||||
| BazaarLink | openai | API Key | ✅ | ✅ | ❌ | ❌ |
|
||||
| Petals | openai | None | ✅ | ✅ | ❌ | ❌ |
|
||||
| Qoder | openai | OAuth / PAT | ✅ | ✅ | ✅ | ⚠️ Per request |
|
||||
| OpenCode (Go/Zen) | openai | OAuth | ✅ | ✅ | ✅ | ❌ |
|
||||
| CLIProxyAPI | openai | Custom | ✅ | ✅ | ❌ | ❌ |
|
||||
|
||||
## Format Translation Coverage
|
||||
|
||||
|
||||
201
docs/AUTHZ_GUIDE.md
Normal file
201
docs/AUTHZ_GUIDE.md
Normal file
@@ -0,0 +1,201 @@
|
||||
# Authorization Guide
|
||||
|
||||
> **Source of truth:** `src/server/authz/`, `src/shared/constants/publicApiRoutes.ts`, `src/lib/api/requireManagementAuth.ts`, `src/shared/utils/apiAuth.ts`
|
||||
> **Last updated:** 2026-05-13 — v3.8.0
|
||||
|
||||
OmniRoute has a route-aware authorization pipeline that gates every API request. Classification is **deterministic** and **fail-closed** — anything that cannot be classified ends up as `MANAGEMENT` and demands a session or management-grade token. This page explains the model for engineers maintaining routes or designing new endpoints.
|
||||
|
||||
## Two Auth Modes
|
||||
|
||||
### 1. API Key (Bearer)
|
||||
|
||||
Used for the OpenAI/Anthropic/Gemini-compatible client APIs and a few management routes when the key has the `manage` scope.
|
||||
|
||||
```
|
||||
Authorization: Bearer <api-key>
|
||||
```
|
||||
|
||||
Validated by `isValidApiKey()` / `extractApiKey()` in `src/sse/services/auth.ts` and re-exported through `src/shared/utils/apiAuth.ts`. The validator also accepts the `OMNIROUTE_API_KEY` / `ROUTER_API_KEY` env vars as persistent passthrough keys (issue #1350).
|
||||
|
||||
### 2. Dashboard Session (auth_token cookie)
|
||||
|
||||
For dashboard pages and admin operations.
|
||||
|
||||
```
|
||||
Cookie: auth_token=<JWT signed with JWT_SECRET>
|
||||
```
|
||||
|
||||
Verified by `isDashboardSessionAuthenticated()` in `src/shared/utils/apiAuth.ts`. The pipeline auto-refreshes the JWT when it has fewer than 7 days left in its 30-day lifetime.
|
||||
|
||||
Some management routes accept **either** mode: cookie OR `Bearer <key>` when the API key has the `manage` (or `admin`) scope. This is what enables the "configurable via API calls" workflow added in v3.8.
|
||||
|
||||
## Route Classes
|
||||
|
||||
`src/server/authz/types.ts` defines three classes; any route that cannot be classified deterministically falls back to `MANAGEMENT`.
|
||||
|
||||
| Class | Description | Auth required |
|
||||
| ------------ | ---------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------- |
|
||||
| `PUBLIC` | Explicitly safe routes — login, logout, status, init, health, onboarding bootstrap. | None |
|
||||
| `CLIENT_API` | Model-serving endpoints — `/api/v1/*`, plus aliases `/v1/*`, `/chat/completions`, `/responses`, `/models`, `/codex/*`. | Bearer key (unless `REQUIRE_API_KEY != "true"`) |
|
||||
| `MANAGEMENT` | Dashboard pages, settings, providers, keys, admin and diagnostics endpoints. | Dashboard session OR Bearer with `manage` scope |
|
||||
|
||||
## Pipeline
|
||||
|
||||
```
|
||||
Incoming request → src/middleware.ts
|
||||
→ runAuthzPipeline() in src/server/authz/pipeline.ts
|
||||
1. Strip trusted internal headers (x-omniroute-auth-*, x-omniroute-route-class)
|
||||
2. Generate request id, classify route via classifyRoute()
|
||||
3. If pathname == "/" → redirect /dashboard
|
||||
4. If draining (graceful shutdown) and /api/* → 503
|
||||
5. If non-GET /api/* → checkBodySize() guard
|
||||
6. If OPTIONS → CORS preflight 204
|
||||
7. If options.enforce == false → pass-through with route-class headers
|
||||
8. Otherwise: POLICIES[routeClass].evaluate(ctx)
|
||||
- allow → stamp x-omniroute-auth-{kind,id,label,scopes} → NextResponse.next()
|
||||
- reject → JSON error w/ correlation_id (dashboard pages → 302 /login)
|
||||
```
|
||||
|
||||
Trusted internal headers (defined in `src/server/authz/headers.ts`) are **stripped from incoming requests** before classification — clients cannot pre-populate `x-omniroute-auth-*` to impersonate a subject.
|
||||
|
||||
### Policy contracts
|
||||
|
||||
Each route class has a policy in `src/server/authz/policies/`:
|
||||
|
||||
- **`publicPolicy`** (`policies/public.ts`) — always returns `allow({ kind: "anonymous", id: "anonymous" })`.
|
||||
- **`clientApiPolicy`** (`policies/clientApi.ts`) — extracts Bearer, validates via `validateApiKey()`. Falls through to anonymous if `REQUIRE_API_KEY != "true"`. Allows dashboard-session GET on `/api/v1/models` (used by the dashboard model catalog).
|
||||
- **`managementPolicy`** (`policies/management.ts`) — accepts dashboard session, internal model-sync requests (matched against `/api/providers/[name]/(sync-models|models)`), or skips entirely if `isAuthRequired()` returns false. Returns 403 (`AUTH_001`) when a Bearer token is present but invalid, 401 otherwise.
|
||||
|
||||
A successful policy returns `AuthSubject` with `kind ∈ { client_api_key, dashboard_session, management_key, anonymous }`. Downstream handlers can read it via `assertAuth(request, "CLIENT_API")` in `src/server/authz/assertAuth.ts` instead of re-running auth logic.
|
||||
|
||||
## Public Routes List
|
||||
|
||||
`src/shared/constants/publicApiRoutes.ts` is the explicit allowlist:
|
||||
|
||||
```ts
|
||||
PUBLIC_API_ROUTE_PREFIXES = [
|
||||
"/api/auth/login",
|
||||
"/api/auth/logout",
|
||||
"/api/auth/status",
|
||||
"/api/init",
|
||||
"/api/v1/", // treated as CLIENT_API in classify, not as "no-auth public"
|
||||
"/api/cloud/",
|
||||
"/api/sync/bundle",
|
||||
"/api/oauth/",
|
||||
];
|
||||
|
||||
PUBLIC_READONLY_API_ROUTE_PREFIXES = ["/api/monitoring/health", "/api/settings/require-login"];
|
||||
|
||||
PUBLIC_READONLY_METHODS = new Set(["GET", "HEAD", "OPTIONS"]);
|
||||
```
|
||||
|
||||
Read-only prefixes are public **only** for safe methods. Note: `classifyRoute()` excludes `/api/v1/*` from the PUBLIC fall-through — those are always `CLIENT_API` so the Bearer-key policy still applies.
|
||||
|
||||
## Adding a New Route
|
||||
|
||||
### Pattern 1 — Public client API endpoint (Bearer-auth)
|
||||
|
||||
Routes under `/api/v1/` are classified `CLIENT_API` automatically. The middleware enforces the Bearer check; route handlers don't need to redo it but can read the subject if useful.
|
||||
|
||||
```typescript
|
||||
// src/app/api/v1/your-route/route.ts
|
||||
import { NextRequest, NextResponse } from "next/server";
|
||||
import { assertAuth } from "@/server/authz/assertAuth";
|
||||
|
||||
export async function POST(req: NextRequest) {
|
||||
const subject = assertAuth(req, "CLIENT_API");
|
||||
// subject.kind === "client_api_key" | "anonymous" | "dashboard_session"
|
||||
// ... handler logic
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 2 — Management endpoint (session or Bearer + manage)
|
||||
|
||||
Use `requireManagementAuth()` from `src/lib/api/requireManagementAuth.ts`:
|
||||
|
||||
```typescript
|
||||
import { requireManagementAuth } from "@/lib/api/requireManagementAuth";
|
||||
|
||||
export async function POST(request: Request) {
|
||||
const rejection = await requireManagementAuth(request);
|
||||
if (rejection) return rejection;
|
||||
// ... handler logic
|
||||
}
|
||||
```
|
||||
|
||||
`requireManagementAuth()` returns `null` on success or a JSON error `Response`:
|
||||
|
||||
- 401 `AUTH_001` "Authentication required" — no credentials at all
|
||||
- 403 — invalid Bearer **or** Bearer present but key lacks the `manage` / `admin` scope
|
||||
|
||||
`hasManageScope(scopes)` returns true for `"manage"` or `"admin"`.
|
||||
|
||||
### Pattern 3 — Adding to the public allowlist
|
||||
|
||||
Add the prefix to `PUBLIC_API_ROUTE_PREFIXES` (or `PUBLIC_READONLY_API_ROUTE_PREFIXES` for GET-only). Update unit tests at `tests/unit/public-api-routes.test.ts` and `tests/unit/authz/classify.test.ts`.
|
||||
|
||||
## Scopes
|
||||
|
||||
API keys carry a `scopes` array (stored as JSON in `api_keys.scopes`, see `src/lib/db/apiKeys.ts`).
|
||||
|
||||
### Management scope
|
||||
|
||||
- `manage` / `admin` — grants the key access to management API endpoints when sent as Bearer.
|
||||
|
||||
### MCP scopes (`src/shared/constants/mcpScopes.ts`)
|
||||
|
||||
Each MCP tool requires specific scopes via `MCP_TOOL_SCOPES`. Full list (`MCP_SCOPE_LIST`):
|
||||
|
||||
```
|
||||
read:health, read:combos, write:combos, read:quota, read:usage,
|
||||
read:models, execute:completions, execute:search, write:budget,
|
||||
write:resilience, pricing:write, read:cache, write:cache,
|
||||
read:compression, write:compression, read:proxies
|
||||
```
|
||||
|
||||
Preset bundles (`MCP_SCOPE_PRESETS`): `readonly`, `full`, `monitor`, `agent`. Use `hasRequiredScopes(granted, toolName)` and `getMissingScopes()` for enforcement inside MCP handlers.
|
||||
|
||||
## Auth Required Toggle
|
||||
|
||||
`isAuthRequired()` in `src/shared/utils/apiAuth.ts` decides whether **any** auth is enforced for a request:
|
||||
|
||||
- `settings.requireLogin === false` → auth is globally disabled.
|
||||
- No password configured **and** no `INITIAL_PASSWORD` env var → bootstrap mode allows the onboarding wizard and loopback requests, but exposed network requests still need credentials.
|
||||
- Any DB error → fails closed (secure-by-default).
|
||||
|
||||
## Breaking Change — v3.8.0
|
||||
|
||||
The `/api/v1/agents/tasks/*` and `/api/resilience/model-cooldowns` endpoints **now require management auth** (commit `588a0333`). Clients previously sending a normal API key without the `manage` scope receive `403`. Migration: either issue the key the `manage` scope in the API Manager dashboard, or use a logged-in dashboard session.
|
||||
|
||||
## Testing
|
||||
|
||||
- Unit tests: `tests/unit/authz/` — `classify.test.ts`, `pipeline.test.ts`, `client-api-policy.test.ts`, `management-policy.test.ts`, `public-policy.test.ts`.
|
||||
- Public allowlist: `tests/unit/public-api-routes.test.ts`.
|
||||
- Run focused: `node --import tsx/esm --test tests/unit/authz/classify.test.ts`.
|
||||
|
||||
## Debugging
|
||||
|
||||
The pipeline always stamps responses with:
|
||||
|
||||
```
|
||||
x-request-id: <correlation id, echoed in error bodies>
|
||||
x-omniroute-route-class: PUBLIC | CLIENT_API | MANAGEMENT
|
||||
```
|
||||
|
||||
For authenticated requests the upstream (handler-side) request headers also include:
|
||||
|
||||
```
|
||||
x-omniroute-auth-kind: client_api_key | dashboard_session | management_key | anonymous
|
||||
x-omniroute-auth-id: key_<last-4> | "dashboard" | "anonymous"
|
||||
x-omniroute-auth-label: (optional)
|
||||
x-omniroute-auth-scopes: comma-separated list
|
||||
```
|
||||
|
||||
Use `assertAuth(req, expectedClass)` inside handlers — it throws `AuthzAssertionError` with code `AUTHZ_NOT_INITIALIZED` if the middleware was bypassed (helpful for catching configuration regressions in tests).
|
||||
|
||||
## See Also
|
||||
|
||||
- [API_REFERENCE.md](./API_REFERENCE.md) — auth marker per endpoint
|
||||
- [COMPLIANCE.md](./COMPLIANCE.md) — audit log for auth events
|
||||
- [MCP-SERVER.md](./MCP-SERVER.md) — MCP scope enforcement details
|
||||
- Source: `src/server/authz/`, `src/lib/api/requireManagementAuth.ts`
|
||||
@@ -75,25 +75,81 @@ Auto-scoring selects best provider/model per request
|
||||
|
||||
## How It Works (Persisted Auto-Combos)
|
||||
|
||||
The Auto-Combo Engine dynamically selects the best provider/model for each request using a **6-factor scoring function**:
|
||||
The Auto-Combo Engine dynamically selects the best provider/model for each request using a **9-factor scoring function** (defined in `open-sse/services/autoCombo/scoring.ts` → `DEFAULT_WEIGHTS`). All weights sum to **1.0**.
|
||||
|
||||
| Factor | Weight | Description |
|
||||
| :--------- | :----- | :---------------------------------------------- |
|
||||
| Quota | 0.20 | Remaining capacity [0..1] |
|
||||
| Health | 0.25 | Circuit breaker: CLOSED=1.0, HALF=0.5, OPEN=0.0 |
|
||||
| CostInv | 0.20 | Inverse cost (cheaper = higher score) |
|
||||
| LatencyInv | 0.15 | Inverse p95 latency (faster = higher) |
|
||||
| TaskFit | 0.10 | Model × task type fitness score |
|
||||
| Stability | 0.10 | Low variance in latency/errors |
|
||||
| Factor | Default Weight | Description |
|
||||
| :----------------- | :------------- | :---------------------------------------------------------------------- |
|
||||
| `health` | 0.22 | Health score from circuit breaker (CLOSED=1.0, HALF_OPEN=0.5, OPEN=0.0) |
|
||||
| `quota` | 0.17 | Remaining quota / rate-limit headroom [0..1] |
|
||||
| `costInv` | 0.17 | Inverse cost normalized to pool — cheaper = higher score |
|
||||
| `latencyInv` | 0.13 | Inverse p95 latency normalized to pool — faster = higher score |
|
||||
| `taskFit` | 0.08 | Task-type fitness (coding, review, planning, analysis, debugging, docs) |
|
||||
| `specificityMatch` | 0.08 | Match between request specificity (manifest hint) and model tier |
|
||||
| `stability` | 0.05 | Variance-based stability (low latency stdDev / error rate) |
|
||||
| `tierPriority` | 0.05 | Account-tier priority — Ultra=1.0, Pro=0.67, Standard=0.33, Free=0.0 |
|
||||
| `tierAffinity` | 0.05 | Affinity between the candidate's tier and the manifest-recommended tier |
|
||||
|
||||
**Sum:** `0.22 + 0.17 + 0.17 + 0.13 + 0.08 + 0.08 + 0.05 + 0.05 + 0.05 = 1.0` (validated by `validateWeights()`).
|
||||
|
||||
## Mode Packs
|
||||
|
||||
| Pack | Focus | Key Weight |
|
||||
| :---------------------- | :----------- | :--------------- |
|
||||
| 🚀 **Ship Fast** | Speed | latencyInv: 0.35 |
|
||||
| 💰 **Cost Saver** | Economy | costInv: 0.40 |
|
||||
| 🎯 **Quality First** | Best model | taskFit: 0.40 |
|
||||
| 📡 **Offline Friendly** | Availability | quota: 0.40 |
|
||||
Four pre-defined weight profiles in `open-sse/services/autoCombo/modePacks.ts`. Each pack overrides the default weights to bias selection toward a specific goal. Below are the **full weight tables per pack** (each row sums to 1.0).
|
||||
|
||||
| Factor | ship-fast | cost-saver | quality-first | offline-friendly |
|
||||
| :----------- | :-------- | :--------- | :------------ | :--------------- |
|
||||
| quota | 0.15 | 0.15 | 0.10 | **0.40** |
|
||||
| health | 0.30 | 0.20 | 0.20 | 0.30 |
|
||||
| costInv | 0.05 | **0.40** | 0.05 | 0.10 |
|
||||
| latencyInv | **0.35** | 0.05 | 0.05 | 0.05 |
|
||||
| taskFit | 0.10 | 0.10 | **0.40** | 0.00 |
|
||||
| stability | 0.00 | 0.05 | 0.15 | 0.10 |
|
||||
| tierPriority | 0.05 | 0.05 | 0.05 | 0.05 |
|
||||
|
||||
Notes:
|
||||
|
||||
- `tierAffinity` and `specificityMatch` are not set in mode packs — `calculateScore()` treats them as `?? 0` when absent.
|
||||
- Each pack's emphasis at a glance:
|
||||
- **ship-fast** → latencyInv 0.35 + health 0.30 (low-latency, healthy connections)
|
||||
- **cost-saver** → costInv 0.40 (cheapest tokens win)
|
||||
- **quality-first** → taskFit 0.40 + stability 0.15 (best model for the task, consistent)
|
||||
- **offline-friendly** → quota 0.40 + health 0.30 (max headroom regardless of speed/cost)
|
||||
|
||||
## All Routing Strategies
|
||||
|
||||
OmniRoute's combo engine supports **14 routing strategies** (declared in `src/shared/constants/routingStrategies.ts` → `ROUTING_STRATEGY_VALUES`). The Auto Combo engine itself is exposed under the `auto` strategy; the others are available for persisted combos.
|
||||
|
||||
| Strategy | Description |
|
||||
| :------------------ | :----------------------------------------------------------------- |
|
||||
| `priority` | First-target ordered list with explicit priority |
|
||||
| `weighted` | Weighted random by per-target weight |
|
||||
| `round-robin` | Cycle through targets in order |
|
||||
| `context-relay` | Hand off context across targets (long conversations) |
|
||||
| `fill-first` | Fill each target's quota before moving to next |
|
||||
| `p2c` | Power-of-2-choices random load balancing |
|
||||
| `random` | Uniform random selection |
|
||||
| `least-used` | Pick target with lowest current load |
|
||||
| `cost-optimized` | Minimize $ per request given catalog pricing |
|
||||
| `reset-aware` ⭐ | Prioritize by quota reset time — short reset windows ranked higher |
|
||||
| `strict-random` | Random without deduplication of repeats |
|
||||
| `auto` | Use Auto Combo scoring (9-factor) — **recommended** |
|
||||
| `lkgp` | Last-Known-Good Path (sticky route to last successful target) |
|
||||
| `context-optimized` | Pick target with best fit for current context size |
|
||||
|
||||
⭐ = New in v3.8.0
|
||||
|
||||
## Virtual Auto-Combo Factory
|
||||
|
||||
The Auto Combo engine doesn't require pre-defined combos. Instead, `open-sse/services/autoCombo/virtualFactory.ts` builds candidates on-the-fly:
|
||||
|
||||
1. Pulls `getProviderConnections({ isActive: true })` (all enabled connections)
|
||||
2. Filters to those with valid credentials (API key or non-expired OAuth token via `hasUsableOAuthToken()`)
|
||||
3. Cross-references with `getProviderRegistry()` for model availability + pricing
|
||||
4. For each tuple `(provider, model, connection)`, builds a `VirtualAutoComboCandidate`
|
||||
5. Picks `connection.defaultModel` (or the registry's first model) as the dispatch target
|
||||
6. Scores each candidate using the 9-factor `scorePool()` and the variant's weight pack
|
||||
7. Returns the resulting in-memory `AutoComboConfig` for `handleComboChat()` — never persisted to DB
|
||||
|
||||
This means **adding a new provider with `auto/*` enabled automatically expands the candidate pool** — no manual combo editing needed. The virtual combo is rebuilt per request, so newly-added or newly-healthy connections are picked up immediately.
|
||||
|
||||
## Self-Healing
|
||||
|
||||
@@ -108,27 +164,48 @@ The Auto-Combo Engine dynamically selects the best provider/model for each reque
|
||||
|
||||
## API
|
||||
|
||||
```bash
|
||||
# Create auto-combo
|
||||
curl -X POST http://localhost:20128/api/combos/auto \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"id":"my-auto","name":"Auto Coder","candidatePool":["anthropic","google","openai"],"modePack":"ship-fast"}'
|
||||
There is **no dedicated `POST /api/combos/auto` endpoint** — Auto-Combo is consumed in two ways:
|
||||
|
||||
# List auto-combos
|
||||
curl http://localhost:20128/api/combos/auto
|
||||
1. **Zero-config (recommended):** Send any chat completion request with `model: "auto"` or `model: "auto/<variant>"`. The virtual factory builds the combo per request — no persistence, no API calls needed.
|
||||
|
||||
2. **Persisted combo with `strategy: "auto"`:** Create a regular combo via `POST /api/combos` and set `strategy: "auto"` plus `config.auto.weights` / `config.auto.candidatePool`. The same scoring engine is used; the combo is stored in `combos` and reusable by ID.
|
||||
|
||||
```bash
|
||||
# Zero-config usage (no combo creation)
|
||||
curl -X POST http://localhost:20128/v1/chat/completions \
|
||||
-H "Authorization: Bearer <key>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model":"auto/coding","messages":[{"role":"user","content":"Hello"}]}'
|
||||
|
||||
# Persisted auto combo via the regular combos endpoint
|
||||
curl -X POST http://localhost:20128/api/combos \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"id":"my-auto","name":"Auto Coder","strategy":"auto","config":{"auto":{"candidatePool":["anthropic","google","openai"],"weights":{"quota":0.15,"health":0.3,"costInv":0.05,"latencyInv":0.35,"taskFit":0.1,"stability":0,"tierPriority":0.05}}}}'
|
||||
```
|
||||
|
||||
## Task Fitness
|
||||
|
||||
30+ models scored across 6 task types (`coding`, `review`, `planning`, `analysis`, `debugging`, `documentation`). Supports wildcard patterns (e.g., `*-coder` → high coding score).
|
||||
|
||||
## Auto Variants Recap
|
||||
|
||||
Including the bare `auto` (default) plus the 6 `AutoVariant` values declared in `autoPrefix.ts`, there are **7 invokable model IDs**:
|
||||
|
||||
`auto`, `auto/coding`, `auto/fast`, `auto/cheap`, `auto/offline`, `auto/smart`, `auto/lkgp`
|
||||
|
||||
(`AutoVariant` itself enumerates 6 values; the 7th option is "no variant" — bare `auto` — handled by `parseAutoPrefix()` as `variant: undefined`.)
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
| :------------------------------------------- | :------------------------------------ |
|
||||
| `open-sse/services/autoCombo/scoring.ts` | Scoring function & pool normalization |
|
||||
| `open-sse/services/autoCombo/taskFitness.ts` | Model × task fitness lookup |
|
||||
| `open-sse/services/autoCombo/engine.ts` | Selection logic, bandit, budget cap |
|
||||
| `open-sse/services/autoCombo/selfHealing.ts` | Exclusion, probes, incident mode |
|
||||
| `open-sse/services/autoCombo/modePacks.ts` | 4 weight profiles |
|
||||
| `src/app/api/combos/auto/route.ts` | REST API |
|
||||
| File | Purpose |
|
||||
| :-------------------------------------------------------- | :------------------------------------------------------------------------- |
|
||||
| `open-sse/services/autoCombo/scoring.ts` | 9-factor scoring function, `DEFAULT_WEIGHTS`, pool norm |
|
||||
| `open-sse/services/autoCombo/taskFitness.ts` | Model × task fitness lookup |
|
||||
| `open-sse/services/autoCombo/engine.ts` | Selection logic, bandit, budget cap |
|
||||
| `open-sse/services/autoCombo/selfHealing.ts` | Exclusion, probes, incident mode |
|
||||
| `open-sse/services/autoCombo/modePacks.ts` | 4 weight profiles (ship-fast, cost-saver, quality-first, offline-friendly) |
|
||||
| `open-sse/services/autoCombo/autoPrefix.ts` | `auto/` prefix parser + 6 variants |
|
||||
| `open-sse/services/autoCombo/virtualFactory.ts` | Builds in-memory `AutoComboConfig` from live connections |
|
||||
| `open-sse/services/autoCombo/providerRegistryAccessor.ts` | Test hook for mocking provider registry |
|
||||
| `src/shared/constants/routingStrategies.ts` | `ROUTING_STRATEGY_VALUES` (14 strategies) |
|
||||
| `src/sse/handlers/chat.ts` | Integration: auto-prefix short-circuit |
|
||||
|
||||
@@ -1,15 +1,18 @@
|
||||
# CLI Tools Setup Guide — OmniRoute
|
||||
# CLI Tools — OmniRoute v3.8.0
|
||||
|
||||
This guide explains how to install and configure all supported AI coding CLI tools
|
||||
to use **OmniRoute** as the unified backend, giving you centralized key management,
|
||||
cost tracking, model switching, and request logging across every tool.
|
||||
Last updated: 2026-05-13
|
||||
|
||||
OmniRoute integrates with two categories of CLI tools:
|
||||
|
||||
1. **External CLI integrations** — third-party CLIs (Cursor, Cline, Codex, Claude Code, Qwen Code, Windsurf, Hermes, Amp, etc.) that you point at OmniRoute's local OpenAI-compatible endpoint.
|
||||
2. **Internal OmniRoute CLI** — commands bundled with the `omniroute` binary for server lifecycle, setup, diagnostics, and provider management.
|
||||
|
||||
---
|
||||
|
||||
## How It Works
|
||||
|
||||
```
|
||||
Claude / Codex / OpenCode / Cline / KiloCode / Continue / Kiro / Cursor / Copilot
|
||||
Claude / Codex / OpenCode / Cline / KiloCode / Continue / Cursor / Windsurf / Hermes / Amp / Qwen
|
||||
│
|
||||
▼ (all point to OmniRoute)
|
||||
http://YOUR_SERVER:20128/v1
|
||||
@@ -23,47 +26,70 @@ Claude / Codex / OpenCode / Cline / KiloCode / Continue / Kiro / Cursor / Copilo
|
||||
- One API key to manage all tools
|
||||
- Cost tracking across all CLIs in the dashboard
|
||||
- Model switching without reconfiguring every tool
|
||||
- Works locally and on remote servers (VPS)
|
||||
- Works locally and on remote servers (VPS, Docker, Akamai, Cloudflare Tunnel)
|
||||
|
||||
---
|
||||
|
||||
## Supported Tools (Dashboard Source of Truth)
|
||||
## 1. External CLI Integrations
|
||||
|
||||
The dashboard cards in `/dashboard/cli-tools` are generated from `src/shared/constants/cliTools.ts`.
|
||||
Current list (v3.0.0-rc.16):
|
||||
### Source of Truth
|
||||
|
||||
| Tool | ID | Command | Setup Mode | Install Method |
|
||||
| ------------------ | ------------- | ---------- | ---------- | -------------- |
|
||||
| **Claude Code** | `claude` | `claude` | env | npm |
|
||||
| **OpenAI Codex** | `codex` | `codex` | custom | npm |
|
||||
| **Factory Droid** | `droid` | `droid` | custom | bundled/CLI |
|
||||
| **OpenClaw** | `openclaw` | `openclaw` | custom | bundled/CLI |
|
||||
| **Cursor** | `cursor` | app | guide | desktop app |
|
||||
| **Cline** | `cline` | `cline` | custom | npm |
|
||||
| **Kilo Code** | `kilo` | `kilocode` | custom | npm |
|
||||
| **Continue** | `continue` | extension | guide | VS Code |
|
||||
| **Antigravity** | `antigravity` | internal | mitm | OmniRoute |
|
||||
| **GitHub Copilot** | `copilot` | extension | custom | VS Code |
|
||||
| **OpenCode** | `opencode` | `opencode` | guide | npm |
|
||||
| **Kiro AI** | `kiro` | app/cli | mitm | desktop/CLI |
|
||||
| **Qwen Code** | `qwen` | `qwen` | custom | npm |
|
||||
The dashboard cards in `/dashboard/cli-tools` are generated from
|
||||
`src/shared/constants/cliTools.ts`. The internal helper `bin/cli-commands.mjs`
|
||||
keeps the small set of "fully scriptable" tools that `omniroute setup` can write
|
||||
config files for automatically.
|
||||
|
||||
### Current Catalog (v3.8.0)
|
||||
|
||||
| Tool | ID | Type / Config | Install / Access | Auth |
|
||||
| ------------------ | ------------- | ------------------- | ------------------------------------ | ----------------------------------- |
|
||||
| **Claude Code** | `claude` | env / settings.json | `npm i -g @anthropic-ai/claude-code` | API key (Anthropic gateway) |
|
||||
| **OpenAI Codex** | `codex` | custom (toml) | `npm i -g @openai/codex` | API key (OpenAI) |
|
||||
| **Factory Droid** | `droid` | custom | bundled / CLI | API key |
|
||||
| **Open Claw** | `openclaw` | custom | bundled / CLI | API key |
|
||||
| **Cursor** | `cursor` | guide (Cloud) | Cursor desktop app | API key (Cloud Endpoint) |
|
||||
| **Windsurf** | `windsurf` | guide | Windsurf desktop IDE | API key (BYOK) |
|
||||
| **Cline** | `cline` | custom / VS Code | `npm i -g cline` + VS Code ext | API key |
|
||||
| **Kilo Code** | `kilo` | custom / VS Code | `npm i -g kilocode` + VS Code ext | API key |
|
||||
| **Continue** | `continue` | guide (config.yaml) | VS Code extension | API key |
|
||||
| **Antigravity** | `antigravity` | MITM | OmniRoute built-in | API key (MITM proxy) |
|
||||
| **GitHub Copilot** | `copilot` | custom / VS Code | VS Code extension | API key (CLI fingerprint: `github`) |
|
||||
| **OpenCode** | `opencode` | guide (json) | `npm i -g opencode-ai` | API key (OpenAI-compatible) |
|
||||
| **Hermes** | `hermes` | guide (json) | install per docs | API key (OpenAI-compatible) |
|
||||
| **Amp CLI** | `amp` | guide (env) | install per Sourcegraph docs | API key (OpenAI-compatible) |
|
||||
| **Kiro AI** | `kiro` | MITM | Amazon Kiro IDE / CLI | API key (MITM proxy) |
|
||||
| **Qwen Code** | `qwen` | guide (json/env) | `npm i -g @qwen-code/qwen-code` | API key (OpenAI-compatible) |
|
||||
| **Custom CLI** | `custom` | custom-builder | any OpenAI-compatible client | API key |
|
||||
|
||||
> Notes:
|
||||
>
|
||||
> - "Web wrappers" like ChatGPT/Claude/Grok/Perplexity browser sessions are not
|
||||
> listed here. OmniRoute can proxy them through the `chatgpt-web`,
|
||||
> `claude-web`, `grok-web`, `perplexity-web`, `blackbox-web`,
|
||||
> `muse-spark-web` provider connections, but those are **provider connections**
|
||||
> (configured under `/dashboard/providers`), not CLI tools. They do not surface
|
||||
> as cards under `/dashboard/cli-tools`.
|
||||
> - Tools marked **MITM** (Antigravity, Kiro) intercept the desktop app traffic
|
||||
> locally and require enabling the corresponding mitm endpoint in
|
||||
> `/dashboard/settings`.
|
||||
|
||||
### CLI fingerprint sync (Agents + Settings)
|
||||
|
||||
`/dashboard/agents` and `Settings > CLI Fingerprint` use `src/shared/constants/cliCompatProviders.ts`.
|
||||
This keeps provider IDs aligned with CLI cards and legacy IDs.
|
||||
`/dashboard/agents` and `Settings > CLI Fingerprint` use
|
||||
`src/shared/constants/cliCompatProviders.ts`. This keeps provider IDs aligned
|
||||
with the CLI cards and legacy IDs.
|
||||
|
||||
| CLI ID | Fingerprint Provider ID |
|
||||
| ---------------------------------------------------------------------------------------------------- | ----------------------- |
|
||||
| `kilo` | `kilocode` |
|
||||
| `copilot` | `github` |
|
||||
| `claude` / `codex` / `antigravity` / `kiro` / `cursor` / `cline` / `opencode` / `droid` / `openclaw` | same ID |
|
||||
| CLI ID | Fingerprint Provider ID |
|
||||
| --------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- |
|
||||
| `kilo` | `kilocode` |
|
||||
| `copilot` | `github` |
|
||||
| `claude` / `codex` / `antigravity` / `kiro` / `cursor` / `windsurf` / `cline` / `opencode` / `hermes` / `amp` / `qwen` / `droid` / `openclaw` | same ID |
|
||||
|
||||
Legacy IDs still accepted for compatibility: `copilot`, `kimi-coding`, `qwen`.
|
||||
|
||||
---
|
||||
|
||||
## Step 1 — Get an OmniRoute API Key
|
||||
### Step 1 — Get an OmniRoute API Key
|
||||
|
||||
1. Open the OmniRoute dashboard → **API Manager** (`/dashboard/api-manager`)
|
||||
2. Click **Create API Key**
|
||||
@@ -74,9 +100,9 @@ Legacy IDs still accepted for compatibility: `copilot`, `kimi-coding`, `qwen`.
|
||||
|
||||
---
|
||||
|
||||
## Step 2 — Install CLI Tools
|
||||
### Step 2 — Install CLI Tools
|
||||
|
||||
All npm-based tools require Node.js 18+:
|
||||
All npm-based tools require Node.js 20.20.2+, 22.22.2+ or 24.x:
|
||||
|
||||
```bash
|
||||
# Claude Code (Anthropic)
|
||||
@@ -94,6 +120,9 @@ npm install -g cline
|
||||
# KiloCode
|
||||
npm install -g kilocode
|
||||
|
||||
# Qwen Code (Alibaba)
|
||||
npm install -g @qwen-code/qwen-code
|
||||
|
||||
# Kiro CLI (Amazon — requires curl + unzip)
|
||||
apt-get install -y unzip # on Debian/Ubuntu
|
||||
curl -fsSL https://cli.kiro.dev/install | bash
|
||||
@@ -108,12 +137,13 @@ codex --version # 0.x.x
|
||||
opencode --version # x.x.x
|
||||
cline --version # 2.x.x
|
||||
kilocode --version # x.x.x (or: kilo --version)
|
||||
qwen --version # x.x.x
|
||||
kiro-cli --version # 1.x.x
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 3 — Set Global Environment Variables
|
||||
### Step 3 — Set Global Environment Variables
|
||||
|
||||
Add to `~/.bashrc` (or `~/.zshrc`), then run `source ~/.bashrc`:
|
||||
|
||||
@@ -132,9 +162,9 @@ export GEMINI_API_KEY="sk-your-omniroute-key"
|
||||
|
||||
---
|
||||
|
||||
## Step 4 — Configure Each Tool
|
||||
### Step 4 — Configure Each Tool
|
||||
|
||||
### Claude Code
|
||||
#### Claude Code
|
||||
|
||||
```bash
|
||||
# Create ~/.claude/settings.json:
|
||||
@@ -154,7 +184,7 @@ Use the unified Anthropic gateway root for Claude Code. Do not append `/v1` here
|
||||
|
||||
---
|
||||
|
||||
### OpenAI Codex
|
||||
#### OpenAI Codex
|
||||
|
||||
```bash
|
||||
mkdir -p ~/.codex && cat > ~/.codex/config.yaml << EOF
|
||||
@@ -168,21 +198,39 @@ EOF
|
||||
|
||||
---
|
||||
|
||||
### OpenCode
|
||||
#### OpenCode
|
||||
|
||||
```bash
|
||||
mkdir -p ~/.config/opencode && cat > ~/.config/opencode/config.toml << EOF
|
||||
[provider.openai]
|
||||
base_url = "http://localhost:20128/v1"
|
||||
api_key = "sk-your-omniroute-key"
|
||||
mkdir -p ~/.config/opencode && cat > ~/.config/opencode/opencode.json << EOF
|
||||
{
|
||||
"\$schema": "https://opencode.ai/config.json",
|
||||
"provider": {
|
||||
"omniroute": {
|
||||
"npm": "@ai-sdk/openai-compatible",
|
||||
"name": "OmniRoute",
|
||||
"options": {
|
||||
"baseURL": "http://localhost:20128/v1",
|
||||
"apiKey": "sk-your-omniroute-key"
|
||||
},
|
||||
"models": {
|
||||
"claude-sonnet-4-5": { "name": "claude-sonnet-4-5" },
|
||||
"claude-sonnet-4-5-thinking": { "name": "claude-sonnet-4-5-thinking" },
|
||||
"gemini-3-flash": { "name": "gemini-3-flash" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
EOF
|
||||
```
|
||||
|
||||
**Test:** `opencode`
|
||||
|
||||
> Use `opencode run "your prompt" --model omniroute/claude-sonnet-4-5-thinking --variant high`
|
||||
> to send thinking variants.
|
||||
|
||||
---
|
||||
|
||||
### Cline (CLI or VS Code)
|
||||
#### Cline (CLI or VS Code)
|
||||
|
||||
**CLI mode:**
|
||||
|
||||
@@ -203,7 +251,7 @@ Or use the OmniRoute dashboard → **CLI Tools → Cline → Apply Config**.
|
||||
|
||||
---
|
||||
|
||||
### KiloCode (CLI or VS Code)
|
||||
#### KiloCode (CLI or VS Code)
|
||||
|
||||
**CLI mode:**
|
||||
|
||||
@@ -224,7 +272,7 @@ Or use the OmniRoute dashboard → **CLI Tools → KiloCode → Apply Config**.
|
||||
|
||||
---
|
||||
|
||||
### Continue (VS Code Extension)
|
||||
#### Continue (VS Code Extension)
|
||||
|
||||
Edit `~/.continue/config.yaml`:
|
||||
|
||||
@@ -242,7 +290,7 @@ Restart VS Code after editing.
|
||||
|
||||
---
|
||||
|
||||
### Kiro CLI (Amazon)
|
||||
#### Kiro CLI (Amazon)
|
||||
|
||||
```bash
|
||||
# Login to your AWS/Kiro account:
|
||||
@@ -253,12 +301,18 @@ kiro-cli login
|
||||
kiro-cli status
|
||||
```
|
||||
|
||||
For the **Kiro IDE** desktop app, use the MITM endpoint exposed by OmniRoute
|
||||
under `/dashboard/cli-tools → Kiro`.
|
||||
|
||||
---
|
||||
|
||||
### Qwen Code (Alibaba)
|
||||
#### Qwen Code (Alibaba)
|
||||
|
||||
Qwen Code supports OpenAI-compatible API endpoints via environment variables or `settings.json`.
|
||||
|
||||
> Qwen OAuth free tier was discontinued on 2026-04-15. Use OmniRoute with
|
||||
> `alicode` / `openrouter` / `anthropic` / `gemini` providers instead.
|
||||
|
||||
**Option 1: Environment variables (`~/.qwen/.env`)**
|
||||
|
||||
```bash
|
||||
@@ -269,24 +323,20 @@ OPENAI_MODEL="auto"
|
||||
EOF
|
||||
```
|
||||
|
||||
**Option 2: `settings.json` with model providers**
|
||||
**Option 2: `settings.json` with `security.auth`**
|
||||
|
||||
```json
|
||||
// ~/.qwen/settings.json
|
||||
{
|
||||
"env": {
|
||||
"OPENAI_API_KEY": "sk-your-omniroute-key",
|
||||
"OPENAI_BASE_URL": "http://localhost:20128/v1"
|
||||
"security": {
|
||||
"auth": {
|
||||
"selectedType": "openai",
|
||||
"apiKey": "sk-your-omniroute-key",
|
||||
"baseUrl": "http://localhost:20128/v1"
|
||||
}
|
||||
},
|
||||
"modelProviders": {
|
||||
"openai": [
|
||||
{
|
||||
"id": "omniroute-default",
|
||||
"name": "OmniRoute (Auto)",
|
||||
"envKey": "OPENAI_API_KEY",
|
||||
"baseUrl": "http://localhost:20128/v1"
|
||||
}
|
||||
]
|
||||
"model": {
|
||||
"name": "claude-sonnet-4-6"
|
||||
}
|
||||
}
|
||||
```
|
||||
@@ -304,7 +354,9 @@ qwen
|
||||
|
||||
**Test:** `qwen "say hello"`
|
||||
|
||||
### Cursor (Desktop App)
|
||||
---
|
||||
|
||||
#### Cursor (Desktop App)
|
||||
|
||||
> **Note:** Cursor routes requests through its cloud. For OmniRoute integration,
|
||||
> enable **Cloud Endpoint** in OmniRoute Settings and use your public domain URL.
|
||||
@@ -316,22 +368,69 @@ Via GUI: **Settings → Models → OpenAI API Key**
|
||||
|
||||
---
|
||||
|
||||
## Dashboard Auto-Configuration
|
||||
#### Windsurf (Desktop IDE)
|
||||
|
||||
> Official Windsurf docs currently describe BYOK for select Claude models plus
|
||||
> enterprise URL/token settings, not a generic custom OpenAI-compatible provider.
|
||||
> Test BYOK behavior in your environment before relying on this integration.
|
||||
|
||||
1. Open AI Settings inside Windsurf.
|
||||
2. Select **Add custom provider** (OpenAI-compatible).
|
||||
3. Base URL: `http://localhost:20128/v1`
|
||||
4. API Key: your OmniRoute key
|
||||
5. Pick a model from the OmniRoute catalog.
|
||||
|
||||
---
|
||||
|
||||
#### Hermes
|
||||
|
||||
```json
|
||||
// Hermes config file
|
||||
{
|
||||
"provider": {
|
||||
"type": "openai",
|
||||
"baseURL": "http://localhost:20128/v1",
|
||||
"apiKey": "sk-your-omniroute-key",
|
||||
"model": "claude-sonnet-4-6"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### Amp CLI (Sourcegraph)
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY="sk-your-omniroute-key"
|
||||
export OPENAI_BASE_URL="http://localhost:20128/v1"
|
||||
amp --model "claude-sonnet-4-6"
|
||||
|
||||
# Suggested shorthand aliases you can map locally:
|
||||
# g25p -> gemini/gemini-2.5-pro
|
||||
# g25f -> gemini/gemini-2.5-flash
|
||||
# cs45 -> cc/claude-sonnet-4-5-20250929
|
||||
# g54 -> gemini/gemini-3.1-pro-high
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Dashboard Auto-Configuration
|
||||
|
||||
The OmniRoute dashboard automates configuration for most tools:
|
||||
|
||||
1. Go to `http://localhost:20128/dashboard/cli-tools`
|
||||
2. Expand any tool card
|
||||
3. Select your API key from the dropdown
|
||||
4. Click **Apply Config** (if tool is detected as installed)
|
||||
4. Click **Apply Config** (if the tool is detected as installed)
|
||||
5. Or copy the generated config snippet manually
|
||||
|
||||
---
|
||||
|
||||
## Built-in Agents: Droid & OpenClaw
|
||||
### Built-in Agents: Droid & Open Claw
|
||||
|
||||
**Droid** and **OpenClaw** are AI agents built directly into OmniRoute — no installation needed.
|
||||
They run as internal routes and use OmniRoute's model routing automatically.
|
||||
**Droid** and **Open Claw** are AI agents built directly into OmniRoute — no
|
||||
installation needed. They run as internal routes and use OmniRoute's model
|
||||
routing automatically.
|
||||
|
||||
- Access: `http://localhost:20128/dashboard/agents`
|
||||
- Configure: same combos and providers as all other tools
|
||||
@@ -339,6 +438,152 @@ They run as internal routes and use OmniRoute's model routing automatically.
|
||||
|
||||
---
|
||||
|
||||
## 2. Internal OmniRoute CLI
|
||||
|
||||
The `omniroute` binary (installed via `npm install -g omniroute` or bundled
|
||||
with the desktop app) provides commands beyond running the server. The full
|
||||
matrix is implemented in:
|
||||
|
||||
- `bin/omniroute.mjs` — entry point and `--help` text
|
||||
- `bin/cli/index.mjs` — dispatcher for the supported subcommands
|
||||
- `bin/cli/commands/setup.mjs`, `bin/cli/commands/doctor.mjs`,
|
||||
`bin/cli/commands/providers.mjs` — the three core subcommands
|
||||
|
||||
Other subcommands listed in `--help` (status, logs, combo, keys, mcp, a2a,
|
||||
tunnel, backup, restore, quota, health, cache, env, completion, dashboard,
|
||||
serve, stop, restart, open, update, test) are wired through
|
||||
`bin/cli-commands.mjs` and require a running server for most of them.
|
||||
|
||||
### Server Lifecycle
|
||||
|
||||
```bash
|
||||
omniroute # Start server (default port 20128)
|
||||
omniroute --port 3000 # Override port
|
||||
omniroute --no-open # Don't auto-open browser
|
||||
omniroute --mcp # Start as MCP server (stdio transport)
|
||||
omniroute serve # Same as `omniroute`
|
||||
omniroute stop # Stop the running server
|
||||
omniroute restart # Restart the server
|
||||
omniroute dashboard # Open dashboard in default browser
|
||||
omniroute open # Alias for `dashboard`
|
||||
omniroute --version # Print version
|
||||
omniroute --help # Show all commands
|
||||
```
|
||||
|
||||
### Setup & Initialization
|
||||
|
||||
```bash
|
||||
omniroute setup # Interactive setup wizard
|
||||
omniroute setup --non-interactive # CI/automation mode (reads env vars + flags)
|
||||
omniroute setup --password '<value>' # Set admin password directly
|
||||
omniroute setup --add-provider \
|
||||
--provider openai \
|
||||
--api-key '<value>' \
|
||||
--test-provider # Add and test a provider in one shot
|
||||
```
|
||||
|
||||
Recognized environment variables for non-interactive setup:
|
||||
|
||||
| Var | Purpose |
|
||||
| ----------------------------- | -------------------------------------------- |
|
||||
| `OMNIROUTE_SETUP_PASSWORD` | Admin password (>=8 chars) |
|
||||
| `OMNIROUTE_PROVIDER` | Provider id (e.g. `openai`, `anthropic`) |
|
||||
| `OMNIROUTE_PROVIDER_NAME` | Display name for the connection |
|
||||
| `OMNIROUTE_PROVIDER_BASE_URL` | Optional OpenAI-compatible base URL override |
|
||||
| `OMNIROUTE_API_KEY` | Provider API key |
|
||||
| `OMNIROUTE_DEFAULT_MODEL` | Optional default model |
|
||||
| `DATA_DIR` | Override the OmniRoute data directory |
|
||||
|
||||
### Diagnostics
|
||||
|
||||
```bash
|
||||
omniroute doctor # Check config, DB, ports, runtime, memory, liveness
|
||||
omniroute doctor --json # Machine-readable JSON
|
||||
omniroute doctor --no-liveness # Skip the HTTP health probe
|
||||
omniroute doctor --host 0.0.0.0 # Override liveness host
|
||||
omniroute doctor --liveness-url <url> # Full health endpoint URL override
|
||||
```
|
||||
|
||||
The doctor runs these checks: `Config`, `Database`, `Storage/encryption`,
|
||||
`Port availability`, `Node runtime`, `Native binary` (better-sqlite3),
|
||||
`Memory`, and `Server liveness`. It exits non-zero if any check is `fail`.
|
||||
|
||||
### Provider Management
|
||||
|
||||
```bash
|
||||
omniroute providers available # OmniRoute provider catalog
|
||||
omniroute providers available --search openai # Filter catalog by id/name/alias/category
|
||||
omniroute providers available --category api-key # Filter by category (api-key, oauth, free, ...)
|
||||
omniroute providers available --json # Machine-readable JSON
|
||||
|
||||
omniroute providers list # Configured provider connections
|
||||
omniroute providers list --json
|
||||
|
||||
omniroute providers test <id|name> # Test one configured connection
|
||||
omniroute providers test-all # Test every active connection
|
||||
omniroute providers validate # Local-only structural validation
|
||||
```
|
||||
|
||||
> `providers available` reads the OmniRoute catalog; `providers list/test/test-all/validate`
|
||||
> read the local SQLite database directly and do not require the server to be running.
|
||||
|
||||
### Recovery & Reset
|
||||
|
||||
```bash
|
||||
omniroute reset-password # Reset the admin password (legacy alias still works)
|
||||
omniroute reset-encrypted-columns # Show warning + dry-run for encrypted credential reset
|
||||
omniroute reset-encrypted-columns --force # Actually null out encrypted credentials in SQLite
|
||||
```
|
||||
|
||||
### Other subcommands (via `cli-commands.mjs`)
|
||||
|
||||
These are dispatched in `bin/cli-commands.mjs` and assume a running OmniRoute
|
||||
server, unless noted otherwise:
|
||||
|
||||
```bash
|
||||
omniroute status # Comprehensive runtime status
|
||||
omniroute logs # Stream request logs (--json, --search, --follow)
|
||||
omniroute config show # Display current configuration
|
||||
|
||||
omniroute provider list # List available providers (alias of providers list)
|
||||
omniroute provider add # Register OmniRoute as a provider on a tool
|
||||
omniroute keys add | list | remove # Manage API keys
|
||||
omniroute models [provider] # List models (--json, --search)
|
||||
omniroute combo list | switch | create | delete
|
||||
|
||||
omniroute backup # Snapshot config + DB
|
||||
omniroute restore # Restore from a previous snapshot
|
||||
|
||||
omniroute health # Detailed health (breakers, cache, memory)
|
||||
omniroute quota # Provider quota usage
|
||||
omniroute cache # Cache status
|
||||
omniroute cache clear # Clear semantic + signature caches
|
||||
|
||||
omniroute mcp status | restart # MCP server status / restart
|
||||
omniroute a2a status | card # A2A server status / agent card
|
||||
|
||||
omniroute tunnel list | create | stop # Manage tunnels (cloudflare/tailscale/ngrok)
|
||||
omniroute env show | get <k> | set <k> <v> # Inspect / set env vars (temporary)
|
||||
|
||||
omniroute test # Provider connectivity smoke test
|
||||
omniroute update # Check for updates
|
||||
omniroute completion # Generate shell completion
|
||||
```
|
||||
|
||||
### Common flags
|
||||
|
||||
| Flag | Description |
|
||||
| ------------------- | ------------------------------------------------------ |
|
||||
| `--no-open` | Don't auto-open the browser on start |
|
||||
| `--port <n>` | Override the API port (default 20128) |
|
||||
| `--mcp` | Run as MCP server over stdio (for IDEs) |
|
||||
| `--non-interactive` | CI mode (no prompts; reads from env/flags) |
|
||||
| `--json` | Machine-readable JSON output (doctor, providers, etc.) |
|
||||
| `--help`, `-h` | Show command-specific help |
|
||||
| `--version`, `-v` | Print the installed version |
|
||||
|
||||
---
|
||||
|
||||
## Available API Endpoints
|
||||
|
||||
| Endpoint | Description | Use For |
|
||||
@@ -355,42 +600,89 @@ They run as internal routes and use OmniRoute's model routing automatically.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Error | Cause | Fix |
|
||||
| ------------------------- | ----------------------- | ------------------------------------------ |
|
||||
| `Connection refused` | OmniRoute not running | `pm2 start omniroute` |
|
||||
| `401 Unauthorized` | Wrong API key | Check in `/dashboard/api-manager` |
|
||||
| `No combo configured` | No active routing combo | Set up in `/dashboard/combos` |
|
||||
| `invalid model` | Model not in catalog | Use `auto` or check `/dashboard/providers` |
|
||||
| CLI shows "not installed" | Binary not in PATH | Check `which <command>` |
|
||||
| `kiro-cli: not found` | Not in PATH | `export PATH="$HOME/.local/bin:$PATH"` |
|
||||
| Error | Cause | Fix |
|
||||
| ------------------------------------------------- | --------------------------- | --------------------------------------------------------------------------- |
|
||||
| `Connection refused` | OmniRoute not running | `omniroute serve` or `pm2 start omniroute` |
|
||||
| `401 Unauthorized` | Wrong API key | Check in `/dashboard/api-manager` |
|
||||
| `No combo configured` | No active routing combo | Set up in `/dashboard/combos` |
|
||||
| `invalid model` | Model not in catalog | Use `auto` or check `/dashboard/providers` |
|
||||
| CLI shows "not installed" | Binary not in PATH | Check `which <command>` |
|
||||
| `kiro-cli: not found` | Not in PATH | `export PATH="$HOME/.local/bin:$PATH"` |
|
||||
| `doctor` reports SQLite incompatible | Wrong native binary | `cd app && npm rebuild better-sqlite3` |
|
||||
| `doctor` reports `STORAGE_ENCRYPTION_KEY` missing | Encrypted creds without key | Set `STORAGE_ENCRYPTION_KEY` or `omniroute reset-encrypted-columns --force` |
|
||||
|
||||
---
|
||||
|
||||
## Quick Setup Script (One Command)
|
||||
|
||||
```bash
|
||||
# Install all CLIs and configure for OmniRoute (replace with your key and server URL)
|
||||
cat > my-setup.sh <<'EOF'
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# === Edit these ===
|
||||
OMNIROUTE_URL="http://localhost:20128/v1"
|
||||
OMNIROUTE_ANTHROPIC_URL="http://localhost:20128"
|
||||
OMNIROUTE_KEY="sk-your-omniroute-key"
|
||||
# ==================
|
||||
|
||||
npm install -g @anthropic-ai/claude-code @openai/codex opencode-ai cline kilocode @qwen-code/qwen-code
|
||||
# 1. Install the external CLIs
|
||||
npm install -g \
|
||||
@anthropic-ai/claude-code \
|
||||
@openai/codex \
|
||||
opencode-ai \
|
||||
cline \
|
||||
kilocode \
|
||||
@qwen-code/qwen-code
|
||||
|
||||
# Kiro CLI
|
||||
apt-get install -y unzip 2>/dev/null; curl -fsSL https://cli.kiro.dev/install | bash
|
||||
# 2. Optional: Kiro CLI (needs unzip)
|
||||
if ! command -v unzip >/dev/null 2>&1; then
|
||||
sudo apt-get install -y unzip
|
||||
fi
|
||||
curl -fsSL https://cli.kiro.dev/install | bash
|
||||
|
||||
# Write configs
|
||||
mkdir -p ~/.claude ~/.codex ~/.config/opencode ~/.continue
|
||||
# 3. Write the per-tool config files
|
||||
mkdir -p ~/.claude ~/.codex ~/.config/opencode ~/.continue ~/.qwen
|
||||
|
||||
cat > ~/.claude/settings.json <<< "{\"env\":{\"ANTHROPIC_BASE_URL\":\"$OMNIROUTE_ANTHROPIC_URL\",\"ANTHROPIC_AUTH_TOKEN\":\"$OMNIROUTE_KEY\"}}"
|
||||
cat > ~/.codex/config.yaml <<< "model: auto\napiKey: $OMNIROUTE_KEY\napiBaseUrl: $OMNIROUTE_URL"
|
||||
cat >> ~/.bashrc << EOF
|
||||
export OPENAI_BASE_URL="$OMNIROUTE_URL"
|
||||
export OPENAI_API_KEY="$OMNIROUTE_KEY"
|
||||
export ANTHROPIC_BASE_URL="$OMNIROUTE_ANTHROPIC_URL"
|
||||
export ANTHROPIC_AUTH_TOKEN="$OMNIROUTE_KEY"
|
||||
cat > ~/.claude/settings.json <<JSON
|
||||
{
|
||||
"env": {
|
||||
"ANTHROPIC_BASE_URL": "${OMNIROUTE_ANTHROPIC_URL}",
|
||||
"ANTHROPIC_AUTH_TOKEN": "${OMNIROUTE_KEY}"
|
||||
}
|
||||
}
|
||||
JSON
|
||||
|
||||
cat > ~/.codex/config.yaml <<YAML
|
||||
model: auto
|
||||
apiKey: ${OMNIROUTE_KEY}
|
||||
apiBaseUrl: ${OMNIROUTE_URL}
|
||||
YAML
|
||||
|
||||
cat > ~/.qwen/.env <<ENV
|
||||
OPENAI_API_KEY="${OMNIROUTE_KEY}"
|
||||
OPENAI_BASE_URL="${OMNIROUTE_URL}"
|
||||
OPENAI_MODEL="auto"
|
||||
ENV
|
||||
|
||||
# 4. Append global env vars (idempotent guard)
|
||||
if ! grep -q "OmniRoute Universal Endpoint" ~/.bashrc 2>/dev/null; then
|
||||
cat >> ~/.bashrc <<ENV
|
||||
|
||||
# OmniRoute Universal Endpoint
|
||||
export OPENAI_BASE_URL="${OMNIROUTE_URL}"
|
||||
export OPENAI_API_KEY="${OMNIROUTE_KEY}"
|
||||
export ANTHROPIC_BASE_URL="${OMNIROUTE_ANTHROPIC_URL}"
|
||||
export ANTHROPIC_AUTH_TOKEN="${OMNIROUTE_KEY}"
|
||||
ENV
|
||||
fi
|
||||
|
||||
# 5. Validate via the internal CLI
|
||||
omniroute doctor || true
|
||||
omniroute providers list || true
|
||||
|
||||
echo "All CLIs installed and configured for OmniRoute"
|
||||
EOF
|
||||
|
||||
source ~/.bashrc
|
||||
echo "✅ All CLIs installed and configured for OmniRoute"
|
||||
chmod +x my-setup.sh
|
||||
./my-setup.sh
|
||||
```
|
||||
|
||||
372
docs/CLOUD_AGENT.md
Normal file
372
docs/CLOUD_AGENT.md
Normal file
@@ -0,0 +1,372 @@
|
||||
# Cloud Agents
|
||||
|
||||
> **Source of truth:** `src/lib/cloudAgent/` and `src/app/api/v1/agents/tasks/`
|
||||
> **Last updated:** 2026-05-13 — v3.8.0
|
||||
|
||||
OmniRoute orchestrates third-party cloud-hosted coding agents (Codex Cloud, Devin,
|
||||
Jules) as long-running tasks. Each agent is wrapped behind a uniform interface so
|
||||
clients can submit a prompt + repo URL and receive results without dealing with
|
||||
provider-specific APIs.
|
||||
|
||||
A Cloud Agent task is **not** a regular chat completion. It is a durable, multi-step
|
||||
unit of work that may take minutes to hours, can produce a Pull Request as its
|
||||
artifact, and supports follow-up messages and (in some providers) plan approval gates.
|
||||
|
||||
## Supported Agents
|
||||
|
||||
| Provider ID | Class | Source | Upstream Base URL | Plan Approval |
|
||||
| ------------- | ----------------- | ------------------------------------ | --------------------------------------- | ------------- |
|
||||
| `jules` | `JulesAgent` | `src/lib/cloudAgent/agents/jules.ts` | `https://jules.googleapis.com/v1alpha` | Yes |
|
||||
| `devin` | `DevinAgent` | `src/lib/cloudAgent/agents/devin.ts` | `https://api.devin.ai/v1` | Yes |
|
||||
| `codex-cloud` | `CodexCloudAgent` | `src/lib/cloudAgent/agents/codex.ts` | `https://api.openai.com/v1/codex/cloud` | No (auto) |
|
||||
|
||||
Registry: `src/lib/cloudAgent/registry.ts` — exports `getAgent(providerId)`,
|
||||
`getAvailableAgents()`, and `isCloudAgentProvider(providerId)`. The registry is a
|
||||
plain in-memory `Record<string, CloudAgentBase>` populated at module load.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Client (Dashboard / CLI / API)
|
||||
→ POST /api/v1/agents/tasks (management auth required)
|
||||
→ CreateCloudAgentTaskSchema validation (Zod)
|
||||
→ registry.getAgent(providerId)
|
||||
→ getCloudAgentCredentials(providerId)
|
||||
└─ pulls from getProviderConnections({ provider, isActive: true })
|
||||
(apiKey first, fallback to accessToken)
|
||||
→ agent.createTask({ prompt, source, options }, credentials)
|
||||
└─ HTTP POST to upstream provider API
|
||||
└─ returns CloudAgentTask with internal id + externalId
|
||||
→ insertCloudAgentTask(...) into cloud_agent_tasks (SQLite)
|
||||
|
||||
Polling (lazy sync on read):
|
||||
GET /api/v1/agents/tasks/[id]
|
||||
→ getCloudAgentTaskById(id)
|
||||
→ agent.getStatus(externalId, credentials) // refreshes status + activities
|
||||
→ updateCloudAgentTask(...) with new status, result, completed_at
|
||||
→ return serialized task
|
||||
|
||||
Interactions:
|
||||
POST /api/v1/agents/tasks/[id] body: { action: "approve" | "message" | "cancel" }
|
||||
→ agent.approvePlan(externalId, credentials) for "approve"
|
||||
→ agent.sendMessage(externalId, message, credentials) for "message"
|
||||
→ status flips to "cancelled" for "cancel" (local-only)
|
||||
```
|
||||
|
||||
Sync is **lazy**: status is refreshed from the upstream on every `GET /tasks/[id]`.
|
||||
There is no background poller. Dashboards that need fresh state should poll the GET
|
||||
endpoint at a sensible interval.
|
||||
|
||||
## `CloudAgentBase` Interface
|
||||
|
||||
Source: `src/lib/cloudAgent/baseAgent.ts`
|
||||
|
||||
```typescript
|
||||
export interface AgentCredentials {
|
||||
apiKey: string;
|
||||
baseUrl?: string;
|
||||
}
|
||||
|
||||
export interface CreateTaskParams {
|
||||
prompt: string;
|
||||
source: CloudAgentSource;
|
||||
options: {
|
||||
autoCreatePr?: boolean;
|
||||
planApprovalRequired?: boolean;
|
||||
environment?: Record<string, string>;
|
||||
};
|
||||
}
|
||||
|
||||
export interface GetStatusResult {
|
||||
status: CloudAgentStatus;
|
||||
externalId?: string;
|
||||
result?: CloudAgentResult;
|
||||
activities: CloudAgentActivity[];
|
||||
error?: string;
|
||||
}
|
||||
|
||||
export abstract class CloudAgentBase {
|
||||
abstract readonly providerId: string;
|
||||
abstract readonly baseUrl: string;
|
||||
|
||||
abstract createTask(p: CreateTaskParams, c: AgentCredentials): Promise<CloudAgentTask>;
|
||||
abstract getStatus(externalId: string, c: AgentCredentials): Promise<GetStatusResult>;
|
||||
abstract approvePlan(externalId: string, c: AgentCredentials): Promise<void>;
|
||||
abstract sendMessage(
|
||||
externalId: string,
|
||||
message: string,
|
||||
c: AgentCredentials
|
||||
): Promise<CloudAgentActivity>;
|
||||
abstract listSources(
|
||||
c: AgentCredentials
|
||||
): Promise<{ name: string; url: string; branch?: string }[]>;
|
||||
|
||||
protected mapStatus(raw: string): CloudAgentStatus; // heuristic upstream-string → enum
|
||||
protected generateTaskId(): string; // `task_<ts>_<rand>`
|
||||
protected generateActivityId(): string; // `act_<ts>_<rand>`
|
||||
}
|
||||
```
|
||||
|
||||
`CodexCloudAgent.approvePlan` intentionally throws — Codex Cloud auto-plans and has
|
||||
no approval gate. `CodexCloudAgent.listSources` returns `[]`.
|
||||
|
||||
## Domain Types
|
||||
|
||||
Source: `src/lib/cloudAgent/types.ts`
|
||||
|
||||
```typescript
|
||||
export const CLOUD_AGENT_STATUS = {
|
||||
QUEUED: "queued",
|
||||
RUNNING: "running",
|
||||
AWAITING_APPROVAL: "awaiting_approval",
|
||||
COMPLETED: "completed",
|
||||
FAILED: "failed",
|
||||
CANCELLED: "cancelled",
|
||||
} as const;
|
||||
|
||||
export interface CloudAgentSource {
|
||||
repoName: string;
|
||||
repoUrl: string; // must be a valid URL
|
||||
branch?: string;
|
||||
}
|
||||
|
||||
export interface CloudAgentResult {
|
||||
prUrl?: string;
|
||||
prNumber?: number;
|
||||
commitMessage?: string;
|
||||
diffUrl?: string;
|
||||
summary?: string;
|
||||
duration?: number; // seconds, positive int
|
||||
cost?: number; // positive float
|
||||
}
|
||||
|
||||
export interface CloudAgentActivity {
|
||||
id: string;
|
||||
type: "plan" | "command" | "code_change" | "message" | "error" | "completion";
|
||||
content: string;
|
||||
timestamp: string; // ISO 8601
|
||||
metadata?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
export interface CloudAgentTask {
|
||||
id: string; // internal `task_...` id
|
||||
providerId: "jules" | "devin" | "codex-cloud";
|
||||
externalId?: string; // upstream provider's id
|
||||
status: CloudAgentStatus;
|
||||
prompt: string; // 1..10000 chars
|
||||
source: CloudAgentSource;
|
||||
options: {
|
||||
autoCreatePr?: boolean;
|
||||
planApprovalRequired?: boolean;
|
||||
environment?: Record<string, string>;
|
||||
};
|
||||
result?: CloudAgentResult;
|
||||
activities: CloudAgentActivity[];
|
||||
error?: string;
|
||||
createdAt: string;
|
||||
updatedAt: string;
|
||||
completedAt?: string;
|
||||
}
|
||||
```
|
||||
|
||||
Validation schemas (`CreateCloudAgentTaskSchema`, `UpdateCloudAgentTaskSchema`) are
|
||||
exported alongside the types and are used by the route handlers.
|
||||
|
||||
## Database
|
||||
|
||||
Source: `src/lib/cloudAgent/db.ts` — table is created lazily via
|
||||
`createCloudAgentTaskTable()` (also called from `src/lib/cloudAgent/index.ts` at
|
||||
module import).
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS cloud_agent_tasks (
|
||||
id TEXT PRIMARY KEY,
|
||||
provider_id TEXT NOT NULL,
|
||||
external_id TEXT,
|
||||
status TEXT NOT NULL DEFAULT 'queued',
|
||||
prompt TEXT NOT NULL,
|
||||
source TEXT NOT NULL, -- JSON
|
||||
options TEXT DEFAULT '{}', -- JSON
|
||||
result TEXT, -- JSON
|
||||
activities TEXT DEFAULT '[]', -- JSON
|
||||
error TEXT,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
updated_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
completed_at TEXT
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_cloud_agent_tasks_provider ON cloud_agent_tasks(provider_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_cloud_agent_tasks_status ON cloud_agent_tasks(status);
|
||||
CREATE INDEX IF NOT EXISTS idx_cloud_agent_tasks_created ON cloud_agent_tasks(created_at DESC);
|
||||
```
|
||||
|
||||
`updateCloudAgentTask` enforces a **column whitelist** to prevent SQL injection:
|
||||
`status`, `prompt`, `source`, `options`, `result`, `activities`, `error`,
|
||||
`completed_at`. Any other key in the partial update is silently dropped.
|
||||
|
||||
## REST API — Task Lifecycle
|
||||
|
||||
**Auth:** All `/api/v1/agents/tasks*` endpoints require **management auth**
|
||||
(`requireCloudAgentManagementAuth` wraps `requireManagementAuth` from
|
||||
`src/lib/api/requireManagementAuth`). This is enforced after commit `588a0333`
|
||||
(_"fix(auth): require management auth for agent and cooldown APIs"_).
|
||||
|
||||
| Method | Path | Purpose |
|
||||
| ------- | ----------------------------- | ------------------------------------------------------ |
|
||||
| OPTIONS | `/api/v1/agents/tasks` | CORS preflight |
|
||||
| GET | `/api/v1/agents/tasks` | List tasks (filter: `provider`, `status`, `limit≤500`) |
|
||||
| POST | `/api/v1/agents/tasks` | Create task (dispatches to upstream + persists) |
|
||||
| DELETE | `/api/v1/agents/tasks?id=...` | Delete task by query id (does **not** cancel upstream) |
|
||||
| OPTIONS | `/api/v1/agents/tasks/[id]` | CORS preflight |
|
||||
| GET | `/api/v1/agents/tasks/[id]` | Read task + lazy-sync status from upstream |
|
||||
| POST | `/api/v1/agents/tasks/[id]` | Action: `approve` / `message` / `cancel` |
|
||||
| DELETE | `/api/v1/agents/tasks/[id]` | Delete task by path id |
|
||||
|
||||
### Create task
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/api/v1/agents/tasks \
|
||||
-H "Cookie: auth_token=..." \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"providerId": "devin",
|
||||
"prompt": "Fix the bug in src/foo.ts where the parser returns null",
|
||||
"source": {
|
||||
"repoName": "user/repo",
|
||||
"repoUrl": "https://github.com/user/repo",
|
||||
"branch": "main"
|
||||
},
|
||||
"options": {
|
||||
"autoCreatePr": true,
|
||||
"planApprovalRequired": false
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
Response `201`:
|
||||
|
||||
```json
|
||||
{
|
||||
"data": {
|
||||
"id": "task_1731512345678_abc123def",
|
||||
"providerId": "devin",
|
||||
"externalId": "session_xyz",
|
||||
"status": "queued",
|
||||
"prompt": "...",
|
||||
"source": { "repoName": "user/repo", "repoUrl": "...", "branch": "main" },
|
||||
"options": { "autoCreatePr": true },
|
||||
"createdAt": "2026-05-13T12:34:56.789Z"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Approve a plan
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/api/v1/agents/tasks/<id> \
|
||||
-H "Cookie: auth_token=..." \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"action":"approve"}'
|
||||
```
|
||||
|
||||
### Send a follow-up message
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/api/v1/agents/tasks/<id> \
|
||||
-d '{"action":"message","message":"Also add a unit test for the parser"}'
|
||||
```
|
||||
|
||||
### Cancel (local status only)
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/api/v1/agents/tasks/<id> \
|
||||
-d '{"action":"cancel"}'
|
||||
```
|
||||
|
||||
`cancel` flips `status` to `"cancelled"` in the local DB but does **not** call the
|
||||
upstream provider — there is no abort RPC in `CloudAgentBase`. To stop billing
|
||||
upstream, terminate the task in the provider's own console.
|
||||
|
||||
## REST API — Cloud Provider Plumbing
|
||||
|
||||
These auxiliary endpoints under `src/app/api/cloud/` are used by remote clients
|
||||
(the CLI, the Electron app, or sync workers) to read provider connection metadata
|
||||
and resolve model aliases. They are authenticated with a **regular API key**
|
||||
(via `validateApiKey`), not the management auth used by the task endpoints.
|
||||
|
||||
| Method | Path | Purpose |
|
||||
| ------ | ------------------------------- | ------------------------------------------------------------------- |
|
||||
| POST | `/api/cloud/auth` | Validate API key, return masked connection metadata + model aliases |
|
||||
| PUT | `/api/cloud/credentials/update` | Refresh `accessToken` / `refreshToken` / `expiresAt` |
|
||||
| POST | `/api/cloud/model/resolve` | Resolve a model alias to `{ provider, model }` |
|
||||
| GET | `/api/cloud/models/alias` | List all model aliases |
|
||||
| PUT | `/api/cloud/models/alias` | Set a model alias (and auto-sync to Cloud if enabled) |
|
||||
|
||||
`/api/cloud/auth` never returns raw `apiKey` / `accessToken` / `refreshToken`. It
|
||||
returns `hasApiKey`, `hasAccessToken`, `hasRefreshToken`, and a masked preview
|
||||
(`maskedApiKey`: first 4 + `****` + last 4).
|
||||
|
||||
## Credentials Resolution
|
||||
|
||||
`getCloudAgentCredentials(providerId)` in `src/lib/cloudAgent/api.ts`:
|
||||
|
||||
1. Loads active provider connections via `getProviderConnections({ provider: providerId, isActive: true })`.
|
||||
2. For each connection, prefers `apiKey` (trimmed). Falls back to `accessToken`.
|
||||
3. Returns the first non-empty token wrapped as `{ apiKey: token }`.
|
||||
4. Returns `null` if no usable token is found — the API responds `400` with
|
||||
`"No active credentials configured for cloud agent provider: <id>"`.
|
||||
|
||||
This means Cloud Agents reuse the same Provider Connection table as regular LLM
|
||||
providers. To enable Jules, create an active connection with `provider: "jules"`
|
||||
and a populated `apiKey`.
|
||||
|
||||
## Dashboard
|
||||
|
||||
Source: `src/app/(dashboard)/dashboard/cloud-agents/page.tsx`
|
||||
|
||||
A `"use client"` React page that:
|
||||
|
||||
- Lists tasks (polled via `GET /api/v1/agents/tasks`).
|
||||
- Submits new tasks via a form that maps to `CreateCloudAgentTaskSchema`.
|
||||
- Shows status badges (`queued`, `running`, `awaiting_approval`, `completed`,
|
||||
`failed`, `cancelled`) and renders the `activities[]` timeline.
|
||||
- Surfaces the `result.prUrl` / `commitMessage` / `summary` when `status === "completed"`.
|
||||
|
||||
## Integration with A2A
|
||||
|
||||
Cloud Agents can be exposed as A2A skills by registering an A2A skill that delegates
|
||||
its `tasks/send` handler to `getAgent(...).createTask(...)` and translates A2A task
|
||||
status events to the JSON-RPC 2.0 protocol. See [A2A-SERVER.md](./A2A-SERVER.md).
|
||||
|
||||
## Adding a New Cloud Agent
|
||||
|
||||
1. Create `src/lib/cloudAgent/agents/<name>.ts` extending `CloudAgentBase`.
|
||||
2. Implement `createTask`, `getStatus`, `approvePlan` (or throw if N/A),
|
||||
`sendMessage`, `listSources`. Use `this.mapStatus(...)` for status normalization.
|
||||
3. Register in `src/lib/cloudAgent/registry.ts` under a stable `providerId`.
|
||||
4. Extend the `providerId` literal union in `src/lib/cloudAgent/types.ts`
|
||||
(`CloudAgentTask.providerId` and `CreateCloudAgentTaskSchema`).
|
||||
5. Add the provider to `src/shared/constants/providers.ts` if it needs a connection
|
||||
record. OAuth-based providers also need `src/lib/oauth/providers/`.
|
||||
6. Add tests under `tests/unit/cloud-agent-*.test.ts`.
|
||||
7. Update this doc and the dashboard's `CLOUD_AGENTS` constant.
|
||||
|
||||
## Configuration
|
||||
|
||||
| Env Var | Purpose |
|
||||
| ---------------- | ----------------------------------------------------------- |
|
||||
| `DATA_DIR` | Location of the SQLite database holding `cloud_agent_tasks` |
|
||||
| `JWT_SECRET` | Required for management auth on task endpoints |
|
||||
| `API_KEY_SECRET` | Required to encrypt provider connection credentials at rest |
|
||||
|
||||
No Cloud-Agent-specific env vars exist today — every secret lives in the
|
||||
`provider_connections` table.
|
||||
|
||||
## See Also
|
||||
|
||||
- [A2A-SERVER.md](./A2A-SERVER.md)
|
||||
- [API_REFERENCE.md](./API_REFERENCE.md)
|
||||
- [SKILLS.md](./SKILLS.md)
|
||||
- [MEMORY.md](./MEMORY.md)
|
||||
- Source: `src/lib/cloudAgent/`
|
||||
- Routes: `src/app/api/v1/agents/tasks/`, `src/app/api/cloud/`
|
||||
- Dashboard: `src/app/(dashboard)/dashboard/cloud-agents/page.tsx`
|
||||
224
docs/COMPLIANCE.md
Normal file
224
docs/COMPLIANCE.md
Normal file
@@ -0,0 +1,224 @@
|
||||
# Compliance & Audit
|
||||
|
||||
> **Source of truth:** `src/lib/compliance/`, `src/app/api/compliance/`
|
||||
> **Last updated:** 2026-05-13 — v3.8.0
|
||||
|
||||
OmniRoute records administrative actions, authentication events, provider
|
||||
credential lifecycle changes, and MCP tool invocations to SQLite-backed audit
|
||||
tables. This page covers what gets logged, where it lives, how long it is
|
||||
retained, how API keys can opt out, and how to query the data.
|
||||
|
||||
The implementation lives in `src/lib/compliance/index.ts` (T-43 — "Compliance
|
||||
Controls") and `src/lib/compliance/providerAudit.ts`. Audit writes never throw:
|
||||
on any failure the call is silently swallowed so audit logging cannot break the
|
||||
main request flow.
|
||||
|
||||
## What Gets Logged
|
||||
|
||||
### Administrative audit events (`audit_log`)
|
||||
|
||||
Every call to `logAuditEvent({ action, actor, target, details, ... })` produces
|
||||
one row. Action strings follow a `domain.verb` (or `domain.verb.outcome`)
|
||||
pattern. Confirmed in-tree action types include:
|
||||
|
||||
| Action | Source |
|
||||
| ------------------------------------ | --------------------------------------- |
|
||||
| `auth.login.success` | `src/app/api/auth/login/route.ts` |
|
||||
| `auth.login.failed` | `src/app/api/auth/login/route.ts` |
|
||||
| `auth.login.locked` | `src/app/api/auth/login/route.ts` |
|
||||
| `auth.login.error` | `src/app/api/auth/login/route.ts` |
|
||||
| `auth.login.misconfigured` | `src/app/api/auth/login/route.ts` |
|
||||
| `auth.login.setup_required` | `src/app/api/auth/login/route.ts` |
|
||||
| `auth.logout.success` | `src/app/api/auth/logout/route.ts` |
|
||||
| `provider.credentials.created` | `src/app/api/providers/route.ts` |
|
||||
| `provider.credentials.updated` | `src/app/api/providers/[id]/route.ts` |
|
||||
| `provider.credentials.revoked` | `src/app/api/providers/[id]/route.ts` |
|
||||
| `provider.credentials.batch_revoked` | `src/app/api/providers/route.ts` |
|
||||
| `sync.token.created` | `src/app/api/sync/tokens/route.ts` |
|
||||
| `sync.token.revoked` | `src/app/api/sync/tokens/[id]/route.ts` |
|
||||
| `compliance.cleanup` | `src/lib/compliance/index.ts` |
|
||||
|
||||
Each entry captures `action`, `actor` (defaults to `"system"`), `target`,
|
||||
`details`/`metadata` (JSON), `ip_address`, `resource_type`, `status`,
|
||||
`request_id`, and `timestamp`. Sensitive keys (`apiKey`, `accessToken`,
|
||||
`refreshToken`, `password`, anything matching `*token`/`*secret`/`*apikey`,
|
||||
etc.) are recursively redacted to `"[redacted]"` before the row is written.
|
||||
|
||||
### MCP tool calls (`mcp_tool_audit`)
|
||||
|
||||
Every MCP tool invocation writes a row through
|
||||
`open-sse/mcp-server/audit.ts`. Schema (from
|
||||
`src/lib/db/migrations/002_mcp_a2a_tables.sql`):
|
||||
|
||||
| Column | Notes |
|
||||
| ---------------- | ----------------------------------- |
|
||||
| `id` | autoincrement |
|
||||
| `tool_name` | MCP tool identifier |
|
||||
| `input_hash` | sha256 of input (no payload stored) |
|
||||
| `output_summary` | short, truncated summary |
|
||||
| `duration_ms` | wall time |
|
||||
| `api_key_id` | caller (nullable) |
|
||||
| `success` | `1` / `0` |
|
||||
| `error_code` | terminal error code on failure |
|
||||
| `created_at` | ISO timestamp |
|
||||
|
||||
### Request / usage logs
|
||||
|
||||
These are operational telemetry (not strictly admin audit) but share the same
|
||||
retention pipeline:
|
||||
|
||||
- `usage_history` — per-request usage roll-up
|
||||
- `call_logs` — full per-request log (subject to row-cap, see below)
|
||||
- `proxy_logs` — proxy traffic log (subject to row-cap)
|
||||
- `request_detail_logs` — legacy detailed request log (still pruned if present)
|
||||
|
||||
## Storage Schema
|
||||
|
||||
`audit_log` is created lazily by `ensureAuditLogSchema()` on first use:
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS audit_log (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
timestamp TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
action TEXT NOT NULL,
|
||||
actor TEXT NOT NULL DEFAULT 'system',
|
||||
target TEXT,
|
||||
details TEXT,
|
||||
ip_address TEXT,
|
||||
resource_type TEXT,
|
||||
status TEXT,
|
||||
request_id TEXT,
|
||||
metadata TEXT
|
||||
);
|
||||
```
|
||||
|
||||
Indexes are created on `timestamp`, `action`, `actor`, `resource_type`,
|
||||
`status`, and `request_id`. Missing columns on legacy DBs are added via
|
||||
`ALTER TABLE` on demand.
|
||||
|
||||
## Retention & Cleanup
|
||||
|
||||
Two separate retention windows are honoured:
|
||||
|
||||
| Env var | Default | Applies to |
|
||||
| --------------------------- | -------- | ----------------------------------------------------------------- |
|
||||
| `APP_LOG_RETENTION_DAYS` | `7` | `audit_log`, `mcp_tool_audit` |
|
||||
| `CALL_LOG_RETENTION_DAYS` | `7` | `usage_history`, `call_logs`, `proxy_logs`, `request_detail_logs` |
|
||||
| `CALL_LOGS_TABLE_MAX_ROWS` | `100000` | Row-cap trim for `call_logs` |
|
||||
| `PROXY_LOGS_TABLE_MAX_ROWS` | `100000` | Row-cap trim for `proxy_logs` |
|
||||
|
||||
`cleanupExpiredLogs()` runs the retention pass. It is invoked on server startup
|
||||
from `src/server-init.ts` and `src/instrumentation-node.ts`. Each run logs a
|
||||
`compliance.cleanup` audit event with the per-table delete counts. Proxy/call
|
||||
log trimming is batched (`BATCH_SIZE = 5000`) to avoid long write locks.
|
||||
|
||||
Defaults are defined in `src/lib/logEnv.ts`
|
||||
(`DEFAULT_APP_LOG_RETENTION_DAYS = 7`, `DEFAULT_CALL_LOG_RETENTION_DAYS = 7`).
|
||||
|
||||
## `noLog` Opt-Out (per API key)
|
||||
|
||||
API keys can be flagged so their downstream call traffic is not logged. The
|
||||
flag lives on the `api_keys` table (`no_log INTEGER DEFAULT 0`) and is mirrored
|
||||
into an in-memory set for hot-path lookups.
|
||||
|
||||
```bash
|
||||
# Create a no-log key (management auth required)
|
||||
curl -X POST http://localhost:20128/api/keys \
|
||||
-H "Cookie: auth_token=..." \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"name": "Privacy key", "noLog": true}'
|
||||
```
|
||||
|
||||
Helpers (`src/lib/compliance/index.ts`):
|
||||
|
||||
- `setNoLog(apiKeyId, true|false)` — toggle the in-memory entry
|
||||
- `isNoLog(apiKeyId)` — checked on the request path; falls back to a 30 s
|
||||
cached read from `api_keys.no_log`
|
||||
- `NO_LOG_API_KEY_IDS` (env, comma-separated) — preloaded into the in-memory
|
||||
set on boot; useful when you cannot toggle the column directly
|
||||
|
||||
Administrative audit events (login, provider changes, MCP tool calls, etc.)
|
||||
are **not** affected by `noLog` — only per-request traffic logging is opted
|
||||
out.
|
||||
|
||||
## REST API
|
||||
|
||||
| Endpoint | Method | Description | Auth |
|
||||
| --------------------------- | ------ | ------------------------------------------ | ---------- |
|
||||
| `/api/compliance/audit-log` | `GET` | Paginated admin audit entries with filters | management |
|
||||
| `/api/mcp/audit` | `GET` | Paginated MCP tool audit entries | (open-sse) |
|
||||
| `/api/mcp/audit/stats` | `GET` | Aggregated MCP audit stats | (open-sse) |
|
||||
|
||||
No CSV export endpoint is shipped today — export from the dashboard or query
|
||||
the SQLite database directly.
|
||||
|
||||
### Querying `/api/compliance/audit-log`
|
||||
|
||||
Supported query params (all optional, all use `LIKE %value%` matching for
|
||||
text filters):
|
||||
|
||||
- `action`, `actor`, `target`, `resourceType` (or `resource_type`),
|
||||
`status`, `requestId` (or `request_id`)
|
||||
- `from` / `since`, `to` / `until` — ISO timestamps
|
||||
- `limit` (default `50`, min `1`, max `500`)
|
||||
- `offset` (default `0`, max `10_000`)
|
||||
|
||||
The response is a JSON array. Pagination metadata is returned in headers:
|
||||
`x-total-count`, `x-page-limit`, `x-page-offset`.
|
||||
|
||||
```bash
|
||||
curl "http://localhost:20128/api/compliance/audit-log?action=provider.credentials&from=2026-05-01" \
|
||||
-H "Cookie: auth_token=..."
|
||||
```
|
||||
|
||||
## Dashboard
|
||||
|
||||
The dashboard exposes audit data at **`/dashboard/audit`**
|
||||
(`src/app/(dashboard)/dashboard/audit/page.tsx`). The page has two tabs:
|
||||
|
||||
- **Compliance** (`ComplianceTab.tsx`) — admin audit events from
|
||||
`/api/compliance/audit-log`. Filters by event type, severity (info / warning
|
||||
/ critical, derived from action + status), and date range. Severity is
|
||||
computed client-side from the action/status strings.
|
||||
- **MCP** (`McpAuditTab.tsx`) — MCP tool audit from `/api/mcp/audit`, with
|
||||
filters by tool name and success/failure.
|
||||
|
||||
Both tabs paginate with page sizes of `50` (compliance) and `25` (MCP).
|
||||
|
||||
## Provider Credential Helpers
|
||||
|
||||
`src/lib/compliance/providerAudit.ts` provides shaping helpers used by the
|
||||
provider-management routes when they emit credential events:
|
||||
|
||||
- `summarizeProviderConnectionForAudit(connection)` — strips `apiKey`,
|
||||
`accessToken`, `refreshToken`, `idToken`, and
|
||||
`providerSpecificData.consoleApiKey` before the connection snapshot is
|
||||
written to `details`.
|
||||
- `getProviderAuditTarget(connection)` — composes a stable
|
||||
`"<provider>:<name|id>"` string for the `target` field.
|
||||
- `extractProviderWarnings(...payloads)` — scans provider responses for
|
||||
policy/safety warnings (`[sanitizer]`, `prompt injection detected`,
|
||||
`content has been filtered`, `safety filter`, `policy violation`) and
|
||||
surfaces up to 5 hits, each truncated to 400 chars.
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Flag API keys handling PII (legal, medical, etc.) with `noLog: true`.
|
||||
- Tune `APP_LOG_RETENTION_DAYS` / `CALL_LOG_RETENTION_DAYS` to meet your
|
||||
retention policy. The 7-day defaults are conservative.
|
||||
- Export the audit table off-platform (`sqlite3 dump`) on whatever cadence
|
||||
your compliance program requires — no built-in archival exists.
|
||||
- Track `auth.login.failed` and `auth.login.locked` counts for brute-force
|
||||
detection.
|
||||
- When adding new admin endpoints, call `logAuditEvent({ ... })` with a stable
|
||||
`domain.verb.outcome` action string and pass the request context via
|
||||
`getAuditRequestContext(request)` so IP and `requestId` are captured
|
||||
automatically.
|
||||
|
||||
## See Also
|
||||
|
||||
- [`docs/GUARDRAILS.md`](./GUARDRAILS.md) — PII masking, prompt injection
|
||||
- [`docs/MCP-SERVER.md`](./MCP-SERVER.md) — MCP tool catalog and scopes
|
||||
- [`docs/ENVIRONMENT.md`](./ENVIRONMENT.md) — full env var reference
|
||||
- Source: `src/lib/compliance/`, `src/app/api/compliance/`,
|
||||
`src/app/api/mcp/audit/`, `src/lib/logEnv.ts`
|
||||
@@ -21,13 +21,24 @@ The registry lives in `open-sse/services/compression/engines/registry.ts`. Engin
|
||||
contract:
|
||||
|
||||
- `id`: stable engine id such as `caveman` or `rtk`
|
||||
- `label`: dashboard-readable name
|
||||
- `supports(mode)`: whether the engine can execute a compression mode
|
||||
- `compress(input)`: transforms text/messages and returns stats
|
||||
- `apply(text, config)`: legacy execution path used by stacked pipelines
|
||||
- `compress(input, config)`: primary execution path returning text + stats
|
||||
- `getConfigSchema()`: returns the JSON-Schema-like shape of valid config
|
||||
- `validateConfig(config)`: returns `{ valid, errors[] }`
|
||||
|
||||
Registration uses `registerCompressionEngine(engine)` (or `registerEngine` for advanced cases),
|
||||
which calls `assertValidEngine()` and `validateConfig(defaultConfig)` before accepting.
|
||||
Use `unregisterCompressionEngine(id)` to remove an engine at runtime.
|
||||
|
||||
`strategySelector.ts` registers the built-in engines before compression runs. This lets preview,
|
||||
runtime compression, stacked mode, tests, and future engines use the same execution path.
|
||||
|
||||
### MCP description compression (related)
|
||||
|
||||
A separate registry compresses MCP tool description metadata at registry-level — see
|
||||
`open-sse/mcp-server/descriptionCompressor.ts` and [MCP-SERVER.md](./MCP-SERVER.md). It reuses
|
||||
Caveman rules but operates on tool metadata, not request payloads.
|
||||
|
||||
## Caveman
|
||||
|
||||
Caveman mode focuses on semantic condensation of normal prose:
|
||||
|
||||
@@ -12,16 +12,20 @@ Language packs live under:
|
||||
open-sse/services/compression/rules/<language>/
|
||||
```
|
||||
|
||||
Current shipped packs include:
|
||||
Current shipped packs (verified against `rules/` directory contents):
|
||||
|
||||
| Language | Directory |
|
||||
| ------------------- | -------------- |
|
||||
| English | `rules/en/` |
|
||||
| Portuguese (Brazil) | `rules/pt-BR/` |
|
||||
| Spanish | `rules/es/` |
|
||||
| German | `rules/de/` |
|
||||
| French | `rules/fr/` |
|
||||
| Japanese | `rules/ja/` |
|
||||
| Language | Directory | Rule categories present |
|
||||
| ------------------- | -------------- | --------------------------------------------------- |
|
||||
| English | `rules/en/` | `context`, `dedup`, `filler`, `structural`, `ultra` |
|
||||
| Spanish | `rules/es/` | `context`, `dedup`, `filler`, `structural`, `ultra` |
|
||||
| Portuguese (Brazil) | `rules/pt-BR/` | `context`, `filler`, `structural` |
|
||||
| German | `rules/de/` | `context`, `filler`, `structural` |
|
||||
| French | `rules/fr/` | `context`, `filler`, `structural` |
|
||||
| Japanese | `rules/ja/` | `context`, `filler`, `structural` |
|
||||
|
||||
> **Parity note:** `en` and `es` packs have the full 5 categories; `pt-BR`, `de`, `fr`, `ja` ship 3 categories. The missing `dedup` and `ultra` categories silently fall back to the English built-ins. Contributions welcome to add `dedup.json` and `ultra.json` for the smaller packs.
|
||||
>
|
||||
> The canonical category list and per-category schema live in [`open-sse/services/compression/rules/_schema.json`](../open-sse/services/compression/rules/_schema.json) (JSON Schema draft 2020-12).
|
||||
|
||||
## Language Detection
|
||||
|
||||
|
||||
@@ -3,6 +3,9 @@
|
||||
Compression rules are JSON files loaded at runtime. They are intentionally data-only so new
|
||||
language packs and RTK command filters can be reviewed without changing engine code.
|
||||
|
||||
> **Canonical schema (source of truth):** [`open-sse/services/compression/rules/_schema.json`](../open-sse/services/compression/rules/_schema.json) (JSON Schema draft 2020-12).
|
||||
> The examples below are illustrative — when in doubt, validate your pack against `_schema.json`.
|
||||
|
||||
## Caveman Rule Packs
|
||||
|
||||
Caveman rule packs live under:
|
||||
|
||||
269
docs/ELECTRON_GUIDE.md
Normal file
269
docs/ELECTRON_GUIDE.md
Normal file
@@ -0,0 +1,269 @@
|
||||
# Electron Desktop Guide
|
||||
|
||||
> **Source of truth:** `electron/` workspace
|
||||
> **Last updated:** 2026-05-13 — v3.8.0
|
||||
|
||||
OmniRoute ships a cross-platform desktop app (Windows / macOS / Linux) built on
|
||||
**Electron 41** + **electron-builder 26.10**. The desktop app spawns the Next.js
|
||||
standalone server as a child process, points a `BrowserWindow` at it, and adds a
|
||||
system tray, auto-updater, IPC bridge, and zero-config secret bootstrap.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────┐
|
||||
│ Electron main process (electron/main.js) │
|
||||
│ ├─ Single-instance lock │
|
||||
│ ├─ Child process: Next.js standalone server │
|
||||
│ │ (spawned with Electron's Node runtime) │
|
||||
│ ├─ BrowserWindow → http://localhost:PORT │
|
||||
│ ├─ System tray + context menu │
|
||||
│ ├─ Auto-update via electron-updater │
|
||||
│ ├─ Content Security Policy (session headers) │
|
||||
│ └─ Secret bootstrap (JWT / API_KEY_SECRET) │
|
||||
└──────────────────────────────────────────────┘
|
||||
↕ IPC bridge (electron/preload.js)
|
||||
┌──────────────────────────────────────────────┐
|
||||
│ Renderer (Next.js dashboard) │
|
||||
│ window.electronAPI.* (contextIsolation) │
|
||||
└──────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Versions
|
||||
|
||||
Confirmed from `electron/package.json`:
|
||||
|
||||
| Package | Version |
|
||||
| ------------------ | -------------------------- |
|
||||
| `electron` | `^41.5.1` |
|
||||
| `electron-builder` | `^26.10.0` |
|
||||
| `electron-updater` | `^6.8.5` |
|
||||
| `better-sqlite3` | `^12.9.0` |
|
||||
| App version | `3.8.0` |
|
||||
| App id | `online.omniroute.desktop` |
|
||||
| Product name | `OmniRoute` |
|
||||
|
||||
## Scripts (root `package.json`)
|
||||
|
||||
| Script | Purpose |
|
||||
| --------------------------------- | -------------------------------------------------------------------------- |
|
||||
| `npm run electron:dev` | Starts `npm run dev` + waits for `localhost:20128` + launches Electron |
|
||||
| `npm run electron:build` | Builds Next.js then runs `electron-builder` for the current OS |
|
||||
| `npm run electron:build:win` | Builds Windows NSIS installer + portable (x64) |
|
||||
| `npm run electron:build:mac` | Builds macOS DMG (Intel + Apple Silicon) |
|
||||
| `npm run electron:build:linux` | Builds Linux AppImage + DEB (x64 + arm64) |
|
||||
| `npm run electron:smoke:packaged` | Launches packaged binary and probes `/login` for HTTP 200, then shuts down |
|
||||
|
||||
The `electron/` workspace also exposes:
|
||||
|
||||
- `npm run prepare:bundle` — runs `scripts/prepare-electron-standalone.mjs`
|
||||
- `npm run build:mac-x64` / `build:mac-arm64` — single-arch macOS builds
|
||||
- `npm run pack` — directory-only build for local testing (no installer)
|
||||
|
||||
## Directory Layout
|
||||
|
||||
```
|
||||
electron/
|
||||
├── package.json # Electron deps + electron-builder config
|
||||
├── main.js # Main process (24 KB — see annotations below)
|
||||
├── preload.js # contextBridge IPC bridge
|
||||
├── types.d.ts # AppInfo / ServerStatus / ElectronAPI types
|
||||
├── README.md # In-workspace notes
|
||||
├── assets/ # icon.png, icon.ico, icon.icns, tray-icon.png
|
||||
└── dist-electron/ # electron-builder output (gitignored)
|
||||
|
||||
scripts/
|
||||
├── prepare-electron-standalone.mjs # Stages .next/electron-standalone bundle
|
||||
└── smoke-electron-packaged.mjs # Post-build smoke test
|
||||
```
|
||||
|
||||
Both `main.js` and `preload.js` are **CommonJS `.js` files**, not TypeScript. The
|
||||
renderer-side typings live in `electron/types.d.ts`.
|
||||
|
||||
## IPC Bridge (`preload.js`)
|
||||
|
||||
The preload exposes a whitelisted API on `window.electronAPI` using `contextBridge`
|
||||
with `contextIsolation: true` and `nodeIntegration: false`.
|
||||
|
||||
```javascript
|
||||
const VALID_CHANNELS = {
|
||||
invoke: [
|
||||
"get-app-info",
|
||||
"open-external",
|
||||
"get-data-dir",
|
||||
"restart-server",
|
||||
"check-for-updates",
|
||||
"download-update",
|
||||
"install-update",
|
||||
"get-app-version",
|
||||
],
|
||||
send: ["window-minimize", "window-maximize", "window-close"],
|
||||
receive: ["server-status", "port-changed", "update-status"],
|
||||
};
|
||||
```
|
||||
|
||||
Exposed methods:
|
||||
|
||||
| Renderer call | Type |
|
||||
| ----------------------------------------------------------------- | -------------------------- |
|
||||
| `getAppInfo()` → `{ name, version, platform, isDev, port }` | invoke |
|
||||
| `openExternal(url)` | invoke |
|
||||
| `getDataDir()` | invoke |
|
||||
| `restartServer()` | invoke |
|
||||
| `getAppVersion()` | invoke |
|
||||
| `checkForUpdates()` / `downloadUpdate()` / `installUpdate()` | invoke |
|
||||
| `minimizeWindow()` / `maximizeWindow()` / `closeWindow()` | send |
|
||||
| `onServerStatus(cb)` / `onPortChanged(cb)` / `onUpdateStatus(cb)` | receive (returns disposer) |
|
||||
|
||||
The receive helpers return a **disposer function** rather than relying on
|
||||
`removeAllListeners` — this prevents listener accumulation when React components
|
||||
remount.
|
||||
|
||||
## Server Lifecycle
|
||||
|
||||
`main.js` spawns the Next.js standalone bundle directly with the Electron Node
|
||||
runtime to avoid native-module ABI mismatch with system Node:
|
||||
|
||||
```js
|
||||
spawn(process.execPath, [serverScript], {
|
||||
cwd: NEXT_SERVER_PATH,
|
||||
env: { ...serverEnv, PORT, NODE_ENV: "production", ELECTRON_RUN_AS_NODE: "1", NODE_PATH },
|
||||
stdio: "pipe",
|
||||
});
|
||||
```
|
||||
|
||||
Highlights:
|
||||
|
||||
- `waitForServer()` polls the URL up to 30 s before showing the window (no blank screen on cold start).
|
||||
- `stdio: "pipe"` captures stdout/stderr; ready phrases (`Ready` / `listening`) emit `server-status: running` over IPC.
|
||||
- `before-quit` waits up to 5 s for graceful SIGTERM (WAL checkpoint) then sends SIGKILL.
|
||||
- Port switcher in the tray (`20128`, `3000`, `8080`) stops and restarts the server, then reloads the BrowserWindow.
|
||||
|
||||
## Zero-config Secret Bootstrap
|
||||
|
||||
On first launch, the main process auto-generates and persists missing secrets:
|
||||
|
||||
| Secret | Source |
|
||||
| ------------------------ | ----------------------------------------------------------------------------------- |
|
||||
| `JWT_SECRET` | `crypto.randomBytes(64).toString("hex")` |
|
||||
| `STORAGE_ENCRYPTION_KEY` | `crypto.randomBytes(32).toString("hex")` (refuses if encrypted creds already exist) |
|
||||
| `API_KEY_SECRET` | `crypto.randomBytes(32).toString("hex")` |
|
||||
|
||||
Persisted to `<DATA_DIR>/server.env`. `DATA_DIR` resolves to:
|
||||
|
||||
- Windows: `%APPDATA%\omniroute`
|
||||
- Linux: `$XDG_CONFIG_HOME/omniroute` or `~/.omniroute`
|
||||
- macOS: `~/.omniroute`
|
||||
|
||||
## Window & Tray
|
||||
|
||||
- `BrowserWindow`: 1400×900 (min 1024×700), `backgroundColor: "#0a0a0a"`.
|
||||
- macOS: `titleBarStyle: "hiddenInset"`, traffic-light at `{ x: 16, y: 16 }`.
|
||||
- Windows/Linux: native title bar.
|
||||
- Close button minimizes to tray; the tray menu has **Open OmniRoute**, **Open Dashboard** (external browser), **Server Port** submenu, **Check for Updates**, **Quit**.
|
||||
|
||||
## Content Security Policy
|
||||
|
||||
Set via `session.defaultSession.webRequest.onHeadersReceived`. Notable directives:
|
||||
|
||||
- `frame-ancestors 'none'`, `object-src 'none'`, `child-src 'none'`
|
||||
- `connect-src 'self' http://localhost:* http://127.0.0.1:* ws://localhost:* ws://127.0.0.1:* https://*.omniroute.online https://*.omniroute.dev`
|
||||
- Dev mode adds `'unsafe-eval'` to `script-src` only
|
||||
|
||||
## Auto-update
|
||||
|
||||
Uses `electron-updater` with the GitHub provider (`diegosouzapw/OmniRoute`).
|
||||
|
||||
- `autoDownload = false`, `autoInstallOnAppQuit = true`
|
||||
- Events forwarded to renderer via `update-status` IPC:
|
||||
`checking`, `available`, `not-available`, `downloading` (with `percent`), `downloaded`, `error`
|
||||
- `installUpdate()` kills the server then calls `autoUpdater.quitAndInstall()`
|
||||
- Skipped in dev mode (`!app.isPackaged`)
|
||||
|
||||
## Build Pipeline
|
||||
|
||||
1. `npm run build` → Next.js standalone in `.next/standalone`.
|
||||
2. `prepare-electron-standalone.mjs` → re-stages into `.next/electron-standalone` and rewrites absolute paths inside `server.js` + `required-server-files.json` so the bundle is relocatable.
|
||||
3. `electron-builder` packages `main.js`, `preload.js`, `node_modules`, and `extraResources: { ../.next/electron-standalone → app }`.
|
||||
|
||||
### Build targets
|
||||
|
||||
| OS | Targets |
|
||||
| ------- | ----------------------------------------- |
|
||||
| Windows | NSIS installer + portable (x64) |
|
||||
| macOS | DMG (Intel + arm64, drag-to-Applications) |
|
||||
| Linux | AppImage + DEB (x64 + arm64) |
|
||||
|
||||
NSIS settings: `oneClick: false`, lets the user choose the install directory, creates Desktop and Start-Menu shortcuts.
|
||||
|
||||
## Smoke Testing Packaged Build
|
||||
|
||||
```bash
|
||||
npm run electron:smoke:packaged
|
||||
```
|
||||
|
||||
`scripts/smoke-electron-packaged.mjs`:
|
||||
|
||||
- Auto-discovers the packaged binary in `electron/dist-electron/` for the current platform.
|
||||
- Launches with isolated `HOME`/`APPDATA`/`XDG_*` directories so it doesn't touch developer data.
|
||||
- Polls `http://127.0.0.1:20128/login` for HTTP 200 within 45 s.
|
||||
- Watches stderr/stdout for fatal patterns (`Cannot find module`, `MODULE_NOT_FOUND`, `ERR_DLOPEN_FAILED`, `Failed to start server`, etc.).
|
||||
- Waits 2 s of stable runtime after readiness, then issues SIGTERM and waits for the port to free.
|
||||
- In CI, automatically passes `--no-sandbox --disable-gpu` (and `--disable-dev-shm-usage` on Linux).
|
||||
|
||||
Env overrides: `ELECTRON_SMOKE_APP_EXECUTABLE`, `ELECTRON_SMOKE_URL`, `ELECTRON_SMOKE_TIMEOUT_MS`, `ELECTRON_SMOKE_SETTLE_MS`, `ELECTRON_SMOKE_DATA_DIR`, `ELECTRON_SMOKE_KEEP_DATA`, `ELECTRON_SMOKE_STREAM_LOGS`.
|
||||
|
||||
## Code Signing
|
||||
|
||||
`electron/package.json` does **not** wire signing credentials directly. Pass them via env vars to `electron-builder`:
|
||||
|
||||
### macOS
|
||||
|
||||
```bash
|
||||
export APPLE_ID=<email>
|
||||
export APPLE_APP_SPECIFIC_PASSWORD=<password>
|
||||
export APPLE_TEAM_ID=<id>
|
||||
export CSC_LINK=path/to/cert.p12
|
||||
export CSC_KEY_PASSWORD=<cert-password>
|
||||
npm run electron:build:mac
|
||||
```
|
||||
|
||||
### Windows
|
||||
|
||||
```bash
|
||||
export CSC_LINK=path/to/cert.pfx
|
||||
export CSC_KEY_PASSWORD=<cert-password>
|
||||
npm run electron:build:win
|
||||
```
|
||||
|
||||
### Linux
|
||||
|
||||
AppImage signing is optional — set `LINUX_GPG_KEY` if signing.
|
||||
|
||||
## Distribution
|
||||
|
||||
Artifacts land in `electron/dist-electron/`:
|
||||
|
||||
- `OmniRoute Setup X.Y.Z.exe`, `OmniRoute-X.Y.Z-portable.exe` (Windows)
|
||||
- `OmniRoute-X.Y.Z-mac.dmg`, `OmniRoute-X.Y.Z-arm64-mac.dmg` (macOS)
|
||||
- `OmniRoute-X.Y.Z.AppImage`, `omniroute-desktop_X.Y.Z_amd64.deb` (Linux)
|
||||
|
||||
Releases are published to GitHub Releases (`diegosouzapw/OmniRoute`), which is also where `electron-updater` checks for new versions.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Fix |
|
||||
| --------------------------------------------------------------- | --------------------------------------------------------------------------- |
|
||||
| `Cannot find module 'better-sqlite3'` after Electron major bump | `cd electron && npm rebuild` |
|
||||
| `ERR_DLOPEN_FAILED` for native module | Re-run `prepare:bundle` and verify ABI matches Electron's Node |
|
||||
| Window appears blank on Linux | Confirm Next.js server actually bound to PORT (check `[Server]` logs) |
|
||||
| macOS notarization stalls | Ensure `APPLE_*` vars are exported, not just in `.env` |
|
||||
| Windows SmartScreen warning | Sign with EV cert, or users right-click → "Run anyway" |
|
||||
| Smoke test fails with port-in-use | Stop any local dev server on 20128 before running `electron:smoke:packaged` |
|
||||
|
||||
## See Also
|
||||
|
||||
- [SETUP_GUIDE.md](./SETUP_GUIDE.md)
|
||||
- [RELEASE_CHECKLIST.md](./RELEASE_CHECKLIST.md)
|
||||
- Source: `electron/main.js`, `electron/preload.js`, `electron/package.json`
|
||||
- Helpers: `scripts/prepare-electron-standalone.mjs`, `scripts/smoke-electron-packaged.mjs`
|
||||
244
docs/EVALS.md
Normal file
244
docs/EVALS.md
Normal file
@@ -0,0 +1,244 @@
|
||||
# Evaluations (Evals)
|
||||
|
||||
> **Source of truth:** `src/lib/evals/`, `src/lib/db/evals.ts`, `src/app/api/evals/`
|
||||
> **Last updated:** 2026-05-13 — v3.8.0
|
||||
|
||||
OmniRoute ships a generic evaluation framework you can use to benchmark routing
|
||||
configurations, single providers/models, or the bundled "golden set" suites.
|
||||
Use it to verify routing changes, validate new providers, and gate releases
|
||||
before promoting them to production traffic.
|
||||
|
||||
The framework is implemented as:
|
||||
|
||||
- A pure runner (`src/lib/evals/evalRunner.ts`) that registers in-memory
|
||||
built-in suites, evaluates outputs against expected criteria, and aggregates
|
||||
scorecards.
|
||||
- A persistence layer (`src/lib/db/evals.ts`) for custom (user-defined) suites
|
||||
and historical runs in SQLite.
|
||||
- An orchestration layer (`src/lib/evals/runtime.ts`) that executes each case
|
||||
by dispatching real calls to `POST /v1/chat/completions`, captures latency
|
||||
and outputs, and persists the run.
|
||||
- REST endpoints under `/api/evals/*` (management-auth only).
|
||||
- A dashboard surface at `Dashboard → Usage → Evals` (`EvalsTab.tsx`).
|
||||
|
||||
## Concepts
|
||||
|
||||
### Suite
|
||||
|
||||
A suite is a named collection of test cases with a `description` and one or
|
||||
more cases. Suites come from two sources:
|
||||
|
||||
| Source | Where defined | Mutable at runtime? |
|
||||
| ---------- | --------------------------------------------- | ------------------- |
|
||||
| `built-in` | Registered via `registerSuite()` at boot | No (code-defined) |
|
||||
| `custom` | Stored in SQLite `eval_suites` + `eval_cases` | Yes (via API/UI) |
|
||||
|
||||
The current built-in suites (see `src/lib/evals/evalRunner.ts`):
|
||||
|
||||
- `golden-set` — 10 baseline cases across greeting/math/translation/safety
|
||||
- `coding-proficiency` — Python/JS/SQL/TS/bug detection
|
||||
- `reasoning-logic` — syllogisms, word problems, pattern recognition
|
||||
- `multilingual` — translation and language detection
|
||||
- `safety-guardrails` — PII, jailbreak, refusal, bias awareness
|
||||
- `instruction-following` — JSON-only, numbered lists, language constraints
|
||||
- `codex-comparison` — head-to-head coding tasks intended for compare mode
|
||||
|
||||
### Case
|
||||
|
||||
Each case carries:
|
||||
|
||||
| Field | Description |
|
||||
| ---------- | ------------------------------------------------------------ |
|
||||
| `id` | Stable identifier (used to key outputs and metrics) |
|
||||
| `name` | Human-readable label |
|
||||
| `model` | Default model when the run uses `suite-default` targeting |
|
||||
| `input` | `{ messages, max_tokens? }` — sent to `/v1/chat/completions` |
|
||||
| `expected` | `{ strategy, value }` — scoring rubric (see below) |
|
||||
| `tags` | Optional labels (e.g. `safety`, `pii`, `jailbreak`) |
|
||||
|
||||
### Target
|
||||
|
||||
The same suite can be run against different targets. The target schema is
|
||||
`evalTargetSchema` in `src/shared/validation/schemas.ts`:
|
||||
|
||||
| Target type | `id` | Behavior |
|
||||
| --------------- | ---------- | --------------------------------------------------------------- |
|
||||
| `suite-default` | `null` | Each case uses its built-in `model` field |
|
||||
| `model` | model name | Force every case through one direct model (e.g. `gpt-4o`) |
|
||||
| `combo` | combo name | Run every case through one combo (exercises the routing engine) |
|
||||
|
||||
For `model` and `combo`, the `id` field is required (enforced by Zod
|
||||
`superRefine`). When `compareTarget` is provided, both targets must differ —
|
||||
the runner persists both runs under the same `runGroupId` for A/B comparison.
|
||||
|
||||
## Scoring Rubrics
|
||||
|
||||
Implemented in `evaluateCase()` (evalRunner.ts):
|
||||
|
||||
| Strategy | Pass when… |
|
||||
| ---------- | -------------------------------------------------------------------- |
|
||||
| `exact` | `actualOutput === expected.value` |
|
||||
| `contains` | `actualOutput.toLowerCase().includes(expected.value.toLowerCase())` |
|
||||
| `regex` | `new RegExp(expected.value).test(actualOutput)` is truthy |
|
||||
| `custom` | `expected.fn(actualOutput, evalCase)` returns truthy (built-in only) |
|
||||
|
||||
**Note:** Custom-function scoring is reserved for code-defined (built-in)
|
||||
suites because functions cannot be serialized through the API. The
|
||||
`evalCaseBuilderSchema` only accepts `contains | exact | regex` for
|
||||
user-created suites.
|
||||
|
||||
There is no LLM-as-judge or embedding-based similarity scorer today — it would
|
||||
be a clean extension point in `evaluateCase()`.
|
||||
|
||||
## Database Schema
|
||||
|
||||
Three tables (migrations `030_create_eval_runs.sql` and
|
||||
`031_create_eval_suites.sql`):
|
||||
|
||||
| Table | Purpose |
|
||||
| ------------- | ---------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `eval_suites` | Custom suite metadata (`id`, `name`, `description`) |
|
||||
| `eval_cases` | Cases per suite — `input_json`, `expected_*`, `tags_json` |
|
||||
| `eval_runs` | Historical runs — `pass_rate`, `total`, `passed`, `failed`, `avg_latency_ms`, `summary_json`, `results_json`, `outputs_json` |
|
||||
|
||||
Built-in suites are **not** stored in the DB. They live in memory and are
|
||||
re-registered every time `evalRunner.ts` is imported.
|
||||
|
||||
## REST API
|
||||
|
||||
All endpoints require management auth (`requireManagementAuth`) — they are not
|
||||
part of the public proxy surface.
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
| ----------------------------- | -------- | ------------------------------------------------------------- |
|
||||
| `/api/evals` | `GET` | List suites + recent runs + scorecard + targets + keys |
|
||||
| `/api/evals` | `POST` | Run a suite (single or compare) — schema `evalRunSuiteSchema` |
|
||||
| `/api/evals/{suiteId}` | `GET` | Fetch one suite (built-in or custom) |
|
||||
| `/api/evals/suites` | `POST` | Create a custom suite — schema `evalSuiteSaveSchema` |
|
||||
| `/api/evals/suites/{suiteId}` | `GET` | Fetch a custom suite |
|
||||
| `/api/evals/suites/{suiteId}` | `PUT` | Replace a custom suite (cases get re-inserted) |
|
||||
| `/api/evals/suites/{suiteId}` | `DELETE` | Delete a custom suite and its cases |
|
||||
|
||||
### Running a suite
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/api/evals \
|
||||
-H "Cookie: auth_token=..." \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"suiteId": "golden-set",
|
||||
"target": { "type": "combo", "id": "my-combo" },
|
||||
"apiKeyId": "optional-api-key-uuid"
|
||||
}'
|
||||
```
|
||||
|
||||
Optional fields:
|
||||
|
||||
- `outputs` — `Record<caseId, string>` of pre-computed outputs. When provided,
|
||||
the runner **skips dispatch** and only scores the cached outputs (useful for
|
||||
offline evaluation).
|
||||
- `compareTarget` — second target to run in parallel; both runs share a
|
||||
generated `runGroupId` for head-to-head viewing.
|
||||
- `apiKeyId` — internal API key used to authenticate the dispatched
|
||||
`/v1/chat/completions` calls. Required when `REQUIRE_API_KEY` is enabled.
|
||||
|
||||
### Creating a custom suite
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/api/evals/suites \
|
||||
-H "Cookie: auth_token=..." \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "Production smoke",
|
||||
"description": "Quick sanity check before deploy",
|
||||
"cases": [
|
||||
{
|
||||
"name": "JSON shape",
|
||||
"model": "gpt-4o",
|
||||
"input": { "messages": [{ "role": "user", "content": "Reply with {\"ok\": true}" }] },
|
||||
"expected": { "strategy": "regex", "value": "\"ok\"\\s*:\\s*true" }
|
||||
}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
## Dispatch Pipeline
|
||||
|
||||
`runEvalSuiteAgainstTarget()` (`src/lib/evals/runtime.ts`):
|
||||
|
||||
1. Resolves the suite (built-in or custom).
|
||||
2. For each case, builds a `Request` to `/v1/chat/completions` with the case's
|
||||
`messages`, the resolved `model`, `stream: false`, and `max_tokens: 512`
|
||||
(or the case override).
|
||||
3. Calls the chat handler directly (in-process — no extra HTTP hop).
|
||||
4. Captures latency and extracts text from either `choices[0].message.content`
|
||||
or the Responses-API `output[]` payload.
|
||||
5. Scores all outputs via `runSuite()`, then persists via `saveEvalRun()`.
|
||||
|
||||
Cases run **sequentially**. There is no concurrency flag today.
|
||||
|
||||
## Dashboard
|
||||
|
||||
The UI lives at `Dashboard → Usage → Evals`
|
||||
(`src/app/(dashboard)/dashboard/usage/components/EvalsTab.tsx`). From there you
|
||||
can:
|
||||
|
||||
- Browse built-in and custom suites with case-by-case preview.
|
||||
- Create/edit/delete custom suites with the case builder.
|
||||
- Pick a target (suite defaults / model / combo), optionally a second
|
||||
`compareTarget`, optionally an API key, then run on demand.
|
||||
- Inspect run history, per-case pass/fail, latency, and captured outputs.
|
||||
- See the rolling scorecard aggregated across the latest run per
|
||||
`(suite, target)` scope.
|
||||
|
||||
## Relationship with the Auto-Assessment RFC
|
||||
|
||||
A separate, narrower assessment subsystem lives at `src/domain/assessment/`
|
||||
and is documented in [docs/RFC-AUTO-ASSESSMENT.md](./RFC-AUTO-ASSESSMENT.md).
|
||||
That RFC targets the Auto Combo engine — automatically scoring providers and
|
||||
models so combos can self-heal when upstreams fail. It uses its own runner,
|
||||
its own categorizer, and its own scoring logic.
|
||||
|
||||
The Evals framework documented here is the **broader, general-purpose
|
||||
testing surface**. Prefer it for arbitrary regression suites, A/B comparisons,
|
||||
and per-release smoke tests. Use the Auto-Assessment subsystem when you need
|
||||
real-time provider health to influence routing decisions.
|
||||
|
||||
## CI Integration
|
||||
|
||||
There is no dedicated `eval:ci` npm script today. Two paths if you want to
|
||||
gate releases on eval results:
|
||||
|
||||
- **HTTP path**: stand up the server, hit `POST /api/evals` with a known
|
||||
`suiteId` + `target`, and assert `runs[].summary.passRate >= N` in the
|
||||
response.
|
||||
- **In-process path**: import `runEvalSuiteAgainstTarget()` from
|
||||
`@/lib/evals/runtime` from a script, run against a test DB, and check the
|
||||
returned `PersistedEvalRun.summary`.
|
||||
|
||||
Tests covering the route and history live at
|
||||
`tests/unit/evals-route.test.ts` and `tests/unit/evals-history.test.ts`.
|
||||
|
||||
## Extension Points
|
||||
|
||||
Common changes and where to make them:
|
||||
|
||||
- **New scoring strategy** — extend the `switch (evalCase.expected.strategy)`
|
||||
block in `evaluateCase()` (`evalRunner.ts`) and widen `EvalCaseStrategy` in
|
||||
`src/lib/db/evals.ts` plus `evalCaseBuilderSchema` in `schemas.ts`.
|
||||
- **New built-in suite** — define a suite object and call `registerSuite()` at
|
||||
the bottom of `evalRunner.ts`. It will be auto-discovered by `listSuites()`.
|
||||
- **Run with concurrency** — change the sequential `for` loop in
|
||||
`runEvalSuiteAgainstTarget()` to a bounded `Promise.all` (no concurrency
|
||||
control exists today).
|
||||
- **Stream/tool-call cases** — currently the runner forces `stream: false`.
|
||||
Streaming or tool-aware evaluation would require changes in `runtime.ts`
|
||||
(capture and aggregate SSE chunks before scoring).
|
||||
|
||||
## See Also
|
||||
|
||||
- [USER_GUIDE.md](./USER_GUIDE.md) — overall product walkthrough
|
||||
- [ARCHITECTURE.md](./ARCHITECTURE.md) — request pipeline reference
|
||||
- [RFC-AUTO-ASSESSMENT.md](./RFC-AUTO-ASSESSMENT.md) — Auto Combo scoring
|
||||
- Source: `src/lib/evals/`, `src/lib/db/evals.ts`, `src/app/api/evals/`
|
||||
- UI: `src/app/(dashboard)/dashboard/usage/components/EvalsTab.tsx`
|
||||
@@ -4,6 +4,43 @@
|
||||
|
||||
Visual guide to every section of the OmniRoute dashboard.
|
||||
|
||||
> 📅 **Last updated:** 2026-05-13 — **v3.8.0**
|
||||
|
||||
---
|
||||
|
||||
## ✨ v3.8.0 Highlights
|
||||
|
||||
The v3.7.x → v3.8.0 cycle added zero-config auto routing, new providers, OAuth flows, deeper resilience, and a much richer CLI experience. Headline features below — full details further in the document and in linked specs.
|
||||
|
||||
- 🤖 **Auto Combo / Zero-config auto-routing** — use prefixes `auto/coding`, `auto/fast`, `auto/cheap`, `auto/offline`, `auto/smart`, `auto/lkgp`. Backed by a 9-factor scoring engine and 4 curated **mode packs** (ship-fast, cost-saver, quality-first, offline-friendly)
|
||||
- 🆕 **Command Code provider** (#2199) — first-class registration with model catalog and quota tracking
|
||||
- 🆕 **Z.AI provider** — new free-tier provider with quota labels
|
||||
- 🎬 **KIE media expansion** — extended catalog including video generation models
|
||||
- 🔐 **Windsurf + Devin CLI OAuth flows** (#2168) — end-to-end browser-based login
|
||||
- 🆓 **9 new free providers** — LLM7, Lepton, Kluster, UncloseAI, BazaarLink, Completions, Enally, FreeTheAi, Command Code
|
||||
- 🎯 **Manifest-aware tier routing W1–W4** — provider manifests drive weighted tier selection
|
||||
- 🎨 **Cursor full OpenAI parity** — tool calls, streaming, session management end-to-end
|
||||
- 📊 **Cursor Pro plan usage** — quota & cycle data surfaced in the provider-limits dashboard
|
||||
- ⚡ **Service tier breakdown / Codex fast tier analytics** — per-tier consumption visibility
|
||||
- 📌 **Per-session sticky routing** — Codex sessions pin to the same account between turns
|
||||
- 🔊 **Inworld TTS enhancements** — voice catalogs, streaming, and latency improvements
|
||||
- 🔑 **Kiro headless auth** — login via local `kiro-cli` SQLite store, no browser required
|
||||
- 📉 **DeepSeek quota and limit monitoring** — daily/monthly usage exposed via dashboard
|
||||
- 🔄 **Reset-aware routing strategy** — combos now prefer accounts whose quota window resets soonest
|
||||
- ⏱️ **`fallbackDelayMs`** and **dynamic tool limit detection** — finer fallback timing + per-provider tool-count limits
|
||||
- 🔧 **Background mode degradation (Responses API)** — falls back to synchronous mode with a structured warning when an upstream lacks background polling
|
||||
- 🚦 **Per-provider 429 classification** + `useUpstream429BreakerHints` toggle — finer breaker behavior using upstream rate-limit hints
|
||||
- 🩺 **Model cooldowns dashboard** — observe per-model lockouts and manually re-enable from the UI
|
||||
- 🔒 **MITM dynamic Linux cert detection** — works across Debian/Ubuntu, Fedora/RHEL, Arch, and other distros
|
||||
- 💻 **CLI enhancement suite** — 20+ commands including `omniroute providers`, `omniroute combos`, `omniroute doctor`, `omniroute setup`
|
||||
- 🔍 **Qdrant embedding model discovery** — automatic vector-store model probe
|
||||
- 🔑 **API Manager / Bearer keys with `manage` scope** — perform admin operations programmatically via API
|
||||
- 🏥 **Combo target health analytics** + **structured combo builder** — per-target health & UI builder for assembling `(provider, model, connection)` steps
|
||||
- 🤝 **GitLab Duo OAuth provider** — login with GitLab credentials
|
||||
- 🧠 **Reasoning Replay Cache** — hybrid in-memory + SQLite persistence of reasoning traces
|
||||
|
||||
📚 **Related docs:** [Skills Framework](./SKILLS.md) · [Memory System](./MEMORY.md) · [Cloud Agents](./CLOUD_AGENT.md) · [Webhooks](./WEBHOOKS.md) · [Reasoning Replay Cache](./REASONING_REPLAY.md)
|
||||
|
||||
---
|
||||
|
||||
## 🔌 Providers
|
||||
@@ -67,14 +104,15 @@ Customizable color themes for the entire dashboard. Choose from 7 preset colors
|
||||
|
||||
## ⚙️ Settings
|
||||
|
||||
Comprehensive settings panel with tabs:
|
||||
Comprehensive settings panel with **7 tabs**:
|
||||
|
||||
- **General** — System storage, backup management (export/import database)
|
||||
- **Appearance** — Theme selector (dark/light/system), color theme presets and custom colors, health log visibility, sidebar item visibility controls, Endpoint tunnel visibility controls
|
||||
- **AI** — AI assistant features, default routing presets (Auto Combo `auto/coding`, `auto/fast`, `auto/cheap`, `auto/smart`), reasoning replay cache, and skill/memory toggles
|
||||
- **Security** — API endpoint protection, custom provider blocking, IP filtering, session info
|
||||
- **Routing** — Model aliases, background task degradation
|
||||
- **Resilience** — Rate limit persistence, circuit breaker tuning, auto-disable banned accounts, provider expiration monitoring, **Context Relay** handoff threshold and summary model configuration
|
||||
- **Advanced** — Configuration overrides, configuration audit trail, fallback degradation mode
|
||||
- **Routing** — Model aliases, background task degradation, manifest-aware tier routing (W1–W4), `fallbackDelayMs`, per-session sticky routing
|
||||
- **Resilience** — Rate limit persistence, circuit breaker tuning, auto-disable banned accounts, provider expiration monitoring, **Context Relay** handoff threshold and summary model configuration, per-provider 429 classification & `useUpstream429BreakerHints` toggle, model cooldowns
|
||||
- **Advanced** — Configuration overrides, configuration audit trail, fallback degradation mode, background mode degradation for Responses API
|
||||
|
||||
|
||||
|
||||
@@ -90,12 +128,13 @@ One-click configuration for AI coding tools: Claude Code, Codex CLI, Gemini CLI,
|
||||
|
||||
## 🤖 CLI Agents _(v2.0.11+)_
|
||||
|
||||
Dashboard for discovering and managing CLI agents. Shows a grid of 14 built-in agents (Codex, Claude, Goose, Gemini CLI, OpenClaw, Aider, OpenCode, Cline, Qwen Code, ForgeCode, Amazon Q, Open Interpreter, Cursor CLI, Warp) with:
|
||||
Dashboard for discovering and managing CLI agents. Shows a grid of 18 built-in agents (Codex, Claude, Goose, Gemini CLI, OpenClaw, Aider, OpenCode, Cline, Qwen Code, ForgeCode, Amazon Q, Open Interpreter, Cursor CLI, Warp, **Windsurf**, **Devin CLI**, **Kimi Coding**, **Command Code**) with:
|
||||
|
||||
- **Installation status** — Installed / Not Found with version detection
|
||||
- **Protocol badges** — stdio, HTTP, etc.
|
||||
- **Custom agents** — Register any CLI tool via form (name, binary, version command, spawn args)
|
||||
- **CLI Fingerprint Matching** — Per-provider toggle to match native CLI request signatures, reducing ban risk while preserving proxy IP
|
||||
- **OAuth-backed agents** — Windsurf & Devin CLI now use browser OAuth flows for authentication (v3.8.0+)
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -1,407 +1,232 @@
|
||||
# 🆓 Free LLM API Providers — Consolidated Directory
|
||||
# Free Tiers
|
||||
|
||||
> **The ultimate aggregated reference for all permanently free LLM API providers.**
|
||||
> Consolidated from 6 community repositories. Use with OmniRoute to route through 25+ free providers simultaneously.
|
||||
> **Last consolidated:** 2026-05-13 — OmniRoute v3.8.0
|
||||
> **Source of truth:** `src/shared/constants/providers.ts` (`FREE_PROVIDERS`, `OAUTH_PROVIDERS`, and `APIKEY_PROVIDERS` entries flagged with `hasFree: true` + `freeNote`)
|
||||
|
||||
_Last consolidated: May 2026 · Sources: awesome-free-llm-apis, awesome-free-llm-apis2, free-llm-api-resources, Free-LLM-Collection, FREE-LLM-API-Provider, gpt4free_
|
||||
This page lists providers with usable free tiers shipped in OmniRoute v3.8.0. The data is derived from the provider catalog. If a provider does not appear here, it either has no free tier in the catalog or its `hasFree` flag is `false`.
|
||||
|
||||
Add credentials from the dashboard (`/dashboard/providers/new`) — OmniRoute reads keys from the database, not from per-provider environment variables. The only env vars that influence provider behavior are listed in the [Environment Variables](#environment-variables) section.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
## How free providers are wired
|
||||
|
||||
- [Quick Comparison](#quick-comparison)
|
||||
- [Provider APIs (First-Party)](#provider-apis-first-party)
|
||||
- [Inference Providers (Third-Party)](#inference-providers-third-party)
|
||||
- [China-Based Providers](#china-based-providers)
|
||||
- [Trial Credit Providers](#trial-credit-providers)
|
||||
- [Using with OmniRoute](#using-with-omniroute)
|
||||
- [Glossary](#glossary)
|
||||
OmniRoute classifies providers into the following groups in `src/shared/constants/providers.ts`:
|
||||
|
||||
| Group | Auth | Example IDs |
|
||||
| ------------------ | ------------------------------ | ------------------------------------------- |
|
||||
| `FREE_PROVIDERS` | OAuth or vendor account | `qoder`, `gemini-cli`, `kiro`, `amazon-q` |
|
||||
| `OAUTH_PROVIDERS` | OAuth | `claude`, `cursor`, `windsurf`, `devin-cli` |
|
||||
| `APIKEY_PROVIDERS` | API key (with `hasFree: true`) | `groq`, `cerebras`, `mistral`, `gemini` |
|
||||
|
||||
A provider appears in the **free pool** when either:
|
||||
|
||||
- It is listed in the `FREE_PROVIDERS` map (and not flagged `deprecated: true`), or
|
||||
- It is listed in `APIKEY_PROVIDERS` with `hasFree: true` and a `freeNote` string, or
|
||||
- It is an OAuth provider whose vendor offers a free tier on top of OAuth sign-in.
|
||||
|
||||
---
|
||||
|
||||
## Quick Comparison
|
||||
## Quick reference (API key providers with `hasFree: true`)
|
||||
|
||||
All free providers at a glance, sorted by generosity of free tier:
|
||||
| Provider | ID | Free tier note |
|
||||
| ------------------ | --------------- | ---------------------------------------------------------------------------------------------------- |
|
||||
| AgentRouter | `agentrouter` | $200 free credits on signup — multi-model routing gateway |
|
||||
| AI21 Labs | `ai21` | $10 trial credits on signup (valid 3 months), no credit card required |
|
||||
| AI/ML API | `aimlapi` | $0.025/day free credits — 200+ models via single endpoint |
|
||||
| BazaarLink | `bazaarlink` | Free tier with `auto:free` routing — zero-cost inference, no credit card required |
|
||||
| Baseten | `baseten` | $30 free trial credits for GPU inference |
|
||||
| Blackbox AI | `blackbox` | Free tier: unlimited basic chat plus Minimax-M2.5, no credit card required |
|
||||
| Bytez | `bytez` | $1 free credits, refreshes every 4 weeks |
|
||||
| Cerebras | `cerebras` | Free: 1M tokens/day, 60K TPM — world's fastest inference |
|
||||
| Cloudflare AI | `cloudflare-ai` | Free 10K Neurons/day: ~150 LLM responses or 500s Whisper audio |
|
||||
| Cohere | `cohere` | Free Trial: 1,000 API calls/month for testing, no credit card required |
|
||||
| Completions.me | `completions` | Free unlimited access to Claude, GPT, Gemini — no credit card, no rate limits |
|
||||
| DeepInfra | `deepinfra` | Free signup credits for API testing and model exploration |
|
||||
| DeepSeek | `deepseek` | 5M free tokens on signup — no credit card required |
|
||||
| Enally AI | `enally` | Free for students and developers — no credit card, OTP verification |
|
||||
| Fireworks AI | `fireworks` | $1 free starter credits on signup for API testing |
|
||||
| FreeTheAi | `freetheai` | Community-run — free forever, no paid tiers, no credit card |
|
||||
| Gemini (AI Studio) | `gemini` | Free forever: 1,500 req/day for Gemini 2.5 Flash — no credit card |
|
||||
| GLHF Chat | `glhf` | Free tier for open-source model inference |
|
||||
| Groq | `groq` | Free tier: 30 RPM / 14.4K RPD — no credit card |
|
||||
| HuggingFace | `huggingface` | Free Inference API for thousands of models (Whisper, VITS, SDXL…) |
|
||||
| Hyperbolic | `hyperbolic` | $1–5 trial credits on signup for serverless inference |
|
||||
| Inference.net | `inference-net` | $25 free credits on signup plus research grants available |
|
||||
| Jina AI | `jina-ai` | 10M free tokens on signup (non-commercial), no credit card required |
|
||||
| Kluster AI | `kluster` | $5 free credits on signup — DeepSeek R1, Llama 4 Maverick/Scout, Qwen3 235B |
|
||||
| Lepton AI | `lepton` | Free tier available — fast inference on custom hardware |
|
||||
| LLM7.io | `llm7` | No signup required — 2 req/s, 20 RPM, 100 req/hr free tier |
|
||||
| LongCat AI | `longcat` | 50M tokens/day (Flash-Lite) + 500K/day (Chat/Thinking) — 100% free while public beta |
|
||||
| Mistral | `mistral` | Free Experiment tier: rate-limited access to all models, no credit card required |
|
||||
| Modal | `modal` | $30/month free credits for new accounts |
|
||||
| Morph | `morph` | Free tier: 250K credits/month, $0 |
|
||||
| Nebius | `nebius` | ~$1 trial credits on signup for API testing |
|
||||
| NLP Cloud | `nlpcloud` | Trial credits for new accounts |
|
||||
| Nous Research | `nous-research` | Free tier: 50 RPM, 500,000 TPM — no credit card |
|
||||
| Novita AI | `novita` | $0.50 trial credits on signup (valid about 1 year) |
|
||||
| nScale | `nscale` | $5 free credits on signup for inference testing |
|
||||
| NVIDIA NIM | `nvidia` | Free dev access: ~40 RPM, 70+ models (Kimi K2.5, GLM 4.7, DeepSeek V3.2…) |
|
||||
| OpenRouter | `openrouter` | Free models at $0/token with `:free` suffix — 20 RPM / 200 RPD |
|
||||
| Pollinations AI | `pollinations` | No API key required for free public endpoint. Optional Spore tier: ~0.01 pollen/hour |
|
||||
| Predibase | `predibase` | $25 free trial credits (30-day validity) |
|
||||
| PublicAI | `publicai` | Free community inference tier for testing |
|
||||
| Puter AI | `puter` | 500+ models (GPT-5, Claude Opus 4, Gemini 3 Pro, Grok 4, DeepSeek V3…) — users pay via Puter account |
|
||||
| Reka | `reka` | $10/month recurring free API credits |
|
||||
| SambaNova | `sambanova` | $5 free credits on signup (30-day validity), no credit card required |
|
||||
| Scaleway AI | `scaleway` | 1M free tokens for new accounts — EU/GDPR compliant (Paris), Qwen3 235B & Llama 70B |
|
||||
| SiliconFlow | `siliconflow` | $1 free credits plus permanently free models after identity verification |
|
||||
| Together AI | `together` | $25 signup credits + 3 permanently free models: Llama 3.3 70B, Vision, DeepSeek-R1 distill |
|
||||
| UncloseAI | `uncloseai` | Free forever — no signup, no credit card. OpenAI-compatible endpoints |
|
||||
| Voyage AI | `voyage-ai` | 200M free tokens for embeddings and reranking |
|
||||
|
||||
| Provider | Type | Best Free Model | RPM | RPD | Tokens | OpenAI Compat | Speed |
|
||||
| ----------------- | --------- | ---------------------- | ------- | ------------ | ---------------- | --------------- | --------- |
|
||||
| **Groq** | Inference | Llama 3.3 70B | 30 | 14,400 | 6K TPM | ✅ | 🟢 Fast |
|
||||
| **Cerebras** | Inference | Qwen3 235B | 30 | 14,400 | 1M TPD | ✅ | 🟢 Fast |
|
||||
| **Mistral AI** | Provider | Mistral Large 3 | 60 | Unlimited | 1B/month | ✅ | 🟡 Medium |
|
||||
| **Google Gemini** | Provider | Gemini 2.5 Flash | 5–15 | 20–1,500 | 250K TPM | ✅ | 🟢 Fast |
|
||||
| **NVIDIA NIM** | Inference | 129 models | 40 | — | — | ✅ | 🟡 Medium |
|
||||
| **Ollama Cloud** | Inference | 400+ models | — | — | Session limits | ❌ (Ollama API) | 🟡 Medium |
|
||||
| **OpenRouter** | Inference | 35+ free models | 20 | 50–1,000 | — | ✅ | 🟡 Medium |
|
||||
| **GitHub Models** | Inference | GPT-4.1, GPT-5 | 10–15 | 50–150 | 8K in/4K out | ✅ | 🟡 Medium |
|
||||
| **Cloudflare AI** | Inference | 50+ models | — | 10K neurons | — | ⚠️ Partial | 🟡 Medium |
|
||||
| **Hugging Face** | Inference | Thousands | — | — | $0.10/mo credits | ✅ | 🔴 Slow |
|
||||
| **Cohere** | Provider | Command A (111B) | 20 | — | 1K calls/month | ⚠️ Partial | 🟡 Medium |
|
||||
| **Pollinations** | Inference | Text+Image+Video+Audio | — | Hourly reset | — | ✅ | 🟡 Medium |
|
||||
| **Z.AI (Zhipu)** | Provider | GLM-4.7-Flash | — | — | Undocumented | ✅ | 🟡 Medium |
|
||||
| **SiliconFlow** | Inference | Qwen3-8B | 1,000 | — | 50K TPM | ✅ | 🟡 Medium |
|
||||
| **Kilo Code** | Inference | Free auto-router | ~200/hr | — | — | ✅ | 🟡 Medium |
|
||||
| **LLM7.io** | Inference | 30+ models | 15–30 | — | — | ✅ | 🟡 Medium |
|
||||
| **Kluster AI** | Inference | DeepSeek-R1 | — | — | Undocumented | ✅ | 🟡 Medium |
|
||||
| **ModelScope** | Inference | Qwen, DeepSeek | — | 2,000 | ≤500/model/day | ✅ | 🟡 Medium |
|
||||
| **IBM watsonx** | Provider | Granite models | 2/sec | — | 300K/month | ❌ | 🟡 Medium |
|
||||
**Total: 48 API-key providers with `hasFree: true`.**
|
||||
|
||||
> All entries above are copied verbatim from the `freeNote` field in the provider catalog so they stay in sync with the code.
|
||||
|
||||
---
|
||||
|
||||
## Provider APIs (First-Party)
|
||||
## OAuth-based free tiers
|
||||
|
||||
APIs from the companies that train or fine-tune the models.
|
||||
### Always-free OAuth providers (in `FREE_PROVIDERS`)
|
||||
|
||||
### Google Gemini 🇺🇸
|
||||
These providers are designed around a vendor OAuth flow and ship a free tier by default:
|
||||
|
||||
🔗 [Get API Key](https://aistudio.google.com/app/apikey) · Base URL: `https://generativelanguage.googleapis.com/v1beta`
|
||||
| Provider | ID | Notes |
|
||||
| ---------- | ------------ | ----------------------------------------------------------------------------------------------------------- |
|
||||
| Qoder AI | `qoder` | OAuth or Personal Access Token. Free tier on signup. |
|
||||
| Gemini CLI | `gemini-cli` | Uses Gemini CLI OAuth / Cloud Code credentials. Pro models require an eligible Google account or paid plan. |
|
||||
| Kiro AI | `kiro` | AWS Builder ID (Kiro Free tier). |
|
||||
| Amazon Q | `amazon-q` | Same AWS Builder ID / refresh-token flow as Kiro, but kept as separate connections. |
|
||||
|
||||
> ⚠️ Free tier NOT available in EU/UK/Switzerland. Prompts may be used by Google to improve products.
|
||||
### OAuth providers with vendor-controlled free tiers (in `OAUTH_PROVIDERS`)
|
||||
|
||||
| Model | Context | Max Output | Modality | RPM | RPD |
|
||||
| --------------------------------- | ------- | ---------- | ---------------------- | --- | ------ |
|
||||
| Gemini 2.5 Flash / Gemini 3 Flash | 1M | 65K | Text+Image+Audio+Video | 5 | 20 |
|
||||
| Gemini 2.5 Flash-Lite | 1M | 65K | Text+Image+Audio+Video | 10 | 20 |
|
||||
| Gemini 3.1 Flash-Lite | 1M | 65K | Text+Image+Audio+Video | 15 | 1,500 |
|
||||
| Gemma 4 26B/31B | — | — | Text | 15 | 1,500 |
|
||||
| Gemma 3 (1B/4B/12B/27B) | — | — | Text | 30 | 14,400 |
|
||||
The free-tier surface here depends entirely on each vendor's account plan, not on OmniRoute:
|
||||
|
||||
### Mistral AI 🇫🇷
|
||||
|
||||
🔗 [Get API Key](https://console.mistral.ai/api-keys) · Base URL: `https://api.mistral.ai/v1`
|
||||
|
||||
Free "Experiment" plan, no credit card. ~1B tokens/month. Requires phone verification.
|
||||
|
||||
| Model | Context | Max Output | Modality | Rate Limit |
|
||||
| ------------------ | ------- | ---------- | --------------- | --------------- |
|
||||
| Mistral Small 4 | 256K | 256K | Text+Image+Code | 1 RPS, 500K TPM |
|
||||
| Mistral Medium 3 | 128K | 128K | Text | 1 RPS, 500K TPM |
|
||||
| Mistral Large 3 | 256K | 256K | Text | 1 RPS, 500K TPM |
|
||||
| Mistral Nemo (12B) | 128K | 128K | Text | 1 RPS, 500K TPM |
|
||||
| Codestral | 256K | 256K | Code | 30 RPM, 2K RPD |
|
||||
| Pixtral Large | 128K | 128K | Text+Image | 1 RPS, 500K TPM |
|
||||
|
||||
### Cohere 🇨🇦
|
||||
|
||||
🔗 [Get API Key](https://dashboard.cohere.com/api-keys) · Base URL: `https://api.cohere.com/v2`
|
||||
|
||||
Free "Trial" key. 1,000 API calls/month. Non-commercial use only. 20 RPM.
|
||||
|
||||
| Model | Context | Max Output | Modality |
|
||||
| ------------------- | ------- | ---------- | ----------------------- |
|
||||
| Command A (111B) | 256K | 4K | Text |
|
||||
| Command A Reasoning | 256K | 4K | Text (reasoning) |
|
||||
| Command A Vision | 256K | 4K | Text+Image |
|
||||
| Command A Translate | 256K | 4K | Translation |
|
||||
| Command R+ | 128K | 4K | Text |
|
||||
| Command R | 128K | 4K | Text |
|
||||
| Command R7B | 128K | 4K | Text |
|
||||
| Embed 4 | — | — | Embeddings (Text+Image) |
|
||||
| Rerank 3.5 | — | — | Reranking |
|
||||
|
||||
### Z.AI (Zhipu AI) 🇨🇳
|
||||
|
||||
🔗 [Get API Key](https://open.bigmodel.cn/usercenter/apikeys) · Base URL: `https://open.bigmodel.cn/api/paas/v4`
|
||||
|
||||
Permanent free models, no credit card. No published rate limits.
|
||||
|
||||
| Model | Context | Max Output | Modality |
|
||||
| --------------- | ------- | ---------- | ---------- |
|
||||
| GLM-4.7-Flash | 200K | 128K | Text |
|
||||
| GLM-4.5-Flash | 128K | ~8K | Text |
|
||||
| GLM-4.6V-Flash | 128K | ~4K | Text+Image |
|
||||
| GLM-5 / GLM-5.1 | — | — | Text |
|
||||
|
||||
### IBM watsonx 🇺🇸
|
||||
|
||||
🔗 [Pricing](https://www.ibm.com/products/watsonx-ai/pricing)
|
||||
|
||||
Free tier: 2 RPS, 300K tokens/month. Granite foundation models.
|
||||
| Provider | ID | Auth hint |
|
||||
| -------------------- | ------------- | --------------------------------------------------------------------------------------- |
|
||||
| Claude Code | `claude` | OAuth via `platform.claude.com`. Free quota depends on your Anthropic account. |
|
||||
| Antigravity | `antigravity` | Google OAuth (Antigravity). |
|
||||
| OpenAI Codex | `codex` | OAuth via OpenAI (Codex CLI). Subject to ChatGPT plan free credits. |
|
||||
| GitHub Copilot | `github` | OAuth via GitHub. Free for verified students; free trial otherwise. |
|
||||
| GitLab Duo | `gitlab-duo` | OAuth (`ai_features + read_user`). Requires GitLab Duo entitlement. |
|
||||
| Cursor IDE | `cursor` | Cursor OAuth. Free tier limits depend on Cursor plan. |
|
||||
| Kimi Coding | `kimi-coding` | Moonshot OAuth. Free quota on Kimi Coding accounts. |
|
||||
| Kilo Code | `kilocode` | Kilo OAuth — free auto-router available. |
|
||||
| Cline | `cline` | Cline OAuth. |
|
||||
| Windsurf (Devin CLI) | `windsurf` | Sign in at `windsurf.com`, paste your token. Free tier limits set by Windsurf. |
|
||||
| Devin CLI (Official) | `devin-cli` | Uses the official Devin CLI binary or `WINDSURF_API_KEY`. Subject to Devin's free tier. |
|
||||
|
||||
---
|
||||
|
||||
## Inference Providers (Third-Party)
|
||||
## Deprecated / discontinued
|
||||
|
||||
### Groq 🇺🇸
|
||||
### Qwen Code (`qwen`)
|
||||
|
||||
🔗 [Get API Key](https://console.groq.com/keys) · Base URL: `https://api.groq.com/openai/v1`
|
||||
Marked `deprecated: true` in `FREE_PROVIDERS`. Discontinued **2026-04-15**.
|
||||
|
||||
Ultra-fast LPU inference (~300–500 tok/s). No credit card required.
|
||||
> Qwen OAuth free tier was discontinued on 2026-04-15. Use `alicode`, `alicode-intl`, or `openrouter` providers with an API key instead.
|
||||
|
||||
| Model | RPM | RPD | TPM | Modality |
|
||||
| ---------------------------------- | --- | ------ | --- | ---------------- |
|
||||
| llama-3.3-70b-versatile | 30 | 1,000 | 12K | Text |
|
||||
| llama-3.1-8b-instant | 30 | 14,400 | 6K | Text |
|
||||
| llama-4-scout-17b-16e-instruct | 30 | 1,000 | 30K | Text+Vision |
|
||||
| llama-4-maverick-17b-128e-instruct | 30 | 1,000 | 6K | Text+Vision |
|
||||
| qwen3-32b | 60 | 1,000 | 6K | Text |
|
||||
| kimi-k2-instruct | 60 | 1,000 | 10K | Text |
|
||||
| gpt-oss-120b / gpt-oss-20b | 30 | 1,000 | 8K | Text |
|
||||
| deepseek-r1-distill-70b | 30 | 14,400 | — | Text (reasoning) |
|
||||
| whisper-large-v3 / v3-turbo | 20 | 2,000 | — | Audio→Text |
|
||||
Connections of type `qwen` will keep working until their tokens expire, but no new OAuth sign-ins are accepted upstream. Migrate to:
|
||||
|
||||
### Cerebras 🇺🇸
|
||||
|
||||
🔗 [Get API Key](https://cloud.cerebras.ai/) · Base URL: `https://api.cerebras.ai/v1`
|
||||
|
||||
Wafer-scale chip inference (~2,600 tok/s). 1M tokens/day cap.
|
||||
|
||||
| Model | RPM | RPH | RPD | TPM | TPD |
|
||||
| ------------------------------ | --- | --- | ------ | --- | --- |
|
||||
| gpt-oss-120b | 30 | 900 | 14,400 | 64K | 1M |
|
||||
| llama3.1-8b | 30 | 900 | 14,400 | 60K | 1M |
|
||||
| qwen-3-235b-a22b-instruct-2507 | 30 | 900 | 14,400 | 60K | 1M |
|
||||
| zai-glm-4.7 | 10 | 100 | 100 | 60K | 1M |
|
||||
|
||||
### NVIDIA NIM 🇺🇸
|
||||
|
||||
🔗 [Explore Models](https://build.nvidia.com/explore/discover) · Base URL: `https://integrate.api.nvidia.com/v1`
|
||||
|
||||
Free with NVIDIA Developer Program. **129 models**, 40 RPM. Phone verification required.
|
||||
|
||||
**Notable models:** DeepSeek-R1, DeepSeek-V3.2, Nemotron Ultra 253B, Llama 3.1 405B, Qwen3 Coder 480B, Mistral Large 3, Kimi K2, GLM-5.1, MiniMax M2.7, Gemma 4 31B, + 100 more.
|
||||
|
||||
### OpenRouter 🇺🇸
|
||||
|
||||
🔗 [Get API Key](https://openrouter.ai/keys) · Base URL: `https://openrouter.ai/api/v1`
|
||||
|
||||
35+ free models (suffix `:free`). 20 RPM.
|
||||
|
||||
| Credits Purchased | RPD |
|
||||
| ----------------- | ----- |
|
||||
| < $10 | 50 |
|
||||
| ≥ $10 (one-time) | 1,000 |
|
||||
|
||||
**Notable free models:** DeepSeek R1, DeepSeek V3, Qwen3 Coder 480B, Llama 4 Scout/Maverick, GPT-OSS 120B, Nemotron 3 Super 120B, MiniMax M2.5, Gemma 4 31B, Devstral, + 23 more.
|
||||
|
||||
### GitHub Models 🇺🇸
|
||||
|
||||
🔗 [Marketplace](https://github.com/marketplace/models) · Base URL: `https://models.inference.ai.azure.com`
|
||||
|
||||
Free for all GitHub users. 45+ models including frontier models.
|
||||
|
||||
| Tier | RPM | RPD | Tokens/Request |
|
||||
| ----------------------- | --- | --- | -------------- |
|
||||
| Low tier models | 15 | 150 | 8K in / 4K out |
|
||||
| High tier models | 10 | 50 | 8K in / 4K out |
|
||||
| DeepSeek-R1 / MAI-DS-R1 | 1 | 8 | 4K in / 4K out |
|
||||
| Grok-3 | 1 | 15 | 4K in / 4K out |
|
||||
|
||||
**Notable models:** GPT-4.1, GPT-4o, GPT-5, GPT-5-mini, o3-mini, o4-mini, DeepSeek-R1, Llama 4 Scout/Maverick, Codestral, Mistral Medium 3, Phi-4, Grok-3.
|
||||
|
||||
### Cloudflare Workers AI 🇺🇸
|
||||
|
||||
🔗 [Get Token](https://dash.cloudflare.com/profile/api-tokens) · 10,000 Neurons/day free. 50+ models.
|
||||
|
||||
**Notable models:** Llama 3.3 70B, Llama 4 Scout, Qwen3 30B-A3B, QwQ 32B, DeepSeek R1 Distill, Gemma 4 26B, GLM 4.7 Flash, Nemotron 3 120B, Kimi K2.5/K2.6, Mistral Small 3.1, GPT-OSS 120B/20B, + 40 more.
|
||||
|
||||
### Hugging Face 🇺🇸
|
||||
|
||||
🔗 [Get Token](https://huggingface.co/settings/tokens) · Base URL: `https://api-inference.huggingface.co/v1`
|
||||
|
||||
$0.10/month free credits (auto-replenished). Thousands of models. Serverless limited to <10GB models.
|
||||
|
||||
### Ollama Cloud 🇺🇸
|
||||
|
||||
🔗 [Get Key](https://ollama.com/settings/keys) · Base URL: `https://api.ollama.com`
|
||||
|
||||
400+ models. Session/weekly limits (unpublished). NOT OpenAI SDK-compatible.
|
||||
|
||||
**Notable models:** GPT-OSS 120B, DeepSeek V3.2/V4, Kimi K2/K2.5/K2.6, GLM-5/5.1, Qwen3 Coder 480B, Gemini 3 Flash, MiniMax M2.7, Cogito 2.1 671B, Nemotron 3 Super 120B.
|
||||
|
||||
### Pollinations AI 🇩🇪
|
||||
|
||||
🔗 [Get Key](https://enter.pollinations.ai) · Base URL: `https://gen.pollinations.ai/v1`
|
||||
|
||||
No sign-up required for basic use. Unique: **text + image + video + audio** all free.
|
||||
|
||||
**Text models:** openai, openai-large, openai-reasoning, gemini, mistral, llama.
|
||||
**Image models:** flux, gpt-image, seedream, kontext.
|
||||
**Video:** wan-fast. **Audio:** tts-1, 30+ ElevenLabs voices.
|
||||
|
||||
### SiliconFlow 🇨🇳
|
||||
|
||||
🔗 [Get Key](https://cloud.siliconflow.cn/account/ak) · Base URL: `https://api.siliconflow.cn/v1`
|
||||
|
||||
14 CNY signup credits. Permanently free models: 1,000 RPM, 50K TPM.
|
||||
|
||||
| Model | Context | Modality |
|
||||
| --------------------------- | ------- | ---------------- |
|
||||
| Qwen/Qwen3-8B | 131K | Text |
|
||||
| DeepSeek-R1-0528-Qwen3-8B | ~33K | Text (reasoning) |
|
||||
| DeepSeek-R1-Distill-Qwen-7B | 131K | Text (reasoning) |
|
||||
| THUDM/glm-4-9b-chat | 32K | Text |
|
||||
| THUDM/GLM-4.1V-9B-Thinking | 66K | Vision+Text |
|
||||
| DeepSeek-OCR | — | Vision (OCR) |
|
||||
|
||||
### Kilo Code 🇺🇸
|
||||
|
||||
🔗 [Get Key](https://kilo.ai) · Base URL: `https://api.kilo.ai/api/gateway`
|
||||
|
||||
Free models, no credit card. ~200 req/hr. Auto-router `kilo-auto/free`.
|
||||
|
||||
### LLM7.io 🇬🇧
|
||||
|
||||
🔗 [Get Token](https://token.llm7.io) · Base URL: `https://api.llm7.io/v1`
|
||||
|
||||
30+ models. 15 RPM (30 RPM with free token). No registration for basic access.
|
||||
|
||||
### Kluster AI 🇺🇸
|
||||
|
||||
🔗 [Get Key](https://platform.kluster.ai/apikeys) · DeepSeek-R1, Llama 4 Maverick, Qwen3-235B + more.
|
||||
|
||||
### OpenCode Zen
|
||||
|
||||
🔗 [Docs](https://opencode.ai/docs/zen/) · Free models (Big Pickle Stealth, MiniMax M2.5 Free, Arcee Large).
|
||||
|
||||
### Vercel AI Gateway
|
||||
|
||||
🔗 [Docs](https://vercel.com/docs/ai-gateway) · $5/month free credits. Routes to various providers.
|
||||
- `alicode` (Alibaba Cloud Bailian — DashScope)
|
||||
- `alicode-intl` (Alibaba Cloud International)
|
||||
- `openrouter` (Qwen models exposed via OpenRouter)
|
||||
|
||||
---
|
||||
|
||||
## China-Based Providers
|
||||
## Command Code
|
||||
|
||||
### ModelScope (魔搭社区) 🇨🇳
|
||||
`command-code` is a separate API-key provider for the Command Code agent (see `commandcode.ai`). It is not flagged with `hasFree: true` in the catalog, so it does not appear in the free table above, but it is included here because it ships in v3.8.0 alongside the free-tier providers:
|
||||
|
||||
🔗 [Get Token](https://modelscope.cn/my/myaccesstoken) · Base URL: `https://api-inference.modelscope.cn/v1`
|
||||
- ID: `command-code`
|
||||
- Endpoint: Command Code `/alpha/generate`
|
||||
- Auth: Bearer API key, configured from the dashboard.
|
||||
|
||||
2,000 req/day total, ≤500/model/day. Requires Alibaba Cloud account + real-name verification.
|
||||
|
||||
**Models:** DeepSeek V4 Pro/Flash, DeepSeek V3.2, GLM-5/5.1, MiniMax M2.5, Qwen3-235B, Qwen3 Coder 480B, Ling-2.6-1T.
|
||||
|
||||
### Tencent Hunyuan (腾讯混元)
|
||||
|
||||
Hunyuan-Lite: free. Other models: 100M tokens free (1-year expiry).
|
||||
|
||||
### Volcengine (火山引擎)
|
||||
|
||||
500 resource points/day. Tongyi Qwen free (100 calls/day). Doubao models with tiered pricing.
|
||||
|
||||
### ChatAnywhere
|
||||
|
||||
🔗 Base URL: `https://api.chatanywhere.tech` · GPT-5.4-mini, DeepSeek-V4, and more.
|
||||
|
||||
### InternAI (书生)
|
||||
|
||||
🔗 Base URL: `https://chat.intern-ai.org.cn/api/v1` · 10 RPM. Keys valid 6 months.
|
||||
|
||||
**Models:** intern-latest, intern-s1-pro, internvl3.5-241b-a28b.
|
||||
|
||||
### Bigmodel (智谱)
|
||||
|
||||
🔗 Base URL: `https://open.bigmodel.cn/api/paas/v4/` · 30 concurrent requests.
|
||||
|
||||
**Models:** GLM-4-Flash, GLM-4V-Flash, GLM-4.1V-Thinking-Flash, GLM-4.6V-Flash, GLM-4.7-Flash.
|
||||
Check Command Code's website for the current free-tier policy.
|
||||
|
||||
---
|
||||
|
||||
## Trial Credit Providers
|
||||
## Environment variables
|
||||
|
||||
These offer one-time or time-limited credits (not permanent free tiers):
|
||||
|
||||
| Provider | Credits | Expiry | Notable Models |
|
||||
| ---------------------------------------------------------- | ---------------- | -------- | ----------------------------- |
|
||||
| [Baseten](https://app.baseten.co/) | $30 | — | Any model (pay by compute) |
|
||||
| [NLP Cloud](https://nlpcloud.com) | $15 | — | Various open models |
|
||||
| [AI21](https://studio.ai21.com/) | $10 | 3 months | Jamba family |
|
||||
| [Upstage](https://console.upstage.ai/) | $10 | 3 months | Solar Pro/Mini |
|
||||
| [Modal](https://modal.com) | $5/mo | Monthly | Any model (compute time) |
|
||||
| [SambaNova](https://cloud.sambanova.ai/) | $5 | 3 months | Llama 3.3, Qwen3, DeepSeek R1 |
|
||||
| [Scaleway](https://console.scaleway.com/generative-api) | 1M tokens | One-time | Llama 3.3, Gemma 3, GPT-OSS |
|
||||
| [Alibaba Cloud](https://bailian.console.alibabacloud.com/) | 1M tokens/model | — | Qwen family |
|
||||
| [Fireworks](https://fireworks.ai/) | $1 | — | Various open models |
|
||||
| [Nebius](https://tokenfactory.nebius.com/) | $1 | — | Various open models |
|
||||
| [Inference.net](https://inference.net) | $1 (+$25 survey) | — | Various open models |
|
||||
| [Hyperbolic](https://app.hyperbolic.ai/) | $1 | — | DeepSeek V3, Llama 3.3 |
|
||||
| [Novita](https://novita.ai/) | $0.50 | 1 year | Various open models |
|
||||
|
||||
---
|
||||
|
||||
## Using with OmniRoute
|
||||
|
||||
OmniRoute supports **all providers listed above** as connections. Here's how to maximize free usage:
|
||||
|
||||
### 1. Add Multiple Free Providers
|
||||
|
||||
```
|
||||
Dashboard → Providers → Add Connection
|
||||
```
|
||||
|
||||
Add API keys for Groq, Cerebras, Mistral, Google Gemini, OpenRouter, GitHub Models, etc.
|
||||
|
||||
### 2. Create a Free-Tier Combo
|
||||
|
||||
```
|
||||
Dashboard → Combos → Create Combo → Add all free providers as targets
|
||||
```
|
||||
|
||||
Use the **"priority"** or **"round-robin"** strategy to distribute load across free tiers.
|
||||
|
||||
### 3. Recommended Free Combo Strategy
|
||||
|
||||
| Priority | Provider | Why |
|
||||
| -------- | ----------------- | --------------------------------------------- |
|
||||
| 1 | **Groq** | Fastest inference, 14,400 RPD on small models |
|
||||
| 2 | **Cerebras** | 1M TPD, fast wafer-scale chips |
|
||||
| 3 | **Mistral** | 1B tokens/month, large model selection |
|
||||
| 4 | **Google Gemini** | 1M context, multimodal |
|
||||
| 5 | **NVIDIA NIM** | 129 models, 40 RPM |
|
||||
| 6 | **OpenRouter** | 35+ free models as final fallback |
|
||||
|
||||
### 4. Environment Variables
|
||||
OmniRoute v3.8.0 does **not** read provider API keys from environment variables (with one exception below). Keys are stored in the encrypted SQLite database and configured from the dashboard. The env vars listed here are the only ones that affect free-tier behavior:
|
||||
|
||||
```bash
|
||||
# These providers work out of the box with OmniRoute:
|
||||
GROQ_API_KEY=your-key
|
||||
CEREBRAS_API_KEY=your-key
|
||||
MISTRAL_API_KEY=your-key
|
||||
GOOGLE_AI_API_KEY=your-key
|
||||
NVIDIA_API_KEY=your-key
|
||||
OPENROUTER_API_KEY=your-key
|
||||
GITHUB_TOKEN=your-token
|
||||
CLOUDFLARE_API_TOKEN=your-token
|
||||
COHERE_API_KEY=your-key
|
||||
SILICONFLOW_API_KEY=your-key
|
||||
# Windsurf / Devin CLI — Firebase Web API key used by the Secure Token
|
||||
# Service to refresh the Windsurf app. The default value ships in
|
||||
# .env.example (it is a public Firebase Web API key extracted from the
|
||||
# Devin CLI binary, not a real secret); override only if you mirror your
|
||||
# own Windsurf token-refresh service.
|
||||
WINDSURF_FIREBASE_API_KEY=<see .env.example>
|
||||
|
||||
# Optional fallback for the devin-cli executor when no connection key is set.
|
||||
WINDSURF_API_KEY=
|
||||
|
||||
# Optional path to the official Devin CLI binary.
|
||||
CLI_DEVIN_BIN=/usr/local/bin/devin
|
||||
|
||||
# OAuth client overrides (rarely needed — defaults shipped in code)
|
||||
CODEX_OAUTH_CLIENT_ID=
|
||||
GEMINI_OAUTH_CLIENT_ID=
|
||||
GEMINI_OAUTH_CLIENT_SECRET=
|
||||
GEMINI_CLI_OAUTH_CLIENT_ID=
|
||||
GEMINI_CLI_OAUTH_CLIENT_SECRET=
|
||||
QWEN_OAUTH_CLIENT_ID=
|
||||
KIMI_CODING_OAUTH_CLIENT_ID=
|
||||
GITHUB_OAUTH_CLIENT_ID=
|
||||
GITLAB_DUO_OAUTH_CLIENT_ID=
|
||||
GITLAB_DUO_OAUTH_CLIENT_SECRET=
|
||||
QODER_OAUTH_CLIENT_SECRET=
|
||||
QODER_PERSONAL_ACCESS_TOKEN=
|
||||
|
||||
# CLI sidecar binaries
|
||||
CLI_CODEX_BIN=codex
|
||||
CLI_CURSOR_BIN=agent
|
||||
CLI_CLINE_BIN=cline
|
||||
CLI_QODER_BIN=qoder
|
||||
CLI_QWEN_BIN=qwen
|
||||
```
|
||||
|
||||
### 5. Estimated Free Capacity
|
||||
For all other providers (Groq, Cerebras, Mistral, Gemini, Cohere, NVIDIA, OpenRouter, Together, Fireworks, Cloudflare AI, SambaNova, HuggingFace, SiliconFlow, Hyperbolic, Morph, LLM7, Lepton, Kluster, UncloseAI, BazaarLink, Completions, Enally, FreeTheAi, AgentRouter, Command Code, etc.), add the key from `/dashboard/providers/new`.
|
||||
|
||||
With all top-6 providers combined in a combo:
|
||||
---
|
||||
|
||||
| Metric | Combined Free Capacity |
|
||||
| -------------------- | ---------------------- |
|
||||
| **Requests/Day** | ~31,000+ RPD |
|
||||
| **Tokens/Month** | ~32B+ tokens |
|
||||
| **Models Available** | 200+ unique models |
|
||||
| **Cost** | $0.00 |
|
||||
## How to use
|
||||
|
||||
1. Open `/dashboard/providers/new` and pick the provider you want.
|
||||
2. Paste the API key (or complete the OAuth flow). For OAuth providers, follow the dashboard wizard.
|
||||
3. The provider appears in your routing pool automatically and is eligible for combos and auto-routing.
|
||||
4. Track usage at `/dashboard/usage` to see how close you are to free-tier limits.
|
||||
|
||||
### Suggested combos
|
||||
|
||||
| Goal | Strategy | Notes |
|
||||
| ---------------------------- | -------------------- | --------------------------------------------------------------------- |
|
||||
| Cheapest possible chat | `auto/cheap` | Prefers free / lowest-cost providers; falls back automatically. |
|
||||
| Local-only routing | `auto/offline` | Routes only to local providers (Ollama, LM Studio, vLLM, …). |
|
||||
| Redundancy across free tiers | combo `priority` | List Groq → Cerebras → Mistral → Gemini → NVIDIA → OpenRouter. |
|
||||
| High RPM throughput | combo `round-robin` | Spreads requests across all configured free providers. |
|
||||
| Best success rate | combo `lkgp` / `p2c` | Picks last-known-good provider or "power of two choices" rebalancing. |
|
||||
|
||||
### Tips
|
||||
|
||||
- Combine multiple free providers in a combo (`/dashboard/combos`) to maximize daily quota and route around outages.
|
||||
- Use `omniroute doctor` to verify all configured free providers are reachable.
|
||||
- Check provider health in `/dashboard/monitoring/health` — a provider with an open circuit breaker is skipped automatically.
|
||||
- Free-tier limits change frequently; the `freeNote` strings reflect the limits as known at v3.8.0 ship date. Verify with each provider's official docs before relying on a specific number.
|
||||
|
||||
---
|
||||
|
||||
## Glossary
|
||||
|
||||
| Term | Meaning |
|
||||
| ----------- | ------------------------------------------- |
|
||||
| **RPM** | Requests per minute |
|
||||
| **RPD** | Requests per day |
|
||||
| **RPH** | Requests per hour |
|
||||
| **RPS** | Requests per second |
|
||||
| **TPM** | Tokens per minute |
|
||||
| **TPD** | Tokens per day |
|
||||
| **Neurons** | Cloudflare's compute unit (~1 output token) |
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
This document consolidates data from 6 community repositories:
|
||||
|
||||
| Repository | Focus |
|
||||
| -------------------------------------------------------------------------- | ------------------------------------------------ |
|
||||
| [awesome-free-llm-apis](https://github.com/mnfst/awesome-free-llm-apis) | Curated list with detailed model tables |
|
||||
| [awesome-free-llm-apis2](https://github.com/) | Extended list with speed tiers and code snippets |
|
||||
| [free-llm-api-resources](https://github.com/) | Auto-generated model lists with trial credits |
|
||||
| [Free-LLM-Collection](https://github.com/for-the-zero/Free-LLM-Collection) | Chinese + global providers with rate limits |
|
||||
| [FREE-LLM-API-Provider](https://github.com/CYBIRD-D/FREE-LLM-API-Provider) | Deep provider analysis with CN platforms |
|
||||
| [gpt4free](https://github.com/xtekky/gpt4free) | Config-based routing with quota awareness |
|
||||
|
||||
> ⚠️ **Disclaimer:** Rate limits change frequently. Always verify with the provider's official documentation before relying on specific limits. Trial credits and time-limited promotions are separated from permanent free tiers.
|
||||
| Term | Meaning |
|
||||
| ---------- | -------------------------------------------------------- |
|
||||
| **RPM** | Requests per minute |
|
||||
| **RPD** | Requests per day |
|
||||
| **RPH** | Requests per hour |
|
||||
| **RPS** | Requests per second |
|
||||
| **TPM** | Tokens per minute |
|
||||
| **TPD** | Tokens per day |
|
||||
| **Neuron** | Cloudflare's compute unit (~1 output token) |
|
||||
| **LKGP** | Last-known-good provider — auto-combo strategy |
|
||||
| **P2C** | Power-of-two choices — auto-combo load balancer strategy |
|
||||
|
||||
263
docs/GUARDRAILS.md
Normal file
263
docs/GUARDRAILS.md
Normal file
@@ -0,0 +1,263 @@
|
||||
# Guardrails
|
||||
|
||||
> **Source of truth:** `src/lib/guardrails/`
|
||||
> **Last updated:** 2026-05-13 — v3.8.0
|
||||
|
||||
Guardrails enforce safety, policy, and content transformations at the boundary
|
||||
between OmniRoute and upstream providers. Each guardrail can inspect (and
|
||||
optionally reject, transform, or annotate) request payloads (`preCall`) and
|
||||
upstream responses (`postCall`).
|
||||
|
||||
The system is **fail-open**: if a guardrail throws while executing, the registry
|
||||
records the error and continues with the next guardrail rather than failing the
|
||||
request. Blocking is an explicit decision (`block: true`), never an accident.
|
||||
|
||||
## Built-in Guardrails
|
||||
|
||||
The registry auto-loads three guardrails in priority order on import
|
||||
(see `registry.ts` → `registerDefaultGuardrails()`):
|
||||
|
||||
| Priority | Name | Stage(s) | File |
|
||||
| -------- | ------------------ | -------------- | -------------------- |
|
||||
| `5` | `vision-bridge` | `preCall` | `visionBridge.ts` |
|
||||
| `10` | `pii-masker` | `pre` + `post` | `piiMasker.ts` |
|
||||
| `20` | `prompt-injection` | `preCall` | `promptInjection.ts` |
|
||||
|
||||
Lower priority numbers run **first**.
|
||||
|
||||
### Vision Bridge (`visionBridge.ts`)
|
||||
|
||||
Intercepts image-bearing requests aimed at **non-vision models** and replaces
|
||||
the image parts with text descriptions produced by a configurable vision model
|
||||
before the upstream call. This lets text-only providers transparently handle
|
||||
multimodal payloads.
|
||||
|
||||
Flow:
|
||||
|
||||
1. Skip if the target model already supports vision (unless it appears in the
|
||||
forced-bridge list `isVisionBridgeForcedModel`).
|
||||
2. Extract image parts via `extractImageParts(messages)`. Skip if none.
|
||||
3. Load runtime config from `getSettings()` (`visionBridgeEnabled`,
|
||||
`visionBridgeModel`, `visionBridgePrompt`, `visionBridgeTimeout`,
|
||||
`visionBridgeMaxImages`).
|
||||
4. Cap images at `maxImages`, call the vision model **in parallel**
|
||||
(`Promise.allSettled`), and inject `[Image N]: <description>` text parts
|
||||
in their place — failed images become `[Image N]: (unavailable)`.
|
||||
5. Return `modifiedPayload` + meta (`imagesProcessed`, `processingTimeMs`,
|
||||
`visionModel`).
|
||||
|
||||
Defaults live in `src/shared/constants/visionBridgeDefaults.ts`. The guardrail
|
||||
exposes a `deps` constructor option so tests can inject fake `getSettings` and
|
||||
`callVisionModel` implementations.
|
||||
|
||||
### PII Masker (`piiMasker.ts`)
|
||||
|
||||
Runs on **both** stages.
|
||||
|
||||
- **`preCall`** clones the payload, walks `system`, `messages`, and `input`
|
||||
arrays, and applies `processPII()` (from `@/shared/utils/inputSanitizer`) to
|
||||
string `content`/`text` fields. When `PII_REDACTION_ENABLED=true` **and**
|
||||
`INPUT_SANITIZER_MODE=redact`, detected PII is stripped/redacted in the
|
||||
outbound payload. Otherwise the call records detection counts without
|
||||
rewriting content.
|
||||
- **`postCall`** deep-clones the response, runs `sanitizePIIResponse()` plus
|
||||
the Responses-API-shape masker (`maskResponsesOutput` — covers
|
||||
`output_text` and `output[].content[].text`). If any redaction occurs, the
|
||||
modified response replaces the original.
|
||||
|
||||
The guardrail never blocks; it only annotates (`meta.detections`,
|
||||
`meta.redacted`) or rewrites.
|
||||
|
||||
### Prompt Injection (`promptInjection.ts`)
|
||||
|
||||
Detects adversarial structures in user-supplied content and enforces the
|
||||
configured policy. Behavior is driven by environment variables and constructor
|
||||
options:
|
||||
|
||||
| Setting | Env var | Default | Effect |
|
||||
| --------------- | ----------------------------------------------- | ------- | --------------------------------------- |
|
||||
| Enabled | `INPUT_SANITIZER_ENABLED` | `true` | When `false`, guardrail short-circuits. |
|
||||
| Mode | `INJECTION_GUARD_MODE` / `INPUT_SANITIZER_MODE` | `warn` | `block`, `warn`, or `log`. |
|
||||
| Block threshold | `blockThreshold` option | `high` | Minimum severity required to block. |
|
||||
|
||||
Detection sources:
|
||||
|
||||
1. `sanitizeRequest()` from `@/shared/utils/inputSanitizer` (shared detector
|
||||
set used elsewhere in the pipeline).
|
||||
2. Built-in `DEFAULT_GUARD_PATTERNS` (currently `system_override_inline` and
|
||||
`markdown_system_block`, both `high` severity).
|
||||
3. Optional `customPatterns` passed via constructor options (strings, regex,
|
||||
or `{ name, pattern, severity }` records).
|
||||
|
||||
When `mode === "block"` **and** at least one detection meets the severity
|
||||
threshold, `preCall` returns `{ block: true, message: "Request rejected:
|
||||
suspicious content detected" }`. In `warn`/`log` modes the guardrail logs but
|
||||
allows the call. The shared helper `evaluatePromptInjection()` is also exported
|
||||
for callers that need to evaluate prompts without going through the registry.
|
||||
|
||||
## Base Contract (`base.ts`)
|
||||
|
||||
```typescript
|
||||
class BaseGuardrail {
|
||||
enabled: boolean;
|
||||
name: string;
|
||||
priority: number;
|
||||
|
||||
constructor(name: string, options?: { enabled?: boolean; priority?: number });
|
||||
|
||||
async preCall(payload: unknown, context: GuardrailContext): Promise<GuardrailResult | void>;
|
||||
|
||||
async postCall(response: unknown, context: GuardrailContext): Promise<GuardrailResult | void>;
|
||||
}
|
||||
|
||||
interface GuardrailResult<TValue = unknown> {
|
||||
block?: boolean; // true short-circuits the chain
|
||||
message?: string; // surfaced when blocking
|
||||
meta?: Record<string, unknown> | null;
|
||||
modifiedPayload?: TValue; // returned by preCall to rewrite the request
|
||||
modifiedResponse?: TValue; // returned by postCall to rewrite the response
|
||||
}
|
||||
|
||||
interface GuardrailContext {
|
||||
apiKeyInfo?: Record<string, unknown> | null;
|
||||
disabledGuardrails?: string[] | null;
|
||||
endpoint?: string | null;
|
||||
headers?: Headers | Record<string, unknown> | null;
|
||||
log?: GuardrailLog | Console | null;
|
||||
method?: string | null;
|
||||
model?: string | null;
|
||||
provider?: string | null;
|
||||
sourceFormat?: string | null;
|
||||
stream?: boolean;
|
||||
targetFormat?: string | null;
|
||||
}
|
||||
```
|
||||
|
||||
A guardrail signals "no change" by returning either `void`, `{}`, or
|
||||
`{ block: false }`. Returning a `modifiedPayload`/`modifiedResponse` replaces
|
||||
the value flowing through the chain for downstream guardrails.
|
||||
|
||||
## Registry (`registry.ts`)
|
||||
|
||||
The singleton `guardrailRegistry` exposes:
|
||||
|
||||
- `register(guardrail)` — adds (or replaces by normalized name) a guardrail and
|
||||
re-sorts by ascending `priority`.
|
||||
- `clear()` / `list()` — administrative helpers.
|
||||
- `runPreCallHooks(payload, context)` — iterates active guardrails, threads the
|
||||
payload through `modifiedPayload`, and stops on the first `block: true`.
|
||||
- `runPostCallHooks(response, context)` — same flow on the response side.
|
||||
- `resetGuardrailsForTests({ registerDefaults })` — clears state and optionally
|
||||
re-registers the defaults for clean test isolation.
|
||||
|
||||
Both runners return `{ blocked, payload|response, results, guardrail?, message? }`
|
||||
where `results` is an array of `GuardrailExecutionResult` records that include
|
||||
per-guardrail `blocked`, `skipped`, `modified`, `error`, and `meta` fields,
|
||||
useful for tracing.
|
||||
|
||||
### Disabling Guardrails Per-Request
|
||||
|
||||
`resolveDisabledGuardrails({ apiKeyInfo, body, headers })` aggregates a
|
||||
de-duplicated list of guardrail names that should be skipped for the current
|
||||
request. Sources (all optional, all merged):
|
||||
|
||||
- `apiKeyInfo.disabledGuardrails`
|
||||
- Request body `disabledGuardrails` (top-level)
|
||||
- Request body `metadata.disabledGuardrails`
|
||||
- Header `x-omniroute-disabled-guardrails` (or legacy
|
||||
`x-disabled-guardrails`)
|
||||
|
||||
Values may be arrays of strings or a comma-separated string; names are
|
||||
normalized to lowercase kebab-case (`pii_masker` → `pii-masker`). The result
|
||||
is passed through `context.disabledGuardrails` to the registry, which skips
|
||||
matching guardrails (`skipped: true` in `results`).
|
||||
|
||||
## Execution Order
|
||||
|
||||
For each request flowing through `src/sse/handlers/chat.ts` and
|
||||
`open-sse/handlers/chatCore.ts`:
|
||||
|
||||
1. `resolveDisabledGuardrails(...)` builds the skip list from API key, body,
|
||||
and headers.
|
||||
2. `guardrailRegistry.runPreCallHooks(body, ctx)` runs guardrails in ascending
|
||||
priority order:
|
||||
- Disabled guardrails are recorded as `skipped`.
|
||||
- Each guardrail's `preCall` may rewrite the payload via `modifiedPayload`.
|
||||
- The first `block: true` short-circuits the chain and the handler returns
|
||||
a guardrail rejection response.
|
||||
3. The (potentially rewritten) payload flows into combo routing and upstream
|
||||
dispatch.
|
||||
4. After the response is assembled, `guardrailRegistry.runPostCallHooks(...)`
|
||||
runs the same chain on the response. `block: true` here drops the upstream
|
||||
response.
|
||||
|
||||
Guardrails that throw are recorded with `error: <message>` and logged via
|
||||
`logger.warn`, but the chain continues — fail-open by design.
|
||||
|
||||
## Configuration
|
||||
|
||||
Environment variables read by the built-in guardrails:
|
||||
|
||||
| Variable | Used by | Effect |
|
||||
| ------------------------------------- | -------------------------------- | ----------------------------------------------------- |
|
||||
| `INPUT_SANITIZER_ENABLED` | `prompt-injection` | Set `false` to disable detection entirely. |
|
||||
| `INPUT_SANITIZER_MODE` | `prompt-injection`, `pii-masker` | Shared mode: `warn`, `block`, `log`, or `redact`. |
|
||||
| `INJECTION_GUARD_MODE` | `prompt-injection` | Legacy alias for `INPUT_SANITIZER_MODE`. |
|
||||
| `PII_REDACTION_ENABLED` | `pii-masker` | When `true` + mode `redact`, request PII is stripped. |
|
||||
| `PII_RESPONSE_SANITIZATION` / `_MODE` | `pii-masker` (downstream) | Controls response-side masker behavior. |
|
||||
|
||||
The Vision Bridge reads runtime config from the DB-backed settings store
|
||||
(`getSettings()`), not env vars: `visionBridgeEnabled`, `visionBridgeModel`,
|
||||
`visionBridgePrompt`, `visionBridgeTimeout`, `visionBridgeMaxImages`. Defaults
|
||||
live in `src/shared/constants/visionBridgeDefaults.ts`.
|
||||
|
||||
## Custom Guardrails
|
||||
|
||||
```typescript
|
||||
import { BaseGuardrail, guardrailRegistry } from "@/lib/guardrails";
|
||||
|
||||
class BudgetGuardrail extends BaseGuardrail {
|
||||
constructor() {
|
||||
super("budget", { priority: 50 });
|
||||
}
|
||||
|
||||
async preCall(payload, ctx) {
|
||||
if (ctx.apiKeyInfo?.budgetExceeded) {
|
||||
return { block: true, message: "Daily budget exceeded" };
|
||||
}
|
||||
return { block: false };
|
||||
}
|
||||
}
|
||||
|
||||
guardrailRegistry.register(new BudgetGuardrail());
|
||||
```
|
||||
|
||||
Steps:
|
||||
|
||||
1. Create `src/lib/guardrails/myGuardrail.ts` extending `BaseGuardrail`.
|
||||
2. Implement `preCall` and/or `postCall`.
|
||||
3. Either register at import time (push from `registerDefaultGuardrails`) or
|
||||
call `guardrailRegistry.register(...)` at runtime — the registry replaces
|
||||
any prior guardrail with the same normalized name.
|
||||
4. Add tests under `tests/unit/` (existing examples:
|
||||
`tests/unit/guardrails-registry.test.ts`,
|
||||
`tests/unit/prompt-injection-guard.test.ts`,
|
||||
`tests/unit/guardrails/visionBridge.test.ts`).
|
||||
|
||||
## Testing
|
||||
|
||||
Use `resetGuardrailsForTests()` between tests to start from a known state.
|
||||
Pass `{ registerDefaults: false }` to start with an empty registry and
|
||||
register only the guardrails under test. The Vision Bridge guardrail accepts
|
||||
dependency injection (`deps.getSettings`, `deps.callVisionModel`) so tests can
|
||||
exercise the full flow without DB or network access.
|
||||
|
||||
## See Also
|
||||
|
||||
- `src/lib/guardrails/` — implementation
|
||||
- `src/shared/utils/inputSanitizer.ts` — shared detector that powers
|
||||
prompt-injection and PII masking
|
||||
- `src/shared/constants/visionBridgeDefaults.ts` — Vision Bridge defaults and
|
||||
forced-bridge model list
|
||||
- `docs/RESILIENCE_GUIDE.md` — orthogonal layer (circuit breaker, cooldowns)
|
||||
- `docs/ENVIRONMENT.md` — full env var reference
|
||||
@@ -1,6 +1,8 @@
|
||||
# OmniRoute MCP Server Documentation
|
||||
|
||||
> Model Context Protocol server with 37 tools across routing, cache, compression, memory, skills, and proxy operations
|
||||
> Model Context Protocol server with 37 tools across routing, cache, compression, memory, skills, and proxy operations.
|
||||
>
|
||||
> Source of truth: `open-sse/mcp-server/schemas/tools.ts` (30 tools) + `open-sse/mcp-server/tools/memoryTools.ts` (3 tools) + `open-sse/mcp-server/tools/skillTools.ts` (4 tools). Tool registration and scope wiring lives in `open-sse/mcp-server/server.ts`.
|
||||
|
||||
## Installation
|
||||
|
||||
@@ -17,6 +19,18 @@ Or via the open-sse transport:
|
||||
omniroute --dev # MCP auto-starts on /mcp endpoint
|
||||
```
|
||||
|
||||
## Transports
|
||||
|
||||
The MCP server exposes three transports, all backed by the same `createMcpServer()` factory:
|
||||
|
||||
| Transport | Where | When to use |
|
||||
| :---------------- | :------------------------------------------ | :--------------------------------------------------- |
|
||||
| `stdio` | `open-sse/mcp-server/server.ts` | IDE integrations (Claude Desktop, Cursor, etc.) |
|
||||
| `sse` | `POST/GET /api/mcp/sse` via `httpTransport` | Browser/agent clients that need an event stream |
|
||||
| `streamable-http` | `POST/GET/DELETE /api/mcp/stream` | Multi-session HTTP clients (`mcp-session-id` header) |
|
||||
|
||||
The active HTTP transport (`sse` or `streamable-http`) is selected by the `mcpTransport` setting. Switching transports closes existing sessions on the other transport.
|
||||
|
||||
## IDE Configuration
|
||||
|
||||
See [MCP Client Configuration](SETUP_GUIDE.md#mcp-client-configuration) for Claude Desktop,
|
||||
@@ -24,48 +38,57 @@ Cursor, Cline, and compatible MCP client setup.
|
||||
|
||||
---
|
||||
|
||||
## Essential Tools (8)
|
||||
## Essential Tools (8) — Phase 1
|
||||
|
||||
| Tool | Description |
|
||||
| :------------------------------ | :--------------------------------------- |
|
||||
| `omniroute_get_health` | Gateway health, circuit breakers, uptime |
|
||||
| `omniroute_list_combos` | All configured combos with models |
|
||||
| `omniroute_get_combo_metrics` | Performance metrics for a specific combo |
|
||||
| `omniroute_switch_combo` | Switch active combo by ID/name |
|
||||
| `omniroute_check_quota` | Quota status per provider or all |
|
||||
| `omniroute_route_request` | Send a chat completion through OmniRoute |
|
||||
| `omniroute_cost_report` | Cost analytics for a time period |
|
||||
| `omniroute_list_models_catalog` | Full model catalog with capabilities |
|
||||
| Tool | Scopes | Description |
|
||||
| :------------------------------ | :-------------------- | :------------------------------------------------------------ |
|
||||
| `omniroute_get_health` | `read:health` | Uptime, memory, circuit breakers, rate limits, cache stats |
|
||||
| `omniroute_list_combos` | `read:combos` | All configured combos with strategies (optional metrics) |
|
||||
| `omniroute_get_combo_metrics` | `read:combos` | Performance metrics for a specific combo |
|
||||
| `omniroute_switch_combo` | `write:combos` | Activate or deactivate a combo |
|
||||
| `omniroute_check_quota` | `read:quota` | Quota used/total, percent remaining, reset time, token health |
|
||||
| `omniroute_route_request` | `execute:completions` | Send a chat completion through OmniRoute routing |
|
||||
| `omniroute_cost_report` | `read:usage` | Cost report by period (session/day/week/month) |
|
||||
| `omniroute_list_models_catalog` | `read:models` | Full model catalog with capabilities, status, pricing |
|
||||
|
||||
## Advanced Tools (8)
|
||||
## Phase 1 — Search
|
||||
|
||||
| Tool | Description |
|
||||
| :--------------------------------- | :---------------------------------------------------------- |
|
||||
| `omniroute_simulate_route` | Dry-run routing simulation with fallback tree |
|
||||
| `omniroute_set_budget_guard` | Session budget with degrade/block/alert actions |
|
||||
| `omniroute_set_resilience_profile` | Apply conservative/balanced/aggressive preset |
|
||||
| `omniroute_test_combo` | Live-test all models in a combo via a real upstream request |
|
||||
| `omniroute_get_provider_metrics` | Detailed metrics for one provider |
|
||||
| `omniroute_best_combo_for_task` | Task-fitness recommendation with alternatives |
|
||||
| `omniroute_explain_route` | Explain a past routing decision |
|
||||
| `omniroute_get_session_snapshot` | Full session state: costs, tokens, errors |
|
||||
| Tool | Scopes | Description |
|
||||
| :--------------------- | :--------------- | :--------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `omniroute_web_search` | `execute:search` | Web search through OmniRoute search gateway (Serper/Brave/Perplexity/Exa/Tavily/Google PSE/Linkup/SearchAPI/SearXNG) with failover |
|
||||
|
||||
## Advanced Tools (11) — Phase 2
|
||||
|
||||
| Tool | Scopes | Description |
|
||||
| :--------------------------------- | :----------------------------------- | :---------------------------------------------------------------------------------------- |
|
||||
| `omniroute_simulate_route` | `read:health`, `read:combos` | Dry-run routing simulation with fallback tree |
|
||||
| `omniroute_set_budget_guard` | `write:budget` | Session budget with degrade/block/alert action |
|
||||
| `omniroute_set_routing_strategy` | `write:combos` | Update combo strategy at runtime (priority/weighted/auto/etc.) |
|
||||
| `omniroute_set_resilience_profile` | `write:resilience` | Apply `aggressive` / `balanced` / `conservative` resilience preset |
|
||||
| `omniroute_test_combo` | `execute:completions`, `read:combos` | Live test of every provider in a combo using a real upstream call |
|
||||
| `omniroute_get_provider_metrics` | `read:health` | Per-provider metrics with p50/p95/p99 latency and circuit breaker state |
|
||||
| `omniroute_best_combo_for_task` | `read:combos`, `read:health` | Recommend combo by task type with budget/latency constraints |
|
||||
| `omniroute_explain_route` | `read:health`, `read:usage` | Explain why a request was routed to a provider (scoring factors + fallbacks) |
|
||||
| `omniroute_get_session_snapshot` | `read:usage` | Full session snapshot: cost, tokens, top models/providers, errors, budget guard |
|
||||
| `omniroute_db_health_check` | `read:health`, `write:resilience` | Diagnose (and optionally auto-repair) database drift like broken combo refs / orphan rows |
|
||||
| `omniroute_sync_pricing` | `pricing:write` | Sync pricing data from external sources (LiteLLM); supports `dryRun` |
|
||||
|
||||
## Cache Tools (2)
|
||||
|
||||
| Tool | Description |
|
||||
| :---------------------- | :-------------------------------------------------- |
|
||||
| `omniroute_cache_stats` | Semantic cache, prompt-cache, and idempotency stats |
|
||||
| `omniroute_cache_flush` | Flush cache globally or by signature/model |
|
||||
| Tool | Scopes | Description |
|
||||
| :---------------------- | :------------ | :-------------------------------------------------- |
|
||||
| `omniroute_cache_stats` | `read:cache` | Semantic cache, prompt-cache, and idempotency stats |
|
||||
| `omniroute_cache_flush` | `write:cache` | Flush cache globally or by signature/model |
|
||||
|
||||
## Compression Tools (5)
|
||||
|
||||
| Tool | Description |
|
||||
| :---------------------------------- | :------------------------------------------------------------- |
|
||||
| `omniroute_compression_status` | Compression settings, analytics summary, and cache-aware stats |
|
||||
| `omniroute_compression_configure` | Configure compression mode, threshold, and runtime options |
|
||||
| `omniroute_set_compression_engine` | Set Caveman, RTK, or stacked compression mode and pipeline |
|
||||
| `omniroute_list_compression_combos` | List named compression combos and routing assignments |
|
||||
| `omniroute_compression_combo_stats` | Analytics grouped by compression combo and engine |
|
||||
| Tool | Scopes | Description |
|
||||
| :---------------------------------- | :------------------ | :----------------------------------------------------------------------------------------------------------------------- |
|
||||
| `omniroute_compression_status` | `read:compression` | Compression settings, analytics summary, and cache-aware stats (includes `analytics.mcpDescriptionCompression` metadata) |
|
||||
| `omniroute_compression_configure` | `write:compression` | Configure compression mode, threshold, target ratio, system-prompt preservation, MCP description compression toggle |
|
||||
| `omniroute_set_compression_engine` | `write:compression` | Pick the active engine (off/caveman/rtk/stacked) and Caveman/RTK intensity |
|
||||
| `omniroute_list_compression_combos` | `read:compression` | List named compression combos and their engine pipelines |
|
||||
| `omniroute_compression_combo_stats` | `read:compression` | Analytics grouped by compression combo and engine |
|
||||
|
||||
`omniroute_compression_status` reports MCP description compression separately under
|
||||
`analytics.mcpDescriptionCompression`. Those values are metadata-size estimates for MCP listable
|
||||
@@ -75,45 +98,161 @@ receipts and are marked with `source: "mcp_metadata_estimate"`.
|
||||
See [Compression Engines](COMPRESSION_ENGINES.md) and [RTK Compression](RTK_COMPRESSION.md) for
|
||||
the runtime compression model behind these tools.
|
||||
|
||||
## Other Tool Groups
|
||||
## 1Proxy Tools (3)
|
||||
|
||||
The remaining MCP surface includes 1proxy tools, memory tools, and skill tools. The live source of
|
||||
truth is `open-sse/mcp-server/tools/` and `open-sse/mcp-server/schemas/tools.ts`.
|
||||
| Tool | Scopes | Description |
|
||||
| :-------------------------- | :------------- | :-------------------------------------------------------------------------------------- |
|
||||
| `omniroute_oneproxy_fetch` | `read:proxies` | Fetch free proxies from the 1proxy marketplace (protocol/country/quality/limit filters) |
|
||||
| `omniroute_oneproxy_rotate` | `read:proxies` | Get the next available proxy by strategy (`random` / `quality` / `sequential`) |
|
||||
| `omniroute_oneproxy_stats` | `read:proxies` | Pool stats, sync status, distribution by protocol and country |
|
||||
|
||||
## Authentication
|
||||
## Memory Tools (3)
|
||||
|
||||
MCP tools are authenticated via API key scopes. Each tool requires specific scopes:
|
||||
Defined in `open-sse/mcp-server/tools/memoryTools.ts`. Auth/scope is enforced through the standard MCP scope pipeline.
|
||||
|
||||
| Scope | Tools |
|
||||
| :-------------------- | :------------------------------------------------------------------- |
|
||||
| `read:health` | get_health, get_provider_metrics |
|
||||
| `read:combos` | list_combos, get_combo_metrics |
|
||||
| `write:combos` | switch_combo |
|
||||
| `read:quota` | check_quota |
|
||||
| `write:route` | route_request, simulate_route, test_combo |
|
||||
| `read:usage` | cost_report, get_session_snapshot, explain_route |
|
||||
| `write:config` | set_budget_guard, set_resilience_profile |
|
||||
| `read:models` | list_models_catalog, best_combo_for_task |
|
||||
| `read:cache` | cache_stats |
|
||||
| `write:cache` | cache_flush |
|
||||
| `read:compression` | compression_status, list_compression_combos, compression_combo_stats |
|
||||
| `write:compression` | compression_configure, set_compression_engine |
|
||||
| `execute:completions` | route_request, test_combo |
|
||||
| Tool | Description |
|
||||
| :------------------------ | :---------------------------------------------------------------------------------- |
|
||||
| `omniroute_memory_search` | Search memories by query / type / API key with token-budget enforcement |
|
||||
| `omniroute_memory_add` | Add a new memory entry (`factual` / `episodic` / `procedural` / `semantic`) |
|
||||
| `omniroute_memory_clear` | Clear memories for an API key, optionally filtered by type or `olderThan` timestamp |
|
||||
|
||||
## Skill Tools (4)
|
||||
|
||||
Defined in `open-sse/mcp-server/tools/skillTools.ts`. Backed by `src/lib/skills/registry` + `src/lib/skills/executor`.
|
||||
|
||||
| Tool | Description |
|
||||
| :---------------------------- | :-------------------------------------------------------------------------------- |
|
||||
| `omniroute_skills_list` | List registered skills with optional filtering by API key, name, or enabled state |
|
||||
| `omniroute_skills_enable` | Enable or disable a specific skill by ID |
|
||||
| `omniroute_skills_execute` | Execute a skill with provided input and return the execution record |
|
||||
| `omniroute_skills_executions` | List recent skill execution history |
|
||||
|
||||
---
|
||||
|
||||
## REST API Endpoints
|
||||
|
||||
| Endpoint | Method | Description | Auth |
|
||||
| :--------------------- | :-------------------- | :-------------------------------------------------------------------------------------------------- | :------------------------- |
|
||||
| `/api/mcp/status` | `GET` | Server status: heartbeat, HTTP transport state, audit activity summary | Management (session/admin) |
|
||||
| `/api/mcp/tools` | `GET` | Tool catalog (name, description, scopes, phase, source endpoints) | Management |
|
||||
| `/api/mcp/sse` | `GET` / `POST` | SSE transport endpoint (gated by `mcpEnabled` + `mcpTransport === "sse"`) | API key + scopes |
|
||||
| `/api/mcp/stream` | `POST`/`GET`/`DELETE` | Streamable HTTP transport (uses `mcp-session-id` header; `DELETE` ends the session) | API key + scopes |
|
||||
| `/api/mcp/audit` | `GET` | Audit log entries from `mcp_tool_audit` (filters: `limit`, `offset`, `tool`, `success`, `apiKeyId`) | Management |
|
||||
| `/api/mcp/audit/stats` | `GET` | Aggregated audit stats (`totalCalls`, `successRate`, `avgDurationMs`, top tools) | Management |
|
||||
|
||||
Source files: `src/app/api/mcp/{status,tools,sse,stream,audit,audit/stats}/route.ts`.
|
||||
|
||||
Both SSE and Streamable HTTP transports are blocked until the MCP server is enabled in Settings (`mcpEnabled`) and the appropriate `mcpTransport` is selected. If the wrong transport is configured the route returns HTTP 400 with a hint to switch settings.
|
||||
|
||||
---
|
||||
|
||||
## Authentication & Scopes
|
||||
|
||||
MCP tools are authenticated through API key scopes. Scope enforcement is centralized in
|
||||
`open-sse/mcp-server/scopeEnforcement.ts`. Each tool requires specific scopes:
|
||||
|
||||
| Scope | Tools |
|
||||
| :-------------------- | :---------------------------------------------------------------------------------------------------------------- |
|
||||
| `read:health` | `get_health`, `get_provider_metrics`, `simulate_route`, `explain_route`, `best_combo_for_task`, `db_health_check` |
|
||||
| `read:combos` | `list_combos`, `get_combo_metrics`, `simulate_route`, `best_combo_for_task`, `test_combo` |
|
||||
| `write:combos` | `switch_combo`, `set_routing_strategy` |
|
||||
| `read:quota` | `check_quota` |
|
||||
| `read:usage` | `cost_report`, `get_session_snapshot`, `explain_route` |
|
||||
| `read:models` | `list_models_catalog` |
|
||||
| `execute:completions` | `route_request`, `test_combo` |
|
||||
| `execute:search` | `web_search` |
|
||||
| `write:budget` | `set_budget_guard` |
|
||||
| `write:resilience` | `set_resilience_profile`, `db_health_check` |
|
||||
| `pricing:write` | `sync_pricing` |
|
||||
| `read:cache` | `cache_stats` |
|
||||
| `write:cache` | `cache_flush` |
|
||||
| `read:compression` | `compression_status`, `list_compression_combos`, `compression_combo_stats` |
|
||||
| `write:compression` | `compression_configure`, `set_compression_engine` |
|
||||
| `read:proxies` | `oneproxy_fetch`, `oneproxy_rotate`, `oneproxy_stats` |
|
||||
|
||||
Wildcard scopes are supported: `read:*` grants all read-scopes, `*` grants full access.
|
||||
|
||||
Memory and Skill tools currently do not declare static scope requirements in their definitions; access is gated by the caller's API key and audited through the standard MCP audit pipeline.
|
||||
|
||||
---
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
| :-------------------------------------- | :--------------------------------- | :----------------------------------------------------------------------------------------------------------------------- |
|
||||
| `OMNIROUTE_BASE_URL` | `http://localhost:20128` | Base URL the MCP server uses when calling OmniRoute internal APIs |
|
||||
| `OMNIROUTE_API_KEY` | (empty) | API key forwarded as `Authorization: Bearer` to internal API calls |
|
||||
| `OMNIROUTE_MCP_ENFORCE_SCOPES` | `false` (only `"true"` enables it) | When enabled, missing scopes deny tool calls and log `scope_denied:<reason>` in audit log |
|
||||
| `OMNIROUTE_MCP_SCOPES` | (empty) | Comma-separated allowlist of scopes considered "available" by default (used when caller does not provide its own scopes) |
|
||||
| `OMNIROUTE_MCP_COMPRESS_DESCRIPTIONS` | (unset = on) | When set to `0/false/off/no`, disables MCP description compression at registration time |
|
||||
| `OMNIROUTE_MCP_DESCRIPTION_COMPRESSION` | (unset = on) | Alternate alias for the same toggle as above |
|
||||
| `DATA_DIR` | `~/.omniroute` | Heartbeat file is written to `${DATA_DIR}/runtime/mcp-heartbeat.json` |
|
||||
|
||||
---
|
||||
|
||||
## Description Compression
|
||||
|
||||
MCP tool, prompt, and resource registries can compress descriptions at registration/list time to reduce the metadata footprint exposed to clients (and therefore the prompt context cost). The implementation lives in `open-sse/mcp-server/descriptionCompressor.ts` and is wired into the MCP server via `compressMcpRegistryMetadata` inside `createMcpServer()`.
|
||||
|
||||
- Compression runs over the description text using the Caveman ruleset (`getRulesForContext("all", "full")`) with preserved-block extraction (code spans, fenced blocks, etc.) so structural content is not altered.
|
||||
- Toggle per-deployment via the `compression.mcpDescriptionCompressionEnabled` value in the `key_value` settings table (default: enabled) — exposed in the UI as **Analytics → MCP description compression**.
|
||||
- Toggle process-wide via either `OMNIROUTE_MCP_COMPRESS_DESCRIPTIONS=false` or `OMNIROUTE_MCP_DESCRIPTION_COMPRESSION=false`.
|
||||
- Realtime stats are surfaced via `omniroute_compression_status` under `analytics.mcpDescriptionCompression` and tagged `source: "mcp_metadata_estimate"` to disambiguate from real provider usage receipts.
|
||||
|
||||
---
|
||||
|
||||
## Runtime Heartbeat
|
||||
|
||||
The stdio transport persists liveness to `${DATA_DIR}/runtime/mcp-heartbeat.json` every 5 seconds. The dashboard (`/api/mcp/status`) reads this file plus PID liveness to derive `online`. HTTP transports report state from in-process `getMcpHttpStatus()` instead (no file write).
|
||||
|
||||
The heartbeat snapshot contains:
|
||||
|
||||
```json
|
||||
{
|
||||
"pid": 12345,
|
||||
"startedAt": "2026-05-13T12:34:56.000Z",
|
||||
"lastHeartbeatAt": "2026-05-13T12:35:01.000Z",
|
||||
"version": "1.8.1",
|
||||
"transport": "stdio",
|
||||
"scopesEnforced": false,
|
||||
"allowedScopes": [],
|
||||
"toolCount": 37
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Audit Logging
|
||||
|
||||
Every tool call is logged to `mcp_tool_audit` with:
|
||||
Every tool call is logged to the SQLite `mcp_tool_audit` table by `open-sse/mcp-server/audit.ts`:
|
||||
|
||||
- Tool name, arguments, result
|
||||
- Duration (ms), success/failure
|
||||
- Tool name, arguments (hashed/truncated as per per-tool `auditLevel`), result
|
||||
- Duration in ms, success/failure flag, error message (when applicable)
|
||||
- API key hash, timestamp
|
||||
- Scope denials are logged as `scope_denied:<reason>` with the missing scope list
|
||||
|
||||
Use the dashboard or the `/api/mcp/audit` and `/api/mcp/audit/stats` REST endpoints to inspect recent calls.
|
||||
|
||||
---
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
| :------------------------------------------- | :------------------------------------------------ |
|
||||
| `open-sse/mcp-server/server.ts` | MCP server creation and scoped tool registrations |
|
||||
| `open-sse/mcp-server/transport.ts` | Stdio + HTTP transport |
|
||||
| `open-sse/mcp-server/auth.ts` | API key + scope validation |
|
||||
| `open-sse/mcp-server/audit.ts` | Tool call audit logging |
|
||||
| `open-sse/mcp-server/tools/advancedTools.ts` | 8 advanced tool handlers |
|
||||
| File | Purpose |
|
||||
| :---------------------------------------------- | :--------------------------------------------------------------- |
|
||||
| `open-sse/mcp-server/server.ts` | MCP server factory, stdio entry point, scoped tool registrations |
|
||||
| `open-sse/mcp-server/httpTransport.ts` | SSE + Streamable HTTP transport (session management) |
|
||||
| `open-sse/mcp-server/scopeEnforcement.ts` | Tool scope evaluation and caller resolution |
|
||||
| `open-sse/mcp-server/audit.ts` | Tool call audit logging (`mcp_tool_audit`) |
|
||||
| `open-sse/mcp-server/runtimeHeartbeat.ts` | stdio heartbeat writer (`mcp-heartbeat.json`) |
|
||||
| `open-sse/mcp-server/descriptionCompressor.ts` | Description compression for tool / prompt / resource registries |
|
||||
| `open-sse/mcp-server/schemas/tools.ts` | Zod schemas + tool registry (`MCP_TOOLS`, 30 entries) |
|
||||
| `open-sse/mcp-server/tools/advancedTools.ts` | Phase 2 + cache + 1proxy tool handlers |
|
||||
| `open-sse/mcp-server/tools/compressionTools.ts` | Compression tool handlers |
|
||||
| `open-sse/mcp-server/tools/memoryTools.ts` | Memory tool definitions (3 tools) |
|
||||
| `open-sse/mcp-server/tools/skillTools.ts` | Skill tool definitions (4 tools) |
|
||||
| `src/app/api/mcp/status/route.ts` | `/api/mcp/status` endpoint |
|
||||
| `src/app/api/mcp/tools/route.ts` | `/api/mcp/tools` endpoint |
|
||||
| `src/app/api/mcp/sse/route.ts` | `/api/mcp/sse` SSE transport route |
|
||||
| `src/app/api/mcp/stream/route.ts` | `/api/mcp/stream` Streamable HTTP transport route |
|
||||
| `src/app/api/mcp/audit/route.ts` | `/api/mcp/audit` audit log query |
|
||||
| `src/app/api/mcp/audit/stats/route.ts` | `/api/mcp/audit/stats` aggregated audit metrics |
|
||||
|
||||
322
docs/MEMORY.md
Normal file
322
docs/MEMORY.md
Normal file
@@ -0,0 +1,322 @@
|
||||
# Memory System
|
||||
|
||||
> **Source of truth:** `src/lib/memory/` and `src/app/api/memory/`
|
||||
> **Last updated:** 2026-05-13 — v3.8.0
|
||||
|
||||
OmniRoute provides persistent conversational memory keyed by API key (and
|
||||
optionally session id). Memories are extracted automatically from LLM responses
|
||||
via lightweight regex pattern matching and injected back into subsequent
|
||||
requests as a leading system message (or first user message for providers that
|
||||
reject the system role).
|
||||
|
||||
Memory is **scoped per API key**, not per user — every request authenticated
|
||||
with the same API key shares the same memory pool, with optional further
|
||||
scoping by `sessionId`.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Client → /v1/chat/completions (apiKeyInfo resolved upstream)
|
||||
→ handleChatCore() [open-sse/handlers/chatCore.ts]
|
||||
→ resolveMemoryOwnerId(apiKeyInfo) # extracts id
|
||||
→ getMemorySettings() # cached settings
|
||||
→ shouldInjectMemory(body, {enabled}) # gate
|
||||
→ retrieveMemories(apiKeyId, config) # SQL + optional FTS5
|
||||
→ injectMemory(body, memories, provider) # system or user message
|
||||
→ upstream provider call
|
||||
→ on response: extractFacts(text, apiKeyId, sessionId) # non-blocking
|
||||
→ setImmediate → createMemory(fact) per match
|
||||
```
|
||||
|
||||
The injection and extraction call-sites are wired in
|
||||
`open-sse/handlers/chatCore.ts` (look for `retrieveMemories`, `injectMemory`,
|
||||
and `extractFacts`).
|
||||
|
||||
## Storage Layers
|
||||
|
||||
### Primary: SQLite (`memories` table)
|
||||
|
||||
Created by migration `015_create_memories.sql`:
|
||||
|
||||
| Column | Type | Notes |
|
||||
| --------------------------- | ------------------ | -------------------------------------------------------------------- |
|
||||
| `id` | `TEXT PRIMARY KEY` | UUID generated via `crypto.randomUUID()` |
|
||||
| `api_key_id` | `TEXT NOT NULL` | Owning API key |
|
||||
| `session_id` | `TEXT` | Optional per-conversation scope |
|
||||
| `type` | `TEXT NOT NULL` | One of `factual`, `episodic`, `procedural`, `semantic` |
|
||||
| `key` | `TEXT` | Stable upsert key, e.g. `preference:i_prefer_python` |
|
||||
| `content` | `TEXT NOT NULL` | The actual fact text |
|
||||
| `metadata` | `TEXT` | JSON blob (category, extractedAt, source, ...) |
|
||||
| `created_at` / `updated_at` | `TEXT` | ISO 8601 strings |
|
||||
| `expires_at` | `TEXT` | Optional expiry; `NULL` means permanent |
|
||||
| `memory_id` | `INTEGER UNIQUE` | Added by `023_fix_memory_fts_uuid.sql` to bridge UUIDs ↔ FTS5 rowids |
|
||||
|
||||
Indexes: `api_key_id`, `session_id`, `type`, `expires_at`, plus the unique
|
||||
`memory_id` index.
|
||||
|
||||
**Upsert semantics**: `createMemory()` looks for an existing row with the same
|
||||
`(api_key_id, key)` and updates it in place when found (merging `metadata` via
|
||||
shallow spread). This keeps the table from growing unbounded for repeated
|
||||
preference statements.
|
||||
|
||||
### Full-text Search (`memory_fts` virtual table)
|
||||
|
||||
`022_add_memory_fts5.sql` creates an FTS5 virtual table over `content` and
|
||||
`key`. `023_fix_memory_fts_uuid.sql` fixes a real-world bug where the UUID
|
||||
primary key did not join to FTS5's integer rowid — the migration adds the
|
||||
`memory_id` column, recreates the FTS table, and wires triggers
|
||||
(`memory_fts_ai`, `memory_fts_ad`, `memory_fts_au`) that keep FTS in sync on
|
||||
INSERT, DELETE, and UPDATE.
|
||||
|
||||
Used by `retrieval.ts` for the `semantic` and `hybrid` strategies (see below).
|
||||
The retrieval code guards with `hasTable("memory_fts")` and falls back to
|
||||
chronological order if the FTS table is missing or the FTS query throws.
|
||||
|
||||
### Optional: Qdrant (vector store)
|
||||
|
||||
`src/lib/memory/qdrant.ts` implements an optional Qdrant integration for true
|
||||
semantic memory:
|
||||
|
||||
- `upsertSemanticMemoryPoint()` — embed `key + content` with the configured
|
||||
embedding model, ensure the collection exists (creates cosine-distance
|
||||
vectors on first use), and upsert a point with payload `{memoryId,
|
||||
apiKeyId, sessionId, key, content, metadata, createdAtUnix, expiresAtUnix}`.
|
||||
- `searchSemanticMemory(query, topK, scope)` — embed the query, search the
|
||||
collection filtered by `kind = "omniroute_memory"` and optionally by
|
||||
`apiKeyId` / `sessionId`. Caps `topK` to `[1, 20]`.
|
||||
- `deleteSemanticMemoryPoint(id)` — single point delete.
|
||||
- `cleanupSemanticMemoryPoints({retentionDays})` — bulk delete points whose
|
||||
`expiresAtUnix` is in the past or whose `createdAtUnix` is older than the
|
||||
retention cutoff. Counts first so the dashboard can show actual numbers.
|
||||
- `checkQdrantHealth()` — `GET /readyz` health probe with latency.
|
||||
|
||||
> **TODO**: The chat pipeline (`chatCore.ts`) and the in-tree `retrieveMemories()`
|
||||
> implementation do not currently call `upsertSemanticMemoryPoint` or
|
||||
> `searchSemanticMemory`. The Qdrant integration is feature-flagged via
|
||||
> `qdrantEnabled` in settings, but at the time of writing the
|
||||
> `searchSemanticMemory` results are not fused into retrieval — the
|
||||
> `semantic`/`hybrid` retrieval strategies use SQLite FTS5 only. The settings UI
|
||||
> in `dashboard/settings → MemorySkillsTab` exposes Qdrant config, health,
|
||||
> search test, and cleanup, but the corresponding `/api/settings/qdrant`,
|
||||
> `/api/settings/qdrant/health`, `/api/settings/qdrant/search`, and
|
||||
> `/api/settings/qdrant/cleanup` routes are referenced from the UI but **not
|
||||
> present** under `src/app/api/settings/qdrant/` (only `embedding-models/` is
|
||||
> wired). Treat Qdrant as preview/optional plumbing.
|
||||
|
||||
## Memory Types
|
||||
|
||||
`MemoryType` (`src/lib/memory/types.ts`):
|
||||
|
||||
| Type | Used for |
|
||||
| ------------ | ------------------------------------------------------------ |
|
||||
| `factual` | Preferences, stable user facts, behavioral patterns |
|
||||
| `episodic` | Decisions tied to a specific moment ("I chose Postgres") |
|
||||
| `procedural` | Workflow / how-to memory (reserved; no auto-extractor today) |
|
||||
| `semantic` | Reserved for vector-store entries |
|
||||
|
||||
`MemoryConfig` retrieval strategy is one of `exact`, `semantic`, or `hybrid`,
|
||||
and scope is one of `session`, `apiKey`, or `global`. The default scope from
|
||||
`getMemorySettings()` is `apiKey`.
|
||||
|
||||
## Fact Extraction (`extraction.ts`)
|
||||
|
||||
Extraction is **regex-based**, not LLM-based — it runs in-process with
|
||||
`setImmediate()` so it never blocks the response stream:
|
||||
|
||||
- **Preference patterns** → `MemoryType.FACTUAL`
|
||||
(e.g. `I prefer …`, `I really like …`, `my favorite is …`, `I hate …`)
|
||||
- **Decision patterns** → `MemoryType.EPISODIC`
|
||||
(e.g. `I'll use …`, `I chose …`, `I went with …`, `I'm going to adopt …`)
|
||||
- **Pattern patterns** → `MemoryType.FACTUAL`
|
||||
(e.g. `I usually …`, `I always …`, `I tend to …`)
|
||||
|
||||
Each match is sanitised (`trim`, whitespace-collapse, capped at 500 chars),
|
||||
deduplicated within the batch via a stable `factKey(category, content)`, and
|
||||
stored via `createMemory()` with metadata
|
||||
`{category, extractedAt, source: "llm_response"}`. Input text is capped at
|
||||
64 KiB (`MAX_EXTRACTION_TEXT_LENGTH`) — when longer, the **tail** of the text
|
||||
is used so the most recent assistant content always participates.
|
||||
|
||||
`extractFactsFromText(text)` is exported for tests and returns the structured
|
||||
facts without storing them.
|
||||
|
||||
## Retrieval (`retrieval.ts`)
|
||||
|
||||
`retrieveMemories(apiKeyId, config)` is the main entry point. It:
|
||||
|
||||
1. Normalises and validates the config through `MemoryConfigSchema`.
|
||||
2. Returns `[]` immediately when `enabled` is false or `maxTokens <= 0`.
|
||||
3. Clamps `maxTokens` to `[1, 8000]`.
|
||||
4. Detects whether the modern `memories` table exists (vs the legacy `memory`
|
||||
table) so older databases keep working.
|
||||
5. Builds the base query with expiry guard
|
||||
(`expires_at IS NULL OR datetime(expires_at) > datetime('now')`), optional
|
||||
session scope, and optional `retentionDays` cutoff.
|
||||
6. Branches on strategy:
|
||||
- **`exact`** (default): chronological `ORDER BY created_at DESC LIMIT 100`.
|
||||
- **`semantic`**: if `config.query` and `memory_fts` exists, JOIN
|
||||
`memory_fts MATCH ?` and order by FTS rank; fall back to chronological
|
||||
when FTS returns 0 rows.
|
||||
- **`hybrid`**: union of FTS results (higher relevance) and the
|
||||
chronological set, deduplicated by id.
|
||||
7. Computes a keyword relevance score (`getRelevanceScore`) over
|
||||
`content`, `key`, and `metadata` JSON when a query is provided. Rows with
|
||||
zero score are filtered out.
|
||||
8. Sorts by score desc, then `createdAt` desc.
|
||||
9. Walks the ranked list and accepts entries while a running
|
||||
`estimateTokens(content)` (≈ `length / 4`) stays under the budget. Always
|
||||
returns at least one entry when any matched.
|
||||
|
||||
`estimateTokens` is exported and used by retrieval, summarisation, and the MCP
|
||||
`omniroute_memory_search` tool.
|
||||
|
||||
## Injection (`injection.ts`)
|
||||
|
||||
`injectMemory(request, memories, provider)`:
|
||||
|
||||
1. Joins all memory contents into a single `Memory context: …` string.
|
||||
2. Picks a strategy by provider name:
|
||||
- **System message** (default for OpenAI, Anthropic, Gemini, …) — prepends
|
||||
a `{role: "system", content: memoryText}` ahead of any existing system
|
||||
messages so user system prompts still take precedence.
|
||||
- **User message** (fallback) — for providers in
|
||||
`PROVIDERS_WITHOUT_SYSTEM_MESSAGE`: `o1`, `o1-mini`, `o1-preview`,
|
||||
`glm`, `glmt`, `glm-cn`, `zai`, `qianfan`. These reject the system role
|
||||
and would 400 otherwise (cf. issue #1701 for GLM/Zhipu).
|
||||
3. Logs the count, strategy, and model under `memory.injection.injected`.
|
||||
|
||||
`providerSupportsSystemMessage(provider)` is exported for callers that need to
|
||||
make routing decisions of their own. Unknown providers default to `true`
|
||||
(system role allowed) for safety.
|
||||
|
||||
## Settings (`settings.ts`)
|
||||
|
||||
Memory configuration is **stored in the DB settings table**, not in env vars.
|
||||
`getMemorySettings()` reads from `getSettings()` and caches the result
|
||||
in-process; `invalidateMemorySettingsCache()` is called by the settings PUT
|
||||
route after writes.
|
||||
|
||||
| DB key | Type | Default | UI control |
|
||||
| --------------------- | ------- | -------------------------------------------------- | ----------------------------------------------- |
|
||||
| `memoryEnabled` | boolean | `true` | Memory on/off |
|
||||
| `memoryMaxTokens` | integer | `2000` (range `0–16000`) | Token budget for injection |
|
||||
| `memoryRetentionDays` | integer | `30` (range `1–365`) | Retention window |
|
||||
| `memoryStrategy` | enum | `"hybrid"` (one of `recent`, `semantic`, `hybrid`) | Retrieval strategy |
|
||||
| `skillsEnabled` | boolean | `false` | Toggles per-key skill injection (see SKILLS.md) |
|
||||
|
||||
Note: the UI strategy `"recent"` maps to the internal `"exact"` retrieval
|
||||
strategy via `toMemoryRetrievalConfig()` (chronological order).
|
||||
|
||||
Qdrant-related DB keys (`qdrantEnabled`, `qdrantHost`, `qdrantPort`,
|
||||
`qdrantApiKey`, `qdrantCollection` default `"omniroute_memory"`,
|
||||
`qdrantEmbeddingModel` default `"openai/text-embedding-3-small"`) are read by
|
||||
`normalizeQdrantConfig()` in `qdrant.ts`.
|
||||
|
||||
No `MEMORY_*` or `QDRANT_*` env vars exist today — everything is per-instance
|
||||
DB settings. `OMNIROUTE_MEMORY_MB` (commented out in `.env.example`) is
|
||||
unrelated and refers to Node heap sizing.
|
||||
|
||||
## Summarisation (`summarization.ts`)
|
||||
|
||||
`summarizeMemories(apiKeyId, sessionId?, maxTokens = 4000)` compacts older
|
||||
content when the running token total over a key's memories exceeds the
|
||||
budget. It iterates rows DESC by `created_at`, keeps rows that fit, and for
|
||||
the rest replaces `content` in place with the first three sentences of the
|
||||
original. `tokensSaved` is the difference in `estimateTokens` between old and
|
||||
new content.
|
||||
|
||||
This routine is **available but not called automatically** in the current
|
||||
chat pipeline — call it from a cron, an admin action, or
|
||||
`MemoryConfig.autoSummarize` glue if you need ongoing compaction. The data
|
||||
loss is one-way: original text is overwritten.
|
||||
|
||||
## REST API
|
||||
|
||||
All endpoints require management auth (`requireManagementAuth`).
|
||||
|
||||
| Method | Path | Description |
|
||||
| -------- | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `GET` | `/api/memory` | Paginated list with filters: `apiKeyId`, `type`, `sessionId`, `q`, `limit`, `page`, `offset`. Response includes `stats.total` and `stats.byType` |
|
||||
| `POST` | `/api/memory` | Create entry (Zod-validated: `content`, `key`, optional `type`, `sessionId`, `apiKeyId`, `metadata`, `expiresAt`). Calls `createMemory()` which upserts on `(apiKeyId, key)` |
|
||||
| `GET` | `/api/memory/[id]` | Fetch a single entry by UUID |
|
||||
| `DELETE` | `/api/memory/[id]` | Delete an entry; returns 404 when missing |
|
||||
| `GET` | `/api/memory/health` | Runs `verifyExtractionPipeline("health-check")` — round-trip create→list→delete to confirm the store is alive. Returns `{working, latencyMs, error?}` |
|
||||
| `GET` | `/api/settings/memory` | Current normalised `MemorySettings` |
|
||||
| `PUT` | `/api/settings/memory` | Update one or more of `enabled`, `maxTokens`, `retentionDays`, `strategy`, `skillsEnabled` |
|
||||
|
||||
The `/api/memory` list query supports either `page`-based pagination
|
||||
(`parsePaginationParams`) **or** raw `offset` — when `offset` is present it
|
||||
takes precedence and a derived `page` is computed for the response shape.
|
||||
|
||||
## MCP Tools (`open-sse/mcp-server/tools/memoryTools.ts`)
|
||||
|
||||
When the MCP server is enabled, three memory tools are registered:
|
||||
|
||||
- `omniroute_memory_search` — `{apiKeyId, query?, type?, maxTokens?, limit?}`
|
||||
→ wraps `retrieveMemories()` with `retrievalStrategy: "exact"`, optionally
|
||||
filters by `type`, and reports `totalTokens`.
|
||||
- `omniroute_memory_add` — `{apiKeyId, sessionId?, type, key, content,
|
||||
metadata?}` → wraps `createMemory()`.
|
||||
- `omniroute_memory_clear` — `{apiKeyId, type?, olderThan?}` → lists matching
|
||||
entries, optionally filters by created-before timestamp, then deletes each
|
||||
via `deleteMemory()`.
|
||||
|
||||
See [MCP-SERVER.md](./MCP-SERVER.md) for transport and scope details.
|
||||
|
||||
## Dashboard
|
||||
|
||||
`src/app/(dashboard)/dashboard/memory/page.tsx` provides:
|
||||
|
||||
- Real-time list, search, and pagination (debounced 300 ms).
|
||||
- Type filter (`factual` / `episodic` / `procedural` / `semantic` / all).
|
||||
- Add-memory modal (key, content, type).
|
||||
- Delete per row.
|
||||
- JSON export of the current page; JSON import via file picker.
|
||||
- A green/red health dot driven by `GET /api/memory/health`.
|
||||
- Stat cards: `totalEntries`, `tokensUsed`, `hitRate` (the latter two come
|
||||
from the API stats payload).
|
||||
|
||||
Memory and Qdrant settings live under
|
||||
`/dashboard/settings → Memory & Skills` (`MemorySkillsTab.tsx`).
|
||||
|
||||
## Caching
|
||||
|
||||
`src/lib/memory/store.ts` keeps an in-process LRU-ish cache
|
||||
(`MEMORY_CACHE_TTL = 5 min`, `MEMORY_MAX_CACHE_SIZE = 10 000`, with 20 %
|
||||
oldest eviction) for `getMemory(id)` reads, plus a generic key/value
|
||||
`memoryCache` layer (`src/lib/memory/cache.ts`) with `get`/`set`/`invalidate`
|
||||
methods used by callers that want their own scoped cache (1 000-entry LRU,
|
||||
default TTL 5 min).
|
||||
|
||||
## Privacy & Lifecycle
|
||||
|
||||
- Memory ownership is the API key id (`resolveMemoryOwnerId` in
|
||||
`chatCore.ts`). Without an `apiKeyInfo.id` neither retrieval nor injection
|
||||
nor extraction runs.
|
||||
- Entries with a future `expires_at` are filtered out of retrieval; old
|
||||
entries beyond `retentionDays` are excluded by the
|
||||
`created_at >= cutoff` clause in `retrieveMemories`.
|
||||
- For hard deletion, use `DELETE /api/memory/[id]` or `omniroute_memory_clear`.
|
||||
- Extraction is fire-and-forget via `setImmediate`; failures are logged under
|
||||
`memory.extraction.background.failed` and never surface to the caller.
|
||||
- Verification round-trips (`verifyExtractionPipeline`) clean up their own
|
||||
test entries in a `finally` block.
|
||||
|
||||
## See Also
|
||||
|
||||
- [SKILLS.md](./SKILLS.md) — the `skillsEnabled` setting injects tool
|
||||
definitions alongside memory.
|
||||
- [MCP-SERVER.md](./MCP-SERVER.md) — MCP transport / scopes.
|
||||
- [API_REFERENCE.md](./API_REFERENCE.md) — broader API surface.
|
||||
- [Tuto_Qdrant.MD](../Tuto_Qdrant.MD) — repository-root Qdrant setup tutorial.
|
||||
- Source modules:
|
||||
- `src/lib/memory/types.ts`, `schemas.ts`
|
||||
- `src/lib/memory/store.ts`, `retrieval.ts`, `injection.ts`
|
||||
- `src/lib/memory/extraction.ts`, `summarization.ts`, `verify.ts`
|
||||
- `src/lib/memory/settings.ts`, `qdrant.ts`, `cache.ts`
|
||||
- `src/lib/db/migrations/015_create_memories.sql`,
|
||||
`022_add_memory_fts5.sql`, `023_fix_memory_fts_uuid.sql`
|
||||
- `src/app/api/memory/route.ts`, `[id]/route.ts`, `health/route.ts`
|
||||
- `src/app/api/settings/memory/route.ts`
|
||||
- `open-sse/handlers/chatCore.ts` (injection / extraction wiring)
|
||||
- `open-sse/mcp-server/tools/memoryTools.ts`
|
||||
266
docs/PROVIDER_REFERENCE.md
Normal file
266
docs/PROVIDER_REFERENCE.md
Normal file
@@ -0,0 +1,266 @@
|
||||
# Provider Reference
|
||||
|
||||
> **Auto-generated** from `src/shared/constants/providers.ts` — do not edit by hand.
|
||||
> Regenerate with: `npm run gen:provider-reference`
|
||||
> **Last generated:** 2026-05-13
|
||||
|
||||
Total providers: **177**. See category breakdown below.
|
||||
|
||||
## Categories
|
||||
|
||||
- **Free** — free tier with API key (configured via dashboard)
|
||||
- **OAuth** — sign-in flow handled by OmniRoute, no API key needed
|
||||
- **Web cookie** — wraps the provider's web app via cookie auth
|
||||
- **API key** — paid provider configured via API key (free credits may apply)
|
||||
- **Local** — runs on the user's machine (Ollama, LM Studio, vLLM, etc.)
|
||||
- **Search** — web search providers
|
||||
- **Audio** — audio-only providers (TTS/STT)
|
||||
- **Upstream proxy** — providers that proxy to other providers
|
||||
- **Cloud agent** — long-running coding agents (Codex Cloud, Devin, Jules)
|
||||
- **System** — OmniRoute-internal providers (loopback, etc.)
|
||||
|
||||
Additional tags: `image`, `video`, `aggregator`, `enterprise`, `embed/rerank`, `self-hosted`.
|
||||
|
||||
Use the dashboard at `/dashboard/providers` to enable, configure, and test each provider.
|
||||
|
||||
---
|
||||
|
||||
## Free Tier (OAuth-first or no-key) (5)
|
||||
|
||||
| ID | Alias | Name | Tags | Website | Notes |
|
||||
| ------------ | ------------ | ---------- | ---- | ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `amazon-q` | `aq` | Amazon Q | Free | [link](https://aws.amazon.com/q/developer/) | Uses the same AWS Builder ID or imported refresh-token flow as Kiro, but keeps Amazon Q connections separate. |
|
||||
| `gemini-cli` | `gemini-cli` | Gemini CLI | Free | — | Uses Gemini CLI OAuth / Cloud Code credentials. Pro models require an eligible Google account or paid plan. |
|
||||
| `kiro` | `kr` | Kiro AI | Free | — | — |
|
||||
| `qoder` | `if` | Qoder AI | Free | — | — |
|
||||
| `qwen` | `qw` | Qwen Code | Free | — | ⚠️ **DEPRECATED.** Qwen OAuth free tier was discontinued on 2026-04-15. Use 'alicode', 'alicode-intl', or 'openrouter' provider with API key instead. |
|
||||
|
||||
## OAuth Providers (11)
|
||||
|
||||
| ID | Alias | Name | Tags | Website | Notes |
|
||||
| ------------- | ------------ | -------------------- | ----- | ------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `antigravity` | — | Antigravity | OAuth | — | — |
|
||||
| `claude` | `cc` | Claude Code | OAuth | — | — |
|
||||
| `cline` | `cl` | Cline | OAuth | — | — |
|
||||
| `codex` | `cx` | OpenAI Codex | OAuth | — | — |
|
||||
| `cursor` | `cu` | Cursor IDE | OAuth | — | — |
|
||||
| `devin-cli` | `dv` | Devin CLI (Official) | OAuth | [link](https://cli.devin.ai) | Requires the Devin CLI binary. Run `devin auth login` to authenticate, or provide your WINDSURF_API_KEY. Install: https://cli.devin.ai |
|
||||
| `github` | `gh` | GitHub Copilot | OAuth | — | — |
|
||||
| `gitlab-duo` | `gitlab-duo` | GitLab Duo | OAuth | [link](https://docs.gitlab.com/user/duo_agent_platform/code_suggestions/) | OAuth application with ai_features + read_user scopes. Configure GITLAB_DUO_OAUTH_CLIENT_ID and optionally GITLAB_DUO_OAUTH_CLIENT_SECRET on this OmniRoute instance. |
|
||||
| `kilocode` | `kc` | Kilo Code | OAuth | — | — |
|
||||
| `kimi-coding` | `kmc` | Kimi Coding | OAuth | — | — |
|
||||
| `windsurf` | `ws` | Windsurf (Devin CLI) | OAuth | [link](https://windsurf.com) | Sign in at windsurf.com to get your token. Visit windsurf.com/show-auth-token after logging in and paste it here, or use the device-code login flow. |
|
||||
|
||||
## Web Cookie Providers (5)
|
||||
|
||||
| ID | Alias | Name | Tags | Website | Notes |
|
||||
| ---------------- | ---------- | --------------------------- | ---------- | --------------------------------- | ------------------------------------------------------------------------------------------- |
|
||||
| `blackbox-web` | `bb-web` | Blackbox Web (Subscription) | Web cookie | [link](https://app.blackbox.ai) | Paste your \_\_Secure-authjs.session-token value or full cookie header from app.blackbox.ai |
|
||||
| `chatgpt-web` | `cgpt-web` | ChatGPT Web (Plus/Pro) | Web cookie | [link](https://chatgpt.com) | Paste your \_\_Secure-next-auth.session-token cookie value from chatgpt.com |
|
||||
| `grok-web` | `gw` | Grok Web (Subscription) | Web cookie | [link](https://grok.com) | Paste your sso= cookie value from grok.com |
|
||||
| `muse-spark-web` | `ms-web` | Muse Spark Web (Meta AI) | Web cookie | [link](https://www.meta.ai) | Paste your abra_sess value or full cookie header from meta.ai |
|
||||
| `perplexity-web` | `pplx-web` | Perplexity Web (Pro/Max) | Web cookie | [link](https://www.perplexity.ai) | Paste your \_\_Secure-next-auth.session-token cookie value from perplexity.ai |
|
||||
|
||||
## API Key Providers (paid / paid-with-free-credits) (123)
|
||||
|
||||
| ID | Alias | Name | Tags | Website | Notes |
|
||||
| --------------------- | -------------- | -------------------------- | --------------------- | -------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `agentrouter` | `agentrouter` | AgentRouter | API key, aggregator | [link](https://agentrouter.org) | $200 free credits on signup - multi-model routing gateway |
|
||||
| `ai21` | `ai21` | AI21 Labs | API key | [link](https://www.ai21.com) | $10 trial credits on signup (valid 3 months), no credit card required |
|
||||
| `aimlapi` | `aiml` | AI/ML API | API key, aggregator | [link](https://aimlapi.com) | $0.025/day free credits — 200+ models (GPT-4o, Claude, Gemini, Llama) via single endpoint |
|
||||
| `alibaba` | `ali` | Alibaba Cloud (DashScope) | API key | [link](https://dashscope-intl.aliyuncs.com) | — |
|
||||
| `alicode` | `alicode` | Alibaba | API key | [link](https://bailian.console.aliyun.com) | — |
|
||||
| `alicode-intl` | `alicode-intl` | Alibaba Intl | API key | [link](https://modelstudio.console.alibabacloud.com) | — |
|
||||
| `anthropic` | `anthropic` | Anthropic | API key | [link](https://platform.claude.com) | — |
|
||||
| `azure-ai` | `azure-ai` | Azure AI Foundry | API key, enterprise | [link](https://learn.microsoft.com/azure/ai-foundry) | Use your Azure AI Foundry key. Base URL can be https://<resource>.services.ai.azure.com/openai/v1/ or https://<resource>.openai.azure.com/openai/v1/. |
|
||||
| `azure-openai` | `azure` | Azure OpenAI | API key, enterprise | [link](https://azure.microsoft.com/products/ai-services/openai-service) | Use your Azure OpenAI API key. Base URL should be your resource endpoint, for example https://my-resource.openai.azure.com. |
|
||||
| `bailian-coding-plan` | `bcp` | Alibaba Coding Plan | API key | [link](https://www.alibabacloud.com/help/en/model-studio/coding-plan) | — |
|
||||
| `baseten` | `baseten` | Baseten | API key | [link](https://baseten.co) | $30 free trial credits for GPU inference |
|
||||
| `bazaarlink` | `bzl` | BazaarLink | API key | [link](https://bazaarlink.ai) | Free tier with auto:free routing — zero-cost inference, no credit card required |
|
||||
| `bedrock` | `bedrock` | Amazon Bedrock | API key, enterprise | [link](https://aws.amazon.com/bedrock) | Use your Amazon Bedrock API key in Authorization: Bearer <key>. OmniRoute defaults to the OpenAI-compatible bedrock-mantle endpoint in us-east-1; set a regional base URL if your account uses another region or the bedrock-runtime /openai/v1 path. |
|
||||
| `black-forest-labs` | `bfl` | Black Forest Labs | API key, image | [link](https://blackforestlabs.ai) | — |
|
||||
| `blackbox` | `bb` | Blackbox AI | API key | [link](https://blackbox.ai) | Free tier: unlimited basic chat plus Minimax-M2.5, no credit card required |
|
||||
| `bytez` | `bytez` | Bytez | API key | [link](https://bytez.com) | $1 free credits, refreshes every 4 weeks |
|
||||
| `cablyai` | `cablyai` | CablyAI | API key, aggregator | [link](https://cablyai.com) | Bearer API key for the CablyAI OpenAI-compatible gateway. |
|
||||
| `cerebras` | `cerebras` | Cerebras | API key | [link](https://inference.cerebras.ai) | Free: 1M tokens/day, 60K TPM — world's fastest inference |
|
||||
| `chutes` | `chutes` | Chutes.ai | API key, aggregator | [link](https://chutes.ai) | Bearer API key for the Chutes OpenAI-compatible gateway. |
|
||||
| `clarifai` | `clarifai` | Clarifai | API key, enterprise | [link](https://docs.clarifai.com) | Use your Clarifai PAT or app-specific API key. OmniRoute targets the OpenAI-compatible endpoint at https://api.clarifai.com/v2/ext/openai/v1 and authenticates with Authorization: Key <token>. |
|
||||
| `cloudflare-ai` | `cf` | Cloudflare Workers AI | API key | [link](https://developers.cloudflare.com/workers-ai) | Requires API Token AND Account ID (found at dash.cloudflare.com) |
|
||||
| `codestral` | `codestral` | Codestral | API key | [link](https://mistral.ai) | — |
|
||||
| `cohere` | `cohere` | Cohere | API key | [link](https://cohere.com) | Free Trial: 1,000 API calls/month for testing, no credit card required |
|
||||
| `command-code` | `cmd` | Command Code | API key | [link](https://commandcode.ai/) | Use a Command Code API key. Requests are sent to Command Code's /alpha/generate endpoint. |
|
||||
| `completions` | `cpl` | Completions.me | API key | [link](https://completions.me) | Free unlimited access to Claude, GPT, Gemini — no credit card, no rate limits |
|
||||
| `crof` | `crof` | CrofAI | API key | [link](https://crof.ai) | — |
|
||||
| `databricks` | `databricks` | Databricks | API key, enterprise | [link](https://www.databricks.com) | — |
|
||||
| `datarobot` | `datarobot` | DataRobot | API key, enterprise | [link](https://docs.datarobot.com) | Use your DataRobot API token. Optional Base URL can be the account root (for LLM Gateway) or a deployment URL under /api/v2/deployments/<id>. |
|
||||
| `deepinfra` | `deepinfra` | DeepInfra | API key | [link](https://deepinfra.com) | Free signup credits for API testing and model exploration |
|
||||
| `deepseek` | `ds` | DeepSeek | API key | [link](https://platform.deepseek.com) | 5M free tokens on signup - no credit card required |
|
||||
| `empower` | `empower` | Empower | API key, aggregator | [link](https://docs.empower.dev) | Bearer API key for the Empower OpenAI-compatible endpoint. |
|
||||
| `enally` | `enly` | Enally AI | API key | [link](https://ai.enally.in) | Free for students and developers — no credit card, OTP verification |
|
||||
| `fal-ai` | `fal` | Fal.ai | API key, image | [link](https://fal.ai) | — |
|
||||
| `featherless-ai` | `featherless` | Featherless AI | API key | [link](https://featherless.ai) | — |
|
||||
| `fenayai` | `fenayai` | FenayAI | API key, aggregator | [link](https://fenayai.com) | Bearer API key for the FenayAI OpenAI-compatible gateway. |
|
||||
| `fireworks` | `fireworks` | Fireworks AI | API key | [link](https://fireworks.ai) | $1 free starter credits on signup for API testing |
|
||||
| `freetheai` | `fta` | FreeTheAi | API key | [link](https://freetheai.xyz) | Community-run — free forever, no paid tiers, no credit card |
|
||||
| `friendliai` | `friendli` | FriendliAI | API key | [link](https://friendli.ai) | — |
|
||||
| `galadriel` | `galadriel` | Galadriel | API key | [link](https://galadriel.com) | — |
|
||||
| `gemini` | `gemini` | Gemini (Google AI Studio) | API key | [link](https://aistudio.google.com) | Free forever: 1,500 req/day for Gemini 2.5 Flash — no credit card, get key at aistudio.google.com |
|
||||
| `getgoapi` | `ggo` | GoAPI | API key, aggregator | [link](https://api.getgoapi.com) | — |
|
||||
| `gigachat` | `gigachat` | GigaChat (Sber) | API key | [link](https://developers.sber.ru) | — |
|
||||
| `gitlab` | `gitlab` | GitLab Duo PAT | API key | [link](https://docs.gitlab.com/user/duo_agent_platform/code_suggestions/) | GitLab personal access token for the public Code Suggestions API. Configure a self-hosted base URL when not using gitlab.com. |
|
||||
| `glhf` | `glhf` | GLHF Chat | API key, aggregator | [link](https://glhf.chat) | Bearer API key for the GLHF OpenAI-compatible gateway. |
|
||||
| `glm` | `glm` | GLM Coding | API key | [link](https://z.ai/subscribe) | — |
|
||||
| `glm-cn` | `glmcn` | GLM Coding (China) | API key | [link](https://open.bigmodel.cn) | — |
|
||||
| `glmt` | `glmt` | GLM Thinking | API key | [link](https://open.bigmodel.cn) | — |
|
||||
| `groq` | `groq` | Groq | API key | [link](https://groq.com) | Free tier: 30 RPM / 14.4K RPD — no credit card |
|
||||
| `heroku` | `heroku` | Heroku AI | API key, enterprise | [link](https://www.heroku.com) | — |
|
||||
| `huggingface` | `hf` | HuggingFace | API key | [link](https://huggingface.co) | Free Inference API for thousands of models (Whisper, VITS, SDXL…) |
|
||||
| `hyperbolic` | `hyp` | Hyperbolic | API key | [link](https://hyperbolic.xyz) | $1-5 trial credits on signup for serverless inference |
|
||||
| `inference-net` | `inet` | Inference.net | API key | [link](https://inference.net) | $25 free credits on signup plus research grants available |
|
||||
| `jina-ai` | `jina` | Jina AI | API key, embed/rerank | [link](https://jina.ai) | Bearer API key for the Jina AI rerank API. |
|
||||
| `kie` | `kie` | KIE.AI | API key | [link](https://kie.ai) | — |
|
||||
| `kilo-gateway` | `kg` | Kilo Gateway | API key, aggregator | [link](https://kilo.ai) | — |
|
||||
| `kimi` | `kimi` | Kimi | API key | [link](https://platform.moonshot.ai) | — |
|
||||
| `kimi-coding-apikey` | `kmca` | Kimi Coding (API Key) | API key | [link](https://www.kimi.com/code) | — |
|
||||
| `kluster` | `kluster` | Kluster AI | API key | [link](https://kluster.ai) | $5 free credits on signup - DeepSeek R1, Llama 4 Maverick/Scout, Qwen3 235B |
|
||||
| `lambda-ai` | `lambda` | Lambda AI | API key | [link](https://lambda.ai) | — |
|
||||
| `laozhang` | `lz` | LaoZhang AI | API key, aggregator | [link](https://api.laozhang.ai) | — |
|
||||
| `lepton` | `lepton` | Lepton AI | API key | [link](https://lepton.ai) | Free tier available - fast inference on custom hardware |
|
||||
| `llamagate` | `llamagate` | LlamaGate | API key | [link](https://llamagate.ai) | — |
|
||||
| `llm7` | `llm7` | LLM7.io | API key | [link](https://llm7.io) | No signup required - 2 req/s, 20 RPM, 100 req/hr free tier |
|
||||
| `longcat` | `lc` | LongCat AI | API key | [link](https://longcat.chat/platform/docs) | 50M tokens/day (Flash-Lite) + 500K/day (Chat/Thinking) — 100% free while public beta |
|
||||
| `maritalk` | `maritalk` | Maritalk | API key | [link](https://www.maritaca.ai) | — |
|
||||
| `meta-llama` | `meta` | Meta Llama API | API key | [link](https://llama.developer.meta.com) | — |
|
||||
| `minimax` | `minimax` | Minimax Coding | API key | [link](https://www.minimax.io) | — |
|
||||
| `minimax-cn` | `minimax-cn` | Minimax (China) | API key | [link](https://www.minimaxi.com) | — |
|
||||
| `mistral` | `mistral` | Mistral | API key | [link](https://mistral.ai) | Free Experiment tier: rate-limited access to all models, no credit card required |
|
||||
| `modal` | `mdl` | Modal | API key, enterprise | [link](https://modal.com/docs) | Use the bearer token that protects your Modal deployment, if enabled. Base URL should point to your OpenAI-compatible Modal app, for example https://<workspace>--<app>.modal.run/v1. |
|
||||
| `moonshot` | `moonshot` | Moonshot AI | API key | [link](https://platform.moonshot.ai) | — |
|
||||
| `morph` | `morph` | Morph | API key | [link](https://morphllm.com) | Free tier: 250K credits/month, $0 |
|
||||
| `nanobanana` | `nb` | NanoBanana | API key, image | [link](https://nanobananaapi.ai) | — |
|
||||
| `nanogpt` | `nanogpt` | NanoGPT | API key | [link](https://nano-gpt.com) | — |
|
||||
| `nebius` | `nebius` | Nebius AI | API key | [link](https://nebius.com) | ~$1 trial credits on signup for API testing |
|
||||
| `nlpcloud` | `nlpc` | NLP Cloud | API key | [link](https://docs.nlpcloud.com) | Use your NLP Cloud API key in Authorization: Token <key>. OmniRoute targets the chatbot endpoint on https://api.nlpcloud.io/v1/gpu/<model>/chatbot by default. |
|
||||
| `nous-research` | `nous` | Nous Research | API key | [link](https://portal.nousresearch.com/help) | Use your Nous Portal API key. OmniRoute targets the official OpenAI-compatible inference endpoint at https://inference-api.nousresearch.com/v1. |
|
||||
| `novita` | `novita` | Novita AI | API key, aggregator | [link](https://novita.ai) | $0.50 trial credits on signup (valid about 1 year) |
|
||||
| `nscale` | `nscale` | nScale | API key | [link](https://nscale.com) | $5 free credits on signup for inference testing |
|
||||
| `nvidia` | `nvidia` | NVIDIA NIM | API key | [link](https://build.nvidia.com) | Free dev access: ~40 RPM, 70+ models (Kimi K2.5, GLM 4.7, DeepSeek V3.2...) |
|
||||
| `oci` | `oci` | OCI Generative AI | API key, enterprise | [link](https://www.oracle.com/artificial-intelligence/generative-ai) | Use your OCI Generative AI API key or IAM bearer token. Base URL can be https://inference.generativeai.<region>.oci.oraclecloud.com/openai/v1/. |
|
||||
| `ollama-cloud` | `ollamacloud` | Ollama Cloud | API key | [link](https://ollama.com/settings/api-keys) | — |
|
||||
| `openai` | `openai` | OpenAI | API key | [link](https://platform.openai.com) | — |
|
||||
| `opencode-go` | `opencode-go` | OpenCode Go | API key | [link](https://opencode.ai/go) | — |
|
||||
| `opencode-zen` | `opencode-zen` | OpenCode Zen | API key | [link](https://opencode.ai/zen) | — |
|
||||
| `openrouter` | `openrouter` | OpenRouter | API key, aggregator | [link](https://openrouter.ai) | Free models at $0/token with :free suffix - 20 RPM / 200 RPD |
|
||||
| `ovhcloud` | `ovh` | OVHcloud AI | API key | [link](https://www.ovhcloud.com) | — |
|
||||
| `perplexity` | `pplx` | Perplexity | API key | [link](https://www.perplexity.ai) | — |
|
||||
| `petals` | `petals` | Petals | API key | [link](https://chat.petals.dev) | No API key is required for the public research endpoint. Leave the field blank, or provide a bearer token if your self-hosted Petals gateway uses auth. |
|
||||
| `piapi` | `pi` | PiAPI | API key, aggregator | [link](https://piapi.ai) | — |
|
||||
| `poe` | `poe` | Poe | API key, aggregator | [link](https://creator.poe.com/api-reference) | Bearer API key for the Poe OpenAI-compatible API. |
|
||||
| `pollinations` | `pol` | Pollinations AI | API key | [link](https://pollinations.ai) | No API key required for free public endpoint. Optional Spore tier: ~0.01 pollen/hour. |
|
||||
| `predibase` | `predibase` | Predibase | API key | [link](https://predibase.com) | $25 free trial credits (30-day validity) |
|
||||
| `publicai` | `publicai` | PublicAI | API key | [link](https://publicai.co) | Free community inference tier for testing |
|
||||
| `puter` | `pu` | Puter AI | API key | [link](https://puter.com) | Get token at puter.com/dashboard → Copy Auth Token |
|
||||
| `qianfan` | `qianfan` | Baidu Qianfan | API key | [link](https://cloud.baidu.com/product/wenxinworkshop) | — |
|
||||
| `recraft` | `recraft` | Recraft | API key, image | [link](https://recraft.ai) | — |
|
||||
| `reka` | `reka` | Reka | API key | [link](https://docs.reka.ai/chat/overview) | Use your Reka API key. OmniRoute supports the OpenAI-compatible base URL https://api.reka.ai/v1 and sends both Authorization and X-Api-Key headers for compatibility. |
|
||||
| `runwayml` | `runway` | Runway | API key, video | [link](https://docs.dev.runwayml.com) | Use your Runway API key in Authorization: Bearer <key>. OmniRoute targets the current Runway API at https://api.dev.runwayml.com/v1 and sends the required X-Runway-Version header automatically. |
|
||||
| `sambanova` | `samba` | SambaNova | API key | [link](https://sambanova.ai) | $5 free credits on signup (30-day validity), no credit card required |
|
||||
| `sap` | `sap` | SAP Generative AI Hub | API key, enterprise | [link](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/generative-ai-hub-in-sap-ai-core) | Use your SAP AI Core bearer token. Base URL can be your AI_API_URL root or a deploymentUrl from Generative AI Hub. |
|
||||
| `scaleway` | `scw` | Scaleway AI | API key | [link](https://www.scaleway.com/en/ai/generative-apis) | 1M free tokens for new accounts — EU/GDPR compliant (Paris), Qwen3 235B & Llama 70B |
|
||||
| `siliconflow` | `siliconflow` | SiliconFlow | API key | [link](https://cloud.siliconflow.com) | $1 free credits plus permanently free models after identity verification |
|
||||
| `snowflake` | `snowflake` | Snowflake Cortex | API key, enterprise | [link](https://www.snowflake.com) | — |
|
||||
| `stability-ai` | `stability` | Stability AI | API key, image | [link](https://stability.ai) | — |
|
||||
| `synthetic` | `synthetic` | Synthetic | API key, aggregator | [link](https://synthetic.new) | — |
|
||||
| `thebai` | `thebai` | TheB.AI | API key, aggregator | [link](https://theb.ai) | Bearer API key for the TheB.AI OpenAI-compatible gateway. |
|
||||
| `together` | `together` | Together AI | API key | [link](https://www.together.ai) | $25 signup credits + 3 permanently free models: Llama 3.3 70B, Vision, DeepSeek-R1 distill |
|
||||
| `topaz` | `topaz` | Topaz | API key, image | [link](https://topazlabs.com) | — |
|
||||
| `uncloseai` | `unc` | UncloseAI | API key | [link](https://uncloseai.com) | No auth required. API accepts any non-empty string as key for identification. |
|
||||
| `upstage` | `upstage` | Upstage | API key | [link](https://www.upstage.ai) | — |
|
||||
| `v0-vercel` | `v0` | v0 (Vercel) | API key | [link](https://v0.dev) | — |
|
||||
| `venice` | `venice` | Venice.ai | API key | [link](https://venice.ai) | — |
|
||||
| `vercel-ai-gateway` | `vag` | Vercel AI Gateway | API key, aggregator | [link](https://vercel.com/docs/ai-gateway) | — |
|
||||
| `vertex` | `vertex` | Vertex AI | API key, enterprise | [link](https://cloud.google.com/vertex-ai) | Provide Service Account JSON or OAuth access_token |
|
||||
| `vertex-partner` | `vp` | Vertex AI Partners | API key, enterprise | [link](https://cloud.google.com/vertex-ai) | Provide the same Service Account JSON used for Vertex AI partner models. |
|
||||
| `volcengine` | `volcengine` | Volcengine | API key | [link](https://www.volcengine.com) | — |
|
||||
| `voyage-ai` | `voyage` | Voyage AI | API key, embed/rerank | [link](https://www.voyageai.com) | Bearer API key for Voyage AI embeddings and rerank APIs. |
|
||||
| `wandb` | `wandb` | Weights & Biases Inference | API key | [link](https://wandb.ai) | — |
|
||||
| `watsonx` | `watsonx` | IBM watsonx.ai Gateway | API key, enterprise | [link](https://www.ibm.com/products/watsonx-ai) | Use your watsonx bearer token. Base URL can be https://<region>.ml.cloud.ibm.com/ml/gateway/v1/ or a self-managed /ml/gateway/v1 endpoint. |
|
||||
| `xai` | `xai` | xAI (Grok) | API key | [link](https://x.ai) | — |
|
||||
| `xiaomi-mimo` | `mimo` | Xiaomi MiMo | API key | [link](https://mimo.mi.com) | — |
|
||||
| `zai` | `zai` | Z.AI | API key | [link](https://open.bigmodel.cn) | — |
|
||||
|
||||
## Local Providers (10)
|
||||
|
||||
| ID | Alias | Name | Tags | Website | Notes |
|
||||
| --------------------- | ------------ | ------------------- | ------------------ | --------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `comfyui` | `comfyui` | ComfyUI | Local | [link](https://github.com/comfyanonymous/ComfyUI) | No API key required. Configure the local ComfyUI base URL (default: http://localhost:8188). |
|
||||
| `docker-model-runner` | `dmr` | Docker Model Runner | Local, self-hosted | [link](https://docs.docker.com/ai/model-runner/) | API key optional. Configure the local Docker Model Runner OpenAI-compatible base URL (default: http://localhost:12434/v1). |
|
||||
| `lemonade` | `lemonade` | Lemonade Server | Local, self-hosted | [link](https://lemonade-server.ai) | API key optional. Configure the local Lemonade OpenAI-compatible base URL (default: http://localhost:13305/api/v1). |
|
||||
| `llamafile` | `llamafile` | Llamafile | Local, self-hosted | [link](https://github.com/Mozilla-Ocho/llamafile) | API key optional. Configure the local Llamafile OpenAI-compatible base URL (default: http://127.0.0.1:8080/v1). |
|
||||
| `lm-studio` | `lmstudio` | LM Studio | Local, self-hosted | [link](https://lmstudio.ai) | API key optional. Configure the local LM Studio OpenAI-compatible base URL (default: http://localhost:1234/v1). |
|
||||
| `oobabooga` | `ooba` | oobabooga | Local, self-hosted | [link](https://github.com/oobabooga/text-generation-webui) | API key optional. Configure the local oobabooga OpenAI-compatible base URL (default: http://localhost:5000/v1). |
|
||||
| `sdwebui` | `sdwebui` | SD WebUI | Local | [link](https://github.com/AUTOMATIC1111/stable-diffusion-webui) | No API key required. Configure the local WebUI base URL (default: http://localhost:7860). |
|
||||
| `triton` | `triton` | NVIDIA Triton | Local, self-hosted | [link](https://developer.nvidia.com/triton-inference-server) | API key optional. Configure the Triton OpenAI-compatible base URL (default: http://localhost:8000/v1). |
|
||||
| `vllm` | `vllm` | vLLM | Local, self-hosted | [link](https://github.com/vllm-project/vllm) | API key optional. Configure the local vLLM OpenAI-compatible base URL (default: http://localhost:8000/v1). |
|
||||
| `xinference` | `xinference` | XInference | Local, self-hosted | [link](https://inference.readthedocs.io) | API key optional. Configure the local XInference OpenAI-compatible base URL (default: http://localhost:9997/v1). |
|
||||
|
||||
## Search Providers (11)
|
||||
|
||||
| ID | Alias | Name | Tags | Website | Notes |
|
||||
| ------------------- | --------------- | -------------------------- | ------ | --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
|
||||
| `brave-search` | `brave-search` | Brave Search | Search | [link](https://brave.com/search/api) | Subscription token from Brave Search API dashboard |
|
||||
| `exa-search` | `exa-search` | Exa Search | Search | [link](https://exa.ai) | API key from dashboard.exa.ai |
|
||||
| `google-pse-search` | `google-pse` | Google Programmable Search | Search | [link](https://developers.google.com/custom-search/v1/overview) | Requires a Google API key and your Programmable Search Engine ID (cx) |
|
||||
| `linkup-search` | `linkup` | Linkup Search | Search | [link](https://docs.linkup.so) | Bearer API key from the Linkup dashboard |
|
||||
| `ollama-search` | `ollama-search` | Ollama Search | Search | [link](https://ollama.com/settings/api-keys) | Same API key as Ollama Cloud (from ollama.com/settings/api-keys) |
|
||||
| `perplexity-search` | `pplx-search` | Perplexity Search | Search | [link](https://docs.perplexity.ai/guides/search-quickstart) | Same API key as Perplexity (pplx-...) |
|
||||
| `searchapi-search` | `searchapi` | SearchAPI | Search | [link](https://www.searchapi.io/docs) | API key from SearchAPI (query param or Bearer auth) |
|
||||
| `searxng-search` | `searxng` | SearXNG Search | Search | [link](https://docs.searxng.org) | API key is optional. Set your SearXNG base URL. Some instances may require a bearer token for access. |
|
||||
| `serper-search` | `serper-search` | Serper Search | Search | [link](https://serper.dev) | API key from serper.dev dashboard |
|
||||
| `tavily-search` | `tavily-search` | Tavily Search | Search | [link](https://tavily.com) | API key from app.tavily.com (format: tvly-...) |
|
||||
| `youcom-search` | `youcom-search` | You.com Search | Search | [link](https://you.com/docs/search/overview) | X-API-Key from the You.com platform dashboard |
|
||||
|
||||
## Audio-only Providers (7)
|
||||
|
||||
| ID | Alias | Name | Tags | Website | Notes |
|
||||
| ------------ | ---------- | ---------- | ----- | ------------------------------------- | ----------------------------------------------------------------------------------------------- |
|
||||
| `assemblyai` | `aai` | AssemblyAI | Audio | [link](https://assemblyai.com) | — |
|
||||
| `aws-polly` | `polly` | AWS Polly | Audio | [link](https://aws.amazon.com/polly/) | Use AWS Secret Access Key as API key; set providerSpecificData.accessKeyId and optional region. |
|
||||
| `cartesia` | `cartesia` | Cartesia | Audio | [link](https://cartesia.ai) | — |
|
||||
| `deepgram` | `dg` | Deepgram | Audio | [link](https://deepgram.com) | — |
|
||||
| `elevenlabs` | `el` | ElevenLabs | Audio | [link](https://elevenlabs.io) | — |
|
||||
| `inworld` | `inworld` | Inworld | Audio | [link](https://inworld.ai) | — |
|
||||
| `playht` | `playht` | PlayHT | Audio | [link](https://play.ht) | — |
|
||||
|
||||
## Upstream Proxy Providers (1)
|
||||
|
||||
| ID | Alias | Name | Tags | Website | Notes |
|
||||
| ------------- | ----- | ----------- | -------------- | ---------------------------------------------------- | ----- |
|
||||
| `cliproxyapi` | `cpa` | CLIProxyAPI | Upstream proxy | [link](https://github.com/router-for-me/CLIProxyAPI) | — |
|
||||
|
||||
## Cloud Agent Providers (3)
|
||||
|
||||
| ID | Alias | Name | Tags | Website | Notes |
|
||||
| ------------- | ------------- | ------------ | ----------- | -------------------------------- | ----------------------------------------------------------- |
|
||||
| `codex-cloud` | `codex-cloud` | Codex Cloud | Cloud agent | [link](https://openai.com/codex) | OpenAI API key with Codex Cloud task access. |
|
||||
| `devin` | `devin` | Devin | Cloud agent | [link](https://devin.ai) | Devin API key for cloud agent sessions. |
|
||||
| `jules` | `jules` | Google Jules | Cloud agent | [link](https://jules.google) | Jules API key for creating and managing cloud coding tasks. |
|
||||
|
||||
## System Providers (1)
|
||||
|
||||
| ID | Alias | Name | Tags | Website | Notes |
|
||||
| ------ | ------ | ------------------ | ------ | ------- | ----- |
|
||||
| `auto` | `auto` | Auto (Zero-Config) | System | — | — |
|
||||
|
||||
## Sources of truth
|
||||
|
||||
- Catalog: [`src/shared/constants/providers.ts`](../src/shared/constants/providers.ts)
|
||||
- Registry (per-model details): [`open-sse/config/providerRegistry.ts`](../open-sse/config/providerRegistry.ts)
|
||||
- Executors: [`open-sse/executors/`](../open-sse/executors/) (31 files)
|
||||
- Translators: [`open-sse/translator/`](../open-sse/translator/)
|
||||
|
||||
## See Also
|
||||
|
||||
- [FREE_TIERS.md](./FREE_TIERS.md) — curated free-tier guide
|
||||
- [USER_GUIDE.md](./USER_GUIDE.md) — provider setup walkthrough
|
||||
- [ARCHITECTURE.md](./ARCHITECTURE.md) — overall architecture
|
||||
@@ -10,7 +10,7 @@ OmniRoute includes a full-featured proxy management system that lets you route u
|
||||
|
||||
- [Why Use Proxies?](#why-use-proxies)
|
||||
- [Architecture Overview](#architecture-overview)
|
||||
- [3-Level Proxy System](#3-level-proxy-system)
|
||||
- [4-Level Proxy System](#4-level-proxy-system)
|
||||
- [Proxy Registry (CRUD)](#proxy-registry-crud)
|
||||
- [1proxy Free Marketplace](#1proxy-free-proxy-marketplace)
|
||||
- [Proxy Rotation](#proxy-rotation)
|
||||
@@ -78,7 +78,7 @@ Even outside blocked regions, proxies are useful for:
|
||||
|
||||
---
|
||||
|
||||
## 3-Level Proxy System
|
||||
## 4-Level Proxy System
|
||||
|
||||
OmniRoute supports proxy configuration at **four independent scopes**, resolved in priority order:
|
||||
|
||||
@@ -471,6 +471,10 @@ curl -X PUT "http://localhost:20128/api/upstream-proxy/openai" \
|
||||
| `GET` | `/api/v1/management/proxies/assignments` | List assignments |
|
||||
| `GET` | `/api/v1/management/proxies/health` | Proxy health stats |
|
||||
|
||||
### Tunnels API
|
||||
|
||||
For exposing your OmniRoute instance to the public internet (Cloudflare/ngrok/Tailscale) instead of routing outbound through a proxy, see [TUNNELS_GUIDE.md](./TUNNELS_GUIDE.md). The tunnel REST API lives under `/api/tunnels/{cloudflared,ngrok,tailscale}/*` and is orthogonal to the outbound proxy chain documented above.
|
||||
|
||||
### 1proxy API
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
@@ -495,13 +499,13 @@ curl -X PUT "http://localhost:20128/api/upstream-proxy/openai" \
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
| -------------------------------- | ------------------------------------- | ------------------------------- |
|
||||
| `ENABLE_SOCKS5_PROXY` | `false` | Enable SOCKS5 proxy support |
|
||||
| `ONEPROXY_ENABLED` | `true` | Enable 1proxy integration |
|
||||
| `ONEPROXY_API_URL` | `https://1proxy-api.aitradepulse.com` | 1proxy API endpoint |
|
||||
| `ONEPROXY_MAX_PROXIES` | `500` | Maximum proxies to sync |
|
||||
| `ONEPROXY_MIN_QUALITY_THRESHOLD` | `50` | Minimum quality score to import |
|
||||
| Variable | Default | Description |
|
||||
| -------------------------------- | ------------------------------------- | -------------------------------------------------------------- |
|
||||
| `ENABLE_SOCKS5_PROXY` | `true` | Enable SOCKS5 proxy support (default `true` in `.env.example`) |
|
||||
| `ONEPROXY_ENABLED` | `true` | Enable 1proxy integration |
|
||||
| `ONEPROXY_API_URL` | `https://1proxy-api.aitradepulse.com` | 1proxy API endpoint |
|
||||
| `ONEPROXY_MAX_PROXIES` | `500` | Maximum proxies to sync |
|
||||
| `ONEPROXY_MIN_QUALITY_THRESHOLD` | `50` | Minimum quality score to import |
|
||||
|
||||
---
|
||||
|
||||
|
||||
159
docs/REASONING_REPLAY.md
Normal file
159
docs/REASONING_REPLAY.md
Normal file
@@ -0,0 +1,159 @@
|
||||
# Reasoning Replay Cache
|
||||
|
||||
> **Source of truth:** `src/lib/db/reasoningCache.ts`, `open-sse/services/reasoningCache.ts`
|
||||
> **Last updated:** 2026-05-13 — v3.8.0
|
||||
|
||||
OmniRoute captures assistant `reasoning_content` produced by thinking-mode models and replays it transparently on multi-turn requests when the upstream provider requires it. This eliminates the HTTP 400 errors that strict providers raise when a client's conversation history is missing the prior turn's reasoning.
|
||||
|
||||
## Why This Exists
|
||||
|
||||
Several thinking-mode providers reject a follow-up turn unless the **previous assistant message includes the original `reasoning_content`**. The upstream returns 400 with messages like:
|
||||
|
||||
```
|
||||
Param Incorrect: The reasoning_content in the thinking mode must be passed back to the API.
|
||||
```
|
||||
|
||||
But typical clients (Cursor, Cline, Roo Code, OpenAI SDK) strip `reasoning_content` from the history they replay. OmniRoute restores it from a server-side cache so the request the upstream sees is consistent. Issue #1628 introduced the hybrid memory/SQLite persistence so the cache survives process restarts.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Turn N (assistant generates):
|
||||
→ response contains reasoning_content + tool_calls
|
||||
→ cacheReasoningFromAssistantMessage() writes (memory + DB), keyed by every tool_call.id
|
||||
→ forward response to client (which may or may not retain reasoning)
|
||||
|
||||
Turn N+1 (client sends follow-up):
|
||||
→ translator detects: requiresReasoningReplay(provider, model) === true
|
||||
→ for each assistant message with tool_calls and no reasoning_content:
|
||||
lookupReasoning(toolCalls[0].id) → memory → DB
|
||||
hit → msg.reasoning_content = cached; recordReplay()
|
||||
miss → msg.reasoning_content = "" (legacy fallback for older DeepSeek)
|
||||
→ upstream sees consistent history → no 400
|
||||
```
|
||||
|
||||
Capture happens in `open-sse/handlers/chatCore.ts` (two sites, around lines 4093 and 4380). Replay happens in `open-sse/translator/index.ts` after schema coercion but before dispatch.
|
||||
|
||||
## Storage — Hybrid Memory + SQLite
|
||||
|
||||
The hot path uses an in-memory `Map` (LRU-by-creation) backed by a SQLite table for crash recovery and dashboard visibility.
|
||||
|
||||
| Layer | Implementation | Purpose |
|
||||
| ------ | ---------------------------------------------- | -------------------------------------- |
|
||||
| Memory | `Map` in `open-sse/services/reasoningCache.ts` | Fast lookups, evicts oldest at 2000 |
|
||||
| DB | `reasoning_cache` table (`src/lib/db/`) | Persists across restarts, drives stats |
|
||||
|
||||
Writes go to both. Reads consult memory first, then fall back to DB (DB hits are promoted back into memory). DB failures are non-fatal — the in-memory cache continues to serve the hot path.
|
||||
|
||||
**Defaults:**
|
||||
|
||||
- TTL: `2h` (`TTL_MS = 2 * 60 * 60 * 1000`)
|
||||
- Max memory entries: `2000` (`MAX_MEMORY_ENTRIES`)
|
||||
- Eviction: oldest `createdAt` first
|
||||
|
||||
## Database Schema
|
||||
|
||||
Migration: `src/lib/db/migrations/033_create_reasoning_cache.sql`
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS reasoning_cache (
|
||||
tool_call_id TEXT PRIMARY KEY,
|
||||
provider TEXT NOT NULL,
|
||||
model TEXT NOT NULL,
|
||||
reasoning TEXT NOT NULL,
|
||||
char_count INTEGER NOT NULL DEFAULT 0,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
expires_at INTEGER NOT NULL
|
||||
);
|
||||
```
|
||||
|
||||
Indexes: `expires_at`, `provider`, `model`, `created_at`. `expires_at` is stored as Unix epoch seconds; the SELECT layer normalizes legacy text values via `EXPIRES_AT_EPOCH_SQL`.
|
||||
|
||||
## Provider / Model Detection
|
||||
|
||||
Replay is enabled when `requiresReasoningReplay(provider, model)` returns `true`. The function checks two lists in `open-sse/services/reasoningCache.ts`.
|
||||
|
||||
**Provider IDs (exact match, case-insensitive):**
|
||||
|
||||
- `deepseek`
|
||||
- `opencode-go`
|
||||
- `siliconflow`
|
||||
- `nebius`
|
||||
- `deepinfra`
|
||||
- `sambanova`
|
||||
- `fireworks`
|
||||
- `together`
|
||||
- `xiaomi-mimo`
|
||||
|
||||
**Model regex patterns (case-insensitive):**
|
||||
|
||||
- `/deepseek-r1/i`
|
||||
- `/deepseek-reasoner/i`
|
||||
- `/deepseek-chat/i`
|
||||
- `/kimi-k2/i`
|
||||
- `/qwq/i`
|
||||
- `/qwen.*think/i`
|
||||
- `/glm.*think/i`
|
||||
- `/^mimo[-.]?v\d/i`
|
||||
|
||||
Adding a new strict provider/model means appending to one of these lists and writing a unit test asserting replay injection. The PR description should cite the exact upstream 400 string that motivated the change.
|
||||
|
||||
## REST API
|
||||
|
||||
The cache exposes two endpoints under `src/app/api/cache/reasoning/route.ts`. Both require management authentication (`isAuthenticated` from `@/shared/utils/apiAuth`).
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
| ------ | --------------------------------------------------------- | -------------------------------------------------------- |
|
||||
| GET | `/api/cache/reasoning` | Stats + paginated entries |
|
||||
| GET | `/api/cache/reasoning?provider=deepseek&model=...&limit=` | Filtered listing (`limit` clamped to `[1, 200]`) |
|
||||
| DELETE | `/api/cache/reasoning` | Clear everything (memory + DB) and reset hit/miss counts |
|
||||
| DELETE | `/api/cache/reasoning?provider=deepseek` | Clear only entries for one provider |
|
||||
| DELETE | `/api/cache/reasoning?toolCallId=call_abc` | Delete a single entry |
|
||||
|
||||
**GET response shape:**
|
||||
|
||||
```json
|
||||
{
|
||||
"stats": {
|
||||
"memoryEntries": 12,
|
||||
"dbEntries": 47,
|
||||
"totalEntries": 47,
|
||||
"totalChars": 138291,
|
||||
"hits": 84,
|
||||
"misses": 6,
|
||||
"replays": 81,
|
||||
"replayRate": "90.0%",
|
||||
"byProvider": { "deepseek": { "entries": 32, "chars": 98412 } },
|
||||
"byModel": { "deepseek-reasoner": { "entries": 32, "chars": 98412 } },
|
||||
"oldestEntry": "2026-05-13T10:00:00.000Z",
|
||||
"newestEntry": "2026-05-13T11:42:11.000Z"
|
||||
},
|
||||
"entries": [
|
||||
{
|
||||
"toolCallId": "call_abc",
|
||||
"provider": "deepseek",
|
||||
"model": "deepseek-reasoner",
|
||||
"reasoning": "...",
|
||||
"charCount": 3128,
|
||||
"createdAt": "...",
|
||||
"expiresAt": "..."
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Operational Notes
|
||||
|
||||
- **Cleanup:** `cleanupReasoningCache()` purges expired memory entries and runs `DELETE FROM reasoning_cache WHERE expires_at <= unixepoch('now')`. Health-check workers call this periodically.
|
||||
- **Crash recovery:** After a restart, memory is empty but the DB still holds unexpired entries. The first lookup for a given `tool_call_id` is a DB hit; subsequent lookups are memory hits.
|
||||
- **No reasoning, no cache:** `cacheReasoningFromAssistantMessage` returns `0` when the assistant message has no `reasoning_content` / `reasoning` field, so non-thinking responses cost nothing.
|
||||
- **Non-strict providers:** When `requiresReasoningReplay` is `false` and the target format is OpenAI, the translator **strips** any `reasoning_content` field from outgoing messages — OpenAI Chat Completions does not accept it.
|
||||
|
||||
## See Also
|
||||
|
||||
- [RESILIENCE_GUIDE.md](./RESILIENCE_GUIDE.md) — circuit breakers, cooldowns, model lockouts
|
||||
- [TROUBLESHOOTING.md](./TROUBLESHOOTING.md) — diagnosing upstream 400s
|
||||
- Source: `src/lib/db/reasoningCache.ts`, `open-sse/services/reasoningCache.ts`, `open-sse/translator/index.ts`
|
||||
- Migration: `src/lib/db/migrations/033_create_reasoning_cache.sql`
|
||||
- API route: `src/app/api/cache/reasoning/route.ts`
|
||||
- Original issue: #1628
|
||||
@@ -1,37 +1,203 @@
|
||||
# Release Checklist
|
||||
|
||||
Use this checklist before tagging or publishing a new OmniRoute release.
|
||||
> **Last updated:** 2026-05-13 — v3.8.0
|
||||
> Streamlined release flow that leverages Claude Code skills for automation.
|
||||
|
||||
## Version and Changelog
|
||||
## TL;DR
|
||||
|
||||
1. Bump `package.json` version (`x.y.z`) in the release branch.
|
||||
2. Move release notes from `## [Unreleased]` in `CHANGELOG.md` to a dated section:
|
||||
- `## [x.y.z] — YYYY-MM-DD`
|
||||
3. Keep `## [Unreleased]` as the first changelog section for upcoming work.
|
||||
4. Ensure the latest semver section in `CHANGELOG.md` equals `package.json` version.
|
||||
```bash
|
||||
# 1. Bump version + generate CHANGELOG (skill)
|
||||
/version-bump-cc patch # or minor/major
|
||||
|
||||
## API Docs
|
||||
# 2. Run quality gate locally
|
||||
npm run check # lint + tests
|
||||
npm run test:coverage # full coverage gate (75/75/75/70)
|
||||
|
||||
1. Update `docs/openapi.yaml`:
|
||||
- `info.version` must equal `package.json` version.
|
||||
2. Validate endpoint examples if API contracts changed.
|
||||
# 3. Build & smoke
|
||||
npm run build
|
||||
npm run test:e2e # optional but recommended
|
||||
|
||||
## Runtime Docs
|
||||
# 4. Generate release (skill)
|
||||
/generate-release-cc
|
||||
|
||||
1. Review `docs/ARCHITECTURE.md` for storage/runtime drift.
|
||||
2. Review `docs/TROUBLESHOOTING.md` for env var and operational drift.
|
||||
3. Verify the release/runtime Node.js version still satisfies the supported secure floor:
|
||||
- `>=20.20.2 <21`, `>=22.22.2 <23`, or `>=24.0.0 <25`
|
||||
- `npm run check:node-runtime`
|
||||
4. Validate the npm publish artifact after building the standalone package:
|
||||
- `npm run build:cli`
|
||||
- `npm run check:pack-artifact`
|
||||
- confirm no `app.__qa_backup`, `scripts/scratch`, `package-lock.json`, or other local residue
|
||||
5. Update localized docs if source docs changed significantly.
|
||||
# 5. Deploy (skill)
|
||||
/deploy-vps-both-cc # or akamai-cc / local-cc
|
||||
|
||||
## Automated Check
|
||||
# 6. Capture release evidences (skill)
|
||||
/capture-release-evidences-cc
|
||||
```
|
||||
|
||||
Run the sync guard locally before opening PR:
|
||||
## Detailed Checklist
|
||||
|
||||
### Pre-release
|
||||
|
||||
- [ ] All PRs targeted to this release are merged to `release/vX.Y.0`
|
||||
- [ ] All open Linear/issue items for this version are closed or pushed to next milestone
|
||||
- [ ] CI green on `release/vX.Y.0` branch
|
||||
- [ ] No `TODO(release)` markers in code: `grep -r "TODO(release)" src/ open-sse/`
|
||||
- [ ] Docker base image up to date (currently `node:24.15.0-trixie-slim`)
|
||||
|
||||
### Version & Changelog
|
||||
|
||||
- [ ] Run `/version-bump-cc <patch|minor|major>` (Claude Code skill)
|
||||
- Bumps `package.json`, `electron/package.json`
|
||||
- Regenerates `CHANGELOG.md` from git commits since last tag
|
||||
- Updates README.md badges
|
||||
- [ ] Manually review CHANGELOG.md and clean up commit messages if needed
|
||||
- [ ] Ensure the latest semver section in `CHANGELOG.md` equals `package.json` version
|
||||
- [ ] Keep `## [Unreleased]` as the first changelog section for upcoming work
|
||||
- [ ] Update `docs/openapi.yaml` → `info.version` must equal `package.json` version
|
||||
|
||||
### Code Quality
|
||||
|
||||
- [ ] `npm run lint` — 0 errors (warnings are pre-existing)
|
||||
- [ ] `npm run typecheck:core` — clean
|
||||
- [ ] `npm run typecheck:noimplicit:core` — clean (strict)
|
||||
- [ ] `npm run check:cycles` — no circular deps
|
||||
- [ ] `npm run check:any-budget:t11` — within budget
|
||||
- [ ] `npm run check:route-validation:t06` — clean
|
||||
- [ ] `npm run check:node-runtime` — supported floor met (`>=20.20.2 <21`, `>=22.22.2 <23`, `>=24.0.0 <25`)
|
||||
|
||||
### Testing
|
||||
|
||||
- [ ] `npm run test:unit` — pass
|
||||
- [ ] `npm run test:vitest` — pass (MCP server, autoCombo, cache)
|
||||
- [ ] `npm run test:coverage` — gate 75/75/75/70 satisfied (statements/lines/functions/branches)
|
||||
- [ ] `npm run test:integration` — pass (if changes touch DB / handlers)
|
||||
- [ ] `npm run test:e2e` — pass (UI changes)
|
||||
- [ ] `npm run test:protocols:e2e` — pass (MCP/A2A changes)
|
||||
- [ ] `npm run test:ecosystem` — pass
|
||||
|
||||
### Hooks (Husky validated)
|
||||
|
||||
Husky hooks live in `.husky/` and run automatically on git operations.
|
||||
|
||||
- **pre-commit:** `npx lint-staged + node scripts/check-docs-sync.mjs + npm run check:any-budget:t11`
|
||||
- **pre-push:** currently disabled (commented out). When re-enabled, runs `npm run test:unit`.
|
||||
- Run `npm run test:unit` manually before pushing release branches.
|
||||
|
||||
If a hook fails: fix the underlying issue, don't bypass with `--no-verify`.
|
||||
|
||||
### Conventional Commits
|
||||
|
||||
All release-bound commits must follow `type(scope): subject` format.
|
||||
|
||||
**Valid types:** `feat`, `fix`, `refactor`, `docs`, `test`, `chore`, `perf`, `style`, `ci`
|
||||
|
||||
**Valid scopes:** `db`, `sse`, `oauth`, `dashboard`, `api`, `cli`, `docker`, `ci`, `mcp`, `a2a`, `memory`, `skills`, `cloud-agent`, `guardrails`, `compression`, `auto-combo`, `resilience`, `providers`, `executors`, `translator`, `domain`, `authz`
|
||||
|
||||
Breaking changes: add `BREAKING CHANGE:` footer or `!` after the scope (e.g. `feat(api)!: drop /v0`).
|
||||
|
||||
### Documentation
|
||||
|
||||
- [ ] `npm run check:docs-sync` passes (auto-run by pre-commit)
|
||||
- [ ] `docs/ARCHITECTURE.md` reviewed for storage/runtime drift
|
||||
- [ ] `docs/TROUBLESHOOTING.md` reviewed for env var and operational drift
|
||||
- [ ] If `.env.example` changed: `docs/ENVIRONMENT.md` updated
|
||||
- [ ] If new feature has a UI: `docs/USER_GUIDE.md` mentions it
|
||||
- [ ] If new feature has API: `docs/API_REFERENCE.md` + `docs/openapi.yaml` updated
|
||||
- [ ] If new feature is a module: dedicated `docs/<MODULE>.md` exists
|
||||
- [ ] If breaking change: `docs/TROUBLESHOOTING.md` has migration note
|
||||
|
||||
### i18n
|
||||
|
||||
- [ ] Check `docs/i18n/` for major language drift against source docs
|
||||
- [ ] Run `scripts/i18n-check.mjs` if present in repo
|
||||
- [ ] Translation contributions can be deferred to next release if minor (track in CHANGELOG)
|
||||
|
||||
### Database Migrations
|
||||
|
||||
- [ ] If `src/lib/db/migrations/` has new files:
|
||||
- [ ] Each migration is idempotent (`CREATE TABLE IF NOT EXISTS`, etc.)
|
||||
- [ ] Migrations wrapped in transactions
|
||||
- [ ] Numbered correctly (no gaps in sequence)
|
||||
- [ ] Test on fresh install: delete `~/.omniroute/omniroute.db` and run `npm run dev`
|
||||
- [ ] Test on existing install: backup DB, run migration, verify schema
|
||||
- [ ] WAL files (`-wal`, `-shm`) handled correctly if migration rewrites tables
|
||||
|
||||
### Provider Catalog (Zod-validated)
|
||||
|
||||
- [ ] `src/shared/constants/providers.ts` Zod schema valid at load time
|
||||
- [ ] All providers have required fields (`id`, `label`, `kind`, etc.)
|
||||
- [ ] `freeNote` provided for new free providers
|
||||
- [ ] OAuth providers have `oauthConfig` registered in `src/lib/oauth/constants/oauth.ts`
|
||||
- [ ] If new provider added: corresponding executor in `open-sse/executors/`
|
||||
- [ ] If non-OpenAI format: translator in `open-sse/translator/`
|
||||
- [ ] Models registered in `open-sse/config/providerRegistry.ts`
|
||||
- [ ] Unit tests in `tests/unit/` cover provider classification and routing
|
||||
|
||||
### Desktop (Electron)
|
||||
|
||||
If `electron/` changed:
|
||||
|
||||
- [ ] `npm run electron:smoke:packaged` passes
|
||||
- [ ] Builds tested for at least one of `:win`, `:mac`, `:linux`
|
||||
- [ ] Code signing certs not expired (if signing)
|
||||
- [ ] `electron/package.json` version matches root `package.json`
|
||||
- [ ] Auto-update channel pointer updated if releasing to `stable`
|
||||
|
||||
### Artifact Validation
|
||||
|
||||
- [ ] `npm run build:cli` succeeds
|
||||
- [ ] `npm run check:pack-artifact` clean — no `app.__qa_backup`, `scripts/scratch`, `package-lock.json`, or other local residue
|
||||
- [ ] `npm run build` produces a working standalone Next.js bundle
|
||||
|
||||
### Tagging & Release
|
||||
|
||||
- [ ] Run `/generate-release-cc` (Claude Code skill):
|
||||
- Creates tag `vX.Y.Z`
|
||||
- Pushes tag and branch
|
||||
- Opens GitHub Release with changelog body
|
||||
- Attaches Electron installers (if built)
|
||||
- [ ] Or manually:
|
||||
```bash
|
||||
git tag -a vX.Y.Z -m "Release vX.Y.Z"
|
||||
git push origin vX.Y.Z
|
||||
gh release create vX.Y.Z --notes-from-tag
|
||||
```
|
||||
|
||||
### Deploy
|
||||
|
||||
- [ ] Use deploy skill that matches target:
|
||||
- `/deploy-vps-local-cc` — local VPS (192.168.0.15)
|
||||
- `/deploy-vps-akamai-cc` — Akamai VPS (69.164.221.35)
|
||||
- `/deploy-vps-both-cc` — both
|
||||
- [ ] Smoke test deployed instance:
|
||||
- Open `/dashboard/health` → check version string matches release
|
||||
- Run a `/v1/chat/completions` request against a known provider
|
||||
- Verify `/api/monitoring/health` returns `CLOSED` circuit breakers
|
||||
- Confirm MCP transports respond (`/mcp` HTTP, `/mcp-sse` SSE)
|
||||
|
||||
### Post-release
|
||||
|
||||
- [ ] Run `/capture-release-evidences-cc` (Claude Code skill)
|
||||
- Captures WebP screenshots/recordings of new features
|
||||
- Attaches to release notes / blog post
|
||||
- [ ] Update GitHub Discussions / Discord with release announcement
|
||||
- [ ] Open milestone for next version
|
||||
- [ ] If critical: pin discussion or post in `news.json` for in-app banner
|
||||
|
||||
## Rollback
|
||||
|
||||
If release has critical issue:
|
||||
|
||||
1. `gh release edit vX.Y.Z --prerelease` (marks as not latest)
|
||||
2. `git tag -d vX.Y.Z && git push --delete origin vX.Y.Z` (only if not yet adopted by users)
|
||||
3. Or: hotfix on `release/vX.Y.0` → patch release `vX.Y.(Z+1)`
|
||||
4. Communicate in GitHub Discussions and Discord immediately
|
||||
|
||||
## Hard Rules
|
||||
|
||||
- Never commit directly to `main`
|
||||
- Never use `git push --force` to `main` or `release/*` branches
|
||||
- Never skip Husky hooks (`--no-verify`)
|
||||
- Never commit secrets, credentials, or `.env` files
|
||||
- Coverage must stay ≥75/75/75/70 (statements/lines/functions/branches)
|
||||
- Always include or update tests when changing production code in `src/`, `open-sse/`, `electron/`, or `bin/`
|
||||
|
||||
## Automated Sync Check
|
||||
|
||||
Run the docs sync guard locally before opening a PR:
|
||||
|
||||
```bash
|
||||
npm run check:docs-sync
|
||||
|
||||
@@ -1,140 +1,132 @@
|
||||
# 🛡️ Resilience Guide — OmniRoute
|
||||
# Resilience Guide
|
||||
|
||||
> How OmniRoute keeps your AI coding workflow running when providers fail.
|
||||
OmniRoute has three distinct but related resilience mechanisms. Each has a different scope and purpose. Keep them separate when debugging routing behavior.
|
||||
|
||||
## Overview
|
||||
## 1. Provider Circuit Breaker
|
||||
|
||||
OmniRoute implements a multi-layered resilience system that ensures zero downtime:
|
||||
**Scope:** entire provider (e.g., `glm`, `openai`, `anthropic`).
|
||||
|
||||
```
|
||||
Client Request
|
||||
→ Rate Limit Check (per-IP, per-connection)
|
||||
→ Combo Routing (13 strategies)
|
||||
→ Connection Selection (P2C, round-robin, etc.)
|
||||
→ Request Queue & Pacing
|
||||
→ Execute (provider-specific executor)
|
||||
→ On Failure:
|
||||
→ Connection Cooldown (exponential backoff)
|
||||
→ Circuit Breaker (provider-level)
|
||||
→ Wait For Cooldown (auto-retry)
|
||||
→ Next Combo Target (fallback chain)
|
||||
→ Response
|
||||
```
|
||||
**Purpose:** stop sending traffic to a provider that is repeatedly failing at the upstream/service level.
|
||||
|
||||
**Implementation:**
|
||||
|
||||
- Core class: `src/shared/utils/circuitBreaker.ts`
|
||||
- Wiring: `src/sse/handlers/chatHelpers.ts`, `src/sse/handlers/chat.ts`
|
||||
- Status API: `GET /api/monitoring/health`
|
||||
- Reset API: `POST /api/resilience/reset`
|
||||
- Wrappers: `open-sse/services/accountFallback.ts`
|
||||
- DB table: `domain_circuit_breakers`
|
||||
|
||||
**States:**
|
||||
|
||||
- `CLOSED` — normal traffic allowed
|
||||
- `OPEN` — provider temporarily blocked; combo routing skips it
|
||||
- `HALF_OPEN` — reset timeout elapsed; probe request allowed
|
||||
|
||||
**Defaults (`open-sse/config/constants.ts`):**
|
||||
|
||||
| Class | Threshold | Reset timeout |
|
||||
| ------- | ---------- | ------------- |
|
||||
| OAuth | 3 failures | 60s |
|
||||
| API-key | 5 failures | 30s |
|
||||
| Local | 2 failures | 15s |
|
||||
|
||||
**Trip codes:** only provider-level statuses `[408, 500, 502, 503, 504]`. Do NOT trip for account-level errors (most 401/403/429 — those belong to cooldown or lockout).
|
||||
|
||||
**Lazy recovery:** when `OPEN` expires, `getStatus()`, `canExecute()`, `getRetryAfterMs()` refresh state to `HALF_OPEN`. No background timer needed.
|
||||
|
||||
---
|
||||
|
||||
## Request Queue & Pacing
|
||||
## 2. Connection Cooldown
|
||||
|
||||
Per-connection request buckets smooth bursts before they hit upstream rate caps.
|
||||
**Scope:** single provider connection/account/key.
|
||||
|
||||
Configure in `Dashboard → Settings → Resilience`:
|
||||
**Purpose:** skip one bad key while other connections for the same provider keep serving.
|
||||
|
||||
| Setting | Default | Description |
|
||||
| --------------- | ------- | ------------------------------------ |
|
||||
| Queue Size | `10` | Max queued requests per connection |
|
||||
| Pacing Interval | `0ms` | Minimum gap between requests |
|
||||
| Max Concurrent | `5` | Simultaneous requests per connection |
|
||||
**Implementation:**
|
||||
|
||||
- Mark unavailable: `src/sse/services/auth.ts::markAccountUnavailable()`
|
||||
- Selection: `getProviderCredentials*` in same file
|
||||
- Cooldown calc: `open-sse/services/accountFallback.ts::checkFallbackError()`
|
||||
- Settings: `src/lib/resilience/settings.ts`
|
||||
|
||||
**Fields per connection:**
|
||||
|
||||
- `rateLimitedUntil` — timestamp until cooldown expires
|
||||
- `testStatus: "unavailable"`
|
||||
- `lastError`, `lastErrorType`, `errorCode`
|
||||
- `backoffLevel` — exponential backoff counter
|
||||
|
||||
**Default cooldowns:**
|
||||
|
||||
- OAuth base: 5s
|
||||
- API-key base: 3s
|
||||
- API-key 429: prefers upstream `Retry-After`/reset headers/parseable reset text
|
||||
- Backoff: `baseCooldownMs * 2 ** failureIndex`
|
||||
|
||||
**Anti-thundering-herd guard:** prevents concurrent failures from over-extending cooldown or double-incrementing `backoffLevel`.
|
||||
|
||||
**Terminal states (NOT cooldowns):**
|
||||
|
||||
- `banned`
|
||||
- `expired`
|
||||
- `credits_exhausted`
|
||||
|
||||
These persist until credentials change or an operator resets them. Do not overwrite terminal states with transient cooldown state.
|
||||
|
||||
**Lazy recovery:** when `rateLimitedUntil` is past, connection becomes eligible again. On successful use, `clearAccountError()` clears all error fields.
|
||||
|
||||
---
|
||||
|
||||
## Connection Cooldown
|
||||
## 3. Model Lockout
|
||||
|
||||
A single connection cools down after retryable failures. Features:
|
||||
**Scope:** provider + connection + model triple.
|
||||
|
||||
- **Exponential Backoff** — progressively longer cooldowns after each failure
|
||||
- **`Retry-After` Header Support** — respects upstream hints
|
||||
- **Configurable Base/Max** — tune cooldown duration per use case
|
||||
- **Auto-Recovery** — connection automatically becomes available after cooldown expires
|
||||
**Purpose:** avoid disabling a whole connection when only one model is unavailable or quota-limited.
|
||||
|
||||
**Examples:**
|
||||
|
||||
- Per-model quota providers returning 429
|
||||
- Local providers returning 404 for one missing model
|
||||
- Provider-specific mode/model permission failures (e.g., Grok modes)
|
||||
|
||||
**Implementation:** `open-sse/services/accountFallback.ts` — `lockModel()`, `clearModelLock()`, `getAllModelLockouts()`.
|
||||
|
||||
### Model Cooldowns Dashboard (v3.8.0)
|
||||
|
||||
UI: Settings → Model Cooldowns (`src/app/(dashboard)/dashboard/settings/components/ModelCooldownsCard.tsx`)
|
||||
|
||||
Lists active lockouts with: provider, connection, model, reason, expiresAt. Operators can manually re-enable a model from the card.
|
||||
|
||||
**REST API:**
|
||||
|
||||
- `GET /api/resilience/model-cooldowns` — list active lockouts
|
||||
- `DELETE /api/resilience/model-cooldowns` — manual re-enable. Body: `{provider, connection, model}`. Auth: management.
|
||||
|
||||
---
|
||||
|
||||
## Circuit Breaker
|
||||
## Other Resilience Features
|
||||
|
||||
Provider-level protection against cascading failures:
|
||||
|
||||
1. **Connection-scoped `429` rate limits** stay in Connection Cooldown (don't trip the breaker)
|
||||
2. **Provider-wide transient errors** (5xx, network timeouts) increment the failure counter
|
||||
3. **Breaker trips** only after fallback is exhausted AND the provider still fails
|
||||
4. **Recovery** — breaker automatically moves to half-open state after timeout, tests with probe request
|
||||
|
||||
Configure thresholds in `Dashboard → Settings → Resilience`.
|
||||
- **14 routing strategies** (priority, weighted, round-robin, context-relay, fill-first, p2c, random, least-used, cost-optimized, reset-aware, strict-random, auto, lkgp, context-optimized) — see [AUTO-COMBO.md](./AUTO-COMBO.md).
|
||||
- **Reset-aware routing** (v3.8.0) — prioritizes connections by quota reset time.
|
||||
- **Background mode degradation** — Responses API `background: true` degraded to sync with warning.
|
||||
- **Dynamic tool limit detection** — backs off providers when tool count limits hit.
|
||||
|
||||
---
|
||||
|
||||
## Wait For Cooldown
|
||||
## Debugging
|
||||
|
||||
Instead of immediately failing when all connections are in cooldown, OmniRoute can wait for the earliest connection to expire and retry:
|
||||
|
||||
- **Automatic** — server waits for the earliest cooldown to expire
|
||||
- **Transparent** — client sees a slightly delayed response instead of an error
|
||||
- **Configurable** — enable/disable per combo or globally
|
||||
- All keys for a provider skipped → check both circuit breaker state AND each connection's `rateLimitedUntil`/`testStatus`.
|
||||
- Provider permanently excluded after reset window → code reading raw `state` instead of `getStatus()`/`canExecute()`.
|
||||
- One key fails, others should work → prefer connection cooldown over circuit breaker.
|
||||
- Only one model fails → prefer model lockout over connection cooldown.
|
||||
- State should self-recover but doesn't → check for future timestamp + read path that refreshes expired state. Permanent statuses require manual changes.
|
||||
|
||||
---
|
||||
|
||||
## Anti-Thundering Herd
|
||||
## TLS Fingerprinting & Stealth
|
||||
|
||||
When multiple concurrent requests hit a failing provider simultaneously:
|
||||
|
||||
- **Mutex Protection** — only one retry attempt at a time per connection
|
||||
- **Semaphore** — limits concurrent retry storms across connections
|
||||
- **Deduplication** — identical requests within 5s window are deduplicated
|
||||
|
||||
---
|
||||
|
||||
## Combo Fallback Chains
|
||||
|
||||
The primary resilience mechanism. Configure in `Dashboard → Combos`:
|
||||
|
||||
```txt
|
||||
Combo: "always-on"
|
||||
1. cc/claude-opus-4-7 ← Primary (subscription)
|
||||
2. cx/gpt-5.2-codex ← Secondary (subscription)
|
||||
3. glm/glm-4.7 ← Cheap backup ($0.5/1M)
|
||||
4. if/kimi-k2-thinking ← Free fallback (unlimited)
|
||||
```
|
||||
|
||||
When provider #1 fails (quota, rate, or health), OmniRoute automatically routes to #2, then #3, then #4 — with zero manual intervention.
|
||||
|
||||
### 13 Routing Strategies
|
||||
|
||||
| Strategy | Description |
|
||||
| ------------------- | ---------------------------------- |
|
||||
| `priority` | First available in order |
|
||||
| `weighted` | Weighted distribution |
|
||||
| `fill-first` | Fill primary before moving |
|
||||
| `round-robin` | Rotate through all targets |
|
||||
| `p2c` | Power-of-two choices (quota-aware) |
|
||||
| `random` | Random selection |
|
||||
| `least-used` | Least recently used |
|
||||
| `cost-optimized` | Cheapest available |
|
||||
| `strict-random` | True random (no tracking) |
|
||||
| `auto` | OmniRoute selects based on context |
|
||||
| `lkgp` | Last Known Good Provider |
|
||||
| `context-optimized` | Best for current context window |
|
||||
| `context-relay` | Session handoff during rotation |
|
||||
|
||||
---
|
||||
|
||||
## TLS Fingerprint Spoofing
|
||||
|
||||
OmniRoute makes proxied traffic look like legitimate browser/CLI requests:
|
||||
|
||||
- **Browser-like TLS** via `wreq-js` — prevents bot detection
|
||||
- **CLI Fingerprint Matching** — reorders headers and body fields to match native CLI binary signatures (Claude Code, Codex, etc.)
|
||||
- **Proxy IP Preservation** — stealth features work on top of proxy IP masking
|
||||
|
||||
---
|
||||
|
||||
## Health Dashboard
|
||||
|
||||
Monitor all resilience components in real-time at `Dashboard → Health`:
|
||||
|
||||
- **Uptime** — server uptime and last restart
|
||||
- **Provider Breaker States** — open/closed/half-open per provider
|
||||
- **Connection Cooldowns** — active cooldowns with expiry times
|
||||
- **Cache Stats** — signature + semantic cache hit rates
|
||||
- **Lockouts** — API key lockouts and IP bans
|
||||
- **Latency** — p50/p95/p99 percentiles
|
||||
Provider-specific stealth (JA3/JA4, CCH, obfuscation) is separately documented — see [STEALTH_GUIDE.md](./STEALTH_GUIDE.md).
|
||||
|
||||
---
|
||||
|
||||
|
||||
282
docs/SKILLS.md
Normal file
282
docs/SKILLS.md
Normal file
@@ -0,0 +1,282 @@
|
||||
# Skills Framework
|
||||
|
||||
> **Source of truth:** `src/lib/skills/` and `src/app/api/skills/`
|
||||
> **Last updated:** 2026-05-13 — v3.8.0
|
||||
|
||||
OmniRoute exposes an extensible Skills framework that lets language models (and operators) compose reusable capabilities — from filesystem reads and HTTP requests to sandboxed code execution and curated marketplace skills.
|
||||
|
||||
A skill is a versioned, schema-defined unit of work. OmniRoute can inject skills as tool definitions into outbound requests, intercept tool calls coming back from the model, run the matching handler, and feed the result back to the model so the conversation can continue. The model never sees the implementation — only the tool interface.
|
||||
|
||||
---
|
||||
|
||||
## Concepts
|
||||
|
||||
### Skill Sources
|
||||
|
||||
Three sources of skills coexist in the same registry:
|
||||
|
||||
1. **Built-in skills** (`src/lib/skills/builtins.ts`) — shipped with OmniRoute. Cover the common cases:
|
||||
- `file_read`, `file_write` — per-API-key sandbox workspace under `<DATA_DIR>/skills/workspaces/<hashed-key>/`
|
||||
- `http_request` — outbound HTTP through `safeOutboundFetch` with `guard: "public-only"`
|
||||
- `web_search` — pluggable search provider with caching (`executeWebSearch`)
|
||||
- `eval_code` — Docker-sandboxed `node` or `python` execution
|
||||
- `execute_command` — Docker-sandboxed shell command
|
||||
- `browser` — Playwright-backed scaffolding, disabled by default (`builtin/browser.ts`)
|
||||
2. **SkillsMP** (the OmniRoute Marketplace) — fetched from `https://skillsmp.com/api/v1/skills/search`. Requires `skillsmpApiKey` in Settings.
|
||||
3. **SkillsSH** (`skills.sh` community catalog) — fetched from `https://skills.sh/api/search`. No auth needed; SKILL.md content pulled from GitHub raw.
|
||||
|
||||
A single "active provider" controls which catalog the dashboard installs from (`src/lib/skills/providerSettings.ts`). Switch it under **Settings → Memory & Skills**. Default: `skillsmp`.
|
||||
|
||||
### Skill Identity
|
||||
|
||||
Skills are keyed by `name@version` in the in-memory registry (`src/lib/skills/registry.ts`). Version must be semver (`^\d+\.\d+\.\d+$`). `resolveVersion()` understands `^`, `~`, `>`, `>=`, `<`, `<=`, `==`, and exact-match constraints.
|
||||
|
||||
### Skill Mode
|
||||
|
||||
Each skill has a runtime mode that controls when it is injected:
|
||||
|
||||
| Mode | Behavior |
|
||||
| ------ | ------------------------------------------------------------------------------------------ |
|
||||
| `on` | Always injected as a tool definition |
|
||||
| `off` | Never injected, never executable |
|
||||
| `auto` | Scored against the incoming request; injected only if score ≥ `AUTO_MIN_SCORE` (default 3) |
|
||||
|
||||
`auto` is the default for marketplace-installed skills. `enabled=true` and `mode="off"` together mean "registered but inactive" — toggling `enabled` via the legacy column also bumps `mode` so older codepaths stay consistent (`src/app/api/skills/[id]/route.ts`).
|
||||
|
||||
### Status (executions)
|
||||
|
||||
Skill executions are tracked in the `skill_executions` table with the following statuses (`src/lib/skills/types.ts`):
|
||||
|
||||
```ts
|
||||
enum SkillStatus {
|
||||
PENDING = "pending",
|
||||
RUNNING = "running",
|
||||
SUCCESS = "success",
|
||||
ERROR = "error",
|
||||
TIMEOUT = "timeout",
|
||||
}
|
||||
```
|
||||
|
||||
### Registry Cache
|
||||
|
||||
`SkillRegistry` is a singleton with a 60-second TTL cache (`registry.ts:14`). `loadFromDatabase()` is idempotent and dedupes concurrent calls via `pendingLoad`. Any write (`register`/`unregister`/`unregisterById`) invalidates the cache. Look up versions via `getSkillVersions(name)` and `resolveVersion(name, constraint)`.
|
||||
|
||||
### Provider-Aware Injection
|
||||
|
||||
`injectSkills()` in `src/lib/skills/injection.ts` is the entry point that turns registered skills into provider-specific tool definitions:
|
||||
|
||||
- **OpenAI** — `{ type: "function", function: { name, description, parameters } }`
|
||||
- **Anthropic** — `{ name, description, input_schema }`
|
||||
- **Google (Gemini)** — `{ name, description, parameters }`
|
||||
|
||||
The tool name is encoded as `name@version` so the handler can pick the right version when the model calls it back.
|
||||
|
||||
### AUTO Scoring
|
||||
|
||||
When `mode="auto"`, each candidate skill is scored against the request context (`scoreAutoSkill()` in `injection.ts`):
|
||||
|
||||
| Signal | Points |
|
||||
| ---------------------------------------------- | ------------ |
|
||||
| Skill name appears verbatim in context | +6 |
|
||||
| Each name token matches a context token | +2 |
|
||||
| Each tag substring matches context | +3 |
|
||||
| Each description token matches context | +1 |
|
||||
| Background reason matches a name token | +2 per token |
|
||||
| Background reason matches a tag | +2 per token |
|
||||
| Provider hint in tags matches request provider | +2 / −2 |
|
||||
|
||||
Top `AUTO_MAX_SKILLS = 5` skills with `score >= AUTO_MIN_SCORE = 3` are injected. Ties are broken by `installCount` (desc), then alphabetical name (`injection.ts:225-235`).
|
||||
|
||||
### Tool Call Interception
|
||||
|
||||
`handleToolCallExecution()` in `src/lib/skills/interception.ts` is invoked by the chat handler after the upstream returns a tool-calling response:
|
||||
|
||||
1. `extractToolCalls()` reads provider-specific shapes (OpenAI `tool_calls` / Responses `function_call`, Anthropic `tool_use`, Gemini `functionCalls`).
|
||||
2. Built-in tool aliases (e.g. `omniroute_web_search` → `web_search`) are resolved first. Built-in handlers run inline.
|
||||
3. Anything else routes through `skillExecutor.execute(name@version, args, { apiKeyId, sessionId })`.
|
||||
4. Results are spliced back into the response — `tool_results`, `function_call_output` items, or Anthropic `tool_result` blocks as appropriate.
|
||||
|
||||
`customSkillExecutionEnabled` in the execution context can be set to `false` to allow only built-in interception (used by request paths that explicitly disable user-defined handlers).
|
||||
|
||||
---
|
||||
|
||||
## Docker Sandbox
|
||||
|
||||
Non-builtin code paths (`eval_code`, `execute_command`) run inside Docker via `SandboxRunner` (`src/lib/skills/sandbox.ts`). Every container is launched with:
|
||||
|
||||
```
|
||||
--rm --network none|bridge --cap-drop ALL
|
||||
--security-opt no-new-privileges --pids-limit 100
|
||||
--cpus <cpuLimit/1000> --memory <memoryLimit>m
|
||||
--tmpfs /tmp:rw,noexec,nosuid,size=64m
|
||||
--tmpfs /workspace:rw,noexec,nosuid,size=64m
|
||||
--read-only (when readOnly=true)
|
||||
```
|
||||
|
||||
Defaults (`SandboxRunner.DEFAULT_CONFIG`):
|
||||
|
||||
| Field | Default | Notes |
|
||||
| ---------------- | --------------- | ---------------------------------------------------- |
|
||||
| `cpuLimit` | 100 (= 0.1 CPU) | Divided by 1000 before passing to `--cpus` |
|
||||
| `memoryLimit` | 256 MB | Hard limit |
|
||||
| `timeout` | 30000 ms | Soft kill via `SIGTERM` + `docker kill` |
|
||||
| `networkEnabled` | `false` | Becomes `--network none` |
|
||||
| `readOnly` | `true` | Root FS read-only; `/tmp` and `/workspace` are tmpfs |
|
||||
|
||||
`SandboxRunner.kill(id)` and `killAll()` are exposed for shutdown; running containers are tracked in `runningContainers: Map<string, ChildProcess>`.
|
||||
|
||||
### Sandbox Env Vars
|
||||
|
||||
Configured via `process.env` in `src/lib/skills/builtins.ts`:
|
||||
|
||||
| Env Var | Default | Purpose |
|
||||
| --------------------------------- | ---------------- | ------------------------------------------------------------------ |
|
||||
| `SKILLS_MAX_FILE_BYTES` | `1048576` (1 MB) | Cap for `file_read` and `file_write` |
|
||||
| `SKILLS_MAX_HTTP_RESPONSE_BYTES` | `256000` | Cap for `http_request` response body |
|
||||
| `SKILLS_MAX_SANDBOX_OUTPUT_CHARS` | `100000` | Cap for stdout/stderr returned to the caller |
|
||||
| `SKILLS_SANDBOX_TIMEOUT_MS` | `10000` | Default timeout for sandboxed commands; capped at 60 s |
|
||||
| `SKILLS_SANDBOX_NETWORK_ENABLED` | `false` | Master gate for egress. Set `1` or `true` to allow per-call opt-in |
|
||||
| `SKILLS_ALLOWED_SANDBOX_IMAGES` | (see below) | Comma-separated allowlist of Docker images |
|
||||
|
||||
Default allowed images: `alpine:3.20`, `node:22-alpine`, `python:3.12-alpine`. Any additions via `SKILLS_ALLOWED_SANDBOX_IMAGES` are merged with the defaults; unknown images are rejected by `normalizeImage()`.
|
||||
|
||||
> Note: there is no separate `SKILLS_EXECUTION_TIMEOUT_MS` env var. The non-sandbox handler timeout is hard-coded to 30 s in `SkillExecutor` (`executor.ts:13`) but can be overridden at runtime via `skillExecutor.setTimeout(ms)`.
|
||||
|
||||
### Workspace Isolation
|
||||
|
||||
`file_read` and `file_write` resolve every path relative to a per-API-key workspace at `<DATA_DIR>/skills/workspaces/<sha256(apiKeyId).slice(0,24)>/`. Path traversal (`..`) and forbidden segments (`.env`, `.git`, `.ssh`, `.omniroute`, `.codex`, `secrets`) are rejected before any disk I/O.
|
||||
|
||||
### HTTP Hardening
|
||||
|
||||
`http_request` (`builtins.ts:257`):
|
||||
|
||||
- Method allowlist: `GET, HEAD, POST, PUT, PATCH, DELETE`
|
||||
- Blocked outbound headers: `host, connection, content-length, cookie, set-cookie, authorization, proxy-authorization`
|
||||
- Redirects disabled (`allowRedirect: false`)
|
||||
- Routed through `safeOutboundFetch` with `guard: "public-only"` (private/loopback ranges blocked)
|
||||
- Response truncated at `SKILLS_MAX_HTTP_RESPONSE_BYTES`; client sees `truncated: true`
|
||||
|
||||
---
|
||||
|
||||
## Hybrid Executor (preview)
|
||||
|
||||
`src/lib/skills/hybrid.ts` defines a `HybridExecutor` that decides between `direct` (in-process) and `sandbox` execution per call, with an `autoUpgrade` retry path on timeout/memory errors. The wired-in `directExecutor` / `sandboxRunner` implementations are stubs (`executeDirect`, `executeInSandbox` return placeholder objects) — treat this module as a contract under construction. Real execution still goes through `skillExecutor` + `SandboxRunner`.
|
||||
|
||||
---
|
||||
|
||||
## Storage
|
||||
|
||||
Schema lives in two migrations:
|
||||
|
||||
- `src/lib/db/migrations/016_create_skills.sql` — base `skills` and `skill_executions` tables, with indexes on `(api_key_id, name)` and `(skill_id, status, created_at)`.
|
||||
- `src/lib/db/migrations/027_skill_mode_and_metadata.sql` — adds `mode`, `source_provider`, `tags` (JSON), `install_count` to `skills`.
|
||||
|
||||
`skill_executions.status` is constrained at the database level: `CHECK(status IN ('pending', 'running', 'success', 'error', 'timeout'))`.
|
||||
|
||||
---
|
||||
|
||||
## REST API
|
||||
|
||||
All endpoints live under `src/app/api/skills/`. Management endpoints (`/api/skills`, `/api/skills/[id]`, `/api/skills/install`) require **management auth** via `requireManagementAuth()`. The marketplace/install flows use the lighter `isAuthenticated()` (session or API key).
|
||||
|
||||
| Endpoint | Method | Purpose |
|
||||
| --------------------------------- | ------ | ------------------------------------------------------------------------ | --- | ------------------------ | -------- | ------------------ |
|
||||
| `/api/skills` | GET | List registered skills. Supports `?q=`, `?mode=on | off | auto`, `?source=skillsmp | skillssh | local`, pagination |
|
||||
| `/api/skills/[id]` | PUT | Update `enabled` or `mode` |
|
||||
| `/api/skills/[id]` | DELETE | Unregister by id |
|
||||
| `/api/skills/install` | POST | Install a custom skill (handler code + schema) |
|
||||
| `/api/skills/marketplace` | GET | Search the SkillsMP catalog (returns popular defaults when `q` is empty) |
|
||||
| `/api/skills/marketplace/install` | POST | Install a SkillsMP skill (requires active provider = `skillsmp`) |
|
||||
| `/api/skills/skillssh` | GET | Search the skills.sh catalog (`?q=&limit=`, capped at 100) |
|
||||
| `/api/skills/skillssh/install` | POST | Install a skills.sh skill (requires active provider = `skillssh`) |
|
||||
| `/api/skills/executions` | GET | Paginated execution history (`?apiKeyId=`) |
|
||||
| `/api/skills/executions` | POST | Execute a registered skill ad-hoc |
|
||||
|
||||
The `POST /api/skills/executions` endpoint returns HTTP `503` with `{ error: "Skills execution is disabled..." }` when `settings.skillsEnabled === false` (`executor.ts:42-45`). Operators can flip the master switch from **Settings → AI**.
|
||||
|
||||
### Example: install a custom skill
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/api/skills/install \
|
||||
-H "Authorization: Bearer $OMNIROUTE_MGMT_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "reverse-text",
|
||||
"version": "1.0.0",
|
||||
"description": "Reverses a string",
|
||||
"schema": {
|
||||
"input": { "type": "object", "properties": { "text": { "type": "string" } }, "required": ["text"] },
|
||||
"output": { "type": "object", "properties": { "reversed": { "type": "string" } } }
|
||||
},
|
||||
"handlerCode": "echo-handler",
|
||||
"apiKeyId": "your-api-key-id"
|
||||
}'
|
||||
```
|
||||
|
||||
The `handlerCode` string is a **handler name lookup** — not executable code. The executor maps it via `skillExecutor.registerHandler(name, fn)` (`executor.ts:25`). Marketplace installs store the SKILL.md text in this field as documentation and route execution through model-generated tool calls. Arbitrary user-supplied source is not eval'd.
|
||||
|
||||
---
|
||||
|
||||
## MCP Tools
|
||||
|
||||
Four MCP tools wrap the skills surface (`open-sse/mcp-server/tools/skillTools.ts`). They are auto-registered when the MCP server boots.
|
||||
|
||||
| Tool | Description |
|
||||
| ----------------------------- | ------------------------------------------------------------ |
|
||||
| `omniroute_skills_list` | List skills, optional filters: `apiKeyId`, `name`, `enabled` |
|
||||
| `omniroute_skills_enable` | Enable/disable a skill by `skillId` |
|
||||
| `omniroute_skills_execute` | Execute a skill with an input payload |
|
||||
| `omniroute_skills_executions` | Recent execution history (default 50, max 100) |
|
||||
|
||||
See [MCP-SERVER.md](./MCP-SERVER.md) for transport setup and scope assignments.
|
||||
|
||||
---
|
||||
|
||||
## A2A Integration
|
||||
|
||||
`src/lib/skills/a2a.ts` exports the `memory_aware_routing` A2A skill descriptor and a `registerA2ASkill(registry)` helper. Custom A2A skills live in `src/lib/a2a/skills/` and are dispatched via `A2A_SKILL_HANDLERS` (`src/lib/a2a/taskExecution.ts`). See [A2A-SERVER.md](./A2A-SERVER.md) for the full task lifecycle.
|
||||
|
||||
---
|
||||
|
||||
## Adding a New Built-in Skill
|
||||
|
||||
1. **Define the handler** in `src/lib/skills/builtins.ts` (or a sibling file under `src/lib/skills/builtin/`). Signature: `(input, { apiKeyId, sessionId }) => Promise<output>`.
|
||||
2. **Sandboxed code path?** Call `sandboxRunner.run(image, command, env, sandboxConfig({...}))`. Use `normalizeImage()` against the allowlist.
|
||||
3. **Filesystem path?** Always pass through `resolveWorkspacePath(input, context)` before touching disk.
|
||||
4. **Network call?** Use `safeOutboundFetch` with `guard: "public-only"`; sanitize headers via `sanitizeHeaders()`.
|
||||
5. **Register** by adding the entry to `builtinSkills` (or calling `registerBrowserSkill(executor)`-style at boot).
|
||||
6. **Wire built-in tool aliases** (optional) in `BUILTIN_TOOL_ALIASES` (`interception.ts:23`) if the upstream model emits a different name.
|
||||
7. **Tests** in `src/lib/skills/__tests__/` (Vitest).
|
||||
|
||||
---
|
||||
|
||||
## Adding a Custom (Non-Builtin) Skill
|
||||
|
||||
1. Register the handler at process startup:
|
||||
```ts
|
||||
skillExecutor.registerHandler("my-handler", async (input, ctx) => { ... });
|
||||
```
|
||||
2. Insert the skill via `POST /api/skills/install` (the `handlerCode` field must match the registered handler name).
|
||||
3. Toggle `mode` to `on` or `auto` via `PUT /api/skills/[id]`.
|
||||
|
||||
---
|
||||
|
||||
## Operational Tips
|
||||
|
||||
- **Master switch:** `settings.skillsEnabled = false` blocks all execution and returns HTTP `503` on `/api/skills/executions`. The registry continues to load.
|
||||
- **Lock down egress:** keep `SKILLS_SANDBOX_NETWORK_ENABLED` unset (default) for fully air-gapped sandboxing. Per-call `networkEnabled: true` still requires the master gate.
|
||||
- **Allow specific images:** set `SKILLS_ALLOWED_SANDBOX_IMAGES="myorg/sandbox:1.0,node:22-alpine"` to extend the allowlist.
|
||||
- **Audit executions:** `/dashboard/skills/executions` and `omniroute_skills_executions` both query `skill_executions`. Successful runs include `durationMs`; failures include `errorMessage`.
|
||||
- **Cache invalidation:** call `skillRegistry.invalidateCache()` after manual DB edits; otherwise wait 60 s.
|
||||
- **Anonymous workspace:** when `apiKeyId` is empty, all calls hash to the same `"anonymous"` workspace — share-aware code should always pass a real key.
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [MCP-SERVER.md](./MCP-SERVER.md) — MCP tool registration and transports
|
||||
- [A2A-SERVER.md](./A2A-SERVER.md) — A2A task lifecycle and skill dispatch
|
||||
- [USER_GUIDE.md](./USER_GUIDE.md#-skills-system) — user-facing introduction
|
||||
- [ARCHITECTURE.md](./ARCHITECTURE.md) — request pipeline and component map
|
||||
- Source: `src/lib/skills/`, `src/app/api/skills/`, `open-sse/mcp-server/tools/skillTools.ts`
|
||||
- Tests: `src/lib/skills/__tests__/integration.test.ts`
|
||||
244
docs/STEALTH_GUIDE.md
Normal file
244
docs/STEALTH_GUIDE.md
Normal file
@@ -0,0 +1,244 @@
|
||||
# Stealth Guide
|
||||
|
||||
> **Source of truth:** `open-sse/utils/tlsClient.ts`, `open-sse/services/{chatgptTlsClient,claudeCodeCCH,claudeCodeFingerprint,claudeCodeObfuscation,claudeCodeCompatible,antigravityObfuscation}.ts`, `open-sse/config/cliFingerprints.ts`, `src/mitm/`
|
||||
> **Last updated:** 2026-05-13 — v3.8.0
|
||||
> **Audience:** Engineers maintaining provider-specific stealth integrations.
|
||||
|
||||
OmniRoute integrates with providers whose edges actively fingerprint non-official clients (TLS JA3/JA4, header ordering, JSON body shape, integrity tokens). This page documents the stealth surfaces OmniRoute exposes and where they are implemented.
|
||||
|
||||
## Legal and Ethical Notice
|
||||
|
||||
Stealth features exist so OmniRoute can act as a compatibility layer between user-owned official accounts (Claude Code CLI, ChatGPT Desktop/Web, Antigravity, Cursor, etc.) and OmniRoute's unified API. They are **not** for evading fraud detection, sharing credentials, or violating provider Terms of Service. The maintainers expect operators to comply with the upstream ToS they signed when creating accounts.
|
||||
|
||||
---
|
||||
|
||||
## TLS Fingerprinting Layer
|
||||
|
||||
### `open-sse/utils/tlsClient.ts` — wreq-js (Chrome 124)
|
||||
|
||||
Lazy-loaded `wreq-js` session that impersonates **Chrome 124 on macOS**. Used as a generic JA3/JA4 wrapper for upstreams behind Cloudflare. Falls back to native fetch when `wreq-js` is not installed (`available = false`).
|
||||
|
||||
- Singleton session: `browser: "chrome_124", os: "macos"`
|
||||
- Proxy resolution (priority): `HTTPS_PROXY` → `HTTP_PROXY` → `ALL_PROXY` (also lower-case)
|
||||
- Timeout: `TLS_CLIENT_TIMEOUT_MS` (inherits from `FETCH_TIMEOUT_MS`, default 600000)
|
||||
- `wreq-js` Response is fetch-compatible (`headers`, `text()`, `json()`, `clone()`, `body`).
|
||||
|
||||
### `open-sse/services/chatgptTlsClient.ts` — tls-client-node (Firefox 148)
|
||||
|
||||
Dedicated TLS impersonator for `chatgpt.com`. ChatGPT's Cloudflare config pins `cf_clearance` to JA3/JA4 + HTTP/2 SETTINGS frame ordering — undici's handshake gets `cf-mitigated: challenge` even with valid cookies.
|
||||
|
||||
- Profile: `firefox_148` (must match the Firefox 148 `User-Agent` sent)
|
||||
- Mode: `runtimeMode: "native"` (koffi-loaded shared library; avoids managed sidecar HTTP)
|
||||
- `withRandomTLSExtensionOrder: true`
|
||||
- `tlsFetchChatGpt(url, options)` supports streaming (writes body to temp file, tailed as `ReadableStream`)
|
||||
- Hang detection: `raceWithTimeout` + `TlsClientHangError` triggers `resetClientCache()` so the next call respawns the binding
|
||||
- Proxy resolution (priority): per-call `proxyUrl` → `OMNIROUTE_TLS_PROXY_URL` → `HTTPS_PROXY`/`HTTP_PROXY`/`ALL_PROXY` (the native binding does **not** read these envs itself; it must be threaded through)
|
||||
- Errors: `TlsClientUnavailableError` (binary missing), `TlsClientHangError` (binding deadlocked)
|
||||
|
||||
---
|
||||
|
||||
## Claude Code Stealth Bundle
|
||||
|
||||
When `cliCompatMode` is on, OmniRoute reshapes outgoing Claude requests so they are indistinguishable from `claude-cli` traffic. Three modules collaborate:
|
||||
|
||||
### `claudeCodeFingerprint.ts`
|
||||
|
||||
Computes the 3-char `cc_version` fingerprint embedded in the billing header:
|
||||
|
||||
```
|
||||
SHA256(SALT + msg[4] + msg[7] + msg[20] + version)[:3]
|
||||
```
|
||||
|
||||
- `FINGERPRINT_SALT = "59cf53e54c78"` (hardcoded; matches official client)
|
||||
- Inputs: chars at index 4, 7, 20 of the first user message text + version string
|
||||
- Output: 3-char hex prefix
|
||||
|
||||
### `claudeCodeCCH.ts` (Client Content Hash)
|
||||
|
||||
Server-side integrity check the official Claude Code CLI computes via Bun/Zig. OmniRoute reimplements with `xxhash-wasm`:
|
||||
|
||||
1. Serialize body with `cch=00000;` placeholder
|
||||
2. `xxhash64(bytes, seed) & 0xFFFFF`
|
||||
3. Zero-padded 5-char lowercase hex
|
||||
4. Replace `cch=00000;` with the computed token
|
||||
|
||||
Constants:
|
||||
|
||||
- Seed: `0x6e52736ac806831e`
|
||||
- Pattern: `/\bcch=([0-9a-f]{5});/`
|
||||
|
||||
### `claudeCodeObfuscation.ts`
|
||||
|
||||
Inserts a Unicode **zero-width joiner** (`U+200D`) after the first character of "sensitive" client names so upstream filters cannot grep them. Default word list:
|
||||
|
||||
```
|
||||
opencode, open-code, cline, roo-cline, roo_cline, cursor, windsurf,
|
||||
aider, continue.dev, copilot, avante, codecompanion
|
||||
```
|
||||
|
||||
Applied to: `system` blocks, all `messages[].content`, and `tools[].description` / `tools[].function.description`. Operator-overridable via `setSensitiveWords()`.
|
||||
|
||||
### `claudeCodeCompatible.ts` — `anthropic-compatible-cc-*` providers
|
||||
|
||||
For third-party Anthropic relays that only accept "real Claude Code" traffic:
|
||||
|
||||
- `CLAUDE_CODE_COMPATIBLE_USER_AGENT = "claude-cli/2.1.137 (external, sdk-cli)"`
|
||||
- `CLAUDE_CODE_COMPATIBLE_STAINLESS_PACKAGE_VERSION = "0.81.0"`
|
||||
- `CLAUDE_CODE_COMPATIBLE_STAINLESS_RUNTIME_VERSION = "v24.3.0"`
|
||||
- `anthropic-beta = "claude-code-20250219,interleaved-thinking-2025-05-14,effort-2025-11-24"`
|
||||
- `CONTEXT_1M_BETA_HEADER = "context-1m-2025-08-07"` (Opus/Sonnet 4.x family)
|
||||
- Default path: `/v1/messages?beta=true`
|
||||
|
||||
Sister modules in the same bundle:
|
||||
|
||||
- `claudeCodeConstraints.ts` — temperature + cache-control rules
|
||||
- `claudeCodeToolRemapper.ts` — tool-name remapping
|
||||
- `claudeCodeExtraRemap.ts` — extra payload normalization
|
||||
|
||||
---
|
||||
|
||||
## Antigravity Stealth
|
||||
|
||||
### `antigravityObfuscation.ts`
|
||||
|
||||
Same zero-width-joiner trick as Claude Code, but with an expanded word list that also masks: `claude code`, `claude-code`, `kilo code`, `kilocode`, **`omniroute`**. Mirrors ZeroGravity's `ZEROGRAVITY_SENSITIVE_WORDS` and CLIProxyAPI's cloak system.
|
||||
|
||||
### `antigravityHeaderScrub.ts`
|
||||
|
||||
Strips Stainless SDK markers (`x-stainless-lang`, `x-stainless-package-version`, `x-stainless-os`, `x-stainless-arch`, `x-stainless-runtime`, `x-stainless-runtime-version`, `x-stainless-timeout`, `x-stainless-retry-count`, `x-stainless-helper-method`) before forwarding.
|
||||
|
||||
---
|
||||
|
||||
## CLI Fingerprint Registry — `open-sse/config/cliFingerprints.ts`
|
||||
|
||||
Per-provider table that pins **exact** header ordering and JSON body field ordering captured from mitmproxy traces of the official CLIs. Currently registered: `codex`, `claude`, plus runtime-derived profiles in `providerHeaderProfiles.ts` for `antigravity`, `qwen`, `github`.
|
||||
|
||||
```ts
|
||||
interface CliFingerprint {
|
||||
headerOrder: string[]; // case-sensitive
|
||||
bodyFieldOrder: string[]; // top-level JSON keys
|
||||
userAgent?: string | (() => string);
|
||||
extraHeaders?: Record<string, string>;
|
||||
}
|
||||
```
|
||||
|
||||
Toggle per provider via env (see below). When disabled, headers/body keys appear in whatever order Node/JSON gave them — easy to fingerprint.
|
||||
|
||||
---
|
||||
|
||||
## MITM Proxy (Antigravity, Linux/macOS/Windows)
|
||||
|
||||
For CLIs whose binaries cannot be redirected via `OPENAI_BASE_URL`, OmniRoute runs a local TLS-terminating proxy. Endpoints live under `src/app/api/cli-tools/antigravity-mitm/`.
|
||||
|
||||
| Method | Endpoint | Purpose |
|
||||
| ------ | --------------------------------------- | ------------------------------------------------ |
|
||||
| GET | `/api/cli-tools/antigravity-mitm` | Status — running, pid, dnsConfigured, certExists |
|
||||
| POST | `/api/cli-tools/antigravity-mitm` | Start MITM (requires `apiKey` + `sudoPassword`) |
|
||||
| DELETE | `/api/cli-tools/antigravity-mitm` | Stop MITM |
|
||||
| GET | `/api/cli-tools/antigravity-mitm/alias` | List model aliases |
|
||||
| PUT | `/api/cli-tools/antigravity-mitm/alias` | Save model aliases for a tool |
|
||||
|
||||
Target intercepted host: **`daily-cloudcode-pa.googleapis.com`** (Antigravity's upstream).
|
||||
|
||||
### Start sequence (`src/mitm/manager.ts::startMitm`)
|
||||
|
||||
1. Generate self-signed cert via `selfsigned` (RSA-2048, SHA-256, 1y) — `cert/generate.ts`
|
||||
2. Install cert to system trust store — `cert/install.ts`
|
||||
3. Add hosts entry `127.0.0.1 daily-cloudcode-pa.googleapis.com` — `dns/dnsConfig.ts`
|
||||
4. Spawn `src/mitm/server.cjs` with `ROUTER_API_KEY` + `MITM_LOCAL_PORT` (default `443`)
|
||||
5. Persist PID to `<DATA_DIR>/mitm/.mitm.pid`
|
||||
|
||||
### Linux dynamic trust-store detection — `cert/install.ts`
|
||||
|
||||
`getLinuxCertConfig()` walks a priority list and picks the first existing directory:
|
||||
|
||||
| Distro family | Directory | Update command |
|
||||
| ------------------------ | ------------------------------------------- | ------------------------ |
|
||||
| Debian / Ubuntu | `/usr/local/share/ca-certificates` | `update-ca-certificates` |
|
||||
| Arch / CachyOS / Manjaro | `/etc/ca-certificates/trust-source/anchors` | `update-ca-trust` |
|
||||
| Fedora / RHEL / CentOS | `/etc/pki/ca-trust/source/anchors` | `update-ca-trust` |
|
||||
| openSUSE | `/etc/pki/trust/anchors` | `update-ca-certificates` |
|
||||
|
||||
Cert filename: `omniroute-mitm.crt`. Fingerprint match via `getCertFingerprint()` (SHA-1 of DER).
|
||||
|
||||
Additionally, `updateNssDatabases()` installs into per-user NSS DBs when `certutil` is available: `~/.pki/nssdb`, `~/snap/chromium/.../nssdb`, all Firefox profiles (including snap), under the nickname **`OmniRoute MITM Root CA`**.
|
||||
|
||||
### macOS / Windows
|
||||
|
||||
- **macOS:** `security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain`
|
||||
- **Windows:** elevated PowerShell → `certutil -addstore Root`
|
||||
|
||||
### Auth
|
||||
|
||||
All MITM endpoints require management auth (`requireCliToolsAuth`). The sudo password is cached in module scope (never `globalThis`) and cleared on `stopMitm()`.
|
||||
|
||||
---
|
||||
|
||||
## User-Agent Overrides — env vars (`.env.example` section 12)
|
||||
|
||||
| Variable | Default |
|
||||
| ------------------------ | --------------------------------------------- |
|
||||
| `CLAUDE_USER_AGENT` | `claude-cli/2.1.137 (external, cli)` |
|
||||
| `CODEX_USER_AGENT` | `codex-cli/0.130.0 (Windows 10.0.26200; x64)` |
|
||||
| `GITHUB_USER_AGENT` | `GitHubCopilotChat/0.45.1` |
|
||||
| `ANTIGRAVITY_USER_AGENT` | `antigravity/1.23.2 darwin/arm64` |
|
||||
| `KIRO_USER_AGENT` | `AWS-SDK-JS/3.0.0 kiro-ide/1.0.0` |
|
||||
| `QODER_USER_AGENT` | `Qoder-Cli` |
|
||||
| `QWEN_USER_AGENT` | `QwenCode/0.15.9 (linux; x64)` |
|
||||
| `CURSOR_USER_AGENT` | `Cursor/3.3` |
|
||||
| `GEMINI_CLI_USER_AGENT` | `google-api-nodejs-client/10.3.0` |
|
||||
|
||||
Consumed by `open-sse/executors/base.ts::buildHeaders()` via dynamic lookup. **Bump these when providers release new CLI versions** — stale UA strings start getting rejected as outdated clients.
|
||||
|
||||
## CLI Compatibility Mode Toggles (`.env.example` section 13)
|
||||
|
||||
| Variable | Effect |
|
||||
| -------------------------- | ------------------------------- |
|
||||
| `CLI_COMPAT_CODEX=1` | Codex fingerprint |
|
||||
| `CLI_COMPAT_CLAUDE=1` | claude-cli fingerprint |
|
||||
| `CLI_COMPAT_GITHUB=1` | GitHub Copilot Chat fingerprint |
|
||||
| `CLI_COMPAT_ANTIGRAVITY=1` | Antigravity fingerprint |
|
||||
| `CLI_COMPAT_KIRO=1` | Kiro |
|
||||
| `CLI_COMPAT_CURSOR=1` | Cursor |
|
||||
| `CLI_COMPAT_KIMI_CODING=1` | Kimi Coding |
|
||||
| `CLI_COMPAT_KILOCODE=1` | KiloCode |
|
||||
| `CLI_COMPAT_CLINE=1` | Cline |
|
||||
| `CLI_COMPAT_QWEN=1` | Qwen Code |
|
||||
| `CLI_COMPAT_ALL=1` | Enable all of the above |
|
||||
|
||||
The provider IP is **always preserved** — the toggle only reshapes the request wire image, it does not switch IP egress.
|
||||
|
||||
---
|
||||
|
||||
## Inbound Header Sanitization
|
||||
|
||||
OmniRoute scrubs inbound client headers before forwarding so a request that arrives from Cursor doesn't leak `User-Agent: Cursor/X.Y.Z` to a Claude upstream. See `src/shared/constants/upstreamHeaders.ts` for the denylist, kept in lockstep with the Zod schemas and unit tests.
|
||||
|
||||
---
|
||||
|
||||
## Updating Fingerprints When a Provider Rotates
|
||||
|
||||
1. Capture official CLI traffic with `mitmproxy` (TLS interception + dump)
|
||||
2. Extract JA3/JA4 and the literal header order
|
||||
3. Update the relevant `CLI_FINGERPRINTS[...]` entry
|
||||
4. Bump matching `*_USER_AGENT` default in `.env.example`
|
||||
5. If TLS handshake itself changed: update `chatgptTlsClient.ts::CHATGPT_PROFILE` or wreq-js `browser:` option
|
||||
6. Run `chatgptTlsClient.test.ts` and a manual canary against the live provider
|
||||
7. Ship in a patch release; document in `CHANGELOG.md`
|
||||
|
||||
---
|
||||
|
||||
## Tests
|
||||
|
||||
- `open-sse/services/__tests__/chatgptTlsClient.test.ts` — proxy resolution priority, abort handling, hang recovery
|
||||
- `tests/unit/anthropic-cache-fingerprint.test.ts` — fingerprint determinism
|
||||
- `tests/unit/chatgpt-web.test.ts` — end-to-end stealth path for ChatGPT
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [RESILIENCE_GUIDE.md](./RESILIENCE_GUIDE.md) — what happens when a stealth path gets a `403`
|
||||
- [TROUBLESHOOTING.md](./TROUBLESHOOTING.md)
|
||||
- [ENVIRONMENT.md](./ENVIRONMENT.md) — full env reference
|
||||
- [CLI-TOOLS.md](./CLI-TOOLS.md) — operator view of the MITM workflow
|
||||
281
docs/TUNNELS_GUIDE.md
Normal file
281
docs/TUNNELS_GUIDE.md
Normal file
@@ -0,0 +1,281 @@
|
||||
# Tunnels Guide
|
||||
|
||||
> **Source of truth:** `src/lib/{cloudflaredTunnel,ngrokTunnel,tailscaleTunnel}.ts`, `src/app/api/tunnels/`
|
||||
> **Last updated:** 2026-05-13 — v3.8.0
|
||||
|
||||
OmniRoute can expose its local server (`http://localhost:20128`) to the public
|
||||
internet via three tunnel backends. This is useful for:
|
||||
|
||||
- OAuth callbacks from cloud providers (Antigravity, Gemini, Cursor) that need a
|
||||
publicly reachable redirect URL.
|
||||
- Sharing your local instance with teammates without deploying a VM.
|
||||
- Mobile, remote, or cross-network testing.
|
||||
|
||||
All three backends are managed in-process — OmniRoute starts/stops the underlying
|
||||
binary or SDK from the dashboard or REST API. No reverse-proxy or systemd setup
|
||||
is required.
|
||||
|
||||
## Backends at a glance
|
||||
|
||||
| Backend | Persistence | Cost | Setup |
|
||||
| --------------------------- | ------------------------------------------------------ | ----------------- | ----------------------------------------------- |
|
||||
| **Cloudflare Quick Tunnel** | Ephemeral (URL changes each restart) | Free | Zero — auto-installs `cloudflared` |
|
||||
| **ngrok** | Stable while a paid plan or fixed domain is configured | Free tier + paid | Requires ngrok account + authtoken |
|
||||
| **Tailscale Funnel** | Stable per node within your tailnet | Free for personal | Requires Tailscale install + login + Funnel ACL |
|
||||
|
||||
The implementations live in `src/lib/cloudflaredTunnel.ts`,
|
||||
`src/lib/ngrokTunnel.ts`, and `src/lib/tailscaleTunnel.ts`. All three return a
|
||||
common-shaped `status` object with `phase`, `running`, `publicUrl`, `apiUrl`,
|
||||
`targetUrl`, and `lastError` fields, so the dashboard can render them uniformly.
|
||||
|
||||
## 1. Cloudflare Tunnel (Quick Tunnel)
|
||||
|
||||
`src/lib/cloudflaredTunnel.ts` runs `cloudflared tunnel --url
|
||||
http://localhost:<apiPort>` as a child process and parses the assigned
|
||||
`*.trycloudflare.com` URL from stdout.
|
||||
|
||||
Key behaviors:
|
||||
|
||||
- **Auto-install.** On first use, OmniRoute downloads the latest `cloudflared`
|
||||
binary from the official GitHub releases (managed install lives under
|
||||
`DATA_DIR/cloudflared/`). SHA256 of the downloaded asset is verified against the
|
||||
release manifest before execution.
|
||||
- **Quick-tunnel only.** The current implementation runs only the
|
||||
`--url`-style quick tunnel. Named/persistent tunnels (`cloudflared tunnel
|
||||
login` + `cloudflared tunnel route dns ...`) are not orchestrated by
|
||||
OmniRoute. URLs are ephemeral and will change every restart.
|
||||
- **Process supervision.** The cloudflared PID and resolved URL are persisted to
|
||||
`cloudflared-state.json` so the dashboard can resume status across reloads.
|
||||
|
||||
### Enable / disable via REST
|
||||
|
||||
The endpoint uses an `{action: "enable" | "disable"}` body, not separate
|
||||
`start`/`stop` paths. Management auth (admin session or admin API key) is
|
||||
required.
|
||||
|
||||
```bash
|
||||
# Enable
|
||||
curl -X POST http://localhost:20128/api/tunnels/cloudflared \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Cookie: auth_token=..." \
|
||||
-d '{"action":"enable"}'
|
||||
|
||||
# Status
|
||||
curl http://localhost:20128/api/tunnels/cloudflared \
|
||||
-H "Cookie: auth_token=..."
|
||||
|
||||
# Disable
|
||||
curl -X POST http://localhost:20128/api/tunnels/cloudflared \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Cookie: auth_token=..." \
|
||||
-d '{"action":"disable"}'
|
||||
```
|
||||
|
||||
Or via dashboard: **Settings → Tunnels → Cloudflare**.
|
||||
|
||||
### Optional env vars
|
||||
|
||||
| Variable | Purpose |
|
||||
| ---------------------------------------------------- | ------------------------------------------------------------------------------------- |
|
||||
| `CLOUDFLARED_BIN` | Override the binary path. If set and valid, OmniRoute uses it instead of downloading. |
|
||||
| `CLOUDFLARED_PROTOCOL` / `TUNNEL_TRANSPORT_PROTOCOL` | Transport protocol (default `http2`). |
|
||||
|
||||
## 2. ngrok
|
||||
|
||||
`src/lib/ngrokTunnel.ts` uses the **`@ngrok/ngrok` SDK** (in-process, no CLI
|
||||
subprocess). The native module is imported lazily on first start so platforms
|
||||
without prebuilt binaries do not break the app at boot.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
1. Sign up at <https://ngrok.com>.
|
||||
2. Copy your authtoken from the ngrok dashboard.
|
||||
3. Provide it either via:
|
||||
- `.env`: `NGROK_AUTHTOKEN=<token>`, or
|
||||
- Dashboard: **Settings → Tunnels → ngrok**, or
|
||||
- REST body (one-shot): `{"action":"enable","authToken":"<token>"}`.
|
||||
|
||||
If neither is configured, status returns `phase: "needs_auth"`.
|
||||
|
||||
### Enable / disable via REST
|
||||
|
||||
```bash
|
||||
# Enable (uses NGROK_AUTHTOKEN from env)
|
||||
curl -X POST http://localhost:20128/api/tunnels/ngrok \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Cookie: auth_token=..." \
|
||||
-d '{"action":"enable"}'
|
||||
|
||||
# Enable with inline token
|
||||
curl -X POST http://localhost:20128/api/tunnels/ngrok \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Cookie: auth_token=..." \
|
||||
-d '{"action":"enable","authToken":"2abc..."}'
|
||||
|
||||
# Status
|
||||
curl http://localhost:20128/api/tunnels/ngrok \
|
||||
-H "Cookie: auth_token=..."
|
||||
|
||||
# Disable
|
||||
curl -X POST http://localhost:20128/api/tunnels/ngrok \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Cookie: auth_token=..." \
|
||||
-d '{"action":"disable"}'
|
||||
```
|
||||
|
||||
The response includes the assigned `publicUrl` (e.g.
|
||||
`https://abcd-1234.ngrok-free.app`). Custom domains, regions, and policy rules
|
||||
must be configured in the ngrok dashboard — OmniRoute itself only forwards the
|
||||
local target URL to the SDK.
|
||||
|
||||
## 3. Tailscale Funnel
|
||||
|
||||
`src/lib/tailscaleTunnel.ts` orchestrates the system `tailscale` CLI to expose
|
||||
the local API port via **Funnel** (Tailscale's public-internet egress for serve).
|
||||
It supports the full lifecycle: install, login, daemon start, enable, disable.
|
||||
|
||||
The implementation invokes `tailscale funnel --bg <port>` (background mode). The
|
||||
public URL has the shape `https://<machine>.<tailnet>.ts.net/`.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
1. Install Tailscale (or let OmniRoute do it — see `install` endpoint below).
|
||||
2. Sign in (`tailscale login` or via OmniRoute's `login` endpoint).
|
||||
3. Enable Funnel for your tailnet in the Tailscale admin console:
|
||||
<https://login.tailscale.com/admin/settings/features>.
|
||||
|
||||
On Linux and macOS the daemon (`tailscaled`) requires `sudo` to control. The
|
||||
POST endpoints accept an optional `sudoPassword` field which is forwarded to
|
||||
OmniRoute's MITM password cache (`getCachedPassword` / `setCachedPassword`) for
|
||||
the duration of the call. Windows uses the default service install at
|
||||
`C:\Program Files\Tailscale\tailscale.exe`.
|
||||
|
||||
### REST endpoints
|
||||
|
||||
Tailscale has a richer surface than the other backends because installation,
|
||||
login, daemon, and tunnel are separate concerns.
|
||||
|
||||
| Endpoint | Method | Purpose |
|
||||
| ------------------------------------- | ------ | --------------------------------------------------------------- |
|
||||
| `/api/tunnels/tailscale` | `GET` | Aggregated tunnel status (`phase`, `tunnelUrl`, `apiUrl`, etc.) |
|
||||
| `/api/tunnels/tailscale/check` | `GET` | Lower-level check: installed? logged in? daemon running? |
|
||||
| `/api/tunnels/tailscale/install` | `POST` | Install Tailscale (SSE-streamed progress events) — Linux/macOS |
|
||||
| `/api/tunnels/tailscale/start-daemon` | `POST` | Start `tailscaled` on Linux/macOS |
|
||||
| `/api/tunnels/tailscale/login` | `POST` | Begin login flow; returns `authUrl` to open in a browser |
|
||||
| `/api/tunnels/tailscale/enable` | `POST` | Start the Funnel for the API port |
|
||||
| `/api/tunnels/tailscale/disable` | `POST` | Stop the Funnel |
|
||||
|
||||
All Tailscale endpoints require management auth (see `routeUtils.ts ::
|
||||
requireTailscaleAuth`).
|
||||
|
||||
Example enable:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/api/tunnels/tailscale/enable \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Cookie: auth_token=..." \
|
||||
-d '{"sudoPassword":"<linux-pwd>","port":20128}'
|
||||
```
|
||||
|
||||
If Funnel is not enabled in the admin console, the response includes
|
||||
`funnelNotEnabled: true` plus an `enableUrl` to open in a browser.
|
||||
|
||||
### Optional env vars
|
||||
|
||||
| Variable | Purpose |
|
||||
| --------------- | ------------------------------------ |
|
||||
| `TAILSCALE_BIN` | Override the `tailscale` binary path |
|
||||
|
||||
## Endpoint summary
|
||||
|
||||
| Endpoint | Method | Body | Auth |
|
||||
| ------------------------------------- | ------ | ----------------------------------- | ---------- |
|
||||
| `/api/tunnels/cloudflared` | `GET` | — | management |
|
||||
| `/api/tunnels/cloudflared` | `POST` | `{action: "enable" \| "disable"}` | management |
|
||||
| `/api/tunnels/ngrok` | `GET` | — | management |
|
||||
| `/api/tunnels/ngrok` | `POST` | `{action, authToken?}` | management |
|
||||
| `/api/tunnels/tailscale` | `GET` | — | management |
|
||||
| `/api/tunnels/tailscale/check` | `GET` | — | management |
|
||||
| `/api/tunnels/tailscale/install` | `POST` | `{sudoPassword?}` (SSE) | management |
|
||||
| `/api/tunnels/tailscale/start-daemon` | `POST` | `{sudoPassword?}` | management |
|
||||
| `/api/tunnels/tailscale/login` | `POST` | `{hostname?}` | management |
|
||||
| `/api/tunnels/tailscale/enable` | `POST` | `{sudoPassword?, hostname?, port?}` | management |
|
||||
| `/api/tunnels/tailscale/disable` | `POST` | `{sudoPassword?}` | management |
|
||||
|
||||
There is no central `/api/settings/tunnels` endpoint — each backend is
|
||||
independent.
|
||||
|
||||
## OAuth callback considerations
|
||||
|
||||
When you expose OmniRoute through a tunnel, the dashboard and OAuth flows must
|
||||
build callback URLs against the **public** hostname, not `localhost`. Otherwise
|
||||
the OAuth provider redirects the user back to a URL its servers cannot reach,
|
||||
and the handshake fails.
|
||||
|
||||
Set:
|
||||
|
||||
```bash
|
||||
NEXT_PUBLIC_BASE_URL=https://<your-tunnel-host>
|
||||
```
|
||||
|
||||
and restart OmniRoute before initiating OAuth. For ephemeral Cloudflare Quick
|
||||
Tunnels the URL changes after every restart, so prefer ngrok with a reserved
|
||||
domain or Tailscale Funnel for production OAuth use.
|
||||
|
||||
## Health and monitoring
|
||||
|
||||
The dashboard surfaces tunnel state under **Settings → Tunnels**:
|
||||
|
||||
- Active backend(s) and current `phase` (`stopped`, `starting`, `running`,
|
||||
`needs_auth`, `error`).
|
||||
- The current public URL and the derived API URL (`<publicUrl>/v1`).
|
||||
- The local target URL the tunnel is forwarding to.
|
||||
- Last error message, if any.
|
||||
|
||||
For programmatic monitoring poll the per-backend `GET` endpoints. Running more
|
||||
than one backend simultaneously is allowed; OmniRoute will track each
|
||||
independently.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "cloudflared binary not found"
|
||||
|
||||
OmniRoute attempts to auto-install on first use. If the install is blocked
|
||||
(restricted network, no GitHub access), download `cloudflared` manually from
|
||||
<https://github.com/cloudflare/cloudflared/releases> and set
|
||||
`CLOUDFLARED_BIN=/path/to/cloudflared`.
|
||||
|
||||
### "ngrok: authtoken required"
|
||||
|
||||
`phase: "needs_auth"` means no authtoken was found. Set `NGROK_AUTHTOKEN` in
|
||||
`.env`, configure it via the dashboard, or pass `authToken` in the enable POST
|
||||
body.
|
||||
|
||||
### "tailscale: funnel not enabled"
|
||||
|
||||
When the enable response includes `funnelNotEnabled: true`, Funnel is disabled
|
||||
for your tailnet. Open the returned `enableUrl` (or the admin console feature
|
||||
page) and toggle Funnel on.
|
||||
|
||||
### Tunnel URL changes break OAuth
|
||||
|
||||
Use ngrok with a reserved domain or Tailscale Funnel (both stable per-node).
|
||||
Cloudflare Quick Tunnels are ephemeral by design and not recommended for
|
||||
long-lived OAuth callbacks.
|
||||
|
||||
### Permission denied on Linux/macOS for Tailscale
|
||||
|
||||
`tailscaled` needs root. Provide `sudoPassword` to the relevant POST endpoint,
|
||||
or run the daemon yourself (`sudo systemctl start tailscaled`).
|
||||
|
||||
## See also
|
||||
|
||||
- [PROXY_GUIDE.md](./PROXY_GUIDE.md) — outbound proxy (1proxy, SOCKS5, HTTP) for
|
||||
egress traffic.
|
||||
- [ENVIRONMENT.md](./ENVIRONMENT.md) — full list of env vars including
|
||||
`NEXT_PUBLIC_BASE_URL`.
|
||||
- [FLY_IO_DEPLOYMENT_GUIDE.md](./FLY_IO_DEPLOYMENT_GUIDE.md),
|
||||
[DOCKER_GUIDE.md](./DOCKER_GUIDE.md) — alternatives to tunneling for stable
|
||||
public hosting.
|
||||
- Source: `src/lib/{cloudflaredTunnel,ngrokTunnel,tailscaleTunnel}.ts`,
|
||||
`src/app/api/tunnels/`.
|
||||
@@ -15,6 +15,15 @@ Complete guide for configuring providers, creating combos, integrating CLI tools
|
||||
- [Deployment](#-deployment)
|
||||
- [Available Models](#-available-models)
|
||||
- [Advanced Features](#-advanced-features)
|
||||
- [Auto-Routing (Zero-config)](#-auto-routing-zero-config)
|
||||
- [MCP & A2A Integration](#-mcp--a2a-integration)
|
||||
- [Skills System](#-skills-system)
|
||||
- [Memory System](#-memory-system)
|
||||
- [Webhooks](#-webhooks)
|
||||
- [Cloud Agents](#-cloud-agents)
|
||||
- [Programmatic Management](#-programmatic-management)
|
||||
- [Internal CLI](#-internal-cli)
|
||||
- [Desktop Application (Electron)](#-desktop-application-electron)
|
||||
|
||||
---
|
||||
|
||||
@@ -58,7 +67,7 @@ Complete guide for configuring providers, creating combos, integrating CLI tools
|
||||
Combo: "maximize-claude"
|
||||
1. cc/claude-opus-4-7 (use subscription fully)
|
||||
2. glm/glm-4.7 (cheap backup when quota out)
|
||||
3. if/kimi-k2-thinking (free emergency fallback)
|
||||
3. if/kimi-k2 (free emergency fallback)
|
||||
|
||||
Monthly cost: $20 (subscription) + ~$5 (backup) = $25 total
|
||||
vs. $20 + hitting limits = frustration
|
||||
@@ -70,8 +79,8 @@ vs. $20 + hitting limits = frustration
|
||||
|
||||
```
|
||||
Combo: "free-forever"
|
||||
1. gc/gemini-3-flash (180K free/month)
|
||||
2. if/kimi-k2-thinking (unlimited free)
|
||||
1. gemini-cli/gemini-3-flash-preview (180K free/month)
|
||||
2. if/kimi-k2 (unlimited free)
|
||||
3. qw/qwen3-coder-plus (unlimited free)
|
||||
|
||||
Monthly cost: $0
|
||||
@@ -88,7 +97,7 @@ Combo: "always-on"
|
||||
2. cx/gpt-5.5 (second subscription)
|
||||
3. glm/glm-4.7 (cheap, resets daily)
|
||||
4. minimax/MiniMax-M2.1 (cheapest, 5h reset)
|
||||
5. if/kimi-k2-thinking (free unlimited)
|
||||
5. if/kimi-k2 (free unlimited)
|
||||
|
||||
Result: 5 layers of fallback = zero downtime
|
||||
Monthly cost: $20-200 (subscriptions) + $10-20 (backup)
|
||||
@@ -100,9 +109,9 @@ Monthly cost: $20-200 (subscriptions) + $10-20 (backup)
|
||||
|
||||
```
|
||||
Combo: "openclaw-free"
|
||||
1. if/glm-4.7 (unlimited free)
|
||||
2. if/minimax-m2.1 (unlimited free)
|
||||
3. if/kimi-k2-thinking (unlimited free)
|
||||
1. if/qwen3-coder-plus (unlimited free)
|
||||
2. if/deepseek-r1 (unlimited free)
|
||||
3. if/kimi-k2 (unlimited free)
|
||||
|
||||
Monthly cost: $0
|
||||
Access via: WhatsApp, Telegram, Slack, Discord, iMessage, Signal...
|
||||
@@ -137,8 +146,10 @@ Dashboard → Providers → Connect Codex
|
||||
→ 5-hour + weekly reset
|
||||
|
||||
Models:
|
||||
cx/gpt-5-5
|
||||
cx/gpt-5-3-codex-spark
|
||||
cx/gpt-5.5
|
||||
cx/gpt-5.4
|
||||
cx/gpt-5.3-codex
|
||||
cx/gpt-5.3-codex-spark
|
||||
```
|
||||
|
||||
#### Gemini CLI (FREE 180K/month!)
|
||||
@@ -149,8 +160,9 @@ Dashboard → Providers → Connect Gemini CLI
|
||||
→ 180K completions/month + 1K/day
|
||||
|
||||
Models:
|
||||
gc/gemini-3-flash
|
||||
gc/gemini-3.1-flash-lite-preview
|
||||
gemini-cli/gemini-3.1-pro-preview
|
||||
gemini-cli/gemini-3-flash-preview
|
||||
gemini-cli/gemini-3.1-flash-lite-preview
|
||||
```
|
||||
|
||||
**Best Value:** Huge free tier! Use this before paid tiers.
|
||||
@@ -163,8 +175,10 @@ Dashboard → Providers → Connect GitHub
|
||||
→ Monthly reset (1st of month)
|
||||
|
||||
Models:
|
||||
gh/gpt-5.5
|
||||
gh/gpt-5.4
|
||||
gh/claude-4.6-sonnet
|
||||
gh/claude-sonnet-4.6
|
||||
gh/claude-opus-4.7
|
||||
gh/gemini-3.1-pro-preview
|
||||
```
|
||||
|
||||
@@ -206,7 +220,7 @@ Models:
|
||||
```bash
|
||||
Dashboard → Connect Qoder → OAuth login → Unlimited usage
|
||||
|
||||
Models: if/kimi-k2-thinking, if/qwen3-coder-plus, if/glm-4.7, if/minimax-m2, if/deepseek-r1
|
||||
Models: if/kimi-k2, if/qwen3-coder-plus, if/qwen3-max, if/qwen3-235b, if/deepseek-r1, if/deepseek-v3.2
|
||||
```
|
||||
|
||||
#### Kiro (Claude FREE)
|
||||
@@ -242,9 +256,9 @@ Use in CLI: premium-coding
|
||||
```
|
||||
Name: free-combo
|
||||
Models:
|
||||
1. gc/gemini-3-flash-preview (180K free/month)
|
||||
2. if/kimi-k2-thinking (unlimited)
|
||||
3. qw/qwen3-coder-plus (unlimited)
|
||||
1. gemini-cli/gemini-3-flash-preview (180K free/month)
|
||||
2. if/kimi-k2 (unlimited)
|
||||
3. qw/coder-model (unlimited)
|
||||
|
||||
Cost: $0 forever!
|
||||
```
|
||||
@@ -293,7 +307,7 @@ Edit `~/.openclaw/openclaw.json`:
|
||||
{
|
||||
"agents": {
|
||||
"defaults": {
|
||||
"model": { "primary": "omniroute/if/glm-4.7" }
|
||||
"model": { "primary": "omniroute/if/kimi-k2" }
|
||||
}
|
||||
},
|
||||
"models": {
|
||||
@@ -302,7 +316,7 @@ Edit `~/.openclaw/openclaw.json`:
|
||||
"baseUrl": "http://localhost:20128/v1",
|
||||
"apiKey": "your-omniroute-api-key",
|
||||
"api": "openai-completions",
|
||||
"models": [{ "id": "if/glm-4.7", "name": "glm-4.7" }]
|
||||
"models": [{ "id": "if/kimi-k2", "name": "kimi-k2" }]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -432,7 +446,7 @@ Void Linux users can package and install OmniRoute natively using the `xbps-src`
|
||||
```bash
|
||||
# Template file for 'omniroute'
|
||||
pkgname=omniroute
|
||||
version=3.2.4
|
||||
version=3.8.0
|
||||
revision=1
|
||||
hostmakedepends="nodejs python3 make"
|
||||
depends="openssl"
|
||||
@@ -532,8 +546,8 @@ post_install() {
|
||||
| `PORT` | framework default | Service port (`20128` in examples) |
|
||||
| `HOSTNAME` | framework default | Bind host (Docker defaults to `0.0.0.0`) |
|
||||
| `NODE_ENV` | runtime default | Set `production` for deploy |
|
||||
| `BASE_URL` | `http://localhost:20128` | Server-side internal base URL |
|
||||
| `CLOUD_URL` | `https://omniroute.dev` | Cloud sync endpoint base URL |
|
||||
| `NEXT_PUBLIC_BASE_URL` | `http://localhost:20128` | Public base URL surfaced to the dashboard and exposed to the server (replaces legacy `BASE_URL`) |
|
||||
| `NEXT_PUBLIC_CLOUD_URL` | `https://omniroute.dev` | Cloud sync endpoint base URL (replaces legacy `CLOUD_URL`) |
|
||||
| `API_KEY_SECRET` | `endpoint-proxy-api-key-secret` | HMAC secret for generated API keys |
|
||||
| `REQUIRE_API_KEY` | `false` | Enforce Bearer API key on `/v1/*` |
|
||||
| `ALLOW_API_KEY_REVEAL` | `false` | Allow Api Manager to copy full API keys on demand |
|
||||
@@ -556,45 +570,57 @@ For the full environment variable reference, see the [README](../README.md).
|
||||
<details>
|
||||
<summary><b>View all available models</b></summary>
|
||||
|
||||
**Claude Code (`cc/`)** — Pro/Max: `cc/claude-opus-4-7`, `cc/claude-sonnet-4-6`, `cc/claude-haiku-4-5-20251001`
|
||||
> The list below is curated from `open-sse/config/providerRegistry.ts` for v3.8.0. Cloud catalogs (Gemini, OpenRouter, etc.) are synced dynamically — for the full live catalog open **Dashboard → Providers → [provider] → Available Models** or call `GET /api/models/catalog`.
|
||||
|
||||
**Codex (`cx/`)** — Plus/Pro: `cx/gpt-5.5`, `cx/gpt-5.4`, `cx/gpt-5.3-codex-spark`, `cx/gpt-5.3-codex`
|
||||
**Claude Code (`cc/`)** — Pro/Max OAuth: `cc/claude-opus-4-7`, `cc/claude-opus-4-6`, `cc/claude-opus-4-5-20251101`, `cc/claude-sonnet-4-6`, `cc/claude-sonnet-4-5-20250929`, `cc/claude-haiku-4-5-20251001`
|
||||
|
||||
**Gemini CLI (`gc/`)** — FREE: `gc/gemini-3-flash-preview`, `gc/gemini-3.1-flash-lite-preview`
|
||||
**Codex (`cx/`)** — Plus/Pro OAuth: `cx/gpt-5.5` (+ effort tiers: `gpt-5.5-xhigh`, `gpt-5.5-high`, `gpt-5.5-medium`, `gpt-5.5-low`), `cx/gpt-5.4`, `cx/gpt-5.4-mini`, `cx/gpt-5.3-codex`, `cx/gpt-5.3-codex-spark`, `cx/gpt-5.2`
|
||||
|
||||
**GitHub Copilot (`gh/`)**: `gh/gpt-5-5`, `gh/gpt-5-4`, `gh/claude-opus-4.7`, `gh/claude-sonnet-4.6`, `gh/claude-haiku-4.5`
|
||||
**Gemini CLI (`gemini-cli/`)** — FREE OAuth: `gemini-cli/gemini-3.1-pro-preview`, `gemini-cli/gemini-3.1-pro-preview-customtools`, `gemini-cli/gemini-3-flash-preview`, `gemini-cli/gemini-3.1-flash-lite-preview`
|
||||
|
||||
**GLM (`glm/`)** — $0.6/1M: `glm/glm-5.1`
|
||||
**GitHub Copilot (`gh/`)** — OAuth: `gh/gpt-5.5`, `gh/gpt-5.4`, `gh/gpt-5.4-mini`, `gh/gpt-5-mini`, `gh/gpt-5.3-codex`, `gh/claude-opus-4.7`, `gh/claude-opus-4.6`, `gh/claude-opus-4-5-20251101`, `gh/claude-sonnet-4.6`, `gh/claude-sonnet-4.5`, `gh/claude-haiku-4.5`, `gh/gemini-3.1-pro-preview`, `gh/gemini-3-flash-preview`, `gh/oswe-vscode-prime`
|
||||
|
||||
**MiniMax (`minimax/`)** — $0.2/1M: `minimax/MiniMax-M2.7`, `minimax/MiniMax-M2.5`
|
||||
**Kiro (`kr/`)** — FREE OAuth: `kr/auto-kiro`, `kr/claude-opus-4.7`, `kr/claude-opus-4.6`, `kr/claude-sonnet-4.6`, `kr/claude-sonnet-4.5`, `kr/claude-haiku-4.5`, `kr/deepseek-3.2`, `kr/minimax-m2.5`, `kr/minimax-m2.1`, `kr/glm-5`, `kr/qwen3-coder-next`
|
||||
|
||||
**Qoder (`if/`)** — FREE: `if/kimi-k2-thinking`, `if/qwen3-coder-plus`, `if/deepseek-r1`
|
||||
**Qoder (`if/`)** — FREE OAuth: `if/kimi-k2-0905`, `if/kimi-k2`, `if/qwen3-coder-plus`, `if/qwen3-max`, `if/qwen3-max-preview`, `if/qwen3-vl-plus`, `if/qwen3-32b`, `if/qwen3-235b-a22b-thinking-2507`, `if/qwen3-235b-a22b-instruct`, `if/qwen3-235b`, `if/deepseek-v3.2`, `if/deepseek-v3`, `if/deepseek-r1`, `if/qoder-rome-30ba3b`
|
||||
|
||||
**Qwen (`qw/`)**: `qw/qwen3-coder-plus`, `qw/qwen3-coder-flash`
|
||||
**Qwen (`qw/`)** — FREE OAuth (chat.qwen.ai): `qw/coder-model`, `qw/vision-model`
|
||||
|
||||
**Kiro (`kr/`)** — FREE: `kr/claude-sonnet-4.5`, `kr/claude-haiku-4.5`
|
||||
**GLM (`glm/`, `glm-cn/`, `zai/`, `glmt/`)** — $0.2–0.6/1M: `glm/glm-5.1`, `glm/glm-5`, `glm/glm-5-turbo`, `glm/glm-4.7`, `glm/glm-4.7-flash`, `glm/glm-4.6`, `glm/glm-4.6v`, `glm/glm-4.5`, `glm/glm-4.5v`, `glm/glm-4.5-air`
|
||||
|
||||
**DeepSeek (`ds/`)**: `ds/deepseek-v4-pro`, `ds/deepseek-v4-flash`
|
||||
**MiniMax (`minimax/`, `minimax-cn/`)** — $0.2/1M: `minimax/MiniMax-M2.7`, `minimax/MiniMax-M2.7-highspeed`, `minimax/MiniMax-M2.5`, `minimax/MiniMax-M2.5-highspeed`
|
||||
|
||||
**Groq (`groq/`)**: `groq/llama-3.3-70b-versatile`, `groq/llama-4-maverick-17b-128e-instruct`
|
||||
**Kimi (`kimi/`, `kimi-coding/`, `kimi-coding-apikey/`)** — $9/mo flat or per-use: `kimi/kimi-k2.6`, `kimi/kimi-k2.5`
|
||||
|
||||
**xAI (`xai/`)**: `xai/grok-4.3`, `xai/grok-4.20-0309-reasoning`, `xai/grok-4.20-0309-non-reasoning`
|
||||
**DeepSeek (`ds/`)** — API key: `ds/deepseek-v4-pro`, `ds/deepseek-v4-flash`
|
||||
|
||||
**Mistral (`mistral/`)**: `mistral/mistral-large-latest`, `mistral/mistral-medium-3-5`, `mistral/mistral-small-latest`, `mistral/devstral-latest`, `mistral/codestral-latest`
|
||||
**Groq (`groq/`)** — Ultra-fast: `groq/llama-3.3-70b-versatile`, `groq/meta-llama/llama-4-maverick-17b-128e-instruct`, `groq/qwen/qwen3-32b`, `groq/openai/gpt-oss-120b`
|
||||
|
||||
**Perplexity (`pplx/`)**: `pplx/sonar-deep-research`, `pplx/sonar-reasoning-pro`, `pplx/sonar-pro`, `pplx/sonar`
|
||||
**xAI (`xai/`)** — Grok native: `xai/grok-4.3`, `xai/grok-4.20-multi-agent-0309`, `xai/grok-4.20-0309-reasoning`, `xai/grok-4.20-0309-non-reasoning`
|
||||
|
||||
**Together AI (`together/`)**: `together/meta-llama/Llama-3.3-70B-Instruct-Turbo`
|
||||
**Mistral (`mistral/`)** — EU-hosted: `mistral/mistral-large-latest`, `mistral/mistral-medium-3-5`, `mistral/mistral-small-latest`, `mistral/devstral-latest`, `mistral/codestral-latest`
|
||||
|
||||
**Fireworks AI (`fireworks/`)**: `fireworks/accounts/fireworks/models/deepseek-v3p1`
|
||||
**Perplexity (`pplx/`)** — Search-augmented: `pplx/sonar-deep-research`, `pplx/sonar-reasoning-pro`, `pplx/sonar-pro`, `pplx/sonar`
|
||||
|
||||
**Cerebras (`cerebras/`)**: `cerebras/zai-glm-4.7`, `cerebras/gpt-oss-120b`
|
||||
**Together AI (`together/`)** — Open-source: `together/meta-llama/Llama-3.3-70B-Instruct-Turbo-Free` (free), `together/meta-llama/Llama-Vision-Free`, `together/deepseek-ai/DeepSeek-R1-Distill-Llama-70B-Free`, `together/deepseek-ai/DeepSeek-R1`, `together/Qwen/Qwen3-235B-A22B`, `together/meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8`
|
||||
|
||||
**Cohere (`cohere/`)**: `cohere/command-r-plus-08-2024`
|
||||
**Fireworks AI (`fireworks/`)** — Fast inference: `fireworks/accounts/fireworks/models/kimi-k2p6`, `fireworks/accounts/fireworks/models/minimax-m2p7`, `fireworks/accounts/fireworks/models/qwen3p6-plus`, `fireworks/accounts/fireworks/models/glm-5p1`, `fireworks/accounts/fireworks/models/deepseek-v4-pro`
|
||||
|
||||
**NVIDIA NIM (`nvidia/`)**: `nvidia/nvidia/llama-3.3-70b-instruct`
|
||||
**Cerebras (`cerebras/`)** — Wafer-scale: `cerebras/zai-glm-4.7`, `cerebras/gpt-oss-120b`
|
||||
|
||||
**Baidu Qianfan (`qianfan/`)**: `qianfan/ernie-5.1`, `qianfan/ernie-5.0-thinking-latest`, `qianfan/ernie-x1.1`
|
||||
**Cohere (`cohere/`)** — RAG-focused: `cohere/command-a-reasoning-08-2025`, `cohere/command-a-vision-07-2025`, `cohere/command-a-03-2025`, `cohere/command-r-08-2024`
|
||||
|
||||
**NVIDIA NIM (`nvidia/`)** — Enterprise: `nvidia/z-ai/glm-5.1`, `nvidia/minimaxai/minimax-m2.7`, `nvidia/google/gemma-4-31b-it`, `nvidia/mistralai/mistral-small-4-119b-2603`, `nvidia/mistralai/mistral-large-3-675b-instruct-2512`, `nvidia/qwen/qwen3.5-397b-a17b`, `nvidia/deepseek-ai/deepseek-v4-pro`, `nvidia/openai/gpt-oss-120b`, `nvidia/nvidia/nemotron-3-super-120b-a12b`
|
||||
|
||||
**Baidu Qianfan (`qianfan/`)** — ERNIE: `qianfan/ernie-5.1`, `qianfan/ernie-5.0-thinking-latest`, `qianfan/ernie-x1.1`
|
||||
|
||||
**Ollama Cloud (`ollama-cloud/`)**: `ollama-cloud/deepseek-v4-pro`, `ollama-cloud/deepseek-v4-flash`, `ollama-cloud/kimi-k2.6`, `ollama-cloud/glm-5.1`, `ollama-cloud/minimax-m2.7`, `ollama-cloud/gemma4:31b`, `ollama-cloud/qwen3.5:397b`
|
||||
|
||||
**Gemini (Google Cloud `gemini/`)**: Synced live per API key from Google — no static list. Connect a key in **Dashboard → Providers** then use **Available Models** to import the current catalog (e.g. `gemini/gemini-3-pro`, `gemini/gemini-3-flash`).
|
||||
|
||||
**Other compatible providers** (selected): `cohere`, `databricks`, `snowflake`, `together`, `vertex`, `alibaba`, `bedrock` (via `aws-bedrock`), `azure-ai`, `openrouter` (passthrough catalog), `siliconflow`, `hyperbolic`, `huggingface`, `featherless-ai`, `cloudflare-ai`, `scaleway`, `deepinfra`, `vercel-ai-gateway`, `bazaarlink`, `friendliai`, `nous-research`, `reka`, `volcengine`, `ai21`, `gigachat`. Each maintains its own model list in `providerRegistry.ts` and can be auto-synced when the provider exposes a `/models` endpoint.
|
||||
|
||||
**Note on model IDs:** OmniRoute uses provider-native IDs (`claude-opus-4-7`, `gpt-5.5`, `glm-5.1`, `MiniMax-M2.7`, `kimi-k2.5`, `grok-4.20-0309-reasoning`). Some IDs include dotted versions because that is how the upstream API expects them. If a model is not listed above, run `omniroute models --search <term>` or hit `GET /api/models/catalog` to confirm availability.
|
||||
|
||||
</details>
|
||||
|
||||
@@ -665,7 +691,7 @@ Returns models grouped by provider with types (`chat`, `embedding`, `image`).
|
||||
|
||||
- Sync providers, combos, and settings across devices
|
||||
- Automatic background sync with timeout + fail-fast
|
||||
- Prefer server-side `BASE_URL`/`CLOUD_URL` in production
|
||||
- Prefer server-side `NEXT_PUBLIC_BASE_URL`/`NEXT_PUBLIC_CLOUD_URL` in production
|
||||
|
||||
### Cloudflare Quick Tunnel
|
||||
|
||||
@@ -708,7 +734,9 @@ Access via **Dashboard → Translator**. Debug and visualize how OmniRoute trans
|
||||
|
||||
### Routing Strategies
|
||||
|
||||
Configure via **Dashboard → Settings → Routing**.
|
||||
Configure via **Dashboard → Settings → Routing**. The dashboard exposes the six most-used strategies; combos and the auto-router internally support a wider set.
|
||||
|
||||
**Dashboard-visible strategies (account-level routing):**
|
||||
|
||||
| Strategy | Description |
|
||||
| ------------------------------ | ------------------------------------------------------------------------------------------------ |
|
||||
@@ -719,6 +747,19 @@ Configure via **Dashboard → Settings → Routing**.
|
||||
| **Least Used** | Routes to the account with the oldest `lastUsedAt` timestamp, distributing traffic evenly |
|
||||
| **Cost Optimized** | Routes to the account with the lowest priority value, optimizing for lowest-cost providers |
|
||||
|
||||
**Advanced combo and auto strategies** (configurable per combo or via `auto/*` prefixes — see [AUTO-COMBO.md](AUTO-COMBO.md)):
|
||||
|
||||
- `priority` — strict order, never round-robins
|
||||
- `weighted` — proportional traffic split by per-model weights
|
||||
- `fill-first` — drain the first model until limits hit
|
||||
- `round-robin` / `strict-random` / `random`
|
||||
- `p2c` (Power of Two Choices)
|
||||
- `least-used` and `cost-optimized`
|
||||
- `auto` — score-driven across all candidates
|
||||
- `lkgp` (Last Known Good Provider) — sticks to the last successful model per session
|
||||
- `context-optimized` — picks the model with the largest free context window
|
||||
- `context-relay` — chains long-context models for follow-up turns
|
||||
|
||||
#### External Sticky Session Header
|
||||
|
||||
For external session affinity (for example, Claude Code/Codex agents behind reverse proxies), send:
|
||||
@@ -828,16 +869,17 @@ curl -X POST http://localhost:20128/api/db-backups/import \
|
||||
|
||||
### Settings Dashboard
|
||||
|
||||
The settings page is organized into 6 tabs for easy navigation:
|
||||
The settings page is organized into **7 tabs** for easy navigation:
|
||||
|
||||
| Tab | Contents |
|
||||
| -------------- | ------------------------------------------------------------------------------------------------------------- |
|
||||
| **General** | System storage tools, appearance settings, theme controls, sidebar visibility, and Endpoint tunnel visibility |
|
||||
| **Security** | Login/Password settings, IP Access Control, API auth for `/models`, and Provider Blocking |
|
||||
| **Routing** | Global routing strategy (6 options), wildcard model aliases, fallback chains, combo defaults |
|
||||
| **Resilience** | Request queue, connection cooldown, provider breaker config, and wait-for-cooldown behavior |
|
||||
| **AI** | Thinking budget configuration, global system prompt injection, prompt cache stats |
|
||||
| **Advanced** | Global proxy configuration (HTTP/SOCKS5) |
|
||||
| Tab | Contents |
|
||||
| -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **General** | System storage tools, default behavior, Endpoint tunnel visibility |
|
||||
| **Appearance** | Theme controls (light/dark/system), sidebar visibility, panel toggles for Cloudflare/Tailscale/ngrok tunnel cards |
|
||||
| **AI** | Thinking budget configuration, global system prompt injection, prompt cache stats |
|
||||
| **Security** | Login/Password settings, IP Access Control, API auth for `/models`, Provider Blocking, prompt-injection guard |
|
||||
| **Routing** | Global routing strategy (Fill First / Round Robin / P2C / Random / Least Used / Cost Optimized), wildcard model aliases, fallback chains, combo defaults |
|
||||
| **Resilience** | Request queue, connection cooldown, provider breaker config, and wait-for-cooldown behavior |
|
||||
| **Advanced** | Global proxy configuration (HTTP/SOCKS5), per-provider proxy overrides |
|
||||
|
||||
---
|
||||
|
||||
@@ -880,9 +922,34 @@ curl -X POST http://localhost:20128/v1/audio/transcriptions \
|
||||
-F "model=deepgram/nova-3"
|
||||
```
|
||||
|
||||
Available providers: **Deepgram** (`deepgram/`), **AssemblyAI** (`assemblyai/`).
|
||||
**Speech-to-Text (transcription)** providers:
|
||||
|
||||
Supported audio formats: `mp3`, `wav`, `m4a`, `flac`, `ogg`, `webm`.
|
||||
- `openai/` (whisper-compatible)
|
||||
- `groq/` (Groq Whisper Turbo)
|
||||
- `deepgram/` (Nova family)
|
||||
- `assemblyai/`
|
||||
- `nvidia/` (Parakeet, Canary)
|
||||
- `huggingface/` (whisper variants)
|
||||
- `qwen/`
|
||||
|
||||
**Text-to-Speech (`POST /v1/audio/speech`)** providers:
|
||||
|
||||
- `openai/` (tts-1, tts-1-hd)
|
||||
- `hyperbolic/`
|
||||
- `deepgram/` (Aura)
|
||||
- `nvidia/` (Magpie TTS)
|
||||
- `elevenlabs/`
|
||||
- `huggingface/`
|
||||
- `inworld/`
|
||||
- `cartesia/`
|
||||
- `playht/`
|
||||
- `kie/`
|
||||
- `aws-polly/`
|
||||
- `xiaomi-mimo/`
|
||||
- `coqui/`, `tortoise/`
|
||||
- `qwen/`
|
||||
|
||||
Supported audio formats for transcription: `mp3`, `wav`, `m4a`, `flac`, `ogg`, `webm`. TTS output formats depend on the provider (mp3, wav, opus, pcm, mulaw).
|
||||
|
||||
---
|
||||
|
||||
@@ -920,6 +987,186 @@ Access via **Dashboard → Health**. Real-time system health overview with 6 car
|
||||
|
||||
---
|
||||
|
||||
## 🤖 Auto-Routing (Zero-config)
|
||||
|
||||
OmniRoute ships with a **score-driven auto-router** that picks the best model for each request across every connected provider — no combo to maintain. Just send the request with one of the `auto/*` prefixes and OmniRoute will assemble a virtual combo on the fly, scoring candidates on latency, cost, success rate, context fit, model fitness for the task, recent failures, quota, and circuit-breaker state.
|
||||
|
||||
| Prefix | Optimizes for |
|
||||
| -------------- | ----------------------------------------------------------------------------- |
|
||||
| `auto` | Balanced default (latency × cost × success rate) |
|
||||
| `auto/coding` | Coding tasks: prefers Claude, GPT-5, GLM, Kimi, Qwen Coder, DeepSeek coders |
|
||||
| `auto/cheap` | Lowest $/token, accepts higher latency |
|
||||
| `auto/fast` | Lowest latency, ignores cost |
|
||||
| `auto/offline` | Local-only providers (Ollama, vLLM, llama.cpp) — useful for air-gapped setups |
|
||||
| `auto/smart` | Reasoning quality first (Opus, GPT-5 xhigh, R1, GLM 5.1 reasoning) |
|
||||
| `auto/lkgp` | "Last Known Good Provider" — sticky to the most recently successful target |
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/v1/chat/completions \
|
||||
-H "Authorization: Bearer $OMNIROUTE_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "auto/coding",
|
||||
"messages": [{ "role": "user", "content": "Refactor this Python function" }],
|
||||
"stream": true
|
||||
}'
|
||||
```
|
||||
|
||||
The auto-router is fully described in [AUTO-COMBO.md](AUTO-COMBO.md) — including how to tune scoring weights, blacklist providers, and inspect routing decisions in **Dashboard → Auto Combo**.
|
||||
|
||||
---
|
||||
|
||||
## 🔌 MCP & A2A Integration
|
||||
|
||||
OmniRoute is both an **MCP server** (Model Context Protocol) and an **A2A server** (Agent-to-Agent JSON-RPC 2.0). Any MCP-compatible IDE or agent host can call OmniRoute tools directly — no extra wrapper required.
|
||||
|
||||
### MCP transports
|
||||
|
||||
- **SSE**: `http://localhost:20128/api/mcp/sse`
|
||||
- **Streamable HTTP**: `http://localhost:20128/api/mcp/stream`
|
||||
- **stdio**: `omniroute --mcp` (for IDE plugins that prefer stdio)
|
||||
|
||||
### Connect Claude Desktop
|
||||
|
||||
Edit `~/Library/Application Support/Claude/claude_desktop_config.json` (macOS) or the equivalent on Windows/Linux:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"omniroute": {
|
||||
"command": "omniroute",
|
||||
"args": ["--mcp"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Connect Cursor / Continue / VS Code MCP
|
||||
|
||||
Use the SSE URL `http://localhost:20128/api/mcp/sse` and a Bearer API key generated in **Dashboard → API Manager**.
|
||||
|
||||
### Scopes
|
||||
|
||||
MCP tools are grouped into 10 scopes: `analytics`, `auth`, `billing`, `combos`, `health`, `keys`, `memory`, `models`, `providers`, `system`. Each Bearer key can be limited to specific scopes — see [MCP-SERVER.md](MCP-SERVER.md) for the full tool catalog and [A2A-SERVER.md](A2A-SERVER.md) for the JSON-RPC schema.
|
||||
|
||||
---
|
||||
|
||||
## 🧠 Skills System
|
||||
|
||||
OmniRoute exposes an extensible **skill framework** (`src/lib/skills/`) so agents and the A2A endpoint can run domain-specific routines (e.g. `code-review`, `summarize`, `extract-facts`, `web-research`).
|
||||
|
||||
- **Marketplace UI** — Browse and install skills from **Dashboard → Skills**
|
||||
- **Per-key scopes** — Restrict which API keys can invoke which skills
|
||||
- **Custom skills** — Drop a TypeScript file in `src/lib/a2a/skills/`, register it, and it becomes immediately invocable over A2A
|
||||
|
||||
Full reference: [SKILLS.md](SKILLS.md).
|
||||
|
||||
---
|
||||
|
||||
## 💾 Memory System
|
||||
|
||||
OmniRoute persists **long-term conversational memory** with hybrid retrieval:
|
||||
|
||||
- **SQLite FTS5** for keyword search across past turns
|
||||
- **Qdrant vector store** (optional) for semantic recall
|
||||
- **Automatic fact extraction** — entities, preferences, and decisions are summarized after each session and stored in the `memory_facts` table
|
||||
- Memories are scoped per API key and per session
|
||||
|
||||
Manage memories in **Dashboard → Memory** (search, edit, export, purge). The HTTP surface (`/api/memory/*`) lets agents push and query facts programmatically — see [MEMORY.md](MEMORY.md).
|
||||
|
||||
---
|
||||
|
||||
## 🔔 Webhooks
|
||||
|
||||
Subscribe to OmniRoute events for real-time monitoring and automation.
|
||||
|
||||
- Create a webhook in **Dashboard → Webhooks** with target URL and HMAC signing secret
|
||||
- Available events: `request.completed`, `request.failed`, `provider.unavailable`, `budget.exceeded`, `combo.switched`, `circuit_breaker.opened`, `circuit_breaker.closed`
|
||||
- Every payload includes `X-OmniRoute-Signature` (HMAC-SHA256) for verification
|
||||
- Retries: 3 attempts with exponential backoff, then dead-letter queue
|
||||
|
||||
Full schema in [WEBHOOKS.md](WEBHOOKS.md).
|
||||
|
||||
---
|
||||
|
||||
## ☁️ Cloud Agents
|
||||
|
||||
OmniRoute integrates with cloud coding agents (**OpenAI Codex Cloud**, **Devin**, **Jules**, **Antigravity**) so you can dispatch long-running tasks from the same dashboard that handles your local routing.
|
||||
|
||||
- Create tasks in **Dashboard → Cloud Agents** or via `POST /api/v1/agents/tasks`
|
||||
- Track status, logs, and artifacts per task
|
||||
- Bring-your-own API key per provider — credentials never leave the OmniRoute instance
|
||||
|
||||
Full reference: [CLOUD_AGENT.md](CLOUD_AGENT.md).
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ Programmatic Management
|
||||
|
||||
You can manage every OmniRoute resource (providers, combos, keys, settings) over HTTP using a **Bearer key with the `manage` scope**.
|
||||
|
||||
Generate the key in **Dashboard → API Manager → New Key → Scope: manage**, then:
|
||||
|
||||
```bash
|
||||
# List providers
|
||||
curl http://localhost:20128/api/providers \
|
||||
-H "Authorization: Bearer $OMNIROUTE_MANAGE_KEY"
|
||||
|
||||
# Add a provider connection
|
||||
curl -X POST http://localhost:20128/api/providers \
|
||||
-H "Authorization: Bearer $OMNIROUTE_MANAGE_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{ "provider": "openai", "apiKey": "sk-...", "name": "main" }'
|
||||
|
||||
# Create a combo
|
||||
curl -X POST http://localhost:20128/api/combos \
|
||||
-H "Authorization: Bearer $OMNIROUTE_MANAGE_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{ "name": "premium", "strategy": "priority", "models": [{ "model": "cc/claude-opus-4-7" }, { "model": "glm/glm-5.1" }] }'
|
||||
|
||||
# List/create API keys
|
||||
curl http://localhost:20128/api/keys -H "Authorization: Bearer $OMNIROUTE_MANAGE_KEY"
|
||||
curl -X POST http://localhost:20128/api/keys -H "Authorization: Bearer $OMNIROUTE_MANAGE_KEY" \
|
||||
-d '{ "name": "ci-bot", "scopes": ["chat"] }'
|
||||
```
|
||||
|
||||
See [API_REFERENCE.md](API_REFERENCE.md) for the full endpoint catalog and request/response schemas.
|
||||
|
||||
---
|
||||
|
||||
## 💻 Internal CLI
|
||||
|
||||
OmniRoute ships an internal CLI (`omniroute …`) for setup, diagnostics, and runtime control. This is **separate from the "CLI Tools" page in the dashboard**, which configures third-party CLIs (Claude Code, Cursor, Codex, Cline, …) so they can talk to OmniRoute.
|
||||
|
||||
```bash
|
||||
omniroute setup # Interactive wizard (password, providers, combos)
|
||||
omniroute setup --non-interactive # CI-friendly
|
||||
omniroute doctor # Health diagnostics (data dir, DB, providers, ports)
|
||||
omniroute providers available # List supported providers
|
||||
omniroute providers list # List configured connections
|
||||
omniroute providers test <id> # Live test a provider connection
|
||||
omniroute combos list # List combos
|
||||
omniroute combos switch <name> # Set default combo
|
||||
omniroute models # List available models (--json, --search)
|
||||
omniroute keys add | list | remove # Manage API keys from the terminal
|
||||
omniroute backup # Snapshot config + DB
|
||||
omniroute restore [<timestamp>] # Restore from a snapshot
|
||||
omniroute health # Detailed health (breakers, cache, memory)
|
||||
omniroute quota # Provider quota usage
|
||||
omniroute mcp status # MCP server status
|
||||
omniroute a2a status # A2A server status
|
||||
omniroute tunnel list|create|stop # Cloudflare/Tailscale/ngrok tunnels
|
||||
omniroute reset-password # Reset the admin password
|
||||
omniroute --mcp # Start MCP server over stdio
|
||||
omniroute --port 3000 # Start the server on a custom port
|
||||
```
|
||||
|
||||
Tip: pair `omniroute doctor --json` with your monitoring tool to alert on unhealthy provider connections.
|
||||
|
||||
---
|
||||
|
||||
## 🖥️ Desktop Application (Electron)
|
||||
|
||||
OmniRoute is available as a native desktop application for Windows, macOS, and Linux.
|
||||
|
||||
253
docs/WEBHOOKS.md
Normal file
253
docs/WEBHOOKS.md
Normal file
@@ -0,0 +1,253 @@
|
||||
# Webhooks
|
||||
|
||||
> **Source of truth:** `src/lib/webhookDispatcher.ts`, `src/lib/db/webhooks.ts`, `src/app/api/webhooks/`
|
||||
> **Last updated:** 2026-05-13 — v3.8.0
|
||||
|
||||
OmniRoute can fire HTTP webhooks on platform events. Use them to integrate with
|
||||
Slack, PagerDuty, Datadog, internal alerting services, or any HTTP receiver.
|
||||
|
||||
The dispatcher signs each delivery with HMAC-SHA256, retries on transient
|
||||
failures, tracks delivery health per webhook, and auto-disables endpoints that
|
||||
keep failing.
|
||||
|
||||
## Supported Events
|
||||
|
||||
The `WebhookEvent` type (`src/lib/webhookDispatcher.ts`) currently models:
|
||||
|
||||
| Event | Fires when |
|
||||
| -------------------- | --------------------------------------------------------- |
|
||||
| `request.completed` | A proxied request completes successfully |
|
||||
| `request.failed` | A proxied request fails after all retries/fallback |
|
||||
| `provider.error` | A provider returns an error eligible for circuit-breaking |
|
||||
| `provider.recovered` | A previously failing provider returns to a healthy state |
|
||||
| `quota.exceeded` | An API key crosses a budget/quota threshold |
|
||||
| `combo.switched` | A combo strategy switches its primary target |
|
||||
| `test.ping` | Synthetic event used by the test endpoint |
|
||||
|
||||
Subscriptions accept the literal `"*"` to receive every event. Unknown event
|
||||
names in `events` are ignored at dispatch time.
|
||||
|
||||
> Note: the dispatcher API is wired, but production call sites for some of the
|
||||
> non-`test.ping` events are still landing. Check `grep dispatchEvent` to see
|
||||
> which paths currently invoke the dispatcher in your release.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Caller (handler, service, monitor)
|
||||
dispatchEvent(event, data) [src/lib/webhookDispatcher.ts]
|
||||
-> getEnabledWebhooks() [src/lib/db/webhooks.ts]
|
||||
-> filter by webhook.events
|
||||
-> for each match (in parallel):
|
||||
deliverWebhook(url, payload, secret)
|
||||
build payload { event, timestamp, data }
|
||||
sign body with HMAC-SHA256 (if secret present)
|
||||
POST with 10s timeout
|
||||
retry up to 3 times on 5xx / network error
|
||||
recordWebhookDelivery(id, status, success)
|
||||
-> disableWebhooksWithHighFailures(10)
|
||||
```
|
||||
|
||||
Dispatch is fire-and-forget for the caller: `Promise.allSettled` swallows
|
||||
per-webhook errors so one bad receiver cannot block the others.
|
||||
|
||||
## HMAC Signing
|
||||
|
||||
When a webhook has a `secret`, OmniRoute signs the JSON body and sends:
|
||||
|
||||
```
|
||||
Content-Type: application/json
|
||||
User-Agent: OmniRoute-Webhook/1.0
|
||||
X-Webhook-Event: <event>
|
||||
X-Webhook-Timestamp: <ISO-8601>
|
||||
X-Webhook-Signature: sha256=<hex HMAC-SHA256(secret, body)>
|
||||
```
|
||||
|
||||
> Header names use the `X-Webhook-*` prefix (not `X-OmniRoute-*`). The signature
|
||||
> value is `sha256=<hex>` — verify the full prefix.
|
||||
|
||||
If `createWebhook` is called without a secret, the DB module generates one
|
||||
(`whsec_<48 hex>`) so all webhooks are signed by default.
|
||||
|
||||
### Verifying on the receiver
|
||||
|
||||
```typescript
|
||||
import { createHmac, timingSafeEqual } from "node:crypto";
|
||||
|
||||
function verify(rawBody: string, signature: string, secret: string) {
|
||||
const expected = "sha256=" + createHmac("sha256", secret).update(rawBody).digest("hex");
|
||||
const a = Buffer.from(expected);
|
||||
const b = Buffer.from(signature);
|
||||
return a.length === b.length && timingSafeEqual(a, b);
|
||||
}
|
||||
```
|
||||
|
||||
Always verify against the **raw** request body, before any JSON parsing.
|
||||
|
||||
## Retry & Failure Policy
|
||||
|
||||
`deliverWebhook(url, payload, secret, maxRetries = 3)`:
|
||||
|
||||
- 10 second timeout per attempt (`AbortController`).
|
||||
- HTTP 2xx counts as success.
|
||||
- HTTP 3xx/4xx counts as a non-retryable final status — recorded as delivered
|
||||
with `success = res.ok`.
|
||||
- HTTP 5xx and network errors are retried with exponential backoff:
|
||||
`2^attempt * 1000 ms` (1s, 2s, 4s).
|
||||
- After `maxRetries`, the delivery is recorded as failed.
|
||||
- Each delivery updates `last_triggered_at`, `last_status`, and either resets
|
||||
or increments `failure_count`.
|
||||
- The dispatcher calls `disableWebhooksWithHighFailures(10)` after each fan-out,
|
||||
so any webhook with `failure_count >= 10` is automatically disabled.
|
||||
|
||||
## Database
|
||||
|
||||
Table `webhooks` (migration `011_webhooks.sql`):
|
||||
|
||||
| Column | Type | Notes |
|
||||
| ------------------- | ------- | --------------------------------------------- |
|
||||
| `id` | TEXT PK | UUID |
|
||||
| `url` | TEXT | Destination URL |
|
||||
| `events` | TEXT | JSON array; default `["*"]` |
|
||||
| `secret` | TEXT | HMAC secret (auto-generated if not given) |
|
||||
| `enabled` | INT | 0/1; defaults to 1 |
|
||||
| `description` | TEXT | Optional human label |
|
||||
| `created_at` | TEXT | `datetime('now')` |
|
||||
| `last_triggered_at` | TEXT | Updated on every delivery attempt |
|
||||
| `last_status` | INT | HTTP status of the last attempt (0 = network) |
|
||||
| `failure_count` | INT | Resets to 0 on success, +1 on failure |
|
||||
|
||||
There is **no separate `webhook_deliveries` table** in the current schema —
|
||||
delivery history is aggregated on the `webhooks` row. If you need full audit
|
||||
history, consume `request.completed` / `audit` style events from a downstream
|
||||
log store.
|
||||
|
||||
## REST API
|
||||
|
||||
All endpoints require management auth (`requireManagementAuth`).
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
| ------------------------- | ------ | ------------------------------- |
|
||||
| `/api/webhooks` | GET | List webhooks (secrets masked) |
|
||||
| `/api/webhooks` | POST | Create webhook |
|
||||
| `/api/webhooks/[id]` | GET | Webhook detail (full secret) |
|
||||
| `/api/webhooks/[id]` | PUT | Update fields |
|
||||
| `/api/webhooks/[id]` | DELETE | Remove |
|
||||
| `/api/webhooks/[id]/test` | POST | Fire a `test.ping` (no retries) |
|
||||
|
||||
`GET /api/webhooks` masks the secret to `<first 10 chars>...` to avoid leaking
|
||||
on listing pages. Use the `[id]` GET when you actually need the secret.
|
||||
|
||||
### Create webhook
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/api/webhooks \
|
||||
-H "Cookie: auth_token=..." \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"url": "https://hooks.slack.com/services/...",
|
||||
"secret": "whsec_my_shared_secret",
|
||||
"events": ["quota.exceeded", "provider.error"],
|
||||
"description": "Slack alerts"
|
||||
}'
|
||||
```
|
||||
|
||||
If `secret` is omitted, the server generates a `whsec_<hex>` secret and returns
|
||||
it in the response.
|
||||
|
||||
### Test webhook
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/api/webhooks/<id>/test \
|
||||
-H "Cookie: auth_token=..."
|
||||
```
|
||||
|
||||
Returns `{ delivered, status, error }`. No retries are attempted — useful for
|
||||
quickly validating that the receiver accepts the payload and signature.
|
||||
|
||||
## Dashboard
|
||||
|
||||
The dashboard page at `/dashboard/webhooks` (see
|
||||
`src/app/(dashboard)/dashboard/webhooks/page.tsx`) provides:
|
||||
|
||||
- Create/edit webhooks with an event picker
|
||||
- Status indicator (active / inactive / errored) based on `enabled`,
|
||||
`failure_count`, and `last_status`
|
||||
- One-click test delivery
|
||||
- Manual enable/disable toggle
|
||||
|
||||
## Payload Examples
|
||||
|
||||
### request.completed
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "request.completed",
|
||||
"timestamp": "2026-05-13T20:30:00.123Z",
|
||||
"data": {
|
||||
"trace_id": "...",
|
||||
"api_key_id": "...",
|
||||
"provider": "openai",
|
||||
"model": "gpt-5",
|
||||
"status": 200,
|
||||
"tokens_in": 142,
|
||||
"tokens_out": 350,
|
||||
"cost_usd": 0.0042
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### provider.error
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "provider.error",
|
||||
"timestamp": "2026-05-13T20:31:00.000Z",
|
||||
"data": {
|
||||
"provider": "anthropic",
|
||||
"status": 503,
|
||||
"consecutive_failures": 5,
|
||||
"circuit_state": "open"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### test.ping
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "test.ping",
|
||||
"timestamp": "2026-05-13T20:32:00.000Z",
|
||||
"data": {
|
||||
"message": "Test webhook delivery from OmniRoute",
|
||||
"webhookId": "<uuid>"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Field shapes for non-`test.ping` events are defined by the call sites that emit
|
||||
them; treat the `data` object as forward-compatible (add fields, don't depend on
|
||||
absence).
|
||||
|
||||
## Best Practices
|
||||
|
||||
- **Verify the signature on every delivery** against the raw body — prevents
|
||||
spoofed POSTs from anyone who guesses your webhook URL.
|
||||
- **Respond 2xx within ~5 seconds** — the dispatcher times out at 10 s. Slow
|
||||
receivers will eat retries and inflate `failure_count`.
|
||||
- **Make handlers idempotent** — retries and at-least-once delivery semantics
|
||||
mean duplicates are possible.
|
||||
- **Subscribe minimally** — list only events you actually consume; `"*"` will
|
||||
add cost on receivers you do not control.
|
||||
- **Watch `failure_count`** — endpoints are auto-disabled at 10 consecutive
|
||||
failures; reset by calling `PUT /api/webhooks/[id]` with `enabled: true`
|
||||
after fixing the receiver.
|
||||
- **Rotate secrets periodically** — `PUT` a new `secret`, deploy the new value
|
||||
to the receiver, and confirm via the test endpoint.
|
||||
|
||||
## See Also
|
||||
|
||||
- [API_REFERENCE.md](./API_REFERENCE.md) — full management API surface
|
||||
- [RESILIENCE_GUIDE.md](./RESILIENCE_GUIDE.md) — circuit breaker / cooldown
|
||||
semantics that drive `provider.error` / `provider.recovered`
|
||||
- Source: `src/lib/webhookDispatcher.ts`, `src/lib/db/webhooks.ts`
|
||||
@@ -1,5 +1,13 @@
|
||||
# RFC: Auto-Assessment & Self-Healing Combo Engine
|
||||
|
||||
> ⚠️ **ARCHIVED (DEPRECATED) — 2026-05-13**
|
||||
>
|
||||
> This RFC was partially implemented in `src/domain/assessment/` (Phase 1 ~30%: `assessor.ts`, `categorizer.ts`, `selfHealer.ts` exist; persistence is in-memory only, scoped to `auto/*` models).
|
||||
>
|
||||
> **The broader generic eval framework that supplanted this RFC** lives at `src/lib/evals/` and is documented in [EVALS.md](../EVALS.md). For Auto Combo scoring (which absorbed the "self-healing" direction), see [AUTO-COMBO.md](../AUTO-COMBO.md) and `open-sse/services/autoCombo/`.
|
||||
>
|
||||
> Kept here for historical context. Do not implement against this document — refer to the source files and the docs above.
|
||||
|
||||
## Summary
|
||||
|
||||
Omniroute's combo system currently requires manual configuration: users must know which providers and models are actually working, then manually wire them into combo chains. When providers fail (rate limits, auth errors, model deprecation), combos silently degrade — routing to dead endpoints that timeout or return errors. There is no automated way to:
|
||||
@@ -1,6 +1,8 @@
|
||||
# OmniRoute MCP Server
|
||||
|
||||
> **Model Context Protocol server** that exposes OmniRoute's gateway intelligence as **16 tools** for AI agents.
|
||||
> **Model Context Protocol server** that exposes OmniRoute's gateway intelligence as **37 tools** for AI agents.
|
||||
>
|
||||
> **Source of truth for the full tool catalog and REST surface:** [`docs/MCP-SERVER.md`](../../docs/MCP-SERVER.md). This README focuses on architecture, configuration, and integration examples; the catalog below is a summary subset.
|
||||
|
||||
The MCP Server allows any AI agent (Claude Desktop, Cursor, VS Code Copilot, custom agents) to **monitor, control, and optimize** the OmniRoute AI gateway programmatically.
|
||||
|
||||
@@ -18,8 +20,9 @@ The MCP Server allows any AI agent (Claude Desktop, Cursor, VS Code Copilot, cus
|
||||
┌──────────────────────────────────────────────────────────────────┐
|
||||
│ OmniRoute MCP Server │
|
||||
│ ┌──────────────┐ ┌─────────────────┐ ┌────────────────────┐ │
|
||||
│ │ Scope │ │ 16 MCP Tools │ │ Audit Logger │ │
|
||||
│ │ Enforcement │──│ (Phase 1 + 2) │──│ (SHA-256/SQLite) │ │
|
||||
│ │ Scope │ │ 37 MCP Tools │ │ Audit Logger │ │
|
||||
│ │ Enforcement │──│ (core + memory │──│ (SHA-256/SQLite) │ │
|
||||
│ │ │ │ + skills + …) │ │ │ │
|
||||
│ └──────────────┘ └────────┬────────┘ └────────────────────┘ │
|
||||
└─────────────────────────────┼────────────────────────────────────┘
|
||||
│ HTTP (internal)
|
||||
|
||||
@@ -63,6 +63,7 @@
|
||||
"scripts": {
|
||||
"dev": "node scripts/run-next.mjs dev",
|
||||
"prebuild:docs": "node scripts/generate-docs-index.mjs",
|
||||
"gen:provider-reference": "node --import tsx/esm scripts/gen-provider-reference.ts",
|
||||
"build": "node scripts/build-next-isolated.mjs",
|
||||
"build:cli": "node --import tsx/esm scripts/prepublish.ts",
|
||||
"start": "node scripts/run-next.mjs start",
|
||||
|
||||
192
scripts/gen-provider-reference.ts
Normal file
192
scripts/gen-provider-reference.ts
Normal file
@@ -0,0 +1,192 @@
|
||||
#!/usr/bin/env node
|
||||
// Generates docs/PROVIDER_REFERENCE.md from src/shared/constants/providers.ts.
|
||||
// Run: node --import tsx/esm scripts/gen-provider-reference.ts
|
||||
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import {
|
||||
FREE_PROVIDERS,
|
||||
OAUTH_PROVIDERS,
|
||||
WEB_COOKIE_PROVIDERS,
|
||||
APIKEY_PROVIDERS,
|
||||
LOCAL_PROVIDERS,
|
||||
SEARCH_PROVIDERS,
|
||||
AUDIO_ONLY_PROVIDERS,
|
||||
UPSTREAM_PROXY_PROVIDERS,
|
||||
CLOUD_AGENT_PROVIDERS,
|
||||
SYSTEM_PROVIDERS,
|
||||
IMAGE_ONLY_PROVIDER_IDS,
|
||||
AGGREGATOR_PROVIDER_IDS,
|
||||
ENTERPRISE_CLOUD_PROVIDER_IDS,
|
||||
VIDEO_PROVIDER_IDS,
|
||||
EMBEDDING_RERANK_PROVIDER_IDS,
|
||||
SELF_HOSTED_CHAT_PROVIDER_IDS,
|
||||
} from "../src/shared/constants/providers.ts";
|
||||
|
||||
const __dirname = path.dirname(fileURLToPath(import.meta.url));
|
||||
const ROOT = path.resolve(__dirname, "..");
|
||||
const OUT_FILE = path.join(ROOT, "docs", "PROVIDER_REFERENCE.md");
|
||||
|
||||
type ProviderRecord = {
|
||||
id: string;
|
||||
alias?: string | undefined;
|
||||
name: string;
|
||||
icon?: string;
|
||||
color?: string;
|
||||
textIcon?: string;
|
||||
website?: string;
|
||||
authHint?: string;
|
||||
freeNote?: string;
|
||||
hasFree?: boolean;
|
||||
deprecated?: boolean;
|
||||
deprecationReason?: string;
|
||||
[k: string]: unknown;
|
||||
};
|
||||
|
||||
function asRecords(map: Record<string, ProviderRecord>): ProviderRecord[] {
|
||||
return Object.values(map).map((p) => ({ ...p }));
|
||||
}
|
||||
|
||||
function escapeCell(value: string | undefined): string {
|
||||
if (!value) return "—";
|
||||
return value.replace(/\|/g, "\\|").replace(/\n/g, " ");
|
||||
}
|
||||
|
||||
function row(p: ProviderRecord, category: string): string {
|
||||
const alias = p.alias ? `\`${p.alias}\`` : "—";
|
||||
const hint = p.deprecated
|
||||
? `⚠️ **DEPRECATED.** ${escapeCell(p.deprecationReason)}`
|
||||
: escapeCell(p.authHint || p.freeNote);
|
||||
const link = p.website ? `[link](${p.website})` : "—";
|
||||
return `| \`${p.id}\` | ${alias} | ${escapeCell(p.name)} | ${category} | ${link} | ${hint} |`;
|
||||
}
|
||||
|
||||
function categoryTags(id: string): string[] {
|
||||
const tags: string[] = [];
|
||||
if (IMAGE_ONLY_PROVIDER_IDS.has(id)) tags.push("image");
|
||||
if (VIDEO_PROVIDER_IDS.has(id)) tags.push("video");
|
||||
if (AGGREGATOR_PROVIDER_IDS.has(id)) tags.push("aggregator");
|
||||
if (ENTERPRISE_CLOUD_PROVIDER_IDS.has(id)) tags.push("enterprise");
|
||||
if (EMBEDDING_RERANK_PROVIDER_IDS.has(id)) tags.push("embed/rerank");
|
||||
if (SELF_HOSTED_CHAT_PROVIDER_IDS.has(id)) tags.push("self-hosted");
|
||||
return tags;
|
||||
}
|
||||
|
||||
function sortById(rows: ProviderRecord[]): ProviderRecord[] {
|
||||
return [...rows].sort((a, b) => a.id.localeCompare(b.id));
|
||||
}
|
||||
|
||||
function buildSection(title: string, rows: ProviderRecord[], category: string): string {
|
||||
if (rows.length === 0) return "";
|
||||
const lines: string[] = [];
|
||||
lines.push(`## ${title} (${rows.length})\n`);
|
||||
lines.push("| ID | Alias | Name | Tags | Website | Notes |");
|
||||
lines.push("|----|-------|------|------|---------|-------|");
|
||||
for (const p of sortById(rows)) {
|
||||
const tags = [category, ...categoryTags(p.id)].join(", ");
|
||||
lines.push(row(p, tags));
|
||||
}
|
||||
lines.push("");
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
function buildHeader(total: number): string {
|
||||
const date = new Date().toISOString().slice(0, 10);
|
||||
return [
|
||||
"# Provider Reference",
|
||||
"",
|
||||
`> **Auto-generated** from \`src/shared/constants/providers.ts\` — do not edit by hand.`,
|
||||
`> Regenerate with: \`npm run gen:provider-reference\``,
|
||||
`> **Last generated:** ${date}`,
|
||||
"",
|
||||
`Total providers: **${total}**. See category breakdown below.`,
|
||||
"",
|
||||
"## Categories",
|
||||
"",
|
||||
"- **Free** — free tier with API key (configured via dashboard)",
|
||||
"- **OAuth** — sign-in flow handled by OmniRoute, no API key needed",
|
||||
"- **Web cookie** — wraps the provider's web app via cookie auth",
|
||||
"- **API key** — paid provider configured via API key (free credits may apply)",
|
||||
"- **Local** — runs on the user's machine (Ollama, LM Studio, vLLM, etc.)",
|
||||
"- **Search** — web search providers",
|
||||
"- **Audio** — audio-only providers (TTS/STT)",
|
||||
"- **Upstream proxy** — providers that proxy to other providers",
|
||||
"- **Cloud agent** — long-running coding agents (Codex Cloud, Devin, Jules)",
|
||||
"- **System** — OmniRoute-internal providers (loopback, etc.)",
|
||||
"",
|
||||
"Additional tags: `image`, `video`, `aggregator`, `enterprise`, `embed/rerank`, `self-hosted`.",
|
||||
"",
|
||||
"Use the dashboard at `/dashboard/providers` to enable, configure, and test each provider.",
|
||||
"",
|
||||
"---",
|
||||
"",
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
function main() {
|
||||
const free = asRecords(FREE_PROVIDERS);
|
||||
const oauth = asRecords(OAUTH_PROVIDERS);
|
||||
const webCookie = asRecords(WEB_COOKIE_PROVIDERS);
|
||||
const apiKey = asRecords(APIKEY_PROVIDERS);
|
||||
const local = asRecords(LOCAL_PROVIDERS);
|
||||
const search = asRecords(SEARCH_PROVIDERS);
|
||||
const audio = asRecords(AUDIO_ONLY_PROVIDERS);
|
||||
const upstreamProxy = asRecords(UPSTREAM_PROXY_PROVIDERS);
|
||||
const cloudAgent = asRecords(CLOUD_AGENT_PROVIDERS);
|
||||
const system = asRecords(SYSTEM_PROVIDERS);
|
||||
|
||||
const allIds = new Set<string>([
|
||||
...free.map((p) => p.id),
|
||||
...oauth.map((p) => p.id),
|
||||
...webCookie.map((p) => p.id),
|
||||
...apiKey.map((p) => p.id),
|
||||
...local.map((p) => p.id),
|
||||
...search.map((p) => p.id),
|
||||
...audio.map((p) => p.id),
|
||||
...upstreamProxy.map((p) => p.id),
|
||||
...cloudAgent.map((p) => p.id),
|
||||
...system.map((p) => p.id),
|
||||
]);
|
||||
|
||||
const sections = [
|
||||
buildSection("Free Tier (OAuth-first or no-key)", free, "Free"),
|
||||
buildSection("OAuth Providers", oauth, "OAuth"),
|
||||
buildSection("Web Cookie Providers", webCookie, "Web cookie"),
|
||||
buildSection("API Key Providers (paid / paid-with-free-credits)", apiKey, "API key"),
|
||||
buildSection("Local Providers", local, "Local"),
|
||||
buildSection("Search Providers", search, "Search"),
|
||||
buildSection("Audio-only Providers", audio, "Audio"),
|
||||
buildSection("Upstream Proxy Providers", upstreamProxy, "Upstream proxy"),
|
||||
buildSection("Cloud Agent Providers", cloudAgent, "Cloud agent"),
|
||||
buildSection("System Providers", system, "System"),
|
||||
];
|
||||
|
||||
const footer = [
|
||||
"## Sources of truth",
|
||||
"",
|
||||
"- Catalog: [`src/shared/constants/providers.ts`](../src/shared/constants/providers.ts)",
|
||||
"- Registry (per-model details): [`open-sse/config/providerRegistry.ts`](../open-sse/config/providerRegistry.ts)",
|
||||
"- Executors: [`open-sse/executors/`](../open-sse/executors/) (31 files)",
|
||||
"- Translators: [`open-sse/translator/`](../open-sse/translator/)",
|
||||
"",
|
||||
"## See Also",
|
||||
"",
|
||||
"- [FREE_TIERS.md](./FREE_TIERS.md) — curated free-tier guide",
|
||||
"- [USER_GUIDE.md](./USER_GUIDE.md) — provider setup walkthrough",
|
||||
"- [ARCHITECTURE.md](./ARCHITECTURE.md) — overall architecture",
|
||||
"",
|
||||
].join("\n");
|
||||
|
||||
const content = buildHeader(allIds.size) + sections.join("\n") + "\n" + footer;
|
||||
fs.writeFileSync(OUT_FILE, content);
|
||||
console.log(`✓ Wrote ${OUT_FILE}`);
|
||||
console.log(` Providers: ${allIds.size} unique IDs`);
|
||||
console.log(
|
||||
` Sections: free=${free.length}, oauth=${oauth.length}, web=${webCookie.length}, ` +
|
||||
`apikey=${apiKey.length}, local=${local.length}, search=${search.length}, ` +
|
||||
`audio=${audio.length}, proxy=${upstreamProxy.length}, cloud=${cloudAgent.length}, system=${system.length}`
|
||||
);
|
||||
}
|
||||
|
||||
main();
|
||||
Reference in New Issue
Block a user