From a56c76a330e6cf03f1bb76e66459d107ac3ddc36 Mon Sep 17 00:00:00 2001 From: Diego Rodrigues de Sa e Souza <8016841+diegosouzapw@users.noreply.github.com> Date: Sat, 20 Jun 2026 11:02:48 -0300 Subject: [PATCH] =?UTF-8?q?docs:=20feature-documentation=20catch-up=20(v3.?= =?UTF-8?q?8.20=20=E2=86=92=20v3.8.30)=20(#4391)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One-time reconciliation of the docs with every user-facing feature shipped since v3.8.20 (we had never done a dedicated pass, so debt had accumulated): - README: new '✨ What's New' section (curated v3.8.20→v3.8.30 highlights). - New guides: CLI-INTEGRATIONS (all setup-*/launch commands), MITM-TPROXY-DECRYPT (transparent-decrypt epic), CONTEXT_EDITING (delegated Anthropic clear_tool_uses). - Refreshed: AUTO-COMBO (auto/: + Arena-ELO), API_REFERENCE (x-omniroute-no-memory), MEMORY (int8 quantization + off-by-default), RESILIENCE (model-lockout success-decay), RTK, AGENTBRIDGE, TRAFFIC_INSPECTOR, GUARDRAILS, CLOUD_AGENT, ENVIRONMENT, SETUP_GUIDE, CLI-TOOLS, MCP-SERVER. - Regenerated PROVIDER_REFERENCE (231 providers); synced the count in README/CLAUDE/AGENTS. - Allowlisted external-tool env vars (OPENAI_API_BASE, PROMPTFOO_PROVIDER_KEY) and the STREAM_RECOVERY config-object name in the docs-accuracy gates. All claims source-verified; check:docs-all (sync/counts/env/links/fabricated) passes. Going forward this runs every release via generate-release step 6b. --- AGENTS.md | 4 +- CHANGELOG.md | 4 + CLAUDE.md | 2 +- README.md | 40 +- docs/architecture/RESILIENCE_GUIDE.md | 45 ++- docs/compression/CONTEXT_EDITING.md | 195 +++++++++ docs/compression/RTK_COMPRESSION.md | 77 +++- docs/frameworks/AGENTBRIDGE.md | 95 ++++- docs/frameworks/CLOUD_AGENT.md | 6 +- docs/frameworks/MCP-SERVER.md | 33 +- docs/frameworks/MEMORY.md | 45 ++- docs/frameworks/TRAFFIC_INSPECTOR.md | 73 +++- docs/guides/CLI-INTEGRATIONS.md | 208 ++++++++++ docs/guides/SETUP_GUIDE.md | 33 +- docs/reference/API_REFERENCE.md | 1 + docs/reference/CLI-TOOLS.md | 37 +- docs/reference/ENVIRONMENT.md | 30 +- docs/reference/PROVIDER_REFERENCE.md | 508 ++++++++++++------------ docs/routing/AUTO-COMBO.md | 23 +- docs/security/GUARDRAILS.md | 31 +- docs/security/MITM-TPROXY-DECRYPT.md | 381 ++++++++++++++++++ scripts/check/check-env-doc-sync.mjs | 4 + scripts/check/check-fabricated-docs.mjs | 2 + 23 files changed, 1563 insertions(+), 314 deletions(-) create mode 100644 docs/compression/CONTEXT_EDITING.md create mode 100644 docs/guides/CLI-INTEGRATIONS.md create mode 100644 docs/security/MITM-TPROXY-DECRYPT.md diff --git a/AGENTS.md b/AGENTS.md index 662066d6ad..2382d21727 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -3,12 +3,12 @@ ## Project Unified AI proxy/router — route any LLM through one endpoint. Multi-provider support -with **227 provider entries** (OpenAI, Anthropic, Gemini, DeepSeek, Groq, xAI, Mistral, Fireworks, +with **231 provider entries** (OpenAI, Anthropic, Gemini, DeepSeek, Groq, xAI, Mistral, Fireworks, Cohere, NVIDIA, Cerebras, Pollinations, Puter, Cloudflare AI, HuggingFace, DeepInfra, SambaNova, Meta Llama API, Moonshot AI, AI21 Labs, Databricks, Snowflake, and many more) with **MCP Server** (87 tools), **A2A v0.3 Protocol**, and **Electron desktop app**. -> **Live counts (v3.8.24)**: providers 227 · MCP tools 87 · MCP scopes 30 · A2A skills 6 · +> **Live counts (v3.8.31)**: providers 231 · MCP tools 87 · MCP scopes 30 · A2A skills 6 · > open-sse services 115 · routing strategies 15 · auto-combo scoring factors 9 · > DB modules 83 · DB migrations 97 · base tables 17 · search providers 11 · > i18n locales 42. **Refresh with `npm run check:docs-all`.** diff --git a/CHANGELOG.md b/CHANGELOG.md index 5c4803349a..3ad5e73553 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,6 +26,10 @@ _In development — bullets added per PR; finalized at release._ - **fix(db): cascade-delete orphaned model aliases when a provider is removed** — deleting a custom provider removed its connections and node but left behind the imported model-alias rows (stored as `key=`, `value="/"`). Those stale aliases then blocked re-importing the same provider — the import dedup treated them as "already exists", so no new models appeared. A new `deleteModelAliasesForProvider(providerId)` DB helper drops every alias whose stored value begins with `/` (leaving other providers and user-defined settings aliases untouched), and the provider-node DELETE handler now calls it after removing the connections and node, so a fresh import is unblocked. (thanks @nguyenvanhuy0612) - **fix(api): persist `max_input_tokens` / `max_output_tokens` when adding a custom model** — `POST /api/provider-models` silently dropped the per-model token limits set in the "add custom model" form: the handler destructured the rest of the body but never read `max_input_tokens` / `max_output_tokens`, and `addCustomModel()` had no parameter for them, so the values were thrown away on write. The DB layer (`inputTokenLimit` / `outputTokenLimit`) and the `/v1/models` catalog already round-trip these fields — only the write path was missing. The validation schema now accepts the two optional limits, the handler forwards them, and `addCustomModel()` persists them so a custom model's context/output window survives into the catalog. (thanks @codename-zen) +### 📝 Maintenance + +- **docs: one-time feature-documentation catch-up (v3.8.20 → v3.8.30)** — reconciled the docs with every user-facing feature shipped since v3.8.20: a new README **✨ What's New** section; new guides for [CLI integrations](docs/guides/CLI-INTEGRATIONS.md), [MITM TPROXY transparent decrypt](docs/security/MITM-TPROXY-DECRYPT.md) and [delegated Anthropic Context Editing](docs/compression/CONTEXT_EDITING.md); refreshed AUTO-COMBO (`auto/:` + Arena-ELO), API_REFERENCE (`x-omniroute-no-memory`), MEMORY (int8 quantization, off-by-default), RESILIENCE (model-lockout success-decay), RTK, AGENTBRIDGE, TRAFFIC_INSPECTOR, GUARDRAILS, CLOUD_AGENT, ENVIRONMENT; regenerated PROVIDER_REFERENCE (231 providers) and synced the provider count in README/CLAUDE/AGENTS. Going forward this runs every release (generate-release step 6b). ([#4391](https://github.com/diegosouzapw/OmniRoute/pull/4391)) + --- ## [3.8.30] — 2026-06-20 diff --git a/CLAUDE.md b/CLAUDE.md index 3e308c5367..86763792a1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -35,7 +35,7 @@ For full test matrix, see `CONTRIBUTING.md` → "Running Tests". For deep archit ## Project at a Glance -**OmniRoute** — unified AI proxy/router. One endpoint, 227 LLM providers, auto-fallback. +**OmniRoute** — unified AI proxy/router. One endpoint, 231 LLM providers, auto-fallback. | Layer | Location | Purpose | | ------------- | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | diff --git a/README.md b/README.md index 767a985092..fe1f5018e2 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ # 🚀 OmniRoute — The Free AI Gateway -### Never stop coding. Connect every AI tool to **227 providers** — **50+ free** — through one endpoint. +### Never stop coding. Connect every AI tool to **231 providers** — **50+ free** — through one endpoint. **Plug Claude Code, Codex, Cursor, Cline, Copilot & Antigravity into FREE Claude / GPT / Gemini. Auto-fallback.**
@@ -19,8 +19,8 @@
-[![227 AI Providers](https://img.shields.io/badge/227-AI_Providers-6C5CE7?style=for-the-badge)](#-227-ai-providers--50-free) -[![50+ Free](https://img.shields.io/badge/50%2B-Free_Tiers-00B894?style=for-the-badge)](#-227-ai-providers--50-free) +[![231 AI Providers](https://img.shields.io/badge/231-AI_Providers-6C5CE7?style=for-the-badge)](#-231-ai-providers--50-free) +[![50+ Free](https://img.shields.io/badge/50%2B-Free_Tiers-00B894?style=for-the-badge)](#-231-ai-providers--50-free) [![1.6B Free Tokens/mo](https://img.shields.io/badge/1.6B-Free_Tokens%2Fmo-00B894?style=for-the-badge)](docs/reference/FREE_TIERS.md) [![Token Savings](https://img.shields.io/badge/up_to_95%25-Token_Savings-E17055?style=for-the-badge)](#%EF%B8%8F-save-1595-tokens--automatically) [![15 Strategies](https://img.shields.io/badge/15-Routing_Strategies-0984E3?style=for-the-badge)](#-combos--the-flagship) @@ -59,7 +59,7 @@
-[**🚀 Quick Start**](#-quick-start) • [**🎯 Combos**](#-combos--the-flagship) • [**🌐 Providers**](#-227-ai-providers--50-free) • [**🔌 CLI & MCP**](#-full-cli--a2a--mcp) • [**🗜️ Compression**](#%EF%B8%8F-save-1595-tokens--automatically) • [**🌍 Website**](https://omniroute.online) +[**🚀 Quick Start**](#-quick-start) • [**🎯 Combos**](#-combos--the-flagship) • [**🌐 Providers**](#-231-ai-providers--50-free) • [**🔌 CLI & MCP**](#-full-cli--a2a--mcp) • [**🗜️ Compression**](#%EF%B8%8F-save-1595-tokens--automatically) • [**🌍 Website**](https://omniroute.online) [💥 The Promise](#-the-promise) • [🤔 Why](#-why-omniroute) • [🏆 What Sets Apart](#-what-sets-omniroute-apart) • [🤖 Compatible CLIs](#-compatible-clis--coding-agents) • [🖥️ Where It Runs](#%EF%B8%8F-where-omniroute-runs--anywhere) • [🔒 Private](#-private--local-first) • [🎬 In Action](#-omniroute-in-action) • [📚 Explore More](#-explore-more) • [📧 Support](#-support--community) @@ -137,11 +137,11 @@ -> One endpoint. **227 providers.** Never stop building — and let OmniRoute pick the cheapest one that works. +> One endpoint. **231 providers.** Never stop building — and let OmniRoute pick the cheapest one that works. - + @@ -264,7 +264,7 @@ Result: 4 layers of fallback = zero downtime | Feature | OmniRoute | Other routers | | -------------------------------------- | ----------------------------------------------------------- | ------------- | -| 🌐 Providers | **227** | 20–100 | +| 🌐 Providers | **231** | 20–100 | | 🆓 Free providers | **50+ (11 free forever)** | 1–5 | | 🔀 Routing strategies | **15** (priority, weighted, cost-optimized, context-relay…) | 1–3 | | 🗜️ Token compression | **RTK + Caveman stacked (15–95%)** | None / 20–40% | @@ -283,6 +283,26 @@ Result: 4 layers of fallback = zero downtime
+# ✨ What's New + +
+ +> Recent highlights from **v3.8.20 → v3.8.30**. Full history in [`CHANGELOG.md`](CHANGELOG.md). + +- **🤖 One-command CLI/agent setup** — a dedicated `setup-*` command configures each coding tool to route through OmniRoute (Claude Code, Codex, Cline, Continue, Cursor, Roo Code, Kilo Code, Crush, Goose, Qwen Code, Aider, OpenCode, Gemini CLI); `omniroute launch` / `omniroute launch-codex` are zero-config launchers. → [CLI Integrations](docs/guides/CLI-INTEGRATIONS.md) +- **🛰️ Remote mode** — drive a remote OmniRoute from any machine with scoped access tokens (`omniroute connect` / `omniroute contexts` / `omniroute tokens`). → [Remote Mode](docs/guides/REMOTE-MODE.md) +- **🧭 Smarter auto-routing** — OpenRouter-style `auto/:` combos (e.g. `auto/coding:fast`, `auto/reasoning:pro`), live Arena-ELO + models.dev model intelligence, and per-step account allowlists. → [Auto-Combo](docs/routing/AUTO-COMBO.md) +- **🗜️ Pluggable compression** — an async compression pipeline with Compression Studios, a stable LLMLingua-2 ONNX engine, RTK, and delegated Anthropic Context Editing. → [Compression](docs/compression/COMPRESSION_ENGINES.md) +- **🕵️ Transparent MITM decrypt (TPROXY)** — capture & translate traffic from CLIs that ignore proxy env vars, with a per-SNI certificate authority and a trust-store installer. → [MITM/TPROXY](docs/security/MITM-TPROXY-DECRYPT.md) +- **💸 Cost telemetry everywhere** — `X-OmniRoute-*` cost/usage headers on every endpoint (including media), a non-token cost engine, a cache-HIT `X-OmniRoute-Cost-Saved` header, and per-key USD spend quotas. → [API Reference](docs/reference/API_REFERENCE.md) +- **🧠 Memory you control** — opt-in int8 vector quantization (Qdrant + sqlite-vec), memory off by default, and a per-request `x-omniroute-no-memory` header. → [Memory](docs/frameworks/MEMORY.md) +- **🛡️ Security** — a prompt-injection guard across every LLM route (backed by a red-team suite), plus a free DuckDuckGo last-resort web search. → [Guardrails](docs/security/GUARDRAILS.md) +- **🤝 More providers & agents** — Cursor Cloud Agent (a 4th cloud agent), a refreshed 231-provider catalog (OrcaRouter, Wafer AI, OpenAdapter, dit.ai, TokenRouter, …), and Vertex AI media generation (speech / transcription / music / video). → [Providers](docs/reference/PROVIDER_REFERENCE.md) + +
+ +
+ # 🤖 Compatible CLIs & Coding Agents > One config — `http://localhost:20128/v1` — and **every** AI IDE or CLI runs on free & low-cost models. @@ -320,11 +340,11 @@ Result: 4 layers of fallback = zero downtime
-# 🌐 227 AI Providers — 50+ Free +# 🌐 231 AI Providers — 50+ Free
-> The most complete catalog of any open-source router: **227 providers**, **50+ with a free tier**, **11 free forever**. +> The most complete catalog of any open-source router: **231 providers**, **50+ with a free tier**, **11 free forever**.
@@ -767,7 +787,7 @@ Compression: aggressive (~50%) → double your free quota · Cost: $0/mo **Will I be charged by OmniRoute?** No — it's free, open-source software on your machine. You only pay paid providers directly. OmniRoute has no billing system. **Are FREE providers really unlimited?** Yes — Kiro, Qoder, Pollinations, LongCat, Cloudflare. No catch. **Will compression hurt quality?** No — it only compresses the **input**; code, URLs, JSON are always protected. -**Does it work where AI is blocked?** Yes — 3-level proxy + 1proxy marketplace reach all 227 providers. +**Does it work where AI is blocked?** Yes — 3-level proxy + 1proxy marketplace reach all 231 providers. 📖 [User Guide](docs/guides/USER_GUIDE.md) · [API Reference](docs/reference/API_REFERENCE.md) · [Environment Config](docs/reference/ENVIRONMENT.md) diff --git a/docs/architecture/RESILIENCE_GUIDE.md b/docs/architecture/RESILIENCE_GUIDE.md index 92e2ba0779..d64af0a745 100644 --- a/docs/architecture/RESILIENCE_GUIDE.md +++ b/docs/architecture/RESILIENCE_GUIDE.md @@ -1,7 +1,7 @@ --- title: "Resilience Guide" -version: 3.8.18 -lastUpdated: 2026-06-09 +version: 3.8.31 +lastUpdated: 2026-06-20 --- # Resilience Guide @@ -116,6 +116,47 @@ Lists active lockouts with: provider, connection, model, reason, expiresAt. Oper - `GET /api/resilience/model-cooldowns` — list active lockouts - `DELETE /api/resilience/model-cooldowns` — manual re-enable. Body: `{provider, connection, model}`. Auth: management. +### Lockout settings UI + success-decay recovery (v3.8.23) + +Model lockout went from always-on hardcoded behavior to a fully configurable, +opt-in feature with its own settings card and a self-healing recovery path. + +**Settings card:** Settings → Model Lockout +(`src/app/(dashboard)/dashboard/settings/components/ModelLockoutCard.tsx`). +This is **distinct** from the read-only `ModelCooldownsCard` above (which only +*lists* active lockouts) — the new card *configures the parameters*. Defaults +live in `DEFAULT_MODEL_LOCKOUT_SETTINGS` +(`src/lib/resilience/modelLockoutSettings.ts`): + +| Setting | Default | Meaning | +| ----------------------- | -------------------------------- | --------------------------------------------------------------- | +| `enabled` | `false` | Master toggle — model lockout is **off by default**. | +| `errorCodes` | `[403, 404, 429, 502, 503, 504]` | Upstream statuses that count as a model-scoped failure. | +| `baseCooldownMs` | `120_000` (120 s) | Initial lockout duration for the first failure. | +| `maxCooldownMs` | `1_800_000` (30 min) | Cap on the escalated cooldown. | +| `maxBackoffSteps` | `10` | Max exponential-backoff escalation steps. | +| `useExponentialBackoff` | `true` | Whether repeated failures escalate the cooldown exponentially. | + +Settings persist through the normal settings store and validate via the +resilience settings schema; the card clamps `baseCooldownMs`/`maxCooldownMs` +(with `maxCooldownMs ≥ baseCooldownMs`) and `maxBackoffSteps`. + +**Success-decay recovery:** recovery is **not** purely timer expiry. A healthy +response walks the model's failure count back down so a model that recovered +mid-window stops escalating (and clears) before its timer would. On a successful +combo target, `open-sse/services/combo.ts` calls `decayModelFailureCount()` +(`open-sse/services/accountFallback.ts`), which **halves** the stored +`failureCount` (`Math.floor(failureCount / 2)`); when it reaches `0` the lockout +entry is deleted entirely. The counterpart `recordModelLockoutFailure()` +increments the count (and escalates the cooldown) on failures within the +escalation window. This success-decay is in addition to plain timer expiry — +either path can re-enable a model. + +**State:** lockouts are held **in-memory** (per-process `Map`s of +`ModelLockoutEntry` keyed by `provider:connectionId:model`), not persisted to +the DB — they are lost on restart. The *settings* are persisted; the active +lockout *state* is ephemeral. + --- ## Other Resilience Features diff --git a/docs/compression/CONTEXT_EDITING.md b/docs/compression/CONTEXT_EDITING.md new file mode 100644 index 0000000000..66ecec459e --- /dev/null +++ b/docs/compression/CONTEXT_EDITING.md @@ -0,0 +1,195 @@ +--- +title: "Delegated Context Editing (Anthropic)" +version: 3.8.31 +lastUpdated: 2026-06-20 +--- + +# Delegated Context Editing (Anthropic) + +Delegated **Context Editing** is a Claude-only context-management feature. Unlike OmniRoute's local +compression engines (Caveman, RTK, LLMLingua, stacked pipelines) — which rewrite the request body +*before* it leaves the proxy — Context Editing asks the **provider** to clear stale +tool-use / tool-result blocks from its own running context window. OmniRoute only attaches a body +parameter (`context_management.edits[]`); Claude does the actual clearing against its own tokenizer. + +This is a delegated capability by nature: other providers reject the parameter, so OmniRoute scopes +it strictly to Claude and Claude-Code-compatible relays. + +Source of truth: `open-sse/config/contextEditing.ts` (strategy ids, body injection, telemetry +extraction), `open-sse/executors/base.ts` (injection gate + 400-fallback), and +`open-sse/services/compression/types.ts` (config shape + default). + +## What `clear_tool_uses` does + +OmniRoute injects a single edit into the outbound Anthropic Messages body: + +```json +{ + "context_management": { + "edits": [ + { + "type": "clear_tool_uses_20250919", + "trigger": { "type": "input_tokens", "value": 100000 }, + "keep": { "type": "tool_uses", "value": 3 } + } + ] + } +} +``` + +- `type: "clear_tool_uses_20250919"` — the dated Anthropic strategy id (`CLEAR_TOOL_USES_STRATEGY`). +- `trigger.value: 100000` — once the request's input tokens exceed this threshold, Claude begins + clearing old tool-use/result pairs (`CONTEXT_EDITING_DEFAULT_TRIGGER_TOKENS`, Anthropic's default). +- `keep.value: 3` — the N most recent tool-use/result pairs are kept untouched + (`CONTEXT_EDITING_DEFAULT_KEEP_TOOL_USES`). + +The beta is advertised via the `anthropic-beta: context-management-2025-06-27` header, which +OmniRoute already emits on Claude requests. + +Injection is performed by `applyContextEditingToBody()` and is **idempotent**: if a `clear_tool_uses` +edit already exists on the body (added by a previous call or supplied by the client), the body is +left as-is. If a `clear_thinking_20251015` edit is also present, OmniRoute stable-sorts the +`clear_thinking` edit to the front, because Anthropic requires `clear_thinking` to precede +`clear_tool_uses` in the `edits[]` array. + +## The per-combo enable toggle + +Context Editing is **off by default** and opt-in. The toggle is a single boolean carried in the +compression config: + +- Setting key: `contextEditing.enabled` (camelCase — **not** `context_editing` / `context-editing`). +- Type: `ContextEditingConfig { enabled: boolean }` in + `open-sse/services/compression/types.ts`. +- Default: `DEFAULT_CONTEXT_EDITING_CONFIG = { enabled: false }`. +- Zod schema: `contextEditingConfigSchema` in `src/shared/validation/compressionConfigSchemas.ts`. +- Storage: persisted with the rest of the compression settings (normalized in + `src/lib/db/compression.ts`). + +In the dashboard the toggle lives in the compression hub +(`src/app/(dashboard)/dashboard/context/combos/CompressionHub.tsx`) and writes +`{ contextEditing: { enabled: … } }` back through `saveSettings()`. Because it rides on the +compression-settings object, it composes with the per-combo compression profile rather than being a +fully independent surface — the config carries only the on/off flag; all thresholds (`trigger`, +`keep`) are the constants documented above. + +## Claude-only gating + +Injection only happens for genuine Claude or Claude-Code-compatible relays. The gate in +`open-sse/executors/base.ts` is: + +```ts +if ( + (this.provider === "claude" || isClaudeCodeCompatible(this.provider)) && + contextEditing?.enabled && + !contextEditingDisabled +) { + applyContextEditingToBody(transformedBody, { enabled: true }); +} +``` + +- `this.provider === "claude"` — real Anthropic key/OAuth. +- `isClaudeCodeCompatible(this.provider)` — relays whose provider id starts with the + `anthropic-compatible-cc-` prefix (they advertise Claude Code compatibility, so they are the relays + most likely to accept the beta). See `open-sse/services/provider.ts`. + +Deliberately **excluded**: + +- `claude-web` — a browser relay with a `create_conversation_params` request shape that never sees + `context_management`. +- Generic `anthropic-compatible-*` relays (without the `-cc-` prefix) — third-party endpoints with + uncertain beta support. + +Non-Claude providers never receive the `context_management` parameter even when the toggle is on. + +## The 400-fallback / relay coverage + +A Claude-compatible relay may advertise the beta but still reject the `context_management` parameter +with an HTTP 400. To degrade gracefully instead of failing the request, the executor strips the +parameter and retries the same URL **once**: + +```ts +if ( + response.status === HTTP_STATUS.BAD_REQUEST && + contextEditing?.enabled && + !contextEditingDisabled && + transformedBody?.context_management !== undefined +) { + const errText = await response.clone().text().catch(() => ""); + if (/context[_-]management|context editing/i.test(errText)) { + contextEditingDisabled = true; + delete transformedBody.context_management; + let retryBody = JSON.stringify(transformedBody); + if (isClaudeCodeCompatible(this.provider) || this.provider === "claude") { + retryBody = await signRequestBody(retryBody); + } + response = await fetch(url, { ...fetchOptions, body: retryBody }); + } +} +``` + +Behavior: + +1. Fires only on a `400` while context editing is enabled and the body actually carries + `context_management`. +2. The 400 body is read via a `clone()` so the original response stays intact for the non-matching + path. +3. The error text must match `/context[_-]management|context editing/i` — an unrelated 400 (e.g. + `max_tokens must be >= 1`) does **not** trigger the fallback; the original error propagates. +4. On a match it sets `contextEditingDisabled = true` (which suppresses re-injection if a fresh + `transformedBody` is later built for a retry/fallback URL), deletes `context_management`, + re-signs the body for Claude / Claude-Code-compatible relays (`signRequestBody`), and retries the + same URL once. + +Genuine Claude carries the beta in `ANTHROPIC_BETA_BASE` and does not hit this fallback path. + +## `applied_edits` telemetry + +After a Claude response, OmniRoute records how much context the provider actually cleared. This is +**not** streamed — it is extracted from the non-streaming response body, best-effort, and never +affects the response (telemetry failures are swallowed). + +- Extraction: `extractContextEditingTelemetry(responseBody)` in `open-sse/config/contextEditing.ts`. + It probes `applied_edits` in three locations (defensive over the response shape): + - `context_management.applied_edits` + - `usage.context_management.applied_edits` + - `usage.applied_edits` +- Per-edit fields read from each entry: `cleared_input_tokens` and `cleared_tool_uses` + (snake_case, Anthropic-native), with `clearedInputTokens` / `clearedToolUses` camelCase fallbacks. +- Returns `null` when no `applied_edits` array is found or nothing was actually cleared. + +The receipt shape is `ContextEditingTelemetry { editCount, clearedInputTokens, clearedToolUses }`. +Recording happens in `open-sse/handlers/chatCore.ts` (gated to `provider === "claude"`) via +`recordContextEditingTelemetry()` (`src/lib/db/compressionAnalytics.ts`), which writes a compression +analytics row tagged: + +- `mode: "context-editing"` +- `engine: "context-editing"` +- `tokens_saved` / `original_tokens` = the cleared input-token count +- `request_id` suffixed with `::context-editing` + +So delegated clearing shows up in compression analytics alongside the local engines, under the +`context-editing` engine label, and is distinguishable from RTK/Caveman/LLMLingua savings. + +## Relationship to the local compression engines + +| Aspect | Local engines (Caveman / RTK / LLMLingua / stacked) | Delegated Context Editing | +| ----------------- | --------------------------------------------------- | ------------------------------------------ | +| Where it runs | In OmniRoute, before the request leaves the proxy | In the provider (Claude), server-side | +| What it edits | Prompt / context / tool-result text | Old tool-use / tool-result blocks | +| Provider scope | All providers | `claude` + `anthropic-compatible-cc-*` only | +| Toggle | Compression mode settings | `contextEditing.enabled` | +| Failure mode | Fail-open (original text) | 400-fallback: strip param, retry once | +| Savings telemetry | `engine: ` | `engine: "context-editing"` | + +The two are complementary: local engines compress the bytes OmniRoute sends; Context Editing lets +Claude prune the running context across turns. They can be enabled together. + +## See Also + +- [COMPRESSION_ENGINES.md](./COMPRESSION_ENGINES.md) — engine registry and the local compression + engines +- [RTK_COMPRESSION.md](./RTK_COMPRESSION.md) — command/tool-output compression +- [../frameworks/MCP-SERVER.md](../frameworks/MCP-SERVER.md) — MCP description compression and + tool-cardinality reduction +- Source: `open-sse/config/contextEditing.ts`, `open-sse/executors/base.ts`, + `open-sse/services/compression/types.ts`, `src/lib/db/compressionAnalytics.ts` diff --git a/docs/compression/RTK_COMPRESSION.md b/docs/compression/RTK_COMPRESSION.md index dbe2637ed6..1d94d27a57 100644 --- a/docs/compression/RTK_COMPRESSION.md +++ b/docs/compression/RTK_COMPRESSION.md @@ -1,7 +1,7 @@ --- title: "RTK Compression" -version: 3.8.2 -lastUpdated: 2026-05-13 +version: 3.8.31 +lastUpdated: 2026-06-20 --- # RTK Compression @@ -96,6 +96,7 @@ Important fields: | `rules.dropPatterns` | Remove noisy lines | | `rules.includePatterns` | Prefer actionable lines | | `rules.collapsePatterns` | Collapse repeated matching lines | +| `rules.deduplicate` | Per-filter opt-in: collapse consecutive duplicate lines | | `rules.truncateLineAt` | Unicode-safe per-line truncation | | `rules.onEmpty` | Fallback message if all lines are filtered out | | `tests[]` | Inline samples used by the verify gate | @@ -103,6 +104,57 @@ Important fields: Built-in filters are expected to include inline `tests[]` samples. Custom filters should include them too, especially when they are shared across projects. +## Line Deduplication (two layers) + +RTK collapses duplicate lines at two independent layers: + +1. **Per-filter `deduplicate` (opt-in, default `false`).** A filter can set `rules.deduplicate: true` + to collapse consecutive duplicate lines *within that filter's matched output*, before truncation. + This runs inside `lineFilter.ts`. For legacy filters, it is auto-enabled when the filter defines + `collapsePatterns`. Schema: `deduplicate: z.boolean().default(false)` in + `open-sse/services/compression/engines/rtk/filterSchema.ts`. +2. **Engine-wide `deduplicateThreshold` (default `3`).** After all filters run, the engine collapses + any run of `>= deduplicateThreshold` identical consecutive lines across the whole result + (`deduplicateRepeatedLines`, applied in `engines/rtk/index.ts`). The value is bounded to 2–100 on + normalization. + +The per-filter pass runs first (inside the filter), the engine-wide pass runs last (over the joined +output), so the two compose without double-counting. + +## Line Grouping (`enableGrouping`) + +When `rtkConfig.enableGrouping` is `true` (default `false`), RTK runs an additional `groupSimilarLines` +pass over the post-dedup result that collapses runs of *near-equivalent* (not byte-identical) +consecutive lines. `rtkConfig.groupingThreshold` (default `3`) is the minimum run length that triggers +grouping. This is the structural counterpart to `deduplicateThreshold`: dedup handles exact repeats, +grouping handles "the same shape with small differences". Both flags are part of the `rtkConfig` JSON +persisted in the `key_value` table (see Configuration above), so the setting survives restarts. + +## Code Comment Stripping (`stripCodeComments` / `preserveDocstrings`) + +When `rtkConfig.applyToCodeBlocks` is enabled, RTK can also strip comments from fenced code blocks: + +- `stripCodeComments` (default `false`) — opt-in. When `true`, RTK removes comments from JavaScript + and TypeScript fenced blocks. The flag was historically read but never applied, so the default stays + at "preserve" to avoid a silent production change. +- `preserveDocstrings` (default `true`) — when stripping comments, JSDoc/`/** … */` block comments are + kept (they carry API documentation worth more than the bytes they cost). Set to `false` to strip + those too. + +Comment removal is implemented in `open-sse/services/compression/engines/rtk/codeStripper.ts`. It uses +the **TypeScript parser** (not a regex) so that string, template, and regex literals are never mistaken +for comments, and it bails out entirely when JSX is detected (so JSX expression-container comments are +never corrupted). Comment stripping currently applies to **JavaScript and TypeScript only** — other +languages in the stripper's `CodeLanguage` set (Python, Rust, Go, Ruby, Java) have empty-line and +whitespace collapse but no comment removal. The stripped-block run is tagged `rtk:code-strip` in +`rulesApplied`. + +> **Note — GCF / tabular encoding is a separate engine.** RTK does **not** contain the "GCF" +> (Graph Compact Format) tabular/columnar JSON encoder. That encoder — which replaced an older +> `omni-tabular` encoder — lives in the **headroom** engine +> (`open-sse/services/compression/engines/headroom/`, with the vendored codec under +> `headroom/gcf/`). It is unrelated to the RTK filter pipeline documented here. + ## Configuration Global settings are available through `/api/settings/compression`. RTK-specific settings are also @@ -131,13 +183,32 @@ available through `/api/context/rtk/config`. "customFiltersEnabled": true, "trustProjectFilters": false, "rawOutputRetention": "never", - "rawOutputMaxBytes": 1048576 + "rawOutputMaxBytes": 1048576, + "enableGrouping": false, + "groupingThreshold": 3, + "stripCodeComments": false, + "preserveDocstrings": true } } ``` `enabledFilters` and `disabledFilters` use filter ids, for example `test-vitest` or `git-diff`. +The full `rtkConfig` shape is defined by `RtkConfig` / `DEFAULT_RTK_CONFIG` in +`open-sse/services/compression/types.ts`. The whole object is persisted as a single JSON value in +the SQLite `key_value` table under `namespace = "compression"`, `key = "rtkConfig"` +(`src/lib/db/compression.ts`), and normalized on read by `normalizeRtkConfig`. So every field below +— including `enableGrouping`, `groupingThreshold`, `stripCodeComments`, and `preserveDocstrings` — +round-trips through the same store and survives a restart. + +| Key | Default | Purpose | +| ---------------------- | ------- | ----------------------------------------------------------------------------- | +| `deduplicateThreshold` | `3` | Engine-wide: min consecutive identical lines to collapse (bounded 2–100) | +| `enableGrouping` | `false` | Opt-in: collapse runs of near-equivalent consecutive lines | +| `groupingThreshold` | `3` | Min consecutive similar-line run that triggers grouping | +| `stripCodeComments` | `false` | Opt-in: remove comments from fenced code blocks (needs `applyToCodeBlocks`) | +| `preserveDocstrings` | `true` | When stripping comments, keep JSDoc/`/** … */` blocks | + ## API | Route | Method | Purpose | diff --git a/docs/frameworks/AGENTBRIDGE.md b/docs/frameworks/AGENTBRIDGE.md index 9ad93524fd..ecf024be0e 100644 --- a/docs/frameworks/AGENTBRIDGE.md +++ b/docs/frameworks/AGENTBRIDGE.md @@ -1,7 +1,7 @@ --- title: "AgentBridge" -version: 3.8.6 -lastUpdated: 2026-05-28 +version: 3.8.31 +lastUpdated: 2026-06-20 --- # AgentBridge @@ -10,7 +10,7 @@ AgentBridge is OmniRoute's MITM (Man-in-the-Middle) proxy that intercepts HTTPS **Dashboard location:** `/dashboard/tools/agent-bridge` **Sidebar group:** Tools (after Cloud Agents) -**See also:** [`TRAFFIC_INSPECTOR.md`](./TRAFFIC_INSPECTOR.md) — monitor all intercepted traffic in real-time. +**See also:** [`TRAFFIC_INSPECTOR.md`](./TRAFFIC_INSPECTOR.md) — monitor all intercepted traffic in real-time; [`docs/security/MITM-TPROXY-DECRYPT.md`](../security/MITM-TPROXY-DECRYPT.md) — the Linux TPROXY transparent-decrypt capture mode driven by the `/api/tools/agent-bridge/tproxy` route. --- @@ -225,6 +225,64 @@ Wildcard `*` maps any unrecognized model to the specified target. Persisted in ` AgentBridge intercepts credentials (OAuth tokens, API keys) that the IDE uses to authenticate with upstream providers. These are **masked before logging** (see §2.7) but are visible to OmniRoute's MITM layer. First activation of each agent shows a dismissible risk notice modal. +### 3.6 Maintenance & Diagnostics + +The dashboard exposes a **Maintenance & Diagnostics** card (`AgentBridgeMaintenanceCard`, in `src/app/(dashboard)/dashboard/tools/agent-bridge/components/`) that surfaces operational MITM routes which previously had no UI. Its subtitle: *"Self-test the capture pipeline, undo leftover system state, and move your setup between machines."* The card client helpers live in `src/lib/inspector/agentBridgeMaintenanceApi.ts`. + +| Button | Route | What it does | +|--------|-------|--------------| +| **Diagnose** | `GET /api/tools/agent-bridge/diagnose` | Runs the capture-pipeline self-test and shows a per-check report (✓/✗ + remediation hint). | +| **Repair** | `POST /api/tools/agent-bridge/repair` | Undoes orphaned MITM system state (DNS spoof entries, root CA, system proxy) left behind by a crash or SIGKILL. Idempotent — reports "Nothing to repair" when state is clean. | +| **Remove CA** | `DELETE /api/tools/agent-bridge/cert` | Untrusts and removes the MITM root CA from the OS trust store (explicit, idempotent). Shown only when the CA is currently trusted; requires an inline "Remove CA?" confirmation. | +| **Export config** | `GET /api/tools/agent-bridge/config` | Downloads the portable config JSON (see §3.7). | +| **Import config** | `POST /api/tools/agent-bridge/config` | Uploads a previously-exported config JSON (see §3.7). | + +**Diagnostics checks** (`summarizeDiagnostics()` in `src/mitm/inspector/diagnostics.ts`). The route runs the effectful probe for each and feeds the booleans into the pure summarizer; a single `healthy` verdict plus a per-failure hint is returned: + +| Check name | What it verifies | Hint on failure | +|------------|------------------|-----------------| +| `server-running` | The MITM server process is active | "The MITM server is not running. Start it from the AgentBridge tab." | +| `server-reachable` | The MITM server accepts connections on its port (TCP probe) | "The MITM server is not accepting connections on its port. Check that the port is free and that you have privileges to bind it." | +| `cert-exists` | The MITM certificate has been generated on disk | "No MITM certificate has been generated yet. Generate one from the AgentBridge tab." | +| `cert-trusted` | The MITM root CA is in the OS trust store | "The MITM root CA is not trusted by the OS store, so TLS interception will fail. Trust the certificate from the AgentBridge tab." | +| `dns-configured` | Target hostnames are spoofed in `/etc/hosts` | "Target hostnames are not spoofed in /etc/hosts, so traffic never reaches the proxy. Enable DNS for the agent(s) you want to capture." | + +**Orphaned-state banner:** when the page detects state left behind by a crash (DNS spoof / CA / system proxy), the card shows an amber banner — *"A previous session left system state behind (DNS spoof, CA, or system proxy). Run Repair to clean it up."* — and highlights the **Repair** button. `Repair` is the application-layer analogue of ProxyBridge's `--cleanup` flag (it delegates to `repairMitm()` in `src/mitm/manager.ts`). + +> The MITM root CA is kept installed across stop/start to avoid repeated sudo +> prompts (the same behavior as mitmproxy/Charles), so removing it is an explicit +> **Remove CA** action rather than something that happens automatically on stop. + +### 3.7 Portable config import/export + +AgentBridge can serialize the **operator-tunable** state into a versioned JSON blob so a setup can be replicated across machines. The serializer is `src/lib/inspector/configPortability.ts` (`exportConfig()` / `importConfig()`), validated by `AgentBridgeConfigSchema`. + +The export includes exactly three pieces (built-in defaults are intentionally **NOT** exported, so importing never duplicates or fights them): + +| Field | Source | Notes | +|-------|--------|-------| +| `bypassPatterns` | user-defined bypass patterns (`agent_bridge_bypass`) | default bank/gov/okta patterns are excluded | +| `customHosts` | Traffic Inspector custom hosts (`inspector_custom_hosts`) | each: `{ host, kind: "llm"\|"app"\|"custom", label? }` | +| `agentMappings` | per-agent model mappings (`agent_bridge_mappings`) | `{ [agentId]: [{ source, target }] }` for every agent that has mappings | + +```jsonc +// GET /api/tools/agent-bridge/config +{ + "version": 1, + "bypassPatterns": ["*.internal.example.com"], + "customHosts": [{ "host": "api.example.com", "kind": "llm", "label": null }], + "agentMappings": { "copilot": [{ "source": "gpt-4o", "target": "claude-sonnet-4.7" }] } +} +``` + +**Import behavior** (`POST /api/tools/agent-bridge/config`): bypass patterns and per-agent mappings **replace wholesale**; custom hosts are added **idempotently** (`INSERT OR IGNORE`). The response reports how many of each were applied: + +```jsonc +{ "ok": true, "bypassPatterns": 1, "customHosts": 1, "agents": 1 } +``` + +What is **NOT** in the config: server running state, cert paths, per-agent DNS state, upstream CA path, and TPROXY settings — those are host/runtime state, not portable preferences. + --- ## §4 Per-agent reference @@ -364,18 +422,31 @@ Base path: `/api/tools/agent-bridge/` | Method | Path | Description | |--------|------|-------------| -| GET | `/api/tools/agent-bridge/agents` | List all 9 agents with current state | -| GET | `/api/tools/agent-bridge/state` | Global server state (running, port, cert info) | -| POST | `/api/tools/agent-bridge/server` | Start/stop/restart server (`action: "start"\|"stop"\|"restart"\|"trust-cert"\|"regenerate-cert"`) | -| GET | `/api/tools/agent-bridge/agents/{id}` | State of one agent (dns_enabled, cert_trusted, etc.) | +| GET | `/api/tools/agent-bridge/state` | Global server state + per-agent detection/status | +| GET | `/api/tools/agent-bridge/agents` | List registered agents (id, name, hosts, viability, state) | +| GET | `/api/tools/agent-bridge/agents/{id}` | State of one agent (target config + detection + stored state) | +| PATCH | `/api/tools/agent-bridge/agents/{id}` | Update `setup_completed` for agent | +| GET | `/api/tools/agent-bridge/agents/{id}/detect` | Run detection probe for agent (`installed`, `version?`, `path?`) | | POST | `/api/tools/agent-bridge/agents/{id}/dns` | Enable/disable DNS for agent (`{enabled: boolean}`) | | GET | `/api/tools/agent-bridge/agents/{id}/mappings` | Model mappings for agent | -| PUT | `/api/tools/agent-bridge/agents/{id}/mappings` | Update model mappings | -| GET | `/api/tools/agent-bridge/bypass` | List bypass patterns | -| PUT | `/api/tools/agent-bridge/bypass` | Update bypass patterns | -| POST | `/api/tools/agent-bridge/cert` | Download or regenerate CA cert | +| PUT | `/api/tools/agent-bridge/agents/{id}/mappings` | Replace model mappings | +| POST | `/api/tools/agent-bridge/server` | Start/stop/restart server (`action: "start"\|"stop"\|"restart"\|"trust-cert"\|"regenerate-cert"`) | +| GET | `/api/tools/agent-bridge/cert` | Cert status (`exists`, `trusted`, `path`) | +| POST | `/api/tools/agent-bridge/cert` | Trust (install) the MITM root CA | +| DELETE | `/api/tools/agent-bridge/cert` | Untrust (remove) the MITM root CA — idempotent (see §3.6) | +| POST | `/api/tools/agent-bridge/cert/regenerate` | Regenerate the self-signed MITM cert | +| GET | `/api/tools/agent-bridge/cert/download` | Stream the PEM cert for download | +| GET | `/api/tools/agent-bridge/bypass` | List bypass patterns (`default` + `user`) | +| POST | `/api/tools/agent-bridge/bypass` | Replace user-defined bypass patterns wholesale | +| DELETE | `/api/tools/agent-bridge/bypass?pattern=...` | Remove a single user-defined bypass pattern | +| GET | `/api/tools/agent-bridge/diagnose` | Capture-pipeline self-test (see §3.6) | +| POST | `/api/tools/agent-bridge/repair` | Undo orphaned MITM system state (see §3.6) | +| GET | `/api/tools/agent-bridge/config` | Export portable config JSON (see §3.7) | +| POST | `/api/tools/agent-bridge/config` | Import portable config JSON (see §3.7) | | GET | `/api/tools/agent-bridge/upstream-ca` | Get configured upstream CA path | -| POST | `/api/tools/agent-bridge/upstream-ca` | Set upstream CA cert path | +| POST | `/api/tools/agent-bridge/upstream-ca` | Validate + persist upstream CA path | +| POST | `/api/tools/agent-bridge/upstream-ca/test` | Validate-only (dry-run) an upstream CA path — does not persist | +| GET / POST / DELETE | `/api/tools/agent-bridge/tproxy` | TPROXY transparent-decrypt capture mode — see [`docs/security/MITM-TPROXY-DECRYPT.md`](../security/MITM-TPROXY-DECRYPT.md) | Full OpenAPI schemas: `docs/reference/openapi.yaml` → tag `AgentBridge`. diff --git a/docs/frameworks/CLOUD_AGENT.md b/docs/frameworks/CLOUD_AGENT.md index 82570f57e1..ab06bbbd53 100644 --- a/docs/frameworks/CLOUD_AGENT.md +++ b/docs/frameworks/CLOUD_AGENT.md @@ -1,13 +1,13 @@ --- title: "Cloud Agents" -version: 3.8.2 -lastUpdated: 2026-05-13 +version: 3.8.31 +lastUpdated: 2026-06-20 --- # Cloud Agents > **Source of truth:** `src/lib/cloudAgent/` and `src/app/api/v1/agents/tasks/` -> **Last updated:** 2026-05-13 — v3.8.0 +> **Last updated:** 2026-06-20 — v3.8.31 (frontmatter refresh; 4 agents incl. cursor-cloud) OmniRoute orchestrates third-party cloud-hosted coding agents (Codex Cloud, Cursor, Devin, Jules) as long-running tasks. Each agent is wrapped behind a uniform interface so diff --git a/docs/frameworks/MCP-SERVER.md b/docs/frameworks/MCP-SERVER.md index 4c04ba9f11..9ae858648a 100644 --- a/docs/frameworks/MCP-SERVER.md +++ b/docs/frameworks/MCP-SERVER.md @@ -1,7 +1,7 @@ --- title: "OmniRoute MCP Server Documentation" -version: 3.8.8 -lastUpdated: 2026-05-30 +version: 3.8.31 +lastUpdated: 2026-06-20 --- # OmniRoute MCP Server Documentation @@ -310,6 +310,8 @@ Wildcard scopes are supported: `read:*` grants all read-scopes, `*` grants full | `OMNIROUTE_MCP_SCOPES` | (empty) | Comma-separated allowlist of scopes considered "available" by default (used when caller does not provide its own scopes) | | `OMNIROUTE_MCP_COMPRESS_DESCRIPTIONS` | (unset = on) | When set to `0/false/off/no`, disables MCP description compression at registration time | | `OMNIROUTE_MCP_DESCRIPTION_COMPRESSION` | (unset = on) | Alternate alias for the same toggle as above | +| `MCP_TOOL_DENY` | (unset = no filter) | Comma-separated tool names to drop from `tools/list` (tool-cardinality reduction — see below) | +| `MCP_TOOL_ALLOW` | (unset = no filter) | Comma-separated tool names to keep exclusively (allow-list mode — see below) | | `DATA_DIR` | `~/.omniroute` | Heartbeat file is written to `${DATA_DIR}/runtime/mcp-heartbeat.json` | --- @@ -325,6 +327,33 @@ MCP tool, prompt, and resource registries can compress descriptions at registrat --- +## Tool Cardinality Reduction (F4.3) + +Description compression shrinks each tool's metadata; **tool-cardinality reduction** goes one step further by reducing *how many* tools are announced at all. Advertising fewer tools in the `tools/list` manifest cuts the per-request token cost the client's model pays for the tool catalog ("layer 5" compression). The implementation is a pure, stateless filter in `open-sse/mcp-server/toolCardinality.ts` (`reduceToolManifest`), wired into the registration loop in `createMcpServer()` (`open-sse/mcp-server/server.ts`). + +**Opt-in, off by default.** The filter only runs when at least one of two environment variables is set; with neither set, all 87 tools are announced unchanged. + +| Variable | Mode | +| :--------------- | :-------------------------------------------------------------------------------------- | +| `MCP_TOOL_DENY` | Blacklist — comma-separated tool names that are always dropped from `tools/list` | +| `MCP_TOOL_ALLOW` | Allow-list — comma-separated tool names; only these survive, everything else is dropped | + +`deny` takes priority over `allow`. Names are comma-separated, trimmed, and empty entries are ignored. Examples: + +```bash +# Drop two tools from the catalog +MCP_TOOL_DENY="omniroute_get_health,omniroute_list_combos" omniroute --mcp + +# Announce only the routing + quota tools (allow-list mode) +MCP_TOOL_ALLOW="omniroute_route_request,omniroute_check_quota" omniroute --mcp +``` + +**How filtered tools are removed:** registration always succeeds; a tool the profile rejects is then `.disable()`d on the MCP SDK handle, so it never appears in `tools/list` but the wiring stays intact (clean enable/disable, no re-registration). The profile parser is `readMcpToolProfileFromEnv(process.env)`, which returns `null` (no filtering) when both vars are empty. + +The richer `ToolProfile` shape behind `reduceToolManifest` also supports scope-intersection filtering (`allowScopes`, with `read:*`-style wildcard matching) and a deterministic `maxTools` cap, but those two knobs need the full manifest at registration time and are **not** exposed through the environment variables today (a `tools/list`-level hook is a tracked follow-up). `estimateManifestTokens()` is available to compare the manifest token cost before and after reduction. + +--- + ## Runtime Heartbeat The stdio transport persists liveness to `${DATA_DIR}/runtime/mcp-heartbeat.json` every 5 seconds. The dashboard (`/api/mcp/status`) reads this file plus PID liveness to derive `online`. HTTP transports report state from in-process `getMcpHttpStatus()` instead (no file write). diff --git a/docs/frameworks/MEMORY.md b/docs/frameworks/MEMORY.md index 1746f670f9..b38c25887e 100644 --- a/docs/frameworks/MEMORY.md +++ b/docs/frameworks/MEMORY.md @@ -1,13 +1,13 @@ --- title: "Memory System" -version: 3.8.6 -lastUpdated: 2026-05-28 +version: 3.8.31 +lastUpdated: 2026-06-20 --- # Memory System > **Source of truth:** `src/lib/memory/` and `src/app/api/memory/` -> **Last updated:** 2026-05-28 — v3.8.6 (plan 21 — Memory Engine Redesign) +> **Last updated:** 2026-06-20 — v3.8.31 (off-by-default + int8 quantization catch-up) OmniRoute provides persistent conversational memory keyed by API key (and optionally session id). Memories are extracted automatically from LLM responses @@ -15,6 +15,18 @@ via lightweight regex pattern matching and injected back into subsequent requests as a leading system message (or first user message for providers that reject the system role). +> **Memory is OFF by default (v3.8.30+).** `DEFAULT_MEMORY_SETTINGS.enabled` is +> now `false` (`src/lib/memory/settings.ts`). Enabling memory injects up to +> `maxTokens` (~2k) of retrieved context into **every** chat request, which is +> billed — a surprising cost for new installs and for clients that manage their +> own context. Opt in explicitly under **Settings → Memory** (the +> `MemorySkillsTab` shows a token-cost warning callout when memory is enabled). +> A client can opt a single request out with the `x-omniroute-no-memory` +> request header (`true`/`1`/`yes`) — see the request-header table in +> [API_REFERENCE.md](../reference/API_REFERENCE.md). A no-memory request sets +> `memoryOwnerId = null`, which disables **both** memory and skill injection for +> that request (`open-sse/handlers/chatCore/headers.ts::isNoMemoryRequested`). + Memory is **scoped per API key**, not per user — every request authenticated with the same API key shares the same memory pool, with optional further scoping by `sessionId`. @@ -235,6 +247,30 @@ routes under `src/app/api/settings/qdrant/` are all wired as of v3.8.6: | `/api/settings/qdrant/cleanup` | `POST` | Remove expired / old points | | `/api/settings/qdrant/embedding-models` | `GET` | List available embedding models | +### Vector quantization (int8 — opt-in, both backends) + +Both vector backends support **opt-in int8 quantization** to cut the memory +footprint of stored vectors (~4× smaller than Float32) at a small recall cost. +Default is **off** on both — vectors stay full-precision unless explicitly +enabled. + +| Backend | Setting | Type | Default | Where read | +| ------------ | -------------------------------- | ----------------------------- | -------- | --------------------------------------------------- | +| Qdrant | `qdrantQuantization` (DB key) | `"none" \| "int8" \| "binary"` | `"none"` | `src/lib/memory/qdrant.ts::normalizeQdrantConfig()` | +| sqlite-vec | `MEMORY_VEC_QUANTIZATION` (env) | `"none" \| "int8"` | `"none"` | `src/lib/memory/vectorStore.ts::requestedVecQuantization()` | + +- **Qdrant** is configured per-instance via the `qdrantQuantization` setting + key (exposed as the `quantization` field on `PUT /api/settings/qdrant`). When + `"int8"`, `buildQuantizationConfig()` requests scalar quantization + (`always_ram`, quantile `0.99`) and searches enable `rescore: true` so the + full-precision vectors refine the int8 candidate set. +- **sqlite-vec** quantization is **environment-only** (not a DB setting): set + `MEMORY_VEC_QUANTIZATION=int8` to store the local vectors as an `int8[dim]` + column via `vec_quantize_int8(?, 'unit')`. The chosen mode is folded into the + `embedding_signature` (an `:int8` suffix), so switching modes triggers a full + reindex of the `vec_memories` table — the same lazy-backfill path used when + the embedding model changes. + ## Memory Types `MemoryType` (`src/lib/memory/types.ts`): @@ -332,7 +368,7 @@ route after writes. | DB key | Type | Default | UI control | | --------------------- | ------- | -------------------------------------------------- | ----------------------------------------------- | -| `memoryEnabled` | boolean | `true` | Memory on/off | +| `memoryEnabled` | boolean | `false` (off by default since v3.8.30) | Memory on/off | | `memoryMaxTokens` | integer | `2000` (range `0–16000`) | Token budget for injection | | `memoryRetentionDays` | integer | `30` (range `1–365`) | Retention window | | `memoryStrategy` | enum | `"hybrid"` (one of `recent`, `semantic`, `hybrid`) | Retrieval strategy | @@ -373,6 +409,7 @@ Six optional env vars tune the engine's runtime behaviour (documented in `.env.e | `MEMORY_STATIC_CACHE_DIR` | `/embeddings` | Where to store downloaded models | | `MEMORY_VEC_TOP_K` | `20` | Default top-K for vector search | | `MEMORY_RRF_K` | `60` | RRF k constant for hybrid search | +| `MEMORY_VEC_QUANTIZATION` | `none` | Set to `int8` to store local sqlite-vec vectors quantized (~4× smaller; opt-in). Mode change forces a reindex. | ## Summarisation (`summarization.ts`) diff --git a/docs/frameworks/TRAFFIC_INSPECTOR.md b/docs/frameworks/TRAFFIC_INSPECTOR.md index c48b213930..eff615c830 100644 --- a/docs/frameworks/TRAFFIC_INSPECTOR.md +++ b/docs/frameworks/TRAFFIC_INSPECTOR.md @@ -1,12 +1,12 @@ --- title: "Traffic Inspector" -version: 3.8.6 -lastUpdated: 2026-05-28 +version: 3.8.31 +lastUpdated: 2026-06-20 --- # Traffic Inspector -Traffic Inspector is OmniRoute's built-in HTTPS traffic debugger — a Charles Proxy / mitmweb / HTTP Toolkit-like tool that is **LLM-aware** and **agent-aware**. It lives at `/dashboard/tools/traffic-inspector` and receives live traffic from up to 4 simultaneous capture sources. +Traffic Inspector is OmniRoute's built-in HTTPS traffic debugger — a Charles Proxy / mitmweb / HTTP Toolkit-like tool that is **LLM-aware** and **agent-aware**. It lives at `/dashboard/tools/traffic-inspector` and receives live traffic from up to 5 simultaneous capture sources. **Dashboard location:** `/dashboard/tools/traffic-inspector` **Sidebar group:** Tools (after AgentBridge) @@ -42,7 +42,7 @@ The `TrafficBuffer` (`src/mitm/inspector/buffer.ts`) is a shared in-memory ring ## §2 Capture modes -Traffic Inspector supports **4 simultaneous capture sources**. Each is independently toggleable. +Traffic Inspector supports **5 simultaneous capture sources**. Each is independently toggleable. The `source` field on every `InterceptedRequest` (`src/mitm/inspector/types.ts`) is one of `"agent-bridge"`, `"custom-host"`, `"http-proxy"`, `"system-proxy"`, or `"tproxy"`. ### Mode 1 — AgentBridge (default, always on) @@ -99,6 +99,17 @@ export HTTPS_PROXY=http://127.0.0.1:8080 - Dashboard shows "Reverting system proxy" prompt if user navigates away while active - UI shows `⚠ Advanced` badge + explicit confirmation checkbox +### Mode 5 — TPROXY transparent decrypt (Linux, root, opt-in) + +**Source:** Kernel TPROXY + policy routing (`src/mitm/tproxy/`) +**Mechanism:** Marks new local outbound TCP connections to a target port (default `443`) in `mangle OUTPUT`, an `ip rule` reroutes the marked packets to local delivery, and `mangle PREROUTING`'s `TPROXY` target hands them to a transparent (**IP_TRANSPARENT**) listener (default port `8443`). The listener terminates TLS with a leaf certificate issued **per SNI hostname on demand** by a dynamic CA, captures the decrypted exchange, and forwards the request re-encrypted to the original destination. +**Reach:** **Arbitrary** destination hosts on the target port — no `/etc/hosts` spoof, no `HTTP_PROXY` env, no system-wide proxy mutation. The intercepted process needs no config change, but must trust the dynamic CA. +**Note:** `source` = `"tproxy"` + +**Requirements:** Linux only (**IP_TRANSPARENT** is Linux-only), the **CAP_NET_ADMIN** capability (root), and a native N-API addon that must be built with a C toolchain (`npm run build:native:tproxy`). When unavailable, the dashboard toggle is disabled with the tooltip "TPROXY decrypt requires Linux + root + the native addon". The firewall rules apply/revert transactionally (a crash never leaves a `mangle` rule behind) and flush on reboot. An SO_MARK-based anti-loop keeps the proxy's own re-encrypted forward from being re-intercepted. + +This is a substantial subsystem with its own dedicated operator guide — see **[`docs/security/MITM-TPROXY-DECRYPT.md`](../security/MITM-TPROXY-DECRYPT.md)** for the full firewall recipe, the per-SNI dynamic CA + trust-store installer, the local-only route, anti-loop details, and the configuration schema. The toggle is driven by `GET / POST / DELETE /api/tools/agent-bridge/tproxy` (note: the route lives under the AgentBridge prefix, not the Traffic Inspector prefix). + ### Capture mode comparison | Mode | Setup | Sudo? | Reach | Notes | @@ -107,6 +118,7 @@ export HTTPS_PROXY=http://127.0.0.1:8080 | 2. Custom Hosts | Per-host input | Yes (hosts file) | Any app using that host | Persisted in DB | | 3. HTTP_PROXY | `export HTTPS_PROXY=...` | No | Apps respecting env | Port 8080, no TLS decrypt by default | | 4. System-wide | Toggle + confirm | Yes | All apps on machine | Auto-disable in 30 min | +| 5. TPROXY decrypt | Toggle (Linux + native addon) | Yes (root + CA install) | Any host on the target port | Decrypts arbitrary hosts; off by default — see [MITM-TPROXY-DECRYPT.md](../security/MITM-TPROXY-DECRYPT.md) | --- @@ -166,7 +178,8 @@ export HTTPS_PROXY=http://127.0.0.1:8080 | Host filter | Substring match on `host` field | | Agent filter | Dropdown: All / per-agent | | Status filter | All / 2xx / 3xx / 4xx / 5xx / error | -| Source filter | All / agent-bridge / custom-host / http-proxy / system-proxy | +| Source filter | All / agent-bridge / custom-host / http-proxy / system-proxy / tproxy | +| **Live** filter | Show only in-flight (open) requests — `liveOnly` toggle (see §4.6) | ### 3.5 Resizable panels @@ -245,6 +258,47 @@ interface LlmMetadata { } ``` +### 4.6 Live in-flight request filter + +The request `status` field is `number | "in-flight" | "error"` — an entry is +pushed as `"in-flight"` the moment the request starts and **updated in place** +when the response (or error) arrives. The toolbar's **"Live"** toggle +(`liveOnly`, i18n key `trafficInspector.liveOnly`) restricts the list to entries +whose `status === "in-flight"`, letting you watch open connections in real time. + +The filter is a pure, client-side predicate in +`src/lib/inspector/matchesTrafficFilter.ts`: + +```ts +if (f.liveOnly && req.status !== "in-flight") return false; +``` + +The toggle state lives in `useTrafficFilters` (the inspector dashboard hooks) and +combines with the other filters (profile, host, agent, source, status, context). + +### 4.7 Process attribution (Linux) + +On Linux, each intercepted request can be attributed to the **originating local +process**. Two optional fields are added to `InterceptedRequest`: + +```ts +pid?: number; // originating process id (Linux only) +processName?: string; // originating process name (Linux only) +``` + +`src/mitm/inspector/processAttribution.ts` maps the connection's *client* +ephemeral port to a PID + name by: + +1. Reading `/proc/net/tcp` and `/proc/net/tcp6` to find the socket inode for the + port (`parseProcNetTcpForInode`, a pure fixture-testable parser). +2. Scanning `/proc//fd/` for a symlink to `socket:[]`. +3. Reading the process name from `/proc//comm`. + +A 1-second TTL cache bounds the procfs scan cost under load. Attribution is +**best-effort** — any failure resolves to `null` and never blocks capture. On +macOS/Windows the function returns `null` (stub; `lsof`/`GetExtendedTcpTable` +support is a follow-up). + --- ## §5 Sessions @@ -396,10 +450,15 @@ Base path: `/api/tools/traffic-inspector/` | Method | Path | Description | |--------|------|-------------| -| GET | `/capture-modes` | State of all 4 capture modes | +| GET | `/capture-modes` | State of the AgentBridge / custom-hosts / HTTP_PROXY / system-proxy modes + the `tls-intercept` toggle | | POST | `/capture-modes/http-proxy` | Start/stop HTTP_PROXY listener (`{action: "start"\|"stop"}`) | | POST | `/capture-modes/system-proxy` | Apply/revert system-wide proxy (`{action: "apply"\|"revert"}`) | -| POST | `/capture-modes/tls-intercept` | Toggle HTTPS body decryption in proxy mode | +| POST | `/capture-modes/tls-intercept` | Toggle HTTPS body decryption in proxy mode (`{enabled: boolean}`) | + +> **TPROXY decrypt** (capture mode 5) is driven by a **separate** route under the +> AgentBridge prefix — `GET / POST / DELETE /api/tools/agent-bridge/tproxy` — not +> under `/api/tools/traffic-inspector/`. See +> [`docs/security/MITM-TPROXY-DECRYPT.md`](../security/MITM-TPROXY-DECRYPT.md). ### Sessions diff --git a/docs/guides/CLI-INTEGRATIONS.md b/docs/guides/CLI-INTEGRATIONS.md new file mode 100644 index 0000000000..d16ca19d7c --- /dev/null +++ b/docs/guides/CLI-INTEGRATIONS.md @@ -0,0 +1,208 @@ +--- +title: "CLI Integrations — point any coding CLI at OmniRoute" +version: 3.8.31 +lastUpdated: 2026-06-20 +--- + +# CLI Integrations + +OmniRoute ships a family of `setup-*` commands that configure a coding +CLI (Codex, Claude Code, OpenCode, Cline, …) to use OmniRoute as its backend — so +the tool talks to **one** endpoint and OmniRoute routes to the right provider with +auto-fallback. Each command reads the **live** model catalog from a running +OmniRoute (local or remote) and writes the tool's own config file on **your** +machine. The API key is referenced by env var wherever the tool supports it, so the +secret is never written to disk (the exceptions are noted below). + +There are also two launchers — `omniroute launch` (Claude Code) and +`omniroute launch-codex` (Codex) — that spawn the CLI with the right env injected, +without writing any config at all. + +For the one-time, hand-written base setup of the two richest integrations, see the +per-tool deep dives: + +- [Claude Code configuration](./CLAUDE-CODE-CONFIGURATION.md) +- [Codex CLI configuration](./CODEX-CLI-CONFIGURATION.md) +- [Remote Mode](./REMOTE-MODE.md) — drive a remote OmniRoute (VPS / Tailnet) from your laptop + +--- + +## Master table + +Every command honours the **active context** (set with `omniroute connect`, see +[Remote Mode](./REMOTE-MODE.md)) or explicit `--remote --api-key ` flags. +"Local vs remote" below means: with no flags it targets `http://localhost:20128`; +with `--remote` (or an active remote context) it fetches the catalog from that +server and writes the config locally. + +| Command | Tool | What it writes | Key flags | Local vs remote | +|---------|------|----------------|-----------|-----------------| +| `omniroute setup-codex` | OpenAI Codex CLI | `~/.codex/.config.toml` — one profile per matched model (`codex --profile `) | `--remote` `--api-key` `--only` `--dry-run` `--port` `--codex-home` | Both | +| `omniroute setup-claude` | Claude Code | `~/.claude/profiles//settings.json` — one profile per matched model (`CLAUDE_CONFIG_DIR`) | `--remote` `--api-key` `--only` `--dry-run` `--port` `--claude-home` | Both | +| `omniroute setup-opencode` | OpenCode (openai-compatible) | `~/.config/opencode/opencode.json` — `omniroute` provider with every catalog model (`opencode -m omniroute/`) | `--remote` `--api-key` `--only` `--model` `--dry-run` `--port` | Both | +| `omniroute setup-cline` | Cline | `~/.cline/data/{globalState,secrets}.json` (CLI mode) + prints VS Code extension settings | `--remote` `--api-key` `--model` `--yes` `--dry-run` `--port` `--cline-dir` | Both | +| `omniroute setup-kilo` | Kilo Code | `~/.local/share/kilo/auth.json` (CLI) + merges `kilocode.*` into VS Code `settings.json` if present | `--remote` `--api-key` `--model` `--yes` `--dry-run` `--port` `--auth-path` `--vscode-settings` | Both | +| `omniroute setup-continue` | Continue / `cn` CLI | `~/.continue/config.yaml` — `provider: openai` models, key via `${{ secrets.OMNIROUTE_API_KEY }}` | `--remote` `--api-key` `--only` `--dry-run` `--port` `--config-path` | Both | +| `omniroute setup-cursor` | Cursor | Nothing — prints the in-app steps (Cursor config is opaque SQLite) | `--remote` `--api-key` `--only` `--port` | Both | +| `omniroute setup-roo` | Roo Code | `~/.omniroute/roo-settings.json` (import doc) + sets `roo-cline.autoImportSettingsPath` if a VS Code `settings.json` exists | `--remote` `--api-key` `--model` `--yes` `--dry-run` `--port` `--import-path` `--vscode-settings` | Both | +| `omniroute setup-crush` | Crush | `~/.config/crush/crush.json` — `openai-compat` provider, key via `$OMNIROUTE_API_KEY` | `--remote` `--api-key` `--only` `--dry-run` `--port` `--config-path` | Both | +| `omniroute setup-goose` | Goose | `~/.config/goose/config.yaml` (`GOOSE_PROVIDER`/`OPENAI_HOST`/`GOOSE_MODEL`) + prints env recipe | `--remote` `--api-key` `--model` `--yes` `--dry-run` `--port` `--config-path` | Both | +| `omniroute setup-qwen` | Qwen Code | `~/.qwen/settings.json` — openai `modelProvider`, key via `envKey` (`OMNIROUTE_API_KEY`) | `--remote` `--api-key` `--model` `--yes` `--dry-run` `--port` `--config-path` | Both | +| `omniroute setup-aider` | Aider | `~/.aider.conf.yml` (`openai-api-base` + `model: openai/`) + prints env recipe | `--remote` `--api-key` `--model` `--yes` `--dry-run` `--port` `--config-path` | Both | +| `omniroute setup-gemini` | Gemini CLI (native) | `~/.gemini/settings.json` (`model`) + prints env recipe; base URL is env-only | `--remote` `--api-key` `--model` `--yes` `--dry-run` `--port` `--config-path` | Both | +| `omniroute launch` | Claude Code | Nothing — spawns `claude` with `ANTHROPIC_BASE_URL`/`ANTHROPIC_AUTH_TOKEN` injected | `--remote` `--api-key` `--token` `--profile` `--port` | Both | +| `omniroute launch-codex` | OpenAI Codex CLI | Nothing — spawns `codex` with the `omniroute` provider injected via `-c` flags | `--remote` `--api-key` `--profile` (`-p`) `--port` | Both | + +Notes on flags (verified in the command source): + +- `--remote ` — fetch the catalog from a remote OmniRoute (overrides `--port` + and the active context). `--api-key ` supplies the credential for that + server (defaults to the `OMNIROUTE_API_KEY` env var, or the active context's token). +- `--only ` — comma-separated substrings; keep only model IDs that match + (e.g. `--only glm,kimi`). Available on `setup-codex`, `setup-claude`, + `setup-opencode`, `setup-continue`, `setup-cursor`, `setup-crush`. +- `--dry-run` — print exactly what would be written without touching the + filesystem. Available on every `setup-*` command **except** `setup-cursor` + (which never writes a file). +- `--model ` — required (or picked interactively) for the tools that have no + model auto-discovery: Cline, Kilo, Roo, Goose, Qwen, Aider, Gemini. Those tools + also accept `--yes` for non-interactive runs (which then requires `--model`). + `setup-opencode` takes `--model` to set the default top-level model. +- `--port ` — local OmniRoute port (default `20128`, ignored when `--remote` + is set). Present on all `setup-*` and both launchers. +- The two launchers (`launch`, `launch-codex`) accept `--profile ` to select + a profile written by `setup-claude` / `setup-codex`, plus pass-through args for + the underlying `claude` / `codex` binary. + +> `setup-opencode` is the **lightweight openai-compatible** OpenCode integration. +> There is also a richer plugin integration — `omniroute setup opencode` — which +> installs `@omniroute/opencode-plugin`. They are different commands; the table +> above documents `setup-opencode`. + +--- + +## Local usage + +With OmniRoute running on `localhost:20128`, just run the setup command for your +tool. The catalog is fetched from the local server. + +```bash +# Codex: write a profile per matched model into ~/.codex/ +omniroute setup-codex +codex --profile glm52 # use a generated profile + +# Claude Code: write per-model profiles, then launch one +omniroute setup-claude +omniroute launch --profile glm52 + +# OpenCode: write the openai-compatible provider with all catalog models +omniroute setup-opencode +export OMNIROUTE_API_KEY=sk-... # referenced via {env:OMNIROUTE_API_KEY}, never on disk +opencode -m omniroute/glm/glm-5.2 "..." + +# Tools without auto-discovery need an explicit model: +omniroute setup-aider --model glm/glm-5.2 +omniroute setup-qwen --model kmc/kimi-k2.7 + +# Preview without writing anything: +omniroute setup-continue --dry-run +``` + +Launch without writing any config at all (env-injection only): + +```bash +omniroute launch # Claude Code → local OmniRoute +omniroute launch-codex # Codex CLI → local OmniRoute +omniroute launch-codex --profile glm52 +``` + +--- + +## Remote usage + +Point any setup command at a remote OmniRoute with `--remote` + `--api-key`. The +catalog is fetched from the remote; the config is written on your local machine. + +```bash +# OpenCode against a remote VPS, keep only glm/kimi models +omniroute setup-opencode --remote http://192.168.0.15:20128 --api-key oma_live_xxx \ + --only glm,kimi +opencode -m omniroute/glm/glm-5.2 "..." # export OMNIROUTE_API_KEY first + +# Codex profiles from a remote catalog +omniroute setup-codex --remote http://192.168.0.15:20128 --api-key oma_live_xxx + +# Launch a CLI straight against the remote +omniroute launch --remote http://192.168.0.15:20128 --api-key oma_live_xxx +omniroute launch-codex --remote http://192.168.0.15:20128 --api-key oma_live_xxx +``` + +Instead of passing `--remote`/`--api-key` every time, log in once and let the +**active context** supply them automatically: + +```bash +omniroute connect 192.168.0.15 # mints a scoped token, stores the context +omniroute setup-codex # ← now uses the remote catalog +omniroute setup-opencode # ← same +omniroute launch # ← Claude Code against the remote +``` + +See [Remote Mode](./REMOTE-MODE.md) for contexts, scopes, and token management. + +--- + +## Base URL conventions (which tools want `/v1`) + +OmniRoute exposes the OpenAI surface at `/v1`, the Anthropic surface at the root, +and a native Gemini surface at `/v1beta`. Each integration is wired to the form its +tool expects (verified in the command source): + +| Integration | Base URL written | `/v1`? | +|-------------|------------------|--------| +| `setup-cline` (`openAiBaseUrl`) | root | No — Cline appends `/v1/chat/completions` | +| `setup-goose` (`OPENAI_HOST`) | root | No — Goose appends the path | +| `setup-aider` (`OPENAI_API_BASE`) | root | No — LiteLLM appends `/v1/chat/completions` | +| `setup-kilo`, `setup-roo`, `setup-continue`, `setup-crush`, `setup-qwen`, `setup-cursor` | with `/v1` | Yes | +| `setup-claude` (`ANTHROPIC_BASE_URL`), `launch` | root | No — Claude Code appends `/v1/messages` | +| `setup-codex`, `launch-codex` (`model_providers.omniroute.base_url`) | with `/v1` | Yes | +| `setup-gemini` (`GOOGLE_GEMINI_BASE_URL`) | root | No — the genai SDK appends `/v1beta` | + +> Gemini CLI caveat: a cached Google login can make the CLI ignore +> `GOOGLE_GEMINI_BASE_URL`. Run it logged-out / API-key-only so the base URL takes +> effect (`setup-gemini` prints this warning too). + +--- + +## Keeping native deps on update: `--include=optional` + +When you update with `omniroute update` (after confirming, or with `--apply`), +OmniRoute runs the install with `--include=optional` baked in: + +```bash +npm install -g omniroute@latest --include=optional +``` + +This is **not** a flag you pass to `omniroute update` — it is always applied by the +updater. It guarantees the `optionalDependencies` (`better-sqlite3`, `keytar`, +`tls-client`, the LLMLingua SLM stack) survive the update even if your npm config +has `omit=optional` set, which would otherwise silently drop the native SQLite +driver and OS-keyring binding. To preview the exact command without applying: + +```bash +omniroute update --dry-run +# [DRY RUN] Would run: npm install -g omniroute@latest --include=optional +``` + +Other `omniroute update` flags (verified in source): `--check` (exit 1 if +outdated), `--apply` (install without prompting), `--changelog`, `--no-backup`, +`--yes`. + +--- + +## See also + +- [Claude Code configuration](./CLAUDE-CODE-CONFIGURATION.md) — the deeper Claude Code guide +- [Codex CLI configuration](./CODEX-CLI-CONFIGURATION.md) — the one-time `[model_providers.omniroute]` base setup +- [Remote Mode](./REMOTE-MODE.md) — contexts, scoped access tokens, driving a remote server +- [CLI Tools reference](../reference/CLI-TOOLS.md) — the full catalog of supported tools + dashboard pages +- [Setup Guide](./SETUP_GUIDE.md) — install methods and first-run onboarding diff --git a/docs/guides/SETUP_GUIDE.md b/docs/guides/SETUP_GUIDE.md index bf811ae193..33ec1ce7d4 100644 --- a/docs/guides/SETUP_GUIDE.md +++ b/docs/guides/SETUP_GUIDE.md @@ -1,7 +1,7 @@ --- title: "📖 Setup Guide — OmniRoute" -version: 3.8.2 -lastUpdated: 2026-05-13 +version: 3.8.31 +lastUpdated: 2026-06-20 --- # 📖 Setup Guide — OmniRoute @@ -164,6 +164,35 @@ Ollama Tags URL: http://localhost:20128/api/v1/vscode/YOUR_KEY/api/tags Works with Claude Code, Codex CLI, Gemini CLI, Cursor, Cline, OpenClaw, OpenCode, and OpenAI-compatible SDKs. +#### Auto-configure with `setup-*` + +Instead of pasting the base URL and key by hand, let OmniRoute write each tool's +own config from the live model catalog. One command per tool: + +```bash +omniroute setup-codex # ~/.codex/.config.toml profiles +omniroute setup-claude # ~/.claude/profiles//settings.json +omniroute setup-opencode # ~/.config/opencode/opencode.json (openai-compatible) +omniroute setup-cline # Cline CLI + VS Code extension settings +omniroute setup-kilo # Kilo Code +omniroute setup-continue # ~/.continue/config.yaml (Continue / cn) +omniroute setup-cursor # prints Cursor's in-app steps +omniroute setup-roo # Roo Code import + autoImport pointer +omniroute setup-crush # ~/.config/crush/crush.json +omniroute setup-goose # ~/.config/goose/config.yaml +omniroute setup-qwen # ~/.qwen/settings.json +omniroute setup-aider # ~/.aider.conf.yml +omniroute setup-gemini # Gemini CLI (native /v1beta endpoint) +``` + +Each accepts `--remote --api-key ` to configure a local tool against a +**remote** OmniRoute, plus `--dry-run` to preview. The launchers +`omniroute launch` (Claude Code) and `omniroute launch-codex` (Codex) spawn the CLI +with the right env injected, writing no config at all. + +For the full table (what each command writes, every flag, local vs remote, base-URL +`/v1` conventions), see **[CLI Integrations](./CLI-INTEGRATIONS.md)**. + For detailed per-tool configuration (Claude Code, Codex CLI, Cursor, Cline, OpenClaw, Kilo Code, Copilot, and more), see the dedicated **[CLI Tools Guide](../reference/CLI-TOOLS.md)**. --- diff --git a/docs/reference/API_REFERENCE.md b/docs/reference/API_REFERENCE.md index 833f8dd015..3eedb525ce 100644 --- a/docs/reference/API_REFERENCE.md +++ b/docs/reference/API_REFERENCE.md @@ -63,6 +63,7 @@ Content-Type: application/json | Header | Direction | Description | | ------------------------ | --------- | ------------------------------------------------ | | `X-OmniRoute-No-Cache` | Request | Set to `true` to bypass cache | +| `x-omniroute-no-memory` | Request | Set to `true` to skip memory + skills injection for this request (mirrors no-cache; avoids the per-call token/cost overhead) | | `X-OmniRoute-Progress` | Request | Set to `true` for progress events | | `X-Session-Id` | Request | Sticky session key for external session affinity | | `x_session_id` | Request | Underscore variant also accepted (direct HTTP) | diff --git a/docs/reference/CLI-TOOLS.md b/docs/reference/CLI-TOOLS.md index e6e6fa726b..c4456eae8e 100644 --- a/docs/reference/CLI-TOOLS.md +++ b/docs/reference/CLI-TOOLS.md @@ -1,12 +1,12 @@ --- -title: "CLI Tools — OmniRoute v3.8.6" -version: 3.8.6 -lastUpdated: 2026-05-28 +title: "CLI Tools — OmniRoute" +version: 3.8.31 +lastUpdated: 2026-06-20 --- -# CLI Tools — OmniRoute v3.8.6 +# CLI Tools — OmniRoute -Last updated: 2026-05-28 +Last updated: 2026-06-20 OmniRoute integrates with three categories of CLI tools spread across three dedicated dashboard pages: @@ -45,6 +45,33 @@ ACP Agents (reverse spawn flow): --- +## Auto-configure with `setup-*` + +You do not have to write each tool's config by hand. OmniRoute ships a `setup-*` +command per supported CLI that reads the **live** model catalog from a running +OmniRoute (local or remote) and writes the tool's own config on your machine: + +```bash +omniroute setup-codex omniroute setup-claude omniroute setup-opencode +omniroute setup-cline omniroute setup-kilo omniroute setup-continue +omniroute setup-cursor omniroute setup-roo omniroute setup-crush +omniroute setup-goose omniroute setup-qwen omniroute setup-aider +omniroute setup-gemini +``` + +Each accepts `--remote --api-key ` (configure a local tool against a +remote OmniRoute), `--dry-run` (preview without writing), and `--port`. Tools +without model auto-discovery (Cline, Kilo, Roo, Goose, Qwen, Aider, Gemini) take +`--model ` (and `--yes` for non-interactive runs). The launchers +`omniroute launch` (Claude Code) and `omniroute launch-codex` (Codex) spawn the CLI +with the right env injected and write no config at all. + +> **Full reference:** the master table — what each command writes, every flag, +> local vs remote, and which tools want a `/v1` suffix — lives in +> **[CLI Integrations](../guides/CLI-INTEGRATIONS.md)**. + +--- + ## Source of Truth The unified catalog lives in `src/shared/constants/cliTools.ts` as `CLI_TOOLS: Record`. diff --git a/docs/reference/ENVIRONMENT.md b/docs/reference/ENVIRONMENT.md index 632e940a33..73d160e7f6 100644 --- a/docs/reference/ENVIRONMENT.md +++ b/docs/reference/ENVIRONMENT.md @@ -1,7 +1,7 @@ --- title: "Environment Variables Reference" -version: 3.8.2 -lastUpdated: 2026-05-13 +version: 3.8.31 +lastUpdated: 2026-06-20 --- # Environment Variables Reference @@ -766,12 +766,34 @@ Anthropic-compatible provider instead. | `PROVIDER_COOLDOWN_ENABLED` | _(unset → off)_ | `open-sse/services/providerCooldownTracker.ts` | Opt-in global cross-request provider/connection cooldown tracking. OFF by default (overlaps Connection Cooldown / Provider Circuit Breaker). Accepts `true`/`1`/`on` to enable. | | `PROVIDER_COOLDOWN_MIN_MS` | `5000` | `open-sse/services/providerCooldownTracker.ts` | Minimum cooldown (ms) before a failed provider/connection is retried. Scaled exponentially with consecutive failures. Only used when `PROVIDER_COOLDOWN_ENABLED`. | | `PROVIDER_COOLDOWN_MAX_MS` | `300000` (5 min) | `open-sse/services/providerCooldownTracker.ts` | Maximum cooldown (ms) cap before a failed provider/connection is retried regardless. Only used when `PROVIDER_COOLDOWN_ENABLED`. | -| `STREAM_RECOVERY_ENABLED` | _(unset → off)_ | `open-sse/services/streamRecovery.ts` | Opt-in transparent recovery of truncated upstream streams (free-claude-code port). Holds the opening SSE window so an early cutoff is retried invisibly; OFF by default (adds time-to-first-token latency). Accepts `true`/`1`/`on` to enable. | -| `STREAM_RECOVERY_MIDSTREAM_ENABLED` | _(unset → off)_ | `open-sse/services/streamRecovery.ts` | Opt-in mid-stream continuation (Fase 4.4): after a post-commit truncation, re-request with the partial text as an assistant prefill and stitch the missing suffix (plain-text OpenAI-compatible streams only, never with a tool call in flight). OFF by default — the recovered tail arrives as one burst. Independent of `STREAM_RECOVERY_ENABLED`. Accepts `true`/`1`/`on` to enable. | +| `STREAM_RECOVERY_ENABLED` | _(unset → off)_ | `src/lib/resilience/settings.ts` (seed) → `open-sse/services/streamRecovery.ts` (logic) | **What:** transparent recovery of truncated upstream streams (free-claude-code port). Holds the opening SSE window up to `STREAM_RECOVERY.HOLDBACK_MS` (750 ms) so a *pre-commit* cutoff — one that happens before any byte reaches the client — is re-opened and retried invisibly. **When to enable:** flaky/upstreams that frequently 0-byte-truncate at stream start; leave OFF if you cannot afford up to 750 ms of added time-to-first-token on every stream. Accepts `true`/`1`/`on`. Seeds the persisted Resilience setting; the Dashboard setting wins once set. | +| `STREAM_RECOVERY_MIDSTREAM_ENABLED` | _(unset → off)_ | `src/lib/resilience/settings.ts` (seed) → `open-sse/services/streamRecovery.ts` (logic) | **What:** mid-stream continuation (Fase 4.4) — after a *post-commit* truncation (bytes already reached the client), re-request with the partial text as an assistant prefill and stitch the missing suffix. Plain-text OpenAI-compatible streams only; never fires with a tool call in flight. **When to enable:** long generations that get cut mid-answer and you accept the recovered tail arriving as one burst rather than token-by-token. Independent of `STREAM_RECOVERY_ENABLED` (different risk profile). Accepts `true`/`1`/`on`. | | `HEALTHCHECK_STAGGER_MS` | `3000` | `src/lib/tokenHealthCheck.ts` | Stagger interval (ms) between provider token healthchecks at startup. | | `REQUEST_RETRY` | `2` | `src/sse/services/cooldownAwareRetry.ts` | Number of automatic retries on model-scoped cooldown responses before returning error to client. | | `MAX_RETRY_INTERVAL_SEC` | `30` | `src/sse/services/cooldownAwareRetry.ts` | Max backoff interval (seconds) between cooldown retries. Capped by this value regardless of upstream `Retry-After`. | +### Stream-recovery tuning constants (not env vars) + +The two `STREAM_RECOVERY_*` flags above are the only operator-facing toggles. The +recovery behavior is otherwise tuned by hardcoded constants in +`open-sse/config/constants.ts` (`STREAM_RECOVERY`), shown here for reference — +changing them requires a code edit, not an env var: + +- `STREAM_RECOVERY.HOLDBACK_MS = 750` — how long the opening SSE window is held + so an early truncation can be retried before any byte is committed to the client. +- `STREAM_RECOVERY.BUFFER_MAX_BYTES = 65536` — hard cap on the held window; commit + (flush + passthrough) as soon as this many bytes accumulate, regardless of the timer. +- `STREAM_RECOVERY.EARLY_RETRY_MAX = 4` — max transparent re-opens of the upstream + stream while the holdback is still uncommitted. + +> **Per-provider sliding-window rate limit (no env var):** the FCC-ported +> per-provider sliding-window rate-limit *fallback* exists in code +> (`open-sse/services/providerDefaultRateLimit.ts`, wired through +> `open-sse/services/rateLimitManager.ts`) but ships with an **empty default map** +> and has **no operator env var** today — it is enabled only via a test hook / +> code edit. It is intentionally not listed in the table above. The per-`(token, IP)` +> relay limiter that *does* have a knob is `RELAY_IP_PER_MINUTE` (§3 Network & Ports). + --- ## 22. Debugging diff --git a/docs/reference/PROVIDER_REFERENCE.md b/docs/reference/PROVIDER_REFERENCE.md index e0ba3563d1..07699741c0 100644 --- a/docs/reference/PROVIDER_REFERENCE.md +++ b/docs/reference/PROVIDER_REFERENCE.md @@ -1,16 +1,16 @@ --- title: "Provider Reference" -version: 3.8.27 -lastUpdated: 2026-06-17 +version: 3.8.31 +lastUpdated: 2026-06-20 --- # Provider Reference > **Auto-generated** from `src/shared/constants/providers.ts` — do not edit by hand. > Regenerate with: `npm run gen:provider-reference` -> **Last generated:** 2026-06-17 +> **Last generated:** 2026-06-20 -Total providers: **227**. See category breakdown below. +Total providers: **231**. See category breakdown below. ## Categories @@ -33,277 +33,281 @@ Use the dashboard at `/dashboard/providers` to enable, configure, and test each ## OAuth Providers (19) -| ID | Alias | Name | Tags | Website | Notes | -| ------------- | ------------ | -------------------- | ----- | ------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `agy` | `agy` | Antigravity CLI | OAuth | [link](https://antigravity.google) | Import your Antigravity CLI (`agy`) login (paste/upload its token file), auto-detect a local CLI login, or sign in with Google. Shares the Antigravity backend (incl. Claude models). | -| `amazon-q` | `aq` | Amazon Q | OAuth | [link](https://aws.amazon.com/q/developer/) | Uses the same AWS Builder ID or imported refresh-token flow as Kiro, but keeps Amazon Q connections separate. | -| `antigravity` | — | Antigravity | OAuth | — | — | -| `claude` | `cc` | Claude Code | OAuth | — | — | -| `cline` | `cl` | Cline | OAuth | — | — | -| `codex` | `cx` | OpenAI Codex | OAuth | — | — | -| `cursor` | `cu` | Cursor IDE | OAuth | — | — | -| `devin-cli` | `dv` | Devin CLI (Official) | OAuth | [link](https://cli.devin.ai) | Requires the Devin CLI binary. Run `devin auth login` to authenticate, or provide your WINDSURF_API_KEY. Install: https://cli.devin.ai | -| `gemini-cli` | `gemini-cli` | Gemini CLI | OAuth | — | Uses Gemini CLI OAuth / Cloud Code credentials. Pro models require an eligible Google account or paid plan. | -| `github` | `gh` | GitHub Copilot | OAuth | — | — | -| `gitlab-duo` | `gitlab-duo` | GitLab Duo | OAuth | [link](https://docs.gitlab.com/user/duo_agent_platform/code_suggestions/) | OAuth application with ai_features + read_user scopes. Configure GITLAB_DUO_OAUTH_CLIENT_ID and optionally GITLAB_DUO_OAUTH_CLIENT_SECRET on this OmniRoute instance. | -| `kilocode` | `kc` | Kilo Code | OAuth | — | — | -| `kimi-coding` | `kmc` | Kimi Coding | OAuth | — | — | -| `kiro` | `kr` | Kiro AI | OAuth | — | Free tier: 50 credits/month (~25K–100K tokens). ⚠️ Kiro ToS prohibits third-party proxy/harness use. | -| `qoder` | `if` | Qoder AI | OAuth | — | — | -| `qwen` | `qw` | Qwen Code | OAuth | — | ⚠️ **DEPRECATED.** Qwen OAuth free tier was discontinued on 2026-04-15. Use 'bailian-coding-plan', 'alibaba', 'alibaba-cn', or 'openrouter' provider with API key instead. | -| `trae` | `tr` | Trae | OAuth | [link](https://trae.ai) | Trae is an AI-native IDE by ByteDance (SOLO remote agent). Authorize via trae.ai in the popup, or sign in at solo.trae.ai and paste the Cloud-IDE-JWT (sent as 'Authorization: Cloud-IDE-JWT ', ~14-day lifetime) as the access token; web_id/biz_user_id/user_unique_id/scope/tenant/region propagate via providerSpecificData. No headless refresh for pasted tokens — re-paste on expiry. | -| `windsurf` | `ws` | Windsurf (Devin CLI) | OAuth | [link](https://windsurf.com) | In the Windsurf / VS Code IDE, open the command palette and run `Windsurf: Provide Auth Token` (or click the Jupyter "Get Windsurf Authentication Token" button), then copy the shown token and paste it here. Note: opening windsurf.com/show-auth-token directly only renders a "Redirecting" page — the IDE must initiate the flow (it adds a `?state=...` param) for the token to appear. | -| `zed` | `zd` | Zed IDE | OAuth | [link](https://zed.dev) | Zed stores LLM provider credentials (OpenAI, Anthropic, Google, Mistral, xAI) in the OS keychain. Use the Import button below to discover and import them automatically. | +| ID | Alias | Name | Tags | Website | Notes | +|----|-------|------|------|---------|-------| +| `agy` | `agy` | Antigravity CLI | OAuth | [link](https://antigravity.google) | Import your Antigravity CLI (`agy`) login (paste/upload its token file), auto-detect a local CLI login, or sign in with Google. Shares the Antigravity backend (incl. Claude models). | +| `amazon-q` | `aq` | Amazon Q | OAuth | [link](https://aws.amazon.com/q/developer/) | Uses the same AWS Builder ID or imported refresh-token flow as Kiro, but keeps Amazon Q connections separate. | +| `antigravity` | — | Antigravity | OAuth | — | — | +| `claude` | `cc` | Claude Code | OAuth | — | — | +| `cline` | `cl` | Cline | OAuth | — | — | +| `codex` | `cx` | OpenAI Codex | OAuth | — | — | +| `cursor` | `cu` | Cursor IDE | OAuth | — | — | +| `devin-cli` | `dv` | Devin CLI (Official) | OAuth | [link](https://cli.devin.ai) | Requires the Devin CLI binary. Run `devin auth login` to authenticate, or provide your WINDSURF_API_KEY. Install: https://cli.devin.ai | +| `gemini-cli` | `gemini-cli` | Gemini CLI | OAuth | — | Uses Gemini CLI OAuth / Cloud Code credentials. Pro models require an eligible Google account or paid plan. | +| `github` | `gh` | GitHub Copilot | OAuth | — | — | +| `gitlab-duo` | `gitlab-duo` | GitLab Duo | OAuth | [link](https://docs.gitlab.com/user/duo_agent_platform/code_suggestions/) | OAuth application with ai_features + read_user scopes. Configure GITLAB_DUO_OAUTH_CLIENT_ID and optionally GITLAB_DUO_OAUTH_CLIENT_SECRET on this OmniRoute instance. | +| `kilocode` | `kc` | Kilo Code | OAuth | — | — | +| `kimi-coding` | `kmc` | Kimi Coding | OAuth | — | — | +| `kiro` | `kr` | Kiro AI | OAuth | — | Free tier: 50 credits/month (~25K–100K tokens). ⚠️ Kiro ToS prohibits third-party proxy/harness use. | +| `qoder` | `if` | Qoder AI | OAuth | — | — | +| `qwen` | `qw` | Qwen Code | OAuth | — | ⚠️ **DEPRECATED.** Qwen OAuth free tier was discontinued on 2026-04-15. Use 'bailian-coding-plan', 'alibaba', 'alibaba-cn', or 'openrouter' provider with API key instead. | +| `trae` | `tr` | Trae | OAuth | [link](https://trae.ai) | Trae is an AI-native IDE by ByteDance (SOLO remote agent). Authorize via trae.ai in the popup, or sign in at solo.trae.ai and paste the Cloud-IDE-JWT (sent as 'Authorization: Cloud-IDE-JWT ', ~14-day lifetime) as the access token; web_id/biz_user_id/user_unique_id/scope/tenant/region propagate via providerSpecificData. No headless refresh for pasted tokens — re-paste on expiry. | +| `windsurf` | `ws` | Windsurf (Devin CLI) | OAuth | [link](https://windsurf.com) | In the Windsurf / VS Code IDE, open the command palette and run `Windsurf: Provide Auth Token` (or click the Jupyter "Get Windsurf Authentication Token" button), then copy the shown token and paste it here. Note: opening windsurf.com/show-auth-token directly only renders a "Redirecting" page — the IDE must initiate the flow (it adds a `?state=...` param) for the token to appear. | +| `zed` | `zd` | Zed IDE | OAuth | [link](https://zed.dev) | Zed stores LLM provider credentials (OpenAI, Anthropic, Google, Mistral, xAI) in the OS keychain. Use the Import button below to discover and import them automatically. | ## Web Cookie Providers (22) -| ID | Alias | Name | Tags | Website | Notes | -| ----------------- | ------------- | ---------------------------- | ---------- | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `adapta-web` | `adp-web` | Adapta.org (Adapta One Web) | Web cookie | [link](https://agent.adapta.one) | Paste your \_\_client cookie value from .clerk.agent.adapta.one (DevTools → Application → Cookies) | -| `blackbox-web` | `bb-web` | Blackbox Web (Subscription) | Web cookie | [link](https://app.blackbox.ai) | Paste your \_\_Secure-authjs.session-token value or full cookie header from app.blackbox.ai | -| `chatgpt-web` | `cgpt-web` | ChatGPT Web (Plus/Pro) | Web cookie | [link](https://chatgpt.com) | Paste your \_\_Secure-next-auth.session-token cookie value from chatgpt.com | -| `claude-web` | `cw` | Claude Web | Web cookie | [link](https://claude.ai) | Paste your session cookie from claude.ai | -| `copilot-web` | `copilot` | Microsoft Copilot Web | Web cookie | [link](https://copilot.microsoft.com) | Paste your access_token from copilot.microsoft.com (or export a .har file from DevTools while logged in) | -| `deepseek-web` | `ds-web` | DeepSeek Web | Web cookie | [link](https://chat.deepseek.com) | Paste your userToken from chat.deepseek.com — DevTools → Application → Local Storage → userToken | -| `doubao-web` | `db` | Doubao Web (ByteDance) | Web cookie | [link](https://www.doubao.com) | Paste your session cookie from doubao.com (DevTools → Application → Cookies) | -| `gemini-business` | `gembiz` | Gemini Business (Enterprise) | Web cookie | [link](https://business.gemini.google) | From your enterprise account: open business.gemini.google/home/cid/{your-cid}, then copy **Secure-1PSID and **Secure-1PSIDTS cookies from DevTools → Application → Cookies. Paste as a cookie header below. | -| `gemini-web` | `gweb` | Gemini Web (Free) | Web cookie | [link](https://gemini.google.com) | Paste your **Secure-1PSID cookie value from gemini.google.com. Optionally add **Secure-1PSIDTS separated by semicolon. | -| `grok-web` | `gw` | Grok Web (Subscription) | Web cookie | [link](https://grok.com) | Paste the full grok.com cookie line from DevTools → Application → Cookies. Include both `sso` and `sso-rw` (e.g. `sso=...; sso-rw=...`) — Grok's anti-bot rejects `sso` on its own. | -| `huggingchat` | `huggingchat` | HuggingChat (Free) | Web cookie | [link](https://huggingface.co/chat) | Paste your hf-chat cookie value from huggingface.co/chat (DevTools → Application → Cookies → hf-chat). Optional — works without auth for basic use. | -| `inner-ai` | `in-ai` | Inner.ai (Subscription) | Web cookie | [link](https://app.innerai.com) | Paste your token cookie and email separated by a space: open DevTools → Application → Cookies → .innerai.com, copy the token value, then append a space and your Inner.ai login email. Example: eyJhbG... user@example.com | -| `kimi-web` | `kimi-web` | Kimi Web (Moonshot AI) | Web cookie | [link](https://kimi.moonshot.cn) | Paste your session cookie from kimi.moonshot.cn (DevTools → Application → Cookies) | -| `lmarena` | `lma` | LMArena (Free) | Web cookie | [link](https://lmarena.ai) | Paste your session cookie from lmarena.ai (DevTools → Application → Cookies). Optional — works with free tier for basic comparisons. | -| `muse-spark-web` | `ms-web` | Muse Spark Web (Meta AI) | Web cookie | [link](https://www.meta.ai) | Paste your abra_sess value or full cookie header from meta.ai | -| `perplexity-web` | `pplx-web` | Perplexity Web (Pro/Max) | Web cookie | [link](https://www.perplexity.ai) | Paste your \_\_Secure-next-auth.session-token cookie value from perplexity.ai | -| `phind` | `ph` | Phind (Free) | Web cookie | [link](https://www.phind.com) | Paste your session cookie from phind.com (DevTools → Application → Cookies). Optional — works with free tier. | -| `poe-web` | `poe` | Poe Web (Subscription) | Web cookie | [link](https://poe.com) | Paste your p-b cookie value from poe.com (DevTools → Application → Cookies → p-b) | -| `qwen-web` | `qwen-web` | Qwen Web (Free) | Web cookie | [link](https://chat.qwen.ai) | Open chat.qwen.ai, log in, then open DevTools → Application → Local Storage → copy the "token" value (or use tongyi_sso_ticket cookie as Bearer token). | -| `t3-web` | `t3chat` | t3.chat (Pro/Free) | Web cookie | [link](https://t3.chat) | Open t3.chat in your browser, log in, then open DevTools → Application → Local Storage → https://t3.chat. Copy the value of 'convex-session-id'. Also open DevTools → Network, copy the Cookie header from any request. Paste both values here. See provider setup docs for a step-by-step guide. | -| `v0-vercel-web` | `v0` | v0 Vercel Web (Code Gen) | Web cookie | [link](https://v0.dev) | Paste your session cookie from v0.dev (DevTools → Application → Cookies) | -| `venice-web` | `ven` | Venice Web (Privacy) | Web cookie | [link](https://venice.ai) | Paste your session cookie from venice.ai (DevTools → Application → Cookies) | +| ID | Alias | Name | Tags | Website | Notes | +|----|-------|------|------|---------|-------| +| `adapta-web` | `adp-web` | Adapta.org (Adapta One Web) | Web cookie | [link](https://agent.adapta.one) | Paste your __client cookie value from .clerk.agent.adapta.one (DevTools → Application → Cookies) | +| `blackbox-web` | `bb-web` | Blackbox Web (Subscription) | Web cookie | [link](https://app.blackbox.ai) | Paste your __Secure-authjs.session-token value or full cookie header from app.blackbox.ai | +| `chatgpt-web` | `cgpt-web` | ChatGPT Web (Plus/Pro) | Web cookie | [link](https://chatgpt.com) | Paste your __Secure-next-auth.session-token cookie value from chatgpt.com | +| `claude-web` | `cw` | Claude Web | Web cookie | [link](https://claude.ai) | Paste your session cookie from claude.ai | +| `copilot-web` | `copilot` | Microsoft Copilot Web | Web cookie | [link](https://copilot.microsoft.com) | Paste your access_token from copilot.microsoft.com (or export a .har file from DevTools while logged in) | +| `deepseek-web` | `ds-web` | DeepSeek Web | Web cookie | [link](https://chat.deepseek.com) | Paste your userToken from chat.deepseek.com — DevTools → Application → Local Storage → userToken | +| `doubao-web` | `db` | Doubao Web (ByteDance) | Web cookie | [link](https://www.doubao.com) | Paste your session cookie from doubao.com (DevTools → Application → Cookies) | +| `gemini-business` | `gembiz` | Gemini Business (Enterprise) | Web cookie | [link](https://business.gemini.google) | From your enterprise account: open business.gemini.google/home/cid/{your-cid}, then copy __Secure-1PSID and __Secure-1PSIDTS cookies from DevTools → Application → Cookies. Paste as a cookie header below. | +| `gemini-web` | `gweb` | Gemini Web (Free) | Web cookie | [link](https://gemini.google.com) | Paste your __Secure-1PSID cookie value from gemini.google.com. Optionally add __Secure-1PSIDTS separated by semicolon. | +| `grok-web` | `gw` | Grok Web (Subscription) | Web cookie | [link](https://grok.com) | Paste the full grok.com cookie line from DevTools → Application → Cookies. Include both `sso` and `sso-rw` (e.g. `sso=...; sso-rw=...`) — Grok's anti-bot rejects `sso` on its own. | +| `huggingchat` | `huggingchat` | HuggingChat (Free) | Web cookie | [link](https://huggingface.co/chat) | Paste your hf-chat cookie value from huggingface.co/chat (DevTools → Application → Cookies → hf-chat). Optional — works without auth for basic use. | +| `inner-ai` | `in-ai` | Inner.ai (Subscription) | Web cookie | [link](https://app.innerai.com) | Paste your token cookie and email separated by a space: open DevTools → Application → Cookies → .innerai.com, copy the token value, then append a space and your Inner.ai login email. Example: eyJhbG... user@example.com | +| `kimi-web` | `kimi-web` | Kimi Web (Moonshot AI) | Web cookie | [link](https://kimi.moonshot.cn) | Paste your session cookie from kimi.moonshot.cn (DevTools → Application → Cookies) | +| `lmarena` | `lma` | LMArena (Free) | Web cookie | [link](https://lmarena.ai) | Paste the full Cookie header from lmarena.ai (DevTools → Network → request → Cookie). The session is now split across arena-auth-prod-v1.0, .1, … — copy the whole header. Optional — works with free tier for basic comparisons. | +| `muse-spark-web` | `ms-web` | Muse Spark Web (Meta AI) | Web cookie | [link](https://www.meta.ai) | Paste your abra_sess value or full cookie header from meta.ai | +| `perplexity-web` | `pplx-web` | Perplexity Web (Pro/Max) | Web cookie | [link](https://www.perplexity.ai) | Paste your __Secure-next-auth.session-token cookie value from perplexity.ai | +| `phind` | `ph` | Phind (Free) | Web cookie | [link](https://www.phind.com) | ⚠️ **DEPRECATED.** Phind shut down its API (2026-01); the /api/chat endpoint no longer serves (sweep 2026-06-19). | +| `poe-web` | `poe` | Poe Web (Subscription) | Web cookie | [link](https://poe.com) | Paste your p-b cookie value from poe.com (DevTools → Application → Cookies → p-b) | +| `qwen-web` | `qwen-web` | Qwen Web (Free) | Web cookie | [link](https://chat.qwen.ai) | Open chat.qwen.ai, log in, then open DevTools → Application → Local Storage → copy the "token" value (or use tongyi_sso_ticket cookie as Bearer token). | +| `t3-web` | `t3chat` | t3.chat (Pro/Free) | Web cookie | [link](https://t3.chat) | Open t3.chat in your browser, log in, then open DevTools → Application → Local Storage → https://t3.chat. Copy the value of 'convex-session-id'. Also open DevTools → Network, copy the Cookie header from any request. Paste both values here. See provider setup docs for a step-by-step guide. | +| `v0-vercel-web` | `v0` | v0 Vercel Web (Code Gen) | Web cookie | [link](https://v0.dev) | Paste your session cookie from v0.dev (DevTools → Application → Cookies) | +| `venice-web` | `ven` | Venice Web (Privacy) | Web cookie | [link](https://venice.ai) | Paste your session cookie from venice.ai (DevTools → Application → Cookies) | -## API Key Providers (paid / paid-with-free-credits) (153) +## API Key Providers (paid / paid-with-free-credits) (157) -| ID | Alias | Name | Tags | Website | Notes | -| --------------------- | -------------- | ------------------------------- | --------------------- | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `360ai` | `360ai` | 360 AI | API key | [link](https://ai.360.cn) | Get API key at ai.360.cn | -| `agentrouter` | `agentrouter` | AgentRouter | API key, aggregator | [link](https://agentrouter.org) | $200 free credits on signup - multi-model routing gateway | -| `ai21` | `ai21` | AI21 Labs | API key | [link](https://www.ai21.com) | $10 trial credits on signup (valid 3 months), no credit card required | -| `aimlapi` | `aiml` | AI/ML API | API key, aggregator | [link](https://aimlapi.com) | $0.025/day free credits — 200+ models (GPT-4o, Claude, Gemini, Llama) via single endpoint | -| `alibaba` | `ali` | Alibaba | API key | [link](https://dashscope-intl.aliyuncs.com) | — | -| `alibaba-cn` | `ali-cn` | Alibaba (China) | API key | [link](https://dashscope.aliyuncs.com) | — | -| `anthropic` | `anthropic` | Anthropic | API key | [link](https://platform.claude.com) | — | -| `api-airforce` | `af` | Api.airforce | API key | [link](https://api.airforce) | 55 free tier models including Grok-3, Claude 3.7, Qwen3, Kimi-K2, Gemini 2.5 Flash, DeepSeek-V3 | -| `arcee-ai` | `arcee` | Arcee AI | API key | [link](https://arcee.ai) | Get API key at arcee.ai | -| `azure-ai` | `azure-ai` | Azure AI Foundry | API key, enterprise | [link](https://learn.microsoft.com/azure/ai-foundry) | Use your Azure AI Foundry key. Base URL can be https://.services.ai.azure.com/openai/v1/ or https://.openai.azure.com/openai/v1/. | -| `azure-openai` | `azure` | Azure OpenAI | API key, enterprise | [link](https://azure.microsoft.com/products/ai-services/openai-service) | Use your Azure OpenAI API key. Base URL should be your resource endpoint, for example https://my-resource.openai.azure.com. | -| `baichuan` | `baichuan` | Baichuan | API key | [link](https://baichuan.com) | Get API key at platform.baichuan-ai.com | -| `baidu` | `baidu` | Baidu (ERNIE) | API key | [link](https://yiyan.baidu.com) | Get API key at console.bce.baidu.com | -| `bailian-coding-plan` | `bcp` | Alibaba Coding Plan | API key | [link](https://www.alibabacloud.com/help/en/model-studio/coding-plan) | — | -| `baseten` | `baseten` | Baseten | API key | [link](https://baseten.co) | $30 free trial credits for GPU inference | -| `bazaarlink` | `bzl` | BazaarLink | API key | [link](https://bazaarlink.ai) | Free tier with auto:free routing — zero-cost inference, no credit card required | -| `bedrock` | `bedrock` | Amazon Bedrock | API key, enterprise | [link](https://aws.amazon.com/bedrock) | Use your Amazon Bedrock API key and configure the AWS region where your models are enabled (for example eu-west-2). OmniRoute calls Bedrock's native Converse API directly. | -| `black-forest-labs` | `bfl` | Black Forest Labs | API key, image | [link](https://blackforestlabs.ai) | — | -| `blackbox` | `bb` | Blackbox AI | API key | [link](https://blackbox.ai) | Free tier: unlimited basic chat plus Minimax-M2.5, no credit card required | -| `bluesminds` | `bm` | BluesMinds | API key | [link](https://www.bluesminds.com) | Free daily pi credits — supports 200+ models including GPT-4o, GPT-4.1, Claude Sonnet 4.5, Gemini 2.0 Flash, DeepSeek V4, Qwen, Kimi K2 | -| `byteplus` | `bpm` | BytePlus ModelArk | API key | [link](https://console.byteplus.com/ark) | — | -| `bytez` | `bytez` | Bytez | API key | [link](https://bytez.com) | $1 free credits, refreshes every 4 weeks | -| `cablyai` | `cablyai` | CablyAI | API key, aggregator | [link](https://cablyai.com) | Bearer API key for the CablyAI OpenAI-compatible gateway. | -| `cerebras` | `cerebras` | Cerebras | API key | [link](https://inference.cerebras.ai) | Free Trial: 1M tokens/day, 30K TPM, 5 RPM — no credit card. | -| `chutes` | `chutes` | Chutes.ai | API key, aggregator | [link](https://chutes.ai) | Bearer API key for the Chutes OpenAI-compatible gateway. | -| `clarifai` | `clarifai` | Clarifai | API key, enterprise | [link](https://docs.clarifai.com) | Use your Clarifai PAT or app-specific API key. OmniRoute targets the OpenAI-compatible endpoint at https://api.clarifai.com/v2/ext/openai/v1 and authenticates with Authorization: Key . | -| `cloudflare-ai` | `cf` | Cloudflare Workers AI | API key | [link](https://developers.cloudflare.com/workers-ai) | Requires API Token AND Account ID (found at dash.cloudflare.com) | -| `codestral` | `codestral` | Codestral | API key | [link](https://mistral.ai) | — | -| `cohere` | `cohere` | Cohere | API key | [link](https://cohere.com) | Free Trial: 1,000 API calls/month for testing, no credit card required | -| `command-code` | `cmd` | Command Code | API key | [link](https://commandcode.ai/) | Use a Command Code API key. Requests are sent to Command Code's /alpha/generate endpoint. | -| `coze` | `coze` | Coze | API key | [link](https://coze.com) | Get API key at coze.com/open/api | -| `crof` | `crof` | CrofAI | API key | [link](https://crof.ai) | — | -| `databricks` | `databricks` | Databricks | API key, enterprise | [link](https://www.databricks.com) | — | -| `datarobot` | `datarobot` | DataRobot | API key, enterprise | [link](https://docs.datarobot.com) | Use your DataRobot API token. Optional Base URL can be the account root (for LLM Gateway) or a deployment URL under /api/v2/deployments/. | -| `deepinfra` | `deepinfra` | DeepInfra | API key | [link](https://deepinfra.com) | Free signup credits for API testing and model exploration | -| `deepseek` | `ds` | DeepSeek | API key | [link](https://platform.deepseek.com) | 5M free tokens on signup - no credit card required | -| `dify` | `dify` | Dify | API key | [link](https://dify.ai) | Get API key from your Dify instance. | -| `doubao` | `doubao` | Doubao | API key | [link](https://doubao.com) | Get API key at console.volcengine.com | -| `empower` | `empower` | Empower | API key, aggregator | [link](https://docs.empower.dev) | Bearer API key for the Empower OpenAI-compatible endpoint. | -| `fal-ai` | `fal` | Fal.ai | API key, image | [link](https://fal.ai) | — | -| `featherless-ai` | `featherless` | Featherless AI | API key | [link](https://featherless.ai) | Free tier available — no credit card required | -| `fenayai` | `fenayai` | FenayAI | API key, aggregator | [link](https://fenayai.com) | Bearer API key for the FenayAI OpenAI-compatible gateway. | -| `firecrawl` | `fc` | Firecrawl | API key | [link](https://firecrawl.dev) | — | -| `fireworks` | `fireworks` | Fireworks AI | API key | [link](https://fireworks.ai) | $1 free starter credits on signup for API testing | -| `freeaiapikey` | `faik` | FreeAIAPIKey | API key | [link](https://freeaiapikey.com) | — | -| `freemodel-dev` | `fmd` | FreeModel.dev | API key | [link](https://freemodel.dev) | $300 free credits on signup — no credit card required. Access GPT-5.4 and GPT-5.5 (OpenAI's latest flagship models) through an OpenAI-compatible API. | -| `friendliai` | `friendli` | FriendliAI | API key | [link](https://friendli.ai) | Free tier for serverless inference — no credit card required | -| `galadriel` | `galadriel` | Galadriel | API key | [link](https://galadriel.com) | — | -| `gemini` | `gemini` | Gemini (Google AI Studio) | API key | [link](https://aistudio.google.com) | Free forever: 1,500 req/day for Gemini 2.5 Flash — no credit card, get key at aistudio.google.com | -| `getgoapi` | `ggo` | GoAPI | API key, aggregator | [link](https://api.getgoapi.com) | — | -| `gigachat` | `gigachat` | GigaChat (Sber) | API key | [link](https://developers.sber.ru) | — | -| `github-models` | `ghm` | GitHub Models | API key | [link](https://github.com/marketplace/models) | Create a GitHub PAT with 'models: read' scope at github.com/settings/tokens | -| `gitlab` | `gitlab` | GitLab Duo PAT | API key | [link](https://docs.gitlab.com/user/duo_agent_platform/code_suggestions/) | GitLab personal access token for the public Code Suggestions API. Configure a self-hosted base URL when not using gitlab.com. | -| `gitlawb` | `glb` | Gitlawb Opengateway (MiMo) | API key | [link](https://opengateway.gitlawb.com) | Free tier available — no credit card required | -| `gitlawb-gmi` | `glb-gmi` | Gitlawb Opengateway (GMI Cloud) | API key | [link](https://opengateway.gitlawb.com) | Free tier available — no credit card required | -| `glhf` | `glhf` | GLHF Chat | API key, aggregator | [link](https://glhf.chat) | Bearer API key for the GLHF OpenAI-compatible gateway. | -| `glm` | `glm` | GLM Coding | API key | [link](https://z.ai/subscribe) | — | -| `glm-cn` | `glmcn` | GLM Coding (China) | API key | [link](https://open.bigmodel.cn) | — | -| `glmt` | `glmt` | GLM Thinking | API key | [link](https://open.bigmodel.cn) | — | -| `groq` | `groq` | Groq | API key | [link](https://groq.com) | Free tier: 30 RPM / 14.4K RPD — no credit card | -| `hackclub` | `hc` | Hackclub AI | API key, aggregator | [link](https://ai.hackclub.com) | Sign in with your Hack Club account at ai.hackclub.com. | -| `haiper` | `hp` | Haiper | API key, video | [link](https://haiper.ai) | Get API key at haiper.ai/haiper-api | -| `heroku` | `heroku` | Heroku AI | API key, enterprise | [link](https://www.heroku.com) | — | -| `huggingchat` | `huggingchat` | HuggingChat | API key | [link](https://huggingface.co/chat) | No API key required for basic access. | -| `huggingface` | `hf` | HuggingFace | API key | [link](https://huggingface.co) | Free Inference API for thousands of models (Whisper, VITS, SDXL…) | -| `hyperbolic` | `hyp` | Hyperbolic | API key | [link](https://hyperbolic.xyz) | $1-5 trial credits on signup for serverless inference | -| `ideogram` | `ideo` | Ideogram | API key | [link](https://ideogram.ai) | Get API key at ideogram.ai/docs/api | -| `iflytek` | `iflytek` | iFlytek Spark | API key | [link](https://xinghuo.xfyun.cn) | Get API key at console.xfyun.cn | -| `inclusionai` | `inclusion` | InclusionAI | API key | [link](https://inclusionai.com) | Get API key at inclusionai.com | -| `inference-net` | `inet` | Inference.net | API key | [link](https://inference.net) | $25 free credits on signup plus research grants available | -| `jina-ai` | `jina` | Jina AI | API key, embed/rerank | [link](https://jina.ai) | Bearer API key for the Jina AI rerank API. | -| `jina-reader` | `jr` | Jina Reader | API key | [link](https://jina.ai/reader) | — | -| `kie` | `kie` | KIE.AI | API key | [link](https://kie.ai) | — | -| `kilo-gateway` | `kg` | Kilo Gateway | API key, aggregator | [link](https://kilo.ai) | — | -| `kimi` | `kimi` | Kimi | API key | [link](https://platform.moonshot.ai) | — | -| `kimi-coding-apikey` | `kmca` | Kimi Coding (API Key) | API key | [link](https://www.kimi.com/code) | — | -| `kluster` | `kluster` | Kluster AI | API key | [link](https://kluster.ai) | $5 free credits on signup - DeepSeek R1, Llama 4 Maverick/Scout, Qwen3 235B | -| `lambda-ai` | `lambda` | Lambda AI | API key | [link](https://lambda.ai) | — | -| `laozhang` | `lz` | LaoZhang AI | API key, aggregator | [link](https://api.laozhang.ai) | — | -| `leonardo` | `leo` | Leonardo AI | API key, video | [link](https://leonardo.ai) | Get API key at leonardo.ai/developer | -| `liquid` | `liquid` | Liquid AI | API key | [link](https://liquid.ai) | Get API key at liquid.ai | -| `llamagate` | `llamagate` | LlamaGate | API key | [link](https://llamagate.ai) | — | -| `llm7` | `llm7` | LLM7.io | API key | [link](https://llm7.io) | No signup required - 2 req/s, 20 RPM, 100 req/hr free tier | -| `longcat` | `lc` | LongCat AI | API key | [link](https://longcat.chat/platform/docs) | Free: 5M tokens/day on LongCat-2.0-Preview (Flash models retired 2026-05-29); up to 120M/day via feedback. | -| `maritalk` | `maritalk` | Maritalk | API key | [link](https://www.maritaca.ai) | — | -| `meta-llama` | `meta` | Meta Llama API | API key | [link](https://llama.developer.meta.com) | — | -| `minimax` | `minimax` | Minimax Coding | API key, video | [link](https://www.minimax.io) | — | -| `minimax-cn` | `minimax-cn` | Minimax (China) | API key | [link](https://www.minimaxi.com) | — | -| `mistral` | `mistral` | Mistral | API key | [link](https://mistral.ai) | Free Experiment tier: rate-limited access to all models, no credit card required | -| `modal` | `mdl` | Modal | API key, enterprise | [link](https://modal.com/docs) | Use the bearer token that protects your Modal deployment, if enabled. Base URL should point to your OpenAI-compatible Modal app, for example https://--.modal.run/v1. | -| `monsterapi` | `monster` | MonsterAPI | API key | [link](https://monsterapi.ai) | Get API key at monsterapi.ai | -| `moonshot` | `moonshot` | Moonshot AI | API key | [link](https://platform.moonshot.ai) | — | -| `morph` | `morph` | Morph | API key | [link](https://morphllm.com) | Free tier: 250K credits/month, $0 | -| `nanogpt` | `nanogpt` | NanoGPT | API key | [link](https://nano-gpt.com) | — | -| `nebius` | `nebius` | Nebius AI | API key | [link](https://nebius.com) | ~$1 trial credits on signup for API testing | -| `nlpcloud` | `nlpc` | NLP Cloud | API key | [link](https://docs.nlpcloud.com) | Use your NLP Cloud API key in Authorization: Token . OmniRoute targets the chatbot endpoint on https://api.nlpcloud.io/v1/gpu//chatbot by default. | -| `nomic` | `nomic` | Nomic | API key | [link](https://nomic.ai) | Get API key at atlas.nomic.ai | -| `nous-research` | `nous` | Nous Research | API key | [link](https://portal.nousresearch.com/help) | Use your Nous Portal API key. OmniRoute targets the official OpenAI-compatible inference endpoint at https://inference-api.nousresearch.com/v1. | -| `novita` | `novita` | Novita AI | API key, aggregator | [link](https://novita.ai) | $0.50 trial credits on signup (valid about 1 year) | -| `nscale` | `nscale` | nScale | API key | [link](https://nscale.com) | $5 free credits on signup for inference testing | -| `nvidia` | `nvidia` | NVIDIA NIM | API key | [link](https://build.nvidia.com) | Free dev access: ~40 RPM, 70+ models (Kimi K2.5, GLM 4.7, DeepSeek V3.2...) | -| `oci` | `oci` | OCI Generative AI | API key, enterprise | [link](https://www.oracle.com/artificial-intelligence/generative-ai) | Use your OCI Generative AI API key or IAM bearer token. Base URL can be https://inference.generativeai..oci.oraclecloud.com/openai/v1/. | -| `ollama-cloud` | `ollamacloud` | Ollama Cloud | API key | [link](https://ollama.com/settings/api-keys) | — | -| `openai` | `openai` | OpenAI | API key | [link](https://platform.openai.com) | — | -| `opencode-go` | `opencode-go` | OpenCode Go | API key | [link](https://opencode.ai/go) | — | -| `opencode-zen` | `opencode-zen` | OpenCode Zen | API key | [link](https://opencode.ai/zen) | — | -| `openrouter` | `openrouter` | OpenRouter | API key, aggregator | [link](https://openrouter.ai) | Free models at $0/token with :free suffix - 20 RPM / 200 RPD | -| `orcarouter` | `orcarouter` | OrcaRouter | API key | [link](https://www.orcarouter.ai) | — | -| `ovhcloud` | `ovh` | OVHcloud AI | API key | [link](https://www.ovhcloud.com) | — | -| `perplexity` | `pplx` | Perplexity | API key | [link](https://www.perplexity.ai) | — | -| `phind` | `phind` | Phind | API key | [link](https://phind.com) | Get API key at phind.com | -| `piapi` | `pi` | PiAPI | API key, aggregator | [link](https://piapi.ai) | — | -| `poe` | `poe` | Poe | API key, aggregator | [link](https://creator.poe.com/api-reference) | Bearer API key for the Poe OpenAI-compatible API. | -| `pollinations` | `pol` | Pollinations AI | API key, video | [link](https://pollinations.ai) | No API key required for free public endpoint. Optional Spore tier: ~0.01 pollen/hour. | -| `predibase` | `predibase` | Predibase | API key | [link](https://predibase.com) | $25 free trial credits (30-day validity) | -| `publicai` | `publicai` | PublicAI | API key | [link](https://publicai.co) | Requires an API key — one-time signup credit, then paid | -| `puter` | `pu` | Puter AI | API key | [link](https://puter.com) | Get token at puter.com/dashboard → Copy Auth Token | -| `qianfan` | `qianfan` | Baidu Qianfan | API key | [link](https://cloud.baidu.com/product/wenxinworkshop) | — | -| `recraft` | `recraft` | Recraft | API key, image | [link](https://recraft.ai) | — | -| `reka` | `reka` | Reka | API key | [link](https://docs.reka.ai/chat/overview) | Use your Reka API key. OmniRoute supports the OpenAI-compatible base URL https://api.reka.ai/v1 and sends both Authorization and X-Api-Key headers for compatibility. | -| `runwayml` | `runway` | Runway | API key, video | [link](https://docs.dev.runwayml.com) | Use your Runway API key in Authorization: Bearer . OmniRoute targets the current Runway API at https://api.dev.runwayml.com/v1 and sends the required X-Runway-Version header automatically. | -| `sambanova` | `samba` | SambaNova | API key | [link](https://sambanova.ai) | $5 free credits on signup (30-day validity), no credit card required | -| `sap` | `sap` | SAP Generative AI Hub | API key, enterprise | [link](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/generative-ai-hub-in-sap-ai-core) | Use your SAP AI Core bearer token. Base URL can be your AI_API_URL root or a deploymentUrl from Generative AI Hub. | -| `scaleway` | `scw` | Scaleway AI | API key | [link](https://www.scaleway.com/en/ai/generative-apis) | 1M free tokens for new accounts — EU/GDPR compliant (Paris), Qwen3 235B & Llama 70B | -| `sensenova` | `sensenova` | SenseNova | API key | [link](https://platform.sensenova.cn) | Get API key at platform.sensenova.cn | -| `siliconflow` | `siliconflow` | SiliconFlow | API key | [link](https://cloud.siliconflow.com) | $1 free credits plus permanently free models after identity verification | -| `snowflake` | `snowflake` | Snowflake Cortex | API key, enterprise | [link](https://www.snowflake.com) | — | -| `sparkdesk` | `sparkdesk` | SparkDesk | API key | [link](https://xinghuo.xfyun.cn) | Get API key at console.xfyun.cn | -| `stability-ai` | `stability` | Stability AI | API key, image | [link](https://stability.ai) | — | -| `stepfun` | `stepfun` | StepFun | API key | [link](https://stepfun.com) | Get API key at platform.stepfun.com | -| `suno` | `suno` | Suno | API key | [link](https://suno.ai) | Paste session cookie from suno.ai (Clerk auth) | -| `synthetic` | `synthetic` | Synthetic | API key, aggregator | [link](https://synthetic.new) | — | -| `tencent` | `tencent` | Tencent Hunyuan | API key | [link](https://hunyuan.tencent.com) | Get API key at console.cloud.tencent.com | -| `thebai` | `thebai` | TheB.AI | API key, aggregator | [link](https://theb.ai) | Bearer API key for the TheB.AI OpenAI-compatible gateway. | -| `together` | `together` | Together AI | API key, video | [link](https://www.together.ai) | $25 signup credits + 3 permanently free models: Llama 3.3 70B, Vision, DeepSeek-R1 distill | -| `topaz` | `topaz` | Topaz | API key, image | [link](https://topazlabs.com) | — | -| `udio` | `udio` | Udio | API key | [link](https://udio.com) | Paste session cookie from udio.com (Supabase auth) | -| `uncloseai` | `unc` | UncloseAI | API key | [link](https://uncloseai.com) | No auth required. API accepts any non-empty string as key for identification. | -| `upstage` | `upstage` | Upstage | API key | [link](https://www.upstage.ai) | — | -| `v0-vercel` | `v0` | v0 (Vercel) | API key | [link](https://v0.dev) | — | -| `venice` | `venice` | Venice.ai | API key | [link](https://venice.ai) | — | -| `vercel-ai-gateway` | `vag` | Vercel AI Gateway | API key, aggregator | [link](https://vercel.com/docs/ai-gateway) | — | -| `vertex` | `vertex` | Vertex AI | API key, enterprise | [link](https://cloud.google.com/vertex-ai) | Provide Service Account JSON or OAuth access_token | -| `vertex-partner` | `vp` | Vertex AI Partners | API key, enterprise | [link](https://cloud.google.com/vertex-ai) | Provide the same Service Account JSON used for Vertex AI partner models. | -| `volcengine` | `volcengine` | Volcengine | API key | [link](https://www.volcengine.com) | — | -| `voyage-ai` | `voyage` | Voyage AI | API key, embed/rerank | [link](https://www.voyageai.com) | Bearer API key for Voyage AI embeddings and rerank APIs. | -| `wandb` | `wandb` | Weights & Biases Inference | API key | [link](https://wandb.ai) | — | -| `watsonx` | `watsonx` | IBM watsonx.ai Gateway | API key, enterprise | [link](https://www.ibm.com/products/watsonx-ai) | Use your watsonx bearer token. Base URL can be https://.ml.cloud.ibm.com/ml/gateway/v1/ or a self-managed /ml/gateway/v1 endpoint. | -| `xai` | `xai` | xAI (Grok) | API key | [link](https://x.ai) | — | -| `xiaomi-mimo` | `mimo` | Xiaomi MiMo | API key | [link](https://mimo.mi.com) | — | -| `yi` | `yi` | Yi (01.AI) | API key | [link](https://01.ai) | Get API key at platform.lingyiwanwu.com | -| `zai` | `zai` | Z.AI | API key | [link](https://open.bigmodel.cn) | — | -| `zenmux` | `zm` | ZenMux | API key | [link](https://zenmux.ai) | Use your ZenMux API key in Authorization: Bearer . ZenMux is fully OpenAI-compatible. Base URL: https://zenmux.ai/api/v1. | +| ID | Alias | Name | Tags | Website | Notes | +|----|-------|------|------|---------|-------| +| `360ai` | `360ai` | 360 AI | API key | [link](https://ai.360.cn) | Get API key at ai.360.cn | +| `agentrouter` | `agentrouter` | AgentRouter | API key, aggregator | [link](https://agentrouter.org) | $200 free credits on signup - multi-model routing gateway | +| `ai21` | `ai21` | AI21 Labs | API key | [link](https://www.ai21.com) | $10 trial credits on signup (valid 3 months), no credit card required | +| `aimlapi` | `aiml` | AI/ML API | API key, aggregator | [link](https://aimlapi.com) | Free tier paused (2026) — AI/ML API is now pay-as-you-go only (min $20 top-up); no recurring free credits. | +| `alibaba` | `ali` | Alibaba | API key | [link](https://dashscope-intl.aliyuncs.com) | — | +| `alibaba-cn` | `ali-cn` | Alibaba (China) | API key | [link](https://dashscope.aliyuncs.com) | — | +| `anthropic` | `anthropic` | Anthropic | API key | [link](https://platform.claude.com) | — | +| `api-airforce` | `af` | Api.airforce | API key | [link](https://api.airforce) | 55 free tier models including Grok-3, Claude 3.7, Qwen3, Kimi-K2, Gemini 2.5 Flash, DeepSeek-V3 | +| `arcee-ai` | `arcee` | Arcee AI | API key | [link](https://arcee.ai) | Get API key at arcee.ai | +| `azure-ai` | `azure-ai` | Azure AI Foundry | API key, enterprise | [link](https://learn.microsoft.com/azure/ai-foundry) | Use your Azure AI Foundry key. Base URL can be https://.services.ai.azure.com/openai/v1/ or https://.openai.azure.com/openai/v1/. | +| `azure-openai` | `azure` | Azure OpenAI | API key, enterprise | [link](https://azure.microsoft.com/products/ai-services/openai-service) | Use your Azure OpenAI API key. Base URL should be your resource endpoint, for example https://my-resource.openai.azure.com. | +| `baichuan` | `baichuan` | Baichuan | API key | [link](https://baichuan.com) | Get API key at platform.baichuan-ai.com | +| `baidu` | `baidu` | Baidu (ERNIE) | API key | [link](https://yiyan.baidu.com) | Get API key at console.bce.baidu.com | +| `bailian-coding-plan` | `bcp` | Alibaba Coding Plan | API key | [link](https://www.alibabacloud.com/help/en/model-studio/coding-plan) | — | +| `baseten` | `baseten` | Baseten | API key | [link](https://baseten.co) | $30 free trial credits for GPU inference | +| `bazaarlink` | `bzl` | BazaarLink | API key | [link](https://bazaarlink.ai) | Free tier with auto:free routing — zero-cost inference, no credit card required | +| `bedrock` | `bedrock` | Amazon Bedrock | API key, enterprise | [link](https://aws.amazon.com/bedrock) | Use your Amazon Bedrock API key and configure the AWS region where your models are enabled (for example eu-west-2). OmniRoute calls Bedrock's native Converse API directly. | +| `black-forest-labs` | `bfl` | Black Forest Labs | API key, image | [link](https://blackforestlabs.ai) | — | +| `blackbox` | `bb` | Blackbox AI | API key | [link](https://blackbox.ai) | Free tier: unlimited basic chat plus Minimax-M2.5, no credit card required | +| `bluesminds` | `bm` | BluesMinds | API key | [link](https://www.bluesminds.com) | Free daily pi credits — supports 200+ models including GPT-4o, GPT-4.1, Claude Sonnet 4.5, Gemini 2.0 Flash, DeepSeek V4, Qwen, Kimi K2 | +| `byteplus` | `bpm` | BytePlus ModelArk | API key | [link](https://console.byteplus.com/ark) | — | +| `bytez` | `bytez` | Bytez | API key | [link](https://bytez.com) | $1 free credits, refreshes every 4 weeks | +| `cablyai` | `cablyai` | CablyAI | API key, aggregator | [link](https://cablyai.com) | Bearer API key for the CablyAI OpenAI-compatible gateway. | +| `cerebras` | `cerebras` | Cerebras | API key | [link](https://inference.cerebras.ai) | Free Trial: 1M tokens/day, 30K TPM, 5 RPM — no credit card. | +| `chutes` | `chutes` | Chutes.ai | API key, aggregator | [link](https://chutes.ai) | Bearer API key for the Chutes OpenAI-compatible gateway. | +| `clarifai` | `clarifai` | Clarifai | API key, enterprise | [link](https://docs.clarifai.com) | Use your Clarifai PAT or app-specific API key. OmniRoute targets the OpenAI-compatible endpoint at https://api.clarifai.com/v2/ext/openai/v1 and authenticates with Authorization: Key . | +| `cloudflare-ai` | `cf` | Cloudflare Workers AI | API key | [link](https://developers.cloudflare.com/workers-ai) | Requires API Token AND Account ID (found at dash.cloudflare.com) | +| `codestral` | `codestral` | Codestral | API key | [link](https://mistral.ai) | — | +| `cohere` | `cohere` | Cohere | API key | [link](https://cohere.com) | Free Trial: 1,000 API calls/month for testing, no credit card required | +| `command-code` | `cmd` | Command Code | API key | [link](https://commandcode.ai/) | Use a Command Code API key. Requests are sent to Command Code's /alpha/generate endpoint. | +| `coze` | `coze` | Coze | API key | [link](https://coze.com) | Get API key at coze.com/open/api | +| `crof` | `crof` | CrofAI | API key | [link](https://crof.ai) | — | +| `databricks` | `databricks` | Databricks | API key, enterprise | [link](https://www.databricks.com) | — | +| `datarobot` | `datarobot` | DataRobot | API key, enterprise | [link](https://docs.datarobot.com) | Use your DataRobot API token. Optional Base URL can be the account root (for LLM Gateway) or a deployment URL under /api/v2/deployments/. | +| `deepinfra` | `deepinfra` | DeepInfra | API key | [link](https://deepinfra.com) | Free signup credits for API testing and model exploration | +| `deepseek` | `ds` | DeepSeek | API key | [link](https://platform.deepseek.com) | 5M free tokens on signup - no credit card required | +| `dify` | `dify` | Dify | API key | [link](https://dify.ai) | Get API key from your Dify instance. | +| `dit` | `dai` | DIT.ai | API key | [link](https://dit.ai) | Use your dit.ai API key in Authorization: Bearer . Fully OpenAI-compatible — a drop-in replacement, just change the base URL to https://api.dit.ai/v1. | +| `doubao` | `doubao` | Doubao | API key | [link](https://doubao.com) | Get API key at console.volcengine.com | +| `empower` | `empower` | Empower | API key, aggregator | [link](https://docs.empower.dev) | Bearer API key for the Empower OpenAI-compatible endpoint. | +| `fal-ai` | `fal` | Fal.ai | API key, image | [link](https://fal.ai) | — | +| `featherless-ai` | `featherless` | Featherless AI | API key | [link](https://featherless.ai) | Free tier available — no credit card required | +| `fenayai` | `fenayai` | FenayAI | API key, aggregator | [link](https://fenayai.com) | Bearer API key for the FenayAI OpenAI-compatible gateway. | +| `firecrawl` | `fc` | Firecrawl | API key | [link](https://firecrawl.dev) | — | +| `fireworks` | `fireworks` | Fireworks AI | API key | [link](https://fireworks.ai) | $1 free starter credits on signup for API testing | +| `freeaiapikey` | `faik` | FreeAIAPIKey | API key | [link](https://freeaiapikey.com) | — | +| `freemodel-dev` | `fmd` | FreeModel.dev | API key | [link](https://freemodel.dev) | $300 free credits on signup — no credit card required. Access GPT-5.4 and GPT-5.5 (OpenAI's latest flagship models) through an OpenAI-compatible API. | +| `friendliai` | `friendli` | FriendliAI | API key | [link](https://friendli.ai) | Free tier for serverless inference — no credit card required | +| `galadriel` | `galadriel` | Galadriel | API key | [link](https://galadriel.com) | ⚠️ **DEPRECATED.** api.galadriel.ai no longer resolves (sweep 2026-06-19); the inference API appears discontinued. | +| `gemini` | `gemini` | Gemini (Google AI Studio) | API key | [link](https://aistudio.google.com) | Free forever: 1,500 req/day for Gemini 2.5 Flash — no credit card, get key at aistudio.google.com | +| `getgoapi` | `ggo` | GoAPI | API key, aggregator | [link](https://api.getgoapi.com) | — | +| `gigachat` | `gigachat` | GigaChat (Sber) | API key | [link](https://developers.sber.ru) | — | +| `github-models` | `ghm` | GitHub Models | API key | [link](https://github.com/marketplace/models) | Create a GitHub PAT with 'models: read' scope at github.com/settings/tokens | +| `gitlab` | `gitlab` | GitLab Duo PAT | API key | [link](https://docs.gitlab.com/user/duo_agent_platform/code_suggestions/) | GitLab personal access token for the public Code Suggestions API. Configure a self-hosted base URL when not using gitlab.com. | +| `gitlawb` | `glb` | Gitlawb Opengateway (MiMo) | API key | [link](https://opengateway.gitlawb.com) | Free MiMo (xiaomi/mimo-v2.5) revoked 2026-05 — Opengateway is now a pay-as-you-go credit gateway; no recurring free model. | +| `gitlawb-gmi` | `glb-gmi` | Gitlawb Opengateway (GMI Cloud) | API key | [link](https://opengateway.gitlawb.com) | Free Nemotron promo ended 2026-06 — the GMI Cloud route is now pay-as-you-go credit only. | +| `glhf` | `glhf` | GLHF Chat | API key, aggregator | [link](https://glhf.chat) | ⚠️ **DEPRECATED.** glhf.chat shut down (2026); its api.laf.run gateway no longer serves the catalog (sweep 2026-06-19). | +| `glm` | `glm` | GLM Coding | API key | [link](https://z.ai/subscribe) | — | +| `glm-cn` | `glmcn` | GLM Coding (China) | API key | [link](https://open.bigmodel.cn) | — | +| `glmt` | `glmt` | GLM Thinking | API key | [link](https://open.bigmodel.cn) | — | +| `groq` | `groq` | Groq | API key | [link](https://groq.com) | Free tier: 30 RPM / 14.4K RPD — no credit card | +| `hackclub` | `hc` | Hackclub AI | API key, aggregator | [link](https://ai.hackclub.com) | Sign in with your Hack Club account at ai.hackclub.com. | +| `haiper` | `hp` | Haiper | API key, video | [link](https://haiper.ai) | Get API key at haiper.ai/haiper-api | +| `heroku` | `heroku` | Heroku AI | API key, enterprise | [link](https://www.heroku.com) | — | +| `huggingchat` | `huggingchat` | HuggingChat | API key | [link](https://huggingface.co/chat) | No API key required for basic access. | +| `huggingface` | `hf` | HuggingFace | API key | [link](https://huggingface.co) | Free Inference API for thousands of models (Whisper, VITS, SDXL…) | +| `hyperbolic` | `hyp` | Hyperbolic | API key | [link](https://hyperbolic.xyz) | $1-5 trial credits on signup for serverless inference | +| `ideogram` | `ideo` | Ideogram | API key | [link](https://ideogram.ai) | Get API key at ideogram.ai/docs/api | +| `iflytek` | `iflytek` | iFlytek Spark | API key | [link](https://xinghuo.xfyun.cn) | Get API key at console.xfyun.cn | +| `inclusionai` | `inclusion` | InclusionAI | API key | [link](https://inclusionai.com) | ⚠️ **DEPRECATED.** api.inclusionai.tech no longer resolves (sweep 2026-06-19); the inference API appears discontinued. | +| `inference-net` | `inet` | Inference.net | API key | [link](https://inference.net) | $25 free credits on signup plus research grants available | +| `jina-ai` | `jina` | Jina AI | API key, embed/rerank | [link](https://jina.ai) | Bearer API key for the Jina AI rerank API. | +| `jina-reader` | `jr` | Jina Reader | API key | [link](https://jina.ai/reader) | — | +| `kie` | `kie` | KIE.AI | API key | [link](https://kie.ai) | — | +| `kilo-gateway` | `kg` | Kilo Gateway | API key, aggregator | [link](https://kilo.ai) | — | +| `kimi` | `kimi` | Kimi | API key | [link](https://platform.moonshot.ai) | — | +| `kimi-coding-apikey` | `kmca` | Kimi Coding (API Key) | API key | [link](https://www.kimi.com/code) | — | +| `kluster` | `kluster` | Kluster AI | API key | [link](https://kluster.ai) | ⚠️ **DEPRECATED.** kluster.ai shut down (2026-06-09); api.kluster.ai no longer resolves (sweep 2026-06-19). Use another OpenAI-compatible provider. | +| `lambda-ai` | `lambda` | Lambda AI | API key | [link](https://lambda.ai) | — | +| `laozhang` | `lz` | LaoZhang AI | API key, aggregator | [link](https://api.laozhang.ai) | — | +| `leonardo` | `leo` | Leonardo AI | API key, video | [link](https://leonardo.ai) | Get API key at leonardo.ai/developer | +| `liquid` | `liquid` | Liquid AI | API key | [link](https://liquid.ai) | Get API key at liquid.ai | +| `llamagate` | `llamagate` | LlamaGate | API key | [link](https://llamagate.ai) | — | +| `llm7` | `llm7` | LLM7.io | API key | [link](https://llm7.io) | No signup required - 2 req/s, 20 RPM, 100 req/hr free tier | +| `longcat` | `lc` | LongCat AI | API key | [link](https://longcat.chat/platform/docs) | Free: 5M tokens/day on LongCat-2.0-Preview (Flash models retired 2026-05-29); up to 120M/day via feedback. | +| `maritalk` | `maritalk` | Maritalk | API key | [link](https://www.maritaca.ai) | — | +| `meta-llama` | `meta` | Meta Llama API | API key | [link](https://llama.developer.meta.com) | — | +| `minimax` | `minimax` | Minimax Coding | API key, video | [link](https://www.minimax.io) | — | +| `minimax-cn` | `minimax-cn` | Minimax (China) | API key | [link](https://www.minimaxi.com) | — | +| `mistral` | `mistral` | Mistral | API key | [link](https://mistral.ai) | Free Experiment tier: rate-limited access to all models, no credit card required | +| `modal` | `mdl` | Modal | API key, enterprise | [link](https://modal.com/docs) | Use the bearer token that protects your Modal deployment, if enabled. Base URL should point to your OpenAI-compatible Modal app, for example https://--.modal.run/v1. | +| `monsterapi` | `monster` | MonsterAPI | API key | [link](https://monsterapi.ai) | Get API key at monsterapi.ai | +| `moonshot` | `moonshot` | Moonshot AI | API key | [link](https://platform.moonshot.ai) | — | +| `morph` | `morph` | Morph | API key | [link](https://morphllm.com) | Free tier: 250K credits/month, $0 | +| `nanogpt` | `nanogpt` | NanoGPT | API key | [link](https://nano-gpt.com) | — | +| `nebius` | `nebius` | Nebius AI | API key | [link](https://nebius.com) | ~$1 trial credits on signup for API testing | +| `nlpcloud` | `nlpc` | NLP Cloud | API key | [link](https://docs.nlpcloud.com) | Use your NLP Cloud API key in Authorization: Token . OmniRoute targets the chatbot endpoint on https://api.nlpcloud.io/v1/gpu//chatbot by default. | +| `nomic` | `nomic` | Nomic | API key | [link](https://nomic.ai) | Get API key at atlas.nomic.ai | +| `nous-research` | `nous` | Nous Research | API key | [link](https://portal.nousresearch.com/help) | Use your Nous Portal API key. OmniRoute targets the official OpenAI-compatible inference endpoint at https://inference-api.nousresearch.com/v1. | +| `novita` | `novita` | Novita AI | API key, aggregator | [link](https://novita.ai) | $0.50 trial credits on signup (valid about 1 year) | +| `nscale` | `nscale` | nScale | API key | [link](https://nscale.com) | $5 free credits on signup for inference testing | +| `nvidia` | `nvidia` | NVIDIA NIM | API key | [link](https://build.nvidia.com) | Free dev access: ~40 RPM, 70+ models (Kimi K2.5, GLM 4.7, DeepSeek V3.2...) | +| `oci` | `oci` | OCI Generative AI | API key, enterprise | [link](https://www.oracle.com/artificial-intelligence/generative-ai) | Use your OCI Generative AI API key or IAM bearer token. Base URL can be https://inference.generativeai..oci.oraclecloud.com/openai/v1/. | +| `ollama-cloud` | `ollamacloud` | Ollama Cloud | API key | [link](https://ollama.com/settings/api-keys) | — | +| `openadapter` | `oad` | OpenAdapter | API key | [link](https://openadapter.dev) | Use your OpenAdapter API key in Authorization: Bearer sk-cv-. Fully OpenAI-compatible. API base URL: https://api.openadapter.in/v1. | +| `openai` | `openai` | OpenAI | API key | [link](https://platform.openai.com) | — | +| `opencode-go` | `opencode-go` | OpenCode Go | API key | [link](https://opencode.ai/go) | — | +| `opencode-zen` | `opencode-zen` | OpenCode Zen | API key | [link](https://opencode.ai/zen) | — | +| `openrouter` | `openrouter` | OpenRouter | API key, aggregator | [link](https://openrouter.ai) | Free models at $0/token with :free suffix - 20 RPM / 200 RPD | +| `orcarouter` | `orcarouter` | OrcaRouter | API key | [link](https://www.orcarouter.ai) | — | +| `ovhcloud` | `ovh` | OVHcloud AI | API key | [link](https://www.ovhcloud.com) | — | +| `perplexity` | `pplx` | Perplexity | API key | [link](https://www.perplexity.ai) | — | +| `phind` | `phind` | Phind | API key | [link](https://phind.com) | Get API key at phind.com | +| `piapi` | `pi` | PiAPI | API key, aggregator | [link](https://piapi.ai) | — | +| `poe` | `poe` | Poe | API key, aggregator | [link](https://creator.poe.com/api-reference) | Bearer API key for the Poe OpenAI-compatible API. | +| `pollinations` | `pol` | Pollinations AI | API key, video | [link](https://pollinations.ai) | Free keyless tier: openai, openai-fast, openai-large, qwen-coder, mistral, deepseek, grok, gemini-flash-lite-3.1, perplexity-fast, perplexity-reasoning. Premium models (claude, gemini, midijourney) require a Pollinations API key from enter.pollinations.ai. | +| `predibase` | `predibase` | Predibase | API key | [link](https://predibase.com) | ⚠️ **DEPRECATED.** serving.app.predibase.com no longer resolves (sweep 2026-06-19); the managed serving API appears discontinued. | +| `publicai` | `publicai` | PublicAI | API key | [link](https://publicai.co) | Requires an API key — one-time signup credit, then paid | +| `puter` | `pu` | Puter AI | API key | [link](https://puter.com) | Get token at puter.com/dashboard → Copy Auth Token | +| `qianfan` | `qianfan` | Baidu Qianfan | API key | [link](https://cloud.baidu.com/product/wenxinworkshop) | — | +| `recraft` | `recraft` | Recraft | API key, image | [link](https://recraft.ai) | — | +| `reka` | `reka` | Reka | API key | [link](https://docs.reka.ai/chat/overview) | Use your Reka API key. OmniRoute supports the OpenAI-compatible base URL https://api.reka.ai/v1 and sends both Authorization and X-Api-Key headers for compatibility. | +| `runwayml` | `runway` | Runway | API key, video | [link](https://docs.dev.runwayml.com) | Use your Runway API key in Authorization: Bearer . OmniRoute targets the current Runway API at https://api.dev.runwayml.com/v1 and sends the required X-Runway-Version header automatically. | +| `sambanova` | `samba` | SambaNova | API key | [link](https://sambanova.ai) | $5 free credits on signup (30-day validity), no credit card required | +| `sap` | `sap` | SAP Generative AI Hub | API key, enterprise | [link](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/generative-ai-hub-in-sap-ai-core) | Use your SAP AI Core bearer token. Base URL can be your AI_API_URL root or a deploymentUrl from Generative AI Hub. | +| `scaleway` | `scw` | Scaleway AI | API key | [link](https://www.scaleway.com/en/ai/generative-apis) | 1M free tokens for new accounts — EU/GDPR compliant (Paris), Qwen3 235B & Llama 70B | +| `sensenova` | `sensenova` | SenseNova | API key | [link](https://platform.sensenova.cn) | Get API key at platform.sensenova.cn | +| `siliconflow` | `siliconflow` | SiliconFlow | API key | [link](https://cloud.siliconflow.com) | $1 free credits plus permanently free models after identity verification | +| `snowflake` | `snowflake` | Snowflake Cortex | API key, enterprise | [link](https://www.snowflake.com) | — | +| `sparkdesk` | `sparkdesk` | SparkDesk | API key | [link](https://xinghuo.xfyun.cn) | Get API key at console.xfyun.cn | +| `stability-ai` | `stability` | Stability AI | API key, image | [link](https://stability.ai) | — | +| `stepfun` | `stepfun` | StepFun | API key | [link](https://stepfun.com) | Get API key at platform.stepfun.com | +| `suno` | `suno` | Suno | API key | [link](https://suno.ai) | Paste session cookie from suno.ai (Clerk auth) | +| `synthetic` | `synthetic` | Synthetic | API key, aggregator | [link](https://synthetic.new) | — | +| `tencent` | `tencent` | Tencent Hunyuan | API key | [link](https://hunyuan.tencent.com) | Get API key at console.cloud.tencent.com | +| `thebai` | `thebai` | TheB.AI | API key, aggregator | [link](https://theb.ai) | Bearer API key for the TheB.AI OpenAI-compatible gateway. | +| `together` | `together` | Together AI | API key, video | [link](https://www.together.ai) | $25 signup credits + 3 permanently free models: Llama 3.3 70B, Vision, DeepSeek-R1 distill | +| `tokenrouter` | `trk` | TokenRouter | API key | [link](https://tokenrouter.com) | Use your TokenRouter API key in Authorization: Bearer . Fully OpenAI-compatible. API base URL: https://api.tokenrouter.com/v1. | +| `topaz` | `topaz` | Topaz | API key, image | [link](https://topazlabs.com) | — | +| `udio` | `udio` | Udio | API key | [link](https://udio.com) | Paste session cookie from udio.com (Supabase auth) | +| `uncloseai` | `unc` | UncloseAI | API key | [link](https://uncloseai.com) | No auth required. API accepts any non-empty string as key for identification. | +| `upstage` | `upstage` | Upstage | API key | [link](https://www.upstage.ai) | — | +| `v0-vercel` | `v0` | v0 (Vercel) | API key | [link](https://v0.dev) | — | +| `venice` | `venice` | Venice.ai | API key | [link](https://venice.ai) | — | +| `vercel-ai-gateway` | `vag` | Vercel AI Gateway | API key, aggregator | [link](https://vercel.com/docs/ai-gateway) | — | +| `vertex` | `vertex` | Vertex AI | API key, enterprise | [link](https://cloud.google.com/vertex-ai) | Provide Service Account JSON or OAuth access_token | +| `vertex-partner` | `vp` | Vertex AI Partners | API key, enterprise | [link](https://cloud.google.com/vertex-ai) | Provide the same Service Account JSON used for Vertex AI partner models. | +| `volcengine` | `volcengine` | Volcengine | API key | [link](https://www.volcengine.com) | — | +| `voyage-ai` | `voyage` | Voyage AI | API key, embed/rerank | [link](https://www.voyageai.com) | Bearer API key for Voyage AI embeddings and rerank APIs. | +| `wafer` | `wafer` | Wafer AI | API key | [link](https://wafer.ai) | — | +| `wandb` | `wandb` | Weights & Biases Inference | API key | [link](https://wandb.ai) | — | +| `watsonx` | `watsonx` | IBM watsonx.ai Gateway | API key, enterprise | [link](https://www.ibm.com/products/watsonx-ai) | Use your watsonx bearer token. Base URL can be https://.ml.cloud.ibm.com/ml/gateway/v1/ or a self-managed /ml/gateway/v1 endpoint. | +| `xai` | `xai` | xAI (Grok) | API key | [link](https://x.ai) | — | +| `xiaomi-mimo` | `mimo` | Xiaomi MiMo | API key | [link](https://mimo.mi.com) | — | +| `yi` | `yi` | Yi (01.AI) | API key | [link](https://01.ai) | Get API key at platform.lingyiwanwu.com | +| `zai` | `zai` | Z.AI | API key | [link](https://open.bigmodel.cn) | — | +| `zenmux` | `zm` | ZenMux | API key | [link](https://zenmux.ai) | Use your ZenMux API key in Authorization: Bearer . ZenMux is fully OpenAI-compatible. Base URL: https://zenmux.ai/api/v1. | ## Local Providers (11) -| ID | Alias | Name | Tags | Website | Notes | -| --------------------- | ------------ | ------------------- | ------------------ | --------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `comfyui` | `comfyui` | ComfyUI | Local | [link](https://github.com/comfyanonymous/ComfyUI) | No API key required. Configure the local ComfyUI base URL (default: http://localhost:8188). | -| `docker-model-runner` | `dmr` | Docker Model Runner | Local, self-hosted | [link](https://docs.docker.com/ai/model-runner/) | API key optional. Configure the local Docker Model Runner OpenAI-compatible base URL (default: http://localhost:12434/v1). | -| `lemonade` | `lemonade` | Lemonade Server | Local, self-hosted | [link](https://lemonade-server.ai) | API key optional. Configure the local Lemonade OpenAI-compatible base URL (default: http://localhost:13305/api/v1). | -| `llama-cpp` | `llamacpp` | llama.cpp | Local, self-hosted | [link](https://github.com/ggml-org/llama.cpp) | API key optional (use any value, e.g. sk-no-key-required). Configure the llama-server OpenAI-compatible base URL (default: http://127.0.0.1:8080/v1). Note: if Llamafile is also installed, both default to port 8080 — run only one at a time or override the port. | -| `llamafile` | `llamafile` | Llamafile | Local, self-hosted | [link](https://github.com/Mozilla-Ocho/llamafile) | API key optional. Configure the local Llamafile OpenAI-compatible base URL (default: http://127.0.0.1:8080/v1). | -| `lm-studio` | `lmstudio` | LM Studio | Local, self-hosted | [link](https://lmstudio.ai) | API key optional. Configure the local LM Studio OpenAI-compatible base URL (default: http://localhost:1234/v1). | -| `oobabooga` | `ooba` | oobabooga | Local, self-hosted | [link](https://github.com/oobabooga/text-generation-webui) | API key optional. Configure the local oobabooga OpenAI-compatible base URL (default: http://localhost:5000/v1). | -| `sdwebui` | `sdwebui` | SD WebUI | Local | [link](https://github.com/AUTOMATIC1111/stable-diffusion-webui) | No API key required. Configure the local WebUI base URL (default: http://localhost:7860). | -| `triton` | `triton` | NVIDIA Triton | Local, self-hosted | [link](https://developer.nvidia.com/triton-inference-server) | API key optional. Configure the Triton OpenAI-compatible base URL (default: http://localhost:8000/v1). | -| `vllm` | `vllm` | vLLM | Local, self-hosted | [link](https://github.com/vllm-project/vllm) | API key optional. Configure the local vLLM OpenAI-compatible base URL (default: http://localhost:8000/v1). | -| `xinference` | `xinference` | XInference | Local, self-hosted | [link](https://inference.readthedocs.io) | API key optional. Configure the local XInference OpenAI-compatible base URL (default: http://localhost:9997/v1). | +| ID | Alias | Name | Tags | Website | Notes | +|----|-------|------|------|---------|-------| +| `comfyui` | `comfyui` | ComfyUI | Local | [link](https://github.com/comfyanonymous/ComfyUI) | No API key required. Configure the local ComfyUI base URL (default: http://localhost:8188). | +| `docker-model-runner` | `dmr` | Docker Model Runner | Local, self-hosted | [link](https://docs.docker.com/ai/model-runner/) | API key optional. Configure the local Docker Model Runner OpenAI-compatible base URL (default: http://localhost:12434/v1). | +| `lemonade` | `lemonade` | Lemonade Server | Local, self-hosted | [link](https://lemonade-server.ai) | API key optional. Configure the local Lemonade OpenAI-compatible base URL (default: http://localhost:13305/api/v1). | +| `llama-cpp` | `llamacpp` | llama.cpp | Local, self-hosted | [link](https://github.com/ggml-org/llama.cpp) | API key optional (use any value, e.g. sk-no-key-required). Configure the llama-server OpenAI-compatible base URL (default: http://127.0.0.1:8080/v1). Note: if Llamafile is also installed, both default to port 8080 — run only one at a time or override the port. | +| `llamafile` | `llamafile` | Llamafile | Local, self-hosted | [link](https://github.com/Mozilla-Ocho/llamafile) | API key optional. Configure the local Llamafile OpenAI-compatible base URL (default: http://127.0.0.1:8080/v1). | +| `lm-studio` | `lmstudio` | LM Studio | Local, self-hosted | [link](https://lmstudio.ai) | API key optional. Configure the local LM Studio OpenAI-compatible base URL (default: http://localhost:1234/v1). | +| `oobabooga` | `ooba` | oobabooga | Local, self-hosted | [link](https://github.com/oobabooga/text-generation-webui) | API key optional. Configure the local oobabooga OpenAI-compatible base URL (default: http://localhost:5000/v1). | +| `sdwebui` | `sdwebui` | SD WebUI | Local | [link](https://github.com/AUTOMATIC1111/stable-diffusion-webui) | No API key required. Configure the local WebUI base URL (default: http://localhost:7860). | +| `triton` | `triton` | NVIDIA Triton | Local, self-hosted | [link](https://developer.nvidia.com/triton-inference-server) | API key optional. Configure the Triton OpenAI-compatible base URL (default: http://localhost:8000/v1). | +| `vllm` | `vllm` | vLLM | Local, self-hosted | [link](https://github.com/vllm-project/vllm) | API key optional. Configure the local vLLM OpenAI-compatible base URL (default: http://localhost:8000/v1). | +| `xinference` | `xinference` | XInference | Local, self-hosted | [link](https://inference.readthedocs.io) | API key optional. Configure the local XInference OpenAI-compatible base URL (default: http://localhost:9997/v1). | ## Search Providers (11) -| ID | Alias | Name | Tags | Website | Notes | -| ------------------- | --------------- | -------------------------- | ------ | --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | -| `brave-search` | `brave-search` | Brave Search | Search | [link](https://brave.com/search/api) | Subscription token from Brave Search API dashboard | -| `exa-search` | `exa-search` | Exa Search | Search | [link](https://exa.ai) | API key from dashboard.exa.ai | -| `google-pse-search` | `google-pse` | Google Programmable Search | Search | [link](https://developers.google.com/custom-search/v1/overview) | Requires a Google API key and your Programmable Search Engine ID (cx) | -| `linkup-search` | `linkup` | Linkup Search | Search | [link](https://docs.linkup.so) | Bearer API key from the Linkup dashboard | -| `ollama-search` | `ollama-search` | Ollama Search | Search | [link](https://ollama.com/settings/api-keys) | Same API key as Ollama Cloud (from ollama.com/settings/api-keys) | -| `perplexity-search` | `pplx-search` | Perplexity Search | Search | [link](https://docs.perplexity.ai/guides/search-quickstart) | Same API key as Perplexity (pplx-...) | -| `searchapi-search` | `searchapi` | SearchAPI | Search | [link](https://www.searchapi.io/docs) | API key from SearchAPI (query param or Bearer auth) | -| `searxng-search` | `searxng` | SearXNG Search | Search | [link](https://docs.searxng.org) | API key is optional. Set your SearXNG base URL. Some instances may require a bearer token for access. | -| `serper-search` | `serper-search` | Serper Search | Search | [link](https://serper.dev) | API key from serper.dev dashboard | -| `tavily-search` | `tavily-search` | Tavily Search | Search | [link](https://tavily.com) | API key from app.tavily.com (format: tvly-...) | -| `youcom-search` | `youcom-search` | You.com Search | Search | [link](https://you.com/docs/search/overview) | X-API-Key from the You.com platform dashboard | +| ID | Alias | Name | Tags | Website | Notes | +|----|-------|------|------|---------|-------| +| `brave-search` | `brave-search` | Brave Search | Search | [link](https://brave.com/search/api) | Subscription token from Brave Search API dashboard | +| `exa-search` | `exa-search` | Exa Search | Search | [link](https://exa.ai) | API key from dashboard.exa.ai | +| `google-pse-search` | `google-pse` | Google Programmable Search | Search | [link](https://developers.google.com/custom-search/v1/overview) | Requires a Google API key and your Programmable Search Engine ID (cx) | +| `linkup-search` | `linkup` | Linkup Search | Search | [link](https://docs.linkup.so) | Bearer API key from the Linkup dashboard | +| `ollama-search` | `ollama-search` | Ollama Search | Search | [link](https://ollama.com/settings/api-keys) | Same API key as Ollama Cloud (from ollama.com/settings/api-keys) | +| `perplexity-search` | `pplx-search` | Perplexity Search | Search | [link](https://docs.perplexity.ai/guides/search-quickstart) | Same API key as Perplexity (pplx-...) | +| `searchapi-search` | `searchapi` | SearchAPI | Search | [link](https://www.searchapi.io/docs) | API key from SearchAPI (query param or Bearer auth) | +| `searxng-search` | `searxng` | SearXNG Search | Search | [link](https://docs.searxng.org) | API key is optional. Set your SearXNG base URL. Some instances may require a bearer token for access. | +| `serper-search` | `serper-search` | Serper Search | Search | [link](https://serper.dev) | API key from serper.dev dashboard | +| `tavily-search` | `tavily-search` | Tavily Search | Search | [link](https://tavily.com) | API key from app.tavily.com (format: tvly-...) | +| `youcom-search` | `youcom-search` | You.com Search | Search | [link](https://you.com/docs/search/overview) | X-API-Key from the You.com platform dashboard | ## Audio-only Providers (7) -| ID | Alias | Name | Tags | Website | Notes | -| ------------ | ---------- | ---------- | ----- | ------------------------------------- | ----------------------------------------------------------------------------------------------- | -| `assemblyai` | `aai` | AssemblyAI | Audio | [link](https://assemblyai.com) | — | -| `aws-polly` | `polly` | AWS Polly | Audio | [link](https://aws.amazon.com/polly/) | Use AWS Secret Access Key as API key; set providerSpecificData.accessKeyId and optional region. | -| `cartesia` | `cartesia` | Cartesia | Audio | [link](https://cartesia.ai) | — | -| `deepgram` | `dg` | Deepgram | Audio | [link](https://deepgram.com) | — | -| `elevenlabs` | `el` | ElevenLabs | Audio | [link](https://elevenlabs.io) | — | -| `inworld` | `inworld` | Inworld | Audio | [link](https://inworld.ai) | — | -| `playht` | `playht` | PlayHT | Audio | [link](https://play.ht) | — | +| ID | Alias | Name | Tags | Website | Notes | +|----|-------|------|------|---------|-------| +| `assemblyai` | `aai` | AssemblyAI | Audio | [link](https://assemblyai.com) | — | +| `aws-polly` | `polly` | AWS Polly | Audio | [link](https://aws.amazon.com/polly/) | Use AWS Secret Access Key as API key; set providerSpecificData.accessKeyId and optional region. | +| `cartesia` | `cartesia` | Cartesia | Audio | [link](https://cartesia.ai) | — | +| `deepgram` | `dg` | Deepgram | Audio | [link](https://deepgram.com) | — | +| `elevenlabs` | `el` | ElevenLabs | Audio | [link](https://elevenlabs.io) | — | +| `inworld` | `inworld` | Inworld | Audio | [link](https://inworld.ai) | — | +| `playht` | `playht` | PlayHT | Audio | [link](https://play.ht) | — | ## Upstream Proxy Providers (2) -| ID | Alias | Name | Tags | Website | Notes | -| ------------- | ----- | ----------- | -------------- | ---------------------------------------------------- | ----- | -| `9router` | `nr` | 9router | Upstream proxy | [link](https://www.npmjs.com/package/9router) | — | -| `cliproxyapi` | `cpa` | CLIProxyAPI | Upstream proxy | [link](https://github.com/router-for-me/CLIProxyAPI) | — | +| ID | Alias | Name | Tags | Website | Notes | +|----|-------|------|------|---------|-------| +| `9router` | `nr` | 9router | Upstream proxy | [link](https://www.npmjs.com/package/9router) | — | +| `cliproxyapi` | `cpa` | CLIProxyAPI | Upstream proxy | [link](https://github.com/router-for-me/CLIProxyAPI) | — | ## Cloud Agent Providers (3) -| ID | Alias | Name | Tags | Website | Notes | -| ------------- | ------------- | ------------ | ----------- | -------------------------------- | ----------------------------------------------------------- | -| `codex-cloud` | `codex-cloud` | Codex Cloud | Cloud agent | [link](https://openai.com/codex) | OpenAI API key with Codex Cloud task access. | -| `devin` | `devin` | Devin | Cloud agent | [link](https://devin.ai) | Devin API key for cloud agent sessions. | -| `jules` | `jules` | Google Jules | Cloud agent | [link](https://jules.google) | Jules API key for creating and managing cloud coding tasks. | +| ID | Alias | Name | Tags | Website | Notes | +|----|-------|------|------|---------|-------| +| `codex-cloud` | `codex-cloud` | Codex Cloud | Cloud agent | [link](https://openai.com/codex) | OpenAI API key with Codex Cloud task access. | +| `devin` | `devin` | Devin | Cloud agent | [link](https://devin.ai) | Devin API key for cloud agent sessions. | +| `jules` | `jules` | Google Jules | Cloud agent | [link](https://jules.google) | Jules API key for creating and managing cloud coding tasks. | ## System Providers (1) -| ID | Alias | Name | Tags | Website | Notes | -| ------ | ------ | ------------------ | ------ | ------- | ----- | -| `auto` | `auto` | Auto (Zero-Config) | System | — | — | +| ID | Alias | Name | Tags | Website | Notes | +|----|-------|------|------|---------|-------| +| `auto` | `auto` | Auto (Zero-Config) | System | — | — | ## Sources of truth diff --git a/docs/routing/AUTO-COMBO.md b/docs/routing/AUTO-COMBO.md index 94596e756c..ac5b6fe124 100644 --- a/docs/routing/AUTO-COMBO.md +++ b/docs/routing/AUTO-COMBO.md @@ -1,7 +1,7 @@ --- title: "OmniRoute Auto-Combo Engine" -version: 3.8.2 -lastUpdated: 2026-05-13 +version: 3.8.31 +lastUpdated: 2026-06-20 --- # OmniRoute Auto-Combo Engine @@ -26,6 +26,25 @@ lastUpdated: 2026-05-13 | `auto/smart` | smart | Quality-first + higher exploration rate (10%) for better model discovery | | `auto/lkgp` | lkgp | Explicit LKGP (same as default `auto`) | +### Category × Tier Composition (`auto/:`) + +OpenRouter-style suffixes separate **what kind of route** (category) from **how to optimize it** (tier), so you can compose them freely (#4235 Phase B, `open-sse/services/autoCombo/suffixComposition.ts`): + +- **Categories** (filter the candidate pool by capability): `coding` · `reasoning` · `vision` · `chat` · `multimodal`. `vision`/`multimodal` keep vision-capable models; `reasoning` keeps reasoning/thinking models. +- **Tiers** (pick the scoring weights / pool filter): `fast` (ship-fast) · `cheap` (alias `floor`, cost-saver) · `reliable` (circuit-breaker health + latency stability) · `free` / `pro` (filter the pool by model tier via `classifyTier` — free-tier vs. premium). + +| Example | Resolves to | +| ---------------------- | ----------------------------------------------------------------- | +| `auto/coding:fast` | coding pool, low-latency weights | +| `auto/coding:cheap` | coding pool, cost-optimized (alias `auto/coding:floor`) | +| `auto/reasoning:pro` | reasoning/thinking models only, premium tier | +| `auto/vision` | vision-capable models (no tier → balanced weights) | +| `auto/multimodal:free` | multimodal-capable models, free tier only | + +Any valid `auto/[:]` resolves on demand; a curated subset is advertised in `/v1/models` and the dashboard (`AUTO_SUFFIX_VARIANTS` in `open-sse/services/autoCombo/builtinCatalog.ts`). Filtering is **fail-open** — if a constraint matches no connected models, the full pool is used so routing never breaks. The core scorer (`combo.ts`) is unchanged; the category/tier filter is applied in `buildAutoCandidates`. + +> **Live model intelligence:** auto-routing fitness is informed by live **Arena ELO** rankings + **models.dev** tier data when the `ARENA_ELO_SYNC_ENABLED` flag is on (falls back to the static fitness map otherwise). + **How to use:** ```bash diff --git a/docs/security/GUARDRAILS.md b/docs/security/GUARDRAILS.md index 281f916133..d513d03edd 100644 --- a/docs/security/GUARDRAILS.md +++ b/docs/security/GUARDRAILS.md @@ -1,13 +1,13 @@ --- title: "Guardrails" -version: 3.8.2 -lastUpdated: 2026-05-13 +version: 3.8.31 +lastUpdated: 2026-06-20 --- # Guardrails > **Source of truth:** `src/lib/guardrails/` -> **Last updated:** 2026-05-13 — v3.8.0 +> **Last updated:** 2026-06-20 — v3.8.31 (injection-guard coverage + 16 KB scan bound + red-team) Guardrails enforce safety, policy, and content transformations at the boundary between OmniRoute and upstream providers. Each guardrail can inspect (and @@ -109,6 +109,14 @@ suspicious content detected" }`. In `warn`/`log` modes the guardrail logs but allows the call. The shared helper `evaluatePromptInjection()` is also exported for callers that need to evaluate prompts without going through the registry. +**Scan bound (v3.8.20):** the detector only inspects the **first 16 KB** of +joined prompt text — `MAX_INJECTION_SCAN_BYTES = 16 * 1024` (16 384 bytes) in +`src/shared/utils/inputSanitizer.ts`. Both `detectInjection()` and +`evaluatePromptInjection()` `slice(0, MAX_INJECTION_SCAN_BYTES)` before running +the pattern loop. Injection directives sit near the top of an input, so this +caps regex CPU/GC on multi-hundred-KB payloads without weakening detection (cf. +#3932, #4041). + ## Base Contract (`base.ts`) ```typescript @@ -294,3 +302,20 @@ A extração de texto (`extractMessageContents`) cobre `messages`/`input`/`promp o corpus OWASP-LLM em `INJECTION_GUARD_MODE=block`; garak roda probes (skip sem secret). `moderations` é incluída por consistência — operadores em block-mode podem isentá-la via `resolveDisabledGuardrails`. + +The nightly workflow (`.github/workflows/nightly-llm-security.yml`, cron + manual +dispatch) has two jobs: + +- **`promptfoo-guard` (blocking)** — runs `promptfoo eval -c promptfooconfig.yaml` + with `INJECTION_GUARD_MODE=block`. Each adversarial case (e.g. "ignore all + previous instructions…", DAN-style jailbreaks) asserts the response carries + `error.code === "SECURITY_001"`, i.e. the guard actually rejected the request. +- **`garak` (advisory)** — runs garak `--probes promptinject,dan,leakreplay` + against a local OmniRoute instance (`http://localhost:20128/v1`). Gated on a + provider secret (`PROMPTFOO_PROVIDER_KEY`); skips gracefully and is suffixed + `|| true`, so it reports without failing CI. + +Coverage of the guard helper (`createInjectionGuard` / `withInjectionGuard`) +spans every prompt-bearing `/v1` route; prompt text is pulled from +`messages`/`input`/`prompt`/`query`+`documents`/`instructions`/`system` by +`extractMessageContents()` in `src/shared/utils/inputSanitizer.ts`. diff --git a/docs/security/MITM-TPROXY-DECRYPT.md b/docs/security/MITM-TPROXY-DECRYPT.md new file mode 100644 index 0000000000..18e98d6f2c --- /dev/null +++ b/docs/security/MITM-TPROXY-DECRYPT.md @@ -0,0 +1,381 @@ +--- +title: "MITM TPROXY Transparent Decrypt" +version: 3.8.31 +lastUpdated: 2026-06-20 +--- + +# MITM TPROXY Transparent Decrypt + +TPROXY transparent decrypt is OmniRoute's **5th capture mode** for the +[Traffic Inspector](../frameworks/TRAFFIC_INSPECTOR.md) / [AgentBridge](../frameworks/AGENTBRIDGE.md) +MITM stack. It intercepts and **decrypts** local outbound HTTPS traffic on Linux +using kernel TPROXY + policy routing — **without** spoofing `/etc/hosts` and +**without** mutating OS-wide system-proxy settings. It is headless-friendly +(no DNS edits to clean up) and the firewall rules auto-flush on reboot. + +Unlike the other capture modes, TPROXY needs no per-host setup: it transparently +intercepts **arbitrary** destination hosts on a target port, terminates TLS with +a leaf certificate it issues on the fly per SNI hostname, captures the decrypted +exchange, and re-encrypts the request to the original destination. + +> **Linux-only, root-only, opt-in.** This mode requires Linux, a native addon +> built with a C toolchain, and the **CAP_NET_ADMIN** capability (typically root). It is gated +> behind the loopback-only AgentBridge API and disabled by default. A trusted +> MITM CA that can sign any host is a powerful capability — see [§6 Security](#6-security). + +**Source:** `src/mitm/tproxy/` +**API route:** `GET / POST / DELETE /api/tools/agent-bridge/tproxy` +**Dashboard toggle:** Traffic Inspector → capture-modes toolbar → **"TPROXY Decrypt"** ⚠ +**See also:** [`docs/frameworks/TRAFFIC_INSPECTOR.md`](../frameworks/TRAFFIC_INSPECTOR.md), +[`docs/frameworks/AGENTBRIDGE.md`](../frameworks/AGENTBRIDGE.md) + +--- + +## §1 What it is and when to use it + +The other four capture modes each have a limitation: + +| Mode | How traffic is steered | Limitation | +|------|------------------------|------------| +| AgentBridge | `/etc/hosts` DNS spoof of a fixed host set | only the registered IDE-agent hosts | +| Custom Hosts | `/etc/hosts` DNS spoof per host | one entry per host; sudo to edit hosts | +| HTTP_PROXY | `HTTP_PROXY`/`HTTPS_PROXY` env | only apps that honor the env var | +| System-wide proxy | OS proxy settings | mutates global state; needs revert | + +TPROXY transparent decrypt steers traffic at the **kernel** layer instead. It +marks new local outbound TCP connections to a target port (default `443`) in the +`mangle OUTPUT` chain, an `ip rule` reroutes the marked packets to local delivery, +and on re-entry the `mangle PREROUTING` `TPROXY` target hands them to an +**IP_TRANSPARENT** listener — which then terminates TLS and captures the plaintext. + +Use it when you want to capture and decrypt traffic from a process that: + +- talks to a host AgentBridge does not register, and +- does not honor `HTTP_PROXY`, and +- you do not want to disturb with a system-wide proxy change. + +Because interception happens in the kernel, the originating process needs **no +configuration change** — but the process must trust the dynamic CA OmniRoute +installs (see [§4](#4-the-per-sni-dynamic-ca-and-trust-store-installer)). + +--- + +## §2 Requirements + +| Requirement | Detail | +|-------------|--------| +| **OS** | Linux only — **IP_TRANSPARENT** is a Linux-only socket option. The loader returns "unavailable" on every other platform. | +| **Privilege** | The **CAP_NET_ADMIN** capability to create the transparent socket and apply `iptables`/`ip` rules — in practice, run as root. | +| **Native addon** | A tiny N-API addon (`src/mitm/tproxy/native/transparent.c`) must be built or shipped as a prebuild. See [§3](#3-the-native-ip_transparent-addon). | +| **Kernel modules** | `iptables` with the `TPROXY`, `mangle`, and `mark` match support (validated against kernel 6.8.0). | + +**Graceful degradation:** if any requirement is missing (non-Linux, no toolchain, +addon not built), the addon loader (`src/mitm/tproxy/transparentSocket.ts::loadTransparentAddon`) +returns `null` rather than throwing. The capture-mode status then reports +`available: false`, the dashboard toggle is **disabled** with the tooltip +"TPROXY decrypt requires Linux + root + the native addon", and the rest of +OmniRoute keeps working. + +--- + +## §3 The native IP_TRANSPARENT addon + +Node's `net` module cannot `setsockopt(IP_TRANSPARENT)` *before* `bind()`, which +TPROXY requires (otherwise the kernel drops the redirected packets). The addon +(`src/mitm/tproxy/native/transparent.c`, built via `binding.gyp`) is a small N-API +module exposing three functions, consumed through `transparentSocket.ts`: + +| Addon function | Socket work | Used for | +|----------------|-------------|----------| +| `createTransparentListener(ip, port)` | `socket()` + **SO_REUSEADDR** + **IP_TRANSPARENT** + `bind()` + `listen()`, returns the raw fd | the transparent capture listener (Node adopts the fd via `server.listen({ fd })`) | +| `setSocketMark(fd, mark)` | `setsockopt` **SO_MARK** on an existing fd | anti-loop (mark the proxy's own sockets) | +| `connectMarked(ip, port, mark)` | `socket()` + **SO_MARK** **before** a non-blocking `connect()`, returns fd | the re-encrypted upstream forward (the SYN carries the mark) | + +The original destination is read from `socket.localAddress`/`localPort` — TPROXY +preserves it, so there is no **SO_ORIGINAL_DST**/NAT lookup. + +### Building the addon + +```bash +npm run build:native:tproxy # cd src/mitm/tproxy/native && node-gyp rebuild + # -> native/build/Release/transparent.node +``` + +- During `npm run build`, `scripts/build/build-tproxy-native.mjs` runs `node-gyp + rebuild`. It is **Linux-only and non-fatal** — a missing toolchain just leaves + the capture mode unavailable. +- `assembleStandalone.mjs` copies `build/Release/transparent.node` into the + standalone bundle; `transparentSocket.ts` resolves it both module-relative and + cwd-relative (`/src/mitm/tproxy/native/...`). +- `build/` and `prebuilds/` are git-ignored — the binary is **built, never + committed**. + +The loader probes, in priority order: +`native/build/Release/transparent.node`, then `native/prebuilds/transparent.node` +(both module-relative and under `/src/mitm/tproxy/`). + +--- + +## §4 The per-SNI dynamic CA and trust-store installer + +The static AgentBridge MITM cert works only because AgentBridge DNS-spoofs a +**fixed** host set. TPROXY intercepts **arbitrary** hosts, so the listener must +present a valid leaf for whatever SNI the client requests. + +### Dynamic CA (`src/mitm/tproxy/dynamicCert.ts`) + +`DynamicCertStore` runs a local CA (built on the `selfsigned` dependency) that: + +- Generates a long-lived CA via `generateMitmCa()` (CN `"OmniRoute MITM CA"`, + 10-year validity, `basicConstraints CA=true` + `keyUsage keyCertSign,cRLSign`, + 2048-bit RSA / SHA-256). +- Issues a **leaf per SNI hostname on demand** via `issueLeafCert()` (1-year + validity, `subjectAltName` = the SNI host) and caches one `tls.SecureContext` + per hostname. +- Exposes `createSNICallback()` for the TLS-terminating server (see [§5](#5-how-decrypt-and-capture-work)). +- Can be constructed with an `existingCa` to keep the CA stable across restarts + (so the trust store does not need re-installing). + +The CA private key **never leaves the machine**. + +### Trust-store installer (`src/mitm/tproxy/caTrust.ts`) + +The intercepted client must trust the dynamic CA, so starting the capture mode +installs the CA cert into the OS trust store under a **dedicated slot** — +`omniroute-tproxy-ca.crt` (constant `TPROXY_CA_CERT_NAME`) — kept separate from +the static MITM cert's slot (`omniroute-mitm.crt`) so the two never clobber each +other. + +`installTproxyCa(caPem, sudoPassword?)` detects the distro's anchor directory +(in order: Debian-style first) and runs the matching refresh command: + +| Anchor directory | Refresh command | +|------------------|-----------------| +| `/usr/local/share/ca-certificates` | `update-ca-certificates` | +| `/etc/ca-certificates/trust-source/anchors` | `update-ca-trust` | +| `/etc/pki/ca-trust/source/anchors` | `update-ca-trust` | +| `/etc/pki/trust/anchors` | `update-ca-certificates` | + +Install stages the PEM to a temp file, then (privileged) `mkdir -p` the anchor +dir, `cp` the staged file into it, and runs the refresh command. `uninstallTproxyCa()` +removes the dedicated slot only (leaving the static MITM cert untouched) and +refreshes — a no-op on non-Linux. + +All privileged commands run via `execFileWithPassword` (`src/mitm/systemCommands.ts`) +— `spawn` with **arg arrays, no shell, no string interpolation** (Hard Rule #13). +When the process is root (e.g. the VPS) the target runs directly and no password +is needed; on a non-root desktop the `sudoPassword` is passed via `sudo -S` on stdin. + +> The desktop's `sudoPassword` is supplied in the POST body to authorize the +> trust-store install; it is ignored entirely when the process is root. + +--- + +## §5 How decrypt and capture work + +The pipeline (all under `src/mitm/tproxy/`): + +``` +local app ──TCP/443──▶ mangle OUTPUT marks the conn (fwmark) + ip rule → local route table → lo + mangle PREROUTING TPROXY → IP_TRANSPARENT listener (port 8443) + │ captureMode.ts: reads orig dest from socket.localAddress + ▼ + tlsCapture.ts: + 1. TLS-terminate the CLIENT with a per-SNI leaf (dynamicCert) + 2. internal http.Server parses the decrypted plaintext + 3. capture → globalTrafficBuffer.push() with source: "tproxy" + (sanitizeHeaders + maskSecret applied) + 4. forward RE-encrypted to the original destination + over a bypass-marked socket (connectMarked, anti-loop) + │ + ▼ + original upstream (api.example.com) +``` + +- **TLS termination** (`createTlsCaptureServer`): wraps the raw intercepted + socket in a server-side `tls.TLSSocket` using the dynamic CA's SNI callback, + then hands the decrypted stream to an internal `http.Server` (the standard MITM + termination trick). Socket lifetimes are bounded by `MITM_IDLE_TIMEOUT_MS` so a + hung tunnel cannot exhaust file descriptors. +- **Capture** (`handleDecryptedRequest`): pushes an `InterceptedRequest` with + `source: "tproxy"`, status starting `"in-flight"`, headers run through + `sanitizeHeaders()` and bodies through `maskSecret()` before they enter the + buffer. The entry is then updated with the response, sizes, and latency. +- **Re-encrypted forward** (`createForward` / `realForward`): re-encrypts to the + original destination. `rejectUnauthorized` defaults to **`true`** (secure by + default) — the upstream cert is verified against the SNI/Host the client + requested, so the proxy rejects exactly what the original client would. + +### Anti-loop (SO_MARK) + +Because the rules mark new local outbound connections, the proxy's **own** +re-encrypted forward would normally be re-intercepted — an infinite loop. The +forward path defends against this with a bypass socket mark (**SO_MARK**): + +- `realForward` opens its upstream socket via `connectMarked(ip, port, DEFAULT_BYPASS_MARK)` + — `DEFAULT_BYPASS_MARK = 0x539` — which sets the **SO_MARK** **before** `connect()`, + so the forward's SYN carries the bypass mark. +- The `mangle OUTPUT` rule excludes connections already carrying the bypass mark + (`-m mark ! --mark `), so the proxy's forward is **not** re-marked + and does not re-enter TPROXY. + +> Implementation note: the bypass-marked socket must be installed on the agent's +> `createConnection` (`https.request({ createConnection })` is silently ignored +> when an agent is present), or the forward would open an unmarked socket and the +> loop would return. This was the e2e-validated anti-loop fix. + +--- + +## §6 Security + +| Control | Detail | +|---------|--------| +| **Loopback-only API** | `/api/tools/agent-bridge/tproxy` is covered by the `/api/tools/agent-bridge/` prefix in `LOCAL_ONLY_API_PREFIXES` (`src/server/authz/routeGuard.ts`). Loopback enforcement runs **before** auth (Hard Rules #15 + #17) — a leaked JWT over a tunnel cannot start TPROXY capture, which applies `iptables` rules and installs a trust-store CA via child processes. | +| **Dedicated CA slot** | The dynamic CA installs to `omniroute-tproxy-ca.crt`, never clobbering the static MITM cert. | +| **CA key never leaves the host** | `DynamicCertStore` holds the CA key in memory; it is not exported. | +| **Secret masking** | `maskSecret()` on request/response bodies and `sanitizeHeaders()` on headers run **before** `globalTrafficBuffer.push()`. | +| **No shell interpolation** | All `iptables`/`ip`/trust-store commands run via `execFile`/`execFileWithPassword` with arg arrays (Hard Rule #13). | +| **Upstream cert verification** | The re-encrypted forward verifies the upstream cert by default (`rejectUnauthorized: true`). | +| **Error sanitization** | The route's error responses go through `sanitizeErrorMessage()` (Hard Rule #12). | + +**The MITM CA is a powerful capability.** A CA trusted by the OS that can sign any +host means anything OmniRoute intercepts can be decrypted. It is gated behind the +explicit, local-only TPROXY capture mode, off by default, and the trust-store +entry is removed when you stop the mode. + +--- + +## §7 Transactional firewall apply / revert + +A crash must never leave a `mangle` rule or stale route behind. The command builder +(`src/mitm/tproxy/commands.ts`) and runner (`src/mitm/tproxy/setup.ts`) guarantee +**revert is the exact inverse of apply, in reverse order**. + +`applyTproxy(cfg)` runs the apply commands in order; on **any** failure it runs a +best-effort full `revertTproxy(cfg)` and rethrows — so the firewall is either +fully applied or fully reverted, never half-applied. `revertTproxy(cfg)` runs the +inverse commands in reverse order and swallows failures (idempotent — safe to call +unconditionally, e.g. from the AgentBridge `repairMitm()` cleanup). + +`validateTproxyConfig(cfg)` runs before any command: ports must be `1–65535`, +`mark`/`routeTable`/`bypassMark` must be positive integers, and `bypassMark` must +differ from `mark` (anti-loop). + +### Apply commands (in order) + +```bash +ip rule add fwmark lookup +ip route add local 0.0.0.0/0 dev lo table +iptables -t mangle -A OUTPUT -p tcp --dport -m mark ! --mark -j MARK --set-mark +iptables -t mangle -A PREROUTING -p tcp --dport -m mark --mark -j TPROXY --on-port --tproxy-mark +``` + +Revert deletes them in reverse: `PREROUTING -D`, `OUTPUT -D`, `ip route del`, `ip rule del`. + +> The recipe is **OUTPUT-based** because the MITM use case is *local* outbound +> traffic (apps on the same host), which TPROXY in `PREROUTING` alone does not +> see — `PREROUTING` only sees forwarded traffic. The `OUTPUT` chain marks new +> local connections, the `ip rule` reroutes them to local delivery (`lo`), and +> `PREROUTING` then assigns them to the transparent listener. + +--- + +## §8 Configuration + +The start request (`POST /api/tools/agent-bridge/tproxy`) accepts the following +fields, validated by `StartTproxyBodySchema` (`tproxy/route.ts`). All are optional +and fall back to their defaults: + +| Field | Type | Default | Notes | +|-------|------|---------|-------| +| **dport** | int (1–65535) | `443` | Destination TCP port to transparently intercept | +| **mark** | int (≥1) | `0x2333` | Firewall mark set on `OUTPUT`, matched by the `ip rule` + `PREROUTING` | +| **onPort** | int (1–65535) | `8443` | Port the transparent (**IP_TRANSPARENT**) listener binds | +| **routeTable** | int (≥1) | `233` | Policy-routing table id holding the `local 0.0.0.0/0` route | +| **bypassMark** | int (≥1, ≠ `mark`) | `0x539` | The bypass socket mark (**SO_MARK**) the proxy sets on its own upstream conns; excluded in `OUTPUT` (anti-loop) | +| **sudoPassword** | string | — | Non-root desktops only: authorizes the trust-store install; ignored when root | + +There are **no environment variables** for TPROXY — all configuration is via the +POST body or the defaults above. + +--- + +## §9 Enabling from the Traffic Inspector + +1. Open the **Traffic Inspector** (`/dashboard/tools/traffic-inspector`). +2. In the capture-modes toolbar, find the **"TPROXY Decrypt"** ⚠ button + (`src/app/(dashboard)/dashboard/tools/traffic-inspector/components/CaptureModesToolbar.tsx`). + - If it is **disabled** with the tooltip "TPROXY decrypt requires Linux + root + + the native addon", the native addon is unavailable on this host (non-Linux, + no toolchain, or addon not built). See [§2](#2-requirements) and [§3](#3-the-native-ip_transparent-addon). +3. Click the button. It calls `POST /api/tools/agent-bridge/tproxy` via + `startTproxyCaptureMode()` (`src/lib/inspector/tproxyCaptureApi.ts`), which: + builds the dynamic CA, opens the transparent listener, applies the firewall + rules, and installs the CA in the OS trust store. +4. When running, the toggle turns amber and shows the live intercept count + (`· `). Intercepted requests appear in the request list with + `source: "tproxy"`. +5. Click again to stop — `DELETE /api/tools/agent-bridge/tproxy` via + `stopTproxyCaptureMode()` closes the listener, uninstalls the CA, and reverts + the firewall rules. + +The capture-mode status (running / available / intercept count / listener port) comes +from `GET /api/tools/agent-bridge/tproxy` (`getCaptureStatus()` in +`src/mitm/tproxy/captureManager.ts`). Only **one** TPROXY session runs at a time — +starting a second rejects with "TPROXY capture mode is already running". + +--- + +## §10 Troubleshooting + +### Toggle is disabled + +The native addon is not loadable. Confirm: you are on Linux, you built the addon +(`npm run build:native:tproxy`), and the process can load `transparent.node`. +`isTransparentSocketAvailable()` gates the toggle; `GET /api/tools/agent-bridge/tproxy` +returns `available: false` when the addon is missing. + +### Nothing is captured + +- Confirm the intercepted process actually connects to the configured `dport` + (default `443`). +- Confirm the process trusts the dynamic CA. The CA is installed under + `omniroute-tproxy-ca.crt`; apps with their own trust store (Firefox/Chrome NSS) + may need the cert added there too. +- Run the AgentBridge **Diagnose** self-test (see + [`AGENTBRIDGE.md`](../frameworks/AGENTBRIDGE.md)) for cert-trusted / server + health checks. + +### Stale firewall rules after a crash + +`revertTproxy()` is the exact inverse of apply and is idempotent. Stopping the +mode reverts the rules; if OmniRoute was killed mid-session, use the AgentBridge +**Repair** action (`POST /api/tools/agent-bridge/repair`) to undo orphaned system +state (DNS spoof, root CA, system proxy). The TPROXY `mangle` rules and route also +flush automatically on reboot. + +### Infinite loop / the proxy intercepts its own forward + +This is the anti-loop case. Confirm `bypassMark` differs from `mark` (validation +enforces this) and that the forward uses `connectMarked` (it does in `realForward`). +See [§5 Anti-loop](#anti-loop-so_mark). + +--- + +## §11 Source map + +| File | Responsibility | +|------|----------------| +| `src/mitm/tproxy/commands.ts` | Pure `iptables`/`ip` apply + revert command builder; `validateTproxyConfig` | +| `src/mitm/tproxy/setup.ts` | Transactional `applyTproxy` / `revertTproxy` runner (rollback on failure) | +| `src/mitm/tproxy/transparentSocket.ts` | Native-addon loader (`loadTransparentAddon`), `createTransparentListenerFd`, `connectMarked`, `setSocketMark`, `isTransparentSocketAvailable` | +| `src/mitm/tproxy/native/transparent.c` | N-API addon: `createTransparentListener` (IP_TRANSPARENT), `setSocketMark`, `connectMarked` | +| `src/mitm/tproxy/native/binding.gyp` | node-gyp build manifest | +| `src/mitm/tproxy/dynamicCert.ts` | `DynamicCertStore` — per-SNI dynamic CA + leaf cache | +| `src/mitm/tproxy/caTrust.ts` | OS trust-store install/uninstall (`installTproxyCa` / `uninstallTproxyCa`, dedicated slot) | +| `src/mitm/tproxy/tlsCapture.ts` | TLS-terminating decrypt engine + re-encrypted anti-loop forward | +| `src/mitm/tproxy/captureMode.ts` | Transparent-listener orchestration; reads orig dest from `socket.localAddress` | +| `src/mitm/tproxy/captureManager.ts` | Singleton lifecycle: `startCaptureMode` / `stopCaptureMode` / `getCaptureStatus` | +| `src/app/api/tools/agent-bridge/tproxy/route.ts` | `GET` / `POST` / `DELETE` route (LOCAL_ONLY) | +| `src/lib/inspector/tproxyCaptureApi.ts` | Client fetch helpers (`fetchTproxyStatus` / `startTproxyCaptureMode` / `stopTproxyCaptureMode`) | diff --git a/scripts/check/check-env-doc-sync.mjs b/scripts/check/check-env-doc-sync.mjs index 7173200c18..8cba7bc923 100644 --- a/scripts/check/check-env-doc-sync.mjs +++ b/scripts/check/check-env-doc-sync.mjs @@ -177,6 +177,10 @@ const DOC_ONLY_ALLOWLIST = new Set([ "IFLOW_OAUTH_CLIENT_SECRET", // Source-code constants accidentally captured by the doc regex. "CLI_COMPAT_OMITTED_PROVIDER_IDS", + // The stream-recovery tuning object in open-sse/config/constants.ts (`STREAM_RECOVERY.HOLDBACK_MS` + // etc.) — documented for reference; the real operator-facing env vars are STREAM_RECOVERY_ENABLED / + // STREAM_RECOVERY_MIDSTREAM_ENABLED (both in .env.example). The bare prefix is not an env var. + "STREAM_RECOVERY", // Sample default values that look like SHOUTY_NAMES (not env vars). "CHANGEME", // Legacy aliases — present in docs as "would be aliases" but read-only diff --git a/scripts/check/check-fabricated-docs.mjs b/scripts/check/check-fabricated-docs.mjs index 63a8be1638..7a4a867a07 100644 --- a/scripts/check/check-fabricated-docs.mjs +++ b/scripts/check/check-fabricated-docs.mjs @@ -108,6 +108,8 @@ const ENV_VAR_ALLOWLIST = new Set([ "CODEX_HOME", // Codex CLI's own config-home env var (CODEX-CLI-CONFIGURATION.md) "GEMINI_API_KEY", // Gemini CLI's own API-key env var, set by `omniroute setup-gemini` (REMOTE-MODE.md) "GOOGLE_GEMINI_BASE_URL", // Gemini CLI's own base-URL env var, set by `omniroute setup-gemini` (REMOTE-MODE.md) + "OPENAI_API_BASE", // legacy OpenAI base-URL env var some downstream tools (e.g. Aider) read (CLI-INTEGRATIONS.md) + "PROMPTFOO_PROVIDER_KEY", // promptfoo's own provider-key env var, used by the red-team suite (GUARDRAILS.md) "REDIS_PORT", // docker-compose host-port override (DOCKER_GUIDE.md) "AUTO_UPDATE_HOST_REPO_DIR", // docker-compose self-update mount (DOCKER_GUIDE.md) "LINUX_GPG_KEY", // electron AppImage signing key, CI/build only (ELECTRON_GUIDE.md)
🚫 Never hit limits
Auto-fallback across 227 providers in milliseconds. Quota out? Next provider takes over — zero downtime.		🚫 Never hit limits
Auto-fallback across 231 providers in milliseconds. Quota out? Next provider takes over — zero downtime.		 💸 Save up to 95% tokens
RTK + Caveman stacked compression cuts 15–95% of eligible tokens (~89% avg on tool-heavy sessions).		 🆓 $0 to start
50+ providers with a free tier, 11 free forever (Kiro, Qoder, Pollinations, LongCat…). No card needed.