mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-08-26 17:12:27 +03:00
* feat(admission): per-target lane-aware probes for combo/fusion fan-out (#9654 Wave 2)
Combo and fusion fan out N targets without ever consulting the adaptive-admission
layer: the parent request holds one lease, but each fan-out target is dispatched
unconditionally. With virtual lanes enabled (OMNIROUTE_CHAT_VIRTUAL_LANES=1), a
connection whose lane queue is full now SKIPS additional fan-out targets instead
of piling more queued work onto an already-congested session.
Adds PerTargetAdmissionHook (admission/types.ts) + createPerTargetAdmissionHook
factory (chatAdmission.ts): strictly non-blocking (maxWaitMs 0 - skip, never
queue), a no-op when virtual lanes are off, keyed to the parent tenantKey, and
release-on-admit so the probe is a capacity gate, not a hold.
Threaded through every parallel fan-out path:
- priority/weighted executeTarget + round-robin skip chains (combo.ts)
- fusion panel before fan-out (fusion.ts), judge fallback prefers survivors
- chaos parallel panel (autoCombo/chaosEngine.ts)
- tryFusionDispatch / tryRuntimeUnitDispatch / buildBaseOptions (dispatchPrelude.ts)
- chat.ts primary + safety-net redirect call sites
Snapshot exposes virtualLanes so the no-op gate is cheap and honest.
Tests: tests/unit/combo-lane-awareness-9654.test.ts (10 tests) - factory
semantics, priority/RR skip, fusion panel drop + all-skipped 503, no-hook
backward-compat baseline.
* feat(flags): activation UX - env-wins adaptive virtual-lanes flag + env docs (#9654 Wave 2)
U7: make adaptive virtual admission lanes discoverable + activatable.
- New OMNIROUTE_CHAT_VIRTUAL_LANES feature flag (boolean/runtime/requiresRestart) in featureFlagDefinitions + en.json i18n key.
- lib/admissionVirtualLanes.ts: env-wins resolver (env > DB > default) + boot warm folding a DB-sourced override into the process-global runtime env via reloadAdaptiveAdmissionRuntime(options.env) - no process.env mutation, no open-sse changes. Env still wins; DB toggle gates at next boot.
- GET /api/settings/feature-flags special-cases the flag to report the gate true source (ccDiscoveryAliases precedent); flagPayload helper dedupes the payload shape.
- Wire the warm into instrumentation-node registerNodejs (non-fatal, DB-ready).
- Document the master switch in .env.example + ENVIRONMENT.md with the system-1/system-2 distinction; zero new env-doc-sync drift.
- 11 new tests (resolver precedence + warm); 60/60 across feature-flag suites; typecheck core clean; ESLint + doc gates green.
* feat(mcp): surface adaptive admission lane data in omniroute_get_health (#9654 Wave 2)
U8: make adaptive virtual-lane admission visible to agents via the MCP health tool. handleGetHealth now surfaces a curated adaptiveAdmission block from the health payload (which already carried the runtime snapshot but was dropping it): virtualLanes/pressure/utilization/laneCount/laneQueuedCount/laneQueuedCost, laneTenants capped at top-10 by queued cost, admitted/rejected/wouldReject counts, shutdown. Block omitted entirely when the health endpoint reports none.
isLaneFlagOn mirrors the runtime 1|true convention so a string serialization can never invert a boolean lane report. getHealthOutput schema extended with the matching optional shape; tool description updated.
4 new dispatch tests (full block, top-10 cap/order, omission, defensive coercion of string flags + malformed lane entries) - 22/22 in essentialTools.test.ts. README: Adaptive Admission Lane Data table + Skills & Tool Navigability audit (29/43 schema entries covered, 14 undocumented, tool_search keyword runtime discovery, full catalog in docs/frameworks/MCP-SERVER.md).
No new lint errors (4 pre-existing in server.ts), typecheck core clean, doc counts + fabricated-docs gates green.
* docs: add changelog entry for #9654 Wave 2 (#10039)
* fix(codeql): suppress js/insufficient-password-hash false positive in lane-key fingerprinting (#10039)
resolveSessionId sha256-hashes bearer/x-api-key/x-goog-api-key to derive a deterministic, non-reversible per-key lane-bucket ID for virtual admission lanes (#9654). This is not password storage or verification, so the rule is a false positive; suppress it inline (same house style as src/lib/sync/tokens.ts) to clear the codeqlAlerts ratchet (2 > baseline 1) that blocks #10039 and every PR against release/v3.8.50.
* docs(mcp): complete MCP server README tool reference (#10039)
The MCP server README covered only 29 of the 43 schema entries, listing the
remaining tools solely as a gap note with omniroute_tool_search as the runtime
fallback. Add tool-reference tables for the agent-skills trio, oneproxy trio,
web_fetch/web_search, tool_search, create_combo, set_routing_strategy,
pick_fastest_model, sync_pricing, and db_health_check so the README covers the
full schemas catalog, and fold the coverage note into the tool_search discovery
paragraph.
* fix(chat): drop unused correlationId from safety-net combo redirect (#10039)
handleComboChat's HandleComboChatOptions has no correlationId member and
the combo pipeline never consumes it; the property was copied from the
handleSingleModelChat options shape by accident and introduced a new
TS2353 under the open-sse workspace typecheck gate.
* fix(i18n): translate featureFlagChatVirtualLanesEnabledDescription into 42 locales (#10039)
en.json gained the flag description in this PR but the locale catalogs
were never mirrored, failing the pt-BR key-parity (#6695) and vi
completeness gates. Adds a real translation to every locale, keeping the
zh-CN/zh-TW glossary canonical terms (提供者/儀表板) and no ICU drift.
* chore(quality): ratchet open-sse-typecheck baseline down (#10039)
The Wave 2 admission refactor removed 66 baselined open-sse type errors;
re-freeze the baseline so the gate pins the new, tighter state.
* docs: resync provider reference to 341 and CLI tools to 34
The release branch gained an 11th no-auth provider (freeaiapikey registry
resync, #10233) and a 26th CLI Code tool without regenerating the
auto-generated docs, leaving every PR against release/v3.8.50 failing the
Docs Gates strict validator (code 341 vs doc 340, CLI 34 vs "33 tools").
Regenerate docs/reference/PROVIDER_REFERENCE.md and sync the provider/tool
counts across README.md, AGENTS.md, llm.txt plus 42 i18n mirrors,
package.json description, and the four diagram SVGs.
* fix(tests): align count expectations with live catalogs (pre-existing release drift)
Release/v3.8.50 currently fails five gates on its own tree; this PR inherits
them. Fix the stale expectations to match live code:
- feature-flags-settings: 48 -> 49 flags (Wave 2 adds OMNIROUTE_CHAT_VIRTUAL_LANES)
- cli-tools-schema / cli-catalog-counts: 33 -> 34 tools (zcode added; 26 code = 21 visible + 5 none)
- optional-transformers-dependency: onnxruntime-node ~1.24.3 -> ~1.27.0 (bump #10382)
- stryker.conf.json: register chatcore-header-drop-warn-dedupe-10315 test
- check-public-creds: freeze zcodeProtocol clientId false positive (client identifier, not a credential)
* fix(tests): follow release's onnxruntime-node revert to ~1.24.3
release/v3.8.50's #10543 pinned onnxruntime-node back to ~1.24.3 after
#10403's ~1.27.0 bump caused npm to nest a second native copy under
@huggingface/transformers and broke the Docker SONAME contract. This
PR's own drift-alignment commit (57b9c033) predates that revert and
still expected ~1.27.0; the 3-way merge did not flag it as a textual
conflict since only one side touched this exact line, but the merged
tree became internally inconsistent (package.json ~1.24.3 vs test
expecting ~1.27.0). Align the test with the now-canonical release
value.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
* fix(quality): dedupe stryker.conf.json chatcore-header-drop-warn-dedupe entry
The 3-way merge applied both sides' insertion of the same test-file entry
at different positions, producing a duplicate with broken indentation.
Adopted release's clean version of the file.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: Brandon Bennett <brandonbennett@macbookair.myfiosgateway.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Co-authored-by: Brandon Bennett <branben@users.noreply.github.com>
690 lines
31 KiB
Markdown
690 lines
31 KiB
Markdown
# OmniRoute MCP Server
|
||
|
||
> **Model Context Protocol server** that exposes OmniRoute's gateway intelligence as **107 tools** for AI agents.
|
||
>
|
||
> **Source of truth for the full tool catalog and REST surface:** [`docs/frameworks/MCP-SERVER.md`](../../docs/frameworks/MCP-SERVER.md). This README focuses on architecture, configuration, and integration examples; the catalog below is a summary subset.
|
||
|
||
The MCP Server allows any AI agent (Claude Desktop, Cursor, VS Code Copilot, custom agents) to **monitor, control, and optimize** the OmniRoute AI gateway programmatically.
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────────────────┐
|
||
│ AI Agent / IDE │
|
||
│ (Claude Desktop, Cursor, VS Code, Custom) │
|
||
└──────────────────────┬───────────────────────────────────────────┘
|
||
│ MCP Protocol (stdio or HTTP)
|
||
▼
|
||
┌──────────────────────────────────────────────────────────────────┐
|
||
│ OmniRoute MCP Server │
|
||
│ ┌──────────────┐ ┌─────────────────┐ ┌────────────────────┐ │
|
||
│ │ Scope │ │ 107 MCP Tools │ │ Audit Logger │ │
|
||
│ │ Enforcement │──│ (core + memory │──│ (SHA-256/SQLite) │ │
|
||
│ │ │ │ + skills + …) │ │ │ │
|
||
│ └──────────────┘ └────────┬────────┘ └────────────────────┘ │
|
||
└─────────────────────────────┼────────────────────────────────────┘
|
||
│ HTTP (internal)
|
||
▼
|
||
┌──────────────────────────────────────────────────────────────────┐
|
||
│ OmniRoute Gateway (port 20128) │
|
||
│ /v1/chat/completions /api/combos /api/usage ... │
|
||
└──────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## Quick Start
|
||
|
||
### 1. Environment Variables
|
||
|
||
```bash
|
||
# Required: OmniRoute base URL
|
||
export OMNIROUTE_BASE_URL="http://localhost:20128"
|
||
|
||
# Optional: API key for authenticated access
|
||
export OMNIROUTE_API_KEY="your-api-key"
|
||
|
||
# Optional: Scope enforcement (default: disabled)
|
||
export OMNIROUTE_MCP_ENFORCE_SCOPES="true"
|
||
export OMNIROUTE_MCP_SCOPES="read:health,read:combos,read:quota,read:usage,read:models,read:cache,read:compression,read:tools,execute:completions,write:combos,write:budget,write:resilience,write:cache,write:compression"
|
||
```
|
||
|
||
### 2. stdio Transport (IDE Integration)
|
||
|
||
Add to your MCP client configuration:
|
||
|
||
**Claude Desktop** (`claude_desktop_config.json`):
|
||
|
||
```json
|
||
{
|
||
"mcpServers": {
|
||
"omniroute": {
|
||
"command": "node",
|
||
"args": ["path/to/omniroute/open-sse/mcp-server/server.ts"],
|
||
"env": {
|
||
"OMNIROUTE_BASE_URL": "http://localhost:20128",
|
||
"OMNIROUTE_API_KEY": "your-key"
|
||
}
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
**Cursor** (`.cursor/mcp.json`):
|
||
|
||
```json
|
||
{
|
||
"mcpServers": {
|
||
"omniroute": {
|
||
"command": "npx",
|
||
"args": ["tsx", "open-sse/mcp-server/server.ts"],
|
||
"env": {
|
||
"OMNIROUTE_BASE_URL": "http://localhost:20128"
|
||
}
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
**VS Code** (`.vscode/settings.json`):
|
||
|
||
```json
|
||
{
|
||
"mcp": {
|
||
"servers": {
|
||
"omniroute": {
|
||
"command": "npx",
|
||
"args": ["tsx", "open-sse/mcp-server/server.ts"],
|
||
"env": {
|
||
"OMNIROUTE_BASE_URL": "http://localhost:20128"
|
||
}
|
||
}
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
### 3. Start via CLI
|
||
|
||
```bash
|
||
# Direct start (stdio)
|
||
npx tsx open-sse/mcp-server/server.ts
|
||
|
||
# Or via OmniRoute CLI
|
||
omniroute --mcp
|
||
```
|
||
|
||
---
|
||
|
||
## Tool Reference
|
||
|
||
### Phase 1: Essential Tools (8)
|
||
|
||
| # | Tool | Scopes | Description |
|
||
| --- | ------------------------------- | --------------------- | --------------------------------------------------------------------------------------------------- |
|
||
| 1 | `omniroute_get_health` | `read:health` | Gateway health, uptime, memory, circuit breakers, rate limits, cache stats + adaptive lane pressure |
|
||
| 2 | `omniroute_list_combos` | `read:combos` | List all combos (model chains) with strategies and optional metrics |
|
||
| 3 | `omniroute_get_combo_metrics` | `read:combos` | Performance metrics for a specific combo |
|
||
| 4 | `omniroute_switch_combo` | `write:combos` | Activate or deactivate a combo for routing |
|
||
| 5 | `omniroute_check_quota` | `read:quota` | Remaining API quota per provider with token health status |
|
||
| 6 | `omniroute_route_request` | `execute:completions` | Send a chat completion through intelligent routing |
|
||
| 7 | `omniroute_cost_report` | `read:usage` | Cost report by period (session/day/week/month) with per-provider breakdown |
|
||
| 8 | `omniroute_list_models_catalog` | `read:models` | List all available models across providers with capabilities and pricing |
|
||
|
||
### Phase 2: Advanced Tools (8)
|
||
|
||
| # | Tool | Scopes | Description |
|
||
| --- | ---------------------------------- | ------------------------------------ | ---------------------------------------------------------------------------------------------- |
|
||
| 9 | `omniroute_simulate_route` | `read:health`, `read:combos` | Dry-run routing simulation showing fallback tree and estimated costs |
|
||
| 10 | `omniroute_set_budget_guard` | `write:budget` | Set session budget with action on exceed: `degrade`, `block`, or `alert` |
|
||
| 11 | `omniroute_set_resilience_profile` | `write:resilience` | Apply resilience profile: `aggressive`, `balanced`, or `conservative` |
|
||
| 12 | `omniroute_test_combo` | `execute:completions`, `read:combos` | Test each provider in a combo with a real prompt and a real upstream call, report latency/cost |
|
||
| 13 | `omniroute_get_provider_metrics` | `read:health` | Per-provider metrics with latency percentiles (p50/p95/p99), circuit breaker |
|
||
| 14 | `omniroute_best_combo_for_task` | `read:combos`, `read:health` | AI-powered combo recommendation by task type with budget/latency constraints |
|
||
| 15 | `omniroute_explain_route` | `read:health`, `read:usage` | Explain why a request was routed to a provider (scoring factors, fallbacks) |
|
||
| 16 | `omniroute_get_session_snapshot` | `read:usage` | Full session snapshot: cost, tokens, top models, errors, budget status |
|
||
|
||
### Cache and Compression Tools
|
||
|
||
| # | Tool | Scopes | Description |
|
||
| --- | ----------------------------------- | ------------------- | ---------------------------------------------------------------------------- |
|
||
| 21 | `omniroute_cache_stats` | `read:cache` | Semantic cache, prompt-cache, and idempotency statistics |
|
||
| 22 | `omniroute_cache_flush` | `write:cache` | Flush cache entries globally or by signature/model |
|
||
| 23 | `omniroute_compression_status` | `read:compression` | Compression settings, analytics summary, and provider-aware cache statistics |
|
||
| 24 | `omniroute_compression_configure` | `write:compression` | Configure compression mode and trigger thresholds at runtime |
|
||
| 25 | `omniroute_set_compression_engine` | `write:compression` | Set Caveman, RTK, or stacked compression mode and pipeline |
|
||
| 26 | `omniroute_list_compression_combos` | `read:compression` | List named compression combos and routing assignments |
|
||
| 27 | `omniroute_compression_combo_stats` | `read:compression` | Read analytics grouped by compression combo and engine |
|
||
| 28 | `omniroute_ccr_store` | `write:compression` | Store content in the caller-isolated in-memory CCR store |
|
||
| 29 | `omniroute_ccr_retrieve` | `read:compression` | Retrieve full or ranged caller-owned CCR content |
|
||
| 30 | `omniroute_ccr_inspect` | `read:compression` | Inspect CCR metadata without returning content |
|
||
| 31 | `omniroute_ccr_list` | `read:compression` | List paginated caller-owned CCR metadata |
|
||
| 32 | `omniroute_ccr_delete` | `write:compression` | Delete a caller-owned CCR block |
|
||
| 33 | `omniroute_ccr_stats` | `read:compression` | Report caller usage, bounded-store limits, and lifecycle counters |
|
||
|
||
CCR storage is bounded and in-memory only: 2 MiB per block, 16 MiB per principal, 64 MiB global,
|
||
with a 24-hour default TTL. Full MCP retrieval is capped at 256 KiB; larger blocks use ranged or
|
||
grep retrieval. All lifecycle operations are isolated by the authenticated caller principal.
|
||
|
||
MCP listable metadata descriptions are compressed at registration/list time when description
|
||
compression is enabled. `omniroute_compression_status` exposes those savings separately as
|
||
`analytics.mcpDescriptionCompression` with `source: "mcp_metadata_estimate"`, so clients do not
|
||
mistake metadata shrink estimates for provider token receipts.
|
||
|
||
### Discovery & Web Tools
|
||
|
||
| Tool | Scopes | Description |
|
||
| ----------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
|
||
| `omniroute_tool_search` | `read:tools` | Keyword search across the registered MCP tools; returns compact one-line signatures for token-efficient discovery |
|
||
| `omniroute_web_fetch` | `execute:search` | Fetch and extract a URL's content through the web-fetch gateway (Firecrawl, Jina Reader, Tavily, TinyFish) with automatic failover |
|
||
| `omniroute_web_search` | `execute:search` | Web search through the search gateway (Serper, Brave, Perplexity, Exa, Tavily, Google PSE, Linkup, SearchAPI, SearXNG) with failover |
|
||
|
||
### Skills & Catalog Tools
|
||
|
||
| Tool | Scopes | Description |
|
||
| --------------------------------- | -------------- | -------------------------------------------------------------------------------------------------------- |
|
||
| `omniroute_agent_skills_list` | `read:catalog` | List all 42 agent skills with optional `category` (`api`\|`cli`) and `area` filters; metadata + coverage |
|
||
| `omniroute_agent_skills_get` | `read:catalog` | Full metadata + SKILL.md content for a single skill by canonical `id` |
|
||
| `omniroute_agent_skills_coverage` | `read:catalog` | Coverage stats: how many of the 22 API and 20 CLI skills have SKILL.md files on disk vs catalog totals |
|
||
|
||
### Proxy, Pricing & Data Tools
|
||
|
||
| Tool | Scopes | Description |
|
||
| --------------------------- | --------------------------------- | ------------------------------------------------------------------------------------------ |
|
||
| `omniroute_oneproxy_fetch` | `read:proxies` | Fetch free proxies from the 1proxy marketplace (protocol/country/quality/limit filters) |
|
||
| `omniroute_oneproxy_rotate` | `read:proxies` | Get the next available proxy by strategy (`random` / `quality` / `sequential`) |
|
||
| `omniroute_oneproxy_stats` | `read:proxies` | Pool stats, sync status, distribution by protocol and country |
|
||
| `omniroute_sync_pricing` | `pricing:write` | Sync pricing from external sources (LiteLLM) without overwriting user-set prices; `dryRun` |
|
||
| `omniroute_db_health_check` | `read:health`, `write:resilience` | Diagnose (and optionally auto-repair) database drift — broken combo refs, orphan rows |
|
||
|
||
### Combo & Routing Tools
|
||
|
||
| Tool | Scopes | Description |
|
||
| -------------------------------- | ------------------------------------------ | -------------------------------------------------------------------------------------------- |
|
||
| `omniroute_create_combo` | `write:combos` | Register a new combo (model chain) with name, ordered model list, and optional strategy |
|
||
| `omniroute_set_routing_strategy` | `write:combos` | Update combo routing strategy at runtime (`priority` / `weighted` / `auto` / etc.) |
|
||
| `omniroute_pick_fastest_model` | `read:combos`, `read:health`, `read:usage` | Pick the fastest reliable provider-model pair from live telemetry; can apply latency routing |
|
||
|
||
---
|
||
|
||
### Adaptive Admission Lane Data
|
||
|
||
`omniroute_get_health` includes an `adaptiveAdmission` block whenever the gateway's adaptive
|
||
virtual-lane admission is active. It is a curated subset of the live admission snapshot:
|
||
|
||
| Field | Meaning |
|
||
| ------------------ | ---------------------------------------------------------------------- |
|
||
| `virtualLanes` | Whether per-tenant virtual-lane admission is enabled |
|
||
| `pressure` | Current pressure state (e.g. `healthy`, `high`, `critical`) |
|
||
| `utilization` | Current capacity utilization (0.0–1.0) |
|
||
| `laneCount` | Number of live lanes |
|
||
| `laneQueuedCount` | Total requests queued across lanes |
|
||
| `laneQueuedCost` | Total estimated cost queued across lanes |
|
||
| `laneTenants` | Top 10 lanes by queued cost (`tenantKey`, `queuedCount`, `queuedCost`) |
|
||
| `admittedCount` | Requests admitted since boot |
|
||
| `rejectedCount` | Requests rejected since boot |
|
||
| `wouldRejectCount` | Requests that would be rejected under the current limit |
|
||
| `shutdown` | Whether the admission runtime is shutting down |
|
||
|
||
`tenantKey` is an opaque per-API-key derived identifier, never the raw key. The block is omitted
|
||
entirely when the health endpoint reports no adaptive-admission data.
|
||
|
||
### Skills & Tool Navigability
|
||
|
||
The tables above cover the full `schemas/` catalog (43 entries); the authoritative reference with
|
||
scope-enforcement and transport details lives in
|
||
[`docs/frameworks/MCP-SERVER.md`](../../docs/frameworks/MCP-SERVER.md).
|
||
|
||
Agents never need to read this file to find a capability: `omniroute_tool_search` performs keyword
|
||
search across the registered tool set and returns compact one-line signatures (token-efficient
|
||
discovery), so newly added capabilities stay discoverable at runtime.
|
||
|
||
---
|
||
|
||
## Client Examples
|
||
|
||
### Python — Full Agent Workflow
|
||
|
||
```python
|
||
"""
|
||
OmniRoute MCP Client — Python example using the mcp SDK.
|
||
Install: pip install mcp
|
||
"""
|
||
import asyncio
|
||
from mcp import ClientSession, StdioServerParameters
|
||
from mcp.client.stdio import stdio_client
|
||
|
||
async def main():
|
||
server = StdioServerParameters(
|
||
command="npx",
|
||
args=["tsx", "open-sse/mcp-server/server.ts"],
|
||
env={
|
||
"OMNIROUTE_BASE_URL": "http://localhost:20128",
|
||
"OMNIROUTE_API_KEY": "your-key",
|
||
},
|
||
)
|
||
|
||
async with stdio_client(server) as (read, write):
|
||
async with ClientSession(read, write) as session:
|
||
await session.initialize()
|
||
|
||
# 1. Check gateway health
|
||
health = await session.call_tool("omniroute_get_health", {})
|
||
print("Health:", health.content[0].text)
|
||
|
||
# 2. List available combos with metrics
|
||
combos = await session.call_tool("omniroute_list_combos", {
|
||
"includeMetrics": True
|
||
})
|
||
print("Combos:", combos.content[0].text)
|
||
|
||
# 3. Find the best combo for a coding task
|
||
best = await session.call_tool("omniroute_best_combo_for_task", {
|
||
"taskType": "coding",
|
||
"budgetConstraint": 0.50,
|
||
"latencyConstraint": 5000,
|
||
})
|
||
print("Best combo:", best.content[0].text)
|
||
|
||
# 4. Set a session budget guard
|
||
budget = await session.call_tool("omniroute_set_budget_guard", {
|
||
"maxCost": 1.00,
|
||
"action": "degrade",
|
||
"degradeToTier": "cheap",
|
||
})
|
||
print("Budget guard:", budget.content[0].text)
|
||
|
||
# 5. Route a request through intelligent pipeline
|
||
response = await session.call_tool("omniroute_route_request", {
|
||
"model": "claude-sonnet-4",
|
||
"messages": [
|
||
{"role": "user", "content": "Write a Python hello world"}
|
||
],
|
||
"role": "coding",
|
||
})
|
||
print("Response:", response.content[0].text)
|
||
|
||
# 6. Get the session snapshot
|
||
snapshot = await session.call_tool("omniroute_get_session_snapshot", {})
|
||
print("Session:", snapshot.content[0].text)
|
||
|
||
asyncio.run(main())
|
||
```
|
||
|
||
### TypeScript — Programmatic Agent
|
||
|
||
```typescript
|
||
import { Client } from "@modelcontextprotocol/sdk/client/index.js";
|
||
import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js";
|
||
|
||
async function main() {
|
||
const transport = new StdioClientTransport({
|
||
command: "npx",
|
||
args: ["tsx", "open-sse/mcp-server/server.ts"],
|
||
env: {
|
||
OMNIROUTE_BASE_URL: "http://localhost:20128",
|
||
OMNIROUTE_API_KEY: "your-key",
|
||
},
|
||
});
|
||
|
||
const client = new Client({ name: "my-agent", version: "1.0.0" });
|
||
await client.connect(transport);
|
||
|
||
// Check quota before deciding which model to use
|
||
const quota = await client.callTool({
|
||
name: "omniroute_check_quota",
|
||
arguments: { provider: "claude" },
|
||
});
|
||
console.log("Claude quota:", quota.content);
|
||
|
||
// Simulate the route before actually calling
|
||
const simulation = await client.callTool({
|
||
name: "omniroute_simulate_route",
|
||
arguments: {
|
||
model: "claude-sonnet-4",
|
||
promptTokenEstimate: 2000,
|
||
},
|
||
});
|
||
console.log("Route simulation:", simulation.content);
|
||
|
||
// Send the actual request
|
||
const result = await client.callTool({
|
||
name: "omniroute_route_request",
|
||
arguments: {
|
||
model: "claude-sonnet-4",
|
||
messages: [{ role: "user", content: "Explain async/await" }],
|
||
},
|
||
});
|
||
console.log("Result:", result.content);
|
||
|
||
// Cost report
|
||
const costs = await client.callTool({
|
||
name: "omniroute_cost_report",
|
||
arguments: { period: "session" },
|
||
});
|
||
console.log("Costs:", costs.content);
|
||
|
||
await client.close();
|
||
}
|
||
|
||
main();
|
||
```
|
||
|
||
### Go — HTTP Client
|
||
|
||
```go
|
||
package main
|
||
|
||
import (
|
||
"bytes"
|
||
"encoding/json"
|
||
"fmt"
|
||
"io"
|
||
"net/http"
|
||
)
|
||
|
||
// Simplified direct-API approach (bypass MCP, hit OmniRoute APIs directly)
|
||
// Useful if you don't need MCP protocol framing.
|
||
|
||
func callTool(baseURL, tool string, args map[string]any) (string, error) {
|
||
// MCP tools map to OmniRoute APIs:
|
||
endpoints := map[string]string{
|
||
"health": "/api/monitoring/health",
|
||
"combos": "/api/combos",
|
||
"quota": "/api/usage/quota",
|
||
"models": "/v1/models",
|
||
}
|
||
|
||
url := baseURL + endpoints[tool]
|
||
resp, err := http.Get(url)
|
||
if err != nil {
|
||
return "", err
|
||
}
|
||
defer resp.Body.Close()
|
||
body, _ := io.ReadAll(resp.Body)
|
||
return string(body), nil
|
||
}
|
||
|
||
func routeRequest(baseURL, model, prompt string) (string, error) {
|
||
payload := map[string]any{
|
||
"model": model,
|
||
"messages": []map[string]string{
|
||
{"role": "user", "content": prompt},
|
||
},
|
||
"stream": false,
|
||
}
|
||
data, _ := json.Marshal(payload)
|
||
|
||
resp, err := http.Post(
|
||
baseURL+"/v1/chat/completions",
|
||
"application/json",
|
||
bytes.NewReader(data),
|
||
)
|
||
if err != nil {
|
||
return "", err
|
||
}
|
||
defer resp.Body.Close()
|
||
body, _ := io.ReadAll(resp.Body)
|
||
return string(body), nil
|
||
}
|
||
|
||
func main() {
|
||
base := "http://localhost:20128"
|
||
|
||
health, _ := callTool(base, "health", nil)
|
||
fmt.Println("Health:", health)
|
||
|
||
result, _ := routeRequest(base, "auto", "Hello from Go!")
|
||
fmt.Println("Result:", result)
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## Use Cases
|
||
|
||
### 🔄 Use Case 1: Auto-Healing Agent
|
||
|
||
An agent that monitors OmniRoute health and auto-switches combos when providers degrade.
|
||
|
||
```python
|
||
async def auto_healing_loop(session):
|
||
"""Monitor health and react to provider issues."""
|
||
while True:
|
||
# Check health
|
||
health = await session.call_tool("omniroute_get_health", {})
|
||
data = json.loads(health.content[0].text)
|
||
|
||
# Find providers with open circuit breakers
|
||
broken = [
|
||
cb for cb in data["circuitBreakers"]
|
||
if cb["state"] == "OPEN"
|
||
]
|
||
|
||
if broken:
|
||
# Switch to a different resilience profile
|
||
await session.call_tool("omniroute_set_resilience_profile", {
|
||
"profile": "conservative"
|
||
})
|
||
|
||
# Find best alternative combo
|
||
best = await session.call_tool("omniroute_best_combo_for_task", {
|
||
"taskType": "coding"
|
||
})
|
||
best_data = json.loads(best.content[0].text)
|
||
combo_id = best_data["recommendedCombo"]["id"]
|
||
|
||
# Activate it
|
||
await session.call_tool("omniroute_switch_combo", {
|
||
"comboId": combo_id, "active": True
|
||
})
|
||
print(f"⚠️ Auto-healed: switched to {combo_id}")
|
||
|
||
await asyncio.sleep(30) # Check every 30 seconds
|
||
```
|
||
|
||
### 💰 Use Case 2: Budget-Aware Coding Agent
|
||
|
||
An agent that monitors costs in real-time and degrades to cheaper models when nearing budget.
|
||
|
||
```python
|
||
async def budget_aware_coding(session, task: str, max_budget: float):
|
||
"""Complete a coding task within a budget."""
|
||
# Set budget guard
|
||
await session.call_tool("omniroute_set_budget_guard", {
|
||
"maxCost": max_budget,
|
||
"action": "degrade",
|
||
"degradeToTier": "cheap",
|
||
})
|
||
|
||
# Simulate first to estimate cost
|
||
sim = await session.call_tool("omniroute_simulate_route", {
|
||
"model": "claude-sonnet-4",
|
||
"promptTokenEstimate": len(task.split()) * 2,
|
||
})
|
||
sim_data = json.loads(sim.content[0].text)
|
||
estimated_cost = sim_data["fallbackTree"]["bestCaseCost"]
|
||
print(f"Estimated cost: ${estimated_cost:.4f}")
|
||
|
||
# Send request
|
||
result = await session.call_tool("omniroute_route_request", {
|
||
"model": "claude-sonnet-4",
|
||
"messages": [{"role": "user", "content": task}],
|
||
"role": "coding",
|
||
})
|
||
|
||
# Check remaining budget
|
||
snapshot = await session.call_tool("omniroute_get_session_snapshot", {})
|
||
snap_data = json.loads(snapshot.content[0].text)
|
||
print(f"Session cost: ${snap_data['costTotal']:.4f}")
|
||
if snap_data.get("budgetGuard"):
|
||
print(f"Budget remaining: ${snap_data['budgetGuard']['remaining']:.4f}")
|
||
|
||
return json.loads(result.content[0].text)["response"]["content"]
|
||
```
|
||
|
||
### 🧪 Use Case 3: Combo Benchmarking Agent
|
||
|
||
An agent that periodically benchmarks all combos and reports the fastest/cheapest.
|
||
|
||
```python
|
||
async def benchmark_combos(session):
|
||
"""Benchmark all enabled combos and rank them."""
|
||
combos = await session.call_tool("omniroute_list_combos", {
|
||
"includeMetrics": True,
|
||
})
|
||
combo_list = json.loads(combos.content[0].text)["combos"]
|
||
|
||
results = []
|
||
for combo in combo_list:
|
||
if not combo["enabled"]:
|
||
continue
|
||
|
||
test = await session.call_tool("omniroute_test_combo", {
|
||
"comboId": combo["id"],
|
||
"testPrompt": "Return the number 42.",
|
||
})
|
||
test_data = json.loads(test.content[0].text)
|
||
results.append({
|
||
"combo": combo["name"],
|
||
"fastest": test_data["summary"]["fastestProvider"],
|
||
"cheapest": test_data["summary"]["cheapestProvider"],
|
||
"success_rate": f'{test_data["summary"]["successful"]}/{test_data["summary"]["totalProviders"]}',
|
||
})
|
||
|
||
print("📊 Combo Benchmark Results:")
|
||
for r in results:
|
||
print(f" {r['combo']}: fastest={r['fastest']}, cheapest={r['cheapest']}, success={r['success_rate']}")
|
||
```
|
||
|
||
### 🔍 Use Case 4: Post-Mortem Debugging Agent
|
||
|
||
An agent that explains why a request was routed to a specific provider.
|
||
|
||
```typescript
|
||
async function debugRouting(client: Client, requestId: string) {
|
||
// Explain the routing decision
|
||
const explanation = await client.callTool({
|
||
name: "omniroute_explain_route",
|
||
arguments: { requestId },
|
||
});
|
||
const data = JSON.parse(explanation.content[0].text);
|
||
|
||
console.log(`Request ${requestId}:`);
|
||
console.log(` Provider: ${data.decision.providerSelected}`);
|
||
console.log(` Model: ${data.decision.modelUsed}`);
|
||
console.log(` Score: ${data.decision.score}`);
|
||
console.log(` Factors:`);
|
||
for (const factor of data.decision.factors) {
|
||
console.log(` ${factor.name}: ${factor.value} (weight: ${factor.weight})`);
|
||
}
|
||
if (data.decision.fallbacksTriggered.length > 0) {
|
||
console.log(` Fallbacks triggered:`);
|
||
for (const fb of data.decision.fallbacksTriggered) {
|
||
console.log(` ${fb.provider}: ${fb.reason}`);
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
### 📋 Use Case 5: Model Discovery Agent
|
||
|
||
An agent that discovers the cheapest models for a given capability.
|
||
|
||
```python
|
||
async def find_cheapest_models(session, capability="chat"):
|
||
"""Find the cheapest available models for a capability."""
|
||
catalog = await session.call_tool("omniroute_list_models_catalog", {
|
||
"capability": capability,
|
||
})
|
||
models = json.loads(catalog.content[0].text)["models"]
|
||
|
||
# Filter available models with pricing
|
||
priced = [
|
||
m for m in models
|
||
if m["status"] == "available" and m.get("pricing")
|
||
]
|
||
priced.sort(key=lambda m: m["pricing"]["inputPerMillion"] or float("inf"))
|
||
|
||
print(f"💡 Cheapest {capability} models:")
|
||
for m in priced[:5]:
|
||
input_cost = m["pricing"]["inputPerMillion"] or 0
|
||
output_cost = m["pricing"]["outputPerMillion"] or 0
|
||
print(f" {m['id']} ({m['provider']}): ${input_cost}/M in, ${output_cost}/M out")
|
||
```
|
||
|
||
---
|
||
|
||
## Security & Scope Enforcement
|
||
|
||
The MCP server supports **fine-grained scope enforcement** for multi-tenant environments:
|
||
|
||
| Scope | Tools |
|
||
| --------------------- | ---------------------------------------------------------------------------------------------- |
|
||
| `read:health` | `get_health`, `simulate_route`, `get_provider_metrics`, `best_combo_for_task`, `explain_route` |
|
||
| `read:combos` | `list_combos`, `get_combo_metrics`, `simulate_route`, `best_combo_for_task`, `test_combo` |
|
||
| `read:quota` | `check_quota` |
|
||
| `read:usage` | `cost_report`, `explain_route`, `get_session_snapshot` |
|
||
| `read:models` | `list_models_catalog` |
|
||
| `read:cache` | `cache_stats` |
|
||
| `read:compression` | `compression_status`, `list_compression_combos`, `compression_combo_stats` |
|
||
| `write:combos` | `switch_combo` |
|
||
| `write:budget` | `set_budget_guard` |
|
||
| `write:resilience` | `set_resilience_profile` |
|
||
| `write:cache` | `cache_flush` |
|
||
| `write:compression` | `compression_configure`, `set_compression_engine` |
|
||
| `execute:completions` | `route_request`, `test_combo` |
|
||
|
||
**Wildcard scopes:** Use `read:*` to grant all read scopes, or `*` for full access.
|
||
|
||
---
|
||
|
||
## Audit Logging
|
||
|
||
Every tool call is logged to the `mcp_tool_audit` SQLite table:
|
||
|
||
- **Input:** SHA-256 hashed (never stores raw prompts)
|
||
- **Output:** Truncated to 200 chars
|
||
- **Metadata:** Tool name, duration, success/error, API key ID
|
||
|
||
Access audit data via:
|
||
|
||
```typescript
|
||
import { getRecentAuditEntries, getAuditStats } from "./audit";
|
||
|
||
const entries = await getRecentAuditEntries(50);
|
||
const stats = await getAuditStats();
|
||
// stats: { totalCalls, successRate, avgDurationMs, topTools }
|
||
```
|
||
|
||
---
|
||
|
||
## File Structure
|
||
|
||
```
|
||
mcp-server/
|
||
├── server.ts # MCP server setup, essential tool handlers, entry point
|
||
├── index.ts # Barrel export
|
||
├── audit.ts # SQLite audit logger (SHA-256 input hashing)
|
||
├── scopeEnforcement.ts # Fine-grained scope enforcement
|
||
├── schemas/
|
||
│ ├── tools.ts # Zod schemas for core, cache, compression, and proxy tools
|
||
│ ├── a2a.ts # A2A protocol types (Agent Card, Task, JSON-RPC)
|
||
│ ├── audit.ts # Audit & routing decision types + hash helpers
|
||
│ └── index.ts # Schema barrel export
|
||
├── tools/
|
||
│ └── advancedTools.ts # Phase 2 tool handlers (8 advanced tools)
|
||
└── __tests__/
|
||
├── essentialTools.test.ts
|
||
├── advancedTools.test.ts
|
||
└── a2aLifecycle.test.ts
|
||
```
|
||
|
||
---
|
||
|
||
## License
|
||
|
||
Part of [OmniRoute](https://github.com/diegosouzapw/OmniRoute) — MIT License.
|