mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-07-26 09:52:11 +03:00
refactor(docs): create 8 subfolders + diagrams/, move 44 docs preserving history
Group docs into intent-based subfolders so the topic each file covers is visible from the directory layout: architecture/, guides/, reference/, frameworks/, routing/, security/, compression/, ops/. Adds an empty diagrams/ placeholder (populated in FASE 4) and a navigable docs/README.md index. Files were moved with git mv so history is preserved. Internal cross-doc links were rewritten to point at the new subfolder paths. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
diegosouzapw
275
docs/frameworks/A2A-SERVER.md
Normal file
275
docs/frameworks/A2A-SERVER.md
Normal file
@@ -0,0 +1,275 @@
|
||||
# OmniRoute A2A Server Documentation
|
||||
|
||||
> Agent-to-Agent Protocol v0.3 — OmniRoute as an intelligent routing agent
|
||||
|
||||
The A2A surface has two faces:
|
||||
|
||||
- **JSON-RPC 2.0** at `POST /a2a` (canonical entry point, defined in `src/app/a2a/route.ts`).
|
||||
- **REST** under `/api/a2a/*` for dashboards and tooling (status, task list, cancel).
|
||||
|
||||
Tasks are tracked by `A2ATaskManager` (`src/lib/a2a/taskManager.ts`, default 5-minute TTL). Skills are dispatched via `A2A_SKILL_HANDLERS` in `src/lib/a2a/taskExecution.ts`.
|
||||
|
||||
## Agent Discovery
|
||||
|
||||
```bash
|
||||
curl http://localhost:20128/.well-known/agent.json
|
||||
```
|
||||
|
||||
Returns the Agent Card describing OmniRoute's capabilities, skills, and authentication requirements.
|
||||
|
||||
The Agent Card's `version` field is sourced from `process.env.npm_package_version` (see `src/app/.well-known/agent.json/route.ts:13`), so it stays auto-synced with `package.json` on every release.
|
||||
|
||||
---
|
||||
|
||||
## Authentication
|
||||
|
||||
All `/a2a` requests require an API key via the `Authorization` header:
|
||||
|
||||
```
|
||||
Authorization: Bearer YOUR_OMNIROUTE_API_KEY
|
||||
```
|
||||
|
||||
If no API key is configured on the server, authentication is bypassed.
|
||||
|
||||
## Enablement
|
||||
|
||||
A2A is controlled by the **Endpoints → A2A** toggle and is disabled by default. When disabled,
|
||||
`GET /api/a2a/status` reports `status: "disabled"` and `online: false`; JSON-RPC calls to
|
||||
`POST /a2a` return HTTP 503 with JSON-RPC error code `-32000`.
|
||||
|
||||
---
|
||||
|
||||
## JSON-RPC 2.0 Methods
|
||||
|
||||
### `message/send` — Synchronous Execution
|
||||
|
||||
Sends a message to a skill and waits for the complete response.
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/a2a \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer YOUR_KEY" \
|
||||
-d '{
|
||||
"jsonrpc": "2.0",
|
||||
"id": "1",
|
||||
"method": "message/send",
|
||||
"params": {
|
||||
"skill": "smart-routing",
|
||||
"messages": [{"role": "user", "content": "Write a hello world in Python"}],
|
||||
"metadata": {"model": "auto", "combo": "fast-coding"}
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": "1",
|
||||
"result": {
|
||||
"task": { "id": "uuid", "state": "completed" },
|
||||
"artifacts": [{ "type": "text", "content": "..." }],
|
||||
"metadata": {
|
||||
"routing_explanation": "Selected claude-sonnet via provider \"anthropic\" (latency: 1200ms, cost: $0.003)",
|
||||
"cost_envelope": { "estimated": 0.005, "actual": 0.003, "currency": "USD" },
|
||||
"resilience_trace": [
|
||||
{ "event": "primary_selected", "provider": "anthropic", "timestamp": "..." }
|
||||
],
|
||||
"policy_verdict": { "allowed": true, "reason": "within budget and quota limits" }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### `message/stream` — SSE Streaming
|
||||
|
||||
Same as `message/send` but returns Server-Sent Events for real-time streaming.
|
||||
|
||||
```bash
|
||||
curl -N -X POST http://localhost:20128/a2a \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer YOUR_KEY" \
|
||||
-d '{
|
||||
"jsonrpc": "2.0",
|
||||
"id": "1",
|
||||
"method": "message/stream",
|
||||
"params": {
|
||||
"skill": "smart-routing",
|
||||
"messages": [{"role": "user", "content": "Explain quantum computing"}]
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
**SSE Events:**
|
||||
|
||||
```
|
||||
data: {"jsonrpc":"2.0","method":"message/stream","params":{"task":{"id":"...","state":"working"},"chunk":{"type":"text","content":"..."}}}
|
||||
|
||||
: heartbeat 2026-03-03T17:00:00Z
|
||||
|
||||
data: {"jsonrpc":"2.0","method":"message/stream","params":{"task":{"id":"...","state":"completed"},"metadata":{...}}}
|
||||
```
|
||||
|
||||
### `tasks/get` — Query Task Status
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/a2a \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer YOUR_KEY" \
|
||||
-d '{"jsonrpc":"2.0","id":"2","method":"tasks/get","params":{"taskId":"TASK_UUID"}}'
|
||||
```
|
||||
|
||||
### `tasks/cancel` — Cancel a Task
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/a2a \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer YOUR_KEY" \
|
||||
-d '{"jsonrpc":"2.0","id":"3","method":"tasks/cancel","params":{"taskId":"TASK_UUID"}}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Available Skills
|
||||
|
||||
OmniRoute exposes 5 A2A skills wired in `src/lib/a2a/taskExecution.ts::A2A_SKILL_HANDLERS`. Each skill module lives in `src/lib/a2a/skills/`.
|
||||
|
||||
| Skill | ID | Description |
|
||||
| :----------------- | :------------------- | :------------------------------------------------------------------------------------------ |
|
||||
| Smart Routing | `smart-routing` | Routes a prompt through the optimal provider/combo using OmniRoute's combo engine + scoring |
|
||||
| Quota Management | `quota-management` | Reports per-provider quota state, helps callers decide when to throttle/switch |
|
||||
| Provider Discovery | `provider-discovery` | Lists installed providers with capabilities, free-tier flags, OAuth status |
|
||||
| Cost Analysis | `cost-analysis` | Estimates cost of a request/conversation given the catalog + recent usage |
|
||||
| Health Report | `health-report` | Aggregates circuit breaker, cooldown, lockout state per provider |
|
||||
|
||||
> Note: the Agent Card description currently advertises "36+ providers" (`src/app/.well-known/agent.json/route.ts:26` and `:55`). The actual catalog has grown to 180+ providers — the string should be updated in a follow-up change (tracked as a separate doc/code TODO; not modified here).
|
||||
|
||||
---
|
||||
|
||||
## REST API (auxiliary)
|
||||
|
||||
The JSON-RPC endpoint `/a2a` is the canonical A2A entry point. The REST endpoints below provide auxiliary access for dashboards and external tooling:
|
||||
|
||||
| Endpoint | Method | Description | Auth |
|
||||
| :--------------------------- | :----- | :------------------------------- | :--------------------- |
|
||||
| `/api/a2a/status` | GET | Server status, registered skills | (public) |
|
||||
| `/api/a2a/tasks` | GET | List tasks with filters | management |
|
||||
| `/api/a2a/tasks/[id]` | GET | Get task by ID | management |
|
||||
| `/api/a2a/tasks/[id]/cancel` | POST | Cancel running task | management |
|
||||
| `/.well-known/agent.json` | GET | Agent Card (A2A discovery) | (public, cached 3600s) |
|
||||
|
||||
---
|
||||
|
||||
## Adding a New Skill
|
||||
|
||||
1. **Create skill file:** `src/lib/a2a/skills/<your-skill>.ts`
|
||||
|
||||
Export an async function `(task: A2ATask) => Promise<{ artifacts, metadata }>`. Follow the shape of existing skills such as `smartRouting.ts`.
|
||||
|
||||
2. **Register handler:** in `src/lib/a2a/taskExecution.ts`, add an entry to `A2A_SKILL_HANDLERS`:
|
||||
|
||||
```typescript
|
||||
export const A2A_SKILL_HANDLERS = {
|
||||
// ...existing skills
|
||||
"your-skill": async (task) => {
|
||||
const skillModule = await import("./skills/yourSkill");
|
||||
return skillModule.executeYourSkill(task);
|
||||
},
|
||||
};
|
||||
```
|
||||
|
||||
3. **Expose in Agent Card:** in `src/app/.well-known/agent.json/route.ts`, append to the `skills` array:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "your-skill",
|
||||
"name": "Your Skill",
|
||||
"description": "Brief, intent-focused description",
|
||||
"tags": ["routing", "quota"],
|
||||
"examples": ["Sample natural-language invocation"]
|
||||
}
|
||||
```
|
||||
|
||||
4. **Write tests:** `tests/unit/a2a-<your-skill>.test.ts`. Cover happy path + error path.
|
||||
|
||||
5. **Document** the new skill in this file's `Available Skills` table.
|
||||
|
||||
---
|
||||
|
||||
## Task TTL
|
||||
|
||||
Tasks expire after `ttlMinutes` (default 5 min) — configured in the `A2ATaskManager` constructor at `src/lib/a2a/taskManager.ts:82`. To customize, fork the `A2ATaskManager` instantiation and pass a different value (e.g., `new A2ATaskManager(15)` for 15-minute TTL). A background interval sweeps expired tasks every 60 seconds.
|
||||
|
||||
---
|
||||
|
||||
## Task Lifecycle
|
||||
|
||||
```
|
||||
submitted → working → completed
|
||||
→ failed
|
||||
→ cancelled
|
||||
```
|
||||
|
||||
- Tasks expire after 5 minutes by default (see [Task TTL](#task-ttl))
|
||||
- Terminal states: `completed`, `failed`, `cancelled`
|
||||
- Event log tracks every state transition
|
||||
|
||||
---
|
||||
|
||||
## Error Codes
|
||||
|
||||
| Code | Meaning |
|
||||
| :----- | :----------------------------- |
|
||||
| -32700 | Parse error (invalid JSON) |
|
||||
| -32600 | Invalid request / Unauthorized |
|
||||
| -32601 | Method or skill not found |
|
||||
| -32602 | Invalid params |
|
||||
| -32603 | Internal error |
|
||||
| -32000 | A2A endpoint is disabled |
|
||||
|
||||
---
|
||||
|
||||
## Integration Examples
|
||||
|
||||
### Python (requests)
|
||||
|
||||
```python
|
||||
import requests
|
||||
|
||||
resp = requests.post("http://localhost:20128/a2a", json={
|
||||
"jsonrpc": "2.0", "id": "1",
|
||||
"method": "message/send",
|
||||
"params": {
|
||||
"skill": "smart-routing",
|
||||
"messages": [{"role": "user", "content": "Hello"}]
|
||||
}
|
||||
}, headers={"Authorization": "Bearer YOUR_KEY"})
|
||||
|
||||
result = resp.json()["result"]
|
||||
print(result["artifacts"][0]["content"])
|
||||
print(result["metadata"]["routing_explanation"])
|
||||
```
|
||||
|
||||
### TypeScript (fetch)
|
||||
|
||||
```typescript
|
||||
const resp = await fetch("http://localhost:20128/a2a", {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
Authorization: "Bearer YOUR_KEY",
|
||||
},
|
||||
body: JSON.stringify({
|
||||
jsonrpc: "2.0",
|
||||
id: "1",
|
||||
method: "message/send",
|
||||
params: {
|
||||
skill: "smart-routing",
|
||||
messages: [{ role: "user", content: "Hello" }],
|
||||
},
|
||||
}),
|
||||
});
|
||||
const { result } = await resp.json();
|
||||
console.log(result.metadata.routing_explanation);
|
||||
```
|
||||
276
docs/frameworks/AGENT_PROTOCOLS_GUIDE.md
Normal file
276
docs/frameworks/AGENT_PROTOCOLS_GUIDE.md
Normal file
@@ -0,0 +1,276 @@
|
||||
# Agent Protocols Guide
|
||||
|
||||
> **Source:** `src/lib/{a2a,acp,cloudAgent}/`, `src/app/api/{a2a,acp,cloud}/`, `src/app/api/v1/agents/`
|
||||
> **Last updated:** 2026-05-13 — v3.8.0
|
||||
|
||||
OmniRoute exposes three different agent-related surfaces. They look similar at first glance but solve different problems. Use this page to pick the right one.
|
||||
|
||||
## TL;DR
|
||||
|
||||
| Surface | Best for | Transport | Standard |
|
||||
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------- | -------------------- |
|
||||
| **A2A — Agent-to-Agent** | Cross-agent collaboration with peer agents that speak the A2A protocol | JSON-RPC 2.0 over HTTP | A2A v0.3 (open spec) |
|
||||
| **ACP — CLI Agents Registry** | Detecting / registering / launching CLI coding agents installed on the user's machine (Cursor, Cline, Codex CLI, Claude Code, Aider, etc.) | HTTP REST | OmniRoute-specific |
|
||||
| **Cloud Agents** | Submitting long-running coding tasks to external cloud services (Codex Cloud, Devin, Jules) | HTTP REST + DB-backed tasks | OmniRoute-specific |
|
||||
|
||||
The three are independent — pick any subset.
|
||||
|
||||
## Decision Tree
|
||||
|
||||
```
|
||||
Do you need a cloud service to do work outside this machine (Codex Cloud / Devin / Jules)?
|
||||
├─ YES → Cloud Agents (POST /api/v1/agents/tasks)
|
||||
└─ NO → Continue
|
||||
│
|
||||
Do you have a peer agent that speaks A2A and wants to collaborate?
|
||||
├─ YES → A2A (POST /a2a)
|
||||
└─ NO → Continue
|
||||
│
|
||||
Do you need to list / configure CLI coding agents installed locally?
|
||||
├─ YES → ACP (GET /api/acp/agents)
|
||||
└─ NO → Use plain /v1/chat/completions
|
||||
```
|
||||
|
||||
## 1. A2A — Agent-to-Agent
|
||||
|
||||
**Spec:** [A2A v0.3](https://a2a-protocol.org)
|
||||
**OmniRoute endpoint:** `POST /a2a` (JSON-RPC 2.0)
|
||||
**Agent Card:** `GET /.well-known/agent.json`
|
||||
|
||||
### When to use
|
||||
|
||||
- Building a multi-agent system where OmniRoute is one of the peers
|
||||
- Exposing OmniRoute's routing intelligence (smart-routing, quota-management, etc.) to agents in frameworks like Google ADK or generic agent meshes
|
||||
- Wrapping OmniRoute behind a standard discovery + invocation surface
|
||||
|
||||
### Methods
|
||||
|
||||
- `message/send` — submit a message, receive sync response
|
||||
- `message/stream` — submit + receive SSE-streamed progress events
|
||||
- `tasks/get` — read task by ID
|
||||
- `tasks/cancel` — cancel a running task
|
||||
|
||||
### Built-in skills (5)
|
||||
|
||||
- `smart-routing` — route a prompt through the optimal combo
|
||||
- `quota-management` — report per-provider quota state
|
||||
- `provider-discovery` — list installed providers with capabilities
|
||||
- `cost-analysis` — estimate cost of a request/conversation
|
||||
- `health-report` — aggregate breaker/cooldown/lockout state per provider
|
||||
|
||||
### Deep dive
|
||||
|
||||
See [A2A-SERVER.md](./A2A-SERVER.md) for transport details, agent card structure, task TTL config, and the template for adding new skills.
|
||||
|
||||
## 2. ACP — CLI Agents Registry
|
||||
|
||||
**OmniRoute endpoint:** `GET /api/acp/agents`
|
||||
**Source:** `src/lib/acp/{index,manager,registry}.ts`
|
||||
|
||||
### What it is
|
||||
|
||||
ACP is OmniRoute's **local CLI agent inventory**. It detects which coding CLIs are installed on the host (Cursor, Cline, Claude Code, Codex CLI, Continue, etc.), resolves their versions, and surfaces them to the dashboard so the user can wire each CLI to point at OmniRoute.
|
||||
|
||||
This is NOT an external protocol — it's an internal registry that powers the "CLI Tools" UI and the CLI fingerprint tracking (see [CLI-TOOLS.md](../reference/CLI-TOOLS.md)).
|
||||
|
||||
### What it does
|
||||
|
||||
- Probes the host for installed CLI binaries (uses `which` / `where` per OS)
|
||||
- Reads each CLI's version (calls `<bin> --version`)
|
||||
- Optionally accepts user-defined custom agents (binary path + version probe + spawn args)
|
||||
- Persists custom agents in settings
|
||||
- Returns the unified list to the dashboard
|
||||
|
||||
### REST API
|
||||
|
||||
| Endpoint | Method | Description | Auth |
|
||||
| ----------------- | ------ | ------------------------------------------------------------- | ------- |
|
||||
| `/api/acp/agents` | GET | List detected + custom agents (installed/total counts) | API key |
|
||||
| `/api/acp/agents` | POST | Add/update/remove custom agent (action discriminator in body) | API key |
|
||||
|
||||
Body shape for POST (`customAgentBodySchema` in `src/app/api/acp/agents/route.ts`):
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "add|update|remove",
|
||||
"id": "cursor",
|
||||
"name": "Cursor",
|
||||
"binary": "/usr/local/bin/cursor",
|
||||
"versionCommand": "--version",
|
||||
"providerAlias": "cursor",
|
||||
"spawnArgs": ["--api-base", "http://localhost:20128"],
|
||||
"protocol": "stdio"
|
||||
}
|
||||
```
|
||||
|
||||
### Use cases
|
||||
|
||||
- Dashboard "CLI Tools" page lists what's installed and helps you point each at OmniRoute
|
||||
- Custom agents let power users register internal/proprietary CLIs that OmniRoute doesn't know about by default
|
||||
- Detection result fuels the `cli-tools` fingerprint matrix
|
||||
|
||||
### When NOT to use ACP
|
||||
|
||||
- ACP doesn't _run_ tasks. It only detects + configures CLIs. To actually invoke a CLI, you launch it yourself with the env vars OmniRoute provides (`OPENAI_BASE_URL`, `OPENAI_API_KEY`, etc.).
|
||||
|
||||
## 3. Cloud Agents
|
||||
|
||||
**OmniRoute endpoints:** `/api/v1/agents/tasks/*` (lifecycle) + `/api/cloud/*` (plumbing)
|
||||
**Source:** `src/lib/cloudAgent/`
|
||||
|
||||
### What it is
|
||||
|
||||
A uniform interface over third-party cloud coding agents. You submit a prompt + repo URL, OmniRoute dispatches to the right cloud agent, polls status, returns results.
|
||||
|
||||
### Supported agents (3, all confirmed in `src/lib/cloudAgent/agents/`)
|
||||
|
||||
- `codex-cloud` — OpenAI Codex Cloud
|
||||
- `devin` — Cognition Devin
|
||||
- `jules` — Google Jules
|
||||
|
||||
### Lifecycle
|
||||
|
||||
```
|
||||
POST /api/v1/agents/tasks
|
||||
→ BaseAgent.createTask() per agent class
|
||||
→ external service starts work
|
||||
→ task row created in DB (cloud_agent_tasks)
|
||||
↓
|
||||
GET /api/v1/agents/tasks/[id]
|
||||
→ lazy status sync from provider
|
||||
→ returns current status + plan + activity log
|
||||
↓
|
||||
POST /api/v1/agents/tasks/[id] (action: "approve" | "message" | "cancel")
|
||||
→ forwards to provider (or marks cancelled locally)
|
||||
↓
|
||||
DELETE /api/v1/agents/tasks/[id]
|
||||
→ local cancel
|
||||
```
|
||||
|
||||
### Auth
|
||||
|
||||
⚠️ **All `/api/v1/agents/tasks/*` endpoints require management auth** (commit `588a0333`). Bearer-only callers receive 401 since v3.8.0.
|
||||
|
||||
### Deep dive
|
||||
|
||||
See [CLOUD_AGENT.md](./CLOUD_AGENT.md) for the `CloudAgentBase` contract, per-agent specifics, schema details, and credential plumbing endpoints.
|
||||
|
||||
## Comparison: A2A vs Cloud Agents
|
||||
|
||||
Both have "long-running tasks" but at different layers:
|
||||
|
||||
| Aspect | A2A | Cloud Agents |
|
||||
| ------------------ | --------------------------------------------------------------------------------- | ---------------------------------------- |
|
||||
| Standard | Open A2A v0.3 | OmniRoute-specific |
|
||||
| Where compute runs | Inside OmniRoute (uses configured combos) | External (Codex / Devin / Jules servers) |
|
||||
| Task duration | Default TTL 5 min (configurable in `TaskManager`) | Minutes to hours |
|
||||
| Repo-aware | No (passes prompts only) | Yes (repo URL + branch) |
|
||||
| Use case | Cross-agent collab, smart routing as a service | Delegate "implement feature X in repo Y" |
|
||||
| Auth | Optional `OMNIROUTE_API_KEY` for `/a2a`; management for `/api/a2a/*` REST helpers | Always management |
|
||||
|
||||
## Integration Examples
|
||||
|
||||
### Discover OmniRoute's A2A capabilities
|
||||
|
||||
```bash
|
||||
curl http://localhost:20128/.well-known/agent.json
|
||||
```
|
||||
|
||||
Returns the Agent Card with all 5 skills, transports, and version.
|
||||
|
||||
### Call OmniRoute as an A2A agent
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/a2a \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"jsonrpc": "2.0",
|
||||
"method": "message/send",
|
||||
"params": {
|
||||
"messages": [{"role": "user", "content": "Route this prompt"}],
|
||||
"skillId": "smart-routing"
|
||||
},
|
||||
"id": 1
|
||||
}'
|
||||
```
|
||||
|
||||
### List installed CLI agents via ACP
|
||||
|
||||
```bash
|
||||
curl http://localhost:20128/api/acp/agents \
|
||||
-H "Authorization: Bearer <api-key>"
|
||||
```
|
||||
|
||||
### Add a custom CLI agent
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/api/acp/agents \
|
||||
-H "Authorization: Bearer <api-key>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"action": "add",
|
||||
"id": "my-custom-cli",
|
||||
"name": "My Custom CLI",
|
||||
"binary": "/opt/mycli/bin/mycli",
|
||||
"versionCommand": "--version",
|
||||
"providerAlias": "openai"
|
||||
}'
|
||||
```
|
||||
|
||||
### Submit a Cloud Agent task
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/api/v1/agents/tasks \
|
||||
-H "Cookie: auth_token=..." \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"agentId": "devin",
|
||||
"prompt": "Implement feature X in repo Y",
|
||||
"repo": "https://github.com/user/repo",
|
||||
"branch": "main"
|
||||
}'
|
||||
```
|
||||
|
||||
### Poll cloud task status
|
||||
|
||||
```bash
|
||||
curl http://localhost:20128/api/v1/agents/tasks/<task-id> \
|
||||
-H "Cookie: auth_token=..."
|
||||
```
|
||||
|
||||
## When to Use What
|
||||
|
||||
- **Chatbot / copilot frontend** → `/v1/chat/completions` (OpenAI-compat — not an agent protocol)
|
||||
- **Multi-agent collaboration** → A2A
|
||||
- **Listing local CLIs in the dashboard** → ACP
|
||||
- **Delegating long-running coding tasks to cloud services** → Cloud Agents
|
||||
|
||||
## Internal Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────┐
|
||||
│ OmniRoute Core │
|
||||
└─────────────────────┘
|
||||
↑ ↑ ↑
|
||||
┌─────────┘ │ └─────────┐
|
||||
│ │ │
|
||||
┌───────┐ ┌─────────┐ ┌────────────┐
|
||||
│ A2A │ │ ACP │ │ Cloud │
|
||||
│ (/a2a)│ │ (/acp) │ │ Agents │
|
||||
└───────┘ └─────────┘ │ (/v1/agents│
|
||||
│ │ │ /tasks) │
|
||||
↓ ↓ └────────────┘
|
||||
External peer Local CLI │
|
||||
agents that binaries on ↓
|
||||
speak A2A v0.3 the host Codex Cloud,
|
||||
Devin, Jules
|
||||
```
|
||||
|
||||
## See Also
|
||||
|
||||
- [A2A-SERVER.md](./A2A-SERVER.md) — A2A deep dive
|
||||
- [CLOUD_AGENT.md](./CLOUD_AGENT.md) — Cloud Agents deep dive
|
||||
- [CLI-TOOLS.md](../reference/CLI-TOOLS.md) — External CLI integrations (uses ACP)
|
||||
- [SKILLS.md](./SKILLS.md) — Skills framework (different from A2A skills — local execution sandbox)
|
||||
- [API_REFERENCE.md](../reference/API_REFERENCE.md#agents-protocol) — endpoint reference
|
||||
- Source: `src/lib/{a2a,acp,cloudAgent}/`
|
||||
372
docs/frameworks/CLOUD_AGENT.md
Normal file
372
docs/frameworks/CLOUD_AGENT.md
Normal file
@@ -0,0 +1,372 @@
|
||||
# Cloud Agents
|
||||
|
||||
> **Source of truth:** `src/lib/cloudAgent/` and `src/app/api/v1/agents/tasks/`
|
||||
> **Last updated:** 2026-05-13 — v3.8.0
|
||||
|
||||
OmniRoute orchestrates third-party cloud-hosted coding agents (Codex Cloud, Devin,
|
||||
Jules) as long-running tasks. Each agent is wrapped behind a uniform interface so
|
||||
clients can submit a prompt + repo URL and receive results without dealing with
|
||||
provider-specific APIs.
|
||||
|
||||
A Cloud Agent task is **not** a regular chat completion. It is a durable, multi-step
|
||||
unit of work that may take minutes to hours, can produce a Pull Request as its
|
||||
artifact, and supports follow-up messages and (in some providers) plan approval gates.
|
||||
|
||||
## Supported Agents
|
||||
|
||||
| Provider ID | Class | Source | Upstream Base URL | Plan Approval |
|
||||
| ------------- | ----------------- | ------------------------------------ | --------------------------------------- | ------------- |
|
||||
| `jules` | `JulesAgent` | `src/lib/cloudAgent/agents/jules.ts` | `https://jules.googleapis.com/v1alpha` | Yes |
|
||||
| `devin` | `DevinAgent` | `src/lib/cloudAgent/agents/devin.ts` | `https://api.devin.ai/v1` | Yes |
|
||||
| `codex-cloud` | `CodexCloudAgent` | `src/lib/cloudAgent/agents/codex.ts` | `https://api.openai.com/v1/codex/cloud` | No (auto) |
|
||||
|
||||
Registry: `src/lib/cloudAgent/registry.ts` — exports `getAgent(providerId)`,
|
||||
`getAvailableAgents()`, and `isCloudAgentProvider(providerId)`. The registry is a
|
||||
plain in-memory `Record<string, CloudAgentBase>` populated at module load.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Client (Dashboard / CLI / API)
|
||||
→ POST /api/v1/agents/tasks (management auth required)
|
||||
→ CreateCloudAgentTaskSchema validation (Zod)
|
||||
→ registry.getAgent(providerId)
|
||||
→ getCloudAgentCredentials(providerId)
|
||||
└─ pulls from getProviderConnections({ provider, isActive: true })
|
||||
(apiKey first, fallback to accessToken)
|
||||
→ agent.createTask({ prompt, source, options }, credentials)
|
||||
└─ HTTP POST to upstream provider API
|
||||
└─ returns CloudAgentTask with internal id + externalId
|
||||
→ insertCloudAgentTask(...) into cloud_agent_tasks (SQLite)
|
||||
|
||||
Polling (lazy sync on read):
|
||||
GET /api/v1/agents/tasks/[id]
|
||||
→ getCloudAgentTaskById(id)
|
||||
→ agent.getStatus(externalId, credentials) // refreshes status + activities
|
||||
→ updateCloudAgentTask(...) with new status, result, completed_at
|
||||
→ return serialized task
|
||||
|
||||
Interactions:
|
||||
POST /api/v1/agents/tasks/[id] body: { action: "approve" | "message" | "cancel" }
|
||||
→ agent.approvePlan(externalId, credentials) for "approve"
|
||||
→ agent.sendMessage(externalId, message, credentials) for "message"
|
||||
→ status flips to "cancelled" for "cancel" (local-only)
|
||||
```
|
||||
|
||||
Sync is **lazy**: status is refreshed from the upstream on every `GET /tasks/[id]`.
|
||||
There is no background poller. Dashboards that need fresh state should poll the GET
|
||||
endpoint at a sensible interval.
|
||||
|
||||
## `CloudAgentBase` Interface
|
||||
|
||||
Source: `src/lib/cloudAgent/baseAgent.ts`
|
||||
|
||||
```typescript
|
||||
export interface AgentCredentials {
|
||||
apiKey: string;
|
||||
baseUrl?: string;
|
||||
}
|
||||
|
||||
export interface CreateTaskParams {
|
||||
prompt: string;
|
||||
source: CloudAgentSource;
|
||||
options: {
|
||||
autoCreatePr?: boolean;
|
||||
planApprovalRequired?: boolean;
|
||||
environment?: Record<string, string>;
|
||||
};
|
||||
}
|
||||
|
||||
export interface GetStatusResult {
|
||||
status: CloudAgentStatus;
|
||||
externalId?: string;
|
||||
result?: CloudAgentResult;
|
||||
activities: CloudAgentActivity[];
|
||||
error?: string;
|
||||
}
|
||||
|
||||
export abstract class CloudAgentBase {
|
||||
abstract readonly providerId: string;
|
||||
abstract readonly baseUrl: string;
|
||||
|
||||
abstract createTask(p: CreateTaskParams, c: AgentCredentials): Promise<CloudAgentTask>;
|
||||
abstract getStatus(externalId: string, c: AgentCredentials): Promise<GetStatusResult>;
|
||||
abstract approvePlan(externalId: string, c: AgentCredentials): Promise<void>;
|
||||
abstract sendMessage(
|
||||
externalId: string,
|
||||
message: string,
|
||||
c: AgentCredentials
|
||||
): Promise<CloudAgentActivity>;
|
||||
abstract listSources(
|
||||
c: AgentCredentials
|
||||
): Promise<{ name: string; url: string; branch?: string }[]>;
|
||||
|
||||
protected mapStatus(raw: string): CloudAgentStatus; // heuristic upstream-string → enum
|
||||
protected generateTaskId(): string; // `task_<ts>_<rand>`
|
||||
protected generateActivityId(): string; // `act_<ts>_<rand>`
|
||||
}
|
||||
```
|
||||
|
||||
`CodexCloudAgent.approvePlan` intentionally throws — Codex Cloud auto-plans and has
|
||||
no approval gate. `CodexCloudAgent.listSources` returns `[]`.
|
||||
|
||||
## Domain Types
|
||||
|
||||
Source: `src/lib/cloudAgent/types.ts`
|
||||
|
||||
```typescript
|
||||
export const CLOUD_AGENT_STATUS = {
|
||||
QUEUED: "queued",
|
||||
RUNNING: "running",
|
||||
AWAITING_APPROVAL: "awaiting_approval",
|
||||
COMPLETED: "completed",
|
||||
FAILED: "failed",
|
||||
CANCELLED: "cancelled",
|
||||
} as const;
|
||||
|
||||
export interface CloudAgentSource {
|
||||
repoName: string;
|
||||
repoUrl: string; // must be a valid URL
|
||||
branch?: string;
|
||||
}
|
||||
|
||||
export interface CloudAgentResult {
|
||||
prUrl?: string;
|
||||
prNumber?: number;
|
||||
commitMessage?: string;
|
||||
diffUrl?: string;
|
||||
summary?: string;
|
||||
duration?: number; // seconds, positive int
|
||||
cost?: number; // positive float
|
||||
}
|
||||
|
||||
export interface CloudAgentActivity {
|
||||
id: string;
|
||||
type: "plan" | "command" | "code_change" | "message" | "error" | "completion";
|
||||
content: string;
|
||||
timestamp: string; // ISO 8601
|
||||
metadata?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
export interface CloudAgentTask {
|
||||
id: string; // internal `task_...` id
|
||||
providerId: "jules" | "devin" | "codex-cloud";
|
||||
externalId?: string; // upstream provider's id
|
||||
status: CloudAgentStatus;
|
||||
prompt: string; // 1..10000 chars
|
||||
source: CloudAgentSource;
|
||||
options: {
|
||||
autoCreatePr?: boolean;
|
||||
planApprovalRequired?: boolean;
|
||||
environment?: Record<string, string>;
|
||||
};
|
||||
result?: CloudAgentResult;
|
||||
activities: CloudAgentActivity[];
|
||||
error?: string;
|
||||
createdAt: string;
|
||||
updatedAt: string;
|
||||
completedAt?: string;
|
||||
}
|
||||
```
|
||||
|
||||
Validation schemas (`CreateCloudAgentTaskSchema`, `UpdateCloudAgentTaskSchema`) are
|
||||
exported alongside the types and are used by the route handlers.
|
||||
|
||||
## Database
|
||||
|
||||
Source: `src/lib/cloudAgent/db.ts` — table is created lazily via
|
||||
`createCloudAgentTaskTable()` (also called from `src/lib/cloudAgent/index.ts` at
|
||||
module import).
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS cloud_agent_tasks (
|
||||
id TEXT PRIMARY KEY,
|
||||
provider_id TEXT NOT NULL,
|
||||
external_id TEXT,
|
||||
status TEXT NOT NULL DEFAULT 'queued',
|
||||
prompt TEXT NOT NULL,
|
||||
source TEXT NOT NULL, -- JSON
|
||||
options TEXT DEFAULT '{}', -- JSON
|
||||
result TEXT, -- JSON
|
||||
activities TEXT DEFAULT '[]', -- JSON
|
||||
error TEXT,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
updated_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
completed_at TEXT
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_cloud_agent_tasks_provider ON cloud_agent_tasks(provider_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_cloud_agent_tasks_status ON cloud_agent_tasks(status);
|
||||
CREATE INDEX IF NOT EXISTS idx_cloud_agent_tasks_created ON cloud_agent_tasks(created_at DESC);
|
||||
```
|
||||
|
||||
`updateCloudAgentTask` enforces a **column whitelist** to prevent SQL injection:
|
||||
`status`, `prompt`, `source`, `options`, `result`, `activities`, `error`,
|
||||
`completed_at`. Any other key in the partial update is silently dropped.
|
||||
|
||||
## REST API — Task Lifecycle
|
||||
|
||||
**Auth:** All `/api/v1/agents/tasks*` endpoints require **management auth**
|
||||
(`requireCloudAgentManagementAuth` wraps `requireManagementAuth` from
|
||||
`src/lib/api/requireManagementAuth`). This is enforced after commit `588a0333`
|
||||
(_"fix(auth): require management auth for agent and cooldown APIs"_).
|
||||
|
||||
| Method | Path | Purpose |
|
||||
| ------- | ----------------------------- | ------------------------------------------------------ |
|
||||
| OPTIONS | `/api/v1/agents/tasks` | CORS preflight |
|
||||
| GET | `/api/v1/agents/tasks` | List tasks (filter: `provider`, `status`, `limit≤500`) |
|
||||
| POST | `/api/v1/agents/tasks` | Create task (dispatches to upstream + persists) |
|
||||
| DELETE | `/api/v1/agents/tasks?id=...` | Delete task by query id (does **not** cancel upstream) |
|
||||
| OPTIONS | `/api/v1/agents/tasks/[id]` | CORS preflight |
|
||||
| GET | `/api/v1/agents/tasks/[id]` | Read task + lazy-sync status from upstream |
|
||||
| POST | `/api/v1/agents/tasks/[id]` | Action: `approve` / `message` / `cancel` |
|
||||
| DELETE | `/api/v1/agents/tasks/[id]` | Delete task by path id |
|
||||
|
||||
### Create task
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/api/v1/agents/tasks \
|
||||
-H "Cookie: auth_token=..." \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"providerId": "devin",
|
||||
"prompt": "Fix the bug in src/foo.ts where the parser returns null",
|
||||
"source": {
|
||||
"repoName": "user/repo",
|
||||
"repoUrl": "https://github.com/user/repo",
|
||||
"branch": "main"
|
||||
},
|
||||
"options": {
|
||||
"autoCreatePr": true,
|
||||
"planApprovalRequired": false
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
Response `201`:
|
||||
|
||||
```json
|
||||
{
|
||||
"data": {
|
||||
"id": "task_1731512345678_abc123def",
|
||||
"providerId": "devin",
|
||||
"externalId": "session_xyz",
|
||||
"status": "queued",
|
||||
"prompt": "...",
|
||||
"source": { "repoName": "user/repo", "repoUrl": "...", "branch": "main" },
|
||||
"options": { "autoCreatePr": true },
|
||||
"createdAt": "2026-05-13T12:34:56.789Z"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Approve a plan
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/api/v1/agents/tasks/<id> \
|
||||
-H "Cookie: auth_token=..." \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"action":"approve"}'
|
||||
```
|
||||
|
||||
### Send a follow-up message
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/api/v1/agents/tasks/<id> \
|
||||
-d '{"action":"message","message":"Also add a unit test for the parser"}'
|
||||
```
|
||||
|
||||
### Cancel (local status only)
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/api/v1/agents/tasks/<id> \
|
||||
-d '{"action":"cancel"}'
|
||||
```
|
||||
|
||||
`cancel` flips `status` to `"cancelled"` in the local DB but does **not** call the
|
||||
upstream provider — there is no abort RPC in `CloudAgentBase`. To stop billing
|
||||
upstream, terminate the task in the provider's own console.
|
||||
|
||||
## REST API — Cloud Provider Plumbing
|
||||
|
||||
These auxiliary endpoints under `src/app/api/cloud/` are used by remote clients
|
||||
(the CLI, the Electron app, or sync workers) to read provider connection metadata
|
||||
and resolve model aliases. They are authenticated with a **regular API key**
|
||||
(via `validateApiKey`), not the management auth used by the task endpoints.
|
||||
|
||||
| Method | Path | Purpose |
|
||||
| ------ | ------------------------------- | ------------------------------------------------------------------- |
|
||||
| POST | `/api/cloud/auth` | Validate API key, return masked connection metadata + model aliases |
|
||||
| PUT | `/api/cloud/credentials/update` | Refresh `accessToken` / `refreshToken` / `expiresAt` |
|
||||
| POST | `/api/cloud/model/resolve` | Resolve a model alias to `{ provider, model }` |
|
||||
| GET | `/api/cloud/models/alias` | List all model aliases |
|
||||
| PUT | `/api/cloud/models/alias` | Set a model alias (and auto-sync to Cloud if enabled) |
|
||||
|
||||
`/api/cloud/auth` never returns raw `apiKey` / `accessToken` / `refreshToken`. It
|
||||
returns `hasApiKey`, `hasAccessToken`, `hasRefreshToken`, and a masked preview
|
||||
(`maskedApiKey`: first 4 + `****` + last 4).
|
||||
|
||||
## Credentials Resolution
|
||||
|
||||
`getCloudAgentCredentials(providerId)` in `src/lib/cloudAgent/api.ts`:
|
||||
|
||||
1. Loads active provider connections via `getProviderConnections({ provider: providerId, isActive: true })`.
|
||||
2. For each connection, prefers `apiKey` (trimmed). Falls back to `accessToken`.
|
||||
3. Returns the first non-empty token wrapped as `{ apiKey: token }`.
|
||||
4. Returns `null` if no usable token is found — the API responds `400` with
|
||||
`"No active credentials configured for cloud agent provider: <id>"`.
|
||||
|
||||
This means Cloud Agents reuse the same Provider Connection table as regular LLM
|
||||
providers. To enable Jules, create an active connection with `provider: "jules"`
|
||||
and a populated `apiKey`.
|
||||
|
||||
## Dashboard
|
||||
|
||||
Source: `src/app/(dashboard)/dashboard/cloud-agents/page.tsx`
|
||||
|
||||
A `"use client"` React page that:
|
||||
|
||||
- Lists tasks (polled via `GET /api/v1/agents/tasks`).
|
||||
- Submits new tasks via a form that maps to `CreateCloudAgentTaskSchema`.
|
||||
- Shows status badges (`queued`, `running`, `awaiting_approval`, `completed`,
|
||||
`failed`, `cancelled`) and renders the `activities[]` timeline.
|
||||
- Surfaces the `result.prUrl` / `commitMessage` / `summary` when `status === "completed"`.
|
||||
|
||||
## Integration with A2A
|
||||
|
||||
Cloud Agents can be exposed as A2A skills by registering an A2A skill that delegates
|
||||
its `tasks/send` handler to `getAgent(...).createTask(...)` and translates A2A task
|
||||
status events to the JSON-RPC 2.0 protocol. See [A2A-SERVER.md](./A2A-SERVER.md).
|
||||
|
||||
## Adding a New Cloud Agent
|
||||
|
||||
1. Create `src/lib/cloudAgent/agents/<name>.ts` extending `CloudAgentBase`.
|
||||
2. Implement `createTask`, `getStatus`, `approvePlan` (or throw if N/A),
|
||||
`sendMessage`, `listSources`. Use `this.mapStatus(...)` for status normalization.
|
||||
3. Register in `src/lib/cloudAgent/registry.ts` under a stable `providerId`.
|
||||
4. Extend the `providerId` literal union in `src/lib/cloudAgent/types.ts`
|
||||
(`CloudAgentTask.providerId` and `CreateCloudAgentTaskSchema`).
|
||||
5. Add the provider to `src/shared/constants/providers.ts` if it needs a connection
|
||||
record. OAuth-based providers also need `src/lib/oauth/providers/`.
|
||||
6. Add tests under `tests/unit/cloud-agent-*.test.ts`.
|
||||
7. Update this doc and the dashboard's `CLOUD_AGENTS` constant.
|
||||
|
||||
## Configuration
|
||||
|
||||
| Env Var | Purpose |
|
||||
| ---------------- | ----------------------------------------------------------- |
|
||||
| `DATA_DIR` | Location of the SQLite database holding `cloud_agent_tasks` |
|
||||
| `JWT_SECRET` | Required for management auth on task endpoints |
|
||||
| `API_KEY_SECRET` | Required to encrypt provider connection credentials at rest |
|
||||
|
||||
No Cloud-Agent-specific env vars exist today — every secret lives in the
|
||||
`provider_connections` table.
|
||||
|
||||
## See Also
|
||||
|
||||
- [A2A-SERVER.md](./A2A-SERVER.md)
|
||||
- [API_REFERENCE.md](../reference/API_REFERENCE.md)
|
||||
- [SKILLS.md](./SKILLS.md)
|
||||
- [MEMORY.md](./MEMORY.md)
|
||||
- Source: `src/lib/cloudAgent/`
|
||||
- Routes: `src/app/api/v1/agents/tasks/`, `src/app/api/cloud/`
|
||||
- Dashboard: `src/app/(dashboard)/dashboard/cloud-agents/page.tsx`
|
||||
244
docs/frameworks/EVALS.md
Normal file
244
docs/frameworks/EVALS.md
Normal file
@@ -0,0 +1,244 @@
|
||||
# Evaluations (Evals)
|
||||
|
||||
> **Source of truth:** `src/lib/evals/`, `src/lib/db/evals.ts`, `src/app/api/evals/`
|
||||
> **Last updated:** 2026-05-13 — v3.8.0
|
||||
|
||||
OmniRoute ships a generic evaluation framework you can use to benchmark routing
|
||||
configurations, single providers/models, or the bundled "golden set" suites.
|
||||
Use it to verify routing changes, validate new providers, and gate releases
|
||||
before promoting them to production traffic.
|
||||
|
||||
The framework is implemented as:
|
||||
|
||||
- A pure runner (`src/lib/evals/evalRunner.ts`) that registers in-memory
|
||||
built-in suites, evaluates outputs against expected criteria, and aggregates
|
||||
scorecards.
|
||||
- A persistence layer (`src/lib/db/evals.ts`) for custom (user-defined) suites
|
||||
and historical runs in SQLite.
|
||||
- An orchestration layer (`src/lib/evals/runtime.ts`) that executes each case
|
||||
by dispatching real calls to `POST /v1/chat/completions`, captures latency
|
||||
and outputs, and persists the run.
|
||||
- REST endpoints under `/api/evals/*` (management-auth only).
|
||||
- A dashboard surface at `Dashboard → Usage → Evals` (`EvalsTab.tsx`).
|
||||
|
||||
## Concepts
|
||||
|
||||
### Suite
|
||||
|
||||
A suite is a named collection of test cases with a `description` and one or
|
||||
more cases. Suites come from two sources:
|
||||
|
||||
| Source | Where defined | Mutable at runtime? |
|
||||
| ---------- | --------------------------------------------- | ------------------- |
|
||||
| `built-in` | Registered via `registerSuite()` at boot | No (code-defined) |
|
||||
| `custom` | Stored in SQLite `eval_suites` + `eval_cases` | Yes (via API/UI) |
|
||||
|
||||
The current built-in suites (see `src/lib/evals/evalRunner.ts`):
|
||||
|
||||
- `golden-set` — 10 baseline cases across greeting/math/translation/safety
|
||||
- `coding-proficiency` — Python/JS/SQL/TS/bug detection
|
||||
- `reasoning-logic` — syllogisms, word problems, pattern recognition
|
||||
- `multilingual` — translation and language detection
|
||||
- `safety-guardrails` — PII, jailbreak, refusal, bias awareness
|
||||
- `instruction-following` — JSON-only, numbered lists, language constraints
|
||||
- `codex-comparison` — head-to-head coding tasks intended for compare mode
|
||||
|
||||
### Case
|
||||
|
||||
Each case carries:
|
||||
|
||||
| Field | Description |
|
||||
| ---------- | ------------------------------------------------------------ |
|
||||
| `id` | Stable identifier (used to key outputs and metrics) |
|
||||
| `name` | Human-readable label |
|
||||
| `model` | Default model when the run uses `suite-default` targeting |
|
||||
| `input` | `{ messages, max_tokens? }` — sent to `/v1/chat/completions` |
|
||||
| `expected` | `{ strategy, value }` — scoring rubric (see below) |
|
||||
| `tags` | Optional labels (e.g. `safety`, `pii`, `jailbreak`) |
|
||||
|
||||
### Target
|
||||
|
||||
The same suite can be run against different targets. The target schema is
|
||||
`evalTargetSchema` in `src/shared/validation/schemas.ts`:
|
||||
|
||||
| Target type | `id` | Behavior |
|
||||
| --------------- | ---------- | --------------------------------------------------------------- |
|
||||
| `suite-default` | `null` | Each case uses its built-in `model` field |
|
||||
| `model` | model name | Force every case through one direct model (e.g. `gpt-4o`) |
|
||||
| `combo` | combo name | Run every case through one combo (exercises the routing engine) |
|
||||
|
||||
For `model` and `combo`, the `id` field is required (enforced by Zod
|
||||
`superRefine`). When `compareTarget` is provided, both targets must differ —
|
||||
the runner persists both runs under the same `runGroupId` for A/B comparison.
|
||||
|
||||
## Scoring Rubrics
|
||||
|
||||
Implemented in `evaluateCase()` (evalRunner.ts):
|
||||
|
||||
| Strategy | Pass when… |
|
||||
| ---------- | -------------------------------------------------------------------- |
|
||||
| `exact` | `actualOutput === expected.value` |
|
||||
| `contains` | `actualOutput.toLowerCase().includes(expected.value.toLowerCase())` |
|
||||
| `regex` | `new RegExp(expected.value).test(actualOutput)` is truthy |
|
||||
| `custom` | `expected.fn(actualOutput, evalCase)` returns truthy (built-in only) |
|
||||
|
||||
**Note:** Custom-function scoring is reserved for code-defined (built-in)
|
||||
suites because functions cannot be serialized through the API. The
|
||||
`evalCaseBuilderSchema` only accepts `contains | exact | regex` for
|
||||
user-created suites.
|
||||
|
||||
There is no LLM-as-judge or embedding-based similarity scorer today — it would
|
||||
be a clean extension point in `evaluateCase()`.
|
||||
|
||||
## Database Schema
|
||||
|
||||
Three tables (migrations `030_create_eval_runs.sql` and
|
||||
`031_create_eval_suites.sql`):
|
||||
|
||||
| Table | Purpose |
|
||||
| ------------- | ---------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `eval_suites` | Custom suite metadata (`id`, `name`, `description`) |
|
||||
| `eval_cases` | Cases per suite — `input_json`, `expected_*`, `tags_json` |
|
||||
| `eval_runs` | Historical runs — `pass_rate`, `total`, `passed`, `failed`, `avg_latency_ms`, `summary_json`, `results_json`, `outputs_json` |
|
||||
|
||||
Built-in suites are **not** stored in the DB. They live in memory and are
|
||||
re-registered every time `evalRunner.ts` is imported.
|
||||
|
||||
## REST API
|
||||
|
||||
All endpoints require management auth (`requireManagementAuth`) — they are not
|
||||
part of the public proxy surface.
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
| ----------------------------- | -------- | ------------------------------------------------------------- |
|
||||
| `/api/evals` | `GET` | List suites + recent runs + scorecard + targets + keys |
|
||||
| `/api/evals` | `POST` | Run a suite (single or compare) — schema `evalRunSuiteSchema` |
|
||||
| `/api/evals/{suiteId}` | `GET` | Fetch one suite (built-in or custom) |
|
||||
| `/api/evals/suites` | `POST` | Create a custom suite — schema `evalSuiteSaveSchema` |
|
||||
| `/api/evals/suites/{suiteId}` | `GET` | Fetch a custom suite |
|
||||
| `/api/evals/suites/{suiteId}` | `PUT` | Replace a custom suite (cases get re-inserted) |
|
||||
| `/api/evals/suites/{suiteId}` | `DELETE` | Delete a custom suite and its cases |
|
||||
|
||||
### Running a suite
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/api/evals \
|
||||
-H "Cookie: auth_token=..." \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"suiteId": "golden-set",
|
||||
"target": { "type": "combo", "id": "my-combo" },
|
||||
"apiKeyId": "optional-api-key-uuid"
|
||||
}'
|
||||
```
|
||||
|
||||
Optional fields:
|
||||
|
||||
- `outputs` — `Record<caseId, string>` of pre-computed outputs. When provided,
|
||||
the runner **skips dispatch** and only scores the cached outputs (useful for
|
||||
offline evaluation).
|
||||
- `compareTarget` — second target to run in parallel; both runs share a
|
||||
generated `runGroupId` for head-to-head viewing.
|
||||
- `apiKeyId` — internal API key used to authenticate the dispatched
|
||||
`/v1/chat/completions` calls. Required when `REQUIRE_API_KEY` is enabled.
|
||||
|
||||
### Creating a custom suite
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/api/evals/suites \
|
||||
-H "Cookie: auth_token=..." \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "Production smoke",
|
||||
"description": "Quick sanity check before deploy",
|
||||
"cases": [
|
||||
{
|
||||
"name": "JSON shape",
|
||||
"model": "gpt-4o",
|
||||
"input": { "messages": [{ "role": "user", "content": "Reply with {\"ok\": true}" }] },
|
||||
"expected": { "strategy": "regex", "value": "\"ok\"\\s*:\\s*true" }
|
||||
}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
## Dispatch Pipeline
|
||||
|
||||
`runEvalSuiteAgainstTarget()` (`src/lib/evals/runtime.ts`):
|
||||
|
||||
1. Resolves the suite (built-in or custom).
|
||||
2. For each case, builds a `Request` to `/v1/chat/completions` with the case's
|
||||
`messages`, the resolved `model`, `stream: false`, and `max_tokens: 512`
|
||||
(or the case override).
|
||||
3. Calls the chat handler directly (in-process — no extra HTTP hop).
|
||||
4. Captures latency and extracts text from either `choices[0].message.content`
|
||||
or the Responses-API `output[]` payload.
|
||||
5. Scores all outputs via `runSuite()`, then persists via `saveEvalRun()`.
|
||||
|
||||
Cases run **sequentially**. There is no concurrency flag today.
|
||||
|
||||
## Dashboard
|
||||
|
||||
The UI lives at `Dashboard → Usage → Evals`
|
||||
(`src/app/(dashboard)/dashboard/usage/components/EvalsTab.tsx`). From there you
|
||||
can:
|
||||
|
||||
- Browse built-in and custom suites with case-by-case preview.
|
||||
- Create/edit/delete custom suites with the case builder.
|
||||
- Pick a target (suite defaults / model / combo), optionally a second
|
||||
`compareTarget`, optionally an API key, then run on demand.
|
||||
- Inspect run history, per-case pass/fail, latency, and captured outputs.
|
||||
- See the rolling scorecard aggregated across the latest run per
|
||||
`(suite, target)` scope.
|
||||
|
||||
## Relationship with the Auto-Assessment RFC
|
||||
|
||||
A separate, narrower assessment subsystem lives at `src/domain/assessment/`
|
||||
and is documented in [docs/RFC-AUTO-ASSESSMENT.md](./RFC-AUTO-ASSESSMENT.md).
|
||||
That RFC targets the Auto Combo engine — automatically scoring providers and
|
||||
models so combos can self-heal when upstreams fail. It uses its own runner,
|
||||
its own categorizer, and its own scoring logic.
|
||||
|
||||
The Evals framework documented here is the **broader, general-purpose
|
||||
testing surface**. Prefer it for arbitrary regression suites, A/B comparisons,
|
||||
and per-release smoke tests. Use the Auto-Assessment subsystem when you need
|
||||
real-time provider health to influence routing decisions.
|
||||
|
||||
## CI Integration
|
||||
|
||||
There is no dedicated `eval:ci` npm script today. Two paths if you want to
|
||||
gate releases on eval results:
|
||||
|
||||
- **HTTP path**: stand up the server, hit `POST /api/evals` with a known
|
||||
`suiteId` + `target`, and assert `runs[].summary.passRate >= N` in the
|
||||
response.
|
||||
- **In-process path**: import `runEvalSuiteAgainstTarget()` from
|
||||
`@/lib/evals/runtime` from a script, run against a test DB, and check the
|
||||
returned `PersistedEvalRun.summary`.
|
||||
|
||||
Tests covering the route and history live at
|
||||
`tests/unit/evals-route.test.ts` and `tests/unit/evals-history.test.ts`.
|
||||
|
||||
## Extension Points
|
||||
|
||||
Common changes and where to make them:
|
||||
|
||||
- **New scoring strategy** — extend the `switch (evalCase.expected.strategy)`
|
||||
block in `evaluateCase()` (`evalRunner.ts`) and widen `EvalCaseStrategy` in
|
||||
`src/lib/db/evals.ts` plus `evalCaseBuilderSchema` in `schemas.ts`.
|
||||
- **New built-in suite** — define a suite object and call `registerSuite()` at
|
||||
the bottom of `evalRunner.ts`. It will be auto-discovered by `listSuites()`.
|
||||
- **Run with concurrency** — change the sequential `for` loop in
|
||||
`runEvalSuiteAgainstTarget()` to a bounded `Promise.all` (no concurrency
|
||||
control exists today).
|
||||
- **Stream/tool-call cases** — currently the runner forces `stream: false`.
|
||||
Streaming or tool-aware evaluation would require changes in `runtime.ts`
|
||||
(capture and aggregate SSE chunks before scoring).
|
||||
|
||||
## See Also
|
||||
|
||||
- [USER_GUIDE.md](../guides/USER_GUIDE.md) — overall product walkthrough
|
||||
- [ARCHITECTURE.md](../architecture/ARCHITECTURE.md) — request pipeline reference
|
||||
- [RFC-AUTO-ASSESSMENT.md](./RFC-AUTO-ASSESSMENT.md) — Auto Combo scoring
|
||||
- Source: `src/lib/evals/`, `src/lib/db/evals.ts`, `src/app/api/evals/`
|
||||
- UI: `src/app/(dashboard)/dashboard/usage/components/EvalsTab.tsx`
|
||||
258
docs/frameworks/MCP-SERVER.md
Normal file
258
docs/frameworks/MCP-SERVER.md
Normal file
@@ -0,0 +1,258 @@
|
||||
# OmniRoute MCP Server Documentation
|
||||
|
||||
> Model Context Protocol server with 37 tools across routing, cache, compression, memory, skills, and proxy operations.
|
||||
>
|
||||
> Source of truth: `open-sse/mcp-server/schemas/tools.ts` (30 tools) + `open-sse/mcp-server/tools/memoryTools.ts` (3 tools) + `open-sse/mcp-server/tools/skillTools.ts` (4 tools). Tool registration and scope wiring lives in `open-sse/mcp-server/server.ts`.
|
||||
|
||||
## Installation
|
||||
|
||||
OmniRoute MCP is built-in. Start it with:
|
||||
|
||||
```bash
|
||||
omniroute --mcp
|
||||
```
|
||||
|
||||
Or via the open-sse transport:
|
||||
|
||||
```bash
|
||||
# HTTP streamable transport (port 20130)
|
||||
omniroute --dev # MCP auto-starts on /mcp endpoint
|
||||
```
|
||||
|
||||
## Transports
|
||||
|
||||
The MCP server exposes three transports, all backed by the same `createMcpServer()` factory:
|
||||
|
||||
| Transport | Where | When to use |
|
||||
| :---------------- | :------------------------------------------ | :--------------------------------------------------- |
|
||||
| `stdio` | `open-sse/mcp-server/server.ts` | IDE integrations (Claude Desktop, Cursor, etc.) |
|
||||
| `sse` | `POST/GET /api/mcp/sse` via `httpTransport` | Browser/agent clients that need an event stream |
|
||||
| `streamable-http` | `POST/GET/DELETE /api/mcp/stream` | Multi-session HTTP clients (`mcp-session-id` header) |
|
||||
|
||||
The active HTTP transport (`sse` or `streamable-http`) is selected by the `mcpTransport` setting. Switching transports closes existing sessions on the other transport.
|
||||
|
||||
## IDE Configuration
|
||||
|
||||
See [MCP Client Configuration](../guides/SETUP_GUIDE.md#mcp-client-configuration) for Claude Desktop,
|
||||
Cursor, Cline, and compatible MCP client setup.
|
||||
|
||||
---
|
||||
|
||||
## Essential Tools (8) — Phase 1
|
||||
|
||||
| Tool | Scopes | Description |
|
||||
| :------------------------------ | :-------------------- | :------------------------------------------------------------ |
|
||||
| `omniroute_get_health` | `read:health` | Uptime, memory, circuit breakers, rate limits, cache stats |
|
||||
| `omniroute_list_combos` | `read:combos` | All configured combos with strategies (optional metrics) |
|
||||
| `omniroute_get_combo_metrics` | `read:combos` | Performance metrics for a specific combo |
|
||||
| `omniroute_switch_combo` | `write:combos` | Activate or deactivate a combo |
|
||||
| `omniroute_check_quota` | `read:quota` | Quota used/total, percent remaining, reset time, token health |
|
||||
| `omniroute_route_request` | `execute:completions` | Send a chat completion through OmniRoute routing |
|
||||
| `omniroute_cost_report` | `read:usage` | Cost report by period (session/day/week/month) |
|
||||
| `omniroute_list_models_catalog` | `read:models` | Full model catalog with capabilities, status, pricing |
|
||||
|
||||
## Phase 1 — Search
|
||||
|
||||
| Tool | Scopes | Description |
|
||||
| :--------------------- | :--------------- | :--------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `omniroute_web_search` | `execute:search` | Web search through OmniRoute search gateway (Serper/Brave/Perplexity/Exa/Tavily/Google PSE/Linkup/SearchAPI/SearXNG) with failover |
|
||||
|
||||
## Advanced Tools (11) — Phase 2
|
||||
|
||||
| Tool | Scopes | Description |
|
||||
| :--------------------------------- | :----------------------------------- | :---------------------------------------------------------------------------------------- |
|
||||
| `omniroute_simulate_route` | `read:health`, `read:combos` | Dry-run routing simulation with fallback tree |
|
||||
| `omniroute_set_budget_guard` | `write:budget` | Session budget with degrade/block/alert action |
|
||||
| `omniroute_set_routing_strategy` | `write:combos` | Update combo strategy at runtime (priority/weighted/auto/etc.) |
|
||||
| `omniroute_set_resilience_profile` | `write:resilience` | Apply `aggressive` / `balanced` / `conservative` resilience preset |
|
||||
| `omniroute_test_combo` | `execute:completions`, `read:combos` | Live test of every provider in a combo using a real upstream call |
|
||||
| `omniroute_get_provider_metrics` | `read:health` | Per-provider metrics with p50/p95/p99 latency and circuit breaker state |
|
||||
| `omniroute_best_combo_for_task` | `read:combos`, `read:health` | Recommend combo by task type with budget/latency constraints |
|
||||
| `omniroute_explain_route` | `read:health`, `read:usage` | Explain why a request was routed to a provider (scoring factors + fallbacks) |
|
||||
| `omniroute_get_session_snapshot` | `read:usage` | Full session snapshot: cost, tokens, top models/providers, errors, budget guard |
|
||||
| `omniroute_db_health_check` | `read:health`, `write:resilience` | Diagnose (and optionally auto-repair) database drift like broken combo refs / orphan rows |
|
||||
| `omniroute_sync_pricing` | `pricing:write` | Sync pricing data from external sources (LiteLLM); supports `dryRun` |
|
||||
|
||||
## Cache Tools (2)
|
||||
|
||||
| Tool | Scopes | Description |
|
||||
| :---------------------- | :------------ | :-------------------------------------------------- |
|
||||
| `omniroute_cache_stats` | `read:cache` | Semantic cache, prompt-cache, and idempotency stats |
|
||||
| `omniroute_cache_flush` | `write:cache` | Flush cache globally or by signature/model |
|
||||
|
||||
## Compression Tools (5)
|
||||
|
||||
| Tool | Scopes | Description |
|
||||
| :---------------------------------- | :------------------ | :----------------------------------------------------------------------------------------------------------------------- |
|
||||
| `omniroute_compression_status` | `read:compression` | Compression settings, analytics summary, and cache-aware stats (includes `analytics.mcpDescriptionCompression` metadata) |
|
||||
| `omniroute_compression_configure` | `write:compression` | Configure compression mode, threshold, target ratio, system-prompt preservation, MCP description compression toggle |
|
||||
| `omniroute_set_compression_engine` | `write:compression` | Pick the active engine (off/caveman/rtk/stacked) and Caveman/RTK intensity |
|
||||
| `omniroute_list_compression_combos` | `read:compression` | List named compression combos and their engine pipelines |
|
||||
| `omniroute_compression_combo_stats` | `read:compression` | Analytics grouped by compression combo and engine |
|
||||
|
||||
`omniroute_compression_status` reports MCP description compression separately under
|
||||
`analytics.mcpDescriptionCompression`. Those values are metadata-size estimates for MCP listable
|
||||
descriptions (`tools`, `prompts`, `resources`, and `resourceTemplates`); they are not provider usage
|
||||
receipts and are marked with `source: "mcp_metadata_estimate"`.
|
||||
|
||||
See [Compression Engines](../compression/COMPRESSION_ENGINES.md) and [RTK Compression](../compression/RTK_COMPRESSION.md) for
|
||||
the runtime compression model behind these tools.
|
||||
|
||||
## 1Proxy Tools (3)
|
||||
|
||||
| Tool | Scopes | Description |
|
||||
| :-------------------------- | :------------- | :-------------------------------------------------------------------------------------- |
|
||||
| `omniroute_oneproxy_fetch` | `read:proxies` | Fetch free proxies from the 1proxy marketplace (protocol/country/quality/limit filters) |
|
||||
| `omniroute_oneproxy_rotate` | `read:proxies` | Get the next available proxy by strategy (`random` / `quality` / `sequential`) |
|
||||
| `omniroute_oneproxy_stats` | `read:proxies` | Pool stats, sync status, distribution by protocol and country |
|
||||
|
||||
## Memory Tools (3)
|
||||
|
||||
Defined in `open-sse/mcp-server/tools/memoryTools.ts`. Auth/scope is enforced through the standard MCP scope pipeline.
|
||||
|
||||
| Tool | Description |
|
||||
| :------------------------ | :---------------------------------------------------------------------------------- |
|
||||
| `omniroute_memory_search` | Search memories by query / type / API key with token-budget enforcement |
|
||||
| `omniroute_memory_add` | Add a new memory entry (`factual` / `episodic` / `procedural` / `semantic`) |
|
||||
| `omniroute_memory_clear` | Clear memories for an API key, optionally filtered by type or `olderThan` timestamp |
|
||||
|
||||
## Skill Tools (4)
|
||||
|
||||
Defined in `open-sse/mcp-server/tools/skillTools.ts`. Backed by `src/lib/skills/registry` + `src/lib/skills/executor`.
|
||||
|
||||
| Tool | Description |
|
||||
| :---------------------------- | :-------------------------------------------------------------------------------- |
|
||||
| `omniroute_skills_list` | List registered skills with optional filtering by API key, name, or enabled state |
|
||||
| `omniroute_skills_enable` | Enable or disable a specific skill by ID |
|
||||
| `omniroute_skills_execute` | Execute a skill with provided input and return the execution record |
|
||||
| `omniroute_skills_executions` | List recent skill execution history |
|
||||
|
||||
---
|
||||
|
||||
## REST API Endpoints
|
||||
|
||||
| Endpoint | Method | Description | Auth |
|
||||
| :--------------------- | :-------------------- | :-------------------------------------------------------------------------------------------------- | :------------------------- |
|
||||
| `/api/mcp/status` | `GET` | Server status: heartbeat, HTTP transport state, audit activity summary | Management (session/admin) |
|
||||
| `/api/mcp/tools` | `GET` | Tool catalog (name, description, scopes, phase, source endpoints) | Management |
|
||||
| `/api/mcp/sse` | `GET` / `POST` | SSE transport endpoint (gated by `mcpEnabled` + `mcpTransport === "sse"`) | API key + scopes |
|
||||
| `/api/mcp/stream` | `POST`/`GET`/`DELETE` | Streamable HTTP transport (uses `mcp-session-id` header; `DELETE` ends the session) | API key + scopes |
|
||||
| `/api/mcp/audit` | `GET` | Audit log entries from `mcp_tool_audit` (filters: `limit`, `offset`, `tool`, `success`, `apiKeyId`) | Management |
|
||||
| `/api/mcp/audit/stats` | `GET` | Aggregated audit stats (`totalCalls`, `successRate`, `avgDurationMs`, top tools) | Management |
|
||||
|
||||
Source files: `src/app/api/mcp/{status,tools,sse,stream,audit,audit/stats}/route.ts`.
|
||||
|
||||
Both SSE and Streamable HTTP transports are blocked until the MCP server is enabled in Settings (`mcpEnabled`) and the appropriate `mcpTransport` is selected. If the wrong transport is configured the route returns HTTP 400 with a hint to switch settings.
|
||||
|
||||
---
|
||||
|
||||
## Authentication & Scopes
|
||||
|
||||
MCP tools are authenticated through API key scopes. Scope enforcement is centralized in
|
||||
`open-sse/mcp-server/scopeEnforcement.ts`. Each tool requires specific scopes:
|
||||
|
||||
| Scope | Tools |
|
||||
| :-------------------- | :---------------------------------------------------------------------------------------------------------------- |
|
||||
| `read:health` | `get_health`, `get_provider_metrics`, `simulate_route`, `explain_route`, `best_combo_for_task`, `db_health_check` |
|
||||
| `read:combos` | `list_combos`, `get_combo_metrics`, `simulate_route`, `best_combo_for_task`, `test_combo` |
|
||||
| `write:combos` | `switch_combo`, `set_routing_strategy` |
|
||||
| `read:quota` | `check_quota` |
|
||||
| `read:usage` | `cost_report`, `get_session_snapshot`, `explain_route` |
|
||||
| `read:models` | `list_models_catalog` |
|
||||
| `execute:completions` | `route_request`, `test_combo` |
|
||||
| `execute:search` | `web_search` |
|
||||
| `write:budget` | `set_budget_guard` |
|
||||
| `write:resilience` | `set_resilience_profile`, `db_health_check` |
|
||||
| `pricing:write` | `sync_pricing` |
|
||||
| `read:cache` | `cache_stats` |
|
||||
| `write:cache` | `cache_flush` |
|
||||
| `read:compression` | `compression_status`, `list_compression_combos`, `compression_combo_stats` |
|
||||
| `write:compression` | `compression_configure`, `set_compression_engine` |
|
||||
| `read:proxies` | `oneproxy_fetch`, `oneproxy_rotate`, `oneproxy_stats` |
|
||||
|
||||
Wildcard scopes are supported: `read:*` grants all read-scopes, `*` grants full access.
|
||||
|
||||
Memory and Skill tools currently do not declare static scope requirements in their definitions; access is gated by the caller's API key and audited through the standard MCP audit pipeline.
|
||||
|
||||
---
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
| :-------------------------------------- | :--------------------------------- | :----------------------------------------------------------------------------------------------------------------------- |
|
||||
| `OMNIROUTE_BASE_URL` | `http://localhost:20128` | Base URL the MCP server uses when calling OmniRoute internal APIs |
|
||||
| `OMNIROUTE_API_KEY` | (empty) | API key forwarded as `Authorization: Bearer` to internal API calls |
|
||||
| `OMNIROUTE_MCP_ENFORCE_SCOPES` | `false` (only `"true"` enables it) | When enabled, missing scopes deny tool calls and log `scope_denied:<reason>` in audit log |
|
||||
| `OMNIROUTE_MCP_SCOPES` | (empty) | Comma-separated allowlist of scopes considered "available" by default (used when caller does not provide its own scopes) |
|
||||
| `OMNIROUTE_MCP_COMPRESS_DESCRIPTIONS` | (unset = on) | When set to `0/false/off/no`, disables MCP description compression at registration time |
|
||||
| `OMNIROUTE_MCP_DESCRIPTION_COMPRESSION` | (unset = on) | Alternate alias for the same toggle as above |
|
||||
| `DATA_DIR` | `~/.omniroute` | Heartbeat file is written to `${DATA_DIR}/runtime/mcp-heartbeat.json` |
|
||||
|
||||
---
|
||||
|
||||
## Description Compression
|
||||
|
||||
MCP tool, prompt, and resource registries can compress descriptions at registration/list time to reduce the metadata footprint exposed to clients (and therefore the prompt context cost). The implementation lives in `open-sse/mcp-server/descriptionCompressor.ts` and is wired into the MCP server via `compressMcpRegistryMetadata` inside `createMcpServer()`.
|
||||
|
||||
- Compression runs over the description text using the Caveman ruleset (`getRulesForContext("all", "full")`) with preserved-block extraction (code spans, fenced blocks, etc.) so structural content is not altered.
|
||||
- Toggle per-deployment via the `compression.mcpDescriptionCompressionEnabled` value in the `key_value` settings table (default: enabled) — exposed in the UI as **Analytics → MCP description compression**.
|
||||
- Toggle process-wide via either `OMNIROUTE_MCP_COMPRESS_DESCRIPTIONS=false` or `OMNIROUTE_MCP_DESCRIPTION_COMPRESSION=false`.
|
||||
- Realtime stats are surfaced via `omniroute_compression_status` under `analytics.mcpDescriptionCompression` and tagged `source: "mcp_metadata_estimate"` to disambiguate from real provider usage receipts.
|
||||
|
||||
---
|
||||
|
||||
## Runtime Heartbeat
|
||||
|
||||
The stdio transport persists liveness to `${DATA_DIR}/runtime/mcp-heartbeat.json` every 5 seconds. The dashboard (`/api/mcp/status`) reads this file plus PID liveness to derive `online`. HTTP transports report state from in-process `getMcpHttpStatus()` instead (no file write).
|
||||
|
||||
The heartbeat snapshot contains:
|
||||
|
||||
```json
|
||||
{
|
||||
"pid": 12345,
|
||||
"startedAt": "2026-05-13T12:34:56.000Z",
|
||||
"lastHeartbeatAt": "2026-05-13T12:35:01.000Z",
|
||||
"version": "1.8.1",
|
||||
"transport": "stdio",
|
||||
"scopesEnforced": false,
|
||||
"allowedScopes": [],
|
||||
"toolCount": 37
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Audit Logging
|
||||
|
||||
Every tool call is logged to the SQLite `mcp_tool_audit` table by `open-sse/mcp-server/audit.ts`:
|
||||
|
||||
- Tool name, arguments (hashed/truncated as per per-tool `auditLevel`), result
|
||||
- Duration in ms, success/failure flag, error message (when applicable)
|
||||
- API key hash, timestamp
|
||||
- Scope denials are logged as `scope_denied:<reason>` with the missing scope list
|
||||
|
||||
Use the dashboard or the `/api/mcp/audit` and `/api/mcp/audit/stats` REST endpoints to inspect recent calls.
|
||||
|
||||
---
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
| :---------------------------------------------- | :--------------------------------------------------------------- |
|
||||
| `open-sse/mcp-server/server.ts` | MCP server factory, stdio entry point, scoped tool registrations |
|
||||
| `open-sse/mcp-server/httpTransport.ts` | SSE + Streamable HTTP transport (session management) |
|
||||
| `open-sse/mcp-server/scopeEnforcement.ts` | Tool scope evaluation and caller resolution |
|
||||
| `open-sse/mcp-server/audit.ts` | Tool call audit logging (`mcp_tool_audit`) |
|
||||
| `open-sse/mcp-server/runtimeHeartbeat.ts` | stdio heartbeat writer (`mcp-heartbeat.json`) |
|
||||
| `open-sse/mcp-server/descriptionCompressor.ts` | Description compression for tool / prompt / resource registries |
|
||||
| `open-sse/mcp-server/schemas/tools.ts` | Zod schemas + tool registry (`MCP_TOOLS`, 30 entries) |
|
||||
| `open-sse/mcp-server/tools/advancedTools.ts` | Phase 2 + cache + 1proxy tool handlers |
|
||||
| `open-sse/mcp-server/tools/compressionTools.ts` | Compression tool handlers |
|
||||
| `open-sse/mcp-server/tools/memoryTools.ts` | Memory tool definitions (3 tools) |
|
||||
| `open-sse/mcp-server/tools/skillTools.ts` | Skill tool definitions (4 tools) |
|
||||
| `src/app/api/mcp/status/route.ts` | `/api/mcp/status` endpoint |
|
||||
| `src/app/api/mcp/tools/route.ts` | `/api/mcp/tools` endpoint |
|
||||
| `src/app/api/mcp/sse/route.ts` | `/api/mcp/sse` SSE transport route |
|
||||
| `src/app/api/mcp/stream/route.ts` | `/api/mcp/stream` Streamable HTTP transport route |
|
||||
| `src/app/api/mcp/audit/route.ts` | `/api/mcp/audit` audit log query |
|
||||
| `src/app/api/mcp/audit/stats/route.ts` | `/api/mcp/audit/stats` aggregated audit metrics |
|
||||
322
docs/frameworks/MEMORY.md
Normal file
322
docs/frameworks/MEMORY.md
Normal file
@@ -0,0 +1,322 @@
|
||||
# Memory System
|
||||
|
||||
> **Source of truth:** `src/lib/memory/` and `src/app/api/memory/`
|
||||
> **Last updated:** 2026-05-13 — v3.8.0
|
||||
|
||||
OmniRoute provides persistent conversational memory keyed by API key (and
|
||||
optionally session id). Memories are extracted automatically from LLM responses
|
||||
via lightweight regex pattern matching and injected back into subsequent
|
||||
requests as a leading system message (or first user message for providers that
|
||||
reject the system role).
|
||||
|
||||
Memory is **scoped per API key**, not per user — every request authenticated
|
||||
with the same API key shares the same memory pool, with optional further
|
||||
scoping by `sessionId`.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Client → /v1/chat/completions (apiKeyInfo resolved upstream)
|
||||
→ handleChatCore() [open-sse/handlers/chatCore.ts]
|
||||
→ resolveMemoryOwnerId(apiKeyInfo) # extracts id
|
||||
→ getMemorySettings() # cached settings
|
||||
→ shouldInjectMemory(body, {enabled}) # gate
|
||||
→ retrieveMemories(apiKeyId, config) # SQL + optional FTS5
|
||||
→ injectMemory(body, memories, provider) # system or user message
|
||||
→ upstream provider call
|
||||
→ on response: extractFacts(text, apiKeyId, sessionId) # non-blocking
|
||||
→ setImmediate → createMemory(fact) per match
|
||||
```
|
||||
|
||||
The injection and extraction call-sites are wired in
|
||||
`open-sse/handlers/chatCore.ts` (look for `retrieveMemories`, `injectMemory`,
|
||||
and `extractFacts`).
|
||||
|
||||
## Storage Layers
|
||||
|
||||
### Primary: SQLite (`memories` table)
|
||||
|
||||
Created by migration `015_create_memories.sql`:
|
||||
|
||||
| Column | Type | Notes |
|
||||
| --------------------------- | ------------------ | -------------------------------------------------------------------- |
|
||||
| `id` | `TEXT PRIMARY KEY` | UUID generated via `crypto.randomUUID()` |
|
||||
| `api_key_id` | `TEXT NOT NULL` | Owning API key |
|
||||
| `session_id` | `TEXT` | Optional per-conversation scope |
|
||||
| `type` | `TEXT NOT NULL` | One of `factual`, `episodic`, `procedural`, `semantic` |
|
||||
| `key` | `TEXT` | Stable upsert key, e.g. `preference:i_prefer_python` |
|
||||
| `content` | `TEXT NOT NULL` | The actual fact text |
|
||||
| `metadata` | `TEXT` | JSON blob (category, extractedAt, source, ...) |
|
||||
| `created_at` / `updated_at` | `TEXT` | ISO 8601 strings |
|
||||
| `expires_at` | `TEXT` | Optional expiry; `NULL` means permanent |
|
||||
| `memory_id` | `INTEGER UNIQUE` | Added by `023_fix_memory_fts_uuid.sql` to bridge UUIDs ↔ FTS5 rowids |
|
||||
|
||||
Indexes: `api_key_id`, `session_id`, `type`, `expires_at`, plus the unique
|
||||
`memory_id` index.
|
||||
|
||||
**Upsert semantics**: `createMemory()` looks for an existing row with the same
|
||||
`(api_key_id, key)` and updates it in place when found (merging `metadata` via
|
||||
shallow spread). This keeps the table from growing unbounded for repeated
|
||||
preference statements.
|
||||
|
||||
### Full-text Search (`memory_fts` virtual table)
|
||||
|
||||
`022_add_memory_fts5.sql` creates an FTS5 virtual table over `content` and
|
||||
`key`. `023_fix_memory_fts_uuid.sql` fixes a real-world bug where the UUID
|
||||
primary key did not join to FTS5's integer rowid — the migration adds the
|
||||
`memory_id` column, recreates the FTS table, and wires triggers
|
||||
(`memory_fts_ai`, `memory_fts_ad`, `memory_fts_au`) that keep FTS in sync on
|
||||
INSERT, DELETE, and UPDATE.
|
||||
|
||||
Used by `retrieval.ts` for the `semantic` and `hybrid` strategies (see below).
|
||||
The retrieval code guards with `hasTable("memory_fts")` and falls back to
|
||||
chronological order if the FTS table is missing or the FTS query throws.
|
||||
|
||||
### Optional: Qdrant (vector store)
|
||||
|
||||
`src/lib/memory/qdrant.ts` implements an optional Qdrant integration for true
|
||||
semantic memory:
|
||||
|
||||
- `upsertSemanticMemoryPoint()` — embed `key + content` with the configured
|
||||
embedding model, ensure the collection exists (creates cosine-distance
|
||||
vectors on first use), and upsert a point with payload `{memoryId,
|
||||
apiKeyId, sessionId, key, content, metadata, createdAtUnix, expiresAtUnix}`.
|
||||
- `searchSemanticMemory(query, topK, scope)` — embed the query, search the
|
||||
collection filtered by `kind = "omniroute_memory"` and optionally by
|
||||
`apiKeyId` / `sessionId`. Caps `topK` to `[1, 20]`.
|
||||
- `deleteSemanticMemoryPoint(id)` — single point delete.
|
||||
- `cleanupSemanticMemoryPoints({retentionDays})` — bulk delete points whose
|
||||
`expiresAtUnix` is in the past or whose `createdAtUnix` is older than the
|
||||
retention cutoff. Counts first so the dashboard can show actual numbers.
|
||||
- `checkQdrantHealth()` — `GET /readyz` health probe with latency.
|
||||
|
||||
> **TODO**: The chat pipeline (`chatCore.ts`) and the in-tree `retrieveMemories()`
|
||||
> implementation do not currently call `upsertSemanticMemoryPoint` or
|
||||
> `searchSemanticMemory`. The Qdrant integration is feature-flagged via
|
||||
> `qdrantEnabled` in settings, but at the time of writing the
|
||||
> `searchSemanticMemory` results are not fused into retrieval — the
|
||||
> `semantic`/`hybrid` retrieval strategies use SQLite FTS5 only. The settings UI
|
||||
> in `dashboard/settings → MemorySkillsTab` exposes Qdrant config, health,
|
||||
> search test, and cleanup, but the corresponding `/api/settings/qdrant`,
|
||||
> `/api/settings/qdrant/health`, `/api/settings/qdrant/search`, and
|
||||
> `/api/settings/qdrant/cleanup` routes are referenced from the UI but **not
|
||||
> present** under `src/app/api/settings/qdrant/` (only `embedding-models/` is
|
||||
> wired). Treat Qdrant as preview/optional plumbing.
|
||||
|
||||
## Memory Types
|
||||
|
||||
`MemoryType` (`src/lib/memory/types.ts`):
|
||||
|
||||
| Type | Used for |
|
||||
| ------------ | ------------------------------------------------------------ |
|
||||
| `factual` | Preferences, stable user facts, behavioral patterns |
|
||||
| `episodic` | Decisions tied to a specific moment ("I chose Postgres") |
|
||||
| `procedural` | Workflow / how-to memory (reserved; no auto-extractor today) |
|
||||
| `semantic` | Reserved for vector-store entries |
|
||||
|
||||
`MemoryConfig` retrieval strategy is one of `exact`, `semantic`, or `hybrid`,
|
||||
and scope is one of `session`, `apiKey`, or `global`. The default scope from
|
||||
`getMemorySettings()` is `apiKey`.
|
||||
|
||||
## Fact Extraction (`extraction.ts`)
|
||||
|
||||
Extraction is **regex-based**, not LLM-based — it runs in-process with
|
||||
`setImmediate()` so it never blocks the response stream:
|
||||
|
||||
- **Preference patterns** → `MemoryType.FACTUAL`
|
||||
(e.g. `I prefer …`, `I really like …`, `my favorite is …`, `I hate …`)
|
||||
- **Decision patterns** → `MemoryType.EPISODIC`
|
||||
(e.g. `I'll use …`, `I chose …`, `I went with …`, `I'm going to adopt …`)
|
||||
- **Pattern patterns** → `MemoryType.FACTUAL`
|
||||
(e.g. `I usually …`, `I always …`, `I tend to …`)
|
||||
|
||||
Each match is sanitised (`trim`, whitespace-collapse, capped at 500 chars),
|
||||
deduplicated within the batch via a stable `factKey(category, content)`, and
|
||||
stored via `createMemory()` with metadata
|
||||
`{category, extractedAt, source: "llm_response"}`. Input text is capped at
|
||||
64 KiB (`MAX_EXTRACTION_TEXT_LENGTH`) — when longer, the **tail** of the text
|
||||
is used so the most recent assistant content always participates.
|
||||
|
||||
`extractFactsFromText(text)` is exported for tests and returns the structured
|
||||
facts without storing them.
|
||||
|
||||
## Retrieval (`retrieval.ts`)
|
||||
|
||||
`retrieveMemories(apiKeyId, config)` is the main entry point. It:
|
||||
|
||||
1. Normalises and validates the config through `MemoryConfigSchema`.
|
||||
2. Returns `[]` immediately when `enabled` is false or `maxTokens <= 0`.
|
||||
3. Clamps `maxTokens` to `[1, 8000]`.
|
||||
4. Detects whether the modern `memories` table exists (vs the legacy `memory`
|
||||
table) so older databases keep working.
|
||||
5. Builds the base query with expiry guard
|
||||
(`expires_at IS NULL OR datetime(expires_at) > datetime('now')`), optional
|
||||
session scope, and optional `retentionDays` cutoff.
|
||||
6. Branches on strategy:
|
||||
- **`exact`** (default): chronological `ORDER BY created_at DESC LIMIT 100`.
|
||||
- **`semantic`**: if `config.query` and `memory_fts` exists, JOIN
|
||||
`memory_fts MATCH ?` and order by FTS rank; fall back to chronological
|
||||
when FTS returns 0 rows.
|
||||
- **`hybrid`**: union of FTS results (higher relevance) and the
|
||||
chronological set, deduplicated by id.
|
||||
7. Computes a keyword relevance score (`getRelevanceScore`) over
|
||||
`content`, `key`, and `metadata` JSON when a query is provided. Rows with
|
||||
zero score are filtered out.
|
||||
8. Sorts by score desc, then `createdAt` desc.
|
||||
9. Walks the ranked list and accepts entries while a running
|
||||
`estimateTokens(content)` (≈ `length / 4`) stays under the budget. Always
|
||||
returns at least one entry when any matched.
|
||||
|
||||
`estimateTokens` is exported and used by retrieval, summarisation, and the MCP
|
||||
`omniroute_memory_search` tool.
|
||||
|
||||
## Injection (`injection.ts`)
|
||||
|
||||
`injectMemory(request, memories, provider)`:
|
||||
|
||||
1. Joins all memory contents into a single `Memory context: …` string.
|
||||
2. Picks a strategy by provider name:
|
||||
- **System message** (default for OpenAI, Anthropic, Gemini, …) — prepends
|
||||
a `{role: "system", content: memoryText}` ahead of any existing system
|
||||
messages so user system prompts still take precedence.
|
||||
- **User message** (fallback) — for providers in
|
||||
`PROVIDERS_WITHOUT_SYSTEM_MESSAGE`: `o1`, `o1-mini`, `o1-preview`,
|
||||
`glm`, `glmt`, `glm-cn`, `zai`, `qianfan`. These reject the system role
|
||||
and would 400 otherwise (cf. issue #1701 for GLM/Zhipu).
|
||||
3. Logs the count, strategy, and model under `memory.injection.injected`.
|
||||
|
||||
`providerSupportsSystemMessage(provider)` is exported for callers that need to
|
||||
make routing decisions of their own. Unknown providers default to `true`
|
||||
(system role allowed) for safety.
|
||||
|
||||
## Settings (`settings.ts`)
|
||||
|
||||
Memory configuration is **stored in the DB settings table**, not in env vars.
|
||||
`getMemorySettings()` reads from `getSettings()` and caches the result
|
||||
in-process; `invalidateMemorySettingsCache()` is called by the settings PUT
|
||||
route after writes.
|
||||
|
||||
| DB key | Type | Default | UI control |
|
||||
| --------------------- | ------- | -------------------------------------------------- | ----------------------------------------------- |
|
||||
| `memoryEnabled` | boolean | `true` | Memory on/off |
|
||||
| `memoryMaxTokens` | integer | `2000` (range `0–16000`) | Token budget for injection |
|
||||
| `memoryRetentionDays` | integer | `30` (range `1–365`) | Retention window |
|
||||
| `memoryStrategy` | enum | `"hybrid"` (one of `recent`, `semantic`, `hybrid`) | Retrieval strategy |
|
||||
| `skillsEnabled` | boolean | `false` | Toggles per-key skill injection (see SKILLS.md) |
|
||||
|
||||
Note: the UI strategy `"recent"` maps to the internal `"exact"` retrieval
|
||||
strategy via `toMemoryRetrievalConfig()` (chronological order).
|
||||
|
||||
Qdrant-related DB keys (`qdrantEnabled`, `qdrantHost`, `qdrantPort`,
|
||||
`qdrantApiKey`, `qdrantCollection` default `"omniroute_memory"`,
|
||||
`qdrantEmbeddingModel` default `"openai/text-embedding-3-small"`) are read by
|
||||
`normalizeQdrantConfig()` in `qdrant.ts`.
|
||||
|
||||
No `MEMORY_*` or `QDRANT_*` env vars exist today — everything is per-instance
|
||||
DB settings. `OMNIROUTE_MEMORY_MB` (commented out in `.env.example`) is
|
||||
unrelated and refers to Node heap sizing.
|
||||
|
||||
## Summarisation (`summarization.ts`)
|
||||
|
||||
`summarizeMemories(apiKeyId, sessionId?, maxTokens = 4000)` compacts older
|
||||
content when the running token total over a key's memories exceeds the
|
||||
budget. It iterates rows DESC by `created_at`, keeps rows that fit, and for
|
||||
the rest replaces `content` in place with the first three sentences of the
|
||||
original. `tokensSaved` is the difference in `estimateTokens` between old and
|
||||
new content.
|
||||
|
||||
This routine is **available but not called automatically** in the current
|
||||
chat pipeline — call it from a cron, an admin action, or
|
||||
`MemoryConfig.autoSummarize` glue if you need ongoing compaction. The data
|
||||
loss is one-way: original text is overwritten.
|
||||
|
||||
## REST API
|
||||
|
||||
All endpoints require management auth (`requireManagementAuth`).
|
||||
|
||||
| Method | Path | Description |
|
||||
| -------- | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `GET` | `/api/memory` | Paginated list with filters: `apiKeyId`, `type`, `sessionId`, `q`, `limit`, `page`, `offset`. Response includes `stats.total` and `stats.byType` |
|
||||
| `POST` | `/api/memory` | Create entry (Zod-validated: `content`, `key`, optional `type`, `sessionId`, `apiKeyId`, `metadata`, `expiresAt`). Calls `createMemory()` which upserts on `(apiKeyId, key)` |
|
||||
| `GET` | `/api/memory/[id]` | Fetch a single entry by UUID |
|
||||
| `DELETE` | `/api/memory/[id]` | Delete an entry; returns 404 when missing |
|
||||
| `GET` | `/api/memory/health` | Runs `verifyExtractionPipeline("health-check")` — round-trip create→list→delete to confirm the store is alive. Returns `{working, latencyMs, error?}` |
|
||||
| `GET` | `/api/settings/memory` | Current normalised `MemorySettings` |
|
||||
| `PUT` | `/api/settings/memory` | Update one or more of `enabled`, `maxTokens`, `retentionDays`, `strategy`, `skillsEnabled` |
|
||||
|
||||
The `/api/memory` list query supports either `page`-based pagination
|
||||
(`parsePaginationParams`) **or** raw `offset` — when `offset` is present it
|
||||
takes precedence and a derived `page` is computed for the response shape.
|
||||
|
||||
## MCP Tools (`open-sse/mcp-server/tools/memoryTools.ts`)
|
||||
|
||||
When the MCP server is enabled, three memory tools are registered:
|
||||
|
||||
- `omniroute_memory_search` — `{apiKeyId, query?, type?, maxTokens?, limit?}`
|
||||
→ wraps `retrieveMemories()` with `retrievalStrategy: "exact"`, optionally
|
||||
filters by `type`, and reports `totalTokens`.
|
||||
- `omniroute_memory_add` — `{apiKeyId, sessionId?, type, key, content,
|
||||
metadata?}` → wraps `createMemory()`.
|
||||
- `omniroute_memory_clear` — `{apiKeyId, type?, olderThan?}` → lists matching
|
||||
entries, optionally filters by created-before timestamp, then deletes each
|
||||
via `deleteMemory()`.
|
||||
|
||||
See [MCP-SERVER.md](./MCP-SERVER.md) for transport and scope details.
|
||||
|
||||
## Dashboard
|
||||
|
||||
`src/app/(dashboard)/dashboard/memory/page.tsx` provides:
|
||||
|
||||
- Real-time list, search, and pagination (debounced 300 ms).
|
||||
- Type filter (`factual` / `episodic` / `procedural` / `semantic` / all).
|
||||
- Add-memory modal (key, content, type).
|
||||
- Delete per row.
|
||||
- JSON export of the current page; JSON import via file picker.
|
||||
- A green/red health dot driven by `GET /api/memory/health`.
|
||||
- Stat cards: `totalEntries`, `tokensUsed`, `hitRate` (the latter two come
|
||||
from the API stats payload).
|
||||
|
||||
Memory and Qdrant settings live under
|
||||
`/dashboard/settings → Memory & Skills` (`MemorySkillsTab.tsx`).
|
||||
|
||||
## Caching
|
||||
|
||||
`src/lib/memory/store.ts` keeps an in-process LRU-ish cache
|
||||
(`MEMORY_CACHE_TTL = 5 min`, `MEMORY_MAX_CACHE_SIZE = 10 000`, with 20 %
|
||||
oldest eviction) for `getMemory(id)` reads, plus a generic key/value
|
||||
`memoryCache` layer (`src/lib/memory/cache.ts`) with `get`/`set`/`invalidate`
|
||||
methods used by callers that want their own scoped cache (1 000-entry LRU,
|
||||
default TTL 5 min).
|
||||
|
||||
## Privacy & Lifecycle
|
||||
|
||||
- Memory ownership is the API key id (`resolveMemoryOwnerId` in
|
||||
`chatCore.ts`). Without an `apiKeyInfo.id` neither retrieval nor injection
|
||||
nor extraction runs.
|
||||
- Entries with a future `expires_at` are filtered out of retrieval; old
|
||||
entries beyond `retentionDays` are excluded by the
|
||||
`created_at >= cutoff` clause in `retrieveMemories`.
|
||||
- For hard deletion, use `DELETE /api/memory/[id]` or `omniroute_memory_clear`.
|
||||
- Extraction is fire-and-forget via `setImmediate`; failures are logged under
|
||||
`memory.extraction.background.failed` and never surface to the caller.
|
||||
- Verification round-trips (`verifyExtractionPipeline`) clean up their own
|
||||
test entries in a `finally` block.
|
||||
|
||||
## See Also
|
||||
|
||||
- [SKILLS.md](./SKILLS.md) — the `skillsEnabled` setting injects tool
|
||||
definitions alongside memory.
|
||||
- [MCP-SERVER.md](./MCP-SERVER.md) — MCP transport / scopes.
|
||||
- [API_REFERENCE.md](../reference/API_REFERENCE.md) — broader API surface.
|
||||
- [Tuto_Qdrant.md](../Tuto_Qdrant.md) — repository-root Qdrant setup tutorial (integration currently dormant — see status banner at top of that file).
|
||||
- Source modules:
|
||||
- `src/lib/memory/types.ts`, `schemas.ts`
|
||||
- `src/lib/memory/store.ts`, `retrieval.ts`, `injection.ts`
|
||||
- `src/lib/memory/extraction.ts`, `summarization.ts`, `verify.ts`
|
||||
- `src/lib/memory/settings.ts`, `qdrant.ts`, `cache.ts`
|
||||
- `src/lib/db/migrations/015_create_memories.sql`,
|
||||
`022_add_memory_fts5.sql`, `023_fix_memory_fts_uuid.sql`
|
||||
- `src/app/api/memory/route.ts`, `[id]/route.ts`, `health/route.ts`
|
||||
- `src/app/api/settings/memory/route.ts`
|
||||
- `open-sse/handlers/chatCore.ts` (injection / extraction wiring)
|
||||
- `open-sse/mcp-server/tools/memoryTools.ts`
|
||||
282
docs/frameworks/SKILLS.md
Normal file
282
docs/frameworks/SKILLS.md
Normal file
@@ -0,0 +1,282 @@
|
||||
# Skills Framework
|
||||
|
||||
> **Source of truth:** `src/lib/skills/` and `src/app/api/skills/`
|
||||
> **Last updated:** 2026-05-13 — v3.8.0
|
||||
|
||||
OmniRoute exposes an extensible Skills framework that lets language models (and operators) compose reusable capabilities — from filesystem reads and HTTP requests to sandboxed code execution and curated marketplace skills.
|
||||
|
||||
A skill is a versioned, schema-defined unit of work. OmniRoute can inject skills as tool definitions into outbound requests, intercept tool calls coming back from the model, run the matching handler, and feed the result back to the model so the conversation can continue. The model never sees the implementation — only the tool interface.
|
||||
|
||||
---
|
||||
|
||||
## Concepts
|
||||
|
||||
### Skill Sources
|
||||
|
||||
Three sources of skills coexist in the same registry:
|
||||
|
||||
1. **Built-in skills** (`src/lib/skills/builtins.ts`) — shipped with OmniRoute. Cover the common cases:
|
||||
- `file_read`, `file_write` — per-API-key sandbox workspace under `<DATA_DIR>/skills/workspaces/<hashed-key>/`
|
||||
- `http_request` — outbound HTTP through `safeOutboundFetch` with `guard: "public-only"`
|
||||
- `web_search` — pluggable search provider with caching (`executeWebSearch`)
|
||||
- `eval_code` — Docker-sandboxed `node` or `python` execution
|
||||
- `execute_command` — Docker-sandboxed shell command
|
||||
- `browser` — Playwright-backed scaffolding, disabled by default (`builtin/browser.ts`)
|
||||
2. **SkillsMP** (the OmniRoute Marketplace) — fetched from `https://skillsmp.com/api/v1/skills/search`. Requires `skillsmpApiKey` in Settings.
|
||||
3. **SkillsSH** (`skills.sh` community catalog) — fetched from `https://skills.sh/api/search`. No auth needed; SKILL.md content pulled from GitHub raw.
|
||||
|
||||
A single "active provider" controls which catalog the dashboard installs from (`src/lib/skills/providerSettings.ts`). Switch it under **Settings → Memory & Skills**. Default: `skillsmp`.
|
||||
|
||||
### Skill Identity
|
||||
|
||||
Skills are keyed by `name@version` in the in-memory registry (`src/lib/skills/registry.ts`). Version must be semver (`^\d+\.\d+\.\d+$`). `resolveVersion()` understands `^`, `~`, `>`, `>=`, `<`, `<=`, `==`, and exact-match constraints.
|
||||
|
||||
### Skill Mode
|
||||
|
||||
Each skill has a runtime mode that controls when it is injected:
|
||||
|
||||
| Mode | Behavior |
|
||||
| ------ | ------------------------------------------------------------------------------------------ |
|
||||
| `on` | Always injected as a tool definition |
|
||||
| `off` | Never injected, never executable |
|
||||
| `auto` | Scored against the incoming request; injected only if score ≥ `AUTO_MIN_SCORE` (default 3) |
|
||||
|
||||
`auto` is the default for marketplace-installed skills. `enabled=true` and `mode="off"` together mean "registered but inactive" — toggling `enabled` via the legacy column also bumps `mode` so older codepaths stay consistent (`src/app/api/skills/[id]/route.ts`).
|
||||
|
||||
### Status (executions)
|
||||
|
||||
Skill executions are tracked in the `skill_executions` table with the following statuses (`src/lib/skills/types.ts`):
|
||||
|
||||
```ts
|
||||
enum SkillStatus {
|
||||
PENDING = "pending",
|
||||
RUNNING = "running",
|
||||
SUCCESS = "success",
|
||||
ERROR = "error",
|
||||
TIMEOUT = "timeout",
|
||||
}
|
||||
```
|
||||
|
||||
### Registry Cache
|
||||
|
||||
`SkillRegistry` is a singleton with a 60-second TTL cache (`registry.ts:14`). `loadFromDatabase()` is idempotent and dedupes concurrent calls via `pendingLoad`. Any write (`register`/`unregister`/`unregisterById`) invalidates the cache. Look up versions via `getSkillVersions(name)` and `resolveVersion(name, constraint)`.
|
||||
|
||||
### Provider-Aware Injection
|
||||
|
||||
`injectSkills()` in `src/lib/skills/injection.ts` is the entry point that turns registered skills into provider-specific tool definitions:
|
||||
|
||||
- **OpenAI** — `{ type: "function", function: { name, description, parameters } }`
|
||||
- **Anthropic** — `{ name, description, input_schema }`
|
||||
- **Google (Gemini)** — `{ name, description, parameters }`
|
||||
|
||||
The tool name is encoded as `name@version` so the handler can pick the right version when the model calls it back.
|
||||
|
||||
### AUTO Scoring
|
||||
|
||||
When `mode="auto"`, each candidate skill is scored against the request context (`scoreAutoSkill()` in `injection.ts`):
|
||||
|
||||
| Signal | Points |
|
||||
| ---------------------------------------------- | ------------ |
|
||||
| Skill name appears verbatim in context | +6 |
|
||||
| Each name token matches a context token | +2 |
|
||||
| Each tag substring matches context | +3 |
|
||||
| Each description token matches context | +1 |
|
||||
| Background reason matches a name token | +2 per token |
|
||||
| Background reason matches a tag | +2 per token |
|
||||
| Provider hint in tags matches request provider | +2 / −2 |
|
||||
|
||||
Top `AUTO_MAX_SKILLS = 5` skills with `score >= AUTO_MIN_SCORE = 3` are injected. Ties are broken by `installCount` (desc), then alphabetical name (`injection.ts:225-235`).
|
||||
|
||||
### Tool Call Interception
|
||||
|
||||
`handleToolCallExecution()` in `src/lib/skills/interception.ts` is invoked by the chat handler after the upstream returns a tool-calling response:
|
||||
|
||||
1. `extractToolCalls()` reads provider-specific shapes (OpenAI `tool_calls` / Responses `function_call`, Anthropic `tool_use`, Gemini `functionCalls`).
|
||||
2. Built-in tool aliases (e.g. `omniroute_web_search` → `web_search`) are resolved first. Built-in handlers run inline.
|
||||
3. Anything else routes through `skillExecutor.execute(name@version, args, { apiKeyId, sessionId })`.
|
||||
4. Results are spliced back into the response — `tool_results`, `function_call_output` items, or Anthropic `tool_result` blocks as appropriate.
|
||||
|
||||
`customSkillExecutionEnabled` in the execution context can be set to `false` to allow only built-in interception (used by request paths that explicitly disable user-defined handlers).
|
||||
|
||||
---
|
||||
|
||||
## Docker Sandbox
|
||||
|
||||
Non-builtin code paths (`eval_code`, `execute_command`) run inside Docker via `SandboxRunner` (`src/lib/skills/sandbox.ts`). Every container is launched with:
|
||||
|
||||
```
|
||||
--rm --network none|bridge --cap-drop ALL
|
||||
--security-opt no-new-privileges --pids-limit 100
|
||||
--cpus <cpuLimit/1000> --memory <memoryLimit>m
|
||||
--tmpfs /tmp:rw,noexec,nosuid,size=64m
|
||||
--tmpfs /workspace:rw,noexec,nosuid,size=64m
|
||||
--read-only (when readOnly=true)
|
||||
```
|
||||
|
||||
Defaults (`SandboxRunner.DEFAULT_CONFIG`):
|
||||
|
||||
| Field | Default | Notes |
|
||||
| ---------------- | --------------- | ---------------------------------------------------- |
|
||||
| `cpuLimit` | 100 (= 0.1 CPU) | Divided by 1000 before passing to `--cpus` |
|
||||
| `memoryLimit` | 256 MB | Hard limit |
|
||||
| `timeout` | 30000 ms | Soft kill via `SIGTERM` + `docker kill` |
|
||||
| `networkEnabled` | `false` | Becomes `--network none` |
|
||||
| `readOnly` | `true` | Root FS read-only; `/tmp` and `/workspace` are tmpfs |
|
||||
|
||||
`SandboxRunner.kill(id)` and `killAll()` are exposed for shutdown; running containers are tracked in `runningContainers: Map<string, ChildProcess>`.
|
||||
|
||||
### Sandbox Env Vars
|
||||
|
||||
Configured via `process.env` in `src/lib/skills/builtins.ts`:
|
||||
|
||||
| Env Var | Default | Purpose |
|
||||
| --------------------------------- | ---------------- | ------------------------------------------------------------------ |
|
||||
| `SKILLS_MAX_FILE_BYTES` | `1048576` (1 MB) | Cap for `file_read` and `file_write` |
|
||||
| `SKILLS_MAX_HTTP_RESPONSE_BYTES` | `256000` | Cap for `http_request` response body |
|
||||
| `SKILLS_MAX_SANDBOX_OUTPUT_CHARS` | `100000` | Cap for stdout/stderr returned to the caller |
|
||||
| `SKILLS_SANDBOX_TIMEOUT_MS` | `10000` | Default timeout for sandboxed commands; capped at 60 s |
|
||||
| `SKILLS_SANDBOX_NETWORK_ENABLED` | `false` | Master gate for egress. Set `1` or `true` to allow per-call opt-in |
|
||||
| `SKILLS_ALLOWED_SANDBOX_IMAGES` | (see below) | Comma-separated allowlist of Docker images |
|
||||
|
||||
Default allowed images: `alpine:3.20`, `node:22-alpine`, `python:3.12-alpine`. Any additions via `SKILLS_ALLOWED_SANDBOX_IMAGES` are merged with the defaults; unknown images are rejected by `normalizeImage()`.
|
||||
|
||||
> Note: there is no separate `SKILLS_EXECUTION_TIMEOUT_MS` env var. The non-sandbox handler timeout is hard-coded to 30 s in `SkillExecutor` (`executor.ts:13`) but can be overridden at runtime via `skillExecutor.setTimeout(ms)`.
|
||||
|
||||
### Workspace Isolation
|
||||
|
||||
`file_read` and `file_write` resolve every path relative to a per-API-key workspace at `<DATA_DIR>/skills/workspaces/<sha256(apiKeyId).slice(0,24)>/`. Path traversal (`..`) and forbidden segments (`.env`, `.git`, `.ssh`, `.omniroute`, `.codex`, `secrets`) are rejected before any disk I/O.
|
||||
|
||||
### HTTP Hardening
|
||||
|
||||
`http_request` (`builtins.ts:257`):
|
||||
|
||||
- Method allowlist: `GET, HEAD, POST, PUT, PATCH, DELETE`
|
||||
- Blocked outbound headers: `host, connection, content-length, cookie, set-cookie, authorization, proxy-authorization`
|
||||
- Redirects disabled (`allowRedirect: false`)
|
||||
- Routed through `safeOutboundFetch` with `guard: "public-only"` (private/loopback ranges blocked)
|
||||
- Response truncated at `SKILLS_MAX_HTTP_RESPONSE_BYTES`; client sees `truncated: true`
|
||||
|
||||
---
|
||||
|
||||
## Hybrid Executor (preview)
|
||||
|
||||
`src/lib/skills/hybrid.ts` defines a `HybridExecutor` that decides between `direct` (in-process) and `sandbox` execution per call, with an `autoUpgrade` retry path on timeout/memory errors. The wired-in `directExecutor` / `sandboxRunner` implementations are stubs (`executeDirect`, `executeInSandbox` return placeholder objects) — treat this module as a contract under construction. Real execution still goes through `skillExecutor` + `SandboxRunner`.
|
||||
|
||||
---
|
||||
|
||||
## Storage
|
||||
|
||||
Schema lives in two migrations:
|
||||
|
||||
- `src/lib/db/migrations/016_create_skills.sql` — base `skills` and `skill_executions` tables, with indexes on `(api_key_id, name)` and `(skill_id, status, created_at)`.
|
||||
- `src/lib/db/migrations/027_skill_mode_and_metadata.sql` — adds `mode`, `source_provider`, `tags` (JSON), `install_count` to `skills`.
|
||||
|
||||
`skill_executions.status` is constrained at the database level: `CHECK(status IN ('pending', 'running', 'success', 'error', 'timeout'))`.
|
||||
|
||||
---
|
||||
|
||||
## REST API
|
||||
|
||||
All endpoints live under `src/app/api/skills/`. Management endpoints (`/api/skills`, `/api/skills/[id]`, `/api/skills/install`) require **management auth** via `requireManagementAuth()`. The marketplace/install flows use the lighter `isAuthenticated()` (session or API key).
|
||||
|
||||
| Endpoint | Method | Purpose |
|
||||
| --------------------------------- | ------ | ------------------------------------------------------------------------ | --- | ------------------------ | -------- | ------------------ |
|
||||
| `/api/skills` | GET | List registered skills. Supports `?q=`, `?mode=on | off | auto`, `?source=skillsmp | skillssh | local`, pagination |
|
||||
| `/api/skills/[id]` | PUT | Update `enabled` or `mode` |
|
||||
| `/api/skills/[id]` | DELETE | Unregister by id |
|
||||
| `/api/skills/install` | POST | Install a custom skill (handler code + schema) |
|
||||
| `/api/skills/marketplace` | GET | Search the SkillsMP catalog (returns popular defaults when `q` is empty) |
|
||||
| `/api/skills/marketplace/install` | POST | Install a SkillsMP skill (requires active provider = `skillsmp`) |
|
||||
| `/api/skills/skillssh` | GET | Search the skills.sh catalog (`?q=&limit=`, capped at 100) |
|
||||
| `/api/skills/skillssh/install` | POST | Install a skills.sh skill (requires active provider = `skillssh`) |
|
||||
| `/api/skills/executions` | GET | Paginated execution history (`?apiKeyId=`) |
|
||||
| `/api/skills/executions` | POST | Execute a registered skill ad-hoc |
|
||||
|
||||
The `POST /api/skills/executions` endpoint returns HTTP `503` with `{ error: "Skills execution is disabled..." }` when `settings.skillsEnabled === false` (`executor.ts:42-45`). Operators can flip the master switch from **Settings → AI**.
|
||||
|
||||
### Example: install a custom skill
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/api/skills/install \
|
||||
-H "Authorization: Bearer $OMNIROUTE_MGMT_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "reverse-text",
|
||||
"version": "1.0.0",
|
||||
"description": "Reverses a string",
|
||||
"schema": {
|
||||
"input": { "type": "object", "properties": { "text": { "type": "string" } }, "required": ["text"] },
|
||||
"output": { "type": "object", "properties": { "reversed": { "type": "string" } } }
|
||||
},
|
||||
"handlerCode": "echo-handler",
|
||||
"apiKeyId": "your-api-key-id"
|
||||
}'
|
||||
```
|
||||
|
||||
The `handlerCode` string is a **handler name lookup** — not executable code. The executor maps it via `skillExecutor.registerHandler(name, fn)` (`executor.ts:25`). Marketplace installs store the SKILL.md text in this field as documentation and route execution through model-generated tool calls. Arbitrary user-supplied source is not eval'd.
|
||||
|
||||
---
|
||||
|
||||
## MCP Tools
|
||||
|
||||
Four MCP tools wrap the skills surface (`open-sse/mcp-server/tools/skillTools.ts`). They are auto-registered when the MCP server boots.
|
||||
|
||||
| Tool | Description |
|
||||
| ----------------------------- | ------------------------------------------------------------ |
|
||||
| `omniroute_skills_list` | List skills, optional filters: `apiKeyId`, `name`, `enabled` |
|
||||
| `omniroute_skills_enable` | Enable/disable a skill by `skillId` |
|
||||
| `omniroute_skills_execute` | Execute a skill with an input payload |
|
||||
| `omniroute_skills_executions` | Recent execution history (default 50, max 100) |
|
||||
|
||||
See [MCP-SERVER.md](./MCP-SERVER.md) for transport setup and scope assignments.
|
||||
|
||||
---
|
||||
|
||||
## A2A Integration
|
||||
|
||||
`src/lib/skills/a2a.ts` exports the `memory_aware_routing` A2A skill descriptor and a `registerA2ASkill(registry)` helper. Custom A2A skills live in `src/lib/a2a/skills/` and are dispatched via `A2A_SKILL_HANDLERS` (`src/lib/a2a/taskExecution.ts`). See [A2A-SERVER.md](./A2A-SERVER.md) for the full task lifecycle.
|
||||
|
||||
---
|
||||
|
||||
## Adding a New Built-in Skill
|
||||
|
||||
1. **Define the handler** in `src/lib/skills/builtins.ts` (or a sibling file under `src/lib/skills/builtin/`). Signature: `(input, { apiKeyId, sessionId }) => Promise<output>`.
|
||||
2. **Sandboxed code path?** Call `sandboxRunner.run(image, command, env, sandboxConfig({...}))`. Use `normalizeImage()` against the allowlist.
|
||||
3. **Filesystem path?** Always pass through `resolveWorkspacePath(input, context)` before touching disk.
|
||||
4. **Network call?** Use `safeOutboundFetch` with `guard: "public-only"`; sanitize headers via `sanitizeHeaders()`.
|
||||
5. **Register** by adding the entry to `builtinSkills` (or calling `registerBrowserSkill(executor)`-style at boot).
|
||||
6. **Wire built-in tool aliases** (optional) in `BUILTIN_TOOL_ALIASES` (`interception.ts:23`) if the upstream model emits a different name.
|
||||
7. **Tests** in `src/lib/skills/__tests__/` (Vitest).
|
||||
|
||||
---
|
||||
|
||||
## Adding a Custom (Non-Builtin) Skill
|
||||
|
||||
1. Register the handler at process startup:
|
||||
```ts
|
||||
skillExecutor.registerHandler("my-handler", async (input, ctx) => { ... });
|
||||
```
|
||||
2. Insert the skill via `POST /api/skills/install` (the `handlerCode` field must match the registered handler name).
|
||||
3. Toggle `mode` to `on` or `auto` via `PUT /api/skills/[id]`.
|
||||
|
||||
---
|
||||
|
||||
## Operational Tips
|
||||
|
||||
- **Master switch:** `settings.skillsEnabled = false` blocks all execution and returns HTTP `503` on `/api/skills/executions`. The registry continues to load.
|
||||
- **Lock down egress:** keep `SKILLS_SANDBOX_NETWORK_ENABLED` unset (default) for fully air-gapped sandboxing. Per-call `networkEnabled: true` still requires the master gate.
|
||||
- **Allow specific images:** set `SKILLS_ALLOWED_SANDBOX_IMAGES="myorg/sandbox:1.0,node:22-alpine"` to extend the allowlist.
|
||||
- **Audit executions:** `/dashboard/skills/executions` and `omniroute_skills_executions` both query `skill_executions`. Successful runs include `durationMs`; failures include `errorMessage`.
|
||||
- **Cache invalidation:** call `skillRegistry.invalidateCache()` after manual DB edits; otherwise wait 60 s.
|
||||
- **Anonymous workspace:** when `apiKeyId` is empty, all calls hash to the same `"anonymous"` workspace — share-aware code should always pass a real key.
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [MCP-SERVER.md](./MCP-SERVER.md) — MCP tool registration and transports
|
||||
- [A2A-SERVER.md](./A2A-SERVER.md) — A2A task lifecycle and skill dispatch
|
||||
- [USER_GUIDE.md](../guides/USER_GUIDE.md#-skills-system) — user-facing introduction
|
||||
- [ARCHITECTURE.md](../architecture/ARCHITECTURE.md) — request pipeline and component map
|
||||
- Source: `src/lib/skills/`, `src/app/api/skills/`, `open-sse/mcp-server/tools/skillTools.ts`
|
||||
- Tests: `src/lib/skills/__tests__/integration.test.ts`
|
||||
253
docs/frameworks/WEBHOOKS.md
Normal file
253
docs/frameworks/WEBHOOKS.md
Normal file
@@ -0,0 +1,253 @@
|
||||
# Webhooks
|
||||
|
||||
> **Source of truth:** `src/lib/webhookDispatcher.ts`, `src/lib/db/webhooks.ts`, `src/app/api/webhooks/`
|
||||
> **Last updated:** 2026-05-13 — v3.8.0
|
||||
|
||||
OmniRoute can fire HTTP webhooks on platform events. Use them to integrate with
|
||||
Slack, PagerDuty, Datadog, internal alerting services, or any HTTP receiver.
|
||||
|
||||
The dispatcher signs each delivery with HMAC-SHA256, retries on transient
|
||||
failures, tracks delivery health per webhook, and auto-disables endpoints that
|
||||
keep failing.
|
||||
|
||||
## Supported Events
|
||||
|
||||
The `WebhookEvent` type (`src/lib/webhookDispatcher.ts`) currently models:
|
||||
|
||||
| Event | Fires when |
|
||||
| -------------------- | --------------------------------------------------------- |
|
||||
| `request.completed` | A proxied request completes successfully |
|
||||
| `request.failed` | A proxied request fails after all retries/fallback |
|
||||
| `provider.error` | A provider returns an error eligible for circuit-breaking |
|
||||
| `provider.recovered` | A previously failing provider returns to a healthy state |
|
||||
| `quota.exceeded` | An API key crosses a budget/quota threshold |
|
||||
| `combo.switched` | A combo strategy switches its primary target |
|
||||
| `test.ping` | Synthetic event used by the test endpoint |
|
||||
|
||||
Subscriptions accept the literal `"*"` to receive every event. Unknown event
|
||||
names in `events` are ignored at dispatch time.
|
||||
|
||||
> Note: the dispatcher API is wired, but production call sites for some of the
|
||||
> non-`test.ping` events are still landing. Check `grep dispatchEvent` to see
|
||||
> which paths currently invoke the dispatcher in your release.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Caller (handler, service, monitor)
|
||||
dispatchEvent(event, data) [src/lib/webhookDispatcher.ts]
|
||||
-> getEnabledWebhooks() [src/lib/db/webhooks.ts]
|
||||
-> filter by webhook.events
|
||||
-> for each match (in parallel):
|
||||
deliverWebhook(url, payload, secret)
|
||||
build payload { event, timestamp, data }
|
||||
sign body with HMAC-SHA256 (if secret present)
|
||||
POST with 10s timeout
|
||||
retry up to 3 times on 5xx / network error
|
||||
recordWebhookDelivery(id, status, success)
|
||||
-> disableWebhooksWithHighFailures(10)
|
||||
```
|
||||
|
||||
Dispatch is fire-and-forget for the caller: `Promise.allSettled` swallows
|
||||
per-webhook errors so one bad receiver cannot block the others.
|
||||
|
||||
## HMAC Signing
|
||||
|
||||
When a webhook has a `secret`, OmniRoute signs the JSON body and sends:
|
||||
|
||||
```
|
||||
Content-Type: application/json
|
||||
User-Agent: OmniRoute-Webhook/1.0
|
||||
X-Webhook-Event: <event>
|
||||
X-Webhook-Timestamp: <ISO-8601>
|
||||
X-Webhook-Signature: sha256=<hex HMAC-SHA256(secret, body)>
|
||||
```
|
||||
|
||||
> Header names use the `X-Webhook-*` prefix (not `X-OmniRoute-*`). The signature
|
||||
> value is `sha256=<hex>` — verify the full prefix.
|
||||
|
||||
If `createWebhook` is called without a secret, the DB module generates one
|
||||
(`whsec_<48 hex>`) so all webhooks are signed by default.
|
||||
|
||||
### Verifying on the receiver
|
||||
|
||||
```typescript
|
||||
import { createHmac, timingSafeEqual } from "node:crypto";
|
||||
|
||||
function verify(rawBody: string, signature: string, secret: string) {
|
||||
const expected = "sha256=" + createHmac("sha256", secret).update(rawBody).digest("hex");
|
||||
const a = Buffer.from(expected);
|
||||
const b = Buffer.from(signature);
|
||||
return a.length === b.length && timingSafeEqual(a, b);
|
||||
}
|
||||
```
|
||||
|
||||
Always verify against the **raw** request body, before any JSON parsing.
|
||||
|
||||
## Retry & Failure Policy
|
||||
|
||||
`deliverWebhook(url, payload, secret, maxRetries = 3)`:
|
||||
|
||||
- 10 second timeout per attempt (`AbortController`).
|
||||
- HTTP 2xx counts as success.
|
||||
- HTTP 3xx/4xx counts as a non-retryable final status — recorded as delivered
|
||||
with `success = res.ok`.
|
||||
- HTTP 5xx and network errors are retried with exponential backoff:
|
||||
`2^attempt * 1000 ms` (1s, 2s, 4s).
|
||||
- After `maxRetries`, the delivery is recorded as failed.
|
||||
- Each delivery updates `last_triggered_at`, `last_status`, and either resets
|
||||
or increments `failure_count`.
|
||||
- The dispatcher calls `disableWebhooksWithHighFailures(10)` after each fan-out,
|
||||
so any webhook with `failure_count >= 10` is automatically disabled.
|
||||
|
||||
## Database
|
||||
|
||||
Table `webhooks` (migration `011_webhooks.sql`):
|
||||
|
||||
| Column | Type | Notes |
|
||||
| ------------------- | ------- | --------------------------------------------- |
|
||||
| `id` | TEXT PK | UUID |
|
||||
| `url` | TEXT | Destination URL |
|
||||
| `events` | TEXT | JSON array; default `["*"]` |
|
||||
| `secret` | TEXT | HMAC secret (auto-generated if not given) |
|
||||
| `enabled` | INT | 0/1; defaults to 1 |
|
||||
| `description` | TEXT | Optional human label |
|
||||
| `created_at` | TEXT | `datetime('now')` |
|
||||
| `last_triggered_at` | TEXT | Updated on every delivery attempt |
|
||||
| `last_status` | INT | HTTP status of the last attempt (0 = network) |
|
||||
| `failure_count` | INT | Resets to 0 on success, +1 on failure |
|
||||
|
||||
There is **no separate `webhook_deliveries` table** in the current schema —
|
||||
delivery history is aggregated on the `webhooks` row. If you need full audit
|
||||
history, consume `request.completed` / `audit` style events from a downstream
|
||||
log store.
|
||||
|
||||
## REST API
|
||||
|
||||
All endpoints require management auth (`requireManagementAuth`).
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
| ------------------------- | ------ | ------------------------------- |
|
||||
| `/api/webhooks` | GET | List webhooks (secrets masked) |
|
||||
| `/api/webhooks` | POST | Create webhook |
|
||||
| `/api/webhooks/[id]` | GET | Webhook detail (full secret) |
|
||||
| `/api/webhooks/[id]` | PUT | Update fields |
|
||||
| `/api/webhooks/[id]` | DELETE | Remove |
|
||||
| `/api/webhooks/[id]/test` | POST | Fire a `test.ping` (no retries) |
|
||||
|
||||
`GET /api/webhooks` masks the secret to `<first 10 chars>...` to avoid leaking
|
||||
on listing pages. Use the `[id]` GET when you actually need the secret.
|
||||
|
||||
### Create webhook
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/api/webhooks \
|
||||
-H "Cookie: auth_token=..." \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"url": "https://hooks.slack.com/services/...",
|
||||
"secret": "whsec_my_shared_secret",
|
||||
"events": ["quota.exceeded", "provider.error"],
|
||||
"description": "Slack alerts"
|
||||
}'
|
||||
```
|
||||
|
||||
If `secret` is omitted, the server generates a `whsec_<hex>` secret and returns
|
||||
it in the response.
|
||||
|
||||
### Test webhook
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/api/webhooks/<id>/test \
|
||||
-H "Cookie: auth_token=..."
|
||||
```
|
||||
|
||||
Returns `{ delivered, status, error }`. No retries are attempted — useful for
|
||||
quickly validating that the receiver accepts the payload and signature.
|
||||
|
||||
## Dashboard
|
||||
|
||||
The dashboard page at `/dashboard/webhooks` (see
|
||||
`src/app/(dashboard)/dashboard/webhooks/page.tsx`) provides:
|
||||
|
||||
- Create/edit webhooks with an event picker
|
||||
- Status indicator (active / inactive / errored) based on `enabled`,
|
||||
`failure_count`, and `last_status`
|
||||
- One-click test delivery
|
||||
- Manual enable/disable toggle
|
||||
|
||||
## Payload Examples
|
||||
|
||||
### request.completed
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "request.completed",
|
||||
"timestamp": "2026-05-13T20:30:00.123Z",
|
||||
"data": {
|
||||
"trace_id": "...",
|
||||
"api_key_id": "...",
|
||||
"provider": "openai",
|
||||
"model": "gpt-5",
|
||||
"status": 200,
|
||||
"tokens_in": 142,
|
||||
"tokens_out": 350,
|
||||
"cost_usd": 0.0042
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### provider.error
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "provider.error",
|
||||
"timestamp": "2026-05-13T20:31:00.000Z",
|
||||
"data": {
|
||||
"provider": "anthropic",
|
||||
"status": 503,
|
||||
"consecutive_failures": 5,
|
||||
"circuit_state": "open"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### test.ping
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "test.ping",
|
||||
"timestamp": "2026-05-13T20:32:00.000Z",
|
||||
"data": {
|
||||
"message": "Test webhook delivery from OmniRoute",
|
||||
"webhookId": "<uuid>"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Field shapes for non-`test.ping` events are defined by the call sites that emit
|
||||
them; treat the `data` object as forward-compatible (add fields, don't depend on
|
||||
absence).
|
||||
|
||||
## Best Practices
|
||||
|
||||
- **Verify the signature on every delivery** against the raw body — prevents
|
||||
spoofed POSTs from anyone who guesses your webhook URL.
|
||||
- **Respond 2xx within ~5 seconds** — the dispatcher times out at 10 s. Slow
|
||||
receivers will eat retries and inflate `failure_count`.
|
||||
- **Make handlers idempotent** — retries and at-least-once delivery semantics
|
||||
mean duplicates are possible.
|
||||
- **Subscribe minimally** — list only events you actually consume; `"*"` will
|
||||
add cost on receivers you do not control.
|
||||
- **Watch `failure_count`** — endpoints are auto-disabled at 10 consecutive
|
||||
failures; reset by calling `PUT /api/webhooks/[id]` with `enabled: true`
|
||||
after fixing the receiver.
|
||||
- **Rotate secrets periodically** — `PUT` a new `secret`, deploy the new value
|
||||
to the receiver, and confirm via the test endpoint.
|
||||
|
||||
## See Also
|
||||
|
||||
- [API_REFERENCE.md](../reference/API_REFERENCE.md) — full management API surface
|
||||
- [RESILIENCE_GUIDE.md](../architecture/RESILIENCE_GUIDE.md) — circuit breaker / cooldown
|
||||
semantics that drive `provider.error` / `provider.recovered`
|
||||
- Source: `src/lib/webhookDispatcher.ts`, `src/lib/db/webhooks.ts`
|
||||
Reference in New Issue
Block a user