pjandro/OmniRoute
1
0
Fork 0
mirror of https://github.com/diegosouzapw/OmniRoute.git synced 2026-09-20 22:02:19 +03:00
Code Issues Packages Projects Releases Wiki Activity

feat(routing): deterministic routing strategies for self-hosted entry (RIC-740) (#13611)

Browse Source 
* feat(routing): self-hosted unified OpenAI-compatible entry (RIC-738)

Divert /v1/chat/completions through the self-hosted provider adapters when
OMNIROUTE_SELF_HOSTED_PROVIDERS / OMNIROUTE_SELF_HOSTED_PROVIDERS_FILE is set:
one OpenAI-compatible contract in, auto-route to the selected provider
(x-omniroute-provider header, provider/model prefix, or first provider),
standard OpenAI error shape out. Optional OMNIROUTE_SELF_HOSTED_API_KEY guards
the entry (D5 reserved); unset = open loopback route. Upstream credentials stay
runtime-only and are stripped from echoed responses.

Brings in the provider-adapters baseline from sibling branch (RIC-737) that
this entry depends on. Includes 21 passing unit tests (provider selection,
model-prefix forwarding, header hygiene, auth, error normalization, SSE
passthrough, fall-through/misconfig), docs, env example, changelog fragment.

* feat(routing): deterministic routing strategies for self-hosted entry (RIC-740)

Add the M2 deterministic routing strategy engine (D3 可审计路由) to the
self-hosted unified entry: a declarative `strategy:` block expressing five
explainable, non-predictive policies — blacklist/whitelist hard filters,
cooldown circuit breaker, cost-priority, latency-aware ordering, and an
explicit fallback chain. The ordered candidate list is the fallback chain:
a failed primary (network or non-2xx) falls through to the next candidate and
each failure feeds the breaker. Every response carries an
x-omniroute-route-decision header answering "why this model / why not that
one". A pinned provider rejected by a hard filter returns 400 (never a silent
re-route); no eligible providers returns 503 with the full explainable
decision. No ML/predict dependency.

Covers the RIC-740 acceptance: 5 strategy types with unit tests + HTTP
fault-injection tests (primary down -> fallback works), config matching docs,
and no predict/ML deps. Adds docs, .env.example entries, and a changelog
fragment.

* refactor(routing): reduce complexity-ratchet violations in new self-hosted routing files

Extract cost/id validation, pin-blocked resolution, ordering, and env/file
source resolution into small helpers so routingStrategies.ts and
selfHostedEntry.ts stay under the complexity-ratchets cap. No behavior
change — the same 51 routing-strategies/self-hosted-entry/provider-adapters
tests pass unmodified.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* docs(routing): document the 5 self-hosted env vars in ENVIRONMENT.md

check:env-doc-sync failed because OMNIROUTE_SELF_HOSTED_PROVIDERS(_FILE),
OMNIROUTE_SELF_HOSTED_API_KEY and OMNIROUTE_SELF_HOSTED_STRATEGY(_FILE)
were present in .env.example but missing from
docs/reference/ENVIRONMENT.md. Add them under "6. Tool & Routing Policies".

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Co-authored-by: Ant Rich <ant@richants.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Co-authored-by: luyuehm <luyuehm@users.noreply.github.com>
This commit is contained in:
luyuehm
2026-09-18 22:59:15 +08:00
committed by GitHub
parent 3ebea07278
commit 5a82da7084
13 changed files with 2282 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
docs/reference/ENVIRONMENT.md
View File

@@ -310,6 +310,11 @@ OmniRoute provides a two-layer defense: request-side injection scanning and resp
| `OMNIROUTE_PREFER_CLAUDE_CODE_FOR_UNPREFIXED_CLAUDE_MODELS` | `false` | `open-sse/services/model.ts` | Opt-in: route bare `claude-*` model IDs from Claude Code clients through the Claude Code OAuth account instead of requiring a provider prefix. Explicit provider prefixes still win. Also configurable via a dashboard toggle on the Claude provider page. |
| `COMBO_CONCURRENCY_PER_MODEL` | `3` | `open-sse/services/comboConfig.ts` | Per-model concurrency cap for round-robin combos (#9100). The round-robin combo semaphore was hard-capped at 3 concurrent requests per model with no override, serializing higher-concurrency traffic behind that cap. Validated to `>= 1`, clamped to `<= 32`. |
| `DISABLE_CONTEXT_WINDOW_CHECKS` | `false` | `open-sse/handlers/chatCore.ts` | Dangerous opt-in that skips OmniRoute's local context-window / max-input-token check for direct single-model requests. Upstream providers still enforce their actual limits; prompt compression and the model's own output-token cap remain active. Effective precedence is Feature Flags DB override > environment variable > default; no restart is required. |
| `OMNIROUTE_SELF_HOSTED_PROVIDERS` | _(unset)_ | `open-sse/services/selfHostedEntry.ts` | Inline YAML `providers:` document (RIC-738, D4). When set (with or without a `strategy:` block), `/v1/chat/completions` diverts to the self-hosted unified OpenAI-compatible entry instead of the cloud pipeline. Unset (the default): the route falls straight through to the existing cloud pipeline. See `docs/routing/SELF_HOSTED_OPENAI_ENTRY.md`. |
| `OMNIROUTE_SELF_HOSTED_PROVIDERS_FILE` | _(unset)_ | `open-sse/services/selfHostedEntry.ts` | Path to a YAML file holding the same `providers:` document as `OMNIROUTE_SELF_HOSTED_PROVIDERS`, for deployments that prefer a file over an inline env var. Either one activates the self-hosted entry. |
| `OMNIROUTE_SELF_HOSTED_API_KEY` | _(unset — open route)_ | `open-sse/services/selfHostedEntry.ts` | Optional shared API key for the unified self-hosted entry (D5 scaffold, reserved for the per-key quota system). When set, requests must include `Authorization: Bearer <key>`. Unset: the route is open, matching the existing self-hosted local-provider pattern (loopback/trusted-network deployment). |
| `OMNIROUTE_SELF_HOSTED_STRATEGY` | _(unset)_ | `open-sse/services/routingStrategies.ts` | Inline YAML `strategy:` document for the deterministic routing engine (M2/RIC-740, D3) — blacklist/whitelist, cooldown breaker, cost-priority, latency-aware, fallback chain. Overrides an inline `strategy:` block nested inside `OMNIROUTE_SELF_HOSTED_PROVIDERS`, per-key. See `docs/routing/DETERMINISTIC_ROUTING.md`. |
| `OMNIROUTE_SELF_HOSTED_STRATEGY_FILE` | _(unset)_ | `open-sse/services/routingStrategies.ts` | Path to a YAML file holding the same `strategy:` document as `OMNIROUTE_SELF_HOSTED_STRATEGY`, for deployments that prefer a file over an inline env var. |
| `OMNIROUTE_DISABLE_CONVERSATION_TRACKING` | _(unset)_ | `open-sse/services/conversationTracker.ts` | Set `1` to stop collecting conversation history. `resolveConversationId()` returns an untracked result before it reads SQLite or parses message history, and client-supplied session IDs are covered too. Routing-session handling is unchanged and existing records are not deleted. For deployments that do not use the dashboard's conversation view and want the turn table to stop growing. |
---

124
docs/routing/DETERMINISTIC_ROUTING.md Normal file
View File

@@ -0,0 +1,124 @@
---
title: "Deterministic Routing Strategies"
---
# Deterministic routing strategies
OmniRoute's self-hosted gateway entry (`/v1/chat/completions`, RIC-738) routes to a
single provider by default. When you configure a `strategy:` block, the route decision
becomes a **deterministic, explainable policy** — the OmniRoute differentiator vs
NotDiamond/Martian "predict" black-box routing.
Every decision is:
- **Deterministic** — the same config + same runtime state yields the same provider.
- **Config-expressed** — one rule per line; no "一体化智能体", no learned model.
- **Explainable** — every response carries `x-omniroute-route-decision`, a one-line
answer to "why this model?" (and "why NOT that one").
This is M2 of the RIC-697 differentiation (D3 可审计路由). There is **no predictive /
ML dependency** — the strategies are pure rules over observable signals
(consecutive failures, declared cost, measured latency).
## Enable
Add a `strategy:` block to the same YAML document as `providers:`, or point
`OMNIROUTE_SELF_HOSTED_STRATEGY` (inline YAML) / `OMNIROUTE_SELF_HOSTED_STRATEGY_FILE`
(a file) at a standalone `strategy:` document. An explicit env strategy merges over
the inline block per-key.
```yaml
# providers.yaml
providers:
- id: cheap
kind: openai
baseUrl: http://127.0.0.1:11434/v1
model: llama3
costPer1MInput: 0.2 # USD per 1M input tokens — used by cost-priority
- id: fast
kind: openai
baseUrl: http://127.0.0.1:8080/v1
model: gpt-4o-mini
costPer1MInput: 0.6
- id: premium
kind: anthropic
baseUrl: http://127.0.0.1:8081
model: claude-sonnet
costPer1MInput: 3.0
strategy:
blacklist: [] # provider ids never used
whitelist: [cheap, fast, premium] # when non-empty, ONLY these are used
costPriority: true # cheapest eligible candidate first
latencyAware:
enabled: true # fastest recent-average candidate first
cooldown:
consecutiveFailures: 2 # breaker trips after this many in a row
cooldownMs: 30000 # …and the provider stays excluded this long
fallbackChain: [cheap, fast, premium] # explicit fallback order
```
## The five strategies
| Strategy | Config | Effect |
| ---------------- | --------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| Blacklist | `blacklist: [id, ...]` | Listed providers are never candidates. |
| Whitelist | `whitelist: [id, ...]` | Non-empty → only listed providers are candidates. |
| Cooldown/breaker | `cooldown: {consecutiveFailures, cooldownMs}` | After N consecutive failures a provider is excluded for the window. A success resets the counter. |
| Cost-priority | `costPriority: true` | Eligible candidates sorted by `costPer1MInput` ascending (declared, never fabricated). |
| Latency-aware | `latencyAware.enabled: true` | Eligible candidates sorted by recent average request latency ascending. Unsampled providers sort last. |
| Fallback chain | `fallbackChain: [id, ...]` | Explicit primary→backup order. Wins over cost/latency ordering. |
Filters (blacklist / whitelist / cooldown) run first and remove candidates. Then the
ordering stage sorts the survivors: explicit `fallbackChain` wins; otherwise
`costPriority` then `latencyAware` apply in that order.
## Pinned providers
The `x-omniroute-provider` header and the `provider/model` model-prefix are **explicit
pins**, not preferences. If a pinned provider is excluded by a hard filter (blacklist /
whitelist / active cooldown), the request fails with `400` and the pin reason — the
gateway never silently re-routes a client who asked for a specific provider. If the pin
survives the filters it is the first candidate, and the remaining candidates serve as
its fallback chain.
## Fallback on failure
The ordered candidate list is the fallback chain. When the first provider fails
(network unreachable, connection refused, DNS/TLS, or any non-2xx), the gateway walks
the next candidate, and so on. Each failure is recorded into the cooldown breaker and
each attempt is latency-sampled — so a broken primary also cools down for _subsequent_
requests. When every candidate fails, the last normalized error (OpenAI shape) is
returned. When no candidate is eligible at all, a `503` with the full explainable
decision is returned.
## Explainability
Every response routed through the strategy engine carries:
```
x-omniroute-route-decision: cheap(cost-priority: cheap #1) -> fast(cost-priority: fast #2); premium excluded: blacklist: premium forbidden
```
The value is the ordered candidate list plus every exclusion reason — the audit trail
for "why this model, why not that one".
## Failure contract (unchanged from RIC-738)
- Upstream non-2xx body is normalized through `parseUpstreamError` + `buildErrorBody`
into the standard OpenAI `error` shape (Hard Rule #12).
- Network-level failures return a normalized `502`; after the whole chain is exhausted,
the last normalized error is surfaced.
- Credential / session headers echoed upstream are stripped from every response.
- Provider config with a malformed `strategy:` block returns `500` — a misconfigured
policy never silently becomes a no-op.
## Tests
- `tests/unit/routing-strategies.test.ts` — 25 unit tests over the pure strategy engine:
parse, blacklist/whitelist, cooldown breaker, cost-priority, latency-aware, fallback
chain, combined pipeline, explainability.
- `tests/unit/self-hosted-entry.test.ts` — fault injection over real HTTP: primary down
→ fallback succeeds; all-down → last error; pinned-blacklisted → 400; cooldown
excludes a failing provider on the next request; no-eligible → 503 with explainable
decision.

93
docs/routing/SELF_HOSTED_OPENAI_ENTRY.md Normal file
View File

@@ -0,0 +1,93 @@
---
title: "Self-Hosted OpenAI-Compatible Entry"
---
# Self-hosted unified OpenAI-compatible entry
When enabled, OmniRoute's existing `/v1/chat/completions` (and the OpenAI-compatible
contract it serves) becomes a **self-hosted gateway**: one OpenAI-compatible request
in, auto-routed to the provider of your choice through the provider adapters, with a
standard OpenAI error shape out. Client code does not change.
This is the D4 (接入即用) differentiator from the RIC-697 design: the same `/v1` path
the OpenAI SDK already targets, backed by your own providers instead of a single
vendored catalog.
## Enable
Set either env var (see `.env.example` for both):
- `OMNIROUTE_SELF_HOSTED_PROVIDERS` — inline YAML document (runtime-only, not logged).
- `OMNIROUTE_SELF_HOSTED_PROVIDERS_FILE` — path to a YAML file.
```yaml
# providers.yaml
providers:
- id: local
kind: openai # openai | anthropic | local
baseUrl: http://127.0.0.1:11434/v1
model: llama3
# apiKey: sk-... # optional, runtime-only
- id: claude
kind: anthropic
baseUrl: http://127.0.0.1:8080
model: claude-sonnet
```
When **either** var is set, the unified entry is active for every request to
`/v1/chat/completions`. Config present but unparseable returns `500` (it never
silently falls through to cloud routing). When neither is set, the route behaves
exactly as before.
## Provider auto-route
Precedence (deterministic, no predictive model):
1. `x-omniroute-provider: <id>` header — exact provider id.
2. `model` prefix: `provider/model` (slash) or `provider::model` (double colon).
3. First configured provider.
The routing prefix is stripped before forwarding — upstream receives the bare model
(`claude-sonnet`, not `claude/claude-sonnet`).
```bash
# via header
curl http://localhost:20128/v1/chat/completions \
-H "x-omniroute-provider: claude" \
-H "content-type: application/json" \
-d '{"model":"claude-sonnet","messages":[{"role":"user","content":"hi"}]}'
# or via model prefix
curl ... -d '{"model":"claude/claude-sonnet","messages":[...]}'
```
## Auth scaffold (D5 reserved)
Optional `OMNIROUTE_SELF_HOSTED_API_KEY`. When set, requests must carry
`Authorization: Bearer <key>`. Unset = open route (loopback / trusted network),
mirroring how OmniRoute's existing local providers work. The D5 quota/quota-自治
key system is expected to take over this header.
## Failure contract
- Upstream non-2xx body is normalized through `parseUpstreamError` + `buildErrorBody`
into the standard OpenAI `error` shape (Hard Rule #12 — never raw upstream text).
- Network-level failures (connection refused / DNS / TLS) return a normalized `502`
with `error.message: "Upstream provider unreachable: …"`.
- Credential / session headers echoed upstream are stripped from every response.
## Tests
`tests/unit/self-hosted-entry.test.ts` covers provider selection, model-prefix
forwarding, header hygiene, auth, upstream-error normalization, SSE streaming
passthrough, and the fall-through / misconfig paths — all over real HTTP against a
local upstream.
## Notes
- Self-hosted models bypass the cloud-only retirement / alias machinery by design:
the divert happens before those cloud checks, so ids like `local/llama3` never
trip cloud-peer 410s or alias rewrites.
- Deterministic routing strategies (fallback / cooldown / cost / latency / blacklist)
are M2's scope (`RIC-740`) and inject into the gateway layer here. Configure them
with a `strategy:` block — see [DETERMINISTIC_ROUTING.md](./DETERMINISTIC_ROUTING.md).