pjandro/OmniRoute
1
0
Fork 0
mirror of https://github.com/diegosouzapw/OmniRoute.git synced 2026-07-26 09:52:11 +03:00
Code Issues Packages Projects Releases Wiki Activity

docs: add relay backend strategy guide (#5533)

Browse Source
This commit is contained in:
KooshaPari
2026-06-29 22:16:09 -07:00
committed by GitHub
parent df2752b1ca
commit baa5ad5023
2 changed files with 65 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/reference/ENVIRONMENT.md
View File

@@ -859,6 +859,8 @@ Allow users to report issues directly from the Dashboard.
## Deployment Scenarios
For relay backend SRE guidance (ts/bifrost/auto behavior, 9router vs CLIProxyAPI placement, and high-throughput fallback strategy), see [Relay Backend Strategy](/docs/reference/RELAY_BACKEND_STRATEGY.md).
### Minimal Local Development
```bash

63
docs/reference/RELAY_BACKEND_STRATEGY.md Normal file
View File

@@ -0,0 +1,63 @@
---
title: "Relay Backend Strategy"
version: 3.8.42
lastUpdated: 2026-06-30
---
# Relay Backend Strategy
## Summary
OmniRoute now supports three relay modes for `/api/v1/relay/chat/completions`:
- `ts`: Use the TypeScript relay in-process.
- `bifrost`: Force the Bifrost gateway.
- `auto`: Prefer Bifrost when available, fall back to TypeScript on failure.
When you are running at high request rate (large number of tokens/day, near-constant throughput), the best strategy is to keep the fallback path explicit and fast so the hot path never blocks on a dead sidecar.
## Mode behavior
- `ts`
- Lowest operational complexity.
- All routing and validation runs in Node.
- No sidecar dependency for availability.
- `bifrost`
- Force all requests through the sidecar gateway.
- No automatic fallback.
- Useful only when sidecar health and latency are guaranteed.
- `auto`
- Sidecar is used when it can be reached and is enabled.
- Failed attempts trigger fallback headers and return traffic to TS so request success remains bounded.
- This mode is the safest choice for production when uptime matters more than strict sidecar-only routing.
## 9router vs CLIPROXYAPI today
9router and CLIPROXYAPI are both integrations that historically exposed compatibility paths for upstream providers.
- 9router is an embedded path for upstream orchestration and compatibility behavior.
- CLIPROXYAPI is a proxy API bridge for CLI / SDK style traffic.
- Bifrost is being stabilized as the externalized path when you need a dedicated sidecar-like hop and low-latency local dispatch.
If you are currently comparing 9router/CLIPROXYAPI:
- Keep request signing, allowlist checks, and DB policy gates in the API route before handoff.
- If a workflow needs strict sidecar behavior and lower per-request variance, use `OMNIROUTE_RELAY_BACKEND=bifrost`.
- If you need sidecar resilience with graceful degradation under incident conditions, use `OMNIROUTE_RELAY_BACKEND=auto`.
## High-throughput guidance
For sustained high RPM/RPS and strict success SLO:
1. Use `auto` with a practical cooldown and failure telemetry.
2. Keep upstream validation and API-key checks on the TypeScript route boundary.
3. Enable explicit headers/counters so your alerting sees fallback frequency and reasons.
4. Tune sidecar timeouts to fail fast, not to wait forever.
5. Keep service auto-restart and health telemetry loop healthy so fallback is truly exceptional.
## Suggested baseline
- `OMNIROUTE_RELAY_BACKEND=auto`
- `BIFROST_ENABLED=1`
- Keep API keys, allowlist, sanitizer, and rate-limit checks enabled in route handlers (they always run before downstream forwarding).
- Export fallback metrics from your reverse proxy and request logs so sidecar outages are visible within one minute.