From baa5ad50233874fce1287365f472d6101b427d4a Mon Sep 17 00:00:00 2001
From: KooshaPari <42529354+KooshaPari@users.noreply.github.com>
Date: Mon, 29 Jun 2026 22:16:09 -0700
Subject: [PATCH] docs: add relay backend strategy guide (#5533)

---
 docs/reference/ENVIRONMENT.md            |  2 +
 docs/reference/RELAY_BACKEND_STRATEGY.md | 63 ++++++++++++++++++++++++
 2 files changed, 65 insertions(+)
 create mode 100644 docs/reference/RELAY_BACKEND_STRATEGY.md

diff --git a/docs/reference/ENVIRONMENT.md b/docs/reference/ENVIRONMENT.md
index 8c2f7043e9..ee62da5052 100644
--- a/docs/reference/ENVIRONMENT.md
+++ b/docs/reference/ENVIRONMENT.md
@@ -859,6 +859,8 @@ Allow users to report issues directly from the Dashboard.
 
 ## Deployment Scenarios
 
+For relay backend SRE guidance (ts/bifrost/auto behavior, 9router vs CLIProxyAPI placement, and high-throughput fallback strategy), see [Relay Backend Strategy](/docs/reference/RELAY_BACKEND_STRATEGY.md).
+
 ### Minimal Local Development
 
 ```bash
diff --git a/docs/reference/RELAY_BACKEND_STRATEGY.md b/docs/reference/RELAY_BACKEND_STRATEGY.md
new file mode 100644
index 0000000000..e497d98bf0
--- /dev/null
+++ b/docs/reference/RELAY_BACKEND_STRATEGY.md
@@ -0,0 +1,63 @@
+---
+title: "Relay Backend Strategy"
+version: 3.8.42
+lastUpdated: 2026-06-30
+---
+
+# Relay Backend Strategy
+
+## Summary
+
+OmniRoute now supports three relay modes for `/api/v1/relay/chat/completions`:
+
+- `ts`: Use the TypeScript relay in-process.
+- `bifrost`: Force the Bifrost gateway.
+- `auto`: Prefer Bifrost when available, fall back to TypeScript on failure.
+
+When you are running at high request rate (large number of tokens/day, near-constant throughput), the best strategy is to keep the fallback path explicit and fast so the hot path never blocks on a dead sidecar.
+
+## Mode behavior
+
+- `ts`
+  - Lowest operational complexity.
+  - All routing and validation runs in Node.
+  - No sidecar dependency for availability.
+- `bifrost`
+  - Force all requests through the sidecar gateway.
+  - No automatic fallback.
+  - Useful only when sidecar health and latency are guaranteed.
+- `auto`
+  - Sidecar is used when it can be reached and is enabled.
+  - Failed attempts trigger fallback headers and return traffic to TS so request success remains bounded.
+  - This mode is the safest choice for production when uptime matters more than strict sidecar-only routing.
+
+## 9router vs CLIPROXYAPI today
+
+9router and CLIPROXYAPI are both integrations that historically exposed compatibility paths for upstream providers.
+
+- 9router is an embedded path for upstream orchestration and compatibility behavior.
+- CLIPROXYAPI is a proxy API bridge for CLI / SDK style traffic.
+- Bifrost is being stabilized as the externalized path when you need a dedicated sidecar-like hop and low-latency local dispatch.
+
+If you are currently comparing 9router/CLIPROXYAPI:
+
+- Keep request signing, allowlist checks, and DB policy gates in the API route before handoff.
+- If a workflow needs strict sidecar behavior and lower per-request variance, use `OMNIROUTE_RELAY_BACKEND=bifrost`.
+- If you need sidecar resilience with graceful degradation under incident conditions, use `OMNIROUTE_RELAY_BACKEND=auto`.
+
+## High-throughput guidance
+
+For sustained high RPM/RPS and strict success SLO:
+
+1. Use `auto` with a practical cooldown and failure telemetry.
+2. Keep upstream validation and API-key checks on the TypeScript route boundary.
+3. Enable explicit headers/counters so your alerting sees fallback frequency and reasons.
+4. Tune sidecar timeouts to fail fast, not to wait forever.
+5. Keep service auto-restart and health telemetry loop healthy so fallback is truly exceptional.
+
+## Suggested baseline
+
+- `OMNIROUTE_RELAY_BACKEND=auto`
+- `BIFROST_ENABLED=1`
+- Keep API keys, allowlist, sanitizer, and rate-limit checks enabled in route handlers (they always run before downstream forwarding).
+- Export fallback metrics from your reverse proxy and request logs so sidecar outages are visible within one minute.