mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-08-22 15:12:23 +03:00
d87b97a786
Obrigado — feature substancial e bem estruturada: separa qualidade operacional (comportamento de wire: 4xx/5xx, 429, respostas malformadas, stream interrompido) de qualidade semântica (só setada por avaliadores externos, nunca inferida do sucesso HTTP), com confidence/sample-awareness para não deixar poucos sucessos de sorte dominarem o ranking. Instrumentação de streaming (TTFT/ITL) threaded até RoutingEvent, endpoint de explicabilidade, e teste E2E determinístico cobrindo degradação→recuperação→blip. Validação (worktree própria a partir de origin/release/v3.8.50, merge limpo, 0 conflitos): - typecheck:core limpo, complexity/cognitive-complexity dentro do baseline - 59/59 testes passando (mlx-provider, routing-adaptive-e2e, routing-events(-concurrency), routing-otel, routing-quality, routing-scoring-quality, stream-timing, auto-combo-scoring-clamp)
351 lines
18 KiB
Markdown
351 lines
18 KiB
Markdown
---
|
||
title: "Adaptive Routing: Routing Events, Quality Feedback & Explainability"
|
||
version: 3.8.50
|
||
lastUpdated: 2026-08-20
|
||
---
|
||
|
||
# Adaptive Routing: Routing Events, Quality Feedback & Explainability
|
||
|
||
This document describes the feedback-driven adaptive routing foundation added to
|
||
OmniRoute. It is deliberately small: it introduces a typed routing-outcome
|
||
channel, an online quality signal that feeds the existing auto-combo scorer, an
|
||
optional OpenTelemetry exporter, and an explainability endpoint. It does **not**
|
||
replace the existing resilience stack (circuit breaker, connection cooldown,
|
||
model lockout, health matrix, autopilot) — it complements it.
|
||
|
||
## 1. Architectural context
|
||
|
||
OmniRoute is a data plane with a **request hot path** and a **control/intelligence
|
||
plane**. The hot path must stay fast, memory-efficient, asynchronous, resilient and
|
||
predictable. Evaluation, quality scoring, experiments and historical analysis belong
|
||
to the control plane.
|
||
|
||
```
|
||
AI Agent / IDE
|
||
│
|
||
▼
|
||
┌─────────────────────┐
|
||
│ OmniRoute │ data plane (fast, sync, in-memory)
|
||
│ routing / failover │
|
||
│ health / guardrail │
|
||
│ cache / streaming │
|
||
└──────────┬──────────┘
|
||
│ RoutingEvent (fire-and-forget, ~0.2µs)
|
||
▼
|
||
┌─────────────────────┐
|
||
│ Feedback sinks │ control plane (async, best-effort)
|
||
│ quality tracker │
|
||
│ OTel exporter │
|
||
│ explain store │
|
||
└──────────┬──────────┘
|
||
▼ quality score
|
||
auto-combo scorer
|
||
```
|
||
|
||
### What was already there (audited, not duplicated)
|
||
|
||
| Concept | Existing implementation |
|
||
| ----------------------------------- | -------------------------------------------------------------------------------------------------- |
|
||
| Availability (can we send traffic?) | Circuit breaker (CLOSED/DEGRADED/OPEN/HALF_OPEN, DB-persisted), connection cooldown, model lockout |
|
||
| Health reporting | `providerHealthMatrix.ts`, `providerHealthAutopilot.ts` |
|
||
| Shadow traffic | `open-sse/services/combo/shadowRouting.ts` |
|
||
| Guardrails | `src/lib/guardrails/` (pre/post hooks) |
|
||
| Exact cache | `src/lib/semanticCache.ts` (signature-based) |
|
||
| Evaluators / eval-driven routing | `src/lib/evals/`, `open-sse/services/evalRouting.ts` |
|
||
| Combo decision explainability | `open-sse/services/combo/decisionTrace.ts` |
|
||
| Dashboard real-time events | `src/lib/events/eventBus.ts` (UI notification channel, `unknown` payloads, 100-entry history) |
|
||
|
||
The routing-event layer is **not** a re-implementation of `eventBus`: that bus is
|
||
the dashboard's real-time notification channel (typed _event names_, opaque
|
||
payloads, UI consumers). `RoutingEvent` is a typed _outcome_ struct
|
||
(latency/tokens/cost/outcome/finish-reason) consumed by the control plane's
|
||
feedback sinks (quality tracker, OTel exporter, explain store).
|
||
|
||
### What was missing (added here)
|
||
|
||
1. A **typed routing-outcome event + sink abstraction** (`RoutingEvent` /
|
||
`RoutingEventSink`). `decisionTrace` is combo-scoped and in-memory-only;
|
||
`comboMetrics` are cumulative counters; `call_logs` is raw async persistence.
|
||
None is a typed, sink-based outcome channel that a quality tracker, an OTel
|
||
exporter, or a Future-AGI-style evaluator can subscribe to.
|
||
2. An **online quality signal** (EWMA) for output quality — the scorer previously
|
||
proxied "quality" only through static task fitness and opt-in eval pass-rates.
|
||
3. An **optional, dependency-free OTel exporter** using GenAI semantic conventions.
|
||
4. An **explainability endpoint** returning the real routing decisions + quality state.
|
||
|
||
## 2. Routing Events (feedback foundation)
|
||
|
||
Files: `open-sse/services/routing/events.ts`, `.../index.ts`
|
||
|
||
A `RoutingEvent` carries only routing metadata:
|
||
|
||
```ts
|
||
interface RoutingEvent {
|
||
requestId: string;
|
||
provider: string;
|
||
model: string;
|
||
strategy: string; // "auto" | "priority" | "direct" | ...
|
||
latencyMs: number;
|
||
ttftMs: number | null;
|
||
inputTokens: number | null;
|
||
outputTokens: number | null;
|
||
cost: number | null;
|
||
retries: number;
|
||
fallbackUsed: boolean;
|
||
outcome: RoutingOutcome; // allowlisted union
|
||
status: number | null;
|
||
finishReason: string | null;
|
||
connectionId: string | null;
|
||
ts: number;
|
||
}
|
||
```
|
||
|
||
`RoutingEventSink` is a `Send+Sync`-style trait in TypeScript:
|
||
|
||
```ts
|
||
interface RoutingEventSink {
|
||
readonly name: string;
|
||
record(event: RoutingEvent): void; // must be O(1), no sync I/O
|
||
}
|
||
```
|
||
|
||
The hot path calls `emitRoutingEvent(event)` once per completed request
|
||
(the streaming-completion callback, the non-streaming success path, and the
|
||
malformed-200 failure path in `handleChatCore`). Dispatch is synchronous fan-out
|
||
to registered sinks, but each sink only enqueues/updates in-memory state. **No
|
||
synchronous database writes, no network I/O on the hot path.**
|
||
|
||
Default sinks:
|
||
|
||
- `MemoryRoutingEventStore` — bounded (500) ring buffer, newest-first, for the
|
||
explain endpoint.
|
||
- `QualityTracker` consumer — updates the EWMA quality estimate.
|
||
- `OtlpHttpsEventSink` — optional, enabled only when `OMNIROUTE_OTEL_ENDPOINT`
|
||
(or `OTEL_EXPORTER_OTLP_ENDPOINT`) is set.
|
||
|
||
### Measured overhead (honest comparison)
|
||
|
||
`npm run bench:routing-events` on this workstation (100k iterations; sub-µs ops
|
||
measured as aggregate µs/op because per-op percentiles are below
|
||
`performance.now()` timer resolution):
|
||
|
||
| Scenario | µs/op | ops/s |
|
||
| --------------------------------- | ------ | ------ |
|
||
| baseline (scoring only) | ~0.045 | ~22 M |
|
||
| baseline + RoutingEvent (2 sinks) | ~0.168 | ~5.9 M |
|
||
| baseline + event + OTel enqueue | ~0.163 | ~6.1 M |
|
||
| concurrent (8 interleaved bursts) | ~0.18 | — |
|
||
|
||
The event-dispatch delta over baseline scoring is ~0.12 µs/request; the OTel sink
|
||
only enqueues (O(1) buffer push), adding nothing measurable. These numbers are
|
||
machine-specific and relative — not a production guarantee. The v1 "~0.2 µs"
|
||
figure was an aggregate estimate; this methodology separates the scoring baseline
|
||
from the event-dispatch cost.
|
||
|
||
## 3. Quality Signal (feedback-driven provider state)
|
||
|
||
Files: `open-sse/services/routing/quality.ts`
|
||
|
||
v2 separates **operational** from **semantic** quality:
|
||
|
||
- **Operational** — derived from the routing hot path (HTTP 4xx/5xx, connection
|
||
failures, 429s, malformed responses, stream interruptions, `finish_reason=length`,
|
||
zero-output successes, latency/TTFT EWMA). A 200 is NOT treated as semantic
|
||
quality.
|
||
- **Semantic** — the actual value of the generated output. ONLY ever produced by
|
||
an evaluator via `setSemanticQuality()`. It is `null` until one provides it and
|
||
never leaks into the operational score.
|
||
|
||
Per-(provider, model) state (EWMA + bounded counters):
|
||
|
||
- `successEwma` — EWMA (α=0.2) of outcome success.
|
||
- `latencyEwma` / `ttftEwma` — EWMA of latency (α=0.1).
|
||
- `samples`, `anomalies`, `rateLimited`, `semantic`, `semanticConfidence`.
|
||
- `recencyMs` — how recently the model was last observed.
|
||
|
||
### Confidence / sample awareness
|
||
|
||
`confidence = clamp01(samples / 50)`, and the score returned to the scorer is
|
||
blended toward the neutral midpoint:
|
||
|
||
```
|
||
score = 0.5 + confidence * (operational - 0.5)
|
||
```
|
||
|
||
Consequences (verified by tests):
|
||
|
||
- A cold provider (0 samples) scores **0.5** — not unfairly penalized, but
|
||
unable to dominate a provider with thousands of solid observations.
|
||
- A provider with 7 lucky successes is pulled toward 0.5 (never dominates from
|
||
optimistic initialization).
|
||
- A provider with 50+ samples converges to its true operational score.
|
||
- Degradation and recovery are gradual (EWMA), and one isolated failure does
|
||
not destroy a healthy provider.
|
||
|
||
`ProviderQuality` exposes `{ operational, semantic, confidence, samples, anomalies,
|
||
rateLimited, successEwma, latencyEwmaMs, ttftEwmaMs, recencyMs }`.
|
||
|
||
This feeds the auto-combo scorer as the `quality` scoring factor:
|
||
|
||
- `ScoringFactors.quality` / `ScoringWeights.quality` in
|
||
`open-sse/services/autoCombo/scoring.ts`.
|
||
- `DEFAULT_WEIGHTS`: `health` 0.1905 → 0.1605, `quality` 0.03. Sum stays 1.0.
|
||
- `buildAutoCandidates` populates `candidate.quality` from the tracker; candidates
|
||
without data default to neutral **0.5** (a cold candidate is neither boosted nor
|
||
penalized).
|
||
|
||
The closed loop:
|
||
|
||
```
|
||
RoutingEvent → QualityTracker → getQualityScore → auto-combo quality factor
|
||
↑ │
|
||
└────── request outcome (handleChatCore) ←────────────┘
|
||
```
|
||
|
||
### Hard exclusion vs soft penalty
|
||
|
||
The quality signal is a **soft adaptive preference** only. Hard exclusion stays
|
||
with the existing resilience stack: circuit breaker OPEN, quota exhausted,
|
||
auth failure, model lockout — none of these are affected by the quality score.
|
||
A provider whose quality score dips temporarily is de-preferenced, never
|
||
hard-disabled.
|
||
|
||
## 3b. Canonical stream timing (TTFT / ITL)
|
||
|
||
Files: `open-sse/utils/streamTiming.ts`
|
||
|
||
`createStreamTiming()` is the single instrumentation seam for the streaming path,
|
||
wired into `createSSEStream` (open-sse/utils/stream.ts):
|
||
|
||
- `markByte()` — first upstream chunk received.
|
||
- `markForward()` — first chunk forwarded to the client (used for TTFT).
|
||
- `markInterrupted()` — stream timeout/abort/error before a clean finish.
|
||
- `ttft()` = first-forwarded-SSE-chunk latency. **This is NOT token-level TTFT** —
|
||
a single SSE chunk may carry zero/one/many tokens. Documented precisely.
|
||
- `avgItlMs()` = mean inter-chunk gap (a chunk-latency proxy for ITL).
|
||
|
||
TTFT/ITL/interrupted flow into the `RoutingEvent` (`ttftMs`, `itlMs`) and are
|
||
exported as GenAI/OmniRoute span attributes by the OTel sink.
|
||
|
||
## 4. OpenTelemetry / GenAI observability
|
||
|
||
Files: `open-sse/services/routing/otel.ts`
|
||
|
||
- Dependency-free OTLP/HTTP JSON exporter (uses global `fetch`, no
|
||
`@opentelemetry/*` SDK).
|
||
- Spans follow GenAI semantic conventions (`gen_ai.provider.name`,
|
||
`gen_ai.request.model`, `gen_ai.usage.input_tokens/output_tokens`,
|
||
`gen_ai.completion.finish_reason`, `gen_ai.system`) plus OmniRoute routing
|
||
attributes (outcome, status, ttft, retries, fallback).
|
||
- `record()` only enqueues into a bounded buffer (O(1)); a background timer
|
||
flushes via `POST {endpoint}/v1/traces` asynchronously. Under overload the
|
||
oldest events are dropped (`dropped` counter) — never backpressure the data
|
||
plane.
|
||
- **Disabled unless configured.** `OMNIROUTE_OTEL_ENDPOINT` (or
|
||
`OTEL_EXPORTER_OTLP_ENDPOINT`) must be set; otherwise the sink is not
|
||
registered and zero OTel code runs.
|
||
|
||
## 5. Explainability
|
||
|
||
- `GET /v1/explain/routing` returns the recent `RoutingEvent`s (the real
|
||
decisions, newest first) and the per-provider/model quality snapshot.
|
||
- Auth mirrors `/v1/combos` (Bearer API key or dashboard session; anonymous on
|
||
single-user local deployments with `REQUIRE_API_KEY=false`).
|
||
- Combo-level per-invocation traces remain available via the existing
|
||
`decisionTrace.ts` (header `X-OmniRoute-Combo-Trace`).
|
||
- Safety: events carry only routing metadata, never prompts/bodies/credentials.
|
||
|
||
## 6. Evaluation-plane integration (Future AGI readiness)
|
||
|
||
OmniRoute treats Future AGI (or any evaluator) as a **potential
|
||
intelligence/evaluation backend, not a dependency**. The seams:
|
||
|
||
- A `RoutingEventSink` can forward events to an evaluator asynchronously.
|
||
- The `MemoryRoutingEventStore` + quality snapshot give an evaluator the raw
|
||
decision stream.
|
||
- A future `Evaluator` (deterministic, local judge, HTTP, WASM) would consume
|
||
events/traces and return a `QualityScore` that feeds the same
|
||
`getQualityScore`/quality-factor path.
|
||
- Existing eval-driven routing (`open-sse/services/evalRouting.ts`) already
|
||
re-orders combo targets by `eval_runs` pass-rates when enabled.
|
||
|
||
No evaluation runs synchronously on the request path, and the gateway operates
|
||
fully with the evaluator absent.
|
||
|
||
## 7. Final architectural review
|
||
|
||
1. **What remains on the synchronous hot path?** Routing/scoring, guardrail
|
||
pre-checks, cache lookup, and one `emitRoutingEvent` fan-out (~0.12 µs over
|
||
baseline scoring) to in-memory sinks.
|
||
2. **What moved to asynchronous processing?** OTel export (timer + fetch),
|
||
`call_logs`/usage persistence, semantic-cache writes, quality is in-memory
|
||
and O(1) (no async needed).
|
||
3. **How does a routing outcome become feedback?** `handleChatCore` emits a
|
||
`RoutingEvent` → `QualityTracker` updates EWMA state → `getQualityScore`
|
||
feeds the auto-combo `quality` factor.
|
||
4. **How does quality influence future routing?** A low quality score reduces
|
||
the weighted score of that provider/model in `scoreAutoTargets`, so degraded
|
||
models are gradually de-preferenced and recover as their EWMA improves.
|
||
5. **How can Future AGI integrate without becoming a dependency?** Via the
|
||
`RoutingEventSink` interface / a future `Evaluator` adapter — no hardcoded
|
||
dependency.
|
||
6. **What happens when the evaluator is unavailable?** Routing is unaffected;
|
||
quality falls back to neutral (1.0) for models with no observed signal.
|
||
7. **What happens when telemetry is unavailable?** The OTel sink simply isn't
|
||
registered; the rest of the routing layer runs unchanged.
|
||
8. **What happens under overload?** The OTel buffer drops oldest events; quality
|
||
and the ring buffer are bounded by construction; no backpressure.
|
||
9. **How does provider state recover after degradation?** EWMA re-converges as
|
||
successes accumulate; warmup keeps cold models neutral; the circuit breaker
|
||
independently recovers via HALF_OPEN probes.
|
||
10. **Which proposed features were intentionally NOT implemented, and why?**
|
||
- Shadow traffic / experiments — already implemented
|
||
(`combo/shadowRouting.ts`); not re-built.
|
||
- Guardrails — already implemented (`src/lib/guardrails/`); not duplicated.
|
||
- Semantic cache — already implemented (`src/lib/semanticCache.ts`); not
|
||
duplicated.
|
||
- A full experiment-management platform, dataset tooling, prompt-optimization
|
||
platform, vector DB, or mandatory external OTel infrastructure — out of
|
||
scope for a lean data plane.
|
||
- A Rust `RoutingEvent` struct — the data plane is TypeScript; the TS type
|
||
is the adapted equivalent.
|
||
|
||
## 8. Configuration reference
|
||
|
||
| Variable | Default | Effect |
|
||
| ----------------------------- | ----------- | ------------------------------------------------------------------------------- |
|
||
| `OMNIROUTE_OTEL_ENDPOINT` | unset | When set, enables the OTLP/HTTP traces exporter (e.g. `http://collector:4318`). |
|
||
| `OTEL_EXPORTER_OTLP_ENDPOINT` | unset | Fallback alias for the OTLP endpoint. |
|
||
| `OTEL_SERVICE_NAME` | `omniroute` | `service.name` resource attribute. |
|
||
|
||
## 9. Tests
|
||
|
||
- `tests/unit/routing-events.test.ts` — event normalization, status
|
||
classification, bounded ring buffer, sink fan-out + isolation.
|
||
- `tests/unit/routing-quality.test.ts` — EWMA warmup, failure/success recovery,
|
||
anomaly penalties, 429 transient handling, snapshot, reset.
|
||
- `tests/unit/routing-scoring-quality.test.ts` — weight integrity, neutral
|
||
default, quality factor ranking.
|
||
- `tests/unit/routing-otel.test.ts` — enable gating, GenAI span payload, async
|
||
flush, drop-under-overload.
|
||
- `tests/unit/routing-events-concurrency.test.ts` — thousands of events, ring
|
||
buffer boundedness, throwing-sink isolation, interleaved async bursts,
|
||
reset-during-inserts.
|
||
- `tests/unit/routing-adaptive-e2e.test.ts` — deterministic end-to-end loop via
|
||
the real `scoreAutoTargets` scorer: healthy → degrade → recover → blip, plus
|
||
cold-start and lucky-cold-provider scenarios.
|
||
- `tests/unit/stream-timing.test.ts` — TTFT (first-forwarded-chunk), ITL,
|
||
first-byte vs first-forward, interruption, malformed/empty chunk safety.
|
||
|
||
## 10. Pre-existing issues status (Phase 18)
|
||
|
||
| Issue | Status | Notes |
|
||
| ----------------------------------------------------- | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||
| `omniglyph` export mismatch | **FIXED (environmental)** | `node_modules` was out of sync with `package-lock.json` (installed 1.3.1 vs locked 1.4.0). Running `npm install omniglyph@1.4.0` restored the locked version; type errors dropped to 0. Manifests unchanged. |
|
||
| Stale `getKnownContextOverflow` tests | **KNOWN — not fixed** | `combo-context-overflow-compression-probe.test.ts` imports a function that no longer exists in `open-sse/services/combo.ts` (only comments reference it). Fixing requires re-implementing or re-writing those tests — unrelated architectural churn. |
|
||
| `combo-runtime-unit-concurrency.test.ts` DB isolation | **KNOWN — not fixed** | Test-harness SQLite-isolation assertion fails when run directly; fails identically on the base branch. |
|
||
| i18n `llm.txt` drift | **KNOWN — not fixed** | `docs/i18n/*/llm.txt` differ from root; pre-existing, blocks the docs-sync pre-commit gate. |
|
||
|
||
Environmental vs code issues are kept distinct; no unrelated failures are hidden
|
||
behind changed test filters.
|