Compare commits

..

2 Commits

Author SHA1 Message Date
Egor
21c75528a8 docs: add changelog fragment for #10887
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
2026-08-20 17:27:59 -03:00
Egor
9136d84007 fix(memory): enable agent memory save/update via MCP tools + builtin stream guard
- memoryTools: apiKeyId now optional; falls back to caller principal
  (HTTP auth headers / OMNIROUTE_API_KEY env) so agents can store memory
  without knowing their key id
- memorySkillsInjection: server-side memory_* builtins only injected for
  non-stream requests (stream clients execute tools client-side via MCP)
- memoryBuiltins: memory_save/update/search/delete builtin tools with
  per-provider schemas + interception dispatch
- retrieval: fix toFts5MatchQuery import (ReferenceError on FTS5 path)
- tests: MCP auto-owner fallback, cross-principal isolation, stream guard
2026-08-20 17:12:11 -03:00
69 changed files with 136 additions and 4793 deletions

View File

@@ -1361,14 +1361,6 @@ CURSOR_USER_AGENT="Cursor/3.4"
# FETCH_BODY_TIMEOUT_MS=600000 # Time to receive full response body
# FETCH_CONNECT_TIMEOUT_MS=30000 # TCP connection establishment (default: 30s)
# FETCH_KEEPALIVE_TIMEOUT_MS=4000 # Keep-alive socket idle timeout (default: 4s)
# OMNIROUTE_DIRECT_HEADERS_TIMEOUT_MS=30000 # Bounded response-start window per direct
# # (no-proxy) attempt (#10214). A silently-dropped
# # pooled keep-alive socket surfaces no transport
# # error, so without this bound a direct request can
# # stall until undici's headersTimeout (600s) or the
# # caller's deadline; on expiry the request retries
# # once on a fresh no-keep-alive socket. 0 disables
# # the bound (default: 30000 = 30s).
# Default timeout (ms) for src/shared/utils/fetchTimeout.ts. Acts as the
# fallback when FETCH_TIMEOUT_MS is unset. Default: 120000 (2 min).
@@ -1423,14 +1415,6 @@ CURSOR_USER_AGENT="Cursor/3.4"
# OMNIROUTE_PPLX_TLS_TIMEOUT_MS=30000
# OMNIROUTE_PPLX_TLS_GRACE_MS=10000
# ── Perplexity web: built-in-search hint ──
# Used by: open-sse/executors/perplexity-web/protocol.ts — appends "You have
# built-in web search. Answer questions directly using search results." to the
# caller's system message. Off by default: Perplexity's answer engine searches
# anyway, and for coding clients the sentence leaks into replies as
# meta-commentary. Set to 1/true/yes/on to restore the old behavior.
# OMNIROUTE_PPLX_SEARCH_HINT=0
# ── Grok web TLS sidecar (Chrome-fingerprinted client) ──
# Used by: open-sse/services/grokTlsClient.ts — wire-level timeout for the
# bogdanfinn/tls-client koffi binding and the JS-side grace window layered on

View File

@@ -1 +0,0 @@
- **feat(sse):** add GLM-5.3 support (`glm-5.3`, `glm-5.3-high`, `glm-5.3-low`) across the z.ai first-party providers, mapping the upstream `reasoning_effort` request parameter to the existing 5.2 tier UX ([#10896](https://github.com/diegosouzapw/OmniRoute/pull/10896)) — thanks @phuongddx

View File

@@ -1 +0,0 @@
- **feat(home):** add a live **Recent Requests** panel beside the home Provider Topology (polls `GET /api/usage/call-logs?excludeTests=1` every ~3s, gated by the topology appearance toggle + page visibility). `excludeTests` is now an allowlist of real provider inference (`/v1/%` or `/api/v1/%`), applied before `LIMIT`, so connection-test/model-sync/management rows can never leak into the feed ([#10897](https://github.com/diegosouzapw/OmniRoute/pull/10897), extracted from [#8450](https://github.com/diegosouzapw/OmniRoute/pull/8450)) — thanks @nguyenha935

View File

@@ -1 +0,0 @@
- **fix(network):** direct (no-proxy) egress now bounds each attempt's response-start window (default 30s, `OMNIROUTE_DIRECT_HEADERS_TIMEOUT_MS`) and retries once on a fresh no-keep-alive socket, so a silently-dropped pooled keep-alive connection can no longer stall direct providers (opencode-go, command-code) until a service restart ([#10214](https://github.com/diegosouzapw/OmniRoute/issues/10214))

View File

@@ -1 +0,0 @@
- fix(config): exclude cookie-auth image bridges (chatgpt-web, gemini-web) from the unprefixed model scan so a bare id never silently binds to an unofficial web bridge (#10848)

View File

@@ -1 +0,0 @@
- **fix(perplexity-web):** make the built-in-search hint appended to every system message opt-in via `OMNIROUTE_PPLX_SEARCH_HINT` (off by default) — Perplexity's answer engine searches anyway, and the hint leaked into replies as meta-commentary for coding clients ([#10902](https://github.com/diegosouzapw/OmniRoute/pull/10902), extracted from [#8634](https://github.com/diegosouzapw/OmniRoute/pull/8634)) — thanks @danscMax

View File

@@ -1 +0,0 @@
- **fix(translator):** merge consecutive same-role contents in direct Claude to Gemini request translation to prevent upstream HTTP 400 errors

View File

@@ -1,350 +0,0 @@
---
title: "Adaptive Routing: Routing Events, Quality Feedback & Explainability"
version: 3.8.50
lastUpdated: 2026-08-20
---
# Adaptive Routing: Routing Events, Quality Feedback & Explainability
This document describes the feedback-driven adaptive routing foundation added to
OmniRoute. It is deliberately small: it introduces a typed routing-outcome
channel, an online quality signal that feeds the existing auto-combo scorer, an
optional OpenTelemetry exporter, and an explainability endpoint. It does **not**
replace the existing resilience stack (circuit breaker, connection cooldown,
model lockout, health matrix, autopilot) — it complements it.
## 1. Architectural context
OmniRoute is a data plane with a **request hot path** and a **control/intelligence
plane**. The hot path must stay fast, memory-efficient, asynchronous, resilient and
predictable. Evaluation, quality scoring, experiments and historical analysis belong
to the control plane.
```
AI Agent / IDE
┌─────────────────────┐
│ OmniRoute │ data plane (fast, sync, in-memory)
│ routing / failover │
│ health / guardrail │
│ cache / streaming │
└──────────┬──────────┘
│ RoutingEvent (fire-and-forget, ~0.2µs)
┌─────────────────────┐
│ Feedback sinks │ control plane (async, best-effort)
│ quality tracker │
│ OTel exporter │
│ explain store │
└──────────┬──────────┘
▼ quality score
auto-combo scorer
```
### What was already there (audited, not duplicated)
| Concept | Existing implementation |
| ----------------------------------- | -------------------------------------------------------------------------------------------------- |
| Availability (can we send traffic?) | Circuit breaker (CLOSED/DEGRADED/OPEN/HALF_OPEN, DB-persisted), connection cooldown, model lockout |
| Health reporting | `providerHealthMatrix.ts`, `providerHealthAutopilot.ts` |
| Shadow traffic | `open-sse/services/combo/shadowRouting.ts` |
| Guardrails | `src/lib/guardrails/` (pre/post hooks) |
| Exact cache | `src/lib/semanticCache.ts` (signature-based) |
| Evaluators / eval-driven routing | `src/lib/evals/`, `open-sse/services/evalRouting.ts` |
| Combo decision explainability | `open-sse/services/combo/decisionTrace.ts` |
| Dashboard real-time events | `src/lib/events/eventBus.ts` (UI notification channel, `unknown` payloads, 100-entry history) |
The routing-event layer is **not** a re-implementation of `eventBus`: that bus is
the dashboard's real-time notification channel (typed _event names_, opaque
payloads, UI consumers). `RoutingEvent` is a typed _outcome_ struct
(latency/tokens/cost/outcome/finish-reason) consumed by the control plane's
feedback sinks (quality tracker, OTel exporter, explain store).
### What was missing (added here)
1. A **typed routing-outcome event + sink abstraction** (`RoutingEvent` /
`RoutingEventSink`). `decisionTrace` is combo-scoped and in-memory-only;
`comboMetrics` are cumulative counters; `call_logs` is raw async persistence.
None is a typed, sink-based outcome channel that a quality tracker, an OTel
exporter, or a Future-AGI-style evaluator can subscribe to.
2. An **online quality signal** (EWMA) for output quality — the scorer previously
proxied "quality" only through static task fitness and opt-in eval pass-rates.
3. An **optional, dependency-free OTel exporter** using GenAI semantic conventions.
4. An **explainability endpoint** returning the real routing decisions + quality state.
## 2. Routing Events (feedback foundation)
Files: `open-sse/services/routing/events.ts`, `.../index.ts`
A `RoutingEvent` carries only routing metadata:
```ts
interface RoutingEvent {
requestId: string;
provider: string;
model: string;
strategy: string; // "auto" | "priority" | "direct" | ...
latencyMs: number;
ttftMs: number | null;
inputTokens: number | null;
outputTokens: number | null;
cost: number | null;
retries: number;
fallbackUsed: boolean;
outcome: RoutingOutcome; // allowlisted union
status: number | null;
finishReason: string | null;
connectionId: string | null;
ts: number;
}
```
`RoutingEventSink` is a `Send+Sync`-style trait in TypeScript:
```ts
interface RoutingEventSink {
readonly name: string;
record(event: RoutingEvent): void; // must be O(1), no sync I/O
}
```
The hot path calls `emitRoutingEvent(event)` once per completed request
(the streaming-completion callback, the non-streaming success path, and the
malformed-200 failure path in `handleChatCore`). Dispatch is synchronous fan-out
to registered sinks, but each sink only enqueues/updates in-memory state. **No
synchronous database writes, no network I/O on the hot path.**
Default sinks:
- `MemoryRoutingEventStore` — bounded (500) ring buffer, newest-first, for the
explain endpoint.
- `QualityTracker` consumer — updates the EWMA quality estimate.
- `OtlpHttpsEventSink` — optional, enabled only when `OMNIROUTE_OTEL_ENDPOINT`
(or `OTEL_EXPORTER_OTLP_ENDPOINT`) is set.
### Measured overhead (honest comparison)
`npm run bench:routing-events` on this workstation (100k iterations; sub-µs ops
measured as aggregate µs/op because per-op percentiles are below
`performance.now()` timer resolution):
| Scenario | µs/op | ops/s |
| --------------------------------- | ------ | ------ |
| baseline (scoring only) | ~0.045 | ~22 M |
| baseline + RoutingEvent (2 sinks) | ~0.168 | ~5.9 M |
| baseline + event + OTel enqueue | ~0.163 | ~6.1 M |
| concurrent (8 interleaved bursts) | ~0.18 | — |
The event-dispatch delta over baseline scoring is ~0.12 µs/request; the OTel sink
only enqueues (O(1) buffer push), adding nothing measurable. These numbers are
machine-specific and relative — not a production guarantee. The v1 "~0.2 µs"
figure was an aggregate estimate; this methodology separates the scoring baseline
from the event-dispatch cost.
## 3. Quality Signal (feedback-driven provider state)
Files: `open-sse/services/routing/quality.ts`
v2 separates **operational** from **semantic** quality:
- **Operational** — derived from the routing hot path (HTTP 4xx/5xx, connection
failures, 429s, malformed responses, stream interruptions, `finish_reason=length`,
zero-output successes, latency/TTFT EWMA). A 200 is NOT treated as semantic
quality.
- **Semantic** — the actual value of the generated output. ONLY ever produced by
an evaluator via `setSemanticQuality()`. It is `null` until one provides it and
never leaks into the operational score.
Per-(provider, model) state (EWMA + bounded counters):
- `successEwma` — EWMA (α=0.2) of outcome success.
- `latencyEwma` / `ttftEwma` — EWMA of latency (α=0.1).
- `samples`, `anomalies`, `rateLimited`, `semantic`, `semanticConfidence`.
- `recencyMs` — how recently the model was last observed.
### Confidence / sample awareness
`confidence = clamp01(samples / 50)`, and the score returned to the scorer is
blended toward the neutral midpoint:
```
score = 0.5 + confidence * (operational - 0.5)
```
Consequences (verified by tests):
- A cold provider (0 samples) scores **0.5** — not unfairly penalized, but
unable to dominate a provider with thousands of solid observations.
- A provider with 7 lucky successes is pulled toward 0.5 (never dominates from
optimistic initialization).
- A provider with 50+ samples converges to its true operational score.
- Degradation and recovery are gradual (EWMA), and one isolated failure does
not destroy a healthy provider.
`ProviderQuality` exposes `{ operational, semantic, confidence, samples, anomalies,
rateLimited, successEwma, latencyEwmaMs, ttftEwmaMs, recencyMs }`.
This feeds the auto-combo scorer as the `quality` scoring factor:
- `ScoringFactors.quality` / `ScoringWeights.quality` in
`open-sse/services/autoCombo/scoring.ts`.
- `DEFAULT_WEIGHTS`: `health` 0.1905 → 0.1605, `quality` 0.03. Sum stays 1.0.
- `buildAutoCandidates` populates `candidate.quality` from the tracker; candidates
without data default to neutral **0.5** (a cold candidate is neither boosted nor
penalized).
The closed loop:
```
RoutingEvent → QualityTracker → getQualityScore → auto-combo quality factor
↑ │
└────── request outcome (handleChatCore) ←────────────┘
```
### Hard exclusion vs soft penalty
The quality signal is a **soft adaptive preference** only. Hard exclusion stays
with the existing resilience stack: circuit breaker OPEN, quota exhausted,
auth failure, model lockout — none of these are affected by the quality score.
A provider whose quality score dips temporarily is de-preferenced, never
hard-disabled.
## 3b. Canonical stream timing (TTFT / ITL)
Files: `open-sse/utils/streamTiming.ts`
`createStreamTiming()` is the single instrumentation seam for the streaming path,
wired into `createSSEStream` (open-sse/utils/stream.ts):
- `markByte()` — first upstream chunk received.
- `markForward()` — first chunk forwarded to the client (used for TTFT).
- `markInterrupted()` — stream timeout/abort/error before a clean finish.
- `ttft()` = first-forwarded-SSE-chunk latency. **This is NOT token-level TTFT**
a single SSE chunk may carry zero/one/many tokens. Documented precisely.
- `avgItlMs()` = mean inter-chunk gap (a chunk-latency proxy for ITL).
TTFT/ITL/interrupted flow into the `RoutingEvent` (`ttftMs`, `itlMs`) and are
exported as GenAI/OmniRoute span attributes by the OTel sink.
## 4. OpenTelemetry / GenAI observability
Files: `open-sse/services/routing/otel.ts`
- Dependency-free OTLP/HTTP JSON exporter (uses global `fetch`, no
`@opentelemetry/*` SDK).
- Spans follow GenAI semantic conventions (`gen_ai.provider.name`,
`gen_ai.request.model`, `gen_ai.usage.input_tokens/output_tokens`,
`gen_ai.completion.finish_reason`, `gen_ai.system`) plus OmniRoute routing
attributes (outcome, status, ttft, retries, fallback).
- `record()` only enqueues into a bounded buffer (O(1)); a background timer
flushes via `POST {endpoint}/v1/traces` asynchronously. Under overload the
oldest events are dropped (`dropped` counter) — never backpressure the data
plane.
- **Disabled unless configured.** `OMNIROUTE_OTEL_ENDPOINT` (or
`OTEL_EXPORTER_OTLP_ENDPOINT`) must be set; otherwise the sink is not
registered and zero OTel code runs.
## 5. Explainability
- `GET /v1/explain/routing` returns the recent `RoutingEvent`s (the real
decisions, newest first) and the per-provider/model quality snapshot.
- Auth mirrors `/v1/combos` (Bearer API key or dashboard session; anonymous on
single-user local deployments with `REQUIRE_API_KEY=false`).
- Combo-level per-invocation traces remain available via the existing
`decisionTrace.ts` (header `X-OmniRoute-Combo-Trace`).
- Safety: events carry only routing metadata, never prompts/bodies/credentials.
## 6. Evaluation-plane integration (Future AGI readiness)
OmniRoute treats Future AGI (or any evaluator) as a **potential
intelligence/evaluation backend, not a dependency**. The seams:
- A `RoutingEventSink` can forward events to an evaluator asynchronously.
- The `MemoryRoutingEventStore` + quality snapshot give an evaluator the raw
decision stream.
- A future `Evaluator` (deterministic, local judge, HTTP, WASM) would consume
events/traces and return a `QualityScore` that feeds the same
`getQualityScore`/quality-factor path.
- Existing eval-driven routing (`open-sse/services/evalRouting.ts`) already
re-orders combo targets by `eval_runs` pass-rates when enabled.
No evaluation runs synchronously on the request path, and the gateway operates
fully with the evaluator absent.
## 7. Final architectural review
1. **What remains on the synchronous hot path?** Routing/scoring, guardrail
pre-checks, cache lookup, and one `emitRoutingEvent` fan-out (~0.12 µs over
baseline scoring) to in-memory sinks.
2. **What moved to asynchronous processing?** OTel export (timer + fetch),
`call_logs`/usage persistence, semantic-cache writes, quality is in-memory
and O(1) (no async needed).
3. **How does a routing outcome become feedback?** `handleChatCore` emits a
`RoutingEvent``QualityTracker` updates EWMA state → `getQualityScore`
feeds the auto-combo `quality` factor.
4. **How does quality influence future routing?** A low quality score reduces
the weighted score of that provider/model in `scoreAutoTargets`, so degraded
models are gradually de-preferenced and recover as their EWMA improves.
5. **How can Future AGI integrate without becoming a dependency?** Via the
`RoutingEventSink` interface / a future `Evaluator` adapter — no hardcoded
dependency.
6. **What happens when the evaluator is unavailable?** Routing is unaffected;
quality falls back to neutral (1.0) for models with no observed signal.
7. **What happens when telemetry is unavailable?** The OTel sink simply isn't
registered; the rest of the routing layer runs unchanged.
8. **What happens under overload?** The OTel buffer drops oldest events; quality
and the ring buffer are bounded by construction; no backpressure.
9. **How does provider state recover after degradation?** EWMA re-converges as
successes accumulate; warmup keeps cold models neutral; the circuit breaker
independently recovers via HALF_OPEN probes.
10. **Which proposed features were intentionally NOT implemented, and why?**
- Shadow traffic / experiments — already implemented
(`combo/shadowRouting.ts`); not re-built.
- Guardrails — already implemented (`src/lib/guardrails/`); not duplicated.
- Semantic cache — already implemented (`src/lib/semanticCache.ts`); not
duplicated.
- A full experiment-management platform, dataset tooling, prompt-optimization
platform, vector DB, or mandatory external OTel infrastructure — out of
scope for a lean data plane.
- A Rust `RoutingEvent` struct — the data plane is TypeScript; the TS type
is the adapted equivalent.
## 8. Configuration reference
| Variable | Default | Effect |
| ----------------------------- | ----------- | ------------------------------------------------------------------------------- |
| `OMNIROUTE_OTEL_ENDPOINT` | unset | When set, enables the OTLP/HTTP traces exporter (e.g. `http://collector:4318`). |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | unset | Fallback alias for the OTLP endpoint. |
| `OTEL_SERVICE_NAME` | `omniroute` | `service.name` resource attribute. |
## 9. Tests
- `tests/unit/routing-events.test.ts` — event normalization, status
classification, bounded ring buffer, sink fan-out + isolation.
- `tests/unit/routing-quality.test.ts` — EWMA warmup, failure/success recovery,
anomaly penalties, 429 transient handling, snapshot, reset.
- `tests/unit/routing-scoring-quality.test.ts` — weight integrity, neutral
default, quality factor ranking.
- `tests/unit/routing-otel.test.ts` — enable gating, GenAI span payload, async
flush, drop-under-overload.
- `tests/unit/routing-events-concurrency.test.ts` — thousands of events, ring
buffer boundedness, throwing-sink isolation, interleaved async bursts,
reset-during-inserts.
- `tests/unit/routing-adaptive-e2e.test.ts` — deterministic end-to-end loop via
the real `scoreAutoTargets` scorer: healthy → degrade → recover → blip, plus
cold-start and lucky-cold-provider scenarios.
- `tests/unit/stream-timing.test.ts` — TTFT (first-forwarded-chunk), ITL,
first-byte vs first-forward, interruption, malformed/empty chunk safety.
## 10. Pre-existing issues status (Phase 18)
| Issue | Status | Notes |
| ----------------------------------------------------- | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `omniglyph` export mismatch | **FIXED (environmental)** | `node_modules` was out of sync with `package-lock.json` (installed 1.3.1 vs locked 1.4.0). Running `npm install omniglyph@1.4.0` restored the locked version; type errors dropped to 0. Manifests unchanged. |
| Stale `getKnownContextOverflow` tests | **KNOWN — not fixed** | `combo-context-overflow-compression-probe.test.ts` imports a function that no longer exists in `open-sse/services/combo.ts` (only comments reference it). Fixing requires re-implementing or re-writing those tests — unrelated architectural churn. |
| `combo-runtime-unit-concurrency.test.ts` DB isolation | **KNOWN — not fixed** | Test-harness SQLite-isolation assertion fails when run directly; fails identically on the base branch. |
| i18n `llm.txt` drift | **KNOWN — not fixed** | `docs/i18n/*/llm.txt` differ from root; pre-existing, blocks the docs-sync pre-commit gate. |
Environmental vs code issues are kept distinct; no unrelated failures are hidden
behind changed test filters.

View File

@@ -12,7 +12,6 @@
"ROUTER_BACKENDS",
"admission-lanes",
"cluster-decisions",
"persistence-backend-boundary",
"ADAPTIVE_ROUTING"
"persistence-backend-boundary"
]
}

View File

@@ -52,8 +52,6 @@ safely retry only the failures after a partial result.
- **Pollinations** — Free GPT-5, Claude, Gemini (no key needed)
- **LongCat** — 10M tokens free (one-time grant, requires account + KYC)
- **Cloudflare AI** — 50+ models, 10K neurons/day
- **MLX Gemma 26B** — Local Apple Silicon model (~38.5 tok/s, ~15.9GB RAM)
- **MLX Qwen 3.8 27B** — Local Apple Silicon model (~9.1 tok/s, ~13.1GB RAM)
4. Click **Connect**
5. Done! You now have free AI access.
@@ -81,94 +79,6 @@ safely retry only the failures after a partial result.
5. Login with your account
6. Done! You now have access to your subscription models.
### Option D: Local MLX Models (Apple Silicon)
For Apple Silicon Macs with unified memory, OmniRoute supports connecting to local MLX models running via `mlx-lm.server` as regular OpenAI-compatible local providers.
#### Prerequisites
- **Apple Silicon Mac** (M1/M2/M3/M4) with 24GB+ unified memory recommended
- **uv** package manager: `curl -LsSf https://astral.sh/uv/install.sh | sh`
- **mlx-lm**: `uv pip install mlx-lm`
#### Quick Start
1. **Install dependencies**:
```bash
# Install uv if not already installed
curl -LsSf https://astral.sh/uv/install.sh | sh
# Install mlx-lm
uv pip install mlx-lm
```
2. **Start MLX servers manually** (in separate terminals):
```bash
# Terminal 1: Gemma 4 26B A4B IT-QAT (port 11435)
uv run mlx_lm.server --model mlx-community/gemma-4-26B-A4B-it-qat-q4_0-mlx-aligned --port 11435 --host 127.0.0.1
# Terminal 2: Qwen 3.8 27B MLX Mixed (port 11436)
uv run mlx_lm.server --model maglun/Qwen3.8-27B-MLX-Mixed-3.80bpw --port 11436 --host 127.0.0.1
```
3. **Connect in OmniRoute Dashboard**:
- Go to **Providers** → **Add Provider**
- Select **MLX Gemma 26B** or **MLX Qwen 3.8 27B**
- Click **Connect** (no API key needed)
4. **Use with OpenCode**:
```bash
# Configure OpenCode to use OmniRoute
opencode config set api.base_url http://localhost:20128/v1
opencode config set api.key <your-omniroute-api-key>
# Use MLX models
opencode run --model mlx-gemma/gemma-4-26b
opencode run --model mlx-qwen/qwen3.8-27b
```
#### Memory Management
**Important**: With 24GB unified memory, only **one large MLX model can run at a time**.
- Gemma 26B: ~15.9GB peak memory
- Qwen 3.8 27B: ~13.1GB peak memory
You must manage this manually:
- Run only one MLX server at a time, or
- Run both on separate machines, or
- Stop one before starting the other
OmniRoute does not automatically manage MLX server processes — it only routes requests to the OpenAI-compatible endpoints you configure.
#### Tool Calling Support
Both models support OpenAI-compatible tool calling. Test with:
```bash
curl -X POST http://localhost:20128/v1/chat/completions \
-H "Authorization: Bearer <key>" \
-H "Content-Type: application/json" \
-d '{
"model": "mlx-gemma/gemma-4-26b",
"messages": [{"role": "user", "content": "What is 2+2? Use the calculator tool."}],
"tools": [{"type": "function", "function": {"name": "calculator", "description": "Calculate", "parameters": {"type": "object", "properties": {"expression": {"type": "string"}}, "required": ["expression"]}}}]
}'
```
#### Troubleshooting
| Issue | Solution |
| ------------------ | ----------------------------------------------------------------------------- |
| Server won't start | Check `uv run mlx_lm.server --help` and verify model IDs |
| Out of memory | Ensure only one model runs; close other apps; check Activity Monitor |
| Connection refused | Verify server is running on correct port (11435/11436) |
| Slow responses | First request loads model into memory (~30-60s); subsequent requests are fast |
| Tool calling fails | Ensure model supports tools; check OmniRoute logs for translation errors |
---
## Best Free Providers

View File

@@ -732,7 +732,6 @@ REQUEST_TIMEOUT_MS (global override)
| `OMNIROUTE_AGENT_GOAL_STREAM_RECOVERY` | `true` | Enable early stream recovery automatically for detected `/goal` agent runs. Set `false`/`0`/`off` to disable the goal-specific opt-in. This can only ADD recovery on top of the operator default — it never overrides an explicit `STREAM_RECOVERY_ENABLED`/DB settings opt-out. |
| `OMNIROUTE_CODEX_DROP_NONSTANDARD_EVENTS` | _(off)_ | Strip non-standard `codex.*` SSE events (e.g. `codex.rate_limits`) that break the OpenAI SDK's `responses.stream()` with a 502. Set `true`/`1`/`yes` to enable. |
| `FETCH_HEADERS_TIMEOUT_MS` | = `FETCH_TIMEOUT_MS` | Time to receive response headers. |
| `OMNIROUTE_DIRECT_HEADERS_TIMEOUT_MS` | `30000` (30s) | Maximum response-start wait (ms) for each direct no-proxy attempt. A timeout retries once on a fresh socket; set `0` to disable the bound and retain the previous behavior. |
| `FETCH_BODY_TIMEOUT_MS` | = `FETCH_TIMEOUT_MS` | Time to receive the full response body. |
| `FETCH_CONNECT_TIMEOUT_MS` | `30000` | TCP connection establishment timeout. |
| `FETCH_KEEPALIVE_TIMEOUT_MS` | `4000` | Keep-alive socket idle timeout. |
@@ -756,7 +755,6 @@ REQUEST_TIMEOUT_MS (global override)
| `OMNIROUTE_CLAUDE_TLS_GRACE_MS` | `10000` | JS-side grace added on top of the wire timeout when the native binding is wedged. |
| `OMNIROUTE_PPLX_TLS_TIMEOUT_MS` | `30000` | Wire-level timeout for the bogdanfinn/tls-client koffi binding (`perplexityTlsClient.ts`). |
| `OMNIROUTE_PPLX_TLS_GRACE_MS` | `10000` | JS-side grace added on top of the wire timeout when the native binding is wedged. |
| `OMNIROUTE_PPLX_SEARCH_HINT` | `0` (off) | Appends "You have built-in web search. Answer questions directly using search results." to the caller's system message (`perplexity-web/protocol.ts`). Off by default — Perplexity searches anyway, and the sentence leaks into replies as meta-commentary for coding clients. Set `1`/`true`/`yes`/`on` to restore. |
| `OMNIROUTE_GROK_TLS_TIMEOUT_MS` | `60000` | Wire-level timeout for the bogdanfinn/tls-client koffi binding (`grokTlsClient.ts`). |
| `OMNIROUTE_GROK_TLS_GRACE_MS` | `10000` | JS-side grace added on top of the wire timeout when the native binding is wedged. |
| `OMNIROUTE_NOTION_TLS_TIMEOUT_MS` | `30000` | Wire-level timeout for the bogdanfinn/tls-client koffi binding (`notionTlsClient.ts`); the `notion-web` executor raises it per-request to `180000` for long generations. |

View File

@@ -355,7 +355,7 @@ Use the dashboard at `/dashboard/providers` to enable, configure, and test each
| `zerolimitai` | `zerolimitai` | ZeroLimitAI | API key, aggregator | [link](https://www.zerolimitai.com) | Temporary free trial is advertised, but official pages conflict between 3 and 7 days; a 100-calls/day claim is not treated as permanent. |
| `zylo-api` | `zylo` | Zylo API | API key, aggregator | [link](https://zyloai.net) | Basic plan: 10 RPM, 7,200 requests/day and 200,000 tokens/day; limited to Basic text models. |
## Local Providers (14)
## Local Providers (12)
| ID | Alias | Name | Tags | Website | Notes |
|----|-------|------|------|---------|-------|
@@ -365,8 +365,6 @@ Use the dashboard at `/dashboard/providers` to enable, configure, and test each
| `llama-cpp` | `llamacpp` | llama.cpp | Local, self-hosted | [link](https://github.com/ggml-org/llama.cpp) | API key optional (use any value, e.g. sk-no-key-required). Configure the llama-server OpenAI-compatible base URL (default: http://127.0.0.1:8080/v1). Note: if Llamafile is also installed, both default to port 8080 — run only one at a time or override the port. |
| `llamafile` | `llamafile` | Llamafile | Local, self-hosted | [link](https://github.com/Mozilla-Ocho/llamafile) | API key optional. Configure the local Llamafile OpenAI-compatible base URL (default: http://127.0.0.1:8080/v1). |
| `lm-studio` | `lmstudio` | LM Studio | Local, self-hosted | [link](https://lmstudio.ai) | API key optional. Configure the local LM Studio OpenAI-compatible base URL (default: http://localhost:1234/v1). |
| `mlx-gemma` | `mlx-gemma` | MLX Gemma 26B | Local, self-hosted | [link](https://github.com/ml-explore/mlx) | No API key required. Runs mlx-lm server locally on port 11435. Requires `uv` and `mlx-lm` installed. Model: `mlx-community/gemma-4-26B-A4B-it-qat-q4_0-mlx-aligned` (~15.9GB peak memory). |
| `mlx-qwen` | `mlx-qwen` | MLX Qwen 3.8 27B | Local, self-hosted | [link](https://github.com/ml-explore/mlx) | No API key required. Runs mlx-lm server locally on port 11436. Requires `uv` and `mlx-lm` installed. Model: `maglun/Qwen3.8-27B-MLX-Mixed-3.80bpw` (~13.1GB peak memory). |
| `ollama-local` | `ollama` | Ollama | Local, self-hosted | [link](https://ollama.com) | No API key required. Ollama runs locally — configure its OpenAI-compatible base URL (default: http://localhost:11434/v1) and make sure Ollama is running before connecting. |
| `oobabooga` | `ooba` | oobabooga | Local, self-hosted | [link](https://github.com/oobabooga/text-generation-webui) | API key optional. Configure the local oobabooga OpenAI-compatible base URL (default: http://localhost:5000/v1). |
| `sdwebui` | `sdwebui` | SD WebUI | Local | [link](https://github.com/AUTOMATIC1111/stable-diffusion-webui) | No API key required. Configure the local WebUI base URL (default: http://localhost:7860). |

View File

@@ -18,35 +18,6 @@ export const GLM_ANTHROPIC_DEFAULT_BASE_URLS = Object.freeze({
});
export const GLM_SHARED_MODELS = Object.freeze([
{
// GLM-5.3 (2026-08-14): one upstream id; effort is the reasoning_effort
// param (low|high|max, default max) — the -high/-low entries below are
// OmniRoute aliases resolved by GlmExecutor::parseGlmEffortTier.
// Default context window not yet published by Z.ai; 1M mirrored from
// GLM-5.2 (same base model). https://z.ai/blog/glm-5.3
id: "glm-5.3",
name: "GLM 5.3",
contextLength: 1000000,
maxOutputTokens: 131072,
toolCalling: true,
supportsReasoning: true,
},
{
id: "glm-5.3-high",
name: "GLM 5.3 High",
contextLength: 1000000,
maxOutputTokens: 131072,
toolCalling: true,
supportsReasoning: true,
},
{
id: "glm-5.3-low",
name: "GLM 5.3 Low",
contextLength: 1000000,
maxOutputTokens: 131072,
toolCalling: true,
supportsReasoning: true,
},
{
id: "glm-5.2",
name: "GLM 5.2",

View File

@@ -918,9 +918,9 @@ export function parseImageModel(modelStr) {
}
}
// No provider prefix — try to find the model in every provider, excluding cookie-auth (web) bridges
// No provider prefix — try to find the model in every provider
for (const [providerId, config] of Object.entries(IMAGE_PROVIDERS)) {
if (config.authHeader !== "cookie" && (config.routingAliases?.includes(modelStr) || config.models.some((m) => m.id === modelStr))) {
if (config.routingAliases?.includes(modelStr) || config.models.some((m) => m.id === modelStr)) {
return { provider: providerId, model: modelStr };
}
}

View File

@@ -3,8 +3,6 @@ import { unorouterProvider } from "./registry/unorouter/index.ts";
import { aimlapiProvider } from "./registry/aimlapi/index.ts";
import { byteplusProvider } from "./registry/byteplus/index.ts";
import { mlxGemmaProvider } from "./registry/mlx/index.ts";
import { mlxQwenProvider } from "./registry/mlx/index.ts";
import { ollama_cloudProvider } from "./registry/ollama-cloud/index.ts";
import { syntheticProvider } from "./registry/synthetic/index.ts";
import { ideogramProvider } from "./registry/ideogram/index.ts";
@@ -266,8 +264,6 @@ import { helixmindProvider } from "./registry/helixmind/index.ts";
export const REGISTRY: Record<string, RegistryEntry> = {
aimlapi: aimlapiProvider,
"mlx-gemma": mlxGemmaProvider,
"mlx-qwen": mlxQwenProvider,
"ollama-cloud": ollama_cloudProvider,
synthetic: syntheticProvider,
ideogram: ideogramProvider,

View File

@@ -1,66 +0,0 @@
import type { RegistryEntry } from "../../shared.ts";
import { buildOpenAiCompatibleRegistryEntry } from "../../shared.ts";
// MLX ports (deterministic, documented)
const MLX_GEMMA_PORT = 11435;
const MLX_QWEN_PORT = 11436;
// ─────────────────────────────────────────────────────────────────────────────
// Memory-aware context windows for MLX models on 24GB unified memory.
// Based on verified peak memory: Gemma 26B ~15.9GB, Qwen 27B ~13.1GB.
// KV cache estimate: 2 * 2 * layers * kv_heads * head_dim * num_ctx bytes.
// Conservative context windows to leave headroom for OS/other processes.
export const MLX_DEFAULT_CONTEXT_LIMIT = 32768;
const CONTEXT_GEMMA_26B = 8192; // 15.9GB weights + ~3.5GB KV @ 8k = ~19.4GB (safe for 24GB)
const CONTEXT_QWEN_27B = 8192; // 13.1GB weights + ~3.5GB KV @ 8k = ~16.6GB (safe for 24GB)
// ─────────────────────────────────────────────────────────────────────────────
// MLX Gemma 26B Provider
// Model: mlx-community/gemma-4-26B-A4B-it-qat-q4_0-mlx-aligned
// Verified speed: ~38.5 tok/s, peak memory: ~15.9 GB
export const mlxGemmaProvider: RegistryEntry = buildOpenAiCompatibleRegistryEntry({
id: "mlx-gemma",
alias: "mlx-gemma",
baseUrl: `http://localhost:${MLX_GEMMA_PORT}/v1`,
modelsUrl: `http://localhost:${MLX_GEMMA_PORT}/v1/models`,
passthroughModels: false,
defaultContextLength: MLX_DEFAULT_CONTEXT_LIMIT,
models: [
{
id: "mlx-community/gemma-4-26B-A4B-it-qat-q4_0-mlx-aligned",
name: "Gemma 4 26B A4B IT-QAT (MLX)",
toolCalling: true,
supportsVision: false,
supportsReasoning: false,
contextLength: CONTEXT_GEMMA_26B,
maxOutputTokens: 8192,
},
],
timeoutMs: 120000, // Longer timeout for model loading
});
// ─────────────────────────────────────────────────────────────────────────────
// MLX Qwen3.8 27B Provider
// Model: maglun/Qwen3.8-27B-MLX-Mixed-3.80bpw
// Verified speed: ~9.1 tok/s, peak memory: ~13.1 GB
export const mlxQwenProvider: RegistryEntry = buildOpenAiCompatibleRegistryEntry({
id: "mlx-qwen",
alias: "mlx-qwen",
baseUrl: `http://localhost:${MLX_QWEN_PORT}/v1`,
modelsUrl: `http://localhost:${MLX_QWEN_PORT}/v1/models`,
passthroughModels: false,
defaultContextLength: MLX_DEFAULT_CONTEXT_LIMIT,
models: [
{
id: "maglun/Qwen3.8-27B-MLX-Mixed-3.80bpw",
name: "Qwen 3.8 27B MLX Mixed 3.80bpw",
toolCalling: true,
supportsVision: false,
supportsReasoning: false,
contextLength: CONTEXT_QWEN_27B,
maxOutputTokens: 8192,
},
],
timeoutMs: 120000, // Longer timeout for model loading
});

View File

@@ -11,15 +11,13 @@ export const zaiProvider: RegistryEntry = {
authType: "apikey",
authHeader: "x-api-key",
headers: getAnthropicCompatHeaders(),
// Real upstream model IDs only. The effort tiers (glm-5.2-high/-max,
// glm-5.3-high/-low) are intentionally NOT listed here: they are OmniRoute
// aliases resolved by the GlmExecutor (parseGlmEffortTier → base model +
// effort selector). This provider uses the DefaultExecutor, which sends the
// model ID verbatim, so the aliases would reach z.ai's Anthropic endpoint as
// unknown IDs. Use the `glm` provider for effort tiers. Vision models are
// likewise omitted (handled elsewhere).
// Real upstream model IDs only. The effort tiers (glm-5.2-high / glm-5.2-max)
// are intentionally NOT listed here: they are OmniRoute aliases resolved by the
// GlmExecutor (parseGlm52Effort → base "glm-5.2" + effort field). This provider
// uses the DefaultExecutor, which sends the model ID verbatim, so the aliases
// would reach z.ai's Anthropic endpoint as unknown IDs. Use the `glm` provider
// for effort tiers. Vision models are likewise omitted (handled elsewhere).
models: [
{ id: "glm-5.3", name: "GLM 5.3" },
{ id: "glm-5.2", name: "GLM 5.2" },
{ id: "glm-5.1", name: "GLM 5.1" },
{ id: "glm-5", name: "GLM 5" },

View File

@@ -52,41 +52,17 @@ function getEffectiveKey(credentials: ProviderCredentials): string {
return credentials.apiKey || credentials.accessToken || "";
}
export type GlmEffortLevel = "low" | "high" | "max";
type GlmEffortTier = {
baseModel: string;
effort: GlmEffortLevel;
/** Transport where the upstream honors the effort selector for this family. */
transport: GlmTransport;
};
/**
* GLM-5.2 effort tiers (glm-5.2-high/-max) route exclusively through the
* Anthropic transport, where Zhipu maps Claude Code effort selectors (high/max)
* to reasoning intensity. The base model ID sent upstream is always "glm-5.2".
*
* GLM-5.3 replaced tier endpoints with a documented `reasoning_effort` request
* parameter (low|high|max, default max) on the coding chat/completions endpoint,
* so its tiers stay on the OpenAI transport and inject `reasoning_effort` +
* `thinking.type=enabled` (5.3 no longer accepts thinking disabled).
* GLM-5.2 effort tiers route exclusively through the Anthropic transport,
* where Zhipu maps Claude Code effort selectors (high/max) to reasoning
* intensity. The base model ID sent upstream is always "glm-5.2".
*
* https://docs.z.ai/devpack/latest-model
* https://z.ai/blog/glm-5.3
*/
function parseGlmEffortTier(model: string): GlmEffortTier | null {
switch (model) {
case "glm-5.2-high":
return { baseModel: "glm-5.2", effort: "high", transport: "anthropic" };
case "glm-5.2-max":
return { baseModel: "glm-5.2", effort: "max", transport: "anthropic" };
case "glm-5.3-high":
return { baseModel: "glm-5.3", effort: "high", transport: "openai" };
case "glm-5.3-low":
return { baseModel: "glm-5.3", effort: "low", transport: "openai" };
default:
return null;
}
function parseGlm52Effort(model: string): { baseModel: string; effort: "high" | "max" } | null {
if (model === "glm-5.2-high") return { baseModel: "glm-5.2", effort: "high" };
if (model === "glm-5.2-max") return { baseModel: "glm-5.2", effort: "max" };
return null;
}
/**
@@ -302,7 +278,7 @@ export class GlmExecutor extends DefaultExecutor {
credentials: ProviderCredentials,
transport: GlmTransport
) {
const effortTier = parseGlmEffortTier(model);
const effortTier = parseGlm52Effort(model);
const effectiveModel = effortTier ? effortTier.baseModel : model;
const transformed = this.transformRequest(effectiveModel, body, stream, credentials);
@@ -337,14 +313,6 @@ export class GlmExecutor extends DefaultExecutor {
}
if (transport === "openai") {
// GLM-5.3 effort tiers: inject the documented `reasoning_effort` param and
// force thinking on — 5.3 rejects thinking.type "disabled", and an effort
// tier without thinking would silently drop the selector upstream.
if (record && effortTier && effortTier.transport === "openai") {
const existingThinking = asRecord(record.thinking);
record.thinking = { ...existingThinking, type: "enabled" };
record.reasoning_effort = effortTier.effort;
}
if (record && stream && hasTools(record) && record.tool_stream === undefined) {
return { ...record, tool_stream: true };
}
@@ -478,12 +446,7 @@ export class GlmExecutor extends DefaultExecutor {
*/
private async finalizeAnthropicTransportResult(
input: ExecuteInput,
result: {
response: Response;
url: string;
headers: Record<string, string>;
transformedBody: unknown;
}
result: { response: Response; url: string; headers: Record<string, string>; transformedBody: unknown }
): Promise<GlmExecuteResult> {
const { response: rawResponse, url, headers, transformedBody } = result;
const clientHeaders = input.clientHeaders ?? {};
@@ -512,14 +475,13 @@ export class GlmExecutor extends DefaultExecutor {
}
async execute(input: ExecuteInput): Promise<GlmExecuteResult> {
const effortTier = parseGlmEffortTier(input.model);
const effortTier = parseGlm52Effort(input.model);
// Effort tiers route directly through their family's transport (no fallback):
// GLM-5.2 → Anthropic (Zhipu only graduates effort there, via the
// effort-2025-11-24 beta header in GLM_ANTHROPIC_BETA); GLM-5.3 → OpenAI
// coding endpoint (`reasoning_effort` param). See parseGlmEffortTier.
// GLM-5.2 effort tiers route directly through Anthropic transport (no fallback).
// Zhipu only graduates effort on the Anthropic endpoint via the
// effort-2025-11-24 beta header included in GLM_ANTHROPIC_BETA.
if (effortTier) {
return this.executeTransport(input, effortTier.transport);
return this.executeTransport(input, "anthropic");
}
const primaryTransport = getGlmTransport(

View File

@@ -370,29 +370,15 @@ export function buildPplxRequestBody(
};
}
const SEARCH_HINT = "You have built-in web search. Answer questions directly using search results.";
/**
* Whether to append {@link SEARCH_HINT} to the caller's system message.
*
* It used to be unconditional. Perplexity's answer engine is search-first anyway, and
* for coding clients the sentence leaks into replies as meta-commentary ("I need to
* search before responding per my instructions"), so it is now opt-in via
* `OMNIROUTE_PPLX_SEARCH_HINT`. Read per call rather than at module load so the flag
* can be flipped without restarting the server (and so tests can toggle it).
*/
function searchHintEnabled(): boolean {
return /^(1|true|yes|on)$/i.test(process.env.OMNIROUTE_PPLX_SEARCH_HINT ?? "");
}
export function buildQuery(parsed: ParsedMessages, followUpUuid: string | null): string {
if (followUpUuid) return parsed.currentMsg;
const obj: Record<string, unknown> = {};
if (parsed.systemMsg.trim()) {
obj.instructions = searchHintEnabled()
? [parsed.systemMsg.trim(), SEARCH_HINT]
: [parsed.systemMsg.trim()];
obj.instructions = [
parsed.systemMsg.trim(),
"You have built-in web search. Answer questions directly using search results.",
];
}
if (parsed.history.length > 0) {
obj.history = parsed.history;

View File

@@ -34,38 +34,6 @@ import { storeStreamingSemanticCacheResponse } from "./chatCore/streamingSemanti
import { assembleStreamingPipeline } from "./chatCore/streamingPipeline.ts";
import { sanitizeChatRequestBody } from "./chatCore/sanitization.ts";
import { applyResponsesInputPolicy } from "../services/responsesInputPolicy.ts";
import {
createRoutingEvent,
emitRoutingEvent,
outcomeFromStatus,
} from "../services/routing/index.ts";
/**
* Best-effort finish_reason extraction from a (possibly translated) response
* body for routing-event telemetry. Returns null when the shape is unknown.
*/
function routingFinishReason(body: unknown): string | null {
if (!body || typeof body !== "object") return null;
const record = body as Record<string, unknown>;
const choices = record.choices;
if (Array.isArray(choices)) {
const first = choices[0];
if (first && typeof first === "object") {
const fr = (first as Record<string, unknown>).finish_reason;
if (typeof fr === "string") return fr;
}
}
const output = record.output;
if (Array.isArray(output)) {
for (const item of output) {
if (item && typeof item === "object") {
const fr = (item as Record<string, unknown>).finish_reason;
if (typeof fr === "string") return fr;
}
}
}
return null;
}
import {
getHeaderValueCaseInsensitive,
isNoMemoryRequested,
@@ -5089,27 +5057,6 @@ export async function handleChatCore({
});
persistFailureUsage(HTTP_STATUS.BAD_GATEWAY, "malformed_translated_response");
trackPendingRequest(model, provider, pendingConnId, false);
// Routing event (feedback foundation) — record the malformed outcome so
// the quality tracker de-prioritizes this model over time.
void emitRoutingEvent(
createRoutingEvent({
requestId: traceId || pendingRequestId || "unknown",
provider: provider || "unknown",
model: model || "unknown",
strategy: isCombo ? (comboStrategy ?? "combo") : "direct",
latencyMs: Date.now() - startTime,
ttftMs: null,
inputTokens: null,
outputTokens: null,
cost: null,
retries: 0,
fallbackUsed: false, // combo-level fallback tracked by decisionTrace
outcome: "malformed",
status: HTTP_STATUS.BAD_GATEWAY,
finishReason: routingFinishReason(translatedResponse),
connectionId: credentials?.connectionId ?? null,
})
);
return createErrorResult(
HTTP_STATUS.BAD_GATEWAY,
malformedMessage,
@@ -5210,43 +5157,6 @@ export async function handleChatCore({
response: { status: 200, data: translatedResponse },
});
// Routing event (feedback foundation) — fire-and-forget, cheap.
void emitRoutingEvent(
createRoutingEvent({
requestId: traceId || pendingRequestId || "unknown",
provider: provider || "unknown",
model: model || "unknown",
strategy: isCombo ? (comboStrategy ?? "combo") : "direct",
latencyMs: Date.now() - startTime,
ttftMs: null,
inputTokens:
usage && typeof usage === "object"
? (() => {
const promptTokens = (usage as Record<string, unknown>).prompt_tokens;
return typeof promptTokens === "number" && Number.isFinite(promptTokens)
? promptTokens
: null;
})()
: null,
outputTokens:
usage && typeof usage === "object"
? (() => {
const completionTokens = (usage as Record<string, unknown>).completion_tokens;
return typeof completionTokens === "number" && Number.isFinite(completionTokens)
? completionTokens
: null;
})()
: null,
cost: Number.isFinite(estimatedCost) ? estimatedCost : null,
retries: 0,
fallbackUsed: false, // combo-level fallback tracked by decisionTrace
outcome: "success",
status: 200,
finishReason: routingFinishReason(translatedResponse),
connectionId: credentials?.connectionId ?? null,
})
);
return {
success: true,
response: buildNonStreamingJsonResponse(translatedResponse, responseHeaders),
@@ -5367,8 +5277,6 @@ export async function handleChatCore({
error: streamError,
errorCode: streamErrorCode,
ttft,
itlMs: streamItlMs,
interrupted: streamInterrupted,
}) => {
const normalizedStreamStatus = streamStatus || 200;
if (streamCompletionRecorded) return;
@@ -5472,53 +5380,6 @@ export async function handleChatCore({
endpoint: endpointPath,
});
// Routing event (feedback foundation) — fire-and-forget, cheap, never blocks
// the stream. Feeds the quality tracker + optional OTel exporter.
void emitRoutingEvent(
createRoutingEvent({
requestId: traceId || pendingRequestId || "unknown",
provider: provider || "unknown",
model: model || "unknown",
strategy: isCombo ? (comboStrategy ?? "combo") : "direct",
latencyMs: Date.now() - startTime,
ttftMs: typeof ttft === "number" && Number.isFinite(ttft) && ttft >= 0 ? ttft : null,
itlMs:
typeof streamItlMs === "number" && Number.isFinite(streamItlMs) && streamItlMs >= 0
? streamItlMs
: null,
inputTokens:
streamUsage && typeof streamUsage === "object"
? (() => {
const promptTokens = (streamUsage as Record<string, unknown>).prompt_tokens;
return typeof promptTokens === "number" && Number.isFinite(promptTokens)
? promptTokens
: null;
})()
: null,
outputTokens:
streamUsage && typeof streamUsage === "object"
? (() => {
const completionTokens = (streamUsage as Record<string, unknown>).completion_tokens;
return typeof completionTokens === "number" && Number.isFinite(completionTokens)
? completionTokens
: null;
})()
: null,
cost: null,
retries: 0,
fallbackUsed: false, // combo-level fallback tracked by decisionTrace
outcome:
normalizedStreamStatus === 200
? "success"
: streamErrorCode === "stream_interrupted" || streamErrorCode === "aborted"
? "stream_interrupted"
: outcomeFromStatus(normalizedStreamStatus),
status: normalizedStreamStatus,
finishReason: routingFinishReason(streamResponseBody),
connectionId: streamConnectionId ?? credentials?.connectionId ?? null,
})
);
persistAttemptLogs({
status: normalizedStreamStatus,
error: streamError || undefined,

View File

@@ -1346,107 +1346,6 @@ export async function handleOpenAIImageEdit({
return result;
}
/**
* Handle OpenRouter's unified Image API reference-image flow.
*
* OpenRouter does not expose `/images/edits`; image-to-image requests use
* `POST /api/v1/images` with `input_references` containing data-URL images.
* Keep this separate from the generic multipart `/images/edits` forwarder,
* whose contract is used by custom OpenAI-compatible nodes (#10197).
*/
export async function handleOpenRouterImageEdit({
model,
provider,
baseUrl,
credentials,
prompt,
imageBytes,
imageMime,
size,
n = 1,
log,
}: {
model: string;
provider: string;
baseUrl: string;
credentials:
| {
apiKey?: string;
accessToken?: string;
}
| null
| undefined;
prompt: string;
imageBytes: Buffer;
imageMime?: string | null;
size?: string | null;
n?: number;
log?: { info: (tag: string, message: string) => void } | null;
}) {
const startTime = Date.now();
let url = baseUrl.trim();
while (url.endsWith("/")) url = url.slice(0, -1);
if (url.endsWith("/images/generations")) {
url = url.slice(0, -"/images/generations".length) + "/images";
} else if (!url.endsWith("/images")) {
url += "/images";
}
const mime = imageMime || "image/png";
const upstreamBody: Record<string, unknown> = {
model,
prompt,
input_references: [
{
type: "image_url",
image_url: {
url: `data:${mime};base64,${imageBytes.toString("base64")}`,
},
},
],
n: n || 1,
};
if (size) upstreamBody.size = size;
const headers: Record<string, string> = {
"Content-Type": "application/json",
};
const token = credentials?.apiKey || credentials?.accessToken;
if (token) headers.Authorization = `Bearer ${token}`;
log?.info(
"IMAGE",
`${provider}/${model} (reference edit) | prompt: "${prompt.slice(0, 60)}..." -> ${url}`
);
const result = await fetchImageEndpoint(
url,
headers,
JSON.stringify(upstreamBody),
provider,
log
);
saveCallLog({
method: "POST",
path: "/v1/images/edits",
status: result.status || (result.success ? 200 : 502),
model: `${provider}/${model}`,
provider,
duration: Date.now() - startTime,
tokens: { prompt_tokens: 0, completion_tokens: 0 },
error: result.success
? null
: typeof result.error === "string"
? result.error.slice(0, 500)
: null,
requestBody: { model, prompt: prompt.slice(0, 200), size: size || "default", n: n || 1 },
responseBody: result.success ? { images_count: result.data?.data?.length || 0 } : null,
}).catch(() => {});
return result;
}
export async function handleImageEdit({
provider,
model,

View File

@@ -23,12 +23,6 @@ export interface ScoringFactors {
sessionAvailability?: number;
resetWindowAffinity: number;
connectionDensity: number;
/**
* Feedback-driven quality signal [0,1] from the routing-event quality tracker
* (open-sse/services/routing/quality.ts). Optional so cold candidates with no
* observed events default to neutral (1.0) and are never penalized.
*/
quality?: number;
}
export interface ScoringWeights {
@@ -46,13 +40,11 @@ export interface ScoringWeights {
sessionAvailability?: number;
resetWindowAffinity: number;
connectionDensity: number;
/** Weight for the feedback-driven quality factor (#feedback-foundation). */
quality?: number;
}
export const DEFAULT_WEIGHTS: ScoringWeights = {
quota: 0.1429,
health: 0.1605,
health: 0.1905,
costInv: 0.1429,
latencyInv: 0.1143,
taskFit: 0.0762,
@@ -65,10 +57,6 @@ export const DEFAULT_WEIGHTS: ScoringWeights = {
sessionAvailability: 0.0476,
resetWindowAffinity: 0,
connectionDensity: 0.0476,
// Shifted from `health` (0.1905 → 0.1605): availability stays dominant, and
// the new quality signal (observed output quality over time) gets a real,
// if smaller, vote. Sum remains exactly 1.0.
quality: 0.03,
};
/** Normalize independently configured UI weights into a scoring distribution. */
@@ -119,12 +107,6 @@ export interface ProviderCandidate {
sessionAvailability?: number;
/** Score [0..1] for quota reset-window preference; sooner selected reset windows score higher. */
resetWindowAffinity?: number;
/**
* Feedback-driven quality score [0..1] for this provider/model from the
* routing-event quality tracker (open-sse/services/routing). Omitted/undefined
* candidates default to a neutral 1.0 in calculateFactors.
*/
quality?: number;
connectionPoolSize?: number;
connectionId?: string;
}
@@ -159,10 +141,7 @@ export function calculateScore(factors: ScoringFactors, weights: ScoringWeights)
(weights.cacheAffinity ?? 0) * (factors.cacheAffinity ?? 0) +
(weights.sessionAvailability ?? 0) * (factors.sessionAvailability ?? 1) +
(weights.resetWindowAffinity ?? 0) * factors.resetWindowAffinity +
(weights.connectionDensity ?? 0) * factors.connectionDensity +
// Missing quality factor → neutral 0.5: a cold candidate is neither boosted
// (which would let optimistic initialization dominate) nor penalized.
(weights.quality ?? 0) * (factors.quality ?? 0.5)
(weights.connectionDensity ?? 0) * factors.connectionDensity
);
}
@@ -289,9 +268,6 @@ export function calculateFactors(
sessionAvailability: clamp01(candidate.sessionAvailability ?? 1),
resetWindowAffinity: clamp01(candidate.resetWindowAffinity ?? 0.5),
connectionDensity: clamp01(((candidate.connectionPoolSize ?? 1) - 1) / 10),
// Feedback quality signal; neutral 0.5 when the tracker has no data yet
// (cold providers are neither boosted nor unfairly penalized).
quality: clamp01(candidate.quality ?? 0.5),
};
}

View File

@@ -36,7 +36,6 @@ import {
import { buildNoUpstreamResponseDiagnostics, buildRecoveryHint } from "./combo/pinRecovery.ts";
import { buildTargetTimeoutRunner } from "./combo/targetTimeoutRunner.ts";
import { recordComboRequest, recordComboShadowRequest, getComboMetrics } from "./comboMetrics.ts";
import { qualityScoreFor } from "./routing/index.ts";
import {
expandComboSystemPromptIfPresent,
resolveTargetFingerprint,
@@ -579,9 +578,6 @@ export async function buildAutoCandidates(
connectionPoolSize: connectionPoolCounts.get(provider) ?? 1,
connectionId: target.connectionId ?? undefined,
authType,
// Feedback-driven quality signal (routing quality tracker). Neutral 1.0
// before enough samples accumulate — a cold model is never penalized.
quality: qualityScoreFor(provider, model),
};
})
);

View File

@@ -181,7 +181,6 @@ function applyDrr(targets: ResolvedComboTarget[], comboName: string): ResolvedCo
const deficits = getDrrDeficits(comboName);
const totalWeight = targets.reduce((sum, t) => sum + normalizeWeight(t.weight), 0);
if (totalWeight <= 0) return targets.slice();
// Add each target's quantum (weight share) to its deficit.
for (const target of targets) {
@@ -207,9 +206,8 @@ function applyDrr(targets: ResolvedComboTarget[], comboName: string): ResolvedCo
return [winner, ...rest];
}
/** Weights default to 1. Explicit 0 stays 0 so the operator can disable a target. */
/** Weights default to 1 and are floored at 1 to keep quantum math well-defined. */
function normalizeWeight(weight: number | undefined): number {
if (weight === 0) return 0;
return Number.isFinite(weight) && (weight as number) > 0 ? (weight as number) : 1;
}

View File

@@ -1,220 +0,0 @@
/**
* Routing Events — first-class representation of routing outcomes.
*
* Every request that reaches a provider emits one `RoutingEvent` describing what
* happened: which provider/model was used, under which strategy, with what
* latency/tokens/cost, and whether the outcome was a success, an error, a
* malformed response, a timeout, a rate-limit, or a blocked request.
*
* This is the "feedback foundation": the event is cheap to produce (no I/O in
* the emitting call) and is fanned out synchronously to registered sinks, each
* of which must be O(1)-ish and must never perform synchronous I/O. Sinks can
* then do whatever they need asynchronously — buffer to an OTLP exporter,
* update in-memory quality statistics, keep a bounded ring buffer for
* explainability, etc.
*
* DESIGN NOTE (adapted from the Future-AGI-inspired mission, kept deliberately
* lean): the original proposal was a Rust `RoutingEvent` struct + a
* `RoutingEventSink` trait. This module is the TypeScript equivalent, sized to
* the existing codebase: we already persist rich per-request detail in
* `call_logs` (async) and keep per-combo counters in `comboMetrics.ts`. This
* module adds the *typed, structured, sink-based* outcome channel those systems
* lacked, without duplicating either of them.
*
* SAFETY CONTRACT: an event carries ONLY routing metadata — provider, model,
* strategy, timing, token/cost numbers, an allowlisted outcome, finish reason,
* HTTP status, connection id. Never prompts, request/response bodies, headers,
* credentials, or account ids.
*/
/**
* Allowlisted routing outcomes. Keeping this an enum-like union prevents freeform
* strings from leaking into telemetry/quality logic and keeps sinks exhaustive.
*/
export const ROUTING_OUTCOMES = [
"success",
"error",
"malformed",
"timeout",
"rate_limited",
"stream_interrupted",
"guardrail_blocked",
"cancelled",
] as const;
export type RoutingOutcome = (typeof ROUTING_OUTCOMES)[number];
export interface RoutingEvent {
/** Correlation/request id — never a prompt or body. */
requestId: string;
provider: string;
model: string;
/** Combo strategy (e.g. "auto") or "direct" when not routed through a combo. */
strategy: string;
latencyMs: number;
/**
* Time-to-first-forwarded-SSE-chunk in ms (NOT token-level TTFT), or null
* for non-streaming requests / when nothing was forwarded.
*/
ttftMs: number | null;
/**
* Mean inter-chunk gap in ms — a chunk-latency proxy for inter-token latency,
* only meaningful for streaming requests. Null otherwise.
*/
itlMs: number | null;
inputTokens: number | null;
outputTokens: number | null;
cost: number | null;
retries: number;
fallbackUsed: boolean;
outcome: RoutingOutcome;
/** Upstream HTTP status; null when the request never reached a provider. */
status: number | null;
/** finish_reason from the provider response (stop / length / tool_calls / ...). */
finishReason: string | null;
connectionId: string | null;
ts: number;
}
/** A sink consumes routing events. Implementations must never do sync I/O. */
export interface RoutingEventSink {
readonly name: string;
record(event: RoutingEvent): void;
}
const sinks = new Set<RoutingEventSink>();
/**
* Register a sink. Returns an unsubscribe function. Registering the same sink
* instance twice is a no-op (Set semantics).
*/
export function registerRoutingEventSink(sink: RoutingEventSink): () => void {
sinks.add(sink);
return () => {
sinks.delete(sink);
};
}
/** Test/ops hook: list currently registered sink names. */
export function listRoutingEventSinks(): string[] {
return Array.from(sinks, (s) => s.name);
}
/** Test/ops hook: remove every registered sink. */
export function clearRoutingEventSinks(): void {
sinks.clear();
}
/**
* Emit a routing event to every registered sink. Synchronous and allocation-
* friendly so callers can invoke it at the end of the request hot path without
* measurable impact; each sink's `record()` must be cheap (enqueue/buffer only).
* A throwing sink is isolated so one misbehaving sink cannot break the router.
*/
export function dispatchRoutingEvent(event: RoutingEvent): void {
for (const sink of sinks) {
try {
sink.record(event);
} catch {
// Sinks are observability/best-effort — never let one break the data plane.
}
}
}
/**
* Bounded in-memory ring-buffer sink. Holds the most recent N events for
* explainability/debugging (see GET /api/v1/explain/routing). Insert is O(1);
* no TTL sweep needed because the buffer is size-bounded by construction.
*/
export class MemoryRoutingEventStore implements RoutingEventSink {
readonly name = "memory";
private buffer: RoutingEvent[] = [];
private cursor = 0;
constructor(private readonly capacity = 500) {}
record(event: RoutingEvent): void {
if (this.buffer.length < this.capacity) {
this.buffer.push(event);
} else {
this.buffer[this.cursor] = event;
}
this.cursor = (this.cursor + 1) % this.capacity;
}
/** Most recent events, newest first, up to `limit`. */
recent(limit = 50): RoutingEvent[] {
if (this.buffer.length < this.capacity) {
return this.buffer.slice(-limit).reverse();
}
// Ring is full — walk backwards from the cursor.
const out: RoutingEvent[] = [];
for (let i = 0; i < Math.min(limit, this.buffer.length); i++) {
const idx = (this.cursor - 1 - i + this.buffer.length) % this.buffer.length;
out.push(this.buffer[idx]);
}
return out;
}
clear(): void {
this.buffer = [];
this.cursor = 0;
}
get size(): number {
return this.buffer.length;
}
}
/** Create a well-formed event with defaults for unset observability fields. */
export function createRoutingEvent(input: {
requestId: string;
provider: string;
model: string;
strategy?: string | null;
latencyMs: number;
ttftMs?: number | null;
itlMs?: number | null;
inputTokens?: number | null;
outputTokens?: number | null;
cost?: number | null;
retries?: number;
fallbackUsed?: boolean;
outcome: RoutingOutcome;
status?: number | null;
finishReason?: string | null;
connectionId?: string | null;
ts?: number;
}): RoutingEvent {
return {
requestId: input.requestId,
provider: input.provider || "unknown",
model: input.model || "unknown",
strategy: input.strategy ?? "direct",
latencyMs: Math.max(0, input.latencyMs || 0),
ttftMs: input.ttftMs ?? null,
itlMs: input.itlMs ?? null,
inputTokens: input.inputTokens ?? null,
outputTokens: input.outputTokens ?? null,
cost: input.cost ?? null,
retries: input.retries ?? 0,
fallbackUsed: input.fallbackUsed ?? false,
outcome: input.outcome,
status: input.status ?? null,
finishReason: input.finishReason ?? null,
connectionId: input.connectionId ?? null,
ts: input.ts ?? Date.now(),
};
}
/**
* Classify an upstream HTTP status into a RoutingOutcome. Status 200/201 → success;
* 429 → rate_limited; 408/504 → timeout; 4xx/5xx → error; anything else → error.
*/
export function outcomeFromStatus(status: number | null | undefined): RoutingOutcome {
if (status == null) return "error";
if (status === 200 || status === 201) return "success";
if (status === 429) return "rate_limited";
if (status === 408 || status === 504) return "timeout";
return "error";
}

View File

@@ -1,133 +0,0 @@
/**
* Routing feedback foundation — default wiring.
*
* Bootstraps the default routing-event sinks:
* 1. `MemoryRoutingEventStore` — bounded ring buffer for explainability.
* 2. `QualityTracker` consumer — feeds the auto-combo `quality` scoring factor.
* 3. Optional OTel/HTTP exporter — enabled only when an OTLP endpoint is set.
*
* The hot path only calls `emitRoutingEvent()`, which fans out synchronously to
* these cheap in-memory sinks. No synchronous I/O, no external dependencies.
*
* This is the adapter seam Future AGI (or any evaluation backend) can plug into
* later without becoming a dependency: an evaluator would be another
* `RoutingEventSink` (or a consumer of the ring buffer / quality snapshot).
*/
import {
clearRoutingEventSinks,
dispatchRoutingEvent,
listRoutingEventSinks,
MemoryRoutingEventStore,
registerRoutingEventSink,
type RoutingEvent,
type RoutingEventSink,
} from "./events.ts";
import {
getProviderQuality,
getQualityScore,
getQualitySnapshot,
recordQualityEvent,
resetQualityTracker,
setSemanticQuality,
type ProviderQuality,
} from "./quality.ts";
import { isRoutingOtelEnabled, OtlpHttpsEventSink } from "./otel.ts";
const memoryStore = new MemoryRoutingEventStore(500);
// The quality tracker is registered as a sink so it updates inline with the
// event (O(1) math) and the OTel exporter only ever enqueues.
const qualitySink: RoutingEventSink = {
name: "quality",
record(event: RoutingEvent): void {
recordQualityEvent(event);
},
};
let otelSink: OtlpHttpsEventSink | null = null;
let initialized = false;
/** Register the default sinks. Idempotent; safe to call multiple times. */
export function initRoutingObservability(env: NodeJS.ProcessEnv = process.env): {
sinks: string[];
otelEnabled: boolean;
} {
if (initialized) {
return { sinks: listRoutingSinkNames(), otelEnabled: isRoutingOtelEnabled(env) };
}
initialized = true;
registerRoutingEventSink(memoryStore);
registerRoutingEventSink(qualitySink);
if (isRoutingOtelEnabled(env)) {
const endpoint = (env.OMNIROUTE_OTEL_ENDPOINT ?? env.OTEL_EXPORTER_OTLP_ENDPOINT ?? "").trim();
otelSink = new OtlpHttpsEventSink({
endpoint,
serviceName: env.OTEL_SERVICE_NAME ?? "omniroute",
maxBatchSize: 64,
flushIntervalMs: 10_000,
});
registerRoutingEventSink(otelSink);
}
return { sinks: listRoutingSinkNames(), otelEnabled: otelSink != null };
}
/** Emit a routing event to all registered sinks (fire-and-forget, cheap). */
export function emitRoutingEvent(event: RoutingEvent): void {
if (!initialized) initRoutingObservability();
dispatchRoutingEvent(event);
}
/** Neutral default quality used when a model has no observed events. */
export function qualityScoreFor(provider: string, model: string): number {
return getQualityScore(provider, model);
}
/** Full per-provider/model quality view (operational + semantic + confidence). */
export function providerQualityFor(provider: string, model: string): ProviderQuality {
return getProviderQuality(provider, model);
}
/**
* Evaluator seam: record a semantic quality score. NEVER call this from the
* request hot path with HTTP-derived signals — semantic quality is reserved for
* actual evaluation (task success, tool-use correctness, groundedness).
*/
export { setSemanticQuality } from "./quality.ts";
export function routingQualitySnapshot(limit = 200): ReturnType<typeof getQualitySnapshot> {
return getQualitySnapshot(limit);
}
export { classifyQuality, type QualityClassification } from "./quality.ts";
export function recentRoutingEvents(limit = 50): RoutingEvent[] {
return memoryStore.recent(limit);
}
export function routingOtelStats(): { buffered: number; dropped: number } | null {
return otelSink ? otelSink.getStats() : null;
}
function listRoutingSinkNames(): string[] {
return listRoutingEventSinks();
}
/** Test/ops hook: full reset of the routing observability layer. */
export function resetRoutingObservability(): void {
clearRoutingEventSinks();
memoryStore.clear();
resetQualityTracker();
if (otelSink) {
otelSink.stop();
otelSink = null;
}
initialized = false;
}
export type { RoutingEvent, RoutingOutcome, RoutingEventSink } from "./events.ts";
export { createRoutingEvent, outcomeFromStatus } from "./events.ts";

View File

@@ -1,227 +0,0 @@
/**
* Optional OpenTelemetry / GenAI observability sink.
*
* A `RoutingEventSink` that forwards routing events to an OTLP/HTTP collector as
* GenAI semantic-convention spans (semconvgenai: `gen_ai.provider.name`,
* `gen_ai.request.model`, `gen_ai.operation.name`, `gen_ai.usage.input_tokens`,
* `gen_ai.usage.output_tokens`, etc.).
*
* Deliberately lightweight:
* - No `@opentelemetry/*` SDK dependency. Uses the collector's OTLP/HTTP JSON
* (traces) endpoint via global `fetch`, which is already available and async.
* - `record()` only enqueues into a bounded buffer (O(1), never I/O). A single
* background flush timer drains the buffer asynchronously. Under overload the
* oldest events are dropped (never backpressure the data plane).
* - Disabled unless `OMNIROUTE_OTEL_ENDPOINT` (or `OTEL_EXPORTER_OTLP_ENDPOINT`)
* is set — normal lightweight deployments run with zero OTel code executing.
* - No secrets/prompts are ever serialized; only RoutingEvent metadata.
*/
export interface OtlpHttpsExporterConfig {
/** Collector base URL, e.g. https://collector:4318 — spans go to /v1/traces. */
endpoint: string;
/** Export batch size / flush interval. */
maxBatchSize?: number;
flushIntervalMs?: number;
serviceName?: string;
}
/** Resolve whether OTLP export is configured. */
export function isRoutingOtelEnabled(env: NodeJS.ProcessEnv = process.env): boolean {
const endpoint = (env.OMNIROUTE_OTEL_ENDPOINT ?? env.OTEL_EXPORTER_OTLP_ENDPOINT ?? "").trim();
return endpoint.length > 0;
}
interface OtelSpan {
traceId: string;
spanId: string;
name: string;
kind: number;
startTimeUnixNano: string;
endTimeUnixNano: string;
attributes: Array<{
key: string;
value: { stringValue?: string; intValue?: string; doubleValue?: number };
}>;
}
function toHex(bytes: Uint8Array): string {
return Array.from(bytes, (b) => b.toString(16).padStart(2, "0")).join("");
}
function randomId(bytes: number): string {
const arr = new Uint8Array(bytes);
// Use crypto.getRandomValues when available (Node ≥ 19 global), else Math.random.
if (typeof crypto !== "undefined" && typeof crypto.getRandomValues === "function") {
crypto.getRandomValues(arr);
} else {
for (let i = 0; i < bytes; i++) arr[i] = Math.floor(Math.random() * 256);
}
return toHex(arr);
}
export class OtlpHttpsEventSink {
readonly name = "otel";
private readonly endpoint: string;
private readonly maxBatchSize: number;
private readonly serviceName: string;
private buffer: RoutingEventLike[] = [];
private dropped = 0;
private consecutiveFailures = 0;
private flushedBatches = 0;
private timer: ReturnType<typeof setInterval> | null = null;
private flushing = false;
constructor(private readonly config: OtlpHttpsExporterConfig) {
this.endpoint = config.endpoint.replace(/\/+$/, "") + "/v1/traces";
this.maxBatchSize = config.maxBatchSize ?? 64;
this.serviceName = config.serviceName ?? "omniroute";
this.start();
}
/** O(1) enqueue; drops oldest when the buffer is full. Never performs I/O. */
record(event: RoutingEventLike): void {
if (this.buffer.length >= this.maxBatchSize * 4) {
this.buffer.shift();
this.dropped += 1;
}
this.buffer.push(event);
}
getStats(): {
buffered: number;
dropped: number;
consecutiveFailures: number;
flushedBatches: number;
} {
return {
buffered: this.buffer.length,
dropped: this.dropped,
consecutiveFailures: this.consecutiveFailures,
flushedBatches: this.flushedBatches,
};
}
stop(): void {
if (this.timer) {
clearInterval(this.timer);
this.timer = null;
}
void this.flush();
}
private start(): void {
const intervalMs = this.config.flushIntervalMs ?? 10_000;
this.timer = setInterval(() => void this.flush(), intervalMs);
// Do not keep the process alive just for telemetry.
this.timer.unref?.();
}
private async flush(): Promise<void> {
if (this.flushing) return;
if (this.buffer.length === 0) return;
this.flushing = true;
const batch = this.buffer.splice(0, this.maxBatchSize);
try {
const res = await fetch(this.endpoint, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(buildOtlpTracesPayload(batch, this.serviceName)),
signal: AbortSignal.timeout(3000),
});
if (!res.ok) throw new Error(`OTLP collector returned ${res.status}`);
this.consecutiveFailures = 0;
this.flushedBatches += 1;
} catch {
// Telemetry delivery is best-effort. Re-buffer for a retry, but stop after
// MAX_CONSECUTIVE_FAILURES so a permanently-unavailable collector cannot
// grow the buffer without bound. The dropped counter reflects the loss.
this.consecutiveFailures += 1;
if (this.consecutiveFailures >= MAX_CONSECUTIVE_FAILURES) {
this.dropped += batch.length;
} else {
this.buffer.unshift(...batch);
}
} finally {
this.flushing = false;
}
}
}
/** Drop a batch (and count it) after this many consecutive collector failures. */
const MAX_CONSECUTIVE_FAILURES = 5;
// eslint-disable-next-line @typescript-eslint/no-explicit-any
type RoutingEventLike = any;
/**
* Build an OTLP/HTTP traces JSON payload with one span per routing event,
* mapped to GenAI semantic conventions.
*/
export function buildOtlpTracesPayload(events: RoutingEventLike[], serviceName: string): unknown {
const resourceSpans = [
{
resource: {
attributes: [
{ key: "service.name", value: { stringValue: serviceName } },
{ key: "telemetry.sdk.name", value: { stringValue: "omniroute-routing" } },
],
},
scopeSpans: [
{
scope: { name: "omniroute.routing" },
spans: events.map(toSpan),
},
],
},
];
return { resourceSpans };
}
function attr(
key: string,
value: string | number
): { key: string; value: { stringValue?: string; intValue?: string; doubleValue?: number } } {
if (typeof value === "number") {
return Number.isInteger(value)
? { key, value: { intValue: String(value) } }
: { key, value: { doubleValue: value } };
}
return { key, value: { stringValue: String(value) } };
}
function toSpan(event: RoutingEventLike): OtelSpan {
const traceId = randomId(16);
const spanId = randomId(8);
const startNs = BigInt(event.ts) * 1_000_000n;
const endNs = startNs + BigInt(Math.max(0, event.latencyMs || 0)) * 1_000_000n;
const attributes = [
attr("gen_ai.provider.name", event.provider),
attr("gen_ai.request.model", event.model),
attr("gen_ai.operation.name", "chat"),
attr("gen_ai.system", event.strategy || "direct"),
attr("gen_ai.usage.input_tokens", event.inputTokens ?? 0),
attr("gen_ai.usage.output_tokens", event.outputTokens ?? 0),
attr("gen_ai.completion.finish_reason", event.finishReason ?? "unknown"),
attr("gen_ai.request.temperature", 0),
attr("omniroute.routing.outcome", event.outcome),
attr("omniroute.routing.status", event.status ?? 0),
attr("omniroute.routing.ttft_ms", event.ttftMs ?? -1),
attr("omniroute.routing.itl_ms", event.itlMs ?? -1),
attr("omniroute.routing.retries", event.retries ?? 0),
attr("omniroute.routing.fallback_used", event.fallbackUsed ? 1 : 0),
attr("gen_ai.client.token.usage.input_tokens", event.inputTokens ?? 0),
attr("gen_ai.client.token.usage.output_tokens", event.outputTokens ?? 0),
];
if (event.connectionId) attributes.push(attr("omniroute.connection_id", event.connectionId));
return {
traceId,
spanId,
name: `chat ${event.provider}/${event.model}`,
kind: 3, // CLIENT
startTimeUnixNano: startNs.toString(),
endTimeUnixNano: endNs.toString(),
attributes,
};
}

View File

@@ -1,313 +0,0 @@
/**
* Provider/Model Quality Signal — feedback-driven adaptive routing (v2).
*
* v2 separates two distinct concepts that v1 conflated:
*
* - **Operational quality** — derived from the routing hot path (HTTP status,
* connection failures, 429s, malformed responses, stream interruptions,
* finish_reason anomalies, zero-output successes, latency/TTFT). A request
* returning HTTP 200 is NOT necessarily high quality; operational quality
* only says "the wire behaved."
* - **Semantic quality** — the actual value of the generated output
* (evaluator score, task success, tool-use correctness, factual accuracy).
* This is ONLY ever produced by an external evaluator via
* `setSemanticQuality()`. It is never manufactured from HTTP success. It is
* `null` until an evaluator provides a value.
*
* Confidence / sample awareness (v2):
* - `confidence = clamp01(samples / CONFIDENCE_FULL_SAMPLES)`.
* - The score returned to the scorer is blended toward the neutral midpoint
* (0.5): `score = NEUTRAL + confidence * (operational - NEUTRAL)`.
* - Consequences: a cold provider (0 samples) scores neutral 0.5 — it is not
* unfairly penalized, but it also cannot dominate a provider with thousands
* of solid observations. A provider with 7 lucky successes is pulled toward
* 0.5, so it never dominates purely from optimistic initialization.
*
* This complements the existing resilience stack (circuit breaker, connection
* cooldown, model lockout, health matrix): those handle *availability* (hard
* exclusion); this signal handles *soft adaptive preference*.
*
* Statistics are plain arithmetic (EWMA + small counters), O(1) per event, safe
* under the Node event loop's single thread — no lock-free/atomic trickery.
*/
/** EWMA smoothing factor (alpha). Lower = slower adaptation. */
const OPERATIONAL_ALPHA = 0.2;
/** Latency EWMA alpha — slower so transient spikes don't tank quality instantly. */
const LATENCY_ALPHA = 0.1;
/** Samples at which confidence reaches 1.0 (full confidence). */
const CONFIDENCE_FULL_SAMPLES = 50;
/** Neutral score used for cold/unknown providers (midpoint, neither boosted nor penalized). */
const NEUTRAL_SCORE = 0.5;
interface QualityState {
/** EWMA of the success indicator (1 = good, 0 = bad). */
successEwma: number;
/** EWMA of latency in ms. */
latencyEwma: number;
/** EWMA of TTFT in ms (streaming only). */
ttftEwma: number | null;
/** Total events observed for this (provider, model). */
samples: number;
/** Count of operational-anomaly events (malformed / empty / length / interrupted). */
anomalies: number;
/** Rate-limit (429) count — tracked separately for observability. */
rateLimited: number;
/** Semantic quality [0,1] from an external evaluator, if one has provided it. */
semantic: number | null;
/** Confidence [0,1] of the semantic score as reported by the evaluator. */
semanticConfidence: number | null;
lastTs: number;
}
const states = new Map<string, QualityState>();
function keyOf(provider: string, model: string): string {
return `${provider}/${model}`;
}
function getOrCreate(key: string): QualityState {
let state = states.get(key);
if (!state) {
state = {
successEwma: 1,
latencyEwma: 0,
ttftEwma: null,
samples: 0,
anomalies: 0,
rateLimited: 0,
semantic: null,
semanticConfidence: null,
lastTs: 0,
};
states.set(key, state);
}
return state;
}
function isOperationalAnomaly(event: {
outcome: string;
finishReason: string | null;
outputTokens: number | null | undefined;
}): boolean {
if (event.outcome === "malformed" || event.outcome === "stream_interrupted") return true;
// finish_reason=length → the model ran out of output budget (truncated answer).
if (event.outcome === "success" && event.finishReason === "length") return true;
// A "successful" 200 that produced zero output tokens is an empty/invalid output.
// NOTE: we deliberately do NOT treat a missing finish_reason as an anomaly —
// streaming passthrough frequently has no reconstructed finish_reason, so that
// signal would penalize every legitimately streamed request (pure noise).
if (event.outcome === "success" && event.outputTokens === 0) return true;
return false;
}
function successIndicator(event: { outcome: string; status: number | null }): number {
if (event.outcome === "success") return 1;
// 429 is a transient signal, not a quality failure — treat as neutral-positive.
if (event.outcome === "rate_limited" || event.status === 429) return 0.5;
return 0;
}
/** Record one operational routing event into the quality estimate. O(1). */
export function recordQualityEvent(event: {
provider: string;
model: string;
outcome: string;
status: number | null;
latencyMs: number;
ttftMs?: number | null;
finishReason?: string | null;
outputTokens?: number | null;
ts?: number;
}): void {
const key = keyOf(event.provider || "unknown", event.model || "unknown");
const state = getOrCreate(key);
state.samples += 1;
if (
isOperationalAnomaly({
outcome: event.outcome,
finishReason: event.finishReason ?? null,
outputTokens: event.outputTokens ?? undefined,
})
) {
state.anomalies += 1;
}
if (event.outcome === "rate_limited" || event.status === 429) state.rateLimited += 1;
const indicator = successIndicator({ outcome: event.outcome, status: event.status });
// First sample seeds the EWMA directly (no lag toward a default).
state.successEwma =
state.samples === 1
? indicator
: state.successEwma + OPERATIONAL_ALPHA * (indicator - state.successEwma);
const latency = Number.isFinite(event.latencyMs) && event.latencyMs >= 0 ? event.latencyMs : 0;
state.latencyEwma =
state.samples === 1
? latency
: state.latencyEwma + LATENCY_ALPHA * (latency - state.latencyEwma);
const ttft = event.ttftMs;
if (typeof ttft === "number" && Number.isFinite(ttft) && ttft >= 0) {
state.ttftEwma =
state.ttftEwma == null ? ttft : state.ttftEwma + LATENCY_ALPHA * (ttft - state.ttftEwma);
}
state.lastTs = event.ts ?? Date.now();
}
/**
* Evaluator seam: record a semantic quality score for a (provider, model).
* Semantic quality is ONLY ever produced by an evaluator (deterministic scorer,
* local LLM judge, HTTP/Future-AGI adapter, WASM). It is never manufactured from
* operational/HTP success. `confidence` should reflect the evaluator's certainty
* (e.g. number of eval cases backing the score).
*/
export function setSemanticQuality(
provider: string,
model: string,
score: number,
confidence: number
): void {
const state = getOrCreate(keyOf(provider || "unknown", model || "unknown"));
state.semantic = Math.max(0, Math.min(1, Number.isFinite(score) ? score : 0.5));
state.semanticConfidence = Math.max(0, Math.min(1, Number.isFinite(confidence) ? confidence : 0));
}
export interface ProviderQuality {
provider: string;
model: string;
/** Operational score [0,1] (wire behavior) — confidence-adjusted, neutral 0.5 cold. */
operational: number;
/** Semantic score [0,1] from an evaluator, or null when none has been provided. */
semantic: number | null;
/** Confidence [0,1] of the operational score (sample-count based). */
confidence: number;
/** Confidence [0,1] of the semantic score, when an evaluator reported one. */
semanticConfidence: number | null;
samples: number;
anomalies: number;
rateLimited: number;
successEwma: number;
latencyEwmaMs: number;
ttftEwmaMs: number | null;
/** Milliseconds since the last observed event; null when never observed. */
recencyMs: number | null;
lastTs: number;
}
/** Raw operational score before the confidence blend (pure EWMA + penalties). */
function rawOperationalScore(state: QualityState): number {
let score = state.successEwma;
// Latency degradation: soft penalty capped at 0.2 so slow models are discounted, not zeroed.
const latencyPenalty = Math.min(0.2, state.latencyEwma / 60_000);
score -= latencyPenalty;
// Anomaly penalty: capped so a few bad apples don't nuke a provider entirely.
const anomalyRate = state.anomalies / Math.max(1, state.samples);
score -= Math.min(0.25, anomalyRate * 0.5);
return Math.max(0, Math.min(1, score));
}
function confidenceOf(samples: number): number {
return Math.max(0, Math.min(1, samples / CONFIDENCE_FULL_SAMPLES));
}
/**
* Operational quality for a (provider, model), confidence-adjusted and blended
* toward the neutral midpoint. See module docs for the cold-start guarantee.
*/
export function getProviderQuality(provider: string, model: string): ProviderQuality {
const state = states.get(keyOf(provider, model));
const now = Date.now();
if (!state || state.samples === 0) {
return {
provider,
model,
operational: NEUTRAL_SCORE,
semantic: null,
confidence: 0,
semanticConfidence: null,
samples: 0,
anomalies: 0,
rateLimited: 0,
successEwma: 1,
latencyEwmaMs: 0,
ttftEwmaMs: null,
recencyMs: null,
lastTs: 0,
};
}
const confidence = confidenceOf(state.samples);
const raw = rawOperationalScore(state);
const operational = NEUTRAL_SCORE + confidence * (raw - NEUTRAL_SCORE);
return {
provider,
model,
operational,
semantic: state.semantic,
confidence,
semanticConfidence: state.semanticConfidence,
samples: state.samples,
anomalies: state.anomalies,
rateLimited: state.rateLimited,
successEwma: state.successEwma,
latencyEwmaMs: state.latencyEwma,
ttftEwmaMs: state.ttftEwma,
recencyMs: state.samples > 0 ? Math.max(0, now - state.lastTs) : null,
lastTs: state.lastTs,
};
}
/**
* Backward-compatible scalar used by the auto-combo scorer's `quality` factor.
* Returns the confidence-adjusted operational score (neutral 0.5 when cold).
*/
export function getQualityScore(provider: string, model: string): number {
return getProviderQuality(provider, model).operational;
}
/** Full snapshot of the tracker for explainability / dashboard. */
export function getQualitySnapshot(limit = 200): ProviderQuality[] {
const views: ProviderQuality[] = [];
for (const [key] of states) {
const slash = key.indexOf("/");
const provider = slash >= 0 ? key.slice(0, slash) : key;
const model = slash >= 0 ? key.slice(slash + 1) : key;
views.push(getProviderQuality(provider, model));
}
views.sort((a, b) => b.lastTs - a.lastTs);
return views.slice(0, limit);
}
/**
* Classify a provider/model quality state for explainability / dashboard.
* This reflects the SOFT adaptive signal — it says nothing about hard exclusion
* (circuit open / quota / auth), which is owned by the resilience stack.
*
* - "healthy": high confidence + operational quality well above neutral
* - "degraded": operational quality at or below neutral (soft penalty active)
* - "warming": low confidence (few samples) — treated neutrally
* - "cold": never observed — neutral, cannot dominate
*/
export type QualityClassification = "healthy" | "degraded" | "warming" | "cold";
export function classifyQuality(q: ProviderQuality): QualityClassification {
if (q.samples === 0) return "cold";
if (q.confidence < 0.5) return "warming";
if (q.operational < 0.5) return "degraded";
return "healthy";
}
/** Test/ops hook: reset all quality state. */
export function resetQualityTracker(): void {
states.clear();
}
export const QUALITY_WELL_KNOWN = {
CONFIDENCE_FULL_SAMPLES,
NEUTRAL_SCORE,
} as const;

View File

@@ -15,8 +15,6 @@ import { getModelSpec } from "../../../src/shared/constants/modelSpecs.ts";
import {
buildChangedToolNameMap,
buildHistoricalToolResultContext,
mergeConsecutiveSameRoleContents,
type GeminiContent,
} from "./openai-to-gemini/helpers.ts";
/**
@@ -47,7 +45,7 @@ export function claudeToGeminiRequest(model, body, stream, credentials = null) {
: null;
const result: {
model: string;
contents: GeminiContent[];
contents: Array<Record<string, unknown>>;
generationConfig: Record<string, unknown>;
safetySettings: unknown;
systemInstruction?: { role: string; parts: Array<{ text: string }> };
@@ -316,11 +314,6 @@ export function claudeToGeminiRequest(model, body, stream, credentials = null) {
result._toolNameMap = changedToolNameMap;
}
// Gemini strictly rejects requests containing consecutive messages with the same role
// (400 INVALID_ARGUMENT: "Request contains consecutive messages with the same role").
// Normalize adjacent same-role messages by concatenating their parts.
result.contents = mergeConsecutiveSameRoleContents(result.contents);
return result;
}

View File

@@ -39,13 +39,8 @@ import {
escapeHistoricalContextAttribute,
escapeHistoricalContextContent,
buildHistoricalToolResultContext,
type GeminiPart,
type GeminiContent,
mergeConsecutiveSameRoleContents,
} from "./openai-to-gemini/helpers.ts";
export { mergeConsecutiveSameRoleContents, type GeminiContent, type GeminiPart };
// Observed Antigravity wrapper output cap, not an underlying model capability.
// Keep this bridge-local: Antigravity currently caps visible output around 16K.
// See: https://github.com/keisksw/antigravity-output-analysis
@@ -61,6 +56,9 @@ const GEMINI_BUILTIN_TOOL_NAMES = new Set<string>([
"googleSearch",
]);
type GeminiPart = Record<string, unknown>;
type GeminiContent = { role: string; parts: GeminiPart[] };
type GeminiFunctionDeclaration = {
name: string;
description: string;
@@ -160,6 +158,29 @@ type GeminiToolNameOptions = {
supportsSignatureBypass?: boolean;
};
// Gemini-family APIs (incl. Antigravity / Vertex) reject a `contents[]` array that
// has two adjacent entries with the same role:
// 400 INVALID_ARGUMENT "Request contains consecutive messages with the same role".
// Client history that carries consecutive user turns — or a tool-result turn (mapped
// to role:"user") immediately followed by a plain user turn — would otherwise leak
// that invalid alternation through. Merge adjacent same-role entries by concatenating
// their parts, the same normalization the Kiro and Claude request paths already apply
// (9router#2191).
export function mergeConsecutiveSameRoleContents(contents: GeminiContent[]): GeminiContent[] {
const merged: GeminiContent[] = [];
for (const entry of contents) {
const last = merged[merged.length - 1];
if (last && last.role === entry.role) {
last.parts.push(...entry.parts);
} else {
// Shallow-copy the entry and its `parts` array so a later same-role merge
// (`last.parts.push(...)`) never mutates the caller's input objects.
merged.push({ ...entry, parts: [...entry.parts] });
}
}
return merged;
}
// Core: Convert OpenAI request to Gemini format (base for all variants)
function openaiToGeminiBase(
model: string,

View File

@@ -152,29 +152,3 @@ export function buildHistoricalToolResultContext(name: string, response: unknown
"</previous_tool_result_context>",
].join("\n");
}
export type GeminiPart = Record<string, unknown>;
export type GeminiContent = { role: string; parts: GeminiPart[] };
// Gemini-family APIs (incl. Antigravity / Vertex) reject a `contents[]` array that
// has two adjacent entries with the same role:
// 400 INVALID_ARGUMENT "Request contains consecutive messages with the same role".
// Client history that carries consecutive user turns — or a tool-result turn (mapped
// to role:"user") immediately followed by a plain user turn — would otherwise leak
// that invalid alternation through. Merge adjacent same-role entries by concatenating
// their parts, the same normalization the Kiro and Claude request paths already apply
// (9router#2191).
export function mergeConsecutiveSameRoleContents(contents: GeminiContent[]): GeminiContent[] {
const merged: GeminiContent[] = [];
for (const entry of contents) {
const last = merged[merged.length - 1];
if (last && last.role === entry.role) {
last.parts.push(...entry.parts);
} else {
// Shallow-copy the entry and its `parts` array so a later same-role merge
// (`last.parts.push(...)`) never mutates the caller's input objects.
merged.push({ ...entry, parts: [...entry.parts] });
}
}
return merged;
}

View File

@@ -1,77 +0,0 @@
type DirectFetchOptions = RequestInit & { dispatcher?: unknown };
type DirectFetch = (
input: RequestInfo | URL,
options: DirectFetchOptions
) => Promise<Response>;
const DEFAULT_DIRECT_HEADERS_TIMEOUT_MS = 30_000;
const DIRECT_RESPONSE_START_TIMEOUT_CODE = "DIRECT_RESPONSE_START_TIMEOUT";
export function resolveDirectHeadersTimeoutMs(
env: Record<string, string | undefined> = process.env
): number {
const raw = env.OMNIROUTE_DIRECT_HEADERS_TIMEOUT_MS;
if (raw == null || raw.trim() === "") return DEFAULT_DIRECT_HEADERS_TIMEOUT_MS;
const parsed = Number(raw);
return Number.isFinite(parsed) && parsed > 0 ? Math.floor(parsed) : 0;
}
function createDirectResponseStartTimeout(timeoutMs: number): Error & { code: string } {
const err = new Error(
`Direct response did not start within ${timeoutMs}ms — retrying on a fresh socket`
) as Error & { code: string };
err.name = "TimeoutError";
err.code = DIRECT_RESPONSE_START_TIMEOUT_CODE;
return err;
}
export function isDirectResponseStartTimeout(err: unknown): boolean {
return (
!!err &&
typeof err === "object" &&
"code" in err &&
err.code === DIRECT_RESPONSE_START_TIMEOUT_CODE
);
}
function mergeAbortSignals(
primary: AbortSignal | null | undefined,
secondary: AbortSignal
): AbortSignal {
if (!primary) return secondary;
if (primary.aborted) return primary;
const controller = new AbortController();
const onPrimaryAbort = () => controller.abort(primary.reason);
const onSecondaryAbort = () => controller.abort(secondary.reason);
const cleanup = () => {
primary.removeEventListener("abort", onPrimaryAbort);
secondary.removeEventListener("abort", onSecondaryAbort);
};
primary.addEventListener("abort", onPrimaryAbort, { once: true });
secondary.addEventListener("abort", onSecondaryAbort, { once: true });
controller.signal.addEventListener("abort", cleanup, { once: true });
return controller.signal;
}
export async function directFetchWithBoundedResponseStart(
input: RequestInfo | URL,
options: DirectFetchOptions,
fetchImpl: DirectFetch,
timeoutMs: number
): Promise<Response> {
if (!timeoutMs || timeoutMs <= 0) return fetchImpl(input, options);
const attemptController = new AbortController();
const timer = setTimeout(
() => attemptController.abort(createDirectResponseStartTimeout(timeoutMs)),
timeoutMs
);
timer.unref?.();
try {
return await fetchImpl(input, {
...options,
signal: mergeAbortSignals(options.signal, attemptController.signal),
});
} finally {
clearTimeout(timer);
}
}

View File

@@ -19,11 +19,6 @@ import {
isControlPlaneProxyDirectFallbackEnabled,
isFeatureFlagEnabled,
} from "@/shared/utils/featureFlags";
import {
directFetchWithBoundedResponseStart,
isDirectResponseStartTimeout,
resolveDirectHeadersTimeoutMs,
} from "./directResponseStartTimeout.ts";
// #9100: relay egress (Vercel / Deno / Cloudflare edge functions) used to go
// through bare `originalFetch` — NO connection pooling, NO timeout, NO retry.
@@ -159,6 +154,7 @@ type TlsFingerprintStore = {
provider?: string | null;
sessionScope?: string;
};
/**
* #5217 (Gap-secondary): a mutable sink that records the proxy actually applied
* by `runWithProxyContext` for the in-flight request. Executors that pin their
@@ -806,7 +802,15 @@ async function patchedFetch(
(deps.nativeFetch as FetchWithDispatcher | undefined) ?? originalFetchWithDispatcher;
return _nativeFetch(input, options);
}
// Direct undici path: bound response-start, fresh-socket retry, and body guard.
// Direct connection (no proxy) — use undici with custom dispatcher for timeout control.
// Falls back to original native fetch if dispatcher initialization fails (#1054).
// Retries once on transient dispatcher errors before falling back (fix: proxyfetch-undici-retry).
//
// Non-replayable body guard: if the body is stream-like (ReadableStream/Blob)
// or the input is a Request that carries a body, the first dispatcher attempt
// owns that body. Retrying or falling back to native fetch would replay a
// consumed/locked body and can mask the original transport error with
// "Response body object should not be disturbed or locked".
const hasNonReplayableBody = requestHasNonReplayableBody(input, options);
const maxAttempts = hasNonReplayableBody ? 1 : 2;
const _undiciDirect =
@@ -814,44 +818,32 @@ async function patchedFetch(
const _nativeFallback =
(deps.nativeFetch as FetchWithDispatcher | undefined) ?? originalFetchWithDispatcher;
let lastDispatcherError: unknown = null;
const directHeadersTimeoutMs = resolveDirectHeadersTimeoutMs();
let targetHostForLogs = "";
try {
targetHostForLogs = new URL(targetUrl).host;
} catch {
// ignore — logging is best-effort
}
for (let attempt = 0; attempt < maxAttempts; attempt++) {
try {
return await directFetchWithBoundedResponseStart(
input,
{
...options,
dispatcher: attempt === 0 ? getDefaultDispatcher() : getRetryDispatcher(),
},
_undiciDirect,
directHeadersTimeoutMs
);
return await _undiciDirect(input, {
...options,
// #4252: first attempt uses the pooled keep-alive dispatcher; a retry
// (after a transient socket error) uses the no-keep-alive dispatcher so
// it opens a FRESH socket instead of grabbing another stale pooled one
// — the burst pattern was the retry re-hitting a dead pooled socket and
// then falling through to native fetch (which also pools) → 502.
dispatcher: attempt === 0 ? getDefaultDispatcher() : getRetryDispatcher(),
});
} catch (dispatcherError) {
if (isDirectResponseStartTimeout(dispatcherError)) {
if (attempt === 0 && maxAttempts > 1) {
console.warn(
`[ProxyFetch] Direct response-start timeout (${directHeadersTimeoutMs}ms) on pooled dispatcher — retrying on fresh no-keep-alive dispatcher: ${targetHostForLogs}`
);
lastDispatcherError = dispatcherError;
continue;
}
throw dispatcherError;
}
const msg =
dispatcherError instanceof Error ? dispatcherError.message : String(dispatcherError);
// CAUTION: Do NOT fallback to native fetch if the error is a version mismatch (invalid onRequestStart)
// because the native fetch will definitely fail with the undici v8 dispatcher.
if (msg.includes("onRequestStart")) {
console.error(
`[ProxyFetch] Fatal version mismatch: Dispatcher (v8) vs Fetch (v6/native). Hardware upgrade or SOCKS5 config isolation required. Error: ${msg}`
);
throw dispatcherError;
}
// Retry/fallback only for connection errors, never HTTP errors.
// Only retry/fallback for connection/dispatcher errors, not HTTP errors.
// Prefer the .code property when available (more stable across undici
// versions than message-string matching); fall back to substring match
// for errors that lack a structured code.
tagProxyUnreachable(dispatcherError);
const errCode = (dispatcherError as { code?: unknown })?.code;
if (
@@ -862,7 +854,10 @@ async function patchedFetch(
msg.includes("UND_ERR")
) {
if (attempt === 0 && maxAttempts > 1) {
// Retry after a short fixed backoff on a fresh socket.
// First failure — retry once after a short backoff before giving up.
// Delay is OMNIROUTE_RETRY_BACKOFF_MS (default 10ms): a fixed backoff
// beats random jitter here because the retry opens a fresh socket, so
// jitter was pure added latency with no herd benefit.
lastDispatcherError = dispatcherError;
await new Promise((r) => setTimeout(r, RETRY_BACKOFF_MS));
continue;
@@ -878,7 +873,7 @@ async function patchedFetch(
throw tagProxyUnreachable(dispatcherError);
}
// Exhausted attempts: try proxy fallback before native fetch.
// All attempts exhausted — try proxy fallback before native fetch
if (
!tlsDirectFallback &&
source === "direct" &&
@@ -904,14 +899,20 @@ async function patchedFetch(
}
}
}
// Preserve the original monitoring phrase and append the transport cause.
// Preserve original phrase intact for monitoring: "Undici dispatcher failed, falling back to native fetch"
// #4252: append the flattened err.cause (code/syscall/errno/address) — the bare
// "fetch failed" message hides what actually broke, making bursts undiagnosable.
console.warn(
`[ProxyFetch] Undici dispatcher failed, falling back to native fetch (after retry): ${describeFetchCause(dispatcherError)}`
);
try {
return await _nativeFallback(input, options);
} catch (nativeError) {
// Surface both dispatcher and native causes immediately.
// #4252: both the undici dispatcher AND native fetch failed. Surface BOTH
// causes (server log) and tag the propagated error so the combo executor sees
// a diagnosable failure IMMEDIATELY instead of a bare "fetch failed" — the
// latter left jobs sitting until the 30s semaphore queue timeout, which then
// tripped the circuit breaker.
const detail = `dispatcher=[${describeFetchCause(dispatcherError)}] native=[${describeFetchCause(nativeError)}]`;
console.warn(`[ProxyFetch] native fetch fallback ALSO failed: ${detail}`);
if (nativeError instanceof Error) {

View File

@@ -81,7 +81,6 @@ import {
import { restoreClaudeToolName } from "../services/claudeCodeToolRemapper.ts";
import { normalizeFinalOpenAIStreamChunk } from "./openAIStreamChunk.ts";
import { collectClaudeDelta } from "./streamClaudeDelta.ts";
import { createStreamTiming, type StreamTiming } from "./streamTiming.ts";
/**
* Race a response body read against a timeout.
@@ -130,15 +129,7 @@ type StreamCompletePayload = {
clientPayload?: unknown;
error?: string | null;
errorCode?: string | null;
/**
* Time-to-first-forwarded-SSE-chunk in ms, or null when nothing was forwarded.
* NOT token-level TTFT — see open-sse/utils/streamTiming.ts for what is measured.
*/
ttft?: number | null;
/** Mean inter-chunk gap in ms (chunk-latency proxy for ITL), or null. */
itlMs?: number | null;
/** True when the stream was interrupted (timeout/abort/error) before a clean finish. */
interrupted?: boolean;
};
type StreamOptions = {
@@ -586,10 +577,7 @@ function getOpenAIIntermediateChunks(value: unknown): unknown[] {
return Array.isArray(candidate) ? candidate : [];
}
export function restoreClaudePassthroughToolUseName(
parsed: JsonRecord,
toolNameMap: unknown
): boolean {
export function restoreClaudePassthroughToolUseName(parsed: JsonRecord, toolNameMap: unknown): boolean {
const block =
parsed.content_block && typeof parsed.content_block === "object"
? (parsed.content_block as JsonRecord)
@@ -672,16 +660,6 @@ export function createSSEStream(options: StreamOptions = {}) {
performance.clearMarks("omni-request-body-size");
}
// Canonical streaming timing (TTFT / ITL / interruption). One instance per
// stream, marked from the transform below. ttft() = first-forwarded-SSE-chunk
// latency (NOT token-level) — see streamTiming.ts.
const timing: StreamTiming = createStreamTiming();
/** Forward a pre-encoded SSE chunk, marking TTFT/ITL on the way. */
const forward = (controller: TransformStreamDefaultController<Uint8Array>, bytes: Uint8Array) => {
timing.markForward();
controller.enqueue(bytes);
};
// Drop internal commentary-phase Responses output before forwarding (#6199).
// Explicit option wins; otherwise read the feature flag (default on) — resolved once per stream.
const shouldDropResponsesCommentary =
@@ -970,7 +948,7 @@ export function createSSEStream(options: StreamOptions = {}) {
clientPayloadCollector.push(event);
const output = formatSSE(event, FORMATS.CLAUDE);
reqLogger?.appendConvertedChunk?.(output);
forward(controller, encoder.encode(output));
controller.enqueue(encoder.encode(output));
}
};
@@ -995,8 +973,7 @@ export function createSSEStream(options: StreamOptions = {}) {
const errOutput = formatSSE(errorEvent, FORMATS.CLAUDE);
reqLogger?.appendConvertedChunk?.(errOutput);
clientPayloadCollector.push(errorEvent);
forward(controller, encoder.encode(errOutput));
timing.markInterrupted();
controller.enqueue(encoder.encode(errOutput));
let failureHandled = false;
if (onFailure) {
try {
@@ -1057,7 +1034,7 @@ export function createSSEStream(options: StreamOptions = {}) {
clientPayloadCollector.push(itemSanitized);
reqLogger?.appendConvertedChunk?.(output);
forwardedValuableChunk = true;
forward(controller, encoder.encode(output));
controller.enqueue(encoder.encode(output));
};
const emitFinalSseMetadata = async (
@@ -1082,7 +1059,7 @@ export function createSSEStream(options: StreamOptions = {}) {
});
if (!comment) return;
reqLogger?.appendConvertedChunk?.(comment);
forward(controller, encoder.encode(comment));
controller.enqueue(encoder.encode(comment));
};
const getResponsesReasoningKey = (payload: Record<string, unknown>): string | null => {
@@ -1169,7 +1146,7 @@ export function createSSEStream(options: StreamOptions = {}) {
clientPayloadCollector.push(syntheticEvent.body);
const output = `event: ${syntheticEvent.event}\ndata: ${JSON.stringify(syntheticEvent.body)}\n\n`;
reqLogger?.appendConvertedChunk?.(output);
forward(controller, encoder.encode(output));
controller.enqueue(encoder.encode(output));
}
};
@@ -1187,7 +1164,6 @@ export function createSSEStream(options: StreamOptions = {}) {
let failureHandled = false;
if (onFailure) {
try {
timing.markInterrupted();
failureHandled =
onFailure({
status: HTTP_STATUS.GATEWAY_TIMEOUT,
@@ -1219,7 +1195,6 @@ export function createSSEStream(options: StreamOptions = {}) {
transform(chunk, controller) {
if (streamTimedOut) return;
const now = Date.now();
timing.markByte();
lastChunkTime = now;
const text = decoder.decode(chunk, { stream: true });
buffer += text;
@@ -1278,7 +1253,7 @@ export function createSSEStream(options: StreamOptions = {}) {
const pendingOutput = passthroughEventPrefix.flush();
if (pendingOutput) {
reqLogger?.appendConvertedChunk?.(pendingOutput);
forward(controller, encoder.encode(pendingOutput));
controller.enqueue(encoder.encode(pendingOutput));
}
clearPendingPassthroughEvent();
continue;
@@ -1445,7 +1420,7 @@ export function createSSEStream(options: StreamOptions = {}) {
clientPayloadCollector.push(event);
}
reqLogger?.appendConvertedChunk?.(output);
forward(controller, encoder.encode(output));
controller.enqueue(encoder.encode(output));
injectedUsage = true;
} else {
output = `data: ${JSON.stringify(parsed)}\n\n`;
@@ -1734,7 +1709,7 @@ export function createSSEStream(options: StreamOptions = {}) {
clientPayload = parsed;
clientPayloadCollector.push(clientPayload);
reqLogger?.appendConvertedChunk?.(output);
forward(controller, encoder.encode(output));
controller.enqueue(encoder.encode(output));
continue;
}
@@ -1810,7 +1785,7 @@ export function createSSEStream(options: StreamOptions = {}) {
totalContentLength += delta.reasoning_content.length;
clientPayloadCollector.push(reasoningChunk);
reqLogger?.appendConvertedChunk?.(rOutput);
forward(controller, encoder.encode(rOutput));
controller.enqueue(encoder.encode(rOutput));
delete delta.reasoning_content;
splitMixedReasoningContent = true;
}
@@ -1989,7 +1964,7 @@ export function createSSEStream(options: StreamOptions = {}) {
}
reqLogger?.appendConvertedChunk?.(output);
forward(controller, encoder.encode(output));
controller.enqueue(encoder.encode(output));
if (failurePayload) {
let failureHandled = false;
if (onFailure) {
@@ -2029,7 +2004,7 @@ export function createSSEStream(options: StreamOptions = {}) {
if (parsed.error) {
const output = formatTranslatedStreamError(parsed, sourceFormat);
reqLogger?.appendConvertedChunk?.(output);
forward(controller, encoder.encode(output));
controller.enqueue(encoder.encode(output));
upstreamErrorForwarded = true;
doneSent = true;
continue;
@@ -2248,7 +2223,7 @@ export function createSSEStream(options: StreamOptions = {}) {
passthroughEventPrefix,
emitConvertedOutput: (output: string) => {
reqLogger?.appendConvertedChunk?.(output);
forward(controller, encoder.encode(output));
controller.enqueue(encoder.encode(output));
},
pushProviderPayload: (payload: unknown) => providerPayloadCollector.push(payload),
pushClientPayload: (payload: unknown) => clientPayloadCollector.push(payload),
@@ -2361,7 +2336,7 @@ export function createSSEStream(options: StreamOptions = {}) {
output = output.endsWith("\n") ? `${output}\n` : `${output}\n\n`;
}
reqLogger?.appendConvertedChunk?.(output);
forward(controller, encoder.encode(output));
controller.enqueue(encoder.encode(output));
}
if (shouldInjectClaudeEmptyResponseOnFlush(claudeEmptyResponseLifecycle)) {
@@ -2405,7 +2380,7 @@ export function createSSEStream(options: StreamOptions = {}) {
flushOutput = `data: ${JSON.stringify(syntheticChunk)}\n\n`;
}
reqLogger?.appendConvertedChunk?.(flushOutput);
forward(controller, encoder.encode(flushOutput));
controller.enqueue(encoder.encode(flushOutput));
passthroughAccumulatedContent = appendBoundedText(
passthroughAccumulatedContent,
passthroughBufferedTextualToolCallContent
@@ -2422,7 +2397,7 @@ export function createSSEStream(options: StreamOptions = {}) {
totalContentLength += thinkFlush.addedLength;
clientPayloadCollector.push(thinkFlush.syntheticChunk);
reqLogger?.appendConvertedChunk?.(thinkFlush.flushOutput);
forward(controller, encoder.encode(thinkFlush.flushOutput));
controller.enqueue(encoder.encode(thinkFlush.flushOutput));
}
// Estimate usage if provider didn't return valid usage
@@ -2456,7 +2431,7 @@ export function createSSEStream(options: StreamOptions = {}) {
);
const finishOutput = `data: ${JSON.stringify(syntheticFinishChunk)}\n\n`;
reqLogger?.appendConvertedChunk?.(finishOutput);
forward(controller, encoder.encode(finishOutput));
controller.enqueue(encoder.encode(finishOutput));
clientPayloadCollector.push(syntheticFinishChunk);
}
await emitFinalSseMetadata(controller, usage);
@@ -2465,7 +2440,7 @@ export function createSSEStream(options: StreamOptions = {}) {
clientPayloadCollector.push({ done: true });
const doneOutput = "data: [DONE]\n\n";
reqLogger?.appendConvertedChunk?.(doneOutput);
forward(controller, encoder.encode(doneOutput));
controller.enqueue(encoder.encode(doneOutput));
}
}
// Notify caller for call log persistence (include full response body with accumulated content)
@@ -2539,9 +2514,6 @@ export function createSSEStream(options: StreamOptions = {}) {
status: 200,
usage,
responseBody,
ttft: timing.ttftMs(),
itlMs: timing.avgItlMs(),
interrupted: timing.interrupted,
// #9315 switched the summary to the accumulated responseBody to avoid
// stale/truncated event data — but responseBody here is synthesized in
// chat-completion shape, which loses the Responses API `response` object.
@@ -2644,7 +2616,6 @@ export function createSSEStream(options: StreamOptions = {}) {
let failureHandled = false;
if (onFailure) {
try {
timing.markInterrupted();
failureHandled =
onFailure({
status: err.status,
@@ -2664,9 +2635,6 @@ export function createSSEStream(options: StreamOptions = {}) {
status: err.status,
usage: state?.usage,
responseBody: errorBody,
ttft: timing.ttftMs(),
itlMs: timing.avgItlMs(),
interrupted: timing.interrupted,
error: err.message,
errorCode: err.code,
providerPayload: providerPayloadCollector.build(
@@ -2763,7 +2731,7 @@ export function createSSEStream(options: StreamOptions = {}) {
clientPayloadCollector.push({ done: true });
const doneOutput = "data: [DONE]\n\n";
reqLogger?.appendConvertedChunk?.(doneOutput);
forward(controller, encoder.encode(doneOutput));
controller.enqueue(encoder.encode(doneOutput));
}
}

View File

@@ -1,83 +0,0 @@
/**
* Canonical streaming timing instrumentation (TTFT / ITL / interruption).
*
* One reusable seam for measuring the streaming path. It is created once per
* stream and marked from the SSE transform:
*
* markByte() — first upstream chunk received (bytes arrived from provider)
* markForward() — first chunk forwarded to the client (first SSE chunk enqueued)
*
* `ttft()` is therefore **first-forwarded-SSE-chunk latency**, NOT token-level
* TTFT. We document that distinction explicitly: a single SSE chunk can carry
* zero, one, or many tokens, and chunk boundaries do not map to token
* boundaries. If a future implementation can measure actual token timing it
* should extend this seam, not bypass it.
*
* ITL (inter-token latency) is approximated by the mean gap between forwarded
* SSE chunks (bounded sample window). It is a chunk-latency proxy, again not
* true token timing — callers must label it as such.
*
* The object is cheap to construct, plain mutable state, and safe under the
* event loop's single thread (each stream owns its own instance).
*/
export interface StreamTiming {
startedAt: number;
firstByteAt: number | null;
firstForwardAt: number | null;
lastForwardAt: number | null;
/** Mean gap between forwarded chunks (ms), bounded window. */
interChunkGaps: number[];
forwardedChunks: number;
interrupted: boolean;
markByte(): void;
markForward(): void;
markInterrupted(): void;
/** First-forwarded-SSE-chunk latency in ms, or null if nothing was forwarded. */
ttftMs(): number | null;
/** Mean inter-chunk gap in ms, or null when fewer than 2 chunks were forwarded. */
avgItlMs(): number | null;
/** Time from stream start to completion (ms). */
totalMs(): number;
}
/** Max number of inter-chunk samples kept (bounds memory). */
const MAX_INTER_CHUNK_GAPS = 32;
export function createStreamTiming(): StreamTiming {
const timing: StreamTiming = {
startedAt: Date.now(),
firstByteAt: null,
firstForwardAt: null,
lastForwardAt: null,
interChunkGaps: [],
forwardedChunks: 0,
interrupted: false,
markByte() {
if (this.firstByteAt === null) this.firstByteAt = Date.now();
},
markForward() {
const now = Date.now();
if (this.firstForwardAt === null) this.firstForwardAt = now;
if (this.lastForwardAt !== null && this.interChunkGaps.length < MAX_INTER_CHUNK_GAPS) {
this.interChunkGaps.push(now - this.lastForwardAt);
}
this.lastForwardAt = now;
this.forwardedChunks += 1;
},
markInterrupted() {
this.interrupted = true;
},
ttftMs() {
return this.firstForwardAt === null ? null : this.firstForwardAt - this.startedAt;
},
avgItlMs() {
if (this.interChunkGaps.length === 0) return null;
const sum = this.interChunkGaps.reduce((a, b) => a + b, 0);
return sum / this.interChunkGaps.length;
},
totalMs() {
return Date.now() - this.startedAt;
},
};
return timing;
}

View File

@@ -89,7 +89,6 @@
"gen:provider-reference": "bun scripts/docs/gen-provider-reference.ts",
"bench:compression": "bun scripts/compression/benchmark.ts",
"bench:heap-body": "node --expose-gc --import tsx/esm scripts/perf/request-body-heap.ts",
"bench:routing-events": "node --import tsx/esm scripts/perf/routing-events-bench.ts",
"eval:compression": "node --import tsx scripts/compression-eval/index.ts",
"eval:router": "node --import tsx scripts/router-eval/index.ts",
"eval:router:compare": "node --import tsx scripts/router-eval/compare.ts",

View File

@@ -1,175 +0,0 @@
/**
* Routing feedback foundation benchmark (v2 — honest comparison).
*
* v1 reported a single "~0.2µs/request" figure. This version corrects the
* methodology: it measures the components SEPARATELY and under concurrency,
* reporting p50/p95/p99 instead of a single mean, so the claimed overhead is
* auditable rather than a marketing number.
*
* Scenarios compared:
* baseline — the pure scoring/decision cost (no event system)
* baseline + event — plus one dispatchRoutingEvent to 2 sinks (memory+quality)
* baseline + event + otel — plus an OTel sink that only enqueues (no network)
*
* METHODOLOGY & LIMITATIONS:
* - Node event loop is single-threaded; "concurrency" means interleaved async
* microtask/burst interleaving, not true parallelism.
* - p95/p99 are measured per-op over a big N with high-resolution timers.
* - No network I/O is performed (OTel flush is deliberately not fired).
* - Numbers are machine-specific; treat them as relative, not absolute.
*
* Usage:
* npm run bench:routing-events
* npm run bench:routing-events -- --events 200000
*/
import { performance } from "node:perf_hooks";
import {
dispatchRoutingEvent,
MemoryRoutingEventStore,
registerRoutingEventSink,
type RoutingEvent,
type RoutingEventSink,
} from "../../open-sse/services/routing/events.ts";
import { recordQualityEvent } from "../../open-sse/services/routing/quality.ts";
import { OtlpHttpsEventSink } from "../../open-sse/services/routing/otel.ts";
import {
calculateFactors,
calculateScore,
DEFAULT_WEIGHTS,
type ProviderCandidate,
} from "../../open-sse/services/autoCombo/scoring.ts";
const N = Number(process.argv[2] === "--events" ? (process.argv[3] ?? 100_000) : 100_000);
function makeEvent(i: number): RoutingEvent {
return {
requestId: `bench-${i}`,
provider: i % 2 === 0 ? "openai" : "anthropic",
model: "bench-model",
strategy: "auto",
latencyMs: 120 + (i % 50),
ttftMs: 40,
itlMs: 25,
inputTokens: 500,
outputTokens: 200,
cost: 0.01,
retries: 0,
fallbackUsed: false,
outcome: i % 100 === 0 ? "malformed" : "success",
status: 200,
finishReason: "stop",
connectionId: null,
ts: Date.now(),
};
}
function bench(name: string, iterations: number, fn: (i: number) => number): void {
// Warmup
for (let i = 0; i < Math.min(10_000, iterations); i++) fn(i);
const start = performance.now();
for (let i = 0; i < iterations; i++) fn(i);
const elapsedMs = performance.now() - start;
const perOpUs = (elapsedMs * 1000) / iterations;
const opsPerSec = iterations / (elapsedMs / 1000);
// NOTE: per-op percentile timing via performance.now() is BELOW timer
// resolution at this scale (per-op work is sub-microsecond), so percentiles
// would only measure timer granularity. Aggregate µs/op + throughput are the
// honest metrics here.
console.log(
`${name.padEnd(46)} ${iterations.toLocaleString()} ops in ${elapsedMs.toFixed(1)}ms | ` +
`${perOpUs.toFixed(3)}µs/op | ${Math.round(opsPerSec).toLocaleString()} ops/s`
);
}
// Shared sink set for the "event" and "otel" scenarios.
const store = new MemoryRoutingEventStore(500);
registerRoutingEventSink(store);
const qualitySink: RoutingEventSink = {
name: "quality",
record: (e) => recordQualityEvent(e),
};
registerRoutingEventSink(qualitySink);
// OTel sink that only enqueues (flush interval set absurdly high; never fires in-run).
const otelSink = new OtlpHttpsEventSink({
endpoint: "http://127.0.0.1:1", // unreachable; record() never touches the network
flushIntervalMs: 1_000_000,
});
registerRoutingEventSink(otelSink);
const candidate = (quality: number): ProviderCandidate => ({
provider: "p",
model: "m",
quotaRemaining: 100,
quotaTotal: 100,
circuitBreakerState: "CLOSED",
costPer1MTokens: 1,
p95LatencyMs: 100,
latencyStdDev: 10,
errorRate: 0,
quality,
});
const pool = [candidate(0.9), candidate(0.5), candidate(0.2)];
console.log(
`\nRouting events benchmark (${N.toLocaleString()} iterations, 2 sinks + otel-enqueue)\n`
);
// baseline: the scoring/decision cost the router already pays WITHOUT the event system.
bench("baseline: calculateFactors+Score", N, (i) => {
const c = pool[i % pool.length];
const f = calculateFactors(c, pool, "general", () => 0.5);
return calculateScore(f, DEFAULT_WEIGHTS);
});
// baseline + event: the production hot-path cost (dispatch to memory+quality sinks).
bench("baseline + RoutingEvent (2 sinks)", N, (i) => {
const c = pool[i % pool.length];
const f = calculateFactors(c, pool, "general", () => 0.5);
const score = calculateScore(f, DEFAULT_WEIGHTS);
dispatchRoutingEvent(makeEvent(i));
return score;
});
// baseline + event + OTel-enqueue: adds the third sink (still no network I/O).
bench("baseline + event + OTel enqueue", N, (i) => {
const c = pool[i % pool.length];
const f = calculateFactors(c, pool, "general", () => 0.5);
const score = calculateScore(f, DEFAULT_WEIGHTS);
dispatchRoutingEvent(makeEvent(i));
return score;
});
// Concurrency: bursts interleaved on the event loop.
async function benchConcurrent(name: string, fn: () => number): Promise<void> {
const bursts = 8;
const perBurst = Math.ceil(N / bursts);
const start = performance.now();
await Promise.all(
Array.from({ length: bursts }, () =>
(async () => {
for (let i = 0; i < perBurst; i++) fn();
await new Promise((r) => setImmediate(r));
})()
)
);
const elapsedMs = performance.now() - start;
const totalOps = bursts * perBurst;
console.log(
`${name.padEnd(46)} ${totalOps.toLocaleString()} ops in ${elapsedMs.toFixed(1)}ms ` +
`(${(elapsedMs * 1000) / totalOps}µs/op aggregate)`
);
}
console.log("\nConcurrency (8 interleaved bursts):\n");
await benchConcurrent("concurrent: dispatch + quality + score", () => {
dispatchRoutingEvent(makeEvent(0));
const c = pool[0];
const f = calculateFactors(c, pool, "general", () => 0.5);
return calculateScore(f, DEFAULT_WEIGHTS);
});
console.log(`\nOTel sink stats: ${JSON.stringify(otelSink.getStats())}`);
otelSink.stop();
console.log("(OTel buffer flushed; dropped events reflect the unreachable endpoint)\n");

View File

@@ -19,7 +19,6 @@ import { getProviderDisplayLabel } from "@/shared/utils/providerDisplayLabel";
import { useIsElectron, useOpenExternal } from "@/shared/hooks/useElectron";
import { HomeProviderTopologySection } from "./HomeProviderTopologySection";
import { shouldShowProviderTopologyOnHome } from "./homeAppearance";
import HomeRecentRequests from "../home/HomeRecentRequests";
type UpdateStep = {
step: string;
@@ -1127,15 +1126,12 @@ export default function HomePageClient({ machineId }: HomePageClientProps) {
)}
{showProviderTopologyOnHome && (
<div className="grid grid-cols-1 lg:grid-cols-[2fr_1fr] gap-3">
<HomeProviderTopologySection
providers={topologyProviders}
lastProvider={lastProvider}
errorProvider={errorProvider}
enabled={showProviderTopologyOnHome}
/>
<HomeRecentRequests enabled={showProviderTopologyOnHome} />
</div>
<HomeProviderTopologySection
providers={topologyProviders}
lastProvider={lastProvider}
errorProvider={errorProvider}
enabled={showProviderTopologyOnHome}
/>
)}
{/* Provider Models Modal */}

View File

@@ -1,207 +0,0 @@
"use client";
import { useCallback, useEffect, useState } from "react";
import { useTranslations } from "next-intl";
import { Card } from "@/shared/components";
import { fmtCompact } from "@/shared/utils/formatting";
/**
* Home-page "Recent Requests" panel — the live request feed that sits beside the
* Provider Topology graph (parity with 9Router's Usage view).
*
* ## Data source
* Fed by POLLING `GET /api/usage/call-logs?limit=N` every ~3s, NOT by the live
* WebSocket. The WS is used elsewhere for the in-flight beam (it emits
* `request.started` and, since the stuck-latch fix, `request.completed`/`request.failed`),
* but its payload carries no tokens/latency/status and no persisted history — so it
* can't back a "recent requests" table on its own. The call-logs endpoint merges the
* in-memory active/completed entries with persisted rows and returns them newest-first
* (active on top), which is exactly this feed.
*
* The poll is gated by `enabled` (the same `showProviderTopologyOnHome` flag that
* gates the topology section) AND page visibility, so a backgrounded tab pauses.
*/
const POLL_INTERVAL_MS = 3000;
// Rows shown after client-side filtering of connection-test rows.
const RECENT_LIMIT = 20;
// Fetch a wider window than we display so filtering out connection-test rows
// (a burst of "Test connection" clicks) can't starve the feed below RECENT_LIMIT.
const FETCH_LIMIT = 60;
type CallLogRow = {
id?: string;
timestamp?: string;
status?: number;
model?: string;
provider?: string;
providerDisplay?: string | null;
path?: string;
sourceFormat?: string | null;
targetFormat?: string | null;
tokens?: { in?: number; out?: number };
error?: string | null;
active?: boolean;
completed?: boolean;
};
/**
* Connection tests write real `call_logs` rows (provider "Test connection" button →
* `/api/providers/[id]/test`) with fixed markers: model `connection-test`, path
* `/api/providers/test`, sourceFormat/targetFormat `test`. Those are health probes,
* not user traffic, so they must not clutter the Recent Requests feed (matching how
* 9Router keeps its Usage list to real calls). Drop any row carrying a test marker.
*/
function isConnectionTestRow(row: CallLogRow): boolean {
return (
row.model === "connection-test" ||
row.sourceFormat === "test" ||
row.targetFormat === "test" ||
row.path === "/api/providers/test"
);
}
type RequestState = "active" | "error" | "ok";
function requestState(row: CallLogRow): RequestState {
if (row.active || row.status === 0) return "active";
if (row.error || (typeof row.status === "number" && row.status >= 400)) return "error";
return "ok";
}
function timeAgo(timestamp: string | undefined, nowMs: number): string {
if (!timestamp) return "";
const then = Date.parse(timestamp);
if (!Number.isFinite(then)) return "";
const diff = Math.max(0, Math.floor((nowMs - then) / 1000));
if (diff < 60) return `${diff}s`;
if (diff < 3600) return `${Math.floor(diff / 60)}m`;
if (diff < 86400) return `${Math.floor(diff / 3600)}h`;
return `${Math.floor(diff / 86400)}d`;
}
const STATE_DOT: Record<RequestState, string> = {
active: "bg-primary animate-pulse",
error: "bg-red-500",
ok: "bg-green-500",
};
export default function HomeRecentRequests({ enabled = true }: { enabled?: boolean }) {
const t = useTranslations("home");
const [rows, setRows] = useState<CallLogRow[]>([]);
const [loaded, setLoaded] = useState(false);
// A ticking clock so the relative "When" column updates without re-fetching.
const [nowMs, setNowMs] = useState(() => Date.now());
useEffect(() => {
if (!enabled) return;
const id = setInterval(() => setNowMs(Date.now()), 1000);
return () => clearInterval(id);
}, [enabled]);
const load = useCallback(async (signal: AbortSignal) => {
try {
const res = await fetch(`/api/usage/call-logs?limit=${FETCH_LIMIT}&excludeTests=1`, {
cache: "no-store",
signal,
});
if (!res.ok) return;
const data = await res.json();
if (signal.aborted) return;
const filtered = Array.isArray(data)
? (data as CallLogRow[]).filter((row) => !isConnectionTestRow(row)).slice(0, RECENT_LIMIT)
: [];
setRows(filtered);
setLoaded(true);
} catch (error) {
const isAbort = error instanceof DOMException && error.name === "AbortError";
if (!isAbort) console.error("Failed to load recent requests:", error);
}
}, []);
useEffect(() => {
if (!enabled) return;
let cancelled = false;
let timeoutId: ReturnType<typeof setTimeout> | null = null;
let controller: AbortController | null = null;
const tick = async () => {
// Pause polling while the tab is backgrounded; resume on next tick.
if (document.visibilityState === "visible") {
const currentController = new AbortController();
controller = currentController;
await load(currentController.signal);
if (controller === currentController) controller = null;
}
if (!cancelled) timeoutId = setTimeout(tick, POLL_INTERVAL_MS);
};
tick();
return () => {
cancelled = true;
if (timeoutId) clearTimeout(timeoutId);
controller?.abort();
};
}, [enabled, load]);
return (
<Card padding="sm" className="flex min-w-0 flex-col overflow-hidden h-[300px] sm:h-[420px]">
<div className="pb-2 mb-1 border-b border-border shrink-0">
<span className="text-xs font-semibold uppercase tracking-wide text-text-muted">
{t("recentRequests")}
</span>
</div>
{loaded && rows.length === 0 ? (
<div className="flex-1 flex items-center justify-center text-sm text-text-muted">
{t("recentRequestsEmpty")}
</div>
) : (
<div className="flex-1 overflow-y-auto -mx-1 px-1">
<table className="w-full min-w-0 border-collapse text-xs">
<thead className="sticky top-0 z-10 bg-surface">
<tr className="border-b border-border text-text-muted">
<th className="w-2 py-1.5" />
<th className="py-1.5 text-left font-semibold">{t("recentRequestsModel")}</th>
<th className="py-1.5 text-right font-semibold whitespace-nowrap">
{t("recentRequestsTokens")}
</th>
<th className="py-1.5 text-right font-semibold">{t("recentRequestsWhen")}</th>
</tr>
</thead>
<tbody className="divide-y divide-border/50">
{rows.map((row, i) => {
const state = requestState(row);
return (
<tr key={row.id || i} className="hover:bg-bg-subtle transition-colors">
<td className="py-1.5">
<span className={`block size-1.5 rounded-full ${STATE_DOT[state]}`} />
</td>
<td
className="py-1.5 font-mono truncate max-w-[140px]"
title={row.model || ""}
>
{row.model || "—"}
</td>
<td className="py-1.5 text-right whitespace-nowrap">
<span className="text-primary">{fmtCompact(row.tokens?.in)}</span>{" "}
<span className="text-green-500">{fmtCompact(row.tokens?.out)}</span>
</td>
<td className="py-1.5 text-right whitespace-nowrap text-text-muted">
{state === "active" ? (
<span className="text-primary"></span>
) : (
timeAgo(row.timestamp, nowMs)
)}
</td>
</tr>
);
})}
</tbody>
</table>
</div>
)}
</Card>
);
}

View File

@@ -143,9 +143,6 @@ export async function GET(request: Request) {
if (searchParams.get("correlationId")) filter.correlationId = searchParams.get("correlationId");
if (searchParams.get("limit")) filter.limit = parseInt(searchParams.get("limit"));
if (searchParams.get("offset")) filter.offset = parseInt(searchParams.get("offset"));
// Home Recent Requests feed sets excludeTests=1 so connection-test probe rows
// are dropped at the SQL layer (before LIMIT), not client-side after slicing.
if (searchParams.get("excludeTests") === "1") filter.excludeTests = true;
const [logs, connections, providerNodes] = await Promise.all([
getCallLogs(filter),

View File

@@ -1,72 +0,0 @@
/**
* GET /v1/explain/routing — routing explainability + feedback state.
*
* Returns the most recent routing events (bounded in-memory ring buffer) and
* the per-provider/model quality snapshot produced by the feedback foundation
* (open-sse/services/routing). This is REAL decision data — the events were
* emitted by the request hot path, not recomputed after the fact.
*
* Safety: only routing metadata (provider/model/strategy/timing/tokens/outcome/
* status/finish_reason). Never prompts, bodies, headers, credentials, accounts.
*
* Auth mirrors /v1/combos: valid Bearer API key or dashboard session. With
* REQUIRE_API_KEY=false (single-user local deployments) anonymous read is
* allowed, matching /v1/models behavior.
*/
import { NextResponse } from "next/server";
import { errorResponse } from "@omniroute/open-sse/utils/error.ts";
import { HTTP_STATUS } from "@omniroute/open-sse/config/constants.ts";
import { extractApiKey, isValidApiKey } from "@/sse/services/auth";
import { isDashboardSessionAuthenticated } from "@/shared/utils/apiAuth";
import { isRequireApiKeyEnabled } from "@/shared/utils/featureFlags";
import {
recentRoutingEvents,
routingQualitySnapshot,
routingOtelStats,
initRoutingObservability,
classifyQuality,
} from "@omniroute/open-sse/services/routing/index.ts";
export async function OPTIONS() {
return new Response(null, {
headers: {
"Access-Control-Allow-Methods": "GET, OPTIONS",
"Access-Control-Allow-Headers": "*",
},
});
}
export async function GET(request: Request) {
const apiKeyRaw = extractApiKey(request);
const apiKeyOk = apiKeyRaw ? await isValidApiKey(apiKeyRaw) : false;
const dashboardOk = !apiKeyOk ? await isDashboardSessionAuthenticated(request) : false;
if (!apiKeyOk && !dashboardOk && isRequireApiKeyEnabled()) {
return errorResponse(HTTP_STATUS.UNAUTHORIZED, "Authentication required");
}
try {
const limit = Math.min(
500,
Math.max(1, Number(new URL(request.url).searchParams.get("limit")) || 50)
);
const { sinks, otelEnabled } = initRoutingObservability();
const quality = routingQualitySnapshot(limit).map((q) => ({
...q,
classification: classifyQuality(q),
}));
return NextResponse.json(
{
object: "routing_explain",
sinks,
otelEnabled,
events: recentRoutingEvents(limit),
quality,
otel: routingOtelStats(),
},
{ headers: { "Cache-Control": "no-store" } }
);
} catch {
return errorResponse(HTTP_STATUS.SERVER_ERROR, "Failed to build routing explain payload");
}
}

View File

@@ -7,61 +7,6 @@ export async function OPTIONS() {
return handleCorsOptions();
}
const DEFAULT_LIST_LIMIT = 20;
const MAX_LIST_LIMIT = 10000;
export function parseFilesListQuery(searchParams: URLSearchParams):
| {
ok: true;
limit: number;
after: string | undefined;
order: "asc" | "desc";
purpose: string | undefined;
}
| { ok: false; response: Response } {
const rawLimit = searchParams.get("limit");
let limit = DEFAULT_LIST_LIMIT;
if (rawLimit !== null) {
if (!/^\d+$/.test(rawLimit)) {
return {
ok: false,
response: NextResponse.json(
{ error: { message: "limit must be a positive integer", type: "invalid_request_error" } },
{ status: 400, headers: CORS_HEADERS }
),
};
}
limit = Number.parseInt(rawLimit, 10);
if (limit < 1 || limit > MAX_LIST_LIMIT) {
return {
ok: false,
response: NextResponse.json(
{
error: {
message: `limit must be between 1 and ${MAX_LIST_LIMIT}`,
type: "invalid_request_error",
},
},
{ status: 400, headers: CORS_HEADERS }
),
};
}
}
const orderParam = searchParams.get("order");
const order = orderParam === "asc" ? "asc" : "desc";
return {
ok: true,
limit,
after: searchParams.get("after") || undefined,
order,
purpose: searchParams.get("purpose") || undefined,
};
}
export async function POST(request: Request) {
const scope = await getApiKeyRequestScope(request);
if (scope.rejection) return scope.rejection;
@@ -133,9 +78,10 @@ export async function GET(request: Request) {
const apiKeyId = scope.apiKeyId;
const { searchParams } = new URL(request.url);
const parsed = parseFilesListQuery(searchParams);
if (!parsed.ok) return parsed.response;
const { limit, after, order, purpose } = parsed;
const limit = Math.min(Number.parseInt(searchParams.get("limit") || "20") || 20, 10000);
const after = searchParams.get("after") || undefined;
const order = (searchParams.get("order") as "asc" | "desc") || "desc";
const purpose = searchParams.get("purpose") || undefined;
// We fetch limit + 1 to check if there are more items
const files = listFiles({

View File

@@ -3,7 +3,6 @@ import {
handleCodexImageEdit,
handleImageEdit,
handleOpenAIImageEdit,
handleOpenRouterImageEdit,
} from "@omniroute/open-sse/handlers/imageGeneration.ts";
import {
handleFalAIImageEdit,
@@ -586,55 +585,6 @@ async function postHandler(request: Request, _context?: unknown) {
});
}
// Built-in OpenRouter uses its unified Image API for reference-image
// edits: POST /api/v1/images with input_references. Forward through the
// provider-specific adapter (#10197), rather than the multipart
// /images/edits path used by custom OpenAI-compatible nodes.
if (providerConfig?.id === "openrouter") {
const credentials = await getProviderCredentialsWithQuotaPreflight(
parsed.provider,
null,
allowedConnections,
resolvedModel
);
if (!credentials) {
return errorResponse(
HTTP_STATUS.UNAUTHORIZED,
`No credentials for provider: ${parsed.provider}`
);
}
if (credentials.allRateLimited) {
return unavailableResponse(
HTTP_STATUS.RATE_LIMITED,
`[${parsed.provider}] All accounts rate limited`,
credentials.retryAfter,
credentials.retryAfterHuman
);
}
const result = await handleOpenRouterImageEdit({
provider: parsed.provider,
model: parsed.model,
baseUrl: providerConfig.baseUrl,
credentials,
prompt,
imageBytes,
imageMime,
size: size ?? undefined,
n: 1,
log,
});
if (result.success) {
await clearRecoveredProviderState(credentials);
return jsonResponse(result.data);
}
return jsonResponse(
toJsonErrorPayload(result.error, "Image edit provider error"),
result.status
);
}
// Other built-in providers do not expose an OpenAI-compatible edit endpoint.
if (providerConfig) {
return errorResponse(

View File

@@ -1800,11 +1800,6 @@
"updateStarted": "Update started...",
"reloadingPageAutomatically": "Reloading page automatically...",
"providerTopology": "Provider Topology",
"recentRequests": "Recent Requests",
"recentRequestsEmpty": "No requests yet.",
"recentRequestsModel": "Model",
"recentRequestsTokens": "In / Out",
"recentRequestsWhen": "When",
"downloadDmg": "Download DMG (macOS)",
"downloadDmgDescription": "A new version of the OmniRoute desktop app is available. Please download and install the macOS DMG installer to update (current: v{version}).",
"downloadExe": "Download EXE (Windows)",

View File

@@ -700,14 +700,6 @@ export async function getCallLogs(filter: any = {}) {
if (filter.combo) {
conditions.push("cl.combo_name IS NOT NULL");
}
if (filter.excludeTests) {
// Home "Recent Requests" is an allowlist of real provider inference, not a
// blacklist of known backend log types. Persisted provider requests enter via
// the public gateway namespaces (/v1/* or /api/v1/*); internal management work
// (connection tests, model sync, and future /api/providers/* jobs) does not.
// Apply this before LIMIT so backend rows can never displace real traffic.
conditions.push(`(cl.path LIKE '/v1/%' OR cl.path LIKE '/api/v1/%')`);
}
if (filter.since) {
conditions.push("cl.timestamp >= @since");
params.since = filter.since instanceof Date ? filter.since.toISOString() : String(filter.since);

View File

@@ -84,7 +84,6 @@ const FACTOR_KEYS: ComboScoringInspectorFactorKey[] = [
"sessionAvailability",
"resetWindowAffinity",
"connectionDensity",
"quality",
];
function roundNumber(value: number, digits = 4): number {
@@ -316,21 +315,14 @@ function factorBreakdown(
weights: ScoringWeights,
context: CandidateContext
): ComboScoringInspectorFactor[] {
return FACTOR_KEYS.map((key) => {
// Optional factors (cacheAffinity/sessionAvailability/quality) default to
// their scoring neutral (1 for a factor, 0 for a weight) so the contribution
// sum stays consistent with calculateScore.
const value = factors[key] ?? 1;
const weight = weights[key] ?? 0;
return {
key,
value: roundNumber(value),
weight: roundNumber(weight),
contribution: roundNumber(value * weight),
source: context.sources[key] ?? "default",
note: context.notes[key],
};
}).sort((left, right) => Math.abs(right.contribution) - Math.abs(left.contribution));
return FACTOR_KEYS.map((key) => ({
key,
value: roundNumber(factors[key]),
weight: roundNumber(weights[key]),
contribution: roundNumber(factors[key] * weights[key]),
source: context.sources[key] ?? "default",
note: context.notes[key],
})).sort((left, right) => Math.abs(right.contribution) - Math.abs(left.contribution));
}
function targetForecastMap(targets: ComboForecastTarget[]): Map<string, ComboForecastTarget> {

View File

@@ -66,14 +66,7 @@ const BEDROCK_CLAUDE_ALIASES = (...modelIds: string[]) => [
// Provider discovery/sync sources can under-report GLM-5.2 IDs as 128K.
// Keep native/bare Z.AI GLM-5.2 context authoritative, but do not blindly apply
// it to every provider-wrapped alias: hosted providers can and do cap lower.
const AUTHORITATIVE_CONTEXT_WINDOW_MODEL_IDS = new Set([
"glm-5.3",
"glm-5.3-high",
"glm-5.3-low",
"glm-5.2",
"glm-5.2-high",
"glm-5.2-max",
]);
const AUTHORITATIVE_CONTEXT_WINDOW_MODEL_IDS = new Set(["glm-5.2", "glm-5.2-high", "glm-5.2-max"]);
const AUTHORITATIVE_PROVIDER_CONTEXT_WINDOWS = new Map<string, number>([
["cloudflare-ai/@cf/zai-org/glm-5.2", 262144],
// Hugging Face Router has 1M-capable backends, but bare routing can select
@@ -574,30 +567,6 @@ export const MODEL_SPECS: Record<string, ModelSpec> = {
supportsTools: true,
},
// ── Z.AI GLM-5.3 (1M context mirrored from 5.2 — same base model; 128K max
// output; effort via reasoning_effort param, tiers are OmniRoute aliases) ──
"glm-5.3": {
maxOutputTokens: 131072,
contextWindow: 1000000,
thinkingBudgetCap: 38912,
supportsThinking: true,
supportsTools: true,
},
"glm-5.3-high": {
maxOutputTokens: 131072,
contextWindow: 1000000,
thinkingBudgetCap: 38912,
supportsThinking: true,
supportsTools: true,
},
"glm-5.3-low": {
maxOutputTokens: 131072,
contextWindow: 1000000,
thinkingBudgetCap: 38912,
supportsThinking: true,
supportsTools: true,
},
// ── Z.AI GLM-5.2 (1M context, 128K max output, effort tiers) ────
"glm-5.2": {
maxOutputTokens: 131072,

View File

@@ -111,30 +111,6 @@ export const CLAUDE_SONNET_5_PRICING = {
};
export const GLM_PRICING = {
// GLM-5.3 (2026-08-14): Z.ai hasn't published 5.3 rates yet — mirrored from
// GLM-5.2 (same base model; 5.1 and 5.2 also share identical rates).
// Correct when https://docs.z.ai/guides/overview/pricing lists glm-5.3.
"glm-5.3": {
input: 1.2,
output: 5,
cached: 0.3,
reasoning: 5,
cache_creation: 1.2,
},
"glm-5.3-high": {
input: 1.2,
output: 5,
cached: 0.3,
reasoning: 5,
cache_creation: 1.2,
},
"glm-5.3-low": {
input: 1.2,
output: 5,
cached: 0.3,
reasoning: 5,
cache_creation: 1.2,
},
"glm-5.2": {
input: 1.2,
output: 5,

View File

@@ -216,8 +216,6 @@ export function isLocalProvider(providerId: unknown): boolean {
}
export const SELF_HOSTED_CHAT_PROVIDER_IDS = new Set([
"mlx-gemma",
"mlx-qwen",
"ollama-local",
"lm-studio",
"vllm",
@@ -274,8 +272,6 @@ export function providerAllowsOptionalApiKey(providerId: unknown): boolean {
const BULK_API_KEY_EXCLUDED = new Set([
"vertex",
"vertex-partner",
"mlx-gemma",
"mlx-qwen",
"ollama-local",
"grok-web",
"perplexity-web",

View File

@@ -3,32 +3,6 @@
* Pure data literal; re-exported by the providers.ts barrel. No behavior change.
*/
export const LOCAL_PROVIDERS = {
"mlx-gemma": {
id: "mlx-gemma",
alias: "mlx-gemma",
name: "MLX Gemma 26B",
icon: "memory",
color: "#8B5CF6",
textIcon: "MG",
website: "https://github.com/ml-explore/mlx",
authHint:
"No API key required. Runs mlx-lm server locally on port 11435. Requires uv and mlx-lm installed. Model: mlx-community/gemma-4-26B-A4B-it-qat-q4_0-mlx-aligned (~15.9GB peak memory).",
localDefault: "http://localhost:11435/v1",
passthroughModels: false,
},
"mlx-qwen": {
id: "mlx-qwen",
alias: "mlx-qwen",
name: "MLX Qwen 3.8 27B",
icon: "memory",
color: "#EC4899",
textIcon: "MQ",
website: "https://github.com/ml-explore/mlx",
authHint:
"No API key required. Runs mlx-lm server locally on port 11436. Requires uv and mlx-lm installed. Model: maglun/Qwen3.8-27B-MLX-Mixed-3.80bpw (~13.1GB peak memory).",
localDefault: "http://localhost:11436/v1",
passthroughModels: false,
},
"ollama-local": {
id: "ollama-local",
alias: "ollama",

View File

@@ -272,8 +272,7 @@ export type ComboScoringInspectorFactorKey =
| "cacheAffinity"
| "sessionAvailability"
| "resetWindowAffinity"
| "connectionDensity"
| "quality";
| "connectionDensity";
export type ComboScoringInspectorSource =
"combo_health" | "combo_forecast" | "combo_autopilot" | "runtime" | "default";

View File

@@ -1,77 +0,0 @@
import { describe, it } from "node:test";
import assert from "node:assert";
import { createFile, deleteFile } from "@/lib/db/files";
import { GET, parseFilesListQuery } from "@/app/api/v1/files/route";
describe("GET /v1/files limit validation", () => {
it("defaults to 20 when limit is absent", () => {
const parsed = parseFilesListQuery(new URLSearchParams("order=asc"));
assert.equal(parsed.ok, true);
if (!parsed.ok) return;
assert.equal(parsed.limit, 20);
});
it("parses an explicit positive integer limit", () => {
const parsed = parseFilesListQuery(new URLSearchParams("limit=2&order=asc&purpose=batch"));
assert.equal(parsed.ok, true);
if (!parsed.ok) return;
assert.equal(parsed.limit, 2);
assert.equal(parsed.order, "asc");
assert.equal(parsed.purpose, "batch");
});
it("rejects non-integer, zero, and oversized limits", async () => {
for (const rawLimit of ["abc", "1.5", "-1", "0", "10001"]) {
const parsed = parseFilesListQuery(
new URLSearchParams(`limit=${encodeURIComponent(rawLimit)}`)
);
assert.equal(parsed.ok, false, `limit=${rawLimit} should be rejected`);
if (parsed.ok) continue;
assert.equal(parsed.response.status, 400);
const body = await parsed.response.json();
assert.equal(body.error.type, "invalid_request_error");
}
});
it("returns only the requested number of files over HTTP", async () => {
const created = [
createFile({
bytes: 1,
filename: "test-files-limit-http-a.txt",
purpose: "assistants",
content: Buffer.from("a"),
mimeType: "text/plain",
}),
createFile({
bytes: 1,
filename: "test-files-limit-http-b.txt",
purpose: "assistants",
content: Buffer.from("b"),
mimeType: "text/plain",
}),
];
try {
const response = await GET(
new Request("http://localhost/v1/files?limit=1&purpose=assistants")
);
assert.equal(response.status, 200);
const body = await response.json();
assert.equal(body.object, "list");
assert.equal(body.data.length, 1);
assert.equal(body.has_more, true);
} finally {
for (const file of created) deleteFile(file.id);
}
});
it("returns 400 over HTTP for an invalid limit instead of listing files", async () => {
const response = await GET(new Request("http://localhost/v1/files?limit=-1"));
assert.equal(response.status, 400);
const body = await response.json();
assert.equal(body.error.type, "invalid_request_error");
});
});

View File

@@ -1,185 +0,0 @@
// #10197 (tiangao88): route-level coverage for the built-in OpenRouter branch
// that /v1/images/edits gained in this PR. Exercises the actual POST(request)
// handler so the credentials / rate-limit / unified-Image-API forwarding branches
// added to route.ts itself are proven, not just the downstream service call.
//
// Before this change: POST /v1/images/edits rejected the built-in `openrouter`
// provider ("Image edit is not supported for built-in provider"), so image
// Combos routing through OpenRouter could generate but never edit. OpenRouter's
// current reference-image contract is POST /api/v1/images with input_references.
import test from "node:test";
import assert from "node:assert/strict";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-openrouter-edits-"));
process.env.DATA_DIR = TEST_DATA_DIR;
process.env.API_KEY_SECRET = process.env.API_KEY_SECRET || "openrouter-edits-test-secret";
const core = await import("../../src/lib/db/core.ts");
const providersDb = await import("../../src/lib/db/providers.ts");
const apiKeysDb = await import("../../src/lib/db/apiKeys.ts");
const imageEditRoute = await import("../../src/app/api/v1/images/edits/route.ts");
const v1ModelsCatalog = await import("../../src/app/api/v1/models/catalog.ts");
interface ErrorResponseBody {
error: { message: string; code?: string };
}
interface ImageResponseBody {
data: Array<{ b64_json?: string; url?: string }>;
}
const originalFetch = globalThis.fetch;
async function resetStorage() {
globalThis.fetch = originalFetch;
apiKeysDb.resetApiKeyState();
core.resetDbInstance();
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
fs.mkdirSync(TEST_DATA_DIR, { recursive: true });
v1ModelsCatalog.__resetCatalogBuilderRunsForTest();
}
function seedOpenRouterConnection(overrides: { rateLimitedUntil?: string | null } = {}) {
return providersDb.createProviderConnection({
provider: "openrouter",
authType: "apikey",
name: "openrouter-test",
apiKey: "sk-or-test-openrouter-edits",
isActive: true,
testStatus: "active",
rateLimitedUntil: overrides.rateLimitedUntil ?? null,
});
}
function dataUrlPng(bytes: number[]): string {
return `data:image/png;base64,${Buffer.from(bytes).toString("base64")}`;
}
const REF_A = dataUrlPng([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a, 1]);
test.beforeEach(async () => {
await resetStorage();
});
test.after(() => {
globalThis.fetch = originalFetch;
apiKeysDb.resetApiKeyState();
core.resetDbInstance();
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
});
test("#10197 v1 image edit POST forwards built-in openrouter edits to the unified Image API", async () => {
await seedOpenRouterConnection();
let hitUrl: string | null = null;
let hitAuth: string | null = null;
let hitBody = "";
globalThis.fetch = async (url, init: RequestInit = {}) => {
hitUrl = String(url);
const headers = init.headers;
hitAuth =
headers instanceof Headers
? String(headers.get("authorization") || "")
: String(
(headers as Record<string, string> | undefined)?.authorization ||
(headers as Record<string, string> | undefined)?.Authorization ||
""
);
// The OpenRouter adapter sends JSON with input_references, not multipart.
const raw = init.body;
if (typeof raw === "string") hitBody = raw;
else if (raw instanceof Uint8Array) hitBody = Buffer.from(raw).toString("utf8");
else if (raw instanceof ArrayBuffer) hitBody = Buffer.from(raw).toString("utf8");
else if (raw && typeof (raw as { arrayBuffer?: unknown }).arrayBuffer === "function") {
hitBody = Buffer.from(await (raw as { arrayBuffer(): Promise<ArrayBuffer> }).arrayBuffer()).toString("utf8");
}
return new Response(
JSON.stringify({ data: [{ b64_json: Buffer.from([0x89, 0x50, 0x4e, 0x47]).toString("base64") }] }),
{ status: 200, headers: { "content-type": "application/json" } }
);
};
const response = await imageEditRoute.POST(
new Request("http://localhost/api/v1/images/edits", {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({
model: "openrouter/google/gemini-3.1-flash-image-preview",
prompt: "add a red hat",
images: [REF_A],
}),
})
);
const body = (await response.json()) as ImageResponseBody;
assert.equal(response.status, 200);
assert.ok(body.data[0].b64_json, "edit must return an image payload");
// OpenRouter's current image-to-image endpoint is the unified Image API.
assert.equal(hitUrl, "https://openrouter.ai/api/v1/images");
// Must carry the OpenRouter connection key as a Bearer token.
assert.equal(hitAuth, "Bearer sk-or-test-openrouter-edits");
assert.ok(hitBody, "JSON body must be captured");
const forwarded = JSON.parse(hitBody) as {
model?: string;
prompt?: string;
input_references?: Array<{ image_url?: { url?: string } }>;
};
assert.equal(forwarded.model, "google/gemini-3.1-flash-image-preview");
assert.equal(forwarded.prompt, "add a red hat");
assert.equal(forwarded.input_references?.length, 1);
assert.match(forwarded.input_references?.[0]?.image_url?.url || "", /^data:image\/png;base64,/);
});
test("#10197 v1 image edit POST surfaces missing openrouter credentials", async () => {
// No openrouter connection seeded at all.
globalThis.fetch = async () => {
throw new Error("Missing-credentials path must not reach upstream");
};
const response = await imageEditRoute.POST(
new Request("http://localhost/api/v1/images/edits", {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({
model: "openrouter/openai/gpt-5-image-mini",
prompt: "edit this",
images: [REF_A],
}),
})
);
const body = (await response.json()) as ErrorResponseBody;
assert.equal(response.status, 401);
assert.match(body.error.message, /No credentials for provider: openrouter/);
// Hard Rule #12 — error responses must never leak a raw stack trace.
assert.ok(!body.error.message.includes("at /"));
});
test("#10197 v1 image edit POST surfaces openrouter rate-limit sentinel", async () => {
await seedOpenRouterConnection({ rateLimitedUntil: new Date(Date.now() + 60_000).toISOString() });
globalThis.fetch = async () => {
throw new Error("Rate-limited path must not reach upstream");
};
const response = await imageEditRoute.POST(
new Request("http://localhost/api/v1/images/edits", {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({
model: "openrouter/openai/gpt-5.4-image-2",
prompt: "edit this",
images: [REF_A],
}),
})
);
const body = (await response.json()) as ErrorResponseBody;
assert.equal(response.status, 429);
assert.match(body.error.message, /All accounts rate limited/);
assert.ok(!body.error.message.includes("at /"));
});

View File

@@ -36,7 +36,6 @@ const ONES: ScoringFactors = {
contextAffinity: 1,
resetWindowAffinity: 1,
connectionDensity: 1,
quality: 1,
};
function candidate(partial: Partial<ProviderCandidate> = {}): ProviderCandidate {

View File

@@ -1,132 +0,0 @@
import test from "node:test";
import assert from "node:assert/strict";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
/**
* Home "Recent Requests" feed passes `excludeTests` to getCallLogs so the panel shows
* ONLY real provider inference — never backend/management log rows. The filter is an
* ALLOWLIST of the public gateway namespaces (`/v1/%` and `/api/v1/%`), applied before
* LIMIT, rather than a blacklist of individual known noise types. This guards against
* the reported regression where model-sync rows (request_type 'model-sync', path
* `/api/providers/*`) leaked into the feed because the old blacklist only dropped
* connection-test rows.
*/
const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-calllogs-allowlist-"));
process.env.DATA_DIR = TEST_DATA_DIR;
process.env.CALL_LOG_RETENTION_DAYS = "3650";
const core = await import("../../src/lib/db/core.ts");
const callLogs = await import("../../src/lib/usage/callLogs.ts");
type SeedRow = {
id: string;
timestamp: string;
path: string;
model: string;
provider: string;
source_format?: string;
request_type?: string | null;
};
function insertCallLog(row: SeedRow) {
const db = core.getDbInstance();
db.prepare(
`
INSERT INTO call_logs (
id, timestamp, method, path, status, model, provider, source_format, request_type, detail_state
)
VALUES (
@id, @timestamp, 'POST', @path, 200, @model, @provider, @source_format, @request_type, 'none'
)
`
).run({
source_format: row.source_format ?? null,
request_type: row.request_type ?? null,
...row,
});
}
test.beforeEach(() => {
core.resetDbInstance();
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
fs.mkdirSync(TEST_DATA_DIR, { recursive: true });
});
test.after(() => {
core.resetDbInstance();
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
});
test("excludeTests keeps only /v1 and /api/v1 inference rows, drops all backend/management rows", async () => {
const base = Date.parse("2026-01-01T00:00:00.000Z");
const iso = (i: number) => new Date(base + i * 1000).toISOString();
// Two REAL provider inference rows — the only rows the feed should keep.
insertCallLog({
id: "real-v1",
timestamp: iso(4),
path: "/v1/chat/completions",
model: "openai/gpt-4.1",
provider: "openai",
});
insertCallLog({
id: "real-api-v1",
timestamp: iso(3),
path: "/api/v1/chat/completions",
model: "anthropic/claude-opus-4-8",
provider: "anthropic",
});
// Backend/management NOISE — must never appear in the feed.
insertCallLog({
id: "noise-model-sync",
timestamp: iso(2),
path: "/api/providers/openai/models",
model: "model-sync",
provider: "openai",
source_format: "-",
request_type: "model-sync",
});
insertCallLog({
id: "noise-connection-test",
timestamp: iso(1),
path: "/api/providers/test",
model: "connection-test",
provider: "openai",
source_format: "test",
});
const rows = await callLogs.getCallLogs({ excludeTests: true, limit: 50 });
const ids = rows.map((row) => row.id).sort();
assert.deepEqual(
ids,
["real-api-v1", "real-v1"],
"only the /v1 and /api/v1 inference rows survive the allowlist"
);
});
test("without excludeTests every row is returned (allowlist is opt-in)", async () => {
const base = Date.parse("2026-02-01T00:00:00.000Z");
insertCallLog({
id: "real",
timestamp: new Date(base).toISOString(),
path: "/v1/chat/completions",
model: "openai/gpt-4.1",
provider: "openai",
});
insertCallLog({
id: "sync",
timestamp: new Date(base + 1000).toISOString(),
path: "/api/providers/openai/models",
model: "model-sync",
provider: "openai",
request_type: "model-sync",
});
const rows = await callLogs.getCallLogs({ limit: 50 });
assert.equal(rows.length, 2, "no allowlist → backend rows are not filtered out");
});

View File

@@ -1,150 +0,0 @@
import test from "node:test";
import assert from "node:assert/strict";
const { claudeToGeminiRequest } =
await import("../../open-sse/translator/request/claude-to-gemini.ts");
test("Claude -> Gemini merges consecutive user text turns into a single user turn", () => {
const result = claudeToGeminiRequest(
"gemini-2.5-flash",
{
messages: [
{ role: "user", content: "hello" },
{ role: "user", content: [{ type: "text", text: "world" }] },
],
},
false
);
assert.equal(result.contents.length, 1);
assert.equal(result.contents[0].role, "user");
assert.deepEqual(result.contents[0].parts, [{ text: "hello" }, { text: "world" }]);
});
test("Claude -> Gemini merges tool_result and subsequent user instruction into single user turn", () => {
const result = claudeToGeminiRequest(
"gemini-2.5-flash",
{
messages: [
{ role: "user", content: "Calculate 2+2" },
{
role: "assistant",
content: [{ type: "text", text: "I will calculate that." }],
},
{
role: "user",
content: [
{
type: "tool_result",
tool_use_id: "tool_call_1",
content: "4",
},
],
},
{
role: "user",
content: "Now add 10 to that result",
},
],
},
false
);
// Contents must alternate properly and not have consecutive same-role turns
for (let i = 1; i < result.contents.length; i++) {
assert.notEqual(
result.contents[i].role,
result.contents[i - 1].role,
`Consecutive same-role detected at index ${i - 1} and ${i}: ${result.contents[i].role}`
);
}
// The last turn should be a merged user turn containing both the tool context and the text
const lastTurn = result.contents[result.contents.length - 1];
assert.equal(lastTurn.role, "user");
assert.equal(lastTurn.parts.length, 2);
assert.ok(
typeof (lastTurn.parts[0] as { text: string }).text === "string" &&
(lastTurn.parts[0] as { text: string }).text.includes("previous_tool_result_context")
);
assert.deepEqual(lastTurn.parts[1], { text: "Now add 10 to that result" });
});
test("Claude -> Gemini preserves alternating conversation turns without spurious merging", () => {
const result = claudeToGeminiRequest(
"gemini-2.5-flash",
{
messages: [
{ role: "user", content: "Hello" },
{ role: "assistant", content: "Hi! How can I help?" },
{ role: "user", content: "What is the capital of France?" },
],
},
false
);
assert.equal(result.contents.length, 3);
assert.equal(result.contents[0].role, "user");
assert.deepEqual(result.contents[0].parts, [{ text: "Hello" }]);
assert.equal(result.contents[1].role, "model");
assert.deepEqual(result.contents[1].parts, [{ text: "Hi! How can I help?" }]);
assert.equal(result.contents[2].role, "user");
assert.deepEqual(result.contents[2].parts, [{ text: "What is the capital of France?" }]);
});
test("Claude -> Gemini merges three or more consecutive user turns into a single user turn", () => {
const result = claudeToGeminiRequest(
"gemini-2.5-flash",
{
messages: [
{ role: "user", content: "part 1" },
{ role: "user", content: "part 2" },
{ role: "user", content: "part 3" },
],
},
false
);
assert.equal(result.contents.length, 1);
assert.equal(result.contents[0].role, "user");
assert.deepEqual(result.contents[0].parts, [
{ text: "part 1" },
{ text: "part 2" },
{ text: "part 3" },
]);
});
test("Claude -> Gemini handles empty messages array without error", () => {
const result = claudeToGeminiRequest(
"gemini-2.5-flash",
{
messages: [],
},
false
);
assert.deepEqual(result.contents, []);
});
test("Claude -> Gemini merges consecutive assistant turns into a single model turn", () => {
const result = claudeToGeminiRequest(
"gemini-2.5-flash",
{
messages: [
{ role: "user", content: "hello" },
{ role: "assistant", content: "response part 1" },
{ role: "assistant", content: [{ type: "text", text: "response part 2" }] },
],
},
false
);
assert.equal(result.contents.length, 2);
assert.equal(result.contents[0].role, "user");
assert.deepEqual(result.contents[0].parts, [{ text: "hello" }]);
assert.equal(result.contents[1].role, "model");
assert.deepEqual(result.contents[1].parts, [
{ text: "response part 1" },
{ text: "response part 2" },
]);
});

View File

@@ -1,180 +0,0 @@
import test from "node:test";
import assert from "node:assert/strict";
// GLM-5.3 support (released 2026-08-14, https://z.ai/blog/glm-5.3).
//
// Upstream ships ONE model id (`glm-5.3`) — effort is a request parameter
// (`reasoning_effort`: low|high|max, default max) on the coding chat/completions
// endpoint, and `thinking.type: "disabled"` is rejected (converted to low by the
// coding endpoint). OmniRoute keeps the GLM-5.2 tier UX: `glm-5.3-high` /
// `glm-5.3-low` pseudo-ids resolved by the GlmExecutor only. Base `glm-5.3` uses
// the upstream default (max). Unlike the 5.2 tiers (Anthropic-transport effort
// beta header), the 5.3 tiers use the documented `reasoning_effort` param on the
// OpenAI coding transport.
//
// Spec caveat: Z.ai has not yet published the default context window — 1M is
// mirrored from GLM-5.2 (same base model) per operator decision; correct when
// the official spec lands.
const { getRegistryEntry } = await import("../../open-sse/config/providerRegistry.ts");
const { GlmExecutor } = await import("../../open-sse/executors/glm.ts");
const { MODEL_SPECS } = await import("../../src/shared/constants/modelSpecs.ts");
const { GLM_PRICING } = await import("../../src/shared/constants/pricing/shared-tiers.ts");
const GLM_5_3_IDS = ["glm-5.3", "glm-5.3-high", "glm-5.3-low"] as const;
// transformForTransport returns an opaque body; surface only the fields asserted below.
type TransformedRequest = {
model?: string;
reasoning_effort?: string;
thinking?: { type?: string } | null;
max_tokens?: number;
effort?: string;
};
function modelIds(provider: string): string[] {
const entry = getRegistryEntry(provider);
assert.ok(entry, `provider "${provider}" should be registered`);
return (entry.models ?? []).map((m) => m.id);
}
for (const provider of ["glm", "glm-cn", "glmt"]) {
test(`${provider} advertises the GLM-5.3 base model and effort tiers (GLM_SHARED_MODELS)`, () => {
const ids = modelIds(provider);
for (const id of GLM_5_3_IDS) {
assert.ok(ids.includes(id), `${provider} should expose ${id}; got ${ids.join(", ")}`);
}
});
test(`${provider} GLM-5.3 entries mirror the GLM-5.2 shape (1M ctx, 128K out)`, () => {
const models = getRegistryEntry(provider)!.models ?? [];
const base = models.find((m) => m.id === "glm-5.3");
assert.ok(base, "glm-5.3 entry missing");
assert.equal(base.contextLength, 1_000_000);
assert.equal(base.maxOutputTokens, 131_072);
assert.equal(base.toolCalling, true);
assert.equal(base.supportsReasoning, true);
});
}
test("zai advertises the GLM-5.3 base model only (DefaultExecutor sends ids verbatim)", () => {
const ids = modelIds("zai");
assert.ok(ids.includes("glm-5.3"), `zai should advertise glm-5.3; got ${ids.join(", ")}`);
for (const alias of ["glm-5.3-high", "glm-5.3-low"]) {
assert.ok(
!ids.includes(alias),
`zai must not list ${alias}: GlmExecutor-only alias, unknown upstream on the Anthropic endpoint`
);
}
});
test("modelSpecs carries 1M/128K specs for all GLM-5.3 ids", () => {
for (const id of GLM_5_3_IDS) {
const spec = MODEL_SPECS[id];
assert.ok(spec, `MODEL_SPECS should include ${id}`);
assert.equal(spec.contextWindow, 1_000_000);
assert.equal(spec.maxOutputTokens, 131_072);
assert.equal(spec.supportsThinking, true);
}
});
test("GLM_PRICING covers the GLM-5.3 ids with GLM-5.2-parity rates", () => {
const reference = GLM_PRICING["glm-5.2"];
assert.ok(reference, "glm-5.2 pricing reference missing");
for (const id of GLM_5_3_IDS) {
const pricing = GLM_PRICING[id];
assert.ok(pricing, `GLM_PRICING should include ${id}`);
assert.deepEqual(pricing, reference);
}
});
test("GlmExecutor resolves glm-5.3-high to reasoning_effort=high on the OpenAI coding transport", () => {
const executor = new GlmExecutor("glm");
const transformed = executor.transformForTransport(
"glm-5.3-high",
{ messages: [{ role: "user", content: "hi" }] },
false,
{ apiKey: "glm-key" },
"openai"
) as TransformedRequest;
assert.equal(transformed.model, "glm-5.3", "upstream must receive the base model id");
assert.equal(transformed.reasoning_effort, "high");
assert.equal(transformed.thinking?.type, "enabled");
});
test("GlmExecutor resolves glm-5.3-low to reasoning_effort=low with thinking enabled", () => {
const executor = new GlmExecutor("glm");
const transformed = executor.transformForTransport(
"glm-5.3-low",
{ messages: [{ role: "user", content: "hi" }] },
false,
{ apiKey: "glm-key" },
"openai"
) as TransformedRequest;
assert.equal(transformed.model, "glm-5.3");
assert.equal(transformed.reasoning_effort, "low");
assert.equal(transformed.thinking?.type, "enabled");
});
test("GlmExecutor leaves base glm-5.3 without an injected reasoning_effort (upstream default = max)", () => {
const executor = new GlmExecutor("glm");
const transformed = executor.transformForTransport(
"glm-5.3",
{ model: "glm-5.3", messages: [{ role: "user", content: "hi" }] },
false,
{ apiKey: "glm-key" },
"openai"
) as TransformedRequest;
assert.equal(transformed.model, "glm-5.3");
assert.equal(transformed.reasoning_effort, undefined);
// Thinking-model max_tokens default applies to 5.3 (GLM_THINKING_MODEL_PATTERN)
assert.equal(transformed.max_tokens, 131_072);
});
test("GLM-5.3 effort tiers execute on the OpenAI coding transport (no Anthropic-only pinning)", async () => {
const executor = new GlmExecutor("glm");
const originalFetch = globalThis.fetch;
const calls: string[] = [];
globalThis.fetch = async (url) => {
calls.push(String(url));
return new Response(
'data: {"id":"chatcmpl-glm53","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":"ok"}}]}\n\ndata: [DONE]\n\n',
{ headers: { "Content-Type": "text/event-stream" } }
);
};
try {
await executor.execute({
model: "glm-5.3-high",
body: { messages: [{ role: "user", content: "hello" }] },
stream: true,
credentials: {
apiKey: "glm-key",
providerSpecificData: { baseUrl: "https://api.z.ai/api/coding/paas/v4" },
},
});
assert.deepEqual(calls, ["https://api.z.ai/api/coding/paas/v4/chat/completions"]);
} finally {
globalThis.fetch = originalFetch;
}
});
test("GLM-5.2 effort tiers still pin the Anthropic transport (effort beta header) — regression guard", () => {
const executor = new GlmExecutor("glm");
const transformed = executor.transformForTransport(
"glm-5.2-max",
{ messages: [{ role: "user", content: "hi" }] },
false,
{ apiKey: "glm-key" },
"anthropic"
) as TransformedRequest;
assert.equal(transformed.model, "glm-5.2");
assert.equal(transformed.effort, "max");
assert.equal(transformed.thinking?.type, "enabled");
});

View File

@@ -1,57 +0,0 @@
import { describe, it, beforeEach, afterEach } from "node:test";
import assert from "node:assert";
import { getRegistryEntry } from "../../open-sse/config/providerRegistry.ts";
// Mock fetch for health checks
const originalFetch = global.fetch;
describe("MLX Provider Registry Entries", () => {
beforeEach(() => {
global.fetch = async () => ({ ok: false, status: 500 });
});
afterEach(() => {
global.fetch = originalFetch;
});
it("should have mlx-gemma registry entry with correct configuration", async () => {
const entry = getRegistryEntry("mlx-gemma");
assert.ok(entry, "mlx-gemma should be registered");
assert.equal(entry?.id, "mlx-gemma");
assert.equal(entry?.alias, "mlx-gemma");
assert.equal(entry?.format, "openai");
assert.equal(entry?.executor, "default"); // Uses default executor for OpenAI-compatible
assert.equal(entry?.baseUrl, "http://localhost:11435/v1");
assert.equal(entry?.modelsUrl, "http://localhost:11435/v1/models");
assert.equal(entry?.passthroughModels, false);
assert.ok(entry?.models?.length === 1);
assert.equal(entry?.models?.[0]?.id, "mlx-community/gemma-4-26B-A4B-it-qat-q4_0-mlx-aligned");
assert.equal(entry?.models?.[0]?.toolCalling, true);
assert.equal(entry?.models?.[0]?.contextLength, 8192);
});
it("should have mlx-qwen registry entry with correct configuration", async () => {
const entry = getRegistryEntry("mlx-qwen");
assert.ok(entry, "mlx-qwen should be registered");
assert.equal(entry?.id, "mlx-qwen");
assert.equal(entry?.alias, "mlx-qwen");
assert.equal(entry?.format, "openai");
assert.equal(entry?.executor, "default");
assert.equal(entry?.baseUrl, "http://localhost:11436/v1");
assert.equal(entry?.modelsUrl, "http://localhost:11436/v1/models");
assert.equal(entry?.passthroughModels, false);
assert.ok(entry?.models?.length === 1);
assert.equal(entry?.models?.[0]?.id, "maglun/Qwen3.8-27B-MLX-Mixed-3.80bpw");
assert.equal(entry?.models?.[0]?.toolCalling, true);
assert.equal(entry?.models?.[0]?.contextLength, 8192);
});
it("should have both MLX providers in registered providers list", async () => {
const { getRegisteredProviders } = await import("../../open-sse/config/providerRegistry.ts");
const providers = getRegisteredProviders();
assert.ok(providers.includes("mlx-gemma"));
assert.ok(providers.includes("mlx-qwen"));
});
});

View File

@@ -912,35 +912,6 @@ test("Model mapping: thinking mode uses thinking variant", async () => {
}
});
// ─── The search hint is opt-in ──────────────────────────────────────────────
// It used to be appended to every system message and leaked into answers as
// meta-commentary, which is noise for coding clients.
test("buildQuery: search hint is off by default and opt-in via env", async () => {
const { buildQuery } = await import("../../open-sse/executors/perplexity-web/protocol.ts");
const parsed = { systemMsg: "You are terse.", history: [], currentMsg: "hi" };
const HINT = "built-in web search";
const prev = process.env.OMNIROUTE_PPLX_SEARCH_HINT;
try {
delete process.env.OMNIROUTE_PPLX_SEARCH_HINT;
const off = JSON.parse(buildQuery(parsed, null));
assert.deepEqual(off.instructions, ["You are terse."]);
assert.equal(off.query, "hi");
process.env.OMNIROUTE_PPLX_SEARCH_HINT = "1";
const on = JSON.parse(buildQuery(parsed, null));
assert.equal(on.instructions.length, 2);
assert.ok(on.instructions[1].includes(HINT));
process.env.OMNIROUTE_PPLX_SEARCH_HINT = "0";
assert.equal(JSON.parse(buildQuery(parsed, null)).instructions.length, 1);
} finally {
if (prev === undefined) delete process.env.OMNIROUTE_PPLX_SEARCH_HINT;
else process.env.OMNIROUTE_PPLX_SEARCH_HINT = prev;
}
});
// ─── Test: Live multi-step stream (no COMPLETED; text_completed + diffs) ────
test("Live multi-step: reconstructs answer without status COMPLETED", async () => {

View File

@@ -1,143 +0,0 @@
/**
* #10214 — Direct (no-proxy) requests stall on a silently-dropped pooled
* keep-alive socket until the caller's deadline or a service restart.
*
* The default direct dispatcher pools keep-alive sockets for up to
* `fetchKeepAliveTimeoutMs` (4 s). A socket that silently drops (half-open, no
* RST) surfaces NO transport error — undici's headersTimeout (600 s default) is
* the only guard, so the existing fresh-socket retry (which fires on
* UND_ERR/ECONNRESET/fetch-failed) never triggers. Observed live: opencode-go
* and command-code stall 100% of routed requests until `systemctl restart`.
*
* The fix bounds the response-start window per direct attempt
* (`OMNIROUTE_DIRECT_HEADERS_TIMEOUT_MS`, default 30 s) and retries once on the
* fresh no-keep-alive dispatcher (a brand-new socket) when the pooled attempt
* times out — converting the zombie-socket stall into a clean failover.
*/
import { test } from "node:test";
import assert from "node:assert/strict";
import { proxyFetch } from "../../open-sse/utils/proxyFetch.ts";
import { getDefaultDispatcher, getRetryDispatcher } from "../../open-sse/utils/proxyDispatcher.ts";
const DIRECT_RESPONSE_START_TIMEOUT_CODE = "DIRECT_RESPONSE_START_TIMEOUT";
/** Simulates a silent half-open pooled socket: the request never resolves, but
* observes the abort signal like real undici does (rejects with the reason). */
function hangingFetch(capture: {
calls: number;
dispatchers: unknown[];
}): (input: RequestInfo | URL, init?: RequestInit) => Promise<Response> {
return (input, init) => {
capture.calls++;
capture.dispatchers.push((init as { dispatcher?: unknown } | undefined)?.dispatcher);
return new Promise<Response>((_, reject) => {
const signal = init?.signal;
signal?.addEventListener(
"abort",
() =>
reject(signal.reason instanceof Error ? signal.reason : new Error(String(signal.reason))),
{ once: true }
);
// never resolve — the upstream accepted the connection but sends nothing
});
};
}
function withFastTimeout<T>(fn: () => Promise<T>): Promise<T> {
process.env.OMNIROUTE_DIRECT_HEADERS_TIMEOUT_MS = "50";
return fn().finally(() => {
delete process.env.OMNIROUTE_DIRECT_HEADERS_TIMEOUT_MS;
});
}
test("#10214 a response-start timeout on the pooled attempt retries on the FRESH no-keep-alive dispatcher", async () => {
const capture = { calls: 0, dispatchers: [] as unknown[] };
const mockUndici = async (input: RequestInfo | URL, init?: RequestInit): Promise<Response> => {
capture.calls++;
capture.dispatchers.push((init as { dispatcher?: unknown } | undefined)?.dispatcher);
if (capture.calls === 1) {
// First attempt hits a silently-dead pooled socket — hang, no error.
return new Promise<Response>((_, reject) => {
init?.signal?.addEventListener("abort", () => reject(init.signal!.reason), { once: true });
});
}
return new Response("ok", { status: 200 });
};
const mockNative = async (): Promise<Response> =>
new Response("native-should-not-fire", { status: 200 });
const res = await withFastTimeout(() =>
proxyFetch(
"https://opencode.ai/zen/go/v1/chat/completions",
{ method: "POST" },
{ undiciFetch: mockUndici, nativeFetch: mockNative }
)
);
assert.equal(capture.calls, 2, "pooled attempt times out and must retry once");
assert.equal(await res.text(), "ok");
// The regression guard: attempt 0 used the pooled keep-alive dispatcher; the
// retry used the fresh no-keep-alive dispatcher — a DIFFERENT instance, so the
// retry opens a brand-new socket that cannot be the zombie.
assert.equal(
capture.dispatchers[0],
getDefaultDispatcher(),
"first attempt must use the pooled default dispatcher"
);
assert.equal(
capture.dispatchers[1],
getRetryDispatcher(),
"timeout retry must use the fresh no-keep-alive dispatcher"
);
assert.notEqual(capture.dispatchers[0], capture.dispatchers[1]);
});
test("#10214 when the fresh-dispatcher retry also stalls, the timeout surfaces (no native fallback)", async () => {
const capture = { calls: 0, dispatchers: [] as unknown[] };
const mockUndici = hangingFetch(capture);
const mockNative = async (): Promise<Response> =>
new Response("native-should-not-fire", { status: 200 });
await assert.rejects(
withFastTimeout(() =>
proxyFetch(
"https://opencode.ai/zen/go/v1/chat/completions",
{ method: "POST" },
{ undiciFetch: mockUndici, nativeFetch: mockNative }
)
),
(err: unknown) => {
assert.equal(
(err as { code?: unknown }).code,
DIRECT_RESPONSE_START_TIMEOUT_CODE,
"final failure must be the classified direct response-start timeout"
);
return true;
}
);
assert.equal(capture.calls, 2, "both attempts must have been made");
assert.equal(capture.dispatchers[0], getDefaultDispatcher());
assert.equal(capture.dispatchers[1], getRetryDispatcher());
});
test("#10214 a healthy fast response is untouched by the bound (single attempt, no retry)", async () => {
const capture = { calls: 0, dispatchers: [] as unknown[] };
const mockUndici = async (input: RequestInfo | URL, init?: RequestInit): Promise<Response> => {
capture.calls++;
capture.dispatchers.push((init as { dispatcher?: unknown } | undefined)?.dispatcher);
return new Response("ok", { status: 200 });
};
const res = await withFastTimeout(() =>
proxyFetch(
"https://opencode.ai/zen/go/v1/chat/completions",
{ method: "POST" },
{ undiciFetch: mockUndici }
)
);
assert.equal(capture.calls, 1, "healthy request must not retry");
assert.equal(capture.dispatchers[0], getDefaultDispatcher());
assert.equal(await res.text(), "ok");
});

View File

@@ -1,199 +0,0 @@
/**
* tests/unit/routing-adaptive-e2e.test.ts
*
* Deterministic end-to-end adaptive routing test (Phases 5 + 13).
*
* Exercises the REAL production routing path: the routing-event quality tracker
* → auto-combo scoring (`scoreAutoTargets` from open-sse/services/combo/autoStrategy.ts),
* which is what an "auto" combo uses to pick its preferred provider/model.
*
* Scenarios verified:
* 1. Healthy provider A outranks/ties backup B.
* 2. Degradation injected into A (sustained 5xx) → A's score falls below B → B preferred.
* 3. Recovery injected into A (sustained successes) → A's score recovers → A preferred again.
* 4. A cold provider C is neutral — it neither dominates nor is unfairly penalized.
* 5. One isolated failure does not overturn a healthy provider.
*
* The whole loop is deterministic: no network, no DB, only the real tracker +
* the real scorer.
*/
import test from "node:test";
import assert from "node:assert/strict";
import {
resetQualityTracker,
recordQualityEvent,
} from "../../open-sse/services/routing/quality.ts";
import { qualityScoreFor } from "../../open-sse/services/routing/index.ts";
import { scoreAutoTargets } from "../../open-sse/services/combo/autoStrategy.ts";
import { DEFAULT_WEIGHTS } from "../../open-sse/services/autoCombo/scoring.ts";
import type { ResolvedComboTarget } from "../../open-sse/services/combo/types.ts";
import type { AutoProviderCandidate } from "../../open-sse/services/combo/types.ts";
function target(provider: string, model: string, weight = 1): ResolvedComboTarget {
const modelStr = `${provider}/${model}`;
return {
kind: "model",
stepId: modelStr,
executionKey: modelStr,
modelStr,
provider,
providerId: provider,
connectionId: null,
weight,
label: null,
};
}
function candidate(
target: ResolvedComboTarget,
quality: number,
extra: Partial<AutoProviderCandidate> = {}
): AutoProviderCandidate {
return {
stepId: target.stepId,
executionKey: target.executionKey,
modelStr: target.modelStr,
provider: target.provider,
model: target.modelStr.split("/")[1],
quotaRemaining: 100,
quotaTotal: 100,
circuitBreakerState: "CLOSED",
costPer1MTokens: 1,
p95LatencyMs: 100,
latencyStdDev: 10,
errorRate: 0,
accountTier: "standard",
quotaResetIntervalSecs: 86400,
contextAffinity: 0.5,
sessionAvailability: 1,
resetWindowAffinity: 0.5,
connectionPoolSize: 1,
connectionId: null,
quality,
...extra,
};
}
function record(
provider: string,
model: string,
partial: Partial<Parameters<typeof recordQualityEvent>[0]> = {}
): void {
recordQualityEvent({
provider,
model,
outcome: "success",
status: 200,
latencyMs: 100,
finishReason: "stop",
outputTokens: 5,
...partial,
});
}
function bestProvider(targets: ReturnType<typeof scoreAutoTargets>): string {
return targets[0].target.provider;
}
/** Derive the bare model id from a "provider/model" modelStr. */
function modelOf(t: ResolvedComboTarget): string {
return t.modelStr.slice(t.provider.length + 1);
}
/** Score a set of providers using their live quality-tracker signal. */
function scoreProviders(
ts: ResolvedComboTarget[],
weights = DEFAULT_WEIGHTS
): ReturnType<typeof scoreAutoTargets> {
return scoreAutoTargets(
ts,
ts.map((t) => candidate(t, qualityScoreFor(t.provider, modelOf(t)))),
"general",
weights
);
}
test("healthy provider A is preferred over backup B, cold C stays neutral", () => {
resetQualityTracker();
// Warm A to high confidence with solid success.
for (let i = 0; i < 100; i++) record("a", "m");
// B warm but mildly degraded.
for (let i = 0; i < 100; i++)
record("b", "m", {
outcome: i % 5 === 0 ? "error" : "success",
status: i % 5 === 0 ? 500 : 200,
});
const a = target("a", "m");
const b = target("b", "m");
const c = target("c", "m");
const scored = scoreProviders([a, b, c]);
assert.equal(bestProvider(scored), "a", "healthy A must be the top pick");
// Cold C must not be top (neutral 0.5 quality vs A's high quality).
assert.notEqual(bestProvider(scored), "c", "cold provider must not dominate");
// C's quality must be exactly neutral.
assert.equal(qualityScoreFor("c", "m"), 0.5);
});
test("degradation injected into A flips preference to B; recovery flips it back", () => {
resetQualityTracker();
// Phase 0: A and B are otherwise identical; both healthy. A is preferred via
// stable tie-break, and quality is the only differentiator.
for (let i = 0; i < 100; i++) record("a", "m");
for (let i = 0; i < 100; i++) record("b", "m");
const a = target("a", "m");
const b = target("b", "m");
const score = () => scoreProviders([a, b]);
const initial = score();
assert.equal(bestProvider(initial), "a", "initially A is preferred (tie-break on equal quality)");
// Phase 1: degrade A — sustained 5xx. A's quality collapses to ~0, so B
// (identical but healthy) becomes preferred. Gradual: the EWMA smooths the drop.
for (let i = 0; i < 60; i++) record("a", "m", { outcome: "error", status: 500 });
const during = score();
assert.equal(
bestProvider(during),
"b",
"sustained degradation must flip preference to B (gradual, not instant)"
);
const qualityA = qualityScoreFor("a", "m");
assert.ok(qualityA < 0.5, `A quality degraded below neutral, got ${qualityA}`);
// Phase 2: recover A — sustained successes. Quality recovers and A's
// preference is restored.
for (let i = 0; i < 200; i++) record("a", "m");
const after = score();
assert.equal(bestProvider(after), "a", "recovery must restore A's preference");
// Phase 3: one isolated failure must not destroy A — its quality stays healthy
// (well above neutral), it just falls marginally behind the now-equally-tied B.
record("a", "m", { outcome: "error", status: 500 });
const qualityAfterBlip = qualityScoreFor("a", "m");
assert.ok(
qualityAfterBlip > 0.7,
`one isolated failure must not destroy A's quality, got ${qualityAfterBlip}`
);
const afterBlip = score();
const aEntry = afterBlip.find((s) => s.target.provider === "a");
const bEntry = afterBlip.find((s) => s.target.provider === "b");
assert.ok(
Math.abs(aEntry!.score - bEntry!.score) < 0.02,
"A must remain competitive after one isolated failure (not destroyed)"
);
});
test("a provider with insufficient evidence does not dominate from optimistic init", () => {
resetQualityTracker();
// Solid warm provider.
for (let i = 0; i < 200; i++) record("solid", "m");
// Lucky cold provider: 7 flawless successes.
for (let i = 0; i < 7; i++) record("lucky", "m");
const s = target("solid", "m");
const l = target("lucky", "m");
const scored = scoreProviders([s, l]);
assert.equal(bestProvider(scored), "solid", "solid warm provider must beat a lucky cold one");
});

View File

@@ -1,172 +0,0 @@
/**
* tests/unit/routing-events-concurrency.test.ts
*
* Stress the routing-event system (Phase 4):
* - thousands of events through the real dispatch + quality + memory sinks
* - ring-buffer boundedness under sustained load (newest retained)
* - a throwing sink under load is isolated (other sinks keep working)
* - quality tracker updates stay consistent under interleaved async bursts
* - simultaneous reset() during inserts does not throw or corrupt state
*
* Node's event loop is single-threaded, so "concurrency" here is interleaved
* async execution; these tests assert correctness under bursts, not true
* parallelism.
*/
import test from "node:test";
import assert from "node:assert/strict";
import {
dispatchRoutingEvent,
MemoryRoutingEventStore,
registerRoutingEventSink,
clearRoutingEventSinks,
createRoutingEvent,
type RoutingEvent,
type RoutingEventSink,
} from "../../open-sse/services/routing/events.ts";
import {
recordQualityEvent,
getQualityScore,
resetQualityTracker,
getProviderQuality,
} from "../../open-sse/services/routing/quality.ts";
function makeEvent(i: number): RoutingEvent {
return createRoutingEvent({
requestId: `r-${i}`,
provider: i % 2 === 0 ? "openai" : "anthropic",
model: "m",
strategy: "auto",
latencyMs: 50 + (i % 100),
outcome: i % 100 === 0 ? "malformed" : "success",
status: 200,
finishReason: "stop",
outputTokens: 5,
});
}
test("sustained burst of thousands of events is bounded and consistent", async () => {
clearRoutingEventSinks();
resetQualityTracker();
const store = new MemoryRoutingEventStore(100);
registerRoutingEventSink(store);
registerRoutingEventSink({
name: "quality",
record: (e) => recordQualityEvent(e),
});
const N = 10_000;
for (let i = 0; i < N; i++) dispatchRoutingEvent(makeEvent(i));
assert.equal(store.size, 100, "ring buffer must stay bounded at capacity");
const recent = store.recent(5);
assert.equal(recent[0].requestId, `r-${N - 1}`, "newest event must be retained");
const q = getProviderQuality("openai", "m");
assert.equal(q.samples, N / 2, "even-indexed events all landed in the quality tracker");
assert.ok(q.operational > 0.5, "mostly-successful provider should be above neutral");
clearRoutingEventSinks();
});
test("a throwing sink under load does not break other sinks", () => {
clearRoutingEventSinks();
resetQualityTracker();
const seen: string[] = [];
registerRoutingEventSink({
name: "thrower",
record: () => {
throw new Error("sink boom");
},
});
registerRoutingEventSink({
name: "collector",
record: (e) => void seen.push(e.requestId),
});
for (let i = 0; i < 2000; i++) dispatchRoutingEvent(makeEvent(i));
assert.equal(seen.length, 2000, "all events must still reach the good sink");
clearRoutingEventSinks();
});
test("interleaved async bursts keep quality math consistent", async () => {
resetQualityTracker();
const bursts = Array.from({ length: 8 }, (_, b) =>
(async () => {
for (let i = 0; i < 500; i++) {
recordQualityEvent(makeEvent(b * 500 + i));
// Yield occasionally to interleave with the other bursts.
if (i % 50 === 0) await new Promise((r) => setImmediate(r));
}
})()
);
await Promise.all(bursts);
const q = getProviderQuality("openai", "m");
assert.equal(q.samples, 2000, "4 bursts * 500 with i%2==0 → 2000 openai samples");
assert.ok(Number.isFinite(q.operational) && q.operational >= 0 && q.operational <= 1);
// Recency should be non-null and tiny (events were just recorded).
assert.ok(q.recencyMs !== null && q.recencyMs < 5000);
});
test("reset during inserts is safe and state re-initializes cleanly", async () => {
resetQualityTracker();
const store = new MemoryRoutingEventStore(50);
registerRoutingEventSink(store);
registerRoutingEventSink({ name: "quality", record: (e) => recordQualityEvent(e) });
const writer = (async () => {
for (let i = 0; i < 2000; i++) {
dispatchRoutingEvent(makeEvent(i));
if (i % 200 === 0) await new Promise((r) => setImmediate(r));
}
})();
// Fire several resets while the writer is mid-flight.
const resets = Array.from({ length: 3 }, (_, k) =>
(async () => {
await new Promise((r) => setImmediate(r));
resetQualityTracker();
store.clear();
})()
);
await Promise.all(resets);
await writer;
// After a reset the tracker is empty for the reset epoch; post-reset events
// must still record without throwing. We assert non-negative, finite state.
const q = getProviderQuality("openai", "m");
assert.ok(q.samples >= 0);
assert.ok(Number.isFinite(q.operational));
clearRoutingEventSinks();
resetQualityTracker();
});
test("quality score never goes NaN or out of [0,1] under adversarial events", () => {
resetQualityTracker();
const bad: RoutingEvent[] = [
makeEvent(0),
createRoutingEvent({
requestId: "nan-1",
provider: "nanp",
model: "nanm",
latencyMs: NaN,
outcome: "error",
status: NaN,
ttftMs: NaN,
outputTokens: NaN,
}),
];
for (const e of bad) dispatchRoutingEvent(makeEvent(1));
recordQualityEvent({
provider: "nanp",
model: "nanm",
outcome: "error",
status: NaN,
latencyMs: NaN,
ttftMs: NaN,
outputTokens: NaN,
});
const score = getQualityScore("nanp", "nanm");
assert.ok(Number.isFinite(score), `score must be finite, got ${score}`);
assert.ok(score >= 0 && score <= 1, `score in [0,1], got ${score}`);
});

View File

@@ -1,141 +0,0 @@
/**
* tests/unit/routing-events.test.ts
*
* Routing feedback foundation (open-sse/services/routing/events.ts):
* - createRoutingEvent normalizes defaults
* - outcomeFromStatus classifies HTTP statuses
* - MemoryRoutingEventStore is bounded and returns newest-first
* - dispatchRoutingEvent fans out to sinks and isolates a throwing sink
*/
import test from "node:test";
import assert from "node:assert/strict";
import {
MemoryRoutingEventStore,
createRoutingEvent,
outcomeFromStatus,
dispatchRoutingEvent,
registerRoutingEventSink,
listRoutingEventSinks,
clearRoutingEventSinks,
type RoutingEvent,
type RoutingEventSink,
} from "../../open-sse/services/routing/events.ts";
function event(partial: Partial<RoutingEvent> = {}): RoutingEvent {
return createRoutingEvent({
requestId: "req-1",
provider: "openai",
model: "gpt-4o",
strategy: "auto",
latencyMs: 120,
outcome: "success",
status: 200,
...partial,
});
}
test("createRoutingEvent fills observability defaults", () => {
const e = createRoutingEvent({
requestId: "req-x",
provider: "anthropic",
model: "claude-4",
latencyMs: 50,
outcome: "error",
status: 500,
});
assert.equal(e.strategy, "direct");
assert.equal(e.ttftMs, null);
assert.equal(e.inputTokens, null);
assert.equal(e.outputTokens, null);
assert.equal(e.cost, null);
assert.equal(e.retries, 0);
assert.equal(e.fallbackUsed, false);
assert.equal(e.finishReason, null);
assert.equal(e.connectionId, null);
assert.ok(e.ts > 0);
assert.equal(e.status, 500);
});
test("outcomeFromStatus classifies statuses", () => {
assert.equal(outcomeFromStatus(200), "success");
assert.equal(outcomeFromStatus(201), "success");
assert.equal(outcomeFromStatus(429), "rate_limited");
assert.equal(outcomeFromStatus(408), "timeout");
assert.equal(outcomeFromStatus(504), "timeout");
assert.equal(outcomeFromStatus(500), "error");
assert.equal(outcomeFromStatus(400), "error");
assert.equal(outcomeFromStatus(null), "error");
assert.equal(outcomeFromStatus(undefined), "error");
});
test("MemoryRoutingEventStore returns newest-first within capacity", () => {
const store = new MemoryRoutingEventStore(5);
for (let i = 0; i < 5; i++) store.record(event({ requestId: `r-${i}` }));
const recent = store.recent(5);
assert.equal(recent.length, 5);
assert.equal(recent[0].requestId, "r-4");
assert.equal(recent[4].requestId, "r-0");
});
test("MemoryRoutingEventStore is bounded and still newest-first after overflow", () => {
const store = new MemoryRoutingEventStore(3);
for (let i = 0; i < 10; i++) store.record(event({ requestId: `r-${i}` }));
assert.equal(store.size, 3);
const recent = store.recent(3);
assert.deepEqual(
recent.map((e) => e.requestId),
["r-9", "r-8", "r-7"]
);
store.clear();
assert.equal(store.size, 0);
assert.deepEqual(store.recent(), []);
});
test("dispatchRoutingEvent fans out to every registered sink", () => {
const seen: string[] = [];
const sink: RoutingEventSink = {
name: "test-a",
record: (e) => void seen.push(e.requestId),
};
const unsub = registerRoutingEventSink(sink);
try {
dispatchRoutingEvent(event({ requestId: "fan-1" }));
dispatchRoutingEvent(event({ requestId: "fan-2" }));
assert.deepEqual(seen, ["fan-1", "fan-2"]);
} finally {
unsub();
}
});
test("dispatchRoutingEvent isolates a throwing sink", () => {
const badSink: RoutingEventSink = {
name: "test-throw",
record: () => {
throw new Error("boom");
},
};
const goodSeen: string[] = [];
const goodSink: RoutingEventSink = {
name: "test-good",
record: (e) => void goodSeen.push(e.requestId),
};
registerRoutingEventSink(badSink);
registerRoutingEventSink(goodSink);
try {
dispatchRoutingEvent(event({ requestId: "isolated" }));
assert.deepEqual(goodSeen, ["isolated"]);
} finally {
clearRoutingEventSinks();
}
});
test("listRoutingEventSinks reports registered names", () => {
clearRoutingEventSinks();
assert.deepEqual(listRoutingEventSinks(), []);
const unsub = registerRoutingEventSink({ name: "probe", record: () => {} });
try {
assert.deepEqual(listRoutingEventSinks(), ["probe"]);
} finally {
unsub();
}
});

View File

@@ -1,129 +0,0 @@
/**
* tests/unit/routing-otel.test.ts
*
* Optional OpenTelemetry sink (open-sse/services/routing/otel.ts):
* - disabled unless an endpoint is configured
* - buildOtlpTracesPayload emits GenAI semantic-convention spans
* - record() enqueues without performing I/O; stop() flushes via fetch
* - dropped events are counted when the buffer overflows
*/
import test from "node:test";
import assert from "node:assert/strict";
import {
buildOtlpTracesPayload,
isRoutingOtelEnabled,
OtlpHttpsEventSink,
} from "../../open-sse/services/routing/otel.ts";
import type { RoutingEvent } from "../../open-sse/services/routing/events.ts";
function event(partial: Partial<RoutingEvent> = {}): RoutingEvent {
return {
requestId: "req-1",
provider: "openai",
model: "gpt-4o",
strategy: "auto",
latencyMs: 120,
ttftMs: 40,
inputTokens: 10,
outputTokens: 20,
cost: 0.01,
retries: 1,
fallbackUsed: true,
outcome: "success",
status: 200,
finishReason: "stop",
connectionId: "conn-1",
ts: 1_700_000_000_000,
...partial,
};
}
test("isRoutingOtelEnabled is false without an endpoint", () => {
assert.equal(isRoutingOtelEnabled({}), false);
assert.equal(isRoutingOtelEnabled({ OMNIROUTE_OTEL_ENDPOINT: " " }), false);
});
test("isRoutingOtelEnabled honors OMNIROUTE_OTEL_ENDPOINT and OTLP env", () => {
assert.equal(isRoutingOtelEnabled({ OMNIROUTE_OTEL_ENDPOINT: "http://collector:4318" }), true);
assert.equal(
isRoutingOtelEnabled({ OTEL_EXPORTER_OTLP_ENDPOINT: "https://collector:4318" }),
true
);
});
test("buildOtlpTracesPayload emits GenAI semantic-convention spans", () => {
const payload = buildOtlpTracesPayload([event()], "omniroute-test") as {
resourceSpans: Array<{
scopeSpans: Array<{
spans: Array<{
attributes: Array<{ key: string; value: { stringValue?: string; intValue?: string } }>;
}>;
}>;
}>;
};
const span = payload.resourceSpans[0].scopeSpans[0].spans[0];
const attrs = Object.fromEntries(
span.attributes.map((a) => [a.key, a.value.stringValue ?? a.value.intValue])
);
assert.equal(attrs["gen_ai.provider.name"], "openai");
assert.equal(attrs["gen_ai.request.model"], "gpt-4o");
assert.equal(attrs["gen_ai.system"], "auto");
assert.equal(attrs["gen_ai.usage.input_tokens"], "10");
assert.equal(attrs["gen_ai.usage.output_tokens"], "20");
assert.equal(attrs["gen_ai.completion.finish_reason"], "stop");
assert.equal(attrs["omniroute.routing.outcome"], "success");
assert.equal(attrs["omniroute.routing.status"], "200");
assert.equal(attrs["omniroute.routing.retries"], "1");
assert.equal(attrs["omniroute.routing.fallback_used"], "1");
assert.equal(attrs["omniroute.connection_id"], "conn-1");
assert.ok(BigInt(span.startTimeUnixNano) > 0n);
});
test("OtlpHttpsEventSink record() enqueues without I/O and flush sends via fetch", async () => {
const calls: Array<{ url: string; body: string }> = [];
const originalFetch = global.fetch;
// eslint-disable-next-line @typescript-eslint/no-explicit-any
global.fetch = (async (url: any, init: any) => {
calls.push({ url: String(url), body: String(init?.body ?? "") });
return { ok: true } as Response;
}) as typeof fetch;
const sink = new OtlpHttpsEventSink({
endpoint: "http://collector:4318",
flushIntervalMs: 1_000_000, // effectively never auto-flush in the test
});
try {
sink.record(event());
sink.record(event({ requestId: "req-2" }));
assert.equal(sink.getStats().buffered, 2);
// Force an explicit flush via stop().
await new Promise((r) => setTimeout(r, 20));
sink.stop();
await new Promise((r) => setTimeout(r, 50));
assert.equal(calls.length, 1, "one flush should have been sent");
assert.ok(calls[0].url.endsWith("/v1/traces"), calls[0].url);
const body = JSON.parse(calls[0].body);
assert.ok(body.resourceSpans[0].scopeSpans[0].spans.length === 2);
assert.equal(sink.getStats().buffered, 0);
} finally {
global.fetch = originalFetch;
}
});
test("OtlpHttpsEventSink drops oldest when the buffer is saturated", async () => {
const originalFetch = global.fetch;
global.fetch = (async () => ({ ok: true }) as Response) as typeof fetch;
const sink = new OtlpHttpsEventSink({
endpoint: "http://collector:4318",
maxBatchSize: 2,
flushIntervalMs: 1_000_000,
});
try {
for (let i = 0; i < 20; i++) sink.record(event({ requestId: `r-${i}` }));
const stats = sink.getStats();
assert.ok(stats.dropped > 0, "overload must drop events, never block");
sink.stop();
} finally {
global.fetch = originalFetch;
}
});

View File

@@ -1,187 +0,0 @@
/**
* tests/unit/routing-quality.test.ts
*
* Feedback-driven quality signal v2 (open-sse/services/routing/quality.ts):
* - operational vs semantic separation (semantic is NEVER manufactured from HTTP)
* - neutral 0.5 for cold providers (not unfairly penalized, cannot dominate)
* - confidence/sample-awareness (lucky cold provider cannot outrank a solid warm one)
* - success raises / failure lowers the EWMA score
* - malformed / stream-interrupted / empty-output anomalies penalize
* - 429 is transient (far lighter than a 500)
* - confidence ramps with sample count
* - reset clears state
*/
import test from "node:test";
import assert from "node:assert/strict";
import {
recordQualityEvent,
getQualityScore,
getProviderQuality,
setSemanticQuality,
getQualitySnapshot,
resetQualityTracker,
QUALITY_WELL_KNOWN,
} from "../../open-sse/services/routing/quality.ts";
const { CONFIDENCE_FULL_SAMPLES } = QUALITY_WELL_KNOWN;
function record(
provider: string,
model: string,
partial: Partial<Parameters<typeof recordQualityEvent>[0]> = {}
): void {
recordQualityEvent({
provider,
model,
outcome: "success",
status: 200,
latencyMs: 100,
finishReason: "stop",
...partial,
});
}
test("cold provider scores neutral 0.5 (no penalty, no dominance)", () => {
resetQualityTracker();
assert.equal(getQualityScore("openai", "gpt-4o"), 0.5);
const q = getProviderQuality("openai", "gpt-4o");
assert.equal(q.operational, 0.5);
assert.equal(q.confidence, 0);
assert.equal(q.samples, 0);
});
test("below warmup threshold the score is pulled toward neutral (not 1.0)", () => {
resetQualityTracker();
// 7 lucky successes: operational EWMA → 1.0, but confidence is low, so the
// blended score must stay well below 1.0 — it must not dominate a solid warm provider.
for (let i = 0; i < 7; i++) record("openai", "gpt-4o");
const lucky = getQualityScore("openai", "gpt-4o");
assert.ok(lucky > 0.5 && lucky < 0.8, `lucky cold provider should be near-neutral, got ${lucky}`);
});
test("a provider with thousands of solid observations outranks a lucky cold provider", () => {
resetQualityTracker();
// Solid warm provider: 4000 samples, ~91% success.
for (let i = 0; i < 4000; i++) {
record("p", "solid", {
outcome: i % 11 === 0 ? "error" : "success",
status: i % 11 === 0 ? 500 : 200,
});
}
// Lucky cold provider: 7 samples, all success.
for (let i = 0; i < 7; i++) record("p", "lucky");
const solid = getQualityScore("p", "solid");
const lucky = getQualityScore("p", "lucky");
assert.ok(solid > lucky, `solid (${solid}) must outrank lucky (${lucky})`);
assert.ok(solid > 0.8, `solid provider should score high, got ${solid}`);
});
test("sustained failures degrade; sustained successes recover gradually", () => {
resetQualityTracker();
for (let i = 0; i < 20; i++) record("openai", "gpt-4o", { outcome: "error", status: 500 });
const degraded = getQualityScore("openai", "gpt-4o");
assert.ok(degraded < 0.4, `expected degraded score, got ${degraded}`);
for (let i = 0; i < 40; i++) record("openai", "gpt-4o");
const recovered = getQualityScore("openai", "gpt-4o");
assert.ok(recovered > degraded, "successes must recover the score");
assert.ok(recovered > 0.7, `expected recovery toward healthy, got ${recovered}`);
});
test("one isolated failure does not destroy a warm provider", () => {
resetQualityTracker();
for (let i = 0; i < 100; i++) record("p", "m");
const before = getQualityScore("p", "m");
record("p", "m", { outcome: "error", status: 500 });
const after = getQualityScore("p", "m");
assert.ok(after > 0.7, `single failure must not destroy a healthy provider, got ${after}`);
assert.ok(after < before, "the single failure should still register");
});
test("malformed and stream-interrupted outcomes penalize more than a clean error", () => {
resetQualityTracker();
record("p", "m-a", { outcome: "malformed", status: 200, finishReason: "stop" });
for (let i = 0; i < 20; i++) record("p", "m-a");
record("p", "m-b");
for (let i = 0; i < 20; i++) record("p", "m-b");
assert.ok(
getQualityScore("p", "m-a") < getQualityScore("p", "m-b"),
"anomaly history must lower quality below a clean record"
);
});
test("finish_reason=length (truncated output) counts as an anomaly", () => {
resetQualityTracker();
for (let i = 0; i < 20; i++)
record("p", "truncated", { outcome: "success", finishReason: "length" });
for (let i = 0; i < 20; i++) record("p", "clean");
assert.ok(
getQualityScore("p", "truncated") < getQualityScore("p", "clean"),
"length finish_reason must hurt quality"
);
});
test("zero-output successes count as anomalies; missing output does not", () => {
resetQualityTracker();
for (let i = 0; i < 20; i++)
record("p", "empty", { outcome: "success", outputTokens: 0, finishReason: "stop" });
for (let i = 0; i < 20; i++) record("p", "ok", { outcome: "success", outputTokens: 5 });
assert.ok(
getQualityScore("p", "empty") < getQualityScore("p", "ok"),
"zero-output 200 must hurt quality more than a normal 200"
);
});
test("429 is transient (near-neutral), not a quality failure", () => {
resetQualityTracker();
for (let i = 0; i < 50; i++) record("p", "rl", { outcome: "rate_limited", status: 429 });
for (let i = 0; i < 50; i++) record("p", "err", { outcome: "error", status: 500 });
const rateLimited = getQualityScore("p", "rl");
const error = getQualityScore("p", "err");
assert.ok(rateLimited > error, "rate-limited should score better than hard failures");
assert.ok(rateLimited >= 0.45, "rate-limit alone should not tank quality below neutral");
});
test("semantic quality is separate from operational and never manufactured", () => {
resetQualityTracker();
// A provider with perfect operational history but no evaluator → semantic null.
for (let i = 0; i < 100; i++) record("p", "op-only");
const q = getProviderQuality("p", "op-only");
assert.equal(q.semantic, null, "semantic must be null until an evaluator provides it");
assert.ok(q.operational > 0.9, "operational can be high independently");
// An evaluator can then attach a semantic score.
setSemanticQuality("p", "op-only", 0.42, 0.8);
const q2 = getProviderQuality("p", "op-only");
assert.equal(q2.semantic, 0.42);
assert.equal(q2.semanticConfidence, 0.8);
// The operational score must NOT be contaminated by the semantic score.
assert.ok(
Math.abs(q2.operational - q.operational) < 1e-9,
"semantic must not leak into operational"
);
});
test("snapshot reports confidence, samples and anomaly counts", () => {
resetQualityTracker();
for (let i = 0; i < 10; i++) record("snap", "model");
record("snap", "model", { outcome: "malformed" });
const snap = getQualitySnapshot();
const view = snap.find((v) => v.provider === "snap" && v.model === "model");
assert.ok(view, "snapshot must contain the tracked model");
assert.equal(view!.confidence, 11 / CONFIDENCE_FULL_SAMPLES);
assert.ok(view!.samples === 11);
assert.ok(view!.anomalies >= 1);
assert.ok(view!.operational >= 0 && view!.operational <= 1);
});
test("reset clears all tracked state", () => {
resetQualityTracker();
record("p", "m");
assert.equal(getQualitySnapshot().length, 1);
resetQualityTracker();
assert.equal(getQualitySnapshot().length, 0);
assert.equal(getQualityScore("p", "m"), 0.5);
});

View File

@@ -1,96 +0,0 @@
/**
* tests/unit/routing-scoring-quality.test.ts
*
* Scoring integration of the feedback quality signal:
* - DEFAULT_WEIGHTS still sums to ~1.0 (validateWeights) with the new quality weight
* - calculateFactors defaults missing quality to neutral 1.0
* - calculateScore applies the quality factor
* - a low-quality candidate ranks below an identical high-quality one
*/
import test from "node:test";
import assert from "node:assert/strict";
import {
calculateFactors,
calculateScore,
DEFAULT_WEIGHTS,
normalizeScoringWeights,
validateWeights,
type ProviderCandidate,
type ScoringFactors,
} from "../../open-sse/services/autoCombo/scoring.ts";
function candidate(partial: Partial<ProviderCandidate> = {}): ProviderCandidate {
return {
provider: "p",
model: "m",
quotaRemaining: 100,
quotaTotal: 100,
circuitBreakerState: "CLOSED",
costPer1MTokens: 1,
p95LatencyMs: 100,
latencyStdDev: 10,
errorRate: 0,
accountTier: "standard",
quotaResetIntervalSecs: 86400,
...partial,
};
}
test("DEFAULT_WEIGHTS sums to ~1 with the new quality weight", () => {
const sum = Object.values(DEFAULT_WEIGHTS).reduce((a, b) => a + Number(b), 0);
assert.ok(Math.abs(sum - 1) < 1e-9, `expected sum ≈ 1, got ${sum}`);
assert.ok(validateWeights(DEFAULT_WEIGHTS), "validateWeights must accept DEFAULT_WEIGHTS");
assert.ok((DEFAULT_WEIGHTS.quality ?? 0) > 0, "quality weight must be > 0");
});
test("calculateFactors defaults missing quality to neutral 0.5", () => {
const factors = calculateFactors(candidate(), [candidate()], "general", () => 0.5);
assert.equal(factors.quality, 0.5);
});
test("calculateFactors clamps quality to [0,1]", () => {
const low = calculateFactors(candidate({ quality: -2 }), [candidate()], "general", () => 0.5);
assert.equal(low.quality, 0);
const high = calculateFactors(candidate({ quality: 5 }), [candidate()], "general", () => 0.5);
assert.equal(high.quality, 1);
});
test("calculateScore applies the quality factor", () => {
const base: ScoringFactors = {
quota: 0.5,
health: 0.5,
costInv: 0.5,
latencyInv: 0.5,
taskFit: 0.5,
stability: 0.5,
tierPriority: 0.5,
tierAffinity: 0.5,
specificityMatch: 0.5,
contextAffinity: 0.5,
resetWindowAffinity: 0.5,
connectionDensity: 0.5,
};
const good = calculateScore({ ...base, quality: 1 }, DEFAULT_WEIGHTS);
const bad = calculateScore({ ...base, quality: 0 }, DEFAULT_WEIGHTS);
assert.ok(good > bad, "higher quality must score strictly higher");
assert.ok(good >= 0 && good <= 1);
assert.ok(bad >= 0 && bad <= 1);
});
test("low-quality candidate ranks below identical high-quality candidate", () => {
const good = candidate({ provider: "p", model: "good", quality: 1 });
const poor = candidate({ provider: "p", model: "poor", quality: 0.3 });
const pool = [good, poor];
const fg = calculateFactors(good, pool, "general", () => 0.5);
const fp = calculateFactors(poor, pool, "general", () => 0.5);
const sg = calculateScore(fg, DEFAULT_WEIGHTS);
const sp = calculateScore(fp, DEFAULT_WEIGHTS);
assert.ok(sg > sp, `good candidate (${sg}) must outrank poor (${sp})`);
});
test("normalizeScoringWeights keeps quality and renormalizes to 1", () => {
const normalized = normalizeScoringWeights({ quality: 0.1 });
const total = Object.values(normalized).reduce((s, v) => s + Number(v), 0);
assert.ok(Math.abs(total - 1) < 1e-9);
assert.ok((normalized.quality ?? 0) > 0);
});

View File

@@ -1,86 +0,0 @@
/**
* tests/unit/stream-timing.test.ts
*
* Canonical stream instrumentation (open-sse/utils/streamTiming.ts):
* - TTFT = first-forwarded-SSE-chunk latency (NOT token-level) — documented
* - ITL = mean inter-chunk gap (chunk-latency proxy)
* - first-byte vs first-forward distinction
* - interruption marking
* - malformed/empty chunks do not corrupt timing
*/
import test from "node:test";
import assert from "node:assert/strict";
import { createStreamTiming, type StreamTiming } from "../../open-sse/utils/streamTiming.ts";
test("ttft() is null when nothing was forwarded", () => {
const t = createStreamTiming();
t.markByte();
assert.equal(t.ttftMs(), null);
assert.equal(t.avgItlMs(), null);
});
test("ttft() measures first-forwarded-chunk latency (byte vs forward distinguished)", async () => {
const t = createStreamTiming();
t.markByte(); // first upstream byte arrives immediately
await new Promise((r) => setTimeout(r, 20));
t.markForward(); // first chunk forwarded 20ms later
const ttft = t.ttftMs();
assert.ok(ttft !== null && ttft >= 20 && ttft < 5000, `ttft=${ttft}`);
assert.ok(t.firstByteAt !== null);
assert.ok(t.firstByteAt! < t.firstForwardAt!, "first byte precedes first forward");
});
test("avgItlMs() measures mean inter-chunk gap across multiple chunks", async () => {
const t = createStreamTiming();
for (let i = 0; i < 4; i++) {
t.markForward();
await new Promise((r) => setTimeout(r, 10));
}
const itl = t.avgItlMs();
assert.ok(itl !== null && itl >= 8 && itl < 5000, `itl=${itl}`);
assert.equal(t.forwardedChunks, 4);
});
test("empty chunks do not corrupt timing (markByte without forward)", () => {
const t = createStreamTiming();
t.markByte();
t.markByte(); // duplicate bytes are idempotent for first-byte
assert.equal(t.ttftMs(), null, "no forward → no ttft");
t.markForward();
assert.ok(t.ttftMs() !== null);
});
test("malformed/keepalive-only traffic (no forward) yields no ttft", () => {
const t = createStreamTiming();
// Simulate a provider that only sends keepalives/blank lines, never data.
for (let i = 0; i < 5; i++) t.markByte();
assert.equal(t.ttftMs(), null);
assert.equal(t.forwardedChunks, 0);
});
test("interruption is recorded and does not reset other timing", async () => {
const t = createStreamTiming();
t.markForward();
await new Promise((r) => setTimeout(r, 5));
t.markForward();
t.markInterrupted();
assert.equal(t.interrupted, true);
assert.ok(t.ttftMs() !== null);
assert.ok(t.avgItlMs() !== null);
});
test("normal completion: totalMs() is monotonic and >= first-forward latency", async () => {
const t = createStreamTiming();
await new Promise((r) => setTimeout(r, 15));
t.markForward();
const total = t.totalMs();
const ttft = t.ttftMs();
assert.ok(total >= 15);
assert.ok(ttft !== null && ttft <= total, "ttft must be <= total duration");
});
test("max inter-chunk samples are bounded (memory bound)", async () => {
const t = createStreamTiming();
for (let i = 0; i < 200; i++) t.markForward();
assert.ok(t.interChunkGaps.length <= 32, `bounded to 32 samples, got ${t.interChunkGaps.length}`);
});

View File

@@ -1,34 +0,0 @@
import test from "node:test";
import assert from "node:assert/strict";
import { IMAGE_PROVIDERS, parseImageModel } from "../../open-sse/config/imageRegistry.ts";
test("#10848 bare id that only exists on a cookie-auth web bridge should not silently resolve to it", () => {
const chatgptWeb = IMAGE_PROVIDERS["chatgpt-web"];
assert.equal(chatgptWeb.authHeader, "cookie");
const otherProvidersWithSameId = Object.entries(IMAGE_PROVIDERS).filter(
([providerId, config]) =>
providerId !== "chatgpt-web" && config.models.some((m) => m.id === "gpt-5.5")
);
assert.deepEqual(
otherProvidersWithSameId,
[],
"expected only chatgpt-web (cookie) to register gpt-5.5"
);
const resolved = parseImageModel("gpt-5.5");
assert.notDeepEqual(
resolved,
{ provider: "chatgpt-web", model: "gpt-5.5" },
"bare 'gpt-5.5' must not silently bind to the cookie-auth chatgpt-web bridge"
);
assert.deepEqual(parseImageModel("chatgpt-web/gpt-5.5"), {
provider: "chatgpt-web",
model: "gpt-5.5",
});
assert.deepEqual(parseImageModel("cgpt-web/gpt-5.5"), {
provider: "chatgpt-web",
model: "gpt-5.5",
});
});

View File

@@ -10,7 +10,7 @@ import assert from "node:assert/strict";
//
// The `zai` provider uses the DefaultExecutor, which sends the requested model ID
// verbatim. The effort tiers `glm-5.2-high` / `glm-5.2-max` are OmniRoute aliases
// that only the GlmExecutor knows how to resolve (parseGlmEffortTier → base model
// that only the GlmExecutor knows how to resolve (parseGlm52Effort → base model
// "glm-5.2" + `effort` field + effort-2025-11-24 beta header). Listing them under
// `zai` would send unknown model IDs to z.ai's Anthropic endpoint, so they belong
// to the `glm` provider only.