diff --git a/2026-06-20-Unified-Compression-Config-Panel-Design.md b/2026-06-20-Unified-Compression-Config-Panel-Design.md new file mode 100644 index 0000000..ec55b9d --- /dev/null +++ b/2026-06-20-Unified-Compression-Config-Panel-Design.md @@ -0,0 +1,237 @@ +> 🌍 [View in other languages](Languages) + + +# Unified Compression Config Panel β€” Design + +**Status:** approved direction (2026-06-20), pending spec review +**Base branch:** `release/v3.8.31` +**Goal:** Make `/dashboard/context/settings` the single management panel for the master +on/off and each compression engine's on/off + level, deriving the default pipeline from +those toggles. Keep the Combos page solely for *chaining* (ordered named pipelines) + +selecting the globally-active profile, and the per-engine pages solely for *detailed* +config. Plan for (but phase) multiple named profiles with an active selector and a +per-request override header. + +--- + +## 1. Background β€” current state (the problem) + +Compression on/off + level is split across **two stores and several UIs**, with real +duplication: + +- **`/api/settings/compression`** (DB `key_value` ns=`compression`, via `src/lib/db/compression.ts`) + holds: `enabled` (master), `defaultMode`, `autoTriggerTokens`, `cavemanConfig`, + `cavemanOutputMode`, `rtkConfig`, `aggressive`, `ultra`, language config, etc. +- **`/api/context/combos/default`** (the *default combo pipeline*) holds the per-engine + `enabled` + config for the **structural** engines (lite, headroom, session-dedup, ccr, + llmlingua, aggressive, ultra) as pipeline steps. The per-engine detail pages + (`EngineConfigPage.tsx`) read/write here via `setEngineInDefaultCombo`. +- **`compression_combos`** table holds *named* pipelines assigned to routing combos. + +Concrete duplications (from the UI audit): +- Caveman on/off in **3** places (TokenSaverCard, CompressionSettingsTab, CavemanContextPageClient). +- Caveman intensity in **3**; RTK intensity in **2**; Caveman output mode in **2**. +- `lite/session-dedup/headroom/ccr/llmlingua` on/off **absent** from the central settings β€” + only on their per-engine page (which writes the *default combo*, not the central config). + +So "is engine X on?" is answered inconsistently (central config for caveman/rtk; default +combo for the structural engines), and the **Combos page edits the same default pipeline** +that a central panel would β€” the conceptual overlap the redesign must remove. + +Out of scope as a *config* surface: `dashboard/compression/studio` is the Compression +Studio (waterfall/cockpit analytics), not toggles β€” untouched. + +Engines (the 10 stackable units): `lite, caveman, aggressive, ultra, rtk, headroom, +session-dedup, ccr, llmlingua` (registered `CompressionEngine`s) + `mcpAccessibility` +(separate path: compresses MCP tool-result outputs, not the chat pipeline). Each engine's +`stackPriority` defines automatic ordering (session-dedup 3, ccr 4, lite 5, rtk 10, +headroom 15, caveman 20, aggressive 30, llmlingua 35, ultra 40). + +--- + +## 2. Design principles β€” the boundary + +| Surface | Concept | Owns | Never does | +|---|---|---|---| +| **Panel** (`context/settings`) | "My **Default**: what is on + level" | master on/off; per-engine on/off + level | ordering/chaining; per-route assignment | +| **Combos** (`context/combos`, menu #2) | "Named **profiles**: chaining + which is active + per-route assignment" | create/edit ordered named pipelines; pick the globally-active profile; assign to routing combos | engine on/off (inherits the active profile's membership) | +| **Per-engine pages** (menu, after combos) | "Deep config of one engine" | filters, rules, language packs, thresholds | on/off; level (those live in the panel) | + +**No duplication rule:** the **Panel owns the Default profile** (membership + level, order +auto-derived by `stackPriority`). A **Combo is a *named alternative* profile** (membership ++ level + *explicit order*). Different scopes (the one default vs named alternatives) β€” not +the same object edited twice. The editable "default combo" store is **removed**; the +default pipeline becomes **derived** from the panel. + +--- + +## 3. Architecture + +### 3.1 Resolution model (per request, most-specific wins) + +``` +x-omniroute-compression header (per-request override) + β†’ routing-combo override (comboOverrides[comboId]) (per-route) + β†’ active profile (activeComboId: "default" | ) (global) + β†’ Default = derived from panel engines map + β†’ off (master disabled, or zero engines on) +``` + +A single `resolveCompressionPlan(config, { comboId, header })` produces the effective +`{ mode, stackedPipeline }` fed to the existing `applyCompressionAsync`. It supersedes +today's `getEffectiveMode` (which only does combo-override β†’ autoTrigger β†’ defaultMode). + +### 3.2 Data model (Model A β€” single source + derived pipeline) + +**CompressionConfig gains an engines map** (the single source for the Default's on/off + +level), replacing the editable default-combo store: + +```ts +interface EngineToggle { + enabled: boolean; + level?: string; // caveman/cavemanOutput: lite|full|ultra ; rtk: minimal|standard|aggressive ; others: undefined +} +interface CompressionConfig { + enabled: boolean; // master + engines: Record; // NEW β€” the Default profile + activeComboId: string | null; // NEW β€” null/"default" = derived default; else a compression_combos id + autoTriggerTokens: number; + autoTriggerMode?: CompressionMode; // kept (auto-trigger still selects a profile/mode on large prompts) + preserveSystemPrompt: boolean; + comboOverrides: Record; // existing per-routing-combo override + // detailed per-engine config keeps living in its existing sub-objects + // (cavemanConfig, rtkConfig, aggressive, ultra, …) β€” edited by the per-engine pages. +} +``` + +**Deriving the Default pipeline** from `engines` (pure function `deriveDefaultPlan`): +- master `enabled === false` β†’ `off`. +- 0 engines enabled β†’ `off`. +- exactly 1 enabled and it is a single-mode engine (`lite|caveman|aggressive|ultra|rtk`) β†’ + that mode (single path β€” reuses today's `applyCompression(mode)`). +- otherwise β†’ `stacked` with the enabled engines in `stackPriority` order, each step's + `intensity` from its `level` (the global `stackedPipeline` already accepts all 9 engines + after the B-PIPELINE-DIVERGENCE fix). + +The **stored `defaultMode` field** and the editable default combo +(`/api/context/combos/default`, `setEngineInDefaultCombo`) are **removed**; the default is +derived from `engines`. (The `CompressionMode` *type* persists β€” `comboOverrides`, +`autoTriggerMode`, and the resolver's output still use it; only the stored `defaultMode` +field is dropped.) A DB migration backfills `engines` from the current `defaultMode` + +default-combo steps + caveman/rtk config so existing installs keep their behavior. + +**Named combos** (`compression_combos`, existing table): N ordered pipelines. `activeComboId` +selects which profile is globally active (`Default` or a named combo). Per-routing-combo +assignment stays in `comboOverrides`. + +`mcpAccessibility` keeps its own store (`/api/settings/compression/mcp-accessibility`, +migration 056) β€” surfaced in the panel as a row that writes there, with a scope note. + +### 3.3 The per-request header + +`x-omniroute-compression: ` (mirrors the `x-omniroute-no-memory`/`no-cache` pattern, +PR #4290; parsed in the request pipeline alongside the other omniroute headers). Values: +- `off` β†’ no compression for this request. +- `default` β†’ the derived Default profile. +- `` β†’ that named combo. +- `engine:` β†’ a single engine (if that engine is enabled in the Default), e.g. `engine:rtk`. + +Invalid/unknown value β†’ ignored (falls through to the normal resolution); never errors the +request. Header parsing + validation has a unit test asserting each form and the +fall-through. + +--- + +## 4. Screens + +### 4.1 Panel β€” `context/settings` (engine grid) + +A single client component (replacing the scattered `CompressionSettingsTab` + +`CompressionTokenSaverCard` toggles): +- **Master** on/off at top. +- **Engine grid** (one row per engine, ordered by `stackPriority` so the row order mirrors + run order): `[engine name + short desc] [on/off toggle] [level selector if applicable] + [β†’ detail page]`. Level selector appears only for engines with levels + (caveman lite|full|ultra, rtk minimal|standard|aggressive; caveman output mode as its own row). +- **General** (auto-trigger tokens, preserve-system-prompt) below the grid. +- Reads/writes the `engines` map + master via `GET/PUT /api/settings/compression` (single + endpoint). The displayed default pipeline (derived) is shown read-only ("runs: rtk β†’ + caveman β†’ …") so the user sees the effect without editing order here. + +### 4.2 Combos β€” `context/combos` (menu #2) + +- List of named combos; create/edit an **ordered** pipeline (drag/reorder + per-step level) + β€” the only place for explicit chaining. +- **Active profile selector**: `Default (panel)` | `` β†’ writes `activeComboId`. +- Assign a combo to a routing combo (existing `comboOverrides`). +- Reuses the existing `CompressionCombosPageClient` / `comboFlowModel`; the `CompressionHub` + "master mode selector" is removed (mode is now derived; the Hub becomes a read-only + overview or is folded into the panel). + +### 4.3 Per-engine pages (menu, after combos) + +`EngineConfigPage` + the caveman/rtk custom pages: **lose** the on/off + level controls +(moved to the panel) and keep only **detailed** config (filters, rules, language packs, +thresholds, preview). They **stop writing** `/api/context/combos/default`; detailed config +writes to its own sub-object in `/api/settings/compression` (or the existing +caveman/rtk facade routes, which already proxy to it). + +### 4.4 Navigation + +`COMPRESSION_CONTEXT_GROUP` (`src/shared/constants/sidebarVisibility.ts`) reordered: +**Settings (panel) β†’ Combos β†’ per-engine pages β†’ Studio (analytics)**. + +--- + +## 5. Consolidation / migration + +- `CompressionTokenSaverCard` quick toggles β†’ **absorbed into the panel; the card is removed**. +- Duplicate caveman/rtk on/off + intensity in `CompressionSettingsTab` β†’ removed. +- `EngineConfigPage` β†’ on/off+level removed; stops writing the default combo. +- DB migration: backfill `engines` map + `activeComboId="default"` from current state + (defaultMode + default-combo steps + caveman/rtk/ultra/aggressive enabled), so live + installs preserve behavior. The editable default-combo route (`PUT /api/context/combos/default`) + becomes a **read-only shim for one release** (returns the derived default; rejects writes + with a deprecation note), then is removed. + +--- + +## 6. Phasing + +Each phase is its own implementation plan + PR, independently shippable and TDD'd +(Hard Rule #18). `writing-plans` will author **Phase 1 first**; Phases 2–3 get their own +plans later. + +- **Phase 1 (core consolidation):** the `engines` map + `deriveDefaultPlan` + migration; + the engine-grid panel; remove scattered/duplicate toggles; per-engine pages lose on/off; + menu reorder. Delivers the single-panel goal. *No behavior change for existing installs + (migration backfills).* +- **Phase 2 (profiles):** multiple named combos + the `activeComboId` active-profile selector. +- **Phase 3 (header):** the `x-omniroute-compression` per-request override. + +Phases 2–3 reuse the Phase-1 resolution model (`resolveCompressionPlan`), which is built +header/active-aware from the start so later phases only wire UI + header parsing. + +--- + +## 7. Testing + +- **Unit:** `deriveDefaultPlan` (every engines-map shape β†’ expected mode/pipeline); + migration backfill (old config β†’ equivalent engines map); `resolveCompressionPlan` + precedence (header > comboOverride > active > default > off); header parsing (each form + + fall-through); panel reducer (toggle/level edits β†’ config patch). +- **Component (vitest):** the panel renders all engines, toggles persist, derived pipeline + preview updates. +- **Integration:** a config with engines `{rtk,caveman}` on β†’ `applyCompressionAsync` runs the + derived stacked pipeline; single engine on β†’ single-mode path; equivalence with the old + defaultMode behavior for the same logical config. +- Both runners green (`test:unit` + `test:vitest`); typecheck:core clean; lint 0 errors. + +--- + +## 8. Non-goals (YAGNI) + +- No new compression engines in this work (the model just makes adding them trivial later). +- No change to the engines' internal algorithms (the recent fixes are separate). +- The Compression Studio (analytics) is not restructured. +- Phases 2–3 UI polish (combo templates, sharing) is out of scope. diff --git a/2026-06-20-Unified-Compression-Config-Panel-Plan.md b/2026-06-20-Unified-Compression-Config-Panel-Plan.md new file mode 100644 index 0000000..86fa233 --- /dev/null +++ b/2026-06-20-Unified-Compression-Config-Panel-Plan.md @@ -0,0 +1,492 @@ +> 🌍 [View in other languages](Languages) + + +# Unified Compression Config Panel β€” Phase 1 Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make `/dashboard/context/settings` the single source for the master + per-engine on/off + level, with the default compression pipeline DERIVED from those toggles, removing the scattered/duplicate toggles β€” with zero behavior change for existing installs (a migration backfills). + +**Architecture:** Add an `engines` map (+`activeComboId`) to `CompressionConfig` as the single source. A pure `deriveDefaultPlan(engines)` turns it into `{mode, stackedPipeline}`; a pure `resolveCompressionPlan(config, ctx)` applies precedence (headerβ†’combo-overrideβ†’active-profileβ†’derived-defaultβ†’off) and feeds the existing `applyCompressionAsync`. A DB migration backfills the map. The engine-grid panel reads/writes the map via the single `/api/settings/compression` endpoint; per-engine pages lose on/off+level; the editable default-combo route becomes a read-only shim. + +**Tech Stack:** TypeScript, Next.js 16 App Router, Zod, SQLite (better-sqlite3), Node test runner + Vitest (component), React. + +**Base:** worktree `feat/compression-config-panel-v3831` off `release/v3.8.31`. Spec: `docs/compression/2026-06-20-unified-compression-config-panel-design.md`. + +**Conventions:** unit tests run `node --import tsx/esm --test tests/unit/.test.ts`; component tests run `npm run test:vitest`. Each task ends by running the FULL compression suite (`node --import tsx/esm --test tests/unit/compression/*.test.ts`) + `npm run typecheck:core` before commit. Never `--no-verify`. + +--- + +## File Structure + +**Create:** +- `open-sse/services/compression/engineCatalog.ts` β€” pure metadata: per-engine `{ id, label, stackPriority, levels?, isSingleMode }`. One source of truth for "which engines exist, which have levels, which can be a standalone mode". +- `open-sse/services/compression/deriveDefaultPlan.ts` β€” pure: `engines` map β†’ `{ mode, stackedPipeline }`. +- `open-sse/services/compression/resolveCompressionPlan.ts` β€” pure: precedence resolver. +- `src/lib/db/migrations/102_compression_engines_map.sql` β€” backfill `engines` + `activeComboId`. +- `src/app/(dashboard)/dashboard/context/settings/CompressionPanel.tsx` β€” the engine-grid panel. +- `tests/unit/compression/engine-catalog.test.ts`, `derive-default-plan.test.ts`, `resolve-compression-plan.test.ts`, `compression-engines-map-migration.test.ts`. +- `tests/unit/ui/compressionPanel.test.tsx` (vitest). + +**Modify:** +- `open-sse/services/compression/types.ts` β€” add `EngineToggle`, `CompressionConfig.engines`, `activeComboId`; keep `CompressionMode` type; drop stored `defaultMode` usage (derive). +- `src/lib/db/compression.ts` β€” normalize/persist `engines` + `activeComboId`. +- `open-sse/services/compression/strategySelector.ts` β€” `selectCompressionStrategy`/`getEffectiveMode` delegate to `resolveCompressionPlan`. +- `open-sse/handlers/chatCore.ts` β€” call `resolveCompressionPlan`. +- `src/app/api/settings/compression/route.ts` β€” accept/return `engines` + `activeComboId`. +- `src/app/api/context/combos/default/route.ts` β€” read-only shim (reject writes). +- `src/app/(dashboard)/dashboard/context/settings/page.tsx` β€” render `CompressionPanel` (not the old tab). +- `src/shared/components/compression/EngineConfigPage.tsx` + caveman/rtk client pages β€” remove on/off+level; stop writing default combo. +- `src/shared/constants/sidebarVisibility.ts` β€” reorder `COMPRESSION_CONTEXT_GROUP`. + +**Engine reference (`stackPriority`, levels, single-mode):** +`session-dedup`(3,β€”,no) `ccr`(4,β€”,no) `lite`(5,β€”,yes) `rtk`(10,minimal|standard|aggressive,yes) `headroom`(15,β€”,no) `caveman`(20,lite|full|ultra,yes) `aggressive`(30,β€”,yes) `llmlingua`(35,β€”,no) `ultra`(40,β€”,yes). Plus `cavemanOutput`(intensity lite|full|ultra, separate from input caveman) and `mcpAccessibility`(no level, separate store). + +--- + +## Task 1: Engine catalog (single source of engine metadata) + +**Files:** +- Create: `open-sse/services/compression/engineCatalog.ts` +- Test: `tests/unit/compression/engine-catalog.test.ts` + +- [ ] **Step 1: Write the failing test** + +```ts +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { ENGINE_CATALOG, engineMeta, ENGINE_IDS } from "@omniroute/open-sse/services/compression/engineCatalog.ts"; + +test("catalog lists every engine with stackPriority", () => { + for (const id of ["session-dedup","ccr","lite","rtk","headroom","caveman","aggressive","llmlingua","ultra"]) { + assert.ok(engineMeta(id), `${id} present`); + assert.equal(typeof engineMeta(id).stackPriority, "number"); + } +}); +test("levels + single-mode flags are correct", () => { + assert.deepEqual(engineMeta("rtk").levels, ["minimal","standard","aggressive"]); + assert.deepEqual(engineMeta("caveman").levels, ["lite","full","ultra"]); + assert.equal(engineMeta("headroom").levels, undefined); + assert.equal(engineMeta("caveman").isSingleMode, true); + assert.equal(engineMeta("headroom").isSingleMode, false); +}); +test("ENGINE_IDS is ordered by stackPriority", () => { + const ps = ENGINE_IDS.map((id) => engineMeta(id).stackPriority); + assert.deepEqual(ps, [...ps].sort((a,b)=>a-b)); +}); +``` + +- [ ] **Step 2: Run β†’ FAIL** (`node --import tsx/esm --test tests/unit/compression/engine-catalog.test.ts`) β€” module not found. + +- [ ] **Step 3: Implement** `engineCatalog.ts`: + +```ts +export interface EngineMeta { + id: string; + label: string; + stackPriority: number; + levels?: string[]; // intensity options; undefined = no level selector + isSingleMode: boolean; // can be the effective mode when it is the only engine on + description: string; +} +export const ENGINE_CATALOG: Record = { + "session-dedup": { id:"session-dedup", label:"Session Dedup", stackPriority:3, isSingleMode:false, description:"Cross-turn block deduplication." }, + ccr: { id:"ccr", label:"CCR (Retrieval)", stackPriority:4, isSingleMode:false, description:"Content-addressed retrieval markers." }, + lite: { id:"lite", label:"Lite", stackPriority:5, isSingleMode:true, description:"Whitespace/format cleanup." }, + rtk: { id:"rtk", label:"RTK", stackPriority:10, levels:["minimal","standard","aggressive"], isSingleMode:true, description:"Command-output filtering." }, + headroom: { id:"headroom", label:"Headroom", stackPriority:15, isSingleMode:false, description:"Tabular JSON compaction." }, + caveman: { id:"caveman", label:"Caveman", stackPriority:20, levels:["lite","full","ultra"], isSingleMode:true, description:"Rule-based prose compression." }, + aggressive: { id:"aggressive", label:"Aggressive", stackPriority:30, isSingleMode:true, description:"Summarize + age old turns." }, + llmlingua: { id:"llmlingua", label:"LLMLingua (SLM)", stackPriority:35, isSingleMode:false, description:"Semantic pruning (ONNX)." }, + ultra: { id:"ultra", label:"Ultra", stackPriority:40, isSingleMode:true, description:"Heuristic token pruning (+ optional SLM)." }, +}; +export const ENGINE_IDS: string[] = Object.values(ENGINE_CATALOG).sort((a,b)=>a.stackPriority-b.stackPriority).map((e)=>e.id); +export function engineMeta(id: string): EngineMeta { return ENGINE_CATALOG[id]; } +``` + +- [ ] **Step 4: Run β†’ PASS.** +- [ ] **Step 5: Commit** `git add open-sse/services/compression/engineCatalog.ts tests/unit/compression/engine-catalog.test.ts && git commit -m "feat(compression): engine catalog metadata (levels, single-mode, order)"` + +--- + +## Task 2: `EngineToggle` + `engines` map + `activeComboId` on the config type + +**Files:** +- Modify: `open-sse/services/compression/types.ts` (the `CompressionConfig` interface + `DEFAULT_COMPRESSION_CONFIG`) +- Test: `tests/unit/compression/engine-catalog.test.ts` (extend) β€” assert the default config shape. + +- [ ] **Step 1: Add the test** (append): + +```ts +import { DEFAULT_COMPRESSION_CONFIG } from "@omniroute/open-sse/services/compression/types.ts"; +test("default config has an engines map + activeComboId", () => { + assert.equal(typeof DEFAULT_COMPRESSION_CONFIG.engines, "object"); + assert.equal(DEFAULT_COMPRESSION_CONFIG.activeComboId, null); + // default-off: every engine disabled by default (opt-in preserved) + for (const id of ENGINE_IDS) assert.equal(DEFAULT_COMPRESSION_CONFIG.engines[id]?.enabled, false); +}); +``` + +- [ ] **Step 2: Run β†’ FAIL.** +- [ ] **Step 3: Implement** in `types.ts`: add `export interface EngineToggle { enabled: boolean; level?: string }`; add to `CompressionConfig`: `engines: Record;` and `activeComboId: string | null;`. In `DEFAULT_COMPRESSION_CONFIG` add `engines: Object.fromEntries(ENGINE_IDS.map((id)=>[id,{enabled:false}]))` (import `ENGINE_IDS`) and `activeComboId: null`. Keep `defaultMode` field for now (removed in Task 9 once derive is wired) to avoid breaking compilation. + +- [ ] **Step 4: Run β†’ PASS + `npm run typecheck:core`.** +- [ ] **Step 5: Commit** `... -m "feat(compression): add engines map + activeComboId to CompressionConfig"` + +--- + +## Task 3: `deriveDefaultPlan` (the heart β€” engines map β†’ mode/pipeline) + +**Files:** +- Create: `open-sse/services/compression/deriveDefaultPlan.ts` +- Test: `tests/unit/compression/derive-default-plan.test.ts` + +- [ ] **Step 1: Write the failing test** + +```ts +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { deriveDefaultPlan } from "@omniroute/open-sse/services/compression/deriveDefaultPlan.ts"; + +const on = (level) => ({ enabled: true, ...(level?{level}:{}) }); +test("master off / empty / none-on => off", () => { + assert.deepEqual(deriveDefaultPlan({}, false), { mode:"off", stackedPipeline:[] }); + assert.deepEqual(deriveDefaultPlan({}, true), { mode:"off", stackedPipeline:[] }); + assert.deepEqual(deriveDefaultPlan({ rtk:{enabled:false} }, true), { mode:"off", stackedPipeline:[] }); +}); +test("exactly one single-mode engine => that mode", () => { + assert.deepEqual(deriveDefaultPlan({ caveman: on("full") }, true), { mode:"standard", stackedPipeline:[] }); + assert.deepEqual(deriveDefaultPlan({ rtk: on("minimal") }, true), { mode:"rtk", stackedPipeline:[] }); + assert.deepEqual(deriveDefaultPlan({ lite: on() }, true), { mode:"lite", stackedPipeline:[] }); +}); +test("one non-single-mode engine => stacked with that engine", () => { + const p = deriveDefaultPlan({ headroom: on() }, true); + assert.equal(p.mode, "stacked"); + assert.deepEqual(p.stackedPipeline, [{ engine:"headroom" }]); +}); +test("multiple engines => stacked in stackPriority order, levels as intensity", () => { + const p = deriveDefaultPlan({ caveman: on("full"), rtk: on("standard"), headroom: on() }, true); + assert.equal(p.mode, "stacked"); + assert.deepEqual(p.stackedPipeline, [ + { engine:"rtk", intensity:"standard" }, // pri 10 + { engine:"headroom" }, // pri 15 + { engine:"caveman", intensity:"full" }, // pri 20 + ]); +}); +``` + +- [ ] **Step 2: Run β†’ FAIL.** +- [ ] **Step 3: Implement** `deriveDefaultPlan.ts`: + +```ts +import { ENGINE_CATALOG, engineMeta } from "./engineCatalog.ts"; +import type { EngineToggle } from "./types.ts"; + +const SINGLE_MODE_OF: Record = { lite:"lite", caveman:"standard", aggressive:"aggressive", ultra:"ultra", rtk:"rtk" }; + +export interface DerivedPlan { mode: string; stackedPipeline: Array<{ engine: string; intensity?: string }>; } + +export function deriveDefaultPlan(engines: Record, masterEnabled: boolean): DerivedPlan { + if (!masterEnabled) return { mode:"off", stackedPipeline:[] }; + const onIds = Object.keys(ENGINE_CATALOG).filter((id) => engines[id]?.enabled === true); + if (onIds.length === 0) return { mode:"off", stackedPipeline:[] }; + if (onIds.length === 1 && engineMeta(onIds[0]).isSingleMode) { + return { mode: SINGLE_MODE_OF[onIds[0]], stackedPipeline:[] }; + } + const ordered = onIds.sort((a,b)=>engineMeta(a).stackPriority-engineMeta(b).stackPriority); + const stackedPipeline = ordered.map((id) => { + const level = engines[id]?.level; + return level ? { engine:id, intensity:level } : { engine:id }; + }); + return { mode:"stacked", stackedPipeline }; +} +``` + +- [ ] **Step 4: Run β†’ PASS.** +- [ ] **Step 5: Commit** `... -m "feat(compression): deriveDefaultPlan (engines map -> mode/pipeline)"` + +--- + +## Task 4: Migration 102 β€” backfill `engines` + `activeComboId` + +**Files:** +- Create: `src/lib/db/migrations/102_compression_engines_map.sql` +- Test: `tests/unit/compression/compression-engines-map-migration.test.ts` + +- [ ] **Step 1: Write the failing test** (uses `resetDbInstance()` + closes handles in `test.after`, per repo rule): + +```ts +import { test, after } from "node:test"; +import assert from "node:assert/strict"; +import { getDbInstance, resetDbInstance } from "@/lib/db/core.ts"; +import { getCompressionSettings } from "@/lib/db/compression.ts"; + +after(() => resetDbInstance()); +test("migration backfills engines map from prior defaultMode + default combo", () => { + resetDbInstance(); + const db = getDbInstance(); // runs migrations incl. 102 + // simulate a pre-102 install: master on, defaultMode 'standard', caveman enabled + db.prepare("INSERT OR REPLACE INTO key_value(namespace,key,value) VALUES('compression','enabled','true')").run(); + db.prepare("INSERT OR REPLACE INTO key_value(namespace,key,value) VALUES('compression','defaultMode','\"standard\"')").run(); + db.prepare("INSERT OR REPLACE INTO key_value(namespace,key,value) VALUES('compression','cavemanConfig','{\"enabled\":true}')").run(); + const cfg = getCompressionSettings(); + assert.equal(cfg.engines.caveman.enabled, true); + assert.equal(cfg.activeComboId, null); +}); +``` + +- [ ] **Step 2: Run β†’ FAIL.** +- [ ] **Step 3: Implement** the SQL migration (idempotent; the row backfill of derived `engines` is done in `normalizeCompressionSettings` read-path β€” the SQL only seeds `activeComboId` default and a marker). Migration `102_compression_engines_map.sql`: + +```sql +-- Phase 1 of the unified compression panel: the engines map + activeComboId become the +-- single source. The engines map is DERIVED on read (normalizeCompressionSettings) from the +-- legacy defaultMode + default-combo steps + caveman/rtk/ultra/aggressive config, so existing +-- installs keep their behavior. Here we only ensure activeComboId defaults to NULL ("default"). +INSERT OR IGNORE INTO key_value (namespace, key, value) VALUES ('compression', 'activeComboId', 'null'); +``` + +(The read-path derivation in Task 5 is what makes `getCompressionSettings().engines` correct; the migration just guarantees `activeComboId` exists.) + +- [ ] **Step 4: Run β†’ PASS** (after Task 5's normalize is in β€” if running 4 before 5, expect the engines assertion to fail; do Task 5 then re-run). Commit after Task 5. + +--- + +## Task 5: Persist + normalize `engines` / `activeComboId` (read-path derivation) + +**Files:** +- Modify: `src/lib/db/compression.ts` (`getCompressionSettings`/`normalizeCompressionSettings`/`updateCompressionSettings`) +- Test: reuse Task 4's migration test + add a round-trip test in the same file. + +- [ ] **Step 1: Add round-trip test**: + +```ts +import { updateCompressionSettings } from "@/lib/db/compression.ts"; +test("engines map persists round-trip + activeComboId", () => { + resetDbInstance(); getDbInstance(); + updateCompressionSettings({ enabled:true, engines:{ rtk:{enabled:true,level:"standard"}, caveman:{enabled:true,level:"full"} }, activeComboId:null }); + const cfg = getCompressionSettings(); + assert.equal(cfg.engines.rtk.enabled, true); + assert.equal(cfg.engines.rtk.level, "standard"); + assert.equal(cfg.engines.caveman.level, "full"); +}); +``` + +- [ ] **Step 2: Run β†’ FAIL.** +- [ ] **Step 3: Implement** in `compression.ts`: + - In `normalizeCompressionSettings`: read stored `engines` if present; ELSE derive it from legacy fields β€” `engines[id].enabled` from: caveman/rtk/ultra/aggressive `*.enabled`; structural engines from the default-combo steps (read the default combo); single-modes from `defaultMode`. Levels from `cavemanConfig.intensity`/`rtkConfig.intensity`. Read `activeComboId` (default null). + - In `updateCompressionSettings`: accept `engines` (validate each value `{enabled:boolean, level?:string}`) + `activeComboId` and persist as `key_value` rows (`engines` as one JSON row). + - Add a Zod sub-schema `engineToggleSchema = z.object({ enabled: z.boolean(), level: z.string().optional() })` and `engines: z.record(engineToggleSchema).optional()`, `activeComboId: z.string().nullable().optional()`. + +- [ ] **Step 4: Run β†’ PASS** (Task 4 + Task 5 tests). `npm run typecheck:core`. +- [ ] **Step 5: Commit** `git add src/lib/db/migrations/102_compression_engines_map.sql src/lib/db/compression.ts tests/unit/compression/compression-engines-map-migration.test.ts && git commit -m "feat(compression): persist+backfill engines map and activeComboId (migration 102)"` + +--- + +## Task 6: `resolveCompressionPlan` (precedence resolver) + +**Files:** +- Create: `open-sse/services/compression/resolveCompressionPlan.ts` +- Test: `tests/unit/compression/resolve-compression-plan.test.ts` + +- [ ] **Step 1: Write the failing test** + +```ts +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { resolveCompressionPlan } from "@omniroute/open-sse/services/compression/resolveCompressionPlan.ts"; + +const base = { enabled:true, engines:{ caveman:{enabled:true,level:"full"} }, activeComboId:null, comboOverrides:{} }; +test("derived default when no override/active/header", () => { + assert.deepEqual(resolveCompressionPlan(base, {}), { mode:"standard", stackedPipeline:[] }); +}); +test("routing-combo override wins over default", () => { + const cfg = { ...base, comboOverrides:{ cmb:"aggressive" } }; + assert.equal(resolveCompressionPlan(cfg, { comboId:"cmb" }).mode, "aggressive"); +}); +test("active named combo wins over default (Phase 2 wiring uses combos table; here pass it in)", () => { + const cfg = { ...base, activeComboId:"c1" }; + const combos = { c1: [{ engine:"rtk", intensity:"standard" }] }; + const plan = resolveCompressionPlan(cfg, { combos }); + assert.equal(plan.mode, "stacked"); + assert.deepEqual(plan.stackedPipeline, [{ engine:"rtk", intensity:"standard" }]); +}); +test("master off => off regardless", () => { + assert.equal(resolveCompressionPlan({ ...base, enabled:false }, {}).mode, "off"); +}); +``` + +- [ ] **Step 2: Run β†’ FAIL.** +- [ ] **Step 3: Implement** `resolveCompressionPlan.ts`: + +```ts +import { deriveDefaultPlan, type DerivedPlan } from "./deriveDefaultPlan.ts"; + +export interface ResolveCtx { + comboId?: string | null; + header?: string | null; // x-omniroute-compression (Phase 3 parses+passes; Phase 1 callers pass undefined) + combos?: Record>; // named combo pipelines by id +} +export function resolveCompressionPlan(config: any, ctx: ResolveCtx): DerivedPlan { + if (config?.enabled === false) return { mode:"off", stackedPipeline:[] }; + // 1. header (Phase 3 supplies parsed value; here it composes if present) + if (ctx.header) { + if (ctx.header === "off") return { mode:"off", stackedPipeline:[] }; + if (ctx.header !== "default") { + const fromHeader = headerToPlan(ctx.header, config, ctx); + if (fromHeader) return fromHeader; // unknown => fall through + } + } + // 2. routing-combo override + const ov = ctx.comboId ? config?.comboOverrides?.[ctx.comboId] : undefined; + if (ov) return modeToPlan(ov, config); + // 3. active named combo + if (config?.activeComboId && ctx.combos?.[config.activeComboId]) { + return { mode:"stacked", stackedPipeline: ctx.combos[config.activeComboId] }; + } + // 4. derived default + return deriveDefaultPlan(config?.engines ?? {}, config?.enabled !== false); +} +function modeToPlan(mode: string, config: any): DerivedPlan { + return mode === "stacked" + ? { mode:"stacked", stackedPipeline: config?.stackedPipeline ?? [] } + : { mode, stackedPipeline:[] }; +} +function headerToPlan(h: string, config: any, ctx: ResolveCtx): DerivedPlan | null { + if (h.startsWith("engine:")) { const id = h.slice(7); return config?.engines?.[id]?.enabled ? deriveDefaultPlan({ [id]: config.engines[id] }, true) : null; } + if (ctx.combos?.[h]) return { mode:"stacked", stackedPipeline: ctx.combos[h] }; + return null; +} +``` + +- [ ] **Step 4: Run β†’ PASS.** +- [ ] **Step 5: Commit** `... -m "feat(compression): resolveCompressionPlan precedence resolver (header>override>active>default)"` + +--- + +## Task 7: Wire the resolver into strategy selection + chatCore + +**Files:** +- Modify: `open-sse/services/compression/strategySelector.ts` (`selectCompressionStrategy`) +- Modify: `open-sse/handlers/chatCore.ts` (the compression call site) +- Test: `tests/unit/compression/strategySelector.test.ts` (extend with an engines-map case) + +- [ ] **Step 1: Add test** asserting `selectCompressionStrategy` with `engines:{rtk:{enabled:true}}` + master on returns mode `rtk`; with `{rtk,caveman}` returns `stacked`. (Use the existing test's import + harness.) +- [ ] **Step 2: Run β†’ FAIL.** +- [ ] **Step 3: Implement** β€” `selectCompressionStrategy` calls `resolveCompressionPlan(config, { comboId, combos })` and returns its `mode` (and expose the `stackedPipeline` so `applyCompressionAsync` uses the derived pipeline when mode==="stacked"). Load named `combos` from the combos DB module. In `chatCore.ts`, pass the active combo set; keep the `header` arg `undefined` (Phase 3 fills it). Keep `autoTriggerMode` behavior (auto-trigger still overrides to its mode on large prompts β€” apply BEFORE step 4 default). +- [ ] **Step 4: Run β†’ PASS** + full compression suite + typecheck. +- [ ] **Step 5: Commit** `... -m "feat(compression): selectCompressionStrategy uses resolveCompressionPlan"` + +--- + +## Task 8: API β€” `/api/settings/compression` carries `engines` + `activeComboId` + +**Files:** +- Modify: `src/app/api/settings/compression/route.ts` +- Test: `tests/unit/api/compression/compression-api.test.ts` (extend) + +- [ ] **Step 1: Add test**: `PUT` with `{engines:{rtk:{enabled:true,level:"standard"}}}` then `GET` returns it; error body has no stack (`!body.error?.message?.includes("at /")`). +- [ ] **Step 2: Run β†’ FAIL.** +- [ ] **Step 3: Implement** β€” extend the route's Zod body schema with `engines` + `activeComboId` (reuse the db sub-schema); GET returns them; errors via `buildErrorBody`. +- [ ] **Step 4: Run β†’ PASS** + vitest if the route is covered there. +- [ ] **Step 5: Commit** `... -m "feat(api): settings/compression carries engines map + activeComboId"` + +--- + +## Task 9: Remove stored `defaultMode` write-path + default-combo editable route β†’ shim + +**Files:** +- Modify: `open-sse/services/compression/types.ts` (drop `defaultMode` from the persisted shape; keep `CompressionMode` type) +- Modify: `src/app/api/context/combos/default/route.ts` (PUT β†’ 410/deprecation; GET β†’ derived default read-only) +- Test: `tests/unit/api/...` for the shim (PUT rejected, GET returns derived). + +- [ ] **Step 1: Add test**: `PUT /api/context/combos/default` returns a deprecation error (not 200); `GET` returns the derived default pipeline. +- [ ] **Step 2: Run β†’ FAIL.** +- [ ] **Step 3: Implement** β€” `setEngineInDefaultCombo` no longer the write path; PUT route returns `buildErrorBody` "deprecated: edit engines in /api/settings/compression"; GET returns `deriveDefaultPlan(config.engines, config.enabled)`. Remove remaining reads of stored `defaultMode` (derive). +- [ ] **Step 4: Run β†’ PASS** + full suite + typecheck. +- [ ] **Step 5: Commit** `... -m "refactor(compression): derive default; default-combo route is a read-only shim"` + +--- + +## Task 10: The engine-grid panel UI + +**Files:** +- Create: `src/app/(dashboard)/dashboard/context/settings/CompressionPanel.tsx` +- Modify: `src/app/(dashboard)/dashboard/context/settings/page.tsx` (render `CompressionPanel`) +- Test: `tests/unit/ui/compressionPanel.test.tsx` (vitest, `createRoot`+`act`) + +- [ ] **Step 1: Write the failing component test**: render `CompressionPanel` with a stubbed `fetch` returning `{enabled:true, engines:{rtk:{enabled:true,level:"standard"}}}`; assert it renders a row per `ENGINE_IDS`, the rtk level shows "standard", toggling caveman issues a `PUT` with `engines.caveman.enabled:true`, and the derived-pipeline preview text appears. +- [ ] **Step 2: Run β†’ FAIL** (`npm run test:vitest`). +- [ ] **Step 3: Implement** `CompressionPanel.tsx`: master toggle; map `ENGINE_IDS` β†’ a row component `[label+desc][Toggle][LevelSelect if meta.levels][Link β†’ /dashboard/context/]`; a derived-pipeline preview computed client-side via `deriveDefaultPlan` (import the pure fn); a `cavemanOutput` row; an `mcpAccessibility` row (writes its own endpoint, with a "MCP tool outputs" note); general settings (auto-trigger, preserve system prompt). Save via `PUT /api/settings/compression` (debounced, merge-patch like the existing `save()` pattern in `CompressionSettingsTab`). Reuse existing primitives (`Toggle`, segmented control) from the current cards. +- [ ] **Step 4:** Update `page.tsx` to render ``. Run β†’ PASS. +- [ ] **Step 5: Commit** `... -m "feat(dashboard): engine-grid compression panel (single source for on/off + level)"` + +--- + +## Task 11: Consolidate β€” remove scattered/duplicate toggles + per-engine on/off + +**Files:** +- Modify: `src/app/(dashboard)/dashboard/settings/components/CompressionTokenSaverCard.tsx` (remove; or strip to a read-only summary linking to the panel) +- Modify: `CompressionSettingsTab.tsx` (remove duplicate caveman/rtk on/off + intensity sections; keep only things not in the panel, or delete if fully superseded) +- Modify: `src/shared/components/compression/EngineConfigPage.tsx` + `CavemanContextPageClient.tsx` + `RtkContextPageClient.tsx` (remove on/off + level; keep detailed config; stop writing `/api/context/combos/default`) +- Test: vitest render tests for the per-engine pages assert NO enabled toggle is present; the existing tests updated to the new shape (alignment, not masking). + +- [ ] **Step 1:** Update the affected render tests to expect the new (toggle-free) shape; run β†’ FAIL on the still-present toggles. +- [ ] **Step 2:** Implement the removals; per-engine detail save writes detailed config to its facade route (caveman/rtk) or the settings sub-object. +- [ ] **Step 3:** Run vitest + full compression suite β†’ PASS. +- [ ] **Step 4: Commit** `... -m "refactor(dashboard): remove duplicate compression toggles; per-engine pages keep only detailed config"` + +--- + +## Task 12: Menu reorder + integration + full validation + +**Files:** +- Modify: `src/shared/constants/sidebarVisibility.ts` (`COMPRESSION_CONTEXT_GROUP`: Settings β†’ Combos β†’ per-engine β†’ Studio) +- Test: `tests/unit/...` sidebar order test (if one exists) + an integration test. + +- [ ] **Step 1:** Add an integration test: build a config with `engines:{rtk:{enabled:true},caveman:{enabled:true,level:"full"}}`, call `selectCompressionStrategy` + `applyCompressionAsync` on a realistic body, assert the derived stacked pipeline ran (engineBreakdown has rtk+caveman) and equals the behavior of an explicit `[rtk,caveman]` stacked config. Run β†’ (write fails first if any wiring gap). +- [ ] **Step 2:** Reorder the sidebar group; update any sidebar order test (alignment). +- [ ] **Step 3:** FULL validation: `npm run typecheck:core` (clean) Β· `npm run lint` (0 errors) Β· `node --import tsx/esm --test tests/unit/compression/*.test.ts` (green) Β· `npm run test:vitest` (green) Β· the api/integration compression tests. +- [ ] **Step 4: Commit** `... -m "feat(compression): unified panel menu order + integration coverage"` + +--- + +## Self-review notes (done while writing) +- Spec coverage: panel (T10), combos boundary (default derived T3/T9; named/active resolver T6 β€” UI for active selection is Phase 2), per-engine detail-only (T11), header (resolver is header-aware T6; parsing+wiring is Phase 3), migration (T4/T5), menu (T12). βœ“ +- Type consistency: `EngineToggle` (T2) used by `deriveDefaultPlan` (T3), `resolveCompressionPlan` (T6), normalize (T5), panel (T10). `DerivedPlan` shape consistent T3↔T6↔T7. βœ“ +- No placeholders: each task has concrete code/tests. UI tasks (T10/T11) specify exact behavior + assertions; final per-line component code is produced at execution following the existing card patterns. + +--- + +# πŸ“Œ RESUMO β€” o que ESTE plano (Fase 1) entrega + +1. **CatΓ‘logo de engines** (`engineCatalog.ts`) β€” fonte ΓΊnica de metadados (nΓ­veis, single-mode, ordem). +2. **Modelo de dados Fase A** β€” `engines` map + `activeComboId` em `CompressionConfig`, com migraΓ§Γ£o 102 + backfill (zero mudanΓ§a de comportamento). +3. **`deriveDefaultPlan`** β€” pipeline default DERIVADO dos toggles (0/1/N engines β†’ off/modo/stacked). +4. **`resolveCompressionPlan`** β€” resolvedor de precedΓͺncia (header > override por-rota > perfil ativo > default derivado > off), jΓ‘ header/active-aware. +5. **FiaΓ§Γ£o no runtime** β€” `selectCompressionStrategy`/`chatCore` usam o resolvedor. +6. **API** β€” `/api/settings/compression` carrega `engines` + `activeComboId`; rota `combos/default` vira shim read-only; `defaultMode` armazenado removido (derivado). +7. **Painel engine-grid** β€” `CompressionPanel.tsx` como fonte ΓΊnica de master + on/off + nΓ­vel, com preview do pipeline derivado. +8. **ConsolidaΓ§Γ£o** β€” remove toggles duplicados (TokenSaverCard, CompressionSettingsTab) e tira on/off+nΓ­vel das pΓ‘ginas por-engine (que ficam sΓ³ com config detalhada). +9. **Menu** β€” Settings β†’ Combos β†’ pΓ‘ginas por-engine β†’ Studio. + +--- + +# ⏳ PENDÊNCIAS β€” a fazer DEPOIS deste plano (cada uma vira seu prΓ³prio plano/PR) + +### Fase 2 β€” Perfis nomeados + seletor de ativo +- **UI de combos como perfis**: a pΓ‘gina `context/combos` lista N combos nomeados, edita pipeline **ordenado** (drag/reorder + nΓ­vel por step), e tem o seletor **"perfil ativo"** (`Default` | ``) gravando `activeComboId`. O resolvedor (Fase 1) jΓ‘ consome `activeComboId`; falta a UI + o carregamento dos combos nomeados no `selectCompressionStrategy`. Remover o "master mode selector" do `CompressionHub` (modo agora Γ© derivado). + +### Fase 3 β€” Header por-request `x-omniroute-compression` +- **Parsing + wiring do header**: ler `x-omniroute-compression` no pipeline (espelhando `x-omniroute-no-memory`, PR #4290), passar como `ctx.header` ao `resolveCompressionPlan` (que jΓ‘ trata `off`/`default`/``/`engine:`). Doc no `API_REFERENCE` + teste de fetch-capture provando precedΓͺncia por-request. + +### Itens de compressΓ£o deferidos (do ciclo de fixes, independentes deste painel) +- **B-OBSERVABILITY** (telemetria): engines no-op somem do `engineBreakdown` (nΓ£o dΓ‘ pra distinguir "rodou 0%" de "pulou"). Exige um refactor do modelo de breakdown (campo `ran`/`skipped`) que toca a UI Studio + testes que asseram `.length` β€” deferido conscientemente. +- **B-CAVEMAN-PACKS**: `de`/`fr`/`ja` sem `dedup.json`+`ultra.json` (ultra==full nessas lΓ­nguas). ConteΓΊdo linguΓ­stico β€” adicionar os packs (ou contribuiΓ§Γ£o), **sem** fallback EN que mutilaria. +- **js-tiktoken 1.0.21β†’1.0.22**: bump trivial (o range `^1.0.20` jΓ‘ permite; sΓ³ pinar o lockfile num install real). + +### DecisΓ£o operacional pendente (nΓ£o-cΓ³digo) +- **Ligar o SLM (tier ultra) em produΓ§Γ£o**: validado ao vivo (49,4% real), mas mantido **OFF** por sua escolha. Quando decidir o trade-off (custo/latΓͺncia/qualidade do pruning), ligar via o painel (apΓ³s Fase 1) ou pela escrita de config β€” comeΓ§ando conservador (β‰₯2000 tok). + +### Portes upstream (opcionais, do audit β€” fora do escopo deste painel) +- headroom *safety-rails*/BM25; filtros novos do rtk/token-savior; conformance GCF v3.1 (jΓ‘ cobrimos o `[..]:`); transformers.js 3.5.2β†’4.x (arriscado, major). diff --git a/Cluster-Decisions.md b/Cluster-Decisions.md new file mode 100644 index 0000000..321a5eb --- /dev/null +++ b/Cluster-Decisions.md @@ -0,0 +1,98 @@ +> 🌍 [View in other languages](Languages) + + +# Cluster Decisions β€” Optional Sidecar Profiles + +**Status:** proposal (awaiting @diegosouzapw review) +**Date:** 2026-06-20 +**Refs:** [#3932](https://github.com/diegosouzapw/OmniRoute/issues/3932), PR #4381 + +## TL;DR + +Two opt-in compose profiles (`memory`, `bifrost`) for the existing 8-service deployment in [`docker-compose.yml`](../../docker-compose.yml). Default-up behaviour is **unchanged**: 3 Γ— `omniroute` replicas + Caddy + Redis + CliproxyAPI. The two new profiles add Qdrant and Bifrost as optional sidecars, gated by `docker compose --profile up`. **No existing service is removed or replaced.** + +## Why this is conservative + +OmniRoute's existing deployment shape is already lean and proven: + +- **`redis:7-alpine`** handles the rate-limit/cache workload at production scale. +- **SQLite + sqlite-vec + FTS5** cover local memory + vector + text-search (see [`src/lib/memory/vectorStore.ts:108`](../../src/lib/memory/vectorStore.ts)). +- **Caddy** is already the LB + TLS terminator ([`docker-compose.yml`](../../docker-compose.yml)). +- **Bifrost** is already integrated as the Tier-1 router in [`src/app/api/v1/relay/chat/completions/bifrost/route.ts`](../../src/app/api/v1/relay/chat/completions/bifrost/route.ts) (sidecar proxy with kill switch via `BIFROST_ENABLED` env var β€” set `=0` to bypass the sidecar and fall through to the TS path). + +The two profiles here are **scale-out options for deployments that hit the SQLite ceiling** β€” not migrations. Both are default-off. + +## The two profiles + +### `memory` β€” Qdrant Vector Memory Sidecar + +**When to flip on:** + +- >1M embeddings per deployment (sqlite-vec starts to slow at scale). +- Multi-replica deployment that needs shared vector state across `omniroute-1/2/3`. +- You already have an external Qdrant cluster (Qdrant Cloud, on-prem). + +**What it adds:** + +| Service | Image | Ports | Notes | +| --------------- | ------------------------ | ----------- | ---------------------------------------------------------------------- | +| `qdrant` | `qdrant/qdrant:v1.12.4` | `6333` HTTP | HNSW index; persistent volume `omniroute_qdrant_data` | + +**Activation:** flip `qdrantEnabled = true` in the Settings UI **or** set `QDRANT_HOST=qdrant` env. See [`src/lib/memory/qdrant.ts:60`](../../src/lib/memory/qdrant.ts) for the precedence rules (settings table β†’ env var β†’ default). + +**Env vars:** `QDRANT_HOST`, `QDRANT_PORT`, `QDRANT_API_KEY`, `QDRANT_COLLECTION`, `QDRANT_VECTOR_SIZE`, `QDRANT_HNSW_EF_CONSTRUCT` (see `.env.example` lines 1672-1683). + +### `bifrost` β€” Bifrost Tier-1 Router Sidecar + +**When to flip on:** + +- You run β‰₯3 `omniroute` replicas and want provider rotation centralised in a single Go process. +- You want a single audit/logging surface for upstream-provider requests across all replicas. +- You want horizontal scaling of the Tier-1 routing layer independent of the OmniRoute replicas. + +**What it adds:** + +| Service | Image | Ports | Notes | +| --------- | ------------------------------------------- | ---------- | -------------------------------------------------------------------------- | +| `bifrost` | `ghcr.io/maximhq/bifrost:1.5.21` | `8080` | Go-based Tier-1 router; persistent logs volume `omniroute_bifrost_logs` | + +**Activation:** set `BIFROST_BASE_URL=http://bifrost:8080` in `.env.example`. The existing sidecar proxy route at [`src/app/api/v1/relay/chat/completions/bifrost/route.ts`](../../src/app/api/v1/relay/chat/completions/bifrost/route.ts) (added in PR #4381) will pick this up automatically. + +**Env vars:** `BIFROST_BASE_URL`, `BIFROST_API_KEY`, `BIFROST_STREAMING_ENABLED`, `BIFROST_TIMEOUT_MS` (see `.env.example` lines 1685-1695). + +## What this PR explicitly does NOT do + +The original issue thread floated a larger cluster rewrite. After auditing the actual workload shape, the following are **rejected** for the reasons given: + +| Component | Verdict | Reason | +| --------------- | --------- | ------------------------------------------------------------------------------------------------------------ | +| **Dragonfly** | **DROP** | `redis:7-alpine` is already fine for the rate-limit workload at production scale; no ceiling to break. | +| **NATS** | **DROP** | Each `omniroute` replica is a single Node.js process; no multi-process pub/sub workload exists. | +| **PostgreSQL** | **DROP** | SQLite + sqlite-vec + FTS5 cover all 3 use cases; 97 migrations + Electron packaging block migration. | +| **Neo4j** | **DROP** | Routing is a 5-table join; recursive CTE on SQLite is sufficient. | +| **MinIO** | **DROP** | No multi-MB blob workload; images/audio are passthrough proxies. | +| **pgvector / pg_ai / pg_textsearch** | **DROP** | Same SQLite-ceiling reason as PostgreSQL; pgvector ecosystem fragmented. | +| **HAProxy / Envoy** | **DROP** | Caddy already does LB + TLS; both were explicitly rejected as Tier-1 routers (see `AGENTS.md`). | + +If a future use case proves out one of these, this doc is the place to amend. + +## 4-week rollout (if approved) + +1. **Wk 1** β€” Land this PR + verification of opt-in profiles with a 3-replica compose stack. +2. **Wk 2** β€” Bifrost full activation for OpenAI/Claude/Gemini/Ollama (4 of 14+ providers) using the sidecar proxy route at [`src/app/api/v1/relay/chat/completions/bifrost/route.ts`](../../src/app/api/v1/relay/chat/completions/bifrost/route.ts) (gated by `BIFROST_ENABLED`, kill-switchable at runtime). +3. **Wk 3** β€” Qdrant memory profile enabled in a single test deployment; measure latency delta vs sqlite-vec. +4. **Wk 4** β€” Observability healthchecks (`docker compose ps` exit codes + `wget` smoke tests); 71-pillar refresh per ADR-041. + +## Files changed in this PR + +| File | Change | +| ----------------------------------------------------------- | --------------------------------------------------------------------- | +| `docker-compose.yml` | +30 lines: `memory` profile (Qdrant), `bifrost` profile (Bifrost), persistent volumes, healthchecks. | +| `.env.example` | +24 lines: `QDRANT_*` (6 vars), `BIFROST_*` (4 vars). | +| `docs/reference/ENVIRONMENT.md` | +6 rows in section 25 for the `QDRANT_*` env vars. | +| `src/lib/memory/qdrant.ts` | +33 lines: env-var fallback chain (settings β†’ env β†’ default) for `QDRANT_HOST`/`QDRANT_PORT`/`QDRANT_API_KEY`/`QDRANT_COLLECTION`/`QDRANT_VECTOR_SIZE`/`QDRANT_HNSW_EF_CONSTRUCT`/`QDRANT_EMBEDDING_MODEL`. | +| `src/lib/memory/__tests__/qdrant-wiring.test.ts` | +88 lines: 9 new test cases pinning the env-var fallback precedence. | +| `docs/architecture/cluster-decisions.md` (this file) | NEW β€” decision record for the opt-in profiles. | +| `AGENTS.md` | +1 line: pointer to this doc in the reference documentation table. | + +**Net touched code:** 4 production files (`docker-compose.yml`, `qdrant.ts`, `.env.example`, `ENVIRONMENT.md`), 1 test file (`qdrant-wiring.test.ts`), 2 doc files (`cluster-decisions.md`, `AGENTS.md`). diff --git a/Database-Guide.md b/Database-Guide.md new file mode 100644 index 0000000..e3b97d0 --- /dev/null +++ b/Database-Guide.md @@ -0,0 +1,621 @@ +> 🌍 [View in other languages](Languages) + + +# Database Schema & Operations Guide + +> **TL;DR**: OmniRoute uses **SQLite with WAL journaling** as its primary store, with **AES-256-GCM** encryption at rest for sensitive fields. This guide covers the schema, migrations, backup/recovery, and operational runbooks. + +**Sources:** +- `src/lib/db/core.ts` β€” singleton + SCHEMA_SQL (17 base tables) +- `src/lib/db/migrationRunner.ts` β€” versioned migrations +- `src/lib/db/migrations/` β€” 94 versioned SQL files +- `src/lib/db/encryption.ts` β€” encryption helpers +- `src/lib/db/backup.ts` β€” backup export/import +- `src/lib/db/healthCheck.ts` β€” health diagnostics + +--- + +## Why SQLite? + +OmniRoute chose SQLite over PostgreSQL/MySQL for several reasons: + +| Factor | SQLite | PostgreSQL | +|--------|--------|-----------| +| **Deployment** | Embedded β€” no separate server | Requires server setup | +| **Encryption** | Application-layer (AES-256-GCM) | Built-in TDE | +| **Performance** | Faster for small/medium workloads | Better for huge concurrent writes | +| **Concurrency** | WAL mode allows concurrent reads | Full MVCC | +| **Backup** | Single-file copy | `pg_dump` or filesystem snapshot | +| **Use case** | Per-user install, embedded | Multi-tenant SaaS | + +For **single-user, single-instance** deployments (the primary OmniRoute use case), SQLite is simpler and faster. + +### WAL Journaling + +`core.ts` opens the database with **WAL (Write-Ahead Logging) mode**: + +```ts +// src/lib/db/core.ts +db.pragma("journal_mode = WAL"); +db.pragma("busy_timeout = 5000"); +db.pragma("synchronous = NORMAL"); +db.pragma("cache_size = -2048"); +``` + +WAL allows **concurrent reads** during writes β€” important for the dashboard, which queries while requests are being recorded. + +--- + +## Database Location + +The SQLite file is stored at: + +| OS | Path | +|----|------| +| Linux | `~/.omniroute/storage.sqlite` | +| macOS | `~/.omniroute/storage.sqlite` | +| Windows | `%USERPROFILE%\.omniroute\storage.sqlite` | +| Docker | `/app/data/storage.sqlite` (configurable via `DATA_DIR`) | + +Companion files: + +- `storage.sqlite-wal` β€” write-ahead log +- `storage.sqlite-shm` β€” shared memory file +- `call_logs/` β€” request payload artifacts (if enabled) + +**Override the location:** + +```bash +DATA_DIR=/custom/path omniroute +``` + +--- + +## Domain Module Architecture + +OmniRoute's database has **76 domain modules** in `src/lib/db/`. Each module: + +- Owns one or more specific tables +- Exports typed CRUD functions +- Never touches another module's tables +- Uses `getDbInstance()` from `core.ts` to access the DB + +### The 76 DB Modules + +OmniRoute has **76 module files** in `src/lib/db/`. Below is a sampling of core modules; see the directory listing for the complete list: + +| Module | Tables | Responsibility | +|--------|--------|----------------| +| `providers.ts` | `provider_connections` | OAuth/API key provider registration and credentials | +| `models.ts` | `key_value` (model data) | Model definitions, capabilities, pricing | +| `combos.ts` | `combos` | Combo routing configs and ordering | +| `apiKeys.ts` | `api_keys` | API key lifecycle, scopes, quota tracking | +| `settings.ts` | `key_value`, `api_keys`, `combos` | System configuration and shared KV store | +| `backup.ts` | β€” | Backup export/import operations | +| `proxies.ts` | `proxy_registry`, `proxy_assignments`, `provider_connections` | Proxy configs and routing rules | +| `prompts.ts` | `prompt_templates` | Reusable prompt templates, versioning | +| `webhooks.ts` | `webhooks` | Event-driven webhook subscriptions and logs | +| `detailedLogs.ts` | `request_detail_logs` | Per-request audit logging (optional, high volume) | +| `domainState.ts` | `domain_*` (5 tables) | Domain budgets, circuit breakers, lockouts, fallback chains, cost history | +| `registeredKeys.ts` | `registered_keys`, `account_key_limits`, `provider_key_limits` | Whitelisted API keys for MCP/A2A | +| `quotaSnapshots.ts` | `quota_snapshots` | Historical quota usage | +| `modelComboMappings.ts` | `model_combo_mappings` | Map models to combo defaults | +| `cliToolState.ts` | `cli_tool_state` | CLI-specific persistent state | +| `encryption.ts` | β€” | Helpers for encrypting/decrypting fields | +| `readCache.ts` | β€” | In-memory cache for read-heavy ops | +| `secrets.ts` | `key_value` (encrypted entries) | Encrypted secret storage | +| `stateReset.ts` | β€” | Wipe/reset DB state for testing | +| `contextHandoffs.ts` | `context_handoffs` | Session context for agent handoff | +| `usage*.ts` | `usage_history`, `call_logs`, `proxy_logs` | Usage tracking | +| `compression*.ts` | `compression_settings`, `compression_combos` | Compression config | + +### Module Boundaries + +A core architectural rule: **modules don't access each other's tables directly**. To work with another module's data, import the function from that module. + +```ts +// ❌ WRONG: direct SQL from another module +db.prepare("SELECT * FROM provider_connections").all(); + +// βœ… RIGHT: use the providers module function +import { listProviders } from "@/lib/db/providers"; +const providers = await listProviders(); +``` + +This rule is enforced by code review β€” there's no static check, but violations are flagged. + +--- + +## Base Schema (17 tables) + +`core.ts` defines the 17 base tables in `SCHEMA_SQL`. These are created by migration `001_initial_schema.sql` and form the core schema. + +### Core Tables (created in initial migration) + +| Table | Purpose | Key columns | +|-------|---------|-------------| +| `provider_connections` | Provider credentials (encrypted) | `id`, `provider`, `auth_type`, `api_key`, `is_active` | +| `provider_nodes` | Provider node routing info | `id`, `type`, `name`, `base_url`, `created_at` | +| `key_value` | General KV store | `namespace`, `key`, `value` | +| `combos` | Routing combo definitions | `id`, `name`, `data`, `sort_order` | +| `api_keys` | API keys for the gateway | `id`, `name`, `key`, `machine_id`, `allowed_models` | +| `db_meta` | Database metadata | `key`, `value` | +| `usage_history` | Request usage records | `id`, `provider`, `model`, `tokens_input`, `tokens_output`, `timestamp` | +| `call_logs` | Request payloads & responses | `id`, `timestamp`, `status`, `model`, `provider`, `latency_ms` | +| `proxy_logs` | Proxy request logs | `id`, `timestamp`, `proxy_type`, `status`, `provider` | +| `domain_fallback_chains` | Model-to-provider chains | `model`, `chain` | +| `domain_budgets` | Per-domain spend budgets | `api_key_id`, `daily_limit_usd`, `warning_threshold`, `reset_interval` | +| `domain_budget_reset_logs` | Budget reset history | `id`, `api_key_id`, `reset_interval`, `previous_spend`, `reset_at` | +| `domain_cost_history` | Per-domain cost tracking | `id`, `api_key_id`, `cost`, `timestamp` | +| `domain_lockout_state` | Domain rate-limit state | `identifier`, `attempts`, `locked_until` | +| `domain_circuit_breakers` | Circuit breaker state per domain | `name`, `state`, `failure_count`, `last_failure_time` | +| `semantic_cache` | LLM response cache | `id`, `signature`, `model`, `prompt_hash`, `response` | +| `quota_snapshots` | Historical quota snapshots | `id`, `provider`, `connection_id`, `window_key`, `remaining_percentage` | + +### Additional Tables (added by later migrations) + +Subsequent migrations add tables such as: +- `cli_tool_state` (migration 011) β€” CLI tool state +- `mcp_*` tables β€” MCP server audit +- `a2a_*` tables β€” A2A task state +- `usage_*` tables β€” usage tracking +- `plugin_*` tables β€” plugin system +- `skill_executions` β€” skill execution history +- `memory_*` tables β€” memory system +- `compression_*` tables β€” compression system +- `webhook_*` tables β€” webhook delivery log +- `acp_*` tables β€” Agent Client Protocol +- `oneproxy_*` tables β€” 1proxy marketplace +- `proxy_assignments` β€” proxy scope bindings +- `detailed_call_artifacts` β€” call log artifacts metadata +- `quota_alert_history` β€” quota alert audit +- `command_code_auth_sessions` β€” Command Code OAuth sessions + +The full list of ~30+ tables is in `src/lib/db/migrations/`. + +--- + +## Migrations + +OmniRoute uses **versioned, idempotent migrations** in `src/lib/db/migrations/`. Each migration is a single SQL file named `NNN_description.sql`. + +### Migration Naming + +``` +001_initial_schema.sql +002_mcp_a2a_tables.sql +003_provider_node_custom_paths.sql +... +021_combo_call_log_targets.sql +``` + +### How Migrations Run + +At startup, `migrationRunner.ts`: + +1. Creates `_omniroute_migrations` table if not exists +2. Queries for already-applied migrations +3. Applies any new migrations in order, each in a transaction +4. Records each applied migration with timestamp + +```ts +// src/lib/db/migrationRunner.ts (simplified) +export async function runMigrations(db: SqliteDatabase, migrationsDir: string) { + const applied = getAppliedMigrations(db); + const available = readMigrationFiles(migrationsDir); + + for (const migration of available) { + if (applied.includes(migration.id)) continue; + db.transaction(() => { + db.exec(migration.sql); + recordAppliedMigration(db, migration.id); + })(); + } +} +``` + +### Idempotency + +Migrations must be **idempotent** β€” running them twice should be a no-op: + +```sql +-- 004_proxy_registry.sql +CREATE TABLE IF NOT EXISTS proxy_registry ( + id TEXT PRIMARY KEY, + host TEXT NOT NULL, + port INTEGER NOT NULL, + ... +); +``` + +Use `IF NOT EXISTS`, `IF EXISTS`, and `OR IGNORE` / `OR REPLACE` clauses liberally. + +### Adding a New Migration + +1. **Identify the next number**: `ls src/lib/db/migrations/ | tail -1` +2. **Create the file**: `NNN_my_change.sql` +3. **Use safe DDL**: `CREATE TABLE IF NOT EXISTS`, `ALTER TABLE ... ADD COLUMN` +4. **Backfill data carefully**: use `UPDATE ... WHERE ...` to handle existing rows +5. **Test on a copy**: never run untested migrations on production + +Example: + +```sql +-- 022_add_combo_priority.sql +ALTER TABLE combos ADD COLUMN priority INTEGER DEFAULT 100; +UPDATE combos SET priority = 100 WHERE priority IS NULL; +CREATE INDEX IF NOT EXISTS idx_combos_priority ON combos(priority); +``` + +> **Backwards-incompatible changes** (e.g., dropping columns) are tricky. OmniRoute does NOT support downgrade β€” once a migration is applied, the schema change is permanent. Plan accordingly. + +--- + +## Encryption at Rest + +Sensitive fields (API keys, OAuth tokens, connection strings) are encrypted at rest using **AES-256-GCM**. + +### How It Works + +```ts +// src/lib/db/encryption.ts (simplified) +const key = deriveKeyFromPassphrase(passphrase, salt); +const iv = randomBytes(12); +const cipher = createCipheriv("aes-256-gcm", key, iv); +const encrypted = Buffer.concat([cipher.update(plaintext), cipher.final()]); +const authTag = cipher.getAuthTag(); +return { encrypted, iv, authTag }; +``` + +### Where It's Used + +- `provider_connections.api_key` β€” encrypted at application level +- `provider_connections.access_token`, `refresh_token`, `id_token` β€” encrypted at application level +- `key_value` entries with `namespace = "secrets"` β€” encrypted at application level +- `proxy_registry.auth` β€” encrypted at application level (if present) + +### Encryption Key + +The encryption key is derived from a **passphrase** (set via `STORAGE_ENCRYPTION_KEY` env var) and a **salt** (stored in the DB). Both are required to decrypt data. + +```bash +# Generate a secure passphrase +openssl rand -hex 32 + +# Set in .env +STORAGE_ENCRYPTION_KEY= +``` + +> **Critical**: Losing the encryption key means losing access to all encrypted data. **Back up the key separately from the database**. + +### What's NOT Encrypted + +For performance reasons, the following are stored in plaintext: + +- Provider display names +- Model definitions (already public) +- Routing rules +- Usage records (no PII) + +--- + +## Encryption Caveats (v3.8.16+) + +OmniRoute uses **`migrateLegacyEncryptedString()`** to handle two encryption schemes transparently: + +- **Legacy** (pre-v3.5.0): XOR-based "encryption" (not real crypto) +- **Current**: AES-256-GCM with proper IV and auth tag + +The migration helper detects the legacy format and re-encrypts with the new scheme on first read. This means you can upgrade an old database without losing credentials. + +--- + +## Read Cache + +For frequently-read data (models, providers, settings), `readCache.ts` provides an **in-memory cache**: + +```ts +// Cached at startup, invalidated on write +const providers = await getCachedProviders(); // Fast, in-memory +const fresh = await listProviders(); // Slow, hits DB +``` + +| Cached entity | Cache key | TTL | +|---------------|-----------|-----| +| `models` | `models:v1` | Until write | +| `provider_connections` | `providers:v1` | Until write | +| `settings` | `settings:v1` | Until write | +| `combos` | `combos:v1` | Until write | + +Cache is invalidated on every write to the corresponding table. + +--- + +## Backup and Recovery + +### Manual Backup + +```bash +# Use the CLI to create a local backup +omniroute backup create --name pre-migration + +# Or via the API +curl -X PUT http://localhost:20128/api/db-backups \ + -H "Authorization: Bearer $MANAGEMENT_KEY" \ + -H "Content-Type: application/json" \ + -d '{"name": "pre-migration"}' +``` + +The backup file includes: + +- All DB tables (serialized to JSON) +- Call log artifacts (base64-encoded, optional) +- Settings + secrets (encrypted) +- Plugin configuration + +### Restore + +```bash +# Via CLI +omniroute restore pre-migration + +# Via API +curl -X POST http://localhost:20128/api/db-backups/restore \ + -H "Authorization: Bearer $MANAGEMENT_KEY" \ + -H "Content-Type: application/json" \ + -d '{"name": "pre-migration"}' +``` + +> **Warning**: Restore overwrites the entire DB. Stop all clients first. + +### Automated Backups + +```bash +# Enable automated daily backups via CLI +omniroute backup auto enable --cron "0 2 * * *" --retention 7 +``` + +### SQLite Hot Backup + +For zero-downtime backup of a live DB: + +```bash +sqlite3 ~/.omniroute/storage.sqlite ".backup /backups/omniroute-hot.db" +``` + +This uses SQLite's online backup API β€” safe to run while OmniRoute is running. + +--- + +## Performance Tuning + +### WAL Mode + +WAL is enabled by default. For high-write workloads, consider: + +```sql +PRAGMA wal_autocheckpoint = 1000; -- Checkpoint every 1000 pages +PRAGMA journal_size_limit = 67108864; -- 64MB WAL cap +``` + +### Indexes + +Key indexes for performance (auto-created by migrations): + +- `idx_models_provider` β€” model lookups by provider +- `idx_combo_targets_combo_id` β€” combo target expansion +- `idx_usage_history_api_key_timestamp` β€” usage analytics +- `idx_quota_snapshots_api_key_window` β€” quota tracking +- `idx_call_logs_timestamp` β€” call log queries + +To add a new index, create a migration: + +```sql +-- 023_add_my_index.sql +CREATE INDEX IF NOT EXISTS idx_my_table_my_column ON my_table(my_column); +``` + +### Memory-Mapped I/O + +For very large databases (>10GB), memory mapping can be adjusted via SQLite pragma: + +```sql +-- Set via SQLite pragma (adjust in core.ts or runtime) +PRAGMA mmap_size = 268435456; -- 256MB +``` + +### Compaction + +Long-running OmniRoute instances benefit from occasional `VACUUM`: + +```bash +sqlite3 ~/.omniroute/storage.sqlite "VACUUM;" +``` + +Run monthly during low-traffic windows. (WAL mode reduces the need, but doesn't eliminate it.) + +--- + +## Health Check + +`src/lib/db/healthCheck.ts` provides **DB-level health diagnostics**: + +```bash +GET /api/db/health + +Returns: + +```json +{ + "status": "healthy", + "checks": { + "writable": { "status": "pass" }, + "integrity": { "status": "pass", "result": "ok" }, + "foreign_keys": { "status": "pass", "violations": 0 }, + "orphaned_artifacts": { "status": "warn", "count": 12 }, + "table_sizes": { + "usage_history": { "rows": 12345, "size_mb": 12.3 }, + "call_logs": { "rows": 567, "size_mb": 2.1 } + } + } +} +``` + +Run `PRAGMA integrity_check` to detect corruption: + +```bash +sqlite3 ~/.omniroute/storage.sqlite "PRAGMA integrity_check;" +# Should print: ok +``` + +If it returns anything other than `ok`, **stop using the database immediately** and restore from backup. + +--- + +## Disaster Recovery + +### Scenario 1: WAL File Lost + +The `-wal` file is missing but `-shm` and main DB are intact: + +```bash +# Recovers automatically on next open +omniroute +``` + +If SQLite can't auto-recover: + +```bash +sqlite3 ~/.omniroute/storage.sqlite ".recover" > recovered.sql +sqlite3 recovered.db < recovered.sql +mv recovered.db ~/.omniroute/storage.sqlite +``` + +### Scenario 2: Main DB File Corrupted + +Restore from backup: + +```bash +omniroute sync pull --merge # or: omniroute backup restore +``` + +### Scenario 3: Encryption Key Lost + +**No recovery possible** without the key. The encrypted fields are unreadable. Re-add all providers manually with new credentials. + +> **Mitigation**: Always back up the encryption key separately, ideally in a password manager or KMS. + +### Scenario 4: Disk Full + +SQLite will return `SQLITE_FULL` errors. Free disk space, then: + +```bash +# Checkpoint WAL to free up space +sqlite3 ~/.omniroute/storage.sqlite "PRAGMA wal_checkpoint(TRUNCATE);" +``` + +--- + +## Common Operations + +### Inspect a Table + +```bash +sqlite3 ~/.omniroute/storage.sqlite "SELECT * FROM api_keys LIMIT 5;" +``` + +### Count Rows in All Tables + +```bash +sqlite3 ~/.omniroute/storage.sqlite < 🌍 [View in other languages](Languages) + + +# Monitoring & Observability Guide + +> **TL;DR**: OmniRoute ships with built-in health monitoring, provider autopilot, quota tracking, and observability hooks. This guide covers the dashboard, alerts, and troubleshooting. + +**Sources:** +- `src/lib/monitoring/observability.ts` β€” observability snapshot +- `src/lib/monitoring/comboHealthAutopilot.ts` β€” combo health autopilot +- `src/lib/monitoring/providerHealthAutopilot.ts` β€” provider autopilot +- `src/lib/monitoring/providerHealthMatrix.ts` β€” provider health matrix +- `src/lib/localHealthCheck.ts` β€” local health check +- `src/lib/tokenHealthCheck.ts` β€” token refresh health +- `src/lib/proxyHealth.ts` β€” proxy health cache (covered in PROXY_GUIDE.md) + +--- + +## Overview + +OmniRoute has **3 layers of monitoring**: + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Layer 1: System Health (server-level) β”‚ +β”‚ β”œβ”€ localHealthCheck.ts β€” DB, ports, native deps β”‚ +β”‚ β”œβ”€ db/healthCheck.ts β€” integrity, FK, orphaned artifacts β”‚ +β”‚ └─ Dashboard: /dashboard/health β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚ Layer 2: Provider Health (per-provider resilience) β”‚ +β”‚ β”œβ”€ providerHealthAutopilot.ts β€” circuit breaker, cooldowns β”‚ +β”‚ β”œβ”€ providerHealthMatrix.ts β€” health scores by provider/model β”‚ +β”‚ └─ Dashboard: /dashboard/providers β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚ Layer 3: Live Observability (runtime snapshots) β”‚ +β”‚ β”œβ”€ observability.ts β€” circuit breakers, sessions, quota β”‚ +β”‚ β”œβ”€ tokenHealthCheck.ts β€” OAuth token refresh health β”‚ +β”‚ └─ MCP tools: omniroute_get_health, omniroute_get_session_snapshot β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +--- + +## Dashboard Pages + +### `/dashboard/health` (System Health) + +The top-level health dashboard shows: + +| Section | What it shows | +|---------|---------------| +| **Server status** | Uptime, version, port, active connections | +| **Database** | Connection, integrity, WAL size, recent migrations | +| **Provider summary** | Active count, healthy count, breaker open count | +| **Quota monitors** | Active sessions, alerting, exhausted | +| **Recent errors** | Last 10 errors with stack traces | +| **Resource usage** | Memory, CPU, heap pressure indicator | + +### `/dashboard/providers` (Provider Health) + +Per-provider dashboard: + +| Column | Description | +|--------|-------------| +| Provider | Provider ID + display name | +| Health | Green/yellow/red status | +| Circuit | Open/closed/half-open state | +| Connections | Count of connections, last refresh | +| Models | Available models, health per model | +| Cost | Today's cost, 7-day trend | +| Errors | Last 24h error count, top error class | + +Click a provider to see: +- Recent requests with latency breakdown +- Per-connection health scores +- Per-model lockouts +- Autopilot recommendations + +### `/dashboard/quota` (Quota Tracking) + +For each API key: + +- Current usage vs limit (progress bar) +- Quota trend (30-day chart) +- Next reset time +- Alert history + +### `/dashboard/combos` (Combo Health) + +Per-combo: + +- Strategy + targets +- Health per target +- Recent fallback events +- Success rate (24h, 7d, 30d) + +--- + +## Health Check API + +> **Note:** Only `GET /api/monitoring/health` is exposed as a REST endpoint. All other monitoring data (provider health, autopilot issues, quota monitors, token health, latency) is accessed via the **MCP tool** `observability_snapshot` or the **dashboard** pages β€” there are no dedicated REST routes for these. + +### System Health + +```bash +GET /api/monitoring/health +``` + +Response: + +```json +{ + "status": "healthy", + "version": "3.8.16", + "uptime": 123456, + "checks": { + "database": { "status": "pass", "latency_ms": 2 }, + "writeable": { "status": "pass" }, + "integrity": { "status": "pass", "result": "ok" }, + "foreign_keys": { "status": "pass", "violations": 0 }, + "heap_pressure": { "status": "pass", "usage_mb": 142, "threshold_mb": 512 }, + "active_sessions": 12, + "providers": { + "total": 7, + "healthy": 6, + "degraded": 1, + "down": 0 + } + } +} +``` + +### Provider Health + +> **No REST endpoint.** Provider health data is available via the MCP tool `observability_snapshot` or the dashboard `/dashboard/providers` page. + +### Provider Detail + +> **No REST endpoint.** Per-provider detail is available via the dashboard `/dashboard/providers` page. + +--- + +## Provider Health Autopilot + +The `providerHealthAutopilot.ts` module is a **self-healing system** that: + +1. Detects provider issues (circuit open, cooldowns, lockouts, quota warnings) +2. Generates **recommended actions** to resolve them +3. Optionally **auto-executes** low-risk actions + +### Issue Types Detected + +| Issue kind | Severity | Example condition | +|------------|----------|-------------------| +| `provider_circuit_open` | critical | Circuit breaker open after 5 failures | +| `provider_circuit_half_open` | warning | Circuit testing recovery | +| `connection_cooldown` | warning | Connection in cooldown after 429 | +| `stale_connection_error` | warning | Last refresh failed 30+ minutes ago | +| `terminal_connection_error` | critical | OAuth revoked, key invalid | +| `inactive_connection` | info | Connection disabled in settings | +| `model_lockout` | warning | Specific model in quarantine | +| `quota_monitor_warning` | warning | Quota at 80%+ usage | + +### Action Types Generated + +| Action | Risk | Description | +|--------|------|-------------| +| `clear_provider_breaker` | medium | Reset the circuit breaker to closed | +| `clear_connection_cooldown` | low | Remove cooldown from a connection | +| `clear_stale_connection_error` | low | Clear stale error flag | +| `clear_model_lockout` | low | Re-enable a quarantined model | +| `reactivate_connection` | medium | Re-enable a deactivated connection | +| `deactivate_connection` | high | Disable a problematic connection | + +### API + +> **No REST endpoint.** Autopilot issues are available via the MCP tool `observability_snapshot` or the dashboard. The autopilot runs internally; its behavior is configured via the settings DB (per-connection `autopilotMode` field), not environment variables β€” `grep -rn` for an autopilot-mode env var returns zero hits. + +### Autopilot Mode + +The autopilot operates in **manual mode** by default β€” it detects issues and generates recommended actions, but does not auto-apply them. Actions can be applied via the dashboard. + +--- + +## Combo Health Autopilot + +`comboHealthAutopilot.ts` is the **combo-specific** equivalent of the provider autopilot. It: + +- Detects unhealthy combos +- Recommends target reordering +- Suggests disabling broken targets +- Auto-removes dead targets after N failures + +### Combo Issue Examples + +``` +Combo "always-on" (priority strategy) +β”œβ”€ Target 1: openai/gpt-5 (healthy) +β”œβ”€ Target 2: anthropic/claude-opus-4-6 (⚠️ model lockout until 14:00) +└─ Target 3: kiro/claude-sonnet-4-5 (healthy) + +Recommended action: Reorder β€” move kiro above anthropic until lockout expires +``` + +--- + +## Quota Monitors + +`observability.ts` exposes **per-session quota monitors** for subscription providers (Claude Code, Codex, GitHub Copilot): + +```ts +interface QuotaMonitorSnapshot { + sessionId: string; + provider: string; + accountId: string; + status: "starting" | "idle" | "healthy" | "warning" | "exhausted" | "error"; + lastQuotaPercent: number | null; // 0-100 + lastQuotaUsed: number | null; + lastQuotaTotal: number | null; + lastResetAt: string | null; + nextPollAt: string | null; + totalPolls: number; + totalAlerts: number; + consecutiveFailures: number; +} +``` + +### Status Meanings + +| Status | When | UI action | +|--------|------|-----------| +| `starting` | Initial poll in progress | Spinner | +| `idle` | No recent activity | Hidden from dashboard | +| `healthy` | Quota > 50% remaining | Green dot | +| `warning` | Quota < 50% remaining | Yellow alert | +| `exhausted` | Quota = 0% | Red block, route to next provider | +| `error` | Polling failed | Red dot, retry soon | + +### API + +> **No REST endpoint.** Quota monitor data is available via the MCP tool `observability_snapshot` or the dashboard. + +--- + +## Observability Snapshot + +The MCP tool `observability_snapshot` returns a **complete system snapshot** for AI agents: + +```json +{ + "circuitBreakers": [ + { + "name": "openai", + "state": "closed", + "failureCount": 0, + "lastFailureTime": null, + "retryAfterMs": null + } + ], + "sessions": [ + { + "sessionId": "sess-123", + "createdAt": 1234567890, + "lastActive": 1234567999, + "requestCount": 42, + "connectionId": "conn-456", + "ageMs": 109 + } + ], + "quotaMonitors": { /* see above */ }, + "uptime": 12345, + "version": "3.8.16" +} +``` + +Agents use this to make **routing decisions** β€” for example, "if openai's circuit is open, route to anthropic first". + +--- + +## Token Health Check + +OAuth providers (Claude Code, GitHub Copilot, Cursor) need **periodic token refresh**. `src/lib/tokenHealthCheck.ts` runs a background scheduler: + +- **Sweep tick**: every 60 seconds (sweep in `TICK_MS = 60 * 1000` at `src/lib/tokenHealthCheck.ts:30`) +- **Per-connection health check interval**: default 60 minutes (`DEFAULT_HEALTH_CHECK_INTERVAL_MIN = 60`); configurable via the settings DB +- **Pre-emptive refresh on 401**: handled by the per-connection interceptor + +### Token Health Status + +```ts +interface TokenHealth { + connectionId: string; + provider: string; + status: "valid" | "expiring_soon" | "expired" | "refresh_failed"; + expiresAt: string; + lastRefresh: string; + nextRefresh: string; + consecutiveFailures: number; +} +``` + +### Configuration + +Token health check configuration is handled internally by `tokenHealthCheck.ts`. + +### Token Health + +> **No REST endpoint.** Token health data is available via the dashboard or the MCP tool `observability_snapshot`. + +--- + +## Alerting + +### Built-in Channels + +OmniRoute supports **3 alert channels**: + +| Channel | Setup | Use case | +|---------|-------|----------| +| Dashboard banner | Always on | In-app notifications | +| Webhook | Configure URL | Slack, Discord, PagerDuty | +| Log | Default | For external log aggregation | + +### Webhook Configuration + +> **Note:** Webhook alerting configuration is handled via the dashboard Settings page. See the Settings UI for webhook URL, event filtering, and payload customization. + +### Alert Types + +| Alert | When | Default severity | +|-------|------|------------------| +| `provider_circuit_open` | Circuit opens | critical | +| `provider_circuit_half_open` | Circuit testing recovery | info | +| `quota_warning` | Quota at 80%+ | warning | +| `quota_exhausted` | Quota at 100% | critical | +| `token_refresh_failed` | 3+ consecutive refresh failures | warning | +| `token_expired` | Token past expiry | critical | +| `combo_target_unhealthy` | Combo target in cooldown for 1h+ | warning | +| `db_integrity_warning` | FK violations > 0 | warning | +| `heap_pressure` | Heap usage > 80% of threshold | warning | + +--- + +## Performance Metrics + +### Tracked Metrics + +| Metric | Type | Source | +|--------|------|--------| +| `request_count` | counter | `services/usage.ts` | +| `request_latency_ms` | histogram | `services/usage.ts` | +| `tokens_consumed` | counter | `services/usage.ts` | +| `cost_usd` | counter | `services/usage.ts` | +| `provider_errors` | counter | `services/errorClassifier.ts` | +| `circuit_state_changes` | counter | `services/resilience.ts` | +| `cache_hits` | counter | `services/signatureCache.ts` | +| `compression_savings` | histogram | `services/compression/stats.ts` | +| `quota_used` | gauge | `services/quotaMonitor.ts` | +| `memory_used_mb` | gauge | `observability.ts` | + +### Latency Percentiles (p50/p95/p99) + +> **No REST endpoint.** Latency percentile data is available via the dashboard `/dashboard/health` page. Prometheus/OpenTelemetry export is planned for v3.9. + +### Prometheus / OpenTelemetry Export (Phase 2) + +Planned for v3.9: native export to Prometheus, OpenTelemetry, Datadog. + +For now, scrape `/api/monitoring/health` with any HTTP-based monitoring system (Prometheus blackbox exporter, Datadog HTTP check, etc.). + +--- + +## Alerting Recipes + +### Slack +> **Note:** Webhook alerting is configured through the dashboard Settings page β€” there are no dedicated webhook env vars (`grep -rn` returns zero hits). See the Settings UI for webhook URL, event filtering, and payload customization. +### Discord +> Webhook alerting uses the same Settings UI flow as Slack. Discord accepts the same JSON payload shape. +### PagerDuty +> Webhook alerting uses the same Settings UI flow. PagerDuty Events API v2 routing keys are configured in the Settings UI. +### Custom Webhook (JSON) +> Any HTTP endpoint that accepts POST with JSON body will work. Configure the URL in the Settings UI. + +--- + +## Dashboard Configuration + +### Customize the Health Dashboard + +Create a `~/.omniroute/dashboard.json`: + +```json +{ + "health": { + "sections": [ + "server_status", + "database", + "providers", + "quota_monitors", + "recent_errors" + ], + "refresh_interval_ms": 5000 + } +} +``` + +### Pin a Provider to the Top + +```json +{ + "health": { + "pinned_providers": ["openai", "anthropic"] + } +} +``` + +--- + +## Troubleshooting + +### "Provider says healthy but requests fail" + +1. Check the **autopilot issues** β€” maybe a model is locked out +2. Look at **recent errors** for the specific error class +3. Try the **connection test** in the provider card +4. Check if the provider is **rate-limited at upstream** (not visible locally) + +### "Quota says healthy but I see 429s" + +- 429 means the provider says you've used your quota +- OmniRoute's quota tracking may be **stale** β€” the provider's truth is upstream +- Quota data refreshes automatically via the internal quota monitor + +### "Combo is failing but all targets look healthy" + +- Check **combo health** dashboard for target ordering issues +- Look at **fallback events** β€” maybe the combo is exhausting too quickly +- Verify the **strategy** matches your use case (priority vs round-robin vs auto) + +### "Database health check is failing" + +- Run `sqlite3 ~/.omniroute/storage.sqlite "PRAGMA integrity_check;"` +- If "ok" β€” false alarm, the health check is being too strict +- If anything else β€” **stop OmniRoute** and follow the [disaster recovery guide](./DATABASE_GUIDE.md#disaster-recovery) + +### "Memory heap pressure is critical" + +```bash +# Check current heap +node -e "console.log(process.memoryUsage())" + +# Trigger manual GC (if --expose-gc) +node --expose-gc -e "global.gc(); console.log(process.memoryUsage())" + +# Reduce concurrent requests (set via the dashboard Settings page, not an env var) +# There is no `MAX_CONCURRENT_REQUESTS` env var β€” configure it in Settings β†’ Concurrency. +``` + +--- + +## See Also + +- [USAGE_QUOTA_GUIDE.md](../features/USAGE_QUOTA_GUIDE.md) β€” usage & cost tracking +- [DATABASE_GUIDE.md](./DATABASE_GUIDE.md) β€” DB schema + health +- [PROXY_GUIDE.md](./PROXY_GUIDE.md) β€” proxy health (separate cache) +- [ARCHITECTURE.md](../architecture/ARCHITECTURE.md) β€” system architecture +- [RESILIENCE_GUIDE.md](../architecture/RESILIENCE_GUIDE.md) β€” circuit breaker details +- Source: `src/lib/monitoring/` (4 files, 2121 LOC) diff --git a/Open-SSE-Architecture.md b/Open-SSE-Architecture.md new file mode 100644 index 0000000..3aa047b --- /dev/null +++ b/Open-SSE-Architecture.md @@ -0,0 +1,564 @@ +> 🌍 [View in other languages](Languages) + + +# open-sse Architecture + +> **TL;DR**: `open-sse/` is the core streaming engine that powers every LLM request in OmniRoute. It contains ~406 files implementing the request pipeline, executors, services, MCP server, and translation layer. This guide explains how the pieces fit together. + +**Source:** `open-sse/` (workspace package, ~143K LOC across 406 files) + +--- + +## Why a Separate Workspace Package? + +`open-sse/` is a **standalone workspace** in the OmniRoute monorepo for several reasons: + +1. **Reusability** β€” `open-sse` is published as `@omniroute/open-sse` on npm, so other projects can use it independently +2. **Clean boundaries** β€” the streaming engine is decoupled from the OmniRoute-specific UI/DB layer +3. **Performance** β€” the engine has no Next.js dependencies, enabling faster cold starts in CLI/serverless contexts +4. **Versioning** β€” `open-sse` can release on its own cadence + +```json +// package.json +"workspaces": ["open-sse"] +``` + +--- + +## Top-Level Structure + +``` +open-sse/ +β”œβ”€β”€ index.ts # Public entry point +β”œβ”€β”€ types.d.ts # Public type exports +β”œβ”€β”€ package.json # @omniroute/open-sse +β”œβ”€β”€ config/ # Provider configs, constants, registries +β”œβ”€β”€ executors/ # Per-provider HTTP executors (59 files) +β”œβ”€β”€ handlers/ # Request handlers (chatCore, responses, etc.) +β”œβ”€β”€ lib/ # Internal utilities +β”œβ”€β”€ mcp-server/ # Model Context Protocol server +β”œβ”€β”€ services/ # ~114 service modules +β”œβ”€β”€ transformer/ # Responses API format transformer +β”œβ”€β”€ translator/ # Format translation (OpenAI ↔ Claude ↔ Gemini) +└── utils/ # Shared utilities (logging, error, stream, etc.) +``` + +### Module Counts + +| Directory | Files | Purpose | +| `executors/` | 62 | Per-provider HTTP executors (unified via DefaultExecutor factory) | +| `handlers/` | ~15 | Request entry points (chatCore, responses, embeddings) | +| `services/` | ~114 | Routing, caching, rate limiting, refresh, etc. | +| `translator/` | ~10 | Format conversion (OpenAI ↔ Claude ↔ Gemini) | +| `mcp-server/` | 30 | MCP tools and transports | +| `utils/` | ~30 | Cross-cutting utilities (logging, error, stream) | +| `config/` | ~10 | Provider configs, constants, registries | + +--- + +## The Request Pipeline + +Every LLM request flows through a **5-stage pipeline**: + +``` + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + HTTP request β”‚ 1. ROUTE β”‚ combo resolution, model selection + (Next.js route) β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ 2. TRANSLATEβ”‚ format conversion (OpenAI ↔ Claude ↔ Gemini) + β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ 3. EXECUTE β”‚ provider executor, HTTP, retry, breaker + β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ 4. STREAM β”‚ SSE transformation, backpressure + β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ 5. RECORD β”‚ usage tracking, call log, error classification + β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό + HTTP response (SSE or JSON) +``` + +### Stage 1: Route (services/combo.ts) + +**Entry point**: `handleComboChat()` in `services/combo.ts` + +Resolves the request to a concrete `(provider, model, account, credentials)` tuple: + +- Look up the combo by ID (or build a virtual combo for `auto/*` models) +- Apply routing strategy (priority, weighted, round-robin, etc.) +- Filter out unhealthy providers (circuit breaker) +- Pick the next viable target + +For `auto/*` models, this stage also: +- Runs the **9-factor scoring** algorithm (`services/autoCombo/`) +- Selects a `provider+model` pair based on health, cost, latency, etc. + +### Stage 2: Translate (translator/) + +If the source format (e.g., OpenAI) differs from the target format (e.g., Claude), the request is **translated**: + +- System prompt β†’ system message +- Tool definitions β†’ provider-specific tool format +- Reasoning/thinking parameters β†’ provider-specific equivalents +- Message role normalization (`developer` β†’ `system` for non-OpenAI) + +The `translator/index.ts` exposes: + +```ts +translateRequest(body, sourceFormat, targetFormat): TranslatedRequest +needsTranslation(source, target): boolean +``` + +### Stage 3: Execute (executors/) + +**Entry point**: `getExecutor(providerId).execute(request, options)` + +All providers use `DefaultExecutor` (`executors/default.ts`) via the `getExecutor()` factory fallback. The executor: + +- Builds the upstream URL (`buildUrl()`) +- Adds provider-specific headers (`buildHeaders()`) +- Transforms the request body (`transformRequest()`) +- Sends the HTTP request with retry + exponential backoff +- Handles auth refresh if needed (OAuth providers) + +All executors extend `BaseExecutor` (`executors/base.ts`, 1170 LOC) which provides: +- Common retry logic +- Proxy integration +- Circuit breaker integration +- Usage recording hooks + +### Stage 4: Stream (utils/stream.ts) + +For streaming responses, the executor returns a **ReadableStream**. The handler: + +- Pipes through an SSE transform (`createSSETransformStreamWithLogger`) +- Applies heartbeat pings to detect dead connections +- Handles client disconnect gracefully (`pipeWithDisconnect`) +- Transforms SSE β†’ JSON for non-streaming clients + +For non-streaming responses, the executor returns a parsed JSON object that is passed through unchanged. + +### Stage 5: Record (services/usage.ts) + +After the response (success or failure), usage is recorded: + +- `prompt_tokens`, `completion_tokens`, `cached_tokens` from the response +- `cost_usd` computed from pricing data +- `latency_ms`, `status`, `error_class` if failed +- Persisted to `usage_history` table + +Call log artifacts (if enabled) are written to `${DATA_DIR}/call_logs/`. + +--- + +## Key Files Deep-Dive + +### chatCore.ts (5977 lines) + +The **main request handler**. Despite its size, it has a clear structure: + +```ts +// Pseudo-structure of chatCore.ts +export async function handleChat(request: NextRequest) { + // 1. Auth + CORS + await authenticateRequest(request); + applyCorsHeaders(response); + + // 2. Body validation + const body = await parseRequestBody(request); + + // 3. Format detection + translation + const sourceFormat = detectFormat(request); + const targetFormat = getTargetFormat(providerId); + if (needsTranslation(sourceFormat, targetFormat)) { + body = translateRequest(body, sourceFormat, targetFormat); + } + + // 4. Combo routing + const targets = await resolveComboTargets(comboId, body); + for (const target of targets) { + try { + const result = await executeOnTarget(target, body); + await recordUsage(result); + return result; + } catch (err) { + // Continue to next target + } + } + + // 5. Emergency fallback + return await emergencyFallback(body); +} +``` + +Despite being one giant function, it's organized into **commented sections** that map to the 5-stage pipeline. + +### combo.ts (4456 LOC) + +The **routing engine** that resolves a combo to ordered targets. + +```ts +// services/combo.ts +export async function handleComboChat(body, comboId): Promise { + const targets = await resolveComboTargets(comboId, body); + for (const target of targets) { + try { + return await handleSingleModel(target, body); + } catch (err) { + log.warn("target failed, trying next", { target, err }); + } + } + throw new ComboExhaustedError("All targets failed"); +} +``` + +Supports **15 routing strategies** (see `src/shared/constants/routingStrategies.ts`): + +| Strategy | Behavior | +|----------|----------| +| `priority` | First-target ordered list | +| `weighted` | Probabilistic by per-target weight | +| `round-robin` | Cycle through targets in order | +| `context-relay` | Hand off context across targets | +| `fill-first` | Fill quota before moving to next | +| `p2c` | Power of two choices | +| `random` | Uniform random | +| `least-used` | Pick the one with fewest recent uses | +| `cost-optimized` | Cheapest healthy target first | +| `reset-aware` | Aware of provider reset windows | +| `reset-window` | Reset window-based routing | +| `strict-random` | Truly uniform (no quality weighting) | +| `auto` | Use 9-factor scoring (`autoCombo/`) | +| `lkgp` | Last known good provider first | +| `context-optimized` | Best for long-context requests | + +### base.ts (1170 LOC) + +The **abstract executor** that all 59 executors extend. It contains: + +- `buildUrl()` β€” default URL construction (subclasses override for custom) +- `buildHeaders()` β€” default headers (auth, content-type) +- `transformRequest()` β€” pass-through by default +- `execute()` β€” the main HTTP loop with retry/backoff/breaker + +```ts +// open-sse/executors/default.ts +export class DefaultExecutor extends BaseExecutor { + // Handles all OpenAI/Anthropic-compatible providers + // Providers register configurations (URL, auth, headers) but share executor logic +} +``` + +Provider-specific behavior (auth headers, base URL, version headers) is configured via the provider registry, not separate executor classes. +``` + +--- + +## Services (117 modules) + +Services are **focused, single-purpose modules** that handlers compose. The big categories: + +### Routing & Combo + +- `combo.ts` β€” entry point for combo-routed requests +- `services/autoCombo/` β€” 9-factor scoring, 8 auto routing strategies +- `wildcardRouter.ts` β€” matches wildcard routes (`gpt-*`) +- `modelFamilyFallback.ts` β€” T5 intra-family fallback + +### Rate Limiting & Quota + +- `rateLimitManager.ts` β€” token bucket per key+provider +- `usage.ts` β€” usage recording +- `quotaCache.ts` β€” in-memory quota snapshots + +### Account & Token + +- `tokenRefresh.ts` β€” OAuth refresh on 401 +- `accountFallback.ts` β€” switch to alternate account +- `sessionManager.ts` β€” multi-turn session state + +### Intelligence + +- `intentClassifier.ts` β€” classify request intent +- `taskAwareRouter.ts` β€” route by task type +- `thinkingBudget.ts` β€” allocate thinking tokens +- `contextManager.ts` β€” inject routing context + +### Resilience + +- `resilience.ts` β€” retry, backoff, breaker orchestration +- `emergencyFallback.ts` β€” last-resort fallback +- `modelDeprecation.ts` β€” auto-route to successor models + +### State + +- `signatureCache.ts` β€” dedup by request signature +- `volumeDetector.ts` β€” load shedding +- `contextHandoff.ts` β€” session serialization + +### Compression + +- `compression/` (subdirectory) β€” full compression pipeline +- 39 files covering engines, rule packs, adapters + +### Skills + +- (covered in [SKILLS.md](./SKILLS.md)) + +### Memory + +- (covered in [MEMORY.md](./MEMORY.md)) + +--- + +## Executors (75+ files) + +One file per provider. They all extend `BaseExecutor` and override what differs. + +### Common Patterns + +Providers are resolved via `getExecutor(providerId)`, which returns the configured executor. OpenAI/Anthropic-compatible providers use `DefaultExecutor` (`executors/default.ts`). Provider-specific behavior (base URL, auth headers, API version) is configured in `open-sse/config/providers/`, while request body transformations are handled in `open-sse/translator/`. + +**Custom URL** is set via provider configuration: + +```ts +// Provider config in open-sse/config/providers/ +export default { + id: "together", + baseURL: "https://api.together.xyz/v1/chat/completions", +} +``` + +**Custom auth** is handled through the provider registry's auth configuration (API key, OAuth, header profiles). + +**Custom request body** transformations (e.g., Anthropic separating `system` from `messages`) are registered per-provider in `open-sse/translator/`. +``` + +### The Executor Factory + +`executors/index.ts` exports `getExecutor(providerId)`: + +```ts +import { getExecutor } from "@omniroute/open-sse/executors"; + +const executor = getExecutor("anthropic"); +const result = await executor.execute({ + model: "claude-sonnet-4-5", + messages: [...], +}); +``` + +The factory is generated from `config/providerRegistry.ts` which lists all 212+ providers and their executor class. + +--- + +## Translators + +Translate between **3 formats**: OpenAI, Anthropic, Gemini, plus the new Responses API. + +### When Translation Happens + +```ts +import { needsTranslation, translateRequest } from "@omniroute/open-sse/translator"; + +if (needsTranslation(sourceFormat, targetFormat)) { + body = translateRequest(body, sourceFormat, targetFormat); +} +``` + +Common translations: +- `OpenAI β†’ Anthropic`: separate `system` field, `x-api-key` header +- `OpenAI β†’ Gemini`: `contents` instead of `messages`, `systemInstruction` +- `OpenAI β†’ Responses API`: `input` array, `previous_response_id` state + +### Edge Cases Handled + +- `developer` role β†’ `system` for non-OpenAI +- `system` role β†’ merged into first user message for GLM/ERNIE +- `json_schema` β†’ Gemini's `responseMimeType` + `responseSchema` +- `tools` β†’ provider-specific tool format +- Thinking parameters (o1, Claude) β†’ provider-specific equivalents + +--- + +## MCP Server + +`open-sse/mcp-server/` implements the **Model Context Protocol** server: + +- **30+ tools** (provider management, combos, memory, cache, compression, 1proxy, skills) +- **3 transports**: stdio, SSE, Streamable HTTP +- **13 scopes** for fine-grained authorization +### Tool Registration + +Tools are registered as standalone files in `open-sse/mcp-server/tools/`, each exporting a name, schema, handler, and scope: + +```ts +// open-sse/mcp-server/tools/getHealth.ts +import { z } from "zod"; +export default { + name: "omniroute_get_health", + description: "Get system health snapshot", + scope: "read:health", + inputSchema: z.object({}), + handler: async (_args, ctx) => { + return await getSystemHealth(); + }, +}; +``` + +### Transports + +```ts +// stdio (CLI usage) +startMcpStdio(server); + +// SSE (HTTP-based streaming) +startMcpSse(server, port); + +// Streamable HTTP (modern MCP) +startMcpStreamable(server, port); +``` + +### Authorization + +Every tool call goes through scope checks (`open-sse/mcp-server/auth/`): + +```ts +if (!hasScope(apiKey, "providers:read")) { + throw new Error("Insufficient scope"); +} +``` + +--- + +## Transformers + +`open-sse/transformer/` converts between **Chat Completions** and **Responses API** formats. + +### Why a Separate Transformer? + +The Responses API is OpenAI's new format with **stateful conversations** (`previous_response_id`). When a client sends a Responses request, OmniRoute: + +1. Converts Responses β†’ Chat Completions internally +2. Sends to provider (any provider that supports Chat Completions) +3. Converts the response back to Responses format +4. Streams the converted response to the client + +The transformer (`transformer/responsesTransformer.ts`) provides: + +```ts +createResponsesApiTransformStream(): TransformStream +``` + +This handles: +- `response.output_item.added` events +- `response.output_text.delta` events +- `response.completed` event +- Tool call mapping (`function_call` ↔ `tool_calls`) + +--- + +## Configuration + +`open-sse/config/` holds the configuration layer: + +| File | Purpose | +|------|---------| +| `providerRegistry.ts` | 212+ provider definitions | +| `providerModels.ts` | Model aliases, format mapping | +| `constants.ts` | Timeouts, limits, status codes | +| `defaultThinkingSignature.ts` | Default Claude thinking signature | +| `modelStrip.ts` (in services) | Per-provider field stripping | + +### Provider Registry Schema + +```ts +interface ProviderConfig { + id: string; + name: string; + baseUrl: string; + authType: "bearer" | "api-key" | "oauth" | "cookie"; + executorClass: string; + defaultModel: string; + capabilities: ProviderCapabilities; + models: ModelDefinition[]; +} +``` + +Zod validation at module load ensures all provider configs are valid. + +--- + +## Performance Constraints + +The routing engine has strict performance budgets: + +| Operation | Target | Measurement | +|-----------|--------|-------------| +| Combo resolution | <10ms | For 50 targets | +| Rate limit check | <1ms | In-memory token bucket | +| Model family fallback | <5ms | Cached family definitions | +| Request routing dispatch | <2ms | Hot path | +| **No blocking I/O in routing hot path** | β€” | All async | + +--- + +## Anti-Patterns + +❌ **Synchronous DB calls in `combo.ts`** β€” pre-compute and cache +❌ **Retry logic in handlers** β€” use `retry()` from resilience service +❌ **Direct provider config access** β€” use `providerRegistry` getters +❌ **Hardcoded fallback chains** β€” define in `modelFamilyFallback.ts` +❌ **State mutations across concurrent requests** β€” use request-scoped context only + +--- + +## Adding a New Component + +### Adding a New Service + +1. Create `open-sse/services/[serviceName].ts` with focused responsibility +2. Export main handler function and any constants +3. Add unit tests in `tests/unit/services/[serviceName].test.mjs` +4. Integrate into request pipeline in `handlers/chatCore.ts` (if routing-related) +5. Update routing logic in `combo.ts` if service affects target selection +6. Document in this file + +### Adding a New Executor + +1. Create `open-sse/executors/[provider].ts` extending `BaseExecutor` +2. Register in `config/providerRegistry.ts` +3. Add to `executors/index.ts` factory +4. Add unit tests for the executor +5. Document in `docs/architecture/ARCHITECTURE.md` + +### Adding a New MCP Tool + +1. Create or update `open-sse/mcp-server/tools/[category]Tools.ts` +2. Define Zod schema for inputs +3. Register tool in `mcp-server/index.ts` +4. Add to scope matrix in `mcp-server/auth/` +5. Add unit tests + +--- + +## See Also + +- [ARCHITECTURE.md](../architecture/ARCHITECTURE.md) β€” high-level architecture +- [CODEBASE_DOCUMENTATION.md](../architecture/CODEBASE_DOCUMENTATION.md) β€” engineering reference +- [REPOSITORY_MAP.md](../architecture/REPOSITORY_MAP.md) β€” directory-by-directory +- [AUTO-COMBO.md](../routing/AUTO-COMBO.md) β€” 9-factor scoring +- [MCP-SERVER.md](./MCP-SERVER.md) β€” MCP server +- [A2A-SERVER.md](./A2A-SERVER.md) β€” A2A server +- Source: `open-sse/` (400+ files, ~143K LOC) diff --git a/Usage-Quota-Guide.md b/Usage-Quota-Guide.md new file mode 100644 index 0000000..554d9cc --- /dev/null +++ b/Usage-Quota-Guide.md @@ -0,0 +1,427 @@ +> 🌍 [View in other languages](Languages) + + +# Usage, Quota & Spend Tracking + +> **TL;DR**: OmniRoute tracks every request's token usage, computes cost, enforces per-API-key quota, and surfaces analytics in the dashboard. This guide explains how it all works. + +**Sources:** +- `open-sse/services/usage.ts` (~70KB) β€” main usage tracking +- `src/lib/usageAnalytics.ts` (~10KB) β€” aggregation for dashboard +- `src/lib/db/quotaSnapshots.ts` β€” historical quota data +- `src/lib/db/usage*.ts` β€” multiple usage-related DB modules + +--- + +## Overview + +Every request that flows through OmniRoute generates a **usage record** that captures: + +- **Identity**: which API key, provider, model, combo +- **Tokens**: prompt tokens, completion tokens, cached tokens, total +- **Cost**: USD amount (computed from pricing data) +- **Timing**: latency, start/end timestamps +- **Status**: success, error, rate-limited, etc. + +These records are aggregated into **analytics**, persisted as **quota snapshots**, and used to enforce **per-key budget limits**. + +``` +Request ──▢ chatCore ──▢ usage.record() ──▢ SQLite + β”‚ + β”Œβ”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β” + β–Ό β–Ό β–Ό + analytics quota billing + (dashboard) (enforce) (export) +``` + +--- + +## What Gets Recorded + +The `usage.ts` service captures a **usage event** for every request: + +| Field | Type | Source | +|-------|------|--------| +| `id` | string | UUID generated on record | +| `apiKeyId` | string | The API key that initiated the request | +| `provider` | string | Provider ID (openai, anthropic, etc.) | +| `model` | string | Model ID (gpt-5, claude-opus-4-6, etc.) | +| `comboId` | string? | Combo ID if routed through a combo | +| `promptTokens` | number | From upstream response | +| `completionTokens` | number | From upstream response | +| `cachedTokens` | number | Cache hit tokens (Anthropic prompt caching, etc.) | +| `totalTokens` | number | prompt + completion | +| `costUsd` | number | Computed from pricing data | +| `latencyMs` | number | End-to-end request duration | +| `status` | enum | `success`, `error`, `rate_limited`, `timeout`, `cancelled` | +| `errorClass` | string? | Error class if status != success | +| `timestamp` | string | ISO 8601 UTC | +| `metadata` | object | Custom plugin-injected data | + +### Where Tokens Come From + +Tokens are extracted from the upstream provider's response in the **response handler**: + +```ts +// From open-sse/handlers/chatCore.ts +const response = await providerExecutor.execute(provider, request); +const usage = response.usage || { + prompt_tokens: 0, + completion_tokens: 0, + cached_tokens: 0, +}; +``` + +For providers that don't return usage (some web-cookie providers), OmniRoute **estimates** tokens using a `~4 chars per token` heuristic (see `open-sse/services/autoCombo/pipelineRouter.ts`). + +### Cached Tokens + +OmniRoute tracks `cached_tokens` separately from `prompt_tokens` because: + +- Anthropic prompt caching charges a reduced rate for cached tokens (10% of normal) +- Some providers return `cache_read_input_tokens` that should be priced differently +- Analytics can show the **cache hit rate** = `cached_tokens / prompt_tokens` + +--- + +## Cost Calculation + +Costs are computed from **pricing data** synced from LiteLLM (`src/lib/pricingSync.ts`): + +| Model | Input $/1M | Output $/1M | Cached $/1M | +|-------|-----------|-------------|-------------| +| gpt-5 | $2.50 | $10.00 | β€” | +| claude-opus-4-6 | $15.00 | $75.00 | $1.50 | +| claude-sonnet-4-5 | $3.00 | $15.00 | $0.30 | +| gemini-2.5-pro | $1.25 | $10.00 | β€” | + +The cost formula (`src/lib/usage/costCalculator.ts`): + +```ts +cost = (prompt_tokens - cached_tokens) * input_price + + cached_tokens * cached_price + + completion_tokens * output_price +``` + +> **Why subtract cached from prompt?** The cached portion is priced separately; charging input price on the whole prompt would over-count. + +### Pricing Sync + +Pricing data is auto-synced from LiteLLM via the `/api/pricing/sync` endpoint (triggered by the built-in cron task, not a user-facing env var): + +```bash +# Manual trigger +curl -X POST http://localhost:20128/api/pricing/sync +``` + +For models with no pricing data, OmniRoute falls back to **estimating cost** using internal average rates (sourced from LiteLLM's pricing data). + + +--- + +## Date Range Aggregation + +The `usageAnalytics.ts` module computes dashboard widgets from raw usage data. It supports 7 time ranges: + +| Range | Window | Use case | +|-------|--------|----------| +| `1d` | Last 24 hours | Hourly cost spike detection | +| `7d` | Last 7 days | Weekly review | +| `30d` | Last 30 days | Monthly billing | +| `90d` | Last 90 days | Quarterly analysis | +| `ytd` | Since Jan 1 of current year | Annual budget tracking | +| `all` | All time | Lifetime stats | +| `custom` | User-defined start/end | Audits, ad-hoc queries | + +### Dashboard Widgets Computed + +For any date range, the analytics layer computes: + +| Widget | Description | +|--------|-------------| +| **Summary cards** | Total requests, total cost, total tokens, success rate | +| **Daily trend chart** | Cost + tokens per day, stacked by model | +| **Activity heatmap** | Hour-of-day Γ— day-of-week grid, color = request count | +| **Model breakdown** | Pie chart of cost by model | +| **Provider breakdown** | Bar chart of requests by provider | +| **Top API keys** | Table of top 10 keys by cost | +| **Error analysis** | Error rate over time, top error classes | + +### Programmatic Access + +```ts +import { computeAnalytics } from "@/lib/usageAnalytics"; + +const analytics = await computeAnalytics( + history, // usage history records + "7d", // time range: "1d" | "7d" | "30d" | "90d" | "ytd" | "all" | "custom" + connectionMap, // provider connection map (connectionId β†’ account name) + { + startDate: "2025-01-01", // optional: for "custom" range + endDate: "2025-06-01", // optional: for "custom" range + } +); + +console.log(analytics.summary.totalCost); // 12.34 (cents) +console.log(analytics.byModel[0]); // { model, cost, requests, promptTokens, completionTokens } + +--- + +## Quota Enforcement + +Per-API-key quota is enforced in two places: + +1. **Soft limit** (`quotaWarnAt`): dashboard warning when usage exceeds threshold +2. **Hard limit** (`quotaLimit`): request rejected with HTTP 429 when exceeded + +### Configuration + +```ts +// Per API key +await updateApiKey(keyId, { + quotaWarnAt: 5_00, // $5.00 β€” show warning + quotaLimit: 10_00, // $10.00 β€” hard stop + quotaWindow: "month", // "day" | "week" | "month" | "all" +}); +``` + +### Enforcement Flow + +``` +Request ──▢ quotaCheck() + β”‚ + β”œβ”€β”€ Within limit? ──▢ allow + β”‚ + └── Over limit? ──▢ 429 Too Many Requests + with Retry-After header +``` + +### Quota Snapshots + +`quotaSnapshots` table stores **historical quota state** for trend analysis: + +| Field | Description | +|-------|-------------| +| `apiKeyId` | The key being tracked | +| `window` | "day" | "week" | "month" | +| `used` | Cost used in this window (cents) | +| `limit` | The limit (cents) | +| `resetAt` | When the window resets | +| `createdAt` | When the snapshot was taken | + +Snapshots are taken **on every request** that uses > 0 cost, and used to: + +- Render the quota progress bar in the dashboard +- Show 30-day quota trend charts +- Trigger alerts when usage approaches the limit + +--- + +## REST API + +### List Usage Records + +```bash +GET /api/usage?range=7d&limit=100 +GET /api/usage?apiKeyId=key-123&range=30d +GET /api/usage?provider=openai&range=1d +``` + +Response: + +```json +{ + "records": [ + { + "id": "uuid", + "apiKeyId": "key-123", + "provider": "openai", + "model": "gpt-5", + "promptTokens": 1234, + "completionTokens": 567, + "totalTokens": 1801, + "costUsd": 0.0050, + "latencyMs": 1234, + "status": "success", + "timestamp": "2026-06-08T12:00:00Z" + } + ], + "total": 1234, + "nextCursor": "..." +} +``` + +### Get Analytics Summary + +```bash +GET /api/usage/analytics?range=7d&groupBy=model +``` + +Response: + +```json +{ + "summary": { + "totalCost": 12.34, + "totalRequests": 5678, + "totalTokens": 12345678, + "successRate": 0.987, + "avgLatencyMs": 1234 + }, + "models": [ + { "model": "gpt-5", "cost": 8.50, "requests": 1234, "tokens": 4567890 }, + { "model": "claude-opus-4-6", "cost": 3.84, "requests": 234, "tokens": 234567 } + ], + "daily": [ + { "date": "2026-06-01", "cost": 1.50, "requests": 800 }, + { "date": "2026-06-02", "cost": 2.00, "requests": 1000 } + ] +} +``` + +### Query Usage Analytics + +Usage data is accessed via the dashboard or MCP tools, not direct REST export endpoints. Available analytics: + +- **`/api/usage/analytics`** β€” aggregated usage metrics (group by model, provider, key) +- **`/api/usage/quota`** β€” current quota status per API key +- **`/api/usage/history`** β€” request history logs + +--- + +## MCP Tools + +Two MCP tools expose usage data to agents (see `open-sse/mcp-server/tools/`): + +| Tool | Description | +|------|-------------| +| `omniroute_cost_report` | Generates a per-key cost report for a given period | +| `omniroute_check_quota` | Returns current quota status for an API key | + +Example agent invocation: + +```json +{ + "tool": "omniroute_cost_report", + "args": { "period": "week" } +} +``` + +--- + +## Retention and Cleanup + +Usage data grows ~1-10KB per request. At scale, this can be significant. + +### Retention Settings + +Usage history retention is configured via the Database Settings in the UI or via `/api/settings/database`. + +By default, usage history is retained for **90 days**. + +### Cleanup + +Old records are cleaned up by `src/lib/db/cleanup.ts`: + +- Triggered by the background cron process +- Deletes records from `usage_history` older than the configured `usageHistory` retention setting +### Storage Estimation + +| Request rate | 30-day storage | 90-day storage | +|--------------|----------------|----------------| +| 100 req/day | ~3MB | ~9MB | +| 1,000 req/day | ~30MB | ~90MB | +| 10,000 req/day | ~300MB | ~900MB | +| 100,000 req/day | ~3GB | ~9GB | + +For very high traffic, consider: + +- Reducing the retention period via Database Settings +- Using `aggregated_metrics` instead of raw records (only for analytics) + +--- + +## Cost Optimization Tips + +### 1. Use the Right Model + +```bash +# Quick answer β€” use cheap + fast +curl -d '{"model":"auto/fast","messages":[...]}' + +# Complex task β€” use quality +curl -d '{"model":"auto/smart","messages":[...]}' +``` + +### 2. Enable Caching + +Anthropic prompt caching saves **90% on repeated context**: + +```ts +// The caching is automatic β€” just include the same large system prompt +const response = await openai.chat({ + model: "claude-sonnet-4-5", + system: longSystemPrompt, // Will be cached automatically + messages: [{ role: "user", content: "..." }] +}); +``` + +### 3. Use Compression + +RTK + Caveman compression saves **15-95% on tool-heavy sessions**: + +```ts +const config = { + compression: { + engine: "rtk", + intensity: "aggressive" + } +}; +``` + +### 4. Set Per-Key Quotas + +Always set `quotaLimit` to prevent runaway costs: + +```ts +await updateApiKey(keyId, { quotaLimit: 10_00 }); // $10/month cap +``` + +### 5. Audit Top Consumers + +Use the dashboard or **`/api/usage/analytics`** to group by API key and sort by cost: + +```bash +GET /api/usage/analytics?groupBy=apiKey +``` + +--- + +## Troubleshooting + +### "Cost is higher than expected" + +1. Check **`/api/usage/analytics?groupBy=model`** β€” find the expensive model +2. Check **`/api/usage/analytics?groupBy=apiKey`** β€” find the heavy consumer +3. Verify pricing data is up to date: `POST /api/pricing/sync` + +### "Records missing" +- Check DB retention settings under Dashboard β†’ Database β†’ Cleanup β€” old records are deleted by the periodic cleanup task (`src/lib/db/cleanup.ts`) +- Check for errors in `src/lib/db/usage*.ts` β€” DB write failures are logged but not surfaced +- Verify the request actually reached `chatCore` β€” check combo routing + +### "Quota not enforcing" + +- Check the key's `quotaLimit` setting +- Verify `quotaWindow` is set correctly +- Look for `quotaSnapshots` records β€” they should be created on every request + +--- + +## See Also + +- [DATABASE_GUIDE.md](../ops/DATABASE_GUIDE.md) β€” Schema for usage tables +- [ENVIRONMENT.md](../reference/ENVIRONMENT.md#18-pricing-sync) β€” pricing sync env vars +- [AUTO-COMBO.md](../routing/AUTO-COMBO.md) β€” How `auto/fast`, `auto/cheap` reduce cost +- [API_REFERENCE.md](../reference/API_REFERENCE.md) β€” Full `/api/usage/*` reference +- Source: `open-sse/services/usage.ts`, `src/lib/usageAnalytics.ts`, `src/lib/db/usage*.ts`