mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-08-05 23:02:10 +03:00
docs(wiki): auto-sync pages + cover counts with docs
github-actions[bot]
237
2026-06-20-Unified-Compression-Config-Panel-Design.md
Normal file
237
2026-06-20-Unified-Compression-Config-Panel-Design.md
Normal file
@@ -0,0 +1,237 @@
|
||||
> 🌍 [View in other languages](Languages)
|
||||
|
||||
|
||||
# Unified Compression Config Panel — Design
|
||||
|
||||
**Status:** approved direction (2026-06-20), pending spec review
|
||||
**Base branch:** `release/v3.8.31`
|
||||
**Goal:** Make `/dashboard/context/settings` the single management panel for the master
|
||||
on/off and each compression engine's on/off + level, deriving the default pipeline from
|
||||
those toggles. Keep the Combos page solely for *chaining* (ordered named pipelines) +
|
||||
selecting the globally-active profile, and the per-engine pages solely for *detailed*
|
||||
config. Plan for (but phase) multiple named profiles with an active selector and a
|
||||
per-request override header.
|
||||
|
||||
---
|
||||
|
||||
## 1. Background — current state (the problem)
|
||||
|
||||
Compression on/off + level is split across **two stores and several UIs**, with real
|
||||
duplication:
|
||||
|
||||
- **`/api/settings/compression`** (DB `key_value` ns=`compression`, via `src/lib/db/compression.ts`)
|
||||
holds: `enabled` (master), `defaultMode`, `autoTriggerTokens`, `cavemanConfig`,
|
||||
`cavemanOutputMode`, `rtkConfig`, `aggressive`, `ultra`, language config, etc.
|
||||
- **`/api/context/combos/default`** (the *default combo pipeline*) holds the per-engine
|
||||
`enabled` + config for the **structural** engines (lite, headroom, session-dedup, ccr,
|
||||
llmlingua, aggressive, ultra) as pipeline steps. The per-engine detail pages
|
||||
(`EngineConfigPage.tsx`) read/write here via `setEngineInDefaultCombo`.
|
||||
- **`compression_combos`** table holds *named* pipelines assigned to routing combos.
|
||||
|
||||
Concrete duplications (from the UI audit):
|
||||
- Caveman on/off in **3** places (TokenSaverCard, CompressionSettingsTab, CavemanContextPageClient).
|
||||
- Caveman intensity in **3**; RTK intensity in **2**; Caveman output mode in **2**.
|
||||
- `lite/session-dedup/headroom/ccr/llmlingua` on/off **absent** from the central settings —
|
||||
only on their per-engine page (which writes the *default combo*, not the central config).
|
||||
|
||||
So "is engine X on?" is answered inconsistently (central config for caveman/rtk; default
|
||||
combo for the structural engines), and the **Combos page edits the same default pipeline**
|
||||
that a central panel would — the conceptual overlap the redesign must remove.
|
||||
|
||||
Out of scope as a *config* surface: `dashboard/compression/studio` is the Compression
|
||||
Studio (waterfall/cockpit analytics), not toggles — untouched.
|
||||
|
||||
Engines (the 10 stackable units): `lite, caveman, aggressive, ultra, rtk, headroom,
|
||||
session-dedup, ccr, llmlingua` (registered `CompressionEngine`s) + `mcpAccessibility`
|
||||
(separate path: compresses MCP tool-result outputs, not the chat pipeline). Each engine's
|
||||
`stackPriority` defines automatic ordering (session-dedup 3, ccr 4, lite 5, rtk 10,
|
||||
headroom 15, caveman 20, aggressive 30, llmlingua 35, ultra 40).
|
||||
|
||||
---
|
||||
|
||||
## 2. Design principles — the boundary
|
||||
|
||||
| Surface | Concept | Owns | Never does |
|
||||
|---|---|---|---|
|
||||
| **Panel** (`context/settings`) | "My **Default**: what is on + level" | master on/off; per-engine on/off + level | ordering/chaining; per-route assignment |
|
||||
| **Combos** (`context/combos`, menu #2) | "Named **profiles**: chaining + which is active + per-route assignment" | create/edit ordered named pipelines; pick the globally-active profile; assign to routing combos | engine on/off (inherits the active profile's membership) |
|
||||
| **Per-engine pages** (menu, after combos) | "Deep config of one engine" | filters, rules, language packs, thresholds | on/off; level (those live in the panel) |
|
||||
|
||||
**No duplication rule:** the **Panel owns the Default profile** (membership + level, order
|
||||
auto-derived by `stackPriority`). A **Combo is a *named alternative* profile** (membership
|
||||
+ level + *explicit order*). Different scopes (the one default vs named alternatives) — not
|
||||
the same object edited twice. The editable "default combo" store is **removed**; the
|
||||
default pipeline becomes **derived** from the panel.
|
||||
|
||||
---
|
||||
|
||||
## 3. Architecture
|
||||
|
||||
### 3.1 Resolution model (per request, most-specific wins)
|
||||
|
||||
```
|
||||
x-omniroute-compression header (per-request override)
|
||||
→ routing-combo override (comboOverrides[comboId]) (per-route)
|
||||
→ active profile (activeComboId: "default" | <comboId>) (global)
|
||||
→ Default = derived from panel engines map
|
||||
→ off (master disabled, or zero engines on)
|
||||
```
|
||||
|
||||
A single `resolveCompressionPlan(config, { comboId, header })` produces the effective
|
||||
`{ mode, stackedPipeline }` fed to the existing `applyCompressionAsync`. It supersedes
|
||||
today's `getEffectiveMode` (which only does combo-override → autoTrigger → defaultMode).
|
||||
|
||||
### 3.2 Data model (Model A — single source + derived pipeline)
|
||||
|
||||
**CompressionConfig gains an engines map** (the single source for the Default's on/off +
|
||||
level), replacing the editable default-combo store:
|
||||
|
||||
```ts
|
||||
interface EngineToggle {
|
||||
enabled: boolean;
|
||||
level?: string; // caveman/cavemanOutput: lite|full|ultra ; rtk: minimal|standard|aggressive ; others: undefined
|
||||
}
|
||||
interface CompressionConfig {
|
||||
enabled: boolean; // master
|
||||
engines: Record<CompressionEngineId, EngineToggle>; // NEW — the Default profile
|
||||
activeComboId: string | null; // NEW — null/"default" = derived default; else a compression_combos id
|
||||
autoTriggerTokens: number;
|
||||
autoTriggerMode?: CompressionMode; // kept (auto-trigger still selects a profile/mode on large prompts)
|
||||
preserveSystemPrompt: boolean;
|
||||
comboOverrides: Record<string, CompressionMode>; // existing per-routing-combo override
|
||||
// detailed per-engine config keeps living in its existing sub-objects
|
||||
// (cavemanConfig, rtkConfig, aggressive, ultra, …) — edited by the per-engine pages.
|
||||
}
|
||||
```
|
||||
|
||||
**Deriving the Default pipeline** from `engines` (pure function `deriveDefaultPlan`):
|
||||
- master `enabled === false` → `off`.
|
||||
- 0 engines enabled → `off`.
|
||||
- exactly 1 enabled and it is a single-mode engine (`lite|caveman|aggressive|ultra|rtk`) →
|
||||
that mode (single path — reuses today's `applyCompression(mode)`).
|
||||
- otherwise → `stacked` with the enabled engines in `stackPriority` order, each step's
|
||||
`intensity` from its `level` (the global `stackedPipeline` already accepts all 9 engines
|
||||
after the B-PIPELINE-DIVERGENCE fix).
|
||||
|
||||
The **stored `defaultMode` field** and the editable default combo
|
||||
(`/api/context/combos/default`, `setEngineInDefaultCombo`) are **removed**; the default is
|
||||
derived from `engines`. (The `CompressionMode` *type* persists — `comboOverrides`,
|
||||
`autoTriggerMode`, and the resolver's output still use it; only the stored `defaultMode`
|
||||
field is dropped.) A DB migration backfills `engines` from the current `defaultMode` +
|
||||
default-combo steps + caveman/rtk config so existing installs keep their behavior.
|
||||
|
||||
**Named combos** (`compression_combos`, existing table): N ordered pipelines. `activeComboId`
|
||||
selects which profile is globally active (`Default` or a named combo). Per-routing-combo
|
||||
assignment stays in `comboOverrides`.
|
||||
|
||||
`mcpAccessibility` keeps its own store (`/api/settings/compression/mcp-accessibility`,
|
||||
migration 056) — surfaced in the panel as a row that writes there, with a scope note.
|
||||
|
||||
### 3.3 The per-request header
|
||||
|
||||
`x-omniroute-compression: <value>` (mirrors the `x-omniroute-no-memory`/`no-cache` pattern,
|
||||
PR #4290; parsed in the request pipeline alongside the other omniroute headers). Values:
|
||||
- `off` → no compression for this request.
|
||||
- `default` → the derived Default profile.
|
||||
- `<combo-name|id>` → that named combo.
|
||||
- `engine:<id>` → a single engine (if that engine is enabled in the Default), e.g. `engine:rtk`.
|
||||
|
||||
Invalid/unknown value → ignored (falls through to the normal resolution); never errors the
|
||||
request. Header parsing + validation has a unit test asserting each form and the
|
||||
fall-through.
|
||||
|
||||
---
|
||||
|
||||
## 4. Screens
|
||||
|
||||
### 4.1 Panel — `context/settings` (engine grid)
|
||||
|
||||
A single client component (replacing the scattered `CompressionSettingsTab` +
|
||||
`CompressionTokenSaverCard` toggles):
|
||||
- **Master** on/off at top.
|
||||
- **Engine grid** (one row per engine, ordered by `stackPriority` so the row order mirrors
|
||||
run order): `[engine name + short desc] [on/off toggle] [level selector if applicable]
|
||||
[→ detail page]`. Level selector appears only for engines with levels
|
||||
(caveman lite|full|ultra, rtk minimal|standard|aggressive; caveman output mode as its own row).
|
||||
- **General** (auto-trigger tokens, preserve-system-prompt) below the grid.
|
||||
- Reads/writes the `engines` map + master via `GET/PUT /api/settings/compression` (single
|
||||
endpoint). The displayed default pipeline (derived) is shown read-only ("runs: rtk →
|
||||
caveman → …") so the user sees the effect without editing order here.
|
||||
|
||||
### 4.2 Combos — `context/combos` (menu #2)
|
||||
|
||||
- List of named combos; create/edit an **ordered** pipeline (drag/reorder + per-step level)
|
||||
— the only place for explicit chaining.
|
||||
- **Active profile selector**: `Default (panel)` | `<combo>` → writes `activeComboId`.
|
||||
- Assign a combo to a routing combo (existing `comboOverrides`).
|
||||
- Reuses the existing `CompressionCombosPageClient` / `comboFlowModel`; the `CompressionHub`
|
||||
"master mode selector" is removed (mode is now derived; the Hub becomes a read-only
|
||||
overview or is folded into the panel).
|
||||
|
||||
### 4.3 Per-engine pages (menu, after combos)
|
||||
|
||||
`EngineConfigPage` + the caveman/rtk custom pages: **lose** the on/off + level controls
|
||||
(moved to the panel) and keep only **detailed** config (filters, rules, language packs,
|
||||
thresholds, preview). They **stop writing** `/api/context/combos/default`; detailed config
|
||||
writes to its own sub-object in `/api/settings/compression` (or the existing
|
||||
caveman/rtk facade routes, which already proxy to it).
|
||||
|
||||
### 4.4 Navigation
|
||||
|
||||
`COMPRESSION_CONTEXT_GROUP` (`src/shared/constants/sidebarVisibility.ts`) reordered:
|
||||
**Settings (panel) → Combos → per-engine pages → Studio (analytics)**.
|
||||
|
||||
---
|
||||
|
||||
## 5. Consolidation / migration
|
||||
|
||||
- `CompressionTokenSaverCard` quick toggles → **absorbed into the panel; the card is removed**.
|
||||
- Duplicate caveman/rtk on/off + intensity in `CompressionSettingsTab` → removed.
|
||||
- `EngineConfigPage` → on/off+level removed; stops writing the default combo.
|
||||
- DB migration: backfill `engines` map + `activeComboId="default"` from current state
|
||||
(defaultMode + default-combo steps + caveman/rtk/ultra/aggressive enabled), so live
|
||||
installs preserve behavior. The editable default-combo route (`PUT /api/context/combos/default`)
|
||||
becomes a **read-only shim for one release** (returns the derived default; rejects writes
|
||||
with a deprecation note), then is removed.
|
||||
|
||||
---
|
||||
|
||||
## 6. Phasing
|
||||
|
||||
Each phase is its own implementation plan + PR, independently shippable and TDD'd
|
||||
(Hard Rule #18). `writing-plans` will author **Phase 1 first**; Phases 2–3 get their own
|
||||
plans later.
|
||||
|
||||
- **Phase 1 (core consolidation):** the `engines` map + `deriveDefaultPlan` + migration;
|
||||
the engine-grid panel; remove scattered/duplicate toggles; per-engine pages lose on/off;
|
||||
menu reorder. Delivers the single-panel goal. *No behavior change for existing installs
|
||||
(migration backfills).*
|
||||
- **Phase 2 (profiles):** multiple named combos + the `activeComboId` active-profile selector.
|
||||
- **Phase 3 (header):** the `x-omniroute-compression` per-request override.
|
||||
|
||||
Phases 2–3 reuse the Phase-1 resolution model (`resolveCompressionPlan`), which is built
|
||||
header/active-aware from the start so later phases only wire UI + header parsing.
|
||||
|
||||
---
|
||||
|
||||
## 7. Testing
|
||||
|
||||
- **Unit:** `deriveDefaultPlan` (every engines-map shape → expected mode/pipeline);
|
||||
migration backfill (old config → equivalent engines map); `resolveCompressionPlan`
|
||||
precedence (header > comboOverride > active > default > off); header parsing (each form +
|
||||
fall-through); panel reducer (toggle/level edits → config patch).
|
||||
- **Component (vitest):** the panel renders all engines, toggles persist, derived pipeline
|
||||
preview updates.
|
||||
- **Integration:** a config with engines `{rtk,caveman}` on → `applyCompressionAsync` runs the
|
||||
derived stacked pipeline; single engine on → single-mode path; equivalence with the old
|
||||
defaultMode behavior for the same logical config.
|
||||
- Both runners green (`test:unit` + `test:vitest`); typecheck:core clean; lint 0 errors.
|
||||
|
||||
---
|
||||
|
||||
## 8. Non-goals (YAGNI)
|
||||
|
||||
- No new compression engines in this work (the model just makes adding them trivial later).
|
||||
- No change to the engines' internal algorithms (the recent fixes are separate).
|
||||
- The Compression Studio (analytics) is not restructured.
|
||||
- Phases 2–3 UI polish (combo templates, sharing) is out of scope.
|
||||
492
2026-06-20-Unified-Compression-Config-Panel-Plan.md
Normal file
492
2026-06-20-Unified-Compression-Config-Panel-Plan.md
Normal file
@@ -0,0 +1,492 @@
|
||||
> 🌍 [View in other languages](Languages)
|
||||
|
||||
|
||||
# Unified Compression Config Panel — Phase 1 Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Make `/dashboard/context/settings` the single source for the master + per-engine on/off + level, with the default compression pipeline DERIVED from those toggles, removing the scattered/duplicate toggles — with zero behavior change for existing installs (a migration backfills).
|
||||
|
||||
**Architecture:** Add an `engines` map (+`activeComboId`) to `CompressionConfig` as the single source. A pure `deriveDefaultPlan(engines)` turns it into `{mode, stackedPipeline}`; a pure `resolveCompressionPlan(config, ctx)` applies precedence (header→combo-override→active-profile→derived-default→off) and feeds the existing `applyCompressionAsync`. A DB migration backfills the map. The engine-grid panel reads/writes the map via the single `/api/settings/compression` endpoint; per-engine pages lose on/off+level; the editable default-combo route becomes a read-only shim.
|
||||
|
||||
**Tech Stack:** TypeScript, Next.js 16 App Router, Zod, SQLite (better-sqlite3), Node test runner + Vitest (component), React.
|
||||
|
||||
**Base:** worktree `feat/compression-config-panel-v3831` off `release/v3.8.31`. Spec: `docs/compression/2026-06-20-unified-compression-config-panel-design.md`.
|
||||
|
||||
**Conventions:** unit tests run `node --import tsx/esm --test tests/unit/<f>.test.ts`; component tests run `npm run test:vitest`. Each task ends by running the FULL compression suite (`node --import tsx/esm --test tests/unit/compression/*.test.ts`) + `npm run typecheck:core` before commit. Never `--no-verify`.
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
**Create:**
|
||||
- `open-sse/services/compression/engineCatalog.ts` — pure metadata: per-engine `{ id, label, stackPriority, levels?, isSingleMode }`. One source of truth for "which engines exist, which have levels, which can be a standalone mode".
|
||||
- `open-sse/services/compression/deriveDefaultPlan.ts` — pure: `engines` map → `{ mode, stackedPipeline }`.
|
||||
- `open-sse/services/compression/resolveCompressionPlan.ts` — pure: precedence resolver.
|
||||
- `src/lib/db/migrations/102_compression_engines_map.sql` — backfill `engines` + `activeComboId`.
|
||||
- `src/app/(dashboard)/dashboard/context/settings/CompressionPanel.tsx` — the engine-grid panel.
|
||||
- `tests/unit/compression/engine-catalog.test.ts`, `derive-default-plan.test.ts`, `resolve-compression-plan.test.ts`, `compression-engines-map-migration.test.ts`.
|
||||
- `tests/unit/ui/compressionPanel.test.tsx` (vitest).
|
||||
|
||||
**Modify:**
|
||||
- `open-sse/services/compression/types.ts` — add `EngineToggle`, `CompressionConfig.engines`, `activeComboId`; keep `CompressionMode` type; drop stored `defaultMode` usage (derive).
|
||||
- `src/lib/db/compression.ts` — normalize/persist `engines` + `activeComboId`.
|
||||
- `open-sse/services/compression/strategySelector.ts` — `selectCompressionStrategy`/`getEffectiveMode` delegate to `resolveCompressionPlan`.
|
||||
- `open-sse/handlers/chatCore.ts` — call `resolveCompressionPlan`.
|
||||
- `src/app/api/settings/compression/route.ts` — accept/return `engines` + `activeComboId`.
|
||||
- `src/app/api/context/combos/default/route.ts` — read-only shim (reject writes).
|
||||
- `src/app/(dashboard)/dashboard/context/settings/page.tsx` — render `CompressionPanel` (not the old tab).
|
||||
- `src/shared/components/compression/EngineConfigPage.tsx` + caveman/rtk client pages — remove on/off+level; stop writing default combo.
|
||||
- `src/shared/constants/sidebarVisibility.ts` — reorder `COMPRESSION_CONTEXT_GROUP`.
|
||||
|
||||
**Engine reference (`stackPriority`, levels, single-mode):**
|
||||
`session-dedup`(3,—,no) `ccr`(4,—,no) `lite`(5,—,yes) `rtk`(10,minimal|standard|aggressive,yes) `headroom`(15,—,no) `caveman`(20,lite|full|ultra,yes) `aggressive`(30,—,yes) `llmlingua`(35,—,no) `ultra`(40,—,yes). Plus `cavemanOutput`(intensity lite|full|ultra, separate from input caveman) and `mcpAccessibility`(no level, separate store).
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Engine catalog (single source of engine metadata)
|
||||
|
||||
**Files:**
|
||||
- Create: `open-sse/services/compression/engineCatalog.ts`
|
||||
- Test: `tests/unit/compression/engine-catalog.test.ts`
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
```ts
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { ENGINE_CATALOG, engineMeta, ENGINE_IDS } from "@omniroute/open-sse/services/compression/engineCatalog.ts";
|
||||
|
||||
test("catalog lists every engine with stackPriority", () => {
|
||||
for (const id of ["session-dedup","ccr","lite","rtk","headroom","caveman","aggressive","llmlingua","ultra"]) {
|
||||
assert.ok(engineMeta(id), `${id} present`);
|
||||
assert.equal(typeof engineMeta(id).stackPriority, "number");
|
||||
}
|
||||
});
|
||||
test("levels + single-mode flags are correct", () => {
|
||||
assert.deepEqual(engineMeta("rtk").levels, ["minimal","standard","aggressive"]);
|
||||
assert.deepEqual(engineMeta("caveman").levels, ["lite","full","ultra"]);
|
||||
assert.equal(engineMeta("headroom").levels, undefined);
|
||||
assert.equal(engineMeta("caveman").isSingleMode, true);
|
||||
assert.equal(engineMeta("headroom").isSingleMode, false);
|
||||
});
|
||||
test("ENGINE_IDS is ordered by stackPriority", () => {
|
||||
const ps = ENGINE_IDS.map((id) => engineMeta(id).stackPriority);
|
||||
assert.deepEqual(ps, [...ps].sort((a,b)=>a-b));
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run → FAIL** (`node --import tsx/esm --test tests/unit/compression/engine-catalog.test.ts`) — module not found.
|
||||
|
||||
- [ ] **Step 3: Implement** `engineCatalog.ts`:
|
||||
|
||||
```ts
|
||||
export interface EngineMeta {
|
||||
id: string;
|
||||
label: string;
|
||||
stackPriority: number;
|
||||
levels?: string[]; // intensity options; undefined = no level selector
|
||||
isSingleMode: boolean; // can be the effective mode when it is the only engine on
|
||||
description: string;
|
||||
}
|
||||
export const ENGINE_CATALOG: Record<string, EngineMeta> = {
|
||||
"session-dedup": { id:"session-dedup", label:"Session Dedup", stackPriority:3, isSingleMode:false, description:"Cross-turn block deduplication." },
|
||||
ccr: { id:"ccr", label:"CCR (Retrieval)", stackPriority:4, isSingleMode:false, description:"Content-addressed retrieval markers." },
|
||||
lite: { id:"lite", label:"Lite", stackPriority:5, isSingleMode:true, description:"Whitespace/format cleanup." },
|
||||
rtk: { id:"rtk", label:"RTK", stackPriority:10, levels:["minimal","standard","aggressive"], isSingleMode:true, description:"Command-output filtering." },
|
||||
headroom: { id:"headroom", label:"Headroom", stackPriority:15, isSingleMode:false, description:"Tabular JSON compaction." },
|
||||
caveman: { id:"caveman", label:"Caveman", stackPriority:20, levels:["lite","full","ultra"], isSingleMode:true, description:"Rule-based prose compression." },
|
||||
aggressive: { id:"aggressive", label:"Aggressive", stackPriority:30, isSingleMode:true, description:"Summarize + age old turns." },
|
||||
llmlingua: { id:"llmlingua", label:"LLMLingua (SLM)", stackPriority:35, isSingleMode:false, description:"Semantic pruning (ONNX)." },
|
||||
ultra: { id:"ultra", label:"Ultra", stackPriority:40, isSingleMode:true, description:"Heuristic token pruning (+ optional SLM)." },
|
||||
};
|
||||
export const ENGINE_IDS: string[] = Object.values(ENGINE_CATALOG).sort((a,b)=>a.stackPriority-b.stackPriority).map((e)=>e.id);
|
||||
export function engineMeta(id: string): EngineMeta { return ENGINE_CATALOG[id]; }
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run → PASS.**
|
||||
- [ ] **Step 5: Commit** `git add open-sse/services/compression/engineCatalog.ts tests/unit/compression/engine-catalog.test.ts && git commit -m "feat(compression): engine catalog metadata (levels, single-mode, order)"`
|
||||
|
||||
---
|
||||
|
||||
## Task 2: `EngineToggle` + `engines` map + `activeComboId` on the config type
|
||||
|
||||
**Files:**
|
||||
- Modify: `open-sse/services/compression/types.ts` (the `CompressionConfig` interface + `DEFAULT_COMPRESSION_CONFIG`)
|
||||
- Test: `tests/unit/compression/engine-catalog.test.ts` (extend) — assert the default config shape.
|
||||
|
||||
- [ ] **Step 1: Add the test** (append):
|
||||
|
||||
```ts
|
||||
import { DEFAULT_COMPRESSION_CONFIG } from "@omniroute/open-sse/services/compression/types.ts";
|
||||
test("default config has an engines map + activeComboId", () => {
|
||||
assert.equal(typeof DEFAULT_COMPRESSION_CONFIG.engines, "object");
|
||||
assert.equal(DEFAULT_COMPRESSION_CONFIG.activeComboId, null);
|
||||
// default-off: every engine disabled by default (opt-in preserved)
|
||||
for (const id of ENGINE_IDS) assert.equal(DEFAULT_COMPRESSION_CONFIG.engines[id]?.enabled, false);
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run → FAIL.**
|
||||
- [ ] **Step 3: Implement** in `types.ts`: add `export interface EngineToggle { enabled: boolean; level?: string }`; add to `CompressionConfig`: `engines: Record<string, EngineToggle>;` and `activeComboId: string | null;`. In `DEFAULT_COMPRESSION_CONFIG` add `engines: Object.fromEntries(ENGINE_IDS.map((id)=>[id,{enabled:false}]))` (import `ENGINE_IDS`) and `activeComboId: null`. Keep `defaultMode` field for now (removed in Task 9 once derive is wired) to avoid breaking compilation.
|
||||
|
||||
- [ ] **Step 4: Run → PASS + `npm run typecheck:core`.**
|
||||
- [ ] **Step 5: Commit** `... -m "feat(compression): add engines map + activeComboId to CompressionConfig"`
|
||||
|
||||
---
|
||||
|
||||
## Task 3: `deriveDefaultPlan` (the heart — engines map → mode/pipeline)
|
||||
|
||||
**Files:**
|
||||
- Create: `open-sse/services/compression/deriveDefaultPlan.ts`
|
||||
- Test: `tests/unit/compression/derive-default-plan.test.ts`
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
```ts
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { deriveDefaultPlan } from "@omniroute/open-sse/services/compression/deriveDefaultPlan.ts";
|
||||
|
||||
const on = (level) => ({ enabled: true, ...(level?{level}:{}) });
|
||||
test("master off / empty / none-on => off", () => {
|
||||
assert.deepEqual(deriveDefaultPlan({}, false), { mode:"off", stackedPipeline:[] });
|
||||
assert.deepEqual(deriveDefaultPlan({}, true), { mode:"off", stackedPipeline:[] });
|
||||
assert.deepEqual(deriveDefaultPlan({ rtk:{enabled:false} }, true), { mode:"off", stackedPipeline:[] });
|
||||
});
|
||||
test("exactly one single-mode engine => that mode", () => {
|
||||
assert.deepEqual(deriveDefaultPlan({ caveman: on("full") }, true), { mode:"standard", stackedPipeline:[] });
|
||||
assert.deepEqual(deriveDefaultPlan({ rtk: on("minimal") }, true), { mode:"rtk", stackedPipeline:[] });
|
||||
assert.deepEqual(deriveDefaultPlan({ lite: on() }, true), { mode:"lite", stackedPipeline:[] });
|
||||
});
|
||||
test("one non-single-mode engine => stacked with that engine", () => {
|
||||
const p = deriveDefaultPlan({ headroom: on() }, true);
|
||||
assert.equal(p.mode, "stacked");
|
||||
assert.deepEqual(p.stackedPipeline, [{ engine:"headroom" }]);
|
||||
});
|
||||
test("multiple engines => stacked in stackPriority order, levels as intensity", () => {
|
||||
const p = deriveDefaultPlan({ caveman: on("full"), rtk: on("standard"), headroom: on() }, true);
|
||||
assert.equal(p.mode, "stacked");
|
||||
assert.deepEqual(p.stackedPipeline, [
|
||||
{ engine:"rtk", intensity:"standard" }, // pri 10
|
||||
{ engine:"headroom" }, // pri 15
|
||||
{ engine:"caveman", intensity:"full" }, // pri 20
|
||||
]);
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run → FAIL.**
|
||||
- [ ] **Step 3: Implement** `deriveDefaultPlan.ts`:
|
||||
|
||||
```ts
|
||||
import { ENGINE_CATALOG, engineMeta } from "./engineCatalog.ts";
|
||||
import type { EngineToggle } from "./types.ts";
|
||||
|
||||
const SINGLE_MODE_OF: Record<string,string> = { lite:"lite", caveman:"standard", aggressive:"aggressive", ultra:"ultra", rtk:"rtk" };
|
||||
|
||||
export interface DerivedPlan { mode: string; stackedPipeline: Array<{ engine: string; intensity?: string }>; }
|
||||
|
||||
export function deriveDefaultPlan(engines: Record<string, EngineToggle>, masterEnabled: boolean): DerivedPlan {
|
||||
if (!masterEnabled) return { mode:"off", stackedPipeline:[] };
|
||||
const onIds = Object.keys(ENGINE_CATALOG).filter((id) => engines[id]?.enabled === true);
|
||||
if (onIds.length === 0) return { mode:"off", stackedPipeline:[] };
|
||||
if (onIds.length === 1 && engineMeta(onIds[0]).isSingleMode) {
|
||||
return { mode: SINGLE_MODE_OF[onIds[0]], stackedPipeline:[] };
|
||||
}
|
||||
const ordered = onIds.sort((a,b)=>engineMeta(a).stackPriority-engineMeta(b).stackPriority);
|
||||
const stackedPipeline = ordered.map((id) => {
|
||||
const level = engines[id]?.level;
|
||||
return level ? { engine:id, intensity:level } : { engine:id };
|
||||
});
|
||||
return { mode:"stacked", stackedPipeline };
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run → PASS.**
|
||||
- [ ] **Step 5: Commit** `... -m "feat(compression): deriveDefaultPlan (engines map -> mode/pipeline)"`
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Migration 102 — backfill `engines` + `activeComboId`
|
||||
|
||||
**Files:**
|
||||
- Create: `src/lib/db/migrations/102_compression_engines_map.sql`
|
||||
- Test: `tests/unit/compression/compression-engines-map-migration.test.ts`
|
||||
|
||||
- [ ] **Step 1: Write the failing test** (uses `resetDbInstance()` + closes handles in `test.after`, per repo rule):
|
||||
|
||||
```ts
|
||||
import { test, after } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { getDbInstance, resetDbInstance } from "@/lib/db/core.ts";
|
||||
import { getCompressionSettings } from "@/lib/db/compression.ts";
|
||||
|
||||
after(() => resetDbInstance());
|
||||
test("migration backfills engines map from prior defaultMode + default combo", () => {
|
||||
resetDbInstance();
|
||||
const db = getDbInstance(); // runs migrations incl. 102
|
||||
// simulate a pre-102 install: master on, defaultMode 'standard', caveman enabled
|
||||
db.prepare("INSERT OR REPLACE INTO key_value(namespace,key,value) VALUES('compression','enabled','true')").run();
|
||||
db.prepare("INSERT OR REPLACE INTO key_value(namespace,key,value) VALUES('compression','defaultMode','\"standard\"')").run();
|
||||
db.prepare("INSERT OR REPLACE INTO key_value(namespace,key,value) VALUES('compression','cavemanConfig','{\"enabled\":true}')").run();
|
||||
const cfg = getCompressionSettings();
|
||||
assert.equal(cfg.engines.caveman.enabled, true);
|
||||
assert.equal(cfg.activeComboId, null);
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run → FAIL.**
|
||||
- [ ] **Step 3: Implement** the SQL migration (idempotent; the row backfill of derived `engines` is done in `normalizeCompressionSettings` read-path — the SQL only seeds `activeComboId` default and a marker). Migration `102_compression_engines_map.sql`:
|
||||
|
||||
```sql
|
||||
-- Phase 1 of the unified compression panel: the engines map + activeComboId become the
|
||||
-- single source. The engines map is DERIVED on read (normalizeCompressionSettings) from the
|
||||
-- legacy defaultMode + default-combo steps + caveman/rtk/ultra/aggressive config, so existing
|
||||
-- installs keep their behavior. Here we only ensure activeComboId defaults to NULL ("default").
|
||||
INSERT OR IGNORE INTO key_value (namespace, key, value) VALUES ('compression', 'activeComboId', 'null');
|
||||
```
|
||||
|
||||
(The read-path derivation in Task 5 is what makes `getCompressionSettings().engines` correct; the migration just guarantees `activeComboId` exists.)
|
||||
|
||||
- [ ] **Step 4: Run → PASS** (after Task 5's normalize is in — if running 4 before 5, expect the engines assertion to fail; do Task 5 then re-run). Commit after Task 5.
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Persist + normalize `engines` / `activeComboId` (read-path derivation)
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/lib/db/compression.ts` (`getCompressionSettings`/`normalizeCompressionSettings`/`updateCompressionSettings`)
|
||||
- Test: reuse Task 4's migration test + add a round-trip test in the same file.
|
||||
|
||||
- [ ] **Step 1: Add round-trip test**:
|
||||
|
||||
```ts
|
||||
import { updateCompressionSettings } from "@/lib/db/compression.ts";
|
||||
test("engines map persists round-trip + activeComboId", () => {
|
||||
resetDbInstance(); getDbInstance();
|
||||
updateCompressionSettings({ enabled:true, engines:{ rtk:{enabled:true,level:"standard"}, caveman:{enabled:true,level:"full"} }, activeComboId:null });
|
||||
const cfg = getCompressionSettings();
|
||||
assert.equal(cfg.engines.rtk.enabled, true);
|
||||
assert.equal(cfg.engines.rtk.level, "standard");
|
||||
assert.equal(cfg.engines.caveman.level, "full");
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run → FAIL.**
|
||||
- [ ] **Step 3: Implement** in `compression.ts`:
|
||||
- In `normalizeCompressionSettings`: read stored `engines` if present; ELSE derive it from legacy fields — `engines[id].enabled` from: caveman/rtk/ultra/aggressive `*.enabled`; structural engines from the default-combo steps (read the default combo); single-modes from `defaultMode`. Levels from `cavemanConfig.intensity`/`rtkConfig.intensity`. Read `activeComboId` (default null).
|
||||
- In `updateCompressionSettings`: accept `engines` (validate each value `{enabled:boolean, level?:string}`) + `activeComboId` and persist as `key_value` rows (`engines` as one JSON row).
|
||||
- Add a Zod sub-schema `engineToggleSchema = z.object({ enabled: z.boolean(), level: z.string().optional() })` and `engines: z.record(engineToggleSchema).optional()`, `activeComboId: z.string().nullable().optional()`.
|
||||
|
||||
- [ ] **Step 4: Run → PASS** (Task 4 + Task 5 tests). `npm run typecheck:core`.
|
||||
- [ ] **Step 5: Commit** `git add src/lib/db/migrations/102_compression_engines_map.sql src/lib/db/compression.ts tests/unit/compression/compression-engines-map-migration.test.ts && git commit -m "feat(compression): persist+backfill engines map and activeComboId (migration 102)"`
|
||||
|
||||
---
|
||||
|
||||
## Task 6: `resolveCompressionPlan` (precedence resolver)
|
||||
|
||||
**Files:**
|
||||
- Create: `open-sse/services/compression/resolveCompressionPlan.ts`
|
||||
- Test: `tests/unit/compression/resolve-compression-plan.test.ts`
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
```ts
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { resolveCompressionPlan } from "@omniroute/open-sse/services/compression/resolveCompressionPlan.ts";
|
||||
|
||||
const base = { enabled:true, engines:{ caveman:{enabled:true,level:"full"} }, activeComboId:null, comboOverrides:{} };
|
||||
test("derived default when no override/active/header", () => {
|
||||
assert.deepEqual(resolveCompressionPlan(base, {}), { mode:"standard", stackedPipeline:[] });
|
||||
});
|
||||
test("routing-combo override wins over default", () => {
|
||||
const cfg = { ...base, comboOverrides:{ cmb:"aggressive" } };
|
||||
assert.equal(resolveCompressionPlan(cfg, { comboId:"cmb" }).mode, "aggressive");
|
||||
});
|
||||
test("active named combo wins over default (Phase 2 wiring uses combos table; here pass it in)", () => {
|
||||
const cfg = { ...base, activeComboId:"c1" };
|
||||
const combos = { c1: [{ engine:"rtk", intensity:"standard" }] };
|
||||
const plan = resolveCompressionPlan(cfg, { combos });
|
||||
assert.equal(plan.mode, "stacked");
|
||||
assert.deepEqual(plan.stackedPipeline, [{ engine:"rtk", intensity:"standard" }]);
|
||||
});
|
||||
test("master off => off regardless", () => {
|
||||
assert.equal(resolveCompressionPlan({ ...base, enabled:false }, {}).mode, "off");
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run → FAIL.**
|
||||
- [ ] **Step 3: Implement** `resolveCompressionPlan.ts`:
|
||||
|
||||
```ts
|
||||
import { deriveDefaultPlan, type DerivedPlan } from "./deriveDefaultPlan.ts";
|
||||
|
||||
export interface ResolveCtx {
|
||||
comboId?: string | null;
|
||||
header?: string | null; // x-omniroute-compression (Phase 3 parses+passes; Phase 1 callers pass undefined)
|
||||
combos?: Record<string, Array<{ engine:string; intensity?:string }>>; // named combo pipelines by id
|
||||
}
|
||||
export function resolveCompressionPlan(config: any, ctx: ResolveCtx): DerivedPlan {
|
||||
if (config?.enabled === false) return { mode:"off", stackedPipeline:[] };
|
||||
// 1. header (Phase 3 supplies parsed value; here it composes if present)
|
||||
if (ctx.header) {
|
||||
if (ctx.header === "off") return { mode:"off", stackedPipeline:[] };
|
||||
if (ctx.header !== "default") {
|
||||
const fromHeader = headerToPlan(ctx.header, config, ctx);
|
||||
if (fromHeader) return fromHeader; // unknown => fall through
|
||||
}
|
||||
}
|
||||
// 2. routing-combo override
|
||||
const ov = ctx.comboId ? config?.comboOverrides?.[ctx.comboId] : undefined;
|
||||
if (ov) return modeToPlan(ov, config);
|
||||
// 3. active named combo
|
||||
if (config?.activeComboId && ctx.combos?.[config.activeComboId]) {
|
||||
return { mode:"stacked", stackedPipeline: ctx.combos[config.activeComboId] };
|
||||
}
|
||||
// 4. derived default
|
||||
return deriveDefaultPlan(config?.engines ?? {}, config?.enabled !== false);
|
||||
}
|
||||
function modeToPlan(mode: string, config: any): DerivedPlan {
|
||||
return mode === "stacked"
|
||||
? { mode:"stacked", stackedPipeline: config?.stackedPipeline ?? [] }
|
||||
: { mode, stackedPipeline:[] };
|
||||
}
|
||||
function headerToPlan(h: string, config: any, ctx: ResolveCtx): DerivedPlan | null {
|
||||
if (h.startsWith("engine:")) { const id = h.slice(7); return config?.engines?.[id]?.enabled ? deriveDefaultPlan({ [id]: config.engines[id] }, true) : null; }
|
||||
if (ctx.combos?.[h]) return { mode:"stacked", stackedPipeline: ctx.combos[h] };
|
||||
return null;
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run → PASS.**
|
||||
- [ ] **Step 5: Commit** `... -m "feat(compression): resolveCompressionPlan precedence resolver (header>override>active>default)"`
|
||||
|
||||
---
|
||||
|
||||
## Task 7: Wire the resolver into strategy selection + chatCore
|
||||
|
||||
**Files:**
|
||||
- Modify: `open-sse/services/compression/strategySelector.ts` (`selectCompressionStrategy`)
|
||||
- Modify: `open-sse/handlers/chatCore.ts` (the compression call site)
|
||||
- Test: `tests/unit/compression/strategySelector.test.ts` (extend with an engines-map case)
|
||||
|
||||
- [ ] **Step 1: Add test** asserting `selectCompressionStrategy` with `engines:{rtk:{enabled:true}}` + master on returns mode `rtk`; with `{rtk,caveman}` returns `stacked`. (Use the existing test's import + harness.)
|
||||
- [ ] **Step 2: Run → FAIL.**
|
||||
- [ ] **Step 3: Implement** — `selectCompressionStrategy` calls `resolveCompressionPlan(config, { comboId, combos })` and returns its `mode` (and expose the `stackedPipeline` so `applyCompressionAsync` uses the derived pipeline when mode==="stacked"). Load named `combos` from the combos DB module. In `chatCore.ts`, pass the active combo set; keep the `header` arg `undefined` (Phase 3 fills it). Keep `autoTriggerMode` behavior (auto-trigger still overrides to its mode on large prompts — apply BEFORE step 4 default).
|
||||
- [ ] **Step 4: Run → PASS** + full compression suite + typecheck.
|
||||
- [ ] **Step 5: Commit** `... -m "feat(compression): selectCompressionStrategy uses resolveCompressionPlan"`
|
||||
|
||||
---
|
||||
|
||||
## Task 8: API — `/api/settings/compression` carries `engines` + `activeComboId`
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/app/api/settings/compression/route.ts`
|
||||
- Test: `tests/unit/api/compression/compression-api.test.ts` (extend)
|
||||
|
||||
- [ ] **Step 1: Add test**: `PUT` with `{engines:{rtk:{enabled:true,level:"standard"}}}` then `GET` returns it; error body has no stack (`!body.error?.message?.includes("at /")`).
|
||||
- [ ] **Step 2: Run → FAIL.**
|
||||
- [ ] **Step 3: Implement** — extend the route's Zod body schema with `engines` + `activeComboId` (reuse the db sub-schema); GET returns them; errors via `buildErrorBody`.
|
||||
- [ ] **Step 4: Run → PASS** + vitest if the route is covered there.
|
||||
- [ ] **Step 5: Commit** `... -m "feat(api): settings/compression carries engines map + activeComboId"`
|
||||
|
||||
---
|
||||
|
||||
## Task 9: Remove stored `defaultMode` write-path + default-combo editable route → shim
|
||||
|
||||
**Files:**
|
||||
- Modify: `open-sse/services/compression/types.ts` (drop `defaultMode` from the persisted shape; keep `CompressionMode` type)
|
||||
- Modify: `src/app/api/context/combos/default/route.ts` (PUT → 410/deprecation; GET → derived default read-only)
|
||||
- Test: `tests/unit/api/...` for the shim (PUT rejected, GET returns derived).
|
||||
|
||||
- [ ] **Step 1: Add test**: `PUT /api/context/combos/default` returns a deprecation error (not 200); `GET` returns the derived default pipeline.
|
||||
- [ ] **Step 2: Run → FAIL.**
|
||||
- [ ] **Step 3: Implement** — `setEngineInDefaultCombo` no longer the write path; PUT route returns `buildErrorBody` "deprecated: edit engines in /api/settings/compression"; GET returns `deriveDefaultPlan(config.engines, config.enabled)`. Remove remaining reads of stored `defaultMode` (derive).
|
||||
- [ ] **Step 4: Run → PASS** + full suite + typecheck.
|
||||
- [ ] **Step 5: Commit** `... -m "refactor(compression): derive default; default-combo route is a read-only shim"`
|
||||
|
||||
---
|
||||
|
||||
## Task 10: The engine-grid panel UI
|
||||
|
||||
**Files:**
|
||||
- Create: `src/app/(dashboard)/dashboard/context/settings/CompressionPanel.tsx`
|
||||
- Modify: `src/app/(dashboard)/dashboard/context/settings/page.tsx` (render `CompressionPanel`)
|
||||
- Test: `tests/unit/ui/compressionPanel.test.tsx` (vitest, `createRoot`+`act`)
|
||||
|
||||
- [ ] **Step 1: Write the failing component test**: render `CompressionPanel` with a stubbed `fetch` returning `{enabled:true, engines:{rtk:{enabled:true,level:"standard"}}}`; assert it renders a row per `ENGINE_IDS`, the rtk level shows "standard", toggling caveman issues a `PUT` with `engines.caveman.enabled:true`, and the derived-pipeline preview text appears.
|
||||
- [ ] **Step 2: Run → FAIL** (`npm run test:vitest`).
|
||||
- [ ] **Step 3: Implement** `CompressionPanel.tsx`: master toggle; map `ENGINE_IDS` → a row component `[label+desc][Toggle][LevelSelect if meta.levels][Link → /dashboard/context/<id>]`; a derived-pipeline preview computed client-side via `deriveDefaultPlan` (import the pure fn); a `cavemanOutput` row; an `mcpAccessibility` row (writes its own endpoint, with a "MCP tool outputs" note); general settings (auto-trigger, preserve system prompt). Save via `PUT /api/settings/compression` (debounced, merge-patch like the existing `save()` pattern in `CompressionSettingsTab`). Reuse existing primitives (`Toggle`, segmented control) from the current cards.
|
||||
- [ ] **Step 4:** Update `page.tsx` to render `<CompressionPanel/>`. Run → PASS.
|
||||
- [ ] **Step 5: Commit** `... -m "feat(dashboard): engine-grid compression panel (single source for on/off + level)"`
|
||||
|
||||
---
|
||||
|
||||
## Task 11: Consolidate — remove scattered/duplicate toggles + per-engine on/off
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/app/(dashboard)/dashboard/settings/components/CompressionTokenSaverCard.tsx` (remove; or strip to a read-only summary linking to the panel)
|
||||
- Modify: `CompressionSettingsTab.tsx` (remove duplicate caveman/rtk on/off + intensity sections; keep only things not in the panel, or delete if fully superseded)
|
||||
- Modify: `src/shared/components/compression/EngineConfigPage.tsx` + `CavemanContextPageClient.tsx` + `RtkContextPageClient.tsx` (remove on/off + level; keep detailed config; stop writing `/api/context/combos/default`)
|
||||
- Test: vitest render tests for the per-engine pages assert NO enabled toggle is present; the existing tests updated to the new shape (alignment, not masking).
|
||||
|
||||
- [ ] **Step 1:** Update the affected render tests to expect the new (toggle-free) shape; run → FAIL on the still-present toggles.
|
||||
- [ ] **Step 2:** Implement the removals; per-engine detail save writes detailed config to its facade route (caveman/rtk) or the settings sub-object.
|
||||
- [ ] **Step 3:** Run vitest + full compression suite → PASS.
|
||||
- [ ] **Step 4: Commit** `... -m "refactor(dashboard): remove duplicate compression toggles; per-engine pages keep only detailed config"`
|
||||
|
||||
---
|
||||
|
||||
## Task 12: Menu reorder + integration + full validation
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/shared/constants/sidebarVisibility.ts` (`COMPRESSION_CONTEXT_GROUP`: Settings → Combos → per-engine → Studio)
|
||||
- Test: `tests/unit/...` sidebar order test (if one exists) + an integration test.
|
||||
|
||||
- [ ] **Step 1:** Add an integration test: build a config with `engines:{rtk:{enabled:true},caveman:{enabled:true,level:"full"}}`, call `selectCompressionStrategy` + `applyCompressionAsync` on a realistic body, assert the derived stacked pipeline ran (engineBreakdown has rtk+caveman) and equals the behavior of an explicit `[rtk,caveman]` stacked config. Run → (write fails first if any wiring gap).
|
||||
- [ ] **Step 2:** Reorder the sidebar group; update any sidebar order test (alignment).
|
||||
- [ ] **Step 3:** FULL validation: `npm run typecheck:core` (clean) · `npm run lint` (0 errors) · `node --import tsx/esm --test tests/unit/compression/*.test.ts` (green) · `npm run test:vitest` (green) · the api/integration compression tests.
|
||||
- [ ] **Step 4: Commit** `... -m "feat(compression): unified panel menu order + integration coverage"`
|
||||
|
||||
---
|
||||
|
||||
## Self-review notes (done while writing)
|
||||
- Spec coverage: panel (T10), combos boundary (default derived T3/T9; named/active resolver T6 — UI for active selection is Phase 2), per-engine detail-only (T11), header (resolver is header-aware T6; parsing+wiring is Phase 3), migration (T4/T5), menu (T12). ✓
|
||||
- Type consistency: `EngineToggle` (T2) used by `deriveDefaultPlan` (T3), `resolveCompressionPlan` (T6), normalize (T5), panel (T10). `DerivedPlan` shape consistent T3↔T6↔T7. ✓
|
||||
- No placeholders: each task has concrete code/tests. UI tasks (T10/T11) specify exact behavior + assertions; final per-line component code is produced at execution following the existing card patterns.
|
||||
|
||||
---
|
||||
|
||||
# 📌 RESUMO — o que ESTE plano (Fase 1) entrega
|
||||
|
||||
1. **Catálogo de engines** (`engineCatalog.ts`) — fonte única de metadados (níveis, single-mode, ordem).
|
||||
2. **Modelo de dados Fase A** — `engines` map + `activeComboId` em `CompressionConfig`, com migração 102 + backfill (zero mudança de comportamento).
|
||||
3. **`deriveDefaultPlan`** — pipeline default DERIVADO dos toggles (0/1/N engines → off/modo/stacked).
|
||||
4. **`resolveCompressionPlan`** — resolvedor de precedência (header > override por-rota > perfil ativo > default derivado > off), já header/active-aware.
|
||||
5. **Fiação no runtime** — `selectCompressionStrategy`/`chatCore` usam o resolvedor.
|
||||
6. **API** — `/api/settings/compression` carrega `engines` + `activeComboId`; rota `combos/default` vira shim read-only; `defaultMode` armazenado removido (derivado).
|
||||
7. **Painel engine-grid** — `CompressionPanel.tsx` como fonte única de master + on/off + nível, com preview do pipeline derivado.
|
||||
8. **Consolidação** — remove toggles duplicados (TokenSaverCard, CompressionSettingsTab) e tira on/off+nível das páginas por-engine (que ficam só com config detalhada).
|
||||
9. **Menu** — Settings → Combos → páginas por-engine → Studio.
|
||||
|
||||
---
|
||||
|
||||
# ⏳ PENDÊNCIAS — a fazer DEPOIS deste plano (cada uma vira seu próprio plano/PR)
|
||||
|
||||
### Fase 2 — Perfis nomeados + seletor de ativo
|
||||
- **UI de combos como perfis**: a página `context/combos` lista N combos nomeados, edita pipeline **ordenado** (drag/reorder + nível por step), e tem o seletor **"perfil ativo"** (`Default` | `<combo>`) gravando `activeComboId`. O resolvedor (Fase 1) já consome `activeComboId`; falta a UI + o carregamento dos combos nomeados no `selectCompressionStrategy`. Remover o "master mode selector" do `CompressionHub` (modo agora é derivado).
|
||||
|
||||
### Fase 3 — Header por-request `x-omniroute-compression`
|
||||
- **Parsing + wiring do header**: ler `x-omniroute-compression` no pipeline (espelhando `x-omniroute-no-memory`, PR #4290), passar como `ctx.header` ao `resolveCompressionPlan` (que já trata `off`/`default`/`<combo>`/`engine:<id>`). Doc no `API_REFERENCE` + teste de fetch-capture provando precedência por-request.
|
||||
|
||||
### Itens de compressão deferidos (do ciclo de fixes, independentes deste painel)
|
||||
- **B-OBSERVABILITY** (telemetria): engines no-op somem do `engineBreakdown` (não dá pra distinguir "rodou 0%" de "pulou"). Exige um refactor do modelo de breakdown (campo `ran`/`skipped`) que toca a UI Studio + testes que asseram `.length` — deferido conscientemente.
|
||||
- **B-CAVEMAN-PACKS**: `de`/`fr`/`ja` sem `dedup.json`+`ultra.json` (ultra==full nessas línguas). Conteúdo linguístico — adicionar os packs (ou contribuição), **sem** fallback EN que mutilaria.
|
||||
- **js-tiktoken 1.0.21→1.0.22**: bump trivial (o range `^1.0.20` já permite; só pinar o lockfile num install real).
|
||||
|
||||
### Decisão operacional pendente (não-código)
|
||||
- **Ligar o SLM (tier ultra) em produção**: validado ao vivo (49,4% real), mas mantido **OFF** por sua escolha. Quando decidir o trade-off (custo/latência/qualidade do pruning), ligar via o painel (após Fase 1) ou pela escrita de config — começando conservador (≥2000 tok).
|
||||
|
||||
### Portes upstream (opcionais, do audit — fora do escopo deste painel)
|
||||
- headroom *safety-rails*/BM25; filtros novos do rtk/token-savior; conformance GCF v3.1 (já cobrimos o `[..]:`); transformers.js 3.5.2→4.x (arriscado, major).
|
||||
98
Cluster-Decisions.md
Normal file
98
Cluster-Decisions.md
Normal file
@@ -0,0 +1,98 @@
|
||||
> 🌍 [View in other languages](Languages)
|
||||
|
||||
|
||||
# Cluster Decisions — Optional Sidecar Profiles
|
||||
|
||||
**Status:** proposal (awaiting @diegosouzapw review)
|
||||
**Date:** 2026-06-20
|
||||
**Refs:** [#3932](https://github.com/diegosouzapw/OmniRoute/issues/3932), PR #4381
|
||||
|
||||
## TL;DR
|
||||
|
||||
Two opt-in compose profiles (`memory`, `bifrost`) for the existing 8-service deployment in [`docker-compose.yml`](../../docker-compose.yml). Default-up behaviour is **unchanged**: 3 × `omniroute` replicas + Caddy + Redis + CliproxyAPI. The two new profiles add Qdrant and Bifrost as optional sidecars, gated by `docker compose --profile <name> up`. **No existing service is removed or replaced.**
|
||||
|
||||
## Why this is conservative
|
||||
|
||||
OmniRoute's existing deployment shape is already lean and proven:
|
||||
|
||||
- **`redis:7-alpine`** handles the rate-limit/cache workload at production scale.
|
||||
- **SQLite + sqlite-vec + FTS5** cover local memory + vector + text-search (see [`src/lib/memory/vectorStore.ts:108`](../../src/lib/memory/vectorStore.ts)).
|
||||
- **Caddy** is already the LB + TLS terminator ([`docker-compose.yml`](../../docker-compose.yml)).
|
||||
- **Bifrost** is already integrated as the Tier-1 router in [`src/app/api/v1/relay/chat/completions/bifrost/route.ts`](../../src/app/api/v1/relay/chat/completions/bifrost/route.ts) (sidecar proxy with kill switch via `BIFROST_ENABLED` env var — set `=0` to bypass the sidecar and fall through to the TS path).
|
||||
|
||||
The two profiles here are **scale-out options for deployments that hit the SQLite ceiling** — not migrations. Both are default-off.
|
||||
|
||||
## The two profiles
|
||||
|
||||
### `memory` — Qdrant Vector Memory Sidecar
|
||||
|
||||
**When to flip on:**
|
||||
|
||||
- >1M embeddings per deployment (sqlite-vec starts to slow at scale).
|
||||
- Multi-replica deployment that needs shared vector state across `omniroute-1/2/3`.
|
||||
- You already have an external Qdrant cluster (Qdrant Cloud, on-prem).
|
||||
|
||||
**What it adds:**
|
||||
|
||||
| Service | Image | Ports | Notes |
|
||||
| --------------- | ------------------------ | ----------- | ---------------------------------------------------------------------- |
|
||||
| `qdrant` | `qdrant/qdrant:v1.12.4` | `6333` HTTP | HNSW index; persistent volume `omniroute_qdrant_data` |
|
||||
|
||||
**Activation:** flip `qdrantEnabled = true` in the Settings UI **or** set `QDRANT_HOST=qdrant` env. See [`src/lib/memory/qdrant.ts:60`](../../src/lib/memory/qdrant.ts) for the precedence rules (settings table → env var → default).
|
||||
|
||||
**Env vars:** `QDRANT_HOST`, `QDRANT_PORT`, `QDRANT_API_KEY`, `QDRANT_COLLECTION`, `QDRANT_VECTOR_SIZE`, `QDRANT_HNSW_EF_CONSTRUCT` (see `.env.example` lines 1672-1683).
|
||||
|
||||
### `bifrost` — Bifrost Tier-1 Router Sidecar
|
||||
|
||||
**When to flip on:**
|
||||
|
||||
- You run ≥3 `omniroute` replicas and want provider rotation centralised in a single Go process.
|
||||
- You want a single audit/logging surface for upstream-provider requests across all replicas.
|
||||
- You want horizontal scaling of the Tier-1 routing layer independent of the OmniRoute replicas.
|
||||
|
||||
**What it adds:**
|
||||
|
||||
| Service | Image | Ports | Notes |
|
||||
| --------- | ------------------------------------------- | ---------- | -------------------------------------------------------------------------- |
|
||||
| `bifrost` | `ghcr.io/maximhq/bifrost:1.5.21` | `8080` | Go-based Tier-1 router; persistent logs volume `omniroute_bifrost_logs` |
|
||||
|
||||
**Activation:** set `BIFROST_BASE_URL=http://bifrost:8080` in `.env.example`. The existing sidecar proxy route at [`src/app/api/v1/relay/chat/completions/bifrost/route.ts`](../../src/app/api/v1/relay/chat/completions/bifrost/route.ts) (added in PR #4381) will pick this up automatically.
|
||||
|
||||
**Env vars:** `BIFROST_BASE_URL`, `BIFROST_API_KEY`, `BIFROST_STREAMING_ENABLED`, `BIFROST_TIMEOUT_MS` (see `.env.example` lines 1685-1695).
|
||||
|
||||
## What this PR explicitly does NOT do
|
||||
|
||||
The original issue thread floated a larger cluster rewrite. After auditing the actual workload shape, the following are **rejected** for the reasons given:
|
||||
|
||||
| Component | Verdict | Reason |
|
||||
| --------------- | --------- | ------------------------------------------------------------------------------------------------------------ |
|
||||
| **Dragonfly** | **DROP** | `redis:7-alpine` is already fine for the rate-limit workload at production scale; no ceiling to break. |
|
||||
| **NATS** | **DROP** | Each `omniroute` replica is a single Node.js process; no multi-process pub/sub workload exists. |
|
||||
| **PostgreSQL** | **DROP** | SQLite + sqlite-vec + FTS5 cover all 3 use cases; 97 migrations + Electron packaging block migration. |
|
||||
| **Neo4j** | **DROP** | Routing is a 5-table join; recursive CTE on SQLite is sufficient. |
|
||||
| **MinIO** | **DROP** | No multi-MB blob workload; images/audio are passthrough proxies. |
|
||||
| **pgvector / pg_ai / pg_textsearch** | **DROP** | Same SQLite-ceiling reason as PostgreSQL; pgvector ecosystem fragmented. |
|
||||
| **HAProxy / Envoy** | **DROP** | Caddy already does LB + TLS; both were explicitly rejected as Tier-1 routers (see `AGENTS.md`). |
|
||||
|
||||
If a future use case proves out one of these, this doc is the place to amend.
|
||||
|
||||
## 4-week rollout (if approved)
|
||||
|
||||
1. **Wk 1** — Land this PR + verification of opt-in profiles with a 3-replica compose stack.
|
||||
2. **Wk 2** — Bifrost full activation for OpenAI/Claude/Gemini/Ollama (4 of 14+ providers) using the sidecar proxy route at [`src/app/api/v1/relay/chat/completions/bifrost/route.ts`](../../src/app/api/v1/relay/chat/completions/bifrost/route.ts) (gated by `BIFROST_ENABLED`, kill-switchable at runtime).
|
||||
3. **Wk 3** — Qdrant memory profile enabled in a single test deployment; measure latency delta vs sqlite-vec.
|
||||
4. **Wk 4** — Observability healthchecks (`docker compose ps` exit codes + `wget` smoke tests); 71-pillar refresh per ADR-041.
|
||||
|
||||
## Files changed in this PR
|
||||
|
||||
| File | Change |
|
||||
| ----------------------------------------------------------- | --------------------------------------------------------------------- |
|
||||
| `docker-compose.yml` | +30 lines: `memory` profile (Qdrant), `bifrost` profile (Bifrost), persistent volumes, healthchecks. |
|
||||
| `.env.example` | +24 lines: `QDRANT_*` (6 vars), `BIFROST_*` (4 vars). |
|
||||
| `docs/reference/ENVIRONMENT.md` | +6 rows in section 25 for the `QDRANT_*` env vars. |
|
||||
| `src/lib/memory/qdrant.ts` | +33 lines: env-var fallback chain (settings → env → default) for `QDRANT_HOST`/`QDRANT_PORT`/`QDRANT_API_KEY`/`QDRANT_COLLECTION`/`QDRANT_VECTOR_SIZE`/`QDRANT_HNSW_EF_CONSTRUCT`/`QDRANT_EMBEDDING_MODEL`. |
|
||||
| `src/lib/memory/__tests__/qdrant-wiring.test.ts` | +88 lines: 9 new test cases pinning the env-var fallback precedence. |
|
||||
| `docs/architecture/cluster-decisions.md` (this file) | NEW — decision record for the opt-in profiles. |
|
||||
| `AGENTS.md` | +1 line: pointer to this doc in the reference documentation table. |
|
||||
|
||||
**Net touched code:** 4 production files (`docker-compose.yml`, `qdrant.ts`, `.env.example`, `ENVIRONMENT.md`), 1 test file (`qdrant-wiring.test.ts`), 2 doc files (`cluster-decisions.md`, `AGENTS.md`).
|
||||
621
Database-Guide.md
Normal file
621
Database-Guide.md
Normal file
@@ -0,0 +1,621 @@
|
||||
> 🌍 [View in other languages](Languages)
|
||||
|
||||
|
||||
# Database Schema & Operations Guide
|
||||
|
||||
> **TL;DR**: OmniRoute uses **SQLite with WAL journaling** as its primary store, with **AES-256-GCM** encryption at rest for sensitive fields. This guide covers the schema, migrations, backup/recovery, and operational runbooks.
|
||||
|
||||
**Sources:**
|
||||
- `src/lib/db/core.ts` — singleton + SCHEMA_SQL (17 base tables)
|
||||
- `src/lib/db/migrationRunner.ts` — versioned migrations
|
||||
- `src/lib/db/migrations/` — 94 versioned SQL files
|
||||
- `src/lib/db/encryption.ts` — encryption helpers
|
||||
- `src/lib/db/backup.ts` — backup export/import
|
||||
- `src/lib/db/healthCheck.ts` — health diagnostics
|
||||
|
||||
---
|
||||
|
||||
## Why SQLite?
|
||||
|
||||
OmniRoute chose SQLite over PostgreSQL/MySQL for several reasons:
|
||||
|
||||
| Factor | SQLite | PostgreSQL |
|
||||
|--------|--------|-----------|
|
||||
| **Deployment** | Embedded — no separate server | Requires server setup |
|
||||
| **Encryption** | Application-layer (AES-256-GCM) | Built-in TDE |
|
||||
| **Performance** | Faster for small/medium workloads | Better for huge concurrent writes |
|
||||
| **Concurrency** | WAL mode allows concurrent reads | Full MVCC |
|
||||
| **Backup** | Single-file copy | `pg_dump` or filesystem snapshot |
|
||||
| **Use case** | Per-user install, embedded | Multi-tenant SaaS |
|
||||
|
||||
For **single-user, single-instance** deployments (the primary OmniRoute use case), SQLite is simpler and faster.
|
||||
|
||||
### WAL Journaling
|
||||
|
||||
`core.ts` opens the database with **WAL (Write-Ahead Logging) mode**:
|
||||
|
||||
```ts
|
||||
// src/lib/db/core.ts
|
||||
db.pragma("journal_mode = WAL");
|
||||
db.pragma("busy_timeout = 5000");
|
||||
db.pragma("synchronous = NORMAL");
|
||||
db.pragma("cache_size = -2048");
|
||||
```
|
||||
|
||||
WAL allows **concurrent reads** during writes — important for the dashboard, which queries while requests are being recorded.
|
||||
|
||||
---
|
||||
|
||||
## Database Location
|
||||
|
||||
The SQLite file is stored at:
|
||||
|
||||
| OS | Path |
|
||||
|----|------|
|
||||
| Linux | `~/.omniroute/storage.sqlite` |
|
||||
| macOS | `~/.omniroute/storage.sqlite` |
|
||||
| Windows | `%USERPROFILE%\.omniroute\storage.sqlite` |
|
||||
| Docker | `/app/data/storage.sqlite` (configurable via `DATA_DIR`) |
|
||||
|
||||
Companion files:
|
||||
|
||||
- `storage.sqlite-wal` — write-ahead log
|
||||
- `storage.sqlite-shm` — shared memory file
|
||||
- `call_logs/` — request payload artifacts (if enabled)
|
||||
|
||||
**Override the location:**
|
||||
|
||||
```bash
|
||||
DATA_DIR=/custom/path omniroute
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Domain Module Architecture
|
||||
|
||||
OmniRoute's database has **76 domain modules** in `src/lib/db/`. Each module:
|
||||
|
||||
- Owns one or more specific tables
|
||||
- Exports typed CRUD functions
|
||||
- Never touches another module's tables
|
||||
- Uses `getDbInstance()` from `core.ts` to access the DB
|
||||
|
||||
### The 76 DB Modules
|
||||
|
||||
OmniRoute has **76 module files** in `src/lib/db/`. Below is a sampling of core modules; see the directory listing for the complete list:
|
||||
|
||||
| Module | Tables | Responsibility |
|
||||
|--------|--------|----------------|
|
||||
| `providers.ts` | `provider_connections` | OAuth/API key provider registration and credentials |
|
||||
| `models.ts` | `key_value` (model data) | Model definitions, capabilities, pricing |
|
||||
| `combos.ts` | `combos` | Combo routing configs and ordering |
|
||||
| `apiKeys.ts` | `api_keys` | API key lifecycle, scopes, quota tracking |
|
||||
| `settings.ts` | `key_value`, `api_keys`, `combos` | System configuration and shared KV store |
|
||||
| `backup.ts` | — | Backup export/import operations |
|
||||
| `proxies.ts` | `proxy_registry`, `proxy_assignments`, `provider_connections` | Proxy configs and routing rules |
|
||||
| `prompts.ts` | `prompt_templates` | Reusable prompt templates, versioning |
|
||||
| `webhooks.ts` | `webhooks` | Event-driven webhook subscriptions and logs |
|
||||
| `detailedLogs.ts` | `request_detail_logs` | Per-request audit logging (optional, high volume) |
|
||||
| `domainState.ts` | `domain_*` (5 tables) | Domain budgets, circuit breakers, lockouts, fallback chains, cost history |
|
||||
| `registeredKeys.ts` | `registered_keys`, `account_key_limits`, `provider_key_limits` | Whitelisted API keys for MCP/A2A |
|
||||
| `quotaSnapshots.ts` | `quota_snapshots` | Historical quota usage |
|
||||
| `modelComboMappings.ts` | `model_combo_mappings` | Map models to combo defaults |
|
||||
| `cliToolState.ts` | `cli_tool_state` | CLI-specific persistent state |
|
||||
| `encryption.ts` | — | Helpers for encrypting/decrypting fields |
|
||||
| `readCache.ts` | — | In-memory cache for read-heavy ops |
|
||||
| `secrets.ts` | `key_value` (encrypted entries) | Encrypted secret storage |
|
||||
| `stateReset.ts` | — | Wipe/reset DB state for testing |
|
||||
| `contextHandoffs.ts` | `context_handoffs` | Session context for agent handoff |
|
||||
| `usage*.ts` | `usage_history`, `call_logs`, `proxy_logs` | Usage tracking |
|
||||
| `compression*.ts` | `compression_settings`, `compression_combos` | Compression config |
|
||||
|
||||
### Module Boundaries
|
||||
|
||||
A core architectural rule: **modules don't access each other's tables directly**. To work with another module's data, import the function from that module.
|
||||
|
||||
```ts
|
||||
// ❌ WRONG: direct SQL from another module
|
||||
db.prepare("SELECT * FROM provider_connections").all();
|
||||
|
||||
// ✅ RIGHT: use the providers module function
|
||||
import { listProviders } from "@/lib/db/providers";
|
||||
const providers = await listProviders();
|
||||
```
|
||||
|
||||
This rule is enforced by code review — there's no static check, but violations are flagged.
|
||||
|
||||
---
|
||||
|
||||
## Base Schema (17 tables)
|
||||
|
||||
`core.ts` defines the 17 base tables in `SCHEMA_SQL`. These are created by migration `001_initial_schema.sql` and form the core schema.
|
||||
|
||||
### Core Tables (created in initial migration)
|
||||
|
||||
| Table | Purpose | Key columns |
|
||||
|-------|---------|-------------|
|
||||
| `provider_connections` | Provider credentials (encrypted) | `id`, `provider`, `auth_type`, `api_key`, `is_active` |
|
||||
| `provider_nodes` | Provider node routing info | `id`, `type`, `name`, `base_url`, `created_at` |
|
||||
| `key_value` | General KV store | `namespace`, `key`, `value` |
|
||||
| `combos` | Routing combo definitions | `id`, `name`, `data`, `sort_order` |
|
||||
| `api_keys` | API keys for the gateway | `id`, `name`, `key`, `machine_id`, `allowed_models` |
|
||||
| `db_meta` | Database metadata | `key`, `value` |
|
||||
| `usage_history` | Request usage records | `id`, `provider`, `model`, `tokens_input`, `tokens_output`, `timestamp` |
|
||||
| `call_logs` | Request payloads & responses | `id`, `timestamp`, `status`, `model`, `provider`, `latency_ms` |
|
||||
| `proxy_logs` | Proxy request logs | `id`, `timestamp`, `proxy_type`, `status`, `provider` |
|
||||
| `domain_fallback_chains` | Model-to-provider chains | `model`, `chain` |
|
||||
| `domain_budgets` | Per-domain spend budgets | `api_key_id`, `daily_limit_usd`, `warning_threshold`, `reset_interval` |
|
||||
| `domain_budget_reset_logs` | Budget reset history | `id`, `api_key_id`, `reset_interval`, `previous_spend`, `reset_at` |
|
||||
| `domain_cost_history` | Per-domain cost tracking | `id`, `api_key_id`, `cost`, `timestamp` |
|
||||
| `domain_lockout_state` | Domain rate-limit state | `identifier`, `attempts`, `locked_until` |
|
||||
| `domain_circuit_breakers` | Circuit breaker state per domain | `name`, `state`, `failure_count`, `last_failure_time` |
|
||||
| `semantic_cache` | LLM response cache | `id`, `signature`, `model`, `prompt_hash`, `response` |
|
||||
| `quota_snapshots` | Historical quota snapshots | `id`, `provider`, `connection_id`, `window_key`, `remaining_percentage` |
|
||||
|
||||
### Additional Tables (added by later migrations)
|
||||
|
||||
Subsequent migrations add tables such as:
|
||||
- `cli_tool_state` (migration 011) — CLI tool state
|
||||
- `mcp_*` tables — MCP server audit
|
||||
- `a2a_*` tables — A2A task state
|
||||
- `usage_*` tables — usage tracking
|
||||
- `plugin_*` tables — plugin system
|
||||
- `skill_executions` — skill execution history
|
||||
- `memory_*` tables — memory system
|
||||
- `compression_*` tables — compression system
|
||||
- `webhook_*` tables — webhook delivery log
|
||||
- `acp_*` tables — Agent Client Protocol
|
||||
- `oneproxy_*` tables — 1proxy marketplace
|
||||
- `proxy_assignments` — proxy scope bindings
|
||||
- `detailed_call_artifacts` — call log artifacts metadata
|
||||
- `quota_alert_history` — quota alert audit
|
||||
- `command_code_auth_sessions` — Command Code OAuth sessions
|
||||
|
||||
The full list of ~30+ tables is in `src/lib/db/migrations/`.
|
||||
|
||||
---
|
||||
|
||||
## Migrations
|
||||
|
||||
OmniRoute uses **versioned, idempotent migrations** in `src/lib/db/migrations/`. Each migration is a single SQL file named `NNN_description.sql`.
|
||||
|
||||
### Migration Naming
|
||||
|
||||
```
|
||||
001_initial_schema.sql
|
||||
002_mcp_a2a_tables.sql
|
||||
003_provider_node_custom_paths.sql
|
||||
...
|
||||
021_combo_call_log_targets.sql
|
||||
```
|
||||
|
||||
### How Migrations Run
|
||||
|
||||
At startup, `migrationRunner.ts`:
|
||||
|
||||
1. Creates `_omniroute_migrations` table if not exists
|
||||
2. Queries for already-applied migrations
|
||||
3. Applies any new migrations in order, each in a transaction
|
||||
4. Records each applied migration with timestamp
|
||||
|
||||
```ts
|
||||
// src/lib/db/migrationRunner.ts (simplified)
|
||||
export async function runMigrations(db: SqliteDatabase, migrationsDir: string) {
|
||||
const applied = getAppliedMigrations(db);
|
||||
const available = readMigrationFiles(migrationsDir);
|
||||
|
||||
for (const migration of available) {
|
||||
if (applied.includes(migration.id)) continue;
|
||||
db.transaction(() => {
|
||||
db.exec(migration.sql);
|
||||
recordAppliedMigration(db, migration.id);
|
||||
})();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Idempotency
|
||||
|
||||
Migrations must be **idempotent** — running them twice should be a no-op:
|
||||
|
||||
```sql
|
||||
-- 004_proxy_registry.sql
|
||||
CREATE TABLE IF NOT EXISTS proxy_registry (
|
||||
id TEXT PRIMARY KEY,
|
||||
host TEXT NOT NULL,
|
||||
port INTEGER NOT NULL,
|
||||
...
|
||||
);
|
||||
```
|
||||
|
||||
Use `IF NOT EXISTS`, `IF EXISTS`, and `OR IGNORE` / `OR REPLACE` clauses liberally.
|
||||
|
||||
### Adding a New Migration
|
||||
|
||||
1. **Identify the next number**: `ls src/lib/db/migrations/ | tail -1`
|
||||
2. **Create the file**: `NNN_my_change.sql`
|
||||
3. **Use safe DDL**: `CREATE TABLE IF NOT EXISTS`, `ALTER TABLE ... ADD COLUMN`
|
||||
4. **Backfill data carefully**: use `UPDATE ... WHERE ...` to handle existing rows
|
||||
5. **Test on a copy**: never run untested migrations on production
|
||||
|
||||
Example:
|
||||
|
||||
```sql
|
||||
-- 022_add_combo_priority.sql
|
||||
ALTER TABLE combos ADD COLUMN priority INTEGER DEFAULT 100;
|
||||
UPDATE combos SET priority = 100 WHERE priority IS NULL;
|
||||
CREATE INDEX IF NOT EXISTS idx_combos_priority ON combos(priority);
|
||||
```
|
||||
|
||||
> **Backwards-incompatible changes** (e.g., dropping columns) are tricky. OmniRoute does NOT support downgrade — once a migration is applied, the schema change is permanent. Plan accordingly.
|
||||
|
||||
---
|
||||
|
||||
## Encryption at Rest
|
||||
|
||||
Sensitive fields (API keys, OAuth tokens, connection strings) are encrypted at rest using **AES-256-GCM**.
|
||||
|
||||
### How It Works
|
||||
|
||||
```ts
|
||||
// src/lib/db/encryption.ts (simplified)
|
||||
const key = deriveKeyFromPassphrase(passphrase, salt);
|
||||
const iv = randomBytes(12);
|
||||
const cipher = createCipheriv("aes-256-gcm", key, iv);
|
||||
const encrypted = Buffer.concat([cipher.update(plaintext), cipher.final()]);
|
||||
const authTag = cipher.getAuthTag();
|
||||
return { encrypted, iv, authTag };
|
||||
```
|
||||
|
||||
### Where It's Used
|
||||
|
||||
- `provider_connections.api_key` — encrypted at application level
|
||||
- `provider_connections.access_token`, `refresh_token`, `id_token` — encrypted at application level
|
||||
- `key_value` entries with `namespace = "secrets"` — encrypted at application level
|
||||
- `proxy_registry.auth` — encrypted at application level (if present)
|
||||
|
||||
### Encryption Key
|
||||
|
||||
The encryption key is derived from a **passphrase** (set via `STORAGE_ENCRYPTION_KEY` env var) and a **salt** (stored in the DB). Both are required to decrypt data.
|
||||
|
||||
```bash
|
||||
# Generate a secure passphrase
|
||||
openssl rand -hex 32
|
||||
|
||||
# Set in .env
|
||||
STORAGE_ENCRYPTION_KEY=<your-key>
|
||||
```
|
||||
|
||||
> **Critical**: Losing the encryption key means losing access to all encrypted data. **Back up the key separately from the database**.
|
||||
|
||||
### What's NOT Encrypted
|
||||
|
||||
For performance reasons, the following are stored in plaintext:
|
||||
|
||||
- Provider display names
|
||||
- Model definitions (already public)
|
||||
- Routing rules
|
||||
- Usage records (no PII)
|
||||
|
||||
---
|
||||
|
||||
## Encryption Caveats (v3.8.16+)
|
||||
|
||||
OmniRoute uses **`migrateLegacyEncryptedString()`** to handle two encryption schemes transparently:
|
||||
|
||||
- **Legacy** (pre-v3.5.0): XOR-based "encryption" (not real crypto)
|
||||
- **Current**: AES-256-GCM with proper IV and auth tag
|
||||
|
||||
The migration helper detects the legacy format and re-encrypts with the new scheme on first read. This means you can upgrade an old database without losing credentials.
|
||||
|
||||
---
|
||||
|
||||
## Read Cache
|
||||
|
||||
For frequently-read data (models, providers, settings), `readCache.ts` provides an **in-memory cache**:
|
||||
|
||||
```ts
|
||||
// Cached at startup, invalidated on write
|
||||
const providers = await getCachedProviders(); // Fast, in-memory
|
||||
const fresh = await listProviders(); // Slow, hits DB
|
||||
```
|
||||
|
||||
| Cached entity | Cache key | TTL |
|
||||
|---------------|-----------|-----|
|
||||
| `models` | `models:v1` | Until write |
|
||||
| `provider_connections` | `providers:v1` | Until write |
|
||||
| `settings` | `settings:v1` | Until write |
|
||||
| `combos` | `combos:v1` | Until write |
|
||||
|
||||
Cache is invalidated on every write to the corresponding table.
|
||||
|
||||
---
|
||||
|
||||
## Backup and Recovery
|
||||
|
||||
### Manual Backup
|
||||
|
||||
```bash
|
||||
# Use the CLI to create a local backup
|
||||
omniroute backup create --name pre-migration
|
||||
|
||||
# Or via the API
|
||||
curl -X PUT http://localhost:20128/api/db-backups \
|
||||
-H "Authorization: Bearer $MANAGEMENT_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"name": "pre-migration"}'
|
||||
```
|
||||
|
||||
The backup file includes:
|
||||
|
||||
- All DB tables (serialized to JSON)
|
||||
- Call log artifacts (base64-encoded, optional)
|
||||
- Settings + secrets (encrypted)
|
||||
- Plugin configuration
|
||||
|
||||
### Restore
|
||||
|
||||
```bash
|
||||
# Via CLI
|
||||
omniroute restore pre-migration
|
||||
|
||||
# Via API
|
||||
curl -X POST http://localhost:20128/api/db-backups/restore \
|
||||
-H "Authorization: Bearer $MANAGEMENT_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"name": "pre-migration"}'
|
||||
```
|
||||
|
||||
> **Warning**: Restore overwrites the entire DB. Stop all clients first.
|
||||
|
||||
### Automated Backups
|
||||
|
||||
```bash
|
||||
# Enable automated daily backups via CLI
|
||||
omniroute backup auto enable --cron "0 2 * * *" --retention 7
|
||||
```
|
||||
|
||||
### SQLite Hot Backup
|
||||
|
||||
For zero-downtime backup of a live DB:
|
||||
|
||||
```bash
|
||||
sqlite3 ~/.omniroute/storage.sqlite ".backup /backups/omniroute-hot.db"
|
||||
```
|
||||
|
||||
This uses SQLite's online backup API — safe to run while OmniRoute is running.
|
||||
|
||||
---
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### WAL Mode
|
||||
|
||||
WAL is enabled by default. For high-write workloads, consider:
|
||||
|
||||
```sql
|
||||
PRAGMA wal_autocheckpoint = 1000; -- Checkpoint every 1000 pages
|
||||
PRAGMA journal_size_limit = 67108864; -- 64MB WAL cap
|
||||
```
|
||||
|
||||
### Indexes
|
||||
|
||||
Key indexes for performance (auto-created by migrations):
|
||||
|
||||
- `idx_models_provider` — model lookups by provider
|
||||
- `idx_combo_targets_combo_id` — combo target expansion
|
||||
- `idx_usage_history_api_key_timestamp` — usage analytics
|
||||
- `idx_quota_snapshots_api_key_window` — quota tracking
|
||||
- `idx_call_logs_timestamp` — call log queries
|
||||
|
||||
To add a new index, create a migration:
|
||||
|
||||
```sql
|
||||
-- 023_add_my_index.sql
|
||||
CREATE INDEX IF NOT EXISTS idx_my_table_my_column ON my_table(my_column);
|
||||
```
|
||||
|
||||
### Memory-Mapped I/O
|
||||
|
||||
For very large databases (>10GB), memory mapping can be adjusted via SQLite pragma:
|
||||
|
||||
```sql
|
||||
-- Set via SQLite pragma (adjust in core.ts or runtime)
|
||||
PRAGMA mmap_size = 268435456; -- 256MB
|
||||
```
|
||||
|
||||
### Compaction
|
||||
|
||||
Long-running OmniRoute instances benefit from occasional `VACUUM`:
|
||||
|
||||
```bash
|
||||
sqlite3 ~/.omniroute/storage.sqlite "VACUUM;"
|
||||
```
|
||||
|
||||
Run monthly during low-traffic windows. (WAL mode reduces the need, but doesn't eliminate it.)
|
||||
|
||||
---
|
||||
|
||||
## Health Check
|
||||
|
||||
`src/lib/db/healthCheck.ts` provides **DB-level health diagnostics**:
|
||||
|
||||
```bash
|
||||
GET /api/db/health
|
||||
|
||||
Returns:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "healthy",
|
||||
"checks": {
|
||||
"writable": { "status": "pass" },
|
||||
"integrity": { "status": "pass", "result": "ok" },
|
||||
"foreign_keys": { "status": "pass", "violations": 0 },
|
||||
"orphaned_artifacts": { "status": "warn", "count": 12 },
|
||||
"table_sizes": {
|
||||
"usage_history": { "rows": 12345, "size_mb": 12.3 },
|
||||
"call_logs": { "rows": 567, "size_mb": 2.1 }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Run `PRAGMA integrity_check` to detect corruption:
|
||||
|
||||
```bash
|
||||
sqlite3 ~/.omniroute/storage.sqlite "PRAGMA integrity_check;"
|
||||
# Should print: ok
|
||||
```
|
||||
|
||||
If it returns anything other than `ok`, **stop using the database immediately** and restore from backup.
|
||||
|
||||
---
|
||||
|
||||
## Disaster Recovery
|
||||
|
||||
### Scenario 1: WAL File Lost
|
||||
|
||||
The `-wal` file is missing but `-shm` and main DB are intact:
|
||||
|
||||
```bash
|
||||
# Recovers automatically on next open
|
||||
omniroute
|
||||
```
|
||||
|
||||
If SQLite can't auto-recover:
|
||||
|
||||
```bash
|
||||
sqlite3 ~/.omniroute/storage.sqlite ".recover" > recovered.sql
|
||||
sqlite3 recovered.db < recovered.sql
|
||||
mv recovered.db ~/.omniroute/storage.sqlite
|
||||
```
|
||||
|
||||
### Scenario 2: Main DB File Corrupted
|
||||
|
||||
Restore from backup:
|
||||
|
||||
```bash
|
||||
omniroute sync pull --merge # or: omniroute backup restore <backup-id>
|
||||
```
|
||||
|
||||
### Scenario 3: Encryption Key Lost
|
||||
|
||||
**No recovery possible** without the key. The encrypted fields are unreadable. Re-add all providers manually with new credentials.
|
||||
|
||||
> **Mitigation**: Always back up the encryption key separately, ideally in a password manager or KMS.
|
||||
|
||||
### Scenario 4: Disk Full
|
||||
|
||||
SQLite will return `SQLITE_FULL` errors. Free disk space, then:
|
||||
|
||||
```bash
|
||||
# Checkpoint WAL to free up space
|
||||
sqlite3 ~/.omniroute/storage.sqlite "PRAGMA wal_checkpoint(TRUNCATE);"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Operations
|
||||
|
||||
### Inspect a Table
|
||||
|
||||
```bash
|
||||
sqlite3 ~/.omniroute/storage.sqlite "SELECT * FROM api_keys LIMIT 5;"
|
||||
```
|
||||
|
||||
### Count Rows in All Tables
|
||||
|
||||
```bash
|
||||
sqlite3 ~/.omniroute/storage.sqlite <<EOF
|
||||
SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%';
|
||||
EOF
|
||||
```
|
||||
|
||||
|
||||
### Reset (Wipe) All Data
|
||||
|
||||
```bash
|
||||
# Stop OmniRoute first
|
||||
omniroute stop
|
||||
|
||||
# Delete the DB file
|
||||
rm ~/.omniroute/storage.sqlite*
|
||||
|
||||
# Restart (will recreate empty DB)
|
||||
omniroute
|
||||
```
|
||||
|
||||
For a **selective** reset (keep providers, wipe usage):
|
||||
|
||||
```bash
|
||||
DELETE FROM usage_history WHERE timestamp < datetime('now', '-30 day');
|
||||
DELETE FROM call_logs WHERE timestamp < datetime('now', '-30 day');
|
||||
DELETE FROM proxy_logs WHERE timestamp < datetime('now', '-30 day');
|
||||
```
|
||||
|
||||
### Export Single Table
|
||||
|
||||
```bash
|
||||
sqlite3 ~/.omniroute/storage.sqlite <<EOF
|
||||
.mode csv
|
||||
.output api_keys.csv
|
||||
SELECT * FROM api_keys;
|
||||
EOF
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Database is locked"
|
||||
|
||||
Another process is holding a write lock. Either:
|
||||
|
||||
- Wait for the other process to finish (check `lsof | grep storage.sqlite`)
|
||||
- Kill the other process
|
||||
- If persistent, restart OmniRoute
|
||||
|
||||
### "Foreign key constraint failed"
|
||||
|
||||
A domain module is violating referential integrity. Check:
|
||||
|
||||
- Orphaned rows in dependent tables
|
||||
- Cascading deletes that didn't propagate
|
||||
- Recent migration that changed a foreign key
|
||||
|
||||
Run `PRAGMA foreign_key_check;` to find violations.
|
||||
|
||||
### "Out of memory"
|
||||
|
||||
SQLite's memory-mapped I/O is exceeding the OS limit. Reduce via SQLite pragma:
|
||||
|
||||
```sql
|
||||
PRAGMA mmap_size = 134217728; -- 128MB instead of 256MB
|
||||
```
|
||||
|
||||
Or disable:
|
||||
|
||||
```sql
|
||||
PRAGMA mmap_size = 0;
|
||||
```
|
||||
|
||||
### "Migration failed mid-way"
|
||||
|
||||
The migration ran in a transaction, so it should have rolled back. If not:
|
||||
|
||||
1. **Stop OmniRoute** (prevent further attempts)
|
||||
2. **Check the DB state** with `sqlite3`
|
||||
3. **Manually fix** the partial migration
|
||||
4. **Re-run** OmniRoute (the migration will be retried)
|
||||
|
||||
To prevent this, always test migrations on a copy first.
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [USAGE_QUOTA_GUIDE.md](../features/USAGE_QUOTA_GUIDE.md) — usage tables
|
||||
- [MONITORING_GUIDE.md](./MONITORING_GUIDE.md) — health monitoring
|
||||
- [RELEASE_CHECKLIST.md](./RELEASE_CHECKLIST.md) — release flow
|
||||
- Source: `src/lib/db/` (80+ files, ~25K LOC)
|
||||
468
Monitoring-Guide.md
Normal file
468
Monitoring-Guide.md
Normal file
@@ -0,0 +1,468 @@
|
||||
> 🌍 [View in other languages](Languages)
|
||||
|
||||
|
||||
# Monitoring & Observability Guide
|
||||
|
||||
> **TL;DR**: OmniRoute ships with built-in health monitoring, provider autopilot, quota tracking, and observability hooks. This guide covers the dashboard, alerts, and troubleshooting.
|
||||
|
||||
**Sources:**
|
||||
- `src/lib/monitoring/observability.ts` — observability snapshot
|
||||
- `src/lib/monitoring/comboHealthAutopilot.ts` — combo health autopilot
|
||||
- `src/lib/monitoring/providerHealthAutopilot.ts` — provider autopilot
|
||||
- `src/lib/monitoring/providerHealthMatrix.ts` — provider health matrix
|
||||
- `src/lib/localHealthCheck.ts` — local health check
|
||||
- `src/lib/tokenHealthCheck.ts` — token refresh health
|
||||
- `src/lib/proxyHealth.ts` — proxy health cache (covered in PROXY_GUIDE.md)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
OmniRoute has **3 layers of monitoring**:
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ Layer 1: System Health (server-level) │
|
||||
│ ├─ localHealthCheck.ts — DB, ports, native deps │
|
||||
│ ├─ db/healthCheck.ts — integrity, FK, orphaned artifacts │
|
||||
│ └─ Dashboard: /dashboard/health │
|
||||
├──────────────────────────────────────────────────────────────┤
|
||||
│ Layer 2: Provider Health (per-provider resilience) │
|
||||
│ ├─ providerHealthAutopilot.ts — circuit breaker, cooldowns │
|
||||
│ ├─ providerHealthMatrix.ts — health scores by provider/model │
|
||||
│ └─ Dashboard: /dashboard/providers │
|
||||
├──────────────────────────────────────────────────────────────┤
|
||||
│ Layer 3: Live Observability (runtime snapshots) │
|
||||
│ ├─ observability.ts — circuit breakers, sessions, quota │
|
||||
│ ├─ tokenHealthCheck.ts — OAuth token refresh health │
|
||||
│ └─ MCP tools: omniroute_get_health, omniroute_get_session_snapshot │
|
||||
└──────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dashboard Pages
|
||||
|
||||
### `/dashboard/health` (System Health)
|
||||
|
||||
The top-level health dashboard shows:
|
||||
|
||||
| Section | What it shows |
|
||||
|---------|---------------|
|
||||
| **Server status** | Uptime, version, port, active connections |
|
||||
| **Database** | Connection, integrity, WAL size, recent migrations |
|
||||
| **Provider summary** | Active count, healthy count, breaker open count |
|
||||
| **Quota monitors** | Active sessions, alerting, exhausted |
|
||||
| **Recent errors** | Last 10 errors with stack traces |
|
||||
| **Resource usage** | Memory, CPU, heap pressure indicator |
|
||||
|
||||
### `/dashboard/providers` (Provider Health)
|
||||
|
||||
Per-provider dashboard:
|
||||
|
||||
| Column | Description |
|
||||
|--------|-------------|
|
||||
| Provider | Provider ID + display name |
|
||||
| Health | Green/yellow/red status |
|
||||
| Circuit | Open/closed/half-open state |
|
||||
| Connections | Count of connections, last refresh |
|
||||
| Models | Available models, health per model |
|
||||
| Cost | Today's cost, 7-day trend |
|
||||
| Errors | Last 24h error count, top error class |
|
||||
|
||||
Click a provider to see:
|
||||
- Recent requests with latency breakdown
|
||||
- Per-connection health scores
|
||||
- Per-model lockouts
|
||||
- Autopilot recommendations
|
||||
|
||||
### `/dashboard/quota` (Quota Tracking)
|
||||
|
||||
For each API key:
|
||||
|
||||
- Current usage vs limit (progress bar)
|
||||
- Quota trend (30-day chart)
|
||||
- Next reset time
|
||||
- Alert history
|
||||
|
||||
### `/dashboard/combos` (Combo Health)
|
||||
|
||||
Per-combo:
|
||||
|
||||
- Strategy + targets
|
||||
- Health per target
|
||||
- Recent fallback events
|
||||
- Success rate (24h, 7d, 30d)
|
||||
|
||||
---
|
||||
|
||||
## Health Check API
|
||||
|
||||
> **Note:** Only `GET /api/monitoring/health` is exposed as a REST endpoint. All other monitoring data (provider health, autopilot issues, quota monitors, token health, latency) is accessed via the **MCP tool** `observability_snapshot` or the **dashboard** pages — there are no dedicated REST routes for these.
|
||||
|
||||
### System Health
|
||||
|
||||
```bash
|
||||
GET /api/monitoring/health
|
||||
```
|
||||
|
||||
Response:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "healthy",
|
||||
"version": "3.8.16",
|
||||
"uptime": 123456,
|
||||
"checks": {
|
||||
"database": { "status": "pass", "latency_ms": 2 },
|
||||
"writeable": { "status": "pass" },
|
||||
"integrity": { "status": "pass", "result": "ok" },
|
||||
"foreign_keys": { "status": "pass", "violations": 0 },
|
||||
"heap_pressure": { "status": "pass", "usage_mb": 142, "threshold_mb": 512 },
|
||||
"active_sessions": 12,
|
||||
"providers": {
|
||||
"total": 7,
|
||||
"healthy": 6,
|
||||
"degraded": 1,
|
||||
"down": 0
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Provider Health
|
||||
|
||||
> **No REST endpoint.** Provider health data is available via the MCP tool `observability_snapshot` or the dashboard `/dashboard/providers` page.
|
||||
|
||||
### Provider Detail
|
||||
|
||||
> **No REST endpoint.** Per-provider detail is available via the dashboard `/dashboard/providers` page.
|
||||
|
||||
---
|
||||
|
||||
## Provider Health Autopilot
|
||||
|
||||
The `providerHealthAutopilot.ts` module is a **self-healing system** that:
|
||||
|
||||
1. Detects provider issues (circuit open, cooldowns, lockouts, quota warnings)
|
||||
2. Generates **recommended actions** to resolve them
|
||||
3. Optionally **auto-executes** low-risk actions
|
||||
|
||||
### Issue Types Detected
|
||||
|
||||
| Issue kind | Severity | Example condition |
|
||||
|------------|----------|-------------------|
|
||||
| `provider_circuit_open` | critical | Circuit breaker open after 5 failures |
|
||||
| `provider_circuit_half_open` | warning | Circuit testing recovery |
|
||||
| `connection_cooldown` | warning | Connection in cooldown after 429 |
|
||||
| `stale_connection_error` | warning | Last refresh failed 30+ minutes ago |
|
||||
| `terminal_connection_error` | critical | OAuth revoked, key invalid |
|
||||
| `inactive_connection` | info | Connection disabled in settings |
|
||||
| `model_lockout` | warning | Specific model in quarantine |
|
||||
| `quota_monitor_warning` | warning | Quota at 80%+ usage |
|
||||
|
||||
### Action Types Generated
|
||||
|
||||
| Action | Risk | Description |
|
||||
|--------|------|-------------|
|
||||
| `clear_provider_breaker` | medium | Reset the circuit breaker to closed |
|
||||
| `clear_connection_cooldown` | low | Remove cooldown from a connection |
|
||||
| `clear_stale_connection_error` | low | Clear stale error flag |
|
||||
| `clear_model_lockout` | low | Re-enable a quarantined model |
|
||||
| `reactivate_connection` | medium | Re-enable a deactivated connection |
|
||||
| `deactivate_connection` | high | Disable a problematic connection |
|
||||
|
||||
### API
|
||||
|
||||
> **No REST endpoint.** Autopilot issues are available via the MCP tool `observability_snapshot` or the dashboard. The autopilot runs internally; its behavior is configured via the settings DB (per-connection `autopilotMode` field), not environment variables — `grep -rn` for an autopilot-mode env var returns zero hits.
|
||||
|
||||
### Autopilot Mode
|
||||
|
||||
The autopilot operates in **manual mode** by default — it detects issues and generates recommended actions, but does not auto-apply them. Actions can be applied via the dashboard.
|
||||
|
||||
---
|
||||
|
||||
## Combo Health Autopilot
|
||||
|
||||
`comboHealthAutopilot.ts` is the **combo-specific** equivalent of the provider autopilot. It:
|
||||
|
||||
- Detects unhealthy combos
|
||||
- Recommends target reordering
|
||||
- Suggests disabling broken targets
|
||||
- Auto-removes dead targets after N failures
|
||||
|
||||
### Combo Issue Examples
|
||||
|
||||
```
|
||||
Combo "always-on" (priority strategy)
|
||||
├─ Target 1: openai/gpt-5 (healthy)
|
||||
├─ Target 2: anthropic/claude-opus-4-6 (⚠️ model lockout until 14:00)
|
||||
└─ Target 3: kiro/claude-sonnet-4-5 (healthy)
|
||||
|
||||
Recommended action: Reorder — move kiro above anthropic until lockout expires
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quota Monitors
|
||||
|
||||
`observability.ts` exposes **per-session quota monitors** for subscription providers (Claude Code, Codex, GitHub Copilot):
|
||||
|
||||
```ts
|
||||
interface QuotaMonitorSnapshot {
|
||||
sessionId: string;
|
||||
provider: string;
|
||||
accountId: string;
|
||||
status: "starting" | "idle" | "healthy" | "warning" | "exhausted" | "error";
|
||||
lastQuotaPercent: number | null; // 0-100
|
||||
lastQuotaUsed: number | null;
|
||||
lastQuotaTotal: number | null;
|
||||
lastResetAt: string | null;
|
||||
nextPollAt: string | null;
|
||||
totalPolls: number;
|
||||
totalAlerts: number;
|
||||
consecutiveFailures: number;
|
||||
}
|
||||
```
|
||||
|
||||
### Status Meanings
|
||||
|
||||
| Status | When | UI action |
|
||||
|--------|------|-----------|
|
||||
| `starting` | Initial poll in progress | Spinner |
|
||||
| `idle` | No recent activity | Hidden from dashboard |
|
||||
| `healthy` | Quota > 50% remaining | Green dot |
|
||||
| `warning` | Quota < 50% remaining | Yellow alert |
|
||||
| `exhausted` | Quota = 0% | Red block, route to next provider |
|
||||
| `error` | Polling failed | Red dot, retry soon |
|
||||
|
||||
### API
|
||||
|
||||
> **No REST endpoint.** Quota monitor data is available via the MCP tool `observability_snapshot` or the dashboard.
|
||||
|
||||
---
|
||||
|
||||
## Observability Snapshot
|
||||
|
||||
The MCP tool `observability_snapshot` returns a **complete system snapshot** for AI agents:
|
||||
|
||||
```json
|
||||
{
|
||||
"circuitBreakers": [
|
||||
{
|
||||
"name": "openai",
|
||||
"state": "closed",
|
||||
"failureCount": 0,
|
||||
"lastFailureTime": null,
|
||||
"retryAfterMs": null
|
||||
}
|
||||
],
|
||||
"sessions": [
|
||||
{
|
||||
"sessionId": "sess-123",
|
||||
"createdAt": 1234567890,
|
||||
"lastActive": 1234567999,
|
||||
"requestCount": 42,
|
||||
"connectionId": "conn-456",
|
||||
"ageMs": 109
|
||||
}
|
||||
],
|
||||
"quotaMonitors": { /* see above */ },
|
||||
"uptime": 12345,
|
||||
"version": "3.8.16"
|
||||
}
|
||||
```
|
||||
|
||||
Agents use this to make **routing decisions** — for example, "if openai's circuit is open, route to anthropic first".
|
||||
|
||||
---
|
||||
|
||||
## Token Health Check
|
||||
|
||||
OAuth providers (Claude Code, GitHub Copilot, Cursor) need **periodic token refresh**. `src/lib/tokenHealthCheck.ts` runs a background scheduler:
|
||||
|
||||
- **Sweep tick**: every 60 seconds (sweep in `TICK_MS = 60 * 1000` at `src/lib/tokenHealthCheck.ts:30`)
|
||||
- **Per-connection health check interval**: default 60 minutes (`DEFAULT_HEALTH_CHECK_INTERVAL_MIN = 60`); configurable via the settings DB
|
||||
- **Pre-emptive refresh on 401**: handled by the per-connection interceptor
|
||||
|
||||
### Token Health Status
|
||||
|
||||
```ts
|
||||
interface TokenHealth {
|
||||
connectionId: string;
|
||||
provider: string;
|
||||
status: "valid" | "expiring_soon" | "expired" | "refresh_failed";
|
||||
expiresAt: string;
|
||||
lastRefresh: string;
|
||||
nextRefresh: string;
|
||||
consecutiveFailures: number;
|
||||
}
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
Token health check configuration is handled internally by `tokenHealthCheck.ts`.
|
||||
|
||||
### Token Health
|
||||
|
||||
> **No REST endpoint.** Token health data is available via the dashboard or the MCP tool `observability_snapshot`.
|
||||
|
||||
---
|
||||
|
||||
## Alerting
|
||||
|
||||
### Built-in Channels
|
||||
|
||||
OmniRoute supports **3 alert channels**:
|
||||
|
||||
| Channel | Setup | Use case |
|
||||
|---------|-------|----------|
|
||||
| Dashboard banner | Always on | In-app notifications |
|
||||
| Webhook | Configure URL | Slack, Discord, PagerDuty |
|
||||
| Log | Default | For external log aggregation |
|
||||
|
||||
### Webhook Configuration
|
||||
|
||||
> **Note:** Webhook alerting configuration is handled via the dashboard Settings page. See the Settings UI for webhook URL, event filtering, and payload customization.
|
||||
|
||||
### Alert Types
|
||||
|
||||
| Alert | When | Default severity |
|
||||
|-------|------|------------------|
|
||||
| `provider_circuit_open` | Circuit opens | critical |
|
||||
| `provider_circuit_half_open` | Circuit testing recovery | info |
|
||||
| `quota_warning` | Quota at 80%+ | warning |
|
||||
| `quota_exhausted` | Quota at 100% | critical |
|
||||
| `token_refresh_failed` | 3+ consecutive refresh failures | warning |
|
||||
| `token_expired` | Token past expiry | critical |
|
||||
| `combo_target_unhealthy` | Combo target in cooldown for 1h+ | warning |
|
||||
| `db_integrity_warning` | FK violations > 0 | warning |
|
||||
| `heap_pressure` | Heap usage > 80% of threshold | warning |
|
||||
|
||||
---
|
||||
|
||||
## Performance Metrics
|
||||
|
||||
### Tracked Metrics
|
||||
|
||||
| Metric | Type | Source |
|
||||
|--------|------|--------|
|
||||
| `request_count` | counter | `services/usage.ts` |
|
||||
| `request_latency_ms` | histogram | `services/usage.ts` |
|
||||
| `tokens_consumed` | counter | `services/usage.ts` |
|
||||
| `cost_usd` | counter | `services/usage.ts` |
|
||||
| `provider_errors` | counter | `services/errorClassifier.ts` |
|
||||
| `circuit_state_changes` | counter | `services/resilience.ts` |
|
||||
| `cache_hits` | counter | `services/signatureCache.ts` |
|
||||
| `compression_savings` | histogram | `services/compression/stats.ts` |
|
||||
| `quota_used` | gauge | `services/quotaMonitor.ts` |
|
||||
| `memory_used_mb` | gauge | `observability.ts` |
|
||||
|
||||
### Latency Percentiles (p50/p95/p99)
|
||||
|
||||
> **No REST endpoint.** Latency percentile data is available via the dashboard `/dashboard/health` page. Prometheus/OpenTelemetry export is planned for v3.9.
|
||||
|
||||
### Prometheus / OpenTelemetry Export (Phase 2)
|
||||
|
||||
Planned for v3.9: native export to Prometheus, OpenTelemetry, Datadog.
|
||||
|
||||
For now, scrape `/api/monitoring/health` with any HTTP-based monitoring system (Prometheus blackbox exporter, Datadog HTTP check, etc.).
|
||||
|
||||
---
|
||||
|
||||
## Alerting Recipes
|
||||
|
||||
### Slack
|
||||
> **Note:** Webhook alerting is configured through the dashboard Settings page — there are no dedicated webhook env vars (`grep -rn` returns zero hits). See the Settings UI for webhook URL, event filtering, and payload customization.
|
||||
### Discord
|
||||
> Webhook alerting uses the same Settings UI flow as Slack. Discord accepts the same JSON payload shape.
|
||||
### PagerDuty
|
||||
> Webhook alerting uses the same Settings UI flow. PagerDuty Events API v2 routing keys are configured in the Settings UI.
|
||||
### Custom Webhook (JSON)
|
||||
> Any HTTP endpoint that accepts POST with JSON body will work. Configure the URL in the Settings UI.
|
||||
|
||||
---
|
||||
|
||||
## Dashboard Configuration
|
||||
|
||||
### Customize the Health Dashboard
|
||||
|
||||
Create a `~/.omniroute/dashboard.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"health": {
|
||||
"sections": [
|
||||
"server_status",
|
||||
"database",
|
||||
"providers",
|
||||
"quota_monitors",
|
||||
"recent_errors"
|
||||
],
|
||||
"refresh_interval_ms": 5000
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Pin a Provider to the Top
|
||||
|
||||
```json
|
||||
{
|
||||
"health": {
|
||||
"pinned_providers": ["openai", "anthropic"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Provider says healthy but requests fail"
|
||||
|
||||
1. Check the **autopilot issues** — maybe a model is locked out
|
||||
2. Look at **recent errors** for the specific error class
|
||||
3. Try the **connection test** in the provider card
|
||||
4. Check if the provider is **rate-limited at upstream** (not visible locally)
|
||||
|
||||
### "Quota says healthy but I see 429s"
|
||||
|
||||
- 429 means the provider says you've used your quota
|
||||
- OmniRoute's quota tracking may be **stale** — the provider's truth is upstream
|
||||
- Quota data refreshes automatically via the internal quota monitor
|
||||
|
||||
### "Combo is failing but all targets look healthy"
|
||||
|
||||
- Check **combo health** dashboard for target ordering issues
|
||||
- Look at **fallback events** — maybe the combo is exhausting too quickly
|
||||
- Verify the **strategy** matches your use case (priority vs round-robin vs auto)
|
||||
|
||||
### "Database health check is failing"
|
||||
|
||||
- Run `sqlite3 ~/.omniroute/storage.sqlite "PRAGMA integrity_check;"`
|
||||
- If "ok" — false alarm, the health check is being too strict
|
||||
- If anything else — **stop OmniRoute** and follow the [disaster recovery guide](./DATABASE_GUIDE.md#disaster-recovery)
|
||||
|
||||
### "Memory heap pressure is critical"
|
||||
|
||||
```bash
|
||||
# Check current heap
|
||||
node -e "console.log(process.memoryUsage())"
|
||||
|
||||
# Trigger manual GC (if --expose-gc)
|
||||
node --expose-gc -e "global.gc(); console.log(process.memoryUsage())"
|
||||
|
||||
# Reduce concurrent requests (set via the dashboard Settings page, not an env var)
|
||||
# There is no `MAX_CONCURRENT_REQUESTS` env var — configure it in Settings → Concurrency.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [USAGE_QUOTA_GUIDE.md](../features/USAGE_QUOTA_GUIDE.md) — usage & cost tracking
|
||||
- [DATABASE_GUIDE.md](./DATABASE_GUIDE.md) — DB schema + health
|
||||
- [PROXY_GUIDE.md](./PROXY_GUIDE.md) — proxy health (separate cache)
|
||||
- [ARCHITECTURE.md](../architecture/ARCHITECTURE.md) — system architecture
|
||||
- [RESILIENCE_GUIDE.md](../architecture/RESILIENCE_GUIDE.md) — circuit breaker details
|
||||
- Source: `src/lib/monitoring/` (4 files, 2121 LOC)
|
||||
564
Open-SSE-Architecture.md
Normal file
564
Open-SSE-Architecture.md
Normal file
@@ -0,0 +1,564 @@
|
||||
> 🌍 [View in other languages](Languages)
|
||||
|
||||
|
||||
# open-sse Architecture
|
||||
|
||||
> **TL;DR**: `open-sse/` is the core streaming engine that powers every LLM request in OmniRoute. It contains ~406 files implementing the request pipeline, executors, services, MCP server, and translation layer. This guide explains how the pieces fit together.
|
||||
|
||||
**Source:** `open-sse/` (workspace package, ~143K LOC across 406 files)
|
||||
|
||||
---
|
||||
|
||||
## Why a Separate Workspace Package?
|
||||
|
||||
`open-sse/` is a **standalone workspace** in the OmniRoute monorepo for several reasons:
|
||||
|
||||
1. **Reusability** — `open-sse` is published as `@omniroute/open-sse` on npm, so other projects can use it independently
|
||||
2. **Clean boundaries** — the streaming engine is decoupled from the OmniRoute-specific UI/DB layer
|
||||
3. **Performance** — the engine has no Next.js dependencies, enabling faster cold starts in CLI/serverless contexts
|
||||
4. **Versioning** — `open-sse` can release on its own cadence
|
||||
|
||||
```json
|
||||
// package.json
|
||||
"workspaces": ["open-sse"]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Top-Level Structure
|
||||
|
||||
```
|
||||
open-sse/
|
||||
├── index.ts # Public entry point
|
||||
├── types.d.ts # Public type exports
|
||||
├── package.json # @omniroute/open-sse
|
||||
├── config/ # Provider configs, constants, registries
|
||||
├── executors/ # Per-provider HTTP executors (59 files)
|
||||
├── handlers/ # Request handlers (chatCore, responses, etc.)
|
||||
├── lib/ # Internal utilities
|
||||
├── mcp-server/ # Model Context Protocol server
|
||||
├── services/ # ~114 service modules
|
||||
├── transformer/ # Responses API format transformer
|
||||
├── translator/ # Format translation (OpenAI ↔ Claude ↔ Gemini)
|
||||
└── utils/ # Shared utilities (logging, error, stream, etc.)
|
||||
```
|
||||
|
||||
### Module Counts
|
||||
|
||||
| Directory | Files | Purpose |
|
||||
| `executors/` | 62 | Per-provider HTTP executors (unified via DefaultExecutor factory) |
|
||||
| `handlers/` | ~15 | Request entry points (chatCore, responses, embeddings) |
|
||||
| `services/` | ~114 | Routing, caching, rate limiting, refresh, etc. |
|
||||
| `translator/` | ~10 | Format conversion (OpenAI ↔ Claude ↔ Gemini) |
|
||||
| `mcp-server/` | 30 | MCP tools and transports |
|
||||
| `utils/` | ~30 | Cross-cutting utilities (logging, error, stream) |
|
||||
| `config/` | ~10 | Provider configs, constants, registries |
|
||||
|
||||
---
|
||||
|
||||
## The Request Pipeline
|
||||
|
||||
Every LLM request flows through a **5-stage pipeline**:
|
||||
|
||||
```
|
||||
┌──────────────┐
|
||||
HTTP request │ 1. ROUTE │ combo resolution, model selection
|
||||
(Next.js route) └──────┬───────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ 2. TRANSLATE│ format conversion (OpenAI ↔ Claude ↔ Gemini)
|
||||
└──────┬───────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ 3. EXECUTE │ provider executor, HTTP, retry, breaker
|
||||
└──────┬───────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ 4. STREAM │ SSE transformation, backpressure
|
||||
└──────┬───────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ 5. RECORD │ usage tracking, call log, error classification
|
||||
└──────┬───────┘
|
||||
│
|
||||
▼
|
||||
HTTP response (SSE or JSON)
|
||||
```
|
||||
|
||||
### Stage 1: Route (services/combo.ts)
|
||||
|
||||
**Entry point**: `handleComboChat()` in `services/combo.ts`
|
||||
|
||||
Resolves the request to a concrete `(provider, model, account, credentials)` tuple:
|
||||
|
||||
- Look up the combo by ID (or build a virtual combo for `auto/*` models)
|
||||
- Apply routing strategy (priority, weighted, round-robin, etc.)
|
||||
- Filter out unhealthy providers (circuit breaker)
|
||||
- Pick the next viable target
|
||||
|
||||
For `auto/*` models, this stage also:
|
||||
- Runs the **9-factor scoring** algorithm (`services/autoCombo/`)
|
||||
- Selects a `provider+model` pair based on health, cost, latency, etc.
|
||||
|
||||
### Stage 2: Translate (translator/)
|
||||
|
||||
If the source format (e.g., OpenAI) differs from the target format (e.g., Claude), the request is **translated**:
|
||||
|
||||
- System prompt → system message
|
||||
- Tool definitions → provider-specific tool format
|
||||
- Reasoning/thinking parameters → provider-specific equivalents
|
||||
- Message role normalization (`developer` → `system` for non-OpenAI)
|
||||
|
||||
The `translator/index.ts` exposes:
|
||||
|
||||
```ts
|
||||
translateRequest(body, sourceFormat, targetFormat): TranslatedRequest
|
||||
needsTranslation(source, target): boolean
|
||||
```
|
||||
|
||||
### Stage 3: Execute (executors/)
|
||||
|
||||
**Entry point**: `getExecutor(providerId).execute(request, options)`
|
||||
|
||||
All providers use `DefaultExecutor` (`executors/default.ts`) via the `getExecutor()` factory fallback. The executor:
|
||||
|
||||
- Builds the upstream URL (`buildUrl()`)
|
||||
- Adds provider-specific headers (`buildHeaders()`)
|
||||
- Transforms the request body (`transformRequest()`)
|
||||
- Sends the HTTP request with retry + exponential backoff
|
||||
- Handles auth refresh if needed (OAuth providers)
|
||||
|
||||
All executors extend `BaseExecutor` (`executors/base.ts`, 1170 LOC) which provides:
|
||||
- Common retry logic
|
||||
- Proxy integration
|
||||
- Circuit breaker integration
|
||||
- Usage recording hooks
|
||||
|
||||
### Stage 4: Stream (utils/stream.ts)
|
||||
|
||||
For streaming responses, the executor returns a **ReadableStream**. The handler:
|
||||
|
||||
- Pipes through an SSE transform (`createSSETransformStreamWithLogger`)
|
||||
- Applies heartbeat pings to detect dead connections
|
||||
- Handles client disconnect gracefully (`pipeWithDisconnect`)
|
||||
- Transforms SSE → JSON for non-streaming clients
|
||||
|
||||
For non-streaming responses, the executor returns a parsed JSON object that is passed through unchanged.
|
||||
|
||||
### Stage 5: Record (services/usage.ts)
|
||||
|
||||
After the response (success or failure), usage is recorded:
|
||||
|
||||
- `prompt_tokens`, `completion_tokens`, `cached_tokens` from the response
|
||||
- `cost_usd` computed from pricing data
|
||||
- `latency_ms`, `status`, `error_class` if failed
|
||||
- Persisted to `usage_history` table
|
||||
|
||||
Call log artifacts (if enabled) are written to `${DATA_DIR}/call_logs/`.
|
||||
|
||||
---
|
||||
|
||||
## Key Files Deep-Dive
|
||||
|
||||
### chatCore.ts (5977 lines)
|
||||
|
||||
The **main request handler**. Despite its size, it has a clear structure:
|
||||
|
||||
```ts
|
||||
// Pseudo-structure of chatCore.ts
|
||||
export async function handleChat(request: NextRequest) {
|
||||
// 1. Auth + CORS
|
||||
await authenticateRequest(request);
|
||||
applyCorsHeaders(response);
|
||||
|
||||
// 2. Body validation
|
||||
const body = await parseRequestBody(request);
|
||||
|
||||
// 3. Format detection + translation
|
||||
const sourceFormat = detectFormat(request);
|
||||
const targetFormat = getTargetFormat(providerId);
|
||||
if (needsTranslation(sourceFormat, targetFormat)) {
|
||||
body = translateRequest(body, sourceFormat, targetFormat);
|
||||
}
|
||||
|
||||
// 4. Combo routing
|
||||
const targets = await resolveComboTargets(comboId, body);
|
||||
for (const target of targets) {
|
||||
try {
|
||||
const result = await executeOnTarget(target, body);
|
||||
await recordUsage(result);
|
||||
return result;
|
||||
} catch (err) {
|
||||
// Continue to next target
|
||||
}
|
||||
}
|
||||
|
||||
// 5. Emergency fallback
|
||||
return await emergencyFallback(body);
|
||||
}
|
||||
```
|
||||
|
||||
Despite being one giant function, it's organized into **commented sections** that map to the 5-stage pipeline.
|
||||
|
||||
### combo.ts (4456 LOC)
|
||||
|
||||
The **routing engine** that resolves a combo to ordered targets.
|
||||
|
||||
```ts
|
||||
// services/combo.ts
|
||||
export async function handleComboChat(body, comboId): Promise<ChatResult> {
|
||||
const targets = await resolveComboTargets(comboId, body);
|
||||
for (const target of targets) {
|
||||
try {
|
||||
return await handleSingleModel(target, body);
|
||||
} catch (err) {
|
||||
log.warn("target failed, trying next", { target, err });
|
||||
}
|
||||
}
|
||||
throw new ComboExhaustedError("All targets failed");
|
||||
}
|
||||
```
|
||||
|
||||
Supports **15 routing strategies** (see `src/shared/constants/routingStrategies.ts`):
|
||||
|
||||
| Strategy | Behavior |
|
||||
|----------|----------|
|
||||
| `priority` | First-target ordered list |
|
||||
| `weighted` | Probabilistic by per-target weight |
|
||||
| `round-robin` | Cycle through targets in order |
|
||||
| `context-relay` | Hand off context across targets |
|
||||
| `fill-first` | Fill quota before moving to next |
|
||||
| `p2c` | Power of two choices |
|
||||
| `random` | Uniform random |
|
||||
| `least-used` | Pick the one with fewest recent uses |
|
||||
| `cost-optimized` | Cheapest healthy target first |
|
||||
| `reset-aware` | Aware of provider reset windows |
|
||||
| `reset-window` | Reset window-based routing |
|
||||
| `strict-random` | Truly uniform (no quality weighting) |
|
||||
| `auto` | Use 9-factor scoring (`autoCombo/`) |
|
||||
| `lkgp` | Last known good provider first |
|
||||
| `context-optimized` | Best for long-context requests |
|
||||
|
||||
### base.ts (1170 LOC)
|
||||
|
||||
The **abstract executor** that all 59 executors extend. It contains:
|
||||
|
||||
- `buildUrl()` — default URL construction (subclasses override for custom)
|
||||
- `buildHeaders()` — default headers (auth, content-type)
|
||||
- `transformRequest()` — pass-through by default
|
||||
- `execute()` — the main HTTP loop with retry/backoff/breaker
|
||||
|
||||
```ts
|
||||
// open-sse/executors/default.ts
|
||||
export class DefaultExecutor extends BaseExecutor {
|
||||
// Handles all OpenAI/Anthropic-compatible providers
|
||||
// Providers register configurations (URL, auth, headers) but share executor logic
|
||||
}
|
||||
```
|
||||
|
||||
Provider-specific behavior (auth headers, base URL, version headers) is configured via the provider registry, not separate executor classes.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Services (117 modules)
|
||||
|
||||
Services are **focused, single-purpose modules** that handlers compose. The big categories:
|
||||
|
||||
### Routing & Combo
|
||||
|
||||
- `combo.ts` — entry point for combo-routed requests
|
||||
- `services/autoCombo/` — 9-factor scoring, 8 auto routing strategies
|
||||
- `wildcardRouter.ts` — matches wildcard routes (`gpt-*`)
|
||||
- `modelFamilyFallback.ts` — T5 intra-family fallback
|
||||
|
||||
### Rate Limiting & Quota
|
||||
|
||||
- `rateLimitManager.ts` — token bucket per key+provider
|
||||
- `usage.ts` — usage recording
|
||||
- `quotaCache.ts` — in-memory quota snapshots
|
||||
|
||||
### Account & Token
|
||||
|
||||
- `tokenRefresh.ts` — OAuth refresh on 401
|
||||
- `accountFallback.ts` — switch to alternate account
|
||||
- `sessionManager.ts` — multi-turn session state
|
||||
|
||||
### Intelligence
|
||||
|
||||
- `intentClassifier.ts` — classify request intent
|
||||
- `taskAwareRouter.ts` — route by task type
|
||||
- `thinkingBudget.ts` — allocate thinking tokens
|
||||
- `contextManager.ts` — inject routing context
|
||||
|
||||
### Resilience
|
||||
|
||||
- `resilience.ts` — retry, backoff, breaker orchestration
|
||||
- `emergencyFallback.ts` — last-resort fallback
|
||||
- `modelDeprecation.ts` — auto-route to successor models
|
||||
|
||||
### State
|
||||
|
||||
- `signatureCache.ts` — dedup by request signature
|
||||
- `volumeDetector.ts` — load shedding
|
||||
- `contextHandoff.ts` — session serialization
|
||||
|
||||
### Compression
|
||||
|
||||
- `compression/` (subdirectory) — full compression pipeline
|
||||
- 39 files covering engines, rule packs, adapters
|
||||
|
||||
### Skills
|
||||
|
||||
- (covered in [SKILLS.md](./SKILLS.md))
|
||||
|
||||
### Memory
|
||||
|
||||
- (covered in [MEMORY.md](./MEMORY.md))
|
||||
|
||||
---
|
||||
|
||||
## Executors (75+ files)
|
||||
|
||||
One file per provider. They all extend `BaseExecutor` and override what differs.
|
||||
|
||||
### Common Patterns
|
||||
|
||||
Providers are resolved via `getExecutor(providerId)`, which returns the configured executor. OpenAI/Anthropic-compatible providers use `DefaultExecutor` (`executors/default.ts`). Provider-specific behavior (base URL, auth headers, API version) is configured in `open-sse/config/providers/`, while request body transformations are handled in `open-sse/translator/`.
|
||||
|
||||
**Custom URL** is set via provider configuration:
|
||||
|
||||
```ts
|
||||
// Provider config in open-sse/config/providers/
|
||||
export default {
|
||||
id: "together",
|
||||
baseURL: "https://api.together.xyz/v1/chat/completions",
|
||||
}
|
||||
```
|
||||
|
||||
**Custom auth** is handled through the provider registry's auth configuration (API key, OAuth, header profiles).
|
||||
|
||||
**Custom request body** transformations (e.g., Anthropic separating `system` from `messages`) are registered per-provider in `open-sse/translator/`.
|
||||
```
|
||||
|
||||
### The Executor Factory
|
||||
|
||||
`executors/index.ts` exports `getExecutor(providerId)`:
|
||||
|
||||
```ts
|
||||
import { getExecutor } from "@omniroute/open-sse/executors";
|
||||
|
||||
const executor = getExecutor("anthropic");
|
||||
const result = await executor.execute({
|
||||
model: "claude-sonnet-4-5",
|
||||
messages: [...],
|
||||
});
|
||||
```
|
||||
|
||||
The factory is generated from `config/providerRegistry.ts` which lists all 212+ providers and their executor class.
|
||||
|
||||
---
|
||||
|
||||
## Translators
|
||||
|
||||
Translate between **3 formats**: OpenAI, Anthropic, Gemini, plus the new Responses API.
|
||||
|
||||
### When Translation Happens
|
||||
|
||||
```ts
|
||||
import { needsTranslation, translateRequest } from "@omniroute/open-sse/translator";
|
||||
|
||||
if (needsTranslation(sourceFormat, targetFormat)) {
|
||||
body = translateRequest(body, sourceFormat, targetFormat);
|
||||
}
|
||||
```
|
||||
|
||||
Common translations:
|
||||
- `OpenAI → Anthropic`: separate `system` field, `x-api-key` header
|
||||
- `OpenAI → Gemini`: `contents` instead of `messages`, `systemInstruction`
|
||||
- `OpenAI → Responses API`: `input` array, `previous_response_id` state
|
||||
|
||||
### Edge Cases Handled
|
||||
|
||||
- `developer` role → `system` for non-OpenAI
|
||||
- `system` role → merged into first user message for GLM/ERNIE
|
||||
- `json_schema` → Gemini's `responseMimeType` + `responseSchema`
|
||||
- `tools` → provider-specific tool format
|
||||
- Thinking parameters (o1, Claude) → provider-specific equivalents
|
||||
|
||||
---
|
||||
|
||||
## MCP Server
|
||||
|
||||
`open-sse/mcp-server/` implements the **Model Context Protocol** server:
|
||||
|
||||
- **30+ tools** (provider management, combos, memory, cache, compression, 1proxy, skills)
|
||||
- **3 transports**: stdio, SSE, Streamable HTTP
|
||||
- **13 scopes** for fine-grained authorization
|
||||
### Tool Registration
|
||||
|
||||
Tools are registered as standalone files in `open-sse/mcp-server/tools/`, each exporting a name, schema, handler, and scope:
|
||||
|
||||
```ts
|
||||
// open-sse/mcp-server/tools/getHealth.ts
|
||||
import { z } from "zod";
|
||||
export default {
|
||||
name: "omniroute_get_health",
|
||||
description: "Get system health snapshot",
|
||||
scope: "read:health",
|
||||
inputSchema: z.object({}),
|
||||
handler: async (_args, ctx) => {
|
||||
return await getSystemHealth();
|
||||
},
|
||||
};
|
||||
```
|
||||
|
||||
### Transports
|
||||
|
||||
```ts
|
||||
// stdio (CLI usage)
|
||||
startMcpStdio(server);
|
||||
|
||||
// SSE (HTTP-based streaming)
|
||||
startMcpSse(server, port);
|
||||
|
||||
// Streamable HTTP (modern MCP)
|
||||
startMcpStreamable(server, port);
|
||||
```
|
||||
|
||||
### Authorization
|
||||
|
||||
Every tool call goes through scope checks (`open-sse/mcp-server/auth/`):
|
||||
|
||||
```ts
|
||||
if (!hasScope(apiKey, "providers:read")) {
|
||||
throw new Error("Insufficient scope");
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Transformers
|
||||
|
||||
`open-sse/transformer/` converts between **Chat Completions** and **Responses API** formats.
|
||||
|
||||
### Why a Separate Transformer?
|
||||
|
||||
The Responses API is OpenAI's new format with **stateful conversations** (`previous_response_id`). When a client sends a Responses request, OmniRoute:
|
||||
|
||||
1. Converts Responses → Chat Completions internally
|
||||
2. Sends to provider (any provider that supports Chat Completions)
|
||||
3. Converts the response back to Responses format
|
||||
4. Streams the converted response to the client
|
||||
|
||||
The transformer (`transformer/responsesTransformer.ts`) provides:
|
||||
|
||||
```ts
|
||||
createResponsesApiTransformStream(): TransformStream
|
||||
```
|
||||
|
||||
This handles:
|
||||
- `response.output_item.added` events
|
||||
- `response.output_text.delta` events
|
||||
- `response.completed` event
|
||||
- Tool call mapping (`function_call` ↔ `tool_calls`)
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
`open-sse/config/` holds the configuration layer:
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `providerRegistry.ts` | 212+ provider definitions |
|
||||
| `providerModels.ts` | Model aliases, format mapping |
|
||||
| `constants.ts` | Timeouts, limits, status codes |
|
||||
| `defaultThinkingSignature.ts` | Default Claude thinking signature |
|
||||
| `modelStrip.ts` (in services) | Per-provider field stripping |
|
||||
|
||||
### Provider Registry Schema
|
||||
|
||||
```ts
|
||||
interface ProviderConfig {
|
||||
id: string;
|
||||
name: string;
|
||||
baseUrl: string;
|
||||
authType: "bearer" | "api-key" | "oauth" | "cookie";
|
||||
executorClass: string;
|
||||
defaultModel: string;
|
||||
capabilities: ProviderCapabilities;
|
||||
models: ModelDefinition[];
|
||||
}
|
||||
```
|
||||
|
||||
Zod validation at module load ensures all provider configs are valid.
|
||||
|
||||
---
|
||||
|
||||
## Performance Constraints
|
||||
|
||||
The routing engine has strict performance budgets:
|
||||
|
||||
| Operation | Target | Measurement |
|
||||
|-----------|--------|-------------|
|
||||
| Combo resolution | <10ms | For 50 targets |
|
||||
| Rate limit check | <1ms | In-memory token bucket |
|
||||
| Model family fallback | <5ms | Cached family definitions |
|
||||
| Request routing dispatch | <2ms | Hot path |
|
||||
| **No blocking I/O in routing hot path** | — | All async |
|
||||
|
||||
---
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
❌ **Synchronous DB calls in `combo.ts`** — pre-compute and cache
|
||||
❌ **Retry logic in handlers** — use `retry()` from resilience service
|
||||
❌ **Direct provider config access** — use `providerRegistry` getters
|
||||
❌ **Hardcoded fallback chains** — define in `modelFamilyFallback.ts`
|
||||
❌ **State mutations across concurrent requests** — use request-scoped context only
|
||||
|
||||
---
|
||||
|
||||
## Adding a New Component
|
||||
|
||||
### Adding a New Service
|
||||
|
||||
1. Create `open-sse/services/[serviceName].ts` with focused responsibility
|
||||
2. Export main handler function and any constants
|
||||
3. Add unit tests in `tests/unit/services/[serviceName].test.mjs`
|
||||
4. Integrate into request pipeline in `handlers/chatCore.ts` (if routing-related)
|
||||
5. Update routing logic in `combo.ts` if service affects target selection
|
||||
6. Document in this file
|
||||
|
||||
### Adding a New Executor
|
||||
|
||||
1. Create `open-sse/executors/[provider].ts` extending `BaseExecutor`
|
||||
2. Register in `config/providerRegistry.ts`
|
||||
3. Add to `executors/index.ts` factory
|
||||
4. Add unit tests for the executor
|
||||
5. Document in `docs/architecture/ARCHITECTURE.md`
|
||||
|
||||
### Adding a New MCP Tool
|
||||
|
||||
1. Create or update `open-sse/mcp-server/tools/[category]Tools.ts`
|
||||
2. Define Zod schema for inputs
|
||||
3. Register tool in `mcp-server/index.ts`
|
||||
4. Add to scope matrix in `mcp-server/auth/`
|
||||
5. Add unit tests
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [ARCHITECTURE.md](../architecture/ARCHITECTURE.md) — high-level architecture
|
||||
- [CODEBASE_DOCUMENTATION.md](../architecture/CODEBASE_DOCUMENTATION.md) — engineering reference
|
||||
- [REPOSITORY_MAP.md](../architecture/REPOSITORY_MAP.md) — directory-by-directory
|
||||
- [AUTO-COMBO.md](../routing/AUTO-COMBO.md) — 9-factor scoring
|
||||
- [MCP-SERVER.md](./MCP-SERVER.md) — MCP server
|
||||
- [A2A-SERVER.md](./A2A-SERVER.md) — A2A server
|
||||
- Source: `open-sse/` (400+ files, ~143K LOC)
|
||||
427
Usage-Quota-Guide.md
Normal file
427
Usage-Quota-Guide.md
Normal file
@@ -0,0 +1,427 @@
|
||||
> 🌍 [View in other languages](Languages)
|
||||
|
||||
|
||||
# Usage, Quota & Spend Tracking
|
||||
|
||||
> **TL;DR**: OmniRoute tracks every request's token usage, computes cost, enforces per-API-key quota, and surfaces analytics in the dashboard. This guide explains how it all works.
|
||||
|
||||
**Sources:**
|
||||
- `open-sse/services/usage.ts` (~70KB) — main usage tracking
|
||||
- `src/lib/usageAnalytics.ts` (~10KB) — aggregation for dashboard
|
||||
- `src/lib/db/quotaSnapshots.ts` — historical quota data
|
||||
- `src/lib/db/usage*.ts` — multiple usage-related DB modules
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Every request that flows through OmniRoute generates a **usage record** that captures:
|
||||
|
||||
- **Identity**: which API key, provider, model, combo
|
||||
- **Tokens**: prompt tokens, completion tokens, cached tokens, total
|
||||
- **Cost**: USD amount (computed from pricing data)
|
||||
- **Timing**: latency, start/end timestamps
|
||||
- **Status**: success, error, rate-limited, etc.
|
||||
|
||||
These records are aggregated into **analytics**, persisted as **quota snapshots**, and used to enforce **per-key budget limits**.
|
||||
|
||||
```
|
||||
Request ──▶ chatCore ──▶ usage.record() ──▶ SQLite
|
||||
│
|
||||
┌───────┼───────┐
|
||||
▼ ▼ ▼
|
||||
analytics quota billing
|
||||
(dashboard) (enforce) (export)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## What Gets Recorded
|
||||
|
||||
The `usage.ts` service captures a **usage event** for every request:
|
||||
|
||||
| Field | Type | Source |
|
||||
|-------|------|--------|
|
||||
| `id` | string | UUID generated on record |
|
||||
| `apiKeyId` | string | The API key that initiated the request |
|
||||
| `provider` | string | Provider ID (openai, anthropic, etc.) |
|
||||
| `model` | string | Model ID (gpt-5, claude-opus-4-6, etc.) |
|
||||
| `comboId` | string? | Combo ID if routed through a combo |
|
||||
| `promptTokens` | number | From upstream response |
|
||||
| `completionTokens` | number | From upstream response |
|
||||
| `cachedTokens` | number | Cache hit tokens (Anthropic prompt caching, etc.) |
|
||||
| `totalTokens` | number | prompt + completion |
|
||||
| `costUsd` | number | Computed from pricing data |
|
||||
| `latencyMs` | number | End-to-end request duration |
|
||||
| `status` | enum | `success`, `error`, `rate_limited`, `timeout`, `cancelled` |
|
||||
| `errorClass` | string? | Error class if status != success |
|
||||
| `timestamp` | string | ISO 8601 UTC |
|
||||
| `metadata` | object | Custom plugin-injected data |
|
||||
|
||||
### Where Tokens Come From
|
||||
|
||||
Tokens are extracted from the upstream provider's response in the **response handler**:
|
||||
|
||||
```ts
|
||||
// From open-sse/handlers/chatCore.ts
|
||||
const response = await providerExecutor.execute(provider, request);
|
||||
const usage = response.usage || {
|
||||
prompt_tokens: 0,
|
||||
completion_tokens: 0,
|
||||
cached_tokens: 0,
|
||||
};
|
||||
```
|
||||
|
||||
For providers that don't return usage (some web-cookie providers), OmniRoute **estimates** tokens using a `~4 chars per token` heuristic (see `open-sse/services/autoCombo/pipelineRouter.ts`).
|
||||
|
||||
### Cached Tokens
|
||||
|
||||
OmniRoute tracks `cached_tokens` separately from `prompt_tokens` because:
|
||||
|
||||
- Anthropic prompt caching charges a reduced rate for cached tokens (10% of normal)
|
||||
- Some providers return `cache_read_input_tokens` that should be priced differently
|
||||
- Analytics can show the **cache hit rate** = `cached_tokens / prompt_tokens`
|
||||
|
||||
---
|
||||
|
||||
## Cost Calculation
|
||||
|
||||
Costs are computed from **pricing data** synced from LiteLLM (`src/lib/pricingSync.ts`):
|
||||
|
||||
| Model | Input $/1M | Output $/1M | Cached $/1M |
|
||||
|-------|-----------|-------------|-------------|
|
||||
| gpt-5 | $2.50 | $10.00 | — |
|
||||
| claude-opus-4-6 | $15.00 | $75.00 | $1.50 |
|
||||
| claude-sonnet-4-5 | $3.00 | $15.00 | $0.30 |
|
||||
| gemini-2.5-pro | $1.25 | $10.00 | — |
|
||||
|
||||
The cost formula (`src/lib/usage/costCalculator.ts`):
|
||||
|
||||
```ts
|
||||
cost = (prompt_tokens - cached_tokens) * input_price
|
||||
+ cached_tokens * cached_price
|
||||
+ completion_tokens * output_price
|
||||
```
|
||||
|
||||
> **Why subtract cached from prompt?** The cached portion is priced separately; charging input price on the whole prompt would over-count.
|
||||
|
||||
### Pricing Sync
|
||||
|
||||
Pricing data is auto-synced from LiteLLM via the `/api/pricing/sync` endpoint (triggered by the built-in cron task, not a user-facing env var):
|
||||
|
||||
```bash
|
||||
# Manual trigger
|
||||
curl -X POST http://localhost:20128/api/pricing/sync
|
||||
```
|
||||
|
||||
For models with no pricing data, OmniRoute falls back to **estimating cost** using internal average rates (sourced from LiteLLM's pricing data).
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Date Range Aggregation
|
||||
|
||||
The `usageAnalytics.ts` module computes dashboard widgets from raw usage data. It supports 7 time ranges:
|
||||
|
||||
| Range | Window | Use case |
|
||||
|-------|--------|----------|
|
||||
| `1d` | Last 24 hours | Hourly cost spike detection |
|
||||
| `7d` | Last 7 days | Weekly review |
|
||||
| `30d` | Last 30 days | Monthly billing |
|
||||
| `90d` | Last 90 days | Quarterly analysis |
|
||||
| `ytd` | Since Jan 1 of current year | Annual budget tracking |
|
||||
| `all` | All time | Lifetime stats |
|
||||
| `custom` | User-defined start/end | Audits, ad-hoc queries |
|
||||
|
||||
### Dashboard Widgets Computed
|
||||
|
||||
For any date range, the analytics layer computes:
|
||||
|
||||
| Widget | Description |
|
||||
|--------|-------------|
|
||||
| **Summary cards** | Total requests, total cost, total tokens, success rate |
|
||||
| **Daily trend chart** | Cost + tokens per day, stacked by model |
|
||||
| **Activity heatmap** | Hour-of-day × day-of-week grid, color = request count |
|
||||
| **Model breakdown** | Pie chart of cost by model |
|
||||
| **Provider breakdown** | Bar chart of requests by provider |
|
||||
| **Top API keys** | Table of top 10 keys by cost |
|
||||
| **Error analysis** | Error rate over time, top error classes |
|
||||
|
||||
### Programmatic Access
|
||||
|
||||
```ts
|
||||
import { computeAnalytics } from "@/lib/usageAnalytics";
|
||||
|
||||
const analytics = await computeAnalytics(
|
||||
history, // usage history records
|
||||
"7d", // time range: "1d" | "7d" | "30d" | "90d" | "ytd" | "all" | "custom"
|
||||
connectionMap, // provider connection map (connectionId → account name)
|
||||
{
|
||||
startDate: "2025-01-01", // optional: for "custom" range
|
||||
endDate: "2025-06-01", // optional: for "custom" range
|
||||
}
|
||||
);
|
||||
|
||||
console.log(analytics.summary.totalCost); // 12.34 (cents)
|
||||
console.log(analytics.byModel[0]); // { model, cost, requests, promptTokens, completionTokens }
|
||||
|
||||
---
|
||||
|
||||
## Quota Enforcement
|
||||
|
||||
Per-API-key quota is enforced in two places:
|
||||
|
||||
1. **Soft limit** (`quotaWarnAt`): dashboard warning when usage exceeds threshold
|
||||
2. **Hard limit** (`quotaLimit`): request rejected with HTTP 429 when exceeded
|
||||
|
||||
### Configuration
|
||||
|
||||
```ts
|
||||
// Per API key
|
||||
await updateApiKey(keyId, {
|
||||
quotaWarnAt: 5_00, // $5.00 — show warning
|
||||
quotaLimit: 10_00, // $10.00 — hard stop
|
||||
quotaWindow: "month", // "day" | "week" | "month" | "all"
|
||||
});
|
||||
```
|
||||
|
||||
### Enforcement Flow
|
||||
|
||||
```
|
||||
Request ──▶ quotaCheck()
|
||||
│
|
||||
├── Within limit? ──▶ allow
|
||||
│
|
||||
└── Over limit? ──▶ 429 Too Many Requests
|
||||
with Retry-After header
|
||||
```
|
||||
|
||||
### Quota Snapshots
|
||||
|
||||
`quotaSnapshots` table stores **historical quota state** for trend analysis:
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| `apiKeyId` | The key being tracked |
|
||||
| `window` | "day" | "week" | "month" |
|
||||
| `used` | Cost used in this window (cents) |
|
||||
| `limit` | The limit (cents) |
|
||||
| `resetAt` | When the window resets |
|
||||
| `createdAt` | When the snapshot was taken |
|
||||
|
||||
Snapshots are taken **on every request** that uses > 0 cost, and used to:
|
||||
|
||||
- Render the quota progress bar in the dashboard
|
||||
- Show 30-day quota trend charts
|
||||
- Trigger alerts when usage approaches the limit
|
||||
|
||||
---
|
||||
|
||||
## REST API
|
||||
|
||||
### List Usage Records
|
||||
|
||||
```bash
|
||||
GET /api/usage?range=7d&limit=100
|
||||
GET /api/usage?apiKeyId=key-123&range=30d
|
||||
GET /api/usage?provider=openai&range=1d
|
||||
```
|
||||
|
||||
Response:
|
||||
|
||||
```json
|
||||
{
|
||||
"records": [
|
||||
{
|
||||
"id": "uuid",
|
||||
"apiKeyId": "key-123",
|
||||
"provider": "openai",
|
||||
"model": "gpt-5",
|
||||
"promptTokens": 1234,
|
||||
"completionTokens": 567,
|
||||
"totalTokens": 1801,
|
||||
"costUsd": 0.0050,
|
||||
"latencyMs": 1234,
|
||||
"status": "success",
|
||||
"timestamp": "2026-06-08T12:00:00Z"
|
||||
}
|
||||
],
|
||||
"total": 1234,
|
||||
"nextCursor": "..."
|
||||
}
|
||||
```
|
||||
|
||||
### Get Analytics Summary
|
||||
|
||||
```bash
|
||||
GET /api/usage/analytics?range=7d&groupBy=model
|
||||
```
|
||||
|
||||
Response:
|
||||
|
||||
```json
|
||||
{
|
||||
"summary": {
|
||||
"totalCost": 12.34,
|
||||
"totalRequests": 5678,
|
||||
"totalTokens": 12345678,
|
||||
"successRate": 0.987,
|
||||
"avgLatencyMs": 1234
|
||||
},
|
||||
"models": [
|
||||
{ "model": "gpt-5", "cost": 8.50, "requests": 1234, "tokens": 4567890 },
|
||||
{ "model": "claude-opus-4-6", "cost": 3.84, "requests": 234, "tokens": 234567 }
|
||||
],
|
||||
"daily": [
|
||||
{ "date": "2026-06-01", "cost": 1.50, "requests": 800 },
|
||||
{ "date": "2026-06-02", "cost": 2.00, "requests": 1000 }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Query Usage Analytics
|
||||
|
||||
Usage data is accessed via the dashboard or MCP tools, not direct REST export endpoints. Available analytics:
|
||||
|
||||
- **`/api/usage/analytics`** — aggregated usage metrics (group by model, provider, key)
|
||||
- **`/api/usage/quota`** — current quota status per API key
|
||||
- **`/api/usage/history`** — request history logs
|
||||
|
||||
---
|
||||
|
||||
## MCP Tools
|
||||
|
||||
Two MCP tools expose usage data to agents (see `open-sse/mcp-server/tools/`):
|
||||
|
||||
| Tool | Description |
|
||||
|------|-------------|
|
||||
| `omniroute_cost_report` | Generates a per-key cost report for a given period |
|
||||
| `omniroute_check_quota` | Returns current quota status for an API key |
|
||||
|
||||
Example agent invocation:
|
||||
|
||||
```json
|
||||
{
|
||||
"tool": "omniroute_cost_report",
|
||||
"args": { "period": "week" }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Retention and Cleanup
|
||||
|
||||
Usage data grows ~1-10KB per request. At scale, this can be significant.
|
||||
|
||||
### Retention Settings
|
||||
|
||||
Usage history retention is configured via the Database Settings in the UI or via `/api/settings/database`.
|
||||
|
||||
By default, usage history is retained for **90 days**.
|
||||
|
||||
### Cleanup
|
||||
|
||||
Old records are cleaned up by `src/lib/db/cleanup.ts`:
|
||||
|
||||
- Triggered by the background cron process
|
||||
- Deletes records from `usage_history` older than the configured `usageHistory` retention setting
|
||||
### Storage Estimation
|
||||
|
||||
| Request rate | 30-day storage | 90-day storage |
|
||||
|--------------|----------------|----------------|
|
||||
| 100 req/day | ~3MB | ~9MB |
|
||||
| 1,000 req/day | ~30MB | ~90MB |
|
||||
| 10,000 req/day | ~300MB | ~900MB |
|
||||
| 100,000 req/day | ~3GB | ~9GB |
|
||||
|
||||
For very high traffic, consider:
|
||||
|
||||
- Reducing the retention period via Database Settings
|
||||
- Using `aggregated_metrics` instead of raw records (only for analytics)
|
||||
|
||||
---
|
||||
|
||||
## Cost Optimization Tips
|
||||
|
||||
### 1. Use the Right Model
|
||||
|
||||
```bash
|
||||
# Quick answer — use cheap + fast
|
||||
curl -d '{"model":"auto/fast","messages":[...]}'
|
||||
|
||||
# Complex task — use quality
|
||||
curl -d '{"model":"auto/smart","messages":[...]}'
|
||||
```
|
||||
|
||||
### 2. Enable Caching
|
||||
|
||||
Anthropic prompt caching saves **90% on repeated context**:
|
||||
|
||||
```ts
|
||||
// The caching is automatic — just include the same large system prompt
|
||||
const response = await openai.chat({
|
||||
model: "claude-sonnet-4-5",
|
||||
system: longSystemPrompt, // Will be cached automatically
|
||||
messages: [{ role: "user", content: "..." }]
|
||||
});
|
||||
```
|
||||
|
||||
### 3. Use Compression
|
||||
|
||||
RTK + Caveman compression saves **15-95% on tool-heavy sessions**:
|
||||
|
||||
```ts
|
||||
const config = {
|
||||
compression: {
|
||||
engine: "rtk",
|
||||
intensity: "aggressive"
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
### 4. Set Per-Key Quotas
|
||||
|
||||
Always set `quotaLimit` to prevent runaway costs:
|
||||
|
||||
```ts
|
||||
await updateApiKey(keyId, { quotaLimit: 10_00 }); // $10/month cap
|
||||
```
|
||||
|
||||
### 5. Audit Top Consumers
|
||||
|
||||
Use the dashboard or **`/api/usage/analytics`** to group by API key and sort by cost:
|
||||
|
||||
```bash
|
||||
GET /api/usage/analytics?groupBy=apiKey
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Cost is higher than expected"
|
||||
|
||||
1. Check **`/api/usage/analytics?groupBy=model`** — find the expensive model
|
||||
2. Check **`/api/usage/analytics?groupBy=apiKey`** — find the heavy consumer
|
||||
3. Verify pricing data is up to date: `POST /api/pricing/sync`
|
||||
|
||||
### "Records missing"
|
||||
- Check DB retention settings under Dashboard → Database → Cleanup — old records are deleted by the periodic cleanup task (`src/lib/db/cleanup.ts`)
|
||||
- Check for errors in `src/lib/db/usage*.ts` — DB write failures are logged but not surfaced
|
||||
- Verify the request actually reached `chatCore` — check combo routing
|
||||
|
||||
### "Quota not enforcing"
|
||||
|
||||
- Check the key's `quotaLimit` setting
|
||||
- Verify `quotaWindow` is set correctly
|
||||
- Look for `quotaSnapshots` records — they should be created on every request
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [DATABASE_GUIDE.md](../ops/DATABASE_GUIDE.md) — Schema for usage tables
|
||||
- [ENVIRONMENT.md](../reference/ENVIRONMENT.md#18-pricing-sync) — pricing sync env vars
|
||||
- [AUTO-COMBO.md](../routing/AUTO-COMBO.md) — How `auto/fast`, `auto/cheap` reduce cost
|
||||
- [API_REFERENCE.md](../reference/API_REFERENCE.md) — Full `/api/usage/*` reference
|
||||
- Source: `open-sse/services/usage.ts`, `src/lib/usageAnalytics.ts`, `src/lib/db/usage*.ts`
|
||||
Reference in New Issue
Block a user