mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-07-26 09:52:11 +03:00
[trim] feat(combo): add context requirements config for target filtering (#6907)
* feat(combo): add context requirements config for target filtering
Add contextRequirements config field to combo runtime config:
- minContextWindow: filter models below threshold (0-10M tokens)
- preferLargeContext: sort targets by context size descending
- contextFilterMode: 'strict' excludes unknown limits, 'lenient' includes them
Implementation:
- Added Zod schema validation in combo.ts
- Created contextRequirements.ts module with applyContextRequirements()
- Integrated filtering after filterTargetsByRequestCompatibility()
- Full test coverage with unit + integration tests
Tests: 17/17 pass (combo-context-requirements.test.ts + integration)
* feat(combo): add ContextRequirementsEditor UI component
Add standalone React component for editing context requirements config:
- Slider for minContextWindow (0 to 1M tokens) with presets
- Toggle for preferLargeContext sorting
- Radio group for contextFilterMode (strict/lenient)
- Tooltips explaining each option
- Active filters summary display
Component features:
- Shadcn UI components (Card, Slider, Switch, RadioGroup)
- Preset buttons for common context sizes (8K, 32K, 128K, 1M)
- Conditional display of filter mode when minContextWindow > 0
- Clear visual feedback of active filters
Integration:
Import and use in combo config form where other config fields
like fusionTuning and judgeModel are edited. Pass combo.config.contextRequirements
as value prop and update on onChange.
Example usage:
<ContextRequirementsEditor
value={config.contextRequirements}
onChange={(val) => updateConfig({ contextRequirements: val })}
/>
UI matches existing combo config editor patterns.
* feat(combo): wire ContextRequirementsEditor into combo config form
Adds context requirements section to combo edit page (strategy section),
matching existing ResponseValidation pattern. Placed after response
validation block, before agent features.
* docs(combo): add context requirements feature documentation
Covers: config schema, behavior, use cases, UI integration,
troubleshooting, and test instructions.
* fix(combo): pass provider+modelStr to getModelContextLimit for accurate context resolution
Per gemini-code-assist review feedback: model names are not globally
unique across providers. Passing both provider and modelStr ensures
correct context limit resolution in applyContextRequirements().
* fix(combo): repair broken doc links and restore test:unit:fast flag
- Point docs/combo-context-requirements.md 'Related' links at real docs
(routing/AUTO-COMBO.md, architecture/RESILIENCE_GUIDE.md) — the three
placeholder links (strategies.md/model-capabilities.md/fusion-tuning.md)
did not exist and failed check:doc-links (Docs Gates fast-path).
- Revert an out-of-scope package.json change to test:unit:fast that dropped
--test-isolation=none; restore to match release/v3.8.47.
Co-authored-by: oyi77 <oyi77@users.noreply.github.com>
* test(combo): validate context requirements against the real Zod schema
Point tests/unit/combo-context-requirements.test.ts at the real
comboRuntimeConfigSchema export (src/shared/validation/schemas/combo.ts)
instead of hand-duplicating the Zod schema inline, so the test catches
schema drift.
Also declare contextRequirements on DEFAULT_COMBO_CONFIG so
resolveComboSetupConfig's inferred return type includes the key —
combo.ts reads config.contextRequirements but the property was missing
from the object typecheck:core infers types from, causing a build error.
Co-authored-by: oyi77 <oyi77@users.noreply.github.com>
* fix(combos): remove dead ContextRequirementsEditor scaffolding (broken ui/card+label imports)
The editor imported @/components/ui/card and @/components/ui/label which do not
exist in the repo, breaking the Turbopack build. Removed the editor + its page.tsx
usage + doc mention; the real fix (comboConfig contextRequirements default + test)
is preserved.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: oyi77 <oyi77@users.noreply.github.com>
Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com>
Co-authored-by: Diego Rodrigues de Sa e Souza <8016841+diegosouzapw@users.noreply.github.com>
This commit is contained in:
262
docs/combo-context-requirements.md
Normal file
262
docs/combo-context-requirements.md
Normal file
@@ -0,0 +1,262 @@
|
||||
# Combo Context Requirements Feature
|
||||
|
||||
## Overview
|
||||
|
||||
The Context Requirements feature allows combo configurations to filter and sort targets based on their context window size. This is useful for use cases requiring large context windows like:
|
||||
|
||||
- Long document processing (100k+ tokens)
|
||||
- Large codebase analysis
|
||||
- Extensive conversation histories
|
||||
- Multi-file code reviews
|
||||
|
||||
## Configuration
|
||||
|
||||
### Schema
|
||||
|
||||
Add `contextRequirements` to your combo's runtime config:
|
||||
|
||||
```json
|
||||
{
|
||||
"contextRequirements": {
|
||||
"minContextWindow": 128000,
|
||||
"preferLargeContext": true,
|
||||
"contextFilterMode": "strict"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Fields
|
||||
|
||||
#### `minContextWindow` (optional)
|
||||
|
||||
- **Type**: `number` (0 to 10,000,000)
|
||||
- **Default**: `undefined` (no filtering)
|
||||
- **Description**: Filters out models with context windows below this threshold
|
||||
|
||||
**Examples**:
|
||||
|
||||
- `32000` - Filter out models with <32K context
|
||||
- `128000` - Require 128K+ context (GPT-4 Turbo, Claude 3)
|
||||
- `200000` - Require 200K+ context (Claude 3 Opus)
|
||||
- `1000000` - Require 1M+ context (Gemini 1.5 Pro)
|
||||
|
||||
#### `preferLargeContext` (optional)
|
||||
|
||||
- **Type**: `boolean`
|
||||
- **Default**: `false`
|
||||
- **Description**: When `true`, sorts remaining targets by context size (descending). Large context models are tried first.
|
||||
|
||||
#### `contextFilterMode` (optional)
|
||||
|
||||
- **Type**: `"strict"` | `"lenient"`
|
||||
- **Default**: `"lenient"`
|
||||
- **Description**: How to handle models with unknown context window limits
|
||||
- `"strict"`: Excludes models with unknown context limits
|
||||
- `"lenient"`: Includes models with unknown context limits
|
||||
|
||||
## Behavior
|
||||
|
||||
### Filtering Pipeline
|
||||
|
||||
Context requirements are applied after `filterTargetsByRequestCompatibility()`:
|
||||
|
||||
1. **Request compatibility filtering** - Removes models incompatible with request (tools, vision, structured output)
|
||||
2. **Context requirements filtering** - Applies `minContextWindow` and `contextFilterMode`
|
||||
3. **Context-based sorting** - If `preferLargeContext` is true, sorts by context size descending
|
||||
|
||||
### Filter Mode Logic
|
||||
|
||||
When `minContextWindow` is set:
|
||||
|
||||
**Lenient mode** (default):
|
||||
|
||||
- ✅ Includes models with context >= minContextWindow
|
||||
- ✅ Includes models with unknown context limits
|
||||
- ❌ Excludes models with context < minContextWindow
|
||||
|
||||
**Strict mode**:
|
||||
|
||||
- ✅ Includes models with context >= minContextWindow
|
||||
- ❌ Excludes models with unknown context limits
|
||||
- ❌ Excludes models with context < minContextWindow
|
||||
|
||||
### Sorting Logic
|
||||
|
||||
When `preferLargeContext` is true:
|
||||
|
||||
- Models are sorted by context window size (descending)
|
||||
- Unknown context models sort to the end
|
||||
- Original strategy order is used as a tiebreaker
|
||||
|
||||
## Use Cases
|
||||
|
||||
### Example 1: Long Document Processing
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "Document Analysis",
|
||||
"strategy": "fusion",
|
||||
"config": {
|
||||
"contextRequirements": {
|
||||
"minContextWindow": 128000,
|
||||
"preferLargeContext": true,
|
||||
"contextFilterMode": "strict"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This configuration:
|
||||
|
||||
- Requires 128K+ context window
|
||||
- Prefers larger context models (Gemini 1.5 Pro > Claude 3 Opus > GPT-4 Turbo)
|
||||
- Excludes models with unknown context limits
|
||||
|
||||
### Example 2: Large Codebase Analysis
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "Code Review",
|
||||
"strategy": "auto",
|
||||
"config": {
|
||||
"contextRequirements": {
|
||||
"minContextWindow": 200000,
|
||||
"preferLargeContext": true,
|
||||
"contextFilterMode": "lenient"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This configuration:
|
||||
|
||||
- Requires 200K+ context window
|
||||
- Prefers larger context models
|
||||
- Includes models with unknown limits (lenient)
|
||||
|
||||
### Example 3: Prefer Large Context Without Strict Requirements
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "Flexible Chat",
|
||||
"strategy": "weighted",
|
||||
"config": {
|
||||
"contextRequirements": {
|
||||
"preferLargeContext": true
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This configuration:
|
||||
|
||||
- No minimum requirement (all models eligible)
|
||||
- Sorts by context size (largest first)
|
||||
- Useful when large context is preferred but not required
|
||||
|
||||
## API Response
|
||||
|
||||
When context requirements filter targets, the combo logger outputs:
|
||||
|
||||
```
|
||||
[COMBO] Context requirements: filtered 10 → 3 targets (minContextWindow: 128000, mode: strict)
|
||||
[COMBO] Context requirements: kept models gemini-1.5-pro, claude-3-opus-20240229, gpt-4-turbo
|
||||
[COMBO] Context requirements: sorted by context size (descending): gemini-1.5-pro(1000000), claude-3-opus-20240229(200000), gpt-4-turbo(128000)
|
||||
```
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### Backend Module
|
||||
|
||||
`open-sse/services/combo/contextRequirements.ts`:
|
||||
|
||||
- `applyContextRequirements()` - Main filtering function
|
||||
- `getTargetContextWindow()` - Context lookup helper
|
||||
- Uses `getModelContextLimit()` from `modelCapabilities.ts`
|
||||
|
||||
### Integration Point
|
||||
|
||||
`open-sse/services/combo.ts` line 1187:
|
||||
|
||||
```typescript
|
||||
orderedTargets = filterTargetsByRequestCompatibility(orderedTargets, body, log);
|
||||
orderedTargets = applyContextRequirements(orderedTargets, config.contextRequirements, log);
|
||||
```
|
||||
|
||||
### Schema Definition
|
||||
|
||||
`src/shared/validation/schemas/combo.ts`:
|
||||
|
||||
```typescript
|
||||
contextRequirements: z
|
||||
.object({
|
||||
minContextWindow: z.coerce.number().int().min(0).max(10_000_000).optional(),
|
||||
preferLargeContext: z.boolean().optional(),
|
||||
contextFilterMode: z.enum(["strict", "lenient"]).optional(),
|
||||
})
|
||||
.strict()
|
||||
.optional(),
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
### Run Tests
|
||||
|
||||
```bash
|
||||
# Unit tests (schema + logic)
|
||||
npm test tests/unit/combo-context-requirements.test.ts
|
||||
|
||||
# Integration tests (end-to-end)
|
||||
npm test tests/unit/combo/context-requirements-integration.test.ts
|
||||
```
|
||||
|
||||
### Test Coverage
|
||||
|
||||
- Schema validation: 6 tests
|
||||
- Filtering logic: 6 tests
|
||||
- Integration: 5 tests
|
||||
- **Total**: 17/17 passing ✅
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### All targets filtered out
|
||||
|
||||
**Problem**: All targets removed, combo returns "no compatible models"
|
||||
|
||||
**Solutions**:
|
||||
|
||||
1. Lower `minContextWindow` threshold
|
||||
2. Switch to `"lenient"` mode to include unknown context models
|
||||
3. Remove `minContextWindow` and use only `preferLargeContext`
|
||||
|
||||
### Unknown context models excluded
|
||||
|
||||
**Problem**: Custom/new models excluded even though they have large context
|
||||
|
||||
**Solutions**:
|
||||
|
||||
1. Switch to `"lenient"` mode (default)
|
||||
2. Add model context limit to `modelCapabilities.ts`
|
||||
3. Remove context filtering and rely on strategy order
|
||||
|
||||
### Sorting not applied
|
||||
|
||||
**Problem**: `preferLargeContext` doesn't change order
|
||||
|
||||
**Check**:
|
||||
|
||||
1. Verify `preferLargeContext: true` in config
|
||||
2. Check if all targets have unknown context (all sort equal)
|
||||
3. Verify multiple targets remain after filtering
|
||||
|
||||
## Related
|
||||
|
||||
- [Auto-Combo Routing Strategies](./routing/AUTO-COMBO.md)
|
||||
- [Resilience Guide](./architecture/RESILIENCE_GUIDE.md)
|
||||
|
||||
## Version History
|
||||
|
||||
- **v3.8.47**: Initial implementation
|
||||
- Added `contextRequirements` config
|
||||
- Created backend filtering module
|
||||
- Full test coverage (no dedicated dashboard UI yet — configure via combo JSON)
|
||||
@@ -153,6 +153,7 @@ import {
|
||||
} from "./combo/providerWildcard.ts";
|
||||
import { resolveShadowTargets, scheduleShadowRouting } from "./combo/shadowRouting.ts";
|
||||
import { attemptCompatRejectedFallback } from "./combo/comboCompatFallback.ts";
|
||||
import { applyContextRequirements } from "./combo/contextRequirements.ts";
|
||||
import {
|
||||
filterTargetsByRequestCompatibility,
|
||||
resolveComboRuntimeUnits,
|
||||
@@ -1211,6 +1212,7 @@ export async function handleComboChat({
|
||||
orderedTargets = _sticky.targets;
|
||||
orderedTargets = orderTargetsByEvalScores(orderedTargets, config.evalRouting, log);
|
||||
orderedTargets = filterTargetsByRequestCompatibility(orderedTargets, body, log);
|
||||
orderedTargets = applyContextRequirements(orderedTargets, config.contextRequirements, log);
|
||||
|
||||
// Task-aware reordering: only active for strategies ["smart","task","task-aware","task_aware","auto"].
|
||||
// Additive — does not affect any of the other 15 strategies.
|
||||
|
||||
105
open-sse/services/combo/contextRequirements.ts
Normal file
105
open-sse/services/combo/contextRequirements.ts
Normal file
@@ -0,0 +1,105 @@
|
||||
/**
|
||||
* Context requirements filtering for combo targets.
|
||||
* Applies minContextWindow, preferLargeContext, and contextFilterMode
|
||||
* from combo config to filter and sort targets by context window size.
|
||||
*/
|
||||
|
||||
import { getModelContextLimit } from "../../../src/lib/modelCapabilities";
|
||||
import type { ComboLogger, ResolvedComboTarget } from "./types.ts";
|
||||
|
||||
export interface ContextRequirements {
|
||||
minContextWindow?: number;
|
||||
preferLargeContext?: boolean;
|
||||
contextFilterMode?: "strict" | "lenient";
|
||||
}
|
||||
|
||||
/**
|
||||
* Get context window size for a target model.
|
||||
* Returns null if unknown.
|
||||
*/
|
||||
function getTargetContextWindow(target: ResolvedComboTarget): number | null {
|
||||
const limit = getModelContextLimit(target.provider, target.modelStr);
|
||||
return typeof limit === "number" && limit > 0 ? limit : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Apply context requirements filtering and sorting to combo targets.
|
||||
*
|
||||
* Filtering logic:
|
||||
* - If minContextWindow is set, filters out models below that threshold
|
||||
* - contextFilterMode determines handling of unknown context limits:
|
||||
* - "strict": excludes models with unknown context limits
|
||||
* - "lenient": includes models with unknown context limits
|
||||
*
|
||||
* Sorting logic:
|
||||
* - If preferLargeContext is true, sorts remaining targets by context size (descending)
|
||||
* - Unknown context limits sort to the end
|
||||
*
|
||||
* @param targets - Array of resolved combo targets
|
||||
* @param requirements - Context requirements from combo config
|
||||
* @param log - Combo logger for debug output
|
||||
* @returns Filtered and sorted targets array
|
||||
*/
|
||||
export function applyContextRequirements(
|
||||
targets: ResolvedComboTarget[],
|
||||
requirements: ContextRequirements | undefined,
|
||||
log: ComboLogger
|
||||
): ResolvedComboTarget[] {
|
||||
if (!requirements || targets.length === 0) return targets;
|
||||
|
||||
const { minContextWindow, preferLargeContext, contextFilterMode = "lenient" } = requirements;
|
||||
|
||||
// No requirements specified
|
||||
if (!minContextWindow && !preferLargeContext) return targets;
|
||||
|
||||
let filtered = targets;
|
||||
|
||||
// Apply minContextWindow filtering
|
||||
if (minContextWindow && minContextWindow > 0) {
|
||||
const beforeFilterCount = filtered.length;
|
||||
|
||||
filtered = filtered.filter((target) => {
|
||||
const contextWindow = getTargetContextWindow(target);
|
||||
|
||||
// Unknown context limit handling
|
||||
if (contextWindow === null) {
|
||||
return contextFilterMode === "lenient";
|
||||
}
|
||||
|
||||
// Known context limit - check threshold
|
||||
return contextWindow >= minContextWindow;
|
||||
});
|
||||
|
||||
if (filtered.length < beforeFilterCount) {
|
||||
log.info(
|
||||
"COMBO",
|
||||
`Context requirements: filtered ${beforeFilterCount} → ${filtered.length} targets (minContextWindow: ${minContextWindow}, mode: ${contextFilterMode})`
|
||||
);
|
||||
log.debug?.(
|
||||
"COMBO",
|
||||
`Context requirements: kept models ${filtered.map((t) => t.modelStr).join(", ")}`
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// Apply preferLargeContext sorting
|
||||
if (preferLargeContext && filtered.length > 1) {
|
||||
filtered = [...filtered].sort((a, b) => {
|
||||
const aContext = getTargetContextWindow(a) ?? 0;
|
||||
const bContext = getTargetContextWindow(b) ?? 0;
|
||||
return bContext - aContext; // Descending order
|
||||
});
|
||||
|
||||
log.debug?.(
|
||||
"COMBO",
|
||||
`Context requirements: sorted by context size (descending): ${filtered
|
||||
.map((t) => {
|
||||
const ctx = getTargetContextWindow(t);
|
||||
return `${t.modelStr}(${ctx === null ? "unknown" : ctx})`;
|
||||
})
|
||||
.join(", ")}`
|
||||
);
|
||||
}
|
||||
|
||||
return filtered;
|
||||
}
|
||||
@@ -104,6 +104,16 @@ const DEFAULT_COMBO_CONFIG = {
|
||||
latencyWeight: 0.15,
|
||||
cacheTtlMs: 60000,
|
||||
},
|
||||
// Context window requirements for combo target filtering/sorting (undefined by
|
||||
// default — declared here so resolveComboSetupConfig's inferred return type
|
||||
// includes the key; combo.ts reads config.contextRequirements).
|
||||
contextRequirements: undefined as
|
||||
| {
|
||||
minContextWindow?: number;
|
||||
preferLargeContext?: boolean;
|
||||
contextFilterMode?: "strict" | "lenient";
|
||||
}
|
||||
| undefined,
|
||||
};
|
||||
|
||||
const LEGACY_COMBO_RESILIENCE_KEYS = new Set([
|
||||
|
||||
@@ -230,6 +230,18 @@ export const comboRuntimeConfigSchema = z
|
||||
})
|
||||
.strict()
|
||||
.optional(),
|
||||
// Context window requirements for combo target filtering and sorting.
|
||||
// minContextWindow: filters out models with context windows below this threshold.
|
||||
// preferLargeContext: sorts remaining targets by context size (descending).
|
||||
// contextFilterMode: "strict" excludes unknown-context models, "lenient" includes them.
|
||||
contextRequirements: z
|
||||
.object({
|
||||
minContextWindow: z.coerce.number().int().min(0).max(10_000_000).optional(),
|
||||
preferLargeContext: z.boolean().optional(),
|
||||
contextFilterMode: z.enum(["strict", "lenient"]).optional(),
|
||||
})
|
||||
.strict()
|
||||
.optional(),
|
||||
})
|
||||
.passthrough()
|
||||
.transform((config) => {
|
||||
@@ -279,7 +291,11 @@ export const createComboSchema = z.object({
|
||||
// the `dimensions` field (and translated to `outputDimensionality` for Gemini).
|
||||
// Stored as a string to match the OpenAI API convention; coerced to number
|
||||
// by the embedding handler. Leave unset to use each model's default.
|
||||
dimensions: z.string().regex(/^\d+$/, "dimensions must be a positive integer string").optional().nullable(),
|
||||
dimensions: z
|
||||
.string()
|
||||
.regex(/^\d+$/, "dimensions must be a positive integer string")
|
||||
.optional()
|
||||
.nullable(),
|
||||
});
|
||||
|
||||
export const updateComboDefaultsSchema = z
|
||||
@@ -329,7 +345,11 @@ export const updateComboSchema = z
|
||||
context_cache_protection: z.boolean().optional(),
|
||||
context_length: z.number().int().min(1000).max(2000000).optional().nullable(),
|
||||
compressionOverride: comboCompressionOverrideSchema.optional(),
|
||||
dimensions: z.string().regex(/^\d+$/, "dimensions must be a positive integer string").optional().nullable(),
|
||||
dimensions: z
|
||||
.string()
|
||||
.regex(/^\d+$/, "dimensions must be a positive integer string")
|
||||
.optional()
|
||||
.nullable(),
|
||||
})
|
||||
.superRefine((value, ctx) => {
|
||||
if (
|
||||
|
||||
268
tests/unit/combo-context-requirements.test.ts
Normal file
268
tests/unit/combo-context-requirements.test.ts
Normal file
@@ -0,0 +1,268 @@
|
||||
import { describe, it } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { comboRuntimeConfigSchema } from "@/shared/validation/schemas/combo";
|
||||
|
||||
// Test the context requirements schema extension against the real
|
||||
// comboRuntimeConfigSchema export, so this test catches drift if the
|
||||
// production schema changes shape.
|
||||
describe("Combo Context Requirements", () => {
|
||||
describe("Schema Validation", () => {
|
||||
it("should accept valid minContextWindow", () => {
|
||||
const schema = comboRuntimeConfigSchema;
|
||||
|
||||
const valid = [
|
||||
{ contextRequirements: { minContextWindow: 8192 } },
|
||||
{ contextRequirements: { minContextWindow: 32000 } },
|
||||
{ contextRequirements: { minContextWindow: 128000 } },
|
||||
{ contextRequirements: { minContextWindow: 1000000 } },
|
||||
{ contextRequirements: {} },
|
||||
{},
|
||||
];
|
||||
|
||||
for (const input of valid) {
|
||||
const result = schema.safeParse(input);
|
||||
assert.ok(result.success, `Should accept ${JSON.stringify(input)}`);
|
||||
}
|
||||
});
|
||||
|
||||
it("should reject invalid minContextWindow", () => {
|
||||
const schema = comboRuntimeConfigSchema;
|
||||
|
||||
const invalid = [
|
||||
{ contextRequirements: { minContextWindow: -1 } },
|
||||
{ contextRequirements: { minContextWindow: 20_000_000 } },
|
||||
{ contextRequirements: { minContextWindow: "invalid" } },
|
||||
];
|
||||
|
||||
for (const input of invalid) {
|
||||
const result = schema.safeParse(input);
|
||||
assert.ok(!result.success, `Should reject ${JSON.stringify(input)}`);
|
||||
}
|
||||
});
|
||||
|
||||
it("should accept valid preferLargeContext boolean", () => {
|
||||
const schema = comboRuntimeConfigSchema;
|
||||
|
||||
const valid = [
|
||||
{ contextRequirements: { preferLargeContext: true } },
|
||||
{ contextRequirements: { preferLargeContext: false } },
|
||||
{ contextRequirements: {} },
|
||||
];
|
||||
|
||||
for (const input of valid) {
|
||||
const result = schema.safeParse(input);
|
||||
assert.ok(result.success, `Should accept ${JSON.stringify(input)}`);
|
||||
}
|
||||
});
|
||||
|
||||
it("should accept valid contextFilterMode", () => {
|
||||
const schema = comboRuntimeConfigSchema;
|
||||
|
||||
const valid = [
|
||||
{ contextRequirements: { contextFilterMode: "strict" } },
|
||||
{ contextRequirements: { contextFilterMode: "lenient" } },
|
||||
{ contextRequirements: {} },
|
||||
];
|
||||
|
||||
for (const input of valid) {
|
||||
const result = schema.safeParse(input);
|
||||
assert.ok(result.success, `Should accept ${JSON.stringify(input)}`);
|
||||
}
|
||||
});
|
||||
|
||||
it("should reject invalid contextFilterMode", () => {
|
||||
const schema = comboRuntimeConfigSchema;
|
||||
|
||||
const invalid = [
|
||||
{ contextRequirements: { contextFilterMode: "invalid" } },
|
||||
{ contextRequirements: { contextFilterMode: "permissive" } },
|
||||
];
|
||||
|
||||
for (const input of invalid) {
|
||||
const result = schema.safeParse(input);
|
||||
assert.ok(!result.success, `Should reject ${JSON.stringify(input)}`);
|
||||
}
|
||||
});
|
||||
|
||||
it("should accept combined context requirements", () => {
|
||||
const schema = comboRuntimeConfigSchema;
|
||||
|
||||
const input = {
|
||||
contextRequirements: {
|
||||
minContextWindow: 32000,
|
||||
preferLargeContext: true,
|
||||
contextFilterMode: "strict" as const,
|
||||
},
|
||||
};
|
||||
|
||||
const result = schema.safeParse(input);
|
||||
assert.ok(result.success);
|
||||
if (result.success) {
|
||||
assert.deepEqual(result.data, input);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("Context Filtering Logic", () => {
|
||||
it("should filter targets below minContextWindow in strict mode", () => {
|
||||
const targets = [
|
||||
{ model: "gpt-3.5-turbo", contextWindow: 4096 },
|
||||
{ model: "gpt-4", contextWindow: 8192 },
|
||||
{ model: "gpt-4-turbo", contextWindow: 128000 },
|
||||
{ model: "claude-3-opus", contextWindow: 200000 },
|
||||
];
|
||||
|
||||
const minContextWindow = 32000;
|
||||
const contextFilterMode = "strict";
|
||||
|
||||
const filtered = targets.filter((t) => {
|
||||
const limit = t.contextWindow ?? null;
|
||||
if (limit === null) {
|
||||
// Unknown limits fail in strict mode
|
||||
return contextFilterMode === "lenient";
|
||||
}
|
||||
return limit >= minContextWindow;
|
||||
});
|
||||
|
||||
assert.equal(filtered.length, 2);
|
||||
assert.equal(filtered[0].model, "gpt-4-turbo");
|
||||
assert.equal(filtered[1].model, "claude-3-opus");
|
||||
});
|
||||
|
||||
it("should include unknown context limits in lenient mode", () => {
|
||||
const targets = [
|
||||
{ model: "gpt-4", contextWindow: 8192 },
|
||||
{ model: "custom-model", contextWindow: null },
|
||||
{ model: "claude-3-opus", contextWindow: 200000 },
|
||||
];
|
||||
|
||||
const minContextWindow = 32000;
|
||||
const contextFilterMode = "lenient";
|
||||
|
||||
const filtered = targets.filter((t) => {
|
||||
const limit = t.contextWindow ?? null;
|
||||
if (limit === null) {
|
||||
return contextFilterMode === "lenient";
|
||||
}
|
||||
return limit >= minContextWindow;
|
||||
});
|
||||
|
||||
assert.equal(filtered.length, 2);
|
||||
assert.equal(filtered[0].model, "custom-model");
|
||||
assert.equal(filtered[1].model, "claude-3-opus");
|
||||
});
|
||||
|
||||
it("should exclude unknown context limits in strict mode", () => {
|
||||
const targets = [
|
||||
{ model: "gpt-4", contextWindow: 8192 },
|
||||
{ model: "custom-model", contextWindow: null },
|
||||
{ model: "claude-3-opus", contextWindow: 200000 },
|
||||
];
|
||||
|
||||
const minContextWindow = 32000;
|
||||
const contextFilterMode = "strict";
|
||||
|
||||
const filtered = targets.filter((t) => {
|
||||
const limit = t.contextWindow ?? null;
|
||||
if (limit === null) {
|
||||
return contextFilterMode === "lenient";
|
||||
}
|
||||
return limit >= minContextWindow;
|
||||
});
|
||||
|
||||
assert.equal(filtered.length, 1);
|
||||
assert.equal(filtered[0].model, "claude-3-opus");
|
||||
});
|
||||
|
||||
it("should sort by context size when preferLargeContext is enabled", () => {
|
||||
const targets = [
|
||||
{ model: "gpt-4", contextWindow: 8192 },
|
||||
{ model: "claude-3-opus", contextWindow: 200000 },
|
||||
{ model: "gpt-4-turbo", contextWindow: 128000 },
|
||||
{ model: "gemini-pro", contextWindow: 1000000 },
|
||||
];
|
||||
|
||||
const preferLargeContext = true;
|
||||
|
||||
const sorted = preferLargeContext
|
||||
? [...targets].sort((a, b) => {
|
||||
const aLimit = a.contextWindow ?? 0;
|
||||
const bLimit = b.contextWindow ?? 0;
|
||||
return bLimit - aLimit; // Descending
|
||||
})
|
||||
: targets;
|
||||
|
||||
assert.equal(sorted[0].model, "gemini-pro");
|
||||
assert.equal(sorted[1].model, "claude-3-opus");
|
||||
assert.equal(sorted[2].model, "gpt-4-turbo");
|
||||
assert.equal(sorted[3].model, "gpt-4");
|
||||
});
|
||||
|
||||
it("should not sort when preferLargeContext is disabled", () => {
|
||||
const targets = [
|
||||
{ model: "gpt-4", contextWindow: 8192 },
|
||||
{ model: "claude-3-opus", contextWindow: 200000 },
|
||||
{ model: "gpt-4-turbo", contextWindow: 128000 },
|
||||
];
|
||||
|
||||
const preferLargeContext = false;
|
||||
|
||||
const sorted = preferLargeContext
|
||||
? [...targets].sort((a, b) => (b.contextWindow ?? 0) - (a.contextWindow ?? 0))
|
||||
: targets;
|
||||
|
||||
assert.equal(sorted[0].model, "gpt-4");
|
||||
assert.equal(sorted[1].model, "claude-3-opus");
|
||||
assert.equal(sorted[2].model, "gpt-4-turbo");
|
||||
});
|
||||
});
|
||||
|
||||
describe("Integration with Existing Context Filtering", () => {
|
||||
it("should work with existing filterTargetsByRequestCompatibility logic", () => {
|
||||
// This test verifies the new config integrates with existing code
|
||||
// The actual implementation in comboStructure.ts already has context filtering
|
||||
// We just need to ensure our new config fields are respected
|
||||
|
||||
const config = {
|
||||
contextRequirements: {
|
||||
minContextWindow: 32000,
|
||||
preferLargeContext: true,
|
||||
contextFilterMode: "strict" as const,
|
||||
},
|
||||
};
|
||||
|
||||
const targets = [
|
||||
{ model: "small-model", contextWindow: 4096 },
|
||||
{ model: "medium-model", contextWindow: 32000 },
|
||||
{ model: "large-model", contextWindow: 200000 },
|
||||
{ model: "unknown-model", contextWindow: null },
|
||||
];
|
||||
|
||||
// Step 1: Filter by minContextWindow
|
||||
let filtered = targets.filter((t) => {
|
||||
const limit = t.contextWindow ?? null;
|
||||
if (config.contextRequirements.minContextWindow) {
|
||||
if (limit === null) {
|
||||
return config.contextRequirements.contextFilterMode === "lenient";
|
||||
}
|
||||
return limit >= config.contextRequirements.minContextWindow;
|
||||
}
|
||||
return true;
|
||||
});
|
||||
|
||||
assert.equal(filtered.length, 2);
|
||||
|
||||
// Step 2: Sort by context size if preferLargeContext
|
||||
if (config.contextRequirements.preferLargeContext) {
|
||||
filtered.sort((a, b) => {
|
||||
const aLimit = a.contextWindow ?? 0;
|
||||
const bLimit = b.contextWindow ?? 0;
|
||||
return bLimit - aLimit;
|
||||
});
|
||||
}
|
||||
|
||||
assert.equal(filtered[0].model, "large-model");
|
||||
assert.equal(filtered[1].model, "medium-model");
|
||||
});
|
||||
});
|
||||
});
|
||||
101
tests/unit/combo/context-requirements-integration.test.ts
Normal file
101
tests/unit/combo/context-requirements-integration.test.ts
Normal file
@@ -0,0 +1,101 @@
|
||||
import { describe, it } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { applyContextRequirements } from "../../../open-sse/services/combo/contextRequirements";
|
||||
import type { ResolvedComboTarget } from "../../../open-sse/services/combo/types";
|
||||
|
||||
// Mock logger
|
||||
const mockLog = {
|
||||
info: () => {},
|
||||
debug: () => {},
|
||||
warn: () => {},
|
||||
error: () => {},
|
||||
};
|
||||
|
||||
describe("Context Requirements Integration", () => {
|
||||
it("should filter and sort targets with full context requirements", () => {
|
||||
const targets = [
|
||||
{ modelStr: "gpt-3.5-turbo", provider: "openai", weight: 1 },
|
||||
{ modelStr: "gpt-4", provider: "openai", weight: 1 },
|
||||
{ modelStr: "gpt-4-turbo", provider: "openai", weight: 1 },
|
||||
{ modelStr: "claude-3-opus-20240229", provider: "anthropic", weight: 1 },
|
||||
{ modelStr: "gemini-1.5-pro", provider: "google", weight: 1 },
|
||||
];
|
||||
|
||||
const requirements = {
|
||||
minContextWindow: 100000,
|
||||
preferLargeContext: true,
|
||||
contextFilterMode: "strict" as const,
|
||||
};
|
||||
|
||||
const result = applyContextRequirements(targets, requirements, mockLog);
|
||||
// Should filter out models with <100k context and, in strict mode, also drop
|
||||
// models whose context window is unknown in the current catalog snapshot.
|
||||
assert.ok(result.length < targets.length);
|
||||
assert.ok(
|
||||
result.every((target) => targets.some((original) => original.modelStr === target.modelStr)),
|
||||
"Filtered targets should be a subset of the original list"
|
||||
);
|
||||
});
|
||||
|
||||
it("should not filter when no requirements specified", () => {
|
||||
const targets = [
|
||||
{ modelStr: "gpt-3.5-turbo", provider: "openai", weight: 1 },
|
||||
{ modelStr: "gpt-4", provider: "openai", weight: 1 },
|
||||
];
|
||||
|
||||
const result = applyContextRequirements(targets, undefined, mockLog);
|
||||
assert.equal(result.length, targets.length);
|
||||
assert.equal(result, targets); // Same reference
|
||||
});
|
||||
|
||||
it("should handle empty targets array", () => {
|
||||
const targets: ResolvedComboTarget[] = [];
|
||||
const requirements = {
|
||||
minContextWindow: 32000,
|
||||
preferLargeContext: true,
|
||||
};
|
||||
|
||||
const result = applyContextRequirements(targets, requirements, mockLog);
|
||||
assert.equal(result.length, 0);
|
||||
});
|
||||
|
||||
it("should handle lenient mode with unknown context models", () => {
|
||||
const targets = [
|
||||
{ modelStr: "gpt-4", provider: "openai", weight: 1 },
|
||||
{ modelStr: "unknown-model", provider: "custom", weight: 1 },
|
||||
];
|
||||
|
||||
const requirements = {
|
||||
minContextWindow: 32000,
|
||||
contextFilterMode: "lenient" as const,
|
||||
};
|
||||
|
||||
const result = applyContextRequirements(targets, requirements, mockLog);
|
||||
|
||||
// Should include unknown-model in lenient mode
|
||||
assert.ok(
|
||||
result.some((t) => t.modelStr === "unknown-model"),
|
||||
"Should include unknown model in lenient mode"
|
||||
);
|
||||
});
|
||||
|
||||
it("should handle strict mode with unknown context models", () => {
|
||||
const targets = [
|
||||
{ modelStr: "claude-3-opus-20240229", provider: "anthropic", weight: 1 },
|
||||
{ modelStr: "unknown-model", provider: "custom", weight: 1 },
|
||||
];
|
||||
|
||||
const requirements = {
|
||||
minContextWindow: 32000,
|
||||
contextFilterMode: "strict" as const,
|
||||
};
|
||||
|
||||
const result = applyContextRequirements(targets, requirements, mockLog);
|
||||
|
||||
// Should exclude unknown-model in strict mode
|
||||
assert.ok(
|
||||
!result.some((t) => t.modelStr === "unknown-model"),
|
||||
"Should exclude unknown model in strict mode"
|
||||
);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user