[trim] feat(combo): add context requirements config for target filtering (#6907)

* feat(combo): add context requirements config for target filtering

Add contextRequirements config field to combo runtime config:
- minContextWindow: filter models below threshold (0-10M tokens)
- preferLargeContext: sort targets by context size descending
- contextFilterMode: 'strict' excludes unknown limits, 'lenient' includes them

Implementation:
- Added Zod schema validation in combo.ts
- Created contextRequirements.ts module with applyContextRequirements()
- Integrated filtering after filterTargetsByRequestCompatibility()
- Full test coverage with unit + integration tests

Tests: 17/17 pass (combo-context-requirements.test.ts + integration)

* feat(combo): add ContextRequirementsEditor UI component

Add standalone React component for editing context requirements config:
- Slider for minContextWindow (0 to 1M tokens) with presets
- Toggle for preferLargeContext sorting
- Radio group for contextFilterMode (strict/lenient)
- Tooltips explaining each option
- Active filters summary display

Component features:
- Shadcn UI components (Card, Slider, Switch, RadioGroup)
- Preset buttons for common context sizes (8K, 32K, 128K, 1M)
- Conditional display of filter mode when minContextWindow > 0
- Clear visual feedback of active filters

Integration:
Import and use in combo config form where other config fields
like fusionTuning and judgeModel are edited. Pass combo.config.contextRequirements
as value prop and update on onChange.

Example usage:
<ContextRequirementsEditor
  value={config.contextRequirements}
  onChange={(val) => updateConfig({ contextRequirements: val })}
/>

UI matches existing combo config editor patterns.

* feat(combo): wire ContextRequirementsEditor into combo config form

Adds context requirements section to combo edit page (strategy section),
matching existing ResponseValidation pattern. Placed after response
validation block, before agent features.

* docs(combo): add context requirements feature documentation

Covers: config schema, behavior, use cases, UI integration,
troubleshooting, and test instructions.

* fix(combo): pass provider+modelStr to getModelContextLimit for accurate context resolution

Per gemini-code-assist review feedback: model names are not globally
unique across providers. Passing both provider and modelStr ensures
correct context limit resolution in applyContextRequirements().

* fix(combo): repair broken doc links and restore test:unit:fast flag

- Point docs/combo-context-requirements.md 'Related' links at real docs
  (routing/AUTO-COMBO.md, architecture/RESILIENCE_GUIDE.md) — the three
  placeholder links (strategies.md/model-capabilities.md/fusion-tuning.md)
  did not exist and failed check:doc-links (Docs Gates fast-path).
- Revert an out-of-scope package.json change to test:unit:fast that dropped
  --test-isolation=none; restore to match release/v3.8.47.

Co-authored-by: oyi77 <oyi77@users.noreply.github.com>

* test(combo): validate context requirements against the real Zod schema

Point tests/unit/combo-context-requirements.test.ts at the real
comboRuntimeConfigSchema export (src/shared/validation/schemas/combo.ts)
instead of hand-duplicating the Zod schema inline, so the test catches
schema drift.

Also declare contextRequirements on DEFAULT_COMBO_CONFIG so
resolveComboSetupConfig's inferred return type includes the key —
combo.ts reads config.contextRequirements but the property was missing
from the object typecheck:core infers types from, causing a build error.

Co-authored-by: oyi77 <oyi77@users.noreply.github.com>

* fix(combos): remove dead ContextRequirementsEditor scaffolding (broken ui/card+label imports)

The editor imported @/components/ui/card and @/components/ui/label which do not
exist in the repo, breaking the Turbopack build. Removed the editor + its page.tsx
usage + doc mention; the real fix (comboConfig contextRequirements default + test)
is preserved.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Co-authored-by: oyi77 <oyi77@users.noreply.github.com>
Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com>
Co-authored-by: Diego Rodrigues de Sa e Souza <8016841+diegosouzapw@users.noreply.github.com>
This commit is contained in:
Paijo
2026-07-12 13:41:50 +07:00
committed by GitHub
parent ecf226ece7
commit 10cb2447b1
7 changed files with 770 additions and 2 deletions

View File

@@ -0,0 +1,262 @@
# Combo Context Requirements Feature
## Overview
The Context Requirements feature allows combo configurations to filter and sort targets based on their context window size. This is useful for use cases requiring large context windows like:
- Long document processing (100k+ tokens)
- Large codebase analysis
- Extensive conversation histories
- Multi-file code reviews
## Configuration
### Schema
Add `contextRequirements` to your combo's runtime config:
```json
{
"contextRequirements": {
"minContextWindow": 128000,
"preferLargeContext": true,
"contextFilterMode": "strict"
}
}
```
### Fields
#### `minContextWindow` (optional)
- **Type**: `number` (0 to 10,000,000)
- **Default**: `undefined` (no filtering)
- **Description**: Filters out models with context windows below this threshold
**Examples**:
- `32000` - Filter out models with <32K context
- `128000` - Require 128K+ context (GPT-4 Turbo, Claude 3)
- `200000` - Require 200K+ context (Claude 3 Opus)
- `1000000` - Require 1M+ context (Gemini 1.5 Pro)
#### `preferLargeContext` (optional)
- **Type**: `boolean`
- **Default**: `false`
- **Description**: When `true`, sorts remaining targets by context size (descending). Large context models are tried first.
#### `contextFilterMode` (optional)
- **Type**: `"strict"` | `"lenient"`
- **Default**: `"lenient"`
- **Description**: How to handle models with unknown context window limits
- `"strict"`: Excludes models with unknown context limits
- `"lenient"`: Includes models with unknown context limits
## Behavior
### Filtering Pipeline
Context requirements are applied after `filterTargetsByRequestCompatibility()`:
1. **Request compatibility filtering** - Removes models incompatible with request (tools, vision, structured output)
2. **Context requirements filtering** - Applies `minContextWindow` and `contextFilterMode`
3. **Context-based sorting** - If `preferLargeContext` is true, sorts by context size descending
### Filter Mode Logic
When `minContextWindow` is set:
**Lenient mode** (default):
- ✅ Includes models with context >= minContextWindow
- ✅ Includes models with unknown context limits
- ❌ Excludes models with context < minContextWindow
**Strict mode**:
- ✅ Includes models with context >= minContextWindow
- ❌ Excludes models with unknown context limits
- ❌ Excludes models with context < minContextWindow
### Sorting Logic
When `preferLargeContext` is true:
- Models are sorted by context window size (descending)
- Unknown context models sort to the end
- Original strategy order is used as a tiebreaker
## Use Cases
### Example 1: Long Document Processing
```json
{
"name": "Document Analysis",
"strategy": "fusion",
"config": {
"contextRequirements": {
"minContextWindow": 128000,
"preferLargeContext": true,
"contextFilterMode": "strict"
}
}
}
```
This configuration:
- Requires 128K+ context window
- Prefers larger context models (Gemini 1.5 Pro > Claude 3 Opus > GPT-4 Turbo)
- Excludes models with unknown context limits
### Example 2: Large Codebase Analysis
```json
{
"name": "Code Review",
"strategy": "auto",
"config": {
"contextRequirements": {
"minContextWindow": 200000,
"preferLargeContext": true,
"contextFilterMode": "lenient"
}
}
}
```
This configuration:
- Requires 200K+ context window
- Prefers larger context models
- Includes models with unknown limits (lenient)
### Example 3: Prefer Large Context Without Strict Requirements
```json
{
"name": "Flexible Chat",
"strategy": "weighted",
"config": {
"contextRequirements": {
"preferLargeContext": true
}
}
}
```
This configuration:
- No minimum requirement (all models eligible)
- Sorts by context size (largest first)
- Useful when large context is preferred but not required
## API Response
When context requirements filter targets, the combo logger outputs:
```
[COMBO] Context requirements: filtered 10 → 3 targets (minContextWindow: 128000, mode: strict)
[COMBO] Context requirements: kept models gemini-1.5-pro, claude-3-opus-20240229, gpt-4-turbo
[COMBO] Context requirements: sorted by context size (descending): gemini-1.5-pro(1000000), claude-3-opus-20240229(200000), gpt-4-turbo(128000)
```
## Implementation Details
### Backend Module
`open-sse/services/combo/contextRequirements.ts`:
- `applyContextRequirements()` - Main filtering function
- `getTargetContextWindow()` - Context lookup helper
- Uses `getModelContextLimit()` from `modelCapabilities.ts`
### Integration Point
`open-sse/services/combo.ts` line 1187:
```typescript
orderedTargets = filterTargetsByRequestCompatibility(orderedTargets, body, log);
orderedTargets = applyContextRequirements(orderedTargets, config.contextRequirements, log);
```
### Schema Definition
`src/shared/validation/schemas/combo.ts`:
```typescript
contextRequirements: z
.object({
minContextWindow: z.coerce.number().int().min(0).max(10_000_000).optional(),
preferLargeContext: z.boolean().optional(),
contextFilterMode: z.enum(["strict", "lenient"]).optional(),
})
.strict()
.optional(),
```
## Testing
### Run Tests
```bash
# Unit tests (schema + logic)
npm test tests/unit/combo-context-requirements.test.ts
# Integration tests (end-to-end)
npm test tests/unit/combo/context-requirements-integration.test.ts
```
### Test Coverage
- Schema validation: 6 tests
- Filtering logic: 6 tests
- Integration: 5 tests
- **Total**: 17/17 passing ✅
## Troubleshooting
### All targets filtered out
**Problem**: All targets removed, combo returns "no compatible models"
**Solutions**:
1. Lower `minContextWindow` threshold
2. Switch to `"lenient"` mode to include unknown context models
3. Remove `minContextWindow` and use only `preferLargeContext`
### Unknown context models excluded
**Problem**: Custom/new models excluded even though they have large context
**Solutions**:
1. Switch to `"lenient"` mode (default)
2. Add model context limit to `modelCapabilities.ts`
3. Remove context filtering and rely on strategy order
### Sorting not applied
**Problem**: `preferLargeContext` doesn't change order
**Check**:
1. Verify `preferLargeContext: true` in config
2. Check if all targets have unknown context (all sort equal)
3. Verify multiple targets remain after filtering
## Related
- [Auto-Combo Routing Strategies](./routing/AUTO-COMBO.md)
- [Resilience Guide](./architecture/RESILIENCE_GUIDE.md)
## Version History
- **v3.8.47**: Initial implementation
- Added `contextRequirements` config
- Created backend filtering module
- Full test coverage (no dedicated dashboard UI yet — configure via combo JSON)

View File

@@ -153,6 +153,7 @@ import {
} from "./combo/providerWildcard.ts";
import { resolveShadowTargets, scheduleShadowRouting } from "./combo/shadowRouting.ts";
import { attemptCompatRejectedFallback } from "./combo/comboCompatFallback.ts";
import { applyContextRequirements } from "./combo/contextRequirements.ts";
import {
filterTargetsByRequestCompatibility,
resolveComboRuntimeUnits,
@@ -1211,6 +1212,7 @@ export async function handleComboChat({
orderedTargets = _sticky.targets;
orderedTargets = orderTargetsByEvalScores(orderedTargets, config.evalRouting, log);
orderedTargets = filterTargetsByRequestCompatibility(orderedTargets, body, log);
orderedTargets = applyContextRequirements(orderedTargets, config.contextRequirements, log);
// Task-aware reordering: only active for strategies ["smart","task","task-aware","task_aware","auto"].
// Additive — does not affect any of the other 15 strategies.

View File

@@ -0,0 +1,105 @@
/**
* Context requirements filtering for combo targets.
* Applies minContextWindow, preferLargeContext, and contextFilterMode
* from combo config to filter and sort targets by context window size.
*/
import { getModelContextLimit } from "../../../src/lib/modelCapabilities";
import type { ComboLogger, ResolvedComboTarget } from "./types.ts";
export interface ContextRequirements {
minContextWindow?: number;
preferLargeContext?: boolean;
contextFilterMode?: "strict" | "lenient";
}
/**
* Get context window size for a target model.
* Returns null if unknown.
*/
function getTargetContextWindow(target: ResolvedComboTarget): number | null {
const limit = getModelContextLimit(target.provider, target.modelStr);
return typeof limit === "number" && limit > 0 ? limit : null;
}
/**
* Apply context requirements filtering and sorting to combo targets.
*
* Filtering logic:
* - If minContextWindow is set, filters out models below that threshold
* - contextFilterMode determines handling of unknown context limits:
* - "strict": excludes models with unknown context limits
* - "lenient": includes models with unknown context limits
*
* Sorting logic:
* - If preferLargeContext is true, sorts remaining targets by context size (descending)
* - Unknown context limits sort to the end
*
* @param targets - Array of resolved combo targets
* @param requirements - Context requirements from combo config
* @param log - Combo logger for debug output
* @returns Filtered and sorted targets array
*/
export function applyContextRequirements(
targets: ResolvedComboTarget[],
requirements: ContextRequirements | undefined,
log: ComboLogger
): ResolvedComboTarget[] {
if (!requirements || targets.length === 0) return targets;
const { minContextWindow, preferLargeContext, contextFilterMode = "lenient" } = requirements;
// No requirements specified
if (!minContextWindow && !preferLargeContext) return targets;
let filtered = targets;
// Apply minContextWindow filtering
if (minContextWindow && minContextWindow > 0) {
const beforeFilterCount = filtered.length;
filtered = filtered.filter((target) => {
const contextWindow = getTargetContextWindow(target);
// Unknown context limit handling
if (contextWindow === null) {
return contextFilterMode === "lenient";
}
// Known context limit - check threshold
return contextWindow >= minContextWindow;
});
if (filtered.length < beforeFilterCount) {
log.info(
"COMBO",
`Context requirements: filtered ${beforeFilterCount}${filtered.length} targets (minContextWindow: ${minContextWindow}, mode: ${contextFilterMode})`
);
log.debug?.(
"COMBO",
`Context requirements: kept models ${filtered.map((t) => t.modelStr).join(", ")}`
);
}
}
// Apply preferLargeContext sorting
if (preferLargeContext && filtered.length > 1) {
filtered = [...filtered].sort((a, b) => {
const aContext = getTargetContextWindow(a) ?? 0;
const bContext = getTargetContextWindow(b) ?? 0;
return bContext - aContext; // Descending order
});
log.debug?.(
"COMBO",
`Context requirements: sorted by context size (descending): ${filtered
.map((t) => {
const ctx = getTargetContextWindow(t);
return `${t.modelStr}(${ctx === null ? "unknown" : ctx})`;
})
.join(", ")}`
);
}
return filtered;
}

View File

@@ -104,6 +104,16 @@ const DEFAULT_COMBO_CONFIG = {
latencyWeight: 0.15,
cacheTtlMs: 60000,
},
// Context window requirements for combo target filtering/sorting (undefined by
// default — declared here so resolveComboSetupConfig's inferred return type
// includes the key; combo.ts reads config.contextRequirements).
contextRequirements: undefined as
| {
minContextWindow?: number;
preferLargeContext?: boolean;
contextFilterMode?: "strict" | "lenient";
}
| undefined,
};
const LEGACY_COMBO_RESILIENCE_KEYS = new Set([

View File

@@ -230,6 +230,18 @@ export const comboRuntimeConfigSchema = z
})
.strict()
.optional(),
// Context window requirements for combo target filtering and sorting.
// minContextWindow: filters out models with context windows below this threshold.
// preferLargeContext: sorts remaining targets by context size (descending).
// contextFilterMode: "strict" excludes unknown-context models, "lenient" includes them.
contextRequirements: z
.object({
minContextWindow: z.coerce.number().int().min(0).max(10_000_000).optional(),
preferLargeContext: z.boolean().optional(),
contextFilterMode: z.enum(["strict", "lenient"]).optional(),
})
.strict()
.optional(),
})
.passthrough()
.transform((config) => {
@@ -279,7 +291,11 @@ export const createComboSchema = z.object({
// the `dimensions` field (and translated to `outputDimensionality` for Gemini).
// Stored as a string to match the OpenAI API convention; coerced to number
// by the embedding handler. Leave unset to use each model's default.
dimensions: z.string().regex(/^\d+$/, "dimensions must be a positive integer string").optional().nullable(),
dimensions: z
.string()
.regex(/^\d+$/, "dimensions must be a positive integer string")
.optional()
.nullable(),
});
export const updateComboDefaultsSchema = z
@@ -329,7 +345,11 @@ export const updateComboSchema = z
context_cache_protection: z.boolean().optional(),
context_length: z.number().int().min(1000).max(2000000).optional().nullable(),
compressionOverride: comboCompressionOverrideSchema.optional(),
dimensions: z.string().regex(/^\d+$/, "dimensions must be a positive integer string").optional().nullable(),
dimensions: z
.string()
.regex(/^\d+$/, "dimensions must be a positive integer string")
.optional()
.nullable(),
})
.superRefine((value, ctx) => {
if (

View File

@@ -0,0 +1,268 @@
import { describe, it } from "node:test";
import assert from "node:assert/strict";
import { comboRuntimeConfigSchema } from "@/shared/validation/schemas/combo";
// Test the context requirements schema extension against the real
// comboRuntimeConfigSchema export, so this test catches drift if the
// production schema changes shape.
describe("Combo Context Requirements", () => {
describe("Schema Validation", () => {
it("should accept valid minContextWindow", () => {
const schema = comboRuntimeConfigSchema;
const valid = [
{ contextRequirements: { minContextWindow: 8192 } },
{ contextRequirements: { minContextWindow: 32000 } },
{ contextRequirements: { minContextWindow: 128000 } },
{ contextRequirements: { minContextWindow: 1000000 } },
{ contextRequirements: {} },
{},
];
for (const input of valid) {
const result = schema.safeParse(input);
assert.ok(result.success, `Should accept ${JSON.stringify(input)}`);
}
});
it("should reject invalid minContextWindow", () => {
const schema = comboRuntimeConfigSchema;
const invalid = [
{ contextRequirements: { minContextWindow: -1 } },
{ contextRequirements: { minContextWindow: 20_000_000 } },
{ contextRequirements: { minContextWindow: "invalid" } },
];
for (const input of invalid) {
const result = schema.safeParse(input);
assert.ok(!result.success, `Should reject ${JSON.stringify(input)}`);
}
});
it("should accept valid preferLargeContext boolean", () => {
const schema = comboRuntimeConfigSchema;
const valid = [
{ contextRequirements: { preferLargeContext: true } },
{ contextRequirements: { preferLargeContext: false } },
{ contextRequirements: {} },
];
for (const input of valid) {
const result = schema.safeParse(input);
assert.ok(result.success, `Should accept ${JSON.stringify(input)}`);
}
});
it("should accept valid contextFilterMode", () => {
const schema = comboRuntimeConfigSchema;
const valid = [
{ contextRequirements: { contextFilterMode: "strict" } },
{ contextRequirements: { contextFilterMode: "lenient" } },
{ contextRequirements: {} },
];
for (const input of valid) {
const result = schema.safeParse(input);
assert.ok(result.success, `Should accept ${JSON.stringify(input)}`);
}
});
it("should reject invalid contextFilterMode", () => {
const schema = comboRuntimeConfigSchema;
const invalid = [
{ contextRequirements: { contextFilterMode: "invalid" } },
{ contextRequirements: { contextFilterMode: "permissive" } },
];
for (const input of invalid) {
const result = schema.safeParse(input);
assert.ok(!result.success, `Should reject ${JSON.stringify(input)}`);
}
});
it("should accept combined context requirements", () => {
const schema = comboRuntimeConfigSchema;
const input = {
contextRequirements: {
minContextWindow: 32000,
preferLargeContext: true,
contextFilterMode: "strict" as const,
},
};
const result = schema.safeParse(input);
assert.ok(result.success);
if (result.success) {
assert.deepEqual(result.data, input);
}
});
});
describe("Context Filtering Logic", () => {
it("should filter targets below minContextWindow in strict mode", () => {
const targets = [
{ model: "gpt-3.5-turbo", contextWindow: 4096 },
{ model: "gpt-4", contextWindow: 8192 },
{ model: "gpt-4-turbo", contextWindow: 128000 },
{ model: "claude-3-opus", contextWindow: 200000 },
];
const minContextWindow = 32000;
const contextFilterMode = "strict";
const filtered = targets.filter((t) => {
const limit = t.contextWindow ?? null;
if (limit === null) {
// Unknown limits fail in strict mode
return contextFilterMode === "lenient";
}
return limit >= minContextWindow;
});
assert.equal(filtered.length, 2);
assert.equal(filtered[0].model, "gpt-4-turbo");
assert.equal(filtered[1].model, "claude-3-opus");
});
it("should include unknown context limits in lenient mode", () => {
const targets = [
{ model: "gpt-4", contextWindow: 8192 },
{ model: "custom-model", contextWindow: null },
{ model: "claude-3-opus", contextWindow: 200000 },
];
const minContextWindow = 32000;
const contextFilterMode = "lenient";
const filtered = targets.filter((t) => {
const limit = t.contextWindow ?? null;
if (limit === null) {
return contextFilterMode === "lenient";
}
return limit >= minContextWindow;
});
assert.equal(filtered.length, 2);
assert.equal(filtered[0].model, "custom-model");
assert.equal(filtered[1].model, "claude-3-opus");
});
it("should exclude unknown context limits in strict mode", () => {
const targets = [
{ model: "gpt-4", contextWindow: 8192 },
{ model: "custom-model", contextWindow: null },
{ model: "claude-3-opus", contextWindow: 200000 },
];
const minContextWindow = 32000;
const contextFilterMode = "strict";
const filtered = targets.filter((t) => {
const limit = t.contextWindow ?? null;
if (limit === null) {
return contextFilterMode === "lenient";
}
return limit >= minContextWindow;
});
assert.equal(filtered.length, 1);
assert.equal(filtered[0].model, "claude-3-opus");
});
it("should sort by context size when preferLargeContext is enabled", () => {
const targets = [
{ model: "gpt-4", contextWindow: 8192 },
{ model: "claude-3-opus", contextWindow: 200000 },
{ model: "gpt-4-turbo", contextWindow: 128000 },
{ model: "gemini-pro", contextWindow: 1000000 },
];
const preferLargeContext = true;
const sorted = preferLargeContext
? [...targets].sort((a, b) => {
const aLimit = a.contextWindow ?? 0;
const bLimit = b.contextWindow ?? 0;
return bLimit - aLimit; // Descending
})
: targets;
assert.equal(sorted[0].model, "gemini-pro");
assert.equal(sorted[1].model, "claude-3-opus");
assert.equal(sorted[2].model, "gpt-4-turbo");
assert.equal(sorted[3].model, "gpt-4");
});
it("should not sort when preferLargeContext is disabled", () => {
const targets = [
{ model: "gpt-4", contextWindow: 8192 },
{ model: "claude-3-opus", contextWindow: 200000 },
{ model: "gpt-4-turbo", contextWindow: 128000 },
];
const preferLargeContext = false;
const sorted = preferLargeContext
? [...targets].sort((a, b) => (b.contextWindow ?? 0) - (a.contextWindow ?? 0))
: targets;
assert.equal(sorted[0].model, "gpt-4");
assert.equal(sorted[1].model, "claude-3-opus");
assert.equal(sorted[2].model, "gpt-4-turbo");
});
});
describe("Integration with Existing Context Filtering", () => {
it("should work with existing filterTargetsByRequestCompatibility logic", () => {
// This test verifies the new config integrates with existing code
// The actual implementation in comboStructure.ts already has context filtering
// We just need to ensure our new config fields are respected
const config = {
contextRequirements: {
minContextWindow: 32000,
preferLargeContext: true,
contextFilterMode: "strict" as const,
},
};
const targets = [
{ model: "small-model", contextWindow: 4096 },
{ model: "medium-model", contextWindow: 32000 },
{ model: "large-model", contextWindow: 200000 },
{ model: "unknown-model", contextWindow: null },
];
// Step 1: Filter by minContextWindow
let filtered = targets.filter((t) => {
const limit = t.contextWindow ?? null;
if (config.contextRequirements.minContextWindow) {
if (limit === null) {
return config.contextRequirements.contextFilterMode === "lenient";
}
return limit >= config.contextRequirements.minContextWindow;
}
return true;
});
assert.equal(filtered.length, 2);
// Step 2: Sort by context size if preferLargeContext
if (config.contextRequirements.preferLargeContext) {
filtered.sort((a, b) => {
const aLimit = a.contextWindow ?? 0;
const bLimit = b.contextWindow ?? 0;
return bLimit - aLimit;
});
}
assert.equal(filtered[0].model, "large-model");
assert.equal(filtered[1].model, "medium-model");
});
});
});

View File

@@ -0,0 +1,101 @@
import { describe, it } from "node:test";
import assert from "node:assert/strict";
import { applyContextRequirements } from "../../../open-sse/services/combo/contextRequirements";
import type { ResolvedComboTarget } from "../../../open-sse/services/combo/types";
// Mock logger
const mockLog = {
info: () => {},
debug: () => {},
warn: () => {},
error: () => {},
};
describe("Context Requirements Integration", () => {
it("should filter and sort targets with full context requirements", () => {
const targets = [
{ modelStr: "gpt-3.5-turbo", provider: "openai", weight: 1 },
{ modelStr: "gpt-4", provider: "openai", weight: 1 },
{ modelStr: "gpt-4-turbo", provider: "openai", weight: 1 },
{ modelStr: "claude-3-opus-20240229", provider: "anthropic", weight: 1 },
{ modelStr: "gemini-1.5-pro", provider: "google", weight: 1 },
];
const requirements = {
minContextWindow: 100000,
preferLargeContext: true,
contextFilterMode: "strict" as const,
};
const result = applyContextRequirements(targets, requirements, mockLog);
// Should filter out models with <100k context and, in strict mode, also drop
// models whose context window is unknown in the current catalog snapshot.
assert.ok(result.length < targets.length);
assert.ok(
result.every((target) => targets.some((original) => original.modelStr === target.modelStr)),
"Filtered targets should be a subset of the original list"
);
});
it("should not filter when no requirements specified", () => {
const targets = [
{ modelStr: "gpt-3.5-turbo", provider: "openai", weight: 1 },
{ modelStr: "gpt-4", provider: "openai", weight: 1 },
];
const result = applyContextRequirements(targets, undefined, mockLog);
assert.equal(result.length, targets.length);
assert.equal(result, targets); // Same reference
});
it("should handle empty targets array", () => {
const targets: ResolvedComboTarget[] = [];
const requirements = {
minContextWindow: 32000,
preferLargeContext: true,
};
const result = applyContextRequirements(targets, requirements, mockLog);
assert.equal(result.length, 0);
});
it("should handle lenient mode with unknown context models", () => {
const targets = [
{ modelStr: "gpt-4", provider: "openai", weight: 1 },
{ modelStr: "unknown-model", provider: "custom", weight: 1 },
];
const requirements = {
minContextWindow: 32000,
contextFilterMode: "lenient" as const,
};
const result = applyContextRequirements(targets, requirements, mockLog);
// Should include unknown-model in lenient mode
assert.ok(
result.some((t) => t.modelStr === "unknown-model"),
"Should include unknown model in lenient mode"
);
});
it("should handle strict mode with unknown context models", () => {
const targets = [
{ modelStr: "claude-3-opus-20240229", provider: "anthropic", weight: 1 },
{ modelStr: "unknown-model", provider: "custom", weight: 1 },
];
const requirements = {
minContextWindow: 32000,
contextFilterMode: "strict" as const,
};
const result = applyContextRequirements(targets, requirements, mockLog);
// Should exclude unknown-model in strict mode
assert.ok(
!result.some((t) => t.modelStr === "unknown-model"),
"Should exclude unknown model in strict mode"
);
});
});