mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-07-31 04:12:10 +03:00
feat(chaos): big update - optimize, fix bugs, add features, enhance UX (#6728)
* feat(chaos): add Chaos Mode — multi-model parallel/collaborative execution - New DB column chaos_mode_enabled on api_keys table - API key create/PATCH routes support chaosModeEnabled toggle - Core library src/lib/chaos/chaosConfig.ts for persistent config - API routes: GET/PUT/DELETE /api/chaos/config - Chaos execution POST /api/skills/collect/chaos with key auth - Dashboard page at /dashboard/chaos with full config UI - Sidebar entry in Agentic Features section - Chaos mode toggle in API Key editor permissions panel - i18n keys for chaos config (en.json) * feat(chaos): big update — optimize, fix bugs, add features === Changes === 1. NEW: src/lib/chaos/chaosExecutor.ts — shared execution engine - Removed ~150 lines of duplicate dispatch logic between two API routes - Single executeChaosRun() function used by both endpoints - Added concurrency limit (max 10 parallel requests) - Added proper TypeScript interfaces (ChaosRunInput, ChaosRunResult) - Added error logging throughout 2. FIX: src/app/api/skills/collect/chaos/route.ts - Was MISSING logger import (log.error was undefined at runtime) - Reduced from 388 lines → 142 lines by delegating to shared executor - Added maxTokens support in schema validation 3. REFACTOR: src/app/api/chaos/run/route.ts - Simplified to thin wrapper: auth + validate + delegate to executor - Added maxTokens support 4. ENHANCE: src/lib/chaos/chaosConfig.ts - Added maxTokens config field (256-128k, default 4096) - Persisted per-instance via settings table 5. ENHANCE: UI — ChaosConfigPageClient.tsx - Loads available providers from /api/models for dropdown autocomplete - Added datalist-based provider selector in overrides section - Added Max Tokens configuration input - Added expandable provider list showing all detected providers - Fixed duplicate override detection * fix(chaos): fetch providers from /api/providers instead of /api/keys * fix(chaos): remove dead code isOverrideDuplicate, fix maxTokens fallback to include global config * fix(chaos): resetConfig now shows error on HTTP failure (was silent) * feat(dashboard): Chaos Mode — multi-model parallel/collaborative execution Splits the PR down to only the genuinely new Chaos Mode feature (drops the duplicate Skill Collector/GitHub-discovery portion already shipped via #6186). Replaces the loopback fetch() dispatch (hardcoded to the wrong port) with the established in-process synthetic-Request/route-handler pattern used by src/lib/batches/dispatch.ts, moves settings persistence off raw SQL, and adds unit test coverage for chaosConfig, chaosExecutor and the 3 chaos API routes. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * fix(chaos): fix external Bearer-auth bypass and stale config cache in tests validateApiKey() returns a plain boolean for both the deployment-time env key and a DB-backed key, so branching on `keyInfo === true` in verifyChaosKey() (src/app/api/skills/collect/chaos/route.ts) treated every valid API key as having full env-key access, silently skipping the chaosModeEnabled permission check entirely. Now always resolves through getApiKeyMetadata() and only bypasses the per-key check for the synthesized env-key record (id: "env-key"). Also exports invalidateChaosConfigCache() from chaosConfig.ts and wires it into the route tests' resetStorage() — the in-process config cache was surviving DB resets between tests, causing state to leak across cases. Fixes CHANGELOG-eat from the release merge (re-inserted the Chaos Mode bullet against the base CHANGELOG.md, verified additive via check-changelog-integrity.mjs) and re-syncs against release/v3.8.47 tip. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * docs(changelog): Chaos Mode overhaul bullet referencing #6728 after release sync Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * fix(merge): restore #6126 clinepass files reverted by release auto-resolve + baseline re-merge The release sync's auto-resolve reverted sibling PR #6126's clinepass work (registry, catalog, oauth constants, clineAuth.ts, token-refresh case, tests) and the file-size baseline — all outside this PR's scope. Restored to the release versions, re-applied only this PR's own baseline entries, restored the #6126 CHANGELOG bullet (re-inserting only this PR's own). Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * fix(dashboard): chaos client hook must not import the server Pino logger useChaosConfigData ("use client") pulled @/sse/utils/logger → shared Pino → logRotation/dataPaths → node:fs into the browser bundle, breaking next build (Turbopack: Can't resolve 'fs') — caught by the DAST smoke's isolated build. console.error matches every other dashboard client component. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * test(api-manager): align switch-count invariant with the extracted toggle components The Self-service block now renders 4 inline switches; the #5731 quota-bypass and #6728 chaos-access toggles were extracted into dedicated components. The type="button" invariant is preserved AND extended: the test now also asserts each extracted component's switches declare type="button". Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * chore(sync): merge release tip + restore own CHANGELOG bullet Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> --------- Co-authored-by: Moseyuh333 <Moseyuh333@users.noreply.github.com> Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com>
This commit is contained in:
@@ -18,6 +18,7 @@ _Living section — bullets land here as PRs merge into `release/v3.8.47` (paral
|
||||
- **ClinePass dual-auth**: ClinePass now offers both sign-in methods on its dashboard page — OAuth (reusing the Cline WorkOS flow) as the primary "Connect" path, or a pasted BYOK API key via "Manual API key", instead of only the API-key-only provider shipped in #5942. The registry alias was aligned to `cp` (matching the `OAUTH_PROVIDERS` catalog alias) so `<alias>/<modelId>` routing resolves correctly, the OAuth refresh dispatch now routes `clinepass` to the shared Cline refresh flow, and the duplicate API-key-only catalog entry was removed to keep ClinePass listed once. Regression guard: `tests/unit/clinepass-provider.test.ts`. (#6126 — thanks @hajilok)
|
||||
- **feat(oauth):** Kiro/Amazon Q auto-import now supports enterprise **External IdP** ("Your organization") logins via Microsoft Entra/Okta/Auth0/OneLogin/Ping/Google/Cognito — these org-issued tokens are not AWS SSO tokens (no `aorAAAAAG`-prefixed refresh token) and can't refresh through the AWS OIDC/Kiro-social path, so `tryAwsSsoCache()` now detects them (`authMethod`/`provider === "externalidp"`) and refreshes via the org IdP's own `tokenEndpoint` (public-client OAuth2 refresh grant, no client secret), persisting `TokenType: EXTERNAL_IDP` gating so the runtime executor sends the header the AWS CodeWhisperer API requires for these accounts; `tokenEndpoint` is SSRF-guarded against an HTTPS + known-IdP-host-suffix allowlist. (#6363 — thanks @artickc)
|
||||
- **Kiro long-lived API key auth**: new `/api/oauth/kiro/api-key` route + `KiroService.validateApiKey` let a Kiro account be linked with a long-lived AWS CodeWhisperer/Kiro API key instead of the interactive OAuth device flow, with live per-account model discovery (`ListAvailableModels`, 5-minute cache) layered over the existing static registry fallback (#6587 — thanks @strangersp)
|
||||
- **Chaos Mode**: multi-model parallel/collaborative task execution — dispatches a task to every active provider connection at once (parallel) or chains outputs sequentially so each model builds on the previous one's answer (collaborative), configurable via Dashboard → Chaos Mode (`GET`/`PUT`/`DELETE /api/chaos/config`) and gated per-API-key via a new `chaosModeEnabled` permission (opt-in — disabled by default globally and per key). `POST /api/chaos/run` (dashboard session) and `POST /api/skills/collect/chaos` (external Bearer-token) delegate to a shared `executeChaosRun()` engine (`src/lib/chaos/chaosExecutor.ts`) that dispatches in-process via the established synthetic-Request/route-handler pattern (no network hop, no hardcoded port), with a concurrency cap (max 10 parallel), configurable `max_tokens` (256–128k), a clear error when `stream` is requested, and collaborative-chain info (provider order + input size). Fixes external Bearer-auth bypass and stale config-cache leakage. Regression guard: `tests/unit/chaos-config.test.ts`, `tests/unit/chaos-executor.test.ts`, `tests/unit/chaos-api-routes.test.ts`. ([#6728](https://github.com/diegosouzapw/OmniRoute/pull/6728) — thanks @Moseyuh333)
|
||||
|
||||
### 🐛 Bug Fixes
|
||||
|
||||
|
||||
@@ -25,6 +25,8 @@ import { SELF_ACCOUNT_QUOTA_SCOPE, SELF_USAGE_SCOPE } from "@/shared/constants/s
|
||||
import { extractApiErrorMessage } from "@/shared/http/apiErrorMessage";
|
||||
import { hasProviderQuotaBypassScope } from "@/shared/constants/apiKeyPolicyScopes";
|
||||
import { UsageLimitSettings } from "./components/UsageLimitSettings";
|
||||
import { ChaosModeAccessToggle } from "./components/ChaosModeAccessToggle";
|
||||
import { BypassProviderQuotaToggle } from "./components/BypassProviderQuotaToggle";
|
||||
|
||||
// Constants for validation
|
||||
const MAX_KEY_NAME_LENGTH = 200;
|
||||
@@ -125,6 +127,7 @@ interface ApiKey {
|
||||
streamDefaultMode?: StreamDefaultMode;
|
||||
disableNonPublicModels?: boolean;
|
||||
allowUsageCommand?: boolean;
|
||||
chaosModeEnabled?: boolean;
|
||||
usageLimitEnabled?: boolean;
|
||||
dailyUsageLimitUsd?: number | null;
|
||||
weeklyUsageLimitUsd?: number | null;
|
||||
@@ -518,7 +521,8 @@ export default function ApiManagerPageClient() {
|
||||
const res = await fetch(`/api/keys/${encodeURIComponent(key.id)}/devices`);
|
||||
if (!res.ok) return [key.id, 0] as const;
|
||||
const data = await res.json();
|
||||
const count = typeof data?.count === "number" && Number.isFinite(data.count) ? data.count : 0;
|
||||
const count =
|
||||
typeof data?.count === "number" && Number.isFinite(data.count) ? data.count : 0;
|
||||
return [key.id, count] as const;
|
||||
} catch {
|
||||
return [key.id, 0] as const;
|
||||
@@ -791,7 +795,8 @@ export default function ApiManagerPageClient() {
|
||||
usageLimitEnabled: boolean,
|
||||
dailyUsageLimitUsd: number | null,
|
||||
weeklyUsageLimitUsd: number | null,
|
||||
blockedModels: string[]
|
||||
blockedModels: string[],
|
||||
chaosModeEnabled: boolean
|
||||
) => {
|
||||
if (!editingKey || !editingKey.id) return;
|
||||
|
||||
@@ -862,6 +867,7 @@ export default function ApiManagerPageClient() {
|
||||
usageLimitEnabled,
|
||||
dailyUsageLimitUsd,
|
||||
weeklyUsageLimitUsd,
|
||||
chaosModeEnabled,
|
||||
}),
|
||||
});
|
||||
|
||||
@@ -1645,7 +1651,8 @@ const PermissionsModal = memo(function PermissionsModal({
|
||||
usageLimitEnabled: boolean,
|
||||
dailyUsageLimitUsd: number | null,
|
||||
weeklyUsageLimitUsd: number | null,
|
||||
blockedModels: string[]
|
||||
blockedModels: string[],
|
||||
chaosModeEnabled: boolean
|
||||
) => void;
|
||||
}) {
|
||||
const t = useTranslations("apiManager");
|
||||
@@ -1731,6 +1738,7 @@ const PermissionsModal = memo(function PermissionsModal({
|
||||
const [usageCommandEnabled, setUsageCommandEnabled] = useState(
|
||||
apiKey?.allowUsageCommand === true
|
||||
);
|
||||
const [chaosModeEnabled, setChaosModeEnabled] = useState(apiKey?.chaosModeEnabled === true);
|
||||
const [usageLimitEnabled, setUsageLimitEnabled] = useState(apiKey?.usageLimitEnabled === true);
|
||||
const [dailyUsageLimitUsd, setDailyUsageLimitUsd] = useState(
|
||||
typeof apiKey?.dailyUsageLimitUsd === "number" && apiKey.dailyUsageLimitUsd > 0
|
||||
@@ -1935,7 +1943,8 @@ const PermissionsModal = memo(function PermissionsModal({
|
||||
usageLimitEnabled,
|
||||
parseUsdLimitInput(dailyUsageLimitUsd),
|
||||
parseUsdLimitInput(weeklyUsageLimitUsd),
|
||||
blockedModels
|
||||
blockedModels,
|
||||
chaosModeEnabled
|
||||
);
|
||||
}, [
|
||||
onSave,
|
||||
@@ -1974,6 +1983,7 @@ const PermissionsModal = memo(function PermissionsModal({
|
||||
parseUsdLimitInput,
|
||||
blockedClaudeCodeFamilies,
|
||||
initialBlockedModels,
|
||||
chaosModeEnabled,
|
||||
apiKey?.scopes,
|
||||
t,
|
||||
]);
|
||||
@@ -2563,30 +2573,17 @@ const PermissionsModal = memo(function PermissionsModal({
|
||||
/>
|
||||
</div>
|
||||
|
||||
{/* Chaos Mode Access Toggle */}
|
||||
<ChaosModeAccessToggle
|
||||
enabled={chaosModeEnabled}
|
||||
onToggle={() => setChaosModeEnabled((prev) => !prev)}
|
||||
/>
|
||||
|
||||
{/* Advanced Provider Quota Policy Override */}
|
||||
<div className="flex items-start justify-between gap-3 p-3 rounded-lg border border-amber-500/20 bg-amber-500/5">
|
||||
<div className="flex flex-col gap-1 pr-2">
|
||||
<p className="text-sm font-medium text-text-main">Bypass provider quota cutoffs</p>
|
||||
<p className="text-xs text-text-muted">
|
||||
Allows this key to ignore upstream provider/account cutoff policy during routing. API
|
||||
key USD quotas still apply.
|
||||
</p>
|
||||
</div>
|
||||
<button
|
||||
type="button"
|
||||
role="switch"
|
||||
aria-checked={bypassProviderQuotaPolicyEnabled}
|
||||
onClick={() => setBypassProviderQuotaPolicyEnabled((prev) => !prev)}
|
||||
className={`inline-flex shrink-0 items-center gap-1.5 px-2.5 py-1.5 rounded-md text-xs font-semibold transition-colors ${
|
||||
bypassProviderQuotaPolicyEnabled
|
||||
? "bg-amber-500/15 text-amber-700 dark:text-amber-300 border border-amber-500/30"
|
||||
: "bg-black/5 dark:bg-white/5 text-text-muted border border-border"
|
||||
}`}
|
||||
>
|
||||
<span className="material-symbols-outlined text-[14px]">alt_route</span>
|
||||
{bypassProviderQuotaPolicyEnabled ? tc("enabled") : tc("disabled")}
|
||||
</button>
|
||||
</div>
|
||||
<BypassProviderQuotaToggle
|
||||
enabled={bypassProviderQuotaPolicyEnabled}
|
||||
onToggle={() => setBypassProviderQuotaPolicyEnabled((prev) => !prev)}
|
||||
/>
|
||||
|
||||
{/* Disable Non-Public Models Toggle */}
|
||||
<div className="flex items-start justify-between gap-3 p-3 rounded-lg border border-border bg-surface/40">
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
"use client";
|
||||
|
||||
import { useTranslations } from "next-intl";
|
||||
|
||||
/**
|
||||
* "Bypass provider quota cutoffs" toggle for the API Key permissions modal.
|
||||
* Extracted out of ApiManagerPageClient.tsx (frozen god-file — see
|
||||
* config/quality/file-size-baseline.json) following the same pattern as
|
||||
* UsageLimitSettings.tsx — pure UI move, no behavior change.
|
||||
*/
|
||||
export function BypassProviderQuotaToggle({
|
||||
enabled,
|
||||
onToggle,
|
||||
}: {
|
||||
enabled: boolean;
|
||||
onToggle: () => void;
|
||||
}) {
|
||||
const tc = useTranslations("common");
|
||||
|
||||
return (
|
||||
<div className="flex items-start justify-between gap-3 p-3 rounded-lg border border-amber-500/20 bg-amber-500/5">
|
||||
<div className="flex flex-col gap-1 pr-2">
|
||||
<p className="text-sm font-medium text-text-main">Bypass provider quota cutoffs</p>
|
||||
<p className="text-xs text-text-muted">
|
||||
Allows this key to ignore upstream provider/account cutoff policy during routing. API key
|
||||
USD quotas still apply.
|
||||
</p>
|
||||
</div>
|
||||
<button
|
||||
type="button"
|
||||
role="switch"
|
||||
aria-checked={enabled}
|
||||
onClick={onToggle}
|
||||
className={`inline-flex shrink-0 items-center gap-1.5 px-2.5 py-1.5 rounded-md text-xs font-semibold transition-colors ${
|
||||
enabled
|
||||
? "bg-amber-500/15 text-amber-700 dark:text-amber-300 border border-amber-500/30"
|
||||
: "bg-black/5 dark:bg-white/5 text-text-muted border border-border"
|
||||
}`}
|
||||
>
|
||||
<span className="material-symbols-outlined text-[14px]">alt_route</span>
|
||||
{enabled ? tc("enabled") : tc("disabled")}
|
||||
</button>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
"use client";
|
||||
|
||||
import { useTranslations } from "next-intl";
|
||||
|
||||
/**
|
||||
* Chaos Mode access toggle for the API Key permissions modal — gates a single
|
||||
* API key's ability to call the Chaos Mode dispatch endpoints
|
||||
* (`POST /api/chaos/run`, `POST /api/skills/collect/chaos`) via the
|
||||
* `chaosModeEnabled` permission. Extracted out of ApiManagerPageClient.tsx
|
||||
* (frozen god-file — see config/quality/file-size-baseline.json) following the
|
||||
* same pattern as UsageLimitSettings.tsx.
|
||||
*/
|
||||
export function ChaosModeAccessToggle({
|
||||
enabled,
|
||||
onToggle,
|
||||
}: {
|
||||
enabled: boolean;
|
||||
onToggle: () => void;
|
||||
}) {
|
||||
const tChaos = useTranslations("chaosConfig");
|
||||
const tc = useTranslations("common");
|
||||
|
||||
return (
|
||||
<div className="flex items-start justify-between gap-3 p-3 rounded-lg border border-amber-500/20 bg-amber-500/5">
|
||||
<div className="flex flex-col gap-1">
|
||||
<p className="text-sm font-medium text-text-main">{tChaos("keyPermission")}</p>
|
||||
<p className="text-xs text-text-muted">{tChaos("keyPermissionDesc")}</p>
|
||||
</div>
|
||||
<button
|
||||
type="button"
|
||||
role="switch"
|
||||
aria-checked={enabled}
|
||||
onClick={onToggle}
|
||||
className={`inline-flex shrink-0 items-center gap-1.5 px-2.5 py-1.5 rounded-md text-xs font-semibold transition-colors ${
|
||||
enabled
|
||||
? "bg-amber-500/15 text-amber-700 dark:text-amber-300 border border-amber-500/30"
|
||||
: "bg-black/5 dark:bg-white/5 text-text-muted border border-border"
|
||||
}`}
|
||||
>
|
||||
<span className="material-symbols-outlined text-[14px]">blender</span>
|
||||
{tChaos("pageTitle")} - {enabled ? tc("enabled") : tc("disabled")}
|
||||
</button>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
107
src/app/(dashboard)/dashboard/chaos/ChaosConfigPageClient.tsx
Normal file
107
src/app/(dashboard)/dashboard/chaos/ChaosConfigPageClient.tsx
Normal file
@@ -0,0 +1,107 @@
|
||||
/**
|
||||
* /dashboard/chaos — Chaos Mode Configuration Page
|
||||
*
|
||||
* Allows users to:
|
||||
* - Enable/disable chaos mode globally
|
||||
* - Set default mode (parallel/collaborative)
|
||||
* - Override provider models for chaos mode
|
||||
* - Set custom system prompt and max tokens
|
||||
* - Configure timeout
|
||||
* - Test chaos mode with a simple task
|
||||
*
|
||||
* State + handlers live in useChaosConfigPage.ts and the JSX sections are
|
||||
* split into ./components/* — this file stays a thin composition/render
|
||||
* function under the complexity/size ratchet
|
||||
* (config/quality/complexity-baseline.json).
|
||||
*/
|
||||
"use client";
|
||||
|
||||
import { useChaosConfigPage } from "./useChaosConfigPage";
|
||||
import { ChaosModeSelector } from "./components/ChaosModeSelector";
|
||||
import { ChaosTestResultsPanel } from "./components/ChaosTestResultsPanel";
|
||||
import { ChaosProviderOverridesPanel } from "./components/ChaosProviderOverridesPanel";
|
||||
import { ChaosBasicSettingsFields } from "./components/ChaosBasicSettingsFields";
|
||||
import { ChaosConfigActionsBar } from "./components/ChaosConfigActionsBar";
|
||||
import { ChaosStatusMessage } from "./components/ChaosStatusMessage";
|
||||
|
||||
export default function ChaosConfigPage() {
|
||||
const {
|
||||
t,
|
||||
config,
|
||||
setConfig,
|
||||
availableProviders,
|
||||
loading,
|
||||
saving,
|
||||
testing,
|
||||
testResult,
|
||||
message,
|
||||
saveConfig,
|
||||
resetConfig,
|
||||
testChaos,
|
||||
addOverride,
|
||||
updateOverride,
|
||||
removeOverride,
|
||||
} = useChaosConfigPage();
|
||||
|
||||
if (loading) {
|
||||
return (
|
||||
<div className="flex items-center justify-center h-64">
|
||||
<div className="text-text-muted animate-pulse">{t("loadingProviderModels")}</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="max-w-3xl mx-auto p-4 sm:p-6 space-y-6">
|
||||
{/* Header */}
|
||||
<div>
|
||||
<h1 className="text-2xl font-bold text-text-main">{t("pageTitle")}</h1>
|
||||
<p className="text-sm text-text-muted mt-1">{t("pageSubtitle")}</p>
|
||||
</div>
|
||||
|
||||
{/* Status Message */}
|
||||
<ChaosStatusMessage message={message} />
|
||||
|
||||
{/* Enable toggle + timeout + max tokens + system prompt */}
|
||||
<ChaosBasicSettingsFields
|
||||
settings={config}
|
||||
onChange={(patch) => setConfig((prev) => ({ ...prev, ...patch }))}
|
||||
/>
|
||||
|
||||
{/* Default Mode Selector */}
|
||||
<ChaosModeSelector
|
||||
mode={config.defaultMode}
|
||||
onChange={(defaultMode) => setConfig((prev) => ({ ...prev, defaultMode }))}
|
||||
label={t("mode")}
|
||||
parallelLabel={t("modeParallel")}
|
||||
parallelDesc={t("modeParallelDesc")}
|
||||
collaborativeLabel={t("modeCollaborative")}
|
||||
collaborativeDesc={t("modeCollaborativeDesc")}
|
||||
/>
|
||||
|
||||
<ChaosConfigActionsBar
|
||||
saving={saving}
|
||||
testing={testing}
|
||||
testDisabled={!config.enabled}
|
||||
onSave={saveConfig}
|
||||
onReset={resetConfig}
|
||||
onTest={testChaos}
|
||||
/>
|
||||
|
||||
{/* Test Results */}
|
||||
{testResult && <ChaosTestResultsPanel result={testResult} />}
|
||||
|
||||
{/* Provider Overrides */}
|
||||
<ChaosProviderOverridesPanel
|
||||
overrides={config.providerOverrides}
|
||||
availableProviders={availableProviders}
|
||||
title={t("providerOverrides")}
|
||||
description={t("providerOverridesDesc")}
|
||||
addLabel={t("addProvider")}
|
||||
onAdd={addOverride}
|
||||
onUpdate={updateOverride}
|
||||
onRemove={removeOverride}
|
||||
/>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
28
src/app/(dashboard)/dashboard/chaos/chaosPageTypes.ts
Normal file
28
src/app/(dashboard)/dashboard/chaos/chaosPageTypes.ts
Normal file
@@ -0,0 +1,28 @@
|
||||
import type { ChaosProviderOverride } from "./components/ChaosProviderOverridesPanel";
|
||||
|
||||
export interface ChaosProviderInfo {
|
||||
id: string;
|
||||
name: string;
|
||||
provider: string;
|
||||
defaultModel: string | null;
|
||||
}
|
||||
|
||||
export interface ChaosPageConfig {
|
||||
enabled: boolean;
|
||||
defaultMode: "parallel" | "collaborative";
|
||||
providerOverrides: ChaosProviderOverride[];
|
||||
systemPrompt?: string;
|
||||
timeoutMs: number;
|
||||
maxTokens: number;
|
||||
}
|
||||
|
||||
export const DEFAULT_CHAOS_PAGE_CONFIG: ChaosPageConfig = {
|
||||
enabled: false,
|
||||
defaultMode: "parallel",
|
||||
providerOverrides: [],
|
||||
systemPrompt: "",
|
||||
timeoutMs: 120_000,
|
||||
maxTokens: 4096,
|
||||
};
|
||||
|
||||
export type ChaosPageMessage = { type: "success" | "error"; text: string } | null;
|
||||
@@ -0,0 +1,122 @@
|
||||
"use client";
|
||||
|
||||
import { useTranslations } from "next-intl";
|
||||
|
||||
export interface ChaosBasicSettings {
|
||||
enabled: boolean;
|
||||
timeoutMs: number;
|
||||
maxTokens: number;
|
||||
systemPrompt?: string;
|
||||
}
|
||||
|
||||
function ChaosSystemPromptField({
|
||||
value,
|
||||
onChange,
|
||||
}: {
|
||||
value: string | undefined;
|
||||
onChange: (value: string) => void;
|
||||
}) {
|
||||
const t = useTranslations("chaosConfig");
|
||||
return (
|
||||
<div className="p-3 rounded-lg border border-border bg-surface/40">
|
||||
<p className="text-sm font-medium text-text-main">{t("systemPrompt")}</p>
|
||||
<p className="text-xs text-text-muted mb-2">{t("systemPromptDesc")}</p>
|
||||
<textarea
|
||||
value={value || ""}
|
||||
onChange={(e) => onChange(e.target.value)}
|
||||
rows={3}
|
||||
className="w-full px-3 py-1.5 rounded-md border border-border bg-surface text-sm text-text-main resize-y"
|
||||
placeholder="Optional: override the default chaos mode system prompt..."
|
||||
/>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Enable toggle + timeout + max tokens + system prompt fields for the Chaos
|
||||
* Mode config page. Extracted out of ChaosConfigPageClient.tsx to keep the
|
||||
* page component under the complexity/size ratchet
|
||||
* (config/quality/complexity-baseline.json).
|
||||
*/
|
||||
export function ChaosBasicSettingsFields({
|
||||
settings,
|
||||
onChange,
|
||||
}: {
|
||||
settings: ChaosBasicSettings;
|
||||
onChange: (patch: Partial<ChaosBasicSettings>) => void;
|
||||
}) {
|
||||
const t = useTranslations("chaosConfig");
|
||||
|
||||
return (
|
||||
<>
|
||||
{/* Enable/Disable Toggle */}
|
||||
<div className="flex items-start justify-between gap-3 p-3 rounded-lg border border-border bg-surface/40">
|
||||
<div className="flex flex-col gap-1">
|
||||
<p className="text-sm font-medium text-text-main">{t("enableChaos")}</p>
|
||||
<p className="text-xs text-text-muted">{t("enableChaosDesc")}</p>
|
||||
</div>
|
||||
<button
|
||||
type="button"
|
||||
role="switch"
|
||||
aria-checked={settings.enabled}
|
||||
onClick={() => onChange({ enabled: !settings.enabled })}
|
||||
className={`inline-flex shrink-0 items-center gap-1.5 px-2.5 py-1.5 rounded-md text-xs font-semibold transition-colors ${
|
||||
settings.enabled
|
||||
? "bg-amber-500/15 text-amber-700 dark:text-amber-300 border border-amber-500/30"
|
||||
: "bg-black/5 dark:bg-white/5 text-text-muted border border-border"
|
||||
}`}
|
||||
>
|
||||
<span className="material-symbols-outlined text-[14px]">
|
||||
{settings.enabled ? "toggle_on" : "toggle_off"}
|
||||
</span>
|
||||
{settings.enabled ? "Enabled" : "Disabled"}
|
||||
</button>
|
||||
</div>
|
||||
|
||||
{/* Timeout */}
|
||||
<div className="p-3 rounded-lg border border-border bg-surface/40">
|
||||
<p className="text-sm font-medium text-text-main">{t("timeout")}</p>
|
||||
<p className="text-xs text-text-muted mb-2">{t("timeoutDesc")}</p>
|
||||
<input
|
||||
type="number"
|
||||
min={5000}
|
||||
max={600000}
|
||||
step={5000}
|
||||
value={settings.timeoutMs}
|
||||
onChange={(e) =>
|
||||
onChange({
|
||||
timeoutMs: Math.max(5000, Math.min(600000, Number(e.target.value) || 120000)),
|
||||
})
|
||||
}
|
||||
className="w-full px-3 py-1.5 rounded-md border border-border bg-surface text-sm text-text-main"
|
||||
/>
|
||||
</div>
|
||||
|
||||
{/* Max Tokens */}
|
||||
<div className="p-3 rounded-lg border border-border bg-surface/40">
|
||||
<p className="text-sm font-medium text-text-main">Max Tokens</p>
|
||||
<p className="text-xs text-text-muted mb-2">
|
||||
Maximum tokens per model response. Higher values cost more and take longer.
|
||||
</p>
|
||||
<input
|
||||
type="number"
|
||||
min={256}
|
||||
max={128000}
|
||||
step={256}
|
||||
value={settings.maxTokens}
|
||||
onChange={(e) =>
|
||||
onChange({
|
||||
maxTokens: Math.max(256, Math.min(128000, Number(e.target.value) || 4096)),
|
||||
})
|
||||
}
|
||||
className="w-full px-3 py-1.5 rounded-md border border-border bg-surface text-sm text-text-main"
|
||||
/>
|
||||
</div>
|
||||
|
||||
<ChaosSystemPromptField
|
||||
value={settings.systemPrompt}
|
||||
onChange={(systemPrompt) => onChange({ systemPrompt })}
|
||||
/>
|
||||
</>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,74 @@
|
||||
"use client";
|
||||
|
||||
import { useTranslations } from "next-intl";
|
||||
|
||||
/**
|
||||
* Save/Reset + Test-run action buttons for the Chaos Mode config page.
|
||||
* Extracted out of ChaosConfigPageClient.tsx to keep the page component under
|
||||
* the complexity/size ratchet (config/quality/complexity-baseline.json).
|
||||
*/
|
||||
export function ChaosConfigActionsBar({
|
||||
saving,
|
||||
testing,
|
||||
testDisabled,
|
||||
onSave,
|
||||
onReset,
|
||||
onTest,
|
||||
}: {
|
||||
saving: boolean;
|
||||
testing: boolean;
|
||||
testDisabled: boolean;
|
||||
onSave: () => void;
|
||||
onReset: () => void;
|
||||
onTest: () => void;
|
||||
}) {
|
||||
const t = useTranslations("chaosConfig");
|
||||
|
||||
return (
|
||||
<>
|
||||
{/* Actions */}
|
||||
<div className="flex gap-3">
|
||||
<button
|
||||
type="button"
|
||||
onClick={onSave}
|
||||
disabled={saving}
|
||||
className="flex items-center gap-2 px-4 py-2 rounded-lg bg-primary text-white text-sm font-semibold hover:opacity-90 disabled:opacity-50"
|
||||
>
|
||||
{saving ? (
|
||||
<span className="material-symbols-outlined text-[16px] animate-spin">sync</span>
|
||||
) : (
|
||||
<span className="material-symbols-outlined text-[16px]">save</span>
|
||||
)}
|
||||
{t("saveConfig")}
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
onClick={onReset}
|
||||
disabled={saving}
|
||||
className="flex items-center gap-2 px-4 py-2 rounded-lg border border-border text-text-muted text-sm hover:bg-black/5 dark:hover:bg-white/5 disabled:opacity-50"
|
||||
>
|
||||
<span className="material-symbols-outlined text-[16px]">restart_alt</span>
|
||||
{t("configReset")}
|
||||
</button>
|
||||
</div>
|
||||
|
||||
{/* Test Button */}
|
||||
<div className="p-3 rounded-lg border border-border bg-surface/40">
|
||||
<p className="text-sm font-medium text-text-main mb-2">{t("testButton")}</p>
|
||||
<button
|
||||
type="button"
|
||||
onClick={onTest}
|
||||
disabled={testing || testDisabled}
|
||||
className="flex items-center gap-2 px-4 py-2 rounded-lg bg-amber-500/15 text-amber-700 dark:text-amber-300 border border-amber-500/30 text-sm font-semibold hover:bg-amber-500/25 disabled:opacity-50"
|
||||
>
|
||||
{testing ? (
|
||||
<span className="material-symbols-outlined text-[16px] animate-spin">sync</span>
|
||||
) : (
|
||||
<span className="material-symbols-outlined text-[16px]">play_arrow</span>
|
||||
)}
|
||||
{testing ? "Running..." : t("testButton")}
|
||||
</button>
|
||||
</div>
|
||||
</>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,60 @@
|
||||
"use client";
|
||||
|
||||
/**
|
||||
* Parallel / Collaborative mode selector for the Chaos Mode config page.
|
||||
* Extracted out of ChaosConfigPageClient.tsx to keep the page component under
|
||||
* the complexity/size ratchet (config/quality/complexity-baseline.json).
|
||||
*/
|
||||
export function ChaosModeSelector({
|
||||
mode,
|
||||
onChange,
|
||||
label,
|
||||
parallelLabel,
|
||||
parallelDesc,
|
||||
collaborativeLabel,
|
||||
collaborativeDesc,
|
||||
}: {
|
||||
mode: "parallel" | "collaborative";
|
||||
onChange: (mode: "parallel" | "collaborative") => void;
|
||||
label: string;
|
||||
parallelLabel: string;
|
||||
parallelDesc: string;
|
||||
collaborativeLabel: string;
|
||||
collaborativeDesc: string;
|
||||
}) {
|
||||
return (
|
||||
<div className="p-3 rounded-lg border border-border bg-surface/40">
|
||||
<p className="text-sm font-medium text-text-main mb-2">{label}</p>
|
||||
<div className="flex gap-2">
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => onChange("parallel")}
|
||||
className={`flex-1 px-3 py-2 rounded-md text-xs font-semibold transition-all ${
|
||||
mode === "parallel"
|
||||
? "bg-primary text-white"
|
||||
: "bg-black/5 dark:bg-white/5 text-text-muted hover:bg-black/10 dark:hover:bg-white/10"
|
||||
}`}
|
||||
>
|
||||
<span className="material-symbols-outlined text-[16px] align-middle mr-1">
|
||||
call_split
|
||||
</span>
|
||||
{parallelLabel}
|
||||
<p className="text-[10px] opacity-70 mt-0.5">{parallelDesc}</p>
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => onChange("collaborative")}
|
||||
className={`flex-1 px-3 py-2 rounded-md text-xs font-semibold transition-all ${
|
||||
mode === "collaborative"
|
||||
? "bg-primary text-white"
|
||||
: "bg-black/5 dark:bg-white/5 text-text-muted hover:bg-black/10 dark:hover:bg-white/10"
|
||||
}`}
|
||||
>
|
||||
<span className="material-symbols-outlined text-[16px] align-middle mr-1">merge</span>
|
||||
{collaborativeLabel}
|
||||
<p className="text-[10px] opacity-70 mt-0.5">{collaborativeDesc}</p>
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,156 @@
|
||||
"use client";
|
||||
|
||||
export interface ChaosProviderInfo {
|
||||
id: string;
|
||||
name: string;
|
||||
provider: string;
|
||||
defaultModel: string | null;
|
||||
}
|
||||
|
||||
export interface ChaosProviderOverride {
|
||||
providerId: string;
|
||||
modelId?: string;
|
||||
enabled: boolean;
|
||||
}
|
||||
|
||||
type UpdateOverride = (index: number, field: keyof ChaosProviderOverride, value: any) => void;
|
||||
|
||||
function ChaosProviderOverrideRow({
|
||||
override,
|
||||
index,
|
||||
availableProviders,
|
||||
onUpdate,
|
||||
onRemove,
|
||||
}: {
|
||||
override: ChaosProviderOverride;
|
||||
index: number;
|
||||
availableProviders: ChaosProviderInfo[];
|
||||
onUpdate: UpdateOverride;
|
||||
onRemove: (index: number) => void;
|
||||
}) {
|
||||
return (
|
||||
<div className="flex items-center gap-2 p-2 rounded-md bg-black/5 dark:bg-white/5">
|
||||
{/* Provider dropdown with available options */}
|
||||
<div className="flex-1 relative">
|
||||
<input
|
||||
type="text"
|
||||
list={`provider-list-${index}`}
|
||||
placeholder="Provider ID (type or select)"
|
||||
value={override.providerId}
|
||||
onChange={(e) => onUpdate(index, "providerId", e.target.value)}
|
||||
className="w-full px-2 py-1 rounded border border-border bg-surface text-xs text-text-main"
|
||||
/>
|
||||
<datalist id={`provider-list-${index}`}>
|
||||
{availableProviders.map((p) => (
|
||||
<option key={p.id} value={p.provider} />
|
||||
))}
|
||||
</datalist>
|
||||
</div>
|
||||
<input
|
||||
type="text"
|
||||
placeholder="Model ID (optional)"
|
||||
value={override.modelId || ""}
|
||||
onChange={(e) => onUpdate(index, "modelId", e.target.value)}
|
||||
className="flex-1 px-2 py-1 rounded border border-border bg-surface text-xs text-text-main"
|
||||
/>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => onUpdate(index, "enabled", !override.enabled)}
|
||||
className={`px-2 py-1 rounded text-xs ${
|
||||
override.enabled ? "bg-green-500/10 text-green-600" : "bg-red-500/10 text-red-600"
|
||||
}`}
|
||||
>
|
||||
{override.enabled ? "ON" : "OFF"}
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => onRemove(index)}
|
||||
className="px-2 py-1 rounded text-xs text-red-500 hover:bg-red-500/10"
|
||||
>
|
||||
<span className="material-symbols-outlined text-[14px]">close</span>
|
||||
</button>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function ChaosAvailableProvidersHint({ availableProviders }: { availableProviders: ChaosProviderInfo[] }) {
|
||||
if (availableProviders.length === 0) return null;
|
||||
return (
|
||||
<details className="text-xs text-text-muted">
|
||||
<summary className="cursor-pointer hover:text-text-main">
|
||||
Available providers ({availableProviders.length})
|
||||
</summary>
|
||||
<div className="mt-1 flex flex-wrap gap-1">
|
||||
{availableProviders.map((p) => (
|
||||
<span key={p.id} className="px-1.5 py-0.5 rounded bg-black/5 dark:bg-white/5">
|
||||
{p.provider}
|
||||
{p.defaultModel && <span className="opacity-60 ml-1">({p.defaultModel})</span>}
|
||||
</span>
|
||||
))}
|
||||
</div>
|
||||
</details>
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-provider model override editor for the Chaos Mode config page.
|
||||
* Extracted out of ChaosConfigPageClient.tsx to keep the page component under
|
||||
* the complexity/size ratchet (config/quality/complexity-baseline.json).
|
||||
*/
|
||||
export function ChaosProviderOverridesPanel({
|
||||
overrides,
|
||||
availableProviders,
|
||||
title,
|
||||
description,
|
||||
addLabel,
|
||||
onAdd,
|
||||
onUpdate,
|
||||
onRemove,
|
||||
}: {
|
||||
overrides: ChaosProviderOverride[];
|
||||
availableProviders: ChaosProviderInfo[];
|
||||
title: string;
|
||||
description: string;
|
||||
addLabel: string;
|
||||
onAdd: () => void;
|
||||
onUpdate: UpdateOverride;
|
||||
onRemove: (index: number) => void;
|
||||
}) {
|
||||
return (
|
||||
<div className="p-3 rounded-lg border border-border bg-surface/40 space-y-3">
|
||||
<div className="flex items-center justify-between">
|
||||
<div>
|
||||
<p className="text-sm font-medium text-text-main">{title}</p>
|
||||
<p className="text-xs text-text-muted">{description}</p>
|
||||
</div>
|
||||
<button
|
||||
type="button"
|
||||
onClick={onAdd}
|
||||
className="flex items-center gap-1 px-2 py-1 rounded-md text-xs font-semibold bg-primary/10 text-primary hover:bg-primary/20"
|
||||
>
|
||||
<span className="material-symbols-outlined text-[14px]">add</span>
|
||||
{addLabel}
|
||||
</button>
|
||||
</div>
|
||||
|
||||
{overrides.length === 0 && (
|
||||
<p className="text-xs text-text-muted italic">
|
||||
No overrides — all active providers will participate with their default models
|
||||
</p>
|
||||
)}
|
||||
|
||||
{overrides.map((override, idx) => (
|
||||
<ChaosProviderOverrideRow
|
||||
key={idx}
|
||||
override={override}
|
||||
index={idx}
|
||||
availableProviders={availableProviders}
|
||||
onUpdate={onUpdate}
|
||||
onRemove={onRemove}
|
||||
/>
|
||||
))}
|
||||
|
||||
<ChaosAvailableProvidersHint availableProviders={availableProviders} />
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
"use client";
|
||||
|
||||
import type { ChaosPageMessage } from "../chaosPageTypes";
|
||||
|
||||
/**
|
||||
* Success/error status banner for the Chaos Mode config page. Extracted out
|
||||
* of ChaosConfigPageClient.tsx to keep the page component under the
|
||||
* complexity/size ratchet (config/quality/complexity-baseline.json).
|
||||
*/
|
||||
export function ChaosStatusMessage({ message }: { message: ChaosPageMessage }) {
|
||||
if (!message) return null;
|
||||
return (
|
||||
<div
|
||||
className={`p-3 rounded-lg text-sm font-medium ${
|
||||
message.type === "success"
|
||||
? "bg-green-500/10 text-green-700 dark:text-green-300 border border-green-500/20"
|
||||
: "bg-red-500/10 text-red-700 dark:text-red-300 border border-red-500/20"
|
||||
}`}
|
||||
>
|
||||
{message.text}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
"use client";
|
||||
|
||||
export interface ChaosModelResult {
|
||||
providerId: string;
|
||||
providerName: string;
|
||||
modelId: string;
|
||||
status: "success" | "error" | "skipped";
|
||||
content: string | null;
|
||||
error?: string;
|
||||
durationMs: number;
|
||||
}
|
||||
|
||||
export interface ChaosTestResult {
|
||||
task: string;
|
||||
mode: string;
|
||||
startedAt: string;
|
||||
totalProviders: number;
|
||||
totalResults: number;
|
||||
models: ChaosModelResult[];
|
||||
summary?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Test-run results panel for the Chaos Mode config page. Extracted out of
|
||||
* ChaosConfigPageClient.tsx to keep the page component under the
|
||||
* complexity/size ratchet (config/quality/complexity-baseline.json).
|
||||
*/
|
||||
export function ChaosTestResultsPanel({ result }: { result: ChaosTestResult }) {
|
||||
return (
|
||||
<div className="p-3 rounded-lg border border-border bg-surface/40 space-y-3">
|
||||
<h3 className="text-sm font-bold text-text-main">
|
||||
Test Results — {result.mode} mode ({result.totalProviders} providers)
|
||||
</h3>
|
||||
<div className="text-xs text-text-muted">
|
||||
Started: {new Date(result.startedAt).toLocaleTimeString()}
|
||||
</div>
|
||||
{result.models.map((model, idx) => (
|
||||
<div
|
||||
key={idx}
|
||||
className={`p-2 rounded-md text-xs ${
|
||||
model.status === "success"
|
||||
? "bg-green-500/5 border border-green-500/20"
|
||||
: "bg-red-500/5 border border-red-500/20"
|
||||
}`}
|
||||
>
|
||||
<div className="font-medium text-text-main">
|
||||
[{idx + 1}] {model.providerName} / {model.modelId}
|
||||
<span className="ml-2 text-text-muted">({model.durationMs}ms)</span>
|
||||
<span
|
||||
className={`ml-2 ${model.status === "success" ? "text-green-500" : "text-red-500"}`}
|
||||
>
|
||||
{model.status}
|
||||
</span>
|
||||
</div>
|
||||
{model.status === "success" && model.content && (
|
||||
<div className="mt-1 text-text-muted line-clamp-3 whitespace-pre-wrap">
|
||||
{model.content.slice(0, 300)}
|
||||
{model.content.length > 300 ? "..." : ""}
|
||||
</div>
|
||||
)}
|
||||
{model.error && <div className="mt-1 text-red-500">{model.error}</div>}
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
12
src/app/(dashboard)/dashboard/chaos/page.tsx
Normal file
12
src/app/(dashboard)/dashboard/chaos/page.tsx
Normal file
@@ -0,0 +1,12 @@
|
||||
/**
|
||||
* /dashboard/chaos/page.tsx — Chaos Mode Configuration
|
||||
*/
|
||||
import ChaosConfigPageClient from "./ChaosConfigPageClient";
|
||||
|
||||
export const metadata = {
|
||||
title: "Chaos Mode — OmniRoute",
|
||||
};
|
||||
|
||||
export default function Page() {
|
||||
return <ChaosConfigPageClient />;
|
||||
}
|
||||
96
src/app/(dashboard)/dashboard/chaos/useChaosConfigData.ts
Normal file
96
src/app/(dashboard)/dashboard/chaos/useChaosConfigData.ts
Normal file
@@ -0,0 +1,96 @@
|
||||
"use client";
|
||||
|
||||
import { useEffect, useMemo, useState } from "react";
|
||||
import {
|
||||
DEFAULT_CHAOS_PAGE_CONFIG,
|
||||
type ChaosPageConfig,
|
||||
type ChaosProviderInfo,
|
||||
} from "./chaosPageTypes";
|
||||
|
||||
function pickProviderName(conn: any): string {
|
||||
return conn.provider || conn.name || conn.id || "";
|
||||
}
|
||||
|
||||
function toProviderInfo(conn: any, providerName: string): ChaosProviderInfo {
|
||||
return {
|
||||
id: conn.id,
|
||||
name: conn.name || conn.provider || providerName,
|
||||
provider: conn.provider || providerName,
|
||||
defaultModel: conn.defaultModel || null,
|
||||
};
|
||||
}
|
||||
|
||||
/** Extract a deduplicated ChaosProviderInfo[] from the /api/providers connections payload. */
|
||||
function extractProvidersFromPayload(payload: unknown): ChaosProviderInfo[] {
|
||||
const allConnections = (payload as any)?.connections || (payload as any)?.data || payload;
|
||||
if (!Array.isArray(allConnections)) return [];
|
||||
|
||||
const extracted: ChaosProviderInfo[] = [];
|
||||
const seenProvider = new Set<string>();
|
||||
for (const conn of allConnections) {
|
||||
const providerName = pickProviderName(conn);
|
||||
if (!providerName || seenProvider.has(providerName.toLowerCase())) continue;
|
||||
seenProvider.add(providerName.toLowerCase());
|
||||
extracted.push(toProviderInfo(conn, providerName));
|
||||
}
|
||||
return extracted;
|
||||
}
|
||||
|
||||
/** Fetch the persisted chaos config, or null on failure (caller keeps the default). */
|
||||
async function fetchChaosConfig(): Promise<ChaosPageConfig | null> {
|
||||
const res = await fetch("/api/chaos/config");
|
||||
if (!res.ok) return null;
|
||||
const data = await res.json();
|
||||
return data.config || DEFAULT_CHAOS_PAGE_CONFIG;
|
||||
}
|
||||
|
||||
/** Fetch active provider connections, or [] on failure/empty. */
|
||||
async function fetchChaosProviders(): Promise<ChaosProviderInfo[]> {
|
||||
const res = await fetch("/api/providers");
|
||||
if (!res.ok) return [];
|
||||
const data = await res.json();
|
||||
return extractProvidersFromPayload(data);
|
||||
}
|
||||
|
||||
/**
|
||||
* Chaos config + provider-connection data loading for the Chaos Mode config
|
||||
* page. Extracted out of the page component to keep it under the
|
||||
* complexity/size ratchet (config/quality/complexity-baseline.json).
|
||||
*/
|
||||
export function useChaosConfigData() {
|
||||
const [config, setConfig] = useState<ChaosPageConfig>(DEFAULT_CHAOS_PAGE_CONFIG);
|
||||
const [providers, setProviders] = useState<ChaosProviderInfo[]>([]);
|
||||
const [loading, setLoading] = useState(true);
|
||||
|
||||
// Build a unique list of provider names from active connections for the dropdown
|
||||
const availableProviders = useMemo(() => {
|
||||
const seen = new Set<string>();
|
||||
return providers.filter((p) => {
|
||||
const key = p.provider || p.name || p.id;
|
||||
if (seen.has(key)) return false;
|
||||
seen.add(key);
|
||||
return true;
|
||||
});
|
||||
}, [providers]);
|
||||
|
||||
// Fetch current config + providers on mount
|
||||
useEffect(() => {
|
||||
async function load() {
|
||||
try {
|
||||
const [nextConfig, nextProviders] = await Promise.all([
|
||||
fetchChaosConfig(),
|
||||
fetchChaosProviders(),
|
||||
]);
|
||||
if (nextConfig) setConfig(nextConfig);
|
||||
if (nextProviders.length > 0) setProviders(nextProviders);
|
||||
} catch (err) {
|
||||
console.error("[chaos] Failed to load config", err);
|
||||
} finally {
|
||||
setLoading(false);
|
||||
}
|
||||
}
|
||||
load();
|
||||
}, []);
|
||||
|
||||
return { config, setConfig, availableProviders, loading };
|
||||
}
|
||||
72
src/app/(dashboard)/dashboard/chaos/useChaosConfigPage.ts
Normal file
72
src/app/(dashboard)/dashboard/chaos/useChaosConfigPage.ts
Normal file
@@ -0,0 +1,72 @@
|
||||
"use client";
|
||||
|
||||
import { useCallback } from "react";
|
||||
import type { ChaosProviderOverride } from "./components/ChaosProviderOverridesPanel";
|
||||
import { useChaosConfigData } from "./useChaosConfigData";
|
||||
import { useChaosConfigPersistence } from "./useChaosConfigPersistence";
|
||||
import { useChaosTestRun } from "./useChaosTestRun";
|
||||
|
||||
export type { ChaosProviderInfo, ChaosPageConfig } from "./chaosPageTypes";
|
||||
export { DEFAULT_CHAOS_PAGE_CONFIG } from "./chaosPageTypes";
|
||||
|
||||
/**
|
||||
* All state + handlers for the Chaos Mode config page
|
||||
* (src/app/(dashboard)/dashboard/chaos/ChaosConfigPageClient.tsx). Composes
|
||||
* useChaosConfigData/useChaosConfigPersistence/useChaosTestRun so the page
|
||||
* component itself stays a thin render function under the complexity/size
|
||||
* ratchet (config/quality/complexity-baseline.json).
|
||||
*/
|
||||
export function useChaosConfigPage() {
|
||||
const { config, setConfig, availableProviders, loading } = useChaosConfigData();
|
||||
const { t, saving, message, setMessage, saveConfig, resetConfig } = useChaosConfigPersistence(
|
||||
config,
|
||||
setConfig
|
||||
);
|
||||
const { testing, testResult, testChaos } = useChaosTestRun(config, setMessage);
|
||||
|
||||
const addOverride = useCallback(() => {
|
||||
setConfig((prev) => ({
|
||||
...prev,
|
||||
providerOverrides: [...prev.providerOverrides, { providerId: "", modelId: "", enabled: true }],
|
||||
}));
|
||||
}, [setConfig]);
|
||||
|
||||
const updateOverride = useCallback(
|
||||
(index: number, field: keyof ChaosProviderOverride, value: any) => {
|
||||
setConfig((prev) => {
|
||||
const overrides = [...prev.providerOverrides];
|
||||
overrides[index] = { ...overrides[index], [field]: value };
|
||||
return { ...prev, providerOverrides: overrides };
|
||||
});
|
||||
},
|
||||
[setConfig]
|
||||
);
|
||||
|
||||
const removeOverride = useCallback(
|
||||
(index: number) => {
|
||||
setConfig((prev) => ({
|
||||
...prev,
|
||||
providerOverrides: prev.providerOverrides.filter((_, i) => i !== index),
|
||||
}));
|
||||
},
|
||||
[setConfig]
|
||||
);
|
||||
|
||||
return {
|
||||
t,
|
||||
config,
|
||||
setConfig,
|
||||
availableProviders,
|
||||
loading,
|
||||
saving,
|
||||
testing,
|
||||
testResult,
|
||||
message,
|
||||
saveConfig,
|
||||
resetConfig,
|
||||
testChaos,
|
||||
addOverride,
|
||||
updateOverride,
|
||||
removeOverride,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
"use client";
|
||||
|
||||
import { useCallback, useState, type Dispatch, type SetStateAction } from "react";
|
||||
import { useTranslations } from "next-intl";
|
||||
import type { ChaosPageConfig, ChaosPageMessage } from "./chaosPageTypes";
|
||||
|
||||
/**
|
||||
* Save / reset persistence for the Chaos Mode config page. Extracted out of
|
||||
* the page component to keep it under the complexity/size ratchet
|
||||
* (config/quality/complexity-baseline.json).
|
||||
*/
|
||||
export function useChaosConfigPersistence(
|
||||
config: ChaosPageConfig,
|
||||
setConfig: Dispatch<SetStateAction<ChaosPageConfig>>
|
||||
) {
|
||||
const t = useTranslations("chaosConfig");
|
||||
const [saving, setSaving] = useState(false);
|
||||
const [message, setMessage] = useState<ChaosPageMessage>(null);
|
||||
|
||||
const saveConfig = useCallback(async () => {
|
||||
setSaving(true);
|
||||
setMessage(null);
|
||||
try {
|
||||
const res = await fetch("/api/chaos/config", {
|
||||
method: "PUT",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify(config),
|
||||
});
|
||||
|
||||
if (res.ok) {
|
||||
const data = await res.json();
|
||||
setConfig(data.config);
|
||||
setMessage({ type: "success", text: t("configSaved") });
|
||||
} else {
|
||||
setMessage({ type: "error", text: t("configError") });
|
||||
}
|
||||
} catch {
|
||||
setMessage({ type: "error", text: t("configError") });
|
||||
} finally {
|
||||
setSaving(false);
|
||||
}
|
||||
}, [config, setConfig, t]);
|
||||
|
||||
const resetConfig = useCallback(async () => {
|
||||
setSaving(true);
|
||||
setMessage(null);
|
||||
try {
|
||||
const res = await fetch("/api/chaos/config", { method: "DELETE" });
|
||||
if (res.ok) {
|
||||
const data = await res.json();
|
||||
setConfig(data.config);
|
||||
setMessage({ type: "success", text: "Config reset to defaults" });
|
||||
} else {
|
||||
const err = await res.json().catch(() => ({ error: "Reset failed" }));
|
||||
setMessage({ type: "error", text: err.error || "Reset failed" });
|
||||
}
|
||||
} catch {
|
||||
setMessage({ type: "error", text: "Failed to reset config" });
|
||||
} finally {
|
||||
setSaving(false);
|
||||
}
|
||||
}, [setConfig]);
|
||||
|
||||
return { t, saving, message, setMessage, saveConfig, resetConfig };
|
||||
}
|
||||
48
src/app/(dashboard)/dashboard/chaos/useChaosTestRun.ts
Normal file
48
src/app/(dashboard)/dashboard/chaos/useChaosTestRun.ts
Normal file
@@ -0,0 +1,48 @@
|
||||
"use client";
|
||||
|
||||
import { useCallback, useState } from "react";
|
||||
import { useTranslations } from "next-intl";
|
||||
import type { ChaosTestResult } from "./components/ChaosTestResultsPanel";
|
||||
import type { ChaosPageConfig, ChaosPageMessage } from "./chaosPageTypes";
|
||||
|
||||
/**
|
||||
* "Test Chaos Mode" run trigger for the Chaos Mode config page. Extracted out
|
||||
* of the page component to keep it under the complexity/size ratchet
|
||||
* (config/quality/complexity-baseline.json).
|
||||
*/
|
||||
export function useChaosTestRun(config: ChaosPageConfig, setMessage: (message: ChaosPageMessage) => void) {
|
||||
const t = useTranslations("chaosConfig");
|
||||
const [testing, setTesting] = useState(false);
|
||||
const [testResult, setTestResult] = useState<ChaosTestResult | null>(null);
|
||||
|
||||
const testChaos = useCallback(async () => {
|
||||
setTesting(true);
|
||||
setTestResult(null);
|
||||
setMessage(null);
|
||||
try {
|
||||
const res = await fetch("/api/chaos/run", {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
task: t("testTask"),
|
||||
mode: config.defaultMode,
|
||||
maxTokens: config.maxTokens,
|
||||
}),
|
||||
});
|
||||
|
||||
if (res.ok) {
|
||||
const data: ChaosTestResult = await res.json();
|
||||
setTestResult(data);
|
||||
} else {
|
||||
const err = await res.json().catch(() => ({ error: "Unknown error" }));
|
||||
setMessage({ type: "error", text: err.error || "Test failed" });
|
||||
}
|
||||
} catch (err: any) {
|
||||
setMessage({ type: "error", text: err.message || "Test failed" });
|
||||
} finally {
|
||||
setTesting(false);
|
||||
}
|
||||
}, [config.defaultMode, config.maxTokens, setMessage, t]);
|
||||
|
||||
return { testing, testResult, testChaos };
|
||||
}
|
||||
84
src/app/api/chaos/config/route.ts
Normal file
84
src/app/api/chaos/config/route.ts
Normal file
@@ -0,0 +1,84 @@
|
||||
/**
|
||||
* GET /api/chaos/config — Get chaos mode configuration
|
||||
* PUT /api/chaos/config — Update chaos mode configuration
|
||||
* DELETE /api/chaos/config — Reset to defaults
|
||||
*
|
||||
* Chaos Mode global settings: which providers/models participate,
|
||||
* default mode (parallel/collaborative), system prompt, timeout.
|
||||
*/
|
||||
|
||||
import { NextResponse } from "next/server";
|
||||
import { requireManagementAuth } from "@/lib/api/requireManagementAuth";
|
||||
import { buildErrorBody, sanitizeErrorMessage } from "@omniroute/open-sse/utils/error";
|
||||
import {
|
||||
getChaosConfig,
|
||||
setChaosConfig,
|
||||
resetChaosConfig,
|
||||
chaosConfigSchema,
|
||||
} from "@/lib/chaos/chaosConfig";
|
||||
import { validateBody, isValidationFailure } from "@/shared/validation/helpers";
|
||||
import * as log from "@/sse/utils/logger";
|
||||
|
||||
export const dynamic = "force-dynamic";
|
||||
|
||||
/**
|
||||
* GET /api/chaos/config
|
||||
* Returns the current chaos mode configuration.
|
||||
*/
|
||||
export async function GET(request: Request) {
|
||||
const authError = await requireManagementAuth(request);
|
||||
if (authError) return authError;
|
||||
|
||||
try {
|
||||
const config = await getChaosConfig();
|
||||
return NextResponse.json({ config });
|
||||
} catch (err) {
|
||||
const msg = sanitizeErrorMessage(err);
|
||||
log.error("chaos", "Error fetching chaos config", err);
|
||||
return NextResponse.json(buildErrorBody(500, msg), { status: 500 });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* PUT /api/chaos/config
|
||||
* Update chaos mode configuration.
|
||||
*/
|
||||
export async function PUT(request: Request) {
|
||||
const authError = await requireManagementAuth(request);
|
||||
if (authError) return authError;
|
||||
|
||||
try {
|
||||
const rawBody = await request.json();
|
||||
const validation = validateBody(chaosConfigSchema, rawBody);
|
||||
if (isValidationFailure(validation)) {
|
||||
return NextResponse.json(buildErrorBody(400, validation.error.message), {
|
||||
status: 400,
|
||||
});
|
||||
}
|
||||
|
||||
const config = await setChaosConfig(validation.data);
|
||||
return NextResponse.json({ config, message: "Chaos config updated" });
|
||||
} catch (err) {
|
||||
const msg = sanitizeErrorMessage(err);
|
||||
log.error("chaos", "Error updating chaos config", err);
|
||||
return NextResponse.json(buildErrorBody(500, msg), { status: 500 });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* DELETE /api/chaos/config
|
||||
* Reset chaos config to defaults.
|
||||
*/
|
||||
export async function DELETE(request: Request) {
|
||||
const authError = await requireManagementAuth(request);
|
||||
if (authError) return authError;
|
||||
|
||||
try {
|
||||
const config = await resetChaosConfig();
|
||||
return NextResponse.json({ config, message: "Chaos config reset to defaults" });
|
||||
} catch (err) {
|
||||
const msg = sanitizeErrorMessage(err);
|
||||
log.error("chaos", "Error resetting chaos config", err);
|
||||
return NextResponse.json(buildErrorBody(500, msg), { status: 500 });
|
||||
}
|
||||
}
|
||||
73
src/app/api/chaos/run/route.ts
Normal file
73
src/app/api/chaos/run/route.ts
Normal file
@@ -0,0 +1,73 @@
|
||||
/**
|
||||
* POST /api/chaos/run — Unified Chaos Mode execution endpoint.
|
||||
*
|
||||
* Dashboard-friendly: uses the current management session (cookie) for auth.
|
||||
* Delegates all execution logic to the shared chaosExecutor library.
|
||||
*
|
||||
* Body (JSON):
|
||||
* task: string // REQUIRED — the task/goal
|
||||
* providers?: string[] // Optional — filter specific providers
|
||||
* mode?: "parallel" | "collaborative" // Optional — override global default
|
||||
* systemPrompt?: string // Optional — override global system prompt
|
||||
* maxTokens?: number // Optional — override max_tokens per model call
|
||||
*
|
||||
* Returns the same shape as POST /api/skills/collect/chaos.
|
||||
*/
|
||||
import { NextResponse } from "next/server";
|
||||
import { z } from "zod";
|
||||
import { requireManagementAuth } from "@/lib/api/requireManagementAuth";
|
||||
import { validateBody, isValidationFailure } from "@/shared/validation/helpers";
|
||||
import { buildErrorBody, sanitizeErrorMessage } from "@omniroute/open-sse/utils/error";
|
||||
import { getChaosConfig } from "@/lib/chaos/chaosConfig";
|
||||
import { executeChaosRun, type ChaosRunResult } from "@/lib/chaos/chaosExecutor";
|
||||
import * as log from "@/sse/utils/logger";
|
||||
|
||||
export const dynamic = "force-dynamic";
|
||||
|
||||
const runSchema = z.object({
|
||||
task: z.string().min(1, "Task is required").max(100_000, "task too long"),
|
||||
providers: z.array(z.string().min(1)).max(50).optional(),
|
||||
mode: z.enum(["parallel", "collaborative"]).optional(),
|
||||
systemPrompt: z.string().max(10_000).optional(),
|
||||
maxTokens: z.number().int().min(256).max(128_000).optional(),
|
||||
});
|
||||
|
||||
export async function POST(request: Request) {
|
||||
// Require dashboard management auth (cookie-based)
|
||||
const authError = await requireManagementAuth(request);
|
||||
if (authError) return authError;
|
||||
|
||||
try {
|
||||
// Load global chaos config
|
||||
const globalConfig = await getChaosConfig();
|
||||
if (!globalConfig.enabled) {
|
||||
return NextResponse.json(
|
||||
buildErrorBody(400, "Chaos Mode is not enabled. Enable it in Dashboard → Chaos Mode."),
|
||||
{ status: 400 }
|
||||
);
|
||||
}
|
||||
|
||||
const rawBody = await request.json();
|
||||
const validation = validateBody(runSchema, rawBody);
|
||||
if (isValidationFailure(validation)) {
|
||||
return NextResponse.json(buildErrorBody(400, validation.error.message), { status: 400 });
|
||||
}
|
||||
|
||||
const { task, providers, mode, systemPrompt, maxTokens } = validation.data;
|
||||
|
||||
const result: ChaosRunResult = await executeChaosRun({
|
||||
task,
|
||||
providers,
|
||||
mode,
|
||||
systemPrompt,
|
||||
timeoutMs: globalConfig.timeoutMs,
|
||||
maxTokens: maxTokens || globalConfig.maxTokens,
|
||||
});
|
||||
|
||||
return NextResponse.json(result);
|
||||
} catch (err) {
|
||||
const msg = sanitizeErrorMessage(err);
|
||||
log.error("chaos", "Chaos run error", err);
|
||||
return NextResponse.json(buildErrorBody(500, msg), { status: 500 });
|
||||
}
|
||||
}
|
||||
@@ -86,6 +86,7 @@ export async function PATCH(request, { params }) {
|
||||
usageLimitEnabled,
|
||||
dailyUsageLimitUsd,
|
||||
weeklyUsageLimitUsd,
|
||||
chaosModeEnabled,
|
||||
} = validation.data;
|
||||
|
||||
const payload: Parameters<typeof updateApiKeyPermissions>[1] = {};
|
||||
@@ -112,6 +113,7 @@ export async function PATCH(request, { params }) {
|
||||
if (usageLimitEnabled !== undefined) payload.usageLimitEnabled = usageLimitEnabled;
|
||||
if (dailyUsageLimitUsd !== undefined) payload.dailyUsageLimitUsd = dailyUsageLimitUsd;
|
||||
if (weeklyUsageLimitUsd !== undefined) payload.weeklyUsageLimitUsd = weeklyUsageLimitUsd;
|
||||
if (chaosModeEnabled !== undefined) payload.chaosModeEnabled = chaosModeEnabled;
|
||||
|
||||
const updated = await updateApiKeyPermissions(id, payload);
|
||||
if (!updated) {
|
||||
@@ -145,6 +147,7 @@ export async function PATCH(request, { params }) {
|
||||
...(usageLimitEnabled !== undefined && { usageLimitEnabled }),
|
||||
...(dailyUsageLimitUsd !== undefined && { dailyUsageLimitUsd }),
|
||||
...(weeklyUsageLimitUsd !== undefined && { weeklyUsageLimitUsd }),
|
||||
...(chaosModeEnabled !== undefined && { chaosModeEnabled }),
|
||||
});
|
||||
} catch (error) {
|
||||
log.error("keys", "Error updating key permissions", error);
|
||||
|
||||
@@ -71,6 +71,7 @@ export async function POST(request) {
|
||||
usageLimitEnabled,
|
||||
dailyUsageLimitUsd,
|
||||
weeklyUsageLimitUsd,
|
||||
chaosModeEnabled,
|
||||
} = validation.data;
|
||||
|
||||
// Always get machineId from server
|
||||
@@ -82,7 +83,8 @@ export async function POST(request) {
|
||||
allowUsageCommand === true ||
|
||||
usageLimitEnabled === true ||
|
||||
dailyUsageLimitUsd !== undefined ||
|
||||
weeklyUsageLimitUsd !== undefined
|
||||
weeklyUsageLimitUsd !== undefined ||
|
||||
chaosModeEnabled === true
|
||||
) {
|
||||
await updateApiKeyPermissions(apiKey.id, {
|
||||
...(noLog === true && { noLog: true }),
|
||||
@@ -90,6 +92,7 @@ export async function POST(request) {
|
||||
...(usageLimitEnabled === true && { usageLimitEnabled: true }),
|
||||
...(dailyUsageLimitUsd !== undefined && { dailyUsageLimitUsd }),
|
||||
...(weeklyUsageLimitUsd !== undefined && { weeklyUsageLimitUsd }),
|
||||
...(chaosModeEnabled === true && { chaosModeEnabled: true }),
|
||||
});
|
||||
}
|
||||
|
||||
@@ -114,6 +117,7 @@ export async function POST(request) {
|
||||
usageLimitEnabled: usageLimitEnabled === true,
|
||||
dailyUsageLimitUsd: dailyUsageLimitUsd ?? null,
|
||||
weeklyUsageLimitUsd: weeklyUsageLimitUsd ?? null,
|
||||
chaosModeEnabled: chaosModeEnabled === true,
|
||||
streamDefaultMode: "legacy",
|
||||
},
|
||||
{ status: 201 }
|
||||
|
||||
147
src/app/api/skills/collect/chaos/route.ts
Normal file
147
src/app/api/skills/collect/chaos/route.ts
Normal file
@@ -0,0 +1,147 @@
|
||||
/**
|
||||
* POST /api/skills/collect/chaos
|
||||
*
|
||||
* Chaos Mode — spawn multiple models across providers for parallel or collaborative
|
||||
* task execution. Each active provider contributes one model instance; all models
|
||||
* work on the same task simultaneously (parallel) or in a chain where each sees
|
||||
* the previous model's output (collaborative).
|
||||
*
|
||||
* External API: uses Bearer token auth (API key with chaos_mode_enabled).
|
||||
*
|
||||
* Body (JSON):
|
||||
* task: string // REQUIRED — the task/goal for all models
|
||||
* providers?: string[] // Optional filter — only these provider IDs
|
||||
* mode?: "parallel" | "collaborative" // Default: from global config
|
||||
* systemPrompt?: string // Optional custom system prompt override
|
||||
* maxTokens?: number // Optional — max_tokens per model call
|
||||
*
|
||||
* Returns:
|
||||
* {
|
||||
* task, mode, startedAt,
|
||||
* totalProviders, totalResults,
|
||||
* models: [{ providerId, providerName, modelId, status, content, error?, durationMs }],
|
||||
* summary?: string
|
||||
* }
|
||||
*/
|
||||
import { NextResponse } from "next/server";
|
||||
import { z } from "zod";
|
||||
import { validateBody, isValidationFailure } from "@/shared/validation/helpers";
|
||||
import { buildErrorBody, sanitizeErrorMessage } from "@omniroute/open-sse/utils/error";
|
||||
import { validateApiKey, getApiKeyMetadata } from "@/lib/localDb";
|
||||
import { getChaosConfig } from "@/lib/chaos/chaosConfig";
|
||||
import { executeChaosRun, type ChaosRunResult } from "@/lib/chaos/chaosExecutor";
|
||||
import * as log from "@/sse/utils/logger";
|
||||
|
||||
export const dynamic = "force-dynamic";
|
||||
|
||||
// ── Schema ───────────────────────────────────────────────────────────────────
|
||||
|
||||
const chaosSchema = z.object({
|
||||
task: z.string().min(1, "task is required").max(100_000, "task too long"),
|
||||
providers: z.array(z.string().min(1)).max(50).optional(),
|
||||
mode: z.enum(["parallel", "collaborative"]).optional(),
|
||||
systemPrompt: z.string().max(10_000).optional(),
|
||||
maxTokens: z.number().int().min(256).max(128_000).optional(),
|
||||
});
|
||||
|
||||
// ── Auth helpers ─────────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Extract Bearer token from Authorization header.
|
||||
*/
|
||||
function extractBearerToken(request: Request): string | null {
|
||||
const auth = request.headers.get("Authorization");
|
||||
if (!auth || !auth.startsWith("Bearer ")) return null;
|
||||
return auth.slice(7).trim();
|
||||
}
|
||||
|
||||
/**
|
||||
* Verify API key has chaos mode enabled.
|
||||
* validateApiKey returns a plain boolean for BOTH the deployment-time env key and a
|
||||
* DB-backed key (see src/lib/db/apiKeys.ts::validateApiKey) — it never returns the
|
||||
* key record, so the env-key/DB-key distinction has to be made via getApiKeyMetadata,
|
||||
* whose synthesized env-key record is tagged `id: "env-key"` (src/lib/db/apiKeys.ts::
|
||||
* getApiKeyMetadata) and always carries "manage" scope.
|
||||
*/
|
||||
async function verifyChaosKey(bearerToken: string): Promise<{ ok: boolean; error?: string }> {
|
||||
const isValid = await validateApiKey(bearerToken);
|
||||
if (!isValid) {
|
||||
return { ok: false, error: "Invalid API key" };
|
||||
}
|
||||
|
||||
const metadata = await getApiKeyMetadata(bearerToken);
|
||||
if (!metadata) {
|
||||
return { ok: false, error: "Invalid API key" };
|
||||
}
|
||||
|
||||
// Env key has full access (see getApiKeyMetadata's synthesized "env-key" record).
|
||||
if (metadata.id === "env-key") {
|
||||
return { ok: true };
|
||||
}
|
||||
|
||||
if (!metadata.chaosModeEnabled) {
|
||||
return {
|
||||
ok: false,
|
||||
error: "Chaos Mode is not enabled for this API key. Enable it in API Key settings.",
|
||||
};
|
||||
}
|
||||
|
||||
return { ok: true };
|
||||
}
|
||||
|
||||
// ── Main handler ─────────────────────────────────────────────────────────────
|
||||
|
||||
export async function POST(request: Request) {
|
||||
try {
|
||||
// ── API Key auth check ─────────────────────────────────────────────
|
||||
const bearerToken = extractBearerToken(request);
|
||||
if (!bearerToken) {
|
||||
return NextResponse.json(
|
||||
buildErrorBody(401, "Missing or invalid Authorization header — Bearer token required"),
|
||||
{ status: 401 }
|
||||
);
|
||||
}
|
||||
|
||||
const auth = await verifyChaosKey(bearerToken);
|
||||
if (!auth.ok) {
|
||||
return NextResponse.json(buildErrorBody(403, auth.error!), { status: 403 });
|
||||
}
|
||||
|
||||
// ── Parse request body ─────────────────────────────────────────────
|
||||
const rawBody = await request.json();
|
||||
const validation = validateBody(chaosSchema, rawBody);
|
||||
if (isValidationFailure(validation)) {
|
||||
return NextResponse.json(buildErrorBody(400, validation.error.message), {
|
||||
status: 400,
|
||||
});
|
||||
}
|
||||
|
||||
const { task, providers, mode, systemPrompt, maxTokens } = validation.data;
|
||||
|
||||
// ── Load global chaos config ───────────────────────────────────────
|
||||
const globalConfig = await getChaosConfig();
|
||||
if (!globalConfig.enabled) {
|
||||
return NextResponse.json(
|
||||
buildErrorBody(400, "Chaos Mode is not enabled globally. Enable it in Dashboard → Chaos Mode."),
|
||||
{ status: 400 }
|
||||
);
|
||||
}
|
||||
|
||||
// ── Execute via the shared executor ────────────────────────────────
|
||||
const result: ChaosRunResult = await executeChaosRun({
|
||||
task,
|
||||
providers,
|
||||
mode,
|
||||
systemPrompt,
|
||||
timeoutMs: globalConfig.timeoutMs,
|
||||
maxTokens: maxTokens || globalConfig.maxTokens,
|
||||
apiKey: bearerToken,
|
||||
});
|
||||
|
||||
return NextResponse.json(result);
|
||||
} catch (err) {
|
||||
const msg = sanitizeErrorMessage(err);
|
||||
log.error("chaos", "Chaos external API error", err);
|
||||
return NextResponse.json(buildErrorBody(500, msg), { status: 500 });
|
||||
}
|
||||
}
|
||||
@@ -871,6 +871,7 @@
|
||||
"skills": "Skills",
|
||||
"omniSkills": "OmniSkills",
|
||||
"agentSkills": "AgentSkills",
|
||||
"chaosConfig": "Chaos Mode",
|
||||
"docs": "Docs",
|
||||
"issues": "Issues",
|
||||
"endpoints": "Endpoints",
|
||||
@@ -9149,5 +9150,35 @@
|
||||
"auth": "Auth",
|
||||
"feasibility": "Feasibility",
|
||||
"models": "Models"
|
||||
},
|
||||
"chaosConfig": {
|
||||
"pageTitle": "Chaos Mode",
|
||||
"pageSubtitle": "Run multiple AI models in parallel or collaboratively on the same task",
|
||||
"enableChaos": "Enable Chaos Mode",
|
||||
"enableChaosDesc": "Allow API keys with chaos mode enabled to use this feature",
|
||||
"mode": "Default Mode",
|
||||
"modeParallel": "Parallel",
|
||||
"modeCollaborative": "Collaborative",
|
||||
"modeParallelDesc": "All models run simultaneously — fastest results",
|
||||
"modeCollaborativeDesc": "Models chain outputs — each sees the previous result",
|
||||
"timeout": "Timeout (ms)",
|
||||
"timeoutDesc": "Max time per model call (5000-600000ms)",
|
||||
"systemPrompt": "System Prompt (optional)",
|
||||
"systemPromptDesc": "Custom instructions for all chaos mode model instances",
|
||||
"providerOverrides": "Provider Overrides",
|
||||
"providerOverridesDesc": "Select specific models per provider for chaos mode",
|
||||
"providerId": "Provider",
|
||||
"modelId": "Model",
|
||||
"addProvider": "Add Provider",
|
||||
"removeProvider": "Remove",
|
||||
"saveConfig": "Save Configuration",
|
||||
"configSaved": "Chaos configuration saved successfully",
|
||||
"configError": "Failed to save chaos configuration",
|
||||
"configReset": "Reset to Defaults",
|
||||
"keyPermission": "Chaos Mode Access",
|
||||
"keyPermissionDesc": "Allow this API key to use Chaos Mode (multi-model parallel execution)",
|
||||
"testButton": "Test Chaos Mode",
|
||||
"testTask": "Write a short poem about artificial intelligence",
|
||||
"loadingProviderModels": "Loading providers..."
|
||||
}
|
||||
}
|
||||
|
||||
119
src/lib/chaos/chaosConfig.ts
Normal file
119
src/lib/chaos/chaosConfig.ts
Normal file
@@ -0,0 +1,119 @@
|
||||
/**
|
||||
* lib/chaos/chaosConfig.ts
|
||||
*
|
||||
* Chaos Mode configuration — persisted per-instance settings for:
|
||||
* - Which providers/models participate
|
||||
* - Default mode (parallel vs collaborative)
|
||||
* - System prompt overrides
|
||||
* - Max timeout per model call
|
||||
*/
|
||||
|
||||
import { z } from "zod";
|
||||
import { getSettings, updateSettings } from "@/lib/db/settings";
|
||||
|
||||
// ── Schema ───────────────────────────────────────────────────────────────────
|
||||
|
||||
export const chaosConfigSchema = z.object({
|
||||
enabled: z.boolean().default(false),
|
||||
defaultMode: z.enum(["parallel", "collaborative"]).default("parallel"),
|
||||
providerOverrides: z
|
||||
.array(
|
||||
z.object({
|
||||
providerId: z.string().min(1),
|
||||
modelId: z.string().optional(),
|
||||
enabled: z.boolean().default(true),
|
||||
})
|
||||
)
|
||||
.max(200)
|
||||
.default([]),
|
||||
systemPrompt: z.string().max(10_000).optional(),
|
||||
timeoutMs: z.number().int().min(5_000).max(600_000).default(120_000),
|
||||
maxTokens: z.number().int().min(256).max(128_000).default(4096),
|
||||
});
|
||||
|
||||
export type ChaosConfig = z.infer<typeof chaosConfigSchema>;
|
||||
|
||||
export const DEFAULT_CHAOS_CONFIG: ChaosConfig = {
|
||||
enabled: false,
|
||||
defaultMode: "parallel",
|
||||
providerOverrides: [],
|
||||
systemPrompt: undefined,
|
||||
timeoutMs: 120_000,
|
||||
maxTokens: 4096,
|
||||
};
|
||||
|
||||
// ── Persistence ──────────────────────────────────────────────────────────────
|
||||
//
|
||||
// Persisted via the shared settings store (src/lib/db/settings.ts::getSettings/
|
||||
// updateSettings — the `key_value` table, namespace 'settings') rather than
|
||||
// hand-rolled SQL against a nonexistent `settings` table (the original PR queried
|
||||
// a table that was never created — every read silently fell back to defaults and
|
||||
// every write/reset threw). Follows the repo convention of routing all settings
|
||||
// reads/writes through src/lib/db/settings.ts (see CLAUDE.md → Database).
|
||||
|
||||
const CONFIG_KEY = "chaosModeConfig";
|
||||
|
||||
let _configCache: ChaosConfig | null = null;
|
||||
|
||||
/**
|
||||
* Get the current Chaos Mode configuration.
|
||||
*/
|
||||
export async function getChaosConfig(): Promise<ChaosConfig> {
|
||||
if (_configCache) return _configCache;
|
||||
|
||||
try {
|
||||
const settings = await getSettings();
|
||||
const raw = settings[CONFIG_KEY];
|
||||
|
||||
if (raw === undefined || raw === null) {
|
||||
_configCache = DEFAULT_CHAOS_CONFIG;
|
||||
return _configCache;
|
||||
}
|
||||
|
||||
const result = chaosConfigSchema.safeParse(raw);
|
||||
if (result.success) {
|
||||
_configCache = result.data;
|
||||
return result.data;
|
||||
}
|
||||
|
||||
// Fall back to default if stored config is invalid
|
||||
_configCache = DEFAULT_CHAOS_CONFIG;
|
||||
return _configCache;
|
||||
} catch {
|
||||
_configCache = DEFAULT_CHAOS_CONFIG;
|
||||
return _configCache;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Update the Chaos Mode configuration.
|
||||
*/
|
||||
export async function setChaosConfig(config: ChaosConfig): Promise<ChaosConfig> {
|
||||
const validated = chaosConfigSchema.parse(config);
|
||||
|
||||
await updateSettings({ [CONFIG_KEY]: validated });
|
||||
|
||||
// Invalidate cache
|
||||
_configCache = null;
|
||||
|
||||
return validated;
|
||||
}
|
||||
|
||||
/**
|
||||
* Reset chaos config to defaults.
|
||||
*/
|
||||
export async function resetChaosConfig(): Promise<ChaosConfig> {
|
||||
await updateSettings({ [CONFIG_KEY]: null });
|
||||
_configCache = null;
|
||||
return DEFAULT_CHAOS_CONFIG;
|
||||
}
|
||||
|
||||
/**
|
||||
* Invalidate the in-memory config cache without touching persisted settings.
|
||||
* Needed whenever the underlying DB/settings store is reset out-of-band (e.g.
|
||||
* test teardown calling resetDbInstance()) — otherwise getChaosConfig() keeps
|
||||
* serving a stale in-memory value after the store it was read from is gone.
|
||||
*/
|
||||
export function invalidateChaosConfigCache(): void {
|
||||
_configCache = null;
|
||||
}
|
||||
439
src/lib/chaos/chaosExecutor.ts
Normal file
439
src/lib/chaos/chaosExecutor.ts
Normal file
@@ -0,0 +1,439 @@
|
||||
/**
|
||||
* lib/chaos/chaosExecutor.ts
|
||||
*
|
||||
* Shared Chaos Mode execution engine — used by BOTH:
|
||||
* - POST /api/chaos/run (dashboard, management-session auth)
|
||||
* - POST /api/skills/collect/chaos (external, Bearer-token auth)
|
||||
*
|
||||
* Eliminates the ~150 lines of duplicate dispatch logic that previously existed
|
||||
* in both route files.
|
||||
*/
|
||||
import { getProviderConnections } from "@/models";
|
||||
import { getChaosConfig, type ChaosConfig } from "@/lib/chaos/chaosConfig";
|
||||
import { POST as postChatCompletion } from "@/app/api/v1/chat/completions/route";
|
||||
|
||||
// Wrapped in an object (rather than called as a bare imported function) so unit
|
||||
// tests can swap it out via `mock.method(chatDispatch, "postChatCompletion", ...)`
|
||||
// without hitting real upstream providers — the same pattern src/lib/batches/
|
||||
// dispatch.ts uses for its `dispatch` export (ES module named bindings are
|
||||
// read-only and cannot be mocked directly).
|
||||
export const chatDispatch = {
|
||||
postChatCompletion,
|
||||
};
|
||||
|
||||
// ── Exported types ───────────────────────────────────────────────────────────
|
||||
|
||||
export interface ProviderInfo {
|
||||
id: string;
|
||||
name: string;
|
||||
provider: string;
|
||||
defaultModel: string | null;
|
||||
}
|
||||
|
||||
export interface ModelResult {
|
||||
providerId: string;
|
||||
providerName: string;
|
||||
modelId: string;
|
||||
status: "success" | "error" | "skipped";
|
||||
content: string | null;
|
||||
error?: string;
|
||||
durationMs: number;
|
||||
}
|
||||
|
||||
export type ChaosMode = "parallel" | "collaborative";
|
||||
|
||||
export interface ChaosRunInput {
|
||||
task: string;
|
||||
providers?: string[];
|
||||
mode?: ChaosMode;
|
||||
systemPrompt?: string;
|
||||
/** Override the global timeout for this single run */
|
||||
timeoutMs?: number;
|
||||
/** Override max_tokens sent to each model (default 4096) */
|
||||
maxTokens?: number;
|
||||
/**
|
||||
* API key to attribute the in-process dispatch calls to (usage accounting,
|
||||
* per-key policy). Optional — omitted for dashboard-initiated runs, which fall
|
||||
* back to the same "local mode" (no Authorization header) path used by
|
||||
* src/lib/evals/runtime.ts and src/lib/batches/dispatch.ts.
|
||||
*/
|
||||
apiKey?: string | null;
|
||||
}
|
||||
|
||||
export interface ChaosRunResult {
|
||||
task: string;
|
||||
mode: ChaosMode;
|
||||
startedAt: string;
|
||||
totalProviders: number;
|
||||
totalResults: number;
|
||||
models: ModelResult[];
|
||||
summary?: string;
|
||||
}
|
||||
|
||||
interface EffectiveRunParams {
|
||||
mode: ChaosMode;
|
||||
timeoutMs: number;
|
||||
maxTokens: number;
|
||||
effectiveSystemPrompt: string;
|
||||
}
|
||||
|
||||
// ── Constants ─────────────────────────────────────────────────────────────────
|
||||
|
||||
export const DEFAULT_SYSTEM_PROMPT = `You are one of several AI models working in CHAOS MODE.
|
||||
|
||||
Your job:
|
||||
1. Analyze the user's task thoroughly
|
||||
2. Produce the best possible response using your unique strengths
|
||||
3. Be concise but complete — your output will be combined with other models' outputs
|
||||
4. Do NOT refer to "other models" or "CHAOS MODE" in your response — just answer the task directly`;
|
||||
|
||||
export const COLLABORATIVE_SYSTEM_PROMPT = `You are one of several AI models working in CHAOS MODE (collaborative).
|
||||
|
||||
Your job:
|
||||
1. You will see the task AND the previous model's output
|
||||
2. Build upon, refine, critique, or extend the previous work
|
||||
3. Add new insights, fix issues, or provide an alternative perspective
|
||||
4. Do NOT refer to "CHAOS MODE" or other models explicitly — just contribute your part naturally`;
|
||||
|
||||
const DEFAULT_MAX_TOKENS = 4096;
|
||||
/** Maximum concurrent fetch requests in parallel mode */
|
||||
const MAX_CONCURRENCY = 10;
|
||||
|
||||
// ── Internal helpers — config/param resolution ──────────────────────────────
|
||||
|
||||
function resolveEffectiveRunParams(
|
||||
input: ChaosRunInput,
|
||||
globalConfig: ChaosConfig
|
||||
): EffectiveRunParams {
|
||||
const mode: ChaosMode = input.mode || globalConfig.defaultMode || "parallel";
|
||||
const timeoutMs = input.timeoutMs || globalConfig.timeoutMs || 120_000;
|
||||
const maxTokens = input.maxTokens || globalConfig.maxTokens || DEFAULT_MAX_TOKENS;
|
||||
const effectiveSystemPrompt =
|
||||
input.systemPrompt ||
|
||||
globalConfig.systemPrompt ||
|
||||
(mode === "collaborative" ? COLLABORATIVE_SYSTEM_PROMPT : DEFAULT_SYSTEM_PROMPT);
|
||||
|
||||
return { mode, timeoutMs, maxTokens, effectiveSystemPrompt };
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the best model ID for a provider connection.
|
||||
* Applies provider overrides from global chaos config if present.
|
||||
*/
|
||||
function resolveModelId(
|
||||
conn: { provider?: string; id?: string; defaultModel?: string | null },
|
||||
overrides: ChaosConfig["providerOverrides"]
|
||||
): string {
|
||||
const override = overrides.find(
|
||||
(o) =>
|
||||
o.enabled &&
|
||||
(o.providerId.toLowerCase() === (conn.provider || "").toLowerCase() ||
|
||||
o.providerId.toLowerCase() === (conn.id || "").toLowerCase())
|
||||
);
|
||||
if (override?.modelId) return override.modelId;
|
||||
return conn.defaultModel || conn.provider || conn.id || "unknown";
|
||||
}
|
||||
|
||||
// ── Internal helpers — provider selection ───────────────────────────────────
|
||||
|
||||
/** Narrow `active` connections down to the caller-requested `providers` filter. */
|
||||
function filterByRequestedProviders(active: any[], requested: string[]): any[] {
|
||||
const filterSet = new Set(requested.map((p: string) => p.toLowerCase()));
|
||||
const selected = active.filter((c: any) => filterSet.has((c.provider ?? "").toLowerCase()));
|
||||
if (selected.length === 0) {
|
||||
throw new Error(`None of the specified providers are active: ${requested.join(", ")}`);
|
||||
}
|
||||
return selected;
|
||||
}
|
||||
|
||||
/** Narrow `active` connections down to the enabled global-config overrides (soft filter). */
|
||||
function filterByEnabledOverrides(
|
||||
active: any[],
|
||||
enabledOverrides: ChaosConfig["providerOverrides"]
|
||||
): any[] {
|
||||
const overrideIds = new Set(enabledOverrides.map((o) => o.providerId.toLowerCase()));
|
||||
const selected = active.filter(
|
||||
(c: any) =>
|
||||
overrideIds.has((c.provider ?? "").toLowerCase()) || overrideIds.has((c.id ?? "").toLowerCase())
|
||||
);
|
||||
return selected.length > 0 ? selected : active; // fallback to all active
|
||||
}
|
||||
|
||||
function toProviderInfoList(selected: any[]): ProviderInfo[] {
|
||||
const providerMap = new Map<string, ProviderInfo>();
|
||||
for (const conn of selected) {
|
||||
const providerKey = conn.provider || conn.id;
|
||||
if (!providerMap.has(providerKey)) {
|
||||
providerMap.set(providerKey, {
|
||||
id: conn.id,
|
||||
name: conn.name || conn.provider || providerKey,
|
||||
provider: conn.provider || providerKey,
|
||||
defaultModel: conn.defaultModel || null,
|
||||
});
|
||||
}
|
||||
}
|
||||
return Array.from(providerMap.values());
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch active provider connections and narrow them down to the requested
|
||||
* (explicit `providers` filter, else enabled global-config overrides, else all
|
||||
* active) set, deduplicated by provider id.
|
||||
*/
|
||||
async function selectChaosProviders(
|
||||
input: ChaosRunInput,
|
||||
globalConfig: ChaosConfig
|
||||
): Promise<{ providers: ProviderInfo[]; enabledOverrides: ChaosConfig["providerOverrides"] }> {
|
||||
const allConnections = await getProviderConnections().catch(() => [] as any[]);
|
||||
const active = (Array.isArray(allConnections) ? allConnections : []).filter(
|
||||
(c: any) => c.isActive !== false
|
||||
);
|
||||
|
||||
if (active.length === 0) {
|
||||
throw new Error("No active provider connections found");
|
||||
}
|
||||
|
||||
const enabledOverrides = globalConfig.providerOverrides.filter((o) => o.enabled);
|
||||
|
||||
let selected = active;
|
||||
if (input.providers && input.providers.length > 0) {
|
||||
selected = filterByRequestedProviders(active, input.providers);
|
||||
} else if (enabledOverrides.length > 0) {
|
||||
selected = filterByEnabledOverrides(active, enabledOverrides);
|
||||
}
|
||||
|
||||
return { providers: toProviderInfoList(selected), enabledOverrides };
|
||||
}
|
||||
|
||||
// ── Internal helpers — single-model dispatch ────────────────────────────────
|
||||
|
||||
function buildDispatchRequest(
|
||||
model: string,
|
||||
messages: { role: string; content: string }[],
|
||||
maxTokens: number,
|
||||
timeoutMs: number,
|
||||
apiKey?: string | null
|
||||
): Request {
|
||||
const headers = new Headers({ "Content-Type": "application/json" });
|
||||
if (apiKey) {
|
||||
headers.set("Authorization", `Bearer ${apiKey}`);
|
||||
}
|
||||
return new Request("http://localhost/api/v1/chat/completions", {
|
||||
method: "POST",
|
||||
headers,
|
||||
body: JSON.stringify({ model, messages, stream: false, max_tokens: maxTokens }),
|
||||
signal: AbortSignal.timeout(timeoutMs),
|
||||
});
|
||||
}
|
||||
|
||||
async function parseDispatchResponse(
|
||||
res: Response,
|
||||
ctx: { providerId: string; providerName: string; modelId: string },
|
||||
start: number
|
||||
): Promise<ModelResult> {
|
||||
if (!res.ok) {
|
||||
const errText = await res.text().catch(() => "unknown error");
|
||||
return {
|
||||
...ctx,
|
||||
status: "error",
|
||||
content: null,
|
||||
error: `API ${res.status}: ${errText.slice(0, 500)}`,
|
||||
durationMs: Math.round(performance.now() - start),
|
||||
};
|
||||
}
|
||||
|
||||
const data = await res.json();
|
||||
const content =
|
||||
data.choices?.[0]?.message?.content ?? data.choices?.[0]?.text ?? JSON.stringify(data);
|
||||
|
||||
return {
|
||||
...ctx,
|
||||
status: "success",
|
||||
content,
|
||||
durationMs: Math.round(performance.now() - start),
|
||||
};
|
||||
}
|
||||
|
||||
function buildDispatchErrorResult(
|
||||
err: unknown,
|
||||
ctx: { providerId: string; providerName: string; modelId: string },
|
||||
start: number,
|
||||
timeoutMs: number
|
||||
): ModelResult {
|
||||
const errObj = err as { name?: string; type?: string; message?: string };
|
||||
const isAbort = errObj?.name === "AbortError" || errObj?.type === "aborted";
|
||||
return {
|
||||
...ctx,
|
||||
status: "error",
|
||||
content: null,
|
||||
error: isAbort ? `timeout (${timeoutMs}ms)` : (errObj?.message ?? String(err)),
|
||||
durationMs: Math.round(performance.now() - start),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Dispatch to OmniRoute's own /v1/chat/completions handler for a given
|
||||
* provider+model — in-process, via a synthetic Request handed directly to the
|
||||
* route's POST handler. No network hop, no port dependency. Mirrors the
|
||||
* established pattern in src/lib/batches/dispatch.ts and src/lib/evals/runtime.ts
|
||||
* (which the codebase's outbound-self-call convention requires — see #6679 review).
|
||||
*/
|
||||
async function dispatchToModel(
|
||||
providerId: string,
|
||||
providerName: string,
|
||||
modelId: string,
|
||||
messages: { role: string; content: string }[],
|
||||
timeoutMs: number,
|
||||
maxTokens: number,
|
||||
apiKey?: string | null
|
||||
): Promise<ModelResult> {
|
||||
const start = performance.now();
|
||||
const ctx = { providerId, providerName, modelId };
|
||||
|
||||
try {
|
||||
const model = modelId || providerId;
|
||||
const request = buildDispatchRequest(model, messages, maxTokens, timeoutMs, apiKey);
|
||||
const res = await chatDispatch.postChatCompletion(request);
|
||||
return await parseDispatchResponse(res, ctx, start);
|
||||
} catch (err: unknown) {
|
||||
return buildDispatchErrorResult(err, ctx, start, timeoutMs);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Run an array of async functions with a concurrency limit.
|
||||
* Uses a simple pooling approach: start up to `limit` tasks at once,
|
||||
* and as each completes, start the next one.
|
||||
*/
|
||||
async function runWithConcurrencyLimit<T>(tasks: (() => Promise<T>)[], limit: number): Promise<T[]> {
|
||||
const results: T[] = new Array(tasks.length);
|
||||
let nextIndex = 0;
|
||||
|
||||
async function worker(): Promise<void> {
|
||||
while (nextIndex < tasks.length) {
|
||||
const idx = nextIndex++;
|
||||
results[idx] = await tasks[idx]();
|
||||
}
|
||||
}
|
||||
|
||||
const workers = Array.from({ length: Math.min(limit, tasks.length) }, () => worker());
|
||||
await Promise.all(workers);
|
||||
return results;
|
||||
}
|
||||
|
||||
// ── Internal helpers — mode dispatch ────────────────────────────────────────
|
||||
|
||||
function buildMessages(
|
||||
systemPrompt: string,
|
||||
userContent: string
|
||||
): { role: "system" | "user"; content: string }[] {
|
||||
return [
|
||||
{ role: "system", content: systemPrompt },
|
||||
{ role: "user", content: userContent },
|
||||
];
|
||||
}
|
||||
|
||||
async function dispatchParallel(
|
||||
providers: ProviderInfo[],
|
||||
input: ChaosRunInput,
|
||||
overrides: ChaosConfig["providerOverrides"],
|
||||
params: EffectiveRunParams
|
||||
): Promise<ModelResult[]> {
|
||||
const tasks = providers.map((p) => () => {
|
||||
const modelId = resolveModelId(p, overrides);
|
||||
const messages = buildMessages(params.effectiveSystemPrompt, input.task);
|
||||
return dispatchToModel(
|
||||
p.id,
|
||||
p.name,
|
||||
modelId,
|
||||
messages,
|
||||
params.timeoutMs,
|
||||
params.maxTokens,
|
||||
input.apiKey
|
||||
);
|
||||
});
|
||||
|
||||
const results = await runWithConcurrencyLimit(tasks, MAX_CONCURRENCY);
|
||||
|
||||
// Sort: successes first, then errors, then by duration
|
||||
results.sort((a, b) => {
|
||||
if (a.status === "success" && b.status !== "success") return -1;
|
||||
if (a.status !== "success" && b.status === "success") return 1;
|
||||
return a.durationMs - b.durationMs;
|
||||
});
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
async function dispatchCollaborative(
|
||||
providers: ProviderInfo[],
|
||||
input: ChaosRunInput,
|
||||
overrides: ChaosConfig["providerOverrides"],
|
||||
params: EffectiveRunParams
|
||||
): Promise<ModelResult[]> {
|
||||
const results: ModelResult[] = [];
|
||||
let context = input.task;
|
||||
|
||||
for (const p of providers) {
|
||||
const modelId = resolveModelId(p, overrides);
|
||||
const messages = buildMessages(params.effectiveSystemPrompt, context);
|
||||
|
||||
const result = await dispatchToModel(
|
||||
p.id,
|
||||
p.name,
|
||||
modelId,
|
||||
messages,
|
||||
params.timeoutMs,
|
||||
params.maxTokens,
|
||||
input.apiKey
|
||||
);
|
||||
results.push(result);
|
||||
|
||||
if (result.status === "success" && result.content) {
|
||||
context = `Task: ${input.task}\n\nPrevious model's output:\n${result.content}\n\n---\n\nPlease refine, extend, critique, or provide an alternative perspective on the above.`;
|
||||
}
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
function buildCollaborativeSummary(mode: ChaosMode, results: ModelResult[]): string | undefined {
|
||||
if (mode !== "collaborative") return undefined;
|
||||
return (
|
||||
results
|
||||
.filter((r) => r.status === "success" && r.content)
|
||||
.map((r) => r.content!)
|
||||
.join("\n\n---\n\n") || undefined
|
||||
);
|
||||
}
|
||||
|
||||
// ── Main execution function ──────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Execute a Chaos Mode run.
|
||||
*
|
||||
* This is the single shared implementation used by both API routes.
|
||||
*/
|
||||
export async function executeChaosRun(input: ChaosRunInput): Promise<ChaosRunResult> {
|
||||
const globalConfig = await getChaosConfig();
|
||||
const params = resolveEffectiveRunParams(input, globalConfig);
|
||||
const { providers, enabledOverrides } = await selectChaosProviders(input, globalConfig);
|
||||
|
||||
const startedAt = new Date().toISOString();
|
||||
const results =
|
||||
params.mode === "parallel"
|
||||
? await dispatchParallel(providers, input, enabledOverrides, params)
|
||||
: await dispatchCollaborative(providers, input, enabledOverrides, params);
|
||||
|
||||
const summary = buildCollaborativeSummary(params.mode, results);
|
||||
|
||||
return {
|
||||
task: input.task,
|
||||
mode: params.mode,
|
||||
startedAt,
|
||||
totalProviders: providers.length,
|
||||
totalResults: results.length,
|
||||
models: results,
|
||||
...(summary ? { summary } : {}),
|
||||
};
|
||||
}
|
||||
@@ -44,4 +44,8 @@ export const API_KEY_COLUMN_FALLBACKS = [
|
||||
name: "weekly_usage_limit_usd",
|
||||
definition: "weekly_usage_limit_usd REAL",
|
||||
},
|
||||
{
|
||||
name: "chaos_mode_enabled",
|
||||
definition: "chaos_mode_enabled INTEGER NOT NULL DEFAULT 0",
|
||||
},
|
||||
] as const;
|
||||
|
||||
@@ -43,6 +43,7 @@ import {
|
||||
parseNullableTimestamp,
|
||||
parseIsBanned,
|
||||
parseStreamDefaultMode,
|
||||
parseChaosModeEnabled,
|
||||
} from "./apiKeys/rowParsers";
|
||||
import type { AccessSchedule, RateLimitRule } from "./apiKeys/types";
|
||||
|
||||
@@ -95,6 +96,7 @@ interface ApiKeyMetadata {
|
||||
usageLimitEnabled: boolean;
|
||||
dailyUsageLimitUsd: number | null;
|
||||
weeklyUsageLimitUsd: number | null;
|
||||
chaosModeEnabled: boolean;
|
||||
}
|
||||
|
||||
interface ApiKeyRow extends JsonRecord {
|
||||
@@ -134,6 +136,8 @@ interface ApiKeyRow extends JsonRecord {
|
||||
dailyUsageLimitUsd?: unknown;
|
||||
weekly_usage_limit_usd?: unknown;
|
||||
weeklyUsageLimitUsd?: unknown;
|
||||
chaos_mode_enabled?: unknown;
|
||||
chaosModeEnabled?: unknown;
|
||||
}
|
||||
|
||||
interface StatementLike<TRow = unknown> {
|
||||
@@ -180,6 +184,7 @@ interface ApiKeyView extends JsonRecord {
|
||||
usageLimitEnabled?: boolean;
|
||||
dailyUsageLimitUsd?: number | null;
|
||||
weeklyUsageLimitUsd?: number | null;
|
||||
chaosModeEnabled?: boolean;
|
||||
}
|
||||
|
||||
// LRU cache for API key validation (valid keys only)
|
||||
@@ -393,7 +398,7 @@ function getPreparedStatements(db: ApiKeysDbLike): ApiKeysStatements {
|
||||
"SELECT id, expires_at, revoked_at, is_active, is_banned FROM api_keys WHERE key = ? OR key_hash = ?"
|
||||
);
|
||||
_stmtGetKeyMetadata = db.prepare<ApiKeyRow>(
|
||||
"SELECT id, name, machine_id, allowed_models, blocked_models, allowed_combos, allowed_connections, allowed_quotas, no_log, auto_resolve, is_active, access_schedule, max_requests_per_day, max_requests_per_minute, throttle_delay_ms, max_sessions, revoked_at, expires_at, ip_allowlist, scopes, rate_limits, is_banned, key_hash, allowed_endpoints, stream_default_mode, disable_non_public_models, allow_usage_command, usage_limit_enabled, daily_usage_limit_usd, weekly_usage_limit_usd, proxy_id FROM api_keys WHERE key = ? OR key_hash = ?"
|
||||
"SELECT id, name, machine_id, allowed_models, blocked_models, allowed_combos, allowed_connections, allowed_quotas, no_log, auto_resolve, is_active, access_schedule, max_requests_per_day, max_requests_per_minute, throttle_delay_ms, max_sessions, revoked_at, expires_at, ip_allowlist, scopes, rate_limits, is_banned, key_hash, allowed_endpoints, stream_default_mode, disable_non_public_models, allow_usage_command, usage_limit_enabled, daily_usage_limit_usd, weekly_usage_limit_usd, chaos_mode_enabled, proxy_id FROM api_keys WHERE key = ? OR key_hash = ?"
|
||||
);
|
||||
_stmtInsertKey = db.prepare(
|
||||
"INSERT INTO api_keys (id, name, key, machine_id, allowed_models, no_log, created_at, key_prefix, key_hash, scopes) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
|
||||
@@ -446,6 +451,7 @@ export async function getApiKeys() {
|
||||
(camelRow as JsonRecord).disableNonPublicModels
|
||||
);
|
||||
camelRow.allowUsageCommand = parseAllowUsageCommand((camelRow as JsonRecord).allowUsageCommand);
|
||||
camelRow.chaosModeEnabled = parseChaosModeEnabled((camelRow as JsonRecord).chaosModeEnabled);
|
||||
Object.assign(camelRow, parseApiKeyUsageLimitFields(camelRow));
|
||||
if (typeof camelRow.id === "string" && camelRow.id.length > 0) {
|
||||
setNoLog(camelRow.id, camelRow.noLog === true);
|
||||
@@ -558,6 +564,7 @@ export async function getApiKeyById(id: string) {
|
||||
(camelRow as JsonRecord).disableNonPublicModels
|
||||
);
|
||||
camelRow.allowUsageCommand = parseAllowUsageCommand((camelRow as JsonRecord).allowUsageCommand);
|
||||
camelRow.chaosModeEnabled = parseChaosModeEnabled((camelRow as JsonRecord).chaosModeEnabled);
|
||||
Object.assign(camelRow, parseApiKeyUsageLimitFields(camelRow));
|
||||
if (typeof camelRow.id === "string" && camelRow.id.length > 0) {
|
||||
setNoLog(camelRow.id, camelRow.noLog === true);
|
||||
@@ -684,6 +691,7 @@ export async function updateApiKeyPermissions(
|
||||
usageLimitEnabled?: boolean;
|
||||
dailyUsageLimitUsd?: number | null;
|
||||
weeklyUsageLimitUsd?: number | null;
|
||||
chaosModeEnabled?: boolean;
|
||||
}
|
||||
) {
|
||||
const db = getDbInstance() as ApiKeysDbLike;
|
||||
@@ -722,6 +730,7 @@ export async function updateApiKeyPermissions(
|
||||
dailyUsageLimitUsd: (update as { dailyUsageLimitUsd?: number | null }).dailyUsageLimitUsd,
|
||||
weeklyUsageLimitUsd: (update as { weeklyUsageLimitUsd?: number | null })
|
||||
.weeklyUsageLimitUsd,
|
||||
chaosModeEnabled: (update as { chaosModeEnabled?: boolean }).chaosModeEnabled,
|
||||
};
|
||||
|
||||
if (
|
||||
@@ -748,6 +757,7 @@ export async function updateApiKeyPermissions(
|
||||
(normalized as Record<string, unknown>).streamDefaultMode === undefined &&
|
||||
normalized.disableNonPublicModels === undefined &&
|
||||
normalized.allowUsageCommand === undefined &&
|
||||
normalized.chaosModeEnabled === undefined &&
|
||||
!hasUsageLimitUpdate(normalized as Record<string, unknown>)
|
||||
) {
|
||||
return false;
|
||||
@@ -781,6 +791,7 @@ export async function updateApiKeyPermissions(
|
||||
usageLimitEnabled?: number;
|
||||
dailyUsageLimitUsd?: number | null;
|
||||
weeklyUsageLimitUsd?: number | null;
|
||||
chaosModeEnabled?: number;
|
||||
} = { id };
|
||||
|
||||
if (normalized.name !== undefined) {
|
||||
@@ -884,6 +895,11 @@ export async function updateApiKeyPermissions(
|
||||
params.allowUsageCommand = normalized.allowUsageCommand ? 1 : 0;
|
||||
}
|
||||
|
||||
if (normalized.chaosModeEnabled !== undefined) {
|
||||
updates.push("chaos_mode_enabled = @chaosModeEnabled");
|
||||
params.chaosModeEnabled = normalized.chaosModeEnabled ? 1 : 0;
|
||||
}
|
||||
|
||||
appendUsageLimitUpdates(normalized as Record<string, unknown>, updates, params);
|
||||
|
||||
const maxSessionsUpdate = (normalized as Record<string, unknown>).maxSessions;
|
||||
@@ -1270,6 +1286,7 @@ export async function getApiKeyMetadata(
|
||||
usageLimitEnabled: false,
|
||||
dailyUsageLimitUsd: null,
|
||||
weeklyUsageLimitUsd: null,
|
||||
chaosModeEnabled: false,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -1343,6 +1360,9 @@ export async function getApiKeyMetadata(
|
||||
allowUsageCommand: parseAllowUsageCommand(
|
||||
(record as JsonRecord).allow_usage_command ?? (record as JsonRecord).allowUsageCommand
|
||||
),
|
||||
chaosModeEnabled: parseChaosModeEnabled(
|
||||
(record as JsonRecord).chaos_mode_enabled ?? (record as JsonRecord).chaosModeEnabled
|
||||
),
|
||||
...parseApiKeyUsageLimitFields(record as JsonRecord),
|
||||
};
|
||||
|
||||
|
||||
@@ -47,6 +47,10 @@ export function parseAllowUsageCommand(value: unknown): boolean {
|
||||
return value === true || value === 1 || value === "1";
|
||||
}
|
||||
|
||||
export function parseChaosModeEnabled(value: unknown): boolean {
|
||||
return value === true || value === 1 || value === "1";
|
||||
}
|
||||
|
||||
export function parseIsActive(value: unknown): boolean {
|
||||
// DEFAULT 1 — active unless explicitly set to 0
|
||||
if (value === 0 || value === "0" || value === false) return false;
|
||||
|
||||
@@ -17,6 +17,13 @@ const PUBLIC_API_ROUTE_PREFIXES = [
|
||||
// The handler enforces its own auth via extractUsageCommandApiKey/isValidApiKey
|
||||
// and the allowUsageCommand flag — it must not be gated by management auth.
|
||||
"/api/usage/om-usage",
|
||||
// Chaos Mode external dispatch endpoint (POST /api/skills/collect/chaos).
|
||||
// This entry only bypasses the dashboard requireLogin (cookie) gate — the
|
||||
// handler enforces its own Bearer-token auth (validateApiKey +
|
||||
// chaosModeEnabled check) before doing any work. See src/app/api/skills/
|
||||
// collect/chaos/route.ts. Do not widen this prefix to cover other
|
||||
// /api/skills/collect/* routes without the same per-handler auth.
|
||||
"/api/skills/collect/chaos",
|
||||
];
|
||||
|
||||
const PUBLIC_READONLY_API_ROUTE_PREFIXES = [
|
||||
|
||||
@@ -515,6 +515,14 @@ const AGENTIC_FEATURES_ITEMS: readonly SidebarSectionChild[] = [
|
||||
subtitleKey: "agentSkillsSubtitle",
|
||||
icon: "share",
|
||||
},
|
||||
{
|
||||
id: "chaos-config",
|
||||
href: "/dashboard/chaos",
|
||||
i18nKey: "chaosConfig",
|
||||
labelFallback: "Chaos Mode",
|
||||
subtitleFallback: "Multi-model parallel execution",
|
||||
icon: "blender",
|
||||
},
|
||||
{
|
||||
id: "skills",
|
||||
href: "/dashboard/omni-skills",
|
||||
|
||||
@@ -74,6 +74,7 @@ export const HIDEABLE_SIDEBAR_ITEM_IDS = [
|
||||
"memory",
|
||||
"skills",
|
||||
"agent-skills",
|
||||
"chaos-config",
|
||||
"mcp",
|
||||
"a2a",
|
||||
"plugins",
|
||||
|
||||
@@ -25,6 +25,7 @@ export const createKeySchema = z.object({
|
||||
usageLimitEnabled: z.boolean().optional(),
|
||||
dailyUsageLimitUsd: z.coerce.number().min(0).optional().nullable(),
|
||||
weeklyUsageLimitUsd: z.coerce.number().min(0).optional().nullable(),
|
||||
chaosModeEnabled: z.boolean().optional(),
|
||||
scopes: z.array(z.string().trim().min(1).max(64)).max(32).optional(),
|
||||
});
|
||||
|
||||
@@ -110,6 +111,7 @@ export const updateKeyPermissionsSchema = z
|
||||
usageLimitEnabled: z.boolean().optional(),
|
||||
dailyUsageLimitUsd: z.coerce.number().min(0).optional().nullable(),
|
||||
weeklyUsageLimitUsd: z.coerce.number().min(0).optional().nullable(),
|
||||
chaosModeEnabled: z.boolean().optional(),
|
||||
})
|
||||
.superRefine((value, ctx) => {
|
||||
if (
|
||||
@@ -133,7 +135,8 @@ export const updateKeyPermissionsSchema = z
|
||||
value.allowUsageCommand === undefined &&
|
||||
value.usageLimitEnabled === undefined &&
|
||||
value.dailyUsageLimitUsd === undefined &&
|
||||
value.weeklyUsageLimitUsd === undefined
|
||||
value.weeklyUsageLimitUsd === undefined &&
|
||||
value.chaosModeEnabled === undefined
|
||||
) {
|
||||
ctx.addIssue({
|
||||
code: z.ZodIssueCode.custom,
|
||||
|
||||
@@ -61,14 +61,28 @@ test("permissions modal switch buttons declare button type", () => {
|
||||
selfServiceBlock.match(/<button\s+type="button"\s+role="switch"/g) ?? []
|
||||
).length;
|
||||
|
||||
// Self-service Visibility block has 5 switches: own-usage visibility,
|
||||
// shared-account quota visibility, disable-non-public-models (#3041), the
|
||||
// per-key local usage command allowance (#4034), and the API-key provider
|
||||
// quota-policy bypass scope (#5731).
|
||||
// Self-service Visibility block has 4 inline switches: own-usage visibility,
|
||||
// shared-account quota visibility, disable-non-public-models (#3041), and the
|
||||
// per-key local usage command allowance (#4034). The API-key provider
|
||||
// quota-policy bypass scope (#5731) and the Chaos Mode access scope (#6728)
|
||||
// were extracted into dedicated toggle components (asserted below).
|
||||
// The invariant is that every switch declares type="button"
|
||||
// (typedSwitchButtonCount === switchButtonCount) to avoid implicit submit.
|
||||
assert.equal(switchButtonCount, 5);
|
||||
assert.equal(typedSwitchButtonCount, 5);
|
||||
assert.equal(switchButtonCount, 4);
|
||||
assert.equal(typedSwitchButtonCount, 4);
|
||||
|
||||
// The extracted toggle components keep the same invariant.
|
||||
for (const rel of [
|
||||
"src/app/(dashboard)/dashboard/api-manager/components/BypassProviderQuotaToggle.tsx",
|
||||
"src/app/(dashboard)/dashboard/api-manager/components/ChaosModeAccessToggle.tsx",
|
||||
]) {
|
||||
const componentSource = fs.readFileSync(path.join(repoRoot, rel), "utf8");
|
||||
const compSwitches = (componentSource.match(/role="switch"/g) ?? []).length;
|
||||
const compTyped = (componentSource.match(/<button\s+type="button"\s+role="switch"/g) ?? [])
|
||||
.length;
|
||||
assert.ok(compSwitches >= 1, `${rel} must render a switch`);
|
||||
assert.equal(compTyped, compSwitches, `${rel}: every switch declares type="button"`);
|
||||
}
|
||||
});
|
||||
|
||||
test("permissions modal exposes Claude Code default wildcard model", () => {
|
||||
|
||||
320
tests/unit/chaos-api-routes.test.ts
Normal file
320
tests/unit/chaos-api-routes.test.ts
Normal file
@@ -0,0 +1,320 @@
|
||||
/**
|
||||
* Unit tests for the 3 Chaos Mode API routes:
|
||||
* - GET/PUT/DELETE /api/chaos/config (management-session auth)
|
||||
* - POST /api/chaos/run (management-session auth)
|
||||
* - POST /api/skills/collect/chaos (external Bearer-token auth)
|
||||
*
|
||||
* Covers auth, Zod validation, and the "Chaos Mode disabled" 400 short-circuit
|
||||
* per Hard Rule #18 / the #6679 review checklist. Upstream model dispatch is
|
||||
* stubbed via chaosExecutor's `chatDispatch` mock point (see chaos-executor.test.ts)
|
||||
* so these tests never hit a real provider.
|
||||
*/
|
||||
import test, { mock } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import fs from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
|
||||
const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-chaos-routes-"));
|
||||
const ORIGINAL_DATA_DIR = process.env.DATA_DIR;
|
||||
const ORIGINAL_API_KEY_SECRET = process.env.API_KEY_SECRET;
|
||||
const ORIGINAL_INITIAL_PASSWORD = process.env.INITIAL_PASSWORD;
|
||||
|
||||
process.env.DATA_DIR = TEST_DATA_DIR;
|
||||
process.env.API_KEY_SECRET = process.env.API_KEY_SECRET ?? "chaos-routes-test-secret";
|
||||
|
||||
const core = await import("../../src/lib/db/core.ts");
|
||||
const apiKeysDb = await import("../../src/lib/db/apiKeys.ts");
|
||||
const providersDb = await import("../../src/lib/db/providers.ts");
|
||||
const chaosConfig = await import("../../src/lib/chaos/chaosConfig.ts");
|
||||
const chaosExecutor = await import("../../src/lib/chaos/chaosExecutor.ts");
|
||||
|
||||
const configRoute = await import("../../src/app/api/chaos/config/route.ts");
|
||||
const runRoute = await import("../../src/app/api/chaos/run/route.ts");
|
||||
const skillsChaosRoute = await import("../../src/app/api/skills/collect/chaos/route.ts");
|
||||
|
||||
async function resetStorage() {
|
||||
core.resetDbInstance();
|
||||
apiKeysDb.resetApiKeyState();
|
||||
// The DB is about to be wiped out from under it — drop the in-memory chaos
|
||||
// config cache too, or getChaosConfig() keeps serving a stale value (e.g. a
|
||||
// prior test's `enabled: true`) after resetDbInstance() below.
|
||||
chaosConfig.invalidateChaosConfigCache();
|
||||
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
|
||||
fs.mkdirSync(TEST_DATA_DIR, { recursive: true });
|
||||
delete process.env.INITIAL_PASSWORD;
|
||||
}
|
||||
|
||||
function makeRequest(method: string, url: string, body?: unknown, headers: Record<string, string> = {}) {
|
||||
return new Request(url, {
|
||||
method,
|
||||
headers: {
|
||||
...(body !== undefined ? { "content-type": "application/json" } : {}),
|
||||
...headers,
|
||||
},
|
||||
body: body !== undefined ? JSON.stringify(body) : undefined,
|
||||
});
|
||||
}
|
||||
|
||||
function jsonResponse(body: unknown, status = 200): Response {
|
||||
return new Response(JSON.stringify(body), {
|
||||
status,
|
||||
headers: { "Content-Type": "application/json" },
|
||||
});
|
||||
}
|
||||
|
||||
test.beforeEach(async () => {
|
||||
await resetStorage();
|
||||
});
|
||||
|
||||
test.afterEach(() => {
|
||||
mock.restoreAll();
|
||||
});
|
||||
|
||||
test.after(() => {
|
||||
core.resetDbInstance();
|
||||
apiKeysDb.resetApiKeyState();
|
||||
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
|
||||
|
||||
if (ORIGINAL_DATA_DIR === undefined) {
|
||||
delete process.env.DATA_DIR;
|
||||
} else {
|
||||
process.env.DATA_DIR = ORIGINAL_DATA_DIR;
|
||||
}
|
||||
if (ORIGINAL_API_KEY_SECRET === undefined) {
|
||||
delete process.env.API_KEY_SECRET;
|
||||
} else {
|
||||
process.env.API_KEY_SECRET = ORIGINAL_API_KEY_SECRET;
|
||||
}
|
||||
if (ORIGINAL_INITIAL_PASSWORD === undefined) {
|
||||
delete process.env.INITIAL_PASSWORD;
|
||||
} else {
|
||||
process.env.INITIAL_PASSWORD = ORIGINAL_INITIAL_PASSWORD;
|
||||
}
|
||||
});
|
||||
|
||||
// ═════════════════════════════════════════════════════════════════════════════
|
||||
// GET/PUT/DELETE /api/chaos/config
|
||||
// ═════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
test("GET /api/chaos/config — requires management auth when a password is configured", async () => {
|
||||
process.env.INITIAL_PASSWORD = "chaos-config-requires-login";
|
||||
|
||||
const res = await configRoute.GET(makeRequest("GET", "http://localhost/api/chaos/config"));
|
||||
assert.ok(res.status === 401 || res.status === 403, `expected 401/403, got ${res.status}`);
|
||||
|
||||
const body = (await res.json()) as { error: { message: string } | string };
|
||||
const errorMsg = typeof body.error === "string" ? body.error : body.error.message;
|
||||
assert.ok(!errorMsg.includes("at /"), "Hard Rule #12: no stack trace exposure");
|
||||
});
|
||||
|
||||
test("GET /api/chaos/config — returns defaults, PUT updates, DELETE resets", async () => {
|
||||
const getRes = await configRoute.GET(makeRequest("GET", "http://localhost/api/chaos/config"));
|
||||
assert.equal(getRes.status, 200);
|
||||
const getBody = (await getRes.json()) as { config: typeof chaosConfig.DEFAULT_CHAOS_CONFIG };
|
||||
// JSON.stringify drops keys whose value is `undefined` (systemPrompt), so compare
|
||||
// against the JSON round-tripped shape rather than the raw in-memory default.
|
||||
assert.deepEqual(
|
||||
getBody.config,
|
||||
JSON.parse(JSON.stringify(chaosConfig.DEFAULT_CHAOS_CONFIG))
|
||||
);
|
||||
|
||||
const putRes = await configRoute.PUT(
|
||||
makeRequest("PUT", "http://localhost/api/chaos/config", {
|
||||
enabled: true,
|
||||
defaultMode: "collaborative",
|
||||
providerOverrides: [],
|
||||
timeoutMs: 60_000,
|
||||
maxTokens: 8192,
|
||||
})
|
||||
);
|
||||
assert.equal(putRes.status, 200);
|
||||
const putBody = (await putRes.json()) as { config: { enabled: boolean; defaultMode: string } };
|
||||
assert.equal(putBody.config.enabled, true);
|
||||
assert.equal(putBody.config.defaultMode, "collaborative");
|
||||
|
||||
const deleteRes = await configRoute.DELETE(
|
||||
makeRequest("DELETE", "http://localhost/api/chaos/config")
|
||||
);
|
||||
assert.equal(deleteRes.status, 200);
|
||||
const deleteBody = (await deleteRes.json()) as { config: typeof chaosConfig.DEFAULT_CHAOS_CONFIG };
|
||||
// Same JSON.stringify undefined-key drop as the GET assertion above.
|
||||
assert.deepEqual(
|
||||
deleteBody.config,
|
||||
JSON.parse(JSON.stringify(chaosConfig.DEFAULT_CHAOS_CONFIG))
|
||||
);
|
||||
});
|
||||
|
||||
test("PUT /api/chaos/config — 400 on schema validation failure", async () => {
|
||||
const res = await configRoute.PUT(
|
||||
makeRequest("PUT", "http://localhost/api/chaos/config", {
|
||||
enabled: true,
|
||||
defaultMode: "not-a-real-mode",
|
||||
})
|
||||
);
|
||||
assert.equal(res.status, 400);
|
||||
});
|
||||
|
||||
// ═════════════════════════════════════════════════════════════════════════════
|
||||
// POST /api/chaos/run
|
||||
// ═════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
test("POST /api/chaos/run — requires management auth when a password is configured", async () => {
|
||||
process.env.INITIAL_PASSWORD = "chaos-run-requires-login";
|
||||
|
||||
const res = await runRoute.POST(
|
||||
makeRequest("POST", "http://localhost/api/chaos/run", { task: "hello" })
|
||||
);
|
||||
assert.ok(res.status === 401 || res.status === 403, `expected 401/403, got ${res.status}`);
|
||||
});
|
||||
|
||||
test("POST /api/chaos/run — 400 when Chaos Mode is not enabled globally", async () => {
|
||||
const res = await runRoute.POST(
|
||||
makeRequest("POST", "http://localhost/api/chaos/run", { task: "hello" })
|
||||
);
|
||||
assert.equal(res.status, 400);
|
||||
const body = (await res.json()) as { error: { message: string } };
|
||||
assert.match(body.error.message, /not enabled/i);
|
||||
});
|
||||
|
||||
test("POST /api/chaos/run — 400 on invalid body (missing task)", async () => {
|
||||
await chaosConfig.setChaosConfig({
|
||||
enabled: true,
|
||||
defaultMode: "parallel",
|
||||
providerOverrides: [],
|
||||
timeoutMs: 120_000,
|
||||
maxTokens: 4096,
|
||||
});
|
||||
|
||||
const res = await runRoute.POST(makeRequest("POST", "http://localhost/api/chaos/run", {}));
|
||||
assert.equal(res.status, 400);
|
||||
});
|
||||
|
||||
test("POST /api/chaos/run — 200 happy path dispatches without an Authorization header (dashboard/local mode)", async () => {
|
||||
await chaosConfig.setChaosConfig({
|
||||
enabled: true,
|
||||
defaultMode: "parallel",
|
||||
providerOverrides: [],
|
||||
timeoutMs: 120_000,
|
||||
maxTokens: 4096,
|
||||
});
|
||||
await providersDb.createProviderConnection({
|
||||
provider: "openai",
|
||||
authType: "apikey",
|
||||
name: "Run Route Provider",
|
||||
apiKey: "sk-run-route",
|
||||
defaultModel: "gpt-4o-mini",
|
||||
});
|
||||
|
||||
let capturedAuth: string | null | undefined;
|
||||
mock.method(chaosExecutor.chatDispatch, "postChatCompletion", async (req: Request) => {
|
||||
capturedAuth = req.headers.get("Authorization");
|
||||
return jsonResponse({ choices: [{ message: { content: "dashboard result" } }] });
|
||||
});
|
||||
|
||||
const res = await runRoute.POST(
|
||||
makeRequest("POST", "http://localhost/api/chaos/run", { task: "Summarize" })
|
||||
);
|
||||
assert.equal(res.status, 200);
|
||||
const body = (await res.json()) as { models: { status: string; content: string }[] };
|
||||
assert.equal(body.models[0].status, "success");
|
||||
assert.equal(body.models[0].content, "dashboard result");
|
||||
assert.equal(capturedAuth, null);
|
||||
});
|
||||
|
||||
// ═════════════════════════════════════════════════════════════════════════════
|
||||
// POST /api/skills/collect/chaos
|
||||
// ═════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
test("POST /api/skills/collect/chaos — 401 without an Authorization header", async () => {
|
||||
const res = await skillsChaosRoute.POST(
|
||||
makeRequest("POST", "http://localhost/api/skills/collect/chaos", { task: "hello" })
|
||||
);
|
||||
assert.equal(res.status, 401);
|
||||
});
|
||||
|
||||
test("POST /api/skills/collect/chaos — 403 with an invalid API key", async () => {
|
||||
const res = await skillsChaosRoute.POST(
|
||||
makeRequest(
|
||||
"POST",
|
||||
"http://localhost/api/skills/collect/chaos",
|
||||
{ task: "hello" },
|
||||
{ Authorization: "Bearer sk-does-not-exist" }
|
||||
)
|
||||
);
|
||||
assert.equal(res.status, 403);
|
||||
});
|
||||
|
||||
test("POST /api/skills/collect/chaos — 403 when the API key does not have chaosModeEnabled", async () => {
|
||||
const created = await apiKeysDb.createApiKey("No Chaos Key", "machine-chaos-01");
|
||||
|
||||
const res = await skillsChaosRoute.POST(
|
||||
makeRequest(
|
||||
"POST",
|
||||
"http://localhost/api/skills/collect/chaos",
|
||||
{ task: "hello" },
|
||||
{ Authorization: `Bearer ${created.key}` }
|
||||
)
|
||||
);
|
||||
assert.equal(res.status, 403);
|
||||
const body = (await res.json()) as { error: { message: string } };
|
||||
assert.match(body.error.message, /Chaos Mode is not enabled for this API key/);
|
||||
});
|
||||
|
||||
test("POST /api/skills/collect/chaos — 400 when Chaos Mode is disabled globally even with a valid key", async () => {
|
||||
const created = await apiKeysDb.createApiKey("Chaos Key", "machine-chaos-02");
|
||||
await apiKeysDb.updateApiKeyPermissions(created.id, { chaosModeEnabled: true });
|
||||
apiKeysDb.clearApiKeyCaches();
|
||||
|
||||
const res = await skillsChaosRoute.POST(
|
||||
makeRequest(
|
||||
"POST",
|
||||
"http://localhost/api/skills/collect/chaos",
|
||||
{ task: "hello" },
|
||||
{ Authorization: `Bearer ${created.key}` }
|
||||
)
|
||||
);
|
||||
assert.equal(res.status, 400);
|
||||
});
|
||||
|
||||
test("POST /api/skills/collect/chaos — 200 happy path forwards the caller's key to the in-process dispatch", async () => {
|
||||
await chaosConfig.setChaosConfig({
|
||||
enabled: true,
|
||||
defaultMode: "parallel",
|
||||
providerOverrides: [],
|
||||
timeoutMs: 120_000,
|
||||
maxTokens: 4096,
|
||||
});
|
||||
await providersDb.createProviderConnection({
|
||||
provider: "openai",
|
||||
authType: "apikey",
|
||||
name: "Skills Route Provider",
|
||||
apiKey: "sk-skills-route",
|
||||
defaultModel: "gpt-4o-mini",
|
||||
});
|
||||
|
||||
const created = await apiKeysDb.createApiKey("Chaos Key External", "machine-chaos-03");
|
||||
await apiKeysDb.updateApiKeyPermissions(created.id, { chaosModeEnabled: true });
|
||||
apiKeysDb.clearApiKeyCaches();
|
||||
|
||||
let capturedAuth: string | null | undefined;
|
||||
mock.method(chaosExecutor.chatDispatch, "postChatCompletion", async (req: Request) => {
|
||||
capturedAuth = req.headers.get("Authorization");
|
||||
return jsonResponse({ choices: [{ message: { content: "external result" } }] });
|
||||
});
|
||||
|
||||
const res = await skillsChaosRoute.POST(
|
||||
makeRequest(
|
||||
"POST",
|
||||
"http://localhost/api/skills/collect/chaos",
|
||||
{ task: "Summarize" },
|
||||
{ Authorization: `Bearer ${created.key}` }
|
||||
)
|
||||
);
|
||||
|
||||
assert.equal(res.status, 200);
|
||||
const body = (await res.json()) as { models: { status: string; content: string }[] };
|
||||
assert.equal(body.models[0].status, "success");
|
||||
assert.equal(body.models[0].content, "external result");
|
||||
assert.equal(capturedAuth, `Bearer ${created.key}`);
|
||||
});
|
||||
126
tests/unit/chaos-config.test.ts
Normal file
126
tests/unit/chaos-config.test.ts
Normal file
@@ -0,0 +1,126 @@
|
||||
/**
|
||||
* Unit tests for src/lib/chaos/chaosConfig.ts
|
||||
*
|
||||
* Regression coverage for #6679: the original PR queried `SELECT/INSERT/DELETE
|
||||
* ... FROM settings ...` against a table that was never created in this schema
|
||||
* (only `key_value` exists) — every read silently fell back to
|
||||
* DEFAULT_CHAOS_CONFIG and every write/reset threw at runtime. This module now
|
||||
* routes persistence through src/lib/db/settings.ts::getSettings/updateSettings
|
||||
* (the `key_value` table, namespace 'settings'), matching the repo convention
|
||||
* (CLAUDE.md → Database: never write raw SQL outside src/lib/db/).
|
||||
*/
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import fs from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
|
||||
const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-chaos-config-"));
|
||||
const ORIGINAL_DATA_DIR = process.env.DATA_DIR;
|
||||
process.env.DATA_DIR = TEST_DATA_DIR;
|
||||
|
||||
const core = await import("../../src/lib/db/core.ts");
|
||||
const chaosConfig = await import("../../src/lib/chaos/chaosConfig.ts");
|
||||
|
||||
async function resetStorage() {
|
||||
core.resetDbInstance();
|
||||
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
|
||||
fs.mkdirSync(TEST_DATA_DIR, { recursive: true });
|
||||
}
|
||||
|
||||
test.beforeEach(async () => {
|
||||
await resetStorage();
|
||||
});
|
||||
|
||||
test.after(() => {
|
||||
core.resetDbInstance();
|
||||
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
|
||||
if (ORIGINAL_DATA_DIR === undefined) {
|
||||
delete process.env.DATA_DIR;
|
||||
} else {
|
||||
process.env.DATA_DIR = ORIGINAL_DATA_DIR;
|
||||
}
|
||||
});
|
||||
|
||||
test("getChaosConfig returns DEFAULT_CHAOS_CONFIG when nothing is persisted yet", async () => {
|
||||
const config = await chaosConfig.getChaosConfig();
|
||||
assert.deepEqual(config, chaosConfig.DEFAULT_CHAOS_CONFIG);
|
||||
});
|
||||
|
||||
test("setChaosConfig persists via the settings store and getChaosConfig reads it back", async () => {
|
||||
const settingsDb = await import("../../src/lib/db/settings.ts");
|
||||
|
||||
const written = await chaosConfig.setChaosConfig({
|
||||
enabled: true,
|
||||
defaultMode: "collaborative",
|
||||
providerOverrides: [{ providerId: "openai", modelId: "gpt-4o", enabled: true }],
|
||||
systemPrompt: "Custom prompt",
|
||||
timeoutMs: 60_000,
|
||||
maxTokens: 8192,
|
||||
});
|
||||
|
||||
assert.equal(written.enabled, true);
|
||||
assert.equal(written.defaultMode, "collaborative");
|
||||
|
||||
const readBack = await chaosConfig.getChaosConfig();
|
||||
assert.deepEqual(readBack, written);
|
||||
|
||||
// Confirms the write actually lands in key_value (namespace 'settings'), not a
|
||||
// nonexistent `settings` table (the bug this fix replaces).
|
||||
const rawSettings = await settingsDb.getSettings();
|
||||
assert.equal((rawSettings as Record<string, unknown>).chaosModeConfig !== undefined, true);
|
||||
});
|
||||
|
||||
test("setChaosConfig rejects invalid config (Zod parse, not raw SQL failure)", async () => {
|
||||
const invalidConfig = {
|
||||
enabled: true,
|
||||
defaultMode: "not-a-real-mode",
|
||||
providerOverrides: [],
|
||||
timeoutMs: 60_000,
|
||||
maxTokens: 4096,
|
||||
} as unknown as Awaited<ReturnType<typeof chaosConfig.getChaosConfig>>;
|
||||
|
||||
await assert.rejects(() => chaosConfig.setChaosConfig(invalidConfig));
|
||||
});
|
||||
|
||||
test("resetChaosConfig clears the stored config back to defaults", async () => {
|
||||
await chaosConfig.setChaosConfig({
|
||||
enabled: true,
|
||||
defaultMode: "parallel",
|
||||
providerOverrides: [],
|
||||
timeoutMs: 90_000,
|
||||
maxTokens: 2048,
|
||||
});
|
||||
assert.equal((await chaosConfig.getChaosConfig()).enabled, true);
|
||||
|
||||
const reset = await chaosConfig.resetChaosConfig();
|
||||
assert.deepEqual(reset, chaosConfig.DEFAULT_CHAOS_CONFIG);
|
||||
assert.deepEqual(await chaosConfig.getChaosConfig(), chaosConfig.DEFAULT_CHAOS_CONFIG);
|
||||
});
|
||||
|
||||
test("getChaosConfig falls back to defaults when the stored payload fails schema validation", async () => {
|
||||
const settingsDb = await import("../../src/lib/db/settings.ts");
|
||||
|
||||
await settingsDb.updateSettings({
|
||||
chaosModeConfig: { enabled: "not-a-boolean", defaultMode: "parallel" },
|
||||
});
|
||||
|
||||
const config = await chaosConfig.getChaosConfig();
|
||||
assert.deepEqual(config, chaosConfig.DEFAULT_CHAOS_CONFIG);
|
||||
});
|
||||
|
||||
test("in-memory cache is invalidated on write so config updates are visible immediately", async () => {
|
||||
const first = await chaosConfig.getChaosConfig();
|
||||
assert.equal(first.enabled, false);
|
||||
|
||||
await chaosConfig.setChaosConfig({
|
||||
enabled: true,
|
||||
defaultMode: "parallel",
|
||||
providerOverrides: [],
|
||||
timeoutMs: 120_000,
|
||||
maxTokens: 4096,
|
||||
});
|
||||
|
||||
const second = await chaosConfig.getChaosConfig();
|
||||
assert.equal(second.enabled, true);
|
||||
});
|
||||
243
tests/unit/chaos-executor.test.ts
Normal file
243
tests/unit/chaos-executor.test.ts
Normal file
@@ -0,0 +1,243 @@
|
||||
/**
|
||||
* Unit tests for src/lib/chaos/chaosExecutor.ts
|
||||
*
|
||||
* Regression coverage for #6679: the original PR dispatched to models via a real
|
||||
* network fetch() to `${OMNIROUTE_INTERNAL_URL || "http://localhost:30129"}` —
|
||||
* 30129 is the author's personal dev port, not the project's default port 20128,
|
||||
* so Chaos Mode silently failed on any standard install. This now dispatches
|
||||
* in-process via the established synthetic-Request/route-handler pattern (see
|
||||
* src/lib/batches/dispatch.ts, src/lib/evals/runtime.ts) — no network hop, no
|
||||
* port dependency. `chatDispatch.postChatCompletion` is mocked here (same
|
||||
* technique tests/unit/batch-processor.test.ts uses for dispatch.ts) so these
|
||||
* tests never hit a real upstream provider.
|
||||
*/
|
||||
import test, { mock } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import fs from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
|
||||
const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-chaos-executor-"));
|
||||
const ORIGINAL_DATA_DIR = process.env.DATA_DIR;
|
||||
process.env.DATA_DIR = TEST_DATA_DIR;
|
||||
|
||||
const core = await import("../../src/lib/db/core.ts");
|
||||
const providersDb = await import("../../src/lib/db/providers.ts");
|
||||
const chaosConfig = await import("../../src/lib/chaos/chaosConfig.ts");
|
||||
const chaosExecutor = await import("../../src/lib/chaos/chaosExecutor.ts");
|
||||
|
||||
async function resetStorage() {
|
||||
core.resetDbInstance();
|
||||
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
|
||||
fs.mkdirSync(TEST_DATA_DIR, { recursive: true });
|
||||
}
|
||||
|
||||
test.beforeEach(async () => {
|
||||
await resetStorage();
|
||||
});
|
||||
|
||||
test.afterEach(() => {
|
||||
mock.restoreAll();
|
||||
});
|
||||
|
||||
test.after(() => {
|
||||
core.resetDbInstance();
|
||||
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
|
||||
if (ORIGINAL_DATA_DIR === undefined) {
|
||||
delete process.env.DATA_DIR;
|
||||
} else {
|
||||
process.env.DATA_DIR = ORIGINAL_DATA_DIR;
|
||||
}
|
||||
});
|
||||
|
||||
function jsonResponse(body: unknown, status = 200): Response {
|
||||
return new Response(JSON.stringify(body), {
|
||||
status,
|
||||
headers: { "Content-Type": "application/json" },
|
||||
});
|
||||
}
|
||||
|
||||
test("executeChaosRun throws when no active provider connections exist", async () => {
|
||||
await assert.rejects(
|
||||
() => chaosExecutor.executeChaosRun({ task: "hello" }),
|
||||
/No active provider connections found/
|
||||
);
|
||||
});
|
||||
|
||||
test("executeChaosRun dispatches in-process (no fetch/network call) and returns per-provider results", async () => {
|
||||
await providersDb.createProviderConnection({
|
||||
provider: "openai",
|
||||
authType: "apikey",
|
||||
name: "Test OpenAI",
|
||||
apiKey: "sk-test-openai",
|
||||
defaultModel: "gpt-4o-mini",
|
||||
});
|
||||
|
||||
let capturedRequest: Request | null = null;
|
||||
mock.method(chaosExecutor.chatDispatch, "postChatCompletion", async (req: Request) => {
|
||||
capturedRequest = req;
|
||||
return jsonResponse({ choices: [{ message: { content: "hello from openai" } }] });
|
||||
});
|
||||
|
||||
const result = await chaosExecutor.executeChaosRun({ task: "Summarize this repo" });
|
||||
|
||||
assert.equal(result.mode, "parallel");
|
||||
assert.equal(result.totalProviders, 1);
|
||||
assert.equal(result.models.length, 1);
|
||||
assert.equal(result.models[0].status, "success");
|
||||
assert.equal(result.models[0].content, "hello from openai");
|
||||
|
||||
assert.ok(capturedRequest, "postChatCompletion should have been invoked in-process");
|
||||
assert.equal(
|
||||
(capturedRequest as unknown as Request).url,
|
||||
"http://localhost/api/v1/chat/completions"
|
||||
);
|
||||
});
|
||||
|
||||
test("executeChaosRun forwards the caller's apiKey as a Bearer Authorization header", async () => {
|
||||
await providersDb.createProviderConnection({
|
||||
provider: "anthropic",
|
||||
authType: "apikey",
|
||||
name: "Test Anthropic",
|
||||
apiKey: "sk-test-anthropic",
|
||||
defaultModel: "claude-3-5-sonnet",
|
||||
});
|
||||
|
||||
let capturedAuth: string | null = null;
|
||||
mock.method(chaosExecutor.chatDispatch, "postChatCompletion", async (req: Request) => {
|
||||
capturedAuth = req.headers.get("Authorization");
|
||||
return jsonResponse({ choices: [{ message: { content: "ok" } }] });
|
||||
});
|
||||
|
||||
await chaosExecutor.executeChaosRun({ task: "task", apiKey: "sk-caller-key" });
|
||||
|
||||
assert.equal(capturedAuth, "Bearer sk-caller-key");
|
||||
});
|
||||
|
||||
test("executeChaosRun omits Authorization header when no apiKey is provided (dashboard/local mode)", async () => {
|
||||
await providersDb.createProviderConnection({
|
||||
provider: "openai",
|
||||
authType: "apikey",
|
||||
name: "Test OpenAI",
|
||||
apiKey: "sk-test-openai-2",
|
||||
defaultModel: "gpt-4o-mini",
|
||||
});
|
||||
|
||||
let capturedAuth: string | null | undefined;
|
||||
mock.method(chaosExecutor.chatDispatch, "postChatCompletion", async (req: Request) => {
|
||||
capturedAuth = req.headers.get("Authorization");
|
||||
return jsonResponse({ choices: [{ message: { content: "ok" } }] });
|
||||
});
|
||||
|
||||
await chaosExecutor.executeChaosRun({ task: "task" });
|
||||
|
||||
assert.equal(capturedAuth, null);
|
||||
});
|
||||
|
||||
test("executeChaosRun surfaces upstream errors per-model instead of throwing", async () => {
|
||||
await providersDb.createProviderConnection({
|
||||
provider: "openai",
|
||||
authType: "apikey",
|
||||
name: "Failing OpenAI",
|
||||
apiKey: "sk-test-failing",
|
||||
defaultModel: "gpt-4o-mini",
|
||||
});
|
||||
|
||||
mock.method(chaosExecutor.chatDispatch, "postChatCompletion", async () =>
|
||||
jsonResponse({ error: "upstream exploded" }, 502)
|
||||
);
|
||||
|
||||
const result = await chaosExecutor.executeChaosRun({ task: "task" });
|
||||
|
||||
assert.equal(result.models[0].status, "error");
|
||||
assert.match(result.models[0].error ?? "", /API 502/);
|
||||
});
|
||||
|
||||
test("executeChaosRun collaborative mode chains context between sequential model calls", async () => {
|
||||
await providersDb.createProviderConnection({
|
||||
provider: "openai",
|
||||
authType: "apikey",
|
||||
name: "Model A",
|
||||
apiKey: "sk-test-a",
|
||||
defaultModel: "gpt-4o-mini",
|
||||
});
|
||||
await providersDb.createProviderConnection({
|
||||
provider: "anthropic",
|
||||
authType: "apikey",
|
||||
name: "Model B",
|
||||
apiKey: "sk-test-b",
|
||||
defaultModel: "claude-3-5-sonnet",
|
||||
});
|
||||
|
||||
const seenUserMessages: string[] = [];
|
||||
mock.method(chaosExecutor.chatDispatch, "postChatCompletion", async (req: Request) => {
|
||||
const body = (await req.json()) as { messages: { role: string; content: string }[] };
|
||||
const userMessage = body.messages.find((m) => m.role === "user");
|
||||
seenUserMessages.push(userMessage?.content ?? "");
|
||||
return jsonResponse({
|
||||
choices: [{ message: { content: `reply-${seenUserMessages.length}` } }],
|
||||
});
|
||||
});
|
||||
|
||||
const result = await chaosExecutor.executeChaosRun({
|
||||
task: "Original task",
|
||||
mode: "collaborative",
|
||||
});
|
||||
|
||||
assert.equal(result.mode, "collaborative");
|
||||
assert.equal(result.models.length, 2);
|
||||
assert.equal(seenUserMessages[0], "Original task");
|
||||
assert.match(seenUserMessages[1], /Previous model's output:\nreply-1/);
|
||||
assert.ok(result.summary && result.summary.includes("reply-2"));
|
||||
});
|
||||
|
||||
test("executeChaosRun respects an explicit providers filter and errors when none match", async () => {
|
||||
await providersDb.createProviderConnection({
|
||||
provider: "openai",
|
||||
authType: "apikey",
|
||||
name: "Only OpenAI",
|
||||
apiKey: "sk-test-only-openai",
|
||||
defaultModel: "gpt-4o-mini",
|
||||
});
|
||||
|
||||
mock.method(chaosExecutor.chatDispatch, "postChatCompletion", async () =>
|
||||
jsonResponse({ choices: [{ message: { content: "ok" } }] })
|
||||
);
|
||||
|
||||
await assert.rejects(
|
||||
() => chaosExecutor.executeChaosRun({ task: "task", providers: ["anthropic"] }),
|
||||
/None of the specified providers are active/
|
||||
);
|
||||
|
||||
const result = await chaosExecutor.executeChaosRun({ task: "task", providers: ["openai"] });
|
||||
assert.equal(result.models.length, 1);
|
||||
});
|
||||
|
||||
test("executeChaosRun applies global config provider overrides for model resolution", async () => {
|
||||
await providersDb.createProviderConnection({
|
||||
provider: "openai",
|
||||
authType: "apikey",
|
||||
name: "Override target",
|
||||
apiKey: "sk-test-override",
|
||||
defaultModel: "gpt-4o-mini",
|
||||
});
|
||||
|
||||
await chaosConfig.setChaosConfig({
|
||||
enabled: true,
|
||||
defaultMode: "parallel",
|
||||
providerOverrides: [{ providerId: "openai", modelId: "gpt-4-turbo", enabled: true }],
|
||||
timeoutMs: 120_000,
|
||||
maxTokens: 4096,
|
||||
});
|
||||
|
||||
let capturedModel: string | undefined;
|
||||
mock.method(chaosExecutor.chatDispatch, "postChatCompletion", async (req: Request) => {
|
||||
const body = (await req.json()) as { model: string };
|
||||
capturedModel = body.model;
|
||||
return jsonResponse({ choices: [{ message: { content: "ok" } }] });
|
||||
});
|
||||
|
||||
await chaosExecutor.executeChaosRun({ task: "task" });
|
||||
|
||||
assert.equal(capturedModel, "gpt-4-turbo");
|
||||
});
|
||||
Reference in New Issue
Block a user