feat(combo): sanitized diagnostic trace on auto-combo terminal failure (#6545)

Merged — thank you, @developerjillur! Sanitized diagnostic trace on an auto-combo terminal failure (ids/reason-codes only, capped), plus an actionable reasoning-budget-exhausted message. Integrated into release/v3.8.47.

CHANGELOG.md

@@ -22,6 +22,7 @@ _Living section — bullets land here as PRs merge into `release/v3.8.47` (paral
 - **fix(api):** tool-call arguments could render as `[object Object]` sequences instead of the real JSON through the `/anthropic` (Anthropic-shape `/messages`) routing path ([#6459](https://github.com/diegosouzapw/OmniRoute/issues/6459)) — `appendToolCallArgumentDelta()` (`open-sse/utils/toolCallArguments.ts`), the shared accumulator the streaming `openai-to-claude` response translator, `openai-responses` translator, and `responsesTransformer` all call to build up a tool call's `arguments`/`input_json_delta` buffer, treated any non-string `incoming` fragment as an empty string. Some upstreams deliver the full `tool_calls[].function.arguments` value as an already-parsed JSON object/array instead of the OpenAI-contracted JSON-encoded string; the old code silently discarded that fragment, leaving `tool_use.input` empty, and left downstream buffers open to a plain string coercion of the object (`[object Object]`) once client-side concatenation kicked in. `appendToolCallArgumentDelta()` now `JSON.stringify()`s a non-string, non-null object/array fragment into a valid JSON fragment instead of dropping it, so the assembled `partial_json` always parses back into the original structured value. Regression guard: `tests/unit/anthropic-toolcall-args-6459.test.ts`. (thanks @chirag127)
 - **fix(providers):** `fusion` combo returned the opaque `"All fusion panel models failed"` 503 even when only a minority of panel members were actually cooling down / rate-limited, and a user-supplied `fusionTuning.minPanel=1` was silently overridden ([#6454](https://github.com/diegosouzapw/OmniRoute/issues/6454)) — `handleFusionChat()` hard-clamped the quorum floor via `Math.min(Math.max(2, cfg.minPanel), panel.length)`, so an operator-configured `minPanel=1` never took effect: `collectPanel()`'s straggler-grace timer only starts once `ok >= minPanel`, and with the floor forced to 2 a single fast success plus N slow-failing stragglers never reached quorum, so the panel sat waiting instead of degrading to the survivor. Per-member failure reasons (`straggler_dropped`/`timeout`/`threw`/`status_XXX`/`empty_content`/`unparseable`) were also logged server-side but never surfaced in the 503 body, leaving operators unable to tell a rate-limit fan-fail from a broader outage. Fixed by honoring `Math.max(1, cfg.minPanel)` and threading a `failures: Array<{ model, reason }>` collector into the 503 message (`model=reason` per entry) — production fix already merged via #6521; this entry backfills the missing CHANGELOG bullet and adds an 11-member, `fusion-free`-scale regression test matching the original repro shape (a cooling minority must not sink a healthy majority; a genuinely all-failed panel still returns the documented 503). Regression guard: `tests/unit/services/fusion-min-panel-and-failure-detail.test.ts` + `tests/unit/fusion-partial-panel-failure-6454.test.ts`. (thanks @chirag127)
 - **fix(providers):** `fusion` combo strategy silently returned a panel member's raw answer instead of the configured `config.judgeModel` synthesis ([#6455](https://github.com/diegosouzapw/OmniRoute/issues/6455)) — `handleFusionChat()`'s single-survivor "degrade gracefully" path (added for #6454) returned the lone panel answer directly whenever only one panelist succeeded, regardless of whether an explicit `judgeModel` was configured; with the default `minPanel: 2` and a 2-model panel, any single flaky/rate-limited panelist forced this path on every request, so the configured judge (e.g. `auto/claude-opus`) was never invoked and the client-visible `.model` reflected whichever panelist happened to survive. The judge is now still invoked to synthesize a lone surviving answer whenever `judgeModel` is explicitly configured; the cheap direct-answer shortcut is kept only for the implicit case (no `judgeModel` set, where the "judge" is just `panel[0]`). Regression guard: `tests/unit/fusion-judge-model-6455.test.ts` + updated `tests/unit/combo-fusion-strategy.test.ts`. (thanks @chirag127)
+- **feat(combo):** sanitized diagnostic trace on an auto-combo terminal failure — instead of an opaque 503, a terminal combo failure now returns a whitelist-projected trace (candidate pool size, attempted count, excluded provider/reason codes, attempt order, and a terminal-reason code) via the new `errorResponseWithComboDiagnostics()`/`sanitizeComboDiagnostics()` in `open-sse/utils/error.ts` — provider/model ids and enumerated reason codes only, never keys/tokens/bodies, length- and count-capped. A reasoning-budget-exhausted panel now returns an actionable "increase max_tokens" message rather than a blind retry-limit 503. Regression guard: `tests/unit/combo-diagnostics-trace.test.ts`. ([#6545](https://github.com/diegosouzapw/OmniRoute/pull/6545)) — see PR. (thanks @developerjillur)
 - **fix(providers):** image/diffusion models discovered from an upstream catalog (e.g. HuggingFace's live `/v1/models`) are no longer advertised as chat models ([#6457](https://github.com/diegosouzapw/OmniRoute/issues/6457)) — the chat catalog builder defaulted synced models with no modality info to `endpoints: ["chat"]`, so `huggingface/stabilityai/stable-diffusion-xl-base-1.0` showed up in the chat `/v1/models` listing and returned `400 "not a chat model"` when called. `catalog.ts` now skips any synced model already registered as an image model for that provider (via the new `isRegisteredImageModel()`), leaving `getAllImageModels()` to list it with the correct `type: "image"`. Regression guard: `tests/unit/image-model-not-in-chat-catalog-6457.test.ts`.
 - **fix(test):** replace the bare `expect(true).toBe(true)` tautology in `playground-api-tab.test.tsx`'s SSE test and close the `check:test-masking` gap that let it slip through for a full cycle ([#6404](https://github.com/diegosouzapw/OmniRoute/issues/6404)) — a prior pass (#6548) had already swapped the literal to `expect(sendBtn).toBeDefined()`, but that stayed just as vacuous: the test's fetch mock returned an empty `/v1/models` list, so `ApiTab`'s Send button is always `disabled` (`!selectedModel`) and the SSE branch never runs — the "SSE infra is verified" comment was never true. The test now mocks a real model, drives the model `<select>` to enable Send, asserts `sendBtn.disabled === false` before clicking, and asserts the streamed SSE delta (`"Hello!"`) actually reached the response editor. Root cause on the detector side: `check-test-masking.mjs`'s tautology subcheck only compares base-vs-HEAD counts within a PR's own diff (`headExtTaut > baseExtTaut`) and no-ops locally when `GITHUB_BASE_SHA`/`GITHUB_BASE_REF` are unset ("sem base ref — pulando") — so a tautology merged once, or checked with a bare local run, was invisible forever after. Added a new always-on, PR-independent absolute-floor scan (`scanBareTautologies` + `countBareTautologies`) over every git-tracked test file for the bare `expect(true).toBe(true)` / `assert.equal(1,1)` / `assert.strictEqual(1,1)` patterns specifically (deliberately excluding `assert.ok(true)`, which has ~15 pre-existing verified-legitimate try/catch-fallback uses repo-wide and stays governed by the lenient diff-only subcheck) — verified zero pre-existing hits repo-wide once this file was fixed, so the new floor is safe to enforce unconditionally. Regression guard: `tests/unit/check-test-masking.test.ts` (new `scanBareTautologies`/`countBareTautologies` cases) + `tests/unit/ui/playground-api-tab.test.tsx`. (thanks @chirag127)
 - **fix(oauth):** Codex/ChatGPT (and every other OAuth provider) connection stays stuck showing "Auth Failed" even after a genuinely successful token refresh ([#6352](https://github.com/diegosouzapw/OmniRoute/issues/6352)) — `updateProviderCredentials()` (the shared `onPersist` callback for the manual "Refresh token" route, the reactive per-request refresh in `chat.ts`, and the Codex/Claude auth-file importers) correctly reused the stored `refresh_token`, persisted the new `access_token`, and replaced a rotated `refresh_token`, but never cleared the stale `testStatus`/`lastError*`/`errorCode` fields left over from a prior expired/invalid refresh or upstream 401/403 — only the separate background health-check sweep did that clearing. A successful refresh now resets `testStatus` to `"active"` and clears `lastError`, `lastErrorAt`, `lastErrorType`, `lastErrorSource`, and `errorCode` (an explicit `testStatus` from the caller still wins). Regression guard: `tests/unit/codex-oauth-refresh-persist-6352.test.ts`.

open-sse/services/combo.ts

@@ -18,7 +18,12 @@ import {
   recordProviderFailure,
   selectLockoutCooldownMs,
 } from "./accountFallback.ts";
-import { errorResponse, unavailableResponse } from "../utils/error.ts";
+import {
+  errorResponse,
+  unavailableResponse,
+  errorResponseWithComboDiagnostics,
+} from "../utils/error.ts";
+import type { ComboDiagnostics } from "../utils/error.ts";
 import { buildTargetTimeoutRunner } from "./combo/targetTimeoutRunner.ts";
 import { recordComboRequest, recordComboShadowRequest, getComboMetrics } from "./comboMetrics.ts";
 import {
@@ -1222,8 +1227,24 @@ export async function handleComboChat({
   // 16 strategies (priority, weighted, etc.) that funnel through executeTarget.
   const quotaCutoffResetWindowConfig = resolveResetWindowConfig(config as Record<string, unknown>);
 
+  // QA P0 diagnostics: record the order in which targets were actually attempted
+  // (provider/model ids only) so a terminal combo failure can report the attempt
+  // sequence alongside pool size + exhaustion reasons. Accumulates across set retries.
+  const comboAttemptOrder: Array<{ provider: string; model: string }> = [];
+
   if (orderedTargets.length === 0) {
-    return comboModelNotFoundResponse("Combo has no executable targets");
+    return errorResponseWithComboDiagnostics(
+      404,
+      "Combo has no executable targets",
+      {
+        poolSize: 0,
+        attempted: 0,
+        excluded: [],
+        attemptOrder: [],
+        terminalReason: "no_executable_targets",
+      },
+      { code: "model_not_found", type: "invalid_request_error" }
+    );
   }
 
   scheduleShadowRouting(
@@ -1306,6 +1327,23 @@ export async function handleComboChat({
       let fallbackCount = 0;
       let recordedAttempts = 0;
 
+      // QA P0: assemble a sanitized diagnostic trace from the state already in scope
+      // (pool size + this set-try's exhausted providers/connections + attempt order +
+      // a terminal-reason code). Never touches keys/tokens — provider/model ids only.
+      const buildComboDiag = (terminalReason: string): ComboDiagnostics => ({
+        poolSize: orderedTargets.length,
+        attempted: recordedAttempts,
+        excluded: [
+          ...[...exhaustedProviders].map((p) => ({ provider: p, reason: "exhausted" })),
+          ...[...exhaustedConnections].map((c) => ({
+            provider: "unknown",
+            reason: `exhausted_connection:${String(c).slice(0, 8)}`,
+          })),
+        ],
+        attemptOrder: comboAttemptOrder,
+        terminalReason,
+      });
+
       let globalResolve: ((res: Response) => void) | null = null;
       const globalPromise = new Promise<Response>((res) => {
         globalResolve = res;
@@ -1442,7 +1480,23 @@ export async function handleComboChat({
               "COMBO",
               `Maximum combo attempts (${MAX_GLOBAL_ATTEMPTS}) exceeded across all targets and fallbacks. Terminating loop to prevent runaway background requests.`
             );
-            return { ok: false, response: errorResponse(503, "Maximum combo retry limit reached") };
+            // Actionable failure instead of an opaque 503 when every candidate
+            // failed the same recoverable way. If the dominant cause was reasoning
+            // models exhausting a too-small max_tokens budget (no content output),
+            // retrying other models can't help — tell the caller to raise max_tokens.
+            const reasoningExhausted = /reasoning consumed \d+\/\d+ tokens/.test(lastError || "");
+            return {
+              ok: false,
+              response: errorResponseWithComboDiagnostics(
+                503,
+                reasoningExhausted
+                  ? "All combo candidates exhausted their token budget on reasoning without producing content. Increase max_tokens — reasoning models need a larger budget to emit content."
+                  : "Maximum combo retry limit reached",
+                buildComboDiag(
+                  reasoningExhausted ? "reasoning_budget_exhausted" : "max_attempts_exceeded"
+                )
+              ),
+            };
           }
 
           // Predictive TTFT Circuit Breaker (skip slow models)
@@ -1500,6 +1554,8 @@ export async function handleComboChat({
             timestamp: Date.now(),
             strategy,
           });
+          // QA P0 diagnostics: capture the attempt order (provider/model ids only).
+          comboAttemptOrder.push({ provider: provider ?? "unknown", model: modelStr });
 
           // Deep clone the body to ensure context preservation and prevent mutations
           // from affecting other targets in the combo. structuredClone avoids the
@@ -2244,15 +2300,11 @@ export async function handleComboChat({
           latencyMs,
           fallbackCount,
         });
-        return new Response(
-          JSON.stringify({
-            error: {
-              message: "Service temporarily unavailable: all upstream accounts are inactive",
-              type: "service_unavailable",
-              code: "ALL_ACCOUNTS_INACTIVE",
-            },
-          }),
-          { status: 503, headers: { "Content-Type": "application/json" } }
+        return errorResponseWithComboDiagnostics(
+          503,
+          "Service temporarily unavailable: all upstream accounts are inactive",
+          buildComboDiag("all_accounts_inactive"),
+          { code: "ALL_ACCOUNTS_INACTIVE", type: "service_unavailable" }
         );
       }
 
@@ -2303,10 +2355,11 @@ export async function handleComboChat({
       }
 
       log.warn("COMBO", `All models failed | ${msg}`);
-      return new Response(JSON.stringify({ error: { message: msg } }), {
+      return errorResponseWithComboDiagnostics(
         status,
-        headers: { "Content-Type": "application/json" },
-      });
+        msg,
+        buildComboDiag(lastError ?? "all_models_failed")
+      );
     }
 
     return errorResponse(503, "Combo routing completed without an upstream response");

open-sse/utils/error.ts

@@ -120,6 +120,88 @@ export function buildErrorBody(
   return body;
 }
 
+/**
+ * Sanitized auto-combo diagnostic trace surfaced on a combo terminal failure.
+ * Contains ONLY provider/model ids, enumerated reason codes, and counts — never
+ * keys, tokens, cookies, credentials, or upstream bodies. Fields are length- and
+ * count-capped so the projection is safe to place in HTTP headers too. (QA P0:
+ * "Add a sanitized combo diagnostic trace … candidate pool count, excluded
+ * provider/model reasons, selected attempt order, terminal failure summary.")
+ */
+export interface ComboExclusion {
+  provider: string;
+  model?: string;
+  reason: string;
+}
+export interface ComboDiagnostics {
+  poolSize: number;
+  attempted: number;
+  excluded: ComboExclusion[];
+  attemptOrder: Array<{ provider: string; model: string }>;
+  terminalReason: string;
+}
+
+function clampDiagStr(v: unknown, max = 128): string {
+  return typeof v === "string" ? v.slice(0, max).replace(/[\r\n]+/g, " ") : "";
+}
+
+/**
+ * Whitelist projection — guarantees only id/reason string primitives + integer
+ * counts can escape, regardless of what the caller assembled. This is the secret
+ * containment boundary for the diagnostic trace.
+ */
+export function sanitizeComboDiagnostics(d: ComboDiagnostics): ComboDiagnostics {
+  return {
+    poolSize: Number.isFinite(d?.poolSize) ? d.poolSize : 0,
+    attempted: Number.isFinite(d?.attempted) ? d.attempted : 0,
+    excluded: (d?.excluded ?? []).slice(0, 64).map((e) => ({
+      provider: clampDiagStr(e?.provider, 64),
+      ...(e?.model ? { model: clampDiagStr(e.model, 96) } : {}),
+      reason: clampDiagStr(e?.reason, 64),
+    })),
+    attemptOrder: (d?.attemptOrder ?? [])
+      .slice(0, 64)
+      .map((a) => ({ provider: clampDiagStr(a?.provider, 64), model: clampDiagStr(a?.model, 96) })),
+    terminalReason: clampDiagStr(d?.terminalReason, 200),
+  };
+}
+
+/**
+ * errorResponse variant that attaches a sanitized combo diagnostic trace as BOTH
+ * `x-omniroute-combo-*` headers and a `diagnostics` field in the OpenAI-shaped
+ * error body (extra field — backward-compatible with standard error parsers).
+ * `opts.code`/`opts.type` override the status-derived defaults (e.g. to preserve
+ * the `ALL_ACCOUNTS_INACTIVE` code on the 503 terminal path).
+ */
+export function errorResponseWithComboDiagnostics(
+  statusCode: number,
+  message: string,
+  diagnostics: ComboDiagnostics,
+  opts: { code?: string; type?: string } = {}
+): Response {
+  const safe = sanitizeComboDiagnostics(diagnostics);
+  const body = buildErrorBody(statusCode, message) as ErrorResponseBody & {
+    diagnostics?: ComboDiagnostics;
+  };
+  if (opts.code) body.error.code = opts.code;
+  if (opts.type) body.error.type = opts.type;
+  body.diagnostics = safe;
+  const excludedHeader = safe.excluded
+    .map((e) => `${e.provider}${e.model ? `/${e.model}` : ""}:${e.reason}`)
+    .join(",")
+    .slice(0, 900);
+  return new Response(JSON.stringify(body), {
+    status: statusCode,
+    headers: {
+      "Content-Type": "application/json",
+      "x-omniroute-combo-pool-size": String(safe.poolSize),
+      "x-omniroute-combo-attempted": String(safe.attempted),
+      "x-omniroute-combo-excluded": excludedHeader,
+      "x-omniroute-combo-terminal-reason": safe.terminalReason.slice(0, 200),
+    },
+  });
+}
+
 /**
  * Create error Response object (for non-streaming)
  * @param {number} statusCode - HTTP status code

tests/unit/combo-diagnostics-trace.test.ts

@@ -0,0 +1,81 @@
+/**
+ * QA P0 — sanitized auto-combo diagnostic trace.
+ * Guards the new `errorResponseWithComboDiagnostics` / `sanitizeComboDiagnostics`
+ * helpers: they must surface pool size + attempt order + exclusion reasons as
+ * both `x-omniroute-combo-*` headers and a `diagnostics` body field, while the
+ * sanitizer is the secret-containment boundary (only provider/model/reason ids +
+ * counts may ever escape — never keys/tokens/credentials).
+ */
+import test from "node:test";
+import assert from "node:assert/strict";
+
+const { errorResponseWithComboDiagnostics, sanitizeComboDiagnostics } = await import(
+  "../../open-sse/utils/error.ts"
+);
+
+test("combo diagnostics: headers + body carry the sanitized trace (code override preserved)", async () => {
+  const res = errorResponseWithComboDiagnostics(
+    503,
+    "all upstream accounts inactive",
+    {
+      poolSize: 3,
+      attempted: 2,
+      excluded: [{ provider: "openai", model: "gpt-x", reason: "exhausted" }],
+      attemptOrder: [{ provider: "openai", model: "gpt-x" }],
+      terminalReason: "all_accounts_inactive",
+    },
+    { code: "ALL_ACCOUNTS_INACTIVE", type: "service_unavailable" }
+  );
+
+  assert.equal(res.status, 503);
+  assert.equal(res.headers.get("x-omniroute-combo-pool-size"), "3");
+  assert.equal(res.headers.get("x-omniroute-combo-attempted"), "2");
+  assert.match(res.headers.get("x-omniroute-combo-excluded") || "", /openai\/gpt-x:exhausted/);
+  assert.equal(res.headers.get("x-omniroute-combo-terminal-reason"), "all_accounts_inactive");
+
+  const body = await res.json();
+  assert.equal(body.error.code, "ALL_ACCOUNTS_INACTIVE");
+  assert.equal(body.error.type, "service_unavailable");
+  assert.ok(body.diagnostics, "diagnostics field present in body");
+  assert.equal(body.diagnostics.poolSize, 3);
+  assert.equal(body.diagnostics.attempted, 2);
+  assert.equal(body.diagnostics.terminalReason, "all_accounts_inactive");
+  assert.equal(body.diagnostics.attemptOrder[0].provider, "openai");
+});
+
+test("combo diagnostics: sanitizer caps sizes + keeps only the whitelist keys", () => {
+  const dirty = {
+    poolSize: 1,
+    attempted: 1,
+    excluded: Array.from({ length: 200 }, (_, i) => ({
+      provider: "p" + i,
+      reason: "r".repeat(500),
+    })),
+    attemptOrder: Array.from({ length: 200 }, () => ({ provider: "p", model: "m" })),
+    terminalReason: "x".repeat(1000),
+  };
+  const safe = sanitizeComboDiagnostics(dirty as never);
+  assert.ok(safe.excluded.length <= 64, "excluded capped at 64");
+  assert.ok(safe.attemptOrder.length <= 64, "attemptOrder capped at 64");
+  assert.ok(safe.excluded[0].reason.length <= 64, "reason length clamped");
+  assert.ok(safe.terminalReason.length <= 200, "terminalReason length clamped");
+  assert.deepEqual(Object.keys(safe.excluded[0]).sort(), ["provider", "reason"]);
+});
+
+test("combo diagnostics: secret containment — non-whitelisted fields never survive", () => {
+  const leaky = {
+    poolSize: 1,
+    attempted: 1,
+    excluded: [
+      { provider: "openai", reason: "exhausted", apiKey: "sk-SECRET-KEY", token: "SECRET-TOK" },
+    ],
+    attemptOrder: [{ provider: "openai", model: "m", accessToken: "SECRET-OAUTH" }],
+    terminalReason: "t",
+  };
+  const safe = sanitizeComboDiagnostics(leaky as never);
+  const serialized = JSON.stringify(safe);
+  assert.ok(!serialized.includes("SECRET"), "no secret VALUES survive the projection");
+  assert.ok(!serialized.includes("apiKey"), "no apiKey KEY survives");
+  assert.ok(!serialized.includes("accessToken"), "no accessToken KEY survives");
+  assert.ok(!serialized.includes("token"), "no token KEY survives");
+});