Files
OmniRoute/tests/unit/combo-test-health.test.ts
Jan Leon ea650253af Stream model health probes for slow providers (#7377)
* fix(model-test): stream slow chat probes

* fix(model-test): handle JSON responses for streaming probes

* fix(model-test): preserve transient errors from streaming probes

* fix(model-test): keep transient probe failures visible

* chore(ci): rerun pull request checks

* docs(model-test): clarify transient failure handling

* chore(ci): rerun pull request checks

* test(model-test): cover slow timeout response path

* chore(quality): register base-branch mutation tests

* fix(sse): stop real-network leak and DOMException crash in model-test-runner timeout path

Two new tests added by this PR fail against the current release tip:

- tests/unit/model-test-runner.test.ts's slow-timeout regression test races
  the cold-start cost of the chat-completions pipeline (SSE translators,
  compression settings, etc. all lazily init on the first real request in a
  process). With a 1s AbortController timeout, the abort can fire before
  chatCore ever reaches the executor's fetch() call; the mocked fetch is then
  invoked after the test's own `finally` block has already restored the real
  fetch, so the assertion on the mock never fires and the request leaks onto
  the real network. Warm up the pipeline with one fast, resolving mock call
  before timing the 1s scenario.

- Once the warm-up unblocks that race, a second, real bug surfaces:
  withRateLimit's abort handling (open-sse/services/rateLimitManager.ts)
  mutates `reason.name = "AbortError"` in place. When `AbortController.abort()`
  is called with no explicit reason (as modelTestRunner's timeout path does),
  the default reason is a native DOMException, whose `name` is a read-only
  getter — the mutation throws `TypeError: Cannot set property name of
  [object DOMException] which has only a getter` instead of rejecting
  cleanly. Build a fresh Error instead of mutating the caller-supplied
  reason, preserving the original as `.cause`.

Adds a focused regression test in tests/unit/rate-limit-manager.test.ts that
reproduces the DOMException crash directly against withRateLimit.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
2026-07-18 21:18:42 -03:00

160 lines
4.0 KiB
TypeScript

import test from "node:test";
import assert from "node:assert/strict";
const {
buildComboTestRequestBody,
extractComboTestResponseText,
extractComboTestStreamResult,
extractComboTestStreamText,
} = await import("../../src/lib/combos/testHealth.ts");
test("combo test helper builds a realistic smoke payload", () => {
const originalRandom = Math.random;
let callCount = 0;
let body;
try {
Math.random = () => {
callCount += 1;
return callCount === 1 ? 0.4680222223 : 0.2677;
};
body = buildComboTestRequestBody("openrouter/openai/gpt-5.4");
} finally {
Math.random = originalRandom;
}
assert.equal(body.model, "openrouter/openai/gpt-5.4");
assert.equal(body.messages[0].content, "Calculate 52122+34093, and reply with the result only.");
assert.equal(body.max_tokens, 2048);
assert.equal("temperature" in body, false);
assert.equal(body.stream, false);
});
test("combo test helper builds a small streaming model probe", () => {
const body = buildComboTestRequestBody("nvidia/z-ai/glm-5.2", false, { stream: true });
assert.equal(body.stream, true);
assert.equal(body.max_tokens, 64);
});
test("combo test helper ignores keepalives and extracts streamed model content", () => {
const text = extractComboTestStreamText(
': omniroute-keepalive\n\ndata: {"choices":[{"delta":{"content":"O"}}]}\n\n' +
'data: {"choices":[{"delta":{"content":"K"}}]}\n\ndata: [DONE]\n\n'
);
assert.equal(text, "OK");
});
test("combo test helper preserves streamed upstream errors", () => {
const result = extractComboTestStreamResult(
'data: {"error":{"message":"Rate limit exceeded","code":"429"}}\n\n'
);
assert.deepEqual(result, {
text: "",
error: { message: "Rate limit exceeded", statusCode: 429 },
});
});
test("combo test helper extracts text from chat-completions responses", () => {
const text = extractComboTestResponseText({
choices: [
{
message: {
role: "assistant",
content: "OK",
},
},
],
});
assert.equal(text, "OK");
});
test("combo test helper extracts text from block-based responses", () => {
const text = extractComboTestResponseText({
choices: [
{
message: {
role: "assistant",
content: [
{ type: "text", text: "OK" },
{ type: "output_text", text: "Confirmed." },
],
},
},
],
});
assert.equal(text, "OK\nConfirmed.");
});
test("combo test helper extracts reasoning content when visible text is absent", () => {
const text = extractComboTestResponseText({
choices: [
{
message: {
role: "assistant",
content: null,
reasoning_content: "Working through the request.\nOK",
},
},
],
});
assert.equal(text, "Working through the request.\nOK");
});
test("combo test helper extracts reasoning_text aliases from GitHub-style responses", () => {
const text = extractComboTestResponseText({
choices: [
{
message: {
role: "assistant",
content: "",
reasoning_text: "Reasoning trace",
},
},
],
});
assert.equal(text, "Reasoning trace");
});
test("combo test helper treats reasoning-only completions as a healthy signal", () => {
const text = extractComboTestResponseText({
choices: [
{
finish_reason: "length",
message: {
role: "assistant",
content: "",
},
},
],
usage: {
prompt_tokens: 6,
completion_tokens: 12,
total_tokens: 18,
completion_tokens_details: {
reasoning_tokens: 12,
},
},
});
assert.equal(text, "[reasoning-only completion]");
});
test("combo test helper returns empty string when no text content exists", () => {
const text = extractComboTestResponseText({
choices: [
{
message: {
role: "assistant",
content: [{ type: "tool_call", id: "call_1" }],
},
},
],
});
assert.equal(text, "");
});