From 706dc75c13906d83e1932e05e40c0c65b98ae41e Mon Sep 17 00:00:00 2001 From: Beexly Date: Fri, 18 Sep 2026 11:13:06 -0500 Subject: [PATCH] fix(chatCore): stop executeWithUpstreamStartTimeout leaking its abortPromise listener (hedge-cancelled process exit) (#12406) * fix(sse): stop mergeAbortSignals from leaking abort listeners mergeAbortSignals() attached "abort" listeners to its primary/secondary signals but never removed them once the merged signal settled. Every executor fetch attempt calls this (fetchWithStartTimeout, once per URL/retry), so a busy combo request accumulated one live listener per call on the long-lived combo/client signal. A leaked listener still fires when that signal is later aborted (e.g. a hedge cancellation arriving after this merge's own caller already finished), for a merged output nothing is watching anymore. Mirrors the already-correct self-cleaning pattern in open-sse/utils/directResponseStartTimeout.ts's local mergeAbortSignals. Regression test measures listener growth across repeated merges of the same long-lived signal: 25 merges leaked exactly 25 listeners pre-fix, 0 post-fix. (cherry picked from commit 07969655147d3236969b38bfb41280ab4fb52b79) * fix(server): stop the crash guard re-throwing combo abort reasons Production crash 2026-08-31 (omniroute.log): on a client disconnect, handleDisconnect aborted the combo controller and a late abort listener threw the abort reason on an empty stack: Error [AbortError]: hedge-cancelled at ... AbortController.abort ... handleDisconnect file:///.../src/shared/utils/httpClientAbortGuard.mjs:130 throw err; isClientAbortError() only knew Node's stream codes and "aborted", so shouldSwallowUncaught() said false and the guard re-threw, taking the whole server down. - Port upstream's AbortError line (name "AbortError" + abort-flavoured message) so request_signal_aborted / DOMException aborts are absorbed. - Add an exact-message match for the combo abort reasons from open-sse/services/combo/comboAbortReasons.ts ("hedge-cancelled", "combo-per-model-timeout"), name-agnostic because the raw reason is a plain Error that only gets name="AbortError" stamped on the way out. A losing hedge / stalled target is never a server fault. Inlined so this .mjs stays dependency-free for scripts/dev/run-next.mjs. Tests: port upstream's guard tests, add the exact crash shape, a child-process replay of the crash (dies pre-fix, survives post-fix), a genuine-error case that must still crash, and a sync check against comboAbortReasons.ts. The child-process helper passes a file:// URL, not a bare path, so the tests run on Windows. Co-Authored-By: Claude Fable 5.1 (cherry picked from commit 90c9bce8c474b60c37cc63f4421d50feae4c0ad2) * fix(chatCore): stop executeWithUpstreamStartTimeout leaking its abortPromise listener Root cause of the 2026-08-31 production exit (Error [AbortError]: hedge-cancelled), verified by mapping the crash frames in .build/next/server/chunks/13721.js back to this file: - The abortPromise abort listener registered on the long-lived client / stream signal was never removed in the finally block (only abortListener and timeoutAbortListener were), so every executor attempt (and every retry) leaked one listener onto that signal. - Promise.race only subscribes to abortPromise/timeoutPromise once the array literal has been evaluated. When execute() threw synchronously the race never ran, abortPromise was orphaned, and the next hedge cancellation / client disconnect aborted the signal with the string reason streamHandler.ts forwards; createAbortError() rebuilt it as an AbortError-named Error and rejected a promise nothing awaited. That unhandledRejection reached the process crash guard, which re-threw it as an uncaughtException and exited with code 7. Keep a handle to the listener and remove it with the others, and mark the two race-loser promises as handled so a synchronous throw from execute() can never orphan them. Race semantics are unchanged (the race still observes their rejections). Regression tests: (1) a resolving execute leaves the listener count on the client signal unchanged; (2) a synchronously throwing execute leaks no listener and a later abort with the string "hedge-cancelled" produces no unhandledRejection. Both fail against the previous implementation. Note: commit 079696551 (mergeAbortSignals cleanup) is correct listener hygiene but is not on this crash path; this is the fix for the incident. Co-Authored-By: Claude Fable 5.1 (cherry picked from commit e68a50ad854a1945c23e747da4ef15a820cc148d) * fix(server): absorb raw string abort reasons in the crash guard; document the verified crash path Follow-ups from the adversarial review of 90c9bce8c: - open-sse/utils/streamHandler.ts aborts the stream controller with a raw string reason (getClientAbortReason / handleDisconnect) and undici rejects with signal.reason verbatim, so a cancellation can reach process level as a bare string. isClientAbortError() returned false for every non-object, which would still have exited the process. Absorb the combo abort reasons and the stream-handler disconnect reasons when they arrive as strings. - Correct the mechanism comment: the 2026-08-31 exit was a leaked upstreamTimeouts.ts abortPromise listener rejecting a promise nothing awaited (unhandledRejection), escalated by this guard, not a listener throwing synchronously. The leak is fixed at the source in the previous commit; this guard remains the last-resort net. - Reword the inlining rationale (plain node launcher, no reliance on type-stripping for the .ts constants module). - Tests: the child-process replay now also exercises the unhandledRejection route with the exact production error shape and with raw string reasons; add unit coverage for string reasons and non-object look-alikes. Co-Authored-By: Claude Fable 5.1 (cherry picked from commit 696fcc8fe1b9b9bc43e6e5f5f5e44b619a86e68a) --------- Co-authored-by: Claude Fable 5.1 Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> Co-authored-by: Beexly --- ...6-upstream-timeouts-abort-listener-leak.md | 1 + open-sse/executors/base.ts | 25 +-- open-sse/executors/base/mergeAbortSignals.ts | 47 +++++ .../handlers/chatCore/upstreamTimeouts.ts | 22 ++- src/shared/utils/httpClientAbortGuard.mjs | 36 ++++ tests/unit/chatcore-upstream-timeouts.test.ts | 65 +++++++ tests/unit/executor-base-utils.test.ts | 38 ++++ tests/unit/httpClientAbortGuard.test.mjs | 164 ++++++++++++++---- 8 files changed, 332 insertions(+), 66 deletions(-) create mode 100644 changelog.d/fixes/12406-upstream-timeouts-abort-listener-leak.md create mode 100644 open-sse/executors/base/mergeAbortSignals.ts diff --git a/changelog.d/fixes/12406-upstream-timeouts-abort-listener-leak.md b/changelog.d/fixes/12406-upstream-timeouts-abort-listener-leak.md new file mode 100644 index 0000000000..16e246bf60 --- /dev/null +++ b/changelog.d/fixes/12406-upstream-timeouts-abort-listener-leak.md @@ -0,0 +1 @@ +- **fix(chatCore):** stop `executeWithUpstreamStartTimeout` leaking its abortPromise listener onto the long-lived client/stream signal, and stop `mergeAbortSignals` leaking per-attempt abort listeners, so a later hedge cancellation or client disconnect cannot reject an orphaned promise and take the process down (`Error [AbortError]: hedge-cancelled`). The crash guard also absorbs combo abort reasons (`hedge-cancelled`, `combo-per-model-timeout`) and raw string disconnect reasons as a last-resort net ([#12406](https://github.com/diegosouzapw/OmniRoute/pull/12406) — thanks @Beexly) diff --git a/open-sse/executors/base.ts b/open-sse/executors/base.ts index 19c8bc9fd9..dc65898295 100644 --- a/open-sse/executors/base.ts +++ b/open-sse/executors/base.ts @@ -130,6 +130,8 @@ import { sanitizeReasoningEffortForProvider } from "./base/reasoningEffort.ts"; // Reasoning-effort sanitation extracted to a pure leaf; re-exported for external // importers (mimoThinking service + tests) that import it from "./base.ts". export { sanitizeReasoningEffortForProvider } from "./base/reasoningEffort.ts"; +import { mergeAbortSignals } from "./base/mergeAbortSignals.ts"; +export { mergeAbortSignals } from "./base/mergeAbortSignals.ts"; /** * Sanitizes a custom API path to prevent path traversal attacks. @@ -229,29 +231,6 @@ export type CountTokensInput = { signal?: AbortSignal | null; }; -export function mergeAbortSignals(primary: AbortSignal, secondary: AbortSignal): AbortSignal { - const controller = new AbortController(); - - const abortFrom = (source: AbortSignal) => { - if (!controller.signal.aborted) { - controller.abort(source.reason); - } - }; - - if (primary.aborted) { - abortFrom(primary); - return controller.signal; - } - if (secondary.aborted) { - abortFrom(secondary); - return controller.signal; - } - - primary.addEventListener("abort", () => abortFrom(primary), { once: true }); - secondary.addEventListener("abort", () => abortFrom(secondary), { once: true }); - return controller.signal; -} - import { hasActiveClaudeThinking, readNestedThinkingBudget, diff --git a/open-sse/executors/base/mergeAbortSignals.ts b/open-sse/executors/base/mergeAbortSignals.ts new file mode 100644 index 0000000000..a6581c7129 --- /dev/null +++ b/open-sse/executors/base/mergeAbortSignals.ts @@ -0,0 +1,47 @@ +// Self-cleaning abort-signal merge. Extracted from base.ts so the frozen +// executor host does not grow past its file-size cap for this leak fix. + +/** + * Merge two abort signals into one that fires when either does. + * + * The `primary`/`secondary` abort listeners registered below MUST be removed once the + * merged controller settles — otherwise they outlive this call and stay attached to + * whichever input signal is longer-lived (typically the combo/client signal, which + * stays open for the whole request while this merge is re-done per executor fetch + * attempt/retry). A hedge cancellation or client disconnect arriving after this + * particular merge's caller has already moved on then still fires the listener, + * detached from anything that's still awaiting it. Mirrors the already-correct + * self-cleaning pattern in `open-sse/utils/directResponseStartTimeout.ts`'s local + * `mergeAbortSignals` (#hedge-cancelled-abort-listener-leak). + */ +export function mergeAbortSignals(primary: AbortSignal, secondary: AbortSignal): AbortSignal { + const controller = new AbortController(); + + const cleanup = () => { + primary.removeEventListener("abort", onPrimaryAbort); + secondary.removeEventListener("abort", onSecondaryAbort); + }; + + const abortFrom = (source: AbortSignal) => { + if (!controller.signal.aborted) { + controller.abort(source.reason); + } + cleanup(); + }; + + const onPrimaryAbort = () => abortFrom(primary); + const onSecondaryAbort = () => abortFrom(secondary); + + if (primary.aborted) { + abortFrom(primary); + return controller.signal; + } + if (secondary.aborted) { + abortFrom(secondary); + return controller.signal; + } + + primary.addEventListener("abort", onPrimaryAbort, { once: true }); + secondary.addEventListener("abort", onSecondaryAbort, { once: true }); + return controller.signal; +} diff --git a/open-sse/handlers/chatCore/upstreamTimeouts.ts b/open-sse/handlers/chatCore/upstreamTimeouts.ts index b1a8da548d..72ee7849ef 100644 --- a/open-sse/handlers/chatCore/upstreamTimeouts.ts +++ b/open-sse/handlers/chatCore/upstreamTimeouts.ts @@ -123,10 +123,7 @@ export function getExecutorTimeoutMs( // Defensive backstop for direct callers: resolveConnectionTimeoutMs is the // gate (it rejects out-of-range values so the chain falls through); this // clamp only caps values a future caller could pass unvetted. - return Math.min( - Math.max(0, Math.floor(connectionTimeoutMs)), - MAX_PROVIDER_SPECIFIC_TIMEOUT_MS - ); + return Math.min(Math.max(0, Math.floor(connectionTimeoutMs)), MAX_PROVIDER_SPECIFIC_TIMEOUT_MS); } const modelOverride = resolveModelTimeoutOverride(provider, model); if (modelOverride !== undefined) return modelOverride; @@ -271,15 +268,30 @@ export async function executeWithUpstreamStartTimeout({ }, timeoutMs); }); + let abortPromiseListener: (() => void) | null = null; const abortPromise = new Promise((_, reject) => { - signal.addEventListener("abort", () => reject(createAbortError(signal)), { once: true }); + abortPromiseListener = () => reject(createAbortError(signal)); + signal.addEventListener("abort", abortPromiseListener, { once: true }); }); + // Promise.race only subscribes to timeoutPromise/abortPromise once the array + // literal below has been evaluated. If execute() throws synchronously the race + // never runs, both promises are orphaned, and a later abort of the long-lived + // client signal surfaces as an unhandledRejection. That was the 2026-08-31 + // production exit: a hedge sibling won after a client disconnect, the leaked + // listener below rebuilt the string reason as an AbortError, and nothing was + // awaiting the promise it rejected. Marking them handled keeps the race + // semantics (it still observes the rejections) while closing that path. + abortPromise.catch(() => {}); + timeoutPromise.catch(() => {}); try { return await Promise.race([execute(combinedController.signal), timeoutPromise, abortPromise]); } finally { if (timeoutId) clearTimeout(timeoutId); if (abortListener) signal.removeEventListener("abort", abortListener); + // Never removed before this fix: one listener leaked onto the client signal + // per call (chatCore.ts invokes this once per executor attempt, plus retries). + if (abortPromiseListener) signal.removeEventListener("abort", abortPromiseListener); if (timeoutAbortListener) { timeoutController.signal.removeEventListener("abort", timeoutAbortListener); } diff --git a/src/shared/utils/httpClientAbortGuard.mjs b/src/shared/utils/httpClientAbortGuard.mjs index 9465ff3208..206146f6a8 100644 --- a/src/shared/utils/httpClientAbortGuard.mjs +++ b/src/shared/utils/httpClientAbortGuard.mjs @@ -48,12 +48,31 @@ * @module */ +/** + * Abort reasons raised by combo target dispatch. Mirror of + * open-sse/services/combo/comboAbortReasons.ts — keep the two in sync. + */ +const COMBO_ABORT_REASONS = new Set(["hedge-cancelled", "combo-per-model-timeout"]); + +/** + * Raw string reasons open-sse/utils/streamHandler.ts passes to + * handleDisconnect() / abortController.abort() when the client goes away. + */ +const CLIENT_DISCONNECT_REASONS = new Set(["request_signal_aborted", "client_closed", "cancelled"]); + /** * @param {unknown} err * @returns {boolean} true when `err` represents a client closing the * connection rather than a server-side fault. */ export function isClientAbortError(err) { + // open-sse/utils/streamHandler.ts aborts the stream controller with a RAW + // STRING reason (getClientAbortReason / handleDisconnect), and undici rejects + // with signal.reason verbatim, so a cancellation can surface at the process + // level as a bare string rather than an Error object. + if (typeof err === "string") { + return COMBO_ABORT_REASONS.has(err) || CLIENT_DISCONNECT_REASONS.has(err); + } if (!err || typeof err !== "object") return false; const e = /** @type {NodeJS.ErrnoException} */ (err); // Node emits `Error: aborted` (no code) from http.Server#abortIncoming. @@ -65,6 +84,23 @@ export function isClientAbortError(err) { // `Error: aborted` — an emitter-left 'error' event on any of these used to // kill the process (#fix-dev-server-aborted). if (e.name === "AbortError" && /abort/i.test(String(e.message))) return true; + // Combo dispatch cancels a losing hedged target / a stalled target by + // aborting with `new Error(reason)` for one of the reasons in + // open-sse/services/combo/comboAbortReasons.ts (targetTimeoutRunner.ts). + // streamHandler.ts forwards that reason to the stream controller as a plain + // string, and a leaked abort listener in + // open-sse/handlers/chatCore/upstreamTimeouts.ts (executeWithUpstreamStartTimeout) + // rebuilt it via createAbortError() as an AbortError-named Error to reject a + // promise nothing was awaiting. That unhandledRejection reached this guard, + // which re-threw it as an uncaughtException. Production exit 2026-08-31: + // Error [AbortError]: hedge-cancelled + // The listener leak is fixed at the source; this stays as the last-resort net. + // A sibling target winning / a target stalling is never a server fault, so + // match the exact reason text whatever `name` the thrower stamped on it. + // Inlined rather than imported from comboAbortReasons.ts: this file runs + // under plain node (scripts/dev/run-next.mjs, no tsx) and must not depend on + // type-stripping being available for that .ts module. + if (COMBO_ABORT_REASONS.has(String(e.message))) return true; switch (e.code) { case "ERR_STREAM_PREMATURE_CLOSE": case "ECONNRESET": diff --git a/tests/unit/chatcore-upstream-timeouts.test.ts b/tests/unit/chatcore-upstream-timeouts.test.ts index affad73611..5525824209 100644 --- a/tests/unit/chatcore-upstream-timeouts.test.ts +++ b/tests/unit/chatcore-upstream-timeouts.test.ts @@ -1,10 +1,12 @@ import test from "node:test"; import assert from "node:assert/strict"; +import { getEventListeners } from "node:events"; import { createBodyTimeoutError, createUpstreamStartTimeoutError, createAbortError, + executeWithUpstreamStartTimeout, computeBillableTokens, getExecutorTimeoutMs, normalizeExecutorResult, @@ -57,3 +59,66 @@ test("normalizeExecutorResult rejects malformed executor output", () => { /must contain a Response/ ); }); +test("executeWithUpstreamStartTimeout leaves no abort listener on the client signal after a resolving execute", async () => { + const client = new AbortController(); + const before = getEventListeners(client.signal, "abort").length; + const result = await executeWithUpstreamStartTimeout({ + executor: {}, + provider: "test-provider", + model: "test-model", + connectionTimeoutMs: 5_000, + signal: client.signal, + execute: async () => "ok", + }); + assert.equal(result, "ok"); + assert.equal( + getEventListeners(client.signal, "abort").length, + before, + "every listener registered for the race must be removed once it settles" + ); +}); + +test("executeWithUpstreamStartTimeout: a synchronously throwing execute cannot orphan abortPromise (2026-08-31 hedge-cancelled exit)", async () => { + // Production shape: execute() threw before Promise.race subscribed, so the + // abortPromise listener stayed on the long-lived client signal with nobody + // awaiting its rejection. The next hedge cancellation / client disconnect + // then aborted that signal with a string reason and the rejection became an + // unhandledRejection: `Error [AbortError]: hedge-cancelled`. + const client = new AbortController(); + const before = getEventListeners(client.signal, "abort").length; + await assert.rejects( + executeWithUpstreamStartTimeout({ + executor: {}, + provider: "test-provider", + model: "test-model", + connectionTimeoutMs: 5_000, + signal: client.signal, + execute: () => { + throw new Error("sync failure before the race subscribed"); + }, + }), + /sync failure before the race subscribed/ + ); + assert.equal( + getEventListeners(client.signal, "abort").length, + before, + "no listener may leak onto the client signal when execute() throws synchronously" + ); + + let unhandled: unknown = null; + const onUnhandled = (reason: unknown) => { + unhandled = reason; + }; + process.on("unhandledRejection", onUnhandled); + try { + client.abort("hedge-cancelled"); + await new Promise((resolve) => setTimeout(resolve, 25)); + } finally { + process.off("unhandledRejection", onUnhandled); + } + assert.equal( + unhandled, + null, + "a late abort must not surface as an unhandledRejection: " + String(unhandled) + ); +}); diff --git a/tests/unit/executor-base-utils.test.ts b/tests/unit/executor-base-utils.test.ts index fd86922898..ee5c50a78e 100644 --- a/tests/unit/executor-base-utils.test.ts +++ b/tests/unit/executor-base-utils.test.ts @@ -1,5 +1,7 @@ import test from "node:test"; import assert from "node:assert/strict"; +import { getEventListeners } from "node:events"; +import type { EventEmitter } from "node:events"; const base = await import("../../open-sse/executors/base.ts"); @@ -116,6 +118,42 @@ test("mergeAbortSignals aborts when secondary fires", () => { assert.ok(merged.aborted); }); +// Regression: mergeAbortSignals used to attach its "abort" listeners to `primary`/ +// `secondary` and never remove them. Every executor fetch attempt calls this (once +// per URL/retry — open-sse/executors/base.ts's fetchWithStartTimeout), so a busy +// combo request accumulated one live listener per call on the long-lived combo/ +// client signal. Each leaked listener still fires when that signal is aborted later +// (e.g. a hedge cancellation — COMBO_HEDGE_CANCELLED_REASON, "hedge-cancelled" — +// arriving after this particular merge's own caller already finished), for a +// `merged` output nothing is watching anymore. Proven here by counting real +// listener growth (via node:events' getEventListeners) across repeated merges of +// the SAME long-lived primary signal — exactly the fetchWithStartTimeout pattern. +test("mergeAbortSignals removes its listeners once the merged signal settles (no leak across repeated merges)", () => { + const countAbortListeners = (target: AbortSignal) => + getEventListeners(target as unknown as EventEmitter, "abort").length; + + const longLived = new AbortController(); + const before = countAbortListeners(longLived.signal); + + for (let i = 0; i < 25; i++) { + const perAttempt = new AbortController(); + const merged = base.mergeAbortSignals(longLived.signal, perAttempt.signal); + assert.ok(!merged.aborted); + // Simulate this attempt finishing on its own (its own timeout/abort fires), + // the way each fetchWithStartTimeout call's timeoutController does. + perAttempt.abort(new Error(`attempt-${i}-timeout`)); + assert.ok(merged.aborted); + } + + const after = countAbortListeners(longLived.signal); + assert.ok( + after <= before + 1, + `expected mergeAbortSignals to clean up its "abort" listener on the long-lived ` + + `signal after each merge settles; listener count grew from ~${before} to ${after} ` + + `across 25 merges (leak)` + ); +}); + test("sanitizeReasoningEffortForProvider passes through body without reasoning_effort", () => { const body = { model: "gpt-4o", temperature: 0.7 }; const result = base.sanitizeReasoningEffortForProvider(body, "openai", "gpt-4o"); diff --git a/tests/unit/httpClientAbortGuard.test.mjs b/tests/unit/httpClientAbortGuard.test.mjs index 9345ccef22..2c595ce089 100644 --- a/tests/unit/httpClientAbortGuard.test.mjs +++ b/tests/unit/httpClientAbortGuard.test.mjs @@ -103,7 +103,10 @@ test("shouldSwallowUncaught absorbs the real 'aborted' uncaughtException signatu assert.equal(shouldSwallowUncaught(abortErr, "uncaughtException"), true); assert.equal(shouldSwallowUncaught(abortErr, undefined), true); assert.equal( - shouldSwallowUncaught(Object.assign(new Error("ECONNRESET"), { code: "ECONNRESET" }), "uncaughtException"), + shouldSwallowUncaught( + Object.assign(new Error("ECONNRESET"), { code: "ECONNRESET" }), + "uncaughtException" + ), true ); }); @@ -117,6 +120,7 @@ test("shouldSwallowUncaught preserves crash semantics for genuine errors", () => test("installProcessCrashGuard does not throw on import and is idempotent", () => { assert.doesNotThrow(() => installProcessCrashGuard(() => {})); + assert.doesNotThrow(() => installProcessCrashGuard(() => {})); }); test("isClientAbortError matches OmniRoute SSE AbortError shapes (#fix-crash-guard-logger-7)", () => { // Exact production shape from the 2026-08-31 crash log: @@ -136,18 +140,82 @@ test("shouldSwallowUncaught absorbs SSE AbortError rejections", () => { assert.equal(shouldSwallowUncaught(sseAbort, "unhandledRejection"), true); }); -// Production crash (2026-08-25 → 08-31, ~170 restarts, exit code 7): +test("isClientAbortError absorbs combo abort reasons (exact 2026-08-31 hedge-cancelled crash shape)", () => { + // omniroute.log, 2026-08-31T23:47Z: the process died at this guard's own + // uncaughtException re-throw with + // Error [AbortError]: hedge-cancelled + // at ... AbortController.abort ... handleDisconnect + // targetTimeoutRunner.ts aborts with new Error("hedge-cancelled"); by the + // time it escapes, the thrower has stamped name = "AbortError" on it. + const relabelled = Object.assign(new Error("hedge-cancelled"), { name: "AbortError" }); + assert.equal(isClientAbortError(relabelled), true, "relabelled AbortError must be absorbed"); + const raw = new Error("hedge-cancelled"); + assert.equal(isClientAbortError(raw), true, "raw abort reason (name=Error) must be absorbed"); + const stalled = new Error("combo-per-model-timeout"); + assert.equal( + isClientAbortError(stalled), + true, + "per-model timeout abort reason must be absorbed" + ); + // Look-alikes that are genuine faults keep crash semantics (exact match only). + assert.equal(isClientAbortError(new Error("hedge-cancelled: unexpected state")), false); + assert.equal( + isClientAbortError(new TypeError("Cannot read properties of undefined (reading 'hedge')")), + false + ); + assert.equal( + isClientAbortError(Object.assign(new Error("disk full"), { code: "ENOSPC" })), + false + ); +}); + +test("combo abort reasons in the guard stay in sync with comboAbortReasons.ts", async (t) => { + let mod; + try { + mod = await import("../../open-sse/services/combo/comboAbortReasons.ts"); + } catch { + t.skip("TypeScript loader (tsx) not active in this run"); + return; + } + for (const reason of [mod.COMBO_HEDGE_CANCELLED_REASON, mod.COMBO_PER_MODEL_TIMEOUT_REASON]) { + assert.equal(typeof reason, "string"); + assert.equal(isClientAbortError(new Error(reason)), true, reason + " must be absorbed"); + } +}); + +test("shouldSwallowUncaught absorbs the hedge-cancelled uncaughtException (2026-08-31 crash)", () => { + const hedged = Object.assign(new Error("hedge-cancelled"), { name: "AbortError" }); + assert.equal(shouldSwallowUncaught(hedged, "uncaughtException"), true); + assert.equal(shouldSwallowUncaught(hedged, "unhandledRejection"), true); + assert.equal(shouldSwallowUncaught(hedged, undefined), true); +}); + +function runGuardChild(script) { + // Hand the child a file:// URL, not a filesystem path: dynamic import() of a + // bare Windows path ("C:...") fails with ERR_UNSUPPORTED_ESM_URL_SCHEME. + const guardUrl = new URL("../../src/shared/utils/httpClientAbortGuard.mjs", import.meta.url).href; + return new Promise((resolve, reject) => { + const child = spawn(process.execPath, ["--input-type=module", "-e", script, guardUrl], { + stdio: ["ignore", "pipe", "pipe"], + }); + let out = ""; + let err = ""; + child.stdout.on("data", (d) => (out += d)); + child.stderr.on("data", (d) => (err += d)); + child.on("close", (status) => resolve({ status, stdout: out, stderr: err })); + child.on("error", reject); + }); +} + +// Production crash (2026-08-25 -> 08-31, ~170 restarts, exit code 7): // every real call site installs the guard with NO logger, so the old -// `const logger = log ?? console` default invoked the console OBJECT as a -// function inside the uncaughtException handler → TypeError inside -// process._fatalException → Node exit code 7. These children run the REAL +// "const logger = log ?? console" default invoked the console OBJECT as a +// function inside the uncaughtException handler -> TypeError inside +// process._fatalException -> Node exit code 7. These children run the REAL // production call shape; the process must survive benign aborts and still // crash on genuine errors. test("installProcessCrashGuard() with no logger swallows aborts instead of dying (exit-7 regression)", async () => { - const guardPath = fileURLToPath( - new URL("../../src/shared/utils/httpClientAbortGuard.mjs", import.meta.url) - ); - const script = ` + const { status, stdout, stderr } = await runGuardChild(` const { installProcessCrashGuard } = await import(process.argv[1]); installProcessCrashGuard(); // production call sites pass NO logger process.emit( @@ -162,46 +230,66 @@ test("installProcessCrashGuard() with no logger swallows aborts instead of dying ); console.log("ALIVE"); process.exit(0); - `; - const { status, stdout, stderr } = await new Promise((resolve, reject) => { - const child = spawn(process.execPath, ["--input-type=module", "-e", script, guardPath], { - stdio: ["ignore", "pipe", "pipe"], - }); - let out = ""; - let err = ""; - child.stdout.on("data", (d) => (out += d)); - child.stderr.on("data", (d) => (err += d)); - child.on("close", (status) => resolve({ status, stdout: out, stderr: err })); - child.on("error", reject); - }); - assert.equal(status, 0, `child must survive benign aborts; stderr: ${stderr}`); + `); + assert.equal(status, 0, "child must survive benign aborts; stderr: " + stderr); + assert.match(stdout, /ALIVE/); +}); + +test("installProcessCrashGuard() survives the real hedge-cancelled uncaughtException", async () => { + // Replays the 2026-08-31 production crash through a real process: the abort + // reason escapes an abort listener as an uncaughtException with the exact + // name/message the log recorded. Pre-fix this exits non-zero at the re-throw. + const { status, stdout, stderr } = await runGuardChild(` + const { installProcessCrashGuard } = await import(process.argv[1]); + installProcessCrashGuard(); + const ctl = new AbortController(); + ctl.signal.addEventListener("abort", () => { + const err = ctl.signal.reason; + err.name = "AbortError"; + throw err; // escapes the listener -> uncaughtException + }, { once: true }); + process.once("exit", (code) => { if (code === 0) console.log("ALIVE"); }); + setTimeout(() => process.exit(0), 50); + ctl.abort(new Error("hedge-cancelled")); + // The verified production route: a leaked upstreamTimeouts abortPromise + // listener rejected a promise nothing awaited -> unhandledRejection. + Promise.reject(Object.assign(new Error("hedge-cancelled"), { name: "AbortError" })); + // streamHandler.ts aborts with raw strings; undici can reject with them verbatim. + Promise.reject("hedge-cancelled"); + Promise.reject("request_signal_aborted"); + `); + assert.equal(status, 0, "child must survive hedge-cancelled; stderr: " + stderr); assert.match(stdout, /ALIVE/); }); test("installProcessCrashGuard still crashes on genuine errors (no over-swallowing)", async () => { - const guardPath = fileURLToPath( - new URL("../../src/shared/utils/httpClientAbortGuard.mjs", import.meta.url) - ); - const script = ` + const { status, stdout } = await runGuardChild(` const { installProcessCrashGuard } = await import(process.argv[1]); installProcessCrashGuard(); + console.log("LOADED"); // proves the failure below is the re-throw, not a module-load error process.emit("uncaughtException", new Error("genuine failure"), "uncaughtException"); console.log("SHOULD_NOT_REACH"); - `; - const { status, stdout, stderr: _stderr } = await new Promise((resolve, reject) => { - const child = spawn(process.execPath, ["--input-type=module", "-e", script, guardPath], { - stdio: ["ignore", "pipe", "pipe"], - }); - let out = ""; - let err = ""; - child.stdout.on("data", (d) => (out += d)); - child.stderr.on("data", (d) => (err += d)); - child.on("close", (status) => resolve({ status, stdout: out, stderr: err })); - child.on("error", reject); - }); + `); + assert.match(stdout, /LOADED/, "guard must have loaded before the genuine error was raised"); assert.notEqual(status, 0, "genuine errors must keep crash semantics"); assert.doesNotMatch(stdout, /SHOULD_NOT_REACH/); }); +test("isClientAbortError absorbs raw string abort reasons from streamHandler (undici rejects with signal.reason verbatim)", () => { + for (const reason of [ + "hedge-cancelled", + "combo-per-model-timeout", + "request_signal_aborted", + "client_closed", + "cancelled", + ]) { + assert.equal(isClientAbortError(reason), true, reason + " must be absorbed"); + assert.equal(shouldSwallowUncaught(reason, "unhandledRejection"), true); + } + assert.equal(isClientAbortError("genuine failure"), false); + assert.equal(isClientAbortError(""), false); + assert.equal(isClientAbortError(42), false); + assert.equal(isClientAbortError(null), false); +}); // A swallowed error is the ONLY evidence it ever happened; logging just // code/message throws away the stack. The logger must receive the full