fix(chatCore): stop executeWithUpstreamStartTimeout leaking its abortPromise listener (hedge-cancelled process exit) (#12406)

* fix(sse): stop mergeAbortSignals from leaking abort listeners

mergeAbortSignals() attached "abort" listeners to its primary/secondary
signals but never removed them once the merged signal settled. Every
executor fetch attempt calls this (fetchWithStartTimeout, once per
URL/retry), so a busy combo request accumulated one live listener per
call on the long-lived combo/client signal. A leaked listener still
fires when that signal is later aborted (e.g. a hedge cancellation
arriving after this merge's own caller already finished), for a merged
output nothing is watching anymore.

Mirrors the already-correct self-cleaning pattern in
open-sse/utils/directResponseStartTimeout.ts's local mergeAbortSignals.

Regression test measures listener growth across repeated merges of the
same long-lived signal: 25 merges leaked exactly 25 listeners pre-fix,
0 post-fix.

(cherry picked from commit 07969655147d3236969b38bfb41280ab4fb52b79)

* fix(server): stop the crash guard re-throwing combo abort reasons

Production crash 2026-08-31 (omniroute.log): on a client disconnect,
handleDisconnect aborted the combo controller and a late abort listener
threw the abort reason on an empty stack:

    Error [AbortError]: hedge-cancelled
        at ... AbortController.abort ... handleDisconnect
    file:///.../src/shared/utils/httpClientAbortGuard.mjs:130  throw err;

isClientAbortError() only knew Node's stream codes and "aborted", so
shouldSwallowUncaught() said false and the guard re-threw, taking the
whole server down.

- Port upstream's AbortError line (name "AbortError" + abort-flavoured
  message) so request_signal_aborted / DOMException aborts are absorbed.
- Add an exact-message match for the combo abort reasons from
  open-sse/services/combo/comboAbortReasons.ts ("hedge-cancelled",
  "combo-per-model-timeout"), name-agnostic because the raw reason is a
  plain Error that only gets name="AbortError" stamped on the way out.
  A losing hedge / stalled target is never a server fault. Inlined so
  this .mjs stays dependency-free for scripts/dev/run-next.mjs.

Tests: port upstream's guard tests, add the exact crash shape, a
child-process replay of the crash (dies pre-fix, survives post-fix), a
genuine-error case that must still crash, and a sync check against
comboAbortReasons.ts. The child-process helper passes a file:// URL, not
a bare path, so the tests run on Windows.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 90c9bce8c474b60c37cc63f4421d50feae4c0ad2)

* fix(chatCore): stop executeWithUpstreamStartTimeout leaking its abortPromise listener

Root cause of the 2026-08-31 production exit (Error [AbortError]:
hedge-cancelled), verified by mapping the crash frames in
.build/next/server/chunks/13721.js back to this file:

- The abortPromise abort listener registered on the long-lived client /
  stream signal was never removed in the finally block (only abortListener
  and timeoutAbortListener were), so every executor attempt (and every
  retry) leaked one listener onto that signal.
- Promise.race only subscribes to abortPromise/timeoutPromise once the
  array literal has been evaluated. When execute() threw synchronously the
  race never ran, abortPromise was orphaned, and the next hedge
  cancellation / client disconnect aborted the signal with the string
  reason streamHandler.ts forwards; createAbortError() rebuilt it as an
  AbortError-named Error and rejected a promise nothing awaited. That
  unhandledRejection reached the process crash guard, which re-threw it as
  an uncaughtException and exited with code 7.

Keep a handle to the listener and remove it with the others, and mark the
two race-loser promises as handled so a synchronous throw from execute()
can never orphan them. Race semantics are unchanged (the race still
observes their rejections).

Regression tests: (1) a resolving execute leaves the listener count on
the client signal unchanged; (2) a synchronously throwing execute leaks
no listener and a later abort with the string "hedge-cancelled" produces
no unhandledRejection. Both fail against the previous implementation.

Note: commit 079696551 (mergeAbortSignals cleanup) is correct listener
hygiene but is not on this crash path; this is the fix for the incident.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit e68a50ad854a1945c23e747da4ef15a820cc148d)

* fix(server): absorb raw string abort reasons in the crash guard; document the verified crash path

Follow-ups from the adversarial review of 90c9bce8c:

- open-sse/utils/streamHandler.ts aborts the stream controller with a raw
  string reason (getClientAbortReason / handleDisconnect) and undici
  rejects with signal.reason verbatim, so a cancellation can reach
  process level as a bare string. isClientAbortError() returned false for
  every non-object, which would still have exited the process. Absorb the
  combo abort reasons and the stream-handler disconnect reasons when they
  arrive as strings.
- Correct the mechanism comment: the 2026-08-31 exit was a leaked
  upstreamTimeouts.ts abortPromise listener rejecting a promise nothing
  awaited (unhandledRejection), escalated by this guard, not a listener
  throwing synchronously. The leak is fixed at the source in the previous
  commit; this guard remains the last-resort net.
- Reword the inlining rationale (plain node launcher, no reliance on
  type-stripping for the .ts constants module).
- Tests: the child-process replay now also exercises the
  unhandledRejection route with the exact production error shape and with
  raw string reasons; add unit coverage for string reasons and non-object
  look-alikes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 696fcc8fe1b9b9bc43e6e5f5f5e44b619a86e68a)

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Co-authored-by: Beexly <Beexly@users.noreply.github.com>
This commit is contained in:
Beexly
2026-09-18 11:13:06 -05:00
committed by GitHub
parent eb4e3be5dc
commit 706dc75c13
8 changed files with 332 additions and 66 deletions

View File

@@ -0,0 +1 @@
- **fix(chatCore):** stop `executeWithUpstreamStartTimeout` leaking its abortPromise listener onto the long-lived client/stream signal, and stop `mergeAbortSignals` leaking per-attempt abort listeners, so a later hedge cancellation or client disconnect cannot reject an orphaned promise and take the process down (`Error [AbortError]: hedge-cancelled`). The crash guard also absorbs combo abort reasons (`hedge-cancelled`, `combo-per-model-timeout`) and raw string disconnect reasons as a last-resort net ([#12406](https://github.com/diegosouzapw/OmniRoute/pull/12406) — thanks @Beexly)

View File

@@ -130,6 +130,8 @@ import { sanitizeReasoningEffortForProvider } from "./base/reasoningEffort.ts";
// Reasoning-effort sanitation extracted to a pure leaf; re-exported for external
// importers (mimoThinking service + tests) that import it from "./base.ts".
export { sanitizeReasoningEffortForProvider } from "./base/reasoningEffort.ts";
import { mergeAbortSignals } from "./base/mergeAbortSignals.ts";
export { mergeAbortSignals } from "./base/mergeAbortSignals.ts";
/**
* Sanitizes a custom API path to prevent path traversal attacks.
@@ -229,29 +231,6 @@ export type CountTokensInput = {
signal?: AbortSignal | null;
};
export function mergeAbortSignals(primary: AbortSignal, secondary: AbortSignal): AbortSignal {
const controller = new AbortController();
const abortFrom = (source: AbortSignal) => {
if (!controller.signal.aborted) {
controller.abort(source.reason);
}
};
if (primary.aborted) {
abortFrom(primary);
return controller.signal;
}
if (secondary.aborted) {
abortFrom(secondary);
return controller.signal;
}
primary.addEventListener("abort", () => abortFrom(primary), { once: true });
secondary.addEventListener("abort", () => abortFrom(secondary), { once: true });
return controller.signal;
}
import {
hasActiveClaudeThinking,
readNestedThinkingBudget,

View File

@@ -0,0 +1,47 @@
// Self-cleaning abort-signal merge. Extracted from base.ts so the frozen
// executor host does not grow past its file-size cap for this leak fix.
/**
* Merge two abort signals into one that fires when either does.
*
* The `primary`/`secondary` abort listeners registered below MUST be removed once the
* merged controller settles — otherwise they outlive this call and stay attached to
* whichever input signal is longer-lived (typically the combo/client signal, which
* stays open for the whole request while this merge is re-done per executor fetch
* attempt/retry). A hedge cancellation or client disconnect arriving after this
* particular merge's caller has already moved on then still fires the listener,
* detached from anything that's still awaiting it. Mirrors the already-correct
* self-cleaning pattern in `open-sse/utils/directResponseStartTimeout.ts`'s local
* `mergeAbortSignals` (#hedge-cancelled-abort-listener-leak).
*/
export function mergeAbortSignals(primary: AbortSignal, secondary: AbortSignal): AbortSignal {
const controller = new AbortController();
const cleanup = () => {
primary.removeEventListener("abort", onPrimaryAbort);
secondary.removeEventListener("abort", onSecondaryAbort);
};
const abortFrom = (source: AbortSignal) => {
if (!controller.signal.aborted) {
controller.abort(source.reason);
}
cleanup();
};
const onPrimaryAbort = () => abortFrom(primary);
const onSecondaryAbort = () => abortFrom(secondary);
if (primary.aborted) {
abortFrom(primary);
return controller.signal;
}
if (secondary.aborted) {
abortFrom(secondary);
return controller.signal;
}
primary.addEventListener("abort", onPrimaryAbort, { once: true });
secondary.addEventListener("abort", onSecondaryAbort, { once: true });
return controller.signal;
}

View File

@@ -123,10 +123,7 @@ export function getExecutorTimeoutMs(
// Defensive backstop for direct callers: resolveConnectionTimeoutMs is the
// gate (it rejects out-of-range values so the chain falls through); this
// clamp only caps values a future caller could pass unvetted.
return Math.min(
Math.max(0, Math.floor(connectionTimeoutMs)),
MAX_PROVIDER_SPECIFIC_TIMEOUT_MS
);
return Math.min(Math.max(0, Math.floor(connectionTimeoutMs)), MAX_PROVIDER_SPECIFIC_TIMEOUT_MS);
}
const modelOverride = resolveModelTimeoutOverride(provider, model);
if (modelOverride !== undefined) return modelOverride;
@@ -271,15 +268,30 @@ export async function executeWithUpstreamStartTimeout<T>({
}, timeoutMs);
});
let abortPromiseListener: (() => void) | null = null;
const abortPromise = new Promise<never>((_, reject) => {
signal.addEventListener("abort", () => reject(createAbortError(signal)), { once: true });
abortPromiseListener = () => reject(createAbortError(signal));
signal.addEventListener("abort", abortPromiseListener, { once: true });
});
// Promise.race only subscribes to timeoutPromise/abortPromise once the array
// literal below has been evaluated. If execute() throws synchronously the race
// never runs, both promises are orphaned, and a later abort of the long-lived
// client signal surfaces as an unhandledRejection. That was the 2026-08-31
// production exit: a hedge sibling won after a client disconnect, the leaked
// listener below rebuilt the string reason as an AbortError, and nothing was
// awaiting the promise it rejected. Marking them handled keeps the race
// semantics (it still observes the rejections) while closing that path.
abortPromise.catch(() => {});
timeoutPromise.catch(() => {});
try {
return await Promise.race([execute(combinedController.signal), timeoutPromise, abortPromise]);
} finally {
if (timeoutId) clearTimeout(timeoutId);
if (abortListener) signal.removeEventListener("abort", abortListener);
// Never removed before this fix: one listener leaked onto the client signal
// per call (chatCore.ts invokes this once per executor attempt, plus retries).
if (abortPromiseListener) signal.removeEventListener("abort", abortPromiseListener);
if (timeoutAbortListener) {
timeoutController.signal.removeEventListener("abort", timeoutAbortListener);
}

View File

@@ -48,12 +48,31 @@
* @module
*/
/**
* Abort reasons raised by combo target dispatch. Mirror of
* open-sse/services/combo/comboAbortReasons.ts — keep the two in sync.
*/
const COMBO_ABORT_REASONS = new Set(["hedge-cancelled", "combo-per-model-timeout"]);
/**
* Raw string reasons open-sse/utils/streamHandler.ts passes to
* handleDisconnect() / abortController.abort() when the client goes away.
*/
const CLIENT_DISCONNECT_REASONS = new Set(["request_signal_aborted", "client_closed", "cancelled"]);
/**
* @param {unknown} err
* @returns {boolean} true when `err` represents a client closing the
* connection rather than a server-side fault.
*/
export function isClientAbortError(err) {
// open-sse/utils/streamHandler.ts aborts the stream controller with a RAW
// STRING reason (getClientAbortReason / handleDisconnect), and undici rejects
// with signal.reason verbatim, so a cancellation can surface at the process
// level as a bare string rather than an Error object.
if (typeof err === "string") {
return COMBO_ABORT_REASONS.has(err) || CLIENT_DISCONNECT_REASONS.has(err);
}
if (!err || typeof err !== "object") return false;
const e = /** @type {NodeJS.ErrnoException} */ (err);
// Node emits `Error: aborted` (no code) from http.Server#abortIncoming.
@@ -65,6 +84,23 @@ export function isClientAbortError(err) {
// `Error: aborted` — an emitter-left 'error' event on any of these used to
// kill the process (#fix-dev-server-aborted).
if (e.name === "AbortError" && /abort/i.test(String(e.message))) return true;
// Combo dispatch cancels a losing hedged target / a stalled target by
// aborting with `new Error(reason)` for one of the reasons in
// open-sse/services/combo/comboAbortReasons.ts (targetTimeoutRunner.ts).
// streamHandler.ts forwards that reason to the stream controller as a plain
// string, and a leaked abort listener in
// open-sse/handlers/chatCore/upstreamTimeouts.ts (executeWithUpstreamStartTimeout)
// rebuilt it via createAbortError() as an AbortError-named Error to reject a
// promise nothing was awaiting. That unhandledRejection reached this guard,
// which re-threw it as an uncaughtException. Production exit 2026-08-31:
// Error [AbortError]: hedge-cancelled
// The listener leak is fixed at the source; this stays as the last-resort net.
// A sibling target winning / a target stalling is never a server fault, so
// match the exact reason text whatever `name` the thrower stamped on it.
// Inlined rather than imported from comboAbortReasons.ts: this file runs
// under plain node (scripts/dev/run-next.mjs, no tsx) and must not depend on
// type-stripping being available for that .ts module.
if (COMBO_ABORT_REASONS.has(String(e.message))) return true;
switch (e.code) {
case "ERR_STREAM_PREMATURE_CLOSE":
case "ECONNRESET":

View File

@@ -1,10 +1,12 @@
import test from "node:test";
import assert from "node:assert/strict";
import { getEventListeners } from "node:events";
import {
createBodyTimeoutError,
createUpstreamStartTimeoutError,
createAbortError,
executeWithUpstreamStartTimeout,
computeBillableTokens,
getExecutorTimeoutMs,
normalizeExecutorResult,
@@ -57,3 +59,66 @@ test("normalizeExecutorResult rejects malformed executor output", () => {
/must contain a Response/
);
});
test("executeWithUpstreamStartTimeout leaves no abort listener on the client signal after a resolving execute", async () => {
const client = new AbortController();
const before = getEventListeners(client.signal, "abort").length;
const result = await executeWithUpstreamStartTimeout({
executor: {},
provider: "test-provider",
model: "test-model",
connectionTimeoutMs: 5_000,
signal: client.signal,
execute: async () => "ok",
});
assert.equal(result, "ok");
assert.equal(
getEventListeners(client.signal, "abort").length,
before,
"every listener registered for the race must be removed once it settles"
);
});
test("executeWithUpstreamStartTimeout: a synchronously throwing execute cannot orphan abortPromise (2026-08-31 hedge-cancelled exit)", async () => {
// Production shape: execute() threw before Promise.race subscribed, so the
// abortPromise listener stayed on the long-lived client signal with nobody
// awaiting its rejection. The next hedge cancellation / client disconnect
// then aborted that signal with a string reason and the rejection became an
// unhandledRejection: `Error [AbortError]: hedge-cancelled`.
const client = new AbortController();
const before = getEventListeners(client.signal, "abort").length;
await assert.rejects(
executeWithUpstreamStartTimeout({
executor: {},
provider: "test-provider",
model: "test-model",
connectionTimeoutMs: 5_000,
signal: client.signal,
execute: () => {
throw new Error("sync failure before the race subscribed");
},
}),
/sync failure before the race subscribed/
);
assert.equal(
getEventListeners(client.signal, "abort").length,
before,
"no listener may leak onto the client signal when execute() throws synchronously"
);
let unhandled: unknown = null;
const onUnhandled = (reason: unknown) => {
unhandled = reason;
};
process.on("unhandledRejection", onUnhandled);
try {
client.abort("hedge-cancelled");
await new Promise((resolve) => setTimeout(resolve, 25));
} finally {
process.off("unhandledRejection", onUnhandled);
}
assert.equal(
unhandled,
null,
"a late abort must not surface as an unhandledRejection: " + String(unhandled)
);
});

View File

@@ -1,5 +1,7 @@
import test from "node:test";
import assert from "node:assert/strict";
import { getEventListeners } from "node:events";
import type { EventEmitter } from "node:events";
const base = await import("../../open-sse/executors/base.ts");
@@ -116,6 +118,42 @@ test("mergeAbortSignals aborts when secondary fires", () => {
assert.ok(merged.aborted);
});
// Regression: mergeAbortSignals used to attach its "abort" listeners to `primary`/
// `secondary` and never remove them. Every executor fetch attempt calls this (once
// per URL/retry — open-sse/executors/base.ts's fetchWithStartTimeout), so a busy
// combo request accumulated one live listener per call on the long-lived combo/
// client signal. Each leaked listener still fires when that signal is aborted later
// (e.g. a hedge cancellation — COMBO_HEDGE_CANCELLED_REASON, "hedge-cancelled" —
// arriving after this particular merge's own caller already finished), for a
// `merged` output nothing is watching anymore. Proven here by counting real
// listener growth (via node:events' getEventListeners) across repeated merges of
// the SAME long-lived primary signal — exactly the fetchWithStartTimeout pattern.
test("mergeAbortSignals removes its listeners once the merged signal settles (no leak across repeated merges)", () => {
const countAbortListeners = (target: AbortSignal) =>
getEventListeners(target as unknown as EventEmitter, "abort").length;
const longLived = new AbortController();
const before = countAbortListeners(longLived.signal);
for (let i = 0; i < 25; i++) {
const perAttempt = new AbortController();
const merged = base.mergeAbortSignals(longLived.signal, perAttempt.signal);
assert.ok(!merged.aborted);
// Simulate this attempt finishing on its own (its own timeout/abort fires),
// the way each fetchWithStartTimeout call's timeoutController does.
perAttempt.abort(new Error(`attempt-${i}-timeout`));
assert.ok(merged.aborted);
}
const after = countAbortListeners(longLived.signal);
assert.ok(
after <= before + 1,
`expected mergeAbortSignals to clean up its "abort" listener on the long-lived ` +
`signal after each merge settles; listener count grew from ~${before} to ${after} ` +
`across 25 merges (leak)`
);
});
test("sanitizeReasoningEffortForProvider passes through body without reasoning_effort", () => {
const body = { model: "gpt-4o", temperature: 0.7 };
const result = base.sanitizeReasoningEffortForProvider(body, "openai", "gpt-4o");

View File

@@ -103,7 +103,10 @@ test("shouldSwallowUncaught absorbs the real 'aborted' uncaughtException signatu
assert.equal(shouldSwallowUncaught(abortErr, "uncaughtException"), true);
assert.equal(shouldSwallowUncaught(abortErr, undefined), true);
assert.equal(
shouldSwallowUncaught(Object.assign(new Error("ECONNRESET"), { code: "ECONNRESET" }), "uncaughtException"),
shouldSwallowUncaught(
Object.assign(new Error("ECONNRESET"), { code: "ECONNRESET" }),
"uncaughtException"
),
true
);
});
@@ -117,6 +120,7 @@ test("shouldSwallowUncaught preserves crash semantics for genuine errors", () =>
test("installProcessCrashGuard does not throw on import and is idempotent", () => {
assert.doesNotThrow(() => installProcessCrashGuard(() => {}));
assert.doesNotThrow(() => installProcessCrashGuard(() => {}));
});
test("isClientAbortError matches OmniRoute SSE AbortError shapes (#fix-crash-guard-logger-7)", () => {
// Exact production shape from the 2026-08-31 crash log:
@@ -136,18 +140,82 @@ test("shouldSwallowUncaught absorbs SSE AbortError rejections", () => {
assert.equal(shouldSwallowUncaught(sseAbort, "unhandledRejection"), true);
});
// Production crash (2026-08-25 → 08-31, ~170 restarts, exit code 7):
test("isClientAbortError absorbs combo abort reasons (exact 2026-08-31 hedge-cancelled crash shape)", () => {
// omniroute.log, 2026-08-31T23:47Z: the process died at this guard's own
// uncaughtException re-throw with
// Error [AbortError]: hedge-cancelled
// at ... AbortController.abort ... handleDisconnect
// targetTimeoutRunner.ts aborts with new Error("hedge-cancelled"); by the
// time it escapes, the thrower has stamped name = "AbortError" on it.
const relabelled = Object.assign(new Error("hedge-cancelled"), { name: "AbortError" });
assert.equal(isClientAbortError(relabelled), true, "relabelled AbortError must be absorbed");
const raw = new Error("hedge-cancelled");
assert.equal(isClientAbortError(raw), true, "raw abort reason (name=Error) must be absorbed");
const stalled = new Error("combo-per-model-timeout");
assert.equal(
isClientAbortError(stalled),
true,
"per-model timeout abort reason must be absorbed"
);
// Look-alikes that are genuine faults keep crash semantics (exact match only).
assert.equal(isClientAbortError(new Error("hedge-cancelled: unexpected state")), false);
assert.equal(
isClientAbortError(new TypeError("Cannot read properties of undefined (reading 'hedge')")),
false
);
assert.equal(
isClientAbortError(Object.assign(new Error("disk full"), { code: "ENOSPC" })),
false
);
});
test("combo abort reasons in the guard stay in sync with comboAbortReasons.ts", async (t) => {
let mod;
try {
mod = await import("../../open-sse/services/combo/comboAbortReasons.ts");
} catch {
t.skip("TypeScript loader (tsx) not active in this run");
return;
}
for (const reason of [mod.COMBO_HEDGE_CANCELLED_REASON, mod.COMBO_PER_MODEL_TIMEOUT_REASON]) {
assert.equal(typeof reason, "string");
assert.equal(isClientAbortError(new Error(reason)), true, reason + " must be absorbed");
}
});
test("shouldSwallowUncaught absorbs the hedge-cancelled uncaughtException (2026-08-31 crash)", () => {
const hedged = Object.assign(new Error("hedge-cancelled"), { name: "AbortError" });
assert.equal(shouldSwallowUncaught(hedged, "uncaughtException"), true);
assert.equal(shouldSwallowUncaught(hedged, "unhandledRejection"), true);
assert.equal(shouldSwallowUncaught(hedged, undefined), true);
});
function runGuardChild(script) {
// Hand the child a file:// URL, not a filesystem path: dynamic import() of a
// bare Windows path ("C:...") fails with ERR_UNSUPPORTED_ESM_URL_SCHEME.
const guardUrl = new URL("../../src/shared/utils/httpClientAbortGuard.mjs", import.meta.url).href;
return new Promise((resolve, reject) => {
const child = spawn(process.execPath, ["--input-type=module", "-e", script, guardUrl], {
stdio: ["ignore", "pipe", "pipe"],
});
let out = "";
let err = "";
child.stdout.on("data", (d) => (out += d));
child.stderr.on("data", (d) => (err += d));
child.on("close", (status) => resolve({ status, stdout: out, stderr: err }));
child.on("error", reject);
});
}
// Production crash (2026-08-25 -> 08-31, ~170 restarts, exit code 7):
// every real call site installs the guard with NO logger, so the old
// `const logger = log ?? console` default invoked the console OBJECT as a
// function inside the uncaughtException handler TypeError inside
// process._fatalException Node exit code 7. These children run the REAL
// "const logger = log ?? console" default invoked the console OBJECT as a
// function inside the uncaughtException handler -> TypeError inside
// process._fatalException -> Node exit code 7. These children run the REAL
// production call shape; the process must survive benign aborts and still
// crash on genuine errors.
test("installProcessCrashGuard() with no logger swallows aborts instead of dying (exit-7 regression)", async () => {
const guardPath = fileURLToPath(
new URL("../../src/shared/utils/httpClientAbortGuard.mjs", import.meta.url)
);
const script = `
const { status, stdout, stderr } = await runGuardChild(`
const { installProcessCrashGuard } = await import(process.argv[1]);
installProcessCrashGuard(); // production call sites pass NO logger
process.emit(
@@ -162,46 +230,66 @@ test("installProcessCrashGuard() with no logger swallows aborts instead of dying
);
console.log("ALIVE");
process.exit(0);
`;
const { status, stdout, stderr } = await new Promise((resolve, reject) => {
const child = spawn(process.execPath, ["--input-type=module", "-e", script, guardPath], {
stdio: ["ignore", "pipe", "pipe"],
});
let out = "";
let err = "";
child.stdout.on("data", (d) => (out += d));
child.stderr.on("data", (d) => (err += d));
child.on("close", (status) => resolve({ status, stdout: out, stderr: err }));
child.on("error", reject);
});
assert.equal(status, 0, `child must survive benign aborts; stderr: ${stderr}`);
`);
assert.equal(status, 0, "child must survive benign aborts; stderr: " + stderr);
assert.match(stdout, /ALIVE/);
});
test("installProcessCrashGuard() survives the real hedge-cancelled uncaughtException", async () => {
// Replays the 2026-08-31 production crash through a real process: the abort
// reason escapes an abort listener as an uncaughtException with the exact
// name/message the log recorded. Pre-fix this exits non-zero at the re-throw.
const { status, stdout, stderr } = await runGuardChild(`
const { installProcessCrashGuard } = await import(process.argv[1]);
installProcessCrashGuard();
const ctl = new AbortController();
ctl.signal.addEventListener("abort", () => {
const err = ctl.signal.reason;
err.name = "AbortError";
throw err; // escapes the listener -> uncaughtException
}, { once: true });
process.once("exit", (code) => { if (code === 0) console.log("ALIVE"); });
setTimeout(() => process.exit(0), 50);
ctl.abort(new Error("hedge-cancelled"));
// The verified production route: a leaked upstreamTimeouts abortPromise
// listener rejected a promise nothing awaited -> unhandledRejection.
Promise.reject(Object.assign(new Error("hedge-cancelled"), { name: "AbortError" }));
// streamHandler.ts aborts with raw strings; undici can reject with them verbatim.
Promise.reject("hedge-cancelled");
Promise.reject("request_signal_aborted");
`);
assert.equal(status, 0, "child must survive hedge-cancelled; stderr: " + stderr);
assert.match(stdout, /ALIVE/);
});
test("installProcessCrashGuard still crashes on genuine errors (no over-swallowing)", async () => {
const guardPath = fileURLToPath(
new URL("../../src/shared/utils/httpClientAbortGuard.mjs", import.meta.url)
);
const script = `
const { status, stdout } = await runGuardChild(`
const { installProcessCrashGuard } = await import(process.argv[1]);
installProcessCrashGuard();
console.log("LOADED"); // proves the failure below is the re-throw, not a module-load error
process.emit("uncaughtException", new Error("genuine failure"), "uncaughtException");
console.log("SHOULD_NOT_REACH");
`;
const { status, stdout, stderr: _stderr } = await new Promise((resolve, reject) => {
const child = spawn(process.execPath, ["--input-type=module", "-e", script, guardPath], {
stdio: ["ignore", "pipe", "pipe"],
});
let out = "";
let err = "";
child.stdout.on("data", (d) => (out += d));
child.stderr.on("data", (d) => (err += d));
child.on("close", (status) => resolve({ status, stdout: out, stderr: err }));
child.on("error", reject);
});
`);
assert.match(stdout, /LOADED/, "guard must have loaded before the genuine error was raised");
assert.notEqual(status, 0, "genuine errors must keep crash semantics");
assert.doesNotMatch(stdout, /SHOULD_NOT_REACH/);
});
test("isClientAbortError absorbs raw string abort reasons from streamHandler (undici rejects with signal.reason verbatim)", () => {
for (const reason of [
"hedge-cancelled",
"combo-per-model-timeout",
"request_signal_aborted",
"client_closed",
"cancelled",
]) {
assert.equal(isClientAbortError(reason), true, reason + " must be absorbed");
assert.equal(shouldSwallowUncaught(reason, "unhandledRejection"), true);
}
assert.equal(isClientAbortError("genuine failure"), false);
assert.equal(isClientAbortError(""), false);
assert.equal(isClientAbortError(42), false);
assert.equal(isClientAbortError(null), false);
});
// A swallowed error is the ONLY evidence it ever happened; logging just
// code/message throws away the stack. The logger must receive the full