mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-09-19 05:12:16 +03:00
fix(chatCore): stop executeWithUpstreamStartTimeout leaking its abortPromise listener (hedge-cancelled process exit) (#12406)
* fix(sse): stop mergeAbortSignals from leaking abort listeners
mergeAbortSignals() attached "abort" listeners to its primary/secondary
signals but never removed them once the merged signal settled. Every
executor fetch attempt calls this (fetchWithStartTimeout, once per
URL/retry), so a busy combo request accumulated one live listener per
call on the long-lived combo/client signal. A leaked listener still
fires when that signal is later aborted (e.g. a hedge cancellation
arriving after this merge's own caller already finished), for a merged
output nothing is watching anymore.
Mirrors the already-correct self-cleaning pattern in
open-sse/utils/directResponseStartTimeout.ts's local mergeAbortSignals.
Regression test measures listener growth across repeated merges of the
same long-lived signal: 25 merges leaked exactly 25 listeners pre-fix,
0 post-fix.
(cherry picked from commit 07969655147d3236969b38bfb41280ab4fb52b79)
* fix(server): stop the crash guard re-throwing combo abort reasons
Production crash 2026-08-31 (omniroute.log): on a client disconnect,
handleDisconnect aborted the combo controller and a late abort listener
threw the abort reason on an empty stack:
Error [AbortError]: hedge-cancelled
at ... AbortController.abort ... handleDisconnect
file:///.../src/shared/utils/httpClientAbortGuard.mjs:130 throw err;
isClientAbortError() only knew Node's stream codes and "aborted", so
shouldSwallowUncaught() said false and the guard re-threw, taking the
whole server down.
- Port upstream's AbortError line (name "AbortError" + abort-flavoured
message) so request_signal_aborted / DOMException aborts are absorbed.
- Add an exact-message match for the combo abort reasons from
open-sse/services/combo/comboAbortReasons.ts ("hedge-cancelled",
"combo-per-model-timeout"), name-agnostic because the raw reason is a
plain Error that only gets name="AbortError" stamped on the way out.
A losing hedge / stalled target is never a server fault. Inlined so
this .mjs stays dependency-free for scripts/dev/run-next.mjs.
Tests: port upstream's guard tests, add the exact crash shape, a
child-process replay of the crash (dies pre-fix, survives post-fix), a
genuine-error case that must still crash, and a sync check against
comboAbortReasons.ts. The child-process helper passes a file:// URL, not
a bare path, so the tests run on Windows.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 90c9bce8c474b60c37cc63f4421d50feae4c0ad2)
* fix(chatCore): stop executeWithUpstreamStartTimeout leaking its abortPromise listener
Root cause of the 2026-08-31 production exit (Error [AbortError]:
hedge-cancelled), verified by mapping the crash frames in
.build/next/server/chunks/13721.js back to this file:
- The abortPromise abort listener registered on the long-lived client /
stream signal was never removed in the finally block (only abortListener
and timeoutAbortListener were), so every executor attempt (and every
retry) leaked one listener onto that signal.
- Promise.race only subscribes to abortPromise/timeoutPromise once the
array literal has been evaluated. When execute() threw synchronously the
race never ran, abortPromise was orphaned, and the next hedge
cancellation / client disconnect aborted the signal with the string
reason streamHandler.ts forwards; createAbortError() rebuilt it as an
AbortError-named Error and rejected a promise nothing awaited. That
unhandledRejection reached the process crash guard, which re-threw it as
an uncaughtException and exited with code 7.
Keep a handle to the listener and remove it with the others, and mark the
two race-loser promises as handled so a synchronous throw from execute()
can never orphan them. Race semantics are unchanged (the race still
observes their rejections).
Regression tests: (1) a resolving execute leaves the listener count on
the client signal unchanged; (2) a synchronously throwing execute leaks
no listener and a later abort with the string "hedge-cancelled" produces
no unhandledRejection. Both fail against the previous implementation.
Note: commit 079696551 (mergeAbortSignals cleanup) is correct listener
hygiene but is not on this crash path; this is the fix for the incident.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit e68a50ad854a1945c23e747da4ef15a820cc148d)
* fix(server): absorb raw string abort reasons in the crash guard; document the verified crash path
Follow-ups from the adversarial review of 90c9bce8c:
- open-sse/utils/streamHandler.ts aborts the stream controller with a raw
string reason (getClientAbortReason / handleDisconnect) and undici
rejects with signal.reason verbatim, so a cancellation can reach
process level as a bare string. isClientAbortError() returned false for
every non-object, which would still have exited the process. Absorb the
combo abort reasons and the stream-handler disconnect reasons when they
arrive as strings.
- Correct the mechanism comment: the 2026-08-31 exit was a leaked
upstreamTimeouts.ts abortPromise listener rejecting a promise nothing
awaited (unhandledRejection), escalated by this guard, not a listener
throwing synchronously. The leak is fixed at the source in the previous
commit; this guard remains the last-resort net.
- Reword the inlining rationale (plain node launcher, no reliance on
type-stripping for the .ts constants module).
- Tests: the child-process replay now also exercises the
unhandledRejection route with the exact production error shape and with
raw string reasons; add unit coverage for string reasons and non-object
look-alikes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 696fcc8fe1b9b9bc43e6e5f5f5e44b619a86e68a)
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Co-authored-by: Beexly <Beexly@users.noreply.github.com>
This commit is contained in:
@@ -0,0 +1 @@
|
||||
- **fix(chatCore):** stop `executeWithUpstreamStartTimeout` leaking its abortPromise listener onto the long-lived client/stream signal, and stop `mergeAbortSignals` leaking per-attempt abort listeners, so a later hedge cancellation or client disconnect cannot reject an orphaned promise and take the process down (`Error [AbortError]: hedge-cancelled`). The crash guard also absorbs combo abort reasons (`hedge-cancelled`, `combo-per-model-timeout`) and raw string disconnect reasons as a last-resort net ([#12406](https://github.com/diegosouzapw/OmniRoute/pull/12406) — thanks @Beexly)
|
||||
@@ -130,6 +130,8 @@ import { sanitizeReasoningEffortForProvider } from "./base/reasoningEffort.ts";
|
||||
// Reasoning-effort sanitation extracted to a pure leaf; re-exported for external
|
||||
// importers (mimoThinking service + tests) that import it from "./base.ts".
|
||||
export { sanitizeReasoningEffortForProvider } from "./base/reasoningEffort.ts";
|
||||
import { mergeAbortSignals } from "./base/mergeAbortSignals.ts";
|
||||
export { mergeAbortSignals } from "./base/mergeAbortSignals.ts";
|
||||
|
||||
/**
|
||||
* Sanitizes a custom API path to prevent path traversal attacks.
|
||||
@@ -229,29 +231,6 @@ export type CountTokensInput = {
|
||||
signal?: AbortSignal | null;
|
||||
};
|
||||
|
||||
export function mergeAbortSignals(primary: AbortSignal, secondary: AbortSignal): AbortSignal {
|
||||
const controller = new AbortController();
|
||||
|
||||
const abortFrom = (source: AbortSignal) => {
|
||||
if (!controller.signal.aborted) {
|
||||
controller.abort(source.reason);
|
||||
}
|
||||
};
|
||||
|
||||
if (primary.aborted) {
|
||||
abortFrom(primary);
|
||||
return controller.signal;
|
||||
}
|
||||
if (secondary.aborted) {
|
||||
abortFrom(secondary);
|
||||
return controller.signal;
|
||||
}
|
||||
|
||||
primary.addEventListener("abort", () => abortFrom(primary), { once: true });
|
||||
secondary.addEventListener("abort", () => abortFrom(secondary), { once: true });
|
||||
return controller.signal;
|
||||
}
|
||||
|
||||
import {
|
||||
hasActiveClaudeThinking,
|
||||
readNestedThinkingBudget,
|
||||
|
||||
47
open-sse/executors/base/mergeAbortSignals.ts
Normal file
47
open-sse/executors/base/mergeAbortSignals.ts
Normal file
@@ -0,0 +1,47 @@
|
||||
// Self-cleaning abort-signal merge. Extracted from base.ts so the frozen
|
||||
// executor host does not grow past its file-size cap for this leak fix.
|
||||
|
||||
/**
|
||||
* Merge two abort signals into one that fires when either does.
|
||||
*
|
||||
* The `primary`/`secondary` abort listeners registered below MUST be removed once the
|
||||
* merged controller settles — otherwise they outlive this call and stay attached to
|
||||
* whichever input signal is longer-lived (typically the combo/client signal, which
|
||||
* stays open for the whole request while this merge is re-done per executor fetch
|
||||
* attempt/retry). A hedge cancellation or client disconnect arriving after this
|
||||
* particular merge's caller has already moved on then still fires the listener,
|
||||
* detached from anything that's still awaiting it. Mirrors the already-correct
|
||||
* self-cleaning pattern in `open-sse/utils/directResponseStartTimeout.ts`'s local
|
||||
* `mergeAbortSignals` (#hedge-cancelled-abort-listener-leak).
|
||||
*/
|
||||
export function mergeAbortSignals(primary: AbortSignal, secondary: AbortSignal): AbortSignal {
|
||||
const controller = new AbortController();
|
||||
|
||||
const cleanup = () => {
|
||||
primary.removeEventListener("abort", onPrimaryAbort);
|
||||
secondary.removeEventListener("abort", onSecondaryAbort);
|
||||
};
|
||||
|
||||
const abortFrom = (source: AbortSignal) => {
|
||||
if (!controller.signal.aborted) {
|
||||
controller.abort(source.reason);
|
||||
}
|
||||
cleanup();
|
||||
};
|
||||
|
||||
const onPrimaryAbort = () => abortFrom(primary);
|
||||
const onSecondaryAbort = () => abortFrom(secondary);
|
||||
|
||||
if (primary.aborted) {
|
||||
abortFrom(primary);
|
||||
return controller.signal;
|
||||
}
|
||||
if (secondary.aborted) {
|
||||
abortFrom(secondary);
|
||||
return controller.signal;
|
||||
}
|
||||
|
||||
primary.addEventListener("abort", onPrimaryAbort, { once: true });
|
||||
secondary.addEventListener("abort", onSecondaryAbort, { once: true });
|
||||
return controller.signal;
|
||||
}
|
||||
@@ -123,10 +123,7 @@ export function getExecutorTimeoutMs(
|
||||
// Defensive backstop for direct callers: resolveConnectionTimeoutMs is the
|
||||
// gate (it rejects out-of-range values so the chain falls through); this
|
||||
// clamp only caps values a future caller could pass unvetted.
|
||||
return Math.min(
|
||||
Math.max(0, Math.floor(connectionTimeoutMs)),
|
||||
MAX_PROVIDER_SPECIFIC_TIMEOUT_MS
|
||||
);
|
||||
return Math.min(Math.max(0, Math.floor(connectionTimeoutMs)), MAX_PROVIDER_SPECIFIC_TIMEOUT_MS);
|
||||
}
|
||||
const modelOverride = resolveModelTimeoutOverride(provider, model);
|
||||
if (modelOverride !== undefined) return modelOverride;
|
||||
@@ -271,15 +268,30 @@ export async function executeWithUpstreamStartTimeout<T>({
|
||||
}, timeoutMs);
|
||||
});
|
||||
|
||||
let abortPromiseListener: (() => void) | null = null;
|
||||
const abortPromise = new Promise<never>((_, reject) => {
|
||||
signal.addEventListener("abort", () => reject(createAbortError(signal)), { once: true });
|
||||
abortPromiseListener = () => reject(createAbortError(signal));
|
||||
signal.addEventListener("abort", abortPromiseListener, { once: true });
|
||||
});
|
||||
// Promise.race only subscribes to timeoutPromise/abortPromise once the array
|
||||
// literal below has been evaluated. If execute() throws synchronously the race
|
||||
// never runs, both promises are orphaned, and a later abort of the long-lived
|
||||
// client signal surfaces as an unhandledRejection. That was the 2026-08-31
|
||||
// production exit: a hedge sibling won after a client disconnect, the leaked
|
||||
// listener below rebuilt the string reason as an AbortError, and nothing was
|
||||
// awaiting the promise it rejected. Marking them handled keeps the race
|
||||
// semantics (it still observes the rejections) while closing that path.
|
||||
abortPromise.catch(() => {});
|
||||
timeoutPromise.catch(() => {});
|
||||
|
||||
try {
|
||||
return await Promise.race([execute(combinedController.signal), timeoutPromise, abortPromise]);
|
||||
} finally {
|
||||
if (timeoutId) clearTimeout(timeoutId);
|
||||
if (abortListener) signal.removeEventListener("abort", abortListener);
|
||||
// Never removed before this fix: one listener leaked onto the client signal
|
||||
// per call (chatCore.ts invokes this once per executor attempt, plus retries).
|
||||
if (abortPromiseListener) signal.removeEventListener("abort", abortPromiseListener);
|
||||
if (timeoutAbortListener) {
|
||||
timeoutController.signal.removeEventListener("abort", timeoutAbortListener);
|
||||
}
|
||||
|
||||
@@ -48,12 +48,31 @@
|
||||
* @module
|
||||
*/
|
||||
|
||||
/**
|
||||
* Abort reasons raised by combo target dispatch. Mirror of
|
||||
* open-sse/services/combo/comboAbortReasons.ts — keep the two in sync.
|
||||
*/
|
||||
const COMBO_ABORT_REASONS = new Set(["hedge-cancelled", "combo-per-model-timeout"]);
|
||||
|
||||
/**
|
||||
* Raw string reasons open-sse/utils/streamHandler.ts passes to
|
||||
* handleDisconnect() / abortController.abort() when the client goes away.
|
||||
*/
|
||||
const CLIENT_DISCONNECT_REASONS = new Set(["request_signal_aborted", "client_closed", "cancelled"]);
|
||||
|
||||
/**
|
||||
* @param {unknown} err
|
||||
* @returns {boolean} true when `err` represents a client closing the
|
||||
* connection rather than a server-side fault.
|
||||
*/
|
||||
export function isClientAbortError(err) {
|
||||
// open-sse/utils/streamHandler.ts aborts the stream controller with a RAW
|
||||
// STRING reason (getClientAbortReason / handleDisconnect), and undici rejects
|
||||
// with signal.reason verbatim, so a cancellation can surface at the process
|
||||
// level as a bare string rather than an Error object.
|
||||
if (typeof err === "string") {
|
||||
return COMBO_ABORT_REASONS.has(err) || CLIENT_DISCONNECT_REASONS.has(err);
|
||||
}
|
||||
if (!err || typeof err !== "object") return false;
|
||||
const e = /** @type {NodeJS.ErrnoException} */ (err);
|
||||
// Node emits `Error: aborted` (no code) from http.Server#abortIncoming.
|
||||
@@ -65,6 +84,23 @@ export function isClientAbortError(err) {
|
||||
// `Error: aborted` — an emitter-left 'error' event on any of these used to
|
||||
// kill the process (#fix-dev-server-aborted).
|
||||
if (e.name === "AbortError" && /abort/i.test(String(e.message))) return true;
|
||||
// Combo dispatch cancels a losing hedged target / a stalled target by
|
||||
// aborting with `new Error(reason)` for one of the reasons in
|
||||
// open-sse/services/combo/comboAbortReasons.ts (targetTimeoutRunner.ts).
|
||||
// streamHandler.ts forwards that reason to the stream controller as a plain
|
||||
// string, and a leaked abort listener in
|
||||
// open-sse/handlers/chatCore/upstreamTimeouts.ts (executeWithUpstreamStartTimeout)
|
||||
// rebuilt it via createAbortError() as an AbortError-named Error to reject a
|
||||
// promise nothing was awaiting. That unhandledRejection reached this guard,
|
||||
// which re-threw it as an uncaughtException. Production exit 2026-08-31:
|
||||
// Error [AbortError]: hedge-cancelled
|
||||
// The listener leak is fixed at the source; this stays as the last-resort net.
|
||||
// A sibling target winning / a target stalling is never a server fault, so
|
||||
// match the exact reason text whatever `name` the thrower stamped on it.
|
||||
// Inlined rather than imported from comboAbortReasons.ts: this file runs
|
||||
// under plain node (scripts/dev/run-next.mjs, no tsx) and must not depend on
|
||||
// type-stripping being available for that .ts module.
|
||||
if (COMBO_ABORT_REASONS.has(String(e.message))) return true;
|
||||
switch (e.code) {
|
||||
case "ERR_STREAM_PREMATURE_CLOSE":
|
||||
case "ECONNRESET":
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { getEventListeners } from "node:events";
|
||||
|
||||
import {
|
||||
createBodyTimeoutError,
|
||||
createUpstreamStartTimeoutError,
|
||||
createAbortError,
|
||||
executeWithUpstreamStartTimeout,
|
||||
computeBillableTokens,
|
||||
getExecutorTimeoutMs,
|
||||
normalizeExecutorResult,
|
||||
@@ -57,3 +59,66 @@ test("normalizeExecutorResult rejects malformed executor output", () => {
|
||||
/must contain a Response/
|
||||
);
|
||||
});
|
||||
test("executeWithUpstreamStartTimeout leaves no abort listener on the client signal after a resolving execute", async () => {
|
||||
const client = new AbortController();
|
||||
const before = getEventListeners(client.signal, "abort").length;
|
||||
const result = await executeWithUpstreamStartTimeout({
|
||||
executor: {},
|
||||
provider: "test-provider",
|
||||
model: "test-model",
|
||||
connectionTimeoutMs: 5_000,
|
||||
signal: client.signal,
|
||||
execute: async () => "ok",
|
||||
});
|
||||
assert.equal(result, "ok");
|
||||
assert.equal(
|
||||
getEventListeners(client.signal, "abort").length,
|
||||
before,
|
||||
"every listener registered for the race must be removed once it settles"
|
||||
);
|
||||
});
|
||||
|
||||
test("executeWithUpstreamStartTimeout: a synchronously throwing execute cannot orphan abortPromise (2026-08-31 hedge-cancelled exit)", async () => {
|
||||
// Production shape: execute() threw before Promise.race subscribed, so the
|
||||
// abortPromise listener stayed on the long-lived client signal with nobody
|
||||
// awaiting its rejection. The next hedge cancellation / client disconnect
|
||||
// then aborted that signal with a string reason and the rejection became an
|
||||
// unhandledRejection: `Error [AbortError]: hedge-cancelled`.
|
||||
const client = new AbortController();
|
||||
const before = getEventListeners(client.signal, "abort").length;
|
||||
await assert.rejects(
|
||||
executeWithUpstreamStartTimeout({
|
||||
executor: {},
|
||||
provider: "test-provider",
|
||||
model: "test-model",
|
||||
connectionTimeoutMs: 5_000,
|
||||
signal: client.signal,
|
||||
execute: () => {
|
||||
throw new Error("sync failure before the race subscribed");
|
||||
},
|
||||
}),
|
||||
/sync failure before the race subscribed/
|
||||
);
|
||||
assert.equal(
|
||||
getEventListeners(client.signal, "abort").length,
|
||||
before,
|
||||
"no listener may leak onto the client signal when execute() throws synchronously"
|
||||
);
|
||||
|
||||
let unhandled: unknown = null;
|
||||
const onUnhandled = (reason: unknown) => {
|
||||
unhandled = reason;
|
||||
};
|
||||
process.on("unhandledRejection", onUnhandled);
|
||||
try {
|
||||
client.abort("hedge-cancelled");
|
||||
await new Promise((resolve) => setTimeout(resolve, 25));
|
||||
} finally {
|
||||
process.off("unhandledRejection", onUnhandled);
|
||||
}
|
||||
assert.equal(
|
||||
unhandled,
|
||||
null,
|
||||
"a late abort must not surface as an unhandledRejection: " + String(unhandled)
|
||||
);
|
||||
});
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { getEventListeners } from "node:events";
|
||||
import type { EventEmitter } from "node:events";
|
||||
|
||||
const base = await import("../../open-sse/executors/base.ts");
|
||||
|
||||
@@ -116,6 +118,42 @@ test("mergeAbortSignals aborts when secondary fires", () => {
|
||||
assert.ok(merged.aborted);
|
||||
});
|
||||
|
||||
// Regression: mergeAbortSignals used to attach its "abort" listeners to `primary`/
|
||||
// `secondary` and never remove them. Every executor fetch attempt calls this (once
|
||||
// per URL/retry — open-sse/executors/base.ts's fetchWithStartTimeout), so a busy
|
||||
// combo request accumulated one live listener per call on the long-lived combo/
|
||||
// client signal. Each leaked listener still fires when that signal is aborted later
|
||||
// (e.g. a hedge cancellation — COMBO_HEDGE_CANCELLED_REASON, "hedge-cancelled" —
|
||||
// arriving after this particular merge's own caller already finished), for a
|
||||
// `merged` output nothing is watching anymore. Proven here by counting real
|
||||
// listener growth (via node:events' getEventListeners) across repeated merges of
|
||||
// the SAME long-lived primary signal — exactly the fetchWithStartTimeout pattern.
|
||||
test("mergeAbortSignals removes its listeners once the merged signal settles (no leak across repeated merges)", () => {
|
||||
const countAbortListeners = (target: AbortSignal) =>
|
||||
getEventListeners(target as unknown as EventEmitter, "abort").length;
|
||||
|
||||
const longLived = new AbortController();
|
||||
const before = countAbortListeners(longLived.signal);
|
||||
|
||||
for (let i = 0; i < 25; i++) {
|
||||
const perAttempt = new AbortController();
|
||||
const merged = base.mergeAbortSignals(longLived.signal, perAttempt.signal);
|
||||
assert.ok(!merged.aborted);
|
||||
// Simulate this attempt finishing on its own (its own timeout/abort fires),
|
||||
// the way each fetchWithStartTimeout call's timeoutController does.
|
||||
perAttempt.abort(new Error(`attempt-${i}-timeout`));
|
||||
assert.ok(merged.aborted);
|
||||
}
|
||||
|
||||
const after = countAbortListeners(longLived.signal);
|
||||
assert.ok(
|
||||
after <= before + 1,
|
||||
`expected mergeAbortSignals to clean up its "abort" listener on the long-lived ` +
|
||||
`signal after each merge settles; listener count grew from ~${before} to ${after} ` +
|
||||
`across 25 merges (leak)`
|
||||
);
|
||||
});
|
||||
|
||||
test("sanitizeReasoningEffortForProvider passes through body without reasoning_effort", () => {
|
||||
const body = { model: "gpt-4o", temperature: 0.7 };
|
||||
const result = base.sanitizeReasoningEffortForProvider(body, "openai", "gpt-4o");
|
||||
|
||||
@@ -103,7 +103,10 @@ test("shouldSwallowUncaught absorbs the real 'aborted' uncaughtException signatu
|
||||
assert.equal(shouldSwallowUncaught(abortErr, "uncaughtException"), true);
|
||||
assert.equal(shouldSwallowUncaught(abortErr, undefined), true);
|
||||
assert.equal(
|
||||
shouldSwallowUncaught(Object.assign(new Error("ECONNRESET"), { code: "ECONNRESET" }), "uncaughtException"),
|
||||
shouldSwallowUncaught(
|
||||
Object.assign(new Error("ECONNRESET"), { code: "ECONNRESET" }),
|
||||
"uncaughtException"
|
||||
),
|
||||
true
|
||||
);
|
||||
});
|
||||
@@ -117,6 +120,7 @@ test("shouldSwallowUncaught preserves crash semantics for genuine errors", () =>
|
||||
|
||||
test("installProcessCrashGuard does not throw on import and is idempotent", () => {
|
||||
assert.doesNotThrow(() => installProcessCrashGuard(() => {}));
|
||||
assert.doesNotThrow(() => installProcessCrashGuard(() => {}));
|
||||
});
|
||||
test("isClientAbortError matches OmniRoute SSE AbortError shapes (#fix-crash-guard-logger-7)", () => {
|
||||
// Exact production shape from the 2026-08-31 crash log:
|
||||
@@ -136,18 +140,82 @@ test("shouldSwallowUncaught absorbs SSE AbortError rejections", () => {
|
||||
assert.equal(shouldSwallowUncaught(sseAbort, "unhandledRejection"), true);
|
||||
});
|
||||
|
||||
// Production crash (2026-08-25 → 08-31, ~170 restarts, exit code 7):
|
||||
test("isClientAbortError absorbs combo abort reasons (exact 2026-08-31 hedge-cancelled crash shape)", () => {
|
||||
// omniroute.log, 2026-08-31T23:47Z: the process died at this guard's own
|
||||
// uncaughtException re-throw with
|
||||
// Error [AbortError]: hedge-cancelled
|
||||
// at ... AbortController.abort ... handleDisconnect
|
||||
// targetTimeoutRunner.ts aborts with new Error("hedge-cancelled"); by the
|
||||
// time it escapes, the thrower has stamped name = "AbortError" on it.
|
||||
const relabelled = Object.assign(new Error("hedge-cancelled"), { name: "AbortError" });
|
||||
assert.equal(isClientAbortError(relabelled), true, "relabelled AbortError must be absorbed");
|
||||
const raw = new Error("hedge-cancelled");
|
||||
assert.equal(isClientAbortError(raw), true, "raw abort reason (name=Error) must be absorbed");
|
||||
const stalled = new Error("combo-per-model-timeout");
|
||||
assert.equal(
|
||||
isClientAbortError(stalled),
|
||||
true,
|
||||
"per-model timeout abort reason must be absorbed"
|
||||
);
|
||||
// Look-alikes that are genuine faults keep crash semantics (exact match only).
|
||||
assert.equal(isClientAbortError(new Error("hedge-cancelled: unexpected state")), false);
|
||||
assert.equal(
|
||||
isClientAbortError(new TypeError("Cannot read properties of undefined (reading 'hedge')")),
|
||||
false
|
||||
);
|
||||
assert.equal(
|
||||
isClientAbortError(Object.assign(new Error("disk full"), { code: "ENOSPC" })),
|
||||
false
|
||||
);
|
||||
});
|
||||
|
||||
test("combo abort reasons in the guard stay in sync with comboAbortReasons.ts", async (t) => {
|
||||
let mod;
|
||||
try {
|
||||
mod = await import("../../open-sse/services/combo/comboAbortReasons.ts");
|
||||
} catch {
|
||||
t.skip("TypeScript loader (tsx) not active in this run");
|
||||
return;
|
||||
}
|
||||
for (const reason of [mod.COMBO_HEDGE_CANCELLED_REASON, mod.COMBO_PER_MODEL_TIMEOUT_REASON]) {
|
||||
assert.equal(typeof reason, "string");
|
||||
assert.equal(isClientAbortError(new Error(reason)), true, reason + " must be absorbed");
|
||||
}
|
||||
});
|
||||
|
||||
test("shouldSwallowUncaught absorbs the hedge-cancelled uncaughtException (2026-08-31 crash)", () => {
|
||||
const hedged = Object.assign(new Error("hedge-cancelled"), { name: "AbortError" });
|
||||
assert.equal(shouldSwallowUncaught(hedged, "uncaughtException"), true);
|
||||
assert.equal(shouldSwallowUncaught(hedged, "unhandledRejection"), true);
|
||||
assert.equal(shouldSwallowUncaught(hedged, undefined), true);
|
||||
});
|
||||
|
||||
function runGuardChild(script) {
|
||||
// Hand the child a file:// URL, not a filesystem path: dynamic import() of a
|
||||
// bare Windows path ("C:...") fails with ERR_UNSUPPORTED_ESM_URL_SCHEME.
|
||||
const guardUrl = new URL("../../src/shared/utils/httpClientAbortGuard.mjs", import.meta.url).href;
|
||||
return new Promise((resolve, reject) => {
|
||||
const child = spawn(process.execPath, ["--input-type=module", "-e", script, guardUrl], {
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
});
|
||||
let out = "";
|
||||
let err = "";
|
||||
child.stdout.on("data", (d) => (out += d));
|
||||
child.stderr.on("data", (d) => (err += d));
|
||||
child.on("close", (status) => resolve({ status, stdout: out, stderr: err }));
|
||||
child.on("error", reject);
|
||||
});
|
||||
}
|
||||
|
||||
// Production crash (2026-08-25 -> 08-31, ~170 restarts, exit code 7):
|
||||
// every real call site installs the guard with NO logger, so the old
|
||||
// `const logger = log ?? console` default invoked the console OBJECT as a
|
||||
// function inside the uncaughtException handler → TypeError inside
|
||||
// process._fatalException → Node exit code 7. These children run the REAL
|
||||
// "const logger = log ?? console" default invoked the console OBJECT as a
|
||||
// function inside the uncaughtException handler -> TypeError inside
|
||||
// process._fatalException -> Node exit code 7. These children run the REAL
|
||||
// production call shape; the process must survive benign aborts and still
|
||||
// crash on genuine errors.
|
||||
test("installProcessCrashGuard() with no logger swallows aborts instead of dying (exit-7 regression)", async () => {
|
||||
const guardPath = fileURLToPath(
|
||||
new URL("../../src/shared/utils/httpClientAbortGuard.mjs", import.meta.url)
|
||||
);
|
||||
const script = `
|
||||
const { status, stdout, stderr } = await runGuardChild(`
|
||||
const { installProcessCrashGuard } = await import(process.argv[1]);
|
||||
installProcessCrashGuard(); // production call sites pass NO logger
|
||||
process.emit(
|
||||
@@ -162,46 +230,66 @@ test("installProcessCrashGuard() with no logger swallows aborts instead of dying
|
||||
);
|
||||
console.log("ALIVE");
|
||||
process.exit(0);
|
||||
`;
|
||||
const { status, stdout, stderr } = await new Promise((resolve, reject) => {
|
||||
const child = spawn(process.execPath, ["--input-type=module", "-e", script, guardPath], {
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
});
|
||||
let out = "";
|
||||
let err = "";
|
||||
child.stdout.on("data", (d) => (out += d));
|
||||
child.stderr.on("data", (d) => (err += d));
|
||||
child.on("close", (status) => resolve({ status, stdout: out, stderr: err }));
|
||||
child.on("error", reject);
|
||||
});
|
||||
assert.equal(status, 0, `child must survive benign aborts; stderr: ${stderr}`);
|
||||
`);
|
||||
assert.equal(status, 0, "child must survive benign aborts; stderr: " + stderr);
|
||||
assert.match(stdout, /ALIVE/);
|
||||
});
|
||||
|
||||
test("installProcessCrashGuard() survives the real hedge-cancelled uncaughtException", async () => {
|
||||
// Replays the 2026-08-31 production crash through a real process: the abort
|
||||
// reason escapes an abort listener as an uncaughtException with the exact
|
||||
// name/message the log recorded. Pre-fix this exits non-zero at the re-throw.
|
||||
const { status, stdout, stderr } = await runGuardChild(`
|
||||
const { installProcessCrashGuard } = await import(process.argv[1]);
|
||||
installProcessCrashGuard();
|
||||
const ctl = new AbortController();
|
||||
ctl.signal.addEventListener("abort", () => {
|
||||
const err = ctl.signal.reason;
|
||||
err.name = "AbortError";
|
||||
throw err; // escapes the listener -> uncaughtException
|
||||
}, { once: true });
|
||||
process.once("exit", (code) => { if (code === 0) console.log("ALIVE"); });
|
||||
setTimeout(() => process.exit(0), 50);
|
||||
ctl.abort(new Error("hedge-cancelled"));
|
||||
// The verified production route: a leaked upstreamTimeouts abortPromise
|
||||
// listener rejected a promise nothing awaited -> unhandledRejection.
|
||||
Promise.reject(Object.assign(new Error("hedge-cancelled"), { name: "AbortError" }));
|
||||
// streamHandler.ts aborts with raw strings; undici can reject with them verbatim.
|
||||
Promise.reject("hedge-cancelled");
|
||||
Promise.reject("request_signal_aborted");
|
||||
`);
|
||||
assert.equal(status, 0, "child must survive hedge-cancelled; stderr: " + stderr);
|
||||
assert.match(stdout, /ALIVE/);
|
||||
});
|
||||
|
||||
test("installProcessCrashGuard still crashes on genuine errors (no over-swallowing)", async () => {
|
||||
const guardPath = fileURLToPath(
|
||||
new URL("../../src/shared/utils/httpClientAbortGuard.mjs", import.meta.url)
|
||||
);
|
||||
const script = `
|
||||
const { status, stdout } = await runGuardChild(`
|
||||
const { installProcessCrashGuard } = await import(process.argv[1]);
|
||||
installProcessCrashGuard();
|
||||
console.log("LOADED"); // proves the failure below is the re-throw, not a module-load error
|
||||
process.emit("uncaughtException", new Error("genuine failure"), "uncaughtException");
|
||||
console.log("SHOULD_NOT_REACH");
|
||||
`;
|
||||
const { status, stdout, stderr: _stderr } = await new Promise((resolve, reject) => {
|
||||
const child = spawn(process.execPath, ["--input-type=module", "-e", script, guardPath], {
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
});
|
||||
let out = "";
|
||||
let err = "";
|
||||
child.stdout.on("data", (d) => (out += d));
|
||||
child.stderr.on("data", (d) => (err += d));
|
||||
child.on("close", (status) => resolve({ status, stdout: out, stderr: err }));
|
||||
child.on("error", reject);
|
||||
});
|
||||
`);
|
||||
assert.match(stdout, /LOADED/, "guard must have loaded before the genuine error was raised");
|
||||
assert.notEqual(status, 0, "genuine errors must keep crash semantics");
|
||||
assert.doesNotMatch(stdout, /SHOULD_NOT_REACH/);
|
||||
});
|
||||
test("isClientAbortError absorbs raw string abort reasons from streamHandler (undici rejects with signal.reason verbatim)", () => {
|
||||
for (const reason of [
|
||||
"hedge-cancelled",
|
||||
"combo-per-model-timeout",
|
||||
"request_signal_aborted",
|
||||
"client_closed",
|
||||
"cancelled",
|
||||
]) {
|
||||
assert.equal(isClientAbortError(reason), true, reason + " must be absorbed");
|
||||
assert.equal(shouldSwallowUncaught(reason, "unhandledRejection"), true);
|
||||
}
|
||||
assert.equal(isClientAbortError("genuine failure"), false);
|
||||
assert.equal(isClientAbortError(""), false);
|
||||
assert.equal(isClientAbortError(42), false);
|
||||
assert.equal(isClientAbortError(null), false);
|
||||
});
|
||||
|
||||
// A swallowed error is the ONLY evidence it ever happened; logging just
|
||||
// code/message throws away the stack. The logger must receive the full
|
||||
|
||||
Reference in New Issue
Block a user