pjandro/OmniRoute
1
0
Fork 0
mirror of https://github.com/diegosouzapw/OmniRoute.git synced 2026-09-21 14:22:14 +03:00
Code Issues Packages Projects Releases Wiki Activity

fix(resilience): extend process crash guard to combo hedge cancels and upstream fetch failures (#13636) (#14064)

Browse Source 
Closes a real process-killer: the direct-response start timeout could fire after the fetch promise had already settled, and aborting at that point delivered the abort reason to a promise nobody was awaiting — Node promotes that to an `unhandledRejection` → `uncaughtException` and the process dies (#12861). The timer is now a no-op once the attempt has settled, and the same guard is extended to combo hedge cancels and upstream fetch failures.

Validated as a combined board first (this PR merged with the 11 siblings of the same batch on the release tip): eslint on every changed file with the suppressions file, typecheck:core, check:open-sse-typecheck, complexity, cognitive-complexity, changelog-integrity, i18n new-key coverage, docs-sync, migration-numbering, provider-consistency and a duplicate-identifier audit all green, plus 275 passing / 0 failing focused node:test cases across the 28 test files the batch touches. Then re-validated alone on the fresh tip before this merge: conflicts re-resolved, file sizes rebaselined for this PR's own growth, eslint and this PR's focused tests re-run.

Thanks @HouMinXi!

Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com>
This commit is contained in:
Bob.Hou
2026-09-18 09:21:32 -04:00
committed by GitHub
parent 9b92477b40
commit 815dedd3c9
10 changed files with 522 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files

134
src/shared/utils/httpClientAbortGuard.mjs
View File

@@ -1,7 +1,8 @@
"use strict";
/**
* HTTP client-abort crash guard (#fix-dev-server-aborted).
* HTTP client-abort / recoverable-upstream-timeout crash guard
* (#fix-dev-server-aborted, #12861).
*
* Node's http.Server turns an 'error' event on an IncomingMessage/ServerResponse
* into an uncaughtException (and therefore a process exit) WHENEVER the emitter
@@ -16,14 +17,29 @@
* connections + a live WebSocket; stray client-side socket closes during
* navigation/HMR were taking the dev server down.
*
* Two more categories were added after the 2026-09-14 agnes-cn upstream storm
* produced two sibling escapes in production: an intentional combo hedge
* cancellation (`AbortError: hedge-cancelled` — the sibling leg already won,
* so the cancellation is expected, not a fault) and undici fetch failures
* (`TypeError: fetch failed` with a socket-level code) against a flapping
* upstream. Both are runtime/environmental conditions the request layer
* already handles; neither is a process-fatal logic bug.
*
* A further, unrelated category covers #12861: `directFetchWithBoundedResponseStart`'s
* response-start timeout (`DIRECT_RESPONSE_START_TIMEOUT`) is a *recoverable*
* signal `proxyFetch.ts` already retries on a fresh socket — but a narrow
* timer/promise-settlement race can still deliver its abort reason to a
* promise nobody is awaiting anymore, which otherwise kills the whole process
* over a single upstream stall that the retry path was built to handle.
*
* Two layers:
* 1. `attachRequestStreamGuards(req, res)` — per-request listeners that absorb
* client-abort errors so they never bubble to the process level. Call it
* inside every `http.createServer((req, res) => …)` request listener.
* 2. `installProcessCrashGuard()` — a last-resort safety net on
* `process.on('uncaughtException' | 'unhandledRejection')` that swallows
* the same benign client-abort errors but otherwise preserves the existing
* crash semantics (so genuine bugs still surface). Idempotent.
* the same benign errors but otherwise preserves the existing crash
* semantics (so genuine bugs still surface). Idempotent.
*
* Kept as a `.mjs` module (no build step) so it is importable both from the
* Node-only dev server (`scripts/dev/run-next.mjs`) and from the TypeScript
@@ -63,9 +79,100 @@ export function isClientAbortError(err) {
}
}
/**
* #12861: a recoverable upstream-fetch timeout that `proxyFetch.ts` already
* retries on a fresh socket (see `open-sse/utils/directResponseStartTimeout.ts`).
* A narrow timer/promise-settlement race can still deliver its abort reason to
* a promise nobody is awaiting anymore, which otherwise surfaces here as an
* unhandledRejection/uncaughtException — even though the retry path already
* handles this exact condition and normally logs it as a plain 504.
*
* Kept as a bare string-code check (no import of the `.ts` source of truth)
* because this file has to stay build-free/plain-JS-loadable — see the module
* docstring. `DIRECT_RESPONSE_START_TIMEOUT_CODE` in
* `open-sse/utils/directResponseStartTimeout.ts` is the canonical definition;
* keep this string literal in sync with it.
*
* @param {unknown} err
* @returns {boolean}
*/
export function isRecoverableUpstreamTimeoutError(err) {
// Same reason-shape tolerance as isIntentionalComboAbort: a bare string
// reason rejects waiters with the string itself, not an Error object.
if (err === "DIRECT_RESPONSE_START_TIMEOUT") return true;
if (!err || typeof err !== "object") return false;
return /** @type {NodeJS.ErrnoException} */ (err).code === "DIRECT_RESPONSE_START_TIMEOUT";
}
/**
* Intentional combo-leg cancellation. When a combo dispatches hedged targets,
* the losing legs are aborted with a distinctive reason once a sibling wins
* (`hedge-cancelled`) or exceeds its per-model budget (`combo-per-model-timeout`)
* — see `COMBO_HEDGE_CANCELLED_REASON` / `COMBO_PER_MODEL_TIMEOUT_REASON` in
* `open-sse/services/combo/comboAbortReasons.ts` (bare literals duplicated here
* because this file must stay build-free; keep in sync). On 2026-09-14 such a
* cancellation escaped its promise chain and killed production with
* `Error [AbortError]: hedge-cancelled` — the request it belonged to had
* already completed 200 via the winning leg.
*
* Distinct from a *client* abort: only these exact reasons qualify, so an
* AbortError from an unknown subsystem still crashes loudly.
*
* @param {unknown} err
* @returns {boolean}
*/
export function isIntentionalComboAbort(err) {
const reasons = new Set(["hedge-cancelled", "combo-per-model-timeout"]);
// AbortSignal.reason is whatever was handed to abort(): a raw string
// reason rejects waiters with the string itself, not an Error object.
if (typeof err === "string") return reasons.has(err);
if (!err || typeof err !== "object") return false;
const e = /** @type {NodeJS.ErrnoException} */ (err);
if (e.name !== "AbortError") return false;
if (reasons.has(String(e.message))) return true;
const cause = /** @type {{ cause?: unknown }} */ (err).cause;
return typeof cause === "string" && reasons.has(cause);
}
/**
* A network/IO failure against an upstream or its proxy — undici surfaces it
* as `TypeError: fetch failed` (fixed message; the syscall code rides on
* `cause`) or as an error carrying a `PROXY_UNREACHABLE` / `UND_ERR_*` code.
* On 2026-09-14 one of these (`PROXY_UNREACHABLE` / ECONNRESET to
* api.agnes-ai.cn) escaped as an uncaughtException and killed production.
* The request that triggered the fetch already fails through the normal
* error path; the stray copy delivered to nobody must not be process-fatal.
*
* The "fetch failed" message match is exact on purpose: it is undici's fixed
* wrapping message, so arbitrary TypeErrors still crash loudly.
*
* @param {unknown} err
* @returns {boolean}
*/
export function isUpstreamNetworkError(err) {
if (!err || typeof err !== "object") return false;
const e = /** @type {NodeJS.ErrnoException} */ (err);
if (e.name === "TypeError" && e.message === "fetch failed") return true;
switch (e.code) {
case "PROXY_UNREACHABLE":
case "UND_ERR_SOCKET":
case "UND_ERR_CONNECT_TIMEOUT":
case "UND_ERR_HEADERS_TIMEOUT":
case "UND_ERR_BODY_TIMEOUT":
case "ECONNREFUSED":
case "EHOSTUNREACH":
case "ENETUNREACH":
case "EAI_AGAIN":
return true;
default:
return false;
}
}
/**
* Decide whether a process-level uncaughtException/unhandledRejection should be
* swallowed (benign client-abort) or allowed to surface (genuine bug).
* swallowed (benign client-abort, or a recoverable upstream timeout that a
* retry path already handles — #12861) or allowed to surface (genuine bug).
*
* Pure + exported so it can be unit-tested without poking process listeners.
*
@@ -75,7 +182,14 @@ export function isClientAbortError(err) {
* @returns {boolean} true => swallow (log only), false => re-throw / let crash.
*/
export function shouldSwallowUncaught(err, origin) {
if (!isClientAbortError(err)) return false;
if (
!isClientAbortError(err) &&
!isRecoverableUpstreamTimeoutError(err) &&
!isIntentionalComboAbort(err) &&
!isUpstreamNetworkError(err)
) {
return false;
}
// Only swallow when the origin matches what the guard installed for. If some
// other subsystem raised it (e.g. a deliberate `throw` in a domain), keep the
// existing crash semantics.
@@ -131,7 +245,9 @@ export function installProcessCrashGuard(log) {
process.on("uncaughtException", (err, origin) => {
if (shouldSwallowUncaught(err, origin)) {
logger("warn", "[server] swallowed client-abort uncaughtException:", err?.message ?? err);
// The warn line is the only evidence a swallowed error ever happened;
// pass the full error object so the stack survives.
logger("warn", "[server] swallowed benign uncaughtException:", err);
return;
}
throw err;
@@ -139,11 +255,7 @@ export function installProcessCrashGuard(log) {
process.on("unhandledRejection", (reason) => {
if (shouldSwallowUncaught(reason, "unhandledRejection")) {
logger(
"warn",
"[server] swallowed client-abort unhandledRejection:",
reason?.message ?? reason
);
logger("warn", "[server] swallowed benign unhandledRejection:", reason);
return;
}
throw reason;