Commit Graph

6 Commits

Author SHA1 Message Date
Beexly
706dc75c13 fix(chatCore): stop executeWithUpstreamStartTimeout leaking its abortPromise listener (hedge-cancelled process exit) (#12406)
* fix(sse): stop mergeAbortSignals from leaking abort listeners

mergeAbortSignals() attached "abort" listeners to its primary/secondary
signals but never removed them once the merged signal settled. Every
executor fetch attempt calls this (fetchWithStartTimeout, once per
URL/retry), so a busy combo request accumulated one live listener per
call on the long-lived combo/client signal. A leaked listener still
fires when that signal is later aborted (e.g. a hedge cancellation
arriving after this merge's own caller already finished), for a merged
output nothing is watching anymore.

Mirrors the already-correct self-cleaning pattern in
open-sse/utils/directResponseStartTimeout.ts's local mergeAbortSignals.

Regression test measures listener growth across repeated merges of the
same long-lived signal: 25 merges leaked exactly 25 listeners pre-fix,
0 post-fix.

(cherry picked from commit 07969655147d3236969b38bfb41280ab4fb52b79)

* fix(server): stop the crash guard re-throwing combo abort reasons

Production crash 2026-08-31 (omniroute.log): on a client disconnect,
handleDisconnect aborted the combo controller and a late abort listener
threw the abort reason on an empty stack:

    Error [AbortError]: hedge-cancelled
        at ... AbortController.abort ... handleDisconnect
    file:///.../src/shared/utils/httpClientAbortGuard.mjs:130  throw err;

isClientAbortError() only knew Node's stream codes and "aborted", so
shouldSwallowUncaught() said false and the guard re-threw, taking the
whole server down.

- Port upstream's AbortError line (name "AbortError" + abort-flavoured
  message) so request_signal_aborted / DOMException aborts are absorbed.
- Add an exact-message match for the combo abort reasons from
  open-sse/services/combo/comboAbortReasons.ts ("hedge-cancelled",
  "combo-per-model-timeout"), name-agnostic because the raw reason is a
  plain Error that only gets name="AbortError" stamped on the way out.
  A losing hedge / stalled target is never a server fault. Inlined so
  this .mjs stays dependency-free for scripts/dev/run-next.mjs.

Tests: port upstream's guard tests, add the exact crash shape, a
child-process replay of the crash (dies pre-fix, survives post-fix), a
genuine-error case that must still crash, and a sync check against
comboAbortReasons.ts. The child-process helper passes a file:// URL, not
a bare path, so the tests run on Windows.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 90c9bce8c474b60c37cc63f4421d50feae4c0ad2)

* fix(chatCore): stop executeWithUpstreamStartTimeout leaking its abortPromise listener

Root cause of the 2026-08-31 production exit (Error [AbortError]:
hedge-cancelled), verified by mapping the crash frames in
.build/next/server/chunks/13721.js back to this file:

- The abortPromise abort listener registered on the long-lived client /
  stream signal was never removed in the finally block (only abortListener
  and timeoutAbortListener were), so every executor attempt (and every
  retry) leaked one listener onto that signal.
- Promise.race only subscribes to abortPromise/timeoutPromise once the
  array literal has been evaluated. When execute() threw synchronously the
  race never ran, abortPromise was orphaned, and the next hedge
  cancellation / client disconnect aborted the signal with the string
  reason streamHandler.ts forwards; createAbortError() rebuilt it as an
  AbortError-named Error and rejected a promise nothing awaited. That
  unhandledRejection reached the process crash guard, which re-threw it as
  an uncaughtException and exited with code 7.

Keep a handle to the listener and remove it with the others, and mark the
two race-loser promises as handled so a synchronous throw from execute()
can never orphan them. Race semantics are unchanged (the race still
observes their rejections).

Regression tests: (1) a resolving execute leaves the listener count on
the client signal unchanged; (2) a synchronously throwing execute leaks
no listener and a later abort with the string "hedge-cancelled" produces
no unhandledRejection. Both fail against the previous implementation.

Note: commit 079696551 (mergeAbortSignals cleanup) is correct listener
hygiene but is not on this crash path; this is the fix for the incident.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit e68a50ad854a1945c23e747da4ef15a820cc148d)

* fix(server): absorb raw string abort reasons in the crash guard; document the verified crash path

Follow-ups from the adversarial review of 90c9bce8c:

- open-sse/utils/streamHandler.ts aborts the stream controller with a raw
  string reason (getClientAbortReason / handleDisconnect) and undici
  rejects with signal.reason verbatim, so a cancellation can reach
  process level as a bare string. isClientAbortError() returned false for
  every non-object, which would still have exited the process. Absorb the
  combo abort reasons and the stream-handler disconnect reasons when they
  arrive as strings.
- Correct the mechanism comment: the 2026-08-31 exit was a leaked
  upstreamTimeouts.ts abortPromise listener rejecting a promise nothing
  awaited (unhandledRejection), escalated by this guard, not a listener
  throwing synchronously. The leak is fixed at the source in the previous
  commit; this guard remains the last-resort net.
- Reword the inlining rationale (plain node launcher, no reliance on
  type-stripping for the .ts constants module).
- Tests: the child-process replay now also exercises the
  unhandledRejection route with the exact production error shape and with
  raw string reasons; add unit coverage for string reasons and non-object
  look-alikes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 696fcc8fe1b9b9bc43e6e5f5f5e44b619a86e68a)

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Co-authored-by: Beexly <Beexly@users.noreply.github.com>
2026-09-18 13:13:06 -03:00
Bob.Hou
815dedd3c9 fix(resilience): extend process crash guard to combo hedge cancels and upstream fetch failures (#13636) (#14064)
Closes a real process-killer: the direct-response start timeout could fire after the fetch promise had already settled, and aborting at that point delivered the abort reason to a promise nobody was awaiting — Node promotes that to an `unhandledRejection` → `uncaughtException` and the process dies (#12861). The timer is now a no-op once the attempt has settled, and the same guard is extended to combo hedge cancels and upstream fetch failures.

Validated as a combined board first (this PR merged with the 11 siblings of the same batch on the release tip): eslint on every changed file with the suppressions file, typecheck:core, check:open-sse-typecheck, complexity, cognitive-complexity, changelog-integrity, i18n new-key coverage, docs-sync, migration-numbering, provider-consistency and a duplicate-identifier audit all green, plus 275 passing / 0 failing focused node:test cases across the 28 test files the batch touches. Then re-validated alone on the fresh tip before this merge: conflicts re-resolved, file sizes rebaselined for this PR's own growth, eslint and this PR's focused tests re-run.

Thanks @HouMinXi!

Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com>
2026-09-18 10:21:32 -03:00
Bob.Hou
e7999c477b fix(db): bound health scans and isolate native diagnostics (#13717)
Merged after a maintainer rework that kept every one of @HouMinXi's commits intact.

**What the rework added on top of the contribution:** the new DB health-check behaviour is gated behind a default-off feature flag (`src/shared/constants/featureFlagDefinitions.ts`, `defaultValue: "false"`), documented in `docs/reference/FEATURE_FLAGS.md` with the description key carried into all 66 locales, so the release default is unchanged and the new bounds only apply when an operator opts in. The optional-FTS5 migration set was reconciled by hand with the "180" entry that landed meanwhile (`src/lib/db/migrationRunner/constants.ts`).

**Carried from your rebased head:** the `/api/db/health` local-only classification in `src/server/authz/routeGuard.ts` plus its `routeGuard` assertion — `runManagedDbHealthCheck()` forks native diagnostics into a child process, so Hard Rules #15/#17 apply. Re-verified here: 37 pass / 0 fail.

Validated as a combined board first (this PR merged with the 21 siblings of the same wave on the release tip): eslint with the frozen suppressions, typecheck:core, check:open-sse-typecheck, complexity, cognitive-complexity, changelog-integrity, i18n new-key coverage, docs-counts, docs-sync, migration-numbering, provider-consistency and a duplicate-identifier audit all green, plus 176 passing / 0 failing focused node:test cases across the 25 test files the wave touches and the dashboard test under Vitest (2/0). Then re-validated alone on the fresh tip before this merge: conflicts re-resolved, file sizes rebaselined for this PR's own growth, eslint and this PR's focused tests re-run.

Thank you for the depth of this one — the resource-bounds suite and the sql.js startup/backup coverage are the kind of tests that keep a database layer honest.
2026-09-16 12:59:45 -03:00
Bob.Hou
f36fcd8c31 fix(resilience): extend process crash guard to combo hedge cancels and upstream fetch failures (#13636)
Closes a real process-killer: the direct-response start timeout could fire after the fetch promise had already settled, and aborting at that point delivered the abort reason to a promise nobody was awaiting — Node promotes that to an `unhandledRejection` → `uncaughtException` and the process dies (#12861). The timer is now a no-op once the attempt has settled, and the same guard is extended to combo hedge cancels and upstream fetch failures.

Validated as a combined board first (this PR merged with the 11 siblings of the same batch on the release tip): eslint on every changed file with the suppressions file, typecheck:core, check:open-sse-typecheck, complexity, cognitive-complexity, changelog-integrity, i18n new-key coverage, docs-sync, migration-numbering, provider-consistency and a duplicate-identifier audit all green, plus 275 passing / 0 failing focused node:test cases across the 28 test files the batch touches. Then re-validated alone on the fresh tip before this merge: conflicts re-resolved, file sizes rebaselined for this PR's own growth, eslint and this PR's focused tests re-run.

Thanks @HouMinXi!
2026-09-16 03:03:07 -03:00
Alvin T. Veroy
668beed5b8 fix(sse): absorb AbortError/request_signal_aborted in the client-abort crash guard (#12165)
OmniRoute's SSE teardown aborts in-flight legs with
`Error [AbortError]: request_signal_aborted` on client disconnects
(open-sse/utils/streamHandler.ts getClientAbortReason), and fetch/DOM
cancellation surfaces as AbortError with an abort-flavoured message.
isClientAbortError() only matched message 'aborted'/'Aborted' plus errno
codes, so these shapes fell through shouldSwallowUncaught() and were
re-thrown from the process-level uncaughtException/unhandledRejection
handlers — killing the whole server on a routine client disconnect
(observed as repeated exit-code-7 crashes with
'uncaughtException: Error [AbortError]: request_signal_aborted').

Match AbortError by name when the message is abort-flavoured; genuine
errors that merely mention 'abort' (e.g. TypeError) still crash loudly.

Tests: new unit cases for the SSE/DOM AbortError shapes, a child-process
regression proving the process survives both benign emissions with the
production no-logger install shape, and a child-process test proving
genuine errors keep crash semantics.
2026-08-31 14:10:33 -03:00
Az1muth
2d185de9d9 fix(sse): prevent client-abort (aborted/ECONNRESET) from crashing the server (#11556)
Merged via /merge-batch (lote 2026-08-26, v3.8.51). Boarded no worktree combinado junto com outras ~30 PRs; validação única: typecheck/complexity/cognitive-complexity/changelog-integrity verdes, file-size rebaseado onde necessário (crescimento legítimo), lint com os mesmos 228 achados pré-existentes confirmados via sonda contra o tip puro (não introduzidos por este lote), e ~370 testes focados (unit + vitest) passando. Obrigado pela contribuição.
2026-08-26 08:11:14 -03:00