mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-08-18 05:02:15 +03:00
fix(sse): gate structural chat admission shedding on real heap pressure
Closes #10183, Closes #10268 3.8.49 (#9654/#9940) replaced the 3.8.48 heap-ratio shed (heapUsed/heapLimit >= 0.75) in chatBodyAdmission.ts with an unconditional CHAT_MAX_HEAVY_IN_FLIGHT=1 structural lease. A second concurrent "structurally heavy" chat request (>=200 messages, >=64 tools, or >=32k estimated tokens — routine for coding-agent fan-out like Hermes/Cursor/Claude Code) was hard-rejected with a retryable HTTP 503 chat_admission_busy/structure_limit regardless of actual heap pressure, even on a host with ample free RAM. Restore the heap-conditional gate as an ADDITIONAL check layered on top of (not a replacement for) the #9654 bounded-concurrency / per-connection-lane protection: when heavyweight capacity is busy, only enter the bounded-wait/shed path when a live heap-pressure probe (heapUsed / v8 heap_size_limit >= OMNIROUTE_CHAT_ADMISSION_HEAP_SHED_RATIO, default 0.75) confirms real pressure. A healthy heap now admits the second heavy request immediately via a no-op lease instead of parking or shedding it. The probe is injectable via admitChatStructure({ heapPressureCheck }) for deterministic tests. Regression tests: - tests/unit/bug-10183-admission-heavy-healthy-heap.test.ts (new, permanent): healthy-heap 2nd heavy request now admitted (was RED); genuinely pressured heap still sheds it. - tests/unit/probe-10268-structural-503.test.ts (promoted to permanent): the exact reported 503 chat_admission_busy shape is still produced under real heap pressure, and the same fan-out is admitted on a healthy heap. - tests/unit/chat-body-admission.test.ts, tests/unit/chat-body-admission-queue.test.ts, tests/unit/per-connection-admission-9654.test.ts updated to inject heapPressureCheck: () => true where they exercise the busy/shed path, preserving #9654/#4380 coverage. Gates run: npm run typecheck:core (clean), eslint --suppressions-location config/quality/eslint-suppressions.json on changed files (clean), scripts/check/check-file-size.mjs (OK), scripts/check/check-test-discovery.mjs (OK), focused admission suite (68/68 passing) and npm run test:unit (in progress at commit time under heavy shared-devbox contention from a 13-way parallel session fan-out; no admission-related failures observed through 1873 lines of output, the sole failure seen was a pre-existing unrelated proxy/search timeout consistent with known load-induced flakiness, not a regression from this change). ⚠️ base-red inherited: #9985 — ESLint errors (2) from #10250
This commit is contained in:
@@ -17,6 +17,7 @@
|
||||
|
||||
import { CORS_HEADERS } from "../utils/cors";
|
||||
import { createHash } from "crypto";
|
||||
import v8 from "node:v8";
|
||||
|
||||
function parsePositiveInt(value: string | undefined, fallback: number): number {
|
||||
const parsed = Number.parseInt(String(value), 10);
|
||||
@@ -80,6 +81,41 @@ export const CHAT_HEAVY_ESTIMATED_TOKENS = parsePositiveInt(
|
||||
process.env.OMNIROUTE_CHAT_HEAVY_ESTIMATED_TOKENS,
|
||||
32_000
|
||||
);
|
||||
|
||||
/**
|
||||
* Heap-pressure shed ratio for the structural admission gate (#10183, #10268).
|
||||
*
|
||||
* 3.8.48 only shed a heavy request once `heapUsed / heapLimit >= shedRatio` (0.75).
|
||||
* 3.8.49 (#9654/#9940) replaced that heap-conditional shed with an unconditional
|
||||
* `CHAT_MAX_HEAVY_IN_FLIGHT=1` structural lease, so a second concurrent "heavy"
|
||||
* request (coding-agent fan-out is the common trigger) was hard-rejected with a
|
||||
* retryable 503 even on a host with ample free RAM. This restores the heap
|
||||
* condition as an ADDITIONAL gate layered on top of the bounded-concurrency /
|
||||
* per-connection-lane protection from #9654 (that protection stays in force —
|
||||
* this constant only decides whether a *busy* lease is still shed with a 503 or
|
||||
* admitted anyway because the heap has real headroom).
|
||||
*/
|
||||
export const CHAT_ADMISSION_HEAP_SHED_RATIO = (() => {
|
||||
const parsed = Number(process.env.OMNIROUTE_CHAT_ADMISSION_HEAP_SHED_RATIO);
|
||||
return Number.isFinite(parsed) && parsed > 0 && parsed <= 1 ? parsed : 0.75;
|
||||
})();
|
||||
|
||||
/**
|
||||
* Live `heapUsed / heap_size_limit` pressure probe, injectable for deterministic
|
||||
* tests (`admitChatStructure({ heapPressureCheck })`). Defaults to the real V8
|
||||
* heap statistics. Any read failure is treated as "not under pressure" so a
|
||||
* transient stats error never turns into a false structural shed.
|
||||
*/
|
||||
export function defaultHeapPressureCheck(): boolean {
|
||||
try {
|
||||
const heapUsed = process.memoryUsage().heapUsed;
|
||||
const heapLimit = v8.getHeapStatistics().heap_size_limit;
|
||||
if (!Number.isFinite(heapLimit) || heapLimit <= 0) return false;
|
||||
return heapUsed / heapLimit >= CHAT_ADMISSION_HEAP_SHED_RATIO;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
/**
|
||||
* Optional per-deployment history cap. `0` (the default) disables it.
|
||||
*
|
||||
@@ -523,6 +559,12 @@ export async function admitChatStructure(
|
||||
heavyTokens?: number;
|
||||
queueMs?: number;
|
||||
signal?: AbortSignal;
|
||||
/**
|
||||
* Heap-pressure probe consulted only when heavyweight capacity is busy
|
||||
* (#10183, #10268). Defaults to `defaultHeapPressureCheck` (live V8 heap
|
||||
* stats). Tests inject a deterministic override.
|
||||
*/
|
||||
heapPressureCheck?: () => boolean;
|
||||
} = {}
|
||||
): Promise<ChatStructureAdmission> {
|
||||
if (!body || typeof body !== "object" || Array.isArray(body)) return { admit: true, lease };
|
||||
@@ -560,6 +602,26 @@ export async function admitChatStructure(
|
||||
(options.sessionId
|
||||
? perConnectionAdmissionController.getController(options.sessionId)
|
||||
: defaultAdmissionController);
|
||||
|
||||
// Uncontended fast path: capacity is free, no need to consult heap pressure at all.
|
||||
const immediate = controller.tryAcquireHeavy();
|
||||
if (immediate) return { admit: true, lease: immediate };
|
||||
|
||||
// Heavyweight capacity is momentarily busy (a concurrent heavy request holds the
|
||||
// lease). #10183 / #10268: only enter the bounded-wait / shed path — with its
|
||||
// queued-bytes heap valve and abort handling (#9654) — when the heap is
|
||||
// GENUINELY under pressure. This restores the 3.8.48 `heapUsed/heapLimit >=
|
||||
// shedRatio` condition as an additional gate on top of (never a replacement
|
||||
// for) the bounded-concurrency / per-connection-lane protection above. A
|
||||
// healthy heap has real headroom for a second heavy request even while the
|
||||
// single lease is momentarily busy, so admit it immediately instead of
|
||||
// parking/shedding a request that has nothing to do with actual resource
|
||||
// pressure.
|
||||
const heapPressureCheck = options.heapPressureCheck ?? defaultHeapPressureCheck;
|
||||
if (!heapPressureCheck()) {
|
||||
return { admit: true, lease: createNoopLease() };
|
||||
}
|
||||
|
||||
// Structural-only waits happen on byte-light bodies (a byte-heavy body already
|
||||
// holds the byte-stage lease), so the conservative 256KB weight bounds the
|
||||
// parsed JSON the waiter keeps resident while parked.
|
||||
@@ -633,14 +695,18 @@ export function resolveSelfLoopBearer(): string {
|
||||
* gap that kept the Zoo Code / api-key describe call failing even after the byte
|
||||
* stage was bypassed. Release is a no-op; capacity was never reserved.
|
||||
*/
|
||||
const NULL_LEASE: ChatAdmissionLease = {
|
||||
get released() {
|
||||
return true;
|
||||
},
|
||||
release() {
|
||||
// No-op: the sentinel never reserved heavyweight capacity.
|
||||
},
|
||||
};
|
||||
function createNoopLease(): ChatAdmissionLease {
|
||||
return {
|
||||
get released() {
|
||||
return true;
|
||||
},
|
||||
release() {
|
||||
// No-op: this sentinel never reserved heavyweight capacity.
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
const NULL_LEASE: ChatAdmissionLease = createNoopLease();
|
||||
|
||||
/**
|
||||
* True when the request is a trusted in-process self-loop sub-request that must
|
||||
|
||||
Reference in New Issue
Block a user