pjandro/OmniRoute
1
0
Fork 0
mirror of https://github.com/diegosouzapw/OmniRoute.git synced 2026-08-12 02:02:13 +03:00
Code Issues Packages Projects Releases Wiki Activity

fix(admission): queue heavyweight chat requests before 503 busy (#9845)

Browse Source 
Agent clients (OpenCode, Claude Code, Cursor) fan out heavy sub-requests
that land on the admission gate together. With the single heavyweight
slot, concurrent heavy requests were rejected immediately with a
retryable 503; clients burn their retry budget in seconds and the agent
dies mid-task.

Heavy requests now wait up to OMNIROUTE_CHAT_ADMISSION_QUEUE_MS (default
5000ms) for a slot before the 503, served FIFO; 0 restores the legacy
immediate-reject behaviour. Applied to both the byte-based path
(admitChatRequest) and the structure-based path (admitChatStructure, now
async).

Co-authored-by: herjarsa <herjarsa@users.noreply.github.com>
This commit is contained in:
Diego Rodrigues de Sa e Souza
2026-08-09 09:50:37 -03:00
committed by GitHub
parent 580162548a
commit 247f2606cd
5 changed files with 270 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
docs/guides/TROUBLESHOOTING.md
View File

@@ -523,6 +523,12 @@ exhausts its bounds of `10,000` visited nodes or depth `12`.
Each process uses a process-local guard to reserve limited heavyweight capacity before retaining
and parsing a large request body. A heavyweight lease remains held for the lifetime of an SSE
response.
When capacity is busy, a heavyweight request first waits up to
`OMNIROUTE_CHAT_ADMISSION_QUEUE_MS` (default `5000`, `0` disables the wait) for a slot to free up
before answering the retryable `503`. The bounded wait exists so agent-style clients
(OpenCode, Claude Code, Cursor) that fan out heavy sub-requests concurrently serialize the burst
instead of burning their whole retry budget on immediate rejections and dying mid-task.
Current heavyweight lease occupancy is not surfaced in the dashboard.
Settings → Resilience → Request Queue → Concurrent Requests does not control this; that setting
governs a separate provider request-queue mechanism.
@@ -530,13 +536,18 @@ governs a separate provider request-queue mechanism.
**Fix:**
1. Retry first. Clients should honor `Retry-After` and use backoff rather than immediately
repeating the request.
repeating the request. Note that with the default `OMNIROUTE_CHAT_ADMISSION_QUEUE_MS=5000`
a heavy request already waited up to 5 seconds before the `503`, so a client retry loop should
back off beyond that instead of hammering.
2. If normal deployment traffic repeatedly exhausts the guard, you can cautiously raise
`OMNIROUTE_CHAT_MAX_HEAVY_IN_FLIGHT` from its default of `1`. Increase it one step at a time,
restart OmniRoute after each change, and observe memory headroom under representative load.
Every additional heavyweight request can increase concurrent V8 heap use and container or
host OOM risk. No value is safe for every deployment; validate the setting against your own
traffic and memory limits rather than assuming that `2` is universally safe.
3. Prefer widening the wait (`OMNIROUTE_CHAT_ADMISSION_QUEUE_MS`) over raising the in-flight
limit when bursts are short: waiting costs latency, while an extra concurrent heavyweight
request costs heap residency for the whole request lifetime.
See the [environment-variable reference](../reference/ENVIRONMENT.md#4-security--authentication)
for the authoritative admission settings. Loosening the heavyweight classification thresholds

3
docs/reference/ENVIRONMENT.md
View File

@@ -190,7 +190,8 @@ OmniRoute uses **SQLite** (via `better-sqlite3`) for all persistence. These vari
| `MAX_BODY_SIZE_BYTES` | `10485760` (10 MB) | `src/shared/middleware/bodySizeGuard.ts` | Maximum allowed request body size. Rejects payloads exceeding this limit. |
| `OMNIROUTE_CHAT_LARGE_BODY_BYTES` | `262144` (256 KB) | `src/shared/middleware/chatBodyAdmission.ts` | Actual request bodies at or above this threshold require an atomic process-local heavyweight admission lease before JSON parsing. |
| `OMNIROUTE_CHAT_HARD_MAX_BODY_BYTES` | `52428800` (50 MB) | `src/shared/middleware/chatBodyAdmission.ts` | Chat-route hard cap enforced against bytes read during bounded ingestion, including requests with missing, invalid, or dishonest `Content-Length`; excess receives `413`. |
| `OMNIROUTE_CHAT_MAX_HEAVY_IN_FLIGHT` | `1` | `src/shared/middleware/chatBodyAdmission.ts` | Maximum heavyweight chat requests admitted concurrently in one process. When capacity is unavailable, OmniRoute returns retryable `503` with `Retry-After`. |
| `OMNIROUTE_CHAT_MAX_HEAVY_IN_FLIGHT` | `1` | `src/shared/middleware/chatBodyAdmission.ts` | Maximum heavyweight chat requests admitted concurrently in one process. When capacity is unavailable, OmniRoute waits up to `OMNIROUTE_CHAT_ADMISSION_QUEUE_MS` for a slot, then returns retryable `503` with `Retry-After`. |
| `OMNIROUTE_CHAT_ADMISSION_QUEUE_MS` | `5000` | `src/shared/middleware/chatBodyAdmission.ts` | How long a heavyweight chat request waits for an admission slot before the retryable `503`. A bounded wait serializes agent bursts (OpenCode, Claude Code, Cursor sub-requests) that would otherwise burn their client retry budget on immediate rejections; `0` restores the legacy immediate-reject behaviour. |
| `OMNIROUTE_CHAT_HEAVY_MESSAGE_COUNT` | `200` | `src/shared/middleware/chatBodyAdmission.ts` | Message count that classifies a chat request as heavyweight even when its body is below the byte threshold. |
| `OMNIROUTE_CHAT_HEAVY_TOOL_COUNT` | `64` | `src/shared/middleware/chatBodyAdmission.ts` | Tool count that classifies a chat request as heavyweight even when its body is below the byte threshold. |
| `OMNIROUTE_CHAT_HEAVY_ESTIMATED_TOKENS` | `32000` | `src/shared/middleware/chatBodyAdmission.ts` | Conservative string-size token estimate that classifies a request as heavyweight; this is an admission-cost proxy, not provider billing tokenization. |

9
src/app/api/v1/chat/completions/route.ts
View File

@@ -17,6 +17,7 @@ import { resolveKeepaliveThreshold } from "@omniroute/open-sse/utils/keepaliveTh
import {
admitChatRequest,
admitChatStructure,
CHAT_ADMISSION_QUEUE_MAX_MS,
releaseChatAdmissionAfterHandler,
releaseChatAdmissionWhenDone,
} from "@/shared/middleware/chatBodyAdmission";
@@ -99,7 +100,9 @@ export async function POST(request) {
// Reserve heavyweight capacity atomically and ingest the body with a hard byte bound
// BEFORE JSON parsing. Missing or dishonest Content-Length values cannot bypass
// the actual-byte limit. Capacity exhaustion is retryable rather than process-fatal.
const admissionResult = await admitChatRequest(request);
const admissionResult = await admitChatRequest(request, {
queueMs: CHAT_ADMISSION_QUEUE_MAX_MS,
});
if (admissionResult.admit === false) return admissionResult.response;
const admission = admissionResult;
request = admission.request;
@@ -142,7 +145,9 @@ export async function POST(request) {
}
}
const structuralAdmission = admitChatStructure(parsedBody, admission.lease);
const structuralAdmission = await admitChatStructure(parsedBody, admission.lease, {
queueMs: CHAT_ADMISSION_QUEUE_MAX_MS,
});
if (structuralAdmission.admit === false) {
admission.lease?.release();
return finishAdmission(structuralAdmission.response);

72
src/shared/middleware/chatBodyAdmission.ts
View File

@@ -15,6 +15,11 @@ function parsePositiveInt(value: string | undefined, fallback: number): number {
return Number.isSafeInteger(parsed) && parsed > 0 ? parsed : fallback;
}
function parseNonNegativeInt(value: string | undefined, fallback: number): number {
const parsed = Number.parseInt(String(value), 10);
return Number.isSafeInteger(parsed) && parsed >= 0 ? parsed : fallback;
}
export const CHAT_LARGE_BODY_BYTES = parsePositiveInt(
process.env.OMNIROUTE_CHAT_LARGE_BODY_BYTES,
256 * 1024
@@ -30,6 +35,18 @@ const CHAT_MAX_HEAVY_IN_FLIGHT = parsePositiveInt(
1
);
/**
* How long a heavy request waits for heavyweight capacity before giving up with a
* retryable 503. Agent loops (OpenCode, Claude Code, Cursor…) fan out sub-requests
* that routinely land on the admission gate together; an immediate 503 makes the
* client burn its retry budget in seconds and the agent dies mid-task. A short
* bounded wait serializes the burst instead. `0` (legacy) rejects immediately.
*/
export const CHAT_ADMISSION_QUEUE_MAX_MS = parseNonNegativeInt(
process.env.OMNIROUTE_CHAT_ADMISSION_QUEUE_MS,
5000
);
export const CHAT_HEAVY_MESSAGE_COUNT = parsePositiveInt(
process.env.OMNIROUTE_CHAT_HEAVY_MESSAGE_COUNT,
200
@@ -71,10 +88,13 @@ export interface ChatAdmissionLease {
/**
* Process-local heavyweight reservation. The capacity check and increment execute in one
* synchronous JavaScript turn, making acquisition atomic within an OmniRoute process.
* Queueing is intentionally separate: unavailable capacity is a retryable 503.
* Unavailable capacity is a bounded wait (see `acquireHeavyWithin`) and only then a
* retryable 503, so short agent bursts serialize instead of killing the client's
* retry budget.
*/
export class ChatAdmissionController {
#activeHeavy = 0;
#waiters: Array<() => void> = [];
constructor(readonly maxHeavyInFlight = 1) {
if (!Number.isSafeInteger(maxHeavyInFlight) || maxHeavyInFlight < 1) {
@@ -98,9 +118,40 @@ export class ChatAdmissionController {
if (released) return;
released = true;
this.#activeHeavy = Math.max(0, this.#activeHeavy - 1);
this.#waiters.shift()?.();
},
};
}
/**
* Wait up to `timeoutMs` for heavyweight capacity, retrying atomically on each
* release. Resolves `null` when the deadline expires with no capacity freed, in
* which case the caller answers the retryable 503. `timeoutMs <= 0` is the
* legacy immediate-reject path. Waiters are served FIFO.
*/
async acquireHeavyWithin(timeoutMs: number): Promise<ChatAdmissionLease | null> {
const deadline = Date.now() + Math.max(0, Math.floor(timeoutMs));
for (;;) {
const lease = this.tryAcquireHeavy();
if (lease) return lease;
const remaining = deadline - Date.now();
if (remaining <= 0) return null;
let resolver: (() => void) | null = null;
const released = new Promise<void>((resolve) => {
resolver = () => resolve();
this.#waiters.push(resolver);
});
const timedOut = await Promise.race([
released.then(() => false),
new Promise<boolean>((resolve) => setTimeout(() => resolve(true), remaining)),
]);
if (resolver) {
const index = this.#waiters.indexOf(resolver);
if (index >= 0) this.#waiters.splice(index, 1);
}
if (timedOut) return null;
}
}
}
const defaultAdmissionController = new ChatAdmissionController(CHAT_MAX_HEAVY_IN_FLIGHT);
@@ -208,7 +259,7 @@ function estimateStructureTokens(value: unknown, limit: number): TokenEstimate {
return { tokens, exhausted: stack.length > 0 && tokens < limit };
}
export function admitChatStructure(
export async function admitChatStructure(
body: unknown,
lease: ChatAdmissionLease | null,
options: {
@@ -217,8 +268,9 @@ export function admitChatStructure(
heavyMessages?: number;
heavyTools?: number;
heavyTokens?: number;
queueMs?: number;
} = {}
): ChatStructureAdmission {
): Promise<ChatStructureAdmission> {
if (!body || typeof body !== "object" || Array.isArray(body)) return { admit: true, lease };
const record = body as Record<string, unknown>;
@@ -249,7 +301,9 @@ export function admitChatStructure(
estimatedTokens >= heavyTokens;
if (!heavy || lease) return { admit: true, lease };
const acquired = (options.controller ?? defaultAdmissionController).tryAcquireHeavy();
const acquired = await (options.controller ?? defaultAdmissionController).acquireHeavyWithin(
options.queueMs ?? 0
);
return acquired
? { admit: true, lease: acquired }
: { admit: false, response: structuralRejectionResponse(503, maxMessages) };
@@ -361,11 +415,13 @@ export async function admitChatRequest(
controller?: ChatAdmissionController;
largeBodyBytes?: number;
hardMaxBytes?: number;
queueMs?: number;
} = {}
): Promise<ChatRequestAdmission> {
const controller = options.controller ?? defaultAdmissionController;
const largeBodyBytes = options.largeBodyBytes ?? CHAT_LARGE_BODY_BYTES;
const hardMaxBytes = options.hardMaxBytes ?? CHAT_HARD_MAX_BODY_BYTES;
const queueMs = options.queueMs ?? 0;
const internalBypass = isInternalAdmissionBypass(request);
const contentLength = parseContentLength(request.headers.get("content-length"));
@@ -411,15 +467,15 @@ export async function admitChatRequest(
}
let lease: ChatAdmissionLease | null = null;
const reserve = (): boolean => {
const reserve = async (): Promise<boolean> => {
if (lease) return true;
lease = controller.tryAcquireHeavy();
lease = await controller.acquireHeavyWithin(queueMs);
return lease !== null;
};
// A known-large declaration can reserve before ingestion. Unknown lengths are boundedly
// sniffed below; this avoids consuming scarce heavyweight capacity for small chunked bodies.
if (contentLength !== null && contentLength >= largeBodyBytes && !reserve()) {
if (contentLength !== null && contentLength >= largeBodyBytes && !(await reserve())) {
return { admit: false, response: rejectionResponse(503, hardMaxBytes) };
}
@@ -438,7 +494,7 @@ export async function admitChatRequest(
lease?.release();
return { admit: false, response: rejectionResponse(413, hardMaxBytes) };
}
if (totalBytes >= largeBodyBytes && !reserve()) {
if (totalBytes >= largeBodyBytes && !(await reserve())) {
await reader.cancel("chat admission capacity unavailable").catch(() => undefined);
return { admit: false, response: rejectionResponse(503, hardMaxBytes) };
}

205
tests/unit/chat-body-admission.test.ts
View File

@@ -60,7 +60,7 @@ test("small known body is admitted without consuming heavyweight capacity", asyn
test("a byte-light request above the message threshold acquires heavyweight capacity", async () => {
const controller = new ChatAdmissionController(1);
const result = admitChatStructure(
const result = await admitChatStructure(
{
messages: [
{ role: "user", content: "one" },
@@ -82,7 +82,7 @@ test("a byte-light request above the tool threshold is rejected when heavy capac
const occupied = controller.tryAcquireHeavy();
assert.ok(occupied);
const result = admitChatStructure(
const result = await admitChatStructure(
{ messages: [], tools: [{ type: "function" }, { type: "function" }] },
null,
{ controller, maxMessages: 10, heavyMessages: 10, heavyTools: 2, heavyTokens: 10_000 }
@@ -98,7 +98,7 @@ test("a byte-light request above the tool threshold is rejected when heavy capac
test("an opt-in history cap still returns the structured compact-required 413", async () => {
const controller = new ChatAdmissionController(1);
const result = admitChatStructure(
const result = await admitChatStructure(
{ messages: Array.from({ length: 3 }, () => ({ role: "user", content: "x" })) },
null,
{ controller, maxMessages: 2, heavyMessages: 1, heavyTools: 10, heavyTokens: 10_000 }
@@ -120,7 +120,7 @@ test("no history cap is enforced by default; long conversations are admitted", a
assert.equal(CHAT_HARD_MAX_MESSAGES, 0, "the shipped default must not cap history");
const controller = new ChatAdmissionController(1);
const result = admitChatStructure(
const result = await admitChatStructure(
{ messages: Array.from({ length: 5_000 }, () => ({ role: "user", content: "x" })) },
null,
{ controller, heavyMessages: 200, heavyTools: 64, heavyTokens: 32_000 }
@@ -137,7 +137,7 @@ test("an uncapped oversized conversation still yields to occupied heavyweight ca
const occupied = controller.tryAcquireHeavy();
assert.ok(occupied);
const result = admitChatStructure(
const result = await admitChatStructure(
{ messages: Array.from({ length: 5_000 }, () => ({ role: "user", content: "x" })) },
null,
{ controller, maxMessages: 0, heavyMessages: 200, heavyTools: 64, heavyTokens: 32_000 }
@@ -153,9 +153,9 @@ test("an uncapped oversized conversation still yields to occupied heavyweight ca
occupied.release();
});
test("maxMessages: 0 explicitly disables the history cap", () => {
test("maxMessages: 0 explicitly disables the history cap", async () => {
const controller = new ChatAdmissionController(1);
const result = admitChatStructure(
const result = await admitChatStructure(
{ messages: Array.from({ length: 3 }, () => ({ role: "user", content: "x" })) },
null,
{ controller, maxMessages: 0, heavyMessages: 1, heavyTools: 10, heavyTokens: 10_000 }
@@ -165,9 +165,9 @@ test("maxMessages: 0 explicitly disables the history cap", () => {
if (result.admit) result.lease?.release();
});
test("a conservative token estimate classifies string messages and tool schemas as heavy", () => {
test("a conservative token estimate classifies string messages and tool schemas as heavy", async () => {
const controller = new ChatAdmissionController(1);
const result = admitChatStructure(
const result = await admitChatStructure(
{
messages: [{ role: "user", content: "abcdefgh" }],
tools: [{ type: "function", function: { name: "tool", description: "abcdefgh" } }],
@@ -181,9 +181,9 @@ test("a conservative token estimate classifies string messages and tool schemas
if (result.admit) result.lease?.release();
});
test("exhausting the bounded structural inspection is conservatively heavyweight", () => {
test("exhausting the bounded structural inspection is conservatively heavyweight", async () => {
const controller = new ChatAdmissionController(1);
const result = admitChatStructure(
const result = await admitChatStructure(
{
messages: [
{
@@ -201,12 +201,12 @@ test("exhausting the bounded structural inspection is conservatively heavyweight
if (result.admit) result.lease?.release();
});
test("tool-schema property names contribute to the conservative token estimate", () => {
test("tool-schema property names contribute to the conservative token estimate", async () => {
const controller = new ChatAdmissionController(1);
const properties = Object.fromEntries(
Array.from({ length: 5 }, (_, index) => [`${index}${"k".repeat(99)}`, {}])
);
const result = admitChatStructure(
const result = await admitChatStructure(
{ messages: [], tools: [{ function: { parameters: { properties } } }] },
null,
{ controller, maxMessages: 10, heavyMessages: 10, heavyTools: 10, heavyTokens: 100 }
@@ -217,9 +217,9 @@ test("tool-schema property names contribute to the conservative token estimate",
if (result.admit) result.lease?.release();
});
test("non-ASCII strings use a conservative UTF-8 token estimate", () => {
test("non-ASCII strings use a conservative UTF-8 token estimate", async () => {
const controller = new ChatAdmissionController(1);
const result = admitChatStructure(
const result = await admitChatStructure(
{ messages: [{ role: "user", content: "漢".repeat(100) }] },
null,
{ controller, maxMessages: 10, heavyMessages: 10, heavyTools: 10, heavyTokens: 100 }
@@ -230,10 +230,10 @@ test("non-ASCII strings use a conservative UTF-8 token estimate", () => {
if (result.admit) result.lease?.release();
});
test("wide objects exhaust bounded inspection without materializing all property values", () => {
test("wide objects exhaust bounded inspection without materializing all property values", async () => {
const controller = new ChatAdmissionController(1);
const wide = Object.fromEntries(Array.from({ length: 10_001 }, (_, index) => [`k${index}`, 0]));
const result = admitChatStructure({ messages: [{ role: "user", content: wide }] }, null, {
const result = await admitChatStructure({ messages: [{ role: "user", content: wide }] }, null, {
controller,
maxMessages: 10,
heavyMessages: 10,
@@ -246,12 +246,12 @@ test("wide objects exhaust bounded inspection without materializing all property
if (result.admit) result.lease?.release();
});
test("an existing byte-heavy lease is reused for structure-heavy admission", () => {
test("an existing byte-heavy lease is reused for structure-heavy admission", async () => {
const controller = new ChatAdmissionController(1);
const lease = controller.tryAcquireHeavy();
assert.ok(lease);
const result = admitChatStructure(
const result = await admitChatStructure(
{
messages: [
{ role: "user", content: "one" },
@@ -637,7 +637,7 @@ test("bypass describe call passes the structural stage while the parent holds th
// Structural stage (the route's admitChatStructure(parsedBody, admission.lease)):
// the sentinel lease must prevent the heavy body from re-acquiring → no 503.
const structural = admitChatStructure(JSON.parse(body), admission.lease, { controller });
const structural = await admitChatStructure(JSON.parse(body), admission.lease, { controller });
assert.equal(structural.admit, true);
assert.equal(controller.activeHeavy, 1);
@@ -813,3 +813,168 @@ test("sk_omniroute sentinel is rejected once an env key is configured (REQUIRE_A
restore();
}
});
test("a heavy structural request waits for capacity instead of failing immediately", async () => {
const controller = new ChatAdmissionController(1);
const held = controller.tryAcquireHeavy();
assert.ok(held);
const pending = admitChatStructure(
{
messages: [
{ role: "user", content: "one" },
{ role: "user", content: "two" },
],
},
null,
{
controller,
maxMessages: 10,
heavyMessages: 2,
heavyTools: 10,
heavyTokens: 10_000,
queueMs: 500,
}
);
// Capacity is still busy: the request must not have resolved (admit/reject) yet.
let settled = false;
void pending.then(() => {
settled = true;
});
await new Promise((resolve) => setTimeout(resolve, 30));
assert.equal(settled, false, "must wait while capacity is busy");
held.release();
const result = await pending;
assert.equal(result.admit, true);
if (result.admit) {
assert.equal(controller.activeHeavy, 1, "waiting request acquires the freed lease");
result.lease?.release();
}
assert.equal(controller.activeHeavy, 0);
});
test("waiting for admission times out into a retryable 503", async () => {
const controller = new ChatAdmissionController(1);
const held = controller.tryAcquireHeavy();
assert.ok(held);
const started = Date.now();
const result = await admitChatStructure(
{
messages: [
{ role: "user", content: "one" },
{ role: "user", content: "two" },
],
},
null,
{
controller,
maxMessages: 10,
heavyMessages: 2,
heavyTools: 10,
heavyTokens: 10_000,
queueMs: 50,
}
);
assert.equal(result.admit, false);
if (!result.admit) {
assert.equal(result.response.status, 503);
assert.equal(result.response.headers.get("retry-after"), "1");
assert.equal((await result.response.json()).error.code, "chat_admission_busy");
}
assert.ok(Date.now() - started >= 40, "must wait for the queue deadline before rejecting");
assert.equal(controller.activeHeavy, 1, "the holder keeps its lease");
held.release();
assert.equal(controller.activeHeavy, 0);
});
test("byte-heavy admission waits for capacity when queueMs is set", async () => {
const controller = new ChatAdmissionController(1);
const body = JSON.stringify({ messages: [{ role: "user", content: "x".repeat(40) }] });
const options = { controller, largeBodyBytes: 32, hardMaxBytes: 1024, queueMs: 500 };
const first = await admitChatRequest(chatRequest(body), options);
assert.equal(first.admit, true);
if (!first.admit) return;
const second = admitChatRequest(chatRequest(body), options);
let secondSettled = false;
void second.then(() => {
secondSettled = true;
});
await new Promise((resolve) => setTimeout(resolve, 30));
assert.equal(
secondSettled,
false,
"second heavy request must queue while the first holds capacity"
);
first.lease?.release();
const secondResult = await second;
assert.equal(secondResult.admit, true, "second request acquires capacity after release");
if (secondResult.admit) secondResult.lease?.release();
assert.equal(controller.activeHeavy, 0);
});
test("expired admission queue keeps the legacy immediate 503 behaviour", async () => {
const controller = new ChatAdmissionController(1);
const held = controller.tryAcquireHeavy();
assert.ok(held);
const result = await admitChatStructure(
{
messages: [
{ role: "user", content: "one" },
{ role: "user", content: "two" },
],
},
null,
{
controller,
maxMessages: 10,
heavyMessages: 2,
heavyTools: 10,
heavyTokens: 10_000,
queueMs: 0,
}
);
assert.equal(result.admit, false);
if (!result.admit) assert.equal(result.response.status, 503);
held.release();
});
test("admission waiters are served FIFO as capacity frees", async () => {
const controller = new ChatAdmissionController(1);
const held = controller.tryAcquireHeavy();
assert.ok(held);
const body = {
messages: [
{ role: "user", content: "one" },
{ role: "user", content: "two" },
],
};
const options = {
controller,
maxMessages: 10,
heavyMessages: 2,
heavyTools: 10,
heavyTokens: 10_000,
queueMs: 500,
};
const first = admitChatStructure(body, null, options);
const second = admitChatStructure(body, null, options);
held.release();
const firstResult = await first;
assert.equal(firstResult.admit, true);
if (firstResult.admit) firstResult.lease?.release();
const secondResult = await second;
assert.equal(secondResult.admit, true);
if (secondResult.admit) secondResult.lease?.release();
assert.equal(controller.activeHeavy, 0);
});