Files
OmniRoute/open-sse/services/admission/controller.ts
Brandon Bennett 6615a5445b feat: combo-lane awareness + activation UX + MCP visibility (Wave 2 of #9654) (#10039)
* feat(admission): per-target lane-aware probes for combo/fusion fan-out (#9654 Wave 2)

Combo and fusion fan out N targets without ever consulting the adaptive-admission
layer: the parent request holds one lease, but each fan-out target is dispatched
unconditionally. With virtual lanes enabled (OMNIROUTE_CHAT_VIRTUAL_LANES=1), a
connection whose lane queue is full now SKIPS additional fan-out targets instead
of piling more queued work onto an already-congested session.

Adds PerTargetAdmissionHook (admission/types.ts) + createPerTargetAdmissionHook
factory (chatAdmission.ts): strictly non-blocking (maxWaitMs 0 - skip, never
queue), a no-op when virtual lanes are off, keyed to the parent tenantKey, and
release-on-admit so the probe is a capacity gate, not a hold.

Threaded through every parallel fan-out path:
- priority/weighted executeTarget + round-robin skip chains (combo.ts)
- fusion panel before fan-out (fusion.ts), judge fallback prefers survivors
- chaos parallel panel (autoCombo/chaosEngine.ts)
- tryFusionDispatch / tryRuntimeUnitDispatch / buildBaseOptions (dispatchPrelude.ts)
- chat.ts primary + safety-net redirect call sites

Snapshot exposes virtualLanes so the no-op gate is cheap and honest.

Tests: tests/unit/combo-lane-awareness-9654.test.ts (10 tests) - factory
semantics, priority/RR skip, fusion panel drop + all-skipped 503, no-hook
backward-compat baseline.

* feat(flags): activation UX - env-wins adaptive virtual-lanes flag + env docs (#9654 Wave 2)

U7: make adaptive virtual admission lanes discoverable + activatable.
- New OMNIROUTE_CHAT_VIRTUAL_LANES feature flag (boolean/runtime/requiresRestart) in featureFlagDefinitions + en.json i18n key.
- lib/admissionVirtualLanes.ts: env-wins resolver (env > DB > default) + boot warm folding a DB-sourced override into the process-global runtime env via reloadAdaptiveAdmissionRuntime(options.env) - no process.env mutation, no open-sse changes. Env still wins; DB toggle gates at next boot.
- GET /api/settings/feature-flags special-cases the flag to report the gate true source (ccDiscoveryAliases precedent); flagPayload helper dedupes the payload shape.
- Wire the warm into instrumentation-node registerNodejs (non-fatal, DB-ready).
- Document the master switch in .env.example + ENVIRONMENT.md with the system-1/system-2 distinction; zero new env-doc-sync drift.
- 11 new tests (resolver precedence + warm); 60/60 across feature-flag suites; typecheck core clean; ESLint + doc gates green.

* feat(mcp): surface adaptive admission lane data in omniroute_get_health (#9654 Wave 2)

U8: make adaptive virtual-lane admission visible to agents via the MCP health tool. handleGetHealth now surfaces a curated adaptiveAdmission block from the health payload (which already carried the runtime snapshot but was dropping it): virtualLanes/pressure/utilization/laneCount/laneQueuedCount/laneQueuedCost, laneTenants capped at top-10 by queued cost, admitted/rejected/wouldReject counts, shutdown. Block omitted entirely when the health endpoint reports none.

isLaneFlagOn mirrors the runtime 1|true convention so a string serialization can never invert a boolean lane report. getHealthOutput schema extended with the matching optional shape; tool description updated.

4 new dispatch tests (full block, top-10 cap/order, omission, defensive coercion of string flags + malformed lane entries) - 22/22 in essentialTools.test.ts. README: Adaptive Admission Lane Data table + Skills & Tool Navigability audit (29/43 schema entries covered, 14 undocumented, tool_search keyword runtime discovery, full catalog in docs/frameworks/MCP-SERVER.md).

No new lint errors (4 pre-existing in server.ts), typecheck core clean, doc counts + fabricated-docs gates green.

* docs: add changelog entry for #9654 Wave 2 (#10039)

* fix(codeql): suppress js/insufficient-password-hash false positive in lane-key fingerprinting (#10039)

resolveSessionId sha256-hashes bearer/x-api-key/x-goog-api-key to derive a deterministic, non-reversible per-key lane-bucket ID for virtual admission lanes (#9654). This is not password storage or verification, so the rule is a false positive; suppress it inline (same house style as src/lib/sync/tokens.ts) to clear the codeqlAlerts ratchet (2 > baseline 1) that blocks #10039 and every PR against release/v3.8.50.

* docs(mcp): complete MCP server README tool reference (#10039)

The MCP server README covered only 29 of the 43 schema entries, listing the
remaining tools solely as a gap note with omniroute_tool_search as the runtime
fallback. Add tool-reference tables for the agent-skills trio, oneproxy trio,
web_fetch/web_search, tool_search, create_combo, set_routing_strategy,
pick_fastest_model, sync_pricing, and db_health_check so the README covers the
full schemas catalog, and fold the coverage note into the tool_search discovery
paragraph.

* fix(chat): drop unused correlationId from safety-net combo redirect (#10039)

handleComboChat's HandleComboChatOptions has no correlationId member and
the combo pipeline never consumes it; the property was copied from the
handleSingleModelChat options shape by accident and introduced a new
TS2353 under the open-sse workspace typecheck gate.

* fix(i18n): translate featureFlagChatVirtualLanesEnabledDescription into 42 locales (#10039)

en.json gained the flag description in this PR but the locale catalogs
were never mirrored, failing the pt-BR key-parity (#6695) and vi
completeness gates. Adds a real translation to every locale, keeping the
zh-CN/zh-TW glossary canonical terms (提供者/儀表板) and no ICU drift.

* chore(quality): ratchet open-sse-typecheck baseline down (#10039)

The Wave 2 admission refactor removed 66 baselined open-sse type errors;
re-freeze the baseline so the gate pins the new, tighter state.

* docs: resync provider reference to 341 and CLI tools to 34

The release branch gained an 11th no-auth provider (freeaiapikey registry
resync, #10233) and a 26th CLI Code tool without regenerating the
auto-generated docs, leaving every PR against release/v3.8.50 failing the
Docs Gates strict validator (code 341 vs doc 340, CLI 34 vs "33 tools").

Regenerate docs/reference/PROVIDER_REFERENCE.md and sync the provider/tool
counts across README.md, AGENTS.md, llm.txt plus 42 i18n mirrors,
package.json description, and the four diagram SVGs.

* fix(tests): align count expectations with live catalogs (pre-existing release drift)

Release/v3.8.50 currently fails five gates on its own tree; this PR inherits
them. Fix the stale expectations to match live code:

- feature-flags-settings: 48 -> 49 flags (Wave 2 adds OMNIROUTE_CHAT_VIRTUAL_LANES)
- cli-tools-schema / cli-catalog-counts: 33 -> 34 tools (zcode added; 26 code = 21 visible + 5 none)
- optional-transformers-dependency: onnxruntime-node ~1.24.3 -> ~1.27.0 (bump #10382)
- stryker.conf.json: register chatcore-header-drop-warn-dedupe-10315 test
- check-public-creds: freeze zcodeProtocol clientId false positive (client identifier, not a credential)

* fix(tests): follow release's onnxruntime-node revert to ~1.24.3

release/v3.8.50's #10543 pinned onnxruntime-node back to ~1.24.3 after
#10403's ~1.27.0 bump caused npm to nest a second native copy under
@huggingface/transformers and broke the Docker SONAME contract. This
PR's own drift-alignment commit (57b9c033) predates that revert and
still expected ~1.27.0; the 3-way merge did not flag it as a textual
conflict since only one side touched this exact line, but the merged
tree became internally inconsistent (package.json ~1.24.3 vs test
expecting ~1.27.0). Align the test with the now-canonical release
value.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* fix(quality): dedupe stryker.conf.json chatcore-header-drop-warn-dedupe entry

The 3-way merge applied both sides' insertion of the same test-file entry
at different positions, producing a duplicate with broken indentation.
Adopted release's clean version of the file.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Co-authored-by: Brandon Bennett <brandonbennett@macbookair.myfiosgateway.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Co-authored-by: Brandon Bennett <branben@users.noreply.github.com>
2026-08-18 11:31:46 -03:00

893 lines
30 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import {
clampLimit,
closeAdaptationWindow,
createAdaptationState,
noteLatency,
noteOutcome,
sampleActiveIntegral,
setPressure,
type AdaptationState,
} from "./adaptation.ts";
import { validateConfig, type ValidatedConfig } from "./config.ts";
import { estimateAdmissionCost, normalizeRequestCost } from "./cost.ts";
import { FairCostQueue, type QueueEntry } from "./queue.ts";
import {
MAX_ADMISSION_WINDOW_MS,
createAdmissionRejectError,
type AdaptiveAdmissionConfig,
type AdmissionAcquireResult,
type AdmissionAdmitted,
type AdmissionClock,
type AdmissionLease,
type AdmissionPressure,
type AdmissionRejectCode,
type AdmissionReleaseMeta,
type AdmissionReleaseOutcome,
type AdmissionRequest,
type AdmissionSnapshot,
type ShadowDecision,
} from "./types.ts";
/**
* Idle TTL for per-tenant virtual admission lanes (#9654).
*/
const ADMISSION_LANE_TTL_MS = 60_000;
/** Bounded per-tenant lane map to prevent unbounded memory growth (#9654). */
const ADMISSION_LANE_MAX_SESSIONS = 1_000;
type VirtualDisposition = "active" | "queued" | "rejected" | "none";
const MAX_SAFE_BIGINT = BigInt(Number.MAX_SAFE_INTEGER);
/** Snapshot numbers are always finite safe integers; never emit rounded unsafe Number values. */
function saturateSnapshotNumber(value: number): number {
if (!Number.isFinite(value) || value <= 0) return 0;
if (value >= Number.MAX_SAFE_INTEGER) return Number.MAX_SAFE_INTEGER;
return Math.floor(value);
}
function bigintToSnapshotNumber(value: bigint): number {
if (value <= 0n) return 0;
if (value >= MAX_SAFE_BIGINT) return Number.MAX_SAFE_INTEGER;
return Number(value);
}
function addSaturated(total: number, delta: number): number {
if (delta <= 0) return saturateSnapshotNumber(total);
if (total >= Number.MAX_SAFE_INTEGER - delta) return Number.MAX_SAFE_INTEGER;
return total + delta;
}
interface ActiveLeaseRecord {
id: string;
cost: number;
released: boolean;
admittedAtMs: number;
virtualDisposition: VirtualDisposition;
}
interface QueuedPayload {
resolve: (value: AdmissionAdmitted) => void;
reject: (err: Error) => void;
signal?: AbortSignal;
onAbort?: () => void;
}
let leaseSeq = 0;
function nextId(prefix: string): string {
leaseSeq += 1;
return `${prefix}-${leaseSeq}`;
}
function defaultClock(): AdmissionClock {
return {
now: () => Date.now(),
setTimer: (fn, delayMs) => {
const handle = setTimeout(fn, delayMs);
// Window/deadline timers must not pin the event loop open when idle.
if (typeof handle.unref === "function") handle.unref();
return handle;
},
clearTimer: (id) => clearTimeout(id as ReturnType<typeof setTimeout>),
};
}
/**
* Dependency-injected weighted adaptive admission controller.
* Pure in-process core: no env/settings/route wiring.
*/
export class AdaptiveAdmissionController {
private config: ValidatedConfig;
private readonly clock: AdmissionClock;
private adaptation: AdaptationState;
private queue: FairCostQueue<QueuedPayload>;
private virtualQueue: FairCostQueue<{ recordId: string }>;
/** Per-tenant virtual admission lanes (#9654). */
private readonly virtualLanes = new Map<
string,
{
queue: FairCostQueue<QueuedPayload>;
lastUsedMs: number;
}
>();
/** Eviction timer for idle lanes; re-armed when a lane is created. */
private laneEvictionTimer: unknown = undefined;
private readonly active = new Map<string, ActiveLeaseRecord>();
private activeCost = 0n;
private virtualActiveCost = 0;
private virtualActiveCount = 0;
private lastSampleMs: number;
private windowTimer: unknown = undefined;
private shutDown = false;
private admittedCount = 0;
private rejectedCount = 0;
private wouldAdmitCount = 0;
private wouldQueueCount = 0;
private wouldRejectCount = 0;
constructor(config: AdaptiveAdmissionConfig, clock?: Partial<AdmissionClock>) {
this.config = validateConfig(config);
this.clock = {
now: clock?.now ?? defaultClock().now,
setTimer: clock?.setTimer ?? defaultClock().setTimer,
clearTimer: clock?.clearTimer ?? defaultClock().clearTimer,
};
const now = this.clock.now();
this.adaptation = createAdaptationState(
this.config.initialLimit,
this.config.minLimit,
this.config.maxLimit,
now
);
this.queue = new FairCostQueue(this.config.maxQueueCount, this.config.maxQueueCost);
this.virtualQueue = new FairCostQueue(this.config.maxQueueCount, this.config.maxQueueCost);
this.lastSampleMs = now;
this.armWindowTimer();
}
updateConfig(config: AdaptiveAdmissionConfig): void {
const next = validateConfig(config);
this.sampleIntegral();
this.config = next;
this.adaptation.currentLimit = Math.min(
next.maxLimit,
Math.max(next.minLimit, this.adaptation.currentLimit)
);
// #10111: the idle-recovery ceiling must track a new initialLimit (and the
// possibly-also-new min/maxLimit) instead of staying pinned to the value computed
// at construction time — otherwise a raised initialLimit can never recover past the
// stale ceiling, and a lowered one leaves the ceiling above the new maxLimit.
this.adaptation.recoveryCeiling = clampLimit(next.initialLimit, next.minLimit, next.maxLimit);
this.adaptation.windowStartMs = this.clock.now();
this.adaptation.windowActiveCostIntegral = 0;
this.adaptation.windowCompleted = 0;
this.adaptation.windowLatencySamples = 0;
this.adaptation.freezeGrowth = false;
this.adaptation.criticalDecreaseConsumed = false;
this.adaptation.pressure = "normal";
this.lastSampleMs = this.clock.now();
const drained = this.queue.drain();
this.queue = new FairCostQueue(next.maxQueueCount, next.maxQueueCost);
// Drain per-tenant virtual lane queues (#9654).
for (const [, lane] of this.virtualLanes) {
for (const entry of lane.queue.drain()) {
drained.push(entry);
}
}
this.virtualLanes.clear();
this.clearLaneEviction();
for (const entry of drained) {
if (next.mode !== "enforce") {
this.clearEntryTimer(entry);
this.detachAbort(entry);
entry.payload.resolve(this.admit(entry.cost));
continue;
}
// Cost above the new enforce limit must fail closed immediately, never strand until deadline.
if (entry.cost > this.adaptation.currentLimit) {
this.failQueued(
entry,
"ADMISSION_OVERSIZED",
"request cost exceeds max budget after config update"
);
continue;
}
if (!this.queue.enqueue(entry)) {
this.failQueued(entry, "ADMISSION_QUEUE_FULL", "queue capacity reduced");
}
}
this.rebuildVirtualState(next.mode === "shadow");
this.armWindowTimer();
if (next.mode === "enforce") {
this.dispatch();
}
}
snapshot(): AdmissionSnapshot {
this.sampleIntegral();
return {
mode: this.config.mode,
currentLimit: this.adaptation.currentLimit,
minLimit: this.config.minLimit,
maxLimit: this.config.maxLimit,
activeCost: bigintToSnapshotNumber(this.activeCost),
activeCount: saturateSnapshotNumber(this.active.size),
queuedCost: saturateSnapshotNumber(this.queue.totalCost),
queuedCount: saturateSnapshotNumber(this.queue.size),
virtualActiveCost: saturateSnapshotNumber(this.virtualActiveCost),
virtualActiveCount: saturateSnapshotNumber(this.virtualActiveCount),
virtualQueuedCost: saturateSnapshotNumber(this.virtualQueue.totalCost),
virtualQueuedCount: saturateSnapshotNumber(this.virtualQueue.size),
virtualLanes: this.config.virtualLanes === true,
laneCount: saturateSnapshotNumber(this.virtualLanes.size),
laneQueuedCost: saturateSnapshotNumber(this.laneTotalQueuedCost()),
laneQueuedCount: saturateSnapshotNumber(this.laneTotalQueuedCount()),
laneTenants: this.laneTenantSnapshot(),
admittedCount: saturateSnapshotNumber(this.admittedCount),
rejectedCount: saturateSnapshotNumber(this.rejectedCount),
wouldAdmitCount: saturateSnapshotNumber(this.wouldAdmitCount),
wouldQueueCount: saturateSnapshotNumber(this.wouldQueueCount),
wouldRejectCount: saturateSnapshotNumber(this.wouldRejectCount),
shortLatencyEwma: this.adaptation.shortLatencyEwma,
longLatencyEwma: this.adaptation.longLatencyEwma,
utilization: this.adaptation.utilization,
pressure: this.adaptation.pressure,
shutdown: this.shutDown,
};
}
observePressure(pressure: AdmissionPressure): void {
setPressure(this.adaptation, pressure);
if (pressure === "critical") {
// Immediate fast decrease once per window; window close must not re-apply it.
if (!this.adaptation.criticalDecreaseConsumed) {
this.adaptation.currentLimit = Math.max(
this.config.minLimit,
Math.floor(this.adaptation.currentLimit * this.config.adaptation.criticalDecreaseFactor)
);
this.adaptation.criticalDecreaseConsumed = true;
this.dispatch();
this.dispatchVirtual();
}
}
}
/** Deterministic window tick for tests / injected clocks. */
tick(): void {
this.sampleIntegral();
this.evictIdleLanes();
closeAdaptationWindow(this.adaptation, this.config.adaptation, this.clock.now());
// Real queue first, then virtual: raised limits must promote shadow-queued work
// before newer arrivals are classified against the updated budget.
this.dispatch();
this.dispatchVirtual();
}
async acquire(request: AdmissionRequest): Promise<AdmissionAcquireResult> {
if (this.shutDown) {
return this.reject("ADMISSION_SHUTDOWN", "admission controller is shut down");
}
if (request.signal?.aborted) {
return this.reject("ADMISSION_ABORTED", "request aborted before acquire");
}
if (request.pressure) setPressure(this.adaptation, request.pressure);
const cost = this.resolveCost(request);
const mode = this.config.mode;
if (mode === "off") {
return this.admitVirtual(cost);
}
const limit = this.adaptation.currentLimit;
if (mode === "shadow") {
return this.acquireShadow(request, cost, limit);
}
// enforce
if (cost > limit) {
// #10111 solo-progress: the adaptive aggregate limit can collapse below an
// individually-valid request (a slow-provider turn shrinks currentLimit via the
// latency gradient, and no increase can fire because every path to "completed"
// requires an admission). A request within the healthy aggregate ceiling must never
// be terminally rejected as oversized while the system is otherwise idle — admit a
// single bounded solo request so the pipeline keeps making progress and the limit can
// recover. The hard per-request ceiling (maxLimit), the critical/high pressure fuse,
// and a busy system (active/queued work present) all take precedence over solo.
if (this.shouldAdmitSolo(cost)) {
return this.admit(cost);
}
return this.reject("ADMISSION_OVERSIZED", "request cost exceeds max budget");
}
// Once work is queued, every newer request joins the same fair queue even if it
// currently fits. This makes bounded bypass accounting effective and prevents
// direct arrivals from indefinitely jumping an older reserved weighted request.
if (this.queue.size === 0 && this.activeCost + BigInt(cost) <= BigInt(limit)) {
return this.admit(cost);
}
if (!this.queue.canAccept(cost)) {
return this.reject("ADMISSION_QUEUE_FULL", "admission queue is full");
}
return this.enqueue(request, cost);
}
shutdown(): void {
if (this.shutDown) return;
this.shutDown = true;
if (this.windowTimer !== undefined) {
this.clock.clearTimer(this.windowTimer);
this.windowTimer = undefined;
}
const drained = this.queue.drain();
for (const entry of drained) {
this.clearEntryTimer(entry);
this.detachAbort(entry);
entry.payload.reject(
createAdmissionRejectError("ADMISSION_SHUTDOWN", "admission controller shut down")
);
this.rejectedCount += 1;
}
// Drain per-tenant virtual lane queues (#9654).
for (const [, lane] of this.virtualLanes) {
for (const entry of lane.queue.drain()) {
this.clearEntryTimer(entry);
this.detachAbort(entry);
entry.payload.reject(
createAdmissionRejectError("ADMISSION_SHUTDOWN", "admission controller shut down")
);
this.rejectedCount += 1;
}
}
this.virtualLanes.clear();
this.clearLaneEviction();
}
/**
* #10111: whether a request that currently exceeds the temporary aggregate limit may run
* solo. True only when the request fits the healthy aggregate ceiling (maxLimit), the
* system is otherwise idle (no active/queued/lane work) and pressure is normal — so an
* individually-valid request is not terminally rejected as oversized just because a
* latency-gradient decrease collapsed the temporary limit. Under genuine load, critical
* pressure, or an over-ceiling request the caller falls through to the terminal reject.
*/
private shouldAdmitSolo(cost: number): boolean {
return (
cost <= this.config.maxLimit &&
this.active.size === 0 &&
this.queue.size === 0 &&
this.laneTotalQueuedCount() === 0 &&
this.adaptation.pressure === "normal"
);
}
private resolveCost(request: AdmissionRequest): number {
if (request.cost !== undefined) {
return normalizeRequestCost(request.cost, this.config.maxRequestCost);
}
if (request.features) {
return estimateAdmissionCost(request.features, this.config.costConfig);
}
return 1;
}
private acquireShadow(request: AdmissionRequest, cost: number, limit: number): AdmissionAdmitted {
let decision: ShadowDecision;
let disposition: VirtualDisposition;
if (cost > limit || !Number.isSafeInteger(cost)) {
decision = "would-reject";
disposition = "rejected";
this.wouldRejectCount += 1;
} else if (this.virtualActiveCost + cost <= limit) {
decision = "would-admit";
disposition = "active";
this.virtualActiveCost = addSaturated(this.virtualActiveCost, cost);
this.virtualActiveCount = addSaturated(this.virtualActiveCount, 1);
this.wouldAdmitCount = addSaturated(this.wouldAdmitCount, 1);
} else if (this.virtualQueue.canAccept(cost)) {
decision = "would-queue";
disposition = "queued";
this.wouldQueueCount += 1;
} else {
decision = "would-reject";
disposition = "rejected";
this.wouldRejectCount += 1;
}
const admitted = this.admit(cost, disposition);
if (disposition === "queued") {
this.virtualQueue.enqueue({
id: admitted.lease.id,
tenantKey: request.tenantKey || "_default",
cost,
enqueuedAtMs: this.clock.now(),
deadlineMs: Number.MAX_SAFE_INTEGER,
payload: { recordId: admitted.lease.id },
});
}
return { ...admitted, shadowDecision: decision };
}
private admitVirtual(cost: number): AdmissionAdmitted {
// Mode off: no accounting.
const id = nextId("lease");
const lease: AdmissionLease = {
id,
cost,
get released() {
return true;
},
release: () => {
/* no-op */
},
};
this.admittedCount += 1;
return { status: "admitted", lease };
}
private admit(cost: number, virtualDisposition: VirtualDisposition = "none"): AdmissionAdmitted {
this.sampleIntegral();
const id = nextId("lease");
const record: ActiveLeaseRecord = {
id,
cost,
released: false,
admittedAtMs: this.clock.now(),
virtualDisposition,
};
this.active.set(id, record);
this.activeCost += BigInt(cost);
this.admittedCount += 1;
const controller = this;
const lease: AdmissionLease = {
id,
cost,
get released() {
return record.released;
},
release(outcome: AdmissionReleaseOutcome = "success", meta?: AdmissionReleaseMeta) {
controller.releaseLease(record, outcome, meta);
},
};
return { status: "admitted", lease };
}
private releaseLease(
record: ActiveLeaseRecord,
outcome: AdmissionReleaseOutcome,
meta?: AdmissionReleaseMeta
): void {
if (record.released) return;
record.released = true;
// Sample while the lease still contributes to activeCost so utilization EWMA sees load.
this.sampleIntegral();
if (this.active.has(record.id)) {
this.active.delete(record.id);
this.activeCost -= BigInt(record.cost);
}
const latency =
meta?.latencyMs !== undefined
? meta.latencyMs
: Math.max(0, this.clock.now() - record.admittedAtMs);
noteLatency(this.adaptation, latency, this.config.adaptation);
noteOutcome(this.adaptation, outcome);
this.adaptation.windowCompleted += 1;
if (meta?.pressure) setPressure(this.adaptation, meta.pressure);
this.releaseVirtual(record);
this.dispatch();
}
private enqueue(request: AdmissionRequest, cost: number): AdmissionAcquireResult {
const id = nextId("q");
const maxWait = normalizeRequestCost(
request.maxWaitMs ?? this.config.defaultMaxWaitMs,
MAX_ADMISSION_WINDOW_MS
);
const now = this.clock.now();
const deadlineMs = Math.min(Number.MAX_SAFE_INTEGER, now + maxWait);
let settle: {
resolve: (v: AdmissionAdmitted) => void;
reject: (e: Error) => void;
};
const promise = new Promise<AdmissionAdmitted>((resolve, reject) => {
settle = { resolve, reject };
});
const entry: QueueEntry<QueuedPayload> = {
id,
tenantKey: request.tenantKey && request.tenantKey.length > 0 ? request.tenantKey : "_default",
cost,
enqueuedAtMs: now,
deadlineMs,
payload: {
resolve: (v) => settle.resolve(v),
reject: (e) => settle.reject(e),
signal: request.signal,
},
};
// Per-tenant virtual admission lanes (#9654): when enabled via
// OMNIROUTE_CHAT_VIRTUAL_LANES=1, requests with a tenantKey are enqueued into
// a per-tenant lane queue instead of the shared queue, so one tenant's
// burst does not 503 other sessions. Lanes are bounded by
// ADMISSION_LANE_MAX_SESSIONS and idle-evicted after ADMISSION_LANE_TTL_MS.
// Default: OFF — preserves the shared FairCostQueue round-robin behavior.
if (entry.tenantKey !== "_default" && this.config.virtualLanes) {
const lane = this.getOrCreateLane(entry.tenantKey);
if (!lane.queue.enqueue(entry)) {
this.removeEmptyLane(entry.tenantKey);
return this.reject("ADMISSION_QUEUE_FULL", "admission lane queue is full");
}
this.armLaneEviction();
} else if (!this.queue.enqueue(entry)) {
return this.reject("ADMISSION_QUEUE_FULL", "admission queue is full");
}
this.dispatch();
entry.timerId = this.clock.setTimer(
() => {
this.expireEntry(id, "ADMISSION_DEADLINE", "admission wait deadline exceeded");
},
Math.max(0, deadlineMs - now)
);
if (request.signal) {
const onAbort = () => {
this.expireEntry(id, "ADMISSION_ABORTED", "request aborted while queued");
};
entry.payload.onAbort = onAbort;
request.signal.addEventListener("abort", onAbort, { once: true });
}
// Capacity may have freed between check and enqueue in concurrent hosts; try dispatch.
this.dispatch();
return { status: "queued", promise };
}
private expireEntry(id: string, code: AdmissionRejectCode, message: string): void {
let entry = this.queue.removeById(id);
if (!entry) {
// Search per-tenant lane queues (#9654).
for (const [, lane] of this.virtualLanes) {
entry = lane.queue.removeById(id);
if (entry) {
this.removeEmptyLane(entry.tenantKey);
break;
}
}
}
if (!entry) return;
this.clearEntryTimer(entry);
this.detachAbort(entry);
entry.payload.reject(createAdmissionRejectError(code, message));
this.rejectedCount += 1;
// Resume enforce dispatch so a now-fitting successor is not stranded until
// unrelated activity. dispatch() is a no-op after shutdown / non-enforce.
this.dispatch();
}
private failQueued(
entry: QueueEntry<QueuedPayload>,
code: AdmissionRejectCode,
message: string
): void {
this.clearEntryTimer(entry);
this.detachAbort(entry);
entry.payload.reject(createAdmissionRejectError(code, message));
this.rejectedCount += 1;
}
private dispatch(): void {
if (this.shutDown || this.config.mode !== "enforce") return;
while (this.queue.size > 0) {
const limit = this.adaptation.currentLimit;
const available = BigInt(limit) - this.activeCost;
if (available <= 0n) return;
const entry = this.queue.dequeue(Number(available));
if (!entry) return;
this.clearEntryTimer(entry);
this.detachAbort(entry);
if (entry.payload.signal?.aborted) {
entry.payload.reject(
createAdmissionRejectError("ADMISSION_ABORTED", "request aborted while queued")
);
this.rejectedCount += 1;
continue;
}
if (this.clock.now() >= entry.deadlineMs) {
entry.payload.reject(
createAdmissionRejectError("ADMISSION_DEADLINE", "admission wait deadline exceeded")
);
this.rejectedCount += 1;
continue;
}
entry.payload.resolve(this.admit(entry.cost));
}
this.dispatchLanes();
}
/** Round-robin dispatch across per-tenant virtual lane queues (#9654). */
private dispatchLanes(): void {
if (this.shutDown || this.config.mode !== "enforce") return;
if (this.virtualLanes.size === 0) return;
const keys = Array.from(this.virtualLanes.keys());
for (const key of keys) {
const lane = this.virtualLanes.get(key);
if (!lane) continue;
// Dispatch as many entries from this lane as capacity allows,
// then break to give other lanes a fair share.
while (lane.queue.size > 0) {
const limit = this.adaptation.currentLimit;
const available = BigInt(limit) - this.activeCost;
if (available <= 0n) return;
const entry = lane.queue.dequeue(Number(available));
if (!entry) break; // head doesn't fit
this.clearEntryTimer(entry);
this.detachAbort(entry);
if (entry.payload.signal?.aborted) {
entry.payload.reject(
createAdmissionRejectError("ADMISSION_ABORTED", "request aborted while queued")
);
this.rejectedCount += 1;
continue;
}
if (this.clock.now() >= entry.deadlineMs) {
entry.payload.reject(
createAdmissionRejectError("ADMISSION_DEADLINE", "admission wait deadline exceeded")
);
this.rejectedCount += 1;
continue;
}
entry.payload.resolve(this.admit(entry.cost));
break; // yield to next lane for fairness
}
this.removeEmptyLane(key);
}
}
private getOrCreateLane(tenantKey: string): {
queue: FairCostQueue<QueuedPayload>;
lastUsedMs: number;
} {
let lane = this.virtualLanes.get(tenantKey);
if (!lane) {
// Evict oldest lane if at capacity (LRU).
if (this.virtualLanes.size >= ADMISSION_LANE_MAX_SESSIONS) {
const oldestKey = this.oldestLaneKey();
if (oldestKey) {
this.deleteLane(oldestKey);
}
}
// Per-lane queue uses the same maxQueueCount/maxQueueCost as the shared
// queue. Total memory is bounded by ADMISSION_LANE_MAX_SESSIONS (1000)
// × per-lane queue caps — each lane's FairCostQueue rejects when full.
lane = {
queue: new FairCostQueue(this.config.maxQueueCount, this.config.maxQueueCost),
lastUsedMs: this.clock.now(),
};
this.virtualLanes.set(tenantKey, lane);
}
lane.lastUsedMs = this.clock.now();
return lane;
}
private removeEmptyLane(tenantKey: string): void {
const lane = this.virtualLanes.get(tenantKey);
if (lane && lane.queue.size === 0) {
this.virtualLanes.delete(tenantKey);
}
}
/** Drain and reject all pending entries in a lane before removing it from the map. */
private deleteLane(tenantKey: string): void {
const lane = this.virtualLanes.get(tenantKey);
if (!lane) return;
for (const entry of lane.queue.drain()) {
this.clearEntryTimer(entry);
this.detachAbort(entry);
entry.payload.reject(
createAdmissionRejectError("ADMISSION_LANE_EVICTED", "connection lane evicted")
);
this.rejectedCount += 1;
}
this.virtualLanes.delete(tenantKey);
}
private oldestLaneKey(): string | undefined {
let oldest: string | undefined;
let oldestMs = Infinity;
for (const [key, lane] of this.virtualLanes) {
if (lane.lastUsedMs <= oldestMs) {
oldestMs = lane.lastUsedMs;
oldest = key;
}
}
return oldest;
}
private evictIdleLanes(): void {
const now = this.clock.now();
const keysToDelete: string[] = [];
for (const [key, lane] of this.virtualLanes) {
if (now - lane.lastUsedMs >= ADMISSION_LANE_TTL_MS) {
keysToDelete.push(key);
}
}
for (const key of keysToDelete) {
this.deleteLane(key);
}
if (this.virtualLanes.size > 0) {
this.armLaneEviction();
} else {
this.clearLaneEviction();
}
}
private armLaneEviction(): void {
this.clearLaneEviction();
this.laneEvictionTimer = this.clock.setTimer(
() => this.evictIdleLanes(),
ADMISSION_LANE_TTL_MS
);
}
private clearLaneEviction(): void {
if (this.laneEvictionTimer !== undefined) {
this.clock.clearTimer(this.laneEvictionTimer);
this.laneEvictionTimer = undefined;
}
}
private laneTotalQueuedCost(): number {
let total = 0;
for (const [, lane] of this.virtualLanes) {
total = addSaturated(total, lane.queue.totalCost);
}
return total;
}
private laneTotalQueuedCount(): number {
let count = 0;
for (const [, lane] of this.virtualLanes) {
count = addSaturated(count, lane.queue.size);
}
return count;
}
private laneTenantSnapshot(): ReadonlyArray<{
tenantKey: string;
queuedCount: number;
queuedCost: number;
}> {
const arr: { tenantKey: string; queuedCount: number; queuedCost: number }[] = [];
for (const [tenantKey, lane] of this.virtualLanes) {
arr.push({
tenantKey,
queuedCount: saturateSnapshotNumber(lane.queue.size),
queuedCost: saturateSnapshotNumber(lane.queue.totalCost),
});
}
return arr;
}
private releaseVirtual(record: ActiveLeaseRecord): void {
if (record.virtualDisposition === "active") {
this.virtualActiveCost -= record.cost;
this.virtualActiveCount -= 1;
} else if (record.virtualDisposition === "queued") {
this.virtualQueue.removeById(record.id);
}
record.virtualDisposition = "none";
this.dispatchVirtual();
}
private dispatchVirtual(): void {
while (this.virtualQueue.size > 0) {
const available = this.adaptation.currentLimit - this.virtualActiveCost;
if (available <= 0) return;
const entry = this.virtualQueue.dequeue(available);
if (!entry) return;
const record = this.active.get(entry.payload.recordId);
if (!record || record.released) continue;
record.virtualDisposition = "active";
this.virtualActiveCost = addSaturated(this.virtualActiveCost, record.cost);
this.virtualActiveCount = addSaturated(this.virtualActiveCount, 1);
}
}
private rebuildVirtualState(enable: boolean): void {
this.virtualQueue = new FairCostQueue(this.config.maxQueueCount, this.config.maxQueueCost);
this.virtualActiveCost = 0;
this.virtualActiveCount = 0;
for (const record of this.active.values()) record.virtualDisposition = "none";
if (!enable) return;
for (const record of this.active.values()) {
// Individually oversized work is virtual-rejected, never virtually queued.
if (record.cost > this.adaptation.currentLimit) {
record.virtualDisposition = "rejected";
continue;
}
if (record.cost <= this.adaptation.currentLimit - this.virtualActiveCost) {
record.virtualDisposition = "active";
this.virtualActiveCost = addSaturated(this.virtualActiveCost, record.cost);
this.virtualActiveCount = addSaturated(this.virtualActiveCount, 1);
} else if (
this.virtualQueue.enqueue({
id: record.id,
tenantKey: "_existing",
cost: record.cost,
enqueuedAtMs: record.admittedAtMs,
deadlineMs: Number.MAX_SAFE_INTEGER,
payload: { recordId: record.id },
})
) {
record.virtualDisposition = "queued";
} else {
record.virtualDisposition = "rejected";
}
}
}
private reject(code: AdmissionRejectCode, message: string): AdmissionAcquireResult {
this.rejectedCount += 1;
return { status: "rejected", code, message };
}
private clearEntryTimer(entry: QueueEntry<QueuedPayload>): void {
if (entry.timerId !== undefined) {
this.clock.clearTimer(entry.timerId);
entry.timerId = undefined;
}
}
private detachAbort(entry: QueueEntry<QueuedPayload>): void {
if (entry.payload.signal && entry.payload.onAbort) {
entry.payload.signal.removeEventListener("abort", entry.payload.onAbort);
entry.payload.onAbort = undefined;
}
}
private sampleIntegral(): void {
const now = this.clock.now();
const dt = now - this.lastSampleMs;
if (dt > 0) {
// Cap at currentLimit before Number conversion so shadow oversubscription never
// feeds an unsafe rounded activeCost into the utilization integral.
const limit = this.adaptation.currentLimit;
const activeForIntegral = this.activeCost >= BigInt(limit) ? limit : Number(this.activeCost);
sampleActiveIntegral(this.adaptation, activeForIntegral, dt);
this.lastSampleMs = now;
}
}
private armWindowTimer(): void {
if (this.windowTimer !== undefined) {
this.clock.clearTimer(this.windowTimer);
this.windowTimer = undefined;
}
if (this.shutDown || this.config.mode === "off") return;
const tick = () => {
this.tick();
if (!this.shutDown && this.config.mode !== "off") {
this.windowTimer = this.clock.setTimer(tick, this.config.windowMs);
}
};
this.windowTimer = this.clock.setTimer(tick, this.config.windowMs);
}
}