mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-08-26 00:52:18 +03:00
* feat(admission): per-target lane-aware probes for combo/fusion fan-out (#9654 Wave 2)
Combo and fusion fan out N targets without ever consulting the adaptive-admission
layer: the parent request holds one lease, but each fan-out target is dispatched
unconditionally. With virtual lanes enabled (OMNIROUTE_CHAT_VIRTUAL_LANES=1), a
connection whose lane queue is full now SKIPS additional fan-out targets instead
of piling more queued work onto an already-congested session.
Adds PerTargetAdmissionHook (admission/types.ts) + createPerTargetAdmissionHook
factory (chatAdmission.ts): strictly non-blocking (maxWaitMs 0 - skip, never
queue), a no-op when virtual lanes are off, keyed to the parent tenantKey, and
release-on-admit so the probe is a capacity gate, not a hold.
Threaded through every parallel fan-out path:
- priority/weighted executeTarget + round-robin skip chains (combo.ts)
- fusion panel before fan-out (fusion.ts), judge fallback prefers survivors
- chaos parallel panel (autoCombo/chaosEngine.ts)
- tryFusionDispatch / tryRuntimeUnitDispatch / buildBaseOptions (dispatchPrelude.ts)
- chat.ts primary + safety-net redirect call sites
Snapshot exposes virtualLanes so the no-op gate is cheap and honest.
Tests: tests/unit/combo-lane-awareness-9654.test.ts (10 tests) - factory
semantics, priority/RR skip, fusion panel drop + all-skipped 503, no-hook
backward-compat baseline.
* feat(flags): activation UX - env-wins adaptive virtual-lanes flag + env docs (#9654 Wave 2)
U7: make adaptive virtual admission lanes discoverable + activatable.
- New OMNIROUTE_CHAT_VIRTUAL_LANES feature flag (boolean/runtime/requiresRestart) in featureFlagDefinitions + en.json i18n key.
- lib/admissionVirtualLanes.ts: env-wins resolver (env > DB > default) + boot warm folding a DB-sourced override into the process-global runtime env via reloadAdaptiveAdmissionRuntime(options.env) - no process.env mutation, no open-sse changes. Env still wins; DB toggle gates at next boot.
- GET /api/settings/feature-flags special-cases the flag to report the gate true source (ccDiscoveryAliases precedent); flagPayload helper dedupes the payload shape.
- Wire the warm into instrumentation-node registerNodejs (non-fatal, DB-ready).
- Document the master switch in .env.example + ENVIRONMENT.md with the system-1/system-2 distinction; zero new env-doc-sync drift.
- 11 new tests (resolver precedence + warm); 60/60 across feature-flag suites; typecheck core clean; ESLint + doc gates green.
* feat(mcp): surface adaptive admission lane data in omniroute_get_health (#9654 Wave 2)
U8: make adaptive virtual-lane admission visible to agents via the MCP health tool. handleGetHealth now surfaces a curated adaptiveAdmission block from the health payload (which already carried the runtime snapshot but was dropping it): virtualLanes/pressure/utilization/laneCount/laneQueuedCount/laneQueuedCost, laneTenants capped at top-10 by queued cost, admitted/rejected/wouldReject counts, shutdown. Block omitted entirely when the health endpoint reports none.
isLaneFlagOn mirrors the runtime 1|true convention so a string serialization can never invert a boolean lane report. getHealthOutput schema extended with the matching optional shape; tool description updated.
4 new dispatch tests (full block, top-10 cap/order, omission, defensive coercion of string flags + malformed lane entries) - 22/22 in essentialTools.test.ts. README: Adaptive Admission Lane Data table + Skills & Tool Navigability audit (29/43 schema entries covered, 14 undocumented, tool_search keyword runtime discovery, full catalog in docs/frameworks/MCP-SERVER.md).
No new lint errors (4 pre-existing in server.ts), typecheck core clean, doc counts + fabricated-docs gates green.
* docs: add changelog entry for #9654 Wave 2 (#10039)
* fix(codeql): suppress js/insufficient-password-hash false positive in lane-key fingerprinting (#10039)
resolveSessionId sha256-hashes bearer/x-api-key/x-goog-api-key to derive a deterministic, non-reversible per-key lane-bucket ID for virtual admission lanes (#9654). This is not password storage or verification, so the rule is a false positive; suppress it inline (same house style as src/lib/sync/tokens.ts) to clear the codeqlAlerts ratchet (2 > baseline 1) that blocks #10039 and every PR against release/v3.8.50.
* docs(mcp): complete MCP server README tool reference (#10039)
The MCP server README covered only 29 of the 43 schema entries, listing the
remaining tools solely as a gap note with omniroute_tool_search as the runtime
fallback. Add tool-reference tables for the agent-skills trio, oneproxy trio,
web_fetch/web_search, tool_search, create_combo, set_routing_strategy,
pick_fastest_model, sync_pricing, and db_health_check so the README covers the
full schemas catalog, and fold the coverage note into the tool_search discovery
paragraph.
* fix(chat): drop unused correlationId from safety-net combo redirect (#10039)
handleComboChat's HandleComboChatOptions has no correlationId member and
the combo pipeline never consumes it; the property was copied from the
handleSingleModelChat options shape by accident and introduced a new
TS2353 under the open-sse workspace typecheck gate.
* fix(i18n): translate featureFlagChatVirtualLanesEnabledDescription into 42 locales (#10039)
en.json gained the flag description in this PR but the locale catalogs
were never mirrored, failing the pt-BR key-parity (#6695) and vi
completeness gates. Adds a real translation to every locale, keeping the
zh-CN/zh-TW glossary canonical terms (提供者/儀表板) and no ICU drift.
* chore(quality): ratchet open-sse-typecheck baseline down (#10039)
The Wave 2 admission refactor removed 66 baselined open-sse type errors;
re-freeze the baseline so the gate pins the new, tighter state.
* docs: resync provider reference to 341 and CLI tools to 34
The release branch gained an 11th no-auth provider (freeaiapikey registry
resync, #10233) and a 26th CLI Code tool without regenerating the
auto-generated docs, leaving every PR against release/v3.8.50 failing the
Docs Gates strict validator (code 341 vs doc 340, CLI 34 vs "33 tools").
Regenerate docs/reference/PROVIDER_REFERENCE.md and sync the provider/tool
counts across README.md, AGENTS.md, llm.txt plus 42 i18n mirrors,
package.json description, and the four diagram SVGs.
* fix(tests): align count expectations with live catalogs (pre-existing release drift)
Release/v3.8.50 currently fails five gates on its own tree; this PR inherits
them. Fix the stale expectations to match live code:
- feature-flags-settings: 48 -> 49 flags (Wave 2 adds OMNIROUTE_CHAT_VIRTUAL_LANES)
- cli-tools-schema / cli-catalog-counts: 33 -> 34 tools (zcode added; 26 code = 21 visible + 5 none)
- optional-transformers-dependency: onnxruntime-node ~1.24.3 -> ~1.27.0 (bump #10382)
- stryker.conf.json: register chatcore-header-drop-warn-dedupe-10315 test
- check-public-creds: freeze zcodeProtocol clientId false positive (client identifier, not a credential)
* fix(tests): follow release's onnxruntime-node revert to ~1.24.3
release/v3.8.50's #10543 pinned onnxruntime-node back to ~1.24.3 after
#10403's ~1.27.0 bump caused npm to nest a second native copy under
@huggingface/transformers and broke the Docker SONAME contract. This
PR's own drift-alignment commit (57b9c033) predates that revert and
still expected ~1.27.0; the 3-way merge did not flag it as a textual
conflict since only one side touched this exact line, but the merged
tree became internally inconsistent (package.json ~1.24.3 vs test
expecting ~1.27.0). Align the test with the now-canonical release
value.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
* fix(quality): dedupe stryker.conf.json chatcore-header-drop-warn-dedupe entry
The 3-way merge applied both sides' insertion of the same test-file entry
at different positions, producing a duplicate with broken indentation.
Adopted release's clean version of the file.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: Brandon Bennett <brandonbennett@macbookair.myfiosgateway.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Co-authored-by: Brandon Bennett <branben@users.noreply.github.com>
893 lines
30 KiB
TypeScript
893 lines
30 KiB
TypeScript
import {
|
||
clampLimit,
|
||
closeAdaptationWindow,
|
||
createAdaptationState,
|
||
noteLatency,
|
||
noteOutcome,
|
||
sampleActiveIntegral,
|
||
setPressure,
|
||
type AdaptationState,
|
||
} from "./adaptation.ts";
|
||
import { validateConfig, type ValidatedConfig } from "./config.ts";
|
||
import { estimateAdmissionCost, normalizeRequestCost } from "./cost.ts";
|
||
import { FairCostQueue, type QueueEntry } from "./queue.ts";
|
||
import {
|
||
MAX_ADMISSION_WINDOW_MS,
|
||
createAdmissionRejectError,
|
||
type AdaptiveAdmissionConfig,
|
||
type AdmissionAcquireResult,
|
||
type AdmissionAdmitted,
|
||
type AdmissionClock,
|
||
type AdmissionLease,
|
||
type AdmissionPressure,
|
||
type AdmissionRejectCode,
|
||
type AdmissionReleaseMeta,
|
||
type AdmissionReleaseOutcome,
|
||
type AdmissionRequest,
|
||
type AdmissionSnapshot,
|
||
type ShadowDecision,
|
||
} from "./types.ts";
|
||
|
||
/**
|
||
* Idle TTL for per-tenant virtual admission lanes (#9654).
|
||
*/
|
||
const ADMISSION_LANE_TTL_MS = 60_000;
|
||
/** Bounded per-tenant lane map to prevent unbounded memory growth (#9654). */
|
||
const ADMISSION_LANE_MAX_SESSIONS = 1_000;
|
||
|
||
type VirtualDisposition = "active" | "queued" | "rejected" | "none";
|
||
|
||
const MAX_SAFE_BIGINT = BigInt(Number.MAX_SAFE_INTEGER);
|
||
|
||
/** Snapshot numbers are always finite safe integers; never emit rounded unsafe Number values. */
|
||
function saturateSnapshotNumber(value: number): number {
|
||
if (!Number.isFinite(value) || value <= 0) return 0;
|
||
if (value >= Number.MAX_SAFE_INTEGER) return Number.MAX_SAFE_INTEGER;
|
||
return Math.floor(value);
|
||
}
|
||
|
||
function bigintToSnapshotNumber(value: bigint): number {
|
||
if (value <= 0n) return 0;
|
||
if (value >= MAX_SAFE_BIGINT) return Number.MAX_SAFE_INTEGER;
|
||
return Number(value);
|
||
}
|
||
|
||
function addSaturated(total: number, delta: number): number {
|
||
if (delta <= 0) return saturateSnapshotNumber(total);
|
||
if (total >= Number.MAX_SAFE_INTEGER - delta) return Number.MAX_SAFE_INTEGER;
|
||
return total + delta;
|
||
}
|
||
|
||
interface ActiveLeaseRecord {
|
||
id: string;
|
||
cost: number;
|
||
released: boolean;
|
||
admittedAtMs: number;
|
||
virtualDisposition: VirtualDisposition;
|
||
}
|
||
|
||
interface QueuedPayload {
|
||
resolve: (value: AdmissionAdmitted) => void;
|
||
reject: (err: Error) => void;
|
||
signal?: AbortSignal;
|
||
onAbort?: () => void;
|
||
}
|
||
|
||
let leaseSeq = 0;
|
||
|
||
function nextId(prefix: string): string {
|
||
leaseSeq += 1;
|
||
return `${prefix}-${leaseSeq}`;
|
||
}
|
||
|
||
function defaultClock(): AdmissionClock {
|
||
return {
|
||
now: () => Date.now(),
|
||
setTimer: (fn, delayMs) => {
|
||
const handle = setTimeout(fn, delayMs);
|
||
// Window/deadline timers must not pin the event loop open when idle.
|
||
if (typeof handle.unref === "function") handle.unref();
|
||
return handle;
|
||
},
|
||
clearTimer: (id) => clearTimeout(id as ReturnType<typeof setTimeout>),
|
||
};
|
||
}
|
||
|
||
/**
|
||
* Dependency-injected weighted adaptive admission controller.
|
||
* Pure in-process core: no env/settings/route wiring.
|
||
*/
|
||
export class AdaptiveAdmissionController {
|
||
private config: ValidatedConfig;
|
||
private readonly clock: AdmissionClock;
|
||
private adaptation: AdaptationState;
|
||
private queue: FairCostQueue<QueuedPayload>;
|
||
private virtualQueue: FairCostQueue<{ recordId: string }>;
|
||
/** Per-tenant virtual admission lanes (#9654). */
|
||
private readonly virtualLanes = new Map<
|
||
string,
|
||
{
|
||
queue: FairCostQueue<QueuedPayload>;
|
||
lastUsedMs: number;
|
||
}
|
||
>();
|
||
/** Eviction timer for idle lanes; re-armed when a lane is created. */
|
||
private laneEvictionTimer: unknown = undefined;
|
||
private readonly active = new Map<string, ActiveLeaseRecord>();
|
||
private activeCost = 0n;
|
||
private virtualActiveCost = 0;
|
||
private virtualActiveCount = 0;
|
||
private lastSampleMs: number;
|
||
private windowTimer: unknown = undefined;
|
||
private shutDown = false;
|
||
|
||
private admittedCount = 0;
|
||
private rejectedCount = 0;
|
||
private wouldAdmitCount = 0;
|
||
private wouldQueueCount = 0;
|
||
private wouldRejectCount = 0;
|
||
|
||
constructor(config: AdaptiveAdmissionConfig, clock?: Partial<AdmissionClock>) {
|
||
this.config = validateConfig(config);
|
||
this.clock = {
|
||
now: clock?.now ?? defaultClock().now,
|
||
setTimer: clock?.setTimer ?? defaultClock().setTimer,
|
||
clearTimer: clock?.clearTimer ?? defaultClock().clearTimer,
|
||
};
|
||
const now = this.clock.now();
|
||
this.adaptation = createAdaptationState(
|
||
this.config.initialLimit,
|
||
this.config.minLimit,
|
||
this.config.maxLimit,
|
||
now
|
||
);
|
||
this.queue = new FairCostQueue(this.config.maxQueueCount, this.config.maxQueueCost);
|
||
this.virtualQueue = new FairCostQueue(this.config.maxQueueCount, this.config.maxQueueCost);
|
||
this.lastSampleMs = now;
|
||
this.armWindowTimer();
|
||
}
|
||
|
||
updateConfig(config: AdaptiveAdmissionConfig): void {
|
||
const next = validateConfig(config);
|
||
this.sampleIntegral();
|
||
this.config = next;
|
||
this.adaptation.currentLimit = Math.min(
|
||
next.maxLimit,
|
||
Math.max(next.minLimit, this.adaptation.currentLimit)
|
||
);
|
||
// #10111: the idle-recovery ceiling must track a new initialLimit (and the
|
||
// possibly-also-new min/maxLimit) instead of staying pinned to the value computed
|
||
// at construction time — otherwise a raised initialLimit can never recover past the
|
||
// stale ceiling, and a lowered one leaves the ceiling above the new maxLimit.
|
||
this.adaptation.recoveryCeiling = clampLimit(next.initialLimit, next.minLimit, next.maxLimit);
|
||
this.adaptation.windowStartMs = this.clock.now();
|
||
this.adaptation.windowActiveCostIntegral = 0;
|
||
this.adaptation.windowCompleted = 0;
|
||
this.adaptation.windowLatencySamples = 0;
|
||
this.adaptation.freezeGrowth = false;
|
||
this.adaptation.criticalDecreaseConsumed = false;
|
||
this.adaptation.pressure = "normal";
|
||
this.lastSampleMs = this.clock.now();
|
||
|
||
const drained = this.queue.drain();
|
||
this.queue = new FairCostQueue(next.maxQueueCount, next.maxQueueCost);
|
||
// Drain per-tenant virtual lane queues (#9654).
|
||
for (const [, lane] of this.virtualLanes) {
|
||
for (const entry of lane.queue.drain()) {
|
||
drained.push(entry);
|
||
}
|
||
}
|
||
this.virtualLanes.clear();
|
||
this.clearLaneEviction();
|
||
for (const entry of drained) {
|
||
if (next.mode !== "enforce") {
|
||
this.clearEntryTimer(entry);
|
||
this.detachAbort(entry);
|
||
entry.payload.resolve(this.admit(entry.cost));
|
||
continue;
|
||
}
|
||
// Cost above the new enforce limit must fail closed immediately, never strand until deadline.
|
||
if (entry.cost > this.adaptation.currentLimit) {
|
||
this.failQueued(
|
||
entry,
|
||
"ADMISSION_OVERSIZED",
|
||
"request cost exceeds max budget after config update"
|
||
);
|
||
continue;
|
||
}
|
||
if (!this.queue.enqueue(entry)) {
|
||
this.failQueued(entry, "ADMISSION_QUEUE_FULL", "queue capacity reduced");
|
||
}
|
||
}
|
||
|
||
this.rebuildVirtualState(next.mode === "shadow");
|
||
this.armWindowTimer();
|
||
if (next.mode === "enforce") {
|
||
this.dispatch();
|
||
}
|
||
}
|
||
|
||
snapshot(): AdmissionSnapshot {
|
||
this.sampleIntegral();
|
||
return {
|
||
mode: this.config.mode,
|
||
currentLimit: this.adaptation.currentLimit,
|
||
minLimit: this.config.minLimit,
|
||
maxLimit: this.config.maxLimit,
|
||
activeCost: bigintToSnapshotNumber(this.activeCost),
|
||
activeCount: saturateSnapshotNumber(this.active.size),
|
||
queuedCost: saturateSnapshotNumber(this.queue.totalCost),
|
||
queuedCount: saturateSnapshotNumber(this.queue.size),
|
||
virtualActiveCost: saturateSnapshotNumber(this.virtualActiveCost),
|
||
virtualActiveCount: saturateSnapshotNumber(this.virtualActiveCount),
|
||
virtualQueuedCost: saturateSnapshotNumber(this.virtualQueue.totalCost),
|
||
virtualQueuedCount: saturateSnapshotNumber(this.virtualQueue.size),
|
||
virtualLanes: this.config.virtualLanes === true,
|
||
laneCount: saturateSnapshotNumber(this.virtualLanes.size),
|
||
laneQueuedCost: saturateSnapshotNumber(this.laneTotalQueuedCost()),
|
||
laneQueuedCount: saturateSnapshotNumber(this.laneTotalQueuedCount()),
|
||
laneTenants: this.laneTenantSnapshot(),
|
||
admittedCount: saturateSnapshotNumber(this.admittedCount),
|
||
rejectedCount: saturateSnapshotNumber(this.rejectedCount),
|
||
wouldAdmitCount: saturateSnapshotNumber(this.wouldAdmitCount),
|
||
wouldQueueCount: saturateSnapshotNumber(this.wouldQueueCount),
|
||
wouldRejectCount: saturateSnapshotNumber(this.wouldRejectCount),
|
||
shortLatencyEwma: this.adaptation.shortLatencyEwma,
|
||
longLatencyEwma: this.adaptation.longLatencyEwma,
|
||
utilization: this.adaptation.utilization,
|
||
pressure: this.adaptation.pressure,
|
||
shutdown: this.shutDown,
|
||
};
|
||
}
|
||
|
||
observePressure(pressure: AdmissionPressure): void {
|
||
setPressure(this.adaptation, pressure);
|
||
if (pressure === "critical") {
|
||
// Immediate fast decrease once per window; window close must not re-apply it.
|
||
if (!this.adaptation.criticalDecreaseConsumed) {
|
||
this.adaptation.currentLimit = Math.max(
|
||
this.config.minLimit,
|
||
Math.floor(this.adaptation.currentLimit * this.config.adaptation.criticalDecreaseFactor)
|
||
);
|
||
this.adaptation.criticalDecreaseConsumed = true;
|
||
this.dispatch();
|
||
this.dispatchVirtual();
|
||
}
|
||
}
|
||
}
|
||
|
||
/** Deterministic window tick for tests / injected clocks. */
|
||
tick(): void {
|
||
this.sampleIntegral();
|
||
this.evictIdleLanes();
|
||
closeAdaptationWindow(this.adaptation, this.config.adaptation, this.clock.now());
|
||
// Real queue first, then virtual: raised limits must promote shadow-queued work
|
||
// before newer arrivals are classified against the updated budget.
|
||
this.dispatch();
|
||
this.dispatchVirtual();
|
||
}
|
||
|
||
async acquire(request: AdmissionRequest): Promise<AdmissionAcquireResult> {
|
||
if (this.shutDown) {
|
||
return this.reject("ADMISSION_SHUTDOWN", "admission controller is shut down");
|
||
}
|
||
|
||
if (request.signal?.aborted) {
|
||
return this.reject("ADMISSION_ABORTED", "request aborted before acquire");
|
||
}
|
||
|
||
if (request.pressure) setPressure(this.adaptation, request.pressure);
|
||
|
||
const cost = this.resolveCost(request);
|
||
const mode = this.config.mode;
|
||
|
||
if (mode === "off") {
|
||
return this.admitVirtual(cost);
|
||
}
|
||
|
||
const limit = this.adaptation.currentLimit;
|
||
|
||
if (mode === "shadow") {
|
||
return this.acquireShadow(request, cost, limit);
|
||
}
|
||
|
||
// enforce
|
||
if (cost > limit) {
|
||
// #10111 solo-progress: the adaptive aggregate limit can collapse below an
|
||
// individually-valid request (a slow-provider turn shrinks currentLimit via the
|
||
// latency gradient, and no increase can fire because every path to "completed"
|
||
// requires an admission). A request within the healthy aggregate ceiling must never
|
||
// be terminally rejected as oversized while the system is otherwise idle — admit a
|
||
// single bounded solo request so the pipeline keeps making progress and the limit can
|
||
// recover. The hard per-request ceiling (maxLimit), the critical/high pressure fuse,
|
||
// and a busy system (active/queued work present) all take precedence over solo.
|
||
if (this.shouldAdmitSolo(cost)) {
|
||
return this.admit(cost);
|
||
}
|
||
return this.reject("ADMISSION_OVERSIZED", "request cost exceeds max budget");
|
||
}
|
||
|
||
// Once work is queued, every newer request joins the same fair queue even if it
|
||
// currently fits. This makes bounded bypass accounting effective and prevents
|
||
// direct arrivals from indefinitely jumping an older reserved weighted request.
|
||
if (this.queue.size === 0 && this.activeCost + BigInt(cost) <= BigInt(limit)) {
|
||
return this.admit(cost);
|
||
}
|
||
|
||
if (!this.queue.canAccept(cost)) {
|
||
return this.reject("ADMISSION_QUEUE_FULL", "admission queue is full");
|
||
}
|
||
|
||
return this.enqueue(request, cost);
|
||
}
|
||
|
||
shutdown(): void {
|
||
if (this.shutDown) return;
|
||
this.shutDown = true;
|
||
if (this.windowTimer !== undefined) {
|
||
this.clock.clearTimer(this.windowTimer);
|
||
this.windowTimer = undefined;
|
||
}
|
||
const drained = this.queue.drain();
|
||
for (const entry of drained) {
|
||
this.clearEntryTimer(entry);
|
||
this.detachAbort(entry);
|
||
entry.payload.reject(
|
||
createAdmissionRejectError("ADMISSION_SHUTDOWN", "admission controller shut down")
|
||
);
|
||
this.rejectedCount += 1;
|
||
}
|
||
// Drain per-tenant virtual lane queues (#9654).
|
||
for (const [, lane] of this.virtualLanes) {
|
||
for (const entry of lane.queue.drain()) {
|
||
this.clearEntryTimer(entry);
|
||
this.detachAbort(entry);
|
||
entry.payload.reject(
|
||
createAdmissionRejectError("ADMISSION_SHUTDOWN", "admission controller shut down")
|
||
);
|
||
this.rejectedCount += 1;
|
||
}
|
||
}
|
||
this.virtualLanes.clear();
|
||
this.clearLaneEviction();
|
||
}
|
||
|
||
/**
|
||
* #10111: whether a request that currently exceeds the temporary aggregate limit may run
|
||
* solo. True only when the request fits the healthy aggregate ceiling (maxLimit), the
|
||
* system is otherwise idle (no active/queued/lane work) and pressure is normal — so an
|
||
* individually-valid request is not terminally rejected as oversized just because a
|
||
* latency-gradient decrease collapsed the temporary limit. Under genuine load, critical
|
||
* pressure, or an over-ceiling request the caller falls through to the terminal reject.
|
||
*/
|
||
private shouldAdmitSolo(cost: number): boolean {
|
||
return (
|
||
cost <= this.config.maxLimit &&
|
||
this.active.size === 0 &&
|
||
this.queue.size === 0 &&
|
||
this.laneTotalQueuedCount() === 0 &&
|
||
this.adaptation.pressure === "normal"
|
||
);
|
||
}
|
||
|
||
private resolveCost(request: AdmissionRequest): number {
|
||
if (request.cost !== undefined) {
|
||
return normalizeRequestCost(request.cost, this.config.maxRequestCost);
|
||
}
|
||
if (request.features) {
|
||
return estimateAdmissionCost(request.features, this.config.costConfig);
|
||
}
|
||
return 1;
|
||
}
|
||
|
||
private acquireShadow(request: AdmissionRequest, cost: number, limit: number): AdmissionAdmitted {
|
||
let decision: ShadowDecision;
|
||
let disposition: VirtualDisposition;
|
||
if (cost > limit || !Number.isSafeInteger(cost)) {
|
||
decision = "would-reject";
|
||
disposition = "rejected";
|
||
this.wouldRejectCount += 1;
|
||
} else if (this.virtualActiveCost + cost <= limit) {
|
||
decision = "would-admit";
|
||
disposition = "active";
|
||
this.virtualActiveCost = addSaturated(this.virtualActiveCost, cost);
|
||
this.virtualActiveCount = addSaturated(this.virtualActiveCount, 1);
|
||
this.wouldAdmitCount = addSaturated(this.wouldAdmitCount, 1);
|
||
} else if (this.virtualQueue.canAccept(cost)) {
|
||
decision = "would-queue";
|
||
disposition = "queued";
|
||
this.wouldQueueCount += 1;
|
||
} else {
|
||
decision = "would-reject";
|
||
disposition = "rejected";
|
||
this.wouldRejectCount += 1;
|
||
}
|
||
|
||
const admitted = this.admit(cost, disposition);
|
||
if (disposition === "queued") {
|
||
this.virtualQueue.enqueue({
|
||
id: admitted.lease.id,
|
||
tenantKey: request.tenantKey || "_default",
|
||
cost,
|
||
enqueuedAtMs: this.clock.now(),
|
||
deadlineMs: Number.MAX_SAFE_INTEGER,
|
||
payload: { recordId: admitted.lease.id },
|
||
});
|
||
}
|
||
return { ...admitted, shadowDecision: decision };
|
||
}
|
||
|
||
private admitVirtual(cost: number): AdmissionAdmitted {
|
||
// Mode off: no accounting.
|
||
const id = nextId("lease");
|
||
const lease: AdmissionLease = {
|
||
id,
|
||
cost,
|
||
get released() {
|
||
return true;
|
||
},
|
||
release: () => {
|
||
/* no-op */
|
||
},
|
||
};
|
||
this.admittedCount += 1;
|
||
return { status: "admitted", lease };
|
||
}
|
||
|
||
private admit(cost: number, virtualDisposition: VirtualDisposition = "none"): AdmissionAdmitted {
|
||
this.sampleIntegral();
|
||
const id = nextId("lease");
|
||
const record: ActiveLeaseRecord = {
|
||
id,
|
||
cost,
|
||
released: false,
|
||
admittedAtMs: this.clock.now(),
|
||
virtualDisposition,
|
||
};
|
||
this.active.set(id, record);
|
||
this.activeCost += BigInt(cost);
|
||
this.admittedCount += 1;
|
||
|
||
const controller = this;
|
||
const lease: AdmissionLease = {
|
||
id,
|
||
cost,
|
||
get released() {
|
||
return record.released;
|
||
},
|
||
release(outcome: AdmissionReleaseOutcome = "success", meta?: AdmissionReleaseMeta) {
|
||
controller.releaseLease(record, outcome, meta);
|
||
},
|
||
};
|
||
return { status: "admitted", lease };
|
||
}
|
||
|
||
private releaseLease(
|
||
record: ActiveLeaseRecord,
|
||
outcome: AdmissionReleaseOutcome,
|
||
meta?: AdmissionReleaseMeta
|
||
): void {
|
||
if (record.released) return;
|
||
record.released = true;
|
||
// Sample while the lease still contributes to activeCost so utilization EWMA sees load.
|
||
this.sampleIntegral();
|
||
if (this.active.has(record.id)) {
|
||
this.active.delete(record.id);
|
||
this.activeCost -= BigInt(record.cost);
|
||
}
|
||
|
||
const latency =
|
||
meta?.latencyMs !== undefined
|
||
? meta.latencyMs
|
||
: Math.max(0, this.clock.now() - record.admittedAtMs);
|
||
noteLatency(this.adaptation, latency, this.config.adaptation);
|
||
noteOutcome(this.adaptation, outcome);
|
||
this.adaptation.windowCompleted += 1;
|
||
if (meta?.pressure) setPressure(this.adaptation, meta.pressure);
|
||
this.releaseVirtual(record);
|
||
|
||
this.dispatch();
|
||
}
|
||
|
||
private enqueue(request: AdmissionRequest, cost: number): AdmissionAcquireResult {
|
||
const id = nextId("q");
|
||
const maxWait = normalizeRequestCost(
|
||
request.maxWaitMs ?? this.config.defaultMaxWaitMs,
|
||
MAX_ADMISSION_WINDOW_MS
|
||
);
|
||
const now = this.clock.now();
|
||
const deadlineMs = Math.min(Number.MAX_SAFE_INTEGER, now + maxWait);
|
||
|
||
let settle: {
|
||
resolve: (v: AdmissionAdmitted) => void;
|
||
reject: (e: Error) => void;
|
||
};
|
||
const promise = new Promise<AdmissionAdmitted>((resolve, reject) => {
|
||
settle = { resolve, reject };
|
||
});
|
||
|
||
const entry: QueueEntry<QueuedPayload> = {
|
||
id,
|
||
tenantKey: request.tenantKey && request.tenantKey.length > 0 ? request.tenantKey : "_default",
|
||
cost,
|
||
enqueuedAtMs: now,
|
||
deadlineMs,
|
||
payload: {
|
||
resolve: (v) => settle.resolve(v),
|
||
reject: (e) => settle.reject(e),
|
||
signal: request.signal,
|
||
},
|
||
};
|
||
|
||
// Per-tenant virtual admission lanes (#9654): when enabled via
|
||
// OMNIROUTE_CHAT_VIRTUAL_LANES=1, requests with a tenantKey are enqueued into
|
||
// a per-tenant lane queue instead of the shared queue, so one tenant's
|
||
// burst does not 503 other sessions. Lanes are bounded by
|
||
// ADMISSION_LANE_MAX_SESSIONS and idle-evicted after ADMISSION_LANE_TTL_MS.
|
||
// Default: OFF — preserves the shared FairCostQueue round-robin behavior.
|
||
if (entry.tenantKey !== "_default" && this.config.virtualLanes) {
|
||
const lane = this.getOrCreateLane(entry.tenantKey);
|
||
if (!lane.queue.enqueue(entry)) {
|
||
this.removeEmptyLane(entry.tenantKey);
|
||
return this.reject("ADMISSION_QUEUE_FULL", "admission lane queue is full");
|
||
}
|
||
this.armLaneEviction();
|
||
} else if (!this.queue.enqueue(entry)) {
|
||
return this.reject("ADMISSION_QUEUE_FULL", "admission queue is full");
|
||
}
|
||
this.dispatch();
|
||
|
||
entry.timerId = this.clock.setTimer(
|
||
() => {
|
||
this.expireEntry(id, "ADMISSION_DEADLINE", "admission wait deadline exceeded");
|
||
},
|
||
Math.max(0, deadlineMs - now)
|
||
);
|
||
|
||
if (request.signal) {
|
||
const onAbort = () => {
|
||
this.expireEntry(id, "ADMISSION_ABORTED", "request aborted while queued");
|
||
};
|
||
entry.payload.onAbort = onAbort;
|
||
request.signal.addEventListener("abort", onAbort, { once: true });
|
||
}
|
||
|
||
// Capacity may have freed between check and enqueue in concurrent hosts; try dispatch.
|
||
this.dispatch();
|
||
|
||
return { status: "queued", promise };
|
||
}
|
||
|
||
private expireEntry(id: string, code: AdmissionRejectCode, message: string): void {
|
||
let entry = this.queue.removeById(id);
|
||
if (!entry) {
|
||
// Search per-tenant lane queues (#9654).
|
||
for (const [, lane] of this.virtualLanes) {
|
||
entry = lane.queue.removeById(id);
|
||
if (entry) {
|
||
this.removeEmptyLane(entry.tenantKey);
|
||
break;
|
||
}
|
||
}
|
||
}
|
||
if (!entry) return;
|
||
this.clearEntryTimer(entry);
|
||
this.detachAbort(entry);
|
||
entry.payload.reject(createAdmissionRejectError(code, message));
|
||
this.rejectedCount += 1;
|
||
// Resume enforce dispatch so a now-fitting successor is not stranded until
|
||
// unrelated activity. dispatch() is a no-op after shutdown / non-enforce.
|
||
this.dispatch();
|
||
}
|
||
|
||
private failQueued(
|
||
entry: QueueEntry<QueuedPayload>,
|
||
code: AdmissionRejectCode,
|
||
message: string
|
||
): void {
|
||
this.clearEntryTimer(entry);
|
||
this.detachAbort(entry);
|
||
entry.payload.reject(createAdmissionRejectError(code, message));
|
||
this.rejectedCount += 1;
|
||
}
|
||
|
||
private dispatch(): void {
|
||
if (this.shutDown || this.config.mode !== "enforce") return;
|
||
while (this.queue.size > 0) {
|
||
const limit = this.adaptation.currentLimit;
|
||
const available = BigInt(limit) - this.activeCost;
|
||
if (available <= 0n) return;
|
||
const entry = this.queue.dequeue(Number(available));
|
||
if (!entry) return;
|
||
this.clearEntryTimer(entry);
|
||
this.detachAbort(entry);
|
||
if (entry.payload.signal?.aborted) {
|
||
entry.payload.reject(
|
||
createAdmissionRejectError("ADMISSION_ABORTED", "request aborted while queued")
|
||
);
|
||
this.rejectedCount += 1;
|
||
continue;
|
||
}
|
||
if (this.clock.now() >= entry.deadlineMs) {
|
||
entry.payload.reject(
|
||
createAdmissionRejectError("ADMISSION_DEADLINE", "admission wait deadline exceeded")
|
||
);
|
||
this.rejectedCount += 1;
|
||
continue;
|
||
}
|
||
entry.payload.resolve(this.admit(entry.cost));
|
||
}
|
||
this.dispatchLanes();
|
||
}
|
||
|
||
/** Round-robin dispatch across per-tenant virtual lane queues (#9654). */
|
||
private dispatchLanes(): void {
|
||
if (this.shutDown || this.config.mode !== "enforce") return;
|
||
if (this.virtualLanes.size === 0) return;
|
||
|
||
const keys = Array.from(this.virtualLanes.keys());
|
||
for (const key of keys) {
|
||
const lane = this.virtualLanes.get(key);
|
||
if (!lane) continue;
|
||
// Dispatch as many entries from this lane as capacity allows,
|
||
// then break to give other lanes a fair share.
|
||
while (lane.queue.size > 0) {
|
||
const limit = this.adaptation.currentLimit;
|
||
const available = BigInt(limit) - this.activeCost;
|
||
if (available <= 0n) return;
|
||
const entry = lane.queue.dequeue(Number(available));
|
||
if (!entry) break; // head doesn't fit
|
||
this.clearEntryTimer(entry);
|
||
this.detachAbort(entry);
|
||
if (entry.payload.signal?.aborted) {
|
||
entry.payload.reject(
|
||
createAdmissionRejectError("ADMISSION_ABORTED", "request aborted while queued")
|
||
);
|
||
this.rejectedCount += 1;
|
||
continue;
|
||
}
|
||
if (this.clock.now() >= entry.deadlineMs) {
|
||
entry.payload.reject(
|
||
createAdmissionRejectError("ADMISSION_DEADLINE", "admission wait deadline exceeded")
|
||
);
|
||
this.rejectedCount += 1;
|
||
continue;
|
||
}
|
||
entry.payload.resolve(this.admit(entry.cost));
|
||
break; // yield to next lane for fairness
|
||
}
|
||
this.removeEmptyLane(key);
|
||
}
|
||
}
|
||
|
||
private getOrCreateLane(tenantKey: string): {
|
||
queue: FairCostQueue<QueuedPayload>;
|
||
lastUsedMs: number;
|
||
} {
|
||
let lane = this.virtualLanes.get(tenantKey);
|
||
if (!lane) {
|
||
// Evict oldest lane if at capacity (LRU).
|
||
if (this.virtualLanes.size >= ADMISSION_LANE_MAX_SESSIONS) {
|
||
const oldestKey = this.oldestLaneKey();
|
||
if (oldestKey) {
|
||
this.deleteLane(oldestKey);
|
||
}
|
||
}
|
||
// Per-lane queue uses the same maxQueueCount/maxQueueCost as the shared
|
||
// queue. Total memory is bounded by ADMISSION_LANE_MAX_SESSIONS (1000)
|
||
// × per-lane queue caps — each lane's FairCostQueue rejects when full.
|
||
lane = {
|
||
queue: new FairCostQueue(this.config.maxQueueCount, this.config.maxQueueCost),
|
||
lastUsedMs: this.clock.now(),
|
||
};
|
||
this.virtualLanes.set(tenantKey, lane);
|
||
}
|
||
lane.lastUsedMs = this.clock.now();
|
||
return lane;
|
||
}
|
||
|
||
private removeEmptyLane(tenantKey: string): void {
|
||
const lane = this.virtualLanes.get(tenantKey);
|
||
if (lane && lane.queue.size === 0) {
|
||
this.virtualLanes.delete(tenantKey);
|
||
}
|
||
}
|
||
|
||
/** Drain and reject all pending entries in a lane before removing it from the map. */
|
||
private deleteLane(tenantKey: string): void {
|
||
const lane = this.virtualLanes.get(tenantKey);
|
||
if (!lane) return;
|
||
for (const entry of lane.queue.drain()) {
|
||
this.clearEntryTimer(entry);
|
||
this.detachAbort(entry);
|
||
entry.payload.reject(
|
||
createAdmissionRejectError("ADMISSION_LANE_EVICTED", "connection lane evicted")
|
||
);
|
||
this.rejectedCount += 1;
|
||
}
|
||
this.virtualLanes.delete(tenantKey);
|
||
}
|
||
|
||
private oldestLaneKey(): string | undefined {
|
||
let oldest: string | undefined;
|
||
let oldestMs = Infinity;
|
||
for (const [key, lane] of this.virtualLanes) {
|
||
if (lane.lastUsedMs <= oldestMs) {
|
||
oldestMs = lane.lastUsedMs;
|
||
oldest = key;
|
||
}
|
||
}
|
||
return oldest;
|
||
}
|
||
|
||
private evictIdleLanes(): void {
|
||
const now = this.clock.now();
|
||
const keysToDelete: string[] = [];
|
||
for (const [key, lane] of this.virtualLanes) {
|
||
if (now - lane.lastUsedMs >= ADMISSION_LANE_TTL_MS) {
|
||
keysToDelete.push(key);
|
||
}
|
||
}
|
||
for (const key of keysToDelete) {
|
||
this.deleteLane(key);
|
||
}
|
||
if (this.virtualLanes.size > 0) {
|
||
this.armLaneEviction();
|
||
} else {
|
||
this.clearLaneEviction();
|
||
}
|
||
}
|
||
|
||
private armLaneEviction(): void {
|
||
this.clearLaneEviction();
|
||
this.laneEvictionTimer = this.clock.setTimer(
|
||
() => this.evictIdleLanes(),
|
||
ADMISSION_LANE_TTL_MS
|
||
);
|
||
}
|
||
|
||
private clearLaneEviction(): void {
|
||
if (this.laneEvictionTimer !== undefined) {
|
||
this.clock.clearTimer(this.laneEvictionTimer);
|
||
this.laneEvictionTimer = undefined;
|
||
}
|
||
}
|
||
|
||
private laneTotalQueuedCost(): number {
|
||
let total = 0;
|
||
for (const [, lane] of this.virtualLanes) {
|
||
total = addSaturated(total, lane.queue.totalCost);
|
||
}
|
||
return total;
|
||
}
|
||
|
||
private laneTotalQueuedCount(): number {
|
||
let count = 0;
|
||
for (const [, lane] of this.virtualLanes) {
|
||
count = addSaturated(count, lane.queue.size);
|
||
}
|
||
return count;
|
||
}
|
||
|
||
private laneTenantSnapshot(): ReadonlyArray<{
|
||
tenantKey: string;
|
||
queuedCount: number;
|
||
queuedCost: number;
|
||
}> {
|
||
const arr: { tenantKey: string; queuedCount: number; queuedCost: number }[] = [];
|
||
for (const [tenantKey, lane] of this.virtualLanes) {
|
||
arr.push({
|
||
tenantKey,
|
||
queuedCount: saturateSnapshotNumber(lane.queue.size),
|
||
queuedCost: saturateSnapshotNumber(lane.queue.totalCost),
|
||
});
|
||
}
|
||
return arr;
|
||
}
|
||
|
||
private releaseVirtual(record: ActiveLeaseRecord): void {
|
||
if (record.virtualDisposition === "active") {
|
||
this.virtualActiveCost -= record.cost;
|
||
this.virtualActiveCount -= 1;
|
||
} else if (record.virtualDisposition === "queued") {
|
||
this.virtualQueue.removeById(record.id);
|
||
}
|
||
record.virtualDisposition = "none";
|
||
this.dispatchVirtual();
|
||
}
|
||
|
||
private dispatchVirtual(): void {
|
||
while (this.virtualQueue.size > 0) {
|
||
const available = this.adaptation.currentLimit - this.virtualActiveCost;
|
||
if (available <= 0) return;
|
||
const entry = this.virtualQueue.dequeue(available);
|
||
if (!entry) return;
|
||
const record = this.active.get(entry.payload.recordId);
|
||
if (!record || record.released) continue;
|
||
record.virtualDisposition = "active";
|
||
this.virtualActiveCost = addSaturated(this.virtualActiveCost, record.cost);
|
||
this.virtualActiveCount = addSaturated(this.virtualActiveCount, 1);
|
||
}
|
||
}
|
||
|
||
private rebuildVirtualState(enable: boolean): void {
|
||
this.virtualQueue = new FairCostQueue(this.config.maxQueueCount, this.config.maxQueueCost);
|
||
this.virtualActiveCost = 0;
|
||
this.virtualActiveCount = 0;
|
||
for (const record of this.active.values()) record.virtualDisposition = "none";
|
||
if (!enable) return;
|
||
for (const record of this.active.values()) {
|
||
// Individually oversized work is virtual-rejected, never virtually queued.
|
||
if (record.cost > this.adaptation.currentLimit) {
|
||
record.virtualDisposition = "rejected";
|
||
continue;
|
||
}
|
||
if (record.cost <= this.adaptation.currentLimit - this.virtualActiveCost) {
|
||
record.virtualDisposition = "active";
|
||
this.virtualActiveCost = addSaturated(this.virtualActiveCost, record.cost);
|
||
this.virtualActiveCount = addSaturated(this.virtualActiveCount, 1);
|
||
} else if (
|
||
this.virtualQueue.enqueue({
|
||
id: record.id,
|
||
tenantKey: "_existing",
|
||
cost: record.cost,
|
||
enqueuedAtMs: record.admittedAtMs,
|
||
deadlineMs: Number.MAX_SAFE_INTEGER,
|
||
payload: { recordId: record.id },
|
||
})
|
||
) {
|
||
record.virtualDisposition = "queued";
|
||
} else {
|
||
record.virtualDisposition = "rejected";
|
||
}
|
||
}
|
||
}
|
||
|
||
private reject(code: AdmissionRejectCode, message: string): AdmissionAcquireResult {
|
||
this.rejectedCount += 1;
|
||
return { status: "rejected", code, message };
|
||
}
|
||
|
||
private clearEntryTimer(entry: QueueEntry<QueuedPayload>): void {
|
||
if (entry.timerId !== undefined) {
|
||
this.clock.clearTimer(entry.timerId);
|
||
entry.timerId = undefined;
|
||
}
|
||
}
|
||
|
||
private detachAbort(entry: QueueEntry<QueuedPayload>): void {
|
||
if (entry.payload.signal && entry.payload.onAbort) {
|
||
entry.payload.signal.removeEventListener("abort", entry.payload.onAbort);
|
||
entry.payload.onAbort = undefined;
|
||
}
|
||
}
|
||
|
||
private sampleIntegral(): void {
|
||
const now = this.clock.now();
|
||
const dt = now - this.lastSampleMs;
|
||
if (dt > 0) {
|
||
// Cap at currentLimit before Number conversion so shadow oversubscription never
|
||
// feeds an unsafe rounded activeCost into the utilization integral.
|
||
const limit = this.adaptation.currentLimit;
|
||
const activeForIntegral = this.activeCost >= BigInt(limit) ? limit : Number(this.activeCost);
|
||
sampleActiveIntegral(this.adaptation, activeForIntegral, dt);
|
||
this.lastSampleMs = now;
|
||
}
|
||
}
|
||
|
||
private armWindowTimer(): void {
|
||
if (this.windowTimer !== undefined) {
|
||
this.clock.clearTimer(this.windowTimer);
|
||
this.windowTimer = undefined;
|
||
}
|
||
if (this.shutDown || this.config.mode === "off") return;
|
||
const tick = () => {
|
||
this.tick();
|
||
if (!this.shutDown && this.config.mode !== "off") {
|
||
this.windowTimer = this.clock.setTimer(tick, this.config.windowMs);
|
||
}
|
||
};
|
||
this.windowTimer = this.clock.setTimer(tick, this.config.windowMs);
|
||
}
|
||
}
|