fix: per-connection virtual admission lanes (#9654) (#9940)

* fix: add per-connection virtual admission lanes (#9654)

Worst-day-ever analysis to harden AdaptiveAdmissionController:

- Guard expireEntry() against null entry (CRITICAL null deref)
- Add deleteLane() to drain+reject on LRU eviction (HIGH orphaned promises)
- Fix Map mutation during evictIdleLanes iteration (MEDIUM safety)
- Add ADMISSION_LANE_EVICTED reject code (MEDIUM clarity)
- Pass sessionId to admitChatRequest in route.ts
- virtualLanes defaults to false in validateConfig
- 7 new controller tests + 14 new byte-level admission tests
- Assertions tightened from >= to === (Matt Pocock methodology)

Debunked 2 false positives: concurrency race (single-threaded JS)
and memory amplification (FairCostQueue bounds per-lane).

Fixes #9654

* fix(admission): restore bounded queue-wait on per-connection lanes (#9654)

The per-connection lane refactor dropped the bounded queue-wait
(acquireHeavyWithin / #waiters / queueMs). #9654's acceptance criteria and
#9608 section C prefer server-side wait/pacing up to defaultMaxWaitMs over
an instant retryable 503.

- ChatAdmissionController: re-add #waiters FIFO + acquireHeavyWithin(timeoutMs);
  queueMs: 0 preserves the instant-503 path
- admitChatStructure and admitChatRequest.reserve are async again and take queueMs
- route: pass CHAT_ADMISSION_QUEUE_MAX_MS and await the admission calls
- per-connection lane tests await the async admitChatStructure

Admission suite: 114/114 pass (bun test, 7 files).

* chore: re-trigger CI after dast-smoke infra cancellation (#9654)

* feat(admission): cancel queue-wait on client abort (#9654)

U2 from KC plan 2026-08-09-001. Thread the request AbortSignal through
acquireHeavyWithin so a disconnected client stops parking in the FIFO
for the full queueMs.

- acquireHeavyWithin(timeoutMs, signal?): on abort the waiter is removed
  from the FIFO immediately and the promise resolves null early;
  pre-aborted signals never park; the deadline timer is cleared when
  abort/release wins the race
- admitChatRequest reserve() passes request.signal; admitChatStructure
  gains options.signal; the route threads request.signal
- 5 exact-assertion tests (settle-early, pre-aborted, byte-heavy,
  structural, FIFO-preservation): 119/119 across the 7-file suite

* fix(admission): bound queued bytes for the queue-wait heap valve (#9654)

U3 from KC plan 2026-08-09-001. The restored queue-wait parks fully-buffered
bodies; without a cap, several large coding-agent bodies (~750 KB) waiting at
once recreates the #4380 heap amplification this module was built to stop.

- acquireHeavyWithin(timeoutMs, signal?, queuedBytes): each parked waiter is
  charged its buffered size against CHAT_ADMISSION_MAX_QUEUED_BYTES (default
  4 MB); over-budget waits reject immediately with a retryable 503 and never
  park. The charge is released on wake, abort, or timeout.
- Real sizes threaded from admitChatRequest (declared length / sniffed bytes);
  structural waits charge the conservative 256 KB weight.
- Lower default OMNIROUTE_CHAT_ADMISSION_QUEUE_MS to 2000ms (was 5000ms).
- Env vars documented in .env.example; 6 exact-assertion tests: 125/125 across
  the 7-file admission suite (was 119).

* docs: map the two admission-lane systems for operators (#9654)

U5 from KC plan 2026-08-09-001. Verifies lane metrics are exposed by the health
payload (GET /api/monitoring/health -> adaptiveAdmission -> lane* fields) and
records which lane system reports where: byte-level per-connection lanes (always
on, memory scope) vs adaptive virtual lanes (opt-in via OMNIROUTE_CHAT_VIRTUAL_LANES,
dispatch scope) plus the explicit opt-in ops note.

* docs: add required frontmatter to admission-lanes doc (dast-smoke build fix)

* docs: sync env vars with .env.example and ENVIRONMENT.md (docs gate fix)

* fix(admission): complete REJECT_MAP, literal lane env read, split oversized test file

Three CI-gate fixes surfaced by the post-merge check run (head 3de77166e):

1. open-sse-typecheck (TS2741): REJECT_MAP was missing the ADMISSION_LANE_EVICTED
   entry that controller.ts:662 emits on lane eviction. Add the 503 mapping so the
   Record<AdmissionRejectCode, RejectHttpMapping> is total.
2. Docs Gates fabricated-claim: OMNIROUTE_CHAT_VIRTUAL_LANES was read dynamically
   via ENV_KEYS.virtualLanes (env[key]), invisible to the literal env.X scanner.
   Read it literally — behavior-identical, doc claim now verifiable.
3. check:file-size: chat-body-admission.test.ts (1307 lines) exceeded the 1000-line
   new-file cap. Split the queue-wait/abort/heap-valve section into
   chat-body-admission-queue.test.ts (818 + 513 lines, both under cap).

Suite: 125/125 across 8 files. All three checkers pass locally.

* refactor(admission): drop dead ENV_KEYS.virtualLanes entry + lock lane-evicted mapping test

Code-review follow-up on 50c93d266:

1. ENV_KEYS.virtualLanes is now unreferenced since the literal env read landed;
   remove it so the config map only lists keys actually read through the map.
2. Add an exact-assertion runtime test for the ADMISSION_LANE_EVICTED mapping:
   a queued lane waiter evicted by the 60s idle TTL rejects with 503 /
   admission_lane_evicted / Retry-After 1 / sanitized body (no raw tenant key).
   Proves the REJECT_MAP entry end-to-end through buildAdmissionRejectResponse.

Suite: 126/126 (17 in runtime file, 125 in the 8-file admission suite).

---------

Co-authored-by: Brandon Bennett <brandonbennett@macbookair.myfiosgateway.com>

src/app/api/v1/chat/completions/route.ts

@@ -20,6 +20,7 @@ import {
   CHAT_ADMISSION_QUEUE_MAX_MS,
   releaseChatAdmissionAfterHandler,
   releaseChatAdmissionWhenDone,
+  resolveSessionId,
 } from "@/shared/middleware/chatBodyAdmission";
 import {
   readCompressionRequestHeader,
@@ -101,7 +102,9 @@ export async function POST(request) {
   // Reserve heavyweight capacity atomically and ingest the body with a hard byte bound
   // BEFORE JSON parsing. Missing or dishonest Content-Length values cannot bypass
   // the actual-byte limit. Capacity exhaustion is retryable rather than process-fatal.
+  const sessionId = resolveSessionId(request);
   const admissionResult = await admitChatRequest(request, {
+    sessionId,
     queueMs: CHAT_ADMISSION_QUEUE_MAX_MS,
   });
   if (admissionResult.admit === false) return admissionResult.response;
@@ -147,7 +150,9 @@ export async function POST(request) {
         }
 
         const structuralAdmission = await admitChatStructure(parsedBody, admission.lease, {
+          sessionId,
           queueMs: CHAT_ADMISSION_QUEUE_MAX_MS,
+          signal: request.signal,
         });
         if (structuralAdmission.admit === false) {
           admission.lease?.release();

src/shared/middleware/chatBodyAdmission.ts

@@ -1,14 +1,25 @@
 /**
- * Bounded admission for POST /v1/chat/completions.
+ * Process-local bounded admission for POST /v1/chat/completions.
  *
  * Large chat bodies amplify into multiple transient representations while they are parsed,
  * translated, compressed, and dispatched. A heap snapshot alone cannot prevent two healthy
  * requests from entering that allocation-heavy path together. This module reserves process-
  * local heavyweight capacity before parsing and enforces the hard limit against bytes read,
  * not an untrusted Content-Length header.
+ *
+ * Per-connection virtual admission lanes (#9654): each distinct API-key (or anonymous)
+ * bucket gets its own FairCostQueue so one connection cannot exhaust heavyweight capacity
+ * and starve others. Idle sessions are auto-evicted after a TTL.
  */
 
 import { CORS_HEADERS } from "../utils/cors";
+import { createHash } from "crypto";
+
+
+const OMNIROUTE_CHAT_VIRTUAL_TTL_MS = parsePositiveInt(
+  process.env.OMNIROUTE_CHAT_VIRTUAL_TTL_MS,
+  60_000
+);
 
 function parsePositiveInt(value: string | undefined, fallback: number): number {
   const parsed = Number.parseInt(String(value), 10);
@@ -30,7 +41,7 @@ export const CHAT_HARD_MAX_BODY_BYTES = parsePositiveInt(
   50 * 1024 * 1024
 );
 
-const CHAT_MAX_HEAVY_IN_FLIGHT = parsePositiveInt(
+export const CHAT_MAX_HEAVY_IN_FLIGHT = parsePositiveInt(
   process.env.OMNIROUTE_CHAT_MAX_HEAVY_IN_FLIGHT,
   1
 );
@@ -44,7 +55,20 @@ const CHAT_MAX_HEAVY_IN_FLIGHT = parsePositiveInt(
  */
 export const CHAT_ADMISSION_QUEUE_MAX_MS = parseNonNegativeInt(
   process.env.OMNIROUTE_CHAT_ADMISSION_QUEUE_MS,
-  5000
+  2000
+);
+
+/**
+ * Queued-bytes budget for the admission wait (#9654 / U3). A parked waiter holds a
+ * fully-buffered request body; several large coding-agent bodies (~750 KB) waiting at
+ * once is exactly the heap-amplification scenario chatBodyAdmission was built to stop
+ * (#4380). Each lane's controller charges every parked waiter's buffered size against
+ * this budget and rejects over-budget waits immediately (retryable 503) instead of
+ * parking. Bytes are released when a waiter wakes, aborts, or times out.
+ */
+export const CHAT_ADMISSION_MAX_QUEUED_BYTES = parsePositiveInt(
+  process.env.OMNIROUTE_CHAT_ADMISSION_MAX_QUEUED_BYTES,
+  4 * 1024 * 1024
 );
 
 export const CHAT_HEAVY_MESSAGE_COUNT = parsePositiveInt(
@@ -94,18 +118,30 @@ export interface ChatAdmissionLease {
  */
 export class ChatAdmissionController {
   #activeHeavy = 0;
+  #queuedBytes = 0;
   #waiters: Array<() => void> = [];
 
-  constructor(readonly maxHeavyInFlight = 1) {
+  constructor(
+    readonly maxHeavyInFlight = 1,
+    readonly maxQueuedBytes = CHAT_ADMISSION_MAX_QUEUED_BYTES
+  ) {
     if (!Number.isSafeInteger(maxHeavyInFlight) || maxHeavyInFlight < 1) {
       throw new RangeError("maxHeavyInFlight must be a positive integer");
     }
+    if (!Number.isSafeInteger(maxQueuedBytes) || maxQueuedBytes < 0) {
+      throw new RangeError("maxQueuedBytes must be a non-negative integer");
+    }
   }
 
   get activeHeavy(): number {
     return this.#activeHeavy;
   }
 
+  /** Total buffered bytes currently parked in the FIFO (heap valve accounting). */
+  get queuedBytes(): number {
+    return this.#queuedBytes;
+  }
+
   tryAcquireHeavy(): ChatAdmissionLease | null {
     if (this.#activeHeavy >= this.maxHeavyInFlight) return null;
     this.#activeHeavy += 1;
@@ -128,27 +164,71 @@ export class ChatAdmissionController {
    * release. Resolves `null` when the deadline expires with no capacity freed, in
    * which case the caller answers the retryable 503. `timeoutMs <= 0` is the
    * legacy immediate-reject path. Waiters are served FIFO.
+   *
+   * When `signal` aborts while parked (client disconnect), the waiter is removed
+   * from the FIFO immediately and the promise resolves `null` early instead of
+   * parking for the full `timeoutMs` — the caller's 503 is dropped on the dead
+   * connection, so no capacity is consumed and the freed slot never wakes a
+   * waiter the client no longer needs. A signal that is already aborted never
+   * parks at all.
+   *
+   * `queuedBytes` is the buffered body size this waiter will hold while parked;
+   * it is charged against `maxQueuedBytes` so a burst of large bodies cannot
+   * amplify the heap (#4380). An over-budget wait is rejected immediately with
+   * `null` (retryable 503) and never parks; the charge is released on wake,
+   * abort, or timeout.
    */
-  async acquireHeavyWithin(timeoutMs: number): Promise<ChatAdmissionLease | null> {
+  async acquireHeavyWithin(
+    timeoutMs: number,
+    signal?: AbortSignal,
+    queuedBytes = 0
+  ): Promise<ChatAdmissionLease | null> {
     const deadline = Date.now() + Math.max(0, Math.floor(timeoutMs));
     for (;;) {
+      if (signal?.aborted) return null;
       const lease = this.tryAcquireHeavy();
       if (lease) return lease;
       const remaining = deadline - Date.now();
       if (remaining <= 0) return null;
+      // Heap valve: refuse to park when the queued-bytes budget is exhausted.
+      if (queuedBytes > 0 && this.#queuedBytes + queuedBytes > this.maxQueuedBytes) {
+        return null;
+      }
+      this.#queuedBytes += queuedBytes;
       let resolver: (() => void) | null = null;
       const released = new Promise<void>((resolve) => {
         resolver = () => resolve();
         this.#waiters.push(resolver);
       });
-      const timedOut = await Promise.race([
+      let deadlineTimer: ReturnType<typeof setTimeout> | null = null;
+      const races: Array<Promise<boolean>> = [
         released.then(() => false),
-        new Promise<boolean>((resolve) => setTimeout(() => resolve(true), remaining)),
-      ]);
+        new Promise<boolean>((resolve) => {
+          deadlineTimer = setTimeout(() => resolve(true), remaining);
+        }),
+      ];
+      let onAbort: (() => void) | null = null;
+      if (signal) {
+        races.push(
+          new Promise<boolean>((resolve) => {
+            const listener = () => resolve(true);
+            onAbort = listener;
+            signal.addEventListener("abort", listener, { once: true });
+            // Already-aborted signals must settle without parking.
+            if (signal.aborted) resolve(true);
+          })
+        );
+      }
+      const timedOut = await Promise.race(races);
+      // The waiter has left the FIFO (wake, abort, or timeout) — release its charge.
+      this.#queuedBytes = Math.max(0, this.#queuedBytes - queuedBytes);
       if (resolver) {
         const index = this.#waiters.indexOf(resolver);
         if (index >= 0) this.#waiters.splice(index, 1);
       }
+      // Cancel the deadline timer when abort/release wins; a fired timer is a no-op.
+      if (deadlineTimer) clearTimeout(deadlineTimer);
+      if (onAbort) signal?.removeEventListener("abort", onAbort);
       if (timedOut) return null;
     }
   }
@@ -156,6 +236,142 @@ export class ChatAdmissionController {
 
 const defaultAdmissionController = new ChatAdmissionController(CHAT_MAX_HEAVY_IN_FLIGHT);
 
+/**
+ * Per-connection virtual admission lanes (#9654).
+ *
+ * Maps a sessionId (API-key hash or "anonymous") → ChatAdmissionController.
+   Each connection gets its own bounded heavyweight capacity so one connection
+ * cannot exhaust `CHAT_MAX_HEAVY_IN_FLIGHT` and starve others at the byte-level
+ * admission stage.
+ *
+ * Idle sessions are auto-evicted after OMNIROUTE_CHAT_VIRTUAL_TTL_MS
+ * (default 60s) to prevent unbounded Map growth.
+ */
+const OMNIROUTE_CHAT_VIRTUAL_MAX_SESSIONS = parsePositiveInt(
+  process.env.OMNIROUTE_CHAT_VIRTUAL_MAX_SESSIONS,
+  64
+);
+
+export function resolveSessionId(request: Request): string {
+  // Reuse the existing internal-bypass auth extraction: bearer token from
+  // Authorization, x-api-key (Anthropic-style), or Google API key header.
+  const authHeader = request.headers.get("authorization") || "";
+  const bearerMatch = /^bearer\s+(\S+)$/i.exec(authHeader.trim());
+  if (bearerMatch) {
+    return "key_" + createHash("sha256").update(bearerMatch[1]).digest("hex").slice(0, 16);
+  }
+  const xApiKey = request.headers.get("x-api-key") || "";
+  if (xApiKey.trim().length > 0) {
+    return "key_" + createHash("sha256").update(xApiKey.trim()).digest("hex").slice(0, 16);
+  }
+  const xGoogApiKey = request.headers.get("x-goog-api-key") || "";
+  if (xGoogApiKey.trim().length > 0) {
+    return "key_" + createHash("sha256").update(xGoogApiKey.trim()).digest("hex").slice(0, 16);
+  }
+  return "anonymous";
+}
+
+interface SessionRecord {
+  controller: ChatAdmissionController;
+  lastUsedMs: number;
+}
+
+export class PerConnectionAdmissionController {
+  #sessions = new Map<string, SessionRecord>();
+  #evictionTimer: ReturnType<typeof setTimeout> | null = null;
+  readonly maxSessions: number;
+  readonly sessionTtlMs: number;
+
+  constructor(
+    readonly maxHeavyPerSession: number,
+    opts?: { maxSessions?: number; sessionTtlMs?: number }
+  ) {
+    this.maxSessions = opts?.maxSessions ?? OMNIROUTE_CHAT_VIRTUAL_MAX_SESSIONS;
+    this.sessionTtlMs = opts?.sessionTtlMs ?? OMNIROUTE_CHAT_VIRTUAL_TTL_MS;
+  }
+
+  getController(sessionId: string): ChatAdmissionController {
+    this.evictIfDue();
+    const existing = this.#sessions.get(sessionId);
+    if (existing) {
+      existing.lastUsedMs = Date.now();
+      return existing.controller;
+    }
+    // Evict oldest if at capacity (LRU fallback when TTL hasn't fired).
+    if (this.#sessions.size >= this.maxSessions) {
+      const oldestKey = this.oldestKey();
+      if (oldestKey) this.#sessions.delete(oldestKey);
+    }
+    const controller = new ChatAdmissionController(this.maxHeavyPerSession);
+    this.#sessions.set(sessionId, { controller, lastUsedMs: Date.now() });
+    this.armEviction();
+    return controller;
+  }
+
+  /** Snapshot for observability — never exposes raw API keys. */
+  snapshot(): ReadonlyArray<{ sessionId: string; activeHeavy: number; idleMs: number }> {
+    const now = Date.now();
+    const arr: Array<{ sessionId: string; activeHeavy: number; idleMs: number }> = [];
+    for (const [sessionId, record] of this.#sessions) {
+      arr.push({
+        sessionId,
+        activeHeavy: record.controller.activeHeavy,
+        idleMs: now - record.lastUsedMs,
+      });
+    }
+    return arr;
+  }
+
+  get sessionCount(): number {
+    return this.#sessions.size;
+  }
+
+  private oldestKey(): string | undefined {
+    let oldest: string | undefined;
+    let oldestMs = Infinity;
+    for (const [key, record] of this.#sessions) {
+      // Use <= so that for equal timestamps, later-inserted entries win,
+      // preserving LRU semantics when Date.now() returns the same value.
+      if (record.lastUsedMs <= oldestMs) {
+        oldestMs = record.lastUsedMs;
+        oldest = key;
+      }
+    }
+    return oldest;
+  }
+
+  private evictIfDue(): void {
+    const now = Date.now();
+    let evicted = false;
+    for (const [sessionId, record] of this.#sessions) {
+      if (now - record.lastUsedMs >= this.sessionTtlMs) {
+        this.#sessions.delete(sessionId);
+        evicted = true;
+      }
+    }
+    if (evicted) this.armEviction();
+  }
+
+  private armEviction(): void {
+    if (this.#evictionTimer !== null) return;
+    this.#evictionTimer = setTimeout(() => {
+      this.#evictionTimer = null;
+      this.evictIfDue();
+    }, this.sessionTtlMs).unref();
+  }
+
+  /** Force cleanup of all sessions (used by shutdown / tests). */
+  dispose(): void {
+    this.#sessions.clear();
+    if (this.#evictionTimer !== null) {
+      clearTimeout(this.#evictionTimer);
+      this.#evictionTimer = null;
+    }
+  }
+}
+
+export const perConnectionAdmissionController = new PerConnectionAdmissionController(CHAT_MAX_HEAVY_IN_FLIGHT);
+
 export type ChatRequestAdmission =
   | { admit: true; request: Request; lease: ChatAdmissionLease | null }
   | { admit: false; response: Response };
@@ -264,11 +480,13 @@ export async function admitChatStructure(
   lease: ChatAdmissionLease | null,
   options: {
     controller?: ChatAdmissionController;
+    sessionId?: string;
     maxMessages?: number;
     heavyMessages?: number;
     heavyTools?: number;
     heavyTokens?: number;
     queueMs?: number;
+    signal?: AbortSignal;
   } = {}
 ): Promise<ChatStructureAdmission> {
   if (!body || typeof body !== "object" || Array.isArray(body)) return { admit: true, lease };
@@ -301,8 +519,18 @@ export async function admitChatStructure(
     estimatedTokens >= heavyTokens;
   if (!heavy || lease) return { admit: true, lease };
 
-  const acquired = await (options.controller ?? defaultAdmissionController).acquireHeavyWithin(
-    options.queueMs ?? 0
+  const controller =
+    options.controller ??
+    (options.sessionId
+      ? perConnectionAdmissionController.getController(options.sessionId)
+      : defaultAdmissionController);
+  // Structural-only waits happen on byte-light bodies (a byte-heavy body already
+  // holds the byte-stage lease), so the conservative 256KB weight bounds the
+  // parsed JSON the waiter keeps resident while parked.
+  const acquired = await controller.acquireHeavyWithin(
+    options.queueMs ?? 0,
+    options.signal,
+    CHAT_LARGE_BODY_BYTES
   );
   return acquired
     ? { admit: true, lease: acquired }
@@ -413,12 +641,15 @@ export async function admitChatRequest(
   request: Request,
   options: {
     controller?: ChatAdmissionController;
+    sessionId?: string;
     largeBodyBytes?: number;
     hardMaxBytes?: number;
     queueMs?: number;
   } = {}
 ): Promise<ChatRequestAdmission> {
-  const controller = options.controller ?? defaultAdmissionController;
+  const sessionId = options.sessionId ?? resolveSessionId(request);
+  const controller =
+    options.controller ?? perConnectionAdmissionController.getController(sessionId);
   const largeBodyBytes = options.largeBodyBytes ?? CHAT_LARGE_BODY_BYTES;
   const hardMaxBytes = options.hardMaxBytes ?? CHAT_HARD_MAX_BODY_BYTES;
   const queueMs = options.queueMs ?? 0;
@@ -467,15 +698,19 @@ export async function admitChatRequest(
   }
 
   let lease: ChatAdmissionLease | null = null;
-  const reserve = async (): Promise<boolean> => {
+  const reserve = async (bytes = 0): Promise<boolean> => {
     if (lease) return true;
-    lease = await controller.acquireHeavyWithin(queueMs);
+    lease = await controller.acquireHeavyWithin(queueMs, request.signal, bytes);
     return lease !== null;
   };
 
   // A known-large declaration can reserve before ingestion. Unknown lengths are boundedly
   // sniffed below; this avoids consuming scarce heavyweight capacity for small chunked bodies.
-  if (contentLength !== null && contentLength >= largeBodyBytes && !(await reserve())) {
+  if (
+    contentLength !== null &&
+    contentLength >= largeBodyBytes &&
+    !(await reserve(Math.min(contentLength, hardMaxBytes)))
+  ) {
     return { admit: false, response: rejectionResponse(503, hardMaxBytes) };
   }
 
@@ -494,7 +729,7 @@ export async function admitChatRequest(
         lease?.release();
         return { admit: false, response: rejectionResponse(413, hardMaxBytes) };
       }
-      if (totalBytes >= largeBodyBytes && !(await reserve())) {
+      if (totalBytes >= largeBodyBytes && !(await reserve(totalBytes))) {
         await reader.cancel("chat admission capacity unavailable").catch(() => undefined);
         return { admit: false, response: rejectionResponse(503, hardMaxBytes) };
       }