mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-08-19 05:32:19 +03:00
8d78e3dfd3
* fix: add per-connection virtual admission lanes (#9654) Worst-day-ever analysis to harden AdaptiveAdmissionController: - Guard expireEntry() against null entry (CRITICAL null deref) - Add deleteLane() to drain+reject on LRU eviction (HIGH orphaned promises) - Fix Map mutation during evictIdleLanes iteration (MEDIUM safety) - Add ADMISSION_LANE_EVICTED reject code (MEDIUM clarity) - Pass sessionId to admitChatRequest in route.ts - virtualLanes defaults to false in validateConfig - 7 new controller tests + 14 new byte-level admission tests - Assertions tightened from >= to === (Matt Pocock methodology) Debunked 2 false positives: concurrency race (single-threaded JS) and memory amplification (FairCostQueue bounds per-lane). Fixes #9654 * fix(admission): restore bounded queue-wait on per-connection lanes (#9654) The per-connection lane refactor dropped the bounded queue-wait (acquireHeavyWithin / #waiters / queueMs). #9654's acceptance criteria and #9608 section C prefer server-side wait/pacing up to defaultMaxWaitMs over an instant retryable 503. - ChatAdmissionController: re-add #waiters FIFO + acquireHeavyWithin(timeoutMs); queueMs: 0 preserves the instant-503 path - admitChatStructure and admitChatRequest.reserve are async again and take queueMs - route: pass CHAT_ADMISSION_QUEUE_MAX_MS and await the admission calls - per-connection lane tests await the async admitChatStructure Admission suite: 114/114 pass (bun test, 7 files). * chore: re-trigger CI after dast-smoke infra cancellation (#9654) * feat(admission): cancel queue-wait on client abort (#9654) U2 from KC plan 2026-08-09-001. Thread the request AbortSignal through acquireHeavyWithin so a disconnected client stops parking in the FIFO for the full queueMs. - acquireHeavyWithin(timeoutMs, signal?): on abort the waiter is removed from the FIFO immediately and the promise resolves null early; pre-aborted signals never park; the deadline timer is cleared when abort/release wins the race - admitChatRequest reserve() passes request.signal; admitChatStructure gains options.signal; the route threads request.signal - 5 exact-assertion tests (settle-early, pre-aborted, byte-heavy, structural, FIFO-preservation): 119/119 across the 7-file suite * fix(admission): bound queued bytes for the queue-wait heap valve (#9654) U3 from KC plan 2026-08-09-001. The restored queue-wait parks fully-buffered bodies; without a cap, several large coding-agent bodies (~750 KB) waiting at once recreates the #4380 heap amplification this module was built to stop. - acquireHeavyWithin(timeoutMs, signal?, queuedBytes): each parked waiter is charged its buffered size against CHAT_ADMISSION_MAX_QUEUED_BYTES (default 4 MB); over-budget waits reject immediately with a retryable 503 and never park. The charge is released on wake, abort, or timeout. - Real sizes threaded from admitChatRequest (declared length / sniffed bytes); structural waits charge the conservative 256 KB weight. - Lower default OMNIROUTE_CHAT_ADMISSION_QUEUE_MS to 2000ms (was 5000ms). - Env vars documented in .env.example; 6 exact-assertion tests: 125/125 across the 7-file admission suite (was 119). * docs: map the two admission-lane systems for operators (#9654) U5 from KC plan 2026-08-09-001. Verifies lane metrics are exposed by the health payload (GET /api/monitoring/health -> adaptiveAdmission -> lane* fields) and records which lane system reports where: byte-level per-connection lanes (always on, memory scope) vs adaptive virtual lanes (opt-in via OMNIROUTE_CHAT_VIRTUAL_LANES, dispatch scope) plus the explicit opt-in ops note. * docs: add required frontmatter to admission-lanes doc (dast-smoke build fix) * docs: sync env vars with .env.example and ENVIRONMENT.md (docs gate fix) * fix(admission): complete REJECT_MAP, literal lane env read, split oversized test file Three CI-gate fixes surfaced by the post-merge check run (head3de77166e): 1. open-sse-typecheck (TS2741): REJECT_MAP was missing the ADMISSION_LANE_EVICTED entry that controller.ts:662 emits on lane eviction. Add the 503 mapping so the Record<AdmissionRejectCode, RejectHttpMapping> is total. 2. Docs Gates fabricated-claim: OMNIROUTE_CHAT_VIRTUAL_LANES was read dynamically via ENV_KEYS.virtualLanes (env[key]), invisible to the literal env.X scanner. Read it literally — behavior-identical, doc claim now verifiable. 3. check:file-size: chat-body-admission.test.ts (1307 lines) exceeded the 1000-line new-file cap. Split the queue-wait/abort/heap-valve section into chat-body-admission-queue.test.ts (818 + 513 lines, both under cap). Suite: 125/125 across 8 files. All three checkers pass locally. * refactor(admission): drop dead ENV_KEYS.virtualLanes entry + lock lane-evicted mapping test Code-review follow-up on50c93d266: 1. ENV_KEYS.virtualLanes is now unreferenced since the literal env read landed; remove it so the config map only lists keys actually read through the map. 2. Add an exact-assertion runtime test for the ADMISSION_LANE_EVICTED mapping: a queued lane waiter evicted by the 60s idle TTL rejects with 503 / admission_lane_evicted / Retry-After 1 / sanitized body (no raw tenant key). Proves the REJECT_MAP entry end-to-end through buildAdmissionRejectResponse. Suite: 126/126 (17 in runtime file, 125 in the 8-file admission suite). --------- Co-authored-by: Brandon Bennett <brandonbennett@macbookair.myfiosgateway.com>
2.6 KiB
2.6 KiB
title, status, lastUpdated
| title | status | lastUpdated |
|---|---|---|
| Admission lanes — two lane systems, what gates each, where each reports | active | 2026-08-09 |
Admission lanes (#9654) — two lane systems, what gates each, where each reports
OmniRoute has two process-local lane systems with different scopes. They are complementary; operators should know which one they are looking at.
1. Byte-level per-connection lanes (chatBodyAdmission.ts)
- Scope: the buffered-body/heap path for
POST /v1/chat/completions. Guards against heap amplification from large coding-agent bodies (#4380). - Gate: always on. Each distinct API key (hashed) — or
anonymous— gets its own lane withCHAT_MAX_HEAVY_IN_FLIGHTcapacity, so one session's burst cannot starve another session's heavyweight slot. - Tuning:
OMNIROUTE_CHAT_VIRTUAL_TTL_MS— idle-lane eviction (default 60000)OMNIROUTE_CHAT_VIRTUAL_MAX_SESSIONS— lane count cap (default 64)OMNIROUTE_CHAT_ADMISSION_QUEUE_MS— queue-wait before 503 (default 2000)OMNIROUTE_CHAT_ADMISSION_MAX_QUEUED_BYTES— queued-bytes heap valve (default 4 MB)
- Reports: not in
GET /api/monitoring/healthtoday; observable viaPerConnectionAdmissionController.snapshot()(sessionId hash, activeHeavy, idleMs).
2. Adaptive runtime virtual lanes (open-sse/services/admission)
- Scope: tenant-key admission for provider dispatch — queue cost, latency-guided limit adaptation, lane queueing, and lane metrics.
- Gate: opt-in. Disabled unless
OMNIROUTE_CHAT_VIRTUAL_LANES=true. Without it, the adaptive controller keeps the shared queue behavior (criterion 1 of #9654 only holds once an operator enables lanes). - Tuning:
OMNIROUTE_CHAT_VIRTUAL_LANES+ adaptive config (maxQueueCount,maxQueueCost,defaultMaxWaitMs, …). - Reports:
GET /api/monitoring/health→adaptiveAdmission→laneCount,laneQueuedCount,laneQueuedCost,laneTenants(opaque lane IDs, never raw keys).
Which one is showing in a dashboard
adaptiveAdmission.laneCount/laneTenants→ adaptive virtual lanes (system 2).- A health payload with no
adaptiveAdmission.lane*fields usually meansOMNIROUTE_CHAT_VIRTUAL_LANESis unset — the byte-level lanes (system 1) are still active, but nothing underadaptiveAdmissionwill report lane data until it is enabled.
Why both exist
The byte-level lanes bound the memory-heavy parse/compress path; the adaptive lanes bound dispatch cost per tenant. #9654's criterion 1 ("one session's burst does not 503 another") is enforced by system 1 unconditionally and by system 2 once opt-in is enabled.