docs: map the two admission-lane systems for operators (#9654)

U5 from KC plan 2026-08-09-001. Verifies lane metrics are exposed by the health
payload (GET /api/monitoring/health -> adaptiveAdmission -> lane* fields) and
records which lane system reports where: byte-level per-connection lanes (always
on, memory scope) vs adaptive virtual lanes (opt-in via OMNIROUTE_CHAT_VIRTUAL_LANES,
dispatch scope) plus the explicit opt-in ops note.
This commit is contained in:
Brandon Bennett
2026-08-09 17:52:07 -07:00
parent 74c3a04d0d
commit f652f59bd0

View File

@@ -0,0 +1,44 @@
# Admission lanes (#9654) — two lane systems, what gates each, where each reports
OmniRoute has **two** process-local lane systems with different scopes. They are
complementary; operators should know which one they are looking at.
## 1. Byte-level per-connection lanes (`chatBodyAdmission.ts`)
- **Scope:** the buffered-body/heap path for `POST /v1/chat/completions`. Guards
against heap amplification from large coding-agent bodies (#4380).
- **Gate:** **always on.** Each distinct API key (hashed) — or `anonymous` — gets its
own lane with `CHAT_MAX_HEAVY_IN_FLIGHT` capacity, so one session's burst cannot
starve another session's heavyweight slot.
- **Tuning:**
- `OMNIROUTE_CHAT_VIRTUAL_TTL_MS` — idle-lane eviction (default 60000)
- `OMNIROUTE_CHAT_VIRTUAL_MAX_SESSIONS` — lane count cap (default 64)
- `OMNIROUTE_CHAT_ADMISSION_QUEUE_MS` — queue-wait before 503 (default 2000)
- `OMNIROUTE_CHAT_ADMISSION_MAX_QUEUED_BYTES` — queued-bytes heap valve (default 4 MB)
- **Reports:** not in `GET /api/monitoring/health` today; observable via
`PerConnectionAdmissionController.snapshot()` (sessionId hash, activeHeavy, idleMs).
## 2. Adaptive runtime virtual lanes (`open-sse/services/admission`)
- **Scope:** tenant-key admission for provider dispatch — queue cost, latency-guided
limit adaptation, lane queueing, and lane metrics.
- **Gate:** **opt-in.** Disabled unless `OMNIROUTE_CHAT_VIRTUAL_LANES=true`. Without it,
the adaptive controller keeps the shared queue behavior (criterion 1 of #9654 only
holds once an operator enables lanes).
- **Tuning:** `OMNIROUTE_CHAT_VIRTUAL_LANES` + adaptive config (`maxQueueCount`,
`maxQueueCost`, `defaultMaxWaitMs`, …).
- **Reports:** `GET /api/monitoring/health``adaptiveAdmission``laneCount`,
`laneQueuedCount`, `laneQueuedCost`, `laneTenants` (opaque lane IDs, never raw keys).
## Which one is showing in a dashboard
- `adaptiveAdmission.laneCount` / `laneTenants`**adaptive virtual lanes** (system 2).
- A health payload with **no** `adaptiveAdmission.lane*` fields usually means
`OMNIROUTE_CHAT_VIRTUAL_LANES` is unset — the byte-level lanes (system 1) are still
active, but nothing under `adaptiveAdmission` will report lane data until it is enabled.
## Why both exist
The byte-level lanes bound the memory-heavy parse/compress path; the adaptive lanes
bound dispatch cost per tenant. #9654's criterion 1 ("one session's burst does not 503
another") is enforced by system 1 unconditionally and by system 2 once opt-in is enabled.