Acquire admission once after API-key policy, preserve lazy raw-request snapshots, and bind lease settlement to JSON, SSE, abort, deadline, and failure lifecycles. Expose a low-cardinality health summary and preserve non-SSE Ollama errors unchanged.
Add bounded weighted admission with fair queuing, deadline and cancellation handling, exact lease accounting, and a default-shadow runtime. Keep asynchronous resource-pressure shedding as an independent safety fuse and bound request feature estimation.
* fix(reasoning): sanitize streamed K3 think tags
* refactor(stream): move think-tag helpers into thinkTagParser leaf module
open-sse/utils/stream.ts is frozen at 2887 lines with zero headroom, and the
kimi-coding-apikey think-tag handling added here pushed it to 2951. The parsing
half of that work has no SSE dependency, so it moves to thinkTagParser.ts (an
unfrozen leaf): the open-tag lookahead predicate and the end-of-stream flush
delta assembly. stream.ts keeps only the SSE envelope - enqueue, payload
collection, logging.
Getting back under the frozen ceiling needed more than just the PR's own
added lines, since even the fully self-gating helpers still cost a handful of
call-site lines stream.ts has zero room for. Along the way this also
deduplicates a synthetic chat-completion-chunk literal that was copy-pasted
three times in stream.ts (textual tool-call flush, think-tag flush, terminal
finish_reason synthesis) into one buildSyntheticChatChunk() in
streamHelpers.ts - pure DRY, no behavior change.
Behavior is unchanged: same tests, same counts.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: rinseaid <rinseaid@rinseaid.net>
Co-authored-by: diegosouzapw <diegosouzapw@users.noreply.github.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Fixes#8788.
- Added 'UND_ERR_SOCKET' to PROXY_UNREACHABLE_ERROR_CODES so socket disconnects and mid-stream closes are properly tagged with code PROXY_UNREACHABLE and errorCode proxy_unreachable.
- Ensured tagProxyUnreachable normalizes error code to PROXY_UNREACHABLE and errorCode to proxy_unreachable when catching socket-level disconnects.
- Added test in proxyfetch-undici-retry.test.ts verifying UND_ERR_SOCKET classification.
Wire NextAuth session-token merge/persist on successful perplexity-web
responses so rotated cookies survive in provider_connections, matching
chatgpt-web behavior.
synthOpenAIErrorChunk() sets error.type to upstream_empty_response but omits
error.code. The isRequestScopedUpstreamFailure guard only checks type for
context_length_exceeded, so an upstream_empty_response error slips past it.
- Added code: upstream_empty_response to the error object in synthOpenAIErrorChunk()
- Added test assertion verifying error.code matches error.type
This ensures the guard correctly identifies upstream empty responses as
request-scoped failures.
Closes#8469
An `auto/*` combo whose first step lands on an uncredentialed backend returns
HTTP 200 with `finish_reason: "stop"`, `content: null` and `error: null`. The
agent sees a clean empty assistant turn, has no error to stop on, and retries to
its cap.
The non-streaming path already refuses this: `isEmptyContentResponse` rewrites a
200-with-no-content into a 502 "Provider returned empty content", which the combo
layer classifies as a model-level transient and fails over on (#5085). The
streaming path had no equivalent, and neither existing guard covers it:
- `ensureStreamReadiness` is a LIVENESS probe, not a content one. Its failure
message says so — "Stream ended before producing a non-ping SSE event" — and
`hasStreamReadinessSignal` returns true for a bare `delta:{"role":"assistant"}`.
- `createDisconnectAwareStream`'s #7699 branch fires on a MISSING terminal marker
and is scoped to the Claude client format, because for other formats a
marker-less close is genuinely ambiguous.
The reported stream trips neither: OpenAI format, terminates with
`finish_reason: "stop"` and `[DONE]`, contains nothing.
"Completed normally but emitted zero content" is not ambiguous the way a missing
marker is, so this guard is format-agnostic. It reuses `hasUsefulStreamContent`,
which already existed in streamReadiness.ts — exported, correct, and wired to
nothing — and which already counts tool-call-only and reasoning-only output as
real (#2520). A watcher wraps it to handle frames split across network chunks and
to spot the terminal states where emptiness is legitimate, kept in step with
errorClassifier.ts's `LEGIT_EMPTY_OPENAI_FINISH` / `LEGIT_EMPTY_CLAUDE_STOP`:
length, tool_calls, content_filter, max_tokens, tool_use.
Two guards keep it from over-firing, one of which caught a real regression while
building this: the check applies only when bytes were forwarded, and only when
the body actually looked like SSE. A plain JSON completion travels through the
same wrapper and has no `data:` frames, so "no content seen" says nothing about
it — without the SSE gate, four existing stream tests failed.
The `if (done)` branch's reasoning moved into `resolveSilentCloseReason()`, which
also drops `pull` back under the function-length ceiling; cyclomatic lands at
2187 against a baseline of 2188.
This surfaces the error rather than failing over. Failing over would mean holding
every stream until its first content token, since the combo has already returned
leg 1's response by then — a much larger change. Surfacing the error satisfies
the issue's stated expectation ("surface the upstream error OR fail over") and
stops the retry loop, which is the reported harm.
Closes#8649
* fix(sse): stop thrashing the provider prompt cache for caching-aware clients
- shouldPreserveCacheControl: preserve client cache_control markers for every
combo strategy. The deterministic-strategy gate forced marker rewrites whose
per-request breakpoint positions are not stable turn-over-turn, thrashing the
upstream prompt cache (observed in production as ~200k cache_write tokens per
turn on quota-share combos). Preserving is never worse: on a stable target
the client's breakpoints advance deterministically; on a target switch both
approaches miss equally.
- prepareClaudeRequest: translator-path opt-in fallback — when preserve-mode
has nothing to preserve (client sent no cache_control anywhere), apply the
standard heuristic so requests never ship with zero cache breakpoints. The
claude-code-compatible relay path keeps its no-supplement contract.
- anthropic-beta: forward the client-negotiated context-1m-2025-08-07 through
the allowlist merge so a /model <id>[1m] client keeps its long-context
negotiation behind the proxy (never forced when the client did not send it).
* fix(sse): surface cache tokens in non-streaming OpenAI usage + finalize pending on quota-share block
- translateNonStreamingResponse (claude→openai): fold cache_read into
prompt_tokens and expose prompt_tokens_details.cached_tokens /
cache_creation_tokens, mirroring the streaming contract (#1426/#2215).
Non-streaming OpenAI clients behind a cached Claude upstream previously saw
prompt_tokens=<uncached remainder> (e.g. 23 for a ~9k request) with no cache
visibility.
- chatCore quota-share block: finalize the pending-request slot before
returning the policy 429 — the path never reaches upstream and the orphaned
pending lingered as a status-0 call-log row until the reaper swept it.
Validated live on the staging box (repro: blocked key → orphan row; after:
clean).
---------
Co-authored-by: diegosouzapw <diegosouzapw24@gmail.com>
`kieExecutor.createTask()` returns `JsonObject` (`Record<string, unknown>`), so
`createData.data` is `unknown` and the `createData?.data?.taskId` read that
image, video and music generation each duplicated could not compile. The same
three-line expression appeared verbatim in all three handlers.
`open-sse/utils/kieTask.ts` already holds two helpers with exactly this shape —
`normalizeKieTaskState()` and `parseKieResultJson()` both take `unknown`, guard
with `isJsonObject()` and return a declared type. `getKieTaskId()` follows them,
so the three handlers now share one guarded read instead of three unguarded ones.
`getKieCallbackUrl()` took `KieCallbackBody`, a weak type (all properties
optional). Passing a request body whose declared keys are `prompt` /
`timeout_ms` / `poll_interval_ms` tripped TS2559 "no properties in common" at
both music call sites. It receives arbitrary upstream request bodies, so it now
takes `unknown` and guards the same way its neighbours do; `KieCallbackBody`
had no other reference and is gone.
Behaviour is unchanged. `isJsonObject()` rejects arrays and null exactly where
optional chaining already yielded `undefined`, and the callers' `String(taskId)`
coercion moved inside the helper, so a numeric id still reaches `pollTask()` as
a string and a falsy id still takes the 502 branch.
Fixes 5 of the 208 `tsc -p open-sse/tsconfig.json` diagnostics with no new ones:
3 x TS2339 `taskId` on `unknown`, 2 x TS2559 on `KieCallbackBody`.
Refs #8484
Squash merge from local merge-train (Hard Rule owner-approved). Tip 029cdf4215cf465f0e1716ac9f84a84692b1e881 validated on 192.168.0.113: 26631/26653 pass.
* feat(db): cc discovery alias gate storage + EXPOSE_CC_DISCOVERY_ALIASES flag
Adds the gate for claude/<provider>/<model> discovery-alias mirror ids on
the /v1/models catalog: a new runtime feature flag (env forces on and wins
over the dashboard DB override), per-provider and per-model "on"/"off"/null
overrides stored in key_value under the ccDiscoveryAliases namespace, and a
pure precedence resolver (model > provider > global). Catalog wiring is a
separate follow-up task; this only lands the gate + storage.
* feat(sse): synthesize claude/ discovery aliases for the model catalog
* feat(api): advertise cc discovery aliases on /v1/models behind the 3-level gate
* feat(sse): resolve claude/ discovery aliases on the request path
* fix(sse): import getComboByName from db/combos, not the localDb barrel
* fix(sse): cover custom-node prefixes and the Codex WS bridge in cc discovery alias resolution
* feat(dashboard): cc discovery alias toggles + flag-screen env warning
Adds the operator-facing UI/API layer for the Claude Code discovery-alias
gate (claude/<provider>/<model> mirror ids on /v1/models): REST endpoint
for provider/model overrides, a provider-detail card with 3-state
(inherit/on/off) toggles, an info button on the Claude Code tool card
linking to Feature Flags, and an env-source warning on the
EXPOSE_CC_DISCOVERY_ALIASES flag card when it's forced on via env.
* feat(api): cc discovery usage metrics
* fix(api): record cc alias metric in the production wrapper + atomic counter upsert
* docs: document cc discovery aliases (Claude Code guide + feature flag catalog)
* fix(sse): don't mirror built-in auto/* combos as discovery aliases (advertised-but-unroutable)
* i18n(vi): translate the discovery-alias strings instead of shipping placeholders
vi is the one locale with a strict "no internal missing markers" test, so the 17
__MISSING__ entries this branch added (the provider ccAlias panel, the info
button, the feature-flag description and the env warning) would have turned that
test red the moment the base itself was repaired. Translated, keeping every ICU
placeholder ({modelId}, {error}) and the literal claude/<provider>/<model> id
shape intact.
* chore(quality): raise the frozen caps this feature legitimately grows
catalog.ts 1615 -> 1639: the alias synthesis is wired into the catalog builder,
which is where the per-key-filtered list is assembled — the only place the mirror
entries can be appended after model hiding has been applied.
localDb.ts 808 -> 810: two re-export lines for the new ccDiscoveryAliases db
module, which is exactly what the "Adding a New DB Module" recipe prescribes.
* refactor(dashboard,api): keep the complexity ratchets flat
The feature added four cyclomatic violations and one cognitive one, which the
ratchets reject — the baseline only moves when a metric improves. Split the new
code instead:
- appendCcDiscoveryAliases: the four skip-guards become isMirrorableId().
- resolveCcDiscoveryAliasStripWith: alias parsing and gate resolution become
parseCcAliasTarget() and resolveGateFor(), replacing a chain of ternaries that
each re-tested isComboAlias.
- FeatureFlagCard: the env-precedence warning becomes its own component instead
of a conditional branch inside an already-large render.
- ProviderCcAliasSection: the loader moves to useCcAliasData(), and the override
list and add-row become ModelOverrideList / AddOverrideRow, bringing both
oversized functions back under the 80-line rule.
Behavior unchanged — the 74 discovery-alias tests pass untouched. Both ratchets
now sit exactly at baseline (2188 / 971).
* feat(sse): selective upstream 4xx error passthrough util
* feat(sse): relay upstream 4xx error bodies verbatim on the Anthropic request path
createErrorResult() gains an opt-in 7th param opts.passthrough; when set and
the upstream body is eligible (per shouldPassthroughUpstreamError), the
returned response body is the upstream 4xx body verbatim instead of the
sanitized wrapper. Internal classification fields (error/rawMessage/
errorType/errorCode) are never affected.
Wired into chatCore.ts's 7 upstream-error createErrorResult call sites
(model_unavailable / context_overflow / generic upstream-error branches),
gated on sourceFormat === FORMATS.CLAUDE so only /v1/messages requests get
the verbatim body — OpenAI-format paths are unchanged.
* test(quality): register upstream-error-passthrough in stryker tap.testFiles
`tests/unit/upstream-error-passthrough.test.ts` covers `open-sse/utils/error.ts`,
an instrumented module, so `check:mutation-test-coverage --strict` requires it in
`tap.testFiles` — the gate flagged the drift on this PR.
* fix(sse): label HTTP 499 disconnects as client_disconnected
Preserve caller-supplied error type/code in buildErrorBody so stream
abort classification is not overwritten by the status-code table.
* docs(security): document buildErrorBody classification arg
---------
Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com>
buildClientRawRequest deep-cloned the ENTIRE request body on every chat request, unbounded.
On the #7847 incident payload (3.05 MiB, 729 messages, 86 tools) that retains 3.19 MiB per
request, and it is pure waste: every consumer of clientRawRequest.body is observability and
none of them keeps the full payload.
chatCore.ts -> reqLogger.logClientRawRequest no-op when the logger is disabled, otherwise
re-clones via cloneBoundedForLog (0.08 MiB)
chatCore.ts -> trackPendingRequest clientRequest, surfaced by /api/logs/[id]
chat.ts -> recordRejectedRequestUsage requestBody
None feeds dispatch, translation or the upstream request, so the snapshot is now taken with
cloneBoundedForLog: 3.19 MiB -> 0.08 MiB, a 41x reduction, and retention no longer scales with
history length. It stays a clone rather than an alias because body is rewritten downstream
(plugin onRequest hook, compression) and the log must show what the client actually sent.
Bounding at the entry means the logger re-bounds an already-bounded value, which exposed that
cloneBoundedForLog was NOT idempotent -- each container exceeded its own bound once the marker
was added, so a second pass truncated again:
arrays [marker, ...24 items] is 25 entries > 24, so the marker and one real item were
dropped and originalLength was rewritten as 25 instead of the true 729
objects 80 keys + _omniroute_truncated_keys is 81 > 80, so a real key was evicted to make
room for the marker and the dropped count was reported as 1 instead of 20
strings the marker was appended AFTER slicing to maxLength, so the bounded string was
longer than the bound
Without this the persisted log payload would have changed shape versus before the fix. All
three now keep the marker inside the budget and treat an already-bounded value as final;
verified end to end -- the marker still reports originalLength 729.
TDD: tests/unit/repro-7847-bound-client-raw-request.test.ts was written first and failed on
three assertions (unbounded retention, retention scaling with history, and the idempotence
precondition) before either change.
Two changes with one root cause: several hot paths built a full JSON string only to read
its .length, and one of them silently changed the answer.
1. CORRECTNESS -- combo's fallback-compression trigger
estimateTokens(JSON.stringify(attemptBody)) took the STRING branch of estimateTokens,
which is ceil(length / CHARS_PER_TOKEN) over the raw JSON. An inline base64 image is
then charged as if every character of the data URL were prose. Measured on a 200 KB
inline image:
via string (before) 50,039 tokens
via object (after) 1,231 tokens
a 40x over-count, tripping fallback compression on requests nowhere near the context
window. This is the same class #8368/#8401 fixed on the request path; the combo call
site was missed. Passing the object routes through extractImageTokens, which charges
images structurally. Text-only bodies are unaffected -- verified identical, and pinned
by a test.
2. ALLOCATION -- jsonLength()
Adds an exact serialized-length walker: same O(n) scan, no string. Used by
estimateTokens' object branch and by streamReadinessPolicy (which runs on every
streaming request and only ever used .length).
Exactness matters because every consumer feeds a threshold, so this is property-tested
against JSON.stringify over 4000 generated structures covering escaping, lone
surrogates, omitted values, non-finite numbers, toJSON, Date, Map, cycles and BigInt.
Anything outside the plain-JSON subset falls back to JSON.stringify for THAT SUBTREE
only, so an exotic leaf never forces the message history back onto the allocating path.
Honest scoping of the memory win: the string was always transient, and V8 collects it
efficiently, so this is not 3 MiB of retained heap. Measured allocation churn over 20 calls
on a 3.06 MiB body: 3.1 MiB -> 0.5 MiB, about 6x less. The #8549 benchmark row for this
mechanism measures a HELD string and therefore overstates it; the correctness fix above is
the larger deliverable here.
Eight TS2339s in stream.ts read properties off `never`. `never` here did not
mean unreachable code — it meant a type predicate was lying.
function isClaudeEventPayload(payload: unknown): payload is JsonRecord
...
const flushedParsed = bufferedPayload as JsonRecord;
const isClaude = isClaudeEventPayload(flushedParsed);
} else if (!isClaude) { // JsonRecord minus JsonRecord = never
The predicate answers "does this payload carry a Claude event type?" — a
question about contents, not type. A Claude event and a non-Claude one are both
JsonRecord, so `payload is JsonRecord` narrows nothing; applied to an argument
already typed JsonRecord it collapsed the negative branch to `never`, taking
`.id` and `.choices` with it.
Changed the return type to `boolean`. No call site depended on the narrowing:
both other callers pass the value to updateClaudeEmptyResponseLifecycle(), whose
parameter is `unknown`, and passthroughTailProcessor.ts already declares this
function as `(payload: unknown) => boolean` — so this aligns the definition with
how the codebase already consumes it.
280 -> 272, zero new, on a line-number-agnostic diff of the full tsc error set.
The diff is one line, and a type predicate has no runtime representation, so the
emitted JavaScript is unchanged.
No test added, and no gap to fill: the `never`-typed branch is real, reachable
and already covered from both sides — stream-numeric-ids.test.ts exercises
"normalizes numeric id in final chunk without trailing newline" (the flush path
with a non-Claude payload, i.e. the branch TS believed impossible) and "Claude
passthrough does not normalize numeric ids" (the other arm). 338/338 across the
48 SSE/passthrough suites.
No explanatory comment either: stream.ts sits exactly at its frozen file-size
cap (2887), so a single added line fails check:file-size. The reasoning lives
here and in the PR rather than costing a baseline bump on a frozen file.
Co-authored-by: backryun <busan011@ormbiz.co.kr>
First slice of the TypeScript 7 migration split requested on #7697: resolve the
type diagnostics under `open-sse/tsconfig.json` in the lowest-risk modules, with
no toolchain change. 12 diagnostics across 8 files, all outside the hot path —
`chatCore.ts` and `stream.ts` are deliberately left for a later, standalone slice.
Fixes, by cause:
* `Transformer.cancel` (progressTracker, sseHeartbeat, and stream.ts's existing
handler) — the WHATWG Streams standard defines `transformer.cancel(reason)` and
Node implements it (verified on v24: cancelling the readable side invokes it),
but `lib.dom.d.ts` still omits it from `Transformer`, so every such handler was
TS2353. These handlers clear the heartbeat/progress intervals when an SSE client
disconnects, so deleting them to satisfy the checker would leak a timer per
abandoned stream. The interface is patched in `open-sse/types.d.ts` instead.
* `earlyStreamKeepalive` — `SettledHandler` was discriminated by `ok: true | false`.
This workspace compiles with `strictNullChecks: false`, where a boolean-literal
discriminant narrows the positive branch but not the negative one, so reading
`.error` off the rejected arm did not type-check (the two `.response` reads
elsewhere in the file did, which is why only one site errored). Retagged with a
string discriminant, which narrows both branches under the same settings.
* `toolCallShim` / `openai-responses` — assigning back to a property declared
`unknown` resets the `typeof` narrowing, so the following comparison no longer
saw a number/array. Both now read through a local. The `Read` limit clamp is
behavior-identical: its two branches are mutually exclusive at READ_MAX_LIMIT 2000.
* `sanitizeToolResultId` — takes `unknown` but forwards to a `string` parameter; a
non-string id previously reached `.replace()` and threw. Coerced instead.
* `openaiHelper` — `opts = {}` inferred `{}`; typed as `FilterToOpenAIFormatOptions`.
* `cursorAgentProtobuf` — `Buffer.alloc(0)` infers `Buffer<ArrayBuffer>` under
@types/node 26 while the decoded field is `Buffer<ArrayBufferLike>`; the locals
now use bare `Buffer`, matching `requestMetadata` a few lines above.
Validation: 335 -> 321 diagnostics with zero new errors (full tsc error-set diff
against the base config). typecheck:core clean, lint clean, check:type-coverage
92.17% -> 94.17%. All 114 existing test files that import a touched module pass;
`plan3-p0.test.ts` fails identically with and without this change (it reads the
developer's real ~/.omniroute DB instead of a test-scoped DATA_DIR).
The new test covers the three behavioral surfaces rather than the refactors the
existing keepalive/heartbeat suites already hold: that `transformer.cancel()`
really fires and can clear an interval, the id coercion, and the limit-clamp bounds.
* fix(api): stop the 2000-token safety buffer from inflating usage.prompt_tokens in the client response (#8331)
* fix(sse): scope #8331's usage-buffer fix around Claude-Code-compatible providers
The #8331 fix correctly stopped folding the 2000-token context-window safety
margin into client-visible prompt_tokens/input_tokens/total_tokens for normal
API metering clients. But it also silently changed the response shape for
Claude-Code-compatible providers, whose own context accounting reads the
buffered number straight out of usage — regressing
tests/unit/cc-compatible-provider.test.ts (expected 2007, got 7).
Fold the computed context_budget_* fields back into the visible usage fields
for that one path only (applyClientUsageBuffer's new
preserveContextBudgetInVisibleUsage option, gated on the existing
isClaudeCodeCompatible flag in chatCore.ts). Every other caller keeps the
real, unbuffered #8331 numbers.
Live incident: streamed assistant text was losing the spaces BETWEEN words
(e.g. "Bilden är en riktig JPEG nu" -> "Bildenärenriktig JPEG nu") on the
Responses-API and Claude streaming paths.
stripInternalReasoningPlaceholder() (#8081/#8162) is called on every
individual delta.content chunk, and unconditionally called .trim() even when
its sentinel ("(prior reasoning summary unavailable)") was never present in
that chunk. Tokenizers commonly emit sub-word tokens with a leading space as
part of the token (e.g. " en", " riktig") -- each such chunk got its only
whitespace character (the inter-word space) silently trimmed away before
being appended to the accumulated message, while the words themselves stayed
intact. Punctuation-only chunks were largely unaffected, matching what was
observed live.
Only trims when the sentinel is actually present -- preserves the original
#8081 intent (collapse a placeholder-only chunk to "") without touching the
overwhelming majority of chunks that never contain it.
Co-authored-by: Markus Hartung <markus.hartream@gmail.com>
Regression: #8162 (port of #8081) added an unconditional `.trim()` to
stripInternalReasoningPlaceholder(), applied to every streaming
delta.content chunk across 3 call-sites (openai-to-claude.ts,
openai-responses.ts, responsesTransformer.ts). Leading/trailing
whitespace at a chunk boundary is a real word boundary between
streaming fragments; trimming it glues adjacent chunks together on
the client ("Hello, " + "world." + " Bye." -> "Hello,world.Bye.").
Fix: early-return via .includes() before the replaceAll+trim, so the
function is a true no-op when the sentinel is absent from the chunk.
Behavior when the sentinel IS present is unchanged.
Validation:
- tests/unit/streaming-reasoning-dedup-5786.test.ts: the "(A-guard)"
test was RED on the base branch ('Hello,world.Bye.' vs
'Hello, world. Bye.'); GREEN after the fix (4/4 passing).
- tests/unit/translator-resp-openai-to-claude.test.ts: added a new
multi-chunk boundary-whitespace regression test, proven RED against
the pre-fix code (12/13), GREEN after (13/13).
- No regressions in responses-transformer.test.ts (17/17),
responses-transformer-dense-output.test.ts (3/3), or the other
suites exercising the shared placeholder utility (160/160 total
across all consumers).
Refs #8162
Refs #8081
Claude→OpenAI translation was emitting a literal </think> into
delta.content for ordinary Chat Completions clients. Reasoning already
ships as reasoning_content, so default to suppress and keep
x-omniroute-thinking-marker: on as the #4633 opt-in.
