Acquire admission once after API-key policy, preserve lazy raw-request snapshots, and bind lease settlement to JSON, SSE, abort, deadline, and failure lifecycles. Expose a low-cardinality health summary and preserve non-SSE Ollama errors unchanged.
Add bounded weighted admission with fair queuing, deadline and cancellation handling, exact lease accounting, and a default-shadow runtime. Keep asynchronous resource-pressure shedding as an independent safety fuse and bound request feature estimation.
* fix(reasoning): sanitize streamed K3 think tags
* refactor(stream): move think-tag helpers into thinkTagParser leaf module
open-sse/utils/stream.ts is frozen at 2887 lines with zero headroom, and the
kimi-coding-apikey think-tag handling added here pushed it to 2951. The parsing
half of that work has no SSE dependency, so it moves to thinkTagParser.ts (an
unfrozen leaf): the open-tag lookahead predicate and the end-of-stream flush
delta assembly. stream.ts keeps only the SSE envelope - enqueue, payload
collection, logging.
Getting back under the frozen ceiling needed more than just the PR's own
added lines, since even the fully self-gating helpers still cost a handful of
call-site lines stream.ts has zero room for. Along the way this also
deduplicates a synthetic chat-completion-chunk literal that was copy-pasted
three times in stream.ts (textual tool-call flush, think-tag flush, terminal
finish_reason synthesis) into one buildSyntheticChatChunk() in
streamHelpers.ts - pure DRY, no behavior change.
Behavior is unchanged: same tests, same counts.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: rinseaid <rinseaid@rinseaid.net>
Co-authored-by: diegosouzapw <diegosouzapw@users.noreply.github.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Fixes#8788.
- Added 'UND_ERR_SOCKET' to PROXY_UNREACHABLE_ERROR_CODES so socket disconnects and mid-stream closes are properly tagged with code PROXY_UNREACHABLE and errorCode proxy_unreachable.
- Ensured tagProxyUnreachable normalizes error code to PROXY_UNREACHABLE and errorCode to proxy_unreachable when catching socket-level disconnects.
- Added test in proxyfetch-undici-retry.test.ts verifying UND_ERR_SOCKET classification.
Wire NextAuth session-token merge/persist on successful perplexity-web
responses so rotated cookies survive in provider_connections, matching
chatgpt-web behavior.
synthOpenAIErrorChunk() sets error.type to upstream_empty_response but omits
error.code. The isRequestScopedUpstreamFailure guard only checks type for
context_length_exceeded, so an upstream_empty_response error slips past it.
- Added code: upstream_empty_response to the error object in synthOpenAIErrorChunk()
- Added test assertion verifying error.code matches error.type
This ensures the guard correctly identifies upstream empty responses as
request-scoped failures.
Closes#8469
An `auto/*` combo whose first step lands on an uncredentialed backend returns
HTTP 200 with `finish_reason: "stop"`, `content: null` and `error: null`. The
agent sees a clean empty assistant turn, has no error to stop on, and retries to
its cap.
The non-streaming path already refuses this: `isEmptyContentResponse` rewrites a
200-with-no-content into a 502 "Provider returned empty content", which the combo
layer classifies as a model-level transient and fails over on (#5085). The
streaming path had no equivalent, and neither existing guard covers it:
- `ensureStreamReadiness` is a LIVENESS probe, not a content one. Its failure
message says so — "Stream ended before producing a non-ping SSE event" — and
`hasStreamReadinessSignal` returns true for a bare `delta:{"role":"assistant"}`.
- `createDisconnectAwareStream`'s #7699 branch fires on a MISSING terminal marker
and is scoped to the Claude client format, because for other formats a
marker-less close is genuinely ambiguous.
The reported stream trips neither: OpenAI format, terminates with
`finish_reason: "stop"` and `[DONE]`, contains nothing.
"Completed normally but emitted zero content" is not ambiguous the way a missing
marker is, so this guard is format-agnostic. It reuses `hasUsefulStreamContent`,
which already existed in streamReadiness.ts — exported, correct, and wired to
nothing — and which already counts tool-call-only and reasoning-only output as
real (#2520). A watcher wraps it to handle frames split across network chunks and
to spot the terminal states where emptiness is legitimate, kept in step with
errorClassifier.ts's `LEGIT_EMPTY_OPENAI_FINISH` / `LEGIT_EMPTY_CLAUDE_STOP`:
length, tool_calls, content_filter, max_tokens, tool_use.
Two guards keep it from over-firing, one of which caught a real regression while
building this: the check applies only when bytes were forwarded, and only when
the body actually looked like SSE. A plain JSON completion travels through the
same wrapper and has no `data:` frames, so "no content seen" says nothing about
it — without the SSE gate, four existing stream tests failed.
The `if (done)` branch's reasoning moved into `resolveSilentCloseReason()`, which
also drops `pull` back under the function-length ceiling; cyclomatic lands at
2187 against a baseline of 2188.
This surfaces the error rather than failing over. Failing over would mean holding
every stream until its first content token, since the combo has already returned
leg 1's response by then — a much larger change. Surfacing the error satisfies
the issue's stated expectation ("surface the upstream error OR fail over") and
stops the retry loop, which is the reported harm.
Closes#8649
* fix(sse): stop thrashing the provider prompt cache for caching-aware clients
- shouldPreserveCacheControl: preserve client cache_control markers for every
combo strategy. The deterministic-strategy gate forced marker rewrites whose
per-request breakpoint positions are not stable turn-over-turn, thrashing the
upstream prompt cache (observed in production as ~200k cache_write tokens per
turn on quota-share combos). Preserving is never worse: on a stable target
the client's breakpoints advance deterministically; on a target switch both
approaches miss equally.
- prepareClaudeRequest: translator-path opt-in fallback — when preserve-mode
has nothing to preserve (client sent no cache_control anywhere), apply the
standard heuristic so requests never ship with zero cache breakpoints. The
claude-code-compatible relay path keeps its no-supplement contract.
- anthropic-beta: forward the client-negotiated context-1m-2025-08-07 through
the allowlist merge so a /model <id>[1m] client keeps its long-context
negotiation behind the proxy (never forced when the client did not send it).
* fix(sse): surface cache tokens in non-streaming OpenAI usage + finalize pending on quota-share block
- translateNonStreamingResponse (claude→openai): fold cache_read into
prompt_tokens and expose prompt_tokens_details.cached_tokens /
cache_creation_tokens, mirroring the streaming contract (#1426/#2215).
Non-streaming OpenAI clients behind a cached Claude upstream previously saw
prompt_tokens=<uncached remainder> (e.g. 23 for a ~9k request) with no cache
visibility.
- chatCore quota-share block: finalize the pending-request slot before
returning the policy 429 — the path never reaches upstream and the orphaned
pending lingered as a status-0 call-log row until the reaper swept it.
Validated live on the staging box (repro: blocked key → orphan row; after:
clean).
---------
Co-authored-by: diegosouzapw <diegosouzapw24@gmail.com>
`kieExecutor.createTask()` returns `JsonObject` (`Record<string, unknown>`), so
`createData.data` is `unknown` and the `createData?.data?.taskId` read that
image, video and music generation each duplicated could not compile. The same
three-line expression appeared verbatim in all three handlers.
`open-sse/utils/kieTask.ts` already holds two helpers with exactly this shape —
`normalizeKieTaskState()` and `parseKieResultJson()` both take `unknown`, guard
with `isJsonObject()` and return a declared type. `getKieTaskId()` follows them,
so the three handlers now share one guarded read instead of three unguarded ones.
`getKieCallbackUrl()` took `KieCallbackBody`, a weak type (all properties
optional). Passing a request body whose declared keys are `prompt` /
`timeout_ms` / `poll_interval_ms` tripped TS2559 "no properties in common" at
both music call sites. It receives arbitrary upstream request bodies, so it now
takes `unknown` and guards the same way its neighbours do; `KieCallbackBody`
had no other reference and is gone.
Behaviour is unchanged. `isJsonObject()` rejects arrays and null exactly where
optional chaining already yielded `undefined`, and the callers' `String(taskId)`
coercion moved inside the helper, so a numeric id still reaches `pollTask()` as
a string and a falsy id still takes the 502 branch.
Fixes 5 of the 208 `tsc -p open-sse/tsconfig.json` diagnostics with no new ones:
3 x TS2339 `taskId` on `unknown`, 2 x TS2559 on `KieCallbackBody`.
Refs #8484
Squash merge from local merge-train (Hard Rule owner-approved). Tip 029cdf4215cf465f0e1716ac9f84a84692b1e881 validated on 192.168.0.113: 26631/26653 pass.
* feat(db): cc discovery alias gate storage + EXPOSE_CC_DISCOVERY_ALIASES flag
Adds the gate for claude/<provider>/<model> discovery-alias mirror ids on
the /v1/models catalog: a new runtime feature flag (env forces on and wins
over the dashboard DB override), per-provider and per-model "on"/"off"/null
overrides stored in key_value under the ccDiscoveryAliases namespace, and a
pure precedence resolver (model > provider > global). Catalog wiring is a
separate follow-up task; this only lands the gate + storage.
* feat(sse): synthesize claude/ discovery aliases for the model catalog
* feat(api): advertise cc discovery aliases on /v1/models behind the 3-level gate
* feat(sse): resolve claude/ discovery aliases on the request path
* fix(sse): import getComboByName from db/combos, not the localDb barrel
* fix(sse): cover custom-node prefixes and the Codex WS bridge in cc discovery alias resolution
* feat(dashboard): cc discovery alias toggles + flag-screen env warning
Adds the operator-facing UI/API layer for the Claude Code discovery-alias
gate (claude/<provider>/<model> mirror ids on /v1/models): REST endpoint
for provider/model overrides, a provider-detail card with 3-state
(inherit/on/off) toggles, an info button on the Claude Code tool card
linking to Feature Flags, and an env-source warning on the
EXPOSE_CC_DISCOVERY_ALIASES flag card when it's forced on via env.
* feat(api): cc discovery usage metrics
* fix(api): record cc alias metric in the production wrapper + atomic counter upsert
* docs: document cc discovery aliases (Claude Code guide + feature flag catalog)
* fix(sse): don't mirror built-in auto/* combos as discovery aliases (advertised-but-unroutable)
* i18n(vi): translate the discovery-alias strings instead of shipping placeholders
vi is the one locale with a strict "no internal missing markers" test, so the 17
__MISSING__ entries this branch added (the provider ccAlias panel, the info
button, the feature-flag description and the env warning) would have turned that
test red the moment the base itself was repaired. Translated, keeping every ICU
placeholder ({modelId}, {error}) and the literal claude/<provider>/<model> id
shape intact.
* chore(quality): raise the frozen caps this feature legitimately grows
catalog.ts 1615 -> 1639: the alias synthesis is wired into the catalog builder,
which is where the per-key-filtered list is assembled — the only place the mirror
entries can be appended after model hiding has been applied.
localDb.ts 808 -> 810: two re-export lines for the new ccDiscoveryAliases db
module, which is exactly what the "Adding a New DB Module" recipe prescribes.
* refactor(dashboard,api): keep the complexity ratchets flat
The feature added four cyclomatic violations and one cognitive one, which the
ratchets reject — the baseline only moves when a metric improves. Split the new
code instead:
- appendCcDiscoveryAliases: the four skip-guards become isMirrorableId().
- resolveCcDiscoveryAliasStripWith: alias parsing and gate resolution become
parseCcAliasTarget() and resolveGateFor(), replacing a chain of ternaries that
each re-tested isComboAlias.
- FeatureFlagCard: the env-precedence warning becomes its own component instead
of a conditional branch inside an already-large render.
- ProviderCcAliasSection: the loader moves to useCcAliasData(), and the override
list and add-row become ModelOverrideList / AddOverrideRow, bringing both
oversized functions back under the 80-line rule.
Behavior unchanged — the 74 discovery-alias tests pass untouched. Both ratchets
now sit exactly at baseline (2188 / 971).
* feat(sse): selective upstream 4xx error passthrough util
* feat(sse): relay upstream 4xx error bodies verbatim on the Anthropic request path
createErrorResult() gains an opt-in 7th param opts.passthrough; when set and
the upstream body is eligible (per shouldPassthroughUpstreamError), the
returned response body is the upstream 4xx body verbatim instead of the
sanitized wrapper. Internal classification fields (error/rawMessage/
errorType/errorCode) are never affected.
Wired into chatCore.ts's 7 upstream-error createErrorResult call sites
(model_unavailable / context_overflow / generic upstream-error branches),
gated on sourceFormat === FORMATS.CLAUDE so only /v1/messages requests get
the verbatim body — OpenAI-format paths are unchanged.
* test(quality): register upstream-error-passthrough in stryker tap.testFiles
`tests/unit/upstream-error-passthrough.test.ts` covers `open-sse/utils/error.ts`,
an instrumented module, so `check:mutation-test-coverage --strict` requires it in
`tap.testFiles` — the gate flagged the drift on this PR.
* fix(sse): label HTTP 499 disconnects as client_disconnected
Preserve caller-supplied error type/code in buildErrorBody so stream
abort classification is not overwritten by the status-code table.
* docs(security): document buildErrorBody classification arg
---------
Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com>
buildClientRawRequest deep-cloned the ENTIRE request body on every chat request, unbounded.
On the #7847 incident payload (3.05 MiB, 729 messages, 86 tools) that retains 3.19 MiB per
request, and it is pure waste: every consumer of clientRawRequest.body is observability and
none of them keeps the full payload.
chatCore.ts -> reqLogger.logClientRawRequest no-op when the logger is disabled, otherwise
re-clones via cloneBoundedForLog (0.08 MiB)
chatCore.ts -> trackPendingRequest clientRequest, surfaced by /api/logs/[id]
chat.ts -> recordRejectedRequestUsage requestBody
None feeds dispatch, translation or the upstream request, so the snapshot is now taken with
cloneBoundedForLog: 3.19 MiB -> 0.08 MiB, a 41x reduction, and retention no longer scales with
history length. It stays a clone rather than an alias because body is rewritten downstream
(plugin onRequest hook, compression) and the log must show what the client actually sent.
Bounding at the entry means the logger re-bounds an already-bounded value, which exposed that
cloneBoundedForLog was NOT idempotent -- each container exceeded its own bound once the marker
was added, so a second pass truncated again:
arrays [marker, ...24 items] is 25 entries > 24, so the marker and one real item were
dropped and originalLength was rewritten as 25 instead of the true 729
objects 80 keys + _omniroute_truncated_keys is 81 > 80, so a real key was evicted to make
room for the marker and the dropped count was reported as 1 instead of 20
strings the marker was appended AFTER slicing to maxLength, so the bounded string was
longer than the bound
Without this the persisted log payload would have changed shape versus before the fix. All
three now keep the marker inside the budget and treat an already-bounded value as final;
verified end to end -- the marker still reports originalLength 729.
TDD: tests/unit/repro-7847-bound-client-raw-request.test.ts was written first and failed on
three assertions (unbounded retention, retention scaling with history, and the idempotence
precondition) before either change.
Two changes with one root cause: several hot paths built a full JSON string only to read
its .length, and one of them silently changed the answer.
1. CORRECTNESS -- combo's fallback-compression trigger
estimateTokens(JSON.stringify(attemptBody)) took the STRING branch of estimateTokens,
which is ceil(length / CHARS_PER_TOKEN) over the raw JSON. An inline base64 image is
then charged as if every character of the data URL were prose. Measured on a 200 KB
inline image:
via string (before) 50,039 tokens
via object (after) 1,231 tokens
a 40x over-count, tripping fallback compression on requests nowhere near the context
window. This is the same class #8368/#8401 fixed on the request path; the combo call
site was missed. Passing the object routes through extractImageTokens, which charges
images structurally. Text-only bodies are unaffected -- verified identical, and pinned
by a test.
2. ALLOCATION -- jsonLength()
Adds an exact serialized-length walker: same O(n) scan, no string. Used by
estimateTokens' object branch and by streamReadinessPolicy (which runs on every
streaming request and only ever used .length).
Exactness matters because every consumer feeds a threshold, so this is property-tested
against JSON.stringify over 4000 generated structures covering escaping, lone
surrogates, omitted values, non-finite numbers, toJSON, Date, Map, cycles and BigInt.
Anything outside the plain-JSON subset falls back to JSON.stringify for THAT SUBTREE
only, so an exotic leaf never forces the message history back onto the allocating path.
Honest scoping of the memory win: the string was always transient, and V8 collects it
efficiently, so this is not 3 MiB of retained heap. Measured allocation churn over 20 calls
on a 3.06 MiB body: 3.1 MiB -> 0.5 MiB, about 6x less. The #8549 benchmark row for this
mechanism measures a HELD string and therefore overstates it; the correctness fix above is
the larger deliverable here.
Eight TS2339s in stream.ts read properties off `never`. `never` here did not
mean unreachable code — it meant a type predicate was lying.
function isClaudeEventPayload(payload: unknown): payload is JsonRecord
...
const flushedParsed = bufferedPayload as JsonRecord;
const isClaude = isClaudeEventPayload(flushedParsed);
} else if (!isClaude) { // JsonRecord minus JsonRecord = never
The predicate answers "does this payload carry a Claude event type?" — a
question about contents, not type. A Claude event and a non-Claude one are both
JsonRecord, so `payload is JsonRecord` narrows nothing; applied to an argument
already typed JsonRecord it collapsed the negative branch to `never`, taking
`.id` and `.choices` with it.
Changed the return type to `boolean`. No call site depended on the narrowing:
both other callers pass the value to updateClaudeEmptyResponseLifecycle(), whose
parameter is `unknown`, and passthroughTailProcessor.ts already declares this
function as `(payload: unknown) => boolean` — so this aligns the definition with
how the codebase already consumes it.
280 -> 272, zero new, on a line-number-agnostic diff of the full tsc error set.
The diff is one line, and a type predicate has no runtime representation, so the
emitted JavaScript is unchanged.
No test added, and no gap to fill: the `never`-typed branch is real, reachable
and already covered from both sides — stream-numeric-ids.test.ts exercises
"normalizes numeric id in final chunk without trailing newline" (the flush path
with a non-Claude payload, i.e. the branch TS believed impossible) and "Claude
passthrough does not normalize numeric ids" (the other arm). 338/338 across the
48 SSE/passthrough suites.
No explanatory comment either: stream.ts sits exactly at its frozen file-size
cap (2887), so a single added line fails check:file-size. The reasoning lives
here and in the PR rather than costing a baseline bump on a frozen file.
Co-authored-by: backryun <busan011@ormbiz.co.kr>
First slice of the TypeScript 7 migration split requested on #7697: resolve the
type diagnostics under `open-sse/tsconfig.json` in the lowest-risk modules, with
no toolchain change. 12 diagnostics across 8 files, all outside the hot path —
`chatCore.ts` and `stream.ts` are deliberately left for a later, standalone slice.
Fixes, by cause:
* `Transformer.cancel` (progressTracker, sseHeartbeat, and stream.ts's existing
handler) — the WHATWG Streams standard defines `transformer.cancel(reason)` and
Node implements it (verified on v24: cancelling the readable side invokes it),
but `lib.dom.d.ts` still omits it from `Transformer`, so every such handler was
TS2353. These handlers clear the heartbeat/progress intervals when an SSE client
disconnects, so deleting them to satisfy the checker would leak a timer per
abandoned stream. The interface is patched in `open-sse/types.d.ts` instead.
* `earlyStreamKeepalive` — `SettledHandler` was discriminated by `ok: true | false`.
This workspace compiles with `strictNullChecks: false`, where a boolean-literal
discriminant narrows the positive branch but not the negative one, so reading
`.error` off the rejected arm did not type-check (the two `.response` reads
elsewhere in the file did, which is why only one site errored). Retagged with a
string discriminant, which narrows both branches under the same settings.
* `toolCallShim` / `openai-responses` — assigning back to a property declared
`unknown` resets the `typeof` narrowing, so the following comparison no longer
saw a number/array. Both now read through a local. The `Read` limit clamp is
behavior-identical: its two branches are mutually exclusive at READ_MAX_LIMIT 2000.
* `sanitizeToolResultId` — takes `unknown` but forwards to a `string` parameter; a
non-string id previously reached `.replace()` and threw. Coerced instead.
* `openaiHelper` — `opts = {}` inferred `{}`; typed as `FilterToOpenAIFormatOptions`.
* `cursorAgentProtobuf` — `Buffer.alloc(0)` infers `Buffer<ArrayBuffer>` under
@types/node 26 while the decoded field is `Buffer<ArrayBufferLike>`; the locals
now use bare `Buffer`, matching `requestMetadata` a few lines above.
Validation: 335 -> 321 diagnostics with zero new errors (full tsc error-set diff
against the base config). typecheck:core clean, lint clean, check:type-coverage
92.17% -> 94.17%. All 114 existing test files that import a touched module pass;
`plan3-p0.test.ts` fails identically with and without this change (it reads the
developer's real ~/.omniroute DB instead of a test-scoped DATA_DIR).
The new test covers the three behavioral surfaces rather than the refactors the
existing keepalive/heartbeat suites already hold: that `transformer.cancel()`
really fires and can clear an interval, the id coercion, and the limit-clamp bounds.
* fix(api): stop the 2000-token safety buffer from inflating usage.prompt_tokens in the client response (#8331)
* fix(sse): scope #8331's usage-buffer fix around Claude-Code-compatible providers
The #8331 fix correctly stopped folding the 2000-token context-window safety
margin into client-visible prompt_tokens/input_tokens/total_tokens for normal
API metering clients. But it also silently changed the response shape for
Claude-Code-compatible providers, whose own context accounting reads the
buffered number straight out of usage — regressing
tests/unit/cc-compatible-provider.test.ts (expected 2007, got 7).
Fold the computed context_budget_* fields back into the visible usage fields
for that one path only (applyClientUsageBuffer's new
preserveContextBudgetInVisibleUsage option, gated on the existing
isClaudeCodeCompatible flag in chatCore.ts). Every other caller keeps the
real, unbuffered #8331 numbers.
Live incident: streamed assistant text was losing the spaces BETWEEN words
(e.g. "Bilden är en riktig JPEG nu" -> "Bildenärenriktig JPEG nu") on the
Responses-API and Claude streaming paths.
stripInternalReasoningPlaceholder() (#8081/#8162) is called on every
individual delta.content chunk, and unconditionally called .trim() even when
its sentinel ("(prior reasoning summary unavailable)") was never present in
that chunk. Tokenizers commonly emit sub-word tokens with a leading space as
part of the token (e.g. " en", " riktig") -- each such chunk got its only
whitespace character (the inter-word space) silently trimmed away before
being appended to the accumulated message, while the words themselves stayed
intact. Punctuation-only chunks were largely unaffected, matching what was
observed live.
Only trims when the sentinel is actually present -- preserves the original
#8081 intent (collapse a placeholder-only chunk to "") without touching the
overwhelming majority of chunks that never contain it.
Co-authored-by: Markus Hartung <markus.hartream@gmail.com>
Regression: #8162 (port of #8081) added an unconditional `.trim()` to
stripInternalReasoningPlaceholder(), applied to every streaming
delta.content chunk across 3 call-sites (openai-to-claude.ts,
openai-responses.ts, responsesTransformer.ts). Leading/trailing
whitespace at a chunk boundary is a real word boundary between
streaming fragments; trimming it glues adjacent chunks together on
the client ("Hello, " + "world." + " Bye." -> "Hello,world.Bye.").
Fix: early-return via .includes() before the replaceAll+trim, so the
function is a true no-op when the sentinel is absent from the chunk.
Behavior when the sentinel IS present is unchanged.
Validation:
- tests/unit/streaming-reasoning-dedup-5786.test.ts: the "(A-guard)"
test was RED on the base branch ('Hello,world.Bye.' vs
'Hello, world. Bye.'); GREEN after the fix (4/4 passing).
- tests/unit/translator-resp-openai-to-claude.test.ts: added a new
multi-chunk boundary-whitespace regression test, proven RED against
the pre-fix code (12/13), GREEN after (13/13).
- No regressions in responses-transformer.test.ts (17/17),
responses-transformer-dense-output.test.ts (3/3), or the other
suites exercising the shared placeholder utility (160/160 total
across all consumers).
Refs #8162
Refs #8081
Claude→OpenAI translation was emitting a literal </think> into
delta.content for ordinary Chat Completions clients. Reasoning already
ships as reasoning_content, so default to suppress and keep
x-omniroute-thinking-marker: on as the #4633 opt-in.
* fix(ci): resolve upstream-inherited check failures
* fix(streamHandler): log trackPendingRequest decrement failures instead of swallowing
The clearPendingRequest function had an empty catch block around the
trackPendingRequest decrement call. If it threw, the pending request
counter stayed incremented — causing drift, false-positive rate limiting,
and masked overload conditions.
Now logs the error with context for observability.
Fixes#8141
---------
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
* fix(sse): Gemini TPM classification, combo-cooldown-wait for auto/quota-share, and target-timeout floor
Gemini TPM/RPM 429s were misclassified as QUOTA_EXHAUSTED because
sanitizeErrorMessage() truncates to the first line, hiding Google's
metric name and retry hint on lines 2-3. Added a rawMessage field
(internal-only, never reaches the client) and classifyGeminiQuotaMetricFromText()
to classify from the untruncated text, reordered ahead of the generic
credits/daily-quota checks.
Widened comboCooldownWaitEnabled (wait out a short transient cooldown
instead of crystallizing a 429/503) from quota-share-only to also cover
auto-strategy combos, and raised the wait ceiling to 65s/130s-budget/90s-cap
to match Gemini's ~60s TPM/RPM windows.
The per-target timeout (DEFAULT_COMBO_TARGET_TIMEOUT_MS, 120s) was shorter
than the new 130s cooldown-wait budget, so a target could get cut off
mid-wait with a synthetic 524 instead of completing the retry. Added
resolveComboTargetTimeoutMsForCombo()/isComboCooldownWaitEligible() in
comboConfig.ts to raise the per-target floor to budgetMs+buffer only for
wait-eligible strategies (auto/quota-share), verified live: a 12-request
concurrent burst against a TPM-exhausted combo went from 2/12 succeeding
(10 x 524) to 12/12 succeeding with zero 503/524.
Also: liveGeminiShared.ts's sendAndValidate now fails fast on a 503
instead of retrying past it, and the health dashboard + request logger
surface TPM stats alongside RPM/RPD.
Co-authored-by: Markus Hartung <markus.hartung@gmail.com>
* fix(sse): combo-exhausted rejection logs now capture request body + attempted models
recordRejectedRequestUsage() (the fast path for combo requests that never
reach handleChatCore, e.g. all targets locked by resilience cooldown)
hardcoded provider: "-" and never passed a request body to saveCallLog(),
so /dashboard/logs entries for these failures were nearly useless for
debugging: no way to see the client's request or which models were tried.
- recordRejectedRequestUsage() now accepts requestBody and persists it
through the existing saveCallLog() artifact mechanism (same path
handleChatCore's own logging uses).
- Added summarizeComboAttemptedModels(), which reads the combo's own model
list (always available, unlike the response's combo-diagnostics headers —
a model-level resilience-lockout skip never touches the
exhaustedProviders/exhaustedConnections sets those headers are built
from) to populate a real "provider" value instead of "-".
- Wired both into the call site in src/sse/handlers/chat.ts.
NOTE: unrelated to the Gemini TPM/combo-cooldown-wait fix on this branch —
landed here per operator request, to be split into its own branch/PR.
Co-authored-by: Markus Hartung <markus.hartung@gmail.com>
* feat(sse): synthetic streaming keep-alive event + 5-minute Gemini cooldown-wait ceiling
Many clients enforce a first-SSE-byte timeout, which made it unsafe to wait
out a longer upstream rate-limit cooldown on a streaming request — the
client would abandon the connection before any bytes arrived. This landed
in two parts:
1. Synthetic startup "thinking" event (OpenAI chat/completions format):
the already-existing withEarlyStreamKeepalive wrapper (open-sse/utils/
earlyStreamKeepalive.ts, wired into /v1/chat/completions, /v1/messages,
/v1/responses since #2544) opens the SSE stream immediately once a
request runs past its threshold, but only ever sent empty/no-op
keepalive frames. Added a `startupFrame` option (defaults to
`keepaliveFrame` — zero behavior change unless a route opts in) so the
very first frame can carry real content instead. Wired
OPENAI_STARTUP_THINKING_FRAME (a reasoning_content delta: "OmniRoute:
got request, sending to provider") into /v1/chat/completions only —
Claude Messages and Responses API formats both require a preceding
envelope event (message_start / response.created) that a synthetic
pre-dispatch frame can't safely fabricate without risking a duplicate
envelope once the real stream arrives, so those two routes keep their
existing (safe, proven) keepalive frames unchanged.
2. Raised the "wait out a known cooldown, then retry" ceiling to 5 minutes
for both retry mechanisms, now that a client-side first-byte timeout is
no longer a risk on the (opted-in) route:
- comboCooldownWait (auto/quota-share combos, open-sse/services/combo.ts):
maxWaitMs hard clamp raised 90s -> 300s (src/lib/resilience/settings/
normalize.ts); defaults raised to maxWaitMs:90s/maxAttempts:5/
budgetMs:300s. comboConfig.ts's resolveComboTargetTimeoutMsForCombo
already derives the per-target timeout floor from budgetMs, so it
tracks the new ceiling with no further changes.
- waitForCooldown (direct, non-combo model requests, src/sse/handlers/
chat.ts): this mechanism had NO cumulative cap before — only a
per-wait cap (maxRetryWaitMs) and a retry count (maxRetries), so
maxRetries x maxRetryWaitMs could exceed 5 minutes with no ceiling.
Added a budgetMs field (mirrors comboCooldownWait) to
WaitForCooldownSettings/CooldownAwareRetrySettings, threaded a
requestRetryBudgetLeftMs tracker through chat.ts's requestAttemptLoop
(mirrors combo.ts's comboCooldownBudgetLeftMs), and made
getCooldownAwareRetryDecision refuse to wait once the cumulative
budget is exhausted even if the single wait is under maxRetryWaitMs.
Co-authored-by: Markus Hartung <markus.hartung@gmail.com>
* fix(sse): extend the synthetic keep-alive thinking event to /v1/responses
Live incident (OpenClaw, log id 1784407081908-cbc24f): a /v1/responses
request to gemini/gemma-4-31b-it took 56s to produce a first byte and the
client disconnected (499 request_signal_aborted) — the same client-first-byte-
timeout problem the previous commit fixed for /v1/chat/completions, but
/v1/responses only had the generic bare-comment keepalive (no content), so it
wasn't covered.
Added RESPONSES_STARTUP_THINKING_FRAME: a self-contained synthetic reasoning
item (response.output_item.added -> reasoning_summary_part.added ->
reasoning_summary_text.delta -> reasoning_summary_part.done), opened AND
closed within this one frame rather than left dangling — it never carries a
response_id, so it can't collide with the real upstream response's own
independent response.created lifecycle that follows. Mirrors the abbreviated
delta+part.done close pattern open-sse/utils/stream.ts's own
emitSyntheticResponsesReasoningSummary already uses for real mid-stream
reasoning content.
Wired into src/app/api/v1/responses/route.ts via the startupFrame option
added in the previous commit.
Co-authored-by: Markus Hartung <markus.hartung@gmail.com>
* fix(sse): combo cooldown-wait vars reset every setTry, crystallizing a bogus 503 instead of waiting
Live incident (log id 1784416706646-51): a request to the "default" combo
(strategy=auto, maxSetRetries=3) hit a real Gemini TPM 429 on both gemma-4
targets, correctly classified as a short 40s rate_limit lockout — then
crystallized a 503 "all upstream accounts are inactive" in 6.9s instead of
ever reaching the cooldown-aware wait.
Root cause: `lastError`/`earliestRetryAfter`/`lastStatus` were declared with
`let` INSIDE the `for (setTry...)` loop body, so they reset to null at the
start of every set-try. When both targets lock out on setTry 0, every
subsequent setTry (1..maxSetRetries) pre-skips both targets via the
isModelLocked check with no real dispatch — so on the FINAL setTry (the only
one whose values the post-loop decision reads, since it's gated behind
`if (setTry < maxSetRetries) continue`), lastStatus was null, hitting the
"!lastStatus" branch (ALL_ACCOUNTS_INACTIVE 503) and completely bypassing the
comboCooldownWaitEnabled / earliestRetryAfter wait logic — even though a
real 429 with a known ~40s retry-after WAS observed on setTry 0.
This bug predates today's Gemini TPM work (any combo with maxSetRetries > 0
whose targets all lock out on the first pass was affected) but was masked in
existing tests: the "auto strategy (2 models...)" regression test uses
maxSetRetries: 0, so it only ever runs ONE setTry iteration and never
exercises the reset-on-retry path. It also explains why the dedicated
12-concurrent-request burst test passed cleanly — with concurrent requests,
timing variance meant some request's FINAL setTry iteration still had a live
target to dispatch to, giving lastStatus/earliestRetryAfter fresh data. A
single isolated request has no such luck.
Fix: hoist lastError/earliestRetryAfter/lastStatus to just inside
dispatchWithCooldownRetry, before the setTry loop, so they persist across
set-tries (still reset fresh on each recursive dispatchWithCooldownRetry()
call after a wait, which is correct). recordedAttempts/fallbackCount/
exhaustedProviders etc. are intentionally left per-iteration (unrelated to
this bug).
New regression test in tests/unit/combo-quota-share-cooldown-wait.test.ts
reproduces the exact live scenario (2 targets, both lock out on setTry 0,
maxSetRetries: 3) — confirmed red (503) against the pre-fix code, green
(200, waits and retries) against the fix.
Co-authored-by: Markus Hartung <markus.hartung@gmail.com>
* test(sse): extend live Gemini workload to Responses API + add large-context TPM test
Two additions to the live Gemini test suite, both live-verified against the
dev instance:
1. sendAndValidate() (tests/integration/liveGeminiShared.ts) now accepts an
apiFormat: "chat" | "responses" parameter, building the Responses-API
request shape (input array, max_output_tokens) and parsing its SSE events
(response.output_text.delta / response.reasoning_summary_text.delta /
response.completed) via the new readResponsesSSEStream(). Wired into two
new tests in live-gemini-workload.test.ts ([30]/[31]), mirroring the
existing Chat Completions streaming coverage. Verified live: 24/25 + 5/5
payloads succeeded end-to-end through the new code path (the one failure
was a ~300s test-client fetch timeout unrelated to the Responses API code
itself — a separate, not-yet-addressed test-harness limitation).
2. genHugeContextMessage() builds a single message large enough (~4
chars/token estimate) to approach or exceed Gemini's free-tier TPM ceiling
(16000 input tokens/min for gemma-4) by itself. Every other prompt
generator in this file tops out around 1-2k tokens — nowhere near that
ceiling — so none of the existing workload tests ever exercised a REAL TPM
429, only RPM-style rate limiting. tests/integration/gemini-large-context-tpm.test.ts
sends two ~12-13k-token requests back-to-back (comfortably exceeding
16000/min together) to exercise the full path against production Gemini:
TPM classification, the comboCooldownWait retry, and the synthetic
keep-alive frame on a genuinely slow request.
Co-authored-by: Markus Hartung <markus.hartung@gmail.com>
* fix(sse): abandoned combo target dispatch now observes its own per-target timeout, fixing a permanent "pending" dashboard leak
Live incident (dashboard log id 1784418258231-14961a, reported as "an ongoing
request even though there's already a 200"): a combo target dispatch
abandoned by comboTargetTimeoutMs (open-sse/services/combo/targetTimeoutRunner.ts)
left a permanent phantom "pending" entry in the dashboard, even after the
overall combo request had already succeeded via a different retry.
Root cause: chatCore.ts's createStreamController — and everything downstream
that depends on it (withRateLimit's Promise.race against Bottleneck,
acquireAccountSemaphore) — only ever watches clientRawRequest.signal, which
is the ORIGINAL client's request signal (set once via buildClientRawRequest
and reused unchanged across every target dispatch in a combo). It has no
connection to targetTimeoutRunner.ts's OWN AbortController
(target.modelAbortSignal), which is what actually fires when
comboTargetTimeoutMs (300s) elapses. src/sse/handlers/chat.ts's
handleSingleModel bridge between combo.ts and handleSingleModelChat received
`target.modelAbortSignal` but silently dropped it — never forwarded it
anywhere. So when a target got abandoned (e.g. stuck inside a wedged
Bottleneck rate-limiter queue, see the WEDGED force-reset log line from the
same incident), its per-target timeout fired and let the COMBO move on and
retry successfully elsewhere — but the abandoned dispatch's own promise
chain never learned it had been superseded, so it hung forever waiting on a
signal that was never going to fire, and trackPendingRequest(false) (the
finalize call) never ran.
Fix: thread target.modelAbortSignal through as a new modelAbortSignal
runtimeOption, and merge it into clientRawRequest.signal (via the existing
mergeAbortSignals helper from open-sse/executors/base.ts) right before
dispatch, so an abandoned target's own promise chain now observes its abort
and can reach its cleanup path — new resolveDispatchClientRawRequest() makes
this mechanically testable in isolation.
Co-authored-by: Markus Hartung <markus.hartung@gmail.com>
* fix(sse): combo cooldown-wait state recording, rate-limit wedge recovery, OpenAI-format SSE error frames
Five related fixes surfaced by live incidents (dashboard log ids 1784457764961-73,
1784465227489-a2cbc0, 1784504040241-6f8b9a) while validating the Gemini TPM/cooldown-wait
work on this branch against real OpenClaw traffic:
- combo.ts: the model-lockout bail-out branches in dispatchWithCooldownRetry never
recorded lastStatus, so once every target in a set hit an existing lockout the final
check crystallized a bogus ALL_ACCOUNTS_INACTIVE 503 instead of reaching the
cooldown-wait decision, even with a real 429 + short retry-after observed.
- combo.ts/combo/types.ts: the "all credentials cooling down" pre-dispatch rejection
(buildModelCooldownBody) nests its retry hint as error.retry_after/reset_seconds, not
the top-level retryAfter every other 429 shape uses — combo's extraction only read the
latter, so earliestRetryAfter stayed null for this shape even after lastStatus was fixed.
- rateLimitManager.ts: the wedge-recovery watchdog used disconnect(), which releases the
heartbeat timer but never rejects jobs already QUEUED on that instance — orphaned
dispatches hung until the outer ~300s per-target timeout, well past real clients'
patience. Switched to stop({ dropWaitingJobs: true }), safe because the wedge condition
already requires RUNNING===0 && EXECUTING===0.
- earlyStreamKeepalive.ts: the in-band error frame emitted after committing to a 200 SSE
stream was hardcoded to Anthropic's `event: error` convention for every route, including
the OpenAI-format ones (/v1/chat/completions, /v1/responses) where that framing is
either invisible or malformed to a plain data-line parser. Added per-route
OPENAI_CHAT_ERROR_FRAME / OPENAI_RESPONSES_ERROR_FRAME and wired them in.
- chatCore.ts: persisted a synthetic clientResponse error body even when the client had
already disconnected (AbortError) before that body was ever computed — misleading the
dashboard into showing "what the client received" for a response that was never sent.
Also: RequestLoggerDetail.tsx — Provider/Client Event Stream panes lost their collapse
toggle when StreamSection replaced the collapsible PayloadSection (692d6be80, unifying
active/finished request views) without carrying the toggle over.
Each fix has a TDD regression test with a confirmed red-before-green cycle.
Co-authored-by: Markus Hartung <markus.hartung@gmail.com>
* test(sse): free-tier model + gemma-4 TPM-ceiling benchmark harness
Adds a live benchmark comparing free models OmniRoute exposes across
configured providers plus previously-unexercised no-auth providers
(felo-web, aihorde, opencode, duckduckgo-web — none need a connection
row, they were just never tried). Reuses liveGeminiShared.ts's SSE
parsers and CASE_BUILDERS instead of duplicating them.
Also adds a targeted TPM-stress test firing back-to-back large-context
prompts at the gemma-4-31b model across its 3 free hosts (Gemini,
NVIDIA, AI Horde) to isolate whether the documented 16k-tokens/minute
free-tier ceiling is Gemini-specific enforcement or an inherent
model property.
FORCE_TOOL_CHOICE_REQUIRED is a test-only, default-off env flag added
to liveGeminiShared.ts and live-gemini-agentic-loop.test.ts for an
earlier live A/B comparison of tool_choice: required vs unset — kept
as a reusable knob for future runs.
Co-Authored-By: Markus Hartung <markus.hartung@gmail.com>
* test(sse): benchmark for the 2026-07-22 newly-enabled provider batch
Adds NEWLY_ENABLED_MODELS to freeModelBenchmarkShared.ts (Mistral
Leanstral, OpenRouter's live "free"-tagged roster, OpenCode Zen's
current free models — refetched live from
https://opencode.ai/zen/v1/models since the static catalog had
drifted) and a dedicated workload benchmark test for them.
Co-Authored-By: Markus Hartung <markus.hartung@gmail.com>
* test(sse): sync geminiRateLimitTracker tests with e74a1722b's corrected Gemma 4 limits
e74a1722b updated geminiRateLimits.json's gemma-4-* entries from the stale
15/1500/-1 (rpm/rpd/tpm) to the real published free-tier values
16000/14400/16000, but never updated the tests asserting the old numbers.
Surfaced by running the full test:unit suite as a post-rebase sanity check.
Co-Authored-By: Markus Hartung <markus.hartream@gmail.com>
* chore(quality): file-size baseline for own-growth (#8213)
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: Markus Hartung <markus.hartung@gmail.com>
Co-authored-by: Markus Hartung <markus.hartream@gmail.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
* fix(sse): synthesize tool_calls for Gemini's malformed function-call abort reasons
Live incident (dashboard log id 1784489701456-d8c0e9): Gemini terminates a
stream with finishReason MALFORMED_FUNCTION_CALL/UNEXPECTED_TOOL_CALL when its
own parser rejects an attempted tool call — there's no real functionCall part,
only a human-readable finishMessage. gemini-to-openai.ts passed this through
raw as finish_reason (9router#2462's fix, correctly keeping it off a clean
"stop"/Claude end_turn), but a raw "malformed_function_call" isn't one of
OpenAI's 5 documented finish_reason values, so a real OpenAI-format client
(OpenClaw) has no handling for it at all and silently never notices the turn
failed — confirmed live via tests/integration/live-gemini-workload.test.ts's
[28] streaming case after the Gemini TPM/rebase work on this branch.
Fix: synthesize a tool_calls entry (arguments carry the error code + Gemini's
finishMessage, valid JSON) and finish_reason: "tool_calls" instead, routing
the failure into the ordinary "tool call arguments didn't parse" path every
OpenAI-compatible agent loop already handles. Defers to a real tool call if
one already completed earlier in the same turn — the real call wins, no
synthetic entry piles on top of it.
Tests (TDD, each confirmed red-before-green):
- 5 new unit tests in the existing 9router#2462 regression file, covering the
synthesis itself, UNEXPECTED_TOOL_CALL, the real-tool-call-wins edge case,
and no-regression on a clean STOP.
- New fixture (tests/fixtures/translation/gemini-malformed-function-call-stream.json):
the real 6-chunk event series from the live incident, sanitized (personal
paths/URLs replaced with generic placeholders, structure preserved exactly).
- New integration test chains the real translator into the real Responses API
transformer using that same fixture, proving correct behavior on BOTH
/v1/chat/completions and /v1/responses from one shared ground-truth event
series.
Co-authored-by: Markus Hartung <markus.hartung@gmail.com>
* fix(sse): don't drop a malformed tool-call failure when it lands beside a real one
Live incident (dashboard log id 1784589106014-2a42f8), analyzing why the
prior malformed-function-call fix (3568c7259) still wasn't reaching the
client in this case: Gemini can emit a REAL, valid functionCall AND finish
the SAME candidate with MALFORMED_FUNCTION_CALL — the model attempted
multiple tool calls in one turn (here: a real status-check call plus a
malformed "exec"+"cron" multi-call attempt), one parsed cleanly and the
other didn't.
The first fix version skipped synthesizing a failure signal whenever a real
tool call already existed (state.toolCalls.size > 0), on the assumption
that meant the model was retrying a LATER, separate attempt after an
earlier one already succeeded. That's indistinguishable, from the
translator's state, from this same-turn case — so it silently discarded
the malformed attempt's information entirely: the client saw the real call
succeed and never learned the other tool calls were attempted and rejected.
Fix: always synthesize the failure entry when a malformed abort reason is
seen, appending it alongside any real tool call rather than skipping it.
Multiple tool_calls in one response is normal, well-supported OpenAI
behavior (parallel tool calls), so this adds the failure as an additional
entry instead of replacing or hiding the real one.
Tests (TDD, confirmed red-before-green):
- Rewrote the unit test that encoded the old (wrong) assumption to assert
both the real and synthesized calls are present.
- New fixture (gemini-malformed-function-call-parallel-real-call-stream.json):
the real event series from this incident, sanitized.
- New integration tests (same file as 3568c7259's) prove both
/v1/chat/completions and /v1/responses surface both tool calls correctly
from this fixture.
Co-authored-by: Markus Hartung <markus.hartung@gmail.com>
* feat(sse): honor tool_choice when translating OpenAI requests to Gemini
Investigating a live report that gemini-3.1-flash-lite frequently narrates
an intended tool call in plain text instead of actually emitting one
(dashboard log id 1784591483850-49c408 — 9 raw provider chunks, all plain
text, zero functionCall parts, clean finishReason STOP): body.tool_choice
was never read anywhere in the OpenAI->Gemini request translator.
result.toolConfig was unconditionally hardcoded to
{ functionCallingConfig: { mode: "VALIDATED" } } whenever tools were
present, regardless of what the caller sent. VALIDATED lets the model
respond with plain text OR a schema-validated function call at its own
discretion — it never forces a call the way OpenAI's tool_choice:
"required" (Gemini's ANY mode) does, so a caller had no way to compel a
tool call even when explicitly requesting one.
Added convertOpenAIToolChoiceToGemini(), mirroring the existing
convertOpenAIToolChoice() in openai-to-claude.ts for the same OpenAI
tool_choice shapes (string "auto"/"none"/"required", or
{type:"function",function:{name}} to force one specific tool):
- unset/"auto" -> VALIDATED (unchanged default, no regression)
- "required"/"any" -> ANY (forces a call)
- "none" -> NONE (disables function calling)
- {type:"function",...} -> ANY + allowedFunctionNames: [name]
Wired into both Gemini request paths: the direct/base translator
(openaiToGeminiBase) and the Antigravity/Cloud Code envelope
(wrapInCloudCodeEnvelope), which now reuses the base translator's already-
computed toolConfig instead of re-deriving its own hardcoded VALIDATED.
This unblocks (but does not itself resolve) the live question — a
tool_choice: "required" A/B test against gemini-3.1-flash-lite follows to
confirm ANY mode actually changes the narrate-vs-act behavior in practice.
Also updates the T11 any-budget allowlist for this file: the "any" string
comparisons (tool_choice value "any", not a TypeScript type) are the same
documented false-positive pattern already carved out for executors/base.ts.
Tests (TDD, confirmed red-before-green): 9 new unit tests covering all
tool_choice shapes on both the direct and Antigravity/Cloud Code paths,
plus the no-tools and unset-default no-regression cases.
Co-authored-by: Markus Hartung <markus.hartung@gmail.com>
* chore(quality): file-size baseline for own-growth (#8211)
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: Markus Hartung <markus.hartung@gmail.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
* fix: strip internal reasoning placeholder from user-visible content (#8081)
The internal reasoning replay sentinel '(prior reasoning summary unavailable)'
can leak into user-visible assistant content when a model echoes it through
ordinary message.content / delta.content. Existing suppression only checked
reasoning_content fields and reasoning-specific events.
Changes:
- Add stripInternalReasoningPlaceholder() to reasoningPlaceholder.ts —
removes all occurrences of the sentinel and trims; returns '' when
nothing meaningful remains
- Streaming: strip in responsesTransformer.ts, openai-responses.ts, and
openai-to-claude.ts at the delta.content entry point; skip emission
entirely when only the placeholder was present
- Non-streaming: strip in responseSanitizer.ts sanitizeMessageContent()
and sanitizeResponsesMessageContent() (all three text paths)
translateText is unaffected (uses mode='translate' via plain newsClient).
The per-provider reasoning_content check remains as defense-in-depth.
* fix: skip only empty content block on reasoning-placeholder, keep finish_reason/tool_calls (#8081)
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
* chore(quality): rebaseline openai-responses.ts own-growth (#8081 guard)
---------
Co-authored-by: Austin Liu <austinliu@Austins-MacBook-Air-3.local>
Co-authored-by: Probe Test <probe@example.com>
Co-authored-by: Dingding-leo <Dingding-leo@users.noreply.github.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
* fix(responses): close namespace round-trip for Responses-Chat translation (#7936)
The #7905 custom-tool-call path landed in release/v3.8.49 but left #7936
open: Responses namespace sub-tools were flattened to a bare leaf on the
Chat wire with no response-side closure, so Codex's adjudicator rejected
every namespace sub-tool call with `unsupported call` -- it only has a
dispatch entry for the header bits (namespace+name), no entry for the
bare leaf.
This patch closes the round-trip without mutating the Chat wire name
(alignment with #7905's bare-leaf contract and with the issue author's
proposed fix):
* request side (openai-responses.ts): keep tool.function.name as the bare
leaf, populate a side-band namespaceToolIdentityMap keyed on that leaf,
and thread it through translatedBody._toolNameMap.
* request -> response seam (chatCore.ts): extract the identity map before
dispatch and pass it through to the non-stream completion path and to
all three stream pipelines (translate openai-responses, translate
other, passthrough).
* response translator (response/openai-responses.ts): in emitToolCall
(response.output_item.added) and closeToolCall (custom_tool_call /
function_call output_item.done), call resolveRequestToolIdentity() to
rewrite the bare leaf back to {namespace,name} and emit codex-compatible
independent fields.
* passthrough (utils/stream.ts): add a response passthrough rewriter
restoreResponsesPassthroughFunctionCallIdentity that intercepts
response.output_item.added, response.output_item.done, and
response.completed and stamps the same {namespace,name} tuple.
* helper (requestToolIdentity.ts): a 20-line stateless resolver; never
parses a name.
The wire-visible Chat tool.function.name stays the bare leaf -- non-OpenAI
providers (NVIDIA, GLM, Kimi, Gemini, ...) frequently truncate or rewrite
long __-dotted names; bare leaves avoid that failure mode entirely. The
codex ResponseItem::FunctionCall schema (models.rs) declares an
independent namespace: Option<String> field and has a
function_call_deserializes_optional_namespace round-trip test, so
emitting it separately matches the codex adjudicator dispatch.
Includes 19 new test cases across 4 files:
- request-side bare-leaf wire + side-band ledger construction
- response-side tuple emit + unmapped passthrough + apply_patch exclusion
- ambiguous-leaf collision safety (entry dropped, leaf emits verbatim)
- per-request isolation between concurrent streams
- a precompiled Atlassian-style nested namespace override
* fix(responses): skip tool_search_call input items instead of 400 (#7936 addendum)
Codex 0.42+ emits `tool_search_call` (and later `tool_search_result`) input
items when the model uses the dynamic tool-search optimization. They are
metadata-only: they record that the model queried a subset of the
available tools, and carry nothing that OpenAI Chat Completions can
represent.
Without an explicit skip in openai-responses.ts, the input loop threw
Unsupported Responses API feature: input item type 'tool_search_call'
cannot be represented in Chat Completions -- and because these items
stay in the Responses API `input` for every follow-up turn, the whole
server returned 400 on EVERY subsequent /v1/responses in the same
session until the user cleared history.
Observed in the wild:
/v1/responses 400 "Unsupported Responses API feature: input item
type 'tool_search_call' cannot be represented in Chat Completions
[longcat/LongCat-2.0 (400), longcat/LongCat-2.0 (400)]"
The meituan combo (longcat fallback) was the most visible victim, but
the underlying throw is source-format-side and hits any Responses-API
consumer whose upstream does not natively support Responses.
Fix: stop on the item type the same way `reasoning` is skipped --
display-only metadata, no chat side-effect. Covers both
`tool_search_call` and the follow-up `tool_search_result` shapes.
Adds 3 unit tests:
- tool_search_call is silently skipped (no 400)
- tool_search_result is silently skipped
- tool_search_call items interspersed with real messages are skipped
in order; real messages survive
* chore(quality): rebaseline openai-responses.ts + stream.ts own-growth (#7936 namespace round-trip)
---------
Co-authored-by: TonPro <hello@tonpro.fu>
Co-authored-by: RCrushMe <RCrushMe@users.noreply.github.com>
* feat(media): Adobe Firefly image + video generation provider
Add unofficial adobe-firefly media provider with full OpenAI-compatible
image and video generation: Nano Banana / GPT Image families, Sora 2,
Veo 3.1 (standard/fast/reference), and Kling 3.0.
Supports browser session cookies (auto IMS token exchange) or direct
IMS access tokens, async submit-and-poll against Firefly 3P endpoints,
aspect-ratio and resolution controls, and multi-account web-session UX.
Chat completions are intentionally rejected (media-only surface).
Includes unit coverage for registry wiring, payload builders, auth
resolution, and mocked generate happy-paths.
* fix(adobe-firefly): clio auth, discovery fallback, credits balance
Root-cause 401 invalid token against live firefly.adobe.com captures:
generate/discovery use x-api-key + IMS client_id clio-playground-web
(not projectx_webapp). Align headers/origin, dual cookie to IMS exchange
(clio first, Express fallback), BKS poll rewrite for /jobs/result.
Models: parse POST /v2/models/discovery + static fallback catalog from
adobe/get_models.txt; expand image/video registries.
Limits: GET firefly.adobe.io/v1/credits/balance (SunbreakWebUI1) with
total/remaining + free/plan detail quotas. Clarify cookie vs JWT UX.
Unit tests: 27/27 pass.
* fix(adobe-firefly): reject guest tokens from page-only cookies
Live repro with firefly.adobe.com Cookie export: IMS check with
guest_allowed=true returns account_type=guest (no AdobeID). That token
fails generate (401 invalid token) and credits/balance (403
ErrMismatchOauthToken). guest_allowed=false needs adobelogin.com IMS
session cookies which are not present in a page-only Cookie paste.
- Detect/reject guest JWTs; clear error tells user to paste Bearer JWT
- Prefer user JWT from HAR/mixed paste; improve credential extraction
- Update web-cookie + credential UX to recommend Authorization Bearer
Unit tests 29/29.
* fix(adobe-firefly): production auth, Limits, and 408 load handling
Live validation against firefly.adobe.com + packaged VibeProxy:
Auth / credentials
- Prefer IMS user JWT (Bearer from firefly-3p); reject guest tokens from
page-only cookies with an actionable error
- Extract JWT from Bearer, access_token=, IMS sessionStorage tokenValue,
and mixed HAR pastes; prefer non-guest tokens
- Strip JWT from Cookie header (undici Headers.append crash on mixed paste)
- Keep sherlockToken → x-arp-session-id + sanitized Cookie for generate
Limits
- credits/balance → Record quotas (firefly_total / free / plan) so
providerLimits caches them (arrays were ignored)
- Allowlist adobe-firefly + firefly in USAGE_SUPPORTED + APIKEY limits
- Live: 10000 plan credits parsed end-to-end after refresh
Generate
- Browser-shaped gpt-image body (size auto, no extra top-level size)
- Exponential 408 "system under load" retries (8 attempts) with clear
client message that 408 is Adobe capacity, not invalid token
- Live: generate returns proper 408 under load; balance/models stay 200
Tests: adobe-firefly unit suite 33/33 pass.
* fix(adobe-firefly): match live capture headers; add gpt-image-2
- Do not send firefly.adobe.com Cookie to firefly-3p (wrong-origin; soft 408)
- Lift sherlockToken only into x-arp-session-id
- Poll headers match status_check.txt (Bearer + accept, no x-api-key)
- Catalog gpt-image-2 alias → upstream modelVersion "2" (GPT Image 2)
- Shorter 408 retry budget so clients fail fast with clear message
- Unit suite 34/34
* fix(adobe-firefly): always send x-arp-session-id on generate (fixes 408)
Root cause of Bearer JWT → HTTP 408 colligo "system under load":
submit only set x-arp-session-id when sherlockToken was present in a
cookie paste. JWT-only credentials never sent the header, and Adobe
soft-blocks those requests with instant 408 (x-colligo-timeout:0.0).
A/B against a real user IMS token:
- det nonce + synthetic ARP → 200
- random nonce + synthetic ARP → 200
- det nonce without ARP → 408
Match adobe2api / GPT2Image-Pro:
- buildAdobeSubmitNonce = sha256(user_id + prompt[:256])
- buildAdobeArpSessionId = base64({sid, ftr}) synthetic session
- buildAdobeSubmitHeaders always sets both headers
Live adobeFireflyGenerateImage end-to-end: submit + poll → S3 presigned URL.
* fix(adobe-firefly): drop literal cred fallbacks + type-clean tests
Addresses pre-merge review feedback on #8006:
- Removes the `|| "literal"` fallback after resolvePublicCred() in
adobeFireflyApiKey()/adobeFireflyExpressClientId()/adobeFireflyBalanceApiKey()
(open-sse/services/adobeFireflyClient.ts). resolvePublicCred() already
always returns the decoded embedded default, so the literal fallback
was dead code that reproduced the exact env-or-literal anti-pattern
docs/security/PUBLIC_CREDS.md documents as BAD (Hard Rule #11).
- Replaces the 11 `@typescript-eslint/no-explicit-any` casts in
tests/unit/adobe-firefly.test.ts with concrete types
(Record<string, unknown>, Headers, Error-narrowing on the
assert.rejects predicate), matching the pattern already used
elsewhere in this suite. `no-explicit-any` is a hard ESLint error
under tests/ in this repo.
- Freezes file-size baseline entries for the new
open-sse/services/adobeFireflyClient.ts (1958 LOC, new-file cap 800,
mirrors the qoderCli.ts precedent for a legitimately large new
provider client), open-sse/config/imageRegistry.ts (800->821, new
adobe-firefly registry entry) and the +3 LOC growth in
src/lib/usage/providerLimits.ts (1000->1003).
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com>
Co-authored-by: artickc <artickc@users.noreply.github.com>
* perf: startup parallelization, stream TextEncoder lift, auth middleware bottlenecks
Startup (~100-300ms faster cold start):
- Parallelize 4 early imports via Promise.all() in registerNodejs()
- Parallelize 10 independent background services via Promise.allSettled()
- Each service has independent try/catch — no failure domino effect
Streaming pipeline (8 fewer TextEncoder GC allocations per SSE event):
- Lift new TextEncoder() from per-chunk inside buildClaudeStreamingResponse
to function scope alongside existing decoder singleton
Auth middleware bottlenecks (from PerfBottleneckAnalysis):
- Backoff decay loop: replace updateProviderConnection (full CRUD:
SELECT+encrypt+cache-invalidate+backup) with resetConnectionBackoff
(targeted UPDATE of backoff/error columns only)
- Dual .filter() for quota: replace two passes calling
isQuotaExhaustedForRequest per connection with a single for loop
partitioning into withQuota/exhaustedQuota
- Debug-log filter recomputation: capture connectionFilterStatus Map
during the filter pass; debug loop reads 6 string comparisons instead
of 6 function calls per connection
Supporting:
- Add resetConnectionBackoff to src/lib/db/providers.ts (patterned after
clearConnectionErrorIfUnchanged, no CAS check)
- Re-export resetConnectionBackoff from src/lib/localDb.ts
- Update integration-wiring.test.ts regex for parallelized dynamic import
* perf: P2C quota cache, lazy provider init, structuredClone elimination, getSettings→getCachedSettings
- **auth.ts: P2C quota re-evaluation cache** — quotaResults Map threaded
through selectPoolSubset → compareP2CConnections → getP2CConnectionScore.
Populated during filter + partition passes, eliminating redundant
evaluateQuotaLimitPolicy / isQuotaExhaustedForRequest calls when the
P2C comparator re-evaluates previously-scored connections.
- **constants.ts: lazy PROVIDERS via Proxy** — replaces eager
generateLegacyProviders() + loadProviderCredentials() at module load
with Proxy delegating to deferred init on first property access.
- **providerModels.ts: lazy PROVIDER_MODELS + PROVIDER_ID_TO_ALIAS** —
same Proxy pattern for both exports; generateModels()/generateAliasMap()
deferred until first read.
- **stream.ts: structuredClone → minimal object spread** — replaces
O(n) deep clone of SSE response chunks with targeted reconstruction
of only mutated fields (usage, delta.content, finish_reason).
- **progressTracker.ts: TextDecoder lift** — module-level decoder
instead of per-chunk new TextDecoder().
- **Route files: getSettings() → getCachedSettings()** — 13 API route
files converted from uncached per-request DB reads to TTL-cached
wrapper (5s default), eliminating redundant queries on every request.
- **settings.ts: re-export getCachedSettings** from readCache for
non-localDb consumers.
- **Remove settingsCache.ts** — dead file, no imports reference it.
TS compile: 0 errors. Auth tests: 225/225 pass. Services: 269/269 pass.
* perf: Phase 1 tangible wins — egressCache eviction, mmap_size PRAGMA, composite indexes, proxyFallback lazy import
- egressCache: lazy TTL eviction on getCachedEgressIp access (bounds memory
leak to distinct proxy URLs, typically <100)
- mmap_size: apply stored PRAGMA from key_value table (256MiB default) after
applyStoredDatabaseOptimizationSettings — setting was stored but never applied
- schemaColumns: add idx_uh_provider_model_timestamp (covers getModelLatencyStats)
and idx_pc_provider_auth_type (covers 6+ provider_connections queries)
- proxyFallback: convert static import to dynamic import() inside error handler
(defers 210ms module load from startup to first proxy-retry scenario)
* perf: add dedup expression index, unref() sweep timers
- Add COALESCE expression index idx_uh_dedup on usage_history
matching the exact dedup query pattern. Eliminates FULL TABLE
SCAN on every saveRequestUsage insert.
- Add composite idx_uh_provider_model_timestamp on usage_history.
- Add composite idx_pc_provider_auth_type on provider_connections.
- Add .unref() to setInterval in batchProcessor.ts (polling loop).
- Add .unref() to setInterval in runtimeHeartbeat.ts (heartbeat).
* perf: bump SQLite cache_size default from 16MB to 64MB
New installs now start with 64MB page cache (was 16MB). Existing
users' stored settings are unchanged. Reduces disk reads for the
typical ~250MB database by keeping ~25% of pages in memory.
Also resolved pre-existing merge conflict in webhooks.ts.
* docs: add Redis production config guide and proxy port clash investigation report
- docs/redis-production-config.md: comprehensive Redis tuning guide
covering client options, server config, Docker settings, scaling,
and monitoring for all three Redis workloads (rate limiting,
auth cache, quota store)
- docs/proxy-port-clash-report.md: investigation confirming proxy
subsystem has no port binding issues; real EADDRINUSE history
traced to process supervisor crash-loop restart race (#4425) and
live-dashboard port clash (#6324), both already fixed
* fix: address PR #7893 review — add Proxy traps, extract migrations to reduce providers.ts size
- Add set trap to PROVIDER_ID_TO_ALIAS Proxy (providerModels.ts)
- Add deleteProperty traps to all three lazy Proxies (PROVIDERS,
PROVIDER_MODELS, PROVIDER_ID_TO_ALIAS)
- Extract autoMigrateLegacyEncryptedConnections and getGheCopilotHosts
from providers.ts (1129→1036 lines, -93) into providers/migrations.ts
- Both functions re-exported via providers.ts for backward compat
File-size ratchet resolved: src/lib/db/providers.ts now 1036 lines.
* fix: resolve merge conflict markers in 3 route/test files
- model-combo-mappings/route.ts: kept upstream version (Zod pagination
via validateBody + isValidationFailure), restored missing return
statement for GET handler
- playground/presets/route.ts: kept stashed version details (satisfies
type-narrowing + inlined Response) — functionally identical
- error-sanitization.test.ts: matches upstream exactly (no diff)
Test verification: same 7 pre-existing failures confirmed on upstream
baseline (c1bdd91e7). Zero regressions from conflict resolution.
Closes remaining uncommitted work from PR #7046 rebase.
* test(db): add resetConnectionBackoff coverage + fix file-size ratchet regression
- Add tests/unit/reset-connection-backoff.test.ts: covers the new
resetConnectionBackoff lightweight-UPDATE helper (clears backoff/error
columns and re-activates a connection, unconditional-write behavior on
terminal statuses, no-op for empty/unknown ids). Zero prior coverage
per pre-merge review of PR #7893 (Hard Rule #8).
- open-sse/services/batchProcessor.ts: fold the new unref() call into the
existing setInterval(...).unref() chain instead of a separate statement,
keeping the file at the frozen 915-line ratchet (Timeout.unref() already
returns `this`, so no type cast is needed).
- src/lib/localDb.ts: drop one blank separator line so the new
resetConnectionBackoff re-export stays within the frozen 808-line ratchet.
Pre-merge fix for PR #7893 (perf/startup-stream-auth-optimizations) per
/green-prs plan-file _tasks/pipeline/prs/1-analyzed/7893-*.plan.md.
Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: oyi77 <oyi77@users.noreply.github.com>
Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com>
Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouzapw@users.noreply.github.com>
