mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-08-06 23:32:12 +03:00
fix(quality): 2 production bugs + 24 unit base-reds + measured gate ceilings (#9529)
* fix(quality): resolve net-new lint errors and allowlist #9343 assert rewrite Two `no-explicit-any` errors landed with #9407 and #9320 after the suppressions inventory was generated. Project policy is to fix new violations rather than freeze them, so both are typed instead: - #9407: `executor as unknown as Record<string, unknown>` - #9320: `(k: { name?: string })` Also allowlists the net-assert reduction in web-tools-translation-2820 (39->35). #9343 inverted the contract — bare JSON must no longer be promoted to tool_calls without an explicit <tool> envelope — so the tests were rewritten to assert non-promotion, which costs fewer asserts than validating a promoted object. More restrictive, not weaker. * fix(quality): raise integration ceiling to 40min and unpin codex-cli version in test The integration gate's 20min ceiling killed a healthy run: measured 22m08s hermetic on an idle 16-core box (935 tests across 112 files, strictly serial at --test-concurrency=1 because ~16 of them bind a port or share a DB). The "~3-10min" estimate in the code was stale by ~3x. 40min keeps the ceiling's real purpose — turning a genuine hang into a visible failure — without failing a long-but-healthy suite. Also fixes a base-red in chat-pipeline:564c204efebumped DEFAULT_CODEX_CLIENT_VERSION to 0.146.0 but the User-Agent assertion still pinned 0.144.1. The line two above already read the constant via getCodexClientVersion(); this one duplicated the literal. Deriving it from the same source stops the next bump from breaking the test again. * fix(ratelimit): re-arm Bottleneck reservoir heartbeat after updateSettings Bottleneck 2.19.5 (frozen upstream dependency, no release since 2019) has a bug in LocalDatastore#_startHeartbeat() (node_modules/bottleneck/lib/ LocalDatastore.js:29,56): the guard `if (this.heartbeat == null && ...)` only (re)creates the periodic reservoir-refresh interval the first time it runs. Every later call -- including the one updateSettings() itself triggers internally -- falls into the else branch and does clearInterval(this.heartbeat) WITHOUT resetting the reference back to null. Because the stale reference sticks around, every future _startHeartbeat() call keeps taking the same dead else branch: the periodic reservoir refresh is gone forever after the first manual updateSettings() call on a limiter. Every limiter created by this file starts with a live heartbeat (buildLimiterDefaults() always sets reservoirRefreshInterval/ reservoirRefreshAmount), so the very first updateFromHeaders() / updateFromResponseBody() / applyRequestQueueSettings() call against a limiter permanently kills its refresh. In production this wedges the request queue once the reservoir hits 0: an auto-enrolled apikey connection accumulates its default 60 requests, the reservoir zeroes, the queue freezes for ~120s, the watchdog fires a synthetic 502 (RATE_LIMIT_QUEUE_WEDGED), the connection cools down and gets excluded from weighted combo pools -- turning a configured 70/30 split into ~50/50. Add applyLimiterSettings(), a module-local wrapper around limiter.updateSettings() that nulls the stale heartbeat reference and re-invokes _startHeartbeat() afterward so it takes the "start a fresh interval" branch again. Route all 5 updateSettings() call sites through it (updateAllLimiterSettings, both updateFromHeaders() branches, loadPersistedLimits(), and updateFromResponseBody()). updateAllLimiterSettings is now async and awaited by its two callers (initializeRateLimits, applyRequestQueueSettings); the sync call sites use the existing trackAsyncOperation() fire-and-forget tracking pattern. tests/integration/combo-matrix/weighted.test.ts is the E2E proof: the "weighted: 70/30" case now passes with zero WEDGED/RATE_LIMIT_QUEUE/502 log lines across 200 sequential requests (previously the wedge/recovery cycle inflated its runtime and skewed the distribution toward ~50/50). Refs #8213 * fix(tests): remove stray TDD probes committed by accident inf4e93f339dThree TDD repro/probe test files landed on the release tip viaf4e93f339d(docs: add management authentication terminology guide, files from a worktree. Each file is a pre-fix TDD probe that belongs to a *different*, still-in-flight fix branch/PR and duplicates a file path that PR already owns and will properly update on merge: - tests/unit/authz/probe-9033-repro.test.ts: probe for #9033 (IP blacklist direct-connection bypass). 3/4 asserts fail against this tree (D1, D2, Bonus — all assert the not-yet-implemented target behavior); D3 passes (pre-existing behavior). Owned by PR #9385 (open, unmerged), which modifies this exact path. - tests/unit/repro-8522.test.ts: probe for #8522 (absolute file-size baseline reds innocent PRs on inherited drift). First test fails against this tree's evaluateFileSizes (still absolute-only); second (sanity: real growth still flags) passes. #8522 is actually CLOSED upstream — PR #9355 merged the real fix into release/v3.8.50 today (2026-08-05T15:53Z) modifying this exact path — but this branch's merge-base with release/v3.8.50 (6b0e11e378) predates that merge, so the fix has not synced into this tree yet. - tests/unit/repro-8956.test.ts: probe for #8956 (resolveProjectRoot stops at synthetic Next.js standalone package.json). First test fails against this tree; second (sanity: named package.json still resolves) passes. Owned by PR #9354 (open, unmerged), which modifies this exact path. Each deleted file's real implementation + passing version already exists in its owning PR and will land normally through that PR's own merge — deleting the premature copy here does not lose any coverage. No config/quality/test-masking-allowlist.json entry was added: the _deletedWithReplacement schema only supports `replacement` (a test file that must already exist in this tree's HEAD — none does, the real versions live in the unmerged sibling PRs above) or `sourceRemoved` (production files that must be absent from HEAD — they are not, none of the three issues are implemented in this tree). Neither shape fits an "owned by an in-flight sibling PR" deletion, so the CI test-masking gate will flag these 3 deletions for mandatory human review on this branch's next PR diff against release/v3.8.50 — flagged for the owner rather than inventing a new allowlist shape. Refs #9033, #8522, #8956, #7786 * fix(tests): align 8189-classifier-compat with #9276 always-mode semantics tests/unit/8189-classifier-compat-auto-narrow.test.ts was a test-sibling forgotten when #9276 (commit6b531fbacd) removed the unconditional `if (mode === "always") return true` branch from shouldDefaultAllowClassifier(). tests/unit/claude-classifier-compat.test.ts was updated in that same commit; this file was not. Old contract: 'always' mode short-circuited every Claude-format request unconditionally (operator opt-in was treated as sufficient on its own). New contract: 'always' now requires the same SECURITY_MONITOR_MARKER system-prompt text as 'auto' — the marker-optional behavior let a normal chat request through /v1/messages be silently swallowed by an operator's 'always' opt-in. The single 'always' test (1 assert, no-marker body expecting true) is replaced by two tests mirroring the depth already used for 'auto' mode in the same file: no-marker/false and marker-present/true. Net effect is +1 assert, not a reduction — the new pair verifies both directions of the narrowed contract instead of only the now-incorrect unconditional case. Before: 3/4 pass (the 'always' test failed: expected true, got false). After: 5/5 pass. Refs #9276 * fix(tests): align deepseek-web-tools-execute with #9343 tool envelope contract tests/unit/deepseek-web-tools-execute-2820.test.ts (executor level) was a test-sibling forgotten when #9343 (commitd969555417) hardened tool-call parsing: bare JSON with no explicit <tool>/<tool_call> envelope is never promoted to tool_calls anymore (previously it was, whenever a tools[] set was requested — a security gap allowing prose/code-fenced JSON echoed back by the model, or a copy-attack, to trigger real tool execution). Three siblings were updated in the same commit: web-tools-translation.test.ts and web-tools-translation-2820.test.ts (parseToolCallsFromText, the shared translator), and deepseek-web-tools-variants.test.ts (parseDeepSeekToolCalls, deepseek-specific parser) — all inverted their bare-JSON assertions to `toolCalls === null` + `content === text` (preserved verbatim, not stripped). This file calls the executor's execute() (full HTTP round trip through buildToolAwareResult), so it was not touched by that diff and kept asserting the old contract (finish_reason: "tool_calls", content: null). Verified against source (open-sse/executors/deepseek-web.ts buildToolAwareResult): when parseDeepSeekToolCalls returns toolCalls=null, hasCalls is false, so finish_reason is "stop", message.tool_calls is never set, and message.content is the parser's returned content — which for text with no <tool>/<tool_call> tag at all is the original string, unchanged (parseToolCallsFromText's early-return branch). The test now asserts exactly that shape, at the same executor level as the rest of the file's tool_calls that make sense at that level as the rest of the file's tool_calls Refs #9343 * fix(tests): align visionBridge tests with #8430 contract (partial — see note) Two test-siblings were forgotten when #8430 (commit7e55abbc41) hardened Vision Bridge's vision-model selection: getBestVisionModel() now validates that a candidate has a usable active connection (hasUsableCredentialsForModel, DB-backed) before returning it, instead of unconditionally returning the fixedModel or a hardcoded "openai/gpt-4o-mini" default. Three siblings were updated in the same commit (visionBridgeRouter.test.ts, the new repro-8430.test.ts, vision-bridge-preserve-on-failure-4012.test.ts); these two were not. tests/unit/guardrails/visionBridgeHelpers.callVisionModel.test.ts (8 failures, all "No vision-capable provider connected"): callVisionModel()'s `routerConfig` param only merges into getBestVisionModel's CONFIG argument, never its `deps` argument, so there is no way to inject a credentials stub through this function's public signature (unlike the guardrail class and getBestVisionModel itself, which do accept an injectable `hasUsableCredentials`). These tests exercise callVisionModel's own request/response handling, not credential routing (already covered elsewhere), so the fix seeds one real usable `provider_connections` row per provider the file exercises (openai, anthropic) via createProviderConnection in a test.before() hook, with resetDbInstance() in test.after() per the DB-handle-cleanup convention. All 8 now pass. tests/unit/guardrails/visionBridge.test.ts (7 failures): 1 of the 7 (VB-S03) is a genuine forgotten-contract case, fixed here — same semantic flip already applied to vision-bridge-preserve-on-failure-4012.test.ts: in the combo describe path, when EVERY describe call fails, the raw image is now replaced with an "(unavailable)" stub instead of preserved, because that path is only reached for confirmed non-vision targets. Assertions inverted to match (imagePart undefined, unavailable-stub present), same assert count, no weakening. *** THE OTHER 6 (VB-S12, VB-S12b, VB-S01, VB-S13, VB-S07, VB-S10) ARE DELIBERATELY LEFT FAILING. *** These are NOT a #8430 contract change — root- caused to what looks like a separate, unintentional regression: the ONE call to getBestVisionModel() in visionBridge.ts's whole-request-reroute path (line 244, `getBestVisionModel({ fixedModel: configuredModel })`) does not pass a `deps` second argument, so it always uses the real DB-backed hasUsableCredentialsForModel instead of this.deps.hasUsableCredentials — even though the two adjacent checks in the very same function (`checkCreds(model)` at line 226, `checkCreds(bestModel)` at line 246) DO honor the injectable override. In this suite's empty-but-readable isolated test DB, that real check deterministically returns `false` (not the indeterminate `null` the file's own createGuardrail() comment says these tests rely on: "Fail-open (null) so classic VB-S01/S07/S10 reroute tests keep working without a live credential DB"), so getBestVisionModel silently returns null, the reroute branch's `if (bestModel && ...)` guard never fires, and every test that expects a reroute observes a silent no-op instead. Evidence this is a source gap, not a test that needs updating: - The file's own pre-existing comment names VB-S01/S07/S10 as tests the `null` fail-open default is SUPPOSED to keep green. - VB-CRED-01/02 (the file's only two tests that actually inject a non-default hasUsableCredentials mock) both pass today, but neither one's assertions distinguish "mock honored" from "mock ignored, real check also says no" — they don't prove the threading works, they just don't happen to notice it's missing. - visionBridgeRouter.test.ts, repro-8430.test.ts, and vision-bridge-preserve-on-failure-4012.test.ts (22 tests, all green) all either call getBestVisionModel directly with explicit deps, or mock callVisionModel wholesale (bypassing getBestVisionModel entirely) — none of them exercises this exact call site through the guardrail's own deps. Per instructions, this was intentionally NOT "fixed" by weakening these 6 tests' assertions (that would mask the gap) or by seeding fake DB credentials to route around it (that would hide a real production DI inconsistency behind a test-only workaround) or by touching src/lib/guardrails/visionBridge.ts (a production behavior change outside a test-alignment task's scope, and Hard Rule #18 requires its own TDD/validation cycle). Flagging for the owner: the likely one-line fix is threading `{ hasUsableCredentials: this.deps.hasUsableCredentials }` as getBestVisionModel's second argument at visionBridge.ts:244, mirroring the two adjacent call sites in the same function. Before: 15 failures (7 + 8). After: 9 pass added (1 + 8), 6 still fail (unchanged, by design). Refs #8430 * feat(quality): add strayFromCommit deletion allowlist form to test-masking gate The deletion allowlist supported two shapes: replacement (test rewritten elsewhere) and sourceRemoved (feature deleted). Neither fits a third legitimate case surfaced today: test files that entered the repo BY ACCIDENT — commitf4e93f339d(#7786 docs) swept another session's worktree artifacts into the release, including TDD probes owned by open fix PRs (probe-9033-repro -> PR #9385, repro-8956 -> PR #9354, repro-8522 -> PR #9355). Those probes fail by design until their owning PR merges, so every unit run on the release tip broke on them. The new strayFromCommit form is verified, not trusted: the gate asks git which commit actually ADDED the file (git log --diff-filter=A) and only exempts the deletion when it matches the declared hash; a non-empty reason naming the owning PR/issue is mandatory. Also allowlists the deepseek-web-tools-execute assert reduction (23->21) fromed661f2126— same #9343 contract-inversion class as the existing web-tools-translation entry. Gate unit tests: 55/55 pass. Full gate vs main: OK. * fix(guardrails): pass credential deps to getBestVisionModel at reroute call site The individual-model reroute path in VisionBridgeGuardrail.preCall() calls getBestVisionModel({ fixedModel: configuredModel }) without its second `deps` argument, so the router always falls back to the real DB-backed hasUsableCredentialsForModel instead of an injected `deps.hasUsableCredentials` override. The two adjacent credential checks in the same function (the original-model check and the best-model check, both via the local `checkCreds` binding) already thread deps correctly — only this middle call, added in #8430, was left out. Pass the same resolved `checkCreds` used by those two adjacent checks as `getBestVisionModel`'s deps argument so all three credential checks in this reroute path stay consistent. Fixes 6 tests in tests/unit/guardrails/visionBridge.test.ts that depended on the injected hasUsableCredentials mock being honored on this path: VB-S12, VB-S12b, VB-S01, VB-S13, VB-S07, VB-S10. Refs #8430 * fix(quality): raise unit ceiling to 100min and align 2 more forgotten sibling tests Unit ceiling 45->100min: a hermetic-env measurement on the loaded devbox (load 7-26) was still inside invocation 1 of 3 at 76min when killed; contention factor 2-3x measured, no idle measurement exists. The pre-flight's real condition is exactly that contended one (unit runs in Promise.all with integration+vitest), and there 45min provably killed a healthy suite and fabricated a false base-red. The 45min value came from v3.8.43 as an estimate never validated by measurement. TODO in-code: re-tighten after an idle run on the .113 box. Also aligns the 6th and 7th occurrences of the same systemic pattern (behavior change merged updating only part of the sibling tests): - issue-7859-gemini-web-redirect-valid: #9407 refined ServiceLogin redirects to mean expired session; the #7859 regression coverage is preserved via a non-ServiceLogin public redirect variant. - provider-validation-specialty claude-web 429: #9406 inverted the contract (rate-limited session is unhealthy); the dedicated repro file owns the full contract, this sibling now matches it. Also carries the file-size rebaseline for #9323's base.ts growth (1578->1623, WAF retry + burst guard) and the eslintWarnings baseline tightened 5000->0 (real measured value with the TS7 suppressions in place — 5000 left the ratchet inert). Refs #9407, #9406, #9323 * fix(tests): restore the 3 TDD probes now owned by merged fixes and drop their stray allowlist entries The base advanced while this PR was open: the real fixes for the three issues behind the stray probes all merged into release/v3.8.50 — #9385 (issue 9033), #9355 (issue 8522) and #9354 (issue 8956). - probe-9033-repro / repro-8522: the base rewrote both probes into the regression tests of their merged fixes, so the delete side of the rebase conflict was dropped and the base versions kept. - repro-8956: #9354 only realigned one fixture line in auto-update.test.ts (package.json marker now needs a name field) and added no test for the new skip-synthetic behavior — the probe is the ONLY regression coverage of that merged fix (2/2 green on the base), so deleting it would remove real coverage. Restored. With no test-file deletions left in the PR diff, the three strayFromCommit allowlist entries are stale and removed. The strayFromCommit form support in check-test-masking.mjs stays (covered by its own fixtures). * fix(quality): rebaseline file-size for PR #9529 own growth The base sits exactly at the old frozen values, so the base-relative mode (#8522) does not cover this growth — it is this PR's own: - open-sse/services/rateLimitManager.ts 1060->1105: the applyLimiterSettings() helper that re-arms the reservoir heartbeat after updateSettings (Bottleneck 2.19.5 fix, TDD in ratelimit-reservoir-refresh.test.ts). - tests/integration/chat-pipeline.test.ts 1592->1598: codex User-Agent derived from getCodexClientVersion() instead of a pinned literal. - tests/unit/provider-validation-specialty.test.ts 2980->2985: new claude-web 429 -> valid:false coverage (#9406). * fix(docs): sync provider count to 291 in README and CLAUDE The live catalog counts 291 providers but README.md/CLAUDE.md still said 290, so the STRICT 'Docs Gates (fast-path)' check reds EVERY open PR against release/v3.8.50 (verified on #9537/#9539 as well — inherited base-red, not introduced by this PR). Updated all provider-count mentions including the section anchor. * fix(tests): align launch-codex 6312 guard with the async #9454 spawn contract #9454 made resolveCodexSpawn async (PATH-probes a native codex.exe before the .cmd shim) and updated its own tests, but left this older sibling calling the function synchronously — destructuring the Promise yields undefined and reds Unit fast-path (1/4) for EVERY open PR against the release (verified on #9537/#9539; inherited base-red). Realigned to the async contract with an injected probe; keeps the original #6312 fallback guard plus the only non-Windows codex coverage (now also asserting the probe never runs off Windows). * fix(translator): move state-mutating reasoning summary helper out of the pure leaf #9500 added buildResponsesReasoningSummaryDelta(state, ...) to pureHelpers.ts, but the function reads AND mutates stream state (reasoningSummaryIndex map) — violating the leaf contract declared in the file header ('no host imports, no stream state') and guarded by response-openai-responses-purehelpers-split.test.ts, which reds Unit fast-path (4/4) for every open PR (inherited base-red, verified on #9537/#9539). Moved verbatim to the host next to the other stream-state helpers (markResponsesReasoningDeltaEmitted); the host was its only consumer. Behavior unchanged: repro-9500-reasoning-separator 3/3 green, leaf/host architecture tests green. * fix(quality): rebaseline openai-responses.ts for the leaf-state relocation The #9500 helper moved from pureHelpers.ts into the host (previous commit) grows the host file 1174->1204 while the leaf shrinks by the same amount — net-zero LOC across the pair, but the per-file frozen ratchet only sees the growing side. * fix(tests): let the 9442 cert-mode test see past the harness trust-store guard tests/_setup/isolateDataDir.ts sets OMNIROUTE_SKIP_SYSTEM_TRUST=1 globally, which makes installCert() return before issuing any command — so the #9442 install-gap test captured nothing and could NEVER pass under npm run test:unit (it only passed invoked directly, harness-less; inherited base-red on Unit fast-path 3/4, verified on #9537/#9539). Clear the flag for this file only (restored in test.after): safe because every spawned command is a logging stub on PATH and OMNIROUTE_NO_SUDO=1 strips sudo, so nothing touches the real trust store. 6/6 under the CI harness including system-trust-test-guard. --------- Co-authored-by: diegosouzapw <diegosouzapw@users.noreply.github.com>
This commit is contained in:
committed by
GitHub
parent
88a2fd26c7
commit
8180b49ce1
@@ -46,7 +46,7 @@ Repository map and Reference Documentation sections below.
|
||||
|
||||
## Project at a Glance
|
||||
|
||||
**OmniRoute** — unified AI proxy/router. One endpoint, 290 LLM providers, auto-fallback.
|
||||
**OmniRoute** — unified AI proxy/router. One endpoint, 291 LLM providers, auto-fallback.
|
||||
|
||||
| Layer | Location | Purpose |
|
||||
| ------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
|
||||
14
README.md
14
README.md
@@ -7,7 +7,7 @@
|
||||
|
||||
# 🚀 OmniRoute — The Free AI Gateway
|
||||
|
||||
<img src="./docs/diagrams/readme-hero.svg" width="100%" alt="OmniRoute — Never stop coding. Every AI tool → 290 providers — 90+ free — through one endpoint. Claude Code, Codex, Cursor, Cline, Copilot & Antigravity into FREE Claude / GPT / Gemini with auto-fallback. RTK + Caveman stacked compression saves 15–95% tokens (~89% avg) — never hit limits. 290 AI providers · 90+ free tiers · ~1.53B free tokens/mo · 19 routing strategies · $0 to start."/>
|
||||
<img src="./docs/diagrams/readme-hero.svg" width="100%" alt="OmniRoute — Never stop coding. Every AI tool → 291 providers — 90+ free — through one endpoint. Claude Code, Codex, Cursor, Cline, Copilot & Antigravity into FREE Claude / GPT / Gemini with auto-fallback. RTK + Caveman stacked compression saves 15–95% tokens (~89% avg) — never hit limits. 291 AI providers · 90+ free tiers · ~1.53B free tokens/mo · 19 routing strategies · $0 to start."/>
|
||||
|
||||
</div>
|
||||
|
||||
@@ -81,7 +81,7 @@
|
||||
<tr>
|
||||
<td align="right"><b>⚙️ Features</b></td>
|
||||
<td align="center"><a href="#-combos--the-flagship">🎯 Combos</a></td>
|
||||
<td align="center"><a href="#-290-ai-providers--90-free">🌐 Providers</a></td>
|
||||
<td align="center"><a href="#-291-ai-providers--90-free">🌐 Providers</a></td>
|
||||
<td align="center"><a href="#-full-cli--a2a--mcp">🔌 CLI & MCP</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
@@ -188,7 +188,7 @@ curl http://localhost:20128/v1/chat/completions \
|
||||
|
||||
</div>
|
||||
|
||||
<img src="./docs/diagrams/promise-pillars.svg" width="100%" alt="The Promise — One endpoint. 290 providers. Never stop building — OmniRoute picks the cheapest one that works. Six pillars: Never hit limits (auto-fallback across 290 providers in milliseconds, zero downtime) · Save up to 95% tokens (RTK + Caveman stacked compression cuts 15–95%, ~89% avg on tool-heavy sessions) · $0 to start (90+ free tiers, 40+ free forever — no card needed) · Every tool works (33 coding agents through one config) · One endpoint (OpenAI ↔ Claude ↔ Gemini ↔ Responses API at /v1) · Production-grade (circuit breakers, TLS stealth, MCP 104 tools, A2A, memory, guardrails, evals — 25,000+ tests)."/>
|
||||
<img src="./docs/diagrams/promise-pillars.svg" width="100%" alt="The Promise — One endpoint. 291 providers. Never stop building — OmniRoute picks the cheapest one that works. Six pillars: Never hit limits (auto-fallback across 291 providers in milliseconds, zero downtime) · Save up to 95% tokens (RTK + Caveman stacked compression cuts 15–95%, ~89% avg on tool-heavy sessions) · $0 to start (90+ free tiers, 40+ free forever — no card needed) · Every tool works (33 coding agents through one config) · One endpoint (OpenAI ↔ Claude ↔ Gemini ↔ Responses API at /v1) · Production-grade (circuit breakers, TLS stealth, MCP 104 tools, A2A, memory, guardrails, evals — 25,000+ tests)."/>
|
||||
|
||||
<br/>
|
||||
<br/>
|
||||
@@ -439,7 +439,7 @@ All **19** strategies — mix & match per combo step:
|
||||
|
||||
</div>
|
||||
|
||||
<img src="./docs/diagrams/comparison-table.svg" width="100%" alt="What sets OmniRoute apart — comparison table vs 9router, OpenRouter, CLIProxyAPI and LiteLLM across 13 capabilities. OmniRoute: 290 providers, 90+ free providers built-in, 19 routing strategies, 12-engine token compression, built-in MCP server with 104 tools, A2A agent protocol, persistent memory, guardrails, cloud agents, TLS fingerprint stealth, Desktop/Termux/PWA, 43 i18n UI locales, 100% MIT self-hosted. OmniRoute is the only one with the full set; competitors show a mix of checks, partials and crosses. Verified from each project's docs."/>
|
||||
<img src="./docs/diagrams/comparison-table.svg" width="100%" alt="What sets OmniRoute apart — comparison table vs 9router, OpenRouter, CLIProxyAPI and LiteLLM across 13 capabilities. OmniRoute: 291 providers, 90+ free providers built-in, 19 routing strategies, 12-engine token compression, built-in MCP server with 104 tools, A2A agent protocol, persistent memory, guardrails, cloud agents, TLS fingerprint stealth, Desktop/Termux/PWA, 43 i18n UI locales, 100% MIT self-hosted. OmniRoute is the only one with the full set; competitors show a mix of checks, partials and crosses. Verified from each project's docs."/>
|
||||
|
||||
<sub>📊 Full methodology & per-feature detail vs 9router, OpenRouter, CLIProxyAPI & LiteLLM → [`docs/comparison/OMNIROUTE_VS_ALTERNATIVES.md`](docs/comparison/OMNIROUTE_VS_ALTERNATIVES.md)</sub>
|
||||
|
||||
@@ -513,7 +513,7 @@ Pix copia-e-cola:
|
||||
- **🖼️ New endpoints** — `/v1/ocr` (Mistral OCR) and `/v1/audio/translations` (Whisper-style) round out the media surface. → [API Reference](docs/reference/API_REFERENCE.md)
|
||||
- **🎨 Image / video / audio generation** — one API for media: xAI Grok Imagine & Novita AI video, ComfyUI, Freepik, Adobe Firefly, Microsoft Designer, Google Imagen, Segmind, EdgeTTS. → [API Reference](docs/reference/API_REFERENCE.md)
|
||||
- **🌍 Deployment & ops** — reverse-proxy `basePath`, browser-language auto-detect, per-key device tracking, root-less MITM trust, zh-TW localization. → [Environment](docs/reference/ENVIRONMENT.md)
|
||||
- **🤝 More providers & agents** — Cursor Cloud Agent, Grok Build (xAI) with browser + OAuth login, Ollama first-class card, Claude Opus 5 & Sonnet 5, Kimi official partnership (Code/Web/Moonshot), Zed, Requesty, SenseNova, Yuanbao, Agnes AI… and a refreshed **290-provider catalog**. → [Providers](docs/reference/PROVIDER_REFERENCE.md)
|
||||
- **🤝 More providers & agents** — Cursor Cloud Agent, Grok Build (xAI) with browser + OAuth login, Ollama first-class card, Claude Opus 5 & Sonnet 5, Kimi official partnership (Code/Web/Moonshot), Zed, Requesty, SenseNova, Yuanbao, Agnes AI… and a refreshed **291-provider catalog**. → [Providers](docs/reference/PROVIDER_REFERENCE.md)
|
||||
- **📡 Routing transparency** — every response carries an `X-OmniRoute-Decision` header naming the strategy/provider/latency that served it, a new `cache-optimized` combo strategy + Auto-Combo `cacheAffinity` factor route repeat requests back to the connection holding the cached prefix, and a read-only `/v1/auto-combo/{channel}/candidates` endpoint exposes an `auto/*` channel's live candidate pool. → [Auto-Combo](docs/routing/AUTO-COMBO.md)
|
||||
- **⚡ Local performance & infra** — one-click local Redis, Cloudflare Workers / Deno Deploy relay deployers, Bifrost & Mux as supervised embedded services. → [Embedded Services](docs/frameworks/EMBEDDED-SERVICES.md)
|
||||
|
||||
@@ -574,11 +574,11 @@ Pix copia-e-cola:
|
||||
|
||||
<div align="center">
|
||||
|
||||
## 🌐 290 AI Providers — 90+ Free
|
||||
## 🌐 291 AI Providers — 90+ Free
|
||||
|
||||
</div>
|
||||
|
||||
> The most complete catalog of any open-source router: **290 providers**, **90+ with a free tier**, **40+ free forever**.
|
||||
> The most complete catalog of any open-source router: **291 providers**, **90+ with a free tier**, **40+ free forever**.
|
||||
|
||||
<div align="center">
|
||||
|
||||
|
||||
@@ -172,7 +172,7 @@
|
||||
"_rebaseline_2026_07_25_8510_adobe_firefly_reference_images_tests": "#8510 (artickc, feat/adobe-firefly-reference-images) own test growth: tests/unit/adobe-firefly.test.ts 711->871 (+159, entirely this PR's diff — new referenceBlobs upload/dispatch coverage for handleAdobeFireflyImageGeneration, resolveAdobeSourceImageIds, and the storage-upload wire contract). Route-level /v1/images/edits coverage (credentials/rate-limit/4-ref-cap branches added to route.ts) lives in the new tests/unit/8510-adobe-firefly-edits-route.test.ts instead of growing this file further.",
|
||||
"_rebaseline_basered_codebuddy_cn": "Base-red fix (#4664 CodeBuddy CN): oauth-providers-config.test.ts 867->870 (+3) to align the EXPECTED provider list/config with the codebuddy-cn provider that #4664 added to the registry without updating this test (it asserts 'exactly once').",
|
||||
"_rebaseline_pr4613_compatible_provider_groups": "Reconcile #4613 already-merged growth: providers-page-utils.test.ts 1004->1052 (+48, buildCompatibleProviderGroups partition unit test). Fast-gate PR->release does not run check:file-size, so this surfaced post-merge.",
|
||||
"tests/integration/chat-pipeline.test.ts": 1592,
|
||||
"tests/integration/chat-pipeline.test.ts": 1598,
|
||||
"tests/integration/chatcore-compression-integration.test.ts": 1114,
|
||||
"tests/unit/account-fallback-service.test.ts": 1563,
|
||||
"tests/unit/batch_api.test.ts": 1324,
|
||||
@@ -190,7 +190,7 @@
|
||||
"tests/unit/models-catalog-route.test.ts": 1636,
|
||||
"tests/unit/perplexity-web.test.ts": 1355,
|
||||
"tests/unit/provider-models-route.test.ts": 1784,
|
||||
"tests/unit/provider-validation-specialty.test.ts": 2980,
|
||||
"tests/unit/provider-validation-specialty.test.ts": 2985,
|
||||
"tests/unit/providers-page-utils.test.ts": 1106,
|
||||
"tests/unit/response-sanitizer.test.ts": 1063,
|
||||
"tests/unit/route-edge-coverage.test.ts": 1241,
|
||||
@@ -343,7 +343,7 @@
|
||||
"_rebaseline_pr1043_minimax_tts": "Upstream port decolua/9router#1043 (toanalien) own growth: audioSpeech.ts 965->1061 (+96). Adds MiniMax T2A v2 TTS dispatch (handleMinimaxSpeech + hexToBytes helper) — provider entry was already in audioRegistry (format: minimax-tts) but no handler existed, falling through to the OpenAI-compatible default that fails (T2A has custom shape + hex-encoded audio + base_resp envelope). New branch sits next to the other inline provider branches (xiaomi-mimo, coqui, tortoise, aws-polly) — extracting would just create indirection. Covered by tests/unit/minimax-tts-1043.test.ts (3 tests, GREEN: success, base_resp error, invalid-hex).",
|
||||
"_rebaseline_pr4592_exclude_exhausted_auto": "Reconcile #4592 already-merged growth: combo.ts 2991->3036 (+45, terminal-status quota-cutoff exclusion in buildAutoCandidates + opt-in gate). Fast-gate PR->release does not run check:file-size.",
|
||||
"open-sse/executors/antigravity.ts": 1528,
|
||||
"open-sse/executors/base.ts": 1578,
|
||||
"open-sse/executors/base.ts": 1623,
|
||||
"open-sse/executors/chatgpt-web.ts": 3241,
|
||||
"open-sse/executors/codex.ts": 1534,
|
||||
"open-sse/executors/cursor.ts": 1560,
|
||||
@@ -363,8 +363,8 @@
|
||||
"open-sse/services/claudeCodeCompatible.ts": 1202,
|
||||
"open-sse/services/combo.ts": 3648,
|
||||
"open-sse/services/compression/strategySelector.ts": 1060,
|
||||
"open-sse/services/rateLimitManager.ts": 1060,
|
||||
"open-sse/translator/response/openai-responses.ts": 1174,
|
||||
"open-sse/services/rateLimitManager.ts": 1105,
|
||||
"open-sse/translator/response/openai-responses.ts": 1204,
|
||||
"open-sse/utils/cursorAgentProtobuf.ts": 1505,
|
||||
"open-sse/utils/stream.ts": 2889,
|
||||
"src/app/(dashboard)/dashboard/HomePageClient.tsx": 1381,
|
||||
@@ -405,7 +405,7 @@
|
||||
"src/sse/handlers/chat.ts": 1845,
|
||||
"src/sse/services/auth.ts": 2508,
|
||||
"tests/unit/account-fallback-service.test.ts": 1572,
|
||||
"tests/unit/provider-validation-specialty.test.ts": 2980,
|
||||
"tests/unit/provider-validation-specialty.test.ts": 2985,
|
||||
"open-sse/executors/hyperagent.ts": 1026
|
||||
},
|
||||
"_rebaseline_2026_07_27_v3849_train2": "Merge-train 2 (7 PRs) — owner-approved 2026-07-27. Single entry: chatCore.ts 4955->5006 (#8595, Responses multi-turn image compaction before the context hard-reject). Genuine irreducible growth at the existing compaction chokepoint in handleChatCore — the PR adds a last-resort retry against the concrete budget plus the estimateFinalInputTokens helper, both wired at the pre-existing call site rather than a new branch. Covered by tests/unit/8560-responses-image-compaction.test.ts (4 tests).",
|
||||
@@ -418,5 +418,7 @@
|
||||
"_rebaseline_2026_07_29_8281_home_quickstart_prefetch": "Release v3.8.49 base-red fix (no PR — captain sweep): src/app/(dashboard)/dashboard/HomePageClient.tsx 1377->1381 (+4). #8292 added prefetch={false} to the sidebar but left /home's five quick-start Links prefetching, so first paint still fired 12 speculative RSC requests — caught by navigation.spec.ts only after the e2e helper bug (APP_ROUTE_PATTERN missing /home) was repaired in the same cycle. Growth is the five prefetch attributes; it was offset first by extracting the repeated className literals (INLINE_LINK x4, DOCS_LINK x1), which collapsed five wrapped <Link> blocks back to one line each — a naive fix measured 1391. Guard: tests/unit/sidebar-prefetch-policy-8281.test.ts.",
|
||||
"_rebaseline_2026_08_01_8964_xai_agent_tools": "PR #8964 own growth: chatCore.ts 5020->5034 at the existing native-passthrough chokepoint. Adds xAI Agent Tools passthrough for /v1/responses (xai/xai-oauth/xao): resolve nativeXaiResponsesPassthrough, force openai-responses targetFormat, stamp body marker, and OR into the existing nativeCodexPassthrough sites (web-search bypass + requestEndpointPath). Leaf logic in passthroughHelpers, responsesEndpoint, targetFormat, xai executor, responseSanitizer, usageTracking. Cohesive wiring at the Codex passthrough boundary.",
|
||||
"_rebaseline_2026_08_01_8964_response_sanitizer": "PR #8964 own growth: responseSanitizer.ts 1115->1128. Keep cost_in_usd_ticks / server_side_tool_usage(_details) through sanitizeResponsesApiResponse allowlists so native xAI tool responses retain usage.",
|
||||
"_rebaseline_2026_08_02_v3850_agentrouter_responses": "Release v3.8.50 AgentRouter/Codex compatibility reconciliation. open-sse/executors/base.ts 1562->1578: #9190 wires AgentRouter's selected Claude/OpenAI/Responses protocol through the existing executor URL, auth, identity-header and fingerprint chokepoints; the reusable alternate resolver remains outside base.ts. open-sse/utils/stream.ts 2887->2889: #9213 evaluates Responses ID and usage normalization independently so response.completed always receives finite usage.total_tokens instead of short-circuiting after an ID rewrite. tests/unit/chatcore-translation-paths.test.ts 2769->2776: #9191 updates the existing Claude-Code bridge assertions for the dynamic AgentRouter wire image. PR #9224 offsets its own chatCore growth by extracting the AgentRouter protocol decisions into chatCore/agentRouterProtocol.ts, leaving chatCore below its frozen ceiling. Covered by agentrouter executor/chatCore protocol tests, chatcore translation-path tests, and responses-commentary-passthrough tests."
|
||||
"_rebaseline_2026_08_02_v3850_agentrouter_responses": "Release v3.8.50 AgentRouter/Codex compatibility reconciliation. open-sse/executors/base.ts 1562->1578: #9190 wires AgentRouter's selected Claude/OpenAI/Responses protocol through the existing executor URL, auth, identity-header and fingerprint chokepoints; the reusable alternate resolver remains outside base.ts. open-sse/utils/stream.ts 2887->2889: #9213 evaluates Responses ID and usage normalization independently so response.completed always receives finite usage.total_tokens instead of short-circuiting after an ID rewrite. tests/unit/chatcore-translation-paths.test.ts 2769->2776: #9191 updates the existing Claude-Code bridge assertions for the dynamic AgentRouter wire image. PR #9224 offsets its own chatCore growth by extracting the AgentRouter protocol decisions into chatCore/agentRouterProtocol.ts, leaving chatCore below its frozen ceiling. Covered by agentrouter executor/chatCore protocol tests, chatcore translation-path tests, and responses-commentary-passthrough tests.",
|
||||
"_rebaseline_2026_08_05_9323_agentrouter_waf_retry": "PR #9323 (fix(agentrouter): retry on 400 content-blocked + burst guard) own growth: open-sse/executors/base.ts 1578->1623 (check-file-size.mjs conta via split(\"\\n\").length; wc -l ve 1622). As +45 linhas sao o WAF_RETRY_CONFIG + o burst guard via gateOutboundRequest() para o WAF do agentrouter.org, com comentarios explicando o porque de cada mitigacao e cobertos por tests/unit/base-executor-waf-retry.test.ts e tests/unit/wafRateLimit.test.ts. Crescimento funcional legitimo, nao inchaco.",
|
||||
"_rebaseline_2026_08_05_9529_own_growth": "PR #9529 own growth (base release/v3.8.50 medida EXATAMENTE nos frozen antigos, entao o modo base-relative #8522 nao cobre): open-sse/services/rateLimitManager.ts 1060->1105 (+45: helper applyLimiterSettings() que re-arma o heartbeat do reservoir apos updateSettings — fix do bug Bottleneck 2.19.5 que congelava a fila weighted; TDD em tests/unit/ratelimit-reservoir-refresh.test.ts); tests/integration/chat-pipeline.test.ts 1592->1598 (+6: User-Agent do codex derivado de getCodexClientVersion() em vez de literal pinado — teste-irmao alinhado ao contrato); tests/unit/provider-validation-specialty.test.ts 2980->2985 (+5: cobertura NOVA claude-web 429 -> valid:false, alinhamento #9406); open-sse/translator/response/openai-responses.ts 1174->1204 (+30: buildResponsesReasoningSummaryDelta MOVIDA do leaf pureHelpers.ts para o host — a funcao do #9500 muta stream state e violava o contrato do leaf puro; o LOC total do par host+leaf nao cresceu, o pureHelpers encolheu o mesmo tanto). Crescimento por fix de producao + cobertura adicional + realocacao arquitetural, nao inchaco."
|
||||
}
|
||||
|
||||
@@ -109,5 +109,7 @@
|
||||
"tests/unit/usage-service-hardening.test.ts": "v3.8.49 #7866/#8565/#8013: qwen removido (−3 asserts); o Kimi/Kiro builder-id (uso profileless) passou a ter SUCESSO real em vez de erro de ARN — supportsProfilelessKiroUsage(\"builder-id\") retorna true —, trocando 1 assert de regex de erro por 3 asserts de valor; e os ids de bucket de quota do Antigravity foram atualizados para o catálogo atual. Rodado no HEAD: 23/23 passam. Net 210→209. Verificado legítimo. Prune após v3.8.49 mergear para main.",
|
||||
"tests/unit/virtual-auto-combo.test.ts": "v3.8.49 #7928/#8183: o pooling de contas passou a agrupar conexões web-session do mesmo provider numa entrada lógica com allowedConnectionIds (campo confirmado em open-sse/services/autoCombo/virtualFactory.ts), e o pool no-auth virou uma allowlist fixa (AUTO_COMBO_NOAUTH_ALLOWLIST = opencode, felo-web) — os testes antigos esperavam duplicatas e a inclusão de duckduckgo-web/theoldllm/chipotle, que hoje são corretamente excluídos. Guard dedicado em noauth-autocombo-allowlist.test.ts. Rodado no HEAD: 10/10 passam. Net 39→31. Verificado legítimo. Prune após v3.8.49 mergear para main.",
|
||||
"open-sse/services/__tests__/tierResolver.test.ts": "v3.8.49 #7866: refactor(qwen) remove o provider OAuth legado — o teste \"classifies Qwen as free\" e a entrada de qwen na lista do batch saíram junto com o provider, e os índices do batch desceram de 10 para 9 elementos (net 61→59). Superfície extinta, não enfraquecimento. Verificado legítimo. Prune após v3.8.49 mergear para main.",
|
||||
"tests/unit/plugins-welcome-banner-e2e.test.ts": "v3.8.50 #9126 (commit 8fac6bcd48): o teste único 'BUILTIN_EVENTS has all 14 events' (13 asserts .ok/.equal) foi reestruturado em 3 testes mais específicos — 'contains only emitted/public events' (assert.deepEqual da lista completa), 'does not advertise dead events' (7 asserts .equal(false) para eventos sem emissor real: onModelSelect/onComboResolve/onRateLimit/onQuotaExhaust/onProviderError/onStreamStart/onStreamEnd) e 'lifecycle events remain represented' (4 asserts .ok). Contrato mais forte (agora também nega presença dos eventos mortos), não mais fraco — a contagem líquida cai (73→61) porque o assert.deepEqual único substitui múltiplos assert.ok redundantes com a mesma cobertura. Asserts restruturados, não removidos sem substituição. Verificado legítimo."
|
||||
"tests/unit/plugins-welcome-banner-e2e.test.ts": "v3.8.50 #9126 (commit 8fac6bcd48): o teste único 'BUILTIN_EVENTS has all 14 events' (13 asserts .ok/.equal) foi reestruturado em 3 testes mais específicos — 'contains only emitted/public events' (assert.deepEqual da lista completa), 'does not advertise dead events' (7 asserts .equal(false) para eventos sem emissor real: onModelSelect/onComboResolve/onRateLimit/onQuotaExhaust/onProviderError/onStreamStart/onStreamEnd) e 'lifecycle events remain represented' (4 asserts .ok). Contrato mais forte (agora também nega presença dos eventos mortos), não mais fraco — a contagem líquida cai (73→61) porque o assert.deepEqual único substitui múltiplos assert.ok redundantes com a mesma cobertura. Asserts restruturados, não removidos sem substituição. Verificado legítimo.",
|
||||
"tests/unit/web-tools-translation-2820.test.ts": "v3.8.50 #9343 (commit d969555417): fix(security) exige envelope <tool> explicito — JSON puro NAO deve mais ser promovido a tool_calls. Os 5 testes foram REESCRITOS para o contrato oposto (antes: 'promove e valida name/arguments'; agora: 'toolCalls === null e content preservado'), o que naturalmente usa menos asserts: verificar a NAO-promocao custa 2 asserts, verificar o objeto promovido custava 4. Contrato mais restritivo, nao mais fraco (39->35). Verificado legitimo — a inversao esta explicita nos proprios nomes dos testes ('does NOT promote ... (#9343)').",
|
||||
"tests/unit/deepseek-web-tools-execute-2820.test.ts": "v3.8.50 ed661f2126 (alinhamento ao #9343): o teste 'parses bare JSON reply into OpenAI tool_calls' foi reescrito para o contrato INVERTIDO do fix de seguranca #9343 — JSON puro sem envelope <tool> NAO deve mais ser promovido. Verificar a nao-promocao custa 3 asserts (finish_reason stop, sem tool_calls, content preservado verbatim) onde validar o objeto promovido custava 5 (23->21). Mesma classe da entrada web-tools-translation-2820 acima. Contrato mais restritivo, nao mais fraco."
|
||||
}
|
||||
|
||||
@@ -164,13 +164,58 @@ function buildLimiterDefaults() {
|
||||
};
|
||||
}
|
||||
|
||||
function updateAllLimiterSettings() {
|
||||
const defaults = buildLimiterDefaults();
|
||||
for (const limiter of limiters.values()) {
|
||||
limiter.updateSettings(defaults);
|
||||
/**
|
||||
* Apply new settings to a Bottleneck limiter and re-arm its reservoir-refresh
|
||||
* heartbeat.
|
||||
*
|
||||
* Bottleneck 2.19.5 (frozen upstream dependency, no release since 2019) has a
|
||||
* bug in `LocalDatastore#_startHeartbeat()`
|
||||
* (node_modules/bottleneck/lib/LocalDatastore.js:29,56): the guard
|
||||
* `if (this.heartbeat == null && ...)` only (re)creates the periodic
|
||||
* reservoir-refresh interval the FIRST time it runs. Every later call —
|
||||
* including the one `updateSettings()` itself triggers internally — falls
|
||||
* into the `else` branch and does `clearInterval(this.heartbeat)` WITHOUT
|
||||
* resetting `this.heartbeat` back to `null`. Because the stale reference is
|
||||
* left in place, every future `_startHeartbeat()` call keeps taking the same
|
||||
* dead `else` branch: the periodic reservoir refresh is gone forever after
|
||||
* the FIRST manual `updateSettings()` call on a limiter — every limiter here
|
||||
* starts with a live heartbeat (buildLimiterDefaults() always sets
|
||||
* reservoirRefreshInterval/reservoirRefreshAmount), so that "first call" is
|
||||
* whichever of the 5 updateSettings() call sites in this file runs first.
|
||||
*
|
||||
* Work around it here instead of patching node_modules: null out the stale
|
||||
* reference ourselves and re-invoke `_startHeartbeat()` so it takes the
|
||||
* "start a fresh interval" branch again. Every `limiter.updateSettings(...)`
|
||||
* call in this file MUST go through this helper, never Bottleneck's method
|
||||
* directly.
|
||||
*/
|
||||
async function applyLimiterSettings(
|
||||
limiter: Bottleneck,
|
||||
updates: Bottleneck.ConstructorOptions
|
||||
): Promise<void> {
|
||||
await limiter.updateSettings(updates);
|
||||
const store = (
|
||||
limiter as unknown as {
|
||||
_store?: {
|
||||
heartbeat?: ReturnType<typeof setInterval> | null;
|
||||
_startHeartbeat?: () => void;
|
||||
};
|
||||
}
|
||||
)._store;
|
||||
if (store && typeof store._startHeartbeat === "function") {
|
||||
if (store.heartbeat != null) clearInterval(store.heartbeat);
|
||||
store.heartbeat = null;
|
||||
store._startHeartbeat();
|
||||
}
|
||||
}
|
||||
|
||||
async function updateAllLimiterSettings() {
|
||||
const defaults = buildLimiterDefaults();
|
||||
await Promise.all(
|
||||
Array.from(limiters.values(), (limiter) => applyLimiterSettings(limiter, defaults))
|
||||
);
|
||||
}
|
||||
|
||||
function reconcileEnabledConnections(
|
||||
connectionsRaw: unknown[],
|
||||
requestQueueSettings: RequestQueueSettings
|
||||
@@ -381,7 +426,7 @@ export async function initializeRateLimits() {
|
||||
connections as unknown[],
|
||||
currentRequestQueueSettings
|
||||
);
|
||||
updateAllLimiterSettings();
|
||||
await updateAllLimiterSettings();
|
||||
|
||||
// Load per-connection rate limit overrides
|
||||
connectionRateLimitOverrides.clear();
|
||||
@@ -414,7 +459,7 @@ export async function applyRequestQueueSettings(nextSettings: RequestQueueSettin
|
||||
const { getCachedProviderConnections } = await import("@/lib/localDb");
|
||||
const connections = await getCachedProviderConnections();
|
||||
reconcileEnabledConnections(connections as unknown[], currentRequestQueueSettings);
|
||||
updateAllLimiterSettings();
|
||||
await updateAllLimiterSettings();
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -779,9 +824,7 @@ export function updateFromHeaders(provider, connectionId, headers, status, model
|
||||
logRateLimit(
|
||||
`⚠️ [RATE-LIMIT] ${provider}:${connectionId.slice(0, 8)} — near capacity, slowing down`
|
||||
);
|
||||
limiter.updateSettings({
|
||||
minTime: 200, // Add 200ms between requests
|
||||
});
|
||||
trackAsyncOperation(applyLimiterSettings(limiter, { minTime: 200 }));
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -812,7 +855,7 @@ export function updateFromHeaders(provider, connectionId, headers, status, model
|
||||
}
|
||||
}
|
||||
|
||||
limiter.updateSettings(updates);
|
||||
trackAsyncOperation(applyLimiterSettings(limiter, updates));
|
||||
|
||||
// Persist learned limits (debounced)
|
||||
recordLearnedLimit(
|
||||
@@ -1014,7 +1057,7 @@ async function loadPersistedLimits() {
|
||||
const limiter = limiters.get(key);
|
||||
if (limiter && limit > 0) {
|
||||
const inferredMinTime = minTime || Math.max(0, Math.floor(60000 / limit) - 10);
|
||||
limiter.updateSettings({ minTime: inferredMinTime });
|
||||
await applyLimiterSettings(limiter, { minTime: inferredMinTime });
|
||||
count++;
|
||||
}
|
||||
}
|
||||
@@ -1050,10 +1093,12 @@ export function updateFromResponseBody(provider, connectionId, responseBody, sta
|
||||
`🚫 [RATE-LIMIT] ${provider}:${connectionId.slice(0, 8)} — body-parsed retry: ${Math.ceil(retryAfterMs / 1000)}s (${reason})`
|
||||
);
|
||||
|
||||
limiter.updateSettings({
|
||||
reservoir: 0,
|
||||
reservoirRefreshAmount: 60,
|
||||
reservoirRefreshInterval: retryAfterMs,
|
||||
});
|
||||
trackAsyncOperation(
|
||||
applyLimiterSettings(limiter, {
|
||||
reservoir: 0,
|
||||
reservoirRefreshAmount: 60,
|
||||
reservoirRefreshInterval: retryAfterMs,
|
||||
})
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -18,7 +18,6 @@ import {
|
||||
normalizeOutputIndex,
|
||||
normalizeUpstreamFailure,
|
||||
getVisibleResponsesReasoningSummaryText,
|
||||
buildResponsesReasoningSummaryDelta,
|
||||
} from "./openai-responses/pureHelpers.ts";
|
||||
import { createEventEmitter } from "./openai-responses/eventEmitter.ts";
|
||||
import { buildResponsesToolCallItem } from "./responsesToolItem.ts";
|
||||
@@ -727,6 +726,37 @@ function markResponsesReasoningDeltaEmitted(state, itemId) {
|
||||
state.reasoningItemsWithDelta.add(id);
|
||||
}
|
||||
|
||||
// #9500 — streaming separator helper. When summary_index increments mid-stream
|
||||
// for a given item_id, a new reasoning segment begins; prefix "\n\n" so segments
|
||||
// don't arrive back-to-back. Only prefixes when a delta was already emitted for
|
||||
// the item AND the index advanced — never on the first segment. Lives here (not
|
||||
// in pureHelpers.ts) because it reads and mutates stream state, which the pure
|
||||
// leaf must not hold.
|
||||
function buildResponsesReasoningSummaryDelta(state, data, reasoningDelta) {
|
||||
const itemId = data.item_id != null ? String(data.item_id) : "";
|
||||
const summaryIndex = typeof data.summary_index === "number" ? data.summary_index : null;
|
||||
if (!(state.reasoningSummaryIndex instanceof Map)) {
|
||||
state.reasoningSummaryIndex = new Map();
|
||||
}
|
||||
const lastIndex = itemId ? state.reasoningSummaryIndex.get(itemId) : undefined;
|
||||
const alreadyEmittedForItem = itemId
|
||||
? state.reasoningItemsWithDelta instanceof Set && state.reasoningItemsWithDelta.has(itemId)
|
||||
: Boolean(state.reasoningDeltaEmitted);
|
||||
let deltaText = reasoningDelta;
|
||||
if (
|
||||
summaryIndex !== null &&
|
||||
lastIndex !== undefined &&
|
||||
summaryIndex > lastIndex &&
|
||||
alreadyEmittedForItem
|
||||
) {
|
||||
deltaText = `\n\n${reasoningDelta}`;
|
||||
}
|
||||
if (itemId && (lastIndex === undefined || summaryIndex > lastIndex)) {
|
||||
state.reasoningSummaryIndex.set(itemId, summaryIndex);
|
||||
}
|
||||
return deltaText;
|
||||
}
|
||||
|
||||
// #5786 — build a Chat-format reasoning delta chunk in the shape the client renders in
|
||||
// its thinking panel (`reasoning_content`, or `reasoning_text` for Copilot-compatible
|
||||
// clients). Mirrors the `response.reasoning_summary_text.delta` branch.
|
||||
|
||||
@@ -177,37 +177,6 @@ export function extractResponsesReasoningSummaryText(item) {
|
||||
.join("\n\n");
|
||||
}
|
||||
|
||||
// #9500 — streaming separator helper. When summary_index increments mid-stream
|
||||
// for a given item_id, a new reasoning segment begins; prefix "\n\n" so segments
|
||||
// don't arrive back-to-back. Only prefixes when a delta was already emitted for
|
||||
// the item AND the index advanced — never on the first segment.
|
||||
export function buildResponsesReasoningSummaryDelta(state, data, reasoningDelta) {
|
||||
const itemId = data.item_id != null ? String(data.item_id) : "";
|
||||
const summaryIndex =
|
||||
typeof data.summary_index === "number" ? data.summary_index : null;
|
||||
if (!(state.reasoningSummaryIndex instanceof Map)) {
|
||||
state.reasoningSummaryIndex = new Map();
|
||||
}
|
||||
const lastIndex = itemId ? state.reasoningSummaryIndex.get(itemId) : undefined;
|
||||
const alreadyEmittedForItem = itemId
|
||||
? state.reasoningItemsWithDelta instanceof Set &&
|
||||
state.reasoningItemsWithDelta.has(itemId)
|
||||
: Boolean(state.reasoningDeltaEmitted);
|
||||
let deltaText = reasoningDelta;
|
||||
if (
|
||||
summaryIndex !== null &&
|
||||
lastIndex !== undefined &&
|
||||
summaryIndex > lastIndex &&
|
||||
alreadyEmittedForItem
|
||||
) {
|
||||
deltaText = `\n\n${reasoningDelta}`;
|
||||
}
|
||||
if (itemId && (lastIndex === undefined || summaryIndex > lastIndex)) {
|
||||
state.reasoningSummaryIndex.set(itemId, summaryIndex);
|
||||
}
|
||||
return deltaText;
|
||||
}
|
||||
|
||||
// #7095/#7176 — when Codex exposes a reasoning item only as encrypted private
|
||||
// reasoning (no plaintext summary), chat clients would otherwise see nothing in
|
||||
// their thinking panel. Reconciles two goals that used to be in tension:
|
||||
|
||||
@@ -251,7 +251,7 @@ export function findReimplementedConditions(prodSources, testSource, testImports
|
||||
* (filtro D do git diff --diff-filter=MDR).
|
||||
*
|
||||
* `deletionAllowlist` (`_deletedWithReplacement` no test-masking-allowlist.json)
|
||||
* isenta uma deleção de duas formas, cada uma com sua própria verificação:
|
||||
* isenta uma deleção de três formas, cada uma com sua própria verificação:
|
||||
* 1. `replacement` (path string) — o substituto declarado existe no HEAD e é
|
||||
* ele próprio um arquivo de teste — o caso "reescrito em outro path sem
|
||||
* rename detectável" (conteúdo novo demais para o -M do git).
|
||||
@@ -259,12 +259,19 @@ export function findReimplementedConditions(prodSources, testSource, testImports
|
||||
* os arquivos de produção listados precisam estar ausentes no HEAD (sem
|
||||
* substituto porque não há mais código a testar). Usar apenas quando a
|
||||
* remoção do código-fonte está confirmada na mesma commit/PR.
|
||||
* 3. `strayFromCommit` (hash) + `reason` (não-vazio) — o arquivo entrou no
|
||||
* repositório POR ACIDENTE no commit declarado (ex.: um commit de docs
|
||||
* que varreu artefatos de worktree de outra sessão, caso f4e93f339d) e a
|
||||
* deleção devolve o arquivo ao seu fluxo dono (um PR/issue aberto). O
|
||||
* gate verifica via git que o commit declarado é exatamente o que ADICIONOU
|
||||
* o arquivo; o `reason` deve nomear o PR/issue dono para a revisão humana.
|
||||
* Qualquer entrada cuja condição declarada não se verifique continua flagada.
|
||||
*/
|
||||
export function evaluateDeletedFiles(
|
||||
deletedPaths,
|
||||
deletionAllowlist = {},
|
||||
fileExists = fs.existsSync
|
||||
fileExists = fs.existsSync,
|
||||
addedByCommit = lookupAddedByCommit
|
||||
) {
|
||||
const flags = [];
|
||||
for (const f of deletedPaths) {
|
||||
@@ -285,6 +292,21 @@ export function evaluateDeletedFiles(
|
||||
);
|
||||
continue;
|
||||
}
|
||||
if (entry && typeof entry.strayFromCommit === "string" && entry.strayFromCommit.trim()) {
|
||||
if (typeof entry.reason !== "string" || !entry.reason.trim()) {
|
||||
flags.push(
|
||||
`${f}: deleção allowlistada como stray mas sem \`reason\` — nomeie o PR/issue dono do arquivo`
|
||||
);
|
||||
continue;
|
||||
}
|
||||
const actual = addedByCommit(f);
|
||||
const declared = entry.strayFromCommit.trim();
|
||||
if (actual && (actual === declared || actual.startsWith(declared))) continue;
|
||||
flags.push(
|
||||
`${f}: deleção allowlistada como stray de ${declared} mas o commit que adicionou o arquivo é ${actual ?? "desconhecido"}`
|
||||
);
|
||||
continue;
|
||||
}
|
||||
flags.push(
|
||||
`${f}: arquivo de teste deletado — revisão humana obrigatória (mascaramento alto-sinal)`
|
||||
);
|
||||
@@ -292,6 +314,26 @@ export function evaluateDeletedFiles(
|
||||
return flags;
|
||||
}
|
||||
|
||||
/**
|
||||
* (subcheck 1, forma 3) Hash COMPLETO do commit que adicionou `path` (o add
|
||||
* mais recente — cobre o caso deletado-e-readicionado). `null` quando o git
|
||||
* não conhece o path.
|
||||
*/
|
||||
function lookupAddedByCommit(path) {
|
||||
try {
|
||||
const out = execFileSync("git", ["log", "--diff-filter=A", "--format=%H", "--", path], {
|
||||
encoding: "utf8",
|
||||
});
|
||||
const hashes = out
|
||||
.split("\n")
|
||||
.map((s) => s.trim())
|
||||
.filter(Boolean);
|
||||
return hashes.length ? hashes[0] : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse `git diff --name-status -M --diff-filter=DR` output, separating TRUE
|
||||
* test-file deletions ("D\tpath") from RENAMES ("R<score>\told\tnew").
|
||||
|
||||
@@ -96,9 +96,7 @@ export function firstFailureLine(out) {
|
||||
.split("\n")
|
||||
.map((l) => l.trim())
|
||||
.filter(Boolean);
|
||||
const hit = lines.find((l) =>
|
||||
/✖|✗|not ok|AssertionError|error TS|FAIL|Error:|REGRESS/i.test(l)
|
||||
);
|
||||
const hit = lines.find((l) => /✖|✗|not ok|AssertionError|error TS|FAIL|Error:|REGRESS/i.test(l));
|
||||
return (hit || lines[lines.length - 1] || "failed").slice(0, 200);
|
||||
}
|
||||
|
||||
@@ -570,10 +568,20 @@ async function main() {
|
||||
// release — that is why it is a HARD pre-flight gate.
|
||||
const slow = [
|
||||
{
|
||||
// Raised 45→100min 2026-08-05: a hermetic-env run on the loaded devbox
|
||||
// (load 7-26) was still inside invocation 1 of 3 at 76min when killed;
|
||||
// contention factor 2-3× was measured against idle windows, and no idle
|
||||
// measurement exists yet. The pre-flight's REAL condition is exactly
|
||||
// this contended one (unit runs in Promise.all with integration+vitest
|
||||
// plus whatever else the devbox carries), and there 45min provably
|
||||
// killed a healthy suite and fabricated a false base-red. The ceiling's
|
||||
// purpose — turning a genuine hang (stuck SQLite handle = zero progress
|
||||
// forever) into a visible failure — survives at 100min.
|
||||
// TODO: measure on the idle .113 box and re-tighten to ~1.8× measured.
|
||||
id: "unit",
|
||||
label: "Unit tests (full suite, CI concurrency — runs ~20-35min silently)",
|
||||
label: "Unit tests (full suite, CI concurrency — ~30-50min idle, up to ~100min under load)",
|
||||
args: ["run", "test:unit:ci"],
|
||||
timeout: 45 * 60 * 1000,
|
||||
timeout: 100 * 60 * 1000,
|
||||
},
|
||||
{
|
||||
id: "vitest",
|
||||
@@ -582,10 +590,16 @@ async function main() {
|
||||
timeout: 15 * 60 * 1000,
|
||||
},
|
||||
{
|
||||
// Measured 2026-08-05 on an idle 16-core box: 22m08s hermetic (935 tests,
|
||||
// 112 files at --test-concurrency=1, i.e. strictly serial because ~16 of
|
||||
// them bind a port or share a DB). The old "~3-10min" estimate was stale by
|
||||
// ~3x and the 20min ceiling killed a healthy run. 40min keeps the ceiling's
|
||||
// real purpose — turning a genuine hang (unreleased DB handle) into a
|
||||
// visible failure — without punishing a long-but-healthy suite.
|
||||
id: "integration",
|
||||
label: "Integration tests (~3-10min)",
|
||||
label: "Integration tests (~20-25min)",
|
||||
args: ["run", "test:integration"],
|
||||
timeout: 20 * 60 * 1000,
|
||||
timeout: 40 * 60 * 1000,
|
||||
},
|
||||
];
|
||||
if (WITH_BUILD) {
|
||||
|
||||
@@ -241,7 +241,14 @@ export class VisionBridgeGuardrail extends BaseGuardrail {
|
||||
typeof settings.visionBridgeModel === "string" && settings.visionBridgeModel.trim()
|
||||
? settings.visionBridgeModel.trim()
|
||||
: undefined;
|
||||
const bestModel = await getBestVisionModel({ fixedModel: configuredModel });
|
||||
// Propagate the same resolved credential check used by the adjacent
|
||||
// checkCreds() calls above/below (#8430) — without this, the router
|
||||
// falls back to the real DB-backed hasUsableCredentialsForModel and
|
||||
// ignores an injected `deps.hasUsableCredentials` test/DI override.
|
||||
const bestModel = await getBestVisionModel(
|
||||
{ fixedModel: configuredModel },
|
||||
{ hasUsableCredentials: checkCreds }
|
||||
);
|
||||
if (bestModel && bestModel !== model) {
|
||||
const bestUsable = await checkCreds(bestModel);
|
||||
// Only block the reroute when we KNOW the target is unusable (false).
|
||||
|
||||
@@ -689,7 +689,13 @@ test("chat pipeline applies Codex CLI fingerprint to OAuth responses requests",
|
||||
assert.equal(call.headers.Version, getCodexClientVersion());
|
||||
assert.equal(call.headers["Openai-Beta"], "responses=experimental");
|
||||
assert.equal(call.headers["X-Codex-Beta-Features"], "responses_websockets");
|
||||
assert.equal(call.headers["User-Agent"], "codex-cli/0.144.1 (Windows 10.0.26200; x64)");
|
||||
// Derive from the same source the code reads (see getCodexClientVersion() two
|
||||
// lines above) instead of pinning the literal — #9323's version bump to 0.146.0
|
||||
// broke this assertion while the rest of the test kept passing.
|
||||
assert.equal(
|
||||
call.headers["User-Agent"],
|
||||
`codex-cli/${getCodexClientVersion()} (Windows 10.0.26200; x64)`
|
||||
);
|
||||
assert.equal(call.headers["x-codex-window-id"], "conv_codex_fingerprint:0");
|
||||
assert.ok(call.headers["x-client-request-id"], "expected Codex request id header");
|
||||
assert.ok(call.headers["x-codex-turn-metadata"], "expected Codex turn metadata header");
|
||||
|
||||
@@ -11,6 +11,12 @@
|
||||
*
|
||||
* Fix: in "auto" mode, the SECURITY_MONITOR_MARKER system-prompt text is now a
|
||||
* necessary condition. `stop_sequences` alone is no longer sufficient.
|
||||
*
|
||||
* Follow-up (#9276): "always" mode previously short-circuited EVERY Claude-format
|
||||
* request unconditionally, regardless of signal shape. That let a normal chat
|
||||
* request through /v1/messages be silently swallowed by an operator's "always"
|
||||
* opt-in. The unconditional `if (mode === "always") return true` branch was
|
||||
* removed — "always" now requires the same SECURITY_MONITOR_MARKER as "auto".
|
||||
*/
|
||||
|
||||
import test from "node:test";
|
||||
@@ -52,24 +58,37 @@ test("issue #8189: 'auto' mode still short-circuits when the security-monitor ma
|
||||
);
|
||||
});
|
||||
|
||||
test("issue #8189: 'always' mode is unaffected — every Claude-format request still short-circuits (operator opted in)", () => {
|
||||
test("issue #9276: 'always' mode does NOT short-circuit without the security-monitor marker (narrowed to match 'auto')", () => {
|
||||
const body = {
|
||||
system: "You are a helpful assistant that writes CMS page templates.",
|
||||
stop_sequences: ["</block>"],
|
||||
messages: [{ role: "user", content: "hello" }],
|
||||
};
|
||||
assert.equal(
|
||||
shouldDefaultAllowClassifier(FORMATS.CLAUDE, body, "always"),
|
||||
false,
|
||||
"mode='always' must NOT short-circuit a request with no security-monitor marker, even " +
|
||||
"with stop_sequences=['</block>'] — #9276 removed the unconditional always-mode return"
|
||||
);
|
||||
});
|
||||
|
||||
test("issue #9276: 'always' mode still short-circuits when the security-monitor marker is present (operator opted in)", () => {
|
||||
const body = {
|
||||
system:
|
||||
"You are a security monitor for autonomous AI coding agents. Evaluate the following action.",
|
||||
stop_sequences: [],
|
||||
messages: [{ role: "user", content: "<transcript>Bash rm -rf /</transcript>" }],
|
||||
};
|
||||
assert.equal(
|
||||
shouldDefaultAllowClassifier(FORMATS.CLAUDE, body, "always"),
|
||||
true,
|
||||
"mode='always' must short-circuit every Claude-format request regardless of signal shape"
|
||||
"mode='always' must still short-circuit when the security-monitor marker is present"
|
||||
);
|
||||
});
|
||||
|
||||
test("issue #8189: 'off' mode (shipped default) never short-circuits", () => {
|
||||
const body = {
|
||||
system: [
|
||||
{ type: "text", text: "You are a security monitor for autonomous AI coding agents." },
|
||||
],
|
||||
system: [{ type: "text", text: "You are a security monitor for autonomous AI coding agents." }],
|
||||
stop_sequences: ["</block>"],
|
||||
};
|
||||
assert.equal(shouldDefaultAllowClassifier(FORMATS.CLAUDE, body, "off"), false);
|
||||
|
||||
@@ -132,7 +132,9 @@ test("execute (non-stream) parses <tool> reply into OpenAI tool_calls", async ()
|
||||
assert.equal(choice.finish_reason, "tool_calls");
|
||||
assert.equal(choice.message.tool_calls.length, 1);
|
||||
assert.equal(choice.message.tool_calls[0].function.name, "get_weather");
|
||||
assert.deepEqual(JSON.parse(choice.message.tool_calls[0].function.arguments), { city: "Paris" });
|
||||
assert.deepEqual(JSON.parse(choice.message.tool_calls[0].function.arguments), {
|
||||
city: "Paris",
|
||||
});
|
||||
assert.ok(
|
||||
!String(choice.message.content || "").includes("<tool>"),
|
||||
"raw tool block stripped from content"
|
||||
@@ -142,8 +144,9 @@ test("execute (non-stream) parses <tool> reply into OpenAI tool_calls", async ()
|
||||
}
|
||||
});
|
||||
|
||||
test("execute (non-stream) parses bare JSON reply into OpenAI tool_calls", async () => {
|
||||
const mock = installMock('{"name":"getWeather","arguments":{"city":"Paris"}}');
|
||||
test("execute (non-stream) does NOT promote bare JSON reply to tool_calls (#9343)", async () => {
|
||||
const bareJson = '{"name":"getWeather","arguments":{"city":"Paris"}}';
|
||||
const mock = installMock(bareJson);
|
||||
try {
|
||||
const executor = new DeepSeekWebExecutor();
|
||||
const result = await executor.execute({
|
||||
@@ -156,11 +159,17 @@ test("execute (non-stream) parses bare JSON reply into OpenAI tool_calls", async
|
||||
assert.ok(result.response.ok);
|
||||
const json = JSON.parse(await result.response.text());
|
||||
const choice = json.choices[0];
|
||||
assert.equal(choice.finish_reason, "tool_calls");
|
||||
assert.equal(choice.message.tool_calls.length, 1);
|
||||
assert.equal(choice.message.tool_calls[0].function.name, "get_weather");
|
||||
assert.deepEqual(JSON.parse(choice.message.tool_calls[0].function.arguments), { city: "Paris" });
|
||||
assert.equal(choice.message.content, null, "bare JSON tool call is stripped from content");
|
||||
assert.equal(
|
||||
choice.finish_reason,
|
||||
"stop",
|
||||
"bare JSON with no <tool> envelope must not be promoted to a tool call"
|
||||
);
|
||||
assert.ok(!choice.message.tool_calls, "no tool_calls on a bare JSON reply (#9343)");
|
||||
assert.equal(
|
||||
choice.message.content,
|
||||
bareJson,
|
||||
"bare JSON must be preserved verbatim as content, not stripped or promoted"
|
||||
);
|
||||
} finally {
|
||||
mock.restore();
|
||||
}
|
||||
|
||||
@@ -428,7 +428,7 @@ test("VB-S07: reroutes base64 image to vision model", async () => {
|
||||
|
||||
// ── VB-S03: Fail-open on vision error (via combo mapping path) ────────────
|
||||
|
||||
test("VB-S03: preserves the original image when the vision API fails (#4012)", async () => {
|
||||
test("VB-S03/#8430: combo-mapping describe failure replaces the image with an error stub (not preserved)", async () => {
|
||||
shouldVisionFail = true;
|
||||
const guardrail = createGuardrail({
|
||||
deps: {
|
||||
@@ -464,12 +464,22 @@ test("VB-S03: preserves the original image when the vision API fails (#4012)", a
|
||||
text?: string;
|
||||
}>;
|
||||
|
||||
// #4012: a failed describe must NOT replace the image with an "(unavailable)"
|
||||
// stub — the original image is preserved so a vision-capable upstream can see it.
|
||||
// SEMANTIC CHANGE (#8430): in the combo describe path (forced here via
|
||||
// checkModelHasComboMapping), when EVERY describe call fails, the upstream is
|
||||
// a confirmed non-vision model that cannot handle raw images — the raw
|
||||
// image_url part is now replaced with an "(unavailable)" error stub instead
|
||||
// of being preserved. The original #4012 preserve-raw behavior still applies
|
||||
// to the reroute path, where the upstream model might still be vision-capable
|
||||
// (see tests/unit/vision-bridge-preserve-on-failure-4012.test.ts, updated by
|
||||
// the same #8430 commit).
|
||||
const imagePart = content.find((p) => p.type === "image_url");
|
||||
assert.ok(imagePart, "original image_url part must be preserved on describe failure");
|
||||
assert.strictEqual(
|
||||
imagePart,
|
||||
undefined,
|
||||
"raw image_url must be replaced when every describe call fails in the combo path"
|
||||
);
|
||||
const unavailPart = content.find((p) => p.type === "text" && p.text?.includes("unavailable"));
|
||||
assert.strictEqual(unavailPart, undefined);
|
||||
assert.ok(unavailPart, "an 'unavailable' error stub should be present when describe fails");
|
||||
});
|
||||
|
||||
test("VB-S03: logs warning when vision API fails (via combo mapping)", async () => {
|
||||
@@ -771,21 +781,11 @@ test("VB-CRED-02: does NOT reroute to a vision model known to lack credentials",
|
||||
});
|
||||
|
||||
test("isProviderConnectionUsable rejects noauth without api key", async () => {
|
||||
const { isProviderConnectionUsable } = await import(
|
||||
"../../../src/lib/guardrails/visionBridge.ts"
|
||||
);
|
||||
assert.strictEqual(
|
||||
isProviderConnectionUsable({ authType: "noauth", apiKey: null }),
|
||||
false
|
||||
);
|
||||
assert.strictEqual(
|
||||
isProviderConnectionUsable({ authType: "apikey", apiKey: "sk-real" }),
|
||||
true
|
||||
);
|
||||
assert.strictEqual(
|
||||
isProviderConnectionUsable({ authType: "oauth", refreshToken: "rt" }),
|
||||
true
|
||||
);
|
||||
const { isProviderConnectionUsable } =
|
||||
await import("../../../src/lib/guardrails/visionBridge.ts");
|
||||
assert.strictEqual(isProviderConnectionUsable({ authType: "noauth", apiKey: null }), false);
|
||||
assert.strictEqual(isProviderConnectionUsable({ authType: "apikey", apiKey: "sk-real" }), true);
|
||||
assert.strictEqual(isProviderConnectionUsable({ authType: "oauth", refreshToken: "rt" }), true);
|
||||
assert.strictEqual(
|
||||
isProviderConnectionUsable({ authType: "apikey", apiKey: "x", testStatus: "banned" }),
|
||||
false
|
||||
|
||||
@@ -6,6 +6,8 @@ import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import dns from "node:dns";
|
||||
import { callVisionModel, type VisionModelConfig } from "@/lib/guardrails/visionBridgeHelpers";
|
||||
import { createProviderConnection } from "@/lib/db/providers";
|
||||
import { resetDbInstance } from "@/lib/db/core";
|
||||
|
||||
// Store original fetch
|
||||
const originalFetch = globalThis.fetch;
|
||||
@@ -30,6 +32,37 @@ process.on("exit", () => {
|
||||
(dns.promises as { lookup: unknown }).lookup = originalDnsLookup;
|
||||
});
|
||||
|
||||
// (#8430) getBestVisionModel now validates that a `fixedModel` has a usable
|
||||
// connection (via hasUsableCredentialsForModel, which queries the real DB)
|
||||
// before returning it — an unreachable fixedModel falls through to
|
||||
// auto-selection and, with nothing else configured either, resolves to `null`,
|
||||
// which callVisionModel turns into a hard "No vision-capable provider
|
||||
// connected" error before it ever reaches the HTTP call these tests mock.
|
||||
// The router/credential-selection logic itself is already covered by
|
||||
// visionBridgeRouter.test.ts and repro-8430.test.ts; these tests exercise
|
||||
// callVisionModel's own request/response handling, so they just need one
|
||||
// usable connection seeded per provider they use ("openai/gpt-4o-mini",
|
||||
// "anthropic/claude-3-haiku") so getBestVisionModel resolves the requested
|
||||
// fixedModel unchanged instead of null.
|
||||
test.before(async () => {
|
||||
await createProviderConnection({
|
||||
provider: "openai",
|
||||
authType: "apikey",
|
||||
name: "vision-bridge-test-openai",
|
||||
apiKey: "sk-test-openai",
|
||||
});
|
||||
await createProviderConnection({
|
||||
provider: "anthropic",
|
||||
authType: "apikey",
|
||||
name: "vision-bridge-test-anthropic",
|
||||
apiKey: "sk-test-anthropic",
|
||||
});
|
||||
});
|
||||
|
||||
test.after(() => {
|
||||
resetDbInstance();
|
||||
});
|
||||
|
||||
test("callVisionModel returns description on success", async () => {
|
||||
// Mock global fetch
|
||||
const mockResponse = {
|
||||
|
||||
@@ -15,9 +15,8 @@
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
const { validateGeminiWebProvider } = await import(
|
||||
"../../src/lib/providers/validation/webProvidersB.ts"
|
||||
);
|
||||
const { validateGeminiWebProvider } =
|
||||
await import("../../src/lib/providers/validation/webProvidersB.ts");
|
||||
|
||||
const originalFetch = globalThis.fetch;
|
||||
|
||||
@@ -25,13 +24,38 @@ test.afterEach(() => {
|
||||
globalThis.fetch = originalFetch;
|
||||
});
|
||||
|
||||
// #9407 refined this contract: a redirect to accounts.google.com/ServiceLogin is
|
||||
// specifically an EXPIRED session (valid:false with re-paste guidance), while other
|
||||
// public accounts.google.com paths remain valid-with-warning. The original #7859
|
||||
// regression (public redirect must not fall through to the generic catch → invalid)
|
||||
// is still covered — by the non-ServiceLogin variant below.
|
||||
test("gemini-web validator: 302 redirect to ServiceLogin → expired session (#9407)", async () => {
|
||||
globalThis.fetch = async (url) => {
|
||||
const target = String(url);
|
||||
if (target.includes("gemini.google.com/app")) {
|
||||
return new Response(null, {
|
||||
status: 302,
|
||||
headers: { location: "https://accounts.google.com/ServiceLogin" },
|
||||
});
|
||||
}
|
||||
throw new Error(`unexpected fetch: ${target}`);
|
||||
};
|
||||
|
||||
const result = await validateGeminiWebProvider({
|
||||
apiKey: "__Secure-1PSID=eyJvalidsession",
|
||||
});
|
||||
|
||||
assert.equal(result.valid, false);
|
||||
assert.match(result.error || "", /Session expired/i);
|
||||
});
|
||||
|
||||
test("gemini-web validator: 302 redirect to a PUBLIC host → valid (regression #7859)", async () => {
|
||||
globalThis.fetch = async (url) => {
|
||||
const target = String(url);
|
||||
if (target.includes("gemini.google.com/app")) {
|
||||
return new Response(null, {
|
||||
status: 302,
|
||||
headers: { location: "https://accounts.google.com/ServiceLogin" },
|
||||
headers: { location: "https://accounts.google.com/signin/continue" },
|
||||
});
|
||||
}
|
||||
throw new Error(`unexpected fetch: ${target}`);
|
||||
|
||||
@@ -5,16 +5,26 @@ import { resolveCodexSpawn } from "../../bin/cli/commands/launch-codex.mjs";
|
||||
|
||||
// Regression guard for #6312: on Windows the `codex` binary is an npm `.cmd`
|
||||
// shim that `spawn` cannot resolve without a shell (bare "codex" → ENOENT).
|
||||
test("resolveCodexSpawn: win32 spawns codex.cmd through a shell", () => {
|
||||
const { command, shell } = resolveCodexSpawn("win32");
|
||||
// Since #9454 resolveCodexSpawn is async and probes PATH for a native .exe
|
||||
// first; the .cmd+shell fallback below is the original #6312 contract (the
|
||||
// .exe-preferred path is covered in cli/launch-claude-exe-windows-9454.test.ts).
|
||||
test("resolveCodexSpawn: win32 spawns codex.cmd through a shell when no .exe is found", async () => {
|
||||
const { command, shell } = await resolveCodexSpawn("win32", { probe: async () => null });
|
||||
assert.equal(command, "codex.cmd");
|
||||
assert.equal(shell, true);
|
||||
});
|
||||
|
||||
test("resolveCodexSpawn: non-Windows platforms spawn the bare binary without a shell", () => {
|
||||
test("resolveCodexSpawn: non-Windows platforms spawn the bare binary without a shell", async () => {
|
||||
for (const platform of ["linux", "darwin", "freebsd"]) {
|
||||
const { command, shell } = resolveCodexSpawn(platform);
|
||||
let probeCalls = 0;
|
||||
const { command, shell } = await resolveCodexSpawn(platform, {
|
||||
probe: async () => {
|
||||
probeCalls++;
|
||||
return null;
|
||||
},
|
||||
});
|
||||
assert.equal(command, "codex", `${platform} command`);
|
||||
assert.equal(shell, undefined, `${platform} shell`);
|
||||
assert.equal(probeCalls, 0, `${platform} must not probe PATH off Windows`);
|
||||
}
|
||||
});
|
||||
|
||||
@@ -31,6 +31,16 @@ import { execFileSync } from "node:child_process";
|
||||
const originalPlatformDescriptor = Object.getOwnPropertyDescriptor(process, "platform")!;
|
||||
const originalPath = process.env.PATH;
|
||||
const originalNoSudo = process.env.OMNIROUTE_NO_SUDO;
|
||||
// The global test harness (tests/_setup/isolateDataDir.ts) sets
|
||||
// OMNIROUTE_SKIP_SYSTEM_TRUST=1 so no test mutates the host trust store —
|
||||
// which makes installCert() return before issuing any command, so this file
|
||||
// captures nothing and its install-gap assert can never pass under `npm run
|
||||
// test:unit` (it only passed when invoked directly, without the harness).
|
||||
// Clearing it here is safe: every spawned command (cp/mkdir/chmod/update-ca-*)
|
||||
// is a logging stub on PATH and OMNIROUTE_NO_SUDO=1 strips sudo, so nothing
|
||||
// touches the real system. Restored in test.after below.
|
||||
const originalSkipSystemTrust = process.env.OMNIROUTE_SKIP_SYSTEM_TRUST;
|
||||
delete process.env.OMNIROUTE_SKIP_SYSTEM_TRUST;
|
||||
|
||||
const tmpRoot = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-9442-"));
|
||||
const binDir = path.join(tmpRoot, "bin");
|
||||
@@ -68,6 +78,8 @@ test.after(() => {
|
||||
process.env.PATH = originalPath;
|
||||
if (originalNoSudo === undefined) delete process.env.OMNIROUTE_NO_SUDO;
|
||||
else process.env.OMNIROUTE_NO_SUDO = originalNoSudo;
|
||||
if (originalSkipSystemTrust === undefined) delete process.env.OMNIROUTE_SKIP_SYSTEM_TRUST;
|
||||
else process.env.OMNIROUTE_SKIP_SYSTEM_TRUST = originalSkipSystemTrust;
|
||||
fs.rmSync(tmpRoot, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
@@ -87,7 +99,10 @@ function fakeCertFile(seed: string): string {
|
||||
const der = crypto.createHash("sha256").update(seed).digest();
|
||||
const pem =
|
||||
"-----BEGIN CERTIFICATE-----\n" +
|
||||
der.toString("base64").match(/.{1,64}/g)!.join("\n") +
|
||||
der
|
||||
.toString("base64")
|
||||
.match(/.{1,64}/g)!
|
||||
.join("\n") +
|
||||
"\n-----END CERTIFICATE-----\n";
|
||||
const certPath = path.join(tmpRoot, `${seed}.crt`);
|
||||
fs.writeFileSync(certPath, pem);
|
||||
@@ -156,11 +171,7 @@ test("filesystem proof: cp under umask 0077 creates mode 0600 (why the fix is ne
|
||||
// the bare `cp` on PATH below is a logging stub from the install tests.
|
||||
execFileSync("/usr/bin/cp", [src, dst]);
|
||||
const mode = fs.statSync(dst).mode & 0o777;
|
||||
assert.equal(
|
||||
mode,
|
||||
0o600,
|
||||
"cp under umask 0077 must produce 0600 — the bug this fix repairs"
|
||||
);
|
||||
assert.equal(mode, 0o600, "cp under umask 0077 must produce 0600 — the bug this fix repairs");
|
||||
} finally {
|
||||
process.umask(oldUmask);
|
||||
}
|
||||
|
||||
@@ -2415,7 +2415,12 @@ test("claude-web validator: 401 → invalid session cookie", async () => {
|
||||
__setClaudeTlsFetchOverride(null);
|
||||
});
|
||||
|
||||
test("claude-web validator: 429 → valid (rate limited means auth passed)", async () => {
|
||||
// #9406 inverted this contract: a 429 session shows as UNHEALTHY (valid:false)
|
||||
// so the dashboard stops painting rate-limited sessions green. The dedicated
|
||||
// repro (tests/unit/repro-9406-claude-web-429-valid.test.ts) owns the full
|
||||
// contract incl. Retry-After forwarding; this sibling keeps the validator-level
|
||||
// assertion aligned with it.
|
||||
test("claude-web validator: 429 → invalid (rate limited session is not healthy, #9406)", async () => {
|
||||
__setClaudeTlsFetchOverride(async () =>
|
||||
makeClaudeTlsResponse(429, JSON.stringify({ error: "rate limited" }))
|
||||
);
|
||||
@@ -2425,7 +2430,7 @@ test("claude-web validator: 429 → valid (rate limited means auth passed)", asy
|
||||
apiKey: "sessionKey=sk-ant-sid02-good-key",
|
||||
});
|
||||
|
||||
assert.equal(result.valid, true);
|
||||
assert.equal(result.valid, false);
|
||||
__setClaudeTlsFetchOverride(null);
|
||||
});
|
||||
|
||||
|
||||
132
tests/unit/ratelimit-reservoir-refresh.test.ts
Normal file
132
tests/unit/ratelimit-reservoir-refresh.test.ts
Normal file
@@ -0,0 +1,132 @@
|
||||
/**
|
||||
* TDD regression test — Bottleneck reservoir heartbeat death after updateSettings().
|
||||
*
|
||||
* Bug: Bottleneck 2.19.5 (frozen upstream dependency, no release since 2019) has a
|
||||
* defect in `LocalDatastore#_startHeartbeat()`
|
||||
* (node_modules/bottleneck/lib/LocalDatastore.js:26-58). The guard
|
||||
* `if (this.heartbeat == null && ...)` only (re)creates the periodic
|
||||
* reservoir-refresh `setInterval` the FIRST time it runs. Every later call —
|
||||
* including the one `updateSettings()` itself triggers internally via
|
||||
* `__updateSettings__` — falls into the `else` branch and does
|
||||
* `clearInterval(this.heartbeat)` WITHOUT resetting `this.heartbeat` back to
|
||||
* `null`. Because the stale (now-invalid) reference is left in place, every
|
||||
* future `_startHeartbeat()` call keeps taking the same dead `else` branch:
|
||||
* the periodic reservoir refresh is gone forever after the FIRST manual
|
||||
* `limiter.updateSettings()` call.
|
||||
*
|
||||
* Every limiter created by rateLimitManager.ts starts with a live heartbeat
|
||||
* (the constructor call inside `getLimiter()` always sets
|
||||
* reservoirRefreshInterval/reservoirRefreshAmount — see buildLimiterDefaults()),
|
||||
* so the very first `updateFromHeaders()`/`updateFromResponseBody()`/
|
||||
* `applyRequestQueueSettings()` call against that limiter permanently kills its
|
||||
* refresh. Once the reservoir then hits 0, it never refills again.
|
||||
*
|
||||
* Production symptom: an auto-enrolled apikey connection accumulates its
|
||||
* default 60 requests, the reservoir zeroes, the request queue freezes for
|
||||
* ~120s, the watchdog fires a synthetic 502 (RATE_LIMIT_QUEUE_WEDGED), the
|
||||
* connection cools down and is excluded from weighted combo pools — turning a
|
||||
* configured 70/30 split into ~50/50 (see
|
||||
* tests/integration/combo-matrix/weighted.test.ts, the E2E proof for this
|
||||
* same bug).
|
||||
*
|
||||
* This test drives the exact same sequence directly against
|
||||
* open-sse/services/rateLimitManager.ts's public surface, without any DB or
|
||||
* HTTP layer, to isolate the Bottleneck heartbeat defect on its own.
|
||||
*/
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
const rateLimitManager = await import("../../open-sse/services/rateLimitManager.ts");
|
||||
|
||||
function wait(ms: number): Promise<void> {
|
||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
const PROVIDER = "reservoir-refresh-test-provider";
|
||||
const CONNECTION_ID = "reservoir-refresh-test-conn";
|
||||
|
||||
test.after(async () => {
|
||||
await rateLimitManager.__resetRateLimitManagerForTests();
|
||||
});
|
||||
|
||||
test("reservoir keeps refreshing after updateSettings() touches an already-heartbeating limiter", async () => {
|
||||
rateLimitManager.enableRateLimitProtection(CONNECTION_ID);
|
||||
|
||||
// 1. First call creates the limiter. Bottleneck's LocalDatastore constructor
|
||||
// starts heartbeat #1 (alive) because the default reservoirRefreshInterval/
|
||||
// reservoirRefreshAmount are always set (buildLimiterDefaults()).
|
||||
const warmup = await rateLimitManager.withRateLimit(
|
||||
PROVIDER,
|
||||
CONNECTION_ID,
|
||||
null,
|
||||
async () => "warmup"
|
||||
);
|
||||
assert.equal(warmup, "warmup");
|
||||
|
||||
// 2. Header-learned update — the first *manual* updateSettings() call on this
|
||||
// limiter. remaining(2) < limit(6000)*0.1 takes updateFromHeaders' "throttle"
|
||||
// branch, which sets a real reservoir=2 with a 1s refresh window (limit=6000
|
||||
// keeps minTime at 0 so it doesn't pace the slot consumption below). This is
|
||||
// exactly the call that kills the heartbeat under the unfixed Bottleneck bug.
|
||||
rateLimitManager.updateFromHeaders(
|
||||
PROVIDER,
|
||||
CONNECTION_ID,
|
||||
{
|
||||
"x-ratelimit-limit-requests": "6000",
|
||||
"x-ratelimit-remaining-requests": "2",
|
||||
"x-ratelimit-reset-requests": "1s",
|
||||
},
|
||||
200
|
||||
);
|
||||
|
||||
// updateFromHeaders applies the limiter update asynchronously (fire-and-forget
|
||||
// — see trackAsyncOperation in rateLimitManager.ts). Poll the test-only state
|
||||
// hook until the reservoir actually lands at 2 instead of assuming a fixed
|
||||
// number of event-loop ticks: Bottleneck's own updateSettings() goes through
|
||||
// at least one real setTimeout(0) (yieldLoop) before storeOptions reflects the
|
||||
// new value.
|
||||
const pollDeadline = Date.now() + 2000;
|
||||
let state = await rateLimitManager.__getLimiterStateForTests(PROVIDER, CONNECTION_ID, null);
|
||||
while (state?.reservoir !== 2 && Date.now() < pollDeadline) {
|
||||
await wait(10);
|
||||
state = await rateLimitManager.__getLimiterStateForTests(PROVIDER, CONNECTION_ID, null);
|
||||
}
|
||||
assert.equal(state?.reservoir, 2, "reservoir must land at 2 before the slots below are consumed");
|
||||
|
||||
// 3. Consume both reservoir slots.
|
||||
assert.equal(
|
||||
await rateLimitManager.withRateLimit(PROVIDER, CONNECTION_ID, null, async () => "slot-1"),
|
||||
"slot-1"
|
||||
);
|
||||
assert.equal(
|
||||
await rateLimitManager.withRateLimit(PROVIDER, CONNECTION_ID, null, async () => "slot-2"),
|
||||
"slot-2"
|
||||
);
|
||||
|
||||
// 4. Reservoir is now 0. A healthy Bottleneck heartbeat refills it ~1s later
|
||||
// from the reservoirRefreshInterval/reservoirRefreshAmount configured above.
|
||||
// Race a 3rd request against a 5s timer: if the heartbeat died (unfixed bug),
|
||||
// the request stays QUEUED forever and the timer wins instead.
|
||||
const RACE_TIMEOUT_MS = 5000;
|
||||
let timeoutHandle: ReturnType<typeof setTimeout> | undefined;
|
||||
const timeout = new Promise<"timed-out">((resolve) => {
|
||||
timeoutHandle = setTimeout(() => resolve("timed-out"), RACE_TIMEOUT_MS);
|
||||
});
|
||||
const request = rateLimitManager.withRateLimit(
|
||||
PROVIDER,
|
||||
CONNECTION_ID,
|
||||
null,
|
||||
async () => "slot-3" as const
|
||||
);
|
||||
|
||||
const result = await Promise.race([request, timeout]);
|
||||
if (timeoutHandle) clearTimeout(timeoutHandle);
|
||||
|
||||
assert.equal(
|
||||
result,
|
||||
"slot-3",
|
||||
'reservoir must refresh ~1s after being exhausted; "timed-out" means the Bottleneck ' +
|
||||
"heartbeat died after updateSettings() and the reservoir never refilled " +
|
||||
"(node_modules/bottleneck/lib/LocalDatastore.js _startHeartbeat clearInterval-without-null bug)"
|
||||
);
|
||||
});
|
||||
Reference in New Issue
Block a user