mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-07-26 09:52:11 +03:00
577bbf3e47
* feat(combo): universal cooldown-aware retry & auto-strategy combo-ref guard
Two changes:
1. Universal cooldown-aware retry (combo.ts):
- Remove strategy==="quota-share" gate from comboCooldownWaitEnabled
- Enables all 18 combo strategies (priority, weighted, round-robin, etc.)
to wait out a short transient cooldown and retry the full set
- Non-quota-share strategies use shouldWaitForComboCooldown directly
with earliestRetryAfter and reason="rate_limit" (no per-model lockout)
- quota-share retains its existing per-target model lockout logic
2. Auto-strategy combo-ref guard (autoStrategy.ts):
- expandAutoComboCandidatePool now detects kind==="combo-ref" entries
- When present, returns eligibleTargets without expanding to ALL providers
- Fixes scenario where an "auto" combo with combo-ref delegates to a
sub-combo but pulls in every model from every active provider
* feat(combo): global comboTimeoutMs + aggregated error diagnostics
Adds two features to improve combo resilience and debuggability:
1. Global combo timeout (comboTimeoutMs)
- Configurable per-combo via DEFAULT_COMBO_CONFIG (default 0 = disabled)
- After each target completes, checks if total elapsed time exceeds limit
- When exceeded, stops trying further targets and returns 504 immediately
- Backward-compatible: 0 preserves legacy unlimited-iteration behavior
2. Aggregated error diagnostics
- comboErrors array accumulates per-model failure details (model, status, error)
- On combo timeout or all-models-exhausted, returns a message listing the
first (up to 5) model-level errors with their HTTP status codes
- Enables operators to see WHICH models failed and WHY without digging
through individual server logs
* test(combo): cover universal cooldown-aware retry, comboTimeoutMs, and combo-ref guard
Adds/updates automated coverage for this PR's production changes (PR Test
Policy requires tests alongside src/open-sse/electron/bin changes):
- Update the "preserves the first failure status" expectation in
combo-routing-engine.test.ts: the aggregated per-model error-diagnostics
suffix is new intended output, not a regression.
- Add two new tests for the global comboTimeoutMs feature: the combo stops
dispatching further targets and returns 504/COMBO_TIMEOUT with aggregated
diagnostics once the ceiling trips, and comboTimeoutMs=0 (default) never
trips it.
- Rewrite the non-quota-share (priority) cooldown-wait scenario in
combo-quota-share-cooldown-wait-timing.test.ts: comboCooldownWait is no
longer gated on strategy === "quota-share", so a priority combo now waits
out a short 429 and re-dispatches too (via shouldWaitForComboCooldown with
reason "rate_limit"), instead of propagating immediately as before. Also
adds the disabled-flag counterpart for parity with the quota-share suite.
- Add coverage for the #COMBO-REF guard in expandAutoComboCandidatePool:
a combo whose models array contains a kind:"combo-ref" entry must not be
expanded to every model of every active provider.
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
* test(combo): keep the new comboTimeoutMs tests free of no-explicit-any
The two new tests initially copied the neighbouring tests' `any`-typed
handleSingleModel params / json() casts. Those neighbours are pre-existing
violations frozen in config/quality/eslint-suppressions.json at a count of 261
for this file, so the 7 new occurrences pushed it to 268 and broke `npm run
lint` (no-explicit-any is an error in tests/ since #6218; new violations must
be fixed, not re-frozen).
Type the new tests properly instead: `unknown`/`string` params, a
ComboErrorPayload interface for the parsed body, and drop the unused
`relayOptions: null as any` (the sibling combo cooldown suites already omit
it). Back to exactly 261 — the suppression file is untouched.
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
* fix(combo): gate the universal cooldown retry on the REAL lock reason, not a hardcoded "rate_limit"
The universal cooldown-aware retry kept the quota-share path on
resolveComboCooldownWaitDecision but gave every OTHER strategy a shortcut that
hardcoded `reason: "rate_limit"` and fed shouldWaitForComboCooldown the
earliestRetryAfter directly.
comboCooldownRetry.ts documents TWO deliberate barriers ("SECURITY —
quota_exhausted must be excluded"): (1) the reason allow-list, and (2) the
maxWaitMs ceiling, explicitly called the SECOND barrier. Hardcoding the reason
removed barrier 1 for 17 of the 18 strategies and left only the ceiling — which
does NOT cover a quota_exhausted lock whose wait lands under maxWaitMs. In that
case the combo waits, redispatches against a model that is locked until the
quota resets, and burns the retry budget for nothing.
The shortcut's premise ("non-quota-share combos have no per-connection model
lockout tracking") is also false: recordModelLockoutFailure in the target loop
is not gated on quota-share, so every strategy records model lockouts and the
real reason is always available.
Fix: one decision path for every strategy, always through
resolveComboCooldownWaitDecision, so the reason always crosses the allow-list.
The lock lookup is now keyed on each TARGET's own model (via a new third
`target` arg on lookupLock) — quota-share combos are single-model/multi-account
so this is identical to the previous orderedTargets[0] behavior, but
heterogeneous combos (priority, weighted, round-robin, …) carry a different
model per target and would otherwise miss every lock but the first.
Regression guard (tests/unit/serial/combo-quota-share-cooldown-wait-timing.test.ts):
a priority combo where modelLockout.errorCodes=[403] leaves the 403's
quota_exhausted lock as the only one in play while a 429 crystallizes the
status, and the resulting wait is short enough that the ceiling lets it through
— so only the allow-list can stop it. Verified failing-then-passing: with the
hardcoded reason the combo makes 6 dispatches (wait+redispatch x2); with the
real reason it makes 2 and propagates the 429.
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
* chore(quality): rebaseline file-size for PR #7301 own growth (combo +91, combo-routing-engine test +68)
---------
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>