* feat(combo): universal cooldown-aware retry & auto-strategy combo-ref guard
Two changes:
1. Universal cooldown-aware retry (combo.ts):
- Remove strategy==="quota-share" gate from comboCooldownWaitEnabled
- Enables all 18 combo strategies (priority, weighted, round-robin, etc.)
to wait out a short transient cooldown and retry the full set
- Non-quota-share strategies use shouldWaitForComboCooldown directly
with earliestRetryAfter and reason="rate_limit" (no per-model lockout)
- quota-share retains its existing per-target model lockout logic
2. Auto-strategy combo-ref guard (autoStrategy.ts):
- expandAutoComboCandidatePool now detects kind==="combo-ref" entries
- When present, returns eligibleTargets without expanding to ALL providers
- Fixes scenario where an "auto" combo with combo-ref delegates to a
sub-combo but pulls in every model from every active provider
* feat(combo): global comboTimeoutMs + aggregated error diagnostics
Adds two features to improve combo resilience and debuggability:
1. Global combo timeout (comboTimeoutMs)
- Configurable per-combo via DEFAULT_COMBO_CONFIG (default 0 = disabled)
- After each target completes, checks if total elapsed time exceeds limit
- When exceeded, stops trying further targets and returns 504 immediately
- Backward-compatible: 0 preserves legacy unlimited-iteration behavior
2. Aggregated error diagnostics
- comboErrors array accumulates per-model failure details (model, status, error)
- On combo timeout or all-models-exhausted, returns a message listing the
first (up to 5) model-level errors with their HTTP status codes
- Enables operators to see WHICH models failed and WHY without digging
through individual server logs
* test(combo): cover universal cooldown-aware retry, comboTimeoutMs, and combo-ref guard
Adds/updates automated coverage for this PR's production changes (PR Test
Policy requires tests alongside src/open-sse/electron/bin changes):
- Update the "preserves the first failure status" expectation in
combo-routing-engine.test.ts: the aggregated per-model error-diagnostics
suffix is new intended output, not a regression.
- Add two new tests for the global comboTimeoutMs feature: the combo stops
dispatching further targets and returns 504/COMBO_TIMEOUT with aggregated
diagnostics once the ceiling trips, and comboTimeoutMs=0 (default) never
trips it.
- Rewrite the non-quota-share (priority) cooldown-wait scenario in
combo-quota-share-cooldown-wait-timing.test.ts: comboCooldownWait is no
longer gated on strategy === "quota-share", so a priority combo now waits
out a short 429 and re-dispatches too (via shouldWaitForComboCooldown with
reason "rate_limit"), instead of propagating immediately as before. Also
adds the disabled-flag counterpart for parity with the quota-share suite.
- Add coverage for the #COMBO-REF guard in expandAutoComboCandidatePool:
a combo whose models array contains a kind:"combo-ref" entry must not be
expanded to every model of every active provider.
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
* test(combo): keep the new comboTimeoutMs tests free of no-explicit-any
The two new tests initially copied the neighbouring tests' `any`-typed
handleSingleModel params / json() casts. Those neighbours are pre-existing
violations frozen in config/quality/eslint-suppressions.json at a count of 261
for this file, so the 7 new occurrences pushed it to 268 and broke `npm run
lint` (no-explicit-any is an error in tests/ since #6218; new violations must
be fixed, not re-frozen).
Type the new tests properly instead: `unknown`/`string` params, a
ComboErrorPayload interface for the parsed body, and drop the unused
`relayOptions: null as any` (the sibling combo cooldown suites already omit
it). Back to exactly 261 — the suppression file is untouched.
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
* fix(combo): gate the universal cooldown retry on the REAL lock reason, not a hardcoded "rate_limit"
The universal cooldown-aware retry kept the quota-share path on
resolveComboCooldownWaitDecision but gave every OTHER strategy a shortcut that
hardcoded `reason: "rate_limit"` and fed shouldWaitForComboCooldown the
earliestRetryAfter directly.
comboCooldownRetry.ts documents TWO deliberate barriers ("SECURITY —
quota_exhausted must be excluded"): (1) the reason allow-list, and (2) the
maxWaitMs ceiling, explicitly called the SECOND barrier. Hardcoding the reason
removed barrier 1 for 17 of the 18 strategies and left only the ceiling — which
does NOT cover a quota_exhausted lock whose wait lands under maxWaitMs. In that
case the combo waits, redispatches against a model that is locked until the
quota resets, and burns the retry budget for nothing.
The shortcut's premise ("non-quota-share combos have no per-connection model
lockout tracking") is also false: recordModelLockoutFailure in the target loop
is not gated on quota-share, so every strategy records model lockouts and the
real reason is always available.
Fix: one decision path for every strategy, always through
resolveComboCooldownWaitDecision, so the reason always crosses the allow-list.
The lock lookup is now keyed on each TARGET's own model (via a new third
`target` arg on lookupLock) — quota-share combos are single-model/multi-account
so this is identical to the previous orderedTargets[0] behavior, but
heterogeneous combos (priority, weighted, round-robin, …) carry a different
model per target and would otherwise miss every lock but the first.
Regression guard (tests/unit/serial/combo-quota-share-cooldown-wait-timing.test.ts):
a priority combo where modelLockout.errorCodes=[403] leaves the 403's
quota_exhausted lock as the only one in play while a 429 crystallizes the
status, and the resulting wait is short enough that the ceiling lets it through
— so only the allow-list can stop it. Verified failing-then-passing: with the
hardcoded reason the combo makes 6 dispatches (wait+redispatch x2); with the
real reason it makes 2 and propagates the 429.
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
* chore(quality): rebaseline file-size for PR #7301 own growth (combo +91, combo-routing-engine test +68)
---------
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
* fix(sse): de-flake timing-sensitive combo cooldown/breaker tests (#6803)
Extracts 3 wall-clock-sensitive assertions (combo-quota-share cooldown
ceiling x2, circuit-breaker HALF_OPEN race) into tests/unit/serial/
(--test-concurrency=1, the repo's established remedy for this class of
test) and widens their margins, since a starved CI-runner event loop can
blow even a serialized test's timing window. Also adds an explicit 30s
vitest timeout to the MCP audit shutdown test, which had no override and
inherited vitest's 5000ms default.
Regression proof: reproduced RED locally under real devbox CPU
contention (2644ms/1796ms elapsed vs the old 1500ms ceiling, exactly the
reported failure mode); confirmed GREEN after the fix under the same
contention.
* fix(quality): register new serial timing tests + prune stale any-suppression count
- stryker.conf.json: add tests/unit/serial/combo-quota-share-cooldown-wait-timing.test.ts
and tests/unit/serial/combo-strategy-fallbacks-half-open-timing.test.ts to
tap.testFiles so their mutant kills count for accountFallback.ts and
circuitBreaker.ts (PR #6897 added these files but didn't register them).
- eslint-suppressions.json: combo-strategy-fallbacks.test.ts's no-explicit-any
suppression count was stale (35) after this PR trimmed 2 any-usages out of
the file when extracting the half-open timing test; corrected to 33.
Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>
---------
Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>
* test(ci): quarantine concurrency-sensitive flakes into a serial pass (tests/unit/serial/)
glm-coding-plan-monthly-3580, quota-division-blocks and provider-health-autopilot
fail under --test-concurrency>1 CPU contention but pass isolated (v3.8.45 release
benchmark: this class cost ~28min re-runs per CI round). They now live in
tests/unit/serial/ and run in a dedicated --test-concurrency=1 step appended to
every runner (test:unit, :ci, :ci:shard, :fast, :shard:1/2, coverage:runner —
sharded variants shard the serial pass too, so concurrent shard jobs never
self-collide). Discovery + TIA gates know the new glob; stryker.conf.json path
updated. Guard: tests/unit/test-serial-quarantine.test.ts (4 tests).
* test(ci): quarantine combo-health-autopilot too (async logger writes after test end under load)
Fresh evidence from this PR's own CI: all 3 subtests pass but an async log write
lands after the test ends (ENOENT app.log -> uncaughtException) — same
concurrency-flake family. Moved to tests/unit/serial/, imports adjusted,
stryker path updated, guard test now asserts 4 quarantined files.
