Api-key 403 bodies with Cloudflare error 1010 / browser_signature_banned /
retryable:false were treated as short AUTH_ERROR cooldowns, so the chat loop
waited ~21–33s before falling through. Return cooldownMs:0 so the tier fails
fast without permanently banning the account.
* perf(db): cap modelLockouts eviction at 1000 entries
- Add MODEL_LOCKOUT_EVICTION_CAP constant set to 1000
- Evict oldest entries in insertion order when cap exceeded
- modelFailureState eviction skips entries still in modelLockouts
- Prevents unbounded memory growth under sustained load
* test(db): add lockout eviction test, export helpers
- Extract evictModelLockoutOverflow() from ensureCleanupTimer for testability
- Add getModelLockoutSize() and export MODEL_LOCKOUT_EVICTION_CAP
- 3 tests: overflow eviction, under-cap idempotent, keeps recent entries
* fix(resilience): never evict a still-active model lockout in evictModelLockoutOverflow()
evictModelLockoutOverflow() walked modelLockouts in raw insertion order
and deleted the oldest N regardless of entry.until. If the map exceeded
1000 entries while some of the oldest were still well within their
active cooldown window, eviction silently deleted them — isModelLocked()
would then report the model as unlocked even though it was still
rate-limited/quota-exhausted, undermining the Model Lockout resilience
layer. Reproduced live: lock a "victim" model first, lock 1000 more
distinct models, call evictModelLockoutOverflow(), and isModelLocked()
on the victim flips from true to false despite ~60s of cooldown left.
Fix: only entries whose `until` has already elapsed are eviction
candidates. ensureCleanupTimer()'s tick already runs
cleanupModelLockKey() on every key immediately before calling this
function, which removes genuinely-expired entries — so anything active
left over the cap is, by construction, a real in-progress cooldown and
must never be silently dropped. If the map is still over cap purely
from active entries, the cap becomes a (rare-case) soft bound rather
than trading away correctness.
The 3 existing tests only asserted Map.size shrank to the cap, which
is exactly the buggy behavior being fixed (they created only
active/never-expiring locks and expected mass eviction regardless).
Rewrote them to use lockModel()'s cooldownMs sign to construct
deterministic active vs. already-expired entries (no real sleeps
needed), and added a direct regression test asserting a specific
still-active key survives eviction via isModelLocked() while an
overflow of expired fillers is correctly evicted down to the cap.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
* refactor(resilience): extract lockout eviction to module (file-size cap)
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: oyi77 <oyi77@users.noreply.github.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
