mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-08-02 05:12:11 +03:00
294751d8a3
The auto-enabled Bottleneck safety net for API-key providers can desync
its internal state and silently stop dispatching jobs (queued > 0,
running == 0, executing == 0). Once that happens nothing recovers without
a process restart. This change adds a self-heal layer plus operator
visibility, and keeps existing behavior intact by default.
- Watchdog (30s tick) detects wedged limiters and force-resets them with
stop({dropWaitingJobs:true}) so queued callers actually fail rather than
stall forever.
- RATE_LIMIT_AUTO_ENABLE env var overrides the dashboard auto-enable
toggle (highest precedence). Lets operators flip the safety net off
during an incident without needing dashboard access.
- disableRateLimitProtection now uses stop({dropWaitingJobs:true})
instead of disconnect(); disconnect leaks queued promises (observed
while debugging this).
- STAGE_TRACE checkpoints in chatCore (post_injection, post_translation,
pre/post_semaphore, pre/inside/post_rate_limit, pre/post_executor)
pinpoint which await a hung request was stuck on.
- New /api/admin/concurrency endpoint exposes per-limiter counts and
semaphore stats so operators can see liveness in real time.
- Test coverage for the env-var override.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>