From b1a2ff68870cb6d1d46f8b1ca02363ca65ab0f0a Mon Sep 17 00:00:00 2001 From: Gi99lin <74502520+Gi99lin@users.noreply.github.com> Date: Mon, 17 Aug 2026 14:02:16 +0300 Subject: [PATCH] feat(proxy): non-destructive auto-disable mode for the proxy health scheduler (#10342) * feat(proxy): add non-destructive auto-disable mode for the proxy health scheduler PROXY_AUTO_REMOVE was the only opt-in action the background proxy health scheduler could take on a consistently failing proxy, and it deletes the row. For a manually-maintained proxy chain (multi-proxy pool/rotation, #6365) that is too destructive just to exclude a temporarily-dead member. Add PROXY_AUTO_DISABLE as a sibling flag: at the same consecutive-failure threshold it soft-disables the proxy (status "dead") instead of removing it. "dead" is already one of the statuses the pool/rotation alive-filter excludes, so a disabled proxy drops out of the active chain immediately with no other code changes. The scheduler keeps probing dead proxies on its normal interval, and the existing recovery branch (previously autoRemove-only) re-activates it automatically once it starts answering again. decision.ts's decideProxyHealthAction() gets an optional `autoDisable` input (defaults to false, so existing callers are unaffected) and a "dead" status value; scheduler.ts wires the new PROXY_AUTO_DISABLE env flag through. If both flags are set, auto-remove wins. getProxyHealthStats() now also surfaces the registry `status` so operators can see when a proxy was auto-disabled, and ProxyStatusBadge now treats the full "not alive" status set (not just the literal string "inactive") as inactive in the dashboard. * test(proxy): assert registry status in getProxyHealthStats output The non-destructive auto-disable change added the live registry status to the stats object returned by getProxyHealthStats. Align the pre-existing db-proxies-crud assertion with the intended output shape. Co-authored-by: diegosouzapw * fix(proxy): preserve auto-disabled status in dashboard edits Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> --------- Co-authored-by: adevwithpurpose Co-authored-by: Gi99lin Co-authored-by: diegosouzapw Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> --- .env.example | 7 ++ docs/ops/PROXY_GUIDE.md | 44 ++++++++ docs/reference/ENVIRONMENT.md | 1 + .../components/ProxyRegistryManager.tsx | 8 +- .../settings/components/ProxyStatusBadge.tsx | 9 +- src/lib/db/proxies.ts | 12 ++- src/lib/proxyHealth/decision.ts | 51 ++++++--- src/lib/proxyHealth/scheduler.ts | 52 ++++++--- src/shared/validation/schemas/proxy.ts | 2 +- tests/unit/db-proxies-crud.test.ts | 1 + ...proxy-health-auto-disable-decision.test.ts | 102 ++++++++++++++++++ tests/unit/proxy-registry.test.ts | 13 ++- ...gistryManager-credential-autofill.test.tsx | 70 +++++++++++- 13 files changed, 336 insertions(+), 36 deletions(-) create mode 100644 tests/unit/proxy-health-auto-disable-decision.test.ts diff --git a/.env.example b/.env.example index d415d6b3c7..483dd426ca 100644 --- a/.env.example +++ b/.env.example @@ -1894,6 +1894,13 @@ APP_LOG_TO_FILE=true # PROXY_AUTO_REMOVE=false # Consecutive failures before an auto-remove fires. Default: 3. # PROXY_AUTO_REMOVE_AFTER=3 +# Set "true" to let the scheduler auto-disable (status "dead") proxies after +# repeated failures instead of deleting them. Non-destructive alternative to +# PROXY_AUTO_REMOVE — the row stays in the registry, drops out of pool/rotation +# resolution immediately, and is automatically re-activated once it starts +# answering probes again. Shares the PROXY_AUTO_REMOVE_AFTER threshold above. +# If both PROXY_AUTO_REMOVE and PROXY_AUTO_DISABLE are "true", auto-remove wins. +# PROXY_AUTO_DISABLE=false # Let automated reachability probes (the scheduler + the "Test All" button) WRITE # a proxy's status. Default "false": probes are read-only and never deactivate a # proxy — only the operator sets active/inactive (a flaky probe must not strand an diff --git a/docs/ops/PROXY_GUIDE.md b/docs/ops/PROXY_GUIDE.md index 81535d1cbf..075759fad0 100644 --- a/docs/ops/PROXY_GUIDE.md +++ b/docs/ops/PROXY_GUIDE.md @@ -817,6 +817,50 @@ The proxy is **not deleted** — it's marked unhealthy and won't be selected unt --- +## Automatic Failure Exclusion for Your Own Proxies + +`failOneproxyProxy()` above only covers the 1proxy marketplace pool, which already +auto-degrades on failure (see [Proxy Quality Scores](#proxy-quality-scores)). For +proxies **you** added to the registry, the background health scheduler +(`src/lib/proxyHealth/scheduler.ts`) provides the same "exclude a dead member from +the chain automatically" behavior, without deleting anything: + +```bash +# .env — soft-disable a proxy after 3 consecutive failed probes, re-enable it +# automatically once it starts answering probes again. +PROXY_AUTO_DISABLE=true +PROXY_AUTO_REMOVE_AFTER=3 +``` + +How it fits into a multi-proxy chain: + +1. The scheduler probes every registered proxy every `PROXY_HEALTH_INTERVAL_MS` + (default 10 min; minimum 1 min). +2. After `PROXY_AUTO_REMOVE_AFTER` consecutive **conclusive** failures (a real + connection failure — a timeout or the probe target's own 5xx never counts, see + [Proxy Health Checking](#proxy-health-checking-v3816)), the proxy's `status` is + set to `dead`. +3. `dead` is one of the statuses the alive-status filter used by pool/rotation + resolution excludes, so a scope's rotation (round-robin / random / sticky / + latency — see [Rotation Strategy Decision Tree](#rotation-strategy-decision-tree)) + immediately stops handing that proxy to new requests. No other proxies in the + pool are affected, and the whole pool never silently falls back to a direct + connection — see the [4-Level Proxy System](#4-level-proxy-system) fail-closed + guard. +4. The scheduler keeps probing `dead` proxies on the same interval. The next + successful probe flips `status` back to `active` and it re-enters rotation — + no manual re-add required. + +This is deliberately **opt-in and non-destructive**: by default the scheduler only +counts and logs failures (see policy C in `decision.ts`), and `PROXY_AUTO_DISABLE` +never deletes a row — that is what the separate, more aggressive +`PROXY_AUTO_REMOVE` flag is for. If both are set to `true`, `PROXY_AUTO_REMOVE` +wins (a proxy about to be deleted has no use for a soft-disable in between). See +the [Environment Config](../reference/ENVIRONMENT.md) reference for the full +variable list. + +--- + > 📖 **Related documentation:** > > - [User Guide](../guides/USER_GUIDE.md) — General setup and configuration diff --git a/docs/reference/ENVIRONMENT.md b/docs/reference/ENVIRONMENT.md index 6923092247..7c39edd8d1 100644 --- a/docs/reference/ENVIRONMENT.md +++ b/docs/reference/ENVIRONMENT.md @@ -999,6 +999,7 @@ Anthropic-compatible provider instead. | `PROXY_HEALTH_AUTO_DEACTIVATE` | `false` | `src/lib/proxyHealth/statusPolicy.ts` | When `false` (default), automated reachability probes (the scheduler + the `/api/settings/proxies/auto-test` "Test All" button) are **read-only** and never write a proxy's status — only the operator sets active/inactive, so a flaky probe can't strand an assigned proxy (#6246). Set `true` to restore the legacy test-and-set behaviour. | | `PROXY_AUTO_REMOVE` | `false` | `src/lib/proxyHealth/scheduler.ts` | Set `true` to let the scheduler auto-remove proxies after repeated consecutive failures. | | `PROXY_AUTO_REMOVE_AFTER` | `3` | `src/lib/proxyHealth/scheduler.ts` | Consecutive failures before the scheduler auto-removes a proxy (when `PROXY_AUTO_REMOVE=true`). | +| `PROXY_AUTO_DISABLE` | `false` | `src/lib/proxyHealth/scheduler.ts` | Set `true` to let the scheduler soft-disable (status `dead`, never deleted) a proxy after repeated consecutive failures, instead of removing it. Non-destructive alternative to `PROXY_AUTO_REMOVE`: the proxy drops out of pool/rotation resolution immediately (the alive-status filter used by scope-pool resolution already excludes it) and is automatically re-activated once it starts passing probes again. Shares the `PROXY_AUTO_REMOVE_AFTER` threshold. If both flags are `true`, `PROXY_AUTO_REMOVE` wins. | | `OMNIROUTE_CONTROL_PLANE_PROXY_DIRECT_FALLBACK` | `false` | `src/shared/constants/featureFlagDefinitions.ts` | Allow OAuth and provider validation flows to bypass a pinned proxy and connect directly when proxy reachability pre-checks fail. Effective precedence is Feature Flags DB override > env var > default. | | `RATE_LIMIT_MAX_WAIT_MS` | `15000` (15s) | `open-sse/services/rateLimitManager.ts` | Max time to wait on a 429 before failing the request. | | `RATE_LIMIT_MAX_QUEUE_DEPTH` | `0` (disabled) | `open-sse/services/rateLimitManager.ts` | Queue admission cap: reject with a 429 `queue_full` once this many requests are already queued. `0` = unbounded (default). | diff --git a/src/app/(dashboard)/dashboard/settings/components/ProxyRegistryManager.tsx b/src/app/(dashboard)/dashboard/settings/components/ProxyRegistryManager.tsx index 47d0334041..990d17988d 100644 --- a/src/app/(dashboard)/dashboard/settings/components/ProxyRegistryManager.tsx +++ b/src/app/(dashboard)/dashboard/settings/components/ProxyRegistryManager.tsx @@ -1046,9 +1046,11 @@ import { className="w-full px-3 py-2 rounded bg-bg-subtle border border-border" value={form.status} onChange={(e) => setForm((prev) => ({ ...prev, status: e.target.value }))} + data-testid="proxy-registry-status-select" > + {form.status === "dead" && } @@ -1281,7 +1283,11 @@ import { > {items - .filter((item) => !poolMembers.includes(item.id)) + .filter( + (item) => + !poolMembers.includes(item.id) && + (item.status ?? "").toLowerCase() !== "dead" + ) .map((item) => (