diff --git a/.env.example b/.env.example index d415d6b3c7..483dd426ca 100644 --- a/.env.example +++ b/.env.example @@ -1894,6 +1894,13 @@ APP_LOG_TO_FILE=true # PROXY_AUTO_REMOVE=false # Consecutive failures before an auto-remove fires. Default: 3. # PROXY_AUTO_REMOVE_AFTER=3 +# Set "true" to let the scheduler auto-disable (status "dead") proxies after +# repeated failures instead of deleting them. Non-destructive alternative to +# PROXY_AUTO_REMOVE — the row stays in the registry, drops out of pool/rotation +# resolution immediately, and is automatically re-activated once it starts +# answering probes again. Shares the PROXY_AUTO_REMOVE_AFTER threshold above. +# If both PROXY_AUTO_REMOVE and PROXY_AUTO_DISABLE are "true", auto-remove wins. +# PROXY_AUTO_DISABLE=false # Let automated reachability probes (the scheduler + the "Test All" button) WRITE # a proxy's status. Default "false": probes are read-only and never deactivate a # proxy — only the operator sets active/inactive (a flaky probe must not strand an diff --git a/docs/ops/PROXY_GUIDE.md b/docs/ops/PROXY_GUIDE.md index 81535d1cbf..075759fad0 100644 --- a/docs/ops/PROXY_GUIDE.md +++ b/docs/ops/PROXY_GUIDE.md @@ -817,6 +817,50 @@ The proxy is **not deleted** — it's marked unhealthy and won't be selected unt --- +## Automatic Failure Exclusion for Your Own Proxies + +`failOneproxyProxy()` above only covers the 1proxy marketplace pool, which already +auto-degrades on failure (see [Proxy Quality Scores](#proxy-quality-scores)). For +proxies **you** added to the registry, the background health scheduler +(`src/lib/proxyHealth/scheduler.ts`) provides the same "exclude a dead member from +the chain automatically" behavior, without deleting anything: + +```bash +# .env — soft-disable a proxy after 3 consecutive failed probes, re-enable it +# automatically once it starts answering probes again. +PROXY_AUTO_DISABLE=true +PROXY_AUTO_REMOVE_AFTER=3 +``` + +How it fits into a multi-proxy chain: + +1. The scheduler probes every registered proxy every `PROXY_HEALTH_INTERVAL_MS` + (default 10 min; minimum 1 min). +2. After `PROXY_AUTO_REMOVE_AFTER` consecutive **conclusive** failures (a real + connection failure — a timeout or the probe target's own 5xx never counts, see + [Proxy Health Checking](#proxy-health-checking-v3816)), the proxy's `status` is + set to `dead`. +3. `dead` is one of the statuses the alive-status filter used by pool/rotation + resolution excludes, so a scope's rotation (round-robin / random / sticky / + latency — see [Rotation Strategy Decision Tree](#rotation-strategy-decision-tree)) + immediately stops handing that proxy to new requests. No other proxies in the + pool are affected, and the whole pool never silently falls back to a direct + connection — see the [4-Level Proxy System](#4-level-proxy-system) fail-closed + guard. +4. The scheduler keeps probing `dead` proxies on the same interval. The next + successful probe flips `status` back to `active` and it re-enters rotation — + no manual re-add required. + +This is deliberately **opt-in and non-destructive**: by default the scheduler only +counts and logs failures (see policy C in `decision.ts`), and `PROXY_AUTO_DISABLE` +never deletes a row — that is what the separate, more aggressive +`PROXY_AUTO_REMOVE` flag is for. If both are set to `true`, `PROXY_AUTO_REMOVE` +wins (a proxy about to be deleted has no use for a soft-disable in between). See +the [Environment Config](../reference/ENVIRONMENT.md) reference for the full +variable list. + +--- + > 📖 **Related documentation:** > > - [User Guide](../guides/USER_GUIDE.md) — General setup and configuration diff --git a/docs/reference/ENVIRONMENT.md b/docs/reference/ENVIRONMENT.md index 6923092247..7c39edd8d1 100644 --- a/docs/reference/ENVIRONMENT.md +++ b/docs/reference/ENVIRONMENT.md @@ -999,6 +999,7 @@ Anthropic-compatible provider instead. | `PROXY_HEALTH_AUTO_DEACTIVATE` | `false` | `src/lib/proxyHealth/statusPolicy.ts` | When `false` (default), automated reachability probes (the scheduler + the `/api/settings/proxies/auto-test` "Test All" button) are **read-only** and never write a proxy's status — only the operator sets active/inactive, so a flaky probe can't strand an assigned proxy (#6246). Set `true` to restore the legacy test-and-set behaviour. | | `PROXY_AUTO_REMOVE` | `false` | `src/lib/proxyHealth/scheduler.ts` | Set `true` to let the scheduler auto-remove proxies after repeated consecutive failures. | | `PROXY_AUTO_REMOVE_AFTER` | `3` | `src/lib/proxyHealth/scheduler.ts` | Consecutive failures before the scheduler auto-removes a proxy (when `PROXY_AUTO_REMOVE=true`). | +| `PROXY_AUTO_DISABLE` | `false` | `src/lib/proxyHealth/scheduler.ts` | Set `true` to let the scheduler soft-disable (status `dead`, never deleted) a proxy after repeated consecutive failures, instead of removing it. Non-destructive alternative to `PROXY_AUTO_REMOVE`: the proxy drops out of pool/rotation resolution immediately (the alive-status filter used by scope-pool resolution already excludes it) and is automatically re-activated once it starts passing probes again. Shares the `PROXY_AUTO_REMOVE_AFTER` threshold. If both flags are `true`, `PROXY_AUTO_REMOVE` wins. | | `OMNIROUTE_CONTROL_PLANE_PROXY_DIRECT_FALLBACK` | `false` | `src/shared/constants/featureFlagDefinitions.ts` | Allow OAuth and provider validation flows to bypass a pinned proxy and connect directly when proxy reachability pre-checks fail. Effective precedence is Feature Flags DB override > env var > default. | | `RATE_LIMIT_MAX_WAIT_MS` | `15000` (15s) | `open-sse/services/rateLimitManager.ts` | Max time to wait on a 429 before failing the request. | | `RATE_LIMIT_MAX_QUEUE_DEPTH` | `0` (disabled) | `open-sse/services/rateLimitManager.ts` | Queue admission cap: reject with a 429 `queue_full` once this many requests are already queued. `0` = unbounded (default). | diff --git a/src/app/(dashboard)/dashboard/settings/components/ProxyRegistryManager.tsx b/src/app/(dashboard)/dashboard/settings/components/ProxyRegistryManager.tsx index 47d0334041..990d17988d 100644 --- a/src/app/(dashboard)/dashboard/settings/components/ProxyRegistryManager.tsx +++ b/src/app/(dashboard)/dashboard/settings/components/ProxyRegistryManager.tsx @@ -1046,9 +1046,11 @@ import { className="w-full px-3 py-2 rounded bg-bg-subtle border border-border" value={form.status} onChange={(e) => setForm((prev) => ({ ...prev, status: e.target.value }))} + data-testid="proxy-registry-status-select" > + {form.status === "dead" && } @@ -1281,7 +1283,11 @@ import { > {items - .filter((item) => !poolMembers.includes(item.id)) + .filter( + (item) => + !poolMembers.includes(item.id) && + (item.status ?? "").toLowerCase() !== "dead" + ) .map((item) => (