mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-08-17 20:52:15 +03:00
feat(proxy): non-destructive auto-disable mode for the proxy health scheduler (#10342)
* feat(proxy): add non-destructive auto-disable mode for the proxy health scheduler PROXY_AUTO_REMOVE was the only opt-in action the background proxy health scheduler could take on a consistently failing proxy, and it deletes the row. For a manually-maintained proxy chain (multi-proxy pool/rotation, #6365) that is too destructive just to exclude a temporarily-dead member. Add PROXY_AUTO_DISABLE as a sibling flag: at the same consecutive-failure threshold it soft-disables the proxy (status "dead") instead of removing it. "dead" is already one of the statuses the pool/rotation alive-filter excludes, so a disabled proxy drops out of the active chain immediately with no other code changes. The scheduler keeps probing dead proxies on its normal interval, and the existing recovery branch (previously autoRemove-only) re-activates it automatically once it starts answering again. decision.ts's decideProxyHealthAction() gets an optional `autoDisable` input (defaults to false, so existing callers are unaffected) and a "dead" status value; scheduler.ts wires the new PROXY_AUTO_DISABLE env flag through. If both flags are set, auto-remove wins. getProxyHealthStats() now also surfaces the registry `status` so operators can see when a proxy was auto-disabled, and ProxyStatusBadge now treats the full "not alive" status set (not just the literal string "inactive") as inactive in the dashboard. * test(proxy): assert registry status in getProxyHealthStats output The non-destructive auto-disable change added the live registry status to the stats object returned by getProxyHealthStats. Align the pre-existing db-proxies-crud assertion with the intended output shape. Co-authored-by: diegosouzapw <diegosouzapw@users.noreply.github.com> * fix(proxy): preserve auto-disabled status in dashboard edits Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> --------- Co-authored-by: adevwithpurpose <adevwithpurpose@users.noreply.github.com> Co-authored-by: Gi99lin <Gi99lin@users.noreply.github.com> Co-authored-by: diegosouzapw <diegosouzapw@users.noreply.github.com> Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
This commit is contained in:
Gi99lin
@@ -1894,6 +1894,13 @@ APP_LOG_TO_FILE=true
|
||||
# PROXY_AUTO_REMOVE=false
|
||||
# Consecutive failures before an auto-remove fires. Default: 3.
|
||||
# PROXY_AUTO_REMOVE_AFTER=3
|
||||
# Set "true" to let the scheduler auto-disable (status "dead") proxies after
|
||||
# repeated failures instead of deleting them. Non-destructive alternative to
|
||||
# PROXY_AUTO_REMOVE — the row stays in the registry, drops out of pool/rotation
|
||||
# resolution immediately, and is automatically re-activated once it starts
|
||||
# answering probes again. Shares the PROXY_AUTO_REMOVE_AFTER threshold above.
|
||||
# If both PROXY_AUTO_REMOVE and PROXY_AUTO_DISABLE are "true", auto-remove wins.
|
||||
# PROXY_AUTO_DISABLE=false
|
||||
# Let automated reachability probes (the scheduler + the "Test All" button) WRITE
|
||||
# a proxy's status. Default "false": probes are read-only and never deactivate a
|
||||
# proxy — only the operator sets active/inactive (a flaky probe must not strand an
|
||||
|
||||
@@ -817,6 +817,50 @@ The proxy is **not deleted** — it's marked unhealthy and won't be selected unt
|
||||
|
||||
---
|
||||
|
||||
## Automatic Failure Exclusion for Your Own Proxies
|
||||
|
||||
`failOneproxyProxy()` above only covers the 1proxy marketplace pool, which already
|
||||
auto-degrades on failure (see [Proxy Quality Scores](#proxy-quality-scores)). For
|
||||
proxies **you** added to the registry, the background health scheduler
|
||||
(`src/lib/proxyHealth/scheduler.ts`) provides the same "exclude a dead member from
|
||||
the chain automatically" behavior, without deleting anything:
|
||||
|
||||
```bash
|
||||
# .env — soft-disable a proxy after 3 consecutive failed probes, re-enable it
|
||||
# automatically once it starts answering probes again.
|
||||
PROXY_AUTO_DISABLE=true
|
||||
PROXY_AUTO_REMOVE_AFTER=3
|
||||
```
|
||||
|
||||
How it fits into a multi-proxy chain:
|
||||
|
||||
1. The scheduler probes every registered proxy every `PROXY_HEALTH_INTERVAL_MS`
|
||||
(default 10 min; minimum 1 min).
|
||||
2. After `PROXY_AUTO_REMOVE_AFTER` consecutive **conclusive** failures (a real
|
||||
connection failure — a timeout or the probe target's own 5xx never counts, see
|
||||
[Proxy Health Checking](#proxy-health-checking-v3816)), the proxy's `status` is
|
||||
set to `dead`.
|
||||
3. `dead` is one of the statuses the alive-status filter used by pool/rotation
|
||||
resolution excludes, so a scope's rotation (round-robin / random / sticky /
|
||||
latency — see [Rotation Strategy Decision Tree](#rotation-strategy-decision-tree))
|
||||
immediately stops handing that proxy to new requests. No other proxies in the
|
||||
pool are affected, and the whole pool never silently falls back to a direct
|
||||
connection — see the [4-Level Proxy System](#4-level-proxy-system) fail-closed
|
||||
guard.
|
||||
4. The scheduler keeps probing `dead` proxies on the same interval. The next
|
||||
successful probe flips `status` back to `active` and it re-enters rotation —
|
||||
no manual re-add required.
|
||||
|
||||
This is deliberately **opt-in and non-destructive**: by default the scheduler only
|
||||
counts and logs failures (see policy C in `decision.ts`), and `PROXY_AUTO_DISABLE`
|
||||
never deletes a row — that is what the separate, more aggressive
|
||||
`PROXY_AUTO_REMOVE` flag is for. If both are set to `true`, `PROXY_AUTO_REMOVE`
|
||||
wins (a proxy about to be deleted has no use for a soft-disable in between). See
|
||||
the [Environment Config](../reference/ENVIRONMENT.md) reference for the full
|
||||
variable list.
|
||||
|
||||
---
|
||||
|
||||
> 📖 **Related documentation:**
|
||||
>
|
||||
> - [User Guide](../guides/USER_GUIDE.md) — General setup and configuration
|
||||
|
||||
@@ -999,6 +999,7 @@ Anthropic-compatible provider instead.
|
||||
| `PROXY_HEALTH_AUTO_DEACTIVATE` | `false` | `src/lib/proxyHealth/statusPolicy.ts` | When `false` (default), automated reachability probes (the scheduler + the `/api/settings/proxies/auto-test` "Test All" button) are **read-only** and never write a proxy's status — only the operator sets active/inactive, so a flaky probe can't strand an assigned proxy (#6246). Set `true` to restore the legacy test-and-set behaviour. |
|
||||
| `PROXY_AUTO_REMOVE` | `false` | `src/lib/proxyHealth/scheduler.ts` | Set `true` to let the scheduler auto-remove proxies after repeated consecutive failures. |
|
||||
| `PROXY_AUTO_REMOVE_AFTER` | `3` | `src/lib/proxyHealth/scheduler.ts` | Consecutive failures before the scheduler auto-removes a proxy (when `PROXY_AUTO_REMOVE=true`). |
|
||||
| `PROXY_AUTO_DISABLE` | `false` | `src/lib/proxyHealth/scheduler.ts` | Set `true` to let the scheduler soft-disable (status `dead`, never deleted) a proxy after repeated consecutive failures, instead of removing it. Non-destructive alternative to `PROXY_AUTO_REMOVE`: the proxy drops out of pool/rotation resolution immediately (the alive-status filter used by scope-pool resolution already excludes it) and is automatically re-activated once it starts passing probes again. Shares the `PROXY_AUTO_REMOVE_AFTER` threshold. If both flags are `true`, `PROXY_AUTO_REMOVE` wins. |
|
||||
| `OMNIROUTE_CONTROL_PLANE_PROXY_DIRECT_FALLBACK` | `false` | `src/shared/constants/featureFlagDefinitions.ts` | Allow OAuth and provider validation flows to bypass a pinned proxy and connect directly when proxy reachability pre-checks fail. Effective precedence is Feature Flags DB override > env var > default. |
|
||||
| `RATE_LIMIT_MAX_WAIT_MS` | `15000` (15s) | `open-sse/services/rateLimitManager.ts` | Max time to wait on a 429 before failing the request. |
|
||||
| `RATE_LIMIT_MAX_QUEUE_DEPTH` | `0` (disabled) | `open-sse/services/rateLimitManager.ts` | Queue admission cap: reject with a 429 `queue_full` once this many requests are already queued. `0` = unbounded (default). |
|
||||
|
||||
@@ -1046,9 +1046,11 @@ import {
|
||||
className="w-full px-3 py-2 rounded bg-bg-subtle border border-border"
|
||||
value={form.status}
|
||||
onChange={(e) => setForm((prev) => ({ ...prev, status: e.target.value }))}
|
||||
data-testid="proxy-registry-status-select"
|
||||
>
|
||||
<option value="active">{t("statusActive")}</option>
|
||||
<option value="inactive">{t("statusInactive")}</option>
|
||||
{form.status === "dead" && <option value="dead">dead</option>}
|
||||
</select>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1281,7 +1283,11 @@ import {
|
||||
>
|
||||
<option value="">{t("poolSelectProxy")}</option>
|
||||
{items
|
||||
.filter((item) => !poolMembers.includes(item.id))
|
||||
.filter(
|
||||
(item) =>
|
||||
!poolMembers.includes(item.id) &&
|
||||
(item.status ?? "").toLowerCase() !== "dead"
|
||||
)
|
||||
.map((item) => (
|
||||
<option key={item.id} value={item.id}>
|
||||
{item.name} ({item.type}://{item.host}:{item.port})
|
||||
|
||||
@@ -6,9 +6,16 @@ interface ProxyStatusBadgeProps {
|
||||
status?: string;
|
||||
}
|
||||
|
||||
// Mirrors PROXY_ALIVE_PREDICATE (src/lib/db/proxies/guards.ts) — kept as a small
|
||||
// client-side duplicate rather than importing the server DB module into a "use
|
||||
// client" component (same pattern as RELAY_PROXY_TYPES in proxies/mappers.ts).
|
||||
// Any status in this set (including "dead", written by PROXY_AUTO_DISABLE) is
|
||||
// excluded from pool/rotation resolution, so it must not render as "Active".
|
||||
const NOT_ALIVE_STATUSES = new Set(["inactive", "error", "disabled", "dead", "down"]);
|
||||
|
||||
export function ProxyStatusBadge({ status }: ProxyStatusBadgeProps) {
|
||||
const t = useTranslations("settings");
|
||||
const isInactive = status === "inactive";
|
||||
const isInactive = NOT_ALIVE_STATUSES.has((status ?? "").toLowerCase());
|
||||
return (
|
||||
<span
|
||||
className={`inline-flex items-center gap-1.5 text-xs px-2 py-1 rounded border ${
|
||||
|
||||
@@ -35,7 +35,10 @@ export {
|
||||
} from "./proxies/guards";
|
||||
export { extractRelayAuth, redactProxySecrets } from "./proxies/mappers";
|
||||
export { addProxiesToScopePool } from "./proxySubscriptions";
|
||||
export { bumpProxyRegistryGeneration, getProxyRegistryGeneration } from "./proxies/registryGeneration";
|
||||
export {
|
||||
bumpProxyRegistryGeneration,
|
||||
getProxyRegistryGeneration,
|
||||
} from "./proxies/registryGeneration";
|
||||
import {
|
||||
normalizeRotationScopeId,
|
||||
clearRotationState,
|
||||
@@ -269,9 +272,7 @@ export async function listProxies(options?: {
|
||||
params.push(limit, offset);
|
||||
}
|
||||
const rows = db.prepare(sql).all(...params) as unknown[];
|
||||
const total = (
|
||||
db.prepare("SELECT count(*) as cnt FROM proxy_registry").get() as CountResult
|
||||
).cnt;
|
||||
const total = (db.prepare("SELECT count(*) as cnt FROM proxy_registry").get() as CountResult).cnt;
|
||||
const proxies = rows.map(mapProxyRow);
|
||||
return { items: includeSecrets ? proxies : proxies.map(redactProxySecrets), total };
|
||||
}
|
||||
@@ -685,7 +686,6 @@ export async function deleteProxyById(id: string, options?: { force?: boolean })
|
||||
return result.changes > 0;
|
||||
}
|
||||
|
||||
|
||||
export async function migrateLegacyProxyConfigToRegistry(options?: { force?: boolean }) {
|
||||
const force = options?.force === true;
|
||||
const db = getDbInstance();
|
||||
@@ -770,6 +770,7 @@ export async function getProxyHealthStats(options?: { hours?: number }) {
|
||||
p.type as proxy_type,
|
||||
p.host as proxy_host,
|
||||
p.port as proxy_port,
|
||||
p.status as proxy_status,
|
||||
COUNT(l.id) as total_requests,
|
||||
SUM(CASE WHEN l.status = 'success' THEN 1 ELSE 0 END) as success_count,
|
||||
SUM(CASE WHEN l.status = 'error' THEN 1 ELSE 0 END) as error_count,
|
||||
@@ -800,6 +801,7 @@ export async function getProxyHealthStats(options?: { hours?: number }) {
|
||||
type: String(row.proxy_type || "http"),
|
||||
host: String(row.proxy_host || ""),
|
||||
port: Number(row.proxy_port || 0),
|
||||
status: String(row.proxy_status || "active"),
|
||||
totalRequests: total,
|
||||
successCount: success,
|
||||
errorCount: error,
|
||||
|
||||
@@ -5,15 +5,28 @@
|
||||
* exhaustively without any I/O. The sweep classifies each probe into a tri-state
|
||||
* {@link ProxyProbeOutcome} and applies the returned {@link ProxyHealthDecision}.
|
||||
*
|
||||
* Policy (agreed for #6246):
|
||||
* Policy (agreed for #6246, extended for the auto-disable mode below):
|
||||
* A — downgrade only after `removeAfter` CONSECUTIVE conclusive failures.
|
||||
* B — an `inconclusive` probe (our own timeout/abort, or the probe TARGET
|
||||
* erroring) never penalizes: it neither counts nor changes status.
|
||||
* C — by DEFAULT (auto-remove off) the health check NEVER mutates a proxy's
|
||||
* status. It only counts failures for logging. A proxy is downgraded to
|
||||
* `inactive` (and removed) only when the operator opts in via
|
||||
* PROXY_AUTO_REMOVE=true. This mirrors how accounts are only auto-disabled
|
||||
* when the operator allows it — the operator owns their (often paid) proxies.
|
||||
* C — by DEFAULT (both auto-remove and auto-disable off) the health check
|
||||
* NEVER mutates a proxy's status. It only counts failures for logging.
|
||||
* A proxy's status is only touched once the operator opts in via
|
||||
* PROXY_AUTO_REMOVE=true or PROXY_AUTO_DISABLE=true. This mirrors how
|
||||
* accounts are only auto-disabled when the operator allows it — the
|
||||
* operator owns their (often paid) proxies.
|
||||
* D — PROXY_AUTO_DISABLE=true is the non-destructive sibling of
|
||||
* PROXY_AUTO_REMOVE: at the same consecutive-failure threshold it writes
|
||||
* `status: "dead"` instead of deleting the row. `"dead"` is already one
|
||||
* of the statuses PROXY_ALIVE_PREDICATE excludes (src/lib/db/proxies/guards.ts),
|
||||
* so a disabled proxy drops out of pool/rotation resolution immediately
|
||||
* with no other code changes. Because the sweep keeps probing every
|
||||
* registered proxy regardless of status, a "dead" proxy that starts
|
||||
* answering again is picked back up by the same `outcome === "ok"`
|
||||
* branch that already re-activates proxies for auto-remove — recovery
|
||||
* is free once autoDisable participates in `managesStatus` below. If
|
||||
* both flags are set, auto-remove (destructive) wins: a proxy that is
|
||||
* about to be deleted has no use for a soft-disable in between.
|
||||
*/
|
||||
|
||||
export type ProxyProbeOutcome = "ok" | "fail" | "inconclusive";
|
||||
@@ -23,8 +36,14 @@ export interface ProxyHealthDecisionInput {
|
||||
outcome: ProxyProbeOutcome;
|
||||
/** Consecutive failure count recorded BEFORE this probe. */
|
||||
priorFailures: number;
|
||||
/** PROXY_AUTO_REMOVE === "true" — operator opted into status management. */
|
||||
/** PROXY_AUTO_REMOVE === "true" — operator opted into delete-on-death. */
|
||||
autoRemove: boolean;
|
||||
/**
|
||||
* PROXY_AUTO_DISABLE === "true" — operator opted into soft-disable-on-death
|
||||
* (status "dead", never deleted). Optional/defaults to `false` so existing
|
||||
* callers that predate this flag keep their exact prior behavior.
|
||||
*/
|
||||
autoDisable?: boolean;
|
||||
/** Consecutive conclusive failures required before a downgrade/removal. */
|
||||
removeAfter: number;
|
||||
}
|
||||
@@ -35,14 +54,16 @@ export interface ProxyHealthDecision {
|
||||
/** Whether to drop this proxy from the consecutive-failure map. */
|
||||
clearFailures: boolean;
|
||||
/** Status to write, or `null` to leave the operator-controlled status untouched. */
|
||||
setStatus: "active" | "inactive" | null;
|
||||
setStatus: "active" | "inactive" | "dead" | null;
|
||||
/** Whether to auto-remove the proxy (only ever true when autoRemove is on). */
|
||||
remove: boolean;
|
||||
}
|
||||
|
||||
export function decideProxyHealthAction(input: ProxyHealthDecisionInput): ProxyHealthDecision {
|
||||
const { outcome, priorFailures, autoRemove, removeAfter } = input;
|
||||
const { outcome, priorFailures, autoRemove, autoDisable = false, removeAfter } = input;
|
||||
const threshold = Number.isFinite(removeAfter) && removeAfter > 0 ? removeAfter : 3;
|
||||
// Either opt-in flag hands status control from the operator to the sweep.
|
||||
const managesStatus = autoRemove || autoDisable;
|
||||
|
||||
// B: inconclusive probes are neutral — do not touch count or status.
|
||||
if (outcome === "inconclusive") {
|
||||
@@ -55,7 +76,7 @@ export function decideProxyHealthAction(input: ProxyHealthDecisionInput): ProxyH
|
||||
return {
|
||||
failures: 0,
|
||||
clearFailures: true,
|
||||
setStatus: autoRemove ? "active" : null,
|
||||
setStatus: managesStatus ? "active" : null,
|
||||
remove: false,
|
||||
};
|
||||
}
|
||||
@@ -64,13 +85,17 @@ export function decideProxyHealthAction(input: ProxyHealthDecisionInput): ProxyH
|
||||
const failures = priorFailures + 1;
|
||||
|
||||
// C: default mode only counts/logs — never downgrades.
|
||||
if (!autoRemove) {
|
||||
if (!managesStatus) {
|
||||
return { failures, clearFailures: false, setStatus: null, remove: false };
|
||||
}
|
||||
|
||||
// A: downgrade + remove only once the consecutive threshold is reached.
|
||||
// A/D: act only once the consecutive threshold is reached. Auto-remove
|
||||
// (destructive) takes precedence over auto-disable when both are enabled.
|
||||
if (failures >= threshold) {
|
||||
return { failures, clearFailures: false, setStatus: "inactive", remove: true };
|
||||
if (autoRemove) {
|
||||
return { failures, clearFailures: false, setStatus: "inactive", remove: true };
|
||||
}
|
||||
return { failures, clearFailures: false, setStatus: "dead", remove: false };
|
||||
}
|
||||
|
||||
return { failures, clearFailures: false, setStatus: null, remove: false };
|
||||
|
||||
@@ -2,22 +2,34 @@
|
||||
* Proxy Health Check Scheduler
|
||||
*
|
||||
* Periodically tests all proxy registry entries and automatically
|
||||
* removes proxies that have been failing consecutively.
|
||||
* removes (or soft-disables) proxies that have been failing consecutively.
|
||||
*
|
||||
* Config via environment:
|
||||
* PROXY_HEALTH_INTERVAL_MS — sweep interval (default: 600000 = 10min)
|
||||
* PROXY_HEALTH_ENABLED — set "false" to disable
|
||||
* PROXY_AUTO_REMOVE — set "true" to auto-remove dead proxies
|
||||
* PROXY_AUTO_REMOVE_AFTER — consecutive failures before removal (default: 3)
|
||||
* PROXY_AUTO_REMOVE — set "true" to auto-remove dead proxies (destructive)
|
||||
* PROXY_AUTO_DISABLE — set "true" to auto-disable dead proxies instead of
|
||||
* deleting them (status → "dead", already excluded from
|
||||
* pool/rotation resolution by PROXY_ALIVE_PREDICATE). The
|
||||
* row is never deleted, and the same recovery check that
|
||||
* re-activates proxies for PROXY_AUTO_REMOVE flips it back
|
||||
* to "active" once it starts answering probes again — no
|
||||
* manual re-add needed. If both flags are set, auto-remove
|
||||
* wins (see decision.ts).
|
||||
* PROXY_AUTO_REMOVE_AFTER — consecutive failures before the action above fires
|
||||
* (default: 3). Shared by both PROXY_AUTO_REMOVE and
|
||||
* PROXY_AUTO_DISABLE — they are alternative actions at the
|
||||
* same threshold, not independently tunable.
|
||||
*/
|
||||
|
||||
import { deleteProxyById, listProxies, updateProxy } from "@/lib/localDb";
|
||||
import { createProxyDispatcher, clearDispatcherCache, proxyConfigToUrl } from "@omniroute/open-sse/utils/proxyDispatcher";
|
||||
import { fetch as undiciFetch } from "undici";
|
||||
import {
|
||||
decideProxyHealthAction,
|
||||
type ProxyProbeOutcome,
|
||||
} from "./decision.ts";
|
||||
createProxyDispatcher,
|
||||
clearDispatcherCache,
|
||||
proxyConfigToUrl,
|
||||
} from "@omniroute/open-sse/utils/proxyDispatcher";
|
||||
import { fetch as undiciFetch } from "undici";
|
||||
import { decideProxyHealthAction, type ProxyProbeOutcome } from "./decision.ts";
|
||||
|
||||
// #6246: a HEAD to the public probe target through a legit (often loaded) proxy
|
||||
// can exceed a few seconds; the old 5s ceiling produced false negatives that
|
||||
@@ -58,6 +70,10 @@ function isAutoRemoveEnabled(): boolean {
|
||||
return process.env.PROXY_AUTO_REMOVE === "true";
|
||||
}
|
||||
|
||||
function isAutoDisableEnabled(): boolean {
|
||||
return process.env.PROXY_AUTO_DISABLE === "true";
|
||||
}
|
||||
|
||||
function getRemoveAfter(): number {
|
||||
const raw = parseInt(process.env.PROXY_AUTO_REMOVE_AFTER ?? "", 10);
|
||||
return Number.isFinite(raw) && raw > 0 ? raw : DEFAULT_REMOVE_AFTER;
|
||||
@@ -127,11 +143,13 @@ async function sweep(): Promise<void> {
|
||||
const failureMap = getFailureMap();
|
||||
const removeAfter = getRemoveAfter();
|
||||
const autoRemove = isAutoRemoveEnabled();
|
||||
const autoDisable = isAutoDisableEnabled();
|
||||
|
||||
let tested = 0;
|
||||
let alive = 0;
|
||||
let inconclusive = 0;
|
||||
let removed = 0;
|
||||
let disabled = 0;
|
||||
|
||||
for (let i = 0; i < proxies.length; i += CONCURRENCY) {
|
||||
const batch = proxies.slice(i, i + CONCURRENCY);
|
||||
@@ -153,31 +171,39 @@ async function sweep(): Promise<void> {
|
||||
outcome,
|
||||
priorFailures: failureMap.get(id) ?? 0,
|
||||
autoRemove,
|
||||
autoDisable,
|
||||
removeAfter,
|
||||
});
|
||||
|
||||
if (decision.clearFailures) failureMap.delete(id);
|
||||
else failureMap.set(id, decision.failures);
|
||||
|
||||
// #6246 (policy C): only mutate the operator-owned status when the decision
|
||||
// explicitly asks for it. By default (auto-remove off) setStatus is null, so
|
||||
// a transient probe failure never flips a healthy proxy to inactive.
|
||||
// #6246 (policy C) / auto-disable (policy D): only mutate the operator-owned
|
||||
// status when the decision explicitly asks for it. With both flags off,
|
||||
// setStatus is null, so a transient probe failure never flips a healthy
|
||||
// proxy's status.
|
||||
if (decision.setStatus) {
|
||||
await updateProxy(id, { status: decision.setStatus }).catch(() => {});
|
||||
if (decision.setStatus === "dead") disabled++;
|
||||
}
|
||||
|
||||
if (decision.remove) {
|
||||
if (await deleteProxyById(id, { force: true }).catch(() => false)) {
|
||||
failureMap.delete(id);
|
||||
removed++;
|
||||
try { clearDispatcherCache(); } catch { /* non-critical */ }
|
||||
try {
|
||||
clearDispatcherCache();
|
||||
} catch {
|
||||
/* non-critical */
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
console.log(
|
||||
`${LOG_PREFIX} Sweep complete: ${tested} tested, ${alive} alive, ${inconclusive} inconclusive, ${removed} auto-removed`
|
||||
`${LOG_PREFIX} Sweep complete: ${tested} tested, ${alive} alive, ${inconclusive} inconclusive, ` +
|
||||
`${removed} auto-removed, ${disabled} auto-disabled`
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
@@ -115,7 +115,7 @@ export const proxyRegistryFieldsSchema = z
|
||||
password: z.string().optional(),
|
||||
region: z.string().trim().max(64).nullable().optional(),
|
||||
notes: z.string().trim().max(1000).nullable().optional(),
|
||||
status: z.enum(["active", "inactive"]).optional().default("active"),
|
||||
status: z.enum(["active", "inactive", "dead"]).optional().default("active"),
|
||||
source: z
|
||||
.enum([
|
||||
"manual",
|
||||
|
||||
@@ -223,6 +223,7 @@ test("proxy health stats aggregate proxy_logs and force delete removes assignmen
|
||||
type: "http",
|
||||
host: "stats.local",
|
||||
port: 8080,
|
||||
status: "active",
|
||||
totalRequests: 3,
|
||||
successCount: 1,
|
||||
errorCount: 1,
|
||||
|
||||
102
tests/unit/proxy-health-auto-disable-decision.test.ts
Normal file
102
tests/unit/proxy-health-auto-disable-decision.test.ts
Normal file
@@ -0,0 +1,102 @@
|
||||
/**
|
||||
* PROXY_AUTO_DISABLE — non-destructive sibling of PROXY_AUTO_REMOVE (#6246 policy D).
|
||||
*
|
||||
* Mirrors tests/unit/proxy-health-decide-action-6246.test.ts. Where that suite
|
||||
* covers the existing autoRemove-only behavior (untouched here), this suite
|
||||
* covers the new `autoDisable` input: same consecutive-failure threshold, but
|
||||
* the action at threshold is a soft `status: "dead"` write instead of deletion,
|
||||
* and recovery re-activates exactly like autoRemove's does.
|
||||
*/
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
const { decideProxyHealthAction } = await import("../../src/lib/proxyHealth/decision.ts");
|
||||
|
||||
test("D: default (autoDisable off, autoRemove off) never mutates status on failure", () => {
|
||||
const d = decideProxyHealthAction({
|
||||
outcome: "fail",
|
||||
priorFailures: 0,
|
||||
autoRemove: false,
|
||||
autoDisable: false,
|
||||
removeAfter: 3,
|
||||
});
|
||||
assert.equal(d.setStatus, null);
|
||||
assert.equal(d.remove, false);
|
||||
assert.equal(d.failures, 1);
|
||||
});
|
||||
|
||||
test("D: with autoDisable on, does NOT downgrade before the consecutive threshold", () => {
|
||||
const d = decideProxyHealthAction({
|
||||
outcome: "fail",
|
||||
priorFailures: 1, // this probe makes it 2, threshold is 3
|
||||
autoRemove: false,
|
||||
autoDisable: true,
|
||||
removeAfter: 3,
|
||||
});
|
||||
assert.equal(d.setStatus, null, "2 < 3 failures must not flip dead");
|
||||
assert.equal(d.remove, false);
|
||||
assert.equal(d.failures, 2);
|
||||
});
|
||||
|
||||
test("D: with autoDisable on, soft-disables (status=dead) at the threshold — never removes", () => {
|
||||
const d = decideProxyHealthAction({
|
||||
outcome: "fail",
|
||||
priorFailures: 2, // this probe makes it 3 == threshold
|
||||
autoRemove: false,
|
||||
autoDisable: true,
|
||||
removeAfter: 3,
|
||||
});
|
||||
assert.equal(d.setStatus, "dead");
|
||||
assert.equal(d.remove, false, "auto-disable must never delete the proxy row");
|
||||
assert.equal(d.failures, 3);
|
||||
});
|
||||
|
||||
test("D: when both autoRemove and autoDisable are on, the destructive action wins", () => {
|
||||
const d = decideProxyHealthAction({
|
||||
outcome: "fail",
|
||||
priorFailures: 2,
|
||||
autoRemove: true,
|
||||
autoDisable: true,
|
||||
removeAfter: 3,
|
||||
});
|
||||
assert.equal(d.setStatus, "inactive");
|
||||
assert.equal(d.remove, true, "auto-remove takes precedence when both flags are set");
|
||||
});
|
||||
|
||||
test("D: ok probe re-activates a proxy that autoDisable manages, and resets the streak", () => {
|
||||
const d = decideProxyHealthAction({
|
||||
outcome: "ok",
|
||||
priorFailures: 5,
|
||||
autoRemove: false,
|
||||
autoDisable: true,
|
||||
removeAfter: 3,
|
||||
});
|
||||
assert.equal(d.clearFailures, true);
|
||||
assert.equal(d.setStatus, "active", "recovery flips a soft-disabled proxy back to active");
|
||||
assert.equal(d.failures, 0);
|
||||
});
|
||||
|
||||
test("D: inconclusive probes stay neutral under autoDisable, same as under autoRemove", () => {
|
||||
const d = decideProxyHealthAction({
|
||||
outcome: "inconclusive",
|
||||
priorFailures: 2,
|
||||
autoRemove: false,
|
||||
autoDisable: true,
|
||||
removeAfter: 3,
|
||||
});
|
||||
assert.equal(d.setStatus, null);
|
||||
assert.equal(d.remove, false);
|
||||
assert.equal(d.failures, 2);
|
||||
assert.equal(d.clearFailures, false);
|
||||
});
|
||||
|
||||
test("D: autoDisable defaults to false when omitted — behaves exactly like pre-existing callers", () => {
|
||||
const d = decideProxyHealthAction({
|
||||
outcome: "fail",
|
||||
priorFailures: 2,
|
||||
autoRemove: false,
|
||||
removeAfter: 3,
|
||||
});
|
||||
assert.equal(d.setStatus, null, "omitting autoDisable must not silently opt in");
|
||||
assert.equal(d.remove, false);
|
||||
});
|
||||
@@ -14,7 +14,9 @@ const proxiesDb = await import("../../src/lib/db/proxies.ts");
|
||||
const settingsDb = await import("../../src/lib/db/settings.ts");
|
||||
const apiKeysDb = await import("../../src/lib/db/apiKeys.ts");
|
||||
const proxiesRoute = await import("../../src/app/api/settings/proxies/route.ts");
|
||||
const { createProxyRegistrySchema } = await import("../../src/shared/validation/schemas.ts");
|
||||
const { createProxyRegistrySchema, updateProxyRegistrySchema } = await import(
|
||||
"../../src/shared/validation/schemas.ts"
|
||||
);
|
||||
|
||||
async function resetStorage() {
|
||||
delete process.env.INITIAL_PASSWORD;
|
||||
@@ -523,6 +525,15 @@ test("createProxyRegistrySchema accepts type:vercel and source:vercel-relay (sch
|
||||
}
|
||||
});
|
||||
|
||||
test("updateProxyRegistrySchema accepts dead status for dashboard round-trip", () => {
|
||||
const result = updateProxyRegistrySchema.safeParse({
|
||||
id: "dead-proxy",
|
||||
status: "dead",
|
||||
});
|
||||
|
||||
assert.equal(result.success, true, "auto-disabled proxies must remain editable");
|
||||
});
|
||||
|
||||
test("createProxy persists type:vercel and source:vercel-relay to DB (schema gap-06)", async () => {
|
||||
await resetStorage();
|
||||
|
||||
|
||||
@@ -21,9 +21,18 @@ const SEEDED_PROXY = {
|
||||
family: "auto",
|
||||
};
|
||||
|
||||
const DEAD_PROXY = {
|
||||
...SEEDED_PROXY,
|
||||
id: "proxy-dead-10342",
|
||||
name: "Auto-disabled proxy",
|
||||
status: "dead",
|
||||
};
|
||||
|
||||
let root: Root;
|
||||
let container: HTMLDivElement;
|
||||
let postBody: Record<string, unknown> | undefined;
|
||||
let patchBody: Record<string, unknown> | undefined;
|
||||
let responseItems: Array<typeof SEEDED_PROXY>;
|
||||
|
||||
function jsonResponse(body: unknown): Response {
|
||||
return { ok: true, json: async () => body } as Response;
|
||||
@@ -83,6 +92,8 @@ beforeEach(() => {
|
||||
globalThis as typeof globalThis & { IS_REACT_ACT_ENVIRONMENT?: boolean }
|
||||
).IS_REACT_ACT_ENVIRONMENT = true;
|
||||
postBody = undefined;
|
||||
patchBody = undefined;
|
||||
responseItems = [SEEDED_PROXY];
|
||||
container = document.createElement("div");
|
||||
document.body.appendChild(container);
|
||||
root = createRoot(container);
|
||||
@@ -95,8 +106,15 @@ beforeEach(() => {
|
||||
postBody = JSON.parse(String(init.body));
|
||||
return jsonResponse({ item: { ...SEEDED_PROXY, ...postBody } });
|
||||
}
|
||||
if (url === "/api/settings/proxies" && init?.method === "PATCH") {
|
||||
patchBody = JSON.parse(String(init.body));
|
||||
return jsonResponse({ item: { ...responseItems[0], ...patchBody } });
|
||||
}
|
||||
if (url === "/api/settings/proxies") {
|
||||
return jsonResponse({ items: [SEEDED_PROXY] });
|
||||
return jsonResponse({ items: responseItems });
|
||||
}
|
||||
if (url.startsWith("/api/settings/proxies/pool?")) {
|
||||
return jsonResponse({ members: [], strategy: "round-robin" });
|
||||
}
|
||||
if (url.startsWith("/api/settings/proxies/health")) {
|
||||
return jsonResponse({ items: [] });
|
||||
@@ -167,4 +185,54 @@ describe("ProxyRegistryManager credential autofill regression #8855", () => {
|
||||
expect(postBody?.username).not.toBe("edit-user-sentinel");
|
||||
expect(postBody?.password).not.toBe("edit-password-sentinel");
|
||||
});
|
||||
|
||||
it("round-trips dead status through Edit and excludes it from pool candidates", async () => {
|
||||
responseItems = [DEAD_PROXY];
|
||||
|
||||
const { default: ProxyRegistryManager } =
|
||||
await import("@/app/(dashboard)/dashboard/settings/components/ProxyRegistryManager");
|
||||
|
||||
await act(async () => {
|
||||
root.render(<ProxyRegistryManager />);
|
||||
});
|
||||
await waitFor(() => expect(container.textContent).toContain(DEAD_PROXY.name));
|
||||
|
||||
await click(findButton("edit"));
|
||||
const statusSelect = container.querySelector<HTMLSelectElement>(
|
||||
'[data-testid="proxy-registry-status-select"]'
|
||||
);
|
||||
expect(statusSelect).not.toBeNull();
|
||||
expect(statusSelect?.value).toBe("dead");
|
||||
expect(statusSelect?.querySelector('option[value="dead"]')).not.toBeNull();
|
||||
|
||||
await click(findButton("save"));
|
||||
await waitFor(() => expect(patchBody).toBeDefined());
|
||||
expect(patchBody).toMatchObject({ id: DEAD_PROXY.id, status: "dead" });
|
||||
|
||||
await click(findButton("managePool"));
|
||||
const scopeSelect = container.querySelector<HTMLSelectElement>(
|
||||
'[data-testid="proxy-registry-pool-scope"]'
|
||||
);
|
||||
expect(scopeSelect).not.toBeNull();
|
||||
const setter = Object.getOwnPropertyDescriptor(
|
||||
window.HTMLSelectElement.prototype,
|
||||
"value"
|
||||
)?.set;
|
||||
if (!setter || !scopeSelect) throw new Error("Pool scope select is unavailable");
|
||||
act(() => {
|
||||
setter.call(scopeSelect, "global");
|
||||
scopeSelect.dispatchEvent(new Event("change", { bubbles: true }));
|
||||
});
|
||||
await click(container.querySelector<HTMLButtonElement>(
|
||||
'[data-testid="proxy-registry-pool-load"]'
|
||||
)!);
|
||||
await waitFor(() =>
|
||||
expect(container.querySelector('[data-testid="proxy-registry-pool-add-select"]')).not.toBeNull()
|
||||
);
|
||||
|
||||
const poolAddSelect = container.querySelector<HTMLSelectElement>(
|
||||
'[data-testid="proxy-registry-pool-add-select"]'
|
||||
);
|
||||
expect(poolAddSelect?.querySelector(`option[value="${DEAD_PROXY.id}"]`)).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user