mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-08-21 22:52:19 +03:00
Compare commits
5 Commits
fix/sec-ad
...
release/v3
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3caa59107e | ||
|
|
9469b9c79e | ||
|
|
4ac157b76d | ||
|
|
8f0d0a0d03 | ||
|
|
00dfdadf93 |
12
CHANGELOG.md
12
CHANGELOG.md
@@ -2,6 +2,18 @@
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### ✨ New Features
|
||||
|
||||
- **feat(sse): STRICT_ZERO_COST** — opt-in, off-by-default `freeAccessPolicy: "strict"` setting
|
||||
that hard-verifies every auto-combo candidate against live quota state and per-connection
|
||||
economic safety before it can be dispatched, going beyond `hidePaidModels`'s static catalog
|
||||
check. Adds curated `hardStopGuaranteed` metadata to `FREE_MODEL_BUDGETS`, a short-TTL quota
|
||||
cache reusing `getUsageForProvider()`, and a connection-safety guarantee: a candidate backed
|
||||
by multiple accounts has its `allowedConnectionIds` narrowed to exactly the connections
|
||||
independently verified `SAFE`, so dispatch can never use an unverified account. An
|
||||
`excludeTosAvoid` guard (default `false`) is available separately for contractual risk. See
|
||||
`docs/routing/STRICT_ZERO_COST.md`.
|
||||
|
||||
---
|
||||
|
||||
## [3.8.50] — TBD
|
||||
|
||||
1
changelog.d/fixes/command-code-effort-capabilities.md
Normal file
1
changelog.d/fixes/command-code-effort-capabilities.md
Normal file
@@ -0,0 +1 @@
|
||||
- fix(combo): resolve effort-suffixed command-code variants (e.g. `deepseek-v4-flash-max`) to their base model for capability lookups, so tool-bearing combo requests keep the declared priority order instead of reordering behind models with confirmed capabilities
|
||||
1
changelog.d/fixes/opencode-merge-provider-guard.md
Normal file
1
changelog.d/fixes/opencode-merge-provider-guard.md
Normal file
@@ -0,0 +1 @@
|
||||
- **OpenCode config merge:** stop `mergeOpenCodeConfig` splaying a malformed `provider` block into index keys. The root was already guarded against a non-object; the `provider` branch it spreads one level down was not, so an existing `"provider": ["a", "b"]` merged to `{"0": "a", "1": "b", …}`. Its sibling `mergeOpenCodeConfigText` already refuses the same input.
|
||||
146
docs/routing/STRICT_ZERO_COST.md
Normal file
146
docs/routing/STRICT_ZERO_COST.md
Normal file
@@ -0,0 +1,146 @@
|
||||
---
|
||||
title: "STRICT_ZERO_COST"
|
||||
version: 3.8.50
|
||||
lastUpdated: 2026-08-20
|
||||
---
|
||||
|
||||
# STRICT_ZERO_COST
|
||||
|
||||
> Opt-in, off by default (`settings.freeAccessPolicy !== "strict"` leaves every `auto/*`
|
||||
> candidate pool byte-identical). A stricter sibling of `hidePaidModels`
|
||||
> (`open-sse/services/autoCombo/paidModelFilter.ts`, #6512) for operators who need a hard
|
||||
> guarantee against ANY incremental monetary spend, not just "documented as free".
|
||||
|
||||
## Why this exists, and why `hidePaidModels` alone isn't enough
|
||||
|
||||
`hidePaidModels` answers "is this model classified free in `FREE_MODEL_BUDGETS` right now?" —
|
||||
a point-in-time catalog fact, checked via `isFreeModel()`/`providerHasFreeModels()`
|
||||
(`src/shared/utils/freeModels.ts`). It says nothing about two real risks:
|
||||
|
||||
1. A `recurring-*`/`one-time-initial` free tier's allowance can be **exhausted** — the catalog
|
||||
still lists the model as free, but the account behind it has no headroom left.
|
||||
2. Exceeding a free tier is not always a hard stop. Some providers document explicitly that no
|
||||
payment method can ever be attached ("no credit card required"); others don't say, and a
|
||||
handful bill automatically past the free allowance.
|
||||
|
||||
`hidePaidModels` cannot distinguish these — it was never meant to. STRICT_ZERO_COST adds exactly
|
||||
these two checks, evaluated per candidate, **before** category/tier ranking and **before**
|
||||
dispatch — never after a request has already gone out.
|
||||
|
||||
## Candidate classification
|
||||
|
||||
For every candidate in the pool (`open-sse/services/autoCombo/virtualFactory.ts::buildPreparedPool`,
|
||||
right after `filterPaidOnlyCandidates`):
|
||||
|
||||
1. **Not in `FREE_MODEL_BUDGETS` at all** → excluded. This covers genuinely paid models and any
|
||||
provider/model OmniRoute hasn't classified yet — new candidates start excluded, not included.
|
||||
2. **`freeType: "keyless"`** → passes immediately, **but only for a candidate that genuinely
|
||||
arrived via the no-auth path** (`connectionId === SYNTHETIC_NOAUTH_CONNECTION_ID`,
|
||||
`open-sse/services/autoCombo/resilienceCandidateFilter.ts`). No credential exists for that
|
||||
candidate, so no request against it can ever be billed — no runtime check is needed or
|
||||
possible. The same catalogued `keyless` provider/model reached through a **real** DB
|
||||
connection (`connectionId` is an actual connection id, or the candidate carries
|
||||
`allowedConnectionIds`) does **not** get this shortcut — `keyless` metadata describes the
|
||||
no-auth path specifically, not the provider in general, and never authorizes a real,
|
||||
credentialed account. Such a candidate falls through to check 3 like any other, where it is
|
||||
excluded unless the catalog entry separately carries `hardStopGuaranteed: true` (real
|
||||
`keyless` entries never do — the shortcut was their only path to safety).
|
||||
3. **Any other `freeType`** (`recurring-daily`, `recurring-monthly`, `recurring-credit`,
|
||||
`recurring-uncapped`, `one-time-initial`, and any future type this module doesn't
|
||||
special-case) → passes only if **all** of the following hold:
|
||||
- `hardStopGuaranteed: true` is set on the catalog entry (`FreeModelBudget.hardStopGuaranteed`,
|
||||
`open-sse/config/freeModelCatalog.ts`) — a **curated, hand-set fact** about the provider's
|
||||
own published terms (e.g. an explicit "no credit card required" claim), never derived from
|
||||
`freeType` or from a live API response. Unset (`undefined`) and `false` are both treated as
|
||||
"not guaranteed".
|
||||
- A usage adapter exists for the provider in `USAGE_FETCHER_PROVIDERS`
|
||||
(`open-sse/services/usage.ts`) — the same registry that already backs the quota dashboard and
|
||||
`getUsageForProvider()`. No adapter → excluded, permanently, until one is added.
|
||||
- The live, cached `FreeAccessState` for **the specific connection actually being
|
||||
evaluated** is `status: "SAFE"`, was checked within
|
||||
`settings.autoRefreshProviderQuotaInterval` (default 180s — the existing setting, not a new
|
||||
number), and reports `remainingFreeAllowance` above a small safety margin.
|
||||
4. **`freeType: "discontinued"`** → always excluded.
|
||||
|
||||
## Connection safety (per-connection verification, never per-candidate)
|
||||
|
||||
A candidate in the auto-combo pool is not always tied to one connection. A "logical" candidate
|
||||
(`connectionId: null`) carries an `allowedConnectionIds` allowlist — one or more actual
|
||||
provider connections/accounts any of which could serve the request — and the account actually
|
||||
used is decided later, at dispatch time, by `open-sse/services/combo/autoStrategy.ts`
|
||||
(intersecting `allowedConnectionIds` against its own connection-selection logic, ~line 315-331).
|
||||
|
||||
STRICT_ZERO_COST verifies the free-access state of **each connection in that allowlist
|
||||
individually** (`evaluateCandidateConnections()` in `strictZeroCostFilter.ts`) and rewrites
|
||||
`allowedConnectionIds` down to exactly the subset that came back `SAFE` — never the full
|
||||
original list, and never a single arbitrarily-chosen member. Concretely:
|
||||
|
||||
- Account A `SAFE`, account B `UNKNOWN`/exhausted/billable → only A remains selectable.
|
||||
- All accounts `UNKNOWN` → the candidate is dropped entirely (empty safe set).
|
||||
- A single-connection candidate (`connectionId` set directly, no allowlist) that fails is
|
||||
dropped outright, never returned with an empty `allowedConnectionIds`.
|
||||
|
||||
Because `autoStrategy.ts` already enforces `allowedConnectionIds` as a hard allowlist before
|
||||
selecting a connection to dispatch to, rewriting it to the verified-SAFE subset is sufficient to
|
||||
guarantee the connection actually used at dispatch is always one this filter itself verified —
|
||||
never a different, unverified account on the same candidate. See
|
||||
`tests/unit/autoCombo/strict-zero-cost-connection-safety.test.ts` for the regression proof
|
||||
(keyless-bypass cases A/B/C, multi-account cases 1-5).
|
||||
|
||||
`discovered automatically`: a provider/model shipped tomorrow with the right metadata (in the
|
||||
catalog, with a usage adapter, `hardStopGuaranteed: true`) is usable the moment OmniRoute knows
|
||||
about it — no code change, no whitelist entry, nothing to edit in this module. One removed from
|
||||
the catalog disappears the same way. See
|
||||
`tests/unit/autoCombo/strict-zero-cost-autodiscovery.test.ts` for the regression proof (via
|
||||
injectable fixtures, not by mutating the real catalog).
|
||||
|
||||
## Quota caching (`open-sse/services/autoCombo/freeAccessQuota.ts`)
|
||||
|
||||
Reuses `getUsageForProvider()` — no second quota system. A short, in-memory,
|
||||
process-lifetime cache sits in front of it (TTL equal to the default
|
||||
`autoRefreshProviderQuotaInterval`) so a Telegram-scale request rate never triggers a live
|
||||
billing-API call per candidate per request. Reads are synchronous: a cache miss returns
|
||||
`undefined` (→ excluded, fail-closed) and kicks off a background refresh for the _next_ read —
|
||||
nothing in the candidate-pool build path ever awaits a network call.
|
||||
|
||||
`invalidateFreeAccessState(provider, connectionId)` is called from
|
||||
`src/sse/services/auth.ts::markAccountUnavailable()` the moment a connection fails for any
|
||||
reason, so the very next pool build reads a clean cache miss instead of a stale `SAFE` entry —
|
||||
no waiting out the TTL after a 402/403/quota-exhausted response.
|
||||
|
||||
## ToS guard (independent of economic safety)
|
||||
|
||||
`excludeTosAvoid` (default `false`) drops any candidate whose curated `tos` verdict
|
||||
(`FreeModelBudget.tos`) is `"avoid"` — reuses the same field `hidePaidModels`'s sibling docs
|
||||
(`docs/reference/FREE_TIERS.md`) already populate. Deliberately separate from
|
||||
`freeAccessPolicy`: a candidate can be economically `SAFE` and still excluded here for
|
||||
contractual reasons, or left in when this guard is off even with `freeAccessPolicy: "strict"` on.
|
||||
|
||||
## What passes today
|
||||
|
||||
Run `npx tsx scripts/ad-hoc/dry-run-strict-zero-cost.ts` against a live instance's
|
||||
`GET /v1/auto-combo/{channel}/candidates` output for a real before/after — the script now reads
|
||||
each candidate's real `connectionId`, so it also proves the connection-safety fix live, not just
|
||||
in unit tests. As of 2026-08-20, only `freeType: "keyless"` candidates pass in practice (7 of 29
|
||||
live candidates on this instance: `opencode/big-pickle`, `opencode/deepseek-v4-flash-free`, and
|
||||
5 `felo-web` models — all confirmed arriving with the genuine no-auth `connectionId`, never a
|
||||
real connection) — no currently-catalogued `recurring-*` provider both has a usage adapter
|
||||
registered in `USAGE_FETCHER_PROVIDERS` **and** `hardStopGuaranteed: true` declared (e.g. `groq`
|
||||
has neither the adapter registered here nor is fetched offline in this dry run; `kiro` lacks
|
||||
`hardStopGuaranteed`). This is not a bug: it's the honest state of two independently-curated
|
||||
metadata sets that happen not to overlap yet, not a limitation of the filter itself.
|
||||
|
||||
With `excludeTosAvoid: true` added on top of the same live pool, the count drops from 7 to 0 —
|
||||
every one of the 7 surviving candidates is curated `tos: "avoid"` today (`felo-web`, `opencode`).
|
||||
This is a real, expected trade-off of turning the ToS guard on, not a bug: the guard is
|
||||
`false` by default for exactly this reason (see "ToS guard" above).
|
||||
|
||||
## Enabling
|
||||
|
||||
```json
|
||||
PUT /api/settings
|
||||
{ "freeAccessPolicy": "strict", "excludeTosAvoid": false }
|
||||
```
|
||||
|
||||
Both new settings default to their pre-feature values (`"off"` / `false`) — enabling neither
|
||||
changes any existing `auto/*` routing behavior.
|
||||
@@ -108,8 +108,9 @@ export const FREE_MODEL_BUDGETS: FreeModelBudget[] = [
|
||||
{ provider: "bytez", modelId: "meta-llama/Llama-3.3-70B-Instruct", displayName: "meta-llama/Llama-3.3-70B-Instruct", monthlyTokens: 0, creditTokens: 1000000, freeType: "recurring-credit", poolKey: "bytez", tos: "ambiguous" },
|
||||
{ provider: "bytez", modelId: "mistralai/Mistral-7B-Instruct-v0.3", displayName: "mistralai/Mistral-7B-Instruct-v0.3", monthlyTokens: 0, creditTokens: 1000000, freeType: "recurring-credit", poolKey: "bytez", tos: "ambiguous" },
|
||||
{ provider: "bytez", modelId: "Qwen/Qwen2.5-72B-Instruct", displayName: "Qwen/Qwen2.5-72B-Instruct", monthlyTokens: 0, creditTokens: 1000000, freeType: "recurring-credit", poolKey: "bytez", tos: "ambiguous" },
|
||||
{ provider: "cerebras", modelId: "zai-glm-4.7", displayName: "GLM 4.7", monthlyTokens: 30000000, creditTokens: 0, freeType: "recurring-daily", poolKey: "cerebras", tos: "caution" },
|
||||
{ provider: "cerebras", modelId: "gpt-oss-120b", displayName: "GPT OSS 120B", monthlyTokens: 30000000, creditTokens: 0, freeType: "recurring-daily", poolKey: "cerebras", tos: "caution" },
|
||||
// hardStopGuaranteed: Cerebras pricing page states "Free Trial: 1M tokens/day... no credit card" (open-sse/services/../providers/apikey/inference-hosts.ts:74-84).
|
||||
{ provider: "cerebras", modelId: "zai-glm-4.7", displayName: "GLM 4.7", monthlyTokens: 30000000, creditTokens: 0, freeType: "recurring-daily", poolKey: "cerebras", tos: "caution", hardStopGuaranteed: true },
|
||||
{ provider: "cerebras", modelId: "gpt-oss-120b", displayName: "GPT OSS 120B", monthlyTokens: 30000000, creditTokens: 0, freeType: "recurring-daily", poolKey: "cerebras", tos: "caution", hardStopGuaranteed: true },
|
||||
// #8717: drop dead Workers AI ids (400/403/410). Keep Neurons/day budget on fp8-fast.
|
||||
{ provider: "cloudflare-ai", modelId: "@cf/mistral/mistral-7b-instruct-v0.2-lora", displayName: "Mistral 7B (🆓)", monthlyTokens: 0, creditTokens: 0, freeType: "recurring-daily", poolKey: "cloudflare-ai", tos: "caution" },
|
||||
{ provider: "cloudflare-ai", modelId: "@cf/qwen/qwen2.5-coder-32b-instruct", displayName: "Qwen 2.5 Coder 32B (🆓)", monthlyTokens: 0, creditTokens: 0, freeType: "recurring-daily", poolKey: "cloudflare-ai", tos: "caution" },
|
||||
@@ -187,11 +188,12 @@ export const FREE_MODEL_BUDGETS: FreeModelBudget[] = [
|
||||
{ provider: "glm-cn", modelId: "glm-4.5-flash", displayName: "GLM-4.5-Flash", monthlyTokens: 0, creditTokens: 0, freeType: "recurring-uncapped", poolKey: "zhipu-flash-free", tos: "ok" },
|
||||
{ provider: "glm-cn", modelId: "glm-4.7-flash", displayName: "GLM-4.7-Flash", monthlyTokens: 0, creditTokens: 0, freeType: "recurring-uncapped", poolKey: "zhipu-flash-free", tos: "ok" },
|
||||
{ provider: "glm-cn", modelId: "glm-signup-bonus", displayName: "Z.AI — 20M signup bonus", monthlyTokens: 0, creditTokens: 20000000, freeType: "one-time-initial", poolKey: "zhipu-signup", tos: "ok" },
|
||||
{ provider: "groq", modelId: "meta-llama/llama-4-scout-17b-16e-instruct", displayName: "Llama 4 Scout", monthlyTokens: 15000000, creditTokens: 0, freeType: "recurring-daily", poolKey: "groq", tos: "caution" },
|
||||
{ provider: "groq", modelId: "llama-3.3-70b-versatile", displayName: "Llama 3.3 70B", monthlyTokens: 15000000, creditTokens: 0, freeType: "recurring-daily", poolKey: "groq", tos: "caution" },
|
||||
{ provider: "groq", modelId: "openai/gpt-oss-120b", displayName: "GPT-OSS 120B", monthlyTokens: 15000000, creditTokens: 0, freeType: "recurring-daily", poolKey: "groq", tos: "caution" },
|
||||
{ provider: "groq", modelId: "openai/gpt-oss-20b", displayName: "GPT-OSS 20B", monthlyTokens: 15000000, creditTokens: 0, freeType: "recurring-daily", poolKey: "groq", tos: "caution" },
|
||||
{ provider: "groq", modelId: "qwen/qwen3-32b", displayName: "Qwen3 32B", monthlyTokens: 15000000, creditTokens: 0, freeType: "recurring-daily", poolKey: "groq", tos: "caution" },
|
||||
// hardStopGuaranteed: Groq pricing page states "Free tier: 30 RPM / 14.4K RPD — no credit card" (open-sse/services/../providers/apikey/frontier-labs.ts:71-81).
|
||||
{ provider: "groq", modelId: "meta-llama/llama-4-scout-17b-16e-instruct", displayName: "Llama 4 Scout", monthlyTokens: 15000000, creditTokens: 0, freeType: "recurring-daily", poolKey: "groq", tos: "caution", hardStopGuaranteed: true },
|
||||
{ provider: "groq", modelId: "llama-3.3-70b-versatile", displayName: "Llama 3.3 70B", monthlyTokens: 15000000, creditTokens: 0, freeType: "recurring-daily", poolKey: "groq", tos: "caution", hardStopGuaranteed: true },
|
||||
{ provider: "groq", modelId: "openai/gpt-oss-120b", displayName: "GPT-OSS 120B", monthlyTokens: 15000000, creditTokens: 0, freeType: "recurring-daily", poolKey: "groq", tos: "caution", hardStopGuaranteed: true },
|
||||
{ provider: "groq", modelId: "openai/gpt-oss-20b", displayName: "GPT-OSS 20B", monthlyTokens: 15000000, creditTokens: 0, freeType: "recurring-daily", poolKey: "groq", tos: "caution", hardStopGuaranteed: true },
|
||||
{ provider: "groq", modelId: "qwen/qwen3-32b", displayName: "Qwen3 32B", monthlyTokens: 15000000, creditTokens: 0, freeType: "recurring-daily", poolKey: "groq", tos: "caution", hardStopGuaranteed: true },
|
||||
{ provider: "hackclub", modelId: "meta-llama/llama-3.3-70b-instruct", displayName: "Llama 3.3 70B", monthlyTokens: 0, creditTokens: 0, freeType: "keyless", poolKey: "hackclub", tos: "caution" },
|
||||
{ provider: "hackclub", modelId: "mistralai/mistral-7b-instruct", displayName: "Mistral 7B", monthlyTokens: 0, creditTokens: 0, freeType: "keyless", poolKey: "hackclub", tos: "caution" },
|
||||
{ provider: "hackclub", modelId: "deepseek-ai/deepseek-coder-33b", displayName: "DeepSeek Coder 33B", monthlyTokens: 0, creditTokens: 0, freeType: "keyless", poolKey: "hackclub", tos: "caution" },
|
||||
|
||||
@@ -26,6 +26,20 @@ export interface FreeModelBudget {
|
||||
* reports this per model as `mayTrainOnYourPrompts` on its public catalog.
|
||||
*/
|
||||
trainsOnPrompts?: boolean;
|
||||
/**
|
||||
* True only when the provider's own published terms document that exceeding
|
||||
* the free allowance is a hard stop (request refused / rate-limited) and NOT
|
||||
* automatic pay-as-you-go billing — e.g. an explicit "no credit card
|
||||
* required" claim on the provider's pricing page. This is a curated fact
|
||||
* about the upstream provider, not something derivable from `freeType` or
|
||||
* from any live API response, so it must be set by hand per entry with the
|
||||
* source of the claim in a comment. Leave unset (undefined) whenever this
|
||||
* isn't independently documented — `undefined` and `false` are both treated
|
||||
* as "not guaranteed" by `strictZeroCostFilter.ts`; never default to `true`
|
||||
* to grow the catalog. See STRICT_ZERO_COST in
|
||||
* `open-sse/services/autoCombo/strictZeroCostFilter.ts`.
|
||||
*/
|
||||
hardStopGuaranteed?: boolean;
|
||||
}
|
||||
|
||||
export interface FreeModelTotals {
|
||||
@@ -80,7 +94,7 @@ function fmt(n: number): string {
|
||||
function dedupedSum(
|
||||
models: FreeModelBudget[],
|
||||
pick: (m: FreeModelBudget) => number,
|
||||
include: (m: FreeModelBudget) => boolean,
|
||||
include: (m: FreeModelBudget) => boolean
|
||||
): number {
|
||||
const poolMax = new Map<string, number>();
|
||||
let loose = 0;
|
||||
@@ -100,30 +114,30 @@ export function computeFreeModelTotals(opts: { excludeTosAvoid?: boolean } = {})
|
||||
const steadyRecurringTokens = dedupedSum(
|
||||
models,
|
||||
(m) => m.monthlyTokens,
|
||||
(m) => RECURRING.has(m.freeType),
|
||||
(m) => RECURRING.has(m.freeType)
|
||||
);
|
||||
const recurringCredits = dedupedSum(
|
||||
models,
|
||||
(m) => m.creditTokens,
|
||||
(m) => m.freeType === "recurring-credit",
|
||||
(m) => m.freeType === "recurring-credit"
|
||||
);
|
||||
const oneTimeCredits = dedupedSum(
|
||||
models,
|
||||
(m) => m.creditTokens,
|
||||
(m) => m.freeType === "one-time-initial",
|
||||
(m) => m.freeType === "one-time-initial"
|
||||
);
|
||||
|
||||
const steadyWithRecurringCreditsTokens = steadyRecurringTokens + recurringCredits;
|
||||
const firstMonthRealisticTokens = steadyWithRecurringCreditsTokens + oneTimeCredits;
|
||||
|
||||
const poolCount = new Set(
|
||||
models.filter((m) => RECURRING.has(m.freeType) && m.poolKey).map((m) => m.poolKey),
|
||||
models.filter((m) => RECURRING.has(m.freeType) && m.poolKey).map((m) => m.poolKey)
|
||||
).size;
|
||||
|
||||
// Deposit-unlock boost: sum the FREE_TIER_BOOSTS whose pool still has a live
|
||||
// recurring model in the (optionally ToS-filtered) set.
|
||||
const livePools = new Set(
|
||||
models.filter((m) => RECURRING.has(m.freeType) && m.poolKey).map((m) => m.poolKey),
|
||||
models.filter((m) => RECURRING.has(m.freeType) && m.poolKey).map((m) => m.poolKey)
|
||||
);
|
||||
const boostMonthlyTokens = Object.entries(FREE_TIER_BOOSTS)
|
||||
.filter(([pool]) => livePools.has(pool))
|
||||
|
||||
@@ -56,12 +56,21 @@ export function stripGroqUnsupportedFields<T extends Record<string, unknown>>(bo
|
||||
delete next.top_logprobs;
|
||||
if (Array.isArray(next.messages)) {
|
||||
next.messages = next.messages.map((m) => {
|
||||
if (m && typeof m === "object" && "name" in m) {
|
||||
const { name: _name, ...rest } = m as Record<string, unknown>;
|
||||
if (m && typeof m === "object") {
|
||||
const {
|
||||
name: _name,
|
||||
model: _model,
|
||||
messageId: _msgId,
|
||||
sender: _sender,
|
||||
...rest
|
||||
} = m as Record<string, unknown>;
|
||||
|
||||
return rest;
|
||||
}
|
||||
|
||||
return m;
|
||||
});
|
||||
}
|
||||
return next as T;
|
||||
}
|
||||
|
||||
|
||||
209
open-sse/services/autoCombo/freeAccessQuota.ts
Normal file
209
open-sse/services/autoCombo/freeAccessQuota.ts
Normal file
@@ -0,0 +1,209 @@
|
||||
/**
|
||||
* Live wiring for STRICT_ZERO_COST's quota-based branch.
|
||||
*
|
||||
* Reuses the existing `getUsageForProvider()` (`open-sse/services/usage.ts`)
|
||||
* instead of building a second quota system — this module only adds a short
|
||||
* TTL cache in front of it (so a Telegram-scale request rate never triggers a
|
||||
* live billing-API call per candidate per request) and an invalidation hook
|
||||
* for the resilience layer to call the moment a 402/403/quota-exhausted
|
||||
* response is observed (`accountFallback.ts`).
|
||||
*
|
||||
* The cache is intentionally synchronous to read: `resolveFreeAccessState()`
|
||||
* never awaits. A cache miss returns `undefined` (→ UNKNOWN → excluded,
|
||||
* fail-closed) and kicks off a background refresh for the *next* read —
|
||||
* nothing here can make `strictZeroCostFilter.ts`'s pool build block on a
|
||||
* network call.
|
||||
*/
|
||||
import {
|
||||
getUsageForProvider,
|
||||
USAGE_FETCHER_PROVIDERS,
|
||||
type UsageFetcherProvider,
|
||||
} from "./../usage.ts";
|
||||
import { getCachedProviderConnections } from "@/lib/db/readCache";
|
||||
import { defaultLogger as log } from "@omniroute/open-sse/utils/logger";
|
||||
import type { FreeAccessState } from "./strictZeroCostFilter";
|
||||
|
||||
const USAGE_FETCHER_PROVIDER_SET = new Set<string>(USAGE_FETCHER_PROVIDERS);
|
||||
|
||||
/** Default cache TTL, reused verbatim from the already-shipped
|
||||
* `settings.autoRefreshProviderQuotaInterval` (180s default,
|
||||
* `src/lib/db/settings.ts`) instead of inventing a new number. */
|
||||
const FALLBACK_TTL_MS = 180_000;
|
||||
|
||||
/** Cold-cache thundering-herd guard: after a process restart every candidate
|
||||
* in a pool build is a simultaneous cache miss, which without a cap would
|
||||
* fire one `getUsageForProvider()` call per distinct (provider, connection)
|
||||
* pair in the same tick. Capping concurrent background refreshes spreads
|
||||
* that burst out — a skipped refresh here just means this candidate stays
|
||||
* UNKNOWN (excluded, fail-closed) until a later pool build tries again, never
|
||||
* a correctness issue. */
|
||||
const MAX_CONCURRENT_REFRESHES = 4;
|
||||
|
||||
/** Entries older than this are pruned outright even if nothing ever triggers
|
||||
* a fresh refresh for that exact key again (e.g. the connection was deleted
|
||||
* and no candidate references it anymore, so a normal stale-triggered
|
||||
* refresh — which self-heals inside one TTL window — never fires). A sweep,
|
||||
* not a timer: piggybacks on `resolveFreeAccessState` calls so this module
|
||||
* never owns its own background interval. */
|
||||
const HARD_EVICTION_AGE_MS = FALLBACK_TTL_MS * 20; // 1 hour at the default TTL
|
||||
const SWEEP_EVERY_N_CALLS = 200;
|
||||
|
||||
interface CacheEntry {
|
||||
state: FreeAccessState;
|
||||
fetchedAtMs: number;
|
||||
}
|
||||
|
||||
// Keyed by `${provider}::${connectionId}` — module-level, process-lifetime
|
||||
// cache. Cleared per-entry by `invalidateFreeAccessState`, on a failed
|
||||
// refresh, or by the periodic sweep below; never wholesale.
|
||||
const cache = new Map<string, CacheEntry>();
|
||||
const inFlight = new Set<string>();
|
||||
let resolveCallCount = 0;
|
||||
|
||||
function cacheKey(provider: string, connectionId: string): string {
|
||||
return `${provider}::${connectionId}`;
|
||||
}
|
||||
|
||||
// `getSettings()` is async (DB-backed); reading it synchronously here isn't
|
||||
// possible without changing `resolveFreeAccessState`'s synchronous contract.
|
||||
// Using the fallback unconditionally is equivalent in practice: it's the same
|
||||
// number as `settings.autoRefreshProviderQuotaInterval`'s own default
|
||||
// (`src/lib/db/settings.ts`), and `strictZeroCostFilter.ts`'s own
|
||||
// `maxStateAgeMs` (passed the real, live setting from `virtualFactory.ts`)
|
||||
// is the check that actually gates staleness for the STRICT_ZERO_COST
|
||||
// decision — this cache TTL only bounds how long a background refresh is
|
||||
// skipped, a looser, non-safety-critical concern.
|
||||
function ttlMs(): number {
|
||||
return FALLBACK_TTL_MS;
|
||||
}
|
||||
|
||||
/** Opportunistic sweep of very stale entries, run every N calls instead of on
|
||||
* a timer. Cheap (a single Map iteration) and only ever removes entries no
|
||||
* live candidate can plausibly still be waiting on. */
|
||||
function sweepIfDue(): void {
|
||||
resolveCallCount += 1;
|
||||
if (resolveCallCount % SWEEP_EVERY_N_CALLS !== 0) return;
|
||||
const now = Date.now();
|
||||
for (const [key, entry] of cache) {
|
||||
if (now - entry.fetchedAtMs > HARD_EVICTION_AGE_MS) cache.delete(key);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Best-effort, provider-agnostic extraction of "how much free allowance is
|
||||
* left" from whatever shape `getUsageForProvider()` returns for this
|
||||
* provider today. Adapters were written for a human-readable quota display,
|
||||
* not for this filter, so their payloads are heterogeneous; this function
|
||||
* recognizes the two shapes already used by other read paths in this
|
||||
* codebase (`quotas.*.remainingPercentage` / `.total`+`.remaining`, mirroring
|
||||
* `quota_omniroute.py`'s own parsing) and returns `null` — never a guess —
|
||||
* for anything else. `null` is treated as "not proven safe" by the filter.
|
||||
*/
|
||||
function extractRemainingAllowance(usage: unknown): number | null {
|
||||
if (!usage || typeof usage !== "object") return null;
|
||||
const quotas = (usage as Record<string, unknown>).quotas;
|
||||
if (!quotas || typeof quotas !== "object") return null;
|
||||
|
||||
let worstPercent: number | null = null;
|
||||
for (const raw of Object.values(quotas as Record<string, unknown>)) {
|
||||
if (!raw || typeof raw !== "object") continue;
|
||||
const q = raw as Record<string, unknown>;
|
||||
if (q.unlimited === true) continue;
|
||||
let pct: number | null =
|
||||
typeof q.remainingPercentage === "number" ? q.remainingPercentage : null;
|
||||
if (
|
||||
pct === null &&
|
||||
typeof q.total === "number" &&
|
||||
typeof q.remaining === "number" &&
|
||||
q.total > 0
|
||||
) {
|
||||
pct = (100 * q.remaining) / q.total;
|
||||
}
|
||||
if (pct === null) continue;
|
||||
worstPercent = worstPercent === null ? pct : Math.min(worstPercent, pct);
|
||||
}
|
||||
return worstPercent; // percentage points; the filter's threshold is compared against this unit
|
||||
}
|
||||
|
||||
async function refresh(provider: string, connectionId: string): Promise<void> {
|
||||
const key = cacheKey(provider, connectionId);
|
||||
if (inFlight.has(key)) return;
|
||||
if (inFlight.size >= MAX_CONCURRENT_REFRESHES) return; // thundering-herd guard — see const doc above
|
||||
inFlight.add(key);
|
||||
try {
|
||||
const connections = await getCachedProviderConnections();
|
||||
const connection = connections.find(
|
||||
(c): c is Record<string, unknown> =>
|
||||
!!c &&
|
||||
typeof c === "object" &&
|
||||
(c as Record<string, unknown>).id === connectionId &&
|
||||
(c as Record<string, unknown>).provider === provider
|
||||
);
|
||||
if (!connection) {
|
||||
cache.delete(key);
|
||||
return;
|
||||
}
|
||||
const usage = await getUsageForProvider(
|
||||
connection as unknown as Parameters<typeof getUsageForProvider>[0],
|
||||
{ forceRefresh: false }
|
||||
);
|
||||
const remaining = extractRemainingAllowance(usage);
|
||||
const state: FreeAccessState = {
|
||||
status: remaining === null ? "UNKNOWN" : remaining > 0 ? "SAFE" : "EXHAUSTED",
|
||||
remainingFreeAllowance: remaining,
|
||||
resetAt:
|
||||
usage &&
|
||||
typeof usage === "object" &&
|
||||
typeof (usage as Record<string, unknown>).resetAt === "string"
|
||||
? ((usage as Record<string, unknown>).resetAt as string)
|
||||
: null,
|
||||
checkedAt: new Date().toISOString(),
|
||||
};
|
||||
cache.set(key, { state, fetchedAtMs: Date.now() });
|
||||
} catch (err) {
|
||||
// A failed lookup must never leave a stale SAFE entry behind — drop it so
|
||||
// the next read is a clean cache miss (UNKNOWN), not a lucky reuse.
|
||||
cache.delete(key);
|
||||
log.warn("AUTO", "STRICT_ZERO_COST: usage refresh failed, treating as UNKNOWN", {
|
||||
provider,
|
||||
err: err instanceof Error ? err.message : String(err),
|
||||
});
|
||||
} finally {
|
||||
inFlight.delete(key);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Synchronous read for `strictZeroCostFilter.ts`. Returns `undefined` when
|
||||
* there's no usage adapter for this provider at all (a permanent UNKNOWN, no
|
||||
* point ever refreshing), or on a cold/stale cache — in both cases a
|
||||
* background refresh is kicked off (fire-and-forget, subject to the
|
||||
* concurrency cap above) so a *later* read can benefit, but this call itself
|
||||
* never blocks or throws.
|
||||
*/
|
||||
export function resolveFreeAccessState(
|
||||
provider: string,
|
||||
connectionId: string | undefined
|
||||
): FreeAccessState | undefined {
|
||||
sweepIfDue();
|
||||
if (!USAGE_FETCHER_PROVIDER_SET.has(provider as UsageFetcherProvider)) return undefined;
|
||||
if (!connectionId) return undefined;
|
||||
|
||||
const key = cacheKey(provider, connectionId);
|
||||
const entry = cache.get(key);
|
||||
const fresh = entry && Date.now() - entry.fetchedAtMs <= ttlMs();
|
||||
if (!fresh) {
|
||||
void refresh(provider, connectionId);
|
||||
}
|
||||
return fresh ? entry.state : undefined;
|
||||
}
|
||||
|
||||
/** Called by `accountFallback.ts` the moment a 402/403/quota-exhausted
|
||||
* response is classified for a connection — drops the cached entry
|
||||
* immediately instead of waiting out the TTL, so the very next candidate-pool
|
||||
* build reads a clean cache miss (UNKNOWN) rather than a stale SAFE. */
|
||||
export function invalidateFreeAccessState(provider: string, connectionId: string): void {
|
||||
cache.delete(cacheKey(provider, connectionId));
|
||||
}
|
||||
|
||||
export const __testing = { cache, extractRemainingAllowance, sweepIfDue };
|
||||
283
open-sse/services/autoCombo/strictZeroCostFilter.ts
Normal file
283
open-sse/services/autoCombo/strictZeroCostFilter.ts
Normal file
@@ -0,0 +1,283 @@
|
||||
/**
|
||||
* STRICT_ZERO_COST — an opt-in, stricter sibling of `hidePaidModels`
|
||||
* (`paidModelFilter.ts`) for operators who need a hard guarantee against ANY
|
||||
* incremental monetary spend, not just "documented as free".
|
||||
*
|
||||
* `hidePaidModels` answers "is this model classified free in FREE_MODEL_BUDGETS
|
||||
* right now?" — a point-in-time catalog fact. It says nothing about whether a
|
||||
* `recurring-*`/`one-time-initial` candidate's allowance has since been
|
||||
* consumed, and nothing about whether exceeding it is a hard stop or silent
|
||||
* pay-as-you-go billing. STRICT_ZERO_COST adds exactly those two checks,
|
||||
* before ranking, before dispatch — never after.
|
||||
*
|
||||
* Design, kept deliberately close to `filterPaidOnlyCandidates`'s own stated
|
||||
* goal: "a pure, dependency-light function so the filter is unit-testable in
|
||||
* isolation". The live quota lookup (`getUsageForProvider`, cached with a TTL)
|
||||
* lives in `freeAccessQuota.ts` and is injected here as a plain function —
|
||||
* this file never imports the DB or makes a network call itself.
|
||||
*
|
||||
* No provider or model name appears anywhere in this file. A candidate passes
|
||||
* or fails purely on the metadata it carries (`freeType`, `tos`,
|
||||
* `hardStopGuaranteed`) plus, for quota-based types, a `FreeAccessState`
|
||||
* resolved elsewhere. A future provider that ships correct metadata is
|
||||
* handled automatically; one that doesn't is excluded automatically — see
|
||||
* `docs/routing/STRICT_ZERO_COST.md`.
|
||||
*
|
||||
* ## Connection safety (fixed after code review, see `docs/routing/STRICT_ZERO_COST.md`)
|
||||
*
|
||||
* A candidate from `virtualFactory.ts`'s connection-based pool represents ONE
|
||||
* provider/model pair with a set of *eligible* connections
|
||||
* (`allowedConnectionIds`) — the actual connection used at dispatch is chosen
|
||||
* later (session stickiness/LKGP), not by this filter. Two invariants follow:
|
||||
*
|
||||
* 1. The `keyless` shortcut (no live check needed, because no credential
|
||||
* exists) is valid ONLY for candidates that genuinely came from the
|
||||
* no-auth path — identified by `connectionId === SYNTHETIC_NOAUTH_CONNECTION_ID`
|
||||
* (`resilienceCandidateFilter.ts`). A `keyless`-catalogued model reached
|
||||
* through a real DB connection (the same provider also has a
|
||||
* credentialed connection) does NOT get the shortcut — it falls through
|
||||
* to the normal quota-based check like any other freeType, and is
|
||||
* excluded unless that specific connection independently proves SAFE.
|
||||
* 2. For a multi-account candidate (`connectionId: null`,
|
||||
* `allowedConnectionIds: [...]`), each connection is checked
|
||||
* INDIVIDUALLY. The returned candidate's `allowedConnectionIds` is
|
||||
* REWRITTEN to exactly the subset proven SAFE — never the full original
|
||||
* list. `autoStrategy.ts` (`open-sse/services/combo/autoStrategy.ts:315-331`)
|
||||
* already intersects further routing against `allowedConnectionIds`
|
||||
* before connection selection, so rewriting it here is enough to make
|
||||
* "verified this connection" and "dispatch used this connection" the
|
||||
* same set, by construction — no new enforcement point needed.
|
||||
*/
|
||||
import {
|
||||
FREE_MODEL_BUDGETS,
|
||||
type FreeModelBudget,
|
||||
} from "@omniroute/open-sse/config/freeModelCatalog.ts";
|
||||
import { SYNTHETIC_NOAUTH_CONNECTION_ID } from "./resilienceCandidateFilter";
|
||||
|
||||
/** Types whose allowance needs no runtime verification: no credential exists
|
||||
* for the candidate at all, so no request against it can ever be billed. */
|
||||
const KEYLESS_FREE_TYPES = new Set<FreeModelBudget["freeType"]>(["keyless"]);
|
||||
|
||||
export type FreeAccessStatus = "SAFE" | "EXHAUSTED" | "UNKNOWN";
|
||||
|
||||
/** Live-checked allowance state for one (provider, connection) pair. Resolved
|
||||
* and cached by `freeAccessQuota.ts`; passed in here as plain data so this
|
||||
* module stays free of DB/network dependencies. */
|
||||
export interface FreeAccessState {
|
||||
status: FreeAccessStatus;
|
||||
/** Remaining free allowance in the provider's own unit (tokens, requests, or
|
||||
* USD-equivalent) — whatever `getUsageForProvider()` reports. `null` when
|
||||
* the provider's usage payload doesn't expose a numeric remaining figure. */
|
||||
remainingFreeAllowance: number | null;
|
||||
/** When the allowance next resets, if the provider reports it. */
|
||||
resetAt: string | null;
|
||||
/** When this state was fetched (ISO 8601) — used to detect staleness. */
|
||||
checkedAt: string;
|
||||
}
|
||||
|
||||
/** A candidate as this module needs to see it — a structural subset of
|
||||
* `VirtualAutoComboCandidate` (`virtualFactory.ts`) so this file has no
|
||||
* dependency on that module's full type. */
|
||||
export interface StrictZeroCostCandidate {
|
||||
provider: string;
|
||||
model: string;
|
||||
connectionId: string | null;
|
||||
allowedConnectionIds?: string[];
|
||||
}
|
||||
|
||||
export interface StrictZeroCostOptions {
|
||||
/** Master switch — mirrors `hidePaidModels`'s own off-by-default shape. */
|
||||
enabled: boolean;
|
||||
/**
|
||||
* Resolves the live allowance state for ONE specific (provider, connection)
|
||||
* pair. Returns `undefined` when no usage capability exists for the
|
||||
* provider at all (no adapter registered in `USAGE_FETCHER_PROVIDERS`), or
|
||||
* when the cache has nothing fresh for this exact connection — both are a
|
||||
* meaningful, terminal UNKNOWN for that connection, not an error to retry.
|
||||
*
|
||||
* Synchronous by design: the caller (`virtualFactory.ts`) resolves and
|
||||
* caches state per candidate up front, once per pool build, so this filter
|
||||
* itself never awaits a network call and stays trivially testable.
|
||||
*/
|
||||
resolveFreeAccessState: (provider: string, connectionId: string) => FreeAccessState | undefined;
|
||||
/** Minimum remaining allowance (in the unit `resolveFreeAccessState` reports
|
||||
* — percentage points for the built-in `freeAccessQuota.ts` resolver) a
|
||||
* quota-based connection must exceed to pass. Must be >= 0; a fully-exhausted
|
||||
* account (`remainingFreeAllowance === 0`) fails at any non-negative
|
||||
* threshold via the strict `>` comparison below. */
|
||||
minRemainingAllowance: number;
|
||||
/** Maximum age, in ms, a `FreeAccessState.checkedAt` may have before it's
|
||||
* treated as stale (→ UNKNOWN, excluded). */
|
||||
maxStateAgeMs: number;
|
||||
/** `now` injection for deterministic tests; defaults to `Date.now`. */
|
||||
now?: () => number;
|
||||
/**
|
||||
* The free-model catalog to look candidates up against. Defaults to the
|
||||
* real, live `FREE_MODEL_BUDGETS` — overridable so tests can prove the
|
||||
* autodiscovery contract (a provider/model that appears in the catalog is
|
||||
* automatically considered; one that's removed automatically disappears)
|
||||
* with synthetic fixtures instead of mutating global state. Production
|
||||
* callers should never pass this. Threaded through by
|
||||
* `filterStrictZeroCostCandidates` (previously accepted but silently
|
||||
* ignored — fixed alongside the connection-safety review).
|
||||
*/
|
||||
catalog?: readonly FreeModelBudget[];
|
||||
}
|
||||
|
||||
export function findBudgetEntry(
|
||||
candidate: Pick<StrictZeroCostCandidate, "provider" | "model">,
|
||||
catalog: readonly FreeModelBudget[] = FREE_MODEL_BUDGETS
|
||||
): FreeModelBudget | undefined {
|
||||
return catalog.find((m) => m.provider === candidate.provider && m.modelId === candidate.model);
|
||||
}
|
||||
|
||||
function isConnectionStateSafe(
|
||||
provider: string,
|
||||
connectionId: string,
|
||||
resolveFreeAccessState: StrictZeroCostOptions["resolveFreeAccessState"],
|
||||
options: Pick<StrictZeroCostOptions, "minRemainingAllowance" | "maxStateAgeMs" | "now">
|
||||
): boolean {
|
||||
const state = resolveFreeAccessState(provider, connectionId);
|
||||
if (!state) return false; // no usage adapter for this provider, or lookup never ran/is stale
|
||||
if (state.status !== "SAFE") return false;
|
||||
|
||||
const now = (options.now ?? Date.now)();
|
||||
const checkedAtMs = Date.parse(state.checkedAt);
|
||||
if (!Number.isFinite(checkedAtMs) || now - checkedAtMs > options.maxStateAgeMs) return false;
|
||||
|
||||
if (state.remainingFreeAllowance === null) return false;
|
||||
// A negative threshold would let a negative/garbage reading pass; a caller
|
||||
// that genuinely wants "any allowance greater than zero" should pass 0.
|
||||
if (options.minRemainingAllowance < 0) return false;
|
||||
return state.remainingFreeAllowance > options.minRemainingAllowance;
|
||||
}
|
||||
|
||||
/**
|
||||
* Decide which of a candidate's connections satisfy STRICT_ZERO_COST. Pure —
|
||||
* `resolveFreeAccessState` is the only injected side-effecting dependency,
|
||||
* and it's a synchronous cache read (see `StrictZeroCostOptions` above).
|
||||
*
|
||||
* Returns the list of connection ids proven SAFE right now:
|
||||
* - `[SYNTHETIC_NOAUTH_CONNECTION_ID]` for a genuine no-auth candidate whose
|
||||
* catalog entry is `keyless` — no live check needed or possible.
|
||||
* - a (possibly empty) subset of the candidate's real connection id(s) for
|
||||
* every other case, each individually verified.
|
||||
* An empty array means the caller must exclude the candidate entirely.
|
||||
*/
|
||||
export function evaluateCandidateConnections(
|
||||
candidate: StrictZeroCostCandidate,
|
||||
budgetEntry: FreeModelBudget | undefined,
|
||||
resolveFreeAccessState: StrictZeroCostOptions["resolveFreeAccessState"],
|
||||
options: Pick<StrictZeroCostOptions, "minRemainingAllowance" | "maxStateAgeMs" | "now">
|
||||
): string[] {
|
||||
if (!budgetEntry) return []; // not in the catalog at all → paid, or genuinely unknown
|
||||
|
||||
const isGenuineNoAuthCandidate = candidate.connectionId === SYNTHETIC_NOAUTH_CONNECTION_ID;
|
||||
if (KEYLESS_FREE_TYPES.has(budgetEntry.freeType)) {
|
||||
// The keyless shortcut is trustworthy ONLY when this specific candidate
|
||||
// instance actually has no credential behind it. A `keyless`-catalogued
|
||||
// model reached through a real DB connection (connectionId is a real id,
|
||||
// or the candidate carries allowedConnectionIds at all) must NOT take
|
||||
// this shortcut — it falls through to the quota-based check below like
|
||||
// any other freeType, and is excluded there unless hardStopGuaranteed is
|
||||
// also set for it (which the curated catalog does not do for keyless
|
||||
// entries today, so it will correctly exclude).
|
||||
if (isGenuineNoAuthCandidate) return [SYNTHETIC_NOAUTH_CONNECTION_ID];
|
||||
}
|
||||
if (budgetEntry.freeType === "discontinued") return [];
|
||||
if (isGenuineNoAuthCandidate) return []; // no-auth path but a non-keyless catalog entry: contradictory metadata, fail closed
|
||||
|
||||
// Every remaining freeType (recurring-*, one-time-initial, a keyless entry
|
||||
// reached via a real connection, and any future type this module doesn't
|
||||
// special-case) requires a documented hard stop before any live check even
|
||||
// runs — no point burning a quota lookup on a connection we could never
|
||||
// trust regardless of its answer.
|
||||
if (budgetEntry.hardStopGuaranteed !== true) return [];
|
||||
|
||||
const candidateConnectionIds = candidate.connectionId
|
||||
? [candidate.connectionId]
|
||||
: (candidate.allowedConnectionIds ?? []);
|
||||
|
||||
const safe: string[] = [];
|
||||
for (const connectionId of candidateConnectionIds) {
|
||||
if (connectionId === SYNTHETIC_NOAUTH_CONNECTION_ID) continue; // never reachable here, defensive
|
||||
if (isConnectionStateSafe(candidate.provider, connectionId, resolveFreeAccessState, options)) {
|
||||
safe.push(connectionId);
|
||||
}
|
||||
}
|
||||
return safe;
|
||||
}
|
||||
|
||||
/**
|
||||
* Pool-level filter, same off-by-default identity contract as
|
||||
* `filterPaidOnlyCandidates`. For a candidate that survives with a NARROWED
|
||||
* connection set (the multi-account case), the returned object has
|
||||
* `allowedConnectionIds` rewritten to exactly the SAFE subset — dispatch can
|
||||
* then never select a connection this filter didn't verify, because
|
||||
* `autoStrategy.ts` already enforces `allowedConnectionIds` as a hard
|
||||
* allowlist downstream (see the module docstring above).
|
||||
*/
|
||||
export function filterStrictZeroCostCandidates<T extends StrictZeroCostCandidate>(
|
||||
pool: T[],
|
||||
options: StrictZeroCostOptions
|
||||
): T[] {
|
||||
if (!options.enabled) return pool;
|
||||
|
||||
const kept: T[] = [];
|
||||
let changed = false;
|
||||
for (const candidate of pool) {
|
||||
const budgetEntry = findBudgetEntry(candidate, options.catalog);
|
||||
const safeConnectionIds = evaluateCandidateConnections(
|
||||
candidate,
|
||||
budgetEntry,
|
||||
options.resolveFreeAccessState,
|
||||
options
|
||||
);
|
||||
if (safeConnectionIds.length === 0) {
|
||||
changed = true;
|
||||
continue;
|
||||
}
|
||||
|
||||
const isGenuineNoAuthCandidate = candidate.connectionId === SYNTHETIC_NOAUTH_CONNECTION_ID;
|
||||
const isSingleConnectionCandidate = candidate.connectionId !== null;
|
||||
if (isGenuineNoAuthCandidate || isSingleConnectionCandidate) {
|
||||
// Nothing to narrow — either the no-auth sentinel, or a candidate that
|
||||
// already pointed at exactly one connection which proved safe.
|
||||
kept.push(candidate);
|
||||
continue;
|
||||
}
|
||||
|
||||
// Multi-account candidate: only rewrite if the safe subset is actually
|
||||
// narrower than what was there before, to preserve the same
|
||||
// identity-when-nothing-changed contract as `filterPaidOnlyCandidates`.
|
||||
const original = candidate.allowedConnectionIds ?? [];
|
||||
const isSameSet =
|
||||
original.length === safeConnectionIds.length &&
|
||||
safeConnectionIds.every((id) => original.includes(id));
|
||||
if (isSameSet) {
|
||||
kept.push(candidate);
|
||||
} else {
|
||||
changed = true;
|
||||
kept.push({ ...candidate, allowedConnectionIds: safeConnectionIds });
|
||||
}
|
||||
}
|
||||
return changed ? kept : pool;
|
||||
}
|
||||
|
||||
/**
|
||||
* Separate, optional ToS guard — kept independent from economic safety on
|
||||
* purpose (Marco's requirement): a model can be economically SAFE and still
|
||||
* excluded here for ToS reasons, or left in when this guard is off even if
|
||||
* STRICT_ZERO_COST is on. Reuses the same curated `tos` field, no new data.
|
||||
*/
|
||||
export function filterTosAvoidCandidates<T extends StrictZeroCostCandidate>(
|
||||
pool: T[],
|
||||
excludeTosAvoid: boolean,
|
||||
catalog?: readonly FreeModelBudget[]
|
||||
): T[] {
|
||||
if (!excludeTosAvoid) return pool;
|
||||
return pool.filter((candidate) => {
|
||||
const budgetEntry = findBudgetEntry(candidate, catalog);
|
||||
return budgetEntry?.tos !== "avoid";
|
||||
});
|
||||
}
|
||||
@@ -26,6 +26,8 @@ import { buildFamilyCandidateFilter, type ModelFamily } from "./modelFamily";
|
||||
import { getHiddenModelsByProvider } from "@/models";
|
||||
import { getSyncedAvailableModelsByConnection, getCustomModels } from "@/lib/db/models";
|
||||
import { filterPaidOnlyCandidates } from "./paidModelFilter";
|
||||
import { filterStrictZeroCostCandidates, filterTosAvoidCandidates } from "./strictZeroCostFilter";
|
||||
import { resolveFreeAccessState } from "./freeAccessQuota";
|
||||
import { isModelExcludedByConnection } from "@/domain/connectionModelRules";
|
||||
import { resolveProviderAlias } from "../model.ts";
|
||||
import { filterExcludedCandidates } from "./candidateOverrides";
|
||||
@@ -590,6 +592,29 @@ export async function prepareVirtualAutoComboInputs(
|
||||
// exclude paid-only backends from EVERY `auto/*` candidate pool.
|
||||
const paidFilteredPool = filterPaidOnlyCandidates(pool, settings.hidePaidModels === true);
|
||||
if (paidFilteredPool !== pool) pool = paidFilteredPool;
|
||||
|
||||
// STRICT_ZERO_COST: opt-in, off by default (`settings.freeAccessPolicy !== "strict"`
|
||||
// leaves `pool` byte-identical, same contract as `hidePaidModels`). See
|
||||
// `strictZeroCostFilter.ts` for why this is stricter than `hidePaidModels` alone —
|
||||
// including the connection-safety invariant it enforces per-connection, not just
|
||||
// per-candidate: `resolveFreeAccessState` here is a raw pass-through of the real
|
||||
// per-(provider,connectionId) resolver; the filter itself decides which connection(s)
|
||||
// on each candidate to check and rewrites `allowedConnectionIds` to the SAFE subset.
|
||||
const strictFilteredPool = filterStrictZeroCostCandidates(pool, {
|
||||
enabled: settings.freeAccessPolicy === "strict",
|
||||
resolveFreeAccessState,
|
||||
// 1 percentage point of headroom, not 0: `freeAccessQuota.ts` reports
|
||||
// remaining allowance as a percentage, and a raw ">0" comparison would
|
||||
// let a reading of e.g. 0.3% (rounding noise, not real headroom) pass.
|
||||
minRemainingAllowance: 1,
|
||||
maxStateAgeMs: (settings.autoRefreshProviderQuotaInterval ?? 180) * 1000,
|
||||
});
|
||||
if (strictFilteredPool !== pool) pool = strictFilteredPool;
|
||||
|
||||
// Separate, optional ToS guard — independent of economic safety on purpose.
|
||||
const tosFilteredPool = filterTosAvoidCandidates(pool, settings.excludeTosAvoid === true);
|
||||
if (tosFilteredPool !== pool) pool = tosFilteredPool;
|
||||
|
||||
return pool;
|
||||
};
|
||||
|
||||
|
||||
@@ -589,10 +589,7 @@ function resolveSilentCloseOutcome(input: {
|
||||
if (!input.bytesWereForwarded) return null;
|
||||
|
||||
if (!input.clientTerminalSeen) {
|
||||
if (
|
||||
input.clientResponseFormat === FORMATS.CLAUDE &&
|
||||
input.contentWatcher.sawContent()
|
||||
) {
|
||||
if (input.clientResponseFormat === FORMATS.CLAUDE && input.contentWatcher.sawContent()) {
|
||||
// #7699 — upstream dropped after content reached the client on a Claude
|
||||
// stream. Keep the partial response: emit a clean max_tokens completion
|
||||
// instead of an error frame so Anthropic SDK / Claude Code don't report
|
||||
@@ -611,6 +608,16 @@ function resolveSilentCloseOutcome(input: {
|
||||
if (input.clientResponseFormat === FORMATS.OPENAI && input.contentWatcher.sawContent()) {
|
||||
return { kind: "error", reason: "Upstream stream ended without a terminal marker" };
|
||||
}
|
||||
// Responses-format clients (Codex CLI and other /v1/responses consumers):
|
||||
// a healthy OpenAI Responses stream ALWAYS terminates with an explicit
|
||||
// `response.completed` event — it is the format's only terminal marker and
|
||||
// carries the final status/usage. Content forwarded without it is an
|
||||
// upstream drop, the same class as #10443 for chat completions; surface a
|
||||
// synthetic response.failed instead of a silent close so clients report
|
||||
// the break instead of waiting on a completion event that never comes.
|
||||
if (isResponsesClientFormat(input.clientResponseFormat) && input.contentWatcher.sawContent()) {
|
||||
return { kind: "error", reason: "Upstream stream ended without a terminal marker" };
|
||||
}
|
||||
}
|
||||
|
||||
const watcher = input.contentWatcher;
|
||||
|
||||
105
scripts/ad-hoc/dry-run-strict-zero-cost.ts
Normal file
105
scripts/ad-hoc/dry-run-strict-zero-cost.ts
Normal file
@@ -0,0 +1,105 @@
|
||||
/**
|
||||
* Ad-hoc, one-shot dry run of STRICT_ZERO_COST against the real candidate
|
||||
* pools currently served by this OmniRoute instance (fetched via the
|
||||
* existing read-only `GET /v1/auto-combo/{channel}/candidates` endpoint —
|
||||
* no changes made, no billable calls). Not wired into any test suite.
|
||||
*
|
||||
* Simulates the filter offline: no live usage-quota state is available
|
||||
* (that adapter only runs inside the deployed container), so
|
||||
* `resolveFreeAccessState` always returns `undefined` here — meaning any
|
||||
* quota-based candidate is reported UNKNOWN unless it lacks even a usage
|
||||
* adapter, in which case it's reported UNKNOWN for that reason instead. This
|
||||
* intentionally shows the current, honest ceiling of what's usable today.
|
||||
*
|
||||
* Uses each candidate's REAL `connectionId` from the live endpoint (rather
|
||||
* than assuming) to also exercise the post-code-review connection-safety
|
||||
* check: a `keyless`-catalogued model whose live `connectionId` is NOT the
|
||||
* no-auth sentinel is correctly reported as excluded here too.
|
||||
*/
|
||||
import { readFileSync } from "node:fs";
|
||||
import {
|
||||
evaluateCandidateConnections,
|
||||
findBudgetEntry,
|
||||
} from "../../open-sse/services/autoCombo/strictZeroCostFilter.ts";
|
||||
import { SYNTHETIC_NOAUTH_CONNECTION_ID } from "../../open-sse/services/autoCombo/resilienceCandidateFilter.ts";
|
||||
import { USAGE_FETCHER_PROVIDERS } from "../../open-sse/services/usage.ts";
|
||||
|
||||
const usageProviders = new Set<string>(USAGE_FETCHER_PROVIDERS);
|
||||
const OPTIONS = { minRemainingAllowance: 1, maxStateAgeMs: 180_000 };
|
||||
|
||||
interface Candidate {
|
||||
provider: string;
|
||||
model: string;
|
||||
connectionId: string;
|
||||
}
|
||||
|
||||
function loadCandidates(path: string): Candidate[] {
|
||||
const raw = JSON.parse(readFileSync(path, "utf8"));
|
||||
const list = Array.isArray(raw) ? raw : raw.candidates;
|
||||
// The candidates endpoint's `model` field is the FULL "<providerOrAlias>/<modelId>"
|
||||
// string (`modelStr` — the leading segment is sometimes the provider id,
|
||||
// e.g. "groq/...", sometimes its short alias, e.g. "oc/..." for opencode);
|
||||
// FREE_MODEL_BUDGETS.modelId is always bare. Strip exactly the first "/"
|
||||
// segment (whichever form it is) so e.g. "groq/meta-llama/llama-4-scout..."
|
||||
// becomes "meta-llama/llama-4-scout..." and "oc/big-pickle" becomes
|
||||
// "big-pickle", matching the catalog's modelId either way.
|
||||
return list.map((c: { provider: string; model: string; connectionId?: string }) => {
|
||||
const slash = c.model.indexOf("/");
|
||||
return {
|
||||
provider: c.provider,
|
||||
model: slash === -1 ? c.model : c.model.slice(slash + 1),
|
||||
connectionId: c.connectionId ?? SYNTHETIC_NOAUTH_CONNECTION_ID,
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
function run(label: string, path: string): void {
|
||||
const candidates = loadCandidates(path);
|
||||
console.log(`\n=== ${label} — ${candidates.length} candidati live ===`);
|
||||
|
||||
const kept: Candidate[] = [];
|
||||
const excluded: { candidate: Candidate; reason: string }[] = [];
|
||||
|
||||
for (const c of candidates) {
|
||||
const entry = findBudgetEntry(c);
|
||||
if (!entry) {
|
||||
excluded.push({ candidate: c, reason: "non presente nel catalogo free curato" });
|
||||
continue;
|
||||
}
|
||||
const isNoAuthConnection = c.connectionId === SYNTHETIC_NOAUTH_CONNECTION_ID;
|
||||
if (entry.freeType === "keyless") {
|
||||
const safe = evaluateCandidateConnections(c, entry, () => undefined, OPTIONS);
|
||||
if (safe.length > 0) {
|
||||
kept.push(c);
|
||||
} else if (!isNoAuthConnection) {
|
||||
excluded.push({
|
||||
candidate: c,
|
||||
reason:
|
||||
"keyless nel catalogo ma raggiunto tramite una connessione DB reale (non il sentinel noauth) — shortcut non applicato, richiederebbe hardStopGuaranteed",
|
||||
});
|
||||
} else {
|
||||
excluded.push({ candidate: c, reason: "keyless ma valutazione fallita (inatteso)" });
|
||||
}
|
||||
continue;
|
||||
}
|
||||
const hasAdapter = usageProviders.has(entry.provider);
|
||||
const reason = !hasAdapter
|
||||
? `nessun usage adapter per '${entry.provider}' in USAGE_FETCHER_PROVIDERS`
|
||||
: entry.hardStopGuaranteed !== true
|
||||
? "hardStopGuaranteed non dichiarato per questo modello"
|
||||
: "nessuno stato quota live disponibile in questo dry-run offline (richiederebbe il container reale)";
|
||||
excluded.push({ candidate: c, reason });
|
||||
}
|
||||
|
||||
console.log(`PRIMA (STRICT_ZERO_COST off): ${candidates.length} candidati`);
|
||||
console.log(`DOPO (STRICT_ZERO_COST on): ${kept.length} candidati sopravvissuti`);
|
||||
console.log("Sopravvissuti:");
|
||||
for (const c of kept) console.log(` OK ${c.provider}/${c.model}`);
|
||||
console.log("Esclusi (motivo):");
|
||||
for (const { candidate: c, reason } of excluded) {
|
||||
console.log(` EXCL ${c.provider}/${c.model} — ${reason}`);
|
||||
}
|
||||
}
|
||||
|
||||
run("auto/coding:free", process.argv[2] ?? "/tmp/dryrun_coding_free.json");
|
||||
run("auto/best-free", process.argv[3] ?? "/tmp/dryrun_best-free.json");
|
||||
@@ -238,6 +238,11 @@ export async function getSettings() {
|
||||
// (`:free` suffix, zero-price pricing, or FREE_MODEL_BUDGETS membership). Default
|
||||
// false preserves prior behaviour; opt-in only.
|
||||
hidePaidModels: false,
|
||||
// Opt-in, default off: same shape as hidePaidModels above, but requires a
|
||||
// live hard-stop-guaranteed quota check for non-keyless free candidates.
|
||||
// See open-sse/services/autoCombo/strictZeroCostFilter.ts.
|
||||
freeAccessPolicy: "off",
|
||||
excludeTosAvoid: false,
|
||||
// #9418: Opt-in filter that hides auto/* virtual combos from the /v1/models catalog.
|
||||
// User-defined combos are unaffected; routing still works for hidden ids sent explicitly.
|
||||
hideAutoCombos: false,
|
||||
|
||||
@@ -254,6 +254,39 @@ function leafModelId(modelId: string | null | undefined): string | null {
|
||||
return leaf && leaf !== modelId ? leaf : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Effort suffixes the catalog synthesizes as `<base>-<tier>` variant ids from a
|
||||
* base model's `supportedThinkingEfforts` (mirrors REGISTERED_EFFORT_SUFFIXES
|
||||
* in open-sse/utils/registeredEffortVariants.ts, plus `minimal` for muse).
|
||||
*/
|
||||
const EFFORT_VARIANT_SUFFIXES = [
|
||||
"none",
|
||||
"minimal",
|
||||
"low",
|
||||
"medium",
|
||||
"high",
|
||||
"xhigh",
|
||||
"max",
|
||||
] as const;
|
||||
|
||||
/**
|
||||
* Strip a trailing effort-tier suffix off a model id (e.g.
|
||||
* `deepseek-v4-flash-max` → `deepseek-v4-flash`). Longest token first so
|
||||
* `xhigh` is matched before `high`. Returns null when no known suffix matches
|
||||
* or the id would be left empty.
|
||||
*/
|
||||
function stripKnownEffortSuffix(modelId: string): string | null {
|
||||
const normalized = String(modelId || "").trim();
|
||||
if (!normalized) return null;
|
||||
for (const suffix of EFFORT_VARIANT_SUFFIXES) {
|
||||
const token = `-${suffix}`;
|
||||
if (normalized.length > token.length && normalized.endsWith(token)) {
|
||||
return normalized.slice(0, -token.length);
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function getStaticSpec(modelId: string | null, rawModel: string | null): ModelSpec | undefined {
|
||||
if (modelId) {
|
||||
const byCanonical = getModelSpec(modelId);
|
||||
@@ -704,15 +737,38 @@ export function getResolvedModelCapabilities(
|
||||
// persisted override never feeds back into the comparison that (re)writes it.
|
||||
const usePersistedOverrides = options?.persistedOverrides !== false;
|
||||
const resolved = resolveCapabilityInput(input);
|
||||
const spec = getStaticSpec(resolved.model, resolved.rawModel);
|
||||
const registryModel = getRegistryModel(resolved.provider, resolved.model);
|
||||
const synced = getSyncedCapabilityForResolved(
|
||||
let spec = getStaticSpec(resolved.model, resolved.rawModel);
|
||||
let registryModel = getRegistryModel(resolved.provider, resolved.model);
|
||||
let synced = getSyncedCapabilityForResolved(
|
||||
resolved.provider,
|
||||
resolved.model,
|
||||
resolved.rawModel,
|
||||
snapshot
|
||||
);
|
||||
|
||||
// Effort-suffixed variants (e.g. command-code `deepseek-v4-flash-max`,
|
||||
// `meta/muse-spark-1.2-contributor-xhigh`) are synthesized in the catalog
|
||||
// from the base model's `supportedThinkingEfforts`; they have no registry
|
||||
// row, synced row, or static spec of their own. Without a base-model
|
||||
// fallback the variant resolves with NULL tool/vision/context capabilities,
|
||||
// so a tool-bearing combo request treats the target as incompatible and
|
||||
// silently reorders it behind models with confirmed capabilities. Resolve
|
||||
// the variant's capabilities from its base model when every direct source
|
||||
// misses.
|
||||
if (!spec && !registryModel && !synced && resolved.provider && resolved.model) {
|
||||
const baseModelId = stripKnownEffortSuffix(resolved.model);
|
||||
if (baseModelId && baseModelId !== resolved.model) {
|
||||
spec = getStaticSpec(baseModelId, resolved.rawModel);
|
||||
registryModel = getRegistryModel(resolved.provider, baseModelId);
|
||||
synced = getSyncedCapabilityForResolved(
|
||||
resolved.provider,
|
||||
baseModelId,
|
||||
resolved.rawModel,
|
||||
snapshot
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const modalitiesInput = parseModalities(synced?.modalities_input);
|
||||
const modalitiesOutput = parseModalities(synced?.modalities_output);
|
||||
const lookupKey =
|
||||
|
||||
@@ -91,11 +91,23 @@ export const mergeOpenCodeConfig = (
|
||||
? existingConfig
|
||||
: {};
|
||||
|
||||
// Same guard as the root above, one level down. Spreading a non-object here
|
||||
// does not throw, it splays the value into index keys: an existing
|
||||
// `"provider": ["a", "b"]` merged to `{"0": "a", "1": "b", omniroute: ... }`
|
||||
// and a string was exploded one character per key. mergeOpenCodeConfigText
|
||||
// refuses the same input outright, so the two disagreed on what to do with a
|
||||
// malformed config.
|
||||
const existingProvider = (safeConfig as Record<string, unknown>).provider;
|
||||
const safeProvider =
|
||||
existingProvider && typeof existingProvider === "object" && !Array.isArray(existingProvider)
|
||||
? (existingProvider as Record<string, unknown>)
|
||||
: {};
|
||||
|
||||
return {
|
||||
...safeConfig,
|
||||
$schema: safeConfig.$schema || "https://opencode.ai/config.json",
|
||||
provider: {
|
||||
...((safeConfig as any).provider || {}),
|
||||
...safeProvider,
|
||||
omniroute: buildOpenCodeProviderConfig(input),
|
||||
},
|
||||
};
|
||||
|
||||
@@ -128,6 +128,16 @@ export const updateSettingsSchema = z.object({
|
||||
blockedProviders: z.array(z.string().max(100)).optional(),
|
||||
noAuthFallbackDisabledProviders: z.array(z.string().max(100)).optional(),
|
||||
hidePaidModels: z.boolean().optional(),
|
||||
// STRICT_ZERO_COST (opt-in, default "off"): stricter than hidePaidModels — a
|
||||
// candidate must be keyless (no credential exists, so no request against it
|
||||
// can ever be billed) OR pass a live, fresh, hard-stop-guaranteed quota
|
||||
// check, per candidate, before ranking/dispatch. See
|
||||
// open-sse/services/autoCombo/strictZeroCostFilter.ts.
|
||||
freeAccessPolicy: z.enum(["off", "strict"]).optional(),
|
||||
// Separate from freeAccessPolicy on purpose: excludes candidates whose
|
||||
// curated `tos` verdict is "avoid" (proxy/self-hosted use conflicts with the
|
||||
// provider's own terms) — a contractual concern, not an economic one.
|
||||
excludeTosAvoid: z.boolean().optional(),
|
||||
hideHealthCheckLogs: z.boolean().optional(),
|
||||
hideEndpointCloudflaredTunnel: z.boolean().optional(),
|
||||
hideEndpointTailscaleFunnel: z.boolean().optional(),
|
||||
|
||||
@@ -2453,6 +2453,18 @@ export async function markAccountUnavailable(
|
||||
try {
|
||||
await currentMutex;
|
||||
|
||||
// STRICT_ZERO_COST: this connection just failed (whatever the reason) —
|
||||
// drop any cached "SAFE" free-allowance reading for it immediately rather
|
||||
// than waiting out the TTL, so the very next candidate-pool build reads a
|
||||
// clean cache miss (UNKNOWN → excluded) instead of a stale SAFE. Cheap,
|
||||
// idempotent, and correct to over-invalidate on non-quota failures too —
|
||||
// worst case is one extra background refresh.
|
||||
if (provider) {
|
||||
const { invalidateFreeAccessState } =
|
||||
await import("@omniroute/open-sse/services/autoCombo/freeAccessQuota.ts");
|
||||
invalidateFreeAccessState(provider, connectionId);
|
||||
}
|
||||
|
||||
const resourceBypass = getResource404Bypass(status, errorText, connectionId, log);
|
||||
if (resourceBypass) return resourceBypass;
|
||||
|
||||
|
||||
192
tests/unit/autoCombo/strict-zero-cost-autodiscovery.test.ts
Normal file
192
tests/unit/autoCombo/strict-zero-cost-autodiscovery.test.ts
Normal file
@@ -0,0 +1,192 @@
|
||||
/**
|
||||
* STRICT_ZERO_COST — autodiscovery contract (Fase 5-8 of the design audit):
|
||||
* the filter must never hardcode a provider/model allowlist. It reads
|
||||
* whatever `catalog` (FREE_MODEL_BUDGETS-shaped) it's given, so a new SAFE
|
||||
* entry becomes usable the moment it appears, and a removed one disappears
|
||||
* the moment it's gone — no code change, no KITT change.
|
||||
*
|
||||
* Uses `evaluateCandidateConnections`'s injectable `catalog` argument (added
|
||||
* for exactly this test) with synthetic fixtures instead of mutating the
|
||||
* real, shared `FREE_MODEL_BUDGETS` — that constant is imported by
|
||||
* production code and other test files; mutating it at runtime would be a
|
||||
* global side effect this file has no business causing.
|
||||
*/
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import {
|
||||
evaluateCandidateConnections,
|
||||
findBudgetEntry,
|
||||
type FreeAccessState,
|
||||
type StrictZeroCostCandidate,
|
||||
} from "../../../open-sse/services/autoCombo/strictZeroCostFilter.ts";
|
||||
import { SYNTHETIC_NOAUTH_CONNECTION_ID } from "../../../open-sse/services/autoCombo/resilienceCandidateFilter.ts";
|
||||
import type { FreeModelBudget } from "../../../open-sse/config/freeModelCatalog.ts";
|
||||
|
||||
const NOW = "2026-08-20T00:00:00.000Z";
|
||||
const OPTIONS = { minRemainingAllowance: 1, maxStateAgeMs: 180_000, now: () => Date.parse(NOW) };
|
||||
|
||||
function safeState(): FreeAccessState {
|
||||
return { status: "SAFE", remainingFreeAllowance: 40, resetAt: null, checkedAt: NOW };
|
||||
}
|
||||
|
||||
// `evaluateCandidateConnections` has exactly one source of truth for "is this
|
||||
// candidate documented as free": the `budgetEntry` its caller looked up via
|
||||
// `findBudgetEntry(candidate, catalog)`. These provider ids are otherwise
|
||||
// arbitrary — the fixtures below prove the behavior is driven entirely by
|
||||
// catalog membership, not by any hardcoded provider/model name.
|
||||
const KEYLESS_PROVIDER = "felo-web";
|
||||
const QUOTA_PROVIDER = "groq";
|
||||
const REAL_CONN = "conn-1";
|
||||
|
||||
function keylessEntry(modelId: string): FreeModelBudget {
|
||||
return {
|
||||
provider: KEYLESS_PROVIDER,
|
||||
modelId,
|
||||
displayName: modelId,
|
||||
monthlyTokens: 0,
|
||||
creditTokens: 0,
|
||||
freeType: "keyless",
|
||||
poolKey: null,
|
||||
tos: "ok",
|
||||
};
|
||||
}
|
||||
|
||||
function quotaEntry(modelId: string, hardStopGuaranteed = true): FreeModelBudget {
|
||||
return {
|
||||
provider: QUOTA_PROVIDER,
|
||||
modelId,
|
||||
displayName: modelId,
|
||||
monthlyTokens: 1_000_000,
|
||||
creditTokens: 0,
|
||||
freeType: "recurring-daily",
|
||||
poolKey: null,
|
||||
tos: "ok",
|
||||
hardStopGuaranteed,
|
||||
};
|
||||
}
|
||||
|
||||
// 14/18. New provider/model appears in the catalog → automatically a candidate.
|
||||
test("a brand-new keyless model appearing in the catalog is usable the instant it appears", () => {
|
||||
const modelId = "synthetic-new-model-14";
|
||||
const candidate: StrictZeroCostCandidate = {
|
||||
provider: KEYLESS_PROVIDER,
|
||||
model: modelId,
|
||||
connectionId: SYNTHETIC_NOAUTH_CONNECTION_ID,
|
||||
};
|
||||
const catalogBefore: FreeModelBudget[] = []; // model does not exist yet
|
||||
const catalogAfter: FreeModelBudget[] = [keylessEntry(modelId)]; // "OmniRoute" just added it
|
||||
|
||||
assert.deepEqual(
|
||||
evaluateCandidateConnections(
|
||||
candidate,
|
||||
findBudgetEntry(candidate, catalogBefore),
|
||||
() => undefined,
|
||||
OPTIONS
|
||||
),
|
||||
[],
|
||||
"must be excluded before the catalog knows about it"
|
||||
);
|
||||
assert.deepEqual(
|
||||
evaluateCandidateConnections(
|
||||
candidate,
|
||||
findBudgetEntry(candidate, catalogAfter),
|
||||
() => undefined,
|
||||
OPTIONS
|
||||
),
|
||||
[SYNTHETIC_NOAUTH_CONNECTION_ID],
|
||||
"must pass automatically the moment the SAME code sees it in the catalog — no code change made between these two calls"
|
||||
);
|
||||
});
|
||||
|
||||
// 15/19. Provider/model removed from the catalog → automatically disappears.
|
||||
test("a model removed from the catalog stops being usable, with no code change", () => {
|
||||
const modelId = "synthetic-removed-model-15";
|
||||
const candidate: StrictZeroCostCandidate = {
|
||||
provider: KEYLESS_PROVIDER,
|
||||
model: modelId,
|
||||
connectionId: SYNTHETIC_NOAUTH_CONNECTION_ID,
|
||||
};
|
||||
const catalogBefore: FreeModelBudget[] = [keylessEntry(modelId)];
|
||||
const catalogAfter: FreeModelBudget[] = []; // "OmniRoute" dropped the promo
|
||||
|
||||
assert.deepEqual(
|
||||
evaluateCandidateConnections(
|
||||
candidate,
|
||||
findBudgetEntry(candidate, catalogBefore),
|
||||
() => undefined,
|
||||
OPTIONS
|
||||
),
|
||||
[SYNTHETIC_NOAUTH_CONNECTION_ID]
|
||||
);
|
||||
assert.deepEqual(
|
||||
evaluateCandidateConnections(
|
||||
candidate,
|
||||
findBudgetEntry(candidate, catalogAfter),
|
||||
() => undefined,
|
||||
OPTIONS
|
||||
),
|
||||
[],
|
||||
"must be excluded automatically once gone — no whitelist entry to delete anywhere"
|
||||
);
|
||||
});
|
||||
|
||||
// Case 2 of Fase 8: a quota-based model ships WITH a usage adapter and
|
||||
// hardStopGuaranteed=true and a SAFE live state — passes automatically.
|
||||
test("a new quota-based model with hardStopGuaranteed + SAFE state is usable automatically", () => {
|
||||
const modelId = "synthetic-quota-model-safe";
|
||||
const candidate: StrictZeroCostCandidate = {
|
||||
provider: QUOTA_PROVIDER,
|
||||
model: modelId,
|
||||
connectionId: REAL_CONN,
|
||||
};
|
||||
const catalog: FreeModelBudget[] = [quotaEntry(modelId, true)];
|
||||
assert.deepEqual(
|
||||
evaluateCandidateConnections(
|
||||
candidate,
|
||||
findBudgetEntry(candidate, catalog),
|
||||
() => safeState(),
|
||||
OPTIONS
|
||||
),
|
||||
[REAL_CONN]
|
||||
);
|
||||
});
|
||||
|
||||
// Case 3 of Fase 8: metadata incomplete (no hardStopGuaranteed) → UNKNOWN → excluded automatically.
|
||||
test("a new quota-based model WITHOUT hardStopGuaranteed is excluded automatically, even with a SAFE state", () => {
|
||||
const modelId = "synthetic-quota-model-unguaranteed";
|
||||
const candidate: StrictZeroCostCandidate = {
|
||||
provider: QUOTA_PROVIDER,
|
||||
model: modelId,
|
||||
connectionId: REAL_CONN,
|
||||
};
|
||||
const entry = quotaEntry(modelId);
|
||||
delete entry.hardStopGuaranteed; // simulate metadata that was never set, not defaulted to true
|
||||
const catalog: FreeModelBudget[] = [entry];
|
||||
assert.deepEqual(
|
||||
evaluateCandidateConnections(
|
||||
candidate,
|
||||
findBudgetEntry(candidate, catalog),
|
||||
() => safeState(),
|
||||
OPTIONS
|
||||
),
|
||||
[]
|
||||
);
|
||||
});
|
||||
|
||||
// Production never overrides `catalog` — `findBudgetEntry(candidate)` with no
|
||||
// second argument must read the real, live FREE_MODEL_BUDGETS, so an entirely
|
||||
// invented provider id (one no real OmniRoute release has ever catalogued) is
|
||||
// excluded automatically, with zero whitelist to edit, when the default is used.
|
||||
test("an entirely unknown provider id is excluded automatically when the default (real) catalog is used", () => {
|
||||
const candidate: StrictZeroCostCandidate = {
|
||||
provider: "totally-unheard-of-provider-xyz",
|
||||
model: "whatever",
|
||||
connectionId: REAL_CONN,
|
||||
};
|
||||
assert.deepEqual(
|
||||
evaluateCandidateConnections(candidate, findBudgetEntry(candidate), () => undefined, OPTIONS),
|
||||
[],
|
||||
"findBudgetEntry() with no catalog override reads the real FREE_MODEL_BUDGETS — nothing to add or remove for this provider to stay excluded"
|
||||
);
|
||||
});
|
||||
303
tests/unit/autoCombo/strict-zero-cost-connection-safety.test.ts
Normal file
303
tests/unit/autoCombo/strict-zero-cost-connection-safety.test.ts
Normal file
@@ -0,0 +1,303 @@
|
||||
/**
|
||||
* STRICT_ZERO_COST — connection-safety regression guards for the two
|
||||
* blockers found in code review of commit 5d4f881:
|
||||
*
|
||||
* BLOCKER 1 (keyless bypass): the `keyless` shortcut must apply ONLY to a
|
||||
* candidate that genuinely came from the no-auth path
|
||||
* (`connectionId === SYNTHETIC_NOAUTH_CONNECTION_ID`), never to a
|
||||
* `keyless`-catalogued model reached through a real DB connection.
|
||||
*
|
||||
* BLOCKER 2 (multi-account mismatch): a multi-account candidate's
|
||||
* `allowedConnectionIds` must be rewritten to exactly the SAFE subset, so
|
||||
* whatever `autoStrategy.ts` selects at dispatch time can never be a
|
||||
* connection this filter didn't itself verify.
|
||||
*
|
||||
* Both are pure-function tests against `evaluateCandidateConnections()` /
|
||||
* `filterStrictZeroCostCandidates()` — no DB, no network.
|
||||
*/
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import {
|
||||
evaluateCandidateConnections,
|
||||
filterStrictZeroCostCandidates,
|
||||
type FreeAccessState,
|
||||
type StrictZeroCostCandidate,
|
||||
} from "../../../open-sse/services/autoCombo/strictZeroCostFilter.ts";
|
||||
import { SYNTHETIC_NOAUTH_CONNECTION_ID } from "../../../open-sse/services/autoCombo/resilienceCandidateFilter.ts";
|
||||
import type { FreeModelBudget } from "../../../open-sse/config/freeModelCatalog.ts";
|
||||
|
||||
const NOW = "2026-08-20T00:00:00.000Z";
|
||||
const OPTIONS = { minRemainingAllowance: 1, maxStateAgeMs: 180_000, now: () => Date.parse(NOW) };
|
||||
|
||||
function safeState(overrides: Partial<FreeAccessState> = {}): FreeAccessState {
|
||||
return {
|
||||
status: "SAFE",
|
||||
remainingFreeAllowance: 40,
|
||||
resetAt: null,
|
||||
checkedAt: NOW,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
// A synthetic keyless budget entry — hardStopGuaranteed deliberately absent,
|
||||
// matching the real catalog (keyless entries never set it; the shortcut is
|
||||
// what makes them safe, not this field).
|
||||
function keylessEntry(): FreeModelBudget {
|
||||
return {
|
||||
provider: "kp",
|
||||
modelId: "kp-model",
|
||||
displayName: "kp-model",
|
||||
monthlyTokens: 0,
|
||||
creditTokens: 0,
|
||||
freeType: "keyless",
|
||||
poolKey: null,
|
||||
tos: "ok",
|
||||
};
|
||||
}
|
||||
|
||||
function quotaEntry(hardStopGuaranteed = true): FreeModelBudget {
|
||||
return {
|
||||
provider: "qp",
|
||||
modelId: "qp-model",
|
||||
displayName: "qp-model",
|
||||
monthlyTokens: 1_000_000,
|
||||
creditTokens: 0,
|
||||
freeType: "recurring-daily",
|
||||
poolKey: null,
|
||||
tos: "ok",
|
||||
hardStopGuaranteed,
|
||||
};
|
||||
}
|
||||
|
||||
const REAL_CONN = "real-connection-42";
|
||||
|
||||
// ─── BLOCKER 1 — keyless bypass ─────────────────────────────────────────
|
||||
|
||||
// A. keyless + SYNTHETIC_NOAUTH_CONNECTION_ID → PASS
|
||||
test("A: keyless + SYNTHETIC_NOAUTH_CONNECTION_ID passes with zero live checks", () => {
|
||||
const candidate: StrictZeroCostCandidate = {
|
||||
provider: "kp",
|
||||
model: "kp-model",
|
||||
connectionId: SYNTHETIC_NOAUTH_CONNECTION_ID,
|
||||
};
|
||||
const resolveFreeAccessState = () => {
|
||||
throw new Error("must never be called for a genuine no-auth candidate");
|
||||
};
|
||||
assert.deepEqual(
|
||||
evaluateCandidateConnections(candidate, keylessEntry(), resolveFreeAccessState, OPTIONS),
|
||||
[SYNTHETIC_NOAUTH_CONNECTION_ID]
|
||||
);
|
||||
});
|
||||
|
||||
// B. same provider/model, catalogued keyless, but reached via a real DB
|
||||
// connectionId → must NOT take the keyless shortcut.
|
||||
test("B: keyless-catalogued model reached via a real connectionId does NOT get the keyless shortcut", () => {
|
||||
const candidate: StrictZeroCostCandidate = {
|
||||
provider: "kp",
|
||||
model: "kp-model",
|
||||
connectionId: REAL_CONN, // NOT the sentinel — a genuine DB connection
|
||||
};
|
||||
let called = false;
|
||||
const resolveFreeAccessState = () => {
|
||||
called = true;
|
||||
return safeState(); // even if the connection WOULD report SAFE quota...
|
||||
};
|
||||
const result = evaluateCandidateConnections(
|
||||
candidate,
|
||||
keylessEntry(),
|
||||
resolveFreeAccessState,
|
||||
OPTIONS
|
||||
);
|
||||
// ...the entry has no hardStopGuaranteed (real keyless catalog rows never
|
||||
// set it — the shortcut was their only path to safety), so falling through
|
||||
// to the quota branch must still exclude it.
|
||||
assert.deepEqual(
|
||||
result,
|
||||
[],
|
||||
"must exclude — keyless metadata alone cannot authorize a real connection"
|
||||
);
|
||||
assert.equal(
|
||||
called,
|
||||
false,
|
||||
"must not even bother resolving live state once hardStopGuaranteed is absent"
|
||||
);
|
||||
});
|
||||
|
||||
// C. a connection that later gains a credential still can't exploit the
|
||||
// keyless classification — same code path as B, demonstrated with a
|
||||
// multi-account (allowedConnectionIds) shape to also prove the "logical
|
||||
// candidate" branch, not just the single-connectionId branch, is covered.
|
||||
test("C: a keyless-catalogued model behind allowedConnectionIds (credentialed accounts) is never shortcut-eligible", () => {
|
||||
const candidate: StrictZeroCostCandidate = {
|
||||
provider: "kp",
|
||||
model: "kp-model",
|
||||
connectionId: null,
|
||||
allowedConnectionIds: ["acct-with-credential-1", "acct-with-credential-2"],
|
||||
};
|
||||
const resolveFreeAccessState = () => safeState(); // both accounts report SAFE quota
|
||||
const result = evaluateCandidateConnections(
|
||||
candidate,
|
||||
keylessEntry(),
|
||||
resolveFreeAccessState,
|
||||
OPTIONS
|
||||
);
|
||||
assert.deepEqual(
|
||||
result,
|
||||
[],
|
||||
"keyless metadata must never authorize any real, credentialed account, regardless of what its quota says"
|
||||
);
|
||||
});
|
||||
|
||||
// Sanity: a non-keyless entry reached via the no-auth sentinel is contradictory
|
||||
// metadata (shouldn't happen in practice) and must fail closed, not silently
|
||||
// fall through to a quota check that can never resolve a state for "noauth".
|
||||
test("sanity: non-keyless freeType via the no-auth sentinel connection fails closed", () => {
|
||||
const candidate: StrictZeroCostCandidate = {
|
||||
provider: "qp",
|
||||
model: "qp-model",
|
||||
connectionId: SYNTHETIC_NOAUTH_CONNECTION_ID,
|
||||
};
|
||||
const result = evaluateCandidateConnections(
|
||||
candidate,
|
||||
quotaEntry(true),
|
||||
() => safeState(),
|
||||
OPTIONS
|
||||
);
|
||||
assert.deepEqual(result, []);
|
||||
});
|
||||
|
||||
// ─── BLOCKER 2 — multi-account connection mismatch ─────────────────────
|
||||
|
||||
// 1. account A SAFE, B UNKNOWN → dispatch can use ONLY A.
|
||||
test("1: A SAFE, B UNKNOWN -> safe connection set is exactly [A]", () => {
|
||||
const candidate: StrictZeroCostCandidate = {
|
||||
provider: "qp",
|
||||
model: "qp-model",
|
||||
connectionId: null,
|
||||
allowedConnectionIds: ["A", "B"],
|
||||
};
|
||||
const resolve = (_provider: string, connectionId: string) =>
|
||||
connectionId === "A" ? safeState() : undefined; // B: no state at all == UNKNOWN
|
||||
assert.deepEqual(evaluateCandidateConnections(candidate, quotaEntry(true), resolve, OPTIONS), [
|
||||
"A",
|
||||
]);
|
||||
});
|
||||
|
||||
// 2. A EXHAUSTED, B SAFE → SOLO B.
|
||||
test("2: A EXHAUSTED, B SAFE -> safe connection set is exactly [B]", () => {
|
||||
const candidate: StrictZeroCostCandidate = {
|
||||
provider: "qp",
|
||||
model: "qp-model",
|
||||
connectionId: null,
|
||||
allowedConnectionIds: ["A", "B"],
|
||||
};
|
||||
const resolve = (_provider: string, connectionId: string) =>
|
||||
connectionId === "A"
|
||||
? safeState({ status: "EXHAUSTED", remainingFreeAllowance: 0 })
|
||||
: safeState();
|
||||
assert.deepEqual(evaluateCandidateConnections(candidate, quotaEntry(true), resolve, OPTIONS), [
|
||||
"B",
|
||||
]);
|
||||
});
|
||||
|
||||
// 3. A SAFE, B billable/unverifiable (UNKNOWN) → B not selectable.
|
||||
test("3: A SAFE, B UNKNOWN (billable/unverifiable) -> B never enters the safe set", () => {
|
||||
const candidate: StrictZeroCostCandidate = {
|
||||
provider: "qp",
|
||||
model: "qp-model",
|
||||
connectionId: null,
|
||||
allowedConnectionIds: ["A", "B"],
|
||||
};
|
||||
const resolve = (_provider: string, connectionId: string) =>
|
||||
connectionId === "A" ? safeState() : undefined;
|
||||
const safe = evaluateCandidateConnections(candidate, quotaEntry(true), resolve, OPTIONS);
|
||||
assert.equal(safe.includes("B"), false);
|
||||
assert.deepEqual(safe, ["A"]);
|
||||
});
|
||||
|
||||
// 4. tutte UNKNOWN → candidato escluso.
|
||||
test("4: all connections UNKNOWN -> candidate fully excluded (empty safe set)", () => {
|
||||
const candidate: StrictZeroCostCandidate = {
|
||||
provider: "qp",
|
||||
model: "qp-model",
|
||||
connectionId: null,
|
||||
allowedConnectionIds: ["A", "B", "C"],
|
||||
};
|
||||
assert.deepEqual(
|
||||
evaluateCandidateConnections(candidate, quotaEntry(true), () => undefined, OPTIONS),
|
||||
[]
|
||||
);
|
||||
});
|
||||
|
||||
// 5. account selezionato al dispatch è uno di quelli verificati SAFE — proven
|
||||
// at the pool-filter level: the returned candidate's `allowedConnectionIds`
|
||||
// is rewritten to exactly the safe subset, which is the same field
|
||||
// `autoStrategy.ts` (open-sse/services/combo/autoStrategy.ts:315-331)
|
||||
// already intersects against before connection selection — so whatever it
|
||||
// picks is provably a member of this set, by construction.
|
||||
test("5: filterStrictZeroCostCandidates rewrites allowedConnectionIds to exactly the verified-SAFE subset", () => {
|
||||
const candidate: StrictZeroCostCandidate = {
|
||||
provider: "qp",
|
||||
model: "qp-model",
|
||||
connectionId: null,
|
||||
allowedConnectionIds: ["A", "B", "C"],
|
||||
};
|
||||
const resolve = (_provider: string, connectionId: string) =>
|
||||
connectionId === "B" ? safeState() : undefined; // only B is SAFE
|
||||
const result = filterStrictZeroCostCandidates([candidate], {
|
||||
enabled: true,
|
||||
resolveFreeAccessState: resolve,
|
||||
catalog: [quotaEntry(true)],
|
||||
...OPTIONS,
|
||||
});
|
||||
assert.equal(result.length, 1);
|
||||
assert.deepEqual(
|
||||
result[0].allowedConnectionIds,
|
||||
["B"],
|
||||
"dispatch (via autoStrategy.ts's own allowedConnectionIds intersection) can only ever select B — the one connection this filter actually verified"
|
||||
);
|
||||
assert.equal(
|
||||
result[0].allowedConnectionIds?.includes("A"),
|
||||
false,
|
||||
"A (UNKNOWN) must never remain selectable"
|
||||
);
|
||||
assert.equal(
|
||||
result[0].allowedConnectionIds?.includes("C"),
|
||||
false,
|
||||
"C (UNKNOWN) must never remain selectable"
|
||||
);
|
||||
});
|
||||
|
||||
test("multi-account candidate with an unchanged safe set is returned as the SAME reference (identity contract)", () => {
|
||||
const candidate: StrictZeroCostCandidate = {
|
||||
provider: "qp",
|
||||
model: "qp-model",
|
||||
connectionId: null,
|
||||
allowedConnectionIds: ["A"],
|
||||
};
|
||||
const pool = [candidate];
|
||||
const result = filterStrictZeroCostCandidates(pool, {
|
||||
enabled: true,
|
||||
resolveFreeAccessState: () => safeState(),
|
||||
catalog: [quotaEntry(true)],
|
||||
...OPTIONS,
|
||||
});
|
||||
assert.equal(result, pool, "nothing was narrowed, so the pool array reference must be preserved");
|
||||
assert.equal(result[0], candidate, "the candidate object itself must be preserved, not cloned");
|
||||
});
|
||||
|
||||
test("single-connection candidate that fails is dropped, never returned with an empty allowedConnectionIds", () => {
|
||||
const candidate: StrictZeroCostCandidate = {
|
||||
provider: "qp",
|
||||
model: "qp-model",
|
||||
connectionId: REAL_CONN,
|
||||
};
|
||||
const result = filterStrictZeroCostCandidates([candidate], {
|
||||
enabled: true,
|
||||
resolveFreeAccessState: () => undefined,
|
||||
catalog: [quotaEntry(true)],
|
||||
...OPTIONS,
|
||||
});
|
||||
assert.deepEqual(result, []);
|
||||
});
|
||||
248
tests/unit/autoCombo/strict-zero-cost-filter.test.ts
Normal file
248
tests/unit/autoCombo/strict-zero-cost-filter.test.ts
Normal file
@@ -0,0 +1,248 @@
|
||||
/**
|
||||
* STRICT_ZERO_COST — regression guard for `strictZeroCostFilter.ts`, wired
|
||||
* into `open-sse/services/autoCombo/virtualFactory.ts::buildPreparedPool`
|
||||
* right after `filterPaidOnlyCandidates` (#6512). Mirrors the style/shape of
|
||||
* `tests/unit/autoCombo/paid-model-filter-6512.test.ts`.
|
||||
*
|
||||
* Pure, dependency-light by design: `resolveFreeAccessState` is injected as a
|
||||
* plain function, so no DB/network mocking is needed anywhere in this file.
|
||||
*
|
||||
* Connection-safety-specific cases (the keyless bypass and multi-account
|
||||
* bugs found in code review) live in their own file,
|
||||
* `strict-zero-cost-connection-safety.test.ts` — this file covers the
|
||||
* single-connection / no-connection-ambiguity cases only.
|
||||
*/
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import {
|
||||
evaluateCandidateConnections,
|
||||
filterStrictZeroCostCandidates,
|
||||
filterTosAvoidCandidates,
|
||||
type FreeAccessState,
|
||||
} from "../../../open-sse/services/autoCombo/strictZeroCostFilter.ts";
|
||||
import { SYNTHETIC_NOAUTH_CONNECTION_ID } from "../../../open-sse/services/autoCombo/resilienceCandidateFilter.ts";
|
||||
import { FREE_MODEL_BUDGETS } from "../../../open-sse/config/freeModelCatalog.ts";
|
||||
|
||||
const NOW = "2026-08-20T00:00:00.000Z";
|
||||
const nowMs = () => Date.parse(NOW);
|
||||
|
||||
function freshState(overrides: Partial<FreeAccessState> = {}): FreeAccessState {
|
||||
return {
|
||||
status: "SAFE",
|
||||
remainingFreeAllowance: 50,
|
||||
resetAt: null,
|
||||
checkedAt: NOW,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
const BASE_OPTIONS = { minRemainingAllowance: 1, maxStateAgeMs: 180_000, now: nowMs };
|
||||
|
||||
const REAL_CONN = "conn-real-1";
|
||||
|
||||
// A real keyless entry from the catalog (felo-web, all models keyless/tos=avoid),
|
||||
// as a genuine no-auth candidate (the only shape that legitimately gets the shortcut).
|
||||
const KEYLESS = {
|
||||
provider: "felo-web",
|
||||
model: "felo-chat",
|
||||
connectionId: SYNTHETIC_NOAUTH_CONNECTION_ID,
|
||||
};
|
||||
// A real quota-based entry with hardStopGuaranteed: true (added by this feature),
|
||||
// as a concrete single-connection candidate.
|
||||
const QUOTA_SAFE = { provider: "groq", model: "llama-3.3-70b-versatile", connectionId: REAL_CONN };
|
||||
// A real quota-based entry WITHOUT hardStopGuaranteed (agentrouter: one-time-initial,
|
||||
// no usage adapter, no documented "no credit card" claim — must never pass).
|
||||
const QUOTA_UNGUARANTEED = {
|
||||
provider: "agentrouter",
|
||||
model: "claude-opus-5",
|
||||
connectionId: REAL_CONN,
|
||||
};
|
||||
// Not in FREE_MODEL_BUDGETS at all.
|
||||
const UNKNOWN_MODEL = {
|
||||
provider: "groq",
|
||||
model: "definitely-not-cataloged",
|
||||
connectionId: REAL_CONN,
|
||||
};
|
||||
// A provider with no free models documented anywhere (mirrors paid-model-filter-6512's own PAID fixture).
|
||||
const PAID = { provider: "openai", model: "gpt-4o", connectionId: REAL_CONN };
|
||||
|
||||
test("sanity: fixtures exist in the real catalog with the metadata these tests assume", () => {
|
||||
const groqEntry = FREE_MODEL_BUDGETS.find(
|
||||
(m) => m.provider === "groq" && m.modelId === "llama-3.3-70b-versatile"
|
||||
);
|
||||
assert.equal(groqEntry?.hardStopGuaranteed, true, "groq must carry hardStopGuaranteed: true");
|
||||
const arEntry = FREE_MODEL_BUDGETS.find(
|
||||
(m) => m.provider === "agentrouter" && m.modelId === "claude-opus-5"
|
||||
);
|
||||
assert.notEqual(
|
||||
arEntry?.hardStopGuaranteed,
|
||||
true,
|
||||
"agentrouter must NOT carry hardStopGuaranteed: true (no documented hard-stop guarantee)"
|
||||
);
|
||||
const feloEntry = FREE_MODEL_BUDGETS.find(
|
||||
(m) => m.provider === "felo-web" && m.modelId === "felo-chat"
|
||||
);
|
||||
assert.equal(feloEntry?.freeType, "keyless");
|
||||
assert.equal(feloEntry?.tos, "avoid", "felo-web must be tos=avoid for the ToS-guard tests below");
|
||||
});
|
||||
|
||||
// 1. keyless SAFE (genuine no-auth candidate) → PASS
|
||||
test("keyless candidate from the genuine no-auth path passes with no state at all", () => {
|
||||
const entry = FREE_MODEL_BUDGETS.find(
|
||||
(m) => m.provider === "felo-web" && m.modelId === "felo-chat"
|
||||
);
|
||||
assert.deepEqual(
|
||||
evaluateCandidateConnections(KEYLESS, entry, () => undefined, BASE_OPTIONS),
|
||||
[SYNTHETIC_NOAUTH_CONNECTION_ID]
|
||||
);
|
||||
});
|
||||
|
||||
// 2. paid → EXCLUDE
|
||||
test("paid candidate (no free-model documentation at all) is excluded", () => {
|
||||
assert.deepEqual(
|
||||
evaluateCandidateConnections(PAID, undefined, () => undefined, BASE_OPTIONS),
|
||||
[]
|
||||
);
|
||||
});
|
||||
|
||||
// 3 & 4. unknown model / unknown provider → EXCLUDE
|
||||
test("model absent from the free catalog is excluded even under a known provider", () => {
|
||||
assert.deepEqual(
|
||||
evaluateCandidateConnections(UNKNOWN_MODEL, undefined, () => undefined, BASE_OPTIONS),
|
||||
[]
|
||||
);
|
||||
});
|
||||
|
||||
// 5. quota SAFE + fresh + hardStop → PASS
|
||||
test("quota-based candidate with hardStopGuaranteed, fresh SAFE state above threshold passes", () => {
|
||||
const entry = FREE_MODEL_BUDGETS.find(
|
||||
(m) => m.provider === "groq" && m.modelId === "llama-3.3-70b-versatile"
|
||||
);
|
||||
assert.deepEqual(
|
||||
evaluateCandidateConnections(
|
||||
QUOTA_SAFE,
|
||||
entry,
|
||||
() => freshState({ remainingFreeAllowance: 40 }),
|
||||
BASE_OPTIONS
|
||||
),
|
||||
[REAL_CONN]
|
||||
);
|
||||
});
|
||||
|
||||
// 6. quota exhausted → EXCLUDE
|
||||
test("EXHAUSTED status excludes even with a fresh checkedAt", () => {
|
||||
const entry = FREE_MODEL_BUDGETS.find(
|
||||
(m) => m.provider === "groq" && m.modelId === "llama-3.3-70b-versatile"
|
||||
);
|
||||
const state = freshState({ status: "EXHAUSTED", remainingFreeAllowance: 0 });
|
||||
assert.deepEqual(
|
||||
evaluateCandidateConnections(QUOTA_SAFE, entry, () => state, BASE_OPTIONS),
|
||||
[]
|
||||
);
|
||||
});
|
||||
|
||||
// 7. usage adapter absent (no state resolvable) → EXCLUDE
|
||||
test("quota-based candidate with no resolvable state is excluded, not assumed safe", () => {
|
||||
const entry = FREE_MODEL_BUDGETS.find(
|
||||
(m) => m.provider === "groq" && m.modelId === "llama-3.3-70b-versatile"
|
||||
);
|
||||
assert.deepEqual(
|
||||
evaluateCandidateConnections(QUOTA_SAFE, entry, () => undefined, BASE_OPTIONS),
|
||||
[]
|
||||
);
|
||||
});
|
||||
|
||||
// 8. usage API error → EXCLUDE (modeled as UNKNOWN status; freeAccessQuota.ts
|
||||
// deletes the cache entry on a failed refresh, which resolves to `undefined`
|
||||
// here — same assertion as #7, the important contract is "never falls back to SAFE").
|
||||
test("UNKNOWN status excludes", () => {
|
||||
const entry = FREE_MODEL_BUDGETS.find(
|
||||
(m) => m.provider === "groq" && m.modelId === "llama-3.3-70b-versatile"
|
||||
);
|
||||
const state = freshState({ status: "UNKNOWN", remainingFreeAllowance: null });
|
||||
assert.deepEqual(
|
||||
evaluateCandidateConnections(QUOTA_SAFE, entry, () => state, BASE_OPTIONS),
|
||||
[]
|
||||
);
|
||||
});
|
||||
|
||||
// 9. usage state stale → EXCLUDE
|
||||
test("stale checkedAt excludes even when status is SAFE", () => {
|
||||
const entry = FREE_MODEL_BUDGETS.find(
|
||||
(m) => m.provider === "groq" && m.modelId === "llama-3.3-70b-versatile"
|
||||
);
|
||||
const stale = freshState({ checkedAt: "2026-08-19T00:00:00.000Z" }); // >24h before NOW
|
||||
assert.deepEqual(
|
||||
evaluateCandidateConnections(QUOTA_SAFE, entry, () => stale, BASE_OPTIONS),
|
||||
[]
|
||||
);
|
||||
});
|
||||
|
||||
// 10 & 11. hardStopGuaranteed undefined / false → EXCLUDE
|
||||
test("hardStopGuaranteed undefined excludes even with a perfect SAFE state", () => {
|
||||
const entry = FREE_MODEL_BUDGETS.find(
|
||||
(m) => m.provider === "agentrouter" && m.modelId === "claude-opus-5"
|
||||
);
|
||||
assert.deepEqual(
|
||||
evaluateCandidateConnections(QUOTA_UNGUARANTEED, entry, () => freshState(), BASE_OPTIONS),
|
||||
[]
|
||||
);
|
||||
});
|
||||
|
||||
test("hardStopGuaranteed explicitly false excludes", () => {
|
||||
const entry = {
|
||||
...FREE_MODEL_BUDGETS[0],
|
||||
hardStopGuaranteed: false,
|
||||
freeType: "recurring-monthly" as const,
|
||||
};
|
||||
assert.deepEqual(
|
||||
evaluateCandidateConnections(
|
||||
{ provider: entry.provider, model: entry.modelId, connectionId: REAL_CONN },
|
||||
entry,
|
||||
() => freshState(),
|
||||
BASE_OPTIONS
|
||||
),
|
||||
[]
|
||||
);
|
||||
});
|
||||
|
||||
// 12 & 13. ToS guard, independent of economic evaluation
|
||||
test("tos=avoid + excludeTosAvoid=true excludes a keyless-safe candidate", () => {
|
||||
const result = filterTosAvoidCandidates([KEYLESS], true);
|
||||
assert.deepEqual(result, [], "felo-web (tos=avoid) must be dropped when the guard is on");
|
||||
});
|
||||
|
||||
test("tos=avoid + excludeTosAvoid=false leaves normal economic evaluation untouched", () => {
|
||||
const result = filterTosAvoidCandidates([KEYLESS, PAID], false);
|
||||
assert.deepEqual(result, [KEYLESS, PAID], "guard off must be a pure identity pass-through");
|
||||
});
|
||||
|
||||
// Pool-level filter: default-off regression guard, mirrors paid-model-filter-6512's own first test.
|
||||
test("freeAccessPolicy off (default) returns the pool UNCHANGED (identity, regression guard)", () => {
|
||||
const pool = [KEYLESS, QUOTA_SAFE, PAID];
|
||||
const result = filterStrictZeroCostCandidates(pool, {
|
||||
enabled: false,
|
||||
resolveFreeAccessState: () => undefined,
|
||||
...BASE_OPTIONS,
|
||||
});
|
||||
assert.equal(result, pool, "must return the exact same array reference when opt-in is off");
|
||||
});
|
||||
|
||||
test("strict pool filter keeps only keyless(no-auth) + guaranteed-quota-SAFE candidates", () => {
|
||||
const pool = [KEYLESS, QUOTA_SAFE, QUOTA_UNGUARANTEED, PAID, UNKNOWN_MODEL];
|
||||
const result = filterStrictZeroCostCandidates(pool, {
|
||||
enabled: true,
|
||||
resolveFreeAccessState: (provider) =>
|
||||
provider === "groq" ? freshState({ remainingFreeAllowance: 40 }) : undefined,
|
||||
...BASE_OPTIONS,
|
||||
});
|
||||
assert.deepEqual(result, [KEYLESS, QUOTA_SAFE]);
|
||||
});
|
||||
|
||||
// Autodiscovery cases (14-19) are in
|
||||
// tests/unit/autoCombo/strict-zero-cost-autodiscovery.test.ts, which fabricates
|
||||
// synthetic FREE_MODEL_BUDGETS-shaped fixtures rather than real provider ids —
|
||||
// see that file for why a fixture-based test is the correct tool here.
|
||||
// Blocker 1 (keyless bypass) and Blocker 2 (multi-account) cases are in
|
||||
// tests/unit/autoCombo/strict-zero-cost-connection-safety.test.ts.
|
||||
@@ -74,3 +74,53 @@ test("MiniMax M3 via command-code keeps existing vision capability (no regressio
|
||||
const caps = getResolvedModelCapabilities("command-code/MiniMaxAI/MiniMax-M3");
|
||||
assert.equal(caps.supportsVision, true);
|
||||
});
|
||||
|
||||
test("command-code effort-suffixed variants resolve capabilities from their base model", async () => {
|
||||
// Effort variants (e.g. `-max`, `-xhigh`) are synthesized from the base's
|
||||
// supportedThinkingEfforts and have no registry/synced row of their own.
|
||||
// Without the base-model fallback they resolve NULL tool/vision/context,
|
||||
// which makes a tool-bearing combo request drop them behind confirmed
|
||||
// targets (observed: orchestrator tried opencode-go/mimo-v2.5-max at
|
||||
// position 2 while command-code deepseek sat unused at its declared
|
||||
// priority position 2).
|
||||
//
|
||||
// Seed the models.dev capability store (the source getResolvedModelCapabilities
|
||||
// reads for tool/vision/context) with the base models, then verify the
|
||||
// effort-suffixed variants inherit those capabilities via the base fallback.
|
||||
const modelsDevSync = await import("../../src/lib/modelsDevSync.ts");
|
||||
modelsDevSync.saveModelsDevCapabilities({
|
||||
"command-code": {
|
||||
"deepseek/deepseek-v4-flash": {
|
||||
tool_call: true,
|
||||
reasoning: true,
|
||||
attachment: false,
|
||||
limit_context: 1000000,
|
||||
limit_input: 1000000,
|
||||
limit_output: 131072,
|
||||
},
|
||||
"meta/muse-spark-1.2-contributor": {
|
||||
tool_call: true,
|
||||
reasoning: true,
|
||||
attachment: true,
|
||||
limit_context: 1048576,
|
||||
limit_input: 1048576,
|
||||
limit_output: 1048576,
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
const cases: Array<[string, boolean]> = [
|
||||
["command-code/deepseek/deepseek-v4-flash-max", false], // text-only base
|
||||
["command-code/deepseek/deepseek-v4-flash-high", false],
|
||||
["command-code/meta/muse-spark-1.2-contributor-xhigh", true], // vision base
|
||||
["command-code/meta/muse-spark-1.2-contributor-high", true],
|
||||
];
|
||||
for (const [modelId, vision] of cases) {
|
||||
const caps = getResolvedModelCapabilities(modelId);
|
||||
assert.equal(caps.provider, "command-code", `${modelId} provider`);
|
||||
assert.equal(caps.supportsTools, true, `${modelId} must inherit tool support from base`);
|
||||
assert.equal(caps.supportsVision, vision, `${modelId} must inherit vision from base`);
|
||||
assert.equal(typeof caps.contextWindow, "number", `${modelId} must inherit a context window`);
|
||||
assert.equal(caps.supportsThinking, true, `${modelId} must inherit reasoning from base`);
|
||||
}
|
||||
});
|
||||
|
||||
68
tests/unit/opencode-merge-provider-guard.test.ts
Normal file
68
tests/unit/opencode-merge-provider-guard.test.ts
Normal file
@@ -0,0 +1,68 @@
|
||||
// `mergeOpenCodeConfig` guarded the root against a non-object but not the
|
||||
// `provider` branch it spreads one level down. Spreading a non-object does not
|
||||
// throw, it splays the value into index keys, so an existing config with a
|
||||
// malformed `provider` was rewritten into nonsense instead of being rejected:
|
||||
//
|
||||
// provider: ["a", "b"] -> { "0": "a", "1": "b", omniroute: {...} }
|
||||
// provider: "oops" -> { "0": "o", "1": "o", "2": "p", "3": "s", ... }
|
||||
//
|
||||
// Its sibling `mergeOpenCodeConfigText` throws on the same input
|
||||
// ("Can not add index to parent of type array"), so the two disagreed on what a
|
||||
// malformed config means.
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
const { mergeOpenCodeConfig, mergeOpenCodeConfigText } =
|
||||
await import("../../src/shared/services/opencodeConfig.ts");
|
||||
|
||||
const INPUT = {
|
||||
baseUrl: "http://localhost:20128/v1",
|
||||
apiKey: "sk_test_opencode",
|
||||
model: "claude-sonnet-4-5-thinking",
|
||||
};
|
||||
|
||||
for (const [label, provider] of [
|
||||
["an array", ["a", "b"]],
|
||||
["a string", "oops"],
|
||||
["a number", 7],
|
||||
["null", null],
|
||||
] as const) {
|
||||
test(`mergeOpenCodeConfig drops a provider block that is ${label}`, () => {
|
||||
const merged = mergeOpenCodeConfig({ provider } as never, INPUT);
|
||||
|
||||
assert.deepEqual(
|
||||
Object.keys(merged.provider),
|
||||
["omniroute"],
|
||||
`a provider block that is ${label} must not be splayed into index keys`
|
||||
);
|
||||
assert.equal(merged.provider.omniroute.options.baseURL, "http://localhost:20128/v1");
|
||||
});
|
||||
}
|
||||
|
||||
test("mergeOpenCodeConfig still preserves sibling providers", () => {
|
||||
// The guard must only reject non-objects, never a legitimate provider map.
|
||||
const merged = mergeOpenCodeConfig(
|
||||
{ provider: { custom: { name: "Custom Provider" }, other: { name: "Other" } } } as never,
|
||||
INPUT
|
||||
);
|
||||
|
||||
assert.deepEqual(Object.keys(merged.provider).sort(), ["custom", "omniroute", "other"]);
|
||||
assert.equal(merged.provider.custom.name, "Custom Provider");
|
||||
});
|
||||
|
||||
test("mergeOpenCodeConfig still guards the root itself", () => {
|
||||
for (const existing of [["a"], "oops", 7, null, undefined]) {
|
||||
const merged = mergeOpenCodeConfig(existing as never, INPUT);
|
||||
assert.deepEqual(Object.keys(merged.provider), ["omniroute"]);
|
||||
assert.equal(merged.$schema, "https://opencode.ai/config.json");
|
||||
}
|
||||
});
|
||||
|
||||
test("mergeOpenCodeConfigText keeps refusing the same malformed input", () => {
|
||||
// Pinning the sibling's behaviour: the object variant now drops the bad
|
||||
// block, the text variant still refuses to touch the file. Both are safe;
|
||||
// neither corrupts.
|
||||
for (const text of ['{"provider": ["a","b"]}', '{"provider": "oops"}']) {
|
||||
assert.throws(() => mergeOpenCodeConfigText(text, INPUT));
|
||||
}
|
||||
});
|
||||
@@ -53,3 +53,27 @@ test("stripGroqUnsupportedFields is immutable (does not mutate input)", () => {
|
||||
assert.equal(input.messages[0].name, "bob");
|
||||
assert.equal(input.logprobs, true);
|
||||
});
|
||||
|
||||
test("stripGroqUnsupportedFields drops unsupported messages[].model and other metadata while keeping role and content", () => {
|
||||
const out = stripGroqUnsupportedFields({
|
||||
messages: [
|
||||
{ role: "user", content: "hello" },
|
||||
{
|
||||
role: "assistant",
|
||||
content: "hello!",
|
||||
model: "groq/openai/gpt-oss-20b",
|
||||
messageId: "msg_123",
|
||||
sender: "assistant",
|
||||
},
|
||||
],
|
||||
});
|
||||
assert.equal(out.messages.length, 2);
|
||||
assert.equal(out.messages[0].role, "user");
|
||||
assert.equal(out.messages[0].content, "hello");
|
||||
assert.equal(out.messages[1].role, "assistant");
|
||||
assert.equal(out.messages[1].content, "hello!");
|
||||
assert.equal("model" in out.messages[1], false);
|
||||
assert.equal("messageId" in out.messages[1], false);
|
||||
assert.equal("sender" in out.messages[1], false);
|
||||
});
|
||||
|
||||
|
||||
116
tests/unit/silent-sse-close-responses-no-terminal.test.ts
Normal file
116
tests/unit/silent-sse-close-responses-no-terminal.test.ts
Normal file
@@ -0,0 +1,116 @@
|
||||
/**
|
||||
* Regression test — silent SSE close on OpenAI Responses-format clients.
|
||||
*
|
||||
* Companion to #10443 (OpenAI chat completions) and #7699 (Claude). A healthy
|
||||
* OpenAI Responses stream ALWAYS terminates with an explicit
|
||||
* `response.completed` event; it is the format's only terminal marker and
|
||||
* carries the final status/usage. When the upstream forwards content deltas
|
||||
* and then closes without that event, Responses-format clients (Codex CLI and
|
||||
* other /v1/responses consumers) previously received a silent mid-stream
|
||||
* close — indistinguishable from a healthy end, so the client waits on a
|
||||
* completion event that never arrives. The close must surface a synthetic
|
||||
* `response.failed` instead, keeping everything already forwarded.
|
||||
*/
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
const { createDisconnectAwareStream, createStreamController } =
|
||||
await import("../../open-sse/utils/streamHandler.ts");
|
||||
const { FORMATS } = await import("../../open-sse/translator/formats.ts");
|
||||
|
||||
function createNoopAbortWritableStream(): { getWriter: () => { abort: () => Promise<void> } } {
|
||||
return { getWriter: () => ({ abort: () => Promise.resolve() }) };
|
||||
}
|
||||
|
||||
async function drainStream(stream: ReadableStream<Uint8Array>): Promise<string> {
|
||||
const reader = stream.getReader();
|
||||
const parts: Uint8Array[] = [];
|
||||
for (;;) {
|
||||
const { done, value } = await reader.read();
|
||||
if (done) break;
|
||||
parts.push(value);
|
||||
}
|
||||
return new TextDecoder().decode(
|
||||
parts.reduce((acc, c) => {
|
||||
const merged = new Uint8Array(acc.length + c.length);
|
||||
merged.set(acc, 0);
|
||||
merged.set(c, acc.length);
|
||||
return merged;
|
||||
}, new Uint8Array(0))
|
||||
);
|
||||
}
|
||||
|
||||
async function runClientStream(
|
||||
upstreamChunks: string[],
|
||||
clientResponseFormat: string | null
|
||||
): Promise<string> {
|
||||
const upstream = new ReadableStream<Uint8Array>({
|
||||
start(controller) {
|
||||
const encoder = new TextEncoder();
|
||||
for (const chunk of upstreamChunks) controller.enqueue(encoder.encode(chunk));
|
||||
controller.close();
|
||||
},
|
||||
});
|
||||
|
||||
const transform = new TransformStream<Uint8Array, Uint8Array>({
|
||||
transform(chunk, controller) {
|
||||
controller.enqueue(chunk);
|
||||
},
|
||||
});
|
||||
const transformedBody = upstream.pipeThrough(transform);
|
||||
|
||||
const sc = createStreamController({
|
||||
provider: "opencode-go",
|
||||
model: "muse-spark-1.2-contributor",
|
||||
clientResponseFormat,
|
||||
});
|
||||
|
||||
const wrapped = createDisconnectAwareStream(
|
||||
{ readable: transformedBody, writable: createNoopAbortWritableStream() },
|
||||
sc
|
||||
);
|
||||
|
||||
return drainStream(wrapped);
|
||||
}
|
||||
|
||||
test("Responses format: content then bare close emits response.failed, not a silent close", async () => {
|
||||
const text = await runClientStream(
|
||||
[
|
||||
'event: response.output_text.delta\ndata: {"type":"response.output_text.delta","delta":"partial"}\n\n',
|
||||
],
|
||||
FORMATS.OPENAI_RESPONSES
|
||||
);
|
||||
|
||||
// The forwarded content survives...
|
||||
assert.match(text, /partial/);
|
||||
// ...and the close must be flagged with the format's failure terminal.
|
||||
assert.match(text, /response\.failed/);
|
||||
assert.match(text, /Upstream stream ended without a terminal marker/);
|
||||
});
|
||||
|
||||
test("Responses format: response.completed counts as terminal, no synthetic failure", async () => {
|
||||
const text = await runClientStream(
|
||||
[
|
||||
'event: response.output_text.delta\ndata: {"type":"response.output_text.delta","delta":"done"}\n\n',
|
||||
'event: response.completed\ndata: {"type":"response.completed","response":{"status":"completed"}}\n\n',
|
||||
],
|
||||
FORMATS.OPENAI_RESPONSES
|
||||
);
|
||||
|
||||
assert.match(text, /done/);
|
||||
assert.match(text, /response\.completed/);
|
||||
assert.doesNotMatch(text, /response\.failed/);
|
||||
assert.doesNotMatch(text, /Upstream stream ended without a terminal marker/);
|
||||
});
|
||||
|
||||
test("Responses format: OPENAI_RESPONSE alias gets the same bare-close verdict", async () => {
|
||||
const text = await runClientStream(
|
||||
[
|
||||
'event: response.output_text.delta\ndata: {"type":"response.output_text.delta","delta":"partial"}\n\n',
|
||||
],
|
||||
FORMATS.OPENAI_RESPONSE
|
||||
);
|
||||
|
||||
assert.match(text, /partial/);
|
||||
assert.match(text, /response\.failed/);
|
||||
});
|
||||
Reference in New Issue
Block a user