diff --git a/CHANGELOG.md b/CHANGELOG.md index 0831404938..05d3989c59 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,18 @@ ## [Unreleased] +### ✨ New Features + +- **feat(sse): STRICT_ZERO_COST** — opt-in, off-by-default `freeAccessPolicy: "strict"` setting + that hard-verifies every auto-combo candidate against live quota state and per-connection + economic safety before it can be dispatched, going beyond `hidePaidModels`'s static catalog + check. Adds curated `hardStopGuaranteed` metadata to `FREE_MODEL_BUDGETS`, a short-TTL quota + cache reusing `getUsageForProvider()`, and a connection-safety guarantee: a candidate backed + by multiple accounts has its `allowedConnectionIds` narrowed to exactly the connections + independently verified `SAFE`, so dispatch can never use an unverified account. An + `excludeTosAvoid` guard (default `false`) is available separately for contractual risk. See + `docs/routing/STRICT_ZERO_COST.md`. + --- ## [3.8.50] — TBD diff --git a/docs/routing/STRICT_ZERO_COST.md b/docs/routing/STRICT_ZERO_COST.md new file mode 100644 index 0000000000..50f15778b6 --- /dev/null +++ b/docs/routing/STRICT_ZERO_COST.md @@ -0,0 +1,146 @@ +--- +title: "STRICT_ZERO_COST" +version: 3.8.50 +lastUpdated: 2026-08-20 +--- + +# STRICT_ZERO_COST + +> Opt-in, off by default (`settings.freeAccessPolicy !== "strict"` leaves every `auto/*` +> candidate pool byte-identical). A stricter sibling of `hidePaidModels` +> (`open-sse/services/autoCombo/paidModelFilter.ts`, #6512) for operators who need a hard +> guarantee against ANY incremental monetary spend, not just "documented as free". + +## Why this exists, and why `hidePaidModels` alone isn't enough + +`hidePaidModels` answers "is this model classified free in `FREE_MODEL_BUDGETS` right now?" — +a point-in-time catalog fact, checked via `isFreeModel()`/`providerHasFreeModels()` +(`src/shared/utils/freeModels.ts`). It says nothing about two real risks: + +1. A `recurring-*`/`one-time-initial` free tier's allowance can be **exhausted** — the catalog + still lists the model as free, but the account behind it has no headroom left. +2. Exceeding a free tier is not always a hard stop. Some providers document explicitly that no + payment method can ever be attached ("no credit card required"); others don't say, and a + handful bill automatically past the free allowance. + +`hidePaidModels` cannot distinguish these — it was never meant to. STRICT_ZERO_COST adds exactly +these two checks, evaluated per candidate, **before** category/tier ranking and **before** +dispatch — never after a request has already gone out. + +## Candidate classification + +For every candidate in the pool (`open-sse/services/autoCombo/virtualFactory.ts::buildPreparedPool`, +right after `filterPaidOnlyCandidates`): + +1. **Not in `FREE_MODEL_BUDGETS` at all** → excluded. This covers genuinely paid models and any + provider/model OmniRoute hasn't classified yet — new candidates start excluded, not included. +2. **`freeType: "keyless"`** → passes immediately, **but only for a candidate that genuinely + arrived via the no-auth path** (`connectionId === SYNTHETIC_NOAUTH_CONNECTION_ID`, + `open-sse/services/autoCombo/resilienceCandidateFilter.ts`). No credential exists for that + candidate, so no request against it can ever be billed — no runtime check is needed or + possible. The same catalogued `keyless` provider/model reached through a **real** DB + connection (`connectionId` is an actual connection id, or the candidate carries + `allowedConnectionIds`) does **not** get this shortcut — `keyless` metadata describes the + no-auth path specifically, not the provider in general, and never authorizes a real, + credentialed account. Such a candidate falls through to check 3 like any other, where it is + excluded unless the catalog entry separately carries `hardStopGuaranteed: true` (real + `keyless` entries never do — the shortcut was their only path to safety). +3. **Any other `freeType`** (`recurring-daily`, `recurring-monthly`, `recurring-credit`, + `recurring-uncapped`, `one-time-initial`, and any future type this module doesn't + special-case) → passes only if **all** of the following hold: + - `hardStopGuaranteed: true` is set on the catalog entry (`FreeModelBudget.hardStopGuaranteed`, + `open-sse/config/freeModelCatalog.ts`) — a **curated, hand-set fact** about the provider's + own published terms (e.g. an explicit "no credit card required" claim), never derived from + `freeType` or from a live API response. Unset (`undefined`) and `false` are both treated as + "not guaranteed". + - A usage adapter exists for the provider in `USAGE_FETCHER_PROVIDERS` + (`open-sse/services/usage.ts`) — the same registry that already backs the quota dashboard and + `getUsageForProvider()`. No adapter → excluded, permanently, until one is added. + - The live, cached `FreeAccessState` for **the specific connection actually being + evaluated** is `status: "SAFE"`, was checked within + `settings.autoRefreshProviderQuotaInterval` (default 180s — the existing setting, not a new + number), and reports `remainingFreeAllowance` above a small safety margin. +4. **`freeType: "discontinued"`** → always excluded. + +## Connection safety (per-connection verification, never per-candidate) + +A candidate in the auto-combo pool is not always tied to one connection. A "logical" candidate +(`connectionId: null`) carries an `allowedConnectionIds` allowlist — one or more actual +provider connections/accounts any of which could serve the request — and the account actually +used is decided later, at dispatch time, by `open-sse/services/combo/autoStrategy.ts` +(intersecting `allowedConnectionIds` against its own connection-selection logic, ~line 315-331). + +STRICT_ZERO_COST verifies the free-access state of **each connection in that allowlist +individually** (`evaluateCandidateConnections()` in `strictZeroCostFilter.ts`) and rewrites +`allowedConnectionIds` down to exactly the subset that came back `SAFE` — never the full +original list, and never a single arbitrarily-chosen member. Concretely: + +- Account A `SAFE`, account B `UNKNOWN`/exhausted/billable → only A remains selectable. +- All accounts `UNKNOWN` → the candidate is dropped entirely (empty safe set). +- A single-connection candidate (`connectionId` set directly, no allowlist) that fails is + dropped outright, never returned with an empty `allowedConnectionIds`. + +Because `autoStrategy.ts` already enforces `allowedConnectionIds` as a hard allowlist before +selecting a connection to dispatch to, rewriting it to the verified-SAFE subset is sufficient to +guarantee the connection actually used at dispatch is always one this filter itself verified — +never a different, unverified account on the same candidate. See +`tests/unit/autoCombo/strict-zero-cost-connection-safety.test.ts` for the regression proof +(keyless-bypass cases A/B/C, multi-account cases 1-5). + +`discovered automatically`: a provider/model shipped tomorrow with the right metadata (in the +catalog, with a usage adapter, `hardStopGuaranteed: true`) is usable the moment OmniRoute knows +about it — no code change, no whitelist entry, nothing to edit in this module. One removed from +the catalog disappears the same way. See +`tests/unit/autoCombo/strict-zero-cost-autodiscovery.test.ts` for the regression proof (via +injectable fixtures, not by mutating the real catalog). + +## Quota caching (`open-sse/services/autoCombo/freeAccessQuota.ts`) + +Reuses `getUsageForProvider()` — no second quota system. A short, in-memory, +process-lifetime cache sits in front of it (TTL equal to the default +`autoRefreshProviderQuotaInterval`) so a Telegram-scale request rate never triggers a live +billing-API call per candidate per request. Reads are synchronous: a cache miss returns +`undefined` (→ excluded, fail-closed) and kicks off a background refresh for the _next_ read — +nothing in the candidate-pool build path ever awaits a network call. + +`invalidateFreeAccessState(provider, connectionId)` is called from +`src/sse/services/auth.ts::markAccountUnavailable()` the moment a connection fails for any +reason, so the very next pool build reads a clean cache miss instead of a stale `SAFE` entry — +no waiting out the TTL after a 402/403/quota-exhausted response. + +## ToS guard (independent of economic safety) + +`excludeTosAvoid` (default `false`) drops any candidate whose curated `tos` verdict +(`FreeModelBudget.tos`) is `"avoid"` — reuses the same field `hidePaidModels`'s sibling docs +(`docs/reference/FREE_TIERS.md`) already populate. Deliberately separate from +`freeAccessPolicy`: a candidate can be economically `SAFE` and still excluded here for +contractual reasons, or left in when this guard is off even with `freeAccessPolicy: "strict"` on. + +## What passes today + +Run `npx tsx scripts/ad-hoc/dry-run-strict-zero-cost.ts` against a live instance's +`GET /v1/auto-combo/{channel}/candidates` output for a real before/after — the script now reads +each candidate's real `connectionId`, so it also proves the connection-safety fix live, not just +in unit tests. As of 2026-08-20, only `freeType: "keyless"` candidates pass in practice (7 of 29 +live candidates on this instance: `opencode/big-pickle`, `opencode/deepseek-v4-flash-free`, and +5 `felo-web` models — all confirmed arriving with the genuine no-auth `connectionId`, never a +real connection) — no currently-catalogued `recurring-*` provider both has a usage adapter +registered in `USAGE_FETCHER_PROVIDERS` **and** `hardStopGuaranteed: true` declared (e.g. `groq` +has neither the adapter registered here nor is fetched offline in this dry run; `kiro` lacks +`hardStopGuaranteed`). This is not a bug: it's the honest state of two independently-curated +metadata sets that happen not to overlap yet, not a limitation of the filter itself. + +With `excludeTosAvoid: true` added on top of the same live pool, the count drops from 7 to 0 — +every one of the 7 surviving candidates is curated `tos: "avoid"` today (`felo-web`, `opencode`). +This is a real, expected trade-off of turning the ToS guard on, not a bug: the guard is +`false` by default for exactly this reason (see "ToS guard" above). + +## Enabling + +```json +PUT /api/settings +{ "freeAccessPolicy": "strict", "excludeTosAvoid": false } +``` + +Both new settings default to their pre-feature values (`"off"` / `false`) — enabling neither +changes any existing `auto/*` routing behavior. diff --git a/open-sse/config/freeModelCatalog.data.ts b/open-sse/config/freeModelCatalog.data.ts index 8236a4d5d8..e6ef081aa7 100644 --- a/open-sse/config/freeModelCatalog.data.ts +++ b/open-sse/config/freeModelCatalog.data.ts @@ -108,8 +108,9 @@ export const FREE_MODEL_BUDGETS: FreeModelBudget[] = [ { provider: "bytez", modelId: "meta-llama/Llama-3.3-70B-Instruct", displayName: "meta-llama/Llama-3.3-70B-Instruct", monthlyTokens: 0, creditTokens: 1000000, freeType: "recurring-credit", poolKey: "bytez", tos: "ambiguous" }, { provider: "bytez", modelId: "mistralai/Mistral-7B-Instruct-v0.3", displayName: "mistralai/Mistral-7B-Instruct-v0.3", monthlyTokens: 0, creditTokens: 1000000, freeType: "recurring-credit", poolKey: "bytez", tos: "ambiguous" }, { provider: "bytez", modelId: "Qwen/Qwen2.5-72B-Instruct", displayName: "Qwen/Qwen2.5-72B-Instruct", monthlyTokens: 0, creditTokens: 1000000, freeType: "recurring-credit", poolKey: "bytez", tos: "ambiguous" }, - { provider: "cerebras", modelId: "zai-glm-4.7", displayName: "GLM 4.7", monthlyTokens: 30000000, creditTokens: 0, freeType: "recurring-daily", poolKey: "cerebras", tos: "caution" }, - { provider: "cerebras", modelId: "gpt-oss-120b", displayName: "GPT OSS 120B", monthlyTokens: 30000000, creditTokens: 0, freeType: "recurring-daily", poolKey: "cerebras", tos: "caution" }, + // hardStopGuaranteed: Cerebras pricing page states "Free Trial: 1M tokens/day... no credit card" (open-sse/services/../providers/apikey/inference-hosts.ts:74-84). + { provider: "cerebras", modelId: "zai-glm-4.7", displayName: "GLM 4.7", monthlyTokens: 30000000, creditTokens: 0, freeType: "recurring-daily", poolKey: "cerebras", tos: "caution", hardStopGuaranteed: true }, + { provider: "cerebras", modelId: "gpt-oss-120b", displayName: "GPT OSS 120B", monthlyTokens: 30000000, creditTokens: 0, freeType: "recurring-daily", poolKey: "cerebras", tos: "caution", hardStopGuaranteed: true }, // #8717: drop dead Workers AI ids (400/403/410). Keep Neurons/day budget on fp8-fast. { provider: "cloudflare-ai", modelId: "@cf/mistral/mistral-7b-instruct-v0.2-lora", displayName: "Mistral 7B (🆓)", monthlyTokens: 0, creditTokens: 0, freeType: "recurring-daily", poolKey: "cloudflare-ai", tos: "caution" }, { provider: "cloudflare-ai", modelId: "@cf/qwen/qwen2.5-coder-32b-instruct", displayName: "Qwen 2.5 Coder 32B (🆓)", monthlyTokens: 0, creditTokens: 0, freeType: "recurring-daily", poolKey: "cloudflare-ai", tos: "caution" }, @@ -187,11 +188,12 @@ export const FREE_MODEL_BUDGETS: FreeModelBudget[] = [ { provider: "glm-cn", modelId: "glm-4.5-flash", displayName: "GLM-4.5-Flash", monthlyTokens: 0, creditTokens: 0, freeType: "recurring-uncapped", poolKey: "zhipu-flash-free", tos: "ok" }, { provider: "glm-cn", modelId: "glm-4.7-flash", displayName: "GLM-4.7-Flash", monthlyTokens: 0, creditTokens: 0, freeType: "recurring-uncapped", poolKey: "zhipu-flash-free", tos: "ok" }, { provider: "glm-cn", modelId: "glm-signup-bonus", displayName: "Z.AI — 20M signup bonus", monthlyTokens: 0, creditTokens: 20000000, freeType: "one-time-initial", poolKey: "zhipu-signup", tos: "ok" }, - { provider: "groq", modelId: "meta-llama/llama-4-scout-17b-16e-instruct", displayName: "Llama 4 Scout", monthlyTokens: 15000000, creditTokens: 0, freeType: "recurring-daily", poolKey: "groq", tos: "caution" }, - { provider: "groq", modelId: "llama-3.3-70b-versatile", displayName: "Llama 3.3 70B", monthlyTokens: 15000000, creditTokens: 0, freeType: "recurring-daily", poolKey: "groq", tos: "caution" }, - { provider: "groq", modelId: "openai/gpt-oss-120b", displayName: "GPT-OSS 120B", monthlyTokens: 15000000, creditTokens: 0, freeType: "recurring-daily", poolKey: "groq", tos: "caution" }, - { provider: "groq", modelId: "openai/gpt-oss-20b", displayName: "GPT-OSS 20B", monthlyTokens: 15000000, creditTokens: 0, freeType: "recurring-daily", poolKey: "groq", tos: "caution" }, - { provider: "groq", modelId: "qwen/qwen3-32b", displayName: "Qwen3 32B", monthlyTokens: 15000000, creditTokens: 0, freeType: "recurring-daily", poolKey: "groq", tos: "caution" }, + // hardStopGuaranteed: Groq pricing page states "Free tier: 30 RPM / 14.4K RPD — no credit card" (open-sse/services/../providers/apikey/frontier-labs.ts:71-81). + { provider: "groq", modelId: "meta-llama/llama-4-scout-17b-16e-instruct", displayName: "Llama 4 Scout", monthlyTokens: 15000000, creditTokens: 0, freeType: "recurring-daily", poolKey: "groq", tos: "caution", hardStopGuaranteed: true }, + { provider: "groq", modelId: "llama-3.3-70b-versatile", displayName: "Llama 3.3 70B", monthlyTokens: 15000000, creditTokens: 0, freeType: "recurring-daily", poolKey: "groq", tos: "caution", hardStopGuaranteed: true }, + { provider: "groq", modelId: "openai/gpt-oss-120b", displayName: "GPT-OSS 120B", monthlyTokens: 15000000, creditTokens: 0, freeType: "recurring-daily", poolKey: "groq", tos: "caution", hardStopGuaranteed: true }, + { provider: "groq", modelId: "openai/gpt-oss-20b", displayName: "GPT-OSS 20B", monthlyTokens: 15000000, creditTokens: 0, freeType: "recurring-daily", poolKey: "groq", tos: "caution", hardStopGuaranteed: true }, + { provider: "groq", modelId: "qwen/qwen3-32b", displayName: "Qwen3 32B", monthlyTokens: 15000000, creditTokens: 0, freeType: "recurring-daily", poolKey: "groq", tos: "caution", hardStopGuaranteed: true }, { provider: "hackclub", modelId: "meta-llama/llama-3.3-70b-instruct", displayName: "Llama 3.3 70B", monthlyTokens: 0, creditTokens: 0, freeType: "keyless", poolKey: "hackclub", tos: "caution" }, { provider: "hackclub", modelId: "mistralai/mistral-7b-instruct", displayName: "Mistral 7B", monthlyTokens: 0, creditTokens: 0, freeType: "keyless", poolKey: "hackclub", tos: "caution" }, { provider: "hackclub", modelId: "deepseek-ai/deepseek-coder-33b", displayName: "DeepSeek Coder 33B", monthlyTokens: 0, creditTokens: 0, freeType: "keyless", poolKey: "hackclub", tos: "caution" }, diff --git a/open-sse/config/freeModelCatalog.ts b/open-sse/config/freeModelCatalog.ts index af8b9b0d96..103f3775f7 100644 --- a/open-sse/config/freeModelCatalog.ts +++ b/open-sse/config/freeModelCatalog.ts @@ -26,6 +26,20 @@ export interface FreeModelBudget { * reports this per model as `mayTrainOnYourPrompts` on its public catalog. */ trainsOnPrompts?: boolean; + /** + * True only when the provider's own published terms document that exceeding + * the free allowance is a hard stop (request refused / rate-limited) and NOT + * automatic pay-as-you-go billing — e.g. an explicit "no credit card + * required" claim on the provider's pricing page. This is a curated fact + * about the upstream provider, not something derivable from `freeType` or + * from any live API response, so it must be set by hand per entry with the + * source of the claim in a comment. Leave unset (undefined) whenever this + * isn't independently documented — `undefined` and `false` are both treated + * as "not guaranteed" by `strictZeroCostFilter.ts`; never default to `true` + * to grow the catalog. See STRICT_ZERO_COST in + * `open-sse/services/autoCombo/strictZeroCostFilter.ts`. + */ + hardStopGuaranteed?: boolean; } export interface FreeModelTotals { @@ -80,7 +94,7 @@ function fmt(n: number): string { function dedupedSum( models: FreeModelBudget[], pick: (m: FreeModelBudget) => number, - include: (m: FreeModelBudget) => boolean, + include: (m: FreeModelBudget) => boolean ): number { const poolMax = new Map(); let loose = 0; @@ -100,30 +114,30 @@ export function computeFreeModelTotals(opts: { excludeTosAvoid?: boolean } = {}) const steadyRecurringTokens = dedupedSum( models, (m) => m.monthlyTokens, - (m) => RECURRING.has(m.freeType), + (m) => RECURRING.has(m.freeType) ); const recurringCredits = dedupedSum( models, (m) => m.creditTokens, - (m) => m.freeType === "recurring-credit", + (m) => m.freeType === "recurring-credit" ); const oneTimeCredits = dedupedSum( models, (m) => m.creditTokens, - (m) => m.freeType === "one-time-initial", + (m) => m.freeType === "one-time-initial" ); const steadyWithRecurringCreditsTokens = steadyRecurringTokens + recurringCredits; const firstMonthRealisticTokens = steadyWithRecurringCreditsTokens + oneTimeCredits; const poolCount = new Set( - models.filter((m) => RECURRING.has(m.freeType) && m.poolKey).map((m) => m.poolKey), + models.filter((m) => RECURRING.has(m.freeType) && m.poolKey).map((m) => m.poolKey) ).size; // Deposit-unlock boost: sum the FREE_TIER_BOOSTS whose pool still has a live // recurring model in the (optionally ToS-filtered) set. const livePools = new Set( - models.filter((m) => RECURRING.has(m.freeType) && m.poolKey).map((m) => m.poolKey), + models.filter((m) => RECURRING.has(m.freeType) && m.poolKey).map((m) => m.poolKey) ); const boostMonthlyTokens = Object.entries(FREE_TIER_BOOSTS) .filter(([pool]) => livePools.has(pool)) diff --git a/open-sse/services/autoCombo/freeAccessQuota.ts b/open-sse/services/autoCombo/freeAccessQuota.ts new file mode 100644 index 0000000000..515ec57a87 --- /dev/null +++ b/open-sse/services/autoCombo/freeAccessQuota.ts @@ -0,0 +1,209 @@ +/** + * Live wiring for STRICT_ZERO_COST's quota-based branch. + * + * Reuses the existing `getUsageForProvider()` (`open-sse/services/usage.ts`) + * instead of building a second quota system — this module only adds a short + * TTL cache in front of it (so a Telegram-scale request rate never triggers a + * live billing-API call per candidate per request) and an invalidation hook + * for the resilience layer to call the moment a 402/403/quota-exhausted + * response is observed (`accountFallback.ts`). + * + * The cache is intentionally synchronous to read: `resolveFreeAccessState()` + * never awaits. A cache miss returns `undefined` (→ UNKNOWN → excluded, + * fail-closed) and kicks off a background refresh for the *next* read — + * nothing here can make `strictZeroCostFilter.ts`'s pool build block on a + * network call. + */ +import { + getUsageForProvider, + USAGE_FETCHER_PROVIDERS, + type UsageFetcherProvider, +} from "./../usage.ts"; +import { getCachedProviderConnections } from "@/lib/db/readCache"; +import { defaultLogger as log } from "@omniroute/open-sse/utils/logger"; +import type { FreeAccessState } from "./strictZeroCostFilter"; + +const USAGE_FETCHER_PROVIDER_SET = new Set(USAGE_FETCHER_PROVIDERS); + +/** Default cache TTL, reused verbatim from the already-shipped + * `settings.autoRefreshProviderQuotaInterval` (180s default, + * `src/lib/db/settings.ts`) instead of inventing a new number. */ +const FALLBACK_TTL_MS = 180_000; + +/** Cold-cache thundering-herd guard: after a process restart every candidate + * in a pool build is a simultaneous cache miss, which without a cap would + * fire one `getUsageForProvider()` call per distinct (provider, connection) + * pair in the same tick. Capping concurrent background refreshes spreads + * that burst out — a skipped refresh here just means this candidate stays + * UNKNOWN (excluded, fail-closed) until a later pool build tries again, never + * a correctness issue. */ +const MAX_CONCURRENT_REFRESHES = 4; + +/** Entries older than this are pruned outright even if nothing ever triggers + * a fresh refresh for that exact key again (e.g. the connection was deleted + * and no candidate references it anymore, so a normal stale-triggered + * refresh — which self-heals inside one TTL window — never fires). A sweep, + * not a timer: piggybacks on `resolveFreeAccessState` calls so this module + * never owns its own background interval. */ +const HARD_EVICTION_AGE_MS = FALLBACK_TTL_MS * 20; // 1 hour at the default TTL +const SWEEP_EVERY_N_CALLS = 200; + +interface CacheEntry { + state: FreeAccessState; + fetchedAtMs: number; +} + +// Keyed by `${provider}::${connectionId}` — module-level, process-lifetime +// cache. Cleared per-entry by `invalidateFreeAccessState`, on a failed +// refresh, or by the periodic sweep below; never wholesale. +const cache = new Map(); +const inFlight = new Set(); +let resolveCallCount = 0; + +function cacheKey(provider: string, connectionId: string): string { + return `${provider}::${connectionId}`; +} + +// `getSettings()` is async (DB-backed); reading it synchronously here isn't +// possible without changing `resolveFreeAccessState`'s synchronous contract. +// Using the fallback unconditionally is equivalent in practice: it's the same +// number as `settings.autoRefreshProviderQuotaInterval`'s own default +// (`src/lib/db/settings.ts`), and `strictZeroCostFilter.ts`'s own +// `maxStateAgeMs` (passed the real, live setting from `virtualFactory.ts`) +// is the check that actually gates staleness for the STRICT_ZERO_COST +// decision — this cache TTL only bounds how long a background refresh is +// skipped, a looser, non-safety-critical concern. +function ttlMs(): number { + return FALLBACK_TTL_MS; +} + +/** Opportunistic sweep of very stale entries, run every N calls instead of on + * a timer. Cheap (a single Map iteration) and only ever removes entries no + * live candidate can plausibly still be waiting on. */ +function sweepIfDue(): void { + resolveCallCount += 1; + if (resolveCallCount % SWEEP_EVERY_N_CALLS !== 0) return; + const now = Date.now(); + for (const [key, entry] of cache) { + if (now - entry.fetchedAtMs > HARD_EVICTION_AGE_MS) cache.delete(key); + } +} + +/** + * Best-effort, provider-agnostic extraction of "how much free allowance is + * left" from whatever shape `getUsageForProvider()` returns for this + * provider today. Adapters were written for a human-readable quota display, + * not for this filter, so their payloads are heterogeneous; this function + * recognizes the two shapes already used by other read paths in this + * codebase (`quotas.*.remainingPercentage` / `.total`+`.remaining`, mirroring + * `quota_omniroute.py`'s own parsing) and returns `null` — never a guess — + * for anything else. `null` is treated as "not proven safe" by the filter. + */ +function extractRemainingAllowance(usage: unknown): number | null { + if (!usage || typeof usage !== "object") return null; + const quotas = (usage as Record).quotas; + if (!quotas || typeof quotas !== "object") return null; + + let worstPercent: number | null = null; + for (const raw of Object.values(quotas as Record)) { + if (!raw || typeof raw !== "object") continue; + const q = raw as Record; + if (q.unlimited === true) continue; + let pct: number | null = + typeof q.remainingPercentage === "number" ? q.remainingPercentage : null; + if ( + pct === null && + typeof q.total === "number" && + typeof q.remaining === "number" && + q.total > 0 + ) { + pct = (100 * q.remaining) / q.total; + } + if (pct === null) continue; + worstPercent = worstPercent === null ? pct : Math.min(worstPercent, pct); + } + return worstPercent; // percentage points; the filter's threshold is compared against this unit +} + +async function refresh(provider: string, connectionId: string): Promise { + const key = cacheKey(provider, connectionId); + if (inFlight.has(key)) return; + if (inFlight.size >= MAX_CONCURRENT_REFRESHES) return; // thundering-herd guard — see const doc above + inFlight.add(key); + try { + const connections = await getCachedProviderConnections(); + const connection = connections.find( + (c): c is Record => + !!c && + typeof c === "object" && + (c as Record).id === connectionId && + (c as Record).provider === provider + ); + if (!connection) { + cache.delete(key); + return; + } + const usage = await getUsageForProvider( + connection as unknown as Parameters[0], + { forceRefresh: false } + ); + const remaining = extractRemainingAllowance(usage); + const state: FreeAccessState = { + status: remaining === null ? "UNKNOWN" : remaining > 0 ? "SAFE" : "EXHAUSTED", + remainingFreeAllowance: remaining, + resetAt: + usage && + typeof usage === "object" && + typeof (usage as Record).resetAt === "string" + ? ((usage as Record).resetAt as string) + : null, + checkedAt: new Date().toISOString(), + }; + cache.set(key, { state, fetchedAtMs: Date.now() }); + } catch (err) { + // A failed lookup must never leave a stale SAFE entry behind — drop it so + // the next read is a clean cache miss (UNKNOWN), not a lucky reuse. + cache.delete(key); + log.warn("AUTO", "STRICT_ZERO_COST: usage refresh failed, treating as UNKNOWN", { + provider, + err: err instanceof Error ? err.message : String(err), + }); + } finally { + inFlight.delete(key); + } +} + +/** + * Synchronous read for `strictZeroCostFilter.ts`. Returns `undefined` when + * there's no usage adapter for this provider at all (a permanent UNKNOWN, no + * point ever refreshing), or on a cold/stale cache — in both cases a + * background refresh is kicked off (fire-and-forget, subject to the + * concurrency cap above) so a *later* read can benefit, but this call itself + * never blocks or throws. + */ +export function resolveFreeAccessState( + provider: string, + connectionId: string | undefined +): FreeAccessState | undefined { + sweepIfDue(); + if (!USAGE_FETCHER_PROVIDER_SET.has(provider as UsageFetcherProvider)) return undefined; + if (!connectionId) return undefined; + + const key = cacheKey(provider, connectionId); + const entry = cache.get(key); + const fresh = entry && Date.now() - entry.fetchedAtMs <= ttlMs(); + if (!fresh) { + void refresh(provider, connectionId); + } + return fresh ? entry.state : undefined; +} + +/** Called by `accountFallback.ts` the moment a 402/403/quota-exhausted + * response is classified for a connection — drops the cached entry + * immediately instead of waiting out the TTL, so the very next candidate-pool + * build reads a clean cache miss (UNKNOWN) rather than a stale SAFE. */ +export function invalidateFreeAccessState(provider: string, connectionId: string): void { + cache.delete(cacheKey(provider, connectionId)); +} + +export const __testing = { cache, extractRemainingAllowance, sweepIfDue }; diff --git a/open-sse/services/autoCombo/strictZeroCostFilter.ts b/open-sse/services/autoCombo/strictZeroCostFilter.ts new file mode 100644 index 0000000000..c9bc601bc0 --- /dev/null +++ b/open-sse/services/autoCombo/strictZeroCostFilter.ts @@ -0,0 +1,283 @@ +/** + * STRICT_ZERO_COST — an opt-in, stricter sibling of `hidePaidModels` + * (`paidModelFilter.ts`) for operators who need a hard guarantee against ANY + * incremental monetary spend, not just "documented as free". + * + * `hidePaidModels` answers "is this model classified free in FREE_MODEL_BUDGETS + * right now?" — a point-in-time catalog fact. It says nothing about whether a + * `recurring-*`/`one-time-initial` candidate's allowance has since been + * consumed, and nothing about whether exceeding it is a hard stop or silent + * pay-as-you-go billing. STRICT_ZERO_COST adds exactly those two checks, + * before ranking, before dispatch — never after. + * + * Design, kept deliberately close to `filterPaidOnlyCandidates`'s own stated + * goal: "a pure, dependency-light function so the filter is unit-testable in + * isolation". The live quota lookup (`getUsageForProvider`, cached with a TTL) + * lives in `freeAccessQuota.ts` and is injected here as a plain function — + * this file never imports the DB or makes a network call itself. + * + * No provider or model name appears anywhere in this file. A candidate passes + * or fails purely on the metadata it carries (`freeType`, `tos`, + * `hardStopGuaranteed`) plus, for quota-based types, a `FreeAccessState` + * resolved elsewhere. A future provider that ships correct metadata is + * handled automatically; one that doesn't is excluded automatically — see + * `docs/routing/STRICT_ZERO_COST.md`. + * + * ## Connection safety (fixed after code review, see `docs/routing/STRICT_ZERO_COST.md`) + * + * A candidate from `virtualFactory.ts`'s connection-based pool represents ONE + * provider/model pair with a set of *eligible* connections + * (`allowedConnectionIds`) — the actual connection used at dispatch is chosen + * later (session stickiness/LKGP), not by this filter. Two invariants follow: + * + * 1. The `keyless` shortcut (no live check needed, because no credential + * exists) is valid ONLY for candidates that genuinely came from the + * no-auth path — identified by `connectionId === SYNTHETIC_NOAUTH_CONNECTION_ID` + * (`resilienceCandidateFilter.ts`). A `keyless`-catalogued model reached + * through a real DB connection (the same provider also has a + * credentialed connection) does NOT get the shortcut — it falls through + * to the normal quota-based check like any other freeType, and is + * excluded unless that specific connection independently proves SAFE. + * 2. For a multi-account candidate (`connectionId: null`, + * `allowedConnectionIds: [...]`), each connection is checked + * INDIVIDUALLY. The returned candidate's `allowedConnectionIds` is + * REWRITTEN to exactly the subset proven SAFE — never the full original + * list. `autoStrategy.ts` (`open-sse/services/combo/autoStrategy.ts:315-331`) + * already intersects further routing against `allowedConnectionIds` + * before connection selection, so rewriting it here is enough to make + * "verified this connection" and "dispatch used this connection" the + * same set, by construction — no new enforcement point needed. + */ +import { + FREE_MODEL_BUDGETS, + type FreeModelBudget, +} from "@omniroute/open-sse/config/freeModelCatalog.ts"; +import { SYNTHETIC_NOAUTH_CONNECTION_ID } from "./resilienceCandidateFilter"; + +/** Types whose allowance needs no runtime verification: no credential exists + * for the candidate at all, so no request against it can ever be billed. */ +const KEYLESS_FREE_TYPES = new Set(["keyless"]); + +export type FreeAccessStatus = "SAFE" | "EXHAUSTED" | "UNKNOWN"; + +/** Live-checked allowance state for one (provider, connection) pair. Resolved + * and cached by `freeAccessQuota.ts`; passed in here as plain data so this + * module stays free of DB/network dependencies. */ +export interface FreeAccessState { + status: FreeAccessStatus; + /** Remaining free allowance in the provider's own unit (tokens, requests, or + * USD-equivalent) — whatever `getUsageForProvider()` reports. `null` when + * the provider's usage payload doesn't expose a numeric remaining figure. */ + remainingFreeAllowance: number | null; + /** When the allowance next resets, if the provider reports it. */ + resetAt: string | null; + /** When this state was fetched (ISO 8601) — used to detect staleness. */ + checkedAt: string; +} + +/** A candidate as this module needs to see it — a structural subset of + * `VirtualAutoComboCandidate` (`virtualFactory.ts`) so this file has no + * dependency on that module's full type. */ +export interface StrictZeroCostCandidate { + provider: string; + model: string; + connectionId: string | null; + allowedConnectionIds?: string[]; +} + +export interface StrictZeroCostOptions { + /** Master switch — mirrors `hidePaidModels`'s own off-by-default shape. */ + enabled: boolean; + /** + * Resolves the live allowance state for ONE specific (provider, connection) + * pair. Returns `undefined` when no usage capability exists for the + * provider at all (no adapter registered in `USAGE_FETCHER_PROVIDERS`), or + * when the cache has nothing fresh for this exact connection — both are a + * meaningful, terminal UNKNOWN for that connection, not an error to retry. + * + * Synchronous by design: the caller (`virtualFactory.ts`) resolves and + * caches state per candidate up front, once per pool build, so this filter + * itself never awaits a network call and stays trivially testable. + */ + resolveFreeAccessState: (provider: string, connectionId: string) => FreeAccessState | undefined; + /** Minimum remaining allowance (in the unit `resolveFreeAccessState` reports + * — percentage points for the built-in `freeAccessQuota.ts` resolver) a + * quota-based connection must exceed to pass. Must be >= 0; a fully-exhausted + * account (`remainingFreeAllowance === 0`) fails at any non-negative + * threshold via the strict `>` comparison below. */ + minRemainingAllowance: number; + /** Maximum age, in ms, a `FreeAccessState.checkedAt` may have before it's + * treated as stale (→ UNKNOWN, excluded). */ + maxStateAgeMs: number; + /** `now` injection for deterministic tests; defaults to `Date.now`. */ + now?: () => number; + /** + * The free-model catalog to look candidates up against. Defaults to the + * real, live `FREE_MODEL_BUDGETS` — overridable so tests can prove the + * autodiscovery contract (a provider/model that appears in the catalog is + * automatically considered; one that's removed automatically disappears) + * with synthetic fixtures instead of mutating global state. Production + * callers should never pass this. Threaded through by + * `filterStrictZeroCostCandidates` (previously accepted but silently + * ignored — fixed alongside the connection-safety review). + */ + catalog?: readonly FreeModelBudget[]; +} + +export function findBudgetEntry( + candidate: Pick, + catalog: readonly FreeModelBudget[] = FREE_MODEL_BUDGETS +): FreeModelBudget | undefined { + return catalog.find((m) => m.provider === candidate.provider && m.modelId === candidate.model); +} + +function isConnectionStateSafe( + provider: string, + connectionId: string, + resolveFreeAccessState: StrictZeroCostOptions["resolveFreeAccessState"], + options: Pick +): boolean { + const state = resolveFreeAccessState(provider, connectionId); + if (!state) return false; // no usage adapter for this provider, or lookup never ran/is stale + if (state.status !== "SAFE") return false; + + const now = (options.now ?? Date.now)(); + const checkedAtMs = Date.parse(state.checkedAt); + if (!Number.isFinite(checkedAtMs) || now - checkedAtMs > options.maxStateAgeMs) return false; + + if (state.remainingFreeAllowance === null) return false; + // A negative threshold would let a negative/garbage reading pass; a caller + // that genuinely wants "any allowance greater than zero" should pass 0. + if (options.minRemainingAllowance < 0) return false; + return state.remainingFreeAllowance > options.minRemainingAllowance; +} + +/** + * Decide which of a candidate's connections satisfy STRICT_ZERO_COST. Pure — + * `resolveFreeAccessState` is the only injected side-effecting dependency, + * and it's a synchronous cache read (see `StrictZeroCostOptions` above). + * + * Returns the list of connection ids proven SAFE right now: + * - `[SYNTHETIC_NOAUTH_CONNECTION_ID]` for a genuine no-auth candidate whose + * catalog entry is `keyless` — no live check needed or possible. + * - a (possibly empty) subset of the candidate's real connection id(s) for + * every other case, each individually verified. + * An empty array means the caller must exclude the candidate entirely. + */ +export function evaluateCandidateConnections( + candidate: StrictZeroCostCandidate, + budgetEntry: FreeModelBudget | undefined, + resolveFreeAccessState: StrictZeroCostOptions["resolveFreeAccessState"], + options: Pick +): string[] { + if (!budgetEntry) return []; // not in the catalog at all → paid, or genuinely unknown + + const isGenuineNoAuthCandidate = candidate.connectionId === SYNTHETIC_NOAUTH_CONNECTION_ID; + if (KEYLESS_FREE_TYPES.has(budgetEntry.freeType)) { + // The keyless shortcut is trustworthy ONLY when this specific candidate + // instance actually has no credential behind it. A `keyless`-catalogued + // model reached through a real DB connection (connectionId is a real id, + // or the candidate carries allowedConnectionIds at all) must NOT take + // this shortcut — it falls through to the quota-based check below like + // any other freeType, and is excluded there unless hardStopGuaranteed is + // also set for it (which the curated catalog does not do for keyless + // entries today, so it will correctly exclude). + if (isGenuineNoAuthCandidate) return [SYNTHETIC_NOAUTH_CONNECTION_ID]; + } + if (budgetEntry.freeType === "discontinued") return []; + if (isGenuineNoAuthCandidate) return []; // no-auth path but a non-keyless catalog entry: contradictory metadata, fail closed + + // Every remaining freeType (recurring-*, one-time-initial, a keyless entry + // reached via a real connection, and any future type this module doesn't + // special-case) requires a documented hard stop before any live check even + // runs — no point burning a quota lookup on a connection we could never + // trust regardless of its answer. + if (budgetEntry.hardStopGuaranteed !== true) return []; + + const candidateConnectionIds = candidate.connectionId + ? [candidate.connectionId] + : (candidate.allowedConnectionIds ?? []); + + const safe: string[] = []; + for (const connectionId of candidateConnectionIds) { + if (connectionId === SYNTHETIC_NOAUTH_CONNECTION_ID) continue; // never reachable here, defensive + if (isConnectionStateSafe(candidate.provider, connectionId, resolveFreeAccessState, options)) { + safe.push(connectionId); + } + } + return safe; +} + +/** + * Pool-level filter, same off-by-default identity contract as + * `filterPaidOnlyCandidates`. For a candidate that survives with a NARROWED + * connection set (the multi-account case), the returned object has + * `allowedConnectionIds` rewritten to exactly the SAFE subset — dispatch can + * then never select a connection this filter didn't verify, because + * `autoStrategy.ts` already enforces `allowedConnectionIds` as a hard + * allowlist downstream (see the module docstring above). + */ +export function filterStrictZeroCostCandidates( + pool: T[], + options: StrictZeroCostOptions +): T[] { + if (!options.enabled) return pool; + + const kept: T[] = []; + let changed = false; + for (const candidate of pool) { + const budgetEntry = findBudgetEntry(candidate, options.catalog); + const safeConnectionIds = evaluateCandidateConnections( + candidate, + budgetEntry, + options.resolveFreeAccessState, + options + ); + if (safeConnectionIds.length === 0) { + changed = true; + continue; + } + + const isGenuineNoAuthCandidate = candidate.connectionId === SYNTHETIC_NOAUTH_CONNECTION_ID; + const isSingleConnectionCandidate = candidate.connectionId !== null; + if (isGenuineNoAuthCandidate || isSingleConnectionCandidate) { + // Nothing to narrow — either the no-auth sentinel, or a candidate that + // already pointed at exactly one connection which proved safe. + kept.push(candidate); + continue; + } + + // Multi-account candidate: only rewrite if the safe subset is actually + // narrower than what was there before, to preserve the same + // identity-when-nothing-changed contract as `filterPaidOnlyCandidates`. + const original = candidate.allowedConnectionIds ?? []; + const isSameSet = + original.length === safeConnectionIds.length && + safeConnectionIds.every((id) => original.includes(id)); + if (isSameSet) { + kept.push(candidate); + } else { + changed = true; + kept.push({ ...candidate, allowedConnectionIds: safeConnectionIds }); + } + } + return changed ? kept : pool; +} + +/** + * Separate, optional ToS guard — kept independent from economic safety on + * purpose (Marco's requirement): a model can be economically SAFE and still + * excluded here for ToS reasons, or left in when this guard is off even if + * STRICT_ZERO_COST is on. Reuses the same curated `tos` field, no new data. + */ +export function filterTosAvoidCandidates( + pool: T[], + excludeTosAvoid: boolean, + catalog?: readonly FreeModelBudget[] +): T[] { + if (!excludeTosAvoid) return pool; + return pool.filter((candidate) => { + const budgetEntry = findBudgetEntry(candidate, catalog); + return budgetEntry?.tos !== "avoid"; + }); +} diff --git a/open-sse/services/autoCombo/virtualFactory.ts b/open-sse/services/autoCombo/virtualFactory.ts index c6d10f35dc..9fa1287563 100644 --- a/open-sse/services/autoCombo/virtualFactory.ts +++ b/open-sse/services/autoCombo/virtualFactory.ts @@ -26,6 +26,8 @@ import { buildFamilyCandidateFilter, type ModelFamily } from "./modelFamily"; import { getHiddenModelsByProvider } from "@/models"; import { getSyncedAvailableModelsByConnection, getCustomModels } from "@/lib/db/models"; import { filterPaidOnlyCandidates } from "./paidModelFilter"; +import { filterStrictZeroCostCandidates, filterTosAvoidCandidates } from "./strictZeroCostFilter"; +import { resolveFreeAccessState } from "./freeAccessQuota"; import { isModelExcludedByConnection } from "@/domain/connectionModelRules"; import { resolveProviderAlias } from "../model.ts"; import { filterExcludedCandidates } from "./candidateOverrides"; @@ -590,6 +592,29 @@ export async function prepareVirtualAutoComboInputs( // exclude paid-only backends from EVERY `auto/*` candidate pool. const paidFilteredPool = filterPaidOnlyCandidates(pool, settings.hidePaidModels === true); if (paidFilteredPool !== pool) pool = paidFilteredPool; + + // STRICT_ZERO_COST: opt-in, off by default (`settings.freeAccessPolicy !== "strict"` + // leaves `pool` byte-identical, same contract as `hidePaidModels`). See + // `strictZeroCostFilter.ts` for why this is stricter than `hidePaidModels` alone — + // including the connection-safety invariant it enforces per-connection, not just + // per-candidate: `resolveFreeAccessState` here is a raw pass-through of the real + // per-(provider,connectionId) resolver; the filter itself decides which connection(s) + // on each candidate to check and rewrites `allowedConnectionIds` to the SAFE subset. + const strictFilteredPool = filterStrictZeroCostCandidates(pool, { + enabled: settings.freeAccessPolicy === "strict", + resolveFreeAccessState, + // 1 percentage point of headroom, not 0: `freeAccessQuota.ts` reports + // remaining allowance as a percentage, and a raw ">0" comparison would + // let a reading of e.g. 0.3% (rounding noise, not real headroom) pass. + minRemainingAllowance: 1, + maxStateAgeMs: (settings.autoRefreshProviderQuotaInterval ?? 180) * 1000, + }); + if (strictFilteredPool !== pool) pool = strictFilteredPool; + + // Separate, optional ToS guard — independent of economic safety on purpose. + const tosFilteredPool = filterTosAvoidCandidates(pool, settings.excludeTosAvoid === true); + if (tosFilteredPool !== pool) pool = tosFilteredPool; + return pool; }; diff --git a/scripts/ad-hoc/dry-run-strict-zero-cost.ts b/scripts/ad-hoc/dry-run-strict-zero-cost.ts new file mode 100644 index 0000000000..54ea41f4d3 --- /dev/null +++ b/scripts/ad-hoc/dry-run-strict-zero-cost.ts @@ -0,0 +1,105 @@ +/** + * Ad-hoc, one-shot dry run of STRICT_ZERO_COST against the real candidate + * pools currently served by this OmniRoute instance (fetched via the + * existing read-only `GET /v1/auto-combo/{channel}/candidates` endpoint — + * no changes made, no billable calls). Not wired into any test suite. + * + * Simulates the filter offline: no live usage-quota state is available + * (that adapter only runs inside the deployed container), so + * `resolveFreeAccessState` always returns `undefined` here — meaning any + * quota-based candidate is reported UNKNOWN unless it lacks even a usage + * adapter, in which case it's reported UNKNOWN for that reason instead. This + * intentionally shows the current, honest ceiling of what's usable today. + * + * Uses each candidate's REAL `connectionId` from the live endpoint (rather + * than assuming) to also exercise the post-code-review connection-safety + * check: a `keyless`-catalogued model whose live `connectionId` is NOT the + * no-auth sentinel is correctly reported as excluded here too. + */ +import { readFileSync } from "node:fs"; +import { + evaluateCandidateConnections, + findBudgetEntry, +} from "../../open-sse/services/autoCombo/strictZeroCostFilter.ts"; +import { SYNTHETIC_NOAUTH_CONNECTION_ID } from "../../open-sse/services/autoCombo/resilienceCandidateFilter.ts"; +import { USAGE_FETCHER_PROVIDERS } from "../../open-sse/services/usage.ts"; + +const usageProviders = new Set(USAGE_FETCHER_PROVIDERS); +const OPTIONS = { minRemainingAllowance: 1, maxStateAgeMs: 180_000 }; + +interface Candidate { + provider: string; + model: string; + connectionId: string; +} + +function loadCandidates(path: string): Candidate[] { + const raw = JSON.parse(readFileSync(path, "utf8")); + const list = Array.isArray(raw) ? raw : raw.candidates; + // The candidates endpoint's `model` field is the FULL "/" + // string (`modelStr` — the leading segment is sometimes the provider id, + // e.g. "groq/...", sometimes its short alias, e.g. "oc/..." for opencode); + // FREE_MODEL_BUDGETS.modelId is always bare. Strip exactly the first "/" + // segment (whichever form it is) so e.g. "groq/meta-llama/llama-4-scout..." + // becomes "meta-llama/llama-4-scout..." and "oc/big-pickle" becomes + // "big-pickle", matching the catalog's modelId either way. + return list.map((c: { provider: string; model: string; connectionId?: string }) => { + const slash = c.model.indexOf("/"); + return { + provider: c.provider, + model: slash === -1 ? c.model : c.model.slice(slash + 1), + connectionId: c.connectionId ?? SYNTHETIC_NOAUTH_CONNECTION_ID, + }; + }); +} + +function run(label: string, path: string): void { + const candidates = loadCandidates(path); + console.log(`\n=== ${label} — ${candidates.length} candidati live ===`); + + const kept: Candidate[] = []; + const excluded: { candidate: Candidate; reason: string }[] = []; + + for (const c of candidates) { + const entry = findBudgetEntry(c); + if (!entry) { + excluded.push({ candidate: c, reason: "non presente nel catalogo free curato" }); + continue; + } + const isNoAuthConnection = c.connectionId === SYNTHETIC_NOAUTH_CONNECTION_ID; + if (entry.freeType === "keyless") { + const safe = evaluateCandidateConnections(c, entry, () => undefined, OPTIONS); + if (safe.length > 0) { + kept.push(c); + } else if (!isNoAuthConnection) { + excluded.push({ + candidate: c, + reason: + "keyless nel catalogo ma raggiunto tramite una connessione DB reale (non il sentinel noauth) — shortcut non applicato, richiederebbe hardStopGuaranteed", + }); + } else { + excluded.push({ candidate: c, reason: "keyless ma valutazione fallita (inatteso)" }); + } + continue; + } + const hasAdapter = usageProviders.has(entry.provider); + const reason = !hasAdapter + ? `nessun usage adapter per '${entry.provider}' in USAGE_FETCHER_PROVIDERS` + : entry.hardStopGuaranteed !== true + ? "hardStopGuaranteed non dichiarato per questo modello" + : "nessuno stato quota live disponibile in questo dry-run offline (richiederebbe il container reale)"; + excluded.push({ candidate: c, reason }); + } + + console.log(`PRIMA (STRICT_ZERO_COST off): ${candidates.length} candidati`); + console.log(`DOPO (STRICT_ZERO_COST on): ${kept.length} candidati sopravvissuti`); + console.log("Sopravvissuti:"); + for (const c of kept) console.log(` OK ${c.provider}/${c.model}`); + console.log("Esclusi (motivo):"); + for (const { candidate: c, reason } of excluded) { + console.log(` EXCL ${c.provider}/${c.model} — ${reason}`); + } +} + +run("auto/coding:free", process.argv[2] ?? "/tmp/dryrun_coding_free.json"); +run("auto/best-free", process.argv[3] ?? "/tmp/dryrun_best-free.json"); diff --git a/src/lib/db/settings.ts b/src/lib/db/settings.ts index ce2b889066..b9393db3c8 100644 --- a/src/lib/db/settings.ts +++ b/src/lib/db/settings.ts @@ -238,6 +238,11 @@ export async function getSettings() { // (`:free` suffix, zero-price pricing, or FREE_MODEL_BUDGETS membership). Default // false preserves prior behaviour; opt-in only. hidePaidModels: false, + // Opt-in, default off: same shape as hidePaidModels above, but requires a + // live hard-stop-guaranteed quota check for non-keyless free candidates. + // See open-sse/services/autoCombo/strictZeroCostFilter.ts. + freeAccessPolicy: "off", + excludeTosAvoid: false, // #9418: Opt-in filter that hides auto/* virtual combos from the /v1/models catalog. // User-defined combos are unaffected; routing still works for hidden ids sent explicitly. hideAutoCombos: false, diff --git a/src/shared/validation/settingsSchemas.ts b/src/shared/validation/settingsSchemas.ts index 008cba1bb0..388e3f73c7 100644 --- a/src/shared/validation/settingsSchemas.ts +++ b/src/shared/validation/settingsSchemas.ts @@ -128,6 +128,16 @@ export const updateSettingsSchema = z.object({ blockedProviders: z.array(z.string().max(100)).optional(), noAuthFallbackDisabledProviders: z.array(z.string().max(100)).optional(), hidePaidModels: z.boolean().optional(), + // STRICT_ZERO_COST (opt-in, default "off"): stricter than hidePaidModels — a + // candidate must be keyless (no credential exists, so no request against it + // can ever be billed) OR pass a live, fresh, hard-stop-guaranteed quota + // check, per candidate, before ranking/dispatch. See + // open-sse/services/autoCombo/strictZeroCostFilter.ts. + freeAccessPolicy: z.enum(["off", "strict"]).optional(), + // Separate from freeAccessPolicy on purpose: excludes candidates whose + // curated `tos` verdict is "avoid" (proxy/self-hosted use conflicts with the + // provider's own terms) — a contractual concern, not an economic one. + excludeTosAvoid: z.boolean().optional(), hideHealthCheckLogs: z.boolean().optional(), hideEndpointCloudflaredTunnel: z.boolean().optional(), hideEndpointTailscaleFunnel: z.boolean().optional(), diff --git a/src/sse/services/auth.ts b/src/sse/services/auth.ts index f957c7375c..b1df29552a 100644 --- a/src/sse/services/auth.ts +++ b/src/sse/services/auth.ts @@ -2453,6 +2453,18 @@ export async function markAccountUnavailable( try { await currentMutex; + // STRICT_ZERO_COST: this connection just failed (whatever the reason) — + // drop any cached "SAFE" free-allowance reading for it immediately rather + // than waiting out the TTL, so the very next candidate-pool build reads a + // clean cache miss (UNKNOWN → excluded) instead of a stale SAFE. Cheap, + // idempotent, and correct to over-invalidate on non-quota failures too — + // worst case is one extra background refresh. + if (provider) { + const { invalidateFreeAccessState } = + await import("@omniroute/open-sse/services/autoCombo/freeAccessQuota.ts"); + invalidateFreeAccessState(provider, connectionId); + } + const resourceBypass = getResource404Bypass(status, errorText, connectionId, log); if (resourceBypass) return resourceBypass; diff --git a/tests/unit/autoCombo/strict-zero-cost-autodiscovery.test.ts b/tests/unit/autoCombo/strict-zero-cost-autodiscovery.test.ts new file mode 100644 index 0000000000..bb93e3601b --- /dev/null +++ b/tests/unit/autoCombo/strict-zero-cost-autodiscovery.test.ts @@ -0,0 +1,192 @@ +/** + * STRICT_ZERO_COST — autodiscovery contract (Fase 5-8 of the design audit): + * the filter must never hardcode a provider/model allowlist. It reads + * whatever `catalog` (FREE_MODEL_BUDGETS-shaped) it's given, so a new SAFE + * entry becomes usable the moment it appears, and a removed one disappears + * the moment it's gone — no code change, no KITT change. + * + * Uses `evaluateCandidateConnections`'s injectable `catalog` argument (added + * for exactly this test) with synthetic fixtures instead of mutating the + * real, shared `FREE_MODEL_BUDGETS` — that constant is imported by + * production code and other test files; mutating it at runtime would be a + * global side effect this file has no business causing. + */ +import { test } from "vitest"; +import assert from "node:assert/strict"; + +import { + evaluateCandidateConnections, + findBudgetEntry, + type FreeAccessState, + type StrictZeroCostCandidate, +} from "../../../open-sse/services/autoCombo/strictZeroCostFilter.ts"; +import { SYNTHETIC_NOAUTH_CONNECTION_ID } from "../../../open-sse/services/autoCombo/resilienceCandidateFilter.ts"; +import type { FreeModelBudget } from "../../../open-sse/config/freeModelCatalog.ts"; + +const NOW = "2026-08-20T00:00:00.000Z"; +const OPTIONS = { minRemainingAllowance: 1, maxStateAgeMs: 180_000, now: () => Date.parse(NOW) }; + +function safeState(): FreeAccessState { + return { status: "SAFE", remainingFreeAllowance: 40, resetAt: null, checkedAt: NOW }; +} + +// `evaluateCandidateConnections` has exactly one source of truth for "is this +// candidate documented as free": the `budgetEntry` its caller looked up via +// `findBudgetEntry(candidate, catalog)`. These provider ids are otherwise +// arbitrary — the fixtures below prove the behavior is driven entirely by +// catalog membership, not by any hardcoded provider/model name. +const KEYLESS_PROVIDER = "felo-web"; +const QUOTA_PROVIDER = "groq"; +const REAL_CONN = "conn-1"; + +function keylessEntry(modelId: string): FreeModelBudget { + return { + provider: KEYLESS_PROVIDER, + modelId, + displayName: modelId, + monthlyTokens: 0, + creditTokens: 0, + freeType: "keyless", + poolKey: null, + tos: "ok", + }; +} + +function quotaEntry(modelId: string, hardStopGuaranteed = true): FreeModelBudget { + return { + provider: QUOTA_PROVIDER, + modelId, + displayName: modelId, + monthlyTokens: 1_000_000, + creditTokens: 0, + freeType: "recurring-daily", + poolKey: null, + tos: "ok", + hardStopGuaranteed, + }; +} + +// 14/18. New provider/model appears in the catalog → automatically a candidate. +test("a brand-new keyless model appearing in the catalog is usable the instant it appears", () => { + const modelId = "synthetic-new-model-14"; + const candidate: StrictZeroCostCandidate = { + provider: KEYLESS_PROVIDER, + model: modelId, + connectionId: SYNTHETIC_NOAUTH_CONNECTION_ID, + }; + const catalogBefore: FreeModelBudget[] = []; // model does not exist yet + const catalogAfter: FreeModelBudget[] = [keylessEntry(modelId)]; // "OmniRoute" just added it + + assert.deepEqual( + evaluateCandidateConnections( + candidate, + findBudgetEntry(candidate, catalogBefore), + () => undefined, + OPTIONS + ), + [], + "must be excluded before the catalog knows about it" + ); + assert.deepEqual( + evaluateCandidateConnections( + candidate, + findBudgetEntry(candidate, catalogAfter), + () => undefined, + OPTIONS + ), + [SYNTHETIC_NOAUTH_CONNECTION_ID], + "must pass automatically the moment the SAME code sees it in the catalog — no code change made between these two calls" + ); +}); + +// 15/19. Provider/model removed from the catalog → automatically disappears. +test("a model removed from the catalog stops being usable, with no code change", () => { + const modelId = "synthetic-removed-model-15"; + const candidate: StrictZeroCostCandidate = { + provider: KEYLESS_PROVIDER, + model: modelId, + connectionId: SYNTHETIC_NOAUTH_CONNECTION_ID, + }; + const catalogBefore: FreeModelBudget[] = [keylessEntry(modelId)]; + const catalogAfter: FreeModelBudget[] = []; // "OmniRoute" dropped the promo + + assert.deepEqual( + evaluateCandidateConnections( + candidate, + findBudgetEntry(candidate, catalogBefore), + () => undefined, + OPTIONS + ), + [SYNTHETIC_NOAUTH_CONNECTION_ID] + ); + assert.deepEqual( + evaluateCandidateConnections( + candidate, + findBudgetEntry(candidate, catalogAfter), + () => undefined, + OPTIONS + ), + [], + "must be excluded automatically once gone — no whitelist entry to delete anywhere" + ); +}); + +// Case 2 of Fase 8: a quota-based model ships WITH a usage adapter and +// hardStopGuaranteed=true and a SAFE live state — passes automatically. +test("a new quota-based model with hardStopGuaranteed + SAFE state is usable automatically", () => { + const modelId = "synthetic-quota-model-safe"; + const candidate: StrictZeroCostCandidate = { + provider: QUOTA_PROVIDER, + model: modelId, + connectionId: REAL_CONN, + }; + const catalog: FreeModelBudget[] = [quotaEntry(modelId, true)]; + assert.deepEqual( + evaluateCandidateConnections( + candidate, + findBudgetEntry(candidate, catalog), + () => safeState(), + OPTIONS + ), + [REAL_CONN] + ); +}); + +// Case 3 of Fase 8: metadata incomplete (no hardStopGuaranteed) → UNKNOWN → excluded automatically. +test("a new quota-based model WITHOUT hardStopGuaranteed is excluded automatically, even with a SAFE state", () => { + const modelId = "synthetic-quota-model-unguaranteed"; + const candidate: StrictZeroCostCandidate = { + provider: QUOTA_PROVIDER, + model: modelId, + connectionId: REAL_CONN, + }; + const entry = quotaEntry(modelId); + delete entry.hardStopGuaranteed; // simulate metadata that was never set, not defaulted to true + const catalog: FreeModelBudget[] = [entry]; + assert.deepEqual( + evaluateCandidateConnections( + candidate, + findBudgetEntry(candidate, catalog), + () => safeState(), + OPTIONS + ), + [] + ); +}); + +// Production never overrides `catalog` — `findBudgetEntry(candidate)` with no +// second argument must read the real, live FREE_MODEL_BUDGETS, so an entirely +// invented provider id (one no real OmniRoute release has ever catalogued) is +// excluded automatically, with zero whitelist to edit, when the default is used. +test("an entirely unknown provider id is excluded automatically when the default (real) catalog is used", () => { + const candidate: StrictZeroCostCandidate = { + provider: "totally-unheard-of-provider-xyz", + model: "whatever", + connectionId: REAL_CONN, + }; + assert.deepEqual( + evaluateCandidateConnections(candidate, findBudgetEntry(candidate), () => undefined, OPTIONS), + [], + "findBudgetEntry() with no catalog override reads the real FREE_MODEL_BUDGETS — nothing to add or remove for this provider to stay excluded" + ); +}); diff --git a/tests/unit/autoCombo/strict-zero-cost-connection-safety.test.ts b/tests/unit/autoCombo/strict-zero-cost-connection-safety.test.ts new file mode 100644 index 0000000000..312f83eb51 --- /dev/null +++ b/tests/unit/autoCombo/strict-zero-cost-connection-safety.test.ts @@ -0,0 +1,303 @@ +/** + * STRICT_ZERO_COST — connection-safety regression guards for the two + * blockers found in code review of commit 5d4f881: + * + * BLOCKER 1 (keyless bypass): the `keyless` shortcut must apply ONLY to a + * candidate that genuinely came from the no-auth path + * (`connectionId === SYNTHETIC_NOAUTH_CONNECTION_ID`), never to a + * `keyless`-catalogued model reached through a real DB connection. + * + * BLOCKER 2 (multi-account mismatch): a multi-account candidate's + * `allowedConnectionIds` must be rewritten to exactly the SAFE subset, so + * whatever `autoStrategy.ts` selects at dispatch time can never be a + * connection this filter didn't itself verify. + * + * Both are pure-function tests against `evaluateCandidateConnections()` / + * `filterStrictZeroCostCandidates()` — no DB, no network. + */ +import { test } from "vitest"; +import assert from "node:assert/strict"; + +import { + evaluateCandidateConnections, + filterStrictZeroCostCandidates, + type FreeAccessState, + type StrictZeroCostCandidate, +} from "../../../open-sse/services/autoCombo/strictZeroCostFilter.ts"; +import { SYNTHETIC_NOAUTH_CONNECTION_ID } from "../../../open-sse/services/autoCombo/resilienceCandidateFilter.ts"; +import type { FreeModelBudget } from "../../../open-sse/config/freeModelCatalog.ts"; + +const NOW = "2026-08-20T00:00:00.000Z"; +const OPTIONS = { minRemainingAllowance: 1, maxStateAgeMs: 180_000, now: () => Date.parse(NOW) }; + +function safeState(overrides: Partial = {}): FreeAccessState { + return { + status: "SAFE", + remainingFreeAllowance: 40, + resetAt: null, + checkedAt: NOW, + ...overrides, + }; +} + +// A synthetic keyless budget entry — hardStopGuaranteed deliberately absent, +// matching the real catalog (keyless entries never set it; the shortcut is +// what makes them safe, not this field). +function keylessEntry(): FreeModelBudget { + return { + provider: "kp", + modelId: "kp-model", + displayName: "kp-model", + monthlyTokens: 0, + creditTokens: 0, + freeType: "keyless", + poolKey: null, + tos: "ok", + }; +} + +function quotaEntry(hardStopGuaranteed = true): FreeModelBudget { + return { + provider: "qp", + modelId: "qp-model", + displayName: "qp-model", + monthlyTokens: 1_000_000, + creditTokens: 0, + freeType: "recurring-daily", + poolKey: null, + tos: "ok", + hardStopGuaranteed, + }; +} + +const REAL_CONN = "real-connection-42"; + +// ─── BLOCKER 1 — keyless bypass ───────────────────────────────────────── + +// A. keyless + SYNTHETIC_NOAUTH_CONNECTION_ID → PASS +test("A: keyless + SYNTHETIC_NOAUTH_CONNECTION_ID passes with zero live checks", () => { + const candidate: StrictZeroCostCandidate = { + provider: "kp", + model: "kp-model", + connectionId: SYNTHETIC_NOAUTH_CONNECTION_ID, + }; + const resolveFreeAccessState = () => { + throw new Error("must never be called for a genuine no-auth candidate"); + }; + assert.deepEqual( + evaluateCandidateConnections(candidate, keylessEntry(), resolveFreeAccessState, OPTIONS), + [SYNTHETIC_NOAUTH_CONNECTION_ID] + ); +}); + +// B. same provider/model, catalogued keyless, but reached via a real DB +// connectionId → must NOT take the keyless shortcut. +test("B: keyless-catalogued model reached via a real connectionId does NOT get the keyless shortcut", () => { + const candidate: StrictZeroCostCandidate = { + provider: "kp", + model: "kp-model", + connectionId: REAL_CONN, // NOT the sentinel — a genuine DB connection + }; + let called = false; + const resolveFreeAccessState = () => { + called = true; + return safeState(); // even if the connection WOULD report SAFE quota... + }; + const result = evaluateCandidateConnections( + candidate, + keylessEntry(), + resolveFreeAccessState, + OPTIONS + ); + // ...the entry has no hardStopGuaranteed (real keyless catalog rows never + // set it — the shortcut was their only path to safety), so falling through + // to the quota branch must still exclude it. + assert.deepEqual( + result, + [], + "must exclude — keyless metadata alone cannot authorize a real connection" + ); + assert.equal( + called, + false, + "must not even bother resolving live state once hardStopGuaranteed is absent" + ); +}); + +// C. a connection that later gains a credential still can't exploit the +// keyless classification — same code path as B, demonstrated with a +// multi-account (allowedConnectionIds) shape to also prove the "logical +// candidate" branch, not just the single-connectionId branch, is covered. +test("C: a keyless-catalogued model behind allowedConnectionIds (credentialed accounts) is never shortcut-eligible", () => { + const candidate: StrictZeroCostCandidate = { + provider: "kp", + model: "kp-model", + connectionId: null, + allowedConnectionIds: ["acct-with-credential-1", "acct-with-credential-2"], + }; + const resolveFreeAccessState = () => safeState(); // both accounts report SAFE quota + const result = evaluateCandidateConnections( + candidate, + keylessEntry(), + resolveFreeAccessState, + OPTIONS + ); + assert.deepEqual( + result, + [], + "keyless metadata must never authorize any real, credentialed account, regardless of what its quota says" + ); +}); + +// Sanity: a non-keyless entry reached via the no-auth sentinel is contradictory +// metadata (shouldn't happen in practice) and must fail closed, not silently +// fall through to a quota check that can never resolve a state for "noauth". +test("sanity: non-keyless freeType via the no-auth sentinel connection fails closed", () => { + const candidate: StrictZeroCostCandidate = { + provider: "qp", + model: "qp-model", + connectionId: SYNTHETIC_NOAUTH_CONNECTION_ID, + }; + const result = evaluateCandidateConnections( + candidate, + quotaEntry(true), + () => safeState(), + OPTIONS + ); + assert.deepEqual(result, []); +}); + +// ─── BLOCKER 2 — multi-account connection mismatch ───────────────────── + +// 1. account A SAFE, B UNKNOWN → dispatch can use ONLY A. +test("1: A SAFE, B UNKNOWN -> safe connection set is exactly [A]", () => { + const candidate: StrictZeroCostCandidate = { + provider: "qp", + model: "qp-model", + connectionId: null, + allowedConnectionIds: ["A", "B"], + }; + const resolve = (_provider: string, connectionId: string) => + connectionId === "A" ? safeState() : undefined; // B: no state at all == UNKNOWN + assert.deepEqual(evaluateCandidateConnections(candidate, quotaEntry(true), resolve, OPTIONS), [ + "A", + ]); +}); + +// 2. A EXHAUSTED, B SAFE → SOLO B. +test("2: A EXHAUSTED, B SAFE -> safe connection set is exactly [B]", () => { + const candidate: StrictZeroCostCandidate = { + provider: "qp", + model: "qp-model", + connectionId: null, + allowedConnectionIds: ["A", "B"], + }; + const resolve = (_provider: string, connectionId: string) => + connectionId === "A" + ? safeState({ status: "EXHAUSTED", remainingFreeAllowance: 0 }) + : safeState(); + assert.deepEqual(evaluateCandidateConnections(candidate, quotaEntry(true), resolve, OPTIONS), [ + "B", + ]); +}); + +// 3. A SAFE, B billable/unverifiable (UNKNOWN) → B not selectable. +test("3: A SAFE, B UNKNOWN (billable/unverifiable) -> B never enters the safe set", () => { + const candidate: StrictZeroCostCandidate = { + provider: "qp", + model: "qp-model", + connectionId: null, + allowedConnectionIds: ["A", "B"], + }; + const resolve = (_provider: string, connectionId: string) => + connectionId === "A" ? safeState() : undefined; + const safe = evaluateCandidateConnections(candidate, quotaEntry(true), resolve, OPTIONS); + assert.equal(safe.includes("B"), false); + assert.deepEqual(safe, ["A"]); +}); + +// 4. tutte UNKNOWN → candidato escluso. +test("4: all connections UNKNOWN -> candidate fully excluded (empty safe set)", () => { + const candidate: StrictZeroCostCandidate = { + provider: "qp", + model: "qp-model", + connectionId: null, + allowedConnectionIds: ["A", "B", "C"], + }; + assert.deepEqual( + evaluateCandidateConnections(candidate, quotaEntry(true), () => undefined, OPTIONS), + [] + ); +}); + +// 5. account selezionato al dispatch è uno di quelli verificati SAFE — proven +// at the pool-filter level: the returned candidate's `allowedConnectionIds` +// is rewritten to exactly the safe subset, which is the same field +// `autoStrategy.ts` (open-sse/services/combo/autoStrategy.ts:315-331) +// already intersects against before connection selection — so whatever it +// picks is provably a member of this set, by construction. +test("5: filterStrictZeroCostCandidates rewrites allowedConnectionIds to exactly the verified-SAFE subset", () => { + const candidate: StrictZeroCostCandidate = { + provider: "qp", + model: "qp-model", + connectionId: null, + allowedConnectionIds: ["A", "B", "C"], + }; + const resolve = (_provider: string, connectionId: string) => + connectionId === "B" ? safeState() : undefined; // only B is SAFE + const result = filterStrictZeroCostCandidates([candidate], { + enabled: true, + resolveFreeAccessState: resolve, + catalog: [quotaEntry(true)], + ...OPTIONS, + }); + assert.equal(result.length, 1); + assert.deepEqual( + result[0].allowedConnectionIds, + ["B"], + "dispatch (via autoStrategy.ts's own allowedConnectionIds intersection) can only ever select B — the one connection this filter actually verified" + ); + assert.equal( + result[0].allowedConnectionIds?.includes("A"), + false, + "A (UNKNOWN) must never remain selectable" + ); + assert.equal( + result[0].allowedConnectionIds?.includes("C"), + false, + "C (UNKNOWN) must never remain selectable" + ); +}); + +test("multi-account candidate with an unchanged safe set is returned as the SAME reference (identity contract)", () => { + const candidate: StrictZeroCostCandidate = { + provider: "qp", + model: "qp-model", + connectionId: null, + allowedConnectionIds: ["A"], + }; + const pool = [candidate]; + const result = filterStrictZeroCostCandidates(pool, { + enabled: true, + resolveFreeAccessState: () => safeState(), + catalog: [quotaEntry(true)], + ...OPTIONS, + }); + assert.equal(result, pool, "nothing was narrowed, so the pool array reference must be preserved"); + assert.equal(result[0], candidate, "the candidate object itself must be preserved, not cloned"); +}); + +test("single-connection candidate that fails is dropped, never returned with an empty allowedConnectionIds", () => { + const candidate: StrictZeroCostCandidate = { + provider: "qp", + model: "qp-model", + connectionId: REAL_CONN, + }; + const result = filterStrictZeroCostCandidates([candidate], { + enabled: true, + resolveFreeAccessState: () => undefined, + catalog: [quotaEntry(true)], + ...OPTIONS, + }); + assert.deepEqual(result, []); +}); diff --git a/tests/unit/autoCombo/strict-zero-cost-filter.test.ts b/tests/unit/autoCombo/strict-zero-cost-filter.test.ts new file mode 100644 index 0000000000..dc293a6dda --- /dev/null +++ b/tests/unit/autoCombo/strict-zero-cost-filter.test.ts @@ -0,0 +1,248 @@ +/** + * STRICT_ZERO_COST — regression guard for `strictZeroCostFilter.ts`, wired + * into `open-sse/services/autoCombo/virtualFactory.ts::buildPreparedPool` + * right after `filterPaidOnlyCandidates` (#6512). Mirrors the style/shape of + * `tests/unit/autoCombo/paid-model-filter-6512.test.ts`. + * + * Pure, dependency-light by design: `resolveFreeAccessState` is injected as a + * plain function, so no DB/network mocking is needed anywhere in this file. + * + * Connection-safety-specific cases (the keyless bypass and multi-account + * bugs found in code review) live in their own file, + * `strict-zero-cost-connection-safety.test.ts` — this file covers the + * single-connection / no-connection-ambiguity cases only. + */ +import { test } from "vitest"; +import assert from "node:assert/strict"; + +import { + evaluateCandidateConnections, + filterStrictZeroCostCandidates, + filterTosAvoidCandidates, + type FreeAccessState, +} from "../../../open-sse/services/autoCombo/strictZeroCostFilter.ts"; +import { SYNTHETIC_NOAUTH_CONNECTION_ID } from "../../../open-sse/services/autoCombo/resilienceCandidateFilter.ts"; +import { FREE_MODEL_BUDGETS } from "../../../open-sse/config/freeModelCatalog.ts"; + +const NOW = "2026-08-20T00:00:00.000Z"; +const nowMs = () => Date.parse(NOW); + +function freshState(overrides: Partial = {}): FreeAccessState { + return { + status: "SAFE", + remainingFreeAllowance: 50, + resetAt: null, + checkedAt: NOW, + ...overrides, + }; +} + +const BASE_OPTIONS = { minRemainingAllowance: 1, maxStateAgeMs: 180_000, now: nowMs }; + +const REAL_CONN = "conn-real-1"; + +// A real keyless entry from the catalog (felo-web, all models keyless/tos=avoid), +// as a genuine no-auth candidate (the only shape that legitimately gets the shortcut). +const KEYLESS = { + provider: "felo-web", + model: "felo-chat", + connectionId: SYNTHETIC_NOAUTH_CONNECTION_ID, +}; +// A real quota-based entry with hardStopGuaranteed: true (added by this feature), +// as a concrete single-connection candidate. +const QUOTA_SAFE = { provider: "groq", model: "llama-3.3-70b-versatile", connectionId: REAL_CONN }; +// A real quota-based entry WITHOUT hardStopGuaranteed (agentrouter: one-time-initial, +// no usage adapter, no documented "no credit card" claim — must never pass). +const QUOTA_UNGUARANTEED = { + provider: "agentrouter", + model: "claude-opus-5", + connectionId: REAL_CONN, +}; +// Not in FREE_MODEL_BUDGETS at all. +const UNKNOWN_MODEL = { + provider: "groq", + model: "definitely-not-cataloged", + connectionId: REAL_CONN, +}; +// A provider with no free models documented anywhere (mirrors paid-model-filter-6512's own PAID fixture). +const PAID = { provider: "openai", model: "gpt-4o", connectionId: REAL_CONN }; + +test("sanity: fixtures exist in the real catalog with the metadata these tests assume", () => { + const groqEntry = FREE_MODEL_BUDGETS.find( + (m) => m.provider === "groq" && m.modelId === "llama-3.3-70b-versatile" + ); + assert.equal(groqEntry?.hardStopGuaranteed, true, "groq must carry hardStopGuaranteed: true"); + const arEntry = FREE_MODEL_BUDGETS.find( + (m) => m.provider === "agentrouter" && m.modelId === "claude-opus-5" + ); + assert.notEqual( + arEntry?.hardStopGuaranteed, + true, + "agentrouter must NOT carry hardStopGuaranteed: true (no documented hard-stop guarantee)" + ); + const feloEntry = FREE_MODEL_BUDGETS.find( + (m) => m.provider === "felo-web" && m.modelId === "felo-chat" + ); + assert.equal(feloEntry?.freeType, "keyless"); + assert.equal(feloEntry?.tos, "avoid", "felo-web must be tos=avoid for the ToS-guard tests below"); +}); + +// 1. keyless SAFE (genuine no-auth candidate) → PASS +test("keyless candidate from the genuine no-auth path passes with no state at all", () => { + const entry = FREE_MODEL_BUDGETS.find( + (m) => m.provider === "felo-web" && m.modelId === "felo-chat" + ); + assert.deepEqual( + evaluateCandidateConnections(KEYLESS, entry, () => undefined, BASE_OPTIONS), + [SYNTHETIC_NOAUTH_CONNECTION_ID] + ); +}); + +// 2. paid → EXCLUDE +test("paid candidate (no free-model documentation at all) is excluded", () => { + assert.deepEqual( + evaluateCandidateConnections(PAID, undefined, () => undefined, BASE_OPTIONS), + [] + ); +}); + +// 3 & 4. unknown model / unknown provider → EXCLUDE +test("model absent from the free catalog is excluded even under a known provider", () => { + assert.deepEqual( + evaluateCandidateConnections(UNKNOWN_MODEL, undefined, () => undefined, BASE_OPTIONS), + [] + ); +}); + +// 5. quota SAFE + fresh + hardStop → PASS +test("quota-based candidate with hardStopGuaranteed, fresh SAFE state above threshold passes", () => { + const entry = FREE_MODEL_BUDGETS.find( + (m) => m.provider === "groq" && m.modelId === "llama-3.3-70b-versatile" + ); + assert.deepEqual( + evaluateCandidateConnections( + QUOTA_SAFE, + entry, + () => freshState({ remainingFreeAllowance: 40 }), + BASE_OPTIONS + ), + [REAL_CONN] + ); +}); + +// 6. quota exhausted → EXCLUDE +test("EXHAUSTED status excludes even with a fresh checkedAt", () => { + const entry = FREE_MODEL_BUDGETS.find( + (m) => m.provider === "groq" && m.modelId === "llama-3.3-70b-versatile" + ); + const state = freshState({ status: "EXHAUSTED", remainingFreeAllowance: 0 }); + assert.deepEqual( + evaluateCandidateConnections(QUOTA_SAFE, entry, () => state, BASE_OPTIONS), + [] + ); +}); + +// 7. usage adapter absent (no state resolvable) → EXCLUDE +test("quota-based candidate with no resolvable state is excluded, not assumed safe", () => { + const entry = FREE_MODEL_BUDGETS.find( + (m) => m.provider === "groq" && m.modelId === "llama-3.3-70b-versatile" + ); + assert.deepEqual( + evaluateCandidateConnections(QUOTA_SAFE, entry, () => undefined, BASE_OPTIONS), + [] + ); +}); + +// 8. usage API error → EXCLUDE (modeled as UNKNOWN status; freeAccessQuota.ts +// deletes the cache entry on a failed refresh, which resolves to `undefined` +// here — same assertion as #7, the important contract is "never falls back to SAFE"). +test("UNKNOWN status excludes", () => { + const entry = FREE_MODEL_BUDGETS.find( + (m) => m.provider === "groq" && m.modelId === "llama-3.3-70b-versatile" + ); + const state = freshState({ status: "UNKNOWN", remainingFreeAllowance: null }); + assert.deepEqual( + evaluateCandidateConnections(QUOTA_SAFE, entry, () => state, BASE_OPTIONS), + [] + ); +}); + +// 9. usage state stale → EXCLUDE +test("stale checkedAt excludes even when status is SAFE", () => { + const entry = FREE_MODEL_BUDGETS.find( + (m) => m.provider === "groq" && m.modelId === "llama-3.3-70b-versatile" + ); + const stale = freshState({ checkedAt: "2026-08-19T00:00:00.000Z" }); // >24h before NOW + assert.deepEqual( + evaluateCandidateConnections(QUOTA_SAFE, entry, () => stale, BASE_OPTIONS), + [] + ); +}); + +// 10 & 11. hardStopGuaranteed undefined / false → EXCLUDE +test("hardStopGuaranteed undefined excludes even with a perfect SAFE state", () => { + const entry = FREE_MODEL_BUDGETS.find( + (m) => m.provider === "agentrouter" && m.modelId === "claude-opus-5" + ); + assert.deepEqual( + evaluateCandidateConnections(QUOTA_UNGUARANTEED, entry, () => freshState(), BASE_OPTIONS), + [] + ); +}); + +test("hardStopGuaranteed explicitly false excludes", () => { + const entry = { + ...FREE_MODEL_BUDGETS[0], + hardStopGuaranteed: false, + freeType: "recurring-monthly" as const, + }; + assert.deepEqual( + evaluateCandidateConnections( + { provider: entry.provider, model: entry.modelId, connectionId: REAL_CONN }, + entry, + () => freshState(), + BASE_OPTIONS + ), + [] + ); +}); + +// 12 & 13. ToS guard, independent of economic evaluation +test("tos=avoid + excludeTosAvoid=true excludes a keyless-safe candidate", () => { + const result = filterTosAvoidCandidates([KEYLESS], true); + assert.deepEqual(result, [], "felo-web (tos=avoid) must be dropped when the guard is on"); +}); + +test("tos=avoid + excludeTosAvoid=false leaves normal economic evaluation untouched", () => { + const result = filterTosAvoidCandidates([KEYLESS, PAID], false); + assert.deepEqual(result, [KEYLESS, PAID], "guard off must be a pure identity pass-through"); +}); + +// Pool-level filter: default-off regression guard, mirrors paid-model-filter-6512's own first test. +test("freeAccessPolicy off (default) returns the pool UNCHANGED (identity, regression guard)", () => { + const pool = [KEYLESS, QUOTA_SAFE, PAID]; + const result = filterStrictZeroCostCandidates(pool, { + enabled: false, + resolveFreeAccessState: () => undefined, + ...BASE_OPTIONS, + }); + assert.equal(result, pool, "must return the exact same array reference when opt-in is off"); +}); + +test("strict pool filter keeps only keyless(no-auth) + guaranteed-quota-SAFE candidates", () => { + const pool = [KEYLESS, QUOTA_SAFE, QUOTA_UNGUARANTEED, PAID, UNKNOWN_MODEL]; + const result = filterStrictZeroCostCandidates(pool, { + enabled: true, + resolveFreeAccessState: (provider) => + provider === "groq" ? freshState({ remainingFreeAllowance: 40 }) : undefined, + ...BASE_OPTIONS, + }); + assert.deepEqual(result, [KEYLESS, QUOTA_SAFE]); +}); + +// Autodiscovery cases (14-19) are in +// tests/unit/autoCombo/strict-zero-cost-autodiscovery.test.ts, which fabricates +// synthetic FREE_MODEL_BUDGETS-shaped fixtures rather than real provider ids — +// see that file for why a fixture-based test is the correct tool here. +// Blocker 1 (keyless bypass) and Blocker 2 (multi-account) cases are in +// tests/unit/autoCombo/strict-zero-cost-connection-safety.test.ts.