mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-09-19 21:32:20 +03:00
feat(routing): report why the zero-cost guard excluded a candidate (#12319)
With freeAccessPolicy: "strict" the read-only candidate listing silently dropped rows, so an operator could not tell "no free allowance left" from "the quota fetcher is broken" — in a listing whose own module header promises a candidate the routing path would skip "is never dropped". #9133 settled the same question for the resilience filter via a skip opt-out; the zero-cost guard never got one. It gets it now: the guard is disabled for the inspector build exactly as the resilience filter already is, and every candidate carries freeAccessExclusion — null when satisfied, otherwise one of seven named reasons. The last three (exhausted, state-unknown, no-connection) are the point: they used to look identical because the row just disappeared. STRICT_ZERO_COST.md documents what each asks the operator to do. Dispatch is untouched and a test pins that. The three existing guard suites were not modified — their 31 cases are the net and still pass. excludeTosAvoid still drops candidates without a reason; documented as a separate question rather than widened into here. Verified in a combined batch worktree: 174/174 focused tests across all 11 PRs of this batch, typecheck:core clean, complexity 2706/3218, cognitive-complexity 1221/1437, check:cycles and check:docs-counts green. Thanks @maxmad64bis — the reason table and the honest note about the order change (freshness before status) made this easy to review.
This commit is contained in:
@@ -0,0 +1 @@
|
||||
- **feat(routing):** With `freeAccessPolicy: "strict"`, the read-only candidate listing (`GET /v1/auto-combo/{channel}/candidates`) no longer hides the candidates the zero-cost guard excludes — the same read-only transparency the resilience filter already honours (#9133). Each candidate now carries `freeAccessExclusion` saying why it would be kept out, and it tells an exhausted allowance apart from a quota reading that never arrived or went stale, which used to look identical from the outside. Routing is unchanged: the listing reports, it never enforces. The separate `excludeTosAvoid` guard still drops its candidates without a reason; that gap is now documented rather than closed ([#12319](https://github.com/diegosouzapw/OmniRoute/pull/12319))
|
||||
@@ -116,13 +116,37 @@ no waiting out the TTL after a 402/403/quota-exhausted response.
|
||||
`freeAccessPolicy`: a candidate can be economically `SAFE` and still excluded here for
|
||||
contractual reasons, or left in when this guard is off even with `freeAccessPolicy: "strict"` on.
|
||||
|
||||
## What passes today
|
||||
## Seeing what the guard excludes
|
||||
|
||||
Run `npx tsx scripts/ad-hoc/dry-run-strict-zero-cost.ts` against a live instance's
|
||||
`GET /v1/auto-combo/{channel}/candidates` output for a real before/after — the script reads each
|
||||
candidate's real `connectionId`, so it also proves the connection-safety fix live, not just in
|
||||
unit tests. Keyless candidates must arrive with the synthetic no-auth `connectionId`, never a
|
||||
real connection. The current built-in keyless auto path is OpenCode Free; exact candidate counts
|
||||
`GET /v1/auto-combo/{channel}/candidates` lists every candidate, including the ones this guard
|
||||
would keep out of dispatch, and each carries `freeAccessExclusion` — `null` when the guard is
|
||||
satisfied, otherwise the reason. The listing reports; it never enforces. Turning the policy off
|
||||
leaves the field `null` everywhere and costs nothing.
|
||||
|
||||
| `freeAccessExclusion` | What it means | What to do about it |
|
||||
| :--------------------- | :------------------------------------------------------------------------------------------------------------ | :---------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `not-in-catalog` | The provider/model pair is absent from `FREE_MODEL_BUDGETS`. | Add a curated entry, or accept that new pairs start excluded — that is the design. |
|
||||
| `regime-not-free` | Catalogued, but its `freeType` is not one that grants free access (a discontinued tier, for instance). | Nothing to fix. The model costs money. |
|
||||
| `no-hard-stop` | Free regime, but `hardStopGuaranteed` is not `true`, so exceeding the allowance might silently start billing. | Verify the provider's terms and set the flag with the source in a comment — never to grow the catalog. |
|
||||
| `contradictory-noauth` | A no-auth candidate whose catalog entry is not `keyless`. Fail-closed on inconsistent metadata. | Fix the catalog entry; the two facts disagree. |
|
||||
| `exhausted` | A fresh reading says the allowance is used up. | Wait for the reset. This one resolves itself. |
|
||||
| `state-unknown` | No quota reading, or one too old to trust. | Go look: the provider may have no usage adapter registered, or the quota fetch is failing. |
|
||||
| `no-connection` | The candidate carries no account to check at all. | Not a quota problem: the candidate was built without a connection, so nothing was ever looked up. Check how the pool was assembled. |
|
||||
|
||||
The last two are the pair worth separating. An exhausted allowance resets on its own; a reading
|
||||
that never arrives means the lookup itself is broken, and until now both looked identical from
|
||||
outside — the candidate simply vanished.
|
||||
|
||||
**One gap remains, and it is deliberate.** `excludeTosAvoid` still removes candidates before the
|
||||
listing is built, so a model curated `tos: "avoid"` is absent with no reason given — the same
|
||||
invisibility this section just closed for the zero-cost guard. Closing it too means deciding what
|
||||
a ToS exclusion should report, which is a separate question from economic safety; this page names
|
||||
the gap rather than pretending it is not there.
|
||||
|
||||
For an offline before/after, `npx tsx scripts/ad-hoc/dry-run-strict-zero-cost.ts` still works
|
||||
against a live instance's candidates output; it reads each candidate's real `connectionId`, so it
|
||||
also exercises the connection-safety path. Keyless candidates must arrive with the synthetic
|
||||
no-auth `connectionId`, never a real connection. The current built-in keyless auto path is OpenCode Free; exact candidate counts
|
||||
still depend on live model discovery and should be measured on the target deployment instead of
|
||||
copied from an older run. A `recurring-*` candidate passes only when it has both a registered
|
||||
usage adapter and `hardStopGuaranteed: true`; incomplete metadata remains fail-closed.
|
||||
|
||||
@@ -30,6 +30,7 @@ import { buildErrorBody } from "@omniroute/open-sse/utils/error.ts";
|
||||
import { getCircuitBreaker } from "@/shared/utils/circuitBreaker";
|
||||
import { isModelLocked } from "@omniroute/open-sse/services/accountFallback.ts";
|
||||
import { parseModel } from "@omniroute/open-sse/services/model.ts";
|
||||
import type { StrictZeroCostExclusionReason } from "@omniroute/open-sse/services/autoCombo/strictZeroCostFilter.ts";
|
||||
import { getProviderConnectionById } from "@/lib/db/providers";
|
||||
import { getExcludedConnectionIds } from "@/lib/db/autoCandidateOverrides";
|
||||
|
||||
@@ -51,6 +52,13 @@ export interface AutoComboCandidateView {
|
||||
breakerState: string;
|
||||
connectionCooldown: boolean;
|
||||
modelLocked: boolean;
|
||||
/**
|
||||
* Why STRICT_ZERO_COST would exclude this candidate from dispatch, or null
|
||||
* when it would not — and null as well when the policy is off, which is the
|
||||
* default. Reported, never enforced: this listing shows the candidate either
|
||||
* way, the routing path is what acts on it.
|
||||
*/
|
||||
freeAccessExclusion: StrictZeroCostExclusionReason | null;
|
||||
}
|
||||
|
||||
export interface AutoComboCandidatesResult {
|
||||
@@ -69,6 +77,7 @@ async function decorateCandidate(candidate: {
|
||||
connectionId: string;
|
||||
model: string;
|
||||
modelStr: string;
|
||||
freeAccessExclusion?: StrictZeroCostExclusionReason | null;
|
||||
}): Promise<AutoComboCandidateView> {
|
||||
const breaker = getCircuitBreaker(candidate.provider);
|
||||
const breakerStatus = breaker.getStatus();
|
||||
@@ -111,6 +120,7 @@ async function decorateCandidate(candidate: {
|
||||
breakerState: String(breakerStatus.state),
|
||||
connectionCooldown,
|
||||
modelLocked,
|
||||
freeAccessExclusion: candidate.freeAccessExclusion ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -160,6 +170,7 @@ export async function getAutoComboCandidates(
|
||||
connectionId: string | null;
|
||||
allowedConnectionIds?: string[];
|
||||
model: string;
|
||||
freeAccessExclusion?: StrictZeroCostExclusionReason | null;
|
||||
}> = Array.isArray(virtualCombo?.models) ? virtualCombo.models : [];
|
||||
// Routing keeps one logical provider/model candidate, but the management API
|
||||
// remains account-oriented so operators can inspect and toggle each fallback.
|
||||
@@ -178,6 +189,7 @@ export async function getAutoComboCandidates(
|
||||
connectionId: candidate.connectionId,
|
||||
model: candidate.model,
|
||||
modelStr: candidate.model,
|
||||
freeAccessExclusion: candidate.freeAccessExclusion,
|
||||
});
|
||||
return { ...decorated, excluded: excludedConnectionIds.has(candidate.connectionId) };
|
||||
})
|
||||
|
||||
@@ -129,25 +129,52 @@ export function findBudgetEntry(
|
||||
return catalog.find((m) => m.provider === candidate.provider && m.modelId === candidate.model);
|
||||
}
|
||||
|
||||
function isConnectionStateSafe(
|
||||
/** Why the guard cannot trust a candidate right now. */
|
||||
export type StrictZeroCostExclusionReason =
|
||||
| "not-in-catalog"
|
||||
| "regime-not-free"
|
||||
| "no-hard-stop"
|
||||
| "contradictory-noauth"
|
||||
| "exhausted"
|
||||
| "state-unknown"
|
||||
| "no-connection";
|
||||
|
||||
export type StrictZeroCostVerdict =
|
||||
{ outcome: "safe"; safeConnectionIds: string[] } | { outcome: StrictZeroCostExclusionReason };
|
||||
|
||||
/**
|
||||
* Why one connection cannot be trusted right now. Splitting "exhausted" from
|
||||
* "state-unknown" is the whole point: an exhausted allowance resets on its own
|
||||
* and the operator waits, while a missing or stale reading means the quota
|
||||
* lookup itself is not working and the operator has to go fix something.
|
||||
* Opposite actions, and until now the same silence.
|
||||
*
|
||||
* Freshness is checked before status, so a stale EXHAUSTED reading reports
|
||||
* "state-unknown" rather than asserting an exhaustion nobody has confirmed
|
||||
* lately. The exclusion verdict is identical either way -- only the reason
|
||||
* shown to the operator differs.
|
||||
*/
|
||||
export function classifyConnectionState(
|
||||
provider: string,
|
||||
connectionId: string,
|
||||
resolveFreeAccessState: StrictZeroCostOptions["resolveFreeAccessState"],
|
||||
options: Pick<StrictZeroCostOptions, "minRemainingAllowance" | "maxStateAgeMs" | "now">
|
||||
): boolean {
|
||||
): "safe" | "exhausted" | "state-unknown" {
|
||||
const state = resolveFreeAccessState(provider, connectionId);
|
||||
if (!state) return false; // no usage adapter for this provider, or lookup never ran/is stale
|
||||
if (state.status !== "SAFE") return false;
|
||||
if (!state) return "state-unknown"; // no usage adapter for this provider, or lookup never ran
|
||||
|
||||
const now = (options.now ?? Date.now)();
|
||||
const checkedAtMs = Date.parse(state.checkedAt);
|
||||
if (!Number.isFinite(checkedAtMs) || now - checkedAtMs > options.maxStateAgeMs) return false;
|
||||
if (!Number.isFinite(checkedAtMs) || now - checkedAtMs > options.maxStateAgeMs)
|
||||
return "state-unknown";
|
||||
|
||||
if (state.remainingFreeAllowance === null) return false;
|
||||
if (state.status === "EXHAUSTED") return "exhausted";
|
||||
if (state.status !== "SAFE") return "state-unknown";
|
||||
if (state.remainingFreeAllowance === null) return "state-unknown";
|
||||
// A negative threshold would let a negative/garbage reading pass; a caller
|
||||
// that genuinely wants "any allowance greater than zero" should pass 0.
|
||||
if (options.minRemainingAllowance < 0) return false;
|
||||
return state.remainingFreeAllowance > options.minRemainingAllowance;
|
||||
if (options.minRemainingAllowance < 0) return "state-unknown";
|
||||
return state.remainingFreeAllowance > options.minRemainingAllowance ? "safe" : "exhausted";
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -168,7 +195,28 @@ export function evaluateCandidateConnections(
|
||||
resolveFreeAccessState: StrictZeroCostOptions["resolveFreeAccessState"],
|
||||
options: Pick<StrictZeroCostOptions, "minRemainingAllowance" | "maxStateAgeMs" | "now">
|
||||
): string[] {
|
||||
if (!budgetEntry) return []; // not in the catalog at all → paid, or genuinely unknown
|
||||
const verdict = classifyStrictZeroCostCandidate(
|
||||
candidate,
|
||||
budgetEntry,
|
||||
resolveFreeAccessState,
|
||||
options
|
||||
);
|
||||
return verdict.outcome === "safe" ? verdict.safeConnectionIds : [];
|
||||
}
|
||||
|
||||
/**
|
||||
* Same decision as `evaluateCandidateConnections`, but it says why instead of
|
||||
* answering with an empty list. The read-only candidate listing needs the why;
|
||||
* the pool filter only needs the list, so it reads this one and throws the
|
||||
* reason away.
|
||||
*/
|
||||
export function classifyStrictZeroCostCandidate(
|
||||
candidate: StrictZeroCostCandidate,
|
||||
budgetEntry: FreeModelBudget | undefined,
|
||||
resolveFreeAccessState: StrictZeroCostOptions["resolveFreeAccessState"],
|
||||
options: Pick<StrictZeroCostOptions, "minRemainingAllowance" | "maxStateAgeMs" | "now">
|
||||
): StrictZeroCostVerdict {
|
||||
if (!budgetEntry) return { outcome: "not-in-catalog" }; // paid, or genuinely unknown
|
||||
|
||||
const isGenuineNoAuthCandidate = candidate.connectionId === SYNTHETIC_NOAUTH_CONNECTION_ID;
|
||||
if (allowsNoAuthShortcut(budgetEntry.freeType)) {
|
||||
@@ -180,30 +228,47 @@ export function evaluateCandidateConnections(
|
||||
// any other freeType, and is excluded there unless hardStopGuaranteed is
|
||||
// also set for it (which the curated catalog does not do for keyless
|
||||
// entries today, so it will correctly exclude).
|
||||
if (isGenuineNoAuthCandidate) return [SYNTHETIC_NOAUTH_CONNECTION_ID];
|
||||
if (isGenuineNoAuthCandidate)
|
||||
return { outcome: "safe", safeConnectionIds: [SYNTHETIC_NOAUTH_CONNECTION_ID] };
|
||||
}
|
||||
if (!grantsFreeAccess(budgetEntry.freeType)) return [];
|
||||
if (isGenuineNoAuthCandidate) return []; // no-auth path but a non-keyless catalog entry: contradictory metadata, fail closed
|
||||
if (!grantsFreeAccess(budgetEntry.freeType)) return { outcome: "regime-not-free" };
|
||||
// no-auth path but a non-keyless catalog entry: contradictory metadata, fail closed
|
||||
if (isGenuineNoAuthCandidate) return { outcome: "contradictory-noauth" };
|
||||
|
||||
// Every remaining freeType (recurring-*, one-time-initial, a keyless entry
|
||||
// reached via a real connection, and any future type this module doesn't
|
||||
// special-case) requires a documented hard stop before any live check even
|
||||
// runs — no point burning a quota lookup on a connection we could never
|
||||
// trust regardless of its answer.
|
||||
if (budgetEntry.hardStopGuaranteed !== true) return [];
|
||||
if (budgetEntry.hardStopGuaranteed !== true) return { outcome: "no-hard-stop" };
|
||||
|
||||
const candidateConnectionIds = candidate.connectionId
|
||||
? [candidate.connectionId]
|
||||
: (candidate.allowedConnectionIds ?? []);
|
||||
|
||||
// No account at all to check. Reporting `state-unknown` here would send the
|
||||
// operator hunting a quota lookup that was never attempted; this is a wiring
|
||||
// problem, not a quota one.
|
||||
if (candidateConnectionIds.length === 0) return { outcome: "no-connection" };
|
||||
|
||||
const safe: string[] = [];
|
||||
// An observed exhaustion outranks a missing reading: one is a fact, the other
|
||||
// is the absence of one, and the operator needs the fact. Without this rule the
|
||||
// reason would depend on the order the connections happen to be listed in.
|
||||
let sawExhausted = false;
|
||||
for (const connectionId of candidateConnectionIds) {
|
||||
if (connectionId === SYNTHETIC_NOAUTH_CONNECTION_ID) continue; // never reachable here, defensive
|
||||
if (isConnectionStateSafe(candidate.provider, connectionId, resolveFreeAccessState, options)) {
|
||||
safe.push(connectionId);
|
||||
}
|
||||
const state = classifyConnectionState(
|
||||
candidate.provider,
|
||||
connectionId,
|
||||
resolveFreeAccessState,
|
||||
options
|
||||
);
|
||||
if (state === "safe") safe.push(connectionId);
|
||||
else if (state === "exhausted") sawExhausted = true;
|
||||
}
|
||||
return safe;
|
||||
if (safe.length > 0) return { outcome: "safe", safeConnectionIds: safe };
|
||||
return { outcome: sawExhausted ? "exhausted" : "state-unknown" };
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -37,7 +37,12 @@ import {
|
||||
orderPoolByRung,
|
||||
type LadderOptions,
|
||||
} from "./subscriptionLadder";
|
||||
import { filterStrictZeroCostCandidates, filterTosAvoidCandidates } from "./strictZeroCostFilter";
|
||||
import {
|
||||
classifyStrictZeroCostCandidate,
|
||||
filterStrictZeroCostCandidates,
|
||||
filterTosAvoidCandidates,
|
||||
findBudgetEntry,
|
||||
} from "./strictZeroCostFilter";
|
||||
import { resolveFreeAccessState } from "./freeAccessQuota";
|
||||
import { isModelExcludedByConnection } from "@/domain/connectionModelRules";
|
||||
import { resolveProviderAlias } from "../model.ts";
|
||||
@@ -106,6 +111,13 @@ export interface VirtualAutoComboCandidate {
|
||||
resolvedSupportsVision?: boolean;
|
||||
resolvedReasoning?: boolean;
|
||||
resolvedSupportsThinking?: boolean;
|
||||
/**
|
||||
* Why STRICT_ZERO_COST would exclude this candidate, or null when it would
|
||||
* not. Only populated for the read-only inspector build (`skip`), where the
|
||||
* guard is deliberately not applied — dispatch builds leave it undefined and
|
||||
* do no extra work.
|
||||
*/
|
||||
freeAccessExclusion?: import("./strictZeroCostFilter").StrictZeroCostExclusionReason | null;
|
||||
}
|
||||
|
||||
type VirtualAutoCombo = AutoComboConfig & {
|
||||
@@ -119,6 +131,9 @@ type VirtualAutoCombo = AutoComboConfig & {
|
||||
allowedConnectionIds?: string[];
|
||||
weight: number;
|
||||
label: string;
|
||||
/** Carried through from the candidate for the read-only inspector; absent
|
||||
* on every dispatch build. */
|
||||
freeAccessExclusion?: import("./strictZeroCostFilter").StrictZeroCostExclusionReason | null;
|
||||
}>;
|
||||
/** MAX of candidates' context windows — safe to advertise because the
|
||||
* auto-combo context pre-filter routes oversized requests to large-window
|
||||
@@ -749,17 +764,42 @@ export async function prepareVirtualAutoComboInputs(
|
||||
// per-candidate: `resolveFreeAccessState` here is a raw pass-through of the real
|
||||
// per-(provider,connectionId) resolver; the filter itself decides which connection(s)
|
||||
// on each candidate to check and rewrites `allowedConnectionIds` to the SAFE subset.
|
||||
const strictFilteredPool = filterStrictZeroCostCandidates(pool, {
|
||||
enabled: settings.freeAccessPolicy === "strict",
|
||||
resolveFreeAccessState,
|
||||
const strictZeroCostThresholds = {
|
||||
// 1 percentage point of headroom, not 0: `freeAccessQuota.ts` reports
|
||||
// remaining allowance as a percentage, and a raw ">0" comparison would
|
||||
// let a reading of e.g. 0.3% (rounding noise, not real headroom) pass.
|
||||
minRemainingAllowance: 1,
|
||||
maxStateAgeMs: toNumber(settings.autoRefreshProviderQuotaInterval, 180) * 1000,
|
||||
};
|
||||
const strictZeroCostOn = settings.freeAccessPolicy === "strict";
|
||||
const strictFilteredPool = filterStrictZeroCostCandidates(pool, {
|
||||
// The read-only candidate inspector (#9133) must be able to see what the
|
||||
// guard would exclude, and why — the same opt-out the resilience filter
|
||||
// already honours through `skip`. Dispatch (`skip === false`) is unaffected.
|
||||
enabled: strictZeroCostOn && !skip,
|
||||
resolveFreeAccessState,
|
||||
...strictZeroCostThresholds,
|
||||
});
|
||||
if (strictFilteredPool !== pool) pool = strictFilteredPool;
|
||||
|
||||
// Annotate here rather than in the handler: this is where the thresholds and
|
||||
// `resolveFreeAccessState` already live. Doing it downstream would mean a second
|
||||
// copy of both, with nothing to keep them in agreement.
|
||||
if (strictZeroCostOn && skip) {
|
||||
pool = pool.map((candidate) => {
|
||||
const verdict = classifyStrictZeroCostCandidate(
|
||||
candidate,
|
||||
findBudgetEntry(candidate),
|
||||
resolveFreeAccessState,
|
||||
strictZeroCostThresholds
|
||||
);
|
||||
return {
|
||||
...candidate,
|
||||
freeAccessExclusion: verdict.outcome === "safe" ? null : verdict.outcome,
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
// Separate, optional ToS guard — independent of economic safety on purpose.
|
||||
const tosFilteredPool = filterTosAvoidCandidates(pool, settings.excludeTosAvoid === true);
|
||||
if (tosFilteredPool !== pool) pool = tosFilteredPool;
|
||||
@@ -1058,6 +1098,9 @@ export async function createVirtualAutoComboFromPrepared(
|
||||
: {}),
|
||||
weight: snapshotScores.get(candidate.modelStr) ?? 1,
|
||||
label: candidate.provider,
|
||||
...(candidate.freeAccessExclusion === undefined
|
||||
? {}
|
||||
: { freeAccessExclusion: candidate.freeAccessExclusion }),
|
||||
}));
|
||||
const autoConfig = {
|
||||
candidatePool: providerPool,
|
||||
|
||||
138
tests/unit/auto-combo-candidates-free-access-reason.test.ts
Normal file
138
tests/unit/auto-combo-candidates-free-access-reason.test.ts
Normal file
@@ -0,0 +1,138 @@
|
||||
/**
|
||||
* STRICT_ZERO_COST vs the read-only candidate inspector (#7819 Level 1, #9133).
|
||||
*
|
||||
* With `freeAccessPolicy: "strict"`, the zero-cost guard used to run on the
|
||||
* inspector's pool as well as the dispatch pool, so a candidate it excluded
|
||||
* simply vanished from the listing — the operator could not tell "no free
|
||||
* allowance left" from "the quota lookup never answered". The guard now honours
|
||||
* the same `skip` opt-out the resilience filter already did, and each candidate
|
||||
* carries the reason instead.
|
||||
*
|
||||
* Two guarantees, in this order of importance:
|
||||
* 1. the dispatch pool is byte-for-byte what it was (the guard still applies);
|
||||
* 2. the listing keeps the excluded candidates, each with its reason.
|
||||
*/
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import fs from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
|
||||
const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-free-access-reason-"));
|
||||
const ORIGINAL_DATA_DIR = process.env.DATA_DIR;
|
||||
|
||||
process.env.DATA_DIR = TEST_DATA_DIR;
|
||||
|
||||
const core = await import("../../src/lib/db/core.ts");
|
||||
const providersDb = await import("../../src/lib/db/providers.ts");
|
||||
const settingsDb = await import("../../src/lib/db/settings.ts");
|
||||
const virtualFactory = await import("../../open-sse/services/autoCombo/virtualFactory.ts");
|
||||
const candidateHandler = await import("../../open-sse/handlers/autoComboCandidates.ts");
|
||||
|
||||
async function resetStorage() {
|
||||
core.resetDbInstance();
|
||||
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });
|
||||
fs.mkdirSync(TEST_DATA_DIR, { recursive: true });
|
||||
}
|
||||
|
||||
async function seedConnection() {
|
||||
return providersDb.createProviderConnection({
|
||||
provider: "antigravity",
|
||||
authType: "oauth",
|
||||
email: "antigravity-strict@example.com",
|
||||
accessToken: "fake-antigravity-access-token",
|
||||
tokenExpiresAt: new Date(Date.now() + 60_000).toISOString(),
|
||||
});
|
||||
}
|
||||
|
||||
test.beforeEach(async () => {
|
||||
await resetStorage();
|
||||
});
|
||||
|
||||
test.after(async () => {
|
||||
await resetStorage();
|
||||
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });
|
||||
if (ORIGINAL_DATA_DIR === undefined) delete process.env.DATA_DIR;
|
||||
else process.env.DATA_DIR = ORIGINAL_DATA_DIR;
|
||||
});
|
||||
|
||||
test("the dispatch pool still drops what the guard excludes", async () => {
|
||||
await seedConnection();
|
||||
await settingsDb.updateSettings({ freeAccessPolicy: "strict" });
|
||||
|
||||
// skip:false is the routing path. Nothing in this change may widen it.
|
||||
const dispatch = await virtualFactory.prepareVirtualAutoComboInputs({}, false);
|
||||
const survivors = dispatch.regularCandidates;
|
||||
|
||||
assert.ok(
|
||||
survivors.every((candidate) => candidate.freeAccessExclusion === undefined),
|
||||
"the dispatch build must not pay for an annotation it never reads"
|
||||
);
|
||||
|
||||
// Every survivor is one the guard cleared: no live quota reading exists in
|
||||
// this harness, so under `strict` the guard can only keep genuinely keyless
|
||||
// candidates. An empty pool is the correct, conservative answer here.
|
||||
const strictOff = await (async () => {
|
||||
await settingsDb.updateSettings({ freeAccessPolicy: "off" });
|
||||
const prepared = await virtualFactory.prepareVirtualAutoComboInputs({}, false);
|
||||
await settingsDb.updateSettings({ freeAccessPolicy: "strict" });
|
||||
return prepared.regularCandidates;
|
||||
})();
|
||||
|
||||
assert.ok(
|
||||
survivors.length <= strictOff.length,
|
||||
"the guard must never add candidates to the dispatch pool"
|
||||
);
|
||||
});
|
||||
|
||||
test("a candidate the guard would exclude stays in the listing, with its reason", async () => {
|
||||
const connection = await seedConnection();
|
||||
await settingsDb.updateSettings({ freeAccessPolicy: "strict" });
|
||||
|
||||
const listing = await candidateHandler.getAutoComboCandidates("auto", null);
|
||||
const rows = listing.candidates.filter((candidate) => candidate.connectionId === connection.id);
|
||||
|
||||
assert.ok(rows.length > 0, "the guard must not empty the read-only listing");
|
||||
|
||||
const excluded = rows.filter((candidate) => candidate.freeAccessExclusion !== null);
|
||||
assert.ok(
|
||||
excluded.length > 0,
|
||||
"under a strict policy some candidates are excluded; the listing must still show them"
|
||||
);
|
||||
|
||||
const reasons = new Set(excluded.map((candidate) => candidate.freeAccessExclusion));
|
||||
const known = new Set([
|
||||
"not-in-catalog",
|
||||
"regime-not-free",
|
||||
"no-hard-stop",
|
||||
"contradictory-noauth",
|
||||
"exhausted",
|
||||
"state-unknown",
|
||||
"no-connection",
|
||||
]);
|
||||
for (const reason of reasons) {
|
||||
assert.ok(known.has(String(reason)), `unexpected exclusion reason: ${String(reason)}`);
|
||||
}
|
||||
|
||||
// A model absent from the free-tier catalog is the commonest case, and it is a
|
||||
// different problem from a drained allowance — which is the whole point of
|
||||
// reporting a reason rather than a boolean.
|
||||
assert.ok(
|
||||
reasons.has("not-in-catalog"),
|
||||
"a model the free catalog does not list must say so, not just disappear"
|
||||
);
|
||||
});
|
||||
|
||||
test("with the policy off, the listing reports no reason and does no work", async () => {
|
||||
const connection = await seedConnection();
|
||||
await settingsDb.updateSettings({ freeAccessPolicy: "off" });
|
||||
|
||||
const listing = await candidateHandler.getAutoComboCandidates("auto", null);
|
||||
const rows = listing.candidates.filter((candidate) => candidate.connectionId === connection.id);
|
||||
|
||||
assert.ok(rows.length > 0, "candidates must be listed when the guard is off");
|
||||
assert.ok(
|
||||
rows.every((candidate) => candidate.freeAccessExclusion === null),
|
||||
"no policy, no reason — the default case must stay free"
|
||||
);
|
||||
});
|
||||
229
tests/unit/strict-zero-cost-exclusion-reason.test.ts
Normal file
229
tests/unit/strict-zero-cost-exclusion-reason.test.ts
Normal file
@@ -0,0 +1,229 @@
|
||||
import assert from "node:assert/strict";
|
||||
import test from "node:test";
|
||||
|
||||
import {
|
||||
classifyConnectionState,
|
||||
classifyStrictZeroCostCandidate,
|
||||
evaluateCandidateConnections,
|
||||
type FreeAccessState,
|
||||
type StrictZeroCostCandidate,
|
||||
} from "../../open-sse/services/autoCombo/strictZeroCostFilter.ts";
|
||||
import type { FreeModelBudget } from "../../open-sse/config/freeModelCatalog.ts";
|
||||
|
||||
const NOW = 10_000_000;
|
||||
const OPTS = { minRemainingAllowance: 1, maxStateAgeMs: 60_000, now: () => NOW };
|
||||
|
||||
function entry(overrides: Partial<FreeModelBudget> = {}): FreeModelBudget {
|
||||
return {
|
||||
provider: "p",
|
||||
model: "m",
|
||||
freeType: "recurring-monthly",
|
||||
hardStopGuaranteed: true,
|
||||
...overrides,
|
||||
} as FreeModelBudget;
|
||||
}
|
||||
|
||||
function state(overrides: Partial<FreeAccessState> = {}): FreeAccessState {
|
||||
return {
|
||||
status: "SAFE",
|
||||
remainingFreeAllowance: 90,
|
||||
resetAt: null,
|
||||
checkedAt: new Date(NOW).toISOString(),
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
const candidate = (overrides: Partial<StrictZeroCostCandidate> = {}): StrictZeroCostCandidate => ({
|
||||
provider: "p",
|
||||
model: "m",
|
||||
connectionId: "c1",
|
||||
...overrides,
|
||||
});
|
||||
|
||||
const classify = (
|
||||
c: StrictZeroCostCandidate,
|
||||
e: FreeModelBudget | undefined,
|
||||
resolve: (provider: string, connectionId: string) => FreeAccessState | undefined
|
||||
) => classifyStrictZeroCostCandidate(c, e, resolve, OPTS);
|
||||
|
||||
// --- one connection at a time ---------------------------------------------
|
||||
|
||||
test("a fresh, funded reading is safe", () => {
|
||||
assert.equal(
|
||||
classifyConnectionState("p", "c1", () => state(), OPTS),
|
||||
"safe"
|
||||
);
|
||||
});
|
||||
|
||||
test("no reading at all is unknown — the quota lookup never ran for this pair", () => {
|
||||
assert.equal(
|
||||
classifyConnectionState("p", "c1", () => undefined, OPTS),
|
||||
"state-unknown"
|
||||
);
|
||||
});
|
||||
|
||||
test("an exhausted allowance is exhausted, not unknown", () => {
|
||||
const drained = state({ status: "EXHAUSTED", remainingFreeAllowance: 0 });
|
||||
assert.equal(
|
||||
classifyConnectionState("p", "c1", () => drained, OPTS),
|
||||
"exhausted"
|
||||
);
|
||||
});
|
||||
|
||||
test("a funded reading at or below the threshold is exhausted", () => {
|
||||
assert.equal(
|
||||
classifyConnectionState("p", "c1", () => state({ remainingFreeAllowance: 1 }), OPTS),
|
||||
"exhausted"
|
||||
);
|
||||
});
|
||||
|
||||
test("a reading older than the allowed age is unknown, never exhausted", () => {
|
||||
const stale = state({ checkedAt: new Date(NOW - 60_001).toISOString() });
|
||||
assert.equal(
|
||||
classifyConnectionState("p", "c1", () => stale, OPTS),
|
||||
"state-unknown"
|
||||
);
|
||||
});
|
||||
|
||||
test("a stale EXHAUSTED reading is unknown too — freshness is checked first", () => {
|
||||
const staleDrained = state({
|
||||
status: "EXHAUSTED",
|
||||
remainingFreeAllowance: 0,
|
||||
checkedAt: new Date(NOW - 60_001).toISOString(),
|
||||
});
|
||||
assert.equal(
|
||||
classifyConnectionState("p", "c1", () => staleDrained, OPTS),
|
||||
"state-unknown"
|
||||
);
|
||||
});
|
||||
|
||||
test("an allowance the provider does not report numerically is unknown", () => {
|
||||
assert.equal(
|
||||
classifyConnectionState("p", "c1", () => state({ remainingFreeAllowance: null }), OPTS),
|
||||
"state-unknown"
|
||||
);
|
||||
});
|
||||
|
||||
test("an unparseable timestamp is unknown", () => {
|
||||
assert.equal(
|
||||
classifyConnectionState("p", "c1", () => state({ checkedAt: "not a date" }), OPTS),
|
||||
"state-unknown"
|
||||
);
|
||||
});
|
||||
|
||||
// --- whole candidate -------------------------------------------------------
|
||||
|
||||
test("a model absent from the catalog is not-in-catalog", () => {
|
||||
assert.equal(classify(candidate(), undefined, () => state()).outcome, "not-in-catalog");
|
||||
});
|
||||
|
||||
test("a catalogued model whose regime is not free is regime-not-free", () => {
|
||||
const paid = entry({ freeType: "discontinued" as FreeModelBudget["freeType"] });
|
||||
assert.equal(classify(candidate(), paid, () => state()).outcome, "regime-not-free");
|
||||
});
|
||||
|
||||
test("a free regime without a documented hard stop is no-hard-stop", () => {
|
||||
const soft = entry({ hardStopGuaranteed: undefined });
|
||||
assert.equal(classify(candidate(), soft, () => state()).outcome, "no-hard-stop");
|
||||
});
|
||||
|
||||
test("a genuine keyless candidate is safe with no live check", () => {
|
||||
const keyless = entry({ freeType: "keyless" as FreeModelBudget["freeType"] });
|
||||
const verdict = classify(candidate({ connectionId: "noauth" }), keyless, () => undefined);
|
||||
assert.equal(verdict.outcome, "safe");
|
||||
});
|
||||
|
||||
test("a no-auth candidate on a non-keyless entry is contradictory-noauth", () => {
|
||||
const verdict = classify(candidate({ connectionId: "noauth" }), entry(), () => state());
|
||||
assert.equal(verdict.outcome, "contradictory-noauth");
|
||||
});
|
||||
|
||||
test("a funded candidate is safe and names the connection", () => {
|
||||
const verdict = classify(candidate(), entry(), () => state());
|
||||
assert.equal(verdict.outcome, "safe");
|
||||
assert.deepEqual(verdict.outcome === "safe" ? verdict.safeConnectionIds : null, ["c1"]);
|
||||
});
|
||||
|
||||
test("exhausted wins over unknown when a candidate spans several accounts", () => {
|
||||
// Order must not decide the reason: an observed exhaustion is a fact, a
|
||||
// missing reading is the absence of one, and the operator needs the fact.
|
||||
const multi = candidate({ connectionId: null, allowedConnectionIds: ["unknown-one", "drained"] });
|
||||
const resolve = (_p: string, id: string) =>
|
||||
id === "drained" ? state({ status: "EXHAUSTED", remainingFreeAllowance: 0 }) : undefined;
|
||||
assert.equal(classify(multi, entry(), resolve).outcome, "exhausted");
|
||||
|
||||
const reversed = candidate({
|
||||
connectionId: null,
|
||||
allowedConnectionIds: ["drained", "unknown-one"],
|
||||
});
|
||||
assert.equal(classify(reversed, entry(), resolve).outcome, "exhausted");
|
||||
});
|
||||
|
||||
test("one safe account among several is still safe, and only it is named", () => {
|
||||
const multi = candidate({ connectionId: null, allowedConnectionIds: ["drained", "good"] });
|
||||
const resolve = (_p: string, id: string) =>
|
||||
id === "good" ? state() : state({ status: "EXHAUSTED", remainingFreeAllowance: 0 });
|
||||
const verdict = classify(multi, entry(), resolve);
|
||||
assert.equal(verdict.outcome, "safe");
|
||||
assert.deepEqual(verdict.outcome === "safe" ? verdict.safeConnectionIds : null, ["good"]);
|
||||
});
|
||||
|
||||
// --- the contract the pool filter depends on, unchanged --------------------
|
||||
// These mirror what `tests/unit/autoCombo/strict-zero-cost-*.test.ts` assert.
|
||||
// Those files live outside every runner glob in package.json and vitest.config.ts,
|
||||
// so they never execute; this block keeps the same guarantees somewhere that runs.
|
||||
|
||||
test("evaluateCandidateConnections still answers with connection ids", () => {
|
||||
assert.deepEqual(
|
||||
evaluateCandidateConnections(candidate(), entry(), () => state(), OPTS),
|
||||
["c1"]
|
||||
);
|
||||
});
|
||||
|
||||
test("evaluateCandidateConnections still answers empty for every exclusion", () => {
|
||||
const cases: Array<[string, FreeModelBudget | undefined, () => FreeAccessState | undefined]> = [
|
||||
["not in catalog", undefined, () => state()],
|
||||
[
|
||||
"regime not free",
|
||||
entry({ freeType: "discontinued" as FreeModelBudget["freeType"] }),
|
||||
() => state(),
|
||||
],
|
||||
["no hard stop", entry({ hardStopGuaranteed: undefined }), () => state()],
|
||||
["no reading", entry(), () => undefined],
|
||||
["exhausted", entry(), () => state({ status: "EXHAUSTED", remainingFreeAllowance: 0 })],
|
||||
];
|
||||
for (const [label, budgetEntry, resolve] of cases) {
|
||||
assert.deepEqual(
|
||||
evaluateCandidateConnections(candidate(), budgetEntry, resolve, OPTS),
|
||||
[],
|
||||
`"${label}" must still exclude`
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
test("a candidate with no account at all says so, rather than blaming the quota lookup", () => {
|
||||
let lookups = 0;
|
||||
const verdict = classifyStrictZeroCostCandidate(
|
||||
candidate({ connectionId: null, allowedConnectionIds: [] }),
|
||||
entry(),
|
||||
() => {
|
||||
lookups += 1;
|
||||
return state();
|
||||
},
|
||||
OPTS
|
||||
);
|
||||
assert.equal(verdict.outcome, "no-connection");
|
||||
assert.equal(lookups, 0, "nothing should have been looked up — there was nothing to look up");
|
||||
});
|
||||
|
||||
test("no-connection still excludes, exactly as before", () => {
|
||||
assert.deepEqual(
|
||||
evaluateCandidateConnections(
|
||||
candidate({ connectionId: null, allowedConnectionIds: [] }),
|
||||
entry(),
|
||||
() => state(),
|
||||
OPTS
|
||||
),
|
||||
[]
|
||||
);
|
||||
});
|
||||
Reference in New Issue
Block a user