Files
OmniRoute/open-sse/services/opencodeQuotaFetcher.ts
Diego Rodrigues de Sa e Souza 92a083ab8c fix(resilience): honor dashboard quota snapshots in opencode-go preflight (#11234) (#11267)
The opencode-go quota preflight ignored the dashboard quota snapshots, so
priority combos kept selecting connections whose weekly window was already
drained. Two gaps, two fixes:

A) fetchOpencodeQuota only consulted the live upstream endpoint, which has
   no public quota API (404 — the module JSDoc already admits this). Every
   preflight therefore evaluated null and proceeded (fail-open) even with a
   connection at 0% weekly remaining in plain sight on the dashboard. The
   fetcher now synthesizes its triple-window QuotaInfo from the cached
   dashboard snapshots when the live endpoint yields nothing, mapping
   session→window_5h, weekly→window_weekly, mcp_monthly→window_monthly and
   mirroring getQuotaWindowStatus semantics (expired resetAt = window rolled
   over = must not count as exhausted; fractionReported=false = unknown,
   never exhaustion). Read-only via src/domain/quotaCache.ts accessors —
   never SQL, never a re-scrape on the hot path — and gated on the
   connection actually having dashboard scrape config, so unconfigured
   connections never touch the snapshot store. Fail-open is preserved:
   no snapshots → null, exactly as before. The quotaCache import is dynamic
   because a static edge would close an initialization cycle
   (fetcher → quotaCache → usage.ts → usage/opencode.ts → fetcher).

B) The sibling-selection latency gate in getProviderCredentialsWithQuotaPreflight
   never consulted resilience.quotaPreflight.enabled
   (QUOTA_PREFLIGHT_CUTOFF_ENABLED) — that flag only armed the auto-strategy
   candidate builder and the per-target cutoff for pinned connections, so a
   priority combo over sibling opencode-go connections (connectionId null at
   combo level) skipped preflight entirely. The flag now arms the gate as
   well; the default (flag off) is unchanged.

TDD (Hard Rule #18), tests/unit/quota-exhaustion-cutoff-opencode.test.ts:
- fetcher 404 + seeded snapshots weekly=0%/session=80% → cutoff blocks
  (RED before, GREEN after); also asserts the bridge is read-only (single
  upstream fetch, no re-scrape).
- weekly 0% with next_reset_at in the past → not blocked (window dropped).
- per-window threshold override resolves against the mapped window_weekly
  key (50% override blocks at 40% remaining; factory 2% does not).
- fail-open guard: configured dashboard with no snapshots still returns null.
- selector level: flag on, two opencode-go sisters, priority-1 exhausted →
  selection skips to the healthy one (RED before, GREEN after).

Sibling suites green: opencode-quota-fetcher (18), quota-preflight,
combo-priority-quota-exhaustion-cutoff-5923, issue-6686, 8431, throttle-6911,
sse-auth*, quota fetchers, combo strategies, snapshot/hydration tests
(~500 tests). eslint (with suppressions) and typecheck:core clean.

Closes #11234

Co-authored-by: Xiangzhe <bakryun0718@proton.me>
2026-08-23 14:28:19 -03:00

483 lines
19 KiB
TypeScript

/**
* opencodeQuotaFetcher.ts — OpenCode Go / OpenCode / OpenCode Zen Quota Fetcher
*
* Implements QuotaFetcher for the opencode-go, opencode, and opencode-zen providers
* (quotaPreflight.ts + quotaMonitor.ts).
*
* OpenCode Go has THREE independent quota windows per subscription:
* - 5-hour (rolling): $12 of usage
* - Weekly: $30 of usage
* - Monthly: $60 of usage
*
* Upstream endpoint (defensive — no public API exists yet):
* GET https://opencode.ai/zen/go/v1/quota
* Authorization: Bearer <apiKey>
*
* Expected response shape:
* {
* quota: {
* window_5h: { used: number, limit: number, reset_at: number | null },
* window_weekly: { used: number, limit: number, reset_at: number | null },
* window_monthly: { used: number, limit: number, reset_at: number | null }
* }
* }
*
* NOTE: As of 2026, no public quota API exists for OpenCode Go / OpenCode Zen
* (tracked upstream in anomalyco/opencode#16017, #18648, #31084). The default
* endpoint currently returns HTTP 404. This fetcher is implemented defensively
* so that the dashboard shows "No quota data" gracefully rather than crashing,
* and so that when an endpoint is finally published, only the URL constant
* needs to change.
*
* On a 404 response we log ONE console.warn (latched per process — not per
* request) pointing at the upstream tracking issues, then cache the
* "endpoint unavailable" result for 5 minutes to avoid hammering. On any other
* non-200 / parse failure we return null (fail-open) silently. The first
* call from each server boot is what the operator is most likely to see, so
* we make it count.
*
* Cache: in-memory TTL (60s for success, 5 min for 404).
*
* Override: set OMNIROUTE_OPENCODE_QUOTA_URL to a working endpoint. If
* OpenCode ships a public endpoint (likely in the form of the merged PR
* #16513), the maintainer can update the default.
*
* Registration: call registerOpencodeQuotaFetcher() once at server startup.
*/
import { registerQuotaFetcher, registerQuotaWindows, type QuotaInfo } from "./quotaPreflight.ts";
import { registerMonitorFetcher } from "./quotaMonitor.ts";
import { throttleQuotaFetch } from "./quotaFetchThrottle.ts";
import { resolveOpenCodeGoDashboardConfig } from "./opencodeOllamaUsage.ts";
// OpenCode quota endpoint — same key works across opencode, opencode-go, opencode-zen
// Default points at /zen/go/v1/quota which returns 404 today (no public quota API yet,
// tracked in anomalyco/opencode#16017). Set OMNIROUTE_OPENCODE_QUOTA_URL to override.
const OPENCODE_QUOTA_URL =
process.env.OMNIROUTE_OPENCODE_QUOTA_URL ?? "https://opencode.ai/zen/go/v1/quota";
// Cache TTL — matches Codex / DeepSeek / Bailian pattern (60s)
const CACHE_TTL_MS = 60_000;
// TTL for cached "endpoint unavailable" results (404) — longer to avoid hammering
// a non-existent endpoint
const NO_ENDPOINT_TTL_MS = 5 * 60_000; // 5 minutes
// Window keys as surfaced to the dashboard and quota-window registry
export const OPENCODE_WINDOW_5H = "window_5h";
export const OPENCODE_WINDOW_WEEKLY = "window_weekly";
export const OPENCODE_WINDOW_MONTHLY = "window_monthly";
// Triple-window quota info
export interface OpencodeTripleWindowQuota extends QuotaInfo {
window5h: { percentUsed: number; resetAt: string | null };
windowWeekly: { percentUsed: number; resetAt: string | null };
windowMonthly: { percentUsed: number; resetAt: string | null };
limitReached: boolean;
}
interface CacheEntry {
quota: OpencodeTripleWindowQuota | null;
fetchedAt: number;
/** true when quota is null because the upstream endpoint returned 404 */
noEndpoint?: boolean;
}
// In-memory cache: connectionId → { quota, fetchedAt }
const quotaCache = new Map<string, CacheEntry>();
// One-time 404 warning per URL (avoids spamming on every request)
const _warned404Urls = new Set<string>();
/**
* Reset the 404-warning latch (test-only).
* Exported for unit tests that want to verify the warning fires on each fresh
* 404 response.
*/
export function _resetWarned404Urls(): void {
_warned404Urls.clear();
}
/**
* Check whether a URL has had its 404 warning already emitted (test-only).
*/
export function _hasWarned404(url: string): boolean {
return _warned404Urls.has(url);
}
// Auto-cleanup stale entries every 5 minutes
const _cacheCleanup = setInterval(() => {
const now = Date.now();
for (const [key, entry] of quotaCache) {
if (now - entry.fetchedAt > CACHE_TTL_MS * 5) {
quotaCache.delete(key);
}
}
}, 5 * 60_000);
if (typeof _cacheCleanup === "object" && "unref" in _cacheCleanup) {
(_cacheCleanup as { unref?: () => void }).unref?.();
}
// ─── Helpers ──────────────────────────────────────────────────────────────────
function toNumber(value: unknown, fallback = 0): number {
if (typeof value === "number" && Number.isFinite(value)) return value;
if (typeof value === "string") {
const parsed = parseFloat(value);
if (Number.isFinite(parsed)) return parsed;
}
return fallback;
}
function toRecord(value: unknown): Record<string, unknown> {
return value && typeof value === "object" && !Array.isArray(value)
? (value as Record<string, unknown>)
: {};
}
function parseWindowResetAt(window: Record<string, unknown>): string | null {
const resetAt = toNumber(window["reset_at"] ?? window["resetAt"], 0);
if (resetAt > 0) {
// Unix timestamp in seconds (< 1e12) or milliseconds (>= 1e12)
return new Date(resetAt < 1e12 ? resetAt * 1000 : resetAt).toISOString();
}
const resetAfterSeconds = toNumber(
window["reset_after_seconds"] ?? window["resetAfterSeconds"],
0
);
if (resetAfterSeconds > 0) {
return new Date(Date.now() + resetAfterSeconds * 1000).toISOString();
}
return null;
}
function parseWindowPercent(window: Record<string, unknown>): number {
const used = toNumber(window["used"] ?? window["used_amount"], 0);
const limit = toNumber(window["limit"] ?? window["limit_amount"], 0);
if (limit <= 0) return 0;
return Math.max(0, Math.min(1, used / limit));
}
// ─── Response Parser ──────────────────────────────────────────────────────────
function parseOpencodeQuotaResponse(data: unknown): OpencodeTripleWindowQuota | null {
const obj = toRecord(data);
const quotaObj = toRecord(obj["quota"] ?? obj["data"] ?? obj["usage"]);
// Look for windows under various possible keys
const w5h = toRecord(
quotaObj[OPENCODE_WINDOW_5H] ?? quotaObj["5h"] ?? quotaObj["hourly"] ?? quotaObj["short"]
);
const wWeekly = toRecord(
quotaObj[OPENCODE_WINDOW_WEEKLY] ?? quotaObj["weekly"] ?? quotaObj["week"] ?? quotaObj["wk"]
);
const wMonthly = toRecord(
quotaObj[OPENCODE_WINDOW_MONTHLY] ?? quotaObj["monthly"] ?? quotaObj["month"] ?? quotaObj["mo"]
);
const has5h = Object.keys(w5h).length > 0;
const hasWeekly = Object.keys(wWeekly).length > 0;
const hasMonthly = Object.keys(wMonthly).length > 0;
// Need at least one window to be meaningful
if (!has5h && !hasWeekly && !hasMonthly) return null;
const percent5h = has5h ? parseWindowPercent(w5h) : 0;
const percentWeekly = hasWeekly ? parseWindowPercent(wWeekly) : 0;
const percentMonthly = hasMonthly ? parseWindowPercent(wMonthly) : 0;
const resetAt5h = has5h ? parseWindowResetAt(w5h) : null;
const resetAtWeekly = hasWeekly ? parseWindowResetAt(wWeekly) : null;
const resetAtMonthly = hasMonthly ? parseWindowResetAt(wMonthly) : null;
const worstPercent = Math.max(percent5h, percentWeekly, percentMonthly);
const limitReached =
Boolean(obj["limit_reached"] ?? quotaObj["limit_reached"]) || worstPercent >= 1;
// Dominant reset: pick the window with the worst usage
let dominantResetAt: string | null = null;
if (worstPercent === percent5h) {
dominantResetAt = resetAt5h ?? resetAtWeekly ?? resetAtMonthly;
} else if (worstPercent === percentWeekly) {
dominantResetAt = resetAtWeekly ?? resetAt5h ?? resetAtMonthly;
} else {
dominantResetAt = resetAtMonthly ?? resetAtWeekly ?? resetAt5h;
}
const window5h = { percentUsed: percent5h, resetAt: resetAt5h };
const windowWeekly = { percentUsed: percentWeekly, resetAt: resetAtWeekly };
const windowMonthly = { percentUsed: percentMonthly, resetAt: resetAtMonthly };
const windows: Record<string, { percentUsed: number; resetAt: string | null }> = {};
if (has5h) windows[OPENCODE_WINDOW_5H] = window5h;
if (hasWeekly) windows[OPENCODE_WINDOW_WEEKLY] = windowWeekly;
if (hasMonthly) windows[OPENCODE_WINDOW_MONTHLY] = windowMonthly;
return {
used: worstPercent * 100,
total: 100,
percentUsed: worstPercent,
resetAt: dominantResetAt,
windows,
window5h,
windowWeekly,
windowMonthly,
limitReached,
};
}
// ─── Core Fetcher ─────────────────────────────────────────────────────────────
// ─── Dashboard Snapshot Bridge (#11234) ───────────────────────────────────────
//
// The live endpoint above has no public quota API today (404 — see module
// JSDoc), so without this bridge every preflight evaluated `null` and
// proceeded (fail-open) even when the dashboard already showed a drained
// window. The dashboard scrape (`getOpenCodeGoUsage` in
// opencodeOllamaUsage.ts) persists per-window snapshots through
// `src/domain/quotaCache.ts::setQuotaCache` under the window keys
// session / weekly / mcp_monthly; this bridge synthesizes the same
// OpencodeTripleWindowQuota shape from those cached snapshots so the quota
// cutoff sees them.
//
// Read-only: accessors only, never SQL, never a re-scrape on the hot path.
// Fail-open is preserved — no snapshots means `null`, exactly as before.
// Dashboard snapshot key → fetcher/preflight window key.
const DASHBOARD_SNAPSHOT_WINDOW_MAP: ReadonlyArray<readonly [string, string]> = [
["session", OPENCODE_WINDOW_5H],
["weekly", OPENCODE_WINDOW_WEEKLY],
["mcp_monthly", OPENCODE_WINDOW_MONTHLY],
];
function hasDashboardQuotaConfig(connection?: Record<string, unknown>): boolean {
// Snapshots can only exist when the operator configured the dashboard
// scrape for this connection (or globally via env). Gating on it keeps the
// snapshot read (and its cold-start DB hydration) off connections that
// could never have produced one.
const psd = connection?.providerSpecificData as Record<string, unknown> | undefined;
return resolveOpenCodeGoDashboardConfig(psd).state !== "none";
}
async function synthesizeQuotaFromDashboardSnapshots(
connectionId: string
): Promise<OpencodeTripleWindowQuota | null> {
let quotaCacheDomain: typeof import("../../src/domain/quotaCache.ts");
try {
// Dynamic import: a static edge would close an initialization cycle
// (opencodeQuotaFetcher → quotaCache → usage.ts → usage/opencode.ts →
// opencodeQuotaFetcher).
quotaCacheDomain = await import("../../src/domain/quotaCache.ts");
} catch {
return null;
}
// Hydrate the in-memory cache from persisted snapshots when cold (the
// accessor does this internally), then read the raw per-window rows.
quotaCacheDomain.getQuotaWindowStatus(connectionId, DASHBOARD_SNAPSHOT_WINDOW_MAP[0][0]);
const entry = quotaCacheDomain.getQuotaCache(connectionId);
const quotas = entry?.quotas;
if (!quotas || typeof quotas !== "object") return null;
const now = Date.now();
const windows: Record<string, { percentUsed: number; resetAt: string | null }> = {};
for (const [snapshotKey, windowKey] of DASHBOARD_SNAPSHOT_WINDOW_MAP) {
const raw = quotas[snapshotKey];
if (!raw || typeof raw.remainingPercentage !== "number") continue;
// #10095 mirror: a window whose fraction upstream never reported is
// "unknown", not 0% — it must not count as exhausted.
if (raw.fractionReported === false) continue;
const resetAt = typeof raw.resetAt === "string" && raw.resetAt ? raw.resetAt : null;
if (resetAt) {
const resetMs = Date.parse(resetAt);
// Mirror getQuotaWindowStatus (quotaCache.ts): an expired resetAt means
// the window has rolled into a fresh period — the cached percentage is
// stale and must not count as exhausted.
if (Number.isFinite(resetMs) && resetMs <= now) continue;
}
const remaining = Math.max(0, Math.min(100, raw.remainingPercentage));
windows[windowKey] = { percentUsed: 1 - remaining / 100, resetAt };
}
if (Object.keys(windows).length === 0) return null;
const window5h = windows[OPENCODE_WINDOW_5H] ?? { percentUsed: 0, resetAt: null };
const windowWeekly = windows[OPENCODE_WINDOW_WEEKLY] ?? { percentUsed: 0, resetAt: null };
const windowMonthly = windows[OPENCODE_WINDOW_MONTHLY] ?? { percentUsed: 0, resetAt: null };
const worstPercent = Math.max(
window5h.percentUsed,
windowWeekly.percentUsed,
windowMonthly.percentUsed
);
// Dominant reset: pick the window with the worst usage (same policy as the
// live-response parser above).
let dominantResetAt: string | null = null;
if (worstPercent === window5h.percentUsed) {
dominantResetAt = window5h.resetAt ?? windowWeekly.resetAt ?? windowMonthly.resetAt;
} else if (worstPercent === windowWeekly.percentUsed) {
dominantResetAt = windowWeekly.resetAt ?? window5h.resetAt ?? windowMonthly.resetAt;
} else {
dominantResetAt = windowMonthly.resetAt ?? windowWeekly.resetAt ?? window5h.resetAt;
}
return {
used: worstPercent * 100,
total: 100,
percentUsed: worstPercent,
resetAt: dominantResetAt,
windows,
window5h,
windowWeekly,
windowMonthly,
limitReached: worstPercent >= 1,
};
}
/**
* Fetch current quota for an OpenCode connection.
* Returns percentUsed = max(5h%, weekly%, monthly%) — worst-case across all windows.
*
* Defensive implementation: returns null on any non-200 / parse failure (fail-open).
* See module-level JSDoc for upstream API stability note.
*
* @param connectionId - Connection ID from the DB (used for cache keying)
* @param connection - Optional connection snapshot with apiKey
* @returns OpencodeTripleWindowQuota or null if fetch fails / no credentials
*/
export async function fetchOpencodeQuota(
connectionId: string,
connection?: Record<string, unknown>
): Promise<OpencodeTripleWindowQuota | null> {
// Snapshots can only exist when the dashboard scrape is configured for this
// connection (or globally via env); without it the bridge stays off and the
// fetcher never touches the snapshot store.
const dashboardConfigured = hasDashboardQuotaConfig(connection);
// Check cache first
const cached = quotaCache.get(connectionId);
if (cached) {
// 404 sentinel — use longer TTL to avoid hammering a non-existent endpoint
if (cached.noEndpoint && Date.now() - cached.fetchedAt < NO_ENDPOINT_TTL_MS) {
// The live endpoint is known-absent — serve dashboard snapshots if the
// operator configured the scrape (#11234).
return dashboardConfigured ? synthesizeQuotaFromDashboardSnapshots(connectionId) : null;
}
if (cached.quota !== null && Date.now() - cached.fetchedAt < CACHE_TTL_MS) {
return cached.quota;
}
}
const live = await fetchLiveOpencodeQuota(connectionId, connection);
if (live) return live;
// #11234 — the live endpoint has no public quota API (404) or failed:
// fall back to the operator-configured dashboard snapshots, read-only.
return dashboardConfigured ? synthesizeQuotaFromDashboardSnapshots(connectionId) : null;
}
async function fetchLiveOpencodeQuota(
connectionId: string,
connection?: Record<string, unknown>
): Promise<OpencodeTripleWindowQuota | null> {
// Extract API key from connection
const apiKey =
typeof connection?.apiKey === "string" && connection.apiKey.trim().length > 0
? connection.apiKey
: null;
if (!apiKey) {
return null;
}
try {
// #6911: space concurrent upstream quota fetches (mirrors codexQuotaFetcher.ts).
await throttleQuotaFetch();
const response = await fetch(OPENCODE_QUOTA_URL, {
method: "GET",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
Accept: "application/json",
},
signal: AbortSignal.timeout(8_000),
});
if (!response.ok) {
if (response.status === 404) {
// Upstream doesn't expose this endpoint. Warn once per URL per process so
// operators know the dashboard will be empty for opencode-go connections.
// Cache a 404 sentinel for NO_ENDPOINT_TTL_MS to avoid hammering.
// See opencode issues #10448, #16017, #18648, #31084.
if (!_warned404Urls.has(OPENCODE_QUOTA_URL)) {
_warned404Urls.add(OPENCODE_QUOTA_URL);
console.warn(
`[opencodeQuotaFetcher] ${OPENCODE_QUOTA_URL} returned 404 — opencode-go usage API is not yet public. ` +
`Set OMNIROUTE_OPENCODE_QUOTA_URL to a working endpoint, or follow ` +
`https://github.com/anomalyco/opencode/issues/16017 for upstream status.`
);
}
quotaCache.set(connectionId, {
quota: null,
fetchedAt: Date.now(),
noEndpoint: true,
});
return null;
}
if (response.status === 401 || response.status === 403) {
quotaCache.delete(connectionId);
}
return null;
}
let data: unknown;
try {
data = await response.json();
} catch {
// Malformed JSON — fail open
return null;
}
const quota = parseOpencodeQuotaResponse(data);
if (!quota) return null;
// Store in cache
quotaCache.set(connectionId, { quota, fetchedAt: Date.now() });
return quota;
} catch {
// Network error, timeout, etc. — fail open
return null;
}
}
// ─── Invalidation ─────────────────────────────────────────────────────────────
/**
* Force-invalidate the cache for a connection (e.g., after receiving quota headers).
*/
export function invalidateOpencodeQuotaCache(connectionId: string): void {
quotaCache.delete(connectionId);
}
// ─── Registration ─────────────────────────────────────────────────────────────
/**
* Register the OpenCode quota fetcher with the preflight and monitor systems
* for all three provider variants: opencode-go, opencode, opencode-zen.
*
* Call this once at server startup (in chat.ts, before registerGenericQuotaFetchers).
*/
export function registerOpencodeQuotaFetcher(): void {
for (const provider of ["opencode-go", "opencode", "opencode-zen"] as const) {
registerQuotaFetcher(provider, fetchOpencodeQuota);
registerMonitorFetcher(provider, fetchOpencodeQuota);
registerQuotaWindows(provider, [
OPENCODE_WINDOW_5H,
OPENCODE_WINDOW_WEEKLY,
OPENCODE_WINDOW_MONTHLY,
]);
}
}