The Quota / Providers dashboard (POST /api/usage/provider-limits ->
syncAllProviderLimits, GET /api/usage/[connectionId]) calls
refreshAndUpdateCredentials() per connection, in concurrent chunks. For
rotating-refresh providers (Codex/OpenAI share one Auth0 client_id) the
single-use refresh_token is rotated on every refresh; refreshing siblings
concurrently makes Auth0 revoke the whole token family (openai/codex#9648),
killing every account but the last with [403] <!DOCTYPE html>. On the affected
VM expires_at was persisted as ~0 so needsRefresh() was effectively always
true -> every codex account refreshed on every page visit -> guaranteed cascade.
Fix#1: refreshAndUpdateCredentials skips proactive refresh for rotating
providers (rotationGroupFor) and reuses the current access_token for the quota
fetch; genuine expiry is handled by the reactive, serialized 401 path.
Fix#2 (defense in depth): serializeRefresh inserts a settle gap between two
QUEUED sibling refreshes (default 2000ms, CODEX_REFRESH_SPACING_MS, '0' to opt
out) but releases a lone refresh immediately, adding no latency to the reactive
request path.
Tests: codex-quota-sync-no-proactive-refresh (skip + non-rotating guard),
refresh-serializer-spacing (default/opt-out + lone-vs-queued behavior).
- resolvePoolForSync now resolves groupName via getGroupName(pool.groupId),
falling back to pool.name when the group record is missing.
- Both quotaModelName calls in syncQuotaCombos use groupName instead of
pool.name, so combos are named qtSd/<groupSlug>/<provider>/<model>.
- Prune is now scoped to group+provider: syncing pool A (openrouter) never
deletes pool B (baidu) combos that share the same group.
- removeQuotaCombosForPool likewise scoped to group+provider.
- Updated quota-combos-sync, quota-combo-balancing, quota-combo-cli-providers
tests for the new group-slug naming.
- New tests/unit/quota-combo-groups.test.ts: G1–G4 cover two-pool same-group
naming, provider-scoped prune isolation, default-group fallback, and stale
prune within the same group+provider.
Replace the old quotaShared-<poolSlug>-<provider>/<model> format with
qtSd/<groupSlug>/<provider>/<model>. Adds quotaGroupSlug() as the canonical
helper; keeps quotaPoolSlug() as a delegating alias so existing callers
(quotaKey.ts, quotaCombos.ts) compile without changes. parseQuotaModelName
now returns { groupSlug, provider, model }. Updates quotaCombos.ts,
apiKeyPolicy.ts, and all tests that referenced the old field name or the
old hardcoded prefix literal.
Add src/lib/db/quotaGroups.ts with createGroup/getGroup/getGroupName/
listGroups/renameGroup/deleteGroup; deleteGroup guards against non-empty
groups (throws /pools/i) and protects the 'group-demo' seed. Re-export
all functions + QuotaGroup type from localDb.ts. 15 unit tests in
tests/unit/quota-groups-crud.test.ts all passing.
Introduces first-class quota Group entity: new quota_groups table,
quota_pools.group_id column, GroupDemo seed, backfill of existing
pools, and groupId in QuotaPool/PoolCreate/PoolUpdate with a
group-demo default. Migration runner gains an isSchemaAlreadyApplied
guard for version 087.
Replace the saturation-signal approximation (globalUsedPercent × effectiveLimit,
which is always 0 for requests/tokens/usd) with store.poolConsumedTotal() so
the pool actually saturates and enforceQuotaShare returns "block" when the pool
total hits the effective limit for countable dimensions.
"percent" dimensions continue to use the upstream saturation signal, which is
the authoritative measure for provider-reported utilisation windows.
Add poolConsumedTotal(poolId, dim) to the QuotaStore interface and implement
it in both SqliteQuotaStore and RedisQuotaStore so the enforce path can read
the real pool-wide consumption (sum across all keys) rather than the per-key
saturation signal, which is 0 for countable units and therefore never blocks.
- db/quotaConsumption.ts: add sumPoolDimension() — single SQL COALESCE(SUM)
for curr and prev buckets across all api_key_id rows for a dimensionKey.
- localDb.ts: re-export sumPoolDimension.
- quota/types.ts: add poolConsumedTotal to QuotaStore interface (with JSDoc).
- sqliteQuotaStore.ts: implement using sumPoolDimension + existing
slidingWindowEffective helper — one consistent sliding-window read.
- redisQuotaStore.ts: implement by fetching pool allocations from SQLite (F2),
then issuing a single MGET for all (key, curr-bucket) and (key, prev-bucket)
Redis keys, summing raw values, and applying the sliding-window formula once.
- tests/unit/quota-store-pool-total.test.ts: 4 tests (sum, pool isolation,
unit isolation, peek-no-regression) all passing.
getQuotaStore() is async; enforce.ts used it without await, so store was a Promise
and store.peek/store.consume threw 'not a function' → enforceQuotaShare failed open
on every request and recordConsumption never wrote. Production quota was a silent
no-op (unit tests passed because they inject a sync mock store). Await it + guard.
deletePool now runs a SQLite json_each UPDATE inside the existing
transaction to remove the pool's id from every api_key's allowed_quotas
JSON array, preventing stale pool references after deletion.
Tests: tests/unit/quota-pool-delete-prune.test.ts (7 scenarios).
Runtime (enforce.ts): compute effectiveWeight = 100/N for each key when the
pool's total weight is 0, so pools with all-zero weights (newly created via
UI) are usable immediately without a re-save. Original non-zero weights are
unchanged.
UI (PoolWizard, EditAllocationsModal): addKey now recomputes all weights to
an equal split after adding a key, so saved pools store equal weights and
never persist all-zero allocations.
Tests: tests/unit/quota-equal-split.test.ts (7 scenarios).
getProviderModelIds read PROVIDER_MODELS, which is empty for CLI/OAuth providers
(codex, kimi, claude, …) — so pools built from those providers produced ZERO
quotaShared-* combos and their quota keys saw no models. Read the provider REGISTRY
instead (the same source /v1/models uses), which covers all providers.
Custom models (added via the UI) on opencode-go / openai-compatible nodes always
routed as OpenAI-compatible because there was no per-model targetFormat: addCustomModel
didn't accept it, the API schema stripped it, and getModelTargetFormat (static-registry
only) never saw it — so a custom model needing the Anthropic Messages shape fell back to
the provider default 'openai'.
Thread an optional targetFormat through addCustomModel / replaceCustomModels /
updateCustomModel + the providerModelMutationSchema + the POST/PUT route, surface it from
getModelInfo (one combined custom-model metadata lookup alongside apiFormat), and use it
in chatCore's targetFormat resolution before the provider default.
Closes#2905
A pool with N same-type connections now has an effective budget of
perAccountLimit × N per dimension. Only the limit fed to fair-share
scales — consumption (pool-keyed shared bucket) is unchanged.
Replace the (connId × modelId) upsert loop in syncQuotaCombos with a
model-grouped build: one combo per model with ALL connections as steps
and strategy "fill-first", fixing the collision where a second same-provider
connection overwrote the first (last upsert won, only one account ever used).
- Add assertSingleProvider() helper in quotaPools.ts: queries provider_connections
with DISTINCT provider WHERE id IN (...), throws if >1 provider detected.
- Call guard early in createPool (when connectionIds.length > 1) and updatePool
(when input.connectionIds.length > 1), before any DB writes.
- Add lockedProvider useMemo in PoolWizard.tsx: derived from first selected
connection; availableConnections filtered to same provider once locked.
- Show wizardSingleProviderNote i18n hint in step 1 when lockedProvider is set.
- Add wizardSingleProviderNote to en.json + pt-BR.json (parity).
- New test: tests/unit/quota-pool-single-provider.test.ts (4 cases).
- Update tests/unit/quota-multiprovider.test.ts: all D2 tests now use two
same-provider connections (both PROVIDER_A/openrouter) — the tests were
exercising connection-plumbing (scope, enforce membership, combo fan-out),
not mixed-provider behavior per se; updated to remain valid under Task 3 rule.
Fireworks Fire Pass (fpk_*) keys return 403 "Fire Pass API keys are not
authorized for this route." on /models while still serving chat. Two paths
wrongly penalized the key:
- validateOpenAILikeProvider returned "Invalid API key" for any 403 on the
models endpoint without trying the chat probe.
- checkFallbackError classified the 403 as AUTH_ERROR (retryable cooldown), and
even a generic fall-through hit the transient-cooldown default — marking the
connection unavailable.
Fix: validateOpenAILikeProvider now inspects the 403 body and falls through to
the chat probe for "not authorized for this route" responses (401 and generic
403 still fail fast). checkFallbackError short-circuits such route-restriction
403s to { shouldFallback: false, cooldownMs: 0 } so the connection is not cooled
down. Per CLAUDE.md, a generic api-key 403 should be recoverable unless terminal.
Closes#2929
D2 moved poolSlug collection outside the connection loop, so a pool whose only
connection was deleted still leaked its slug into resolveQuotaKeyScope — that
slug has no quotaShared-* models behind it. Gate the slug on >=1 valid member
connection (restores Phase-A2 behavior for multi-connection pools).
The no-auth OpenCode provider has id "opencode" and alias "oc". The combo
builder built qualifiedModel from the provider id ("opencode/big-pickle"), but
parseModel("opencode/...") resolves to the opencode-zen api-key tier via a
manual ALIAS_TO_PROVIDER_ID override — not the no-auth "opencode" provider.
"oc/<model>" resolves correctly.
Rewrite qualifiedModel to the provider alias for no-auth providers (only when
the alias differs from the id), keeping providerId for getModelIsHidden. Aligned
the route test that previously asserted the buggy "opencode/" prefix.
Closes#2901
Adversarial review of the peer-IP fix surfaced: (1) CRITICAL — cliTokenAuth.ts
still derived loopback from new URL(request.url).hostname (the same spoofable
Host class), letting a remote caller with a stolen CLI token reach management
APIs via Host: 127.0.0.1; now it trusts the middleware-stamped locality verdict
(AUTHZ_HEADER_PEER_LOCALITY, a client-stripped trusted header). (2) HIGH —
isLoopbackHost mangled bare IPv6 (::1, ::ffff:127.0.0.1) via split(":")[0],
a fail-closed DoS on IPv6 deploys. (3) HIGH — the Docker entrypoint ran bare
server.js (no peer stamp); run-standalone.mjs now prefers server-ws.mjs.
big-pickle's OpenCode/Zen upstream runs DeepSeek thinking mode, but the model
id reveals no DeepSeek signal, so requiresReasoningReplay (called with
allowLegacyFallback:false) never triggered. Follow-up/tool-use turns failed
with [400] 'The reasoning_content in the thinking mode must be passed back to
the API'. Note: requiresReasoningReplay does not consume supportsReasoning, so
the registry flag alone would not have fixed it.
Add RegistryModel.interleavedField (mirrors models.dev interleaved_field),
declare interleavedField:'reasoning_content' (+ supportsReasoning:true) on
big-pickle in both opencode and opencode-zen, and surface the registry value
in getResolvedModelCapabilities so requiresReasoningReplay returns true.
Closes#2900
077_api_key_stream_default_mode.sql and 077_quota_pools.sql both claimed
prefix 077, so getMigrationFiles() threw a version-collision error and
getDbInstance() failed at every startup (app would not boot; all DB-touching
unit tests were red on release/v3.8.8).
Renumber the dependency-free, idempotent quota_pools migration 077 -> 085
(no other migration references quota_pools/quota_allocations), keep the
non-idempotent api_key_stream_default_mode ALTER at 077, add a retroactive
isSchemaAlreadyApplied guard (case 085) for DBs that already applied it under
077, and add a regression test enforcing unique migration prefixes.
- quotaKey.ts (resolveQuotaKeyScope): iterate pool.connectionIds (fall back to
[connectionId] for un-backfilled rows); each connection contributes its own
connId + provider to the scope. poolSlugs logic unchanged (one slug per pool).
- quotaCombos.ts (syncQuotaCombos): replace single-connection resolvePoolProvider
with resolvePoolForSync that returns all connectionIds; iterate each connId to
build the desiredNames union across all providers; upsert combos pinned to the
CORRECT per-connection connId; prune against the full union so only truly stale
combos (no longer produced by any current connection) are deleted.
- enforce.ts (enforceQuotaShare + recordConsumption): both pool-matching loops
changed from equality (p.connectionId === input.connectionId) to membership
(p.connectionIds.includes) with fallback for un-backfilled rows. Fail-open
(B16) and pool-level dimension key semantics are preserved unchanged.
- quotaPools.ts: no logic change needed — connectionIds already flows through
getPool (D1); syncQuotaCombosGuarded passes poolId and syncQuotaCombos
resolves the full QuotaPool internally.
- tests/unit/quota-multiprovider.test.ts: 6 new tests covering D2 (scope,
enforce primary/secondary membership, combos 2-provider create + prune).
All 22 tests pass (14 new + 8 existing enforce + pool-connections suites).
Adds migration 086 to create the `quota_pool_connections` join table with a backfill
that seeds every existing pool's single connection_id as its first member. Updates
QuotaPool type with `connectionIds: string[]`, wires createPool/updatePool/deletePool
to maintain the join table transactionally, and keeps `connection_id` as the primary
back-compat column synced to `connectionIds[0]`.
Introduces src/lib/quota/quotaKey.ts with resolveQuotaKeyScope(), a
pure async helper that maps an API key's allowedQuotas pool-ID list to
the concrete connectionIds and provider slugs it is permitted to use.
Covers empty/null/undefined input, missing pools, orphaned connectionIds,
and multi-pool deduplication. No behaviour change to existing code paths.
