From 24a9739604aea24d8acee4a506f3ac2afadffecd Mon Sep 17 00:00:00 2001 From: diegosouzapw Date: Sun, 22 Mar 2026 18:12:50 -0300 Subject: [PATCH] docs: add sub2api gap analysis + 15 implementation tasks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add competitive analysis of sub2api (v0.1.104, 87 contributors) comparing features, open PRs, and model pricing against OmniRoute. Files: - docs/new-features-sub21/gap-analysis.md — full analysis (commits + 38 open PRs) - docs/new-features-sub21/implementation-plan.md — phased plan for all 15 gaps - docs/new-features-sub21/tasks/T01-T15 — detailed task files with: - Problem description + sub2api PR references - Step-by-step implementation with code snippets - Affected files list - Acceptance criteria Priority breakdown: P1 (4): requested_model logs, empty tool_result blocks, x-codex-* headers, X-Session-Id P2 (6): rate-limit persistence, account_deactivated, XFF validation, session limits, Codex/Spark scopes, credits_exhausted P3 (5): max reasoning effort, model pricing, stale quota display, proxy fast-fail, array content Source: https://github.com/Wei-Shaw/sub2api --- .gitignore | 2 + docs/new-features-sub21/gap-analysis.md | 304 ++++++++++++++++++ .../new-features-sub21/implementation-plan.md | 247 ++++++++++++++ .../tasks/T01-requested-model-log.md | 109 +++++++ .../T02-strip-empty-tool-result-blocks.md | 117 +++++++ .../tasks/T03-codex-quota-reset-headers.md | 136 ++++++++ .../tasks/T04-x-session-id-header.md | 107 ++++++ .../tasks/T05-persist-ratelimit-state.md | 127 ++++++++ .../tasks/T06-account-deactivated-status.md | 103 ++++++ .../tasks/T07-xff-validation.md | 103 ++++++ .../tasks/T08-per-key-session-limit.md | 125 +++++++ .../tasks/T09-codex-spark-rate-limit-scope.md | 116 +++++++ .../tasks/T10-credits-exhausted-state.md | 103 ++++++ .../tasks/T11-max-reasoning-effort.md | 86 +++++ .../tasks/T12-model-pricing-updates.md | 95 ++++++ .../tasks/T13-stale-quota-display.md | 90 ++++++ .../tasks/T14-proxy-fast-fail.md | 130 ++++++++ .../tasks/T15-array-content-messages.md | 128 ++++++++ 18 files changed, 2228 insertions(+) create mode 100644 docs/new-features-sub21/gap-analysis.md create mode 100644 docs/new-features-sub21/implementation-plan.md create mode 100644 docs/new-features-sub21/tasks/T01-requested-model-log.md create mode 100644 docs/new-features-sub21/tasks/T02-strip-empty-tool-result-blocks.md create mode 100644 docs/new-features-sub21/tasks/T03-codex-quota-reset-headers.md create mode 100644 docs/new-features-sub21/tasks/T04-x-session-id-header.md create mode 100644 docs/new-features-sub21/tasks/T05-persist-ratelimit-state.md create mode 100644 docs/new-features-sub21/tasks/T06-account-deactivated-status.md create mode 100644 docs/new-features-sub21/tasks/T07-xff-validation.md create mode 100644 docs/new-features-sub21/tasks/T08-per-key-session-limit.md create mode 100644 docs/new-features-sub21/tasks/T09-codex-spark-rate-limit-scope.md create mode 100644 docs/new-features-sub21/tasks/T10-credits-exhausted-state.md create mode 100644 docs/new-features-sub21/tasks/T11-max-reasoning-effort.md create mode 100644 docs/new-features-sub21/tasks/T12-model-pricing-updates.md create mode 100644 docs/new-features-sub21/tasks/T13-stale-quota-display.md create mode 100644 docs/new-features-sub21/tasks/T14-proxy-fast-fail.md create mode 100644 docs/new-features-sub21/tasks/T15-array-content-messages.md diff --git a/.gitignore b/.gitignore index 84df447763..9390e082ee 100644 --- a/.gitignore +++ b/.gitignore @@ -88,6 +88,8 @@ docs/* !docs/AUTO-COMBO.md !docs/MCP-SERVER.md !docs/CLI-TOOLS.md +!docs/new-features-sub21/ +!docs/new-features-sub21/** # open-sse tests open-sse/test/* diff --git a/docs/new-features-sub21/gap-analysis.md b/docs/new-features-sub21/gap-analysis.md new file mode 100644 index 0000000000..21666b2c6f --- /dev/null +++ b/docs/new-features-sub21/gap-analysis.md @@ -0,0 +1,304 @@ +# sub2api vs OmniRoute — Gap Analysis v2 (with Open PRs) + +> **Source:** [github.com/Wei-Shaw/sub2api](https://github.com/Wei-Shaw/sub2api) · v0.1.104 · 87 contributors · 92 releases +> **Analyzed:** merged commits (Mar 20-21, 2026) + **38 open PRs** (pages 1 & 2) +> **Date:** 2026-03-22 + +--- + +## ✅ Already Covered by OmniRoute (no gap) + +| Feature | OmniRoute | +| -------------------------------- | ------------------------------------------------------- | +| Sticky session routing | `sessionManager.ts` + `stickyRoundRobinLimit` | +| Per-model concurrency limiter | `rateLimitSemaphore.ts` (FIFO queue) | +| `reasoning_effort` bidirectional | `thinkingBudget.ts` (low/medium/high ↔ `budget_tokens`) | +| Thinking block support | `claude-to-openai.ts` state machine | +| Circuit breaker | Per-model, exponential backoff | +| OAuth token auto-refresh | 18 providers | +| Request dedup | 5s content-hash window | +| Rate limit detection | Per-provider RPM, min gap, max concurrent | + +--- + +## 🚨 Priority 1 — High Impact, Actionable Now + +### 1. `requested_model` vs `routed_model` in Usage Logs + +**Source:** 8 merged commits (Mar 19-20) — `feat(ent): add requested model to usage log schema` through `fix(dto): fallback to legacy model in usage mapping` + +When a user asks for `openai/gpt-5.2-codex` but the combo falls back to `gpt-5.2-mini`, today's usage log stores only the routed model. sub2api now stores both. + +**Gap:** OmniRoute `call_logs` has a single `model` field. Users can't audit requested vs actual. + +```sql +-- Migration 009 +ALTER TABLE call_logs ADD COLUMN requested_model TEXT DEFAULT NULL; +``` + +- Capture `body.model` at entry as `requestedModel`; store resolved model in existing field +- Show both in Logs dashboard and Analytics filters + +**Priority:** 🔴 Billing transparency / user trust + +--- + +### 2. Empty Text Blocks in Nested `tool_result` → 400 Error + +**Source:** Open PR [#1212](https://github.com/Wei-Shaw/sub2api/pull/1212) — `fix(gateway): strip empty text blocks nested in tool_result.content` + +Anthropic returns `400: "messages: text content blocks must be non-empty"` when a `tool_result.content` array contains `{"type":"text","text":""}`. The fix in PR #1212 is: + +- Add `stripEmptyTextBlocksFromSlice()` that **recursively** filters nested content arrays +- Apply pre-filter on ALL upstream Anthropic paths (gateway, passthrough, count_tokens) + +**OmniRoute gap:** Our `openai-to-claude.ts` translator likely has the same bug — Claude Code in extended tool chains triggers this regularly. + +```typescript +// In openai-to-claude.ts or claude translator +function stripEmptyTextBlocks(content: ContentBlock[]): ContentBlock[] { + return content + .filter((b) => !(b.type === "text" && b.text === "")) + .map((b) => + b.type === "tool_result" && Array.isArray(b.content) + ? { ...b, content: stripEmptyTextBlocks(b.content) } // recurse + : b + ); +} +``` + +**Priority:** 🔴 Active 400 errors for users with tool-heavy agents + +--- + +### 3. `x-codex-*` Header Parsing for Precise Quota Reset Times + +**Source:** Open PR [#357](https://github.com/Wei-Shaw/sub2api/pull/357) — `feat(oauth): persist usage snapshots and window cooldown` + +Codex responses include headers: `x-codex-5h-usage`, `x-codex-5h-limit`, `x-codex-5h-reset-at`, `x-codex-7d-usage`, `x-codex-7d-limit`, `x-codex-7d-reset-at`. + +Today sub2api uses a generic 5-minute fallback when hitting a 429. The PR makes it: + +- Parse these headers after every Codex request +- Persist exact `reset_at` per window (5h / 7d) +- Block the account until that exact timestamp (not a flat 5min guess) +- Also run an optional periodic `OAuthProbeService` for idle accounts + +**OmniRoute gap:** Our `codex.ts` executor parses some quota headers but likely falls back to circuit-breaker cooldown, not precise reset time. + +```typescript +// In codex.ts after each response +const resetAt5h = response.headers.get("x-codex-5h-reset-at"); +const resetAt7d = response.headers.get("x-codex-7d-reset-at"); +const usagePercent = + parseFloat(response.headers.get("x-codex-5h-usage") ?? "0") / + parseFloat(response.headers.get("x-codex-5h-limit") ?? "1"); + +if (usagePercent >= 1 && resetAt5h) { + // block this connection until resetAt (not just 5min) + connection.rateLimitResetAt = new Date(resetAt5h).getTime(); +} +``` + +**Priority:** 🔴 Codex is a top-used provider — bad reset timing hurts UX significantly + +--- + +### 4. `X-Session-Id` HTTP Header for External Sticky Session + +**Source:** sub2api README Nginx note: `underscores_in_headers on` required for `session_id` header + +Clients send `X-Session-Id: abc123` to pin the conversation to the same account. Critical for: + +- Nginx reverse proxy users (Nginx drops underscore headers by default) +- Claude Code / Codex sessions that break on mid-conversation account switches + +**OmniRoute gap:** Our `sessionManager.ts` generates sessions internally (content hash). No way for the client to set a session ID. + +```typescript +// In chatCore.js request handler +const clientSessionId = req.headers["x-session-id"] ?? req.headers["session-id"]; +const sessionId = clientSessionId ? `client:${clientSessionId}` : generateSessionHash(body); +``` + +**Priority:** 🔴 Nginx users lose sticky routing + +--- + +## 🟠 Priority 2 — Medium Impact + +### 5. Rate-Limited Accounts Not Rescheduled After Token Refresh + +**Source:** Open PR [#1218](https://github.com/Wei-Shaw/sub2api/pull/1218) — `fix(openai): prevent rescheduling rate-limited accounts` + +When token refresh runs, it can accidentally clear the `rate_limited` state cached in memory, causing a rate-limited account to be selected again immediately. + +**OmniRoute gap:** Our circuit breaker state is in-memory. A token refresh in `codex.ts` or `auth.ts` could reset the flag. Need to persist account-level rate-limit state in DB (not just in-memory circuit breaker). + +**Priority:** 🟠 Codex users hitting rate limits loop repeatedly + +--- + +### 6. Per-User Session Limit (`max_sessions`) + +**Source:** Open PR [#634](https://github.com/Wei-Shaw/sub2api/pull/634) — `fix: stabilize session hash + add user-level session limit` + +PR adds a `max_sessions` field per user (default 0 = unlimited). When a user opens more than N concurrent sessions, the gateway returns: + +> HTTP 429: "You have reached the maximum number of active sessions. Please close unused sessions or wait." + +Also fixes session hash stability: previously using message content as seed caused same user to appear as multiple sessions across turns. Now uses `ClientIP + APIKeyID`. + +**OmniRoute gap:** No per-key session limit. A single API key can open unlimited sticky sessions. Session hash may be similarly unstable across turns. + +**Priority:** 🟠 Relevant for shared deployments / API key abuse prevention + +--- + +### 7. Codex vs Codex Spark — Separate Rate Limit Scopes + +**Source:** Open PR [#1129](https://github.com/Wei-Shaw/sub2api/pull/1129) — `feat(openai): split codex spark rate limiting from codex` + +Codex has two model tiers: `codex` (standard) and `spark` (more powerful/limited). Today, when `codex` hits its quota, the whole account gets marked rate-limited — even though `spark` quota might still be available. + +**OmniRoute gap:** Our circuit breaker is per-model-alias, so this may partly be handled. But check that hitting `codex` quota doesn't block `codex:spark` or `gpt-5.2-codex` models. + +**Priority:** 🟠 Codex quota management + +--- + +### 8. 401 `account_deactivated` — Distinct Account State + +**Source:** Open PR [#1037](https://github.com/Wei-Shaw/sub2api/pull/1037) — `fix(ratelimit): handle upstream 401 account deactivation` + +When upstream returns 401 with body containing `account_deactivated`, the account should be immediately marked as permanently deactivated — not put in rate-limit cooldown to retry in 5 minutes. + +**OmniRoute gap:** Our auth error handling in executors treats most 401s as token-expired (triggers re-auth). A `account_deactivated` 401 should mark the connection as `expired` permanently. + +```typescript +// In executor error handler +if (status === 401 && body.includes("account_deactivated")) { + await markConnectionStatus(connectionId, "expired"); + throw new PermanentAccountError("Account deactivated by upstream"); +} +``` + +**Priority:** 🟠 Avoids wasted retries on dead accounts + +--- + +### 9. OpenAI-Compatible Chat Completions Mode (Alibaba Coding Plan etc.) + +**Source:** Open PR [#1216](https://github.com/Wei-Shaw/sub2api/pull/1216) — `feat: add support for OpenAI-compatible providers using Chat Completions API` + +Some OpenAI-compatible providers (Alibaba Coding Plan: `coding-intl.dashscope.aliyuncs.com`) support `/v1/chat/completions` but NOT the `/v1/responses` (Responses API). Sub2API adds a per-account flag `openai_chat_completions_mode: true` to skip Responses API and use Chat Completions directly. + +**OmniRoute gap:** We already have separate executors but any OpenAI-Responses API executor that falls back to `/chat/completions` should be configurable per-connection. Check `responsesHandler.js` for providers that don't support Responses. + +**Priority:** 🟠 Compatibility with more OpenAI-compat providers + +--- + +### 10. `X-Forwarded-For` — Skip Invalid Entries for Client IP + +**Source:** Open PR [#1135](https://github.com/Wei-Shaw/sub2api/pull/1135) — `fix: skip invalid X-Forwarded-For entries in client IP detection` + +When `X-Forwarded-For` contains `unknown, 1.2.3.4`, the string `"unknown"` is not a valid IP and should be skipped, falling back to the next entry or to `ClientIP()`. + +**OmniRoute gap:** Our IP extraction for rate limiting — check if we're validating each entry before trusting the first one. + +**Priority:** 🟠 IP-based rate limiting correctness behind proxies + +--- + +## 🟡 Priority 3 — Low-Medium Impact + +### 11. `credits_exhausted` as Distinct Account State + +**Source:** Merged commit `fix: credits-exhausted handling` (PR #1169) + +**Gap:** Distinguish "credits exhausted" from generic error. Set account to `credits_exhausted` state immediately. **Already covered above** in v1 analysis — repeating for completeness. + +--- + +### 12. Model Pricing Updates — Missing from OmniRoute Pricing Table + +**Source:** Open PR [#970](https://github.com/Wei-Shaw/sub2api/pull/970), closed commit `fix: format gpt-5.4 mini fallback pricing` + +Models and pricing sub2api tracks that OmniRoute may be missing: + +| Model | Input $/MTok | Output $/MTok | +| ----------------------------------------- | ------------ | ------------- | +| **MiniMax M2.5** | $0.27 | $0.95 | +| **MiniMax M2.7** (PR #1120 — new default) | TBD | TBD | +| **GLM-4.7** (Zhipu) | $0.38 | $1.98 | +| **GLM-5** (Zhipu) | ~$0.38 | ~$1.98 | +| **Kimi** (Moonshot) | (see PR) | (see PR) | +| **gpt-5.4 mini** | (see commit) | (see commit) | + +**Action:** Add/verify these in OmniRoute's pricing settings and model registry. + +--- + +### 13. `max` Reasoning Effort → `budget_tokens: 100000` (xhigh) + +**Source:** Merged commit `fix(apicompat): 修正 Anthropic→OpenAI 推理级别映射` + PR `fix: openai-messages-effort-max-to-xhigh` + +**Gap:** In `thinkingBudget.ts`, `reasoning_effort: "max"` probably maps to `high` (32K tokens). The correct value is 100K: + +```typescript +const EFFORT_BUDGETS = { low: 1024, medium: 10240, high: 32000, max: 100000 }; +``` + +--- + +### 14. Usage Snapshots Stale After Reset + +**Source:** Merged commit `fix: quota display shows stale cumulative usage after daily/weekly reset` (PR #1171) + +Dashboard can show stale cumulative usage after the upstream quota window resets. Fix: compare `reset_at` with `now()` before displaying counters; auto-zero if window has passed. + +--- + +### 15. Proxy Fast-Fail on Dead Proxy + +**Source:** Merged commit `fix: proxy-fast-fail` (PR #1167) + +When configured proxy is down, requests hang for full timeout (30s). Quick TCP health check before request can short-circuit failures in 2s. + +--- + +## 💡 Architectural Items to Watch (Future) + +| PR | Feature | Relevance | +| ------------------------------------------------------ | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------- | +| [#1010](https://github.com/Wei-Shaw/sub2api/pull/1010) | **OIDC login** — generic OIDC OAuth login + IdP real email | Relevant if OmniRoute adds multi-user login (SSO enterprise integrations) | +| [#977](https://github.com/Wei-Shaw/sub2api/pull/977) | **Webhook custom headers** — custom request headers on webhook notifications | Relevant if OmniRoute adds webhook alerts | +| [#976](https://github.com/Wei-Shaw/sub2api/pull/976) | **Scheduled webhook failure test** — scheduled webhook health self-test | Same | +| [#734](https://github.com/Wei-Shaw/sub2api/pull/734) | **User invite/referral system** — viral growth for SaaS | Future, if OmniRoute goes SaaS | +| [#618](https://github.com/Wei-Shaw/sub2api/pull/618) | **LDAP auth** — enterprise directory login | Future enterprise | +| [#487](https://github.com/Wei-Shaw/sub2api/pull/487) | **Balance management page** — admin credit top-up UI | Future SaaS | +| [#1012](https://github.com/Wei-Shaw/sub2api/pull/1012) | **Subscription benefit plan** — tiered plan benefits management | Future SaaS | +| [#945](https://github.com/Wei-Shaw/sub2api/pull/945) | **Gemini native embeddings** — direct Gemini embedding endpoint | Already in OmniRoute | + +--- + +## 📋 Final Prioritized Action List + +| # | Action | From | Effort | Priority | +| --- | ------------------------------------------------------------------------------------- | --------------- | ------ | -------- | +| 1 | `requested_model` column in call_logs (migration 009) | Merged commits | Medium | 🔴 P1 | +| 2 | Strip empty text blocks from nested `tool_result.content` before forwarding to Claude | PR #1212 | Small | 🔴 P1 | +| 3 | Parse `x-codex-*` headers for precise 5h/7d reset times (not 5min fallback) | PR #357 | Medium | 🔴 P1 | +| 4 | `X-Session-Id` header → external sticky routing | README note | Small | 🔴 P1 | +| 5 | Persist rate-limit state in DB so token refresh doesn't clear it | PR #1218 | Medium | 🟠 P2 | +| 6 | `account_deactivated` → permanent `expired` status, not cooldown | PR #1037 | Small | 🟠 P2 | +| 7 | `X-Forwarded-For` validation — skip non-IP entries | PR #1135 | Tiny | 🟠 P2 | +| 8 | Per-API-key session limit (`max_sessions`) | PR #634 | Medium | 🟠 P2 | +| 9 | Codex vs Spark separate rate limit scopes | PR #1129 | Small | 🟠 P2 | +| 10 | `credits_exhausted` distinct account status | PR #1169 | Small | 🟠 P2 | +| 11 | `max` reasoning_effort → `budget_tokens: 100000` | Merged commit | Tiny | 🟡 P3 | +| 12 | Model pricing: MiniMax M2.5/M2.7, GLM-4.7/5, Kimi, gpt-5.4 mini | PR #970, commit | Tiny | 🟡 P3 | +| 13 | Quota display stale-after-reset fix | PR #1171 | Small | 🟡 P3 | +| 14 | Proxy fast-fail (2s TCP check before hanging 30s) | PR #1167 | Small | 🟡 P3 | +| 15 | Array content in system/tool messages (`content: [{type:"text"}]`) | PR #1197 | Tiny | 🟡 P3 | diff --git a/docs/new-features-sub21/implementation-plan.md b/docs/new-features-sub21/implementation-plan.md new file mode 100644 index 0000000000..1a50eda3fc --- /dev/null +++ b/docs/new-features-sub21/implementation-plan.md @@ -0,0 +1,247 @@ +# OmniRoute — Implementation Plan: sub2api Gap Resolution + +> Based on gap analysis of [github.com/Wei-Shaw/sub2api](https://github.com/Wei-Shaw/sub2api) v0.1.104 +> Analysis: merged commits + 38 open PRs (Mar 2026) +> See: [gap-analysis.md](./gap-analysis.md) + +--- + +## Overview + +15 actionable improvements identified across routing, billing, translator, and provider layers. +Organized in 3 priority tiers with estimated effort and affected files. + +| Priority | Count | Description | +| ---------------- | ----- | -------------------------------------------- | +| 🔴 P1 — Critical | 4 | Active bugs or significant billing/UX impact | +| 🟠 P2 — Medium | 6 | Routing correctness, account state handling | +| 🟡 P3 — Low | 5 | Pricing, display fixes, minor compat | + +--- + +## Tier 1 — Critical (P1) + +### T01 · `requested_model` in Usage Logs + +**Task file:** [tasks/T01-requested-model-log.md](./tasks/T01-requested-model-log.md) + +Add a `requested_model` column to the `call_logs` table so we track the model the client asked for separately from the model that was actually routed. Needed for billing transparency and debugging combo fallbacks. + +**Affected files:** + +- `src/lib/db/migrations/009_requested_model.sql` ← new migration +- `src/lib/db/settings.ts` or `usageDb.ts` ← write requested_model +- `open-sse/chatCore.js` ← capture `body.model` at entry +- `src/app/(dashboard)/dashboard/logs/` ← show new column + +--- + +### T02 · Strip Empty Text Blocks from Nested `tool_result` + +**Task file:** [tasks/T02-strip-empty-tool-result-blocks.md](./tasks/T02-strip-empty-tool-result-blocks.md) + +Anthropic rejects requests where `tool_result.content` arrays contain `{"type":"text","text":""}`. The fix requires a recursive strip function applied before all Anthropic upstream calls. + +**Affected files:** + +- `open-sse/translator/openai-to-claude.ts` ← add `stripEmptyTextBlocks()` +- `open-sse/chatCore.js` ← apply before forwarding to Anthropic +- `open-sse/translator/` ← also apply in responses/passthrough paths + +--- + +### T03 · Parse `x-codex-*` Headers for Precise Quota Reset Times + +**Task file:** [tasks/T03-codex-quota-reset-headers.md](./tasks/T03-codex-quota-reset-headers.md) + +Codex responses include `x-codex-5h-reset-at`, `x-codex-5h-usage`, `x-codex-7d-reset-at`, etc. Parse these after every response to store the exact reset timestamp per window instead of using a 5-minute generic fallback. + +**Affected files:** + +- `open-sse/executors/codex.ts` ← parse response headers +- `src/lib/db/providers.ts` ← persist `rateLimitResetAt` per connection +- `src/app/(dashboard)/dashboard/providers/` ← display countdown + +--- + +### T04 · `X-Session-Id` Header for External Sticky Routing + +**Task file:** [tasks/T04-x-session-id-header.md](./tasks/T04-x-session-id-header.md) + +Accept `X-Session-Id` from the client HTTP request and use it as the session key for sticky account routing instead of the internally generated content hash. Required for Nginx users (Nginx drops underscore headers by default — use `underscores_in_headers on`). + +**Affected files:** + +- `open-sse/chatCore.js` ← read `x-session-id` header +- `open-sse/services/sessionManager.ts` ← accept external session ID +- `docs/` ← document Nginx `underscores_in_headers on` requirement + +--- + +## Tier 2 — Medium (P2) + +### T05 · Persist Rate-Limit State in DB to Survive Token Refresh + +**Task file:** [tasks/T05-persist-ratelimit-state.md](./tasks/T05-persist-ratelimit-state.md) + +Token refresh flows in `codex.ts` and `auth.ts` can accidentally clear in-memory rate-limit state, causing a rate-limited account to be reselected immediately after refresh. Fix by persisting the rate-limit state in the DB and re-checking before account selection. + +**Affected files:** + +- `src/lib/db/providers.ts` ← add `rate_limit_until` column / read/write +- `open-sse/executors/codex.ts` ← check DB state before returning cached account +- `src/sse/services/auth.ts` ← don't clear rate-limit on credentials-only update + +--- + +### T06 · `account_deactivated` → Permanent `expired` Status + +**Task file:** [tasks/T06-account-deactivated-status.md](./tasks/T06-account-deactivated-status.md) + +When upstream returns `401` with `account_deactivated` in the body, mark the connection as permanently `expired` instead of entering cooldown. Avoids repeated retries on dead accounts. + +**Affected files:** + +- `open-sse/chatCore.js` or `open-sse/executors/*.ts` ← detect signal in 401 body +- `src/lib/db/providers.ts` ← `updateConnectionStatus(id, "expired")` +- `src/shared/constants/providers.ts` ← document deactivation error strings + +--- + +### T07 · X-Forwarded-For Validation — Skip Non-IP Entries + +**Task file:** [tasks/T07-xff-validation.md](./tasks/T07-xff-validation.md) + +When `X-Forwarded-For: unknown, 1.2.3.4` is received, the string `"unknown"` is not a valid IP. Skip invalid entries and fall back to the next valid IP or to remote address. + +**Affected files:** + +- `open-sse/chatCore.js` ← client IP extraction +- `src/app/api/` middleware ← IP-based rate limiting extraction + +--- + +### T08 · Per-API-Key Session Limit (`max_sessions`) + +**Task file:** [tasks/T08-per-key-session-limit.md](./tasks/T08-per-key-session-limit.md) + +Add a `max_sessions` field to API keys (default 0 = unlimited). When a key exceeds the configured number of concurrent sessions, return HTTP 429 with a friendly message. Configurable in API Manager dashboard. + +**Affected files:** + +- `src/lib/db/apiKeys.ts` ← add `max_sessions` field +- `src/lib/db/migrations/009_*` ← (can bundle with T01 migration) +- `open-sse/services/sessionManager.ts` ← enforce session count +- `src/app/(dashboard)/dashboard/api-manager/` ← UI field + +--- + +### T09 · Codex vs Codex Spark Separate Rate Limit Scopes + +**Task file:** [tasks/T09-codex-spark-rate-limit-scope.md](./tasks/T09-codex-spark-rate-limit-scope.md) + +Codex has two rate-limit scopes: `codex` (standard) and `spark` (premium). When `codex` quota is exhausted, `spark` quota may still be available. Track these separately to avoid blocking the entire account when only one scope is exhausted. + +**Affected files:** + +- `open-sse/executors/codex.ts` ← scope-aware rate limit tracking +- `open-sse/services/rateLimitSemaphore.ts` ← key by `{accountId}:{scope}` +- `src/lib/db/providers.ts` ← persist per-scope state + +--- + +### T10 · `credits_exhausted` Distinct Account Status + +**Task file:** [tasks/T10-credits-exhausted-state.md](./tasks/T10-credits-exhausted-state.md) + +When a provider returns signals like `insufficient_quota`, `billing_hard_limit_reached`, or `credits exhausted`, mark the connection as `credits_exhausted` (distinct from `error` or `rate_limited`) and skip it immediately without retry. + +**Affected files:** + +- `open-sse/chatCore.js` ← detect exhaustion signals in response body +- `src/lib/db/providers.ts` ← new status value + UI badge +- `src/app/(dashboard)/dashboard/providers/` ← display badge + +--- + +## Tier 3 — Low (P3) + +### T11 · Fix `max` Reasoning Effort → `budget_tokens: 100000` + +**Task file:** [tasks/T11-max-reasoning-effort.md](./tasks/T11-max-reasoning-effort.md) + +`reasoning_effort: "max"` should map to `budget_tokens: 100000` (Claude's "xhigh" level), not `high` (32K). + +**Affected files:** + +- `open-sse/services/thinkingBudget.ts` ← add `max: 100000` to EFFORT_BUDGETS + +--- + +### T12 · Model Pricing Updates + +**Task file:** [tasks/T12-model-pricing-updates.md](./tasks/T12-model-pricing-updates.md) + +Add missing pricing entries: MiniMax M2.5 ($0.27/$0.95), MiniMax M2.7 (TBD), GLM-4.7 ($0.38/$1.98), GLM-5, Kimi, gpt-5.4 mini. + +**Affected files:** + +- `open-sse/config/providerRegistry.ts` ← add new model entries with pricing +- `src/lib/db/settings.ts` ← default pricing table + +--- + +### T13 · Quota Display Stale After Reset + +**Task file:** [tasks/T13-stale-quota-display.md](./tasks/T13-stale-quota-display.md) + +Dashboard shows stale cumulative usage after the upstream provider resets quota windows. Fix by comparing `reset_at` timestamps before displaying counters. + +**Affected files:** + +- `src/app/(dashboard)/dashboard/providers/` ← check `resetAt` before rendering +- `src/lib/db/providers.ts` ← expose `rateLimitResetAt` in provider data + +--- + +### T14 · Proxy Fast-Fail on Dead Proxy + +**Task file:** [tasks/T14-proxy-fast-fail.md](./tasks/T14-proxy-fast-fail.md) + +When a configured proxy is unreachable, fail immediately (2s TCP check) instead of waiting 30s per request. + +**Affected files:** + +- `src/lib/apiBridgeServer.ts` ← add proxy health cache + TCP check +- `open-sse/chatCore.js` or `src/lib/proxyAgent.ts` ← gate requests through health check + +--- + +### T15 · Array Content in System/Tool Messages + +**Task file:** [tasks/T15-array-content-messages.md](./tasks/T15-array-content-messages.md) + +Some SDKs send `content: [{type:"text", text:"..."}]` (array) for system and tool messages instead of `content: "string"`. Both forms must be handled in the translator. + +**Affected files:** + +- `open-sse/translator/openai-to-claude.ts` ← normalize content field +- `open-sse/chatCore.js` ← normalize before dispatch + +--- + +## Implementation Order + +``` +Phase 1 (now, this branch): T01, T02, T03, T04 ← P1 bugs + billing +Phase 2 (next RC): T05, T06, T07, T09, T10 ← account state correctness +Phase 3 (v3.x): T08, T11, T12, T13, T14, T15 ← polish + pricing +``` + +--- + +## References + +- [sub2api GitHub](https://github.com/Wei-Shaw/sub2api) +- [gap-analysis.md](./gap-analysis.md) — full analysis with PR references +- [sub2api PR list](https://github.com/Wei-Shaw/sub2api/pulls) diff --git a/docs/new-features-sub21/tasks/T01-requested-model-log.md b/docs/new-features-sub21/tasks/T01-requested-model-log.md new file mode 100644 index 0000000000..41d9fe6344 --- /dev/null +++ b/docs/new-features-sub21/tasks/T01-requested-model-log.md @@ -0,0 +1,109 @@ +# T01 · `requested_model` in Usage Logs + +**Priority:** 🔴 P1 — Billing Transparency +**Effort:** Medium (2–4h) +**Phase:** 1 — Current branch + +--- + +## Problem + +When a user sends `model: openai/gpt-5.2-codex` but OmniRoute falls back to `gpt-5.2-mini` via combo, the `call_logs` today only stores the routed model. Users and admins cannot audit: + +- What model was requested vs what was actually used +- Which requests triggered fallbacks +- Whether billing reflects actual upstream usage + +sub2api dedicated 8 commits (Mar 19-20, 2026) to solving this across schema, DB, gateway, DTO, and billing layers. + +--- + +## References + +| Source | Link | +| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | +| sub2api PR feat (schema) | [feat(ent): add requested model to usage log schema](https://github.com/Wei-Shaw/sub2api/commit/0b845c2532198b392685aae0129b23518a106d8d) | +| sub2api PR feat (billing) | [fix(usage): preserve requested model in gateway billing paths](https://github.com/Wei-Shaw/sub2api/commit/4edcfe1f7ce50cfc7ac4d1257282b8448bd60d15) | +| sub2api PR fix (DTO) | [fix(dto): fallback to legacy model in usage mapping](https://github.com/Wei-Shaw/sub2api/commit/27948c777e4dfef2e3117a0f74cf59d433a0b524) | +| sub2api PR fix (Gemini/WS) | [fix(provider): retain upstream model for gemini compat and ws](https://github.com/Wei-Shaw/sub2api/commit/2c667a159cf5eae1d030590cf1afc48f90d9c304) | + +--- + +## Implementation Steps + +### Step 1 — DB Migration + +Create `src/lib/db/migrations/009_requested_model.sql`: + +```sql +-- Migration 009: add requested_model to call_logs for billing transparency +ALTER TABLE call_logs ADD COLUMN requested_model TEXT DEFAULT NULL; + +-- Index to allow filtering by requested model in Analytics +CREATE INDEX IF NOT EXISTS idx_call_logs_requested_model + ON call_logs(requested_model); +``` + +### Step 2 — Capture `requested_model` at Request Entry + +In `open-sse/chatCore.js`, at the very start of request handling, before any combo/routing logic: + +```js +// At the top of handleChatRequest() +const requestedModel = body.model ?? null; + +// Pass through to usage logging +const usageContext = { + requestedModel, + // ... existing fields +}; +``` + +### Step 3 — Write to DB on Completion + +In `src/lib/usageDb.ts` or wherever `call_logs` is written: + +```typescript +await db.run(` + INSERT INTO call_logs (model, requested_model, ...) + VALUES (?, ?, ...) +`, [routedModel, requestedModel, ...]); +``` + +### Step 4 — Expose in Logs Dashboard + +In `src/app/(dashboard)/dashboard/logs/`: + +- Add `requested_model` column to request logs table (collapsible, off by default) +- Show pill badge if `requested_model !== model` (fallback occurred) +- Add filter by `requested_model` + +### Step 5 — Analytics + +In usage analytics, add breakdown: + +- "Fallback rate" = requests where `requested_model !== model` / total +- Group by `requested_model` to see most-requested models + +--- + +## Files to Change + +| File | Action | +| ----------------------------------------------- | ----------------------------------- | +| `src/lib/db/migrations/009_requested_model.sql` | NEW — migration | +| `src/lib/db/core.ts` | Auto-applies migration on startup | +| `open-sse/chatCore.js` | Capture `body.model` before routing | +| `src/lib/usageDb.ts` | Write `requested_model` field | +| `src/app/(dashboard)/dashboard/logs/` | Show in table + filter | +| `src/app/(dashboard)/dashboard/analytics/` | Fallback rate metric | + +--- + +## Acceptance Criteria + +- [ ] `call_logs.requested_model` column exists after migration +- [ ] When combo falls back, `requested_model` differs from `model` in logs +- [ ] Logs page shows `requested_model` column +- [ ] Analytics shows fallback rate stat card +- [ ] No regression in existing log writes (field is nullable) diff --git a/docs/new-features-sub21/tasks/T02-strip-empty-tool-result-blocks.md b/docs/new-features-sub21/tasks/T02-strip-empty-tool-result-blocks.md new file mode 100644 index 0000000000..275b6464cb --- /dev/null +++ b/docs/new-features-sub21/tasks/T02-strip-empty-tool-result-blocks.md @@ -0,0 +1,117 @@ +# T02 · Strip Empty Text Blocks from Nested `tool_result` + +**Priority:** 🔴 P1 — Active 400 Errors +**Effort:** Small (1–2h) +**Phase:** 1 — Current branch + +--- + +## Problem + +Anthropic rejects requests where a `tool_result.content` array contains blocks with empty text: + +```json +{ "type": "tool_result", "content": [{ "type": "text", "text": "" }] } +``` + +Returns: **400** `messages: text content blocks must be non-empty` + +This is especially common with Claude Code and Codex in long tool-call chains where intermediate reasoning steps produce empty text blocks that get carried into subsequent requests. + +The fix requires **recursive** filtering — not just top-level, but also inside nested `tool_result.content` arrays. + +--- + +## References + +| Source | Link | +| ---------------- | -------------------------------------------------------------------------------------------------------------------- | +| sub2api PR #1212 | [fix(gateway): strip empty text blocks in nested tool_result.content](https://github.com/Wei-Shaw/sub2api/pull/1212) | +| Anthropic error | `messages: text content blocks must be non-empty` | +| PR detail | Adds `stripEmptyTextBlocksFromSlice()` recursive helper | + +--- + +## Implementation Steps + +### Step 1 — Add `stripEmptyTextBlocks()` utility + +In `open-sse/translator/openai-to-claude.ts` (or a shared `utils.ts`): + +```typescript +/** + * Recursively strips empty text blocks from content arrays. + * Anthropic returns 400 if any text block has text: "". + * Must handle nesting inside tool_result.content arrays. + */ +export function stripEmptyTextBlocks(content: ContentBlock[] | undefined): ContentBlock[] { + if (!content) return []; + return content + .filter((block) => { + // Remove top-level empty text blocks + if (block.type === "text" && (!block.text || block.text === "")) return false; + return true; + }) + .map((block) => { + // Recurse into tool_result.content + if (block.type === "tool_result" && Array.isArray(block.content)) { + return { ...block, content: stripEmptyTextBlocks(block.content) }; + } + return block; + }); +} +``` + +### Step 2 — Apply Before ALL Anthropic Upstream Calls + +Apply `stripEmptyTextBlocks()` on each message's content in the request body before forwarding: + +```typescript +// In openai-to-claude translator, before building anthropic body +const sanitizedMessages = messages.map((msg) => ({ + ...msg, + content: Array.isArray(msg.content) ? stripEmptyTextBlocks(msg.content) : msg.content, +})); +``` + +Apply in ALL paths sending to Anthropic: + +- `open-sse/chatCore.js` main handler +- `open-sse/responsesHandler.js` (Responses API path) +- Any passthrough / count_tokens path + +### Step 3 — Unit Tests + +Add test cases: + +```typescript +describe("stripEmptyTextBlocks", () => { + it("removes top-level empty text blocks", ...); + it("recurses into tool_result.content", ...); + it("handles deeply nested tool_result", ...); + it("preserves non-empty text blocks", ...); + it("is a no-op when no empty blocks", ...); +}); +``` + +--- + +## Files to Change + +| File | Action | +| ----------------------------------------- | ------------------------------------- | +| `open-sse/translator/openai-to-claude.ts` | Add `stripEmptyTextBlocks()` function | +| `open-sse/chatCore.js` | Apply before Anthropic forwarding | +| `open-sse/responsesHandler.js` | Apply before Anthropic forwarding | +| `open-sse/tests/` | Unit tests for new function | + +--- + +## Acceptance Criteria + +- [ ] Empty text blocks at top level are stripped +- [ ] Empty text blocks inside `tool_result.content` are stripped +- [ ] Multi-level nesting is handled recursively +- [ ] No existing non-empty text blocks are modified +- [ ] All existing tests continue to pass +- [ ] Claude Code + tool-heavy scenarios no longer produce 400 diff --git a/docs/new-features-sub21/tasks/T03-codex-quota-reset-headers.md b/docs/new-features-sub21/tasks/T03-codex-quota-reset-headers.md new file mode 100644 index 0000000000..fbf664c12a --- /dev/null +++ b/docs/new-features-sub21/tasks/T03-codex-quota-reset-headers.md @@ -0,0 +1,136 @@ +# T03 · Parse `x-codex-*` Headers for Precise Quota Reset Times + +**Priority:** 🔴 P1 — UX for Codex Users +**Effort:** Medium (3–5h) +**Phase:** 1 — Current branch + +--- + +## Problem + +Codex (ChatGPT) responses include response headers with precise quota information: + +- `x-codex-5h-usage` — tokens used in the 5-hour window +- `x-codex-5h-limit` — token limit for the 5-hour window +- `x-codex-5h-reset-at` — ISO timestamp when the 5h window resets +- `x-codex-7d-usage` — tokens used in the 7-day window +- `x-codex-7d-limit` — token limit for the 7-day window +- `x-codex-7d-reset-at` — ISO timestamp when the 7d window resets + +Today OmniRoute falls back to a generic circuit-breaker cooldown (~5 minutes) when hitting Codex quota. The real reset could be hours away — causing repeated failed retries — or just seconds away — causing unnecessary skip of a valid account. + +sub2api PR #357 implements full parsing of these headers to store exact reset timestamps per account. + +--- + +## References + +| Source | Link | +| --------------- | -------------------------------------------------------------------------------------------------------- | +| sub2api PR #357 | [feat(oauth): persist usage snapshots and window cooldown](https://github.com/Wei-Shaw/sub2api/pull/357) | +| Codex endpoint | `https://chatgpt.com/backend-api/codex/responses` | +| Headers | `x-codex-5h-*`, `x-codex-7d-*` | +| Reset endpoint | 7d priority over 5h | + +--- + +## Implementation Steps + +### Step 1 — Parse Headers in `codex.ts` + +After every Codex response, extract and persist quota data: + +```typescript +// In codex.ts after receiving response +function parseCodexQuotaHeaders(headers: Headers) { + return { + usage5h: parseFloat(headers.get("x-codex-5h-usage") ?? "0"), + limit5h: parseFloat(headers.get("x-codex-5h-limit") ?? "Infinity"), + resetAt5h: headers.get("x-codex-5h-reset-at") ?? null, // ISO string + usage7d: parseFloat(headers.get("x-codex-7d-usage") ?? "0"), + limit7d: parseFloat(headers.get("x-codex-7d-limit") ?? "Infinity"), + resetAt7d: headers.get("x-codex-7d-reset-at") ?? null, // ISO string + }; +} + +const quota = parseCodexQuotaHeaders(response.headers); + +// Write to connection extra/metadata +await updateConnectionQuota(connection.id, { + codex5hUsage: quota.usage5h, + codex5hLimit: quota.limit5h, + codex5hResetAt: quota.resetAt5h ? new Date(quota.resetAt5h).getTime() : null, + codex7dUsage: quota.usage7d, + codex7dLimit: quota.limit7d, + codex7dResetAt: quota.resetAt7d ? new Date(quota.resetAt7d).getTime() : null, +}); +``` + +### Step 2 — Persist to DB + +In `src/lib/db/providers.ts`, extend connection extras to hold quota state: + +```typescript +// Store as JSON in connection.extra column (already exists) +const extra = { + ...existingExtra, + codex_5h_usage: quota.usage5h, + codex_5h_limit: quota.limit5h, + codex_5h_reset_at: quota.resetAt5h, + codex_7d_usage: quota.usage7d, + codex_7d_limit: quota.limit7d, + codex_7d_reset_at: quota.resetAt7d, +}; +await updateConnectionExtra(connection.id, extra); +``` + +### Step 3 — Block Account Until Exact Reset Time + +When Codex returns 429 (quota exhausted), use the parsed header to set precise block: + +```typescript +// In codex.ts error handler on 429 +const resetAt = quota.resetAt7d ?? quota.resetAt5h; +if (resetAt) { + const resetMs = new Date(resetAt).getTime(); + await setConnectionRateLimitUntil(connection.id, resetMs); + // Circuit breaker will check this timestamp before selecting account +} +``` + +### Step 4 — Display in Dashboard + +In the Providers page, show Codex quota progress bars: + +- 5h window: `usage5h / limit5h` with countdown to `resetAt5h` +- 7d window: `usage7d / limit7d` with countdown to `resetAt7d` + +### Step 5 — Add nonce to test requests + +When running connection tests, add a random nonce to avoid upstream response caching: + +```typescript +body.nonce = crypto.randomUUID(); +``` + +--- + +## Files to Change + +| File | Action | +| ------------------------------------------ | ----------------------------------------------------- | +| `open-sse/executors/codex.ts` | Parse `x-codex-*` headers after every response | +| `src/lib/db/providers.ts` | `updateConnectionExtra()` with quota fields | +| `src/lib/tokenHealthCheck.ts` | Check `rateLimitUntil` before selecting Codex account | +| `src/app/(dashboard)/dashboard/providers/` | Quota progress bar + reset countdown | + +--- + +## Acceptance Criteria + +- [ ] After a Codex request, `x-codex-5h-*` and `x-codex-7d-*` headers are parsed +- [ ] Quota state is persisted in connection extras +- [ ] When quota exhausted, account is blocked until exact `reset_at` time (not 5min) +- [ ] Dashboard shows 5h and 7d quota bars for Codex connections +- [ ] Countdown timer visible +- [ ] On 429, the next request correctly skips the account until reset time diff --git a/docs/new-features-sub21/tasks/T04-x-session-id-header.md b/docs/new-features-sub21/tasks/T04-x-session-id-header.md new file mode 100644 index 0000000000..4c1ecd928f --- /dev/null +++ b/docs/new-features-sub21/tasks/T04-x-session-id-header.md @@ -0,0 +1,107 @@ +# T04 · `X-Session-Id` Header for External Sticky Routing + +**Priority:** 🔴 P1 — Nginx Users / Sticky Session +**Effort:** Small (1–2h) +**Phase:** 1 — Current branch + +--- + +## Problem + +OmniRoute generates session IDs internally (content hash of the conversation). There is no way for a client to explicitly pin a conversation to a specific account. This breaks in two scenarios: + +1. **Nginx reverse proxy:** Nginx drops headers with underscores by default. If a client sends `session_id`, it gets dropped silently. sub2api's README explicitly notes: add `underscores_in_headers on` to Nginx config. We should accept the hyphenated `X-Session-Id` form which Nginx passes cleanly. + +2. **Multi-account clients:** Tools like Codex or Claude Code that maintain multi-turn conversations may not send identical first messages — causing OmniRoute to generate a different session ID per turn, defeating sticky routing. + +--- + +## References + +| Source | Link | +| --------------- | ----------------------------------------------------------------------------------------------------- | +| sub2api README | Nginx note: `underscores_in_headers on` for `session_id` header | +| sub2api PR #634 | [stabilize session hash + add user-level session limit](https://github.com/Wei-Shaw/sub2api/pull/634) | +| OmniRoute | `open-sse/services/sessionManager.ts` | + +--- + +## Implementation Steps + +### Step 1 — Read `X-Session-Id` from Incoming Request + +In `open-sse/chatCore.js` at the start of the request handler: + +```js +// Priority: client-provided session ID > internally generated hash +const clientSessionId = + req.headers["x-session-id"] || // hyphenated (Nginx passes this) + req.headers["x_session_id"] || // underscore (direct HTTP, no Nginx) + req.headers["session-id"] || // alternate form + null; + +const sessionId = clientSessionId + ? `ext:${clientSessionId}` // prefix to avoid collision with internal IDs + : generateSessionHash(body, req); // existing logic +``` + +### Step 2 — Pass Session ID to Sticky Routing + +In `open-sse/services/sessionManager.ts`, the `getOrBindConnection()` function already accepts a session ID — just pass the external one: + +```typescript +const connection = await sessionManager.getOrBindConnection({ + sessionId, // external if provided, internal hash otherwise + providerId, + comboId, +}); +``` + +### Step 3 — Propagate Session ID in Response Headers + +Echo back the session ID so clients know what was used: + +```js +// In response handler +res.setHeader("X-OmniRoute-Session-Id", sessionId); +``` + +### Step 4 — Document Nginx Configuration + +Add to `docs/` and the dashboard Endpoints tab: + +```nginx +http { + underscores_in_headers on; # Required for X-Session-Id (or session_id) header + ... +} +``` + +And in the Claude Code / Codex setup guide, document: + +```bash +# Pin your session to the same Codex account for long conversations +export ANTHROPIC_EXTRA_HEADERS='{"X-Session-Id": "my-project-session-1"}' +``` + +--- + +## Files to Change + +| File | Action | +| ----------------------------------------- | ---------------------------------------------- | +| `open-sse/chatCore.js` | Read `x-session-id` header at request entry | +| `open-sse/services/sessionManager.ts` | Accept external session ID as primary key | +| `open-sse/chatCore.js` | Echo back `X-OmniRoute-Session-Id` in response | +| `docs/` or provider setup guides | Document Nginx `underscores_in_headers on` | +| `src/app/(dashboard)/dashboard/endpoint/` | Mention in API Endpoints tab | + +--- + +## Acceptance Criteria + +- [ ] `X-Session-Id: abc123` header binds conversation to same account +- [ ] `x_session_id` underscore form also accepted (direct HTTP without Nginx) +- [ ] Response includes `X-OmniRoute-Session-Id` header +- [ ] Without the header, existing session hash behavior unchanged +- [ ] Nginx setup guide mentions `underscores_in_headers on` diff --git a/docs/new-features-sub21/tasks/T05-persist-ratelimit-state.md b/docs/new-features-sub21/tasks/T05-persist-ratelimit-state.md new file mode 100644 index 0000000000..eaae8c399a --- /dev/null +++ b/docs/new-features-sub21/tasks/T05-persist-ratelimit-state.md @@ -0,0 +1,127 @@ +# T05 · Persist Rate-Limit State in DB (Survives Token Refresh) + +**Priority:** 🟠 P2 — Routing Correctness +**Effort:** Medium (3–4h) +**Phase:** 2 — Next RC + +--- + +## Problem + +When an account hits a rate limit, OmniRoute records the state in memory (circuit breaker). However, when OAuth token refresh runs for that same account, it can accidentally clear the in-memory rate-limit state — causing the freshly-refreshed (but still rate-limited) account to be selected again immediately. + +This creates a "rate-limit loop": the account gets selected, fails with 429, refreshes its token, gets selected again, fails again. + +sub2api PR #1218 solves this by: + +1. Persisting rate-limit state in the DB (not just in-memory) +2. Re-checking DB state before final account selection +3. Ensuring credentials-only updates (token refresh) don't clear rate-limit flags + +--- + +## References + +| Source | Link | +| ---------------- | -------------------------------------------------------------------------------------------------------- | +| sub2api PR #1218 | [fix(openai): prevent rescheduling rate-limited accounts](https://github.com/Wei-Shaw/sub2api/pull/1218) | +| Problem | Token refresh clears in-memory rate-limit state | +| fix | DB-backed rate state + re-check before selection | + +--- + +## Implementation Steps + +### Step 1 — Add `rate_limited_until` to Connection Schema + +In `src/lib/db/providers.ts`: + +```typescript +// On connection write, persist rate limit state +async function setConnectionRateLimitUntil( + connectionId: string, + until: number | null // epoch ms, null = not rate limited +): Promise { + await db.run(`UPDATE provider_connections SET rate_limited_until = ? WHERE id = ?`, [ + until, + connectionId, + ]); +} + +async function isConnectionRateLimited(connectionId: string): Promise { + const row = await db.get(`SELECT rate_limited_until FROM provider_connections WHERE id = ?`, [ + connectionId, + ]); + if (!row?.rate_limited_until) return false; + return Date.now() < row.rate_limited_until; +} +``` + +### Step 2 — Migration + +Add to migration 009 (or a new 010): + +```sql +ALTER TABLE provider_connections ADD COLUMN rate_limited_until INTEGER DEFAULT NULL; +CREATE INDEX IF NOT EXISTS idx_connections_rate_limit ON provider_connections(rate_limited_until); +``` + +### Step 3 — Write on 429 + +In `open-sse/chatCore.js` or executor error handler: + +```js +if (status === 429) { + const resetAt = parseRateLimitReset(response.headers) ?? Date.now() + 5 * 60 * 1000; + await setConnectionRateLimitUntil(connectionId, resetAt); +} +``` + +### Step 4 — Check Before Selection + +In `src/sse/services/auth.ts` account selection logic: + +```typescript +// Before returning an account/connection +const rateLimited = await isConnectionRateLimited(connection.id); +if (rateLimited) { + // Skip this connection, try next + continue; +} +``` + +### Step 5 — Token Refresh Must NOT Clear Rate Limit + +In OAuth token refresh flow: + +```typescript +// Only update tokens — do NOT touch rate_limited_until +await db.run( + `UPDATE provider_connections SET access_token = ?, refresh_token = ?, expires_at = ? + WHERE id = ?`, + [newToken, newRefresh, newExpiry, connectionId] + // NO rate_limited_until update here +); +``` + +--- + +## Files to Change + +| File | Action | +| --------------------------------- | ------------------------------------------------------------ | +| `src/lib/db/migrations/009_*.sql` | Add `rate_limited_until` column | +| `src/lib/db/providers.ts` | `setConnectionRateLimitUntil()`, `isConnectionRateLimited()` | +| `src/sse/services/auth.ts` | Check DB state before account selection | +| `open-sse/chatCore.js` | Write rate-limit on 429 (with header-parsed reset time) | +| `src/lib/oauth/` | Token refresh flows — don't touch rate_limited_until | + +--- + +## Acceptance Criteria + +- [ ] 429 response writes `rate_limited_until` to DB +- [ ] Account selection skips connections with non-expired `rate_limited_until` +- [ ] Token refresh does not clear `rate_limited_until` +- [ ] After `rate_limited_until` expires, account is available again +- [ ] Dashboard shows "Rate Limited until HH:MM" for affected connections diff --git a/docs/new-features-sub21/tasks/T06-account-deactivated-status.md b/docs/new-features-sub21/tasks/T06-account-deactivated-status.md new file mode 100644 index 0000000000..3b9b1364bb --- /dev/null +++ b/docs/new-features-sub21/tasks/T06-account-deactivated-status.md @@ -0,0 +1,103 @@ +# T06 · `account_deactivated` → Permanent `expired` Status on 401 + +**Priority:** 🟠 P2 — Account State Correctness +**Effort:** Small (1–2h) +**Phase:** 2 — Next RC + +--- + +## Problem + +When an upstream provider returns `401` with `"account_deactivated"` in the response body, OmniRoute currently treats this the same as other 401 errors (expired token → trigger re-auth → retry). This causes: + +- Repeated re-auth attempts on a permanently dead account +- Wasted API calls and quota on upstream +- Confusing error messages for users ("failed to refresh token" vs "account is deactivated") + +The correct behavior: immediately mark the connection as permanently `expired` and stop retrying. + +--- + +## References + +| Source | Link | +| ---------------- | --------------------------------------------------------------------------------------------------------- | +| sub2api PR #1037 | [fix(ratelimit): handle upstream 401 account deactivation](https://github.com/Wei-Shaw/sub2api/pull/1037) | +| Error signal | `401` body containing `account_deactivated` | +| Action | Mark as `expired` permanently, skip retries | + +--- + +## Implementation Steps + +### Step 1 — Define Deactivation Signal Strings + +In `src/shared/constants/providers.ts` or a new error constants file: + +```typescript +export const ACCOUNT_DEACTIVATED_SIGNALS = [ + "account_deactivated", + "account_has_been_deactivated", + "your account has been deactivated", + "this account has been disabled", +]; +``` + +### Step 2 — Detect in 401 Handler + +In `open-sse/chatCore.js` or the generic error handler in each executor: + +```typescript +async function handle401Error(connectionId: string, responseBody: string): Promise { + const isDeactivated = ACCOUNT_DEACTIVATED_SIGNALS.some((signal) => + responseBody.toLowerCase().includes(signal) + ); + + if (isDeactivated) { + // Permanent failure — do NOT retry, do NOT re-auth + await updateConnectionStatus(connectionId, "expired"); + throw new PermanentAccountError(`Account deactivated: ${connectionId}`); + } + + // Regular 401 — try token refresh + await triggerTokenRefresh(connectionId); +} +``` + +### Step 3 — Stop Retry Loop + +Ensure `PermanentAccountError` (or equivalent flag) causes the combo to: + +1. Skip this connection for the rest of the request +2. NOT add it back to the retry queue +3. Fall through to the next combo member + +### Step 4 — Show Badge in Dashboard + +In `src/app/(dashboard)/dashboard/providers/`: + +- Show a red "Deactivated" badge for connections with `expired` status that was set due to deactivation (vs normal expiry) +- Add optional tooltip: "Marked as deactivated by upstream. Re-connect to restore." + +--- + +## Files to Change + +| File | Action | +| ------------------------------------------ | ---------------------------------------- | +| `src/shared/constants/providers.ts` | Add `ACCOUNT_DEACTIVATED_SIGNALS` array | +| `open-sse/chatCore.js` | Detect deactivation in 401 handler | +| `open-sse/executors/codex.ts` | Apply same check | +| `open-sse/executors/claude.ts` | Apply same check | +| `src/lib/db/providers.ts` | `updateConnectionStatus("expired")` call | +| `src/app/(dashboard)/dashboard/providers/` | "Deactivated" badge UI | + +--- + +## Acceptance Criteria + +- [ ] 401 with `account_deactivated` marks connection as `expired` immediately +- [ ] No re-auth attempts are made on deactivated accounts +- [ ] Combo correctly falls through to next member after deactivation +- [ ] Dashboard shows "Deactivated" status badge +- [ ] Normal 401 (expired token) still triggers re-auth correctly diff --git a/docs/new-features-sub21/tasks/T07-xff-validation.md b/docs/new-features-sub21/tasks/T07-xff-validation.md new file mode 100644 index 0000000000..fbaa90655a --- /dev/null +++ b/docs/new-features-sub21/tasks/T07-xff-validation.md @@ -0,0 +1,103 @@ +# T07 · X-Forwarded-For Validation — Skip Non-IP Entries + +**Priority:** 🟠 P2 — IP Rate Limiting Correctness +**Effort:** Tiny (<1h) +**Phase:** 2 — Next RC + +--- + +## Problem + +When running behind Nginx, Cloudflare, or other proxies, the `X-Forwarded-For` header may contain invalid entries like `"unknown"`: + +``` +X-Forwarded-For: unknown, 1.2.3.4, 10.0.0.1 +``` + +If OmniRoute naively reads the first entry (`"unknown"`), IP-based rate limiting and IP filtering fail. The correct behavior is to skip non-valid IP entries and fall back to the first valid one, or to `req.socket.remoteAddress`. + +--- + +## References + +| Source | Link | +| ---------------- | ----------------------------------------------------------------------------------------------------------------- | +| sub2api PR #1135 | [fix: skip invalid X-Forwarded-For entries in client IP detection](https://github.com/Wei-Shaw/sub2api/pull/1135) | +| Problem | `unknown` in XFF breaks IP rate limiting | +| Fix | Skip non-IP entries, fall back to Gin `ClientIP()` equivalent | + +--- + +## Implementation Steps + +### Step 1 — Add IP Validation Helper + +In `open-sse/chatCore.js` or a shared `src/lib/ipUtils.ts`: + +```typescript +import { isIP } from "node:net"; // returns 4, 6, or 0 + +/** + * Extract the real client IP from X-Forwarded-For header. + * Skips invalid entries like "unknown" or empty strings. + * Falls back to remoteAddress if no valid IP found. + */ +export function extractClientIp( + xForwardedFor: string | null, + remoteAddress: string | undefined +): string { + if (xForwardedFor) { + const entries = xForwardedFor.split(",").map((s) => s.trim()); + for (const entry of entries) { + if (entry && isIP(entry) !== 0) { + return entry; // first valid IP + } + } + } + return remoteAddress ?? "unknown"; +} +``` + +### Step 2 — Use in Rate Limiter + +Replace any current `req.headers["x-forwarded-for"]?.split(",")[0]` with: + +```typescript +const clientIp = extractClientIp( + (req.headers["x-forwarded-for"] as string) ?? null, + req.socket?.remoteAddress +); +``` + +### Step 3 — Use in IP Allowlist/Blocklist + +Same function should be used for IP filtering middleware. + +### Step 4 — Add IP Whitespace Trimming + +Also related (sub2api PR #1136): trim whitespace before validating patterns: + +```typescript +// When checking IP against allowlist patterns +const cleanIp = clientIp.trim(); // remove accidental whitespace +``` + +--- + +## Files to Change + +| File | Action | +| ------------------------- | ----------------------------------------- | +| `src/lib/ipUtils.ts` | NEW — `extractClientIp()` helper | +| `open-sse/chatCore.js` | Use `extractClientIp()` for rate limiting | +| `src/app/api/` middleware | Use `extractClientIp()` for IP filtering | + +--- + +## Acceptance Criteria + +- [ ] `X-Forwarded-For: unknown, 1.2.3.4` correctly returns `1.2.3.4` +- [ ] All-invalid XFF falls back to `remoteAddress` +- [ ] Empty XFF falls back to `remoteAddress` +- [ ] IP whitelist/blocklist uses the same extraction logic +- [ ] No behavioral change when XFF is a valid single IP diff --git a/docs/new-features-sub21/tasks/T08-per-key-session-limit.md b/docs/new-features-sub21/tasks/T08-per-key-session-limit.md new file mode 100644 index 0000000000..931b0012ec --- /dev/null +++ b/docs/new-features-sub21/tasks/T08-per-key-session-limit.md @@ -0,0 +1,125 @@ +# T08 · Per-API-Key Session Limit (`max_sessions`) + +**Priority:** 🟠 P2 — Abuse Prevention +**Effort:** Medium (3–5h) +**Phase:** 2 — Next RC + +--- + +## Problem + +A single API key can maintain unlimited concurrent sticky sessions. In shared or multi-user setups, this means one key can monopolize all available upstream accounts by opening many parallel conversations. There's no way for an admin to cap concurrency at the key level. + +sub2api PR #634 adds a `max_sessions` field per user, enforced via Redis Lua scripts that count active sessions per key. + +--- + +## References + +| Source | Link | +| --------------- | ---------------------------------------------------------------------------------------------------------- | +| sub2api PR #634 | [fix: stabilize session hash + add user-level session limit](https://github.com/Wei-Shaw/sub2api/pull/634) | +| Redis key | `session_limit:user:{userID}` | +| HTTP response | `429` with message "You have reached the maximum number of active sessions" | +| Default | `max_sessions = 0` = unlimited (backward compatible) | + +--- + +## Implementation Steps + +### Step 1 — Add `max_sessions` to API Keys + +Extend the `api_keys` table in a migration: + +```sql +ALTER TABLE api_keys ADD COLUMN max_sessions INTEGER DEFAULT 0; +-- 0 = unlimited +``` + +### Step 2 — Track Active Sessions Per Key + +In `open-sse/services/sessionManager.ts`, maintain an in-memory map of active sessions per API key: + +```typescript +// Map: apiKeyId → Set +const activeSessionsByKey = new Map>(); + +function getActiveSessionCount(apiKeyId: string): number { + return activeSessionsByKey.get(apiKeyId)?.size ?? 0; +} + +function registerSession(apiKeyId: string, sessionId: string): void { + if (!activeSessionsByKey.has(apiKeyId)) { + activeSessionsByKey.set(apiKeyId, new Set()); + } + activeSessionsByKey.get(apiKeyId)!.add(sessionId); +} + +function unregisterSession(apiKeyId: string, sessionId: string): void { + activeSessionsByKey.get(apiKeyId)?.delete(sessionId); +} +``` + +### Step 3 — Enforce Limit Before Session Creation + +In `open-sse/chatCore.js` before session binding: + +```js +const maxSessions = apiKey.max_sessions ?? 0; +if (maxSessions > 0) { + const current = getActiveSessionCount(apiKeyId); + if (current >= maxSessions) { + return res.status(429).json({ + error: { + message: + `You have reached the maximum number of active sessions (${maxSessions}). ` + + `Please close unused sessions or wait for them to expire.`, + type: "session_limit_exceeded", + code: "SESSION_LIMIT_EXCEEDED", + }, + }); + } +} +// Proceed with session registration +registerSession(apiKeyId, sessionId); +``` + +### Step 4 — Clean Up on Session End + +When a streaming response ends or aborts: + +```js +req.on("close", () => { + unregisterSession(apiKeyId, sessionId); +}); +``` + +### Step 5 — UI in API Manager + +In `src/app/(dashboard)/dashboard/api-manager/`: + +- Add "Max Sessions" input field (0 = unlimited) +- Show current active session count per key +- Show "∞" when max_sessions = 0 + +--- + +## Files to Change + +| File | Action | +| -------------------------------------------- | -------------------------------------- | +| `src/lib/db/migrations/009_*.sql` | Add `max_sessions` to `api_keys` | +| `src/lib/db/apiKeys.ts` | Include `max_sessions` in key metadata | +| `open-sse/services/sessionManager.ts` | Track + enforce active session count | +| `open-sse/chatCore.js` | Check limit before session binding | +| `src/app/(dashboard)/dashboard/api-manager/` | UI field + active session count | + +--- + +## Acceptance Criteria + +- [ ] `max_sessions: 0` allows unlimited sessions (default) +- [ ] `max_sessions: 3` returns 429 on the 4th concurrent session +- [ ] Session count decrements when request closes/finishes +- [ ] API Manager shows `max_sessions` field and active count +- [ ] Existing keys with no `max_sessions` field behave as unlimited diff --git a/docs/new-features-sub21/tasks/T09-codex-spark-rate-limit-scope.md b/docs/new-features-sub21/tasks/T09-codex-spark-rate-limit-scope.md new file mode 100644 index 0000000000..9f653f89a3 --- /dev/null +++ b/docs/new-features-sub21/tasks/T09-codex-spark-rate-limit-scope.md @@ -0,0 +1,116 @@ +# T09 · Codex vs Codex Spark — Separate Rate Limit Scopes + +**Priority:** 🟠 P2 — Codex Quota Efficiency +**Effort:** Small (2h) +**Phase:** 2 — Next RC + +--- + +## Problem + +OpenAI Codex has two model tiers within the same account: + +- `codex` — standard (gpt-5.2-codex, etc.) +- `spark` — premium/more capable (codex-spark models) + +Each tier has its own independent quota. When the `codex` tier exhausts its quota, the whole account gets circuit-breakered — even though the `spark` tier may still have quota available (or vice versa). + +sub2api PR #1129 fixes this by splitting rate-limit state into two scopes and only blocking the account globally when ALL scopes are exhausted. + +--- + +## References + +| Source | Link | +| ---------------- | --------------------------------------------------------------------------------------------------------- | +| sub2api PR #1129 | [feat(openai): split codex spark rate limiting from codex](https://github.com/Wei-Shaw/sub2api/pull/1129) | +| Problem | Exhausting `codex` quota blocks `spark` unnecessarily | +| Fix | Scope-aware rate limit tracking per model family | + +--- + +## Implementation Steps + +### Step 1 — Define Codex Scopes + +In `open-sse/executors/codex.ts`: + +```typescript +const CODEX_SCOPE_PATTERNS: Record = { + // model name pattern → rate limit scope + "gpt-5.2-codex": "codex", + "codex-spark": "spark", + "codex-mini": "codex", + // default (anything else): "codex" +}; + +function getModelScope(model: string): "codex" | "spark" { + for (const [pattern, scope] of Object.entries(CODEX_SCOPE_PATTERNS)) { + if (model.includes(pattern)) return scope as "codex" | "spark"; + } + return "codex"; // default scope +} +``` + +### Step 2 — Scope-keyed Rate Limit State + +In `open-sse/services/rateLimitSemaphore.ts`, change the key from `{accountId}` to `{accountId}:{scope}`: + +```typescript +// Before: rateLimitState.get(accountId) +// After: +const scopeKey = `${accountId}:${scope}`; +rateLimitState.get(scopeKey); +``` + +### Step 3 — Persist Per-Scope State + +In `src/lib/db/providers.ts`, store scope-aware rate limits in connection extras: + +```typescript +// In connection extra JSON: +{ + "rate_limited": { + "codex": { "until": 1234567890 }, + "spark": null + } +} +``` + +### Step 4 — Only Block Account When ALL Scopes Exhausted + +In account selection logic: + +```typescript +function isAccountFullyRateLimited(connection: Connection, requestedScope: string): boolean { + const scopeState = connection.extra?.rate_limited?.[requestedScope]; + if (!scopeState?.until) return false; + return Date.now() < scopeState.until; +} +// Only skip account if the REQUESTED scope is rate-limited +``` + +### Step 5 — Clear Scope on Reset + +When the reset timer fires for one scope, clear only that scope — not all scopes. + +--- + +## Files to Change + +| File | Action | +| ----------------------------------------- | ------------------------------------------- | +| `open-sse/executors/codex.ts` | `getModelScope()`, scope-aware 429 handling | +| `open-sse/services/rateLimitSemaphore.ts` | Scope-suffixed rate limit keys | +| `src/lib/db/providers.ts` | Per-scope state in connection extras | +| `src/sse/services/auth.ts` | Check scope before skipping account | + +--- + +## Acceptance Criteria + +- [ ] `codex` quota exhaustion does NOT block requests to `spark` models +- [ ] `spark` quota exhaustion does NOT block requests to `codex` models +- [ ] Both scopes exhausted → account fully skipped +- [ ] Scope state persists in DB (survives token refresh) +- [ ] Dashboard shows per-scope quota status diff --git a/docs/new-features-sub21/tasks/T10-credits-exhausted-state.md b/docs/new-features-sub21/tasks/T10-credits-exhausted-state.md new file mode 100644 index 0000000000..4afa37d45b --- /dev/null +++ b/docs/new-features-sub21/tasks/T10-credits-exhausted-state.md @@ -0,0 +1,103 @@ +# T10 · `credits_exhausted` Distinct Account Status + +**Priority:** 🟠 P2 — Account State Visibility +**Effort:** Small (2h) +**Phase:** 2 — Next RC + +--- + +## Problem + +When a provider's response indicates the account has exhausted its credits (not a temporary rate limit, but a billing/quota hard stop), OmniRoute currently treats it as a generic error and increments the circuit breaker. This causes: + +- Repeated retries on an account that won't succeed until credits are refilled +- Circuit breaker trips unnecessarily +- No visual distinction in the dashboard between "rate limited" and "out of credits" + +--- + +## References + +| Source | Link | +| -------------- | ----------------------------------------------------------------------------- | +| sub2api commit | `fix: credits-exhausted handling` (PR #1169) | +| Error signals | `insufficient_quota`, `billing_hard_limit_reached`, `credits exhausted`, etc. | +| Action | Mark as `credits_exhausted`, skip immediately, show badge | + +--- + +## Implementation Steps + +### Step 1 — Define Credits Exhaustion Signals + +In `src/shared/constants/providers.ts`: + +```typescript +export const CREDITS_EXHAUSTED_SIGNALS = [ + // OpenAI + "insufficient_quota", + "billing_hard_limit_reached", + "exceeded your current quota", + // Anthropic + "credit_balance_too_low", + "your credit balance is too low", + // Generic + "credits exhausted", + "quota exceeded", + "payment required", + "account balance", +]; +``` + +### Step 2 — Detect in Error Handler + +In `open-sse/chatCore.js`: + +```js +function isCreditsExhausted(status, body) { + if (status !== 402 && status !== 429 && status !== 400) return false; + const bodyLower = (typeof body === "string" ? body : JSON.stringify(body)).toLowerCase(); + return CREDITS_EXHAUSTED_SIGNALS.some((signal) => bodyLower.includes(signal)); +} + +// In error handling: +if (isCreditsExhausted(status, responseBody)) { + await updateConnectionStatus(connectionId, "credits_exhausted"); + // Skip this connection, no retry + skipConnection = true; +} +``` + +### Step 3 — Distinct Status in DB + +In `src/lib/db/providers.ts`, ensure `credits_exhausted` is a valid connection status alongside `active`, `expired`, `error`, `rate_limited`. + +### Step 4 — UI Badge + +In `src/app/(dashboard)/dashboard/providers/`: + +- Show 🟡 amber "Credits Exhausted" badge (distinct from 🔴 "Error" and 🟠 "Rate Limited") +- Badge tooltip: "This account is out of credits. Reload credits on the provider dashboard." +- Allow manual reset to `active` status after user has topped up credits + +--- + +## Files to Change + +| File | Action | +| ------------------------------------------ | ----------------------------------------------- | +| `src/shared/constants/providers.ts` | `CREDITS_EXHAUSTED_SIGNALS` array | +| `open-sse/chatCore.js` | `isCreditsExhausted()` check in error handler | +| `src/lib/db/providers.ts` | Accept `credits_exhausted` as connection status | +| `src/shared/constants/providers.ts` | Add to `ConnectionStatus` enum/union | +| `src/app/(dashboard)/dashboard/providers/` | "Credits Exhausted" badge | + +--- + +## Acceptance Criteria + +- [ ] `402 Payment Required` or `429` with `insufficient_quota` → `credits_exhausted` status +- [ ] Account immediately skipped (no retry, no circuit breaker increment) +- [ ] Dashboard shows amber "Credits Exhausted" badge +- [ ] Manual reset to `active` possible from dashboard +- [ ] Rate-limited (429 without exhaustion signal) still uses circuit breaker normally diff --git a/docs/new-features-sub21/tasks/T11-max-reasoning-effort.md b/docs/new-features-sub21/tasks/T11-max-reasoning-effort.md new file mode 100644 index 0000000000..ab0005273a --- /dev/null +++ b/docs/new-features-sub21/tasks/T11-max-reasoning-effort.md @@ -0,0 +1,86 @@ +# T11 · Fix `max` Reasoning Effort → `budget_tokens: 100000` + +**Priority:** 🟡 P3 — Claude Max Reasoning +**Effort:** Tiny (<30min) +**Phase:** 3 — v3.x + +--- + +## Problem + +`reasoning_effort: "max"` from OpenAI clients should map to Anthropic's highest thinking budget (`budget_tokens: 100000`, internally called "xhigh"). Currently `thinkingBudget.ts` likely maps `max` to `high` (32K tokens), losing 68K thinking tokens. + +sub2api fixed this in a merged commit: `fix(apicompat): 修正 Anthropic→OpenAI 推理级别映射` and `fix: openai-messages-effort-max-to-xhigh`. + +--- + +## References + +| Source | Link | +| -------------- | ------------------------------------------------------------------ | +| sub2api fix | `fix(apicompat): correct Anthropic→OpenAI reasoning level mapping` | +| sub2api fix | `fix: openai-messages-effort-max-to-xhigh` | +| Anthropic max | `budget_tokens: 100000` | +| OmniRoute file | `open-sse/services/thinkingBudget.ts` | + +--- + +## Implementation Steps + +### Step 1 — Update `EFFORT_BUDGETS` Map + +In `open-sse/services/thinkingBudget.ts`: + +```typescript +// Before: +const EFFORT_BUDGETS = { low: 1024, medium: 10240, high: 32000 }; + +// After: +const EFFORT_BUDGETS = { + low: 1024, + medium: 10240, + high: 32000, + max: 100000, // ← Claude "xhigh" = 100K tokens + xhigh: 100000, // ← explicit alias for internal use +}; +``` + +### Step 2 — Reverse Mapping (Anthropic → OpenAI) + +When translating Claude's `budget_tokens` back to OpenAI `reasoning_effort`: + +```typescript +function budgetToEffort(budget: number): string { + if (budget >= 100000) return "max"; // ← was missing this + if (budget >= 32000) return "high"; + if (budget >= 10240) return "medium"; + return "low"; +} +``` + +### Step 3 — Verify `reasoning.effort` field + +Also check the `reasoning: { effort: "max" }` form (OpenAI Responses API): + +```typescript +const effort = body.reasoning?.effort ?? body.reasoning_effort ?? null; +// "max" must map to 100000 +``` + +--- + +## Files to Change + +| File | Action | +| ------------------------------------- | ----------------------------------------- | +| `open-sse/services/thinkingBudget.ts` | Add `max: 100000` to `EFFORT_BUDGETS` | +| `open-sse/services/thinkingBudget.ts` | Fix reverse mapping in `budgetToEffort()` | + +--- + +## Acceptance Criteria + +- [ ] `reasoning_effort: "max"` sends `budget_tokens: 100000` to Anthropic +- [ ] `reasoning: { effort: "max" }` also maps correctly +- [ ] `budget_tokens: 100000` maps back to `reasoning_effort: "max"` in response +- [ ] Existing `low`/`medium`/`high` behavior unchanged diff --git a/docs/new-features-sub21/tasks/T12-model-pricing-updates.md b/docs/new-features-sub21/tasks/T12-model-pricing-updates.md new file mode 100644 index 0000000000..43bf29f4f0 --- /dev/null +++ b/docs/new-features-sub21/tasks/T12-model-pricing-updates.md @@ -0,0 +1,95 @@ +# T12 · Model Pricing Updates (MiniMax, GLM, Kimi, gpt-5.4 mini) + +**Priority:** 🟡 P3 — Catalog Completeness +**Effort:** Tiny (<1h) +**Phase:** 3 — v3.x + +--- + +## Problem + +Several newer models used by our providers are either missing from OmniRoute's pricing table or have incorrect/outdated pricing. Without accurate pricing, cost analytics are wrong for these models. + +--- + +## References + +| Source | Link | +| ---------------- | --------------------------------------------------------------------------------------------------------------------- | +| sub2api PR #970 | [feat(billing): add fallback pricing for MiniMax M2.5, GLM-4.7/5, Kimi](https://github.com/Wei-Shaw/sub2api/pull/970) | +| sub2api PR #1120 | [feat: upgrade MiniMax default model to M2.7](https://github.com/Wei-Shaw/sub2api/pull/1120) | +| sub2api commit | `fix: format gpt-5.4 mini fallback pricing` | + +--- + +## Pricing Data from sub2api + +| Model | Input $/MTok | Output $/MTok | Notes | +| ------------------------ | ------------ | ------------- | ----------------------------- | +| `MiniMax-M2.5` | $0.27 | $0.95 | via `api.minimax.io` | +| `MiniMax-M2.7` | TBD | TBD | New default, `api.minimax.io` | +| `MiniMax-M2.7-highspeed` | TBD | TBD | | +| `GLM-4.7` (Zhipu) | $0.38 | $1.98 | | +| `GLM-5` (Zhipu) | ~$0.38 | ~$1.98 | Check official page | +| `Kimi` (Moonshot) | TBD | TBD | Check moonshot.cn | +| `gpt-5.4` | TBD | TBD | | +| `gpt-5.4-mini` | TBD | TBD | Check openai.com/pricing | + +--- + +## Implementation Steps + +### Step 1 — Verify Current Pricing Table + +Check `open-sse/config/providerRegistry.ts` for existing entries: + +```bash +grep -n "minimax\|glm\|kimi\|gpt-5.4" open-sse/config/providerRegistry.ts +``` + +### Step 2 — Add Missing Models to Provider Registry + +```typescript +// In providerRegistry.ts, minimax section: +{ id: "minimax-m2.5", label: "MiniMax M2.5", inputPrice: 0.27, outputPrice: 0.95 }, +{ id: "minimax-m2.7", label: "MiniMax M2.7", inputPrice: TBD, outputPrice: TBD }, + +// GLM section: +{ id: "glm-4.7", label: "GLM-4.7", inputPrice: 0.38, outputPrice: 1.98 }, +{ id: "glm-5", label: "GLM-5", inputPrice: 0.38, outputPrice: 1.98 }, +``` + +### Step 3 — Verify Official Pricing Sources + +Before committing, verify against official sources: + +- MiniMax: https://platform.minimaxi.com/document/Price +- Zhipu GLM: https://open.bigmodel.cn/pricing +- Moonshot Kimi: https://platform.moonshot.cn/docs/pricing +- OpenAI gpt-5.4: https://openai.com/api/pricing + +### Step 4 — Update Default Model for MiniMax + +If OmniRoute has a "default model" for MiniMax, update to M2.7 (sub2api PR #1120). +New endpoint URL: `api.minimax.io` (update allowlist). + +--- + +## Files to Change + +| File | Action | +| ------------------------------------- | ------------------------------------------ | +| `open-sse/config/providerRegistry.ts` | Add/update model entries | +| `src/lib/db/settings.ts` | Default pricing table entries | +| `src/shared/constants/providers.ts` | Update MiniMax default model if applicable | + +--- + +## Acceptance Criteria + +- [ ] MiniMax M2.5 and M2.7 have pricing entries +- [ ] GLM-4.7 and GLM-5 have pricing entries +- [ ] gpt-5.4 / gpt-5.4-mini have pricing entries +- [ ] Kimi models have pricing entries +- [ ] Pricing verified against official sources before merging +- [ ] Cost analytics shows correct estimates for these models diff --git a/docs/new-features-sub21/tasks/T13-stale-quota-display.md b/docs/new-features-sub21/tasks/T13-stale-quota-display.md new file mode 100644 index 0000000000..5c0973138e --- /dev/null +++ b/docs/new-features-sub21/tasks/T13-stale-quota-display.md @@ -0,0 +1,90 @@ +# T13 · Quota Display Stale After Provider Reset + +**Priority:** 🟡 P3 — Display Accuracy +**Effort:** Small (1–2h) +**Phase:** 3 — v3.x + +--- + +## Problem + +The Providers dashboard may show cumulative token usage from the previous quota window even after the provider resets (e.g., Codex resets every 5h/7d, Claude has weekly resets). This causes the display to show "500K/1M tokens" even though the window just reset and the actual usage is 0. + +sub2api PR #1171 fixes this by comparing the `reset_at` timestamp with `now()` before rendering — if the window has expired, reset the display to 0%. + +--- + +## References + +| Source | Link | +| -------------- | ------------------------------------------------------------------------------------- | +| sub2api commit | `fix: quota display shows stale cumulative usage after daily/weekly reset` (PR #1171) | +| PR #357 | Also related — persist `reset_at` timestamps from `x-codex-*` headers | + +--- + +## Implementation Steps + +### Step 1 — Store `reset_at` with Usage Data + +When persisting quota data (from T03 — Codex headers), always store the corresponding `reset_at`: + +```typescript +interface QuotaSnapshot { + used: number; + limit: number; + resetAt: number | null; // epoch ms + updatedAt: number; // epoch ms — when this snapshot was taken +} +``` + +### Step 2 — Check Reset Before Display + +In the providers dashboard component: + +```typescript +function getEffectiveUsage(snapshot: QuotaSnapshot): number { + if (!snapshot.resetAt) return snapshot.used; + // If reset time has passed, display shows 0 (window reset) + if (Date.now() >= snapshot.resetAt) return 0; + return snapshot.used; +} + +// In the UI: +const displayUsage = getEffectiveUsage(codex5hSnapshot); +const displayPercent = snapshot.limit > 0 ? (displayUsage / snapshot.limit) * 100 : 0; +``` + +### Step 3 — Show "Reset pending" State + +If `resetAt` is in the past and we haven't yet received a new snapshot: + +- Show 0% usage with a "⟳ Refreshing..." indicator +- Trigger a background probe request to update the snapshot + +### Step 4 — Countdown Timer + +Display a countdown to the next reset for active windows: + +``` +[████░░░░░░] 42% — resets in 2h 35m +``` + +--- + +## Files to Change + +| File | Action | +| ------------------------------------------ | --------------------------------------------------------- | +| `src/app/(dashboard)/dashboard/providers/` | `getEffectiveUsage()` check before rendering | +| `src/lib/db/providers.ts` | Include `resetAt` in quota snapshot fields | +| `open-sse/executors/codex.ts` | Persist `reset_at` from response headers (T03 dependency) | + +--- + +## Acceptance Criteria + +- [ ] After a quota window resets, dashboard shows 0% immediately +- [ ] Countdown timer shows time until next reset +- [ ] "⟳ Refreshing..." shown when snapshot is stale post-reset +- [ ] No regression in providers that don't have reset timestamps (show static usage) diff --git a/docs/new-features-sub21/tasks/T14-proxy-fast-fail.md b/docs/new-features-sub21/tasks/T14-proxy-fast-fail.md new file mode 100644 index 0000000000..a0c531a15e --- /dev/null +++ b/docs/new-features-sub21/tasks/T14-proxy-fast-fail.md @@ -0,0 +1,130 @@ +# T14 · Proxy Fast-Fail on Dead Proxy + +**Priority:** 🟡 P3 — UX for Proxy Users +**Effort:** Small (1–2h) +**Phase:** 3 — v3.x + +--- + +## Problem + +When a configured HTTP/SOCKS5 proxy is unreachable, every request through OmniRoute waits for the full `PROXY_TIMEOUT_MS` (default 30s) before failing. In a combo with 3 providers, this means 90 seconds before any response — or 30s of hanging even before the combo starts. + +sub2api PR #1167 adds a "proxy fast-fail" that runs a quick TCP connectivity check before sending requests through the proxy. + +--- + +## References + +| Source | Link | +| ----------------------- | ------------------------------------------------------------------ | +| sub2api PR #1167 | `fix: proxy-fast-fail` | +| OmniRoute proxy timeout | `src/lib/apiBridgeServer.ts` — `DEFAULT_PROXY_TIMEOUT_MS = 30_000` | + +--- + +## Implementation Steps + +### Step 1 — Add Proxy Health Cache + +In `src/lib/proxyHealth.ts` (new file): + +```typescript +import { createConnection } from "node:net"; + +interface ProxyHealthEntry { + healthy: boolean; + checkedAt: number; + ttlMs: number; +} + +const proxyHealthCache = new Map(); + +/** + * Fast TCP check to see if a proxy is reachable. + * Caches results for `cacheTtlMs` to avoid checking on every request. + */ +export async function isProxyReachable( + proxyUrl: string, + timeoutMs = 2000, + cacheTtlMs = 30_000 +): Promise { + const cached = proxyHealthCache.get(proxyUrl); + if (cached && Date.now() - cached.checkedAt < cached.ttlMs) { + return cached.healthy; + } + + const url = new URL(proxyUrl); + const healthy = await new Promise((resolve) => { + const socket = createConnection( + { host: url.hostname, port: parseInt(url.port || "8080") }, + () => { + socket.destroy(); + resolve(true); + } + ); + socket.setTimeout(timeoutMs); + socket.on("error", () => resolve(false)); + socket.on("timeout", () => { + socket.destroy(); + resolve(false); + }); + }); + + proxyHealthCache.set(proxyUrl, { healthy, checkedAt: Date.now(), ttlMs: cacheTtlMs }); + return healthy; +} +``` + +### Step 2 — Gate Requests Through Health Check + +In `open-sse/chatCore.js`, before each request that uses a proxy: + +```js +const proxyUrl = getConfiguredProxy(provider); +if (proxyUrl) { + const proxyOk = await isProxyReachable(proxyUrl, 2000); + if (!proxyOk) { + logger.warn(`[Proxy Fast-Fail] Proxy ${proxyUrl} is unreachable. Skipping provider.`); + // Return error or skip to next combo member + throw new ProxyUnreachableError(`Proxy unreachable: ${proxyUrl}`); + } +} +``` + +### Step 3 — Show Proxy Status in Health Dashboard + +In `src/app/(dashboard)/dashboard/health/`: + +- Show proxy status: 🟢 Reachable / 🔴 Unreachable / ⚪ Not configured +- Trigger manual re-check button + +### Step 4 — Configurable Fast-Fail Timeout + +Via settings or env var: + +``` +PROXY_FAST_FAIL_TIMEOUT_MS=2000 # TCP check timeout +PROXY_HEALTH_CACHE_TTL_MS=30000 # Cache duration +``` + +--- + +## Files to Change + +| File | Action | +| --------------------------------------- | --------------------------------------------------- | +| `src/lib/proxyHealth.ts` | NEW — `isProxyReachable()` with cache | +| `open-sse/chatCore.js` | Gate Anthropic/OpenAI requests through health check | +| `src/app/(dashboard)/dashboard/health/` | Proxy status indicator | +| `src/lib/apiBridgeServer.ts` | Integrate fast-fail for API bridge requests | + +--- + +## Acceptance Criteria + +- [ ] Dead proxy detected in <2s instead of timing out at 30s +- [ ] Health check result cached for 30s (configurable) +- [ ] Clear log message when proxy fast-fail triggers +- [ ] Health dashboard shows proxy reachability +- [ ] Normal requests unaffected when proxy is healthy diff --git a/docs/new-features-sub21/tasks/T15-array-content-messages.md b/docs/new-features-sub21/tasks/T15-array-content-messages.md new file mode 100644 index 0000000000..53107bad19 --- /dev/null +++ b/docs/new-features-sub21/tasks/T15-array-content-messages.md @@ -0,0 +1,128 @@ +# T15 · Array Content in System/Tool Messages + +**Priority:** 🟡 P3 — SDK Compatibility +**Effort:** Tiny (<1h) +**Phase:** 3 — v3.x + +--- + +## Problem + +The OpenAI API spec allows `content` to be either a string OR an array of content blocks: + +```json +// String form (most clients): +{"role": "system", "content": "You are a helpful assistant."} + +// Array form (some SDKs, Cursor, Codex 2.x): +{"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant."}]} +``` + +Some clients (Cursor, certain Python SDK versions, Codex 2.x) send the array form for system and tool messages. If OmniRoute's translator always assumes string content, these requests fail with parse errors or incorrect translations. + +sub2api PR #1197: `fix(apicompat): support array content for system and tool messages`. + +--- + +## References + +| Source | Link | +| ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | +| sub2api PR #1197 | [fix(apicompat): support array content for system and tool messages](https://github.com/Wei-Shaw/sub2api/commit/4feacf221319b67e5239cfe71ebf654d1bbf0cc6) | +| OpenAI spec | Both string and array forms are valid | +| Affected clients | Cursor, some Python SDK versions, Codex 2.x | + +--- + +## Implementation Steps + +### Step 1 — Add `normalizeContent()` Helper + +In `open-sse/translator/openai-to-claude.ts` or a shared util: + +```typescript +/** + * Normalize content to a string. + * Handles both string and array-of-blocks forms. + */ +export function normalizeContentToString( + content: string | ContentBlock[] | null | undefined +): string { + if (!content) return ""; + if (typeof content === "string") return content; + // Array form: concatenate all text blocks + return content + .filter((b) => b.type === "text") + .map((b) => b.text ?? "") + .join("\n"); +} + +/** + * Normalize content to array form. + * Handles both string and array-of-blocks forms. + */ +export function normalizeContentToArray( + content: string | ContentBlock[] | null | undefined +): ContentBlock[] { + if (!content) return []; + if (typeof content === "string") { + return [{ type: "text", text: content }]; + } + return content; +} +``` + +### Step 2 — Apply in Message Translation + +In the translator, when processing system and tool messages: + +```typescript +// When building Claude request from OpenAI format: +const systemContent = normalizeContentToString(body.system ?? messages[0]?.content); + +// When forwarding tool messages: +const toolMessage = { + role: "tool", + content: normalizeContentToArray(msg.content), +}; +``` + +### Step 3 — Apply in Request Body Parsing + +In `open-sse/chatCore.js`, normalize before any downstream processing: + +```js +// Normalize system message if it's in array form +if (Array.isArray(body.system)) { + body.system = body.system + .filter((b) => b.type === "text") + .map((b) => b.text) + .join("\n"); +} + +// Normalize message content +body.messages = body.messages?.map((msg) => ({ + ...msg, + content: normalizeContent(msg.content), +})); +``` + +--- + +## Files to Change + +| File | Action | +| ----------------------------------------- | ------------------------------------------------------------ | +| `open-sse/translator/openai-to-claude.ts` | `normalizeContentToString()` and `normalizeContentToArray()` | +| `open-sse/chatCore.js` | Normalize system + message content at entry | +| `open-sse/translator/` | Apply normalization in all translation paths | + +--- + +## Acceptance Criteria + +- [ ] `content: "string"` continues to work (no regression) +- [ ] `content: [{"type":"text","text":"..."}]` is correctly handled +- [ ] Array content in system messages is normalized to string for Claude +- [ ] Array content in tool messages is preserved as array for Anthropic +- [ ] Mixed content arrays (text + image) handled correctly