mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-08-06 15:22:12 +03:00
f7fbe3946d
The hardcoded BUFFER_TOKENS = 2000 constant inflates prompt_tokens and
input_tokens in every API response, which is helpful for CLI tools that
rely on reported usage to manage context windows but misleading for SDK
users, cost dashboards, and any integration comparing token counts
across providers.
This change makes the buffer configurable via three layered sources
(in priority order):
1. Environment variable: USAGE_TOKEN_BUFFER=0
2. Settings API / Dashboard: PATCH /api/settings { usageTokenBuffer: 0 }
3. Default: 2000 (preserves existing behavior)
Setting the value to 0 disables the buffer entirely, causing OmniRoute
to return raw provider token counts. The setting is cached in-process
with a 30-second TTL and invalidated immediately when updated through
the settings API.
Changes:
- open-sse/utils/usageTracking.ts: replace hardcoded constant with
getBufferTokens() that reads env / DB settings with TTL cache
- src/shared/validation/settingsSchemas.ts: add usageTokenBuffer field
(int, 0–50000) to the Zod update schema
- src/app/api/settings/route.ts: invalidate buffer cache on update