From 6424c825701acb439a69aa82a06b434a8e4ee438 Mon Sep 17 00:00:00 2001 From: diegosouzapw Date: Sun, 3 May 2026 15:59:28 -0300 Subject: [PATCH] docs(compression): document RTK+Caveman stacked savings ranges Refresh the compression docs and README to describe the default stacked pipeline in terms of eligible-context savings instead of the older generic token-saving range. Add upstream RTK and Caveman benchmark references, explain the multiplicative savings math behind the stacked default, and update feature summaries plus package metadata to match the revised positioning. --- README.md | 58 +++++++++++++++++++++++++++---------- docs/COMPRESSION_ENGINES.md | 18 +++++++++++- docs/COMPRESSION_GUIDE.md | 48 ++++++++++++++++++++++++------ docs/FEATURES.md | 2 +- docs/RTK_COMPRESSION.md | 13 ++++++++- package.json | 2 +- 6 files changed, 113 insertions(+), 28 deletions(-) diff --git a/README.md b/README.md index 90af73193d..d794d02d81 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ # πŸš€ OmniRoute β€” The Free AI Gateway -### Never stop coding. Save 15-75% tokens with prompt compression + auto-fallback to **FREE & low-cost AI models**. +### Never stop coding. Save 15-95% eligible tokens with RTK+Caveman compression + auto-fallback to **FREE & low-cost AI models**. _The most complete open-source AI proxy β€” **one endpoint**, **160+ providers**, **13 routing strategies**, zero downtime. Multi-platform: **Web**, **Desktop (Electron)**, **Mobile (PWA + Termux)**. Fully extensible via **MCP Server (37 tools)**, **A2A Protocol**, and **Memory/Skills** systems. Available in **40+ languages**._ @@ -20,7 +20,7 @@ _The most complete open-source AI proxy β€” **one endpoint**, **160+ providers** diegosouzapw%2FOmniRoute | Trendshift -[πŸš€ Quick Start](#-quick-start) β€’ [πŸ’‘ Features](#-key-features) β€’ [πŸ—œοΈ Compression](#%EF%B8%8F-prompt-compression--save-15-75-tokens-automatically) β€’ [πŸ’° Pricing](#-pricing-at-a-glance) β€’ [🎯 Use Cases](#-use-cases--ready-made-combo-playbooks) β€’ [🌍 Proxy](#-bypass-geographic-blocks--use-ai-from-any-country) β€’ [❓ FAQ](#-frequently-asked-questions) β€’ [πŸ“– Docs](#-documentation) β€’ [πŸ’¬ WhatsApp](https://chat.whatsapp.com/JI7cDQ1GyaiDHhVBpLxf8b?mode=gi_t) +[πŸš€ Quick Start](#-quick-start) β€’ [πŸ’‘ Features](#-key-features) β€’ [πŸ—œοΈ Compression](#%EF%B8%8F-prompt-compression--save-15-95-eligible-tokens-automatically) β€’ [πŸ’° Pricing](#-pricing-at-a-glance) β€’ [🎯 Use Cases](#-use-cases--ready-made-combo-playbooks) β€’ [🌍 Proxy](#-bypass-geographic-blocks--use-ai-from-any-country) β€’ [❓ FAQ](#-frequently-asked-questions) β€’ [πŸ“– Docs](#-documentation) β€’ [πŸ’¬ WhatsApp](https://chat.whatsapp.com/JI7cDQ1GyaiDHhVBpLxf8b?mode=gi_t) @@ -219,7 +219,7 @@ _Connect any AI-powered IDE or CLI tool through OmniRoute β€” free API gateway f **OmniRoute solves all of this:** -βœ… **Prompt Compression** β€” auto-compress prompts & tool outputs, save 15-75% tokens per request +βœ… **Prompt Compression** β€” auto-compress prompts & tool outputs, save 15-95% eligible tokens per request with RTK+Caveman stacked mode βœ… **Maximize subscriptions** β€” track quota, use every bit before reset βœ… **Auto fallback** β€” Subscription β†’ API Key β†’ Cheap β†’ Free, zero downtime βœ… **Multi-account** β€” round-robin between accounts per provider @@ -410,7 +410,7 @@ Alibaba Β· Amazon Q Β· AssemblyAI Β· Baidu Qianfan Β· Baseten Β· Black Forest La ↓ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ OmniRoute (Smart Router) β”‚ -β”‚ β€’ πŸ—œοΈ Prompt Compression (save 15-75% tokens) β”‚ +β”‚ β€’ πŸ—œοΈ Prompt Compression (save 15-95% eligible) β”‚ β”‚ β€’ Format translation (OpenAI ↔ Claude ↔ Gemini) β”‚ β”‚ β€’ Quota tracking + Embeddings + Images β”‚ β”‚ β€’ Auto token refresh + Rate limit management β”‚ @@ -424,14 +424,14 @@ Alibaba Β· Amazon Q Β· AssemblyAI Β· Baidu Qianfan Β· Baseten Β· Black Forest La β”‚ ↓ budget limit └─→ [Tier 4: FREE] Qoder, Qwen, Kiro (unlimited) -Result: Never stop coding, minimal cost + 15-75% token savings +Result: Never stop coding, minimal cost + 15-95% eligible token savings ``` --- -## πŸ—œοΈ Prompt Compression β€” Save 15-75% Tokens Automatically +## πŸ—œοΈ Prompt Compression β€” Save 15-95% Eligible Tokens Automatically -> **Why use many token when few token do trick?** OmniRoute's built-in compression pipeline reduces token usage on every request β€” before it even reaches the provider. Inspired by [Caveman](https://github.com/JuliusBrussee/caveman) (⭐ 51K+). +> **Why use many token when few token do trick?** OmniRoute's built-in compression pipeline reduces token usage before requests reach the provider. It combines ideas from [RTK - Rust Token Killer](https://github.com/rtk-ai/rtk) and [Caveman](https://github.com/JuliusBrussee/caveman) (⭐ 51K+). ### How It Works @@ -442,12 +442,12 @@ Every request passes through the compression pipeline **transparently** β€” no c β”‚ Client sends │────▢│ OmniRoute Compression │────▢│ Provider β”‚ β”‚ full prompt β”‚ β”‚ Pipeline (7 options) β”‚ β”‚ receives β”‚ β”‚ (10,000 tok) β”‚ β”‚ β”‚ β”‚ compressed β”‚ -β”‚ β”‚ β”‚ πŸͺΆ Lite ........... ~15% β”‚ β”‚ (2,500 tok) β”‚ +β”‚ β”‚ β”‚ πŸͺΆ Lite ........... ~15% β”‚ β”‚ (~1,080 tok)β”‚ β”‚ β”‚ β”‚ πŸͺ¨ Standard ....... ~30% β”‚ β”‚ β”‚ -β”‚ β”‚ β”‚ ⚑ Aggressive ..... ~50% β”‚ β”‚ πŸ’° 75% saved β”‚ +β”‚ β”‚ β”‚ ⚑ Aggressive ..... ~50% β”‚ β”‚ πŸ’° up to 95%β”‚ β”‚ β”‚ β”‚ πŸ”₯ Ultra .......... ~75% β”‚ β”‚ β”‚ -β”‚ β”‚ β”‚ 🧰 RTK ............ 20-70% β”‚ β”‚ β”‚ -β”‚ β”‚ β”‚ πŸ”— Stacked ........ 30-80% β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ 🧰 RTK ............ 60-90% β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ πŸ”— Stacked ........ 78-95% β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` @@ -460,8 +460,36 @@ Every request passes through the compression pipeline **transparently** β€” no c | **πŸͺ¨ Standard (Caveman)** | ~30% | 30+ regex rules: filler removal, context condensation, structural compression, multi-turn dedup | Daily coding with Claude/Codex | | **⚑ Aggressive** | ~50% | All standard + progressive message aging + tool result summarization + LLM-based compression | Long sessions with many tool calls | | **πŸ”₯ Ultra** | ~75% | All aggressive + heuristic token pruning + stopword removal + score-based filtering | Maximum savings when tokens are scarce | -| **🧰 RTK** | 20-70% | 49 command-aware filters, RTK-style JSON DSL, verify gate, trust-gated custom filters | Shell/test/build/git output in agents | -| **πŸ”— Stacked** | 30-80% | Ordered engine pipeline, usually RTK first then Caveman | Mixed prompts with tool logs + prose | +| **🧰 RTK** | 60-90% | 49 command-aware filters, RTK-style JSON DSL, verify gate, trust-gated custom filters | Shell/test/build/git output in agents | +| **πŸ”— Stacked** | 78-95% | RTK first, then Caveman input condensation; ~89% with upstream average math | Mixed prompts with tool logs + prose | + +### RTK + Caveman Savings Math + +These numbers are based on the upstream project READMEs under `_references/_outros`: + +| Source | Upstream claim used by OmniRoute docs | +| ------- | ------------------------------------------------------------------------------------------------------------------- | +| Caveman | `~75%` fewer output tokens; benchmark average `65%` output savings, range `22-87%`; `~46%` input compression tool | +| RTK | `60-90%` command-output token savings; sample session `~118,000 -> ~23,900` tokens, which is `79.7%` saved (`~80%`) | + +For the default stacked compression combo, OmniRoute runs: + +```txt +RTK -> Caveman +``` + +When both engines can act on the same tool/context payload, the savings compound: + +```txt +combined = 1 - (1 - RTK savings) * (1 - Caveman input savings) +average = 1 - (1 - 0.80) * (1 - 0.46) = 89.2% +range = 1 - (1 - 0.60..0.90) * (1 - 0.46) = 78.4-94.6% +``` + +Caveman output mode is separate from prompt compression. When enabled for responses, use Caveman's +own upstream output numbers: `65%` average, `~75%` headline, `22-87%` observed range. Total bill +savings depend on the prompt/output mix, but coding-agent sessions are often tool-context heavy, so +the `RTK -> Caveman` combo is the best default for maximum context savings. ### Before & After (Standard/Caveman Mode) @@ -516,7 +544,7 @@ Auto-trigger: set `autoTriggerTokens` to automatically enable compression when a Compression combos can also assign a named compression pipeline to routing combos, so a coding combo can use RTK + Caveman while a paid subscription combo stays on lite mode. -> πŸͺ¨ **Fun fact:** The standard/caveman mode is inspired by [Caveman](https://github.com/JuliusBrussee/caveman) β€” the viral project that proved "caveman speak" cuts 65% of tokens while keeping 100% technical accuracy. OmniRoute takes this further with a **7-option pipeline** that goes from gentle whitespace cleanup through RTK tool-output filters and stacked multi-engine compression. +> πŸͺ¨ **Fun fact:** The standard/caveman mode is inspired by [Caveman](https://github.com/JuliusBrussee/caveman) β€” the viral project that reports 65% average output-token savings while keeping technical accuracy. OmniRoute takes this further with a **7-option pipeline** and a default `RTK -> Caveman` combo that can reach ~89% average savings on eligible tool/context payloads. πŸ“– **Full compression documentation:** [`docs/COMPRESSION_GUIDE.md`](docs/COMPRESSION_GUIDE.md) β€’ [`docs/RTK_COMPRESSION.md`](docs/RTK_COMPRESSION.md) β€’ [`docs/COMPRESSION_ENGINES.md`](docs/COMPRESSION_ENGINES.md) β€’ [`docs/COMPRESSION_RULES_FORMAT.md`](docs/COMPRESSION_RULES_FORMAT.md) β€’ [`docs/COMPRESSION_LANGUAGE_PACKS.md`](docs/COMPRESSION_LANGUAGE_PACKS.md) @@ -941,7 +969,7 @@ Then in `/dashboard/media` β†’ **Transcription** tab: upload any audio or video | ---------------------------------------------------------------------------------------------------- | -------------------------------- | | 🧠 **Smart 4-Tier Fallback** β€” Subscription β†’ API β†’ Cheap β†’ Free | Never stop coding, zero downtime | | πŸ”„ **Format Translation** β€” OpenAI ↔ Claude ↔ Gemini ↔ Responses API | Works with ANY CLI tool | -| πŸ—œοΈ **Prompt Compression** β€” 7 options including Caveman, RTK, and stacked pipelines | Save 15-75% tokens automatically | +| πŸ—œοΈ **Prompt Compression** β€” 7 options including Caveman, RTK, and stacked pipelines | Save 15-95% eligible tokens | | πŸ€– **MCP Server** β€” 37 tools, 3 transports (stdio/SSE/HTTP), 10 scopes | IDE/agent tool integration | | πŸ›‘οΈ **Resilience Engine** β€” circuit breakers, cooldowns, TLS spoofing, anti-thundering herd | Auto-recovery from any failure | | 🎡 **10 Multi-Modal APIs** β€” chat, embed, images, video, music, TTS, STT, moderation, rerank, search | One endpoint for everything | diff --git a/docs/COMPRESSION_ENGINES.md b/docs/COMPRESSION_ENGINES.md index da5c934cfb..81dc01c1ba 100644 --- a/docs/COMPRESSION_ENGINES.md +++ b/docs/COMPRESSION_ENGINES.md @@ -13,7 +13,7 @@ OmniRoute compression is built around engine contracts. A mode can run one engin | `aggressive` | Caveman + history/tool summarizers | Long chat sessions | | `ultra` | Caveman + pruning helpers | Context-limit recovery | | `rtk` | RTK | Terminal, shell, build, test, and git output | -| `stacked` | Pipeline, default `rtk -> caveman` | Mixed tool logs and prose | +| `stacked` | Pipeline, default `rtk -> caveman` | Mixed tool logs and prose, max savings | ## Engine Registry @@ -39,6 +39,11 @@ Caveman mode focuses on semantic condensation of normal prose: The dashboard surface is `Dashboard -> Context & Cache -> Caveman`. +Caveman upstream reports `~75%` fewer output tokens, `65%` average output savings in benchmarks +with a `22-87%` range, and a `~46%` input-compression tool. OmniRoute uses the Caveman input-side +number when documenting stacked prompt/context savings; Caveman output mode remains a separate +response-behavior feature. + ## RTK RTK mode focuses on command and tool output: @@ -61,6 +66,9 @@ The dashboard surface is `Dashboard -> Context & Cache -> RTK`. Operational details for custom filters, trust, verify, and raw-output recovery live in [`RTK_COMPRESSION.md`](RTK_COMPRESSION.md). +RTK upstream reports `60-90%` savings for command-output compression. Its README example shows a +30-minute Claude Code session going from `~118,000` tokens to `~23,900`, or `79.7%` saved. + ## Stacked Pipelines Stacked mode runs pipeline steps in order. The default is: @@ -75,6 +83,14 @@ prose. RTK reduces noisy tool logs first, then Caveman compresses remaining natu Pipeline steps are configured with `stackedPipeline` in compression settings or through compression combos. +When both engines reduce the same eligible payload, savings compound: + +```txt +combined = 1 - (1 - RTK savings) * (1 - Caveman input savings) +average = 1 - (1 - 0.80) * (1 - 0.46) = 89.2% +range = 1 - (1 - 0.60..0.90) * (1 - 0.46) = 78.4-94.6% +``` + ## Compression Combos Compression combos are named compression profiles that can be assigned to routing combos: diff --git a/docs/COMPRESSION_GUIDE.md b/docs/COMPRESSION_GUIDE.md index 9cfe6c8a15..88213bfc63 100644 --- a/docs/COMPRESSION_GUIDE.md +++ b/docs/COMPRESSION_GUIDE.md @@ -1,6 +1,6 @@ # πŸ—œοΈ Prompt Compression Guide β€” OmniRoute -> Save 15-75% on token costs automatically. For a quick overview, see the [README Compression section](../README.md#%EF%B8%8F-prompt-compression--save-15-75-tokens-automatically). +> Save 15-95% on eligible context automatically. For a quick overview, see the [README Compression section](../README.md#%EF%B8%8F-prompt-compression--save-15-95-eligible-tokens-automatically). ## Overview @@ -19,8 +19,8 @@ Client Request β†’ Standard: Caveman-speak filler removal (~30%) β†’ Aggressive: History aging + summarization (~50%) β†’ Ultra: Heuristic pruning + code-block thinning (~75%) - β†’ RTK: Command-aware terminal/tool-output filtering (20-70%) - β†’ Stacked: Ordered multi-engine pipeline, usually RTK then Caveman (30-80%) + β†’ RTK: Command-aware terminal/tool-output filtering (60-90% upstream range) + β†’ Stacked: Ordered multi-engine pipeline, usually RTK then Caveman (78-95% eligible range) β†’ Compressed Request β†’ Provider ``` @@ -79,7 +79,7 @@ Maximum compression for token-critical scenarios: **Best for:** When you're hitting context limits repeatedly. -### RTK Mode (20-70% savings) +### RTK Mode (60-90% upstream range) RTK mode is optimized for verbose tool outputs that appear in coding-agent sessions: @@ -94,7 +94,7 @@ RTK mode is optimized for verbose tool outputs that appear in coding-agent sessi **Best for:** Agent sessions with shell, build, test, git, grep, and file-output transcripts. -### Stacked Mode (30-80% savings) +### Stacked Mode (78-95% eligible range) Stacked mode runs multiple compression engines in a deterministic order. The default pipeline is: @@ -110,6 +110,36 @@ compression combos assigned to routing combos. --- +## Upstream Savings Math + +OmniRoute documents compression savings from two sources: upstream project benchmarks and +OmniRoute's own engine composition. + +| Source | Upstream README number used here | +| ------- | --------------------------------------------------------------------------------------------------------------------- | +| Caveman | `~75%` fewer output tokens, `65%` benchmark average output savings, `22-87%` range, and `~46%` input compression tool | +| RTK | `60-90%` command-output savings; sample session `~118,000 -> ~23,900` tokens, or `79.7%` saved (`~80%`) | + +For overlapping tool/context payloads, the default OmniRoute combo stacks the engines: + +```txt +RTK -> Caveman +``` + +The combined savings are multiplicative, not additive: + +```txt +combined = 1 - (1 - RTK savings) * (1 - Caveman input savings) +average = 1 - (1 - 0.80) * (1 - 0.46) = 89.2% +range = 1 - (1 - 0.60..0.90) * (1 - 0.46) = 78.4-94.6% +``` + +That `78-95%` number applies when both RTK and Caveman can reduce the same input/context payload. +Caveman response output mode is separate: when enabled, use Caveman's own output savings (`65%` +average, `~75%` headline, `22-87%` range). Total billing savings depend on your prompt/output mix. + +--- + ## Token Savings Visualization ``` @@ -118,8 +148,8 @@ With Lite: 40K tokens sent (15% saved β€” safe, always-on) With Standard: 33K tokens sent (30% saved β€” caveman-speak rules) With Aggressive: 24K tokens sent (50% saved β€” aging + summarization) With Ultra: 12K tokens sent (75% saved β€” heuristic pruning) -With RTK: varies by output (20-70% saved β€” command-aware filters) -With Stacked: varies by pipeline (30-80% saved β€” multi-engine execution) +With RTK: 19K-5K tokens sent (60-90% saved on command/tool output) +With Stacked: 10K-2.5K tokens sent (78-95% eligible RTK+Caveman range) ``` --- @@ -229,9 +259,9 @@ Every compressed request includes stats in the server logs: ## Acknowledgments -Standard mode compression rules are inspired by **[Caveman](https://github.com/JuliusBrussee/caveman)** by **[JuliusBrussee](https://github.com/JuliusBrussee)** (⭐ 51K+) β€” the viral "why use many token when few token do trick" project. +Standard mode compression rules are inspired by **[Caveman](https://github.com/JuliusBrussee/caveman)** by **[JuliusBrussee](https://github.com/JuliusBrussee)** (⭐ 51K+) β€” the viral "why use many token when few token do trick" project. Caveman reports `~75%` fewer output tokens, `65%` benchmark average output savings, a `22-87%` output range, and a `~46%` input-compression tool. -RTK mode is inspired by **[RTK - Rust Token Killer](https://github.com/rtk-ai/rtk)** by **[RTK AI](https://github.com/rtk-ai)** β€” the high-performance command-output compression project for terminal, build, test, git, and tool-output filtering. +RTK mode is inspired by **[RTK - Rust Token Killer](https://github.com/rtk-ai/rtk)** by **[RTK AI](https://github.com/rtk-ai)** β€” the high-performance command-output compression project for terminal, build, test, git, and tool-output filtering. RTK reports `60-90%` savings, with its README sample session showing `~80%` saved. --- diff --git a/docs/FEATURES.md b/docs/FEATURES.md index ff86c872f1..4c8e9e455f 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -119,7 +119,7 @@ Context & Cache now exposes dedicated pages for Caveman, RTK, and Compression Co - **Caveman** β€” language-aware rule packs, preview, output-mode controls, and analytics - **RTK** β€” command-aware compression for shell, git, test, build, package, Docker, infra, JSON, and stack-trace output -- **Compression Combos** β€” named pipelines such as `rtk -> caveman` assigned to routing combos +- **Compression Combos** β€” named pipelines such as `rtk -> caveman` assigned to routing combos; the default stacked math reaches `~89%` average and `78-95%` eligible-context savings when both engines apply - **Raw-output recovery** β€” optional redacted RTK raw-output pointers for debugging compressed failures See [Compression Guide](COMPRESSION_GUIDE.md), [RTK Compression](RTK_COMPRESSION.md), and diff --git a/docs/RTK_COMPRESSION.md b/docs/RTK_COMPRESSION.md index 9f24139df3..f675c93d25 100644 --- a/docs/RTK_COMPRESSION.md +++ b/docs/RTK_COMPRESSION.md @@ -12,9 +12,20 @@ rtk -> caveman That order compresses noisy machine output first, then lets Caveman condense remaining prose. +Upstream RTK reports `60-90%` command-output savings. Its README sample session goes from +`~118,000` standard tokens to `~23,900` RTK tokens, which is `79.7%` saved (`~80%`). OmniRoute uses +that upstream average for the stacked savings calculation with Caveman input compression: + +```txt +RTK average: 80% saved +Caveman input: 46% saved +Stacked: 1 - (1 - 0.80) * (1 - 0.46) = 89.2% saved +Range: 1 - (1 - 0.60..0.90) * (1 - 0.46) = 78.4-94.6% +``` + ## What It Compresses -The built-in catalog currently ships 39 filters across these categories: +The built-in catalog currently ships 49 filters across these categories: | Category | Examples | | --------- | ------------------------------------------------------------- | diff --git a/package.json b/package.json index 24568eb1fb..12d4fc61f0 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,7 @@ { "name": "omniroute", "version": "3.7.9", - "description": "Smart AI Router with auto fallback β€” route to FREE & cheap models, zero downtime. Works with Cursor, Cline, Claude Desktop, Codex, and any OpenAI-compatible tool.", + "description": "Unified AI router with 160+ providers, RTK+Caveman compression, auto fallback, MCP/A2A, desktop, PWA, and OpenAI-compatible APIs.", "type": "module", "bin": { "omniroute": "bin/omniroute.mjs",