mirror of https://github.com/diegosouzapw/OmniRoute.git synced 2026-07-26 09:52:11 +03:00

Files

Diego Rodrigues de Sa e Souza 743be29852 feat(compression): RTK compression roadmap (#1889 )

* chore(rtk): initialize compression roadmap branch

* feat(compression): add RTK engine and compression combos

Introduce RTK command-aware tool-output compression alongside
stacked RTK -> Caveman pipelines for mixed prompt contexts.

Add engine registration, declarative RTK filter packs, language-aware
Caveman rule loading, compression combo persistence and assignments,
analytics grouped by engine/combo, and new MCP/API endpoints for
configuration, previews, filters, and combo management.

Expose the new capabilities in the dashboard with dedicated Context &
Cache pages for Caveman, RTK, and compression combos, and update docs,
i18n strings, migrations, and tests to cover the expanded compression
surface.

* feat(compression): expand RTK DSL, filter catalog, and recovery APIs

Add RTK parity features across the compression pipeline, dashboard,
and management APIs. This expands the built-in filter catalog, adds
trust-gated custom filter loading, inline filter verification, code
stripping, smarter detection, and optional redacted raw-output
retention for authenticated recovery.

Also extend Caveman with file-based multilingual rule packs, localized
output-mode instructions, stricter preview/config schemas, engine
registry metadata, analytics fields, and broad unit test coverage for
RTK, rule loading, and stacked compression behavior.

* fix(auth): protect oauth routes and health reset operations

Require authenticated dashboard access for OAuth endpoints that can
create or import provider connections when login enforcement is
enabled.

Move `/api/monitoring/health` to the readonly public route list so
safe methods remain public while DELETE now returns 401 for anonymous
requests.

Also update Next.js native `.node` handling to avoid webpack parse
failures from external packages such as ngrok and keytar, and add
coverage for the new auth behavior.

* build(compression): ship RTK rule and filter assets with app bundles

Include compression JSON assets in Next output tracing, prepublish copies,
and pack artifact policy checks so standalone and packaged builds can
load RTK filters and caveman rule packs at runtime.

Also harden compression runtime behavior by resolving alternate asset
directories, scoping rule cache entries by source path, carrying RTK raw
output pointers through stacked runs, degrading oversized preview diffs,
and applying combo language/output mode defaults during chat routing.

Add coverage for packaging rules, provider-scoped model parsing, smart
truncate edge cases, raw output retention, and combo-driven compression
behavior.

* docs(workflows): update local repo paths to OmniRoute

Replace outdated `/home/diegosouzapw/dev/proxys/9router` references
with the current `OmniRoute` directory across deploy, release, and
version bump workflow guides so local command examples match the
renamed repository layout

* feat(compression): complete RTK parity coverage

* test(build): align next config assertions

---------

Co-authored-by: diegosouzapw <diego.souza.pw@gmail.com>

2026-05-03 00:37:08 -03:00

6.5 KiB

Raw Blame History

Compression Engines

OmniRoute compression is built around engine contracts. A mode can run one engine directly (caveman or rtk) or a deterministic stacked pipeline that executes multiple engines in order.

Modes

Mode	Engine path	Intended input
`off`	none	Exact prompt preservation
`lite`	Caveman lite helpers	Low-risk always-on cleanup
`standard`	Caveman	Natural-language prompt condensation
`aggressive`	Caveman + history/tool summarizers	Long chat sessions
`ultra`	Caveman + pruning helpers	Context-limit recovery
`rtk`	RTK	Terminal, shell, build, test, and git output
`stacked`	Pipeline, default `rtk -> caveman`	Mixed tool logs and prose

Engine Registry

The registry lives in open-sse/services/compression/engines/registry.ts. Engines expose a shared contract:

id: stable engine id such as caveman or rtk
label: dashboard-readable name
supports(mode): whether the engine can execute a compression mode
compress(input): transforms text/messages and returns stats

strategySelector.ts registers the built-in engines before compression runs. This lets preview, runtime compression, stacked mode, tests, and future engines use the same execution path.

Caveman

Caveman mode focuses on semantic condensation of normal prose:

preserves code blocks, URLs, JSON, paths, and structured data
removes filler, hedging, repeated context, and verbose connective phrasing
supports language-aware file rule packs in open-sse/services/compression/rules/
remains available through the legacy standard, aggressive, and ultra modes

The dashboard surface is Dashboard -> Context & Cache -> Caveman.

RTK

RTK mode focuses on command and tool output:

detects output classes such as git status, git branch, git diff, Vitest/Jest/Pytest, Cargo/Go tests, TypeScript/Vite/Webpack builds, ESLint, npm audit/installs, Docker logs, shell find/grep, stack traces, and generic logs
applies 39 JSON filters from open-sse/services/compression/engines/rtk/filters/
supports the RTK-style declarative pipeline: ANSI stripping, replace, match-output short-circuit, strip/keep lines, per-line truncation, head/tail/max-line truncation, and on-empty fallback
supports trust-gated project filters in .rtk/filters.json and global filters in DATA_DIR/rtk/filters.json
strips ANSI sequences, progress noise, repeated lines, and unhelpful boilerplate
preserves actionable failures, warnings, summaries, changed files, and tail context
can optionally retain redacted raw output for recovery/debugging through authenticated management routes

The dashboard surface is Dashboard -> Context & Cache -> RTK.

Operational details for custom filters, trust, verify, and raw-output recovery live in rtk-compression.md.

Stacked Pipelines

Stacked mode runs pipeline steps in order. The default is:

rtk -> caveman

Use this for coding-agent sessions where a prompt combines command output with human or assistant prose. RTK reduces noisy tool logs first, then Caveman compresses remaining natural language.

Pipeline steps are configured with stackedPipeline in compression settings or through compression combos.

Compression Combos

Compression combos are named compression profiles that can be assigned to routing combos:

compression_combos: stores mode, pipeline, RTK config, language config, and default marker
compression_combo_assignments: maps a compression combo to a routing combo
runtime integration resolves an assigned compression combo before generic combo overrides
analytics include compression_combo_id and engine

Dashboard surface: Dashboard -> Context & Cache -> Compression Combos.

API Surface

Route	Purpose
`/api/settings/compression`	Global compression settings
`/api/compression/preview`	Preview any compression mode
`/api/compression/language-packs`	List available Caveman language packs
`/api/context/caveman/config`	Caveman settings alias
`/api/context/rtk/config`	RTK defaults and settings
`/api/context/rtk/filters`	RTK filter catalog
`/api/context/rtk/test`	RTK preview/test endpoint
`/api/context/rtk/raw-output/[id]`	Authenticated redacted raw-output recovery
`/api/context/combos`	Compression combo CRUD
`/api/context/combos/[id]/assignments`	Routing-combo assignment CRUD
`/api/context/analytics`	Compression analytics alias

Management routes require management authentication or API-key policy checks.

MCP Tools

Compression exposes five MCP tools:

Tool	Scope	Purpose
`omniroute_compression_status`	`read:compression`	Settings, analytics, cache stats
`omniroute_compression_configure`	`write:compression`	Update global settings
`omniroute_set_compression_engine`	`write:compression`	Set mode and optional pipeline
`omniroute_list_compression_combos`	`read:compression`	List compression combos
`omniroute_compression_combo_stats`	`read:compression`	Read combo/engine analytics

Validation

The focused gates for this area are:

node --import tsx/esm --test tests/unit/compression/rtk-*.test.ts tests/unit/compression/pipeline-integration.test.ts tests/unit/compression/context-compression-api.test.ts
node --import tsx/esm --test tests/unit/compression/*.test.ts tests/golden-set/*.test.ts tests/integration/compression-pipeline.test.ts tests/unit/api/compression/compression-api.test.ts
npm run typecheck:core

6.5 KiB Raw Blame History