OmniRoute/docs/RTK_COMPRESSION.md

RTK Compression

RTK compression is OmniRoute's command-aware compression engine for terminal and tool output. It is designed for coding-agent sessions where most context growth comes from test logs, build output, package manager noise, shell transcripts, Docker output, git output, and stack traces.

RTK can run directly with defaultMode: "rtk" or as the first step in a stacked pipeline, usually:

rtk -> caveman

That order compresses noisy machine output first, then lets Caveman condense remaining prose.

Upstream RTK reports 60-90% command-output savings. Its README sample session goes from ~118,000 standard tokens to ~23,900 RTK tokens, which is 79.7% saved (~80%). OmniRoute uses that upstream average for the stacked savings calculation with Caveman input compression:

RTK average:    80% saved
Caveman input: 46% saved
Stacked:       1 - (1 - 0.80) * (1 - 0.46) = 89.2% saved
Range:         1 - (1 - 0.60..0.90) * (1 - 0.46) = 78.4-94.6%

What It Compresses

The built-in catalog currently ships 49 filters across these categories:

Category	Examples
git	git status, git branch, git diff, git log
test	Vitest, Jest, Pytest, Playwright, Go tests, Cargo tests
build	TypeScript, ESLint, Biome, Prettier, Vite, Webpack, Turbo, Nx
package	npm install, npm audit, pip, uv sync, Poetry, Bundler
shell	ls, find, grep, generic shell logs
docker	docker ps, Docker logs
infra	Terraform, OpenTofu, systemctl status
generic	JSON output, stack traces, generic output fallback

The detector in open-sse/services/compression/engines/rtk/commandDetector.ts classifies output before filter selection. Filters can also match by command pattern or output regex when a command class is not enough.

Filter Resolution

RTK loads filters in this order:

1. Project filters from .rtk/filters.json, only when trusted.
2. Global filters from DATA_DIR/rtk/filters.json.
3. Built-in filters from open-sse/services/compression/engines/rtk/filters/.

Project filters are intentionally trust-gated because regex filters can change how tool output is shown to agents. A project filter file is accepted when one of these is true:

• rtkConfig.trustProjectFilters is true.
• OMNIROUTE_RTK_TRUST_PROJECT_FILTERS=1 is set.
• .rtk/trust.json contains the SHA-256 hash of .rtk/filters.json.

Trust file example:

{
  "filtersSha256": "0123456789abcdef..."
}

Custom filters can be one filter object or an array of filter objects. Invalid custom filters are skipped and reported by /api/context/rtk/filters diagnostics. Invalid built-in filters fail fast.

Filter DSL

Filters use the JSON schema described in Compression Rules Format. The runtime applies these stages in order:

stripAnsi -> filterStderr -> replace -> matchOutput -> drop/include lines
  -> truncateLineAt -> head/tail/maxLines -> onEmpty

Important fields:

Field	Purpose
rules.stripAnsi	Remove terminal color/control sequences before matching
rules.filterStderr	Normalize common stderr prefixes before matching/filtering
rules.replace	Apply ordered regex replacements
rules.matchOutput	Return a compact summary when output matches a known condition
rules.matchOutput[].unless	Skip the shortcut when an error/failure pattern is present
rules.dropPatterns	Remove noisy lines
rules.includePatterns	Prefer actionable lines
rules.collapsePatterns	Collapse repeated matching lines
rules.truncateLineAt	Unicode-safe per-line truncation
rules.onEmpty	Fallback message if all lines are filtered out
tests[]	Inline samples used by the verify gate

Built-in filters are expected to include inline tests[] samples. Custom filters should include them too, especially when they are shared across projects.

Configuration

Global settings are available through /api/settings/compression. RTK-specific settings are also available through /api/context/rtk/config.

{
  "defaultMode": "stacked",
  "autoTriggerMode": "stacked",
  "autoTriggerTokens": 32000,
  "stackedPipeline": [
    { "engine": "rtk", "intensity": "standard" },
    { "engine": "caveman", "intensity": "full" }
  ],
  "rtkConfig": {
    "enabled": true,
    "intensity": "standard",
    "applyToToolResults": true,
    "applyToCodeBlocks": false,
    "applyToAssistantMessages": false,
    "enabledFilters": [],
    "disabledFilters": [],
    "maxLinesPerResult": 120,
    "maxCharsPerResult": 12000,
    "deduplicateThreshold": 3,
    "customFiltersEnabled": true,
    "trustProjectFilters": false,
    "rawOutputRetention": "never",
    "rawOutputMaxBytes": 1048576
  }
}

enabledFilters and disabledFilters use filter ids, for example test-vitest or git-diff.

API

Route	Method	Purpose
/api/context/rtk/config	GET	Read RTK config
/api/context/rtk/config	PUT	Update RTK config
/api/context/rtk/filters	GET	List filter catalog and load diagnostics
/api/context/rtk/test	POST	Preview RTK compression for one text payload
/api/context/rtk/raw-output/[id]	GET	Read retained redacted raw output
/api/compression/preview	POST	Preview any compression mode

RTK test payload:

{
  "command": "npm test",
  "text": "FAIL tests/example.test.ts\nAssertionError: expected true\nTest Files 1 failed",
  "config": {
    "intensity": "standard"
  }
}

Compression preview payload:

{
  "mode": "stacked",
  "messages": [
    {
      "role": "tool",
      "content": "FAIL tests/example.test.ts\nAssertionError: expected true\nTest Files 1 failed"
    }
  ],
  "config": {
    "rtkConfig": {
      "rawOutputRetention": "failures"
    }
  }
}

Management routes require dashboard management auth or the matching API-key policy.

Raw Output Recovery

RTK normally returns only compressed text. For debugging, rawOutputRetention can retain redacted raw output:

Value	Behavior
never	Do not retain raw output
failures	Retain only likely failure output
always	Retain every compressed RTK raw output, after redaction

Retained files are written under:

DATA_DIR/rtk/raw-output/

Secrets are redacted before persistence, including common bearer tokens, API keys, Slack tokens, AWS access keys, and assignment-style token=..., secret=..., password=... values. Analytics stores only the pointer id, size, and hash metadata.

Verify Gate

The focused verify gate runs built-in inline filter tests without shelling out to external commands:

node --import tsx/esm --test tests/unit/compression/rtk-verify.test.ts

The broader RTK gate is:

node --import tsx/esm --test \
  tests/unit/compression/rtk-*.test.ts \
  tests/unit/compression/pipeline-integration.test.ts \
  tests/unit/compression/context-compression-api.test.ts

Run the broad compression gate before release:

node --import tsx/esm --test \
  tests/unit/compression/*.test.ts \
  tests/golden-set/*.test.ts \
  tests/integration/compression-pipeline.test.ts \
  tests/unit/api/compression/compression-api.test.ts

Extending RTK

1. Add or update a filter JSON file.
2. Include at least one tests[] sample that proves the important behavior.
3. Add a fixture under tests/unit/compression/fixtures/rtk/ for new command families.
4. Add command detection coverage when introducing a new output class.
5. Run the verify and broad RTK gates.
6. If the filter is project-local, commit .rtk/filters.json and refresh .rtk/trust.json only after review.