feat(compression): support RTK TOML schema v1 filters (#7281)

* feat(compression): support RTK TOML filters

* chore(ci): sync RTK skill and dependency allowlist

* refactor(compression): reduce RTK import complexity

* fix(i18n): add Portuguese RTK import translations

* fix(compression): improve RTK TOML import validation feedback
This commit is contained in:
Jan Leon
2026-07-18 20:12:43 +02:00
committed by GitHub
parent dc0dec46c7
commit c78f150ac3
22 changed files with 1651 additions and 53 deletions

View File

@@ -114,6 +114,7 @@
"safe-regex",
"selfsigned",
"size-limit",
"smol-toml",
"socks",
"sql.js",
"sqlite-vec",

View File

@@ -93,6 +93,8 @@ RTK mode is optimized for verbose tool outputs that appear in coding-agent sessi
TypeScript/Vite/Webpack builds, ESLint/Biome/Prettier, npm audit/installs, Docker logs, infra
output, and generic shell output
- Applies JSON filter packs from `open-sse/services/compression/engines/rtk/filters/`
- Imports RTK TOML schema v1 filters from project or global `filters.toml` files, with inline-test
validation and trust-gating for project files
- Ships 49 built-in filters with inline verify samples
- Removes ANSI control sequences, progress bars, repeated lines, and non-actionable noise
- Preserves failures, errors, warnings, changed files, summaries, and the tail of long output

View File

@@ -52,28 +52,59 @@ class is not enough.
RTK loads filters in this order:
1. Project filters from `.rtk/filters.json`, only when trusted.
2. Global filters from `DATA_DIR/rtk/filters.json`.
1. Project filters from `.rtk/filters.toml` and `.rtk/filters.json`, only when trusted.
2. Global filters from `DATA_DIR/rtk/filters.toml` and `DATA_DIR/rtk/filters.json`.
3. Built-in filters from `open-sse/services/compression/engines/rtk/filters/`.
Within the same scope, RTK TOML schema v1 filters take precedence over OmniRoute JSON filters. TOML
`match_command` expressions are checked before command-type matching so an imported command-specific
filter can override a broader filter in that scope. Project scope still takes precedence over global
scope, regardless of file format.
Project filters are intentionally trust-gated because regex filters can change how tool output is
shown to agents. A project filter file is accepted when one of these is true:
- `rtkConfig.trustProjectFilters` is `true`.
- `OMNIROUTE_RTK_TRUST_PROJECT_FILTERS=1` is set.
- `.rtk/trust.json` contains the SHA-256 hash of `.rtk/filters.json`.
- `.rtk/trust.json` contains the matching SHA-256 hash for the project filter file.
Trust file example:
```json
{
"filtersSha256": "0123456789abcdef..."
"filtersSha256": "0123456789abcdef...",
"filtersTomlSha256": "fedcba9876543210..."
}
```
The hashes are separate: `filtersSha256` trusts `.rtk/filters.json`, while `filtersTomlSha256`
trusts `.rtk/filters.toml`. Editing either file invalidates only its own trust entry. Global files
are administrator-installed and use the existing global-filter trust behavior.
Custom filters can be one filter object or an array of filter objects. Invalid custom filters are
skipped and reported by `/api/context/rtk/filters` diagnostics. Invalid built-in filters fail fast.
## RTK TOML schema v1 compatibility
OmniRoute can parse, validate, test, and install declarative filter files using RTK TOML schema v1.
The supported fields are `description`, `match_command`, `strip_ansi`, `filter_stderr`,
`strip_lines_matching`, `keep_lines_matching`, `replace`, `match_output`, `truncate_lines_at`,
`head_lines`, `tail_lines`, `max_lines`, `on_empty`, and `[[tests.<filter>]]` inline tests.
Unknown fields, invalid or unsafe regular expressions, simultaneous strip/keep rules, files over
1 MiB, and references to unknown filters are rejected. A file whose inline tests fail can be
validated for inspection but cannot be installed or loaded. Custom-file load failures remain
fail-open: the invalid file is skipped and the remaining filters continue to work.
OmniRoute receives tool output after the client has already captured it, so `filter_stderr = true`
cannot change process capture. The field is accepted as a no-op and validation returns a warning.
This is intentionally described as **RTK TOML schema v1 compatibility**, not full compatibility
with the RTK executable, shell hooks, Rust command implementations, or its trust-store layout.
The dashboard's advanced RTK view accepts pasted or uploaded TOML. Validation is read-only.
Installation writes `DATA_DIR/rtk/filters.toml` atomically with restrictive permissions and refreshes
the live filter catalog without a restart. Replacing an existing file requires explicit `overwrite`
confirmation and creates `DATA_DIR/rtk/filters.toml.bak` first.
## Filter DSL
Filters use the JSON schema described in [Compression Rules Format](./COMPRESSION_RULES_FORMAT.md).
@@ -216,6 +247,7 @@ round-trips through the same store and survives a restart.
| `/api/context/rtk/config` | GET | Read RTK config |
| `/api/context/rtk/config` | PUT | Update RTK config |
| `/api/context/rtk/filters` | GET | List filter catalog and load diagnostics |
| `/api/context/rtk/import` | POST | Validate or install RTK TOML schema v1 files |
| `/api/context/rtk/test` | POST | Preview RTK compression for one text payload |
| `/api/context/rtk/raw-output/[id]` | GET | Read retained redacted raw output |
| `/api/compression/preview` | POST | Preview any compression mode |
@@ -253,6 +285,18 @@ Compression preview payload:
Management routes require dashboard management auth or the matching API-key policy.
RTK TOML validation payload:
```json
{
"action": "validate",
"content": "schema_version = 1\n\n[filters.my-tool]\nmatch_command = \"^my-tool\\\\b\"\nmax_lines = 20\n"
}
```
Use `"action": "install"` to install the validated file globally. Add `"overwrite": true` only
after reviewing and confirming replacement of an existing global file.
## Raw Output Recovery
RTK normally returns only compressed text. For debugging, `rawOutputRetention` can retain redacted

View File

@@ -2201,6 +2201,36 @@ paths:
"200":
description: RTK filter catalog and diagnostics
/api/context/rtk/import:
post:
tags: [Compression]
summary: Validate or install an RTK TOML schema v1 filter file
security:
- ManagementSessionAuth: []
requestBody:
required: true
content:
application/json:
schema:
type: object
required: [action, content]
additionalProperties: false
properties:
action:
type: string
enum: [validate, install]
content:
type: string
maxLength: 1048576
overwrite:
type: boolean
description: Replace an existing global file and create a backup
responses:
"200":
description: Filter metadata, inline-test outcomes, warnings, and installation status
"400":
description: Invalid TOML, schema, regular expression, inline test, or install request
/api/context/rtk/test:
post:
tags: [Compression]

View File

@@ -12,6 +12,7 @@
},
"dependencies": {
"@toon-format/toon": "^2.3.0",
"safe-regex": "^2.1.1"
"safe-regex": "^2.1.1",
"smol-toml": "1.6.1"
}
}

View File

@@ -4,6 +4,7 @@ import os from "node:os";
import crypto from "node:crypto";
import { detectCommandType } from "./commandDetector.ts";
import { validateRtkFilter, type RtkFilterDefinition } from "./filterSchema.ts";
import { parseRtkTomlV1, RtkTomlCompatibilityError } from "./tomlCompatibility.ts";
let cache: RtkFilterDefinition[] | null = null;
let cacheKey: string | null = null;
@@ -29,6 +30,7 @@ function cachedMatchPattern(pattern: string, value: string): boolean {
export interface RtkFilterLoadDiagnostic {
source: "project" | "global" | "builtin";
format?: "omniroute-json" | "rtk-toml-v1";
path?: string;
level: "warning" | "error";
message: string;
@@ -38,6 +40,7 @@ interface FilterSource {
source: "project" | "global" | "builtin";
path: string;
trusted: boolean;
format: "omniroute-json" | "rtk-toml-v1";
}
interface RtkFilterLoadOptions {
@@ -95,8 +98,12 @@ function projectFiltersTrusted(
try {
const filtersHash = sha256(fs.readFileSync(filtersPath, "utf8"));
const trust = JSON.parse(fs.readFileSync(trustPath, "utf8")) as Record<string, unknown>;
const trustedHash =
typeof trust.filtersSha256 === "string"
const isToml = filtersPath.endsWith(".toml");
const trustedHash = isToml
? typeof trust.filtersTomlSha256 === "string"
? trust.filtersTomlSha256
: null
: typeof trust.filtersSha256 === "string"
? trust.filtersSha256
: typeof trust.trustedFiltersSha256 === "string"
? trust.trustedFiltersSha256
@@ -110,29 +117,52 @@ function projectFiltersTrusted(
function collectFilterSources(options: RtkFilterLoadOptions = {}): FilterSource[] {
const sources: FilterSource[] = [];
const projectPath = path.join(process.cwd(), ".rtk", "filters.json");
if (options.customFiltersEnabled !== false && fs.existsSync(projectPath)) {
const trusted = projectFiltersTrusted(projectPath, options.trustProjectFilters === true);
if (options.customFiltersEnabled !== false) {
collectProjectFilterSources(sources, options);
collectGlobalFilterSources(sources);
}
collectBuiltinFilterSources(sources);
return sources;
}
function collectProjectFilterSources(sources: FilterSource[], options: RtkFilterLoadOptions): void {
const projectCandidates = [
{ path: path.join(process.cwd(), ".rtk", "filters.toml"), format: "rtk-toml-v1" as const },
{ path: path.join(process.cwd(), ".rtk", "filters.json"), format: "omniroute-json" as const },
];
for (const candidate of projectCandidates) {
if (!fs.existsSync(candidate.path)) continue;
const trusted = projectFiltersTrusted(candidate.path, options.trustProjectFilters === true);
if (trusted === true) {
sources.push({ source: "project", path: projectPath, trusted: true });
} else {
diagnostics.push({
source: "project",
path: projectPath,
level: "warning",
message:
trusted === "changed"
? "Project RTK filters changed after trust and were skipped"
: "Project RTK filters are untrusted and were skipped",
});
sources.push({ source: "project", ...candidate, trusted: true });
continue;
}
diagnostics.push({
source: "project",
format: candidate.format,
path: candidate.path,
level: "warning",
message:
trusted === "changed"
? "Project RTK filters changed after trust and were skipped"
: "Project RTK filters are untrusted and were skipped",
});
}
}
function collectGlobalFilterSources(sources: FilterSource[]): void {
const globalCandidates = [
{ path: path.join(getDataDir(), "rtk", "filters.toml"), format: "rtk-toml-v1" as const },
{ path: path.join(getDataDir(), "rtk", "filters.json"), format: "omniroute-json" as const },
];
for (const candidate of globalCandidates) {
if (fs.existsSync(candidate.path)) {
sources.push({ source: "global", ...candidate, trusted: true });
}
}
}
const globalPath = path.join(getDataDir(), "rtk", "filters.json");
if (options.customFiltersEnabled !== false && fs.existsSync(globalPath)) {
sources.push({ source: "global", path: globalPath, trusted: true });
}
function collectBuiltinFilterSources(sources: FilterSource[]): void {
const builtinDir = getFiltersDir();
if (fs.existsSync(builtinDir)) {
let builtinFiles: string[] = [];
@@ -152,25 +182,56 @@ function collectFilterSources(options: RtkFilterLoadOptions = {}): FilterSource[
source: "builtin",
path: path.join(builtinDir, file),
trusted: true,
format: "omniroute-json",
});
}
}
return sources;
}
function parseFilterFile(source: FilterSource): RtkFilterDefinition[] {
try {
const parsed = JSON.parse(fs.readFileSync(source.path, "utf8"));
const entries = Array.isArray(parsed) ? parsed : [parsed];
return entries.map(validateRtkFilter);
const content = fs.readFileSync(source.path, "utf8");
const definitions =
source.format === "rtk-toml-v1"
? (() => {
const result = parseRtkTomlV1(content);
if (!result.passed) {
throw new Error("one or more inline tests failed");
}
for (const warning of result.warnings) {
diagnostics.push({
source: source.source,
format: source.format,
path: source.path,
level: "warning",
message: warning,
});
}
return result.filters;
})()
: (() => {
const parsed = JSON.parse(content);
const entries = Array.isArray(parsed) ? parsed : [parsed];
return entries.map(validateRtkFilter);
})();
return definitions.map((definition) => ({
...definition,
source: source.source,
sourceFormat: definition.sourceFormat ?? source.format,
}));
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
const message =
error instanceof RtkTomlCompatibilityError
? error.publicMessage
: error instanceof Error
? error.message
: String(error);
if (source.source === "builtin") {
throw new Error(`Invalid RTK filter ${path.basename(source.path)}: ${message}`);
}
diagnostics.push({
source: source.source,
format: source.format,
path: source.path,
level: "warning",
message: `Invalid custom RTK filter skipped: ${message}`,
@@ -194,7 +255,16 @@ export function loadRtkFilters(options: RtkFilterLoadOptions = {}): RtkFilterDef
filters.push(...parseFilterFile(source));
}
const sorted = filters.sort((a, b) => b.priority - a.priority || a.id.localeCompare(b.id));
const sourceRank = { project: 3, global: 2, builtin: 1 } as const;
const formatRank = { "rtk-toml-v1": 2, "omniroute-json": 1 } as const;
const sorted = filters.sort(
(a, b) =>
sourceRank[b.source ?? "builtin"] - sourceRank[a.source ?? "builtin"] ||
formatRank[b.sourceFormat ?? "omniroute-json"] -
formatRank[a.sourceFormat ?? "omniroute-json"] ||
b.priority - a.priority ||
a.id.localeCompare(b.id)
);
cache = sorted;
cacheKey = currentCacheKey;
return sorted;
@@ -208,7 +278,14 @@ export function getRtkFilterLoadDiagnostics(): RtkFilterLoadDiagnostic[] {
export function getRtkFilterCatalog(): Array<
Pick<
RtkFilterDefinition,
"id" | "name" | "description" | "commandTypes" | "category" | "priority"
| "id"
| "name"
| "description"
| "commandTypes"
| "category"
| "priority"
| "source"
| "sourceFormat"
>
> {
return loadRtkFilters().map((filter) => ({
@@ -218,6 +295,8 @@ export function getRtkFilterCatalog(): Array<
commandTypes: filter.commandTypes,
category: filter.category,
priority: filter.priority,
source: filter.source,
sourceFormat: filter.sourceFormat,
}));
}
@@ -229,17 +308,25 @@ export function matchRtkFilter(
const detection = detectCommandType(text, command);
const detectedCommand = detection.command ?? command ?? "";
const filters = loadRtkFilters(options);
return (
filters.find((filter) => filter.commandTypes.includes(detection.type)) ??
filters.find(
(filter) =>
detectedCommand &&
filter.commandPatterns.some((pattern) => cachedMatchPattern(pattern, detectedCommand))
) ??
filters.find((filter) =>
filter.matchPatterns.some((pattern) => cachedMatchPattern(pattern, text))
) ??
filters.find((filter) => filter.commandTypes.includes("generic-output")) ??
null
);
for (const source of ["project", "global", "builtin"] as const) {
const scoped = filters.filter((filter) => (filter.source ?? "builtin") === source);
const matched =
scoped.find(
(filter) =>
filter.sourceFormat === "rtk-toml-v1" &&
detectedCommand &&
filter.commandPatterns.some((pattern) => cachedMatchPattern(pattern, detectedCommand))
) ??
scoped.find((filter) => filter.commandTypes.includes(detection.type)) ??
scoped.find(
(filter) =>
detectedCommand &&
filter.commandPatterns.some((pattern) => cachedMatchPattern(pattern, detectedCommand))
) ??
scoped.find((filter) =>
filter.matchPatterns.some((pattern) => cachedMatchPattern(pattern, text))
);
if (matched) return matched;
}
return filters.find((filter) => filter.commandTypes.includes("generic-output")) ?? null;
}

View File

@@ -137,6 +137,12 @@ export interface RtkFilterDefinition {
maxLines: number;
preserveHead: number;
preserveTail: number;
/** Exact RTK TOML schema-v1 head/tail stages. Undefined for OmniRoute-native JSON filters. */
rtkTomlHeadLines?: number;
rtkTomlTailLines?: number;
rtkTomlMaxLines?: number;
sourceFormat?: "omniroute-json" | "rtk-toml-v1";
source?: "project" | "global" | "builtin";
tests: Array<{ name: string; input: string; expected: string; command?: string }>;
}

View File

@@ -54,6 +54,41 @@ function normalizeStderrPrefix(line: string): string {
return line.replace(/^\s*(?:stderr|err)\s*(?:\||:)\s*/i, "");
}
function applyRtkTomlLineLimits(
lines: string[],
filter: RtkFilterDefinition,
appliedRules: string[]
): string[] {
const head = filter.rtkTomlHeadLines;
const tail = filter.rtkTomlTailLines;
const total = lines.length;
if (head !== undefined && tail !== undefined) {
if (total > head + tail) {
lines = [
...lines.slice(0, head),
`... (${total - head - tail} lines omitted)`,
...(tail > 0 ? lines.slice(-tail) : []),
];
appliedRules.push(`${filter.id}:rtk-head-tail`);
}
} else if (head !== undefined && total > head) {
lines = [...lines.slice(0, head), `... (${total - head} lines omitted)`];
appliedRules.push(`${filter.id}:rtk-head`);
} else if (tail !== undefined && total > tail) {
lines = [`... (${total - tail} lines omitted)`, ...(tail > 0 ? lines.slice(-tail) : [])];
appliedRules.push(`${filter.id}:rtk-tail`);
}
const maxLines = filter.rtkTomlMaxLines;
if (maxLines !== undefined && lines.length > maxLines) {
const dropped = lines.length - maxLines;
lines = [...lines.slice(0, maxLines), `... (${dropped} lines truncated)`];
appliedRules.push(`${filter.id}:rtk-max-lines`);
}
return lines;
}
function truncateUnicodeSafe(line: string, maxChars: number): string {
if (maxChars <= 0) return line;
const chars = Array.from(line);
@@ -70,6 +105,7 @@ export function applyLineFilter(text: string, filter: RtkFilterDefinition): Line
const appliedRules: string[] = [];
let lines = text.split(/\r?\n/);
if (filter.sourceFormat === "rtk-toml-v1" && lines.at(-1) === "") lines.pop();
const originalLineCount = lines.length;
if (filter.stripAnsi) {
@@ -159,6 +195,18 @@ export function applyLineFilter(text: string, filter: RtkFilterDefinition): Line
}
}
if (filter.sourceFormat === "rtk-toml-v1") {
lines = applyRtkTomlLineLimits(lines, filter, appliedRules);
const output = lines.join("\n");
const finalOutput = output.trim().length === 0 && filter.onEmpty ? filter.onEmpty : output;
return {
text: finalOutput,
strippedLines: Math.max(0, originalLineCount - finalOutput.split(/\r?\n/).length),
keptByRule: keepPatterns.length > 0,
appliedRules,
};
}
const truncated = smartTruncate(lines.join("\n"), {
maxLines: filter.maxLines,
preserveHead: filter.preserveHead,

View File

@@ -0,0 +1,334 @@
import crypto from "node:crypto";
import fs from "node:fs";
import path from "node:path";
import os from "node:os";
import { parse as parseToml } from "smol-toml";
import { z } from "zod";
import { isReDoSProne, type RtkFilterDefinition } from "./filterSchema.ts";
import { applyLineFilter } from "./lineFilter.ts";
const MAX_TOML_BYTES = 1024 * 1024;
const replaceRuleSchema = z
.object({
pattern: z.string().min(1),
replacement: z.string(),
})
.strict();
const matchOutputRuleSchema = z
.object({
pattern: z.string().min(1),
message: z.string(),
unless: z.string().min(1).optional(),
})
.strict();
const filterSchema = z
.object({
description: z.string().optional(),
match_command: z.string().min(1),
strip_ansi: z.boolean().optional(),
filter_stderr: z.boolean().optional(),
strip_lines_matching: z.array(z.string()).optional(),
keep_lines_matching: z.array(z.string()).optional(),
replace: z.array(replaceRuleSchema).optional(),
match_output: z.array(matchOutputRuleSchema).optional(),
truncate_lines_at: z.number().int().min(0).optional(),
head_lines: z.number().int().min(0).optional(),
tail_lines: z.number().int().min(0).optional(),
max_lines: z.number().int().min(0).optional(),
on_empty: z.string().optional(),
})
.strict();
const inlineTestSchema = z
.object({
name: z.string().min(1),
input: z.string(),
expected: z.string(),
})
.strict();
const fileSchema = z
.object({
schema_version: z.literal(1),
filters: z.record(z.string().min(1), filterSchema).default({}),
tests: z.record(z.string().min(1), z.array(inlineTestSchema)).default({}),
})
.strict();
type ParsedFilter = z.infer<typeof filterSchema>;
function arrayOrEmpty<T>(value: T[] | undefined): T[] {
return value ?? [];
}
function numberOrZero(value: number | undefined): number {
return value ?? 0;
}
export interface RtkTomlTestOutcome {
filterId: string;
testName: string;
passed: boolean;
actual: string;
expected: string;
}
export interface RtkTomlCompatibilityResult {
schemaVersion: 1;
sha256: string;
filters: RtkFilterDefinition[];
outcomes: RtkTomlTestOutcome[];
filtersWithoutTests: string[];
warnings: string[];
passed: boolean;
}
export class RtkTomlCompatibilityError extends Error {
readonly publicMessage: string;
constructor(message: string) {
super("RTK TOML schema v1 compatibility error");
this.name = "RtkTomlCompatibilityError";
this.publicMessage = message;
}
}
function compatibilityError(message: string): RtkTomlCompatibilityError {
return new RtkTomlCompatibilityError(message);
}
function categoryFor(name: string, commandPattern: string): RtkFilterDefinition["category"] {
const value = `${name} ${commandPattern}`.toLowerCase();
if (/\b(?:git|gh)\b/.test(value)) return "git";
if (/\b(?:test|jest|vitest|pytest|cargo test|go test|rspec|playwright)\b/.test(value)) {
return "test";
}
if (/\b(?:build|tsc|eslint|ruff|clippy|gradle|make|next|vite|webpack)\b/.test(value)) {
return "build";
}
if (/\b(?:docker|kubectl|podman|compose)\b/.test(value)) return "docker";
if (/\b(?:npm|pnpm|yarn|bun|pip|poetry|uv|bundle|composer)\b/.test(value)) {
return "package";
}
if (/\b(?:terraform|tofu|ansible|helm|pulumi)\b/.test(value)) return "infra";
if (/\b(?:aws|gcloud|az|cloudflare)\b/.test(value)) return "cloud";
if (/\b(?:ls|find|grep|rg|df|du|ps|systemctl|ssh|rsync)\b/.test(value)) return "shell";
return "generic";
}
function regexFields(filter: ParsedFilter): Array<{ field: string; pattern: string }> {
return [
{ field: "match_command", pattern: filter.match_command },
...(filter.strip_lines_matching ?? []).map((pattern) => ({
field: "strip_lines_matching",
pattern,
})),
...(filter.keep_lines_matching ?? []).map((pattern) => ({
field: "keep_lines_matching",
pattern,
})),
...(filter.replace ?? []).map(({ pattern }) => ({ field: "replace.pattern", pattern })),
...(filter.match_output ?? []).flatMap(({ pattern, unless }) => [
{ field: "match_output.pattern", pattern },
...(unless ? [{ field: "match_output.unless", pattern: unless }] : []),
]),
];
}
function validateRegexes(name: string, filter: ParsedFilter): void {
for (const { field, pattern } of regexFields(filter)) {
if (isReDoSProne(pattern)) {
throw compatibilityError(`filter '${name}' has an unsafe regex in ${field}`);
}
try {
new RegExp(pattern);
} catch {
throw compatibilityError(`filter '${name}' has an invalid regex in ${field}`);
}
}
}
function toDefinition(
name: string,
filter: ParsedFilter,
tests: z.infer<typeof inlineTestSchema>[]
): RtkFilterDefinition {
if (
(filter.strip_lines_matching?.length ?? 0) > 0 &&
(filter.keep_lines_matching?.length ?? 0) > 0
) {
throw compatibilityError(
`filter '${name}' cannot combine strip_lines_matching with keep_lines_matching`
);
}
validateRegexes(name, filter);
return {
id: name,
name,
description: filter.description ?? "",
commandTypes: [],
commandPatterns: [filter.match_command],
matchPatterns: [],
category: categoryFor(name, filter.match_command),
priority: 50,
stripPatterns: arrayOrEmpty(filter.strip_lines_matching),
keepPatterns: arrayOrEmpty(filter.keep_lines_matching),
priorityPatterns: [],
collapsePatterns: [],
stripAnsi: filter.strip_ansi ?? false,
replace: arrayOrEmpty(filter.replace),
matchOutput: arrayOrEmpty(filter.match_output),
truncateLineAt: numberOrZero(filter.truncate_lines_at),
onEmpty: filter.on_empty ?? "",
filterStderr: false,
deduplicate: false,
maxLines: numberOrZero(filter.max_lines),
preserveHead: 0,
preserveTail: 0,
rtkTomlHeadLines: filter.head_lines,
rtkTomlTailLines: filter.tail_lines,
rtkTomlMaxLines: filter.max_lines,
sourceFormat: "rtk-toml-v1",
tests,
};
}
function comparable(value: string): string {
return value.replace(/\n+$/g, "");
}
function tomlSyntaxLocation(error: unknown): string {
if (typeof error !== "object" || error === null) return "";
const { line, column } = error as { line?: unknown; column?: unknown };
if (!Number.isSafeInteger(line) || !Number.isSafeInteger(column)) return "";
return ` (line ${line}, column ${column})`;
}
export function parseRtkTomlV1(content: string): RtkTomlCompatibilityResult {
if (Buffer.byteLength(content, "utf8") > MAX_TOML_BYTES) {
throw compatibilityError(`file exceeds the ${MAX_TOML_BYTES}-byte limit`);
}
let raw: unknown;
try {
raw = parseToml(content);
} catch (error) {
throw compatibilityError(`invalid TOML syntax${tomlSyntaxLocation(error)}`);
}
const parsed = fileSchema.safeParse(raw);
if (!parsed.success) {
const issue = parsed.error.issues[0];
const field = issue?.path.length ? issue.path.join(".") : "document";
throw compatibilityError(`${field}: ${issue?.message ?? "invalid document"}`);
}
if (Object.keys(parsed.data.filters).length === 0) {
throw compatibilityError("document contains no filters");
}
for (const testName of Object.keys(parsed.data.tests)) {
if (!(testName in parsed.data.filters)) {
throw compatibilityError(`tests reference unknown filter '${testName}'`);
}
}
const filters = Object.entries(parsed.data.filters).map(([name, filter]) =>
toDefinition(name, filter, parsed.data.tests[name] ?? [])
);
const outcomes = filters.flatMap((filter) =>
filter.tests.map((test) => {
const actual = comparable(applyLineFilter(test.input, filter).text);
const expected = comparable(test.expected);
return {
filterId: filter.id,
testName: test.name,
passed: actual === expected,
actual,
expected,
};
})
);
const filtersWithoutTests = filters
.filter((filter) => filter.tests.length === 0)
.map((filter) => filter.id);
const warnings = filtersWithoutTests.map(
(id) => `Filter '${id}' has no inline tests and should be reviewed before installation`
);
for (const [id, filter] of Object.entries(parsed.data.filters)) {
if (filter.filter_stderr) {
warnings.push(
`Filter '${id}': filter_stderr is accepted as a no-op because OmniRoute receives already-captured tool output`
);
}
}
return {
schemaVersion: 1,
sha256: crypto.createHash("sha256").update(content).digest("hex"),
filters,
outcomes,
filtersWithoutTests,
warnings,
passed: outcomes.every((outcome) => outcome.passed),
};
}
function getDataDir(): string {
return process.env.DATA_DIR || path.join(os.homedir(), ".omniroute");
}
export function getGlobalRtkTomlPath(): string {
return path.join(getDataDir(), "rtk", "filters.toml");
}
export function installGlobalRtkTomlV1(
content: string,
options: { overwrite?: boolean } = {}
): RtkTomlCompatibilityResult & { installedPath: string; backupCreated: boolean } {
const result = parseRtkTomlV1(content);
if (!result.passed) {
throw compatibilityError("one or more inline tests failed");
}
const target = getGlobalRtkTomlPath();
const directory = path.dirname(target);
fs.mkdirSync(directory, { recursive: true, mode: 0o700 });
try {
fs.chmodSync(directory, 0o700);
} catch {
// Best effort on filesystems that do not support POSIX permissions.
}
if (fs.existsSync(target) && !options.overwrite) {
throw compatibilityError("filters.toml already exists; confirm overwrite to replace it");
}
let backupCreated = false;
if (fs.existsSync(target)) {
fs.copyFileSync(target, `${target}.bak`);
try {
fs.chmodSync(`${target}.bak`, 0o600);
} catch {
// Best effort on filesystems that do not support POSIX permissions.
}
backupCreated = true;
}
const temporary = `${target}.${process.pid}.${crypto.randomUUID()}.tmp`;
try {
fs.writeFileSync(temporary, content, { encoding: "utf8", mode: 0o600, flag: "wx" });
fs.renameSync(temporary, target);
try {
fs.chmodSync(target, 0o600);
} catch {
// Best effort on filesystems that do not support POSIX permissions.
}
} finally {
fs.rmSync(temporary, { force: true });
}
return { ...result, installedPath: "rtk/filters.toml", backupCreated };
}
export const RTK_TOML_MAX_BYTES = MAX_TOML_BYTES;

5
package-lock.json generated
View File

@@ -74,6 +74,7 @@
"recharts": "^3.8.1",
"safe-regex": "^2.1.1",
"selfsigned": "^5.5.0",
"smol-toml": "1.6.1",
"socks": "^2.8.7",
"sql.js": "^1.14.1",
"sqlite-vec": "^0.1.9",
@@ -33779,7 +33780,6 @@
"version": "1.6.1",
"resolved": "https://registry.npmjs.org/smol-toml/-/smol-toml-1.6.1.tgz",
"integrity": "sha512-dWUG8F5sIIARXih1DTaQAX4SsiTXhInKf1buxdY9DIg4ZYPZK5nGM1VRIYmEbDbsHt7USo99xSLFu5Q1IqTmsg==",
"dev": true,
"license": "BSD-3-Clause",
"engines": {
"node": ">= 18"
@@ -37620,7 +37620,8 @@
"version": "3.8.49",
"dependencies": {
"@toon-format/toon": "^2.3.0",
"safe-regex": "^2.1.1"
"safe-regex": "^2.1.1",
"smol-toml": "1.6.1"
}
}
}

View File

@@ -285,6 +285,7 @@
"recharts": "^3.8.1",
"safe-regex": "^2.1.1",
"selfsigned": "^5.5.0",
"smol-toml": "1.6.1",
"socks": "^2.8.7",
"sql.js": "^1.14.1",
"sqlite-vec": "^0.1.9",

View File

@@ -43,6 +43,17 @@ curl https://localhost:20128/api/context/rtk/filters \
-H "Authorization: Bearer $OMNIROUTE_TOKEN"
```
### POST /api/context/rtk/import
Validate or install an RTK TOML schema v1 filter file
```bash
curl -X POST https://localhost:20128/api/context/rtk/import \
-H "Authorization: Bearer $OMNIROUTE_TOKEN"
-H "Content-Type: application/json" \
-d '{}'
```
### POST /api/context/rtk/test
Run RTK compression preview for text

View File

@@ -4,6 +4,7 @@ import { useEffect, useMemo, useState } from "react";
import { useTranslations } from "next-intl";
import { SegmentedControl, Collapsible } from "@/shared/components";
import RtkLearnDiscoverCard from "./RtkLearnDiscoverCard";
import RtkTomlImportCard from "./RtkTomlImportCard";
type RtkFilter = {
id: string;
@@ -78,11 +79,14 @@ export default function RtkContextPageClient() {
.catch(() => {});
}, []);
useEffect(() => {
const loadFilters = () =>
fetch("/api/context/rtk/filters")
.then((res) => (res.ok ? res.json() : null))
.then((data) => setFilters(Array.isArray(data?.filters) ? data.filters : []))
.catch(() => {});
useEffect(() => {
void loadFilters();
fetch("/api/context/rtk/config")
.then((res) => (res.ok ? res.json() : null))
.then((data) => setConfig(data))
@@ -363,6 +367,8 @@ export default function RtkContextPageClient() {
</div>
</Collapsible>
{viewMode === "advanced" && <RtkTomlImportCard onInstalled={loadFilters} />}
<RtkLearnDiscoverCard />
</div>
);

View File

@@ -0,0 +1,255 @@
"use client";
import { useState } from "react";
import { useTranslations } from "next-intl";
const RTK_TOML_MAX_BYTES = 1024 * 1024;
interface ImportFilterSummary {
id: string;
description: string;
category: string;
commandPatterns: string[];
testCount: number;
}
interface ImportTestOutcome {
filterId: string;
testName: string;
passed: boolean;
}
interface ImportResult {
sha256: string;
passed: boolean;
filters: ImportFilterSummary[];
outcomes: ImportTestOutcome[];
warnings: string[];
installedPath?: string;
backupCreated?: boolean;
}
interface RtkTomlImportCardProps {
onInstalled?: () => void | Promise<void>;
}
interface RtkTomlEditorProps {
content: string;
processing: "validate" | "install" | null;
overwrite: boolean;
onContentChange: (content: string) => void;
onFileChange: (file: File | undefined) => void;
onProcess: (action: "validate" | "install") => void;
onOverwriteChange: (overwrite: boolean) => void;
}
async function readErrorMessage(response: Response): Promise<string | null> {
try {
const body = (await response.json()) as { error?: { message?: unknown } };
return typeof body.error?.message === "string" ? body.error.message : null;
} catch {
return null;
}
}
function RtkTomlEditor({
content,
processing,
overwrite,
onContentChange,
onFileChange,
onProcess,
onOverwriteChange,
}: RtkTomlEditorProps) {
const t = useTranslations("contextRtk");
return (
<>
<div className="mt-3 flex flex-col gap-3">
<label className="text-xs font-medium text-text-main">
{t("tomlChooseFile")}
<input
type="file"
accept=".toml,text/plain,application/toml"
onChange={(event) => onFileChange(event.target.files?.[0])}
data-testid="rtk-toml-file"
className="mt-1 block w-full text-xs text-text-muted file:mr-3 file:rounded file:border file:border-border file:bg-bg file:px-2.5 file:py-1 file:text-xs file:text-text-main"
/>
</label>
<textarea
value={content}
onChange={(event) => onContentChange(event.target.value)}
placeholder={t("tomlImportPlaceholder")}
data-testid="rtk-toml-content"
className="h-56 w-full rounded-lg border border-border bg-bg p-3 font-mono text-xs text-text-main"
/>
</div>
<div className="mt-3 flex flex-wrap items-center gap-3">
<button
type="button"
onClick={() => onProcess("validate")}
disabled={processing !== null || !content.trim()}
data-testid="rtk-toml-validate"
className="rounded border border-border px-3 py-1.5 text-xs font-medium text-text-main hover:bg-surface-hover disabled:opacity-50"
>
{processing === "validate" ? t("tomlValidating") : t("tomlValidate")}
</button>
<button
type="button"
onClick={() => onProcess("install")}
disabled={processing !== null || !content.trim()}
data-testid="rtk-toml-install"
className="rounded bg-primary px-3 py-1.5 text-xs font-medium text-white disabled:opacity-50"
>
{processing === "install" ? t("tomlInstalling") : t("tomlInstall")}
</button>
<label className="flex items-center gap-2 text-xs text-text-main">
<input
type="checkbox"
checked={overwrite}
onChange={(event) => onOverwriteChange(event.target.checked)}
data-testid="rtk-toml-overwrite"
/>
{t("tomlConfirmOverwrite")}
</label>
</div>
</>
);
}
function RtkTomlResult({ result }: { result: ImportResult }) {
const t = useTranslations("contextRtk");
return (
<div className="mt-4 rounded-lg border border-border bg-bg p-3" data-testid="rtk-toml-result">
<p className="text-xs font-medium text-text-main">
{result.installedPath
? t("tomlInstalled", { path: result.installedPath })
: result.passed
? t("tomlValidationPassed")
: t("tomlValidationFailed")}
</p>
<p className="mt-1 text-[11px] text-text-muted">
{t("tomlValidationSummary", {
filters: result.filters.length,
tests: result.outcomes.length,
})}
</p>
{result.backupCreated && (
<p className="mt-1 text-[11px] text-text-muted">{t("tomlBackupCreated")}</p>
)}
{result.filters.length > 0 && (
<ul className="mt-2 space-y-1 text-[11px] text-text-main">
{result.filters.map((filter) => (
<li key={filter.id}>
<code>{filter.id}</code> · {filter.category} ·{" "}
{t("tomlTestCount", { count: filter.testCount })}
</li>
))}
</ul>
)}
{result.outcomes.length > 0 && (
<ul className="mt-2 space-y-1 text-[11px] text-text-main">
{result.outcomes.map((outcome) => (
<li key={`${outcome.filterId}:${outcome.testName}`}>
<span
className={
outcome.passed
? "text-emerald-700 dark:text-emerald-300"
: "text-red-600 dark:text-red-400"
}
>
{outcome.passed ? t("tomlTestPassed") : t("tomlTestFailed")}
</span>{" "}
· <code>{outcome.filterId}</code> · {outcome.testName}
</li>
))}
</ul>
)}
{result.warnings.length > 0 && (
<ul className="mt-2 space-y-1 text-[11px] text-amber-700 dark:text-amber-300">
{result.warnings.map((warning) => (
<li key={warning}>{warning}</li>
))}
</ul>
)}
</div>
);
}
export default function RtkTomlImportCard({ onInstalled }: RtkTomlImportCardProps) {
const t = useTranslations("contextRtk");
const [content, setContent] = useState("");
const [result, setResult] = useState<ImportResult | null>(null);
const [error, setError] = useState<string | null>(null);
const [processing, setProcessing] = useState<"validate" | "install" | null>(null);
const [overwrite, setOverwrite] = useState(false);
async function processImport(action: "validate" | "install") {
if (!content.trim()) return;
setProcessing(action);
setError(null);
try {
const response = await fetch("/api/context/rtk/import", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ action, content, overwrite: action === "install" && overwrite }),
});
if (!response.ok) {
throw new Error((await readErrorMessage(response)) ?? t("tomlImportError"));
}
const nextResult = (await response.json()) as ImportResult;
setResult(nextResult);
if (action === "install") await onInstalled?.();
} catch (caught) {
setResult(null);
setError(caught instanceof Error ? caught.message : t("tomlImportError"));
} finally {
setProcessing(null);
}
}
async function loadFile(file: File | undefined) {
if (!file) return;
setError(null);
if (file.size > RTK_TOML_MAX_BYTES) {
setError(t("tomlFileReadError"));
return;
}
try {
setContent(await file.text());
setResult(null);
} catch {
setError(t("tomlFileReadError"));
}
}
return (
<section
className="rounded-lg border border-border bg-surface p-4"
data-testid="rtk-toml-import"
>
<h2 className="text-sm font-semibold text-text-main">{t("tomlImportTitle")}</h2>
<p className="mt-1 text-xs text-text-muted">{t("tomlImportDesc")}</p>
<RtkTomlEditor
content={content}
processing={processing}
overwrite={overwrite}
onContentChange={(nextContent) => {
setContent(nextContent);
setResult(null);
}}
onFileChange={(file) => void loadFile(file)}
onProcess={(action) => void processImport(action)}
onOverwriteChange={setOverwrite}
/>
{error && (
<p className="mt-3 text-xs text-red-600 dark:text-red-400" data-testid="rtk-toml-error">
{error}
</p>
)}
{result && <RtkTomlResult result={result} />}
</section>
);
}

View File

@@ -0,0 +1,81 @@
import { NextResponse } from "next/server";
import { z } from "zod";
import { requireManagementAuth } from "@/lib/api/requireManagementAuth";
import { buildErrorBody } from "@omniroute/open-sse/utils/error";
import { loadRtkFilters } from "@omniroute/open-sse/services/compression/engines/rtk/filterLoader";
import {
installGlobalRtkTomlV1,
parseRtkTomlV1,
RTK_TOML_MAX_BYTES,
RtkTomlCompatibilityError,
type RtkTomlCompatibilityResult,
} from "@omniroute/open-sse/services/compression/engines/rtk/tomlCompatibility";
const RequestSchema = z
.object({
action: z.enum(["validate", "install"]),
content: z.string().min(1).max(RTK_TOML_MAX_BYTES),
overwrite: z.boolean().optional(),
})
.strict();
function responseBody(
result: RtkTomlCompatibilityResult & {
installedPath?: string;
backupCreated?: boolean;
}
) {
return {
schemaVersion: result.schemaVersion,
sha256: result.sha256,
passed: result.passed,
filters: result.filters.map((filter) => ({
id: filter.id,
description: filter.description,
category: filter.category,
commandPatterns: filter.commandPatterns,
testCount: filter.tests.length,
})),
outcomes: result.outcomes,
filtersWithoutTests: result.filtersWithoutTests,
warnings: result.warnings,
installedPath: result.installedPath,
backupCreated: result.backupCreated,
};
}
export async function POST(request: Request) {
const authError = await requireManagementAuth(request);
if (authError) return authError;
let body: unknown;
try {
body = await request.json();
} catch {
return NextResponse.json(buildErrorBody(400, "Invalid JSON body"), { status: 400 });
}
const parsed = RequestSchema.safeParse(body);
if (!parsed.success) {
return NextResponse.json(buildErrorBody(400, "Invalid RTK TOML import request"), {
status: 400,
});
}
try {
const result =
parsed.data.action === "install"
? installGlobalRtkTomlV1(parsed.data.content, { overwrite: parsed.data.overwrite })
: parseRtkTomlV1(parsed.data.content);
if (parsed.data.action === "install") {
loadRtkFilters({ refresh: true });
}
return NextResponse.json(responseBody(result));
} catch (error) {
if (error instanceof RtkTomlCompatibilityError) {
return NextResponse.json(buildErrorBody(400, error.publicMessage), { status: 400 });
}
return NextResponse.json(buildErrorBody(500, "Failed to process RTK TOML filter import"), {
status: 500,
});
}
}

View File

@@ -5609,7 +5609,26 @@
"searchFilters": "Suchfilter...",
"tooltipDedup": "Wie aggressiv wiederholte Zeilen entfernt werden.",
"tooltipMaxChars": "Maximale Zeichen pro Ausgabeblock.",
"tooltipMaxLines": "Maximale Anzahl von Zeilen, die von der Befehlsausgabe ferngehalten werden sollen. Der Überschuss wird abgeschnitten."
"tooltipMaxLines": "Maximale Anzahl von Zeilen, die von der Befehlsausgabe ferngehalten werden sollen. Der Überschuss wird abgeschnitten.",
"tomlImportTitle": "RTK-TOML-Filter importieren",
"tomlImportDesc": "Filter im RTK-TOML-Schema v1 validieren und global installieren. Die Installation schreibt DATA_DIR/rtk/filters.toml; vorhandene Dateien werden nur nach ausdrücklicher Bestätigung ersetzt.",
"tomlChooseFile": "TOML-Datei auswählen",
"tomlImportPlaceholder": "Inhalt einer RTK-filters.toml hier einfügen ...",
"tomlValidate": "Validieren",
"tomlValidating": "Wird validiert …",
"tomlInstall": "Global installieren",
"tomlInstalling": "Wird installiert …",
"tomlConfirmOverwrite": "Vorhandene globale Datei ersetzen und Sicherung erstellen",
"tomlValidationPassed": "Validierung erfolgreich",
"tomlValidationFailed": "Validierung mit fehlgeschlagenen Inline-Tests abgeschlossen",
"tomlValidationSummary": "{filters} Filter, {tests} Inline-Tests",
"tomlInstalled": "Unter {path} installiert",
"tomlBackupCreated": "Eine Sicherung der vorherigen Datei wurde erstellt.",
"tomlTestCount": "{count} Tests",
"tomlTestPassed": "Bestanden",
"tomlTestFailed": "Fehlgeschlagen",
"tomlImportError": "Die RTK-TOML-Datei konnte nicht verarbeitet werden.",
"tomlFileReadError": "Die ausgewählte TOML-Datei konnte nicht gelesen werden."
},
"contextCombos": {
"title": "Compression Combos",

View File

@@ -6352,7 +6352,26 @@
"learnButton": "Suggest filter",
"learnEmpty": "No suggestion yet. Type a command you have run and scan.",
"learnSamplesUsed": "Learned from {count} matching sample(s)",
"suggestionError": "Could not load suggestions. Check management auth and try again."
"suggestionError": "Could not load suggestions. Check management auth and try again.",
"tomlImportTitle": "Import RTK TOML filters",
"tomlImportDesc": "Validate RTK TOML schema v1 filters and install them globally. Installation writes DATA_DIR/rtk/filters.toml; existing files are replaced only after explicit confirmation.",
"tomlChooseFile": "Choose a TOML file",
"tomlImportPlaceholder": "Paste RTK filters.toml content here...",
"tomlValidate": "Validate",
"tomlValidating": "Validating…",
"tomlInstall": "Install globally",
"tomlInstalling": "Installing…",
"tomlConfirmOverwrite": "Replace the existing global file and create a backup",
"tomlValidationPassed": "Validation passed",
"tomlValidationFailed": "Validation completed with failing inline tests",
"tomlValidationSummary": "{filters} filter(s), {tests} inline test(s)",
"tomlInstalled": "Installed at {path}",
"tomlBackupCreated": "A backup of the previous file was created.",
"tomlTestCount": "{count} test(s)",
"tomlTestPassed": "Passed",
"tomlTestFailed": "Failed",
"tomlImportError": "Could not process the RTK TOML file.",
"tomlFileReadError": "Could not read the selected TOML file."
},
"contextCombos": {
"title": "Compression Combos",

View File

@@ -6352,7 +6352,26 @@
"learnButton": "__MISSING__:Suggest filter",
"learnEmpty": "__MISSING__:No suggestion yet. Type a command you have run and scan.",
"learnSamplesUsed": "__MISSING__:Learned from {count} matching sample(s)",
"suggestionError": "__MISSING__:Could not load suggestions. Check management auth and try again."
"suggestionError": "__MISSING__:Could not load suggestions. Check management auth and try again.",
"tomlImportTitle": "Importar filtros TOML do RTK",
"tomlImportDesc": "Valide os filtros do esquema TOML v1 do RTK e instale-os globalmente. A instalação grava em DATA_DIR/rtk/filters.toml; arquivos existentes só são substituídos após confirmação explícita.",
"tomlChooseFile": "Escolher um arquivo TOML",
"tomlImportPlaceholder": "Cole aqui o conteúdo do arquivo filters.toml do RTK...",
"tomlValidate": "Validar",
"tomlValidating": "Validando…",
"tomlInstall": "Instalar globalmente",
"tomlInstalling": "Instalando…",
"tomlConfirmOverwrite": "Substituir o arquivo global existente e criar um backup",
"tomlValidationPassed": "Validação bem-sucedida",
"tomlValidationFailed": "Validação concluída com falhas nos testes embutidos",
"tomlValidationSummary": "{filters} filtro(s), {tests} teste(s) embutido(s)",
"tomlInstalled": "Instalado em {path}",
"tomlBackupCreated": "Foi criado um backup do arquivo anterior.",
"tomlTestCount": "{count} teste(s)",
"tomlTestPassed": "Aprovado",
"tomlTestFailed": "Reprovado",
"tomlImportError": "Não foi possível processar o arquivo TOML do RTK.",
"tomlFileReadError": "Não foi possível ler o arquivo TOML selecionado."
},
"contextCombos": {
"title": "Combos de Compressao",

View File

@@ -0,0 +1,125 @@
import test from "node:test";
import assert from "node:assert/strict";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { makeManagementSessionRequest } from "../../../helpers/managementSession.ts";
const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-rtk-toml-route-"));
const ORIGINAL_DATA_DIR = process.env.DATA_DIR;
const ORIGINAL_INITIAL_PASSWORD = process.env.INITIAL_PASSWORD;
process.env.DATA_DIR = TEST_DATA_DIR;
const core = await import("../../../../src/lib/db/core.ts");
const settingsDb = await import("../../../../src/lib/db/settings.ts");
const importRoute = await import("../../../../src/app/api/context/rtk/import/route.ts");
const SAMPLE = `schema_version = 1
[filters.route-test]
description = "Route test"
match_command = "^route-test"
strip_lines_matching = ["^noise"]
[[tests.route-test]]
name = "removes noise"
input = "noise\\nkept"
expected = "kept"
`;
async function reset() {
core.resetDbInstance();
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
fs.mkdirSync(TEST_DATA_DIR, { recursive: true });
delete process.env.INITIAL_PASSWORD;
}
async function post(body: unknown): Promise<Response> {
return importRoute.POST(
await makeManagementSessionRequest("http://localhost/api/context/rtk/import", {
method: "POST",
body,
})
);
}
test.beforeEach(reset);
test.after(() => {
core.resetDbInstance();
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
if (ORIGINAL_DATA_DIR === undefined) delete process.env.DATA_DIR;
else process.env.DATA_DIR = ORIGINAL_DATA_DIR;
if (ORIGINAL_INITIAL_PASSWORD === undefined) delete process.env.INITIAL_PASSWORD;
else process.env.INITIAL_PASSWORD = ORIGINAL_INITIAL_PASSWORD;
});
test("POST requires management authentication when login is configured", async () => {
process.env.INITIAL_PASSWORD = "rtk-toml-route-password";
await settingsDb.updateSettings({ requireLogin: true });
const response = await importRoute.POST(
new Request("http://localhost/api/context/rtk/import", {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ action: "validate", content: SAMPLE }),
})
);
assert.equal(response.status, 401);
});
test("validate returns tests and does not write filters.toml", async () => {
const response = await post({ action: "validate", content: SAMPLE });
const body = await response.json();
assert.equal(response.status, 200);
assert.equal(body.passed, true);
assert.equal(body.filters[0].id, "route-test");
assert.equal(body.outcomes[0].passed, true);
assert.equal(fs.existsSync(path.join(TEST_DATA_DIR, "rtk", "filters.toml")), false);
});
test("install writes the global file, refuses implicit overwrite, and creates a backup", async () => {
const first = await post({ action: "install", content: SAMPLE });
assert.equal(first.status, 200);
assert.equal(fs.readFileSync(path.join(TEST_DATA_DIR, "rtk", "filters.toml"), "utf8"), SAMPLE);
const refused = await post({ action: "install", content: SAMPLE });
assert.equal(refused.status, 400);
const replacement = SAMPLE.replace("Route test", "Replacement");
const replaced = await post({ action: "install", content: replacement, overwrite: true });
const replacedBody = await replaced.json();
assert.equal(replaced.status, 200);
assert.equal(replacedBody.backupCreated, true);
assert.equal(
fs.readFileSync(path.join(TEST_DATA_DIR, "rtk", "filters.toml.bak"), "utf8"),
SAMPLE
);
});
test("invalid TOML, unsafe regex, failing tests, and oversized input return safe 400 errors", async () => {
const inputs = [
"not = [valid",
`schema_version = 1\n[filters.bad]\nmatch_command = "(a+)+$"`,
`schema_version = 1
[filters.bad]
match_command = "^bad"
strip_lines_matching = ["^noise"]
[[tests.bad]]
name = "fails"
input = "noise\\nkept"
expected = "wrong"`,
"x".repeat(1024 * 1024 + 1),
];
for (const content of inputs) {
const response = await post({ action: "install", content });
const body = await response.json();
assert.equal(response.status, 400);
assert.equal(typeof body.error?.message, "string");
assert.ok(!body.error.message.includes(" at "));
assert.ok(!body.error.message.includes(process.cwd()));
}
});

View File

@@ -36,4 +36,20 @@ describe("RTK line filters", () => {
assert.ok(result.text.includes("On branch"));
assert.ok(!result.text.includes("nothing added"));
});
it("does not truncate within a combined RTK TOML head and tail limit", () => {
const filter = {
...loadRtkFilters().find((item) => item.id === "generic-output")!,
id: "toml-head-tail",
sourceFormat: "rtk-toml-v1" as const,
rtkTomlHeadLines: 2,
rtkTomlTailLines: 2,
};
const result = applyLineFilter("first\nsecond\nthird", filter);
assert.equal(result.text, "first\nsecond\nthird");
assert.ok(!result.appliedRules.includes("toml-head-tail:rtk-head"));
assert.ok(!result.appliedRules.includes("toml-head-tail:rtk-head-tail"));
});
});

View File

@@ -0,0 +1,313 @@
import { afterEach, describe, it } from "node:test";
import assert from "node:assert/strict";
import crypto from "node:crypto";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { applyLineFilter } from "../../../open-sse/services/compression/engines/rtk/lineFilter.ts";
import {
getRtkFilterCatalog,
getRtkFilterLoadDiagnostics,
loadRtkFilters,
matchRtkFilter,
} from "../../../open-sse/services/compression/engines/rtk/filterLoader.ts";
import {
installGlobalRtkTomlV1,
parseRtkTomlV1,
RtkTomlCompatibilityError,
} from "../../../open-sse/services/compression/engines/rtk/tomlCompatibility.ts";
const originalCwd = process.cwd();
const originalDataDir = process.env.DATA_DIR;
const SAMPLE = `schema_version = 1
[filters.my-tool]
description = "Compact my-tool output"
match_command = "^my-tool\\\\s+build"
strip_ansi = true
strip_lines_matching = ["^noise", "^\\\\s*$"]
replace = [{ pattern = "duration: ([0-9]+)ms", replacement = "t=$1ms" }]
match_output = [{ pattern = "all good", message = "my-tool: ok", unless = "error" }]
truncate_lines_at = 12
head_lines = 2
tail_lines = 1
max_lines = 4
on_empty = "my-tool: empty"
[[tests.my-tool]]
name = "filters and truncates"
input = "noise banner\\nfirst line\\nsecond line\\nthird line\\nfourth line\\n"
expected = "first line\\nsecond line\\n... (1 lines omitted)\\nfourth line"
[[tests.my-tool]]
name = "short circuits success"
input = "all good"
expected = "my-tool: ok"
`;
afterEach(() => {
process.chdir(originalCwd);
if (originalDataDir === undefined) delete process.env.DATA_DIR;
else process.env.DATA_DIR = originalDataDir;
loadRtkFilters({ refresh: true, customFiltersEnabled: false });
});
describe("RTK TOML schema v1 compatibility", () => {
it("reports a safe location for invalid TOML syntax", () => {
assert.throws(
() => parseRtkTomlV1("not = [valid"),
(error: unknown) =>
error instanceof RtkTomlCompatibilityError &&
error.publicMessage === "invalid TOML syntax (line 1, column 8)"
);
});
it("parses the official declarative fields and passes inline tests", () => {
const result = parseRtkTomlV1(SAMPLE);
assert.equal(result.schemaVersion, 1);
assert.equal(result.filters.length, 1);
assert.equal(result.outcomes.length, 2);
assert.ok(result.outcomes.every((outcome) => outcome.passed));
assert.equal(result.passed, true);
assert.deepEqual(result.filtersWithoutTests, []);
assert.equal(result.filters[0].sourceFormat, "rtk-toml-v1");
});
it("matches imported TOML before a detected builtin command type", () => {
const project = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-rtk-toml-project-"));
process.chdir(project);
const rtkDir = path.join(project, ".rtk");
const filterPath = path.join(rtkDir, "filters.toml");
fs.mkdirSync(rtkDir, { recursive: true });
fs.writeFileSync(
filterPath,
`schema_version = 1
[filters.git-status-custom]
match_command = "^git\\\\s+status"
strip_lines_matching = ["^noise"]
`
);
fs.writeFileSync(
path.join(rtkDir, "trust.json"),
JSON.stringify({
filtersTomlSha256: crypto
.createHash("sha256")
.update(fs.readFileSync(filterPath))
.digest("hex"),
})
);
const filter = matchRtkFilter("noise\n M file.ts", "git status", { refresh: true });
assert.equal(filter?.id, "git-status-custom");
assert.equal(filter?.source, "project");
});
it("requires a separate trust hash for project TOML files", () => {
const project = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-rtk-toml-project-"));
process.chdir(project);
const rtkDir = path.join(project, ".rtk");
fs.mkdirSync(rtkDir, { recursive: true });
fs.writeFileSync(path.join(rtkDir, "filters.toml"), SAMPLE);
fs.writeFileSync(
path.join(rtkDir, "trust.json"),
JSON.stringify({ filtersSha256: "wrong-key" })
);
const filters = loadRtkFilters({ refresh: true });
const diagnostics = getRtkFilterLoadDiagnostics();
assert.ok(!filters.some((filter) => filter.id === "my-tool"));
assert.ok(
diagnostics.some(
(entry) => entry.format === "rtk-toml-v1" && entry.message.includes("untrusted")
)
);
});
it("loads global TOML and exposes its source format in the catalog", () => {
const project = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-rtk-toml-project-"));
const dataDir = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-rtk-toml-data-"));
process.chdir(project);
process.env.DATA_DIR = dataDir;
fs.mkdirSync(path.join(dataDir, "rtk"), { recursive: true });
fs.writeFileSync(path.join(dataDir, "rtk", "filters.toml"), SAMPLE);
loadRtkFilters({ refresh: true });
const entry = getRtkFilterCatalog().find((filter) => filter.id === "my-tool");
assert.equal(entry?.source, "global");
assert.equal(entry?.sourceFormat, "rtk-toml-v1");
});
it("prioritizes project TOML over a higher-priority global JSON match", () => {
const project = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-rtk-toml-project-"));
const dataDir = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-rtk-toml-data-"));
process.chdir(project);
process.env.DATA_DIR = dataDir;
const projectDir = path.join(project, ".rtk");
fs.mkdirSync(projectDir, { recursive: true });
const projectPath = path.join(projectDir, "filters.toml");
fs.writeFileSync(
projectPath,
`schema_version = 1
[filters.project-match]
match_command = "^scope-tool"
`
);
fs.writeFileSync(
path.join(projectDir, "trust.json"),
JSON.stringify({
filtersTomlSha256: crypto
.createHash("sha256")
.update(fs.readFileSync(projectPath))
.digest("hex"),
})
);
fs.mkdirSync(path.join(dataDir, "rtk"), { recursive: true });
fs.writeFileSync(
path.join(dataDir, "rtk", "filters.json"),
JSON.stringify({
id: "global-match",
label: "Global match",
description: "",
category: "generic",
priority: 100,
match: { commands: ["^scope-tool"], patterns: [], outputTypes: [] },
})
);
const filter = matchRtkFilter("output", "scope-tool", { refresh: true });
assert.equal(filter?.id, "project-match");
});
it("keeps project JSON ahead of a matching global TOML filter", () => {
const project = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-rtk-toml-project-"));
const dataDir = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-rtk-toml-data-"));
process.chdir(project);
process.env.DATA_DIR = dataDir;
const projectDir = path.join(project, ".rtk");
fs.mkdirSync(projectDir, { recursive: true });
const projectPath = path.join(projectDir, "filters.json");
fs.writeFileSync(
projectPath,
JSON.stringify({
id: "project-json-match",
label: "Project JSON match",
description: "",
category: "generic",
match: { commands: ["^scope-tool"], patterns: [], outputTypes: [] },
rules: {},
preserve: { errorPatterns: [], summaryPatterns: [] },
tests: [],
})
);
fs.writeFileSync(
path.join(projectDir, "trust.json"),
JSON.stringify({
filtersSha256: crypto
.createHash("sha256")
.update(fs.readFileSync(projectPath))
.digest("hex"),
})
);
fs.mkdirSync(path.join(dataDir, "rtk"), { recursive: true });
fs.writeFileSync(
path.join(dataDir, "rtk", "filters.toml"),
`schema_version = 1
[filters.global-toml-match]
match_command = "^scope-tool"
`
);
const filter = matchRtkFilter("output", "scope-tool", { refresh: true });
assert.equal(filter?.id, "project-json-match");
});
it("installs atomically, protects the file and refuses an implicit overwrite", () => {
const dataDir = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-rtk-toml-data-"));
process.env.DATA_DIR = dataDir;
const installed = installGlobalRtkTomlV1(SAMPLE);
const target = path.join(dataDir, "rtk", "filters.toml");
assert.equal(installed.installedPath, "rtk/filters.toml");
assert.equal(fs.readFileSync(target, "utf8"), SAMPLE);
assert.equal(fs.statSync(target).mode & 0o777, 0o600);
assert.throws(
() => installGlobalRtkTomlV1(SAMPLE),
(error: unknown) =>
error instanceof RtkTomlCompatibilityError && error.publicMessage.includes("already exists")
);
});
it("rejects unknown fields, unsafe regexes and failing inline tests", () => {
assert.throws(() =>
parseRtkTomlV1(`schema_version = 1
[filters.bad]
match_command = "^bad"
run = "rm -rf /"
`)
);
assert.throws(() =>
parseRtkTomlV1(`schema_version = 1
[filters.bad]
match_command = "(a+)+$"
`)
);
const failed = parseRtkTomlV1(`schema_version = 1
[filters.bad]
match_command = "^bad"
strip_lines_matching = ["^noise"]
[[tests.bad]]
name = "wrong"
input = "noise\\nkept"
expected = "different"
`);
assert.equal(failed.passed, false);
assert.throws(() =>
installGlobalRtkTomlV1(`schema_version = 1
[filters.bad]
match_command = "^bad"
strip_lines_matching = ["^noise"]
[[tests.bad]]
name = "wrong"
input = "noise\\nkept"
expected = "different"
`)
);
});
it("keeps filter_stderr as a documented proxy no-op", () => {
const result = parseRtkTomlV1(`schema_version = 1
[filters.stderr-tool]
match_command = "^stderr-tool"
filter_stderr = true
`);
assert.equal(result.filters[0].filterStderr, false);
assert.ok(result.warnings.some((warning) => warning.includes("filter_stderr")));
assert.equal(applyLineFilter("stderr: error", result.filters[0]).text, "stderr: error");
});
it("rejects non-schema fields in inline tests", () => {
assert.throws(() =>
parseRtkTomlV1(`schema_version = 1
[filters.bad]
match_command = "^bad"
[[tests.bad]]
name = "extra field"
input = "input"
expected = "input"
command = "bad"
`)
);
});
});

View File

@@ -0,0 +1,179 @@
// @vitest-environment jsdom
import React, { act } from "react";
import { createRoot } from "react-dom/client";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
vi.mock("next-intl", () => ({
useTranslations: () => (key: string, params?: Record<string, unknown>) =>
params ? `${key}:${JSON.stringify(params)}` : key,
}));
const { default: RtkTomlImportCard } =
await import("@/app/(dashboard)/dashboard/context/rtk/RtkTomlImportCard");
const containers: HTMLElement[] = [];
function mount(ui: React.ReactElement): HTMLElement {
const container = document.createElement("div");
document.body.appendChild(container);
containers.push(container);
act(() => createRoot(container).render(ui));
return container;
}
async function setTextarea(container: HTMLElement, value: string) {
const input = container.querySelector("[data-testid='rtk-toml-content']") as HTMLTextAreaElement;
await act(async () => {
const setter = Object.getOwnPropertyDescriptor(
window.HTMLTextAreaElement.prototype,
"value"
)!.set!;
setter.call(input, value);
input.dispatchEvent(new Event("input", { bubbles: true }));
});
}
async function click(element: Element | null) {
await act(async () => (element as HTMLElement).click());
}
async function selectFile(container: HTMLElement, file: File) {
const input = container.querySelector("[data-testid='rtk-toml-file']") as HTMLInputElement;
Object.defineProperty(input, "files", { configurable: true, value: [file] });
await act(async () => input.dispatchEvent(new Event("change", { bubbles: true })));
}
beforeEach(() => {
(
globalThis as typeof globalThis & { IS_REACT_ACT_ENVIRONMENT?: boolean }
).IS_REACT_ACT_ENVIRONMENT = true;
});
afterEach(() => {
while (containers.length) containers.pop()?.remove();
document.body.innerHTML = "";
vi.restoreAllMocks();
vi.unstubAllGlobals();
});
describe("RtkTomlImportCard", () => {
it("validates TOML without requesting installation", async () => {
const fetchMock = vi.fn(async () =>
Response.json({
sha256: "abc",
passed: true,
filters: [
{
id: "my-tool",
description: "",
category: "generic",
commandPatterns: ["^my-tool"],
testCount: 1,
},
],
outcomes: [{ filterId: "my-tool", testName: "works", passed: true }],
warnings: [],
})
);
vi.stubGlobal("fetch", fetchMock);
const container = mount(<RtkTomlImportCard />);
await setTextarea(container, "schema_version = 1");
await click(container.querySelector("[data-testid='rtk-toml-validate']"));
const [, options] = fetchMock.mock.calls[0];
expect(JSON.parse(String(options?.body))).toMatchObject({
action: "validate",
overwrite: false,
});
expect(container.querySelector("[data-testid='rtk-toml-result']")?.textContent).toContain(
"my-tool"
);
});
it("installs with explicit overwrite and refreshes the filter catalog", async () => {
const onInstalled = vi.fn(async () => {});
const fetchMock = vi.fn(async () =>
Response.json({
sha256: "abc",
passed: true,
filters: [],
outcomes: [],
warnings: [],
installedPath: "rtk/filters.toml",
backupCreated: true,
})
);
vi.stubGlobal("fetch", fetchMock);
const container = mount(<RtkTomlImportCard onInstalled={onInstalled} />);
await setTextarea(container, "schema_version = 1");
await click(container.querySelector("[data-testid='rtk-toml-overwrite']"));
await click(container.querySelector("[data-testid='rtk-toml-install']"));
const [, options] = fetchMock.mock.calls[0];
expect(JSON.parse(String(options?.body))).toMatchObject({ action: "install", overwrite: true });
expect(onInstalled).toHaveBeenCalledOnce();
expect(container.querySelector("[data-testid='rtk-toml-result']")?.textContent).toContain(
"tomlBackupCreated"
);
});
it("shows the sanitized API error message", async () => {
vi.stubGlobal(
"fetch",
vi.fn(async () =>
Response.json({ error: { message: "filter 'bad' has an unsafe regex" } }, { status: 400 })
)
);
const container = mount(<RtkTomlImportCard />);
await setTextarea(container, "schema_version = 1");
await click(container.querySelector("[data-testid='rtk-toml-validate']"));
expect(container.querySelector("[data-testid='rtk-toml-error']")?.textContent).toContain(
"unsafe regex"
);
});
it("shows failing inline-test outcomes without reporting a successful validation", async () => {
vi.stubGlobal(
"fetch",
vi.fn(async () =>
Response.json({
sha256: "abc",
passed: false,
filters: [],
outcomes: [{ filterId: "bad", testName: "expected output", passed: false }],
warnings: [],
})
)
);
const container = mount(<RtkTomlImportCard />);
await setTextarea(container, "schema_version = 1");
await click(container.querySelector("[data-testid='rtk-toml-validate']"));
const result = container.querySelector("[data-testid='rtk-toml-result']")?.textContent ?? "";
expect(result).toContain("tomlValidationFailed");
expect(result).toContain("tomlTestFailed");
expect(result).not.toContain("tomlValidationPassed");
});
it("rejects oversized files before reading them", async () => {
const container = mount(<RtkTomlImportCard />);
await setTextarea(container, "existing content");
const file = new File([new Uint8Array(1024 * 1024 + 1)], "filters.toml");
const text = vi.spyOn(file, "text");
await selectFile(container, file);
expect(text).not.toHaveBeenCalled();
expect(container.querySelector("[data-testid='rtk-toml-error']")?.textContent).toContain(
"tomlFileReadError"
);
expect(
(container.querySelector("[data-testid='rtk-toml-content']") as HTMLTextAreaElement).value
).toBe("existing content");
});
});