mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-07-26 09:52:11 +03:00
feat(compression): support RTK TOML schema v1 filters (#7281)
* feat(compression): support RTK TOML filters * chore(ci): sync RTK skill and dependency allowlist * refactor(compression): reduce RTK import complexity * fix(i18n): add Portuguese RTK import translations * fix(compression): improve RTK TOML import validation feedback
This commit is contained in:
@@ -114,6 +114,7 @@
|
||||
"safe-regex",
|
||||
"selfsigned",
|
||||
"size-limit",
|
||||
"smol-toml",
|
||||
"socks",
|
||||
"sql.js",
|
||||
"sqlite-vec",
|
||||
|
||||
@@ -93,6 +93,8 @@ RTK mode is optimized for verbose tool outputs that appear in coding-agent sessi
|
||||
TypeScript/Vite/Webpack builds, ESLint/Biome/Prettier, npm audit/installs, Docker logs, infra
|
||||
output, and generic shell output
|
||||
- Applies JSON filter packs from `open-sse/services/compression/engines/rtk/filters/`
|
||||
- Imports RTK TOML schema v1 filters from project or global `filters.toml` files, with inline-test
|
||||
validation and trust-gating for project files
|
||||
- Ships 49 built-in filters with inline verify samples
|
||||
- Removes ANSI control sequences, progress bars, repeated lines, and non-actionable noise
|
||||
- Preserves failures, errors, warnings, changed files, summaries, and the tail of long output
|
||||
|
||||
@@ -52,28 +52,59 @@ class is not enough.
|
||||
|
||||
RTK loads filters in this order:
|
||||
|
||||
1. Project filters from `.rtk/filters.json`, only when trusted.
|
||||
2. Global filters from `DATA_DIR/rtk/filters.json`.
|
||||
1. Project filters from `.rtk/filters.toml` and `.rtk/filters.json`, only when trusted.
|
||||
2. Global filters from `DATA_DIR/rtk/filters.toml` and `DATA_DIR/rtk/filters.json`.
|
||||
3. Built-in filters from `open-sse/services/compression/engines/rtk/filters/`.
|
||||
|
||||
Within the same scope, RTK TOML schema v1 filters take precedence over OmniRoute JSON filters. TOML
|
||||
`match_command` expressions are checked before command-type matching so an imported command-specific
|
||||
filter can override a broader filter in that scope. Project scope still takes precedence over global
|
||||
scope, regardless of file format.
|
||||
|
||||
Project filters are intentionally trust-gated because regex filters can change how tool output is
|
||||
shown to agents. A project filter file is accepted when one of these is true:
|
||||
|
||||
- `rtkConfig.trustProjectFilters` is `true`.
|
||||
- `OMNIROUTE_RTK_TRUST_PROJECT_FILTERS=1` is set.
|
||||
- `.rtk/trust.json` contains the SHA-256 hash of `.rtk/filters.json`.
|
||||
- `.rtk/trust.json` contains the matching SHA-256 hash for the project filter file.
|
||||
|
||||
Trust file example:
|
||||
|
||||
```json
|
||||
{
|
||||
"filtersSha256": "0123456789abcdef..."
|
||||
"filtersSha256": "0123456789abcdef...",
|
||||
"filtersTomlSha256": "fedcba9876543210..."
|
||||
}
|
||||
```
|
||||
|
||||
The hashes are separate: `filtersSha256` trusts `.rtk/filters.json`, while `filtersTomlSha256`
|
||||
trusts `.rtk/filters.toml`. Editing either file invalidates only its own trust entry. Global files
|
||||
are administrator-installed and use the existing global-filter trust behavior.
|
||||
|
||||
Custom filters can be one filter object or an array of filter objects. Invalid custom filters are
|
||||
skipped and reported by `/api/context/rtk/filters` diagnostics. Invalid built-in filters fail fast.
|
||||
|
||||
## RTK TOML schema v1 compatibility
|
||||
|
||||
OmniRoute can parse, validate, test, and install declarative filter files using RTK TOML schema v1.
|
||||
The supported fields are `description`, `match_command`, `strip_ansi`, `filter_stderr`,
|
||||
`strip_lines_matching`, `keep_lines_matching`, `replace`, `match_output`, `truncate_lines_at`,
|
||||
`head_lines`, `tail_lines`, `max_lines`, `on_empty`, and `[[tests.<filter>]]` inline tests.
|
||||
Unknown fields, invalid or unsafe regular expressions, simultaneous strip/keep rules, files over
|
||||
1 MiB, and references to unknown filters are rejected. A file whose inline tests fail can be
|
||||
validated for inspection but cannot be installed or loaded. Custom-file load failures remain
|
||||
fail-open: the invalid file is skipped and the remaining filters continue to work.
|
||||
|
||||
OmniRoute receives tool output after the client has already captured it, so `filter_stderr = true`
|
||||
cannot change process capture. The field is accepted as a no-op and validation returns a warning.
|
||||
This is intentionally described as **RTK TOML schema v1 compatibility**, not full compatibility
|
||||
with the RTK executable, shell hooks, Rust command implementations, or its trust-store layout.
|
||||
|
||||
The dashboard's advanced RTK view accepts pasted or uploaded TOML. Validation is read-only.
|
||||
Installation writes `DATA_DIR/rtk/filters.toml` atomically with restrictive permissions and refreshes
|
||||
the live filter catalog without a restart. Replacing an existing file requires explicit `overwrite`
|
||||
confirmation and creates `DATA_DIR/rtk/filters.toml.bak` first.
|
||||
|
||||
## Filter DSL
|
||||
|
||||
Filters use the JSON schema described in [Compression Rules Format](./COMPRESSION_RULES_FORMAT.md).
|
||||
@@ -216,6 +247,7 @@ round-trips through the same store and survives a restart.
|
||||
| `/api/context/rtk/config` | GET | Read RTK config |
|
||||
| `/api/context/rtk/config` | PUT | Update RTK config |
|
||||
| `/api/context/rtk/filters` | GET | List filter catalog and load diagnostics |
|
||||
| `/api/context/rtk/import` | POST | Validate or install RTK TOML schema v1 files |
|
||||
| `/api/context/rtk/test` | POST | Preview RTK compression for one text payload |
|
||||
| `/api/context/rtk/raw-output/[id]` | GET | Read retained redacted raw output |
|
||||
| `/api/compression/preview` | POST | Preview any compression mode |
|
||||
@@ -253,6 +285,18 @@ Compression preview payload:
|
||||
|
||||
Management routes require dashboard management auth or the matching API-key policy.
|
||||
|
||||
RTK TOML validation payload:
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "validate",
|
||||
"content": "schema_version = 1\n\n[filters.my-tool]\nmatch_command = \"^my-tool\\\\b\"\nmax_lines = 20\n"
|
||||
}
|
||||
```
|
||||
|
||||
Use `"action": "install"` to install the validated file globally. Add `"overwrite": true` only
|
||||
after reviewing and confirming replacement of an existing global file.
|
||||
|
||||
## Raw Output Recovery
|
||||
|
||||
RTK normally returns only compressed text. For debugging, `rawOutputRetention` can retain redacted
|
||||
|
||||
@@ -2201,6 +2201,36 @@ paths:
|
||||
"200":
|
||||
description: RTK filter catalog and diagnostics
|
||||
|
||||
/api/context/rtk/import:
|
||||
post:
|
||||
tags: [Compression]
|
||||
summary: Validate or install an RTK TOML schema v1 filter file
|
||||
security:
|
||||
- ManagementSessionAuth: []
|
||||
requestBody:
|
||||
required: true
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
type: object
|
||||
required: [action, content]
|
||||
additionalProperties: false
|
||||
properties:
|
||||
action:
|
||||
type: string
|
||||
enum: [validate, install]
|
||||
content:
|
||||
type: string
|
||||
maxLength: 1048576
|
||||
overwrite:
|
||||
type: boolean
|
||||
description: Replace an existing global file and create a backup
|
||||
responses:
|
||||
"200":
|
||||
description: Filter metadata, inline-test outcomes, warnings, and installation status
|
||||
"400":
|
||||
description: Invalid TOML, schema, regular expression, inline test, or install request
|
||||
|
||||
/api/context/rtk/test:
|
||||
post:
|
||||
tags: [Compression]
|
||||
|
||||
@@ -12,6 +12,7 @@
|
||||
},
|
||||
"dependencies": {
|
||||
"@toon-format/toon": "^2.3.0",
|
||||
"safe-regex": "^2.1.1"
|
||||
"safe-regex": "^2.1.1",
|
||||
"smol-toml": "1.6.1"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -4,6 +4,7 @@ import os from "node:os";
|
||||
import crypto from "node:crypto";
|
||||
import { detectCommandType } from "./commandDetector.ts";
|
||||
import { validateRtkFilter, type RtkFilterDefinition } from "./filterSchema.ts";
|
||||
import { parseRtkTomlV1, RtkTomlCompatibilityError } from "./tomlCompatibility.ts";
|
||||
|
||||
let cache: RtkFilterDefinition[] | null = null;
|
||||
let cacheKey: string | null = null;
|
||||
@@ -29,6 +30,7 @@ function cachedMatchPattern(pattern: string, value: string): boolean {
|
||||
|
||||
export interface RtkFilterLoadDiagnostic {
|
||||
source: "project" | "global" | "builtin";
|
||||
format?: "omniroute-json" | "rtk-toml-v1";
|
||||
path?: string;
|
||||
level: "warning" | "error";
|
||||
message: string;
|
||||
@@ -38,6 +40,7 @@ interface FilterSource {
|
||||
source: "project" | "global" | "builtin";
|
||||
path: string;
|
||||
trusted: boolean;
|
||||
format: "omniroute-json" | "rtk-toml-v1";
|
||||
}
|
||||
|
||||
interface RtkFilterLoadOptions {
|
||||
@@ -95,8 +98,12 @@ function projectFiltersTrusted(
|
||||
try {
|
||||
const filtersHash = sha256(fs.readFileSync(filtersPath, "utf8"));
|
||||
const trust = JSON.parse(fs.readFileSync(trustPath, "utf8")) as Record<string, unknown>;
|
||||
const trustedHash =
|
||||
typeof trust.filtersSha256 === "string"
|
||||
const isToml = filtersPath.endsWith(".toml");
|
||||
const trustedHash = isToml
|
||||
? typeof trust.filtersTomlSha256 === "string"
|
||||
? trust.filtersTomlSha256
|
||||
: null
|
||||
: typeof trust.filtersSha256 === "string"
|
||||
? trust.filtersSha256
|
||||
: typeof trust.trustedFiltersSha256 === "string"
|
||||
? trust.trustedFiltersSha256
|
||||
@@ -110,29 +117,52 @@ function projectFiltersTrusted(
|
||||
|
||||
function collectFilterSources(options: RtkFilterLoadOptions = {}): FilterSource[] {
|
||||
const sources: FilterSource[] = [];
|
||||
const projectPath = path.join(process.cwd(), ".rtk", "filters.json");
|
||||
if (options.customFiltersEnabled !== false && fs.existsSync(projectPath)) {
|
||||
const trusted = projectFiltersTrusted(projectPath, options.trustProjectFilters === true);
|
||||
if (options.customFiltersEnabled !== false) {
|
||||
collectProjectFilterSources(sources, options);
|
||||
collectGlobalFilterSources(sources);
|
||||
}
|
||||
collectBuiltinFilterSources(sources);
|
||||
return sources;
|
||||
}
|
||||
|
||||
function collectProjectFilterSources(sources: FilterSource[], options: RtkFilterLoadOptions): void {
|
||||
const projectCandidates = [
|
||||
{ path: path.join(process.cwd(), ".rtk", "filters.toml"), format: "rtk-toml-v1" as const },
|
||||
{ path: path.join(process.cwd(), ".rtk", "filters.json"), format: "omniroute-json" as const },
|
||||
];
|
||||
for (const candidate of projectCandidates) {
|
||||
if (!fs.existsSync(candidate.path)) continue;
|
||||
const trusted = projectFiltersTrusted(candidate.path, options.trustProjectFilters === true);
|
||||
if (trusted === true) {
|
||||
sources.push({ source: "project", path: projectPath, trusted: true });
|
||||
} else {
|
||||
diagnostics.push({
|
||||
source: "project",
|
||||
path: projectPath,
|
||||
level: "warning",
|
||||
message:
|
||||
trusted === "changed"
|
||||
? "Project RTK filters changed after trust and were skipped"
|
||||
: "Project RTK filters are untrusted and were skipped",
|
||||
});
|
||||
sources.push({ source: "project", ...candidate, trusted: true });
|
||||
continue;
|
||||
}
|
||||
diagnostics.push({
|
||||
source: "project",
|
||||
format: candidate.format,
|
||||
path: candidate.path,
|
||||
level: "warning",
|
||||
message:
|
||||
trusted === "changed"
|
||||
? "Project RTK filters changed after trust and were skipped"
|
||||
: "Project RTK filters are untrusted and were skipped",
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
function collectGlobalFilterSources(sources: FilterSource[]): void {
|
||||
const globalCandidates = [
|
||||
{ path: path.join(getDataDir(), "rtk", "filters.toml"), format: "rtk-toml-v1" as const },
|
||||
{ path: path.join(getDataDir(), "rtk", "filters.json"), format: "omniroute-json" as const },
|
||||
];
|
||||
for (const candidate of globalCandidates) {
|
||||
if (fs.existsSync(candidate.path)) {
|
||||
sources.push({ source: "global", ...candidate, trusted: true });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const globalPath = path.join(getDataDir(), "rtk", "filters.json");
|
||||
if (options.customFiltersEnabled !== false && fs.existsSync(globalPath)) {
|
||||
sources.push({ source: "global", path: globalPath, trusted: true });
|
||||
}
|
||||
|
||||
function collectBuiltinFilterSources(sources: FilterSource[]): void {
|
||||
const builtinDir = getFiltersDir();
|
||||
if (fs.existsSync(builtinDir)) {
|
||||
let builtinFiles: string[] = [];
|
||||
@@ -152,25 +182,56 @@ function collectFilterSources(options: RtkFilterLoadOptions = {}): FilterSource[
|
||||
source: "builtin",
|
||||
path: path.join(builtinDir, file),
|
||||
trusted: true,
|
||||
format: "omniroute-json",
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return sources;
|
||||
}
|
||||
|
||||
function parseFilterFile(source: FilterSource): RtkFilterDefinition[] {
|
||||
try {
|
||||
const parsed = JSON.parse(fs.readFileSync(source.path, "utf8"));
|
||||
const entries = Array.isArray(parsed) ? parsed : [parsed];
|
||||
return entries.map(validateRtkFilter);
|
||||
const content = fs.readFileSync(source.path, "utf8");
|
||||
const definitions =
|
||||
source.format === "rtk-toml-v1"
|
||||
? (() => {
|
||||
const result = parseRtkTomlV1(content);
|
||||
if (!result.passed) {
|
||||
throw new Error("one or more inline tests failed");
|
||||
}
|
||||
for (const warning of result.warnings) {
|
||||
diagnostics.push({
|
||||
source: source.source,
|
||||
format: source.format,
|
||||
path: source.path,
|
||||
level: "warning",
|
||||
message: warning,
|
||||
});
|
||||
}
|
||||
return result.filters;
|
||||
})()
|
||||
: (() => {
|
||||
const parsed = JSON.parse(content);
|
||||
const entries = Array.isArray(parsed) ? parsed : [parsed];
|
||||
return entries.map(validateRtkFilter);
|
||||
})();
|
||||
return definitions.map((definition) => ({
|
||||
...definition,
|
||||
source: source.source,
|
||||
sourceFormat: definition.sourceFormat ?? source.format,
|
||||
}));
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
const message =
|
||||
error instanceof RtkTomlCompatibilityError
|
||||
? error.publicMessage
|
||||
: error instanceof Error
|
||||
? error.message
|
||||
: String(error);
|
||||
if (source.source === "builtin") {
|
||||
throw new Error(`Invalid RTK filter ${path.basename(source.path)}: ${message}`);
|
||||
}
|
||||
diagnostics.push({
|
||||
source: source.source,
|
||||
format: source.format,
|
||||
path: source.path,
|
||||
level: "warning",
|
||||
message: `Invalid custom RTK filter skipped: ${message}`,
|
||||
@@ -194,7 +255,16 @@ export function loadRtkFilters(options: RtkFilterLoadOptions = {}): RtkFilterDef
|
||||
filters.push(...parseFilterFile(source));
|
||||
}
|
||||
|
||||
const sorted = filters.sort((a, b) => b.priority - a.priority || a.id.localeCompare(b.id));
|
||||
const sourceRank = { project: 3, global: 2, builtin: 1 } as const;
|
||||
const formatRank = { "rtk-toml-v1": 2, "omniroute-json": 1 } as const;
|
||||
const sorted = filters.sort(
|
||||
(a, b) =>
|
||||
sourceRank[b.source ?? "builtin"] - sourceRank[a.source ?? "builtin"] ||
|
||||
formatRank[b.sourceFormat ?? "omniroute-json"] -
|
||||
formatRank[a.sourceFormat ?? "omniroute-json"] ||
|
||||
b.priority - a.priority ||
|
||||
a.id.localeCompare(b.id)
|
||||
);
|
||||
cache = sorted;
|
||||
cacheKey = currentCacheKey;
|
||||
return sorted;
|
||||
@@ -208,7 +278,14 @@ export function getRtkFilterLoadDiagnostics(): RtkFilterLoadDiagnostic[] {
|
||||
export function getRtkFilterCatalog(): Array<
|
||||
Pick<
|
||||
RtkFilterDefinition,
|
||||
"id" | "name" | "description" | "commandTypes" | "category" | "priority"
|
||||
| "id"
|
||||
| "name"
|
||||
| "description"
|
||||
| "commandTypes"
|
||||
| "category"
|
||||
| "priority"
|
||||
| "source"
|
||||
| "sourceFormat"
|
||||
>
|
||||
> {
|
||||
return loadRtkFilters().map((filter) => ({
|
||||
@@ -218,6 +295,8 @@ export function getRtkFilterCatalog(): Array<
|
||||
commandTypes: filter.commandTypes,
|
||||
category: filter.category,
|
||||
priority: filter.priority,
|
||||
source: filter.source,
|
||||
sourceFormat: filter.sourceFormat,
|
||||
}));
|
||||
}
|
||||
|
||||
@@ -229,17 +308,25 @@ export function matchRtkFilter(
|
||||
const detection = detectCommandType(text, command);
|
||||
const detectedCommand = detection.command ?? command ?? "";
|
||||
const filters = loadRtkFilters(options);
|
||||
return (
|
||||
filters.find((filter) => filter.commandTypes.includes(detection.type)) ??
|
||||
filters.find(
|
||||
(filter) =>
|
||||
detectedCommand &&
|
||||
filter.commandPatterns.some((pattern) => cachedMatchPattern(pattern, detectedCommand))
|
||||
) ??
|
||||
filters.find((filter) =>
|
||||
filter.matchPatterns.some((pattern) => cachedMatchPattern(pattern, text))
|
||||
) ??
|
||||
filters.find((filter) => filter.commandTypes.includes("generic-output")) ??
|
||||
null
|
||||
);
|
||||
for (const source of ["project", "global", "builtin"] as const) {
|
||||
const scoped = filters.filter((filter) => (filter.source ?? "builtin") === source);
|
||||
const matched =
|
||||
scoped.find(
|
||||
(filter) =>
|
||||
filter.sourceFormat === "rtk-toml-v1" &&
|
||||
detectedCommand &&
|
||||
filter.commandPatterns.some((pattern) => cachedMatchPattern(pattern, detectedCommand))
|
||||
) ??
|
||||
scoped.find((filter) => filter.commandTypes.includes(detection.type)) ??
|
||||
scoped.find(
|
||||
(filter) =>
|
||||
detectedCommand &&
|
||||
filter.commandPatterns.some((pattern) => cachedMatchPattern(pattern, detectedCommand))
|
||||
) ??
|
||||
scoped.find((filter) =>
|
||||
filter.matchPatterns.some((pattern) => cachedMatchPattern(pattern, text))
|
||||
);
|
||||
if (matched) return matched;
|
||||
}
|
||||
return filters.find((filter) => filter.commandTypes.includes("generic-output")) ?? null;
|
||||
}
|
||||
|
||||
@@ -137,6 +137,12 @@ export interface RtkFilterDefinition {
|
||||
maxLines: number;
|
||||
preserveHead: number;
|
||||
preserveTail: number;
|
||||
/** Exact RTK TOML schema-v1 head/tail stages. Undefined for OmniRoute-native JSON filters. */
|
||||
rtkTomlHeadLines?: number;
|
||||
rtkTomlTailLines?: number;
|
||||
rtkTomlMaxLines?: number;
|
||||
sourceFormat?: "omniroute-json" | "rtk-toml-v1";
|
||||
source?: "project" | "global" | "builtin";
|
||||
tests: Array<{ name: string; input: string; expected: string; command?: string }>;
|
||||
}
|
||||
|
||||
|
||||
@@ -54,6 +54,41 @@ function normalizeStderrPrefix(line: string): string {
|
||||
return line.replace(/^\s*(?:stderr|err)\s*(?:\||:)\s*/i, "");
|
||||
}
|
||||
|
||||
function applyRtkTomlLineLimits(
|
||||
lines: string[],
|
||||
filter: RtkFilterDefinition,
|
||||
appliedRules: string[]
|
||||
): string[] {
|
||||
const head = filter.rtkTomlHeadLines;
|
||||
const tail = filter.rtkTomlTailLines;
|
||||
const total = lines.length;
|
||||
|
||||
if (head !== undefined && tail !== undefined) {
|
||||
if (total > head + tail) {
|
||||
lines = [
|
||||
...lines.slice(0, head),
|
||||
`... (${total - head - tail} lines omitted)`,
|
||||
...(tail > 0 ? lines.slice(-tail) : []),
|
||||
];
|
||||
appliedRules.push(`${filter.id}:rtk-head-tail`);
|
||||
}
|
||||
} else if (head !== undefined && total > head) {
|
||||
lines = [...lines.slice(0, head), `... (${total - head} lines omitted)`];
|
||||
appliedRules.push(`${filter.id}:rtk-head`);
|
||||
} else if (tail !== undefined && total > tail) {
|
||||
lines = [`... (${total - tail} lines omitted)`, ...(tail > 0 ? lines.slice(-tail) : [])];
|
||||
appliedRules.push(`${filter.id}:rtk-tail`);
|
||||
}
|
||||
|
||||
const maxLines = filter.rtkTomlMaxLines;
|
||||
if (maxLines !== undefined && lines.length > maxLines) {
|
||||
const dropped = lines.length - maxLines;
|
||||
lines = [...lines.slice(0, maxLines), `... (${dropped} lines truncated)`];
|
||||
appliedRules.push(`${filter.id}:rtk-max-lines`);
|
||||
}
|
||||
return lines;
|
||||
}
|
||||
|
||||
function truncateUnicodeSafe(line: string, maxChars: number): string {
|
||||
if (maxChars <= 0) return line;
|
||||
const chars = Array.from(line);
|
||||
@@ -70,6 +105,7 @@ export function applyLineFilter(text: string, filter: RtkFilterDefinition): Line
|
||||
const appliedRules: string[] = [];
|
||||
|
||||
let lines = text.split(/\r?\n/);
|
||||
if (filter.sourceFormat === "rtk-toml-v1" && lines.at(-1) === "") lines.pop();
|
||||
const originalLineCount = lines.length;
|
||||
|
||||
if (filter.stripAnsi) {
|
||||
@@ -159,6 +195,18 @@ export function applyLineFilter(text: string, filter: RtkFilterDefinition): Line
|
||||
}
|
||||
}
|
||||
|
||||
if (filter.sourceFormat === "rtk-toml-v1") {
|
||||
lines = applyRtkTomlLineLimits(lines, filter, appliedRules);
|
||||
const output = lines.join("\n");
|
||||
const finalOutput = output.trim().length === 0 && filter.onEmpty ? filter.onEmpty : output;
|
||||
return {
|
||||
text: finalOutput,
|
||||
strippedLines: Math.max(0, originalLineCount - finalOutput.split(/\r?\n/).length),
|
||||
keptByRule: keepPatterns.length > 0,
|
||||
appliedRules,
|
||||
};
|
||||
}
|
||||
|
||||
const truncated = smartTruncate(lines.join("\n"), {
|
||||
maxLines: filter.maxLines,
|
||||
preserveHead: filter.preserveHead,
|
||||
|
||||
334
open-sse/services/compression/engines/rtk/tomlCompatibility.ts
Normal file
334
open-sse/services/compression/engines/rtk/tomlCompatibility.ts
Normal file
@@ -0,0 +1,334 @@
|
||||
import crypto from "node:crypto";
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import os from "node:os";
|
||||
import { parse as parseToml } from "smol-toml";
|
||||
import { z } from "zod";
|
||||
import { isReDoSProne, type RtkFilterDefinition } from "./filterSchema.ts";
|
||||
import { applyLineFilter } from "./lineFilter.ts";
|
||||
|
||||
const MAX_TOML_BYTES = 1024 * 1024;
|
||||
|
||||
const replaceRuleSchema = z
|
||||
.object({
|
||||
pattern: z.string().min(1),
|
||||
replacement: z.string(),
|
||||
})
|
||||
.strict();
|
||||
|
||||
const matchOutputRuleSchema = z
|
||||
.object({
|
||||
pattern: z.string().min(1),
|
||||
message: z.string(),
|
||||
unless: z.string().min(1).optional(),
|
||||
})
|
||||
.strict();
|
||||
|
||||
const filterSchema = z
|
||||
.object({
|
||||
description: z.string().optional(),
|
||||
match_command: z.string().min(1),
|
||||
strip_ansi: z.boolean().optional(),
|
||||
filter_stderr: z.boolean().optional(),
|
||||
strip_lines_matching: z.array(z.string()).optional(),
|
||||
keep_lines_matching: z.array(z.string()).optional(),
|
||||
replace: z.array(replaceRuleSchema).optional(),
|
||||
match_output: z.array(matchOutputRuleSchema).optional(),
|
||||
truncate_lines_at: z.number().int().min(0).optional(),
|
||||
head_lines: z.number().int().min(0).optional(),
|
||||
tail_lines: z.number().int().min(0).optional(),
|
||||
max_lines: z.number().int().min(0).optional(),
|
||||
on_empty: z.string().optional(),
|
||||
})
|
||||
.strict();
|
||||
|
||||
const inlineTestSchema = z
|
||||
.object({
|
||||
name: z.string().min(1),
|
||||
input: z.string(),
|
||||
expected: z.string(),
|
||||
})
|
||||
.strict();
|
||||
|
||||
const fileSchema = z
|
||||
.object({
|
||||
schema_version: z.literal(1),
|
||||
filters: z.record(z.string().min(1), filterSchema).default({}),
|
||||
tests: z.record(z.string().min(1), z.array(inlineTestSchema)).default({}),
|
||||
})
|
||||
.strict();
|
||||
|
||||
type ParsedFilter = z.infer<typeof filterSchema>;
|
||||
|
||||
function arrayOrEmpty<T>(value: T[] | undefined): T[] {
|
||||
return value ?? [];
|
||||
}
|
||||
|
||||
function numberOrZero(value: number | undefined): number {
|
||||
return value ?? 0;
|
||||
}
|
||||
|
||||
export interface RtkTomlTestOutcome {
|
||||
filterId: string;
|
||||
testName: string;
|
||||
passed: boolean;
|
||||
actual: string;
|
||||
expected: string;
|
||||
}
|
||||
|
||||
export interface RtkTomlCompatibilityResult {
|
||||
schemaVersion: 1;
|
||||
sha256: string;
|
||||
filters: RtkFilterDefinition[];
|
||||
outcomes: RtkTomlTestOutcome[];
|
||||
filtersWithoutTests: string[];
|
||||
warnings: string[];
|
||||
passed: boolean;
|
||||
}
|
||||
|
||||
export class RtkTomlCompatibilityError extends Error {
|
||||
readonly publicMessage: string;
|
||||
|
||||
constructor(message: string) {
|
||||
super("RTK TOML schema v1 compatibility error");
|
||||
this.name = "RtkTomlCompatibilityError";
|
||||
this.publicMessage = message;
|
||||
}
|
||||
}
|
||||
|
||||
function compatibilityError(message: string): RtkTomlCompatibilityError {
|
||||
return new RtkTomlCompatibilityError(message);
|
||||
}
|
||||
|
||||
function categoryFor(name: string, commandPattern: string): RtkFilterDefinition["category"] {
|
||||
const value = `${name} ${commandPattern}`.toLowerCase();
|
||||
if (/\b(?:git|gh)\b/.test(value)) return "git";
|
||||
if (/\b(?:test|jest|vitest|pytest|cargo test|go test|rspec|playwright)\b/.test(value)) {
|
||||
return "test";
|
||||
}
|
||||
if (/\b(?:build|tsc|eslint|ruff|clippy|gradle|make|next|vite|webpack)\b/.test(value)) {
|
||||
return "build";
|
||||
}
|
||||
if (/\b(?:docker|kubectl|podman|compose)\b/.test(value)) return "docker";
|
||||
if (/\b(?:npm|pnpm|yarn|bun|pip|poetry|uv|bundle|composer)\b/.test(value)) {
|
||||
return "package";
|
||||
}
|
||||
if (/\b(?:terraform|tofu|ansible|helm|pulumi)\b/.test(value)) return "infra";
|
||||
if (/\b(?:aws|gcloud|az|cloudflare)\b/.test(value)) return "cloud";
|
||||
if (/\b(?:ls|find|grep|rg|df|du|ps|systemctl|ssh|rsync)\b/.test(value)) return "shell";
|
||||
return "generic";
|
||||
}
|
||||
|
||||
function regexFields(filter: ParsedFilter): Array<{ field: string; pattern: string }> {
|
||||
return [
|
||||
{ field: "match_command", pattern: filter.match_command },
|
||||
...(filter.strip_lines_matching ?? []).map((pattern) => ({
|
||||
field: "strip_lines_matching",
|
||||
pattern,
|
||||
})),
|
||||
...(filter.keep_lines_matching ?? []).map((pattern) => ({
|
||||
field: "keep_lines_matching",
|
||||
pattern,
|
||||
})),
|
||||
...(filter.replace ?? []).map(({ pattern }) => ({ field: "replace.pattern", pattern })),
|
||||
...(filter.match_output ?? []).flatMap(({ pattern, unless }) => [
|
||||
{ field: "match_output.pattern", pattern },
|
||||
...(unless ? [{ field: "match_output.unless", pattern: unless }] : []),
|
||||
]),
|
||||
];
|
||||
}
|
||||
|
||||
function validateRegexes(name: string, filter: ParsedFilter): void {
|
||||
for (const { field, pattern } of regexFields(filter)) {
|
||||
if (isReDoSProne(pattern)) {
|
||||
throw compatibilityError(`filter '${name}' has an unsafe regex in ${field}`);
|
||||
}
|
||||
try {
|
||||
new RegExp(pattern);
|
||||
} catch {
|
||||
throw compatibilityError(`filter '${name}' has an invalid regex in ${field}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function toDefinition(
|
||||
name: string,
|
||||
filter: ParsedFilter,
|
||||
tests: z.infer<typeof inlineTestSchema>[]
|
||||
): RtkFilterDefinition {
|
||||
if (
|
||||
(filter.strip_lines_matching?.length ?? 0) > 0 &&
|
||||
(filter.keep_lines_matching?.length ?? 0) > 0
|
||||
) {
|
||||
throw compatibilityError(
|
||||
`filter '${name}' cannot combine strip_lines_matching with keep_lines_matching`
|
||||
);
|
||||
}
|
||||
validateRegexes(name, filter);
|
||||
return {
|
||||
id: name,
|
||||
name,
|
||||
description: filter.description ?? "",
|
||||
commandTypes: [],
|
||||
commandPatterns: [filter.match_command],
|
||||
matchPatterns: [],
|
||||
category: categoryFor(name, filter.match_command),
|
||||
priority: 50,
|
||||
stripPatterns: arrayOrEmpty(filter.strip_lines_matching),
|
||||
keepPatterns: arrayOrEmpty(filter.keep_lines_matching),
|
||||
priorityPatterns: [],
|
||||
collapsePatterns: [],
|
||||
stripAnsi: filter.strip_ansi ?? false,
|
||||
replace: arrayOrEmpty(filter.replace),
|
||||
matchOutput: arrayOrEmpty(filter.match_output),
|
||||
truncateLineAt: numberOrZero(filter.truncate_lines_at),
|
||||
onEmpty: filter.on_empty ?? "",
|
||||
filterStderr: false,
|
||||
deduplicate: false,
|
||||
maxLines: numberOrZero(filter.max_lines),
|
||||
preserveHead: 0,
|
||||
preserveTail: 0,
|
||||
rtkTomlHeadLines: filter.head_lines,
|
||||
rtkTomlTailLines: filter.tail_lines,
|
||||
rtkTomlMaxLines: filter.max_lines,
|
||||
sourceFormat: "rtk-toml-v1",
|
||||
tests,
|
||||
};
|
||||
}
|
||||
|
||||
function comparable(value: string): string {
|
||||
return value.replace(/\n+$/g, "");
|
||||
}
|
||||
|
||||
function tomlSyntaxLocation(error: unknown): string {
|
||||
if (typeof error !== "object" || error === null) return "";
|
||||
const { line, column } = error as { line?: unknown; column?: unknown };
|
||||
if (!Number.isSafeInteger(line) || !Number.isSafeInteger(column)) return "";
|
||||
return ` (line ${line}, column ${column})`;
|
||||
}
|
||||
|
||||
export function parseRtkTomlV1(content: string): RtkTomlCompatibilityResult {
|
||||
if (Buffer.byteLength(content, "utf8") > MAX_TOML_BYTES) {
|
||||
throw compatibilityError(`file exceeds the ${MAX_TOML_BYTES}-byte limit`);
|
||||
}
|
||||
|
||||
let raw: unknown;
|
||||
try {
|
||||
raw = parseToml(content);
|
||||
} catch (error) {
|
||||
throw compatibilityError(`invalid TOML syntax${tomlSyntaxLocation(error)}`);
|
||||
}
|
||||
|
||||
const parsed = fileSchema.safeParse(raw);
|
||||
if (!parsed.success) {
|
||||
const issue = parsed.error.issues[0];
|
||||
const field = issue?.path.length ? issue.path.join(".") : "document";
|
||||
throw compatibilityError(`${field}: ${issue?.message ?? "invalid document"}`);
|
||||
}
|
||||
if (Object.keys(parsed.data.filters).length === 0) {
|
||||
throw compatibilityError("document contains no filters");
|
||||
}
|
||||
|
||||
for (const testName of Object.keys(parsed.data.tests)) {
|
||||
if (!(testName in parsed.data.filters)) {
|
||||
throw compatibilityError(`tests reference unknown filter '${testName}'`);
|
||||
}
|
||||
}
|
||||
|
||||
const filters = Object.entries(parsed.data.filters).map(([name, filter]) =>
|
||||
toDefinition(name, filter, parsed.data.tests[name] ?? [])
|
||||
);
|
||||
const outcomes = filters.flatMap((filter) =>
|
||||
filter.tests.map((test) => {
|
||||
const actual = comparable(applyLineFilter(test.input, filter).text);
|
||||
const expected = comparable(test.expected);
|
||||
return {
|
||||
filterId: filter.id,
|
||||
testName: test.name,
|
||||
passed: actual === expected,
|
||||
actual,
|
||||
expected,
|
||||
};
|
||||
})
|
||||
);
|
||||
const filtersWithoutTests = filters
|
||||
.filter((filter) => filter.tests.length === 0)
|
||||
.map((filter) => filter.id);
|
||||
const warnings = filtersWithoutTests.map(
|
||||
(id) => `Filter '${id}' has no inline tests and should be reviewed before installation`
|
||||
);
|
||||
for (const [id, filter] of Object.entries(parsed.data.filters)) {
|
||||
if (filter.filter_stderr) {
|
||||
warnings.push(
|
||||
`Filter '${id}': filter_stderr is accepted as a no-op because OmniRoute receives already-captured tool output`
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
schemaVersion: 1,
|
||||
sha256: crypto.createHash("sha256").update(content).digest("hex"),
|
||||
filters,
|
||||
outcomes,
|
||||
filtersWithoutTests,
|
||||
warnings,
|
||||
passed: outcomes.every((outcome) => outcome.passed),
|
||||
};
|
||||
}
|
||||
|
||||
function getDataDir(): string {
|
||||
return process.env.DATA_DIR || path.join(os.homedir(), ".omniroute");
|
||||
}
|
||||
|
||||
export function getGlobalRtkTomlPath(): string {
|
||||
return path.join(getDataDir(), "rtk", "filters.toml");
|
||||
}
|
||||
|
||||
export function installGlobalRtkTomlV1(
|
||||
content: string,
|
||||
options: { overwrite?: boolean } = {}
|
||||
): RtkTomlCompatibilityResult & { installedPath: string; backupCreated: boolean } {
|
||||
const result = parseRtkTomlV1(content);
|
||||
if (!result.passed) {
|
||||
throw compatibilityError("one or more inline tests failed");
|
||||
}
|
||||
|
||||
const target = getGlobalRtkTomlPath();
|
||||
const directory = path.dirname(target);
|
||||
fs.mkdirSync(directory, { recursive: true, mode: 0o700 });
|
||||
try {
|
||||
fs.chmodSync(directory, 0o700);
|
||||
} catch {
|
||||
// Best effort on filesystems that do not support POSIX permissions.
|
||||
}
|
||||
if (fs.existsSync(target) && !options.overwrite) {
|
||||
throw compatibilityError("filters.toml already exists; confirm overwrite to replace it");
|
||||
}
|
||||
|
||||
let backupCreated = false;
|
||||
if (fs.existsSync(target)) {
|
||||
fs.copyFileSync(target, `${target}.bak`);
|
||||
try {
|
||||
fs.chmodSync(`${target}.bak`, 0o600);
|
||||
} catch {
|
||||
// Best effort on filesystems that do not support POSIX permissions.
|
||||
}
|
||||
backupCreated = true;
|
||||
}
|
||||
const temporary = `${target}.${process.pid}.${crypto.randomUUID()}.tmp`;
|
||||
try {
|
||||
fs.writeFileSync(temporary, content, { encoding: "utf8", mode: 0o600, flag: "wx" });
|
||||
fs.renameSync(temporary, target);
|
||||
try {
|
||||
fs.chmodSync(target, 0o600);
|
||||
} catch {
|
||||
// Best effort on filesystems that do not support POSIX permissions.
|
||||
}
|
||||
} finally {
|
||||
fs.rmSync(temporary, { force: true });
|
||||
}
|
||||
|
||||
return { ...result, installedPath: "rtk/filters.toml", backupCreated };
|
||||
}
|
||||
|
||||
export const RTK_TOML_MAX_BYTES = MAX_TOML_BYTES;
|
||||
5
package-lock.json
generated
5
package-lock.json
generated
@@ -74,6 +74,7 @@
|
||||
"recharts": "^3.8.1",
|
||||
"safe-regex": "^2.1.1",
|
||||
"selfsigned": "^5.5.0",
|
||||
"smol-toml": "1.6.1",
|
||||
"socks": "^2.8.7",
|
||||
"sql.js": "^1.14.1",
|
||||
"sqlite-vec": "^0.1.9",
|
||||
@@ -33779,7 +33780,6 @@
|
||||
"version": "1.6.1",
|
||||
"resolved": "https://registry.npmjs.org/smol-toml/-/smol-toml-1.6.1.tgz",
|
||||
"integrity": "sha512-dWUG8F5sIIARXih1DTaQAX4SsiTXhInKf1buxdY9DIg4ZYPZK5nGM1VRIYmEbDbsHt7USo99xSLFu5Q1IqTmsg==",
|
||||
"dev": true,
|
||||
"license": "BSD-3-Clause",
|
||||
"engines": {
|
||||
"node": ">= 18"
|
||||
@@ -37620,7 +37620,8 @@
|
||||
"version": "3.8.49",
|
||||
"dependencies": {
|
||||
"@toon-format/toon": "^2.3.0",
|
||||
"safe-regex": "^2.1.1"
|
||||
"safe-regex": "^2.1.1",
|
||||
"smol-toml": "1.6.1"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -285,6 +285,7 @@
|
||||
"recharts": "^3.8.1",
|
||||
"safe-regex": "^2.1.1",
|
||||
"selfsigned": "^5.5.0",
|
||||
"smol-toml": "1.6.1",
|
||||
"socks": "^2.8.7",
|
||||
"sql.js": "^1.14.1",
|
||||
"sqlite-vec": "^0.1.9",
|
||||
|
||||
@@ -43,6 +43,17 @@ curl https://localhost:20128/api/context/rtk/filters \
|
||||
-H "Authorization: Bearer $OMNIROUTE_TOKEN"
|
||||
```
|
||||
|
||||
### POST /api/context/rtk/import
|
||||
|
||||
Validate or install an RTK TOML schema v1 filter file
|
||||
|
||||
```bash
|
||||
curl -X POST https://localhost:20128/api/context/rtk/import \
|
||||
-H "Authorization: Bearer $OMNIROUTE_TOKEN"
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{}'
|
||||
```
|
||||
|
||||
### POST /api/context/rtk/test
|
||||
|
||||
Run RTK compression preview for text
|
||||
|
||||
@@ -4,6 +4,7 @@ import { useEffect, useMemo, useState } from "react";
|
||||
import { useTranslations } from "next-intl";
|
||||
import { SegmentedControl, Collapsible } from "@/shared/components";
|
||||
import RtkLearnDiscoverCard from "./RtkLearnDiscoverCard";
|
||||
import RtkTomlImportCard from "./RtkTomlImportCard";
|
||||
|
||||
type RtkFilter = {
|
||||
id: string;
|
||||
@@ -78,11 +79,14 @@ export default function RtkContextPageClient() {
|
||||
.catch(() => {});
|
||||
}, []);
|
||||
|
||||
useEffect(() => {
|
||||
const loadFilters = () =>
|
||||
fetch("/api/context/rtk/filters")
|
||||
.then((res) => (res.ok ? res.json() : null))
|
||||
.then((data) => setFilters(Array.isArray(data?.filters) ? data.filters : []))
|
||||
.catch(() => {});
|
||||
|
||||
useEffect(() => {
|
||||
void loadFilters();
|
||||
fetch("/api/context/rtk/config")
|
||||
.then((res) => (res.ok ? res.json() : null))
|
||||
.then((data) => setConfig(data))
|
||||
@@ -363,6 +367,8 @@ export default function RtkContextPageClient() {
|
||||
</div>
|
||||
</Collapsible>
|
||||
|
||||
{viewMode === "advanced" && <RtkTomlImportCard onInstalled={loadFilters} />}
|
||||
|
||||
<RtkLearnDiscoverCard />
|
||||
</div>
|
||||
);
|
||||
|
||||
255
src/app/(dashboard)/dashboard/context/rtk/RtkTomlImportCard.tsx
Normal file
255
src/app/(dashboard)/dashboard/context/rtk/RtkTomlImportCard.tsx
Normal file
@@ -0,0 +1,255 @@
|
||||
"use client";
|
||||
|
||||
import { useState } from "react";
|
||||
import { useTranslations } from "next-intl";
|
||||
|
||||
const RTK_TOML_MAX_BYTES = 1024 * 1024;
|
||||
|
||||
interface ImportFilterSummary {
|
||||
id: string;
|
||||
description: string;
|
||||
category: string;
|
||||
commandPatterns: string[];
|
||||
testCount: number;
|
||||
}
|
||||
|
||||
interface ImportTestOutcome {
|
||||
filterId: string;
|
||||
testName: string;
|
||||
passed: boolean;
|
||||
}
|
||||
|
||||
interface ImportResult {
|
||||
sha256: string;
|
||||
passed: boolean;
|
||||
filters: ImportFilterSummary[];
|
||||
outcomes: ImportTestOutcome[];
|
||||
warnings: string[];
|
||||
installedPath?: string;
|
||||
backupCreated?: boolean;
|
||||
}
|
||||
|
||||
interface RtkTomlImportCardProps {
|
||||
onInstalled?: () => void | Promise<void>;
|
||||
}
|
||||
|
||||
interface RtkTomlEditorProps {
|
||||
content: string;
|
||||
processing: "validate" | "install" | null;
|
||||
overwrite: boolean;
|
||||
onContentChange: (content: string) => void;
|
||||
onFileChange: (file: File | undefined) => void;
|
||||
onProcess: (action: "validate" | "install") => void;
|
||||
onOverwriteChange: (overwrite: boolean) => void;
|
||||
}
|
||||
|
||||
async function readErrorMessage(response: Response): Promise<string | null> {
|
||||
try {
|
||||
const body = (await response.json()) as { error?: { message?: unknown } };
|
||||
return typeof body.error?.message === "string" ? body.error.message : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function RtkTomlEditor({
|
||||
content,
|
||||
processing,
|
||||
overwrite,
|
||||
onContentChange,
|
||||
onFileChange,
|
||||
onProcess,
|
||||
onOverwriteChange,
|
||||
}: RtkTomlEditorProps) {
|
||||
const t = useTranslations("contextRtk");
|
||||
return (
|
||||
<>
|
||||
<div className="mt-3 flex flex-col gap-3">
|
||||
<label className="text-xs font-medium text-text-main">
|
||||
{t("tomlChooseFile")}
|
||||
<input
|
||||
type="file"
|
||||
accept=".toml,text/plain,application/toml"
|
||||
onChange={(event) => onFileChange(event.target.files?.[0])}
|
||||
data-testid="rtk-toml-file"
|
||||
className="mt-1 block w-full text-xs text-text-muted file:mr-3 file:rounded file:border file:border-border file:bg-bg file:px-2.5 file:py-1 file:text-xs file:text-text-main"
|
||||
/>
|
||||
</label>
|
||||
<textarea
|
||||
value={content}
|
||||
onChange={(event) => onContentChange(event.target.value)}
|
||||
placeholder={t("tomlImportPlaceholder")}
|
||||
data-testid="rtk-toml-content"
|
||||
className="h-56 w-full rounded-lg border border-border bg-bg p-3 font-mono text-xs text-text-main"
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div className="mt-3 flex flex-wrap items-center gap-3">
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => onProcess("validate")}
|
||||
disabled={processing !== null || !content.trim()}
|
||||
data-testid="rtk-toml-validate"
|
||||
className="rounded border border-border px-3 py-1.5 text-xs font-medium text-text-main hover:bg-surface-hover disabled:opacity-50"
|
||||
>
|
||||
{processing === "validate" ? t("tomlValidating") : t("tomlValidate")}
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => onProcess("install")}
|
||||
disabled={processing !== null || !content.trim()}
|
||||
data-testid="rtk-toml-install"
|
||||
className="rounded bg-primary px-3 py-1.5 text-xs font-medium text-white disabled:opacity-50"
|
||||
>
|
||||
{processing === "install" ? t("tomlInstalling") : t("tomlInstall")}
|
||||
</button>
|
||||
<label className="flex items-center gap-2 text-xs text-text-main">
|
||||
<input
|
||||
type="checkbox"
|
||||
checked={overwrite}
|
||||
onChange={(event) => onOverwriteChange(event.target.checked)}
|
||||
data-testid="rtk-toml-overwrite"
|
||||
/>
|
||||
{t("tomlConfirmOverwrite")}
|
||||
</label>
|
||||
</div>
|
||||
</>
|
||||
);
|
||||
}
|
||||
|
||||
function RtkTomlResult({ result }: { result: ImportResult }) {
|
||||
const t = useTranslations("contextRtk");
|
||||
return (
|
||||
<div className="mt-4 rounded-lg border border-border bg-bg p-3" data-testid="rtk-toml-result">
|
||||
<p className="text-xs font-medium text-text-main">
|
||||
{result.installedPath
|
||||
? t("tomlInstalled", { path: result.installedPath })
|
||||
: result.passed
|
||||
? t("tomlValidationPassed")
|
||||
: t("tomlValidationFailed")}
|
||||
</p>
|
||||
<p className="mt-1 text-[11px] text-text-muted">
|
||||
{t("tomlValidationSummary", {
|
||||
filters: result.filters.length,
|
||||
tests: result.outcomes.length,
|
||||
})}
|
||||
</p>
|
||||
{result.backupCreated && (
|
||||
<p className="mt-1 text-[11px] text-text-muted">{t("tomlBackupCreated")}</p>
|
||||
)}
|
||||
{result.filters.length > 0 && (
|
||||
<ul className="mt-2 space-y-1 text-[11px] text-text-main">
|
||||
{result.filters.map((filter) => (
|
||||
<li key={filter.id}>
|
||||
<code>{filter.id}</code> · {filter.category} ·{" "}
|
||||
{t("tomlTestCount", { count: filter.testCount })}
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
)}
|
||||
{result.outcomes.length > 0 && (
|
||||
<ul className="mt-2 space-y-1 text-[11px] text-text-main">
|
||||
{result.outcomes.map((outcome) => (
|
||||
<li key={`${outcome.filterId}:${outcome.testName}`}>
|
||||
<span
|
||||
className={
|
||||
outcome.passed
|
||||
? "text-emerald-700 dark:text-emerald-300"
|
||||
: "text-red-600 dark:text-red-400"
|
||||
}
|
||||
>
|
||||
{outcome.passed ? t("tomlTestPassed") : t("tomlTestFailed")}
|
||||
</span>{" "}
|
||||
· <code>{outcome.filterId}</code> · {outcome.testName}
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
)}
|
||||
{result.warnings.length > 0 && (
|
||||
<ul className="mt-2 space-y-1 text-[11px] text-amber-700 dark:text-amber-300">
|
||||
{result.warnings.map((warning) => (
|
||||
<li key={warning}>{warning}</li>
|
||||
))}
|
||||
</ul>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
export default function RtkTomlImportCard({ onInstalled }: RtkTomlImportCardProps) {
|
||||
const t = useTranslations("contextRtk");
|
||||
const [content, setContent] = useState("");
|
||||
const [result, setResult] = useState<ImportResult | null>(null);
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
const [processing, setProcessing] = useState<"validate" | "install" | null>(null);
|
||||
const [overwrite, setOverwrite] = useState(false);
|
||||
|
||||
async function processImport(action: "validate" | "install") {
|
||||
if (!content.trim()) return;
|
||||
setProcessing(action);
|
||||
setError(null);
|
||||
try {
|
||||
const response = await fetch("/api/context/rtk/import", {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ action, content, overwrite: action === "install" && overwrite }),
|
||||
});
|
||||
if (!response.ok) {
|
||||
throw new Error((await readErrorMessage(response)) ?? t("tomlImportError"));
|
||||
}
|
||||
const nextResult = (await response.json()) as ImportResult;
|
||||
setResult(nextResult);
|
||||
if (action === "install") await onInstalled?.();
|
||||
} catch (caught) {
|
||||
setResult(null);
|
||||
setError(caught instanceof Error ? caught.message : t("tomlImportError"));
|
||||
} finally {
|
||||
setProcessing(null);
|
||||
}
|
||||
}
|
||||
|
||||
async function loadFile(file: File | undefined) {
|
||||
if (!file) return;
|
||||
setError(null);
|
||||
if (file.size > RTK_TOML_MAX_BYTES) {
|
||||
setError(t("tomlFileReadError"));
|
||||
return;
|
||||
}
|
||||
try {
|
||||
setContent(await file.text());
|
||||
setResult(null);
|
||||
} catch {
|
||||
setError(t("tomlFileReadError"));
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<section
|
||||
className="rounded-lg border border-border bg-surface p-4"
|
||||
data-testid="rtk-toml-import"
|
||||
>
|
||||
<h2 className="text-sm font-semibold text-text-main">{t("tomlImportTitle")}</h2>
|
||||
<p className="mt-1 text-xs text-text-muted">{t("tomlImportDesc")}</p>
|
||||
<RtkTomlEditor
|
||||
content={content}
|
||||
processing={processing}
|
||||
overwrite={overwrite}
|
||||
onContentChange={(nextContent) => {
|
||||
setContent(nextContent);
|
||||
setResult(null);
|
||||
}}
|
||||
onFileChange={(file) => void loadFile(file)}
|
||||
onProcess={(action) => void processImport(action)}
|
||||
onOverwriteChange={setOverwrite}
|
||||
/>
|
||||
|
||||
{error && (
|
||||
<p className="mt-3 text-xs text-red-600 dark:text-red-400" data-testid="rtk-toml-error">
|
||||
{error}
|
||||
</p>
|
||||
)}
|
||||
|
||||
{result && <RtkTomlResult result={result} />}
|
||||
</section>
|
||||
);
|
||||
}
|
||||
81
src/app/api/context/rtk/import/route.ts
Normal file
81
src/app/api/context/rtk/import/route.ts
Normal file
@@ -0,0 +1,81 @@
|
||||
import { NextResponse } from "next/server";
|
||||
import { z } from "zod";
|
||||
import { requireManagementAuth } from "@/lib/api/requireManagementAuth";
|
||||
import { buildErrorBody } from "@omniroute/open-sse/utils/error";
|
||||
import { loadRtkFilters } from "@omniroute/open-sse/services/compression/engines/rtk/filterLoader";
|
||||
import {
|
||||
installGlobalRtkTomlV1,
|
||||
parseRtkTomlV1,
|
||||
RTK_TOML_MAX_BYTES,
|
||||
RtkTomlCompatibilityError,
|
||||
type RtkTomlCompatibilityResult,
|
||||
} from "@omniroute/open-sse/services/compression/engines/rtk/tomlCompatibility";
|
||||
|
||||
const RequestSchema = z
|
||||
.object({
|
||||
action: z.enum(["validate", "install"]),
|
||||
content: z.string().min(1).max(RTK_TOML_MAX_BYTES),
|
||||
overwrite: z.boolean().optional(),
|
||||
})
|
||||
.strict();
|
||||
|
||||
function responseBody(
|
||||
result: RtkTomlCompatibilityResult & {
|
||||
installedPath?: string;
|
||||
backupCreated?: boolean;
|
||||
}
|
||||
) {
|
||||
return {
|
||||
schemaVersion: result.schemaVersion,
|
||||
sha256: result.sha256,
|
||||
passed: result.passed,
|
||||
filters: result.filters.map((filter) => ({
|
||||
id: filter.id,
|
||||
description: filter.description,
|
||||
category: filter.category,
|
||||
commandPatterns: filter.commandPatterns,
|
||||
testCount: filter.tests.length,
|
||||
})),
|
||||
outcomes: result.outcomes,
|
||||
filtersWithoutTests: result.filtersWithoutTests,
|
||||
warnings: result.warnings,
|
||||
installedPath: result.installedPath,
|
||||
backupCreated: result.backupCreated,
|
||||
};
|
||||
}
|
||||
|
||||
export async function POST(request: Request) {
|
||||
const authError = await requireManagementAuth(request);
|
||||
if (authError) return authError;
|
||||
|
||||
let body: unknown;
|
||||
try {
|
||||
body = await request.json();
|
||||
} catch {
|
||||
return NextResponse.json(buildErrorBody(400, "Invalid JSON body"), { status: 400 });
|
||||
}
|
||||
const parsed = RequestSchema.safeParse(body);
|
||||
if (!parsed.success) {
|
||||
return NextResponse.json(buildErrorBody(400, "Invalid RTK TOML import request"), {
|
||||
status: 400,
|
||||
});
|
||||
}
|
||||
|
||||
try {
|
||||
const result =
|
||||
parsed.data.action === "install"
|
||||
? installGlobalRtkTomlV1(parsed.data.content, { overwrite: parsed.data.overwrite })
|
||||
: parseRtkTomlV1(parsed.data.content);
|
||||
if (parsed.data.action === "install") {
|
||||
loadRtkFilters({ refresh: true });
|
||||
}
|
||||
return NextResponse.json(responseBody(result));
|
||||
} catch (error) {
|
||||
if (error instanceof RtkTomlCompatibilityError) {
|
||||
return NextResponse.json(buildErrorBody(400, error.publicMessage), { status: 400 });
|
||||
}
|
||||
return NextResponse.json(buildErrorBody(500, "Failed to process RTK TOML filter import"), {
|
||||
status: 500,
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -5609,7 +5609,26 @@
|
||||
"searchFilters": "Suchfilter...",
|
||||
"tooltipDedup": "Wie aggressiv wiederholte Zeilen entfernt werden.",
|
||||
"tooltipMaxChars": "Maximale Zeichen pro Ausgabeblock.",
|
||||
"tooltipMaxLines": "Maximale Anzahl von Zeilen, die von der Befehlsausgabe ferngehalten werden sollen. Der Überschuss wird abgeschnitten."
|
||||
"tooltipMaxLines": "Maximale Anzahl von Zeilen, die von der Befehlsausgabe ferngehalten werden sollen. Der Überschuss wird abgeschnitten.",
|
||||
"tomlImportTitle": "RTK-TOML-Filter importieren",
|
||||
"tomlImportDesc": "Filter im RTK-TOML-Schema v1 validieren und global installieren. Die Installation schreibt DATA_DIR/rtk/filters.toml; vorhandene Dateien werden nur nach ausdrücklicher Bestätigung ersetzt.",
|
||||
"tomlChooseFile": "TOML-Datei auswählen",
|
||||
"tomlImportPlaceholder": "Inhalt einer RTK-filters.toml hier einfügen ...",
|
||||
"tomlValidate": "Validieren",
|
||||
"tomlValidating": "Wird validiert …",
|
||||
"tomlInstall": "Global installieren",
|
||||
"tomlInstalling": "Wird installiert …",
|
||||
"tomlConfirmOverwrite": "Vorhandene globale Datei ersetzen und Sicherung erstellen",
|
||||
"tomlValidationPassed": "Validierung erfolgreich",
|
||||
"tomlValidationFailed": "Validierung mit fehlgeschlagenen Inline-Tests abgeschlossen",
|
||||
"tomlValidationSummary": "{filters} Filter, {tests} Inline-Tests",
|
||||
"tomlInstalled": "Unter {path} installiert",
|
||||
"tomlBackupCreated": "Eine Sicherung der vorherigen Datei wurde erstellt.",
|
||||
"tomlTestCount": "{count} Tests",
|
||||
"tomlTestPassed": "Bestanden",
|
||||
"tomlTestFailed": "Fehlgeschlagen",
|
||||
"tomlImportError": "Die RTK-TOML-Datei konnte nicht verarbeitet werden.",
|
||||
"tomlFileReadError": "Die ausgewählte TOML-Datei konnte nicht gelesen werden."
|
||||
},
|
||||
"contextCombos": {
|
||||
"title": "Compression Combos",
|
||||
|
||||
@@ -6352,7 +6352,26 @@
|
||||
"learnButton": "Suggest filter",
|
||||
"learnEmpty": "No suggestion yet. Type a command you have run and scan.",
|
||||
"learnSamplesUsed": "Learned from {count} matching sample(s)",
|
||||
"suggestionError": "Could not load suggestions. Check management auth and try again."
|
||||
"suggestionError": "Could not load suggestions. Check management auth and try again.",
|
||||
"tomlImportTitle": "Import RTK TOML filters",
|
||||
"tomlImportDesc": "Validate RTK TOML schema v1 filters and install them globally. Installation writes DATA_DIR/rtk/filters.toml; existing files are replaced only after explicit confirmation.",
|
||||
"tomlChooseFile": "Choose a TOML file",
|
||||
"tomlImportPlaceholder": "Paste RTK filters.toml content here...",
|
||||
"tomlValidate": "Validate",
|
||||
"tomlValidating": "Validating…",
|
||||
"tomlInstall": "Install globally",
|
||||
"tomlInstalling": "Installing…",
|
||||
"tomlConfirmOverwrite": "Replace the existing global file and create a backup",
|
||||
"tomlValidationPassed": "Validation passed",
|
||||
"tomlValidationFailed": "Validation completed with failing inline tests",
|
||||
"tomlValidationSummary": "{filters} filter(s), {tests} inline test(s)",
|
||||
"tomlInstalled": "Installed at {path}",
|
||||
"tomlBackupCreated": "A backup of the previous file was created.",
|
||||
"tomlTestCount": "{count} test(s)",
|
||||
"tomlTestPassed": "Passed",
|
||||
"tomlTestFailed": "Failed",
|
||||
"tomlImportError": "Could not process the RTK TOML file.",
|
||||
"tomlFileReadError": "Could not read the selected TOML file."
|
||||
},
|
||||
"contextCombos": {
|
||||
"title": "Compression Combos",
|
||||
|
||||
@@ -6352,7 +6352,26 @@
|
||||
"learnButton": "__MISSING__:Suggest filter",
|
||||
"learnEmpty": "__MISSING__:No suggestion yet. Type a command you have run and scan.",
|
||||
"learnSamplesUsed": "__MISSING__:Learned from {count} matching sample(s)",
|
||||
"suggestionError": "__MISSING__:Could not load suggestions. Check management auth and try again."
|
||||
"suggestionError": "__MISSING__:Could not load suggestions. Check management auth and try again.",
|
||||
"tomlImportTitle": "Importar filtros TOML do RTK",
|
||||
"tomlImportDesc": "Valide os filtros do esquema TOML v1 do RTK e instale-os globalmente. A instalação grava em DATA_DIR/rtk/filters.toml; arquivos existentes só são substituídos após confirmação explícita.",
|
||||
"tomlChooseFile": "Escolher um arquivo TOML",
|
||||
"tomlImportPlaceholder": "Cole aqui o conteúdo do arquivo filters.toml do RTK...",
|
||||
"tomlValidate": "Validar",
|
||||
"tomlValidating": "Validando…",
|
||||
"tomlInstall": "Instalar globalmente",
|
||||
"tomlInstalling": "Instalando…",
|
||||
"tomlConfirmOverwrite": "Substituir o arquivo global existente e criar um backup",
|
||||
"tomlValidationPassed": "Validação bem-sucedida",
|
||||
"tomlValidationFailed": "Validação concluída com falhas nos testes embutidos",
|
||||
"tomlValidationSummary": "{filters} filtro(s), {tests} teste(s) embutido(s)",
|
||||
"tomlInstalled": "Instalado em {path}",
|
||||
"tomlBackupCreated": "Foi criado um backup do arquivo anterior.",
|
||||
"tomlTestCount": "{count} teste(s)",
|
||||
"tomlTestPassed": "Aprovado",
|
||||
"tomlTestFailed": "Reprovado",
|
||||
"tomlImportError": "Não foi possível processar o arquivo TOML do RTK.",
|
||||
"tomlFileReadError": "Não foi possível ler o arquivo TOML selecionado."
|
||||
},
|
||||
"contextCombos": {
|
||||
"title": "Combos de Compressao",
|
||||
|
||||
125
tests/unit/api/compression/rtk-toml-import-route.test.ts
Normal file
125
tests/unit/api/compression/rtk-toml-import-route.test.ts
Normal file
@@ -0,0 +1,125 @@
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import fs from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
|
||||
import { makeManagementSessionRequest } from "../../../helpers/managementSession.ts";
|
||||
|
||||
const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-rtk-toml-route-"));
|
||||
const ORIGINAL_DATA_DIR = process.env.DATA_DIR;
|
||||
const ORIGINAL_INITIAL_PASSWORD = process.env.INITIAL_PASSWORD;
|
||||
process.env.DATA_DIR = TEST_DATA_DIR;
|
||||
|
||||
const core = await import("../../../../src/lib/db/core.ts");
|
||||
const settingsDb = await import("../../../../src/lib/db/settings.ts");
|
||||
const importRoute = await import("../../../../src/app/api/context/rtk/import/route.ts");
|
||||
|
||||
const SAMPLE = `schema_version = 1
|
||||
[filters.route-test]
|
||||
description = "Route test"
|
||||
match_command = "^route-test"
|
||||
strip_lines_matching = ["^noise"]
|
||||
|
||||
[[tests.route-test]]
|
||||
name = "removes noise"
|
||||
input = "noise\\nkept"
|
||||
expected = "kept"
|
||||
`;
|
||||
|
||||
async function reset() {
|
||||
core.resetDbInstance();
|
||||
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
|
||||
fs.mkdirSync(TEST_DATA_DIR, { recursive: true });
|
||||
delete process.env.INITIAL_PASSWORD;
|
||||
}
|
||||
|
||||
async function post(body: unknown): Promise<Response> {
|
||||
return importRoute.POST(
|
||||
await makeManagementSessionRequest("http://localhost/api/context/rtk/import", {
|
||||
method: "POST",
|
||||
body,
|
||||
})
|
||||
);
|
||||
}
|
||||
|
||||
test.beforeEach(reset);
|
||||
|
||||
test.after(() => {
|
||||
core.resetDbInstance();
|
||||
fs.rmSync(TEST_DATA_DIR, { recursive: true, force: true });
|
||||
if (ORIGINAL_DATA_DIR === undefined) delete process.env.DATA_DIR;
|
||||
else process.env.DATA_DIR = ORIGINAL_DATA_DIR;
|
||||
if (ORIGINAL_INITIAL_PASSWORD === undefined) delete process.env.INITIAL_PASSWORD;
|
||||
else process.env.INITIAL_PASSWORD = ORIGINAL_INITIAL_PASSWORD;
|
||||
});
|
||||
|
||||
test("POST requires management authentication when login is configured", async () => {
|
||||
process.env.INITIAL_PASSWORD = "rtk-toml-route-password";
|
||||
await settingsDb.updateSettings({ requireLogin: true });
|
||||
|
||||
const response = await importRoute.POST(
|
||||
new Request("http://localhost/api/context/rtk/import", {
|
||||
method: "POST",
|
||||
headers: { "content-type": "application/json" },
|
||||
body: JSON.stringify({ action: "validate", content: SAMPLE }),
|
||||
})
|
||||
);
|
||||
|
||||
assert.equal(response.status, 401);
|
||||
});
|
||||
|
||||
test("validate returns tests and does not write filters.toml", async () => {
|
||||
const response = await post({ action: "validate", content: SAMPLE });
|
||||
const body = await response.json();
|
||||
|
||||
assert.equal(response.status, 200);
|
||||
assert.equal(body.passed, true);
|
||||
assert.equal(body.filters[0].id, "route-test");
|
||||
assert.equal(body.outcomes[0].passed, true);
|
||||
assert.equal(fs.existsSync(path.join(TEST_DATA_DIR, "rtk", "filters.toml")), false);
|
||||
});
|
||||
|
||||
test("install writes the global file, refuses implicit overwrite, and creates a backup", async () => {
|
||||
const first = await post({ action: "install", content: SAMPLE });
|
||||
assert.equal(first.status, 200);
|
||||
assert.equal(fs.readFileSync(path.join(TEST_DATA_DIR, "rtk", "filters.toml"), "utf8"), SAMPLE);
|
||||
|
||||
const refused = await post({ action: "install", content: SAMPLE });
|
||||
assert.equal(refused.status, 400);
|
||||
|
||||
const replacement = SAMPLE.replace("Route test", "Replacement");
|
||||
const replaced = await post({ action: "install", content: replacement, overwrite: true });
|
||||
const replacedBody = await replaced.json();
|
||||
assert.equal(replaced.status, 200);
|
||||
assert.equal(replacedBody.backupCreated, true);
|
||||
assert.equal(
|
||||
fs.readFileSync(path.join(TEST_DATA_DIR, "rtk", "filters.toml.bak"), "utf8"),
|
||||
SAMPLE
|
||||
);
|
||||
});
|
||||
|
||||
test("invalid TOML, unsafe regex, failing tests, and oversized input return safe 400 errors", async () => {
|
||||
const inputs = [
|
||||
"not = [valid",
|
||||
`schema_version = 1\n[filters.bad]\nmatch_command = "(a+)+$"`,
|
||||
`schema_version = 1
|
||||
[filters.bad]
|
||||
match_command = "^bad"
|
||||
strip_lines_matching = ["^noise"]
|
||||
[[tests.bad]]
|
||||
name = "fails"
|
||||
input = "noise\\nkept"
|
||||
expected = "wrong"`,
|
||||
"x".repeat(1024 * 1024 + 1),
|
||||
];
|
||||
|
||||
for (const content of inputs) {
|
||||
const response = await post({ action: "install", content });
|
||||
const body = await response.json();
|
||||
assert.equal(response.status, 400);
|
||||
assert.equal(typeof body.error?.message, "string");
|
||||
assert.ok(!body.error.message.includes(" at "));
|
||||
assert.ok(!body.error.message.includes(process.cwd()));
|
||||
}
|
||||
});
|
||||
@@ -36,4 +36,20 @@ describe("RTK line filters", () => {
|
||||
assert.ok(result.text.includes("On branch"));
|
||||
assert.ok(!result.text.includes("nothing added"));
|
||||
});
|
||||
|
||||
it("does not truncate within a combined RTK TOML head and tail limit", () => {
|
||||
const filter = {
|
||||
...loadRtkFilters().find((item) => item.id === "generic-output")!,
|
||||
id: "toml-head-tail",
|
||||
sourceFormat: "rtk-toml-v1" as const,
|
||||
rtkTomlHeadLines: 2,
|
||||
rtkTomlTailLines: 2,
|
||||
};
|
||||
|
||||
const result = applyLineFilter("first\nsecond\nthird", filter);
|
||||
|
||||
assert.equal(result.text, "first\nsecond\nthird");
|
||||
assert.ok(!result.appliedRules.includes("toml-head-tail:rtk-head"));
|
||||
assert.ok(!result.appliedRules.includes("toml-head-tail:rtk-head-tail"));
|
||||
});
|
||||
});
|
||||
|
||||
313
tests/unit/compression/rtk-toml-compatibility.test.ts
Normal file
313
tests/unit/compression/rtk-toml-compatibility.test.ts
Normal file
@@ -0,0 +1,313 @@
|
||||
import { afterEach, describe, it } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import crypto from "node:crypto";
|
||||
import fs from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
|
||||
import { applyLineFilter } from "../../../open-sse/services/compression/engines/rtk/lineFilter.ts";
|
||||
import {
|
||||
getRtkFilterCatalog,
|
||||
getRtkFilterLoadDiagnostics,
|
||||
loadRtkFilters,
|
||||
matchRtkFilter,
|
||||
} from "../../../open-sse/services/compression/engines/rtk/filterLoader.ts";
|
||||
import {
|
||||
installGlobalRtkTomlV1,
|
||||
parseRtkTomlV1,
|
||||
RtkTomlCompatibilityError,
|
||||
} from "../../../open-sse/services/compression/engines/rtk/tomlCompatibility.ts";
|
||||
|
||||
const originalCwd = process.cwd();
|
||||
const originalDataDir = process.env.DATA_DIR;
|
||||
|
||||
const SAMPLE = `schema_version = 1
|
||||
|
||||
[filters.my-tool]
|
||||
description = "Compact my-tool output"
|
||||
match_command = "^my-tool\\\\s+build"
|
||||
strip_ansi = true
|
||||
strip_lines_matching = ["^noise", "^\\\\s*$"]
|
||||
replace = [{ pattern = "duration: ([0-9]+)ms", replacement = "t=$1ms" }]
|
||||
match_output = [{ pattern = "all good", message = "my-tool: ok", unless = "error" }]
|
||||
truncate_lines_at = 12
|
||||
head_lines = 2
|
||||
tail_lines = 1
|
||||
max_lines = 4
|
||||
on_empty = "my-tool: empty"
|
||||
|
||||
[[tests.my-tool]]
|
||||
name = "filters and truncates"
|
||||
input = "noise banner\\nfirst line\\nsecond line\\nthird line\\nfourth line\\n"
|
||||
expected = "first line\\nsecond line\\n... (1 lines omitted)\\nfourth line"
|
||||
|
||||
[[tests.my-tool]]
|
||||
name = "short circuits success"
|
||||
input = "all good"
|
||||
expected = "my-tool: ok"
|
||||
`;
|
||||
|
||||
afterEach(() => {
|
||||
process.chdir(originalCwd);
|
||||
if (originalDataDir === undefined) delete process.env.DATA_DIR;
|
||||
else process.env.DATA_DIR = originalDataDir;
|
||||
loadRtkFilters({ refresh: true, customFiltersEnabled: false });
|
||||
});
|
||||
|
||||
describe("RTK TOML schema v1 compatibility", () => {
|
||||
it("reports a safe location for invalid TOML syntax", () => {
|
||||
assert.throws(
|
||||
() => parseRtkTomlV1("not = [valid"),
|
||||
(error: unknown) =>
|
||||
error instanceof RtkTomlCompatibilityError &&
|
||||
error.publicMessage === "invalid TOML syntax (line 1, column 8)"
|
||||
);
|
||||
});
|
||||
|
||||
it("parses the official declarative fields and passes inline tests", () => {
|
||||
const result = parseRtkTomlV1(SAMPLE);
|
||||
|
||||
assert.equal(result.schemaVersion, 1);
|
||||
assert.equal(result.filters.length, 1);
|
||||
assert.equal(result.outcomes.length, 2);
|
||||
assert.ok(result.outcomes.every((outcome) => outcome.passed));
|
||||
assert.equal(result.passed, true);
|
||||
assert.deepEqual(result.filtersWithoutTests, []);
|
||||
assert.equal(result.filters[0].sourceFormat, "rtk-toml-v1");
|
||||
});
|
||||
|
||||
it("matches imported TOML before a detected builtin command type", () => {
|
||||
const project = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-rtk-toml-project-"));
|
||||
process.chdir(project);
|
||||
const rtkDir = path.join(project, ".rtk");
|
||||
const filterPath = path.join(rtkDir, "filters.toml");
|
||||
fs.mkdirSync(rtkDir, { recursive: true });
|
||||
fs.writeFileSync(
|
||||
filterPath,
|
||||
`schema_version = 1
|
||||
[filters.git-status-custom]
|
||||
match_command = "^git\\\\s+status"
|
||||
strip_lines_matching = ["^noise"]
|
||||
`
|
||||
);
|
||||
fs.writeFileSync(
|
||||
path.join(rtkDir, "trust.json"),
|
||||
JSON.stringify({
|
||||
filtersTomlSha256: crypto
|
||||
.createHash("sha256")
|
||||
.update(fs.readFileSync(filterPath))
|
||||
.digest("hex"),
|
||||
})
|
||||
);
|
||||
|
||||
const filter = matchRtkFilter("noise\n M file.ts", "git status", { refresh: true });
|
||||
|
||||
assert.equal(filter?.id, "git-status-custom");
|
||||
assert.equal(filter?.source, "project");
|
||||
});
|
||||
|
||||
it("requires a separate trust hash for project TOML files", () => {
|
||||
const project = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-rtk-toml-project-"));
|
||||
process.chdir(project);
|
||||
const rtkDir = path.join(project, ".rtk");
|
||||
fs.mkdirSync(rtkDir, { recursive: true });
|
||||
fs.writeFileSync(path.join(rtkDir, "filters.toml"), SAMPLE);
|
||||
fs.writeFileSync(
|
||||
path.join(rtkDir, "trust.json"),
|
||||
JSON.stringify({ filtersSha256: "wrong-key" })
|
||||
);
|
||||
|
||||
const filters = loadRtkFilters({ refresh: true });
|
||||
const diagnostics = getRtkFilterLoadDiagnostics();
|
||||
|
||||
assert.ok(!filters.some((filter) => filter.id === "my-tool"));
|
||||
assert.ok(
|
||||
diagnostics.some(
|
||||
(entry) => entry.format === "rtk-toml-v1" && entry.message.includes("untrusted")
|
||||
)
|
||||
);
|
||||
});
|
||||
|
||||
it("loads global TOML and exposes its source format in the catalog", () => {
|
||||
const project = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-rtk-toml-project-"));
|
||||
const dataDir = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-rtk-toml-data-"));
|
||||
process.chdir(project);
|
||||
process.env.DATA_DIR = dataDir;
|
||||
fs.mkdirSync(path.join(dataDir, "rtk"), { recursive: true });
|
||||
fs.writeFileSync(path.join(dataDir, "rtk", "filters.toml"), SAMPLE);
|
||||
|
||||
loadRtkFilters({ refresh: true });
|
||||
const entry = getRtkFilterCatalog().find((filter) => filter.id === "my-tool");
|
||||
|
||||
assert.equal(entry?.source, "global");
|
||||
assert.equal(entry?.sourceFormat, "rtk-toml-v1");
|
||||
});
|
||||
|
||||
it("prioritizes project TOML over a higher-priority global JSON match", () => {
|
||||
const project = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-rtk-toml-project-"));
|
||||
const dataDir = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-rtk-toml-data-"));
|
||||
process.chdir(project);
|
||||
process.env.DATA_DIR = dataDir;
|
||||
|
||||
const projectDir = path.join(project, ".rtk");
|
||||
fs.mkdirSync(projectDir, { recursive: true });
|
||||
const projectPath = path.join(projectDir, "filters.toml");
|
||||
fs.writeFileSync(
|
||||
projectPath,
|
||||
`schema_version = 1
|
||||
[filters.project-match]
|
||||
match_command = "^scope-tool"
|
||||
`
|
||||
);
|
||||
fs.writeFileSync(
|
||||
path.join(projectDir, "trust.json"),
|
||||
JSON.stringify({
|
||||
filtersTomlSha256: crypto
|
||||
.createHash("sha256")
|
||||
.update(fs.readFileSync(projectPath))
|
||||
.digest("hex"),
|
||||
})
|
||||
);
|
||||
|
||||
fs.mkdirSync(path.join(dataDir, "rtk"), { recursive: true });
|
||||
fs.writeFileSync(
|
||||
path.join(dataDir, "rtk", "filters.json"),
|
||||
JSON.stringify({
|
||||
id: "global-match",
|
||||
label: "Global match",
|
||||
description: "",
|
||||
category: "generic",
|
||||
priority: 100,
|
||||
match: { commands: ["^scope-tool"], patterns: [], outputTypes: [] },
|
||||
})
|
||||
);
|
||||
|
||||
const filter = matchRtkFilter("output", "scope-tool", { refresh: true });
|
||||
assert.equal(filter?.id, "project-match");
|
||||
});
|
||||
|
||||
it("keeps project JSON ahead of a matching global TOML filter", () => {
|
||||
const project = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-rtk-toml-project-"));
|
||||
const dataDir = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-rtk-toml-data-"));
|
||||
process.chdir(project);
|
||||
process.env.DATA_DIR = dataDir;
|
||||
|
||||
const projectDir = path.join(project, ".rtk");
|
||||
fs.mkdirSync(projectDir, { recursive: true });
|
||||
const projectPath = path.join(projectDir, "filters.json");
|
||||
fs.writeFileSync(
|
||||
projectPath,
|
||||
JSON.stringify({
|
||||
id: "project-json-match",
|
||||
label: "Project JSON match",
|
||||
description: "",
|
||||
category: "generic",
|
||||
match: { commands: ["^scope-tool"], patterns: [], outputTypes: [] },
|
||||
rules: {},
|
||||
preserve: { errorPatterns: [], summaryPatterns: [] },
|
||||
tests: [],
|
||||
})
|
||||
);
|
||||
fs.writeFileSync(
|
||||
path.join(projectDir, "trust.json"),
|
||||
JSON.stringify({
|
||||
filtersSha256: crypto
|
||||
.createHash("sha256")
|
||||
.update(fs.readFileSync(projectPath))
|
||||
.digest("hex"),
|
||||
})
|
||||
);
|
||||
|
||||
fs.mkdirSync(path.join(dataDir, "rtk"), { recursive: true });
|
||||
fs.writeFileSync(
|
||||
path.join(dataDir, "rtk", "filters.toml"),
|
||||
`schema_version = 1
|
||||
[filters.global-toml-match]
|
||||
match_command = "^scope-tool"
|
||||
`
|
||||
);
|
||||
|
||||
const filter = matchRtkFilter("output", "scope-tool", { refresh: true });
|
||||
assert.equal(filter?.id, "project-json-match");
|
||||
});
|
||||
|
||||
it("installs atomically, protects the file and refuses an implicit overwrite", () => {
|
||||
const dataDir = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-rtk-toml-data-"));
|
||||
process.env.DATA_DIR = dataDir;
|
||||
|
||||
const installed = installGlobalRtkTomlV1(SAMPLE);
|
||||
const target = path.join(dataDir, "rtk", "filters.toml");
|
||||
|
||||
assert.equal(installed.installedPath, "rtk/filters.toml");
|
||||
assert.equal(fs.readFileSync(target, "utf8"), SAMPLE);
|
||||
assert.equal(fs.statSync(target).mode & 0o777, 0o600);
|
||||
assert.throws(
|
||||
() => installGlobalRtkTomlV1(SAMPLE),
|
||||
(error: unknown) =>
|
||||
error instanceof RtkTomlCompatibilityError && error.publicMessage.includes("already exists")
|
||||
);
|
||||
});
|
||||
|
||||
it("rejects unknown fields, unsafe regexes and failing inline tests", () => {
|
||||
assert.throws(() =>
|
||||
parseRtkTomlV1(`schema_version = 1
|
||||
[filters.bad]
|
||||
match_command = "^bad"
|
||||
run = "rm -rf /"
|
||||
`)
|
||||
);
|
||||
assert.throws(() =>
|
||||
parseRtkTomlV1(`schema_version = 1
|
||||
[filters.bad]
|
||||
match_command = "(a+)+$"
|
||||
`)
|
||||
);
|
||||
const failed = parseRtkTomlV1(`schema_version = 1
|
||||
[filters.bad]
|
||||
match_command = "^bad"
|
||||
strip_lines_matching = ["^noise"]
|
||||
[[tests.bad]]
|
||||
name = "wrong"
|
||||
input = "noise\\nkept"
|
||||
expected = "different"
|
||||
`);
|
||||
assert.equal(failed.passed, false);
|
||||
assert.throws(() =>
|
||||
installGlobalRtkTomlV1(`schema_version = 1
|
||||
[filters.bad]
|
||||
match_command = "^bad"
|
||||
strip_lines_matching = ["^noise"]
|
||||
[[tests.bad]]
|
||||
name = "wrong"
|
||||
input = "noise\\nkept"
|
||||
expected = "different"
|
||||
`)
|
||||
);
|
||||
});
|
||||
|
||||
it("keeps filter_stderr as a documented proxy no-op", () => {
|
||||
const result = parseRtkTomlV1(`schema_version = 1
|
||||
[filters.stderr-tool]
|
||||
match_command = "^stderr-tool"
|
||||
filter_stderr = true
|
||||
`);
|
||||
|
||||
assert.equal(result.filters[0].filterStderr, false);
|
||||
assert.ok(result.warnings.some((warning) => warning.includes("filter_stderr")));
|
||||
assert.equal(applyLineFilter("stderr: error", result.filters[0]).text, "stderr: error");
|
||||
});
|
||||
|
||||
it("rejects non-schema fields in inline tests", () => {
|
||||
assert.throws(() =>
|
||||
parseRtkTomlV1(`schema_version = 1
|
||||
[filters.bad]
|
||||
match_command = "^bad"
|
||||
[[tests.bad]]
|
||||
name = "extra field"
|
||||
input = "input"
|
||||
expected = "input"
|
||||
command = "bad"
|
||||
`)
|
||||
);
|
||||
});
|
||||
});
|
||||
179
tests/unit/ui/rtkTomlImportCard.test.tsx
Normal file
179
tests/unit/ui/rtkTomlImportCard.test.tsx
Normal file
@@ -0,0 +1,179 @@
|
||||
// @vitest-environment jsdom
|
||||
import React, { act } from "react";
|
||||
import { createRoot } from "react-dom/client";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
vi.mock("next-intl", () => ({
|
||||
useTranslations: () => (key: string, params?: Record<string, unknown>) =>
|
||||
params ? `${key}:${JSON.stringify(params)}` : key,
|
||||
}));
|
||||
|
||||
const { default: RtkTomlImportCard } =
|
||||
await import("@/app/(dashboard)/dashboard/context/rtk/RtkTomlImportCard");
|
||||
|
||||
const containers: HTMLElement[] = [];
|
||||
|
||||
function mount(ui: React.ReactElement): HTMLElement {
|
||||
const container = document.createElement("div");
|
||||
document.body.appendChild(container);
|
||||
containers.push(container);
|
||||
act(() => createRoot(container).render(ui));
|
||||
return container;
|
||||
}
|
||||
|
||||
async function setTextarea(container: HTMLElement, value: string) {
|
||||
const input = container.querySelector("[data-testid='rtk-toml-content']") as HTMLTextAreaElement;
|
||||
await act(async () => {
|
||||
const setter = Object.getOwnPropertyDescriptor(
|
||||
window.HTMLTextAreaElement.prototype,
|
||||
"value"
|
||||
)!.set!;
|
||||
setter.call(input, value);
|
||||
input.dispatchEvent(new Event("input", { bubbles: true }));
|
||||
});
|
||||
}
|
||||
|
||||
async function click(element: Element | null) {
|
||||
await act(async () => (element as HTMLElement).click());
|
||||
}
|
||||
|
||||
async function selectFile(container: HTMLElement, file: File) {
|
||||
const input = container.querySelector("[data-testid='rtk-toml-file']") as HTMLInputElement;
|
||||
Object.defineProperty(input, "files", { configurable: true, value: [file] });
|
||||
await act(async () => input.dispatchEvent(new Event("change", { bubbles: true })));
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
(
|
||||
globalThis as typeof globalThis & { IS_REACT_ACT_ENVIRONMENT?: boolean }
|
||||
).IS_REACT_ACT_ENVIRONMENT = true;
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
while (containers.length) containers.pop()?.remove();
|
||||
document.body.innerHTML = "";
|
||||
vi.restoreAllMocks();
|
||||
vi.unstubAllGlobals();
|
||||
});
|
||||
|
||||
describe("RtkTomlImportCard", () => {
|
||||
it("validates TOML without requesting installation", async () => {
|
||||
const fetchMock = vi.fn(async () =>
|
||||
Response.json({
|
||||
sha256: "abc",
|
||||
passed: true,
|
||||
filters: [
|
||||
{
|
||||
id: "my-tool",
|
||||
description: "",
|
||||
category: "generic",
|
||||
commandPatterns: ["^my-tool"],
|
||||
testCount: 1,
|
||||
},
|
||||
],
|
||||
outcomes: [{ filterId: "my-tool", testName: "works", passed: true }],
|
||||
warnings: [],
|
||||
})
|
||||
);
|
||||
vi.stubGlobal("fetch", fetchMock);
|
||||
const container = mount(<RtkTomlImportCard />);
|
||||
|
||||
await setTextarea(container, "schema_version = 1");
|
||||
await click(container.querySelector("[data-testid='rtk-toml-validate']"));
|
||||
|
||||
const [, options] = fetchMock.mock.calls[0];
|
||||
expect(JSON.parse(String(options?.body))).toMatchObject({
|
||||
action: "validate",
|
||||
overwrite: false,
|
||||
});
|
||||
expect(container.querySelector("[data-testid='rtk-toml-result']")?.textContent).toContain(
|
||||
"my-tool"
|
||||
);
|
||||
});
|
||||
|
||||
it("installs with explicit overwrite and refreshes the filter catalog", async () => {
|
||||
const onInstalled = vi.fn(async () => {});
|
||||
const fetchMock = vi.fn(async () =>
|
||||
Response.json({
|
||||
sha256: "abc",
|
||||
passed: true,
|
||||
filters: [],
|
||||
outcomes: [],
|
||||
warnings: [],
|
||||
installedPath: "rtk/filters.toml",
|
||||
backupCreated: true,
|
||||
})
|
||||
);
|
||||
vi.stubGlobal("fetch", fetchMock);
|
||||
const container = mount(<RtkTomlImportCard onInstalled={onInstalled} />);
|
||||
|
||||
await setTextarea(container, "schema_version = 1");
|
||||
await click(container.querySelector("[data-testid='rtk-toml-overwrite']"));
|
||||
await click(container.querySelector("[data-testid='rtk-toml-install']"));
|
||||
|
||||
const [, options] = fetchMock.mock.calls[0];
|
||||
expect(JSON.parse(String(options?.body))).toMatchObject({ action: "install", overwrite: true });
|
||||
expect(onInstalled).toHaveBeenCalledOnce();
|
||||
expect(container.querySelector("[data-testid='rtk-toml-result']")?.textContent).toContain(
|
||||
"tomlBackupCreated"
|
||||
);
|
||||
});
|
||||
|
||||
it("shows the sanitized API error message", async () => {
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn(async () =>
|
||||
Response.json({ error: { message: "filter 'bad' has an unsafe regex" } }, { status: 400 })
|
||||
)
|
||||
);
|
||||
const container = mount(<RtkTomlImportCard />);
|
||||
|
||||
await setTextarea(container, "schema_version = 1");
|
||||
await click(container.querySelector("[data-testid='rtk-toml-validate']"));
|
||||
|
||||
expect(container.querySelector("[data-testid='rtk-toml-error']")?.textContent).toContain(
|
||||
"unsafe regex"
|
||||
);
|
||||
});
|
||||
|
||||
it("shows failing inline-test outcomes without reporting a successful validation", async () => {
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn(async () =>
|
||||
Response.json({
|
||||
sha256: "abc",
|
||||
passed: false,
|
||||
filters: [],
|
||||
outcomes: [{ filterId: "bad", testName: "expected output", passed: false }],
|
||||
warnings: [],
|
||||
})
|
||||
)
|
||||
);
|
||||
const container = mount(<RtkTomlImportCard />);
|
||||
|
||||
await setTextarea(container, "schema_version = 1");
|
||||
await click(container.querySelector("[data-testid='rtk-toml-validate']"));
|
||||
|
||||
const result = container.querySelector("[data-testid='rtk-toml-result']")?.textContent ?? "";
|
||||
expect(result).toContain("tomlValidationFailed");
|
||||
expect(result).toContain("tomlTestFailed");
|
||||
expect(result).not.toContain("tomlValidationPassed");
|
||||
});
|
||||
|
||||
it("rejects oversized files before reading them", async () => {
|
||||
const container = mount(<RtkTomlImportCard />);
|
||||
await setTextarea(container, "existing content");
|
||||
const file = new File([new Uint8Array(1024 * 1024 + 1)], "filters.toml");
|
||||
const text = vi.spyOn(file, "text");
|
||||
|
||||
await selectFile(container, file);
|
||||
|
||||
expect(text).not.toHaveBeenCalled();
|
||||
expect(container.querySelector("[data-testid='rtk-toml-error']")?.textContent).toContain(
|
||||
"tomlFileReadError"
|
||||
);
|
||||
expect(
|
||||
(container.querySelector("[data-testid='rtk-toml-content']") as HTMLTextAreaElement).value
|
||||
).toBe("existing content");
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user