mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-07-26 09:52:11 +03:00
* chore(ci): add .mergify.yml to main — Mergify only reads config from the default branch (#7168)
* feat: scaffold issue agent and router eval provenance
* feat: wire recorded issue triage runner
* feat: ingest recorded issue context
* feat: persist issue agent audit log
* feat: import recorded github issue exports
* docs: document issue agent env toggle
* fix(issue-agent): validate run requests
* fix(ci): add the auto-enqueue pull_request_rule to the Mergify config (queue_conditions alone are eligibility-only) (#7179)
* fix: validate issue agent run requests
* docs: add issue agent execution traceability
* feat(issue-agent): route recorded triage through chat
* test(issue-agent): verify recorded triage through chat route
* docs(issue-agent): add executable triage session artifacts
* fix(ci): migrate Mergify auto-enqueue to merge_protections_settings.auto_merge_conditions (rules-based path is EOL 2026-07-16) (#7216)
* fix(ci): drop Mergify batch settings (batching is a paid-tier feature; free plan queue is serial) (#7220)
* feat(issue-agent): surface RecordedTriageTimeoutError as 504
When the recorded-triage chat completion times out, the AbortController
fires an AbortError that previously surfaced as a generic 400 to the
caller. This change:
* Adds a `RecordedTriageTimeoutError` that wraps the AbortError
with the timeoutMs context.
* Re-throws it from `executeRecordedTriageChatCompletion` so the
caller can distinguish timeouts from other failures.
* In the runs route, catches it and returns a 504 with code
`ISSUE_AGENT_TIMEOUT` so clients can render a useful error.
Tests:
* issue-agent-execution.test.ts — verifies the typed error
* issue-agent-route-execution.test.ts — covers timeout path
* issue-agent-runs-route.test.ts — verifies 504 mapping
* fix(ci): merge queue tolerates the advisory dast-smoke failure (its GH-hosted build hang dequeued every attempt) (#7225)
* test(ci): make the #6634 selfref guard hermetic — main's copy hard-fails every PR (#7341)
main's copy of this test still does git I/O inside a unit test:
const baseSrc = git(['show', 'origin/main:' + FILE]);
Runners check out a shallow single ref, so origin/main does not resolve and the
test dies with 'fatal: invalid object name origin/main'. Every PR into main
fails Unit Tests (7/8) on it — today that is #7313, #7315, #7316, #7334, #7336
and #7337, six PRs red on a defect none of them introduced. #7313 has no other
red at all.
release/v3.8.49 already carries a fix (2e42b8efc, #7174: try/catch, fetch
origin/main on demand, t.skip() when unreachable), but it only reaches main at
release time — so main stays broken for the whole cycle. Cherry-picking it would
also import a new problem: PR Test Policy classifies t.skip() as a silenced
assertion, which we watched it correctly catch on #7300 today.
This is the hermetic version instead (ported from #7327, which does the same for
the release branch): read the file straight off disk, compare against an empty
base so baseTaut/baseExtTaut are 0 — the strictest possible comparison point —
and call evaluateMasking() directly. No git ref, no fetch, no skip, nothing the
runner's checkout depth can break.
The #6634 regression stays covered: the guard's logic lives in
SELF_TEST_FIXTURE_RE (check-test-masking.mjs:337), not in the test. Proven both
ways on main before committing — neutralise SELF_TEST_FIXTURE_RE to /$^/ and
the test FAILS; restore it and it passes 2/2, with check-test-masking.mjs left
byte-identical.
Co-authored-by: growab <nekron@icloud.com>
* chore(quality): tighten main's coverage baseline to the CI's real numbers (#7347)
main's ratchet had been failing --require-tighten on every PR: 11 metrics
improved but the baseline was never tightened. Same class as the #6634
selfref guard — an infra fix that lands only on the release branch leaves
main red for the whole cycle, and every PR into main pays for it.
Values are the merged-coverage numbers from a run on main itself (a local
run measures ~68% vs CI's ~80%; the baseline's own note warns about that
gap). Only the 11 coverage values change — gitleaks and semgrepFindings
keep main's own state.
No changelog fragment: #7326 carries it on release/v3.8.49, and a second
one here would double the entry at release time.
* fix(issue-agent): dryRun-explicit test fixture + sanitize the generic error catch
Two independent Hard Rule #12/#18 fixes on the recorded-triage runs route:
1. tests/unit/issue-agent-route-execution.test.ts's "preserves the normal
chat route provider failure response" test omitted dryRun from its
request body. createRecordedTriageRun() computes `dryRun: input.dryRun
!== false`, so an omitted dryRun defaults to true (dry-run mode) and the
route returns the deterministic dry-run summary WITHOUT ever calling
executeRecordedTriageChatCompletion() -- the mocked 429 fetch was never
invoked, so the test always observed the 200 dry-run response instead.
Add the missing `dryRun: false`, matching the sibling test above it. The
normal chat-completions route also enriches upstream errors with a
"[provider/model] [status]:" prefix and a connection-cooldown hint
(RESILIENCE_GUIDE.md) rather than passing them through byte-for-byte, so
the assertion now checks the original message survives (status +
substring) instead of exact-matching the mocked JSON shape.
2. src/app/api/issue-agent/runs/route.ts's generic (non-timeout) catch
returned raw `error.message` straight to the client. Validation failures
thrown by this module (bad issue URL, malformed GitHub export) are safe,
curated messages -- but appendIssueAgentAuditRecord()'s mkdir/appendFile
under DATA_DIR can throw a real Node fs error (ENOENT/EACCES/EEXIST, ...)
whose raw `.message` embeds the server's absolute filesystem path. Add
isNodeSystemError() (keys off NodeJS.ErrnoException's `.code`, which only
Node's own fs/system errors set) to replace that class of error with a
generic message, and route everything else through sanitizeErrorMessage()
per Hard Rule #12. Add a regression test that forces a real audit-write
failure (pre-creating a file where audit.ts expects to mkdir) and asserts
the response contains neither the errno code nor the DATA_DIR path.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: Diego Rodrigues de Sa e Souza <8016841+diegosouzapw@users.noreply.github.com>
Co-authored-by: KooshaPari <koosha@example.com>
Co-authored-by: growab <nekron@icloud.com>
286 lines
11 KiB
TypeScript
286 lines
11 KiB
TypeScript
import test from "node:test";
|
|
import assert from "node:assert/strict";
|
|
import { mkdtempSync, writeFileSync } from "node:fs";
|
|
import { tmpdir } from "node:os";
|
|
import { join } from "node:path";
|
|
|
|
// Bun evaluates test files in one process. Set DATA_DIR before importing the
|
|
// route so its import-time dependencies never probe the developer's real DB.
|
|
const TEST_DATA_DIR =
|
|
process.env.DATA_DIR ?? mkdtempSync(join(tmpdir(), "issue-agent-runs-route-"));
|
|
process.env.DATA_DIR = TEST_DATA_DIR;
|
|
process.env.APP_LOG_TO_FILE = "false";
|
|
|
|
const { GET, POST } = await import("../../src/app/api/issue-agent/runs/route.ts");
|
|
|
|
async function json(response: Response): Promise<Record<string, unknown>> {
|
|
return (await response.json()) as Record<string, unknown>;
|
|
}
|
|
|
|
test("issue-agent status reports default-off recorded triage support", async () => {
|
|
const previous = process.env.OMNIROUTE_ISSUE_AGENT_ENABLED;
|
|
delete process.env.OMNIROUTE_ISSUE_AGENT_ENABLED;
|
|
try {
|
|
const response = await GET();
|
|
const body = await json(response);
|
|
|
|
assert.equal(response.status, 200);
|
|
assert.equal(body.ok, true);
|
|
assert.equal(body.enabled, false);
|
|
assert.deepEqual(body.supportedModes, ["recorded-triage"]);
|
|
} finally {
|
|
if (previous === undefined) delete process.env.OMNIROUTE_ISSUE_AGENT_ENABLED;
|
|
else process.env.OMNIROUTE_ISSUE_AGENT_ENABLED = previous;
|
|
}
|
|
});
|
|
|
|
test("issue-agent run rejects invalid JSON", async () => {
|
|
const response = await POST(
|
|
new Request("http://localhost/api/issue-agent/runs", {
|
|
method: "POST",
|
|
body: "{",
|
|
})
|
|
);
|
|
const body = await json(response);
|
|
|
|
assert.equal(response.status, 400);
|
|
assert.equal(body.error, "Invalid JSON body");
|
|
});
|
|
|
|
test("issue-agent run rejects invalid request field types", async () => {
|
|
const response = await POST(
|
|
new Request("http://localhost/api/issue-agent/runs", {
|
|
method: "POST",
|
|
body: JSON.stringify({ mode: 42 }),
|
|
})
|
|
);
|
|
|
|
assert.equal(response.status, 400);
|
|
});
|
|
|
|
test("issue-agent run accepts only recorded-triage mode", async () => {
|
|
const response = await POST(
|
|
new Request("http://localhost/api/issue-agent/runs", {
|
|
method: "POST",
|
|
body: JSON.stringify({ mode: "live-shell" }),
|
|
})
|
|
);
|
|
const body = await json(response);
|
|
|
|
assert.equal(response.status, 400);
|
|
assert.equal(body.error, "Unsupported issue-agent mode");
|
|
});
|
|
|
|
test("issue-agent run rejects invalid recorded-triage field types", async () => {
|
|
const response = await POST(
|
|
new Request("http://localhost/api/issue-agent/runs", {
|
|
method: "POST",
|
|
body: JSON.stringify({ mode: "recorded-triage", dryRun: "true" }),
|
|
})
|
|
);
|
|
const body = await json(response);
|
|
const error = body.error as Record<string, unknown>;
|
|
const details = error.details as Array<Record<string, unknown>>;
|
|
|
|
assert.equal(response.status, 400);
|
|
assert.equal(error.message, "Invalid request");
|
|
assert.deepEqual(details, [
|
|
{ field: "dryRun", message: "Invalid input: expected boolean, received string" },
|
|
]);
|
|
});
|
|
|
|
test("issue-agent run is disabled by default", async () => {
|
|
const previous = process.env.OMNIROUTE_ISSUE_AGENT_ENABLED;
|
|
delete process.env.OMNIROUTE_ISSUE_AGENT_ENABLED;
|
|
try {
|
|
const response = await POST(
|
|
new Request("http://localhost/api/issue-agent/runs", {
|
|
method: "POST",
|
|
body: JSON.stringify({
|
|
mode: "recorded-triage",
|
|
issueUrl: "https://github.com/x/y/issues/1",
|
|
}),
|
|
})
|
|
);
|
|
const body = await json(response);
|
|
|
|
assert.equal(response.status, 403);
|
|
assert.equal(body.enabled, false);
|
|
assert.equal(body.requiredEnv, "OMNIROUTE_ISSUE_AGENT_ENABLED=true");
|
|
} finally {
|
|
if (previous === undefined) delete process.env.OMNIROUTE_ISSUE_AGENT_ENABLED;
|
|
else process.env.OMNIROUTE_ISSUE_AGENT_ENABLED = previous;
|
|
}
|
|
});
|
|
|
|
test("issue-agent run returns deterministic recorded-triage plan when enabled", async () => {
|
|
const previous = process.env.OMNIROUTE_ISSUE_AGENT_ENABLED;
|
|
process.env.OMNIROUTE_ISSUE_AGENT_ENABLED = "true";
|
|
try {
|
|
const response = await POST(
|
|
new Request("http://localhost/api/issue-agent/runs", {
|
|
method: "POST",
|
|
body: JSON.stringify({
|
|
mode: "recorded-triage",
|
|
issueUrl: "https://github.com/KooshaPari/OmniRoute/issues/6059",
|
|
dryRun: true,
|
|
}),
|
|
})
|
|
);
|
|
const body = await json(response);
|
|
|
|
assert.equal(response.status, 200);
|
|
assert.equal(body.accepted, true);
|
|
assert.equal(body.mode, "recorded-triage");
|
|
assert.equal(body.repository, "KooshaPari/OmniRoute");
|
|
assert.equal(body.issueNumber, 6059);
|
|
assert.match(String(body.runId), /^issue-agent-recorded-triage-[a-f0-9]{16}$/);
|
|
} finally {
|
|
if (previous === undefined) delete process.env.OMNIROUTE_ISSUE_AGENT_ENABLED;
|
|
else process.env.OMNIROUTE_ISSUE_AGENT_ENABLED = previous;
|
|
}
|
|
});
|
|
|
|
test("issue-agent run rejects invalid enabled recorded-triage URL", async () => {
|
|
const previous = process.env.OMNIROUTE_ISSUE_AGENT_ENABLED;
|
|
process.env.OMNIROUTE_ISSUE_AGENT_ENABLED = "true";
|
|
try {
|
|
const response = await POST(
|
|
new Request("http://localhost/api/issue-agent/runs", {
|
|
method: "POST",
|
|
body: JSON.stringify({ mode: "recorded-triage", issueUrl: "https://example.com/nope" }),
|
|
})
|
|
);
|
|
const body = await json(response);
|
|
|
|
assert.equal(response.status, 400);
|
|
assert.equal(body.error, "Expected a GitHub issue or pull request URL");
|
|
} finally {
|
|
if (previous === undefined) delete process.env.OMNIROUTE_ISSUE_AGENT_ENABLED;
|
|
else process.env.OMNIROUTE_ISSUE_AGENT_ENABLED = previous;
|
|
}
|
|
});
|
|
|
|
test("issue-agent run returns recorded context summary with redaction", async () => {
|
|
const previous = process.env.OMNIROUTE_ISSUE_AGENT_ENABLED;
|
|
const previousDataDir = process.env.DATA_DIR;
|
|
process.env.OMNIROUTE_ISSUE_AGENT_ENABLED = "true";
|
|
process.env.DATA_DIR = mkdtempSync(join(tmpdir(), "issue-agent-route-"));
|
|
try {
|
|
const response = await POST(
|
|
new Request("http://localhost/api/issue-agent/runs", {
|
|
method: "POST",
|
|
body: JSON.stringify({
|
|
mode: "recorded-triage",
|
|
issueUrl: "https://github.com/KooshaPari/OmniRoute/issues/7",
|
|
recordedContext: {
|
|
title: "Review PR mention",
|
|
body: "Authorization: Bearer sk-routeSecret1234567890",
|
|
comments: [{ author: "maintainer", body: "please review", isBot: false }],
|
|
},
|
|
}),
|
|
})
|
|
);
|
|
const body = await json(response);
|
|
const context = body.context as Record<string, unknown>;
|
|
|
|
assert.equal(response.status, 200);
|
|
assert.equal(body.auditPath, join(process.env.DATA_DIR, "issue-agent", "audit.jsonl"));
|
|
assert.equal(context.issueTitle, "Review PR mention");
|
|
assert.equal(context.intent, "review");
|
|
assert.equal(context.humanCommentCount, 1);
|
|
assert.doesNotMatch(String(context.redactedDigestSource), /sk-routeSecret/);
|
|
assert.match(String(context.redactedDigestSource), /\[REDACTED\]/);
|
|
} finally {
|
|
if (previous === undefined) delete process.env.OMNIROUTE_ISSUE_AGENT_ENABLED;
|
|
else process.env.OMNIROUTE_ISSUE_AGENT_ENABLED = previous;
|
|
if (previousDataDir === undefined) delete process.env.DATA_DIR;
|
|
else process.env.DATA_DIR = previousDataDir;
|
|
}
|
|
});
|
|
|
|
test("issue-agent run accepts recorded GitHub export payloads", async () => {
|
|
const previous = process.env.OMNIROUTE_ISSUE_AGENT_ENABLED;
|
|
const previousDataDir = process.env.DATA_DIR;
|
|
process.env.OMNIROUTE_ISSUE_AGENT_ENABLED = "true";
|
|
process.env.DATA_DIR = mkdtempSync(join(tmpdir(), "issue-agent-export-route-"));
|
|
try {
|
|
const response = await POST(
|
|
new Request("http://localhost/api/issue-agent/runs", {
|
|
method: "POST",
|
|
body: JSON.stringify({
|
|
mode: "recorded-triage",
|
|
githubExport: {
|
|
issue: {
|
|
title: "Fix GitHub export importer",
|
|
body: "bug in recorded context path",
|
|
html_url: "https://github.com/KooshaPari/OmniRoute/issues/88",
|
|
},
|
|
comments: [{ user: { login: "maintainer", type: "User" }, body: "please patch" }],
|
|
},
|
|
}),
|
|
})
|
|
);
|
|
const body = await json(response);
|
|
const context = body.context as Record<string, unknown>;
|
|
|
|
assert.equal(response.status, 200);
|
|
assert.equal(body.issueUrl, "https://github.com/KooshaPari/OmniRoute/issues/88");
|
|
assert.equal(body.issueNumber, 88);
|
|
assert.equal(context.issueTitle, "Fix GitHub export importer");
|
|
assert.equal(context.intent, "bugfix");
|
|
assert.equal(body.auditPath, join(process.env.DATA_DIR, "issue-agent", "audit.jsonl"));
|
|
} finally {
|
|
if (previous === undefined) delete process.env.OMNIROUTE_ISSUE_AGENT_ENABLED;
|
|
else process.env.OMNIROUTE_ISSUE_AGENT_ENABLED = previous;
|
|
if (previousDataDir === undefined) delete process.env.DATA_DIR;
|
|
else process.env.DATA_DIR = previousDataDir;
|
|
}
|
|
});
|
|
|
|
test("issue-agent run sanitizes a forced audit-write failure instead of leaking its absolute path", async () => {
|
|
const previous = process.env.OMNIROUTE_ISSUE_AGENT_ENABLED;
|
|
const previousDataDir = process.env.DATA_DIR;
|
|
process.env.OMNIROUTE_ISSUE_AGENT_ENABLED = "true";
|
|
const auditBlockerDir = mkdtempSync(join(tmpdir(), "issue-agent-audit-blocker-"));
|
|
// appendIssueAgentAuditRecord() (src/lib/issueAgent/audit.ts) does
|
|
// `mkdir(join(DATA_DIR, "issue-agent"), { recursive: true })`. Pre-creating a
|
|
// FILE at that exact path forces a real Node fs error (EEXIST) whose raw
|
|
// `.message` embeds this directory's absolute path -- the same leak shape a
|
|
// permission-denied or disk-full failure would produce (Hard Rule #12).
|
|
writeFileSync(join(auditBlockerDir, "issue-agent"), "blocking file, not a directory");
|
|
process.env.DATA_DIR = auditBlockerDir;
|
|
|
|
try {
|
|
const response = await POST(
|
|
new Request("http://localhost/api/issue-agent/runs", {
|
|
method: "POST",
|
|
body: JSON.stringify({
|
|
mode: "recorded-triage",
|
|
issueUrl: "https://github.com/KooshaPari/OmniRoute/issues/1",
|
|
}),
|
|
})
|
|
);
|
|
const body = await json(response);
|
|
|
|
assert.equal(response.status, 400);
|
|
assert.equal(typeof body.error, "string");
|
|
const errorMessage = String(body.error);
|
|
assert.doesNotMatch(errorMessage, /EEXIST|ENOENT|EACCES/, "must not leak the raw errno code");
|
|
assert.ok(
|
|
!errorMessage.includes(auditBlockerDir),
|
|
"must not leak the DATA_DIR absolute path"
|
|
);
|
|
assert.ok(
|
|
!errorMessage.includes("/issue-agent/"),
|
|
"must not leak the audit subdirectory path"
|
|
);
|
|
assert.equal(errorMessage, "Issue Agent request failed due to an internal error");
|
|
} finally {
|
|
if (previous === undefined) delete process.env.OMNIROUTE_ISSUE_AGENT_ENABLED;
|
|
else process.env.OMNIROUTE_ISSUE_AGENT_ENABLED = previous;
|
|
if (previousDataDir === undefined) delete process.env.DATA_DIR;
|
|
else process.env.DATA_DIR = previousDataDir;
|
|
}
|
|
});
|