fix(evals): mark eval-runner requests as self-managed so cases measure the model (#13139) (#13206)

* fix(evals): mark eval-runner requests as self-managed so cases measure the model

executeEvalCase() built its request with only Content-Type and Authorization, so
every graded case picked up the chat path's contextual injections: a selected
output style was prepended as a system message (gated on
`x-omniroute-compression`) and, once the request carried an API key, retrieved
memory plus the built-in `memory_*` tools were appended (gated on
`x-omniroute-no-memory`). An evaluation therefore measured the operator's
injected context as much as the model, and passing an API key to a run made its
score worse, because the key is what gives the request a memory owner (Refs #13139).

Both are documented request-header opt-outs, so the runner now sets them on every
case. Request construction moves to an exported buildEvalCaseRequest() so the
header contract is testable without invoking the chat route.

* docs(changelog): add the eval-runner self-managed-context fragment (#13206)

---------

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

src/lib/evals/runtime.ts

@@ -183,11 +183,28 @@ function resolveCaseModel(evalCase: Record<string, unknown>, target: EvalTargetI
   return caseModel || "gpt-4o";
 }
 
-async function executeEvalCase(
+/**
+ * Build the chat-completions request for one eval case.
+ *
+ * The runner manages its own context: a graded case must measure the model
+ * answering the case, not the operator's injected context. Two injections
+ * otherwise apply on the ordinary chat path — a selected output style is
+ * prepended as a system message (gated on `x-omniroute-compression`), and
+ * retrieved memory plus the built-in `memory_*` tools are appended once the
+ * request carries an API key (gated on `x-omniroute-no-memory`). Both are
+ * request-header opt-outs, so the runner sets them on every case. Without them
+ * a graded case answers in the configured persona or spends its turn calling
+ * `memory_*`, and a run that passes an API key scores *worse* than one that
+ * does not, because the key is what gives the request a memory owner (#13139).
+ *
+ * Exported so the header contract can be asserted without invoking the chat
+ * route — see tests/unit/evals-runtime-self-managed-headers-13139.test.ts.
+ */
+export function buildEvalCaseRequest(
   evalCase: Record<string, unknown>,
   target: EvalTargetInput,
   apiKey: string | null
-): Promise<{ output: string; durationMs: number; error?: string }> {
+): Request {
   const input =
     evalCase.input && typeof evalCase.input === "object" && !Array.isArray(evalCase.input)
       ? (evalCase.input as Record<string, unknown>)
@@ -195,9 +212,7 @@ async function executeEvalCase(
   const model = resolveCaseModel(evalCase, target);
   const headers = new Headers({
     "Content-Type": "application/json",
-    // #13139 — Eval cases must measure the model, not injected context.
-    // Disable output-style injection (persona system messages) and memory
-    // injection (retrieved context + memory_* tools) so grading is clean.
+    // Self-managed context — see the docblock above.
     "x-omniroute-compression": "off",
     "x-omniroute-no-memory": "true",
   });
@@ -206,7 +221,7 @@ async function executeEvalCase(
     headers.set("Authorization", `Bearer ${apiKey}`);
   }
 
-  const request = new Request("http://localhost/api/v1/chat/completions", {
+  return new Request("http://localhost/api/v1/chat/completions", {
     method: "POST",
     headers,
     body: JSON.stringify({
@@ -219,6 +234,14 @@ async function executeEvalCase(
           : 512,
     }),
   });
+}
+
+async function executeEvalCase(
+  evalCase: Record<string, unknown>,
+  target: EvalTargetInput,
+  apiKey: string | null
+): Promise<{ output: string; durationMs: number; error?: string }> {
+  const request = buildEvalCaseRequest(evalCase, target, apiKey);
 
   const startedAt = Date.now();
   const response = await postChatCompletion(request);