Files
OmniRoute/tests/unit
Wu Shuwen 36493a6270 fix(evals): fail a case whose model call errored instead of scoring it passed (#13201)
* fix(evals): fail a case whose model call errored instead of scoring it passed

runSuite() attached caseMetrics[id].error to the graded result but never forced
`passed` to false. executeEvalCase() returns a failed call as an ordinary output
string ("[ERROR] <message>"), so any expected pattern that happened to match that
text was recorded as a pass. That inflates the reported pass rate, and reports a
non-zero score for a run in which no model was ever reached.

Built-in codex-comparison case codex-07 reproduces it: its pattern is
"try|catch|throw|error|Error" and the provider-resolution failure text ends with
"...added as a combo entry.", so the `try` alternative matches and the case is
scored as passed while carrying a non-empty error.

A case that never reached a model has no measured behaviour to grade, so a
failure is the only honest score.

Refs #13137

* docs(changelog): add fragment for the errored-eval-case fix (#13201)
2026-09-18 11:30:35 -03:00
..
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00
2026-08-26 14:25:01 -03:00