* test(infra): retry recursive temp-dir removal instead of failing a shard on ENOTEMPTY (#11966)
Two shards on release/v3.8.51 went red in one day with the same signature —
"ENOTEMPTY, Directory not empty: /tmp/omniroute-<test>-XXXXXX" — from
combo-same-provider-cascade (Unit Tests fast-path 4/4, on a PR that touches only
.github/) and auth-policy-embeddings-webfetch-7785 (the 20k-test TIA step). Both pass
alone and on re-run: the cleanup races something still writing into the directory
(SQLite WAL/-shm checkpoint, a worker, the backup) and under a loaded hosted runner
the window opens. 1154 test files do their own cleanup with
fs.rmSync(dir, { recursive: true, force: true }); 57 already asked for retries.
One-shot codemod (scripts/ad-hoc/codemod-rm-maxretries.mjs, kept for the record):
every rm / rmSync / rmdirSync option object with `recursive: true` and no
`maxRetries` gains `maxRetries: 5, retryDelay: 100` — Node itself then retries
ENOTEMPTY/EBUSY/EPERM for up to ~0.5 s before giving up. 2243 call sites in 1292
files under tests/, the shared tests/_setup/isolateDataDir.ts exit hook included.
Only the option object changes: no call site, assertion or import is touched.
Validation: prettier and ESLint (with the frozen suppressions) clean on all 1292
files; a random 20-file sample runs green (quota-redis-store hangs identically on
the untouched tree — it needs a Redis on localhost, an environment matter). The
four unit shards on this PR are the full run.
* fix(quality): let check-forgotten-sibling-tests read a 1,000-file diff
The gate shells out to `git diff` through execFileSync with Node's default 1 MB
maxBuffer; the 1,292-file codemod in this PR is the first diff large enough to
overflow it, and the gate died with `spawnSync git ENOBUFS` before comparing
anything. 64 MB is far above any real PR and costs nothing when unused.
The router-eval CLI test spawns the CLI with spawnSync and asserts stderr stays empty. NODE_TEST_CONTEXT is inherited by those children, so since #10432 (guard #10428) resolveWritableDataDir() detects a test context with no DATA_DIR and warns on stderr before falling back to a throwaway dir - 194 chars that broke three cases. Pass an isolated DATA_DIR in the child env (the resolution the guard message itself prescribes) instead of loosening the assertions.
Merged after conflict resolution: the 5 conflicting test files were the base-red drains that #11201 already landed on the tip — kept the tip versions; the feature content is untouched. Validated on the combined batch board + this branch: codex-app-server + codex-gpt56-catalog 25/25, typecheck:core clean, docs-counts green (351 providers), provider-consistency 268/351/0. The opt-in codex-app-server transport (JSON-RPC-over-WS, turn/completed-awaited close, Responses SSE bridge) leaves the default codex path untouched. Thank you @arminanton — a 3.4k-line transport with the docs wave and tests to match!
* feat(eval): add router-eval harness (AIQ scoring, regression gate, Pareto search)
Extracts a standalone router-eval evaluation tool that replays routing
decisions (from NDJSON corpora or the usage_history/call_logs SQLite
tables) into an AIQ (success/latency/cost) score, compares baseline vs.
candidate router configs with a retained-run regression gate, and ranks
Pareto-optimal candidates across a search space — a sibling to the
existing eval:compression harness.
New scripts: scripts/router-eval/{index,compare,patch-compare,search,
trends}.ts, scripts/check/check-router-eval-regression.ts, and
src/lib/routerEval/index.ts, wired via 6 new package.json entries
(eval:router, eval:router:compare, eval:router:patch-compare,
eval:router:search, eval:router:trends, check:router-eval).
Reconstructed onto current release/v3.8.49 from the original ~142-commit
stale PR branch: only the genuinely new router-eval payload (17 files)
was extracted — the other ~560 changed files in the original diff were
base-drift already present on release in newer form. The new package.json
scripts now invoke `node --import tsx` instead of `bun`, matching the
`eval:compression` precedent (Bun is reserved for a closed 5-script
allowlist). The runtime-detection shim (`"Bun" in globalThis`) already
present in the harness gracefully falls back to better-sqlite3 under
Node, so no logic changes were needed there; the retained-run manifest's
previously-hardcoded `runtime: "bun"` field and matching CLI help text
were corrected to reflect the actual invocation.
All 27 existing router-eval unit tests pass unchanged.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
* refactor(router-eval): decompose toRouterObservation below complexity gate
toRouterObservation had cyclomatic complexity 18 (gate max is 15). Extract
the per-field parsing/normalization into pure helpers (sampleId, model
fields, latency, cost derivation, success) so the entry point is a plain
sequential assembly of a RouterObservation. Behavior is unchanged — same
tests pass, same fallbacks, same precedence between input aliases.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
