* fix(plugins): delete the host script synchronously and stop two tests leaking child processes
Three related leaks in the plugin child-process lifecycle, found while tracing 27 node
processes on a developer machine.
1. loader.ts removed the generated omniroute-plugin-host-*.mjs with a fire-and-forget
`rm(...).catch(() => {})`. That unlink loses the race against process exit: test:unit
runs with --test-force-exit, which tears the process down before the promise settles,
so every plugin load leaked one temp .mjs into TMPDIR. Measured at 6 files per
full-suite run, 40 accumulated over a handful of local runs. rmSync closes the race;
the throw stays swallowed because an exception raised from a child "exit" handler
would take the server down, and a leftover temp script would not.
2. plugins-manager-lifecycle.test.ts "activates an installed plugin" called activate() --
which spawns the plugin's child process -- but never deactivate(). deactivate() is the
only path that reaches the loader's cleanup(), so the child outlived the test and its
IPC channel kept the test process's event loop alive.
3. plugins-manager-restart-reload-7806.test.ts simulateRestart() deleted the entry from
loadedPlugins without calling cleanup(), dropping the only handle that can kill child
#1. The reload then spawned child #2, and the finally block's deactivate() could reach
only child #2 -- one dangling child per test. A real restart takes the whole process
tree down, so calling cleanup() here is both the faithful simulation and the fix.
Combined effect: a run without --test-force-exit deadlocks. The test process cannot exit
while its child holds the IPC channel open, and the child waits for messages that never
come. Observed as three plugin hosts alive for 4h51m under a runner that never finished.
Validation (Hard Rule #18, TDD): the new plugins-loader.test.ts case fails against the old
async unlink ("must delete the host script synchronously, not on a later tick") and passes
with rmSync. It redirects TMPDIR/TEMP/TMP to a private directory before counting, because
test:unit runs at --test-concurrency=20 and a concurrent file's host scripts would
otherwise land in the counted directory and flake the assertion.
After: 19/19 pass across the three files, 0 temp scripts created, 0 orphan processes.
tests/unit/build/** 334/334; typecheck:core and eslint clean.
* docs(changelog): add fragment for plugin host script sync delete
Previously checked directly, ignoring database feature flag overrides configured via settings/UI. This update routes the check through so DB overrides are respected as intended.
Co-authored-by: diegosouzapw <diegosouzapw@users.noreply.github.com>
* fix(db): preserve node sqlite open semantics
* test(db): keep node sqlite coverage off Bun
* chore(changelog): number native sqlite fallback
* test(db): register node sqlite tests only under Node
The runner reads exactly one directory and records the bare numeric prefix as the
version, so the numeric slots are a single global namespace. Any distribution that
ships its own migrations next to the upstream set has to draw from that same range
while upstream keeps appending to it — and when both sides claim a number, the
runner records one name for it and treats the other as already applied. That
migration then never runs, silently, on every already-provisioned database.
OMNIROUTE_EXTRA_MIGRATIONS_DIRS registers additional directories as
`namespace=dir` entries separated by the platform path delimiter. Files found
there are recorded as `<namespace>-<number>` (e.g. `ee-134`), a version space that
cannot collide with the upstream numeric one, and they are applied after the core
set. Unset — the default, and the only case for a plain install — nothing changes:
no filesystem access, identical behaviour.
Misconfiguration throws instead of being skipped. A malformed entry, a namespace
outside [a-z][a-z0-9]*, a duplicate namespace, or a directory that does not exist
aborts startup, because silently missing schema is the exact failure this exists to
prevent. Two files sharing a number inside the SAME namespace still collide and
throw, mirroring the runner's own guard.
Also fixes an inconsistency the tests surfaced: a missing core directory returned
early and took the extra directories down with it. They are an independent set.
The version-namespaced strings need no further plumbing — the applied set, the gap
reconciliation and the name-mismatch check all key on the version string, and the
numeric-only paths (`Number.parseInt`, the "032"/"041"/"042" special cases,
`isSchemaAlreadyApplied`) ignore them by construction.
11 new tests; the 7 neighbouring migration suites stay green (62 tests).
Squash merge from local merge-train (Hard Rule owner-approved). Tip 029cdf4215cf465f0e1716ac9f84a84692b1e881 validated on 192.168.0.113: 26631/26653 pass.
Squash merge from local merge-train (Hard Rule owner-approved). Tip 029cdf4215cf465f0e1716ac9f84a84692b1e881 validated on 192.168.0.113: 26631/26653 pass.
Squash merge from local merge-train (Hard Rule owner-approved). Tip 029cdf4215cf465f0e1716ac9f84a84692b1e881 validated on 192.168.0.113: 26631/26653 pass.
Squash merge from local merge-train (Hard Rule owner-approved). Tip 029cdf4215cf465f0e1716ac9f84a84692b1e881 validated on 192.168.0.113: 26631/26653 pass.
* feat(db): cc discovery alias gate storage + EXPOSE_CC_DISCOVERY_ALIASES flag
Adds the gate for claude/<provider>/<model> discovery-alias mirror ids on
the /v1/models catalog: a new runtime feature flag (env forces on and wins
over the dashboard DB override), per-provider and per-model "on"/"off"/null
overrides stored in key_value under the ccDiscoveryAliases namespace, and a
pure precedence resolver (model > provider > global). Catalog wiring is a
separate follow-up task; this only lands the gate + storage.
* feat(sse): synthesize claude/ discovery aliases for the model catalog
* feat(api): advertise cc discovery aliases on /v1/models behind the 3-level gate
* feat(sse): resolve claude/ discovery aliases on the request path
* fix(sse): import getComboByName from db/combos, not the localDb barrel
* fix(sse): cover custom-node prefixes and the Codex WS bridge in cc discovery alias resolution
* feat(dashboard): cc discovery alias toggles + flag-screen env warning
Adds the operator-facing UI/API layer for the Claude Code discovery-alias
gate (claude/<provider>/<model> mirror ids on /v1/models): REST endpoint
for provider/model overrides, a provider-detail card with 3-state
(inherit/on/off) toggles, an info button on the Claude Code tool card
linking to Feature Flags, and an env-source warning on the
EXPOSE_CC_DISCOVERY_ALIASES flag card when it's forced on via env.
* feat(api): cc discovery usage metrics
* fix(api): record cc alias metric in the production wrapper + atomic counter upsert
* docs: document cc discovery aliases (Claude Code guide + feature flag catalog)
* fix(sse): don't mirror built-in auto/* combos as discovery aliases (advertised-but-unroutable)
* i18n(vi): translate the discovery-alias strings instead of shipping placeholders
vi is the one locale with a strict "no internal missing markers" test, so the 17
__MISSING__ entries this branch added (the provider ccAlias panel, the info
button, the feature-flag description and the env warning) would have turned that
test red the moment the base itself was repaired. Translated, keeping every ICU
placeholder ({modelId}, {error}) and the literal claude/<provider>/<model> id
shape intact.
* chore(quality): raise the frozen caps this feature legitimately grows
catalog.ts 1615 -> 1639: the alias synthesis is wired into the catalog builder,
which is where the per-key-filtered list is assembled — the only place the mirror
entries can be appended after model hiding has been applied.
localDb.ts 808 -> 810: two re-export lines for the new ccDiscoveryAliases db
module, which is exactly what the "Adding a New DB Module" recipe prescribes.
* refactor(dashboard,api): keep the complexity ratchets flat
The feature added four cyclomatic violations and one cognitive one, which the
ratchets reject — the baseline only moves when a metric improves. Split the new
code instead:
- appendCcDiscoveryAliases: the four skip-guards become isMirrorableId().
- resolveCcDiscoveryAliasStripWith: alias parsing and gate resolution become
parseCcAliasTarget() and resolveGateFor(), replacing a chain of ternaries that
each re-tested isComboAlias.
- FeatureFlagCard: the env-precedence warning becomes its own component instead
of a conditional branch inside an already-large render.
- ProviderCcAliasSection: the loader moves to useCcAliasData(), and the override
list and add-row become ModelOverrideList / AddOverrideRow, bringing both
oversized functions back under the 80-line rule.
Behavior unchanged — the 74 discovery-alias tests pass untouched. Both ratchets
now sit exactly at baseline (2188 / 971).
* fix(guardrails): Vision Bridge describe-fallback ignores unreachable candidates
getVisionCapableModels() scanned the entire static PROVIDER_MODELS catalog
for a describe-fallback model without checking whether the provider has a
usable active connection on this instance. On an instance with no `openai`
provider connected, this let the hardcoded default `openai/gpt-4o-mini` win
selection every time, the describe call would fail, and
replaceImageParts()'s describe-failure fallback (#4012) intentionally
preserves the raw image part — which then reaches a non-vision backend and
gets rejected with an opaque upstream error (e.g. "unknown variant
`image_url`, expected `text`").
Extract the credential-usability check already used by the whole-request
reroute path (visionBridge.ts's hasUsableCredentialsForModel /
isProviderConnectionUsable) into a shared visionBridgeCredentials.ts module,
and apply the same check to the describe-fallback candidate list in
visionBridgeRouter.ts. A confirmed-unusable connection (`false`) excludes a
candidate; an indeterminate result (`null`, e.g. no DB) fails open to
preserve existing behavior.
getVisionCapableModels/getBestVisionModel/getFallbackModels become async to
support the credential lookup; call sites and the existing unit test suite
are updated accordingly, plus new coverage for the exclusion behavior.
* fix(guardrails): fix flaky credential-mock race and move Vision Bridge router tests to a CI-blocking runner
The two new assertions added in this PR (excludes a candidate with no
usable active connection / selects a credentialed candidate over an
uncredentialed one) were correct — the failure was a genuine Vitest
race: getVisionCapableModels() fans out to hasUsableCredentialsForModel()
once per catalog entry via Promise.all, and dozens of concurrent
first-load `await import("@/lib/db/providers")` calls for the same
specifier under vi.mock() nondeterministically resolved against the real
module instead of the mock for some callers. Memoize the dynamic import
in visionBridgeCredentials.ts so it resolves exactly once and is reused,
which removes the race entirely (and is cheaper at runtime too).
Also: tests/unit/guardrails/visionBridgeRouter.test.tsx was never
collected by any CI-blocking gate — `test:unit`'s guardrails glob is
`*.test.ts` only, and `test:vitest` (vitest.mcp.config.ts) doesn't
include this directory; only the advisory `test:vitest:ui` picked it up.
Moved the suite to visionBridgeRouter.test.ts under node:test, threading
an optional `deps.hasUsableCredentials` injection point through
getBestVisionModel()/getFallbackModels() (consistent with the existing
deps pattern in visionBridge.ts) since this project's native test runner
has no supported ESM module-mocking mechanism.
Running the full guardrails suite together also surfaced that this PR's
own credential-exclusion feature silently broke the pre-existing
vision-bridge-callmodel.test.ts fallback-retry test: with zero seeded
provider connections in that test's isolated DATA_DIR, every fallback
candidate is now confirmed-unusable and excluded, leaving no fallback to
retry. Seeded one credentialed connection there so the fallback-retry
mechanics stay independent of the (unrelated) credential filter.
Refs #8433
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com>
* fix(oauth): explain the LAN-IP loopback mismatch with an actionable panel
#8046 already stops the doomed login when a PKCE_CALLBACK_SERVER_PROVIDERS
provider (codex / xai-oauth / grok-cli) is connected from a LAN IP, but it
explained itself as one long English sentence rendered in the generic red
"Connection failed" step. Two concrete problems with that:
- the operator had to parse prose to work out WHICH ports to forward, and the
command shipped with `<port>` / `<omniroute-host>` placeholders to resolve
by hand;
- it forwarded a single port. Both are required: the dashboard port is what
makes the origin true-localhost (a LAN origin never reaches the
callback-server branch at all), and the provider's fixed callback port is
where the browser is actually sent back to. Forwarding either one alone
still fails.
buildPkceLoopbackMismatchHint() now returns the diagnosis as structured data
with the detected host and both ports already filled in, and a dedicated
OAuthLoopbackMismatchPanel renders it as: what happened -> how to fix, in
three numbered steps with copy-to-clipboard fields. No "Try again" button —
retrying the same origin fails identically. The panel yields to the
paste-token tab so grok-cli (which is in both provider sets) never stacks the
two views.
The flat warning string stays exported for non-UI callers.
docs: REMOTE-MODE.md gains a "Connecting Codex / Grok on a remote install"
section with the fixed-callback table and the two-port tunnel, mirroring the
existing Antigravity section.
i18n: 9 new oauthModal keys, hand-written for en + pt-BR and propagated to the
remaining 40 locales as `__MISSING__:` sentinels (runtime falls back to the
clean English value per #7258).
* fix(oauth): correct the Antigravity remote-login guidance and drop the stale i18n copy
Same LAN-origin family as the codex fix in this branch, different mechanism and a
worse failure mode.
Google providers (antigravity / agy) have no fixed foreign port: OAuthModal builds
`http://127.0.0.1:<dashboardPort>/callback`. On a LAN origin that 127.0.0.1 is the
BROWSER's machine, and Google's firstparty/nativeapp consent only releases the code
once the loopback is reachable from the approving browser. When it is not, the consent
never redirects at all — it hangs. So unlike an ordinary provider there is no error
page and no callback URL in the address bar.
That made the existing copy actively wrong. `googleOAuthWarning` was corrected when
the login helper shipped (#5203), but a changed English value does not invalidate
existing translations and `i18n:sync-ui` only fills keys that are ABSENT, never ones
that are STALE — so 39 of 43 locales (pt-BR, pt, es, de, fr, ja, zh-CN, …) kept the
original "wait for the redirect, copy the full URL and paste it below", instructing a
flow that cannot complete. The drift gate that should have caught this is a no-op:
`check-translation-drift.mjs` needs `.i18n-state.json`, which is not in the repo, and
it runs `--warn`.
Because the key's MEANING changed, it is renamed rather than edited — a new key cannot
inherit a stale translation. `googleOAuthWarning` is removed from all 43 locales and
replaced by 7 `googleLoopback*` keys, hand-written for en + pt-BR and marked
`__MISSING__:` elsewhere so the runtime falls back to correct English (#7258).
UI: `OAuthGoogleLoopbackNotice` states what is happening and surfaces both real
remedies with the detected host and port filled in — the local login helper
(recommended; its blob is what the Step 2 field accepts) and a single-port SSH forward.
It also REPLACES `remoteAccessInfo` for this family instead of stacking on top of it:
that notice promises an error page whose URL you copy, true for ordinary providers and
false here.
`agy` deliberately gets no helper command. bin/cli/commands/login.mjs pins
PROVIDER = "antigravity" and parsePastedCredentials() rejects a blob whose embedded
provider does not match the route provider, so advertising the helper there would send
the operator to a blob guaranteed to be refused. It keeps the tunnel path.
Refactor: the shared `resolveDashboardPort` / `buildSshLocalForward` helpers move to
`loopbackTunnel.ts`, used by both hint builders. The codex builder's behaviour is
unchanged (its 11 tests still pass untouched).
docs: REMOTE-MODE.md notes that the dashboard now surfaces the remedies, states that
one forward is enough for Antigravity (contrasting the two-port codex case), and aligns
Option B's command on 127.0.0.1 to match what the UI generates.
---------
Co-authored-by: ikelvingo <im.kelvinwong@gmail.com>
* feat(combos): add select all / unselect all in Browse Catalog
* fix(dashboard): guard combo Select all + test the real batch handlers (#8526)
Select all had no cap — with "Show configured only" off, or a large
provider catalog, one click could add hundreds of models to a combo.
ModelSelectModal now confirms above SELECT_ALL_CONFIRM_THRESHOLD (20)
before batch-adding, matching the native confirm() pattern already
used for bulk/destructive actions elsewhere in the dashboard.
Also extracts ComboFormModal's handleAddModels/handleDeselectModels
batching logic into computeBatchAddModelSteps/computeBatchDeselectModelSteps
(src/lib/combos/builderDraft.ts) so unit tests exercise the real
implementation instead of a hand-maintained mirror that could drift
from the component and stay green while production code broke.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com>
* feat(cli-tools): add all Hermes Agent auxiliary model roles
Extend HERMES_AGENT_ROLES from 7 to 18 slots to match the full
auxiliary.* set in Hermes Agent config.yaml:
- add: mcp, title_generation, memory_query_rewrite, tts_audio_tags,
triage_specifier, kanban_decomposer, profile_describer, goal_judge,
curator, monitor, background_review
- reorder: web_extract before compression (match upstream docs)
Backend generator/reader are generic on auxiliary.<role> — only the
role catalog, UI card, and i18n (en + pl) needed updating.
* fix(i18n): translate the 12 new Hermes auxiliary roles into vi and pt-BR
The 22 new en.json keys landed only in pl.json, breaking the vi and pt-BR
key-parity guards. Adds the same keys with real translations (no placeholder
strings), keeping both parity assertions exact.
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
* test(cli-helper): guard HERMES role catalog parity between backend and UI
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
* fix(providers): resolve native vision for path-shaped multimodal model ids
Leaf-id static/registry metadata now wins over synced attachment=false without
modalities, so cp/cline-pass/kimi-k3 forwards images natively.
Closes#8032
* fix(providers): scope path-shaped leaf lookup to vision only
Move leaf MODEL_SPECS fallback out of shared getStaticSpec() so
aihorde/deepseek/deepseek-v4-flash no longer inherits DeepSeek tool-calling
metadata (#8212). Vision still resolves via getVisionStaticSpec().
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
---------
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
POST /api/combos and PUT /api/combos/[id] had zero validation or
observability when a combo name collided with a real model id, and
sseModelService.getComboForModel() always resolves the combo first. That
combo-first precedence is not a bug: #6940 documents a combo named after a
bare model id (e.g. `gpt-5.5`) as the supported mechanism for per-model
provider fallback, reusing the #3227/#3233 machinery and covered by
tests/unit/responses-combo-resolution-3227.test.ts and
tests/unit/combo-name-codex-responses-rewrite.test.ts. Hard-rejecting a
colliding name (as #8530's literal acceptance criteria requested) would
regress that documented workflow.
Instead, both routes now attach a non-blocking `warning` field
(`COMBO_NAME_SHADOWS_MODEL`) to the create/rename response when the name
collides with a real model id, and a new boot-time scan
(scanComboModelNameCollisionsAtBoot in src/instrumentation-node.ts) logs a
startup warning enumerating existing collisions — so an operator who hits
this by accident has a signal, while the #6940-sanctioned pattern keeps
working exactly as before.
New tests/unit/combo-model-name-collision-8530.test.ts proves both: the
sanctioned shapes (create/rename to a colliding name) still return
201/200 with the warning attached, and non-colliding names get no warning
field at all.
Co-authored-by: ikelvingo <im.kelvinwong@gmail.com>
* chore(auth): condense #8407 note in tokenHealthCheck to fit frozen file-size baseline
The 3-line comment added by #8426 pushed tokenHealthCheck.ts to 843 LOC,
2 over its frozen baseline of 841 (base-red on release/v3.8.49). Fold the
devin-cli rationale into one line — same meaning, gate green again.
* refactor(dashboard): extract provider card highlight into HighlightableProviderCard
#8349 left the back-navigation highlight concern inline in providers/page.tsx
(state + two callbacks + onCardClick/ref props repeated at 18 card call
sites), pushing the page to 1990 LOC over its frozen baseline of 1927.
Move the concern into a HighlightableProviderCard wrapper: each card instance
keeps its own copy of the highlighted id and only the card whose provider id
matches scrolls into view and highlights — same semantics as the single
page-level state, since resolveHighlightedCard already gates on the id match.
page.tsx drops to 1917 LOC (under the frozen baseline, no rebaseline needed).
* test(dashboard): cover HighlightableProviderCard wiring
Direct coverage for the wrapper extracted in the previous refactor:
mount-time scroll+highlight only when history.state.providerId matches,
no reaction on mismatch/null state, isolation across multiple card
instances, and click-time navigation recording via history.replaceState.
Note: the jsdom scrollIntoView shim is a plain no-op rather than a
vi.fn() — vi.restoreAllMocks() would otherwise restore the shared mock
and let the next test's spy inherit its accumulated call count.
`BaseGuardrail.preCall`/`postCall` return `GuardrailResult<unknown> | void`,
and that `| void` is deliberate: docs/security/GUARDRAILS.md documents returning
nothing as one of the three "no change" signals, and CredentialMaskerGuardrail
still declares it. So the union stays.
The problem is the consumption site. `registry.ts` reads `result?.block`,
`result?.message`, `result?.meta`, `result?.modifiedPayload` — but optional
chaining does not make `void` inspectable, and neither does a truthiness test.
That is all 12 TS2339s in the file: one union, six property reads, two dispatch
loops.
Fixed by funnelling the return through `unknown` once, in a local
`asGuardrailResult()` helper, so the loops work against a plain
`GuardrailResult | undefined`. The public contract is untouched — no signature
narrowed, no guardrail changed, callers outside this file unaffected.
280 -> 268, zero new, on a line-number-agnostic diff of the full tsc error set.
Tests: the `void` arm of the documented contract had NO coverage —
guardrails-registry.test.ts exercises guardrails that return a result and one
that throws, never one that returns nothing. Added
guardrails-void-no-change-contract.test.ts (4 tests): silent preCall/postCall
pass through untouched and are recorded as ran-not-skipped-not-errored;
returning `{}` produces an identical execution record; a silent guardrail does
not stop a later one from modifying. They pass against the parent commit's
registry.ts too, which is the evidence for no behavior change.
75/75 across the 13 guardrail suites.
Co-authored-by: backryun <busan011@ormbiz.co.kr>
* fix(resilience): honor comboCooldownWait for every combo strategy
The cooldown-wait path claimed all strategies, but isComboCooldownWaitEligible still gated on quota-share/auto. Widen the gate, raise the per-target timeout floor with it, and consult the allow-list from earliestRetryAfter so a later 403 cannot skip the decision. Fixes#8541.
* test(combo): fold cooldown-wait strategy assertions into a loop
Keep combo-config.test.ts under the file-size ceiling while covering every
routing strategy (including internal quota-share) for the #8541 gate fix.
* chore(combo): trim cooldown-wait comment under file-size ceiling
After rebase onto tip the net +2 in combo.ts sat one line over the frozen
check:file-size count (split-based); fold the SECURITY note into the block.
reverseModelsDevProviders() only matched MODELS_DEV_PROVIDER_MAP entries
by the canonical OmniRoute provider id, but the map's RHS for the OAuth
CLI providers (codex/claude) only lists their alias (cx/cc), never the
canonical id itself. Since the models.dev sync job writes
model_capabilities rows under openai/cx and anthropic/cc (never
codex/claude), and the auto-combo gate canonicalizes a codex/... or
claude/... target's provider to "codex"/"claude" before the lookup,
the synced capability row was unreachable for those two providers.
Also probe the provider's alias (via the already-imported
PROVIDER_ID_TO_ALIAS) when scanning the map, so a canonical id like
"codex"/"claude" still matches entries keyed only by their alias.
Co-authored-by: ikelvingo <im.kelvinwong@gmail.com>
* fix(providers): prevent health sweep from expiring devin-cli local credentials
* fix(auth): drop devin-cli from supportsTokenRefresh explicit set (#8407)
Root cause: listing "devin-cli" as refresh-capable made tokenHealthCheck
force-expire local CLI connections that never have a refresh token.
Remove it from the explicit set (keep windsurf) and drop the health-check
provider hardcode — the existing supportsTokenRefresh=false guard is enough.
Live incident: memory.vec.upsert.fail {"error":"no such table: vec_memories"}
recurred repeatedly right after restarts, even though ensureReady() is called
immediately beforehand. ensureReady()'s signature-check-then-maybe-recreate
logic (resetForSignature does DROP TABLE IF EXISTS + CREATE VIRTUAL TABLE) is
not synchronized against a concurrent caller's upsertVector/deleteVector -- a
second in-flight memory write that independently decides (from a stale read
of memory_vec_meta) it also needs to reset the table can drop it out from
under another write's insert. Confirmed live: memory_vec_meta showed
vec_loaded=0 for the entire session across many restarts, then flipped to 1
mid-investigation once one attempt finally completed without interruption --
consistent with an intermittent race, not a permanently broken path (verified
the underlying sqlite-vec extension and CREATE VIRTUAL TABLE statement work
correctly in isolation, both on the host and inside the production container).
Rather than chase the exact interleaving (every underlying SQLite call is
synchronous via better-sqlite3, so the race window is narrow and did not
reproduce under simple Promise.all stress tests), makes the write path
resilient to arriving after the table was dropped: on a "no such table"
error, recreate vec_memories from the last-known-good memory_vec_meta
dimension and retry once.
Co-authored-by: Markus Hartung <markus.hartream@gmail.com>
* fix(providers): reconcile Kimi K3 vision when attachment contradicts modalities
Synced models.dev rows for kimi-coding*/k3 can ship attachment=false while
modalities_input still lists image/video. Prefer the modality signal (and
normalize at sync + resolve) so supportsVision, attachment, and exposed
modalities agree.
Closes#8250
* fix(providers): keep Kimi K3 static fallback text-only (#8250)
The Kimi K3 vision reconciliation added supportsVision=true directly
to the kimi-coding registry entry for id "k3". That entry is the
static/stable fallback catalog used when discovered capabilities are
unavailable, and it must stay text-only per #4071 — the vision fix is
already applied correctly on the discovered path via
MODEL_SPECS["kimi-k3"] (aliases: ["k3"]) and
modelCapabilities.ts::resolveVisionCapability.
Restores the invariants guarded by
tests/unit/kimi-k2.7-code-registration.test.ts ("Kimi Code k3 fallback
leaves discovered capabilities unset") and
tests/unit/catalog-updates-v3829-kimi-qwen.test.ts ("kmca stable
fallback only carries documented static capabilities").
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Cause: #8182's terminal-connection guard in checkConnection() returns
early for any testStatus in {credits_exhausted, banned, expired} to
stop the sweep from wasting CPU/network probing connections that can
never self-heal. But testStatus="expired" + errorCode="no_refresh_token"
is exactly the state the pre-existing GitHub Copilot self-heal targets
(isGitHubAccessTokenOnlyConnection + canClearGitHubNoRefreshTokenState,
~line 83-97 / 413-492): a Copilot connection with no OAuth refresh
token but a still-valid copilotToken, which the sweep is supposed to
flip back to "active". With the new guard placed ahead of that block
unconditionally, the self-heal became unreachable for exactly the
state it exists to clear.
Impact: healthy GitHub Copilot connections that once lost their OAuth
refresh token got stuck at testStatus="expired" in the dashboard
forever, even though their Copilot sub-token kept working and the
sweep would have cleared the stale status back to "active" every
cycle before #8182.
Fix: carve out the exact recoverable shape from the terminal-skip
guard — testStatus==="expired" && errorCode==="no_refresh_token" &&
isGitHubAccessTokenOnlyConnection(conn) — so the guard still skips
every other terminal case (credits_exhausted, banned, and "expired"
for any other reason) untouched, matching #8182's original intent.
Validation: tests/unit/token-health-no-refresh-token-expired-5326.test.ts
was red (1 fail / 4 pass) before the fix — "checkConnection clears
stale no_refresh_token state for usable GitHub Copilot connections"
asserted testStatus flips back to "active" but got "expired". Green
after the fix (6/6, including a new boundary test proving a GitHub
Copilot connection expired for any OTHER reason, e.g. errorCode
"invalid_grant", is still skipped untouched). Also reran the adjacent
checkConnection/tokenHealthCheck suites (token-health-check.test.ts,
token-health-check-circuit-breaker.test.ts,
apikey-connection-health-check.test.ts, token-health-check-sweep.test.ts,
token-health-check-tickms-defined.test.ts, tokenHealthCheck-batchSize.test.ts,
codex-oauth-refresh-persist-6352.test.ts, oauth-providers-error-handling.test.ts)
— all green, confirming #8182's terminal-skip behavior is otherwise
unchanged. npm run typecheck:core clean.
Refs #8182
Refs #8286
Refs #5326
* fix(bifrost): send v-prefixed transport version, not bare semver
bifrost.ts resolveSpawnArgs() set BIFROST_TRANSPORT_VERSION straight from
getInstalledVersionSync(), which reads the raw "version" field out of
node_modules/@maximhq/bifrost/package.json - always bare semver per npm
convention (e.g. "1.6.3").
@maximhq/bifrost's own bin.js validates that env var against
/^v\d+\.\d+\.\d+(?:-[0-9A-Za-z.-]+)?$/ or the literal "latest" and rejects
anything else with "Invalid transport version format", exiting immediately.
Every embedded Bifrost instance failed to start as a result.
Adds formatTransportVersion() to normalize at the call site that owns the
env var, so bifrost.ts and the upstream @maximhq/bifrost package both stay
exactly as designed - no changes needed to bifrost itself.
Strengthens the existing resolveSpawnArgs test to assert the actual v-prefix
format (it previously only checked for a non-empty string, the same gap
#6877 called out for cliproxy's pre-existing test), and adds a dedicated
regression test file covering the pure formatTransportVersion() helper plus
a real-filesystem resolveSpawnArgs() integration check.
Found and fixed while self-hosting OmniRoute and diagnosing why bifrost
kept crash-looping on startup.
* fix(bifrost): resolve install dir lazily so version read honors runtime DATA_DIR
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Co-authored-by: seanford <seanford@users.noreply.github.com>
* chore(ci): add .mergify.yml to main — Mergify only reads config from the default branch (#7168)
* fix(ci): add the auto-enqueue pull_request_rule to the Mergify config (queue_conditions alone are eligibility-only) (#7179)
* fix(ci): migrate Mergify auto-enqueue to merge_protections_settings.auto_merge_conditions (rules-based path is EOL 2026-07-16) (#7216)
* fix(ci): drop Mergify batch settings (batching is a paid-tier feature; free plan queue is serial) (#7220)
* fix(ci): merge queue tolerates the advisory dast-smoke failure (its GH-hosted build hang dequeued every attempt) (#7225)
* fix(usage): correct token/request counting for 30D/90D/YTD/ALL ranges
Two bugs caused incorrect usage statistics for date ranges beyond the
raw data retention window:
1. Cutoff mismatch: the analytics route computed rawCutoffDate from
aggregation.rawDataRetentionDays (migration 046 seeds =7) while
cleanupUsageHistory rolls up and deletes at retention.usageHistory
(=30). The window [day-30, day-7) existed in usage_history but was
excluded from BOTH UNION legs — raw leg floored at day-7, aggregated
leg ended at day-7 — producing undercounted token sums for 30D,
90D, YTD, and ALL ranges.
Fix: use dbSettings.retention.usageHistory for the raw cutoff in
both route.ts and getRawDataCutoffDate() (aggregateHistory.ts),
matching the actual cleanup boundary.
2. Request undercount: COUNT(*) on the unified source counted each
daily_usage_summary row as 1, not total_requests. A day with 50
rolled-up requests counted as 1.
Fix: add a 'requests' column to both UNION legs (raw: 1, aggregated:
total_requests), change COUNT(*) to SUM(requests) in 6 query
functions, and change successfulRequests from
SUM(CASE WHEN success=1 THEN 1 ELSE 0 END) to
SUM(CASE WHEN success=1 THEN requests ELSE 0 END). Also set
agg leg latency_ms to NULL so AVG(latency_ms) is not skewed.
Tests: 33/33 source-level tests pass (db-usageanalytics-split.test.ts),
verifying 'requests' column presence and SUM(requests) usage in all
affected queries. DB-level integration test added to
usage-analytics.test.ts (requires node + better-sqlite3).
* fix(ci): green CI reds on #7300 — file-size ratchet, stale test fixture, shallow-checkout selfref test
- src/app/api/usage/analytics/route.ts: trim the new comment to keep the file
at the frozen file-size baseline (942 lines) after the retention.usageHistory
cutoff fix — no logic change.
- tests/unit/usage-analytics-route.test.ts: the pre-existing "does not
double-count raw and aggregated rows" test hardcoded a 30-day cutoff that
matched the OLD (buggy) aggregation.rawDataRetentionDays default. Now that
the raw/aggregated boundary correctly uses retention.usageHistory (365 days
by default, matching cleanupUsageHistory's actual rollup/delete boundary),
the fixture's synthetic "old" row was within the raw window and got
excluded from the aggregated leg. Read the real retention setting instead
of hardcoding 30 so the fixture reflects the corrected boundary. Same
assertions (still expects no double-counting, totalRequests=2,
totalTokens=185) — only the fixture dates change.
- tests/unit/check-test-masking-selfref-6634.test.ts: tolerate the shallow/
single-ref checkout used by GitHub-hosted Unit Tests runners (no local
origin/main ref) by fetching it on demand and skipping (never failing) when
unreachable offline. Matches the fix already applied on another branch
(2e42b8efc/#7174) for the same root cause, not yet on main.
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
* test(ci): make the #6634 selfref test checkout-independent (read the real file, no git ref)
The previous on-demand `git fetch origin main` + t.skip() fallback cleared the
shallow-checkout failure but tripped the PR Test Policy's test-masking gate
(a new .skip counts as a silenced assert — correctly so).
Drop the git dependency entirely instead: read the REAL current source of
tests/unit/check-test-masking.test.ts from disk (so the actual #6404 fixture
literals stay under test) and model the pre-#6404 state with an empty base,
which maximizes headTaut - baseTaut — the strictest input for the exclusion
this test asserts. No skip, no weakened assertion, same deepEqual guarantee.
Verified non-vacuous: neutralizing SELF_TEST_FIXTURE_RE in
scripts/check/check-test-masking.mjs makes this test fail; restoring it makes
it pass.
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
* test(ci): make the #6634 selfref guard hermetic — main's copy hard-fails every PR (#7341)
main's copy of this test still does git I/O inside a unit test:
const baseSrc = git(['show', 'origin/main:' + FILE]);
Runners check out a shallow single ref, so origin/main does not resolve and the
test dies with 'fatal: invalid object name origin/main'. Every PR into main
fails Unit Tests (7/8) on it — today that is #7313, #7315, #7316, #7334, #7336
and #7337, six PRs red on a defect none of them introduced. #7313 has no other
red at all.
release/v3.8.49 already carries a fix (2e42b8efc, #7174: try/catch, fetch
origin/main on demand, t.skip() when unreachable), but it only reaches main at
release time — so main stays broken for the whole cycle. Cherry-picking it would
also import a new problem: PR Test Policy classifies t.skip() as a silenced
assertion, which we watched it correctly catch on #7300 today.
This is the hermetic version instead (ported from #7327, which does the same for
the release branch): read the file straight off disk, compare against an empty
base so baseTaut/baseExtTaut are 0 — the strictest possible comparison point —
and call evaluateMasking() directly. No git ref, no fetch, no skip, nothing the
runner's checkout depth can break.
The #6634 regression stays covered: the guard's logic lives in
SELF_TEST_FIXTURE_RE (check-test-masking.mjs:337), not in the test. Proven both
ways on main before committing — neutralise SELF_TEST_FIXTURE_RE to /$^/ and
the test FAILS; restore it and it passes 2/2, with check-test-masking.mjs left
byte-identical.
Co-authored-by: growab <nekron@icloud.com>
* chore(quality): tighten main's coverage baseline to the CI's real numbers (#7347)
main's ratchet had been failing --require-tighten on every PR: 11 metrics
improved but the baseline was never tightened. Same class as the #6634
selfref guard — an infra fix that lands only on the release branch leaves
main red for the whole cycle, and every PR into main pays for it.
Values are the merged-coverage numbers from a run on main itself (a local
run measures ~68% vs CI's ~80%; the baseline's own note warns about that
gap). Only the 11 coverage values change — gitleaks and semgrepFindings
keep main's own state.
No changelog fragment: #7326 carries it on release/v3.8.49, and a second
one here would double the entry at release time.
---------
Co-authored-by: Diego Rodrigues de Sa e Souza <8016841+diegosouzapw@users.noreply.github.com>
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
Background token health check sweep was probing connections with
terminal statuses (credits_exhausted / banned / expired) on every
cycle, wasting CPU and network. These connections can never self-heal
via token refresh — they need manual re-auth or credit top-up.
Add a terminal status guard in checkConnection() that mirrors the
existing isTerminalConnectionStatus() in auth.ts and
TERMINAL_CONNECTION_STATUSES in connectionRecovery.ts.
Verified by reporter: after manually disabling 11 credits_exhausted
connections, CPU dropped from ~53% to ~2.7%. This fix automates that
skip so the sweep never touches terminal connections in the first place.
Co-authored-by: Austin Liu <austinliu@Austins-MacBook-Air-3.local>
* feat(settings): configurable model catalog cache TTL
Add modelCatalogCacheTtlMs to DatabaseSettings with default 1500ms.
Extend cache-config API route to accept the new field.
Replace hardcoded catalog cache TTL with dynamic settings value.
Add 'Cache' settings tab at /dashboard/settings/cache with sidebar entry,
i18n keys, header description, and legacy route redirect.
* feat(combo): add model connection filter toggle to ModelSelectModal
Adds a 'Show configured only' checkbox below the search bar in
ModelSelectModal that filters each provider group's models through
hasEligibleConnectionForModel. Toggle state persists in localStorage.
- Import hasEligibleConnectionForModel from domain/connectionModelRules
- showConfiguredOnly state + localStorage persistence
- connectionFilteredGroups memo layered on filteredGroups
- Renders both provider section and empty state from connectionFilteredGroups
* test(combo): add connection filter toggle tests for ModelSelectModal
Three test cases: (1) hide excluded models when toggle on,
(2) show empty state when all models excluded,
(3) drop provider group when all its models excluded.
All 3 tests pass.
* fix(settings): correct cache-config route import + add route/tab coverage
The cache-config route imported get/update helpers from a nonexistent
module (@/lib/localDb/databaseSettings) and called an undefined
updateSettings() in PUT, crashing every request. Import the real
databaseSettings module (matching the sibling database/route.ts
convention) and call updateDatabaseSettings(); idempotencyWindowMs is
routed through the flat @/lib/db/settings module instead, since that is
where it is actually read at runtime (idempotencyLayer.ts,
runtimeSettings.ts) — it was never part of the databaseSettings "cache"
section type.
Also fixes a dashboard-typecheck regression in ModelSelectModal.tsx: the
new connection-filter toggle called hasEligibleConnectionForModel() with
activeProviders entries typed too narrowly to include
providerSpecificData, which real connection objects carry at runtime.
Adds:
- tests/unit/cache-config-route-8219.test.ts: GET/PUT resolve without
crashing, modelCatalogCacheTtlMs and idempotencyWindowMs round-trip.
- tests/unit/ui/cache-settings-tab-bounds-8219.test.tsx: CacheSettingsTab
min/max TTL bounds (100ms/60000ms) gate the Save button and surface a
validation message; in-bounds values PUT correctly.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
* chore(quality): rebaseline sections.ts for #8219 own-growth
---------
Co-authored-by: oyi77 <oyi77@users.noreply.github.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Use getEmbeddingDimension() in resolveEmbeddingSource so sqlite-vec can
create vec_memories before the first upsert, and abort reindex batches
when ensureReady returns ready=false instead of wasting embed credits.
ServiceSupervisor.start(), when probeBeforeSpawn detects an already-healthy
instance on the target port, "adopts" it (marks the service running without
spawning a duplicate that would die with EADDRINUSE). This adopt path calls
setToolStatus(tool, "running") with no pid argument at all, so this.pid stays
at its constructor default of null for the rest of that supervisor's life --
only the spawn path (a genuinely new child process) ever sets a real pid.
In production this "adopt" path is common, not an edge case: any embedded
sidecar service (cliproxy, 9router, bifrost, mux) whose child process
survives a `systemctl --user restart omniroute.service` gets adopted by the
new supervisor instance on the next start(), and its pid is lost from that
point on -- even though the service is genuinely healthy and running. The
observed symptom: a service shows state "running" but pid null, and
something downstream that keys liveness tracking off pid eventually treats
it as untrustworthy/stale despite nothing actually being wrong.
Fix: resolve the real pid of the process holding the port (via `lsof -ti
:<port>`, best-effort -- a lookup failure leaves pid null rather than
blocking adoption) and record it the same way the spawn path does.
An existing test asserted pid === null on adopt. That was accurate for the
old behavior but reflected a missing resolution, not a deliberate "adopted
services never get a pid" design choice -- updated its assertion to match
the corrected behavior and added a dedicated regression test proving the
resolved pid matches the real process actually bound to the port.
* fix(sse): Gemini TPM classification, combo-cooldown-wait for auto/quota-share, and target-timeout floor
Gemini TPM/RPM 429s were misclassified as QUOTA_EXHAUSTED because
sanitizeErrorMessage() truncates to the first line, hiding Google's
metric name and retry hint on lines 2-3. Added a rawMessage field
(internal-only, never reaches the client) and classifyGeminiQuotaMetricFromText()
to classify from the untruncated text, reordered ahead of the generic
credits/daily-quota checks.
Widened comboCooldownWaitEnabled (wait out a short transient cooldown
instead of crystallizing a 429/503) from quota-share-only to also cover
auto-strategy combos, and raised the wait ceiling to 65s/130s-budget/90s-cap
to match Gemini's ~60s TPM/RPM windows.
The per-target timeout (DEFAULT_COMBO_TARGET_TIMEOUT_MS, 120s) was shorter
than the new 130s cooldown-wait budget, so a target could get cut off
mid-wait with a synthetic 524 instead of completing the retry. Added
resolveComboTargetTimeoutMsForCombo()/isComboCooldownWaitEligible() in
comboConfig.ts to raise the per-target floor to budgetMs+buffer only for
wait-eligible strategies (auto/quota-share), verified live: a 12-request
concurrent burst against a TPM-exhausted combo went from 2/12 succeeding
(10 x 524) to 12/12 succeeding with zero 503/524.
Also: liveGeminiShared.ts's sendAndValidate now fails fast on a 503
instead of retrying past it, and the health dashboard + request logger
surface TPM stats alongside RPM/RPD.
Co-authored-by: Markus Hartung <markus.hartung@gmail.com>
* fix(sse): combo-exhausted rejection logs now capture request body + attempted models
recordRejectedRequestUsage() (the fast path for combo requests that never
reach handleChatCore, e.g. all targets locked by resilience cooldown)
hardcoded provider: "-" and never passed a request body to saveCallLog(),
so /dashboard/logs entries for these failures were nearly useless for
debugging: no way to see the client's request or which models were tried.
- recordRejectedRequestUsage() now accepts requestBody and persists it
through the existing saveCallLog() artifact mechanism (same path
handleChatCore's own logging uses).
- Added summarizeComboAttemptedModels(), which reads the combo's own model
list (always available, unlike the response's combo-diagnostics headers —
a model-level resilience-lockout skip never touches the
exhaustedProviders/exhaustedConnections sets those headers are built
from) to populate a real "provider" value instead of "-".
- Wired both into the call site in src/sse/handlers/chat.ts.
NOTE: unrelated to the Gemini TPM/combo-cooldown-wait fix on this branch —
landed here per operator request, to be split into its own branch/PR.
Co-authored-by: Markus Hartung <markus.hartung@gmail.com>
* feat(sse): synthetic streaming keep-alive event + 5-minute Gemini cooldown-wait ceiling
Many clients enforce a first-SSE-byte timeout, which made it unsafe to wait
out a longer upstream rate-limit cooldown on a streaming request — the
client would abandon the connection before any bytes arrived. This landed
in two parts:
1. Synthetic startup "thinking" event (OpenAI chat/completions format):
the already-existing withEarlyStreamKeepalive wrapper (open-sse/utils/
earlyStreamKeepalive.ts, wired into /v1/chat/completions, /v1/messages,
/v1/responses since #2544) opens the SSE stream immediately once a
request runs past its threshold, but only ever sent empty/no-op
keepalive frames. Added a `startupFrame` option (defaults to
`keepaliveFrame` — zero behavior change unless a route opts in) so the
very first frame can carry real content instead. Wired
OPENAI_STARTUP_THINKING_FRAME (a reasoning_content delta: "OmniRoute:
got request, sending to provider") into /v1/chat/completions only —
Claude Messages and Responses API formats both require a preceding
envelope event (message_start / response.created) that a synthetic
pre-dispatch frame can't safely fabricate without risking a duplicate
envelope once the real stream arrives, so those two routes keep their
existing (safe, proven) keepalive frames unchanged.
2. Raised the "wait out a known cooldown, then retry" ceiling to 5 minutes
for both retry mechanisms, now that a client-side first-byte timeout is
no longer a risk on the (opted-in) route:
- comboCooldownWait (auto/quota-share combos, open-sse/services/combo.ts):
maxWaitMs hard clamp raised 90s -> 300s (src/lib/resilience/settings/
normalize.ts); defaults raised to maxWaitMs:90s/maxAttempts:5/
budgetMs:300s. comboConfig.ts's resolveComboTargetTimeoutMsForCombo
already derives the per-target timeout floor from budgetMs, so it
tracks the new ceiling with no further changes.
- waitForCooldown (direct, non-combo model requests, src/sse/handlers/
chat.ts): this mechanism had NO cumulative cap before — only a
per-wait cap (maxRetryWaitMs) and a retry count (maxRetries), so
maxRetries x maxRetryWaitMs could exceed 5 minutes with no ceiling.
Added a budgetMs field (mirrors comboCooldownWait) to
WaitForCooldownSettings/CooldownAwareRetrySettings, threaded a
requestRetryBudgetLeftMs tracker through chat.ts's requestAttemptLoop
(mirrors combo.ts's comboCooldownBudgetLeftMs), and made
getCooldownAwareRetryDecision refuse to wait once the cumulative
budget is exhausted even if the single wait is under maxRetryWaitMs.
Co-authored-by: Markus Hartung <markus.hartung@gmail.com>
* fix(sse): extend the synthetic keep-alive thinking event to /v1/responses
Live incident (OpenClaw, log id 1784407081908-cbc24f): a /v1/responses
request to gemini/gemma-4-31b-it took 56s to produce a first byte and the
client disconnected (499 request_signal_aborted) — the same client-first-byte-
timeout problem the previous commit fixed for /v1/chat/completions, but
/v1/responses only had the generic bare-comment keepalive (no content), so it
wasn't covered.
Added RESPONSES_STARTUP_THINKING_FRAME: a self-contained synthetic reasoning
item (response.output_item.added -> reasoning_summary_part.added ->
reasoning_summary_text.delta -> reasoning_summary_part.done), opened AND
closed within this one frame rather than left dangling — it never carries a
response_id, so it can't collide with the real upstream response's own
independent response.created lifecycle that follows. Mirrors the abbreviated
delta+part.done close pattern open-sse/utils/stream.ts's own
emitSyntheticResponsesReasoningSummary already uses for real mid-stream
reasoning content.
Wired into src/app/api/v1/responses/route.ts via the startupFrame option
added in the previous commit.
Co-authored-by: Markus Hartung <markus.hartung@gmail.com>
* fix(sse): combo cooldown-wait vars reset every setTry, crystallizing a bogus 503 instead of waiting
Live incident (log id 1784416706646-51): a request to the "default" combo
(strategy=auto, maxSetRetries=3) hit a real Gemini TPM 429 on both gemma-4
targets, correctly classified as a short 40s rate_limit lockout — then
crystallized a 503 "all upstream accounts are inactive" in 6.9s instead of
ever reaching the cooldown-aware wait.
Root cause: `lastError`/`earliestRetryAfter`/`lastStatus` were declared with
`let` INSIDE the `for (setTry...)` loop body, so they reset to null at the
start of every set-try. When both targets lock out on setTry 0, every
subsequent setTry (1..maxSetRetries) pre-skips both targets via the
isModelLocked check with no real dispatch — so on the FINAL setTry (the only
one whose values the post-loop decision reads, since it's gated behind
`if (setTry < maxSetRetries) continue`), lastStatus was null, hitting the
"!lastStatus" branch (ALL_ACCOUNTS_INACTIVE 503) and completely bypassing the
comboCooldownWaitEnabled / earliestRetryAfter wait logic — even though a
real 429 with a known ~40s retry-after WAS observed on setTry 0.
This bug predates today's Gemini TPM work (any combo with maxSetRetries > 0
whose targets all lock out on the first pass was affected) but was masked in
existing tests: the "auto strategy (2 models...)" regression test uses
maxSetRetries: 0, so it only ever runs ONE setTry iteration and never
exercises the reset-on-retry path. It also explains why the dedicated
12-concurrent-request burst test passed cleanly — with concurrent requests,
timing variance meant some request's FINAL setTry iteration still had a live
target to dispatch to, giving lastStatus/earliestRetryAfter fresh data. A
single isolated request has no such luck.
Fix: hoist lastError/earliestRetryAfter/lastStatus to just inside
dispatchWithCooldownRetry, before the setTry loop, so they persist across
set-tries (still reset fresh on each recursive dispatchWithCooldownRetry()
call after a wait, which is correct). recordedAttempts/fallbackCount/
exhaustedProviders etc. are intentionally left per-iteration (unrelated to
this bug).
New regression test in tests/unit/combo-quota-share-cooldown-wait.test.ts
reproduces the exact live scenario (2 targets, both lock out on setTry 0,
maxSetRetries: 3) — confirmed red (503) against the pre-fix code, green
(200, waits and retries) against the fix.
Co-authored-by: Markus Hartung <markus.hartung@gmail.com>
* test(sse): extend live Gemini workload to Responses API + add large-context TPM test
Two additions to the live Gemini test suite, both live-verified against the
dev instance:
1. sendAndValidate() (tests/integration/liveGeminiShared.ts) now accepts an
apiFormat: "chat" | "responses" parameter, building the Responses-API
request shape (input array, max_output_tokens) and parsing its SSE events
(response.output_text.delta / response.reasoning_summary_text.delta /
response.completed) via the new readResponsesSSEStream(). Wired into two
new tests in live-gemini-workload.test.ts ([30]/[31]), mirroring the
existing Chat Completions streaming coverage. Verified live: 24/25 + 5/5
payloads succeeded end-to-end through the new code path (the one failure
was a ~300s test-client fetch timeout unrelated to the Responses API code
itself — a separate, not-yet-addressed test-harness limitation).
2. genHugeContextMessage() builds a single message large enough (~4
chars/token estimate) to approach or exceed Gemini's free-tier TPM ceiling
(16000 input tokens/min for gemma-4) by itself. Every other prompt
generator in this file tops out around 1-2k tokens — nowhere near that
ceiling — so none of the existing workload tests ever exercised a REAL TPM
429, only RPM-style rate limiting. tests/integration/gemini-large-context-tpm.test.ts
sends two ~12-13k-token requests back-to-back (comfortably exceeding
16000/min together) to exercise the full path against production Gemini:
TPM classification, the comboCooldownWait retry, and the synthetic
keep-alive frame on a genuinely slow request.
Co-authored-by: Markus Hartung <markus.hartung@gmail.com>
* fix(sse): abandoned combo target dispatch now observes its own per-target timeout, fixing a permanent "pending" dashboard leak
Live incident (dashboard log id 1784418258231-14961a, reported as "an ongoing
request even though there's already a 200"): a combo target dispatch
abandoned by comboTargetTimeoutMs (open-sse/services/combo/targetTimeoutRunner.ts)
left a permanent phantom "pending" entry in the dashboard, even after the
overall combo request had already succeeded via a different retry.
Root cause: chatCore.ts's createStreamController — and everything downstream
that depends on it (withRateLimit's Promise.race against Bottleneck,
acquireAccountSemaphore) — only ever watches clientRawRequest.signal, which
is the ORIGINAL client's request signal (set once via buildClientRawRequest
and reused unchanged across every target dispatch in a combo). It has no
connection to targetTimeoutRunner.ts's OWN AbortController
(target.modelAbortSignal), which is what actually fires when
comboTargetTimeoutMs (300s) elapses. src/sse/handlers/chat.ts's
handleSingleModel bridge between combo.ts and handleSingleModelChat received
`target.modelAbortSignal` but silently dropped it — never forwarded it
anywhere. So when a target got abandoned (e.g. stuck inside a wedged
Bottleneck rate-limiter queue, see the WEDGED force-reset log line from the
same incident), its per-target timeout fired and let the COMBO move on and
retry successfully elsewhere — but the abandoned dispatch's own promise
chain never learned it had been superseded, so it hung forever waiting on a
signal that was never going to fire, and trackPendingRequest(false) (the
finalize call) never ran.
Fix: thread target.modelAbortSignal through as a new modelAbortSignal
runtimeOption, and merge it into clientRawRequest.signal (via the existing
mergeAbortSignals helper from open-sse/executors/base.ts) right before
dispatch, so an abandoned target's own promise chain now observes its abort
and can reach its cleanup path — new resolveDispatchClientRawRequest() makes
this mechanically testable in isolation.
Co-authored-by: Markus Hartung <markus.hartung@gmail.com>
* fix(sse): combo cooldown-wait state recording, rate-limit wedge recovery, OpenAI-format SSE error frames
Five related fixes surfaced by live incidents (dashboard log ids 1784457764961-73,
1784465227489-a2cbc0, 1784504040241-6f8b9a) while validating the Gemini TPM/cooldown-wait
work on this branch against real OpenClaw traffic:
- combo.ts: the model-lockout bail-out branches in dispatchWithCooldownRetry never
recorded lastStatus, so once every target in a set hit an existing lockout the final
check crystallized a bogus ALL_ACCOUNTS_INACTIVE 503 instead of reaching the
cooldown-wait decision, even with a real 429 + short retry-after observed.
- combo.ts/combo/types.ts: the "all credentials cooling down" pre-dispatch rejection
(buildModelCooldownBody) nests its retry hint as error.retry_after/reset_seconds, not
the top-level retryAfter every other 429 shape uses — combo's extraction only read the
latter, so earliestRetryAfter stayed null for this shape even after lastStatus was fixed.
- rateLimitManager.ts: the wedge-recovery watchdog used disconnect(), which releases the
heartbeat timer but never rejects jobs already QUEUED on that instance — orphaned
dispatches hung until the outer ~300s per-target timeout, well past real clients'
patience. Switched to stop({ dropWaitingJobs: true }), safe because the wedge condition
already requires RUNNING===0 && EXECUTING===0.
- earlyStreamKeepalive.ts: the in-band error frame emitted after committing to a 200 SSE
stream was hardcoded to Anthropic's `event: error` convention for every route, including
the OpenAI-format ones (/v1/chat/completions, /v1/responses) where that framing is
either invisible or malformed to a plain data-line parser. Added per-route
OPENAI_CHAT_ERROR_FRAME / OPENAI_RESPONSES_ERROR_FRAME and wired them in.
- chatCore.ts: persisted a synthetic clientResponse error body even when the client had
already disconnected (AbortError) before that body was ever computed — misleading the
dashboard into showing "what the client received" for a response that was never sent.
Also: RequestLoggerDetail.tsx — Provider/Client Event Stream panes lost their collapse
toggle when StreamSection replaced the collapsible PayloadSection (692d6be80, unifying
active/finished request views) without carrying the toggle over.
Each fix has a TDD regression test with a confirmed red-before-green cycle.
Co-authored-by: Markus Hartung <markus.hartung@gmail.com>
* test(sse): free-tier model + gemma-4 TPM-ceiling benchmark harness
Adds a live benchmark comparing free models OmniRoute exposes across
configured providers plus previously-unexercised no-auth providers
(felo-web, aihorde, opencode, duckduckgo-web — none need a connection
row, they were just never tried). Reuses liveGeminiShared.ts's SSE
parsers and CASE_BUILDERS instead of duplicating them.
Also adds a targeted TPM-stress test firing back-to-back large-context
prompts at the gemma-4-31b model across its 3 free hosts (Gemini,
NVIDIA, AI Horde) to isolate whether the documented 16k-tokens/minute
free-tier ceiling is Gemini-specific enforcement or an inherent
model property.
FORCE_TOOL_CHOICE_REQUIRED is a test-only, default-off env flag added
to liveGeminiShared.ts and live-gemini-agentic-loop.test.ts for an
earlier live A/B comparison of tool_choice: required vs unset — kept
as a reusable knob for future runs.
Co-Authored-By: Markus Hartung <markus.hartung@gmail.com>
* test(sse): benchmark for the 2026-07-22 newly-enabled provider batch
Adds NEWLY_ENABLED_MODELS to freeModelBenchmarkShared.ts (Mistral
Leanstral, OpenRouter's live "free"-tagged roster, OpenCode Zen's
current free models — refetched live from
https://opencode.ai/zen/v1/models since the static catalog had
drifted) and a dedicated workload benchmark test for them.
Co-Authored-By: Markus Hartung <markus.hartung@gmail.com>
* test(sse): sync geminiRateLimitTracker tests with e74a1722b's corrected Gemma 4 limits
e74a1722b updated geminiRateLimits.json's gemma-4-* entries from the stale
15/1500/-1 (rpm/rpd/tpm) to the real published free-tier values
16000/14400/16000, but never updated the tests asserting the old numbers.
Surfaced by running the full test:unit suite as a post-rebase sanity check.
Co-Authored-By: Markus Hartung <markus.hartream@gmail.com>
* chore(quality): file-size baseline for own-growth (#8213)
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: Markus Hartung <markus.hartung@gmail.com>
Co-authored-by: Markus Hartung <markus.hartream@gmail.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
* fix(logs): stop an async EPIPE becoming an uncaughtException loop
A raw process.stderr.write into a broken pipe fails asynchronously, so the
try/catch around it never sees the failure. The stream emits 'error'; with no
listener on process.stderr Node re-throws it as an uncaughtException; the
framework's handler logs that through console.error; and the patched console
writes back into the same dead stream. That closes a self-sustaining loop.
Attach an 'error' listener to process.stdout and process.stderr. Node only
converts a stream 'error' into an uncaughtException when the emitter has no
listener, so the listener alone terminates the cycle. Measured in a spawn
harness over 1.5s: 3,387 uncaught exceptions before, 0 after.
Absorb EPIPE only. Attaching a listener otherwise makes every stream error on
those streams non-fatal process-wide, so ENOSPC, EBADF and the rest are
re-raised on a fresh stack to preserve today's crash semantics. The
accompanying test asserts that in a child process, because node:test
attributes any in-process uncaughtException to the running test.
Add a test-only reset() to undo the patched console and the listeners:
test:unit:fast runs --test-isolation=none, so leaked state would reach every
subsequent test file.
Refs #8181
* fix(logs): bound interceptor disk writes and self-heal a missing log dir
Two write-path defects in the same file, both independent of the loop itself.
writeEntry appended with no rate limit, so while the loop spun it wrote
unbounded lines to disk (4.3 GB in 90 minutes in the reported incident). Apply
the same policy #1006 established in structuredLogger -- 50 writes/sec, a 5s
dedup window, a bounded tracking map -- but scoped to `error` entries only.
That scoping is deliberate: structuredLogger applies its limiter solely to
error() and fatal(), whereas writeEntry serves all five of
log/info/warn/error/debug across ~800 non-error call sites. Capping those would
silently drop routine logging from the Console Log Viewer's file. A test
asserts non-error levels stay unlimited.
ensureDir() ran once in initConsoleInterceptor and never again, so a log
directory removed while the process was alive made every later append throw
ENOENT into a bare catch -- console file-logging then stopped permanently with
nothing surfaced anywhere. Recreate the directory and retry once, and report
the failure exactly once through the unpatched stderr so it cannot recurse
through the patched console or become a flood of its own.
Refs #8181
* fix(logs): skip raw stderr writes to a stream already known dead
error() and fatal() write with a raw process.stderr.write wrapped in
try {} catch {}. The comment on that line says the raw write exists to avoid
Next.js console patching "that triggers EPIPE loops" -- but on a broken pipe
the write fails asynchronously, so the catch never sees it, and the resulting
stream error is what ignites the loop.
Skip the write when the stream is already destroyed or ended, falling through
to the file sink as before. The listener added earlier is what breaks the
cycle; this stops the ignition point firing into a dead stream in the first
place. The existing try/catch is retained for the synchronous cases it always
covered.
The #1006 suppression policy and its call sites are untouched.
Refs #8181
* fix(logs): install the stdio guard independently of console interception
initConsoleInterceptor() returns early when APP_LOG_TO_FILE=false, and when the
log directory cannot be created. The stdio 'error' listeners were installed
after that return, so in those supported configurations no listener was
attached at all.
structuredLogger's raw stderr writes still happen there, and an ordinary broken
pipe raises an async EPIPE without destroyed or writableEnded being set first,
so the guard in safeStderrWrite does not cover it either. The loop this change
exists to prevent was therefore still reachable with file logging turned off.
Extract installStdioErrorGuard() and call it before the early return. It is
idempotent and cleared by reset(). A new test asserts, in a child process, that
both listeners are present when APP_LOG_TO_FILE=false.
Also restore APP_LOG_TO_FILE and APP_LOG_FILE_PATH in the test's after() hook.
test:unit:fast runs with --test-isolation=none, so the previous top-level
mutations leaked into later test files, leaving file logging enabled against a
path this file deletes.
Refs #8181
* fix(sse): collapse single-text-part Responses-API content to a plain string
Every /v1/responses request — even the simplest single-string input —
got 500'd by AI Horde's Aphrodite-backed facade. Root cause:
normalizeResponsesInputForChat() always wraps a plain string input as
`content: [{ type: "input_text", text: value }]` (a one-element array),
and openaiResponsesToOpenAIRequest() mapped that straight through to
`content: [{ type: "text", text: value }]` on the Chat Completions side
— an array. That's spec-valid (OpenAI's own API accepts both shapes),
but strict/naive OpenAI-compatible backends like AI Horde's only
implement the plain-string form and reject the array form outright.
A single-text-part array and a plain string are semantically
identical, so collapse is safe. Real multi-part messages (text+image,
text+file) are left untouched.
Regression test: tests/unit/openai-responses-single-text-content-string.test.ts
(RED before the fix — every collapsed-content assertion failed with an
object instead of a string; GREEN after).
Also adds a deeper AI Horde load-test suite (sequential/concurrent/
cross-model/sustained-throughput/new-capable-model-candidates) that
surfaced this bug via real live traffic after Behemoth-X-123B was
temporarily added to the "default" combo for evaluation.
Co-Authored-By: Markus Hartung <markus.hartung@gmail.com>
* fix(sse): unsupportedParams provider-level fallback for aihorde's live-discovered models
Real OpenClaw traffic against the newly-added Behemoth-X-123B combo
target kept 500ing on every attempt even after the Responses-API
content-array fix landed. The pipeline artifact showed why: `tools`
was still present, unstripped, in the request actually sent to AI
Horde's Aphrodite backend.
Root cause: `unsupportedParams: ["tools", "tool_choice",
"parallel_tool_calls"]` was only declared on the 3 models statically
listed in the aihorde registry entry (Cydonia-24B, Skyfall-31B,
google/gemma-4-31b). AI Horde uses `passthroughModels: true` — its
live worker roster changes constantly — so Behemoth-X-123B, like every
other dynamically-discovered aihorde model, had no model-specific
unsupportedParams entry, and getUnsupportedParams() returned [] for
it. But "the workers run raw text-completion backends" (no tool
calling) is true of every model AI Horde serves, not just the 3
catalogued ones.
Adds a provider-level `unsupportedParams` fallback on RegistryEntry,
checked by getUnsupportedParams() after the per-model lookup misses.
Set on the aihorde entry so it covers its entire live-discovered
roster, present and future, without needing a static per-model catalog
entry for each one.
Regression test: tests/unit/aihorde-tools-unsupported-provider-fallback.test.ts
(RED before the fix — Behemoth-X and deepseek-v4-flash both returned
[] instead of the stripped param list; GREEN after, with a control
case confirming the fallback doesn't leak to unrelated providers).
Co-Authored-By: Markus Hartung <markus.hartung@gmail.com>
* fix(sse): flatten leftover tool-call history when stripping unsupported tools
Third bug in the same AI Horde/Behemoth-X saga: even after tools/
tool_choice were correctly stripped from the live request (previous
fix), real combo traffic still 500'd. The conversation history itself
carried a prior turn's role:"assistant" tool_calls and role:"tool"
result messages, left over from before the combo failed over from a
tool-capable model (Gemini) to a non-tool-capable one (AI Horde). Its
raw completion backend doesn't understand those message shapes at all,
independent of whether live `tools` is present — confirmed by
reproducing with a role:"tool" message and NO tools param at all.
flattenToolHistory() (open-sse/utils/flattenToolHistory.ts) already
existed for exactly this, fully unit-tested — it just had zero call
sites anywhere in the request pipeline. Extracts the unsupported-params
strip into a small testable module
(open-sse/handlers/chatCore/unsupportedParamsStrip.ts, following the
existing chatCore god-file decomposition pattern e.g.
executorClientHeaders.ts) that now also flattens tool-call history
whenever "tools" was among the stripped params.
Regression test: tests/unit/chatcore-unsupported-params-strip.test.ts
(RED before the fix — the flattening test failed with the raw
tool_calls array still present; GREEN after). All 434 existing
chatcore-*.test.ts tests still pass.
Co-Authored-By: Markus Hartung <markus.hartung@gmail.com>
* fix(sse): gate tool-history flattening on unsupported, not on stripped-this-request
The previous commit's flattening only fired when "tools" was actually
present-and-stripped on THIS request. A second live reproduction
against AI Horde had no live `tools` param at all — only stale
tool_calls/tool-result messages inherited from before a combo
failover — and still 500'd, because that condition never triggered.
A model that can't do tool calling can't do it whether or not the
current request happens to carry a `tools` array. Gate on the
unsupported-params list itself (unsupported.includes("tools")) instead
of the subset that was actually present-and-deleted this time.
Regression test added to the same file (RED before — the no-live-tools
case left tool_calls/role:"tool" untouched; GREEN after). All 435
chatcore-*.test.ts still pass.
Co-Authored-By: Markus Hartung <markus.hartung@gmail.com>
* fix(sse): skip tool-incapable combo targets, error clearly on direct requests
Two complementary fixes for a model that structurally can't do tool
calling at all (e.g. AI Horde's raw completion backends) rather than
silently degrading — following up on the earlier strip/flatten fix,
which stopped the crashes but let a tool-incapable target still get
selected and return a 200 that narrates a fake tool call in prose
instead of erroring or being skipped.
1. Root cause, combo routing: getResolvedModelCapabilities()'s
`supportsTools` resolution only checked per-model registry entries,
synced capabilities, and static specs — none of which exist for a
dynamically-discovered model (AI Horde's passthroughModels roster
changes as workers come and go). It fell through to
heuristicToolCalling(), which optimistically defaults to `true` for
any unrecognized model (TOOL_CALLING_UNSUPPORTED_PATTERNS is empty).
Added a provider-level fallback reusing the same unsupportedParams
signal the request-time strip already relies on. This makes the
EXISTING filterTargetsByRequestCompatibility (comboStructure.ts) —
which already correctly excludes non-tool-capable targets when a
request requires tools — actually work for these models; no combo.ts
changes were needed, it was only ever fed bad capability data.
2. Direct/pinned requests: filterTargetsByRequestCompatibility only
protects combo routing. A direct request naming an exact
tool-incapable model has no other target to fail over to — added
checkToolCallingRequiredButUnsupported (chatCore/toolCallingRequiredCheck.ts),
gated on isCombo: false, returning a clear 400 instead of a 200 that
silently can't do what was asked.
Regression tests (both RED before, GREEN after):
- tests/unit/model-capabilities-provider-unsupported-tools.test.ts
- tests/unit/chatcore-tool-calling-required-check.test.ts
All 463 chatcore-*/model-capabilities-*.test.ts and 31 combo
compatibility-filter tests still pass.
Co-Authored-By: Markus Hartung <markus.hartung@gmail.com>
* fix(sse): correct handleChatCore return shape for the tool-calling-blocked error
handleChatCore's documented contract is `{ success, response, status,
error }`, not a raw Response — returning `new Response(...)` directly
(copied from a different early-return whose surrounding context turned
out not to share this function's top-level contract) produced "No
response is returned from route handler ... Expected a Response object
but received 'undefined'" and a bare 500 with an empty body, caught
immediately when verifying the previous commit live.
Uses createErrorResult() (already used by the adjacent
translation-failure branch a few lines up) instead of hand-building the
Response, matching the same pattern already established in this
function for early error returns.
Co-Authored-By: Markus Hartung <markus.hartung@gmail.com>
* fix(sse): scope Responses single-text-content collapse to providers that need it
The single-text-part content array -> plain string collapse (added for AI
Horde's Aphrodite facade, which 500s on the array form) was applied
unconditionally to every provider, silently breaking the standard OpenAI
array-shaped content contract that other providers and existing tests
depend on. Added RegistryEntry.requiresPlainStringContent, gated the
collapse on it (true only for aihorde), and threaded modelInfo.provider
through responsesHandler -> responsesApiHelper -> the translator so the
real /v1/responses call site can identify the provider.
Co-Authored-By: Markus Hartung <markus.hartream@gmail.com>
---------
Co-authored-by: Markus Hartung <markus.hartung@gmail.com>
Co-authored-by: Markus Hartung <markus.hartream@gmail.com>
