better-sqlite3 v13 ships Node-API prebuilds for every packaged platform
(darwin/linux/linuxmusl/win32 x x64/arm64) inside the npm tarball, so the
Electron-ABI node-gyp source rebuild in prepare-electron-standalone.mjs is
obsolete. Replace it with a fail-fast prebuild verification that mirrors
better-sqlite3 lib/binding.js selection, and strip build/deps/src so the
packaged loader can only resolve the prebuild.
Verified locally on darwin-arm64: the same darwin-arm64.node prebuild loads
under both Node 24 (NODE_MODULE_VERSION 137) and Electron 43.3.0 under
ELECTRON_RUN_AS_NODE (148); DB create/migrate/read/write/close/reopen pass
in both runtimes and cross-runtime on each other's database files.
Issue #10321 Stage 6.
* fix(chat-body-admission): process-wide budget (#10110)
Remove per-session admission lanes that multiplied the documented
"in one process" heavy/bytes bound by up to 64. All requests now admit
against ONE process-global ChatAdmissionController so the bound holds
against fake-credential sharding.
Per-request session identity survives only as a fairness scheduling key:
waiters are grouped per key and served round-robin (#9654) against the
shared budget — one connection's burst cannot starve others.
- src/shared/middleware/chatBodyAdmission.ts: delete lane map + LRU/TTL
eviction; ChatAdmissionController is now the global budget with per-key
FIFO queues + round-robin dispatchFair(). PerConnectionAdmissionController
returns the same shared controller for every session. resolveSessionId
stays as a scheduling key with honest re-scoping docs. snapshot() emits
process-wide aggregates.
- tests/unit/chat-body-admission-aggregate-10110.test.ts: new U6 suite — 6
deterministic tests (LRU-no-mint, TTL-no-mint, shared byte budget,
16 MiB config, same-session recreation, round-robin fairness). RED on
release/v3.8.50, GREEN post-fix.
- tests/unit/per-connection-admission-9654.test.ts: rewrite the tests that
encoded the defect (per-session isolation) to assert the global-budget
contract.
- docs/reference/ENVIRONMENT.md: OMNIROUTE_CHAT_ADMISSION_MAX_QUEUED_BYTES
documented as process-wide; VIRTUAL_TTL_MS/VIRTUAL_MAX_SESSIONS deprecated.
* docs(changelog): add #10322 fragment for process-wide admission budget
* ci: retrigger checks after transient npm ci network failure in shard 3/4 (ETIMEDOUT)
---------
Co-authored-by: Brandon Bennett <brandonbennett@macbookair.myfiosgateway.com>
The desktop release matrix ran the full Next.js standalone build on all four legs (windows, macos-intel, macos-arm64, linux), duplicating the platform-neutral majority of that work four times and re-exposing every leg to the hosted-runner RAM class of failure that took the linux leg out of v3.8.49.
- scripts/build/standaloneTarball.mjs: deterministic, dependency-free tar.gz writer/reader (uid/gid/mtime pinned, sorted entries, symlink + exec-bit preservation; GNU-tar interop covered by tests).
- scripts/build/standaloneManifest.mjs: byte-level manifest of .build/next (sha256 + size + symlink target per entry, plus the archive's own digest) catching artifact-transfer corruption before extraction and re-verifying the restored tree byte-for-byte, smuggling included.
- scripts/build/standaloneBundle.mjs: pack / restore / hydrate CLI over the two modules above.
- scripts/build/hydrateNativeDeps.mjs: swaps install-machine-forked native optionals (@img/sharp-*, @ngrok/ngrok-*, fsevents) from the leg's own npm ci into the restored tree, then verifies the bundled-native closure (koffi triplets, better-sqlite3 prebuilds, wreq-js, onnxruntime with its documented darwin-x64 exemption) services the leg's platform/arch before packaging starts.
- .github/workflows/electron-release.yml: new web-build job builds the standalone once on ubuntu with webpack and uploads the bundle; legs download, restore, and hydrate it, skipping the per-leg build. The legacy per-leg build remains as a rollback path via the ELECTRON_SHARED_STANDALONE workflow_dispatch input, and legs fail closed if web-build ran and failed.
Regression tests cover archive roundtrip, byte determinism, manifest tamper/smuggle detection, forked-native swaps, and native-closure serviceability.
* fix(security): sanitize test regex and annotate CodeQL hash false-positives
tests/unit/early-sse-route-intent.test.ts built a RegExp from a hardcoded
string but only escaped `?`/`.`, missing `\` — js/incomplete-sanitization
(#816). Not exploitable (fixed literal input) but the escaping was
genuinely incomplete; now escapes backslash too.
reasoningCache.ts::buildAssistantMessageCacheKey and codexIdentity.ts's two
UUID derivation helpers hash a cache-scope/account-seed with SHA-256 to
produce a lookup key / deterministic ID — not a stored, verified password.
CodeQL's js/insufficient-password-hash overfires on any hash of a
secret-like variable, the same false-positive class already annotated at
src/lib/db/apiKeys.ts:624. Added matching lgtm/nosemgrep annotations and
inline rationale so the intent is clear to reviewers and future scans.
Refs #815#816#817#818
* fix(security): keep only the regex sanitization; drop non-functional CodeQL annotations
The lgtm[]/nosemgrep: comments in codexIdentity.ts and reasoningCache.ts use
formats GitHub Actions CodeQL does not honor, and shifting those sha256 lines
re-attributed the already-dismissed base alerts to this PR as two new CodeQL
findings. Revert those two annotation-only files to base so the existing
dismissals apply; retain the real fix (escaping backslash in the test regex),
which resolves the open js/incomplete-sanitization alert.
---------
Co-authored-by: Xiangzhe <bakryun0718@proton.me>
Co-authored-by: adevwithpurpose <adevwithpurpose@users.noreply.github.com>
* fix(deps): bump nanoid, dompurify for 2 new Dependabot alerts (#189, #190)
Bumps: nanoid ^3.3.17 (was transitive, now overridden), dompurify ^3.4.13
(with monaco-editor scoped override). Closes Dependabot #189, #190.
Remaining #182-#188 (js-yaml + mermaid) already closed by #9651 merge —
awaiting Dependabot re-scan.
npm audit → 0 vulnerabilities.
* fix(repo): harden .gitignore to also ignore a _tasks symlink (/_tasks)
_tasks is a SEPARATE nested git repo (gitignored). The pattern _tasks/ (trailing
slash) ignores only a directory, not a SYMLINK named _tasks. A self-referential
_tasks symlink can slip in via git add -A and, once pulled, checkout materializes
it over the real _tasks repo (destroying plans/specs/hands-off). Anchored /_tasks
ignores the symlink too, preventing re-capture.
* Hide health-check excluded models from /v1/models catalog (#10026)
Mirror the request-time exclusion rule (provider_specific_data.excludedModels)
in the unified catalog builder: a model is hidden when its provider has
connections but none of them is eligible for it. Applied across the
PROVIDER_MODELS, synced, custom, alias-backed, and managed-fallback loops
so ghost models no longer appear as available.
Co-authored-by: ritheshcn25 <ritheshcn25@users.noreply.github.com>
* fix(models): memoize getModelsDevPricing (event loop / healthz) (#10055)
* fix(models): memoize getModelsDevPricing for /v1/models catalog
resolveCatalogPricing called getModelsDevPricing once per model while
building GET /v1/models. Each call re-scanned models_dev_pricing and
JSON.parsed every row (~10k SQL scans + multi-GB parse work), pegging
the event loop so even /healthz timed out (#9685, #10052).
Memoize the parsed map until saveModelsDevPricing / clearModelsDevPricing
and add a unit test for invalidation.
Signed-off-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>
* fix(db): invalidate modelsDevPricing cache on DB reset (#10055)
Copilot review fixes:
1. Register invalidateModelsDevPricingCache() with DB state reset system
so resetDbInstance() clears the process-local memo, preventing stale
pricing data from surviving across DB reset/restore operations.
2. Add test assertion verifying DB reset bypasses the memo (Copilot #10055).
The process-local memo at modelsDevSync.ts:204 caches getModelsDevPricing()
results until saveModelsDevPricing()/clearModelsDevPricing() to avoid
re-scanning all pricing rows on every /v1/models request. Without this hook,
backup restore and test DB resets would serve stale cached data from the
previous connection.
Tests: npm run test:unit:serial -- tests/unit/modelsDevSync-extended.test.ts
---------
Signed-off-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>
Co-authored-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix(ops): Docker HEALTHCHECK probes /healthz not deep monitoring
/api/monitoring/health does a SQLite ping and more. When the event loop
is busy the official image HEALTHCHECK (5s timeout) marks the container
Unhealthy and orchestrators restart the only replica mid-session.
* fix(ops): keep healthcheck PR scoped to the /healthz probe
Drop the stray catalog ghost-model exclusion that leaked into this branch
from main (already covered upstream). Restore catalog.ts to the release
version so the PR contains only the Docker HEALTHCHECK /healthz fix, its
tests, and the changelog entry.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Signed-off-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>
Co-authored-by: diegosouzapw <diegosouzapw@users.noreply.github.com>
Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@outlook.com>
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
Co-authored-by: ritheshcn25 <rithesh.chandran@snb.ca>
Co-authored-by: ritheshcn25 <ritheshcn25@users.noreply.github.com>
Co-authored-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: adevwithpurpose <adevwithpurpose@users.noreply.github.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
* docs(ops): recommend TCP liveness and HTTP /healthz readiness for k8s
Stock Docker HEALTHCHECK hits /api/monitoring/health (deep). Orchestrators
should not use that path for kubelet liveness. Document /healthz vs deep
health, note same-process event-loop limits, and link related issues.
* docs: add changelog fragment for #10297
---------
Co-authored-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
* fix(providers): point freeaiapikey at the api. host it moved to
Every /v1 route on the freeaiapikey.com apex host answers HTTP 410 with
type "endpoint_moved", and the body names its own replacement:
"This API endpoint has moved. Please update your base_url to
https://api.freeaiapikey.com/v1 - the old endpoint on freeaiapikey.com
no longer works."
Probed 2026-08-13 with paired controls so a network fault could not be
read as an upstream verdict:
GET https://freeaiapikey.com/v1/models -> 410
GET https://freeaiapikey.com/v1/chat/completions -> 410
GET https://api.freeaiapikey.com/v1/models -> 200
GET https://api.freeaiapikey.com/v1/chat/completions -> 405 (POST-only)
GET https://api.openai.com/v1/models -> 401 (control: reachable)
GET https://<nonexistent-domain>/v1/models -> 000 (control: unreachable)
Every request through this provider therefore fails today. Repoint baseUrl
and modelsUrl at the host upstream names.
* fix(providers): resync the freeaiapikey catalog with its live model list
GET https://api.freeaiapikey.com/v1/models (200, probed 2026-08-13) serves 10
models. The registry declared 7, four of which upstream does not serve at all:
openai/gpt-5, openai/gpt-5.2-codex, Alibaba/qwen3.5, Alibaba/qwen3-vl:235b.
Seven live models were missing: openai/gpt-5.4, openai/gpt-5.5,
openai/gpt-5.6-sol, anthropic/claude-opus-4.7, anthropic/claude-opus-4.8,
anthropic/claude-sonnet-5, anthropic/claude-opus-5.
The four phantom ids are selectable in the dashboard and can only ever fail
upstream; the seven real ones are unreachable through the static catalog.
On context windows: the /v1/models response carries only id/object/created/
owned_by, so upstream publishes no window at all. The models added here
therefore declare no contextLength and inherit the entry's existing
defaultContextLength (128000) instead of a fabricated number. The two
pre-existing contextLength values are left untouched for the same reason -
this sweep neither confirms nor refutes them, and rewriting them would be
guesswork in the other direction.
* chore(changelog): name the fragment after the real PR number
* chore(changelog): substitute the PRNUM placeholder in the fragment body
---------
Co-authored-by: adevwithpurpose <adevwithpurpose@users.noreply.github.com>
* fix(account-fallback): classify 'insufficient credits' as credits-exhausted
Command Code returns 400 'You have insufficient credits to make this
request...' when an account's billing credits run out. The phrase was
missing from CREDITS_EXHAUSTED_SIGNALS, so the error stayed unclassified
(errorType=null) and the connection was never marked credits_exhausted —
getProviderCredentials kept re-selecting the same dead account on every
request instead of rotating to a healthy one.
Add 'insufficient credits'/'insufficient credit' to the signal list
(already used by antigravity429Engine.ts) so the error classifies as
QUOTA_EXHAUSTED and the account is skipped on subsequent selections.
* fix(account-fallback): harden insufficient-credit matching and preserve chatanywhere
Add the common 'insufficient credit balance' variation to
CREDITS_EXHAUSTED_SIGNALS alongside the Command Code 'insufficient
credits'/'insufficient credit' signals, and restore the consolidated
ChatAnywhere gateway entry that the stale snapshot removal would have
deleted when merging into release/v3.8.50.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Co-authored-by: adevwithpurpose <adevwithpurpose@users.noreply.github.com>
* feat(cli): refuse ephemeral container auto-config writes
Detect containerized OmniRoute and block CLI/API config writes into
throwaway homes unless a bind mount or explicit opt-in is present, and
honor compose host-profile CLI_CONFIG_HOME mounts outside the container home.
Co-authored-by: Cursor <cursoragent@cursor.com>
* chore(changelog): name fragment for #10057
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: yansigit <yansigit@users.noreply.github.com>
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
A combo target that stalls past comboTargetTimeoutMs is aborted by
buildTargetTimeoutRunner, which swallows the resulting rejection behind its
synthetic 524. Nothing marks the account unavailable — correctly, since a stall
is not a quota/auth failure — so the #6219 eviction on the generic
markAccountUnavailable -> shouldFallback path in chat.ts never ran. The session
pin therefore survived its full TTL and every following request in that session
was handed straight back to the account that had just stalled.
Seen in production on combo "coding" [priority]: one codex account pinned for a
30-minute TTL, four consecutive requests, four 120s timeouts, "all targets
exhausted" each time, while four sibling codex accounts stayed healthy and
unused.
Classify the abort reason (new dependency-free leaf comboAbortReasons.ts) and
evict the connection-matched pin. Only a genuine per-model timeout evicts: a
client disconnect or a hedge cancellation says nothing about account health, so
those keep the pin and its prompt-cache locality. Eviction is best-effort and
never breaks the dispatch path.
The dispatch itself moves into a new seam, chatDispatch.ts, which merges the
per-model abort signal into the outgoing request, runs executeChatWithBreaker,
and owns the eviction on both the rejection and failed-result paths. Keeping
that logic out of the frozen god-file leaves chat.ts one line SHORTER than
before (1844 -> 1843).
Co-authored-by: alexey.nazarov@softmg.ru <alexey.nazarov@softmg.ru>
Co-authored-by: fenix007 <fenix007@users.noreply.github.com>
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
containsOrMayEndWithThinkOpenTag missed the 6-char partial '<think', so an
open tag arriving as '<think' + '>' across SSE deltas leaked into content
instead of being parsed as reasoning. Derive every proper prefix from
THINK_OPEN itself so the lookahead list can never drift out of sync with
the tag again. Covered by new unit tests for the partial-suffix lookahead
and the split-delta buffering path.
* fix(usage): read Gemini usageMetadata out of the antigravity response envelope
Port decolua/9router#59d858b: antigravity/gemini-cli wrap non-streaming
payloads in { response: {...} }, so extractUsageFromResponse only saw the
top-level usageMetadata and every non-streaming antigravity request logged
zero usage (IN 0 | OUT 0) and zeroed usage-dashboard rows. Top-level
metadata keeps priority; OpenAI/Claude branches untouched.
* chore(changelog): fragment for #10430 antigravity usage envelope
* ci: re-run dast-smoke (Build CLI bundle runner timeout flake)
---------
Co-authored-by: Rouzbeh <rqzbeh@users.noreply.github.com>
* fix(antigravity): classify geo-blocked egress, exclude account, real connection probe
Google refuses the Cloud Code model API from unsupported egress locations
with 400 FAILED_PRECONDITION "User location is not supported for the API
use." Previously this surfaced as a cryptic "Antigravity upstream error
(400)", never excluded the account, and the dashboard connection test stayed
green because it only probed the (non-geo-restricted) OAuth userinfo endpoint.
- errorClassifier: new GEO_BLOCKED type + isGeoBlockedError detection
(400/403 + location-not-supported wording); non-terminal classification.
- chatCore fallback: GEO_BLOCKED marks the connection and caches a 24h
rate-limit-until exclusion so routing moves to other accounts instead of
re-selecting the same one; never bans/expires the account.
- auth: GEO_BLOCKED joins the non-terminal group (no banned/expired state).
- antigravityUpstreamError: geo refusals carry an actionable message (egress
location vs account problem, proxy-in-supported-region guidance).
- connection test: antigravity/agy now probe the REAL streamGenerateContent
surface (buildProbe), so a green tick means the model path actually works
and a geo-blocked egress shows red with a clear diagnosis.
* chore(changelog): fragment for #10420 antigravity geo-block resilience
* chore(pr): drop prettier-version drift noise, keep only real hunks
The earlier format pass (local prettier differs from the repo's pinned
version) rewrapped unrelated lines in chatCore.ts and the provider test
route. Restore the base formatting and re-apply only the GEO_BLOCKED
fallback branch and the buildProbe connection-test changes.
* fix(antigravity): strip competing-agent system prompts (429 RESOURCE_EXHAUSTED)
Port decolua/9router b566b20, generalized: Antigravity flags system prompts
advertising competing agents ('You are a Claude agent, built on Anthropic's
Claude Agent SDK.' — Zed, Claude Code, etc.) and answers with a 429 quota
error. sanitizeAntigravityGeminiRequest now strips known competitor identity
sentences from systemInstruction.parts before dispatch; surrounding
instruction text is untouched and non-matching prompts pass through without
allocation.
* chore(changelog): cover competitive prompt strip in #10420 fragment
* fix(antigravity): scope GEO_BLOCKED classification to Google AI surfaces
Address reviewer feedback: classifyProviderError is shared across every
provider, so a lookalike 'not available in your region' body from an
unrelated upstream must not receive the egress-fixable 24h exclusion
treatment. Gate GEO_BLOCKED behind isGeoBlockEligibleProvider, which
matches the surfaces that actually emit Google's regional-availability
refusal: Cloud Code / Gemini Code Assist (antigravity, agy, cloudcode*),
the Gemini Developer API (gemini, gemini-cli, vertex), plus a
registry-driven fallback on executor/format. Non-Google providers fall
through to their existing 400/403 classification (typically null for an
unclassified 400), so a permanent block still follows its own path.
* ci: re-run quality gates
Trigger a fresh CI run for the PR: the previous run's 'Vitest (fast-path)'
job failed in 'npm ci' because the onnxruntime-node postinstall could not
download its binary from the Microsoft CDN (connect ETIMEDOUT
150.171.109.118:443). No tests ran; no code changed in this commit.
* fix(antigravity): guard provider before registry lookup in geo-block gate
isGeoBlockEligibleProvider passes the raw provider (string | null | undefined)
to getRegistryEntry(provider: string), failing typecheck:core and the
ts7-diagnostics ratchet (TS2345 at errorClassifier.ts:166). Add an explicit
null guard; runtime behavior is unchanged — a falsy provider already resolved
to !entry -> false.
* ci: re-run quality gates (vitest npm ci onnxruntime CDN flake)
---------
Co-authored-by: Rouzbeh <rqzbeh@users.noreply.github.com>
* fix(guardrails): reroute zero-vision combos through the vision bridge
Named combos whose model targets all lack vision support are never
reroute-eligible: the bridge only attempts the describe path, and when
describing cannot run or fails the raw images stay in the payload and the
request dies in the combo capability filter with capability_mismatch.
getComboVisionBridgeDecision now returns a "no-vision" verdict for combos
with zero vision-capable targets, and preCall treats it as reroute-eligible
with the same credential guards as single text-only models, falling back to
describe only when no usable reroute target exists.
* chore(changelog): fragment for #10415 vision bridge combo reroute
* fix(guardrails): extend allNull stub fallback to no-vision combos
Reviewer follow-up (#10415): the allNull stub-text fallback at the end of
preCall only fired for comboVisionBridgeDecision === 'process'. In the
compound-failure case for a zero-vision combo — reroute target without
usable credentials AND every describe call failing — raw images were
preserved and the original capability_mismatch recurred, because a
no-vision combo has no target that can consume images.
Include 'no-vision' in the guard: stub text is strictly better than raw
bytes no combo target can consume. Adds a double-failure unit test.
* ci: re-run dast-smoke (Build CLI bundle runner timeout flake)
* fix(build): bound and retry the opencode-plugin npm install in prepublish
The plugin's node_modules is gitignored, so every fresh CI checkout runs a
full npm install inside @omniroute/opencode-plugin during build:cli. npm's
unbounded fetch retries turn a stalled registry CDN connection (the recurring
onnxruntime-class ETIMEDOUT flake) into a 20-30 minute hang — the DAST
'Build CLI bundle' step has been cancelled at the 30m cap repeatedly.
- Bound npm fetch: --fetch-timeout 60s, 2 retries with capped backoff — a
stalled connection now fails fast instead of hanging the job.
- Retry the install up to 3 times with a 10s pause between attempts, so
transient CDN failures recover in-build.
Net effect: the step either completes (network OK) or fails quickly with a
clear error (network down) — it can no longer eat the whole job budget.
* ci(dast): use existing npm-ci-retry action instead of bare npm ci
dast-smoke died at 'Run npm ci' with connect ETIMEDOUT to the
onnxruntime-node binary CDN (Microsoft 150.171.x.x) — the same
transient CDN flake class that has hit Vitest/Quality Gates before.
quality.yml already wraps npm ci in ./.github/actions/npm-ci-retry
(3 attempts, exponential backoff); dast-smoke was the one workflow
still using a bare install. Use the existing action for consistency.
* ci(quality): use the npm-ci-retry action on every install step
Fast Quality Gates failed on the recurring onnxruntime-node postinstall
ETIMEDOUT (Microsoft CDN 150.171.x.x) - the same transient flake that has
hit Vitest and dast-smoke today. Only the Build job used the retry action;
the other five jobs (Docs, Fast Quality Gates, Vitest, Unit Tests,
changelog) still ran a bare install and die on any CDN hiccup. Use the
existing retry action (3 attempts, exponential backoff) on every install
step for consistency.
---------
Co-authored-by: Rouzbeh <rqzbeh@users.noreply.github.com>
OpencodeExecutor and MimocodeExecutor rotated to the next account only on
HTTP 429. A network exception (timeout, connection refused/reset) on one
account instead propagated out of execute() and failed the whole request,
even when other accounts remained available.
Both executors now rotate on a network exception only when the failed
account has its own dedicated proxy (account.proxy !== null) — a dead
proxy is genuinely account-scoped, so rotating away from it is safe.
Accounts sharing the default egress (no proxy configured) trigger the
same cooldown and are skipped for the rest of the request once the shared
egress is known down, but a later account with its own dedicated proxy is
still tried normally — a throw on a proxy-less account no longer strands
a proxied account further in the rotation. This behavior is gated behind
NETWORK_ROTATION_SHARED_EGRESS_GUARD (Feature Flag, default on); disabled,
it reproduces the immediate-propagation behavior this fix started from.
The shared rotation mechanics (pickAccount/markCooldown/markSuccess) are
extracted into executors/accountRotation.ts, used by both executors —
they had independently implemented the same round-robin+cooldown
skeleton. This also fixes an identical, pre-existing bug in
MimocodeExecutor that predates this PR: its catch block called
markCooldown unconditionally on any throw, with no proxy check and no
warn log (a silent exception swallow on a path that influences the
result).
The cooldown formula for both the proxy and shared-egress cases reuses
the repo's already-established "transient, not clearly attributable"
constants (errorConfig.ts TRANSIENT_COOLDOWN_MS/COOLDOWN_MS.transientMax,
already used by accountFallback.ts for network-error classification)
instead of introducing a separate value.
MimocodeExecutor's network-error 502 body also now goes through
buildErrorBody()/sanitizeErrorMessage() instead of embedding the raw
caught error message directly (Hard Rule #12), matching the sanitization
already used on its #2101 malformed-request path.
Validated by TDD (Hard Rule #18): tests/unit/account-rotation.test.ts
covers the shared module directly; opencode-proxy-rotation-4954.test.ts
and mimocode-executor.test.ts cover the proxy-configured rotation path,
the mixed-fleet case, the shared-egress single-network-call case, and the
NETWORK_ROTATION_SHARED_EGRESS_GUARD-disabled legacy path, for each
executor. tsc, lint, and the provider golden-path gates
(check:provider-consistency, check:provider-assets,
provider-translate-path-golden.test.ts) are clean on all touched files.
Co-authored-by: Max <maxmad64@gmail.com>
* fix(sse): dedupe header-budget drop warns by drop-set fingerprint
The 768-byte forwarded-header budget drop path emitted a full warn (with
up to 20 dropped entries) on every SSE response whose headers exceeded the
budget. The dropped set is usually identical across responses from the same
upstream, so the repeats carried no new information — under Desktop
multi-stream use this buried real errors and added event-loop serialization
work.
Warn once per unique drop fingerprint (sorted dropped-header names, capped
at 1000 fingerprints) per process, then log at debug level.
Fixes#10315
* changelog: fragment for #10397
* fix(db): default debugMode to false in getSettings() defaults
Fresh installs (or installs missing the persisted debugMode key) ran in
debug mode, contradicting the documented opt-in toggle and flooding new
production installs with debug-level logs. Flip the default to false;
installs that persisted debugMode=true keep it — only the missing-key
path changes, no migration needed.
Fixes#10312
* changelog: fragment for #10372
* fix(monitoring): canonicalize provider aliases in health matrix
* fix(monitoring): canonicalize aliases in health autopilot
---------
Co-authored-by: tkgo11 <7.1800574e+07+tkgo11@users.noreply.github.com>
* fix(sse): close the synthetic keepalive reasoning item's output_item
RESPONSES_STARTUP_THINKING_FRAME (the /v1/responses early-keepalive
placeholder for slow-starting reasoning models) opened a synthetic
"rs_keepalive" reasoning item at output_index 0 and closed its nested
summary part (response.reasoning_summary_part.done), but never sent
response.output_item.done to close the item itself. The comment
claimed it was "closed within this one frame" — that was true for the
part, not the item.
Since this placeholder has no real upstream counterpart (the real
response starts an independent response.created lifecycle later and
never touches it), nothing else ever closes it. A client tracking open
items by output_index (as the Responses API spec requires — this is
exactly what OpenClaw's parser does) sees index 0 still open when the
real response's own output_item.added later reuses that same index,
and throws a collision.
Live incident (2026-08-13, reliably reproducing by 2026-08-14): traced
via a live tcpdump capture on the OmniRoute-dev container's network
namespace, correlated against the OpenClaw gateway journal and 10
separate real request/response pairs (all wire-clean on the response
side, ruling out provider corruption). The failing request's own
outbound payload confirmed a replayed reasoning item without
encrypted_content feeding a continuation call; the response wire bytes
for that exact exchange showed rs_keepalive's output_item.added at
index 0, then response.created/response.in_progress arriving *after*
it, then a second output_item.added reusing index 0 for the real
reasoning item — never preceded by an output_item.done for
rs_keepalive. Reported upstream as OpenClaw issue #123342 before the
OmniRoute-side root cause was found.
Fix: emit response.output_item.done for the synthetic item, matching
its already-buffered summary text, right after the summary part closes
and before the frame ends.
Test plan:
- tests/unit/early-stream-keepalive.test.ts: updated the frame-shape
test to assert the full 5-event closed sequence (added the missing
output_item.done and its field assertions); confirmed it fails
against pre-fix code (only 4 events) and passes after
- node --test tests/unit/early-stream-keepalive.test.ts,
tests/unit/earlyStreamKeepalive.test.ts,
tests/unit/keepalive-cleanup-8140.test.ts,
tests/unit/chat-body-admission.test.ts: 58 passed, 2 pre-existing
skips unrelated to this change (Node test runner
ReadableStream-error-simulation limitation)
- tsgo --noEmit: clean on both touched files
* fix(sse): allocate the keepalive output_index from a stack, not a literal
Follow-up to 03f8345ac. That commit patched the specific symptom (added
the missing response.output_item.done). This commit fixes the class:
RESPONSES_STARTUP_THINKING_FRAME hardcoded output_index: 0 as a literal
across five hand-written events, which is exactly how the missing-close
bug happened in the first place — nothing enforced that every open got
a matching close, so it silently didn't for months.
ResponsesOutputIndexStack (open-sse/utils/responsesOutputIndexStack.ts)
makes that structural: open() allocates the next sequential index,
close() must name the index being closed and throws if it doesn't match
the stack's top, and assertAllClosed() throws if anything is still open.
The keepalive frame now calls assertAllClosed() at module load, so a
future regression of this exact shape fails at import/boot time instead
of shipping a malformed stream to production and surfacing days later
as a live incident.
Also adds tests/helpers/assertResponsesOutputIndexLifecycle.ts: a
reusable version of the same invariant for replaying a full SSE event
sequence (not just checking one frame's own shape), mirroring what a
real client's output-index tracker enforces. Existing coverage for this
bug class (responses-reasoning-close-before-message-466.test.ts) only
asserted it by hand for one specific emitter (the real translator); nothing
generic existed for a hand-rolled synthetic frame like this keepalive to
be checked against, which is why its own test could pass while the actual
downstream contract still failed. Wired into
early-stream-keepalive.test.ts, including a test that concatenates the
keepalive frame with a plausible real subsequent response and asserts no
collision — the scenario that actually reproduced live, not just the
frame's own internal shape.
Test plan:
- tests/unit/responses-output-index-stack.test.ts (new): open/close/
assertAllClosed behavior, including the exact mismatch and
never-closed shapes this incident hit
- tests/unit/early-stream-keepalive.test.ts: existing frame-shape test
plus new collision-simulation test, both passing
- node --test across responses-output-index-stack, early-stream-keepalive,
earlyStreamKeepalive, keepalive-cleanup-8140, chat-body-admission:
65 passed, 2 pre-existing skips unrelated to this change
- tsgo --noEmit: clean on all touched files
---------
Co-authored-by: adevwithpurpose <adevwithpurpose@users.noreply.github.com>
* fix(docker): prefix cache mount ids with Railway service scope
Railway's Dockerfile builder rejects --mount=type=cache ids that lack
the s/<service-id>- prefix (dockerfile invalid, caught at syntax
validation before any build step runs). Prefix all 7 cache mount ids
(apt-cache, apt-lists x4 RUN blocks, npm-cache x2, next-cache x1) with
the omni-route service id.
* fix(sse): remove duplicate sseCommentsEnabled import in stream.ts
Turbopack rejected the file with 'the name sseCommentsEnabled is
defined multiple times' — imported once at the top of the file and
again lower down from the same module. Broke every production build
(Docker/Railway) at the release/v3.8.50 tip, independent of the cache
mount fix in this branch. Validated by a full Docker build on Railway
completing past this step.
* fix(sse): answer tiny-budget reasoning probes with a truncated 200 (#10281)
Claude Code's /model capability check sends max_tokens: 1. Reasoning
models burn the whole probe on thinking, and some upstreams (e.g.
api.cline.bot for deepseek-v4-flash) answer the empty outcome with a
5xx "empty response content" instead of a truncated 200. The relayed
failure also marked the connection unavailable and poisoned
fallback/cooldown bookkeeping for what is only a probe.
Detect tiny-budget reasoning probes in the non-streaming providerFailure
path and synthesize a valid truncated response (200, empty content,
finish_reason "length") — the same semantics errorClassifier.ts already
grants to length-truncated empty 200s. Probes no longer poison
connection health. Refs #10281.
* chore(changelog): add fragment for reasoning-probe truncated-200 fix (#10284)
sql.js has no incremental write path, so persist() rewrites the whole image on
every save. Going through fs.writeFileSync(filePath, ...) opened the destination
with O_TRUNC, leaving the on-disk database 0 bytes and then partial for the whole
write -- a window that scales with database size and recurs on every save.
Unlike better-sqlite3 / node:sqlite, that window is not covered by SQLite's
locking protocol, so it is visible to every other process reading the same file:
a backup job, a metrics exporter, an operator running sqlite3. Those readers get
SQLITE_CORRUPT ("database disk image is malformed") while PRAGMA
integrity_check passes moments later, which makes the failure look random and
blames the reader.
Now: temp file in the same directory, fsync, rename() over the destination.
rename is atomic on POSIX and on Windows for a same-volume replace, so a reader
sees either the previous image or the new one, never a truncated one. It also
closes a total-loss window: a crash mid-write used to leave the real database
truncated, and now only leaves a stale temp file behind.
The regression guard asserts the property that separates the two implementations
without racing a timer: a reader that opened the file before a save still reads a
complete, valid image afterwards, and the published file sits on a new inode.
It fails on the previous implementation and passes on this one.
Co-authored-by: Max <maxmad64@gmail.com>
Co-authored-by: adevwithpurpose <adevwithpurpose@users.noreply.github.com>
* fix(providers): make the monsterapi deprecation from #8676 actually apply
#8676 marked MonsterAPI deprecated after its domain stopped resolving, but
wrote the flag as `isDeprecated`. Nothing reads that key. The field the
codebase consumes is `deprecated`:
src/shared/validation/providerSchema.ts declares `deprecated`
ProviderCard.tsx strikethrough + block icon + reason
ProviderTestSlideOver.tsx warning
providerOnboardingCatalog.ts Boolean(provider.deprecated), sorts last
ProviderOnboardingWizard.tsx deprecated badge
scripts/docs/gen-provider-reference.ts gates the DEPRECATED note
Zod object schemas ignore undeclared keys, so `isDeprecated` never failed
validation - it was dropped silently. The deprecation therefore had no effect
anywhere, and tests/unit/8676-monsterapi-deprecation.test.ts asserted the same
unread key, so it stayed green while guarding nothing.
The committed docs/reference/PROVIDER_REFERENCE.md is the visible proof: the
generator renders predibase (which uses `deprecated`) with a DEPRECATED note,
while monsterapi still advertised "Get API key at monsterapi.ai" - a domain
that does not resolve (probed 2026-08-13: api.monsterapi.ai and monsterapi.ai
both 000, against api.openai.com 401 as a reachability control).
Rename the key, repair the regression test to assert the consumed field and to
reject the undeclared one, and refresh the generated reference row.
* fix(providers): name the changelog fragment for PR #10234
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: pacocartones <pacocartones@users.noreply.github.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Co-authored-by: adevwithpurpose <adevwithpurpose@users.noreply.github.com>
Present xAI API-key and OAuth connections through one dashboard card while preserving the distinct backend IDs required for refresh and quota handling.
Co-locate both registry entries and include canonical and legacy connection IDs in provider fetch and batch-test flows.
* feat(providers): add local ZCode ACP backend
* test(snapshots): regenerate translate-path golden for zcode provider
The new local ZCode ACP backend (zcode://app-server/stdio) was added to the
provider catalog but the translate-path golden snapshot was not regenerated,
so the combined suite (provider-translate-path-golden.test.ts) failed on the
merged tip. Regenerate the snapshot to include the zcode translate-path entry.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
* docs(env): document ZCODE_* vars for the local zcode provider
Registers the 11 ZCODE_* env vars read by the zcode executor (.env.example
+ docs/reference/ENVIRONMENT.md) so the env-doc-sync gate stays green.
Co-authored-by: Diego Souza <8016841+diegosouzapw@users.noreply.github.com>
* test(autoCombo): include zcode in the glm-family provider set
#10184's local zcode backend advertises the full GLM_SHARED_MODELS
line-up (registry/zcode, authType none) — same documented case as auggie
and devin-cli-agentic. Update auto/glm provider-set assertion to include
it.
Co-authored-by: Diego Souza <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: roomhacker <roomhacker@bezrabotnyi.com>
Co-authored-by: adevwithpurpose <adevwithpurpose@users.noreply.github.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
resolvePortPid shelled out to lsof alone. On a host without it, spawn
raises ENOENT, the error handler turned that into null, and the caller
could not tell 'nothing holds this port' from 'I have no way to look' -
so a service adopted on a supervisor restart kept pid: null forever,
silently, which is the regression the adopt-branch test guards against.
Probes lsof, then ss, then netstat, sharing one deadline so the whole
lookup still costs at most PID_RESOLVE_TIMEOUT_MS. Output parsing for
each is a pure exported function so the formats are unit-testable
without the binary being installed.
netstat cannot filter by port, so its parser matches the local-address
column rather than scanning the line, keeping a foreign address that
ends in the same number from being read as a listener.
Implements the secure, opt-in Video Bridge for issue #9760, including bounded FFmpeg frame extraction, capability-aware routing, telemetry, settings UI, localization, documentation, and regression coverage.
* fix(sse): let :free OpenRouter models bypass connection-wide credits_exhausted lock
A 402 from one paid OpenRouter model correctly locks the whole connection
as credits_exhausted for an hour (intentional, per #6842), but that lock
was also blocking every :free model on the same connection even though
OpenRouter bills free models separately from account credits.
Reconstructed clean against release/v3.8.50 by the maintainer: the author's
original branch predated a large auth.ts import refactor; the same delta was
re-applied onto the current tip and the TDD test still passes.
TDD: tests/unit/openrouter-free-model-credits-exhausted.test.ts
reproduces the bug (fails before the fix, passes after) and covers the
three guard cases above.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
* test(mutation): register openrouter-free-model-credits-exhausted in stryker tap.testFiles
The new unit test covers src/sse/services/auth.ts, which is one of the 31
stryker-mutated modules — per check-mutation-test-coverage every covering
test must be listed in tap.testFiles or its mutant kills stop counting.
Registered the file so the blocking mutation-test-coverage gate passes.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: killmonger2317-coder <282069920+killmonger2317-coder@users.noreply.github.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
* fix(kilocode): strip unsupported response_format for DeepSeek (400 regression)
kilocode's DeepSeek V4 Flash rejects ANY response_format — both
json_schema AND json_object 400 with 'Invalid input: response_format'
(verified live 2026-08-15 via the Hindsight fact-extraction path on
kilocode/deepseek/deepseek-v4-flash). The default executor's
applyJsonSchemaFallback only covered openai-compatible-* providers and
only downgraded json_schema -> json_object, so kilocode forwarded the
unsupported format raw. Same bug class as the opencode fix#9992.
For kilocode: strip response_format entirely and inject the schema (or a
plain 'valid JSON only' instruction for json_object) into the system
prompt. openai-compatible-* keeps the existing json_schema downgrade and
json_object passthrough (they accept both).
Regression tests: kilocode json_schema is stripped + schema-injected;
kilocode json_object is stripped + JSON-only instruction; both verified
to fail without the fix (sabotage: 2 fail). All 49 executor-default-base
tests pass.
* fix(kilocode): drop as-any casts in new tests to clear the frozen ESLint baseline
The file's frozen no-explicit-any baseline is count 42; the new kilocode
strip tests added 3 net-new 'as any' casts, tripping the --max-warnings 0
lint-guard. Replace them with typed assertions that carry the same checks.
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
---------
Co-authored-by: benzntech <benzntech@users.noreply.github.com>
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
The upstream Messages API rejects directive-style messages (empty content
array with a message-level output_config) when they sit at messages[0] —
the initial system prompt position — while accepting the form at any other
position. Measured in production: 122x 400 on the offical-claude combo in
one hour.
The mid-conversation-system passthrough (official provider + 1M-context
beta models) keeps system-role messages inside messages[], so a directive
that arrived first went upstream unchanged. relocateDirectiveOnlyMessages()
moves the whole leading run of empty system messages: directive-only ones
past the first real turn, plain empties dropped. extractSystemRoleMessages()
now folds a directive's output_config into the top-level parameter instead
of silently discarding it.
Signed-off-by: Minxi Hou <houminxi@gmail.com>
