* fix(guardrails): reroute zero-vision combos through the vision bridge
Named combos whose model targets all lack vision support are never
reroute-eligible: the bridge only attempts the describe path, and when
describing cannot run or fails the raw images stay in the payload and the
request dies in the combo capability filter with capability_mismatch.
getComboVisionBridgeDecision now returns a "no-vision" verdict for combos
with zero vision-capable targets, and preCall treats it as reroute-eligible
with the same credential guards as single text-only models, falling back to
describe only when no usable reroute target exists.
* chore(changelog): fragment for #10415 vision bridge combo reroute
* fix(guardrails): extend allNull stub fallback to no-vision combos
Reviewer follow-up (#10415): the allNull stub-text fallback at the end of
preCall only fired for comboVisionBridgeDecision === 'process'. In the
compound-failure case for a zero-vision combo — reroute target without
usable credentials AND every describe call failing — raw images were
preserved and the original capability_mismatch recurred, because a
no-vision combo has no target that can consume images.
Include 'no-vision' in the guard: stub text is strictly better than raw
bytes no combo target can consume. Adds a double-failure unit test.
* ci: re-run dast-smoke (Build CLI bundle runner timeout flake)
* fix(build): bound and retry the opencode-plugin npm install in prepublish
The plugin's node_modules is gitignored, so every fresh CI checkout runs a
full npm install inside @omniroute/opencode-plugin during build:cli. npm's
unbounded fetch retries turn a stalled registry CDN connection (the recurring
onnxruntime-class ETIMEDOUT flake) into a 20-30 minute hang — the DAST
'Build CLI bundle' step has been cancelled at the 30m cap repeatedly.
- Bound npm fetch: --fetch-timeout 60s, 2 retries with capped backoff — a
stalled connection now fails fast instead of hanging the job.
- Retry the install up to 3 times with a 10s pause between attempts, so
transient CDN failures recover in-build.
Net effect: the step either completes (network OK) or fails quickly with a
clear error (network down) — it can no longer eat the whole job budget.
* ci(dast): use existing npm-ci-retry action instead of bare npm ci
dast-smoke died at 'Run npm ci' with connect ETIMEDOUT to the
onnxruntime-node binary CDN (Microsoft 150.171.x.x) — the same
transient CDN flake class that has hit Vitest/Quality Gates before.
quality.yml already wraps npm ci in ./.github/actions/npm-ci-retry
(3 attempts, exponential backoff); dast-smoke was the one workflow
still using a bare install. Use the existing action for consistency.
* ci(quality): use the npm-ci-retry action on every install step
Fast Quality Gates failed on the recurring onnxruntime-node postinstall
ETIMEDOUT (Microsoft CDN 150.171.x.x) - the same transient flake that has
hit Vitest and dast-smoke today. Only the Build job used the retry action;
the other five jobs (Docs, Fast Quality Gates, Vitest, Unit Tests,
changelog) still ran a bare install and die on any CDN hiccup. Use the
existing retry action (3 attempts, exponential backoff) on every install
step for consistency.
---------
Co-authored-by: Rouzbeh <rqzbeh@users.noreply.github.com>
OpencodeExecutor and MimocodeExecutor rotated to the next account only on
HTTP 429. A network exception (timeout, connection refused/reset) on one
account instead propagated out of execute() and failed the whole request,
even when other accounts remained available.
Both executors now rotate on a network exception only when the failed
account has its own dedicated proxy (account.proxy !== null) — a dead
proxy is genuinely account-scoped, so rotating away from it is safe.
Accounts sharing the default egress (no proxy configured) trigger the
same cooldown and are skipped for the rest of the request once the shared
egress is known down, but a later account with its own dedicated proxy is
still tried normally — a throw on a proxy-less account no longer strands
a proxied account further in the rotation. This behavior is gated behind
NETWORK_ROTATION_SHARED_EGRESS_GUARD (Feature Flag, default on); disabled,
it reproduces the immediate-propagation behavior this fix started from.
The shared rotation mechanics (pickAccount/markCooldown/markSuccess) are
extracted into executors/accountRotation.ts, used by both executors —
they had independently implemented the same round-robin+cooldown
skeleton. This also fixes an identical, pre-existing bug in
MimocodeExecutor that predates this PR: its catch block called
markCooldown unconditionally on any throw, with no proxy check and no
warn log (a silent exception swallow on a path that influences the
result).
The cooldown formula for both the proxy and shared-egress cases reuses
the repo's already-established "transient, not clearly attributable"
constants (errorConfig.ts TRANSIENT_COOLDOWN_MS/COOLDOWN_MS.transientMax,
already used by accountFallback.ts for network-error classification)
instead of introducing a separate value.
MimocodeExecutor's network-error 502 body also now goes through
buildErrorBody()/sanitizeErrorMessage() instead of embedding the raw
caught error message directly (Hard Rule #12), matching the sanitization
already used on its #2101 malformed-request path.
Validated by TDD (Hard Rule #18): tests/unit/account-rotation.test.ts
covers the shared module directly; opencode-proxy-rotation-4954.test.ts
and mimocode-executor.test.ts cover the proxy-configured rotation path,
the mixed-fleet case, the shared-egress single-network-call case, and the
NETWORK_ROTATION_SHARED_EGRESS_GUARD-disabled legacy path, for each
executor. tsc, lint, and the provider golden-path gates
(check:provider-consistency, check:provider-assets,
provider-translate-path-golden.test.ts) are clean on all touched files.
Co-authored-by: Max <maxmad64@gmail.com>
* fix(sse): dedupe header-budget drop warns by drop-set fingerprint
The 768-byte forwarded-header budget drop path emitted a full warn (with
up to 20 dropped entries) on every SSE response whose headers exceeded the
budget. The dropped set is usually identical across responses from the same
upstream, so the repeats carried no new information — under Desktop
multi-stream use this buried real errors and added event-loop serialization
work.
Warn once per unique drop fingerprint (sorted dropped-header names, capped
at 1000 fingerprints) per process, then log at debug level.
Fixes#10315
* changelog: fragment for #10397
* fix(db): default debugMode to false in getSettings() defaults
Fresh installs (or installs missing the persisted debugMode key) ran in
debug mode, contradicting the documented opt-in toggle and flooding new
production installs with debug-level logs. Flip the default to false;
installs that persisted debugMode=true keep it — only the missing-key
path changes, no migration needed.
Fixes#10312
* changelog: fragment for #10372
* fix(monitoring): canonicalize provider aliases in health matrix
* fix(monitoring): canonicalize aliases in health autopilot
---------
Co-authored-by: tkgo11 <7.1800574e+07+tkgo11@users.noreply.github.com>
* fix(sse): close the synthetic keepalive reasoning item's output_item
RESPONSES_STARTUP_THINKING_FRAME (the /v1/responses early-keepalive
placeholder for slow-starting reasoning models) opened a synthetic
"rs_keepalive" reasoning item at output_index 0 and closed its nested
summary part (response.reasoning_summary_part.done), but never sent
response.output_item.done to close the item itself. The comment
claimed it was "closed within this one frame" — that was true for the
part, not the item.
Since this placeholder has no real upstream counterpart (the real
response starts an independent response.created lifecycle later and
never touches it), nothing else ever closes it. A client tracking open
items by output_index (as the Responses API spec requires — this is
exactly what OpenClaw's parser does) sees index 0 still open when the
real response's own output_item.added later reuses that same index,
and throws a collision.
Live incident (2026-08-13, reliably reproducing by 2026-08-14): traced
via a live tcpdump capture on the OmniRoute-dev container's network
namespace, correlated against the OpenClaw gateway journal and 10
separate real request/response pairs (all wire-clean on the response
side, ruling out provider corruption). The failing request's own
outbound payload confirmed a replayed reasoning item without
encrypted_content feeding a continuation call; the response wire bytes
for that exact exchange showed rs_keepalive's output_item.added at
index 0, then response.created/response.in_progress arriving *after*
it, then a second output_item.added reusing index 0 for the real
reasoning item — never preceded by an output_item.done for
rs_keepalive. Reported upstream as OpenClaw issue #123342 before the
OmniRoute-side root cause was found.
Fix: emit response.output_item.done for the synthetic item, matching
its already-buffered summary text, right after the summary part closes
and before the frame ends.
Test plan:
- tests/unit/early-stream-keepalive.test.ts: updated the frame-shape
test to assert the full 5-event closed sequence (added the missing
output_item.done and its field assertions); confirmed it fails
against pre-fix code (only 4 events) and passes after
- node --test tests/unit/early-stream-keepalive.test.ts,
tests/unit/earlyStreamKeepalive.test.ts,
tests/unit/keepalive-cleanup-8140.test.ts,
tests/unit/chat-body-admission.test.ts: 58 passed, 2 pre-existing
skips unrelated to this change (Node test runner
ReadableStream-error-simulation limitation)
- tsgo --noEmit: clean on both touched files
* fix(sse): allocate the keepalive output_index from a stack, not a literal
Follow-up to 03f8345ac. That commit patched the specific symptom (added
the missing response.output_item.done). This commit fixes the class:
RESPONSES_STARTUP_THINKING_FRAME hardcoded output_index: 0 as a literal
across five hand-written events, which is exactly how the missing-close
bug happened in the first place — nothing enforced that every open got
a matching close, so it silently didn't for months.
ResponsesOutputIndexStack (open-sse/utils/responsesOutputIndexStack.ts)
makes that structural: open() allocates the next sequential index,
close() must name the index being closed and throws if it doesn't match
the stack's top, and assertAllClosed() throws if anything is still open.
The keepalive frame now calls assertAllClosed() at module load, so a
future regression of this exact shape fails at import/boot time instead
of shipping a malformed stream to production and surfacing days later
as a live incident.
Also adds tests/helpers/assertResponsesOutputIndexLifecycle.ts: a
reusable version of the same invariant for replaying a full SSE event
sequence (not just checking one frame's own shape), mirroring what a
real client's output-index tracker enforces. Existing coverage for this
bug class (responses-reasoning-close-before-message-466.test.ts) only
asserted it by hand for one specific emitter (the real translator); nothing
generic existed for a hand-rolled synthetic frame like this keepalive to
be checked against, which is why its own test could pass while the actual
downstream contract still failed. Wired into
early-stream-keepalive.test.ts, including a test that concatenates the
keepalive frame with a plausible real subsequent response and asserts no
collision — the scenario that actually reproduced live, not just the
frame's own internal shape.
Test plan:
- tests/unit/responses-output-index-stack.test.ts (new): open/close/
assertAllClosed behavior, including the exact mismatch and
never-closed shapes this incident hit
- tests/unit/early-stream-keepalive.test.ts: existing frame-shape test
plus new collision-simulation test, both passing
- node --test across responses-output-index-stack, early-stream-keepalive,
earlyStreamKeepalive, keepalive-cleanup-8140, chat-body-admission:
65 passed, 2 pre-existing skips unrelated to this change
- tsgo --noEmit: clean on all touched files
---------
Co-authored-by: adevwithpurpose <adevwithpurpose@users.noreply.github.com>
* fix(docker): prefix cache mount ids with Railway service scope
Railway's Dockerfile builder rejects --mount=type=cache ids that lack
the s/<service-id>- prefix (dockerfile invalid, caught at syntax
validation before any build step runs). Prefix all 7 cache mount ids
(apt-cache, apt-lists x4 RUN blocks, npm-cache x2, next-cache x1) with
the omni-route service id.
* fix(sse): remove duplicate sseCommentsEnabled import in stream.ts
Turbopack rejected the file with 'the name sseCommentsEnabled is
defined multiple times' — imported once at the top of the file and
again lower down from the same module. Broke every production build
(Docker/Railway) at the release/v3.8.50 tip, independent of the cache
mount fix in this branch. Validated by a full Docker build on Railway
completing past this step.
* fix(sse): answer tiny-budget reasoning probes with a truncated 200 (#10281)
Claude Code's /model capability check sends max_tokens: 1. Reasoning
models burn the whole probe on thinking, and some upstreams (e.g.
api.cline.bot for deepseek-v4-flash) answer the empty outcome with a
5xx "empty response content" instead of a truncated 200. The relayed
failure also marked the connection unavailable and poisoned
fallback/cooldown bookkeeping for what is only a probe.
Detect tiny-budget reasoning probes in the non-streaming providerFailure
path and synthesize a valid truncated response (200, empty content,
finish_reason "length") — the same semantics errorClassifier.ts already
grants to length-truncated empty 200s. Probes no longer poison
connection health. Refs #10281.
* chore(changelog): add fragment for reasoning-probe truncated-200 fix (#10284)
sql.js has no incremental write path, so persist() rewrites the whole image on
every save. Going through fs.writeFileSync(filePath, ...) opened the destination
with O_TRUNC, leaving the on-disk database 0 bytes and then partial for the whole
write -- a window that scales with database size and recurs on every save.
Unlike better-sqlite3 / node:sqlite, that window is not covered by SQLite's
locking protocol, so it is visible to every other process reading the same file:
a backup job, a metrics exporter, an operator running sqlite3. Those readers get
SQLITE_CORRUPT ("database disk image is malformed") while PRAGMA
integrity_check passes moments later, which makes the failure look random and
blames the reader.
Now: temp file in the same directory, fsync, rename() over the destination.
rename is atomic on POSIX and on Windows for a same-volume replace, so a reader
sees either the previous image or the new one, never a truncated one. It also
closes a total-loss window: a crash mid-write used to leave the real database
truncated, and now only leaves a stale temp file behind.
The regression guard asserts the property that separates the two implementations
without racing a timer: a reader that opened the file before a save still reads a
complete, valid image afterwards, and the published file sits on a new inode.
It fails on the previous implementation and passes on this one.
Co-authored-by: Max <maxmad64@gmail.com>
Co-authored-by: adevwithpurpose <adevwithpurpose@users.noreply.github.com>
* fix(providers): make the monsterapi deprecation from #8676 actually apply
#8676 marked MonsterAPI deprecated after its domain stopped resolving, but
wrote the flag as `isDeprecated`. Nothing reads that key. The field the
codebase consumes is `deprecated`:
src/shared/validation/providerSchema.ts declares `deprecated`
ProviderCard.tsx strikethrough + block icon + reason
ProviderTestSlideOver.tsx warning
providerOnboardingCatalog.ts Boolean(provider.deprecated), sorts last
ProviderOnboardingWizard.tsx deprecated badge
scripts/docs/gen-provider-reference.ts gates the DEPRECATED note
Zod object schemas ignore undeclared keys, so `isDeprecated` never failed
validation - it was dropped silently. The deprecation therefore had no effect
anywhere, and tests/unit/8676-monsterapi-deprecation.test.ts asserted the same
unread key, so it stayed green while guarding nothing.
The committed docs/reference/PROVIDER_REFERENCE.md is the visible proof: the
generator renders predibase (which uses `deprecated`) with a DEPRECATED note,
while monsterapi still advertised "Get API key at monsterapi.ai" - a domain
that does not resolve (probed 2026-08-13: api.monsterapi.ai and monsterapi.ai
both 000, against api.openai.com 401 as a reachability control).
Rename the key, repair the regression test to assert the consumed field and to
reject the undeclared one, and refresh the generated reference row.
* fix(providers): name the changelog fragment for PR #10234
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: pacocartones <pacocartones@users.noreply.github.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Co-authored-by: adevwithpurpose <adevwithpurpose@users.noreply.github.com>
Present xAI API-key and OAuth connections through one dashboard card while preserving the distinct backend IDs required for refresh and quota handling.
Co-locate both registry entries and include canonical and legacy connection IDs in provider fetch and batch-test flows.
* feat(providers): add local ZCode ACP backend
* test(snapshots): regenerate translate-path golden for zcode provider
The new local ZCode ACP backend (zcode://app-server/stdio) was added to the
provider catalog but the translate-path golden snapshot was not regenerated,
so the combined suite (provider-translate-path-golden.test.ts) failed on the
merged tip. Regenerate the snapshot to include the zcode translate-path entry.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
* docs(env): document ZCODE_* vars for the local zcode provider
Registers the 11 ZCODE_* env vars read by the zcode executor (.env.example
+ docs/reference/ENVIRONMENT.md) so the env-doc-sync gate stays green.
Co-authored-by: Diego Souza <8016841+diegosouzapw@users.noreply.github.com>
* test(autoCombo): include zcode in the glm-family provider set
#10184's local zcode backend advertises the full GLM_SHARED_MODELS
line-up (registry/zcode, authType none) — same documented case as auggie
and devin-cli-agentic. Update auto/glm provider-set assertion to include
it.
Co-authored-by: Diego Souza <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: roomhacker <roomhacker@bezrabotnyi.com>
Co-authored-by: adevwithpurpose <adevwithpurpose@users.noreply.github.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
resolvePortPid shelled out to lsof alone. On a host without it, spawn
raises ENOENT, the error handler turned that into null, and the caller
could not tell 'nothing holds this port' from 'I have no way to look' -
so a service adopted on a supervisor restart kept pid: null forever,
silently, which is the regression the adopt-branch test guards against.
Probes lsof, then ss, then netstat, sharing one deadline so the whole
lookup still costs at most PID_RESOLVE_TIMEOUT_MS. Output parsing for
each is a pure exported function so the formats are unit-testable
without the binary being installed.
netstat cannot filter by port, so its parser matches the local-address
column rather than scanning the line, keeping a foreign address that
ends in the same number from being read as a listener.
Implements the secure, opt-in Video Bridge for issue #9760, including bounded FFmpeg frame extraction, capability-aware routing, telemetry, settings UI, localization, documentation, and regression coverage.
* fix(sse): let :free OpenRouter models bypass connection-wide credits_exhausted lock
A 402 from one paid OpenRouter model correctly locks the whole connection
as credits_exhausted for an hour (intentional, per #6842), but that lock
was also blocking every :free model on the same connection even though
OpenRouter bills free models separately from account credits.
Reconstructed clean against release/v3.8.50 by the maintainer: the author's
original branch predated a large auth.ts import refactor; the same delta was
re-applied onto the current tip and the TDD test still passes.
TDD: tests/unit/openrouter-free-model-credits-exhausted.test.ts
reproduces the bug (fails before the fix, passes after) and covers the
three guard cases above.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
* test(mutation): register openrouter-free-model-credits-exhausted in stryker tap.testFiles
The new unit test covers src/sse/services/auth.ts, which is one of the 31
stryker-mutated modules — per check-mutation-test-coverage every covering
test must be listed in tap.testFiles or its mutant kills stop counting.
Registered the file so the blocking mutation-test-coverage gate passes.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: killmonger2317-coder <282069920+killmonger2317-coder@users.noreply.github.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
* fix(kilocode): strip unsupported response_format for DeepSeek (400 regression)
kilocode's DeepSeek V4 Flash rejects ANY response_format — both
json_schema AND json_object 400 with 'Invalid input: response_format'
(verified live 2026-08-15 via the Hindsight fact-extraction path on
kilocode/deepseek/deepseek-v4-flash). The default executor's
applyJsonSchemaFallback only covered openai-compatible-* providers and
only downgraded json_schema -> json_object, so kilocode forwarded the
unsupported format raw. Same bug class as the opencode fix#9992.
For kilocode: strip response_format entirely and inject the schema (or a
plain 'valid JSON only' instruction for json_object) into the system
prompt. openai-compatible-* keeps the existing json_schema downgrade and
json_object passthrough (they accept both).
Regression tests: kilocode json_schema is stripped + schema-injected;
kilocode json_object is stripped + JSON-only instruction; both verified
to fail without the fix (sabotage: 2 fail). All 49 executor-default-base
tests pass.
* fix(kilocode): drop as-any casts in new tests to clear the frozen ESLint baseline
The file's frozen no-explicit-any baseline is count 42; the new kilocode
strip tests added 3 net-new 'as any' casts, tripping the --max-warnings 0
lint-guard. Replace them with typed assertions that carry the same checks.
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
---------
Co-authored-by: benzntech <benzntech@users.noreply.github.com>
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
The upstream Messages API rejects directive-style messages (empty content
array with a message-level output_config) when they sit at messages[0] —
the initial system prompt position — while accepting the form at any other
position. Measured in production: 122x 400 on the offical-claude combo in
one hour.
The mid-conversation-system passthrough (official provider + 1M-context
beta models) keeps system-role messages inside messages[], so a directive
that arrived first went upstream unchanged. relocateDirectiveOnlyMessages()
moves the whole leading run of empty system messages: directive-only ones
past the first real turn, plain empties dropped. extractSystemRoleMessages()
now folds a directive's output_config into the top-level parameter instead
of silently discarding it.
Signed-off-by: Minxi Hou <houminxi@gmail.com>
Auto combos (virtual auto/* pools via virtualFactory and pure-auto named
combos via expandAutoComboCandidatePool) expanded their candidate pool from
the provider's STATIC registry catalog, which can include models the operator
never synced or approved (e.g. openrouter/auto). The visibility filter
(getHiddenModelsByProvider) only caught models explicitly flagged isHidden,
so catalog-only models passed through and got routed upstream.
Build the credentialed pool from the models the user actually has available
(synced + custom non-hidden), falling back to the static catalog only when
the operator has no synced/custom models for that provider. Applies to every
provider uniformly (openai, kilocode, openrouter, ...), with per-connection
scoping for synced models. Provider wildcards (providerWildcard.ts) already
used the active synced catalog as the authoritative source.
Regression coverage: tests/unit/combo-auto-pool-visible-only.test.ts
Co-authored-by: herjarsa <herjarsa@users.noreply.github.com>
* feat(providers): add tencent-aistudio-web cookie provider (tasw)
* fix(sse): remove orphaned DevinDesktopExecutor import from executor index
The "devin-desktop" executor key is unused (devin-desktop provider config
resolves to executor "devin-cli"); the imported ./devin-desktop.ts file
was never present, so executors/index.ts failed to load (ERR_MODULE_NOT_FOUND)
and broke every unit test that imports the executor registry (e.g.
tests/unit/deepseek-web.test.ts). Stale base sync carried this into the branch.
Remove the dead import/registration/export.
* fix(providers): restore DevinDesktopExecutor registration in executor index
The previous commit removed the devin-desktop executor import/registration/
export from open-sse/executors/index.ts, but the devin-desktop provider
registry still resolves executor "devin-desktop" and
tests/unit/devin-providers.test.ts asserts hasSpecializedExecutor("devin-desktop")
is true. The removal broke 6 tests in that file. Restore the three lines so
the live Devin Desktop executor keeps serving the provider.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
* fix(providers): correct tencent-aistudio-web wrapper shape + provider count sync
Return {response,url,headers,transformedBody} instead of a raw fetch Response
(the executor contract every other executor in this file follows) and
re-wrap the upstream body so it uses the local Response constructor, not the
undici-patched one from globalThis.fetch.
Regenerate docs/reference/PROVIDER_REFERENCE.md and sync the 339->340
provider-count claims (README, AGENTS.md, llm.txt + 42 i18n mirrors,
package.json, promise-pillars/comparison-table/cli-terminal SVGs) that this
PR's new provider invalidated.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
* docs(providers): sync readme-hero.svg provider count claim (339->340)
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
* fix(providers): register tencent-aistudio-web web-session credential metadata + golden
Add the WEB_SESSION_CREDENTIAL_REQUIREMENTS entry for tencent-aistudio-web
(cookie-based, matching the executor's raw Cookie-header credential) and
regenerate the translate-path golden snapshot to include the new provider —
both were failing CI unit tests that enumerate every registered provider.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
* fix(providers): align tencent-aistudio-web test with the wrapper-shape contract
The test asserted res.status/res.json() directly against executor.execute()'s
return value, matching the pre-fix (broken) raw-Response shape. Update it to
read res.response.status/res.response.json() — the {response,url,headers,
transformedBody} contract every executor in this codebase follows.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: MeRezaRezaei <MeRezaRezaei@users.noreply.github.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Co-authored-by: adevwithpurpose <adevwithpurpose@users.noreply.github.com>
Deploying the internal gateway was a manual build/pack/scp/npm-i/pm2-restart sequence with no record of what landed and no proof it served traffic. On 2026-08-14 that shipped a package built from a branch predating #10373: the process came up, health said 'healthy', and every request returned 502 until a human hit it.
scripts/ops/deployCanary.ts holds the policy as pure functions — refuse an artifact that is not traceable to the release line (reusing #10427), and grade the deploy on health PLUS at least one real completion. Zero probes fails: 'no probe ran' must never read as 'everything is fine', which is exactly how a broken egress path hides behind a green health check. Remote steps are argv arrays, never shell strings (Hard Rule #13), ordered so the rollback anchor is captured before the install overwrites it.
scripts/ops/deploy-canary.mjs performs the side effects, supports --dry-run, and prints the rollback command when the smoke fails.
Closes#10429
The packaged artifact stamped dist/BUILD_SHA but nothing verified the SHA belonged to the release line, so a tarball built from a feature branch installed and served traffic indistinguishably from a release build. That is how the internal gateway ended up running a build that predated #10373 and answered every request with 502 'Executor result must contain a Response' — identifying it required SSH plus grepping the compiled chunks.
scripts/build/buildProvenance.ts classifies a build SHA against the release ref (pure functions, injected git probe). A missing SHA fails even with the canary override: an unidentifiable artifact cannot be vouched for. validate-pack-artifact enforces it on real packs (skipped under --policy-only, which runs without a build); OMNIROUTE_ALLOW_CANARY_BUILD=1 records a deliberate off-release-line build instead of failing it. /api/monitoring/health now exposes system.buildSha — absent when unknown, never fabricated.
Closes#10427
Any process that opened the DB without setting DATA_DIR resolved to ~/.omniroute/storage.sqlite — the operator's live database, provider credentials included. tests/_setup/isolateDataDir.ts only covers the npm scripts; the documented single-file test command and ad-hoc probes bypassed it (one did exactly that during #10334).
resolveWritableDataDir now redirects a test-context process with no DATA_DIR to a throwaway temp dir, stable per process. Redirect rather than throw, so the documented single-file command keeps working; OMNIROUTE_ALLOW_DEFAULT_DATA_DIR=1 opts back in and records the intent.
Closes#10428
Audit of every numeric claim in the README, AGENTS.md and the README SVGs against
live code, plus a changelog/credit reconciliation over the full v3.8.50 cycle.
Corrected numbers (all measured, not estimated):
- Provider circuit breaker thresholds were scaled up in code for 500+ connection
deployments (`providerFailureThreshold`: OAuth 3 -> 10, API key 5 -> 15) but the
docs still published the pre-scale values. Fixed in AGENTS.md (now a table that
also separates the provider-level threshold from the per-connection one and lists
the provider cooldowns), in the README alt text and inside resilience-layers.svg
(visible label and aria-label).
- "40+ free forever" was unsourced. Measured from the free-tier catalog as every
provider whose free access renews or needs no key (recurring-monthly, -daily,
-uncapped, -credit, keyless; one-time signup credits and discontinued pools
excluded): 56. Updated in the README and promise-pillars.svg.
- Cycle-evolution table: v3.8.49 shipped 290 providers, not 291, and the model row
compared the v3.8.49 free-tier catalog (516) against today's full catalog. Both
columns now use the same metric - distinct documented models, 1185 -> 1202.
- Tech-stack row: 95 domain modules -> 117.
The free-forever count is now enforced by check:docs-counts so it cannot drift
again; it is derived from freeType in the live catalog, like every other gated
number.
Changelog reconciliation (`scripts/release/list-uncovered-commits.mjs`):
uncovered cycle commits drop from 149 to 62. 75 user-facing commits gained a bullet
with author attribution, the 45 ref-less direct pushes and 29 chore/ci/test/docs
commits were consolidated into rollup bullets, and the contributors table grew from
147 to 161 rows - 14 contributors who had landed work with no credit at all
(including @amartinawi, @pacocartones and @excessivechaos) are now credited.
The remaining 62 carry no PR/issue ref, which is the ceiling of ref-based coverage.
CHANGELOG.md and its i18n mirrors are added to .prettierignore: check:changelog-
integrity compares base bullets as exact strings, and Prettier normalizes markdown
emphasis inside them (*from* -> _from_), so any PR that staged the changelog turned
the merge-integrity job red. scripts/release/* is the changelog's formatter of
record, the same precedent already used for ENVIRONMENT.md and PROVIDER_REFERENCE.md.
Co-authored-by: Xiangzhe <bakryun0718@proton.me>
* docs: add rate limiting guide for free providers (429/400/401)
Community-reported troubleshooting for auto-discovered issues when
rotating through free/no-auth providers (opencode, felo-web, auggie).
Documents the verified env-var combo that eliminates
intermittent 429/400/401 failures in cron/agent automation:
OMNIROUTE_ROTATE_ON_400=true,
OMNIROUTE_CHAT_MAX_HEAVY_IN_FLIGHT=4,
OMNIROUTE_STRUCTURE_LIMIT=off
Includes root-cause breakdown (provider quota vs passthrough 401 vs
concurrency amplification), verification steps via /monitoring/health,
and escalation for hard quota exhaustion.
* docs(providers): fix fabricated env var and breaker states in rate-limit guide
Replace OMNIROUTE_STRUCTURE_LIMIT (does not exist in the codebase) with
OMNIROUTE_CHAT_ADMISSION_QUEUE_MS and document the real rate-limit knobs
(RATE_LIMIT_MAX_WAIT_MS / RATE_LIMIT_MAX_QUEUE_DEPTH / RATE_LIMIT_AUTO_ENABLE).
Correct the circuit breaker states to the actual enum (CLOSED/DEGRADED/OPEN/HALF_OPEN)
and point the health-check note at circuitBreakers.providerBreakers[].state.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: Bruno <bruno@nousresearch.com>
Co-authored-by: mrcram2021 <mrcram2021@users.noreply.github.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Co-authored-by: adevwithpurpose <adevwithpurpose@users.noreply.github.com>
* fix(db): prune pre-migration backups so db_backups stops growing unbounded
createPreMigrationBackup() wrote a VACUUM INTO snapshot on every migration run
and never pruned. On a long-lived instance db_backups/ reached 48.999 files /
204 GB against a 5,3 MB live database; a second devbox showed the same shape
(5.711 files / 24 GB).
The retention policy already existed in cleanupDbBackups() but nothing on the
migration path reached it — its only callers are backup.ts and the
/api/db-backups route, neither of which runs during a migration.
migrationRunner.ts cannot import backup.ts: core.ts imports migrationRunner.ts
and backup.ts imports core.ts, so that edge would close a cycle. The policy
therefore moves to a new core-free module, backupRetention.ts, which both call
sites share — cleanupDbBackups() now delegates to it rather than duplicating it.
At the migration call site the operator's maxFiles/retentionDays are read
through the adapter already open for the run; going through getDbInstance()
would re-enter database initialization. Pruning never throws, so housekeeping
cannot fail a migration.
Closes#10421
* chore(db): declare backupRetention as an intentionally-internal db module
check:db-rules requires every src/lib/db/ module to be either re-exported by
localDb.ts or listed in INTENTIONALLY_INTERNAL. backupRetention.ts is a shared
primitive consumed only by db/backup.ts and db/migrationRunner.ts — the same
category as the migrationRunner entry — so it belongs in the allowlist rather
than in the public re-export surface.
* test(db): include backupRetention in the audited INTENTIONALLY_INTERNAL list
check-db-rules-classification.test.ts freezes the exact membership of
INTENTIONALLY_INTERNAL, so adding the 40th entry has to be reflected there too
— the gate script and this test pin the same contract from opposite sides.
---------
Co-authored-by: Xiangzhe <bakryun0718@proton.me>
