Files
OmniRoute/docs/reference/RELAY_TROUBLESHOOTING.md
Paijo b5e75bb8fe fix(proxy): relay repair + free-pool UX + relay awareness (#6909)
* feat(proxy): relay repair + free-pool UX + relay awareness

* fix(proxy): preserve existing notes fields on relay repair; fix cpu limit /1000 across all 5 container providers

* feat(proxy): extract bulk-import and pool-modal hooks (#6625)

* fix: prevent relay type normalization to http on PATCH

Bug: updateProxyRegistrySchema inherited .default("http") from the base
schema, causing PATCH to silently overwrite relay types (vercel/deno/cloudflare)
with "http" when the client didn't send a type field.

- Move .default('http') from proxyRegistryFieldsSchema to
  createProxyRegistrySchema (only applies to new proxies)
- Strip undefined keys from validated changes before passing to
  updateProxy — .partial() leaves absent fields as undefined, which
  the spread merge in updateProxyRow would propagate to the DB
- Add console.warn in extractRelayAuth when decrypt fails on a
  known-encrypted relayAuthEnc blob

Closes #6905

* feat: replace Load More with page-number pagination in FreePoolTab

- Adds page-number pagination controls with prev/next buttons
- Shows per-page summary with total counts
- Resets to page 1 on filter change via wrapper setters

* fix(proxy): restore free-proxy sync-error tracking reverted by pagination commit

commit 2c8e79f13 (page-number pagination for FreePoolTab) accidentally
reverted the recordFreeProxySyncErrors/clearFreeProxySyncErrors/
getFreeProxySyncErrors functions and the search/sortBy list options that
an earlier commit (6f9ce75f3) in this same PR had added to
src/lib/db/freeProxies.ts and src/app/api/settings/free-proxies/route.ts.

localDb.ts still re-exported the three sync-error functions, so every
module that transitively imports it (which is most of the unit test
suite, plus the Next.js build used by dast-smoke) failed at load time
with "The requested module './db/freeProxies' does not provide an
export named 'clearFreeProxySyncErrors'".

Restores the reverted implementation (verbatim) and fixes the two new
"requires management auth" tests, which asserted 401 for an
unauthenticated request without configuring INITIAL_PASSWORD +
requireLogin first — on a fresh DB with no password configured,
isAuthRequired() treats a loopback request as the pre-setup bootstrap
path and allows it through, so the assertion needs the same
password/requireLogin setup already used by
tests/unit/api/settings-audit.test.ts.

Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>

* fix(proxy): trim unrelated scope from relay-repair/free-pool PR

Remove two subsystems that are unrelated to relay repair and the
free-pool UX and were never wired into the app:

- open-sse/services/combo/capabilityRequirements.ts and
  CapabilityRequirementsEditor.tsx: a combo capability-filtering
  feature never imported by combo.ts or combos/page.tsx, with no
  test coverage.
- src/lib/skills/containerProvider.ts CPU-limit rescale: an
  untested change to sandbox resource limits, unrelated to the
  proxy/relay subsystem this PR targets. Restored to the
  release baseline.

Also renumber the free_proxy_sync_errors migration 121 -> 122 to
avoid colliding with #6855, which independently claims 121 on the
same release branch.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Co-authored-by: oyi77 <oyi77@users.noreply.github.com>
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
2026-07-12 02:28:26 -03:00

4.4 KiB

title, version, lastUpdated
title version lastUpdated
Relay Troubleshooting 3.8.43 2026-07-11

Relay Troubleshooting

Relays (Vercel, Deno, Cloudflare) terminate the upstream connection on a serverless backend so OmniRoute can egress from a stable region while keeping the provider API key server-side. This document covers the two failure modes that operators hit in production and the recovery paths OmniRoute ships for each.

How relay auth is stored

When you deploy a relay from Settings → Proxy Pool → Deploy Relay, the deploy flow stores the relay's auth token in the proxy notes JSON field:

  • If STORAGE_ENCRYPTION_KEY is set, the token is written as relayAuthEnc (AES-encrypted at rest).
  • Otherwise it is written as plaintext relayAuth.

At request time extractRelayAuth(notes) returns whichever form is present, so the relay keeps working across restarts.

Failure mode 1 — undecryptable token after key rotation

Symptom: relays that previously worked now return upstream 401/auth errors after an environment or secret-manager rotation of STORAGE_ENCRYPTION_KEY. The stored relayAuthEnc blob can no longer be decrypted, so extractRelayAuth returns the empty string and the relay sends no auth.

Recovery — repair in place (no redeploy):

  1. Open Settings → Proxy Pool.
  2. Relay rows whose auth is missing show a yellow auth missing badge and a Repair button.
  3. Click Repair. OmniRoute calls POST /api/settings/proxies/[id]/repair-relay, which:
    • decrypts relayAuthEnc with the current key,
    • writes the plaintext relayAuth back into notes,
    • returns { repaired: true, mode: "recovered" }.

The relay resumes serving without re-entering any deploy credentials.

This only works if the current STORAGE_ENCRYPTION_KEY can still decrypt the blob. If you rotated the key without a migration, the blob is unrecoverable.

Failure mode 2 — unrecoverable token (key rotated away)

Symptom: you clicked Repair and got { repaired: false, mode: "redeploy", status: 409 }.

Recovery — redeploy:

The stored token cannot be recovered with the current key. Re-deploy the relay from the same modal you used originally (Deploy Relay menu → Vercel / Deno / Cloudflare). The deploy flow writes a fresh token (encrypted with the current key) and the row's auth missing badge clears.

In the UI, the Repair button itself triggers the redeploy modal when the token is unrecoverable, so you never have to hunt for it manually.

Failure mode 3 — relay reachable but unhealthy

Symptom: the proxy test (speed button) shows the relay up but requests still fail intermittently.

Check the relay awareness headers returned by OmniRoute's auto-test probe (see Settings → Proxy Pool and the relayTested / relayAlive counters):

  • x-relay-url — which relay backend answered.
  • x-relay-modets | bifrost | auto for that request.
  • x-relay-attempts — how many relay hops were tried before success.
  • x-relay-fallbacktrue when the request fell back from the preferred backend to the TypeScript relay.

A high x-relay-fallback rate with low relayAlive means the sidecar backend is unhealthy and you should either fix it or switch the relay backend strategy to ts (see RELAY_BACKEND_STRATEGY.md).

API reference

Method Path Body Success Failure
POST /api/settings/proxies/[id]/repair-relay { "id": "<proxyId>" } 200 { repaired: true, mode: "recovered" } when re-derived; 200 { repaired: false, mode: "noop" } when plaintext already present 409 { mode: "redeploy" } unrecoverable; 400 not a relay type; 404 not found

The list route GET /api/settings/proxies attaches a secret-free relayInfo: { isRelay, authMissing, repairMode } to each item so the dashboard can render the repair affordance without ever exposing the token.