Files
OmniRoute/docs/architecture
Innokentiy Solntsev dd70dbdaa0 fix(sse): Anthropic OAuth 403 "Request not allowed" is a per-request refusal — cooldown with backoff instead of an instant ban (#12859) (#12864)
* fix(sse): Anthropic OAuth 403 "Request not allowed" is a per-request refusal, not a ban

A single upstream 403 on the `claude` OAuth connection was classified
FORBIDDEN and written as the terminal `banned` connection state
(chatCore -> writeTerminalStatus). From then on every request to that
provider was short-circuited with "All 1 connection(s) banned by
upstream - please reconnect in the dashboard" without touching Anthropic,
until an operator reconnected.

Anthropic's OAuth surface answers a small fraction of otherwise-valid
requests with 403 {"type":"permission_error","message":"Request not
allowed"}. On the reporting install the same token returned 200 forty
seconds before the 403 and again right after the connection was
re-enabled; a revoked or expired token is a 401 authentication_error, not
this. It is a refusal of one request, not of the credential.

Classify it as the new non-terminal PROVIDER_ERROR_TYPES.REQUEST_REJECTED
(scoped to provider `claude` and the "Request not allowed" body) and list
that type in authTerminalStatus.isNonTerminalProviderError, mirroring the
Cloudflare FINGERPRINT_REJECTION precedent. The combo layer still falls
through to the next target for the failing request; the connection stays
active for the next one. Any other claude 403 keeps its previous
classification.

Tests: error-classifier.test.ts covers the Anthropic body, the
gateway-flattened "[403]: Request not allowed" message, the same body from
a non-Anthropic provider (still FORBIDDEN), other claude 403s (unchanged),
and the helpers; anthropic-request-not-allowed-not-a-ban.test.ts pins
resolveTerminalConnectionStatus() -> null for the new type even with a
`permanent` fallback verdict, and `banned` for a generic claude 403.

* fix(sse): cooldown with backoff and streak escalation for REQUEST_REJECTED (#12859)

Not "ignore the 403" either: if Anthropic ever made "Request not allowed"
systematic, re-sending every request into it would be the wrong thing to
do to an OAuth account. chatCore now handles REQUEST_REJECTED explicitly:

- exclude the connection via setConnectionRateLimitUntil for a growing
  cooldown (5 -> 15 -> 45 min) so a sporadic refusal costs minutes, not a
  reconnect, and a systematic one cannot become a stream of 403s;
- escalate to the terminal `banned` state only for 3 refusals within a
  60-minute window (services/requestRejectedStreak.ts, in-memory per
  connection; a restart forgets the streak, erring towards more cooldowns
  rather than an operator-undone ban), with a last_error that says so;
- probe-origin failures record but never cool down or ban (#9817).

The existing "request not allowed" text rule (5 s) is unaffected:
markAccountUnavailable skips a connection that already has a future
rateLimitedUntil, so the minute-scale cooldown written here wins.

Tests: request-rejected-streak.test.ts pins the window/threshold/backoff
arithmetic; anthropic-request-not-allowed-cooldown-escalation.test.ts drives
the real chat route against a mocked 403 upstream on a `claude` OAuth
connection: 300 s cooldown, then 900 s, then banned on the third refusal;
a different claude 403 body still bans on the first response.

* chore(changelog): name the #12859 fragment after its PR (#12864)

* refactor(sse): move the REQUEST_REJECTED branch into a chatCore leaf; register its tests for mutation coverage

chatCore.ts is frozen at 5984 lines by the file-size ratchet; the branch
body now lives in open-sse/handlers/chatCore/requestRejectedFailure.ts
(chatCore: 5974 -> 5983). stryker.conf.json tap.testFiles gains the two new
DB-backed tests so their mutant kills count (check:mutation-test-coverage).

* fix(sse): count refusal episodes, reset on success, keep the dashboard honest (#12859 review)

Review findings on the first cut of the REQUEST_REJECTED handling:

- A burst of in-flight requests that all got the 403 within seconds
  produced streak 1, 2, 3 and a ban from one upstream event. The streak
  now counts cooldown *episodes*: a refusal that lands while the
  connection is already excluded is the same event and is not counted.
- Nothing reset the streak on a healthy response, so sporadic refusals
  on a busy install could still accumulate to a ban. chatHelpers'
  onRequestSuccess now clears it (only a real success does - the recovery
  tick's clearAccountError is an elapsed cooldown, not a success).
  Clearing the cooldown by hand in the dashboard clears it too.
- The third rung of the ladder was unreachable (the third refusal
  escalates): the ladder is now 5 -> 15 min, sourced from COOLDOWN_MS next
  to the existing 5 s "request not allowed" rule, with a note on why that
  rule is superseded for claude. The 60-min window becomes a 24 h
  staleness bound - "consecutive" is defined by successes, not by time.
- Probe-origin refusals no longer touch the streak (#9817).
- The cooldown is written like every other connection-level cooldown:
  ISO rateLimitedUntil + testStatus "unavailable" (+ lastErrorAt), so the
  dashboard shows the countdown and the recovery tick restores "active".
- One refusal is re-seeded from the persisted row after a restart so a
  crash loop cannot reset the count on every boot.

Docs: RESILIENCE_GUIDE terminal states + CODEBASE_DOCUMENTATION resilience
row mention the streak module. Tests cover the burst, the success reset,
the seed, and the ISO/unavailable shape end-to-end through the chat route.

* chore(sse): drop unrelated Prettier churn in auth.ts / providers route

* style(api): keep providers route Prettier-clean

* refactor(sse): share the "exclude connection for a cooldown" leaf between GEO_BLOCKED, GCP_PROJECT_REQUIRED and the new branch

The release tip moved chatCore.ts to its frozen 5984 lines, so the
REQUEST_REJECTED branch cannot add a single net line. The GEO_BLOCKED and
GCP_PROJECT_REQUIRED branches were the same eight statements with different
constants and log wording; both now call
open-sse/handlers/chatCore/connectionCooldown.ts::excludeConnectionForCooldown
(behaviour, probe guard and log lines preserved verbatim). chatCore.ts ends
9 lines below the base it branched from.

* chore(chatCore): tighten the cooldown comments to keep the file under its size ceiling

After merging release/v3.8.51, chatCore.ts sat at 6150 lines against a
frozen ceiling of 6146. Condense the explanatory comments this PR added
to the GEO_BLOCKED and GCP_PROJECT_REQUIRED branches; no code change.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Co-authored-by: insoln <is@careerum.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
2026-09-17 16:26:08 -03:00
..
2026-07-13 09:12:40 -03:00
2026-07-02 10:47:13 -03:00