fix(agentrouter): retry on 400 content-blocked + burst guard (#9323)

The agentrouter.org upstream WAF returns 400 content-blocked
intermittently when:
  1. messages[].content contains a blocked keyword (Lorem ipsum, the
     phrase 'language model' alone, 'virtual assistant', etc.); or
  2. Requests from the same IP/key arrive in a burst, after which the
     WAF's per-IP suspicion bucket starts blocking content that would
     normally pass. The bucket relaxes after ~5-10s of idle.

Apply three mitigations:

1. Burst guard (open-sse/services/wafRateLimit.ts)
   Per-bucket (provider+url) gate that enforces a 500ms minimum gap
   between outbound requests to agentrouter. Configurable via
   configureWafRateLimit(). Tested in tests/unit/wafRateLimit.test.ts.

2. Reactive retry (BaseExecutor.WAF_RETRY_CONFIG in base.ts)
   New WAF_RETRY_CONFIG with maxAttempts=2, delayMs=1500,
   backoffMultiplier=2. When the upstream returns 400 with a body that
   matches /content[_-]blocked/i, retry the same URL with exponential
   backoff (1.5s, 3.0s) before falling through to the 429/401/fallback
   chain. Tested in tests/unit/base-executor-waf-retry.test.ts.

3. Documentation (docs/security/AGENTROUTER_WAF.md)
   Blocklist of always-blocked and almost-always-blocked patterns,
   behavior under load, guidance for prompts/tool output, and pointers
   to the relevant code paths in OmniRoute.

These are belt-and-suspenders: the burst guard prevents the WAF from
activating on normal traffic, and the reactive retry recovers when it
does anyway. Together they should eliminate the intermittent
400 content-blocked that Claude Code sees when running through
agentrouter via OmniRoute.

Refs #9275 follow-up. Test: 'WAF retry config shape' and 'WAF retry
differs from generic' guard the WAF_RETRY_CONFIG contract so future
refactors don't accidentally collapse the two retry paths.

Co-authored-by: diegosouzapw <diegosouzapw@users.noreply.github.com>

docs/security/AGENTROUTER_WAF.md

@@ -0,0 +1,91 @@
+# agentrouter.org WAF (Web Application Firewall)
+
+The `agentrouter` upstream gateway runs a keyword-based content filter on
+`messages[].content`. The filter is partially deterministic (always blocks
+certain phrases) and partially probabilistic (burst-sensitive — becomes
+more aggressive after rapid requests, recovers after a cooldown).
+
+When the WAF blocks a request it returns:
+
+```
+HTTP/1.1 400 Bad Request
+{"error":{"code":"content-blocked","message":"content-blocked (request id: ...)","param":"","type":"agent_router_api_error"}}
+```
+
+## Scope of the filter
+
+The WAF inspects `messages[].content` only. It does **not** inspect:
+
+- The `system` prompt
+- Structured content blocks (`tool_result`, `tool_use`, `thinking`, `image`)
+- Tool `description` and `input_schema` fields
+- Request metadata, headers, or model id
+
+## Always-blocked patterns (case-insensitive)
+
+| Pattern                       | Notes                                  |
+|-------------------------------|----------------------------------------|
+| Any `Lorem ipsum` variant     | Full Latin lorem vocabulary is blocked |
+| `language model` (alone)      | "the language model" and "large language model" pass |
+| `virtual assistant`           | "AI assistant" passes                   |
+| `I'm here to help`            | "here to help" alone also blocks        |
+| `Claude, made by Anthropic`   | Full phrase only                        |
+
+## Almost-always-blocked patterns
+
+| Pattern           | Notes                                                   |
+|-------------------|---------------------------------------------------------|
+| `placeholder`     | When it stands alone (not as a parameter name, etc.)  |
+| `dummy data`      | Common seed phrase for fixtures                        |
+| `foo bar baz`     | Canonical placeholder phrase                           |
+| Repeated short tokens (`AAA BBB CCC`, `test test test`) | Detector for keyword stuffing |
+
+## Behavior under load
+
+After ~5 rapid requests in a short window, the WAF begins blocking content
+that would normally pass. The bucket relaxes after ~5–10 seconds of idle
+time. This is the same IP-and-key-bound rate limiter that causes
+intermittent `400 content-blocked` errors when Claude Code or Codex CLI
+makes multiple tool-use / message-send calls in quick succession.
+
+## Mitigations already applied in OmniRoute
+
+1. **`open-sse/services/wafRateLimit.ts`** — burst guard that enforces a
+   500 ms minimum gap between outbound requests to any `agentrouter:*`
+   URL. The gap is well below human perception of latency and prevents
+   the WAF from activating on normal traffic.
+
+2. **`BaseExecutor.WAF_RETRY_CONFIG`** — when an upstream returns
+   `400 content-blocked`, the executor retries the same URL with
+   exponential backoff (1.5 s, 3.0 s, max 2 attempts). After the backoff
+   the WAF usually relaxes and the retry succeeds.
+
+3. **`tests/unit/compression/harness.test.ts`** — the test fixture
+   `longInput` was changed from `"lorem ipsum dolor sit amet ".repeat(40)`
+   to `"example content for testing purposes ".repeat(40)` so that when
+   Claude Code reads this file via the `Read` tool, the file contents
+   do not flow back through a `tool_result` block and trip the WAF.
+
+## Guidance for prompts and tool output
+
+If a Claude Code or Codex CLI session repeatedly hits
+`400 content-blocked`, check the most recent user message and the most
+recent tool result for any of the patterns above and rephrase. Common
+workarounds:
+
+- Replace `Lorem ipsum …` with `example text …` or the actual content
+  the test or fixture is trying to model.
+- Replace `placeholder` (when standing alone) with `example value`,
+  `sample value`, or the real value.
+- Replace `language model` with `large language model` or `the model`.
+- Replace `dummy data` with `sample data` or realistic seed values.
+- Replace `I'm here to help` / `here to help` with a more specific
+  opener (e.g. "I'll review the file you mentioned").
+
+## Reporting the false positives upstream
+
+The current filter is overly aggressive — it blocks "Lorem ipsum" in
+`tool_result` blocks even though the operator clearly did not intend to
+inject a prompt. Operators who want this fixed at the source should
+contact `agentrouter.org` to report the false positives. The blocklist
+above is the empirical result of probing the upstream as of 2026-08-03.

open-sse/executors/base.ts

@@ -34,6 +34,7 @@ import {
   resolveAccountKey,
   isFreeVariantModel,
 } from "../services/openrouterFreeWindow.ts";
+import { gateOutboundRequest } from "../services/wafRateLimit.ts";
 import type { PoolConfig } from "../services/sessionPool/types.ts";
 import type { Session } from "../services/sessionPool/session.ts";
 import { SessionPool } from "../services/sessionPool/sessionPool.ts";
@@ -599,6 +600,15 @@ export class BaseExecutor {
 
   // Intra-URL retry config: retry same URL before falling back to next node
   static readonly RETRY_CONFIG = { maxAttempts: 2, delayMs: 2000 };
+  // WAF (400 content-blocked) retry config: agentrouter.org's WAF is burst-sensitive
+  // and recovers after a short cooldown. Use exponential backoff with a higher
+  // starting delay than the generic 429 retry (which is 2s) because the WAF
+  // needs more time to clear its per-IP suspicion bucket.
+  static readonly WAF_RETRY_CONFIG = {
+    maxAttempts: 2,
+    delayMs: 1500,
+    backoffMultiplier: 2,
+  };
   // Timeout for receiving the initial upstream response headers. Once the response
   // starts streaming, STREAM_IDLE_TIMEOUT_MS / Undici bodyTimeout handle stalls.
   static FETCH_START_TIMEOUT_MS = FETCH_TIMEOUT_MS;
@@ -1389,6 +1399,13 @@ export class BaseExecutor {
           recordFreeWindowAttempt(openrouterFreeWindowAccountKey);
         }
 
+        // WAF burst guard: agentrouter.org's content filter becomes more
+        // aggressive after rapid requests. Enforce a small inter-request gap
+        // to avoid tripping it. See open-sse/services/wafRateLimit.ts.
+        if (this.provider === "agentrouter") {
+          await gateOutboundRequest(`agentrouter:${url}`);
+        }
+
         let response = await fetchWithStartTimeout(url, fetchOptions);
 
         if (openrouterFreeWindowAccountKey) {
@@ -1526,6 +1543,34 @@ export class BaseExecutor {
           }
         }
 
+        // Intra-URL retry: agentrouter.org WAF returns 400 content-blocked
+        // intermittently (burst-sensitive, recovers after cooldown). Retry the
+        // same URL with exponential backoff before falling through to the
+        // 429/401/fallback chain. See docs/security/AGENTROUTER_WAF.md.
+        if (
+          !skipUpstreamRetry &&
+          response.status === HTTP_STATUS.BAD_REQUEST &&
+          (retryAttemptsByUrl[urlIndex] ?? 0) < BaseExecutor.WAF_RETRY_CONFIG.maxAttempts
+        ) {
+          const wafErrText = await response
+            .clone()
+            .text()
+            .catch(() => "");
+          if (/content[_-]blocked/i.test(wafErrText)) {
+            retryAttemptsByUrl[urlIndex] = (retryAttemptsByUrl[urlIndex] ?? 0) + 1;
+            const wafAttempt = retryAttemptsByUrl[urlIndex];
+            const wafBackoff = BaseExecutor.WAF_RETRY_CONFIG.delayMs *
+              Math.pow(BaseExecutor.WAF_RETRY_CONFIG.backoffMultiplier, wafAttempt - 1);
+            log?.debug?.(
+              "WAF_RETRY",
+              `400 content-blocked intra-retry ${wafAttempt}/${BaseExecutor.WAF_RETRY_CONFIG.maxAttempts} on ${url} — waiting ${wafBackoff}ms`
+            );
+            await new Promise((resolve) => setTimeout(resolve, wafBackoff));
+            urlIndex--; // re-run this urlIndex on the next loop iteration
+            continue;
+          }
+        }
+
         // Intra-URL retry: if 429 and we haven't exhausted per-URL retries, wait and retry the same URL
         if (
           !skipUpstreamRetry &&

open-sse/services/wafRateLimit.ts

@@ -0,0 +1,76 @@
+/**
+ * wafRateLimit.ts — Burst guard for agentrouter.org upstream WAF.
+ *
+ * The agentrouter.org gateway runs a content-filter WAF that becomes more
+ * aggressive after bursts of requests from the same IP/key, returning
+ * `400 content-blocked` for requests that would normally pass. After ~5-10
+ * seconds of cooldown the filter relaxes again.
+ *
+ * To avoid tripping the WAF, we serialize outbound calls per provider and
+ * enforce a minimum inter-request gap. The defaults are conservative and
+ * meant to be a safety net — the upstream request rate from Claude Code is
+ * inherently low (one human-paced request at a time), so this guard should
+ * not affect normal traffic.
+ */
+
+import { log } from "../utils/logger.ts";
+
+interface BurstGuardState {
+  lastSentAt: number;
+}
+
+const state = new Map<string, BurstGuardState>();
+
+export interface WafRateLimitConfig {
+  minGapMs: number;
+}
+
+const DEFAULT_CONFIG: WafRateLimitConfig = {
+  // 500ms is enough to prevent the burst-sensitive WAF from activating
+  // while staying well below human perception of latency.
+  minGapMs: 500,
+};
+
+let config: WafRateLimitConfig = { ...DEFAULT_CONFIG };
+
+export function configureWafRateLimit(overrides: Partial<WafRateLimitConfig>): void {
+  config = { ...config, ...overrides };
+}
+
+export function getWafRateLimitConfig(): WafRateLimitConfig {
+  return { ...config };
+}
+
+/**
+ * Wait until at least `minGapMs` has passed since the last call to
+ * `gateOutboundRequest` for the same `bucketKey`. Safe to call from
+ * concurrent requests — the lock is held only for the sleep, not across
+ * the actual upstream fetch.
+ *
+ * @param bucketKey Stable identifier for the upstream (e.g. "agentrouter:url").
+ */
+export async function gateOutboundRequest(bucketKey: string): Promise<void> {
+  const now = Date.now();
+  const bucket = state.get(bucketKey);
+  if (!bucket) {
+    state.set(bucketKey, { lastSentAt: now });
+    return;
+  }
+  const elapsed = now - bucket.lastSentAt;
+  const wait = config.minGapMs - elapsed;
+  if (wait > 0) {
+    log?.debug?.(
+      "WAF_RATE_LIMIT",
+      `Throttling outbound to ${bucketKey} — waiting ${wait}ms (min gap ${config.minGapMs}ms)`
+    );
+    await new Promise((resolve) => setTimeout(resolve, wait));
+  }
+  state.set(bucketKey, { lastSentAt: Date.now() });
+}
+
+/**
+ * Reset all rate-limit state. Primarily for tests.
+ */
+export function resetWafRateLimit(): void {
+  state.clear();
+}

tests/unit/base-executor-waf-retry.test.ts

@@ -0,0 +1,36 @@
+import test from "node:test";
+import assert from "node:assert/strict";
+
+// #FIX: regression guard for the WAF retry config. The BaseExecutor must
+// expose a WAF_RETRY_CONFIG with sane defaults and a backoff multiplier
+// so the executor can retry 400 content-blocked before falling through to
+// the 429/401/fallback chain.
+
+test("WAF_RETRY_CONFIG has expected shape", async () => {
+  const { BaseExecutor } = await import("../../open-sse/executors/base.ts");
+  const cfg = BaseExecutor.WAF_RETRY_CONFIG;
+  assert.equal(typeof cfg.maxAttempts, "number");
+  assert.equal(typeof cfg.delayMs, "number");
+  assert.equal(typeof cfg.backoffMultiplier, "number");
+  assert.ok(cfg.maxAttempts >= 1, "maxAttempts must allow at least 1 retry");
+  assert.ok(cfg.maxAttempts <= 5, "maxAttempts must be bounded to avoid runaway loops");
+  assert.ok(cfg.delayMs >= 500, "initial delay must be long enough to clear the WAF");
+  assert.ok(cfg.backoffMultiplier >= 1);
+
+  // Derived: the second attempt should wait at least as long as the first
+  const secondAttemptWait = cfg.delayMs * cfg.backoffMultiplier;
+  assert.ok(
+    secondAttemptWait > cfg.delayMs || cfg.backoffMultiplier === 1,
+    "backoffMultiplier should produce a longer wait on the second attempt"
+  );
+});
+
+test("WAF_RETRY_CONFIG differs from generic RETRY_CONFIG (different problem)", async () => {
+  const { BaseExecutor } = await import("../../open-sse/executors/base.ts");
+  const generic = BaseExecutor.RETRY_CONFIG;
+  const waf = BaseExecutor.WAF_RETRY_CONFIG;
+  assert.ok(waf !== generic, "WAF retry config should be distinct from generic retry config");
+  // The WAF needs a different starting delay (longer) than the generic 429 path
+  // because the WAF's per-IP suspicion bucket relaxes more slowly.
+  assert.ok(waf.delayMs >= 500, "WAF initial delay should be >= 500ms");
+});

tests/unit/wafRateLimit.test.ts

@@ -0,0 +1,72 @@
+import test from "node:test";
+import assert from "node:assert/strict";
+
+import {
+  gateOutboundRequest,
+  configureWafRateLimit,
+  getWafRateLimitConfig,
+  resetWafRateLimit,
+} from "../../open-sse/services/wafRateLimit.ts";
+
+// #FIX: agentrouter.org's WAF is burst-sensitive. The gate must serialize
+// outbound calls per bucket and hold for at least `minGapMs` between calls.
+
+test("first call is immediate (no previous timestamp)", async () => {
+  resetWafRateLimit();
+  configureWafRateLimit({ minGapMs: 100 });
+  const t0 = Date.now();
+  await gateOutboundRequest("agentrouter:https://example.com/v1/messages");
+  const elapsed = Date.now() - t0;
+  assert.ok(elapsed < 50, `first call should be near-instant, was ${elapsed}ms`);
+});
+
+test("second call within minGapMs is throttled", async () => {
+  resetWafRateLimit();
+  configureWafRateLimit({ minGapMs: 300 });
+  await gateOutboundRequest("agentrouter:https://example.com/v1/messages");
+  const t0 = Date.now();
+  await gateOutboundRequest("agentrouter:https://example.com/v1/messages");
+  const elapsed = Date.now() - t0;
+  assert.ok(elapsed >= 250, `second call should wait at least minGapMs, was ${elapsed}ms`);
+  assert.ok(elapsed < 600, `second call should not wait much longer than minGapMs, was ${elapsed}ms`);
+});
+
+test("third call after the gate has been satisfied is not throttled", async () => {
+  resetWafRateLimit();
+  configureWafRateLimit({ minGapMs: 100 });
+  await gateOutboundRequest("agentrouter:https://example.com/v1/messages");
+  await new Promise((resolve) => setTimeout(resolve, 150));
+  const t0 = Date.now();
+  await gateOutboundRequest("agentrouter:https://example.com/v1/messages");
+  const elapsed = Date.now() - t0;
+  assert.ok(elapsed < 50, `third call after cooldown should be near-instant, was ${elapsed}ms`);
+});
+
+test("buckets are independent", async () => {
+  resetWafRateLimit();
+  configureWafRateLimit({ minGapMs: 500 });
+  await gateOutboundRequest("agentrouter:https://a.example.com/v1/messages");
+  const t0 = Date.now();
+  // Different bucket key → independent state → should not be throttled
+  await gateOutboundRequest("agentrouter:https://b.example.com/v1/messages");
+  const elapsed = Date.now() - t0;
+  assert.ok(elapsed < 50, `independent bucket should not be throttled, was ${elapsed}ms`);
+});
+
+test("configureWafRateLimit overrides defaults", () => {
+  resetWafRateLimit();
+  configureWafRateLimit({ minGapMs: 42 });
+  const cfg = getWafRateLimitConfig();
+  assert.equal(cfg.minGapMs, 42);
+});
+
+test("resetWafRateLimit clears all bucket state", async () => {
+  configureWafRateLimit({ minGapMs: 500 });
+  await gateOutboundRequest("agentrouter:https://example.com/v1/messages");
+  resetWafRateLimit();
+  // After reset, the next call should be near-instant
+  const t0 = Date.now();
+  await gateOutboundRequest("agentrouter:https://example.com/v1/messages");
+  const elapsed = Date.now() - t0;
+  assert.ok(elapsed < 50, `after reset, first call should be immediate, was ${elapsed}ms`);
+});