Files
OmniRoute/open-sse/executors/gemini-web.ts
Abhishek4512009 885cd8c411 feat(gemini-web): expose image generation through /v1/images/generations (closes #10466) (#10494)
* feat(providers): add Cloudflare AI Playground as No Auth provider (closes #10389)

Reverse-engineered access to the free, anonymous Cloudflare AI Playground:
chat runs over a PartySocket WebSocket speaking Cloudflare's cf_agent RPC
protocol with zero credentials (no account, no API key, no cookies). The
WS upgrade is gated on a browser-grade TLS fingerprint, so the executor
drives a headless Chromium via Playwright and speaks the protocol from
inside the page context.

- registry entry: cloudflare-playground (alias cfp), authType none,
  curated 20-model catalog (GLM 5.2, Kimi K2.7 Code, DeepSeek V4 Pro,
  gpt-oss-120B, Llama 3.3 70B, Qwen2.5 Coder 32B, ...) captured from the
  live getModels RPC (2026-08-15)
- executor: cf_agent frame stream -> OpenAI SSE translation, id-filtered
  parser (RPC done:true frames cannot kill the stream), in-band upstream
  errors mapped to HTTP 429/502, abort + timeout handling, clean errors
- noauth UI entry with reverse-engineered-endpoint notice
- tests: 12 unit tests using real captured frames (incl. the 3021
  rate-limit error) + fake transport; ESLint clean; open-sse typecheck clean

* fix(providers): define __name helper in page context before evaluate

Bundlers with keepNames (esbuild/tsx, webpack) inject a __name() call into
serialized function bodies. page.evaluate(openPlaygroundSession) therefore
threw ReferenceError: __name is not defined in real browser sessions.
Define the helper on window before evaluating the session opener.

* fix(providers): sync docs counts, golden snapshots and add reasoning_content support for cloudflare-playground

* chore: remove ad-hoc cfp-shim debug script per review feedback

The standalone shim duplicated the executor's frame-parsing and transport
logic and is superseded by open-sse/executors/cloudflare-playground.ts.
Requested in PR #10442 review.

* feat(gemini-web): expose image generation through /v1/images/generations (closes #10466)

Adds a gemini-web image-generation path following the chatgpt-web precedent:

- imageRegistry: gemini-web provider entry (format gemini-web, cookie auth)
  with the nano-banana-web model. The -web suffix keeps the bare
  nano-banana id owned by adobe-firefly (operator decision 2026-07-31).
- gemini-web executor: new parseStreamResponseImages() extracts generated
  image URLs from the StreamGenerate candidate extension block
  (inner[4][0][12][7][0], url at entry[0][3][3] — string or list form),
  dedupes cumulative frames, upgrades to =s2048, and deliberately skips
  web-search thumbnails at [12][1]. Image mode (x_gemini_web_image_mode)
  captures every StreamGenerate frame, resolves on first image, and gets
  a 90s window; chat mode is byte-for-byte unchanged.
- handlers/imageGeneration/providers/geminiWeb.ts: drives the executor in
  image mode with an explicit generation directive prompt (the web UI
  otherwise answers with web-search images), caps n at 4, returns URLs or
  b64_json (downloads the public googleusercontent asset), and surfaces
  refusal text when no image was produced.
- Dispatch branch on format gemini-web in handleImageGeneration.

Tests: 21 new tests with fixtures built from the documented frame layout
(string/list url forms, cumulative-frame dedupe, web-image exclusion,
size-directive handling, refusal visibility, n-cap, b64_json, registry
wiring incl. the bare nano-banana → adobe-firefly regression guard).
Adjacent suites: gemini-web (6 files), chatgpt-web image, image handler,
route, registry, adobe-firefly, freepik, designer — all green.
ESLint clean on touched files (2 pre-existing any warnings unchanged);
tsc -p open-sse 0 errors.

* fix(media): close browser leak, surface timeout errors, and fall back accounts for gemini-web images

Addresses pre-merge review findings on #10494 (closes #10466):

- cloudflare-playground executor: close the launched browser on EVERY
  non-success start() path, including the detected Cloudflare "Attention
  Required" challenge branch (was leaking a Chromium process per blocked
  request).
- cloudflare-playground executor: a streaming chat timeout now emits an
  explicit timeout_error SSE chunk before [DONE] instead of silently
  completing, so a client can no longer mistake an empty/partial timed-out
  stream for a successful answer. Timeout duration is now injectable for
  deterministic tests.
- gemini-web image handler + imageCredentialRetry: classify the underlying
  GeminiWebExecutor's expired/blocked-session failure modes (400/500, per
  its own Playwright timeout/catch-all branches) as retryable, so
  executeImageWithCredentialFallback advances to the next eligible account
  instead of only doing so on a plain 401.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* docs: regenerate provider counts after merging release/v3.8.50 (341 -> 342)

The previous merge commit resolved all 51 auto-generated-file conflicts by
taking release/v3.8.50's content, which still said 341 providers. Merging in
this branch's Cloudflare Playground provider brings the live catalog to 342,
so npm run check:docs-counts-sync now flags stale claims. Fix:

- docs/reference/PROVIDER_REFERENCE.md: regenerated via
  `npm run gen:provider-reference`.
- README.md/AGENTS.md/llm.txt/package.json description: 341 -> 342.
- docs/diagrams/{readme-hero,promise-pillars,comparison-table,cli-terminal}.svg:
  341 -> 342 in the embedded "NNN providers" text (targeted replace, matched
  against the exact pattern check-docs-counts-sync.mjs validates).

check:docs-counts-sync and check:changelog-integrity are both clean after
this commit.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* docs(env): document CLOUDFLARE_PLAYGROUND_CHROME_PATH

Used by open-sse/executors/cloudflare-playground.ts but missing from
.env.example and docs/reference/ENVIRONMENT.md, caught by the
env-doc-sync gate when combined with other PRs in the release
merge-train.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Co-authored-by: user.email <freakymustard67@gmail.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
2026-08-18 11:43:16 -03:00

817 lines
31 KiB
TypeScript

/**
* GeminiWebExecutor — Gemini Web Session Provider
*
* Routes requests through Google Gemini's web interface using browser
* cookies + Playwright automation. Translates between OpenAI chat
* completions format and Gemini's web UI.
*
* Auth: Cookie-based (__Secure-1PSID + __Secure-1PSIDTS from gemini.google.com)
* Method: Playwright browser automation
*
* Note: Streaming is pseudo-streaming — waits for full Gemini response then
* sends as single SSE chunk. Gemini's StreamGenerate endpoint returns complete
* responses, not chunked streams.
*/
import { BaseExecutor, type ExecuteInput } from "./base.ts";
import { buildErrorBody, sanitizeErrorMessage } from "../utils/error.ts";
import { prepareToolMessages } from "../translator/webTools.ts";
import { buildToolModeResponse } from "./chatgptWebTools.ts";
import {
checkGeminiWebUnsupportedControls,
GEMINI_WEB_UNSUPPORTED_CONTROL_CODE,
} from "./gemini-web/capabilities.ts";
// ─── Constants ──────────────────────────────────────────────────────────────
const GEMINI_URL = "https://gemini.google.com/app";
/**
* Whether an error came from Playwright failing to launch because the browser binary is not
* installed (`chromium.launch: Executable doesn't exist at ...`). This is a host/config
* problem, not a transient upstream fault, so the executor must NOT surface it as a retryable
* 500 (which marks the account unavailable and loops / trips the provider breaker). See #3516.
*/
export function isMissingBrowserExecutable(message: string): boolean {
if (!message) return false;
return /executable doesn't exist|executablenotfound|playwright install|chromium.*download/i.test(
message
);
}
const GEMINI_USER_AGENT =
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36";
// ─── Types ──────────────────────────────────────────────────────────────────
interface GeminiMessage {
role: string;
content: string;
}
interface GeminiRequestBody {
messages: GeminiMessage[];
model?: string;
stream?: boolean;
}
// ─── Helpers ────────────────────────────────────────────────────────────────
function formatChatCompletion(content: string, model: string, finishReason = "stop") {
return {
id: `chatcmpl-${Date.now()}`,
object: "chat.completion",
created: Math.floor(Date.now() / 1000),
model,
choices: [{ index: 0, message: { role: "assistant", content }, finish_reason: finishReason }],
usage: { prompt_tokens: 0, completion_tokens: 0, total_tokens: 0 },
};
}
function formatStreamChunk(content: string, model: string, finishReason: string | null = null) {
return {
id: `chatcmpl-${Date.now()}`,
object: "chat.completion.chunk",
created: Math.floor(Date.now() / 1000),
model,
choices: [{ index: 0, delta: content ? { content } : {}, finish_reason: finishReason }],
};
}
/**
* Flatten the OpenAI-style multi-turn `messages[]` into the single plain-text
* prompt typed into the Gemini web UI (#8371).
*
* gemini-web drives a real browser page and captures only the FIRST
* `StreamGenerate` response, so — unlike claude-web — it has no upstream
* conversation id to thread across turns. It is therefore a stateless,
* single-turn provider: the previous code forwarded only the last user message
* (`messages.filter(m => m.role === "user").pop()`), so follow-up questions
* lost all prior context ("I am in Berlin" → "What should I wear today?" was
* answered without Berlin). This implements the issue's accepted fallback (b):
* flatten the full history into one prompt so the web UI still sees the
* conversation.
*
* Single-turn requests are preserved byte-for-byte (only the final user message
* is returned) — the regression guard for the pre-existing no-tools path.
* Multi-turn requests emit a labeled transcript:
*
* System:
* <system text>
*
* Previous conversation:
* User: ...
* Assistant: ...
*
* Current user message:
* <last user message>
*/
export function buildGeminiPrompt(messages: Array<{ role: string; content: unknown }>): string {
const textMessages = messages.filter(
(m) => typeof m.content === "string" && (m.content as string).trim().length > 0
) as Array<{ role: string; content: string }>;
const userMessages = textMessages.filter((m) => m.role === "user");
const lastUser = userMessages[userMessages.length - 1];
const lastUserContent = lastUser?.content ?? "";
const lastUserIdx = lastUser ? textMessages.lastIndexOf(lastUser) : -1;
// Prior conversation = every user/assistant turn before the final user turn.
const priorTurns = textMessages.filter(
(m, i) => i < lastUserIdx && (m.role === "user" || m.role === "assistant")
);
// Single-turn (no earlier user/assistant turns): byte-for-byte the original
// single-message derivation. Do NOT prepend system text here — the old
// no-tools path ignored a system-only prefix on the first turn.
if (priorTurns.length === 0) return lastUserContent;
const systemText = textMessages
.filter((m) => m.role === "system")
.map((m) => m.content)
.join("\n\n");
const historyLines = priorTurns.map(
(m) => `${m.role === "assistant" ? "Assistant" : "User"}: ${m.content}`
);
const parts: string[] = [];
if (systemText) parts.push(`System:\n${systemText}`);
parts.push(`Previous conversation:\n${historyLines.join("\n\n")}`);
parts.push(`Current user message:\n${lastUserContent}`);
return parts.join("\n\n");
}
/**
* Build the plain-text prompt typed into the Gemini web UI when a tool
* contract is active — the synthetic system message injected by
* `prepareToolMessages()` prepended to the last user message. gemini-web
* only ever sends a single flat string (no native message array), so the
* tool contract and the user's ask are concatenated (#7286).
*/
export function buildGeminiToolPrompt(
effectiveMessages: Array<{ role: string; content: unknown }>
): string {
const toolSystemMsg = effectiveMessages.find((m) => m.role === "system");
const lastUserMsg = [...effectiveMessages].reverse().find((m) => m.role === "user");
const userText = typeof lastUserMsg?.content === "string" ? lastUserMsg.content : "";
const toolPrompt = typeof toolSystemMsg?.content === "string" ? toolSystemMsg.content : "";
return toolPrompt ? `${toolPrompt}\n\n${userText}` : userText;
}
/**
* Tool mode: wrap the buffered Gemini response text in the standard OpenAI
* completion shape, then delegate to the shared `buildToolModeResponse()`
* (`chatgptWebTools.ts`) — parses `<tool>{...}</tool>` blocks out of the
* text into `tool_calls` (malformed JSON degrades to ordinary `content`,
* never throws) and replays either buffered JSON or a terminal SSE chunk
* depending on `stream` (#7286). Exported standalone so the branching logic
* is testable without a full Playwright mock.
*/
export async function buildGeminiToolResponse(
responseText: string,
requestedTools: unknown,
stream: boolean,
model: string,
cid: string,
created: number
): Promise<Response> {
const bufferedJson = new Response(JSON.stringify(formatChatCompletion(responseText, model)), {
status: 200,
headers: { "Content-Type": "application/json" },
});
return buildToolModeResponse(bufferedJson, requestedTools, stream, {
cid,
created,
model,
idSeed: "gwe",
});
}
/**
* Parse cookie string, stripping attributes (Path, Domain, Expires, etc.)
* Input: full browser cookie string or just "name=value; name2=value2"
* Output: array of { name, value } pairs
*/
function parseCookies(raw: string): Array<{ name: string; value: string }> {
return raw
.split(";")
.map((part) => part.trim())
.filter(Boolean)
.map((part) => {
const eqIdx = part.indexOf("=");
if (eqIdx === -1) return null;
const name = part.substring(0, eqIdx).trim();
const value = part.substring(eqIdx + 1).trim();
// Skip cookie attributes that aren't name=value pairs
if (!name || !value) return null;
const lowerName = name.toLowerCase();
if (
["path", "domain", "expires", "max-age", "secure", "httponly", "samesite"].includes(
lowerName
)
) {
return null;
}
return { name, value };
})
.filter(Boolean) as Array<{ name: string; value: string }>;
}
/**
* Parse Gemini StreamGenerate response text.
*
* Response format:
* )]}'
* <length>
* [["wrb.fr", null, "<JSON string>"]]
* <length>
* [["wrb.fr", null, "<JSON string>"]]
*
* The JSON string contains nested array: inner[4][0][1] = ["text chunks"].
* Each wrb.fr line is a CUMULATIVE snapshot of the whole answer generated so
* far (not an independent delta), so we keep only the text from the LAST
* frame that yields non-empty text instead of concatenating every frame —
* concatenating would reproduce the same growing text with each snapshot
* (see #7163).
*/
export function parseStreamResponse(raw: string): string {
const lines = raw.split("\n");
let lastText = "";
for (const rawLine of lines) {
const line = rawLine.trim();
if (!line || line === ")]}'" || /^\d+$/.test(line)) continue;
if (!line.includes("wrb.fr")) continue;
try {
const arr = JSON.parse(line);
if (!Array.isArray(arr) || !Array.isArray(arr[0]) || arr[0][0] !== "wrb.fr") continue;
const payload = arr[0]?.[2];
if (typeof payload !== "string") continue;
const inner = JSON.parse(payload);
// Defensive: check each level before accessing
const responseArray = inner?.[4]?.[0]?.[1];
if (!Array.isArray(responseArray)) continue;
const text = responseArray.filter((c: unknown) => typeof c === "string").join("");
if (text) lastText = text;
} catch {
// Skip unparseable lines
}
}
return lastText;
}
/**
* Extract generated-image URLs from a Gemini StreamGenerate response (#10466).
*
* When the web UI generates images (Nano Banana), the model's answer frames
* carry the assets in the candidate's extension block, NOT in the text:
*
* inner[4][0][12][7][0] → array of generated-image entries
* entry[0][3][3] → the image URL — either a plain string or a
* list of strings (take the first http(s) one)
*
* This path is corroborated by the two maintained reverse-engineered clients
* (gpt4free's Gemini provider and HanaokaYuzu/Gemini-API's _parse_candidate).
* Deliberately NOT collected: `inner[4][0][12][1]` — those are web-search
* result thumbnails, not generated content; mixing them in would serve
* scraped images as "generated" (#10466 acceptance criteria).
*
* Frames are cumulative snapshots, so later frames repeat earlier images;
* we dedupe while preserving first-seen order. A `=s2048` size suffix is
* appended (gpt4free's proven heuristic) so callers get full-resolution
* assets instead of UI thumbnails.
*/
export function parseStreamResponseImages(raw: string): string[] {
const urls: string[] = [];
const seen = new Set<string>();
const lines = raw.split("\n");
for (const rawLine of lines) {
const line = rawLine.trim();
if (!line || line === ")]}'" || /^\d+$/.test(line)) continue;
if (!line.includes("wrb.fr")) continue;
try {
const arr = JSON.parse(line);
if (!Array.isArray(arr) || !Array.isArray(arr[0]) || arr[0][0] !== "wrb.fr") continue;
const payload = arr[0]?.[2];
if (typeof payload !== "string") continue;
const inner = JSON.parse(payload);
const imageEntries = inner?.[4]?.[0]?.[12]?.[7]?.[0];
if (!Array.isArray(imageEntries)) continue;
for (const entry of imageEntries) {
const urlField = entry?.[0]?.[3]?.[3];
let url = "";
if (typeof urlField === "string") {
url = urlField;
} else if (Array.isArray(urlField)) {
const firstHttp = urlField.find(
(u: unknown) => typeof u === "string" && /^https?:\/\//.test(u)
);
url = typeof firstHttp === "string" ? firstHttp : "";
}
if (!url || !/^https?:\/\//.test(url)) continue;
// Upgrade to full resolution unless a size directive is already present
// (googleusercontent size syntax: trailing `=s2048`, `=w1024-h512`, ...).
if (!/=[swh]\d+/.test(url)) url += "=s2048";
if (seen.has(url)) continue;
seen.add(url);
urls.push(url);
}
} catch {
// Skip unparseable lines
}
}
return urls;
}
function readCredentialString(value: unknown): string {
if (typeof value !== "string") return "";
const trimmed = value.trim();
return trimmed.length > 0 ? trimmed : "";
}
function readProviderSpecificString(
providerSpecificData: unknown,
keys: readonly string[]
): string {
if (
!providerSpecificData ||
typeof providerSpecificData !== "object" ||
Array.isArray(providerSpecificData)
) {
return "";
}
const data = providerSpecificData as Record<string, unknown>;
for (const key of keys) {
const value = readCredentialString(data[key]);
if (value) return value;
}
return "";
}
/**
* Merge rotated __Secure-1PSID* cookies read back from the live Playwright
* cookie jar into the original cookie string. Only the three long-lived
* Gemini auth cookies are considered — pulling in the entire jar would risk
* treating short-lived Google analytics/consent cookies as credentials
* (#7676). Cookies the jar didn't return, or that are unchanged, are left
* untouched in the original string.
*/
export function mergeRotatedGeminiCookies(
originalCookie: string,
jarCookies: Array<{ name: string; value: string }>
): string {
const ROTATABLE_NAMES = ["__Secure-1PSID", "__Secure-1PSIDTS", "__Secure-1PSIDCC"];
const jarByName = new Map(jarCookies.map((c) => [c.name, c.value]));
const pairs = parseCookies(originalCookie);
const seen = new Set<string>();
const merged = pairs.map(({ name, value }) => {
seen.add(name);
if (ROTATABLE_NAMES.includes(name) && jarByName.has(name)) {
return { name, value: jarByName.get(name) as string };
}
return { name, value };
});
for (const name of ROTATABLE_NAMES) {
if (!seen.has(name) && jarByName.has(name)) {
merged.push({ name, value: jarByName.get(name) as string });
}
}
return merged.map(({ name, value }) => `${name}=${value}`).join("; ");
}
function normalizeGeminiCookieInput(raw: string, cookieName = "__Secure-1PSID"): string {
const trimmed = raw.trim();
if (!trimmed) return "";
return trimmed.includes("=") ? trimmed : `${cookieName}=${trimmed}`;
}
function resolveGeminiWebCookie(credentials: ExecuteInput["credentials"]): string {
const directCookie =
readCredentialString(credentials?.apiKey) ||
readCredentialString((credentials as Record<string, unknown> | undefined)?.cookie);
if (directCookie) return normalizeGeminiCookieInput(directCookie);
const providerSpecificData = credentials?.providerSpecificData;
const cookie = readProviderSpecificString(providerSpecificData, ["cookie"]);
if (cookie) return normalizeGeminiCookieInput(cookie);
const psid = readProviderSpecificString(providerSpecificData, ["__Secure-1PSID"]);
const psidts = readProviderSpecificString(providerSpecificData, ["__Secure-1PSIDTS"]);
return [
psid ? normalizeGeminiCookieInput(psid, "__Secure-1PSID") : "",
psidts ? normalizeGeminiCookieInput(psidts, "__Secure-1PSIDTS") : "",
]
.filter(Boolean)
.join("; ");
}
// ─── Executor ───────────────────────────────────────────────────────────────
export class GeminiWebExecutor extends BaseExecutor {
constructor() {
super("gemini-web", { id: "gemini-web", baseUrl: GEMINI_URL });
}
/**
* testConnection — validates the cookie format without making a network call
* or launching Playwright. Returns true when the cookie is non-empty and
* contains at least one name=value pair with a non-empty value. This is a
* lightweight pre-check before the browser automation path; full session
* validation is done by validateGeminiWebProvider in the connection test
* flow (#9407).
*/
async testConnection(
credentials: Record<string, unknown>,
_signal?: AbortSignal
): Promise<boolean> {
try {
const cookie = resolveGeminiWebCookie(credentials as unknown as ExecuteInput["credentials"]);
if (!cookie) return false;
const pairs = parseCookies(cookie);
return pairs.some((p) => p.value.length > 0);
} catch {
return false;
}
}
/**
* Read the live Playwright cookie jar back after a successful run and, if
* Google rotated any of the __Secure-1PSID* cookies, forward the merged
* cookie string through onCredentialsRefreshed so it gets persisted to the
* encrypted provider_connections.api_key field. Mirrors the rotate-and-
* persist pattern already shipped in chatgpt-web.ts. A persistence failure
* must never fail the user-facing response (#7676).
*/
private async persistRotatedCookies(
context: import("playwright").BrowserContext,
cookie: string,
credentials: ExecuteInput["credentials"],
onCredentialsRefreshed: ExecuteInput["onCredentialsRefreshed"],
log: ExecuteInput["log"]
): Promise<void> {
if (!onCredentialsRefreshed) return;
try {
const jarCookies = await context.cookies();
const mergedCookie = mergeRotatedGeminiCookies(cookie, jarCookies);
if (mergedCookie && mergedCookie !== cookie) {
await onCredentialsRefreshed({ ...credentials, apiKey: mergedCookie });
}
} catch (err) {
log?.warn?.(
"GEMINI-WEB",
`Failed to persist rotated cookie: ${err instanceof Error ? err.message : String(err)}`
);
}
}
async execute(input: ExecuteInput) {
const { model, body, stream, credentials, signal, log, onCredentialsRefreshed } = input;
const requestBody = body as GeminiRequestBody;
// #9356: fail fast on controls this provider cannot honor (reasoning_effort
// above "minimal", forced tool_choice). Runs before the credential check and
// before Playwright launches — the request is unservable no matter which
// cookie is used, and answering 200 with ordinary prose made agents believe
// their reasoning/tool requirements had been met. See ./gemini-web/capabilities.ts.
const violation = checkGeminiWebUnsupportedControls(body as Record<string, unknown>);
if (violation) {
log?.warn?.(
"GEMINI-WEB",
`Rejected request: "${violation.param}" is not supported by this provider`
);
return {
response: new Response(
JSON.stringify(
buildErrorBody(400, violation.message, null, {
type: "invalid_request_error",
code: GEMINI_WEB_UNSUPPORTED_CONTROL_CODE,
})
),
{ status: 400, headers: { "Content-Type": "application/json" } }
),
url: GEMINI_URL,
headers: {},
transformedBody: body,
};
}
const cookie = resolveGeminiWebCookie(credentials);
if (!cookie) {
return {
response: new Response(JSON.stringify({ error: "Missing Gemini cookies" }), {
status: 401,
headers: { "Content-Type": "application/json" },
}),
url: GEMINI_URL,
headers: {},
transformedBody: body,
};
}
const messages = requestBody.messages || [];
const { hasTools, requestedTools, effectiveMessages } = prepareToolMessages(
body as Record<string, unknown>,
messages
);
// hasTools === false: flatten the full multi-turn history into the single
// prompt so gemini-web (a stateless web-cookie provider that captures only
// the first StreamGenerate response) preserves prior context across turns
// (#8371). Single-turn requests stay byte-for-byte identical to the original
// derivation, keeping the #7286 no-tools regression guard intact.
const prompt = hasTools
? buildGeminiToolPrompt(effectiveMessages)
: buildGeminiPrompt(messages);
if (!prompt) {
return {
response: new Response(JSON.stringify({ error: "No user message found" }), {
status: 400,
headers: { "Content-Type": "application/json" },
}),
url: GEMINI_URL,
headers: {},
transformedBody: body,
};
}
let browser: any = null;
let abortBrowser: (() => void) | null = null;
try {
if (signal?.aborted) {
throw signal.reason instanceof Error ? signal.reason : new Error("Request aborted");
}
const { chromium } = await import("playwright");
browser = await chromium.launch({ headless: true });
abortBrowser = () => {
void browser?.close().catch(() => {});
};
signal?.addEventListener("abort", abortBrowser, { once: true });
const context = await browser.newContext({ userAgent: GEMINI_USER_AGENT });
// Parse cookies — strips attributes like Path, Domain, Expires
const cookiePairs = parseCookies(cookie);
await context.addCookies(
cookiePairs.map(({ name, value }) => ({
name,
value,
domain: ".google.com",
path: "/",
secure: true,
}))
);
const page = await context.newPage();
// #10466: image mode — the /v1/images/generations handler sets
// x_gemini_web_image_mode. Generated images arrive in the candidate's
// extension block ([12][7][0]) of the StreamGenerate frames, sometimes
// only in a LATER frame of the stream (or a follow-up StreamGenerate
// call), so image mode captures every StreamGenerate response, merges
// image URLs across frames, and resolves as soon as one is found.
// Chat mode keeps the original first-response-only behavior.
const imageMode = (body as Record<string, unknown>)?.x_gemini_web_image_mode === true;
// Capture first StreamGenerate response
let responseText = "";
const responseImages: string[] = [];
let captured = false;
const responsePromise = new Promise<void>((resolve) => {
page.on("response", async (resp: any) => {
if (!resp.url().includes("StreamGenerate")) return;
if (!imageMode && captured) return;
if (imageMode) {
// Image mode: merge text + image URLs across every frame and
// resolve as soon as an image appears (images can land in a
// later frame than the text).
try {
const raw = await resp.text();
const text = parseStreamResponse(raw);
if (text) responseText = text;
for (const url of parseStreamResponseImages(raw)) {
if (!responseImages.includes(url)) responseImages.push(url);
}
} catch {
/* ignore unreadable frames */
}
if (responseImages.length > 0) resolve();
} else {
// Chat mode: byte-for-byte the original first-response capture —
// resolve even if reading the body throws, so the flow falls
// through to the "No response from Gemini" 502 instead of
// burning the full wait window.
captured = true;
try {
const raw = await resp.text();
responseText = parseStreamResponse(raw);
} catch {
/* ignore */
}
resolve();
}
});
});
await page.goto(GEMINI_URL, { waitUntil: "domcontentloaded", timeout: 20000 });
if (signal?.aborted) {
throw signal.reason instanceof Error ? signal.reason : new Error("Request aborted");
}
await page.waitForTimeout(3000);
// Type and send message
const inputEl = await page.waitForSelector(".ql-editor, [contenteditable='true']", {
timeout: 10000,
});
await inputEl.click();
await page.keyboard.type(prompt, { delay: 10 });
await page.waitForTimeout(300);
await page.keyboard.press("Enter");
// Wait for response or timeout. Image generation (Nano Banana) is
// noticeably slower than text — the UI renders the asset only after
// the full generation completes — so image mode gets a wider window.
await Promise.race([responsePromise, page.waitForTimeout(imageMode ? 90000 : 30000)]);
if (signal?.aborted) {
throw signal.reason instanceof Error ? signal.reason : new Error("Request aborted");
}
// #10466 image mode: return the captured image URLs to the image
// handler via a custom field (same precedent as chatgpt-web's
// x_image_resolution_failed). An image-only answer can carry little or
// no text, so the empty-text 502 below must not fire when images
// were captured.
if (imageMode) {
await this.persistRotatedCookies(context, cookie, credentials, onCredentialsRefreshed, log);
const modelId = model || "gemini-2.5-pro";
return {
response: new Response(
JSON.stringify({
...formatChatCompletion(responseText, modelId),
x_gemini_web_image_urls: responseImages,
}),
{ status: 200, headers: { "Content-Type": "application/json" } }
),
url: GEMINI_URL,
headers: {},
transformedBody: body,
};
}
if (!responseText) {
return {
response: new Response(JSON.stringify({ error: "No response from Gemini" }), {
status: 502,
headers: { "Content-Type": "application/json" },
}),
url: GEMINI_URL,
headers: {},
transformedBody: body,
};
}
await this.persistRotatedCookies(context, cookie, credentials, onCredentialsRefreshed, log);
const modelId = model || "gemini-2.5-pro";
if (hasTools) {
const cid = `chatcmpl-gwe-${crypto.randomUUID().slice(0, 12)}`;
const created = Math.floor(Date.now() / 1000);
const toolResponse = await buildGeminiToolResponse(
responseText,
requestedTools,
Boolean(stream),
modelId,
cid,
created
);
return { response: toolResponse, url: GEMINI_URL, headers: {}, transformedBody: body };
}
if (stream) {
// Pseudo-streaming: send complete response as single SSE chunk
// Gemini's StreamGenerate returns complete responses, not chunked streams
const encoder = new TextEncoder();
const readable = new ReadableStream(
{
start(controller) {
controller.enqueue(
encoder.encode(
`data: ${JSON.stringify(formatStreamChunk(responseText, modelId))}\n\n`
)
);
controller.enqueue(
encoder.encode(
`data: ${JSON.stringify(formatStreamChunk("", modelId, "stop"))}\n\n`
)
);
controller.enqueue(encoder.encode("data: [DONE]\n\n"));
controller.close();
},
},
{ highWaterMark: 16384 }
);
return {
response: new Response(readable, {
status: 200,
headers: {
"Content-Type": "text/event-stream",
"Cache-Control": "no-cache",
Connection: "keep-alive",
},
}),
url: GEMINI_URL,
headers: {},
transformedBody: body,
};
}
return {
response: new Response(JSON.stringify(formatChatCompletion(responseText, modelId)), {
status: 200,
headers: { "Content-Type": "application/json" },
}),
url: GEMINI_URL,
headers: {},
transformedBody: body,
};
} catch (error) {
const rawMessage = error instanceof Error ? error.message : "Unknown error";
// #3516: a missing Playwright browser is a host/config problem, not a transient upstream
// fault. Surface an actionable error and tag it with the connection-cooldown hint so
// accountFallback skips the provider circuit breaker and applies a short, non-exponential
// cooldown instead of looping on a retryable 500.
if (isMissingBrowserExecutable(rawMessage)) {
return {
response: new Response(
JSON.stringify({
error:
"Gemini Web requires the Playwright Chromium browser, which is not installed. " +
"Run `npx playwright install chromium` on the host (or rebuild the Docker image with browsers).",
}),
{
status: 503,
headers: {
"Content-Type": "application/json",
"X-Omni-Fallback-Hint": "connection_cooldown",
},
}
),
url: GEMINI_URL,
headers: {},
transformedBody: body,
};
}
// #9407: Playwright selector/click timeout errors are terminal — they indicate
// the page DOM does not match expectations (e.g. Gemini changed their UI or
// the session is so expired it lands on a different page). Return 400 so the
// account-fallback system does NOT retry this request as a transient 5xx.
if (
error instanceof Error &&
(error.name === "TimeoutError" ||
rawMessage.includes("waitForSelector") ||
rawMessage.includes("Timeout") ||
rawMessage.includes("actionability") ||
rawMessage.includes("interception"))
) {
return {
response: new Response(
JSON.stringify({
error: sanitizeErrorMessage(rawMessage),
}),
{ status: 400, headers: { "Content-Type": "application/json" } }
),
url: GEMINI_URL,
headers: {},
transformedBody: body,
};
}
return {
response: new Response(
JSON.stringify({
error: sanitizeErrorMessage(rawMessage),
}),
{ status: 500, headers: { "Content-Type": "application/json" } }
),
url: GEMINI_URL,
headers: {},
transformedBody: body,
};
} finally {
if (abortBrowser) signal?.removeEventListener("abort", abortBrowser);
// Always close browser to prevent resource leaks
if (browser) {
try {
await browser.close();
} catch {
/* ignore close errors */
}
}
}
}
}