* fix(claude-web): add charset=utf-8 to Content-Type headers to fix Arabic/Persian UTF-8 mojibake
Fixes#13416
The Claude Web endpoint and the outer SSE streaming pipeline were returning
Content-Type headers without an explicit charset parameter:
- stream.ts responseHeaders() returned 'application/json' and
'text/event-stream' without charset
- responseHeaders.ts buildStreamingResponseHeaders() returned
'text/event-stream' without charset
While RFC 8259 defaults JSON to UTF-8 and the SSE spec defaults
text/event-stream to UTF-8, some HTTP clients (notably VS Code Chat on
Windows) fall back to ISO-8859-1/Latin-1 when no charset is declared,
causing multi-byte UTF-8 characters to appear as mojibake.
For example, the Persian word for hello (سلام, UTF-8 bytes D8 B3 D9 84 D8 A7
D9 85) was decoded as Latin-1, producing the garbled output 'سلام'.
Fix:
- claude-web/stream.ts: append '; charset=utf-8' to the Content-Type
header in the responseHeaders() helper, with a guard to avoid double
appending if the caller already includes a charset
- chatCore/responseHeaders.ts: hardcode 'text/event-stream; charset=utf-8'
in buildStreamingResponseHeaders()
Tests:
- 11 new regression tests in claude-web-utf8-mojibake-13416.test.ts
covering Persian, Arabic, mixed-script, emoji, and chunk-boundary-split
scenarios across both streaming and buffered response paths
- All 7 existing claude-web-stream tests pass
- All 13 response header tests pass
- All 8 adaptive-admission-lifecycle tests pass
* test(sse): confirm streaming charset header and add byte-level UTF-8 repro (#13416)
Independently verified the mojibake root cause before trusting the charset
fix: OmniRoute's claude-web decoder already reconstructs a Persian/Arabic
multi-byte UTF-8 sequence split across a chunk boundary correctly because
it decodes with TextDecoder({ stream: true }); a naive per-chunk decode
(without stream state) is what actually produces the U+FFFD garbling.
Updated the 2 exact Content-Type assertions in chat-pipeline.test.ts to
match the new "text/event-stream; charset=utf-8" header, which is already
the convention used by the other streaming executors (uc.ts, maxai.ts,
codex-app-server.ts, etc).
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: Koosha Pari <koosha@phenotype.ai>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
20f3900889