mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-08-06 07:12:12 +03:00
Two changes with one root cause: several hot paths built a full JSON string only to read
its .length, and one of them silently changed the answer.
1. CORRECTNESS -- combo's fallback-compression trigger
estimateTokens(JSON.stringify(attemptBody)) took the STRING branch of estimateTokens,
which is ceil(length / CHARS_PER_TOKEN) over the raw JSON. An inline base64 image is
then charged as if every character of the data URL were prose. Measured on a 200 KB
inline image:
via string (before) 50,039 tokens
via object (after) 1,231 tokens
a 40x over-count, tripping fallback compression on requests nowhere near the context
window. This is the same class #8368/#8401 fixed on the request path; the combo call
site was missed. Passing the object routes through extractImageTokens, which charges
images structurally. Text-only bodies are unaffected -- verified identical, and pinned
by a test.
2. ALLOCATION -- jsonLength()
Adds an exact serialized-length walker: same O(n) scan, no string. Used by
estimateTokens' object branch and by streamReadinessPolicy (which runs on every
streaming request and only ever used .length).
Exactness matters because every consumer feeds a threshold, so this is property-tested
against JSON.stringify over 4000 generated structures covering escaping, lone
surrogates, omitted values, non-finite numbers, toJSON, Date, Map, cycles and BigInt.
Anything outside the plain-JSON subset falls back to JSON.stringify for THAT SUBTREE
only, so an exotic leaf never forces the message history back onto the allocating path.
Honest scoping of the memory win: the string was always transient, and V8 collects it
efficiently, so this is not 3 MiB of retained heap. Measured allocation churn over 20 calls
on a 3.06 MiB body: 3.1 MiB -> 0.5 MiB, about 6x less. The #8549 benchmark row for this
mechanism measures a HELD string and therefore overstates it; the correctness fix above is
the larger deliverable here.
196 lines
6.3 KiB
TypeScript
196 lines
6.3 KiB
TypeScript
// jsonLength() must equal JSON.stringify().length exactly (#7847).
|
||
//
|
||
// Every consumer feeds a threshold — an adaptive readiness timeout, a compression trigger, a
|
||
// payload-size metric. An approximation would silently shift routing decisions, so "close
|
||
// enough" is not acceptable and this is property-tested rather than spot-checked.
|
||
import { test } from "node:test";
|
||
import assert from "node:assert/strict";
|
||
|
||
const { jsonLength } = await import("../../open-sse/utils/jsonSize.ts");
|
||
|
||
const exact = (value: unknown, label?: string) => {
|
||
const expected = JSON.stringify(value);
|
||
assert.equal(
|
||
jsonLength(value),
|
||
expected === undefined ? 0 : expected.length,
|
||
label ?? `mismatch for ${expected === undefined ? "undefined" : expected.slice(0, 120)}`
|
||
);
|
||
};
|
||
|
||
// ── Deterministic generator (no Math.random: a flaky property test is worthless) ──
|
||
function lcg(seed: number): () => number {
|
||
let s = seed >>> 0;
|
||
return () => (s = (s * 1664525 + 1013904223) >>> 0) / 0x100000000;
|
||
}
|
||
|
||
const NASTY_STRINGS = [
|
||
"",
|
||
"plain",
|
||
'has "quotes"',
|
||
"back\\slash",
|
||
"new\nline\ttab\r\b\f",
|
||
" |