mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-08-01 12:52:11 +03:00
* test(compression): TDD failing tests for hard-budget post-pass (#17) node:test suite for applyHardBudget — targetTokens, targetRatio, no-op, force-preserve, determinism, both-wins, techniquesUsed, integration seam. All tests fail (module not yet created) — TDD red step. * feat(compression): hard-budget post-pass (#17) — compress to exactly N tokens - types.ts: add targetTokens? and targetRatio? to CompressionConfig after contextBudget - hardBudget.ts: applyHardBudget(body, {targetTokens?,targetRatio?}) → CompressionResult; splits prose into sentences/lines, ranks by avg scoreToken ascending, drops lowest- saliency units until ≤ target; UNIT_PRESERVE_RE guards numbers/URLs/errors/code; targetTokens wins when both set; techniquesUsed:["hard-budget"] - strategySelector.ts: hard-budget post-pass in runStackedCompression + runStackedCompressionAsync after engine loop, before finalizeStackedResult; gated on config.targetTokens || config.targetRatio; mergeStackStep + compressed=true * fix(compression): preserve digit-less sensitive lines in hard-budget UNIT_PRESERVE_RE only matched digits/URLs/error-headers/code-fences, so stack-trace at-frames, key=value credential lines, and digit-less paths were droppable and could be cut to hit the budget. Add specific anchors (^\s*at\s, \/[\w.-]+\/, [A-Za-z_]\w*=\S) that never match a bare end-of-sentence period (which would re-break the feature into a no-op). Regression tests drive each unit to target=1 (drops every non-preserved unit) and assert the sensitive line survives; plus a guard that plain prose ending in a period stays droppable. * fix(compression): distribute hard-budget across messages + warn when unreachable Two related correctness fixes in applyHardBudget: - Aggregate target was passed verbatim to compressText for EACH message, so an N-message body could come back ~N× over budget. Distribute the target proportionally per message (floor(target * msgTokens/total)) so the SUM stays <= target. - When every unit is preserve-guarded (or a single oversized preserved unit), the result still exceeds target with no signal. Measure the result and push a validationWarnings entry when it remains over budget, so callers are not silently left over the limit. Regression tests: a 4-message body with target 200 ends with TOTAL <= 200; an all-numeric (all-preserved) body emits the 'could not reach target' warning. * fix(compression): run hard-budget post-pass when targetTokens/targetRatio is 0 The seam gate used `||`, so targetTokens:0 or targetRatio:0 (both falsy) silently skipped the post-pass in runStackedCompression and runStackedCompressionAsync. Switch to `!= null` so an explicit 0 still engages the pass. Regression test: applyStackedCompression with config.targetTokens:0 must report 'hard-budget' in techniquesUsed. * docs(changelog): restore hard-budget bullet (#17, eaten by rebase)
This commit is contained in:
Diego Rodrigues de Sa e Souza
committed by
GitHub
GitHub
parent
c8833fd3e0
commit
fa94f38de5
@@ -14,6 +14,7 @@ _In development — bullets added per PR; finalized at release._
|
||||
|
||||
### ✨ New Features
|
||||
|
||||
- **feat(compression): hard-budget mode — compress to ≤ N tokens** — a deterministic post-pass (`targetTokens` / `targetRatio`, default unset → no-op) that trims a body to a token budget. It ranks sentences/lines by average `scoreToken` ascending and drops the lowest-saliency ones until the body fits (measured by the exact cl100k `countTextTokens`), preserving original order. Lines carrying real signal (digits, URLs, `Error:`-family, code fences, stack `at`-frames, multi-segment paths, `key=value`) are never dropped; the budget is distributed proportionally across messages so the total stays ≤ target; an unreachable target (all-preserved) surfaces a `validationWarnings` note instead of failing silently. Does NOT touch the `estimateCompressionTokens` budget-gate estimator. Tier-3 item of the compression feature-extraction roadmap (#17).
|
||||
- **feat(compression): result memoization for deterministic engines (opt-in)** — caches `(input, config) → result` for provably pure, stateless modes (`lite`/`standard`/`rtk` and stacked pipelines of `{lite,caveman,rtk}`) to skip recompute on the hot path. Opt-in via `memoizeCompressionResults` (default off → zero behavior change). Conservative opt-in whitelist (stateful `ccr`/`session-dedup` — which write the cross-request CCR store — and model-backed `ultra`/`aggressive`/`llmlingua` are never cached), principal-scoped (skipped without a principal, so no cross-principal body leak), and clone-on-store + clone-on-read. Tier-3 item of the compression feature-extraction roadmap (#21).
|
||||
- **feat(compression): inline transparency annotation** — surfaces `tokens=847→312; rules: filler×8, dedup×2` derived from existing compression stats. The `X-OmniRoute-Compression` response header is extended **append-only** (the `mode; source=X` prefix stays byte-identical, so existing header parsers don't break) and the compression studio cockpit shows a matching badge. Zero new computation — it aggregates the `rulesApplied`/`techniquesUsed` already on the stats. Tier-3 item of the compression feature-extraction roadmap (#18).
|
||||
- **feat(compression): saliency heatmap in the compression studio** — the preview studio can now color each token by saliency: `ultra` per-token `scoreToken` (0–1, green→red gradient) or universal kept/removed from the existing diff. A dry-run visualization behind a toggle (no cost on a normal preview; backward-compatible when off). Completes the visualization half of roadmap item #13 (the A/B comparison shipped in [#5080](https://github.com/diegosouzapw/OmniRoute/pull/5080)).
|
||||
|
||||
196
open-sse/services/compression/hardBudget.ts
Normal file
196
open-sse/services/compression/hardBudget.ts
Normal file
@@ -0,0 +1,196 @@
|
||||
/**
|
||||
* Hard-budget post-pass (#17): compress to ≤ N cl100k tokens.
|
||||
*
|
||||
* Deterministic, independent of the relevance engine. Splits prose into
|
||||
* sentences/lines, ranks by average scoreToken ascending, drops the lowest-
|
||||
* saliency units until the body fits the target, then reconstructs original order.
|
||||
* Units containing FORCE_PRESERVE_RE anchors (errors, numbers, URLs, code) are
|
||||
* never dropped.
|
||||
*/
|
||||
|
||||
import type { CompressionResult } from "./types.ts";
|
||||
import { scoreToken } from "./ultraHeuristic.ts";
|
||||
import { countTextTokens } from "../../../src/shared/utils/tiktokenCounter.ts";
|
||||
import { createCompressionStats } from "./stats.ts";
|
||||
|
||||
interface HardBudgetOptions {
|
||||
targetTokens?: number;
|
||||
targetRatio?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Units containing these patterns must never be dropped.
|
||||
* Anchored to meaningful signals only — never matches a bare end-of-sentence
|
||||
* period (which would make every prose line "preserve" → permanent no-op):
|
||||
* - `\d` digits (numbers, line refs, ports)
|
||||
* - `https?:\/\/` URLs
|
||||
* - `(?:Error|…|Traceback):` error/exception headers
|
||||
* - "```" code fences
|
||||
* - `^\s*at\s` stack-trace frames (digit-less)
|
||||
* - `\/[\w.-]+\/` real multi-segment paths (but NOT "and/or")
|
||||
* - `[A-Za-z_]\w*=\S` key=value (credential/config lines, digit-less)
|
||||
*/
|
||||
const UNIT_PRESERVE_RE =
|
||||
/\d|https?:\/\/|(?:Error|Exception|TypeError|RangeError|SyntaxError|ReferenceError|Traceback):|```|^\s*at\s|\/[\w.-]+\/|[A-Za-z_]\w*=\S/i;
|
||||
|
||||
/** Average scoreToken for each word in the unit (sentence/line). */
|
||||
function scoreUnit(unit: string): number {
|
||||
const words = unit.split(/\s+/).filter(Boolean);
|
||||
if (words.length === 0) return 0.5;
|
||||
const total = words.reduce((sum, w) => sum + scoreToken(w), 0);
|
||||
return total / words.length;
|
||||
}
|
||||
|
||||
/** Returns true when the unit must never be dropped. */
|
||||
function mustPreserve(unit: string): boolean {
|
||||
return UNIT_PRESERVE_RE.test(unit);
|
||||
}
|
||||
|
||||
/**
|
||||
* Split text into droppable units (lines, optionally sentence-split for long prose lines).
|
||||
*/
|
||||
function splitUnits(text: string): string[] {
|
||||
return text.split(/\n/).flatMap((line) => {
|
||||
if (line.trim() === "") return [line];
|
||||
// Only sentence-split pure prose lines (no numbers, URLs, error patterns, code)
|
||||
if (!UNIT_PRESERVE_RE.test(line) && line.length > 60) {
|
||||
const sentences = line.split(/(?<=[.!?])\s+/);
|
||||
return sentences.length > 1 ? sentences : [line];
|
||||
}
|
||||
return [line];
|
||||
});
|
||||
}
|
||||
|
||||
interface TaggedUnit {
|
||||
i: number;
|
||||
u: string;
|
||||
tokens: number;
|
||||
score: number;
|
||||
preserve: boolean;
|
||||
}
|
||||
|
||||
function tagUnits(units: string[]): TaggedUnit[] {
|
||||
return units.map((u, i) => ({
|
||||
i,
|
||||
u,
|
||||
tokens: countTextTokens(u),
|
||||
score: scoreUnit(u),
|
||||
preserve: mustPreserve(u),
|
||||
}));
|
||||
}
|
||||
|
||||
function dropToTarget(tagged: TaggedUnit[], targetTokens: number): Set<number> {
|
||||
const dropped = new Set<number>();
|
||||
let tokCount = tagged.reduce((s, x) => s + x.tokens, 0);
|
||||
|
||||
// Sort droppable candidates by score ascending (lowest first = drop first)
|
||||
const candidates = tagged
|
||||
.filter((x) => !x.preserve)
|
||||
.sort((a, b) => a.score - b.score);
|
||||
|
||||
for (const candidate of candidates) {
|
||||
if (tokCount <= targetTokens) break;
|
||||
dropped.add(candidate.i);
|
||||
tokCount -= candidate.tokens;
|
||||
}
|
||||
|
||||
return dropped;
|
||||
}
|
||||
|
||||
function rebuildText(tagged: TaggedUnit[], dropped: Set<number>): string {
|
||||
return tagged
|
||||
.filter((x) => !dropped.has(x.i))
|
||||
.map((x) => x.u)
|
||||
.join("\n");
|
||||
}
|
||||
|
||||
function compressText(text: string, targetTokens: number): string {
|
||||
const currentTokens = countTextTokens(text);
|
||||
if (currentTokens <= targetTokens) return text;
|
||||
|
||||
const units = splitUnits(text);
|
||||
if (units.length <= 1) return text;
|
||||
|
||||
const tagged = tagUnits(units);
|
||||
const dropped = dropToTarget(tagged, targetTokens);
|
||||
if (dropped.size === 0) return text;
|
||||
|
||||
return rebuildText(tagged, dropped);
|
||||
}
|
||||
|
||||
function extractMessages(body: Record<string, unknown>): Array<{ role: string; content: unknown }> {
|
||||
const msgs = body.messages;
|
||||
if (!Array.isArray(msgs)) return [];
|
||||
return msgs as Array<{ role: string; content: unknown }>;
|
||||
}
|
||||
|
||||
export function applyHardBudget(
|
||||
body: Record<string, unknown>,
|
||||
opts: HardBudgetOptions
|
||||
): CompressionResult {
|
||||
const { targetTokens, targetRatio } = opts;
|
||||
|
||||
if (targetTokens == null && targetRatio == null) {
|
||||
return { body, compressed: false, stats: null };
|
||||
}
|
||||
|
||||
const messages = extractMessages(body);
|
||||
if (messages.length === 0) return { body, compressed: false, stats: null };
|
||||
|
||||
// Measure total tokens across all messages
|
||||
const totalText = messages
|
||||
.map((m) => (typeof m.content === "string" ? m.content : JSON.stringify(m.content)))
|
||||
.join(" ");
|
||||
const totalTokens = countTextTokens(totalText);
|
||||
|
||||
// targetTokens wins when both are set
|
||||
const effectiveTarget =
|
||||
targetTokens != null
|
||||
? targetTokens
|
||||
: Math.floor(totalTokens * (targetRatio as number));
|
||||
|
||||
if (totalTokens <= effectiveTarget) {
|
||||
return { body, compressed: false, stats: null };
|
||||
}
|
||||
|
||||
// Distribute the aggregate budget proportionally per message so the SUM stays
|
||||
// ≤ target (passing the full target to each message would let an N-message body
|
||||
// come back N× over budget).
|
||||
const newMessages = messages.map((m) => {
|
||||
if (typeof m.content !== "string") return m;
|
||||
const msgTokens = countTextTokens(m.content);
|
||||
const perMsgTarget =
|
||||
totalTokens > 0 ? Math.floor(effectiveTarget * (msgTokens / totalTokens)) : effectiveTarget;
|
||||
const out = compressText(m.content, perMsgTarget);
|
||||
return out === m.content ? m : { ...m, content: out };
|
||||
});
|
||||
|
||||
const changed = newMessages.some(
|
||||
(m, i) => JSON.stringify(m) !== JSON.stringify(messages[i])
|
||||
);
|
||||
|
||||
// Measure the result to detect when preserve-guarded content makes the target
|
||||
// unreachable, so callers are not silently left over budget.
|
||||
const usedMessages = changed ? newMessages : messages;
|
||||
const resultTokens = countTextTokens(
|
||||
usedMessages
|
||||
.map((m) => (typeof m.content === "string" ? m.content : JSON.stringify(m.content)))
|
||||
.join(" ")
|
||||
);
|
||||
const overBudget = resultTokens > effectiveTarget;
|
||||
|
||||
// Nothing changed and we are within budget → genuine no-op.
|
||||
if (!changed && !overBudget) {
|
||||
return { body, compressed: false, stats: null };
|
||||
}
|
||||
|
||||
const newBody = changed ? { ...body, messages: newMessages } : body;
|
||||
const stats = createCompressionStats(body, newBody, "stacked", ["hard-budget"]);
|
||||
|
||||
if (overBudget) {
|
||||
const warning = `hard-budget: could not reach target (${resultTokens} > ${effectiveTarget}; preserved content exceeds budget)`;
|
||||
stats.validationWarnings = [...(stats.validationWarnings ?? []), warning];
|
||||
}
|
||||
|
||||
return { body: newBody, compressed: changed, stats };
|
||||
}
|
||||
@@ -5,6 +5,7 @@ import type {
|
||||
CompressionResult,
|
||||
CompressionStats,
|
||||
} from "./types.ts";
|
||||
import { applyHardBudget } from "./hardBudget.ts";
|
||||
import { type FidelityGateConfig } from "./fidelityGate.ts";
|
||||
import { gateAdvance } from "./fidelityGateStep.ts";
|
||||
import type { CompressionEngineApplyOptions } from "./engines/types.ts";
|
||||
@@ -859,6 +860,19 @@ function runStackedCompression(
|
||||
}
|
||||
}
|
||||
|
||||
// Hard-budget post-pass (#17): runs after all engines, before finalize.
|
||||
if (options?.config?.targetTokens != null || options?.config?.targetRatio != null) {
|
||||
const hbResult = applyHardBudget(currentBody, {
|
||||
targetTokens: options.config.targetTokens,
|
||||
targetRatio: options.config.targetRatio,
|
||||
});
|
||||
if (hbResult.compressed) {
|
||||
mergeStackStep(acc, "hard-budget", hbResult);
|
||||
currentBody = hbResult.body;
|
||||
compressed = true;
|
||||
}
|
||||
}
|
||||
|
||||
return finalizeStackedResult(
|
||||
body,
|
||||
currentBody,
|
||||
@@ -948,6 +962,19 @@ async function runStackedCompressionAsync(
|
||||
}
|
||||
}
|
||||
|
||||
// Hard-budget post-pass (#17): runs after all engines, before finalize.
|
||||
if (options?.config?.targetTokens != null || options?.config?.targetRatio != null) {
|
||||
const hbResult = applyHardBudget(currentBody, {
|
||||
targetTokens: options.config.targetTokens,
|
||||
targetRatio: options.config.targetRatio,
|
||||
});
|
||||
if (hbResult.compressed) {
|
||||
mergeStackStep(acc, "hard-budget", hbResult);
|
||||
currentBody = hbResult.body;
|
||||
compressed = true;
|
||||
}
|
||||
}
|
||||
|
||||
return finalizeStackedResult(
|
||||
body,
|
||||
currentBody,
|
||||
|
||||
@@ -183,6 +183,18 @@ export interface CompressionConfig {
|
||||
* shouldAutoTrigger branch is bypassed.
|
||||
*/
|
||||
contextBudget?: ContextBudgetConfig;
|
||||
/**
|
||||
* Hard-budget post-pass (#17): compress to at most this many cl100k tokens.
|
||||
* Runs after all stacked engines. Absent → no-op.
|
||||
* When both targetTokens and targetRatio are set, targetTokens wins.
|
||||
*/
|
||||
targetTokens?: number;
|
||||
/**
|
||||
* Hard-budget post-pass (#17): compress to at most this fraction (0–1) of original tokens.
|
||||
* Runs after all stacked engines. Absent → no-op.
|
||||
* When both targetTokens and targetRatio are set, targetTokens wins.
|
||||
*/
|
||||
targetRatio?: number;
|
||||
/**
|
||||
* Phase 4 (B): which tier the `ultra` mode uses.
|
||||
* "heuristic" = Tier-A token pruner (`pruneByScore`, default, byte-identical to pre-B).
|
||||
|
||||
339
tests/unit/compression/hard-budget.test.ts
Normal file
339
tests/unit/compression/hard-budget.test.ts
Normal file
@@ -0,0 +1,339 @@
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { applyHardBudget } from "../../../open-sse/services/compression/hardBudget.ts";
|
||||
import { countTextTokens } from "../../../src/shared/utils/tiktokenCounter.ts";
|
||||
import { applyStackedCompression } from "../../../open-sse/services/compression/strategySelector.ts";
|
||||
|
||||
// ~400-token prose fixture (longer sentences to ensure measurable token count)
|
||||
const PROSE = [
|
||||
"The quick brown fox jumps over the lazy dog and runs through the forest.",
|
||||
"Artificial intelligence systems can process large amounts of information efficiently.",
|
||||
"Machine learning models require substantial computational resources for training.",
|
||||
"Error: something went wrong in the processing pipeline at line 42.",
|
||||
"Natural language processing enables computers to understand human language semantically.",
|
||||
"Deep learning architectures consist of multiple interconnected layers of neurons.",
|
||||
"The database contains approximately 1234567 records from the past fiscal year.",
|
||||
"Transformer models have revolutionized the field of natural language understanding.",
|
||||
"Reinforcement learning allows agents to learn optimal policies through experience.",
|
||||
"The gradient descent algorithm iteratively minimizes the loss function during training.",
|
||||
"Convolutional neural networks excel at image recognition and classification tasks.",
|
||||
"Transfer learning leverages pre-trained models to accelerate new task learning.",
|
||||
"Data preprocessing steps include normalization, tokenization, and feature extraction.",
|
||||
"Hyperparameter tuning is essential for optimizing model performance and generalization.",
|
||||
"The attention mechanism allows models to focus on relevant input sequence parts.",
|
||||
"Batch normalization stabilizes training by normalizing activations within mini-batches.",
|
||||
"Dropout regularization helps prevent overfitting in deep neural network architectures.",
|
||||
"The validation set monitors model performance and guides hyperparameter selection.",
|
||||
"Cross-entropy loss measures the difference between predicted and actual probability distributions.",
|
||||
"Stochastic gradient descent updates model parameters using randomly sampled mini-batches.",
|
||||
].join("\n");
|
||||
|
||||
function makeBody(content: string) {
|
||||
return {
|
||||
messages: [
|
||||
{ role: "user", content },
|
||||
],
|
||||
};
|
||||
}
|
||||
|
||||
test("targetTokens: cuts body to ≤ targetTokens", () => {
|
||||
const body = makeBody(PROSE);
|
||||
const proseTokens = countTextTokens(PROSE);
|
||||
assert.ok(proseTokens > 200, `Fixture too small: ${proseTokens} tokens`);
|
||||
|
||||
const result = applyHardBudget(body, { targetTokens: 200 });
|
||||
assert.ok(result.compressed, "should be compressed");
|
||||
|
||||
const msgs = result.body.messages as Array<{ content: string }>;
|
||||
const outTokens = countTextTokens(msgs.map((m) => m.content).join(" "));
|
||||
assert.ok(
|
||||
outTokens <= 200,
|
||||
`Output tokens ${outTokens} exceed targetTokens 200`
|
||||
);
|
||||
});
|
||||
|
||||
test("targetTokens: preserves highest-saliency sentences", () => {
|
||||
const body = makeBody(PROSE);
|
||||
const result = applyHardBudget(body, { targetTokens: 200 });
|
||||
const msgs = result.body.messages as Array<{ content: string }>;
|
||||
const out = msgs.map((m) => m.content).join("\n");
|
||||
// The error line must be preserved (FORCE_PRESERVE_RE: "Error:")
|
||||
assert.ok(out.includes("Error:"), "Error: line must be preserved");
|
||||
});
|
||||
|
||||
test("targetRatio:0.5 halves the token count", () => {
|
||||
const body = makeBody(PROSE);
|
||||
const proseTokens = countTextTokens(PROSE);
|
||||
|
||||
const result = applyHardBudget(body, { targetRatio: 0.5 });
|
||||
const msgs = result.body.messages as Array<{ content: string }>;
|
||||
const outTokens = countTextTokens(msgs.map((m) => m.content).join(" "));
|
||||
|
||||
assert.ok(result.compressed, "should be compressed");
|
||||
assert.ok(
|
||||
outTokens <= Math.ceil(proseTokens * 0.5),
|
||||
`Output tokens ${outTokens} exceed ratio target ${Math.ceil(proseTokens * 0.5)}`
|
||||
);
|
||||
});
|
||||
|
||||
test("no target → byte-identical no-op", () => {
|
||||
const body = makeBody(PROSE);
|
||||
const result = applyHardBudget(body, {});
|
||||
assert.equal(result.compressed, false);
|
||||
assert.equal(JSON.stringify(result.body), JSON.stringify(body));
|
||||
assert.equal(result.stats, null);
|
||||
});
|
||||
|
||||
test("already-under-target → no-op", () => {
|
||||
const short = "Hello world.";
|
||||
const body = makeBody(short);
|
||||
const result = applyHardBudget(body, { targetTokens: 1000 });
|
||||
assert.equal(result.compressed, false);
|
||||
assert.equal(JSON.stringify(result.body), JSON.stringify(body));
|
||||
});
|
||||
|
||||
test("never drops a line matching FORCE_PRESERVE_RE (Error:, number, https://)", () => {
|
||||
const sensitive = [
|
||||
"Error: critical failure occurred",
|
||||
"The quick brown fox jumps over the lazy dog and runs.",
|
||||
"Amount: 99999",
|
||||
"Irrelevant filler text here.",
|
||||
"Another line of boring unimportant low-signal content.",
|
||||
"https://example.com/api/endpoint",
|
||||
"More filler words to pad the token count here.",
|
||||
"And yet more words that are clearly not important at all.",
|
||||
].join("\n");
|
||||
|
||||
const body = makeBody(sensitive);
|
||||
const totalTokens = countTextTokens(sensitive);
|
||||
// Force aggressive cut to half
|
||||
const result = applyHardBudget(body, { targetTokens: Math.floor(totalTokens * 0.4) });
|
||||
|
||||
const out = (result.body.messages as Array<{ content: string }>).map((m) => m.content).join("\n");
|
||||
assert.ok(out.includes("Error:"), "Error: line must survive");
|
||||
assert.ok(out.includes("99999"), "Number line must survive");
|
||||
assert.ok(out.includes("https://"), "URL line must survive");
|
||||
});
|
||||
|
||||
test("determinism: same input always produces same output", () => {
|
||||
const body = makeBody(PROSE);
|
||||
const r1 = applyHardBudget(body, { targetTokens: 200 });
|
||||
const r2 = applyHardBudget(body, { targetTokens: 200 });
|
||||
assert.equal(JSON.stringify(r1.body), JSON.stringify(r2.body));
|
||||
});
|
||||
|
||||
test("targetTokens wins when both targetTokens and targetRatio are set", () => {
|
||||
const body = makeBody(PROSE);
|
||||
const proseTokens = countTextTokens(PROSE);
|
||||
|
||||
// targetTokens=200 vs targetRatio=0.9 → targetTokens (200) should win
|
||||
const result = applyHardBudget(body, { targetTokens: 200, targetRatio: 0.9 });
|
||||
const msgs = result.body.messages as Array<{ content: string }>;
|
||||
const outTokens = countTextTokens(msgs.map((m) => m.content).join(" "));
|
||||
// If ratio=0.9 won, output would be ~90% of original; targetTokens=200 is more aggressive
|
||||
const ratioTarget = Math.ceil(proseTokens * 0.9);
|
||||
assert.ok(
|
||||
outTokens <= 200,
|
||||
`targetTokens should win: outTokens=${outTokens} should be ≤200, not ≤${ratioTarget}`
|
||||
);
|
||||
});
|
||||
|
||||
test("techniquesUsed includes hard-budget", () => {
|
||||
const body = makeBody(PROSE);
|
||||
const result = applyHardBudget(body, { targetTokens: 200 });
|
||||
assert.ok(result.stats !== null, "stats should be present");
|
||||
assert.ok(
|
||||
result.stats!.techniquesUsed.includes("hard-budget"),
|
||||
`techniquesUsed should include 'hard-budget', got: ${result.stats!.techniquesUsed}`
|
||||
);
|
||||
});
|
||||
|
||||
// --- Review fix #1: digit-less sensitive lines must be preserved ---
|
||||
|
||||
// Filler made of very common words → low saliency → first to be dropped
|
||||
// unless a unit is explicitly preserve-guarded. This forces the regex to
|
||||
// be the thing that protects the sensitive line (not its saliency score).
|
||||
const LOW_SIGNAL_FILLER = [
|
||||
"the and the for and the with the and to the of the in the on the by the.",
|
||||
"to the for the and the of the with the and the in the on the and the by.",
|
||||
"and the of the to the for the with the in the on the and the by the the.",
|
||||
"for the and the to the of the with the and the on the in the by the the.",
|
||||
"of the to the and the for the with the on the and the in the by the the.",
|
||||
"in the and the to the for the of the with the by the and the on the the.",
|
||||
];
|
||||
|
||||
// Target=1 forces EVERY droppable unit out; only preserve-guarded units can
|
||||
// survive. This makes the regex (not the saliency score) the thing under test.
|
||||
test("review#1: never drops a stack-trace at-frame line (digit-less)", () => {
|
||||
const lines = ["at processTicksAndRemainders foo bar baz qux", ...LOW_SIGNAL_FILLER].join("\n");
|
||||
const body = makeBody(lines);
|
||||
const result = applyHardBudget(body, { targetTokens: 1 });
|
||||
const out = (result.body.messages as Array<{ content: string }>).map((m) => m.content).join("\n");
|
||||
assert.ok(out.includes("at processTicksAndRemainders"), "at-frame must survive target=1");
|
||||
});
|
||||
|
||||
test("review#1: never drops a KEY=value credential line (digit-less)", () => {
|
||||
const lines = ["SECRET_KEY=abc the the the the the", ...LOW_SIGNAL_FILLER].join("\n");
|
||||
const body = makeBody(lines);
|
||||
const result = applyHardBudget(body, { targetTokens: 1 });
|
||||
const out = (result.body.messages as Array<{ content: string }>).map((m) => m.content).join("\n");
|
||||
assert.ok(out.includes("SECRET_KEY=abc"), "KEY=value must survive target=1");
|
||||
});
|
||||
|
||||
test("review#1: never drops a multi-slash path line (digit-less)", () => {
|
||||
const lines = ["/usr/local/lib/foo the the the the the", ...LOW_SIGNAL_FILLER].join("\n");
|
||||
const body = makeBody(lines);
|
||||
const result = applyHardBudget(body, { targetTokens: 1 });
|
||||
const out = (result.body.messages as Array<{ content: string }>).map((m) => m.content).join("\n");
|
||||
assert.ok(out.includes("/usr/local/lib/foo"), "multi-slash path must survive target=1");
|
||||
});
|
||||
|
||||
test("review#1: plain prose ending in a period is still droppable", () => {
|
||||
// Regression guard for the deviation: end-of-sentence period must NOT match.
|
||||
const proseLine = "This is plain prose here.";
|
||||
assert.equal(
|
||||
/\d|https?:\/\/|(?:Error|Exception|TypeError|RangeError|SyntaxError|ReferenceError|Traceback):|```|^\s*at\s|\/[\w.-]+\/|[A-Za-z_]\w*=\S/i.test(
|
||||
proseLine
|
||||
),
|
||||
false,
|
||||
"plain prose with a trailing period must remain droppable"
|
||||
);
|
||||
});
|
||||
|
||||
// --- Review fix #2: aggregate target distributed across messages ---
|
||||
|
||||
test("review#2: aggregate target keeps TOTAL ≤ target across multiple messages", () => {
|
||||
// 4 messages of distinct, splittable droppable prose (multi-line so each
|
||||
// message can be cut independently). Pre-fix, the full target was passed to
|
||||
// EACH message, letting the total come back ~N× over budget.
|
||||
const block = (tag: string) =>
|
||||
[
|
||||
`The ${tag} alpha system processes large amounts of information efficiently every day.`,
|
||||
`The ${tag} beta module requires substantial computational resources for routine training.`,
|
||||
`The ${tag} gamma layer enables programs to understand structured language semantically here.`,
|
||||
`The ${tag} delta network consists of multiple interconnected processing components today.`,
|
||||
`The ${tag} epsilon routine accelerates new task learning through transfer of prior knowledge.`,
|
||||
`The ${tag} zeta pipeline stabilizes throughput by normalizing activations within batches.`,
|
||||
].join("\n");
|
||||
const body = {
|
||||
messages: [
|
||||
{ role: "user", content: block("one") },
|
||||
{ role: "assistant", content: block("two") },
|
||||
{ role: "user", content: block("three") },
|
||||
{ role: "assistant", content: block("four") },
|
||||
],
|
||||
};
|
||||
const target = 200;
|
||||
const totalBefore = (body.messages as Array<{ content: string }>).reduce(
|
||||
(s, m) => s + countTextTokens(m.content),
|
||||
0
|
||||
);
|
||||
assert.ok(totalBefore > target, `Fixture too small: ${totalBefore} tokens`);
|
||||
|
||||
const result = applyHardBudget(body, { targetTokens: target });
|
||||
const msgs = result.body.messages as Array<{ content: string }>;
|
||||
const total = msgs.reduce((s, m) => s + countTextTokens(m.content), 0);
|
||||
assert.ok(result.compressed, "should be compressed");
|
||||
assert.ok(
|
||||
total <= target,
|
||||
`aggregate TOTAL ${total} must stay ≤ target ${target}`
|
||||
);
|
||||
});
|
||||
|
||||
// --- Review fix #3: signal a warning when target is impossible ---
|
||||
|
||||
test("review#3: warns when preserved content exceeds budget", () => {
|
||||
// Every line is preserve-guarded (numbers), so the target cannot be reached.
|
||||
const lines = [
|
||||
"Value 11111 is here",
|
||||
"Value 22222 is here",
|
||||
"Value 33333 is here",
|
||||
"Value 44444 is here",
|
||||
"Value 55555 is here",
|
||||
"Value 66666 is here",
|
||||
].join("\n");
|
||||
const body = makeBody(lines);
|
||||
const totalTokens = countTextTokens(lines);
|
||||
const result = applyHardBudget(body, { targetTokens: Math.floor(totalTokens * 0.3) });
|
||||
assert.ok(result.stats !== null, "stats should be present");
|
||||
const warnings = result.stats!.validationWarnings ?? [];
|
||||
assert.ok(
|
||||
warnings.some((w) => w.includes("hard-budget") && w.includes("could not reach target")),
|
||||
`expected a hard-budget warning, got: ${JSON.stringify(warnings)}`
|
||||
);
|
||||
});
|
||||
|
||||
// --- Review fix #4: targetTokens:0 must NOT silently skip ---
|
||||
|
||||
test("review#4: targetTokens:0 attempts compression (not a silent no-op)", () => {
|
||||
const body = makeBody(PROSE);
|
||||
const result = applyHardBudget(body, { targetTokens: 0 });
|
||||
// Target 0 is impossible to fully reach (preserved lines remain), but the
|
||||
// post-pass MUST engage: either it cut something, or it emitted a warning.
|
||||
const warnings = result.stats?.validationWarnings ?? [];
|
||||
assert.ok(
|
||||
result.compressed || warnings.length > 0,
|
||||
"targetTokens:0 must engage the post-pass, not silently skip"
|
||||
);
|
||||
});
|
||||
|
||||
test("review#4: seam runs hard-budget post-pass when config.targetTokens is 0 (falsy)", () => {
|
||||
// The seam gate must use != null, not ||, or targetTokens:0 silently skips.
|
||||
const body = makeBody(PROSE);
|
||||
const config = {
|
||||
enabled: true,
|
||||
defaultMode: "stacked" as const,
|
||||
autoTriggerMode: "lite" as const,
|
||||
autoTriggerTokens: 0,
|
||||
cacheMinutes: 5,
|
||||
preserveSystemPrompt: true,
|
||||
comboOverrides: {},
|
||||
compressionComboId: null,
|
||||
stackedPipeline: [{ engine: "caveman" as const, intensity: "lite" as const }],
|
||||
engines: {},
|
||||
activeComboId: null,
|
||||
targetTokens: 0,
|
||||
};
|
||||
const result = applyStackedCompression(body, config.stackedPipeline, { config });
|
||||
const techniques = result.stats?.techniquesUsed ?? [];
|
||||
assert.ok(
|
||||
techniques.includes("hard-budget"),
|
||||
`seam must run hard-budget for targetTokens:0, got techniques: ${techniques}`
|
||||
);
|
||||
});
|
||||
|
||||
test("integration: applyStackedCompression with config.targetTokens cuts at end of pipeline", () => {
|
||||
const body = makeBody(PROSE);
|
||||
const proseTokens = countTextTokens(PROSE);
|
||||
assert.ok(proseTokens > 150, `Fixture too small: ${proseTokens} tokens`);
|
||||
|
||||
const config = {
|
||||
enabled: true,
|
||||
defaultMode: "stacked" as const,
|
||||
autoTriggerMode: "lite" as const,
|
||||
autoTriggerTokens: 0,
|
||||
cacheMinutes: 5,
|
||||
preserveSystemPrompt: true,
|
||||
comboOverrides: {},
|
||||
compressionComboId: null,
|
||||
stackedPipeline: [{ engine: "caveman" as const, intensity: "lite" as const }],
|
||||
engines: {},
|
||||
activeComboId: null,
|
||||
targetTokens: 150,
|
||||
};
|
||||
|
||||
const result = applyStackedCompression(body, config.stackedPipeline, { config });
|
||||
const msgs = result.body.messages as Array<{ content: string }>;
|
||||
const outTokens = countTextTokens(msgs.map((m) => m.content).join(" "));
|
||||
assert.ok(
|
||||
outTokens <= 150,
|
||||
`Integration: output tokens ${outTokens} exceed targetTokens 150`
|
||||
);
|
||||
assert.ok(result.compressed, "Integration: result should be compressed");
|
||||
|
||||
const techniques = result.stats?.techniquesUsed ?? [];
|
||||
assert.ok(
|
||||
techniques.includes("hard-budget"),
|
||||
`Integration: techniquesUsed should include 'hard-budget', got: ${techniques}`
|
||||
);
|
||||
});
|
||||
Reference in New Issue
Block a user