Files
OmniRoute/src/lib/db/compression.ts
Diego Rodrigues de Sa e Souza cadc3f10b7 Release v3.8.35 (#4743)
* chore(release): open v3.8.35 development cycle

* fix db vacuum scheduler settings (#4726)

Scheduled VACUUM now follows Storage page settings (scheduledVacuum/vacuumHour) as single source of truth; env-flag control path removed. 11/11 vacuum-scheduler tests pass against release/v3.8.35 tip; no orphaned env refs. Integrated into release/v3.8.35.

* fix(tier): noAuth providers count as free; free filter returns empty … (#4753)

noAuth providers now classified free (union of legacy list + NOAUTH_PROVIDERS chat-tier derivation), -free arena_elo alias, and auto/<cat>:free returns an empty pool when no free candidate matches (opt-in legacy fallback via OMNIROUTE_AUTO_FREE_FALLBACK_TO_FULL_POOL). New env var documented in .env.example + ENVIRONMENT.md; CHANGELOG bullet added (maintainer co-author). 46/46 node + 56/56 vitest tests pass on release tip; env-doc-sync, docs-sync, typecheck:core, lint, file-size all green. Integrated into release/v3.8.35.

* refactor(chatCore): extrai 11 helpers de nível superior para 6 leaves puros (#3501) (#4571)

chatCore god-file decomposition (#3501): extract 6 pure leaves (cacheUsageMeta, executorClientHeaders, nonStreamingResponseBody, skillsFormat, streamErrorResult, streamFinalize) from chatCore.ts. Rebased onto release/v3.8.35 tip (resolved single chatCore.ts conflict — removed now-extracted inline buildExecutorClientHeaders). 265/265 chatcore tests, 26/26 new leaf tests, typecheck:core, cycles, file-size all green. Integrated into release/v3.8.35.

* refactor(chatCore): extrai resolveExecutorWithProxy + getExecutionCredentials para leaves (#3501) (#4646)

chatCore #3501: extract resolveExecutorWithProxy + getExecutionCredentials to leaves (executorProxy.ts, executionCredentials.ts). Clean cherry-pick onto release tip post-#4571. 12/12 new leaf tests, typecheck:core, cycles, file-size green. Integrated into release/v3.8.35.

* refactor(chatCore): extrai transforms de mensagens Claude p/ leaf (#3501) (#4708)

chatCore #3501: extract Claude upstream-message transforms to leaf (claudeUpstreamMessages.ts + claudeMessageTypes.ts). Clean cherry-pick post-#4646. 8/8 new leaf tests, typecheck/cycles/file-size green. Integrated into release/v3.8.35.

* refactor(chatCore): extrai persistAttemptLogs para leaf (#3501) (#4717)

chatCore #3501: extract persistAttemptLogs to leaf (attemptLogging.ts). Rebased onto release tip post-#4708 (resolved imports conflict: kept tip's resolveCompressionHeader from compression Phase 3, dropped now-unused logTruncation import moved into the leaf). 288/288 chatcore tests, typecheck/cycles/file-size green. Integrated into release/v3.8.35.

* refactor(chatCore): extrai stageTrace + compressionUsageReceipt para leaves (#3501) (#4721)

chatCore #3501: extract stageTrace + compressionUsageReceipt to leaves. Clean cherry-pick post-#4717. 6/6 new leaf tests, typecheck/cycles/file-size green. Integrated into release/v3.8.35.

* refactor(chatCore): extrai prepareUpstreamBody (1ª sub-fatia do executeProviderRequest, #3501) (#4730)

chatCore #3501: extract prepareUpstreamBody (first sub-slice of executeProviderRequest) to leaf (upstreamBody.ts). Clean cherry-pick post-#4721. 7/7 new leaf tests, full 301/301 chatcore suite, typecheck/cycles/file-size green. Completes the 6-PR chatCore decomposition stack into release/v3.8.35.

* fix(db): make db-backup import size cap configurable (#4719) (#4757)

Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>

* chore(quality): expand check:release-green to the FULL release-PR gate set (#4758)

The release-green pre-flight (Solution C) previously covered only a subset of the
gates that run exclusively on the release PR (PR→main), so reds still accrued
silently on release/** and surfaced in ~40-min layers at release time (v3.8.34:
3 CI rounds — CodeQL sanitization, then the fail-fast Quality Ratchet revealing
openapi then cyclomatic-complexity one push at a time, plus zizmor/integration).

Now check:release-green reproduces the COMPLETE release-PR gate set and reports
EVERY red in one pass (collected, not fail-fast):

- New DRIFT ratchets (report-only, rebaselined at release, never block):
  cyclomatic complexity, dead-code, type-coverage, compression-budget,
  openapi-coverage, workflow-lint (zizmor), codeql-ratchet.
- New HARD gates (real defects): docs-all (fabricated-docs strict + i18n mirror
  sync) and the integration test suite (gated behind !--quick).

The only release-PR gates it still cannot reproduce locally are GitHub-side CodeQL
semantic analysis and SonarQube/SonarCloud (external services).

The nightly-release-green workflow and /green-prs inherit the expanded coverage
automatically (they invoke this script), so cycle drift is now surfaced
continuously and the release PR is green on its first CI run.

Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>

* fix(dashboard): add missing onboarding.tiers step title (#4698) (#4755)

Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>

* feat(compression): Output Styles registry + D0 telemetry (Phase 4A) (#4694)

Phase 4A: Output Styles registry + D0 telemetry. Integrated into release/v3.8.35.

* feat(compression): SLM tier for ultra (Phase 4B) [stacked on #4694] (#4707)

Phase 4B: SLM tier for ultra. Integrated into release/v3.8.35.

* feat(compression): context-budget adaptive compression (Phase 4C) [stacked on #4707] (#4716)

Phase 4C: adaptive context-budget compression. Integrated into release/v3.8.35.

* feat(compression): offline evaluation harness (Phase 4 D1) [stacked on #4716] (#4720)

Phase 4 D1: offline evaluation harness. Integrated into release/v3.8.35.

* fix(sse): deepseek-web folds role:tool results into prompt transcript (#4712) (#4756)

Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>

* fix(dashboard): remove dead unconditional useLiveRequests call in HomePageClient (#4759, #4745, #4596) (#4761)

Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>

* fix(dashboard): dedupe provider nodes by id on compatible-provider add (#4746) (#4768)

Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>

* chore(db): re-export compressionRunTelemetry from localDb to satisfy db-rules (#4775)

Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>

* docs(security): add canonical STRIDE-based threat model (#4783)

Canonical STRIDE threat model. Integrated into release/v3.8.35.

* test(dashboard): add smoke test for home client dashboard (#4793)

Smoke test guarding the dashboard home client render (regression #4745/#4759). Code fix already landed via #4761; this PR's jsdom smoke test is the net-new regression guard. Integrated into release/v3.8.35.

* fix(combos): auto-promote zeroLatencyOptimizationsEnabled so legacy configs (pre-3.8.33 fallbackCompressionMode="lite") round-trip on the first GUI edit (#4774)

Auto-promote zeroLatencyOptimizationsEnabled + strip v3.8.31-era removed keys so legacy combo configs round-trip through PUT /api/combos/{id} on first GUI edit (closes #4382 followup). Pre-merge: rewrote the now-stale reject test to assert auto-promotion + added passthrough/round-trip regression guards; reconciled combos/page.tsx file-size baseline. Integrated into release/v3.8.35.

* refactor(chatCore): extrai parse + usage-stats não-streaming do executeProviderRequest (#3501) (#4762)

chatCore #3501: extract parseNonStreamingResponseBody + recordNonStreamingUsageStats. Integrated into release/v3.8.35.

* refactor(chatCore): extrai recordContextEditingTelemetryHook (#3501) (#4779)

chatCore #3501: extract recordContextEditingTelemetryHook. Integrated into release/v3.8.35.

* refactor(chatCore): extrai recordCompressionCacheStats (#3501) (#4792)

chatCore #3501: extract recordCompressionCacheStats. Integrated into release/v3.8.35.

* refactor(chatCore): extrai writeCavemanOutputAnalytics (#3501) (#4794)

chatCore #3501: extract writeCavemanOutputAnalytics. Integrated into release/v3.8.35.

* refactor(chatCore): extrai scheduleQuotaShareConsumption (POST-hook não-streaming, #3501) (#4780)

chatCore #3501: extract scheduleQuotaShareConsumption (non-streaming POST-hook). Integrated into release/v3.8.35.

* refactor(chatCore): extrai emitRequestGamificationEvent (helper compartilhado DRY, #3501) (#4776)

chatCore #3501: extract emitRequestGamificationEvent (DRY streaming/non-streaming). Integrated into release/v3.8.35.

* refactor(chatCore): extrai runPluginOnResponseHook (#3501) (#4782)

chatCore #3501: extract runPluginOnResponseHook. Integrated into release/v3.8.35.

* refactor(chatCore): extrai scheduleStreamingQuotaShareConsumption (POST-hook streaming, #3501) (#4784)

chatCore #3501: extract scheduleStreamingQuotaShareConsumption (streaming POST-hook). Integrated into release/v3.8.35.

* refactor(chatCore): extrai recordStreamingUsageStats (analytics de usage streaming, #3501) (#4791)

chatCore #3501: extract recordStreamingUsageStats. Integrated into release/v3.8.35.

* refactor(chatCore): extrai recordStreamingCost (custo por-request streaming, #3501) (#4790)

chatCore #3501: extract recordStreamingCost (per-request streaming cost). Integrated into release/v3.8.35.

* docs(readme): credit ponytail + OmniCompress; restore env-doc-sync release-green (#4799)

README compression credits (ponytail/OmniCompress) + env-doc-sync ignore for eval-only OMNIROUTE_EVAL_CREDENTIALS (restores release-green after #4720). Integrated into release/v3.8.35.

* chore(quality): trim combo-config.test.ts comments under file-size cap (#4774 follow-up) (#4800)

Restore file-size release-green. Integrated into release/v3.8.35.

* feat(api-docs): Redoc-rendered /api/docs + consolidate OpenAPI spec to docs/openapi.yaml (#4781)

Redoc /api/docs + OpenAPI spec consolidated to docs/openapi.yaml (canonical 201-path complete spec; old path → legacy fallback). All refs/gates/tests/CI updated. Integrated into release/v3.8.35.

* docs(compression): declare Phase 4 layers — Output Styles, adaptive dial, per-request control (#4801)

The README compression section listed the 9 input engines but not the Phase 4
layers now in production:
- Output Styles (output-axis steering: terse-prose / less-code / terse-cjk, lite/full/ultra)
- adaptive context-budget dial (reserve-output|percentage|absolute · floor|replace-autotrigger|off)
- per-request x-omniroute-compression precedence + the offline eval harness
Also bumped the highlights range to v3.8.35, expanded the compression feature bullet,
and marked the GUIDE's Phase 4 row Shipped (was 'Planned' — it's merged on v3.8.35).

Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore(release): finalize v3.8.35 CHANGELOG + docs reconciliation

- CHANGELOG: complete 3.8.35 section (all 35 commits since v3.8.34,
  contributor attribution: @rdself @megamen32 @KooshaPari @JxnLexn)
- docs(security): align THREAT_MODEL.md refs with real code
  (routeGuard.ts, tokenLimits.ts, /api/monitoring/health) — fabricated-docs gate
- check:fabricated-docs: skip docs/superpowers/specs (dated research reports)
- i18n: sync 3.8.35 section into 41 CHANGELOG mirrors (docs-sync size gate)
- ratchet rebaseline: cyclomatic 1916->1920, eslintWarnings 3907->3912
  (inherited cycle drift; release-finalize diff is docs-only)

* fix(release): resolve inherited base-reds surfaced by v3.8.35 release CI

Cycle base-reds that only run on PR→main (not the PR→release fast-path):

- test(autoCombo): suffixComposition-4517 used node:test in a vitest-only dir
  (#4753) → vitest found no suite. Switch to the vitest API. (Vitest job)
- test(agentSkills): openapiParser fixture wrote docs/reference/openapi.yaml;
  parser reads docs/openapi.yaml since #4781 → point fixture at the new path.
  (Unit/Coverage/Node24/Node26 shard 4)
- test(integration): proxy-pipeline source-scan expected inline streaming-cost
  code that #4790/#3501 extracted to the recordStreamingCost leaf → assert the
  delegation instead. (Integration 1/2)
- fix(chatCore): derive the log trace id from crypto, not Math.random
  (CodeQL js/insecure-randomness — log-correlation id, not a secret).
- test(resilience): circuit-breaker invalid-cooldown fallback asserted t>29000,
  flaking on slow CI where ~1.6s elapsed gave t=28401 → tolerate wall-clock
  drift (t>25000). (Unit 6/8)

* fix(usage): derive pending-request id from crypto, not Math.random

CodeQL js/insecure-randomness (#669): the pending-request id generated in
trackPendingRequest (usageHistory.ts) flows into attempt logging and was flagged
as insecure randomness in a security context. It's a log-correlation id, not a
secret — switch to crypto RNG to clear the alert. Pairs with the chatCore traceId
fix in 37c49781a (same sink).

---------

Co-authored-by: Diego Rodrigues de Sa e Souza <souzamiriamrodrigues790@gmail.com>
Co-authored-by: Randi <55005611+rdself@users.noreply.github.com>
Co-authored-by: Demiurge The Single <megamen932@gmail.com>
Co-authored-by: KooshaPari <42529354+KooshaPari@users.noreply.github.com>
Co-authored-by: Jan Leon <Jan.gaschler@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 17:06:18 -03:00

753 lines
27 KiB
TypeScript

import { backupDbFile } from "./backup";
import { getDefaultCompressionCombo } from "./compressionCombos";
import { getDbInstance } from "./core";
import { invalidateDbCache } from "./readCache";
import {
ENGINE_IDS,
DEFAULT_AGGRESSIVE_CONFIG,
DEFAULT_CAVEMAN_CONFIG,
DEFAULT_CAVEMAN_OUTPUT_MODE_CONFIG,
DEFAULT_COMPRESSION_LANGUAGE_CONFIG,
DEFAULT_COMPRESSION_CONFIG,
DEFAULT_CONTEXT_EDITING_CONFIG,
DEFAULT_MCP_ACCESSIBILITY_CONFIG,
DEFAULT_RTK_CONFIG,
DEFAULT_ULTRA_CONFIG,
clampMcpAccessibilityConfig,
type AggressiveConfig,
type CavemanConfig,
type CavemanOutputModeConfig,
type OutputStyleSelectionEntry,
type CompressionLanguageConfig,
type CompressionPipelineStep,
type CompressionConfig,
type CompressionMode,
type ContextEditingConfig,
type EngineToggle,
type McpAccessibilityConfig,
type RtkConfig,
type UltraConfig,
} from "@omniroute/open-sse/services/compression/types.ts";
import { maybePrewarmUltraSlmOnConfig } from "@omniroute/open-sse/services/compression/ultra.ts";
const NAMESPACE = "compression";
const COMPRESSION_MODES = new Set<CompressionMode>([
"off",
"lite",
"standard",
"aggressive",
"ultra",
"rtk",
"stacked",
]);
type JsonRecord = Record<string, unknown>;
// TTL cache for compression settings (5s)
let compressionSettingsCache: {
value: CompressionConfig;
expiresAt: number;
dbRef: WeakRef<object>;
} | null = null;
// Phase 4 (B): one cold-start SLM pre-warm attempt per process. The save path fires
// on every enable transition; this guard keeps the read path from re-warming on every
// cache miss (the read path runs at most once per 5s, but a cold start should warm once,
// not repeatedly). Best-effort either way (`maybePrewarmUltraSlmOnConfig` never throws).
let _ultraSlmColdPrewarmAttempted = false;
function toRecord(value: unknown): JsonRecord {
return value && typeof value === "object" ? (value as JsonRecord) : {};
}
function parseJsonSafe(raw: string | null): unknown {
if (!raw) return undefined;
try {
return JSON.parse(raw);
} catch {
return undefined;
}
}
function normalizeCavemanConfig(value: unknown): CavemanConfig {
const record = toRecord(value);
const intensity =
record.intensity === "lite" || record.intensity === "full" || record.intensity === "ultra"
? record.intensity
: DEFAULT_CAVEMAN_CONFIG.intensity;
return {
...DEFAULT_CAVEMAN_CONFIG,
...record,
compressRoles: Array.isArray(record.compressRoles)
? record.compressRoles.filter(
(role): role is "user" | "assistant" | "system" =>
role === "user" || role === "assistant" || role === "system"
)
: DEFAULT_CAVEMAN_CONFIG.compressRoles,
skipRules: Array.isArray(record.skipRules)
? record.skipRules.filter((rule): rule is string => typeof rule === "string")
: DEFAULT_CAVEMAN_CONFIG.skipRules,
minMessageLength:
typeof record.minMessageLength === "number" && Number.isFinite(record.minMessageLength)
? Math.max(0, Math.floor(record.minMessageLength))
: DEFAULT_CAVEMAN_CONFIG.minMessageLength,
preservePatterns: Array.isArray(record.preservePatterns)
? record.preservePatterns.filter((pattern): pattern is string => typeof pattern === "string")
: DEFAULT_CAVEMAN_CONFIG.preservePatterns,
intensity,
};
}
function normalizeCavemanOutputModeConfig(value: unknown): CavemanOutputModeConfig {
const record = toRecord(value);
return {
...DEFAULT_CAVEMAN_OUTPUT_MODE_CONFIG,
enabled:
typeof record.enabled === "boolean"
? record.enabled
: DEFAULT_CAVEMAN_OUTPUT_MODE_CONFIG.enabled,
intensity:
record.intensity === "lite" || record.intensity === "full" || record.intensity === "ultra"
? record.intensity
: DEFAULT_CAVEMAN_OUTPUT_MODE_CONFIG.intensity,
autoClarity:
typeof record.autoClarity === "boolean"
? record.autoClarity
: DEFAULT_CAVEMAN_OUTPUT_MODE_CONFIG.autoClarity,
};
}
function normalizeOutputStyleSelection(value: unknown): OutputStyleSelectionEntry[] {
if (!Array.isArray(value)) return [];
const out: OutputStyleSelectionEntry[] = [];
for (const raw of value) {
const record = toRecord(raw);
const id = typeof record.id === "string" ? record.id.trim() : "";
const level =
record.level === "lite" || record.level === "full" || record.level === "ultra"
? record.level
: null;
if (id && level) out.push({ id, level });
}
return out;
}
function normalizeRtkConfig(value: unknown): RtkConfig {
const record = toRecord(value);
return {
...DEFAULT_RTK_CONFIG,
enabled: typeof record.enabled === "boolean" ? record.enabled : DEFAULT_RTK_CONFIG.enabled,
intensity:
record.intensity === "minimal" ||
record.intensity === "standard" ||
record.intensity === "aggressive"
? record.intensity
: DEFAULT_RTK_CONFIG.intensity,
applyToToolResults:
typeof record.applyToToolResults === "boolean"
? record.applyToToolResults
: DEFAULT_RTK_CONFIG.applyToToolResults,
applyToCodeBlocks:
typeof record.applyToCodeBlocks === "boolean"
? record.applyToCodeBlocks
: DEFAULT_RTK_CONFIG.applyToCodeBlocks,
applyToAssistantMessages:
typeof record.applyToAssistantMessages === "boolean"
? record.applyToAssistantMessages
: DEFAULT_RTK_CONFIG.applyToAssistantMessages,
enabledFilters: Array.isArray(record.enabledFilters)
? record.enabledFilters.filter((filter): filter is string => typeof filter === "string")
: DEFAULT_RTK_CONFIG.enabledFilters,
disabledFilters: Array.isArray(record.disabledFilters)
? record.disabledFilters.filter((filter): filter is string => typeof filter === "string")
: DEFAULT_RTK_CONFIG.disabledFilters,
maxLinesPerResult: boundedInt(
record.maxLinesPerResult,
DEFAULT_RTK_CONFIG.maxLinesPerResult,
0,
100000
),
maxCharsPerResult: boundedInt(
record.maxCharsPerResult,
DEFAULT_RTK_CONFIG.maxCharsPerResult,
0,
1000000
),
deduplicateThreshold: boundedInt(
record.deduplicateThreshold,
DEFAULT_RTK_CONFIG.deduplicateThreshold,
2,
100
),
customFiltersEnabled:
typeof record.customFiltersEnabled === "boolean"
? record.customFiltersEnabled
: DEFAULT_RTK_CONFIG.customFiltersEnabled,
trustProjectFilters:
typeof record.trustProjectFilters === "boolean"
? record.trustProjectFilters
: DEFAULT_RTK_CONFIG.trustProjectFilters,
rawOutputRetention:
record.rawOutputRetention === "never" ||
record.rawOutputRetention === "failures" ||
record.rawOutputRetention === "always"
? record.rawOutputRetention
: DEFAULT_RTK_CONFIG.rawOutputRetention,
rawOutputMaxBytes: boundedInt(
record.rawOutputMaxBytes,
DEFAULT_RTK_CONFIG.rawOutputMaxBytes,
1024,
10_000_000
),
enableGrouping:
typeof record.enableGrouping === "boolean"
? record.enableGrouping
: (DEFAULT_RTK_CONFIG.enableGrouping ?? false),
groupingThreshold: boundedInt(
record.groupingThreshold,
DEFAULT_RTK_CONFIG.groupingThreshold ?? 3,
2,
100
),
stripCodeComments:
typeof record.stripCodeComments === "boolean"
? record.stripCodeComments
: (DEFAULT_RTK_CONFIG.stripCodeComments ?? false),
preserveDocstrings:
typeof record.preserveDocstrings === "boolean"
? record.preserveDocstrings
: (DEFAULT_RTK_CONFIG.preserveDocstrings ?? true),
};
}
function normalizeLanguageConfig(value: unknown): CompressionLanguageConfig {
const record = toRecord(value);
const defaultLanguage =
typeof record.defaultLanguage === "string" && record.defaultLanguage.trim()
? record.defaultLanguage.trim()
: DEFAULT_COMPRESSION_LANGUAGE_CONFIG.defaultLanguage;
const enabledPacks = Array.isArray(record.enabledPacks)
? record.enabledPacks
.filter((pack): pack is string => typeof pack === "string" && pack.trim().length > 0)
.map((pack) => pack.trim())
: DEFAULT_COMPRESSION_LANGUAGE_CONFIG.enabledPacks;
return {
...DEFAULT_COMPRESSION_LANGUAGE_CONFIG,
enabled:
typeof record.enabled === "boolean"
? record.enabled
: DEFAULT_COMPRESSION_LANGUAGE_CONFIG.enabled,
defaultLanguage,
autoDetect:
typeof record.autoDetect === "boolean"
? record.autoDetect
: DEFAULT_COMPRESSION_LANGUAGE_CONFIG.autoDetect,
enabledPacks: [...new Set(enabledPacks.length > 0 ? enabledPacks : ["en"])],
};
}
function normalizeContextEditingConfig(value: unknown): ContextEditingConfig {
const record = toRecord(value);
return {
...DEFAULT_CONTEXT_EDITING_CONFIG,
enabled:
typeof record.enabled === "boolean" ? record.enabled : DEFAULT_CONTEXT_EDITING_CONFIG.enabled,
};
}
// Engines allowed in the global stackedPipeline setting. MUST stay in sync with the
// compression-combo KNOWN_ENGINE_IDS (src/lib/db/compressionCombos.ts) — otherwise the
// global setting silently strips engines the combo path accepts (B-PIPELINE-DIVERGENCE).
const STACKED_PIPELINE_ENGINE_IDS = new Set([
"lite",
"caveman",
"aggressive",
"ultra",
"rtk",
"headroom",
"session-dedup",
"ccr",
"llmlingua",
]);
export function normalizeStackedPipeline(value: unknown): CompressionPipelineStep[] {
const source = Array.isArray(value) ? value : (DEFAULT_COMPRESSION_CONFIG.stackedPipeline ?? []);
const pipeline: CompressionPipelineStep[] = [];
for (const entry of source) {
const record = toRecord(entry);
const engine = record.engine;
if (typeof engine !== "string" || !STACKED_PIPELINE_ENGINE_IDS.has(engine)) {
continue;
}
pipeline.push({
engine: engine as CompressionPipelineStep["engine"],
...(typeof record.intensity === "string"
? { intensity: record.intensity as CompressionPipelineStep["intensity"] }
: {}),
...(record.config && typeof record.config === "object"
? { config: record.config as Record<string, unknown> }
: {}),
});
}
return pipeline.length > 0 ? pipeline : (DEFAULT_COMPRESSION_CONFIG.stackedPipeline ?? []);
}
function boundedInt(value: unknown, fallback: number, min: number, max: number): number {
if (typeof value !== "number" || !Number.isFinite(value)) return fallback;
return Math.min(max, Math.max(min, Math.floor(value)));
}
function boundedNumber(value: unknown, fallback: number, min: number, max: number): number {
if (typeof value !== "number" || !Number.isFinite(value)) return fallback;
return Math.min(max, Math.max(min, value));
}
function normalizeAggressiveConfig(value: unknown): AggressiveConfig {
const record = toRecord(value);
const thresholds = toRecord(record.thresholds);
const toolStrategies = toRecord(record.toolStrategies);
return {
...DEFAULT_AGGRESSIVE_CONFIG,
thresholds: {
fullSummary: boundedInt(
thresholds.fullSummary,
DEFAULT_AGGRESSIVE_CONFIG.thresholds.fullSummary,
1,
100
),
moderate: boundedInt(
thresholds.moderate,
DEFAULT_AGGRESSIVE_CONFIG.thresholds.moderate,
1,
100
),
light: boundedInt(thresholds.light, DEFAULT_AGGRESSIVE_CONFIG.thresholds.light, 1, 100),
verbatim: boundedInt(
thresholds.verbatim,
DEFAULT_AGGRESSIVE_CONFIG.thresholds.verbatim,
1,
100
),
},
toolStrategies: {
fileContent:
typeof toolStrategies.fileContent === "boolean"
? toolStrategies.fileContent
: DEFAULT_AGGRESSIVE_CONFIG.toolStrategies.fileContent,
grepSearch:
typeof toolStrategies.grepSearch === "boolean"
? toolStrategies.grepSearch
: DEFAULT_AGGRESSIVE_CONFIG.toolStrategies.grepSearch,
shellOutput:
typeof toolStrategies.shellOutput === "boolean"
? toolStrategies.shellOutput
: DEFAULT_AGGRESSIVE_CONFIG.toolStrategies.shellOutput,
json:
typeof toolStrategies.json === "boolean"
? toolStrategies.json
: DEFAULT_AGGRESSIVE_CONFIG.toolStrategies.json,
errorMessage:
typeof toolStrategies.errorMessage === "boolean"
? toolStrategies.errorMessage
: DEFAULT_AGGRESSIVE_CONFIG.toolStrategies.errorMessage,
},
summarizerEnabled:
typeof record.summarizerEnabled === "boolean"
? record.summarizerEnabled
: DEFAULT_AGGRESSIVE_CONFIG.summarizerEnabled,
maxTokensPerMessage: boundedInt(
record.maxTokensPerMessage,
DEFAULT_AGGRESSIVE_CONFIG.maxTokensPerMessage,
256,
32768
),
minSavingsThreshold: boundedNumber(
record.minSavingsThreshold,
DEFAULT_AGGRESSIVE_CONFIG.minSavingsThreshold,
0,
1
),
};
}
function normalizeUltraConfig(value: unknown): UltraConfig {
const record = toRecord(value);
const modelPath = typeof record.modelPath === "string" ? record.modelPath.trim() : "";
return {
...DEFAULT_ULTRA_CONFIG,
enabled: typeof record.enabled === "boolean" ? record.enabled : DEFAULT_ULTRA_CONFIG.enabled,
compressionRate: boundedNumber(
record.compressionRate,
DEFAULT_ULTRA_CONFIG.compressionRate,
0,
1
),
minScoreThreshold: boundedNumber(
record.minScoreThreshold,
DEFAULT_ULTRA_CONFIG.minScoreThreshold,
0,
1
),
slmFallbackToAggressive:
typeof record.slmFallbackToAggressive === "boolean"
? record.slmFallbackToAggressive
: DEFAULT_ULTRA_CONFIG.slmFallbackToAggressive,
...(modelPath ? { modelPath } : {}),
maxTokensPerMessage: boundedInt(
record.maxTokensPerMessage,
DEFAULT_ULTRA_CONFIG.maxTokensPerMessage,
0,
32768
),
};
}
// Single-mode → engine id mapping. Mirrors deriveDefaultPlan's SINGLE_MODE_OF: a legacy
// install whose only signal is `defaultMode` should turn on the engine that mode runs, so the
// derived engines map matches the old behavior. Keep conservative — these are the only modes
// that map 1:1 to a single engine.
const SINGLE_MODE_ENGINE: Partial<Record<CompressionMode, string>> = {
lite: "lite",
standard: "caveman",
aggressive: "aggressive",
ultra: "ultra",
rtk: "rtk",
};
function normalizeEngineToggle(value: unknown): EngineToggle | null {
const record = toRecord(value);
if (typeof record.enabled !== "boolean") return null;
return {
enabled: record.enabled,
...(typeof record.level === "string" ? { level: record.level } : {}),
};
}
// Sanitize an engines map for persistence: keep only known engine ids with a well-formed
// `{enabled, level?}` toggle. Mirrors the read-path validation so a malformed write can't poison
// the stored row.
function sanitizeEnginesForWrite(value: unknown): Record<string, EngineToggle> {
const record = toRecord(value);
const out: Record<string, EngineToggle> = {};
for (const id of ENGINE_IDS) {
const toggle = normalizeEngineToggle(record[id]);
if (toggle) out[id] = toggle;
}
return out;
}
// Read the stored `engines` JSON row, keeping only well-formed `{enabled, level?}` entries for
// known engine ids. Returns null when no usable row exists so the caller falls back to deriving
// the map from the legacy fields (B-backfill, migration 102).
function parseStoredEnginesMap(value: unknown): Record<string, EngineToggle> | null {
if (!value || typeof value !== "object") return null;
const out: Record<string, EngineToggle> = {};
let any = false;
for (const id of ENGINE_IDS) {
const toggle = normalizeEngineToggle((value as JsonRecord)[id]);
if (toggle) {
out[id] = toggle;
any = true;
}
}
return any ? out : null;
}
// Derive the per-engine toggle map from the legacy compression fields so existing installs keep
// their behavior before they ever write an `engines` row. Single-engine modes (caveman/rtk/ultra/
// aggressive) come from their dedicated config blocks; structural engines (lite/headroom/
// session-dedup/ccr/llmlingua) come from the default-combo pipeline. `defaultMode` is a last-resort
// signal that turns on its single-mode engine when nothing else already did.
function deriveEnginesMap(config: CompressionConfig): Record<string, EngineToggle> {
let defaultComboEngines = new Set<string>();
try {
const combo = getDefaultCompressionCombo();
if (combo) {
defaultComboEngines = new Set(combo.pipeline.map((step) => step.engine));
}
} catch {
defaultComboEngines = new Set<string>();
}
const engines: Record<string, EngineToggle> = {};
for (const id of ENGINE_IDS) {
let enabled = false;
let level: string | undefined;
switch (id) {
case "caveman":
enabled = config.cavemanConfig?.enabled === true;
if (typeof config.cavemanConfig?.intensity === "string") {
level = config.cavemanConfig.intensity;
}
break;
case "rtk":
enabled = config.rtkConfig?.enabled === true;
if (typeof config.rtkConfig?.intensity === "string") {
level = config.rtkConfig.intensity;
}
break;
case "ultra":
enabled = config.ultra?.enabled === true;
break;
case "aggressive":
enabled = aggressiveEnabled(config.aggressive);
break;
default:
// Structural engines (lite/headroom/session-dedup/ccr/llmlingua): on when present in the
// default-combo pipeline.
enabled = defaultComboEngines.has(id);
break;
}
engines[id] = { enabled, ...(level !== undefined ? { level } : {}) };
}
// Last-resort defaultMode signal: if the legacy install only set defaultMode (no engine config),
// turn on the engine that mode actually ran so the derived default matches the old behavior.
const fallbackEngine = SINGLE_MODE_ENGINE[config.defaultMode];
if (fallbackEngine && engines[fallbackEngine] && !engines[fallbackEngine].enabled) {
engines[fallbackEngine] = { ...engines[fallbackEngine], enabled: true };
}
return engines;
}
// `aggressive` config doesn't carry a top-level `enabled` flag in its type, but legacy installs may
// have stored one. Read it defensively for the derived engines map.
function aggressiveEnabled(value: AggressiveConfig | undefined): boolean {
return toRecord(value).enabled === true;
}
export async function getCompressionSettings(): Promise<CompressionConfig> {
const db = getDbInstance();
if (
compressionSettingsCache &&
Date.now() < compressionSettingsCache.expiresAt &&
compressionSettingsCache.dbRef.deref() === db
) {
return compressionSettingsCache.value;
}
compressionSettingsCache = null;
const rows = db.prepare("SELECT key, value FROM key_value WHERE namespace = ?").all(NAMESPACE);
const config: CompressionConfig = {
...DEFAULT_COMPRESSION_CONFIG,
cavemanConfig: { ...DEFAULT_CAVEMAN_CONFIG },
cavemanOutputMode: { ...DEFAULT_CAVEMAN_OUTPUT_MODE_CONFIG },
outputStyles: [],
rtkConfig: { ...DEFAULT_RTK_CONFIG },
languageConfig: { ...DEFAULT_COMPRESSION_LANGUAGE_CONFIG },
stackedPipeline: normalizeStackedPipeline(undefined),
aggressive: normalizeAggressiveConfig(undefined),
ultra: normalizeUltraConfig(undefined),
contextEditing: { ...DEFAULT_CONTEXT_EDITING_CONFIG },
engines: {},
activeComboId: null,
};
// Tracks whether a usable stored `engines` row was found. When absent (pre-migration-102 install)
// we derive the engines map from the legacy fields below so behavior is preserved.
let storedEngines: Record<string, EngineToggle> | null = null;
for (const row of rows) {
const record = toRecord(row);
const key = typeof record.key === "string" ? record.key : null;
const rawValue = typeof record.value === "string" ? record.value : null;
if (!key || rawValue === null) continue;
const parsed = parseJsonSafe(rawValue);
if (parsed === undefined) continue;
switch (key) {
case "enabled":
config.enabled = parsed === true;
break;
case "defaultMode":
if (typeof parsed === "string" && COMPRESSION_MODES.has(parsed as CompressionMode)) {
config.defaultMode = parsed as CompressionMode;
}
break;
case "autoTriggerMode":
if (typeof parsed === "string" && COMPRESSION_MODES.has(parsed as CompressionMode)) {
config.autoTriggerMode = parsed as CompressionMode;
}
break;
case "autoTriggerTokens":
config.autoTriggerTokens =
typeof parsed === "number" && Number.isFinite(parsed)
? Math.max(0, Math.floor(parsed))
: 0;
break;
case "cacheMinutes":
config.cacheMinutes =
typeof parsed === "number" && Number.isFinite(parsed)
? Math.max(1, Math.floor(parsed))
: DEFAULT_COMPRESSION_CONFIG.cacheMinutes;
break;
case "preserveSystemPrompt":
config.preserveSystemPrompt = parsed !== false;
break;
case "mcpDescriptionCompressionEnabled":
config.mcpDescriptionCompressionEnabled = parsed !== false;
break;
case "comboOverrides":
if (parsed && typeof parsed === "object") {
const overrides: Record<string, CompressionMode> = {};
for (const [comboId, mode] of Object.entries(parsed as Record<string, unknown>)) {
if (typeof mode === "string" && COMPRESSION_MODES.has(mode as CompressionMode)) {
overrides[comboId] = mode as CompressionMode;
}
}
config.comboOverrides = overrides;
}
break;
case "compressionComboId":
config.compressionComboId =
typeof parsed === "string" && parsed.trim() ? parsed.trim() : null;
break;
case "stackedPipeline":
config.stackedPipeline = normalizeStackedPipeline(parsed);
break;
case "cavemanConfig":
config.cavemanConfig = normalizeCavemanConfig(parsed);
break;
case "cavemanOutputMode":
config.cavemanOutputMode = normalizeCavemanOutputModeConfig(parsed);
break;
case "outputStyles":
config.outputStyles = normalizeOutputStyleSelection(parsed);
break;
case "rtkConfig":
config.rtkConfig = normalizeRtkConfig(parsed);
break;
case "languageConfig":
config.languageConfig = normalizeLanguageConfig(parsed);
break;
case "aggressive":
case "aggressiveConfig":
config.aggressive = normalizeAggressiveConfig(parsed);
break;
case "ultra":
case "ultraConfig":
config.ultra = normalizeUltraConfig(parsed);
break;
case "contextEditing":
config.contextEditing = normalizeContextEditingConfig(parsed);
break;
case "engines":
storedEngines = parseStoredEnginesMap(parsed);
break;
case "activeComboId":
config.activeComboId =
typeof parsed === "string" && parsed.trim() ? parsed.trim() : null;
break;
case "ultraEngine":
// Phase 4 (B): SLM tier selector. Only the two known values; anything else
// falls back to the heuristic default so a malformed row can never enable SLM.
config.ultraEngine = parsed === "slm" ? "slm" : "heuristic";
break;
case "ultraSlmPrewarm":
config.ultraSlmPrewarm = parsed === true;
break;
}
}
// Engines map: prefer the stored row; otherwise derive from the legacy fields (migration 102
// backfill on the read path). Always fill EVERY id in ENGINE_IDS so the shape matches
// DEFAULT_COMPRESSION_CONFIG.
const derived = storedEngines ?? deriveEnginesMap(config);
const engines: Record<string, EngineToggle> = {};
for (const id of ENGINE_IDS) {
engines[id] = derived[id] ?? { enabled: false };
}
config.engines = engines;
// Runtime-only marker: dispatch trusts the engines map only when it was explicitly stored
// (panel-saved). A backfilled map (no stored row) is display-only — dispatch stays on the
// legacy defaultMode/default-combo path so existing installs keep their behaviour.
config.enginesExplicit = storedEngines !== null;
// Store in TTL cache (5s expiry)
compressionSettingsCache = {
value: config,
expiresAt: Date.now() + 5000,
dbRef: new WeakRef(db),
};
// Phase 4 (B): cold-restart pre-warm — when the stored config already selects the SLM
// tier with pre-warm on, warm the model once (best-effort, fire-and-forget, guarded so
// a frequently-hit read path warms at most once per process). Cache hits return above.
if (!_ultraSlmColdPrewarmAttempted) {
_ultraSlmColdPrewarmAttempted = true;
void maybePrewarmUltraSlmOnConfig({
ultraEngine: config.ultraEngine,
ultraSlmPrewarm: config.ultraSlmPrewarm,
});
}
return config;
}
export async function updateCompressionSettings(
updates: Partial<CompressionConfig>
): Promise<CompressionConfig> {
const db = getDbInstance();
const insert = db.prepare(
"INSERT OR REPLACE INTO key_value (namespace, key, value) VALUES (?, ?, ?)"
);
const tx = db.transaction(() => {
for (const [key, value] of Object.entries(updates)) {
if (value === undefined) continue;
// Persist the engines map as ONE sanitized JSON row so the read path always gets
// well-formed { enabled, level? } toggles for known engine ids.
if (key === "engines") {
insert.run(NAMESPACE, key, JSON.stringify(sanitizeEnginesForWrite(value)));
continue;
}
insert.run(NAMESPACE, key, JSON.stringify(value));
}
});
tx();
backupDbFile("pre-write");
compressionSettingsCache = null;
invalidateDbCache();
const next = await getCompressionSettings();
// Phase 4 (B): the SAVE path covers the enable transition — if this write turns the
// SLM tier + pre-warm on, warm the model once (best-effort, fire-and-forget).
void maybePrewarmUltraSlmOnConfig({
ultraEngine: next.ultraEngine,
ultraSlmPrewarm: next.ultraSlmPrewarm,
});
return next;
}
function normalizeMcpAccessibilityConfig(value: unknown): McpAccessibilityConfig {
// clampMcpAccessibilityConfig (engine layer) owns the numeric floors so the DB normalizer and
// the live MCP-server read path agree — in particular it floors maxTextChars to a sane minimum
// (a value below the tail reservation would make smartFilterText truncate the whole text away).
return clampMcpAccessibilityConfig(value);
}
export async function getMcpAccessibilityConfig(): Promise<McpAccessibilityConfig> {
const db = getDbInstance();
const row = db
.prepare("SELECT value FROM key_value WHERE namespace = ? AND key = ?")
.get(NAMESPACE, "mcpAccessibility") as { value: string } | undefined;
return normalizeMcpAccessibilityConfig(parseJsonSafe(row?.value ?? null));
}
export async function setMcpAccessibilityConfig(
value: Partial<McpAccessibilityConfig>
): Promise<void> {
const next = normalizeMcpAccessibilityConfig({ ...DEFAULT_MCP_ACCESSIBILITY_CONFIG, ...value });
const db = getDbInstance();
db.prepare("INSERT OR REPLACE INTO key_value (namespace, key, value) VALUES (?, ?, ?)").run(
NAMESPACE,
"mcpAccessibility",
JSON.stringify(next)
);
compressionSettingsCache = null;
invalidateDbCache();
}