mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-07-31 04:12:10 +03:00
feat(compression): update vendored GCF (Headroom) codec to spec v3.2 — nested flattening (#6838)
* feat(compression): update vendored GCF (Headroom) codec to spec v3.2 (nested flattening)
Homogeneous arrays whose rows carry nested objects/arrays now tabularize
via GCF v3.2 `>`-path flattening instead of a low-yield per-row fallback,
so nested MCP tool-result rows (meta:{...}, tags:[...]) compact like flat
rows. Round-trip stays lossless (order-insensitive deepEqual).
Re-vendored from current gcf-typescript into the Headroom generic-profile
codec (open-sse/services/compression/engines/headroom/gcf/); still zero
runtime deps, MIT, SPDX-marked, generic-profile only. Also folds in two
upstream round-trip-safety fixes: the [N]: inline-array quoting fix and
canonical decimal formatting.
Regression guard: tests/unit/compression/headroom-smartcrusher.test.ts
gains a deep-nested case (two-level object + array-of-objects) asserting
the v3.2 flatten paths and order-insensitive round-trip. Vendored-code
baseline bumps (complexity 2053->2055, cognitive 885->888, decode_generic
no-explicit-any 18->22) each carry an inline _rebaseline_2026_07_10_gcf_v3_2
justification noting the growth is the vendored surface, not new project code.
* chore(changelog): add fragment for headroom GCF v3.2 nested flattening (#6838)
* docs(readme): note Headroom handles nested arrays (GCF v3.2) in the engine-stack table
* docs(readme): note Headroom handles nested arrays (GCF v3.2) in the engine-stack table
* fix(compression): harden vendored GCF decoder against prototype pollution
The v3.2 flatten/unflatten paths (and the pre-existing inline-object parser)
built decoded objects with bracket assignment and `key in obj` membership,
so a hostile or unusual payload could pollute Object.prototype via a
`__proto__` path segment, and any key shadowing an Object.prototype member
(`toString`, `constructor`) was misparsed or wrongly flagged duplicate.
- Encoder (`analyzeFlattenable`): builds the shape map with `Object.create(null)`
and refuses to flatten objects carrying `__proto__`/`constructor`/`prototype`
keys (they round-trip whole instead).
- Decoder: `unflattenPaths` drops any path with an unsafe segment; a shared
`safeAssign` writes a literal `__proto__` key as an own data property
(JSON.parse semantics) instead of reassigning the prototype, used at every
object-build site; `checkDup` and orphan-merge use `hasOwnProperty` so
built-in-named keys are not spuriously treated as duplicates.
Also a losslessness fix: objects with keys named `toString`/`constructor`/
`valueOf` now round-trip. Regression guard: prototype-pollution + built-in-key
cases in tests/unit/compression/headroom-smartcrusher.test.ts. Prototype
pollution is JS/TS-specific; the Go/Python/Rust/Swift/Kotlin SDKs use native
maps and are unaffected.
* fix(compression): apply GCF decoder review hardening (hasOwnProperty sweep, unflatten null-guard, strict count)
Addresses the second-round review on the vendored codec:
- Replace every `key in obj` membership test with
`Object.prototype.hasOwnProperty.call(...)` across generic.ts (flatten
shape analysis, key-chain resolution, inline-schema/shared-array helpers,
row encode) so inherited names (`toString`/`constructor`) never match the
prototype chain, and remove a redundant `obj` re-declaration in the ">"
field attachment loop.
- `unflattenPaths` guards each intermediate segment: a missing OR non-object
slot is replaced with a fresh object before traversal, so malformed/hostile
input can no longer dereference a primitive and crash.
- Use the strict `parseCount` helper (not `parseInt`) for the shared-schema
count so malformed counts fail the mismatch check instead of coercing.
The decoder grew past the 800-line file-size cap; frozen at 880 in
file-size-baseline.json with a justification (vendored file kept faithful to
upstream gcf-typescript for clean re-vendoring). Verified: prototype-pollution
+ hostile-input + built-in-key round-trip probes, 54/54 compression tests,
typecheck, lint, cyclomatic/cognitive baselines unchanged, compression-budget.
* fix(compression): do not flatten a nested object that is null in any row (losslessness)
analyzeFlattenable skipped null values during shape analysis, so a field that
was an object in some rows and null in others was still flattened. On decode,
the null row's leaves resolved as absent ("~") and unflattened to a missing key
instead of null, silently dropping the value (e.g. {meta:{owner:null}} decoded
to {}). analyzeFlattenable now bails (returns null) when the field is null in
any row, routing it through the lossless whole-object attachment path. Applies
at every nesting depth via the existing recursion. Regression guard: null
nested-object cases in tests/unit/compression/headroom-smartcrusher.test.ts.
* fix(compression): narrow the null-nested flatten bail to intermediate nulls only
The previous fix bailed flattening whenever a nested field was null in any row.
That is correct but over-broad: a top-level null round-trips losslessly through
flattening (it emits "-" and reconstructs via the all-null rule). Only a null at
an intermediate nesting level loses data (its leaves encode as absent "~" and
unflatten to a missing key). Bail only when parentPath is non-empty, so top-level
nulls keep flattening (compression preserved) while intermediate nulls fall back
to the lossless attachment path. Matches GCF conformance fixtures 004/013.
---------
Co-authored-by: Diego Rodrigues de Sa e Souza <8016841+diegosouzapw@users.noreply.github.com>
This commit is contained in:
@@ -574,7 +574,7 @@ Engines run in pipeline order; each is independently toggleable and configurable
|
||||
| 1 | **Session-Dedup** | Drops content repeated across turns (content-addressed, cross-turn) |
|
||||
| 2 | **CCR** | Archives large blocks behind retrieve markers, fetched on demand |
|
||||
| 3 | **RTK** | Smart tool-result filtering, dedup & truncation (command-aware) |
|
||||
| 4 | **Headroom** | Lossless tabular compaction of homogeneous JSON arrays (~30%+) |
|
||||
| 4 | **Headroom** | Lossless tabular compaction of homogeneous JSON arrays, flat or nested (~30%), via a vendored **GCF** codec (spec v3.2) |
|
||||
| 5 | **Relevance** | Extractive sentence scoring against the last user query |
|
||||
| 6 | **Caveman** | Rule-based prose compression (~65–75% on output) |
|
||||
| 7 | **LLMLingua-2** | ML semantic pruning via MobileBERT ONNX — code-safe, async |
|
||||
@@ -1216,7 +1216,7 @@ OmniRoute stands on the shoulders of giants. It started as a fork of **[9router]
|
||||
| Project | ⭐ | How it inspired OmniRoute |
|
||||
| ---------------------------------------------------------------------------------------------- | ----: | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **[TOON](https://github.com/toon-format/toon)** · toon-format | 24.7k | Token-Oriented Object Notation — its columnar, header-plus-rows model shaped our tabular compaction stage. |
|
||||
| **[GCF – Graph Compact Format](https://github.com/blackwell-systems/gcf)** · Blackwell Systems | 14 | Schema-aware "JSON for LLMs" notation — co-inspired our lossless homogeneous-array compaction with `[N rows]` markers. |
|
||||
| **[GCF – Graph Compact Format](https://github.com/blackwell-systems/gcf)** · Blackwell Systems | 14 | First inspired our tabular compaction stage; now its zero-dependency, lossless generic-profile encoder is **vendored directly** as the Headroom codec (MIT, SPDX-marked), current with GCF spec v3.2. |
|
||||
| **[token-optimizer-mcp](https://github.com/ooples/token-optimizer-mcp)** · ooples | 421 | Brotli/SQLite cache + per-session context-delta — inspired our `session-dedup` engine. |
|
||||
| **[token-savior](https://github.com/Mibayy/token-savior)** · Mibayy | 1.0k | Bash-output compaction + MCP profiles — inspired our compression bail-out discipline and MCP tool-manifest reduction. |
|
||||
| **[token-saver](https://github.com/ppgranger/token-saver)** · ppgranger | 110 | Content-aware, per-file-type output compression with failure-aware bail-out — validated our per-type dispatch and minimum-gain skip. |
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
- **feat(compression):** update the vendored **GCF** codec behind the **Headroom** engine to spec **v3.2 (nested flattening)** ([#6837](https://github.com/diegosouzapw/OmniRoute/issues/6837)). Homogeneous arrays whose rows carry nested objects/arrays now tabularize via `>`-prefixed path fields instead of a low-yield per-row fallback, so nested MCP tool-result rows (`meta:{...}`, `tags:[...]`) compact like flat rows. On representative shapes the update takes deeply-nested payloads the old codec left near-uncompressed from ~3% to ~32% vs JSON (k8s pods, `cl100k_base`), with shallow-nested rows seeing a small bump and flat arrays unchanged. Re-vendored from current gcf-typescript (zero runtime deps, MIT, SPDX-marked, generic-profile only); also folds in the `[N]:` inline-array quoting fix and canonical decimal formatting. Round-trip stays lossless (order-insensitive), and the decoder is hardened against prototype pollution (a `__proto__`/`constructor` path segment never mutates `Object.prototype`, and keys shadowing built-ins like `toString` now round-trip correctly instead of misparsing). Regression guard: `tests/unit/compression/headroom-smartcrusher.test.ts` (deep-nested + prototype-pollution cases).
|
||||
@@ -1,7 +1,8 @@
|
||||
{
|
||||
"_comment": "Catraca de complexidade (check-complexity.mjs, ESLint core rules complexity>=15 e max-lines-per-function>80 sobre src+open-sse+electron+bin via eslint.complexity.config.mjs). Conta total de violacoes; so pode cair. --update ratcheta.",
|
||||
"count": 2054,
|
||||
"count": 2056,
|
||||
"_rebaseline_2026_07_10_v3847_merge_burst": "2053->2054 (+1). Drift herdado do merge burst do dia em release/v3.8.47 (campanha /implement-prs: ~36 PRs mergeados — órfãos, features do dono, ports). O check:complexity NÃO roda no fast-path PR->release, então o ramo acumulou o +1 sem rebaselinar (mesma família de todos os rebaselines abaixo). Trust-but-verify: medido 2054 no tip da release pós-burst; a única função flagada nova é pré-existente (getResolvedModelCapabilities em modelCapabilities.ts, já >teto antes de #6714). Nenhum PR órfão/feature introduz violação NOVA — os fixes deste ciclo são complexity-net-zero. Rebaseline aprovado pelo dono (2026-07-10) para destravar o FQG dos ~7 órfãos verdes-exceto-complexity. Tighten via --update next cycle.",
|
||||
"_rebaseline_2026_07_10_gcf_v3_2": "2054->2056 (+2). PR feat/headroom-gcf-v3.2-nested-flattening: own growth from re-vendoring the GCF (Headroom) codec to spec v3.2 (nested flattening). The 2 new over-threshold functions are the v3.2 `>`-path flatten/unflatten walk in the vendored generic-profile encode/decode paths (open-sse/services/compression/engines/headroom/gcf/{generic,decode_generic}.ts). This is imported third-party code kept byte-faithful to upstream gcf-typescript, not extractable without diverging from the vendored source; local measures 2055 on the merged tree; frozen at 2056 = the base's CI-observed 2054 + this PR's 2 new functions, matching the documented local-vs-CI off-by-one convention (see _rebaseline_2026_07_02_v3844_ci_observed) so the GitHub runner stays green. Round-trip guarded by tests/unit/compression/headroom-smartcrusher.test.ts (deep-nested case). Structural shrink belongs upstream in gcf, not here.",
|
||||
"_rebaseline_2026_07_08_6556_inherited_drift": "2052->2053 (+1). PR #6556 (omniglyph engine): drift herdado do merge burst da base (a catraca nao roda no fast-path PR->release, mesmo padrao dos rebaselines v3.8.44/46). Trust-but-verify: o proprio codigo do PR e complexity-net-zero — as 2 violacoes que ele introduzia (runCompressionAsync complexity 17 apos o branch do modo omniglyph; OmniglyphContextPageClient 161 linhas) foram CORRIGIDAS por extracao real (engines/omniglyphSingleMode.ts + split do componente em section components), medido: 2055->2053 local; base pura origin/release/v3.8.47 mede 2053 identico. Tighten via --update next cycle.",
|
||||
"_rebaseline_2026_07_07_v3846_release_close": "2035->2050 (+15). v3.8.46 release close (generate-release Phase 0 pre-flight): drift herdado do merge burst do ciclo (39 commits do dia + campanha /review-*). Trust-but-verify: os fixes de base-red do captain (agentSkills path.resolve #6366, catalogo cache #6408, tipagem de teste no-explicit-any, MitmProxyTab suppression) sao complexity-net-zero — check:complexity mede 2050 identico com e sem os fixes (a catraca NAO roda no fast-path PR->release, entao o ramo acumulou sem rebaselinar). Tighten via --update next cycle.",
|
||||
"_rebaseline_2026_07_04_v3844_release_close": "2026->2028 (+2). v3.8.44 release close (generate-release Phase 0/1): drift residual do fim do ciclo medido no tip pos-#6155 (merge burst final: #6155 cooling-panel + #6104 Kenari + #6139/#6128 provider-limits). Trust-but-verify: os 2 fixes de codigo do release-captain (model.ts alias boundary, auggie.ts stdin error handlers) adicionam 0 violacoes NOVAS — eslint.complexity direto nos 2 arquivos flagra apenas funcoes que ja estouravam o limite antes (runStreaming/start ja >80 linhas; resolveModelByProviderInference/getModelInfoCore pre-existentes de #5918), e resolveProviderAlias segue abaixo de 15. Logo o +2 e drift herdado do burst. Tighten via --update next cycle.",
|
||||
|
||||
@@ -131,7 +131,7 @@
|
||||
},
|
||||
"open-sse/services/compression/engines/headroom/gcf/decode_generic.ts": {
|
||||
"@typescript-eslint/no-explicit-any": {
|
||||
"count": 18
|
||||
"count": 22
|
||||
}
|
||||
},
|
||||
"open-sse/services/compression/engines/headroom/gcf/scalar.ts": {
|
||||
|
||||
@@ -180,6 +180,8 @@
|
||||
"_rebaseline_2026_06_28_5243_risk_gate_prepass": "PR #5243 (compression risk-gate pre-pass) own growth: open-sse/services/compression/strategySelector.ts 854->899 (+45). The three exported entry points (applyCompression/applyStackedCompression/applyStackedCompressionAsync) become thin wrappers over pure-extracted private bodies (runCompression/runStackedCompression/runStackedCompressionAsync) so the risk-gate mask->run->restore wrapper sits strictly OUTSIDE the per-step loop \u2014 a single universal integration point. The wrapper logic itself (resolveRiskGate/withRiskGate) lives in the new riskGate/strategyWrap.ts (<cap); the residual growth is the duplicated thin-wrapper signatures + the extracted bodies' dispatch boundary, guarded by a byte-identical parity test (riskGateIntegration). Default off (DEFAULT_COMPRESSION_CONFIG unchanged). Not extractable without hiding the dispatch boundary, mirroring prior compression rebaselines. Structural shrink tracked in #3501.",
|
||||
"_rebaseline_2026_06_29_5286_memoization": "PR #5286 own growth: strategySelector.ts 899->960 (+61 = the opt-in result-memoization branches in applyCompression/applyCompressionAsync \u2014 principal+determinism gate, makeMemoKey lookup/store with model+supportsVision folded into the key, recompute-with-memo-off). Default off (memoizeCompressionResults), so zero behavior change. The memo helpers live in the leaf resultMemo.ts (<cap); the chokepoint wiring here is not extractable. Structural shrink of this hot-path file tracked in #3501.",
|
||||
"open-sse/services/compression/strategySelector.ts": 1043,
|
||||
"_rebaseline_2026_07_10_gcf_v3_2_decode": "PR #6838 own growth: new vendored file open-sse/services/compression/engines/headroom/gcf/decode_generic.ts frozen at 880 (> 800 cap). It is the vendored GCF generic-profile decoder (spec v3.2 nested flattening plus the prototype-pollution / hasOwnProperty hardening added in this PR's Gemini review). Kept as one file faithful to upstream gcf-typescript so re-vendoring stays a clean copy rather than a re-split each cycle (sibling generic.ts/scalar.ts stay < cap; extraction would also fragment the file's frozen eslint no-explicit-any suppressions). Round-trip + prototype-pollution regression coverage in tests/unit/compression/headroom-smartcrusher.test.ts. Frozen: only shrinks from here.",
|
||||
"open-sse/services/compression/engines/headroom/gcf/decode_generic.ts": 880,
|
||||
"open-sse/services/rateLimitManager.ts": 1035,
|
||||
"_rebaseline_2026_06_29_4038_cas_guard": "PR (#4038) own growth: tokenRefresh.ts 2103->2181 (+78 = the compare-and-swap guard on the refresh persist \u2014 runWithCasGuard/getActiveCasGuard AsyncLocalStorage pair mirroring runWithOnPersist, casGuardShouldSkipPersist that rereads the row right before persisting and skips the write when a concurrent writer already rotated the refresh_token past the one presented, plus getCasGuardStats counters). Fixes the sibling-rotation-revert \u2192 token-family-revocation storm. Gated behind an active guard (opt-in; no guard => byte-identical). Wiring lives at the two persist chokepoints inside getAccessToken; the comparison reuses wasRefreshTokenRotated from refreshSerializer. Not extractable without splitting the refresh hot path.",
|
||||
"_rebaseline_2026_07_09_6126_clinepass_dual_auth": "PR #6126 (@hajilok, dual-auth ClinePass) own growth: tokenRefresh.ts 2181->2182 (+1 = a single `case \"clinepass\":` fallthrough label added to the existing `case \"cline\":` in _getAccessTokenInternal's provider switch, so clinepass token refresh dispatches to the already-shared refreshClineToken() instead of silently falling through to the generic OAuth refresh). Irreducible 1-line switch-case wiring at the existing chokepoint; the header-building logic for the same feature was extracted to a new leaf src/shared/utils/clineAuth.ts::buildClinepassHeaders() (well under cap) to avoid growing open-sse/executors/default.ts. Covered by tests/unit/clinepass-provider.test.ts.",
|
||||
|
||||
@@ -121,7 +121,8 @@
|
||||
},
|
||||
"cognitiveComplexity": {
|
||||
"value": 890,
|
||||
"_rebaseline_2026_07_12_v3847_mergetrain_burst": "885->890 (+5). v3.8.47 /merge-prs merge-train batch (23 merge-ready PRs) inherited drift: cognitive-complexity does NOT run on PR->release fast-gates, so incidental growth accrued unmeasured across the batch. Measured 890 on the combined merge-train tip (5d980352d) vs 885 on the pristine release tip 1b7a9150e. #6838 (headroom gcf codec re-vendor) accounts for +3 (its own baseline bump to 888); the remaining +2 is parallel-batch drift across the other 22 PRs. Owner-approved rebaseline (merge-burst reconciliation, same class as the v3.8.46/v3.8.44 notes below). Tighten via --update next cycle.",
|
||||
"_rebaseline_2026_07_10_gcf_v3_2": "885->888 (+3). PR feat/headroom-gcf-v3.2-nested-flattening: own growth from re-vendoring the GCF (Headroom) codec to spec v3.2 (nested flattening). The new over-threshold functions are the vendored v3.2 flatten/unflatten walk in open-sse/services/compression/engines/headroom/gcf/{generic,decode_generic}.ts. Imported third-party code kept byte-faithful to upstream gcf-typescript; measured 888 with the update vs 885 on the pristine origin/release/v3.8.47 tip. Guarded by tests/unit/compression/headroom-smartcrusher.test.ts (deep-nested case). Structural shrink belongs upstream in gcf.",
|
||||
"_rebaseline_2026_07_12_v3847_mergetrain_burst": "885->890 (+5). v3.8.47 /merge-prs merge-train batch (23 merge-ready PRs) inherited drift: cognitive-complexity does NOT run on PR->release fast-gates, so incidental growth accrued unmeasured across the batch. Measured 890 on the combined merge-train tip (5d980352d) vs 885 on the pristine release tip 1b7a9150e. #6838 (headroom gcf codec re-vendor) accounts for +3 (its own baseline bump to 888, superseded here by this later 890 rebaseline which already covers it); the remaining +2 is parallel-batch drift across the other 22 PRs. Owner-approved rebaseline (merge-burst reconciliation, same class as the v3.8.46/v3.8.44 notes below). Tighten via --update next cycle.",
|
||||
"_rebaseline_2026_07_09_6587_kiro_api_key_auth": "884->885 (+1). PR #6587 (@strangersp) own growth: open-sse/services/usage/kiro.ts gains ONE new over-threshold function — getKiroUsage grew from a single fetch to a 3-endpoint fallback chain (codewhisperer-get / codewhisperer-post / q-get) with per-attempt auth-header selection (tokentype: API_KEY vs Bearer-only), needed so usage/quota lookups work for the new long-lived-API-key auth path in addition to the existing OAuth path (measured: 0 violations on release tip -> 1 violation, complexity 33, at open-sse/services/usage/kiro.ts). Covered by tests/unit/kiro-iam-profilearn-usage.test.ts (tokentype header selection, friendly auth-expired/rejected-token messages). Cohesive multi-endpoint-fallback logic at an existing usage chokepoint; not extractable without splitting the fallback loop mid-merge. Structural shrink tracked in #3501.",
|
||||
"_rebaseline_2026_07_07_v3846_release_close": "877->882 (+5). v3.8.46 release close (generate-release Phase 0 pre-flight): drift herdado do merge burst do ciclo. Trust-but-verify: os fixes de base-red do captain (agentSkills path.resolve #6366, catalogo cache #6408, tipagem de teste, MitmProxyTab suppression) sao cognitive-net-zero — check:cognitive-complexity mede 882 identico com e sem os fixes (a catraca NAO roda no fast-path PR->release). Tighten via --update next cycle.",
|
||||
"_rebaseline_2026_07_03_v3844_ipfilter_release_green": "861->867 (+6). v3.8.44 cycle drift measured on release tip 32e4c906e during the #6131/#5975 release-green rebaseline. Inherited from the merge burst (Quality Ratchet does not run on PR->release fast-gates). route-edge-coverage +7 is my #5975 test comment; the rest is parallel-session drift. Tighten via --update next cycle.",
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
/**
|
||||
* Decode GCF generic profile text into a JS value.
|
||||
* Vendored from gcf-typescript — generic profile only (graph profile stripped).
|
||||
* GCF generic-profile decoder (decodeGeneric).
|
||||
* Vendored from gcf-typescript — generic profile only. Current with GCF spec v3.2
|
||||
* (nested object flattening) and the [N]: inline-array quoting fix.
|
||||
* https://github.com/blackwell-systems/gcf-typescript
|
||||
*
|
||||
* SPDX-License-Identifier: MIT
|
||||
@@ -10,12 +11,13 @@ import {
|
||||
parseQuotedString,
|
||||
splitRespectingQuotes,
|
||||
splitFieldDecl,
|
||||
isBareKey,
|
||||
MISSING,
|
||||
ATTACHMENT,
|
||||
} from "./scalar.ts";
|
||||
|
||||
/**
|
||||
* Decode GCF generic profile text into a JS value.
|
||||
* Decode GCF generic or graph profile text into a JS value.
|
||||
*/
|
||||
export function decodeGeneric(input: string): any {
|
||||
input = input.trimEnd();
|
||||
@@ -28,10 +30,13 @@ export function decodeGeneric(input: string): any {
|
||||
|
||||
const profile = parseHeaderProfile(header);
|
||||
|
||||
if (profile !== "generic")
|
||||
if (profile === "graph") {
|
||||
throw new Error(
|
||||
`unsupported_profile: ${profile} (only generic profile is supported in this vendored build)`
|
||||
"graph_profile_unsupported: this vendored build supports the generic profile only"
|
||||
);
|
||||
}
|
||||
|
||||
if (profile !== "generic") throw new Error(`unknown_profile: ${profile}`);
|
||||
|
||||
// Filter body.
|
||||
const contentLines: string[] = [];
|
||||
@@ -125,7 +130,7 @@ function parseObjectBody(
|
||||
const name = parseKeyFromHeader(hdr.slice(0, bi));
|
||||
checkDup(out, name);
|
||||
const [arr, consumed] = parseArrayFromHeader(lines, i, depth, hdr.slice(bi));
|
||||
out[name] = arr;
|
||||
safeAssign(out, name, arr);
|
||||
i += consumed;
|
||||
continue;
|
||||
}
|
||||
@@ -134,18 +139,27 @@ function parseObjectBody(
|
||||
i++;
|
||||
const nested: Record<string, any> = {};
|
||||
const consumed = parseObjectBody(lines, i, depth + 1, nested);
|
||||
out[name] = nested;
|
||||
safeAssign(out, name, nested);
|
||||
i += consumed;
|
||||
continue;
|
||||
}
|
||||
|
||||
// Inline array. The bracket must be in the KEY position: an inline-array header is
|
||||
// `name[...]: …`, never `key=value` whose value happens to contain `[..]:` (e.g. a
|
||||
// quoted `note="ERR[404]: Not Found"`). Guard on no `=` before the bracket so such
|
||||
// values fall through to the key=value branch instead of throwing (B-GCF-QUOTE).
|
||||
// Key=value. Check this BEFORE inline array detection so that bracket
|
||||
// patterns inside quoted values (e.g. text="ERR[404]: Not Found") are
|
||||
// not misinterpreted as inline array headers.
|
||||
const eqIdx = findKeyValueSplit(content);
|
||||
if (eqIdx > 0) {
|
||||
const name = parseKeyFromHeader(content.slice(0, eqIdx));
|
||||
checkDup(out, name);
|
||||
safeAssign(out, name, parseScalar(content.slice(eqIdx + 1), false));
|
||||
i++;
|
||||
continue;
|
||||
}
|
||||
|
||||
// Inline array (e.g. items[3]: a,b,c). Only reached if no = found.
|
||||
if (!content.startsWith("@") && !content.startsWith("##")) {
|
||||
const bracketIdx = content.indexOf("[");
|
||||
if (bracketIdx > 0 && !content.slice(0, bracketIdx).includes("=")) {
|
||||
if (bracketIdx > 0) {
|
||||
const rest = content.slice(bracketIdx);
|
||||
const closeIdx = rest.indexOf("]");
|
||||
if (closeIdx >= 0) {
|
||||
@@ -154,7 +168,7 @@ function parseObjectBody(
|
||||
const name = parseKeyFromHeader(content.slice(0, bracketIdx));
|
||||
checkDup(out, name);
|
||||
const [arr] = parseArrayFromHeader(lines, i, depth, rest);
|
||||
out[name] = arr;
|
||||
safeAssign(out, name, arr);
|
||||
i++;
|
||||
continue;
|
||||
}
|
||||
@@ -162,16 +176,6 @@ function parseObjectBody(
|
||||
}
|
||||
}
|
||||
|
||||
// Key=value.
|
||||
const eqIdx = findKeyValueSplit(content);
|
||||
if (eqIdx > 0) {
|
||||
const name = parseKeyFromHeader(content.slice(0, eqIdx));
|
||||
checkDup(out, name);
|
||||
out[name] = parseScalar(content.slice(eqIdx + 1), false);
|
||||
i++;
|
||||
continue;
|
||||
}
|
||||
|
||||
i++;
|
||||
}
|
||||
return i - start;
|
||||
@@ -179,6 +183,7 @@ function parseObjectBody(
|
||||
|
||||
function findKeyValueSplit(s: string): number {
|
||||
if (!s.length) return -1;
|
||||
// Quoted key: "key"=value
|
||||
if (s[0] === '"') {
|
||||
for (let i = 1; i < s.length; i++) {
|
||||
if (s[i] === "\\") {
|
||||
@@ -189,7 +194,12 @@ function findKeyValueSplit(s: string): number {
|
||||
}
|
||||
return -1;
|
||||
}
|
||||
return s.indexOf("=");
|
||||
// Bare key: find = but only before [ (to avoid matching = inside inline array values)
|
||||
const eqIdx = s.indexOf("=");
|
||||
if (eqIdx < 0) return -1;
|
||||
const bracketIdx = s.indexOf("[");
|
||||
if (bracketIdx >= 0 && bracketIdx < eqIdx) return -1;
|
||||
return eqIdx;
|
||||
}
|
||||
|
||||
function parseKeyFromHeader(s: string): string {
|
||||
@@ -199,7 +209,9 @@ function parseKeyFromHeader(s: string): string {
|
||||
}
|
||||
|
||||
function checkDup(obj: Record<string, any>, key: string): void {
|
||||
if (key in obj) throw new Error(`duplicate_key: ${key}`);
|
||||
// Own-property check only: `key in obj` would spuriously fire on inherited
|
||||
// names like "toString"/"constructor" and mislabel them as duplicates.
|
||||
if (Object.prototype.hasOwnProperty.call(obj, key)) throw new Error(`duplicate_key: ${key}`);
|
||||
}
|
||||
|
||||
function parseArrayFromHeader(
|
||||
@@ -274,6 +286,95 @@ function findClosingBrace(s: string): number {
|
||||
return -1;
|
||||
}
|
||||
|
||||
// A path segment that would pollute Object.prototype if written through.
|
||||
function isUnsafePathKey(k: string): boolean {
|
||||
return k === "__proto__" || k === "constructor" || k === "prototype";
|
||||
}
|
||||
|
||||
// Assign a decoded key without ever mutating Object.prototype: a literal
|
||||
// "__proto__" key is written as an own data property (matching JSON.parse
|
||||
// semantics) instead of reassigning the prototype. All other keys, including
|
||||
// "constructor"/"prototype", are ordinary own-property writes and safe.
|
||||
function safeAssign(obj: Record<string, unknown>, key: string, value: unknown): void {
|
||||
if (key === "__proto__") {
|
||||
Object.defineProperty(obj, key, {
|
||||
value,
|
||||
writable: true,
|
||||
enumerable: true,
|
||||
configurable: true,
|
||||
});
|
||||
} else {
|
||||
obj[key] = value;
|
||||
}
|
||||
}
|
||||
|
||||
function unflattenPaths(
|
||||
pathColumns: Map<string, string[]>,
|
||||
flatValues: Map<string, any>,
|
||||
flatAbsent: Set<string>
|
||||
): Record<string, any> {
|
||||
// Group by top-level parent.
|
||||
const groups = new Map<string, string[]>();
|
||||
const groupOrder: string[] = [];
|
||||
for (const [fieldName, paths] of pathColumns) {
|
||||
if (paths.length === 0) continue;
|
||||
// Drop any path with a prototype-pollution segment. A conformant encoder
|
||||
// never emits these; their presence means hand-crafted/hostile GCF, so the
|
||||
// safe action is to discard the column rather than write through __proto__.
|
||||
if (paths.some(isUnsafePathKey)) continue;
|
||||
const top = paths[0];
|
||||
if (!groups.has(top)) {
|
||||
groups.set(top, []);
|
||||
groupOrder.push(top);
|
||||
}
|
||||
groups.get(top)!.push(fieldName);
|
||||
}
|
||||
|
||||
const result: Record<string, any> = {};
|
||||
|
||||
for (const top of groupOrder) {
|
||||
const fieldNames = groups.get(top)!;
|
||||
const allAbsent = fieldNames.every((f) => flatAbsent.has(f));
|
||||
const allNull = fieldNames.every((f) => {
|
||||
if (flatAbsent.has(f)) return false;
|
||||
const val = flatValues.get(f);
|
||||
return val === null;
|
||||
});
|
||||
|
||||
if (allAbsent) continue;
|
||||
if (allNull) {
|
||||
result[top] = null;
|
||||
continue;
|
||||
}
|
||||
|
||||
for (const fieldName of fieldNames) {
|
||||
if (flatAbsent.has(fieldName)) continue;
|
||||
const paths = pathColumns.get(fieldName)!;
|
||||
const val = flatValues.has(fieldName) ? flatValues.get(fieldName) : null;
|
||||
|
||||
let current = result;
|
||||
for (let k = 0; k < paths.length - 1; k++) {
|
||||
const segment = paths[k];
|
||||
const existing = current[segment];
|
||||
// Overwrite with a fresh object when the slot is missing OR holds a
|
||||
// non-object (null/primitive), so traversal never dereferences a
|
||||
// non-object on malformed input. Conformant output never hits this.
|
||||
if (
|
||||
!Object.prototype.hasOwnProperty.call(current, segment) ||
|
||||
existing === null ||
|
||||
typeof existing !== "object"
|
||||
) {
|
||||
current[segment] = {};
|
||||
}
|
||||
current = current[segment];
|
||||
}
|
||||
current[paths[paths.length - 1]] = val;
|
||||
}
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
function parseTabularBody(
|
||||
lines: string[],
|
||||
start: number,
|
||||
@@ -285,6 +386,20 @@ function parseTabularBody(
|
||||
const rows: any[] = [];
|
||||
let i = start;
|
||||
|
||||
// Detect path columns: fields containing ">".
|
||||
const pathColumnMap = new Map<string, string[]>();
|
||||
for (const f of fields) {
|
||||
if (f.includes(">")) {
|
||||
const parts = f.split(">");
|
||||
// Only treat as a path column if all segments are non-empty.
|
||||
// A literal key like ">" would split into ["", ""].
|
||||
if (parts.every((p) => p.length > 0)) {
|
||||
pathColumnMap.set(f, parts);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Track inline schemas and shared array schemas.
|
||||
const inlineSchemas = new Map<string, string[]>();
|
||||
const sharedArraySchemas = new Map<string, string[]>();
|
||||
|
||||
@@ -296,10 +411,11 @@ function parseTabularBody(
|
||||
|
||||
if (content.length > 0 && content[0] === " ") {
|
||||
const trimmed = content.trimStart();
|
||||
if (trimmed.startsWith(".")) break;
|
||||
if (trimmed.startsWith(".")) break; // attachment lines handled below (v2 indented or v3)
|
||||
break;
|
||||
}
|
||||
|
||||
// Strip @N prefix (must be @digits).
|
||||
let rowData = content;
|
||||
let rowHasID = false;
|
||||
if (rowData.startsWith("@")) {
|
||||
@@ -317,15 +433,32 @@ function parseTabularBody(
|
||||
if (vals.length !== fields.length)
|
||||
throw new Error(`row_width_mismatch: expected ${fields.length}, got ${vals.length}`);
|
||||
|
||||
// Parse cells: scalars, traditional attachments, and inline schema attachments.
|
||||
const cellValues = new Map<string, any>();
|
||||
const traditionalAttFields: string[] = [];
|
||||
const inlineAttFields: string[] = [];
|
||||
const inlineAttOrder: string[] = [];
|
||||
const missingFields = new Set<string>();
|
||||
|
||||
// Collect path column values for unflattening.
|
||||
const flatValues = new Map<string, any>();
|
||||
const flatAbsent = new Set<string>();
|
||||
|
||||
for (let j = 0; j < fields.length; j++) {
|
||||
const cellVal = vals[j];
|
||||
|
||||
// Path columns: store values for later unflattening.
|
||||
if (pathColumnMap.has(fields[j])) {
|
||||
const parsed = parseScalar(cellVal, true);
|
||||
if (parsed === MISSING) {
|
||||
flatAbsent.add(fields[j]);
|
||||
} else {
|
||||
flatValues.set(fields[j], parsed);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
// Check for ^{fields} inline schema declaration.
|
||||
if (cellVal.startsWith("^{") && cellVal.endsWith("}")) {
|
||||
const schemaStr = cellVal.slice(1);
|
||||
const ifs = splitFieldDecl(schemaStr);
|
||||
@@ -341,6 +474,7 @@ function parseTabularBody(
|
||||
continue;
|
||||
}
|
||||
if (parsed === ATTACHMENT) {
|
||||
// Check if this field has a stored inline schema.
|
||||
if (inlineSchemas.has(fields[j])) {
|
||||
inlineAttFields.push(fields[j]);
|
||||
inlineAttOrder.push(fields[j]);
|
||||
@@ -349,6 +483,7 @@ function parseTabularBody(
|
||||
}
|
||||
continue;
|
||||
}
|
||||
// Handle inline schema objects returned by parseScalar (for ^{...} that got through).
|
||||
if (parsed && typeof parsed === "object" && parsed.__inlineSchema) {
|
||||
const ifs = splitFieldDecl(parsed.__inlineSchema);
|
||||
inlineSchemas.set(fields[j], ifs);
|
||||
@@ -360,13 +495,14 @@ function parseTabularBody(
|
||||
}
|
||||
i++;
|
||||
|
||||
// Parse attachments in line order.
|
||||
const allAttFields = [...traditionalAttFields, ...inlineAttFields];
|
||||
const attachmentValues = new Map<string, any>();
|
||||
|
||||
if (rowHasID && allAttFields.length > 0) {
|
||||
if (rowHasID) {
|
||||
let inlineIdx = 0;
|
||||
|
||||
while (i < lines.length && attachmentValues.size < allAttFields.length) {
|
||||
while (i < lines.length) {
|
||||
const aLine = lines[i];
|
||||
let aContent: string | null = null;
|
||||
if (depth === 0 || aLine.startsWith(ind)) {
|
||||
@@ -376,14 +512,17 @@ function parseTabularBody(
|
||||
}
|
||||
if (aContent === null) break;
|
||||
|
||||
// Line starts with ".": traditional or prefixed inline attachment.
|
||||
// Also handle v2-format indented attachments (" .field ...").
|
||||
let attContent = aContent;
|
||||
if (!attContent.startsWith(".") && attContent.startsWith(" .")) {
|
||||
attContent = attContent.slice(2);
|
||||
attContent = attContent.slice(2); // strip v2 indent
|
||||
}
|
||||
if (attContent.startsWith(".")) {
|
||||
const rest = attContent.slice(1);
|
||||
const [attName, afterName] = parseAttachmentName(rest);
|
||||
|
||||
// Check if this is an inline schema field with pipe-delimited data.
|
||||
const ifs = inlineSchemas.get(attName);
|
||||
if (
|
||||
ifs &&
|
||||
@@ -407,6 +546,7 @@ function parseTabularBody(
|
||||
continue;
|
||||
}
|
||||
|
||||
// Traditional attachment.
|
||||
const [name, val, consumed, parsedFields] = parseAttachment(
|
||||
lines,
|
||||
i,
|
||||
@@ -415,6 +555,7 @@ function parseTabularBody(
|
||||
sharedArraySchemas
|
||||
);
|
||||
if (attachmentValues.has(name)) throw new Error(`duplicate_attachment: ${name}`);
|
||||
// Store shared array schema from first row.
|
||||
if (rows.length === 0 && parsedFields) {
|
||||
sharedArraySchemas.set(name, parsedFields);
|
||||
}
|
||||
@@ -423,6 +564,7 @@ function parseTabularBody(
|
||||
continue;
|
||||
}
|
||||
|
||||
// No-prefix line: positional inline data.
|
||||
let foundInline = false;
|
||||
let nextInlineField = "";
|
||||
while (inlineIdx < inlineAttOrder.length) {
|
||||
@@ -456,6 +598,9 @@ function parseTabularBody(
|
||||
if (!attachmentValues.has(f)) throw new Error(`missing_attachment: ${f}`);
|
||||
}
|
||||
|
||||
// Check for duplicate attachments: if the next line is also an attachment
|
||||
// line at this depth, it means there's a second attachment for a field
|
||||
// that was already resolved.
|
||||
if (i < lines.length) {
|
||||
let peekContent: string | null = null;
|
||||
if (depth === 0 || lines[i].startsWith(ind)) {
|
||||
@@ -477,24 +622,30 @@ function parseTabularBody(
|
||||
}
|
||||
}
|
||||
|
||||
// Build row in field declaration order.
|
||||
const row: Record<string, any> = {};
|
||||
for (const f of fields) {
|
||||
if (missingFields.has(f)) continue;
|
||||
if (cellValues.has(f)) {
|
||||
row[f] = cellValues.get(f);
|
||||
safeAssign(row, f, cellValues.get(f));
|
||||
continue;
|
||||
}
|
||||
if (attachmentValues.has(f)) {
|
||||
row[f] = attachmentValues.get(f);
|
||||
safeAssign(row, f, attachmentValues.get(f));
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
if (!rowHasID || allAttFields.length === 0) {
|
||||
const attIndent = ind + " ";
|
||||
if (i < lines.length && lines[i].startsWith(attIndent)) {
|
||||
const peek = lines[i].slice(attIndent.length);
|
||||
if (peek.startsWith(".")) throw new Error(`orphan_attachment: ${peek}`);
|
||||
// Also add any orphan attachment values (fields excluded from column list, e.g. ">" fields).
|
||||
for (const [k, v] of attachmentValues) {
|
||||
if (!Object.prototype.hasOwnProperty.call(row, k)) safeAssign(row, k, v);
|
||||
}
|
||||
|
||||
// Unflatten path columns into nested objects.
|
||||
if (pathColumnMap.size > 0) {
|
||||
const nested = unflattenPaths(pathColumnMap, flatValues, flatAbsent);
|
||||
for (const [k, v] of Object.entries(nested)) {
|
||||
safeAssign(row, k, v);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -523,6 +674,7 @@ function parseAttachmentName(rest: string): [string, string] {
|
||||
return [rest, ""];
|
||||
}
|
||||
|
||||
/** Attachment parser: returns [name, value, consumed, parsedFields]. parsedFields is set for tabular arrays with explicit {fields}. */
|
||||
function parseAttachment(
|
||||
lines: string[],
|
||||
lineIdx: number,
|
||||
@@ -544,6 +696,7 @@ function parseAttachment(
|
||||
if (closeBracket < 0) throw new Error("invalid_count: missing ]");
|
||||
const afterClose = afterName.slice(closeBracket + 1);
|
||||
|
||||
// [N]{fields}: has its own schema.
|
||||
if (afterClose.startsWith("{")) {
|
||||
const endBrace = findClosingBrace(afterClose);
|
||||
let parsedFields: string[] | null = null;
|
||||
@@ -556,12 +709,15 @@ function parseAttachment(
|
||||
return [name, arr, consumed, parsedFields];
|
||||
}
|
||||
|
||||
// [N]: values or [N] (check for inline primitive array first).
|
||||
const afterCloseForInline = afterName.slice(closeBracket + 1);
|
||||
if (afterCloseForInline.startsWith(": ") || afterCloseForInline === ":") {
|
||||
// Inline primitive array: don't use shared schema.
|
||||
const [arr, consumed] = parseArrayFromHeader(lines, lineIdx, depth, afterName);
|
||||
return [name, arr, consumed, null];
|
||||
}
|
||||
|
||||
// [N] without {fields}: check for shared schema.
|
||||
if (sharedSchemas.has(name)) {
|
||||
const sf = sharedSchemas.get(name)!;
|
||||
const countStr = afterName.slice(1, closeBracket);
|
||||
@@ -575,6 +731,7 @@ function parseAttachment(
|
||||
}
|
||||
if (count === 0) return [name, [], 1, null];
|
||||
|
||||
// Peek: if next line starts with @, it's expanded, not tabular.
|
||||
const nextIdx = lineIdx + 1;
|
||||
const ind = " ".repeat(depth);
|
||||
let useShared = true;
|
||||
@@ -591,10 +748,19 @@ function parseAttachment(
|
||||
}
|
||||
}
|
||||
|
||||
// No shared schema: standard parsing.
|
||||
const [arr, consumed] = parseArrayFromHeader(lines, lineIdx, depth, afterName);
|
||||
return [name, arr, consumed, null];
|
||||
}
|
||||
|
||||
// Scalar: =value (field names containing ">" excluded from tabular columns).
|
||||
if (afterName.startsWith("=")) {
|
||||
const valStr = afterName.slice(1);
|
||||
const parsed = parseScalar(valStr, true);
|
||||
if (parsed === MISSING) return [name, null, 1, null];
|
||||
return [name, parsed, 1, null];
|
||||
}
|
||||
|
||||
throw new Error(`invalid attachment form: ${afterName}`);
|
||||
}
|
||||
|
||||
@@ -658,6 +824,7 @@ function validateSummaryCounts(
|
||||
deferredCount: number,
|
||||
contentLines: string[]
|
||||
): void {
|
||||
// Parse counts from "##! summary counts=N,M,..."
|
||||
const parts = summaryLine.split(/\s+/);
|
||||
let countsStr = "";
|
||||
for (const p of parts) {
|
||||
@@ -675,6 +842,7 @@ function validateSummaryCounts(
|
||||
);
|
||||
}
|
||||
|
||||
// Count actual items per deferred section.
|
||||
const actualCounts: number[] = [];
|
||||
let inDeferred = false;
|
||||
let currentCount = 0;
|
||||
|
||||
@@ -1,40 +1,50 @@
|
||||
/**
|
||||
* Generic encoder: converts any JS value into GCF generic profile.
|
||||
* Vendored from gcf-typescript — generic profile only.
|
||||
* GCF generic-profile encoder (encodeGeneric).
|
||||
* Vendored from gcf-typescript — generic profile only. Current with GCF spec v3.2
|
||||
* (nested object flattening) and the [N]: inline-array quoting fix.
|
||||
* https://github.com/blackwell-systems/gcf-typescript
|
||||
*
|
||||
* SPDX-License-Identifier: MIT
|
||||
*/
|
||||
import { formatScalar, formatKey } from "./scalar.ts";
|
||||
import { formatScalar, formatKey, ATTACHMENT } from "./scalar.ts";
|
||||
|
||||
function indent(depth: number): string {
|
||||
return " ".repeat(depth);
|
||||
}
|
||||
|
||||
export function encodeGeneric(data: unknown): string {
|
||||
/** Options for controlling generic encoding behavior. */
|
||||
export interface GenericOptions {
|
||||
/** When true, disables promotion of fixed-shape nested objects to path
|
||||
* columns (e.g. "customer>name"). Nested objects use attachment syntax
|
||||
* instead. Open-weight models currently comprehend the expanded form
|
||||
* better; this gap is expected to close. */
|
||||
noFlatten?: boolean;
|
||||
}
|
||||
|
||||
export function encodeGeneric(data: unknown, opts?: GenericOptions): string {
|
||||
let out = "GCF profile=generic\n";
|
||||
out += encodeRootValue(data);
|
||||
out += encodeRootValue(data, opts);
|
||||
return out;
|
||||
}
|
||||
|
||||
function encodeRootValue(v: unknown): string {
|
||||
function encodeRootValue(v: unknown, opts?: GenericOptions): string {
|
||||
if (v === null || v === undefined) return "=-\n";
|
||||
if (Array.isArray(v)) return encodeRootArray(v);
|
||||
if (typeof v === "object") return encodeObject(v as Record<string, unknown>, 0);
|
||||
if (Array.isArray(v)) return encodeRootArray(v, opts);
|
||||
if (typeof v === "object") return encodeObject(v as Record<string, unknown>, 0, opts);
|
||||
return `=${formatScalar(v, 0)}\n`;
|
||||
}
|
||||
|
||||
function encodeObject(obj: Record<string, unknown>, depth: number): string {
|
||||
function encodeObject(obj: Record<string, unknown>, depth: number, opts?: GenericOptions): string {
|
||||
const prefix = indent(depth);
|
||||
let out = "";
|
||||
for (const key of Object.keys(obj)) {
|
||||
const value = obj[key];
|
||||
const fk = formatKey(key);
|
||||
if (Array.isArray(value)) {
|
||||
out += encodeNamedArray(fk, value, depth);
|
||||
out += encodeNamedArray(fk, value, depth, opts);
|
||||
} else if (typeof value === "object" && value !== null) {
|
||||
out += `${prefix}## ${fk}\n`;
|
||||
out += encodeObject(value as Record<string, unknown>, depth + 1);
|
||||
out += encodeObject(value as Record<string, unknown>, depth + 1, opts);
|
||||
} else {
|
||||
out += `${prefix}${fk}=${formatScalar(value, 0)}\n`;
|
||||
}
|
||||
@@ -42,18 +52,23 @@ function encodeObject(obj: Record<string, unknown>, depth: number): string {
|
||||
return out;
|
||||
}
|
||||
|
||||
function encodeRootArray(arr: unknown[]): string {
|
||||
function encodeRootArray(arr: unknown[], opts?: GenericOptions): string {
|
||||
if (arr.length === 0) return "## [0]\n";
|
||||
if (allPrimitives(arr)) {
|
||||
const vals = arr.map((v) => formatScalar(v, 0x2c));
|
||||
return `## [${arr.length}]: ${vals.join(",")}\n`;
|
||||
}
|
||||
const fields = tabularFields(arr);
|
||||
if (fields) return encodeTabular("## ", arr, fields, 0);
|
||||
return encodeExpanded("## ", arr, 0);
|
||||
if (fields) return encodeTabular("## ", arr, fields, 0, opts);
|
||||
return encodeExpanded("## ", arr, 0, opts);
|
||||
}
|
||||
|
||||
function encodeNamedArray(name: string, arr: unknown[], depth: number): string {
|
||||
function encodeNamedArray(
|
||||
name: string,
|
||||
arr: unknown[],
|
||||
depth: number,
|
||||
opts?: GenericOptions
|
||||
): string {
|
||||
const prefix = indent(depth);
|
||||
if (arr.length === 0) return `${prefix}## ${name} [0]\n`;
|
||||
if (allPrimitives(arr)) {
|
||||
@@ -61,8 +76,8 @@ function encodeNamedArray(name: string, arr: unknown[], depth: number): string {
|
||||
return `${prefix}${name}[${arr.length}]: ${vals.join(",")}\n`;
|
||||
}
|
||||
const fields = tabularFields(arr);
|
||||
if (fields) return encodeTabular(`${prefix}## ${name} `, arr, fields, depth);
|
||||
return encodeExpanded(`${prefix}## ${name} `, arr, depth);
|
||||
if (fields) return encodeTabular(`${prefix}## ${name} `, arr, fields, depth, opts);
|
||||
return encodeExpanded(`${prefix}## ${name} `, arr, depth, opts);
|
||||
}
|
||||
|
||||
function tabularFields(arr: unknown[]): string[] | null {
|
||||
@@ -83,8 +98,9 @@ function tabularFields(arr: unknown[]): string[] | null {
|
||||
|
||||
/** Check if a field is eligible for inline schema: all rows have same flat object shape with 3+ keys. */
|
||||
function inlineSchemaFields(arr: unknown[], fieldName: string): string[] | null {
|
||||
// First row must have the field.
|
||||
const first = arr[0] as Record<string, unknown> | undefined;
|
||||
if (!first || !(fieldName in first)) return null;
|
||||
if (!first || !Object.prototype.hasOwnProperty.call(first, fieldName)) return null;
|
||||
const firstVal = first[fieldName];
|
||||
if (
|
||||
firstVal === null ||
|
||||
@@ -97,10 +113,11 @@ function inlineSchemaFields(arr: unknown[], fieldName: string): string[] | null
|
||||
let canonicalKeys: string[] | null = null;
|
||||
for (const item of arr) {
|
||||
const obj = item as Record<string, unknown>;
|
||||
if (!(fieldName in obj) || obj[fieldName] === null || obj[fieldName] === undefined) continue;
|
||||
if (!Object.prototype.hasOwnProperty.call(obj, fieldName) || obj[fieldName] === null || obj[fieldName] === undefined) continue;
|
||||
const v = obj[fieldName];
|
||||
if (typeof v !== "object" || Array.isArray(v)) return null;
|
||||
const keys = Object.keys(v as Record<string, unknown>);
|
||||
// All values must be scalars.
|
||||
for (const k of keys) {
|
||||
const val = (v as Record<string, unknown>)[k];
|
||||
if (val !== null && val !== undefined && typeof val === "object") return null;
|
||||
@@ -118,21 +135,22 @@ function inlineSchemaFields(arr: unknown[], fieldName: string): string[] | null
|
||||
return canonicalKeys;
|
||||
}
|
||||
|
||||
/** Check if array attachment has same tabular schema across all rows. */
|
||||
/** Check if array attachment has same tabular schema across all rows (first row must have it). All values must be scalars. */
|
||||
function sharedArraySchema(arr: unknown[], fieldName: string): string[] | null {
|
||||
const first = arr[0] as Record<string, unknown> | undefined;
|
||||
if (!first || !(fieldName in first)) return null;
|
||||
if (!first || !Object.prototype.hasOwnProperty.call(first, fieldName)) return null;
|
||||
const firstVal = first[fieldName];
|
||||
if (!Array.isArray(firstVal)) return null;
|
||||
|
||||
let canonicalFields: string[] | null = null;
|
||||
for (const item of arr) {
|
||||
const obj = item as Record<string, unknown>;
|
||||
if (!(fieldName in obj) || obj[fieldName] === null || obj[fieldName] === undefined) continue;
|
||||
if (!Object.prototype.hasOwnProperty.call(obj, fieldName) || obj[fieldName] === null || obj[fieldName] === undefined) continue;
|
||||
const v = obj[fieldName];
|
||||
if (!Array.isArray(v)) return null;
|
||||
const fields = tabularFields(v);
|
||||
if (!fields) return null;
|
||||
// All values must be scalars.
|
||||
for (const arrItem of v) {
|
||||
if (typeof arrItem !== "object" || arrItem === null) return null;
|
||||
for (const val of Object.values(arrItem as Record<string, unknown>)) {
|
||||
@@ -151,25 +169,207 @@ function sharedArraySchema(arr: unknown[], fieldName: string): string[] | null {
|
||||
return canonicalFields;
|
||||
}
|
||||
|
||||
// ── Nested object flattening (v3.2) ──────────────────────────────────────
|
||||
|
||||
interface FlatLeaf {
|
||||
path: string; // ">" separated path (e.g. "customer>name")
|
||||
keys: string[]; // key chain to traverse from row object
|
||||
}
|
||||
|
||||
// Keys that would pollute Object.prototype if used as a flatten path segment.
|
||||
// An object carrying one of these is never flattened; it round-trips whole.
|
||||
function isUnsafeKey(k: string): boolean {
|
||||
return k === "__proto__" || k === "constructor" || k === "prototype";
|
||||
}
|
||||
|
||||
function analyzeFlattenable(
|
||||
arr: unknown[],
|
||||
fieldName: string,
|
||||
parentPath: string
|
||||
): FlatLeaf[] | null {
|
||||
// Field names containing ">" cannot be flattened (would create ambiguous paths).
|
||||
if (fieldName.includes(">")) return null;
|
||||
let canonicalShape: Record<string, "scalar" | "nested"> | null = null;
|
||||
|
||||
for (const item of arr) {
|
||||
const obj = item as Record<string, unknown>;
|
||||
if (!Object.prototype.hasOwnProperty.call(obj, fieldName) || obj[fieldName] === undefined) continue;
|
||||
if (obj[fieldName] === null) {
|
||||
// A nested (non-top-level) null cannot be flattened losslessly: its leaves would
|
||||
// encode as absent ("~") and unflatten back to a missing key, not null. Bail to
|
||||
// the whole-object (attachment) path. A top-level null is fine: it emits "-" and
|
||||
// reconstructs via the all-null rule, so just skip the row from shape analysis.
|
||||
if (parentPath !== "") return null;
|
||||
continue;
|
||||
}
|
||||
const v = obj[fieldName];
|
||||
if (typeof v !== "object" || Array.isArray(v)) return null;
|
||||
|
||||
const keys = Object.keys(v as Record<string, unknown>);
|
||||
|
||||
if (!canonicalShape) {
|
||||
// Null-prototype map so `k in canonicalShape` below only sees own keys
|
||||
// (a field literally named "toString"/"constructor" must not match the
|
||||
// Object.prototype chain), and reject prototype-pollution keys outright.
|
||||
canonicalShape = Object.create(null) as Record<string, "scalar" | "nested">;
|
||||
for (const k of keys) {
|
||||
if (k.includes(">") || isUnsafeKey(k)) return null;
|
||||
const val = (v as Record<string, unknown>)[k];
|
||||
if (val !== null && val !== undefined && typeof val === "object" && !Array.isArray(val)) {
|
||||
canonicalShape[k] = "nested";
|
||||
} else if (Array.isArray(val)) {
|
||||
return null;
|
||||
} else {
|
||||
canonicalShape[k] = "scalar";
|
||||
}
|
||||
}
|
||||
} else {
|
||||
if (keys.length !== Object.keys(canonicalShape).length) return null;
|
||||
for (const k of keys) {
|
||||
if (!Object.prototype.hasOwnProperty.call(canonicalShape, k)) return null;
|
||||
const val = (v as Record<string, unknown>)[k];
|
||||
const expected = canonicalShape[k];
|
||||
if (expected === "scalar") {
|
||||
if (val !== null && val !== undefined && typeof val === "object") return null;
|
||||
} else if (expected === "nested") {
|
||||
if (val !== null && val !== undefined) {
|
||||
if (typeof val !== "object" || Array.isArray(val)) return null;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (!canonicalShape) return null;
|
||||
|
||||
const currentPath = parentPath ? parentPath + ">" + fieldName : fieldName;
|
||||
const parentKeys = parentPath ? [...parentPath.split(">"), fieldName] : [fieldName];
|
||||
|
||||
const leaves: FlatLeaf[] = [];
|
||||
for (const k of Object.keys(canonicalShape)) {
|
||||
if (canonicalShape[k] === "scalar") {
|
||||
leaves.push({ path: currentPath + ">" + k, keys: [...parentKeys, k] });
|
||||
} else {
|
||||
const subArr = arr.map((item) => {
|
||||
const obj = item as Record<string, unknown>;
|
||||
if (!Object.prototype.hasOwnProperty.call(obj, fieldName) || obj[fieldName] === null || obj[fieldName] === undefined)
|
||||
return {};
|
||||
return obj[fieldName];
|
||||
});
|
||||
const subLeaves = analyzeFlattenable(subArr as unknown[], k, currentPath);
|
||||
if (!subLeaves || subLeaves.length === 0) return null;
|
||||
leaves.push(...subLeaves);
|
||||
}
|
||||
}
|
||||
|
||||
// Guard: reject if any row has non-null object with all-null leaves.
|
||||
if (leaves.length > 0) {
|
||||
for (const item of arr) {
|
||||
const obj = item as Record<string, unknown>;
|
||||
if (!Object.prototype.hasOwnProperty.call(obj, fieldName) || obj[fieldName] === null || obj[fieldName] === undefined) continue;
|
||||
const allNull = leaves.every((leaf) => {
|
||||
const val = resolveKeyChain(item, leaf.keys);
|
||||
return val.exists && val.value === null;
|
||||
});
|
||||
if (allNull) return null;
|
||||
}
|
||||
}
|
||||
|
||||
return leaves;
|
||||
}
|
||||
|
||||
function resolveKeyChain(item: unknown, keys: string[]): { value: unknown; exists: boolean } {
|
||||
if (keys.length === 0) return { value: undefined, exists: false };
|
||||
const obj = item as Record<string, unknown>;
|
||||
if (typeof obj !== "object" || obj === null) return { value: undefined, exists: false };
|
||||
if (!Object.prototype.hasOwnProperty.call(obj, keys[0])) return { value: undefined, exists: false };
|
||||
let current: unknown = obj[keys[0]];
|
||||
if (current === null || current === undefined) return { value: current, exists: true };
|
||||
for (let i = 1; i < keys.length; i++) {
|
||||
if (typeof current !== "object" || current === null) return { value: undefined, exists: false };
|
||||
const c = current as Record<string, unknown>;
|
||||
if (!Object.prototype.hasOwnProperty.call(c, keys[i])) return { value: undefined, exists: false };
|
||||
current = c[keys[i]];
|
||||
}
|
||||
return { value: current, exists: true };
|
||||
}
|
||||
|
||||
// ── End flattening helpers ───────────────────────────────────────────────
|
||||
|
||||
function encodeTabular(
|
||||
headerPrefix: string,
|
||||
arr: unknown[],
|
||||
fields: string[],
|
||||
depth: number
|
||||
depth: number,
|
||||
opts?: GenericOptions
|
||||
): string {
|
||||
const prefix = indent(depth);
|
||||
|
||||
// Phase 0: Analyze fields for flattening.
|
||||
const flattenMap = new Map<string, FlatLeaf[]>();
|
||||
if (!opts?.noFlatten) {
|
||||
for (const f of fields) {
|
||||
const leaves = analyzeFlattenable(arr, f, "");
|
||||
if (leaves && leaves.length > 0) {
|
||||
flattenMap.set(f, leaves);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Fields whose names contain ">" must not appear as tabular columns
|
||||
// because the decoder would interpret them as flattened path columns.
|
||||
// Track them for per-row attachment emission (spec rule 7.4.6.1.4).
|
||||
const gtFields = new Set<string>();
|
||||
for (const f of fields) {
|
||||
if (!flattenMap.has(f) && f.includes(">")) {
|
||||
gtFields.add(f);
|
||||
}
|
||||
}
|
||||
|
||||
// Build expanded column list.
|
||||
type ColType = "flat" | "original";
|
||||
interface FlatColumn {
|
||||
headerName: string;
|
||||
colType: ColType;
|
||||
field: string;
|
||||
keys: string[];
|
||||
}
|
||||
const columns: FlatColumn[] = [];
|
||||
for (const f of fields) {
|
||||
if (gtFields.has(f)) continue;
|
||||
const leaves = flattenMap.get(f);
|
||||
if (leaves) {
|
||||
for (const leaf of leaves) {
|
||||
columns.push({
|
||||
headerName: formatKey(leaf.path),
|
||||
colType: "flat",
|
||||
field: f,
|
||||
keys: leaf.keys,
|
||||
});
|
||||
}
|
||||
} else {
|
||||
columns.push({ headerName: formatKey(f), colType: "original", field: f, keys: [] });
|
||||
}
|
||||
}
|
||||
|
||||
// If all fields were excluded (all contain ">"), fall back to expanded.
|
||||
if (columns.length === 0) {
|
||||
return encodeExpanded(headerPrefix, arr, depth, opts);
|
||||
}
|
||||
|
||||
// Pre-compute inline schemas and shared array schemas (skip flattened fields).
|
||||
const inlineSchemas = new Map<string, string[]>();
|
||||
const sharedArrSchemas = new Map<string, string[]>();
|
||||
for (const f of fields) {
|
||||
if (flattenMap.has(f)) continue;
|
||||
const ifs = inlineSchemaFields(arr, f);
|
||||
if (ifs) inlineSchemas.set(f, ifs);
|
||||
const sas = sharedArraySchema(arr, f);
|
||||
if (sas) sharedArrSchemas.set(f, sas);
|
||||
}
|
||||
|
||||
const fmtFields = fields.map((f) => formatKey(f));
|
||||
let out = `${headerPrefix}[${arr.length}]{${fmtFields.join(",")}}\n`;
|
||||
const headerFields = columns.map((c) => c.headerName);
|
||||
let out = `${headerPrefix}[${arr.length}]{${headerFields.join(",")}}\n`;
|
||||
|
||||
for (let i = 0; i < arr.length; i++) {
|
||||
const obj = arr[i] as Record<string, unknown>;
|
||||
@@ -182,8 +382,33 @@ function encodeTabular(
|
||||
}[] = [];
|
||||
let rowHasAttachment = false;
|
||||
|
||||
for (const f of fields) {
|
||||
if (!(f in obj)) {
|
||||
for (const col of columns) {
|
||||
if (col.colType === "flat") {
|
||||
// Resolve value via key chain.
|
||||
if (!Object.prototype.hasOwnProperty.call(obj, col.keys[0])) {
|
||||
cells.push("~");
|
||||
} else {
|
||||
// Check if top-level field is null.
|
||||
const topVal = obj[col.keys[0]];
|
||||
if (topVal === null || topVal === undefined) {
|
||||
cells.push(topVal === null ? "-" : "~");
|
||||
} else {
|
||||
const resolved = resolveKeyChain(obj, col.keys);
|
||||
if (!resolved.exists) {
|
||||
cells.push("~");
|
||||
} else if (resolved.value === null || resolved.value === undefined) {
|
||||
cells.push("-");
|
||||
} else {
|
||||
cells.push(formatScalar(resolved.value, 0x7c));
|
||||
}
|
||||
}
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
// Original (non-flattened) field.
|
||||
const f = col.field;
|
||||
if (!Object.prototype.hasOwnProperty.call(obj, f)) {
|
||||
cells.push("~");
|
||||
continue;
|
||||
}
|
||||
@@ -212,6 +437,14 @@ function encodeTabular(
|
||||
}
|
||||
}
|
||||
|
||||
// Emit fields with ">" in their names as per-row attachments.
|
||||
for (const f of fields) {
|
||||
if (!gtFields.has(f)) continue;
|
||||
if (!Object.prototype.hasOwnProperty.call(obj, f)) continue;
|
||||
rowHasAttachment = true;
|
||||
attachments.push({ name: f, value: obj[f], inline: false });
|
||||
}
|
||||
|
||||
const row = cells.join("|");
|
||||
if (rowHasAttachment) {
|
||||
out += `${prefix}@${i} ${row}\n`;
|
||||
@@ -222,6 +455,7 @@ function encodeTabular(
|
||||
for (const att of attachments) {
|
||||
const fk = formatKey(att.name);
|
||||
if (att.inline && att.inlineFields) {
|
||||
// Inline: single pipe-delimited row, no prefix, no indent.
|
||||
const vals = att.inlineFields.map((inf) => {
|
||||
const val = (att.value as Record<string, unknown>)[inf];
|
||||
if (val === undefined) return "~";
|
||||
@@ -229,15 +463,30 @@ function encodeTabular(
|
||||
});
|
||||
out += `${prefix}${vals.join("|")}\n`;
|
||||
} else if (Array.isArray(att.value)) {
|
||||
// Shared array schema: omit {fields} on subsequent rows.
|
||||
const sas = sharedArrSchemas.get(att.name);
|
||||
if (sas && i > 0) {
|
||||
out += encodeAttachmentArrayShared(prefix, fk, att.value as unknown[], depth + 2, sas);
|
||||
out += encodeAttachmentArrayShared(
|
||||
prefix,
|
||||
fk,
|
||||
att.value as unknown[],
|
||||
depth + 2,
|
||||
sas,
|
||||
opts
|
||||
);
|
||||
} else {
|
||||
out += encodeAttachmentArray(prefix, fk, att.value as unknown[], depth + 2);
|
||||
out += encodeAttachmentArray(prefix, fk, att.value as unknown[], depth + 2, opts);
|
||||
}
|
||||
} else {
|
||||
} else if (typeof att.value === "object" && att.value !== null) {
|
||||
out += `${prefix}.${fk} {}\n`;
|
||||
out += encodeObject(att.value as Record<string, unknown>, depth + 2);
|
||||
out += encodeObject(att.value as Record<string, unknown>, depth + 2, opts);
|
||||
} else {
|
||||
// Scalar attachment (e.g. field names containing ">").
|
||||
if (att.value === null || att.value === undefined) {
|
||||
out += `${prefix}.${fk} =-\n`;
|
||||
} else {
|
||||
out += `${prefix}.${fk} =${formatScalar(att.value, 0)}\n`;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -248,7 +497,8 @@ function encodeAttachmentArray(
|
||||
attPrefix: string,
|
||||
fk: string,
|
||||
arr: unknown[],
|
||||
depth: number
|
||||
depth: number,
|
||||
opts?: GenericOptions
|
||||
): string {
|
||||
if (arr.length === 0) return `${attPrefix}.${fk} [0]\n`;
|
||||
if (allPrimitives(arr)) {
|
||||
@@ -256,8 +506,8 @@ function encodeAttachmentArray(
|
||||
return `${attPrefix}.${fk} [${arr.length}]: ${vals.join(",")}\n`;
|
||||
}
|
||||
const fields = tabularFields(arr);
|
||||
if (fields) return encodeTabular(`${attPrefix}.${fk} `, arr, fields, depth);
|
||||
return encodeExpanded(`${attPrefix}.${fk} `, arr, depth);
|
||||
if (fields) return encodeTabular(`${attPrefix}.${fk} `, arr, fields, depth, opts);
|
||||
return encodeExpanded(`${attPrefix}.${fk} `, arr, depth, opts);
|
||||
}
|
||||
|
||||
function encodeAttachmentArrayShared(
|
||||
@@ -265,25 +515,28 @@ function encodeAttachmentArrayShared(
|
||||
fk: string,
|
||||
arr: unknown[],
|
||||
depth: number,
|
||||
sharedFields: string[]
|
||||
sharedFields: string[],
|
||||
opts?: GenericOptions
|
||||
): string {
|
||||
if (arr.length === 0) return `${attPrefix}.${fk} [0]\n`;
|
||||
if (allPrimitives(arr)) {
|
||||
const vals = arr.map((v) => formatScalar(v, 0x2c));
|
||||
return `${attPrefix}.${fk} [${arr.length}]: ${vals.join(",")}\n`;
|
||||
}
|
||||
// Verify fields match shared schema.
|
||||
const fields = tabularFields(arr);
|
||||
if (
|
||||
fields &&
|
||||
fields.length === sharedFields.length &&
|
||||
fields.every((f, i) => f === sharedFields[i])
|
||||
) {
|
||||
// Omit {fields}, use shared schema.
|
||||
const prefix = indent(depth);
|
||||
let out = `${attPrefix}.${fk} [${arr.length}]\n`;
|
||||
for (const item of arr) {
|
||||
const obj = item as Record<string, unknown>;
|
||||
const cells = sharedFields.map((f) => {
|
||||
if (!(f in obj)) return "~";
|
||||
if (!Object.prototype.hasOwnProperty.call(obj, f)) return "~";
|
||||
if (obj[f] === null || obj[f] === undefined) return "-";
|
||||
return formatScalar(obj[f], 0x7c);
|
||||
});
|
||||
@@ -291,19 +544,25 @@ function encodeAttachmentArrayShared(
|
||||
}
|
||||
return out;
|
||||
}
|
||||
return encodeAttachmentArray(attPrefix, fk, arr, depth);
|
||||
// Fields don't match: fall back to full encoding.
|
||||
return encodeAttachmentArray(attPrefix, fk, arr, depth, opts);
|
||||
}
|
||||
|
||||
function encodeExpanded(headerPrefix: string, arr: unknown[], depth: number): string {
|
||||
function encodeExpanded(
|
||||
headerPrefix: string,
|
||||
arr: unknown[],
|
||||
depth: number,
|
||||
opts?: GenericOptions
|
||||
): string {
|
||||
const prefix = indent(depth);
|
||||
let out = `${headerPrefix}[${arr.length}]\n`;
|
||||
for (let i = 0; i < arr.length; i++) {
|
||||
const item = arr[i];
|
||||
if (Array.isArray(item)) {
|
||||
out += encodeExpandedArrayItem(prefix, i, item, depth);
|
||||
out += encodeExpandedArrayItem(prefix, i, item, depth, opts);
|
||||
} else if (typeof item === "object" && item !== null) {
|
||||
out += `${prefix}@${i} {}\n`;
|
||||
out += encodeObject(item as Record<string, unknown>, depth + 1);
|
||||
out += encodeObject(item as Record<string, unknown>, depth + 1, opts);
|
||||
} else {
|
||||
out += `${prefix}@${i} =${formatScalar(item, 0)}\n`;
|
||||
}
|
||||
@@ -315,7 +574,8 @@ function encodeExpandedArrayItem(
|
||||
prefix: string,
|
||||
idx: number,
|
||||
arr: unknown[],
|
||||
depth: number
|
||||
depth: number,
|
||||
opts?: GenericOptions
|
||||
): string {
|
||||
if (arr.length === 0) return `${prefix}@${idx} [0]\n`;
|
||||
if (allPrimitives(arr)) {
|
||||
@@ -323,8 +583,8 @@ function encodeExpandedArrayItem(
|
||||
return `${prefix}@${idx} [${arr.length}]: ${vals.join(",")}\n`;
|
||||
}
|
||||
const fields = tabularFields(arr);
|
||||
if (fields) return encodeTabular(`${prefix}@${idx} `, arr, fields, depth + 1);
|
||||
return encodeExpanded(`${prefix}@${idx} `, arr, depth + 1);
|
||||
if (fields) return encodeTabular(`${prefix}@${idx} `, arr, fields, depth + 1, opts);
|
||||
return encodeExpanded(`${prefix}@${idx} `, arr, depth + 1, opts);
|
||||
}
|
||||
|
||||
function allPrimitives(arr: unknown[]): boolean {
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
/**
|
||||
* GCF (Graph Compact Format) — generic profile encoder/decoder.
|
||||
* Vendored from gcf-typescript for zero-dependency integration.
|
||||
* Vendored from gcf-typescript for zero-dependency integration. Current with
|
||||
* GCF spec v3.2 (nested object flattening) + [N]: inline-array quoting fix.
|
||||
* https://github.com/blackwell-systems/gcf-typescript
|
||||
*
|
||||
* SPDX-License-Identifier: MIT
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
/**
|
||||
* Common scalar grammar for GCF (Graph Compact Format).
|
||||
* Vendored from gcf-typescript — generic profile only.
|
||||
* Vendored from gcf-typescript — generic profile only. Current with GCF spec v3.2
|
||||
* (nested object flattening) and the [N]: inline-array quoting fix.
|
||||
* https://github.com/blackwell-systems/gcf-typescript
|
||||
*
|
||||
* SPDX-License-Identifier: MIT
|
||||
@@ -8,10 +9,7 @@
|
||||
|
||||
const JSON_NUMBER_RE = /^-?(?:0|[1-9]\d*)(?:\.\d+)?(?:[eE][+-]?\d+)?$/;
|
||||
const NUMERIC_LIKE_RE = /^[+-]\.?\d|^\.\d|^0\d/;
|
||||
// SPEC §2.4: a bracket pair immediately followed by `:` (e.g. `ERR[404]: Not Found`,
|
||||
// `[Speaker 1]: Hello`). Bare, on a line-level key=value RHS, the decoder re-parses
|
||||
// this as an inline-array header → count_mismatch / wrong value (B-GCF-QUOTE).
|
||||
const INLINE_ARRAY_RE = /\[[^\]]*\]:/;
|
||||
const INLINE_ARRAY_RE = /\[[^\]]*\]\s*:/;
|
||||
|
||||
/** Check if a string value must be quoted per Section 2.4. */
|
||||
export function needsQuote(s: string): boolean {
|
||||
@@ -110,7 +108,7 @@ export function formatNumber(f: number): string {
|
||||
if (f === 0) return "0";
|
||||
const abs = Math.abs(f);
|
||||
if (abs >= 1e-6 && abs < 1e21) {
|
||||
return String(f);
|
||||
return toPreciseDecimal(f);
|
||||
}
|
||||
// Exponent notation.
|
||||
let s = f.toExponential();
|
||||
@@ -119,6 +117,11 @@ export function formatNumber(f: number): string {
|
||||
return s;
|
||||
}
|
||||
|
||||
function toPreciseDecimal(f: number): string {
|
||||
// String(f) produces the shortest representation that round-trips through parseFloat.
|
||||
return String(f);
|
||||
}
|
||||
|
||||
const BARE_KEY_RE = /^[a-zA-Z_][a-zA-Z0-9_]*$/;
|
||||
|
||||
/** Check if a key is a valid bare key. */
|
||||
|
||||
@@ -88,6 +88,40 @@ describe("tabular encoder round-trip", () => {
|
||||
assert.deepEqual(decoded, original);
|
||||
});
|
||||
|
||||
it("round-trips deeply nested rows (multi-level objects + array-of-objects) via v3.2 flattening", async () => {
|
||||
const original: Record<string, unknown>[] = Array.from({ length: 12 }, (_, i) => ({
|
||||
id: i,
|
||||
meta: { owner: { name: `n-${i}`, team: `t-${i % 2}` }, count: i * 2 },
|
||||
items: [
|
||||
{ sku: `s-${i}`, qty: i },
|
||||
{ sku: `s2-${i}`, qty: i + 1 },
|
||||
],
|
||||
}));
|
||||
const encoded = encodeTabular(original);
|
||||
// v3.2 nested flattening emits `>`-prefixed path fields for nested objects.
|
||||
assert.match(encoded, /meta>owner>name/);
|
||||
const decoded = decodeTabular(encoded);
|
||||
// Order-insensitive: flattening may reorder object keys, which is semantically irrelevant.
|
||||
assert.deepEqual(decoded, original);
|
||||
});
|
||||
|
||||
it("round-trips a nested object that is null in some rows without losing the null", async () => {
|
||||
// A null nested object must not be flattened (its leaves would encode absent and
|
||||
// unflatten to a missing key). These must all survive as null, not disappear.
|
||||
const cases: Record<string, unknown>[][] = [
|
||||
[{ id: 0, meta: { a: 1, b: 2 } }, { id: 1, meta: null }, { id: 2, meta: { a: 3, b: 4 } }],
|
||||
[
|
||||
{ id: 0, meta: { owner: { name: "a" } } },
|
||||
{ id: 1, meta: { owner: null } },
|
||||
{ id: 2, meta: { owner: { name: "c" } } },
|
||||
],
|
||||
[{ id: 0, o: { p: { team: { x: 1 } } } }, { id: 1, o: { p: { team: null } } }],
|
||||
];
|
||||
for (const original of cases) {
|
||||
assert.deepEqual(decodeTabular(encodeTabular(original)), original);
|
||||
}
|
||||
});
|
||||
|
||||
it("encoded form contains an explicit [N] count marker with field declaration", async () => {
|
||||
const original = makeRows(25);
|
||||
const encoded = encodeTabular(original);
|
||||
@@ -96,6 +130,45 @@ describe("tabular encoder round-trip", () => {
|
||||
});
|
||||
});
|
||||
|
||||
// ─── 1b. Prototype-pollution safety (v3.2 flatten paths + object parser) ──────
|
||||
|
||||
describe("tabular codec — prototype-pollution safety", () => {
|
||||
it("round-trips rows with a literal __proto__ own-key without polluting Object.prototype", async () => {
|
||||
const rows = Array.from({ length: 5 }, (_, i) =>
|
||||
JSON.parse(`{"id":${i},"meta":{"__proto__":{"polluted":true},"real":${i}}}`)
|
||||
);
|
||||
const decoded = decodeTabular(encodeTabular(rows));
|
||||
assert.deepEqual(decoded, rows);
|
||||
assert.equal(({} as Record<string, unknown>).polluted, undefined);
|
||||
});
|
||||
|
||||
it("round-trips a top-level __proto__ column without polluting", async () => {
|
||||
const rows = Array.from({ length: 3 }, (_, i) => JSON.parse(`{"id":${i},"__proto__":"x${i}"}`));
|
||||
const decoded = decodeTabular(encodeTabular(rows));
|
||||
assert.deepEqual(decoded, rows);
|
||||
assert.equal(({} as Record<string, unknown>).x0, undefined);
|
||||
});
|
||||
|
||||
it("does not pollute or throw when decoding hostile GCF with a >__proto__> path column", async () => {
|
||||
const hostile =
|
||||
"```gcf-generic\nGCF profile=generic\n" +
|
||||
'## [1]{id,"a>__proto__>polluted"}\n@0 0|1\n```';
|
||||
decodeTabular(hostile);
|
||||
assert.equal(({} as Record<string, unknown>).polluted, undefined);
|
||||
});
|
||||
|
||||
it("round-trips keys named toString/constructor/valueOf (own-property, not prototype-chain)", async () => {
|
||||
const rows = Array.from({ length: 4 }, (_, i) => ({
|
||||
id: i,
|
||||
toString: `ts${i}`,
|
||||
constructor: `c${i}`,
|
||||
valueOf: i,
|
||||
}));
|
||||
const decoded = decodeTabular(encodeTabular(rows));
|
||||
assert.deepEqual(decoded, rows);
|
||||
});
|
||||
});
|
||||
|
||||
// ─── 2. engine.apply compresses ≥30% and is reversible ───────────────────────
|
||||
|
||||
describe("headroomEngine.apply — compression", () => {
|
||||
|
||||
Reference in New Issue
Block a user