diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f9317e3102..9937d3512d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -533,6 +533,15 @@ jobs: env: BASE_REF: ${{ github.base_ref && format('origin/{0}', github.base_ref) || '' }} run: node scripts/i18n/check-new-key-coverage.mjs + # Absolute complement of the two gates above: every locale must carry exactly the key + # set of en.json, whatever the age of the key. A locale batch is generated from the + # en.json of the day the branch is cut and translates for days while the base keeps + # adding keys — the batch PR adds no key itself, so the new-key gate stays silent and + # 43 absent keys out of ~13,000 still read 99.7 % coverage. Incident 2026-09-15: + # batch 1 (#13044) landed 43 keys short in nine locales, batch 2 (#13660) 10 keys short + # in eight. Fix is `sync-ui-keys --locale= --translate-markers`. + - name: i18n key completeness (every locale carries every en.json key) + run: node scripts/i18n/check-key-completeness.mjs # #8038: cheap glossary/protected-terms consistency gate — # complements i18n-ui-coverage (key parity) and the ICU `i18n` job below diff --git a/changelog.d/features/13827-i18n-key-completeness-gate.md b/changelog.d/features/13827-i18n-key-completeness-gate.md new file mode 100644 index 0000000000..f7a405faac --- /dev/null +++ b/changelog.d/features/13827-i18n-key-completeness-gate.md @@ -0,0 +1 @@ +- **feat(i18n):** new blocking gate `i18n:check-keys` (`scripts/i18n/check-key-completeness.mjs`) — every locale catalog must carry exactly the key set of `en.json`, whatever the age of the key; the percentage and new-key gates let batch 1 (#13044) ship 43 keys short and batch 2 (#13660) 10 keys short. The i18n guide now documents the post-merge re-sync and the retranslation flow. (#13827) diff --git a/docs/architecture/QUALITY_GATES.md b/docs/architecture/QUALITY_GATES.md index 2038c437a7..d3f053e28d 100644 --- a/docs/architecture/QUALITY_GATES.md +++ b/docs/architecture/QUALITY_GATES.md @@ -145,6 +145,7 @@ Runs on every PR to `main`. Blocks merge on failure. | `check-ui-keys-coverage` (inline) | UI i18n key coverage is ≥ 65% | Yes | | `check-ui-value-drift` (inline) | A rewritten English **value** leaves no stale translation behind | Yes | | `check-new-key-coverage` (inline) | A **new** English key reaches every locale | Yes | +| `check-key-completeness` (inline) | Every locale carries exactly the key set of `en.json` (absent key = defect, whatever its age; `__MISSING__` counts as present) | Yes | | `check-translation-ratio` | Real-translation ratio per locale (identical-to-English / placeholder / missing leaves outside the allowlist) must not exceed `config/quality/i18n-translation-baseline.json` + slack | **Advisory** | Needs `fetch-depth: 0` — the value-drift gate diffs `en.json` against the merge base. @@ -536,6 +537,21 @@ several "obvious" merges turned out to hide debt and are **not** clean drop-ins. - Supply-chain (provenance, SBOM, Trivy, Scorecard): [`docs/security/SUPPLY_CHAIN.md`](../security/SUPPLY_CHAIN.md) +#### `check-key-completeness` — key-set parity gate + +`scripts/i18n/check-key-completeness.mjs` (`npm run i18n:check-keys`, job `i18n-ui-coverage`). +Compares the leaf key set of every `src/i18n/messages/.json` with `en.json` and fails +on any absent or extra leaf, regardless of when the key was added. `__MISSING__:` placeholders +count as present (their content is the ratio gate's business). It is the absolute complement +of the two diff-based/percentage gates: `check-ui-keys-coverage` enforces an 80 % floor per +locale (43 absent keys out of ~13,000 still read 99.7 %) and `check-new-key-coverage` judges +only the keys a PR adds to `en.json`. A locale batch is generated from the `en.json` of the day +its branch is cut and translates for days while the base keeps adding keys; the batch PR adds no +key itself, so both siblings stayed silent when batch 1 (#13044) landed 43 keys short in nine +locales and batch 2 (#13660) 10 keys short in eight (2026-09-15). Fix a red with +`node scripts/i18n/sync-ui-keys.mjs --locale= --translate-markers`; an `extra` leaf +means the source dropped it — delete it from the locale. `--warn` reports without failing. + #### `check-new-key-coverage` — new-key i18n gate Sibling of `check-ui-value-drift`. That one catches an English value that was **rewritten** diff --git a/docs/guides/I18N.md b/docs/guides/I18N.md index d518238449..627ff57162 100644 --- a/docs/guides/I18N.md +++ b/docs/guides/I18N.md @@ -211,6 +211,38 @@ npm run i18n:check-ui-coverage && npm run i18n:check-ratio && npm run check:docs adapter and must not be edited by hand. The Google-Translate generator (`generate-multilang.mjs`) is deprecated and is not part of this flow. +## Keeping catalogs complete and retranslating English copies + +Three gates guard the catalogs, and they see different things: + +| Gate | Sees | +| -------------------------------- | ----------------------------------------------------------------------- | +| `npm run i18n:check-ui-coverage` | ≥ 80 % of leaves translated per locale | +| `npm run i18n:check-new-keys` | a key the PR adds to `en.json` reached every locale | +| `npm run i18n:check-keys` | every locale carries exactly the key set of `en.json`, whatever the age | +| `npm run i18n:check-ratio` | share of leaves still identical to English may only fall (ratchet) | + +**After every merge of the base into a locale branch**, re-sync the locales the branch owns — +the base keeps adding keys while a batch translates: + +```bash +node scripts/i18n/sync-ui-keys.mjs --locale=km,kn,ml --translate-markers --batch-size=40 +npm run i18n:check-keys +``` + +**Retranslating verbatim-English leaves** (`--retranslate-identical`) turns every leaf that is +still byte-identical to `en.json` — outside `scripts/i18n/untranslatable-keys.json` — into a +`__MISSING__:` placeholder and translates it in the same run. Before a bulk run, put every key +that a test pins to its English value (product, engine and flag names — e.g. the Vietnamese +sidebar engines in `dashboard-localization-contract.test.ts`, the pt-BR label in +`server-owned-tool-loop-flag.test.ts`) into the allowlist first, then: + +```bash +node scripts/i18n/sync-ui-keys.mjs --locale=es --retranslate-identical --translate-markers --batch-size=40 +npm run i18n:check-ratio:update # tighten the baseline once the locale improved +npm run i18n:check-glossary # zh-CN / zh-TW / ko protected terms +``` + ## Auto-Translation Pipeline ### generate-multilang.mjs (Google Translate) diff --git a/package.json b/package.json index a42bf3a0cf..acc9cd9b4b 100644 --- a/package.json +++ b/package.json @@ -286,7 +286,8 @@ "test:unit:serial": "cross-env DISABLE_SQLITE_AUTO_BACKUP=true node --max-old-space-size=8192 --import tsx/esm --import ./open-sse/utils/setupPolyfill.ts --import ./tests/_setup/isolateDataDir.ts --test --test-force-exit --test-concurrency=1 \"tests/unit/serial/**/*.test.ts\"", "alibaba:sync-allowlist": "node --import tsx/esm scripts/ops/sync-alibaba-allowlist.mjs", "check:vitest-exclusions": "node scripts/check/check-vitest-exclusions.mjs", - "i18n:check-new-keys": "node scripts/i18n/check-new-key-coverage.mjs" + "i18n:check-new-keys": "node scripts/i18n/check-new-key-coverage.mjs", + "i18n:check-keys": "node scripts/i18n/check-key-completeness.mjs" }, "dependencies": { "@aws-sdk/client-bedrock-runtime": "^3.1120.0", diff --git a/scripts/i18n/check-key-completeness.mjs b/scripts/i18n/check-key-completeness.mjs new file mode 100644 index 0000000000..bd9e006c3e --- /dev/null +++ b/scripts/i18n/check-key-completeness.mjs @@ -0,0 +1,132 @@ +#!/usr/bin/env node +/** + * OmniRoute — i18n key COMPLETENESS gate (CI gate, blocking). + * + * Every `src/i18n/messages/.json` must carry exactly the key set of `en.json`: + * no leaf absent, no leaf the source no longer has. A `__MISSING__:` placeholder counts as + * present (the ratio gate judges its content); an ABSENT key is the defect this gate names. + * + * Why the two sibling gates cannot see it (the incident it encodes, 2026-09-15): + * - `check-ui-keys-coverage.mjs` enforces an 80 % floor per locale — 43 absent keys out of + * ~13,000 still reads 99.7 %. + * - `check-new-key-coverage.mjs` judges only the keys a PR ADDS to en.json. A locale batch + * is generated from the en.json of the moment the branch is cut; while its translation + * runs for days the base keeps adding keys, and the batch PR adds none itself — so the + * nine batch-1 catalogs (#13044) landed 43 keys short and the eight batch-2 catalogs + * (#13660) 10 keys short. The home widget test was the first thing that noticed. + * + * This gate is absolute, not diff-based: it compares the tree as it is. + * + * Usage: + * node scripts/i18n/check-key-completeness.mjs # blocking + * node scripts/i18n/check-key-completeness.mjs --warn # report only, exit 0 + * npm run i18n:check-keys + */ + +import { promises as fs } from "node:fs"; +import path from "node:path"; +import process from "node:process"; +import { fileURLToPath, pathToFileURL } from "node:url"; + +const SCRIPT_DIR = path.dirname(fileURLToPath(import.meta.url)); +const ROOT = path.resolve(SCRIPT_DIR, "..", ".."); +const MESSAGES_DIR = path.join(ROOT, "src", "i18n", "messages"); +const SOURCE_LOCALE = "en"; + +function isPlainObject(value) { + return value !== null && typeof value === "object" && !Array.isArray(value); +} + +/** Dotted leaf paths of a catalog tree (objects recurse, everything else is a leaf). */ +export function leafPaths(node, prefix = "", out = new Set()) { + if (!isPlainObject(node)) return out; + for (const [key, value] of Object.entries(node)) { + const dotted = prefix ? `${prefix}.${key}` : key; + if (isPlainObject(value)) leafPaths(value, dotted, out); + else out.add(dotted); + } + return out; +} + +/** + * Pure core. `en` is the source catalog, `locales` maps locale code → catalog. Returns one + * entry per locale whose key set differs from the source, sorted by locale, with sorted + * `missing` (in en, absent in the locale) and `extra` (in the locale, gone from en) lists. + * Locales with an identical key set are not listed. + */ +export function findIncompleteLocales({ en, locales }) { + const source = leafPaths(en); + const gaps = []; + for (const locale of Object.keys(locales).sort()) { + const target = leafPaths(locales[locale]); + const missing = [...source].filter((k) => !target.has(k)).sort(); + const extra = [...target].filter((k) => !source.has(k)).sort(); + if (missing.length || extra.length) gaps.push({ locale, missing, extra }); + } + return gaps; +} + +async function readCatalogs() { + const files = (await fs.readdir(MESSAGES_DIR)).filter((f) => f.endsWith(".json")).sort(); + const locales = {}; + let en = null; + for (const file of files) { + const code = file.slice(0, -".json".length); + const parsed = JSON.parse(await fs.readFile(path.join(MESSAGES_DIR, file), "utf8")); + if (code === SOURCE_LOCALE) en = parsed; + else locales[code] = parsed; + } + if (!en) throw new Error(`[i18n-keys] ${SOURCE_LOCALE}.json not found in ${MESSAGES_DIR}`); + return { en, locales }; +} + +function formatReport(gaps, sample = 5) { + const lines = []; + for (const { locale, missing, extra } of gaps) { + const parts = []; + if (missing.length) { + parts.push( + `${missing.length} missing (${missing.slice(0, sample).join(", ")}${missing.length > sample ? ", …" : ""})` + ); + } + if (extra.length) { + parts.push( + `${extra.length} extra (${extra.slice(0, sample).join(", ")}${extra.length > sample ? ", …" : ""})` + ); + } + lines.push(` - ${locale}: ${parts.join("; ")}`); + } + return lines.join("\n"); +} + +async function main() { + const warnOnly = process.argv.includes("--warn"); + const { en, locales } = await readCatalogs(); + const gaps = findIncompleteLocales({ en, locales }); + const total = leafPaths(en).size; + const count = Object.keys(locales).length; + if (gaps.length === 0) { + console.log( + `[i18n-keys] OK — ${count} locales carry all ${total} keys of en.json, none extra.` + ); + return; + } + const missingTotal = gaps.reduce((s, g) => s + g.missing.length, 0); + const extraTotal = gaps.reduce((s, g) => s + g.extra.length, 0); + console.error( + `[i18n-keys] ${warnOnly ? "WARN" : "FAIL"} — ${gaps.length}/${count} locales differ from en.json (${missingTotal} missing, ${extraTotal} extra leaves):` + ); + console.error(formatReport(gaps)); + console.error( + "[i18n-keys] Fix: node scripts/i18n/sync-ui-keys.mjs --locale= --translate-markers (adds the missing keys and translates them); extra keys mean the source dropped them — remove them from the locale." + ); + if (!warnOnly) process.exitCode = 1; +} + +const isDirectRun = process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href; +if (isDirectRun) { + main().catch((err) => { + console.error(`[i18n-keys] ${err.message}`); + process.exitCode = 1; + }); +} diff --git a/tests/unit/i18n-key-completeness.test.ts b/tests/unit/i18n-key-completeness.test.ts new file mode 100644 index 0000000000..bda8b71c12 --- /dev/null +++ b/tests/unit/i18n-key-completeness.test.ts @@ -0,0 +1,63 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { findIncompleteLocales, leafPaths } from "../../scripts/i18n/check-key-completeness.mjs"; + +// Absolute key-set parity between en.json and every locale catalog. Unlike the new-key gate +// (diff-based) and the coverage gate (80 % floor), this one names an ABSENT key regardless of +// when it was added — the defect the batch-1/batch-2 locale PRs (#13044, #13660) shipped. + +const en = { home: { title: "Home", legend: { active: "Active" } }, common: { save: "Save" } }; + +test("leafPaths flattens nested objects into dotted leaves and ignores non-objects", () => { + assert.deepEqual([...leafPaths(en)].sort(), ["common.save", "home.legend.active", "home.title"]); + assert.deepEqual([...leafPaths("not a tree")], []); +}); + +test("a locale with exactly the source key set is not listed", () => { + const gaps = findIncompleteLocales({ + en, + locales: { + pt: { home: { title: "Início", legend: { active: "Ativo" } }, common: { save: "Salvar" } }, + }, + }); + assert.deepEqual(gaps, []); +}); + +test("a __MISSING__ placeholder counts as present — the ratio gate judges its content", () => { + const gaps = findIncompleteLocales({ + en, + locales: { + pt: { + home: { title: "__MISSING__:Home", legend: { active: "Ativo" } }, + common: { save: "Salvar" }, + }, + }, + }); + assert.deepEqual(gaps, []); +}); + +test("absent leaves are reported per locale, sorted, whatever their age", () => { + const gaps = findIncompleteLocales({ + en, + locales: { + km: { home: { title: "ទំព័រដើម" }, common: { save: "រក្សាទុក" } }, + de: { home: { title: "Start", legend: { active: "Aktiv" } }, common: { save: "Speichern" } }, + }, + }); + assert.deepEqual(gaps, [{ locale: "km", missing: ["home.legend.active"], extra: [] }]); +}); + +test("leaves the source dropped are reported as extra, and a wrong shape counts as missing", () => { + const gaps = findIncompleteLocales({ + en, + locales: { + fr: { + home: { title: "Accueil", legend: "Légende" }, + common: { save: "Enregistrer", cancel: "Annuler" }, + }, + }, + }); + assert.deepEqual(gaps, [ + { locale: "fr", missing: ["home.legend.active"], extra: ["common.cancel", "home.legend"] }, + ]); +});