Compare commits

..

2 Commits

Author SHA1 Message Date
diegosouzapw
455aade660 fix(i18n): split oversized docs sections before translating
chunkMarkdown only cut on '## ' headings, so a single long section (README.md
carries a 16 KB one, USER_GUIDE.md a 20 KB one) became one oversized request.
On the slow fallback model that request could not finish inside the backend's
10-minute fetch timeout, and the three biggest docs of every new locale failed
with 'fetch failed' on every retry — Odia burned eight attempts on them.

A section still longer than maxChars is now split again on sub-headings and
paragraph boundaries, never inside a fenced code block; a block that is itself
larger than the limit stays whole rather than being cut mid-paragraph.
2026-09-11 16:08:49 -03:00
diegosouzapw
e89a4da2b0 fix(i18n): keep the ICU literal escape around angle placeholders when translating
English writes '<name>' — the single quotes are ICU's escape, so the span
renders as literal text. Every backend drops them, and the translated message
then parses as an unclosed ICU tag: the nine locales of the first batch each
shipped two such strings and only CI caught them.

translateString and translateBatch now restore the quoting, doubling an
apostrophe inside the span so it does not close the literal early (Estonian
OmniRoute'i, Irish d'eochair). Messages whose English mixes real markup with a
literal span are left untouched, since there is no safe way to tell the two
apart.
2026-09-10 12:39:15 -03:00
6 changed files with 211 additions and 116 deletions

View File

@@ -578,31 +578,14 @@ own dedicated branch, and you MUST confirm the base branch with the operator bef
# HARD LINKS (`cp -al`), never a symlink: ~5s for the whole tree and near-zero extra
# disk (the inodes are shared), and unlike a symlink it does not break the dev server.
cp -al "$(git -C <main_checkout> rev-parse --show-toplevel)/node_modules" node_modules
# `.husky/_` is gitignored, so a fresh worktree does NOT have it and
# `core.hooksPath=.husky/_` then points at a directory that does not exist —
# every pre-commit gate goes silently mute. Copy it too.
cp -a "$(git -C <main_checkout> rev-parse --show-toplevel)/.husky/_" .husky/_
```
`scripts/dev/new-worktree.sh <branch> [base]` does all of the above (canonical path,
hard-linked `node_modules`, `.husky/_`) and then **verifies** the hook is actually
executable, so prefer it over running the steps by hand.
**Never `ln -s` node_modules.** Turbopack rejects a symlink that resolves outside the
project root, so `npm run dev` dies with a FATAL panic (`Symlink [project]/node_modules
is invalid, it points out of the filesystem root`) while typecheck, lint and the test
runners all keep passing — the error names "filesystem root", not the worktree, so it
reads like a Next/build bug and costs real time to trace (incident 2026-07-31, #9043).
**A worktree without `.husky/_` runs NO pre-commit gate — and says nothing.** `git`
resolves `core.hooksPath` relative to the worktree top; when the directory is missing it
simply finds no hook and commits. Nothing is printed, the commit succeeds, and the
identity/lint/docs gates never ran. This is how 59 commits carrying a stale identity
override (name of a contributor + the maintainer's e-mail) got past
`scripts/check/check-git-identity.sh` between 2026-08-29 and 09-02 — they were all made in
`cp -al` worktrees. Verify with `ls .husky/_/pre-commit` inside a new worktree, or just use
`scripts/dev/new-worktree.sh`, which fails loudly when the hook is not executable.
3. **Work, commit, push, open the PR — all from inside the worktree.** Never `git checkout` a
different branch inside a worktree another session might share.
4. **Tear down only your own** worktree + branch when done, from the main checkout:

View File

@@ -1,92 +0,0 @@
#!/usr/bin/env sh
# Cria uma worktree isolada seguindo o protocolo obrigatório do AGENTS.md
# (Git Workflow → "Worktree isolation" / Hard Rule #19), incluindo os dois
# passos que são fáceis de esquecer e falham em silêncio:
#
# 1. node_modules por HARD LINK (`cp -al`), nunca symlink — um symlink que
# resolve fora da raiz mata o Turbopack com um FATAL que culpa a
# "filesystem root" e não a worktree (incidente 2026-07-31, #9043).
# 2. `.husky/_` copiado — é gitignored, então uma worktree nova NÃO o tem, e
# `core.hooksPath=.husky/_` aponta para um diretório inexistente: TODOS os
# hooks de pre-commit ficam mudos, sem aviso nenhum. Foi assim que 59
# commits com identidade trocada passaram pelo gate entre 29/08 e 02/09
# (ver .mailmap e scripts/check/check-git-identity.sh).
#
# Uso: scripts/dev/new-worktree.sh <branch> [base-branch]
# Ex.: scripts/dev/new-worktree.sh fix/12345-algo release/v3.8.51
set -e
BRANCH="$1"
BASE="$2"
if [ -z "$BRANCH" ]; then
echo "uso: scripts/dev/new-worktree.sh <branch> [base-branch]" >&2
echo " ex: scripts/dev/new-worktree.sh fix/12345-algo release/v3.8.51" >&2
exit 1
fi
# O checkout PRINCIPAL, mesmo quando este script roda de dentro de outra worktree:
# `--show-toplevel` devolveria a worktree atual, e a nova nasceria aninhada nela.
MAIN=$(dirname "$(git rev-parse --path-format=absolute --git-common-dir)")
cd "$MAIN"
# Sem base explícita, usa a release ativa (maior release/* por semver) — nunca
# `main` e nunca "a branch em que eu estou", conforme a Hard Rule #19.
if [ -z "$BASE" ]; then
BASE=$(git ls-remote --heads origin 'refs/heads/release/*' \
| sed 's#.*refs/heads/##' | sort -V | tail -1)
[ -z "$BASE" ] && { echo "não consegui resolver a release ativa; passe a base explicitamente" >&2; exit 1; }
echo "base não informada — usando a release ativa: $BASE"
fi
DIR=".claude/worktrees/${BRANCH##*/}"
[ -e "$DIR" ] && { echo "já existe: $DIR" >&2; exit 1; }
git fetch origin "$BASE" --quiet
git worktree add "$DIR" -b "$BRANCH" "origin/$BASE"
# `.husky/_` PRIMEIRO: é minúsculo e é o que decide se os gates locais rodam.
# Copiar node_modules antes seria arriscar abortar (set -e) numa árvore de ~10 GB
# e deixar a worktree sem hook nenhum — exatamente o defeito que este script existe
# para impedir.
if [ -d "$MAIN/.husky/_" ]; then
cp -a "$MAIN/.husky/_" "$DIR/.husky/_"
else
echo "AVISO: .husky/_ não existe no checkout principal — rode 'npm install' lá primeiro" >&2
fi
# node_modules: hard links, ~5s e disco quase zero (inodes compartilhados).
# Um `cp -al SRC DEST` com DEST já existente aninharia SRC DENTRO dele
# (node_modules/node_modules), então DEST não pode existir aqui.
if [ -d "$MAIN/node_modules/node_modules" ]; then
echo "AVISO: $MAIN/node_modules/node_modules existe — resíduo de um cp -al aninhado." >&2
echo " Ele infla a cópia e esgota o limite de hard links; convém removê-lo." >&2
fi
if [ -d "$MAIN/node_modules" ]; then
# Falha parcial (limite de hard links, disco) não pode derrubar a worktree inteira:
# os hooks já estão no lugar e o npm install continua sendo uma saída válida.
if cp -al "$MAIN/node_modules" "$DIR/node_modules" 2>"$DIR/.cp-node-modules.log"; then
echo "node_modules: $(ls "$DIR/node_modules" | wc -l) entradas (hard links)"
rm -f "$DIR/.cp-node-modules.log"
else
echo "AVISO: a cópia de node_modules falhou parcialmente (veja $DIR/.cp-node-modules.log)." >&2
echo " Primeiras linhas:" >&2
head -3 "$DIR/.cp-node-modules.log" >&2
fi
else
echo "AVISO: node_modules não existe no checkout principal — rode 'npm install' lá primeiro" >&2
fi
# Verificação: o hook precisa estar REALMENTE ativo, não apenas presente.
HOOKS_PATH=$(git -C "$DIR" config --get core.hooksPath || echo ".git/hooks")
if [ -x "$DIR/$HOOKS_PATH/pre-commit" ]; then
echo "hooks: ativos ($HOOKS_PATH/pre-commit)"
else
echo "AVISO: pre-commit NÃO está ativo em $DIR/$HOOKS_PATH — os gates locais não vão rodar" >&2
exit 1
fi
echo
echo "pronto: $DIR (branch $BRANCH, base $BASE)"
echo " cd $DIR"

View File

@@ -119,6 +119,32 @@ export const TRANSLATION_SYSTEM = (englishName, native) =>
`Keep punctuation and trailing whitespace identical to the source.`,
].join(" ");
/**
* Restores the ICU literal escape the backends drop around angle placeholders.
*
* English writes `'<name>'`: those single quotes are ICU's escape, so the span
* renders as the literal text `<name>`. Translations come back as a bare
* `<nome>`, which ICU then parses as an (unclosed) tag and the message fails to
* compile — every locale in the first batch shipped two of these.
*
* Only messages whose English side quotes EVERY angle span are touched: when the
* source mixes real markup (`<b>`) with a literal span there is no safe way to
* tell which is which, so the translation is left exactly as it came back. An
* apostrophe inside the span is doubled, otherwise it closes the literal early.
*/
export function preserveIcuLiteralQuotes(englishValue, translated) {
if (typeof englishValue !== "string" || typeof translated !== "string") return translated;
if (!englishValue.includes("'<")) return translated;
// Every "<" in the source must be the start of an escaped span.
for (let i = 0; i < englishValue.length; i++) {
if (englishValue[i] === "<" && englishValue[i - 1] !== "'") return translated;
}
return translated.replace(
/(?<!')<([^<>]*)>(?!')/g,
(_m, inner) => `'<${inner.replace(/'/g, "''")}>'`
);
}
export async function translateString(englishValue, localeEntry, backend) {
const englishName = localeEntry.english ?? localeEntry.name;
const native = localeEntry.native ?? localeEntry.name;
@@ -127,7 +153,7 @@ export async function translateString(englishValue, localeEntry, backend) {
{ role: "user", content: englishValue },
];
const out = await callChat(messages, backend);
return out.trim();
return preserveIcuLiteralQuotes(englishValue, out.trim());
}
// ----- Batch mode ----------------------------------------------------------
@@ -198,8 +224,14 @@ export async function translateBatch(entries, localeEntry, backend) {
{ role: "user", content: JSON.stringify(payload) },
];
const out = await callChat(messages, backend);
return parseBatchResponse(
const parsed = parseBatchResponse(
out,
entries.map((e) => e.id)
);
for (const entry of entries) {
if (typeof parsed[entry.id] === "string") {
parsed[entry.id] = preserveIcuLiteralQuotes(entry.text, parsed[entry.id]);
}
}
return parsed;
}

View File

@@ -380,16 +380,22 @@ const SYSTEM_PROMPT = (englishName, native) =>
`Return ONLY the translated markdown — no preamble, no explanation, no surrounding fences.`,
].join(" ");
// Splits a markdown body into chunks of <= maxChars, breaking on top-level `## ` headings only.
function chunkMarkdown(markdown, maxChars = 6000) {
// Splits a markdown body into chunks of <= maxChars. Top-level `## ` headings
// are the preferred cut; a section that is still longer than maxChars is then
// split again on `### ` headings and paragraph boundaries, never inside a
// fenced code block. Before the second pass a single long section (README.md
// has a 16 KB one, USER_GUIDE.md a 20 KB one) became one oversized request
// that the slow fallback model could not answer inside the backend's
// 10-minute fetch timeout, and the biggest docs failed on every retry.
export function chunkMarkdown(markdown, maxChars = 6000) {
if (markdown.length <= maxChars) return [markdown];
const lines = markdown.split("\n");
const chunks = [];
const sections = [];
let buf = [];
let size = 0;
for (const line of lines) {
if (line.startsWith("## ") && size > maxChars * 0.5) {
chunks.push(buf.join("\n"));
sections.push(buf.join("\n"));
buf = [line];
size = line.length;
} else {
@@ -397,7 +403,65 @@ function chunkMarkdown(markdown, maxChars = 6000) {
size += line.length + 1;
}
}
if (buf.length) chunks.push(buf.join("\n"));
if (buf.length) sections.push(buf.join("\n"));
return sections.flatMap((section) =>
section.length <= maxChars ? [section] : splitOversizedSection(section, maxChars)
);
}
const FENCE_LINE = /^\s*(```|~~~)/;
// Groups a section into blocks — a whole fenced code block, a heading-led run,
// or a paragraph ending at a blank line — and packs them greedily. A block that
// is itself larger than maxChars stays whole: cutting mid-paragraph or inside a
// fence would hand the model a fragment it cannot translate faithfully.
function splitOversizedSection(section, maxChars) {
const blocks = [];
let block = [];
let inFence = false;
for (const line of section.split("\n")) {
const isFence = FENCE_LINE.test(line);
if (inFence) {
block.push(line);
if (isFence) {
inFence = false;
blocks.push(block);
block = [];
}
continue;
}
if (isFence) {
if (block.length) blocks.push(block);
block = [line];
inFence = true;
continue;
}
if (/^##+ /.test(line) && block.length) {
blocks.push(block);
block = [];
}
block.push(line);
if (line.trim() === "") {
blocks.push(block);
block = [];
}
}
if (block.length) blocks.push(block);
const chunks = [];
let current = [];
let size = 0;
for (const lines of blocks) {
const length = lines.join("\n").length + 1;
if (size > 0 && size + length > maxChars) {
chunks.push(current.join("\n"));
current = [];
size = 0;
}
current.push(...lines);
size += length;
}
if (current.length) chunks.push(current.join("\n"));
return chunks;
}

View File

@@ -0,0 +1,65 @@
import test from "node:test";
import assert from "node:assert/strict";
import { chunkMarkdown } from "../../scripts/i18n/run-translation.mjs";
// The docs translator splits a page into chunks so one upstream call stays
// short. It used to break only on `## ` headings, so a single long section
// (README.md carries a 16 KB one, USER_GUIDE.md a 20 KB one) became one
// oversized request that the slow fallback model could not answer inside the
// backend's 10-minute fetch timeout — the three biggest docs of every locale
// then failed with "fetch failed" on every retry.
const para = (label: string, n = 12) =>
Array.from({ length: n }, (_, i) => `${label} sentence ${i + 1} with some filler text.`).join(
" "
);
test("a section longer than maxChars is split on sub-headings and paragraphs", () => {
const body = [
"## Big",
para("a"),
"",
"### Part one",
para("b"),
"",
para("c"),
"",
"### Part two",
para("d"),
].join("\n");
const chunks = chunkMarkdown(body, 700);
assert.ok(chunks.length > 1, "must split an oversized section");
for (const chunk of chunks) assert.ok(chunk.length <= 700, `chunk too big: ${chunk.length}`);
// Nothing lost: re-joining with blank lines reproduces every line of the source.
const lines = (s: string) => s.split("\n").filter((l) => l.trim() !== "");
assert.deepEqual(lines(chunks.join("\n\n")), lines(body));
});
test("never splits inside a fenced code block", () => {
const code = [
"```ts",
...Array.from({ length: 30 }, (_, i) => `const v${i} = ${i};`),
"```",
].join("\n");
const body = ["## Code", para("x"), "", code, "", para("y")].join("\n");
const chunks = chunkMarkdown(body, 500);
const withFence = chunks.filter((c) => c.includes("```"));
for (const chunk of withFence) {
assert.equal(
(chunk.match(/```/g) ?? []).length % 2,
0,
"fence must open and close in the same chunk"
);
}
});
test("keeps the old behaviour for short pages and for normal ## sections", () => {
assert.deepEqual(chunkMarkdown("# Title\n\nshort", 6000), ["# Title\n\nshort"]);
const body = ["## A", para("a", 4), "", "## B", para("b", 4), "", "## C", para("c", 4)].join(
"\n"
);
const chunks = chunkMarkdown(body, 400);
assert.ok(chunks.every((c) => c.length <= 400));
assert.ok(chunks.length > 1);
for (const chunk of chunks.slice(1)) assert.match(chunk, /^## /, "cuts land on ## boundaries");
});

View File

@@ -0,0 +1,43 @@
import test from "node:test";
import assert from "node:assert/strict";
import { preserveIcuLiteralQuotes } from "../../scripts/i18n/lib/translate-backend.mjs";
// The English catalog escapes angle placeholders for ICU: the single quotes in
// '<name>' make the span literal text. Translation backends drop them, and the
// message then parses as an unclosed ICU tag — every locale added in batch 1
// shipped two such strings before CI caught them.
test("re-escapes an angle span the translation left bare", () => {
assert.equal(
preserveIcuLiteralQuotes(
"Regenerate ~/.claude/profiles/'<name>'/settings.json",
"Regenerar ~/.claude/profiles/<nome>/settings.json"
),
"Regenerar ~/.claude/profiles/'<nome>'/settings.json"
);
});
test("doubles an apostrophe inside the span, which would close the literal early", () => {
assert.equal(
preserveIcuLiteralQuotes("'<your OmniRoute API key>'", "<teie OmniRoute'i API-võti>"),
"'<teie OmniRoute''i API-võti>'"
);
});
test("leaves a translation that already carries the quotes untouched", () => {
const already = "'<seu token>'";
assert.equal(preserveIcuLiteralQuotes("'<your token>'", already), already);
});
test("does not touch messages whose English has a real unquoted tag", () => {
// <b> here is markup the translation must keep as markup, not literal text.
const translated = "Clique em <b>Salvar</b> e em '<nome>'";
assert.equal(preserveIcuLiteralQuotes("Click <b>Save</b> and '<name>'", translated), translated);
});
test("leaves messages without angle spans alone", () => {
assert.equal(preserveIcuLiteralQuotes("Settings", "Nastavitve"), "Nastavitve");
assert.equal(
preserveIcuLiteralQuotes("Restricted to {count} endpoints", "Omejeno na {count} točk"),
"Omejeno na {count} točk"
);
});