feat(ci): give the review bot a severity scale and a tally

REVIEW.md said what blocks and what does not, but never how to mark a finding,
so every review invented its own shape and none carried a severity. It now
names the three markers the hosted Code Review service uses - Important, Nit,
Pre-existing - and keys them to what the pull request did rather than to how
alarming a defect looks alone: a defect it introduces or worsens is Important,
one it merely brought into view is Pre-existing and cannot be a reason to hold
it. Pre-existing was missing entirely, and checking what this panel emits means
reading far outside the diff, so those findings had nowhere to go except a
wrong Important or silence.

The volume cap said how many and never which. It now collapses a nit repeated
across files into one finding, prefers a nit in code the pull request wrote
over one in code it only moved, caps pre-existing findings at three, and states
that Important findings are never capped - a section listing two caps otherwise
reads as licence to trim what matters. The review opens with a tally so the
author sees the shape before the detail.

Two contradictions went with it. The file told the reviewer to skip what CI
enforces and then to check that a new i18n key reaches all 13 locales, which
i18n-dead-keys.test.ts pins in both directions - the rule moves to "Do not
report" with the reason. "Anything CI already enforces: npm audit" overstated
what runs; CI audits production dependencies at high and above, so a
dev-dependency advisory is out of scope by design.

The reviewer could not read its own CI. Only postgres-durable-first runs
against PostgreSQL, and XRAY_E2E_BINARY and XUI_SCALE_TEST are set by no job,
so a dialect or migration change can carry a wall of green while the paths it
touches never executed. That belongs to the verification bar, next to the rule
that a behaviour claim needs a file:line citation, and "CI passed" now needs a
run actually read. Also names the two house choices no linter defends: neither
golangci-lint nor oxlint rejects a testify or Tailwind import.

Both kinds of claim rot on a rename, so a test pins them the way
repo-context.md's claims are already pinned - the CI jobs REVIEW.md names must
exist in ci.yml, the skip gates it calls unset must stay unset, and the locale
count must match the directory.

The review itself moves from high to max effort, and the prompt records why it
names REVIEW.md at all: the code-review skill reads CLAUDE.md on its own but
not REVIEW.md, so dropping that clause would silently stop the file applying.
Drops a CLAUDE.md reference to tools/seedperf/, which no longer exists - the
review reads that file as project context, so a stale path there misleads it.
This commit is contained in:
Sanaei
2026-08-21 03:19:33 +02:00
parent 73a971c2d1
commit 5321665d5b
4 changed files with 134 additions and 23 deletions

View File

@@ -153,3 +153,46 @@ func TestBotContextSkipGatesExist(t *testing.T) {
})
}
}
// REVIEW.md tells the reviewer which CI job proves what, and which skip gates
// mean a green run proved nothing. Both go stale silently on a rename.
func TestReviewNamesRealCIJobsAndGates(t *testing.T) {
doc := readRepoFile(t, reviewPath)
ci := readRepoFile(t, ciWorkflowPath)
// Hyphenated only: a single-word job name is indistinguishable from prose.
jobs := regexp.MustCompile("`([a-z0-9]+(?:-[a-z0-9]+)+)`").FindAllStringSubmatch(doc, -1)
if len(jobs) < 2 {
t.Fatalf("expected %s to name at least 2 CI jobs in backticks, found %d", reviewPath, len(jobs))
}
for _, j := range jobs {
t.Run(j[1], func(t *testing.T) {
if !strings.Contains(ci, "\n "+j[1]+":\n") {
t.Errorf("%s names a CI job %q that %s does not define", reviewPath, j[1], ciWorkflowPath)
}
})
}
for _, g := range regexp.MustCompile("`((?:XUI|XRAY)_[A-Z0-9_]+)`").FindAllStringSubmatch(doc, -1) {
t.Run(g[1], func(t *testing.T) {
if strings.Contains(ci, g[1]) {
t.Errorf("%s claims %s is never set in CI, but %s sets it", reviewPath, g[1], ciWorkflowPath)
}
})
}
}
// The i18n rule is the one REVIEW.md states as a number, so it is the one that
// goes wrong silently when a locale is added.
func TestReviewLocaleFileCount(t *testing.T) {
doc := readRepoFile(t, reviewPath)
m := regexp.MustCompile(`(\d+) locale files`).FindStringSubmatch(doc)
if m == nil {
t.Fatalf("%s no longer states the i18n rule as \"N locale files\"", reviewPath)
}
files, err := filepath.Glob("internal/web/translation/*.json")
if err != nil {
t.Fatalf("glob locales: %v", err)
}
if got := len(files); m[1] != itoa(got) {
t.Errorf("%s tells the reviewer to expect %s locale files, internal/web/translation/ holds %d", reviewPath, m[1], got)
}
}