docs(quality): bind operational skills to exact-candidate evidence

This commit is contained in:
diegosouzapw
2026-09-21 20:17:02 -03:00
parent 5e41262117
commit 87b53673e4
7 changed files with 248 additions and 0 deletions

View File

@@ -2,6 +2,11 @@
"schemaVersion": 1,
"scope": "Validation-related npm aliases, not workflow jobs or matrix instances. Profiles list static scans only; no profile here proves release acceptance.",
"aliases": [
{
"name": "check:quality-skill-contract",
"command": "node scripts/quality/check-skill-contract.mjs",
"disposition": "separately-invoked"
},
{
"name": "check",
"command": "npm run lint && npm run test",

View File

@@ -0,0 +1,50 @@
# Exact-candidate base health
An issue is an incident index, not a certificate of the current branch state.
No open issue means UNKNOWN until current execution evidence proves otherwise.
## Resolve the evidence
1. Resolve the live base SHA and record the timestamp. Resolve the contributor
head and combined candidate separately when working on a PR.
2. Find the applicable workflow runs for that exact SHA. Read their event,
profile, attempt, terminal job results and retained artifacts.
3. For the Release-Green observer, use its versioned report verifier when present;
compare the recorded validator exit, candidate SHA and event/profile. A quick
push receipt covers only quick checks. Even a valid full observer receipt does
not replace applicable CI jobs absent from that observer.
4. Read the incident body and latest comments for diagnostic history. An issue
from an older SHA, a closed issue, an empty search or outer workflow success
alone does not establish PASS. If artifacts expired or identity is missing,
report UNKNOWN/INCOMPLETE and state the missing evidence.
5. Re-resolve the branch before any admission decision. A changed SHA invalidates
the previous candidate's acceptance.
## Causal classification
- BASE-RED: the same failure signature reproduced on the exact base and candidate
under comparable command, runtime and configuration.
- PR-RED: candidate fails while its exact base passes the comparable requirement.
- INFRA-RED: positive evidence of runner, installation, network or resource failure.
- OBSERVER-RED: execution and parser/artifact/aggregate disagree.
- DRIFT: a measured policy/budget difference; acceptance follows the actual required
gate, never an automatic baseline increase.
- FLAKY: controlled repeated runs demonstrate alternating results.
- UNKNOWN: evidence is insufficient for the classifications above.
Queued, pending, in-progress, cancelled and missing results are not PASS.
A skipped lane is acceptable only when the applicable policy explicitly excludes
it; record that reason. Draft fast paths cannot certify a code candidate.
Fork `action_required` means execution awaits approval, not contributor code failure.
## Recovery and handoff
Repair confirmed base failures in their own branch/worktree and preserve
contributor attribution. Record unverified signatures as hypotheses, not inherited
defects. Compare each affected requirement instead of copying an old issue's
failure list onto every new PR.
A base problem may explain a PR's red but does not turn required failure into
acceptance. Merge, approval, release, deploy and administrative ruleset changes
still require their own authority. No issue closure or baseline update is a
substitute for a successful exact-candidate execution.

View File

@@ -0,0 +1,66 @@
---
name: quality-scan
description: Run a scoped, read-only quality scan and report exact-candidate evidence, failures and frozen debt. Use for an ad-hoc gate sweep or release preparation; a static scan is not full release acceptance.
---
# Quality scan
## Choose the scope
Read `../_shared/base-green.md` before interpreting base or PR health.
- Default: the static `quality-scan` profile.
- Fast: `quality-scan-fast`; explicitly report omitted gates.
- A named npm alias: that alias only.
- Base status: inspect remote evidence; no local suite is implied.
- Full release/PR acceptance: inspect the applicable workflow lanes and use
`../validate-release-green/SKILL.md` for the local observer's limited scope.
When present, `config/quality/gate-manifest.json` owns static profile membership
and the npm-alias inventory. `package.json` owns commands and runtimes.
Use `npm run check:gate-manifest` and the scan's `--list` mode to measure the
current inventory. An alias, workflow job, matrix instance and test case are
different units. Never substitute a remembered count for this inventory.
If the candidate predates the manifest, inspect its actual package scripts,
aggregator and workflows; label the inventory provisional. Do not invent a
missing command or claim manifest enforcement on an older branch.
## Execute and preserve evidence
Pin the base and candidate SHAs in an isolated worktree. Run the actual npm
entrypoints: direct file invocation can change the required Node/Bun runtime.
Record command, cwd, versions, start/end, exit/signal, selected scope and log path.
Use the gate process supervisor when the candidate provides it. Otherwise capture
the command's own exit before displaying a summary, for example:
```bash
scan_log=$(mktemp)
if npm run quality:scan >"$scan_log" 2>&1; then
scan_exit=0
else
scan_exit=$?
fi
tail -n 30 "$scan_log"
printf 'scan_exit=%s log=%s\n' "$scan_exit" "$scan_log"
```
Missing dependencies, timeout, cancellation and unavailable artifacts are
incomplete evidence, not PASS. A scan may write logs/reports; it does not update
baselines, mutate product code or contact live providers unless explicitly scoped.
## Report the result
Report the selected profile and SHA, executed/failed/not-run counts, raw logs and
the remaining acceptance lanes. PASS applies only to the executed scope.
List frozen debt separately from new findings. A nonzero exit proves a failed
execution, not that the contributor introduced it: use the causal comparison in
the shared base-green protocol.
If the tracked skill-contract checker exists, `npm run check:quality-skill-contract`
compares these private instructions with the versioned templates. Missing private
skills are NOT_INSTALLED, not evidence that their instructions were validated.
Stop after the report for scan-only requests. Fixes require a change request;
baseline changes require their own reviewed justification. Preserve existing
thresholds and assertions throughout recovery.

View File

@@ -0,0 +1,56 @@
---
name: validate-release-green
description: Run or inspect the local Release-Green observer for an exact base or combined PR candidate, separating hard failures, advisory drift and incomplete evidence. Use for maintainer preflight; full CI acceptance remains a separate requirement.
---
# Validate Release-Green
Read `../_shared/base-green.md` before interpreting remote status or attributing
a failure. Resolve the current base and candidate SHA; use an isolated worktree.
The observer validates that working tree, not an abstract branch name.
## Select and execute the actual profile
Inspect `scripts/quality/validate-release-green.mjs`, its npm entrypoint and the
candidate's workflows before choosing flags. On candidates that provide them:
- `npm run check:release-green -- --quick`: partial fast diagnosis.
- `npm run check:release-green -- --full-ci --with-build --hermetic --json`:
the observer's broader static/test/package profile; this is still not a promise
of parity with every workflow lane.
The versioned static gate manifest, if present, describes static scan membership,
not all observer or admission jobs. Report unmodeled coverage explicitly.
Build in a worktree with physical dependencies, never an external node_modules
symlink. Preserve stdout/stderr, process receipts and the validator's own exit.
Read the report even when the outer job is green. Missing, contradictory, stale,
cancelled or timed-out evidence cannot establish PASS.
## Interpret and compare
HARD is the observer's blocking classification; DRIFT is its diagnostic bucket.
Neither class alone proves causality. Compare the same requirement on the exact
base before calling a defect contributor-introduced. A ratchet classified as
advisory locally can still block in the applicable CI workflow.
A green local profile means only that profile passed. Record remaining lanes,
including relevant Node/Vitest suites, coverage, security, build, package/boot and
external checks. Do not stamp merge-ready or pre-greened based on a focused run,
a draft skip, a closed incident or an old head SHA.
For machine reports, use `scripts/ci/release-green-result.mjs` when available on
the candidate. It checks the observed exit, exact SHA and event/profile; supply
the values from the run rather than inventing an exit code. The report verifier
does not establish that omitted workflow requirements ran.
## Handoff
Report SHA, profile, command, versions, raw log/artifact locations, hard failures,
drift, incomplete lanes and causal confidence. Re-query live refs before using
the result for a later decision. A full-current observer PASS may resolve its
own incident only through the authorized workflow; it cannot close unrelated
requirements or authorize a merge, baseline change, release or deployment.
This skill is diagnostic. Implement fixes only when the user requested changes.
Keep thresholds, assertions, contributor credit and action boundaries intact.

View File

@@ -239,6 +239,7 @@
"quality:refresh-model-lifecycle": "node scripts/quality/refresh-model-lifecycle.mjs",
"quality:gate": "npm run quality:collect && npm run quality:ratchet -- --allow-missing",
"check:gate-manifest": "node scripts/quality/gate-manifest.mjs",
"check:quality-skill-contract": "node scripts/quality/check-skill-contract.mjs",
"quality:scan": "node scripts/quality/run-all-gates.mjs",
"quality:scan:fast": "node scripts/quality/run-all-gates.mjs --fast",
"audit:deps": "npm audit --audit-level=critical && (npm audit --audit-level=high || echo '::warning::high-severity advisories present (non-blocking)') && npm run audit:electron",

View File

@@ -0,0 +1,38 @@
#!/usr/bin/env node
import { existsSync, readFileSync } from "node:fs";
import { resolve } from "node:path";
import { fileURLToPath, pathToFileURL } from "node:url";
const TEMPLATES = fileURLToPath(new URL("../../config/quality/skill-templates/", import.meta.url));
const CONTRACTS = [
"quality-scan/SKILL.md",
"validate-release-green/SKILL.md",
"_shared/base-green.md",
];
export function inspectSkillContracts(root, templates = TEMPLATES) {
if (!existsSync(root)) return { ok: false, status: "NOT_INSTALLED", files: [] };
const files = CONTRACTS.map((path) => {
const installed = resolve(root, path);
const expected = readFileSync(resolve(templates, path), "utf8");
const status = !existsSync(installed)
? "MISSING"
: readFileSync(installed, "utf8") === expected
? "MATCH"
: "DRIFT";
return { path, status };
});
const ok = files.every((file) => file.status === "MATCH");
return { ok, status: ok ? "MATCH" : "INCOMPLETE", files };
}
if (import.meta.url === pathToFileURL(process.argv[1] || "").href) {
try {
const result = inspectSkillContracts(process.argv[2] || resolve(".agents/skills"));
console.log(JSON.stringify(result, null, 2));
process.exitCode = result.ok ? 0 : 1;
} catch (error) {
console.error(`[quality-skill-contract] ${error.message}`);
process.exitCode = 1;
}
}

View File

@@ -0,0 +1,32 @@
import assert from "node:assert/strict";
import { cpSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import test from "node:test";
import { inspectSkillContracts } from "../../../scripts/quality/check-skill-contract.mjs";
test("the skill contract distinguishes installed, drifted and missing instructions", () => {
const dir = mkdtempSync(join(tmpdir(), "omniroute-skill-contract-"));
const root = join(dir, "skills");
try {
const absent = inspectSkillContracts(root);
assert.equal(absent.ok, false);
assert.equal(absent.status, "NOT_INSTALLED");
cpSync("config/quality/skill-templates", root, { recursive: true });
assert.equal(inspectSkillContracts(root).ok, true);
const file = join(root, "quality-scan/SKILL.md");
writeFileSync(file, readFileSync(file, "utf8") + "\nUnreviewed local instruction.\n");
const drifted = inspectSkillContracts(root);
assert.equal(drifted.ok, false);
assert.deepEqual(
drifted.files.filter((f) => f.status === "DRIFT").map((f) => f.path),
["quality-scan/SKILL.md"]
);
rmSync(file);
const missing = inspectSkillContracts(root);
assert.equal(missing.ok, false);
assert.equal(missing.files.find((f) => f.path === "quality-scan/SKILL.md")?.status, "MISSING");
} finally {
rmSync(dir, { recursive: true, force: true });
}
});