mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-09-22 06:42:19 +03:00
docs(quality): bind operational skills to exact-candidate evidence
This commit is contained in:
@@ -2,6 +2,11 @@
|
||||
"schemaVersion": 1,
|
||||
"scope": "Validation-related npm aliases, not workflow jobs or matrix instances. Profiles list static scans only; no profile here proves release acceptance.",
|
||||
"aliases": [
|
||||
{
|
||||
"name": "check:quality-skill-contract",
|
||||
"command": "node scripts/quality/check-skill-contract.mjs",
|
||||
"disposition": "separately-invoked"
|
||||
},
|
||||
{
|
||||
"name": "check",
|
||||
"command": "npm run lint && npm run test",
|
||||
|
||||
50
config/quality/skill-templates/_shared/base-green.md
Normal file
50
config/quality/skill-templates/_shared/base-green.md
Normal file
@@ -0,0 +1,50 @@
|
||||
# Exact-candidate base health
|
||||
|
||||
An issue is an incident index, not a certificate of the current branch state.
|
||||
No open issue means UNKNOWN until current execution evidence proves otherwise.
|
||||
|
||||
## Resolve the evidence
|
||||
|
||||
1. Resolve the live base SHA and record the timestamp. Resolve the contributor
|
||||
head and combined candidate separately when working on a PR.
|
||||
2. Find the applicable workflow runs for that exact SHA. Read their event,
|
||||
profile, attempt, terminal job results and retained artifacts.
|
||||
3. For the Release-Green observer, use its versioned report verifier when present;
|
||||
compare the recorded validator exit, candidate SHA and event/profile. A quick
|
||||
push receipt covers only quick checks. Even a valid full observer receipt does
|
||||
not replace applicable CI jobs absent from that observer.
|
||||
4. Read the incident body and latest comments for diagnostic history. An issue
|
||||
from an older SHA, a closed issue, an empty search or outer workflow success
|
||||
alone does not establish PASS. If artifacts expired or identity is missing,
|
||||
report UNKNOWN/INCOMPLETE and state the missing evidence.
|
||||
5. Re-resolve the branch before any admission decision. A changed SHA invalidates
|
||||
the previous candidate's acceptance.
|
||||
|
||||
## Causal classification
|
||||
|
||||
- BASE-RED: the same failure signature reproduced on the exact base and candidate
|
||||
under comparable command, runtime and configuration.
|
||||
- PR-RED: candidate fails while its exact base passes the comparable requirement.
|
||||
- INFRA-RED: positive evidence of runner, installation, network or resource failure.
|
||||
- OBSERVER-RED: execution and parser/artifact/aggregate disagree.
|
||||
- DRIFT: a measured policy/budget difference; acceptance follows the actual required
|
||||
gate, never an automatic baseline increase.
|
||||
- FLAKY: controlled repeated runs demonstrate alternating results.
|
||||
- UNKNOWN: evidence is insufficient for the classifications above.
|
||||
|
||||
Queued, pending, in-progress, cancelled and missing results are not PASS.
|
||||
A skipped lane is acceptable only when the applicable policy explicitly excludes
|
||||
it; record that reason. Draft fast paths cannot certify a code candidate.
|
||||
Fork `action_required` means execution awaits approval, not contributor code failure.
|
||||
|
||||
## Recovery and handoff
|
||||
|
||||
Repair confirmed base failures in their own branch/worktree and preserve
|
||||
contributor attribution. Record unverified signatures as hypotheses, not inherited
|
||||
defects. Compare each affected requirement instead of copying an old issue's
|
||||
failure list onto every new PR.
|
||||
|
||||
A base problem may explain a PR's red but does not turn required failure into
|
||||
acceptance. Merge, approval, release, deploy and administrative ruleset changes
|
||||
still require their own authority. No issue closure or baseline update is a
|
||||
substitute for a successful exact-candidate execution.
|
||||
66
config/quality/skill-templates/quality-scan/SKILL.md
Normal file
66
config/quality/skill-templates/quality-scan/SKILL.md
Normal file
@@ -0,0 +1,66 @@
|
||||
---
|
||||
name: quality-scan
|
||||
description: Run a scoped, read-only quality scan and report exact-candidate evidence, failures and frozen debt. Use for an ad-hoc gate sweep or release preparation; a static scan is not full release acceptance.
|
||||
---
|
||||
|
||||
# Quality scan
|
||||
|
||||
## Choose the scope
|
||||
|
||||
Read `../_shared/base-green.md` before interpreting base or PR health.
|
||||
|
||||
- Default: the static `quality-scan` profile.
|
||||
- Fast: `quality-scan-fast`; explicitly report omitted gates.
|
||||
- A named npm alias: that alias only.
|
||||
- Base status: inspect remote evidence; no local suite is implied.
|
||||
- Full release/PR acceptance: inspect the applicable workflow lanes and use
|
||||
`../validate-release-green/SKILL.md` for the local observer's limited scope.
|
||||
|
||||
When present, `config/quality/gate-manifest.json` owns static profile membership
|
||||
and the npm-alias inventory. `package.json` owns commands and runtimes.
|
||||
Use `npm run check:gate-manifest` and the scan's `--list` mode to measure the
|
||||
current inventory. An alias, workflow job, matrix instance and test case are
|
||||
different units. Never substitute a remembered count for this inventory.
|
||||
|
||||
If the candidate predates the manifest, inspect its actual package scripts,
|
||||
aggregator and workflows; label the inventory provisional. Do not invent a
|
||||
missing command or claim manifest enforcement on an older branch.
|
||||
|
||||
## Execute and preserve evidence
|
||||
|
||||
Pin the base and candidate SHAs in an isolated worktree. Run the actual npm
|
||||
entrypoints: direct file invocation can change the required Node/Bun runtime.
|
||||
Record command, cwd, versions, start/end, exit/signal, selected scope and log path.
|
||||
Use the gate process supervisor when the candidate provides it. Otherwise capture
|
||||
the command's own exit before displaying a summary, for example:
|
||||
|
||||
```bash
|
||||
scan_log=$(mktemp)
|
||||
if npm run quality:scan >"$scan_log" 2>&1; then
|
||||
scan_exit=0
|
||||
else
|
||||
scan_exit=$?
|
||||
fi
|
||||
tail -n 30 "$scan_log"
|
||||
printf 'scan_exit=%s log=%s\n' "$scan_exit" "$scan_log"
|
||||
```
|
||||
|
||||
Missing dependencies, timeout, cancellation and unavailable artifacts are
|
||||
incomplete evidence, not PASS. A scan may write logs/reports; it does not update
|
||||
baselines, mutate product code or contact live providers unless explicitly scoped.
|
||||
|
||||
## Report the result
|
||||
|
||||
Report the selected profile and SHA, executed/failed/not-run counts, raw logs and
|
||||
the remaining acceptance lanes. PASS applies only to the executed scope.
|
||||
List frozen debt separately from new findings. A nonzero exit proves a failed
|
||||
execution, not that the contributor introduced it: use the causal comparison in
|
||||
the shared base-green protocol.
|
||||
|
||||
If the tracked skill-contract checker exists, `npm run check:quality-skill-contract`
|
||||
compares these private instructions with the versioned templates. Missing private
|
||||
skills are NOT_INSTALLED, not evidence that their instructions were validated.
|
||||
|
||||
Stop after the report for scan-only requests. Fixes require a change request;
|
||||
baseline changes require their own reviewed justification. Preserve existing
|
||||
thresholds and assertions throughout recovery.
|
||||
@@ -0,0 +1,56 @@
|
||||
---
|
||||
name: validate-release-green
|
||||
description: Run or inspect the local Release-Green observer for an exact base or combined PR candidate, separating hard failures, advisory drift and incomplete evidence. Use for maintainer preflight; full CI acceptance remains a separate requirement.
|
||||
---
|
||||
|
||||
# Validate Release-Green
|
||||
|
||||
Read `../_shared/base-green.md` before interpreting remote status or attributing
|
||||
a failure. Resolve the current base and candidate SHA; use an isolated worktree.
|
||||
The observer validates that working tree, not an abstract branch name.
|
||||
|
||||
## Select and execute the actual profile
|
||||
|
||||
Inspect `scripts/quality/validate-release-green.mjs`, its npm entrypoint and the
|
||||
candidate's workflows before choosing flags. On candidates that provide them:
|
||||
|
||||
- `npm run check:release-green -- --quick`: partial fast diagnosis.
|
||||
- `npm run check:release-green -- --full-ci --with-build --hermetic --json`:
|
||||
the observer's broader static/test/package profile; this is still not a promise
|
||||
of parity with every workflow lane.
|
||||
|
||||
The versioned static gate manifest, if present, describes static scan membership,
|
||||
not all observer or admission jobs. Report unmodeled coverage explicitly.
|
||||
|
||||
Build in a worktree with physical dependencies, never an external node_modules
|
||||
symlink. Preserve stdout/stderr, process receipts and the validator's own exit.
|
||||
Read the report even when the outer job is green. Missing, contradictory, stale,
|
||||
cancelled or timed-out evidence cannot establish PASS.
|
||||
|
||||
## Interpret and compare
|
||||
|
||||
HARD is the observer's blocking classification; DRIFT is its diagnostic bucket.
|
||||
Neither class alone proves causality. Compare the same requirement on the exact
|
||||
base before calling a defect contributor-introduced. A ratchet classified as
|
||||
advisory locally can still block in the applicable CI workflow.
|
||||
|
||||
A green local profile means only that profile passed. Record remaining lanes,
|
||||
including relevant Node/Vitest suites, coverage, security, build, package/boot and
|
||||
external checks. Do not stamp merge-ready or pre-greened based on a focused run,
|
||||
a draft skip, a closed incident or an old head SHA.
|
||||
|
||||
For machine reports, use `scripts/ci/release-green-result.mjs` when available on
|
||||
the candidate. It checks the observed exit, exact SHA and event/profile; supply
|
||||
the values from the run rather than inventing an exit code. The report verifier
|
||||
does not establish that omitted workflow requirements ran.
|
||||
|
||||
## Handoff
|
||||
|
||||
Report SHA, profile, command, versions, raw log/artifact locations, hard failures,
|
||||
drift, incomplete lanes and causal confidence. Re-query live refs before using
|
||||
the result for a later decision. A full-current observer PASS may resolve its
|
||||
own incident only through the authorized workflow; it cannot close unrelated
|
||||
requirements or authorize a merge, baseline change, release or deployment.
|
||||
|
||||
This skill is diagnostic. Implement fixes only when the user requested changes.
|
||||
Keep thresholds, assertions, contributor credit and action boundaries intact.
|
||||
@@ -239,6 +239,7 @@
|
||||
"quality:refresh-model-lifecycle": "node scripts/quality/refresh-model-lifecycle.mjs",
|
||||
"quality:gate": "npm run quality:collect && npm run quality:ratchet -- --allow-missing",
|
||||
"check:gate-manifest": "node scripts/quality/gate-manifest.mjs",
|
||||
"check:quality-skill-contract": "node scripts/quality/check-skill-contract.mjs",
|
||||
"quality:scan": "node scripts/quality/run-all-gates.mjs",
|
||||
"quality:scan:fast": "node scripts/quality/run-all-gates.mjs --fast",
|
||||
"audit:deps": "npm audit --audit-level=critical && (npm audit --audit-level=high || echo '::warning::high-severity advisories present (non-blocking)') && npm run audit:electron",
|
||||
|
||||
38
scripts/quality/check-skill-contract.mjs
Normal file
38
scripts/quality/check-skill-contract.mjs
Normal file
@@ -0,0 +1,38 @@
|
||||
#!/usr/bin/env node
|
||||
import { existsSync, readFileSync } from "node:fs";
|
||||
import { resolve } from "node:path";
|
||||
import { fileURLToPath, pathToFileURL } from "node:url";
|
||||
|
||||
const TEMPLATES = fileURLToPath(new URL("../../config/quality/skill-templates/", import.meta.url));
|
||||
const CONTRACTS = [
|
||||
"quality-scan/SKILL.md",
|
||||
"validate-release-green/SKILL.md",
|
||||
"_shared/base-green.md",
|
||||
];
|
||||
|
||||
export function inspectSkillContracts(root, templates = TEMPLATES) {
|
||||
if (!existsSync(root)) return { ok: false, status: "NOT_INSTALLED", files: [] };
|
||||
const files = CONTRACTS.map((path) => {
|
||||
const installed = resolve(root, path);
|
||||
const expected = readFileSync(resolve(templates, path), "utf8");
|
||||
const status = !existsSync(installed)
|
||||
? "MISSING"
|
||||
: readFileSync(installed, "utf8") === expected
|
||||
? "MATCH"
|
||||
: "DRIFT";
|
||||
return { path, status };
|
||||
});
|
||||
const ok = files.every((file) => file.status === "MATCH");
|
||||
return { ok, status: ok ? "MATCH" : "INCOMPLETE", files };
|
||||
}
|
||||
|
||||
if (import.meta.url === pathToFileURL(process.argv[1] || "").href) {
|
||||
try {
|
||||
const result = inspectSkillContracts(process.argv[2] || resolve(".agents/skills"));
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
process.exitCode = result.ok ? 0 : 1;
|
||||
} catch (error) {
|
||||
console.error(`[quality-skill-contract] ${error.message}`);
|
||||
process.exitCode = 1;
|
||||
}
|
||||
}
|
||||
32
tests/unit/build/quality-skill-contract.test.ts
Normal file
32
tests/unit/build/quality-skill-contract.test.ts
Normal file
@@ -0,0 +1,32 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { cpSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import test from "node:test";
|
||||
import { inspectSkillContracts } from "../../../scripts/quality/check-skill-contract.mjs";
|
||||
|
||||
test("the skill contract distinguishes installed, drifted and missing instructions", () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), "omniroute-skill-contract-"));
|
||||
const root = join(dir, "skills");
|
||||
try {
|
||||
const absent = inspectSkillContracts(root);
|
||||
assert.equal(absent.ok, false);
|
||||
assert.equal(absent.status, "NOT_INSTALLED");
|
||||
cpSync("config/quality/skill-templates", root, { recursive: true });
|
||||
assert.equal(inspectSkillContracts(root).ok, true);
|
||||
const file = join(root, "quality-scan/SKILL.md");
|
||||
writeFileSync(file, readFileSync(file, "utf8") + "\nUnreviewed local instruction.\n");
|
||||
const drifted = inspectSkillContracts(root);
|
||||
assert.equal(drifted.ok, false);
|
||||
assert.deepEqual(
|
||||
drifted.files.filter((f) => f.status === "DRIFT").map((f) => f.path),
|
||||
["quality-scan/SKILL.md"]
|
||||
);
|
||||
rmSync(file);
|
||||
const missing = inspectSkillContracts(root);
|
||||
assert.equal(missing.ok, false);
|
||||
assert.equal(missing.files.find((f) => f.path === "quality-scan/SKILL.md")?.status, "MISSING");
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
Reference in New Issue
Block a user