From 434978edc6cbd475151b818bbeb8a92ca5db0e7b Mon Sep 17 00:00:00 2001 From: Diego Rodrigues de Sa e Souza Date: Tue, 14 Jul 2026 01:13:54 -0300 Subject: [PATCH] docs(homolog): guia de operacao da suite + fragment de changelog + allowlist env-doc-sync --- changelog.d/features/homolog-e2e-suite.md | 1 + docs/ops/HOMOLOGATION.md | 104 ++++++++++++++++++++++ scripts/check/check-env-doc-sync.mjs | 9 ++ 3 files changed, 114 insertions(+) create mode 100644 changelog.d/features/homolog-e2e-suite.md create mode 100644 docs/ops/HOMOLOGATION.md diff --git a/changelog.d/features/homolog-e2e-suite.md b/changelog.d/features/homolog-e2e-suite.md new file mode 100644 index 0000000000..b2c4637d66 --- /dev/null +++ b/changelog.d/features/homolog-e2e-suite.md @@ -0,0 +1 @@ +- **Homologation suite**: new `npm run homolog` runs the full release-homologation battery against the deployed VPS — health/version parity, API + real SSE streaming with an ephemeral API key (created and revoked by the run), minimal-cost real-provider smoke (promptfoo generated from the live catalog), and a Playwright sweep that loads every dashboard route and exercises the API-key UI flow — emitting a unified CTRF report that backs the release STOP #2 checklist diff --git a/docs/ops/HOMOLOGATION.md b/docs/ops/HOMOLOGATION.md new file mode 100644 index 0000000000..2754d12956 --- /dev/null +++ b/docs/ops/HOMOLOGATION.md @@ -0,0 +1,104 @@ +--- +title: "Homologation Suite (npm run homolog)" +version: 3.8.49 +lastUpdated: 2026-07-14 +--- + +# Homologation Suite (`npm run homolog`) + +Real-environment E2E validation of the OmniRoute deploy running on the homologation VPS +(`HOMOLOG_BASE_URL`, e.g. `http://192.168.0.15:20128`). One command replaces the manual +release STOP #2 checklist with an automated, evidence-producing run. + +## What it covers + +| Layer | What it checks | Implementation | +| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | +| L0 — health/parity | `/api/monitoring/health` responds `200` with `status: "healthy"` and the expected version | `scripts/homolog/lib/parity.mjs` | +| L1a — ephemeral key | Admin login → `POST /api/keys` creates a scoped API key for the run, revoked (`DELETE /api/keys/:id`) in a `finally` block regardless of outcome | `scripts/homolog/lib/adminClient.mjs` | +| L1b — API surface | `/v1/models` catalog, a real non-streaming chat completion (tier-critical model, `max_tokens: 5`), an invalid-key `401`, and public `/api/monitoring/health` | `tests/homolog/api/core.http` (httpYac) | +| L1c — SSE streaming | Real streaming chat completion; asserts `text/event-stream`, at least one content delta, and a `[DONE]` terminator | `scripts/homolog/lib/sseCheck.mjs` | +| L2 — real providers | One minimal-cost chat request per critical provider present in the live `/v1/models` catalog, generated on the fly via promptfoo | `scripts/homolog/gen-promptfoo.mjs` + `scripts/homolog/lib/providerTiers.mjs` | +| L4a — UI auth | Logs in once via the real login form and reuses the session (`storageState`) across the UI layer | `tests/homolog/ui/auth.setup.ts` | +| L4b — UI routes | Every static `page.tsx` under `src/app/(dashboard)/dashboard` (discovered from the filesystem, dynamic `[param]` routes skipped) loads without an HTTP error, a page error, or the Next.js error boundary | `tests/homolog/ui/routes.spec.ts` | +| L4c — UI critical flow | Creates an API key through the dashboard UI and revokes it again (leaves no residue on the VPS) | `tests/homolog/ui/api-key-flow.spec.ts` | +| L5 — unified report | Merges httpYac (via `junit-to-ctrf`), the promptfoo→CTRF adapter, and the Playwright CTRF reporter into one `homolog-ctrf.json`, plus a human-readable `homolog-report/summary.md` | `scripts/homolog/run.mjs` | + +Zero LLM involvement in the replay itself — this is a deterministic regression battery, +not an eval. AI only enters in future maintenance work (see Roadmap below). + +## Prerequisites + +1. Copy `.env.homolog.example` to `.env.homolog` (gitignored — never commit it) and fill in: + - `HOMOLOG_BASE_URL` — the target deploy, e.g. `http://192.168.0.15:20128`. + - `HOMOLOG_ADMIN_PASSWORD` — the dashboard management password for that deploy. + - `HOMOLOG_CRITICAL_PROVIDERS` — comma-separated provider prefixes that get a real + smoke chat request (e.g. `openai,anthropic,gemini,codex,grok,glm,deepseek,openrouter`). + - `HOMOLOG_API_KEY` — leave empty in normal runs; the suite creates and revokes its + own ephemeral key. Only set this to debug a single layer in isolation. +2. `npm install` in the repo (the suite's dependencies — `httpyac`, `promptfoo`, + `playwright-ctrf-json-reporter`, `junit-to-ctrf`, `ctrf` — are regular devDependencies). +3. `npx playwright install` if the browser binaries are not already present. + +## How to run + +```bash +npm run homolog +``` + +To validate against a deploy whose version does not match the local `package.json` +(e.g. a homologation box still on a previous patch release), override the expected +version explicitly: + +```bash +HOMOLOG_EXPECT_VERSION=3.8.47 npm run homolog +``` + +The run exits non-zero if any layer fails, and always attempts to revoke the ephemeral +API key it created, even on failure (`finally` block in `scripts/homolog/run.mjs`). + +## Reading the report + +All output lands in `homolog-report/` (gitignored): + +- `summary.md` — the same table printed to stdout, one row per layer (✅/❌ + detail). +- `homolog-ctrf.json` — the unified CTRF report (merge of API/SSE, provider-smoke, and + UI results) — this is the artifact to attach to a release STOP #2 checklist. +- `httpyac-junit.xml`, `api-ctrf.json`, `providers-ctrf.json`, `ui-ctrf.json` — the + per-layer raw/intermediate reports. +- `promptfooconfig.yaml`, `provider-misses.json` — the generated promptfoo config for + the current run and any critical providers that were missing from the live catalog. + +A failing L0 aborts immediately (no ephemeral key is created) since a version/health +mismatch means every downstream layer would be validating the wrong deploy. + +## Re-baselining when the UI changes legitimately + +L4b (route smoke) and L4c (API-key UI flow) are driven by real DOM locators, not +snapshots, so most legitimate UI changes do not require any suite update. When a change +does break a locator (e.g. a renamed button label or a moved settings page): + +1. Re-confirm the locator against the current source (the specs already document which + file/line each locator was confirmed against — follow the same pattern, don't guess). +2. Update the spec in `tests/homolog/ui/`. +3. Re-run `npm run homolog` (or just the affected Playwright spec) against the VPS to + confirm the fix, then commit. + +There is no visual/pixel baseline in this suite (F1) — see Roadmap for that. + +## Roadmap (F2 / F3) + +Design and phased rollout live in the internal planning spec +`_tasks/superpowers/specs/2026-07-13-homolog-e2e-suite-design.md` (not linked — internal +`_tasks/` artifact, not part of this repo's tracked docs). Summary: + +- **F2** — full walkthrough recording → Playwright Test Agents (`planner`/`generator`) + turn it into flow specs (create combo, test provider, edit settings, MCP tools) + + visual regression baseline (Lost Pixel) with masks over dynamic data (metrics, + timestamps, logs) + a `healer` maintenance routine per release. +- **F3** — resilience/contract/wiring coverage: toxiproxy + a fake OpenAI-compatible + provider on the devbox, a `homolog-resilience` combo on the VPS pointed at it + (injected timeout → assert fallback + circuit breaker open/close via + `/api/monitoring/health`); gated Schemathesis contract testing against + `docs/openapi.yaml` (low `--max-examples`, fixed seeds, non-LLM endpoints only); and + wiring `npm run homolog` + its `summary.md` into the `/generate-release` STOP #2 phase. diff --git a/scripts/check/check-env-doc-sync.mjs b/scripts/check/check-env-doc-sync.mjs index 197d8ddeb9..dbec1b79c7 100644 --- a/scripts/check/check-env-doc-sync.mjs +++ b/scripts/check/check-env-doc-sync.mjs @@ -101,6 +101,15 @@ const IGNORE_FROM_CODE = new Set([ // ("http://192.168.0.15:20128" / null), never OmniRoute runtime config (#5151). "COMBO_LIVE_BASE_URL", "COMBO_LIVE_API_KEY", + // Homologation E2E suite (npm run homolog) vars — configured via the dedicated + // .env.homolog file (template: .env.homolog.example), never in the runtime .env. + // Test/ops-only signals against the homologation VPS, same class as COMBO_LIVE_*. + // See docs/ops/HOMOLOGATION.md. + "HOMOLOG_BASE_URL", + "HOMOLOG_ADMIN_PASSWORD", + "HOMOLOG_API_KEY", + "HOMOLOG_CRITICAL_PROVIDERS", + "HOMOLOG_EXPECT_VERSION", // update-notifier opt-out for the CLI binary. "OMNIROUTE_NO_UPDATE_NOTIFIER", // Headless CLI execution flag for Electron.