mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-07-31 04:12:10 +03:00
c97d2a6ae2
* feat(homolog): scaffolding da suíte de homologação E2E (deps + npm run homolog) * feat(homolog): L0 avaliador de paridade de deploy (TDD) * feat(homolog): L1a ciclo de vida de API key efêmera (login admin -> create -> revoke) * feat(homolog): L1b suite httpYac de API (models, chat, auth de management, health) * feat(homolog): L1c checker SSE de streaming real (TDD no parser) * feat(homolog): L2 smoke de providers reais via promptfoo gerado do catálogo * feat(homolog): L4a Playwright homolog config + login storageState * feat(homolog): L4b smoke de todas as rotas do dashboard (descoberta via fs) * feat(homolog): L4c fluxo criar/revogar API key pela UI * fix(homolog): resiliencia real-environment — stream:false no smoke promptfoo, retry de socket keep-alive, key efemera com sufixo unico * feat(homolog): L5 orquestrador npm run homolog + relatorio CTRF unificado * docs(homolog): guia de operacao da suite + fragment de changelog + allowlist env-doc-sync * fix(homolog): paraleliza o sweep de rotas do dashboard (fullyParallel + 8 workers) * fix(homolog): isola outputs crus em homolog-report/raw para nao quebrar o ctrf merge * fix(homolog): outputDir absoluto do reporter CTRF da UI (path relativo escapava do worktree) * chore(quality): allowlist the 5 homolog-suite devDependencies (ctrf-io trio, httpyac, promptfoo) after registry verification * chore(quality): register the homolog Playwright suite as a test-discovery collector (run.mjs -> tests/homolog/ui)
105 lines
7.7 KiB
Markdown
105 lines
7.7 KiB
Markdown
---
|
|
title: "Homologation Suite (npm run homolog)"
|
|
version: 3.8.49
|
|
lastUpdated: 2026-07-14
|
|
---
|
|
|
|
# Homologation Suite (`npm run homolog`)
|
|
|
|
Real-environment E2E validation of the OmniRoute deploy running on the homologation VPS
|
|
(`HOMOLOG_BASE_URL`, e.g. `http://192.168.0.15:20128`). One command replaces the manual
|
|
release STOP #2 checklist with an automated, evidence-producing run.
|
|
|
|
## What it covers
|
|
|
|
| Layer | What it checks | Implementation |
|
|
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- |
|
|
| L0 — health/parity | `/api/monitoring/health` responds `200` with `status: "healthy"` and the expected version | `scripts/homolog/lib/parity.mjs` |
|
|
| L1a — ephemeral key | Admin login → `POST /api/keys` creates a scoped API key for the run, revoked (`DELETE /api/keys/:id`) in a `finally` block regardless of outcome | `scripts/homolog/lib/adminClient.mjs` |
|
|
| L1b — API surface | `/v1/models` catalog, a real non-streaming chat completion (tier-critical model, `max_tokens: 5`), an invalid-key `401`, and public `/api/monitoring/health` | `tests/homolog/api/core.http` (httpYac) |
|
|
| L1c — SSE streaming | Real streaming chat completion; asserts `text/event-stream`, at least one content delta, and a `[DONE]` terminator | `scripts/homolog/lib/sseCheck.mjs` |
|
|
| L2 — real providers | One minimal-cost chat request per critical provider present in the live `/v1/models` catalog, generated on the fly via promptfoo | `scripts/homolog/gen-promptfoo.mjs` + `scripts/homolog/lib/providerTiers.mjs` |
|
|
| L4a — UI auth | Logs in once via the real login form and reuses the session (`storageState`) across the UI layer | `tests/homolog/ui/auth.setup.ts` |
|
|
| L4b — UI routes | Every static `page.tsx` under `src/app/(dashboard)/dashboard` (discovered from the filesystem, dynamic `[param]` routes skipped) loads without an HTTP error, a page error, or the Next.js error boundary | `tests/homolog/ui/routes.spec.ts` |
|
|
| L4c — UI critical flow | Creates an API key through the dashboard UI and revokes it again (leaves no residue on the VPS) | `tests/homolog/ui/api-key-flow.spec.ts` |
|
|
| L5 — unified report | Merges httpYac (via `junit-to-ctrf`), the promptfoo→CTRF adapter, and the Playwright CTRF reporter into one `homolog-ctrf.json`, plus a human-readable `homolog-report/summary.md` | `scripts/homolog/run.mjs` |
|
|
|
|
Zero LLM involvement in the replay itself — this is a deterministic regression battery,
|
|
not an eval. AI only enters in future maintenance work (see Roadmap below).
|
|
|
|
## Prerequisites
|
|
|
|
1. Copy `.env.homolog.example` to `.env.homolog` (gitignored — never commit it) and fill in:
|
|
- `HOMOLOG_BASE_URL` — the target deploy, e.g. `http://192.168.0.15:20128`.
|
|
- `HOMOLOG_ADMIN_PASSWORD` — the dashboard management password for that deploy.
|
|
- `HOMOLOG_CRITICAL_PROVIDERS` — comma-separated provider prefixes that get a real
|
|
smoke chat request (e.g. `openai,anthropic,gemini,codex,grok,glm,deepseek,openrouter`).
|
|
- `HOMOLOG_API_KEY` — leave empty in normal runs; the suite creates and revokes its
|
|
own ephemeral key. Only set this to debug a single layer in isolation.
|
|
2. `npm install` in the repo (the suite's dependencies — `httpyac`, `promptfoo`,
|
|
`playwright-ctrf-json-reporter`, `junit-to-ctrf`, `ctrf` — are regular devDependencies).
|
|
3. `npx playwright install` if the browser binaries are not already present.
|
|
|
|
## How to run
|
|
|
|
```bash
|
|
npm run homolog
|
|
```
|
|
|
|
To validate against a deploy whose version does not match the local `package.json`
|
|
(e.g. a homologation box still on a previous patch release), override the expected
|
|
version explicitly:
|
|
|
|
```bash
|
|
HOMOLOG_EXPECT_VERSION=3.8.47 npm run homolog
|
|
```
|
|
|
|
The run exits non-zero if any layer fails, and always attempts to revoke the ephemeral
|
|
API key it created, even on failure (`finally` block in `scripts/homolog/run.mjs`).
|
|
|
|
## Reading the report
|
|
|
|
All output lands in `homolog-report/` (gitignored):
|
|
|
|
- `summary.md` — the same table printed to stdout, one row per layer (✅/❌ + detail).
|
|
- `homolog-ctrf.json` — the unified CTRF report (merge of API/SSE, provider-smoke, and
|
|
UI results) — this is the artifact to attach to a release STOP #2 checklist.
|
|
- `httpyac-junit.xml`, `api-ctrf.json`, `providers-ctrf.json`, `ui-ctrf.json` — the
|
|
per-layer raw/intermediate reports.
|
|
- `promptfooconfig.yaml`, `provider-misses.json` — the generated promptfoo config for
|
|
the current run and any critical providers that were missing from the live catalog.
|
|
|
|
A failing L0 aborts immediately (no ephemeral key is created) since a version/health
|
|
mismatch means every downstream layer would be validating the wrong deploy.
|
|
|
|
## Re-baselining when the UI changes legitimately
|
|
|
|
L4b (route smoke) and L4c (API-key UI flow) are driven by real DOM locators, not
|
|
snapshots, so most legitimate UI changes do not require any suite update. When a change
|
|
does break a locator (e.g. a renamed button label or a moved settings page):
|
|
|
|
1. Re-confirm the locator against the current source (the specs already document which
|
|
file/line each locator was confirmed against — follow the same pattern, don't guess).
|
|
2. Update the spec in `tests/homolog/ui/`.
|
|
3. Re-run `npm run homolog` (or just the affected Playwright spec) against the VPS to
|
|
confirm the fix, then commit.
|
|
|
|
There is no visual/pixel baseline in this suite (F1) — see Roadmap for that.
|
|
|
|
## Roadmap (F2 / F3)
|
|
|
|
Design and phased rollout live in the internal planning spec
|
|
`_tasks/superpowers/specs/2026-07-13-homolog-e2e-suite-design.md` (not linked — internal
|
|
`_tasks/` artifact, not part of this repo's tracked docs). Summary:
|
|
|
|
- **F2** — full walkthrough recording → Playwright Test Agents (`planner`/`generator`)
|
|
turn it into flow specs (create combo, test provider, edit settings, MCP tools) +
|
|
visual regression baseline (Lost Pixel) with masks over dynamic data (metrics,
|
|
timestamps, logs) + a `healer` maintenance routine per release.
|
|
- **F3** — resilience/contract/wiring coverage: toxiproxy + a fake OpenAI-compatible
|
|
provider on the devbox, a `homolog-resilience` combo on the VPS pointed at it
|
|
(injected timeout → assert fallback + circuit breaker open/close via
|
|
`/api/monitoring/health`); gated Schemathesis contract testing against
|
|
`docs/openapi.yaml` (low `--max-examples`, fixed seeds, non-LLM endpoints only); and
|
|
wiring `npm run homolog` + its `summary.md` into the `/generate-release` STOP #2 phase.
|