pjandro/OmniRoute
1
0
Fork 0
mirror of https://github.com/diegosouzapw/OmniRoute.git synced 2026-07-26 09:52:11 +03:00
Code Issues Packages Projects Releases Wiki Activity

docs(wiki): auto-sync pages + cover counts with docs

github-actions[bot]
2026-07-17 06:04:12 +00:00
parent 9c5a6ca58c
commit 5b858f3333
3 changed files with 197 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

101
Homologation.md Normal file

@@ -0,0 +1,101 @@
> 🌍 [View in other languages](Languages)
# Homologation Suite (`npm run homolog`)
Real-environment E2E validation of the OmniRoute deploy running on the homologation VPS
(`HOMOLOG_BASE_URL`, e.g. `http://192.168.0.15:20128`). One command replaces the manual
release STOP #2 checklist with an automated, evidence-producing run.
## What it covers
| Layer | What it checks | Implementation |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- |
| L0 — health/parity | `/api/monitoring/health` responds `200` with `status: "healthy"` and the expected version | `scripts/homolog/lib/parity.mjs` |
| L1a — ephemeral key | Admin login → `POST /api/keys` creates a scoped API key for the run, revoked (`DELETE /api/keys/:id`) in a `finally` block regardless of outcome | `scripts/homolog/lib/adminClient.mjs` |
| L1b — API surface | `/v1/models` catalog, a real non-streaming chat completion (tier-critical model, `max_tokens: 5`), an invalid-key `401`, and public `/api/monitoring/health` | `tests/homolog/api/core.http` (httpYac) |
| L1c — SSE streaming | Real streaming chat completion; asserts `text/event-stream`, at least one content delta, and a `[DONE]` terminator | `scripts/homolog/lib/sseCheck.mjs` |
| L2 — real providers | One minimal-cost chat request per critical provider present in the live `/v1/models` catalog, generated on the fly via promptfoo | `scripts/homolog/gen-promptfoo.mjs` + `scripts/homolog/lib/providerTiers.mjs` |
| L4a — UI auth | Logs in once via the real login form and reuses the session (`storageState`) across the UI layer | `tests/homolog/ui/auth.setup.ts` |
| L4b — UI routes | Every static `page.tsx` under `src/app/(dashboard)/dashboard` (discovered from the filesystem, dynamic `[param]` routes skipped) loads without an HTTP error, a page error, or the Next.js error boundary | `tests/homolog/ui/routes.spec.ts` |
| L4c — UI critical flow | Creates an API key through the dashboard UI and revokes it again (leaves no residue on the VPS) | `tests/homolog/ui/api-key-flow.spec.ts` |
| L5 — unified report | Merges httpYac (via `junit-to-ctrf`), the promptfoo→CTRF adapter, and the Playwright CTRF reporter into one `homolog-ctrf.json`, plus a human-readable `homolog-report/summary.md` | `scripts/homolog/run.mjs` |
Zero LLM involvement in the replay itself — this is a deterministic regression battery,
not an eval. AI only enters in future maintenance work (see Roadmap below).
## Prerequisites
1. Copy `.env.homolog.example` to `.env.homolog` (gitignored — never commit it) and fill in:
- `HOMOLOG_BASE_URL` — the target deploy, e.g. `http://192.168.0.15:20128`.
- `HOMOLOG_ADMIN_PASSWORD` — the dashboard management password for that deploy.
- `HOMOLOG_CRITICAL_PROVIDERS` — comma-separated provider prefixes that get a real
smoke chat request (e.g. `openai,anthropic,gemini,codex,grok,glm,deepseek,openrouter`).
- `HOMOLOG_API_KEY` — leave empty in normal runs; the suite creates and revokes its
own ephemeral key. Only set this to debug a single layer in isolation.
2. `npm install` in the repo (the suite's dependencies — `httpyac`, `promptfoo`,
`playwright-ctrf-json-reporter`, `junit-to-ctrf`, `ctrf` — are regular devDependencies).
3. `npx playwright install` if the browser binaries are not already present.
## How to run
```bash
npm run homolog
```
To validate against a deploy whose version does not match the local `package.json`
(e.g. a homologation box still on a previous patch release), override the expected
version explicitly:
```bash
HOMOLOG_EXPECT_VERSION=3.8.47 npm run homolog
```
The run exits non-zero if any layer fails, and always attempts to revoke the ephemeral
API key it created, even on failure (`finally` block in `scripts/homolog/run.mjs`).
## Reading the report
All output lands in `homolog-report/` (gitignored):
- `summary.md` — the same table printed to stdout, one row per layer (✅/❌ + detail).
- `homolog-ctrf.json` — the unified CTRF report (merge of API/SSE, provider-smoke, and
UI results) — this is the artifact to attach to a release STOP #2 checklist.
- `httpyac-junit.xml`, `api-ctrf.json`, `providers-ctrf.json`, `ui-ctrf.json` — the
per-layer raw/intermediate reports.
- `promptfooconfig.yaml`, `provider-misses.json` — the generated promptfoo config for
the current run and any critical providers that were missing from the live catalog.
A failing L0 aborts immediately (no ephemeral key is created) since a version/health
mismatch means every downstream layer would be validating the wrong deploy.
## Re-baselining when the UI changes legitimately
L4b (route smoke) and L4c (API-key UI flow) are driven by real DOM locators, not
snapshots, so most legitimate UI changes do not require any suite update. When a change
does break a locator (e.g. a renamed button label or a moved settings page):
1. Re-confirm the locator against the current source (the specs already document which
file/line each locator was confirmed against — follow the same pattern, don't guess).
2. Update the spec in `tests/homolog/ui/`.
3. Re-run `npm run homolog` (or just the affected Playwright spec) against the VPS to
confirm the fix, then commit.
There is no visual/pixel baseline in this suite (F1) — see Roadmap for that.
## Roadmap (F2 / F3)
Design and phased rollout live in the internal planning spec
`_tasks/superpowers/specs/2026-07-13-homolog-e2e-suite-design.md` (not linked — internal
`_tasks/` artifact, not part of this repo's tracked docs). Summary:
- **F2** — full walkthrough recording → Playwright Test Agents (`planner`/`generator`)
turn it into flow specs (create combo, test provider, edit settings, MCP tools) +
visual regression baseline (Lost Pixel) with masks over dynamic data (metrics,
timestamps, logs) + a `healer` maintenance routine per release.
- **F3** — resilience/contract/wiring coverage: toxiproxy + a fake OpenAI-compatible
provider on the devbox, a `homolog-resilience` combo on the VPS pointed at it
(injected timeout → assert fallback + circuit breaker open/close via
`/api/monitoring/health`); gated Schemathesis contract testing against
`docs/openapi.yaml` (low `--max-examples`, fixed seeds, non-LLM endpoints only); and
wiring `npm run homolog` + its `summary.md` into the `/generate-release` STOP #2 phase.

62
Merge-Train.md Normal file

@@ -0,0 +1,62 @@
> 🌍 [View in other languages](Languages)
# Merge Queue & Manual Merge-Train Runbook
Since v3.8.49 (WS3.2/WS3.4 of the quality/velocity plan) the default merge path for
reviewed PRs into `release/vX.Y.Z` is the **Mergify merge queue** (`.mergify.yml`);
the **manual merge-train** documented below is the FALLBACK — used during incidents,
release freezes, or if the Mergify Open Source plan ever changes.
## Default path: the Mergify queue
1. PR is reviewed/greened by the campaigns and approved by the owner's pre-merge ⭐
gate (the report + per-item decision — see `/merge-prs` Step 0.75).
2. The owner (or the session acting on the owner's decision) applies the **`queue`**
label. The label IS the merge approval; Mergify only executes it.
3. Mergify batches up to 10 queued PRs, validates the batch against the fast-gates,
and merges (squash). A red batch is **bisected automatically** — the offending PR
is isolated in ~log2(N) revalidations and unqueued; the rest proceed.
4. Post-merge, the continuous release-green workflow validates the new tip on push
and opens an attribution issue if the combination regressed (never auto-revert).
Guardrails (mirror `CLAUDE.md` Hard Rules #21/#22):
- **Release freeze open** → do NOT label PRs targeting the frozen branch; retarget to
the active `release/vX+1` first.
- **Another session's in-flight PR** → never label it; only the owning session queues
its own work.
- Tests-only diffs and `hotfix`-labeled PRs already run reduced CI (see
`RELEASE_CHECKLIST.md` → Hotfix Fast-Lane); the queue conditions accept whatever
check set actually ran (`#check-failure=0` + `#check-pending=0`).
## Fallback: the manual merge-train
Used when the queue is unavailable. This codifies the practice that drained 33 PRs in
one day during the v3.8.47 cycle:
1. **Assemble the batch** (~10–30 reviewed+approved PRs). Check `linked:` collisions
(same `tap.testFiles`, same CHANGELOG hunks) and serialize those.
2. **Validate ONCE**: in an isolated worktree off the release tip, merge all batch
heads locally, then run the release-equivalent suite
(`npm run check:release-green`, add `--with-build` before a release).
3. **Green** → merge the PRs in sequence (re-checking `state,headRefOid` before each —
a PR whose head moved re-enters review). Prove the net diff of each merge is the
PR's own change (no auto-resolve reverts: audit `git diff --stat` for
out-of-scope deletions).
4. **Red** → bisect the batch by halves (validate each half) instead of re-validating
one-by-one; drop the offending PR back to the review queue with the evidence.
5. **Never**: merge during a freeze into the frozen branch; `git stash` anywhere;
blanket-rerun CI hoping a red goes away (rule: a red is information).
## Tiering (why the queue is safe with fast-gates only)
- **Per PR** (quality.yml fast-gates): TIA-impacted tests + full unit 4-shard +
vitest + lint bag + typecheck + docs/changelog integrity.
- **Per batch/tip** (continuous release-green): `--quick` HARD gates on every push to
the release branch; full `--with-build --full-ci` sweeps 3×/day.
- **Per release** (ci.yml on the release PR): the complete matrix incl. E2E ×9,
package-artifact + tarball boot-smoke, coverage/ratchets.
Nothing is validated less than before — the heavy surface just runs per batch/tip
instead of per PR, which is what removes the O(N) round-trips.

34
Runner-Box.md Normal file

@@ -0,0 +1,34 @@
> 🌍 [View in other languages](Languages)
# Self-Hosted Runner Box Operations (.113 pool)
The self-hosted pool (`self-hosted, omni-release` labels) runs on the 16 GB box at
`192.168.0.113`. Two failure modes recurred on release days and were, until v3.8.49,
manual discipline; the **janitor script codifies them** (WS3.3 of the quality plan):
1. **Orphaned temp/work dirs** filling the disk → disk-full SQLite errors mid-job.
2. **>4 concurrent runners** → OOM-killed jobs (8-wide killed jobs twice on the
v3.8.47 release day; 4-wide is the proven ceiling).
## Install the janitor (one-time, on the box)
```bash
sudo mkdir -p /opt/omniroute-ops
sudo cp scripts/ops/runner-janitor.sh /opt/omniroute-ops/
sudo chmod +x /opt/omniroute-ops/runner-janitor.sh
( sudo crontab -l 2>/dev/null; echo '*/30 * * * * /opt/omniroute-ops/runner-janitor.sh >> /var/log/runner-janitor.log 2>&1' ) | sudo crontab -
```
What it does every 30min: sweeps runner temp leftovers older than 24h, alerts at
≥85% root-disk usage, and alerts when more than the runner ceiling (default 4, tunable
via the script's own environment) of `Runner.Listener` processes are up. Alerts land in `/var/log/runner-janitor.log`
with a non-zero exit (grep for `⚠`).
## Operating rules
- **Ceiling: 4 runners** on the 16 GB box. Runners 5–8 stay STOPPED except for
explicit off-peak experiments — never during a release window.
- Stopping a runner mid-job cancels the job (observed live): `systemctl stop`
only when its runner is idle (`Runner.Listener` without a `Runner.Worker` child).
- The `.15` VPS is homologation-only — never runs CI runners.