Files
OmniRoute/.github/workflows/nightly-resilience.yml
Diego Rodrigues de Sa e Souza 751710616a fix(ci): run the build-bearing nightly jobs on the box's light pool (#11965) (#11967)
Four nightly jobs run a backend-only `next build` on ubuntu-latest (7 GB):
Schemathesis, promptfoo injection guard, garak probes and the axe a11y suite
(self-building webServer). On release/v3.8.51 three of them died with the hosted
VM shutdown signature and nobody saw it — nightlies have no audience — and the
fourth passes by a margin of minutes. They now target [self-hosted, omni-light]
(hosted fallback when USE_VPS_RUNNER is off), a new two-listener label on the .113
box for jobs that need ~6 GB, not the 14-16 GB of a full build; they run once a day
in the 04:00-06:00 UTC window, when the box is idle.

Fleet reshaped the same day and documented in docs/ops/RUNNER_BOX.md: 4 active
OmniRoute listeners (omniroute-113-5/-6 omni-build, omniroute-113/-2 omni-light),
omniroute-113-3/-4/-7/-8 disabled (systemctl enable --now brings one back), janitor
ceiling MAX_ACTIVE_RUNNERS=4. The remaining headroom limit is the VM's 31 GB of RAM
(2 heavy + 2 light ≈ 42 GB peak, inside the 16 GB swap); more RAM on the Proxmox VM
is the lever that turns the label ceilings into 3 heavy + 2 light.

check:workflows --ratchet unchanged (194/194); check-workflows and
backend-only-smoke-workflows suites pass; docs-sync PASS.
2026-08-28 23:26:52 -03:00

115 lines
3.8 KiB
YAML

name: Nightly Resilience
on:
schedule:
- cron: "41 4 * * *"
workflow_dispatch:
permissions:
contents: read
jobs:
heap:
name: Heap-growth gate
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- uses: actions/setup-node@v7
with:
node-version: "24"
cache: npm
- run: npm ci
- run: npm run test:heap
chaos:
name: Resilience chaos (fault injection)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- uses: actions/setup-node@v7
with:
node-version: "24"
cache: npm
- run: npm ci
- run: npm run test:chaos
k6-soak:
name: k6 load/soak
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- uses: actions/setup-node@v7
with:
node-version: "24"
cache: npm
- run: npm ci
- name: Build CLI bundle
env:
JWT_SECRET: ci-build-secret-with-sufficient-length-for-validation
OMNIROUTE_BUILD_BACKEND_ONLY: "1"
run: npm run build:cli
- name: Start OmniRoute (background)
env:
JWT_SECRET: ci-build-secret-with-sufficient-length-for-validation
PORT: "20128"
run: |
node dist/server.js > server.log 2>&1 &
echo $! > server.pid
for i in $(seq 1 30); do
if curl -sf http://localhost:20128/api/monitoring/health >/dev/null; then echo "server up"; break; fi
sleep 2
done
- name: Install k6
uses: grafana/setup-k6-action@v1
- name: Run k6 soak
run: k6 run tests/load/k6-soak.js
env:
BASE_URL: http://localhost:20128
SOAK_DURATION: "3m"
SOAK_VUS: "10"
- name: Stop server
if: always()
run: kill "$(cat server.pid)" || true
a11y:
name: A11y axe (nightly, freeze-and-alert)
# #11965: this job runs a backend-only `next build`; the hosted 7 GB runner cannot build
# release/v3.8.51 (VM shutdown ~7 min in), so it targets the box's light pool (`omni-light`:
# two listeners, jobs ≤ ~6 GB). Falls back to hosted when USE_VPS_RUNNER is off.
runs-on: ${{ (vars.USE_VPS_RUNNER == 'true' && fromJSON('["self-hosted","omni-light"]')) || 'ubuntu-latest' }}
# The Playwright webServer (`start` mode) builds Next via build-next-isolated.mjs and
# boots the standalone server itself (waits on /api/monitoring/health, 15min webServer
# timeout). Unlike the per-PR test-e2e job, this nightly job has no pre-built artifact,
# so it self-builds — hence the generous job timeout. REQUIRE_AXE=1 makes the suite run
# the real axe analysis (the 4 page tests are gated to nightly so per-PR e2e stays fast)
# and makes the meta-test fail loudly if @axe-core/playwright ever goes missing.
timeout-minutes: 30
env:
JWT_SECRET: ci-test-secret-with-sufficient-length-for-validation
API_KEY_SECRET: ci-test-api-key-secret-long
DISABLE_SQLITE_AUTO_BACKUP: "true"
REQUIRE_AXE: "1"
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- uses: actions/setup-node@v7
with:
node-version: "24"
cache: npm
- run: npm ci
- name: Cache Playwright browsers
uses: actions/cache@v6.1.0
with:
path: ~/.cache/ms-playwright
key: playwright-chromium-${{ runner.os }}-${{ hashFiles('package-lock.json') }}
restore-keys: playwright-chromium-${{ runner.os }}-
- run: npx playwright install --with-deps chromium
- name: Run axe a11y suite (self-building webServer)
run: npx playwright test tests/e2e/a11y.spec.ts