Files
OmniRoute/.github/workflows/nightly-release-green.yml
Diego Rodrigues de Sa e Souza b74c63a39e ci(vps): hermetic nightly pre-flight on the release runner (descoped: e2e/integration/electron stay hosted) (#6305)
* ci(vps): extend the dynamic omni-release runner to e2e/integration/electron + nightly pre-flight

Completes the #6284 rollout to the jobs where the VPS pays the most:
- test-e2e (9 shards, 15-20min/shard hosted — dominated by setup, not tests),
  test-integration (2 shards) and electron-package-smoke now pick the
  self-hosted omni-release runner under the same gate: vars.USE_VPS_RUNNER
  == 'true' AND own-origin (fork PRs never reach self-hosted).
- nightly-release-green (the release pre-flight) becomes runner-dynamic too:
  on the VPS it runs in a clean env — no operator OMNIROUTE_API_KEY, no
  local noauth CLIs — eliminating the machine-specific false positives that
  dominated the 2026-07-05 pre-flight; passes --hermetic (no-op until the
  #6300 validator lands, then belt-and-suspenders).

Validation plan (per operator request): release-runner-up.sh -> full ci.yml
workflow_dispatch on this branch exercising e2e/integration/electron on the
VPS (proves playwright --with-deps + xvfb on the runner) -> down.sh -> VM
off verified.

* fix(mitm): test suite and CI must never mutate the OS trust store (OMNIROUTE_SKIP_SYSTEM_TRUST)

Incident 2026-07-05 on the self-hosted release runner (VM 113): the
integration test 'POST /cert: installs trust when cert exists' exercised the
REAL install path, wrote a 105-byte fake PEM (FakeMITMCertForTestingOnly)
into /usr/local/share/ca-certificates and update-ca-certificates baked the
invalid entry into ca-certificates.crt — breaking ALL system TLS on the VM
(curl error 77, apt cert failures, and the intermittent gzip-corrupted
next-build artifacts that failed 6/9 e2e shards in run 28754447912). Hosted
runners are ephemeral, so the same mutation went unnoticed for months.

- installCert/uninstallCert: skip the OS dispatch under
  OMNIROUTE_SKIP_SYSTEM_TRUST=1 — AFTER the input checks, so the #4546
  environment-skip contract (missing file throws -> structured skip) and the
  already-installed/not-installed early returns are preserved.
- installTproxyCa/uninstallTproxyCa: same guard, only when no run dep is
  injected (DI'd tests keep exercising the full command sequence with mocks).
- tests/_setup/isolateDataDir.ts sets the env for every node:test process;
  ci.yml/quality.yml/nightly-release-green.yml set it workflow-wide (e2e runs
  the real app outside the test setup).

TDD: tests/unit/system-trust-test-guard.test.ts (guard exported to every test
process; guarded install resolves on a real file without touching the OS;
missing-file contract preserved). 82/82 across the affected cert/tproxy/
agent-bridge suites.

* ci(vps): descope — e2e/integration/electron stay on hosted runners; keep the hermetic nightly pre-flight dynamic

Validation verdict (runs 28754447912 + 28757670732, VM 113):
- e2e cannot run >1 per VM: both jobs bind port 20128 ('already used') — needs
  a per-job port in the playwright runner before any VM rollout.
- concurrent ~1GB artifact downloads truncate on the VM uplink (gzip 'invalid
  compressed data' — with 2 runners the e2e shard passed; corruption returned
  at 4) — actions/download-artifact has no integrity retry here.
- integration shard 2 exceeded its 15-min timeout twice on the VM.
The VPS remains a win for whole-machine jobs: build/unit/vitest (already
dynamic via #6284) and nightly-release-green (single job, clean env, hermetic
pre-flight) — which this PR keeps.
2026-07-05 21:49:48 -03:00

147 lines
5.9 KiB
YAML

name: Nightly Release-Green
# Solution D — continuous, NON-BLOCKING drift signal for the active release branch.
#
# WHY: the full gate (ci.yml) only runs on the release PR (PR → main), so reds
# accrue silently on release/** and explode — in layers — at release time. This
# nightly reproduces the release-equivalent validation on the active release branch
# HEAD and, when there are HARD failures, opens/updates a single tracking issue.
#
# It is NOT a required status check and never touches a contributor PR — it only
# reports. Ratchet drift (eslint warnings / cognitive-complexity / file-size) is
# expected mid-cycle and is reported but never raises the alarm on its own; only
# real defects (typecheck / lint errors / unit / vitest / db-rules / public-creds /
# package-artifact) flip the issue open.
on:
schedule:
- cron: "23 5 * * *" # 05:23 UTC daily — off-peak, distinct from other nightlies
workflow_dispatch:
inputs:
branch:
description: "Release branch to validate (default: highest release/vX.Y.Z)"
required: false
type: string
permissions:
contents: read
issues: write
concurrency:
group: nightly-release-green
cancel-in-progress: true
env:
OMNIROUTE_SKIP_SYSTEM_TRUST: "1"
jobs:
release-green:
name: Validate active release branch
# Dynamic runner: with USE_VPS_RUNNER=true (release window / on-demand pre-flight)
# this runs on the dedicated VPS runner — clean env (no operator OMNIROUTE_API_KEY,
# no local noauth CLIs => zero machine-specific false positives) and no contention.
# Nightly cron normally finds the var false (VM off) and falls back to hosted.
runs-on: ${{ (vars.USE_VPS_RUNNER == 'true' && fromJSON('["self-hosted","omni-release"]')) || 'ubuntu-latest' }}
env:
JWT_SECRET: ci-nightly-secret-with-sufficient-length-for-validation
API_KEY_SECRET: ci-nightly-api-key-secret-long
DISABLE_SQLITE_AUTO_BACKUP: "true"
steps:
- uses: actions/checkout@v7
with:
fetch-depth: 0
persist-credentials: false
- name: Resolve active release branch
id: branch
env:
INPUT_BRANCH: ${{ github.event.inputs.branch }}
run: |
set -euo pipefail
if [ -n "${INPUT_BRANCH:-}" ]; then
TARGET="$INPUT_BRANCH"
else
# highest release/vX.Y.Z by semver among remote branches
TARGET=$(git for-each-ref --format='%(refname:short)' 'refs/remotes/origin/release/v*' \
| sed 's#origin/##' \
| sort -t/ -k2 -V \
| tail -1)
fi
if [ -z "$TARGET" ]; then echo "No release/v* branch found"; exit 1; fi
# Strict format guard — reject anything that isn't release/vX.Y.Z (blocks
# ref/command injection via the workflow_dispatch input).
if ! printf '%s' "$TARGET" | grep -qE '^release/v[0-9]+\.[0-9]+\.[0-9]+$'; then
echo "Refusing non-canonical branch name: $TARGET"; exit 1
fi
echo "target=$TARGET" >> "$GITHUB_OUTPUT"
echo "Active release branch: $TARGET"
- name: Checkout the release branch
env:
TARGET: ${{ steps.branch.outputs.target }}
run: |
set -euo pipefail
git checkout "$TARGET"
git log -1 --oneline
- uses: actions/setup-node@v6
with:
node-version: "24"
cache: npm
- uses: ./.github/actions/npm-ci-retry
- name: Release-green validation (full)
id: validate
run: |
set +e
# --hermetic: scrub live-test trigger vars (self-hosted runner may carry
# operator env; hosted ignores the unknown flag before #6300 lands).
node scripts/quality/validate-release-green.mjs --json --with-build --hermetic \
1> release-green.json 2> release-green.log
echo "exit=$?" >> "$GITHUB_OUTPUT"
echo "------- report -------"
cat release-green.log
- name: Open / update tracking issue on HARD failure
if: steps.validate.outputs.exit != '0'
env:
GH_TOKEN: ${{ github.token }}
TARGET: ${{ steps.branch.outputs.target }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
run: |
set -euo pipefail
TITLE="🔴 Release branch not green: ${TARGET}"
{
echo "The nightly **release-green** validation found HARD failures on \`${TARGET}\`."
echo "These are real defects that would block the release PR — fix them in the"
echo "originating PR branch (via co-authorship), not by demanding it from contributors."
echo ""
echo "**Run:** ${RUN_URL}"
echo ""
echo '```'
sed -n '/──────── verdict ────────/,$p' release-green.log || tail -40 release-green.log
echo '```'
echo ""
echo "_Ratchet drift (eslint warnings / cognitive-complexity / file-size) listed above is expected mid-cycle and is rebaselined at release — it is NOT a contributor concern and did not, on its own, open this issue._"
} > issue-body.md
EXISTING=$(gh issue list --repo "$GITHUB_REPOSITORY" --state open \
--search "in:title $TITLE" --json number --jq '.[0].number' 2>/dev/null || echo "")
if [ -n "$EXISTING" ]; then
gh issue comment "$EXISTING" --repo "$GITHUB_REPOSITORY" --body-file issue-body.md
echo "Updated existing issue #$EXISTING"
else
gh issue create --repo "$GITHUB_REPOSITORY" --title "$TITLE" --body-file issue-body.md
fi
- name: Upload report artifact
if: always()
uses: actions/upload-artifact@v4
with:
name: release-green-report
path: |
release-green.json
release-green.log
if-no-files-found: ignore