From c4f29ca82468fc99118723b73aecb73e608f1cf9 Mon Sep 17 00:00:00 2001 From: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> Date: Fri, 28 Aug 2026 10:58:25 -0300 Subject: [PATCH] fix(ci): give main's build its own lane on the self-hosted pool MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The .113 box has 31 GB and a single next-build peaks at 14–16 GB RSS: one build fits with room, two sit at the edge, three take the box down. On 2026-08-28 13:50Z the kernel OOM-killed main's next-build (15.7 GB) while a PR build ran beside it — five Build jobs had been queued by a burst of PRs — and the publish lost its artefact, which sends it into the 40-minute rebuild that OOMs on its own (attempt 5 of this release). Job-level concurrency on `build`, two lanes: heavy-build-main pushes to main — never contended, never behind PR traffic heavy-build-pr pull requests — serialize among themselves cancel-in-progress stays false: a running build is never killed by a newer one. GitHub's own rule for a group is one running + one pending, older pendings cancelled — so under a burst the third PR build shows "cancelled" and needs a re-run. That is the trade-off, stated: a cancelled PR check is re-runnable; a dead main build costs a release. The proper fix remains a label split (omni-build on two runners, omni-light on the rest) so the queue lives on the runner side without cancellations — an operator decision recorded in docs/ops/RUNNER_BOX.md. --- .github/workflows/ci.yml | 11 +++++++++++ changelog.d/maintenance/11897-ci-heavy-build-lane.md | 3 +++ 2 files changed, 14 insertions(+) create mode 100644 changelog.d/maintenance/11897-ci-heavy-build-lane.md diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ab4a44268d..ee0727329c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -613,6 +613,17 @@ jobs: # var unset/false) also falls back to ubuntu-latest. runs-on: ${{ (vars.USE_VPS_RUNNER == 'true' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository)) && fromJSON('["self-hosted","omni-release"]') || 'ubuntu-latest' }} needs: changes + # The .113 pool runs ONE next-build with room to spare and two at the edge: the + # box has 31 GB and a single next-build peaks at 14–16 GB RSS. On 2026-08-28 + # 13:50Z the kernel OOM-killed main's build while a PR build ran beside it + # (five Build jobs had been queued by a burst of PRs). Two lanes: main keeps + # its own so a release is never queued behind PR traffic; PR builds serialize + # among themselves. GitHub keeps one running + one pending per group and + # CANCELS older pendings — a cancelled PR build is re-runnable; a dead main + # build costs the publish its artefact and a 40-minute rebuild that OOMs. + concurrency: + group: heavy-build-${{ github.ref == 'refs/heads/main' && 'main' || 'pr' }} + cancel-in-progress: false if: ${{ github.event_name != 'pull_request' || (needs.changes.outputs.code == 'true' && github.event.pull_request.draft == false) }} steps: - uses: actions/checkout@v7 diff --git a/changelog.d/maintenance/11897-ci-heavy-build-lane.md b/changelog.d/maintenance/11897-ci-heavy-build-lane.md new file mode 100644 index 0000000000..6bb3bbee3b --- /dev/null +++ b/changelog.d/maintenance/11897-ci-heavy-build-lane.md @@ -0,0 +1,3 @@ +- The CI `build` job now runs in two concurrency lanes — `main` and pull requests — + so a release build is never queued behind (or OOM-killed beside) PR builds on the + self-hosted pool, which holds one `next-build` comfortably and two at the edge.