e71be03398
* chore(ops): make the runner janitor act on what it can prove, not advise The .113 janitor already knew the rules and had been shouting them into a log nobody reads: on 2026-08-28 12:00Z it reported "10 listeners > ceiling 8" and "disk 85%" — for hours — while 6.7 GB of dead-run leftovers sat on the 12 GB tmpfs (RAM) because its patterns matched neither e2e-build.tar.gz nor next-build/, its 24 h fuse is a day too long for memory, and its _work/_temp base (/home/*/actions-runner*) does not exist on this box (runners live under /opt). Measured while draining the v3.8.50 npm publish (postmortem, Parte III). What changes: - idle is PROVEN before removal, with ONE lsof snapshot filtered to the swept bases (lsof +D per path walked whole trees and took minutes; 460 candidates grepping a re-printed 83k-line string was the other half). 20 s on the box. Without lsof the janitor removes nothing and says why (exit 1). - tmpfs leftovers go after 3 h, disk _work/_temp after 24 h; both overridable. Patterns gain next-build* and e2e-build.tar.gz; /opt/actions-runner* is swept. - zombie builds: a next-build older than 75 min has no job (a real Build step is ~26 min). On 2026-08-27 one ran 70 min after GitHub had declared its job lost, holding 3.6 GB. KillMode=mixed on the units covers systemctl stop/restart; this covers the lost-connection path. - prunes 48 h-old checkouts under _work of runners whose unit is STOPPED — an active runner is never touched. - alerts on memory PSI (full/avg60) and reports the listener ceiling with an omniroute/other breakdown (the box also hosts OmniHeuris and OmniMind). Enforcing the ceiling stays an operator decision (label split), not cron's. - --dry-run prints exactly what it would do and touches nothing; unknown arguments are rejected. Dry-run on the real box: 460 stale omniroute-* test fixtures (930 MB of RAM) it would reclaim, 0 busy, 0 false "removed" lines, 20 s. The unit suite drives the script against a fixture tree with every base redirected; the sweep branch runs where lsof exists (hosted CI images) and the without-lsof contract everywhere. docs/ops/RUNNER_BOX.md reconciled to the measured box: 31 GB (it said 16), ten listeners, the 14 GB next-build ceiling, the KillMode drop-in, and the rule that nothing is cleaned by hand while a runner is busy. * docs(ops): restore the frontmatter fumadocs requires on RUNNER_BOX.md Rewriting the page whole dropped its `title:` frontmatter, and docs/ is compiled into the Next build by fumadocs-mdx — so Build, Fast Production Build and dast-smoke all died with "[MDX] invalid frontmatter in docs/ops/RUNNER_BOX.md". Same block as before, verbatim.
4.6 KiB
title
| title |
|---|
| Self-Hosted Runner Box Operations |
Self-Hosted Runner Box Operations (.113 pool)
The self-hosted pool (self-hosted, omni-release labels) runs on the .113 box.
Measured 2026-08-28 (v3.8.50 postmortem, Parte III):
| resource | value | what it means for scheduling |
|---|---|---|
| RAM / CPU | 31 GB / 32 cores (was 16 GB when this doc was first written) | one next-build peaks at ~14 GB → 2 concurrent heavy builds saturate the box, 3 take it down (2026-08-28 06:42Z: load 56, two jobs lost) |
| swap | 15 GB | it swapped its way through the v3.8.50 publish; pressure shows in /proc/pressure/memory |
/tmp |
12 GB tmpfs = RAM | anything parked there is memory; leftovers are swept after 3 h |
| disk | 188 GB | _work checkouts of 8 runners reach ~70 GB with no cap |
| runners | 10 listeners: 8 OmniRoute + OmniHeuris + OmniMind | all share the memory above |
Install the janitor (one-time, on the box)
scp scripts/ops/runner-janitor.sh root@192.168.0.113:/opt/omniroute-ops/runner-janitor.sh
ssh root@192.168.0.113 'chmod +x /opt/omniroute-ops/runner-janitor.sh; apt-get install -y lsof'
# cron (root): every 30 min, log to /var/log/runner-janitor.log
*/30 * * * * MAX_ACTIVE_RUNNERS=8 /opt/omniroute-ops/runner-janitor.sh >> /var/log/runner-janitor.log 2>&1
lsof is required: the janitor proves a path is idle with one snapshot of open
files before removing it, and without the tool it removes nothing and says so
(exit 1). Try any change with --dry-run first — it prints exactly what it would
do and touches nothing.
What it does every run: sweeps our own leftovers (runner-*, omniroute-*,
next-build*, e2e-build.tar.gz) after 3 h on tmpfs and 24 h on disk
_work/_temp; kills a next-build older than 75 min (no job runs that long — on
2026-08-27 one ran 70 min after GitHub had declared its job lost); prunes 48 h-old
checkouts of runners whose unit is stopped; alerts on disk ≥ 85 %, memory PSI
full/avg60 ≥ 10 %, and more listeners than MAX_ACTIVE_RUNNERS (with an
omniroute/other breakdown). Exit 1 = attention needed; read the log.
Runner units: KillMode
The runner's default KillMode=process leaves Runner.Worker → npm → next-build
alive when a unit is stopped or restarted — an orphan build keeps eating RAM and
CPU with no job attached. Every OmniRoute unit carries a drop-in
(/etc/systemd/system/actions.runner.diegosouzapw-OmniRoute.<name>.service.d/10-killmode.conf)
with KillMode=mixed: SIGTERM to the listener first, SIGKILL to the whole cgroup at
TimeoutStop. It takes effect on the unit's next restart — restart one runner at
a time, only when idle, with the idle check and the restart in the same command.
Operating rules
- Heavy-build ceiling: 2 at a time. The listener ceiling (
MAX_ACTIVE_RUNNERS=8in cron) is a proxy until jobs are split by label —omni-buildon 2 runners for Build/publish/heavy shards,omni-lighton the rest — which is an operator decision, not something cron should enforce by killing listeners. - Never clean
/tmpor_workby hand while any runner is busy. A check-then-delete with a gap between the two is how a live Build job lost its_workon 2026-08-27. The janitor does the check and the removal in one step; let it. - Stopping a runner mid-job cancels the job (observed live):
systemctl stoponly when its listener has noRunner.Workerchild — and do it in one command. - Workflows must not park artefacts in
/tmp(it is RAM). Download to$RUNNER_TEMP(on disk, per runner) — the 1.3 GBnext-buildartefact took 27–32 minutes to land on the tmpfs and 2 minutes to upload from disk. - The
.15VPS is homologation-only — never runs CI runners.