Clone
1
Runner Box
github-actions[bot] edited this page 2026-07-17 06:04:12 +00:00
This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

🌍 View in other languages

Self-Hosted Runner Box Operations (.113 pool)

The self-hosted pool (self-hosted, omni-release labels) runs on the 16 GB box at 192.168.0.113. Two failure modes recurred on release days and were, until v3.8.49, manual discipline; the janitor script codifies them (WS3.3 of the quality plan):

  1. Orphaned temp/work dirs filling the disk → disk-full SQLite errors mid-job.
  2. >4 concurrent runners → OOM-killed jobs (8-wide killed jobs twice on the v3.8.47 release day; 4-wide is the proven ceiling).

Install the janitor (one-time, on the box)

sudo mkdir -p /opt/omniroute-ops
sudo cp scripts/ops/runner-janitor.sh /opt/omniroute-ops/
sudo chmod +x /opt/omniroute-ops/runner-janitor.sh
( sudo crontab -l 2>/dev/null; echo '*/30 * * * * /opt/omniroute-ops/runner-janitor.sh >> /var/log/runner-janitor.log 2>&1' ) | sudo crontab -

What it does every 30min: sweeps runner temp leftovers older than 24h, alerts at ≥85% root-disk usage, and alerts when more than the runner ceiling (default 4, tunable via the script's own environment) of Runner.Listener processes are up. Alerts land in /var/log/runner-janitor.log with a non-zero exit (grep for ).

Operating rules

  • Ceiling: 4 runners on the 16 GB box. Runners 58 stay STOPPED except for explicit off-peak experiments — never during a release window.
  • Stopping a runner mid-job cancels the job (observed live): systemctl stop only when its runner is idle (Runner.Listener without a Runner.Worker child).
  • The .15 VPS is homologation-only — never runs CI runners.