Files
OmniRoute/scripts/ops
Diego Rodrigues de Sa e Souza e71be03398 chore(ops): make the runner janitor act on what it can prove, not advise (#11893)
* chore(ops): make the runner janitor act on what it can prove, not advise

The .113 janitor already knew the rules and had been shouting them into a log
nobody reads: on 2026-08-28 12:00Z it reported "10 listeners > ceiling 8" and
"disk 85%" — for hours — while 6.7 GB of dead-run leftovers sat on the 12 GB
tmpfs (RAM) because its patterns matched neither e2e-build.tar.gz nor
next-build/, its 24 h fuse is a day too long for memory, and its _work/_temp
base (/home/*/actions-runner*) does not exist on this box (runners live under
/opt). Measured while draining the v3.8.50 npm publish (postmortem, Parte III).

What changes:

- idle is PROVEN before removal, with ONE lsof snapshot filtered to the swept
  bases (lsof +D per path walked whole trees and took minutes; 460 candidates
  grepping a re-printed 83k-line string was the other half). 20 s on the box.
  Without lsof the janitor removes nothing and says why (exit 1).
- tmpfs leftovers go after 3 h, disk _work/_temp after 24 h; both overridable.
  Patterns gain next-build* and e2e-build.tar.gz; /opt/actions-runner* is swept.
- zombie builds: a next-build older than 75 min has no job (a real Build step is
  ~26 min). On 2026-08-27 one ran 70 min after GitHub had declared its job lost,
  holding 3.6 GB. KillMode=mixed on the units covers systemctl stop/restart;
  this covers the lost-connection path.
- prunes 48 h-old checkouts under _work of runners whose unit is STOPPED — an
  active runner is never touched.
- alerts on memory PSI (full/avg60) and reports the listener ceiling with an
  omniroute/other breakdown (the box also hosts OmniHeuris and OmniMind).
  Enforcing the ceiling stays an operator decision (label split), not cron's.
- --dry-run prints exactly what it would do and touches nothing; unknown
  arguments are rejected.

Dry-run on the real box: 460 stale omniroute-* test fixtures (930 MB of RAM) it
would reclaim, 0 busy, 0 false "removed" lines, 20 s. The unit suite drives the
script against a fixture tree with every base redirected; the sweep branch runs
where lsof exists (hosted CI images) and the without-lsof contract everywhere.

docs/ops/RUNNER_BOX.md reconciled to the measured box: 31 GB (it said 16), ten
listeners, the 14 GB next-build ceiling, the KillMode drop-in, and the rule that
nothing is cleaned by hand while a runner is busy.

* docs(ops): restore the frontmatter fumadocs requires on RUNNER_BOX.md

Rewriting the page whole dropped its `title:` frontmatter, and docs/ is
compiled into the Next build by fumadocs-mdx — so Build, Fast Production Build
and dast-smoke all died with "[MDX] invalid frontmatter in
docs/ops/RUNNER_BOX.md". Same block as before, verbatim.
2026-08-28 15:44:29 -03:00
..