diff --git a/changelog.d/maintenance/11892-runner-janitor-act-not-advise.md b/changelog.d/maintenance/11892-runner-janitor-act-not-advise.md new file mode 100644 index 0000000000..4f311fdf29 --- /dev/null +++ b/changelog.d/maintenance/11892-runner-janitor-act-not-advise.md @@ -0,0 +1,5 @@ +- `scripts/ops/runner-janitor.sh` now proves a path is idle with one `lsof` + snapshot and removes stale leftovers itself (tmpfs after 3 h — it is RAM — disk + after 24 h), kills orphan `next-build` processes, prunes checkouts of stopped + runners, and alerts on memory pressure; `--dry-run` shows exactly what it would + do. `docs/ops/RUNNER_BOX.md` reconciled to the measured box (31 GB, 10 listeners). diff --git a/docs/ops/RUNNER_BOX.md b/docs/ops/RUNNER_BOX.md index 07bd70cb04..2012742e7e 100644 --- a/docs/ops/RUNNER_BOX.md +++ b/docs/ops/RUNNER_BOX.md @@ -4,32 +4,62 @@ title: Self-Hosted Runner Box Operations # Self-Hosted Runner Box Operations (.113 pool) -The self-hosted pool (`self-hosted, omni-release` labels) runs on the 16 GB box at -`192.168.0.113`. Two failure modes recurred on release days and were, until v3.8.49, -manual discipline; the **janitor script codifies them** (WS3.3 of the quality plan): +The self-hosted pool (`self-hosted, omni-release` labels) runs on the **.113** box. +Measured 2026-08-28 (v3.8.50 postmortem, Parte III): -1. **Orphaned temp/work dirs** filling the disk → disk-full SQLite errors mid-job. -2. **>4 concurrent runners** → OOM-killed jobs (8-wide killed jobs twice on the - v3.8.47 release day; 4-wide is the proven ceiling). +| resource | value | what it means for scheduling | +| --------- | ---------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | +| RAM / CPU | **31 GB / 32 cores** (was 16 GB when this doc was first written) | one `next-build` peaks at **~14 GB** → 2 concurrent heavy builds saturate the box, 3 take it down (2026-08-28 06:42Z: load 56, two jobs lost) | +| swap | 15 GB | it swapped its way through the v3.8.50 publish; pressure shows in `/proc/pressure/memory` | +| `/tmp` | **12 GB tmpfs = RAM** | anything parked there is memory; leftovers are swept after 3 h | +| disk | 188 GB | `_work` checkouts of 8 runners reach ~70 GB with no cap | +| runners | **10 listeners**: 8 OmniRoute + OmniHeuris + OmniMind | all share the memory above | ## Install the janitor (one-time, on the box) ```bash -sudo mkdir -p /opt/omniroute-ops -sudo cp scripts/ops/runner-janitor.sh /opt/omniroute-ops/ -sudo chmod +x /opt/omniroute-ops/runner-janitor.sh -( sudo crontab -l 2>/dev/null; echo '*/30 * * * * /opt/omniroute-ops/runner-janitor.sh >> /var/log/runner-janitor.log 2>&1' ) | sudo crontab - +scp scripts/ops/runner-janitor.sh root@192.168.0.113:/opt/omniroute-ops/runner-janitor.sh +ssh root@192.168.0.113 'chmod +x /opt/omniroute-ops/runner-janitor.sh; apt-get install -y lsof' +# cron (root): every 30 min, log to /var/log/runner-janitor.log +*/30 * * * * MAX_ACTIVE_RUNNERS=8 /opt/omniroute-ops/runner-janitor.sh >> /var/log/runner-janitor.log 2>&1 ``` -What it does every 30min: sweeps runner temp leftovers older than 24h, alerts at -≥85% root-disk usage, and alerts when more than the runner ceiling (default 4, tunable -via the script's own environment) of `Runner.Listener` processes are up. Alerts land in `/var/log/runner-janitor.log` -with a non-zero exit (grep for `⚠`). +`lsof` is required: the janitor proves a path is idle with one snapshot of open +files before removing it, and without the tool it removes nothing and says so +(exit 1). Try any change with `--dry-run` first — it prints exactly what it would +do and touches nothing. + +What it does every run: sweeps our own leftovers (`runner-*`, `omniroute-*`, +`next-build*`, `e2e-build.tar.gz`) after **3 h on tmpfs** and 24 h on disk +`_work/_temp`; kills a `next-build` older than 75 min (no job runs that long — on +2026-08-27 one ran 70 min after GitHub had declared its job lost); prunes 48 h-old +checkouts of runners whose unit is **stopped**; alerts on disk ≥ 85 %, memory PSI +`full/avg60` ≥ 10 %, and more listeners than `MAX_ACTIVE_RUNNERS` (with an +omniroute/other breakdown). Exit 1 = attention needed; read the log. + +## Runner units: KillMode + +The runner's default `KillMode=process` leaves `Runner.Worker → npm → next-build` +alive when a unit is stopped or restarted — an orphan build keeps eating RAM and +CPU with no job attached. Every OmniRoute unit carries a drop-in +(`/etc/systemd/system/actions.runner.diegosouzapw-OmniRoute..service.d/10-killmode.conf`) +with `KillMode=mixed`: SIGTERM to the listener first, SIGKILL to the whole cgroup at +`TimeoutStop`. It takes effect on the unit's next restart — restart **one runner at +a time, only when idle**, with the idle check and the restart in the same command. ## Operating rules -- **Ceiling: 4 runners** on the 16 GB box. Runners 5–8 stay STOPPED except for - explicit off-peak experiments — never during a release window. -- Stopping a runner mid-job cancels the job (observed live): `systemctl stop` - only when its runner is idle (`Runner.Listener` without a `Runner.Worker` child). +- **Heavy-build ceiling: 2 at a time.** The listener ceiling (`MAX_ACTIVE_RUNNERS=8` + in cron) is a proxy until jobs are split by label — `omni-build` on 2 runners for + Build/publish/heavy shards, `omni-light` on the rest — which is an operator + decision, not something cron should enforce by killing listeners. +- **Never clean `/tmp` or `_work` by hand while any runner is busy.** A + check-then-delete with a gap between the two is how a live Build job lost its + `_work` on 2026-08-27. The janitor does the check and the removal in one step; + let it. +- Stopping a runner mid-job cancels the job (observed live): `systemctl stop` only + when its listener has no `Runner.Worker` child — and do it in one command. +- Workflows must not park artefacts in `/tmp` (it is RAM). Download to + `$RUNNER_TEMP` (on disk, per runner) — the 1.3 GB `next-build` artefact took 27–32 + minutes to land on the tmpfs and 2 minutes to upload from disk. - The `.15` VPS is homologation-only — never runs CI runners. diff --git a/scripts/ops/runner-janitor.sh b/scripts/ops/runner-janitor.sh index 99c081b10f..9a07cf0b60 100755 --- a/scripts/ops/runner-janitor.sh +++ b/scripts/ops/runner-janitor.sh @@ -1,53 +1,172 @@ #!/usr/bin/env bash -# runner-janitor — self-hosted runner box hygiene (WS3.3, v3.8.49 quality plan). +# runner-janitor — self-hosted runner box hygiene for the .113 pool. # -# The .113 runner box has recurring failure modes that until now were manual -# discipline: orphaned tmpfs/work dirs filling the disk, and >4 concurrent -# runners OOM-killing jobs (16 GB box; incidents on the v3.8.47 release day). -# Install via cron on the box (see docs/ops/RUNNER_BOX.md): -# */30 * * * * /opt/omniroute-ops/runner-janitor.sh >> /var/log/runner-janitor.log 2>&1 +# Runs from cron every 30 min (see docs/ops/RUNNER_BOX.md). It ACTS on what it +# can prove is safe and ALERTS on what needs an operator decision. Reads of +# "is this in use?" and the removal happen in the same command, never in two +# passes: a check-then-delete with a gap is how a live Build job lost its _work +# on 2026-08-27. # +# Measured box (2026-08-28): 31 GB RAM, 32 cores, 15 GB swap, /tmp = 12 GB +# tmpfs (RAM!), 188 GB disk. A single `next-build` peaks at ~14 GB, so two +# concurrent heavy builds saturate the box and three take it down (06:42Z that +# day: load 56, two jobs lost). The v3.8.50 postmortem (Parte III) has the numbers. +# +# What it does, in order: +# 1) sweep stale artefacts our tooling leaves behind — tmpfs bases after 3 h +# (they hold RAM), disk _work/_temp bases after 24 h; only names we create, +# only when no process has them open +# 2) kill zombie builds: a `next-build` older than ZOMBIE_BUILD_MAX_MIN has no +# job attached (a real Build step measures ~26 min). On 2026-08-27 one ran +# 70 minutes after GitHub had already declared its job lost, eating 3.6 GB +# and a full core set. KillMode=mixed on the units covers systemctl +# stop/restart; this covers the lost-connection path. +# 3) prune 48 h-old checkouts under _work of runners whose unit is INACTIVE +# (stopped runners cannot be mid-job; active ones are never touched) +# 4) alert: root disk >= DISK_ALERT_PCT, memory PSI full/avg60 >= threshold, +# Runner.Listener count above the ceiling (with a per-project breakdown — +# the box also hosts OmniHeuris and OmniMind runners) +# +# Usage: runner-janitor.sh [--dry-run] [--help] # Exit codes: 0 healthy · 1 attention needed (printed to stdout for the log). set -euo pipefail -MAX_ACTIVE_RUNNERS="${MAX_ACTIVE_RUNNERS:-4}" -DISK_ALERT_PCT="${DISK_ALERT_PCT:-85}" -WORK_DIR_MAX_AGE_HOURS="${WORK_DIR_MAX_AGE_HOURS:-24}" -STATUS=0 - -echo "[janitor] $(date -u +%FT%TZ) start" - -# 1) Sweep stale runner temp/work leftovers (>24h — no legitimate job runs that long). -# Hardened for a root cron on world-writable paths: never follow a symlinked base -# (a compromised runner could plant one), -P + -xdev so the sweep cannot traverse -# out of the filesystem, and patterns narrowed to names OUR tooling creates -# (no generic tmp* — unrelated system temp files are out of scope). -for base in /tmp /home/*/actions-runner*/_work/_temp; do - [ -d "$base" ] || continue - [ -L "$base" ] && { echo "[janitor] skip symlinked base: $base"; continue; } - find -P "$base" -xdev -maxdepth 1 \( -name 'runner-*' -o -name 'omniroute-*' \) \ - ! -type l -mmin +$((WORK_DIR_MAX_AGE_HOURS * 60)) -exec rm -rf {} + 2>/dev/null || true +DRY_RUN=0 +for arg in "$@"; do + case "$arg" in + --dry-run) DRY_RUN=1 ;; + -h|--help) + sed -n '2,32p' "$0" | sed 's/^# \{0,1\}//' + exit 0 ;; + *) echo "unknown argument: $arg" >&2; exit 2 ;; + esac done -echo "[janitor] stale temp sweep done" -# 2) Disk pressure — alert loudly before SQLITE_FULL kills jobs mid-run. -USAGE=$(df --output=pcent / | tail -1 | tr -dc '0-9') -if [ "$USAGE" -ge "$DISK_ALERT_PCT" ]; then - echo "[janitor] ⚠ ROOT DISK ${USAGE}% >= ${DISK_ALERT_PCT}% — clean before the next heavy run" - STATUS=1 +MAX_ACTIVE_RUNNERS="${MAX_ACTIVE_RUNNERS:-8}" +DISK_ALERT_PCT="${DISK_ALERT_PCT:-85}" +TMPFS_MAX_AGE_HOURS="${TMPFS_MAX_AGE_HOURS:-3}" +WORK_TEMP_MAX_AGE_HOURS="${WORK_TEMP_MAX_AGE_HOURS:-24}" +WORK_CHECKOUT_MAX_AGE_HOURS="${WORK_CHECKOUT_MAX_AGE_HOURS:-48}" +ZOMBIE_BUILD_MAX_MIN="${ZOMBIE_BUILD_MAX_MIN:-75}" +ZOMBIE_BUILD_COMM="${ZOMBIE_BUILD_COMM:-next-build}" +PSI_FULL_AVG60_ALERT="${PSI_FULL_AVG60_ALERT:-10}" +# Overridable so the unit test can point everything at a fixture tree. +JANITOR_TMP_BASES="${JANITOR_TMP_BASES-/tmp}" +JANITOR_WORK_TEMP_BASES="${JANITOR_WORK_TEMP_BASES-/opt/actions-runner*/_work/_temp /home/*/actions-runner*/_work/_temp}" +JANITOR_RUNNER_DIRS="${JANITOR_RUNNER_DIRS-/opt/actions-runner*}" +JANITOR_PSI_FILE="${JANITOR_PSI_FILE:-/proc/pressure/memory}" +JANITOR_DF_PATH="${JANITOR_DF_PATH:-/}" + +STATUS=0 +say() { echo "[janitor] $*"; } + +# "Is anything using this?" — ONE snapshot of every open path on the box +# (lsof -Fn), then a prefix match per candidate. `lsof +D ` walks the whole +# tree instead and took minutes on a 5 GB leftover — unusable from cron. An +# absent lsof means "cannot prove idle": the sweep keeps the path and says so. +LSOF_BIN="${JANITOR_LSOF:-lsof}" +have_busy_tools() { command -v "$LSOF_BIN" >/dev/null 2>&1; } +SNAP="" +cleanup() { [ -n "$SNAP" ] && rm -f -- "$SNAP"; } +trap cleanup EXIT +# One lsof for the whole run (~13 s / 83k lines on the box), kept ONLY for the +# bases we sweep — 460 candidates grepping a re-printed 83k-line string was the +# slow part, not lsof itself. +snapshot_open_paths() { + have_busy_tools || return 0 + SNAP=$(mktemp) || return 0 + local prefixes="" b + for b in $JANITOR_TMP_BASES $JANITOR_WORK_TEMP_BASES; do [ -d "$b" ] && prefixes="$prefixes"$'\n'"$b/"; done + # -F n: one "n" line per open file; -w: no warnings + "$LSOF_BIN" -w -Fn 2>/dev/null | sed -n 's/^n//p' | grep -F -f <(printf '%s' "$prefixes" | sed '/^$/d') > "$SNAP" 2>/dev/null || true +} +is_busy() { + local p="$1" + [ -n "$SNAP" ] && [ -s "$SNAP" ] || return 1 + # exact path, or anything beneath it when it is a directory + grep -qxF -- "$p" "$SNAP" && return 0 + [ -d "$p" ] && grep -qF -- "$p/" "$SNAP" +} + +# sweep : only names our tooling creates, never through +# a symlinked base, never across a filesystem, and remove+check in one step. +sweep() { + local base="$1" max_min="$2" p + [ -d "$base" ] || return 0 + [ -L "$base" ] && { say "skip symlinked base: $base"; return 0; } + while IFS= read -r -d '' p; do + if ! have_busy_tools; then say "cannot prove idle (lsof missing — apt install lsof), kept: $p"; STATUS=1; continue; fi + if is_busy "$p"; then say "busy, kept: $p"; continue; fi + if [ "$DRY_RUN" -eq 1 ]; then say "would remove ($(( max_min / 60 ))h+): $p"; else rm -rf -- "$p" && say "removed ($(( max_min / 60 ))h+): $p"; fi + done < <(find -P "$base" -xdev -mindepth 1 -maxdepth 1 \ + \( -name 'runner-*' -o -name 'omniroute-*' -o -name 'next-build*' -o -name 'e2e-build.tar.gz' \) \ + ! -type l -mmin "+$max_min" -print0 2>/dev/null || true) +} + +say "$(date -u +%FT%TZ) start${DRY_RUN:+ (dry-run=$DRY_RUN)} busy-tools=$(have_busy_tools && echo ok || echo MISSING)" + +# 1) stale artefacts — tmpfs is RAM, so it gets the short fuse +snapshot_open_paths +for base in $JANITOR_TMP_BASES; do sweep "$base" $(( TMPFS_MAX_AGE_HOURS * 60 )); done +for base in $JANITOR_WORK_TEMP_BASES; do sweep "$base" $(( WORK_TEMP_MAX_AGE_HOURS * 60 )); done +say "stale temp sweep done" + +# 2) zombie builds +ZOMBIES=0 +while read -r pid etimes comm; do + [ -n "${pid:-}" ] || continue + if [ "$etimes" -gt $(( ZOMBIE_BUILD_MAX_MIN * 60 )) ]; then + say "⚠ zombie build pid=$pid comm=$comm age=$(( etimes / 60 ))min > ${ZOMBIE_BUILD_MAX_MIN}min — no job runs this long" + if [ "$DRY_RUN" -eq 1 ]; then say "[dry-run] would: kill -TERM $pid (then -KILL)"; else + kill -TERM "$pid" 2>/dev/null || true; sleep 10 + kill -0 "$pid" 2>/dev/null && { kill -KILL "$pid" 2>/dev/null || true; say " needed SIGKILL"; } + fi + ZOMBIES=$(( ZOMBIES + 1 )); STATUS=1 + fi +done < <(ps -eo pid=,etimes=,comm= 2>/dev/null | awk -v c="$ZOMBIE_BUILD_COMM" '$3 ~ ("^" c) {print $1, $2, $3}' || true) +say "zombie builds: $ZOMBIES" + +# 3) old checkouts of STOPPED runners +for d in $JANITOR_RUNNER_DIRS; do + [ -d "$d" ] && [ -f "$d/.runner" ] || continue + agent=$(grep -o '"agentName": *"[^"]*"' "$d/.runner" 2>/dev/null | sed 's/.*"\([^"]*\)"$/\1/') + [ -n "$agent" ] || continue + unit=$(systemctl list-units --plain --no-legend "actions.runner.*.${agent}.service" 2>/dev/null | awk 'NR==1{print $1}') + [ -n "$unit" ] || continue + if systemctl is-active --quiet "$unit"; then continue; fi + while IFS= read -r -d '' co; do + if [ "$DRY_RUN" -eq 1 ]; then say "would prune checkout of stopped runner $agent: $co"; else rm -rf -- "$co" && say "pruned checkout of stopped runner $agent: $co"; fi + done < <(find -P "$d/_work" -xdev -mindepth 2 -maxdepth 2 -type d -mmin "+$(( WORK_CHECKOUT_MAX_AGE_HOURS * 60 ))" -print0 2>/dev/null || true) +done + +# 4a) disk +USAGE=$(df --output=pcent "$JANITOR_DF_PATH" 2>/dev/null | tail -1 | tr -dc '0-9') +if [ "${USAGE:-0}" -ge "$DISK_ALERT_PCT" ]; then + say "⚠ ROOT DISK ${USAGE}% >= ${DISK_ALERT_PCT}% — clean before the next heavy run"; STATUS=1 else - echo "[janitor] disk ${USAGE}% OK" + say "disk ${USAGE:-?}% OK" fi -# 3) Concurrency ceiling — 8-wide OOMed the 16 GB box twice on release day; -# 4 is the proven ceiling. This CODIFIES the rule that was manual discipline. +# 4b) memory pressure (PSI) — the box swapped its way through the v3.8.50 publish +if [ -r "$JANITOR_PSI_FILE" ]; then + FULL60=$(awk '/^full/ {for(i=1;i<=NF;i++) if ($i ~ /^avg60=/) {sub("avg60=","",$i); print $i}}' "$JANITOR_PSI_FILE" 2>/dev/null || echo "") + if [ -n "$FULL60" ] && awk -v v="$FULL60" -v t="$PSI_FULL_AVG60_ALERT" 'BEGIN{exit !(v+0 >= t+0)}'; then + say "⚠ MEMORY PRESSURE psi full/avg60=${FULL60}% >= ${PSI_FULL_AVG60_ALERT}% — too many heavy jobs at once"; STATUS=1 + else + say "memory psi full/avg60=${FULL60:-n/a}% OK" + fi +fi + +# 4c) concurrency ceiling — alert with a breakdown; the fix is fewer/labelled +# runners (an operator decision), not killing listeners from cron. ACTIVE=$(pgrep -fc "Runner.Listener" || true) +OMNI=$(pgrep -fc "actions-runner-omniroute[^ ]*/bin[^ ]*/Runner.Listener" || true) if [ "${ACTIVE:-0}" -gt "$MAX_ACTIVE_RUNNERS" ]; then - echo "[janitor] ⚠ ${ACTIVE} Runner.Listener processes > ceiling ${MAX_ACTIVE_RUNNERS} — stop the extra runners (systemctl stop actions.runner.)" + say "⚠ ${ACTIVE} Runner.Listener processes (omniroute=${OMNI:-0}, other=$(( ${ACTIVE:-0} - ${OMNI:-0} ))) > ceiling ${MAX_ACTIVE_RUNNERS} — stop idle extras: systemctl stop only when it has no Runner.Worker child" STATUS=1 else - echo "[janitor] runners active: ${ACTIVE:-0}/${MAX_ACTIVE_RUNNERS} OK" + say "runners active: ${ACTIVE:-0}/${MAX_ACTIVE_RUNNERS} (omniroute=${OMNI:-0}) OK" fi -echo "[janitor] done status=$STATUS" +say "done status=$STATUS" exit "$STATUS" diff --git a/tests/unit/runner-janitor.test.ts b/tests/unit/runner-janitor.test.ts new file mode 100644 index 0000000000..9519414fbc --- /dev/null +++ b/tests/unit/runner-janitor.test.ts @@ -0,0 +1,196 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import { + existsSync, + mkdirSync, + mkdtempSync, + readFileSync, + rmSync, + statSync, + utimesSync, + writeFileSync, +} from "node:fs"; +import os from "node:os"; +import path from "node:path"; + +/** + * scripts/ops/runner-janitor.sh runs from cron on the .113 runner box. This + * suite pins its safety contract against a fixture tree — never the real /tmp: + * every base, the runner dirs, the PSI file and the df path are redirected, the + * zombie pattern is set to a name no process has, and the ceilings are lifted + * so the outcome does not depend on the box the test happens to run on. + */ +const ROOT = path.resolve(import.meta.dirname, "..", ".."); +const SCRIPT = path.join(ROOT, "scripts", "ops", "runner-janitor.sh"); +const HOUR = 3_600_000; +// The sweep needs lsof to PROVE a path is idle (one snapshot of open paths). Hosted CI +// images ship both; a bare devbox may not. Each branch below asserts what must +// hold in that environment — without the tools the contract is "delete nothing, +// say why", which is exactly the behaviour worth pinning. +const HAVE_BUSY_TOOLS = + spawnSync("bash", ["-c", "command -v lsof"], { stdio: "ignore" }).status === 0; + +function fixture() { + const base = mkdtempSync(path.join(os.tmpdir(), "janitor-fixture-")); + const old = new Date(Date.now() - 5 * HOUR); + const mk = (name: string, dir: boolean, when: Date | null) => { + const p = path.join(base, name); + if (dir) { + mkdirSync(p); + writeFileSync(path.join(p, "x"), "x"); + } else writeFileSync(p, "x"); + if (when) utimesSync(p, when, when); + return p; + }; + return { + base, + staleTar: mk("e2e-build.tar.gz", false, old), // fixed-name artefact ci.yml/npm-publish leave behind + staleBuild: mk("next-build-abc", true, old), + staleUpgrade: mk("omniroute-install-upgrade-xyz", true, old), + fresh: mk("omniroute-batch-api-fresh", true, null), // in use right now + unrelated: mk("somebody-elses.log", false, old), // not ours — never touched + }; +} + +function run(args: string[], base: string, extraEnv: Record = {}) { + return spawnSync("bash", [SCRIPT, ...args], { + encoding: "utf8", + stdio: ["ignore", "pipe", "pipe"], + env: { + ...process.env, + JANITOR_TMP_BASES: base, + JANITOR_WORK_TEMP_BASES: "", + JANITOR_RUNNER_DIRS: path.join(base, "no-runners-here-*"), + JANITOR_PSI_FILE: path.join(base, "no-psi"), + JANITOR_DF_PATH: base, + ZOMBIE_BUILD_COMM: "janitor-test-no-such-process", + MAX_ACTIVE_RUNNERS: "9999", + DISK_ALERT_PCT: "101", + ...extraEnv, + }, + }); +} + +describe("runner-janitor.sh", () => { + it("is executable bash with strict mode and prints usage on --help", () => { + assert.ok(existsSync(SCRIPT)); + assert.ok(statSync(SCRIPT).mode & 0o111, "must be chmod +x (cron runs it directly)"); + const body = readFileSync(SCRIPT, "utf8"); + assert.ok(body.startsWith("#!/usr/bin/env bash")); + assert.ok(body.includes("set -euo pipefail")); + const help = run(["--help"], os.tmpdir()); + assert.equal(help.status, 0, help.stderr); + assert.match(help.stdout, /--dry-run/); + }); + + it("without lsof it cannot prove idle, so it deletes nothing and says why (exit 1)", () => { + const f = fixture(); + try { + const r = run([], f.base, { JANITOR_LSOF: "/nonexistent/lsof" }); + assert.equal(r.status, 1, "a janitor that cannot do its job must show up in the cron log"); + assert.match(r.stdout, /busy-tools=MISSING/); + assert.match( + r.stdout, + /cannot prove idle \(lsof missing — apt install lsof\), kept: .*e2e-build\.tar\.gz/ + ); + for (const p of [f.staleTar, f.staleBuild, f.staleUpgrade, f.fresh, f.unrelated]) { + assert.ok(existsSync(p), `must not delete ${p} when idleness cannot be proven`); + } + } finally { + rmSync(f.base, { recursive: true, force: true }); + } + }); + + it("--dry-run names what it WOULD remove and removes nothing", (t) => { + if (!HAVE_BUSY_TOOLS) return t.skip("lsof absent on this box — sweep branch covered in CI"); + const f = fixture(); + try { + const r = run(["--dry-run"], f.base); + assert.equal(r.status, 0, r.stderr + r.stdout); + assert.match(r.stdout, /busy-tools=ok/); + for (const p of [f.staleTar, f.staleBuild, f.staleUpgrade]) { + assert.match( + r.stdout, + new RegExp(`would remove \\(3h\\+\\): ${p.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")}`) + ); + assert.doesNotMatch( + r.stdout, + new RegExp(`removed \\(3h\\+\\): ${p.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")}`), + "dry-run must never claim it removed something" + ); + assert.ok(existsSync(p), `dry-run must not delete ${p}`); + } + assert.doesNotMatch( + r.stdout, + /omniroute-batch-api-fresh/, + "a fresh dir is never a candidate" + ); + assert.doesNotMatch(r.stdout, /somebody-elses\.log/, "only names our tooling creates"); + assert.match(r.stdout, /zombie builds: 0/); + assert.match(r.stdout, /done status=0/); + } finally { + rmSync(f.base, { recursive: true, force: true }); + } + }); + + it("for real: sweeps the three stale artefacts, keeps the fresh one and the stranger", (t) => { + if (!HAVE_BUSY_TOOLS) return t.skip("lsof absent on this box — sweep branch covered in CI"); + const f = fixture(); + try { + const r = run([], f.base); + assert.equal(r.status, 0, r.stderr + r.stdout); + assert.ok(!existsSync(f.staleTar), "stale e2e-build.tar.gz must go (it is RAM on tmpfs)"); + assert.ok(!existsSync(f.staleBuild), "stale next-build dir must go"); + assert.ok(!existsSync(f.staleUpgrade), "stale install-upgrade dir must go"); + assert.ok(existsSync(f.fresh), "a fresh dir must survive"); + assert.ok(existsSync(f.unrelated), "files we did not create must survive even when old"); + } finally { + rmSync(f.base, { recursive: true, force: true }); + } + }); + + it("tmpfs fuse is shorter than the disk fuse (RAM vs disk), both overridable", () => { + const f = fixture(); + try { + // With a 6h tmpfs fuse the 5h-old artefacts are NOT stale yet. + const r = run(["--dry-run"], f.base, { TMPFS_MAX_AGE_HOURS: "6" }); + assert.doesNotMatch( + r.stdout, + /would remove|removed \(|cannot prove idle/, + "nothing is stale under a 6h fuse, so no candidate is even examined" + ); + const body = readFileSync(SCRIPT, "utf8"); + assert.match(body, /TMPFS_MAX_AGE_HOURS:-3\}/, "tmpfs default must stay short — it is RAM"); + assert.match(body, /WORK_TEMP_MAX_AGE_HOURS:-24\}/); + } finally { + rmSync(f.base, { recursive: true, force: true }); + } + }); + + it("alerts (exit 1) on disk and memory pressure thresholds without touching files", () => { + const f = fixture(); + try { + writeFileSync( + path.join(f.base, "psi"), + "some avg10=0.00 avg60=0.00 avg300=0.00 total=1\nfull avg10=0.00 avg60=23.50 avg300=9.00 total=1\n" + ); + const r = run(["--dry-run"], f.base, { + JANITOR_PSI_FILE: path.join(f.base, "psi"), + DISK_ALERT_PCT: "0", + }); + assert.equal(r.status, 1, "attention needed must be exit 1 for the cron log"); + assert.match(r.stdout, /MEMORY PRESSURE psi full\/avg60=23\.50%/); + assert.ok(existsSync(f.fresh) && existsSync(f.unrelated)); + assert.match(r.stdout, /ROOT DISK \d+% >= 0%/); + assert.ok(existsSync(f.staleTar), "alerting never deletes"); + } finally { + rmSync(f.base, { recursive: true, force: true }); + } + }); + + it("rejects unknown arguments instead of silently running", () => { + const r = run(["--yolo"], os.tmpdir()); + assert.equal(r.status, 2); + }); +});