* test(infra): retry recursive temp-dir removal instead of failing a shard on ENOTEMPTY (#11966)
Two shards on release/v3.8.51 went red in one day with the same signature —
"ENOTEMPTY, Directory not empty: /tmp/omniroute-<test>-XXXXXX" — from
combo-same-provider-cascade (Unit Tests fast-path 4/4, on a PR that touches only
.github/) and auth-policy-embeddings-webfetch-7785 (the 20k-test TIA step). Both pass
alone and on re-run: the cleanup races something still writing into the directory
(SQLite WAL/-shm checkpoint, a worker, the backup) and under a loaded hosted runner
the window opens. 1154 test files do their own cleanup with
fs.rmSync(dir, { recursive: true, force: true }); 57 already asked for retries.
One-shot codemod (scripts/ad-hoc/codemod-rm-maxretries.mjs, kept for the record):
every rm / rmSync / rmdirSync option object with `recursive: true` and no
`maxRetries` gains `maxRetries: 5, retryDelay: 100` — Node itself then retries
ENOTEMPTY/EBUSY/EPERM for up to ~0.5 s before giving up. 2243 call sites in 1292
files under tests/, the shared tests/_setup/isolateDataDir.ts exit hook included.
Only the option object changes: no call site, assertion or import is touched.
Validation: prettier and ESLint (with the frozen suppressions) clean on all 1292
files; a random 20-file sample runs green (quota-redis-store hangs identically on
the untouched tree — it needs a Redis on localhost, an environment matter). The
four unit shards on this PR are the full run.
* fix(quality): let check-forgotten-sibling-tests read a 1,000-file diff
The gate shells out to `git diff` through execFileSync with Node's default 1 MB
maxBuffer; the 1,292-file codemod in this PR is the first diff large enough to
overflow it, and the gate died with `spawnSync git ENOBUFS` before comparing
anything. 64 MB is far above any real PR and costs nothing when unused.
* chore(ops): make the runner janitor act on what it can prove, not advise
The .113 janitor already knew the rules and had been shouting them into a log
nobody reads: on 2026-08-28 12:00Z it reported "10 listeners > ceiling 8" and
"disk 85%" — for hours — while 6.7 GB of dead-run leftovers sat on the 12 GB
tmpfs (RAM) because its patterns matched neither e2e-build.tar.gz nor
next-build/, its 24 h fuse is a day too long for memory, and its _work/_temp
base (/home/*/actions-runner*) does not exist on this box (runners live under
/opt). Measured while draining the v3.8.50 npm publish (postmortem, Parte III).
What changes:
- idle is PROVEN before removal, with ONE lsof snapshot filtered to the swept
bases (lsof +D per path walked whole trees and took minutes; 460 candidates
grepping a re-printed 83k-line string was the other half). 20 s on the box.
Without lsof the janitor removes nothing and says why (exit 1).
- tmpfs leftovers go after 3 h, disk _work/_temp after 24 h; both overridable.
Patterns gain next-build* and e2e-build.tar.gz; /opt/actions-runner* is swept.
- zombie builds: a next-build older than 75 min has no job (a real Build step is
~26 min). On 2026-08-27 one ran 70 min after GitHub had declared its job lost,
holding 3.6 GB. KillMode=mixed on the units covers systemctl stop/restart;
this covers the lost-connection path.
- prunes 48 h-old checkouts under _work of runners whose unit is STOPPED — an
active runner is never touched.
- alerts on memory PSI (full/avg60) and reports the listener ceiling with an
omniroute/other breakdown (the box also hosts OmniHeuris and OmniMind).
Enforcing the ceiling stays an operator decision (label split), not cron's.
- --dry-run prints exactly what it would do and touches nothing; unknown
arguments are rejected.
Dry-run on the real box: 460 stale omniroute-* test fixtures (930 MB of RAM) it
would reclaim, 0 busy, 0 false "removed" lines, 20 s. The unit suite drives the
script against a fixture tree with every base redirected; the sweep branch runs
where lsof exists (hosted CI images) and the without-lsof contract everywhere.
docs/ops/RUNNER_BOX.md reconciled to the measured box: 31 GB (it said 16), ten
listeners, the 14 GB next-build ceiling, the KillMode drop-in, and the rule that
nothing is cleaned by hand while a runner is busy.
* docs(ops): restore the frontmatter fumadocs requires on RUNNER_BOX.md
Rewriting the page whole dropped its `title:` frontmatter, and docs/ is
compiled into the Next build by fumadocs-mdx — so Build, Fast Production Build
and dast-smoke all died with "[MDX] invalid frontmatter in
docs/ops/RUNNER_BOX.md". Same block as before, verbatim.
