Files
OmniRoute/tests/unit
Innokentiy Solntsev e7872e57c3 fix(db): reclaim freed pages incrementally instead of a blocking VACUUM in the cleanup scheduler (#12821) (#12830)
* fix(db): reclaim freed pages incrementally instead of a blocking VACUUM in the cleanup scheduler (#12821)

startCleanupScheduler() ran a synchronous whole-database VACUUM on the event
loop whenever a cleanup pass deleted at least one row - 30 s after every
start and every 6 h. With node:sqlite that blocks every route (/healthz
included) for the duration: 7 min 55 s on a 540 MB storage.sqlite to reclaim
six rows. It also bypassed vacuumScheduler, the app-level owner of full
VACUUMs and the operator's scheduledVacuum / vacuumHour settings.

cleanup.ts no longer issues a full VACUUM. After each pass reclaimFreedPages()
branches on PRAGMA auto_vacuum:

- INCREMENTAL: drain the freelist with PRAGMA incremental_vacuum(N) in ~1 MiB
  batches (N from page_size), pausing between batches for as long as the last
  one took (<=250 ms), PASSIVE checkpoint every 64 batches and a TRUNCATE
  checkpoint at the end so the main file shrinks in WAL mode; hard caps of
  2048 batches / 30 s per pass, the remainder waits for the next pass.
- FULL: nothing to do, SQLite reclaims on commit.
- NONE: incremental_vacuum is a no-op, so record a request via the new
  vacuumScheduler.requestFullVacuum(); the rebuild runs in the configured
  window (or via the Storage page button). scheduledVacuum=never is honored.

vacuumScheduler persists fullVacuumRequestedAt / fullVacuumRequestReason,
clears them on the next successful runNow(), and hydrates from key_value
before an early request so it cannot clobber a persisted lastRunAt.

Loop robustness: db.exec() rather than pragma() (bun:sqlite's all() steps a
zero-column pragma once), SQLITE_BUSY/LOCKED and a handle closed under the
pass stop it quietly, other errors stop it with partial progress logged.
Also drops the duplicate cleanupProxyLogs() call in the scheduled pass -
runAutoCleanup() already covers proxy_logs.

Tests: new tests/unit/db/cleanup-reclaim-freed-pages.test.ts (INCREMENTAL
drain/pause/checkpoint, page_size-derived batch, caps, FULL no-op, NONE
defers and leaves page_count untouched, runScheduledCleanupPass() path);
vacuum-scheduler.test.ts covers requestFullVacuum persistence, restart
survival and clearing; cleanup-column-fix.test.mjs now asserts
incremental_vacuum and the absence of a full VACUUM statement.

* chore(changelog): name the #12821 fragment after its PR (#12830)

* fix(db): extract reclaimFreedPages into its own module and fix full-suite regressions

Split the #12821 incremental-vacuum reclamation logic out of cleanup.ts
into src/lib/db/reclaimFreedPages.ts (re-exported for callers/tests) so
cleanup.ts stays under the file-size cap after the #13011 reconciliation
merge grew it past the 1200-line threshold.

Also fixes two full-suite failures surfaced by running the
cleanup/vacuumScheduler/db-health suite post-merge (not just this PR's
own 3 test files, per the plan-file's mandatory item):

- tests/unit/cleanup-column-fix.test.mjs scanned cleanup.ts's raw source
  for the PRAGMA incremental_vacuum invariant, which now lives in the
  extracted module — updated to scan both files.
- tests/unit/db/cleanup-reclaim-freed-pages.test.ts asserted the freelist
  count is byte-for-byte unchanged when auto_vacuum=NONE. The tip's
  runAutoCleanup() now also runs cleanupCompressionRunTelemetry(), which
  lazily creates its table on first use (ensureCompressionRunTelemetryTable)
  — a legitimate one-time page cost from a freshly migrated DB, unrelated
  to reclaimFreedPages()'s own behavior. Loosened the assertion to a small
  tolerance while keeping the page_count assertion that actually guards
  against a full rebuild.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* chore(db): drop the reclaimable-bytes VACUUM gate test superseded by incremental reclaim

tests/unit/vacuum-reclaimable-threshold.test.ts pinned cleanup.ts's
vacuumAfterCleanup()/getReclaimableBytes()/getVacuumMinReclaimableBytes()
(#13079). This branch removes the inline post-cleanup full VACUUM entirely in
favour of reclaimFreedPages() (#12821), which reads the same freelist_count /
page_size signal and defers a full VACUUM to the vacuum scheduler when
auto_vacuum=NONE. With those three exports gone the file cannot compile, and
the behaviour it guarded no longer exists.

---------

Co-authored-by: insoln <is@careerum.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
2026-09-17 18:38:26 -03:00
..