mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-09-18 21:02:50 +03:00
9c5d60027e
* fix(api): stream /api/logs/export with row cap to prevent V8 heap OOM (#13123) Fixes #13123 GET /api/logs/export buffered every matching row into a single JSON.stringify call with pretty-printing (null,2), roughly doubling the string size. On tables with tens of thousands of rows this crashed the Node process with a V8 heap OOM, taking the gateway down for minutes. Changes: - Stream the response via ReadableStream, serializing one row at a time so peak memory stays bounded regardless of table size. - Add a configurable row cap (limit query param, default 10000, max 50000) so callers cannot accidentally request unbounded exports. - Remove pretty-printing (callers can pretty-print client-side). - Include cap metadata (capped, limit, totalAvailable) when the cap fires so callers know they received a truncated result. - Preserve backward-compatible response envelope: { count, hours, type, logs, ... }. * fix(api): push the /api/logs/export row cap down into the DB layer (#13123) The route-layer streaming + cap from the previous pass still called exportCallLogsSince()/exportProxyLogsSince(), which hydrated and buffered EVERY matching row (including rows beyond the limit) before the cap was ever applied — peak V8 heap was essentially unchanged. Adds countCallLogsSince()/countProxyLogsSince() (cheap COUNT(*), no row hydration, used for totalAvailable) and iterateCallLogsSince()/ iterateProxyLogsSince() that bound the query with SQL LIMIT and yield/hydrate one row at a time: a generator over a LIMIT-bounded id list for call_logs, and fixed-size LIMIT/OFFSET pages for proxy_logs (the shared SqliteAdapter only exposes run/get/all, not a `.iterate()` cursor, so LIMIT/OFFSET pagination is the available cursor-equivalent without widening that interface across all 4 driver adapters). The route now streams from these instead, so the full matching row set is never buffered. Also moves capped/limit/totalAvailable into the response header instead of only the trailer, so a client consuming the stream incrementally learns about truncation before processing every row. Rewrote the test to call the real route.GET handler against a seeded test database instead of a local reimplementation of the stream builder, so a regression in the route or its DB-layer delegates is actually caught. Documents the pre-existing (now more clearly load-bearing) breaking change in a changelog fragment: `limit` defaults to 10,000 rows, so exports that previously returned everything are silently truncated. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> --------- Co-authored-by: Koosha Pari <koosha@phenotype.ai> Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>