From a0763764901a9543ed229cf1acf9da3d7fcfc9e5 Mon Sep 17 00:00:00 2001 From: Egor Date: Fri, 31 Jul 2026 15:59:23 +0300 Subject: [PATCH] docs: add Rust port feasibility study and deployment infrastructure notes - rust-port-research.md: hot-path map (10 latency-critical ops), measured tiktoken baseline (37.9ms/57K tokens), engine profiles (RTK/headroom/ ionizer/caveman), architecture decision (compression-core crate + N-API, revised per review), phased roadmap, golden-test strategy, risks - infrastructure.md: Proxmox/LXC topology, component table, push flows for Forgejo/OpenHands/OmniRoute/project-history, access notes, do-not-touch - docs/README.md: index links --- docs/README.md | 2 + docs/infrastructure.md | 99 +++++++++++++++++++ docs/rust-port-research.md | 188 +++++++++++++++++++++++++++++++++++++ 3 files changed, 289 insertions(+) create mode 100644 docs/infrastructure.md create mode 100644 docs/rust-port-research.md diff --git a/docs/README.md b/docs/README.md index 9cb3895425..ba32961063 100644 --- a/docs/README.md +++ b/docs/README.md @@ -64,6 +64,8 @@ How the system is put together — read these to understand the runtime, code la - [QUALITY_GATES.md](architecture/QUALITY_GATES.md) — quality-gate scripts and CI jobs inventory. - [MONITORING_SECTIONS.md](architecture/MONITORING_SECTIONS.md) — monitoring/costs dashboard navigation. - [cluster-decisions.md](architecture/cluster-decisions.md) — optional sidecar/cluster profile decisions. +- [rust-port-research.md](rust-port-research.md) — feasibility study for porting the CPU-bound compression/tokenization core to a standalone Rust library (fork research, 2026-07-31). +- [infrastructure.md](infrastructure.md) — deployment topology (Proxmox / LXC 101 / LXC 106 / Forgejo / Docker) and push flows (fork research, 2026-07-31). ## reference/ diff --git a/docs/infrastructure.md b/docs/infrastructure.md new file mode 100644 index 0000000000..3487e1f5e6 --- /dev/null +++ b/docs/infrastructure.md @@ -0,0 +1,99 @@ +# OmniRoute Deployment & Infrastructure + +> **Date**: 2026-07-31 +> **Scope**: Local Mac dev instance + Proxmox/LXC production layout. Context for anyone resuming work. + +## Topology + +``` +┌─────────────────────────────────────────────────────────┐ +│ HOST Proxmox 100.73.44.17 (pve-n150.tailad1b81.ts.net) │ +│ │ +│ tailscaled → holds :443 (Funnel) │ +│ ├─ / → http://192.168.3.101:80/ (Forgejo) │ +│ └─ /omniroute → http://192.168.3.106:20128/ (OmniRoute)│ +│ │ +│ Docker: │ +│ ├─ openhands :3000 (host-network, --privileged) │ +│ └─ amnezia-awg2 :48243/udp (WireGuard, do NOT touch) │ +└───────┬───────────────────────────────────────────────────┘ + │ LXC (lxc-attach -n ) +┌───────┴──────────┐ ┌────────────────────────────────────┐ +│ LXC 101 │ │ LXC 106 (agent-node, 192.168.3.106)│ +│ Forgejo :80 │ │ Docker: │ +│ (git-repositories)│ │ ├─ omniroute :20128 (data→/opt/ │ +└───────────────────┘ │ │ omniroute/data) │ + │ │ └─ openhands :8000/18000/8002 │ + │ │ (OLD duplicate — DELETE) │ + │ ├─ systemd project-history :43128 │ + │ ├─ component-vault :43133 (old) │ + │ └─ iptables: INPUT DROP + ACCEPT │ + │ for 22,20128,43128,43133,... │ + └────────────────────────────────────┘ +``` + +## Components + +| Component | Where | Details | +|---|---|---| +| **Tailscale** | host | `tailscale serve` with Funnel; certs `/var/lib/tailscale/certs/pve-n150.*` | +| **Caddy** | — | **not installed** (no package, no Caddyfile) — HTTPS handled by Tailscale Serve | +| **OpenHands** | host, docker | image `openhands:fixed` (`4e631813f208`), host-network, privileged; DB in `/opt/openhands/workspace/.openhands-state`; created via `docker run -e LLM_MODEL=ds/deepseek-v4-flash -e LLM_BASE_URL=http://192.168.3.106:20128/v1 -e LLM_API_KEY=sk-d146...` (backup: `/opt/openhands/container-config-backup.txt`) | +| **OmniRoute** | LXC 106, docker | image `diegosouzapw/omniroute`, mount `/opt/omniroute/data→/app/data`, cmd `node dev/run-standalone.mjs`; sources/build: `/opt/omniroute-build` (git + Dockerfile + compose) | +| **Forgejo** | LXC 101 | git server, `http://192.168.3.101`, external `https://pve-n150.tailad1b81.ts.net/` (path prefix `/git/`; Gitea 15.0.1) | +| **project-history** | LXC 106, systemd | Rust, `/opt/project-history` (src + binary + data), port 43128 | + +## Pushing changes + +**1. To Forgejo (any session):** remote `http://192.168.3.101/egorich/.git`. + From Mac: `https://egorich:@pve-n150.tailad1b81.ts.net/git/egorich/.git` + ⚠️ URL-encode `@` in the password as `%40`. + +**2. To OpenHands (code/fixes):** image built via `docker commit`, so change = edit inside container + commit image: +```bash +docker exec -it openhands bash # edit /app/openhands/... +docker commit openhands openhands:fixed # fix patch into image +docker restart openhands # apply +``` +Env config (model, key, CORS): recreate container with same command from `/opt/openhands/container-config-backup.txt` + new `-e`. + +**3. To OmniRoute (LXC 106):** +```bash +lxc-attach -n 106 -- bash +cd /opt/omniroute-build # git pull / checkout pr/fix-pack +docker compose -f docker-compose.prod.yml build +docker compose -f docker-compose.prod.yml up -d +``` +Data (providers, keys) in `/opt/omniroute/data` — survives rebuild (volume). + +**4. To project-history (LXC 106):** +```bash +lxc-attach -n 106 -- bash +cd /opt/project-history # or git clone from Forgejo (no .git there!) +# edit src/, then: +cargo build --release +systemctl restart project-history +curl http://127.0.0.1:43128/api/health +``` + +## Exposing a new path over HTTPS +```bash +tailscale serve --bg --set-path /history http://192.168.3.106:43128/ +``` + +## Do NOT touch +- iptables in LXC 106 (INPUT DROP, persistent rules) +- `omniroute` (needed by OpenHands) +- `amnezia-awg2` +- DB `data/project_history.sqlite3` + +## Access notes (Mac) + +- SSH to Proxmox/LXC **does not work** from this Mac (Tailscale is stopped here; ports time out). +- Forgejo API works over `https://pve-n150.tailad1b81.ts.net/git/api/v1/` (Basic auth `egorich`). +- Everything else reachable only from the Proxmox host / LXC sessions. + +## Forgejo repo (created 2026-07-31) + +- `egorich/OmniRoute` — branches `pr/fix-pack` (PR-ready), `feat/personal-build` (full history) +- GitHub PR: https://github.com/diegosouzapw/OmniRoute/pull/9058 diff --git a/docs/rust-port-research.md b/docs/rust-port-research.md new file mode 100644 index 0000000000..d8be3dc686 --- /dev/null +++ b/docs/rust-port-research.md @@ -0,0 +1,188 @@ +# Rust Port Research — OmniRoute Compute Engine Extraction + +> **Date**: 2026-07-31 +> **Status**: Feasibility study (research only, no code written yet) +> **Author**: Egor (fork `Egorich-print/OmniRoute`, branch `feat/personal-build`) +> **Reviewer feedback**: ChatGPT architecture review incorporated below + +## TL;DR + +OmniRoute's latency-critical path is the **CPU-bound deterministic compression + tokenization layer** — not the backend plumbing. Port these pure algorithms to a **standalone Rust library** (`compression-core`) with a thin N-API binding as the primary integration path. Ship in this order: **tiktoken → ionizer → headroom → caveman → RTK** (RTK last — thousands of rules, highest risk). Golden-test JS↔Rust byte-in-byte before replacing anything. + +Target: an independent OSS crate (`ai-compression-engine` / `context-engine`) usable by OmniRoute, OpenCode, Cline, Roo, and any AI proxy — not `omniroute-rust`. + +--- + +## 1. Measured Baseline + +Benchmark on the dev Mac (2026-07-31): + +| Operation | Input | Cost | Notes | +|---|---|---|---| +| `countTextTokens()` — js-tiktoken `cl100k_base` | 230K chars (~57K tokens) | **37.9 ms/count** | Runs per chat request | +| Rust `tiktoken-rs` (est.) | same | ~1-3 ms | 10-30x faster | + +The token counter runs on **every** chat request. Compression runs per-request when the conversation exceeds budget. + +## 2. Hot Path Map (chat streaming request) + +All operations below are synchronous and block the Node event loop. + +| # | Operation | File:Line | CPU | Freq | Rust portability | +|---|---|---|---|---|---| +| 1 | SSE chunk JSON parse/stringify | `open-sse/utils/stream.ts:2391` | Expensive | per-chunk | High (serde) | +| 2 | Tiktoken token counting | `src/shared/utils/tiktokenCounter.ts:54` | Moderate | per-req | **High** | +| 3 | RTK compression (regex/line filtering) | `open-sse/services/compression/engines/rtk/index.ts:525` | Expensive | per-req | **High** | +| 4 | Headroom tabular compaction | `open-sse/services/compression/engines/headroom/index.ts:114` | Moderate | per-req | High | +| 5 | Request format translation | `open-sse/translator/registry.ts:23` | Moderate | per-req | Moderate | +| 6 | SQLite usage persistence | `src/lib/usage/usageHistory.ts:675` | Moderate | per-req | Low (DB coupling) | +| 7 | PII sanitization (SSE transform) | `open-sse/handlers/chatCore/streamingPipeline.ts:91` | Moderate | per-chunk | High (regex) | +| 8 | Memory/skills injection (context merge) | `open-sse/handlers/chatCore.ts:1065` | Cheap | per-req | Moderate | +| 9 | Idempotency/request hashing | `open-sse/handlers/chatCore.ts:608` | Cheap | per-req | High (crypto) | +| 10 | Usage estimation (fallback counting) | `open-sse/utils/usageTracking.ts:560` | Cheap | per-chunk | High | + +**Where time goes (estimate):** network wait ≫ CPU (compression + tiktoken) > DB > per-chunk overhead. + +## 3. Compression Engine Profiles + +### Tiktoken counter — `src/shared/utils/tiktokenCounter.ts` (62 LOC + lib) +- Library: **js-tiktoken** v1.0.21 — pure JS port of tiktoken, no WASM. +- Mechanism: pre-computed BPE rank tables shipped as base64 binary blobs (~6 MB across 6 rank files); byte-pair merge on `TextEncoder` UTF-8 byte arrays. +- Encodings used: `cl100k_base` (default), `o200k_base` (Codex). +- Node deps: `TextEncoder`/`TextDecoder` (built-ins), `base64-js`. +- **Verdict**: pure deterministic BPE → ideal Rust port (`tiktoken-rs` supports cl100k + o200k natively). + +### RTK — `open-sse/services/compression/engines/rtk/` (20 files, ~4000 LOC) +- `index.ts` 706, `commandDetector.ts` 482, `filterLoader.ts` 332, `tomlCompatibility.ts` 334, `learn.ts` 290, `lineFilter.ts` + more. +- Deterministic rule engine: regex-based line classification, keep-patterns for code blocks/JSON, folding/merging rules, tool-call-aware filtering (bash vs non-shell tools). +- Called per-request on the whole messages array; **sync** (no awaits in the core). +- **Verdict**: port last. High effort, high risk — but regex crate gives linear-time matching (no backtracking blowups) and output equivalence is testable via golden tests. + +### Headroom — `engines/headroom/` (~550 LOC) +- "Tabular compaction": replaces array-of-objects message content with compact columnar blocks (```gcf-generic ... ```). Lossless, conservative (only when strictly smaller), never touches system messages. +- **Verdict**: pure deterministic, port after ionizer. + +### Ionizer — `engines/ionizer/` (124 + 205 LOC) +- Lossy statistical sampling of oversized homogeneous JSON arrays: keeps schema + error rows + first/last rows + seeded uniform middle sample. +- Deterministic: FNV-1a hash + mulberry32 PRNG (no Math.random). +- **Verdict**: trivial port, nearly zero risk — do second. + +### Caveman — `engines/cavemanAdapter.ts` + `caveman.ts` (~250 LOC) +- Regex rule-based compaction for `standard` mode. +- **Verdict**: port after headroom. + +### Other engines (not first-wave) +- `relevance/` — keyword scoring (no embeddings/network). +- `session-dedup/` — dedupe via hash, per-request. +- `llm/`, `llmlingua/` — **LLM-dependent (network)**, opt-in, NOT portable to pure CPU core. +- `ccr/` (Content-Addressable Recovery) — stores full original for reconstruction. + +## 4. Orchestrator + +- Entry points: `applyCompression` (sync) / `applyCompressionAsync` (async) — `open-sse/services/compression/strategySelector.ts:255` / `:459`. +- Exported via `open-sse/services/compression/index.ts:86-91`. +- Mode dispatch: `off | rtk | codex-responses | omniglyph | lite | stacked | standard | aggressive | ultra`. +- `stacked` mode runs engines sequentially by `stackPriority` (rtk=10, ionizer=13, headroom=15, ...). +- Called from `chatCore.ts` via dynamic `import()`; sync CPU-bound → blocks event loop. +- **Verdict**: single pure function `(messages, budget, config) → (messages, metrics)` — clean extraction surface for a Rust core. + +## 5. Architecture Decision (revised per ChatGPT review) + +### Recommendation: Rust library + thin N-API binding (NOT sidecar-first) + +``` +crates/ + compression-core/ ← pure algorithms, no I/O, no OmniRoute knowledge + src/ + tiktoken/ (cl100k_base, o200k_base) + ionizer/ + headroom/ + caveman/ + rtk/ (last) + napi/ ← N-API binding (primary integration path, in-process) + sidecar/ ← optional HTTP/Unix-socket server over the same core + cli/ ← CLI harness (bench, golden tests) +``` + +**Why N-API first (vs Unix-socket sidecar):** +- Every sidecar call pays serialize→socket→deserialize→compute→serialize→deserialize. +- For a 2 ms token count, IPC overhead becomes a large fraction of the call. +- N-API is in-process: zero serialization on the hot path, no process management. +- Keep sidecar only if process isolation / multi-language integration is actually needed. + +**Wire format if sidecar is later added:** `bincode` / `postcard` / MessagePack — NOT JSON. Messages are large; JSON round-trip is wasted work. + +### Independence from OmniRoute + +Make it a **standalone OSS project**: `ai-compression-engine` or `context-engine`. + +```rust +// core API surface +pub fn compress(messages: &[Message], config: &CompressionConfig) -> CompressionResult; +pub fn count_tokens(encoding: Encoding, text: &str) -> u64; +``` + +No OmniRoute imports anywhere in `compression-core`. Consumers: OmniRoute, OpenCode, Cline, Roo, any AI proxy. + +## 6. Phased Roadmap (revised) + +| Phase | Work | Effort | Risk | +|---|---|---|---| +| 0 | Baseline benchmark (latency, event-loop blocking, compression %) | 1 day | — | +| 1 | Extract `compression-core` crate + port **tiktoken**, golden tests | 2-3 days | Minimal | +| 2 | Port **ionizer** | 1-2 days | Nearly zero | +| 3 | Port **headroom** | 2-3 days | Low | +| 4 | Port **caveman** | 2 days | Low | +| 5 | Port **RTK** (biggest, do when harness proven) | 5-7 days | High | +| 6 | N-API binding as primary path; feature-flag integration with fallback to JS | 2-3 days | Low | +| 7 | Optional sidecar (isolation/multi-lang) | 2 days | Low | +| 8 | **Second wave**: SSE parser + OpenAI/Claude/Gemini translators (per-chunk hot path) | TBD | High | + +**Total to full compression replacement: ~2-3 weeks.** First measurable win (tiktoken): 2-3 days. + +## 7. Golden Testing (mandatory) + +``` +fixtures/ + conversation1.json + ... + conversation500.json + +JS run → output_a.json +Rust run → output_b.json +assert_eq(output_a, output_b) // byte-in-byte, 100% required +``` + +- Until 100% match, **do not** switch the runtime to Rust. +- This makes even the RTK rewrite safe. +- Also validates tiktoken rank tables (JS vs `tiktoken-rs`) on 100+ varied texts. + +## 8. Fallback Strategy (for OmniRoute integration) + +- New env: `OMNIROUTE_COMPRESSION_SIDECAR` (optional, off by default) or N-API availability check. +- Node code stays untouched; a client wrapper (`src/lib/compression-rust/`) tries Rust → falls back to existing JS path (`applyRtkCompression`, `applyCompression`). +- Zero risk to the current deployment. + +## 9. Deferred / Stay-in-JS + +- Dashboard (Next.js App Router, ~200 pages) — never ported. +- Skills, memory, MCP, guardrail management, quotas, combo config — not latency-critical. +- `llm`/`llmlingua` compression engines — network/LLM-dependent, stay in JS. +- SQLite usage persistence — DB-coupled, stays. + +## 10. Risks + +| Risk | Mitigation | +|---|---| +| JS↔Rust behavioral divergence | Golden tests (500 fixtures, byte-in-byte) | +| RTK `tomlCompatibility.ts` / `learn.ts` / `filterLoader.ts` | Port whole submodules (deterministic); rules format 1:1 | +| Rust toolchain on LXC 106 (aarch64?) | Check `rustup`/`cargo`; multi-stage Docker build or cross-compile on Mac | +| N-API ABI mismatch (Node 26) | Use `napi-rs` (prebuilt binaries, Node-version tolerant) | +| Sidecar JSON overhead | bincode/postcard if sidecar is adopted | + +## 11. Open Questions + +1. Cargo/rustup present on LXC 106, or cross-compile from Mac? +2. Confirm N-API as primary integration path (vs sidecar)? +3. Want the baseline benchmark included in the roadmap before porting? +4. Repo home for `compression-core`: new repo (`ai-compression-engine`) or `crates/` inside OmniRoute fork first?