mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-08-02 13:22:11 +03:00
5ac4637816
* docs: move superpowers/research artifacts to isolated _tasks repo + docs tree cleanup
- Move docs/superpowers/{plans,specs} and docs/research/* into the gitignored,
separately-versioned _tasks/ repo; untrack the two tracked research design docs.
- Add CLAUDE.md "Planning & Research Artifacts" section overriding the superpowers
default save paths (docs/... -> _tasks/...); align REPOSITORY_MAP and
DOCUMENTATION_OVERHAUL_PLAN with the new convention.
- Drop 4 now-obsolete /api/discovery/* entries from check-docs-symbols allowlist
(stale-enforcement) and refresh code/spec path comments to _tasks/...
- Sweeps in concurrent docs-tree restructuring (root-level provider/guide docs,
compression spec cleanup, .mcp.json.example removal).
* docs: reorganize docs/ tree + fix stale facts across ~26 docs
Phase A — reorganization:
- Move 7 orphan root docs into subfolders (providers/ created; TIERS+USAGE_QUOTA→guides/;
plugins+PLUGIN_SDK→frameworks/); delete 8 obsolete/redundant docs (SUBMIT_PR superseded
by CONTRIBUTING; DOCUMENTATION_OVERHAUL_PLAN; INCIDENT_RESPONSE/PERF_BUDGETS/THREAT_MODEL;
3 ops snapshots). Rebuild README index (was missing ~40 files) + per-folder meta.json nav.
- Clean 14 dangling doc-path references in bin/ ops scripts, scripts/, workflow, tests;
fix the dockerignore-docs-coverage required-docs path (PROVIDERS→providers/CLAUDE_WEB).
Phase B — content accuracy (verified against code, not the audit summary):
- Functional: ENVIRONMENT flag defaults (INPUT_SANITIZER/MCP_ENFORCE_SCOPES=true,
COMPRESS_DESCRIPTIONS=false, dynamic heap); MCP-SERVER notion tool names (omniroute_*→
notion_*) + counts 87→94; coverage gate 75/70→60/60/60/60 (RELEASE_CHECKLIST, COVERAGE_PLAN,
ERROR_SANITIZATION, CONTRIBUTING); pre-push hook description; regenerate PROVIDER_REFERENCE (237).
- Count drift: providers 237, executors 70, migrations 106, db modules 94, oauth 19,
strategies 17, MCP 94, flags 38, TS 6.0, open-sse ~900/services 294 across architecture/
frameworks/ops docs; AUTO-COMBO 9→12 factors w/ correct DEFAULT_WEIGHTS; REASONING +2
patterns; STEALTH UA defaults; AGENT_PROTOCOLS +cursor-cloud/list-capabilities;
LANGUAGE_PACKS +id pack.
- Kept Node 20 (runtime guard accepts 20.20.2+; only engines is stricter) and MCP scopes=13
(mcpScopes.ts) — both were correct in the docs; corrected only the attribution.
* docs: finish content refresh — compression engines, CLI_TOKEN merge, metadata sweep
- Compression: document the additional built-in engines (CCR, headroom, ionizer,
session-dedup) in COMPRESSION_ENGINES; clarify LLMLingua-2 is the ultra-mode SLM
backend + cross-ref the extra engines in EXTENDING_COMPRESSION; add the id
(Indonesian) language pack to LANGUAGE_PACKS.
- AUTO-COMBO: replace the orphan 'How tiers fit' weight table (stale weights) with a
pointer to the canonical 12-factor DEFAULT_WEIGHTS table.
- Security: merge CLI_TOKEN_AUTH.md (legacy 32-char SHA-256 format) into CLI_TOKEN.md
as a 'Legacy format — still accepted' section (server accepts both HMAC + legacy),
delete CLI_TOKEN_AUTH.md, drop it from the index + security nav.
- Metadata: bump stale frontmatter (version/lastUpdated) to 3.8.40/2026-06-28 across the
doc set audited this pass, and normalize the in-body 'Last updated' header lines to match.
* fix(runtime): drop Node 20 from supported range + align all docs/diagrams/counts
- Node minimum is now 22 (aligned with package.json engines). SUPPORTED_NODE_RANGE in
src/shared/utils/nodeRuntimeSupport.ts (and the bin/ mirror) drops the 20.x line →
'>=22.22.2 <23 || >=24.0.0 <27'; getNodeRuntimeSupport now rejects Node 20 as
unsupported-major. Test updated (TDD): node-runtime-support.test.ts asserts Node 20
rejected. Docs aligned (TROUBLESHOOTING ×2, TERMUX, RELEASE_CHECKLIST, CODEBASE,
CLI-TOOLS, README, llm.txt + 42 i18n llm.txt mirrors, skills/cli-serve).
- Diagrams regenerated: mcp-tools-87 -> mcp-tools-94 (34 base + pool 6 = 94) and
auto-combo-9factor -> auto-combo-12factor (correct DEFAULT_WEIGHTS); SVGs re-rendered
via mermaid-cli; doc refs + diagrams/README updated; fixed a pre-existing broken
resilience-3layers image path.
- CLAUDE.md + AGENTS.md aligned to real counts (237 providers, 94 MCP tools / 34 base,
106 migrations, 94 db modules, 12-factor auto-combo, 17 strategies); README provider
count 231 -> 237; executor count corrected to 68 (provider executors, excl base/index)
and OAuth to 18 across architecture docs. check:docs-all now passes (0 strict drift,
0 broken links); removed dead .mcp.json.example doc link.
* fix(services): update installer Node hint to >=22.22.2 (aligned with dropped Node 20)
* docs: realign counts to current release tip after rebase
The release tip advanced while this work was in flight (Gemini CLI provider/executor
removed by #5246, plus other PRs). Re-counted against the current code and updated:
providers 237->236, executors 68->67, OAuth modules 18->17, open-sse services 294->298;
regenerated PROVIDER_REFERENCE.md (236). check:docs-all passes (0 strict drift).
* docs(changelog) + i18n: record Node 20 drop + fix nodeIncompatibleHint
- CHANGELOG: add [3.8.40] entries for the Node 20.x removal (runtime) and the docs
reorganization/accuracy audit.
- i18n: nodeIncompatibleHint across all 42 locales no longer lists Node 20.x as
supported (ASCII + CJK full-width variants), aligned with the dropped Node 20.
* fix(docs): repair CI breakages from the doc moves
- test: cli-plugin-system asserted docs/dev/plugins.md exists; the file moved to
docs/frameworks/PLUGINS.md — point the test at the new path (Unit fast-path 2/2 fix).
- frontmatter: PLUGINS.md and PLUGIN_SDK.md moved into the fumadocs-indexed
docs/frameworks/ which requires a 'title' frontmatter; the missing frontmatter
failed the Next.js MDX build (dast-smoke 'invalid frontmatter'). Added frontmatter
to both, plus the providers/ docs (consistency; that folder is not indexed).
213 lines
12 KiB
Markdown
213 lines
12 KiB
Markdown
---
|
||
title: "Playground Studio"
|
||
version: 3.8.40
|
||
lastUpdated: 2026-06-28
|
||
---
|
||
|
||
# Playground Studio
|
||
|
||
> **Feature:** Playground Studio — unified AI testing workspace for `/dashboard/playground`.
|
||
> **Plans:** `17-playground-studio-redesign.plan.md` + `_orchestration/master-plan-group-C.md`
|
||
> **Status:** Released in v3.8.6
|
||
|
||
---
|
||
|
||
## Overview
|
||
|
||
Playground Studio transforms `/dashboard/playground` from a simple Monaco-based editor into
|
||
a full-featured testing workspace. It replaces the legacy `page.tsx` with a `PlaygroundStudio`
|
||
shell that renders four tabs and a shared config pane.
|
||
|
||
```
|
||
┌ Playground ──────────────────────────────────────────────────────────┐
|
||
│ [💬 Chat] [⚖ Compare] [{} API] [🔧 Build] 142↑ 38↓ · $0.002 </>│
|
||
├──────────────────────────────────────────┬───────────────────────────┤
|
||
│ {active tab content} │ ─ Config │
|
||
│ │ Endpoint [chat ∨] │
|
||
│ │ Model [gpt-5.4 ∨] │
|
||
│ │ System [textarea] │
|
||
│ │ Temp ▕▕▔▔ 0.7 │
|
||
│ │ Presets [▾ load][save] │
|
||
│ │ [✨ Improve prompt] │
|
||
└──────────────────────────────────────────┴───────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## Tabs
|
||
|
||
### Chat Tab
|
||
|
||
Evolves `ChatPlayground.tsx` into a multi-turn streaming workbench:
|
||
|
||
- Full markdown rendering via `MarkdownMessage.tsx` (code blocks, tables, lists, links).
|
||
- System prompt sourced from the shared Config pane.
|
||
- Token/cost per message (prompt + completion tokens).
|
||
- Regenerate last response.
|
||
- Sends to `POST /v1/chat/completions` with SSE streaming.
|
||
|
||
### Compare Tab
|
||
|
||
The key differentiator for a proxy: run 1 prompt across up to **4 models in parallel**.
|
||
|
||
- Up to 4 columns, each independently streaming from `/v1/chat/completions`.
|
||
- `+ Add model` button (Cmd+K shortcut) to add columns.
|
||
- `Run all ▶` triggers all streams simultaneously via `Promise.all` + per-column `AbortController`.
|
||
- Global **Cancel all** aborts every in-flight stream.
|
||
- Per-column `ProviderMetrics` shows TTFT, TPS, tokens, and estimated cost in real time.
|
||
- Metrics labelled **"client-side estimate"** (D12) — measured from first SSE chunk.
|
||
|
||
### API Tab
|
||
|
||
Preserves 100% of the original Monaco editor for power users (D14):
|
||
|
||
- 10 endpoints: chat completions, completions, embeddings, images, audio, speech, transcriptions, moderations, rerank, search.
|
||
- Multimodal file upload.
|
||
- SSE streaming with real-time output.
|
||
- Wrapped as `ApiTab.tsx` (lazy-loaded, `ssr: false`).
|
||
|
||
### Build Tab
|
||
|
||
Tools/function calling and structured output UI:
|
||
|
||
- `ToolsBuilder.tsx` — add/edit/remove `tools[]` with JSON schema editor per tool.
|
||
Validates parameters via `ToolDefinitionSchema` (Zod).
|
||
- `StructuredOutputEditor.tsx` — toggle JSON mode + JSON schema editor.
|
||
Validates response against schema via `StructuredOutputSchema` (Zod).
|
||
- Sends request to `/v1/chat/completions` with `tools[]` and/or `response_format`.
|
||
|
||
---
|
||
|
||
## Config Pane (Shared)
|
||
|
||
`StudioConfigPane.tsx` — always visible, collapsible.
|
||
|
||
| Field | Component | Notes |
|
||
| -------------- | --------------------- | ---------------------------------------------------------------------- |
|
||
| Endpoint | `<select>` | 10 options matching `PlaygroundEndpoint` |
|
||
| Model | `<input>` | free text, e.g. `openai/gpt-4o` |
|
||
| System prompt | `<textarea>` | fed into all tabs |
|
||
| Parameters | `ParamSliders` | temperature, max_tokens, top_p, presence/frequency penalty, seed, stop |
|
||
| Presets | `PresetPicker` | load/save named config snapshots (persisted in DB) |
|
||
| Improve prompt | `ImprovePromptButton` | opens quota-warning modal, calls `/api/playground/improve-prompt` |
|
||
|
||
State is lifted to `PlaygroundStudio.tsx` and passed down to all tabs. Switching tabs
|
||
preserves config state.
|
||
|
||
---
|
||
|
||
## Top Bar
|
||
|
||
`StudioTopBar.tsx`:
|
||
|
||
- Tab switcher (role="tablist").
|
||
- `TokenCostCounter` — live token (↑/↓) and estimated cost display.
|
||
- Export code button (`</>`) — opens `ExportCodeModal`.
|
||
|
||
---
|
||
|
||
## Export Code Modal
|
||
|
||
`ExportCodeModal.tsx` uses `codeExport.ts` to generate curl / Python / TypeScript snippets
|
||
from the current `PlaygroundState`. API key placeholder is always `$OMNIROUTE_API_KEY` (D11).
|
||
|
||
---
|
||
|
||
## Prompt Improver
|
||
|
||
`ImprovePromptButton.tsx` → `useImprovePrompt.ts` → `POST /api/playground/improve-prompt`:
|
||
|
||
1. Modal warns "will consume quota".
|
||
2. On confirm, sends `{ system, prompt, model, tone }` to the route.
|
||
3. Route calls `/v1/chat/completions` internally with `promptImprover.META_SYSTEM_PROMPT`.
|
||
4. Returns `{ improvedSystem?, improvedPrompt?, tokensIn, tokensOut }`.
|
||
5. UI patches the Config pane system prompt and the Chat tab user prompt.
|
||
|
||
---
|
||
|
||
## Presets
|
||
|
||
`PresetPicker.tsx` → `usePresets.ts` → `/api/playground/presets/*`:
|
||
|
||
- Stored in `playground_presets` SQLite table (migration `084_playground_presets.sql`).
|
||
- Each preset stores: `name`, `endpoint`, `model`, `system`, `params_json`, `created_at`.
|
||
- CRUD: `GET` list, `POST` create, `GET /:id`, `PUT /:id`, `DELETE /:id`.
|
||
|
||
---
|
||
|
||
## Stream Metrics
|
||
|
||
`useStreamMetrics.ts` + `streamMetrics.ts` (pure function):
|
||
|
||
- `start()` — records request start time.
|
||
- `onFirstChunk()` — records TTFT.
|
||
- `onChunk(n)` — accumulates completion token count.
|
||
- `finish(usage?)` — computes final metrics: `ttftMs`, `totalMs`, `tps`, `tokensIn`, `tokensOut`, `costUsd`.
|
||
- Pricing from static table in `src/lib/playground/types.ts` (labelled "estimated" — D13).
|
||
|
||
---
|
||
|
||
## Backend Routes
|
||
|
||
| Method | Path | Handler |
|
||
| -------- | -------------------------------- | ----------------------------------------------------------------------------------------- |
|
||
| `POST` | `/api/playground/improve-prompt` | Zod-validates `ImprovePromptRequestSchema`; calls `/v1/chat/completions` with meta-prompt |
|
||
| `GET` | `/api/playground/presets` | Returns `{ presets: PlaygroundPresetListItem[] }` |
|
||
| `POST` | `/api/playground/presets` | Creates preset; validates `PlaygroundPresetCreateSchema` |
|
||
| `GET` | `/api/playground/presets/:id` | Returns one preset or 404 |
|
||
| `PUT` | `/api/playground/presets/:id` | Partial update |
|
||
| `DELETE` | `/api/playground/presets/:id` | 204 |
|
||
|
||
Auth: optional (`REQUIRE_API_KEY`). Errors via `buildErrorBody()` (Hard Rule #12).
|
||
|
||
---
|
||
|
||
## Key Files
|
||
|
||
| Path | Purpose |
|
||
| -------------------------------------------------------------------------- | --------------------------------------------------- |
|
||
| `src/app/(dashboard)/dashboard/playground/PlaygroundStudio.tsx` | Shell component, tab orchestrator |
|
||
| `src/app/(dashboard)/dashboard/playground/components/StudioTopBar.tsx` | Tabs + counter + export button |
|
||
| `src/app/(dashboard)/dashboard/playground/components/StudioConfigPane.tsx` | Shared config panel |
|
||
| `src/app/(dashboard)/dashboard/playground/components/tabs/ChatTab.tsx` | Chat workbench |
|
||
| `src/app/(dashboard)/dashboard/playground/components/tabs/CompareTab.tsx` | Multi-model compare |
|
||
| `src/app/(dashboard)/dashboard/playground/components/tabs/ApiTab.tsx` | Monaco editor (preserved) |
|
||
| `src/app/(dashboard)/dashboard/playground/components/tabs/BuildTab.tsx` | Tools + structured output |
|
||
| `src/app/(dashboard)/dashboard/playground/components/ExportCodeModal.tsx` | Code export modal |
|
||
| `src/app/(dashboard)/dashboard/playground/components/CompareColumn.tsx` | Single compare column |
|
||
| `src/app/(dashboard)/dashboard/playground/components/ProviderMetrics.tsx` | TTFT/TPS display |
|
||
| `src/app/(dashboard)/dashboard/playground/hooks/useStreamMetrics.ts` | Client-side metric hook |
|
||
| `src/app/(dashboard)/dashboard/playground/hooks/usePresets.ts` | Presets CRUD hook |
|
||
| `src/app/(dashboard)/dashboard/playground/hooks/useImprovePrompt.ts` | Improve-prompt hook |
|
||
| `src/lib/playground/codeExport.ts` | curl/Python/TS generator (shared with Search Tools) |
|
||
| `src/lib/playground/promptImprover.ts` | Meta-prompt builder |
|
||
| `src/lib/playground/streamMetrics.ts` | Pure metrics computation |
|
||
| `src/lib/db/playgroundPresets.ts` | DB module (CRUD) |
|
||
| `src/app/api/playground/improve-prompt/route.ts` | Improve-prompt REST route |
|
||
| `src/app/api/playground/presets/route.ts` | Presets list + create |
|
||
| `src/app/api/playground/presets/[id]/route.ts` | Presets get/update/delete |
|
||
| `src/lib/db/migrations/084_playground_presets.sql` | DB migration |
|
||
|
||
---
|
||
|
||
## Troubleshooting
|
||
|
||
| Symptom | Cause | Fix |
|
||
| -------------------------------------- | ----------------------------- | ------------------------------------------------------------------------------- |
|
||
| Monaco editor not rendering in API tab | SSR loaded Monaco | Verify `ApiTab` uses `dynamic(..., { ssr: false })` |
|
||
| Compare streams fire sequentially | Wrong `Promise.all` usage | All stream starts must be dispatched in one `Promise.all` call |
|
||
| Metrics show `null` TTFT | First chunk handler not wired | Check `useStreamMetrics.onFirstChunk()` is called in the SSE reader loop |
|
||
| Preset not persisting | DB migration not run | Run `npm run db:migrate` or restart the server (migration auto-runs on startup) |
|
||
| Improve prompt returns 502 | Model not set in Config | User must enter a model name in the Config pane before improving |
|
||
| Export code shows `MISSING_API_KEY` | Placeholder not inserted | `codeExport.ts` always uses `API_KEY_PLACEHOLDER = "$OMNIROUTE_API_KEY"` |
|
||
|
||
---
|
||
|
||
## References
|
||
|
||
- Master plan: `_tasks/features-v3.8.6/refactorpages/_orchestration/master-plan-group-C.md`
|
||
- Feature plan: `_tasks/features-v3.8.6/refactorpages/17-playground-studio-redesign.plan.md`
|
||
- Code export: `src/lib/playground/codeExport.ts`
|
||
- Prompt improver: `src/lib/playground/promptImprover.ts`
|
||
- Search Tools Studio: `docs/frameworks/SEARCH_TOOLS_STUDIO.md`
|