* test(integration): add general live-test tool for the real "default" combo
Temporary WIP commit on this deferred branch — lands in its own separate
PR once the bug-fix extraction batch is done (never bundled into a
bug-fix PR). Unlike liveGeminiShared.ts (provisions its own narrow
2-model Gemini-only combo), this reads the REAL "default" combo
currently configured on the target instance directly from the DB and
exercises every provider/model step in it directly, bypassing combo
routing, so live-test coverage always matches whatever is actually
configured instead of a hardcoded snapshot.
Live-verified against omniroute-beta (seeded with the real 18-model,
5-provider default combo): 14/18 models pass consistently across
non-streaming + streaming Chat Completions and streaming Responses API.
The 4 consistent failures are real external state (cerebras
credits_exhausted, one deprecated openrouter free-tier model), not code
regressions.
(cherry picked from commit c40b13a48fd897259c56f5122e9e57a3dc7654ba)
* test(integration): add rootless wire-capture correlation to the live-test tool
Temporary WIP commit on this deferred branch — lands in the same final
live-test-tool PR as the general default-combo suite, never bundled into
a bug-fix PR.
liveContainerHarness.ts spins up a dedicated, throwaway podman container
(same runner-base image target as the operator's local dev/beta
containers) so wire-capture tests are fully self-contained: builds the
image if missing, starts the container with a persistent data dir, waits
for health, seeds the real "default" combo + provider connections from
the operator's local omniroute-dev instance (idempotent — only runs once
per data dir), and provisions API keys via the running instance's own
auth flow.
wireCapture.ts captures the container's actual network traffic via
`podman unshare nsenter --net=<container netns> -- tcpdump` — no root
needed, verified working live (this generalizes the root-requiring
`sudo nsenter -t $PID` command scripts/sre/tcp-close-analyzer.py already
documented for the same rootless-Podman netns problem; that script's
docstring now documents both). Capture and analysis needed two real fixes
found only by running the pipeline live: `-U` (unbuffered tcpdump writes)
plus a `pkill -f <pcap path>` fallback, since `podman unshare -> nsenter
-> tcpdump` is a 3-level subprocess chain and SIGTERM to the top-level
process doesn't reach the tcpdump grandchild, leaving an orphaned process
and a truncated/unreadable pcap; and filtering on the container's
internal listening port (20128) rather than the dynamically-assigned host
port, since capture happens inside the container's own network namespace
where only the internal port is meaningful.
live-default-combo-wire-capture.test.ts (gated on RUN_LIVE_WIRE_CAPTURE=1)
ties it together: sends a small representative sample of requests through
the real default combo, then cross-checks each one's app-level JSON
status against the actual HTTP status line observed on the wire via
scripts/sre/tcp-close-analyzer.py's stream reassembly — catching bugs
where the app layer claims success but the wire shows a
truncated/reset stream, not just what liveDefaultComboShared.ts's
existing breadth suite already covers.
Live-verified end-to-end: 4/4 sampled requests correlated correctly
across 8 captured TCP streams, container + capture process fully torn
down afterward (verified no orphaned podman container or tcpdump
process left running).
sendModelRequest/filterActiveModelTargets (liveDefaultComboShared.ts) gain
optional baseUrl/apiKey overrides, defaulting to the existing module-level
omniroute-beta target, so the wire-capture suite can point the same
request-sending logic at its own dedicated container instead.
(cherry picked from commit 914a7e42cbe914f257db9f72eedc902ee1532083)
---------
Co-authored-by: Markus Hartung <mail@hartmark.se>
Dependency-free (stdlib-only) libpcap/Ethernet/IPv4/TCP parser that answers
one question: does OmniRoute or the far end (Caddy, on behalf of whichever
client it's proxying) close the TCP connection first? Dashboard-level 499s
only tell us OmniRoute detected a dropped connection, not which side's FIN/
RST actually landed first -- this settles it from the raw packets.
Handles classic Ethernet and both "Linux cooked" linktypes (SLL/SLL2, what
`tcpdump -i any` produces) since rootless Podman has no host-visible bridge
interface to capture on directly -- the capture instructions in the script
document the nsenter-into-container-netns workaround.
Adds --find to grep every reassembled stream for a literal marker string --
in practice the reliable way to locate one specific request (the
x-correlation-id header isn't echoed on every hop) is dropping a fresh UUID
into an actual chat message and searching for it, then cross-referencing
the matched stream's timing against data/call_logs/<date>/*.json.
Co-authored-by: Markus Hartung <markus.hartream@gmail.com>
