cherry-pick(pr-9744): test(integration): add general live-test tool for the real "default" combo + rootless wire capture (#9862)

* test(integration): add general live-test tool for the real "default" combo

Temporary WIP commit on this deferred branch — lands in its own separate
PR once the bug-fix extraction batch is done (never bundled into a
bug-fix PR). Unlike liveGeminiShared.ts (provisions its own narrow
2-model Gemini-only combo), this reads the REAL "default" combo
currently configured on the target instance directly from the DB and
exercises every provider/model step in it directly, bypassing combo
routing, so live-test coverage always matches whatever is actually
configured instead of a hardcoded snapshot.

Live-verified against omniroute-beta (seeded with the real 18-model,
5-provider default combo): 14/18 models pass consistently across
non-streaming + streaming Chat Completions and streaming Responses API.
The 4 consistent failures are real external state (cerebras
credits_exhausted, one deprecated openrouter free-tier model), not code
regressions.

(cherry picked from commit c40b13a48fd897259c56f5122e9e57a3dc7654ba)

* test(integration): add rootless wire-capture correlation to the live-test tool

Temporary WIP commit on this deferred branch — lands in the same final
live-test-tool PR as the general default-combo suite, never bundled into
a bug-fix PR.

liveContainerHarness.ts spins up a dedicated, throwaway podman container
(same runner-base image target as the operator's local dev/beta
containers) so wire-capture tests are fully self-contained: builds the
image if missing, starts the container with a persistent data dir, waits
for health, seeds the real "default" combo + provider connections from
the operator's local omniroute-dev instance (idempotent — only runs once
per data dir), and provisions API keys via the running instance's own
auth flow.

wireCapture.ts captures the container's actual network traffic via
`podman unshare nsenter --net=<container netns> -- tcpdump` — no root
needed, verified working live (this generalizes the root-requiring
`sudo nsenter -t $PID` command scripts/sre/tcp-close-analyzer.py already
documented for the same rootless-Podman netns problem; that script's
docstring now documents both). Capture and analysis needed two real fixes
found only by running the pipeline live: `-U` (unbuffered tcpdump writes)
plus a `pkill -f <pcap path>` fallback, since `podman unshare -> nsenter
-> tcpdump` is a 3-level subprocess chain and SIGTERM to the top-level
process doesn't reach the tcpdump grandchild, leaving an orphaned process
and a truncated/unreadable pcap; and filtering on the container's
internal listening port (20128) rather than the dynamically-assigned host
port, since capture happens inside the container's own network namespace
where only the internal port is meaningful.

live-default-combo-wire-capture.test.ts (gated on RUN_LIVE_WIRE_CAPTURE=1)
ties it together: sends a small representative sample of requests through
the real default combo, then cross-checks each one's app-level JSON
status against the actual HTTP status line observed on the wire via
scripts/sre/tcp-close-analyzer.py's stream reassembly — catching bugs
where the app layer claims success but the wire shows a
truncated/reset stream, not just what liveDefaultComboShared.ts's
existing breadth suite already covers.

Live-verified end-to-end: 4/4 sampled requests correlated correctly
across 8 captured TCP streams, container + capture process fully torn
down afterward (verified no orphaned podman container or tcpdump
process left running).

sendModelRequest/filterActiveModelTargets (liveDefaultComboShared.ts) gain
optional baseUrl/apiKey overrides, defaulting to the existing module-level
omniroute-beta target, so the wire-capture suite can point the same
request-sending logic at its own dedicated container instead.

(cherry picked from commit 914a7e42cbe914f257db9f72eedc902ee1532083)

---------

Co-authored-by: Markus Hartung <mail@hartmark.se>

scripts/sre/tcp-close-analyzer.py

@@ -17,8 +17,8 @@ parsing the libpcap file format and IPv4/TCP headers directly. Good enough
 for this one question; not a general-purpose pcap toolkit.
 
 ────────────────────────────────────────────────────────────────────────────
-CAPTURING (run this yourself — needs root/sudo for CAP_NET_RAW; also see
---show-capture-cmd)
+CAPTURING (run this yourself — needs root/sudo for CAP_NET_RAW, UNLESS you
+use the rootless method below; also see --show-capture-cmd)
 ────────────────────────────────────────────────────────────────────────────
 
 Rootless Podman gotcha: there is usually NO `podman3`/`podmanN` bridge
@@ -33,6 +33,24 @@ container's OWN namespace via its PID instead:
     sudo nsenter -t "$PID" -n tcpdump -i any -w /tmp/omniroute-capture.pcap \\
         'host <omniroute-container-ip> and port 20128'
 
+Rootless alternative (NO sudo needed): a bare `nsenter -t $PID -n` fails
+with "Invalid argument" for a rootless container, because its network
+namespace lives inside a user namespace you're not in yet. `podman unshare`
+puts you in that same user namespace first, so `nsenter --net=` against the
+container's netns path succeeds as a plain user — verified working live
+(captured a real `POST /v1/chat/completions` request body in cleartext this
+way, no root at any point):
+
+    NETNS=$(podman inspect omniroute-dev --format '{{.NetworkSettings.SandboxKey}}')
+    podman unshare nsenter --net="$NETNS" -- \\
+        tcpdump -i any -w /tmp/omniroute-capture.pcap 'port 20128'
+
+No `sudo chmod` needed afterward either, since the file was never
+root-owned. This is also what
+tests/integration/wireCapture.ts + liveContainerHarness.ts automate for the
+live wire-capture test suite (its own dedicated throwaway container, not
+omniroute-dev) — see RUN_LIVE_WIRE_CAPTURE=1 in that test file.
+
 Find the container's IP first with:
     podman inspect omniroute-dev --format '{{.NetworkSettings.Networks}}'