docs(ops): add canonical incident response runbook (#4868)

Integrated into release/v3.8.36

docs/INCIDENT_RESPONSE.md

@@ -0,0 +1,174 @@
+# Incident Response Runbook — OmniRoute (2026-06-18)
+
+**Status**: Authoritative. The 71-pillar audit (L61) references this doc
+for the `Obs > 2.00` gate.
+**Owner**: observability-circle (lead: security-circle lead).
+**SLOs**: see `docs/PERF_BUDGETS.md` § 1 (top-level SLOs) and
+`ops/slos.yaml` (machine-readable form, generated by the Bifrost team).
+**Disclosure policy**: see `SECURITY.md` (vulnerability disclosure only,
+separate flow).
+
+This runbook is the operational playbook for **non-security** incidents:
+outages, latency regressions, error-budget burn, and provider-side
+failures. Vulnerability disclosure stays on `SECURITY.md`; do not route
+those through this runbook.
+
+---
+
+## 1. Severity ladder
+
+| Sev | Definition | Examples | Page on | Resolve by |
+|---|---|---|---|---|
+| **SEV-1** | User-visible outage; > 50 % of requests failing or > 2x SLO breach for 5 min. | Cluster down; auth layer broken; 5xx flood. | On-call P0 (immediate) | 4 h |
+| **SEV-2** | Significant degradation; 1.5–2x SLO breach for 15 min, or single-tenant impact. | Single provider down; p95 > 1.5x budget; rate-limit runaway. | On-call P1 (15 min) | 24 h |
+| **SEV-3** | Latent bug or near-miss; no current user impact but error budget at risk. | Memory leak trending up; circuit breaker tripping on one provider. | Slack `#omniroute-ops` (next standup) | 7 d |
+| **SEV-4** | Cosmetic / informational. | Log line noise; non-binding UI glitch. | Next weekly review | Next refactor cycle |
+
+**Burn-rate escalation** (per `docs/PERF_BUDGETS.md` § 1): 6x for 5 min
+is SEV-1; 2x for 1 h is SEV-2; sustained < 1x for 7 d demotes to SEV-3.
+
+---
+
+## 2. Detection sources
+
+| Source | Signal | Routing |
+|---|---|---|
+| Prometheus (`/metrics`) | Counter deltas (5xx, latency) | Alertmanager → PagerDuty |
+| Grafana SLO dashboards | SLO burn-rate panels | Slack `#omniroute-ops` |
+| Uptime probe (`/api/health/ping`) | 3 consecutive failures from 3 regions | Alertmanager → PagerDuty |
+| Dependabot | New CVE in dependency | GitHub issue + Slack `#security` |
+| User report (support@) | Manual triage | Slack `#omniroute-triage` |
+| Error budget burn alert | `slo_burn_rate > threshold` | Alertmanager |
+
+Prometheus and Alertmanager are configured in the deploy repo (see
+`docs/operations/DEPLOY.md` once published; currently inline in
+`docker-compose.prod.yml`).
+
+---
+
+## 3. First-15-minutes checklist
+
+When paged, the on-call engineer runs this checklist verbatim. **Do
+not** skip steps; each is timed.
+
+1. **0:00** — Acknowledge the page in PagerDuty. Stops the escalation
+   timer and notifies the secondary.
+2. **0:02** — Open the [SLO dashboard][dash] and the [incident
+   channel][chan] (`#inc-YYYY-MM-DD-slug`). Post a single-line ack
+   with the alert name and the time.
+3. **0:05** — Classify severity per § 1. If SEV-1 or SEV-2, declare
+   the incident in the channel and tag `@incident-commander`.
+4. **0:08** — Capture the alert payload, the most recent deploy SHA,
+   and the top 5 slow / erroring endpoints. Post to the channel.
+5. **0:12** — Decide: **mitigate first, root-cause later**. Choose
+   one of:
+   - **Roll back** to the last green deploy (`bin/rollback.sh vX.Y.Z`).
+   - **Failover** to the healthy replicas (Caddy LB removes the bad
+     replica automatically; verify with `curl /api/health/ping`).
+   - **Disable** a broken provider via `POST /api/providers/{id}/disable`
+     (one-line toggle; safe by default).
+6. **0:15** — Post the chosen mitigation in the channel. If the page
+   is still firing after 5 more minutes, escalate to the secondary.
+
+[chan]: https://phenotype.slack.com/archives/incidents
+[dash]: https://grafana.phenotype.internal/d/omniroute-slos
+
+---
+
+## 4. Mitigation runbooks (per failure mode)
+
+### 4.1 Provider outage (single provider down)
+
+1. `POST /api/providers/{provider}/disable` — toggles the provider off
+   in the registry; all routes re-resolve on next request.
+2. Verify p95 returns to budget within 5 min.
+3. If all providers for a model are down, **disable the model** (see
+   `src/lib/a2a/skills/providerDiscovery.ts` for the disable path).
+4. Update the [status page][status] with a banner if the outage
+   exceeds 15 min.
+
+[status]: https://status.phenotype.dev
+
+### 4.2 Cluster-wide latency regression
+
+1. Check the most recent deploy (`/api/version` returns the SHA).
+2. If p95 doubled vs the 7-day baseline, **roll back** to the prior
+   SHA via `bin/rollback.sh`.
+3. If the regression is provider-side, see § 4.1.
+
+### 4.3 Auth layer broken (5xx on /v1/responses for all keys)
+
+1. Check the authz-inventory endpoint:
+   `curl https://api.omniroute.dev/api/settings/authz-inventory | jq`.
+2. If `policies_active` is empty, restore from the last good backup
+   (`bin/restore-policies.sh <sha>`).
+3. Roll back if the cause is unclear.
+
+### 4.4 Data-layer incident (sqlite corruption, audit log gap)
+
+1. **Stop the cluster** (`docker compose -f docker-compose.prod.yml
+   stop`) — preventing further writes is more important than uptime.
+2. Snapshot the data volume (`bin/snapshot-data.sh`).
+3. Open a SEV-1; this is data-loss territory. Page the data-team.
+4. Restore from the last verified backup (see `docs/BACKUP.md` once
+   published; currently the runbook is `bin/restore-data.sh <sha>`).
+
+### 4.5 Security incident (vulnerability disclosure)
+
+**Stop.** This is the `SECURITY.md` path, not this runbook. Page the
+security on-call (`@security-team`); do not post details to
+`#omniroute-ops`.
+
+---
+
+## 5. Communication
+
+| Audience | Channel | Cadence | Owner |
+|---|---|---|---|
+| Engineering | `#inc-YYYY-MM-DD-slug` | Real-time | Incident commander |
+| Status page | `status.phenotype.dev` | Every 30 min during SEV-1/2 | On-call |
+| Customers (email) | `announce@phenotype.dev` | At SEV-1 start + resolution | Comms lead |
+| Upstream providers | Direct contact | At SEV-1 start | Vendor mgmt |
+| Postmortem | `docs/postmortem/YYYY-MM-DD-slug.md` | Within 5 business days | Incident commander |
+
+Postmortem template is at `docs/postmortem/TEMPLATE.md` (forthcoming;
+see ADR-024 for the cadence and ADR-029 for the postmortem convention).
+
+---
+
+## 6. On-call rotation
+
+| Role | Primary | Secondary | Rotation |
+|---|---|---|---|
+| Engineering on-call | security-circle lead | @open-sse | Weekly, Mon 09:00 PDT |
+| Security on-call | @security-team | — | Weekly |
+| Data on-call | @db-team | — | Weekly |
+| Comms lead | @comms | — | As needed |
+
+**Handoff**: every Monday 09:00 PDT, the outgoing on-call posts a
+written handoff to the incoming in `#omnirouse-ops-handoff` covering:
+open SEV-3/4 items, scheduled maintenance windows, and any
+in-flight mitigations.
+
+---
+
+## 7. Postmortem expectations
+
+- **Blameless**. People did the best they could with the information
+  they had. Focus on systems, signals, and decision points.
+- **Within 5 business days** of resolution. File via
+  `gh issue create --label postmortem --label SEV-1` (or `--label SEV-2`).
+- **Action items** must be assigned, dated, and tracked in
+  `docs/TECH_DEBT.md` (P0 < 30 d, P1 < 90 d per that doc's SLA).
+- **Mandatory attendees**: incident commander, on-call, any engineer
+  who touched the mitigation, and one person who was *not* involved
+  (fresh-eyes review).
+
+---
+
+## 8. Review log
+
+| Date | Reviewer | Change |
+|---|---|---|
+| 2026-06-18 | security-circle lead | Initial runbook; severity ladder + 15-min checklist + 4.1–4.5 mitigation runbooks. Closes 71-pillar audit L61 (1/3 → 2/3). |
+| 2026-07-18 (planned) | observability-circle | Wire on-call rotation into PagerDuty schedule; add the postmortem template. |