From dad55fed0a8d720e595b42bfd0e20b866dbd287f Mon Sep 17 00:00:00 2001 From: KooshaPari <42529354+KooshaPari@users.noreply.github.com> Date: Tue, 23 Jun 2026 18:51:05 -0700 Subject: [PATCH] docs(ops): add canonical incident response runbook (#4868) Integrated into release/v3.8.36 --- docs/INCIDENT_RESPONSE.md | 174 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 174 insertions(+) create mode 100644 docs/INCIDENT_RESPONSE.md diff --git a/docs/INCIDENT_RESPONSE.md b/docs/INCIDENT_RESPONSE.md new file mode 100644 index 0000000000..7b34178248 --- /dev/null +++ b/docs/INCIDENT_RESPONSE.md @@ -0,0 +1,174 @@ +# Incident Response Runbook — OmniRoute (2026-06-18) + +**Status**: Authoritative. The 71-pillar audit (L61) references this doc +for the `Obs > 2.00` gate. +**Owner**: observability-circle (lead: security-circle lead). +**SLOs**: see `docs/PERF_BUDGETS.md` § 1 (top-level SLOs) and +`ops/slos.yaml` (machine-readable form, generated by the Bifrost team). +**Disclosure policy**: see `SECURITY.md` (vulnerability disclosure only, +separate flow). + +This runbook is the operational playbook for **non-security** incidents: +outages, latency regressions, error-budget burn, and provider-side +failures. Vulnerability disclosure stays on `SECURITY.md`; do not route +those through this runbook. + +--- + +## 1. Severity ladder + +| Sev | Definition | Examples | Page on | Resolve by | +|---|---|---|---|---| +| **SEV-1** | User-visible outage; > 50 % of requests failing or > 2x SLO breach for 5 min. | Cluster down; auth layer broken; 5xx flood. | On-call P0 (immediate) | 4 h | +| **SEV-2** | Significant degradation; 1.5–2x SLO breach for 15 min, or single-tenant impact. | Single provider down; p95 > 1.5x budget; rate-limit runaway. | On-call P1 (15 min) | 24 h | +| **SEV-3** | Latent bug or near-miss; no current user impact but error budget at risk. | Memory leak trending up; circuit breaker tripping on one provider. | Slack `#omniroute-ops` (next standup) | 7 d | +| **SEV-4** | Cosmetic / informational. | Log line noise; non-binding UI glitch. | Next weekly review | Next refactor cycle | + +**Burn-rate escalation** (per `docs/PERF_BUDGETS.md` § 1): 6x for 5 min +is SEV-1; 2x for 1 h is SEV-2; sustained < 1x for 7 d demotes to SEV-3. + +--- + +## 2. Detection sources + +| Source | Signal | Routing | +|---|---|---| +| Prometheus (`/metrics`) | Counter deltas (5xx, latency) | Alertmanager → PagerDuty | +| Grafana SLO dashboards | SLO burn-rate panels | Slack `#omniroute-ops` | +| Uptime probe (`/api/health/ping`) | 3 consecutive failures from 3 regions | Alertmanager → PagerDuty | +| Dependabot | New CVE in dependency | GitHub issue + Slack `#security` | +| User report (support@) | Manual triage | Slack `#omniroute-triage` | +| Error budget burn alert | `slo_burn_rate > threshold` | Alertmanager | + +Prometheus and Alertmanager are configured in the deploy repo (see +`docs/operations/DEPLOY.md` once published; currently inline in +`docker-compose.prod.yml`). + +--- + +## 3. First-15-minutes checklist + +When paged, the on-call engineer runs this checklist verbatim. **Do +not** skip steps; each is timed. + +1. **0:00** — Acknowledge the page in PagerDuty. Stops the escalation + timer and notifies the secondary. +2. **0:02** — Open the [SLO dashboard][dash] and the [incident + channel][chan] (`#inc-YYYY-MM-DD-slug`). Post a single-line ack + with the alert name and the time. +3. **0:05** — Classify severity per § 1. If SEV-1 or SEV-2, declare + the incident in the channel and tag `@incident-commander`. +4. **0:08** — Capture the alert payload, the most recent deploy SHA, + and the top 5 slow / erroring endpoints. Post to the channel. +5. **0:12** — Decide: **mitigate first, root-cause later**. Choose + one of: + - **Roll back** to the last green deploy (`bin/rollback.sh vX.Y.Z`). + - **Failover** to the healthy replicas (Caddy LB removes the bad + replica automatically; verify with `curl /api/health/ping`). + - **Disable** a broken provider via `POST /api/providers/{id}/disable` + (one-line toggle; safe by default). +6. **0:15** — Post the chosen mitigation in the channel. If the page + is still firing after 5 more minutes, escalate to the secondary. + +[chan]: https://phenotype.slack.com/archives/incidents +[dash]: https://grafana.phenotype.internal/d/omniroute-slos + +--- + +## 4. Mitigation runbooks (per failure mode) + +### 4.1 Provider outage (single provider down) + +1. `POST /api/providers/{provider}/disable` — toggles the provider off + in the registry; all routes re-resolve on next request. +2. Verify p95 returns to budget within 5 min. +3. If all providers for a model are down, **disable the model** (see + `src/lib/a2a/skills/providerDiscovery.ts` for the disable path). +4. Update the [status page][status] with a banner if the outage + exceeds 15 min. + +[status]: https://status.phenotype.dev + +### 4.2 Cluster-wide latency regression + +1. Check the most recent deploy (`/api/version` returns the SHA). +2. If p95 doubled vs the 7-day baseline, **roll back** to the prior + SHA via `bin/rollback.sh`. +3. If the regression is provider-side, see § 4.1. + +### 4.3 Auth layer broken (5xx on /v1/responses for all keys) + +1. Check the authz-inventory endpoint: + `curl https://api.omniroute.dev/api/settings/authz-inventory | jq`. +2. If `policies_active` is empty, restore from the last good backup + (`bin/restore-policies.sh `). +3. Roll back if the cause is unclear. + +### 4.4 Data-layer incident (sqlite corruption, audit log gap) + +1. **Stop the cluster** (`docker compose -f docker-compose.prod.yml + stop`) — preventing further writes is more important than uptime. +2. Snapshot the data volume (`bin/snapshot-data.sh`). +3. Open a SEV-1; this is data-loss territory. Page the data-team. +4. Restore from the last verified backup (see `docs/BACKUP.md` once + published; currently the runbook is `bin/restore-data.sh `). + +### 4.5 Security incident (vulnerability disclosure) + +**Stop.** This is the `SECURITY.md` path, not this runbook. Page the +security on-call (`@security-team`); do not post details to +`#omniroute-ops`. + +--- + +## 5. Communication + +| Audience | Channel | Cadence | Owner | +|---|---|---|---| +| Engineering | `#inc-YYYY-MM-DD-slug` | Real-time | Incident commander | +| Status page | `status.phenotype.dev` | Every 30 min during SEV-1/2 | On-call | +| Customers (email) | `announce@phenotype.dev` | At SEV-1 start + resolution | Comms lead | +| Upstream providers | Direct contact | At SEV-1 start | Vendor mgmt | +| Postmortem | `docs/postmortem/YYYY-MM-DD-slug.md` | Within 5 business days | Incident commander | + +Postmortem template is at `docs/postmortem/TEMPLATE.md` (forthcoming; +see ADR-024 for the cadence and ADR-029 for the postmortem convention). + +--- + +## 6. On-call rotation + +| Role | Primary | Secondary | Rotation | +|---|---|---|---| +| Engineering on-call | security-circle lead | @open-sse | Weekly, Mon 09:00 PDT | +| Security on-call | @security-team | — | Weekly | +| Data on-call | @db-team | — | Weekly | +| Comms lead | @comms | — | As needed | + +**Handoff**: every Monday 09:00 PDT, the outgoing on-call posts a +written handoff to the incoming in `#omnirouse-ops-handoff` covering: +open SEV-3/4 items, scheduled maintenance windows, and any +in-flight mitigations. + +--- + +## 7. Postmortem expectations + +- **Blameless**. People did the best they could with the information + they had. Focus on systems, signals, and decision points. +- **Within 5 business days** of resolution. File via + `gh issue create --label postmortem --label SEV-1` (or `--label SEV-2`). +- **Action items** must be assigned, dated, and tracked in + `docs/TECH_DEBT.md` (P0 < 30 d, P1 < 90 d per that doc's SLA). +- **Mandatory attendees**: incident commander, on-call, any engineer + who touched the mitigation, and one person who was *not* involved + (fresh-eyes review). + +--- + +## 8. Review log + +| Date | Reviewer | Change | +|---|---|---| +| 2026-06-18 | security-circle lead | Initial runbook; severity ladder + 15-min checklist + 4.1–4.5 mitigation runbooks. Closes 71-pillar audit L61 (1/3 → 2/3). | +| 2026-07-18 (planned) | observability-circle | Wire on-call rotation into PagerDuty schedule; add the postmortem template. |