mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-08-03 05:45:04 +03:00
7fefc6782b
* docs(ops): add canonical incident response runbook
Adds a 5-level severity incident response runbook with role
assignments, communication templates, and post-mortem cadence.
### Files (1 changed, +X / -0)
- docs/INCIDENT_RESPONSE.md — incident classification, response
roles per severity (sev1/sev2/sev3/sev4/sev5), pager rotation,
status page templates, post-mortem schedule (within 5 business
days of sev1/sev2 resolution)
### Why this matters
- diegosouzapw/OmniRoute has no incident response runbook as of 2026-06-23
- The 71-pillar framework (Observability & Ops domain, L56–L63)
flags incident response as P0 for any production-serving surface
- Establishes the on-call rotation + escalation paths in writing
- Post-mortem template is the load-bearing artifact (no-blame
culture, 5-business-day deadline, action item tracking)
### Compatibility
- Pure documentation — no code change, zero behavior change
- Single file, lands in one commit
- No new dependencies
* chore(ci): add .mergify.yml to main — Mergify only reads config from the default branch (#7168)
* fix(ci): add the auto-enqueue pull_request_rule to the Mergify config (queue_conditions alone are eligibility-only) (#7179)
* fix(ci): migrate Mergify auto-enqueue to merge_protections_settings.auto_merge_conditions (rules-based path is EOL 2026-07-16) (#7216)
* fix(ci): drop Mergify batch settings (batching is a paid-tier feature; free plan queue is serial) (#7220)
* fix(ci): merge queue tolerates the advisory dast-smoke failure (its GH-hosted build hang dequeued every attempt) (#7225)
* test(ci): make the #6634 selfref guard hermetic — main's copy hard-fails every PR (#7341)
main's copy of this test still does git I/O inside a unit test:
const baseSrc = git(['show', 'origin/main:' + FILE]);
Runners check out a shallow single ref, so origin/main does not resolve and the
test dies with 'fatal: invalid object name origin/main'. Every PR into main
fails Unit Tests (7/8) on it — today that is #7313, #7315, #7316, #7334, #7336
and #7337, six PRs red on a defect none of them introduced. #7313 has no other
red at all.
release/v3.8.49 already carries a fix (2e42b8efc, #7174: try/catch, fetch
origin/main on demand, t.skip() when unreachable), but it only reaches main at
release time — so main stays broken for the whole cycle. Cherry-picking it would
also import a new problem: PR Test Policy classifies t.skip() as a silenced
assertion, which we watched it correctly catch on #7300 today.
This is the hermetic version instead (ported from #7327, which does the same for
the release branch): read the file straight off disk, compare against an empty
base so baseTaut/baseExtTaut are 0 — the strictest possible comparison point —
and call evaluateMasking() directly. No git ref, no fetch, no skip, nothing the
runner's checkout depth can break.
The #6634 regression stays covered: the guard's logic lives in
SELF_TEST_FIXTURE_RE (check-test-masking.mjs:337), not in the test. Proven both
ways on main before committing — neutralise SELF_TEST_FIXTURE_RE to /$^/ and
the test FAILS; restore it and it passes 2/2, with check-test-masking.mjs left
byte-identical.
Co-authored-by: growab <nekron@icloud.com>
* chore(quality): tighten main's coverage baseline to the CI's real numbers (#7347)
main's ratchet had been failing --require-tighten on every PR: 11 metrics
improved but the baseline was never tightened. Same class as the #6634
selfref guard — an infra fix that lands only on the release branch leaves
main red for the whole cycle, and every PR into main pays for it.
Values are the merged-coverage numbers from a run on main itself (a local
run measures ~68% vs CI's ~80%; the baseline's own note warns about that
gap). Only the 11 coverage values change — gitleaks and semgrepFindings
keep main's own state.
No changelog fragment: #7326 carries it on release/v3.8.49, and a second
one here would double the entry at release time.
* docs(ops): fix incident-response runbook factual accuracy issues
Review on PR #7334 found several fabricated/foreign-template references in
docs/INCIDENT_RESPONSE.md that would misdirect an on-call engineer during a
real incident:
- Sec 3 and 4.1 cited a nonexistent `POST /api/providers/{id}/disable`
endpoint. The real mechanism is per-connection:
`PUT /api/providers/{connectionId}` with `{ "isActive": false }`
(src/app/api/providers/[id]/route.ts). There is no single whole-provider
kill switch, so the steps now say to repeat per connection/key, or rely
on the automatic provider circuit breaker / Model Lockout described in
docs/architecture/RESILIENCE_GUIDE.md. Also drops the equally fabricated
"disable path" pointer at src/lib/a2a/skills/providerDiscovery.ts, which
has no such function.
- Sec 4.3 cited a `policies_active` field on GET /api/settings/authz-inventory
that does not exist; the route actually returns a route-tier inventory
(tiers/bypassEnabled/bypassPrefixes/spawnCapablePrefixes/cors). Rewrote
the check against the real shape and added a fallback signal
(JWT_SECRET/API_KEY_SECRET + isValidApiKey's DB reachability) for a
genuine all-keys auth outage.
- Stripped leftover "phenotype" branding (phenotype.slack.com,
grafana.phenotype.internal, status.phenotype.dev, announce@phenotype.dev)
copy-pasted from another org's template, replacing with explicit TBD
placeholders rather than inventing new unverified URLs.
- Fixed the fabricated ADR-024/ADR-029 citations — this repo has no ADR
directory; pointed at the real convention in
docs/architecture/cluster-decisions.md (ADR-041) instead.
- Fixed the #omnirouse-ops-handoff typo -> #omniroute-ops-handoff.
- Added a changelog.d/ fragment per CONTRIBUTING.md convention.
Docs-only change; no production code touched.
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
---------
Co-authored-by: KooshaPari <kooshapari@users.noreply.github.com>
Co-authored-by: Diego Rodrigues de Sa e Souza <8016841+diegosouzapw@users.noreply.github.com>
Co-authored-by: growab <nekron@icloud.com>
Co-authored-by: KooshaPari <1000+KooshaPari@users.noreply.github.com>
195 lines
9.3 KiB
Markdown
195 lines
9.3 KiB
Markdown
# Incident Response Runbook — OmniRoute (2026-06-18)
|
||
|
||
**Status**: Authoritative. The 71-pillar audit (L61) references this doc
|
||
for the `Obs > 2.00` gate.
|
||
**Owner**: observability-circle (lead: security-circle lead).
|
||
**SLOs**: see `docs/PERF_BUDGETS.md` § 1 (top-level SLOs) and
|
||
`ops/slos.yaml` (machine-readable form, generated by the Bifrost team).
|
||
**Disclosure policy**: see `SECURITY.md` (vulnerability disclosure only,
|
||
separate flow).
|
||
|
||
This runbook is the operational playbook for **non-security** incidents:
|
||
outages, latency regressions, error-budget burn, and provider-side
|
||
failures. Vulnerability disclosure stays on `SECURITY.md`; do not route
|
||
those through this runbook.
|
||
|
||
---
|
||
|
||
## 1. Severity ladder
|
||
|
||
| Sev | Definition | Examples | Page on | Resolve by |
|
||
|---|---|---|---|---|
|
||
| **SEV-1** | User-visible outage; > 50 % of requests failing or > 2x SLO breach for 5 min. | Cluster down; auth layer broken; 5xx flood. | On-call P0 (immediate) | 4 h |
|
||
| **SEV-2** | Significant degradation; 1.5–2x SLO breach for 15 min, or single-tenant impact. | Single provider down; p95 > 1.5x budget; rate-limit runaway. | On-call P1 (15 min) | 24 h |
|
||
| **SEV-3** | Latent bug or near-miss; no current user impact but error budget at risk. | Memory leak trending up; circuit breaker tripping on one provider. | Slack `#omniroute-ops` (next standup) | 7 d |
|
||
| **SEV-4** | Cosmetic / informational. | Log line noise; non-binding UI glitch. | Next weekly review | Next refactor cycle |
|
||
|
||
**Burn-rate escalation** (per `docs/PERF_BUDGETS.md` § 1): 6x for 5 min
|
||
is SEV-1; 2x for 1 h is SEV-2; sustained < 1x for 7 d demotes to SEV-3.
|
||
|
||
---
|
||
|
||
## 2. Detection sources
|
||
|
||
| Source | Signal | Routing |
|
||
|---|---|---|
|
||
| Prometheus (`/metrics`) | Counter deltas (5xx, latency) | Alertmanager → PagerDuty |
|
||
| Grafana SLO dashboards | SLO burn-rate panels | Slack `#omniroute-ops` |
|
||
| Uptime probe (`/api/health/ping`) | 3 consecutive failures from 3 regions | Alertmanager → PagerDuty |
|
||
| Dependabot | New CVE in dependency | GitHub issue + Slack `#security` |
|
||
| User report (support@) | Manual triage | Slack `#omniroute-triage` |
|
||
| Error budget burn alert | `slo_burn_rate > threshold` | Alertmanager |
|
||
|
||
Prometheus and Alertmanager are configured in the deploy repo (see
|
||
`docs/operations/DEPLOY.md` once published; currently inline in
|
||
`docker-compose.prod.yml`).
|
||
|
||
---
|
||
|
||
## 3. First-15-minutes checklist
|
||
|
||
When paged, the on-call engineer runs this checklist verbatim. **Do
|
||
not** skip steps; each is timed.
|
||
|
||
1. **0:00** — Acknowledge the page in PagerDuty. Stops the escalation
|
||
timer and notifies the secondary.
|
||
2. **0:02** — Open the [SLO dashboard][dash] and the [incident
|
||
channel][chan] (`#inc-YYYY-MM-DD-slug`). Post a single-line ack
|
||
with the alert name and the time.
|
||
3. **0:05** — Classify severity per § 1. If SEV-1 or SEV-2, declare
|
||
the incident in the channel and tag `@incident-commander`.
|
||
4. **0:08** — Capture the alert payload, the most recent deploy SHA,
|
||
and the top 5 slow / erroring endpoints. Post to the channel.
|
||
5. **0:12** — Decide: **mitigate first, root-cause later**. Choose
|
||
one of:
|
||
- **Roll back** to the last green deploy (`bin/rollback.sh vX.Y.Z`).
|
||
- **Failover** to the healthy replicas (Caddy LB removes the bad
|
||
replica automatically; verify with `curl /api/health/ping`).
|
||
- **Disable** the broken connection(s) via `PUT /api/providers/{connectionId}`
|
||
with body `{ "isActive": false }` (per-connection toggle, safe by
|
||
default; repeat per key/account — see § 4.1).
|
||
6. **0:15** — Post the chosen mitigation in the channel. If the page
|
||
is still firing after 5 more minutes, escalate to the secondary.
|
||
|
||
[chan]: TBD — set to your team's incident-chat channel (e.g. a Discord/Slack `#inc-*` channel); not provisioned by this repo.
|
||
[dash]: TBD — set to your Grafana/observability dashboard URL; not provisioned by this repo.
|
||
|
||
---
|
||
|
||
## 4. Mitigation runbooks (per failure mode)
|
||
|
||
### 4.1 Provider outage (single provider down)
|
||
|
||
1. `PUT /api/providers/{connectionId}` with body `{ "isActive": false }` —
|
||
deactivates that connection; combo routing and account selection skip it
|
||
on the next request (`src/app/api/providers/[id]/route.ts`). There is no
|
||
single whole-provider kill switch — if the provider has more than one
|
||
key/account, repeat per connection, or let the automatic provider circuit
|
||
breaker trip on its own (`src/shared/utils/circuitBreaker.ts`,
|
||
`domain_circuit_breakers` table; see `docs/architecture/RESILIENCE_GUIDE.md`).
|
||
2. Verify p95 returns to budget within 5 min.
|
||
3. If all connections for a model are down, apply the same `isActive: false`
|
||
toggle to every connection offering that model — there is no separate
|
||
per-model disable endpoint. Combo routing's automatic Model Lockout
|
||
(`open-sse/services/accountFallback.ts`; see
|
||
`docs/architecture/RESILIENCE_GUIDE.md`) also skips a model that keeps
|
||
erroring, without manual action.
|
||
4. Update the status page (if one is configured — see § 5) with a banner if
|
||
the outage exceeds 15 min.
|
||
|
||
### 4.2 Cluster-wide latency regression
|
||
|
||
1. Check the most recent deploy (`/api/version` returns the SHA).
|
||
2. If p95 doubled vs the 7-day baseline, **roll back** to the prior
|
||
SHA via `bin/rollback.sh`.
|
||
3. If the regression is provider-side, see § 4.1.
|
||
|
||
### 4.3 Auth layer broken (5xx on /v1/responses for all keys)
|
||
|
||
1. Check the authz-inventory endpoint:
|
||
`curl https://api.omniroute.dev/api/settings/authz-inventory | jq`.
|
||
It returns a route-tier inventory (`tiers`, `bypassEnabled`,
|
||
`bypassPrefixes`, `spawnCapablePrefixes`, `cors` — see
|
||
`src/app/api/settings/authz-inventory/route.ts`); there is no
|
||
`policies_active` field. A non-200 response, or a `tiers` array that
|
||
fails to populate, means the settings/DB layer the auth pipeline reads
|
||
from is down — not just a single bad key.
|
||
2. If the endpoint itself errors or returns malformed data, restore the
|
||
settings store from the last good backup (`bin/restore-policies.sh <sha>`).
|
||
3. If the endpoint is healthy but requests still 5xx for every key, verify
|
||
`JWT_SECRET` / `API_KEY_SECRET` are set and unchanged for this deploy,
|
||
and that `isValidApiKey` (`src/sse/services/auth.ts`) can reach the DB.
|
||
4. Roll back if the cause is unclear.
|
||
|
||
### 4.4 Data-layer incident (sqlite corruption, audit log gap)
|
||
|
||
1. **Stop the cluster** (`docker compose -f docker-compose.prod.yml
|
||
stop`) — preventing further writes is more important than uptime.
|
||
2. Snapshot the data volume (`bin/snapshot-data.sh`).
|
||
3. Open a SEV-1; this is data-loss territory. Page the data-team.
|
||
4. Restore from the last verified backup (see `docs/BACKUP.md` once
|
||
published; currently the runbook is `bin/restore-data.sh <sha>`).
|
||
|
||
### 4.5 Security incident (vulnerability disclosure)
|
||
|
||
**Stop.** This is the `SECURITY.md` path, not this runbook. Page the
|
||
security on-call (`@security-team`); do not post details to
|
||
`#omniroute-ops`.
|
||
|
||
---
|
||
|
||
## 5. Communication
|
||
|
||
| Audience | Channel | Cadence | Owner |
|
||
|---|---|---|---|
|
||
| Engineering | `#inc-YYYY-MM-DD-slug` | Real-time | Incident commander |
|
||
| Status page | TBD — not provisioned by this repo | Every 30 min during SEV-1/2 | On-call |
|
||
| Customers (email) | TBD — set your announcement list/address | At SEV-1 start + resolution | Comms lead |
|
||
| Upstream providers | Direct contact | At SEV-1 start | Vendor mgmt |
|
||
| Postmortem | `docs/postmortem/YYYY-MM-DD-slug.md` | Within 5 business days | Incident commander |
|
||
|
||
Postmortem template is at `docs/postmortem/TEMPLATE.md` (forthcoming; no
|
||
dedicated ADR covers it yet — once written, register it in
|
||
`docs/architecture/cluster-decisions.md` following this repo's 71-pillar/ADR
|
||
numbering convention, e.g. ADR-041 there).
|
||
|
||
---
|
||
|
||
## 6. On-call rotation
|
||
|
||
| Role | Primary | Secondary | Rotation |
|
||
|---|---|---|---|
|
||
| Engineering on-call | security-circle lead | @open-sse | Weekly, Mon 09:00 PDT |
|
||
| Security on-call | @security-team | — | Weekly |
|
||
| Data on-call | @db-team | — | Weekly |
|
||
| Comms lead | @comms | — | As needed |
|
||
|
||
**Handoff**: every Monday 09:00 PDT, the outgoing on-call posts a
|
||
written handoff to the incoming in `#omniroute-ops-handoff` covering:
|
||
open SEV-3/4 items, scheduled maintenance windows, and any
|
||
in-flight mitigations.
|
||
|
||
---
|
||
|
||
## 7. Postmortem expectations
|
||
|
||
- **Blameless**. People did the best they could with the information
|
||
they had. Focus on systems, signals, and decision points.
|
||
- **Within 5 business days** of resolution. File via
|
||
`gh issue create --label postmortem --label SEV-1` (or `--label SEV-2`).
|
||
- **Action items** must be assigned, dated, and tracked in
|
||
`docs/TECH_DEBT.md` (P0 < 30 d, P1 < 90 d per that doc's SLA).
|
||
- **Mandatory attendees**: incident commander, on-call, any engineer
|
||
who touched the mitigation, and one person who was *not* involved
|
||
(fresh-eyes review).
|
||
|
||
---
|
||
|
||
## 8. Review log
|
||
|
||
| Date | Reviewer | Change |
|
||
|---|---|---|
|
||
| 2026-06-18 | security-circle lead | Initial runbook; severity ladder + 15-min checklist + 4.1–4.5 mitigation runbooks. Closes 71-pillar audit L61 (1/3 → 2/3). |
|
||
| 2026-07-18 | observability-circle | Corrected § 4.1/4.3 to the real provider-disable (`PUT /api/providers/{connectionId}`) and authz-inventory (`tiers`/`bypassEnabled`/`cors`, no `policies_active`) mechanisms; removed foreign branding and the nonexistent ADR-024/029 references. |
|
||
| 2026-07-18 (planned) | observability-circle | Wire on-call rotation into PagerDuty schedule; add the postmortem template. |
|