mirror of https://github.com/diegosouzapw/OmniRoute.git synced 2026-07-31 04:12:10 +03:00

Files

KooshaPari 7fefc6782b feat(incident-response): structured incident response templates (#7334 )

* docs(ops): add canonical incident response runbook

Adds a 5-level severity incident response runbook with role
assignments, communication templates, and post-mortem cadence.

### Files (1 changed, +X / -0)

- docs/INCIDENT_RESPONSE.md — incident classification, response
  roles per severity (sev1/sev2/sev3/sev4/sev5), pager rotation,
  status page templates, post-mortem schedule (within 5 business
  days of sev1/sev2 resolution)

### Why this matters

- diegosouzapw/OmniRoute has no incident response runbook as of 2026-06-23
- The 71-pillar framework (Observability & Ops domain, L56–L63)
  flags incident response as P0 for any production-serving surface
- Establishes the on-call rotation + escalation paths in writing
- Post-mortem template is the load-bearing artifact (no-blame
  culture, 5-business-day deadline, action item tracking)

### Compatibility

- Pure documentation — no code change, zero behavior change
- Single file, lands in one commit
- No new dependencies

* chore(ci): add .mergify.yml to main — Mergify only reads config from the default branch (#7168)

* fix(ci): add the auto-enqueue pull_request_rule to the Mergify config (queue_conditions alone are eligibility-only) (#7179)

* fix(ci): migrate Mergify auto-enqueue to merge_protections_settings.auto_merge_conditions (rules-based path is EOL 2026-07-16) (#7216)

* fix(ci): drop Mergify batch settings (batching is a paid-tier feature; free plan queue is serial) (#7220)

* fix(ci): merge queue tolerates the advisory dast-smoke failure (its GH-hosted build hang dequeued every attempt) (#7225)

* test(ci): make the #6634 selfref guard hermetic — main's copy hard-fails every PR (#7341)

main's copy of this test still does git I/O inside a unit test:

    const baseSrc = git(['show', 'origin/main:' + FILE]);

Runners check out a shallow single ref, so origin/main does not resolve and the
test dies with 'fatal: invalid object name origin/main'. Every PR into main
fails Unit Tests (7/8) on it — today that is #7313, #7315, #7316, #7334, #7336
and #7337, six PRs red on a defect none of them introduced. #7313 has no other
red at all.

release/v3.8.49 already carries a fix (2e42b8efc, #7174: try/catch, fetch
origin/main on demand, t.skip() when unreachable), but it only reaches main at
release time — so main stays broken for the whole cycle. Cherry-picking it would
also import a new problem: PR Test Policy classifies t.skip() as a silenced
assertion, which we watched it correctly catch on #7300 today.

This is the hermetic version instead (ported from #7327, which does the same for
the release branch): read the file straight off disk, compare against an empty
base so baseTaut/baseExtTaut are 0 — the strictest possible comparison point —
and call evaluateMasking() directly. No git ref, no fetch, no skip, nothing the
runner's checkout depth can break.

The #6634 regression stays covered: the guard's logic lives in
SELF_TEST_FIXTURE_RE (check-test-masking.mjs:337), not in the test. Proven both
ways on main before committing — neutralise SELF_TEST_FIXTURE_RE to /$^/ and
the test FAILS; restore it and it passes 2/2, with check-test-masking.mjs left
byte-identical.

Co-authored-by: growab <nekron@icloud.com>

* chore(quality): tighten main's coverage baseline to the CI's real numbers (#7347)

main's ratchet had been failing --require-tighten on every PR: 11 metrics
improved but the baseline was never tightened. Same class as the #6634
selfref guard — an infra fix that lands only on the release branch leaves
main red for the whole cycle, and every PR into main pays for it.

Values are the merged-coverage numbers from a run on main itself (a local
run measures ~68% vs CI's ~80%; the baseline's own note warns about that
gap). Only the 11 coverage values change — gitleaks and semgrepFindings
keep main's own state.

No changelog fragment: #7326 carries it on release/v3.8.49, and a second
one here would double the entry at release time.

* docs(ops): fix incident-response runbook factual accuracy issues

Review on PR #7334 found several fabricated/foreign-template references in
docs/INCIDENT_RESPONSE.md that would misdirect an on-call engineer during a
real incident:

- Sec 3 and 4.1 cited a nonexistent `POST /api/providers/{id}/disable`
  endpoint. The real mechanism is per-connection:
  `PUT /api/providers/{connectionId}` with `{ "isActive": false }`
  (src/app/api/providers/[id]/route.ts). There is no single whole-provider
  kill switch, so the steps now say to repeat per connection/key, or rely
  on the automatic provider circuit breaker / Model Lockout described in
  docs/architecture/RESILIENCE_GUIDE.md. Also drops the equally fabricated
  "disable path" pointer at src/lib/a2a/skills/providerDiscovery.ts, which
  has no such function.
- Sec 4.3 cited a `policies_active` field on GET /api/settings/authz-inventory
  that does not exist; the route actually returns a route-tier inventory
  (tiers/bypassEnabled/bypassPrefixes/spawnCapablePrefixes/cors). Rewrote
  the check against the real shape and added a fallback signal
  (JWT_SECRET/API_KEY_SECRET + isValidApiKey's DB reachability) for a
  genuine all-keys auth outage.
- Stripped leftover "phenotype" branding (phenotype.slack.com,
  grafana.phenotype.internal, status.phenotype.dev, announce@phenotype.dev)
  copy-pasted from another org's template, replacing with explicit TBD
  placeholders rather than inventing new unverified URLs.
- Fixed the fabricated ADR-024/ADR-029 citations — this repo has no ADR
  directory; pointed at the real convention in
  docs/architecture/cluster-decisions.md (ADR-041) instead.
- Fixed the #omnirouse-ops-handoff typo -> #omniroute-ops-handoff.
- Added a changelog.d/ fragment per CONTRIBUTING.md convention.

Docs-only change; no production code touched.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Co-authored-by: KooshaPari <kooshapari@users.noreply.github.com>
Co-authored-by: Diego Rodrigues de Sa e Souza <8016841+diegosouzapw@users.noreply.github.com>
Co-authored-by: growab <nekron@icloud.com>
Co-authored-by: KooshaPari <1000+KooshaPari@users.noreply.github.com>

2026-07-18 15:13:43 -03:00

9.3 KiB

Raw Blame History

Incident Response Runbook — OmniRoute (2026-06-18)

Status: Authoritative. The 71-pillar audit (L61) references this doc for the Obs > 2.00 gate. Owner: observability-circle (lead: security-circle lead). SLOs: see docs/PERF_BUDGETS.md § 1 (top-level SLOs) and ops/slos.yaml (machine-readable form, generated by the Bifrost team). Disclosure policy: see SECURITY.md (vulnerability disclosure only, separate flow).

This runbook is the operational playbook for non-security incidents: outages, latency regressions, error-budget burn, and provider-side failures. Vulnerability disclosure stays on SECURITY.md; do not route those through this runbook.

1. Severity ladder

Sev	Definition	Examples	Page on	Resolve by
SEV-1	User-visible outage; > 50 % of requests failing or > 2x SLO breach for 5 min.	Cluster down; auth layer broken; 5xx flood.	On-call P0 (immediate)	4 h
SEV-2	Significant degradation; 1.5–2x SLO breach for 15 min, or single-tenant impact.	Single provider down; p95 > 1.5x budget; rate-limit runaway.	On-call P1 (15 min)	24 h
SEV-3	Latent bug or near-miss; no current user impact but error budget at risk.	Memory leak trending up; circuit breaker tripping on one provider.	Slack `#omniroute-ops` (next standup)	7 d
SEV-4	Cosmetic / informational.	Log line noise; non-binding UI glitch.	Next weekly review	Next refactor cycle

Burn-rate escalation (per docs/PERF_BUDGETS.md § 1): 6x for 5 min is SEV-1; 2x for 1 h is SEV-2; sustained < 1x for 7 d demotes to SEV-3.

2. Detection sources

Source	Signal	Routing
Prometheus (`/metrics`)	Counter deltas (5xx, latency)	Alertmanager → PagerDuty
Grafana SLO dashboards	SLO burn-rate panels	Slack `#omniroute-ops`
Uptime probe (`/api/health/ping`)	3 consecutive failures from 3 regions	Alertmanager → PagerDuty
Dependabot	New CVE in dependency	GitHub issue + Slack `#security`
User report (support@)	Manual triage	Slack `#omniroute-triage`
Error budget burn alert	`slo_burn_rate > threshold`	Alertmanager

Prometheus and Alertmanager are configured in the deploy repo (see docs/operations/DEPLOY.md once published; currently inline in docker-compose.prod.yml).

3. First-15-minutes checklist

When paged, the on-call engineer runs this checklist verbatim. Do not skip steps; each is timed.

0:00 — Acknowledge the page in PagerDuty. Stops the escalation timer and notifies the secondary.
0:02 — Open the [SLO dashboard][dash] and the [incident channel][chan] (#inc-YYYY-MM-DD-slug). Post a single-line ack with the alert name and the time.
0:05 — Classify severity per § 1. If SEV-1 or SEV-2, declare the incident in the channel and tag @incident-commander.
0:08 — Capture the alert payload, the most recent deploy SHA, and the top 5 slow / erroring endpoints. Post to the channel.
0:12 — Decide: mitigate first, root-cause later. Choose one of:
- Roll back to the last green deploy (bin/rollback.sh vX.Y.Z).
- Failover to the healthy replicas (Caddy LB removes the bad replica automatically; verify with curl /api/health/ping).
- Disable the broken connection(s) via PUT /api/providers/{connectionId} with body { "isActive": false } (per-connection toggle, safe by default; repeat per key/account — see § 4.1).
0:15 — Post the chosen mitigation in the channel. If the page is still firing after 5 more minutes, escalate to the secondary.

[chan]: TBD — set to your team's incident-chat channel (e.g. a Discord/Slack #inc-* channel); not provisioned by this repo. [dash]: TBD — set to your Grafana/observability dashboard URL; not provisioned by this repo.

4. Mitigation runbooks (per failure mode)

4.1 Provider outage (single provider down)

PUT /api/providers/{connectionId} with body { "isActive": false } — deactivates that connection; combo routing and account selection skip it on the next request (src/app/api/providers/[id]/route.ts). There is no single whole-provider kill switch — if the provider has more than one key/account, repeat per connection, or let the automatic provider circuit breaker trip on its own (src/shared/utils/circuitBreaker.ts, domain_circuit_breakers table; see docs/architecture/RESILIENCE_GUIDE.md).
Verify p95 returns to budget within 5 min.
If all connections for a model are down, apply the same isActive: false toggle to every connection offering that model — there is no separate per-model disable endpoint. Combo routing's automatic Model Lockout (open-sse/services/accountFallback.ts; see docs/architecture/RESILIENCE_GUIDE.md) also skips a model that keeps erroring, without manual action.
Update the status page (if one is configured — see § 5) with a banner if the outage exceeds 15 min.

4.2 Cluster-wide latency regression

Check the most recent deploy (/api/version returns the SHA).
If p95 doubled vs the 7-day baseline, roll back to the prior SHA via bin/rollback.sh.
If the regression is provider-side, see § 4.1.

4.3 Auth layer broken (5xx on /v1/responses for all keys)

Check the authz-inventory endpoint: curl https://api.omniroute.dev/api/settings/authz-inventory | jq. It returns a route-tier inventory (tiers, bypassEnabled, bypassPrefixes, spawnCapablePrefixes, cors — see src/app/api/settings/authz-inventory/route.ts); there is no policies_active field. A non-200 response, or a tiers array that fails to populate, means the settings/DB layer the auth pipeline reads from is down — not just a single bad key.
If the endpoint itself errors or returns malformed data, restore the settings store from the last good backup (bin/restore-policies.sh <sha>).
If the endpoint is healthy but requests still 5xx for every key, verify JWT_SECRET / API_KEY_SECRET are set and unchanged for this deploy, and that isValidApiKey (src/sse/services/auth.ts) can reach the DB.
Roll back if the cause is unclear.

4.4 Data-layer incident (sqlite corruption, audit log gap)

Stop the cluster (docker compose -f docker-compose.prod.yml stop) — preventing further writes is more important than uptime.
Snapshot the data volume (bin/snapshot-data.sh).
Open a SEV-1; this is data-loss territory. Page the data-team.
Restore from the last verified backup (see docs/BACKUP.md once published; currently the runbook is bin/restore-data.sh <sha>).

4.5 Security incident (vulnerability disclosure)

Stop. This is the SECURITY.md path, not this runbook. Page the security on-call (@security-team); do not post details to #omniroute-ops.

5. Communication

Audience	Channel	Cadence	Owner
Engineering	`#inc-YYYY-MM-DD-slug`	Real-time	Incident commander
Status page	TBD — not provisioned by this repo	Every 30 min during SEV-1/2	On-call
Customers (email)	TBD — set your announcement list/address	At SEV-1 start + resolution	Comms lead
Upstream providers	Direct contact	At SEV-1 start	Vendor mgmt
Postmortem	`docs/postmortem/YYYY-MM-DD-slug.md`	Within 5 business days	Incident commander

Postmortem template is at docs/postmortem/TEMPLATE.md (forthcoming; no dedicated ADR covers it yet — once written, register it in docs/architecture/cluster-decisions.md following this repo's 71-pillar/ADR numbering convention, e.g. ADR-041 there).

6. On-call rotation

Role	Primary	Secondary	Rotation
Engineering on-call	security-circle lead	@open-sse	Weekly, Mon 09:00 PDT
Security on-call	@security-team	—	Weekly
Data on-call	@db-team	—	Weekly
Comms lead	@comms	—	As needed

Handoff: every Monday 09:00 PDT, the outgoing on-call posts a written handoff to the incoming in #omniroute-ops-handoff covering: open SEV-3/4 items, scheduled maintenance windows, and any in-flight mitigations.

7. Postmortem expectations

Blameless. People did the best they could with the information they had. Focus on systems, signals, and decision points.
Within 5 business days of resolution. File via gh issue create --label postmortem --label SEV-1 (or --label SEV-2).
Action items must be assigned, dated, and tracked in docs/TECH_DEBT.md (P0 < 30 d, P1 < 90 d per that doc's SLA).
Mandatory attendees: incident commander, on-call, any engineer who touched the mitigation, and one person who was not involved (fresh-eyes review).

8. Review log

Date	Reviewer	Change
2026-06-18	security-circle lead	Initial runbook; severity ladder + 15-min checklist + 4.1–4.5 mitigation runbooks. Closes 71-pillar audit L61 (1/3 → 2/3).
2026-07-18	observability-circle	Corrected § 4.1/4.3 to the real provider-disable (`PUT /api/providers/{connectionId}`) and authz-inventory (`tiers`/`bypassEnabled`/`cors`, no `policies_active`) mechanisms; removed foreign branding and the nonexistent ADR-024/029 references.
2026-07-18 (planned)	observability-circle	Wire on-call rotation into PagerDuty schedule; add the postmortem template.

9.3 KiB Raw Blame History Unescape Escape