Operational Runbook & P1 Incident Response Playbook

# Operational Runbook: P1 Incident Response Playbook

## 1. Incident Severity Definitions
- **P1 (Critical Outage):** Core service unavailable for > 1% of users, data corruption risk, or security perimeter breach.
- **P2 (Major Degradation):** Non-critical feature failure with available workaround; elevated API latency > 500ms.
- **P3 (Minor Defect):** Cosmetic UI defect or non-blocking edge-case error.

---

## 2. Immediate Triage Protocol (First 15 Minutes)
1. **Declare Incident:** Incident Commander posts notification in `#incident-response` channel.
2. **Assign Roles:**
   - **Incident Commander (IC):** Directs triage and coordinates technical leads.
   - **Operations Lead:** Executes diagnostics, rolling restarts, or traffic diversion.
   - **Communications Lead:** Updates external status page every 20 minutes.
3. **Establish Isolation:** If database corruption or security intrusion suspected, trigger immediate circuit breaker isolation on affected ingress routes.

---

## 3. Rapid Diagnostic Commands
```bash
# 1. Inspect active ingress error rates
kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx --tail=500 | grep "50[0-4]"

# 2. Check database connection pool health
psql -c "SELECT count(*), state FROM pg_stat_activity GROUP BY state;"

# 3. Check consumer group lag on Kafka
kafka-consumer-groups --bootstrap-server kafka:9092 --describe --group order-processors
```