V2

2026-08-22 12:49:53 +02:00 · 2026-03-24 18:02:50 -04:00
parent 2d209b9258
commit ba9e489584
111 changed files with 48382 additions and 3099 deletions
@@ -11,6 +11,22 @@ metadata:

 Apply SRE rigor to AI systems where incidents include quality regressions, unsafe outputs, and budget explosions.

+## When to Use This Skill
+
+- An LLM endpoint begins returning degraded or hallucinated answers
+- Token spend spikes beyond budget thresholds
+- A model provider goes down and traffic must fail over
+- Safety guardrails fire at abnormal rates
+- A new model deployment causes latency or accuracy regression
+
+## Prerequisites
+
+- Prometheus and Alertmanager deployed with scrape targets for AI services
+- Grafana dashboards for golden signals (latency, error rate, cost, quality)
+- On-call rotation configured in PagerDuty, Opsgenie, or equivalent
+- Runbook repository accessible to responders
+- Rollback mechanism for model and prompt versions (GitOps or feature flags)
+
 ## AI Incident Classes

 - **Availability incident**: model/provider unavailable, timeout storm.
@@ -18,11 +34,14 @@ Apply SRE rigor to AI systems where incidents include quality regressions, unsaf
 - **Safety incident**: harmful or policy-violating outputs increase.
 - **Cost incident**: unexpected token or provider spend spike.

-## Severity Framework (Example)
+## Severity Framework

- **SEV1**: user-facing outage, critical compliance risk, or active data leak.
- **SEV2**: major degradation affecting key flows.
- **SEV3**: limited impact or internal-only issue.
+| Severity | Criteria | Response Time | Notification |
+|----------|----------|---------------|--------------|
+| SEV1 | User-facing outage, compliance risk, data leak | 5 min | Page on-call + incident commander |
+| SEV2 | Major degradation in key flows | 15 min | Page on-call |
+| SEV3 | Limited impact or internal-only issue | 1 hour | Slack alert |
+| SEV4 | Cosmetic or low-priority regression | Next business day | Ticket |

 ## Golden Signals for AI Services

@@ -32,25 +51,200 @@ Apply SRE rigor to AI systems where incidents include quality regressions, unsaf
 - Cost per minute and per tenant
 - Guardrail violation rate

+## Prometheus Alert Rules
+
+```yaml
+# prometheus-ai-alerts.yaml
+groups:
+  - name: ai-service-alerts
+    rules:
+      - alert: ModelEndpointDown
+        expr: up{job="llm-inference"} == 0
+        for: 2m
+        labels:
+          severity: sev1
+        annotations:
+          summary: "LLM inference endpoint {{ $labels.instance }} is down"
+          runbook_url: "https://runbooks.internal/ai/model-outage"
+
+      - alert: HighHallucinationRate
+        expr: |
+          rate(llm_hallucination_detected_total[10m])
+          / rate(llm_requests_total[10m]) > 0.15
+        for: 5m
+        labels:
+          severity: sev2
+        annotations:
+          summary: "Hallucination rate above 15% for {{ $labels.model }}"
+          runbook_url: "https://runbooks.internal/ai/quality-regression"
+
+      - alert: TokenCostExplosion
+        expr: |
+          sum(rate(llm_token_cost_dollars[5m])) by (tenant)
+          > 0.50
+        for: 3m
+        labels:
+          severity: sev2
+        annotations:
+          summary: "Token spend exceeds $0.50/min for tenant {{ $labels.tenant }}"
+          runbook_url: "https://runbooks.internal/ai/cost-spike"
+
+      - alert: LatencyP95Exceeded
+        expr: |
+          histogram_quantile(0.95,
+            rate(llm_request_duration_seconds_bucket[5m])
+          ) > 5
+        for: 5m
+        labels:
+          severity: sev2
+        annotations:
+          summary: "LLM p95 latency exceeds 5s for {{ $labels.service }}"
+
+      - alert: GuardrailViolationSpike
+        expr: |
+          rate(llm_guardrail_violations_total[10m])
+          / rate(llm_requests_total[10m]) > 0.05
+        for: 5m
+        labels:
+          severity: sev1
+        annotations:
+          summary: "Guardrail violations above 5% for {{ $labels.model }}"
+          runbook_url: "https://runbooks.internal/ai/safety-incident"
+
+      - alert: ModelQualityDrop
+        expr: |
+          llm_eval_score{metric="groundedness"} < 0.70
+        for: 10m
+        labels:
+          severity: sev2
+        annotations:
+          summary: "Groundedness score dropped below 0.70 for {{ $labels.model }}"
+
+      - alert: ProviderErrorRateHigh
+        expr: |
+          rate(llm_provider_errors_total[5m])
+          / rate(llm_provider_requests_total[5m]) > 0.10
+        for: 3m
+        labels:
+          severity: sev2
+        annotations:
+          summary: "Provider {{ $labels.provider }} error rate above 10%"
+```
+
 ## Response Playbooks

-### Model Outage
-1. Freeze deployments.
-2. Shift traffic to fallback model/provider.
-3. Enforce stricter rate limits.
-4. Communicate ETA and mitigation.
+### Model Outage Runbook

-### Quality Regression
-1. Roll back prompt/model version.
-2. Disable risky optimization flags.
-3. Increase sampling for trace review.
-4. Re-run latest eval baseline.
+```text
+TRIGGER: ModelEndpointDown fires for > 2 minutes
+RESPONDER: On-call AI platform engineer

-### Cost Spike
-1. Identify top tenants/routes/models.
-2. Enable cache + cheaper fallback path.
-3. Apply temporary token caps.
-4. Open postmortem with prevention actions.
+1. Acknowledge alert in PagerDuty.
+2. Check provider status page (e.g., status.openai.com).
+3. Verify network connectivity:
+     curl -s -o /dev/null -w "%{http_code}" https://api.provider.com/health
+4. If provider is down:
+     a. Enable fallback model route in gateway config.
+     b. kubectl set env deployment/llm-gateway FALLBACK_ENABLED=true
+     c. Verify fallback traffic is flowing via Grafana dashboard.
+5. If self-hosted model is down:
+     a. Check pod status: kubectl get pods -l app=llm-inference -n ai
+     b. Check GPU health: kubectl logs -l app=llm-inference --tail=50
+     c. Restart if OOM: kubectl rollout restart deployment/llm-inference -n ai
+6. Freeze all deployments:
+     kubectl annotate deployment --all deploy-freeze=true -n ai
+7. Communicate ETA in #incident-channel.
+8. When resolved, unfreeze and run smoke tests.
+```
+
+### Quality Regression Runbook (Hallucination Spike)
+
+```text
+TRIGGER: HighHallucinationRate or ModelQualityDrop fires
+RESPONDER: On-call AI engineer + ML lead
+
+1. Acknowledge alert. Open incident ticket.
+2. Identify scope:
+     - Which model version? Check deployment metadata.
+     - Which routes/tenants affected? Filter by labels in Grafana.
+3. Check recent changes:
+     - Model version promotion in last 24h?
+     - Prompt template changes in last 24h?
+     - Retrieval index rebuild in last 24h?
+4. If recent model change:
+     kubectl rollout undo deployment/llm-inference -n ai
+5. If recent prompt change:
+     git revert <commit> && git push  # triggers GitOps redeploy
+6. Increase trace sampling to 100% for affected route:
+     kubectl set env deployment/llm-gateway TRACE_SAMPLE_RATE=1.0
+7. Run offline eval suite against current production:
+     python run_evals.py --target prod --suite quality --compare baseline
+8. Confirm metrics return to baseline before closing.
+```
+
+### Token Cost Explosion Runbook
+
+```text
+TRIGGER: TokenCostExplosion fires
+RESPONDER: On-call platform engineer
+
+1. Identify top consumers:
+     Query: topk(10, sum(rate(llm_token_cost_dollars[15m])) by (tenant, model, route))
+2. Check for runaway loops:
+     - Agent retry storms (exponential token growth per request)
+     - Missing max_tokens caps on new routes
+     - Cache bypass due to config change
+3. Apply immediate caps:
+     kubectl patch configmap llm-quotas -n ai --patch '
+       data:
+         max_tokens_per_request: "4096"
+         rpm_limit: "60"
+     '
+4. Enable semantic cache if disabled:
+     kubectl set env deployment/llm-gateway CACHE_ENABLED=true
+5. Route traffic to cheaper model tier:
+     kubectl set env deployment/llm-gateway DEFAULT_MODEL=gpt-4o-mini
+6. Notify affected tenants of temporary limits.
+7. Open postmortem with cost attribution analysis.
+```
+
+## Escalation Procedures
+
+```text
+Level 1 (0-15 min):  On-call AI platform engineer
+Level 2 (15-30 min): AI platform team lead + affected product owner
+Level 3 (30-60 min): Engineering director + security (if safety incident)
+Level 4 (60+ min):   VP Engineering + legal (if compliance/data incident)
+
+Safety incidents always start at Level 2 minimum.
+Provider-side incidents: open support ticket immediately at Level 1.
+```
+
+## Detection Queries (PromQL)
+
+```promql
+# Request success rate by model
+1 - (
+  sum(rate(llm_requests_total{status="error"}[5m])) by (model)
+  / sum(rate(llm_requests_total[5m])) by (model)
+)
+
+# Cost per successful answer
+sum(rate(llm_token_cost_dollars[5m])) by (route)
+/ sum(rate(llm_requests_total{status="success"}[5m])) by (route)
+
+# Hallucination rate trend (1h window, 5m steps)
+rate(llm_hallucination_detected_total[1h])
+/ rate(llm_requests_total[1h])
+
+# Latency breakdown by stage
+histogram_quantile(0.95, rate(llm_retrieval_duration_seconds_bucket[5m]))
+histogram_quantile(0.95, rate(llm_generation_duration_seconds_bucket[5m]))
+histogram_quantile(0.95, rate(llm_tool_execution_duration_seconds_bucket[5m]))
+
+# Tenant cost leaderboard
+topk(10, sum(rate(llm_token_cost_dollars[1h])) by (tenant))
+```

 ## Postmortem Requirements

@@ -58,9 +252,60 @@ Apply SRE rigor to AI systems where incidents include quality regressions, unsaf
 - Blast radius by tenant and feature
 - Missed signals and alert tuning actions
 - Concrete hardening tasks with owners and due dates
+- Cost impact (dollars, tokens, affected requests)
+- Customer communication log
+
+## Postmortem Template
+
+```markdown
+## Incident Summary
+- **Severity**: SEVx
+- **Duration**: start_time - end_time (Xh Ym)
+- **Detection**: How was it detected? (alert / customer report / manual)
+- **Impact**: X tenants, Y requests, $Z cost
+
+## Timeline
+| Time (UTC) | Event |
+|------------|-------|
+| HH:MM | Alert fired |
+| HH:MM | Responder acknowledged |
+| HH:MM | Root cause identified |
+| HH:MM | Mitigation applied |
+| HH:MM | Incident resolved |
+
+## Root Cause
+[Description]
+
+## Action Items
+| Action | Owner | Due Date | Status |
+|--------|-------|----------|--------|
+| Tune alert threshold | @engineer | YYYY-MM-DD | Open |
+| Add fallback route | @platform | YYYY-MM-DD | Open |
+```
+
+## Chaos Engineering for AI Systems
+
+Regularly test incident readiness:
+
+- **Provider failover drill**: block provider API at network level, verify fallback activates within SLO.
+- **Model rollback drill**: deploy known-bad model version, verify automated quality gate catches it.
+- **Cost cap drill**: simulate runaway token usage, verify quotas trigger before budget threshold.
+- **Cache failure drill**: disable semantic cache, verify system degrades gracefully.
+
+## Troubleshooting
+
+| Symptom | Check | Fix |
+|---------|-------|-----|
+| All requests timing out | Provider status page, DNS resolution | Enable fallback provider |
+| Gradual quality decline | Recent model/prompt deployments | Roll back to last known good |
+| Sudden cost spike | Per-tenant token usage dashboard | Apply emergency token caps |
+| Guardrail violations spike | Model version, prompt injection logs | Enable stricter input filtering |
+| Intermittent 503 errors | Pod restarts, GPU OOM events | Increase memory limits or reduce batch size |

 ## Related Skills

 - [incident-response](../../../security/operations/incident-response/) - Standard incident process and evidence
 - [alerting-oncall](../../observability/alerting-oncall/) - Paging and escalation policy
 - [llm-cost-optimization](../llm-cost-optimization/) - Spend controls and efficiency patterns
+- [agent-observability](../agent-observability/) - Instrument requests, traces, and costs
+- [rag-observability-evals](../rag-observability-evals/) - RAG quality monitoring